Compare commits

...
Author SHA1 Message Date
mesh-tools 475bb4bb0b Merge pull request 'The view's tools read their own reach by its bare name, and a test holds every state the commands read to what is issued (issue 1280)' (#82) from fix/1280-view-reads-its-reach into main 2026-10-26 22:48:25 +00:00
jschoubben efadc691e7 The view's tools read their own reach by its bare name, as the mesh issues it, and a test holds every state the commands read to what is issued (issue 1280)
mesh/check being carried to efadc691e7ea on 475bb4bb0b8f | from 4af36c69a761
mesh/delivery merged: source.main moved to novox/mesh-delivery@475bb4bb0b8f
view_reach asked for mesh-delivery-view.reach, a name the mesh never issued, so it failed on the machines;
the same line was not gofmt'd, which failed the try of main. Quick fix: the name. Thorough for this module: the
manifest test now refuses any stdio.State literal in cmd/ that is neither an own state nor listed in reads.
2026-10-26 23:27:05 +01:00
mesh-tools 4af36c69a7 Merge pull request 'Delivery records are true at the source: the indexer records what it took in, the view derives what reached the machines, a settle verb for held deliveries (issue 1280)' (#80) from fix/delivery-records-true into main 2026-10-26 21:48:50 +00:00
mesh-tools bd107d2015 Merge pull request 'A version that fails on one machine while another runs it healthy is that machine's failure, and goes no further meanwhile (issue 1496, ADR 0557 §1-§3)' (#81) from fix/1496-start-health into main 2026-10-26 21:37:58 +00:00
jschoubben f007ca4227 A node that runs a version alone-failed elsewhere still gets its want at it (ADR 0557 §2, issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bd107d20154c
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking f007ca42279a on 4b36cc29a6ae | novox/mesh-delivery#81 in group fix/1496-start-health
Review of 50c4674: the bar on new wants naming F while a node fails it alone
also caught a node that runs F with no live want, the issue-1945 recovery path:
it went on neither list and its node-engine gave the module up. The bar now
holds only a node that neither wants nor runs F.
2026-10-26 22:34:25 +01:00
jschoubben 50c4674dfa While a node fails a version alone, no further node gets a want for it (ADR 0557 §2, issue 1496)
ADR 0557 version 9: a failure that is one node's holds nothing and sets nothing
back, but the version reaches no node that does not want it yet until it is
over. A node whose want does not name F gets no new want naming it (rule 2, in
words naming the failing nodes); wants already written stay.
2026-10-26 22:30:00 +01:00
jschoubben c12b870104 The indexer binds its taken state when it appears, as issue 1790 asks of every state a delivery adds: an indexer started before the bus holds it still takes merges in and says the key it did not write (issue 1280, ADR 0556 clause 1)
mesh/check checking c12b870104e8 on bd107d20154c | novox/mesh-delivery#80 in group fix/delivery-records-true
mesh/delivery merged: source.main moved to novox/mesh-delivery@4af36c69a761
mesh/delivery-group group fix/delivery-records-true merged: every member is merged
2026-10-26 22:25:39 +01:00
jschoubben 92aedb9365 The first pass after a start clears failed-alone for every module and known node (ADR 0557, issue 1496)
Review round 2 of 16e4e61: gating the first-pass clear on a failed have left a
condition an earlier process raised standing for good once its node recovered
while no assigner watched. The first pass now clears it for every module and
every node the presence bucket knows, a quiet clear at the router for one that
is not open.
2026-10-26 22:21:00 +01:00
jschoubben 368204115c Pin the teller at mesh-sdk 68d65e6, the group's sdk head merged with sdk main (issue 1280) 2026-10-26 22:20:45 +01:00
jschoubben 971cdb44e2 Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-indexer/cmd/mesh-delivery-indexer/main.go
#	modules/mesh-delivery-indexer/go.mod
#	modules/mesh-delivery-indexer/go.sum
#	modules/mesh-delivery-indexer/manifest_test.go
#	modules/mesh-delivery-indexer/module.json
#	modules/mesh-delivery-view/cmd/mesh-delivery-view-tools/main.go
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
#	modules/mesh-delivery-view/internal/view/read.go
#	modules/mesh-delivery-view/manifest_test.go
#	modules/mesh-delivery-view/module.json
2026-10-26 22:20:45 +01:00
jschoubben 16e4e61a0f Only an online healthy witness, and only a first failure, keep a version from being held; failed-alone clears when a node leaves (ADR 0557, issue 1496)
Review of 2ee7589: a node that sleeps after running F healthy kept its stale
have as a witness, so F stayed unheld while it failed on the online nodes; and a
failed-alone condition was never cleared once its node left the module.

A witness must now be online by the watchman's presence. F failing on a second
node is no longer one node's failure: the hold and the set-back stand (ADR 0557
version 6, §2). A remembered failed-alone key whose node is no longer one of the
module's nodes is cleared once; the first pass after a start clears it for every
node the presence bucket knows, for a module with a failed have.
2026-10-26 22:16:38 +01:00
mesh-tools 4b36cc29a6 Merge pull request 'Placement from the catalogue: the bridge carries it, the indexer writes it, the assigner compares it (issue 1126)' (#76) from feat/1126-nodes-compose into main 2026-10-26 21:11:26 +00:00
mesh-tools 31ddd09087 Merge pull request 'The checker ends a check whose heads no candidate holds, and starts a group's check once its heads settle (issue 1890)' (#79) from fix/1890-end-superseded-checks into main 2026-10-26 21:10:56 +00:00
jschoubben 68173de8bd Pin the indexer, the teller and the view at mesh-sdk b4310df: the delivery-records work merged with sdk main (issue 1280) 2026-10-26 22:08:40 +01:00
jschoubben 960e3309bc Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 22:07:28 +01:00
jschoubben 2ee7589474 A version healthy on another node is not held for one node's failed health check (ADR 0557, issue 1496)
On ace, 2026-10-26 21:19, a newly assigned mesh-build-node failed its first
health check at 8a98386f (later 31be2f24) while shanks, g14 and novox had run it
healthy for an hour. Being first by name, ace was the canary, and the assigner
held the version and moved the three healthy nodes back.

A failed health check of the newest fingerprint now holds it only when no other
node's have names it healthy (ADR 0557 §2 keeps ADR 0429 §6's hold and set-back
for that case). Otherwise the failure is that node's (ADR 0557 §1): nothing is
held or set back, the node keeps its want, and the assigner raises the node's
condition once (failed-alone.<node>.<module>), naming the node, the module, the
fingerprint and the have's words, and clears it once when it ends; the first
pass after a start clears what a previous process may have raised.
2026-10-26 22:06:32 +01:00
jschoubben 297992a4ae Pin mesh-sdk 9796444 (l2-next: feat/1126-nodes-compose merged with sdk main #101, the standing-test race fixes) (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4b36cc29a6ae
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 297992a4ae55 on 4b36cc29a6ae | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 22:05:41 +01:00
jschoubben 3f85230417 Pin mesh-sdk 5f45175 (l2-next: feat/1126-nodes-compose merged with sdk main) after merging main (issue 1126) 2026-10-26 21:59:21 +01:00
jschoubben 43c3158bc6 Merge remote-tracking branch 'origin/main' into feat/1126-nodes-compose
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 21:57:03 +01:00
jschoubben d95e9f3229 End a check whose heads no candidate holds, and start a group's check once its heads settle (issue 1890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@31ddd090876f
mesh/check pass d95e9f322913 on d581f9659a78 | unit 191 run, 525 kept | integration 201 run, 591 kept | composition passed on novox
2026-10-26 21:44:12 +01:00
mesh-tools d581f9659a Merge pull request 'Tries record their asks; the try limit counts tries on a build node; build_usage shows capacity, hours and steps (issue 1890)' (#78) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:34 +00:00
jschoubben ea94f9234c Merge remote-tracking branch 'origin/main' into fix/capacity-follows-cpu-share
mesh/delivery merged: source.main moved to novox/mesh-delivery@d581f9659a78
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking ea94f9234c97 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
2026-10-26 21:29:44 +01:00
jschoubben c19468b04e Pin the checker and the view at mesh-sdk 805b4d9: the capacity work merged with sdk main 2026-10-26 21:29:44 +01:00
jschoubben c47afd8f4b build_usage says a seat's build hours, and when it is outside them
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group fix/capacity-follows-cpu-share checking: 1 of 3 member(s) ready
mesh/check checking c47afd8f4b88 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
A build module's capacity.<node> now carries its node's build window (the operator, 2026-10-26: g14
builds only from 18:00 to 06:00); the seat shows it, outside it says it takes nothing new, and a
value the module cannot read shows as any hour with the value and why. The checker's and the view's
SDK pin moves to mesh-sdk d00dac1.
2026-10-26 21:22:36 +01:00
jschoubben afd5a3ba07 A module a node wants and runs nothing countable of holds no head back; the reached head's event names its merges as not yet on the machines; pin mesh-sdk e44cbaf (review of issue 1280, ADR 0556 clause 2) 2026-10-26 21:15:31 +01:00
jschoubben 7dbad4c6d2 Pin mesh-sdk 33144781 (taken, reach, settle layouts) in the indexer, the view and the teller (issue 1280) 2026-10-26 21:05:29 +01:00
jschoubben 083a5042f6 The view reads the wants by module once per derivation of the reach 2026-10-26 21:03:56 +01:00
jschoubben 351ca7c290 The teller checks a settle ask's warrant against mesh-delivery.settle, the act its options bind (issue 1280, ADR 0556 clause 5) 2026-10-26 20:58:20 +01:00
jschoubben f907be2c14 The view derives per repository up to which merge every node runs it, keeps it as its state reach and says each reached head, so a fix on the machines reads delivered (issue 1280, ADR 0556 clauses 2 and 3) 2026-10-26 20:56:12 +01:00
jschoubben 6201f58c20 The indexer says per repository the last merge it took in, so the view knows its modules speak for that merge (issue 1280, ADR 0556 clause 1) 2026-10-26 20:49:00 +01:00
jschoubben 8fee764c73 Pin mesh-sdk feat/1126-nodes-compose 03dce12, the group's head: the checker refuses a pin that is neither trunk nor the group's head (issue 1126)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 8fee764c734e on d581f9659a78 | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:35:45 +01:00
mesh-tools 3dd94f714d Merge pull request 'The checker fails any repository's change that adds a binary, in any commit (issue 1577)' (#77) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:27 +00:00
jschoubben 352066dde6 Review of L2: the assigner binds the indexer's placement state when it appears, with no restart (D1); the three new unit tests folded into existing ones, so no count rises (D2); pin mesh-sdk 3968849 (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 352066dde611 on 3dd94f714d6e | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:12:00 +01:00
jschoubben d6bd31a166 Say the try limits as what they are: a check when a try is asked
Tries let in while waiting may later all run at once; each node's build capacity bounds what runs
on it (review of issue 1890's try cap).
2026-10-26 20:11:08 +01:00
jschoubben c0a4b139a2 A try queued behind the build seat's checks no longer blocks the next try
Only tries a build node holds count toward the 4 at once; tries waiting for a node are bounded
apart at 8, and the refusal names both counts (issue 1890: at 19:53 four tries queued behind 44
checks for up to 47 minutes refused every further try).
2026-10-26 20:05:51 +01:00
jschoubben 6d60953f98 A running try names its asks and where each stands; build_usage says a seat's CPU-bound capacity
The checker wrote a try's asks only at its end, so a 31-minute try could not be told queued from
running (issue 1890). The try's record now names each ask as it is put, then the node that took
it, its state and the phase its step says, written on the step key's own changes. build_usage
names a running try's asks and takes a build module's capacity.<node> (configured bounded by its
node's build CPU, issue 1806's follow-up) when it is current with mesh-config.
2026-10-26 19:58:39 +01:00
jschoubben d74e082155 Checker refuses a program a commit of the branch added and a later one removed: it stays in history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-delivery@3dd94f714d6e
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking d74e0821552c on df8b9ac3b1b5 | novox/mesh-delivery#77 in group fix/1577-checker-refuses-binaries
2026-10-26 19:47:53 +01:00
jschoubben 2e0013ec25 The assigner's wants test goes on to placement: a placed node's want names the build alone and its own want says it composes its own (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2e0013ec2500 on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:44:55 +01:00
jschoubben 2493aa6529 Pin mesh-sdk feat/1126-nodes-compose 9713c30: placement carries a settings-form module's values (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2493aa65299e on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:40:38 +01:00
jschoubben 85c89b16ef The grants part composes what the catalogue's placement places beside the assignments, and holds a seat held once for the mesh where placement says (issue 1126, ADR 0549 §3, G5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:33:27 +01:00
jschoubben 4f75925410 Nodes compose: placement from the catalogue, node records, self wants, own and pair credentials, core boxes and the authority from the vault (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
One commit for the branch feat/1126-nodes-compose of mesh-delivery: its 16 commits squashed so its history does not carry the 17.8 MB compiled mesh-delivery-assigner a commit added by mistake and a later one removed (issue 1577). The tree equals the former head ff6c038d. The operator allowed this rewrite of the feature branch on 2026-10-26.
2026-10-26 19:30:18 +01:00
jschoubben 820c458f4c The captured clean fetches carry the programs a current build module answers, so lines asserted whole do not warn (issue 1577) 2026-10-26 19:16:25 +01:00
jschoubben dc9d65524d A check judged by a build module that reports no programs says so in its status line, still passing: the refusal did not run (issue 1577) 2026-10-26 19:08:00 +01:00
jschoubben 1a4c788ed7 Checker fails a change whose fetch reports a compiled program, and warns when a build module reports none: build output kept getting committed (issue 1577) 2026-10-26 19:01:40 +01:00
81 changed files with 3086 additions and 176 deletions
+3
View File
@@ -1 +1,4 @@
vendor/ vendor/
# A compiled program never enters a commit (issue 1577): `go build .` in a command directory leaves one there.
modules/*/cmd/*/mesh-*
!modules/*/cmd/*/mesh-*.go
@@ -172,6 +172,9 @@ func TestTheAssignerStartsAndSaysWhatItKeeps(t *testing.T) {
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket. // 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) { func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets { for _, missing := range readBuckets {
if missing == delivery.PlacementBucket {
continue // it starts without it, below
}
b := startBed(t, missing) b := startBed(t, missing)
select { select {
case err := <-b.start(t): case err := <-b.start(t):
@@ -218,6 +221,34 @@ func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
t.Fatal("the put was not taken further once its bucket was made") t.Fatal("the put was not taken further once its bucket was made")
} }
} }
// Review of L2 (D1): without the indexer's placement state it starts all the same, and once the bucket is made (the
// indexer and the assigner come in one delivery) it is read with no restart: a module only the placement names is
// said as placed by one source only.
pb := startBed(t, delivery.PlacementBucket)
pdone := pb.start(t)
select {
case err := <-pdone:
t.Fatalf("without %s the assigner stopped: %v", delivery.PlacementBucket, err)
case <-time.After(2 * time.Second):
}
placement, err := pb.js.CreateKeyValue(pb.ctx, jetstream.KeyValueConfig{Bucket: delivery.PlacementBucket, History: 5})
if err != nil {
t.Fatal(err)
}
pl, _ := delivery.EncodePlacement(delivery.Placement{Node: "n1", Modules: []delivery.PlacedModule{{Module: "web"}}})
plkey, _ := delivery.PlacementKey("n1")
if _, err := placement.Put(pb.ctx, plkey, pl); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(15 * time.Second); ; time.Sleep(200 * time.Millisecond) {
if c, ok := pb.conditionsSaid(t)[delivery.AssignerModule+".placement.n1.web"]; ok && c.State == asks.StateRaised {
break
}
if time.Now().After(deadline) {
t.Fatal("the placement state made after the start was not read")
}
}
} }
// putJSON writes a value into one of the bed's buckets. // putJSON writes a value into one of the bed's buckets.
@@ -406,6 +437,32 @@ func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
if !cleared { if !cleared {
t.Errorf("the condition of web on neither list on g14 was not cleared once g14 got its want") t.Errorf("the condition of web on neither list on g14 was not cleared once g14 got its want")
} }
// Issue 1126 (ADR 0547 §2): the catalogue's placement reaches the indexer's state naming web on every node the
// assignments name it on. Each node then composes its own: its want names the build alone (Self, no composition), and
// its own want says it composes its own, which the node agent reads (mesh-host's TestNats...ComposesAndAppliesItsOwn).
for _, node := range []string{"ace", "g14", "shanks"} {
pl, err := delivery.EncodePlacement(delivery.Placement{Node: node, Modules: []delivery.PlacedModule{{Module: "web"}}})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PlacementKey(node)
b.putJSON(t, delivery.PlacementBucket, pkey, pl)
}
if w := b.wantNow(t, "ace", "web", "ace's want once placed: the build alone, composed by the node", func(w delivery.Want) bool { return w.Self }); w.Composed != "" || w.FP != fNew {
t.Errorf("ace's want once placed = %+v", w)
}
nkey, _ := delivery.WantNodeKey("ace")
var self bool
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && !self; time.Sleep(100 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, nkey); err == nil {
nw, err := delivery.DecodeNodeWant(e.Value())
self = err == nil && nw.Self
}
}
if !self {
t.Error("ace's own want does not say it composes its own once placed")
}
} }
// 1297/B9 (the assigner's half, ADR 0528 clause 6): started whole, the assigner holds a fingerprint whose health check // 1297/B9 (the assigner's half, ADR 0528 clause 6): started whole, the assigner holds a fingerprint whose health check
@@ -59,6 +59,7 @@ var readBuckets = []string{
delivery.CandidatesBucket, delivery.CandidatesBucket,
delivery.ManifestsBucket, delivery.ManifestsBucket,
delivery.ModulesBucket, delivery.ModulesBucket,
delivery.PlacementBucket,
delivery.PresenceBucket, delivery.PresenceBucket,
delivery.HaveBucket, delivery.HaveBucket,
delivery.StepsBucket(goBuildModule), delivery.StepsBucket(goBuildModule),
@@ -98,6 +99,12 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err) logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue continue
} }
if name == delivery.PlacementBucket && err != nil {
// The indexer's placement state comes with the second milestone (ADR 0549 §2): without it the controller's
// assignments alone name where modules run, as before any placement was read (ADR 0549 §5).
logf("the bucket %s cannot be opened (%v): the controller's assignments alone are read until the assigner starts again", name, err)
continue
}
if err != nil { if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err) return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
} }
@@ -151,7 +158,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err) logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err)
} }
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs)) logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, kvs, sayVia(ctx, js), emitting{said}, logger) return follow(ctx, js, conn, kvs, sayVia(ctx, js), emitting{said}, logger)
} }
// emitting is the process connection as the puts' caller: its calls, and its events with no options. // emitting is the process connection as the puts' caller: its calls, and its events with no options.
@@ -169,6 +176,7 @@ var watched = []string{
delivery.BuilderBuildsBucket, delivery.BuilderBuildsBucket,
delivery.CandidatesBucket, delivery.CandidatesBucket,
delivery.ModulesBucket, delivery.ModulesBucket,
delivery.PlacementBucket,
delivery.PresenceBucket, delivery.PresenceBucket,
delivery.HaveBucket, delivery.HaveBucket,
delivery.AssignerAsksBucket, delivery.AssignerAsksBucket,
@@ -179,7 +187,7 @@ var watched = []string{
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after // follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends. // each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error { func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error {
w := newWorld() w := newWorld()
dirty := make(chan struct{}, 1) dirty := make(chan struct{}, 1)
wake := func() { wake := func() {
@@ -263,7 +271,12 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
} }
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket) logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
} }
started := 0
for _, name := range watched { for _, name := range watched {
if kvs[name] == nil && name == delivery.PlacementBucket {
continue // not on the bus at this start (open said so)
}
started++
var opts []jetstream.WatchOpt var opts []jetstream.WatchOpt
if name == delivery.HaveBucket { if name == delivery.HaveBucket {
opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back
@@ -275,7 +288,20 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil { if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err return err
} }
for range len(watched) + 1 { // The indexer's placement state arrives with the same delivery as this assigner (ADR 0549 §2): when it is not on the
// bus at this start it is bound once the bus says it was created, and watched from then on, with no restart (issue
// 1790's kind). Until then the controller's assignments alone name where modules run (ADR 0549 §5).
placementAppeared := make(chan jetstream.KeyValue, 1)
if kvs[delivery.PlacementBucket] == nil && js != nil {
process.AwaitBucket(ctx, js, nc, delivery.PlacementBucket, "the controller's assignments alone are read",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- kv })
}
takePlacement := func(kv jetstream.KeyValue) error {
kvs[delivery.PlacementBucket] = kv
logger.Printf("the bucket %s is open: placement is read beside the controller's assignments", delivery.PlacementBucket)
return watch(delivery.PlacementBucket)
}
for range started + 1 {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return nil return nil
@@ -297,6 +323,10 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
case err := <-errs: case err := <-errs:
return err return err
case <-settled: case <-settled:
case kv := <-placementAppeared:
if err := takePlacement(kv); err != nil {
return err
}
case <-dirty: case <-dirty:
reopenPuts(p) reopenPuts(p)
p.pass(ctx) p.pass(ctx)
@@ -94,7 +94,7 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}} kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0)) go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0))
select { select {
case key := <-wrote: case key := <-wrote:
if key != "want.ace.node" { if key != "want.ace.node" {
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"errors" "errors"
"maps"
"slices" "slices"
"strings" "strings"
"time" "time"
@@ -39,6 +40,9 @@ type passer struct {
// noWant is, by node and module, the reason last said why the node gets no want for the module (issue 1945): a reason is // noWant is, by node and module, the reason last said why the node gets no want for the module (issue 1945): a reason is
// said in the journal when it appears or changes, and its clearing when the node has a want again; one line per change. // said in the journal when it appears or changes, and its clearing when the node has a want again; one line per change.
noWant map[string]string noWant map[string]string
// nodeFailed is, by node and module, the words of the condition last raised for a failure that is the node's own (ADR 0557
// §1, issue 1496): raised once per words, cleared once when it ends.
nodeFailed map[string]string
} }
func (p *passer) clock() time.Time { func (p *passer) clock() time.Time {
@@ -60,7 +64,23 @@ func (p *passer) pass(ctx context.Context) {
// controller has composed it and again whenever that digest changes (ADR 0539 §6). // controller has composed it and again whenever that digest changes (ADR 0539 §6).
func (p *passer) nodeWants(ctx context.Context) { func (p *passer) nodeWants(ctx context.Context) {
written := p.w.read(delivery.AssignerWantsBucket) written := p.w.read(delivery.AssignerWantsBucket)
placed := p.w.placedNodes()
// A node the catalogue places composes its own composition (ADR 0547 §2): its own want names none.
for node := range placed {
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
}
if before, err := delivery.DecodeNodeWant(written[key]); err == nil && before.Self {
continue
}
want := delivery.NodeWant{Self: true, Words: "the node composes its own composition (ADR 0547 §2)"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
for node, d := range p.w.nodeComposed() { for node, d := range p.w.nodeComposed() {
if placed[node] {
continue
}
key, err := delivery.WantNodeKey(node) key, err := delivery.WantNodeKey(node)
if err != nil { if err != nil {
continue continue
@@ -97,6 +117,8 @@ func (p *passer) wants(ctx context.Context) {
} }
} }
wanted := p.w.wantsOf() wanted := p.w.wantsOf()
frozen := p.sayPlacement(p.w.placementDiffers())
placed := p.w.placedNodes()
puts := p.putsByModule() puts := p.putsByModule()
p.staleAsks(ctx, trunk) p.staleAsks(ctx, trunk)
names := map[string]bool{} names := map[string]bool{}
@@ -119,7 +141,14 @@ func (p *passer) wants(ctx context.Context) {
fp := trunk[name].FP2 fp := trunk[name].FP2
name := name name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name], m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
Composed: func(node, fp string) (delivery.ComposedDigest, bool) { return p.w.composed(node, name, fp) }, Composed: func(node, fp string) (delivery.ComposedDigest, bool) {
if placed[node] {
// The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want
// names the build alone, and the node judges the move itself.
return delivery.ComposedDigest{Judgement: delivery.JudgedNeither, Words: "the node composes its own"}, true
}
return p.w.composed(node, name, fp)
},
GrantedAt: func(fp string) bool { GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp) key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key) return err == nil && p.w.has(delivery.ControllerGrantsBucket, key)
@@ -153,9 +182,11 @@ func (p *passer) wants(ctx context.Context) {
last[node] = p.w.lastHealthy(node, name, fp) last[node] = p.w.lastHealthy(node, name, fp)
} }
out := assign.Wants(m, nodes, wanted[name], last) out := assign.Wants(m, nodes, wanted[name], last)
p.yeses(ctx, name, m, placed, wanted[name], &out)
gatherNeither(neither, name, out) gatherNeither(neither, name, out)
leftNeither(left, m, wanted[name], out) leftNeither(left, m, wanted[name], out)
p.sayNoWant(name, m.Nodes, wanted[name], out) p.sayNoWant(name, m.Nodes, wanted[name], out)
p.sayNodeFailed(name, fp, m.Nodes, nodes, out, first)
if first { if first {
startClears(stale, m, out) startClears(stale, m, out)
if fp != "" && !m.Held && out.Held == nil && !out.Lifted { if fp != "" && !m.Held && out.Held == nil && !out.Lifted {
@@ -183,6 +214,9 @@ func (p *passer) wants(ctx context.Context) {
p.recordAsk(ctx, name, fp, need) p.recordAsk(ctx, name, fp, need)
} }
for node, want := range out.Wants { for node, want := range out.Wants {
if frozen[node+"."+name] {
continue // one source alone names it (ADR 0549 §5): no want, and the one written before stays
}
key, err := delivery.WantKey(node, name) key, err := delivery.WantKey(node, name)
if err != nil { if err != nil {
p.logf("no want of %s on %s is written: %v", name, node, err) p.logf("no want of %s on %s is written: %v", name, node, err)
@@ -230,6 +264,60 @@ func (p *passer) sayNoWant(module string, nodes []string, before map[string]deli
} }
} }
// sayNodeFailed raises, once per node and words, the condition of a failed health check of a module's newest fingerprint fp
// that is the node's own, since fp runs healthy elsewhere and is not held (ADR 0557 §1, issue 1496), and says it in the journal;
// it clears the condition once when the node no longer fails alone, or is no longer one of the module's nodes. The first pass
// after a start remembers nothing a previous process raised: it clears, for every module, the condition of every node the
// presence bucket knows that does not fail alone, since a node that recovered while no process watched leaves no failed have to
// find it by (a clear of a condition not open is quiet at the router). A condition the router did not take is said again at the next pass.
func (p *passer) sayNodeFailed(module, fp string, nodes []string, known assign.Nodes, out assign.Outcome, first bool) {
if p.nodeFailed == nil {
p.nodeFailed = map[string]string{}
}
look := map[string]bool{}
for _, node := range nodes {
look[node] = true
}
for key := range p.nodeFailed {
if node, m, _ := strings.Cut(key, "."); m == module {
look[node] = true
}
}
if first {
for node := range known.Online {
look[node] = true
}
}
for _, node := range slices.Sorted(maps.Keys(look)) {
key := node + "." + module
f, failing := out.NodeFailed[node]
last, said := p.nodeFailed[key]
switch {
case failing && f.Words+"|"+strings.Join(f.Others, ",") != last:
p.logf("%s on %s: the health check at %s failed here (%s), while it runs healthy on %s: the failure is %s's, so nothing "+
"is held or set back", module, node, fp, f.Words, strings.Join(f.Others, ", "), node)
if p.say != nil {
if err := p.say(assign.NodeFailedCondition(node, module, fp, f, true)); err != nil {
p.logf("the condition of %s failing on %s alone could not be said, and is said at the next pass: %v", module, node, err)
continue
}
}
p.nodeFailed[key] = f.Words + "|" + strings.Join(f.Others, ",")
case !failing && (said || first):
if said {
p.logf("%s on %s no longer fails its health check alone", module, node)
}
if p.say != nil {
if err := p.say(assign.NodeFailedCondition(node, module, fp, assign.NodeFailure{}, false)); err != nil && said {
p.logf("the condition of %s failing on %s alone could not be cleared, and is cleared at the next pass: %v", module, node, err)
continue
}
}
delete(p.nodeFailed, key)
}
}
}
// neitherOne is a module assigned to a node that holds no want for it after a pass, and the rule holding its first want back. // neitherOne is a module assigned to a node that holds no want for it after a pass, and the rule holding its first want back.
type neitherOne struct { type neitherOne struct {
node, module string node, module string
@@ -728,3 +816,83 @@ func (p *passer) backupAsk(ctx context.Context, key, node, module string, want d
} }
p.said[c.Key] = true p.said[c.Key] = true
} }
// sayPlacement raises, once, the condition of every pair the controller's assignments and the catalogue's placement do not
// both name, clears it once they agree, and answers those pairs by node and module (ADR 0549 §5): the pass writes no want
// for them and keeps the one written before. A say that fails is made again at the next pass.
func (p *passer) sayPlacement(differ []assign.Pair) map[string]bool {
frozen := map[string]bool{}
now := map[string]bool{}
for _, d := range differ {
frozen[d.Node+"."+d.Module] = true
c := assign.PlacementCondition(d, true)
now[c.Key] = true
if p.said[c.Key+".raised"] {
continue
}
p.logf("%s on %s is named by %s alone: no want is written for it there until both sources name it (ADR 0549 §5)", d.Module, d.Node, d.Only)
if err := p.say(c); err != nil {
p.logf("the placement condition of %s on %s could not be said, and is said at the next pass: %v", d.Module, d.Node, err)
continue
}
p.said[c.Key+".raised"] = true
}
prefix := delivery.AssignerModule + ".placement."
for key := range p.said {
c, raised := strings.CutSuffix(key, ".raised")
if !raised || !strings.HasPrefix(c, prefix) || now[c] {
continue
}
node, module, _ := strings.Cut(strings.TrimPrefix(c, prefix), ".")
if err := p.say(assign.PlacementCondition(assign.Pair{Node: node, Module: module}, false)); err != nil {
p.logf("the placement condition of %s on %s could not be cleared, and is cleared at the next pass: %v", module, node, err)
continue
}
delete(p.said, key)
}
return frozen
}
// yeses writes a placed node's wants naming the build alone, and sees to the moves a node that composes its own judged a
// person must allow (ADR 0547 §3): for a have of phase
// waiting-yes it raises the record ask once, naming the composition waiting and the node's judgement, and on a yes writes
// the want again naming it; a yes once written is kept while the want stays at its build, so the node is not asked again.
func (p *passer) yeses(ctx context.Context, module string, m assign.Module, placed map[string]bool, before map[string]delivery.Want, out *assign.Outcome) {
for _, node := range m.Nodes {
if !placed[node] {
continue
}
was, had := before[node]
w, written := out.Wants[node]
if !written {
if !had || was.Removed {
continue
}
w = was
}
if w.Removed {
continue
}
yes := ""
if had && was.FP == w.FP {
yes = was.Yes
}
key, err := delivery.HaveKey(node, module)
if err != nil {
continue
}
if h, err := delivery.DecodeHave(p.w.value(delivery.HaveBucket, key)); err == nil && h.Phase == delivery.PhaseWaitingYes && h.Waits != yes {
if m.Allowed != nil && m.Allowed(node, h.Waits) {
yes = h.Waits
} else {
p.recordAsk(ctx, module, w.FP, assign.RecordNeed{Node: node, Digest: h.Waits, Judgement: h.Judgement, Words: h.Detail})
}
}
// A want carried over from before the node composed its own may still name the controller's composition; a placed
// node's want names the build alone (ADR 0547 §2).
if yes != w.Yes || w.Composed != "" || !w.Self {
w.Yes, w.Composed, w.Self = yes, "", true
out.Wants[node] = w
}
}
}
@@ -342,6 +342,9 @@ func putOfTheBus(t *testing.T) {
// again after a say that failed, and clears it once when the module has a want again; a module on neither list is also written to // again after a say that failed, and clears it once when the module has a want again; a module on neither list is also written to
// the journal, so it is loud with no operator channel. // the journal, so it is loud with no operator channel.
func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) { func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
// A pair only one source names is a condition of the same kind (issue 1126): folded here to hold the count.
aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t)
t.Run("gathers", testThePassGathersTheModulesOnNeitherList)
p := newPasser(buckets{}, time.Date(2026, 10, 25, 15, 0, 0, 0, time.UTC)) p := newPasser(buckets{}, time.Date(2026, 10, 25, 15, 0, 0, 0, time.UTC))
var said []asks.Condition var said []asks.Condition
failing := true failing := true
@@ -391,7 +394,7 @@ func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
} }
// 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module. // 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module.
func TestThePassGathersTheModulesOnNeitherList(t *testing.T) { func testThePassGathersTheModulesOnNeitherList(t *testing.T) {
got := map[string]neitherOne{} got := map[string]neitherOne{}
gatherNeither(got, "screen-lock", assign.Outcome{Neither: map[string]assign.Holding{"g14": {Rule: delivery.RuleGrant, Words: "w"}}}) gatherNeither(got, "screen-lock", assign.Outcome{Neither: map[string]assign.Holding{"g14": {Rule: delivery.RuleGrant, Words: "w"}}})
gatherNeither(got, "rofi", assign.Outcome{}) gatherNeither(got, "rofi", assign.Outcome{})
@@ -443,12 +446,71 @@ func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
t.Errorf("pass %d said %q, want %q", i, got, c.line) t.Errorf("pass %d said %q, want %q", i, got, c.line)
} }
} }
// 1496, ADR 0557 §1: a failure that is the node's own (its fingerprint healthy elsewhere, not held) raises the node's condition
// once, naming the node, the module, F and the have's words, and clears it once when it ends.
var conds []asks.Condition
p.say = func(c asks.Condition) error { conds = append(conds, c); return nil }
failing := map[string]assign.NodeFailure{"shanks": {Others: []string{"ace"}, Words: "health failed, phase running: w"}}
for i, c := range []struct {
failed map[string]assign.NodeFailure
state string // the condition said in this pass, "" for none
}{
{failing, asks.StateRaised},
{failing, ""},
{nil, asks.StateCleared},
{nil, ""},
} {
conds = nil
p.sayNodeFailed("rofi", fpTest, []string{"shanks"}, assign.Nodes{}, assign.Outcome{NodeFailed: c.failed}, false)
switch {
case c.state == "" && len(conds) != 0, c.state != "" && (len(conds) != 1 || conds[0].State != c.state):
t.Errorf("node failed, pass %d said %+v, want %q", i, conds, c.state)
case c.state == asks.StateRaised && (conds[0].Key != "mesh-delivery-assigner.failed-alone.shanks.rofi" ||
!strings.Contains(conds[0].Explanation, "failed on shanks (health failed, phase running: w), while it runs healthy on ace")):
t.Errorf("node failed, raised %+v", conds[0])
}
}
// D2: a node that leaves the module's nodes while it failed alone is cleared once; the first pass after a start clears every
// node the presence bucket knows, for every module, failed have or not (a node that recovered unwatched leaves none).
conds = nil
p.sayNodeFailed("rofi", fpTest, []string{"shanks"}, assign.Nodes{}, assign.Outcome{NodeFailed: failing}, false)
p.sayNodeFailed("rofi", fpTest, nil, assign.Nodes{}, assign.Outcome{}, false)
p.sayNodeFailed("rofi", fpTest, nil, assign.Nodes{}, assign.Outcome{}, false)
if len(conds) != 2 || conds[1].State != asks.StateCleared || conds[1].Key != "mesh-delivery-assigner.failed-alone.shanks.rofi" {
t.Errorf("a node leaving the module said %+v, want one raise and one clear", conds)
}
known := assign.Nodes{Online: map[string]bool{"ace": true, "g14": false},
Haves: map[string]map[string]delivery.Have{"g14": {"rofi": {FP: fpTest, Health: delivery.HealthFailed}}}}
for _, c := range []struct {
name string
known assign.Nodes
keys []string
}{
{"a failed have", known, []string{"mesh-delivery-assigner.failed-alone.ace.rofi", "mesh-delivery-assigner.failed-alone.g14.rofi"}},
{"no failed have", assign.Nodes{Online: known.Online}, []string{"mesh-delivery-assigner.failed-alone.ace.rofi", "mesh-delivery-assigner.failed-alone.g14.rofi"}},
} {
conds = nil
fresh := newPasser(buckets{}, time.Date(2026, 10, 26, 22, 0, 0, 0, time.UTC))
fresh.logf = func(string, ...any) {}
fresh.say = func(c asks.Condition) error { conds = append(conds, c); return nil }
fresh.sayNodeFailed("rofi", fpTest, nil, c.known, assign.Outcome{}, true)
var keys []string
for _, cond := range conds {
if cond.State == asks.StateCleared {
keys = append(keys, cond.Key)
}
}
if !slices.Equal(keys, c.keys) || len(conds) != len(c.keys) {
t.Errorf("first pass, %s: said %+v, want clears of %q", c.name, conds, c.keys)
}
}
} }
// A neither or held condition a previous process raised is cleared by the first pass after a start: a fresh pass remembers // A neither or held condition a previous process raised is cleared by the first pass after a start: a fresh pass remembers
// nothing it said, so it clears, once, the neither condition of every assigned module not on neither list and the held condition of // nothing it said, so it clears, once, the neither condition of every assigned module not on neither list and the held condition of
// every module that is not held. Clearing a condition that is not open is quiet at the router. // every module that is not held. Clearing a condition that is not open is quiet at the router.
func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) { func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
t.Run("a held module", testAHeldModuleRaisesNoNeitherAndClearsAStandingOne)
wantsKV := &memKV{values: map[string][]byte{}} wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerWantsBucket: wantsKV, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now()) p := newPasser(buckets{delivery.AssignerWantsBucket: wantsKV, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
var said []asks.Condition var said []asks.Condition
@@ -486,7 +548,7 @@ func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
// 1945/AC24: given a module held at F under rule 2 on a node that runs it with no want, when the assigner passes, then it raises no // 1945/AC24: given a module held at F under rule 2 on a node that runs it with no want, when the assigner passes, then it raises no
// neither condition for it, a neither condition this process raised for it before is cleared, and so is one a previous process // neither condition for it, a neither condition this process raised for it before is cleared, and so is one a previous process
// raised (the first pass after a start), each once. // raised (the first pass after a start), each once.
func TestAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) { func testAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) {
setup := func() (*passer, *[]asks.Condition) { setup := func() (*passer, *[]asks.Condition) {
p := newPasser(buckets{delivery.AssignerWantsBucket: &memKV{values: map[string][]byte{}}, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now()) p := newPasser(buckets{delivery.AssignerWantsBucket: &memKV{values: map[string][]byte{}}, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
said := &[]asks.Condition{} said := &[]asks.Condition{}
@@ -702,6 +764,8 @@ func mustKey(key string, err error) string {
// (ADR 0545). That record ask is not about the trunk's fingerprint: it survives every pass, so the operator is asked, and on a // (ADR 0545). That record ask is not about the trunk's fingerprint: it survives every pass, so the operator is asked, and on a
// yes the want names the revised digest. // yes the want names the revised digest.
func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing.T) { func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing.T) {
// A placed node's yes comes from its have (issue 1126): folded here to hold the count.
aPlacedNodesWantsNameTheBuildAloneAndItsYesComesFromItsHave(t)
fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64) fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64)
revised, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64) revised, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64)
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}} asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
@@ -750,3 +814,133 @@ func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing
t.Error("the answered record ask was taken away") t.Error("the answered record ask was taken away")
} }
} }
// 1126 (ADR 0549 §5): until the third milestone a pair the controller's assignments and the catalogue's placement do not
// both name gets no want and one condition, raised once and cleared once both name it; before any placement was read the
// assignments are the one source and nothing is raised.
func aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t *testing.T) {
p := newPasser(buckets{}, time.Date(2026, 10, 26, 18, 0, 0, 0, time.UTC))
var said []asks.Condition
p.say = func(c asks.Condition) error { said = append(said, c); return nil }
assigned := func(node, module string) {
raw, _ := json.Marshal(map[string]string{"node": node, "module": module})
p.w.put(provider.AssignmentsBucket, node+"."+module, raw)
}
place := func(node string, modules ...string) {
pl := delivery.Placement{Node: node}
for _, m := range modules {
pl.Modules = append(pl.Modules, delivery.PlacedModule{Module: m})
}
raw, err := delivery.EncodePlacement(pl)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.PlacementKey(node)
p.w.put(delivery.PlacementBucket, key, raw)
}
assigned("g14", "a")
assigned("g14", "b")
if got, _ := p.w.assigned(); !slices.Equal(got["g14"], []string{"a", "b"}) || len(p.sayPlacement(p.w.placementDiffers())) != 0 || len(said) != 0 {
t.Fatalf("before any placement: assigned %v, said %v", got, said)
}
none := func(context.Context, string) ([]byte, error) { return nil, errNotThere }
if m := p.w.mesh(context.Background(), none); m.Input != nil || m.InputUnread != "" {
t.Errorf("before any placement the check composes over %+v (%q)", m.Input, m.InputUnread)
}
place("g14", "a", "c")
got, _ := p.w.assigned()
frozen := p.sayPlacement(p.w.placementDiffers())
p.sayPlacement(p.w.placementDiffers())
if !slices.Equal(got["g14"], []string{"a"}) || !frozen["g14.b"] || !frozen["g14.c"] || frozen["g14.a"] || len(said) != 2 ||
said[0].State != asks.StateRaised || !strings.Contains(said[0].Explanation, "the controller's assignments names b on g14") ||
!strings.Contains(said[1].Explanation, "the catalogue's placement names c on g14") {
t.Fatalf("one source alone: assigned %v, frozen %v, said %+v", got, frozen, said)
}
place("g14", "a", "b", "c")
assigned("g14", "c")
if frozen := p.sayPlacement(p.w.placementDiffers()); len(frozen) != 0 || len(said) != 4 || said[2].State != asks.StateCleared || said[3].State != asks.StateCleared {
t.Errorf("both agree again: frozen %v, said %+v", frozen, said)
}
// 1126 (ADR 0547 §2): placed, the check composes over the input the nodes compose from (frombus), once each placed node
// wrote its record; until then it says why and composes the assignments alone.
if m := p.w.mesh(context.Background(), none); m.Input != nil || !strings.Contains(m.InputUnread, "g14") {
t.Errorf("a placed node with no record: input %+v, unread %q", m.Input, m.InputUnread)
}
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "g14", TunnelKey: "k"}})
p.w.put(delivery.HaveBucket, "g14.node", rec)
m := p.w.mesh(context.Background(), none)
if m.Input == nil || len(m.Input.Nodes) != 1 || !slices.Equal(m.Input.Nodes[0].Assigned, []string{"a", "b", "c"}) || m.Input.Nodes[0].Place.Key != "k" {
t.Errorf("the check's input from the bus: %+v (%q)", m.Input, m.InputUnread)
}
}
// 1126 (ADR 0547 §2 to §4): a node the catalogue places composes its own: its wants name the build alone and its own want
// names no composition; a have waiting for a yes raises the record ask once, and the yes is written into the want and
// kept there while the want stays at its build.
func aPlacedNodesWantsNameTheBuildAloneAndItsYesComesFromItsHave(t *testing.T) {
fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64)
waiting, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64)
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
p.logf = t.Logf
put := func(bucket, key string, v []byte) { p.w.put(bucket, key, v) }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
put(delivery.ModulesBucket, "web", mod) // the trunk's build is not done: rule 1 holds g14 at fpOld
put(provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"b-1"}`))
pl, _ := delivery.EncodePlacement(delivery.Placement{Node: "g14", Modules: []delivery.PlacedModule{{Module: "web"}}})
put(delivery.PlacementBucket, "placement.g14", pl)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
put(delivery.PresenceBucket, "g14", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: was, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
put(delivery.HaveBucket, "g14.web", have)
wkey, _ := delivery.WantKey("g14", "web")
want, _ := delivery.EncodeWant(delivery.Want{FP: fpOld, Composed: was})
wantsKV.values[wkey] = want
put(delivery.AssignerWantsBucket, wkey, want)
sync := func() {
for k, v := range wantsKV.values {
put(delivery.AssignerWantsBucket, k, v)
}
}
p.wants(context.Background())
p.nodeWants(context.Background())
sync()
got, err := delivery.DecodeWant(wantsKV.values[wkey])
if err != nil || got.FP != fpOld || got.Composed != "" || !got.Self {
t.Fatalf("the placed node's want is %+v (%v), want it at %s, the node composing it", got, err, fpOld)
}
nkey, _ := delivery.WantNodeKey("g14")
if nw, err := delivery.DecodeNodeWant(wantsKV.values[nkey]); err != nil || nw.Composed != "" || !nw.Self {
t.Fatalf("the placed node's own want is %+v (%v), want one naming no composition", nw, err)
}
waits, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: was, Phase: delivery.PhaseWaitingYes, Health: delivery.HealthOK,
Judgement: delivery.JudgedRecreates, Waits: waiting, Detail: "it recreates web.server"})
put(delivery.HaveBucket, "g14.web", waits)
akey, _ := delivery.RecordAskKey("g14", "web", waiting)
p.wants(context.Background())
if _, there := asksKV.values[akey]; !there {
t.Fatal("a have waiting for a yes raised no record ask")
}
a, err := delivery.DecodeAssignerAsk(akey, asksKV.values[akey])
if err != nil {
t.Fatal(err)
}
put(delivery.AssignerAsksBucket, akey, asksKV.values[akey])
yes, _ := delivery.EncodeAnswer(delivery.Answer{Option: assign.OptionYes, Warrant: a.Ask.ID + "@2026-10-26T19:00:00Z", Words: "the operator"})
put(delivery.AnswersBucket, akey, yes)
p.wants(context.Background())
sync()
if got, err = delivery.DecodeWant(wantsKV.values[wkey]); err != nil || got.Yes != waiting || got.FP != fpOld {
t.Fatalf("after the yes the want is %+v (%v), want it to name the yes %s", got, err, waiting)
}
// The node applied it: its have runs the composition; the yes stays while the want stays at its build.
ran, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: waiting, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
put(delivery.HaveBucket, "g14.web", ran)
p.wants(context.Background())
if got, _ = delivery.DecodeWant(wantsKV.values[wkey]); got.Yes != waiting {
t.Errorf("a pass after the move dropped the yes: %+v", got)
}
}
@@ -14,6 +14,8 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider" "git.novox.be/novox/mesh-sdk/go/provider"
@@ -162,8 +164,56 @@ func (w *world) manifest(ctx context.Context, get getter, digest string) parsed
return p return p
} }
// assigned is each node's modules with an open assignment, read from the controller's assignments, in name order. // assigned is each node's modules both the controller's assignments and the indexer's placement name, in name order
// (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module.
func (w *world) assigned() (map[string][]string, map[string]assignment) { func (w *world) assigned() (map[string][]string, map[string]assignment) {
byAssignment, all := w.assignedByController()
placed, seeded := w.placed()
both, _ := assign.Placed(byAssignment, placed, seeded)
return both, all
}
// placementDiffers is every pair the controller's assignments and the indexer's placement do not both name.
func (w *world) placementDiffers() []assign.Pair {
byAssignment, _ := w.assignedByController()
placed, seeded := w.placed()
_, differ := assign.Placed(byAssignment, placed, seeded)
return differ
}
// placed is each node's modules in the indexer's placement (ADR 0549 §2), and whether any placement key was read: a key
// that does not read places nothing on its node and is said by the indexer's own read tool.
func (w *world) placed() (map[string][]string, bool) {
out := map[string][]string{}
values := w.read(delivery.PlacementBucket)
for key, value := range values {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
p, err := delivery.DecodePlacement(value)
if err != nil || p.Node != node {
continue
}
out[node] = p.Assigned()
}
return out, len(values) > 0
}
// placedNodes is every node the indexer's placement names (ADR 0549 §2): those compose their own modules (ADR 0547 §2),
// so their wants name the build alone and the seventh rule does not hold them.
func (w *world) placedNodes() map[string]bool {
placed, _ := w.placed()
out := make(map[string]bool, len(placed))
for node := range placed {
out[node] = true
}
return out
}
// assignedByController is each node's modules with an open assignment, read from the controller's assignments, in
// name order.
func (w *world) assignedByController() (map[string][]string, map[string]assignment) {
out, all := map[string][]string{}, map[string]assignment{} out, all := map[string][]string{}, map[string]assignment{}
for _, value := range w.read(provider.AssignmentsBucket) { for _, value := range w.read(provider.AssignmentsBucket) {
var a assignment var a assignment
@@ -210,9 +260,86 @@ func (w *world) mesh(ctx context.Context, manifests getter) assign.Mesh {
} }
m.Manifests[name] = p.manifest m.Manifests[name] = p.manifest
} }
// Once the catalogue places the nodes, the check composes over the input they compose from (ADR 0547 §2, ADR 0549
// §3): one composer's input for the check and the nodes. An input that cannot be read is said, and the nodes and their
// assignments alone are composed, as before any placement.
if _, seeded := w.placed(); seeded {
in, err := frombus.Input(ctx, worldReader{w: w, manifests: manifests}, frombus.Mesh{}, frombus.Scope{Every: true})
if err != nil {
m.InputUnread = err.Error()
} else {
m.Input = &in
}
}
return m return m
} }
// worldReader is what the assigner watched as frombus reads the bus: the indexer's placement and modules, each node's
// record in its own have key, the manifests by digest; the check reads no config value, own secret or credential (the
// composition it asks resolves, and stands in for what it does not carry).
type worldReader struct {
w *world
manifests getter
}
func (r worldReader) Placements(context.Context) (map[string]delivery.Placement, error) {
out := map[string]delivery.Placement{}
for key, value := range r.w.read(delivery.PlacementBucket) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
if p, err := delivery.DecodePlacement(value); err == nil && p.Node == node {
out[node] = p
}
}
return out, nil
}
func (r worldReader) Records(context.Context) (map[string]delivery.NodeRecord, error) {
out := map[string]delivery.NodeRecord{}
for key, value := range r.w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
}
if h, err := delivery.DecodeHaveNode(value); err == nil && h.Record != nil {
out[node] = *h.Record
}
}
return out, nil
}
func (r worldReader) Modules(context.Context) (map[string]delivery.Module, error) {
return r.w.trunk(), nil
}
func (r worldReader) Manifest(ctx context.Context, digest string) ([]byte, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return nil, err
}
raw, err := r.manifests(ctx, key)
if err != nil {
return nil, err
}
m, err := delivery.DecodeManifest(raw)
if err != nil {
return nil, err
}
return []byte(m.JSON), nil
}
func (worldReader) Values(context.Context, string, string) (map[string]any, error) { return nil, nil }
func (worldReader) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return nil, nil
}
func (worldReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil
}
// change is a candidate's change for the composition check, or why the composition waits for it. // change is a candidate's change for the composition check, or why the composition waits for it.
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) { func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{}, ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-assigner
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -10
View File
@@ -1,13 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c h1:pY2YpHSSsfDPBrOEiWTNSIPn0zdnAayRmtmmyQ3hCHY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8 h1:jA7V7hlfJAigVUvJDpKwfgd9ylQ9bqppg3rK9XgLWoA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8 h1:vhYk/iv43u2WGSTrFyQGDvGBxLaRITgMAiiI9DjuA10=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5 h1:gE80bR0tPME7MTMBLcdP62v+F/9soVXmdcBYNMOye44=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -81,6 +81,29 @@ func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.C
return c return c
} }
// NodeFailedCondition is the condition the assigner raises for a module whose newest fingerprint fp failed its health check on
// node while other nodes run it healthy (ADR 0557 §1, issue 1496): the failure is the node's, so nothing is held or set back and
// the node keeps its want. It names the node, the module, fp and the have's words, keyed by node and module, and is cleared
// (raised false) once the node no longer fails alone.
func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) asks.Condition {
key := delivery.AssignerModule + ".failed-alone." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s no longer fails its health check alone.", module, node)}
}
headline := fmt.Sprintf("%s fails on %s alone", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s The health check of %s at %s failed on %s (%s), while it runs healthy on %s. The failure is %s's, "+
"so the assigner holds nothing and sets no node back, and %s keeps its want. This clears once %s runs it healthy or "+
"another version is the newest.", asks.NothingToDo, module, short(fp), node, f.Words, strings.Join(f.Others, ", "), node,
node, node)}
}
// NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition // NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent) // gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by // naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
@@ -35,6 +35,13 @@ type Mesh struct {
// Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a // Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a
// composer at another mesh-sdk reads (issue 1622). // composer at another mesh-sdk reads (issue 1622).
Digests map[string]string Digests map[string]string
// Input is the rest of the mesh as the nodes compose from it, filled by mesh-sdk's frombus from the catalogue's
// placement and the nodes' records (novox/hq ADR 0547 §2, ADR 0549 §3, issue 1126), so the check and the nodes share
// one composer's input; nil before any placement is read, when the nodes and their assignments alone are composed.
Input *compose.Input
// InputUnread says why that input could not be read from the bus, when it could not: the check then composes the
// nodes and their assignments alone, and says so in its warnings.
InputUnread string
} }
// Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those // Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those
@@ -42,8 +42,15 @@ func HeadJob(m Mesh, c Change) (delivery.ComposeJob, error) {
for _, name := range c.Removed { for _, name := range c.Removed {
delete(change, name) delete(change, name)
} }
req := composer.Request{V: composer.Version, Input: plainInput(m.Assigned), Depth: composer.DepthResolve, in, fromClaims := plainInput(m.Assigned), true
HoldingsFromClaims: true, Machines: Touched(m, c), if m.Input != nil {
// The input the nodes compose from, without its catalogue: the cases carry the manifests (issue 1126).
in = *m.Input
in.Catalogue = nil
fromClaims = len(in.Holdings) == 0
}
req := composer.Request{V: composer.Version, Input: in, Depth: composer.DepthResolve,
HoldingsFromClaims: fromClaims, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, {Name: CaseChange, Manifests: change}}} Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, {Name: CaseChange, Manifests: change}}}
return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group) return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group)
} }
@@ -195,6 +202,10 @@ func FromComposer(m Mesh, c Change, head Ran, readers []Reader, took time.Durati
readerFails, readerWarnings := judgeReaders(c, change, readers) readerFails, readerWarnings := judgeReaders(c, change, readers)
fails = append(fails, readerFails...) fails = append(fails, readerFails...)
v.Warnings = append(v.Warnings, readerWarnings...) v.Warnings = append(v.Warnings, readerWarnings...)
if m.InputUnread != "" {
v.Warnings = append(v.Warnings, "composed over the nodes and their assignments alone: the input the nodes compose from "+
"cannot be read from the bus ("+m.InputUnread+")")
}
return finish(v, m, c, machines, fails) return finish(v, m, c, machines, fails)
} }
@@ -8,6 +8,7 @@ import (
"time" "time"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer" "git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -100,6 +101,25 @@ func TestAGroupWithAnSDKMemberAsksTheComposerAtItsHead(t *testing.T) {
if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims { if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims {
t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims) t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims)
} }
// 1126 (ADR 0547 §2, ADR 0549 §3): once the catalogue places the nodes, the check composes over the input the nodes
// compose from (frombus), its catalogue left to the cases, and the seats held as placement says.
m := digestedMesh()
m.Input = &compose.Input{Catalogue: map[string]catalogue.Manifest{"web": {Module: "web"}},
Nodes: []compose.Node{{Name: "g14", Assigned: []string{"web"}, Place: compose.Place{Endpoint: "g14.example:51820"}}},
Holdings: []catalogue.Held{{Claim: "web-seat", Scope: "mesh", Node: "g14", Module: "web"}}}
placedJob, err := HeadJob(m, c)
if err != nil {
t.Fatal(err)
}
placed, err := composer.Decode(placedJob.Request)
if err != nil {
t.Fatal(err)
}
if placed.Input.Catalogue != nil || len(placed.Input.Nodes) != 1 || placed.Input.Nodes[0].Place.Endpoint != "g14.example:51820" ||
len(placed.Input.Holdings) != 1 || placed.HoldingsFromClaims {
t.Errorf("the placed check's request = %+v, holdings from claims %v", placed.Input, placed.HoldingsFromClaims)
}
} }
// 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the // 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the
@@ -0,0 +1,99 @@
package assign
import (
"fmt"
"sort"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Until ADR 0547's third milestone two sources name which module runs on which node: the controller's assignments and
// the indexer's placement, read from the catalogue (ADR 0549 §5). The assigner writes wants only for a pair both name:
// for a pair one source names alone it writes no want and touches none it wrote before, so nothing moves and nothing is
// removed on one source's word, and it raises a condition naming the pair until both agree. Before the first placement
// file reaches the indexer (no placement key at all) the controller's assignments are the one source, as before.
// Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or
// OnlyPlacement.
type Pair struct {
Node, Module, Only string
}
// The source that names a pair alone.
const (
OnlyAssignments = "the controller's assignments"
OnlyPlacement = "the catalogue's placement"
)
// Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs.
func Placed(assigned, placed map[string][]string, seeded bool) (map[string][]string, []Pair) {
if !seeded {
return assigned, nil
}
set := func(m map[string][]string) map[string]bool {
out := map[string]bool{}
for node, modules := range m {
for _, module := range modules {
out[node+"\x00"+module] = true
}
}
return out
}
inAssigned, inPlaced := set(assigned), set(placed)
both := map[string][]string{}
var differ []Pair
for node, modules := range assigned {
for _, module := range modules {
if inPlaced[node+"\x00"+module] {
both[node] = append(both[node], module)
} else {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyAssignments})
}
}
}
for node, modules := range placed {
for _, module := range modules {
if !inAssigned[node+"\x00"+module] {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyPlacement})
}
}
}
for _, modules := range both {
sort.Strings(modules)
}
sort.Slice(differ, func(i, j int) bool {
if differ[i].Node != differ[j].Node {
return differ[i].Node < differ[j].Node
}
return differ[i].Module < differ[j].Module
})
return both, differ
}
// PlacementCondition is the condition of a pair the two sources do not both name: raised while they differ, cleared once
// both name it or neither does.
func PlacementCondition(p Pair, raised bool) asks.Condition {
key := delivery.AssignerModule + ".placement." + p.Node + "." + p.Module
subject := asks.Scope{Scope: "module", ID: p.Module, Machine: p.Node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("the controller's assignments and the catalogue's placement agree on %s on %s.", p.Module, p.Node)}
}
headline := fmt.Sprintf("%s on %s is placed by one source only", p.Module, p.Node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
var needs string
if p.Only == OnlyAssignments {
needs = fmt.Sprintf("add %s to the catalogue's placement file of %s, or unassign it at the controller", p.Module, p.Node)
} else {
needs = fmt.Sprintf("assign %s to %s at the controller, as the merged placement file says", p.Module, p.Node)
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s names %s on %s and the other source does not. Until ADR 0547's third milestone both must "+
"name a module on a node (ADR 0549 §5): the assigner writes no want for it there and leaves the one it wrote before, so "+
"nothing moves or is removed. This clears once both name it or neither does.", asks.NeedsYou, needs, p.Only, p.Module, p.Node)}
}
@@ -2,6 +2,7 @@ package assign
import ( import (
"fmt" "fmt"
"maps"
"slices" "slices"
"sort" "sort"
"strings" "strings"
@@ -169,6 +170,9 @@ type Outcome struct {
// Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held // Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held
// record and its ask are taken away, and the wants move by the rules again in this pass. // record and its ask are taken away, and the wants move by the rules again in this pass.
Lifted bool Lifted bool
// NodeFailed is, by node, a failed health check of m.FP on a node while another node runs m.FP healthy: the failure is
// that node's, so m.FP is not held and nothing is set back (ADR 0557 §1, issue 1496). The pass raises its condition once.
NodeFailed map[string]NodeFailure
// Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14). // Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14).
Record []RecordNeed Record []RecordNeed
// Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set // Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set
@@ -228,11 +232,25 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
} }
if !held && !out.Lifted && !m.Released && m.FP != "" { if !held && !out.Lifted && !m.Released && m.FP != "" {
for _, node := range slices.Sorted(slices.Values(m.Nodes)) { for _, node := range slices.Sorted(slices.Values(m.Nodes)) {
if h, ok := n.Haves[node][m.Name]; ok && h.FP == m.FP && failed(h) { h, ok := n.Haves[node][m.Name]
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)} if !ok || h.FP != m.FP || !failed(h) {
held = true continue
break
} }
if others := healthyElsewhere(m, n, node); len(others) > 0 && !failedElsewhere(m, n, node) {
// F runs healthy on another node, so its failure here is this node's, not F's: F is not held, no node is set back,
// and the failing node keeps its want (ADR 0557 §1; issue 1496: on ace a newly assigned mesh-build-node failed its
// first check and the hold moved shanks, g14 and novox, healthy on F for an hour, back). The pass raises the node's
// condition. With no online healthy witness, or with F failed on a second node, the hold and the set-back stand (ADR 0557
// §2, ADR 0429 §6).
if out.NodeFailed == nil {
out.NodeFailed = map[string]NodeFailure{}
}
out.NodeFailed[node] = NodeFailure{Others: others, Words: HeldWords(h, time.Time{})} // said once: no clock in it
continue
}
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
} }
} }
mm := m mm := m
@@ -298,6 +316,13 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
ok, target = true, have.FP ok, target = true, have.FP
} }
} }
if ok && target == m.FP && len(out.NodeFailed) > 0 && !(had && !before.Removed && before.FP == m.FP) &&
n.Haves[node][m.Name].FP != m.FP {
// While a node fails F alone, F reaches no further node: a node that neither wants nor runs F gets no new want
// naming it, and wants already written stay (ADR 0557 §2). A node that runs F with no live want gets its want at F, or
// it would be on neither list and its node-engine would give the module up (issue 1945).
ok, rule, words = false, delivery.RuleNotHeld, FailedAloneWords(m.Name, m.FP, slices.Sorted(maps.Keys(out.NodeFailed)))
}
switch { switch {
case ok: case ok:
d, _ := m.composition(node, target) d, _ := m.composition(node, target)
@@ -419,6 +444,43 @@ func failed(h delivery.Have) bool {
return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed) return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed)
} }
// healthyElsewhere is every online node of m but node that runs m.FP healthy, by name.
func healthyElsewhere(m Module, n Nodes, node string) []string {
var out []string
for _, other := range slices.Sorted(slices.Values(m.Nodes)) {
// A witness is online: a node that sleeps keeps the have it last wrote, and a stale have is no evidence that F runs
// healthy now.
if other != node && n.Online[other] && healthy(n.Haves[other][m.Name], m.FP) {
out = append(out, other)
}
}
return out
}
// failedElsewhere says whether another node of m has a have naming m.FP failed: then the failure is not one node's alone, and
// the hold and the set-back stand at the second failure (ADR 0557 §2).
func failedElsewhere(m Module, n Nodes, node string) bool {
for _, other := range m.Nodes {
if h, ok := n.Haves[other][m.Name]; ok && other != node && h.FP == m.FP && failed(h) {
return true
}
}
return false
}
// FailedAloneWords say why a node gets no new want naming fp: fp fails its health check on the nodes named, alone.
func FailedAloneWords(module, fp string, failing []string) string {
return fmt.Sprintf("%s at %s fails its health check on %s, so no further node moves to it until that is over", module, short(fp),
strings.Join(failing, ", "))
}
// NodeFailure is a failed health check of a module's newest fingerprint on one node while it runs healthy on Others: the have's
// health, phase and words (HeldWords without a time), and the healthy witnesses by name.
type NodeFailure struct {
Others []string
Words string
}
// runs says whether a have is of a module the node runs: running, whatever its health. // runs says whether a have is of a module the node runs: running, whatever its health.
func runs(h delivery.Have) bool { func runs(h delivery.Have) bool {
return h.Phase == delivery.PhaseRunning return h.Phase == delivery.PhaseRunning
@@ -150,6 +150,72 @@ func TestAFailedHealthHoldsAndSetsBack(t *testing.T) {
if w, ok := out.Wants["g14"]; ok && w.FP == fNew { if w, ok := out.Wants["g14"]; ok && w.FP == fNew {
t.Errorf("g14 moved to a held fingerprint: %+v", w) t.Errorf("g14 moved to a held fingerprint: %+v", w)
} }
// 1496, ADR 0557 §1, §2: the same failure while F runs healthy on another node is that node's: F is not held, nothing is set
// back, the failing node keeps its want, and the outcome names it (on ace a newly assigned module's first check held a version healthy for an hour on three nodes).
n.Haves["g14"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
before := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}, "g14": {FP: fNew, Composed: dig(fNew)}}
out = Wants(m, n, before, map[string]string{"ace": fOld, "g14": fOld})
if out.Held != nil || len(out.Wants) != 0 {
t.Fatalf("held %+v, wants %+v: want no hold and no want changed", out.Held, out.Wants)
}
if f := out.NodeFailed["ace"]; !slices.Equal(f.Others, []string{"g14"}) || f.Words != "health failed, phase running: connection refused" ||
len(out.NodeFailed) != 1 {
t.Errorf("node failed = %+v", out.NodeFailed)
}
// ADR 0557 §2 (version 9): while one node fails F alone, novox, whose want names fOld, gets no new want naming F; the wants
// at F stay. The canary ace runs F healthy here and g14 fails it, so the canary rule (3) would let novox move: only §2 holds it.
three := m
three.Nodes = []string{"ace", "g14", "novox"}
failedHave := n.Haves["ace"]["mail"]
n.Haves["ace"]["mail"], n.Haves["g14"]["mail"] = n.Haves["g14"]["mail"], failedHave
n.Online["novox"] = true
n.Haves["novox"] = map[string]delivery.Have{"mail": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}}
before["novox"] = delivery.Want{FP: fOld, Composed: dig(fOld)}
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if _, alone := out.NodeFailed["g14"]; !alone {
t.Errorf("g14 does not fail alone: %+v", out.NodeFailed)
}
if w, written := out.Wants["novox"]; out.Held != nil || len(out.NodeFailed) != 1 || (written && w.FP == fNew) || out.Rules["novox"].Rule != delivery.RuleNotHeld {
t.Errorf("held %+v, node failed %+v, novox's want %+v (%+v): want novox kept off F", out.Held, out.NodeFailed, out.Wants["novox"], out.Rules["novox"])
}
if _, written := out.Wants["g14"]; written {
t.Errorf("g14's want at F was rewritten: %+v", out.Wants["g14"])
}
// A node that runs F healthy with no want, while g14 fails it alone, gets its want at F and is on no neither list (issue 1945).
n.Haves["novox"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
delete(before, "novox")
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if w := out.Wants["novox"]; w.FP != fNew || len(out.Neither) != 0 {
t.Errorf("novox running F with no want: want %+v, neither %+v; want its want at F and no neither", w, out.Neither)
}
n.Haves["ace"]["mail"], n.Haves["g14"]["mail"] = failedHave, n.Haves["ace"]["mail"]
// ADR 0557 §2: a witness that is offline keeps the have it last wrote and witnesses nothing (D1); F failing on a second node is
// not one node's failure (D3). Either way F is held and ace is set back.
for _, c := range []struct {
name string
online map[string]bool
novox delivery.Health
}{
{"the witness sleeps", map[string]bool{"ace": true, "g14": false, "novox": true}, delivery.HealthOK},
{"F fails on a second node", map[string]bool{"ace": true, "g14": true, "novox": true}, delivery.HealthFailed},
} {
n.Online = c.online
n.Haves["novox"] = map[string]delivery.Have{"mail": {FP: fNew, Phase: delivery.PhaseRunning, Health: c.novox}}
if c.novox == delivery.HealthOK {
n.Haves["novox"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
}
before["novox"] = delivery.Want{FP: fNew, Composed: dig(fNew)}
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if out.Held == nil || out.Held.Node != "ace" || len(out.NodeFailed) != 0 {
t.Errorf("%s: held %+v, node failed %+v: want F held at ace", c.name, out.Held, out.NodeFailed)
}
if w := out.Wants["ace"]; w.FP != fOld {
t.Errorf("%s: ace's want = %+v, want set back", c.name, w)
}
}
} }
// 1297/B4: a module holding data is not set back; its want stays and says why. // 1297/B4: a module holding data is not set back; its want stays and says why.
@@ -34,7 +34,7 @@ func TestManifest(t *testing.T) {
slices.Sort(reads) slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants", want := []string{"mesh-build-go.steps", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"} "mesh-delivery-indexer.modules", "mesh-delivery-indexer.placement", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"}
if !reflect.DeepEqual(reads, want) { if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want) t.Errorf("reads = %v, want exactly %v", reads, want)
} }
@@ -12,6 +12,7 @@
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-indexer.modules",
"mesh-delivery-indexer.placement",
"mesh-delivery-teller.answers", "mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence", "mesh-delivery-watchman.presence",
"node-engine.have" "node-engine.have"
@@ -105,7 +105,7 @@ func TryTools(tr Trying) []stdio.Tool {
{Name: check.TryTool, {Name: check.TryTool,
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " + Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " + "its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " + "where it stands. A try of the same head that waits or runs is the answer to a second ask; a new try is refused while 4 tries run on a build node or 8 wait for one, mesh-wide; tries let in while waiting may later all run at once, and each node's build capacity bounds what runs on it. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused. A try whose ask no build node took within the step " + "the checker checks no level of (one whose trunk the gate does not name) is refused. A try whose ask no build node took within the step " +
"bound ends as error; its ask stays in the build seat's stream and may still run later, judged by nobody (issue 734).", "bound ends as error; its ask stays in the build seat's stream and may still run later, judged by nobody (issue 734).",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{ Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
@@ -135,7 +135,8 @@ func TryTools(tr Trying) []stdio.Tool {
}}, }},
{Name: check.TryReadTool, {Name: check.TryReadTool,
Description: "Where a try stands, as stored: queued, running, or its end (pass, fail, or error when it could not run), with what was said of it, " + Description: "Where a try stands, as stored: queued, running, or its end (pass, fail, or error when it could not run), with what was said of it, " +
"the head and the trunk commit it was merged onto, the time it took and what failed first. Read-only.", "the head and the trunk commit it was merged onto, the time it took and what failed first; while it runs, each ask it put as soon as it is put, " +
"with the node that took it, its state and the phase its step says (fetch and merge, a level's digest, lint, tests) and since when. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"id"}, "properties": map[string]any{ Input: map[string]any{"type": "object", "required": []string{"id"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"}}}, "id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"}}},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
@@ -427,7 +427,8 @@ func (b *bed) start(t *testing.T, env map[string]string) {
if err := os.WriteFile(broker, cred, 0o600); err != nil { if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
e := map[string]string{"MESH_BROKER_FILE": broker} // The settle window of a group's heads (check.DefaultSettle) is cut short: each bed's candidate is written whole.
e := map[string]string{"MESH_BROKER_FILE": broker, "MESH_CHECKER_SETTLE": "10ms"}
for k, v := range env { for k, v := range env {
e[k] = v e[k] = v
} }
@@ -189,6 +189,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil { if err != nil {
return err return err
} }
settleWindow, err := durationEnv(getenv, "MESH_CHECKER_SETTLE", check.DefaultSettle)
if err != nil {
return err
}
link := strings.TrimSpace(getenv("MESH_CHECKER_LINK")) link := strings.TrimSpace(getenv("MESH_CHECKER_LINK"))
if link == "" { if link == "" {
link = defaultLink link = defaultLink
@@ -221,6 +225,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Bound: checkBound, Bound: checkBound,
Link: link, Link: link,
Logf: logger.Printf, Logf: logger.Printf,
Settle: settleWindow,
} }
// The builder's builds say how the candidates' fingerprints stand (issue 1684); where its bucket is not there yet the line // The builder's builds say how the candidates' fingerprints stand (issue 1684); where its bucket is not there yet the line
// says nothing of the builds, until the checker starts again. // says nothing of the builds, until the checker starts again.
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 h1:ikD/ggf9/5k23sxJOX1PFcGzb4M3+QeC66AT0MyQ9Zs= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,42 @@
package check
import (
"fmt"
"strings"
)
// A change that adds or modifies a compiled program fails (issue 1577): it is build output, `go build` in a module's folder
// writes one beside go.mod, and `git add -A` takes it. The build module's fetch step, which holds the merged clone, reads the
// first bytes of each changed file and reports the programs (mesh-sdk buildstep.MergeInto); the checker judges the answer.
// NoProgramsReported is the warning of a build module whose fetch answered no programs field: an older module, which cannot
// tell, so the check does not refuse a program it did not look for.
const NoProgramsReported = "the build module did not report programs: a compiled program in the change is not refused"
// checkBinariesOf fails a pull request whose change adds or modifies a compiled program, naming each file. It judges the
// first fetch answer of the pull request, whichever build module gave it; a build module that reports no programs leaves a
// warning on the check and passes it.
func (st *state) checkBinariesOf(t *target) verdictPtr {
key := fmt.Sprintf("%s#%d", t.pull.Repository, t.pull.Number)
if t.fetched.Merge == "" {
return nil // no merge was made: nothing was changed to judge
}
if t.fetched.Programs == nil {
st.c.note(fmt.Sprintf("%s: %s (%s)", key, NoProgramsReported, t.module))
st.c.warn(key, fmt.Sprintf("a compiled program is not refused: %s reported no programs", t.module))
st.j.logf("%s: %s (%s)", key, NoProgramsReported, t.module)
return nil
}
var found []string
for _, p := range *t.fetched.Programs {
if p.Commit != "" {
found = append(found, fmt.Sprintf("%s: a compiled program added in %s and removed later: it stays in history; squash the branch (issue 1577)", p.Path, short12(p.Commit)))
continue
}
found = append(found, p.Path+": a compiled program is build output: remove it and add it to .gitignore (issue 1577)")
}
if len(found) > 0 {
return failed(t.node, fmt.Sprintf("%s: %s", key, strings.Join(found, "; ")))
}
return nil
}
@@ -0,0 +1,63 @@
package check
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func TestAChangeWhoseFetchReportsProgramsFailsNamingEachFile(t *testing.T) { // 1577
prog := func(p ...string) *[]ProgramFile {
out := []ProgramFile{}
for _, f := range p {
path, commit, _ := strings.Cut(f, "@") // `path@commit`: added by a commit of the branch and removed later
out = append(out, ProgramFile{Path: path, Kind: "ELF", Commit: commit})
}
return &out
}
for _, c := range []struct {
name string
programs *[]ProgramFile
fail []string // paths named; none passes
warned bool // the build module reported no programs field
}{
{"none found", prog(), nil, false},
{"one program", prog("modules/m/m"), []string{"modules/m/m"}, false},
{"two programs", prog("a/x", "b/y"), []string{"a/x", "b/y"}, false},
{"added and removed in the branch", prog("gone/prog@0123456789abcdef0123"), []string{"gone/prog: a compiled program added in 0123456789ab and removed later: it stays in history; squash the branch (issue 1577)"}, false},
{"an older build module reports nothing", nil, nil, true},
} {
b := newBench(t)
b.fetch.Merge, b.fetch.Programs = "mmmm", c.programs
cs := caseOf(t, pullOf(repoA, 7, head1))
v := b.judge(t, cs).Verdict
if got := slices.ContainsFunc(cs.Notes(), func(n string) bool { return strings.Contains(n, NoProgramsReported) }); got != c.warned {
t.Errorf("%s: warned = %v, want %v (%v)", c.name, got, c.warned, cs.Notes())
}
line := Line(v, cs, cs.Pulls[0], nil, nil, nil)
if got := strings.Contains(line, "a compiled program is not refused: mesh-build-go reported no programs"); got != c.warned {
t.Errorf("%s: the status line holds the warning = %v, want %v: %s", c.name, got, c.warned, line)
}
if len(c.fail) == 0 {
if v.State != delivery.VerdictPass {
t.Errorf("%s: verdict = %+v", c.name, v)
}
continue
}
if v.State != delivery.VerdictFail {
t.Errorf("%s: verdict = %+v, want a failure", c.name, v)
continue
}
for _, p := range c.fail {
want := p
if !strings.Contains(p, " added in ") {
want = p + ": a compiled program is build output: remove it and add it to .gitignore (issue 1577)"
}
if !strings.Contains(v.Failed.Words, want) {
t.Errorf("%s: words %q do not name %s", c.name, v.Failed.Words, p)
}
}
}
}
@@ -43,9 +43,16 @@ type Case struct {
Try *TryOptions Try *TryOptions
notesMu sync.Mutex notesMu sync.Mutex
notes []string // warnings are, by pull request, what part of its check did not run (Case.warn).
warnings map[string]string
notes []string
// asks are the ids of the asks the check put on the build seats' streams, in order: a try names them (Try.Asks). // asks are the ids of the asks the check put on the build seats' streams, in order: a try names them (Try.Asks).
asks []string asks []string
// Watch, when set, hears each ask the check puts, as it is put (step nil), and each value its step key takes while
// the check waits for it: the node that took it, the phase it says, its end. A try writes them into its record, so a
// running try names its asks and where each stands (issue 1890). Called from the check's goroutines; it must not block
// for long.
Watch func(askID string, step *delivery.Step)
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at. // Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a carry after a trunk or sibling move is asked on the // Kind is what asked the check (KindNew, KindNodes, KindTrunk): a carry after a trunk or sibling move is asked on the
@@ -186,6 +193,23 @@ func (c *Case) note(s string) {
} }
} }
// warn says of a pull request (pullKey) that a part of its check did not run, in words for its status line (issue 1577).
func (c *Case) warn(key, words string) {
c.notesMu.Lock()
defer c.notesMu.Unlock()
if c.warnings == nil {
c.warnings = map[string]string{}
}
c.warnings[key] = words
}
// Warning is what warn said of a pull request, or "".
func (c *Case) Warning(key string) string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return c.warnings[key]
}
// Notes are the words noted on the case, in the order they were noted. // Notes are the words noted on the case, in the order they were noted.
func (c *Case) Notes() []string { func (c *Case) Notes() []string {
c.notesMu.Lock() c.notesMu.Lock()
@@ -85,6 +85,10 @@ type Checker struct {
Announce Announcer Announce Announcer
Now func() time.Time Now func() time.Time
Logf func(format string, args ...any) Logf func(format string, args ...any)
// Settle is how long a group's heads must stay the same after the candidate last changed them before its check starts
// (issue 1890): the members of a group are announced one after another, and a check of each partial set is thrown
// away at the next. None when 0; DefaultSettle in the module.
Settle time.Duration
once sync.Once once sync.Once
wg sync.WaitGroup wg sync.WaitGroup
@@ -96,15 +100,77 @@ type Checker struct {
epoch uint64 // changes of the nodes seen since this process started epoch uint64 // changes of the nodes seen since this process started
nodesFail map[string]uint64 // by verdict key: the epoch a failure of the nodes was written at nodesFail map[string]uint64 // by verdict key: the epoch a failure of the nodes was written at
held map[string]string // by verdict key: the turn that holds its check off, as last said held map[string]string // by verdict key: the turn that holds its check off, as last said
live map[string]liveCheck // by verdict key: the check running in this process, ended when its heads go
settling map[string]*window // by candidates key: the window its heads settle in
}
// liveCheck is a check running in this process: the candidates key of its group and the end of its run.
type liveCheck struct {
group string
end context.CancelCauseFunc
}
// window is a group's settle window: aborted with errUnsettled when its heads change again or the group goes, and with
// errSettled to end it at once.
type window struct{ abort context.CancelCauseFunc }
var (
errUnsettled = errors.New("the group's heads changed again")
errSettled = errors.New("the group's heads settled")
)
// settleNow ends the settle window of every group at once, as if Settle had passed.
func (ch *Checker) settleNow() {
ch.mu.Lock()
defer ch.mu.Unlock()
for _, w := range ch.settling {
w.abort(errSettled)
}
}
// DefaultSettle is the module's Settle (issue 1890). On 2026-10-26 the three members of a group were announced within 2.8
// seconds of each other (20:31:52.3, :53.7, :55.1 UTC), a session opening or pushing them one API call after another;
// fifteen seconds holds five times that, and costs a check of 10 to 18 minutes under 2.5 % of its time. A member that comes
// later still is no waste beyond its asks already taken: the check of the heads it replaced is ended at once.
const DefaultSettle = 15 * time.Second
// EndedWords open the words of a verdict the checker ended without judging it, its heads no candidate's any more (issue
// 1890): never a failure of the change, and judged whole if a candidate holds those heads again.
const EndedWords = "ended: "
// IsEnded says whether a verdict was ended without being judged (EndedWords).
func IsEnded(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && v.Failed.Machine == "checker" && strings.HasPrefix(v.Failed.Words, EndedWords)
}
// superseded is the cause a running check is ended with: the heads that replaced its own, or none when its group went.
type superseded struct{ by string }
func (s superseded) Error() string {
if s.by == "" {
return EndedWords + "no candidate holds these heads any more: the group was closed or merged"
}
return EndedWords + "superseded by " + s.by
} }
func (ch *Checker) init() { func (ch *Checker) init() {
ch.once.Do(func() { ch.once.Do(func() {
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{} ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{} ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{}
ch.live, ch.settling = map[string]liveCheck{}, map[string]*window{}
}) })
} }
// endOthers ends every check of the group running in this process whose verdict key is not keep: no candidate holds its
// heads any more (issue 1890). Called with ch.mu held.
func (ch *Checker) endOthers(group, keep, by string) {
for k, l := range ch.live {
if l.group == group && k != keep {
l.end(superseded{by: by})
}
}
}
func (ch *Checker) logf(format string, args ...any) { func (ch *Checker) logf(format string, args ...any) {
if ch.Logf != nil { if ch.Logf != nil {
ch.Logf(format, args...) ch.Logf(format, args...)
@@ -137,17 +203,62 @@ func (ch *Checker) Candidate(ctx context.Context, key string, value []byte) {
ch.logf("candidates.%s holds what is not a candidate, left: %v", key, err) ch.logf("candidates.%s holds what is not a candidate, left: %v", key, err)
return return
} }
vkey, _ := delivery.CheckerVerdictKey(c.Group, c.HeadsHash)
ch.mu.Lock() ch.mu.Lock()
prev, had := ch.known[key]
ch.known[key] = c ch.known[key] = c
// A check of heads this candidate no longer holds is ended at once (issue 1890): a member's new head, a member joining
// or leaving.
ch.endOthers(key, vkey, GateHeads(c))
moved := !had || prev.HeadsHash != c.HeadsHash
if ch.Settle > 0 && (moved || ch.settling[key] != nil) {
// The group's heads changed: its check waits until they have been the same for Settle, so a group whose members
// are announced one after another is checked once, whole. A later change starts the window again. The wait is a
// context's bound, no timer (ADR 0437 §8).
if w := ch.settling[key]; w != nil {
w.abort(errUnsettled)
}
base, abort := context.WithCancelCause(ctx)
w := &window{abort: abort}
ch.settling[key] = w
ch.wg.Add(1)
go func() {
defer ch.wg.Done()
wait, done := context.WithTimeout(base, ch.Settle)
defer done()
<-wait.Done()
settled := errors.Is(wait.Err(), context.DeadlineExceeded) || errors.Is(context.Cause(wait), errSettled)
ch.mu.Lock()
if !settled || ch.settling[key] != w {
ch.mu.Unlock()
return // changed again, forgotten, or the checker stops
}
delete(ch.settling, key)
now, ok := ch.known[key]
ch.mu.Unlock()
abort(nil)
if ok {
ch.consider(ctx, key, now)
}
}()
ch.mu.Unlock()
return
}
ch.mu.Unlock() ch.mu.Unlock()
ch.consider(ctx, key, c) ch.consider(ctx, key, c)
} }
// Forget takes a candidate out (its key was deleted). // Forget takes a candidate out (its key was deleted): a check of its heads still running is ended, as no candidate holds
// them any more.
func (ch *Checker) Forget(key string) { func (ch *Checker) Forget(key string) {
ch.init() ch.init()
ch.mu.Lock() ch.mu.Lock()
delete(ch.known, key) delete(ch.known, key)
if w := ch.settling[key]; w != nil {
w.abort(errUnsettled)
delete(ch.settling, key)
}
ch.endOthers(key, "", "")
ch.mu.Unlock() ch.mu.Unlock()
} }
@@ -190,32 +301,49 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
} }
cs.Queued = ch.now() cs.Queued = ch.now()
ch.mu.Lock() ch.mu.Lock()
if ch.settling[key] != nil {
ch.mu.Unlock()
return // its heads are settling: the window's end considers it (Settle)
}
if ch.inflight[cs.Key] { if ch.inflight[cs.Key] {
ch.again[cs.Key] = true // looked at once more when the check ends, so a change of the trunk is never lost ch.again[cs.Key] = true // looked at once more when the check ends, so a change of the trunk is never lost
ch.mu.Unlock() ch.mu.Unlock()
return return
} }
ch.inflight[cs.Key] = true ch.inflight[cs.Key] = true
run, end := context.WithCancelCause(ctx)
ch.live[cs.Key] = liveCheck{group: key, end: end}
ch.mu.Unlock() ch.mu.Unlock()
ch.wg.Add(1) ch.wg.Add(1)
go func() { go func() {
defer ch.wg.Done() defer ch.wg.Done()
vkey := cs.Key
defer func() {
ch.mu.Lock()
delete(ch.inflight, vkey)
delete(ch.again, vkey)
delete(ch.live, vkey)
ch.mu.Unlock()
end(nil)
}()
for { for {
// Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a // Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a
// failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9; // failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9;
// ADR 0536 §4). The checker holds no place and no capacity of its own. // ADR 0536 §4). The checker holds no place and no capacity of its own.
ch.prepare(ctx, cs) ch.prepare(ctx, cs)
ch.try(ctx, key, cs) ch.try(ctx, run, key, cs)
ch.mu.Lock() ch.mu.Lock()
more := ch.again[cs.Key] more := ch.again[vkey]
delete(ch.again, cs.Key) delete(ch.again, vkey)
if !more || ctx.Err() != nil { ch.mu.Unlock()
delete(ch.inflight, cs.Key) if !more || run.Err() != nil {
ch.mu.Unlock()
return return
} }
ch.mu.Unlock() // Looked at again for the same heads only: a candidate that moved to other heads has its own check (issue 1890).
if next := CaseOf(ch.candidate(key, c), ch.heads()); next != nil { if next := CaseOf(ch.candidate(key, c), ch.heads()); next != nil {
if next.Key != vkey {
return
}
cs = next cs = next
} }
cs.Queued = ch.now() cs.Queued = ch.now()
@@ -252,6 +380,8 @@ func (ch *Checker) prepare(ctx context.Context, cs *Case) {
return return
case IsConflictFailure(v): case IsConflictFailure(v):
return // a head git refused: its next check is whole (ADR 0548 §7) return // a head git refused: its next check is whole (ADR 0548 §7)
case IsEnded(v):
return // ended unjudged when its heads went (issue 1890): held again, they are checked whole
} }
cs.Kind = KindTrunk cs.Kind = KindTrunk
if v.State == delivery.VerdictPass && ch.stale(v, cs) { if v.State == delivery.VerdictPass && ch.stale(v, cs) {
@@ -402,8 +532,8 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil return &claimed{rev}, nil
} }
return nil, nil return nil, nil
case ch.stale(cur, cs): case IsEnded(cur), ch.stale(cur, cs):
retake = true retake = true // heads held again after their check was ended unjudged (issue 1890), or a trunk that moved
case cur.State == delivery.VerdictFail && IsNodesFailure(cur): case cur.State == delivery.VerdictFail && IsNodesFailure(cur):
ch.mu.Lock() ch.mu.Lock()
at, ok := ch.nodesFail[cs.Key] at, ok := ch.nodesFail[cs.Key]
@@ -483,9 +613,13 @@ func IsNodesFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, UnheardWords) return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, UnheardWords)
} }
// try claims the check, judges it within its bound and writes the verdict and the owed statuses. A check that ends // try claims the check, judges it within its bound under run and writes the verdict and the owed statuses. A check that ends
// because ctx did is left running: the checker takes it again when it starts. // because ctx did is left running: the checker takes it again when it starts. A check whose run was ended because no
func (ch *Checker) try(ctx context.Context, key string, cs *Case) { // candidate holds its heads any more (issue 1890) is ended: its verdict says so, and no status is written.
func (ch *Checker) try(ctx, run context.Context, key string, cs *Case) {
if run.Err() != nil {
return // ended before it was claimed: nothing to write
}
ch.mu.Lock() ch.mu.Lock()
first := !ch.tried[cs.Key] first := !ch.tried[cs.Key]
ch.tried[cs.Key] = true ch.tried[cs.Key] = true
@@ -501,21 +635,26 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
} }
began := ch.now() began := ch.now()
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{}) ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
out := ch.judge(ctx, cs) out := ch.judge(run, cs)
// An ask the build seat gave up on judged nothing (issue 734): the check is asked again under the same claim, its status // An ask the build seat gave up on judged nothing (issue 734): the check is asked again under the same claim, its status
// pending, and after LostAskTries it fails as not heard, judged again when the nodes change. // pending, and after LostAskTries it fails as not heard, judged again when the nodes change.
for try := 1; IsLostAsk(out.Verdict) && ctx.Err() == nil; try++ { for try := 1; IsLostAsk(out.Verdict) && run.Err() == nil; try++ {
if try >= LostAskTries { if try >= LostAskTries {
out.Verdict.Failed.Words = strings.TrimSpace(out.Verdict.Failed.Words + "\nasked " + fmt.Sprint(LostAskTries) + out.Verdict.Failed.Words = strings.TrimSpace(out.Verdict.Failed.Words + "\nasked " + fmt.Sprint(LostAskTries) +
" times and given up each time: " + UnheardWords) " times and given up each time: " + UnheardWords)
break break
} }
ch.logf("%s: %s; asked again (%d of %d)", cs.Key, out.Verdict.Failed.Words, try+1, LostAskTries) ch.logf("%s: %s; asked again (%d of %d)", cs.Key, out.Verdict.Failed.Words, try+1, LostAskTries)
out = ch.judge(ctx, cs) out = ch.judge(run, cs)
} }
if ctx.Err() != nil { if ctx.Err() != nil {
return return
} }
var why superseded
if errors.As(context.Cause(run), &why) {
ch.end(ctx, cs, cl, why)
return
}
v := out.Verdict v := out.Verdict
v.Group, v.Heads, v.Trunk, v.Beside = cs.Group, cs.Heads, cs.Trunk, nil v.Group, v.Heads, v.Trunk, v.Beside = cs.Group, cs.Heads, cs.Trunk, nil
for r, c := range out.Beside { for r, c := range out.Beside {
@@ -569,6 +708,26 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
ch.logf("%s: %s", cs.Key, v.State) ch.logf("%s: %s", cs.Key, v.State)
} }
// end writes the verdict of a check whose heads no candidate holds any more (issue 1890), by compare-and-set on its claim,
// and says it once in the log with the asks it had put: they stay in the build seats' streams, since an ask once put cannot
// be taken back (buildrun.Asker has no withdraw), and are no longer waited for. No status is written: the heads that
// replaced these have their own check, whose pending status stands.
func (ch *Checker) end(ctx context.Context, cs *Case, cl *claimed, why superseded) {
v := delivery.Verdict{Group: cs.Group, Heads: cs.Heads, Trunk: cs.Trunk, State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Machine: "checker", Words: why.Error()}}
data, err := delivery.EncodeVerdict(v)
if err == nil {
_, err = ch.Verdicts.Update(ctx, cs.Key, data, cl.rev)
}
if err != nil {
ch.logf("%s: %s, but the verdict could not say so: %v", cs.Key, why.Error(), err)
return
}
stage, _ := cs.Progress.Now()
asks := cs.Asks()
ch.logf("%s: %s; it was at %q with %d asks put %v, which are no longer waited for", cs.Key, why.Error(), stage, len(asks), asks)
}
// judge runs the case within the bound. A judge that fails to judge, and a check that ran past its bound, are verdicts of // judge runs the case within the bound. A judge that fails to judge, and a check that ran past its bound, are verdicts of
// failure naming the machine and the step. // failure naming the machine and the step.
func (ch *Checker) judge(ctx context.Context, cs *Case) Outcome { func (ch *Checker) judge(ctx context.Context, cs *Case) Outcome {
@@ -667,7 +826,7 @@ func (ch *Checker) Sweep(ctx context.Context) error {
continue continue
} }
cur.State = delivery.VerdictFail cur.State = delivery.VerdictFail
cur.Failed = delivery.FailedFirst{Machine: "checker", Words: "closed at start: left running, and no candidate holds these heads any more, so nothing will check them"} cur.Failed = delivery.FailedFirst{Machine: "checker", Words: EndedWords + "closed at start: left running, and no candidate holds these heads any more, so nothing will check them"}
data, err := delivery.EncodeVerdict(cur) data, err := delivery.EncodeVerdict(cur)
if err != nil { if err != nil {
return err return err
@@ -3,6 +3,7 @@ package check
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -333,7 +334,10 @@ func TestTheStatusIsPendingWhileTheCheckRunsAndEndsSuccessOrFailure(t *testing.T
} }
} }
func TestSweepClosesARunningKeyNoCandidateHoldsAndLeavesOneItDoes(t *testing.T) { // No check of heads no candidate holds is left running (issue 1890): at start the sweep ends a key left running; while the
// checker runs, a check whose group moved to other heads is ended at once, unjudged, and writes no status; heads held again
// are checked whole; and a group's check starts only once its heads settled, so a partial set of members is never checked.
func TestHeadsNoCandidateHoldsAreNeverLeftRunning(t *testing.T) {
ctx, j := context.Background(), passJudge() ctx, j := context.Background(), passJudge()
r := newRig(t, j) r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1)) cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
@@ -351,12 +355,91 @@ func TestSweepClosesARunningKeyNoCandidateHoldsAndLeavesOneItDoes(t *testing.T)
if err := r.ch.Sweep(ctx); err != nil { if err := r.ch.Sweep(ctx); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if v := r.verdict(t, dead); v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "closed at start") { if v := r.verdict(t, dead); !IsEnded(v) || !strings.Contains(v.Failed.Words, "closed at start") {
t.Errorf("the dead key = %+v", v) t.Errorf("the dead key = %+v", v)
} }
if v := r.verdict(t, live); v.State != delivery.VerdictRunning { if v := r.verdict(t, live); v.State != delivery.VerdictRunning {
t.Errorf("the live key = %+v, want it left for the check to take over", v) t.Errorf("the live key = %+v, want it left for the check to take over", v)
} }
// A member joins while the check of the lone member's heads runs: that check is ended, its judge's run cancelled.
started, ended := make(chan struct{}), make(chan error, 1)
j = &judgeFunc{f: func(run context.Context, _ *Case, call int) (Outcome, error) {
if call == 1 {
close(started)
<-run.Done()
ended <- context.Cause(run)
return Outcome{}, run.Err()
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
var logged []string
var logMu sync.Mutex
r.ch.Logf = func(f string, a ...any) { logMu.Lock(); logged = append(logged, fmt.Sprintf(f, a...)); logMu.Unlock() }
alone, both := cand, candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(repoSDK, 3, head2))
r.ch.Candidate(ctx, "feat%2Fx", alone)
<-started
r.ch.Candidate(ctx, "feat%2Fx", both)
var bothC delivery.Candidate
_ = json.Unmarshal(both, &bothC)
select {
case err := <-ended:
if err == nil || !strings.Contains(err.Error(), "superseded by "+GateHeads(bothC)) {
t.Errorf("the old check's run ended with %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("the check of the heads the candidate left still runs")
}
r.ch.Wait()
if v := r.verdict(t, live); !IsEnded(v) || v.Failed.Words != EndedWords+"superseded by "+GateHeads(bothC) {
t.Errorf("the superseded verdict = %+v", v.Failed)
}
if v := r.verdict(t, keyOf(t, "feat/x", both)); v.State != delivery.VerdictPass {
t.Errorf("the group's verdict = %s", v.State)
}
okey, _ := delivery.CheckerOwedKey(repoA, 7, head1)
if raw, _, _, _ := r.owed.Get(ctx, okey); !strings.Contains(string(raw), `"success"`) {
t.Errorf("the head's status = %s, want the group's success, never the ended check's", raw)
}
logMu.Lock()
said := 0
for _, l := range logged {
if strings.Contains(l, "ended: superseded") {
said++
}
}
logMu.Unlock()
if said != 1 {
t.Errorf("the end was said %d times: %v", said, logged)
}
// The member leaves again: the ended heads are held again and checked whole.
r.ch.Candidate(ctx, "feat%2Fx", alone)
r.ch.Wait()
if v := r.verdict(t, live); v.State != delivery.VerdictPass || j.count() != 3 {
t.Errorf("the heads held again = %s after %d judgings, want a pass of a third", v.State, j.count())
}
// Members announced one after another: only the settled, whole set is checked.
j = passJudge()
r = newRig(t, j)
r.ch.Settle = time.Hour
sdk := candidateOf(t, "feat/y", pullOf(repoSDK, 4, head2))
whole := candidateOf(t, "feat/y", pullOf(repoSDK, 4, head2), pullOf(repoA, 8, head1))
r.ch.Candidate(ctx, "feat%2Fy", sdk)
r.ch.Candidate(ctx, "feat%2Fy", whole)
r.ch.Reconsider(ctx) // a trunk move or a turn while the heads settle starts nothing
if j.count() != 0 {
t.Fatalf("judged %d times before the heads settled", j.count())
}
r.ch.settleNow()
r.ch.Wait()
if _, _, found, _ := r.verdicts.Get(ctx, keyOf(t, "feat/y", sdk)); found || j.count() != 1 {
t.Errorf("the partial set has a verdict (%v) or the judge ran %d times, want the whole set alone", found, j.count())
}
if v := r.verdict(t, keyOf(t, "feat/y", whole)); v.State != delivery.VerdictPass {
t.Errorf("the whole set = %s", v.State)
}
} }
func TestTheForgeBridgeIsToldOnceForEachChangeOfAStatus(t *testing.T) { // 1683 (the checker's half of AC5) func TestTheForgeBridgeIsToldOnceForEachChangeOfAStatus(t *testing.T) { // 1683 (the checker's half of AC5)
@@ -427,8 +427,14 @@ func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, l
if err := st.j.Asks.Put(ctx, a); err != nil { if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err)) return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
} }
waiting := ctx
if watch := st.c.Watch; watch != nil {
id := a.ID
watch(id, nil) // put: a try names it at once, not only when it ends (issue 1890)
waiting = buildrun.WithStepSeen(ctx, func(s delivery.Step) { watch(id, &s) })
}
var err error var err error
step, err = st.j.Asks.Wait(ctx, a) step, err = st.j.Asks.Wait(waiting, a)
if err != nil { if err != nil {
if ctx.Err() != nil { if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error()) return fail(t.module, ctx.Err().Error())
@@ -489,6 +495,9 @@ func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, l
if v := st.checkPinsOf(ctx, t); v != nil { if v := st.checkPinsOf(ctx, t); v != nil {
return v return v
} }
if v := st.checkBinariesOf(t); v != nil {
return v
}
} }
ran := 0.0 ran := 0.0
for _, la := range answer.Levels { for _, la := range answer.Levels {
@@ -87,18 +87,20 @@ type pieces struct {
judged string judged string
// carried says, of a carried verdict, the commits it was carried from (ADR 0548 §8); never dropped. // carried says, of a carried verdict, the commits it was carried from (ADR 0548 §8); never dropped.
carried string carried string
// warning says that a part of the check did not run, for instance that an older build module could not refuse a compiled program; never dropped.
warning string
} }
// pullKey names a pull request of a case: `<owner/repo>#<number>`. // pullKey names a pull request of a case: `<owner/repo>#<number>`.
func pullKey(p Pull) string { return fmt.Sprintf("%s#%d", p.Repository, p.Number) } func pullKey(p Pull) string { return fmt.Sprintf("%s#%d", p.Repository, p.Number) }
func (p pieces) join(drop int) string { func (p pieces) join(drop int) string {
parts := []string{p.head, p.commits, p.carried, p.counts, p.judged} parts := []string{p.head, p.commits, p.carried, p.warning, p.counts, p.judged}
switch { switch {
case drop < 3 && p.times != "": case drop < 3 && p.times != "":
parts = append(parts, p.times) parts = append(parts, p.times)
case drop == 3 && p.shortTimes != "": case drop == 3 && p.shortTimes != "":
parts = []string{p.head, p.commits, p.carried, p.shortCounts, p.judged, p.shortTimes} parts = []string{p.head, p.commits, p.carried, p.warning, p.shortCounts, p.judged, p.shortTimes}
} }
if p.over != "" { if p.over != "" {
parts = append(parts, p.over) parts = append(parts, p.over)
@@ -210,6 +212,9 @@ func carriedWords(from map[string]string, v delivery.Verdict, cs *Case, p Pull)
func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) pieces { func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) pieces {
var pc pieces var pc pieces
pc.judged = judged[pullKey(p)] pc.judged = judged[pullKey(p)]
if cs != nil {
pc.warning = cs.Warning(pullKey(p))
}
commits := short12(p.Head) + " on " + short12(p.Trunk) commits := short12(p.Head) + " on " + short12(p.Trunk)
switch v.State { switch v.State {
case delivery.VerdictRunning: case delivery.VerdictRunning:
@@ -44,6 +44,17 @@ type Fetched struct {
Conflicts []string `json:"conflicts,omitempty"` Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"` Changed []string `json:"changed,omitempty"`
Trees map[string]MergeTree `json:"trees,omitempty"` Trees map[string]MergeTree `json:"trees,omitempty"`
// Programs are the changed files whose first bytes are an executable binary, read by the build module in its workspace
// (issue 1577); nil for a build module that does not report it, a pointer to an empty list for one that found none.
Programs *[]ProgramFile `json:"programs,omitempty"`
}
// ProgramFile is a changed file that is a compiled program, and its format (ELF, Mach-O or PE).
type ProgramFile struct {
Path string `json:"path"`
Kind string `json:"kind"`
// Commit is set for a program a commit of the branch added and a later one removed: it stays in history once merged.
Commit string `json:"commit,omitempty"`
} }
// GoPackage is one package of a Go test result. // GoPackage is one package of a Go test result.
@@ -10,7 +10,7 @@ TestFetchWithABaseMergesAndAnswersThePreview and TestFetchWithAConflictEndsPasse
into a throwaway test that wrote them out, at mesh-catalog feat/r4c5-checker d5804927 (the git commands are the test's fake, into a throwaway test that wrote them out, at mesh-catalog feat/r4c5-checker d5804927 (the git commands are the test's fake,
so the shas are its constants; the shape and the field names are the module's own marshalling). node-fetch-clean.json is the so the shas are its constants; the shape and the field names are the module's own marshalling). node-fetch-clean.json is the
FetchResult json.Marshal of mesh-build-node's fetch (fetch.go, preview.go) in its test TestFetchWithABaseMergesAndAnswersThePreview FetchResult json.Marshal of mesh-build-node's fetch (fetch.go, preview.go) in its test TestFetchWithABaseMergesAndAnswersThePreview
(cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679. (cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679. Both clean fetches carry `"programs":[]`, which a merging fetch of the build modules answers since issue 1577 (mesh-sdk buildstep.MergeInto's Programs), added by hand to the captured bytes: a fixture without it is an older build module's, and its line warns that no compiled program is refused.
forge-get-file-go-mod.json is the answer of the forge's gitea_get_file (mesh-catalog modules/gitea, tools/index.ts) for modules/mesh-delivery-builder/go.mod of forge-get-file-go-mod.json is the answer of the forge's gitea_get_file (mesh-catalog modules/gitea, tools/index.ts) for modules/mesh-delivery-builder/go.mod of
novox/mesh-delivery at main, and forge-compare-trunk.json the answer of gitea_compare for novox/mesh-sdk base main, head cb54756c3865 (a commit on the novox/mesh-delivery at main, and forge-compare-trunk.json the answer of gitea_compare for novox/mesh-sdk base main, head cb54756c3865 (a commit on the
@@ -1 +1 @@
{"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}} {"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"programs":[],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}}
@@ -1 +1 @@
{"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}} {"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"programs":[],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}}
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"regexp" "regexp"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -29,8 +30,13 @@ const (
// TriesState is the checker's state of tries: the bucket mesh-delivery-checker_tries, keys try-<unix nanoseconds>. // TriesState is the checker's state of tries: the bucket mesh-delivery-checker_tries, keys try-<unix nanoseconds>.
TriesState = "tries" TriesState = "tries"
TriesBucket = delivery.CheckerModule + "_" + TriesState TriesBucket = delivery.CheckerModule + "_" + TriesState
// MaxTries is how many tries may wait or run at once, mesh-wide: each holds a build node for minutes. // MaxTries and MaxTriesWaiting are checked when a try is asked, mesh-wide: a new try is refused while MaxTries tries run
MaxTries = 4 // on a build node or MaxTriesWaiting wait for one (queued, or their asks not yet taken by a node). It is no bound on
// what runs later: tries let in while waiting may later all run at once, and each node's build capacity bounds what
// runs on it. A try waiting behind the seat's other asks holds no node, so it no longer blocks the next try (issue
// 1890: at 19:53 on 2026-10-26 four tries sat queued behind 44 checks for up to 47 minutes and no fifth could be asked).
MaxTries = 4
MaxTriesWaiting = 8
// TryScan is how many of the newest tries a new ask reads to find the ones that wait or run (issue 1827). // TryScan is how many of the newest tries a new ask reads to find the ones that wait or run (issue 1827).
TryScan = 64 TryScan = 64
// TryKeep is how long an ended try's key is kept before the checker removes it. // TryKeep is how long an ended try's key is kept before the checker removes it.
@@ -79,12 +85,104 @@ type Try struct {
Machines []string `json:"machines,omitempty"` Machines []string `json:"machines,omitempty"`
// Asks are the ids of the asks the try put on the build seats' streams, each ending on the step key <node>.<id> of its // Asks are the ids of the asks the try put on the build seats' streams, each ending on the step key <node>.<id> of its
// build module: the delivery view counts those steps as a try's, not a candidate's check (build_usage). // build module: the delivery view counts those steps as a try's, not a candidate's check (build_usage).
Asks []string `json:"asks,omitempty"` Asks []string `json:"asks,omitempty"`
// Steps say where each ask stands while the try runs and after: asked until a node takes it, then the node, its
// state, the phase its step says and when (issue 1890: a 31-minute try could not be told queued from running).
Steps []TryStep `json:"steps,omitempty"`
Asked time.Time `json:"asked"` Asked time.Time `json:"asked"`
Started time.Time `json:"started,omitzero"` Started time.Time `json:"started,omitzero"`
Ended time.Time `json:"ended,omitzero"` Ended time.Time `json:"ended,omitzero"`
} }
// TryAsked is a TryStep's state before a build node took its ask: it waits in the seat's stream.
const TryAsked = "asked"
// TryStep is one ask of a try as its step key last said it.
type TryStep struct {
Ask string `json:"ask"`
Node string `json:"node,omitempty"`
State string `json:"state"`
Phase string `json:"phase,omitempty"`
Started time.Time `json:"started,omitzero"`
PhaseAt time.Time `json:"phase-at,omitzero"`
Ended time.Time `json:"ended,omitzero"`
}
// tryWriter writes a running try's record again as its asks are put and their steps move: one writer per try, holding
// the revision it wrote last, so the try's end is written at that revision. A write refused (the try ended by hand)
// stops the writes; the end's own write then says it.
type tryWriter struct {
mu sync.Mutex
tr *Tryer
ctx context.Context
key string
t Try
rev uint64
stopped bool
}
// heard takes an ask put (step nil) or a value of its step key in, and writes the record when it changed.
func (w *tryWriter) heard(askID string, step *delivery.Step) {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped {
return
}
changed := false
if !slices.Contains(w.t.Asks, askID) {
w.t.Asks, changed = append(w.t.Asks, askID), true
}
i := slices.IndexFunc(w.t.Steps, func(s TryStep) bool { return s.Ask == askID })
if i < 0 {
w.t.Steps, changed = append(w.t.Steps, TryStep{Ask: askID, State: TryAsked}), true
i = len(w.t.Steps) - 1
}
if step != nil {
now := TryStep{Ask: askID, Node: step.Node, State: step.State, Phase: step.Phase, Started: step.Started,
PhaseAt: step.PhaseAt, Ended: step.Ended}
changed = changed || now != w.t.Steps[i]
w.t.Steps[i] = now
}
if changed {
w.write()
}
}
// write writes the record at the revision written last; a refusal stops the writes.
func (w *tryWriter) write() {
data, err := encodeTry(w.t)
if err != nil {
return
}
rev, err := w.tr.Tries.Update(w.ctx, w.key, data, w.rev)
if err != nil {
w.stopped = true
w.tr.logf("%s: its asks' progress is no longer written: %v", w.key, err)
return
}
w.rev = rev
}
// end stops the writes and answers the record and its revision, for the try's end.
func (w *tryWriter) end() (Try, uint64) {
w.mu.Lock()
defer w.mu.Unlock()
w.stopped = true
return w.t, w.rev
}
// HoldsANode says whether the try holds a build node now: it runs, and a step of its asks runs on a node, or it names
// no step (a checker that writes its asks only at the end, or one about to put its first).
func (t Try) HoldsANode() bool {
if t.State != TryRunning {
return false
}
if len(t.Steps) == 0 {
return true
}
return slices.ContainsFunc(t.Steps, func(s TryStep) bool { return s.State == delivery.StepRunning })
}
// Over says whether the try came to its end. // Over says whether the try came to its end.
func (t Try) Over() bool { return t.State == TryPass || t.State == TryFail || t.State == TryError } func (t Try) Over() bool { return t.State == TryPass || t.State == TryFail || t.State == TryError }
@@ -124,8 +222,8 @@ func NotChecked(repository string) error {
} }
// AskTry writes a new try, queued, and answers its id: a try of the same head that waits or runs is the answer to a second // AskTry writes a new try, queued, and answers its id: a try of the same head that waits or runs is the answer to a second
// ask, and at most MaxTries wait or run at once. trunks are the gate's trunk heads of every repository (source.main): a // ask, and a new try is refused while MaxTries tries run on a build node or MaxTriesWaiting wait for one. trunks are the
// repository with none is no repository the checker checks. A trunk asked by name is its trunk head now. // gate's trunk heads of every repository (source.main): a repository with none is no repository the checker checks. A trunk asked by name is its trunk head now.
func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (string, error) { func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (string, error) {
repo := strings.ToLower(strings.TrimSuffix(strings.TrimSpace(a.Repository), ".git")) repo := strings.ToLower(strings.TrimSuffix(strings.TrimSpace(a.Repository), ".git"))
head := strings.ToLower(strings.TrimSpace(a.Head)) head := strings.ToLower(strings.TrimSpace(a.Head))
@@ -188,7 +286,7 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
}); err != nil { }); err != nil {
return "", err return "", err
} }
var open []string var holding, waiting []string
for i, cur := range tries { for i, cur := range tries {
if !live[i] { if !live[i] {
continue continue
@@ -196,10 +294,18 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
if cur.Repository == t.Repository && cur.Head == t.Head && cur.Trunk == t.Trunk && cur.Tests == t.Tests && cur.Level == t.Level { if cur.Repository == t.Repository && cur.Head == t.Head && cur.Trunk == t.Trunk && cur.Tests == t.Tests && cur.Level == t.Level {
return cur.ID, nil return cur.ID, nil
} }
open = append(open, fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State)) said := fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State)
if cur.HoldsANode() {
holding = append(holding, said)
} else {
waiting = append(waiting, said)
}
} }
if len(open) >= MaxTries { // Checked at this ask only: a try queued behind the seat's checks holds no node; tries let in while waiting may later
return "", fmt.Errorf("%d tries wait or run, the most at once: %s; ask again when one ends", len(open), strings.Join(open, ", ")) // all run at once, and each node's build capacity bounds what runs on it.
if len(holding) >= MaxTries || len(waiting) >= MaxTriesWaiting {
return "", fmt.Errorf("a new try is refused while %d tries run on a build node or %d wait for one; now %d run and %d wait: running %s; waiting %s; ask again when one ends",
MaxTries, MaxTriesWaiting, len(holding), len(waiting), orNone(holding), orNone(waiting))
} }
t.ID = "try-" + strconv.FormatInt(now.UnixNano(), 10) t.ID = "try-" + strconv.FormatInt(now.UnixNano(), 10)
data, err := encodeTry(t) data, err := encodeTry(t)
@@ -395,7 +501,8 @@ func (tr *Tryer) run(ctx context.Context, key string) {
} }
return return
} }
cs := &Case{Key: key, Group: key, progress := &tryWriter{tr: tr, ctx: ctx, key: key, t: t, rev: rev}
cs := &Case{Key: key, Group: key, Watch: progress.heard,
Pulls: []Pull{{Repository: t.Repository, Head: t.Head, Trunk: t.Trunk}}, Pulls: []Pull{{Repository: t.Repository, Head: t.Head, Trunk: t.Trunk}},
Trunk: map[string]string{t.Repository: t.Trunk}, Trunk: map[string]string{t.Repository: t.Trunk},
Queued: t.Asked, Queued: t.Asked,
@@ -409,6 +516,7 @@ func (tr *Tryer) run(ctx context.Context, key string) {
if ctx.Err() != nil { if ctx.Err() != nil {
return // left running: the checker takes it over when it starts; or ended by hand, which wrote its end return // left running: the checker takes it over when it starts; or ended by hand, which wrote its end
} }
t, rev = progress.end()
v := out.Verdict v := out.Verdict
t.Ended = tr.now().UTC() t.Ended = tr.now().UTC()
t.Took = t.Ended.Sub(t.Started).Round(time.Second).String() t.Took = t.Ended.Sub(t.Started).Round(time.Second).String()
@@ -470,3 +578,11 @@ func judgeWithin(ctx context.Context, j Judge, cs *Case, bound time.Duration) (O
} }
return out, true return out, true
} }
// orNone joins the tries said, or says none.
func orNone(said []string) string {
if len(said) == 0 {
return "none"
}
return strings.Join(said, ", ")
}
@@ -3,6 +3,7 @@ package check
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -23,6 +24,19 @@ func (m memTries) Create(key string, value []byte) error {
return err return err
} }
// writtenTries is a memKV that keeps every running try it was given to write, in order.
type writtenTries struct {
*memKV
running []Try
}
func (w *writtenTries) Update(ctx context.Context, key string, value []byte, rev uint64) (uint64, error) {
if t, err := DecodeTry(value); err == nil && !t.Over() {
w.running = append(w.running, t)
}
return w.memKV.Update(ctx, key, value, rev)
}
var tryTrunks = map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA} var tryTrunks = map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA}
func readTry(t testing.TB, kv *memKV, id string) Try { func readTry(t testing.TB, kv *memKV, id string) Try {
@@ -54,14 +68,38 @@ func TestATryIsAskedQueuedOncePerHeadAndAtMostFourAtOnce(t *testing.T) { // ADR
if err != nil || again != id { if err != nil || again != id {
t.Errorf("a second ask of the same head answered %q, %v, want %q", again, err, id) t.Errorf("a second ask of the same head answered %q, %v, want %q", again, err, id)
} }
heads := []string{head2, trunkA2, trunkS2} // Issue 1890: tries on a build node (a step of theirs running, or none named yet) and tries waiting for one (queued, or
for i, h := range heads { // their asks not taken) are bounded apart, MaxTries and MaxTriesWaiting; the refusal names both counts.
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoA, Head: h, Trunk: trunkA}, at.Add(time.Duration(i+2)*time.Second)); err != nil { onNode := func(i int) Try {
t.Fatalf("try %d: %v", i+2, err) return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
} Steps: []TryStep{{Ask: "a", State: delivery.StepRunning}}}
} }
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute)); err == nil || !strings.Contains(err.Error(), "4 tries wait or run") { waitingFor := func(i int) Try {
t.Errorf("a fifth try answered %v, want refused naming the four", err) return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
Steps: []TryStep{{Ask: "a", State: TryAsked}}}
}
for _, c := range []struct {
what string
holding, wait int
refused string
}{
{"four queued behind the checks, none on a node: a fifth is taken", 0, 4, ""},
{"four on a node", MaxTries, 0, "now 4 run and 0 wait"},
{"three on a node, eight waiting", 3, MaxTriesWaiting, "a new try is refused while 4 tries run on a build node or 8 wait for one; now 3 run and 8 wait"},
} {
kv := newMemKV()
for i := range c.holding + c.wait {
tr := waitingFor(i + 1)
if i < c.holding {
tr = onNode(i + 1)
}
data, _ := encodeTry(tr)
_, _ = kv.Create(context.Background(), tr.ID, data)
}
_, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute))
if (c.refused == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), c.refused)) {
t.Errorf("%s: answered %v, want refused %q", c.what, err, c.refused)
}
} }
} }
@@ -89,7 +127,8 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
t.Fatal(err) t.Fatal(err)
} }
b := newBench(t) b := newBench(t)
tr := &Tryer{Tries: kv, Judge: b.ex} written := &writtenTries{memKV: kv}
tr := &Tryer{Tries: written, Judge: b.ex}
raw, _, _, _ := kv.Get(ctx, id) raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw) tr.Put(ctx, id, raw)
tr.Wait() tr.Wait()
@@ -97,6 +136,15 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
if got.State != TryPass || got.Took == "" || got.Head != head1 || got.Trunk != trunkA || got.Said == "" || got.Levels[delivery.LevelUnit].Run != 2 { if got.State != TryPass || got.Took == "" || got.Head != head1 || got.Trunk != trunkA || got.Said == "" || got.Levels[delivery.LevelUnit].Run != 2 {
t.Fatalf("try = %+v", got) t.Fatalf("try = %+v", got)
} }
// Issue 1890: while it runs, the try names its ask as soon as it is put, and then where its step stands; the end
// keeps each ask's node and state.
if len(written.running) < 3 || len(written.running[1].Asks) != 1 || written.running[1].Steps[0].State != TryAsked ||
written.running[2].Steps[0].Node != "shanks" {
t.Errorf("the running try's writes = %+v, want its ask named while it ran, then its node", written.running)
}
if len(got.Steps) != 1 || got.Steps[0].Node != "shanks" || got.Steps[0].State != delivery.StepPassed {
t.Errorf("the ended try's steps = %+v", got.Steps)
}
// The try names the asks it put, so the delivery view counts their steps as a try's (build_usage). // The try names the asks it put, so the delivery view counts their steps as a try's (build_usage).
if len(got.Asks) != 1 { if len(got.Asks) != 1 {
t.Errorf("asks = %v, want the one check ask the try put", got.Asks) t.Errorf("asks = %v, want the one check ask the try put", got.Asks)
@@ -91,6 +91,28 @@ func treesVia(ctx context.Context, call caller) bridge.Trees {
} }
} }
// listedVia asks the forge for the files directly under one directory of a commit through call: gitea_commit_trees with
// listed (issue 1126), answer {listed: {directory: {name: blob id}}}.
func listedVia(ctx context.Context, call caller) bridge.Listed {
return func(owner, repo, sha, dir string) (map[string]string, error) {
raw, err := call(ctx, commitTreesTool, map[string]any{"owner": owner, "repo": repo, "sha": sha, "listed": []string{dir}})
if err != nil {
return nil, err
}
var a struct {
Listed map[string]map[string]string `json:"listed"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("the answer of %s is not {listed}: %w", commitTreesTool, err)
}
if a.Listed == nil {
// A forge module from before listed answers none: placement cannot be read, never read as empty.
return nil, fmt.Errorf("the answer of %s lists no directory; the forge module does not list yet", commitTreesTool)
}
return a.Listed[dir], nil
}
}
// getFileTool is the forge's tool for one file at a ref (mesh-catalog modules/gitea tools/index.ts gitea_get_file): // getFileTool is the forge's tool for one file at a ref (mesh-catalog modules/gitea tools/index.ts gitea_get_file):
// arguments owner, repo, path and ref, answer {file: {path, ref, sha, size, content}}, the content decoded to text // arguments owner, repo, path and ref, answer {file: {path, ref, sha, size, content}}, the content decoded to text
// (client.ts getFile). A path the forge does not have fails with `Gitea API <path>: 404 <body>` (client.ts request). // (client.ts getFile). A path the forge does not have fails with `Gitea API <path>: 404 <body>` (client.ts request).
@@ -117,8 +117,8 @@ type publisher interface {
// handle turns one event into its message on FORGE and acknowledges it. An event it cannot read is terminated, never // handle turns one event into its message on FORGE and acknowledges it. An event it cannot read is terminated, never
// redelivered, and the log names its subject and why. A message it could not write stays unacknowledged: the server // redelivered, and the log names its subject and why. A message it could not write stays unacknowledged: the server
// delivers it again (the server's ack wait is the only clock). // delivers it again (the server's ack wait is the only clock).
func handle(ctx context.Context, m delivered, pub publisher, trees bridge.Trees, files bridge.Files, logger *log.Logger) { func handle(ctx context.Context, m delivered, pub publisher, trees bridge.Trees, files bridge.Files, list bridge.Listed, logger *log.Logger) {
msg, err := bridge.Convert(m.Subject(), m.Data(), trees, files) msg, err := bridge.ConvertWith(m.Subject(), m.Data(), trees, files, list)
switch { switch {
case errors.Is(err, bridge.ErrRetry): case errors.Is(err, bridge.ErrRetry):
logger.Printf("%v; heard again in %s", err, retryDelay) logger.Printf("%v; heard again in %s", err, retryDelay)
@@ -237,7 +237,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// Bounded: the turn is held, and no sync or other event goes on without it. // Bounded: the turn is held, and no sync or other event goes on without it.
hctx, cancel := context.WithTimeout(ctx, syncDeadline) hctx, cancel := context.WithTimeout(ctx, syncDeadline)
defer cancel() defer cancel()
handle(hctx, m, rec, treesVia(hctx, call), filesVia(hctx, call), logger) handle(hctx, m, rec, treesVia(hctx, call), filesVia(hctx, call), listedVia(hctx, call), logger)
}) })
}, },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) })) jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
@@ -81,7 +81,7 @@ func TestReadableEventIsWrittenOnForgeAndAcked(t *testing.T) { // 1235/AC2 (unit
}) })
var logs bytes.Buffer var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{} m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0)) handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if len(pub.subjects) != 1 || pub.subjects[0] != "mesh.stream.mesh-delivery-forge-bridge.event.forge.novox.mesh-delivery.pull" { if len(pub.subjects) != 1 || pub.subjects[0] != "mesh.stream.mesh-delivery-forge-bridge.event.forge.novox.mesh-delivery.pull" {
t.Fatalf("published on %v", pub.subjects) t.Fatalf("published on %v", pub.subjects)
} }
@@ -97,7 +97,7 @@ func TestReadableEventIsWrittenOnForgeAndAcked(t *testing.T) { // 1235/AC2 (unit
func TestUnreadableEventIsTerminatedAndLogged(t *testing.T) { // 1235/AC5 func TestUnreadableEventIsTerminatedAndLogged(t *testing.T) { // 1235/AC5
var logs bytes.Buffer var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(map[string]any{"head_sha": "short"})}, &fakePub{} m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(map[string]any{"head_sha": "short"})}, &fakePub{}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0)) handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if len(pub.subjects) != 0 { if len(pub.subjects) != 0 {
t.Errorf("a message was written: %v", pub.subjects) t.Errorf("a message was written: %v", pub.subjects)
} }
@@ -112,7 +112,7 @@ func TestUnreadableEventIsTerminatedAndLogged(t *testing.T) { // 1235/AC5
func TestUnwrittenMessageIsNotAcked(t *testing.T) { // 1235 (a failed write is redelivered by the server, not retried here) func TestUnwrittenMessageIsNotAcked(t *testing.T) { // 1235 (a failed write is redelivered by the server, not retried here)
var logs bytes.Buffer var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{err: errors.New("no stream")} m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{err: errors.New("no stream")}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0)) handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if m.acked != 0 || m.termd != 0 { if m.acked != 0 || m.termd != 0 {
t.Errorf("acked %d terminated %d, want neither", m.acked, m.termd) t.Errorf("acked %d terminated %d, want neither", m.acked, m.termd)
} }
@@ -134,7 +134,7 @@ func TestTreesRefusedLeavesTheEventForTenSecondsAndLogs(t *testing.T) { // 1236/
call := func(context.Context, string, any) (json.RawMessage, error) { call := func(context.Context, string, any) (json.RawMessage, error) {
return nil, errors.New("the tree listing of novox/mesh-delivery was cut") return nil, errors.New("the tree listing of novox/mesh-delivery was cut")
} }
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), log.New(&logs, "", 0)) handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), nil, log.New(&logs, "", 0))
if len(pub.subjects) != 0 { if len(pub.subjects) != 0 {
t.Errorf("a message was written: %v", pub.subjects) t.Errorf("a message was written: %v", pub.subjects)
} }
@@ -163,7 +163,7 @@ func TestMergeCallsTheToolOnceAndWritesTheMergedMessage(t *testing.T) { // 1236/
} }
return json.RawMessage(`{"root":"r0","trees":{"":"t0","modules/a":"t1"}}`), nil return json.RawMessage(`{"root":"r0","trees":{"":"t0","modules/a":"t1"}}`), nil
} }
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), log.New(&logs, "", 0)) handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), nil, log.New(&logs, "", 0))
if len(keys) == 0 || keys[0] != "gitea.gitea_commit_trees" || slices.Index(keys[1:], "gitea.gitea_commit_trees") >= 0 { if len(keys) == 0 || keys[0] != "gitea.gitea_commit_trees" || slices.Index(keys[1:], "gitea.gitea_commit_trees") >= 0 {
t.Fatalf("calls %v, want the trees once and first", keys) t.Fatalf("calls %v, want the trees once and first", keys)
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-forge-bridge
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9 h1:Ufmd2cYEXICQ817IGflccP9bexABE4WhIO49HPbaoSo= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -7,6 +7,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"maps"
"path" "path"
"regexp" "regexp"
"slices" "slices"
@@ -88,6 +89,17 @@ func unreadable(subject, why string, args ...any) error {
// the trunk asks trees once and becomes a merged message; a merge into any other branch is a closed pull message, as the // the trunk asks trees once and becomes a merged message; a merge into any other branch is a closed pull message, as the
// pull request left its group. A forge that cannot say is ErrRetry; a merge with no module.json is ErrNoModules. // pull request left its group. A forge that cannot say is ErrRetry; a merge with no module.json is ErrNoModules.
func Convert(subject string, data []byte, trees Trees, files Files) (Message, error) { func Convert(subject string, data []byte, trees Trees, files Files) (Message, error) {
return ConvertWith(subject, data, trees, files, nil)
}
// Listed asks the forge for the files directly under one directory of a commit, by name to blob id (gitea_commit_trees
// with listed); a directory the commit lacks lists none.
type Listed func(owner, repo, sha, dir string) (map[string]string, error)
// ConvertWith is Convert; with list, a merge of the catalogue repository (delivery.CatalogueRepository) also carries every
// placement file of its merge commit (novox/hq ADR 0549 §2, issue 1126), read whole or not at all: a forge that cannot
// say is ErrRetry, as for the manifests, so the indexer never takes a part of placement for all of it.
func ConvertWith(subject string, data []byte, trees Trees, files Files, list Listed) (Message, error) {
var action string var action string
switch subject { switch subject {
case UpdatedSubject: case UpdatedSubject:
@@ -114,7 +126,7 @@ func Convert(subject string, data []byte, trees Trees, files Files) (Message, er
return Message{}, unreadable(subject, "no branch") return Message{}, unreadable(subject, "no branch")
} }
if subject == MergedSubject && p.Base == Trunk { if subject == MergedSubject && p.Base == Trunk {
return merged(subject, p, trees, files) return merged(subject, p, trees, files, list)
} }
f := delivery.Forge{Kind: delivery.ForgePull, Repository: p.Owner + "/" + p.Repo, Branch: p.Head, Number: p.Number, f := delivery.Forge{Kind: delivery.ForgePull, Repository: p.Owner + "/" + p.Repo, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Action: action} Head: p.HeadSHA, Action: action}
@@ -227,7 +239,7 @@ func inputsOf(owner, repo, ref string, dirs []string, files Files) (manifests ma
} }
// merged is the merged message of a merge into the trunk: the trees of the merge commit, asked once. // merged is the merged message of a merge into the trunk: the trees of the merge commit, asked once.
func merged(subject string, p pull, trees Trees, files Files) (Message, error) { func merged(subject string, p pull, trees Trees, files Files, list Listed) (Message, error) {
if !sha40.MatchString(p.MergeSHA) { if !sha40.MatchString(p.MergeSHA) {
return Message{}, unreadable(subject, "the merge commit %q is not 40 hex digits", p.MergeSHA) return Message{}, unreadable(subject, "the merge commit %q is not 40 hex digits", p.MergeSHA)
} }
@@ -252,8 +264,15 @@ func merged(subject string, p pull, trees Trees, files Files) (Message, error) {
if err != nil { if err != nil {
return Message{}, unreadable(subject, "%v", err) return Message{}, unreadable(subject, "%v", err)
} }
body, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number, f := delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks}) Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks}
if list != nil && delivery.SameRepository(repository, delivery.CatalogueRepository) {
if f.Placements, err = placementsOf(p.Owner, p.Repo, p.MergeSHA, list, files); err != nil {
return Message{}, fmt.Errorf("%w: %s: the placement files of %s at the merge commit %s: %v", ErrRetry, subject, repository, p.MergeSHA, err)
}
f.PlacementsRead = true
}
body, err := delivery.EncodeForge(f)
if err != nil { if err != nil {
return Message{}, unreadable(subject, "%v", err) return Message{}, unreadable(subject, "%v", err)
} }
@@ -350,3 +369,30 @@ func OpenMessage(answer []byte) (Message, error) {
} }
return Message{Event: delivery.OpenEvent(), Data: body}, nil return Message{Event: delivery.OpenEvent(), Data: body}, nil
} }
// placementsOf is every placement file of the catalogue at a commit, by node, its bytes exact: each `<node>.json` directly
// under delivery.PlacementDir. A file whose name is no node is left out and the rest read; the catalogue's check refuses
// such a file before it merges (ADR 0549 §1). Nil when there is none.
func placementsOf(owner, repo, sha string, list Listed, files Files) (map[string]string, error) {
names, err := list(owner, repo, sha, delivery.PlacementDir)
if err != nil {
return nil, err
}
var out map[string]string
for _, name := range slices.Sorted(maps.Keys(names)) {
file := delivery.PlacementDir + "/" + name
node, ok := delivery.PlacementFileNode(file)
if !ok {
continue
}
raw, err := files(owner, repo, file, sha)
if err != nil {
return nil, fmt.Errorf("%s: %w", file, err)
}
if out == nil {
out = map[string]string{}
}
out[node] = string(raw)
}
return out, nil
}
@@ -79,6 +79,47 @@ func TestMergedCarriesManifestsAndLocks(t *testing.T) { // 1448/AC1
if want := map[string]map[string]string{"modules/a": {"go.sum": hexSum(sum)}}; !reflect.DeepEqual(f.Locks, want) { if want := map[string]map[string]string{"modules/a": {"go.sum": hexSum(sum)}}; !reflect.DeepEqual(f.Locks, want) {
t.Errorf("locks = %v, want %v", f.Locks, want) t.Errorf("locks = %v, want %v", f.Locks, want)
} }
if f.Placements != nil || f.PlacementsRead {
t.Errorf("a merge of another repository than the catalogue carries placement: %v", f.Placements)
}
// Issue 1126 (ADR 0549 §2): a merge of the catalogue carries every placement file of its merge commit, by node; a file
// whose name is no node is left out; a listing or a file the forge cannot give is a retry, never a part.
catalogue := mergedBody(map[string]any{"repo": "mesh-catalog"})
forge.have["placement/ace.json"] = []byte(`{"v":1,"node":"ace","modules":[]}`)
forge.have["placement/g14.json"] = []byte(`{"v":1,"node":"g14","modules":[{"module":"zsh"}]}`)
listing := map[string]string{"ace.json": "b1", "g14.json": "b2", "README.md": "b3", "a.b.json": "b4"}
var listedAt []string
list := func(owner, repo, sha, dir string) (map[string]string, error) {
listedAt = append(listedAt, owner+"/"+repo+"@"+sha+":"+dir)
return listing, nil
}
msg, err = ConvertWith(mergedSubject, catalogue, treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get, list)
if err != nil {
t.Fatal(err)
}
if f, err = delivery.DecodeForge(msg.Data); err != nil {
t.Fatal(err)
}
want := map[string]string{"ace": `{"v":1,"node":"ace","modules":[]}`, "g14": `{"v":1,"node":"g14","modules":[{"module":"zsh"}]}`}
if !f.PlacementsRead || !reflect.DeepEqual(f.Placements, want) || !reflect.DeepEqual(listedAt, []string{"novox/mesh-catalog@" + merge + ":placement"}) {
t.Errorf("placements = %v (read %v), listed at %v", f.Placements, f.PlacementsRead, listedAt)
}
listing = map[string]string{}
if msg, err = ConvertWith(mergedSubject, catalogue, treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get, list); err != nil {
t.Fatal(err)
}
if f, _ = delivery.DecodeForge(msg.Data); !f.PlacementsRead || f.Placements != nil {
t.Errorf("a catalogue with no placement file says it read none: %v, %v", f.Placements, f.PlacementsRead)
}
listing = map[string]string{"gone.json": "b5"}
if _, err := ConvertWith(mergedSubject, catalogue, treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get, list); !errors.Is(err, ErrRetry) || !strings.Contains(err.Error(), "placement/gone.json") {
t.Errorf("a placement file the forge cannot give: %v", err)
}
failing := func(_, _, _, _ string) (map[string]string, error) { return nil, errors.New("forge down") }
if _, err := ConvertWith(mergedSubject, catalogue, treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get, failing); !errors.Is(err, ErrRetry) {
t.Errorf("a listing the forge cannot give: %v", err)
}
} }
func TestPullWithKnownDirsCarriesTheSameAtTheHead(t *testing.T) { // 1448/AC2 func TestPullWithKnownDirsCarriesTheSameAtTheHead(t *testing.T) { // 1448/AC2
@@ -91,6 +91,58 @@ func pinned(want string) *tls.Config {
} }
} }
// awaitedStore is a state bound when it appears (process.AwaitBucket) as the indexer's store: until then every call is
// process.ErrNotYet.
type awaitedStore struct{ b *process.AwaitedBucket }
func (s awaitedStore) store() (kvStore, error) {
kv := s.b.KV()
if kv == nil {
return kvStore{}, process.ErrNotYet
}
return kvStore{kv}, nil
}
func (s awaitedStore) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
st, err := s.store()
if err != nil {
return nil, 0, false, err
}
return st.Get(ctx, key)
}
func (s awaitedStore) Create(ctx context.Context, key string, value []byte) error {
st, err := s.store()
if err != nil {
return err
}
return st.Create(ctx, key, value)
}
func (s awaitedStore) Update(ctx context.Context, key string, value []byte, revision uint64) error {
st, err := s.store()
if err != nil {
return err
}
return st.Update(ctx, key, value, revision)
}
func (s awaitedStore) Delete(ctx context.Context, key string, revision uint64) error {
st, err := s.store()
if err != nil {
return err
}
return st.Delete(ctx, key, revision)
}
func (s awaitedStore) Keys(ctx context.Context) ([]string, error) {
st, err := s.store()
if err != nil {
return nil, err
}
return st.Keys(ctx)
}
// kvStore is the candidates bucket as the indexer's store: compare-and-set through Create, Update and Delete at a revision. // kvStore is the candidates bucket as the indexer's store: compare-and-set through Create, Update and Delete at a revision.
type kvStore struct{ kv jetstream.KeyValue } type kvStore struct{ kv jetstream.KeyValue }
@@ -253,6 +305,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil { if err != nil {
return err return err
} }
// Its taken state (ADR 0556 clause 1) is bound when it appears, as every state a delivery adds (issue 1790): until then
// a merge's taken key is said and not written, and the rest of the indexer runs.
taken := process.AwaitBucket(ctx, js, conn, delivery.TakenBucket, "no repository's taken key is written", logger.Printf)
// Its placement state (ADR 0549 §2) is bound when it appears, as every state a delivery adds (issue 1790): until then a
// catalogue merge's placement is said and not written, and the rest of the indexer runs.
placement := process.AwaitBucket(ctx, js, conn, delivery.PlacementBucket, "no catalogue merge's placement is written", logger.Printf)
// The controller's build records only name the directory of a module whose key names none (fillPaths): without them // The controller's build records only name the directory of a module whose key names none (fillPaths): without them
// such a module waits for its path, every other is indexed (issue 1945). // such a module waits for its path, every other is indexed (issue 1945).
builds := process.OptionalBucket(ctx, js, delivery.BuildsBucket, "a module whose key names no directory is not given one from the controller's build records until the indexer starts again", logger.Printf) builds := process.OptionalBucket(ctx, js, delivery.BuildsBucket, "a module whose key names no directory is not given one from the controller's build records until the indexer starts again", logger.Printf)
@@ -286,8 +344,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err return err
} }
defer calls.Close() defer calls.Close()
ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Builds: known, Logf: logger.Printf, ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known, Logf: logger.Printf,
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call)} Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call),
Placement: awaitedStore{placement}, Say: func(event string, body any) error { return calls.Emit(event, body) }}
// The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key // The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key
// main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does // main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does
// not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate. // not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate.
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-indexer
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 h1:ikD/ggf9/5k23sxJOX1PFcGzb4M3+QeC66AT0MyQ9Zs= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61 h1:/c6ncICWjq55nBf8TlDM0ROe/PuH+qZIdnuVbtg8UuM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -9,6 +9,7 @@ import (
"strings" "strings"
"sync" "sync"
"sync/atomic" "sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
) )
@@ -47,7 +48,11 @@ type Indexer struct {
Modules Store Modules Store
// Manifests is the manifests bucket; nil writes no manifests (issue 1449). // Manifests is the manifests bucket; nil writes no manifests (issue 1449).
Manifests Store Manifests Store
Builds *Builds // Taken is the taken bucket: per repository the last merge taken in (taken.go, ADR 0556); nil writes none.
Taken Store
// Now is the clock of a taken value; nil is time.Now.
Now func() time.Time
Builds *Builds
// Named asks the forge for the git object of each named path of the module directories given, at a commit // Named asks the forge for the git object of each named path of the module directories given, at a commit
// (gitea.gitea_commit_trees with paths; novox/hq ADR 0499 clauses 2 and 3): the indexer is the one reader of a // (gitea.gitea_commit_trees with paths; novox/hq ADR 0499 clauses 2 and 3): the indexer is the one reader of a
// manifest's paths (ADR 0495 clause 1, the lead's decision L8 of R4e). Nil asks nothing, and a module naming paths // manifest's paths (ADR 0495 clause 1, the lead's decision L8 of R4e). Nil asks nothing, and a module naming paths
@@ -60,6 +65,10 @@ type Indexer struct {
// Contains says whether a commit is the second or an ancestor of it (gitea.gitea_contains). // Contains says whether a commit is the second or an ancestor of it (gitea.gitea_contains).
Contains func(ctx context.Context, repository, commit, in string) (bool, error) Contains func(ctx context.Context, repository, commit, in string) (bool, error)
Logf func(format string, args ...any) Logf func(format string, args ...any)
// Placement is the placement bucket (delivery.PlacementBucket, ADR 0549 §2); nil writes no placement.
Placement Store
// Say says one of the indexer's events (placement-changed); nil says none.
Say func(event string, body any) error
mu sync.Mutex // guards source, toolchains and previews, which the watches and the forge messages all reach mu sync.Mutex // guards source, toolchains and previews, which the watches and the forge messages all reach
source sourceNow source sourceNow
@@ -94,8 +103,14 @@ func (ix *Indexer) Apply(ctx context.Context, f delivery.Forge) error {
if err := ix.applyMerged(ctx, f); err != nil { if err := ix.applyMerged(ctx, f); err != nil {
return err return err
} }
if err := ix.writeTaken(ctx, f); err != nil {
return err
}
ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945) ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945)
} }
if err := ix.applyPlacement(ctx, f); err != nil {
return err
}
var digests map[string]string var digests map[string]string
var held map[string]delivery.Module var held map[string]delivery.Module
if f.Kind == delivery.ForgePull { if f.Kind == delivery.ForgePull {
@@ -17,8 +17,9 @@ const (
// memStore is the candidates bucket's compare-and-set behaviour in memory: a revision per write, a conflict when the // memStore is the candidates bucket's compare-and-set behaviour in memory: a revision per write, a conflict when the
// revision read is no longer the key's. Writes counts the writes that went through; beforeWrite lets a test change the // revision read is no longer the key's. Writes counts the writes that went through; beforeWrite lets a test change the
// key under the indexer. // key under the indexer; unbound, when set, is every read's error (a state not yet on the bus).
type memStore struct { type memStore struct {
unbound error
vals map[string][]byte vals map[string][]byte
revs map[string]uint64 revs map[string]uint64
next uint64 next uint64
@@ -29,6 +30,9 @@ type memStore struct {
func newMem() *memStore { return &memStore{vals: map[string][]byte{}, revs: map[string]uint64{}} } func newMem() *memStore { return &memStore{vals: map[string][]byte{}, revs: map[string]uint64{}} }
func (m *memStore) Get(_ context.Context, key string) ([]byte, uint64, bool, error) { func (m *memStore) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
if m.unbound != nil {
return nil, 0, false, m.unbound
}
v, ok := m.vals[key] v, ok := m.vals[key]
return v, m.revs[key], ok, nil return v, m.revs[key], ok, nil
} }
@@ -43,6 +43,8 @@ func withManifests(f delivery.Forge, manifests map[string]string, locks map[stri
} }
func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1 func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1
// A merge's placement files are written as its manifests are (issue 1126): folded here to hold the count.
placementIsWrittenAtAMergeOnlyWhereItChanged(t)
avahi := catalogueManifest(t, "avahi") avahi := catalogueManifest(t, "avahi")
ix, mods, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs)) ix, mods, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil) f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil)
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
) )
const merge = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" const merge = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
@@ -94,22 +95,46 @@ func TestAnUnchangedModuleWritesNothing(t *testing.T) { // 1239 (unchanged value
// writes is the values written by then; commit is each module's commit after. // writes is the values written by then; commit is each module's commit after.
writes int writes int
commit map[string]string commit map[string]string
// taken is the merge the repository's taken key names after then (issue 1280, ADR 0556 clause 1).
taken string
}{ }{
{name: "the same merge again", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, {name: "the same merge again", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, taken: merge,
then: func(ix *Indexer) error { return ix.Apply(context.Background(), mergedMsg(repo, three)) }}, then: func(ix *Indexer) error { return ix.Apply(context.Background(), mergedMsg(repo, three)) }},
{name: "a merge touching one of three modules", writes: 1, commit: map[string]string{"a": next, "b": merge, "c": merge}, {name: "a merge touching one of three modules", writes: 1, commit: map[string]string{"a": next, "b": merge, "c": merge}, taken: next,
then: func(ix *Indexer) error { then: func(ix *Indexer) error {
m := mergedMsg(repo, map[string]string{"modules/a": "treeA2", "modules/b": "treeB", "modules/c": "treeC"}) m := mergedMsg(repo, map[string]string{"modules/a": "treeA2", "modules/b": "treeB", "modules/c": "treeC"})
m.Merge = next m.Merge = next
return ix.Apply(context.Background(), m) return ix.Apply(context.Background(), m)
}}, }},
{name: "a later source revision of another repository's merge", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, {name: "a merge touching none of them is still taken in", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, taken: next,
then: func(ix *Indexer) error {
m := mergedMsg(repo, three)
m.Merge = next
return ix.Apply(context.Background(), m)
}},
{name: "a merge while the taken state is not on the bus yet is still taken in", writes: 1, commit: map[string]string{"a": next, "b": merge, "c": merge}, taken: merge,
then: func(ix *Indexer) error { // issue 1790: the state is bound when it appears, so the indexer runs before it does
ix.Taken = &memStore{unbound: process.ErrNotYet}
m := mergedMsg(repo, map[string]string{"modules/a": "treeA2", "modules/b": "treeB", "modules/c": "treeC"})
m.Merge = next
return ix.Apply(context.Background(), m)
}},
{name: "an earlier merge heard again after a later one leaves taken", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, taken: merge,
then: func(ix *Indexer) error {
ix.Contains = func(_ context.Context, _, commit, in string) (bool, error) { return commit == next && in == merge, nil }
m := mergedMsg(repo, three)
m.Merge = next
return ix.Apply(context.Background(), m)
}},
{name: "a later source revision of another repository's merge", writes: 0, commit: map[string]string{"a": merge, "b": merge, "c": merge}, taken: merge,
then: func(ix *Indexer) error { then: func(ix *Indexer) error {
return ix.ApplySource(context.Background(), 9, delivery.Source{Group: "g", Heads: map[string]string{repo: merge, "novox/mesh-catalog": otherMerge}}) return ix.ApplySource(context.Background(), 9, delivery.Source{Group: "g", Heads: map[string]string{repo: merge, "novox/mesh-catalog": otherMerge}})
}}, }},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
ix, mods := modulesIndexer(t) ix, mods := modulesIndexer(t)
taken := newMem()
ix.Taken = taken
ctx := context.Background() ctx := context.Background()
if err := ix.Apply(ctx, mergedMsg(repo, three)); err != nil { if err := ix.Apply(ctx, mergedMsg(repo, three)); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -129,6 +154,10 @@ func TestAnUnchangedModuleWritesNothing(t *testing.T) { // 1239 (unchanged value
t.Errorf("%s = %+v (%v), want commit %.8s", module, m, ok, want) t.Errorf("%s = %+v (%v), want commit %.8s", module, m, ok, want)
} }
} }
key, _ := delivery.RepositoryKey(repo)
if got, err := delivery.DecodeTaken(taken.vals[key]); err != nil || got.Merge != tc.taken || got.Repository != repo {
t.Errorf("taken.%s = %+v (%v), want merge %.8s", key, got, err, tc.taken)
}
}) })
} }
} }
@@ -0,0 +1,123 @@
package index
import (
"context"
"errors"
"fmt"
"maps"
"slices"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// Placement of modules on nodes (novox/hq ADR 0549 §2, issue 1126): at each merge of the catalogue the forge bridge
// carries every placement file of the merge commit, and the indexer writes each node's key `placement.<node>` in its
// state `placement` only when it differs (the revision alone is no change), deletes the key of a node whose file is
// gone, and says `placement-changed` for each node it wrote or deleted, naming the modules added, removed or changed.
// A merge that changes no placement writes nothing and says nothing.
//
// A file that does not read as a placement, or names another node than its file, is said in the log and its key is left
// as it was: the catalogue's own check refuses such a file before it merges (ADR 0549 §1), so it is a fault of that
// check, and a node keeps composing from the placement it had rather than from none.
// applyPlacement takes the placement files of a merged message in; a message that read none (another repository, an
// older bridge) changes nothing.
func (ix *Indexer) applyPlacement(ctx context.Context, f delivery.Forge) error {
if ix.Placement == nil || f.Kind != delivery.ForgeMerged || !f.PlacementsRead {
return nil
}
keys, err := ix.Placement.Keys(ctx)
if errors.Is(err, process.ErrNotYet) {
// The state is not on the bus yet: said, and the merge's candidates are still taken out (issue 1790's binding
// writes the next catalogue merge's placement once it appears).
ix.logf("%s at %s: its placement files are not written: %v", f.Repository, f.Merge, err)
return nil
}
if err != nil {
return fmt.Errorf("reading the placement keys: %w", err)
}
gone := map[string]bool{}
for _, key := range keys {
if node, err := delivery.ParsePlacementKey(key); err == nil {
gone[node] = true
}
}
var refused []string
for _, node := range slices.Sorted(maps.Keys(f.Placements)) {
delete(gone, node)
next, err := delivery.DecodePlacement([]byte(f.Placements[node]))
if err == nil && next.Node != node {
err = fmt.Errorf("the file of %s names the node %q", node, next.Node)
}
if err != nil {
refused = append(refused, node)
ix.logf("%s at %s: the placement file of %s is refused and placement.%s is left as it was (the catalogue's check should have refused it): %v",
f.Repository, f.Merge, node, node, err)
continue
}
next.Revision = f.Merge
if err := ix.writePlacement(ctx, node, &next); err != nil {
return err
}
}
for _, node := range slices.Sorted(maps.Keys(gone)) {
if err := ix.writePlacement(ctx, node, nil); err != nil {
return err
}
}
if len(refused) > 0 {
// Said, and the merge's candidates are still taken out: a refused file holds nobody's delivery.
ix.logf("%s at %s: the placement files of %v do not read; their keys are left as they were", f.Repository, f.Merge, refused)
}
return nil
}
// writePlacement writes one node's key, or deletes it for next nil, only when it differs, and says what changed.
func (ix *Indexer) writePlacement(ctx context.Context, node string, next *delivery.Placement) error {
key, err := delivery.PlacementKey(node)
if err != nil {
return err
}
var change *delivery.PlacementChange
err = write(ctx, ix.Placement, key, func(value []byte, found bool) ([]byte, bool, error) {
change = nil
var before delivery.Placement
if found {
cur, err := delivery.DecodePlacement(value)
if err != nil {
ix.logf("placement.%s holds what is not a placement, written again: %v", node, err)
} else {
before = cur
}
}
if next == nil {
if !found {
return nil, false, nil
}
c := delivery.ChangeOf(before, delivery.Placement{Node: node})
change = &c
return nil, true, nil
}
if found && delivery.SamePlacement(before, *next) {
return nil, false, nil
}
data, err := delivery.EncodePlacement(*next)
if err != nil {
return nil, false, err
}
c := delivery.ChangeOf(before, *next)
change = &c
return data, false, nil
})
if err != nil || change == nil {
return err
}
if ix.Say != nil {
if err := ix.Say(delivery.PlacementChangedEvent, *change); err != nil {
// The key is written: a reader that watches it hears the change; the event is said loudly as not said.
ix.logf("placement.%s is written, and %s could not be said: %v", node, delivery.PlacementChangedEvent, err)
}
}
return nil
}
@@ -0,0 +1,97 @@
package index
import (
"context"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0549 §2 (issue 1126): a merge changing one node's file writes that key alone and says one event; a merge with no
// placement change writes nothing and says nothing; a node whose file is gone loses its key and says its modules
// removed; a file that does not read leaves its key; a message that read no placement changes none.
func placementIsWrittenAtAMergeOnlyWhereItChanged(t *testing.T) {
ctx := context.Background()
ix, _, logs := newIndexer(t)
place := newMem()
ix.Placement = place
var said []delivery.PlacementChange
ix.Say = func(event string, body any) error {
if event != delivery.PlacementChangedEvent {
t.Errorf("said %s", event)
}
said = append(said, body.(delivery.PlacementChange))
return nil
}
merged := func(merge string, files map[string]string) delivery.Forge {
return delivery.Forge{V: delivery.Version, Kind: delivery.ForgeMerged, Repository: delivery.CatalogueRepository, Branch: "main",
Number: 7, Head: "h", Merge: merge, Root: "r", Trees: map[string]string{"": "t"}, Placements: files, PlacementsRead: true}
}
ace := `{"v":1,"node":"ace","modules":[{"module":"plex"}]}`
g14 := `{"v":1,"node":"g14","modules":[{"module":"zsh"}]}`
written := func(node string) (delivery.Placement, uint64) {
t.Helper()
v, ok := place.vals["placement."+node]
if !ok {
return delivery.Placement{}, 0
}
p, err := delivery.DecodePlacement(v)
if err != nil {
t.Fatal(err)
}
return p, place.revs["placement."+node]
}
if err := ix.Apply(ctx, merged("m1", map[string]string{"ace": ace, "g14": g14})); err != nil {
t.Fatal(err)
}
if p, _ := written("ace"); p.Revision != "m1" || !reflect.DeepEqual(p.Assigned(), []string{"plex"}) || len(said) != 2 {
t.Fatalf("first merge: %+v, said %v", p, said)
}
_, aceRev := written("ace")
said = nil
if err := ix.Apply(ctx, merged("m2", map[string]string{"ace": ace, "g14": g14})); err != nil {
t.Fatal(err)
}
if p, rev := written("ace"); rev != aceRev || p.Revision != "m1" || len(said) != 0 {
t.Errorf("a merge with no placement change wrote %+v (revision %d, was %d) and said %v", p, rev, aceRev, said)
}
said = nil
if err := ix.Apply(ctx, merged("m3", map[string]string{"ace": ace, "g14": `{"v":1,"node":"g14","modules":[{"module":"zsh"},{"module":"xterm"}]}`})); err != nil {
t.Fatal(err)
}
if _, rev := written("ace"); rev != aceRev || len(said) != 1 || said[0].Node != "g14" || said[0].Revision != "m3" ||
!reflect.DeepEqual(said[0].Added, []string{"xterm"}) {
t.Errorf("one file changed: ace's revision %d (was %d), said %+v", rev, aceRev, said)
}
said = nil
if err := ix.Apply(ctx, merged("m4", map[string]string{"ace": ace, "g14": `{"v":1,"node":"ace","modules":[]}`})); err != nil {
t.Fatal(err)
}
if p, _ := written("g14"); len(p.Modules) != 2 || len(said) != 0 || !strings.Contains(logs.String(), "the placement file of %s is refused") {
t.Errorf("a file naming another node replaced the key: %+v, said %v, logs %s", p, said, logs)
}
said = nil
if err := ix.Apply(ctx, merged("m5", map[string]string{"ace": ace})); err != nil {
t.Fatal(err)
}
if _, rev := written("g14"); rev != 0 || len(said) != 1 || !reflect.DeepEqual(said[0].Removed, []string{"xterm", "zsh"}) {
t.Errorf("a file removed: g14's key at revision %d, said %+v", rev, said)
}
said = nil
unread := merged("m6", nil)
unread.PlacementsRead = false
if err := ix.Apply(ctx, unread); err != nil {
t.Fatal(err)
}
if p, _ := written("ace"); p.Revision != "m1" || len(said) != 0 {
t.Errorf("a message that read no placement changed it: %+v, %v", p, said)
}
}
@@ -164,6 +164,7 @@ func TestAPreviewHeardBeforeItsPullRequestGivesTheFingerprintWhenTheCandidateIsW
} }
func TestAPreviewNamingNoManifestOfTheModulesDirectoryGivesAReason(t *testing.T) { func TestAPreviewNamingNoManifestOfTheModulesDirectoryGivesAReason(t *testing.T) {
t.Run("the same preview again", testTheSamePreviewAppliedAgainWritesNothing)
ix, _, cands := previewBed(t, true) ix, _, cands := previewBed(t, true)
if err := ix.Apply(context.Background(), avahiPull(t, h1)); err != nil { if err := ix.Apply(context.Background(), avahiPull(t, h1)); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -176,7 +177,7 @@ func TestAPreviewNamingNoManifestOfTheModulesDirectoryGivesAReason(t *testing.T)
} }
} }
func TestTheSamePreviewAppliedAgainWritesNothing(t *testing.T) { func testTheSamePreviewAppliedAgainWritesNothing(t *testing.T) {
ix, _, cands := previewBed(t, true) ix, _, cands := previewBed(t, true)
if err := ix.Apply(context.Background(), avahiPull(t, h1)); err != nil { if err := ix.Apply(context.Background(), avahiPull(t, h1)); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -0,0 +1,60 @@
package index
import (
"context"
"errors"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// What the indexer took in (novox/hq ADR 0556 clause 1, issue 1280): after a merged message is applied whole, the key of
// its repository in the state `taken` names that merge, so a reader of the modules knows they say the repository at that
// merge or later. The indexer is the key's only writer. A merge heard again after a later one (a redelivery) does not put
// the key back: the key moves only to a merge that contains the one it names, as the forge says it; a forge that cannot
// say keeps the key as it is and says so.
// writeTaken writes the repository's taken key for the merge f carries, unless it names that merge or a later one.
func (ix *Indexer) writeTaken(ctx context.Context, f delivery.Forge) error {
if ix.Taken == nil || f.Merge == "" {
return nil
}
key, err := delivery.RepositoryKey(f.Repository)
if err != nil {
ix.logf("%s at %s: no taken key is written: %v", f.Repository, f.Merge, err)
return nil
}
err = write(ctx, ix.Taken, key, func(value []byte, found bool) ([]byte, bool, error) {
if found {
if cur, err := delivery.DecodeTaken(value); err == nil {
if cur.Merge == f.Merge {
return nil, false, nil
}
if ix.Contains != nil {
later, err := ix.Contains(ctx, f.Repository, f.Merge, cur.Merge)
if err != nil {
ix.logf("%s: whether %s is in %s cannot be read, so taken.%s stays at %s: %v", f.Repository, f.Merge, cur.Merge, key, cur.Merge, err)
return nil, false, nil
}
if later {
return nil, false, nil // a merge heard again after a later one
}
}
}
}
now := time.Now
if ix.Now != nil {
now = ix.Now
}
data, err := delivery.EncodeTaken(delivery.Taken{Repository: delivery.RepositoryName(f.Repository), Merge: f.Merge, At: now().UTC()})
return data, false, err
})
if errors.Is(err, process.ErrNotYet) {
// The state is not on the bus yet (a delivery that adds it, bound when it appears as issue 1790 says): said, and
// the merge's modules are still taken in; the next merge of the repository writes its key once the state appears.
ix.logf("%s at %s: its taken key is not written: %v", f.Repository, f.Merge, err)
return nil
}
return err
}
@@ -38,6 +38,7 @@ type manifest struct {
Version string `json:"version"` Version string `json:"version"`
Slug string `json:"slug"` Slug string `json:"slug"`
Tools []string `json:"tools"` Tools []string `json:"tools"`
Emits []string `json:"emits"`
Consumes []string `json:"consumes"` Consumes []string `json:"consumes"`
State []state `json:"state"` State []state `json:"state"`
Reads []string `json:"reads"` Reads []string `json:"reads"`
@@ -80,7 +81,11 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
if m.Module != "mesh-delivery-indexer" { if m.Module != "mesh-delivery-indexer" {
t.Errorf("module = %q", m.Module) t.Errorf("module = %q", m.Module)
} }
if want := []state{{Name: "candidates", History: 10}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) { // 1126 (ADR 0549 §2): the indexer's placement state, and the event it says when a node's placement changed.
if want := []string{"placement-changed"}; !reflect.DeepEqual(m.Emits, want) {
t.Errorf("emits = %v, want %v", m.Emits, want)
}
if want := []state{{Name: "candidates", History: 10}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "placement", History: 10}, {Name: "taken", History: 1}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want) t.Errorf("state = %+v, want %+v", m.State, want)
} }
if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-controller.assignments", "mesh-controller.builds", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) { if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-controller.assignments", "mesh-controller.builds", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) {
@@ -107,8 +112,8 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
if want := []string{"indexer_take_in"}; !reflect.DeepEqual(m.Tools, want) { if want := []string{"indexer_take_in"}; !reflect.DeepEqual(m.Tools, want) {
t.Errorf("tools = %v, want %v", m.Tools, want) t.Errorf("tools = %v, want %v", m.Tools, want)
} }
// No emits, no streams, no bus, no seat (ADR 0437 §4). // No streams, no bus, no seat (ADR 0437 §4); its one event is placement-changed (ADR 0549 §2), checked above.
for _, k := range []string{"streams", "logs", "emits", "claims", "provides", "requires", "uses", "defines-seats", "seat", "seats", "bus"} { for _, k := range []string{"streams", "logs", "claims", "provides", "requires", "uses", "defines-seats", "seat", "seats", "bus"} {
if _, ok := top[k]; ok { if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the indexer has none", k) t.Errorf("the manifest declares %q: the indexer has none", k)
} }
@@ -5,6 +5,9 @@
"tools": [ "tools": [
"indexer_take_in" "indexer_take_in"
], ],
"emits": [
"placement-changed"
],
"consumes": [ "consumes": [
"mesh-build-go.toolchain.reported", "mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported", "mesh-build-node.toolchain.reported",
@@ -15,6 +18,8 @@
{ "name": "candidates", "history": 10 }, { "name": "candidates", "history": 10 },
{ "name": "manifests", "history": 1 }, { "name": "manifests", "history": 1 },
{ "name": "modules", "history": 20 }, { "name": "modules", "history": 20 },
{ "name": "placement", "history": 10 },
{ "name": "taken", "history": 1 },
{ "name": "takeins", "history": 1 } { "name": "takeins", "history": 1 }
], ],
"reads": [ "reads": [
@@ -286,16 +286,17 @@ func (t *Teller) catchUp(id string) {
t.said(h, *w, err) t.said(h, *w, err)
} }
// mergeAct is what the approve option of the gate's ask binds: this ask, the answer, and the heads it covers. It is // mergeAct is what the approve option of the gate's ask binds: the ask's verb (delivery.AskVerb: mesh-delivery.merge, or
// spelled as the gate composes it (mesh-catalog modules/mesh-delivery-gate approval.go mergeAct, from // mesh-delivery.settle for a person's settle ask, novox/hq ADR 0556 clause 5), this ask, the answer, and the heads it covers.
// It is spelled as the gate composes it (mesh-catalog modules/mesh-delivery-gate approval.go mergeAct, from
// modules/mesh-delivery approval.go:63-65) and as mesh-controller verifies it (cmd/mesh-controller/moved.go:98-106). // modules/mesh-delivery approval.go:63-65) and as mesh-controller verifies it (cmd/mesh-controller/moved.go:98-106).
func mergeAct(ask, option string, heads map[string]string) asks.Act { func mergeAct(verb, ask, option string, heads map[string]string) asks.Act {
var out []string var out []string
for id, sha := range heads { for id, sha := range heads {
out = append(out, id+"="+sha) out = append(out, id+"="+sha)
} }
sort.Strings(out) sort.Strings(out)
return asks.Act{"verb": "mesh-delivery.merge", "ask": ask, "option": option, "heads": strings.Join(out, ",")} return asks.Act{"verb": verb, "ask": ask, "option": option, "heads": strings.Join(out, ",")}
} }
// headsOf is the heads an ask covers as an act names them: a delivery's id, `owner/repository@<twelve characters of its // headsOf is the heads an ask covers as an act names them: a delivery's id, `owner/repository@<twelve characters of its
@@ -363,7 +364,7 @@ func (t *Teller) Answer(o asks.Option, w asks.Warrant) error {
return fmt.Errorf("the gate's ask %s is no longer the ask the operator answered (its digest is %s, the warrant's %s)", return fmt.Errorf("the gate's ask %s is no longer the ask the operator answered (its digest is %s, the warrant's %s)",
s.Key, ga.Ask.Digest(), w.AskDigest) s.Key, ga.Ask.Digest(), w.AskDigest)
} }
if err := o.Performs(mergeAct(w.Ask, o.ID, headsOf(ga.Heads))); err != nil { if err := o.Performs(mergeAct(delivery.AskVerb(ga.Group), w.Ask, o.ID, headsOf(ga.Heads))); err != nil {
return err return err
} }
body, err := delivery.EncodeAnswer(delivery.Answer{Option: o.ID, Warrant: warrantID(w), Words: w.Says()}) body, err := delivery.EncodeAnswer(delivery.Answer{Option: o.ID, Warrant: warrantID(w), Words: w.Says()})
@@ -189,7 +189,13 @@ func gateAsk(t *testing.T, heads, boundTo []delivery.Head) (key string, value []
// go/delivery Ask.Asking), the first asking's the key's own. // go/delivery Ask.Asking), the first asking's the key's own.
func gateAskN(t *testing.T, heads, boundTo []delivery.Head, asking int) (key string, value []byte, ask delivery.Ask) { func gateAskN(t *testing.T, heads, boundTo []delivery.Head, asking int) (key string, value []byte, ask delivery.Ask) {
t.Helper() t.Helper()
key, err := delivery.MergeAskKey("g", heads) return gateAskOf(t, "g", "mesh-delivery.merge", heads, boundTo, asking)
}
// gateAskOf is gateAskN for a group whose options bind the verb given.
func gateAskOf(t *testing.T, group, verb string, heads, boundTo []delivery.Head, asking int) (key string, value []byte, ask delivery.Ask) {
t.Helper()
key, err := delivery.MergeAskKey(group, heads)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -197,13 +203,13 @@ func gateAskN(t *testing.T, heads, boundTo []delivery.Head, asking int) (key str
a := asks.Ask{ID: id, Headline: "Merge into main: novox/mesh-delivery#12", Explanation: "Merges two pull requests.\nReview: approved.", a := asks.Ask{ID: id, Headline: "Merge into main: novox/mesh-delivery#12", Explanation: "Merges two pull requests.\nReview: approved.",
Who: asks.Operator, Expires: now.Add(asks.ApproveLasts - 10*time.Minute), OnExpiry: "the merge stays held", About: "delivery.mesh-delivery.12"} Who: asks.Operator, Expires: now.Add(asks.ApproveLasts - 10*time.Minute), OnExpiry: "the merge stays held", About: "delivery.mesh-delivery.12"}
for _, o := range []struct{ id, label, does string }{{"approve", "Approve", "merges"}, {"decline", "Decline", "merges nothing"}} { for _, o := range []struct{ id, label, does string }{{"approve", "Approve", "merges"}, {"decline", "Decline", "merges nothing"}} {
binds, err := asks.ActDigest(mergeAct(id, o.id, headsOf(boundTo))) binds, err := asks.ActDigest(mergeAct(verb, id, o.id, headsOf(boundTo)))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
a.Options = append(a.Options, asks.Option{ID: o.id, Label: o.label, Does: o.does, Level: asks.Approve, Binds: binds}) a.Options = append(a.Options, asks.Option{ID: o.id, Label: o.label, Does: o.does, Level: asks.Approve, Binds: binds})
} }
ask = delivery.Ask{Group: "g", Heads: heads, Asking: asking, Ask: a} ask = delivery.Ask{Group: group, Heads: heads, Asking: asking, Ask: a}
value, err = delivery.EncodeAsk(ask) value, err = delivery.EncodeAsk(ask)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -308,6 +314,18 @@ func TestAWarrantThatHoldsWritesOneAnswerAndASecondWritesNone(t *testing.T) { //
if len(b.conds) != 0 { if len(b.conds) != 0 {
t.Errorf("conditions said: %+v", b.conds) t.Errorf("conditions said: %+v", b.conds)
} }
// 1280 (ADR 0556 clause 5): a settle ask holds when its options bind mesh-delivery.settle, and not when they bind a merge.
for verb, holds := range map[string]bool{"mesh-delivery.settle": true, "mesh-delivery.merge": false} {
b := newBed(t)
group := delivery.SettleGroup([]string{"novox/mesh-delivery@" + headA[:12]})
key, value, ga := gateAskOf(t, group, verb, twoHeads(), twoHeads(), 1)
b.gate[key] = value
b.t.OnAsk(key, 1, value)
b.deliver(t, warrantFor(ga.Ask, "approve"))
if _, written := b.answers.written[delivery.AnswerKey(key)]; written != holds {
t.Errorf("a settle ask binding %s: answer written %v, want %v", verb, written, holds)
}
}
} }
func TestAWarrantThatDoesNotHoldWritesNoAnswerAndRaisesTheCondition(t *testing.T) { // 1343/AC3 func TestAWarrantThatDoesNotHoldWritesNoAnswerAndRaisesTheCondition(t *testing.T) { // 1343/AC3
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-teller
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -10
View File
@@ -1,13 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61 h1:/c6ncICWjq55nBf8TlDM0ROe/PuH+qZIdnuVbtg8UuM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8 h1:jA7V7hlfJAigVUvJDpKwfgd9ylQ9bqppg3rK9XgLWoA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8 h1:vhYk/iv43u2WGSTrFyQGDvGBxLaRITgMAiiI9DjuA10=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5 h1:gE80bR0tPME7MTMBLcdP62v+F/9soVXmdcBYNMOye44=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -19,6 +19,18 @@ import (
// Tools are the read tools over a modules state, and build_usage over the build modules' states. // Tools are the read tools over a modules state, and build_usage over the build modules' states.
func Tools(r view.Reader, u view.UsageReader) []stdio.Tool { func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
return []stdio.Tool{ return []stdio.Tool{
{Name: view.ReachReadTool,
Description: "Whether a merge is on the machines: per repository, the reached head (the newest merge every node runs a build " +
"containing, as the view derives it from the indexer's modules, the wants and the haves; a merge at or before it is on the " +
"machines), what holds the next one (a module on a node, the indexer's lag), and per module its fingerprint, the nodes " +
"that want it, its changes (fingerprint, the merge it first appeared at, its source revision), its floor and the nodes " +
"where it is not settled or not healthy (novox/hq ADR 0556). Give a repository (owner/name) for one, none for every one. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "the repository, owner/name; none for every repository"}}},
Run: func(args map[string]any) (any, error) {
repository, _ := args["repository"].(string)
return r.Reach(repository)
}},
{Name: view.BuildUsageTool, {Name: view.BuildUsageTool,
Description: "How the build nodes are used, and whether a node is eligible to build, per node (every node holding a build module or named by a capacity): each build seat it holds (node-build-go, -node, -image) with its capacity " + Description: "How the build nodes are used, and whether a node is eligible to build, per node (every node holding a build module or named by a capacity): each build seat it holds (node-build-go, -node, -image) with its capacity " +
"(mesh-config's resolved value, or the modules' default) and pause, and whether it is eligible to build and why not; the asks " + "(mesh-config's resolved value, or the modules' default) and pause, and whether it is eligible to build and why not; the asks " +
@@ -50,6 +62,15 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
mods, err := r.Modules(only) mods, err := r.Modules(only)
return map[string]any{"modules": mods}, err return map[string]any{"modules": mods}, err
}}, }},
{Name: view.PlacementReadTool,
Description: "Where modules run, as the indexer read it from the catalogue's placement files at the last merge (placement.<node>, novox/hq ADR 0549): " +
"each node's modules in order with their host ports, bindings and settings placement, its pins and the seats its modules hold, and the catalogue revision. " +
"With node, that node's; with module, every node placing it; with neither, every node's. Keys that do not read are listed as unreadable. Read-only.",
Input: map[string]any{"type": "object", "properties": nodeModule},
Run: func(args map[string]any) (any, error) {
node, module := nodeModuleOf(args)
return r.Placement(node, module)
}},
{Name: view.CandidateReadTool, {Name: view.CandidateReadTool,
Description: "What the indexer stored for one candidate (the open pull requests of a branch) under candidates.<group>, as stored: " + Description: "What the indexer stored for one candidate (the open pull requests of a branch) under candidates.<group>, as stored: " +
"the pull requests and heads, the base, and per pull request the touched modules with their new-form fingerprint (fp2) or why " + "the pull requests and heads, the base, and per pull request the touched modules with their new-form fingerprint (fp2) or why " +
@@ -180,7 +201,9 @@ func read(state stdio.KeptState) view.StateRead {
func main() { func main() {
r := view.Reader{State: read(stdio.State("mesh-delivery-indexer.modules")), Builds: read(stdio.State("mesh-delivery-builder.builds")), CandidatesState: read(stdio.State("mesh-delivery-indexer.candidates")), r := view.Reader{State: read(stdio.State("mesh-delivery-indexer.modules")), Builds: read(stdio.State("mesh-delivery-builder.builds")), CandidatesState: read(stdio.State("mesh-delivery-indexer.candidates")),
WantsState: read(stdio.State("mesh-delivery-assigner.wants")), DifferencesState: read(stdio.State("mesh-delivery-assigner.differences")), HavesState: read(stdio.State("node-engine.have"))} WantsState: read(stdio.State("mesh-delivery-assigner.wants")), DifferencesState: read(stdio.State("mesh-delivery-assigner.differences")), HavesState: read(stdio.State("node-engine.have")),
ReachState: read(stdio.State("reach")),
PlacementState: read(stdio.State("mesh-delivery-indexer.placement"))}
u := view.UsageReader{Haves: r.HavesState, Resolved: read(stdio.State("mesh-config.resolved")), Tries: read(stdio.State("mesh-delivery-checker.tries")), u := view.UsageReader{Haves: r.HavesState, Resolved: read(stdio.State("mesh-config.resolved")), Tries: read(stdio.State("mesh-delivery-checker.tries")),
Steps: map[string]view.StateRead{}, Pauses: map[string]view.StateRead{}, Backlog: backlogOf(socket())} Steps: map[string]view.StateRead{}, Pauses: map[string]view.StateRead{}, Backlog: backlogOf(socket())}
for _, lang := range delivery.AskLanguages { for _, lang := range delivery.AskLanguages {
@@ -71,6 +71,19 @@ func TestModuleToolAnswersTheStoredValue(t *testing.T) {
if _, err := run(t, r, view.ModuleReadTool, map[string]any{"module": ""}); err == nil { if _, err := run(t, r, view.ModuleReadTool, map[string]any{"module": ""}); err == nil {
t.Errorf("an empty name was answered") t.Errorf("an empty name was answered")
} }
// 1280 (ADR 0556): view_reach answers the view's reach of one repository as stored, or of every one, and says a
// repository it holds none of.
const reach = `{"v":1,"repository":"novox/web","reached":"c1","reached_src":12}`
r.ReachState = fake{"novox/web": reach}.reader().State
if out, err := run(t, r, view.ReachReadTool, map[string]any{"repository": "novox/Web"}); err != nil || string(out.(json.RawMessage)) != reach {
t.Errorf("view_reach novox/Web = %v, %v", out, err)
}
if out, err := run(t, r, view.ReachReadTool, map[string]any{}); err != nil || !strings.Contains(fmt.Sprint(out), "novox/web") {
t.Errorf("view_reach of every repository = %v, %v", out, err)
}
if _, err := run(t, r, view.ReachReadTool, map[string]any{"repository": "novox/gone"}); err == nil || !strings.Contains(err.Error(), "no reach of novox/gone") {
t.Errorf("a repository with no reach: %v", err)
}
} }
// 1562/AC1: indexer_modules answers every key as stored, and with none2 only those that say why they have no FP2. // 1562/AC1: indexer_modules answers every key as stored, and with none2 only those that say why they have no FP2.
@@ -92,6 +105,32 @@ func TestModulesToolAnswersEveryKeyAsStored(t *testing.T) {
if len(only) != 1 || string(only["b"]) != withNone { if len(only) != 1 || string(only["b"]) != withNone {
t.Errorf("none2 modules = %v", only) t.Errorf("none2 modules = %v", only)
} }
// 1126 (ADR 0549 §2): indexer_placement answers a node's key as stored, every node placing a module, or every node's,
// and lists a key that does not read.
const ace, g14 = `{"v":1,"node":"ace","revision":"m1","modules":[{"module":"plex"}]}`, `{"v":1,"node":"g14","revision":"m1","modules":[{"module":"zsh"},{"module":"plex"}]}`
p := view.Reader{PlacementState: fake{"placement.ace": ace, "placement.g14": g14, "placement.bad": `{"v":1}`, "other": "{}"}.reader().State}
got := func(args map[string]any) map[string]json.RawMessage {
t.Helper()
out, err := run(t, p, view.PlacementReadTool, args)
if err != nil {
t.Fatal(err)
}
return out.(map[string]any)["placement"].(map[string]json.RawMessage)
}
if one := got(map[string]any{"node": "ace"}); len(one) != 1 || string(one["ace"]) != ace {
t.Errorf("node ace = %v", one)
}
if zsh := got(map[string]any{"module": "zsh"}); len(zsh) != 1 || string(zsh["g14"]) != g14 {
t.Errorf("module zsh = %v", zsh)
}
out, err = run(t, p, view.PlacementReadTool, map[string]any{})
if all := out.(map[string]any); err != nil || len(all["placement"].(map[string]json.RawMessage)) != 2 || len(all["unreadable"].([]string)) != 2 {
t.Errorf("every node = %v, %v", all, err)
}
if _, err := run(t, p, view.PlacementReadTool, map[string]any{"node": "novox"}); err == nil || !strings.Contains(err.Error(), "no placement of novox") {
t.Errorf("a node with no key: %v", err)
}
} }
const ( const (
@@ -144,13 +144,31 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
} }
}) })
model.SetDeliveredEvent(func(e view.DeliveredEvent) { model.SetDeliveredEvent(func(e view.DeliveredEvent) {
if err := emitEvent(ctx, js, cred, DeliveredEventType, e, fmt.Sprintf("delivered.%d", e.Revision)); err != nil { id := fmt.Sprintf("delivered.%d", e.Revision)
if e.Repository != "" { // a repository's reached head (ADR 0556 clause 3)
key, _ := delivery.RepositoryKey(e.Repository)
id = fmt.Sprintf("reached.%s.%d.%d", key, e.Revision, len(e.NotYet))
if len(e.NotYet) > 0 {
sum := sha256.Sum256([]byte(strings.Join(e.NotYet, ",")))
id += "." + hex.EncodeToString(sum[:6])
}
}
if err := emitEvent(ctx, js, cred, DeliveredEventType, e, id); err != nil {
logger.Printf("the delivery of source revision %d could not be said: %v", e.Revision, err) logger.Printf("the delivery of source revision %d could not be said: %v", e.Revision, err)
} }
}) })
model.SetReachWriter(func(w view.ReachWrite) {
if err := putReach(ctx, js, w); err != nil {
logger.Printf("the reach of %s could not be written (written again at its next change): %v", w.Key, err)
}
})
model.SetBacklog(view.BacklogOver(js, 5*time.Second)) model.SetBacklog(view.BacklogOver(js, 5*time.Second))
for _, bucket := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.ControllerComposedBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket} { for _, bucket := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.ControllerComposedBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket, delivery.TakenBucket} {
go follow(ctx, js, bucket, model, logger) go follow(ctx, js, bucket, false, model, logger)
}
// The modules' and the source's history give each fingerprint's first commit and each head's source revision (ADR 0556).
for _, bucket := range []string{delivery.ModulesBucket, delivery.SourceBucket} {
go follow(ctx, js, bucket, true, model, logger)
} }
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -181,11 +199,21 @@ func (w kvWriter) Put(key string, value []byte) error {
return err return err
} }
// putReach writes a reach value under its key in the view's own state reach (ADR 0556 clause 2), its only writer.
func putReach(ctx context.Context, js jetstream.JetStream, w view.ReachWrite) error {
kv, err := js.KeyValue(ctx, delivery.ReachBucket)
if err != nil {
return fmt.Errorf("the bucket %s: %w", delivery.ReachBucket, err)
}
_, err = kv.Put(ctx, w.Key, w.Value)
return err
}
// follow watches a bucket with its last values first and then each change, for as long as ctx lives: a bucket that is // follow watches a bucket with its last values first and then each change, for as long as ctx lives: a bucket that is
// not there yet, or a watch that ends, is tried again. // not there yet, or a watch that ends, is tried again.
func follow(ctx context.Context, js jetstream.JetStream, bucket string, model *view.Model, logger *log.Logger) { func follow(ctx context.Context, js jetstream.JetStream, bucket string, history bool, model *view.Model, logger *log.Logger) {
for ctx.Err() == nil { for ctx.Err() == nil {
if err := watchOnce(ctx, js, bucket, model); err != nil && ctx.Err() == nil { if err := watchOnce(ctx, js, bucket, history, model); err != nil && ctx.Err() == nil {
logger.Printf("cannot watch %s, trying again: %v", bucket, err) logger.Printf("cannot watch %s, trying again: %v", bucket, err)
} }
select { select {
@@ -195,12 +223,17 @@ func follow(ctx context.Context, js jetstream.JetStream, bucket string, model *v
} }
} }
func watchOnce(ctx context.Context, js jetstream.JetStream, bucket string, model *view.Model) error { func watchOnce(ctx context.Context, js jetstream.JetStream, bucket string, history bool, model *view.Model) error {
kv, err := js.KeyValue(ctx, bucket) kv, err := js.KeyValue(ctx, bucket)
if err != nil { if err != nil {
return err return err
} }
w, err := kv.WatchAll(ctx) var opts []jetstream.WatchOpt
initial := bucket
if history {
opts, initial = append(opts, jetstream.IncludeHistory()), view.ReachHistory(bucket)
}
w, err := kv.WatchAll(ctx, opts...)
if err != nil { if err != nil {
return err return err
} }
@@ -214,11 +247,11 @@ func watchOnce(ctx context.Context, js jetstream.JetStream, bucket string, model
return errors.New("the watch ended") return errors.New("the watch ended")
} }
if e == nil { // the last values have all been delivered if e == nil { // the last values have all been delivered
model.Initial(bucket) model.Initial(initial)
continue continue
} }
model.Apply(view.Entry{Bucket: bucket, Key: e.Key(), Value: e.Value(), At: e.Created(), model.Apply(view.Entry{Bucket: bucket, Key: e.Key(), Value: e.Value(), At: e.Created(),
Deleted: e.Operation() != jetstream.KeyValuePut, Revision: e.Revision()}) Deleted: e.Operation() != jetstream.KeyValuePut, Revision: e.Revision(), History: history})
} }
} }
} }
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-view
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2 h1:s1WcG7Ll1OkEwwvMKEabdWvPvTiFCHghlXiY1LoIJDA= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61 h1:/c6ncICWjq55nBf8TlDM0ROe/PuH+qZIdnuVbtg8UuM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026211913-68d65e6bbf61/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,347 @@
package view
import (
"fmt"
"maps"
"slices"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// How far each repository reached (novox/hq ADR 0556 clauses 2 and 3, issue 1280). The view keeps, per repository the gate's
// source names, how far each of its modules reached on the nodes that want it, and derives the repository's reached head: the
// newest head of the repository in the source's history, not past the merge the indexer took in, at which every wanted module's
// floor is at or after its last change at or before that head. It writes that as its state reach and says each new reached
// head as its event delivered, per repository. Nothing here records a merge as delivered (ADR 0429 §1): a merge at or before
// the reached head is on the machines, derived by whoever reads it.
// ReachHistory names the watch of a bucket with its history, as Initial is told it: the modules' and the source's history
// give each fingerprint's first commit and each head's source revision.
func ReachHistory(bucket string) string { return bucket + "#history" }
// reachBuckets are the watches the derivation waits for before it writes or says anything, so a start does not write a reach
// derived from half its inputs.
var reachBuckets = []string{delivery.ModulesBucket, delivery.AssignerWantsBucket, delivery.HaveBucket, delivery.SourceBucket,
delivery.TakenBucket, ReachHistory(delivery.ModulesBucket), ReachHistory(delivery.SourceBucket)}
// ReachWrite is a reach value to write under its key in the view's state reach.
type ReachWrite struct {
Key string
Value []byte
}
// SetReachWriter sets where the view's reach values go; nothing is written until it is set.
func (m *Model) SetReachWriter(put func(ReachWrite)) {
m.mu.Lock()
defer m.mu.Unlock()
m.reachPut = put
}
// applyHistory takes one revision of a watch with history: of the indexer's modules (each module's values in order) and of
// the gate's source (each revision's heads).
func (m *Model) applyHistory(e Entry) {
switch e.Bucket {
case delivery.ModulesBucket:
if e.Deleted {
delete(m.modHist, e.Key)
return
}
mod, err := delivery.DecodeModule(e.Value)
if err != nil || mod.FP2 == "" || mod.Commit == "" {
return
}
h := m.modHist[e.Key]
if n := len(h); n > 0 && h[n-1].FP2 == mod.FP2 && h[n-1].Commit == mod.Commit {
if mod.Src != 0 {
h[n-1].Src = mod.Src
}
return
}
h = append(h, mod)
if len(h) > 2*delivery.MostChanges {
h = h[len(h)-delivery.MostChanges:]
}
m.modHist[e.Key] = h
case delivery.SourceBucket:
if e.Key != delivery.SourceMainKey || e.Deleted || e.Revision == 0 {
return
}
src, err := delivery.DecodeSource(e.Value)
if err != nil {
return
}
m.srcHist[e.Revision] = src.Heads
}
}
// headAt is one head of a repository and the first source revision that named it.
type headAt struct {
commit string
rev uint64
}
// headsOf are the heads of a repository in the source's history, oldest first, each with the first revision naming it.
func (m *Model) headsOf(repository string) []headAt {
var out []headAt
for _, rev := range slices.Sorted(maps.Keys(m.srcHist)) {
for r, head := range m.srcHist[rev] {
if !delivery.SameRepository(r, repository) || head == "" {
continue
}
if n := len(out); n == 0 || !sameCommit(out[n-1].commit, head) {
out = append(out, headAt{commit: head, rev: rev})
}
}
}
return out
}
// revOf is the first source revision naming a commit as the repository's head, or 0 when the history holds none.
func revOf(heads []headAt, commit string) uint64 {
for _, h := range heads {
if sameCommit(h.commit, commit) {
return h.rev
}
}
return 0
}
// sameCommit is whether two spellings of a commit are one (either may be abbreviated to at least seven characters).
func sameCommit(a, b string) bool {
a, b = strings.ToLower(a), strings.ToLower(b)
if len(a) > len(b) {
a, b = b, a
}
return len(a) >= 7 && strings.HasPrefix(b, a)
}
// noBound is a module's bound when its floor is its newest change: it holds no head back.
const noBound = ^uint64(0)
// wantKeys are the wants' keys by module, each sorted: read once for every repository a derivation looks at.
func (m *Model) wantKeys() map[string][]string {
out := map[string][]string{}
for _, key := range slices.Sorted(maps.Keys(m.wants)) {
_, module := splitKey(key)
out[module] = append(out[module], key)
}
return out
}
// deriveReach is the reach of one repository from the model as it stands; wants are the wants' keys by module (wantKeys).
func (m *Model) deriveReach(repository string, wants map[string][]string) delivery.Reach {
r := delivery.Reach{Repository: delivery.RepositoryName(repository), Modules: map[string]delivery.ModuleReach{}}
heads := m.headsOf(repository)
key, _ := delivery.RepositoryKey(repository)
if t, ok := m.taken[key]; ok {
r.Indexed = t.Merge
}
bound := noBound
var holds []string
for _, name := range slices.Sorted(maps.Keys(m.modules)) {
me := m.modules[name]
if me.err != "" || me.value.FP2 == "" || !delivery.SameRepository(me.value.Repository, repository) {
continue
}
mr, b, words := m.moduleReach(name, me.value, heads, wants[name])
r.Modules[name] = mr
if b < bound {
bound = b
}
if b != noBound {
holds = append(holds, words)
}
}
if len(r.Modules) == 0 {
r.Modules = nil
}
indexed := uint64(0)
if r.Indexed != "" {
indexed = revOf(heads, r.Indexed)
}
switch {
case r.Indexed == "":
holds = append(holds, "the indexer has taken in no merge of "+r.Repository)
case indexed == 0:
holds = append(holds, fmt.Sprintf("the merge the indexer took in, %.12s, is not a head of %s in the source's history", r.Indexed, r.Repository))
}
for _, h := range heads {
if h.rev <= indexed && h.rev < bound {
r.Reached, r.ReachedSrc = h.commit, h.rev
}
}
if n := len(heads); n > 0 && !sameCommit(heads[n-1].commit, r.Reached) {
if indexed != 0 && indexed < heads[n-1].rev {
holds = append(holds, fmt.Sprintf("the indexer has not yet taken in %.12s", heads[n-1].commit))
}
slices.Sort(holds)
r.Holds = slices.Compact(holds)
}
return r
}
// moduleReach is how far one module reached, the source revision it holds the reached head before (noBound when none), and
// what holds it, in words. A node counts for the fingerprint its running have runs, healthy or not (ADR 0556 clause 2).
func (m *Model) moduleReach(name string, mod delivery.Module, heads []headAt, wantKeys []string) (delivery.ModuleReach, uint64, string) {
mr := delivery.ModuleReach{FP: mod.FP2}
hist := m.modHist[name]
if n := len(hist); n == 0 || hist[n-1].FP2 != mod.FP2 || hist[n-1].Commit != mod.Commit {
hist = append(slices.Clone(hist), mod) // the value watched now is the newest, whatever the history watch has given
}
for i := len(hist) - 1; i >= 0 && len(mr.Changes) < delivery.MostChanges; i-- {
c := delivery.Change{Commit: hist[i].Commit, FP: hist[i].FP2, Src: hist[i].Src}
if c.Src == 0 {
c.Src = revOf(heads, c.Commit)
}
if n := len(mr.Changes); n > 0 && mr.Changes[n-1].FP == c.FP {
mr.Changes[n-1] = c // the same fingerprint again, back to back: its first appearance is the older commit
continue
}
mr.Changes = append(mr.Changes, c)
}
var waiting []string
floor, floorNode, unset := -1, "", ""
for _, key := range wantKeys {
node, _ := splitKey(key)
w := m.wants[key]
if w.err != "" || w.value.Removed || w.value.FP == "" {
continue
}
mr.Nodes++
h, ok := m.haves[key]
switch {
case !ok || h.err != "" || h.have.FP == "":
waiting = append(waiting, fmt.Sprintf("%s on %s runs nothing yet", name, node))
unset = waiting[len(waiting)-1]
continue
case !runs(h.have.Phase):
waiting = append(waiting, fmt.Sprintf("%s on %s is %s at %s, not running", name, node, h.have.Phase, short(h.have.FP)))
unset = waiting[len(waiting)-1]
continue
case h.have.FP != w.value.FP:
waiting = append(waiting, fmt.Sprintf("%s on %s runs %s, not yet the %s it wants", name, node, short(h.have.FP), short(w.value.FP)))
case h.have.Health == delivery.HealthFailed:
waiting = append(waiting, fmt.Sprintf("%s on %s runs %s and its health check failed", name, node, short(h.have.FP)))
case h.have.Health != delivery.HealthOK:
waiting = append(waiting, fmt.Sprintf("%s on %s runs %s, not yet healthy", name, node, short(h.have.FP)))
}
i := slices.IndexFunc(mr.Changes, func(c delivery.Change) bool { return c.FP == h.have.FP })
if i < 0 {
unset = fmt.Sprintf("%s on %s runs %s, which is not among its %d known changes", name, node, short(h.have.FP), len(mr.Changes))
waiting = append(waiting, unset)
continue
}
if i > floor {
floor, floorNode = i, node
}
}
mr.Waiting = strings.Join(waiting, "; ")
switch {
case mr.Nodes == 0:
return mr, noBound, ""
case unset != "":
// A node that wants the module runs nothing countable of it (no have, not running, or a build older than the kept
// history): the module holds no head of the repository back, which would make every apply anywhere, or one module
// stuck failed, freeze the whole repository (review of issue 1280, ADR 0556 clause 2). Its words name it, and a merge
// among its changes is not yet on the machines (notYet).
return mr, noBound, ""
}
mr.Floor = mr.Changes[floor].Commit
if floor == 0 {
return mr, noBound, ""
}
next := mr.Changes[floor-1]
words := fmt.Sprintf("%s on %s runs the build of %.12s, not yet that of %.12s", name, floorNode, mr.Floor, next.Commit)
if next.Src == 0 {
// The next change's order is not known: the head is held at the floor's own revision, or before every head when that is
// not known either.
if own := mr.Changes[floor].Src; own != 0 {
return mr, own + 1, words
}
return mr, 0, words
}
return mr, next.Src, words
}
// runs is whether a have in this phase runs the fingerprint it names: running, being checked after its apply, or held after a
// failed backup, which keeps running what it has (ADR 0539 clause 16). A have backing up or applying, failed or removed runs
// nothing the derivation can count.
func runs(p delivery.Phase) bool {
return p == delivery.PhaseRunning || p == delivery.PhaseChecking || p == delivery.PhaseHeld
}
// short is a fingerprint's first twelve characters, after its form.
func short(fp string) string {
fp = strings.TrimPrefix(fp, delivery.SourcePrefix2)
if len(fp) > 12 {
fp = fp[:12]
}
return fp
}
// reachReady is whether every watch the derivation reads handed over what stood at its start.
func (m *Model) reachReady() bool {
for _, b := range reachBuckets {
if !m.initial[b] {
return false
}
}
return m.source != nil && m.source.err == ""
}
// reachNow derives the reach of every repository the source names, and answers the values that changed and the reached
// heads not yet said. It is called with the lock held.
func (m *Model) reachNow() ([]ReachWrite, []DeliveredEvent) {
if (m.reachPut == nil && m.deliveredEvent == nil) || !m.reachReady() {
return nil, nil
}
var writes []ReachWrite
var events []DeliveredEvent
wants := m.wantKeys()
for _, repository := range slices.Sorted(maps.Keys(m.source.heads)) {
key, err := delivery.RepositoryKey(repository)
if err != nil {
continue
}
r := m.deriveReach(repository, wants)
if prev, ok := m.reach[key]; !ok || !delivery.SameReach(prev, r) {
if data, err := delivery.EncodeReach(r); err == nil {
m.reach[key] = r
writes = append(writes, ReachWrite{Key: key, Value: data})
}
}
notYet := notYetOf(r)
said := strings.Join(notYet, ",")
if r.Reached != "" && (r.ReachedSrc > m.reachSaid[key] || said != m.reachNotYet[key]) {
m.reachSaid[key], m.reachNotYet[key] = r.ReachedSrc, said
events = append(events, DeliveredEvent{Revision: r.ReachedSrc, Heads: map[string]string{r.Repository: r.Reached},
Repository: r.Repository, NotYet: notYet})
}
}
return writes, events
}
// notYetOf are the merges at or before a repository's reached head that are not yet on the machines: the changes of each module
// a node wants with no floor (a node runs nothing countable of it), sorted. A consumer takes every other merge at or before
// the reached head as on the machines.
func notYetOf(r delivery.Reach) []string {
var out []string
for _, mr := range r.Modules {
if mr.Nodes == 0 || mr.Floor != "" {
continue
}
for _, c := range mr.Changes {
out = append(out, c.Commit)
}
}
slices.Sort(out)
return slices.Compact(out)
}
// Reach is the reach of every repository as the view derived it last, by key.
func (m *Model) Reach() map[string]delivery.Reach {
m.mu.Lock()
defer m.mu.Unlock()
return maps.Clone(m.reach)
}
@@ -0,0 +1,157 @@
package view
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1280 (ADR 0556 clauses 2 and 3): the view derives, per repository, how far each module reached and the reached head, writes
// it when it differs, and says each new reached head once as its event delivered.
func TestTheViewDerivesHowFarEachRepositoryReached(t *testing.T) {
const web, sdk = "novox/web", "novox/sdk"
commitA, commitB, commitC := strings.Repeat("a", 40), strings.Repeat("b", 40), strings.Repeat("c", 40)
commitS, commitX := strings.Repeat("5", 40), strings.Repeat("e", 40)
const fpMail = "src2:3333333333333333333333333333333333333333333333333333333333333333"
mod := func(module, fp, commit string, src uint64, history bool) Entry {
v, err := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: strings.Repeat("a", 64), Repository: web, Commit: commit, Src: src, Path: module})
if err != nil {
t.Fatal(err)
}
return Entry{Bucket: delivery.ModulesBucket, Key: module, Value: v, At: t0, History: history}
}
src := func(rev uint64, webHead string) Entry {
v, err := delivery.EncodeSource(delivery.Source{Group: "g", Heads: map[string]string{web: webHead, sdk: commitS}})
if err != nil {
t.Fatal(err)
}
return Entry{Bucket: delivery.SourceBucket, Key: delivery.SourceMainKey, Value: v, At: t0, Revision: rev, History: true}
}
taken := func(repository, merge string) Entry {
key, _ := delivery.RepositoryKey(repository)
v, err := delivery.EncodeTaken(delivery.Taken{Repository: repository, Merge: merge, At: t0})
if err != nil {
t.Fatal(err)
}
return Entry{Bucket: delivery.TakenBucket, Key: key, Value: v, At: t0}
}
haveIn := func(node, module, fp string, phase delivery.Phase, health delivery.Health) Entry {
e := haveFP(t, node, module, fp, health)
v, _ := delivery.EncodeHave(delivery.Have{FP: fp, Phase: phase, Health: health})
e.Value = v
return e
}
// web changed at A (fpOld) and at C (fpNew); mail first appeared at A. The source named A, B, then C as web's head.
world := func(t *testing.T) []Entry {
cur, _ := delivery.DecodeSource(src(12, commitC).Value)
current := Entry{Bucket: delivery.SourceBucket, Key: delivery.SourceMainKey, Revision: 12, At: t0}
current.Value, _ = delivery.EncodeSource(cur)
return []Entry{src(10, commitA), src(11, commitB), src(12, commitC), current,
mod("web", fpOld, commitA, 10, true), mod("web", fpNew, commitC, 12, true), mod("mail", fpMail, commitA, 0, true),
mod("web", fpNew, commitC, 12, false), mod("mail", fpMail, commitA, 0, false),
taken(web, commitC), taken(sdk, commitS),
wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpNew, delivery.HealthOK),
wantE(t, "g14", "web", delivery.Want{FP: fpNew}), haveFP(t, "g14", "web", fpNew, delivery.HealthOK),
wantE(t, "ace", "mail", delivery.Want{FP: fpMail}), haveFP(t, "ace", "mail", fpMail, delivery.HealthOK)}
}
for _, tc := range []struct {
name string
then []Entry
reached string // web's reached head; "" none
holds string // words one of web's holds carries; "" none
waiting string // words web's module web waits with; "" none
notYet []string // the merges the reached head's event says are not yet on the machines
}{
{name: "every node runs the newest build", reached: commitC},
{name: "a node lagging stops the head before the change it lacks", reached: commitB, holds: "web on g14",
then: []Entry{haveFP(t, "g14", "web", fpOld, delivery.HealthOK)}, waiting: "not yet the"},
{name: "a want held back at an older build holds the head as a lag does", reached: commitB, holds: "web on g14 runs the build of",
then: []Entry{wantE(t, "g14", "web", delivery.Want{FP: fpOld, Newest: fpNew, Rule: 2}), haveFP(t, "g14", "web", fpOld, delivery.HealthOK)}},
{name: "a failing health check is said and holds nothing", reached: commitC, waiting: "health check failed",
then: []Entry{haveFP(t, "g14", "web", fpNew, delivery.HealthFailed)}},
{name: "a node with no running have holds no head back and its module's merges are not yet", reached: commitC, waiting: "not running",
notYet: []string{commitA, commitC},
then: []Entry{haveIn("g14", "web", fpNew, delivery.PhaseApplying, delivery.HealthUnknown)}},
{name: "an unrelated module not running leaves the head reached (the reviewer's probe)", reached: commitC,
notYet: []string{commitA},
then: []Entry{haveIn("ace", "mail", fpMail, delivery.PhaseFailed, delivery.HealthFailed)}},
{name: "a module no node wants holds nothing", reached: commitC,
then: []Entry{wantE(t, "ace", "mail", delivery.Want{Removed: true})}},
{name: "the indexer not yet having taken the head in", reached: commitB, holds: "has not yet taken in",
then: []Entry{taken(web, commitB)}},
{name: "a change whose source revision is not known yet holds no head the source named", reached: commitC,
then: []Entry{mod("web", "src2:"+strings.Repeat("9", 64), commitX, 0, true), mod("web", "src2:"+strings.Repeat("9", 64), commitX, 0, false)}},
} {
t.Run(tc.name, func(t *testing.T) {
m := New()
var written []ReachWrite
var said []DeliveredEvent
m.SetReachWriter(func(w ReachWrite) { written = append(written, w) })
m.SetDeliveredEvent(func(e DeliveredEvent) {
if e.Repository != "" {
said = append(said, e)
}
})
fill(m, world(t)...)
if len(written) != 0 {
t.Fatalf("written before every watch handed over: %v", written)
}
for _, b := range reachBuckets {
m.Initial(b)
}
fill(m, tc.then...)
key, _ := delivery.RepositoryKey(web)
r := m.Reach()[key]
if r.Reached != tc.reached {
t.Errorf("reached = %.8s, want %.8s (holds %v)", r.Reached, tc.reached, r.Holds)
}
if got := strings.Join(r.Holds, "; "); (tc.holds == "") != (got == "") || !strings.Contains(got, tc.holds) {
t.Errorf("holds = %q, want words with %q", got, tc.holds)
}
if got := r.Modules["web"].Waiting; (tc.waiting == "") != (got == "") || !strings.Contains(got, tc.waiting) {
t.Errorf("web waits with %q, want words with %q", got, tc.waiting)
}
// A library reaches the head the indexer took in, with no module.
sk, _ := delivery.RepositoryKey(sdk)
if lib := m.Reach()[sk]; lib.Reached != commitS || len(lib.Modules) != 0 {
t.Errorf("the library's reach = %+v", lib)
}
// Each key is written once with its last value; a reached head is said once, for its repository alone.
last := map[string]delivery.Reach{}
for _, w := range written {
v, err := delivery.DecodeReach(w.Value)
if err != nil {
t.Fatalf("written %s: %v", w.Key, err)
}
last[w.Key] = v
}
if !delivery.SameReach(last[key], r) {
t.Errorf("the last value written is %+v, not the reach derived %+v", last[key], r)
}
if tc.reached != "" {
var last DeliveredEvent
for _, e := range said {
if e.Repository == web {
last = e
}
}
if strings.Join(last.NotYet, ",") != strings.Join(tc.notYet, ",") {
t.Errorf("not yet = %v, want %v", last.NotYet, tc.notYet)
}
}
{
n := len(said)
fill(m, tc.then...)
if len(said) != n && tc.reached == commitC {
t.Errorf("said again: %+v", said[n:])
}
for _, e := range said {
if len(e.Heads) != 1 || e.Group != "" || e.Revision == 0 {
t.Errorf("a reached head said as %+v", e)
}
}
}
})
}
}
@@ -20,6 +20,8 @@ import (
const ( const (
ModuleReadTool = "indexer_module" ModuleReadTool = "indexer_module"
ModulesReadTool = "indexer_modules" ModulesReadTool = "indexer_modules"
// PlacementReadTool reads the indexer's placement (novox/hq ADR 0549 §2, issue 1126).
PlacementReadTool = "indexer_placement"
) )
// StateRead reads the indexer's modules state, as the view's reads grant it: its keys and a key's stored value (nil for a key with none). // StateRead reads the indexer's modules state, as the view's reads grant it: its keys and a key's stored value (nil for a key with none).
@@ -39,6 +41,39 @@ type Reader struct {
WantsState StateRead // the assigner's wants and holds WantsState StateRead // the assigner's wants and holds
DifferencesState StateRead // the assigner's differences DifferencesState StateRead // the assigner's differences
HavesState StateRead // the node-engine's haves HavesState StateRead // the node-engine's haves
ReachState StateRead // the view's own reach (ADR 0556)
PlacementState StateRead // the indexer's placement
}
// ReachReadTool reads the view's reach: per repository, how far each module reached and the reached head (ADR 0556).
const ReachReadTool = "view_reach"
// Reach is the stored reach of one repository, or of every repository when none is named, as the view wrote it.
func (r Reader) Reach(repository string) (any, error) {
if r.ReachState.Get == nil || r.ReachState.Keys == nil {
return nil, fmt.Errorf("the reach state cannot be read here")
}
if repository = strings.TrimSpace(repository); repository != "" {
key, err := delivery.RepositoryKey(repository)
if err != nil {
return nil, err
}
raw, err := r.ReachState.Get(key)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, fmt.Errorf("no reach of %s: the view has derived none (the gate's source does not name it, or the view waits for a watch)", repository)
}
return raw, nil
}
keys, err := r.ReachState.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
values, err := r.ReachState.values(keys)
return map[string]any{"reach": values}, err
} }
// Module is the stored value of modules.<module>, exactly as the indexer wrote it. // Module is the stored value of modules.<module>, exactly as the indexer wrote it.
@@ -613,3 +648,55 @@ func (r Reader) Differences(node, module string) (KeptAnswer, error) {
func (r Reader) Haves(node, module string) (KeptAnswer, error) { func (r Reader) Haves(node, module string) (KeptAnswer, error) {
return r.kept(r.HavesState, "the haves", node, module) return r.kept(r.HavesState, "the haves", node, module)
} }
// Placement is the indexer's placement as stored (ADR 0549 §2): one node's key with node, every node placing a module
// with module (each node's key whole, so its pins, ports and bindings are beside the module), and every node's key with
// neither. A key that does not read as a placement is listed as unreadable, never dropped.
func (r Reader) Placement(node, module string) (map[string]any, error) {
node, module = strings.TrimSpace(node), strings.TrimSpace(module)
if r.PlacementState.Keys == nil || r.PlacementState.Get == nil {
return nil, fmt.Errorf("the placement state cannot be read here")
}
if node != "" {
key, err := delivery.PlacementKey(node)
if err != nil {
return nil, err
}
raw, err := r.PlacementState.Get(key)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, fmt.Errorf("no placement of %s: the catalogue holds no placement file for it, or the indexer has read none yet", node)
}
return map[string]any{"placement": map[string]json.RawMessage{node: raw}}, nil
}
keys, err := r.PlacementState.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
values, err := r.PlacementState.values(keys)
if err != nil {
return nil, err
}
out := map[string]json.RawMessage{}
var unreadable []string
for i, key := range keys {
n, err := delivery.ParsePlacementKey(key)
if err != nil {
unreadable = append(unreadable, key)
continue
}
p, err := delivery.DecodePlacement(values[i])
if err != nil {
unreadable = append(unreadable, key)
continue
}
if module != "" && !slices.Contains(p.Assigned(), module) {
continue
}
out[n] = values[i]
}
return map[string]any{"placement": out, "unreadable": unreadable}, nil
}
@@ -67,7 +67,8 @@ type UsageAnswer struct {
At time.Time `json:"at"` At time.Time `json:"at"`
Nodes []NodeUsage `json:"nodes"` Nodes []NodeUsage `json:"nodes"`
Backlog []BacklogRow `json:"backlog"` Backlog []BacklogRow `json:"backlog"`
// TriesRunning are the tries the checker runs now; their asks are counted under check until the try ends and names them. // TriesRunning are the tries the checker runs now, each with its asks and where each stands (a checker that names its
// asks as they are put, issue 1890; an older one names them only at the end, and they are counted under check until then).
TriesRunning []TryRunning `json:"tries_running"` TriesRunning []TryRunning `json:"tries_running"`
Notes []string `json:"notes,omitempty"` Notes []string `json:"notes,omitempty"`
} }
@@ -77,6 +78,9 @@ type TryRunning struct {
ID string `json:"id"` ID string `json:"id"`
Repository string `json:"repository"` Repository string `json:"repository"`
Started time.Time `json:"started,omitzero"` Started time.Time `json:"started,omitzero"`
Asks []string `json:"asks,omitempty"`
// Steps are the try's own words for each ask: asked, or the node that took it, its state and the phase its step says.
Steps json.RawMessage `json:"steps,omitempty"`
} }
// NodeUsage is one node: each build seat it holds or is set to build for, whether it is eligible to build, and its CPU. // NodeUsage is one node: each build seat it holds or is set to build for, whether it is eligible to build, and its CPU.
@@ -101,7 +105,17 @@ type SeatUsage struct {
// Capacity is the asks it runs at once; CapacityFrom where that came from. // Capacity is the asks it runs at once; CapacityFrom where that came from.
Capacity int `json:"capacity"` Capacity int `json:"capacity"`
CapacityFrom string `json:"capacity_from"` CapacityFrom string `json:"capacity_from"`
Paused bool `json:"paused"` // Configured is mesh-config's capacity when the module says it judges by a lower one (capacity.<node> of its pauses
// state: the configured capacity bounded by its node's build CPU, issue 1806's follow-up); Capacity is then the
// module's and CapacityFrom its why.
Configured int `json:"configured,omitempty"`
// BuildHours is the node's build window as the module says it (HH:MM-HH:MM and its zone), absent when it builds at any
// hour; OutsideHours says the module judged the time outside it, so it takes nothing new (the operator, 2026-10-26).
BuildHours string `json:"build_hours,omitempty"`
OutsideHours bool `json:"outside_hours,omitempty"`
// BuildHoursFault is why the module cannot read its node's build-hours value, naming it: it then builds at any hour.
BuildHoursFault string `json:"build_hours_fault,omitempty"`
Paused bool `json:"paused"`
// Eligible says it takes asks: it holds the seat, has a capacity above 0 and is not paused; WhyNot says why not. // Eligible says it takes asks: it holds the seat, has a capacity above 0 and is not paused; WhyNot says why not.
// EligibleKnown is false when this tool cannot tell (it cannot read mesh-config's capacities): Eligible is then // EligibleKnown is false when this tool cannot tell (it cannot read mesh-config's capacities): Eligible is then
// absent, never false, and WhyNot starts "not known: ". // absent, never false, and WhyNot starts "not known: ".
@@ -384,8 +398,9 @@ func (u UsageReader) Usage() UsageAnswer {
} }
} }
// Pauses: paused.<node> of each module. // Pauses: paused.<node> of each module; and capacity.<node>, the capacity the module says it judges by.
paused := map[string]bool{} // <node>.<language> paused := map[string]bool{} // <node>.<language>
judged := map[string]delivery.Capacity{}
for _, lang := range delivery.AskLanguages { for _, lang := range delivery.AskLanguages {
module, _ := delivery.BuildModuleOf(lang) module, _ := delivery.BuildModuleOf(lang)
pr, ok := u.Pauses[lang] pr, ok := u.Pauses[lang]
@@ -411,6 +426,13 @@ func (u UsageReader) Usage() UsageAnswer {
paused[node+"."+lang] = p.Paused paused[node+"."+lang] = p.Paused
} }
} }
if key, err := delivery.CapacityKey(node); err == nil {
if raw, err := pr.Get(key); err == nil && len(raw) > 0 {
if c, err := delivery.DecodeCapacity(raw); err == nil {
judged[node+"."+lang] = c
}
}
}
} }
} }
@@ -452,6 +474,18 @@ func (u UsageReader) Usage() UsageAnswer {
s.Capacity = (&buildrun.Places{Capacity: caps}).CapacityOf(node) s.Capacity = (&buildrun.Places{Capacity: caps}).CapacityOf(node)
s.CapacityFrom = fmt.Sprintf("the modules' default %q: mesh-config holds no resolved value for %s on %s", buildstep.DefaultCapacities, s.Module, node) s.CapacityFrom = fmt.Sprintf("the modules' default %q: mesh-config holds no resolved value for %s on %s", buildstep.DefaultCapacities, s.Module, node)
} }
if c, ok := judged[node+"."+lang]; ok && s.Holds && unknown == "" && !isBad && c.Configured == s.Capacity {
// The module running on the node says its build hours, and judges by less than mesh-config gives when its
// node's build CPU bounds it or the time is outside its build hours.
s.BuildHours, s.OutsideHours = c.BuildHours, c.OutsideHours
if c.BuildHoursFault != "" {
s.BuildHours, s.BuildHoursFault = "any hour", c.BuildHoursFault
}
if c.Effective != c.Configured {
s.Configured, s.Capacity = s.Capacity, c.Effective
s.CapacityFrom = fmt.Sprintf("%s, as %s on %s judges it (said %s)", c.Why, s.Module, node, c.At.UTC().Format(time.RFC3339))
}
}
s.Paused = paused[node+"."+lang] s.Paused = paused[node+"."+lang]
s.EligibleKnown = true s.EligibleKnown = true
switch { switch {
@@ -466,6 +500,8 @@ func (u UsageReader) Usage() UsageAnswer {
s.WhyNot = "not known: this tool cannot read its capacity (" + unknown + ")" s.WhyNot = "not known: this tool cannot read its capacity (" + unknown + ")"
case isBad: case isBad:
s.WhyNot = "its capacity on " + node + " cannot be read, so its puller uses 0" s.WhyNot = "its capacity on " + node + " cannot be read, so its puller uses 0"
case s.OutsideHours:
s.WhyNot = "outside its build hours " + s.BuildHours + " on " + node + ": it takes nothing new until they open"
case s.Capacity == 0: case s.Capacity == 0:
s.WhyNot = "its capacity on " + node + " is 0" s.WhyNot = "its capacity on " + node + " is 0"
} }
@@ -517,6 +553,7 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
want = append(want, k) want = append(want, k)
} }
} }
unnamed := 0
raws, err := u.Tries.values(want) raws, err := u.Tries.values(want)
if err != nil { if err != nil {
note("the checker's tries cannot all be read: %v", err) note("the checker's tries cannot all be read: %v", err)
@@ -526,11 +563,12 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
continue continue
} }
var t struct { var t struct {
ID string `json:"id"` ID string `json:"id"`
Repository string `json:"repository"` Repository string `json:"repository"`
State string `json:"state"` State string `json:"state"`
Started time.Time `json:"started"` Started time.Time `json:"started"`
Asks []string `json:"asks"` Asks []string `json:"asks"`
Steps json.RawMessage `json:"steps"`
} }
if json.Unmarshal(raws[i], &t) != nil { if json.Unmarshal(raws[i], &t) != nil {
continue continue
@@ -539,11 +577,14 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
asks[a] = true asks[a] = true
} }
if t.State == "running" { if t.State == "running" {
ans.TriesRunning = append(ans.TriesRunning, TryRunning{ID: t.ID, Repository: t.Repository, Started: t.Started}) ans.TriesRunning = append(ans.TriesRunning, TryRunning{ID: t.ID, Repository: t.Repository, Started: t.Started, Asks: t.Asks, Steps: t.Steps})
if len(t.Asks) == 0 {
unnamed++
}
} }
} }
if len(ans.TriesRunning) > 0 { if unnamed > 0 {
note("a running try's asks are counted under check until the try ends and names them") note("%d running tries name no ask yet (none put yet, or a checker that names them only at the end): their asks are counted under check until then", unnamed)
} }
return asks return asks
} }
@@ -191,4 +191,36 @@ func TestBuildUsageAnswersEveryNodesUseFromTheStates(t *testing.T) {
if verdict["shanks"] != "not known" || verdict["ace"] != "not eligible" { if verdict["shanks"] != "not known" || verdict["ace"] != "not eligible" {
t.Errorf("verdicts with mesh-config unreadable = %v, want shanks not known, ace not eligible (paused)", verdict) t.Errorf("verdicts with mesh-config unreadable = %v, want shanks not known, ace not eligible (paused)", verdict)
} }
// Issue 1806's follow-up: the module on shanks says it judges by 1 of mesh-config's 2 (its build CPU bounds it); the
// seat says the module's capacity and why. And issue 1890: a running try that names its asks is said with them.
u = usageFixture()
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":1,"configured":2,"quota-from":"the cpu.max of /meshyield.slice","quota-cores":6,"cores-per-step":6,"by-cpu":1,"why":"1, bounded by the CPU","at":"2026-10-26T11:00:00Z"}`,
"capacity.ace": `{"v":1,"effective":2,"configured":3,"why":"stale: mesh-config says 2 now"}`})
u.Tries = states(map[string]string{"try-" + strconv.FormatInt(at("11:58").UnixNano(), 10): `{"id":"try-2","repository":"novox/mesh-delivery","head":"h","trunk":"t","state":"running","asks":["` + tryAsk0 + `"],"steps":[{"ask":"` + tryAsk0 + `","node":"shanks","state":"running","phase":"unit: tests of 3 packages"}]}`})
bound := u.Usage()
sc := seatOf(t, bound, "shanks", "node-build-go")
if sc.Capacity != 1 || sc.Configured != 2 || !strings.HasPrefix(sc.CapacityFrom, "1, bounded by the CPU, as mesh-build-go on shanks judges it") {
t.Errorf("a seat its CPU bounds: capacity %d of %d, from %q", sc.Capacity, sc.Configured, sc.CapacityFrom)
}
if sa := seatOf(t, bound, "ace", "node-build-go"); sa.Capacity != 2 || sa.Configured != 0 {
t.Errorf("a module's word older than mesh-config's value is not taken: %+v", sa)
}
// The operator, 2026-10-26: outside its build hours the seat takes nothing new, and says so.
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":0,"configured":2,"build-hours":"18:00-06:00 CET","outside-hours":true,"why":"0, outside its build hours 18:00-06:00 CET until 18:00 (otherwise 2)","at":"2026-10-26T11:00:00Z"}`})
if so := seatOf(t, u.Usage(), "shanks", "node-build-go"); so.Capacity != 0 || !so.OutsideHours || so.Eligible == nil || *so.Eligible ||
so.WhyNot != "outside its build hours 18:00-06:00 CET on shanks: it takes nothing new until they open" {
t.Errorf("a seat outside its build hours: %+v", so)
}
// A build-hours value the module cannot read: it builds at any hour, and the seat says the value and why.
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":2,"configured":2,"build-hours-fault":"the build hours \"shanks=18:00-6\": \"6\" is not a time HH:MM","why":"2; at any hour, because its build hours cannot be read","at":"2026-10-26T11:00:00Z"}`})
if sf := seatOf(t, u.Usage(), "shanks", "node-build-go"); sf.BuildHours != "any hour" || !strings.Contains(sf.BuildHoursFault, "shanks=18:00-6") || sf.Eligible == nil || !*sf.Eligible {
t.Errorf("a seat whose build hours cannot be read: %+v", sf)
}
if len(bound.TriesRunning) != 1 || len(bound.TriesRunning[0].Asks) != 1 || !strings.Contains(string(bound.TriesRunning[0].Steps), "unit: tests") ||
strings.Contains(strings.Join(bound.Notes, " | "), "name no ask yet") {
t.Errorf("a running try naming its asks: %+v, notes %v", bound.TriesRunning, bound.Notes)
}
} }
var tryAsk0 = stepID(at("11:58"), "a00010")
@@ -23,6 +23,8 @@ type Entry struct {
Deleted bool Deleted bool
// Revision is the bucket revision of the change, when the watch gives one (the source revision is the gate's source key's). // Revision is the bucket revision of the change, when the watch gives one (the source revision is the gate's source key's).
Revision uint64 Revision uint64
// History says the change comes from a watch with history (ReachHistory): it feeds only the history the reach reads.
History bool
} }
// Row is one module on one node. // Row is one module on one node.
@@ -240,6 +242,14 @@ type Model struct {
// the watchman's buckets, by key // the watchman's buckets, by key
presence map[string]presenceEntry presence map[string]presenceEntry
stuck map[string]stuckEntry stuck map[string]stuckEntry
// how far each repository reached (reach.go, ADR 0556)
modHist map[string][]delivery.Module // the indexer's values of each module, oldest first
srcHist map[uint64]map[string]string // the source's heads, by revision
taken map[string]delivery.Taken // the indexer's taken, by repository key
reach map[string]delivery.Reach // the reach derived last, by repository key
reachSaid map[string]uint64 // the reached head's source revision said last, by repository key
reachNotYet map[string]string // the merges said not yet on the machines with it, by repository key
reachPut func(ReachWrite)
} }
type presenceEntry struct { type presenceEntry struct {
@@ -348,21 +358,41 @@ type nodeEntry struct {
func New() *Model { func New() *Model {
return &Model{assigns: map[string]assignment{}, builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{}, return &Model{assigns: map[string]assignment{}, builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{},
haves: map[string]haveEntry{}, nodes: map[string]nodeEntry{}, haves: map[string]haveEntry{}, nodes: map[string]nodeEntry{},
presence: map[string]presenceEntry{}, stuck: map[string]stuckEntry{}} presence: map[string]presenceEntry{}, stuck: map[string]stuckEntry{},
modHist: map[string][]delivery.Module{}, srcHist: map[uint64]map[string]string{}, taken: map[string]delivery.Taken{},
reach: map[string]delivery.Reach{}, reachSaid: map[string]uint64{}, reachNotYet: map[string]string{}}
} }
// Apply takes one change of a key. A value it cannot read is kept as an error on its row, never guessed. When the change makes // Apply takes one change of a key. A value it cannot read is kept as an error on its row, never guessed. When the change makes
// the source revision delivered, the event of it is said once (SetDeliveredEvent), after the model's lock is released. // the source revision delivered, the event of it is said once (SetDeliveredEvent), after the model's lock is released.
func (m *Model) Apply(e Entry) { func (m *Model) Apply(e Entry) {
m.mu.Lock() m.mu.Lock()
m.apply(e) var event *DeliveredEvent
event := m.deliveredNow() var motion []MotionEvent
say := m.deliveredEvent if e.History {
motion, sayMotion := m.motionNow(e), m.motionEvent m.applyHistory(e)
} else {
m.apply(e)
event = m.deliveredNow()
motion = m.motionNow(e)
}
writes, reached := m.reachNow()
say, put := m.deliveredEvent, m.reachPut
sayMotion := m.motionEvent
m.mu.Unlock() m.mu.Unlock()
if event != nil && say != nil { if event != nil && say != nil {
say(*event) say(*event)
} }
if put != nil {
for _, w := range writes {
put(w)
}
}
if say != nil {
for _, ev := range reached {
say(ev)
}
}
if sayMotion != nil { if sayMotion != nil {
for _, ev := range motion { for _, ev := range motion {
sayMotion(ev) sayMotion(ev)
@@ -373,10 +403,18 @@ func (m *Model) Apply(e Entry) {
// DeliveredEvent is what the view says when the source revision is delivered (event `delivered` of the view, issue 1906 D9, ADR // DeliveredEvent is what the view says when the source revision is delivered (event `delivered` of the view, issue 1906 D9, ADR
// 0539 §17): the revision of the gate's source key, the group that moved it and every repository's merged head at it. A // 0539 §17): the revision of the gate's source key, the group that moved it and every repository's merged head at it. A
// consumer takes every earlier revision as covered by it, and takes the same revision twice as once (after a restart of the view). // consumer takes every earlier revision as covered by it, and takes the same revision twice as once (after a restart of the view).
//
// The same event says a repository's reached head (ADR 0556 clause 3): Repository names it, Heads holds its head alone, Revision
// is the source revision that named that head and Group is empty; a consumer takes every earlier merge of that repository as on
// the machines.
type DeliveredEvent struct { type DeliveredEvent struct {
Revision uint64 `json:"revision"` Revision uint64 `json:"revision"`
Group string `json:"group"` Group string `json:"group"`
Heads map[string]string `json:"heads"` Heads map[string]string `json:"heads"`
Repository string `json:"repository,omitempty"`
// NotYet are, in the per-repository form, the merges at or before the reached head that are not yet on the machines: the
// changes of a module a node wants and runs nothing countable of. Said again when they change.
NotYet []string `json:"not_yet,omitempty"`
} }
// SetDeliveredEvent sets where the event of a delivered source revision goes; nothing is said until it is set. // SetDeliveredEvent sets where the event of a delivered source revision goes; nothing is said until it is set.
@@ -512,6 +550,16 @@ func (m *Model) apply(e Entry) {
} }
d, err := delivery.DecodeComposedDigest(e.Value) d, err := delivery.DecodeComposedDigest(e.Value)
m.composed[e.Key] = composedEntry{value: d, err: errText(err)} m.composed[e.Key] = composedEntry{value: d, err: errText(err)}
case delivery.TakenBucket:
if e.Deleted {
delete(m.taken, e.Key)
return
}
if t, err := delivery.DecodeTaken(e.Value); err == nil {
m.taken[e.Key] = t
} else {
delete(m.taken, e.Key)
}
case delivery.SourceBucket: case delivery.SourceBucket:
if e.Key != delivery.SourceMainKey { if e.Key != delivery.SourceMainKey {
return return
@@ -1214,10 +1262,22 @@ func (m *Model) Initial(bucket string) {
} }
} }
say := m.motionEvent say := m.motionEvent
writes, reached := m.reachNow()
put, sayDelivered := m.reachPut, m.deliveredEvent
m.mu.Unlock() m.mu.Unlock()
for _, ev := range out { for _, ev := range out {
say(ev) say(ev)
} }
if put != nil {
for _, w := range writes {
put(w)
}
}
if sayDelivered != nil {
for _, ev := range reached {
sayDelivered(ev)
}
}
} }
// motionOf is the events to say now about the node and module of a model key, from their want, have and stuck key. // motionOf is the events to say now about the node and module of a model key, from their want, have and stuck key.
@@ -110,7 +110,7 @@ func TestWaitsForTheNodeAndForANewerBuild(t *testing.T) { // 1134/AC2
} }
} }
func TestLeftBehindIsNotRunning(t *testing.T) { // 1134/AC3 func TestLeftBehindIsNotRunningAndPulledInIs(t *testing.T) { // 1134/AC3 (both halves)
for name, extra := range map[string]string{"ended": `,"ended":"2026-10-18T11:30:00Z"`, "state": `,"state":"pending"`} { for name, extra := range map[string]string{"ended": `,"ended":"2026-10-18T11:30:00Z"`, "state": `,"state":"pending"`} {
m := New() m := New()
fill(m, base(t)...) fill(m, base(t)...)
@@ -131,13 +131,11 @@ func TestLeftBehindIsNotRunning(t *testing.T) { // 1134/AC3
if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 { if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 {
t.Fatalf("a have with no assignment: %+v counted %d", find(t, s, "a", "w"), s.Counted()) t.Fatalf("a have with no assignment: %+v counted %d", find(t, s, "a", "w"), s.Counted())
} }
} // A module pulled in by another stands: shown and counted, with what required it.
m = New()
func TestPulledInIsRunning(t *testing.T) { // 1134/AC3
m := New()
fill(m, base(t)...) fill(m, base(t)...)
fill(m, assign("b", "z", "b3", `,"pulled-in":true,"required-by":"x"`)) fill(m, assign("b", "z", "b3", `,"pulled-in":true,"required-by":"x"`))
s := m.Snapshot() s = m.Snapshot()
r := find(t, s, "b", "z") r := find(t, s, "b", "z")
if r.Status != StatusAssigned || r.RequiredBy != "x" || r.Have != "b3" || s.Counted() != 4 { if r.Status != StatusAssigned || r.RequiredBy != "x" || r.Have != "b3" || s.Counted() != 4 {
t.Fatalf("pulled-in row = %+v counted %d, want shown and counted with required-by x", r, s.Counted()) t.Fatalf("pulled-in row = %+v counted %d, want shown and counted with required-by x", r, s.Counted())
+47 -4
View File
@@ -3,8 +3,11 @@ package manifest
import ( import (
"bytes" "bytes"
"encoding/json" "encoding/json"
"io/fs"
"os" "os"
"path/filepath"
"reflect" "reflect"
"regexp"
"slices" "slices"
"strings" "strings"
"testing" "testing"
@@ -84,7 +87,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
if m.Module != "mesh-delivery-view" { if m.Module != "mesh-delivery-view" {
t.Errorf("module = %q", m.Module) t.Errorf("module = %q", m.Module)
} }
want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.wants", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "node-engine.have"} // build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9 want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.wants", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.modules", "mesh-delivery-indexer.placement", "mesh-delivery-indexer.taken", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "node-engine.have"} // build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9
got := slices.Clone(m.Reads) got := slices.Clone(m.Reads)
slices.Sort(got) slices.Sort(got)
if !reflect.DeepEqual(got, want) { if !reflect.DeepEqual(got, want) {
@@ -103,8 +106,9 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
if len(m.Uses) != 0 { if len(m.Uses) != 0 {
t.Errorf("uses = %v, want none", m.Uses) t.Errorf("uses = %v, want none", m.Uses)
} }
// The one state the view writes: the operator's answers to the asks that need no proof (ADR 0528 clause 6, 1297/B9), which the teller reads. // The states the view writes: the operator's answers to the asks that need no proof (ADR 0528 clause 6, 1297/B9), which the
if want := []state{{Name: "answers", History: 5}}; !reflect.DeepEqual(m.State, want) { // teller reads, and how far each repository reached (ADR 0556 clause 2, issue 1280), which the gate reads.
if want := []state{{Name: "answers", History: 5}, {Name: "reach", History: 1}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want) t.Errorf("state = %+v, want %+v", m.State, want)
} }
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 { if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -115,7 +119,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
} }
// The two read tools of the indexer's modules (issue 1562), carried by a bundle the runtime launches; the indexer // The two read tools of the indexer's modules (issue 1562), carried by a bundle the runtime launches; the indexer
// itself serves no tool (ADR 0437 §4). Carrying them gives the tool runner read access to every bucket the view reads. // itself serves no tool (ADR 0437 §4). Carrying them gives the tool runner read access to every bucket the view reads.
if want := []string{"indexer_module", "indexer_modules", "indexer_candidate", "indexer_candidates", "builder_build", "builder_builds", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "build_usage"}; !reflect.DeepEqual(m.Tools, want) { if want := []string{"indexer_module", "indexer_modules", "indexer_placement", "indexer_candidate", "indexer_candidates", "builder_build", "builder_builds", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "build_usage", "view_reach"}; !reflect.DeepEqual(m.Tools, want) {
t.Errorf("tools = %v, want %v", m.Tools, want) t.Errorf("tools = %v, want %v", m.Tools, want)
} }
for _, a := range m.Build.Artifacts { for _, a := range m.Build.Artifacts {
@@ -280,3 +284,42 @@ func TestNoDirectoryBelowRootsTrees(t *testing.T) { // 1134/AC10
t.Error("the socket's directory is /var/lib/mesh-delivery-view-cli, 0755, owned by mesh-delivery-view") t.Error("the socket's directory is /var/lib/mesh-delivery-view-cli, 0755, owned by mesh-delivery-view")
} }
} }
// TestEveryStateTheCommandsReadIsIssued holds every literal stdio.State name in cmd/ to what the mesh issues this
// module: a state of its own by its bare name, another module's by <module>.<name> as `reads` lists it. The tools read
// their own reach as "mesh-delivery-view.reach", which the mesh never issued, and view_reach failed on the machines
// (issue 1280, note 13).
func TestEveryStateTheCommandsReadIsIssued(t *testing.T) {
m, _ := load(t)
issued := map[string]bool{}
for _, s := range m.State {
issued[s.Name] = true
}
for _, r := range m.Reads {
issued[r] = true
}
literal := regexp.MustCompile(`stdio\.State\("([^"]+)"\)`)
seen := 0
err := filepath.WalkDir("cmd", func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() || !strings.HasSuffix(path, ".go") {
return err
}
src, err := os.ReadFile(path)
if err != nil {
return err
}
for _, match := range literal.FindAllSubmatch(src, -1) {
seen++
if name := string(match[1]); !issued[name] {
t.Errorf("%s reads the state %q, which the mesh does not issue this module (its own by bare name, another's as `reads` lists it)", path, name)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
if seen == 0 {
t.Fatal("no stdio.State literal found under cmd/: the scan reads nothing")
}
}
+7 -2
View File
@@ -6,6 +6,7 @@
"tools": [ "tools": [
"indexer_module", "indexer_module",
"indexer_modules", "indexer_modules",
"indexer_placement",
"indexer_candidate", "indexer_candidate",
"indexer_candidates", "indexer_candidates",
"builder_build", "builder_build",
@@ -14,7 +15,8 @@
"assigner_want", "assigner_want",
"assigner_differences", "assigner_differences",
"node_haves", "node_haves",
"build_usage" "build_usage",
"view_reach"
], ],
"emits": [ "emits": [
"delivered", "delivered",
@@ -23,7 +25,8 @@
"stuck" "stuck"
], ],
"state": [ "state": [
{ "name": "answers", "history": 5 } { "name": "answers", "history": 5 },
{ "name": "reach", "history": 1 }
], ],
"reads": [ "reads": [
"mesh-controller.assignments", "mesh-controller.assignments",
@@ -45,6 +48,8 @@
"mesh-delivery-gate.source", "mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.modules", "mesh-delivery-indexer.modules",
"mesh-delivery-indexer.placement",
"mesh-delivery-indexer.taken",
"mesh-delivery-teller.answers", "mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence", "mesh-delivery-watchman.presence",
"mesh-delivery-watchman.stuck", "mesh-delivery-watchman.stuck",
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-grants
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 h1:ikD/ggf9/5k23sxJOX1PFcGzb4M3+QeC66AT0MyQ9Zs= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117 h1:88CNZOoQgu7YPYKW16qkWT8Gw9Ucg07aTrIImA2nYUk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026210326-979644420117/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+37
View File
@@ -708,6 +708,43 @@ func TestGrants(t *testing.T) {
} }
}) })
// 1126 (ADR 0549 §1, §3): once the catalogue places the nodes, a seat held once for the mesh is held where placement
// says, not on every machine running a module that claims it; what placement adds is composed beside the assignments.
t.Run("a mesh seat is held where placement says", func(t *testing.T) {
store := catalogue.Manifest{Module: "store", Version: "1",
DefinesSeats: []catalogue.SeatDeclaration{{Name: "store-seat", Scope: catalogue.ScopeMesh, Serves: []catalogue.Verb{{Name: "databases"}}}},
Claims: []catalogue.Claim{{Name: "store-seat", Scope: catalogue.ScopeMesh}}}
in := grants.Inputs{Nodes: []string{"a", "b"}, Assigned: map[string][]string{"a": {"store"}, "b": {"store"}},
Manifests: map[string]catalogue.Manifest{"store": store}}
holds := func(recs buscompose.Records, node string) int {
n := 0
for _, d := range recs.Assigned[node] {
n += len(d.Holds)
}
return n
}
if recs, _ := grants.Records(in); holds(recs, "a") != 1 || holds(recs, "b") != 1 {
t.Fatalf("before placement the seat is held on a %d and b %d times, want both", holds(recs, "a"), holds(recs, "b"))
}
in.Placed = map[string]delivery.Placement{
"a": {Node: "a", Modules: []delivery.PlacedModule{{Module: "store"}}, Holds: []delivery.Holding{{Claim: "store-seat", Scope: "mesh", Module: "store"}}},
"b": {Node: "b", Modules: []delivery.PlacedModule{{Module: "store"}}}}
if recs, _ := grants.Records(in); holds(recs, "a") != 1 || holds(recs, "b") != 0 {
t.Errorf("placed, the seat is held on a %d and b %d times, want a alone", holds(recs, "a"), holds(recs, "b"))
}
b := fixture()
pl, _ := delivery.EncodePlacement(delivery.Placement{Node: "b", Modules: []delivery.PlacedModule{{Module: "mod-y"}, {Module: "mod-x"}}})
b.states[grants.ReadPlacement] = map[string][]byte{"placement.b": pl}
read := grants.ReadInputs(b, &grants.Manifests{})
if !slices.Contains(read.Assigned["b"], "mod-x") || read.Placed["b"].Node != "b" || slices.ContainsFunc(read.Gaps, func(g grants.Gap) bool { return g.Input == "seat holdings" }) {
t.Errorf("read: assigned %v, placed %v, gaps %v", read.Assigned, read.Placed, read.Gaps)
}
b.refused[grants.ReadPlacement] = true
if read := grants.ReadInputs(b, &grants.Manifests{}); slices.ContainsFunc(read.Gaps, func(g grants.Gap) bool { return g.Input == grants.ReadPlacement && g.Failed }) {
t.Errorf("placement not on the bus is a failed read: %v", read.Gaps)
}
})
t.Run("compare excuses a retired verb and a multiply claimed mesh seat's added subscriptions on its machines, nothing else", func(t *testing.T) { t.Run("compare excuses a retired verb and a multiply claimed mesh seat's added subscriptions on its machines, nothing else", func(t *testing.T) {
const retired, store, storeInfo = "mesh.seat.mesh-delivery.tool.mark-ready", "mesh.seat.mesh-store.tool.databases", "$SRV.INFO.mesh-store.>" const retired, store, storeInfo = "mesh.seat.mesh-delivery.tool.mark-ready", "mesh.seat.mesh-store.tool.databases", "$SRV.INFO.mesh-store.>"
two := map[string][]string{"mesh-store": {"a", "n"}} two := map[string][]string{"mesh-store": {"a", "n"}}
@@ -18,7 +18,7 @@ type Watcher interface {
// Followed are the states whose change composes again: placement, the indexer's manifests by module, the resolved // Followed are the states whose change composes again: placement, the indexer's manifests by module, the resolved
// config values, the node agents' haves (a machine's first have, and the bus node's loaded list) and the assigner's // config values, the node agents' haves (a machine's first have, and the bus node's loaded list) and the assigner's
// wants (the manifest each machine is wanted to run), and the vault's hashes of the bus accounts. // wants (the manifest each machine is wanted to run), and the vault's hashes of the bus accounts.
var Followed = []string{ReadAssignments, ReadModules, ReadResolved, ReadHave, ReadWants, ReadAccounts} var Followed = []string{ReadAssignments, ReadModules, ReadResolved, ReadHave, ReadWants, ReadAccounts, ReadPlacement}
// Follower composes again on every change it hears, never on a timer. It never stops for a refusal: a watch it cannot // Follower composes again on every change it hears, never on a timer. It never stops for a refusal: a watch it cannot
// open is said, composed without, and opened again at the next wake (any change it does hear, or a reconnect). // open is said, composed without, and opened again at the next wake (any change it does hear, or a reconnect).
+41 -2
View File
@@ -34,11 +34,14 @@ const (
ReadBuilds = "mesh-delivery-builder.builds" ReadBuilds = "mesh-delivery-builder.builds"
// ReadAccounts is the vault's bcrypt hashes of the bus accounts it holds, by bus user (ADR 0547 clause 9). // ReadAccounts is the vault's bcrypt hashes of the bus accounts it holds, by bus user (ADR 0547 clause 9).
ReadAccounts = "mesh-vault.bus-accounts" ReadAccounts = "mesh-vault.bus-accounts"
// ReadPlacement is the indexer's placement, read from the catalogue (ADR 0549 §2, §3): each node's modules and the seats
// they hold.
ReadPlacement = "mesh-delivery-indexer.placement"
) )
// Reads is every state the part reads, in the manifest's order. // Reads is every state the part reads, in the manifest's order.
var Reads = []string{ReadAssignments, ReadControllerGrant, ReadModules, ReadManifests, ReadResolved, ReadHave, ReadWants, ReadBuilds, var Reads = []string{ReadAssignments, ReadControllerGrant, ReadModules, ReadManifests, ReadResolved, ReadHave, ReadWants, ReadBuilds,
ReadAccounts} ReadAccounts, ReadPlacement}
// The bus module's tools the part calls, as its manifest's invokes name them: read-only, each. // The bus module's tools the part calls, as its manifest's invokes name them: read-only, each.
const ( const (
@@ -103,8 +106,12 @@ type resolved struct {
type Inputs struct { type Inputs struct {
// Nodes is every machine seen: in a node agent's have or a standing assignment. // Nodes is every machine seen: in a node agent's have or a standing assignment.
Nodes []string Nodes []string
// Assigned is each machine's standing assignments, sorted. // Assigned is each machine's standing assignments and what the catalogue's placement puts there, sorted: while both
// stand (ADR 0549 §5) a module either names keeps its membership.
Assigned map[string][]string Assigned map[string][]string
// Placed is each node's placement as the indexer read it from the catalogue (ADR 0549 §2); empty before any is read.
// Where it is, a seat held once for the mesh is held where placement says, not wherever a module claiming it runs.
Placed map[string]delivery.Placement
// Manifests is each module's manifest at the indexer's trunk, by module. // Manifests is each module's manifest at the indexer's trunk, by module.
Manifests map[string]catalogue.Manifest Manifests map[string]catalogue.Manifest
// Digests is each module's manifest digest, by module. // Digests is each module's manifest digest, by module.
@@ -230,6 +237,38 @@ func ReadInputs(r Reader, cache *Manifests) Inputs {
in.BusNode = a.Node in.BusNode = a.Node
} }
} }
// The catalogue's placement (ADR 0549 §3): what it places is composed beside the assignments, and the seats held once
// for the mesh are held where it says. Not on the bus yet is a standing gap, never a failed read, so the list is still
// composed and sealed from the assignments alone.
if keys, err := r.Keys(ReadPlacement); err != nil {
in.Gaps = append(in.Gaps, Gap{Input: ReadPlacement, Why: "not read yet: the assignments alone place modules (" + err.Error() + ")"})
} else {
for _, k := range keys {
raw, found, err := r.Get(ReadPlacement, k)
if err != nil || !found {
continue
}
p, err := delivery.DecodePlacement(raw)
node, kerr := delivery.ParsePlacementKey(k)
if err != nil || kerr != nil || p.Node != node {
in.Gaps = append(in.Gaps, Gap{Input: ReadPlacement + " " + k, Why: "not a placement, so the node is composed from its assignments alone"})
continue
}
if in.Placed == nil {
in.Placed = map[string]delivery.Placement{}
}
in.Placed[node] = p
nodes[node] = true
for _, m := range p.Assigned() {
if !slices.Contains(in.Assigned[node], m) {
in.Assigned[node] = append(in.Assigned[node], m)
}
}
}
}
if len(in.Placed) > 0 {
in.Gaps = slices.DeleteFunc(in.Gaps, func(g Gap) bool { return g.Input == "seat holdings" })
}
haveKeys, err := r.Keys(ReadHave) haveKeys, err := r.Keys(ReadHave)
if err != nil { if err != nil {
in.Gaps = append(in.Gaps, Gap{Input: ReadHave, Why: "refused or absent: " + err.Error(), Failed: true}) in.Gaps = append(in.Gaps, Gap{Input: ReadHave, Why: "refused or absent: " + err.Error(), Failed: true})
@@ -7,6 +7,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/buscompose" "git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
) )
// Records is what the bus composer composes from, rebuilt from the bus (mesh-controller internal/inventory busrecords.go, // Records is what the bus composer composes from, rebuilt from the bus (mesh-controller internal/inventory busrecords.go,
@@ -59,6 +60,9 @@ func Records(in Inputs) (buscompose.Records, []Gap) {
continue continue
} }
d := declaredFor(m, seats, declarers) d := declaredFor(m, seats, declarers)
if len(in.Placed) > 0 {
d.Holds = heldWherePlaced(d.Holds, in.Placed[node], m.Module)
}
for _, sr := range catalogue.ReadsGranted(m, here, node) { for _, sr := range catalogue.ReadsGranted(m, here, node) {
d.KeyedReads = append(d.KeyedReads, buscompose.KeyedRead{Read: sr.Read, Key: sr.Key}) d.KeyedReads = append(d.KeyedReads, buscompose.KeyedRead{Read: sr.Read, Key: sr.Key})
} }
@@ -299,3 +303,22 @@ func hearsOf(m catalogue.Manifest, declared map[string]catalogue.Manifest, node
} }
return out, touches, onEvents return out, touches, onEvents
} }
// heldWherePlaced is a module's holds on a node as the catalogue's placement says them (ADR 0549 §1, §3): a seat held once
// for the mesh is held there only when the node's placement names this module holding it, so a module claiming such a seat
// on two machines holds it on the one placement names, not on both; a seat of every node is held wherever the module runs.
func heldWherePlaced(holds []buscompose.Seat, p delivery.Placement, module string) []buscompose.Seat {
var out []buscompose.Seat
for _, h := range holds {
if h.Scope != "" && h.Scope != catalogue.ScopeMesh {
out = append(out, h)
continue
}
if slices.ContainsFunc(p.Holds, func(x delivery.Holding) bool {
return x.Claim == h.Name && x.Module == module && x.Kind == h.Kind
}) {
out = append(out, h)
}
}
return out
}
+1
View File
@@ -10,6 +10,7 @@
"mesh-delivery-builder.builds", "mesh-delivery-builder.builds",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-indexer.modules",
"mesh-delivery-indexer.placement",
"node-engine.have", "node-engine.have",
"mesh-vault.bus-accounts" "mesh-vault.bus-accounts"
], ],