Compare commits

..
Author SHA1 Message Date
jschoubben 3993175e45 Checker, builder and grants part: pin mesh-sdk main, which holds both the dead-letter hearing (#92) and the placement (#91) (issue 734)
mesh/delivery ready, waits for a review at 3993175e457f
mesh/delivery-group group fix/734-lost-ask checking: composed again for its members' new heads
mesh/check failed: ace: the check of novox/mesh-delivery#73 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 14:45:33 +01:00
jschoubben 60c7ffc981 Checker, builder and grants part: hear DEAD_LETTERS again until heard, say when they do not, and compose the same grant (issue 734, review round 1)
mesh-grants moves to the same mesh-sdk (3203358c), so its shadow user list grants the checker and the builder the
DEAD_LETTERS reads the controller's list will, and the take-over is not blocked by a difference. The checker and the
builder hear DEAD_LETTERS through buildrun.DeadLetterHearing: in the background, again at every reconnect and every ask
waited for while deaf, and keep whether they hear it in their new state hearing, which checker_status and builder_stuck
say as dead_letters. checker_try says that a try ended by the not-taken bound leaves its ask in the stream.

Not done as asked: a module health tool. The catalogue refuses a tool check on a module with no runtime of its own and no
other check beside it (ADR 0227 rule 8, ADR 0366 §1), so the deaf state is said in the status tools and the log.
2026-10-26 14:44:53 +01:00
jschoubben 33de9dde4f Checker and builder: wait on the build seats through mesh-sdk's LostAskWaiter, so an ask the seat gave up on ends every asker's wait (issue 734)
The waiter moved to mesh-sdk buildrun (fix/734-lost-ask, d96b9fd6), where every asker of the build seats shares it.
The builder hears DEAD_LETTERS for the go, node and image seats and asks a given-up part of a build again at once, with
the seat's words as why, instead of waiting for the check bound. The checker keeps what it does with a lost ask (a try
ends as error, a check is asked again) and its verdict words.
2026-10-26 14:44:52 +01:00
jschoubben 6b85220fa4 Checker: an ask the build seat gives up on ends its try as error and asks a check again; a try no node takes ends at the step bound; checker_try_end ends one by hand (issue 734)
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
2026-10-26 14:44:40 +01:00
mesh-tools 1cbae7d41c Merge pull request 'The checker asks the packages a node could not test again, on a node that has what they need (issue 890, ADR 0552)' (#72) from feat/890-placement into main 2026-10-26 13:40:45 +00:00
mesh-tools b3a4b09f5b Merge pull request 'The indexer takes in a registered module whose repository became readable, and serves indexer_take_in (issue 1945)' (#71) from fix/indexer-takes-in into main 2026-10-26 13:39:13 +00:00
mesh-tools c8b26735c8 Merge pull request 'The assigner says when it holds the bus back instead of skipping it silently (issue 1945)' (#70) from fix/bus-bar-loud into main 2026-10-26 13:13:22 +00:00
jschoubben 907e092abc mesh-delivery-indexer reads unread registered modules again in one background run, so a forge timeout holds no merged message or source revision past its ack wait (issue 1945, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b3a4b09f5b6f
mesh/check being carried to 907e092abc45 on b3a4b09f5b6f
The take-in tool's bound now says the take-in is still running and where its answer will be.
2026-10-26 14:11:12 +01:00
jschoubben 3073495a06 checker: pin mesh-sdk 33395f68, feat/890-placement merged with main (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@1cbae7d41cd0
mesh/delivery-group group feat/890-placement merged: every member is merged
mesh/check checking 3073495a0670 on 1cbae7d41cd0
2026-10-26 14:06:10 +01:00
jschoubben 16c411e436 Merge remote-tracking branch 'origin/main' into feat/890-placement 2026-10-26 14:05:40 +01:00
jschoubben 2d841870a8 Assigner: only a put still on its steps is named when a node leaves neither list (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c8b26735c8cf
mesh/check pass 2d841870a873 on c8b26735c8cf | carried from 1d416827157d | unit 185 run, 518 kept | integration 195 run, 584 kept | composition passed…
Review round 2: an ended put stays in the bucket and owns no want.
2026-10-26 14:03:48 +01:00
jschoubben 7f8ce7f5c0 checker: the placed ask names the unmet packages, so a package without a kept verdict is not run or counted twice (issue 890, ADR 0552 §4; review round 1). Pins mesh-sdk 45564e69. 2026-10-26 14:02:16 +01:00
jschoubben 07b5f13474 mesh-delivery-indexer reads a registered module again once its repository becomes readable, and takes one in on ask (issue 1945)
A registered module whose repository the forge answered 404 (car-hunter, snake, before the forge's account could read
them) was read again only when the indexer started again or the controller wrote a new build record of it, so the grants
part's composed user list stayed short of their subjects and the take-over was refused. A failed read now keeps the
module as unread, read again on the next source revision and the next merged message (no timer of its own), and the new
tool indexer_take_in (the tools bundle mesh-delivery-indexer-tools, through the indexer's takeins state, as the
builder's builder_ask_again) takes one in now and answers what it stored or why not.
2026-10-26 14:01:34 +01:00
jschoubben f4aac599ca Assigner: name a running put when a node leaves neither list, and say the bus bar in plain words (issue 1945)
Review round 1: while an operator's put runs through its steps it owns the want, so the clear said
'the node does not run it' of a node that runs the bus.
2026-10-26 14:00:32 +01:00
jschoubben 2d63f0ba62 Assigner: say the bus bar instead of skipping it silently, and clear a neither condition only in true words (issue 1945)
A bus whose move only the operator's put may make (ADR 0546 §8) got no want and nothing in the outcome:
nats on novox, running an old-form build, was on no list while the pass cleared its condition and logged
that it had a want again. The bar now names its holding, puts a node without a live want on neither list
(its condition needs the operator: assigner_put), and a neither condition is cleared as 'has a want again'
only when a live want exists.
2026-10-26 13:55:42 +01:00
mesh-tools 1d41682715 Merge pull request 'mesh-grants says loudly when it holds the user list back, and clears it after a restart (issue 1945)' (#69) from fix/grants-hold-loud into main 2026-10-26 12:44:36 +00:00
jschoubben 705765d113 checker: a placed ask that every node declines for longer than its bound (30 min) fails the check naming the need (issue 890, ADR 0552 review) 2026-10-26 13:42:46 +01:00
jschoubben 16a97ed65d checker: ask the packages a node left unmet again of a node that has what their tests need, and put a declined ask again (issue 890)
A check that landed on ace passed with mesh-build-node's integration tests skipped. The build module now leaves such a
package unmet; the checker keeps the passes and asks the levels from the first unmet one with the need named, putting the
ask again under a new id when a node lacking it declines it. Re-pins mesh-sdk feat/890-placement.
2026-10-26 13:40:00 +01:00
mesh-tools ecf532af0e Merge pull request 'checker_status reads the node-engine at its running commit, as the checker decides (issue 568)' (#68) from fix/status-and-holds into main 2026-10-26 12:39:56 +00:00
jschoubben ce600df98a Say the held-list state on a part's first pass, so a warning raised before a restart is cleared
mesh/delivery merged: source.main moved to novox/mesh-delivery@1d416827157d
mesh/check being carried to ce600df98afc on 1d416827157d
2026-10-26 13:37:31 +01:00
jschoubben b4a6a34705 mesh-grants: a user list held back after take-over raises the warning mesh-grants.user-list-held and clears it when the list advances, so the hold is not only in a tool's output (issue 1945) 2026-10-26 13:36:39 +01:00
jschoubben a7e51b416b checker_status reads the node-engine at its running commit, as the checker's decision does, so it stops listing every candidate as waiting for a trunk move (issue 568)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ecf532af0e85
mesh/check being carried to a7e51b416bc1 on ecf532af0e85 | from 71f786f14662
2026-10-26 13:25:18 +01:00
mesh-tools 71f786f146 Merge pull request 'mesh-grants: take-over and hand-back by the operator's Telegram Approve, the sealed list, and every grant the loaded list holds (issue 1945)' (#67) from feat/1945-user-list-switch into main 2026-10-26 12:21:50 +00:00
mesh-tools 848c6fdc56 Merge pull request 'Checker: one check per set of heads, a trunk or sibling move carries the verdict, a merge turn holds a claim (ADR 0548, issue 568)' (#65) from feat/548-one-check into main 2026-10-26 12:01:32 +00:00
mesh-tools ead9ee2d13 Merge pull request 'Indexer: index a module registered from any branch, by its registered ref, never a newer pull request's (issue 1945)' (#66) from fix/indexer-registered-modules-2 into main 2026-10-26 12:01:24 +00:00
mesh-tools 4861d1f78d Merge pull request 'Checker: keep no package whose tests skipped, so a skipped pass is never reused as proof (issue 890)' (#64) from fix/checker-skips-not-kept into main 2026-10-26 12:01:19 +00:00
jschoubben 5c76c9d3ac mesh-grants: a list in force with no users kept is checked against the users the bus loaded, so the first list after take-over keeps every user (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@71f786f14662
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check being carried to 5c76c9d3ac84 on 71f786f14662 | novox/mesh-delivery#67 in group feat/1945-user-list-switch
2026-10-26 12:54:55 +01:00
jschoubben 699cbfb7c1 Indexer: a registered module takes its registered ref, not a newer branch's record; no read at an older commit after a failure
mesh/delivery merged: source.main moved to novox/mesh-delivery@ead9ee2d138d
mesh/check checking 699cbfb7c131 on ead9ee2d138d
2026-10-26 12:52:29 +01:00
jschoubben 319d56be7e Indexer: a registered module is read from every repository it was built from, whatever the branch, and at the registered commit when the head is gone 2026-10-26 12:45:35 +01:00
mesh-tools 11d3678984 Merge pull request 'Indexer: index a module registered from its own repository, so the grants part composes its grants (issue 1945)' (#63) from fix/indexer-registered-modules into main 2026-10-26 11:33:41 +00:00
jschoubben f55f150896 mesh-grants: after take-over the list in force is not replaced by one composed past a failed read or one dropping its users for want of a hash; the switch's effect says when the bus node does not watch the key (review of the switch, issue 1945). Pins mesh-sdk c2a410bb. 2026-10-26 12:31:31 +01:00
jschoubben e1f1b5d18b Checker: a package with skipped tests is withheld from the kept verdicts, not the whole result (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4861d1f78d28
mesh/delivery-group group fix/checker-skips-not-kept merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:29:14 +01:00
jschoubben f0fe454145 Checker: a result with skipped tests keeps no verdict, so a skipped pass is run again (issue 890) 2026-10-26 12:23:57 +01:00
jschoubben 4026f6c188 checker: a check is never held by a turn its own heads wait in, the node-engine read at its running commit is named and carried when it moves, and a gate bucket made after the start is opened before the next claim (ADR 0548 §3, §6; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@848c6fdc5648
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-delivery#65 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:14:50 +01:00
jschoubben 29cfb8c177 Pin mesh-sdk at the group's head 83ee7769 (issue 1945) 2026-10-26 12:06:58 +01:00
jschoubben a092ee9ff9 mesh-grants writes the user list sealed to the bus node and names it in user-list.sealed; the switch turns that key; take-over state says switched only once the bus node loaded it (ADR 0547 clauses 7, 8; issue 1945)
The bus node can only take over a list that exists: each pass composes the accounts block with the vault's hashes, seals
it to the key the bus node says in its have, puts it in the part's own object bucket by digest and writes the key with
digest, revision, user count, users left out and which list to follow. take-over is refused, naming why, while no list is
sealed to the bus node's current key or the loaded list does not grant the bus node its read.

Review of fix/1945-takeover-blockers: no responders on the router's record is an error, not 'no record'; an answer whose
record cannot be reached is not closed; an ask that failed to publish is taken back at the router; an error acting on an
answer is logged. Pins mesh-sdk 175a5567.
2026-10-26 11:59:41 +01:00
jschoubben 6c3cbee35c checker: pin mesh-sdk at the group's head f5c233a9288c, which holds the merge turns and the carried verdict (ADR 0548 §10; issue 568) 2026-10-26 11:45:13 +01:00
jschoubben 4b23db28b3 mesh-grants: an open take-over ask is always closed (router record, expiry, failed publish, any outcome), the switch is said recorded not switched, the refusal condition is cleared, the excuse is limited to the claiming machines' subscriptions (issue 1945) 2026-10-26 11:43:39 +01:00
jschoubben 646f83642a Indexer: read a registered module at the source's head when it names one, else at the registered commit
mesh/delivery merged: source.main moved to novox/mesh-delivery@11d3678984af
mesh/check pass 646f83642a01 on 11d3678984af | unit 171 run, 502 kept | integration 181 run, 568 kept | composition passed in 39ms on novox
2026-10-26 11:41:46 +01:00
jschoubben d9bfba1e62 Indexer: read one directory of a registered module's repository, leave a repository the source names to its merges, accept master as a trunk 2026-10-26 11:36:22 +01:00
jschoubben f1bb7747ae checker: one integration test through the real bus proves a merge turn holds a check off until it goes and a carry of unchanged digests runs nothing; it extends the equal-inputs test, so the count stays (ADR 0548 §3, §5; issue 568 D7) 2026-10-26 11:36:15 +01:00
jschoubben d2eb3e5ecf checker: the line of a carried verdict names the head on the new trunk, then where it was carried from; a verdict being carried is pending and says so (ADR 0548 §6, §8; issue 568 D6) 2026-10-26 11:35:16 +01:00
jschoubben b0b1cf4f01 mesh-grants: a retired verb and a multiply claimed mesh seat's added subjects are named, not counted, in the comparison (issue 1945) 2026-10-26 11:34:52 +01:00
jschoubben d91ea41f64 checker: only a head git refused gets a whole check after a trunk move; a conflicting carry fails naming the files and asks nothing (ADR 0548 §7; issue 568 D5) 2026-10-26 11:34:08 +01:00
jschoubben 5fed429827 checker: a verdict names the siblings it read at their trunk, and a sibling's move is carried as a trunk move is (ADR 0548 §6; issue 568 D4) 2026-10-26 11:33:35 +01:00
jschoubben 9a569bfb8d mesh-grants: derive that a bundle loads from build.artifacts, as the indexer's manifest has no derived bundles, so route-proxy's helper is carried (issue 1945) 2026-10-26 11:33:28 +01:00
jschoubben e2c41ffa5a checker: a carry keeps the composition it passed while no composed fingerprint changed, and composes again otherwise (ADR 0548 §5; issue 568 D3) 2026-10-26 11:32:31 +01:00
jschoubben 13e4ed8515 checker: a trunk move never ends a check; once it ended, its pass is carried and names the commits it was carried from, and a failed carry names the move (ADR 0548 §1, §5; issue 568 D2) 2026-10-26 11:30:46 +01:00
jschoubben c66fe11e16 checker: claim no check of a repository an approved merge holds a turn on, and claim it when the turn goes (ADR 0548 §3; issue 568 D1) 2026-10-26 11:29:25 +01:00
jschoubben 75d0eb32be mesh-grants: take-over and hand-back are asked of the operator on their channel and run on their yes after the refusals; take-over no longer always refused; grants_take_over_state (issue 1945) 2026-10-26 11:26:37 +01:00
jschoubben c44b9f91cd Indexer: gofmt 2026-10-26 11:23:27 +01:00
jschoubben 82da252cf0 Indexer: index the manifests of modules registered from their own repositories, so the grants part has no gap for them 2026-10-26 11:23:21 +01:00
jschoubben ce72b67cef mesh-grants: hear the condition events through the controller's durable consumer and acknowledge them; the provided mesh-wireguard manifest always wins (issue 1945) 2026-10-26 11:20:42 +01:00
jschoubben 2a94d16834 mesh-grants: ask the controller's root-free verb and wake on its condition events, so a verified sender is composed (issue 1945)
The part composed every membership with no root-free input, so telegram's verified-sender was dropped and the membership held as a shadow
difference. It now asks mesh-controller.root-free for the machines a composition needs, per pass, failing closed and said once, and wakes
on root-not-free condition-raised and condition-cleared.
2026-10-26 11:11:43 +01:00
jschoubben da7baa5e39 mesh-grants: take mesh-wireguard's manifest from mesh-sdk, where the mesh provides it, so its node-tools grants are composed (issue 1945) 2026-10-26 11:11:43 +01:00
75 changed files with 4919 additions and 367 deletions
@@ -3,7 +3,6 @@ package main
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"time"
@@ -114,7 +113,8 @@ func (p *passer) wants(ctx context.Context) {
// quiet at the router.
first := p.neither == nil
stale := map[string]neitherOne{}
defer func() { p.sayNeither(neither, stale) }()
left := map[string]string{}
defer func() { p.sayNeither(neither, stale, left) }()
for name := range names {
fp := trunk[name].FP2
name := name
@@ -154,6 +154,7 @@ func (p *passer) wants(ctx context.Context) {
}
out := assign.Wants(m, nodes, wanted[name], last)
gatherNeither(neither, name, out)
leftNeither(left, m, wanted[name], out)
p.sayNoWant(name, m.Nodes, wanted[name], out)
if first {
startClears(stale, m, out)
@@ -210,7 +211,7 @@ func (p *passer) sayNoWant(module string, nodes []string, before map[string]deli
}
reason := ""
if h, held := out.Rules[node]; held && (!written || w.Removed) {
reason = fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
reason = h.Reason()
}
last, said := p.noWant[key]
switch {
@@ -242,6 +243,48 @@ func gatherNeither(into map[string]neitherOne, module string, out assign.Outcome
}
}
// leftNeither says, by node and module, for every node of m not on neither list after the pass, why: "" when the node has a live
// want for it (written before and kept, or written in this pass), else why it has none. sayNeither clears a neither condition in
// these words, so a clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while the bus bar
// left it with none).
func leftNeither(into map[string]string, m assign.Module, before map[string]delivery.Want, out assign.Outcome) {
for _, node := range m.Nodes {
if _, on := out.Neither[node]; on {
continue
}
w, written := out.Wants[node]
if !written {
w, written = before[node]
}
why := ""
h, ruled := out.Rules[node]
switch {
case written && !w.Removed:
case (m.Held && !out.Lifted) || out.Held != nil:
why = "it is held, and its held condition says so"
case slices.ContainsFunc(out.Record, func(r assign.RecordNeed) bool { return r.Node == node }):
why = "its move waits for the operator's yes on its record ask"
case putOn(m, node):
why = "its put waits on its own steps and the operator's answer"
case m.FP == "":
why = "the trunk has no build of it"
case ruled:
why = h.Reason()
default:
why = "the node does not run it"
}
into[node+"."+m.Name] = why
}
}
// putOn says whether an operator's put on node owns m's want there: one still on its steps (ADR 0546), so the rules write none
// meanwhile. An ended put (done, refused, released) stays in the bucket and owns nothing.
func putOn(m assign.Module, node string) bool {
p, put := m.Puts[node]
_, owns := assign.PutOwns(p)
return put && owns
}
// startClears adds, for the first pass after a start, every node of m that is not on neither list after the pass: a neither
// condition a previous process raised for it is cleared. That covers a node with a want and one with none that is not on neither
// list, such as a held module (issue 1945: the hold is said on its own, so a neither raised for it before the restart would stay up
@@ -258,7 +301,9 @@ func startClears(into map[string]neitherOne, m assign.Module, out assign.Outcome
// once, the condition of each one said before that has a want again (issue 1945). A say that fails is made again at the next pass.
// stale is, on the first pass after a start, the modules whose condition a previous process may have raised and that have a want:
// each is cleared once.
func (p *passer) sayNeither(now, stale map[string]neitherOne) {
// left is, by node and module, why each assigned node not on neither list after this pass has no want ("" when it has one); a key
// it does not hold is no longer assigned.
func (p *passer) sayNeither(now, stale map[string]neitherOne, left map[string]string) {
if p.neither == nil {
p.neither = map[string]neitherOne{}
for key, one := range stale {
@@ -278,21 +323,29 @@ func (p *passer) sayNeither(now, stale map[string]neitherOne) {
continue
}
}
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; rule %d holds its first want back: %s",
one.module, one.node, one.holding.Rule, one.holding.Words)
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; what holds its first want back: %s",
one.module, one.node, one.holding.Reason())
p.neither[key] = one
}
for key, one := range p.neither {
if _, still := now[key]; still {
continue
}
why, assigned := left[key]
if !assigned {
why = "it is no longer assigned there"
}
if p.say != nil {
if err := p.say(assign.NeitherCondition(one.node, one.module, assign.Holding{}, false)); err != nil {
if err := p.say(assign.NeitherLeftCondition(one.node, one.module, why)); err != nil {
p.logf("the condition of %s on neither list on %s could not be cleared, and is cleared at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s on %s has a want again", one.module, one.node)
if why == "" {
p.logf("%s on %s has a want again", one.module, one.node)
} else {
p.logf("%s on %s is no longer on neither list and has no want: %s", one.module, one.node, why)
}
delete(p.neither, key)
}
}
@@ -357,20 +357,37 @@ func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
p.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
one := map[string]neitherOne{"g14.screen-lock": {node: "g14", module: "screen-lock",
holding: assign.Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}}}
p.sayNeither(one, nil) // the say fails
p.sayNeither(one, nil)
p.sayNeither(one, nil)
p.sayNeither(one, nil, nil) // the say fails
p.sayNeither(one, nil, nil)
p.sayNeither(one, nil, nil)
if len(said) != 1 || said[0].State != asks.StateRaised || said[0].Subject.Machine != "g14" || said[0].Subject.ID != "screen-lock" {
t.Fatalf("said = %+v, want one raised condition of screen-lock on g14", said)
}
if !strings.Contains(strings.Join(logged, "\n"), "screen-lock") {
t.Errorf("the journal does not name the module on neither list: %q", logged)
}
p.sayNeither(map[string]neitherOne{}, nil)
p.sayNeither(map[string]neitherOne{}, nil)
wants := map[string]string{"g14.screen-lock": ""}
p.sayNeither(map[string]neitherOne{}, nil, wants)
p.sayNeither(map[string]neitherOne{}, nil, wants)
if len(said) != 2 || said[1].State != asks.StateCleared || said[1].Key != said[0].Key {
t.Errorf("said = %+v, want the condition cleared once under its key", said)
}
if last := logged[len(logged)-1]; last != "screen-lock on g14 has a want again" {
t.Errorf("the clear said %q", last)
}
// 1945 (the bus bar): a module that leaves neither list with no want is cleared in words saying so, never "has a want again".
bus := map[string]neitherOne{"novox.nats": {node: "novox", module: "nats", holding: assign.Holding{Bus: true, Words: "the bus moves only by the operator's put"}}}
p.sayNeither(bus, nil, nil)
if c := said[len(said)-1]; c.State != asks.StateRaised || !strings.HasPrefix(c.Explanation, asks.NeedsYou) || !strings.Contains(c.Explanation, "the bus moves only by the operator's put") {
t.Errorf("the bus's neither condition = %+v", c)
}
p.sayNeither(map[string]neitherOne{}, nil, map[string]string{"novox.nats": "it is held, and its held condition says so"})
if c := said[len(said)-1]; c.State != asks.StateCleared || strings.Contains(c.Resolved, "has a want") {
t.Errorf("a module that left neither list with no want was cleared as %+v", c)
}
if last := logged[len(logged)-1]; last != "nats on novox is no longer on neither list and has no want: it is held, and its held condition says so" {
t.Errorf("the clear without a want said %q", last)
}
}
// 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module.
@@ -381,6 +398,25 @@ func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
if len(got) != 1 || got["g14.screen-lock"].node != "g14" || got["g14.screen-lock"].module != "screen-lock" || got["g14.screen-lock"].holding.Rule != delivery.RuleGrant {
t.Errorf("gathered = %+v", got)
}
// 1945 (the bus bar): a node off neither list is cleared in words that say why it has no want; an operator's put running
// through its steps owns the want, so it is named, never "the node does not run it".
for _, c := range []struct {
name string
puts map[string]delivery.Put
before map[string]delivery.Want
why string
}{
{"a live want", nil, map[string]delivery.Want{"novox": {FP: fpTest}}, ""},
{"a put on its steps", map[string]delivery.Put{"novox": {FP: fpTest, Step: delivery.PutPerson}}, nil, "its put waits on its own steps and the operator's answer"},
{"nothing", nil, nil, "the node does not run it"},
{"an ended put", map[string]delivery.Put{"novox": {FP: fpTest, Step: delivery.PutRefused}}, nil, "the node does not run it"},
} {
left := map[string]string{}
leftNeither(left, assign.Module{Name: "nats", FP: fpTest, Nodes: []string{"novox"}, Puts: c.puts}, c.before, assign.Outcome{})
if why, there := left["novox.nats"]; !there || why != c.why {
t.Errorf("%s: left %q (%v), want %q", c.name, why, there, c.why)
}
}
// 1945 (a first want held back silently): the journal says why a node gets no want when the reason appears or changes, and
// when it has a want again: one line per change, not per pass.
p := newPasser(buckets{}, time.Date(2026, 10, 25, 21, 30, 0, 0, time.UTC))
@@ -399,6 +435,7 @@ func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
{nil, assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Rule: delivery.RuleGrant, Words: "no credential"}}}, "rofi on shanks gets no want: rule 6: no credential"},
{nil, wanted, "rofi on shanks has a want"},
{map[string]delivery.Want{"shanks": wanted.Wants["shanks"]}, assign.Outcome{}, ""},
{nil, assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Bus: true, Words: "the bus moves only by the operator's put"}}}, "rofi on shanks gets no want: the bus moves only by the operator's put"},
} {
logged = nil
p.sayNoWant("rofi", []string{"shanks"}, c.before, c.out)
@@ -96,6 +96,14 @@ func NeitherCondition(node, module string, h Holding, raised bool) asks.Conditio
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
if h.Bus {
needs := fmt.Sprintf("put %s on %s with assigner_put", module, node)
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner never moves the bus: %s. This clears once the node has a want for it.",
asks.NeedsYou, needs, module, node, h.Words)}
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
@@ -103,6 +111,17 @@ func NeitherCondition(node, module string, h Holding, raised bool) asks.Conditio
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
}
// NeitherLeftCondition clears the neither condition of module on node when it is no longer on neither list: why is "" when the
// node has a live want for it again, else why it has none (it is held, its move waits for an ask, the node no longer runs it), so
// the clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while it had none).
func NeitherLeftCondition(node, module, why string) asks.Condition {
c := NeitherCondition(node, module, Holding{}, false)
if why != "" {
c.Resolved = fmt.Sprintf("%s on %s is no longer on neither list and has no want: %s.", module, node, why)
}
return c
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
c := HeldCondition(module, fp, "", "", false, false)
@@ -192,10 +192,27 @@ type RecordNeed struct {
Words string
}
// Holding is a rule holding a node back, and why.
// Holding is a rule holding a node back, and why. Bus marks the bar on the bus instead of one of the seven rules (Rule is then
// 0): the bus moves only by the operator's put (ADR 0539 §15, ADR 0546 §8), never by the assigner (issue 1945: nats on novox,
// running an old-form build, got no want and the pass said nothing).
type Holding struct {
Rule int
Words string
Bus bool
}
// Reason is the holding in words: the rule and its words, or for the bus bar its words alone.
func (h Holding) Reason() string {
if h.Bus {
return h.Words
}
return fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
}
// BusBarWords say why the assigner writes no want moving the bus on node: it runs prev, the trunk builds fp, and only the
// operator's put moves it (ADR 0539 §15, ADR 0546 §8). The words reach the operator, so they name no record.
func BusBarWords(node, prev, fp string) string {
return fmt.Sprintf("only your put moves the bus: %s runs %s, the trunk builds %s; put it with assigner_put", node, short(prev), short(fp))
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
@@ -303,6 +320,14 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
} else if prev, barred := at.busBar(asked, hadLive, running.FP); barred {
// The bus is barred: no want moves it but the operator's put. Said, never silent (issue 1945): its holding is named,
// and a node that runs it with no live want has it on neither list, so its condition stays raised.
h := Holding{Bus: true, Words: BusBarWords(node, prev, at.FP)}
out.Rules[node] = h
if !hadLive && running.Phase != delivery.PhaseRemoved && !held {
out.Neither[node] = h
}
}
continue
}
@@ -466,12 +491,7 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
// The fingerprint the bus ran before: its want's, or with no readable want the one its have names (ADR 0539 §15 bars a new one).
prev := before.FP
if !had {
prev = running
}
if catalogue.ProvidesBus(m.Manifest) && prev != m.FP {
if _, barred := m.busBar(before, had, running); barred {
return nil, true // the bus is never moved by the assigner; the operator writes its want (§15)
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
@@ -483,6 +503,16 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// busBar says whether m is the bus and the move would take it off the fingerprint it ran before, prev: its want's, or with no
// readable want the one its have names (ADR 0539 §15 bars a new one).
func (m Module) busBar(before delivery.Want, had bool, running string) (prev string, barred bool) {
prev = before.FP
if !had {
prev = running
}
return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP
}
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
@@ -427,6 +427,23 @@ func TestTheBusIsNotMovedButItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
t.Errorf("a held bus did not take its user-list revision as a put: %+v", w)
}
m.Puts = nil
// 1945 (the bus bar made loud): a node that runs the bus at an old-form build and holds no want for it gets none, since only
// the operator's put moves the bus, and the pass says so: a holding naming the bar, and the node on neither list so its
// condition stays raised. A node with a live want keeps it and is not on neither list.
m.Composed = composedEverywhere
n.Haves["ace"]["mail"] = delivery.Have{FP: "b-1700000000-oldform", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out = Wants(m, n, map[string]delivery.Want{"g14": {FP: fOld, Composed: dig(fOld)}}, nil)
if w, wrote := out.Wants["ace"]; wrote {
t.Errorf("the bus bar wrote ace's want %+v", w)
}
h, on := out.Neither["ace"]
if !on || !h.Bus || out.Rules["ace"] != h || !strings.Contains(h.Words, "only your put moves the bus") || !strings.Contains(h.Words, "ace runs b-1700000000-oldform") ||
!strings.Contains(h.Words, "assigner_put") {
t.Errorf("the barred bus on ace: neither %+v (%v), rules %+v", h, on, out.Rules["ace"])
}
if _, on := out.Neither["g14"]; on || out.Wants["g14"].FP == fNew {
t.Errorf("g14 has a live want: neither %+v, its want %+v", out.Neither, out.Wants["g14"])
}
stepsOfAPut(t)
}
@@ -15,6 +15,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-builder/internal/build"
)
@@ -31,7 +32,8 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
Description: "Every build key that waits (building or queued), oldest first: its module, fingerprint, step words and since when, and each ask it " +
"waits on by language with its id, when it was put, its asking (1 for the first; the builder asks an ask again when it can no longer be " +
"answered, at most 3 times) and what its build module's steps hold of it (none, or its state on its node since its start). The " +
"seat workers' delivery counts are not readable by the builder. With module, that module's keys only. Read-only.",
"seat workers' delivery counts are not readable by the builder. dead_letters says whether the builder hears the asks the build seats give up " +
"on (issue 734), and why not: heard, a given-up ask is asked again at once; deaf, at its check bound. With module, that module's keys only. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's keys"}}},
Run: func(args map[string]any) (any, error) { return r.Stuck(str(args, "module")) }},
@@ -166,7 +168,8 @@ func main() {
steps[bm] = read(stdio.State(bm + ".steps"))
}
reasks := stdio.State(build.ReAsksState)
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps}
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps,
Hearing: read(stdio.State(buildrun.HearingState))}
put := func(key string, value []byte) error {
_, err := reasks.Put(key, json.RawMessage(value))
return err
@@ -149,6 +149,16 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
asks := buildrun.JetStreamAsks{JS: js}
// The asks the build seats gave up on (issue 734): heard from DEAD_LETTERS, each ends the wait for its step at once,
// and the lane asks its part again (Builder.waitOnce) instead of waiting for the check bound.
// Heard in the background, and again at a reconnect or an ask while it is not; whether it is heard is kept in the
// state hearing, which builder_stuck says (dead_letters).
dead := &buildrun.DeadLetters{}
hearingKV := process.OptionalBucket(ctx, js, delivery.BuilderModule+"_"+buildrun.HearingState, "builder_stuck cannot say whether DEAD_LETTERS is heard", logger.Printf)
hearing := buildrun.NewDeadLetterHearing(ctx, buildrun.JetStreamDeadLetters{JS: js}, delivery.AskLanguages, dead, logger.Printf,
buildrun.SayHearing(ctx, hearingKV, logger.Printf))
defer hearing.Stop()
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again() })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
account, err := process.ReadAccount(path)
@@ -162,7 +172,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
defer calls.Close()
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: bound},
Asks: buildrun.Asker{Publisher: asks, Waiter: buildrun.LostAskWaiter{Inner: asks, Dead: dead, Hearing: hearing}, Bound: bound},
Held: heldOnTheBus{js},
// An ask that can no longer be answered is asked again (issue 1945), read at the step bound: a context's bound.
Gone: heldOnTheBus{js},
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1 h1:fzOkrz9CSp3mEjeV7NfZx8yoivI1sD1AOtIPtyyF8RI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530 h1:NwCb+gOh0UYqlyG6hG2/jffE/Gd0c7YWd4iJlaCB0lU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **An ask that can no longer be answered is asked again** (novox/hq issue 1945). A build seat's stream gives an ask up
@@ -20,7 +21,8 @@ import (
// restart take over the same id and wait again. The builder notices it at the check bound (CheckAfter, the step bound),
// a context's bound and no timer of its own: it asks Gone, which reads the bus, and asks such an ask again under a new
// id, written into the key, said in its log and as the event build.asked-again, at most Askings times; then the key is
// failed with why. A person asks the same by hand with the tool builder_ask_again (AskAgain).
// failed with why. An ask DEAD_LETTERS says the seat gave up (buildrun.LostAskWaiter, issue 734) is asked again at once.
// A person asks the same by hand with the tool builder_ask_again (AskAgain).
// DefaultAskings is how many times a build module's part of a build is asked, the first included, when Builder.Askings is 0.
const DefaultAskings = 3
@@ -315,6 +317,11 @@ func (b *Builder) waitOnce(ctx context.Context, ask delivery.StepAsk, again <-ch
defer stop()
}
step, err := b.Asks.Wait(wait, ask)
if err == nil && buildrun.IsLost(step) {
// The build seat gave the ask up (issue 734): it can no longer be answered, so it is asked again now, with the
// words of why, not at the check bound.
return step, step.Reason, errAgain
}
if err == nil || ctx.Err() != nil {
return step, "", err
}
@@ -690,12 +690,14 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
step string
byHand bool
always bool // Gone says every ask cannot be answered, not only the first
dead bool // DEAD_LETTERS keeps the first ask: the seat gave it up (issue 734)
puts int
done bool
asking int // the asking the event says, 0 when none is said
}{
{name: "given up by its stream: asked again under a new id and done from it", puts: 2, done: true, asking: 2},
{name: "asked again by hand: the same, with no check bound", byHand: true, puts: 2, done: true, asking: 2},
{name: "given up into DEAD_LETTERS: its wait ends failed with the words and it is asked again at once", dead: true, puts: 2, done: true, asking: 2},
{name: "never answerable: failed after its askings", askings: 2, always: true, puts: 2},
{name: "taken over at its last asking: failed, not asked again", askings: 3, always: true, step: "asked mesh-build-go again (asking 3): x", puts: 0},
} {
@@ -714,7 +716,11 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r.st.vals[r.key] = data
}
gone := &goneAsks{}
if !c.byHand {
dead := &buildrun.DeadLetters{}
if c.dead {
r.b.Asks = buildrun.Asker{Publisher: r.asks, Waiter: buildrun.LostAskWaiter{Inner: r.asks, Dead: dead}, Bound: 2 * time.Second}
}
if !c.byHand && !c.dead {
gone.why = func(a delivery.StepAsk) string {
if c.always || a.ID == first || first == "" {
return "it waits first in line while a node took a later ask"
@@ -724,6 +730,10 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r.b.Gone, r.b.CheckAfter = gone, 5*time.Millisecond
}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
if c.dead {
dead.Heard(buildrun.GivenUp{AskID: waitForAsks(t, r, 1)[0].ID, Stream: "SEAT_NODE_BUILD_GO", Deliveries: 5,
At: time.Date(2026, 10, 26, 12, 26, 40, 0, time.UTC)})
}
if c.byHand {
waitForAsks(t, r, 1)
for deadline := time.Now().Add(2 * time.Second); len(r.b.AskAgain(r.key, "stuck since 19:06")) == 0; time.Sleep(time.Millisecond) {
@@ -757,13 +767,16 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
mu.Lock()
defer mu.Unlock()
// Note 62: the old message of an ask asked again because its stream gave it up is dropped from the stream.
if gone.mu.Lock(); c.done && !c.byHand && (len(gone.dropped) != 1 || gone.dropped[0] != first) {
if gone.mu.Lock(); c.done && !c.byHand && !c.dead && (len(gone.dropped) != 1 || gone.dropped[0] != first) {
t.Errorf("dropped %v from the stream, want the old ask %s", gone.dropped, first)
}
gone.mu.Unlock()
if c.asking > 0 && (len(said) != 1 || said[0].Asking != c.asking || said[0].New == said[0].Old || said[0].Key != r.key || said[0].Why == "") {
t.Errorf("said %+v, want once at asking %d", said, c.asking)
}
if c.dead && (len(said) != 1 || !strings.Contains(said[0].Why, buildrun.LostWords+" after 5 deliveries at 2026-10-26T12:26:40Z")) {
t.Errorf("said %+v, want the ask asked again saying the seat gave it up", said)
}
})
}
t.Run("what the bus says of an ask", func(t *testing.T) { // 1945
@@ -836,12 +849,17 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
running, _ := json.Marshal(delivery.Step{V: delivery.Version, ID: "mvo53jcm70b5f8", Tool: "delivery ask", Node: "g14", State: delivery.StepRunning, Started: at})
_, reask, _ := NewReAsk("m", fp1, "stuck", "shanks.lead", at)
r := StuckReader{Builds: state(map[string]string{key: string(building), done: string(finished)}), ReAsks: state(map[string]string{key: string(reask)}),
Steps: map[string]StateRead{buildrun.GoBuildModule: state(map[string]string{}), ImageBuildModule: state(map[string]string{"g14.mvo53jcm70b5f8": string(running)})},
Now: func() time.Time { return at.Add(time.Hour) }}
Steps: map[string]StateRead{buildrun.GoBuildModule: state(map[string]string{}), ImageBuildModule: state(map[string]string{"g14.mvo53jcm70b5f8": string(running)})},
Hearing: state(map[string]string{buildrun.HearingKey: `{"heard":false,"streams":["SEAT_NODE_BUILD_GO"],"why":"DEAD_LETTERS is not heard: SEAT_NODE_BUILD_GO: permissions violation","at":"2026-10-26T13:00:00Z"}`}),
Now: func() time.Time { return at.Add(time.Hour) }}
out, err := r.Stuck("")
if err != nil {
t.Fatal(err)
}
// Issue 734, review round 1: a builder that does not hear DEAD_LETTERS says so, with why.
if dl := out["dead_letters"].(map[string]any); dl["heard"] != false || !strings.Contains(dl["why"].(string), "permissions violation") {
t.Errorf("dead_letters = %v, want not heard, saying why", dl)
}
stuck := out["stuck"].([]Stuck)
if len(stuck) != 1 || stuck[0].Key != key || stuck[0].Age != "1h0m0s" || stuck[0].NoneStep != 1 || stuck[0].ReAsk == nil || stuck[0].ReAsk.Why != "stuck" {
t.Fatalf("stuck = %+v, want the building key alone, an hour old, one ask with no step, with its ask again", stuck)
@@ -106,7 +106,9 @@ type StuckReader struct {
ReAsks StateRead
// Steps are the build modules' steps states, by build module.
Steps map[string]StateRead
Now func() time.Time
// Hearing is the builder's state hearing: whether it hears DEAD_LETTERS (issue 734); without it none is said.
Hearing StateRead
Now func() time.Time
// Logf, when set, gets the one line a call logs: the values it read and the time it took.
Logf func(format string, args ...any)
}
@@ -149,6 +151,20 @@ func parallel(n int, read func(i int) error) error {
}
// Stuck is every key building or queued with its asks, oldest first; with module, that module's only.
// deadLetters says whether the builder hears the asks the build seats give up on (issue 734), and why not: deaf, a lane
// whose ask is given up waits for its check bound (buildrun.HearingHealth over the kept hearing).
func (r StuckReader) deadLetters() map[string]any {
if r.Hearing.Get == nil {
return map[string]any{"heard": false, "why": "the builder's state hearing is not readable here"}
}
raw, _, err := r.Hearing.Get(buildrun.HearingKey)
if err != nil {
return map[string]any{"heard": false, "why": "the builder's state hearing cannot be read: " + err.Error()}
}
a := buildrun.HearingHealth(raw)
return map[string]any{"heard": a.Healthy, "why": a.Why}
}
func (r StuckReader) Stuck(module string) (map[string]any, error) {
started := time.Now()
reads := 0
@@ -268,6 +284,7 @@ func (r StuckReader) Stuck(module string) (map[string]any, error) {
})
answer := map[string]any{"stuck": out, "count": len(out),
"delivery_counts": "not readable by the builder: the build seats' workers' information is granted to the delivery view alone (its backlog); an ask given up on is told by the builder at its check bound and asked again"}
answer["dead_letters"] = r.deadLetters()
if cut {
answer["cut"], answer["note"] = true, fmt.Sprintf("read %d of %d build keys in key order and stopped; give a module to read its keys alone", MaxReads, total)
}
@@ -87,7 +87,7 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if m.Module != "mesh-delivery-builder" {
t.Errorf("module = %q", m.Module)
}
if want := []state{{Name: "builds", History: 5}, {Name: "instances", History: 1}, {Name: "queues", History: 1}, {Name: "reasks", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
if want := []state{{Name: "builds", History: 5}, {Name: "hearing", History: 1}, {Name: "instances", History: 1}, {Name: "queues", History: 1}, {Name: "reasks", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
t.Errorf("state = %+v, want %+v", m.State, want)
}
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
@@ -20,6 +20,7 @@
],
"state": [
{ "name": "builds", "history": 5 },
{ "name": "hearing", "history": 1 },
{ "name": "instances", "history": 1 },
{ "name": "queues", "history": 1 },
{ "name": "reasks", "history": 1 }
@@ -12,6 +12,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-checker/internal/check"
)
@@ -53,10 +54,36 @@ func read(s stdio.KeptState) check.StateRead {
}}
}
// runsOf says which commit the mesh runs of a module from the indexer's modules state, as the checker's own record of it does
// (the commit the module's record was computed at); "" when the record is not readable.
type runsOf struct{ modules check.StateRead }
func (r runsOf) CommitOf(module string) string {
if r.modules.Get == nil {
return ""
}
raw, err := r.modules.Get(module)
if err != nil || len(raw) == 0 {
return ""
}
m, err := delivery.DecodeModule(raw)
if err != nil {
return ""
}
return m.Commit
}
// stateStore is a state the tools write: the tries.
type stateStore struct {
check.StateRead
create func(key string, value []byte) error
entry func(key string) (json.RawMessage, uint64, error)
update func(key string, value []byte, revision uint64) error
}
func (s stateStore) Entry(key string) (json.RawMessage, uint64, error) { return s.entry(key) }
func (s stateStore) Update(key string, value []byte, revision uint64) error {
return s.update(key, value, revision)
}
func (s stateStore) Keys() ([]string, error) { return s.StateRead.Keys() }
@@ -66,11 +93,12 @@ func (s stateStore) Create(key string, value []byte) error { return s.create(k
// Trying is what the try's tools work on: the tries, and the gate's trunk heads.
type Trying struct {
Tries check.TryStore
Ender check.TryEnder
Trunks func() (map[string]string, error)
Now func() time.Time
}
// TryTools are the try's two tools (ADR 0501 §5): ask one, read one. mesh-controller's try calls them.
// TryTools are the try's tools (ADR 0501 §5): ask one, read one, and end one by hand (issue 734). mesh-controller's try calls them.
func TryTools(tr Trying) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return s }
return []stdio.Tool{
@@ -78,7 +106,8 @@ func TryTools(tr Trying) []stdio.Tool {
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused.",
"the checker checks no level of (one whose trunk the gate does not name) is refused. A try whose ask no build node took within the step " +
"bound ends as error; its ask stays in the build seat's stream and may still run later, judged by nobody (issue 734).",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"head": map[string]any{"type": "string", "description": "the full commit to try, 40 or 64 hex digits"},
@@ -115,6 +144,28 @@ func TryTools(tr Trying) []stdio.Tool {
}
return check.ReadTry(tr.Tries, str(args, "id"))
}},
{Name: check.TryEndTool,
Description: "End a try that waits or runs, as error, by hand: say why (kept in what it says). Its place among the 4 is free at once, " +
"the checker stops waiting for its asks, and the next " + check.TryTool + " of its head asks anew with a new id. For a try whose ask " +
"was lost (no step runs for it in checker_status); the checker ends one by itself when the build seat gives its ask up, or no node " +
"takes it within the step bound. A try that ended already is refused.",
Input: map[string]any{"type": "object", "required": []string{"id", "why"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"},
"why": map[string]any{"type": "string", "description": "why it is ended, in words"}}},
Run: func(args map[string]any) (any, error) {
if tr.Ender == nil {
return nil, fmt.Errorf("the tries state cannot be written here")
}
now := time.Now
if tr.Now != nil {
now = tr.Now
}
t, err := check.EndTry(tr.Ender, str(args, "id"), str(args, "why"), now())
if err != nil {
return nil, err
}
return map[string]any{"id": t.ID, "state": t.State, "said": t.Said}, nil
}},
}
}
@@ -174,7 +225,10 @@ func Tools(r check.Reader) []stdio.Tool {
"and the candidates whose check is owed by kind (waiting_by_kind: new, nodes, and trunk, a re-judge pulled after the other two). A key a " +
"build module left running (started past the step bound, or its build module not running on its node) is listed apart under " +
"left_running with why, and not counted. Nothing holds the checker and it has no capacity of its own: a check is an ask on the " +
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). Read-only.",
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). held_by_turn names the candidates whose check an approved " +
"merge's turn holds off (novox/hq ADR 0548 §3), with the repository and the merge that waits; the gate's turns tool reads every turn and its " +
"hand-now tool steers one. dead_letters says whether the checker hears the asks the build seats give up on (issue 734), and why not: " +
"deaf, a try or check whose ask is given up waits until its bound. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return r.Status() }},
}
@@ -193,7 +247,9 @@ func main() {
r := check.Reader{VerdictState: read(stdio.State("verdicts")), KeptState: read(stdio.State("kept")), CandidateState: read(stdio.State("mesh-delivery-indexer.candidates")),
PreviewState: read(stdio.State("previews")),
StepStates: map[string]check.StateRead{"mesh-build-go": read(stdio.State("mesh-build-go.steps")), "mesh-build-node": read(stdio.State("mesh-build-node.steps"))},
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound()}
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound(),
TurnState: read(stdio.State("mesh-delivery-gate.turns")), HearingState: read(stdio.State(buildrun.HearingState))}
r.Runs = runsOf{read(stdio.State("mesh-delivery-indexer.modules"))} // the node-engine at the commit the mesh runs, as the checker reads it
tries := stdio.State(check.TriesState)
source := stdio.State("mesh-delivery-gate.source")
trunks := func() (map[string]string, error) {
@@ -205,13 +261,21 @@ func main() {
return s.Heads, err
}
r.Trunks = trunks
trying := Trying{
Tries: stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}},
Trunks: trunks,
}
store := stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}, entry: func(key string) (json.RawMessage, uint64, error) {
reads.Add(1)
e, err := tries.Get(key)
if err != nil || e == nil {
return nil, 0, err
}
return e.Value, e.Revision, nil
}, update: func(key string, value []byte, revision uint64) error {
_, err := tries.Update(key, json.RawMessage(value), revision)
return err
}}
trying := Trying{Tries: store, Ender: store, Trunks: trunks}
if err := stdio.Serve("", logged(append(Tools(r), TryTools(trying)...))); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1)
@@ -53,13 +53,19 @@ func (m *memStore) Create(key string, value []byte) error {
return nil
}
func (m *memStore) Entry(key string) (json.RawMessage, uint64, error) { return m.vals[key], 1, nil }
func (m *memStore) Update(key string, value []byte, revision uint64) error {
m.vals[key] = value
return nil
}
func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed interface with mesh-controller's try
store := &memStore{vals: map[string]json.RawMessage{}}
tools := TryTools(Trying{Tries: store, Now: func() time.Time { return time.Unix(1, 5) },
tools := TryTools(Trying{Tries: store, Ender: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(tools) != 2 || tools[0].Name != "checker_try" || tools[1].Name != "checker_try_read" {
if len(tools) != 3 || tools[0].Name != "checker_try" || tools[1].Name != "checker_try_read" || tools[2].Name != "checker_try_end" {
t.Fatalf("tools = %+v", tools)
}
out, err := tools[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
@@ -74,6 +80,14 @@ func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed i
if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) {
t.Errorf("read = %s, %v", raw, err)
}
// Issue 734: a try whose ask was lost is ended by hand with why, and its head is asked anew.
ended, err := tools[2].Run(map[string]any{"id": id, "why": "its ask was lost"})
if err != nil || ended.(map[string]any)["state"] != "error" {
t.Errorf("end = %v, %v", ended, err)
}
if _, err := tools[2].Run(map[string]any{"id": id, "why": "again"}); err == nil {
t.Error("an ended try was ended again")
}
if _, err := tools[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil ||
err.Error() != "no level of novox/hq is checked by the checker" {
t.Errorf("hq: %v", err)
@@ -24,6 +24,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-checker/internal/check"
)
@@ -79,6 +80,8 @@ type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
conflict atomic.Bool
// hold is a commit whose fetch never answers: its ask stays taken, as one lost in a bus reload (issue 734).
hold atomic.Value
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
testTool *testing.T
@@ -98,6 +101,9 @@ func (f *fakeBuildModules) result(t *testing.T, tool string, args map[string]any
var v any
switch {
case strings.HasSuffix(tool, "_fetch"):
if h, _ := f.hold.Load().(string); h != "" && args["commit"] == h {
<-f.ctx.Done()
}
name := "go-fetch-clean"
if f.conflict.Load() {
name = "go-fetch-conflict"
@@ -298,6 +304,10 @@ func startBed(t *testing.T) *bed {
}
b.tools[module] = kv
}
// What the controller keeps of the messages a consumer gave up on (issue 734).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buildrun.DeadLettersStream, Subjects: []string{"mesh.events.dead.>"}}); err != nil {
t.Fatal(err)
}
// The checker's stream of events, which the indexer and the forge bridge hear.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "CHECKS", Subjects: []string{"mesh.stream.mesh-delivery-checker.event.>"}}); err != nil {
t.Fatal(err)
@@ -305,7 +315,8 @@ func startBed(t *testing.T) *bed {
for name, history := range map[string]int{delivery.CheckerVerdictsBucket: delivery.CheckerVerdictsHistory, delivery.CheckerKeptBucket: delivery.CheckerKeptHistory,
delivery.CheckerOwedBucket: delivery.CheckerOwedHistory, delivery.PreviewsBucket: delivery.PreviewsHistory, delivery.CandidatesBucket: delivery.CandidatesHistory,
delivery.ModulesBucket: delivery.ModulesHistory, delivery.ManifestsBucket: delivery.ManifestsHistory, delivery.SourceBucket: delivery.SourceHistory,
delivery.PresenceBucket: delivery.PresenceHistory, delivery.HaveBucket: delivery.HaveHistory, check.TriesBucket: 1} {
delivery.PresenceBucket: delivery.PresenceHistory, delivery.HaveBucket: delivery.HaveHistory, check.TriesBucket: 1,
delivery.TurnsBucket: delivery.TurnsHistory} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: uint8(history)})
if err != nil {
t.Fatal(err)
@@ -507,9 +518,12 @@ func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300
}
}
// TestACheckOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every package
// taken from the first's kept verdicts: no test step runs and no second kept key appears.
func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2
// TestACheckOrACarryOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every
// package taken from the first's kept verdicts: no test step runs and no second kept key appears. Then, through the real bus
// (novox/hq ADR 0548 §3, §5): a third candidate is not claimed while the gate's merge turn on its repository stands, and is
// claimed when the turn's key is deleted; and a move of the trunk carries the first verdict on the recheck stream by its
// unchanged digests, running no test, the verdict naming the trunk it was carried from.
func TestACheckOrACarryOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2, ADR 0548 §3, §5
b := startBed(t)
first := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
@@ -526,6 +540,42 @@ func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1
if got := keptCount(t, b); got != kept {
t.Errorf("%d kept keys, was %d", got, kept)
}
// §3: the gate's turn on the repository holds a new check off; its delete claims it.
at := time.Now().UTC()
turn, err := delivery.EncodeTurn(delivery.Turn{Repository: repo, Waiting: []delivery.TurnWait{{Group: "fix/other", Heads: "0123456789ab", Approved: at, Since: at}}})
if err != nil {
t.Fatal(err)
}
tk, _ := delivery.TurnsKey(repo)
if _, err := b.buckets[delivery.TurnsBucket].Put(b.ctx, tk, turn); err != nil {
t.Fatal(err)
}
time.Sleep(300 * time.Millisecond) // the checker hears the turn
third := b.candidate(t, "feat/z", pull(repo, 9, "3333333333333333333333333333333333333333"))
time.Sleep(time.Second)
if _, err := b.buckets[delivery.CheckerVerdictsBucket].Get(b.ctx, third); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("verdicts.%s exists (%v) while a merge turn holds its repository", third, err)
}
if err := b.buckets[delivery.TurnsBucket].Delete(b.ctx, tk); err != nil {
t.Fatal(err)
}
b.verdict(t, third, ended)
// §5: the trunk moves; the first verdict is carried by its digests, nothing runs.
tests = b.fake.count("mesh-build-go.build_go_test")
const moved = "abababababababababababababababababababab"
b.setTrunk(t, map[string]string{repo: moved})
c := b.verdict(t, first, func(v delivery.Verdict) bool { return ended(v) && v.Trunk[repo] == moved })
if c.State != delivery.VerdictPass || c.CarriedFrom[repo] != trunk || c.Levels[delivery.LevelUnit].Run != 0 || c.Levels[delivery.LevelIntegration].Run != 0 {
t.Fatalf("the carried verdict = %+v, want a pass carried from %s with no test run", c, trunk[:12])
}
if got := b.fake.count("mesh-build-go.build_go_test"); got != tests {
t.Errorf("build_go_test called %d times on the carry, was %d: a carry of unchanged digests runs nothing", got, tests)
}
if o := b.owed(t, 7, head1, "success"); !strings.HasPrefix(o.Text, "pass 111111111111 on abababababab | carried from aaaaaaaaaaaa") {
t.Errorf("the carried status says %q", o.Text)
}
}
func keptCount(t *testing.T, b *bed) int {
@@ -676,6 +726,49 @@ func TestATryIsJudgedAndItsEndWrittenWithNoStatusOwed(t *testing.T) {
t.Errorf("a try wrote %s: %v", name, keys)
}
}
// Issue 734: a try whose ask the build seat gives up on (the controller keeps its copy in DEAD_LETTERS with the ask's
// message id) ends as error saying so, and its head asked again is a new try.
b.fake.hold.Store(head2)
lost, err := check.AskTry(store, map[string]string{repo: trunk}, check.TryAsk{Repository: repo, Head: head2}, time.Now())
if err != nil {
t.Fatal(err)
}
steps, err := b.js.KeyValue(b.ctx, delivery.StepsBucket("mesh-build-go"))
if err != nil {
t.Fatal(err)
}
askID := ""
for deadline := time.Now().Add(20 * time.Second); askID == "" && time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
keys, _ := steps.Keys(b.ctx)
for _, k := range keys {
if e, err := steps.Get(b.ctx, k); err == nil && strings.Contains(string(e.Value()), `"running"`) {
askID = strings.TrimPrefix(k, "n1.")
}
}
}
if askID == "" {
t.Fatal("the held try's ask was never taken")
}
dead := &nats.Msg{Subject: "mesh.events.dead.SEAT_NODE_BUILD_GO.SEAT_NODE_BUILD_GO_CHECK_worker", Header: nats.Header{}}
dead.Header.Set(buildrun.DeadHeaderMsgID, askID)
dead.Header.Set(buildrun.DeadHeaderDeliveries, "5")
dead.Header.Set(buildrun.DeadHeaderStream, "SEAT_NODE_BUILD_GO")
dead.Header.Set(buildrun.DeadHeaderGaveUp, time.Now().UTC().Format(time.RFC3339Nano))
if _, err := b.js.PublishMsg(b.ctx, dead); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
raw, _ := store.Get(lost)
if last, err = check.DecodeTry(raw); err == nil && last.Over() {
break
}
}
if last.State != check.TryError || !strings.Contains(last.Said, buildrun.LostWords+" after 5 deliveries") {
t.Fatalf("the try whose ask was given up = %+v, want error saying so", last)
}
if again, err := check.AskTry(store, map[string]string{repo: trunk}, check.TryAsk{Repository: repo, Head: head2}, time.Now()); err != nil || again == lost {
t.Errorf("its head asked again answered %q, %v, want a new try", again, err)
}
}
// TestTheCheckerSaysOwedReadableOnTheBus: started, the checker says on its own states subject what it keeps, owed among
@@ -702,7 +795,7 @@ func TestTheCheckerSaysOwedReadableOnTheBus(t *testing.T) { // R2-1
break
}
if said.Module != "mesh-delivery-checker" || said.Node != "n1" || !said.Keeps("owed") || !said.Readable("owed") ||
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || said.Readable("tries") || len(said.States) != 5 {
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || said.Readable("tries") || said.Readable("hearing") || len(said.States) != 6 {
t.Fatalf("the checker said %+v, want its five states with kept, owed, previews and verdicts readable (1929/AC4; 1887 A5b)", said)
}
}
@@ -0,0 +1,50 @@
package main
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// lazyTurns is the gate's merge turns as the checker reads them (novox/hq ADR 0548 §3), opened when the gate's bucket is
// there: before each claim it is opened again until it is, so a checker that started before the gate made the bucket holds
// checks from the gate's first turn on, with no timer and no restart. Until then it holds nothing, and says so once, loudly,
// and once again when it opens.
type lazyTurns struct {
turns *check.Turns
open func() error
logf func(format string, args ...any)
mu sync.Mutex
opened bool
said bool
}
// ensure opens the bucket once it is there; a failure is said once.
func (l *lazyTurns) ensure() {
l.mu.Lock()
defer l.mu.Unlock()
if l.opened || l.open == nil {
return
}
if err := l.open(); err != nil {
if !l.said {
l.said = true
l.logf("NO MERGE TURN HOLDS A CHECK: the gate's bucket %s cannot be read (%v); every check is claimed as if no approved merge waited, "+
"and the bucket is opened again before each claim until it is there", delivery.TurnsBucket, err)
}
return
}
l.opened = true
if l.said {
l.logf("the gate's merge turns are read from %s from now on: an approved merge's turn holds checks off its repositories again", delivery.TurnsBucket)
}
}
// Holding opens the bucket when it is not yet, then answers what the turns hold.
func (l *lazyTurns) Holding(repositories []string, heads string) (string, delivery.TurnWait, bool) {
l.ensure()
return l.turns.Holding(repositories, heads)
}
@@ -116,6 +116,8 @@ func durationEnv(getenv func(string) string, name string, def time.Duration) (ti
var keptStates = []announce.State{
// The kept verdicts are read by the Go build module, which takes them for a check ask's unchanged packages (issue 1887's
// design limit 1); the checker alone writes them (ADR 0444 clause 5).
// Whether DEAD_LETTERS is heard (issue 734): read by the checker's own status tool alone.
{Name: "hearing"},
{Name: "kept", Readable: true},
{Name: "owed", Readable: true},
{Name: "previews", Readable: true},
@@ -193,13 +195,26 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
tools := &buildrun.ToolchainSet{}
presence := &buildrun.PresenceSet{}
trunks := &check.Trunks{}
turns := &check.Turns{}
lazy := &lazyTurns{turns: turns, logf: logger.Printf}
// A check is one ask per pull request and build module on the build seats' check stream, pulled by a build node with a
// free slot (novox/hq ADR 0525 §1, §3): the checker chooses no node and starts no step.
asks := buildrun.JetStreamAsks{JS: js}
// The asks the build seats gave up on (issue 734): heard from DEAD_LETTERS, each ends the step its asker waits for.
// Heard in the background, and again at a reconnect or an ask while it is not; whether it is heard is kept in the
// state hearing, which checker_status says (dead_letters).
dead := &buildrun.DeadLetters{}
hearingKV := process.OptionalBucket(ctx, js, check.HearingBucket, "checker_status cannot say whether DEAD_LETTERS is heard", logger.Printf)
hearing := buildrun.NewDeadLetterHearing(ctx, buildrun.JetStreamDeadLetters{JS: js}, []string{delivery.AskGo, delivery.AskNode}, dead,
logger.Printf, buildrun.SayHearing(ctx, hearingKV, logger.Printf))
defer hearing.Stop()
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again() })
waiter := buildrun.LostAskWaiter{Inner: asks, Dead: dead, Hearing: hearing}
ch := &check.Checker{
Verdicts: kvStore{buckets[delivery.CheckerVerdictsBucket]},
Owed: kvStore{buckets[delivery.CheckerOwedBucket]},
Trunks: trunks,
Turns: lazy,
Bound: checkBound,
Link: link,
Logf: logger.Printf,
@@ -228,7 +243,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return cw, nil
}}
// The tries (ADR 0501 §5) are judged by the same judge, held by the same deliveries.
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, Logf: logger.Printf}
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, NotTaken: stepBound, Logf: logger.Printf}
announce := func(ctx context.Context, event, key string) {
body, _ := json.Marshal(map[string]string{"key": key})
if _, err := js.Publish(ctx, eventSubject(event), body); err != nil {
@@ -236,8 +251,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
}
repos := &repoModules{manifests: kvStore{buckets[delivery.ManifestsBucket]}, logf: logger.Printf}
ch.Runs = repos // the node-engine a check reads beside at its running commit (ADR 0548 §6)
ch.Judge = newJudge(judgeDeps{
asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: stepBound}, tools: tools, trunks: trunks,
asks: buildrun.Asker{Publisher: asks, Waiter: waiter, Bound: stepBound}, tools: tools, trunks: trunks,
kept: kvStore{buckets[delivery.CheckerKeptBucket]}, previews: kvStore{buckets[delivery.PreviewsBucket]},
repos: repos, builds: builds, forge: forgeReads{call: caller.Call}, logf: logger.Printf,
announce: announce, compositions: compositions,
@@ -313,13 +329,42 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}()
}
}
// The gate's merge turns (novox/hq ADR 0548 §3): no check is claimed of a repository an approved merge waits on, and the
// delete of its turn claims it. Read before the candidates, so no check starts past a turn written before this start.
// Where the gate's bucket is not there yet it is read as empty, and opened again before each claim until it is there
// (lazyTurns), so a gate delivered after the checker started holds checks from its first turn on.
lazy.open = func() error {
kv, err := js.KeyValue(ctx, delivery.TurnsBucket)
if err != nil {
return err
}
tw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { turns.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, tw, func(e jetstream.KeyValueEntry) {
if turns.Put(e.Key(), e.Value(), !opOf(e)) {
ch.Reconsider(ctx) // a turn went or changed: a check it held is claimed now
}
})
}()
return nil
}
lazy.ensure()
// The modules say which repository holds a TypeScript module, so the Node build module judges it too.
mw, err := settle(ctx, buckets[delivery.ModulesBucket], func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, mw, func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
errs <- follow(ctx, mw, func(e jetstream.KeyValueEntry) {
before := repos.CommitOf(check.NodeEngineModule)
repos.Put(ctx, e.Key(), e.Value(), !opOf(e))
if after := repos.CommitOf(check.NodeEngineModule); after != before {
ch.TrunkMoved(ctx) // the node-engine the mesh runs moved: a verdict that read it beside is carried (ADR 0548 §6)
}
})
}()
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026095702-79649c3fa9f4
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026095702-79649c3fa9f4 h1:05sVel/UOvz4/g0dxrDOeVUL7BGakVrxPBlqRq02ZhM=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026095702-79649c3fa9f4/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530 h1:NwCb+gOh0UYqlyG6hG2/jffE/Gd0c7YWd4iJlaCB0lU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,355 @@
package check
import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// One check per delivery (novox/hq ADR 0548): the checker's part. A merge turn holds new checks off its repository (§3);
// a trunk or sibling move never ends a running check and carries its verdict once it ended (§1, §5, §6); only a head git
// refused gets a whole check after a move (§7); the line of a carried verdict says so (§8).
// turnOf is a turn's value on a repository for a delivery of the heads hash.
func turnOf(t testing.TB, repo, group, heads string) []byte {
t.Helper()
at := time.Date(2026, 10, 26, 9, 0, 0, 0, time.UTC)
data, err := delivery.EncodeTurn(delivery.Turn{Repository: repo, Waiting: []delivery.TurnWait{{Group: group, Heads: heads, Approved: at, Since: at}}})
if err != nil {
t.Fatal(err)
}
return data
}
func turnKey(t testing.TB, repo string) string {
t.Helper()
k, err := delivery.TurnsKey(repo)
if err != nil {
t.Fatal(err)
}
return k
}
// waitFor polls a condition the checker's goroutines make true, failing after two seconds.
func waitFor(t testing.TB, what string, ok func() bool) {
t.Helper()
for deadline := time.Now().Add(2 * time.Second); !ok(); time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("%s did not happen", what)
}
}
}
// pullWithFP is a pull request touching the module m at the fingerprint `src2:` and 64 times the hex digit d.
func pullWithFP(repo string, number int, head string, d string) delivery.CandidatePull {
p := pullOf(repo, number, head)
p.Modules = []string{"m"}
p.Fps = map[string]delivery.CandidateModule{"m": {FP2: "src2:" + strings.Repeat(d, 64)}}
return p
}
func TestOneCheckPerDelivery(t *testing.T) {
ctx := context.Background()
for name, run := range map[string]func(t *testing.T){
// §3: no claim while a turn names one of its repositories; a claim on the turn's delete event.
"a turn on its repository holds a new check off until the turn goes": func(t *testing.T) {
j := passJudge()
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "fix/other", "0123456789ab"), false)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 0 || len(r.verdicts.vals) != 0 {
t.Fatalf("judged %d times with a turn on %s, verdicts %v", j.count(), repoA, r.verdicts.vals)
}
if held := r.ch.Held()[keyOf(t, "feat/x", cand)]; !strings.Contains(held, repoA) || !strings.Contains(held, "fix/other") {
t.Errorf("the hold is said as %q, want the turn's repository and group", held)
}
if !turns.Put(turnKey(t, repoA), nil, true) {
t.Fatal("the turn's delete changed nothing")
}
r.ch.Reconsider(ctx) // what the watch of turns does on the delete
r.ch.Wait()
if j.count() != 1 || r.verdict(t, keyOf(t, "feat/x", cand)).State != delivery.VerdictPass {
t.Errorf("judged %d times after the turn went, want once", j.count())
}
},
"a turn on another repository, or of its own heads, holds nothing": func(t *testing.T) {
j := passJudge()
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
c, _ := delivery.DecodeCandidate(cand)
turns.Put(turnKey(t, repoSDK), turnOf(t, repoSDK, "fix/other", "0123456789ab"), false)
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "feat/x", GateHeads(c)), false)
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 1 {
t.Errorf("judged %d times, want once: the turns hold another repository and its own merge", j.count())
}
},
// §3, the race: a check claimed before the gate's turn is judged on the commit it read, never restarted.
"a check claimed before a turn was written runs to its verdict": func(t *testing.T) {
release, started := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) {
close(started)
<-release
return Outcome{Verdict: passing()}, nil
}}
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
<-started
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "fix/other", "0123456789ab"), false)
r.ch.Reconsider(ctx)
close(release)
r.ch.Wait()
if j.count() != 1 || r.verdict(t, keyOf(t, "feat/x", cand)).State != delivery.VerdictPass {
t.Errorf("judged %d times; the running check is not ended by the turn", j.count())
}
},
// §1, §5: a trunk move during a running check calls the judge once, writes its verdict, keeps the status pending (a pass
// on an old trunk is never shown), then asks one carry, on the recheck stream, and no check.
"a trunk move during a check ends nothing and asks one carry after it": func(t *testing.T) {
var r *rig
var kinds []string
var from []map[string]string
judged := make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
kinds, from = append(kinds, c.Kind), append(from, c.CarriedFrom)
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
r.ch.TrunkMoved(ctx) // as the watch of the source says it
close(judged)
} else if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status during the carry is %q, want pending: the pass on the old trunk is never shown", s)
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
<-judged
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || kinds[0] != KindNew || kinds[1] != KindTrunk || from[0] != nil || from[1][repoA] != trunkA {
t.Fatalf("judged %d times as %v carried from %v; want a whole check, then one carry from %s", j.count(), kinds, from, trunkA[:12])
}
if v.State != delivery.VerdictPass || v.Trunk[repoA] != trunkA2 || v.CarriedFrom[repoA] != trunkA {
t.Errorf("the verdict is %s on %v carried from %v; want a pass on the new trunk carried from the old", v.State, v.Trunk, v.CarriedFrom)
}
if s := owedState(t, r, repoA, 7, head1); s != "success" {
t.Errorf("the status after the carry is %q, want success", s)
}
if r.verdicts.creates[keyOf(t, "feat/x", cand)] != 1 {
t.Error("the verdict key was made again: a carry rewrites it")
}
},
// §5: the composition is kept when no fingerprint of a module it composed changed, and runs again otherwise.
"a carry keeps a passed composition while the fingerprints stay the same": func(t *testing.T) {
var kept []bool
j := &judgeFunc{f: func(_ context.Context, c *Case, _ int) (Outcome, error) {
kept = append(kept, c.Composition != nil)
v := passing()
v.Composition = &delivery.Composition{State: delivery.CompositionPass, SDK: "v0.1.14-x", FPSet: c.FPSet}
return Outcome{Verdict: v}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullWithFP(repoA, 7, head1, "a"))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx) // the indexer's fingerprints of the candidate stay the same
r.ch.Wait()
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullWithFP(repoA, 7, head1, "b"))) // a fingerprint changed with the move
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if len(kept) != 3 || kept[0] || !kept[1] || kept[2] {
t.Errorf("the composition was kept in the checks %v; want composed, kept on the carry, composed again once a fingerprint changed", kept)
}
},
// §6: a sibling's move puts the verdict out of date, its status pending saying it is carried, and the carry's verdict
// names the sibling's old commit; the status follows the carry's verdict. Which packages run is the digest's (only
// those that name MESH_CHECK_BESIDE change with a sibling), the build module's part.
"a sibling's move is carried and the status follows the carry": func(t *testing.T) {
var r *rig
var from map[string]string
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
if call == 2 {
from = c.CarriedFrom
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status during the sibling's carry is %q, want pending", s)
}
}
return Outcome{Verdict: passing(), Beside: map[string]string{catalogueRepository: r.trunks.Heads()[catalogueRepository]}}, nil
}}
r = newRig(t, j)
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, catalogueRepository: trunkS})
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if b := r.verdict(t, keyOf(t, "feat/x", cand)).Beside; b[catalogueRepository] != trunkS {
t.Fatalf("the verdict names the siblings %v, want the catalogue at %s", b, trunkS[:12])
}
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, catalogueRepository: trunkS2})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || from[catalogueRepository] != trunkS || v.CarriedFrom[catalogueRepository] != trunkS || v.Beside[catalogueRepository] != trunkS2 {
t.Errorf("judged %d times, carried from %v, verdict beside %v carried from %v; want one carry over the catalogue's move", j.count(), from, v.Beside, v.CarriedFrom)
}
if s := owedState(t, r, repoA, 7, head1); s != "success" {
t.Errorf("the status after the carry is %q, want success", s)
}
},
// §7: a carry whose merge conflicts fails naming the files and asks no check; a later move onto which the head merges
// cleanly asks a whole check, not a carry.
"a carry git refuses fails naming the files, and the next move asks a whole check": func(t *testing.T) {
var kinds []string
var from []map[string]string
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
kinds, from = append(kinds, c.Kind), append(from, c.CarriedFrom)
if call == 2 {
return Outcome{Verdict: delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: "ace",
Words: repoA + "#7 " + ConflictWords + "go.mod"}}}, nil
}
return Outcome{Verdict: passing()}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
r.ch.Reconsider(ctx) // anything else that looks again asks nothing of a conflict
r.ch.Wait()
if v := r.verdict(t, keyOf(t, "feat/x", cand)); j.count() != 2 || !IsConflictFailure(v) || !strings.Contains(v.Failed.Words, "go.mod") {
t.Fatalf("judged %d times, verdict %+v; want the carry failed naming go.mod and nothing asked after it", j.count(), v.Failed)
}
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if j.count() != 3 || kinds[1] != KindTrunk || from[1] == nil || kinds[2] != KindNew || from[2] != nil {
t.Errorf("checks %v carried from %v; want a whole check, a carry, then a whole check after the conflict", kinds, from)
}
},
// §8, §6: the line of a carried verdict names the head on the new trunk commit first, as the gate and the merge tool
// read a line, then where it was carried from and what the carry ran and kept; while it is carried it is pending and says so.
"the line of a carry says where it was carried from": func(t *testing.T) {
cs, p := lineCase(t, pullOf(repoA, 7, head1))
v := passVerdict()
v.Trunk = map[string]string{repoA: trunkA}
v.Beside = map[string]string{catalogueRepository: trunkS2}
v.CarriedFrom = map[string]string{repoA: trunkA2, catalogueRepository: trunkS}
line := Line(v, cs, p, nil, nil, nil)
if !strings.HasPrefix(line, "pass 111111111111 on aaaaaaaaaaaa | carried from abababababab, mesh-catalog 555555555555 | unit 12 run") {
t.Errorf("the carried line = %q", line)
}
if head, trunk := checkWordsOf(line); head != head1[:12] || trunk != trunkA[:12] {
t.Errorf("the line names %s on %s first, want the head on the new trunk", head, trunk)
}
cs.CarriedFrom = map[string]string{repoA: trunkA2}
pending := Line(delivery.Verdict{State: delivery.VerdictRunning}, cs, p, nil, nil, nil)
if !strings.HasPrefix(pending, "being carried to 111111111111 on aaaaaaaaaaaa | from abababababab") {
t.Errorf("the pending line of a carry = %q", pending)
}
},
// §3 (review round 1): two approved merges in one turn whose trunk moved each get their carry; neither holds the other's.
"two merges waiting in one turn each get their carry": func(t *testing.T) {
var r *rig
j := &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) { return Outcome{Verdict: passing()}, nil }}
r = newRig(t, j)
candA, candC := candidateOf(t, "feat/a", pullOf(repoA, 7, head1)), candidateOf(t, "feat/c", pullOf(repoA, 8, head2))
r.ch.Candidate(ctx, "feat%2Fa", candA)
r.ch.Candidate(ctx, "feat%2Fc", candC)
r.ch.Wait()
a, _ := delivery.DecodeCandidate(candA)
c, _ := delivery.DecodeCandidate(candC)
at := time.Date(2026, 10, 26, 9, 0, 0, 0, time.UTC)
turn, err := delivery.EncodeTurn(delivery.Turn{Repository: repoA, Waiting: []delivery.TurnWait{
{Group: "feat/a", Heads: GateHeads(a), Approved: at, Since: at}, {Group: "feat/c", Heads: GateHeads(c), Approved: at.Add(time.Minute), Since: at}}})
if err != nil {
t.Fatal(err)
}
turns := &Turns{}
turns.Put(turnKey(t, repoA), turn, false)
r.ch.Turns = turns
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // a third merge moves the trunk under both
r.ch.TrunkMoved(ctx)
r.ch.Wait()
for _, cand := range [][]byte{candA, candC} {
group := map[bool]string{true: "feat/a", false: "feat/c"}[string(cand) == string(candA)]
if v := r.verdict(t, keyOf(t, group, cand)); v.Trunk[repoA] != trunkA2 || v.CarriedFrom[repoA] != trunkA {
t.Errorf("%s: verdict on %v carried from %v; want its carry claimed and done", group, v.Trunk, v.CarriedFrom)
}
}
},
// §6 (review round 1): the node-engine read at the commit the mesh runs is named, and a new running commit is carried.
"a new running node-engine is carried": func(t *testing.T) {
runs := runsAt{nodeEngineModule: trunkS}
j := &judgeFunc{f: func(_ context.Context, c *Case, _ int) (Outcome, error) {
return Outcome{Verdict: passing(), Beside: map[string]string{nodeEngineRepo: runs[nodeEngineModule]}}, nil
}}
r := newRig(t, j)
r.ch.Runs = runs
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.ch.TrunkMoved(ctx) // nothing moved: nothing is carried
r.ch.Wait()
runs[nodeEngineModule] = trunkS2
r.ch.TrunkMoved(ctx) // what the indexer's watch does when mesh-host's running commit changes
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || v.CarriedFrom[nodeEngineRepo] != trunkS || v.Beside[nodeEngineRepo] != trunkS2 {
t.Errorf("judged %d times, verdict beside %v carried from %v; want one carry over the node-engine's new running commit", j.count(), v.Beside, v.CarriedFrom)
}
},
// §5: a carry that fails names the move it failed on.
"a carry that fails names the move": func(t *testing.T) {
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
return Outcome{Verdict: passing()}, nil
}
return Outcome{Verdict: delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Test: "unit: m/a.TestX", Words: "want 4"}}}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if v.State != delivery.VerdictFail || v.CarriedFrom != nil || !strings.HasPrefix(v.Failed.Words, "want 4") ||
!strings.Contains(v.Failed.Words, repoA+" "+trunkA[:12]+" to "+trunkA2[:12]) {
t.Errorf("the failed carry is %s carried from %v with words %q; want the test's words, then the move", v.State, v.CarriedFrom, v.Failed.Words)
}
},
} {
t.Run(name, run)
}
}
// checkWordsOf is the head and trunk a line names first, as the gate reads it (mesh-delivery-gate onestatus.go checkLine).
func checkWordsOf(line string) (string, string) {
for _, part := range strings.Split(line, " | ") {
f := strings.Fields(part)
for i := 0; i+2 < len(f); i++ {
if f[i+1] == "on" && len(f[i]) == 12 && len(f[i+2]) == 12 {
return f[i], f[i+2]
}
}
}
return "", ""
}
@@ -127,6 +127,9 @@ func (ch *Checker) Composed(ctx context.Context, comp delivery.CompositionVerdic
return
}
withComposition(&v, comp)
if v.Composition != nil && v.Composition.State != delivery.CompositionNotRun {
v.Composition.FPSet = cs.FPSet
}
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: the verdict with its composition cannot be written: %v", cs.Key, err)
@@ -12,8 +12,18 @@ type BesideRef struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Ref string `json:"ref,omitempty"`
// atTrunk says the sibling was read at its trunk's head, so a move of that trunk puts the verdict out of date and is
// carried (novox/hq ADR 0548 §6). Never sent: the node-engine at the commit the mesh runs, mesh-sdk at a pin and a
// group's member at its head do not move with a trunk.
atTrunk bool
// runs says the sibling was read at the commit the mesh runs (the node-engine, the indexer's record of mesh-host): a new
// running commit puts the verdict out of date and is carried (ADR 0548 §6). Never sent.
runs bool
}
// NodeEngineModule is the module whose running commit the node-engine sibling is read at.
const NodeEngineModule = nodeEngineModule
const (
catalogueRepository = "novox/mesh-catalog"
nodeEngineRepo = "novox/mesh-host"
@@ -51,18 +61,21 @@ func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef,
switch {
case repo == catalogueRepository:
if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
out = append(out, BesideRef{Repository: repo, Commit: head, atTrunk: c.member(repo) == nil})
}
case repo == nodeEngineRepo:
commit := ""
if j.Runs != nil {
commit = j.Runs.CommitOf(nodeEngineModule)
}
atTrunk := false
if commit == "" {
commit = heads[repo]
commit, atTrunk = heads[repo], true
}
if commit != "" {
out = append(out, BesideRef{Repository: repo, Commit: commit})
// Named in the verdict either way (ADR 0548 §6, issue 568's own example): at the commit the mesh runs, a new
// running commit puts the verdict out of date; at the trunk, a trunk move does (Checker.besideNow).
out = append(out, BesideRef{Repository: repo, Commit: commit, atTrunk: atTrunk, runs: !atTrunk})
}
case repo == SDKRepository && p.Repository == controllerRepo && j.Forge != nil:
text, err := j.Forge.File(ctx, p.Repository, "go.mod", p.Head)
@@ -82,7 +95,7 @@ func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef,
if member := c.member(repo); member != nil {
out = append(out, BesideRef{Repository: repo, Commit: member.Head, Ref: refOf(member.Number)})
} else if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
out = append(out, BesideRef{Repository: repo, Commit: head, atTrunk: true})
} else if repo == controllerRepo && p.Repository == nodeEngineRepo {
// The node-engine's step-nine test judges the controller's own manifest and fails without it (issue 1295): a
// checker that cannot place it says so, where placing none left that test to skip.
@@ -108,3 +121,13 @@ func refOf(number int) string {
}
return "refs/pull/" + strconv.Itoa(number) + "/head"
}
// besideAtTrunk are the siblings of refs read at their trunk's head that the case does not merge, by repository: what a
// verdict names as Beside (ADR 0548 §6).
func besideAtTrunk(c *Case, refs []BesideRef, into map[string]string) {
for _, r := range refs {
if _, merged := c.Trunk[r.Repository]; (r.atTrunk || r.runs) && !merged {
into[r.Repository] = r.Commit
}
}
}
@@ -46,9 +46,17 @@ type Case struct {
notes []string
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a re-judge after a trunk move is asked on the build
// seats' recheck stream, every other check on check (ADR 0536 §4).
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a carry after a trunk or sibling move is asked on the
// build seats' recheck stream, every other check on check (ADR 0536 §4, ADR 0548 §5).
Kind string
// CarriedFrom are, for the carry of a verdict that passed, the trunk and sibling commits it was judged on (ADR 0548 §5);
// nil for a whole check.
CarriedFrom map[string]string
// FPSet is the candidate's fingerprint set (Candidate.FPSet): a composition names the set it judged.
FPSet string
// Composition is, for a carry, the composition that passed on the same fingerprint set: it is kept and not asked again,
// since no fingerprint of a module it composed changed (ADR 0548 §5). Nil composes.
Composition *delivery.Composition
}
// Progress is the step a check is at, written by the check and read when its bound passes.
@@ -80,7 +88,7 @@ func CaseOf(c delivery.Candidate, trunk map[string]string) *Case {
if err != nil {
return nil
}
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}}
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}, FPSet: c.FPSet}
for _, p := range c.Pulls {
switch head, known := trunk[p.Repository]; {
case !known || head == "":
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
@@ -42,6 +43,9 @@ type Outcome struct {
// Judged says, by pull request (pullKey), which program judged a change to a build module (ADR 0535 §5); a pull request
// that changes none is not named.
Judged map[string]string
// Beside are the siblings the check read at their trunk's head, by repository (ADR 0548 §6): the verdict names them, and
// a move of one is carried.
Beside map[string]string
}
// BuildNote is the state of a candidate's module build, by module: built, failed or waiting (ADR 0484 clause 7).
@@ -65,6 +69,14 @@ type Checker struct {
Owed KV
Judge Judge
Trunks interface{ Heads() map[string]string }
// Runs says which commit the mesh runs of a module (the indexer's modules): the node-engine a verdict read beside at its
// running commit is out of date when that commit moves (ADR 0548 §6); nil reads every sibling at its trunk.
Runs interface{ CommitOf(module string) string }
// Turns are the gate's merge turns (ADR 0548 §3): no check is claimed of a repository a turn names, but the check of the
// waiting delivery's own heads; nil holds none.
Turns interface {
Holding(repositories []string, heads string) (string, delivery.TurnWait, bool)
}
// Bound is how long one check may run; none when 0 (DefaultBound).
Bound time.Duration
// Link is the base of the address a status points at; the verdict key follows.
@@ -83,12 +95,13 @@ type Checker struct {
tried map[string]bool // by verdict key: looked at since this process started
epoch uint64 // changes of the nodes seen since this process started
nodesFail map[string]uint64 // by verdict key: the epoch a failure of the nodes was written at
held map[string]string // by verdict key: the turn that holds its check off, as last said
}
func (ch *Checker) init() {
ch.once.Do(func() {
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail = map[string]uint64{}
ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{}
})
}
@@ -191,7 +204,7 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
// Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a
// failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9;
// ADR 0536 §4). The checker holds no place and no capacity of its own.
cs.Kind = ch.kindOf(ctx, cs)
ch.prepare(ctx, cs)
ch.try(ctx, key, cs)
ch.mu.Lock()
more := ch.again[cs.Key]
@@ -211,29 +224,130 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
}
// The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on
// the build seats' check stream, a head asked again only because its trunk moved on the recheck stream, pulled after.
// the build seats' check stream; a carry, a verdict moved to a newer trunk or sibling commit by its package digests (ADR 0548
// §5), on the recheck stream, pulled after.
const (
KindNew = "new"
KindNodes = "nodes"
KindTrunk = "trunk"
)
// kindOf is the kind of a case's check, read from its verdict key: none (or one left running) is a head never judged; a
// failure of the nodes is asked again after the nodes changed; a verdict on other trunk heads is a re-judge after a trunk
// move. A verdict that stands is ranked with the re-judges: its claim finds nothing to do.
func (ch *Checker) kindOf(ctx context.Context, cs *Case) string {
// prepare reads the case's verdict key and says what its check is (ADR 0548 §1, §5, §7): none (or one left running) is a
// head never judged, a whole check; a failure of the nodes is asked again after the nodes changed; a verdict judged on trunk
// or sibling commits that moved since is carried, its packages run only where the move changed their digest, and a pass's
// commits are named as the carry's CarriedFrom. A verdict that stands is ranked with the carries: its claim finds nothing
// to do.
func (ch *Checker) prepare(ctx context.Context, cs *Case) {
cs.Kind, cs.CarriedFrom, cs.Composition = KindNew, nil, nil
value, _, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return KindNew
return
}
v, err := delivery.DecodeVerdict(value)
switch {
case err != nil || v.State == delivery.VerdictRunning:
return KindNew
return
case IsNodesFailure(v):
return KindNodes
cs.Kind = KindNodes
return
case IsConflictFailure(v):
return // a head git refused: its next check is whole (ADR 0548 §7)
}
return KindTrunk
cs.Kind = KindTrunk
if v.State == delivery.VerdictPass && ch.stale(v, cs) {
cs.CarriedFrom = judgedOn(v)
if c := v.Composition; c != nil && c.State == delivery.CompositionPass && c.FPSet != "" && c.FPSet == cs.FPSet {
kept := *c
cs.Composition = &kept
}
}
}
// stale says whether a verdict was judged on commits that are no longer the case's: a trunk it merged moved, or a sibling it
// read at its trunk's head (ADR 0548 §6).
func (ch *Checker) stale(v delivery.Verdict, cs *Case) bool {
return !sameTrunk(v.Trunk, cs.Trunk) || besideMoved(v.Beside, ch.besideNow())
}
// besideNow are the commits a sibling is read at now, by repository: its trunk head, and for the node-engine the commit the
// mesh runs when the indexer names one, as besideFor places them.
func (ch *Checker) besideNow() map[string]string {
return besideAt(ch.heads(), ch.Runs)
}
// besideAt places the commits a sibling is read at: the trunk heads, with the node-engine at the commit the mesh runs when
// runs names one (ADR 0548 §6). The checker's decision and the status read the same commits through it.
func besideAt(now map[string]string, runs interface{ CommitOf(module string) string }) map[string]string {
if runs != nil {
if c := runs.CommitOf(nodeEngineModule); c != "" {
out := make(map[string]string, len(now)+1)
for r, h := range now {
out[r] = h
}
out[nodeEngineRepo] = c
return out
}
}
return now
}
// besideMoved says whether a sibling read at these commits moved: the commit it is read at now is known and another.
func besideMoved(beside, now map[string]string) bool {
for r, c := range beside {
if h := now[r]; h != "" && h != c {
return true
}
}
return false
}
// judgedOn are the trunk and sibling commits a verdict was judged on, by repository.
func judgedOn(v delivery.Verdict) map[string]string {
out := make(map[string]string, len(v.Trunk)+len(v.Beside))
for r, c := range v.Beside {
out[r] = c
}
for r, c := range v.Trunk {
out[r] = c
}
return out
}
// carried is what a carry's verdict names as carried from (ADR 0548 §5): the commits it was judged on before, of the
// repositories the new verdict still judges or reads; nil when none of them moved.
func carried(from map[string]string, v delivery.Verdict) map[string]string {
out, moved := map[string]string{}, false
for r, c := range from {
now, ok := v.Trunk[r]
if !ok {
now, ok = v.Beside[r]
}
if ok && now != "" && c != "" {
out[r] = c
moved = moved || now != c
}
}
if !moved {
return nil
}
return out
}
// moveWords say a carry's moves, for the words of a failure: "carried from <old12> to <new12> of <repository>", each moved
// repository, sorted.
func moveWords(from map[string]string, v delivery.Verdict) string {
var moves []string
for r, c := range from {
now, ok := v.Trunk[r]
if !ok {
now = v.Beside[r]
}
if now != "" && now != c {
moves = append(moves, fmt.Sprintf("%s %s to %s", r, short12(c), short12(now)))
}
}
sort.Strings(moves)
return strings.Join(moves, ", ")
}
func (ch *Checker) candidate(key string, fallback delivery.Candidate) delivery.Candidate {
@@ -265,6 +379,9 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return nil, fmt.Errorf("%s: %w", cs.Key, err)
}
if !found {
if ch.heldByTurn(cs) {
return nil, nil
}
rev, err := ch.Verdicts.Create(ctx, cs.Key, data)
if errors.Is(err, ErrConflict) {
return nil, nil // another checker made it first
@@ -285,7 +402,7 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil
}
return nil, nil
case !sameTrunk(cur.Trunk, cs.Trunk):
case ch.stale(cur, cs):
retake = true
case cur.State == delivery.VerdictFail && IsNodesFailure(cur):
ch.mu.Lock()
@@ -293,7 +410,7 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
ch.mu.Unlock()
retake = !ok || epoch > at
}
if !retake {
if !retake || ch.heldByTurn(cs) {
return nil, nil
}
rev, err = ch.Verdicts.Update(ctx, cs.Key, data, rev)
@@ -306,6 +423,59 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil
}
// heldByTurn says whether a merge turn holds the case's check off (ADR 0548 §3): an approved merge waits on one of the
// repositories it merges heads into, and it is not the merge of the case's own heads. The check is claimed when the turn's
// key goes (Reconsider on the turns watch). Said once per turn in the log. A check already claimed is never held: one
// claimed in the moment before the gate wrote its turn is judged on the commit it read and carried once after.
func (ch *Checker) heldByTurn(cs *Case) bool {
if ch.Turns == nil {
return false
}
repos := make([]string, 0, len(cs.Trunk))
for r := range cs.Trunk {
repos = append(repos, r)
}
repo, w, held := ch.Turns.Holding(repos, cs.Heads)
why := ""
if held {
why = repo + " " + w.Group + " " + w.Heads
}
ch.mu.Lock()
said := ch.held[cs.Key]
if why == "" {
delete(ch.held, cs.Key)
} else {
ch.held[cs.Key] = why
}
ch.mu.Unlock()
if held && said != why {
ch.logf("%s: not claimed: the approved merge of %s (heads %s) holds a turn on %s, so its check waits for that merge (ADR 0548 §3)",
cs.Key, w.Group, w.Heads, repo)
}
return held
}
// Held is every check a merge turn holds off now, by verdict key, with the turn's repository, group and heads.
func (ch *Checker) Held() map[string]string {
ch.init()
ch.mu.Lock()
defer ch.mu.Unlock()
out := make(map[string]string, len(ch.held))
for k, v := range ch.held {
out[k] = v
}
return out
}
// ConflictWords are what the words of a head that git would not merge onto its trunk say before the files in conflict.
const ConflictWords = "does not merge into the trunk: files in conflict: "
// IsConflictFailure says whether a verdict failed because its head did not merge onto its trunk (ADR 0548 §7): it has no
// pass to carry, so its next check, at a push or at a later trunk move, is a whole check of its head.
func IsConflictFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, ConflictWords)
}
// IsNodesFailure says whether a verdict failed because a part of the mesh the check reads was not heard (UnheardWords):
// it is judged again when the nodes change. No check fails for want of a build node any more: its asks wait in the build
// seats' streams (ADR 0525 §6, issue 1912).
@@ -332,11 +502,41 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
began := ch.now()
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
out := ch.judge(ctx, cs)
// An ask the build seat gave up on judged nothing (issue 734): the check is asked again under the same claim, its status
// pending, and after LostAskTries it fails as not heard, judged again when the nodes change.
for try := 1; IsLostAsk(out.Verdict) && ctx.Err() == nil; try++ {
if try >= LostAskTries {
out.Verdict.Failed.Words = strings.TrimSpace(out.Verdict.Failed.Words + "\nasked " + fmt.Sprint(LostAskTries) +
" times and given up each time: " + UnheardWords)
break
}
ch.logf("%s: %s; asked again (%d of %d)", cs.Key, out.Verdict.Failed.Words, try+1, LostAskTries)
out = ch.judge(ctx, cs)
}
if ctx.Err() != nil {
return
}
v := out.Verdict
v.Group, v.Heads, v.Trunk = cs.Group, cs.Heads, cs.Trunk
v.Group, v.Heads, v.Trunk, v.Beside = cs.Group, cs.Heads, cs.Trunk, nil
for r, c := range out.Beside {
if _, merged := cs.Trunk[r]; !merged && c != "" {
if v.Beside == nil {
v.Beside = map[string]string{}
}
v.Beside[r] = c
}
}
if cs.CarriedFrom != nil {
// A carry (ADR 0548 §5): a pass names the commits it was carried from; a failure names the move it failed on.
switch v.State {
case delivery.VerdictPass:
v.CarriedFrom = carried(cs.CarriedFrom, v)
case delivery.VerdictFail:
if moves := moveWords(cs.CarriedFrom, v); moves != "" {
v.Failed.Words = strings.TrimSpace(v.Failed.Words + "\nfailed on the carry over the move of " + moves + ": the move changed what these tests read")
}
}
}
if v.Took.Queue == 0 && !cs.Queued.IsZero() && began.After(cs.Queued) {
v.Took.Queue = began.Sub(cs.Queued).Milliseconds()
}
@@ -358,11 +558,11 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
ch.nodesFail[cs.Key] = epoch
ch.mu.Unlock()
}
if ch.movedSince(cs) {
if ch.movedSince(cs) || besideMoved(v.Beside, ch.besideNow()) {
// Judged on a trunk that moved while it ran: the verdict is kept as judged, and the status stays pending until the
// check of the new trunk, which Reconsider starts, ends.
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending", cs.Key, v.State)
ch.owePending(ctx, cs, v)
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending until it is carried", cs.Key, v.State)
return
}
ch.writeOwed(ctx, cs, v, out)
@@ -502,14 +702,37 @@ func (ch *Checker) TrunkMoved(ctx context.Context) {
if err != nil || !found {
continue // never judged: no status to put out of date
}
if v, err := delivery.DecodeVerdict(value); err == nil && sameTrunk(v.Trunk, cs.Trunk) {
v, err := delivery.DecodeVerdict(value)
if err == nil && !ch.stale(v, cs) {
continue
}
if err == nil && v.State == delivery.VerdictPass {
cs.CarriedFrom = judgedOn(v) // the status says it is being carried to the new commit (ADR 0548 §6)
}
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
ch.Reconsider(ctx)
}
// owePending owes mesh/check pending on the heads of a verdict judged on commits that moved while it ran: a pass says it is
// being carried to the trunk's head now (ADR 0548 §6), anything else that it is checked again.
func (ch *Checker) owePending(ctx context.Context, judged *Case, v delivery.Verdict) {
next := judged
ckey, err := delivery.CandidatesKey(v.Group)
ch.mu.Lock()
c, ok := ch.known[ckey]
ch.mu.Unlock()
if err == nil && ok {
if now := CaseOf(c, ch.heads()); now != nil && now.Key == judged.Key {
next = now
}
}
if v.State == delivery.VerdictPass {
next.CarriedFrom = judgedOn(v)
}
ch.writeOwed(ctx, next, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
// movedSince says whether a trunk the case was judged on is no longer the trunk's head.
func (ch *Checker) movedSince(cs *Case) bool {
now := ch.heads()
@@ -28,6 +28,7 @@ func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
f := &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "v0.1.14-x", Machines: []string{"g14"}, Took: 40}}
b.ex.Compositions = f
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.FPSet = "fps-of-the-candidate"
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionPass || v.Composition.SDK != "v0.1.14-x" || v.Took.Composition != 40 {
t.Fatalf("verdict = %+v, composition %+v", v, v.Composition)
@@ -38,6 +39,15 @@ func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
if !compositionMentions(v, "passed") {
t.Errorf("the line says %q", compositionWords(v.Composition, v.Took.Composition))
}
if v.Composition.FPSet == "" || v.Composition.FPSet != cs.FPSet {
t.Errorf("the composition names the fingerprint set %q, want the candidate's %q (ADR 0548 §5)", v.Composition.FPSet, cs.FPSet)
}
// ADR 0548 §5: a carry whose fingerprints did not change keeps the composition it passed and asks the assigner nothing.
carry := caseOf(t, pullOf(repoA, 7, head1))
carry.CarriedFrom, carry.Composition = map[string]string{repoA: trunkA2}, v.Composition
if kept := b.judge(t, carry).Verdict; kept.Composition == nil || kept.Composition.SDK != "v0.1.14-x" || len(f.asked) != 1 {
t.Errorf("the carry's composition is %+v and the assigner was asked %d times; want the kept one and no second ask", kept.Composition, len(f.asked))
}
}
func compositionMentions(v delivery.Verdict, word string) bool {
@@ -116,6 +116,9 @@ type bench struct {
fetch Fetched
digest map[string]Digested // by level
test func(c stepCall) (any, error)
// declines is how many asks naming needs a node lacking them declines before shanks, which has every need, takes
// one (issue 890); an ask naming none runs on shanks, which then has none.
declines int
}
// refsOf reads the siblings of a fetch's arguments, as they arrive in an ask (JSON) or as the checker gave them.
@@ -151,8 +154,17 @@ func (b *bench) answer(a delivery.StepAsk) delivery.Step {
if module == buildrun.GoBuildModule {
steps.Digest = tool("digest")
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
began := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
if len(a.Needs) > 0 {
if b.declines > 0 {
b.declines--
result, _ := json.Marshal(delivery.StepDeclined{Declined: true, Needs: a.Needs, Lacks: a.Needs})
return delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: "ace", State: delivery.StepRefused, Started: began, Ended: began,
Reason: "declined", Result: result}
}
steps.Has = func(string) bool { return true }
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed, Started: began, Ended: began.Add(2 * time.Second)}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
@@ -276,6 +288,49 @@ func TestOnlyThePackageWithoutAKeptVerdictRuns(t *testing.T) { // 1288/AC1
if u := out.Verdict.Levels[delivery.LevelUnit]; u.Run != 2 || u.Kept != 40 {
t.Errorf("unit = %+v, want 2 run and 40 kept", u)
}
// Issue 890/AC5: a package needing node-build-node is left unmet where the node lacks it; the check keeps the passes
// and asks the integration level again for a node with it, puts it again when a node declines it, and that node tests
// the package alone, its kept passes not counted twice.
// fixture/outside has no kept verdict after its pass (it reads outside itself: its digest moves every call), so a
// placed ask that ran more than its unmet packages would run and count it again.
b = newBench(t)
b.declines = 1
outside := 0
b.steps.handlers["build_go_digest"] = func(c stepCall) (any, error) {
d := echoBeside(b, b.digest[c.args["level"].(string)])
if d.Level == "integration" {
outside++
d.Packages = map[string]string{"fixture/slow": digB, "fixture/node": digC, "fixture/outside": fmt.Sprintf("%064x", outside)}
}
return d, nil
}
b.digest[delivery.LevelIntegration] = Digested{Level: "integration",
Needs: []delivery.PackageNeeds{{Package: "fixture/node", Needs: []string{delivery.NeedNodeBuildNode}}}}
out = b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
asks := b.asks.Asks()
if out.Verdict.State != delivery.VerdictPass || len(asks) != 3 || len(asks[0].Needs) != 0 {
t.Fatalf("verdict %+v, asks %+v: want a pass of three asks, the first naming no need", out.Verdict, asks)
}
for _, a := range asks[1:] {
if strings.Join(a.Needs, ",") != delivery.NeedNodeBuildNode || strings.Join(a.Check.Levels, ",") != delivery.LevelIntegration || strings.Join(a.Check.Packages, ",") != "fixture/node" {
t.Errorf("a placed ask %+v, want the integration level needing node-build-node", a)
}
}
tests = b.steps.tool("test")
if last := tests[len(tests)-1]; len(tests) != 3 || strings.Join(packagesOf(last), ",") != "fixture/node" || last.args["level"] != "integration" {
t.Errorf("test calls = %+v, want the unit level, the integration level of fixture/slow and fixture/outside, then fixture/node alone", tests)
}
slow := captured[Tested](t, "go-test-slow")
if i := out.Verdict.Levels[delivery.LevelIntegration]; i.Run != 2*slow.Tests || i.Kept != 0 {
t.Errorf("integration = %+v, want %d run and nothing counted kept", i, 2*slow.Tests)
}
// And where no node has the need, the check fails once the declines outlast the bound, naming the need.
b.kept, b.declines = newMemKV(), 1000
b.ex.Kept, b.ex.DeclineBound = Kept{Store: b.kept}, 200*time.Millisecond
if v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict; v.State != delivery.VerdictFail || !strings.Contains(fmt.Sprint(v), "needs a node with node-build-node") {
t.Errorf("with no node having the need: %+v", v)
}
}
func TestAReAskRunsOnlyWhatFailedAndWhatChanged(t *testing.T) { // 1288/AC2, AC3
@@ -620,9 +675,8 @@ func TestNothingIsKeptForAWorkspaceGivenSiblingsWhenTheDigestDoesNotSayItCoversT
// Beside): every check now places the change's own repository at its trunk (issue 1887 A12).
func echoBeside(b *bench, d Digested) Digested {
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
if fetches := b.steps.tool("fetch"); len(fetches) > 0 { // the workspace digested is the last fetch's (issue 890: a placed ask fetches again)
for _, r := range refsOf(fetches[len(fetches)-1].args["beside"]) {
lines = append(lines, r.Repository+" "+r.Commit)
}
}
@@ -4,7 +4,6 @@ import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -88,29 +87,3 @@ func TestATrunkMoveMarksEveryOpenPullRequestIntoItPendingAtOnce(t *testing.T) {
t.Errorf("mesh-sdk#13, alone into a trunk that did not move, is %q, want its success to stand", s)
}
}
func TestACheckThatEndsAfterItsTrunkMovedOwesPendingNotItsVerdict(t *testing.T) { // gate (a): a pass on an old trunk is never shown
ctx := context.Background()
var r *rig
release, judged := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
defer close(judged)
} else {
<-release // the check asked again for the new trunk waits in the stream meanwhile
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
<-judged
for deadline := time.Now().Add(2 * time.Second); owedState(t, r, repoA, 7, head1) == "" && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
time.Sleep(20 * time.Millisecond)
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status is %q, want pending: the pass was judged on a trunk that moved", s)
}
close(release)
r.ch.Wait()
}
@@ -80,6 +80,9 @@ var askLanguage = map[string]string{buildrun.GoBuildModule: delivery.AskGo, buil
// stopping at the first failure; it judges the step's answer as it judged the tools' answers.
type Examiner struct {
Asks Asks
// DeclineBound is how long a placed ask may go on being declined, from its first decline, before the check fails
// saying no node with its need took it (issue 890, ADR 0552); DefaultDeclineBound when zero.
DeclineBound time.Duration
// Toolchains are the nodes' toolchains: a kept verdict names the toolchain the nodes agree on.
Toolchains ToolchainEntries
Kept Kept
@@ -203,12 +206,15 @@ type state struct {
nodes map[string]bool
clone time.Duration
slow []string
// beside are the siblings read at their trunk's head, by repository (ADR 0548 §6).
beside map[string]string
}
// Judge runs the case (it is the Judge of a Checker). A failure of the change, of a merge or of a step is a verdict; an error is only a case that cannot
// be judged at all.
func (j *Examiner) Judge(ctx context.Context, c *Case) (Outcome, error) {
st := &state{j: j, c: c, levels: map[string]*acc{delivery.LevelUnit: {}, delivery.LevelIntegration: {}}, nodes: map[string]bool{}}
st := &state{j: j, c: c, levels: map[string]*acc{delivery.LevelUnit: {}, delivery.LevelIntegration: {}}, nodes: map[string]bool{},
beside: map[string]string{}}
if v := st.judgeAll(ctx); v != nil {
return st.finish(ctx, *v), nil
}
@@ -230,6 +236,13 @@ func (st *state) finish(ctx context.Context, v delivery.Verdict) Outcome {
// fails the check, naming the machine it failed at. A try is not composed (it has no candidate); a composition that
// cannot be had leaves it not run, said; with no assigner running it is not judged, said at once.
func (st *state) composed(ctx context.Context, v *delivery.Verdict) {
if st.c.Composition != nil && st.c.Try == nil {
// A carry whose candidate's fingerprints did not change keeps the composition it passed (ADR 0548 §5).
kept := *st.c.Composition
v.Composition = &kept
st.c.note("the composition is kept: no fingerprint of a module it composed changed")
return
}
if st.j.Compositions == nil || st.c.Try != nil {
return
}
@@ -243,6 +256,9 @@ func (st *state) composed(ctx context.Context, v *delivery.Verdict) {
return
}
withComposition(v, comp)
if v.Composition != nil && v.Composition.State != delivery.CompositionNotRun {
v.Composition.FPSet = st.c.FPSet // a carry keeps it while the fingerprints stay the same (ADR 0548 §5)
}
}
// outcome fills the verdict's levels, times, machines and composition from what ran.
@@ -282,6 +298,9 @@ func (st *state) outcome(v delivery.Verdict) Outcome {
}
}
out := Outcome{Verdict: v, Slowest: slow}
if len(st.beside) > 0 {
out.Beside = st.beside
}
for _, t := range st.targets {
if t.judged != "" {
if out.Judged == nil {
@@ -343,6 +362,7 @@ func (st *state) judgeAll(ctx context.Context) *delivery.Verdict {
if len(beside) > 0 {
args["beside"] = beside
t.beside = besideLines(beside)
besideAtTrunk(st.c, beside, st.beside)
}
if p.Repository == catalogueRepository {
t.runner = RunnerWorkspace // used by the build module when the merge changes it (ADR 0535 §1)
@@ -364,11 +384,25 @@ func (st *state) judgeAll(ctx context.Context) *delivery.Verdict {
return nil
}
// DefaultDeclineBound is an Examiner's DeclineBound when it names none: longer than a check runs, so a node with the
// need that is busy is waited for, and short enough that a mesh where no node has it says so.
const DefaultDeclineBound = 30 * time.Minute
// ask puts the target's check ask, waits for the step of the node that pulled it and judges its answer: the fetch, the
// pins of its go.mod files, then each level's digest, kept verdicts and tests.
// pins of its go.mod files, then each level's digest, kept verdicts and tests. When the answer leaves packages unmet,
// their tests needing what that node lacks (issue 890), the passes are kept and the levels from the first such one are
// asked again of a node that has it (placed).
func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *delivery.Verdict {
return st.askNeeding(ctx, t, merged, st.levelsOf(), nil, nil)
}
// askNeeding is ask with the levels to run, what the node must have and the packages it tests: none for the first ask
// of a target; for the placed one the unmet packages' needs and those packages alone, so no package the first ask judged
// is run or counted again (ADR 0552 §4).
func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, levels, needs, packages []string) *delivery.Verdict {
p := t.pull
job := &delivery.CheckJob{Fetch: t.fetchArgs, Levels: st.levelsOf(), Kept: true}
placed := len(needs) > 0
job := &delivery.CheckJob{Fetch: t.fetchArgs, Levels: levels, Kept: true, Packages: packages}
if t.runner != "" {
job.Runner, job.RunnerWhen = t.runner, buildGoDir
}
@@ -380,17 +414,46 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
kind = delivery.AskRecheck // a re-judge after a trunk move goes after every new head's check (ADR 0536 §4)
}
a := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: askLanguage[t.module], Asker: delivery.CheckerModule,
At: st.j.now().UTC(), Check: job}
st.c.Progress.Set(fmt.Sprintf("the %s ask %s of %s#%d", kind, a.ID, p.Repository, p.Number), "")
if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
}
step, err := st.j.Asks.Wait(ctx, a)
if err != nil {
if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error())
At: st.j.now().UTC(), Check: job, Needs: needs}
var step delivery.Step
var firstDecline time.Time
for {
what := fmt.Sprintf("the %s ask %s of %s#%d", kind, a.ID, p.Repository, p.Number)
if placed {
what += " for a node with " + strings.Join(needs, ", ")
}
return fail(t.module, fmt.Sprintf("the check of %s#%d: %v", p.Repository, p.Number, err))
st.c.Progress.Set(what, "")
if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
}
var err error
step, err = st.j.Asks.Wait(ctx, a)
if err != nil {
if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error())
}
return fail(t.module, fmt.Sprintf("the check of %s#%d: %v", p.Repository, p.Number, err))
}
d, declined := delivery.DeclinedOf(step)
if !declined {
break
}
// A node that lacks the need dropped the ask (issue 890): it is put again, for a node that has it to take; the
// node that declined looks at the stream again only after its next pull's wait.
now := st.j.now()
if firstDecline.IsZero() {
firstDecline = now
}
bound := st.j.DeclineBound
if bound <= 0 {
bound = DefaultDeclineBound
}
if now.Sub(firstDecline) > bound {
return fail(step.Node, fmt.Sprintf("the check of %s#%d needs a node with %s, and none took it in %s: every node that pulled it lacks it (last %s)",
p.Repository, p.Number, strings.Join(needs, ", "), bound, step.Node))
}
st.j.logf("%s#%d: the ask %s was declined on %s, which lacks %s; asked again", p.Repository, p.Number, a.ID, step.Node, strings.Join(d.Lacks, ", "))
a.ID, a.At = buildstep.NewID(), now.UTC()
}
// The fetch and the digests are what of the ask's time its levels' running did not take (Took.Clone).
took := time.Duration(0)
@@ -414,18 +477,28 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
if st.c.Try == nil {
st.j.writePreview(ctx, p, append(*merged, t.fetched))
}
return fail(step.Node, fmt.Sprintf("%s#%d does not merge into the trunk: files in conflict: %s", p.Repository, p.Number, strings.Join(t.fetched.Conflicts, ", ")))
return fail(step.Node, fmt.Sprintf("%s#%d %s%s", p.Repository, p.Number, ConflictWords, strings.Join(t.fetched.Conflicts, ", ")))
}
t.changed = t.fetched.Changed
if !changesBuildGo(p.Repository, t.changed) {
t.runner = "" // the build module used no runner: the merge does not change it
}
*merged = append(*merged, t.fetched)
if v := st.checkPinsOf(ctx, t); v != nil {
return v
if !placed { // the placed ask fetched the same merge: its preview and pins were read from the first
*merged = append(*merged, t.fetched)
if v := st.checkPinsOf(ctx, t); v != nil {
return v
}
}
ran := 0.0
for _, la := range answer.Levels {
if placed {
if len(la.Unmet) > 0 {
st.clone += max(0, took-time.Duration(ran*float64(time.Second)))
return fail(step.Node, fmt.Sprintf("the %s level of %s#%d left %s untested on %s, which was asked for a node with %s",
la.Level, p.Repository, p.Number, unmetNames(la.Unmet), step.Node, strings.Join(needs, ", ")))
}
la.Kept = nil // it names its packages and takes no other's kept verdict; this is only a guard against an older module
}
v, elapsed := st.level(ctx, t, la)
ran += elapsed
if v != nil {
@@ -434,9 +507,48 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
}
}
st.clone += max(0, took-time.Duration(ran*float64(time.Second)))
if unmet := delivery.UnmetNeeds(answer.Levels); len(unmet) > 0 && !placed {
// Every level passed, its passes kept, and some packages were left to a node with what their tests need: the levels
// from the first that left one are asked of such a node, which takes the kept passes and tests those (issue 890).
var again, packages []string
seen := map[string]bool{}
for _, la := range answer.Levels {
if len(again) > 0 || len(la.Unmet) > 0 {
again = append(again, la.Level)
}
for _, u := range la.Unmet {
if !seen[u.Package] {
seen[u.Package] = true
packages = append(packages, u.Package)
}
}
}
st.j.logf("%s#%d: %s on %s left untested; asked of a node with %s", p.Repository, p.Number, unmetOf(answer.Levels), step.Node, strings.Join(unmet, ", "))
return st.askNeeding(ctx, t, merged, again, unmet, packages)
}
return nil
}
// unmetNames names the unmet packages and their needs.
func unmetNames(unmet []delivery.PackageNeeds) string {
var out []string
for _, u := range unmet {
out = append(out, u.Package+" (needs "+strings.Join(u.Needs, ", ")+")")
}
return strings.Join(out, ", ")
}
// unmetOf names the unmet packages of every level.
func unmetOf(levels []delivery.LevelAnswer) string {
var out []string
for _, la := range levels {
if len(la.Unmet) > 0 {
out = append(out, "at "+la.Level+" "+unmetNames(la.Unmet))
}
}
return strings.Join(out, "; ")
}
// level judges one level of a check ask's answer as the tools' answers were judged: the digest keys the kept verdicts
// of the packages that passed, the kept ones taken are counted, and every test result is read and counted, also after
// one that did not pass, so the verdict names every failed test of the level (issue 1921). It answers the level's
@@ -56,12 +56,22 @@ func (k Kept) Keep(ctx context.Context, level, digest string, v delivery.Kept) (
// KeepPasses keeps, for the packages of a test result that passed with tests, a verdict under the digest the build module
// named for the package at the level. A package that failed, has no tests or has no digest is never kept (ADR 0444 clause
// 1): it runs every check. It answers the packages it kept now.
// 1): it runs every check. A package with skipped tests is not kept either: a pass that skipped (a test needing a
// toolchain the node lacks) would be reused for its digest and never run anywhere (issue 890). A build module that
// counts skips only per result, not per package, keeps nothing of a result in which a test skipped. It answers the
// packages it kept now.
func (k Kept) KeepPasses(ctx context.Context, repository, level, toolchain string, res Tested, digests map[string]string, at time.Time) ([]string, error) {
var kept []string
perPackage := false
for _, p := range res.Packages {
perPackage = perPackage || p.Skipped > 0
}
if res.Skipped > 0 && !perPackage {
return kept, nil // an older build module: the skip cannot be placed in a package
}
for _, p := range res.Packages {
digest, ok := digests[p.Package]
if !ok || !p.Passed || p.NoTests || p.Tests == 0 {
if !ok || !p.Passed || p.NoTests || p.Tests == 0 || p.Skipped > 0 {
continue
}
if failedIn(res, p.Package) {
@@ -74,6 +74,32 @@ func TestOnlyPassingPackagesWithADigestAreKept(t *testing.T) { // 1287/AC3, AC4
}
}
func TestAResultWithSkippedTestsKeepsNothing(t *testing.T) { // issue 890: a pass that skipped proved nothing
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
res := Tested{Level: "integration", Passed: true, Skipped: 2, Packages: []GoPackage{{Package: "m/a", Passed: true, Tests: 4}}}
kept, err := k.KeepPasses(ctx, "novox/mesh-catalog", "integration", "go1.26.0", res, map[string]string{"m/a": digA}, at0)
if err != nil || len(kept) != 0 {
t.Fatalf("kept %v, %v, want nothing", kept, err)
}
if keys, _ := st.Keys(ctx); len(keys) != 0 {
t.Errorf("the bucket holds %v, want none", keys)
}
}
func TestOnlyTheSkippingPackageOfAResultIsWithheld(t *testing.T) { // issue 890
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
res := Tested{Level: "integration", Passed: true, Skipped: 2, Packages: []GoPackage{
{Package: "m/a", Passed: true, Tests: 4, Skipped: 2},
{Package: "m/b", Passed: true, Tests: 3},
}}
kept, err := k.KeepPasses(ctx, "novox/mesh-catalog", "integration", "go1.26.0", res, map[string]string{"m/a": digA, "m/b": digB}, at0)
if err != nil || strings.Join(kept, ",") != "m/b" {
t.Fatalf("kept %v, %v, want only m/b", kept, err)
}
}
func TestAKeptValueOfAnotherDigestIsNotRead(t *testing.T) { // 1287/AC7
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
@@ -85,18 +85,20 @@ type pieces struct {
head, commits, counts, shortCounts, times, shortTimes, over, builds, composition, shortComposition, group string
// judged names the program that judged a change to a build module (ADR 0535 §5); never dropped.
judged string
// carried says, of a carried verdict, the commits it was carried from (ADR 0548 §8); never dropped.
carried string
}
// pullKey names a pull request of a case: `<owner/repo>#<number>`.
func pullKey(p Pull) string { return fmt.Sprintf("%s#%d", p.Repository, p.Number) }
func (p pieces) join(drop int) string {
parts := []string{p.head, p.commits, p.counts, p.judged}
parts := []string{p.head, p.commits, p.carried, p.counts, p.judged}
switch {
case drop < 3 && p.times != "":
parts = append(parts, p.times)
case drop == 3 && p.shortTimes != "":
parts = []string{p.head, p.commits, p.shortCounts, p.judged, p.shortTimes}
parts = []string{p.head, p.commits, p.carried, p.shortCounts, p.judged, p.shortTimes}
}
if p.over != "" {
parts = append(parts, p.over)
@@ -171,6 +173,39 @@ func buildWords(builds []BuildNote) string {
return "builds: " + strings.Join(parts, ", ")
}
// carriedWords say the commits a verdict was carried from, for the line of pull request p (ADR 0548 §8): its own trunk's
// commit as twelve characters, a sibling's with its name before it; only those that moved, sorted. Empty when none did.
func carriedWords(from map[string]string, v delivery.Verdict, cs *Case, p Pull) string {
var own string
var others []string
for r, old := range from {
now, ok := v.Trunk[r]
if cs != nil && !ok {
now, ok = cs.Trunk[r]
}
if !ok {
now = v.Beside[r]
}
if now == "" || now == old {
continue
}
if strings.EqualFold(r, p.Repository) {
own = short12(old)
continue
}
name := r
if i := strings.LastIndex(r, "/"); i >= 0 {
name = r[i+1:]
}
others = append(others, name+" "+short12(old))
}
slices.Sort(others)
if own != "" {
others = append([]string{own}, others...)
}
return strings.Join(others, ", ")
}
// parts makes the pieces of the line of a pull request of a case.
func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) pieces {
var pc pieces
@@ -179,8 +214,18 @@ func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []s
switch v.State {
case delivery.VerdictRunning:
pc.head = "checking " + commits
if cs != nil && cs.CarriedFrom != nil {
// Out of date and being carried to the new commit (ADR 0548 §6): pending, and saying so.
pc.head = "being carried to " + commits
if w := carriedWords(cs.CarriedFrom, v, cs, p); w != "" {
pc.carried = "from " + w
}
}
case delivery.VerdictPass:
pc.head = "pass " + commits
if w := carriedWords(v.CarriedFrom, v, cs, p); w != "" {
pc.carried = "carried from " + w // the counts that follow say what the carry ran and kept
}
default:
pc.head = "failed: " + firstFailed(v)
if n := failedCount(v); n > 1 {
@@ -0,0 +1,34 @@
package check
import (
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **An ask the build seat gave up on ends its step** (novox/hq issue 734): mesh-sdk buildrun.LostAskWaiter hands the
// waiting asker a failed step on the node buildrun.LostNode when DEAD_LETTERS keeps the ask's copy (buildrun.HearDeadLetters,
// granted by buscompose DeadLetterReads), and, under buildrun.NotTakenBound, when no build node took the ask within the
// bound. What follows is the checker's: a try ends as error (Tryer.run), with the not-taken bound at the step bound
// (Tryer.NotTaken); a check is asked again under its claim, at most LostAskTries times, then fails as not heard and is
// judged again when the nodes change (Checker.try). A check's ask has no not-taken bound: it waits in the stream as long as
// the backlog is (ADR 0525 §9), and a check ended for a backlog would owe its pull request a failure.
// LostAskTries is how often a check asks again whose ask was given up, before it fails as not heard (UnheardWords) and is
// judged again when the nodes change.
const LostAskTries = 3
// IsLostAsk says whether a verdict failed because the build seat gave its ask up (buildrun.LostWords).
func IsLostAsk(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, buildrun.LostWords)
}
// IsNotTaken says whether a verdict failed because no build node took its ask within the bound (buildrun.NotTakenWords).
func IsNotTaken(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, buildrun.NotTakenWords)
}
// HearingBucket is the checker's state hearing: whether it hears DEAD_LETTERS (buildrun.Hearing under
// buildrun.HearingKey), which checker_status says (dead_letters).
const HearingBucket = delivery.CheckerModule + "_" + buildrun.HearingState
@@ -0,0 +1,170 @@
package check
import (
"context"
"encoding/json"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// losingAsks is the build seat over a bus that gives up the first `lose` asks put (DEAD_LETTERS keeps them, as on
// 2026-10-26 during a bus reload: try-1793017340003993788) and answers the rest; with never set it neither answers nor
// gives up (an ask no node takes).
type losingAsks struct {
*buildrun.FakeAsks
answer func(delivery.StepAsk) delivery.Step
dead *buildrun.DeadLetters
never bool
mu sync.Mutex
lose int
}
func (l *losingAsks) PublishAsk(ctx context.Context, subject, id string, data []byte) error {
if err := l.FakeAsks.PublishAsk(ctx, subject, id, data); err != nil {
return err
}
l.mu.Lock()
lost := l.lose > 0
if lost {
l.lose--
}
l.mu.Unlock()
switch {
case l.never:
case lost:
h := map[string][]string{buildrun.DeadHeaderMsgID: {id}, buildrun.DeadHeaderDeliveries: {"5"}, buildrun.DeadHeaderStream: {"SEAT_NODE_BUILD_GO"},
buildrun.DeadHeaderGaveUp: {"2026-10-26T12:26:40Z"}}
g, ok := buildrun.GivenUpOf(h)
if !ok {
return nil
}
go l.dead.Heard(g)
default:
a, err := delivery.DecodeStepAsk(data)
if err != nil {
return err
}
l.Write(l.answer(a))
}
return nil
}
func losing(b *bench, lose int) *losingAsks {
l := &losingAsks{FakeAsks: &buildrun.FakeAsks{}, answer: b.answer, dead: &buildrun.DeadLetters{}, lose: lose}
b.ex.Asks = buildrun.Asker{Publisher: l, Waiter: buildrun.LostAskWaiter{Inner: l.FakeAsks, Dead: l.dead}, Bound: 5 * time.Second}
return l
}
// endable is the tries state as the end tool sees it.
type endable struct{ kv *memKV }
func (e endable) Entry(key string) (json.RawMessage, uint64, error) {
v, rev, _, err := e.kv.Get(context.Background(), key)
return v, rev, err
}
func (e endable) Update(key string, value []byte, revision uint64) error {
_, err := e.kv.Update(context.Background(), key, value, revision)
return err
}
// Issue 734: a try whose ask the build seat gave up on ends as error saying so, and the next ask of its head is a new try;
// one no node takes ends at the step bound; one ended by hand frees its place and stops being waited for.
func TestAnAskGivenUpOrNeverTakenEndsItsTryAsErrorAndItsHeadIsAskedAnew(t *testing.T) {
ctx := context.Background()
for _, row := range []struct {
name string
lose int
never bool
want string
}{
{"given up by the build seat", 1, false, buildrun.LostWords + " after 5 deliveries at 2026-10-26T12:26:40Z"},
{"taken by no build node within the bound", 0, true, buildrun.NotTakenWords + " within 200ms"},
} {
t.Run(row.name, func(t *testing.T) {
kv := newMemKV()
at := time.Date(2026, 10, 26, 12, 22, 20, 0, time.UTC)
id, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, at)
if err != nil {
t.Fatal(err)
}
b := newBench(t)
l := losing(b, row.lose)
l.never = row.never
tr := &Tryer{Tries: kv, Judge: b.ex, NotTaken: 200 * time.Millisecond}
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
tr.Wait()
got := readTry(t, kv, id)
if got.State != TryError || len(got.Failed) != 1 || !strings.Contains(got.Failed[0], row.want) || !strings.Contains(got.Said, row.want) {
t.Fatalf("try = %+v, want error saying %q", got, row.want)
}
again, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, at.Add(time.Minute))
if err != nil || again == id {
t.Errorf("the head asked again answered %q, %v, want a new try", again, err)
}
})
}
t.Run("ended by hand", func(t *testing.T) {
kv := newMemKV()
id, _ := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, time.Now())
b := newBench(t)
losing(b, 0).never = true
tr := &Tryer{Tries: kv, Judge: b.ex} // no bound: only the hand ends it
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
deadline := time.Now().Add(2 * time.Second)
for readTry(t, kv, id).State != TryRunning && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if _, err := EndTry(endable{kv}, id, "", time.Now()); err == nil {
t.Error("an end without a why was taken")
}
ended, err := EndTry(endable{kv}, id, "its ask was lost in the bus reload", time.Now())
if err != nil || ended.State != TryError || !strings.HasPrefix(ended.Said, "ended by hand ("+TryEndTool+"): its ask was lost") {
t.Fatalf("ended = %+v, %v", ended, err)
}
raw, _, _, _ = kv.Get(ctx, id)
tr.Put(ctx, id, raw) // the watch of the tries hands the end over
tr.Wait() // returns: the try's judging stopped
if got := readTry(t, kv, id); got.State != TryError || got.Said != ended.Said {
t.Errorf("try = %+v, want the hand's end kept", got)
}
if _, err := EndTry(endable{kv}, id, "again", time.Now()); err == nil || !strings.Contains(err.Error(), "ended already") {
t.Errorf("a second end answered %v", err)
}
})
}
// Issue 734: a pull request's check whose ask the build seat gave up on is asked again under its claim, its status
// pending, and judged; one given up every time fails as not heard and is judged again when the nodes change.
func TestACheckWhoseAskWasGivenUpIsAskedAgain(t *testing.T) {
ctx := context.Background()
for _, row := range []struct {
lose int
state string
asks int
}{
{1, delivery.VerdictPass, 2},
{LostAskTries, delivery.VerdictFail, LostAskTries},
} {
b := newBench(t)
l := losing(b, row.lose)
r := newRig(t, b.ex)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if v.State != row.state || len(l.Asks()) != row.asks {
t.Errorf("lose %d: verdict %s (%s) after %d asks, want %s after %d", row.lose, v.State, v.Failed.Words, len(l.Asks()), row.state, row.asks)
}
if row.state == delivery.VerdictFail && (!IsLostAsk(v) || !IsNodesFailure(v)) {
t.Errorf("a check given up every time = %q, want lost and judged again when the nodes change", v.Failed.Words)
}
}
}
@@ -240,7 +240,12 @@ func TestTheSiblingsAreWhereTheTestsStateThem(t *testing.T) { // review D2, ADR
catalogue + " go.mod " + head2: "module git.novox.be/novox/mesh-catalog\n"}, onTrunk: map[string]bool{"cb54756c3865": true}}
b.ex.Forge = forge
// A group of the controller and the catalogue: the controller's tests read the catalogue's trunk, not the group's head.
b.judge(t, caseOf(t, pullOf(controller, 7, head1), pullOf(catalogue, 12, head2)))
out := b.judge(t, caseOf(t, pullOf(controller, 7, head1), pullOf(catalogue, 12, head2)))
// ADR 0548 §6, issue 568's own example: the verdict names the node-engine at the commit the mesh runs, and mesh-sdk at the
// trunk the catalogue's tests read it at; the catalogue, a member here, is not named.
if out.Beside[host] != hostRuns || out.Beside[catalogue] != "" || out.Beside[SDKRepository] != "9999999999999999999999999999999999999999" {
t.Errorf("the siblings named for the verdict = %v, want the node-engine at %s and mesh-sdk at its trunk", out.Beside, hostRuns[:12])
}
got := besideOf(t, b, controller)
if c := got[catalogue]; c.Commit != trunkS || c.Ref != "" {
t.Errorf("the catalogue beside the controller = %+v, want its trunk %s", c, trunkS)
@@ -324,7 +329,11 @@ func TestTheChangesOwnRepositoryIsBesideItAtItsTrunk(t *testing.T) {
func TestACatalogueChangeHasTheCatalogueBesideItAtItsTrunk(t *testing.T) {
b := newBench(t)
b.ex.Trunks = trunksOf(t, map[string]string{catalogueRepository: trunkA, SDKRepository: trunkS, controllerRepo: trunkA2})
b.judge(t, caseOf(t, pullOf(catalogueRepository, 12, head1)))
out := b.judge(t, caseOf(t, pullOf(catalogueRepository, 12, head1)))
// ADR 0548 §6: the outcome names the siblings read at their trunk's head, never the repository it merges.
if out.Beside[SDKRepository] != trunkS || out.Beside[controllerRepo] != trunkA2 || out.Beside[catalogueRepository] != "" {
t.Errorf("the siblings named for the verdict = %v, want mesh-sdk and mesh-controller at their trunks and not the catalogue", out.Beside)
}
got := besideOf(t, b, catalogueRepository)
if c := got[catalogueRepository]; c.Commit != trunkA || c.Ref != "" {
t.Fatalf("the catalogue beside its own change = %+v, want its trunk %s", c, trunkA)
@@ -75,3 +75,40 @@ func TestCheckerStatusNamesTheWaitingChecksByKind(t *testing.T) { // 1897/AC4
t.Errorf("waiting = %v, want %v", st.Waiting, want)
}
}
// 568 note 21: a verdict that read the node-engine at its running commit is not waiting for a trunk move while that commit
// is still the one the mesh runs, though the node-engine's trunk has moved on (ADR 0548 §6).
func TestCheckerStatusReadsTheNodeEngineAtItsRunningCommit(t *testing.T) { // 568/21
cand := candidateOf(t, group(1), pullOf(repoA, 1, head1))
ckey, _ := delivery.CandidatesKey(group(1))
vkey := keyOf(t, group(1), cand)
dc, _ := delivery.DecodeCandidate(cand)
judged, err := delivery.EncodeVerdict(delivery.Verdict{Group: group(1), Heads: dc.HeadsHash, Trunk: map[string]string{repoA: trunkA}, Beside: map[string]string{nodeEngineRepo: "run111"}, State: delivery.VerdictPass, Levels: passing().Levels})
if err != nil {
t.Fatal(err)
}
heads := func() (map[string]string, error) {
return map[string]string{repoA: trunkA, nodeEngineRepo: "trunk222"}, nil
}
for _, row := range []struct {
name string
runs runsAt
want int
}{
{"the running commit the verdict read", runsAt{nodeEngineModule: "run111"}, 0},
{"the running commit moved", runsAt{nodeEngineModule: "run999"}, 1},
{"no running commit known: the trunk's", nil, 1},
} {
rd := Reader{CandidateState: fakeState{ckey: cand}.read(), VerdictState: fakeState{vkey: judged}.read(), Trunks: heads}
if row.runs != nil {
rd.Runs = row.runs
}
st, err := rd.Status()
if err != nil {
t.Fatal(err)
}
if got := len(st.Waiting[KindTrunk]); got != row.want {
t.Errorf("%s: waiting for a trunk move = %v, want %d", row.name, st.Waiting[KindTrunk], row.want)
}
}
}
@@ -8,6 +8,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// The read tools (novox/hq ADR 0484 clauses 8 and 13, issue 1686): what the checker stored, as stored, with the line of the
@@ -39,7 +40,14 @@ type Reader struct {
StepBound time.Duration // how long the checker waits for one step; DefaultStepBound when 0
// Trunks are the trunk heads of every repository (the gate's source); nil names no re-judge after a trunk move.
Trunks func() (map[string]string, error)
Now func() time.Time
// TurnState is the gate's merge turns (ADR 0548 §3); without it no check is named as held by a turn.
TurnState StateRead
// Runs says which commit the mesh runs of a module (the indexer's modules), as the checker's decision reads it: the
// node-engine a verdict names is compared at its running commit, not at the trunk's (ADR 0548 §6); nil knows none.
Runs interface{ CommitOf(module string) string }
// HearingState is the checker's state hearing: whether it hears DEAD_LETTERS (issue 734).
HearingState StateRead
Now func() time.Time
}
// VerdictRead is a verdict as stored with the full report of the status of each pull request it judged (Report), when the
@@ -312,7 +320,68 @@ type Status struct {
// Waiting are the candidates whose check waits, by kind (ADR 0536 §4): new (no verdict for its heads) and nodes (failed
// for the nodes, asked again when they change) start before trunk (judged on trunk heads that moved since).
Waiting map[string][]string `json:"waiting_by_kind"`
Notes []string `json:"notes,omitempty"`
// HeldByTurn are the candidates whose check an approved merge's turn holds off (ADR 0548 §3), by group: the repository and
// the group of the merge that waits. Their check is claimed when that turn goes.
HeldByTurn map[string]string `json:"held_by_turn"`
// DeadLetters says whether the checker hears the asks the build seats give up on (issue 734): deaf, a try or check
// whose ask is given up waits until its bound.
DeadLetters DeadLettersHeard `json:"dead_letters"`
Notes []string `json:"notes,omitempty"`
}
// DeadLettersHeard is the checker's hearing of DEAD_LETTERS as the status says it: healthy or not, and why.
type DeadLettersHeard struct {
Heard bool `json:"heard"`
Why string `json:"why"`
}
// deadLetters reads the checker's kept hearing (buildrun.HearingHealth).
func (r Reader) deadLetters() DeadLettersHeard {
if r.HearingState.Get == nil {
return DeadLettersHeard{Why: "the checker's state hearing is not readable here"}
}
raw, err := r.HearingState.Get(buildrun.HearingKey)
if err != nil {
return DeadLettersHeard{Why: "the checker's state hearing cannot be read: " + err.Error()}
}
a := buildrun.HearingHealth(raw)
return DeadLettersHeard{Heard: a.Healthy, Why: a.Why}
}
// turns reads the gate's merge turns, by repository; nil when they cannot be read, said in a note.
func (r Reader) turns(st *Status) *Turns {
if r.TurnState.Keys == nil || r.TurnState.Get == nil {
st.Notes = append(st.Notes, "the gate's merge turns are not readable here, so no check is named as held by one")
return nil
}
keys, err := r.TurnState.Keys()
if err != nil {
st.Notes = append(st.Notes, "the gate's merge turns cannot be listed, so no check is named as held by one: "+err.Error())
return nil
}
t := &Turns{}
for _, k := range keys {
if raw, err := r.TurnState.Get(k); err == nil && len(raw) > 0 {
t.Put(k, raw, false)
}
}
return t
}
// heldBy says the turn that holds a candidate's check off, or "".
func heldBy(t *Turns, c delivery.Candidate, heads map[string]string) string {
cs := CaseOf(c, heads)
if t == nil || cs == nil {
return ""
}
repos := make([]string, 0, len(cs.Trunk))
for r := range cs.Trunk {
repos = append(repos, r)
}
if repo, w, ok := t.Holding(repos, cs.Heads); ok {
return repo + ": the approved merge of " + w.Group + " waits on it"
}
return ""
}
// Status reads the build modules' steps and the candidates against their verdicts.
@@ -383,6 +452,8 @@ func (r Reader) Status() (Status, error) {
}
st.Unjudged = []string{}
st.Waiting = map[string][]string{KindNew: {}, KindNodes: {}, KindTrunk: {}}
st.HeldByTurn = map[string]string{}
turns := r.turns(&st)
var heads map[string]string
if r.Trunks == nil {
st.Notes = append(st.Notes, "the trunk heads are not readable here, so no check is named as waiting for a trunk move")
@@ -435,6 +506,9 @@ func (r Reader) Status() (Status, error) {
if len(raw) == 0 {
st.Unjudged = append(st.Unjudged, c.Group)
st.Waiting[KindNew] = append(st.Waiting[KindNew], c.Group)
if why := heldBy(turns, c, heads); why != "" {
st.HeldByTurn[c.Group] = why
}
continue
}
v, err := delivery.DecodeVerdict(raw)
@@ -443,12 +517,16 @@ func (r Reader) Status() (Status, error) {
case IsNodesFailure(v):
st.Waiting[KindNodes] = append(st.Waiting[KindNodes], c.Group)
case heads != nil:
if cs := CaseOf(c, heads); cs != nil && !sameTrunk(v.Trunk, cs.Trunk) {
if cs := CaseOf(c, heads); cs != nil && (!sameTrunk(v.Trunk, cs.Trunk) || besideMoved(v.Beside, besideAt(heads, r.Runs))) {
st.Waiting[KindTrunk] = append(st.Waiting[KindTrunk], c.Group)
if why := heldBy(turns, c, heads); why != "" {
st.HeldByTurn[c.Group] = why
}
}
}
}
}
st.DeadLetters = r.deadLetters()
st.Notes = append(st.Notes, "nothing holds the checker and it holds no place: running_steps are the build modules' keys that can still be running (started within the step bound, their build module running on the node); a candidate without a verdict has its check ask in a build seat's stream, waiting for a node with a free slot, whose backlog the delivery view shows (ADR 0525 §4, §9)")
return st, nil
}
@@ -144,7 +144,10 @@ func TestTheStatusToolReadsRunningSteps(t *testing.T) {
CandidateState: fakeState{ckey: cand}.read(),
VerdictState: fakeState{}.read(),
HaveState: fakeState{"shanks.mesh-build-go": haveOf(t, delivery.PhaseRunning)}.read(),
Now: func() time.Time { return time.Date(2026, 10, 22, 15, 10, 0, 0, time.UTC) },
HearingState: fakeState{"dead-letters": []byte(`{"heard":false,"streams":["SEAT_NODE_BUILD_GO"],` +
`"why":"DEAD_LETTERS is not heard, so an ask a build seat gives up on is waited for until its bound: SEAT_NODE_BUILD_GO: permissions violation",` +
`"at":"2026-10-26T13:00:00Z"}`)}.read(),
Now: func() time.Time { return time.Date(2026, 10, 22, 15, 10, 0, 0, time.UTC) },
}
st, err := rd.Status()
if err != nil {
@@ -156,9 +159,13 @@ func TestTheStatusToolReadsRunningSteps(t *testing.T) {
if len(st.Unjudged) != 1 || st.Unjudged[0] != "feat/x" {
t.Errorf("unjudged = %v", st.Unjudged)
}
// Issue 734, review round 1: a checker that does not hear DEAD_LETTERS says so in its status, with why.
if st.DeadLetters.Heard || !strings.Contains(st.DeadLetters.Why, "permissions violation") {
t.Errorf("dead letters = %+v, want not heard, saying why", st.DeadLetters)
}
// What cannot be read is said, not guessed.
st, err = Reader{}.Status()
if err != nil || len(st.Notes) == 0 {
if err != nil || len(st.Notes) == 0 || st.DeadLetters.Heard {
t.Errorf("empty status = %+v, %v", st, err)
}
}
@@ -234,6 +241,18 @@ func TestTheStatusCountsAnHQOnlyGroupAsUnjudged(t *testing.T) {
if len(st.Unjudged) != 1 || st.Unjudged[0] != "pb-x" {
t.Errorf("unjudged = %v, want pb-x", st.Unjudged)
}
// ADR 0548 §3: a candidate whose check an approved merge's turn holds off is named with the turn.
held := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
hkey, _ := delivery.CandidatesKey("feat/x")
st, err = Reader{CandidateState: fakeState{hkey: held}.read(), VerdictState: fakeState{}.read(),
Trunks: func() (map[string]string, error) { return map[string]string{repoA: trunkA}, nil },
TurnState: fakeState{turnKey(t, repoA): turnOf(t, repoA, "fix/other", "0123456789ab")}.read()}.Status()
if err != nil {
t.Fatal(err)
}
if why := st.HeldByTurn["feat/x"]; !strings.Contains(why, repoA) || !strings.Contains(why, "fix/other") {
t.Errorf("held_by_turn = %v, want feat/x held by the merge of fix/other on %s", st.HeldByTurn, repoA)
}
}
// slowBucket wraps a state with a Get that takes delay and counts the values read, the lists and the most Gets in flight.
@@ -3,6 +3,8 @@ package check
import (
"encoding/json"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The shapes of what the build modules' steps answer, as their code writes them (mesh-catalog modules/mesh-build-go
@@ -52,6 +54,8 @@ type GoPackage struct {
Elapsed float64 `json:"elapsed"`
Tests int `json:"tests"`
NoTests bool `json:"no_tests,omitempty"`
// Skipped is how many of the package's tests skipped; an older build module names none.
Skipped int `json:"skipped,omitempty"`
}
// Failure is a test that failed, or a package that failed with no test to name.
@@ -107,4 +111,6 @@ type Digested struct {
WithoutTests []string `json:"without_tests,omitempty"`
// JudgedBy is the program that ran the digest, as Tested.JudgedBy says.
JudgedBy string `json:"judged_by,omitempty"`
// Needs are the packages whose tests need something of the node at this level (issue 890).
Needs []delivery.PackageNeeds `json:"needs,omitempty"`
}
@@ -13,6 +13,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// A try is a pushed head judged by the checker as a pull request's head is, before its pull request is opened (ADR 0501
@@ -24,6 +25,7 @@ import (
const (
TryTool = "checker_try"
TryReadTool = "checker_try_read"
TryEndTool = "checker_try_end"
// TriesState is the checker's state of tries: the bucket mesh-delivery-checker_tries, keys try-<unix nanoseconds>.
TriesState = "tries"
TriesBucket = delivery.CheckerModule + "_" + TriesState
@@ -222,6 +224,52 @@ func ReadTry(store TryStore, id string) (json.RawMessage, error) {
return raw, nil
}
// TryEnder is the tries state as the end tool sees it: one key with its revision, and a write at that revision.
type TryEnder interface {
Entry(key string) (json.RawMessage, uint64, error)
Update(key string, value []byte, revision uint64) error
}
// EndTry ends a try that waits or runs as error, saying why and that it was ended by hand (issue 734): its place is
// free at once, and the next checker_try of its head asks anew. A try that ended already is refused, naming its end.
func EndTry(store TryEnder, id, why string, now time.Time) (Try, error) {
if !tryKeyOK.MatchString(id) {
return Try{}, fmt.Errorf("%q is not a try's id, try-<digits>", id)
}
why = strings.TrimSpace(why)
if why == "" {
return Try{}, errors.New("say why the try is ended: the why is kept in what it says")
}
raw, rev, err := store.Entry(id)
if err != nil {
return Try{}, err
}
if len(raw) == 0 {
return Try{}, fmt.Errorf("no try %s: it was never asked, or it ended more than %s ago and was removed", id, TryKeep)
}
t, err := DecodeTry(raw)
if err != nil {
return Try{}, err
}
if t.Over() {
return Try{}, fmt.Errorf("%s ended already, %s at %s", id, t.State, t.Ended.Format(time.RFC3339))
}
t.State, t.Ended = TryError, now.UTC()
if !t.Started.IsZero() {
t.Took = t.Ended.Sub(t.Started).Round(time.Second).String()
}
t.Said = strings.TrimSpace("ended by hand (" + TryEndTool + "): " + why + " | " + t.Said)
t.Said = strings.TrimSuffix(t.Said, " |")
data, err := encodeTry(t)
if err != nil {
return Try{}, err
}
if err := store.Update(id, data, rev); err != nil {
return Try{}, fmt.Errorf("%s changed while it was ended, read it and end it again: %w", id, err)
}
return t, nil
}
// TryOptions narrow what a try runs; a pull request's check has none.
type TryOptions struct {
Level string
@@ -234,12 +282,14 @@ type Tryer struct {
Tries KV
Judge Judge
Bound time.Duration
Now func() time.Time
Logf func(format string, args ...any)
// NotTaken ends a try whose ask no build node took within it as error (issue 734); none when 0.
NotTaken time.Duration
Now func() time.Time
Logf func(format string, args ...any)
mu sync.Mutex
wg sync.WaitGroup
inflight map[string]bool
inflight map[string]context.CancelFunc
seen map[string]bool // looked at since this process started: a try left running is taken over the first time
}
@@ -267,25 +317,31 @@ func (tr *Tryer) Put(ctx context.Context, key string, value []byte) {
tr.logf("tries.%s holds what is not a try, left: %v", key, err)
return
}
if t.Over() {
return
}
tr.mu.Lock()
if tr.inflight == nil {
tr.inflight, tr.seen = map[string]bool{}, map[string]bool{}
tr.inflight, tr.seen = map[string]context.CancelFunc{}, map[string]bool{}
}
if tr.inflight[key] || (t.State == TryRunning && tr.seen[key]) {
if t.Over() {
// Ended while it ran (checker_try_end): its judging stops, and its asks are no longer waited for.
if stop := tr.inflight[key]; stop != nil {
stop()
}
tr.mu.Unlock()
return
}
if tr.inflight[key] != nil || (t.State == TryRunning && tr.seen[key]) {
tr.mu.Unlock()
return
}
run, stop := context.WithCancel(ctx)
tr.seen[key] = true
tr.inflight[key] = true
tr.inflight[key] = stop
tr.mu.Unlock()
tr.wg.Add(1)
go func() {
defer tr.wg.Done()
defer func() { tr.mu.Lock(); delete(tr.inflight, key); tr.mu.Unlock() }()
tr.run(ctx, key)
defer func() { tr.mu.Lock(); delete(tr.inflight, key); tr.mu.Unlock(); stop() }()
tr.run(run, key)
}()
}
@@ -342,9 +398,13 @@ func (tr *Tryer) run(ctx context.Context, key string) {
Queued: t.Asked,
Progress: &Progress{},
Try: &TryOptions{Level: t.Level, Run: t.Tests}}
out, judged := judgeWithin(ctx, tr.Judge, cs, tr.Bound)
judging := ctx
if tr.NotTaken > 0 {
judging = buildrun.NotTakenBound(ctx, tr.NotTaken)
}
out, judged := judgeWithin(judging, tr.Judge, cs, tr.Bound)
if ctx.Err() != nil {
return // left running: the checker takes it over when it starts
return // left running: the checker takes it over when it starts; or ended by hand, which wrote its end
}
v := out.Verdict
t.Ended = tr.now().UTC()
@@ -355,8 +415,12 @@ func (tr *Tryer) run(ctx context.Context, key string) {
t.Said += " | " + n
}
switch {
case !judged || IsNodesFailure(v):
case !judged || IsNodesFailure(v) || IsLostAsk(v) || IsNotTaken(v):
// Not the change's: its ask was given up or never taken (issue 734), or a part of the mesh was not heard.
t.State = TryError
if IsLostAsk(v) || IsNotTaken(v) {
t.Failed = []string{v.Failed.Words}
}
case v.State == delivery.VerdictPass:
t.State = TryPass
default:
@@ -0,0 +1,78 @@
package check
import (
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Turns holds the gate's merge turns (novox/hq ADR 0548 §3), by repository, as a watch of the gate's turns bucket puts them
// in: the checker claims no check of a repository a turn names, except the check of a waiting delivery's own heads, so an
// approved merge goes before a check not yet started. A bucket not there is read as empty.
type Turns struct {
mu sync.Mutex
byRepo map[string]delivery.Turn
}
// Put takes a value of the turns bucket in, or its delete, and answers whether what the checker holds changed. A value that
// is not a turn is dropped as a delete would be: a turn the checker cannot read holds nothing.
func (t *Turns) Put(key string, value []byte, deleted bool) bool {
repository, err := delivery.ParseTurnsKey(key)
if err != nil {
return false
}
t.mu.Lock()
defer t.mu.Unlock()
if t.byRepo == nil {
t.byRepo = map[string]delivery.Turn{}
}
cur, had := t.byRepo[repository]
if !deleted {
if turn, err := delivery.DecodeTurn(value); err == nil {
t.byRepo[repository] = turn
return !had || !delivery.SameTurn(cur, turn)
}
}
delete(t.byRepo, repository)
return had
}
// Holding names the first of the repositories (by name) a turn holds off the check of the heads, and the delivery that
// waits first in it; false when none does.
func (t *Turns) Holding(repositories []string, heads string) (string, delivery.TurnWait, bool) {
if t == nil {
return "", delivery.TurnWait{}, false
}
sorted := make([]string, 0, len(repositories))
for _, r := range repositories {
sorted = append(sorted, strings.ToLower(r))
}
sort.Strings(sorted)
t.mu.Lock()
defer t.mu.Unlock()
for _, r := range sorted {
turn, ok := t.byRepo[r]
if !ok || !delivery.TurnHolds(turn, heads) {
continue
}
for _, w := range turn.Waiting {
if w.Heads != heads {
return r, w, true
}
}
}
return "", delivery.TurnWait{}, false
}
// All is every turn held now, by repository (a copy).
func (t *Turns) All() map[string]delivery.Turn {
t.mu.Lock()
defer t.mu.Unlock()
out := make(map[string]delivery.Turn, len(t.byRepo))
for r, turn := range t.byRepo {
out[r] = turn
}
return out
}
@@ -93,11 +93,11 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
t.Errorf("module = %q", m.Module)
}
// delivery.CheckerKeptHistory, CheckerOwedHistory, PreviewsHistory, CheckerVerdictsHistory; the names follow the SDK's buckets.
if want := []state{{Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
if want := []state{{Name: "hearing", History: 1}, {Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-assigner.compositions", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates",
"mesh-delivery-assigner.compositions", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
@@ -124,8 +124,8 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
t.Errorf("streams = %+v, want %+v", m.Streams, want)
}
// The read tools (issue 1686) are served by the tools bundle, not by the checker's process.
// The try's two tools (ADR 0501 §5) beside them: mesh-controller's try calls them.
if want := []string{"checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
// The try's tools (ADR 0501 §5; the end by hand, issue 734) beside them: mesh-controller's try calls them.
if want := []string{"checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
for _, k := range []string{"logs", "emits", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
@@ -10,6 +10,7 @@
{ "name": "checks", "events": ["preview.**", "owed.**"] }
],
"state": [
{ "name": "hearing", "history": 1 },
{ "name": "kept", "history": 1 },
{ "name": "owed", "history": 5 },
{ "name": "previews", "history": 5 },
@@ -25,6 +26,7 @@
"mesh-delivery-assigner.compositions",
"mesh-delivery-builder.builds",
"mesh-delivery-gate.source",
"mesh-delivery-gate.turns",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
@@ -41,6 +43,7 @@
"checker_pull",
"checker_status",
"checker_try",
"checker_try_end",
"checker_try_read",
"checker_verdicts"
],
@@ -0,0 +1,111 @@
// Command mesh-delivery-indexer-tools serves the indexer's doing tool (novox/hq issue 1945): the runtime launches it for a
// call. indexer_take_in writes one value of the indexer's takeins state and waits for the indexer's answer there. The
// indexer is the other program: it serves no request subject (ADR 0437 §4). The read tools of the indexer's states are
// mesh-delivery-view's (indexer_module, indexer_modules, indexer_candidate, indexer_candidates).
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"mesh-delivery-indexer/internal/index"
)
// asker writes a takeins value and waits for the indexer's answer to it.
type asker func(key string, value []byte) (index.TakeInAsk, error)
// Tools are the indexer's tools.
func Tools(ask asker, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{
{Name: index.TakeInTool,
Description: "Take a registered module in now: the indexer reads its manifest from the repository and commit the controller's " +
"build records register it from (at the head the gate's source names, when it names one), whatever it tried before, and " +
"writes it to its modules state as a merge would. For a module registered from outside the indexed repositories whose " +
"repository the forge could not read (a 404 before the forge's account could read it). Answers stored, from which " +
"repository, commit and path, or why not (held already, no registration, the forge's error, no manifest there). " +
"The indexer reads such a module again by itself on the next source revision and merged message; this asks now. " +
"indexer_module (mesh-delivery-view) reads the result.",
Input: map[string]any{"type": "object", "required": []string{"module"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name, as the catalogue registers it"},
"why": map[string]any{"type": "string", "description": "why it is taken in now, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := index.NewTakeIn(str(args, "module"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
a, err := ask(key, value)
if err != nil {
return nil, err
}
return a.Result, nil
}},
}
}
// askTakeIn writes the takeins value at key and waits, on a watch of the key, at most bound for the indexer's answer.
func askTakeIn(s stdio.KeptState, bound time.Duration) asker {
return func(key string, value []byte) (index.TakeInAsk, error) {
answered := make(chan index.TakeInAsk, 1)
var rev uint64
var mu sync.Mutex
take := func(revision uint64, v []byte) bool {
mu.Lock()
mine := rev
mu.Unlock()
if mine == 0 || revision <= mine {
return false
}
a, err := index.DecodeTakeIn(v)
if err != nil || a.Answered.IsZero() || a.Result == nil {
return false
}
select {
case answered <- a:
default:
}
return true
}
if err := s.Watch(key, func(c stdio.StateChange) error {
if c.Op == "put" {
take(c.Revision, c.Value)
}
return nil
}); err != nil {
return index.TakeInAsk{}, fmt.Errorf("watching %s.%s: %w", index.TakeInsState, key, err)
}
r, err := s.Put(key, json.RawMessage(value))
if err != nil {
return index.TakeInAsk{}, fmt.Errorf("the ask could not be written to %s: %w", index.TakeInsBucket, err)
}
mu.Lock()
rev = r
mu.Unlock()
// The watch may have heard the answer before the revision was known: read it once too.
if e, err := s.Get(key); err == nil && e != nil {
take(e.Revision, e.Value)
}
wait, cancel := context.WithTimeout(context.Background(), bound) // a context's bound, no timer (ADR 0437 §8)
defer cancel()
select {
case a := <-answered:
return a, nil
case <-wait.Done():
return index.TakeInAsk{}, fmt.Errorf("the indexer did not answer within %s: the take-in is still running (a forge read can take 30 s); its answer will be in %s.%s", bound, index.TakeInsBucket, key)
}
}
}
func main() {
if err := stdio.Serve("", Tools(askTakeIn(stdio.State(index.TakeInsState), 25*time.Second), time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-indexer-tools] %v\n", err)
os.Exit(1)
}
}
@@ -1,7 +1,8 @@
// Command mesh-delivery-indexer is delivery's indexer (novox/hq ADR 0437 §4, §5, issue 1238): it hears the forge
// bridge's messages on the stream FORGE and keeps the candidate groups (candidates.<group>): the pull requests that share
// a branch, their heads and the modules each touches. It reads the controller's builds to know which module a changed
// directory is. It serves no request subject and holds no timer: every reaction is a message or a watch (ADR 0437 §8).
// directory is. It serves no request subject and holds no timer: every reaction is a message or a watch (ADR 0437 §8). Its
// tool indexer_take_in (the program mesh-delivery-indexer-tools) asks through its state takeins, which it watches.
//
// Its build of 2026-10-24 00:37 named a program whose push never reached the artifact store, and a retry sends that
// same build again; this paragraph was added only to give the indexer a new build (issue 1933).
@@ -266,11 +267,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
}
errs := make(chan error, 1)
var buildsWatch jetstream.KeyWatcher
if builds != nil {
if w, err := settle(ctx, builds, each); err != nil {
process.NotWatched(logger.Printf, delivery.BuildsBucket, err, "a module whose key names no directory is not given one until the indexer starts again")
} else {
go func() { errs <- follow(ctx, w, each) }()
buildsWatch = w
}
}
@@ -285,7 +287,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
defer calls.Close()
ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Builds: known, Logf: logger.Printf,
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call)}
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call)}
// The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key
// main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does
// not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate.
@@ -336,6 +338,39 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := ix.RecomputeAtStart(ctx); err != nil {
logger.Printf("computing the new-form fingerprints at start: %v", err)
}
// A module registered in the catalogue from its own repository gets its manifest indexed when the controller's records
// name it and the bucket holds none (registered.go); the records that follow are heard below.
if err := ix.IndexRegisteredAll(ctx); err != nil {
logger.Printf("indexing the registered modules at start: %v", err)
}
if buildsWatch != nil {
go func() {
errs <- follow(ctx, buildsWatch, func(e jetstream.KeyValueEntry) {
each(e)
if !opOf(e) {
return
}
var r delivery.BuildRecord
if json.Unmarshal(e.Value(), &r) != nil || r.Module == "" {
return
}
if err := ix.IndexRegisteredModule(ctx, r.Module); err != nil {
logger.Printf("the registered module %s is not indexed: %v", r.Module, err)
}
})
}()
}
// A person takes a registered module in now with indexer_take_in, which writes the takeins state (issue 1945): watched,
// as every other input. A module the forge could not read is also read again by itself on the next source revision and
// merged message (index.RetryUnread).
if kv := process.OptionalBucket(ctx, js, index.TakeInsBucket, "indexer_take_in is not answered until the indexer starts again; a module the forge could not read is still read again by itself", logger.Printf); kv != nil {
eachTakeIn := func(e jetstream.KeyValueEntry) { takenIn(ctx, ix, kv, e, logger.Printf) }
if tw, err := settle(ctx, kv, eachTakeIn); err != nil {
process.NotWatched(logger.Printf, index.TakeInsBucket, err, "indexer_take_in is not answered until the indexer starts again")
} else {
go func() { errs <- follow(ctx, tw, eachTakeIn) }()
}
}
// The modules the trunk no longer holds go from the bucket (issue 1900); a merge does the same from its own message.
if err := ix.Reconcile(ctx); err != nil {
logger.Printf("dropping the modules the trunk does not hold at start: %v", err)
@@ -2,11 +2,16 @@ package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"path"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-indexer/internal/index"
)
// commitTreesTool is the forge's tool for a commit's trees (mesh-catalog modules/gitea): with `paths`, by module directory
@@ -93,3 +98,75 @@ func containsVia(call toolCaller) func(ctx context.Context, repository, commit,
return *a.Contains, nil
}
}
// getFileTool is the forge's tool for one file at a ref (mesh-catalog modules/gitea gitea_get_file): arguments owner,
// repo, path and ref, answer {file: {content}}; a path it does not have fails with a 404 in its words.
const getFileTool = "gitea.gitea_get_file"
// lockFiles are read beside a manifest, as the forge bridge reads them (bridge.LockFiles).
var lockFiles = []string{"go.sum", "package-lock.json"}
// readVia reads a repository at a commit as the forge bridge does for a merged message: the trees by gitea_commit_trees,
// then the module.json and lock files of the one directory dir by gitea_get_file (a repository can hold hundreds). A manifest the
// forge does not have is an error; a directory with no module.json in the trees is answered with no manifest; a lock file it does not have is left out.
func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir string) (index.RepoRead, error) {
return func(_ context.Context, repository, sha, dir string) (index.RepoRead, error) {
owner, repo, ok := strings.Cut(delivery.RepositoryName(repository), "/")
if !ok {
return index.RepoRead{}, fmt.Errorf("%q is not <owner>/<name>", repository)
}
raw, err := call(commitTreesTool, map[string]any{"owner": owner, "repo": repo, "sha": sha})
if err != nil {
return index.RepoRead{}, err
}
var a struct {
Root string `json:"root"`
Trees map[string]string `json:"trees"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return index.RepoRead{}, fmt.Errorf("the answer of %s is not {root, trees}: %w", commitTreesTool, err)
}
out := index.RepoRead{Root: a.Root, Trees: a.Trees, Manifests: map[string]string{}, Locks: map[string]map[string]string{}}
file := func(dir, name string) (string, error) {
p := name
if dir != "" {
p = dir + "/" + name
}
raw, err := call(getFileTool, map[string]string{"owner": owner, "repo": repo, "path": p, "ref": sha})
if err != nil {
return "", err
}
var f struct {
File *struct {
Content *string `json:"content"`
} `json:"file"`
}
if err := json.Unmarshal(raw, &f); err != nil || f.File == nil || f.File.Content == nil {
return "", fmt.Errorf("the answer of %s is not {file: {content}}: %v", getFileTool, err)
}
return *f.File.Content, nil
}
if _, held := a.Trees[dir]; held {
m, err := file(dir, "module.json")
if err != nil {
return index.RepoRead{}, fmt.Errorf("%s: %w", path.Join(dir, "module.json"), err)
}
out.Manifests[dir] = m
for _, name := range lockFiles {
c, err := file(dir, name)
if err != nil {
if strings.Contains(err.Error(), ": 404 ") {
continue
}
return index.RepoRead{}, fmt.Errorf("%s: %w", path.Join(dir, name), err)
}
sum := sha256.Sum256([]byte(c))
if out.Locks[dir] == nil {
out.Locks[dir] = map[string]string{}
}
out.Locks[dir][name] = hex.EncodeToString(sum[:])
}
}
return out, nil
}
}
@@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"slices"
"strings"
"testing"
@@ -50,3 +51,33 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
t.Error("the trunk asked for paths")
}
}
func TestReadViaReadsManifestsAndLocks(t *testing.T) {
var files []string
call := func(key string, body any) (json.RawMessage, error) {
switch key {
case commitTreesTool:
return json.RawMessage(`{"root":"r0","trees":{"":"t1","modules/other":"t2"}}`), nil
case getFileTool:
files = append(files, body.(map[string]string)["path"])
switch body.(map[string]string)["path"] {
case "module.json":
return json.RawMessage(`{"file":{"content":"{\"module\":\"snake\"}"}}`), nil
case "go.sum":
return json.RawMessage(`{"file":{"content":"x"}}`), nil
}
return nil, errors.New("Gitea API /x: 404 not found")
}
return nil, errors.New("unexpected " + key)
}
got, err := readVia(call)(context.Background(), "http://novox.internal:20000/jschoubben/snake.git", "abc", "")
if err != nil {
t.Fatal(err)
}
if len(got.Manifests) != 1 || len(files) != 3 { // the one directory asked: its manifest and two lock files, never the others
t.Fatalf("manifests = %v, files asked = %v", got.Manifests, files)
}
if got.Root != "r0" || got.Manifests[""] != `{"module":"snake"}` || len(got.Locks[""]) != 1 || got.Locks[""]["go.sum"] == "" {
t.Fatalf("read = %+v", got)
}
}
@@ -0,0 +1,44 @@
package main
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-indexer/internal/index"
)
// takerIn is the part of the indexer a takeins value asks through.
type takerIn interface {
TakeIn(ctx context.Context, module string) index.TakenIn
}
// takenIn answers one value of the takeins state (the tool indexer_take_in, issue 1945): a value not yet answered takes the
// module in now and is written back, by compare-and-set at its revision, with what the indexer stored or why not. A value
// answered already (this indexer's own write) or deleted is left.
func takenIn(ctx context.Context, ix takerIn, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
a, err := index.DecodeTakeIn(e.Value())
if err != nil {
logf("%s.%s holds what is not a take-in ask, left: %v", index.TakeInsState, e.Key(), err)
return
}
if !a.Answered.IsZero() {
return
}
r := ix.TakeIn(ctx, a.Module)
a.Result, a.Answered = &r, time.Now().UTC()
logf("%s: a take-in asked by %s (%s): %s", a.Module, a.By, a.Why, r.Answer)
data, err := json.Marshal(a)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", index.TakeInsState, e.Key(), err)
}
}
@@ -2,6 +2,8 @@ package index
import (
"encoding/json"
"maps"
"slices"
"strings"
"sync"
@@ -88,3 +90,102 @@ func (b *Builds) PathOf(module, repository string) (string, bool) {
}
return pathOf(strings.Trim(best.Path, "/")), true
}
// Registration is a module registered in the catalogue from its own repository, as the controller's newest outcome record
// of it names it: the repository, the directory of its manifest there ("" or "." for the root) and the commit it was
// built at.
type Registration struct {
Module, Repository, Path, Commit, Ref string
}
// registered is whether the record is an outcome that did not fail: what registers a module. Unlike IsTrunkBuild it
// asks for no recorded source inputs, which no build records since ADR 0530 clause 1, and for no particular branch: a
// repository's trunk is named main, master, or what its owner chose (the registered photos, de-spiegel and link2pay were
// built from nox-mesh, and every record of them names that ref).
func registered(r delivery.BuildRecord) bool {
return r.State == "" && r.Failed == ""
}
// RegistrationsOf are the places a module was registered from, by its outcome records: the newest record of each
// repository and directory (a module's name was once another repository's, as photos was a directory of the catalogue's).
// Records of a trunk's ref (main, master, trunk, none) come before those of another branch, then the newer first, so the
// first that holds the module's manifest is the one indexed.
func (b *Builds) RegistrationsOf(module string) []Registration {
type place struct{ repository, path string }
byPlace := map[place][]delivery.BuildRecord{}
b.mu.Lock()
for _, r := range b.records {
if r.Module == module && registered(r) {
k := place{delivery.RepositoryName(r.Repository), pathOf(strings.Trim(r.Path, "/"))}
byPlace[k] = append(byPlace[k], r)
}
}
b.mu.Unlock()
// Within one place the record is that of the ref a registration comes from, never a newer pull request's or try's: a
// trunk's ref (main, master, trunk, none), else the ref most of the place's records name (the registered photos, de-spiegel
// and link2pay were built from nox-mesh), the newer of two equal; then its newest record.
best := map[place]delivery.BuildRecord{}
for k, recs := range byPlace {
count, latest := map[string]int{}, map[string]delivery.BuildRecord{}
for _, r := range recs {
count[r.Ref]++
if cur, ok := latest[r.Ref]; !ok || newer(r, cur) {
latest[r.Ref] = r
}
}
var pick string
first := true
for ref, rec := range latest {
switch {
case first:
case trunkLike[ref] != trunkLike[pick]:
if !trunkLike[ref] {
continue
}
case count[ref] != count[pick]:
if count[ref] < count[pick] {
continue
}
case !newer(rec, latest[pick]):
continue
}
pick, first = ref, false
}
best[k] = latest[pick]
}
recs := slices.Collect(maps.Values(best))
slices.SortFunc(recs, func(x, y delivery.BuildRecord) int {
switch {
case trunkLike[x.Ref] != trunkLike[y.Ref]:
if trunkLike[x.Ref] {
return -1
}
return 1
case newer(x, y):
return -1
case newer(y, x):
return 1
}
return strings.Compare(x.Repository+x.Path, y.Repository+y.Path)
})
out := make([]Registration, 0, len(recs))
for _, r := range recs {
out = append(out, Registration{Module: r.Module, Repository: r.Repository, Path: r.Path, Commit: r.Commit, Ref: r.Ref})
}
return out
}
// Registered is the registrations of every module a record names, by module name.
func (b *Builds) Registered() []Registration {
b.mu.Lock()
names := map[string]bool{}
for _, r := range b.records {
names[r.Module] = true
}
b.mu.Unlock()
var out []Registration
for _, name := range slices.Sorted(maps.Keys(names)) {
out = append(out, b.RegistrationsOf(name)...)
}
return out
}
@@ -8,6 +8,7 @@ import (
"slices"
"strings"
"sync"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -53,6 +54,9 @@ type Indexer struct {
// has none.
Named func(ctx context.Context, repository, sha string, paths map[string][]string) (map[string]map[string]string, error)
Trunk func(ctx context.Context, repository, sha string) ([]string, error)
// Read asks the forge for a repository's trees at a commit and the manifest and lock-file hashes of the one module directory dir (the
// root spelled ""), as the forge bridge reads them for a merged message (registered.go). Nil indexes no registered module.
Read func(ctx context.Context, repository, sha, dir string) (RepoRead, error)
// Contains says whether a commit is the second or an ancestor of it (gitea.gitea_contains).
Contains func(ctx context.Context, repository, commit, in string) (bool, error)
Logf func(format string, args ...any)
@@ -61,6 +65,11 @@ type Indexer struct {
source sourceNow
previews map[string]delivery.Preview // by previews key (previews.go)
toolchains map[string]map[string]delivery.Toolchain // language, then node (toolchains.go)
tried map[string]bool // repository, commit and directory a registered module's manifest was tried at (registered.go)
unread map[string]bool // registered modules whose last read the forge could not answer, read again by RetryUnread
retrying atomic.Bool // a RetryUnread runs in the background (kickRetry)
retryAgain atomic.Bool // an event asked for one more RetryUnread after the running one
retryWG sync.WaitGroup // the background RetryUnread, for tests to wait on
recompute sync.Mutex // one Recompute at a time
}
@@ -85,6 +94,7 @@ func (ix *Indexer) Apply(ctx context.Context, f delivery.Forge) error {
if err := ix.applyMerged(ctx, f); err != nil {
return err
}
ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945)
}
var digests map[string]string
var held map[string]delivery.Module
@@ -25,6 +25,13 @@ import (
// A value the module already holds but for its commit is not written again: its commit stays the one its fingerprint first
// appeared at (writeOverlay), so a merge writes only the modules it changes.
func (ix *Indexer) applyMerged(ctx context.Context, f delivery.Forge) error {
return ix.applyManifests(ctx, f, true)
}
// applyManifests is applyMerged; drop says whether the modules of the repository the message holds no manifest for are
// deleted. A message that names only some of a repository's modules (a registered module read on its own, registered.go)
// drops none.
func (ix *Indexer) applyManifests(ctx context.Context, f delivery.Forge, drop bool) error {
digests, err := ix.keepManifests(ctx, f)
if err != nil {
return err
@@ -84,7 +91,7 @@ func (ix *Indexer) applyMerged(ctx context.Context, f delivery.Forge) error {
}
// A module of this repository the merge commit holds no module.json for is gone from the trunk, and so is its key
// (issue 1900). A message that carries no manifests (an older writer) or one that cannot be read whole says nothing.
if len(f.Manifests) > 0 && every {
if drop && len(f.Manifests) > 0 && every {
if err := ix.dropRemoved(ctx, f.Repository, held, func(key string, _ delivery.Module) bool { return named[key] }); err != nil {
return err
}
@@ -0,0 +1,275 @@
package index
import (
"context"
"fmt"
"maps"
"slices"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A module registered in the catalogue from its own repository (the operator registers it with the repository and the
// path of its manifest) had no manifest here: the indexer learns a repository only from a merged message, and no merge
// followed the registration. The controller's outcome records name each registered module's repository, directory and
// commit (builds.go Registered), so the indexer reads that module's manifest at the repository's head (the gate's source names it) or, with none named, at the
// registered commit, and writes the same keys a merge writes. A later merge to the repository's trunk reaches the module through the forge bridge's merged
// message as for every other repository.
// RepoRead is a repository at a commit as the forge bridge reads it for a merged message: the root tree, the tree of each
// directory holding a module.json (the root spelled ""), their manifests and the hex sha256 of the lock files beside them.
type RepoRead struct {
Root string
Trees map[string]string
Manifests map[string]string
Locks map[string]map[string]string
}
// IndexRegistered writes the module of a registration as a merge of its repository would, when the modules bucket holds
// none of that name. Its commit is the head the gate's source names for the repository, else the registration's. The
// repository's other modules are neither written nor dropped: they are the merges'. A manifest that is not at the path or
// names another module is said and left. A forge that could not say (a timeout, or a 404 while the forge's account could
// not read the repository) is an error, and the module is kept as unread: it is read again at the next source revision
// and the next merged message (RetryUnread), not only when the indexer starts again (issue 1945).
func (ix *Indexer) IndexRegistered(ctx context.Context, r Registration) error {
_, err := ix.indexRegistered(ctx, r, false)
return err
}
// taken is what indexRegistered did with one registration: the module written, held already, or the reason it was not.
type taken struct {
stored, held bool
repository, commit, dir string
why string
}
func (ix *Indexer) indexRegistered(ctx context.Context, r Registration, force bool) (taken, error) {
if ix.Modules == nil || ix.Read == nil {
return taken{why: "the indexer reads no forge or writes no modules"}, nil
}
key, err := delivery.ModulesKey(r.Module)
if err != nil {
ix.logf("%s: the registered module has no usable key, skipped: %v", r.Module, err)
return taken{why: fmt.Sprintf("the module has no usable key: %v", err)}, nil
}
if _, _, found, err := ix.Modules.Get(ctx, key); err != nil {
return taken{}, fmt.Errorf("modules.%s: %w", key, err)
} else if found {
ix.markRead(r.Module)
return taken{held: true, why: "the modules state holds it already: a merge to its repository writes it from now on"}, nil
}
repository := delivery.RepositoryName(r.Repository)
commit := r.Commit
dir := strings.Trim(r.Path, "/")
if dir == delivery.RootPath {
dir = ""
}
// The head the gate's source names for the repository, when it names one, else the registration's commit. A module a
// merge removed is absent at the head and is said and left.
ix.mu.Lock()
for repo, head := range ix.source.heads {
if delivery.SameRepository(repo, r.Repository) && head != "" {
commit = head
}
}
ix.mu.Unlock()
t := taken{repository: repository, commit: commit, dir: dir}
if repository == "" || commit == "" {
ix.logf("%s: registered with no repository or commit, so its manifest is not read", r.Module)
t.why = "the controller's record names no repository or commit"
return t, nil
}
// One try for each repository, commit and directory: a repository with no manifest where it was registered is said once,
// not at every record of the controller's builds. A person's take-in (force) reads again.
tried := repository + "@" + commit + ":" + dir
ix.mu.Lock()
if ix.tried[tried] && !force {
ix.mu.Unlock()
t.why = "tried already at this commit and directory"
return t, nil
}
if ix.tried == nil {
ix.tried = map[string]bool{}
}
ix.tried[tried] = true
ix.mu.Unlock()
read, err := ix.Read(ctx, repository, commit, dir)
if err != nil {
ix.mu.Lock()
delete(ix.tried, tried) // a forge that could not say is asked again
if ix.unread == nil {
ix.unread = map[string]bool{}
}
ix.unread[r.Module] = true // and read again on the next source revision or merged message (RetryUnread)
ix.mu.Unlock()
return t, fmt.Errorf("%s: reading %s at %s: %w", r.Module, repository, commit, err)
}
ix.markRead(r.Module)
raw, ok := read.Manifests[dir]
if !ok || read.Trees[dir] == "" {
ix.logf("%s: %s at %s holds no module.json at %q, so the registered module is not indexed", r.Module, repository, commit, dir)
t.why = fmt.Sprintf("%s at %s holds no module.json at %q", repository, commit, dir)
return t, nil
}
if name := manifestModule(raw); name != r.Module {
ix.logf("%s: the manifest at %q of %s at %s names the module %q, so the registered module is not indexed", r.Module, dir, repository, commit, name)
t.why = fmt.Sprintf("the manifest at %q of %s at %s names the module %q", dir, repository, commit, name)
return t, nil
}
f := delivery.Forge{V: delivery.Version, Kind: delivery.ForgeMerged, Repository: repository, Merge: commit, Root: read.Root,
Trees: map[string]string{dir: read.Trees[dir]},
Manifests: map[string]string{dir: raw}}
if l := read.Locks[dir]; len(l) > 0 {
f.Locks = map[string]map[string]string{dir: l}
}
if err := ix.applyManifests(ctx, f, false); err != nil {
return t, err
}
if _, _, found, err := ix.Modules.Get(ctx, key); err != nil {
return t, fmt.Errorf("modules.%s: %w", key, err)
} else if !found {
t.why = "its manifest was read but the modules state holds no value of it (the indexer's log says why)"
return t, nil
}
t.stored = true
ix.logf("%s: the registered module is indexed from %s at %s", r.Module, repository, commit)
return t, nil
}
// markRead takes a module out of the unread ones: the forge answered for it.
func (ix *Indexer) markRead(module string) {
ix.mu.Lock()
delete(ix.unread, module)
ix.mu.Unlock()
}
// RetryUnread reads again every registered module whose last read the forge could not answer (a timeout, or a 404 while
// the forge's account could not read the repository; issue 1945). It is called on the events that already reach the
// indexer, a new source revision and a merged message, so a repository that became readable is taken in without a hand
// act and without a timer of its own (ADR 0437 §8). What still cannot be read is said and kept for the next.
func (ix *Indexer) RetryUnread(ctx context.Context) {
ix.mu.Lock()
names := slices.Sorted(maps.Keys(ix.unread))
ix.mu.Unlock()
for _, module := range names {
if err := ix.IndexRegisteredModule(ctx, module); err != nil {
ix.logf("the registered module %s is still not indexed: %v", module, err)
}
}
}
// kickRetry runs RetryUnread in one background goroutine, so the message or revision that asked it is acknowledged without
// waiting on the forge (each unread module costs up to two forge calls of 30 s each). A kick while one runs makes it run
// once more when it ends; never two at once.
func (ix *Indexer) kickRetry(ctx context.Context) {
ix.mu.Lock()
none := len(ix.unread) == 0
ix.mu.Unlock()
if none {
return
}
ix.retryAgain.Store(true)
if !ix.retrying.CompareAndSwap(false, true) {
return
}
ix.retryWG.Add(1)
go func() {
defer ix.retryWG.Done()
for ix.retryAgain.Swap(false) {
ix.RetryUnread(ctx)
}
ix.retrying.Store(false)
// A kick between the last Swap and the Store found it running and returned: run for it.
if ix.retryAgain.Load() {
ix.kickRetry(ctx)
}
}()
}
// TakenIn is the answer of a take-in (the tool indexer_take_in): whether the module is now in the modules state, from
// where, and otherwise why not.
type TakenIn struct {
Module string `json:"module"`
Stored bool `json:"stored"`
Held bool `json:"held,omitempty"`
Repository string `json:"repository,omitempty"`
Commit string `json:"commit,omitempty"`
Path string `json:"path,omitempty"`
Answer string `json:"answer"`
}
// TakeIn reads a registered module's manifest now, whatever was tried before, and writes it as a merge would (the tool
// indexer_take_in, issue 1945). A module the modules state holds is answered so and left; one with no registration in the
// controller's records, or whose every registration could not be read or holds no manifest of it, is answered with why.
func (ix *Indexer) TakeIn(ctx context.Context, module string) TakenIn {
out := TakenIn{Module: module}
if ix.Builds == nil {
out.Answer = "the indexer reads no controller's build records, so it knows no registration"
return out
}
regs := ix.Builds.RegistrationsOf(module)
if len(regs) == 0 {
if key, err := delivery.ModulesKey(module); err == nil && ix.Modules != nil {
if _, _, found, err := ix.Modules.Get(ctx, key); err == nil && found {
out.Held, out.Answer = true, "the modules state holds it already: a merge to its repository writes it from now on"
return out
}
}
out.Answer = "the controller's build records name no registration of " + module + ": a module from outside the indexed repositories is known only from them, and one inside them from a merge"
return out
}
var whys []string
for _, r := range regs {
t, err := ix.indexRegistered(ctx, r, true)
out.Repository, out.Commit, out.Path = t.repository, t.commit, pathOf(t.dir)
switch {
case err != nil:
whys = append(whys, err.Error())
continue
case t.held:
out.Held, out.Answer = true, t.why
out.Repository, out.Commit, out.Path = "", "", ""
return out
case t.stored:
out.Stored = true
out.Answer = fmt.Sprintf("written to the modules state from %s at %s", t.repository, t.commit)
return out
}
whys = append(whys, t.why)
}
out.Answer = "not taken in: " + strings.Join(whys, "; ")
return out
}
// IndexRegisteredModule indexes the registration the controller's records name for a module, when there is one.
func (ix *Indexer) IndexRegisteredModule(ctx context.Context, module string) error {
if ix.Builds == nil {
return nil
}
var first error
for _, r := range ix.Builds.RegistrationsOf(module) {
if err := ix.IndexRegistered(ctx, r); err != nil && first == nil {
first = err
}
}
return first
}
// IndexRegisteredAll indexes every registered module the modules bucket does not hold. A module that cannot be read is
// said and the others go on; the first error is answered.
func (ix *Indexer) IndexRegisteredAll(ctx context.Context) error {
if ix.Builds == nil {
return nil
}
var first error
for _, r := range ix.Builds.Registered() {
if err := ix.IndexRegistered(ctx, r); err != nil {
ix.logf("the registered module %s is not indexed: %v", r.Module, err)
if first == nil {
first = err
}
}
}
return first
}
@@ -0,0 +1,258 @@
package index
import (
"context"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func TestARegisteredModuleIsIndexedFromItsRepository(t *testing.T) { // the grants part's gaps: modules registered from their own repositories had no manifest here
const forge = "http://novox.internal:20000/jschoubben/snake.git"
snake := `{"module":"snake","version":"1"}`
for _, tc := range []struct {
name string
path string
read RepoRead
err error
ref string // the build record's ref
heads map[string]string // the gate source's heads
wantRead bool // the forge is read
held bool // modules.snake exists already
other bool // modules.other of the same repository exists
want bool // modules.snake is written
}{
{wantRead: true, name: "a module at the root of its repository", path: "", want: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a root spelled as a dot", path: ".", want: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a module in a directory", path: "modules/snake", want: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"modules/snake": "t1"}, Manifests: map[string]string{"modules/snake": snake}}},
{wantRead: true, name: "another module of the repository is neither written nor dropped", path: "", want: true, other: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{name: "a module the bucket holds is left", path: "", held: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "no manifest at the path", path: "modules/x",
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a manifest of another module", path: "",
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": `{"module":"other","version":"1"}`}}},
{wantRead: true, name: "a repository whose default branch is master", path: "", ref: "master", want: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a repository built from a branch of its own (nox-mesh)", path: "", ref: "nox-mesh", want: true,
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a repository the source names, the module unheld and present at the head", path: "", want: true, heads: map[string]string{"jschoubben/snake": h2},
read: RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}},
{wantRead: true, name: "a forge that cannot say", path: "", err: errors.New("down")},
} {
t.Run(tc.name, func(t *testing.T) {
ix, mods := modulesIndexer(t, rec(t, "build-1", "snake", forge, tc.path, tc.ref, "", "2026-10-19T09:00:00Z", ""))
if tc.held {
if err := mods.Create(context.Background(), "snake", mustModule(t, delivery.Module{Repository: "jschoubben/snake", Commit: h1, Path: ".", None2: "x"})); err != nil {
t.Fatal(err)
}
}
if tc.other {
if err := mods.Create(context.Background(), "other", mustModule(t, delivery.Module{Repository: "jschoubben/snake", Commit: h1, Path: "modules/other", None2: "x"})); err != nil {
t.Fatal(err)
}
}
ix.Manifests = newMem()
var asked struct {
repo, sha, dir string
n int
}
ix.Read = func(_ context.Context, repo, sha, dir string) (RepoRead, error) {
asked.repo, asked.sha, asked.dir = repo, sha, dir
asked.n++
return tc.read, tc.err
}
if tc.heads != nil {
ix.source = sourceNow{rev: 1, heads: tc.heads}
}
err := ix.IndexRegisteredAll(context.Background())
if (err != nil) != (tc.err != nil) {
t.Fatalf("error = %v", err)
}
wantCommit := "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" // the record's
if tc.heads != nil {
wantCommit = h2 // the source's head
}
if tc.wantRead && asked.sha != wantCommit {
t.Errorf("the forge was asked at %s, want %s", asked.sha, wantCommit)
}
wantReads := 1
if !tc.wantRead {
wantReads = 0
}
if tc.wantRead && asked.dir != dirOf(tc.path) {
t.Errorf("the forge was asked for directory %q", asked.dir)
}
if asked.n != wantReads {
t.Errorf("forge reads = %d, want %d", asked.n, wantReads)
}
m, ok := moduleOf(t, mods, "snake")
if tc.want {
if !ok || m.Manifest == "" || m.Repository != "jschoubben/snake" || m.Commit != wantCommit || m.Path != pathOf(dirOf(tc.path)) {
t.Fatalf("modules.snake = %+v, found %v", m, ok)
}
if len(ix.Manifests.(*memStore).vals) != 1 {
t.Errorf("manifests written = %d, want 1", len(ix.Manifests.(*memStore).vals))
}
if asked.repo != "jschoubben/snake" {
t.Errorf("the forge was asked for %q", asked.repo)
}
} else if !tc.held && ok {
t.Fatalf("modules.snake written: %+v", m)
}
if _, ok := moduleOf(t, mods, "other"); ok != tc.other {
t.Errorf("modules.other held = %v, want %v", ok, tc.other)
}
})
}
}
func mustModule(t *testing.T, m delivery.Module) []byte {
t.Helper()
b, err := delivery.EncodeModule(withV(m))
if err != nil {
t.Fatal(err)
}
return b
}
func dirOf(p string) string {
if p == "." {
return ""
}
return p
}
func TestARegistrationOfAnotherRepositoryDoesNotHideTheRealOne(t *testing.T) { // photos: its newest trunk record was a directory of the catalogue's, then novox/photos at nox-mesh
const catalogue, own = "http://novox.internal:20000/novox/mesh-catalog.git", "http://novox.internal:20000/novox/Link2Pay.git"
ix, mods := modulesIndexer(t,
rec(t, "build-1", "link2pay", own, "", "nox-mesh", "", "2026-10-06T23:02:00Z", ""),
rec(t, "build-2", "link2pay", catalogue, "modules/link2pay", "main", "", "2026-10-04T16:02:00Z", ""),
rec(t, "build-3", "link2pay", catalogue, "modules/link2pay", "main", "", "2026-10-03T16:02:00Z", ""))
ix.Manifests = newMem()
var asked []string
ix.Read = func(_ context.Context, repo, _, dir string) (RepoRead, error) {
asked = append(asked, repo+":"+dir)
if repo == "novox/Link2Pay" {
return RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": `{"module":"link2pay","version":"1"}`}}, nil
}
return RepoRead{Root: "r0"}, nil
}
if err := ix.IndexRegisteredAll(context.Background()); err != nil {
t.Fatal(err)
}
m, ok := moduleOf(t, mods, "link2pay")
if !ok || m.Repository != "novox/Link2Pay" {
t.Fatalf("modules.link2pay = %+v, found %v", m, ok)
}
if len(asked) != 2 || asked[0] != "novox/mesh-catalog:modules/link2pay" || asked[1] != "novox/Link2Pay:." && asked[1] != "novox/Link2Pay:" {
t.Errorf("the forge was asked %v: want the catalogue's trunk record first, once, then the repository's own", asked)
}
}
func TestAForgeThatCannotSayLeavesTheModuleUnwrittenAndAsksAgain(t *testing.T) { // a timeout or a 404 at the head: no read at an older commit
// Issue 1945: car-hunter and snake answered 404 until the forge's account could read them, and were read again only when
// the indexer started again. Each row's trigger must now take the module in once the forge reads it.
const snake = `{"module":"snake","version":"1"}`
ctx := context.Background()
for _, tc := range []struct{ failure, trigger string }{
{"timeout", "the indexer starts again"},
{"Gitea API /repos/x/git/trees/y: 404 not found", "the indexer starts again"},
{"Gitea API /repos/x/git/trees/y: 404 not found", "a new source revision"},
{"Gitea API /repos/x/git/trees/y: 404 not found", "a merged message of another repository"},
{"Gitea API /repos/x/git/trees/y: 404 not found", "a person's take-in"},
{"timeout", "a person's take-in"},
} {
t.Run(tc.failure+", then "+tc.trigger, func(t *testing.T) {
ix, mods := modulesIndexer(t, rec(t, "build-1", "snake", "http://novox.internal:20000/jschoubben/snake.git", "", "master", "", "2026-10-19T09:00:00Z", ""))
ix.Manifests = newMem()
ix.source = sourceNow{rev: 1, heads: map[string]string{"jschoubben/snake": h2}}
var asked []string
readable := false
ix.Read = func(_ context.Context, _, sha, _ string) (RepoRead, error) {
asked = append(asked, sha)
if readable {
return RepoRead{Root: "r0", Trees: map[string]string{"": "t1"}, Manifests: map[string]string{"": snake}}, nil
}
return RepoRead{}, errors.New(tc.failure)
}
if err := ix.IndexRegisteredAll(ctx); err == nil {
t.Errorf("no error")
}
if _, ok := moduleOf(t, mods, "snake"); ok || len(asked) != 1 || asked[0] != h2 {
t.Errorf("written %v, asked %v", ok, asked)
}
if err := ix.IndexRegisteredAll(ctx); err == nil || len(asked) != 2 {
t.Errorf("not asked again (asked %v)", asked)
}
readable = true // the forge's account reads the repository now
var got TakenIn
switch tc.trigger {
case "the indexer starts again":
if err := ix.IndexRegisteredAll(ctx); err != nil {
t.Fatal(err)
}
case "a new source revision":
if err := ix.ApplySource(ctx, 2, delivery.Source{Heads: map[string]string{"jschoubben/snake": h2, "novox/other": h1}}); err != nil {
t.Fatal(err)
}
ix.retryWG.Wait() // read again in the background
case "a merged message of another repository":
if err := ix.Apply(ctx, mergedMsg("novox/other", map[string]string{"": "t9"})); err != nil {
t.Fatal(err)
}
ix.retryWG.Wait() // read again in the background
case "a person's take-in":
got = ix.TakeIn(ctx, "snake")
if !got.Stored || got.Repository != "jschoubben/snake" || got.Commit != h2 || got.Path != "." || got.Answer == "" {
t.Errorf("take-in answered %+v", got)
}
}
if m, ok := moduleOf(t, mods, "snake"); !ok || m.Manifest == "" || m.Commit != h2 {
t.Fatalf("after %s the readable module is not taken in: %+v, found %v (asked %v)", tc.trigger, m, ok, asked)
}
// Taken in, it is read no more: a further event asks the forge nothing.
n := len(asked)
if err := ix.ApplySource(ctx, 3, delivery.Source{Heads: map[string]string{"jschoubben/snake": h2}}); err != nil {
t.Fatal(err)
}
ix.retryWG.Wait()
if len(asked) != n {
t.Errorf("the forge was asked again after the module was taken in: %v", asked)
}
if again := ix.TakeIn(ctx, "snake"); again.Stored || !again.Held || len(asked) != n {
t.Errorf("a take-in of a held module answered %+v and asked %v", again, asked)
}
if none := ix.TakeIn(ctx, "nobody"); none.Stored || none.Held || !strings.Contains(none.Answer, "no registration") {
t.Errorf("a take-in of an unregistered module answered %+v", none)
}
})
}
}
func TestARecordOfAPullRequestDoesNotReplaceTheRegisteredRef(t *testing.T) { // a newer feat/x record of the same place
const repo = "http://novox.internal:20000/novox/photos.git"
b := NewBuilds()
for _, r := range [][2]any{
rec(t, "b1", "photos", repo, "", "nox-mesh", "", "2026-10-01T09:00:00Z", ""),
rec(t, "b2", "photos", repo, "", "nox-mesh", "", "2026-10-02T09:00:00Z", ""),
rec(t, "b3", "photos", repo, "", "feat/x", "", "2026-10-09T09:00:00Z", ""),
} {
b.Put(r[0].(string), r[1].([]byte))
}
got := b.RegistrationsOf("photos")
if len(got) != 1 || got[0].Ref != "nox-mesh" {
t.Fatalf("registrations = %+v, want the ref most records name, not the newer feat/x", got)
}
k, v := record(t, "b4", "photos", repo, "", "main", "", "2026-10-03T09:00:00Z", "")
b.Put(k, v) // a trunk's ref beats a branch of its own
if got := b.RegistrationsOf("photos"); len(got) != 1 || got[0].Ref != "main" {
t.Fatalf("registrations = %+v", got)
}
}
@@ -47,6 +47,10 @@ func (ix *Indexer) ApplySource(ctx context.Context, rev uint64, s delivery.Sourc
if ix.Modules == nil {
return ix.RecomputeCandidates(ctx)
}
// A registered module the forge could not read is read again at each new source revision (issue 1945), in the
// background: a forge that does not answer must not hold this revision. One taken in later is stamped when it is written
// (applyManifests reads srcOf).
ix.kickRetry(ctx)
held, err := ix.heldModules(ctx)
if err != nil {
return err
@@ -0,0 +1,64 @@
package index
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The indexer's doing tool (novox/hq issue 1945; the instruction "tools are how a module is spoken to"): indexer_take_in
// reads a registered module's manifest now and writes it to the modules state. The tools bundle serves it through the tool
// runner; the indexer's process serves no request subject (ADR 0437 §4), so the tool writes an ask into the indexer's
// state takeins, which the indexer watches and answers there, as the builder answers builder_ask_again. The read tools
// of the indexer's states stay in mesh-delivery-view.
const (
TakeInTool = "indexer_take_in"
// TakeInsState is the indexer's state of take-in asks: the bucket mesh-delivery-indexer_takeins, keyed as the modules state.
TakeInsState = "takeins"
TakeInsBucket = delivery.IndexerModule + "_" + TakeInsState
)
// TakeInAsk is a value of the takeins state: which module a person asked to be taken in, why, who and when, and, once the
// indexer read it, its answer.
type TakeInAsk struct {
V int `json:"v"`
Module string `json:"module"`
Why string `json:"why,omitempty"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Result *TakenIn `json:"result,omitempty"`
Answered time.Time `json:"answered,omitzero"`
}
// DecodeTakeIn reads a takeins value.
func DecodeTakeIn(data []byte) (TakeInAsk, error) {
var a TakeInAsk
if err := json.Unmarshal(data, &a); err != nil {
return a, err
}
if a.V != 1 {
return a, fmt.Errorf("a takeins value of version %d, not 1", a.V)
}
if a.Module == "" {
return a, errors.New("a takeins value naming no module")
}
return a, nil
}
// NewTakeIn is the value indexer_take_in writes for a module: its key and the value.
func NewTakeIn(module, why, by string, at time.Time) (string, []byte, error) {
module = strings.TrimSpace(module)
if module == "" {
return "", nil, errors.New("name the module to take in (module)")
}
key, err := delivery.ModulesKey(module)
if err != nil {
return "", nil, err
}
data, err := json.Marshal(TakeInAsk{V: 1, Module: module, Why: strings.TrimSpace(why), By: by, At: at.UTC()})
return key, data, err
}
+24 -13
View File
@@ -24,18 +24,20 @@ type state struct {
}
type artifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
System string `json:"system"`
From string `json:"from"`
Binary string `json:"binary"`
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
System string `json:"system"`
From string `json:"from"`
Binary string `json:"binary"`
Loads []string `json:"loads"`
}
type manifest struct {
Module string `json:"module"`
Version string `json:"version"`
Slug string `json:"slug"`
Tools []string `json:"tools"`
Consumes []string `json:"consumes"`
State []state `json:"state"`
Reads []string `json:"reads"`
@@ -78,7 +80,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
if m.Module != "mesh-delivery-indexer" {
t.Errorf("module = %q", m.Module)
}
if want := []state{{Name: "candidates", History: 10}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}}; !reflect.DeepEqual(m.State, want) {
if want := []state{{Name: "candidates", History: 10}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-controller.assignments", "mesh-controller.builds", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) {
@@ -95,14 +97,18 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
if m.SecretsOwner != "mesh-delivery-indexer" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// Two invokes: the forge's trees (the named paths' objects, since the indexer is the one reader of a manifest's paths:
// Three invokes: the file of a registered module's manifest and lock files (registered.go), the forge's trees (the named paths' objects, since the indexer is the one reader of a manifest's paths:
// ADR 0499 clauses 2 and 3, ADR 0495 clause 1; and the module directories of the trunk, issue 1900) and whether a
// commit is in another (issue 1900).
if want := []string{"gitea.gitea_commit_trees", "gitea.gitea_contains"}; !reflect.DeepEqual(m.Invokes, want) {
if want := []string{"gitea.gitea_commit_trees", "gitea.gitea_contains", "gitea.gitea_get_file"}; !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
// No emits, no streams, no bus, no seat, no tool (ADR 0437 §4).
for _, k := range []string{"streams", "logs", "emits", "claims", "provides", "requires", "uses", "defines-seats", "tools", "seat", "seats", "bus"} {
// 1945: one tool, indexer_take_in, served by its tools bundle through the takeins state; the process serves none.
if want := []string{"indexer_take_in"}; !reflect.DeepEqual(m.Tools, want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
// No emits, no streams, no bus, no seat (ADR 0437 §4).
for _, k := range []string{"streams", "logs", "emits", "claims", "provides", "requires", "uses", "defines-seats", "seat", "seats", "bus"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the indexer has none", k)
}
@@ -134,8 +140,13 @@ func TestManifestProcess(t *testing.T) { // 1238/AC1
if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential")
}
if len(m.Build.Artifacts) != 1 {
t.Fatalf("artifacts = %+v", m.Build.Artifacts)
if len(m.Build.Artifacts) != 2 {
t.Fatalf("artifacts = %+v, want the program and the tools", m.Build.Artifacts)
}
if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-indexer-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-indexer-tools"}) {
t.Errorf("tools artifact = %+v", a) // 1945
} else if st, err := os.Stat(a.From); err != nil || !st.IsDir() {
t.Errorf("the tools' source %s: %v", a.From, err)
}
a := m.Build.Artifacts[0]
if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-indexer" || !strings.HasSuffix(a.From, a.Binary) {
+18 -2
View File
@@ -2,6 +2,9 @@
"module": "mesh-delivery-indexer",
"version": "1",
"slug": "indexer",
"tools": [
"indexer_take_in"
],
"consumes": [
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported",
@@ -11,7 +14,8 @@
"state": [
{ "name": "candidates", "history": 10 },
{ "name": "manifests", "history": 1 },
{ "name": "modules", "history": 20 }
{ "name": "modules", "history": 20 },
{ "name": "takeins", "history": 1 }
],
"reads": [
"mesh-build-go.toolchains",
@@ -23,7 +27,8 @@
],
"invokes": [
"gitea.gitea_commit_trees",
"gitea.gitea_contains"
"gitea.gitea_contains",
"gitea.gitea_get_file"
],
"own-secrets": {
"broker": "${dir:state}/broker"
@@ -71,6 +76,17 @@
"system": "arch",
"from": "cmd/mesh-delivery-indexer",
"binary": "mesh-delivery-indexer"
},
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-indexer-tools",
"binary": "mesh-delivery-indexer-tools",
"loads": [
"mesh-delivery-indexer-tools"
]
}
]
}
@@ -0,0 +1,83 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/asks"
)
// The three bus adapters for mesh-sdk go/asker on the part's own credential, as mesh-delivery-teller has them (copied: the
// SDK's asker reaches the bus through a bundle's runtime; moving the adapters into the SDK is a change of its own). What
// the account may do is the controller's composed grant for a module that uses the seat operator-channel: publish its own
// ask and condition, hear its own decided event, and read the router's record of its own asks.
const recordWithin = 30 * time.Second
func publishVia(ctx context.Context, js jetstream.JetStream) func(subject string, body any, id string) error {
return func(subject string, body any, id string) error {
data, err := json.Marshal(body)
if err != nil {
return err
}
var opts []jetstream.PublishOpt
if id != "" {
opts = append(opts, jetstream.WithMsgID(id))
}
_, err = js.Publish(ctx, subject, data, opts...)
return err
}
}
func recordVia(ctx context.Context, conn *nats.Conn, module string) func(id string) (*stdio.StateEntry, error) {
return func(id string) (*stdio.StateEntry, error) {
ctx, cancel := context.WithTimeout(ctx, recordWithin)
defer cancel()
msg, err := conn.RequestWithContext(ctx, "$JS.API.DIRECT.GET.KV_asks.$KV.asks."+module+"."+id, nil)
// No responders is the bucket out of reach for a moment, never "no record": read as none, a live ask would be
// closed by Reconcile a minute later and the operator's yes lost.
if errors.Is(err, nats.ErrNoResponders) {
return nil, fmt.Errorf("the router's record of %s cannot be reached: %w", id, err)
}
if err != nil {
return nil, err
}
if status := msg.Header.Get("Status"); status == "404" || len(msg.Data) == 0 {
return nil, nil
}
return &stdio.StateEntry{Key: id, Value: msg.Data}, nil
}
}
func decidedVia(conn *nats.Conn, module string, logf func(string, ...any)) func(pattern string, handle func(stdio.Envelope) error) error {
return func(pattern string, handle func(stdio.Envelope) error) error {
_, err := conn.Subscribe(asks.DecidedSubject(module), func(m *nats.Msg) {
if err := handle(stdio.Envelope{Key: pattern, Body: m.Data}); err != nil {
logf("a warrant could not be handled and is read again from the router's record at the next start: %v", err)
}
})
return err
}
}
func sayVia(ctx context.Context, js jetstream.JetStream, module string) func(c asks.Condition) error {
return func(c asks.Condition) error {
subject, err := asks.ConditionSubject(module)
if err != nil {
return err
}
data, err := asks.EncodeCondition(c)
if err != nil {
return err
}
_, err = js.Publish(ctx, subject, data)
return err
}
}
+31 -5
View File
@@ -7,9 +7,10 @@
// module placed since cannot serve its seat or reach a new state. Each is published on the assignments stream only when
// it differs from the one last published there. **A bucket, log or stream a manifest declares that the bus lacks is
// created** (the operator, 2026-10-26 08:54; issue 1790), once per manifest, with the manifest's limits, on the event
// that brought the manifest; one the bus holds is never changed. **The user list, the rest of the assertions and the
// grants stay in shadow** until
// the operator's take-over, which this milestone always refuses: they are composed, compared with what the bus loaded
// that brought the manifest; one the bus holds is never changed. **The user list is written sealed to the bus node**
// (ADR 0547 clause 7) as an object of the part's own bucket named in `user-list.sealed`, which the bus node follows only
// from take-over. **The rest of the assertions and the grants stay in shadow** until
// the operator's take-over, asked on their channel and run on their yes after its refusals: they are composed, compared with what the bus loaded
// and holds (through the bus module's own read tools) and kept in the part's own state, and take no effect.
//
// It holds no timer, and never exits for a refused read, write or publish: a refusal is said and tried again at the next
@@ -39,6 +40,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-grants/internal/grants"
@@ -115,8 +117,18 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
bus := newBusReader(ctx, js)
f.Part = &grants.Part{Read: bus, Call: conn, Store: ownStore{conn}, Emit: emitting{conn},
Publish: memberships{ctx: ctx, nc: nc, js: js}, Create: creating{ctx: ctx, js: js}}
Publish: memberships{ctx: ctx, nc: nc, js: js}, Create: creating{ctx: ctx, js: js},
Objects: sealedObjects{ctx: ctx, js: js}, Logf: logger.Printf}
f.Watch = bus
// The operator's word on take-over and hand-back is their yes on their channel: the part asks it itself, and acts on
// the warrant once (mesh-sdk asker), after its refusals.
client := &asker.Client{Module: account.Module, State: conn.State(grants.StateTakeover), Publish: publishVia(ctx, js),
Record: recordVia(ctx, nc, account.Module), Subscribe: decidedVia(nc, account.Module, logger.Printf)}
f.Part.Asker, f.Part.Say = client, sayVia(ctx, js, account.Module)
f.Part.Forget = func(id string) error { return conn.State(grants.StateTakeover).Delete("ask." + id) }
if err := client.Listen(f.Part.Warranted, f.Part.Said); err != nil {
logger.Printf("the operator's answers cannot be heard yet, so take-over and hand-back wait: %v", err)
}
f.Asked = func(wake func()) error {
return conn.State(grants.StateUserList).Watch(grants.KeyAsked, func(c stdio.StateChange) error {
if !c.Current {
@@ -125,7 +137,21 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
})
}
logger.Printf("started on the account %s: publishing memberships and creating the declared buckets, logs and streams the bus lacks; the user list, grants and any change to a held stream in shadow", account.User)
// The controller makes this module's durable consumer on EVENTS from the manifest's consumes (filtered on the two
// condition subjects); it is pulled and every message acknowledged, so its pending never grows. Where it is not there
// yet the pass says so and hears again at the next change.
f.Hear = func(handle func(string, []byte)) error {
cons, err := js.Consumer(ctx, "EVENTS", account.Node+"_"+account.Module)
if err != nil {
return fmt.Errorf("the consumer %s_%s on EVENTS: %w", account.Node, account.Module, err)
}
_, err = cons.Consume(func(m jetstream.Msg) {
handle(m.Subject(), m.Data())
_ = m.Ack()
}, jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
return err
}
logger.Printf("started on the account %s: publishing memberships and creating the declared buckets, logs and streams the bus lacks; and writing the user list sealed to the bus node, which the bus node follows from take-over; grants and any change to a held stream in shadow", account.User)
f.Run(ctx)
return nil
}
@@ -0,0 +1,55 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// sealedObjects is the part's object bucket of user lists sealed to the bus node (novox/hq ADR 0547 clause 7), created on
// its first put: the composer grants the part its create, its puts and deletes, and nothing of another bucket.
type sealedObjects struct {
ctx context.Context
js jetstream.JetStream
}
func (o sealedObjects) store(ctx context.Context) (jetstream.ObjectStore, error) {
s, err := o.js.ObjectStore(ctx, delivery.GrantsUserListObjects)
if errors.Is(err, jetstream.ErrBucketNotFound) {
s, err = o.js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: delivery.GrantsUserListObjects,
Description: "the bus's user list composed by the grants part, each sealed to the bus node's key under the hex of its digest (novox/hq ADR 0547 clause 7): written by mesh-grants alone, read by name by the node-engine of the node holding mesh-broker",
Storage: jetstream.FileStorage, MaxBytes: 64 << 20})
}
if err != nil {
return nil, fmt.Errorf("the object bucket %s: %w", delivery.GrantsUserListObjects, err)
}
return s, nil
}
func (o sealedObjects) PutObject(name string, data []byte) error {
ctx, cancel := context.WithTimeout(o.ctx, requestWithin)
defer cancel()
s, err := o.store(ctx)
if err != nil {
return err
}
_, err = s.PutBytes(ctx, name, data)
return err
}
func (o sealedObjects) DeleteObject(name string) error {
ctx, cancel := context.WithTimeout(o.ctx, requestWithin)
defer cancel()
s, err := o.store(ctx)
if err != nil {
return err
}
if err := s.Delete(ctx, name); err != nil && !errors.Is(err, jetstream.ErrObjectNotFound) {
return err
}
return nil
}
+13 -8
View File
@@ -97,20 +97,25 @@ func toolsOf(p *grants.Part) []stdio.Tool {
"changed, a difference is only said. Answers the comparison and the ask; call again to see what the bus holds.",
Input: schema(map[string]string{"module": "the module"}, "module"),
Run: func(a map[string]any) (any, error) { return p.AssertOne(grants.Answer(a, "module")) }},
{Name: grants.ToolTakeOver, Description: "The operator's switch to the grants part's list (ADR 0547 clause 8), refused without the " +
"operator's credential (a person's account), refused while any user of the shadow list differs from the loaded list (but " +
"those placement no longer names), naming them; in this milestone always refused, since the enrolment users and the " +
"persons cannot be composed. Every ask is recorded in the part's state takeover.",
{Name: grants.ToolTakeOver, Description: "Ask the operator to switch the bus node to the grants part's list (ADR 0547 clause 8). " +
"The operator's word is their yes on their channel (a warrant), never a caller's name: this tool only asks, once at a time. " +
"After the yes, the switch is recorded and takes effect when the bus node loads the part's list; it is refused while the loaded list could not be read, a user differs between the lists " +
"(an enrolment user or a person the part cannot compose among them), or the vault holds no hash for a user, and the " +
"refusal is said to the operator naming the users. Every act is recorded in the part's state takeover.",
Input: schema(map[string]string{"why": "why, kept with the act"}, "why"),
Run: func(a map[string]any) (any, error) {
return nil, p.TakeOver(stdio.CallerOf(a), grants.Answer(a, "why"))
return p.TakeOver(stdio.CallerOf(a), grants.Answer(a, "why"))
}},
{Name: grants.ToolHandBack, Description: "The operator's switch back to the controller's list (ADR 0547 clause 8), refused without " +
"the operator's credential; in shadow there is nothing to hand back, and it says so. Every ask is recorded.",
{Name: grants.ToolHandBack, Description: "Ask the operator to switch the bus node back to the controller's list (ADR 0547 clause 8), " +
"by their yes on their channel; in shadow there is nothing to hand back, and it says so. Every act is recorded.",
Input: schema(map[string]string{"why": "why, kept with the act"}, "why"),
Run: func(a map[string]any) (any, error) {
return nil, p.HandBack(stdio.CallerOf(a), grants.Answer(a, "why"))
return p.HandBack(stdio.CallerOf(a), grants.Answer(a, "why"))
}},
{Name: grants.ToolTakeOverState, Description: "Where the switch stands: the list the bus node follows, the ask open now, the request, and " +
"every act with what became of it (asked, switched, declined, expired, refused and why).",
Input: schema(nil),
Run: func(map[string]any) (any, error) { return p.TakeOverState() }},
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-grants
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026080443-136fa8b2a896
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026080443-136fa8b2a896 h1:5rHfru5NJ7SK20QOt8YUJSglcn1gsjtLc9AGmWYGnRQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026080443-136fa8b2a896/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530 h1:NwCb+gOh0UYqlyG6hG2/jffE/Gd0c7YWd4iJlaCB0lU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026134035-21252194d530/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
+482 -39
View File
@@ -16,8 +16,12 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/seal"
"mesh-grants/internal/grants"
)
@@ -48,6 +52,8 @@ type fakeBus struct {
// creates are the streams the part asked to create, refused when refuseCreate.
creates []string
refuseCreate bool
// rootFree answers the controller's root-free verb: nil is no answerer at all (no responders).
rootFree func(machines []string) (any, error)
}
func (b *fakeBus) LastMembership(node, module string) ([]byte, uint64, bool, error) {
@@ -166,6 +172,15 @@ func (b *fakeBus) Call(key string, body any) (json.RawMessage, error) {
return json.Marshal(map[string]any{"users": out})
case grants.CallStreams:
return json.Marshal(map[string]any{"detail": b.streams})
case grants.CallRootFree:
if b.rootFree == nil {
return nil, errors.New("nats: no responders available for request")
}
out, err := b.rootFree(args["machines"].([]string))
if err != nil {
return nil, err
}
return json.Marshal(out)
}
return nil, fmt.Errorf("no tool %s", key)
}
@@ -300,6 +315,7 @@ func TestGrants(t *testing.T) {
}{
{"reads", m.Reads, grants.Reads},
{"invokes", m.Invokes, grants.Invokes},
{"consumes", m.Consumes, grants.Consumes},
{"emits", m.Emits, grants.Emits},
{"tools", m.Tools, grants.Tools},
} {
@@ -327,6 +343,9 @@ func TestGrants(t *testing.T) {
// in create.go (issue 1790).
publishOnly := filepath.Join("cmd", "mesh-grants", "publish.go")
createOnly := filepath.Join("cmd", "mesh-grants", "create.go")
// And the operator's asks and conditions, on the seat operator-channel's subjects, in asking.go (ADR 0547 clause 8).
asksOnly := filepath.Join("cmd", "mesh-grants", "asking.go")
sealedOnly := filepath.Join("cmd", "mesh-grants", "sealed.go")
files := 0
for _, root := range []string{"cmd", "internal"} {
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
@@ -335,8 +354,11 @@ func TestGrants(t *testing.T) {
}
files++
src, err := os.ReadFile(path)
if path != publishOnly && (strings.Contains(string(src), ".Publish(") || strings.Contains(string(src), "mesh.assignment")) {
t.Errorf("%s publishes: only publish.go publishes, and only memberships", path)
if path != publishOnly && path != asksOnly && (strings.Contains(string(src), ".Publish(") || strings.Contains(string(src), "mesh.assignment")) {
t.Errorf("%s publishes: only publish.go publishes memberships, and asking.go the operator's asks", path)
}
if path == asksOnly && (strings.Count(string(src), ".Publish(") != 2 || strings.Contains(string(src), "mesh.assignment")) {
t.Errorf("asking.go publishes other than an ask, a cancel and a condition, through js.Publish twice")
}
if path == publishOnly && strings.Count(string(src), ".Publish(") != 1 {
t.Errorf("publish.go publishes other than one membership")
@@ -344,6 +366,13 @@ func TestGrants(t *testing.T) {
if n := strings.Count(string(src), "CreateStream"); (path == createOnly && n != 1) || (path != createOnly && n != 0) {
t.Errorf("%s holds CreateStream %d times: only create.go creates, once", path, n)
}
// The one object bucket: the part's own, of the user list sealed to the bus node, in sealed.go (ADR 0547 clause 7).
if path == sealedOnly {
if strings.Count(string(src), "CreateObjectStore(") != 1 || !strings.Contains(string(src), "delivery.GrantsUserListObjects") {
t.Errorf("sealed.go creates other than the part's one object bucket")
}
src = []byte(strings.ReplaceAll(string(src), "CreateObjectStore", ""))
}
for _, f := range forbidden {
if strings.Contains(string(src), f) {
t.Errorf("%s holds %s: the part holds no timer and creates or asserts nothing", path, f)
@@ -436,6 +465,94 @@ func TestGrants(t *testing.T) {
}
})
t.Run("root-free: the controller's word fills the composition, failing closed (issue 1945)", func(t *testing.T) {
const router = `{"module":"messenger","version":"1","runs-as":"messenger","resources":[{"id":"account","type":"user","name":"messenger","shell":"/usr/bin/nologin","home":"/var/lib/messenger"}],"own-secrets":{"broker":"/var/lib/m/broker"},"secrets-owner":"messenger",` +
`"seats":[{"name":"operator-channel","scope":"mesh","serves":["open"]},{"name":"channel","scope":"mesh","kinded":true,"accepts":["show"]}],` +
`"claims":[{"name":"operator-channel","scope":"mesh","serves":["open"]}]}`
const telegram = `{"module":"telegram","version":"1","runs-as":"telegram","resources":[{"id":"account","type":"user","name":"telegram","shell":"/usr/bin/nologin","home":"/var/lib/telegram"}],"own-secrets":{"broker":"/var/lib/t/broker"},"secrets-owner":"telegram",` +
`"claims":[{"name":"channel","scope":"mesh","kind":"telegram","capabilities":["deliver","verified-sender"]}]}`
answer := func(free ...string) func([]string) (any, error) {
return func(machines []string) (any, error) {
var vs []map[string]any
for _, m := range machines {
ok := slices.Contains(free, m)
why := "judged free"
if !ok {
why = "its node-engine judged nothing"
}
vs = append(vs, map[string]any{"machine": m, "free": ok, "why": why})
}
return map[string]any{"machines": vs}, nil
}
}
for _, row := range []struct {
name string
routerOn []string
telegramOn string
ask func([]string) (any, error)
keeps bool
note string
}{
{"router and telegram on one free machine", []string{"a"}, "a", answer("a"), true, ""},
{"router and telegram on two free machines", []string{"a"}, "b", answer("a", "b"), true, ""},
{"the holder's machine is not free", []string{"a"}, "b", answer("a"), false, "b: not free, its node-engine judged nothing"},
{"the router's machine is not free", []string{"a"}, "b", answer("b"), false, "a: not free"},
{"an error is not free", []string{"a"}, "a", func([]string) (any, error) { return nil, errors.New("boom") }, false, "boom"},
{"no answerer is not free", []string{"a"}, "a", nil, false, "no responders"},
{"a machine left out of the answer is not free", []string{"a"}, "a",
func([]string) (any, error) { return map[string]any{"machines": []any{}}, nil }, false, "named no verdict"},
{"the router on two machines frees nothing", []string{"a", "b"}, "a", answer("a", "b"), false, ""},
} {
t.Run(row.name, func(t *testing.T) {
b := fixture()
b.manifestOf("messenger", router)
b.manifestOf("telegram", telegram)
for _, node := range row.routerOn {
b.assign(node, "messenger")
}
b.assign(row.telegramOn, "telegram")
b.rootFree = row.ask
res := partOn(b).Compose()
recs, _ := grants.Records(grants.ReadInputs(b, &grants.Manifests{}))
grants.WithRootFree(&recs, b)
kept := false
for _, k := range buscompose.PlacementsOf(recs, recs.Interchangeable).Kinds {
kept = kept || slices.Contains(k.Capabilities, "verified-sender")
}
if kept != row.keeps {
t.Errorf("verified-sender kept = %v, want %v (said %v)", kept, row.keeps, res.RootNotFree)
}
if !strings.Contains(strings.Join(res.RootNotFree, " | "), row.note) || (row.note == "" && row.keeps && len(res.RootNotFree) != 0) {
t.Errorf("the pass says %v, want it to name %q", res.RootNotFree, row.note)
}
})
}
// A root-not-free condition raised or cleared wakes a pass; another kind does not (never a timer).
wb := fixture()
var handle func(string, []byte)
wf := &grants.Follower{Part: partOn(wb), Watch: &fakeWatcher{opened: make(chan string, 16)}, Logf: func(string, ...any) {},
Hear: func(h func(string, []byte)) error { handle = h; return nil }}
wctx, wcancel := context.WithCancel(context.Background())
wdone := make(chan struct{})
go func() { wf.Run(wctx); close(wdone) }()
waitFor(t, func() bool { return wb.count(grants.CallUsers) > 0 && handle != nil })
passes := func() int { return wb.count(grants.CallUsers) }
before := passes()
handle(grants.EventConditionRaised, []byte(`{"kind":"disk-full","key":"machine.a.disk-full"}`))
time.Sleep(100 * time.Millisecond)
if passes() != before {
t.Errorf("a condition of another kind woke a pass")
}
handle(grants.EventConditionCleared, []byte(`{"kind":"root-not-free","key":"machine.a.root-not-free"}`))
waitFor(t, func() bool { return passes() > before })
wcancel()
<-wdone
if !grants.IsRootNotFree([]byte(`{"kind":"root-not-free"}`)) || !grants.IsRootNotFree([]byte(`{"key":"machine.novox.root-not-free"}`)) ||
grants.IsRootNotFree([]byte(`{"kind":"disk-full","key":"machine.novox.disk-full"}`)) {
t.Errorf("only a root-not-free condition wakes a pass")
}
})
t.Run("a predefined-key fallback is read with no list; every refused manifest listed", func(t *testing.T) {
// mod-p's value falls back to a predefined key, as claude-code's managed_settings does, and no list is on the bus
// (issue 1945); mod-z's manifest carries a key mesh-sdk's reader refuses, and mod-z is assigned nowhere.
@@ -446,7 +563,20 @@ func TestGrants(t *testing.T) {
b.manifestOf("mod-p", pManifest)
b.manifestOf("mod-z", zManifest)
b.assign("b", "mod-p")
// mesh-wireguard is provided by the mesh, not read from a repository: no manifest in the indexer, none refused,
// no gap, and its manifest is the one mesh-sdk provides (issue 1945).
b.assign("b", "mesh-wireguard")
in := grants.ReadInputs(b, &grants.Manifests{})
// An indexer row of the same name never wins: the provided manifest is used and the row is said as a gap.
bw := fixture()
bw.manifestOf("mesh-wireguard", `{"module":"mesh-wireguard","version":"9","own-secrets":{"broker":"/x"}}`)
bw.assign("b", "mesh-wireguard")
if iw := grants.ReadInputs(bw, &grants.Manifests{}); !strings.HasPrefix(iw.Digests["mesh-wireguard"], "provided:") || !gapNamed(iw.Gaps, "manifest of mesh-wireguard") || len(iw.Refused) != 0 {
t.Errorf("an indexer manifest of mesh-wireguard won or went unsaid: %v %v", iw.Digests["mesh-wireguard"], iw.Gaps)
}
if _, ok := in.Manifests["mesh-wireguard"]; !ok || gapNamed(in.Gaps, "manifest of mesh-wireguard") || !strings.HasPrefix(in.Digests["mesh-wireguard"], "provided:") {
t.Errorf("mesh-wireguard's manifest is not taken from mesh-sdk: gaps %v", in.Gaps)
}
if len(in.Refused) != 1 || in.Refused[0].Input != "manifest of mod-z" || !gapNamed(in.Gaps, "manifest of mod-z") {
t.Errorf("refused = %v: only mod-z's, and named in the gaps", in.Refused)
}
@@ -528,9 +658,9 @@ func TestGrants(t *testing.T) {
!slices.Contains(cr.LeftOut, "b.mod-y") {
t.Errorf("a hash heard = %+v", after.UserList.Credentials)
}
err := p.TakeOver("person.operator", "hashes")
if err == nil || !strings.Contains(err.Error(), "no hash for") || strings.Contains(err.Error(), "a.mod-x,") {
t.Errorf("take-over = %v, want the users with no hash named", err)
why := strings.Join(p.Decide(), "; ")
if !strings.Contains(why, "no hash for") || strings.Contains(why, "a.mod-x,") {
t.Errorf("take-over is refused for %q, want the users with no hash named", why)
}
for state, keys := range b.own {
for k, v := range keys {
@@ -578,11 +708,61 @@ func TestGrants(t *testing.T) {
}
})
t.Run("compare excuses a retired verb and a multiply claimed mesh seat's added subscriptions on its machines, nothing else", func(t *testing.T) {
const retired, store, storeInfo = "mesh.seat.mesh-delivery.tool.mark-ready", "mesh.seat.mesh-store.tool.databases", "$SRV.INFO.mesh-store.>"
two := map[string][]string{"mesh-store": {"a", "n"}}
for _, row := range []struct {
name string
user string
subShadow, subBus, pubShadow []string
multi map[string][]string
differs bool
excused int
}{
{"a retired verb the shadow dropped", "a.node-tools", []string{"x"}, []string{"x", retired}, nil, nil, false, 1},
{"a verb of a live seat the shadow lost", "a.node-tools", []string{"x"}, []string{"x", store}, nil, two, true, 0},
{"a seat claimed on two machines, added to a user on one", "a.node-tools", []string{"x", store, storeInfo}, []string{"x"}, nil, two, false, 2},
{"the same added to a user on a machine that does not claim it", "q.node-tools", []string{"x", store}, []string{"x"}, nil, two, true, 0},
{"a publish of the seat added is not excused", "a.node-tools", []string{"x"}, []string{"x"}, []string{store}, two, true, 0},
{"the same added for a seat claimed once", "a.node-tools", []string{"x", store}, []string{"x"}, nil, nil, true, 0},
} {
c := grants.Compare([]grants.User{{User: row.user, Subscribe: row.subShadow, Publish: row.pubShadow}}, []grants.User{{User: row.user, Subscribe: row.subBus}}, nil).
Excusing(grants.ExcusedBy(row.multi))
if c.Equal() == row.differs || len(c.Excused) != row.excused {
t.Errorf("%s: equal %v, excused %v", row.name, c.Equal(), c.Excused)
}
}
})
t.Run("tools", func(t *testing.T) {
b := fixture()
b.loaded = shadowOf(t, b)
b.streams = []map[string]any{{"name": "KV_mod-x_kept", "subjects": []string{"$KV.mod-x_kept.>"}, "retention": "limits", "max_per_subject": 3}}
p := partOn(b)
// The bus node says its key in its have of the bus module; the process puts the sealed list (ADR 0547 clause 7).
busKey, err := seal.GenerateKeyPair()
if err != nil {
t.Fatal(err)
}
withKey := func(w delivery.UserListWritten) {
w.Key = busKey.PublicKey
b.put(grants.ReadHave, "a.nats", delivery.Have{V: delivery.Version, Phase: delivery.PhaseRunning, Health: delivery.HealthOK, UserList: &w})
}
loadedBefore := delivery.UserListWritten{Source: "a.nats.src2", Revision: 7, Wanted: "sha256:aa", Digest: "sha256:aa", Users: 6}
withKey(loadedBefore)
objs := &fakeObjects{}
p.Objects = objs
open := func(g delivery.GrantsUserList) string {
t.Helper()
list, err := delivery.OpenUserList(objs.objects[g.Object], g.Digest, func(bx seal.Box) ([]byte, error) {
out, err := seal.Open(bx, busKey.PrivateKey)
return []byte(out), err
})
if err != nil {
t.Fatalf("the sealed list does not open with the bus node's key: %v", err)
}
return string(list)
}
got, _ := p.UserList()
if !strings.Contains(fmt.Sprint(got), "nothing composed yet") {
@@ -592,6 +772,20 @@ func TestGrants(t *testing.T) {
if !first.Written || first.Revision != 1 || !first.UserList.Compare.Equal() || first.UserList.Users != 6 {
t.Fatalf("first composition = %+v", first)
}
// Written in shadow, the bus node told to follow the controller's list; every user left out while the vault holds
// no hash; a tool's part (no object bucket) writes none.
if g, _, had := p.Sealed(); !had || g.Follow != delivery.FollowController || g.Revision != 1 || g.Node != "a" ||
g.Users != 0 || len(g.LeftOut) != 6 || len(objs.objects) != 1 || !strings.Contains(open(g), "accounts {") {
t.Errorf("the sealed list in shadow = %+v, %d objects", g, len(objs.objects))
}
if _, _, had := partOn(fixture()).Sealed(); had {
t.Error("a part read before any pass holds a sealed list")
}
if tp := partOn(fixture()); tp.Compose().Sealed == nil {
t.Error("a tool's pass does not say the sealed list")
} else if _, _, had := tp.Sealed(); had {
t.Error("a tool's part wrote the sealed list")
}
if len(first.Published) != 3 || len(b.publishes) != 3 {
t.Errorf("the first composition published %v, want the three assignments' memberships", first.Published)
}
@@ -642,54 +836,258 @@ func TestGrants(t *testing.T) {
}
b.loaded = append(b.loaded, grants.User{User: "b.gone"})
fa := &fakeAsker{wheres: map[string]*asks.Warrant{}}
var forgot []string
n := 0
p.Nonce, p.Forget = func() string { n++; return fmt.Sprintf("%04d", n) }, func(id string) error { forgot = append(forgot, id); return nil }
var said []asks.Condition
p.Asker, p.Say = fa, func(c asks.Condition) error { said = append(said, c); return nil }
warrant := func(id string) asks.Warrant { return asks.Warrant{Ask: id, Outcome: asks.OutcomeChosen} }
last := func() grants.Act {
raw, _, _, _ := b.ownGet(grants.StateTakeover, grants.KeySwitch)
var s grants.Switch
_ = json.Unmarshal(raw, &s)
if s.Open != "" && s.Acts[len(s.Acts)-1].Said != "asked" {
t.Errorf("an ask is open and its act is not asked: %+v", s)
}
return s.Acts[len(s.Acts)-1]
}
asked := func(act string) string {
t.Helper()
var res struct{ Asked string }
var err error
var out any
if act == "take-over" {
out, err = p.TakeOver("shanks.node-tools", "a row")
} else {
out, err = p.HandBack("shanks.node-tools", "a row")
}
if err != nil {
t.Fatalf("%s: %v", act, err)
}
raw, _ := json.Marshal(out)
_ = json.Unmarshal(raw, &res)
return res.Asked
}
yes, no := asks.Option{ID: "yes"}, asks.Option{ID: "no"}
for _, row := range []struct {
name, act, caller string
says, never []string
name, act string
option asks.Option
setup func()
said, never []string // the refusal's words and the users it must not name
state string
outcome string
}{
{"take-over without the operator's credential", "take-over", "shanks.node-tools", []string{"operator's alone"}, nil},
{"hand-back without the operator's credential", "hand-back", "unproven", []string{"operator's alone"}, nil},
{"take-over while a user differs names it", "take-over", "person.operator", []string{"users differ", "b.mod-x", grants.NotComposable}, []string{"b.gone"}},
{"hand-back in shadow", "hand-back", "person.operator", []string{"nothing to hand back"}, nil},
{"a yes while a user differs is refused and said, naming it", "take-over", yes, func() {}, []string{"users differ", "b.mod-x"}, []string{"b.gone"}, grants.Shadow, "refused"},
{"a no leaves the switch", "take-over", no, func() {}, nil, nil, grants.Shadow, "declined"},
{"a yes with the lists agreeing and every hash held records the switch", "take-over", yes, func() {
b.states[grants.ReadAssignments] = nil
b.assign("a", "nats")
b.assign("a", "mod-x")
b.assign("b", "mod-y")
b.loaded = append(shadowOf(t, b), grants.User{User: "b.gone"})
for _, u := range shadowOf(t, b) {
b.put(grants.ReadAccounts, u.User, map[string]any{"user": u.User, "hash": "$2a$10$x", "generation": 1})
}
}, nil, nil, grants.TookOver, "recorded"},
{"a yes to hand back records the switch back", "hand-back", yes, func() {}, nil, nil, grants.Shadow, "recorded"},
} {
t.Run(row.name, func(t *testing.T) {
var err error
if row.act == "take-over" {
err = p.TakeOver(row.caller, "a row")
} else {
err = p.HandBack(row.caller, "a row")
row.setup()
before := len(fa.asks)
id := asked(row.act)
if _, err := p.TakeOver("shanks.node-tools", "again"); err == nil || !strings.Contains(err.Error(), "already asked") {
t.Errorf("a second ask while one is open = %v", err)
}
if err == nil {
t.Fatalf("%s by %s was not refused", row.act, row.caller)
p.Compose()
p.Compose()
if len(fa.asks) != before+1 || fa.asks[before].ID != id {
t.Fatalf("the operator was asked %d times, want once: %+v", len(fa.asks)-before, fa.asks)
}
for _, s := range row.says {
if !strings.Contains(err.Error(), s) {
t.Errorf("refusal %q does not say %q", err, s)
ask := fa.asks[before]
if err := ask.Check(time.Date(2026, 10, 25, 22, 0, 0, 0, time.UTC)); err != nil {
t.Errorf("the router would refuse the ask: %v", err)
}
if yesOpt, ok := ask.Option("yes"); !ok || yesOpt.Level != asks.Approve || !strings.HasPrefix(yesOpt.Binds, "sha256:") {
t.Errorf("the yes does not approve and bind its act: %+v", yesOpt)
}
said = nil
if err := p.Warranted(row.option, warrant(id)); err != nil {
t.Fatal(err)
}
l := last()
if l.Said != row.outcome || l.Ask != id || p.SwitchState() != row.state {
t.Errorf("the act = %+v, the switch %s; want %s and %s", l, p.SwitchState(), row.outcome, row.state)
}
for _, w := range row.said {
if !strings.Contains(l.Refused, w) || (row.outcome == "refused" && (len(said) != 1 || !strings.Contains(said[0].Explanation, w))) {
t.Errorf("the refusal %q (said %v) does not say %q", l.Refused, said, w)
}
}
for _, s := range row.never {
if strings.Contains(err.Error(), s) {
t.Errorf("refusal %q names %q", err, s)
for _, w := range row.never {
if strings.Contains(l.Refused, w) {
t.Errorf("the refusal names %q", w)
}
}
raw, _, _, _ := b.ownGet(grants.StateTakeover, grants.KeySwitch)
var s grants.Switch
_ = json.Unmarshal(raw, &s)
last := s.Acts[len(s.Acts)-1]
if s.State != grants.Shadow || last.Act != row.act || last.By != row.caller || last.Refused == "" {
t.Errorf("the act is not recorded: %+v", s)
// A recorded switch says no refusal and clears the one a refused yes left raised.
if row.outcome == "recorded" && row.act == "take-over" && (len(said) != 1 || said[0].State != asks.StateCleared || said[0].Key != "mesh-grants.take-over-refused") {
t.Errorf("a recorded switch did not clear the refusal: %v", said)
}
if row.outcome == "recorded" && !strings.Contains(fmt.Sprint(p.TakeOverState()), "bus_node_loaded_grants_list:not yet") {
t.Errorf("the state claims the bus node follows with no loaded list: %v", fmt.Sprint(p.TakeOverState()))
}
// The key the bus node follows turns with the switch at once, and the state says switched only once the bus
// node loaded the grants part's written list.
g, _, _ := p.Sealed()
if want := map[string]string{grants.TookOver: delivery.FollowGrants, grants.Shadow: delivery.FollowController}[p.SwitchState()]; g.Follow != want {
t.Errorf("the key says follow %s with the switch %s", g.Follow, p.SwitchState())
}
effect := func() string { st, _ := p.TakeOverState(); return fmt.Sprint(st.(map[string]any)["effect"]) }
if row.outcome == "recorded" && row.act == "take-over" {
if e := effect(); strings.HasPrefix(e, "switched") || !strings.Contains(e, "not in effect") {
t.Errorf("effect before the bus node loaded = %s", e)
}
if users := strings.Count(open(g), "{ user: "); users != g.Users || g.Users == 0 || len(g.LeftOut) != 0 {
t.Errorf("the sealed list holds %d users, the key says %d, left out %v", users, g.Users, g.LeftOut)
}
withKey(delivery.UserListWritten{Source: delivery.GrantsUserListSource, Revision: g.Revision, Wanted: g.Digest, Digest: g.Digest, Users: g.Users, Follows: delivery.FollowGrants})
if e := effect(); !strings.HasPrefix(e, "switched") {
t.Errorf("effect once the bus node loaded the written list = %s", e)
}
withKey(loadedBefore)
// After take-over a list composed past a failed read, or one dropping a user of the list in force for
// want of a hash, is not sealed: the list in force stays, and the pass says why.
b.refused[grants.ReadAccounts] = true
if res := p.Compose(); res.Sealed == nil || !strings.Contains(res.Sealed.Why, "a read of this pass failed") {
t.Errorf("a pass past a failed read = %+v", res.Sealed)
}
// While the hold lasts the warning stands, said once however many passes find it; it names why.
p.Compose()
nHeld := 0
for _, c := range said {
if c.Key == grants.HeldCondition && c.State == asks.StateRaised && c.Severity == asks.SeverityWarning && strings.Contains(c.Summary, "a read of this pass failed") {
nHeld++
}
}
if nHeld != 1 {
t.Errorf("the held list raised the warning %d times, want once: %+v", nHeld, said)
}
b.refused[grants.ReadAccounts] = false
if now, _, _ := p.Sealed(); now.Digest != g.Digest || now.Object != g.Object {
t.Errorf("the list in force moved past a failed read: %s, was %s", now.Digest, g.Digest)
}
b.mu.Lock()
hash := b.states[grants.ReadAccounts]["a.mod-x"]
delete(b.states[grants.ReadAccounts], "a.mod-x")
b.mu.Unlock()
if res := p.Compose(); res.Sealed == nil || !strings.Contains(res.Sealed.Why, "a.mod-x would be left out") {
t.Errorf("a pass that drops a.mod-x's hash = %+v", res.Sealed)
}
if now, _, _ := p.Sealed(); now.Digest != g.Digest {
t.Errorf("the list in force moved without a.mod-x: %s", now.Digest)
}
// A list in force written before the part kept its users (no sealed-users): checked against the users the
// bus loaded, so the first list after take-over has the same protection.
b.mu.Lock()
delete(b.own[grants.StateUserList], grants.KeyWrittenUsers)
b.mu.Unlock()
if res := p.Compose(); res.Sealed == nil || !strings.Contains(res.Sealed.Why, "a.mod-x would be left out") {
t.Errorf("a pass with no kept users that drops a.mod-x's hash = %+v", res.Sealed)
}
if now, _, _ := p.Sealed(); now.Digest != g.Digest {
t.Errorf("the list in force moved without a.mod-x and no kept users: %s", now.Digest)
}
b.put(grants.ReadAccounts, "a.mod-x", json.RawMessage(hash))
if p.Compose(); func() bool { now, _, _ := p.Sealed(); return now.Digest != g.Digest }() {
t.Errorf("the list in force moved with every read and hash back")
}
if last := said[len(said)-1]; last.Key != grants.HeldCondition || last.State != asks.StateCleared {
t.Errorf("the held warning is not cleared once nothing is held: %+v", last)
}
// A part started while the warning stood (the said state is in memory only) clears it on its first pass.
var fresh []asks.Condition
fp := partOn(b)
fp.Objects = p.Objects
fp.Asker, fp.Say = fa, func(c asks.Condition) error { fresh = append(fresh, c); return nil }
fp.Compose()
if len(fresh) == 0 || fresh[0].Key != grants.HeldCondition || fresh[0].State != asks.StateCleared {
t.Errorf("a fresh part's first pass does not say the held warning is cleared: %+v", fresh)
}
}
})
}
t.Run("take-over with only a user placement no longer names is refused for want of records alone", func(t *testing.T) {
b.states[grants.ReadAssignments] = nil
b.assign("a", "nats")
b.assign("a", "mod-x")
b.assign("b", "mod-y")
b.loaded = append(shadowOf(t, b), grants.User{User: "b.gone"})
err := p.TakeOver("person.operator", "lists agree")
if err == nil || strings.Contains(err.Error(), "differ") || !strings.Contains(err.Error(), grants.NotComposable) {
t.Errorf("take-over = %v, want refused for the enrolment users and persons alone", err)
t.Run("hand-back in shadow is refused at once, with nothing asked", func(t *testing.T) {
n := len(fa.asks)
if _, err := p.HandBack("shanks.node-tools", "a row"); err == nil || !strings.Contains(err.Error(), "nothing to hand back") {
t.Errorf("hand-back in shadow = %v", err)
}
p.Compose()
if len(fa.asks) != n {
t.Errorf("a refused hand-back asked the operator")
}
})
t.Run("an ask that can stay open is closed every way and can be asked again", func(t *testing.T) {
// A warrant the process missed is caught up from the router's record, whatever its outcome.
id := asked("take-over")
p.Compose()
fa.wheres[id] = &asks.Warrant{Ask: id, Outcome: asks.OutcomeChosen, Option: "no"}
p.Compose()
if l := last(); l.Ask != id || l.Said != "declined" {
t.Errorf("a missed warrant was not caught up: %+v", l)
}
// The router ended it with no choice, and the process was not told.
id = asked("take-over")
p.Compose()
fa.wheres[id] = &asks.Warrant{Ask: id, Outcome: asks.OutcomeCancelled}
p.Compose()
if l := last(); l.Ask != id || !strings.Contains(l.Said, "cancelled") {
t.Errorf("an ask the router ended was not closed: %+v", l)
}
// Said closes on any outcome for the open id, and none for another id.
id = asked("take-over")
p.Said(asker.Refused, asks.Warrant{Ask: "other", Outcome: asks.OutcomeRefused}, nil)
if _, err := p.TakeOver("x", "y"); err == nil {
t.Errorf("a warrant for another ask closed the open one")
}
p.Said(asker.Refused, asks.Warrant{Ask: id, Outcome: asks.OutcomeRefused}, nil)
// A warrant for an ask that is not the open one switches nothing.
if err := p.Warranted(yes, warrant("take-over-other")); err == nil {
t.Errorf("a warrant for an ask that is not open was acted on")
}
// Past its expiry and a margin the part closes it itself; the clock moves, not a timer.
id = asked("take-over")
now := p.Now
p.Now = func() time.Time { return now().Add(25 * time.Hour) }
p.Compose()
p.Now = now
if l := last(); l.Ask != id || l.Said != string(asks.OutcomeExpired) {
t.Errorf("an expired ask was not closed: %+v", l)
}
// An ask that could not be published is closed as refused, its half-made record taken away, and is not retried.
fa.fail = true
id = asked("take-over")
p.Compose()
p.Compose()
if l := last(); l.Ask != id || l.Said != "refused" || !slices.Contains(forgot, id) || !slices.Contains(fa.cancels, id) {
t.Errorf("a failed ask stays open, or is not taken back at the router: %+v, forgot %v, cancelled %v", l, forgot, fa.cancels)
}
fa.fail = false
// An answer the part's own record cannot take yet (its state out of reach) leaves the ask open, closed by
// nothing, and is taken at a later pass.
id = asked("take-over")
p.Compose()
fa.wheres[id] = &asks.Warrant{Ask: id, Outcome: asks.OutcomeChosen, Option: "no"}
fa.takeErr = errors.New("nats: no responders available for request")
p.Compose()
if l := last(); l.Ask != id || l.Said != "asked" {
t.Errorf("an answer not taken closed the ask: %+v", l)
}
fa.takeErr = nil
p.Compose()
if l := last(); l.Ask != id || l.Said != "declined" {
t.Errorf("the answer was not taken once its record could be reached: %+v", l)
}
asked("take-over")
})
t.Run("only memberships published, nothing asserted", func(t *testing.T) {
@@ -1000,6 +1398,51 @@ func TestGrants(t *testing.T) {
})
}
type fakeAsker struct {
asks []asks.Ask
wheres map[string]*asks.Warrant // the router's warrants, by ask
fail bool
cancels []string
// takeErr makes Take fail as the asker does when the part's own record cannot be reached: no handling, an error.
takeErr error
}
func (f *fakeAsker) Cancel(id string) error { f.cancels = append(f.cancels, id); return nil }
// fakeObjects is the part's object bucket of sealed lists.
type fakeObjects struct{ objects map[string][]byte }
func (o *fakeObjects) PutObject(name string, data []byte) error {
if o.objects == nil {
o.objects = map[string][]byte{}
}
o.objects[name] = data
return nil
}
func (o *fakeObjects) DeleteObject(name string) error { delete(o.objects, name); return nil }
func (f *fakeAsker) Ask(a asks.Ask) error {
if f.fail {
return errors.New("the bus refused the publish")
}
f.asks = append(f.asks, a)
return nil
}
func (f *fakeAsker) Where(id string) (string, *asks.Warrant, error) { return "open", f.wheres[id], nil }
// Take hands the warrant to act as the SDK's asker does for one that chose, and records any other.
func (f *fakeAsker) Take(w asks.Warrant, act func(asks.Option, asks.Warrant) error) (asker.Handled, error) {
if f.takeErr != nil {
return "", f.takeErr
}
if w.Outcome != asks.OutcomeChosen {
return asker.Recorded, nil
}
return asker.Acted, act(asks.Option{ID: w.Option}, w)
}
func nonNil(s []string) []string {
if s == nil {
return []string{}
+34 -7
View File
@@ -28,14 +28,20 @@ type Follower struct {
Logf func(string, ...any)
// Asked watches the part's own ask to compose now (a tool's compose-now), waking at each; nil watches none.
Asked func(wake func()) error
// Hear hears the controller seat's condition events (Consumes) and hands each, by event name, with its body, to
// handle; nil hears none. A condition of the kind root-not-free raised or cleared wakes a pass: a machine's
// root-free word changed.
Hear func(handle func(event string, body []byte)) error
wake chan struct{}
once sync.Once
mu sync.Mutex
asked bool
watched map[string]bool
refused map[string]string
have map[string]string
wake chan struct{}
once sync.Once
mu sync.Mutex
asked bool
heard_ bool
rootSaid string
watched map[string]bool
refused map[string]string
have map[string]string
// held and notAsserted are what the last pass held back and could not create, said again only when they change.
held string
notAsserted string
@@ -116,6 +122,18 @@ func (f *Follower) open() {
f.asked = true
}
}
if f.Hear != nil && !f.heard_ {
err := f.Hear(func(event string, body []byte) {
if IsRootNotFree(body) {
f.Wake()
}
})
if err != nil {
f.logf("the controller's condition events cannot be heard yet (%v): tried again at the next change", err)
} else {
f.heard_ = true
}
}
for _, state := range Followed {
f.mu.Lock()
open := f.watched[state]
@@ -175,6 +193,15 @@ func (f *Follower) said(res Result) {
}
}
}
if said := strings.Join(res.RootNotFree, "\n"); said != f.rootSaid {
f.rootSaid = said
if len(res.RootNotFree) == 0 {
f.logf("every machine a composition needs a root-free word for is free")
}
for _, n := range res.RootNotFree {
f.logf("root-free: %s (a verified sender is composed as from no root-free machine there)", n)
}
}
for _, a := range res.Asserted {
f.logf("asserted %s", a)
}
+56 -8
View File
@@ -4,10 +4,11 @@
// state it takes effect in two things alone, each by a grant the operator allowed: it publishes memberships (2026-10-25
// 22:31), and creates a bucket, log or stream a manifest declares that the bus lacks (2026-10-26 08:54, issue 1790). It
// changes no stream the bus holds, writes no grant and no user list the bus node reads, until the operator's take-over,
// which this milestone always refuses.
// which the operator asks on their channel and the part runs on their yes, after its refusals.
package grants
import (
"crypto/sha256"
"encoding/json"
"fmt"
"slices"
@@ -18,6 +19,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
)
// The states of other parts the grants part reads, as its manifest's reads name them (`<module>.<state>`).
@@ -46,7 +48,7 @@ const (
)
// Invokes is every tool the part calls.
var Invokes = []string{CallUsers, CallUserCan, CallStreams}
var Invokes = []string{CallUsers, CallUserCan, CallStreams, CallRootFree}
// Reader reads another part's state, named `<module>.<state>`. A refusal (a read not granted yet, a bucket not there) is
// an error the part names as a gap and composes past: it never stops for one.
@@ -64,6 +66,9 @@ type Caller interface {
type Gap struct {
Input string `json:"input"`
Why string `json:"why"`
// Failed says the gap is a read that failed this pass, not one that stands (ADR 0547 clause 7: after take-over a
// list composed past a failed read is not sealed).
Failed bool `json:"failed,omitempty"`
}
// Standing gaps: what the controller composes from its own store, which no bus value carries in this milestone.
@@ -191,14 +196,14 @@ func (c *Manifests) get(r Reader, digest string) (catalogue.Manifest, error) {
func each(r Reader, state string, gaps *[]Gap) map[string][]byte {
keys, err := r.Keys(state)
if err != nil {
*gaps = append(*gaps, Gap{Input: state, Why: "refused or absent: " + err.Error()})
*gaps = append(*gaps, Gap{Input: state, Why: "refused or absent: " + err.Error(), Failed: true})
return map[string][]byte{}
}
out := map[string][]byte{}
for _, k := range keys {
v, found, err := r.Get(state, k)
if err != nil {
*gaps = append(*gaps, Gap{Input: state + " " + k, Why: err.Error()})
*gaps = append(*gaps, Gap{Input: state + " " + k, Why: err.Error(), Failed: true})
continue
}
if found {
@@ -227,7 +232,7 @@ func ReadInputs(r Reader, cache *Manifests) Inputs {
}
haveKeys, err := r.Keys(ReadHave)
if err != nil {
in.Gaps = append(in.Gaps, Gap{Input: ReadHave, Why: "refused or absent: " + err.Error()})
in.Gaps = append(in.Gaps, Gap{Input: ReadHave, Why: "refused or absent: " + err.Error(), Failed: true})
}
for _, k := range haveKeys {
if node, _, isNode, err := delivery.ParseHaveKey(k); err == nil && isNode {
@@ -241,7 +246,7 @@ func ReadInputs(r Reader, cache *Manifests) Inputs {
sort.Strings(in.Nodes)
if in.BusNode != "" {
if raw, found, err := r.Get(ReadHave, in.BusNode+"."+BusModule); err != nil {
in.Gaps = append(in.Gaps, Gap{Input: "the loaded list's have", Why: err.Error()})
in.Gaps = append(in.Gaps, Gap{Input: "the loaded list's have", Why: err.Error(), Failed: true})
} else if found {
if h, err := delivery.DecodeHave(raw); err == nil {
in.Loaded = h.UserList
@@ -264,11 +269,15 @@ func ReadInputs(r Reader, cache *Manifests) Inputs {
// assigned ones, since a seat is declared by one module and held by another.
indexed, err := r.Keys(ReadModules)
if err != nil {
in.Gaps = append(in.Gaps, Gap{Input: ReadModules, Why: "refused or absent: " + err.Error()})
in.Gaps = append(in.Gaps, Gap{Input: ReadModules, Why: "refused or absent: " + err.Error(), Failed: true})
}
for _, module := range indexed {
raw, found, err := r.Get(ReadModules, module)
if err != nil || !found {
if err != nil {
in.Gaps = append(in.Gaps, Gap{Input: ReadModules + " " + module, Why: err.Error(), Failed: true})
continue
}
if !found {
continue
}
v, err := delivery.DecodeModule(raw)
@@ -289,6 +298,31 @@ func ReadInputs(r Reader, cache *Manifests) Inputs {
in.Manifests[module] = m
in.Digests[module] = v.Manifest
}
// A module the mesh provides itself has no repository the indexer reads: its manifest comes with mesh-sdk, and the
// controller's Provide overwrites any row of it at every start, so the provided one always wins here too (issue 1945,
// the private network on every machine). An indexer manifest of the same name is said as a gap, never used.
if slices.Contains(modules, overlay.Name) {
raw, err := json.Marshal(overlay.Manifest())
var m catalogue.Manifest
if err == nil {
m, err = catalogue.ParseManifestForInputs(raw)
}
in.Gaps = slices.DeleteFunc(in.Gaps, func(g Gap) bool { return g.Input == "manifest of "+overlay.Name })
in.Refused = slices.DeleteFunc(in.Refused, func(g Gap) bool { return g.Input == "manifest of "+overlay.Name })
if _, held := in.Digests[overlay.Name]; held {
in.Gaps = append(in.Gaps, Gap{Input: "manifest of " + overlay.Name, Why: "the indexer also holds one, which is not used: the mesh provides this module's manifest itself, and the controller overwrites any row of it"})
}
if err != nil {
g := Gap{Input: "manifest of " + overlay.Name, Why: "the manifest mesh-sdk provides is refused by its own reader: " + err.Error()}
in.Refused = append(in.Refused, g)
in.Gaps = append(in.Gaps, g)
delete(in.Manifests, overlay.Name)
delete(in.Digests, overlay.Name)
} else {
in.Manifests[overlay.Name] = m
in.Digests[overlay.Name] = fmt.Sprintf("provided:sha256:%x", sha256.Sum256(raw))
}
}
for _, module := range modules {
if _, ok := in.Manifests[module]; !ok && !gapNamed(in.Gaps, "manifest of "+module) {
in.Gaps = append(in.Gaps, Gap{Input: "manifest of " + module,
@@ -361,3 +395,17 @@ func gapNamed(gaps []Gap, input string) bool {
func sortGaps(g []Gap) {
sort.SliceStable(g, func(i, j int) bool { return strings.Compare(g[i].Input, g[j].Input) < 0 })
}
// ReadFailures names every read this pass could not make and every manifest refused: what a list composed now may lack.
func (in Inputs) ReadFailures() []string {
var out []string
for _, g := range in.Gaps {
if g.Failed {
out = append(out, g.Input+": "+g.Why)
}
}
for _, g := range in.Refused {
out = append(out, g.Input+": "+g.Why)
}
return out
}
@@ -22,7 +22,7 @@ import (
// MembershipGaps are what a membership carries that the part cannot compose from the bus.
var MembershipGaps = []Gap{
{Input: "membership: private network and received contributions", Why: "kept in each node's composition, which only that node reads: carried over from the membership last published, none for a first one"},
{Input: "membership: root-free machines", Why: "judged by the controller on the bus at its composition, not kept: a verified sender is composed as from no root-free machine"},
{Input: "membership: root-free machines", Why: "asked of the controller's root-free verb at each pass and kept for that pass alone; a machine it does not call free (an error included) is composed as not free, and the pass says why"},
}
// Publisher publishes a membership on the assignments stream (membership.Subject), last per subject, and reads the one
+58 -105
View File
@@ -6,12 +6,13 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -42,9 +43,6 @@ const (
// Emits is every event the part says.
var Emits = []string{EventAsserted, EventMembership, EventUserList}
// Shadow is the switch's one state in this milestone: the bus node follows the controller's list.
const Shadow = "shadow"
// Store is the part's own state, by local name: the only thing the part writes.
type Store interface {
Get(state, key string) ([]byte, uint64, bool, error)
@@ -100,9 +98,14 @@ type Result struct {
NotAsserted []string `json:"not_asserted,omitempty"`
// Refused is every manifest the indexer holds that this pass could not read, `manifest of <module>: <why>`: its
// module counts as placed nowhere (issue 1945).
Refused []string `json:"refused,omitempty"`
Errors []string `json:"errors,omitempty"`
PublishedOrAsserted string `json:"published_or_asserted"`
Refused []string `json:"refused,omitempty"`
// RootNotFree is each machine the composition needed a root-free word for and did not get one, `<machine>: not
// free, <why>`: a verified sender is composed as from no root-free machine there (issue 1945).
RootNotFree []string `json:"root_not_free,omitempty"`
// Sealed is the list sealed to the bus node as the key `user-list.sealed` names it (ADR 0547 clause 7).
Sealed *SealedState `json:"sealed,omitempty"`
Errors []string `json:"errors,omitempty"`
PublishedOrAsserted string `json:"published_or_asserted"`
}
// NothingTakesEffect is what every composition says it did on the bus beyond its own state and the memberships.
@@ -118,7 +121,21 @@ type Part struct {
Publish Publisher
// Create creates a declared stream the bus lacks: the process's connection; nil in the tools, which ask the process.
Create Creator
Now func() time.Time
// Asker asks the operator (take-over, hand-back): the process's client of the router; nil in the tools, which write
// the request and let the process ask it.
Asker Asker
// Logf logs what no state or condition says (an error of acting on an answer): the process's; nil in the tools.
Logf func(string, ...any)
// Objects is the part's object bucket of sealed lists: the process's connection; nil in the tools.
Objects Objects
// Say says a condition to the operator: the process's; nil in the tools.
Say func(asks.Condition) error
// Forget takes away the record the asker kept of an ask it failed to publish; Nonce is a test's id suffix.
Forget func(id string) error
Nonce func() string
// deciding is set while a take-over is decided, so the composition it makes asks and reconciles nothing.
deciding atomic.Bool
Now func() time.Time
cache Manifests
mu sync.Mutex
@@ -126,6 +143,12 @@ type Part struct {
made creations
// askSeen is the revision of the last ask to compose (KeyAsked) this part acted on.
askSeen uint64
// heldSaid is the reason of the held-list condition last said (empty: none raised); heldKnown is whether one was said
// since the start, so the first pass after a start always says its state (a condition raised before a restart is cleared
// by it). holdMu guards both.
holdMu sync.Mutex
heldSaid string
heldKnown bool
}
// Ask is the value of KeyAsked: a tool's ask to compose now, and, from grants_assert, to create one module's missing
@@ -172,7 +195,8 @@ func (p *Part) Compose() Result {
res.Refused = append(res.Refused, g.Input+": "+g.Why)
}
recs, left := Records(in)
st := UserListState{V: 1, Mode: Shadow, LeftOut: left, Loaded: in.Loaded, BusNode: in.BusNode,
res.RootNotFree = WithRootFree(&recs, p.Call)
st := UserListState{V: 1, Mode: p.SwitchState(), LeftOut: left, Loaded: in.Loaded, BusNode: in.BusNode,
Gaps: append(append([]Gap(nil), in.Gaps...), MembershipGaps...), At: p.now()}
if st.LeftOut == nil {
st.LeftOut = []Gap{}
@@ -184,7 +208,7 @@ func (p *Part) Compose() Result {
users = WithHashes(users, in.Hashes)
st.Digest, st.Users = Digest(users), len(users)
loaded, err := LoadedUsers(p.Call)
st.Compare = Compare(users, loaded, Placed(in))
st.Compare = Compare(users, loaded, Placed(in)).Excusing(ExcusedBy(MultiHeld(recs)))
if err != nil {
st.Compare.Unread = err.Error()
}
@@ -207,6 +231,17 @@ func (p *Part) Compose() Result {
}
}
}
// The list itself, sealed to the bus node and named in `user-list.sealed` (ADR 0547 clause 7).
if list, n, leftOut, err := ComposeList(recs, in.Hashes); err != nil {
res.Errors = append(res.Errors, "the user list could not be composed whole: "+err.Error())
} else {
// The users the bus loaded: what a first list after take-over must keep where the part has not yet written one.
var bus []string
if st.Compare.Unread == "" {
bus = names(loaded)
}
p.writeSealed(&res, list, n, leftOut, in, bus)
}
ms := Memberships(recs)
res.Memberships = len(ms)
for _, key := range sortedKeys(ms) {
@@ -255,6 +290,14 @@ func (p *Part) Compose() Result {
res.Errors = append(res.Errors, fmt.Sprintf("its own %s cannot be written: %v", StateTakeover, err))
}
}
if !p.deciding.Load() {
if err := p.RaiseRequested(); err != nil {
res.Errors = append(res.Errors, err.Error())
}
if err := p.Reconcile(); err != nil {
res.Errors = append(res.Errors, err.Error())
}
}
return res
}
@@ -311,100 +354,10 @@ func sortedKeys[V any](m map[string]V) []string {
return out
}
// Switch is the value of `takeover`: which list the bus node follows, and every act on the switch.
type Switch struct {
V int `json:"v"`
State string `json:"state"`
Acts []Act `json:"acts"`
}
// Act is one take-over or hand-back asked, done or refused.
type Act struct {
Act string `json:"act"`
By string `json:"by"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
Refused string `json:"refused,omitempty"`
}
// keptActs is how many acts the switch keeps, newest last.
const keptActs = 50
// Operator says whether a caller is the operator's credential: a person's own bus account (`person.<name>`), which no
// session of an agent holds (an agent calls as its machine's tool runner).
func Operator(caller string) bool {
name, ok := strings.CutPrefix(caller, "person.")
return ok && name != ""
}
// NotComposable is why take-over is refused in this milestone, whatever the lists say.
const NotComposable = "the enrolment users and the persons cannot be composed: they live only in the controller's store, and take-over waits until they have records on the bus (ADR 0547 clause 9, version 26)"
// record keeps an act on the switch; a refused write is said in the answer.
func (p *Part) record(a Act) string {
raw, _, found, err := p.Store.Get(StateTakeover, KeySwitch)
if err != nil {
return fmt.Sprintf("; the act is not recorded: %v", err)
func names(us []User) []string {
out := make([]string, 0, len(us))
for _, u := range us {
out = append(out, u.User)
}
s := Switch{V: 1, State: Shadow}
if found {
_ = json.Unmarshal(raw, &s)
}
s.Acts = append(s.Acts, a)
if len(s.Acts) > keptActs {
s.Acts = s.Acts[len(s.Acts)-keptActs:]
}
data, _ := json.Marshal(s)
if _, err := p.Store.Put(StateTakeover, KeySwitch, data); err != nil {
return fmt.Sprintf("; the act is not recorded: %v", err)
}
return ""
}
// TakeOver is the operator's switch to the grants part's list. It is refused without the operator's credential, while
// any user of the shadow list differs from the loaded list (but those both agree are gone), and in this milestone
// always, since the enrolment users and the persons cannot be composed. Every ask is recorded.
func (p *Part) TakeOver(caller, why string) error {
a := Act{Act: "take-over", By: caller, Why: why, At: p.now()}
if !Operator(caller) {
a.Refused = fmt.Sprintf("take-over is the operator's alone: %s is not a person's account", caller)
return fmt.Errorf("%s%s", a.Refused, p.record(a))
}
res := p.Compose()
var reasons []string
if c := res.UserList.Compare; !c.Equal() {
if c.Unread != "" {
reasons = append(reasons, "the loaded list could not be read: "+c.Unread)
}
if d := c.Differing(); len(d) > 0 {
reasons = append(reasons, fmt.Sprintf("%d users differ between the shadow list and the loaded list: %s", len(d), strings.Join(d, ", ")))
}
}
cr, gone := res.UserList.Credentials, res.UserList.Compare.Gone
var unhashed []string
for _, u := range append(append([]string(nil), cr.LeftOut...), cr.LosesLogin...) {
if !slices.Contains(gone, u) {
unhashed = append(unhashed, u)
}
}
if len(unhashed) > 0 {
sort.Strings(unhashed)
reasons = append(reasons, fmt.Sprintf("the vault holds no hash for %d users (ADR 0547 clause 9: handed or adopted before take-over): %s",
len(unhashed), strings.Join(unhashed, ", ")))
}
reasons = append(reasons, NotComposable)
a.Refused = "take-over refused: " + strings.Join(reasons, "; ")
return fmt.Errorf("%s%s", a.Refused, p.record(a))
}
// HandBack is the operator's switch back to the controller's list. Refused without the operator's credential, and
// while the part is in shadow, since the bus node follows the controller's list already. Every ask is recorded.
func (p *Part) HandBack(caller, why string) error {
a := Act{Act: "hand-back", By: caller, Why: why, At: p.now()}
if !Operator(caller) {
a.Refused = fmt.Sprintf("hand-back is the operator's alone: %s is not a person's account", caller)
return fmt.Errorf("%s%s", a.Refused, p.record(a))
}
a.Refused = "nothing to hand back: the grants part is in shadow and never took over, so the bus node follows the controller's list already"
return fmt.Errorf("%s%s", a.Refused, p.record(a))
return out
}
@@ -0,0 +1,122 @@
package grants
import (
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscompose"
)
// CallRootFree is the controller seat's verb that judges machines root-free (novox/hq ADR 0259 §8): the one judgement
// the composer needs and the part cannot make from the bus (issue 1945). Only the controller answers it, on purpose.
const CallRootFree = "seat:mesh-controller.root-free"
// The controller seat's condition events the part hears (never a timer): a machine's root-free word changes with the
// raising and the clearing of its `root-not-free` condition.
const (
EventConditionRaised = "mesh-controller.condition-raised"
EventConditionCleared = "mesh-controller.condition-cleared"
// KindRootNotFree is the condition kind whose events wake a pass.
KindRootNotFree = "root-not-free"
)
// Consumes is every event the part hears, as its manifest's consumes name them.
var Consumes = []string{EventConditionRaised, EventConditionCleared}
// EventSubject is the bus subject of a controller seat event named `mesh-controller.<event>`.
func EventSubject(event string) string {
return "mesh.seat.mesh-controller.event." + strings.TrimPrefix(event, "mesh-controller.")
}
// IsRootNotFree says a condition event's body names the `root-not-free` kind (its `kind`, or the last part of its
// `key`, `<scope>.<id>.<kind>`). An unreadable body wakes a pass too: a pass is cheap, a missed change is not.
func IsRootNotFree(body []byte) bool {
var c struct {
Kind string `json:"kind"`
Key string `json:"key"`
}
if json.Unmarshal(body, &c) != nil {
return true
}
return c.Kind == KindRootNotFree || (c.Kind == "" && strings.HasSuffix(c.Key, "."+KindRootNotFree)) || (c.Kind == "" && c.Key == "")
}
// RootFreeNodes are the machines a composition needs a root-free word for: where the router's seat is held and where a
// kinded seat's holder declares `verified-sender`. None when no holder declares it. Sorted. Pure.
func RootFreeNodes(r buscompose.Records) []string {
var nodes []string
need := false
for node, declared := range r.Assigned {
for _, d := range declared {
for _, s := range d.Holds {
if s.Name == "operator-channel" && !slices.Contains(nodes, node) {
nodes = append(nodes, node)
}
if s.Kinded && s.Kind != "" && slices.Contains(s.Capabilities, "verified-sender") {
need = true
if !slices.Contains(nodes, node) {
nodes = append(nodes, node)
}
}
}
}
}
if !need {
return nil
}
sort.Strings(nodes)
return nodes
}
// verdict is one machine's answer of the controller's root-free verb.
type verdict struct {
Machine string `json:"machine"`
Free bool `json:"free"`
Why string `json:"why"`
}
// WithRootFree fills r.RootFree for the machines the composition needs by asking the controller once, and answers a
// line for each machine that is not free, `<machine>: <why>`. It fails closed: an error, no answer or a machine
// left out is not free, and says so. Pure but for the one call; the answer is kept for this pass alone.
func WithRootFree(r *buscompose.Records, call Caller) []string {
nodes := RootFreeNodes(*r)
if len(nodes) == 0 {
return nil
}
free := map[string]verdict{}
var failure string
if call == nil {
failure = "no way to ask the controller here"
} else if raw, err := call.Call(CallRootFree, map[string]any{"machines": nodes}); err != nil {
failure = "the controller's root-free word could not be asked: " + err.Error()
} else {
var body struct {
Machines []verdict `json:"machines"`
}
if err := json.Unmarshal(raw, &body); err != nil {
failure = "the controller's root-free answer could not be read: " + err.Error()
}
for _, v := range body.Machines {
free[v.Machine] = v
}
}
r.RootFree = map[string]bool{}
var notes []string
for _, n := range nodes {
v, answered := free[n]
switch {
case answered && v.Free:
r.RootFree[n] = true
case failure != "":
notes = append(notes, fmt.Sprintf("%s: not free, %s", n, failure))
case !answered:
notes = append(notes, fmt.Sprintf("%s: not free, the controller named no verdict for it", n))
default:
notes = append(notes, fmt.Sprintf("%s: not free, %s", n, v.Why))
}
}
return notes
}
@@ -0,0 +1,331 @@
package grants
import (
"encoding/json"
"fmt"
"regexp"
"slices"
"strings"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **The list the bus node loads from take-over** (novox/hq ADR 0547 clause 7, issue 1945). Each pass composes the bus's
// user list with the vault's hashes (the accounts block, buscompose.ComposeAccounts, as the bus module's file holds it),
// seals it to the bus node's key (the key the node says in its have of the bus module) and puts it as one object of the
// part's object bucket, named by its digest; then writes the key `user-list.sealed` naming it, with the revision, the user
// count, the users left out for want of a hash, and which list the bus node follows (the switch, clause 8). The object
// and the key are written in shadow too: the bus node reads the key and follows the controller's list until the key says
// FollowGrants, so a take-over has a list to load at once. Only the process writes them (Objects); a tool asks it.
// KeySealed is the key of the state user-list the bus node reads (delivery.GrantsUserListKey).
const KeySealed = delivery.GrantsUserListKey
// Objects is the part's object bucket of sealed lists: the process's connection; nil in the tools.
type Objects interface {
PutObject(name string, data []byte) error
DeleteObject(name string) error
}
// ComposeList is the bus's user list as the file holds it, with each user's hash from the vault, and the users left out
// for want of one. Pure.
func ComposeList(r buscompose.Records, hashes map[string]HeldHash) ([]byte, int, []string, error) {
principals, err := buscompose.Users(r)
if err != nil {
return nil, 0, nil, err
}
held := map[string]string{}
for user, h := range hashes {
held[user] = h.Hash
}
filled, missing := buscompose.WithPasswords(principals, held)
list, err := buscompose.ComposeAccounts(filled)
if err != nil {
return nil, 0, nil, err
}
slices.Sort(missing)
return []byte(list), len(filled), missing, nil
}
// Sealed is the part's key `user-list.sealed` as written, and whether there is one.
func (p *Part) Sealed() (delivery.GrantsUserList, uint64, bool) {
raw, rev, found, err := p.Store.Get(StateUserList, KeySealed)
if err != nil || !found {
return delivery.GrantsUserList{}, 0, false
}
g, err := delivery.DecodeGrantsUserList(raw)
if err != nil {
return delivery.GrantsUserList{}, 0, false
}
return g, rev, true
}
// followOf is which list the switch says the bus node follows.
func followOf(state string) string {
if state == TookOver {
return delivery.FollowGrants
}
return delivery.FollowController
}
// writeSealed puts the list sealed to the bus node and writes the key, each only when what it says changed. It says in
// res what it wrote, or why it wrote nothing.
func (p *Part) writeSealed(res *Result, list []byte, users int, leftOut []string, in Inputs, bus []string) {
cur, _, had := p.Sealed()
next := cur
next.Follow = followOf(p.SwitchState())
key := ""
if in.Loaded != nil {
key = in.Loaded.Key
}
digest := delivery.UserListDigest(list)
why, held := "", ""
switch {
case p.Objects == nil:
// The tools: the process writes; the follow alone may change here, from a tool's view of the switch, and is not.
res.Sealed = sealedSays(cur, had, "")
return
case in.BusNode == "":
why = "no node runs the bus's module, so the list is sealed to nobody"
case key == "":
why = "the bus node's have of the bus module names no key yet (its node-engine says it from this release)"
case next.Follow == delivery.FollowGrants && cur.Digest != digest && p.holdsBack(in, leftOut, bus) != "":
// After take-over the list in force is never replaced by one composed past a failed read, or one that drops a user
// of it for want of a hash: that user's module would be locked out of the bus (review of the switch).
why = p.holdsBack(in, leftOut, bus)
held = why
case cur.Digest != digest || cur.SealedTo != delivery.SealingKeyID(key) || cur.Node != in.BusNode:
object, err := delivery.SealUserList(list, key)
if err == nil {
err = p.Objects.PutObject(delivery.UserListObjectName(digest), object)
}
if err != nil {
res.Errors = append(res.Errors, "the sealed user list could not be put: "+err.Error())
res.Sealed = sealedSays(cur, had, "")
return
}
if cur.Digest != digest {
next.Revision++
}
next.Digest, next.Object, next.SealedTo, next.Node = digest, delivery.UserListObjectName(digest), delivery.SealingKeyID(key), in.BusNode
}
p.sayHeld(held)
if next.Digest == digest {
next.Users, next.LeftOut = users, leftOut
}
if next.Digest == "" {
next.Users, next.LeftOut = 0, nil
}
if had && sameSealed(cur, next) {
res.Sealed = sealedSays(cur, had, why)
return
}
next.At = p.now()
data, err := delivery.EncodeGrantsUserList(next)
if err == nil {
_, err = p.Store.Put(StateUserList, KeySealed, data)
}
if err != nil {
res.Errors = append(res.Errors, fmt.Sprintf("its own %s %s cannot be written: %v", StateUserList, KeySealed, err))
res.Sealed = sealedSays(cur, had, "")
return
}
if next.Digest == digest && cur.Digest != digest {
p.keepWrittenUsers(res, list)
}
// The object the key named before is taken away once the key names the new one: the bucket keeps the list in force.
if had && cur.Object != "" && cur.Object != next.Object {
if err := p.Objects.DeleteObject(cur.Object); err != nil {
res.Errors = append(res.Errors, "the sealed list before could not be taken away: "+err.Error())
}
}
res.Sealed = sealedSays(next, true, why)
p.say(res, EventUserList, map[string]any{"digest": next.Digest, "revision": next.Revision, "users": next.Users,
"left_out": len(next.LeftOut), "follow": next.Follow, "sealed_to": next.Node})
}
// sayHeld raises the warning condition mesh-grants.user-list-held while the list in force is kept (why, which words the pass's
// `held: ...`), and clears it on the first pass that holds nothing back: the list advanced, or the reason went. The refusal
// condition's way (sayRefused): said once per reason, again when the reason changes, and tried again when a say failed.
func (p *Part) sayHeld(why string) {
if p.Say == nil {
return
}
p.holdMu.Lock()
defer p.holdMu.Unlock()
if p.heldKnown && why == p.heldSaid {
return
}
c := asks.Condition{Key: HeldCondition, Subject: asks.Scope{Scope: "module", ID: "mesh-grants"}}
if why == "" {
c.State = asks.StateCleared
} else {
reason := strings.TrimPrefix(why, "held: ")
c.Severity, c.State = asks.SeverityWarning, asks.StateRaised
c.Summary, c.Headline, c.Needs = "the user list is held: "+reason, "The user list in force is held back", "a look at why"
c.Explanation = asks.NeedsYou + "a look at why. The bus node keeps following the list written before, not the one composed now: " + reason
c.Resolved = "a pass finds nothing to hold back and the list advances"
}
if p.Say(c) == nil {
p.heldSaid, p.heldKnown = why, true
}
}
// HeldCondition is the key of the warning raised while a list in force is held back after take-over.
const HeldCondition = "mesh-grants.user-list-held"
func sameSealed(a, b delivery.GrantsUserList) bool {
a.At, b.At = b.At, a.At
x, _ := json.Marshal(a)
y, _ := json.Marshal(b)
return string(x) == string(y)
}
// SealedState is what a pass says of the sealed list: the key as written, and why nothing was written when nothing was.
type SealedState struct {
Written *delivery.GrantsUserList `json:"written,omitempty"`
Why string `json:"why,omitempty"`
}
func sealedSays(g delivery.GrantsUserList, had bool, why string) *SealedState {
s := &SealedState{Why: why}
if had {
s.Written = &g
}
return s
}
// syncFollow writes the switch into the key at once, so the bus node turns on the operator's yes, not at the next pass.
func (p *Part) syncFollow(state string) error {
cur, _, had := p.Sealed()
if had && cur.Follow == followOf(state) {
return nil
}
if !had {
cur = delivery.GrantsUserList{}
}
cur.Follow, cur.At = followOf(state), p.now()
data, err := delivery.EncodeGrantsUserList(cur)
if err != nil {
return err
}
_, err = p.Store.Put(StateUserList, KeySealed, data)
return err
}
// Effect is where the switch stands on the bus node: "switched" only once the bus node's loaded list is the grants part's
// written one (its source the part's key, its digest the written digest); otherwise what it waits for.
func (p *Part) Effect(state string, loaded *delivery.UserListWritten) string {
g, _, had := p.Sealed()
fromGrants := loaded != nil && loaded.Source == delivery.GrantsUserListSource
switch {
case state == TookOver && had && fromGrants && g.Digest != "" && loaded.Digest == g.Digest && loaded.Error == "":
return "switched: the bus node loaded the grants part's list " + short(g.Digest) + " (revision " + fmt.Sprint(g.Revision) + ")"
case state == TookOver && !had || state == TookOver && g.Digest == "":
return "recorded, not in effect: the grants part has written no sealed list for the bus node yet"
case state == TookOver && loaded == nil:
return "recorded, not in effect: the bus module's have names no loaded list"
case state == TookOver && loaded.Unwatched != "" && !fromGrants:
return "recorded, not in effect: the bus node does not watch the grants part's key: " + loaded.Unwatched
case state == TookOver && loaded.Error != "":
return "recorded, not in effect: the bus node could not load the grants part's list: " + loaded.Error
case state == TookOver:
return "recorded, not in effect yet: the bus node loaded " + short(loaded.Digest) + " from " + loaded.Source + ", the grants part wrote " + short(g.Digest)
case fromGrants:
return "handed back, not in effect yet: the bus node still follows the grants part's list"
}
return "shadow: the bus node follows the controller's list"
}
func short(d string) string {
d = strings.TrimPrefix(d, "sha256:")
if len(d) > 12 {
d = d[:12]
}
return d
}
// followReasons is what a take-over waits for on the sealed list itself: a list written for the bus node, of the list
// composed now, and the loaded list granting the bus node its read (the controller's composer carries ADR 0547 clause 7's
// grants from the operator's second exception; until its delivery the bus node could not read the first list).
func (p *Part) followReasons(res Result, in Inputs) []string {
var out []string
g, _, had := p.Sealed()
switch {
case !had || g.Digest == "":
why := "no list is written"
if res.Sealed != nil && res.Sealed.Why != "" {
why = res.Sealed.Why
}
out = append(out, "the grants part has written no sealed list for the bus node: "+why)
case in.Loaded == nil || in.Loaded.Key == "" || g.SealedTo != delivery.SealingKeyID(in.Loaded.Key):
out = append(out, "the sealed list is not sealed to the key the bus node says now")
}
if in.BusNode != "" {
node := "node." + in.BusNode
for _, d := range res.UserList.Compare.Differ {
if d.User == node && slices.ContainsFunc(d.PublishAdded, func(s string) bool { return strings.Contains(s, delivery.GrantsUserListBucket) }) {
out = append(out, "the loaded list does not grant "+node+" reading the grants part's sealed list (ADR 0547 clause 7): "+
"the controller's composer carries it from its delivery of the operator's second exception")
}
}
}
return out
}
// KeyWrittenUsers is the key of the state user-list naming the users of the list the key `sealed` names: what a list
// composed after take-over must keep.
const KeyWrittenUsers = "sealed-users"
func (p *Part) keepWrittenUsers(res *Result, list []byte) {
data, _ := json.Marshal(listUsers(list))
if _, err := p.Store.Put(StateUserList, KeyWrittenUsers, data); err != nil {
res.Errors = append(res.Errors, fmt.Sprintf("its own %s %s cannot be written: %v", StateUserList, KeyWrittenUsers, err))
}
}
var listUser = regexp.MustCompile(`\{\s*user:\s*"([^"]+)"`)
// listUsers names the users of a list as the composer writes it.
func listUsers(list []byte) []string {
out := []string{}
for _, m := range listUser.FindAllSubmatch(list, -1) {
out = append(out, string(m[1]))
}
return out
}
// holdsBack says why a list composed now may not replace the one in force after take-over: a read failed this pass, or
// a user of the list in force is left out for want of a hash. The list in force is the one the part wrote, by the users
// it kept of it; where it kept none (a list written before this release), the users the bus loaded (bus, nil when
// they could not be read). Empty when it may.
func (p *Part) holdsBack(in Inputs, leftOut, bus []string) string {
if f := in.ReadFailures(); len(f) > 0 {
return "held: the list in force is kept while a read of this pass failed (" + strings.Join(f, "; ") + ")"
}
raw, _, found, err := p.Store.Get(StateUserList, KeyWrittenUsers)
if err != nil {
return "held: the users of the list in force cannot be read: " + err.Error()
}
var had []string
switch {
case found:
_ = json.Unmarshal(raw, &had)
case bus == nil:
return "held: the users of the list in force are not known: the part kept none, and the loaded list could not be read"
default:
had = bus
}
var lost []string
for _, u := range leftOut {
if slices.Contains(had, u) {
lost = append(lost, u)
}
}
if len(lost) > 0 {
return "held: the list in force is kept, since " + strings.Join(lost, ", ") + " would be left out for want of a hash in the vault"
}
return ""
}
@@ -0,0 +1,491 @@
package grants
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asker"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The switch between the controller's user list and the grants part's (ADR 0547 clause 8). The operator's word is a
// warrant: the operator's answer to an ask on a channel that proves who answered, never a caller's name, since an
// operator's session and every agent's call as the same machine account (the operator, 2026-10-26).
// Shadow is the switch's state until take-over: the bus node follows the controller's list. TookOver is its state after.
const (
Shadow = "shadow"
TookOver = "took-over"
)
// KeyRequest is the key in StateTakeover where a tool leaves its request for the process to ask.
const KeyRequest = "request"
// The two acts the operator may warrant.
const (
ActTakeOver = "take-over"
ActHandBack = "hand-back"
)
// Switch is the value of `takeover`: which list the bus node follows, every act on the switch and the ask open now.
type Switch struct {
V int `json:"v"`
State string `json:"state"`
// Open is the id of the ask the operator has not answered yet, empty when none.
Open string `json:"open,omitempty"`
// Refusing are the acts whose refusal condition was said and not cleared yet.
Refusing []string `json:"refusing,omitempty"`
Acts []Act `json:"acts"`
}
// Act is one take-over or hand-back asked, answered, done or refused.
type Act struct {
Act string `json:"act"`
By string `json:"by"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
// Ask is the router's ask of the act; Said what became of it: asked, switched, declined, expired, refused.
Ask string `json:"ask,omitempty"`
Said string `json:"said,omitempty"`
Refused string `json:"refused,omitempty"`
}
// Request is what a tool leaves for the process to ask: written once per act, marked raised when asked.
type Request struct {
ID string `json:"id"`
Act string `json:"act"`
By string `json:"by"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
Raised bool `json:"raised,omitempty"`
// RaisedAt is when it was asked, and Failed why asking it failed (the ask is then closed as refused).
RaisedAt time.Time `json:"raised_at,omitempty"`
Failed string `json:"failed,omitempty"`
}
// Asker is the client of the router the part asks through (mesh-sdk asker.Client).
type Asker interface {
Ask(a asks.Ask) error
// Where is the router's record of an ask: its state and its warrant, if any.
Where(id string) (string, *asks.Warrant, error)
// Take hands a warrant to act once (a warrant the process missed is caught up through it).
Take(w asks.Warrant, act func(asks.Option, asks.Warrant) error) (asker.Handled, error)
// Cancel takes an ask back at the router, so a copy the operator may still see ends there too.
Cancel(id string) error
}
// askMargin is how long past an ask's expiry the router is given to say it ended, before the part closes it itself;
// recordGrace how long a just-asked ask may be missing from the router's record.
const (
askMargin = 5 * time.Minute
recordGrace = time.Minute
)
// keptActs is how many acts the switch keeps, newest last.
const keptActs = 50
func (p *Part) readSwitch() (Switch, error) {
s := Switch{V: 1, State: Shadow}
raw, _, found, err := p.Store.Get(StateTakeover, KeySwitch)
if err != nil {
return s, err
}
if found {
_ = json.Unmarshal(raw, &s)
}
return s, nil
}
func (p *Part) writeSwitch(s Switch) error {
if len(s.Acts) > keptActs {
s.Acts = s.Acts[len(s.Acts)-keptActs:]
}
if s.Acts == nil {
s.Acts = []Act{}
}
data, _ := json.Marshal(s)
_, err := p.Store.Put(StateTakeover, KeySwitch, data)
return err
}
// SwitchState is which list the bus node follows now.
func (p *Part) SwitchState() string {
if s, err := p.readSwitch(); err == nil && s.State != "" {
return s.State
}
return Shadow
}
// TakeOver is the tool: it asks the operator to switch the bus node to the grants part's list. It writes the request,
// and the part's process asks it on the operator's channel; only the operator's yes (a warrant) runs the switch, after
// the refusals of Decide. Asking again while an ask is open is refused.
func (p *Part) TakeOver(caller, why string) (any, error) { return p.request(ActTakeOver, caller, why) }
// HandBack is the tool: it asks the operator to switch the bus node back to the controller's list.
func (p *Part) HandBack(caller, why string) (any, error) { return p.request(ActHandBack, caller, why) }
// nonce is four hex digits that keep two asks of one second apart.
func (p *Part) nonce() string {
if p.Nonce != nil {
return p.Nonce()
}
var b [2]byte
_, _ = rand.Read(b[:])
return hex.EncodeToString(b[:])
}
func (p *Part) request(act, caller, why string) (any, error) {
s, err := p.readSwitch()
if err != nil {
return nil, fmt.Errorf("the switch cannot be read: %w", err)
}
a := Act{Act: act, By: caller, Why: why, At: p.now()}
if s.Open != "" {
return nil, fmt.Errorf("%s is already asked: the ask %s waits for the operator's answer", act, s.Open)
}
if act == ActHandBack && s.State == Shadow {
a.Refused = "nothing to hand back: the grants part is in shadow and never took over, so the bus node follows the controller's list already"
s.Acts = append(s.Acts, a)
_ = p.writeSwitch(s)
return nil, fmt.Errorf("%s", a.Refused)
}
if act == ActTakeOver && s.State == TookOver {
return nil, fmt.Errorf("the bus node follows the grants part's list already")
}
a.Ask = act + "-" + p.now().Format("20060102t150405") + "-" + p.nonce()
a.Said = "asked"
req, _ := json.Marshal(Request{ID: a.Ask, Act: act, By: caller, Why: why, At: a.At})
if _, err := p.Store.Put(StateTakeover, KeyRequest, req); err != nil {
return nil, fmt.Errorf("the request could not be written: %w", err)
}
s.Open = a.Ask
s.Acts = append(s.Acts, a)
if err := p.writeSwitch(s); err != nil {
return nil, fmt.Errorf("the act could not be recorded: %w", err)
}
// The process asks at its next pass: wake it.
_, _ = p.AskCompose(caller)
return map[string]any{"asked": a.Ask, "says": "the part's process asks the operator on their channel; only the operator's yes records the switch, and it takes effect when the bus node loads the grants part's list. grants_take_over_state shows where it stands"}, nil
}
// RaiseRequested is the process's: it asks the operator the request a tool left, once. Where the part has no asker (the
// tools) it does nothing.
func (p *Part) RaiseRequested() error {
if p.Asker == nil {
return nil
}
raw, rev, found, err := p.Store.Get(StateTakeover, KeyRequest)
if err != nil || !found {
if err != nil {
return fmt.Errorf("its own %s %s cannot be read: %v", StateTakeover, KeyRequest, err)
}
return nil
}
var r Request
if json.Unmarshal(raw, &r) != nil || r.Raised || r.Failed != "" || r.ID == "" {
return nil
}
if s, err := p.readSwitch(); err != nil || s.Open != r.ID {
return nil // an ask no longer open is not asked
}
ask, err := p.AskOf(r)
if err != nil {
return err
}
if err := p.Asker.Ask(ask); err != nil {
// Closed as refused, and the half-made record the asker kept before publishing is taken away, so nothing stays open.
r.Failed = err.Error()
data, _ := json.Marshal(r)
_, _ = p.Store.Put(StateTakeover, KeyRequest, data)
if p.Forget != nil {
_ = p.Forget(r.ID)
}
// A publish that timed out may still have reached the router: taken back there, so a yes to it cannot land on an
// ask the part has forgotten.
if cerr := p.Asker.Cancel(r.ID); cerr != nil && p.Logf != nil {
p.Logf("the ask %s that could not be published was not taken back at the router: %v", r.ID, cerr)
}
if s, rerr := p.readSwitch(); rerr == nil {
_ = p.finish(s, Act{Act: r.Act, By: "operator", At: p.now(), Ask: r.ID, Said: "refused", Refused: "the operator could not be asked: " + err.Error()})
}
return fmt.Errorf("the operator could not be asked the %s: %v", r.Act, err)
}
r.Raised, r.RaisedAt = true, p.now()
data, _ := json.Marshal(r)
if _, err := p.Store.Put(StateTakeover, KeyRequest, data); err != nil {
return fmt.Errorf("the ask %s was asked and its request could not be marked: %v", r.ID, err)
}
_ = rev
return nil
}
// Reconcile closes an ask that can stay open for ever, once per pass while one is open: a warrant the process missed is
// caught up from the router's record, one that ended with none is closed, and one past its expiry and a margin is closed
// as expired by the part itself.
func (p *Part) Reconcile() error {
if p.Asker == nil {
return nil
}
s, err := p.readSwitch()
if err != nil || s.Open == "" {
return err
}
act := actOf(s.Open)
at := p.now()
for _, a := range s.Acts {
if a.Ask == s.Open {
at = a.At
break
}
}
state, w, err := p.Asker.Where(s.Open)
if err != nil {
return fmt.Errorf("the router's record of the ask %s cannot be read: %v", s.Open, err)
}
if w != nil {
h, herr := p.Asker.Take(*w, p.Warranted)
if h == "" && herr != nil {
// The part's own record of the ask could not be reached: nothing was taken, and the next pass tries again.
return fmt.Errorf("the operator's answer to %s could not be taken yet: %v", s.Open, herr)
}
p.Said(h, *w, herr)
if cur, rerr := p.readSwitch(); rerr == nil && cur.Open == s.Open {
_ = p.finish(cur, Act{Act: act, By: "operator", At: p.now(), Ask: s.Open, Said: string(w.Outcome)})
}
return nil
}
raw, _, found, _ := p.Store.Get(StateTakeover, KeyRequest)
var r Request
if found {
_ = json.Unmarshal(raw, &r)
}
switch {
case p.now().After(at.Add(asks.ApproveLasts + askMargin)):
return p.finish(s, Act{Act: act, By: "operator", At: p.now(), Ask: s.Open, Said: string(asks.OutcomeExpired)})
case state == "" && r.ID == s.Open && r.Raised && p.now().After(r.RaisedAt.Add(recordGrace)):
return p.finish(s, Act{Act: act, By: "operator", At: p.now(), Ask: s.Open, Said: "ended", Refused: "the router holds no record of the ask"})
}
return nil
}
func actOf(id string) string {
if strings.HasPrefix(id, ActHandBack+"-") {
return ActHandBack
}
return ActTakeOver
}
// Following says whether the bus node has loaded the grants part's list: "yes" when its loaded digest equals the part's,
// "not yet" when it loaded another, "unknown" when its have says none.
func (p *Part) Following() string {
loaded := ReadInputs(p.Read, &p.cache).Loaded
g, _, had := p.Sealed()
if loaded == nil || !had || g.Digest == "" {
return "unknown: the bus module's have names no loaded list, or the grants part wrote none"
}
if loaded.Source == delivery.GrantsUserListSource && loaded.Digest == g.Digest {
return "yes"
}
return "not yet: the bus node loaded another list"
}
// AskOf is the ask of a request, in the words the operator reads: what changes, why, what it risks. Its yes binds the act.
func (p *Part) AskOf(r Request) (asks.Ask, error) {
verb, headline, does, stays, expl := "grants_take_over", "Switch the bus's user list to the grants part?", "The switch is recorded; it takes effect when the bus node loads the grants part's list.",
"The controller's list stays in force.", "The grants part has composed the user list the bus holds today. Your yes records the switch; it takes effect when the bus node loads the part's list.\n"+
"If the lists differ in any user, or a user has no hash in the vault, the switch is refused and you are told which."
if r.Act == ActHandBack {
verb, headline, does, stays, expl = "grants_hand_back", "Give the bus's user list back to the controller?", "The switch back is recorded; it takes effect when the bus node loads the controller's list.",
"The grants part's list stays in force.", "The bus follows the grants part's list now.\nThe controller still writes its list until it loses its lease, so the bus can follow it again."
}
bind, err := asks.ActDigest(asks.Act{"verb": verb, "module": "mesh-grants", "ask": r.ID})
if err != nil {
return asks.Ask{}, err
}
return asks.Ask{ID: r.ID, Headline: headline, Explanation: expl, Who: asks.Operator,
Options: []asks.Option{{ID: "yes", Label: "Yes, switch", Does: does, Level: asks.Approve, Binds: bind},
{ID: "no", Label: "No, leave it", Does: stays, Level: asks.Acknowledge}},
Expires: p.now().Add(asks.ApproveLasts - time.Minute), OnExpiry: stays + " Nothing changes.", About: "mesh-grants-" + r.Act,
Details: "Asked by " + r.By + ": " + r.Why}, nil
}
// Warranted is the process's act on the operator's answer (asker.Client.Listen hands it once, after checking the ask, the
// option and its level). A no is recorded; a yes runs the switch after the refusals, and a refusal is said to the operator.
func (p *Part) Warranted(o asks.Option, w asks.Warrant) error {
s, err := p.readSwitch()
if err != nil {
return err
}
act := ActTakeOver
if strings.HasPrefix(w.Ask, ActHandBack+"-") {
act = ActHandBack
}
if s.Open != w.Ask {
return fmt.Errorf("the warrant answers the ask %s, and the ask open is %q: nothing is switched", w.Ask, s.Open)
}
a := Act{Act: act, By: "operator", At: p.now(), Ask: w.Ask}
if o.ID != "yes" {
a.Said = "declined"
return p.finish(s, a)
}
var reasons []string
if act == ActTakeOver {
reasons = p.Decide()
if s, err = p.readSwitch(); err != nil { // Decide clears a refusal and writes the switch: read it again
return err
}
} else if s.State == Shadow {
reasons = []string{"nothing to hand back: the bus node follows the controller's list already"}
}
if len(reasons) > 0 {
a.Said, a.Refused = "refused", strings.Join(reasons, "; ")
p.sayRefused(&s, act, a.Refused)
return p.finish(s, a)
}
p.clearRefused(&s)
a.Said = "recorded"
if act == ActTakeOver {
s.State = TookOver
} else {
s.State = Shadow
}
if err := p.finish(s, a); err != nil {
return err
}
// The bus node turns on the key: written now, so the switch reaches it on the operator's yes (ADR 0547 clause 8).
if err := p.syncFollow(s.State); err != nil {
return fmt.Errorf("the switch is recorded and the key the bus node follows could not be written (the next pass writes it): %w", err)
}
if p.Emit != nil {
_ = p.Emit.Emit(EventUserList, map[string]any{"switch": s.State})
}
return nil
}
// Said is the process's note of an ask that ended with no yes or no (expired, or refused by the router): the switch
// stays, and the ask is closed so the operator can be asked again.
func (p *Part) Said(h asker.Handled, w asks.Warrant, err error) {
if err != nil && p.Logf != nil {
p.Logf("the answer to %s (%s) ended in an error: %v", w.Ask, h, err)
}
s, err := p.readSwitch()
if err != nil || s.Open != w.Ask {
return
}
_ = p.finish(s, Act{Act: actOf(w.Ask), By: "operator", At: p.now(), Ask: w.Ask, Said: string(w.Outcome) + " (" + string(h) + ")"})
}
func (p *Part) finish(s Switch, a Act) error {
if s.Open == a.Ask {
s.Open = ""
}
s.Acts = append(s.Acts, a)
return p.writeSwitch(s)
}
// Decide is what stands between a take-over and the grants part's list, read from a composition made now: the loaded list
// not read, any user the two lists keep apart (one the loaded list has that the shadow cannot compose among them, such as
// an enrolment user or a person whose records are in the controller's store alone), and any user with no hash in the vault.
// Nothing here but what is true now: a take-over is refused for these and for no other reason. Empty means none.
func (p *Part) Decide() []string {
p.deciding.Store(true)
res := p.Compose()
p.deciding.Store(false)
var reasons []string
if c := res.UserList.Compare; !c.Equal() {
if c.Unread != "" {
reasons = append(reasons, "the loaded list could not be read: "+c.Unread)
}
if d := c.Differing(); len(d) > 0 {
reasons = append(reasons, fmt.Sprintf("%d users differ between the shadow list and the loaded list: %s", len(d), strings.Join(d, ", ")))
}
}
reasons = append(reasons, p.followReasons(res, ReadInputs(p.Read, &p.cache))...)
cr, gone := res.UserList.Credentials, res.UserList.Compare.Gone
var unhashed []string
for _, u := range append(append([]string(nil), cr.LeftOut...), cr.LosesLogin...) {
if !contains(gone, u) {
unhashed = append(unhashed, u)
}
}
if len(unhashed) > 0 {
sort.Strings(unhashed)
reasons = append(reasons, fmt.Sprintf("the vault holds no hash for %d users (ADR 0547 clause 9: handed or adopted before take-over): %s",
len(unhashed), strings.Join(unhashed, ", ")))
}
if len(reasons) == 0 {
if s, err := p.readSwitch(); err == nil && len(s.Refusing) > 0 {
p.clearRefused(&s)
_ = p.writeSwitch(s)
}
}
return reasons
}
func contains(s []string, v string) bool {
for _, x := range s {
if x == v {
return true
}
}
return false
}
// sayRefused tells the operator a yes was given and the switch refused, with the reasons (the users that differ named). The
// condition stays raised until a switch is recorded or a later Decide finds no reason (clearRefused).
func (p *Part) sayRefused(s *Switch, act, why string) {
if p.Say == nil {
return
}
c := asks.Condition{Key: "mesh-grants." + act + "-refused", Severity: asks.SeverityWarning, Subject: asks.Scope{Scope: "module", ID: "mesh-grants"},
State: asks.StateRaised, Summary: act + " refused: " + why, Headline: "The user list switch was refused", Needs: "a look at why",
Explanation: asks.NeedsYou + "a look at why. You said yes to the switch and it was refused: " + why, Resolved: "the switch is asked again and goes through"}
if p.Say(c) == nil && !contains(s.Refusing, act) {
s.Refusing = append(s.Refusing, act)
}
}
// clearRefused clears every refusal condition said and not cleared: on a recorded switch, a hand-back, or a Decide with no reason.
func (p *Part) clearRefused(s *Switch) {
if p.Say == nil {
return
}
var left []string
for _, act := range s.Refusing {
c := asks.Condition{Key: "mesh-grants." + act + "-refused", Subject: asks.Scope{Scope: "module", ID: "mesh-grants"}, State: asks.StateCleared}
if p.Say(c) != nil {
left = append(left, act)
}
}
s.Refusing = left
}
// TakeOverState is the tool take-over-state: the switch, the request, and what became of each ask.
func (p *Part) TakeOverState() (any, error) {
s, err := p.readSwitch()
if err != nil {
return nil, err
}
loaded := ReadInputs(p.Read, &p.cache).Loaded
out := map[string]any{"state": s.State, "open": s.Open, "acts": s.Acts, "bus_node_loaded_grants_list": p.Following(),
"effect": p.Effect(s.State, loaded),
"says": "a recorded switch takes effect when the bus node loads the grants part's list; effect says switched only once the bus node's loaded digest is the grants part's written one"}
if g, rev, had := p.Sealed(); had {
out["written"] = map[string]any{"digest": g.Digest, "revision": g.Revision, "key_revision": rev, "follow": g.Follow,
"users": g.Users, "left_out": g.LeftOut, "sealed_to": g.Node}
}
if loaded != nil {
out["loaded"] = map[string]any{"source": loaded.Source, "digest": loaded.Digest, "revision": loaded.Revision,
"users": loaded.Users, "follows": loaded.Follows, "at": loaded.At, "error": loaded.Error}
}
if raw, _, found, err := p.Store.Get(StateTakeover, KeyRequest); err == nil && found {
out["request"] = json.RawMessage(raw)
}
return out, nil
}
+11 -9
View File
@@ -13,17 +13,18 @@ import (
// The part's tools, as its manifest lists them. Module tools in this milestone: a `mesh-*` seat must be in mesh-sdk's own
// seat list, which the running controller's composer does not know, so the part claims no seat until take-over.
const (
ToolGrants = "grants_grants"
ToolMembership = "grants_membership"
ToolUserList = "grants_user_list"
ToolComposeNow = "grants_compose_now"
ToolAssert = "grants_assert"
ToolTakeOver = "grants_take_over"
ToolHandBack = "grants_hand_back"
ToolGrants = "grants_grants"
ToolMembership = "grants_membership"
ToolUserList = "grants_user_list"
ToolComposeNow = "grants_compose_now"
ToolAssert = "grants_assert"
ToolTakeOver = "grants_take_over"
ToolTakeOverState = "grants_take_over_state"
ToolHandBack = "grants_hand_back"
)
// Tools is every tool, in the manifest's order.
var Tools = []string{ToolGrants, ToolMembership, ToolUserList, ToolComposeNow, ToolAssert, ToolTakeOver, ToolHandBack}
var Tools = []string{ToolGrants, ToolMembership, ToolUserList, ToolComposeNow, ToolAssert, ToolTakeOver, ToolHandBack, ToolTakeOverState}
// WritesNoGrant is what grants says of the part's own grants in this milestone.
const WritesNoGrant = "none: the grants part writes no grants.<module>.<fp> before ADR 0547's third milestone (clause 5); the controller's are beside"
@@ -70,8 +71,9 @@ func (p *Part) MembershipOf(node, module string) (any, error) {
}
in := ReadInputs(p.Read, &p.cache)
recs, left := Records(in)
notFree := WithRootFree(&recs, p.Call)
m, ok := Memberships(recs)[node+"."+module]
out := map[string]any{"node": node, "module": module, "gaps": MembershipGaps}
out := map[string]any{"node": node, "module": module, "gaps": MembershipGaps, "root_free": recs.RootFree, "root_not_free": notFree}
if ok {
out["composed"] = m
} else {
@@ -5,11 +5,13 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"regexp"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// User is one bus user and its grants, as the bus module's nats_users answers them: never a password hash.
@@ -234,7 +236,9 @@ type Comparison struct {
Dropped []string `json:"dropped"`
Gone []string `json:"gone,omitempty"`
Differ []UserDiff `json:"differ"`
Same int `json:"same"`
// Excused are the subjects the comparison does not count, each with why (Comparison.Excusing).
Excused []Excuse `json:"excused,omitempty"`
Same int `json:"same"`
// Unread says the loaded list could not be read, and why: nothing is compared then.
Unread string `json:"unread,omitempty"`
}
@@ -325,3 +329,96 @@ func Placed(in Inputs) func(user string) bool {
return slices.Contains(in.Nodes, parts[0]) && slices.Contains(in.Assigned[parts[0]], parts[1])
}
}
// Excuse is one subject the comparison does not count as a difference between the lists, and why.
type Excuse struct {
User string `json:"user"`
Subject string `json:"subject"`
Why string `json:"why"`
}
// Excusing removes from the differences the subjects f excuses (added: the shadow grants it and the loaded list does not),
// naming each in Excused; a user left with no difference counts as the same. Pure.
func (c Comparison) Excusing(f func(user, subject string, added, subscribe bool) string) Comparison {
var differ []UserDiff
for _, d := range c.Differ {
keep := func(list []string, added, subscribe bool) []string {
var out []string
for _, s := range list {
if why := f(d.User, s, added, subscribe); why != "" {
c.Excused = append(c.Excused, Excuse{User: d.User, Subject: s, Why: why})
continue
}
out = append(out, s)
}
return out
}
d.PublishAdded, d.PublishDropped = keep(d.PublishAdded, true, false), keep(d.PublishDropped, false, false)
d.SubscribeAdded, d.SubscribeDropped = keep(d.SubscribeAdded, true, true), keep(d.SubscribeDropped, false, true)
if len(d.PublishAdded)+len(d.PublishDropped)+len(d.SubscribeAdded)+len(d.SubscribeDropped) == 0 && d.Responses == "" {
c.Same++
continue
}
differ = append(differ, d)
}
if differ == nil {
differ = []UserDiff{}
}
c.Differ = differ
return c
}
var seatSubject = regexp.MustCompile(`^mesh\.seat\.([^.]+)\.(?:call|tool)\.([^.]+)(?:\.\*)?$`)
var seatService = regexp.MustCompile(`^\$SRV\.(?:INFO|PING|STATS)\.([^.]+)(?:\.>)?$`)
// ExcusedBy is the excuse of a comparison until placement lives in the catalogue (the operator, 2026-10-26): a verb the
// seat catalogue retired on purpose, which the shadow no longer grants, is no difference; and what the shadow grants more
// for a mesh seat that is claimed on several machines (the standing gap seat holdings: it composes the seat as held on each)
// is named and not counted, for a user on a machine that claims the seat and only as a subscription (what it serves), since
// a subscription to a seat's verbs on a machine whose holder does not answer them harms nothing and is gone when holdings
// follow the catalogue. Nothing else is excused: an added publish, a grant of a user on another machine, or the loss of any
// other subject, keeps the lists apart.
func ExcusedBy(multiHeld map[string][]string) func(user, subject string, added, subscribe bool) string {
// holds says the user is on a machine where the seat is claimed: the machine's tool runner, or a module of it.
holds := func(user, seat string) bool {
node, _, _ := strings.Cut(user, ".")
return slices.Contains(multiHeld[seat], node)
}
return func(user, subject string, added, subscribe bool) string {
if m := seatSubject.FindStringSubmatch(subject); m != nil {
if !added && subscribe && retiredVerb(m[1], m[2]) {
return "a verb the seat catalogue retired"
}
if added && subscribe && holds(user, m[1]) {
return "seat " + m[1] + " is claimed on several machines: composed as held on each until holdings follow the catalogue"
}
}
if m := seatService.FindStringSubmatch(subject); m != nil && added && subscribe && holds(user, m[1]) {
return "seat " + m[1] + " is claimed on several machines: composed as held on each until holdings follow the catalogue"
}
return ""
}
}
// MultiHeld are the mesh seats (held once for the mesh) that more than one machine's module claims, each with the
// machines that claim it.
func MultiHeld(r buscompose.Records) map[string][]string {
nodes := map[string][]string{}
for node, declared := range r.Assigned {
for _, d := range declared {
for _, s := range d.Holds {
if (s.Scope == "" || s.Scope == catalogue.ScopeMesh) && !slices.Contains(nodes[s.Name], node) {
nodes[s.Name] = append(nodes[s.Name], node)
}
}
}
}
for seat, ns := range nodes {
if len(ns) < 2 {
delete(nodes, seat)
continue
}
sort.Strings(ns)
}
return nodes
}
+10 -1
View File
@@ -13,10 +13,18 @@
"node-engine.have",
"mesh-vault.bus-accounts"
],
"consumes": [
"mesh-controller.condition-raised",
"mesh-controller.condition-cleared"
],
"uses": [
"operator-channel"
],
"invokes": [
"nats.nats_streams",
"nats.nats_user_can",
"nats.nats_users"
"nats.nats_users",
"seat:mesh-controller.root-free"
],
"emits": [
"asserted",
@@ -30,6 +38,7 @@
"grants_compose_now",
"grants_assert",
"grants_take_over",
"grants_take_over_state",
"grants_hand_back"
],
"state": [