Compare commits

...
Author SHA1 Message Date
mesh-tools d581f9659a Merge pull request 'Tries record their asks; the try limit counts tries on a build node; build_usage shows capacity, hours and steps (issue 1890)' (#78) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:34 +00:00
jschoubben ea94f9234c Merge remote-tracking branch 'origin/main' into fix/capacity-follows-cpu-share
mesh/delivery merged: source.main moved to novox/mesh-delivery@d581f9659a78
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking ea94f9234c97 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
2026-10-26 21:29:44 +01:00
jschoubben c19468b04e Pin the checker and the view at mesh-sdk 805b4d9: the capacity work merged with sdk main 2026-10-26 21:29:44 +01:00
jschoubben c47afd8f4b build_usage says a seat's build hours, and when it is outside them
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group fix/capacity-follows-cpu-share checking: 1 of 3 member(s) ready
mesh/check checking c47afd8f4b88 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
A build module's capacity.<node> now carries its node's build window (the operator, 2026-10-26: g14
builds only from 18:00 to 06:00); the seat shows it, outside it says it takes nothing new, and a
value the module cannot read shows as any hour with the value and why. The checker's and the view's
SDK pin moves to mesh-sdk d00dac1.
2026-10-26 21:22:36 +01:00
mesh-tools 3dd94f714d Merge pull request 'The checker fails any repository's change that adds a binary, in any commit (issue 1577)' (#77) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:27 +00:00
jschoubben d6bd31a166 Say the try limits as what they are: a check when a try is asked
Tries let in while waiting may later all run at once; each node's build capacity bounds what runs
on it (review of issue 1890's try cap).
2026-10-26 20:11:08 +01:00
jschoubben c0a4b139a2 A try queued behind the build seat's checks no longer blocks the next try
Only tries a build node holds count toward the 4 at once; tries waiting for a node are bounded
apart at 8, and the refusal names both counts (issue 1890: at 19:53 four tries queued behind 44
checks for up to 47 minutes refused every further try).
2026-10-26 20:05:51 +01:00
jschoubben 6d60953f98 A running try names its asks and where each stands; build_usage says a seat's CPU-bound capacity
The checker wrote a try's asks only at its end, so a 31-minute try could not be told queued from
running (issue 1890). The try's record now names each ask as it is put, then the node that took
it, its state and the phase its step says, written on the step key's own changes. build_usage
names a running try's asks and takes a build module's capacity.<node> (configured bounded by its
node's build CPU, issue 1806's follow-up) when it is current with mesh-config.
2026-10-26 19:58:39 +01:00
jschoubben d74e082155 Checker refuses a program a commit of the branch added and a later one removed: it stays in history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-delivery@3dd94f714d6e
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking d74e0821552c on df8b9ac3b1b5 | novox/mesh-delivery#77 in group fix/1577-checker-refuses-binaries
2026-10-26 19:47:53 +01:00
jschoubben 820c458f4c The captured clean fetches carry the programs a current build module answers, so lines asserted whole do not warn (issue 1577) 2026-10-26 19:16:25 +01:00
jschoubben dc9d65524d A check judged by a build module that reports no programs says so in its status line, still passing: the refusal did not run (issue 1577) 2026-10-26 19:08:00 +01:00
jschoubben 1a4c788ed7 Checker fails a change whose fetch reports a compiled program, and warns when a build module reports none: build output kept getting committed (issue 1577) 2026-10-26 19:01:40 +01:00
18 changed files with 436 additions and 44 deletions
@@ -105,7 +105,7 @@ func TryTools(tr Trying) []stdio.Tool {
{Name: check.TryTool,
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; a new try is refused while 4 tries run on a build node or 8 wait for one, mesh-wide; tries let in while waiting may later all run at once, and each node's build capacity bounds what runs on it. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused. A try whose ask no build node took within the step " +
"bound ends as error; its ask stays in the build seat's stream and may still run later, judged by nobody (issue 734).",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
@@ -135,7 +135,8 @@ func TryTools(tr Trying) []stdio.Tool {
}},
{Name: check.TryReadTool,
Description: "Where a try stands, as stored: queued, running, or its end (pass, fail, or error when it could not run), with what was said of it, " +
"the head and the trunk commit it was merged onto, the time it took and what failed first. Read-only.",
"the head and the trunk commit it was merged onto, the time it took and what failed first; while it runs, each ask it put as soon as it is put, " +
"with the node that took it, its state and the phase its step says (fetch and merge, a level's digest, lint, tests) and since when. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"id"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"}}},
Run: func(args map[string]any) (any, error) {
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583 h1:ikD/ggf9/5k23sxJOX1PFcGzb4M3+QeC66AT0MyQ9Zs=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026142434-65ee43f07583/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,42 @@
package check
import (
"fmt"
"strings"
)
// A change that adds or modifies a compiled program fails (issue 1577): it is build output, `go build` in a module's folder
// writes one beside go.mod, and `git add -A` takes it. The build module's fetch step, which holds the merged clone, reads the
// first bytes of each changed file and reports the programs (mesh-sdk buildstep.MergeInto); the checker judges the answer.
// NoProgramsReported is the warning of a build module whose fetch answered no programs field: an older module, which cannot
// tell, so the check does not refuse a program it did not look for.
const NoProgramsReported = "the build module did not report programs: a compiled program in the change is not refused"
// checkBinariesOf fails a pull request whose change adds or modifies a compiled program, naming each file. It judges the
// first fetch answer of the pull request, whichever build module gave it; a build module that reports no programs leaves a
// warning on the check and passes it.
func (st *state) checkBinariesOf(t *target) verdictPtr {
key := fmt.Sprintf("%s#%d", t.pull.Repository, t.pull.Number)
if t.fetched.Merge == "" {
return nil // no merge was made: nothing was changed to judge
}
if t.fetched.Programs == nil {
st.c.note(fmt.Sprintf("%s: %s (%s)", key, NoProgramsReported, t.module))
st.c.warn(key, fmt.Sprintf("a compiled program is not refused: %s reported no programs", t.module))
st.j.logf("%s: %s (%s)", key, NoProgramsReported, t.module)
return nil
}
var found []string
for _, p := range *t.fetched.Programs {
if p.Commit != "" {
found = append(found, fmt.Sprintf("%s: a compiled program added in %s and removed later: it stays in history; squash the branch (issue 1577)", p.Path, short12(p.Commit)))
continue
}
found = append(found, p.Path+": a compiled program is build output: remove it and add it to .gitignore (issue 1577)")
}
if len(found) > 0 {
return failed(t.node, fmt.Sprintf("%s: %s", key, strings.Join(found, "; ")))
}
return nil
}
@@ -0,0 +1,63 @@
package check
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func TestAChangeWhoseFetchReportsProgramsFailsNamingEachFile(t *testing.T) { // 1577
prog := func(p ...string) *[]ProgramFile {
out := []ProgramFile{}
for _, f := range p {
path, commit, _ := strings.Cut(f, "@") // `path@commit`: added by a commit of the branch and removed later
out = append(out, ProgramFile{Path: path, Kind: "ELF", Commit: commit})
}
return &out
}
for _, c := range []struct {
name string
programs *[]ProgramFile
fail []string // paths named; none passes
warned bool // the build module reported no programs field
}{
{"none found", prog(), nil, false},
{"one program", prog("modules/m/m"), []string{"modules/m/m"}, false},
{"two programs", prog("a/x", "b/y"), []string{"a/x", "b/y"}, false},
{"added and removed in the branch", prog("gone/prog@0123456789abcdef0123"), []string{"gone/prog: a compiled program added in 0123456789ab and removed later: it stays in history; squash the branch (issue 1577)"}, false},
{"an older build module reports nothing", nil, nil, true},
} {
b := newBench(t)
b.fetch.Merge, b.fetch.Programs = "mmmm", c.programs
cs := caseOf(t, pullOf(repoA, 7, head1))
v := b.judge(t, cs).Verdict
if got := slices.ContainsFunc(cs.Notes(), func(n string) bool { return strings.Contains(n, NoProgramsReported) }); got != c.warned {
t.Errorf("%s: warned = %v, want %v (%v)", c.name, got, c.warned, cs.Notes())
}
line := Line(v, cs, cs.Pulls[0], nil, nil, nil)
if got := strings.Contains(line, "a compiled program is not refused: mesh-build-go reported no programs"); got != c.warned {
t.Errorf("%s: the status line holds the warning = %v, want %v: %s", c.name, got, c.warned, line)
}
if len(c.fail) == 0 {
if v.State != delivery.VerdictPass {
t.Errorf("%s: verdict = %+v", c.name, v)
}
continue
}
if v.State != delivery.VerdictFail {
t.Errorf("%s: verdict = %+v, want a failure", c.name, v)
continue
}
for _, p := range c.fail {
want := p
if !strings.Contains(p, " added in ") {
want = p + ": a compiled program is build output: remove it and add it to .gitignore (issue 1577)"
}
if !strings.Contains(v.Failed.Words, want) {
t.Errorf("%s: words %q do not name %s", c.name, v.Failed.Words, p)
}
}
}
}
@@ -43,9 +43,16 @@ type Case struct {
Try *TryOptions
notesMu sync.Mutex
notes []string
// warnings are, by pull request, what part of its check did not run (Case.warn).
warnings map[string]string
notes []string
// asks are the ids of the asks the check put on the build seats' streams, in order: a try names them (Try.Asks).
asks []string
// Watch, when set, hears each ask the check puts, as it is put (step nil), and each value its step key takes while
// the check waits for it: the node that took it, the phase it says, its end. A try writes them into its record, so a
// running try names its asks and where each stands (issue 1890). Called from the check's goroutines; it must not block
// for long.
Watch func(askID string, step *delivery.Step)
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a carry after a trunk or sibling move is asked on the
@@ -186,6 +193,23 @@ func (c *Case) note(s string) {
}
}
// warn says of a pull request (pullKey) that a part of its check did not run, in words for its status line (issue 1577).
func (c *Case) warn(key, words string) {
c.notesMu.Lock()
defer c.notesMu.Unlock()
if c.warnings == nil {
c.warnings = map[string]string{}
}
c.warnings[key] = words
}
// Warning is what warn said of a pull request, or "".
func (c *Case) Warning(key string) string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return c.warnings[key]
}
// Notes are the words noted on the case, in the order they were noted.
func (c *Case) Notes() []string {
c.notesMu.Lock()
@@ -427,8 +427,14 @@ func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, l
if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
}
waiting := ctx
if watch := st.c.Watch; watch != nil {
id := a.ID
watch(id, nil) // put: a try names it at once, not only when it ends (issue 1890)
waiting = buildrun.WithStepSeen(ctx, func(s delivery.Step) { watch(id, &s) })
}
var err error
step, err = st.j.Asks.Wait(ctx, a)
step, err = st.j.Asks.Wait(waiting, a)
if err != nil {
if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error())
@@ -489,6 +495,9 @@ func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, l
if v := st.checkPinsOf(ctx, t); v != nil {
return v
}
if v := st.checkBinariesOf(t); v != nil {
return v
}
}
ran := 0.0
for _, la := range answer.Levels {
@@ -87,18 +87,20 @@ type pieces struct {
judged string
// carried says, of a carried verdict, the commits it was carried from (ADR 0548 §8); never dropped.
carried string
// warning says that a part of the check did not run, for instance that an older build module could not refuse a compiled program; never dropped.
warning string
}
// pullKey names a pull request of a case: `<owner/repo>#<number>`.
func pullKey(p Pull) string { return fmt.Sprintf("%s#%d", p.Repository, p.Number) }
func (p pieces) join(drop int) string {
parts := []string{p.head, p.commits, p.carried, p.counts, p.judged}
parts := []string{p.head, p.commits, p.carried, p.warning, p.counts, p.judged}
switch {
case drop < 3 && p.times != "":
parts = append(parts, p.times)
case drop == 3 && p.shortTimes != "":
parts = []string{p.head, p.commits, p.carried, p.shortCounts, p.judged, p.shortTimes}
parts = []string{p.head, p.commits, p.carried, p.warning, p.shortCounts, p.judged, p.shortTimes}
}
if p.over != "" {
parts = append(parts, p.over)
@@ -210,6 +212,9 @@ func carriedWords(from map[string]string, v delivery.Verdict, cs *Case, p Pull)
func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) pieces {
var pc pieces
pc.judged = judged[pullKey(p)]
if cs != nil {
pc.warning = cs.Warning(pullKey(p))
}
commits := short12(p.Head) + " on " + short12(p.Trunk)
switch v.State {
case delivery.VerdictRunning:
@@ -44,6 +44,17 @@ type Fetched struct {
Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"`
Trees map[string]MergeTree `json:"trees,omitempty"`
// Programs are the changed files whose first bytes are an executable binary, read by the build module in its workspace
// (issue 1577); nil for a build module that does not report it, a pointer to an empty list for one that found none.
Programs *[]ProgramFile `json:"programs,omitempty"`
}
// ProgramFile is a changed file that is a compiled program, and its format (ELF, Mach-O or PE).
type ProgramFile struct {
Path string `json:"path"`
Kind string `json:"kind"`
// Commit is set for a program a commit of the branch added and a later one removed: it stays in history once merged.
Commit string `json:"commit,omitempty"`
}
// GoPackage is one package of a Go test result.
@@ -10,7 +10,7 @@ TestFetchWithABaseMergesAndAnswersThePreview and TestFetchWithAConflictEndsPasse
into a throwaway test that wrote them out, at mesh-catalog feat/r4c5-checker d5804927 (the git commands are the test's fake,
so the shas are its constants; the shape and the field names are the module's own marshalling). node-fetch-clean.json is the
FetchResult json.Marshal of mesh-build-node's fetch (fetch.go, preview.go) in its test TestFetchWithABaseMergesAndAnswersThePreview
(cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679.
(cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679. Both clean fetches carry `"programs":[]`, which a merging fetch of the build modules answers since issue 1577 (mesh-sdk buildstep.MergeInto's Programs), added by hand to the captured bytes: a fixture without it is an older build module's, and its line warns that no compiled program is refused.
forge-get-file-go-mod.json is the answer of the forge's gitea_get_file (mesh-catalog modules/gitea, tools/index.ts) for modules/mesh-delivery-builder/go.mod of
novox/mesh-delivery at main, and forge-compare-trunk.json the answer of gitea_compare for novox/mesh-sdk base main, head cb54756c3865 (a commit on the
@@ -1 +1 @@
{"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}}
{"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"programs":[],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}}
@@ -1 +1 @@
{"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}}
{"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"programs":[],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}}
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"regexp"
"slices"
"sort"
"strconv"
"strings"
@@ -29,8 +30,13 @@ const (
// TriesState is the checker's state of tries: the bucket mesh-delivery-checker_tries, keys try-<unix nanoseconds>.
TriesState = "tries"
TriesBucket = delivery.CheckerModule + "_" + TriesState
// MaxTries is how many tries may wait or run at once, mesh-wide: each holds a build node for minutes.
MaxTries = 4
// MaxTries and MaxTriesWaiting are checked when a try is asked, mesh-wide: a new try is refused while MaxTries tries run
// on a build node or MaxTriesWaiting wait for one (queued, or their asks not yet taken by a node). It is no bound on
// what runs later: tries let in while waiting may later all run at once, and each node's build capacity bounds what
// runs on it. A try waiting behind the seat's other asks holds no node, so it no longer blocks the next try (issue
// 1890: at 19:53 on 2026-10-26 four tries sat queued behind 44 checks for up to 47 minutes and no fifth could be asked).
MaxTries = 4
MaxTriesWaiting = 8
// TryScan is how many of the newest tries a new ask reads to find the ones that wait or run (issue 1827).
TryScan = 64
// TryKeep is how long an ended try's key is kept before the checker removes it.
@@ -79,12 +85,104 @@ type Try struct {
Machines []string `json:"machines,omitempty"`
// Asks are the ids of the asks the try put on the build seats' streams, each ending on the step key <node>.<id> of its
// build module: the delivery view counts those steps as a try's, not a candidate's check (build_usage).
Asks []string `json:"asks,omitempty"`
Asks []string `json:"asks,omitempty"`
// Steps say where each ask stands while the try runs and after: asked until a node takes it, then the node, its
// state, the phase its step says and when (issue 1890: a 31-minute try could not be told queued from running).
Steps []TryStep `json:"steps,omitempty"`
Asked time.Time `json:"asked"`
Started time.Time `json:"started,omitzero"`
Ended time.Time `json:"ended,omitzero"`
}
// TryAsked is a TryStep's state before a build node took its ask: it waits in the seat's stream.
const TryAsked = "asked"
// TryStep is one ask of a try as its step key last said it.
type TryStep struct {
Ask string `json:"ask"`
Node string `json:"node,omitempty"`
State string `json:"state"`
Phase string `json:"phase,omitempty"`
Started time.Time `json:"started,omitzero"`
PhaseAt time.Time `json:"phase-at,omitzero"`
Ended time.Time `json:"ended,omitzero"`
}
// tryWriter writes a running try's record again as its asks are put and their steps move: one writer per try, holding
// the revision it wrote last, so the try's end is written at that revision. A write refused (the try ended by hand)
// stops the writes; the end's own write then says it.
type tryWriter struct {
mu sync.Mutex
tr *Tryer
ctx context.Context
key string
t Try
rev uint64
stopped bool
}
// heard takes an ask put (step nil) or a value of its step key in, and writes the record when it changed.
func (w *tryWriter) heard(askID string, step *delivery.Step) {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped {
return
}
changed := false
if !slices.Contains(w.t.Asks, askID) {
w.t.Asks, changed = append(w.t.Asks, askID), true
}
i := slices.IndexFunc(w.t.Steps, func(s TryStep) bool { return s.Ask == askID })
if i < 0 {
w.t.Steps, changed = append(w.t.Steps, TryStep{Ask: askID, State: TryAsked}), true
i = len(w.t.Steps) - 1
}
if step != nil {
now := TryStep{Ask: askID, Node: step.Node, State: step.State, Phase: step.Phase, Started: step.Started,
PhaseAt: step.PhaseAt, Ended: step.Ended}
changed = changed || now != w.t.Steps[i]
w.t.Steps[i] = now
}
if changed {
w.write()
}
}
// write writes the record at the revision written last; a refusal stops the writes.
func (w *tryWriter) write() {
data, err := encodeTry(w.t)
if err != nil {
return
}
rev, err := w.tr.Tries.Update(w.ctx, w.key, data, w.rev)
if err != nil {
w.stopped = true
w.tr.logf("%s: its asks' progress is no longer written: %v", w.key, err)
return
}
w.rev = rev
}
// end stops the writes and answers the record and its revision, for the try's end.
func (w *tryWriter) end() (Try, uint64) {
w.mu.Lock()
defer w.mu.Unlock()
w.stopped = true
return w.t, w.rev
}
// HoldsANode says whether the try holds a build node now: it runs, and a step of its asks runs on a node, or it names
// no step (a checker that writes its asks only at the end, or one about to put its first).
func (t Try) HoldsANode() bool {
if t.State != TryRunning {
return false
}
if len(t.Steps) == 0 {
return true
}
return slices.ContainsFunc(t.Steps, func(s TryStep) bool { return s.State == delivery.StepRunning })
}
// Over says whether the try came to its end.
func (t Try) Over() bool { return t.State == TryPass || t.State == TryFail || t.State == TryError }
@@ -124,8 +222,8 @@ func NotChecked(repository string) error {
}
// AskTry writes a new try, queued, and answers its id: a try of the same head that waits or runs is the answer to a second
// ask, and at most MaxTries wait or run at once. trunks are the gate's trunk heads of every repository (source.main): a
// repository with none is no repository the checker checks. A trunk asked by name is its trunk head now.
// ask, and a new try is refused while MaxTries tries run on a build node or MaxTriesWaiting wait for one. trunks are the
// gate's trunk heads of every repository (source.main): a repository with none is no repository the checker checks. A trunk asked by name is its trunk head now.
func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (string, error) {
repo := strings.ToLower(strings.TrimSuffix(strings.TrimSpace(a.Repository), ".git"))
head := strings.ToLower(strings.TrimSpace(a.Head))
@@ -188,7 +286,7 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
}); err != nil {
return "", err
}
var open []string
var holding, waiting []string
for i, cur := range tries {
if !live[i] {
continue
@@ -196,10 +294,18 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
if cur.Repository == t.Repository && cur.Head == t.Head && cur.Trunk == t.Trunk && cur.Tests == t.Tests && cur.Level == t.Level {
return cur.ID, nil
}
open = append(open, fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State))
said := fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State)
if cur.HoldsANode() {
holding = append(holding, said)
} else {
waiting = append(waiting, said)
}
}
if len(open) >= MaxTries {
return "", fmt.Errorf("%d tries wait or run, the most at once: %s; ask again when one ends", len(open), strings.Join(open, ", "))
// Checked at this ask only: a try queued behind the seat's checks holds no node; tries let in while waiting may later
// all run at once, and each node's build capacity bounds what runs on it.
if len(holding) >= MaxTries || len(waiting) >= MaxTriesWaiting {
return "", fmt.Errorf("a new try is refused while %d tries run on a build node or %d wait for one; now %d run and %d wait: running %s; waiting %s; ask again when one ends",
MaxTries, MaxTriesWaiting, len(holding), len(waiting), orNone(holding), orNone(waiting))
}
t.ID = "try-" + strconv.FormatInt(now.UnixNano(), 10)
data, err := encodeTry(t)
@@ -395,7 +501,8 @@ func (tr *Tryer) run(ctx context.Context, key string) {
}
return
}
cs := &Case{Key: key, Group: key,
progress := &tryWriter{tr: tr, ctx: ctx, key: key, t: t, rev: rev}
cs := &Case{Key: key, Group: key, Watch: progress.heard,
Pulls: []Pull{{Repository: t.Repository, Head: t.Head, Trunk: t.Trunk}},
Trunk: map[string]string{t.Repository: t.Trunk},
Queued: t.Asked,
@@ -409,6 +516,7 @@ func (tr *Tryer) run(ctx context.Context, key string) {
if ctx.Err() != nil {
return // left running: the checker takes it over when it starts; or ended by hand, which wrote its end
}
t, rev = progress.end()
v := out.Verdict
t.Ended = tr.now().UTC()
t.Took = t.Ended.Sub(t.Started).Round(time.Second).String()
@@ -470,3 +578,11 @@ func judgeWithin(ctx context.Context, j Judge, cs *Case, bound time.Duration) (O
}
return out, true
}
// orNone joins the tries said, or says none.
func orNone(said []string) string {
if len(said) == 0 {
return "none"
}
return strings.Join(said, ", ")
}
@@ -3,6 +3,7 @@ package check
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
@@ -23,6 +24,19 @@ func (m memTries) Create(key string, value []byte) error {
return err
}
// writtenTries is a memKV that keeps every running try it was given to write, in order.
type writtenTries struct {
*memKV
running []Try
}
func (w *writtenTries) Update(ctx context.Context, key string, value []byte, rev uint64) (uint64, error) {
if t, err := DecodeTry(value); err == nil && !t.Over() {
w.running = append(w.running, t)
}
return w.memKV.Update(ctx, key, value, rev)
}
var tryTrunks = map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA}
func readTry(t testing.TB, kv *memKV, id string) Try {
@@ -54,14 +68,38 @@ func TestATryIsAskedQueuedOncePerHeadAndAtMostFourAtOnce(t *testing.T) { // ADR
if err != nil || again != id {
t.Errorf("a second ask of the same head answered %q, %v, want %q", again, err, id)
}
heads := []string{head2, trunkA2, trunkS2}
for i, h := range heads {
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoA, Head: h, Trunk: trunkA}, at.Add(time.Duration(i+2)*time.Second)); err != nil {
t.Fatalf("try %d: %v", i+2, err)
}
// Issue 1890: tries on a build node (a step of theirs running, or none named yet) and tries waiting for one (queued, or
// their asks not taken) are bounded apart, MaxTries and MaxTriesWaiting; the refusal names both counts.
onNode := func(i int) Try {
return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
Steps: []TryStep{{Ask: "a", State: delivery.StepRunning}}}
}
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute)); err == nil || !strings.Contains(err.Error(), "4 tries wait or run") {
t.Errorf("a fifth try answered %v, want refused naming the four", err)
waitingFor := func(i int) Try {
return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
Steps: []TryStep{{Ask: "a", State: TryAsked}}}
}
for _, c := range []struct {
what string
holding, wait int
refused string
}{
{"four queued behind the checks, none on a node: a fifth is taken", 0, 4, ""},
{"four on a node", MaxTries, 0, "now 4 run and 0 wait"},
{"three on a node, eight waiting", 3, MaxTriesWaiting, "a new try is refused while 4 tries run on a build node or 8 wait for one; now 3 run and 8 wait"},
} {
kv := newMemKV()
for i := range c.holding + c.wait {
tr := waitingFor(i + 1)
if i < c.holding {
tr = onNode(i + 1)
}
data, _ := encodeTry(tr)
_, _ = kv.Create(context.Background(), tr.ID, data)
}
_, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute))
if (c.refused == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), c.refused)) {
t.Errorf("%s: answered %v, want refused %q", c.what, err, c.refused)
}
}
}
@@ -89,7 +127,8 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
t.Fatal(err)
}
b := newBench(t)
tr := &Tryer{Tries: kv, Judge: b.ex}
written := &writtenTries{memKV: kv}
tr := &Tryer{Tries: written, Judge: b.ex}
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
tr.Wait()
@@ -97,6 +136,15 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
if got.State != TryPass || got.Took == "" || got.Head != head1 || got.Trunk != trunkA || got.Said == "" || got.Levels[delivery.LevelUnit].Run != 2 {
t.Fatalf("try = %+v", got)
}
// Issue 1890: while it runs, the try names its ask as soon as it is put, and then where its step stands; the end
// keeps each ask's node and state.
if len(written.running) < 3 || len(written.running[1].Asks) != 1 || written.running[1].Steps[0].State != TryAsked ||
written.running[2].Steps[0].Node != "shanks" {
t.Errorf("the running try's writes = %+v, want its ask named while it ran, then its node", written.running)
}
if len(got.Steps) != 1 || got.Steps[0].Node != "shanks" || got.Steps[0].State != delivery.StepPassed {
t.Errorf("the ended try's steps = %+v", got.Steps)
}
// The try names the asks it put, so the delivery view counts their steps as a try's (build_usage).
if len(got.Asks) != 1 {
t.Errorf("asks = %v, want the one check ask the try put", got.Asks)
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-view
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2 h1:s1WcG7Ll1OkEwwvMKEabdWvPvTiFCHghlXiY1LoIJDA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026125731-2aae6f5580f2/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536 h1:+SoDdDXF7Tgj38WP4kRpUSps6DJVl6P+w5u/KfdtwnQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026202658-805b4d9a3536/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -67,7 +67,8 @@ type UsageAnswer struct {
At time.Time `json:"at"`
Nodes []NodeUsage `json:"nodes"`
Backlog []BacklogRow `json:"backlog"`
// TriesRunning are the tries the checker runs now; their asks are counted under check until the try ends and names them.
// TriesRunning are the tries the checker runs now, each with its asks and where each stands (a checker that names its
// asks as they are put, issue 1890; an older one names them only at the end, and they are counted under check until then).
TriesRunning []TryRunning `json:"tries_running"`
Notes []string `json:"notes,omitempty"`
}
@@ -77,6 +78,9 @@ type TryRunning struct {
ID string `json:"id"`
Repository string `json:"repository"`
Started time.Time `json:"started,omitzero"`
Asks []string `json:"asks,omitempty"`
// Steps are the try's own words for each ask: asked, or the node that took it, its state and the phase its step says.
Steps json.RawMessage `json:"steps,omitempty"`
}
// NodeUsage is one node: each build seat it holds or is set to build for, whether it is eligible to build, and its CPU.
@@ -101,7 +105,17 @@ type SeatUsage struct {
// Capacity is the asks it runs at once; CapacityFrom where that came from.
Capacity int `json:"capacity"`
CapacityFrom string `json:"capacity_from"`
Paused bool `json:"paused"`
// Configured is mesh-config's capacity when the module says it judges by a lower one (capacity.<node> of its pauses
// state: the configured capacity bounded by its node's build CPU, issue 1806's follow-up); Capacity is then the
// module's and CapacityFrom its why.
Configured int `json:"configured,omitempty"`
// BuildHours is the node's build window as the module says it (HH:MM-HH:MM and its zone), absent when it builds at any
// hour; OutsideHours says the module judged the time outside it, so it takes nothing new (the operator, 2026-10-26).
BuildHours string `json:"build_hours,omitempty"`
OutsideHours bool `json:"outside_hours,omitempty"`
// BuildHoursFault is why the module cannot read its node's build-hours value, naming it: it then builds at any hour.
BuildHoursFault string `json:"build_hours_fault,omitempty"`
Paused bool `json:"paused"`
// Eligible says it takes asks: it holds the seat, has a capacity above 0 and is not paused; WhyNot says why not.
// EligibleKnown is false when this tool cannot tell (it cannot read mesh-config's capacities): Eligible is then
// absent, never false, and WhyNot starts "not known: ".
@@ -384,8 +398,9 @@ func (u UsageReader) Usage() UsageAnswer {
}
}
// Pauses: paused.<node> of each module.
// Pauses: paused.<node> of each module; and capacity.<node>, the capacity the module says it judges by.
paused := map[string]bool{} // <node>.<language>
judged := map[string]delivery.Capacity{}
for _, lang := range delivery.AskLanguages {
module, _ := delivery.BuildModuleOf(lang)
pr, ok := u.Pauses[lang]
@@ -411,6 +426,13 @@ func (u UsageReader) Usage() UsageAnswer {
paused[node+"."+lang] = p.Paused
}
}
if key, err := delivery.CapacityKey(node); err == nil {
if raw, err := pr.Get(key); err == nil && len(raw) > 0 {
if c, err := delivery.DecodeCapacity(raw); err == nil {
judged[node+"."+lang] = c
}
}
}
}
}
@@ -452,6 +474,18 @@ func (u UsageReader) Usage() UsageAnswer {
s.Capacity = (&buildrun.Places{Capacity: caps}).CapacityOf(node)
s.CapacityFrom = fmt.Sprintf("the modules' default %q: mesh-config holds no resolved value for %s on %s", buildstep.DefaultCapacities, s.Module, node)
}
if c, ok := judged[node+"."+lang]; ok && s.Holds && unknown == "" && !isBad && c.Configured == s.Capacity {
// The module running on the node says its build hours, and judges by less than mesh-config gives when its
// node's build CPU bounds it or the time is outside its build hours.
s.BuildHours, s.OutsideHours = c.BuildHours, c.OutsideHours
if c.BuildHoursFault != "" {
s.BuildHours, s.BuildHoursFault = "any hour", c.BuildHoursFault
}
if c.Effective != c.Configured {
s.Configured, s.Capacity = s.Capacity, c.Effective
s.CapacityFrom = fmt.Sprintf("%s, as %s on %s judges it (said %s)", c.Why, s.Module, node, c.At.UTC().Format(time.RFC3339))
}
}
s.Paused = paused[node+"."+lang]
s.EligibleKnown = true
switch {
@@ -466,6 +500,8 @@ func (u UsageReader) Usage() UsageAnswer {
s.WhyNot = "not known: this tool cannot read its capacity (" + unknown + ")"
case isBad:
s.WhyNot = "its capacity on " + node + " cannot be read, so its puller uses 0"
case s.OutsideHours:
s.WhyNot = "outside its build hours " + s.BuildHours + " on " + node + ": it takes nothing new until they open"
case s.Capacity == 0:
s.WhyNot = "its capacity on " + node + " is 0"
}
@@ -517,6 +553,7 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
want = append(want, k)
}
}
unnamed := 0
raws, err := u.Tries.values(want)
if err != nil {
note("the checker's tries cannot all be read: %v", err)
@@ -526,11 +563,12 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
continue
}
var t struct {
ID string `json:"id"`
Repository string `json:"repository"`
State string `json:"state"`
Started time.Time `json:"started"`
Asks []string `json:"asks"`
ID string `json:"id"`
Repository string `json:"repository"`
State string `json:"state"`
Started time.Time `json:"started"`
Asks []string `json:"asks"`
Steps json.RawMessage `json:"steps"`
}
if json.Unmarshal(raws[i], &t) != nil {
continue
@@ -539,11 +577,14 @@ func (u UsageReader) tries(now time.Time, ans *UsageAnswer, note func(string, ..
asks[a] = true
}
if t.State == "running" {
ans.TriesRunning = append(ans.TriesRunning, TryRunning{ID: t.ID, Repository: t.Repository, Started: t.Started})
ans.TriesRunning = append(ans.TriesRunning, TryRunning{ID: t.ID, Repository: t.Repository, Started: t.Started, Asks: t.Asks, Steps: t.Steps})
if len(t.Asks) == 0 {
unnamed++
}
}
}
if len(ans.TriesRunning) > 0 {
note("a running try's asks are counted under check until the try ends and names them")
if unnamed > 0 {
note("%d running tries name no ask yet (none put yet, or a checker that names them only at the end): their asks are counted under check until then", unnamed)
}
return asks
}
@@ -191,4 +191,36 @@ func TestBuildUsageAnswersEveryNodesUseFromTheStates(t *testing.T) {
if verdict["shanks"] != "not known" || verdict["ace"] != "not eligible" {
t.Errorf("verdicts with mesh-config unreadable = %v, want shanks not known, ace not eligible (paused)", verdict)
}
// Issue 1806's follow-up: the module on shanks says it judges by 1 of mesh-config's 2 (its build CPU bounds it); the
// seat says the module's capacity and why. And issue 1890: a running try that names its asks is said with them.
u = usageFixture()
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":1,"configured":2,"quota-from":"the cpu.max of /meshyield.slice","quota-cores":6,"cores-per-step":6,"by-cpu":1,"why":"1, bounded by the CPU","at":"2026-10-26T11:00:00Z"}`,
"capacity.ace": `{"v":1,"effective":2,"configured":3,"why":"stale: mesh-config says 2 now"}`})
u.Tries = states(map[string]string{"try-" + strconv.FormatInt(at("11:58").UnixNano(), 10): `{"id":"try-2","repository":"novox/mesh-delivery","head":"h","trunk":"t","state":"running","asks":["` + tryAsk0 + `"],"steps":[{"ask":"` + tryAsk0 + `","node":"shanks","state":"running","phase":"unit: tests of 3 packages"}]}`})
bound := u.Usage()
sc := seatOf(t, bound, "shanks", "node-build-go")
if sc.Capacity != 1 || sc.Configured != 2 || !strings.HasPrefix(sc.CapacityFrom, "1, bounded by the CPU, as mesh-build-go on shanks judges it") {
t.Errorf("a seat its CPU bounds: capacity %d of %d, from %q", sc.Capacity, sc.Configured, sc.CapacityFrom)
}
if sa := seatOf(t, bound, "ace", "node-build-go"); sa.Capacity != 2 || sa.Configured != 0 {
t.Errorf("a module's word older than mesh-config's value is not taken: %+v", sa)
}
// The operator, 2026-10-26: outside its build hours the seat takes nothing new, and says so.
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":0,"configured":2,"build-hours":"18:00-06:00 CET","outside-hours":true,"why":"0, outside its build hours 18:00-06:00 CET until 18:00 (otherwise 2)","at":"2026-10-26T11:00:00Z"}`})
if so := seatOf(t, u.Usage(), "shanks", "node-build-go"); so.Capacity != 0 || !so.OutsideHours || so.Eligible == nil || *so.Eligible ||
so.WhyNot != "outside its build hours 18:00-06:00 CET on shanks: it takes nothing new until they open" {
t.Errorf("a seat outside its build hours: %+v", so)
}
// A build-hours value the module cannot read: it builds at any hour, and the seat says the value and why.
u.Pauses["go"] = states(map[string]string{"capacity.shanks": `{"v":1,"effective":2,"configured":2,"build-hours-fault":"the build hours \"shanks=18:00-6\": \"6\" is not a time HH:MM","why":"2; at any hour, because its build hours cannot be read","at":"2026-10-26T11:00:00Z"}`})
if sf := seatOf(t, u.Usage(), "shanks", "node-build-go"); sf.BuildHours != "any hour" || !strings.Contains(sf.BuildHoursFault, "shanks=18:00-6") || sf.Eligible == nil || !*sf.Eligible {
t.Errorf("a seat whose build hours cannot be read: %+v", sf)
}
if len(bound.TriesRunning) != 1 || len(bound.TriesRunning[0].Asks) != 1 || !strings.Contains(string(bound.TriesRunning[0].Steps), "unit: tests") ||
strings.Contains(strings.Join(bound.Notes, " | "), "name no ask yet") {
t.Errorf("a running try naming its asks: %+v, notes %v", bound.TriesRunning, bound.Notes)
}
}
var tryAsk0 = stepID(at("11:58"), "a00010")