Compare commits

..
21 Commits
Author SHA1 Message Date
mesh-tools 2cb6971194 Merge pull request 'Design 47 H1: a receiver reads its store's fill once per transfer (H-A-227, H-A-228)' (#305) from fix/47-H1-receiver-fill-once into main 2026-11-21 01:12:32 +00:00
jschoubben 6a730c33e6 H-A-227, H-A-228: the receiver reads its objects' fill once per transfer
A fill read on every chunk made the tool runner list the registry's
incoming store per chunk. The kept transfer now holds the fill read at its
open and the bytes taken since; each chunk tells their sum, written with
the put already made per chunk, and the store is read again only at the
finish, a drop, or when a part's chunks are dropped.

Builds: design 47 H H-A-227, H-A-228
2026-11-21 02:11:39 +01:00
mesh-tools d1aab1d08d Merge pull request 'Design 47 H1: the validate and composition run asks carry their entry's pins (H-A-203)' (#304) from fix/47-H1-runner-validate-pins into main 2026-11-21 01:09:03 +00:00
jschoubben c6b4b6054b Design 47 H1: the validator's and the composer's run asks are built against their entry's pins (H-A-203)
The validate step's run ask built the manifest validator from the indexer's
tree with no pins, so the Go build module refused it for a requirement of
another repository (main.74's feh validate, main.67's mesh-sdk validate);
the composition step's run ask built the composer from the node-engine's
tree the same way. The seat mesh-wants' holder builds both asks: each now
carries the pins of the entry its program is built from, the indexer's
entry at the step's own revision and the node-engine entry chosen per
machine, those with a version, in the form the entry keeps and the
builder sends.

Builds: design 47 H H-A-203, H-A-192, H-A-205
2026-11-21 02:03:18 +01:00
mesh-tools 4df45a3793 Merge pull request 'Design 47 H1: the builder keeps a reservation for its fingerprint and withdraws a taken-over build (H-A-225, H-A-226)' (#303) from fix/47-H1-builder-reservation-kept into main 2026-11-21 00:41:48 +00:00
jschoubben e2e13ddd29 Drop withdrawn versions the catalog released past; a dropped number stays a gap (H-A-226)
The series key's Withdrawn grew at every take-over and was never pruned. At each write it now keeps only the
withdrawn versions above the catalog's highest released (those at or below it are refused by the plain rule anyway),
carried across a renumbering of the same fingerprint too; a dropped number is never reused (H-A-226 withdraws
H-A-225's freeing). The comment of the rebuild's reservation now states A-490b as H-A-225 amends it.

Builds: design 47 H H-A-225, H-A-226
2026-11-21 01:40:18 +01:00
jschoubben dcf5b7fa8f Keep a reservation for its fingerprint until released; withdraw a taken-over build (H-A-225)
After a builder restart, the run already building a module carried its reserved number while a re-run of the same
fingerprint reserved the next, so its publish was refused "has no version reserved or released" (main.61: the builder
0.1.111 vs 0.1.112, mesh-grants 0.1.52 vs 0.1.54, and others reserved twice within seconds). A take-over by another
fingerprint dropped the older reservation but let the older build run on and publish a number that no longer existed.

Reserve now answers the standing reservation of the same fingerprint while it is unreleased and above the catalog's
highest, a build again or a re-run alike (A-490b amended so). A take-over keeps the dropped version in Withdrawn and
withdraws the older build (ended superseded, ADR 0664 §4); a late publish of it is refused as withdrawn, naming the
fingerprint that took it over.

Builds: design 47 H H-A-225; G A-490b as amended; ADR 0664 §4
2026-11-21 01:36:32 +01:00
mesh-tools 1fe6ade359 Merge pull request 'Design 47 H1: a same-repository module named only by a replace is a requirement of the module graph (H-A-224)' (#302) from fix/47-H1-indexer-replace-required into main 2026-11-21 00:30:50 +00:00
jschoubben 9e09767a4b Indexer: a same-repository module named only by a replace is a requirement
The watchman's go.mod names buildrun in a replace directive with no require
line, and the walk of the module graph followed require lines only, so
buildrun's requirement of the module assigner's client was never pinned and
every test step of the watchman failed "no pin". The walk now takes the
same-repository modules a go.mod replaces by a directory as required, with
the same same-repository test, so their go.mod files are read and their
other-repository requirements pinned, transitively.

Builds: design 47 H-A-224, H-A-201
2026-11-21 01:26:29 +01:00
mesh-tools 59443ab3f7 Merge pull request 'Design 47 H1: a mover a newer revision moved leaves a running pipeline too (H-A-223)' (#301) from fix/47-H1-runner-moved-leaves into main 2026-11-21 00:24:57 +00:00
mesh-tools 92d0ab8602 Merge pull request 'Design 47 H1: the indexer's test fakes answer what the SDK's pull guard reads (H-A-196)' (#300) from fix/47-H1-indexer-fake-consumer into main 2026-11-21 00:24:55 +00:00
jschoubben 3a31331ad3 Design 47 H1: a mover leaving a running pipeline has its build-node asks dropped (H-A-223)
A moved mover's steps ended superseded, but their asks kept running on the
build nodes. Each such step holding its own ask on a node-build seat, one
reading no shared run, now has the seat's drop asked with the newer
pipeline's why, through the drop the runner already bounds and keeps. The
seat mesh-builds serves no drop, so a build ask there runs to its end,
which the runner ignores: the step is no longer open.

Builds: design 47 H H-A-223
2026-11-21 01:22:10 +01:00
jschoubben d1581666d2 Design 47 H1: the start of a newer pipeline is held to take moved movers out, and reads its branch alone (H-A-223)
No test failed when Start stopped calling the supersede of running
pipelines, the path the live runner takes. Two rows now start a newer
pipeline through Start, a take-in's and a release revision's, with the
supersede wired to a hearer over the same store, and hold that the older
running pipeline's moved mover left it. The supersede lists only the keys
of the newer pipeline's branch instead of every pipeline's.

Builds: design 47 H H-A-223
2026-11-21 01:18:55 +01:00
jschoubben d4d7895392 Indexer: the branch consumer fake answers its CachedInfo and ends when stopped
With the SDK's guard of one pull per durable (pullOnce reads the consumer's
CachedInfo and releases on the consume's Closed), the fake of a branch's
durable embedded a nil consumer and a nil consume context, so
TestABranchDurableNotThereIsMadeAndHeard panicked. The fake now answers a
ConsumerInfo with its stream, name and a created time of its own, and its
consume context closes when stopped. No other fake in mesh-delivery embeds
jetstream.Consumer; the assigner's, runner's, builder's, watchman's and
grants' unit suites pass with the new SDK unchanged.

Builds: design 47 H-A-196, H §774
2026-11-21 01:18:22 +01:00
mesh-tools d6c209a10c Merge pull request 'Design 47 H1: the catalog seat's revision verb answers the store; the builder's picture holds a kept revision by number (H-A-222)' (#299) from fix/47-H1-revision-verb-store into main 2026-11-21 00:14:44 +00:00
mesh-tools 51ac838732 Merge pull request 'Design 47 H1: the indexer decodes every hand tool's ask (H-A-208, H-A-210)' (#298) from fix/47-H1-indexer-again-decode into main 2026-11-21 00:14:40 +00:00
jschoubben 06a52a10e0 Design 47 H1: a moved mover leaves a running pipeline too (H-A-223)
A release re-pinned node-tools while main.54 still built it at its old
fingerprint; the build failed and failed the pipeline for its other 119
movers, because supersession took movers out of waiting pipelines only.
Every start of a newer revision's pipeline now also walks the older running
pipelines of its branch: each mover whose fingerprint it moved leaves, its
open steps ending superseded with a why naming the newer pipeline, a step
reading a test run leaving it as a cancel's does, and the mover kept in the
pipeline's superseded. The end judgement counts a left mover's steps as
neither holding nor failed, so the pipeline ends by its other movers alone;
one every mover left ends superseded. A run on request's pipeline is never
touched.

Builds: design 47 H H-A-223, H-A-163, H-A-213
2026-11-21 01:13:57 +01:00
jschoubben 773e2e33a1 Builder: a kept revision of another source or fingerprint fails the ask at once
Review round 1: an ask names the revision and the source of one revision, so a
kept revision read by number whose source differs from the ask's is not the
ask's; the check is restored, naming both sources. And a kept revision naming
another source or other fingerprints is a mismatch no wait can mend: the step
fails at once, loud, instead of waiting out revision-wait-minutes; only a
revision the catalog does not answer is waited for.

Builds: design 47 H-A-222 (2), H-A-218
2026-11-21 01:10:37 +01:00
jschoubben d817d85fca Builder: an ask naming a revision is held by that revision read by number
An ask of the very source the picture holds, whose entries the picture no
longer names (a re-pin keeps a later revision of the same source), waited for
the picture and failed past its bound, because the kept revision was read by
number only for a source below the picture's. Now an ask naming a revision is
read by number whenever the picture does not name its entries, and is built
when the catalog keeps it and it names every fingerprint asked, whatever the
source says; the source is compared only for an ask naming no revision.

Builds: design 47 H-A-222 (2), H-A-205, H-A-213
2026-11-21 01:07:18 +01:00
jschoubben 6d5000871d Indexer: the revision verb answers the newest kept revision from the store
The tools bundle answered the catalog seat's revision verb from its view of the
trunk in memory, which ran one or more revisions behind what the indexer kept
until it heard the event, so the runner and the builder waited on revisions the
bus held all along. The verb now reads which revision is newest from the store
at the time of the ask (a listing and one object) and takes the view's entries
only when the view holds that very revision.

Builds: design 47 H-A-222 (1), H-A-205, H-A-213
2026-11-21 01:05:55 +01:00
jschoubben 5c7648744d Indexer: read a takeins value's hand acts from TakeInAsk's tags, not a hand-kept list
indexer_take_in_again wrote TakeInAsk{Again} and DecodeTakeIn refused it: its
hand-kept list of act fields lacked Again, so the handler never reached its
Again branch. Every field of TakeInAsk now says by its tag takein whether it is
an act, an argument or of the ask; DecodeTakeIn reads the acts from the tags,
and a table test builds every hand tool's value by its own Run path, decodes it
and acts on it through takenIn, and refuses a field left unclassed.

Builds: design 47 H §774, H-A-208, H-A-210
2026-11-21 01:03:32 +01:00
29 changed files with 904 additions and 103 deletions
@@ -216,7 +216,7 @@ type runPrep struct {
seats []json.RawMessage
manifests map[string]composer.Manifest
none map[string]string
programOf func(node string) (buildstep.RunProgram, string, error)
programOf func(node string) (buildstep.RunProgram, string, map[string]string, error)
err error
}
@@ -274,7 +274,7 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
end(node + " gave no composition input: " + in.Why)
return
}
program, engineFp, err := prep.programOf(node)
program, engineFp, pins, err := prep.programOf(node)
if err != nil {
end(node + ": " + err.Error())
return
@@ -317,7 +317,8 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
return
}
a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(),
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the composition of " + node + " by the node-engine at " + program.Commit}}
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the composition of " + node + " by the node-engine at " + program.Commit},
Packages: pins} // the composer is built against its entry's pins, as every build (H-A-203, H-A-192)
if err := r.put(ctx, a); err != nil {
end(node + ": its run ask " + ask + " is not put: " + err.Error())
}
@@ -732,53 +733,76 @@ func programOfEntry(e meshcatalog.CatalogEntry) (buildstep.RunProgram, error) {
return buildstep.RunProgram{Repository: e.Repository, Commit: e.Commit, Dir: dir, Package: composer.Package}, nil
}
// pinsOfEntry are the mesh packages' versions an entry pins, the ones it names with a version (H-A-203, H-A-183): the pins
// a run's program is built against, in the form the entry keeps and the builder sends. nil when it pins none.
func pinsOfEntry(e meshcatalog.CatalogEntry) map[string]string {
var out map[string]string
for path, version := range e.Packages {
if version == "" {
continue
}
if out == nil {
out = map[string]string{}
}
out[path] = version
}
return out
}
// programs answers, per machine, the composer its run builds and the node-engine fingerprint it was chosen from (A-226, A-243,
// A-252, §719): where the revision's node-engine entry differs from the trunk's, that build for every machine; else the build the
// machine's node-engine runs, as its `running` says, its entry read through mesh-catalog's `entry`; a machine whose node-engine's
// `running` is not heard gets the revision's entry, said (A-451).
func (r *compositionRun) programs(ctx context.Context, half mcc.Half) (func(node string) (buildstep.RunProgram, string, error), error) {
func (r *compositionRun) programs(ctx context.Context, half mcc.Half) (func(node string) (buildstep.RunProgram, string, map[string]string, error), error) {
engine, ok := nodeEngineEntry(half.Entries)
if !ok {
return nil, fmt.Errorf("the catalog revision %d holds no module claiming the seat %s", half.Revision.Revision, ne.Seat)
}
if trunk, ok := r.trunk(); !ok || func() bool { t, ok := nodeEngineEntry(trunk.Entries); return !ok || t.Fp != engine.Fp }() {
p, err := programOfEntry(engine)
return func(string) (buildstep.RunProgram, string, error) { return p, engine.Fp, err }, nil
return func(string) (buildstep.RunProgram, string, map[string]string, error) {
return p, engine.Fp, pinsOfEntry(engine), err
}, nil
}
var mu sync.Mutex
byFp := map[string]buildstep.RunProgram{}
return func(node string) (buildstep.RunProgram, string, error) {
type built struct {
program buildstep.RunProgram
pins map[string]string
}
byFp := map[string]built{}
return func(node string) (buildstep.RunProgram, string, map[string]string, error) {
running, ok := r.running(node, engine.Module)
if !ok || running.Fingerprint == "" {
r.logf("%s: no build of %s is heard running there, so its composer is the catalog revision's entry %s (A-451)", node,
engine.Module, engine.Fp)
p, err := programOfEntry(engine)
return p, engine.Fp, err
return p, engine.Fp, pinsOfEntry(engine), err
}
fp := running.Fingerprint
mu.Lock()
p, ok := byFp[fp]
b, ok := byFp[fp]
mu.Unlock()
if ok {
return p, fp, nil
return b.program, fp, b.pins, nil
}
what := "the entry of the node-engine's build " + fp + " is not read"
body, said, err := r.callRetry(ctx, entryVerbKey, meshcatalog.EntryInput{Fingerprint: fp})
if err != nil || body == nil {
return buildstep.RunProgram{}, fp, unreadErr(what, err, said)
return buildstep.RunProgram{}, fp, nil, unreadErr(what, err, said)
}
var a meshcatalog.EntryAnswer
if err := json.Unmarshal(body, &a); err != nil {
return buildstep.RunProgram{}, fp, fmt.Errorf("%s: %w", what, err)
return buildstep.RunProgram{}, fp, nil, fmt.Errorf("%s: %w", what, err)
}
p, err = programOfEntry(a.Entry)
p, err := programOfEntry(a.Entry)
if err != nil {
return p, fp, err
return p, fp, nil, err
}
pins := pinsOfEntry(a.Entry)
mu.Lock()
byFp[fp] = p
byFp[fp] = built{program: p, pins: pins}
mu.Unlock()
return p, fp, nil
return p, fp, pins, nil
}, nil
}
@@ -53,7 +53,7 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
s := &runStand{asks: map[string]buildstep.StepAsk{}, requests: map[string]composer.Request{}}
engine := func(fp, commit string) meshcatalog.CatalogEntry {
return meshcatalog.CatalogEntry{Module: "engine", Fp: fp, Repository: "novox/mesh-host", Commit: commit, Path: ".", Serves: []string{ne.Seat + ".composition"},
Manifest: map[string]any{"module": "engine"}}
Manifest: map[string]any{"module": "engine"}, Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.25-" + fp, "git.novox.be/novox/x/go": ""}}
}
revisionCommit := commitE1
if revisionEngine != "src2:e1" {
@@ -186,6 +186,13 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
if s.asks["g14"].Run.Program.Commit != commitE2 {
t.Errorf("g14's composer is built at %s, want the build its node-engine runs", s.asks["g14"].Run.Program.Commit)
}
// Each composer is built against the pins of the node-engine entry it is built from, a pin with no version left out
// (H-A-203, H-A-192).
for node, fp := range map[string]string{"ace": "src2:e1", "g14": "src2:e2"} {
if got := s.asks[node].Packages; len(got) != 1 || got["git.novox.be/novox/mesh-sdk/go"] != "v0.1.25-"+fp {
t.Errorf("%s's composer is built against the pins %v; want its entry's", node, got)
}
}
if req.Depth != composer.DepthCompose || req.BlameTries != 20 || len(req.Machines) != 1 || req.Machines[0] != "ace" || len(req.Cases) != 1 || req.Cases[0].Name != changeCase ||
string(req.Input) != `{"node":"ace"}` || len(req.Seats) != 1 || string(req.Seats[0]) != `{"seat":"s1"}` {
t.Errorf("the request of ace %+v", req)
@@ -345,7 +345,8 @@ func (r *compositionRun) startValidate(ctx context.Context, key string, run mesh
return
}
a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(),
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the manifest validator of " + step.Pipeline + " at " + program.Commit}}
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the manifest validator of " + step.Pipeline + " at " + program.Commit},
Packages: pinsOfEntry(entry)} // the validator is built against its entry's pins at the step's revision (H-A-203, H-A-192, H-A-205)
if err := r.put(ctx, a); err != nil {
couldNot("its run ask " + ask + " is not put: " + err.Error())
}
@@ -25,7 +25,8 @@ import (
// is judged against an empty case before, said in its end, and no revision below it is asked.
func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
indexer := meshcatalog.CatalogEntry{Module: "indexer", Repository: "novox/mesh-delivery", Commit: commitE3, Path: "modules/mesh-delivery-indexer",
Fp: "src2:ix", Serves: []string{"seat:" + meshcatalog.Seat + ".revision"}, Manifest: map[string]any{"module": "indexer"}, ManifestDigest: "d-ix"}
Fp: "src2:ix", Serves: []string{"seat:" + meshcatalog.Seat + ".revision"}, Manifest: map[string]any{"module": "indexer"}, ManifestDigest: "d-ix",
Packages: map[string]string{"git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient": "v0.1.4"}}
entry := func(module, digest string, edges ...meshcatalog.Edge) meshcatalog.CatalogEntry {
return meshcatalog.CatalogEntry{Module: module, Manifest: map[string]any{"module": module, "d": digest}, ManifestDigest: digest, Edges: edges}
}
@@ -124,6 +125,10 @@ func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
if p := asks[0].Run.Program; p.Commit != commitE3 || p.Dir != "modules/mesh-delivery-indexer" || p.Package != validatorPackage || len(asks[0].Run.Heads) != 0 {
t.Errorf("the validator is built from %+v", asks[0].Run)
}
// The validator is built against the indexer entry's pins at the step's revision (H-A-203, H-A-192, H-A-205).
if got := asks[0].Packages; len(got) != 1 || got["git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"] != "v0.1.4" {
t.Errorf("the validator is built against the pins %v; want its entry's", got)
}
if c := requests[0].Cases; len(c) != 2 || c[0].Name != changeCase || c[1].Name != beforeCase || len(c[1].Manifests) != 3 || requests[0].Predefined != nil || requests[0].Packages != nil {
t.Errorf("the validator's request is %+v", requests[0])
}
@@ -244,6 +244,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return emitSeatEvent(context.WithoutCancel(ctx), js, cred, meshbuilds.SubjectPackagePublished, "package-published-"+p.Module+"-"+p.Version+"-"+p.Artifact, p)
}
b.Reserved = reserved
// A take-over of a series' reservation by another fingerprint withdraws the older build (H-A-225).
reserved.withdraw = b.Withdraw
// The last versions-filled the stream keeps is read at the start (design 47 G §369, §372): a restart does not lose it.
reserved.readFilled(ctx, lastKeptOf(js, cred, logger.Printf), sleeps)
// A state bound later numbers the builds looked at without it.
@@ -50,6 +50,8 @@ type reservations struct {
bound time.Duration
wake func(context.Context)
logf func(string, ...any)
// withdraw ends the running build of a module's fingerprint, superseded, with why (H-A-225); nil ends none.
withdraw func(module, fingerprint, why string)
// call asks a seat's verb: the git seat's tag (design 47 G §370).
call toolCaller
// say says package-published on the seat mesh-builds; nil says none.
@@ -92,6 +94,11 @@ func (rv *reservations) took(ctx context.Context, older, newer versions.Reserved
}
rv.dropBound(ctx, versions.ReservedKey(older.Module, older.Series))
rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, older.Module)
if older.Fingerprint != newer.Fingerprint && rv.withdraw != nil {
// The older build is ended superseded, as the picture's supersession ends one (H-A-225, ADR 0664 §4): nothing publishes
// under the dropped number.
rv.withdraw(older.Module, older.Fingerprint, "its reservation "+older.Version+" was taken over by the build of "+newer.Fingerprint+", which reserves "+newer.Version)
}
}
var _ build.Reservations = (*reservations)(nil)
@@ -266,7 +266,8 @@ func TestAPublishedHeardBeforeTheDigestsIsSaidWhenTheBuildRecordsThem(t *testing
}
// TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo: design 47 G A-455 — a build that reserves while the series is
// reserved by an older build never released drops the older one's bound and finding and says so in the builder's journal.
// reserved by an older build never released drops the older one's bound and finding and says so in the builder's journal; and
// H-A-225 — the older build is withdrawn, so nothing publishes under the dropped number, while a re-run of the newer keeps it.
func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T) {
ctx := context.Background()
st := &memFindings{vals: map[string][]byte{}}
@@ -279,6 +280,8 @@ func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T)
rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, call, func() jetstream.KeyValue { return bounds }, found,
nil, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
rv.filled.Store(true)
var withdrawn []string
rv.withdraw = func(module, fp, why string) { withdrawn = append(withdrawn, module+" "+fp+": "+why) }
older, err := rv.Reserve(ctx, "m", "0.1", []string{"m:a"}, versions.Source{Fingerprint: "fp-old"})
if err != nil {
t.Fatal(err)
@@ -305,6 +308,12 @@ func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T)
if got, ok, _ := rv.ReservedFor(ctx, "m", "fp-new"); !ok || got.Version != "0.1.1" {
t.Errorf("reserved for fp-new: %+v %v", got, ok)
}
if len(withdrawn) != 1 || withdrawn[0] != "m fp-old: its reservation 0.1.0 was taken over by the build of fp-new, which reserves 0.1.1" {
t.Errorf("withdrawn %q; want the build of fp-old withdrawn once", withdrawn)
}
if again, err := rv.Reserve(ctx, "m", "0.1", []string{"m:a"}, versions.Source{Fingerprint: "fp-new"}); err != nil || again.Version != "0.1.1" || len(withdrawn) != 1 {
t.Errorf("a re-run of fp-new reserved %q (%v), withdrew %q; want 0.1.1 kept, nothing withdrawn", again.Version, err, withdrawn)
}
}
// ansKV records the value versionActed writes back to a reasks key.
@@ -362,10 +362,10 @@ func (m reserveMap) Delete(_ context.Context, key string) error {
return nil
}
// TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber: design 47 G A-490b, at the build, through the real
// reserver — a done key whose own fingerprint has an unreleased 0.1.30 standing, the catalog's highest 0.1.16, is built again
// under 0.1.17 (never 0.1.31), the standing reservation taken over.
func TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber(t *testing.T) {
// TestABuildAgainKeepsItsOwnStandingNumber: design 47 G A-490b as H-A-225 amends it, at the build, through the real reserver —
// a done key whose own fingerprint has an unreleased 0.1.30 standing, the catalog's highest 0.1.16, is built again under 0.1.30
// (never 0.1.31, never a new number): the run building carries the standing number until it is released.
func TestABuildAgainKeepsItsOwnStandingNumber(t *testing.T) {
r := newRig(t, goBundle)
r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: "0.1", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
if v := r.consider(t, "modules/m"); v.State != builds.BuildDone {
@@ -381,8 +381,8 @@ func TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber(t *te
r.b.Reserved = real
got := r.b.BuildAgain(context.Background(), "m", fp1, "its publish was refused", "shanks.lead", 2*time.Second)
r.b.Wait()
cur, ok, err := real.Of(context.Background(), "m", "0.1.17")
if got.Refused != "" || err != nil || !ok || cur.Fingerprint != fp1 {
t.Fatalf("answer %+v; reservation of 0.1.17 = %+v (%v, %v); want built again under 0.1.17, the catalog's highest 0.1.16 plus one", got, cur, ok, err)
cur, ok, err := real.Of(context.Background(), "m", "")
if got.Refused != "" || err != nil || !ok || cur.Fingerprint != fp1 || cur.Version != "0.1.30" {
t.Fatalf("answer %+v; reservation = %+v (%v, %v); want built again under its standing 0.1.30", got, cur, ok, err)
}
}
@@ -25,6 +25,12 @@ import (
// below the newest the picture holds, whose entries the picture no longer names, asks the catalog seat's verb revision for
// its catalog revision by number, once, and is built when that kept revision is of its source and names every entry it
// asks; it never waits, since the picture only moves on. Only a revision above the newest is waited for.
//
// **An ask naming a revision is held by its number** (design 47 H H-A-222): when the picture does not name the ask's
// entries, the ask's catalog revision is read by number, and the ask is built when the catalog keeps it and it names every
// fingerprint asked, whatever the source the picture holds says (a re-pin keeps a later revision of the same source, H-A-211,
// H-A-213, so the source alone cannot say it). The source is compared only for an ask naming no revision; an ask whose
// revision the catalog does not answer and whose source is not below the picture's waits as before.
// DefaultRevisionWait is the bound of a step ask's wait for its revision, its setting revision-wait-minutes' default (H-A-156).
const DefaultRevisionWait = 2 * time.Minute
@@ -73,10 +79,13 @@ func (b *Builder) holdsRevision(branch string, source int64, entries []meshbuild
return true
}
// keptRevisionHolds answers no outcome when the catalog keeps the ask's revision (its catalog revision by number, of its
// source revision) and that revision names every entry of the ask; else failed or could not run, with its words.
// keptRevisionHolds answers no outcome when the catalog keeps the ask's revision (its catalog revision by number) and that
// revision names every entry of the ask (H-A-205, H-A-222); else failed or could not run, with its words.
func (b *Builder) keptRevisionHolds(ctx context.Context, run meshbuilds.StepRun, held int64) (outcome, said string) {
below := fmt.Sprintf("revision %d of %s is below the %d the builder's picture holds", run.Source, run.Branch, held)
below := fmt.Sprintf("revision %d of %s is not named by the builder's picture, which holds source revision %d", run.Source, run.Branch, held)
if run.Source < held {
below = fmt.Sprintf("revision %d of %s is below the %d the builder's picture holds", run.Source, run.Branch, held)
}
if b.KeptRevision == nil || run.Revision <= 0 {
return meshbuilds.StepEndedOutcomeFailed, below + ", and the ask names no catalog revision to read it by"
}
@@ -85,11 +94,15 @@ func (b *Builder) keptRevisionHolds(ctx context.Context, run meshbuilds.StepRun,
b.logf("%s: %s, and the catalog's revision %d cannot be read: %v", run.Pipeline, below, run.Revision, err)
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s, and the catalog's revision %d cannot be read: %v", below, run.Revision, err)
}
// An ask names the revision and the source of one revision (H-A-218, H-A-219): a kept revision of another source is
// not the ask's.
if source != run.Source {
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s, and the catalog's revision %d is of source revision %d", below, run.Revision, source)
b.logf("%s: %s, and the catalog's revision %d is of source revision %d, the ask names %d: the step fails, nothing is built", run.Pipeline, below, run.Revision, source, run.Source)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s, and the catalog's revision %d is of source revision %d, the ask names %d", below, run.Revision, source, run.Source)
}
for _, e := range run.Entries {
if fps[e.Module] != e.Fingerprint {
b.logf("%s: %s, and the catalog's revision %d names %s at %q, not %s: the step fails, nothing is built", run.Pipeline, below, run.Revision, e.Module, fps[e.Module], e.Fingerprint)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s, and the catalog's revision %d names %s at %q, not %s", below, run.Revision, e.Module, fps[e.Module], e.Fingerprint)
}
}
@@ -116,8 +129,17 @@ func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (ou
if b.Picture == nil || run.Source <= 0 || run.Branch == "" {
return "", ""
}
if held, ok := b.Picture(run.Branch); ok && run.Source < held && !b.holdsRevision(run.Branch, run.Source, run.Entries) {
return b.keptRevisionHolds(ctx, run, held)
if held, ok := b.Picture(run.Branch); !b.holdsRevision(run.Branch, run.Source, run.Entries) {
below := ok && run.Source < held
if (run.Revision > 0 && b.KeptRevision != nil) || below {
// By its number (H-A-222), whatever the source the picture holds; for an ask naming none, only below the
// picture (H-A-205). A kept revision of another source or naming other fingerprints is a mismatch the wait
// cannot mend: it fails at once; only a revision the catalog does not answer is waited for.
outcome, said := b.keptRevisionHolds(ctx, run, held)
if outcome == "" || below || outcome == meshbuilds.StepEndedOutcomeFailed {
return outcome, said
}
}
}
bound := b.revisionWait()
notHeld := fmt.Sprintf("revision %d of %s not in the builder's picture", run.Source, run.Branch)
@@ -266,6 +266,23 @@ func (b *Builder) EntryGone(branch string, first bool, module string) {
}
}
// Withdraw ends the running build of module's fingerprint, superseded, with why (design 47 H H-A-225, ADR 0664 §4): its key
// fails "withdrawn" and its asks are withdrawn from the build seats. A fingerprint with no build running is left.
func (b *Builder) Withdraw(module, fingerprint, why string) {
key, err := builds.BuilderBuildKey(module, fingerprint)
if err != nil {
return
}
b.init()
b.mu.Lock()
withdraw := b.builds[key]
b.mu.Unlock()
if withdraw != nil {
b.logf("%s at %s: the build is withdrawn: %s", module, fingerprint, why)
withdraw(buildrun.Withdrawn{Why: why})
}
}
func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.ModulesByModuleValue, e []meshcatalog.Edge) {
b.mu.Lock()
hk := o.Branch + "\x00" + key
@@ -707,8 +724,9 @@ func (b *Builder) try(ctx context.Context, module string, m meshcatalog.ModulesB
var reserved versions.ReservedVersion
hasReserved := false
if rebuild && cl.Build.State == builds.BuildBuilding {
// Reserved afresh, above the highest the catalog's versions answers, taking over the older unreleased reservation of this
// fingerprint (A-455), never its standing number kept (A-490b); the build's own publish then runs.
// Reserved as A-490b amended by H-A-225 has it: this fingerprint's standing reservation is kept while it is unreleased
// and above the highest the catalog's versions answers, else the next above that highest is taken (A-455); the build's
// own publish then runs.
if b.Reserved.Filled() {
reserved, hasReserved = b.reserve(ctx, module, m, plan, b.originOf(key), true)
}
@@ -327,6 +327,37 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
}
}
// H-A-225 (ADR 0664 §4): a build withdrawn while it runs (its reservation taken over by another fingerprint) ends failed
// "withdrawn" with why, so it never publishes under the dropped number; a fingerprint with no build running is left.
func TestAWithdrawnBuildEndsWithdrawn(t *testing.T) {
r := newRig(t, goBundle)
asked, release := make(chan struct{}), make(chan struct{})
answer := r.answer
r.asks.Answer = func(a buildstep.StepAsk) buildstep.Step {
close(asked)
<-release
return answer(a)
}
r.b.Withdraw("m", fp1, "nothing runs yet") // left: no build of it runs
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20",
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains)})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
select {
case <-asked:
case <-time.After(5 * time.Second):
t.Fatal("no ask was put")
}
r.b.Withdraw("m", fp1, "its reservation 0.1.3 was taken over by the build of src2:b, which reserves 0.1.4")
close(release)
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != builds.BuildFailed || !strings.Contains(v.Error, "withdrawn: its reservation 0.1.3 was taken over") {
t.Fatalf("the key = %+v, want failed withdrawn naming the take-over", v)
}
}
// 1887/AC1, AC2: a trunk fingerprint with no key is one ask on the Go seat's delivery stream, named in the building key
// before it is put, and the builder calls no node; the step a node wrote for it makes the key done naming its reference.
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
@@ -356,7 +356,7 @@ func aStepAskWaitsForItsRevision(t *testing.T) {
held.Store(40)
asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked)
var keptReads atomic.Int64 // H-A-205: a revision above the newest is waited for, never read by number
var keptReads atomic.Int64 // H-A-205, H-A-222: a revision above the newest is read by number once; not answered, it is waited for
r.b.KeptRevision = func(context.Context, string, int64) (int64, map[string]string, error) {
keptReads.Add(1)
return 0, nil, errors.New("not asked")
@@ -385,8 +385,8 @@ func aStepAskWaitsForItsRevision(t *testing.T) {
t.Fatal("the ask was not woken by the entry the picture took in")
}
r.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 || keptReads.Load() != 0 {
t.Errorf("end = %+v, reserved from %+v, kept revisions read %d; want passed with one version reserved from revision 41, none read", end, res.sources, keptReads.Load())
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 || keptReads.Load() != 1 {
t.Errorf("end = %+v, reserved from %+v, kept revisions read %d; want passed with one version reserved from revision 41, read by number once", end, res.sources, keptReads.Load())
}
}
@@ -425,6 +425,45 @@ func anOlderKeptRevisionIsReadByNumber(t *testing.T) {
if end := r2.b.BuildStep(t.Context(), run); end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, "names m at \"src2:other\"") {
t.Errorf("end = %+v; want failed naming the fingerprint the kept revision names", end)
}
// H-A-222: an ask of the very source the picture holds, whose entries the picture no longer names (a re-pin kept revision 48
// of the same source 46 and the picture took its entries in), is held by its revision read by number, never waited for.
r3, res3, _ := revisionRig(t, &held, make(chan struct{}, 1))
r3.b.RevisionWait = time.Nanosecond
r3.b.FirstBranch = func(branch, module string) bool { return branch == "main" && module == "m" }
var read3 []string
r3.b.KeptRevision = func(_ context.Context, branch string, revision int64) (int64, map[string]string, error) {
read3 = append(read3, fmt.Sprintf("%s %d", branch, revision))
return 46, map[string]string{"m": fp1}, nil
}
same := meshbuilds.StepRun{Pipeline: "main.48.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 48, Source: 46,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}
end3 := r3.b.BuildStep(t.Context(), same)
r3.b.Wait()
if end3.Outcome != meshbuilds.StepEndedOutcomePassed || !slices.Equal(read3, []string{"main 48"}) || len(res3.sources) != 1 || res3.sources[0].CatalogRevision != 48 {
t.Errorf("end = %+v, kept revisions read %v, reserved %+v; want passed, revision 48 of main read once and reserved at it", end3, read3, res3.sources)
}
// H-A-218, H-A-222: a kept revision of another source than the ask names, or naming another fingerprint, is a mismatch:
// the step fails at once naming it, never waiting out its bound (the bound here is long, the ask is not below the picture).
for _, c := range []struct {
name string
source int64
fp string
said string
}{
{"another source", 45, fp1, "the catalog's revision 48 is of source revision 45, the ask names 46"},
{"another fingerprint", 46, "src2:other", "names m at \"src2:other\""},
} {
rc, resc, _ := revisionRig(t, &held, make(chan struct{}, 1))
rc.b.RevisionWait = time.Hour
rc.b.KeptRevision = func(context.Context, string, int64) (int64, map[string]string, error) {
return c.source, map[string]string{"m": c.fp}, nil
}
end := rc.b.BuildStep(t.Context(), same)
rc.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, c.said) || len(resc.sources) != 0 {
t.Errorf("%s: end = %+v, reserved %+v; want failed at once saying %q, nothing reserved", c.name, end, resc.sources, c.said)
}
}
}
func aStepAskPastItsRevisionBoundFails(t *testing.T) {
@@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"sort"
"strconv"
"strings"
@@ -74,6 +76,9 @@ type ReservedVersion struct {
// Kept are the registry seats' published heard while the build had not yet recorded every digest and whose digest marked
// no artifact then: kept as heard, they mark their artifact when the build records its digests (design 47 G A-415).
Kept []KeptPublished `json:"kept-published,omitempty"`
// Withdrawn are the versions of the series a take-over by another fingerprint withdrew, each with the fingerprint whose
// build it was (design 47 H H-A-225): a late publish of one is refused as withdrawn. Kept while the series' key stands.
Withdrawn map[string]string `json:"withdrawn,omitempty"`
}
// KeptPublished is a registry seat's published kept on a reservation, as heard (design 47 G A-415).
@@ -154,7 +159,10 @@ func SplitVersion(version string) (series string, patch int, err error) {
// Reserve reserves the next version of module's series for a build that starts, for artifacts (their names), and answers
// it. The next patch is the one after the higher of the highest released and the one reserved, 0 when there is neither. A
// conflicting write (another build reserved first) is read again and the next taken.
// conflicting write (another build reserved first) is read again and the next taken. A reservation standing for the same
// fingerprint whose version is not released and is above the catalog's highest is kept and answered as it stands, a build again or a re-run after a restart
// alike (design 47 H H-A-225, which amends G A-490b): the run already building carries that number. A reservation another
// fingerprint takes over is withdrawn: its version is kept in Withdrawn, and Took is told so the older build is ended.
func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts []string, src Source) (ReservedVersion, error) {
if r.Filled != nil && !r.Filled() {
return ReservedVersion{}, ErrNotFilled
@@ -175,6 +183,11 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
if err != nil {
return ReservedVersion{}, err
}
if found && cur.Fingerprint == src.Fingerprint && cur.Fingerprint != "" && !slices.Contains(released, cur.Version) {
if _, p, err := SplitVersion(cur.Version); err == nil && p > high {
return cur, nil // kept until released (H-A-225); one at or below the catalog's highest can never be released
}
}
next := high
if found && !(src.Rebuild && cur.Fingerprint == src.Fingerprint) {
if _, p, err := SplitVersion(cur.Version); err == nil && p > next {
@@ -189,6 +202,23 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
for _, a := range artifacts {
out.Artifacts[a] = ""
}
if found {
out.Withdrawn = maps.Clone(cur.Withdrawn)
if cur.Fingerprint != src.Fingerprint {
if out.Withdrawn == nil {
out.Withdrawn = map[string]string{}
}
out.Withdrawn[cur.Version] = cur.Fingerprint // a dropped number stays a gap (H-A-226)
}
// One at or below the catalog's highest released is refused by the plain rule anyway: the map never grows without end.
maps.DeleteFunc(out.Withdrawn, func(v, _ string) bool {
_, p, err := SplitVersion(v)
return err != nil || p <= high
})
if len(out.Withdrawn) == 0 {
out.Withdrawn = nil
}
}
raw, _ := json.Marshal(out)
if found {
_, err = r.Store.Update(ctx, key, raw, rev)
@@ -711,11 +741,24 @@ func AnswerReservedByName(ctx context.Context, r Reader, carriers func(context.C
}
return out, nil
}
if series, _, err := SplitVersion(version); err == nil {
if cur, _, found, err := rs.get(ctx, ReservedKey(module, series)); err == nil && found && hasKey(cur.Withdrawn, version) {
// A late publish of a build a take-over withdrew (H-A-225): refused as what it is.
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) version %s was withdrawn: the build of %s it was reserved for was taken over by %s, which reserves %s",
name, module, version, cur.Withdrawn[version], cur.Fingerprint, cur.Version)}, nil
}
}
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no version %s reserved or released", name, module, version)}, nil
}
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no reserved version", name, module)}, nil
}
// hasKey says whether m holds k.
func hasKey(m map[string]string, k string) bool {
_, ok := m[k]
return ok
}
// NoEntryCarries opens the answer of reserved given a name no entry carries.
const NoEntryCarries = "no entry carries the name "
@@ -221,18 +221,46 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
}
}
// TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas: design 47 G A-490b — a build again of a done key replaces
// its fingerprint's standing reservation (and tells Took) and numbers above the catalog's highest alone, whether the standing number
// is below it or above it; another fingerprint's standing reservation stays a floor.
// TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226 — a dropped number stays a gap, kept in
// the series key's Withdrawn while it is above the catalog's highest released; at each write those at or below it go, so the
// map never grows without end.
func TestTheWithdrawnVersionsAreDroppedOnceReleasedPast(t *testing.T) {
ctx := context.Background()
store := &reserveMem{mem: newMem()}
highest := "0.1.3"
r := &Reserver{Store: store, Filled: func() bool { return true },
ReleasedOf: func(context.Context, string) ([]string, error) { return []string{highest}, nil }}
for _, fp := range []string{"src2:a", "src2:b", "src2:c"} {
if _, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: fp}); err != nil {
t.Fatal(err)
}
}
cur, _, _, _ := r.get(ctx, ReservedKey("m", "0.1"))
if cur.Version != "0.1.6" || len(cur.Withdrawn) != 2 || cur.Withdrawn["0.1.4"] != "src2:a" || cur.Withdrawn["0.1.5"] != "src2:b" {
t.Fatalf("reservation %+v; want 0.1.6 with 0.1.4 and 0.1.5 withdrawn", cur)
}
highest = "0.1.5" // the catalog released past both withdrawn numbers but the newest
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "src2:d"})
if err != nil || got.Version != "0.1.7" || len(got.Withdrawn) != 1 || got.Withdrawn["0.1.6"] != "src2:c" {
t.Errorf("reserved %+v (%v); want 0.1.7 keeping only 0.1.6 withdrawn", got, err)
}
}
// TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas: design 47 G A-490b as H-A-225 amends it — a build again
// of a done key, or a re-run after a restart, keeps its fingerprint's standing reservation while it is unreleased and above the
// catalog's highest; one at or below it is replaced (Took told) by the next above the catalog's highest alone; another
// fingerprint's standing reservation stays a floor and is withdrawn.
func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *testing.T) {
ctx := context.Background()
for _, c := range []struct {
name, standing, standingFp string
want string
rebuild, kept bool
}{
{name: "standing below the catalog's highest", standing: "0.1.0", standingFp: "fp", want: "0.1.17"},
{name: "standing above the catalog's highest", standing: "0.1.30", standingFp: "fp", want: "0.1.17"},
{name: "another fingerprint's standing number stays a floor", standing: "0.1.30", standingFp: "other", want: "0.1.31"},
{name: "standing below the catalog's highest", standing: "0.1.0", standingFp: "fp", want: "0.1.17", rebuild: true},
{name: "H-A-225: standing above the catalog's highest is kept by a build again", standing: "0.1.30", standingFp: "fp", want: "0.1.30", rebuild: true, kept: true},
{name: "H-A-225: a re-run after a restart keeps its standing number", standing: "0.1.111", standingFp: "fp", want: "0.1.111", kept: true},
{name: "another fingerprint's standing number stays a floor", standing: "0.1.30", standingFp: "other", want: "0.1.31", rebuild: true},
} {
t.Run(c.name, func(t *testing.T) {
store := &reserveMem{mem: newMem()}
@@ -246,10 +274,19 @@ func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *
r.Took = func(_ context.Context, older, newer ReservedVersion) {
took = append(took, older.Version+">"+newer.Version)
}
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "fp", Rebuild: true})
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "fp", Rebuild: c.rebuild})
if c.kept {
if err != nil || got.Version != c.want || len(took) != 0 || got.Artifacts["m:a"] != "d" {
t.Fatalf("reserved %+v (%v), took %v; want the standing %s kept as it stands", got, err, took, c.want)
}
return
}
if err != nil || got.Version != c.want || len(took) != 1 || took[0] != c.standing+">"+c.want {
t.Fatalf("reserved %q (%v), took %v; want %s replacing %s", got.Version, err, took, c.want, c.standing)
}
if w := got.Withdrawn[c.standing]; (c.standingFp != "fp") != (w == c.standingFp) {
t.Errorf("withdrawn %v; want %s withdrawn only when another fingerprint stood", got.Withdrawn, c.standing)
}
})
}
}
@@ -67,6 +67,20 @@ func TestReservedByNameAnswersFromTheViewAndTheCatalog(t *testing.T) {
t.Errorf("%s = %+v, want error naming %q alone", name, got, c.want)
}
}
// H-A-225: another fingerprint taking the series over withdraws the older's version; a late publish of it is refused as
// withdrawn, naming the fingerprint that took it over, never as a version that was never reserved.
if _, err := r.Reserve(ctx, "mesh-sdk", "0.1", []string{"mesh-sdk:ts"}, Source{Fingerprint: "src2:a"}); err != nil {
t.Fatal(err)
}
if _, err := r.Reserve(ctx, "mesh-sdk", "0.1", []string{"mesh-sdk:ts"}, Source{Fingerprint: "src2:b"}); err != nil {
t.Fatal(err)
}
if got := ask("@novox/mesh-sdk", "0.1.5"); !strings.Contains(got.Error, "version 0.1.5 was withdrawn: the build of src2:a it was reserved for was taken over by src2:b, which reserves 0.1.6") {
t.Errorf("a late publish of a withdrawn version = %+v, want refused as withdrawn", got)
}
if got := ask("@novox/mesh-sdk", "9.9.9"); !strings.Contains(got.Error, "reserved or released") {
t.Errorf("a version never reserved = %+v, want refused as neither reserved nor released", got)
}
if _, err := AnswerReservedByName(ctx, rd, Carriers(packagesMem(nil)), released, "x", ""); err == nil {
t.Error("an unreadable view was answered as if no entry carried the name")
}
@@ -7,6 +7,8 @@ import (
"reflect"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -172,25 +174,43 @@ func (f *fakeBranchJS) Consumer(_ context.Context, stream, name string) (jetstre
func (f *fakeBranchJS) CreateConsumer(_ context.Context, stream string, cfg jetstream.ConsumerConfig) (jetstream.Consumer, error) {
f.made = append(f.made, cfg)
c := &fakeBranchConsumer{}
c := newFakeBranchConsumer(stream, cfg.Durable)
f.held[stream+"/"+cfg.Durable] = c
return c, nil
}
// fakeBranchConsumer hands one source-moved to the hearing as soon as it is consumed.
// fakeBranchConsumer hands one source-moved to the hearing as soon as it is consumed. It answers its CachedInfo as the bus
// does, with its stream, name and created time, which the SDK's guard of one pull per durable keys by (design 47 H-A-196).
type fakeBranchConsumer struct {
jetstream.Consumer
info jetstream.ConsumerInfo
heard int
}
type fakeConsumeContext struct{ jetstream.ConsumeContext }
// fakeConsumersMade numbers the fakes, so each is created at a time of its own, as each durable the bus makes is.
var fakeConsumersMade atomic.Int64
func (fakeConsumeContext) Stop() {}
func newFakeBranchConsumer(stream, durable string) *fakeBranchConsumer {
return &fakeBranchConsumer{info: jetstream.ConsumerInfo{Stream: stream, Name: durable, Config: jetstream.ConsumerConfig{Durable: durable},
Created: time.Unix(0, fakeConsumersMade.Add(1)).UTC()}}
}
func (c *fakeBranchConsumer) CachedInfo() *jetstream.ConsumerInfo { return &c.info }
// fakeConsumeContext is a consume that ends when stopped, as the guard's release waits for (H-A-196).
type fakeConsumeContext struct {
jetstream.ConsumeContext
stop sync.Once
closed chan struct{}
}
func (c *fakeConsumeContext) Stop() { c.stop.Do(func() { close(c.closed) }) }
func (c *fakeConsumeContext) Closed() <-chan struct{} { return c.closed }
func (c *fakeBranchConsumer) Consume(handle jetstream.MessageHandler, _ ...jetstream.PullConsumeOpt) (jetstream.ConsumeContext, error) {
c.heard++
handle(msgOf{f: &fakeMsg{subject: "mesh.seat.mesh-delivery.event.source-moved.main.2"}})
return fakeConsumeContext{}, nil
return &fakeConsumeContext{closed: make(chan struct{})}, nil
}
// Design 47 H §743, H-A-134: a branch's durable the bus does not hold the indexer makes, as a module's seat-event consumer is
@@ -207,7 +227,7 @@ func TestABranchDurableNotThereIsMadeAndHeard(t *testing.T) {
name := "n1_mesh-delivery-indexer_" + keys.EscapeName("main")
var existing *fakeBranchConsumer
if tc.held {
existing = &fakeBranchConsumer{}
existing = newFakeBranchConsumer(meshdelivery.StreamSourceMoved, name)
js.held[meshdelivery.StreamSourceMoved+"/"+name] = existing
}
var heard []string
@@ -0,0 +1,147 @@
package main
import (
"context"
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-indexer/internal/index"
)
// askedBy is a takerIn that names the hand act it was asked.
type askedBy struct{ asked []string }
func (s *askedBy) did(act string) index.TakenIn {
s.asked = append(s.asked, act)
return index.TakenIn{Module: "m", Stored: true, Answer: act}
}
func (s *askedBy) SeatDefinitionsAgain(context.Context, string) index.TakenIn {
return s.did("seat-definitions")
}
func (s *askedBy) FingerprintsAgain(context.Context, string) index.TakenIn {
return s.did("fingerprints")
}
func (s *askedBy) RecomputeByHand(context.Context, string, string) index.TakenIn {
return s.did("recompute")
}
func (s *askedBy) DropCandidateGraph(context.Context, string, string, string, string) index.TakenIn {
return s.did("candidate-group")
}
func (s *askedBy) NamedDrop(context.Context, string, string, string, string) index.TakenIn {
return s.did("named-fingerprint")
}
func (s *askedBy) PinDrop(context.Context, string, string, string) index.TakenIn {
return s.did("pin-drop")
}
func (s *askedBy) DropRunByHand(context.Context, string, string, string) index.TakenIn {
return s.did("run-drop")
}
func (s *askedBy) RegistryAct(context.Context, index.TakeInAsk) index.TakenIn {
return s.did("registry")
}
func (s *askedBy) HeldAgain(context.Context, string) index.TakenIn { return s.did("held") }
func (s *askedBy) HeldSkip(context.Context, string, int64, string, string) index.TakenIn {
return s.did("held-skip")
}
func (s *askedBy) TakeInAgain(context.Context, string) index.TakenIn { return s.did("again") }
// answeredKV keeps the answer takenIn writes back.
type answeredKV struct {
jetstream.KeyValue
written []byte
}
func (k *answeredKV) Update(_ context.Context, _ string, value []byte, _ uint64) (uint64, error) {
k.written = value
return 2, nil
}
type askEntry struct {
jetstream.KeyValueEntry
key string
value []byte
}
func (e askEntry) Key() string { return e.key }
func (e askEntry) Value() []byte { return e.value }
func (e askEntry) Revision() uint64 { return 1 }
func (e askEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
// Design 47 H-A-208, H-A-210, H §774: every hand tool's takeins value, built by the tool's own Run path (or the constructor it
// calls), is read by DecodeTakeIn and acted on by takenIn as the act it names; indexer_take_in_again's was refused, its Again
// missing from a hand-kept list. Every field of TakeInAsk says by its tag whether it is an act, so an act added is read.
func TestEveryHandToolsAskIsReadAndActedOn(t *testing.T) {
t.Run("every field is classed", func(t *testing.T) {
ty := reflect.TypeFor[index.TakeInAsk]()
for i := range ty.NumField() {
if c := ty.Field(i).Tag.Get("takein"); !slices.Contains([]string{"act", "argument", "ask"}, c) {
t.Errorf("TakeInAsk.%s has takein %q, want act, argument or ask", ty.Field(i).Name, c)
}
}
})
var values [][]byte
ask := func(_ string, value []byte) (index.TakeInAsk, error) {
values = append(values, value)
return index.TakeInAsk{V: 1, Result: &index.TakenIn{Stored: true}, Answered: time.Now()}, nil
}
record := func(string, string) error { return nil }
tools := map[string]func(map[string]any) (any, error){}
for _, tl := range slices.Concat(index.HeldTools(nil, ask, time.Now, record), index.PinsTools(nil, nil, ask, time.Now, record),
index.RunsTools(nil, nil, ask, time.Now, record), index.NamedTools(ask, time.Now, record),
index.FingerprintsTools(ask, time.Now, record, time.Minute), index.CatalogRevisionTools(nil, ask, time.Now, record),
index.RegistrationTools(nil, nil, ask, time.Now, func(string, string, string, string) error { return nil })) {
tools[tl.Name] = tl.Run
}
tools[index.SeatDefinitionsAgainTool] = func(args map[string]any) (any, error) { // its tool lives in the tools bundle
_, value, err := index.NewSeatDefinitionsAgain(args["module"].(string), "why", "by", time.Now())
values = append(values, value)
return nil, err
}
for _, c := range []struct {
tool string
args map[string]any
act string
}{
{index.TakeInAgainTool, map[string]any{"repository": "novox/mesh-host", "why": "rule changed"}, "again"},
{index.TakeInPendingTool, map[string]any{"branch": "main"}, "held"},
{index.TakeInSkipTool, map[string]any{"branch": "main", "source": float64(4), "why": "stale"}, "held-skip"},
{index.PinDropTool, map[string]any{"key": "main.7", "why": "no pipeline"}, "pin-drop"},
{index.RunDropTool, map[string]any{"run": "run-12", "why": "never ends"}, "run-drop"},
{index.NamedDropTool, map[string]any{"fingerprint": "abc", "member": "ace.feh", "why": "gone"}, "named-fingerprint"},
{index.FingerprintsAgainTool, map[string]any{"module": "feh", "why": "unread packages"}, "fingerprints"},
{index.RecomputeTool, map[string]any{"why": "check"}, "recompute"},
{index.DropCandidateGraphTool, map[string]any{"group": "g", "heads": "h", "why": "no gate event"}, "candidate-group"},
{index.SeedAgainTool, map[string]any{"why": "empty"}, "registry"},
{index.SeatDefinitionsAgainTool, map[string]any{"module": "mesh-contracts"}, "seat-definitions"},
} {
t.Run(c.tool, func(t *testing.T) {
run, ok := tools[c.tool]
if !ok {
t.Fatalf("no tool %s", c.tool)
}
values = nil
if _, err := run(c.args); err != nil || len(values) != 1 {
t.Fatalf("%s wrote %d values (%v)", c.tool, len(values), err)
}
if _, err := index.DecodeTakeIn(values[0]); err != nil {
t.Fatalf("DecodeTakeIn refuses %s: %v", values[0], err)
}
ix, kv := &askedBy{}, &answeredKV{}
var said []string
takenIn(t.Context(), ix, kv, askEntry{key: "k", value: values[0]}, func(f string, a ...any) { said = append(said, f) })
if !slices.Equal(ix.asked, []string{c.act}) {
t.Errorf("takenIn asked %v of %s, want %s (said %s)", ix.asked, values[0], c.act, strings.Join(said, "; "))
}
var answered index.TakeInAsk
if err := json.Unmarshal(kv.written, &answered); err != nil || !answered.Done() {
t.Errorf("takenIn wrote back %s (%v), want the answered ask", kv.written, err)
}
})
}
}
@@ -93,16 +93,20 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
if e, ok, err := modules.Get("ca-trust"); err != nil || !ok || e.Revision != 2 {
t.Errorf("after the event ca-trust = %+v %v (%v), want it at revision 2", e, ok, err)
}
// A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held.
// A-487c: the manifests by digest answer from the view held; the verb revision asked of the trunk's newest lists the bucket
// once for which revision is newest (H-A-222) and takes the entries of the view held at that very revision.
listed = objects.n.Load()
if raw, err := kept.Manifest(meshcatalogclient.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err)
}
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("the manifest listed the bucket %d times, want none", got)
}
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 {
t.Errorf("revision of the newest answered %+v (%v), want revision 2 whole", a, err)
}
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("the manifest and the verb revision listed the bucket %d times, want none", got)
if got := objects.n.Load() - listed; got != 1 {
t.Errorf("the verb revision listed the bucket %d times, want once, for the newest", got)
}
if len(said) != 2 || !strings.Contains(said[1], "read revision 2 (was 1), 2 modules, in ") {
t.Errorf("said %q, want each read with its revision, module count and time", said)
@@ -126,6 +130,16 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
if keys, _ := modules.Keys(); len(keys) != 3 {
t.Errorf("the next heard event left the view at %v, want revision 3's three modules", keys)
}
// H-A-222: two revisions kept that the view has not heard are answered by the verb revision from the store, whole; the
// view held stays a cache at its own revision.
keep(4, "avahi", "ca-trust", "dbus")
keep(5, "avahi", "ca-trust", "dbus", "feh")
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision == nil || a.Revision.Revision != 5 || len(a.Entries) != 4 {
t.Errorf("revision of the newest answered %+v (%v) while the view held 3, want revision 5 whole from the store", a, err)
}
if keys, _ := modules.Keys(); len(keys) != 3 {
t.Errorf("the verb moved the view held to %v, want it at revision 3 until it hears the event", keys)
}
}
// A-487: the verb module reads the one module it is asked.
@@ -60,8 +60,9 @@ func (t *goModTree) sameRepositoryDir(required string) (string, bool) {
}
// sameRepositoryRequired are the directories (from the root, sorted, each once) of every same-repository Go module the
// go.mod of each start directory requires, transitively through those modules' own go.mod files (H-A-202, H-A-157). The
// start directories themselves are not among them.
// go.mod of each start directory requires, transitively through those modules' own go.mod files (H-A-202, H-A-157). A
// same-repository module a go.mod names in a replace directive counts as required, whether or not a require line names it
// (H-A-224). The start directories themselves are not among them.
func (t *goModTree) sameRepositoryRequired(start []string) []string {
seen := map[string]bool{}
for _, s := range start {
@@ -72,7 +73,8 @@ func (t *goModTree) sameRepositoryRequired(start []string) []string {
for len(queue) > 0 {
dir := queue[0]
queue = queue[1:]
for _, req := range fingerprint.MeshRequiresOf(t.goMod(dir)) {
raw := t.goMod(dir)
for _, req := range append(fingerprint.MeshRequiresOf(raw), fingerprint.ReplacedSiblings(raw, t.f.Repository)...) {
d, ok := t.sameRepositoryDir(req)
if !ok || seen[d] {
continue
@@ -2,6 +2,7 @@ package index
import (
"context"
"slices"
"strings"
"testing"
@@ -141,4 +142,23 @@ func TestTheEntryPinsEveryRequirementOfTheModuleGraph(t *testing.T) {
if _, pinned := pins[builds]; pinned || len(pins) != 1 {
t.Errorf("pins = %v, want %s only, never a module of the repository", pins, conditions)
}
// H-A-224: a same-repository module the go.mod names only in a replace directive (the watchman's buildrun) counts as
// required: its own requirement of another repository's client is pinned too.
const assigner = "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
objects["modules/d/buildrun"] = strings.Repeat("d", 40)
ix, mods, _ = ptpIndexer(t, objects)
mod["modules/g"] += "\nreplace git.novox.be/novox/mesh-delivery/modules/d/buildrun => ../d/buildrun\n"
mod["modules/d/buildrun"] = "module git.novox.be/novox/mesh-delivery/modules/d/buildrun\n\nrequire " + assigner + ptpZero + "\n"
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := mod[dir]; ok {
return []byte(c), nil
}
return []byte(ptpGoMods[dir]), nil
}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Inputs2 == nil || !slices.Contains(m.Inputs2.Packages, assigner) || !slices.Contains(m.Inputs2.Packages, conditions) {
t.Errorf("packages = %+v, want the replaced sibling's requirement %s pinned beside the client's", m.Inputs2, assigner)
}
}
@@ -23,17 +23,22 @@ const RevisionInvoke = "seat:" + meshcatalog.Seat + "." + meshcatalog.VerbRevisi
// H-A-66: every account may call every verb). With no trunk revision kept yet the answer is `none-yet` alone (design 47 G §530), never an empty
// catalog and never an error.
//
// kept, when not nil, is the trunk's newest revision held in memory: the trunk's newest answers from it with no read of
// the bucket (A-487c); a candidate, a branch and an older number read the bucket.
// The trunk's newest is the newest the bucket keeps at the time of the ask, read from the store (a listing of the names and
// one object, design 47 H-A-222), never the view held in memory, which runs behind until it hears the revision. kept, when
// not nil, is that view: a cache of the entries only, used when it holds the very revision read; else the entries are read
// from the bucket. A candidate, a branch and an older number read the bucket.
func Revision(ctx context.Context, o Objects, kept *KeptTrunk, in meshcatalog.RevisionInput) (meshcatalog.RevisionAnswer, error) {
var out meshcatalog.RevisionAnswer
var newest meshcatalog.CatalogRevision
var ok bool
var trunkEntries, branchEntries []meshcatalog.CatalogEntry
var err error
if kept != nil {
newest, trunkEntries, ok, err = kept.Newest(ctx, o)
} else {
newest, ok, err := newestOf(ctx, o, TrunkBranch)
cached := false
if err == nil && ok && kept != nil {
if held, entries, any, kerr := kept.Newest(ctx, o); kerr == nil && any && held.Revision == newest.Revision {
trunkEntries, cached = entries, true
}
}
if !cached && (err != nil || ok) {
// readNewestOf reads the entries whole, and the newest again when the pin judge removed one under the read.
newest, trunkEntries, ok, err = readNewestOf(ctx, o, TrunkBranch)
}
if err != nil {
@@ -4,6 +4,8 @@ import (
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"time"
)
@@ -18,45 +20,48 @@ const (
)
// TakeInAsk is a value of the takeins state: which hand act a person asked, why, who and when, and, once the indexer read
// it, its answer.
// it, its answer. Every field says by its tag takein what it is: an act (a field that alone names a hand act the indexer
// serves), an argument of an act, or of the ask itself (version, who, why, when, the answer); DecodeTakeIn reads the acts
// from these tags, so a hand act added here is read the moment its field is (H-A-208, H-A-210: indexer_take_in_again's
// Again was refused by a hand-kept list).
type TakeInAsk struct {
V int `json:"v"`
Module string `json:"module,omitempty"`
Why string `json:"why,omitempty"`
V int `json:"v" takein:"ask"`
Module string `json:"module,omitempty" takein:"argument"`
Why string `json:"why,omitempty" takein:"ask"`
// SeatDefinitions asks a contracts module's trunk seat definitions to be read again now (indexer_seat_definitions_again,
// DLV-I19).
SeatDefinitions bool `json:"seat-definitions,omitempty"`
SeatDefinitions bool `json:"seat-definitions,omitempty" takein:"act"`
// Fingerprints asks the fingerprints of Module ("all": every module) to be computed again now, from the published
// packages as they are read (indexer_fingerprints_again, fingerprints.go).
Fingerprints bool `json:"fingerprints,omitempty"`
Fingerprints bool `json:"fingerprints,omitempty" takein:"act"`
// Recompute asks the catalog graph to be computed again now and compared with the newest kept revision (indexer_recompute).
Recompute bool `json:"recompute,omitempty"`
Recompute bool `json:"recompute,omitempty" takein:"act"`
// CandidateGroup and CandidateHeads, with Drop, ask that candidate's catalog half to be dropped (indexer_drop_candidate_graph).
CandidateGroup string `json:"candidate-group,omitempty"`
CandidateHeads string `json:"candidate-heads,omitempty"`
Drop bool `json:"drop,omitempty"`
CandidateGroup string `json:"candidate-group,omitempty" takein:"act"`
CandidateHeads string `json:"candidate-heads,omitempty" takein:"argument"`
Drop bool `json:"drop,omitempty" takein:"argument"`
// NamedFingerprint and NamedMember, with Drop, ask one member to be removed from that fingerprint's set of the state named
// (indexer_named_drop, named.go, design 47 G §675).
NamedFingerprint string `json:"named-fingerprint,omitempty"`
NamedFingerprint string `json:"named-fingerprint,omitempty" takein:"act"`
// PinDrop asks one pin of the state pins, by its key, to be dropped (indexer_pin_drop, design 47 H §765).
PinDrop string `json:"pin-drop,omitempty"`
PinDrop string `json:"pin-drop,omitempty" takein:"act"`
// Held names the watched branch whose held source-moved is taken in again now (indexer_take_in_pending) or, with Skip and
// HeldSource, ended unwritten (indexer_take_in_skip; design 47 H §744).
Held string `json:"held,omitempty"`
HeldSource int64 `json:"held-source,omitempty"`
Skip bool `json:"skip,omitempty"`
NamedMember string `json:"named-member,omitempty"`
Held string `json:"held,omitempty" takein:"act"`
HeldSource int64 `json:"held-source,omitempty" takein:"argument"`
Skip bool `json:"skip,omitempty" takein:"argument"`
NamedMember string `json:"named-member,omitempty" takein:"argument"`
// Again names the repository taken in again at its kept merge commit (indexer_take_in_again, take_in_again.go; H-A-208, H-A-210).
Again string `json:"again,omitempty"`
Again string `json:"again,omitempty" takein:"act"`
// RunDrop asks one run on request, by its group run-<id>, to be dropped with its candidate (indexer_run_drop, runs_tools.go, A-194).
RunDrop string `json:"run-drop,omitempty"`
RunDrop string `json:"run-drop,omitempty" takein:"act"`
// Registry asks one of the registrations' acts (registrations.go): a trusted change of one, the verb registrations,
// indexer_proposal_drop or indexer_seed_again (design 47 G §3, §4, §8, §667).
Registry *RegistryAct `json:"registry,omitempty"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Result *TakenIn `json:"result,omitempty"`
Answered time.Time `json:"answered,omitzero"`
Registry *RegistryAct `json:"registry,omitempty" takein:"act"`
By string `json:"by,omitempty" takein:"ask"`
At time.Time `json:"at" takein:"ask"`
Result *TakenIn `json:"result,omitempty" takein:"ask"`
Answered time.Time `json:"answered,omitzero" takein:"ask"`
}
// Done says the ask is answered.
@@ -71,8 +76,33 @@ func DecodeTakeIn(data []byte) (TakeInAsk, error) {
if a.V != 1 {
return a, fmt.Errorf("a takeins value of version %d, not 1", a.V)
}
if a.Module == "" && !a.Fingerprints && !a.Recompute && a.CandidateGroup == "" && a.NamedFingerprint == "" && a.PinDrop == "" && a.Held == "" && a.RunDrop == "" && a.Registry == nil {
if len(a.Acts()) == 0 {
return a, errors.New("a takeins value naming no hand act the indexer serves")
}
return a, nil
}
// takeInActs are the indices of TakeInAsk's fields tagged takein:"act".
var takeInActs = func() []int {
var out []int
t := reflect.TypeFor[TakeInAsk]()
for i := range t.NumField() {
if t.Field(i).Tag.Get("takein") == "act" {
out = append(out, i)
}
}
return out
}()
// Acts names the hand acts the ask names, by their json names: its fields tagged takein:"act" that are set.
func (a TakeInAsk) Acts() []string {
var out []string
v := reflect.ValueOf(a)
for _, i := range takeInActs {
if !v.Field(i).IsZero() {
name, _, _ := strings.Cut(v.Type().Field(i).Tag.Get("json"), ",")
out = append(out, name)
}
}
return out
}
@@ -160,6 +160,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Kept: keptReader(ctx, kept), Toolchain: toolchainReader(ctx, toolchains),
Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing}
hearer.AskSteps = starter.AskSteps
starter.Supersede = func(newer seat.Pipeline) { hearer.SupersedeRunning(newer) }
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.AskPinHeld = func(key, module string, pins map[string]string) {
if _, err := starter.AskPinHeld(hearer.Store, key, module, pins, widener.Revision); err != nil {
@@ -162,6 +162,9 @@ type Starter struct {
// Ended says a pipeline that ended at its placing (holderless): the step-ended of each step of steps, then its end
// (Hearer.EndedAtPlacing); nil says it in the journal alone.
Ended func(p seat.Pipeline, steps []int) error
// Supersede takes out of every older running pipeline of newer's branch each mover whose fingerprint newer moved
// (H-A-223; Hearer.SupersedeRunning); nil leaves running pipelines as they are.
Supersede func(newer seat.Pipeline)
// Fills refuses a start while a store of the runner is full (§366); nil refuses none.
Fills *Fills
// Recovered says whether a commit is named a recovery (§81, §86): its steps are asked on the build seats' stream delivery.
@@ -538,6 +541,10 @@ func (s *Starter) Start(b Begin) (bool, error) {
// A waiting pipeline says pipeline-started at its write, so its revision is pinned while it waits (H-A-52), and its
// wait on its turn.
s.sayStartedPipeline(p, now)
// A mover whose fingerprint this revision moved leaves every older running pipeline of the branch (H-A-223).
if s.Supersede != nil {
s.Supersede(p)
}
if !place {
s.sayQueueWait(seat.EventWaitHeld, p, now, "")
if _, err := s.supersedeWaiting(p); err != nil {
@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
@@ -611,9 +612,16 @@ func endPipeline(p *seat.Pipeline, now time.Time, unplaced func(seat.Pipeline, s
}
// A mover with no step placed waits on its registration or a publish: it holds the end only while every placed mover's
// required steps passed, since once one failed or could not run the pipeline can no longer pass (H-A-163, H-A-171).
// A mover that left for a newer pipeline (H-A-223) holds nothing and fails nothing: its steps are not judged, and a
// pipeline every mover left ends superseded.
if allLeft(*p) {
p.Outcome, p.Why, p.Ended = seat.PipelineOutcomeSuperseded, "every mover left it for a newer pipeline (H-A-223)", &now
p.EndCount++
return true
}
var waiting []string
for _, m := range p.Movers {
if !placed(*p, m.Module) {
if !placed(*p, m.Module) && !slices.Contains(p.Superseded, m.Module) {
waiting = append(waiting, m.Module)
}
}
@@ -621,6 +629,8 @@ func endPipeline(p *seat.Pipeline, now time.Time, unplaced func(seat.Pipeline, s
var notAsked []int
for i, s := range p.Steps {
switch {
case slices.Contains(p.Superseded, s.Module):
continue
case s.Step == seat.StepResultStepApproval && s.Outcome == seat.StepResultOutcomeWaiting:
notAsked = append(notAsked, i)
continue
@@ -1,7 +1,9 @@
package pipeline
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -377,3 +379,105 @@ func TestARecoverysStepsAreAskedOnTheStreamDelivery(t *testing.T) {
t.Errorf("recovered: said %v", said)
}
}
// A mover whose fingerprint a newer revision moved leaves a running pipeline (H-A-223, H-A-163): its open steps end
// superseded, never failed, the other mover's go on and the pipeline ends by them alone; one every mover left ends
// superseded.
func TestAMovedMoverLeavesARunningPipeline(t *testing.T) {
t0 := time.Date(2026, 11, 21, 1, 0, 0, 0, time.UTC)
running := func() seat.Pipeline {
return seat.Pipeline{V: 1, Pipeline: "main.54", Branch: "main", Source: 54, Revision: 54, Outcome: seat.PipelineOutcomeRunning, Started: &t0,
Movers: []seat.Mover{{Module: "a", Commit: "c1", Fingerprint: "src2:a1"}, {Module: "b", Commit: "c1", Fingerprint: "src2:b1"}},
Steps: []seat.StepResult{
{Module: "a", Step: seat.StepResultStepBuild, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomeRunning, Ask: "x1", Started: &t0},
{Module: "b", Step: seat.StepResultStepBuild, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomeRunning, Ask: "x2", Started: &t0},
{Module: "b", Step: seat.StepResultStepLint, Artifact: "b:program", Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomeRunning, Ask: "x3", Started: &t0,
Seat: nodebuildgo.Seat, Node: "n2"}}}
}
rows := []struct {
name string
newer []seat.Mover
want string
}{
{"a newer revision moving b: b's steps end superseded, a's go on and end the pipeline passed alone",
[]seat.Mover{{Module: "b", Commit: "c2", Fingerprint: "src2:b2"}}, "running; a build running; b build superseded; b lint superseded | passed"},
{"a newer revision moving every mover ends the pipeline superseded",
[]seat.Mover{{Module: "a", Commit: "c2", Fingerprint: "src2:a2"}, {Module: "b", Commit: "c2", Fingerprint: "src2:b2"}},
"superseded; a build superseded; b build superseded; b lint superseded | superseded"},
{"a newer revision at the same fingerprint takes nothing", []seat.Mover{{Module: "b", Commit: "c1", Fingerprint: "src2:b1"}},
"running; a build running; b build running; b lint running | running"},
}
for _, r := range rows {
t.Run(r.name, func(t *testing.T) {
store := memStore{}
store["main.54"] = struct {
p seat.Pipeline
rev uint64
}{running(), 1}
var said, dropped []string
h := &Hearer{Store: store, Say: func(subject string, _ any) error { said = append(said, subject); return nil }, Now: func() time.Time { return t0 },
DropRun: func(buildSeat, node, ask, why string) error {
dropped = append(dropped, buildSeat+" "+node+" "+ask+" "+fmt.Sprint(strings.Contains(why, "main.57")))
return nil
}}
h.SupersedeRunning(seat.Pipeline{Pipeline: "main.57", Branch: "main", Revision: 57, Movers: r.newer})
// b's lint ask on its build node is dropped naming the newer pipeline; the build ask on mesh-builds, which
// serves no drop, is not (H-A-223).
wantDropped := ""
if r.newer[0].Fingerprint != "src2:b1" {
wantDropped = "node-build-go n2 x3 true"
}
if got := strings.Join(dropped, ","); got != wantDropped {
t.Errorf("dropped %q; want %q", got, wantDropped)
}
p, _, _ := store.Get("main.54")
got := []string{p.Outcome}
for _, s := range p.Steps {
got = append(got, s.Module+" "+s.Step+" "+s.Outcome)
if s.Outcome == seat.StepResultOutcomeSuperseded && !strings.Contains(s.Why, "main.57") {
t.Errorf("the superseded step's why %q names no newer pipeline", s.Why)
}
}
// The mover left holds nothing: a's build passing ends the pipeline by a alone.
if p.Outcome == seat.PipelineOutcomeRunning {
p.Steps[0].Outcome, p.Steps[0].Ended = seat.StepResultOutcomePassed, &t0
endPipeline(&p, t0, nil)
}
if s := strings.Join(got, "; ") + " | " + p.Outcome; s != r.want {
t.Errorf("got %s\nwant %s", s, r.want)
}
})
}
// The live path: a newer pipeline started through Start, a take-in's or a release revision's alike, takes the moved
// mover out of the older running pipeline over the same store (H-A-223, H-A-213).
starts := []struct {
name string
begin Begin
}{
{"a take-in", Begin{Branch: "main", Source: 55, Revision: 55, Call: "0123456789abcdef", Movers: []seat.Mover{{Module: "b", Commit: "c2", Fingerprint: "src2:b2"}}}},
{"a release revision", Begin{Branch: "main", Source: 54, Revision: 57, Call: "release of mesh-sdk 0.1.26", Movers: []seat.Mover{{Module: "b", Commit: "c1", Fingerprint: "src2:b2"}}}},
}
for _, r := range starts {
t.Run("started by "+r.name, func(t *testing.T) {
store := &MoverStore{Raw: newMemRaw()}
raw, _ := json.Marshal(running())
if err := store.Create("main.54", raw); err != nil {
t.Fatal(err)
}
h := &Hearer{Store: store, Say: func(string, any) error { return nil }, Now: func() time.Time { return t0 }}
s := &Starter{Pipelines: store, Store: store, Say: func(string, any) error { return nil }, Now: func() time.Time { return t0 },
Supersede: func(newer seat.Pipeline) { h.SupersedeRunning(newer) }}
if ok, err := s.Start(r.begin); !ok || err != nil {
t.Fatalf("start = %v, %v", ok, err)
}
p, _, _ := store.Get("main.54")
var got []string
for _, st := range p.Steps {
got = append(got, st.Module+" "+st.Step+" "+st.Outcome)
}
if s := p.Outcome + "; " + strings.Join(got, "; "); s != "running; a build running; b build superseded; b lint superseded" {
t.Errorf("the older pipeline after %s = %s; want b left it", r.name, s)
}
})
}
}
@@ -9,6 +9,7 @@ import (
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
)
// Drop asks a build seat's `drop` of a superseded step's ask as a kept call (design 47 G §31, §350; A-182): of node's holder
@@ -50,10 +51,120 @@ func (h *Hearer) Taken(ask, node string) error {
func ptrTime(t time.Time) *time.Time { return &t }
// allLeft says whether every mover of p left it for a newer pipeline (H-A-223).
func allLeft(p seat.Pipeline) bool {
if len(p.Movers) == 0 {
return false
}
for _, m := range p.Movers {
if !slices.Contains(p.Superseded, m.Module) {
return false
}
}
return true
}
// SupersedeRunning takes out of every older running pipeline of newer's branch, read by that branch's keys alone, each mover whose fingerprint newer moved
// (H-A-223): its open steps end superseded, naming newer, never failed; a step reading a test run leaves it, as a cancel's
// does; the mover is kept in the pipeline's superseded, so the end judgement (H-A-163) counts it as left and newer carries
// it. A pipeline then judged ends as at a step's end, by its other movers alone; one every mover left ends superseded.
// A run on request's pipeline is never touched. It answers the pipelines it touched.
func (h *Hearer) SupersedeRunning(newer seat.Pipeline) []string {
pkeys, err := branchKeys(h.Store, newer.Branch)
if err != nil {
h.logf("the running pipelines of %s are not listed, so none is superseded by %s: %v", newer.Branch, newer.Pipeline, err)
return nil
}
var touched []string
for _, key := range pkeys {
if key == newer.Pipeline {
continue
}
if h.supersedeRunningOne(key, newer) {
touched = append(touched, key)
}
}
return touched
}
func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
for {
p, rev, err := h.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning || p.Branch != newer.Branch || p.Run != "" || newer.Run != "" ||
p.Revision >= newer.Revision {
return false
}
var moved []string
for _, m := range p.Movers {
if slices.Contains(p.Superseded, m.Module) {
continue
}
if nm, ok := MoverOf(newer, m.Module); ok && nm.Fingerprint != m.Fingerprint {
moved = append(moved, m.Module)
}
}
if len(moved) == 0 {
return false
}
now := h.now()
why := "superseded: the newer " + newer.Pipeline + " moved its fingerprint, and carries it (H-A-223)"
var ended []int
var leave, drop []seat.StepResult
for i := range p.Steps {
st := &p.Steps[i]
if !slices.Contains(moved, st.Module) || (!open(*st) && st.Outcome != seat.StepResultOutcomeWaiting) {
continue
}
switch {
case sharedRun(*st) != "" && st.Seat != wants.Seat:
leave = append(leave, *st)
case st.Ask != "" && BoundSeat(st.Seat):
drop = append(drop, *st) // its own ask on a build node: the seat's drop cancels it
}
st.Outcome, st.Why, st.Ended = seat.StepResultOutcomeSuperseded, why, &now
p.Unlogged = append(p.Unlogged, endEntry(key, *st, now))
ended = append(ended, i)
}
p.Superseded = append(p.Superseded, moved...)
done := h.endPipeline(&p, now)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
h.logf("the running pipeline %s is not written, so its movers moved by %s stay: %v", key, newer.Pipeline, err)
return false
}
h.logf("%s: %s moved by the newer %s leave it, %d of its open steps ended superseded (H-A-223)", key, strings.Join(moved, ", "), newer.Pipeline, len(ended))
for _, i := range ended {
if err := h.say(seat.EventStepEnded, p, &p.Steps[i]); err != nil {
h.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, p.Steps[i].Module, p.Steps[i].Step, err)
}
}
for _, st := range leave {
h.leaveRun(key, st, "the mover "+st.Module+" left "+key+" for the newer "+newer.Pipeline)
}
// Its running asks on the build nodes are cancelled through the seat's drop (H-A-223). The seat mesh-builds serves no
// drop, so a build ask there runs to its end, which the runner ignores: the step is no longer open.
for _, st := range drop {
if h.DropRun == nil {
h.logf("%s: the ask %s of %s's %s is not dropped: the runner has no drop", key, st.Ask, st.Module, st.Step)
continue
}
if err := h.DropRun(st.Seat, st.Node, st.Ask, why); err != nil {
h.logf("%s: the drop of the ask %s of %s's %s is not asked of %s: %v", key, st.Ask, st.Module, st.Step, st.Seat, err)
}
}
if err := h.said(p, nil, done); err != nil {
h.logf("%v", err)
}
h.appendUnlogged(key)
return true
}
}
// supersedeWaiting takes, in each older waiting pipeline of newer's branch, the movers newer also moves out of it (design 47 H
// §748, ADR 0682 §7): a waiting pipeline left with no mover ends superseded, its waiting key deleted, saying wait-ended and its
// pipeline-ended. newer is a pipeline just written waiting; a running pipeline is never touched, it runs to its end. It
// answers the pipelines it ended.
// pipeline-ended. newer is a pipeline just written waiting; a running pipeline's movers leave it by SupersedeRunning
// (H-A-223). It answers the pipelines it ended.
func (s *Starter) supersedeWaiting(newer seat.Pipeline) ([]string, error) {
if s.Queue == nil || newer.Outcome != seat.PipelineOutcomeWaiting {
return nil, nil
+36 -10
View File
@@ -45,9 +45,14 @@ type Transfer[M any] struct {
// chunks taken per part.
Held []string `json:"held,omitempty"`
Chunks map[string]int `json:"chunks,omitempty"`
Opened time.Time `json:"opened"`
Last time.Time `json:"last"`
Shape M `json:"shape"`
// FillAtOpen is the objects' fill read from the store's figures when the transfer opened (or when chunks were dropped
// since), and TakenSince the bytes of the chunks taken after that read: the fill told per chunk is their sum, kept
// with the transfer, never read again per chunk (design 47 H lead answers H-A-227, H-A-228).
FillAtOpen uint64 `json:"fill_at_open,omitempty"`
TakenSince uint64 `json:"taken_since,omitempty"`
Opened time.Time `json:"opened"`
Last time.Time `json:"last"`
Shape M `json:"shape"`
}
// Lacks is the digests of the parts the transfer does not hold yet, sorted.
@@ -197,11 +202,12 @@ func (r *Receiver[M]) Open(id, opener string, parts []Want, shape M) (Received,
}
return Received{ID: id, Lacks: t.Lacks()}, r.arm(id)
}
t := Transfer[M]{ID: id, Opener: opener, Parts: parts, Chunks: map[string]int{}, Opened: r.now(), Last: r.now(), Shape: shape}
used, err := r.Objects.Used()
if err != nil {
return Received{}, fmt.Errorf("the objects' fill cannot be read: %w", err)
}
t := Transfer[M]{ID: id, Opener: opener, Parts: parts, Chunks: map[string]int{}, FillAtOpen: used, Opened: r.now(),
Last: r.now(), Shape: shape}
var size uint64
for _, p := range parts {
size += uint64(max(p.Size, 0))
@@ -219,7 +225,7 @@ func (r *Receiver[M]) Open(id, opener string, parts []Want, shape M) (Received,
if err := r.Transfers.Put(t); err != nil {
return Received{}, fmt.Errorf("transfer %s cannot be kept: %w", id, err)
}
r.fill()
r.tell(used)
if len(t.Lacks()) == 0 {
return Received{ID: id, Ended: true}, r.finish(t)
}
@@ -271,6 +277,7 @@ func (r *Receiver[M]) Take(id, opener, digest string, sequence int, last bool, d
// A part the holder lacks travels again whole from chunk 0 (design 51 Publishing): its earlier chunks are dropped.
r.dropChunks(t, digest)
t.Chunks[digest] = 0
r.reanchor(&t)
}
if err := r.Objects.Put(r.chunkName(id, digest, sequence), data); err != nil {
return Received{}, fmt.Errorf("the objects did not take the chunk: %w", err)
@@ -280,6 +287,7 @@ func (r *Receiver[M]) Take(id, opener, digest string, sequence int, last bool, d
t.Chunks = map[string]int{}
}
t.Chunks[digest] = sequence + 1
t.TakenSince += uint64(len(data))
t.Last = r.now()
if last {
whole, err := r.part(t, digest)
@@ -289,8 +297,9 @@ func (r *Receiver[M]) Take(id, opener, digest string, sequence int, last bool, d
if got := "sha256:" + sum(whole); got != digest {
r.dropChunks(t, digest)
delete(t.Chunks, digest)
r.reanchor(&t)
_ = r.Transfers.Put(t)
r.fill()
r.tell(t.FillAtOpen)
return Received{}, fmt.Errorf("refused: the bytes of part %s hash to %s; send the part again", digest, got)
}
t.Held = append(t.Held, digest)
@@ -298,7 +307,7 @@ func (r *Receiver[M]) Take(id, opener, digest string, sequence int, last bool, d
if err := r.Transfers.Put(t); err != nil {
return Received{}, err
}
r.fill()
r.tell(t.FillAtOpen + t.TakenSince)
if len(t.Lacks()) == 0 {
return Received{ID: id, Ended: true}, r.finish(t)
}
@@ -365,19 +374,36 @@ func (r *Receiver[M]) chunkName(id, digest string, seq int) string {
return fmt.Sprintf("%s%s/%s/%d", r.Prefix, id, strings.TrimPrefix(digest, "sha256:"), seq)
}
// fill tells Filled the objects' fill.
// fill reads the objects' fill from the store's figures and tells Filled: at a transfer's finish and drop, never per
// chunk (H-A-227).
func (r *Receiver[M]) fill() {
cap := r.CapNow()
if r.Filled == nil || cap == 0 {
if r.Filled == nil {
return
}
used, err := r.Objects.Used()
if err != nil {
return
}
r.tell(used)
}
// tell tells Filled a fill already known: read at the open, or that read plus the bytes taken since.
func (r *Receiver[M]) tell(used uint64) {
cap := r.CapNow()
if r.Filled == nil || cap == 0 {
return
}
r.Filled(used, cap, used*100 >= cap*r.fullPct())
}
// reanchor reads the fill again after a transfer's chunks were dropped, whose sizes it does not keep: the read becomes
// the transfer's FillAtOpen and nothing is taken since. A read that fails keeps the figures it had.
func (r *Receiver[M]) reanchor(t *Transfer[M]) {
if used, err := r.Objects.Used(); err == nil {
t.FillAtOpen, t.TakenSince = used, 0
}
}
// Drop drops a transfer, by hand or because it was idle: its chunks and its key.
func (r *Receiver[M]) Drop(id string) (Transfer[M], error) {
t, found, err := r.Transfers.Get(id)
@@ -29,6 +29,14 @@ func (m memObjects) Used() (uint64, error) {
return u, nil
}
// countedObjects counts the reads of the objects' fill.
type countedObjects struct {
memObjects
reads int
}
func (c *countedObjects) Used() (uint64, error) { c.reads++; return c.memObjects.Used() }
// memTransfers keeps each transfer as the bus keeps it, its JSON, so every case reads back what a key-value state
// gives (BusTransfers): an empty map kept with omitempty comes back nil.
type memTransfers[M any] map[string][]byte
@@ -244,6 +252,43 @@ func aHolderTakesAFileInParts(t *testing.T) {
}
})
// Design 47 H lead answers H-A-227, H-A-228: the fill is read from the store once when a transfer opens and kept in the
// transfer with the bytes taken since; each chunk tells that sum and reads nothing, also through a receiver built
// anew per call, as the holders build it; the finish reads it again.
t.Run("the fill is read once at the open and summed per chunk", func(t *testing.T) {
store := memObjects{"other": make([]byte, 100)}
counted := &countedObjects{memObjects: store}
transfers := memTransfers[packageShape]{}
var told []uint64
fresh := func() *Receiver[packageShape] {
return &Receiver[packageShape]{Objects: counted, Transfers: transfers, Cap: 1 << 20, MaxChunk: 100, Now: func() time.Time { return now },
Filled: func(used, _ uint64, _ bool) { told = append(told, used) },
Store: func(Transfer[packageShape], map[string][]byte) error { return nil }}
}
file := []byte(strings.Repeat("b", 1000))
w := Want{Digest: digestOf(file), Size: int64(len(file))}
if _, err := fresh().Open("t9", "builder", []Want{w}, packageShape{"m", "1"}); err != nil {
t.Fatal(err)
}
for seq := 0; seq < 9; seq++ {
if _, err := fresh().Take("t9", "builder", w.Digest, seq, false, file[seq*100:(seq+1)*100], nil); err != nil {
t.Fatal(err)
}
}
if counted.reads != 1 {
t.Errorf("the fill was read %d times across an open and 9 chunks, want once", counted.reads)
}
if want := []uint64{100, 200, 300, 400, 500, 600, 700, 800, 900, 1000}; !reflect.DeepEqual(told, want) {
t.Errorf("fills told %v, want %v (the read at the open plus the bytes taken since)", told, want)
}
if got, err := fresh().Take("t9", "builder", w.Digest, 9, true, file[900:], nil); err != nil || !got.Ended {
t.Fatalf("the last chunk = %+v, %v", got, err)
}
if counted.reads != 2 || told[len(told)-1] != 100 {
t.Errorf("after the finish: %d reads, told %d; want the store read again, 100", counted.reads, told[len(told)-1])
}
})
// An open transfer is bounded by its idle key (Arm); the key's end (Expired) drops it idle, arms it again for the
// rest when it took a chunk since, and leaves a gone one; a key not written fails the open loudly. The cap the
// binding carries (Bound) counts over Cap.