Compare commits

...
45 Commits
Author SHA1 Message Date
mesh-tools e196064e51 Merge pull request 'Design 47 H: the builder's publish queue: live asks first, newest per series, one listing per pass (H-A-290..293)' (#316) from fix/47-H1-builder-publish-queue into main 2026-11-21 18:00:25 +00:00
jschoubben 2eaa41a420 H-A-292, H-A-293: builder_publish_queue reads one page; a build is retired only within its series
The review found two faults. The tool paged on through a state of any size, one replay of the bucket per page; it now
reads one page of each of its own states and, when the state holds more, answers that page capped, with the keys read
and the state's size from its info, never asking a second page and never refusing. And the retiring of superseded done
builds keyed the newest reservation and release by module alone, so a standing 0.2 would retire a done 0.1.x; both are
now found per module and series (MAJOR.MINOR).

Builds: design 47 H H-A-292, H-A-293, H-A-291 (1), H-A-290 (5)
2026-11-21 18:59:58 +01:00
jschoubben 90ff867e7a lock the test fakes' maps
Two tests failed under -race on main too: the re-ask state fake's maps were written by a try on the watch's goroutine while
the test read and deleted them, and the operator-ask test read the build store unlocked and waited on the builder while
the build its answer starts could still be counting itself in. The fake's maps are locked and read through it, the store
is read under its lock, and the builder is waited on only once that build has claimed its key.

Builds: design 47 H H-A-290 (the module's tests pass with -race, as its checks ask)
2026-11-21 18:59:58 +01:00
jschoubben 77fb6c4ab1 H-A-291: a start publishes only each module's newest done build; a live enqueue cuts a start list short
A start asked a publish again for every unreleased done build, older ones whose digests a newer reservation or release
of the same module made dead included, and a live publish queued during a long start pass waited for its end. A done
build a newer reservation or release (the builder's own record) supersedes is now retired at the start and at the pass:
its key dropped, its publish item ended, said in the journal; the start asks only each module's newest. A live enqueue
marks its holder, and a pass about to send a start-class item stops there, lets the sends in flight end and lists again,
so the live one goes next; a start-class enqueue never does.

Builds: design 47 H H-A-291 (1), (2), H-A-290, H-A-275
2026-11-21 18:59:58 +01:00
jschoubben bda61ae64c H-A-290: the builder's publish queue sends live publishes first, in passes, never a withdrawn build
The tool runner's release waited forty minutes behind old versions: every start asked a publish again for every done
build, and each send relisted and reread the whole queue. A publish whose build ended in this run now goes before any
asked again at a start (the class kept on the key's value, a live key of a build before the start made a start key at
the start); a pass lists a holder's keys once and sends them four at a time; a withdrawn or taken-over build is dropped
with its publish item ended at enqueue, at the pass and at the start; a send that returns unsent says why, and every call
it makes honours its one bound; builder_publish_queue reads its states in pages and answers counts per holder and class,
the oldest and newest queue time, a list capped at fifty and one module's items with their class.

Builds: design 47 H H-A-290 (1)-(5), H-A-269, H-A-271, H-A-275
2026-11-21 18:59:58 +01:00
mesh-tools 5547fd028e Merge pull request 'Design 47 H: a pipeline with nothing open ends at once; the end's reads are made once (H-A-288, H-A-289)' (#315) from fix/47-H1-runner-end-when-empty into main 2026-11-21 17:22:44 +00:00
jschoubben d51d389c1b Runner: a pipeline with nothing open ends at once (47-H1-runner-end-when-empty)
main.117 judged its end at 17:45:19 and never wrote it: every mover's widens
was read inside the end's compare-and-set loop, 126 movers each cut on
grants-for, so each try took about eleven minutes and every write was refused
as changed until the restart, whose start pass judged no end.

The reads an end needs are made once, before the compare-and-set, shared by
every mover and every retry; the runner's start pass ends each running
pipeline with nothing open before the queue is placed; every end says its
outcome in the journal.

Builds: design 47 H H-A-288, H-A-289, H-A-223, H-A-243
2026-11-21 18:18:32 +01:00
mesh-tools c6edd5987f Merge pull request 'Design 47 H: a re-pin of the same source keeps a mover in its pipeline (H-A-283..286)' (#314) from fix/47-H1-runner-same-source-repin into main 2026-11-21 17:00:58 +00:00
jschoubben 9a49b0f4f7 Give a kept mover's asks ids of its kept revision and place it from that revision (H-A-286)
Review round 1: a kept mover's ask ids still came from the pipeline's start
time, so a pin-held ask or a composition took the id of an ask the runner
had just dropped for the old fingerprint, and a restart never matched the
steps placed again; and a kept mover placed later read the branch's held
view, so a lagging view gave the new fingerprint the old pins. Every ask id
of a kept mover's steps (build, tests, validate, pin-held asks,
composition, rerun, restart) now mixes its kept revision into the id's
sum, and every placing of a kept mover (at once, later, at a restart, a
rerun) reads its entries at its kept revision by number, waiting while
that revision cannot be read. The test's starter gets a clock that moves.

Builds: design 47 H H-A-286 (amending H-A-284; with H-A-283, H-A-285)
2026-11-21 17:58:07 +01:00
jschoubben 47f81051be Keep a same-source re-pin's mover in its pipeline and build it again there (H-A-283)
A module importing packages its own pipeline releases was re-pinned at
each publish; each re-pin revision of the same source took it out of the
running pipeline, and each newer build withdrew the older reservation, so
nothing ever published. A revision of the same source now moves no mover
out of the open pipeline of that source: the pipeline keeps it at its new
fingerprint, ends every step of it superseded (said at the old
fingerprint, its asks dropped, its runs left) and places them again from
the re-pin revision's entry under new asks, the build first, still naming
the pipeline's revision. The revision the mover is kept at rides in the
record's kept-revisions, beside the movers, where the pin check reads its
entry. A revision whose every mover is kept starts no pipeline; one of a
new source moves movers as before.

Builds: design 47 H H-A-283, H-A-284, H-A-285 (with H-A-223, H-A-244, ADR 0682)
2026-11-21 17:47:54 +01:00
mesh-tools 769d701e2b Merge pull request 'Design 47 H1: a published event reads each reservation once, only the unbuilt ones on a miss (H-A-280)' (#313) from fix/47-H1-builder-published-read-once into main 2026-11-21 15:47:49 +00:00
jschoubben ece9d6b18f Builder: a published reads each reservation once, and only the unbuilt ones on a miss
Live, a published at a version fourteen reservations share cost three seconds:
the fallback by version read every reservation there, and the journal line
read them all again, so the backlog drained at fifty a minute.

The published now hands the reservations it read (each once, a second pass
taking what the first read) to its journal line. The fallback reads the new
index key of the reservations at a version whose build has not recorded every
digest (kept at reserve and where a digest is recorded, written again at the
start), the only ones a published can be kept on; the line counts the rest from
the index by version without reading them. Before that pass the handler says
in progress to the bus.

Builds: design 47 H H-A-280, H-A-271
2026-11-21 16:45:25 +01:00
mesh-tools 93495203b5 Merge pull request 'Design 47 H1: the builder hears a published once by digest; no publish asked again while one is unheard; a held number is never reserved (H-A-269)' (#312) from fix/47-H1-builder-published-once into main 2026-11-21 14:59:17 +00:00
jschoubben 5628ca2919 Builder: a build whose number a registry holds elsewhere is built again, never moved
Every artifact carries its version from its build (a Go module zip's paths, a
program's stamped version, an image's and a bundle's build environment), so
no digest survives a change of number: the move that kept digests would have
published a binary saying 0.1.37 under another number.

The try, on a number a registry holds at another digest, withdraws it on the
reservation (a gap) and raises version-held-elsewhere naming both digests,
then asks the build of the same fingerprint again through the build-again
path of a done key whose version is not released. Its reserve never takes a
withdrawn number again, skips every number a registry holds, and the build
records fresh digests and runs its own publish; the takeover ends the old
number's publish item and bound.

Builds: design 47 H H-A-275, H-A-279, H-A-226
2026-11-21 15:56:04 +01:00
jschoubben 05c59ae000 Builder: the registry's record ends a publish or moves a number it holds elsewhere
A publish item skipped while the registry held the version would stay skipped
for ever once its published event was lost; and node-tools 0.1.37, held by the
artifact registry at another digest than the build's, was asked again for ever
and never released.

When every missing artifact is held at its version with the built digest the
builder marks it published from the registry's record and says
package-published, as a heard published would; the release follows from the
catalog. Before reserving, the registry seats' versions of the module's
artifacts are read and every number one holds is skipped (version-held-elsewhere,
scope module); a read that fails lets the reserve proceed. A done build whose
number a registry holds at another digest moves to the next free number, its
digests kept, the old number withdrawn as a gap; its old item and bound end and
its publish is asked under the new number. The finding clears at the module's
version-released.

Builds: design 47 H H-A-274, H-A-275, H-A-245, H-A-226
2026-11-21 15:51:21 +01:00
jschoubben e1630b57c6 Builder: no publish asked again while its published may be unheard or the registry holds it
The publish item's try asked the registry to publish node-tools 0.1.37 again
every few minutes while the registry had published it and the builder's own
consumer stood thousands of messages behind, adding events to the backlog.
Before each try the builder reads its durable on the events stream (pending
plus unacknowledged): at a hundred or more the try asks nothing. Below it the
registry seats' own record (their verb versions) is read: every missing
artifact held at the version with its built digest asks nothing either. A
skipped try is said in the journal, written again with its pause and spends
no try; a count or record that cannot be read holds no ask.

Builds: design 47 H H-A-269 (2), H-A-272
2026-11-21 15:40:19 +01:00
jschoubben 5c4c4bc821 Builder: a published finds its reservation through an index by digest and by version
The builder's consumer of the registry seats' published stood thousands behind:
each published, and each one that marked nothing, read every reservation.
An index beside the reservations (three-token keys in the same state, which the
reservations' two-token filter never lists) names the reservation keys holding
a built digest and those standing at a version; it is kept at reserve, at a
digest recorded, at release and at drop, and written again once at the start.
A published reads its digest's entry and that reservation; a digest no
reservation holds is judged from its version's entry, never a scan.

The test stores and the tools' read store now read a key filter token by token
as the bus does, which showed the publish queue's resume listing three-token
keys with the two-token filter: it lists them with its own filter now.

Builds: design 47 H H-A-269 (1), H-A-271
2026-11-21 15:37:44 +01:00
mesh-tools 599bf62d23 Merge pull request 'Design 47 H1: a module entering the trunk's graph is registered at its take-in; the runner says registration-missing (H-A-262)' (#311) from fix/47-H1-indexer-register-at-take-in into main 2026-11-21 12:22:04 +00:00
jschoubben 0de195c5aa Design 47 H1: test that a module entering the graph watches its siblings' alike branch
Review round 2. The siblings watched main, which is also the default
branch, so no test held the alike rule; they now watch stable and the
entering module is asserted to watch it.

Builds: design 47 H H-A-265, H-A-266
2026-11-21 13:20:54 +01:00
jschoubben 3160bb1fb8 Design 47 H1: registration-missing clears when no running pipeline waits on the registration
Review round 1. A pipeline that ended (failed, cancelled) with a mover
still waiting left the finding for ever. It is now judged at every
pipeline's end and in Waiting: cleared when the registration is held, or
when no running pipeline still has an unplaced mover of the module.

Builds: design 47 H H-A-262, H-A-265
2026-11-21 13:18:41 +01:00
jschoubben 93efda5db3 Design 47 H1: the graph skips a pending register, the seed's left-out, and reads the branch
Review round 1. A module with a register proposal pending is skipped (the
operator's Approve registers it); a module the seed left out keeps
seed-left-out and is not registered; the branch is the first watched
branch the repository's registered modules hold alike, else its default
branch read through the seat git, and a module whose default branch is not
read is skipped and said, registered at the next kept trunk revision. One
listing of the registrations finds the modules with none, so a revision
whose modules are all registered reads no registration.

Builds: design 47 H H-A-262, H-A-263, H-A-264, H-A-265, H-A-266
2026-11-21 13:18:41 +01:00
jschoubben f01731fd4e Design 47 H1: the runner raises registration-missing while a mover waits on its registration
A mover with no registration had its wait said only in the journal. The
runner now raises the finding registration-missing (scope module, id the
module) through its finding writer as it places a pipeline whose mover has
no registration, and clears it when the module's registration arrives or
the registrations held name it.

Builds: design 47 H H-A-262, G A-327, ADR 0666 §2
2026-11-21 13:09:23 +01:00
jschoubben 251745f859 Design 47 H1: the indexer registers a module that enters the trunk's module graph
Four modules added to a repository after the registrations' seed were never
registered, so a pipeline's steps for them waited on a registration nobody
asked. A kept trunk revision of any kind now registers each of its modules
with no registration, before the revision is said, with no ask: origin its
repository, path its manifest's, the first watched branch, the steps its
repository's other modules hold alike (else the seed's) and a per-artifact
status only where every sibling naming that artifact holds it identically,
said registration-changed. The start does the same once for the current
trunk graph. With no registration held the seed stays the only writer; a
module that leaves the graph keeps its registration.

Builds: design 47 H H-A-262, H-A-263, H-A-264, G A-327, ADR 0666 §2
2026-11-21 13:09:23 +01:00
mesh-tools d1ec9d9072 Merge pull request 'Design 47 H1: the assigner's pass never waits on the keeping of revisions; no want before a revision is kept (H-A-259)' (#310) from fix/47-H1-assigner-pass-on-ended into main 2026-11-21 11:40:59 +00:00
jschoubben 14254dd0ba Assigner: no want before a revision is kept; the full-store verdict stays the pass's
Review round 1: the start handed a removal-only keeping to the background
while passes wrote wants with no revision kept, so their wanted was never
said. A pass now writes no want until the first keeping keeps a revision
(that keeping wakes the pass), the start's removal folds into the first
pass's keeping, and the pass judges the store's fill itself, through a
handle of its own, before it writes. The integration bed makes the
revisions' object store, as the grants part does from the manifest.

Builds: design 47 H H-A-259, H-A-261; ADR 0682
2026-11-21 12:40:15 +01:00
jschoubben 1e50642e3c Assigner: the pass never waits on the keeping of revisions (H-A-259)
A pass that kept a fresh assignment graph revision sat six to eight minutes
keeping it and removing the revisions not current, so the wakes of
pipeline-ended and running waited for it. The keeping now runs off the pass
loop, one at a time and folded; the removal takes each entry's module and
fingerprint from the store's listing instead of reading every entry object;
each pass and each keeping says its duration.

Builds: design 47 H H-A-259; ADR 0682
2026-11-21 12:40:15 +01:00
mesh-tools b0082ec04f Merge pull request 'Design 47 H1: every ask the runner makes is bounded; a shared validate ask is never read as one mover's; a moved mover leaves a waiting pipeline (H-A-235)' (#309) from fix/47-H1-runner-asks-bounded into main 2026-11-21 11:02:21 +00:00
jschoubben 6b6e4c81f0 Judge a missed first bound by the first read's own entries (H-A-238)
The start read the second bound with a get, which says not found both
for a key never armed and for one the holder's ask-taken deleted, so
every later start asked an answered step once more. The first read
keeps each key's last entry; a first bound's expiry is taken only when
its second bound left no entry at all, and with both passed only the
second is taken, ending the step without asking it again.

Builds: design 47 H H-A-238 (5)
2026-11-21 12:02:02 +01:00
jschoubben 374d060cc0 Take an ask's bound that expired while the runner was down (H-A-238)
The bounds were watched for updates only, so an expiry marker written
while the runner was down was never heard and that ask had no bound.
At the start each ask key's last entry is read; an expiry marker is
taken then. A running heard before the seat-holders view is read is
heard again after a delay instead of at once.

Builds: design 47 H H-A-238 (5), H-A-243 (1)
2026-11-21 12:02:02 +01:00
jschoubben 1f878eec58 A validate rerun moves every mover's validate and reopens its compositions (H-A-236)
The rerun put only the named mover's validate under the new ask; the
others stayed failed under the old one, so the pipeline failed again
and a second rerun made a second validate ask, breaking the pipeline's
one validate ask. The compositions its failed validate ended not asked
now wait again, asked once validate passes as at the placing.

Builds: design 47 H H-A-51, H-A-236
2026-11-21 12:02:02 +01:00
jschoubben c9515da86f Take a validate end by its ask alone, ending every mover's validate (H-A-252)
The pipeline's one validate ask names no mover, but its end names the
mover whose body the holder took; the runner compared that mover with
the first step carrying the ask and dropped a passed validate, so the
pipeline held its slot (main.88).

Builds: design 47 H H-A-236, H-A-252
2026-11-21 12:02:02 +01:00
jschoubben 5f7137e676 A moved mover leaves an older waiting pipeline whenever a newer one starts (H-A-244)
Only a newer pipeline that waited took movers out of older waiting ones;
one placed at once left them, so main.84 was placed nine hours later
with a mover a newer revision had moved and built a stale fingerprint.

Builds: design 47 H H-A-223, H-A-244 (1)
2026-11-21 12:02:02 +01:00
jschoubben 902947a044 Bound a silent seat's ask once; put it again when its holder starts (H-A-243)
mesh-builds and mesh-wants say nothing before a step's end, so asking
them again at the short bound would double a running build. Their ask
has one bound, ask-silent-bound, and is put again when the node-engine
says the holder running, since the holder takes asks on a plain
subscription and loses those put while it is down. An ask asked again
on a build seat carries a second id derived in the accepted form.

Builds: design 47 H H-A-235, H-A-238, H-A-241, H-A-243
2026-11-21 12:02:02 +01:00
jschoubben b95b3ef989 Bound every ask the runner puts on a seat (H-A-235)
An ask whose holder lost it (the seat mesh-wants' holder restarted) held
its pipeline's turn for nine hours. Every ask but a test run's now arms
a key of the state bounds before it is put; its expiry asks once more,
said in the journal, and the second ends the step could not run with
the finding call-unanswered, so the pipeline ends and its turn frees.

Builds: design 47 H H-A-235, H-A-238 (1)-(5)
2026-11-21 12:02:02 +01:00
jschoubben 0f167db73d Rerun of validate or composition asks the shared ask (H-A-236)
The rerun verb looked a step's ask up by the mover's id, but the validate
ask is one per pipeline and names no mover, so every validate rerun, and
a composition rerun, was refused "placed no ask now".

Builds: design 47 H H-A-236
2026-11-21 12:02:02 +01:00
mesh-tools e5f54c5cdb Merge pull request 'Design 47 H1: the indexer declares the finding kinds it raises; its unanswered calls get a kind of its own (H-A-247)' (#308) from fix/47-H1-indexer-finding-kinds into main 2026-11-21 10:57:52 +00:00
jschoubben bdb4edd820 Design 47 H1: the indexer's retired call-unanswered names scope call
The tool runner's approval findings of call-unanswered are of scope machine;
the indexer's entry now names scope call, so no approval key is cleared even
where an id form would match. The test's negative ids take the approval form.

Builds: design 47 H H-A-249, G §580
2026-11-21 11:50:27 +01:00
jschoubben 31c2c5c7fe Design 47 H1: the indexer retires call-unanswered by the ids it wrote under it
The daemon's call-unanswered keys from before the rename would be seeded and
stated by the tool runner's writer, which owns that kind, and never cleared.
The retired list now names call-unanswered with the indexer's own id forms
(the seed's node-build-<kind>.steps, the floor's
mesh-catalog.versions-floor.<module>, the versions fill's
mesh-delivery-builder.builder_versions), so those keys clear once and the
tool runner's approval findings stay.

Builds: design 47 H H-A-247 (3), H-A-249, G §580
2026-11-21 11:50:27 +01:00
jschoubben ad0052c6f5 Design 47 H1: the indexer owns the SDK parts' kinds, its own unanswered-call kind and a retired list
The daemon's writer is handed to the process reader (event-unhandled) and the
heard writer (family-not-heard), kinds its list lacked; and call-unanswered
belongs to the tool runner's writer over the same state. The daemon's
unanswered calls are now indexer-call-unanswered (same scope and id), its list
holds the SDK parts' kinds, and merge-untaken is retired, so its live key and
condition clear at the next start. The test reads the SDK parts handed the
writer and refuses a retired kind still raised.

Builds: design 47 H H-A-242, H-A-247 (1) (2) (3), H-A-82, G §580
2026-11-21 11:50:27 +01:00
jschoubben aef8632d95 Design 47 H1: the indexer's writer owns the kinds its code raises, held by a test
The daemon's writer owned every kind while the tool runner's writer shares the
state `findings`; under G §580 each bundle states only its own kinds. The
daemon's writer is now made with FindingKinds, the ten kinds its code raises,
and a unit test reads the daemon's code for every kind it gives the writer and
fails on one the list does not declare.

Builds: design 47 H H-A-240, H-A-242 (2), H-A-233, G §580
2026-11-21 11:50:27 +01:00
mesh-tools dedcc3ead7 Merge pull request 'Design 47 H1: a done build whose series moved says stale, not 'no version is reserved' (H-A-244)' (#307) from fix/47-H1-builder-stale-and-tag into main 2026-11-21 10:37:56 +00:00
jschoubben a40bbec44a Builder: a done build whose series was taken over fails stale, naming the newer build
A pipeline placed long after its take-in found its old build done but holding no
number, and was told to reserve one; the series had moved to a newer fingerprint.
The step now says it is stale and names that fingerprint and its version.

Builds: design 47 H H-A-244 (2), H-A-226
2026-11-21 11:34:25 +01:00
mesh-tools fe6c5961ae Merge pull request 'Design 47 H1: a repository's packages are those in the merge's tree (H-A-232)' (#306) from fix/47-H1-indexer-gone-packages into main 2026-11-21 02:09:12 +00:00
jschoubben 588a6ff8d5 Design 47 H1: only the trunk's merge drops a gone package; a commit's tree tells an all-gone repository (H-A-232, H-A-233)
Review round 1. A branch's take-in runs through the dry indexer, which
shares the live packages state, so a branch cut before main added a
module would have deleted main's key: publish drops a kept key only on
the real indexer at the trunk's merge. And a repository whose every
asked directory is gone (one key, or all gone) returned the whole error
and raised packages-not-published at every take-in: the seat's trees of
the commit now tell a gone directory from a commit with no tree.

Builds: design 47 H H-A-232, H-A-233, H-A-234; issue 1585
2026-11-21 03:07:02 +01:00
jschoubben ced27928d9 Design 47 H1: the indexer publishes a repository's packages in the merge's tree (H-A-232, H-A-233, H-A-234)
A Go module mesh-catalog removed stayed in the packages state, so every
publish of mesh-catalog asked the forge for its directory, the seat git
answered not-found, and the one-directory fallback (issue 1585) ran only
on an answer without the directory: nothing of the repository was
published and its packages' digests froze unsaid.

filesVia takes the seat's not-found as a failure of the tree, so the
fallback runs; publish drops the kept key of a package whose directory
the trunk's merge no longer holds, said in the journal; a listing that
fails for another reason raises the finding packages-not-published
(scope merge, id the repository), cleared at the next publish that lists.

Builds: design 47 H H-A-232, H-A-233, H-A-234; issue 1585
2026-11-21 02:54:58 +01:00
87 changed files with 6164 additions and 488 deletions
@@ -15,7 +15,21 @@ import (
// fillOf is the store's fill now: its bytes and cap, when the bucket says them and has a cap.
func (g *graphKeeper) fillOf(ctx context.Context) (used, capacity int64, ok bool) {
f, is := g.objects.(assigngraph.Filler)
return fillIn(ctx, g.objects)
}
// passObjects is the store as the pass reads its fill: a handle of its own where one is given, so the pass's read never shares one
// with a keeping that runs beside it (design 47 H-A-261).
func (g *graphKeeper) passObjects() assigngraph.Objects {
if g.reads != nil {
return g.reads
}
return g.objects
}
// fillIn is the fill of the bucket o, when it says one and has a cap.
func fillIn(ctx context.Context, o assigngraph.Objects) (used, capacity int64, ok bool) {
f, is := o.(assigngraph.Filler)
if !is {
return 0, 0, false
}
@@ -35,13 +49,13 @@ func (p *passer) sayFill(event, about string, used, capacity int64, refused stri
// its cap), so a fill reached while the process was down is said too.
func (p *passer) fillAtStart(ctx context.Context) {
g := p.graph
used, capacity, ok := g.fillOf(ctx)
used, capacity, ok := fillIn(ctx, g.passObjects())
if !ok {
return
}
switch {
case used >= capacity:
g.full = true
g.setFull(true)
p.sayFill(meshwants.EventFull, "start", used, capacity, "the next revision")
case used*5 >= capacity*4:
p.sayFill(meshwants.EventFilling, "start", used, capacity, "")
@@ -52,20 +66,20 @@ func (p *passer) fillAtStart(ctx context.Context) {
// 80 % again. It answers true while the store is full.
func (p *passer) fullBefore(ctx context.Context, c assigngraph.Computed) bool {
g := p.graph
used, capacity, ok := g.fillOf(ctx)
used, capacity, ok := fillIn(ctx, g.passObjects())
if !ok {
return false
}
if g.full && used*5 < capacity*4 {
g.full = false
if g.isFull() && used*5 < capacity*4 {
g.setFull(false)
p.sayFill(meshwants.EventRoom, fmt.Sprint(used), used, capacity, "")
}
if used+assigngraph.NewBytes(ctx, g.objects, c) <= capacity {
g.full = false
if used+assigngraph.NewBytes(ctx, g.passObjects(), c) <= capacity {
g.setFull(false)
return false
}
if !g.full {
g.full = true
if !g.isFull() {
g.setFull(true)
p.sayFill(meshwants.EventFull, fmt.Sprint(used), used, capacity, "the next revision")
}
p.logf("the assignment graph revision is not kept: %s is full (%d of %d bytes), the last revision stays and no want is written until there is room", assigngraph.RevisionsObjects, used, capacity)
@@ -61,14 +61,14 @@ func TestTheAssignmentGraphStoreSaysFillingFullAndRoom(t *testing.T) {
if len(events) != 0 {
t.Fatalf("events %v", events)
}
o.base = 700
o.base, o.cap = 8500, 10000 // past 80 %, with room for the next revision, which the pass judges itself (H-A-261)
p.graph.started = false
p.keepGraph(ctx)
p.keepGraph(ctx) // a start again: its first pass judges the fill
if len(events) != 1 || events[0] != meshwants.EventFilling {
t.Fatalf("at the start past 80 %%: %v", events)
}
c := assigngraph.Computed{Revision: meshwants.AssignmentRevision{Entries: []meshwants.NodeEntryRef{}}}
o.base = 990
o.base, o.cap = 990, 1000
if !p.fullBefore(ctx, c) || !p.graph.full || events[len(events)-1] != meshwants.EventFull {
t.Fatalf("a revision that does not fit: events %v", events)
}
@@ -76,8 +76,15 @@ func TestTheAssignmentGraphStoreSaysFillingFullAndRoom(t *testing.T) {
if n := len(events); n != 2 {
t.Errorf("full is said once: %v", events)
}
// H-A-261: the verdict is the pass's own: with the store full the pass writes no want, though a revision is kept.
if held := p.keepGraph(ctx); held != heldFull {
t.Errorf("a pass with the store full answers %q, want %q", held, heldFull)
}
o.base = 100
if p.fullBefore(ctx, c) || p.graph.full || events[len(events)-1] != meshwants.EventRoom {
t.Errorf("room: events %v", events)
}
if held := p.keepGraph(ctx); held != "" {
t.Errorf("a pass with room and a revision kept answers %q, want the wants written", held)
}
}
@@ -8,6 +8,7 @@ import (
"fmt"
"slices"
"strings"
"sync"
"time"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
@@ -24,11 +25,18 @@ import (
// that follows are written from the revision it holds.
type graphKeeper struct {
objects assigngraph.Objects
// reads is the same store through a handle of the pass's own, for the pass's read of its fill (graphfill.go); nil reads objects.
reads assigngraph.Objects
// halves is every followed branch's catalog graph revision held now.
halves func() map[string]mcc.Half
// last is the revision last computed, so a pass whose inputs did not change keeps and says nothing.
last meshwants.AssignmentRevision
has bool
// mu guards what the keeping writes and the pass reads: the revision kept, full, and the need handed over (design 47 H-A-259).
mu sync.Mutex
// handedRev is the revision last handed to the keeping, so a pass whose inputs did not change hands nothing; handed is false
// until one is, and again when its keeping failed or found the store full, so the next pass hands it again.
handedRev meshwants.AssignmentRevision
handed bool
// handedOnce says a revision was handed: the first carries the start's removal (H-A-261).
handedOnce bool
// kept is the revision the wants follow, and kept's source the trunk's source revision it carries; ok is false until one is kept.
kept meshwants.AssignmentRevision
source int64
@@ -40,13 +48,55 @@ type graphKeeper struct {
revisions *revisionRemover
// computed is the revision computed by the last pass, with its entries: its edges.
computed assigngraph.Computed
// The keeping off the pass loop (H-A-259): need is what waits to be kept, keeping says one runs; background runs it (nil: at
// once, in the caller), and wake makes a pass once a keeping changed what the pass reads.
need *keepNeed
keeping bool
background func(func())
wake func()
}
// keepGraph computes the revision of this pass and keeps it when the pass's inputs differ from the last computed.
func (p *passer) keepGraph(ctx context.Context) {
// keepNeed is one need of the keeping: a computed revision to keep (nil: none), and whether the revisions not current are judged.
type keepNeed struct {
c *assigngraph.Computed
judge bool
}
// keptNow is the revision the wants follow, with its trunk source, and whether one is kept.
func (g *graphKeeper) keptNow() (meshwants.AssignmentRevision, int64, bool) {
g.mu.Lock()
defer g.mu.Unlock()
return g.kept, g.source, g.ok
}
// isFull says the store did not hold the last revision handed over, so no want is written (design 47 G §181).
func (g *graphKeeper) isFull() bool {
g.mu.Lock()
defer g.mu.Unlock()
return g.full
}
func (g *graphKeeper) setFull(full bool) {
g.mu.Lock()
g.full = full
g.mu.Unlock()
}
// heldNoRevision is the pass's words while no assignment graph revision is kept: it writes no want (design 47 H-A-261).
const heldNoRevision = "held: no assignment graph revision is kept yet, so no want is written until the first keeping ends (H-A-261)"
// heldFull is the pass's words while the next revision does not fit its store (design 47 G §181).
const heldFull = "the assignment graph store is full: no want is written (design 47 G §181)"
// keepGraph computes the revision of this pass, judges the store's fill itself, and hands the revision to the keeping when the
// pass's inputs differ from the last handed. The keeping and the removing of revisions run off the pass loop, one at a time, and the
// pass never waits on them: it judges and writes the wants from the graph it holds (design 47 H-A-259). It answers why the pass
// writes no want, or "" when it writes them: none while the next revision does not fit the store (G §181), and none while no
// revision is kept (H-A-261), whose keeping wakes a pass when it ends.
func (p *passer) keepGraph(ctx context.Context) string {
g := p.graph
if g == nil || g.objects == nil || g.halves == nil {
return
return ""
}
c := assigngraph.Compute(p.graphInputs())
g.computed = c
@@ -56,40 +106,130 @@ func (p *passer) keepGraph(ctx context.Context) {
p.logf("the assignment graph leaves out %s", u)
}
}
if !g.started {
if !g.started { // the first pass judges the fill once (graphfill.go)
g.started = true
p.fillAtStart(ctx)
}
if g.has && g.ok && assigngraph.Same(g.last, c.Revision) {
if p.fullBefore(ctx, c) {
g.mu.Lock()
g.handed = false // handed once it fits
g.mu.Unlock()
return heldFull
}
g.mu.Lock()
same := g.handed && assigngraph.Same(g.handedRev, c.Revision)
first := !g.handedOnce
if !same {
g.handedRev, g.handed, g.handedOnce = c.Revision, true, true
}
g.mu.Unlock()
if !same {
// The start's removal of the revisions not current folds into the first keeping: it keeps the first pass's revision, then
// removes (H-A-261).
p.needKeeping(ctx, keepNeed{c: &c, judge: first})
}
if _, _, ok := g.keptNow(); !ok {
return heldNoRevision
}
return ""
}
// needKeeping hands a need to the keeping: one keeping runs at a time, and a need that comes while one runs is folded into the one
// that waits (the newest revision, and a judging asked by either), so it runs once more (design 47 H-A-259).
func (p *passer) needKeeping(ctx context.Context, n keepNeed) {
g := p.graph
g.mu.Lock()
if w := g.need; w != nil {
n.judge = n.judge || w.judge
if n.c == nil {
n.c = w.c
}
}
g.need = &n
if g.keeping {
g.mu.Unlock()
return
}
g.keeping = true
g.mu.Unlock()
run := func() {
for {
g.mu.Lock()
next := g.need
g.need = nil
if next == nil {
g.keeping = false
g.mu.Unlock()
return
}
g.mu.Unlock()
p.keepOne(ctx, *next)
}
}
if g.background != nil {
g.background(run)
return
}
run()
}
// keepOne keeps one computed revision, says assignment-revision for a revision kept now and removes the
// revisions that ceased to be current; it says its duration in the journal, and wakes a pass when it changed what the pass reads.
func (p *passer) keepOne(ctx context.Context, n keepNeed) {
g := p.graph
start := time.Now()
changed := false
defer func() {
p.logf("the keeping of the assignment graph took %s", time.Since(start).Round(time.Millisecond))
if changed && g.background != nil && g.wake != nil {
g.wake()
}
}()
if n.c != nil {
_, _, had := g.keptNow()
fresh := p.keepRevision(ctx, *n.c)
_, _, has := g.keptNow()
n.judge = n.judge || fresh
changed = fresh || had != has // the first revision kept: the pass writes the wants it held (H-A-261)
}
if n.judge && g.revisions != nil {
g.revisions.judge() // the revision before ceased to be current
}
}
// keepRevision keeps c when the store holds it, and says whether a revision was kept now.
func (p *passer) keepRevision(ctx context.Context, c assigngraph.Computed) bool {
g := p.graph
again := func() { // the next pass hands this revision again
g.mu.Lock()
g.handed = false
g.mu.Unlock()
}
used, _, _ := g.fillOf(ctx)
if p.fullBefore(ctx, c) {
return
}
rev, fresh, err := assigngraph.Keep(ctx, g.objects, c, meshwants.EventAssignmentRevision, p.clock())
if err != nil {
p.logf("the assignment graph revision is not kept, so no want is said as wanted until it is: %v", err)
return
again()
return false
}
g.last, g.has, g.kept, g.ok = c.Revision, true, rev, true
g.source = 0
var source int64
for _, cat := range rev.Catalogs {
if cat.Branch == p.w.trunkName() {
g.source = cat.Source
source = cat.Source
}
}
g.mu.Lock()
g.kept, g.ok, g.source = rev, true, source
g.mu.Unlock()
if !fresh {
return
return false
}
p.fillingAfter(ctx, used)
if g.revisions != nil {
g.revisions.judge() // the revision before ceased to be current
}
id := fmt.Sprintf("%s-%s-%d", meshwants.EventAssignmentRevision, rev.Kind, rev.Revision)
if err := p.wantsSay(meshwants.EventAssignmentRevision, assigngraph.Event(rev), id); err != nil {
p.logf("the assignment revision %d is kept but not said: %v", rev.Revision, err)
}
return true
}
// graphInputs are what the pass's assignment graph is computed from: every followed branch's catalog half, the placements, each
@@ -147,14 +287,20 @@ func (p *passer) sayWanted(node, module, fp, pipeline string, removed bool) {
return
}
g := p.graph
if g == nil || !g.ok {
var kept meshwants.AssignmentRevision
var source int64
ok := false
if g != nil {
kept, source, ok = g.keptNow()
}
if !ok {
p.logf("the want of %s on %s is written, but no assignment graph revision is kept, so wanted is not said", module, node)
return
}
at := p.clock().UTC()
body := meshwants.WantedEvent{V: keys.Version, Node: node, Module: module, Fp: fp, Pipeline: pipeline, Removed: removed, Revision: g.kept.Revision, Source: g.source, At: &at}
body := meshwants.WantedEvent{V: keys.Version, Node: node, Module: module, Fp: fp, Pipeline: pipeline, Removed: removed, Revision: kept.Revision, Source: source, At: &at}
event := strings.TrimPrefix(meshwants.SubjectWanted(keys.EscapeName(node), keys.EscapeName(module)), "mesh.seat."+meshwants.Seat+".event.")
id := fmt.Sprintf("wanted.%s.%s.%s.%t.%d.%s", keys.EscapeName(node), keys.EscapeName(module), fp, removed, g.kept.Revision, at.Format(time.RFC3339Nano))
id := fmt.Sprintf("wanted.%s.%s.%s.%t.%d.%s", keys.EscapeName(node), keys.EscapeName(module), fp, removed, kept.Revision, at.Format(time.RFC3339Nano))
if err := p.wantsSay(event, body, id); err != nil {
p.logf("wanted of %s on %s is not said: %v", module, node, err)
}
@@ -185,7 +331,7 @@ func (p *passer) namedBuild(module, trunkFP string, unreleased ...bool) func(fp
// the mesh-sdk merge moved its fingerprint).
return p.w.released.withParts(module, p.w.ends.madeOf(module, fp))
}
if p.graph == nil || !p.graph.ok {
if _, _, kept := p.graphKept(); !kept {
p.unheardRelease(module, fp, "no assignment graph revision is kept yet, so no release is looked up")
return assign.KeptBuild{}, false
}
@@ -212,3 +358,11 @@ func (p *passer) unheardRelease(module, fp, why string) {
p.logf("the release of %s at fingerprint %s is not heard: %s; its wants and candidates carry no version and no parts", module, fp, why)
}
}
// graphKept is the revision the wants follow, with its trunk source, and whether one is kept; with no graph kept, none is.
func (p *passer) graphKept() (meshwants.AssignmentRevision, int64, bool) {
if p.graph == nil {
return meshwants.AssignmentRevision{}, 0, false
}
return p.graph.keptNow()
}
@@ -3,8 +3,10 @@ package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"sync"
"testing"
"time"
@@ -12,7 +14,9 @@ import (
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
mwc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshwantsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
mpr "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-assigner/internal/assigngraph"
"mesh-delivery-assigner/internal/buckets"
@@ -132,3 +136,141 @@ func TestTheWantsAreSaidWantedFromTheKeptAssignmentGraphRevision(t *testing.T) {
t.Errorf("dropping the run's candidate dropped trunk: %v", err)
}
}
// blockedObjects is an object bucket whose listing waits while a gate stands: a keeping that does not end until it is opened.
type blockedObjects struct {
memObjects
mu sync.Mutex
gate chan struct{}
}
func (b *blockedObjects) Names(ctx context.Context) ([]string, error) {
b.mu.Lock()
gate := b.gate
b.mu.Unlock()
if gate != nil {
<-gate
}
b.mu.Lock()
defer b.mu.Unlock()
return b.memObjects.Names(ctx)
}
func (b *blockedObjects) Put(ctx context.Context, name string, data []byte) error {
b.mu.Lock()
defer b.mu.Unlock()
return b.memObjects.Put(ctx, name, data)
}
func (b *blockedObjects) Get(ctx context.Context, name string) ([]byte, error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.memObjects.Get(ctx, name)
}
// block stands a gate, opened by the function it answers.
func (b *blockedObjects) block() func() {
gate := make(chan struct{})
b.mu.Lock()
b.gate = gate
b.mu.Unlock()
return func() {
b.mu.Lock()
b.gate = nil
b.mu.Unlock()
close(gate)
}
}
// Design 47 H-A-259, H-A-261: the keeping of the assignment graph runs off the pass loop. At the start, while the first keeping is
// blocked, a pass returns and writes no want; the keeping's end wakes a pass, which writes the passed pipeline's want. Later, while a
// keeping of new inputs is blocked, a pass writes the wants from the revision it holds, and a pass of the same inputs hands nothing
// more. Each pass and each keeping says its duration.
func TestAPassWritesWantsWhileAKeepingIsBlocked(t *testing.T) {
fp := mcc.SourcePrefix2 + strings.Repeat("a", 64)
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(kvSet{buckets.Wants(): wantsKV, buckets.Asks(): &memKV{values: map[string][]byte{}}}, time.Now())
var mu sync.Mutex
var journal strings.Builder
p.logf = func(f string, a ...any) { mu.Lock(); fmt.Fprintf(&journal, f+"\n", a...); mu.Unlock() }
said := func(words string) bool {
mu.Lock()
defer mu.Unlock()
return strings.Contains(journal.String(), words)
}
mdigest := strings.Repeat("e", 64)
mod, _ := mcc.EncodeModule(meshcatalog.ModulesByModuleValue{Fp2: fp, Manifest: mdigest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: mcc.RootPath})
putModuleRaw(p.w, trunkBranch, "web", mod)
p.w.manifests[mdigest] = parsed{module: "web", manifest: catalogue.Manifest{Module: "web"}}
placeOn(t, p.w, "ace", "web")
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
p.ends.trunk = p.w.trunkName
_, _ = p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("web", fp, "0.1.2", 1, mpr.StepResultOutcomePassed)))
objects := &blockedObjects{memObjects: memObjects{}}
var kept sync.WaitGroup
woke := make(chan struct{}, 8)
half := mcc.Half{Revision: meshcatalog.CatalogRevision{Branch: trunkBranch, Kind: "trunk", Revision: 4, Source: 9,
Entries: []meshcatalog.EntryRef{{Module: "web", Digest: "sha256:" + strings.Repeat("a", 64)}}},
Entries: map[string]meshcatalog.CatalogEntry{"web": {Module: "web", Fp: fp}}}
p.graph = &graphKeeper{objects: objects, halves: func() map[string]mcc.Half { return map[string]mcc.Half{trunkBranch: half} },
background: func(f func()) { kept.Go(f) }, wake: func() { woke <- struct{}{} }}
pass := func() {
t.Helper()
done := make(chan struct{})
go func() { p.pass(context.Background()); close(done) }()
select {
case <-done:
case <-time.After(10 * time.Second):
t.Fatal("the pass waits on the keeping of the assignment graph")
}
}
wantOf := func(node string) (mwc.Want, error) {
key, _ := mwc.WantKey(node, "web")
return mwc.DecodeWant(wantsKV.values[key])
}
open := objects.block()
t.Cleanup(func() { // the keeping ends before the test does, and says its duration
objects.mu.Lock()
standing := objects.gate != nil
objects.mu.Unlock()
if standing {
open()
}
kept.Wait()
if !said("the keeping of the assignment graph took ") {
t.Error("the keeping does not say its duration")
}
})
pass() // the start: the first keeping is blocked
if _, err := wantOf("ace"); err == nil {
t.Error("a pass at the start wrote a want before an assignment graph revision is kept")
}
open()
select {
case <-woke:
case <-time.After(10 * time.Second):
t.Fatal("the first keeping's end wakes no pass")
}
pass()
if w, err := wantOf("ace"); err != nil || w.Fp != fp || w.Version != "0.1.2" {
t.Errorf("the want of web on ace = %+v (%v), want the passed version once a revision is kept", w, err)
}
open = objects.block() // a keeping of new inputs is blocked
placeOn(t, p.w, "g14", "web")
pass()
pass() // the same inputs again
if w, err := wantOf("g14"); err != nil || w.Fp != fp || w.Version != "0.1.2" {
t.Errorf("the want of web on g14 = %+v (%v), want the passed version while a keeping is blocked", w, err)
}
p.graph.mu.Lock()
keeping, waiting := p.graph.keeping, p.graph.need
p.graph.mu.Unlock()
if !keeping || waiting != nil {
t.Errorf("keeping %v, need waiting %+v: the pass of the same inputs handed its revision again", keeping, waiting)
}
if !said("the pass took ") {
t.Error("the pass does not say its duration")
}
}
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/keys"
"io"
"maps"
"mesh-delivery-assigner/internal/assigngraph"
"mesh-delivery-assigner/internal/assignment"
"mesh-delivery-assigner/internal/buckets"
"mesh-delivery-assigner/internal/puts"
@@ -190,6 +191,11 @@ func startBed(t *testing.T, leaveOut string) *bed {
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: catalogue.BusName("mesh-delivery-assigner", catalogue.StateObjectsName(buckets.InputsState))}); err != nil {
t.Fatal(err)
}
// The assignment graph revisions' object store, which the grants part makes from the manifest's objects: no want is written
// before a revision is kept in it (design 47 H-A-261).
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: catalogue.ObjectsBucket("mesh-delivery-assigner", assigngraph.RevisionsObjects)}); err != nil {
t.Fatal(err)
}
// The builder's module log, as the runtime raises it (ADR 0297): the running commits are read from it (ADR 0571 clause 1).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(builds.BuilderModule, builds.BuilderLog),
Subjects: []string{buscompose.LogSubject(builds.BuilderModule, builds.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
@@ -613,7 +613,10 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
p.publishWants = wantsPublisher(ctx, js, thisNode)
if iv != nil && iv.h != nil {
objects := newBusObjects(js, process.Account{Node: thisNode, Module: assignment.Module()}, assigngraph.RevisionsObjects)
p.graph = &graphKeeper{objects: objects, halves: iv.h.views}
// The keeping and removing of revisions runs off the pass loop, and a keeping that changed what the pass reads wakes
// one (design 47 H-A-259).
p.graph = &graphKeeper{objects: objects, halves: iv.h.views, background: func(f func()) { go f() }, wake: wake,
reads: newBusObjects(js, process.Account{Node: thisNode, Module: assignment.Module()}, assigngraph.RevisionsObjects)}
keep := func(ctx context.Context, ev meshcatalog.CatalogRevisionEvent) error {
return p.keepCandidate(ctx, iv.call, ev)
}
@@ -628,7 +631,7 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
// The revisions that are not current are removed when one ceases to be current and once now (revisions.go,
// design 47 H §764, H-A-86).
p.graph.revisions = &revisionRemover{ctx: ctx, objects: objects, inUse: inUseOf(w), logf: logger.Printf}
p.graph.revisions.judge()
// The start's removal of the revisions not current folds into the first pass's keeping (H-A-261).
}
}
// The node-engine seat's manifest-reads-changed, for one node (design 47 H-A-104): its words are written into the state
@@ -88,6 +88,43 @@ func (b *busObjects) Names(ctx context.Context) ([]string, error) {
return names, nil
}
// PutDescribed stores data under name with the entry's module and fingerprint in its metadata, which the listing says (assigngraph.Describer,
// design 47 H-A-259).
func (b *busObjects) PutDescribed(ctx context.Context, name string, data []byte, module, fp string) error {
o, err := b.open(ctx)
if err != nil {
return err
}
_, err = o.Put(ctx, jetstream.ObjectMeta{Name: name, Metadata: map[string]string{entryModuleMeta: module, entryFpMeta: fp}}, bytes.NewReader(data))
return err
}
// Described is every object with the module and fingerprint its metadata says, in one listing.
func (b *busObjects) Described(ctx context.Context) ([]assigngraph.Described, error) {
o, err := b.open(ctx)
if err != nil {
return nil, err
}
infos, err := o.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return []assigngraph.Described{}, nil
}
if err != nil {
return nil, err
}
out := make([]assigngraph.Described, 0, len(infos))
for _, i := range infos {
out = append(out, assigngraph.Described{Name: i.Name, Module: i.Metadata[entryModuleMeta], Fp: i.Metadata[entryFpMeta]})
}
return out, nil
}
// The metadata keys of an entry object's module and fingerprint.
const (
entryModuleMeta = "module"
entryFpMeta = "fp"
)
// Delete removes the object of that name; one not held is no error (design 47 G §579: a revision nothing pins is removed).
func (b *busObjects) Delete(ctx context.Context, name string) error {
o, err := b.open(ctx)
@@ -141,13 +141,14 @@ func (p *passer) clock() time.Time {
}
func (p *passer) pass(ctx context.Context) {
start := time.Now()
defer func() { p.logf("the pass took %s", time.Since(start).Round(time.Millisecond)) }() // design 47 H-A-259
p.assignmentsSeat(ctx)
p.keepGraph(ctx)
if p.graph == nil || !p.graph.full { // a full assignment graph store keeps the last revision and writes no new want (design 47 G §181)
if held := p.keepGraph(ctx); held == "" {
p.wants(ctx)
p.sayPass(ctx, "judged")
} else {
p.sayPass(ctx, "the assignment graph store is full: no want is written (design 47 G §181)")
p.sayPass(ctx, held) // a full store keeps the last revision (design 47 G §181); none kept yet writes no want (H-A-261)
}
p.puts(ctx)
p.nodeWants(ctx)
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"slices"
"strconv"
@@ -53,7 +54,7 @@ func trunkRevision(name string) (int64, bool) {
// remove removes each kept revision that is not current, and then the entries no kept revision or candidate lists and no want or
// running build names (design 47 G §179).
func (k *revisionRemover) remove() error {
names, err := k.objects.Names(k.ctx)
names, described, err := k.listing()
if err != nil {
return err
}
@@ -107,7 +108,7 @@ func (k *revisionRemover) remove() error {
if !ok || listed[digest] {
continue
}
e, err := assigngraph.ReadEntry(k.ctx, k.objects, digest)
e, err := k.entryOf(n, digest, described)
if err != nil {
return err
}
@@ -138,3 +139,46 @@ func inUseOf(w *world) func(module, fp string) bool {
return false
}
}
// listing is every object's name in one read and, where the bucket says them, each entry object's module and fingerprint by name
// (design 47 H-A-259); nil described where the bucket says none.
func (k *revisionRemover) listing() ([]string, map[string]assigngraph.Described, error) {
d, ok := k.objects.(assigngraph.Describer)
if !ok {
names, err := k.objects.Names(k.ctx)
return names, nil, err
}
listed, err := d.Described(k.ctx)
if err != nil {
return nil, nil, err
}
names := make([]string, 0, len(listed))
described := make(map[string]assigngraph.Described, len(listed))
for _, l := range listed {
names = append(names, l.Name)
described[l.Name] = l
}
return names, described, nil
}
// entryOf is the module and fingerprint of the entry object name: as the listing says them, with no read; an entry put before the
// listing said them is read once and put again with them, so no later judging reads it (design 47 H-A-259).
func (k *revisionRemover) entryOf(name, digest string, described map[string]assigngraph.Described) (meshwants.AssignmentEntry, error) {
if l, ok := described[name]; ok && l.Module != "" && l.Fp != "" {
return meshwants.AssignmentEntry{Module: l.Module, Fp: l.Fp}, nil
}
data, err := k.objects.Get(k.ctx, name)
if err != nil {
return meshwants.AssignmentEntry{}, fmt.Errorf("the entry %s: %w", digest, err)
}
var e meshwants.AssignmentEntry
if err := json.Unmarshal(data, &e); err != nil {
return e, fmt.Errorf("the entry %s: %w", digest, err)
}
if described != nil {
if err := assigngraph.PutEntry(k.ctx, k.objects, name, data, e); err != nil {
k.logf("the entry %s is not put again with its module and fingerprint, so the next judging reads it again: %v", digest, err)
}
}
return e, nil
}
@@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
@@ -12,42 +13,98 @@ import (
func (m memObjects) Delete(_ context.Context, name string) error { delete(m, name); return nil }
// describedObjects is memObjects whose listing says each entry's module and fingerprint (assigngraph.Describer), counting the
// entry objects read.
type describedObjects struct {
memObjects
meta map[string][2]string
reads int
}
func (d *describedObjects) Get(ctx context.Context, name string) ([]byte, error) {
if strings.HasPrefix(name, "entry.") {
d.reads++
}
return d.memObjects.Get(ctx, name)
}
func (d *describedObjects) PutDescribed(_ context.Context, name string, data []byte, module, fp string) error {
d.memObjects[name], d.meta[name] = data, [2]string{module, fp}
return nil
}
func (d *describedObjects) Described(context.Context) ([]assigngraph.Described, error) {
var out []assigngraph.Described
for n := range d.memObjects {
out = append(out, assigngraph.Described{Name: n, Module: d.meta[n][0], Fp: d.meta[n][1]})
}
return out, nil
}
// Design 47 H §764, H-A-86: with the gate's pins gone, an assignment graph revision that is not current is removed, with the
// entries no kept revision or candidate lists and no want or running build names; the current revision, the entries it lists,
// a running build's entry and a run's kept candidate with its entries stay.
// H-A-259: where the listing says each entry's module and fingerprint, the judging reads no entry object; an entry put before it
// said them is read once and put again with them.
func TestARevisionNotCurrentIsRemoved(t *testing.T) {
objects := memObjects{}
entry := func(module, fp string) string {
e := meshwants.AssignmentEntry{Node: "n1", Module: module, Fp: fp}
d := assigngraph.EntryDigest(e)
raw, _ := json.Marshal(e)
objects[assigngraph.EntryObject(d)] = raw
return d
}
shared, old, running, cand := entry("a", "fp-a"), entry("b", "fp-b1"), entry("c", "fp-c1"), entry("d", "fp-d2")
revision := func(n int64, digests ...string) {
rev := meshwants.AssignmentRevision{Revision: n}
for _, d := range digests {
rev.Entries = append(rev.Entries, meshwants.NodeEntryRef{Node: "n1", Digest: d})
for _, listed := range []string{"plain", "described", "described-later"} {
objects := memObjects{}
described := &describedObjects{memObjects: objects, meta: map[string][2]string{}}
entry := func(module, fp string) string {
e := meshwants.AssignmentEntry{Node: "n1", Module: module, Fp: fp}
d := assigngraph.EntryDigest(e)
raw, _ := json.Marshal(e)
objects[assigngraph.EntryObject(d)] = raw
if listed == "described" {
described.meta[assigngraph.EntryObject(d)] = [2]string{module, fp}
}
return d
}
raw, _ := json.Marshal(rev)
objects[assigngraph.TrunkObject(n)] = raw
}
candRaw, _ := json.Marshal(meshwants.AssignmentRevision{Revision: 1, Kind: "candidate", Group: "run-7", Heads: "h1",
Entries: []meshwants.NodeEntryRef{{Node: "n1", Module: "d", Digest: cand}}})
objects[assigngraph.CandidateObject("run-7", "h1")] = candRaw
revision(1, shared, old, running)
revision(2, shared, old)
revision(3, shared)
k := &revisionRemover{ctx: context.Background(), objects: objects, logf: t.Logf,
inUse: func(module, fp string) bool { return module == "c" && fp == "fp-c1" }}
k.judge()
has := func(name string) bool { _, ok := objects[name]; return ok }
for name, want := range map[string]bool{assigngraph.TrunkObject(1): false, assigngraph.TrunkObject(2): false, assigngraph.TrunkObject(3): true,
assigngraph.EntryObject(old): false, assigngraph.EntryObject(running): true, assigngraph.EntryObject(shared): true,
assigngraph.EntryObject(cand): true, assigngraph.CandidateObject("run-7", "h1"): true} {
if has(name) != want {
t.Errorf("%s kept = %v, want %v", name, has(name), want)
shared, old, running, cand := entry("a", "fp-a"), entry("b", "fp-b1"), entry("c", "fp-c1"), entry("d", "fp-d2")
revision := func(n int64, digests ...string) {
rev := meshwants.AssignmentRevision{Revision: n}
for _, d := range digests {
rev.Entries = append(rev.Entries, meshwants.NodeEntryRef{Node: "n1", Digest: d})
}
raw, _ := json.Marshal(rev)
objects[assigngraph.TrunkObject(n)] = raw
}
candRaw, _ := json.Marshal(meshwants.AssignmentRevision{Revision: 1, Kind: "candidate", Group: "run-7", Heads: "h1",
Entries: []meshwants.NodeEntryRef{{Node: "n1", Module: "d", Digest: cand}}})
objects[assigngraph.CandidateObject("run-7", "h1")] = candRaw
revision(1, shared, old, running)
revision(2, shared, old)
revision(3, shared)
var o revisionObjects = objects
if listed != "plain" {
o = described
}
k := &revisionRemover{ctx: context.Background(), objects: o, logf: t.Logf,
inUse: func(module, fp string) bool { return module == "c" && fp == "fp-c1" }}
k.judge()
has := func(name string) bool { _, ok := objects[name]; return ok }
for name, want := range map[string]bool{assigngraph.TrunkObject(1): false, assigngraph.TrunkObject(2): false, assigngraph.TrunkObject(3): true,
assigngraph.EntryObject(old): false, assigngraph.EntryObject(running): true, assigngraph.EntryObject(shared): true,
assigngraph.EntryObject(cand): true, assigngraph.CandidateObject("run-7", "h1"): true} {
if has(name) != want {
t.Errorf("%s: %s kept = %v, want %v", listed, name, has(name), want)
}
}
switch listed {
case "described":
if described.reads != 0 {
t.Errorf("described: the judging read %d entry objects, want none", described.reads)
}
case "described-later":
if described.meta[assigngraph.EntryObject(running)] != [2]string{"c", "fp-c1"} {
t.Errorf("described-later: the entry in use is not put again with its module and fingerprint: %v", described.meta)
}
reads := described.reads
revision(4, shared)
k.judge()
if described.reads != reads {
t.Errorf("described-later: a second judging read %d entry objects again", described.reads-reads)
}
}
}
}
@@ -30,6 +30,29 @@ type Objects interface {
Names(ctx context.Context) ([]string, error)
}
// Described is one object as the bucket's listing says it: its name, and for an entry object the module and fingerprint put beside
// its bytes (empty for one put before them).
type Described struct {
Name, Module, Fp string
}
// Describer is an object bucket whose listing says, in one read, each entry object's module and fingerprint, so the removal of the
// revisions not current reads no object per entry (design 47 H-A-259).
type Describer interface {
// PutDescribed stores data whole under name, with module and fp beside it in the listing.
PutDescribed(ctx context.Context, name string, data []byte, module, fp string) error
// Described is every object, as the listing says it.
Described(ctx context.Context) ([]Described, error)
}
// PutEntry stores the entry object e under name, with its module and fingerprint beside it where the bucket says them.
func PutEntry(ctx context.Context, o Objects, name string, data []byte, e meshwants.AssignmentEntry) error {
if d, ok := o.(Describer); ok {
return d.PutDescribed(ctx, name, data, e.Module, e.Fp)
}
return o.Put(ctx, name, data)
}
// EntryObject is the object name of the entry of that digest (the seat's key form entry.<digest>).
func EntryObject(digest string) string { return "entry." + digest }
@@ -144,7 +167,7 @@ func writeEntries(ctx context.Context, o Objects, c Computed, rev meshwants.Assi
if err != nil {
return err
}
if err := o.Put(ctx, name, data); err != nil {
if err := PutEntry(ctx, o, name, data, e); err != nil {
return fmt.Errorf("the entry of %s/%s in %s: %w", ref.Node, ref.Module, RevisionsObjects, err)
}
held[name] = true
@@ -14,7 +14,8 @@ import (
"mesh-delivery-builder/internal/versions"
)
// readerOf is a kept state as the versions package's reader.
// readerOf is a kept state as the versions package's reader: its values read in one page, one request for many (design 47 H
// H-A-290 (5), H-A-292), where a key-by-key read of a queue of two thousand keys outlasts the tool's call.
func readerOf(s stdio.KeptState) versions.Reader {
return versions.Reader{Keys: s.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := s.Get(key)
@@ -22,7 +23,34 @@ func readerOf(s stdio.KeptState) versions.Reader {
return nil, false, err
}
return e.Value, true, nil
}}
}, All: func() (versions.StatePage, error) { return onePage(s.Page, s.Info) }}
}
// statePageBytes and statePageBudget bound the one page of a state read, so its answer fits one bus message and its time.
const (
statePageBytes = 768 * 1024
statePageBudget = 10 * time.Second
)
// onePage is the first page of a state's values, and never a second (H-A-292, H-A-293): when the state holds more, the page says
// so with the state's size from its info. A runtime that answers no page says errors.ErrUnsupported.
func onePage(page func(after string, maxBytes int, budget time.Duration) (stdio.StatePage, error), info func() (*stdio.StateInfo, error)) (versions.StatePage, error) {
pg, err := page("", statePageBytes, statePageBudget)
if err != nil {
return versions.StatePage{}, err
}
out := versions.StatePage{Values: make(map[string]json.RawMessage, len(pg.Entries)), More: pg.Next != ""}
for _, e := range pg.Entries {
out.Values[e.Key] = e.Value
}
if out.More {
if in, err := info(); err != nil || in == nil {
out.InfoWords = fmt.Sprintf("the state's info cannot be read: %v", err)
} else {
out.Keys, out.Bytes = in.Keys, in.Bytes
}
}
return out, nil
}
// republishTools are builder_publish_again and builder_publish_queue (design 47 G §588). The publish again is a hand act:
@@ -59,7 +87,11 @@ func republishTools(queue, acted, reasks versions.Reader, ask queueAsker, now fu
return map[string]any{"key": key, "queued": a.Queued, "answer": a.Answer}, nil
}},
{Name: versions.PublishQueueTool, Description: versions.PublishQueueDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return versions.AnswerQueue(queue, acted, reasks) }},
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "list only this module's queued versions, each with its class"},
"version": map[string]any{"type": "string", "description": "with module: only this version, MAJOR.MINOR.PATCH"}}},
Run: func(args map[string]any) (any, error) {
return versions.AnswerQueue(queue, acted, reasks, str(args, "module"), str(args, "version"))
}},
}
}
@@ -0,0 +1,44 @@
package main
import (
"encoding/json"
"errors"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// TestAStateIsReadInOnePage: design 47 H H-A-292, H-A-293 — builder_publish_queue reads each of its own states in one page and
// never asks a second: a state past the page answers that page with More and the state's size from its info; one the page holds
// answers whole without reading the info.
func TestAStateIsReadInOnePage(t *testing.T) {
for _, c := range []struct {
name string
next string
}{{"past one page", "k1"}, {"held by one page", ""}} {
t.Run(c.name, func(t *testing.T) {
var asked []string
infos := 0
page := func(after string, maxBytes int, budget time.Duration) (stdio.StatePage, error) {
asked = append(asked, after)
if after != "" {
return stdio.StatePage{Entries: []stdio.StateEntry{{Key: "k2", Value: json.RawMessage(`2`)}}}, nil
}
return stdio.StatePage{Entries: []stdio.StateEntry{{Key: "k1", Value: json.RawMessage(`1`)}}, Next: c.next}, nil
}
info := func() (*stdio.StateInfo, error) { infos++; return &stdio.StateInfo{Keys: 2, Bytes: 640}, nil }
pg, err := onePage(page, info)
more := c.next != ""
if err != nil || len(asked) != 1 || len(pg.Values) != 1 || pg.More != more || (more && (pg.Keys != 2 || pg.Bytes != 640 || infos != 1)) || (!more && infos != 0) {
t.Errorf("asked the pages after %q, read %v, more %v, size %d keys %d bytes, info read %d times (%v): want one page only, its rest said with the state's size", asked, pg.Values, pg.More, pg.Keys, pg.Bytes, infos, err)
}
})
}
unsupported := func(string, int, time.Duration) (stdio.StatePage, error) {
return stdio.StatePage{}, errors.ErrUnsupported
}
if _, err := onePage(unsupported, nil); !errors.Is(err, errors.ErrUnsupported) {
t.Errorf("a runtime with no pages answered %v, want ErrUnsupported so the reader reads key by key", err)
}
}
@@ -239,6 +239,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// and version-released heard; nothing is reserved before the fill.
reservedKV := process.AwaitBucket(ctx, js, conn, own.reserved, "no version is reserved: builds are made unreserved", logger.Printf)
reserved := newReservations(awaitedReserved{awaitedVersions{b: reservedKV, bucket: own.reserved}}, calls.Call, pubBoundsKV.KV, found, b.Reconsider, logger.Printf)
// The index by digest and by version beside the reservations is written again once the state is bound (H-A-269 (1), H-A-271).
reservedKV.Ready(func(jetstream.KeyValue) { reserved.reindex(ctx) })
// Each artifact the holder says published at a reserved version is said as the seat mesh-builds' package-published (design 47 G §52).
reserved.say = func(ctx context.Context, p meshbuilds.PackagePublished) error {
return emitSeatEvent(context.WithoutCancel(ctx), js, cred, meshbuilds.SubjectPackagePublished, "package-published-"+p.Module+"-"+p.Version+"-"+p.Artifact, p)
@@ -259,7 +261,28 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
republish := &versions.Republisher{Reserved: reserved.r, Queue: awaitedQueue{awaitedReserved{awaitedVersions{b: queueKV, bucket: own.queue}}},
Entry: entryVia(calls.Call), Send: sendVia(calls.Call),
Unreleased: reserved.unreleasedWords, Logf: logger.Printf,
Holders: holders} // a registry seat's holder from the seat-holders view, from the start (design 47 G A-416, A-416a)
Holders: holders, // a registry seat's holder from the seat-holders view, from the start (design 47 G A-416, A-416a)
Started: time.Now()} // a build ended from now on is of the live class, sent first (H-A-290 (1))
// A number a registry seat already holds is never reserved (H-A-275): read before reserving, through the seats' verb versions.
reserved.r.HeldNumbers = func(ctx context.Context, module, fp string, refs []string) ([]string, error) {
return republish.HeldNumbers(ctx, module, fp, refs, registryVersionsVia(calls.Call))
}
// A done build whose number a registry holds elsewhere is built again at the next free number (H-A-279), on the builder's context.
reserved.rebuild = func(_ context.Context, module, fp, why string) error {
res := b.BuildAgain(ctx, module, fp, why, "the builder", buildAgainBound)
switch {
case res.Refused != "":
return errors.New(res.Refused)
case res.Failed != "":
return errors.New(res.Failed)
case res.Pending != "":
logger.Printf("%s at %s is built again: %s", module, fp, res.Pending)
}
return nil
}
reserved.r.HeldElsewhere = func(ctx context.Context, module string, skipped []string) {
reserved.heldElsewhere(ctx, module, "the registries hold "+strings.Join(skipped, ", ")+", so the reserve skipped them")
}
publishSet, err := publishSettingsOf(getenv)
if err != nil {
return err
@@ -275,10 +298,16 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return reserved.r.Of(ctx, module, version)
},
republish: republish.Republish, endedWith: reserved.unreleased,
allDone: func(ctx context.Context) ([]versions.ReservedVersion, error) {
all, err := reserved.r.All(ctx)
return builtOf(all), err
}}
// A publish is asked again only while no published of it may be unheard and the registry's record lacks it (H-A-269 (2), H-A-272).
unheard: unheardVia(js, process.EventsStream, cred.EventConsumerName()),
judge: func(ctx context.Context, v versions.ReservedVersion) (versions.Judged, error) {
return republish.Held(ctx, v, registryVersionsVia(calls.Call))
},
markHeld: reserved.markHeld, moveElsewhere: reserved.movedElsewhere,
allDone: func(ctx context.Context) ([]versions.ReservedVersion, error) { return reserved.r.All(ctx) },
retired: reserved.r.Retired} // only each module's newest done build is asked at the start (H-A-291 (1))
// A version that never publishes (withdrawn or taken over) has its publish item ended by the queue too (H-A-290 (2)).
republish.Ended = items.Done
reasksKV.Ready(func(jetstream.KeyValue) { items.AskAllDone(ctx) })
wirePublishItems(reserved, items, awaitedReserved{awaitedVersions{b: actedKV, bucket: own.acted}}, releasedKept)
acts.Republish = republish.Republish
@@ -46,11 +46,25 @@ type publishItems struct {
reserved func(ctx context.Context, module, version string) (versions.ReservedVersion, bool, error)
republish func(ctx context.Context, module, version string) ([]string, error)
endedWith func(ctx context.Context, v versions.ReservedVersion, why string) // raises the finding version-unreleased
allDone func(ctx context.Context) ([]versions.ReservedVersion, error)
set publishSettings
now func() time.Time
after func(time.Duration) <-chan time.Time
logf func(string, ...any)
// allDone are the reservations, done or not: those whose build is done are asked at the start (builtOf).
allDone func(ctx context.Context) ([]versions.ReservedVersion, error)
// retired are the done builds a newer reservation or release of the same module supersedes, with why (H-A-291 (1)); nil
// retires none.
retired func(ctx context.Context, all []versions.ReservedVersion) (map[string]string, error)
// unheard is how many messages the builder's durable on the events stream holds pending and unacknowledged, read before each
// try (design 47 H H-A-269 (2), H-A-272); nil reads none.
unheard func(ctx context.Context) (int, error)
// judge is what the registries' own record says of a version's missing artifacts (H-A-272, H-A-274, H-A-275); nil judges none.
judge func(ctx context.Context, v versions.ReservedVersion) (versions.Judged, error)
// markHeld marks the artifacts the registries hold with the built digest published from their record, as their published heard
// would (H-A-274); moveElsewhere moves a version a registry holds at another digest to the next free number and asks its
// publish there (H-A-275). Nil does neither.
markHeld func(ctx context.Context, v versions.ReservedVersion, held []versions.HeldArtifact)
moveElsewhere func(ctx context.Context, v versions.ReservedVersion, j versions.Judged) error
set publishSettings
now func() time.Time
after func(time.Duration) <-chan time.Time
logf func(string, ...any)
}
func (p *publishItems) wait(ctx context.Context, d time.Duration) bool {
@@ -88,12 +102,28 @@ func (p *publishItems) Ask(ctx context.Context, module, version string) error {
// AskAllDone asks the publish item of every reservation whose build is done and that has no release: at the builder's start, so
// what stood before this change moves too, whatever its artifacts.
func (p *publishItems) AskAllDone(ctx context.Context) {
vs, err := p.allDone(ctx)
all, err := p.allDone(ctx)
if err != nil {
p.logf("the reservations of done builds are not asked to publish at the start: %v", err)
return
}
for _, v := range vs {
var retired map[string]string
if p.retired != nil {
if retired, err = p.retired(ctx, all); err != nil {
p.logf("the superseded builds are not retired at the start: %v", err)
}
}
for _, v := range builtOf(all) {
if why := retired[v.Module+" "+v.Version]; why != "" { // only each module's newest is asked (H-A-291 (1), H-A-275)
p.logf("%s %s is not asked to publish at the start: %s; its publish item is ended", v.Module, v.Version, why)
p.Done(ctx, v.Module, v.Version)
continue
}
if why := versions.NotStanding(v, true); why != "" { // withdrawn: it never publishes (H-A-290 (2))
p.logf("%s %s is not asked to publish at the start: %s; its publish item is ended", v.Module, v.Version, why)
p.Done(ctx, v.Module, v.Version)
continue
}
if err := p.Ask(ctx, v.Module, v.Version); err != nil {
p.logf("%v", err)
}
@@ -127,7 +157,8 @@ func (p *publishItems) Try(ctx context.Context, kv jetstream.KeyValue, e jetstre
p.logf("%s %s: its reservation cannot be read, the publish item waits: %v", r.Module, r.Version, err)
return
}
if !held { // dropped, taken over or released: never because its artifacts are published (A-456b)
if why := versions.NotStanding(v, held); why != "" { // dropped, taken over, released or withdrawn: never because its artifacts are published (A-456b, H-A-290 (2))
p.logf("%s %s: the publish item is ended: %s", r.Module, r.Version, why)
_ = kv.Delete(ctx, e.Key())
return
}
@@ -138,6 +169,12 @@ func (p *publishItems) Try(ctx context.Context, kv jetstream.KeyValue, e jetstre
if p.endedWith != nil {
p.endedWith(ctx, v, "the publish ended after "+fmt.Sprint(r.Tries)+" tries")
}
} else if why, gone := p.notAgain(ctx, v); gone {
return // moved to another number, whose own item is asked (H-A-275)
} else if why != "" {
// A published of the version may be unheard, or the registry's record ended it: no ask, and no try spent (H-A-272, H-A-274).
p.logf("%s %s: the publish is not asked again: %s; tried again after %s", r.Module, r.Version, why, p.set.pauseAfter(r.Tries))
r.NextTry = p.now().UTC().Add(p.set.pauseAfter(r.Tries))
} else {
r.Tries++
if _, err := p.republish(ctx, r.Module, r.Version); err != nil {
@@ -154,6 +191,85 @@ func (p *publishItems) Try(ctx context.Context, kv jetstream.KeyValue, e jetstre
}
}
// notAgain says why a version's publish is not asked again now, empty when it is (design 47 H H-A-269 (2), H-A-272, H-A-274,
// H-A-275): the builder's durable holds UnheardBound messages or more not yet heard, so a published of it may be among them; or the
// registries' own record holds every missing artifact at its version with its built digest, which marks them published from that
// record; or a registry holds the version at another digest, which moves the build to the next free number (gone: this item is
// ended with the old number). A count or a record that cannot be read is said, and does not hold the ask.
func (p *publishItems) notAgain(ctx context.Context, v versions.ReservedVersion) (why string, gone bool) {
if p.unheard != nil {
n, err := p.unheard(ctx)
switch {
case err != nil:
p.logf("%s %s: the builder's events consumer cannot be read, so the publish is asked again as it stands: %v", v.Module, v.Version, err)
case n >= versions.UnheardBound:
return fmt.Sprintf("the builder's events consumer holds %d messages not yet heard (the bound is %d), a published of it may be among them", n, versions.UnheardBound), false
}
}
if p.judge == nil {
return "", false
}
j, err := p.judge(ctx, v)
switch {
case err != nil:
p.logf("%s %s: the registries' record cannot be read, so the publish is asked again: %v", v.Module, v.Version, err)
case len(j.Elsewhere) > 0 && p.moveElsewhere != nil:
if err := p.moveElsewhere(ctx, v, j); err != nil {
p.logf("%s %s: a registry holds the number at another digest, and the build is not moved: %v", v.Module, v.Version, err)
return "a registry holds " + v.Version + " at another digest", false
}
return "", true
case j.All:
if p.markHeld != nil {
p.markHeld(ctx, v, j.Held)
}
return "the registries hold every missing artifact at " + v.Version + " with its built digest: marked published from their record", false
default:
p.logf("%s %s: asked again: %s", v.Module, v.Version, j.Lacks)
}
return "", false
}
// unheardVia reads the builder's durable on the events stream: its messages pending and those given and not yet acknowledged
// (design 47 H H-A-272), what it has not yet heard.
func unheardVia(js jetstream.JetStream, stream, durable string) func(context.Context) (int, error) {
return func(ctx context.Context) (int, error) {
c, err := js.Consumer(ctx, stream, durable)
if err != nil {
return 0, err
}
info, err := c.Info(ctx)
if err != nil {
return 0, err
}
return int(info.NumPending) + info.NumAckPending, nil
}
}
// registryVersionsVia asks a registry seat's verb versions for a name (H-A-272): each version it holds with its digest.
func registryVersionsVia(call toolCaller) func(context.Context, string, string) ([]versions.HeldVersion, error) {
return func(_ context.Context, seat, name string) ([]versions.HeldVersion, error) {
raw, err := call("seat:"+seat+"."+versions.VerbVersions, map[string]any{"name": name})
if err != nil {
return nil, err
}
var a struct { // the four registry seats answer versions alike: each version with its digest
Versions []struct {
Version string `json:"version"`
Digest string `json:"digest"`
} `json:"versions"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s's versions answered what is no list of versions: %w", seat, err)
}
out := make([]versions.HeldVersion, 0, len(a.Versions))
for _, v := range a.Versions {
out = append(out, versions.HeldVersion{Version: v.Version, Digest: v.Digest})
}
return out, nil
}
}
// numberFile reads a setting's number from the file the manifest declares: no file or an empty one is def; a content that is no
// number of at least one is an error naming the file.
func numberFile(path string, def int) (int, error) {
@@ -4,9 +4,12 @@ import (
"context"
"encoding/json"
"errors"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
@@ -19,9 +22,10 @@ import (
"mesh-delivery-builder/internal/versions"
)
// itemKV is the re-ask state in memory: one revision per key.
// itemKV is the re-ask state in memory: one revision per key, its maps locked (a try writes from the re-ask watch's goroutine).
type itemKV struct {
jetstream.KeyValue
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
rev uint64
@@ -42,13 +46,25 @@ func (e itemEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValueP
func (k *itemKV) entry(key string) jetstream.KeyValueEntry {
return itemEntry{key: key, value: k.vals[key], rev: k.revs[key]}
}
// has says whether key holds a value.
func (k *itemKV) has(key string) bool {
k.mu.Lock()
defer k.mu.Unlock()
_, ok := k.vals[key]
return ok
}
func (k *itemKV) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; !ok {
return nil, jetstream.ErrKeyNotFound
}
return k.entry(key), nil
}
func (k *itemKV) Create(_ context.Context, key string, v []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; ok {
return 0, jetstream.ErrKeyExists
}
@@ -57,6 +73,8 @@ func (k *itemKV) Create(_ context.Context, key string, v []byte, _ ...jetstream.
return k.rev, nil
}
func (k *itemKV) Update(_ context.Context, key string, v []byte, rev uint64) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if k.revs[key] != rev {
return 0, jetstream.ErrKeyExists
}
@@ -65,6 +83,8 @@ func (k *itemKV) Update(_ context.Context, key string, v []byte, rev uint64) (ui
return k.rev, nil
}
func (k *itemKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
k.mu.Lock()
defer k.mu.Unlock()
delete(k.vals, key)
return nil
}
@@ -140,21 +160,49 @@ func TestAPublishItemIsTriedWithAGrowingPauseAndEndsLoudly(t *testing.T) {
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
p.allDone = func(context.Context) ([]versions.ReservedVersion, error) {
published := versions.ReservedVersion{Module: "n", Version: "0.2.0", Artifacts: map[string]string{"n:a": "sha256:a"}, Published: map[string]bool{"n:a": true}, BuiltAt: now}
return builtOf([]versions.ReservedVersion{v, published, {Module: "o", Version: "0.1.0"}}), nil // the last build has not ended
withdrawn := versions.ReservedVersion{Module: "w", Version: "0.4.0", BuiltAt: now, Withdrawn: map[string]string{"0.4.0": "fp"}}
return builtOf([]versions.ReservedVersion{v, published, withdrawn, {Module: "o", Version: "0.1.0"}}), nil // the last build has not ended
}
_ = p.Ask(ctx, "w", "0.4.0") // asked before its reservation was withdrawn
p.AskAllDone(ctx)
if len(kv.vals) != 2 {
t.Errorf("the start asked %d items, want one per done unreleased reservation", len(kv.vals))
if _, ok := kv.vals[versions.PublishItemKey("w", "0.4.0")]; len(kv.vals) != 2 || ok {
t.Errorf("the start left %d items (withdrawn's kept: %v), want one per done unreleased reservation and none for a withdrawn one (H-A-290 (2))", len(kv.vals), ok)
}
// A start with three unreleased done versions of one module asks only the newest and retires the two (H-A-291 (1), H-A-275).
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
_ = p.Ask(ctx, "s", "0.1.3") // asked in an earlier run
p.allDone = func(context.Context) ([]versions.ReservedVersion, error) {
var out []versions.ReservedVersion
for _, ver := range []string{"0.1.3", "0.1.10", "0.1.2"} { // one series (H-A-293)
out = append(out, versions.ReservedVersion{Module: "s", Version: ver, Artifacts: map[string]string{"s:a": "sha256:a"}, BuiltAt: now})
}
return out, nil
}
p.retired = (&versions.Reserver{}).Retired
p.AskAllDone(ctx)
if _, ok := kv.vals[versions.PublishItemKey("s", "0.1.10")]; len(kv.vals) != 1 || !ok {
t.Errorf("the start left the items %v, want only the newest, s 0.1.10, asked and the two older retired", slices.Collect(maps.Keys(kv.vals)))
}
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
// A released version's item is deleted; a version no longer reserved has it deleted at its next try.
p.Done(ctx, "m", "0.1.2")
p.Done(ctx, "n", "0.2.0")
if len(kv.vals) != 0 {
t.Errorf("the release left %v", kv.vals)
}
// A withdrawn reservation has its item ended at its try, nothing queued (H-A-290 (2)).
queued = nil
_ = p.Ask(ctx, "m", "0.1.2")
v.Withdrawn = map[string]string{"0.1.2": "fp"}
e, _ := kv.Get(ctx, key)
p.Try(ctx, kv, e, decodeItem(t, e.Value()))
if len(kv.vals) != 0 || len(queued) != 0 {
t.Errorf("a withdrawn reservation left its item %v, queued %v", kv.vals, queued)
}
v.Withdrawn = nil
_ = p.Ask(ctx, "m", "0.1.2")
reserved = false
e, _ := kv.Get(ctx, key)
e, _ = kv.Get(ctx, key)
p.Try(ctx, kv, e, decodeItem(t, e.Value()))
if len(kv.vals) != 0 {
t.Errorf("a dropped reservation left its item %v", kv.vals)
@@ -220,7 +268,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
}
first := done("src2:one")
key := versions.PublishItemKey("m", first.Version)
if _, ok := kv.vals[key]; !ok {
if !kv.has(key) {
t.Fatalf("a done build has no publish item: %v", kv.vals)
}
// A build whose artifacts are all published still gets its item, until its release is written (A-456b).
@@ -231,7 +279,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if err := rv.Built(ctx, "m", first.Version, map[string]string{"m:a": "sha256:a"}, nil); err != nil {
t.Fatal(err)
}
if _, ok := kv.vals[key]; !ok {
if !kv.has(key) {
t.Fatalf("a build with every artifact published has no item: %v", kv.vals)
}
// The watch hands the item to its try, which queues the publish.
@@ -247,11 +295,11 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
}
// A takeover by another build deletes the older's item and asks the newer's.
second := done("src2:two")
if _, ok := kv.vals[key]; ok {
if kv.has(key) {
t.Errorf("the taken-over reservation kept its item")
}
secondKey := versions.PublishItemKey("m", second.Version)
if _, ok := kv.vals[secondKey]; !ok {
if !kv.has(secondKey) {
t.Fatalf("the newer build has no item: %v", kv.vals)
}
// version-released deletes the item and keeps the release.
@@ -259,7 +307,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if !rv.heard(ctx, msg) || !msg.acked {
t.Fatal("version-released was not taken in")
}
if _, ok := kv.vals[secondKey]; ok {
if kv.has(secondKey) {
t.Errorf("a released version kept its publish item")
}
if ok, _ := rv.ReleasedFor(ctx, "m", "src2:two"); !ok {
@@ -270,7 +318,80 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if _, err := rv.dropped(ctx, "m", "0.1", "op", "never", time.Unix(0, 0)); err != nil {
t.Fatal(err)
}
if _, ok := kv.vals[versions.PublishItemKey("m", third.Version)]; ok {
if kv.has(versions.PublishItemKey("m", third.Version)) {
t.Errorf("a dropped reservation kept its item")
}
}
// TestAPublishIsNotAskedAgainWhileItsPublishedMayBeUnheard: design 47 H H-A-269 (2), H-A-272, H-A-274, H-A-275 — a try asks the
// publish again only while the builder's durable holds fewer than a hundred messages not yet heard and the registries' record lacks
// an artifact; a record holding every missing artifact with the built digest marks them published from it; a record holding the
// version at another digest moves the build to the next free number and ends this item; a skipped try spends no try and is written
// again with its pause; a count that cannot be read holds no ask.
func TestAPublishIsNotAskedAgainWhileItsPublishedMayBeUnheard(t *testing.T) {
ctx := context.Background()
now := time.Date(2026, 11, 21, 15, 21, 0, 0, time.UTC)
v := versions.ReservedVersion{Module: "node-tools", Version: "0.1.37", Artifacts: map[string]string{"node-tools:program": "sha256:7c9b10"}, BuiltAt: now}
at := versions.HeldArtifact{Ref: "node-tools:program", Seat: versions.SeatArtifacts, Name: "node-tools/program", Version: "0.1.37", Digest: "sha256:7c9b10"}
other := at
other.Digest = "sha256:83bec1"
for _, c := range []struct {
name string
unheard int
countErr error
judged versions.Judged
wantAsked, wantMark, wantMov bool
}{
{name: "thousands behind: no ask, the registry not read", unheard: 2944},
{name: "at the bound: no ask", unheard: versions.UnheardBound},
{name: "heard up, the registry holds it: marked from its record, no ask", unheard: versions.UnheardBound - 1, judged: versions.Judged{All: true, Held: []versions.HeldArtifact{at}}, wantMark: true},
{name: "the registry holds the number at another digest: moved, no ask", unheard: 3, judged: versions.Judged{Elsewhere: []versions.HeldArtifact{other}}, wantMov: true},
{name: "heard up, the registry lacks it: asked", unheard: 3, judged: versions.Judged{Lacks: "none"}, wantAsked: true},
{name: "the count unreadable, the registry lacks it: asked", countErr: errors.New("no consumer"), judged: versions.Judged{Lacks: "none"}, wantAsked: true},
} {
t.Run(c.name, func(t *testing.T) {
kv := &itemKV{vals: map[string][]byte{}, revs: map[string]uint64{}}
asked, judged, marked, moved := 0, 0, 0, 0
p := &publishItems{reasks: func() jetstream.KeyValue { return kv }, set: defaultPublishSettings, now: func() time.Time { return now }, logf: t.Logf,
reserved: func(context.Context, string, string) (versions.ReservedVersion, bool, error) { return v, true, nil },
republish: func(context.Context, string, string) ([]string, error) { asked++; return nil, nil },
unheard: func(context.Context) (int, error) { return c.unheard, c.countErr },
judge: func(context.Context, versions.ReservedVersion) (versions.Judged, error) {
judged++
return c.judged, nil
},
markHeld: func(_ context.Context, _ versions.ReservedVersion, h []versions.HeldArtifact) {
if len(h) == 1 && h[0].Digest == "sha256:7c9b10" {
marked++
}
},
moveElsewhere: func(context.Context, versions.ReservedVersion, versions.Judged) error {
moved++
delete(kv.vals, versions.PublishItemKey(v.Module, v.Version)) // the takeover ends the old item
return nil
}}
if err := p.Ask(ctx, v.Module, v.Version); err != nil {
t.Fatal(err)
}
key := versions.PublishItemKey(v.Module, v.Version)
e, _ := kv.Get(ctx, key)
p.Try(ctx, kv, e, decodeItem(t, e.Value()))
if (asked == 1) != c.wantAsked || (marked == 1) != c.wantMark || (moved == 1) != c.wantMov {
t.Fatalf("asked %d, marked %d, moved %d; want asked %v, marked %v, moved %v", asked, marked, moved, c.wantAsked, c.wantMark, c.wantMov)
}
if c.wantMov {
if _, ok := kv.vals[key]; ok {
t.Error("the moved version's old item is written again")
}
return
}
got := decodeItem(t, kv.vals[key])
if got.Tries != map[bool]int{true: 1, false: 0}[c.wantAsked] || !got.NextTry.After(now) {
t.Errorf("tries %d, next try %s: want a try spent only when asked, and the item written again with its pause", got.Tries, got.NextTry)
}
if c.unheard >= versions.UnheardBound && judged != 0 {
t.Error("the registry is read while a published may be unheard")
}
})
}
}
@@ -549,8 +549,14 @@ func (m memVersions) Update(_ context.Context, key string, v []byte, _ uint64) (
m[key] = v
return 2, nil
}
func (m memVersions) KeysOf(context.Context, string) ([]string, error) {
return slices.Collect(maps.Keys(m)), nil
func (m memVersions) KeysOf(_ context.Context, filter string) ([]string, error) {
var out []string
for k := range m {
if versions.FilterMatches(filter, k) { // as the bus reads a filter
out = append(out, k)
}
}
return out, nil
}
// memBuilds is the builder's builds in memory.
@@ -33,8 +33,8 @@ func (s awaitedQueue) Bytes(ctx context.Context) (uint64, error) {
// entryVia reads a catalog graph entry by its fingerprint through the seat mesh-catalog's verb entry.
func entryVia(call toolCaller) func(ctx context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
return func(_ context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
raw, err := call("seat:"+meshcatalog.Seat+"."+meshcatalog.VerbEntry, meshcatalog.EntryInput{Fingerprint: fingerprint})
return func(ctx context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
raw, err := callWithin(ctx, call, "seat:"+meshcatalog.Seat+"."+meshcatalog.VerbEntry, meshcatalog.EntryInput{Fingerprint: fingerprint})
if err != nil {
return meshcatalog.CatalogEntry{}, err
}
@@ -50,8 +50,8 @@ func entryVia(call toolCaller) func(ctx context.Context, fingerprint string) (me
// artifact store, addressed by digest. The four seats' publish take one wire form; an answer that lacks parts, or a fault of
// the call, is the holder's refusal, said in its words (not stored when the holder lacks the bytes).
func sendVia(call toolCaller) func(ctx context.Context, p versions.Publish) error {
return func(_ context.Context, p versions.Publish) error {
raw, err := call("seat:"+p.Seat+"."+versions.VerbPublish, publishBody(p))
return func(ctx context.Context, p versions.Publish) error {
raw, err := callWithin(ctx, call, "seat:"+p.Seat+"."+versions.VerbPublish, publishBody(p))
if err != nil {
return fmt.Errorf("%s's publish of %s %s: %w", p.Seat, p.Name, p.Version, err)
}
@@ -66,6 +66,29 @@ func sendVia(call toolCaller) func(ctx context.Context, p versions.Publish) erro
}
}
// callWithin is call within ctx's bound (design 47 H H-A-290 (4)): its answer, or, once the bound ends first, that it was not
// answered within it. The call itself ends by its own bound (the SDK's call timeout), its answer then dropped.
func callWithin(ctx context.Context, call toolCaller, key string, body any) (json.RawMessage, error) {
if err := ctx.Err(); err != nil {
return nil, fmt.Errorf("%s is not called: %w", key, context.Cause(ctx))
}
type answer struct {
raw json.RawMessage
err error
}
done := make(chan answer, 1)
go func() {
raw, err := call(key, body)
done <- answer{raw, err}
}()
select {
case a := <-done:
return a.raw, a.err
case <-ctx.Done():
return nil, fmt.Errorf("%s is not answered within its bound: %w", key, context.Cause(ctx))
}
}
// publishBody is the seat's publish input for p: the digest, its kind and name and version, no parts.
func publishBody(p versions.Publish) gopackageregistry.PublishInput {
return gopackageregistry.PublishInput{Commit: p.Commit, Digest: p.Digest, Fingerprint: p.Fingerprint, Kind: p.Kind, Name: p.Name, Transfer: p.Transfer, Version: p.Version}
@@ -57,6 +57,26 @@ func TestRepublishSend(t *testing.T) {
t.Errorf("%s: asked %q, error %v, want %q", name, asked, err, c.want)
}
}
// A send and its entry read honour the bound they are given (design 47 H H-A-290 (4)): a call still unanswered when it ends
// is said as not answered within it, and the send goes on.
release := make(chan struct{})
defer close(release)
called := make(chan struct{}, 2)
slow := func(string, any) (json.RawMessage, error) {
called <- struct{}{}
<-release
return json.RawMessage(`{}`), nil
}
for name, call := range map[string]func(context.Context) error{
"the publish": func(ctx context.Context) error { return sendVia(slow)(ctx, p) },
"the entry": func(ctx context.Context) error { _, err := entryVia(slow)(ctx, "fp"); return err },
} {
ctx, cancel := context.WithCancel(context.Background())
go func() { <-called; cancel() }()
if err := call(ctx); err == nil || !strings.Contains(err.Error(), "within its bound") {
t.Errorf("%s past its bound answered %v, want it not answered within its bound", name, err)
}
}
if _, _, err := build.NewRepublish("", "0.1.3", "why", "me", versionsNow()); err == nil {
t.Error("a version with no module is accepted")
}
@@ -54,6 +54,8 @@ type reservations struct {
withdraw func(module, fingerprint, why string)
// call asks a seat's verb: the git seat's tag (design 47 G §370).
call toolCaller
// rebuild builds a done key of module at fp again (the build-again path, A-490; H-A-279); nil builds none.
rebuild func(ctx context.Context, module, fp, why string) error
// say says package-published on the seat mesh-builds; nil says none.
say func(context.Context, meshbuilds.PackagePublished) error
}
@@ -77,10 +79,21 @@ func newReservations(store versions.ReserveStore, call toolCaller, bounds func()
}
return out, nil
}}
rv.r.Took = rv.took
rv.r.Took, rv.r.Logf = rv.took, logf
return rv
}
// reindex writes the reservations' index again from the reservations, once at the builder's start (H-A-269 (1), H-A-271), and
// says so.
func (rv *reservations) reindex(ctx context.Context) {
n, err := rv.r.Reindex(ctx)
if err != nil {
rv.logf("the reservations' index is written again from %d reservations, not whole: %v", n, err)
return
}
rv.logf("the reservations' index is written again from %d reservations", n)
}
// took is a build's takeover of the reservation of an older build never released (design 47 G A-455): the older version is
// dropped, its bound ended and its finding cleared (the newer build's reservation has its own), and the builder's log says so.
func (rv *reservations) took(ctx context.Context, older, newer versions.ReservedVersion) {
@@ -117,6 +130,11 @@ func (rv *reservations) ReservedFor(ctx context.Context, module, fingerprint str
return versions.ReservedVersion{}, false, nil
}
// TakenOver is the reservation that took module's series over from the build of fingerprint (H-A-244).
func (rv *reservations) TakenOver(ctx context.Context, module, fingerprint string) (versions.ReservedVersion, bool, error) {
return rv.r.TakenOver(ctx, module, fingerprint)
}
func (rv *reservations) Filled() bool { return rv.filled.Load() }
func (rv *reservations) Reserve(ctx context.Context, module, series string, artifacts []string, src versions.Source) (versions.ReservedVersion, error) {
@@ -260,6 +278,7 @@ func (rv *reservations) heard(ctx context.Context, m jetstream.Msg) bool {
rv.released(ctx, ev.Module, ev.Version)
}
rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, ev.Module)
rv.found.clear(ctx, versions.VersionHeldElsewhere, meshconditionsclient.ScopeModule, ev.Module)
default:
return false
}
@@ -269,19 +288,21 @@ func (rv *reservations) heard(ctx context.Context, m jetstream.Msg) bool {
return true
}
// published marks the artifacts of the reservation a registry seat's published names (its version and digest).
func (rv *reservations) published(ctx context.Context, o versions.Outcome) {
marked, kept, err := rv.r.Published(ctx, o.Name, o.Version, o.Digest)
// published marks the artifacts of the reservation a registry seat's published names (its version and digest). inProgress is told
// before a pass that may read many reservations, so the handler says in progress to the bus (design 47 H H-A-280); nil is told
// nothing.
func (rv *reservations) published(ctx context.Context, o versions.Outcome, inProgress func()) {
out, err := rv.r.PublishedRead(ctx, o.Name, o.Version, o.Digest, inProgress)
if err != nil {
rv.logf("a published of %s@%s is not recorded in the reservations: %v", o.Name, o.Version, err)
}
if len(kept) > 0 {
rv.logf("the published of %s@%s (digest %s, holder %s) is kept on %s until the build records its digests", o.Name, o.Version, o.Digest, o.Holder, strings.Join(kept, ", "))
} else if len(marked) == 0 {
rv.logf("package-published not said: the published of %s@%s (digest %s, holder %s) marks no artifact of a reservation: %s", o.Name, o.Version, o.Digest, o.Holder, rv.noMark(ctx, o))
if len(out.Kept) > 0 {
rv.logf("the published of %s@%s (digest %s, holder %s) is kept on %s until the build records its digests", o.Name, o.Version, o.Digest, o.Holder, strings.Join(out.Kept, ", "))
} else if len(out.Marked) == 0 {
rv.logf("package-published not said: the published of %s@%s (digest %s, holder %s) marks no artifact of a reservation: %s", o.Name, o.Version, o.Digest, o.Holder, rv.noMark(ctx, o, out.Read))
}
// Each artifact the holder's published newly marked is said as the seat mesh-builds' package-published (design 47 G §52, §334).
for _, m := range marked {
for _, m := range out.Marked {
rv.sayPublished(ctx, m, o.Digest)
}
}
@@ -344,27 +365,29 @@ func (rv *reservations) tagged(ctx context.Context, ev meshcatalog.VersionReleas
return nil
}
// noMark says why a published marked nothing: which reservation holds that version, and what it holds of digests.
func (rv *reservations) noMark(ctx context.Context, o versions.Outcome) string {
// noMark says why a published marked nothing, from the reservations the published read (design 47 H H-A-280: no second read):
// what each holds of digests, and how many stand at that version by the index by version (its one key, no reservation read).
func (rv *reservations) noMark(ctx context.Context, o versions.Outcome, read []versions.ReadReservation) string {
version := strings.TrimPrefix(o.Version, "v")
if version == "" || o.Digest == "" {
return "it names no version or no digest (a by-digest publish)"
}
all, err := rv.r.All(ctx)
if err != nil {
return "the reservations cannot be read: " + err.Error()
}
var out []string
for _, v := range all {
for _, r := range read {
v := r.Reserved
if v.Version != version {
continue
}
out = append(out, fmt.Sprintf("%s holds %d built digests %v, published %v (a published heard before the build recorded its digests is kept until it does)", v.Module, len(v.Artifacts), v.Artifacts, v.Published))
out = append(out, fmt.Sprintf("%s holds %d built digests %v, published %v", v.Module, len(v.Artifacts), v.Artifacts, v.Published))
}
standing := "the number standing at it is not read"
if keys, err := rv.r.IndexedAt(ctx, version); err == nil {
standing = fmt.Sprintf("%d reservations stand at it, the built ones not read", len(keys))
}
if len(out) == 0 {
return "no reservation holds version " + version
return "no reservation read holds version " + version + " unbuilt or with that digest (" + standing + ")"
}
return strings.Join(out, "; ")
return strings.Join(out, "; ") + " (" + standing + "; a published heard before the build recorded its digests is kept until it does)"
}
// reserveDoneAct is builder_reserve_done's act (design 47 G A-455): the next version of the series is reserved for a done trunk build
@@ -390,3 +413,49 @@ func reserveDoneAct(reserve func(ctx context.Context, module, fp string) (versio
func (rv *reservations) ReleasedFor(ctx context.Context, module, fingerprint string) (bool, error) {
return rv.r.ReleasedFor(ctx, module, fingerprint)
}
// markHeld marks the artifacts the registries' record holds at their version with the built digest published, as their published
// heard would, and says package-published of each (design 47 H H-A-274): a published heard and lost never leaves the publish
// asked for ever. The version's release follows from the catalog as for any published.
func (rv *reservations) markHeld(ctx context.Context, v versions.ReservedVersion, held []versions.HeldArtifact) {
for _, h := range held {
rv.logf("%s %s: %s holds %s %s with the built digest %s: marked published from its record", v.Module, v.Version, h.Seat, h.Name, h.Version, h.Digest)
rv.published(ctx, versions.Outcome{Name: h.Name, Version: h.Version, Digest: h.Digest, Holder: h.Seat}, nil)
}
}
// heldElsewhere raises version-held-elsewhere about module (design 47 H H-A-275): numbers a registry holds were skipped or moved
// from, said with why.
func (rv *reservations) heldElsewhere(ctx context.Context, module, why string) {
rv.logf("%s: %s", module, why)
rv.found.raise(ctx, finding{kind: versions.VersionHeldElsewhere, scope: meshconditionsclient.ScopeModule, id: module,
summary: module + ": " + why,
headline: "A number a registry already holds is not used",
explanation: "A registry seat holds a version of " + module + " the builder did not publish there: " + why + ".",
resolved: "The catalog says version-released for " + module + ".",
needs: "builder_reserved shows the reservation and its withdrawn numbers; the registry seat's versions shows what it holds."})
}
// movedElsewhere takes a done build whose number a registry holds at another digest off that number (design 47 H H-A-275,
// H-A-279): every artifact carries its version from its build, so no digest survives a move. The number is withdrawn as a gap
// (H-A-226), the finding raised naming both digests, and the build of the same fingerprint asked again (the build-again path of a
// done key whose version is not released, A-490): its reserve takes the next number no registry holds, it records fresh digests
// and its own publish runs; the takeover of the withdrawn reservation ends its publish item and bound.
func (rv *reservations) movedElsewhere(ctx context.Context, v versions.ReservedVersion, j versions.Judged) error {
if rv.rebuild == nil {
return errors.New("no build again is wired")
}
if _, ok, err := rv.r.Withdraw(ctx, v.Module, v.Version); err != nil || !ok {
if err == nil {
err = fmt.Errorf("no reservation of %s stands at %s", v.Module, v.Version)
}
return err
}
var at []string
for _, h := range j.Elsewhere {
at = append(at, fmt.Sprintf("%s holds %s %s at %s, built %s", h.Seat, h.Name, h.Version, h.Digest, v.Artifacts[h.Ref]))
}
why := fmt.Sprintf("%s is withdrawn and the build of %s is built again at the next free number: %s", v.Version, v.Fingerprint, strings.Join(at, "; "))
rv.heldElsewhere(ctx, v.Module, why)
return rv.rebuild(ctx, v.Module, v.Fingerprint, why)
}
@@ -48,7 +48,7 @@ func (m *memReserve) Update(_ context.Context, key string, v []byte, rev uint64)
func (m *memReserve) KeysOf(_ context.Context, filter string) ([]string, error) {
var out []string
for k := range m.vals {
if filter == versions.AllFilter || strings.HasPrefix(k, strings.TrimSuffix(filter, "*")) {
if versions.FilterMatches(filter, k) { // as the bus reads a filter
out = append(out, k)
}
}
@@ -119,10 +119,10 @@ func TestTheCatalogsWordsMoveTheReservations(t *testing.T) {
if !bounds.keys[boundKey] {
t.Fatalf("the publish's bound is not armed: %v", bounds.keys)
}
rv.published(ctx, versions.Outcome{Name: "m-a", Version: "1.4.3", Digest: "sha256:a"})
rv.published(ctx, versions.Outcome{Name: "m-a", Version: "1.4.3", Digest: "sha256:a"}, nil)
// Design 47 G §52, A-57: the holder's published is said once, per artifact, as the seat mesh-builds' package-published with the
// branch and source revision of the reservation; a second published of the same artifact says nothing.
rv.published(ctx, versions.Outcome{Name: "m-a", Version: "1.4.3", Digest: "sha256:a"})
rv.published(ctx, versions.Outcome{Name: "m-a", Version: "1.4.3", Digest: "sha256:a"}, nil)
if len(said) != 1 || said[0].Artifact != "m:a" || said[0].Version != "1.4.3" || said[0].Digest != "sha256:a" || said[0].Branch != "main" || said[0].Source != 41 {
t.Fatalf("package-published said %+v, want one of m:a at 1.4.3 for main at source 41", said)
}
@@ -236,8 +236,8 @@ func TestAPublishedHeardBeforeTheDigestsIsSaidWhenTheBuildRecordsThem(t *testing
if err != nil {
t.Fatal(err)
}
rv.published(ctx, versions.Outcome{Name: "m-a", Version: res.Version, Digest: "sha256:a"})
rv.published(ctx, versions.Outcome{Name: "m-x", Version: res.Version, Digest: "sha256:x"})
rv.published(ctx, versions.Outcome{Name: "m-a", Version: res.Version, Digest: "sha256:a"}, nil)
rv.published(ctx, versions.Outcome{Name: "m-x", Version: res.Version, Digest: "sha256:x"}, nil)
if len(said) != 0 {
t.Fatalf("package-published said before the digests are recorded: %+v", said)
}
@@ -259,7 +259,7 @@ func TestAPublishedHeardBeforeTheDigestsIsSaidWhenTheBuildRecordsThem(t *testing
t.Fatalf("no finding raised for the digest the holder holds: %s", raised)
}
// Taken in once: a published of m:a heard again says nothing more.
rv.published(ctx, versions.Outcome{Name: "m-a", Version: res.Version, Digest: "sha256:a"})
rv.published(ctx, versions.Outcome{Name: "m-a", Version: res.Version, Digest: "sha256:a"}, nil)
if len(said) != 1 {
t.Fatalf("package-published said again: %+v", said)
}
@@ -393,3 +393,142 @@ func TestAReserveDoneIsAnsweredWithTheVersionAndItsQueuedPublish(t *testing.T) {
t.Errorf("the reserving by hand is not said in the log: %q", logged)
}
}
// TestARegistrysRecordEndsOrMovesAPublish: design 47 H H-A-274 — the artifacts a registry's record holds with the built digest are
// marked published from it and package-published is said, as a heard published would; H-A-275 — a done build whose number a
// registry holds at another digest (H-A-279) has its number withdrawn, raises version-held-elsewhere naming both digests, and is
// built again: at the next number no registry holds, with a fresh digest, published there; the old number's item ends.
func TestARegistrysRecordEndsOrMovesAPublish(t *testing.T) {
ctx := context.Background()
fs := &memFindings{vals: map[string][]byte{}}
none := func(string, any) (json.RawMessage, error) {
return json.Marshal(meshcatalog.VersionsAnswer{Module: "node-tools"})
}
bounds := &boundsKV{keys: map[string]bool{}}
rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, none, func() jetstream.KeyValue { return bounds },
&findings{module: builds.BuilderModule, w: fs, logf: t.Logf, now: time.Now}, nil, t.Logf)
var said []meshbuilds.PackagePublished
rv.say = func(_ context.Context, p meshbuilds.PackagePublished) error { said = append(said, p); return nil }
var asked, ended []string
rv.built = func(_ context.Context, m, v string) error { asked = append(asked, m+" "+v); return nil }
rv.released = func(_ context.Context, m, v string) { ended = append(ended, m+" "+v) }
registry := []string{"0.1.37"}
rv.r.HeldNumbers = func(context.Context, string, string, []string) ([]string, error) { return registry, nil }
rv.filled.Store(true)
res, err := rv.Reserve(ctx, "node-tools", "0.1", []string{"node-tools:program"}, versions.Source{Fingerprint: "fp1"})
if err != nil || res.Version != "0.1.0" {
t.Fatalf("reserved %q (%v)", res.Version, err)
}
if err := rv.Built(ctx, "node-tools", res.Version, map[string]string{"node-tools:program": "sha256:7c9b10"}, nil); err != nil {
t.Fatal(err)
}
cur, _, _ := rv.r.Of(ctx, "node-tools", res.Version)
held := versions.HeldArtifact{Ref: "node-tools:program", Seat: versions.SeatArtifacts, Name: "node-tools/program", Version: res.Version, Digest: "sha256:7c9b10"}
rv.markHeld(ctx, cur, []versions.HeldArtifact{held})
if len(said) != 1 || said[0].Artifact != "node-tools:program" || said[0].Version != res.Version {
t.Fatalf("package-published said %+v, want node-tools:program at %s from the registry's record", said, res.Version)
}
if cur, _, _ = rv.r.Of(ctx, "node-tools", res.Version); len(cur.Missing()) != 0 {
t.Errorf("the record marked nothing: missing %v", cur.Missing())
}
// Another reservation of the module, done, whose number the registry holds at another digest: withdrawn, built again at the
// next free number with a fresh digest, and published there (H-A-279).
if _, err := rv.r.Released(ctx, "node-tools", res.Version); err != nil {
t.Fatal(err)
}
asked, ended, said, registry = nil, nil, nil, []string{"0.1.0", "0.1.1", "0.1.2"}
raw, _ := json.Marshal(versions.ReservedVersion{V: 1, Module: "node-tools", Series: "0.1", Version: "0.1.1", Fingerprint: "fp2",
Artifacts: map[string]string{"node-tools:program": "sha256:7c9b10"}, BuiltAt: time.Now()})
if _, err := rv.r.Store.Create(ctx, versions.ReservedKey("node-tools", "0.1"), raw); err != nil {
t.Fatal(err)
}
rv.rebuild = func(ctx context.Context, module, fp, _ string) error {
again, err := rv.Reserve(ctx, module, "0.1", []string{"node-tools:program"}, versions.Source{Fingerprint: fp, Rebuild: true})
if err != nil {
return err
}
if err := rv.Built(ctx, module, again.Version, map[string]string{"node-tools:program": "sha256:fresh"}, nil); err != nil {
return err
}
rv.published(ctx, versions.Outcome{Name: "node-tools/program", Version: again.Version, Digest: "sha256:fresh"}, nil)
return nil
}
cur, _, _ = rv.r.Of(ctx, "node-tools", "0.1.1")
elsewhere := held
elsewhere.Version, elsewhere.Digest = "0.1.1", "sha256:83bec1"
if err := rv.movedElsewhere(ctx, cur, versions.Judged{Elsewhere: []versions.HeldArtifact{elsewhere}}); err != nil {
t.Fatal(err)
}
again, ok, _ := rv.r.Of(ctx, "node-tools", "")
if !ok || again.Version != "0.1.3" || again.Withdrawn["0.1.1"] != "fp2" || again.Artifacts["node-tools:program"] != "sha256:fresh" {
t.Fatalf("built again as %+v, want 0.1.3 with a fresh digest and 0.1.1 withdrawn", again)
}
if len(said) != 1 || said[0].Version != "0.1.3" || said[0].Digest != "sha256:fresh" {
t.Errorf("package-published said %+v, want the fresh build at 0.1.3", said)
}
if len(asked) != 1 || asked[0] != "node-tools 0.1.3" || len(ended) != 1 || ended[0] != "node-tools 0.1.1" {
t.Errorf("asked %v, ended %v: want the new number's publish asked and the old one's item ended", asked, ended)
}
raised := ""
for _, v := range fs.vals {
raised += string(v)
}
if !strings.Contains(raised, versions.VersionHeldElsewhere) || !strings.Contains(raised, "sha256:83bec1") || !strings.Contains(raised, "sha256:7c9b10") {
t.Errorf("no version-held-elsewhere naming both digests: %s", raised)
}
}
// countedReserve counts the reads of reservation keys (two tokens).
type countedReserve struct {
*memReserve
reads int
}
func (c *countedReserve) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
if !strings.HasPrefix(key, "index.") {
c.reads++
}
return c.memReserve.Get(ctx, key)
}
// TestAPublishedThatMarksNothingIsSaidFromWhatItRead: design 47 H H-A-280 — a published whose digest no reservation holds, at a
// version many reservations share, reads only the unbuilt ones there, each once; its journal line is made from those reads with
// no second read; and the handler is told to say in progress before that pass.
func TestAPublishedThatMarksNothingIsSaidFromWhatItRead(t *testing.T) {
ctx := context.Background()
store := &countedReserve{memReserve: &memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}}
none := func(string, any) (json.RawMessage, error) { return json.Marshal(meshcatalog.VersionsAnswer{}) }
var logs []string
rv := newReservations(store, none, func() jetstream.KeyValue { return &boundsKV{keys: map[string]bool{}} },
&findings{module: builds.BuilderModule, w: &memFindings{vals: map[string][]byte{}}, logf: t.Logf, now: time.Now}, nil,
func(f string, a ...any) { logs = append(logs, fmt.Sprintf(f, a...)) })
rv.filled.Store(true)
for i := range 14 {
m := fmt.Sprintf("m%02d", i)
res, err := rv.Reserve(ctx, m, "0.1", []string{m + ":a"}, versions.Source{Fingerprint: "fp" + m})
if err != nil {
t.Fatal(err)
}
if i > 0 { // one stays unbuilt
if err := rv.Built(ctx, m, res.Version, map[string]string{m + ":a": "sha256:" + m}, nil); err != nil {
t.Fatal(err)
}
}
}
store.reads = 0
told := 0
rv.published(ctx, versions.Outcome{Name: "x", Version: "0.1.0", Digest: "sha256:none"}, func() { told++ })
if store.reads != 1 || told != 1 {
t.Errorf("%d reservation reads, in progress told %d times: want the one unbuilt read once, and in progress said before", store.reads, told)
}
if line := strings.Join(logs, "\n"); !strings.Contains(line, "kept on") {
t.Errorf("the journal says %q, want the published kept on the unbuilt reservation", line)
}
logs, store.reads = nil, 0
// A digest of a built one whose artifact is published already marks nothing: the line names it from the one read.
rv.published(ctx, versions.Outcome{Name: "m05-a", Version: "0.1.0", Digest: "sha256:m05"}, nil)
rv.published(ctx, versions.Outcome{Name: "m05-a", Version: "0.1.0", Digest: "sha256:m05"}, nil)
if store.reads != 2 || !strings.Contains(strings.Join(logs, "\n"), "m05 holds 1 built digests") || !strings.Contains(strings.Join(logs, "\n"), "14 reservations stand at it") {
t.Errorf("%d reads for two publisheds, journal %q: want one read each and the line made from it", store.reads, logs)
}
}
@@ -103,7 +103,12 @@ func (h *seatHearer) heard(ctx context.Context, m jetstream.Msg) bool {
h.pub.heard(ctx, se.seat, se.event, o)
}
if h.res != nil && se.event == versions.HeardPublished {
h.res.published(ctx, o)
// A pass over many reservations may outlast a third of the ack wait: the bus is told in progress before it (H-A-280).
h.res.published(ctx, o, func() {
if err := m.InProgress(); err != nil {
h.logf("in progress of %s not said: %v", m.Subject(), err)
}
})
}
if se.event == versions.HeardPublished && o.Kind == versions.KindPackage && h.b != nil {
// A package registry seat's published pins a package an entry names with none, at the version it carries (H-A-192).
@@ -102,7 +102,10 @@ func seed(t *testing.T, st *memStore, b builds.BuilderBuild) string {
func valueOf(t *testing.T, st *memStore, key string) builds.BuilderBuild {
t.Helper()
b, err := builds.DecodeBuilderBuild(st.vals[key])
st.mu.Lock()
raw := st.vals[key]
st.mu.Unlock()
b, err := builds.DecodeBuilderBuild(raw)
if err != nil {
t.Fatal(err)
}
@@ -149,6 +149,9 @@ type Reservations interface {
ReservedFor(ctx context.Context, module, fingerprint string) (versions.ReservedVersion, bool, error)
// ReleasedFor says whether a version was released for the build of fingerprint, read from the builder's own record (A-456a).
ReleasedFor(ctx context.Context, module, fingerprint string) (bool, error)
// TakenOver is the reservation of module's series that took the series over from the build of fingerprint, whose number
// it keeps withdrawn (design 47 H H-A-226), and whether one does (H-A-244).
TakenOver(ctx context.Context, module, fingerprint string) (versions.ReservedVersion, bool, error)
}
// reserves says whether a look at a trunk build is one that reserves: the build has a series of the form MAJOR.MINOR
@@ -29,8 +29,14 @@ type fakeReservations struct {
built map[string]string
sources []versions.Source
deps map[string][]meshbuilds.Dependency
next string // the version Reserve answers when set, else the series' .5
fail error // the reservation cannot be made (the catalog's versions cannot be read)
next string // the version Reserve answers when set, else the series' .5
fail error // the reservation cannot be made (the catalog's versions cannot be read)
tookBy map[string]versions.ReservedVersion // by fingerprint, the reservation that took its series over (H-A-244)
}
func (f *fakeReservations) TakenOver(_ context.Context, _, fp string) (versions.ReservedVersion, bool, error) {
v, ok := f.tookBy[fp]
return v, ok, nil
}
func (f *fakeReservations) ReleasedFor(_ context.Context, _, fp string) (bool, error) {
@@ -354,7 +360,10 @@ func (r *realReserve) Built(ctx context.Context, module, version string, digests
type reserveMap struct{ *memStore }
func (m reserveMap) KeysOf(ctx context.Context, _ string) ([]string, error) { return m.Keys(ctx) }
func (m reserveMap) KeysOf(ctx context.Context, filter string) ([]string, error) {
all, err := m.Keys(ctx)
return slices.DeleteFunc(all, func(k string) bool { return !versions.FilterMatches(filter, k) }), err // as the bus reads a filter
}
func (m reserveMap) Delete(_ context.Context, key string) error {
m.mu.Lock()
defer m.mu.Unlock()
@@ -95,10 +95,14 @@ func theOperatorAnswers(t *testing.T, option string) {
if err != nil || h != occ.Acted {
t.Fatalf("%s: %v, %v", option, h, err)
}
for i := 0; i < 50 && option == OptionBuildOnceMore; i++ { // the build the answer starts runs on its own goroutine
r.b.Wait()
if valueOf(t, r.st, r.key).State == builds.BuildDone {
// The build the answer starts runs on its own goroutine: the builder is waited on only once that build has claimed the key
// (its start counted before its claim, read through the store's lock), never while its start may still be counted.
for i := 0; i < 50 && option == OptionBuildOnceMore; i++ {
if st := valueOf(t, r.st, r.key).State; st == builds.BuildDone {
break
} else if st != builds.BuildFailed {
r.b.Wait()
continue
}
time.Sleep(time.Millisecond)
}
@@ -136,7 +136,8 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins
// passedWithItsVersion is the verdict of a build that ended done (design 47 G A-456): a build that reserves a version (a build
// of the first watched branch's trunk, with a MAJOR.MINOR series and artifacts) passes only when a version is reserved for its
// fingerprint; one that ended done with none fails loudly, naming builder_reserve_done, so no pipeline says passed for a build
// that can never be released. A build that reserves none (a candidate's, a series of before G) passes as it did.
// that can never be released, and one whose series another fingerprint took over fails saying it is stale, naming that
// fingerprint and its version (design 47 H H-A-244). A build that reserves none (a candidate's, a series of before G) passes as it did.
func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.CatalogEntry) (outcome, said string) {
if b.Reserved == nil {
return meshbuilds.StepEndedOutcomePassed, ""
@@ -168,6 +169,12 @@ func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.Ca
} else if released {
return meshbuilds.StepEndedOutcomePassed, "" // a released build counts as reserved (A-456a)
}
if by, took, err := b.Reserved.TakenOver(ctx, entry.Module, entry.Fp); err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the reservations cannot be read: %v", entry.Module, entry.Fp, err)
} else if took {
// H-A-244 (2): the series was taken over by a newer fingerprint, whose number it carries now: the build is stale.
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s: built at %s but its series moved to %s %s: stale", entry.Module, entry.Fp, by.Fingerprint, by.Version)
}
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s: built at %s, but no version is reserved for it; builder_reserve_done reserves one for the done build", entry.Module, entry.Fp)
}
return meshbuilds.StepEndedOutcomePassed, ""
@@ -89,18 +89,23 @@ func TestABuildStepPassesAReservingBuildOnlyWithItsReservedVersion(t *testing.T)
for _, c := range []struct {
name, version string
reserve, released, unreadable bool
takenOver bool
want, said string
}{
{"no version reserved", "0.1", false, false, false, meshbuilds.StepEndedOutcomeFailed, "built at " + fp1 + ", but no version is reserved for it"},
{"a version reserved", "0.1", true, false, false, meshbuilds.StepEndedOutcomePassed, ""},
{"a released build whose reservation is gone", "0.1", false, true, false, meshbuilds.StepEndedOutcomePassed, ""},
{"the plan cannot be read", "0.1", false, false, true, meshbuilds.StepEndedOutcomeCouldNotRun, "plan cannot be read"},
{"a series of before G", "1", false, false, false, meshbuilds.StepEndedOutcomePassed, ""},
{"no version reserved", "0.1", false, false, false, false, meshbuilds.StepEndedOutcomeFailed, "built at " + fp1 + ", but no version is reserved for it"},
{"H-A-244: its series was taken over by a newer fingerprint", "0.1", false, false, false, true, meshbuilds.StepEndedOutcomeFailed, "m: built at " + fp1 + " but its series moved to " + fp2 + " 0.1.7: stale"},
{"a version reserved", "0.1", true, false, false, false, meshbuilds.StepEndedOutcomePassed, ""},
{"a released build whose reservation is gone", "0.1", false, true, false, false, meshbuilds.StepEndedOutcomePassed, ""},
{"the plan cannot be read", "0.1", false, false, true, false, meshbuilds.StepEndedOutcomeCouldNotRun, "plan cannot be read"},
{"a series of before G", "1", false, false, false, false, meshbuilds.StepEndedOutcomePassed, ""},
} {
t.Run(c.name, func(t *testing.T) {
r := newRig(t, goBundle)
r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: c.version, Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
res := &fakeReservations{filled: true, released: map[string]bool{fp1: c.released}}
if c.takenOver {
res.tookBy = map[string]versions.ReservedVersion{fp1: {Module: "m", Series: "0.1", Version: "0.1.7", Fingerprint: fp2}}
}
if v := r.consider(t, "modules/m"); v.State != builds.BuildDone { // built while the builder reserved nothing, as the live key was
t.Fatalf("build = %+v, want done", v)
}
@@ -112,7 +117,7 @@ func TestABuildStepPassesAReservingBuildOnlyWithItsReservedVersion(t *testing.T)
return meshcatalog.CatalogEntry{Module: "m", Repository: "novox/m", Commit: "c1", Fp: fp1, ManifestDigest: digestOf(c.unreadable), Path: "modules/m"}, nil
}
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.1.m", Step: meshbuilds.StepRunStepBuild, Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
if end.Outcome != c.want || !strings.Contains(end.Said, c.said) || (c.want == meshbuilds.StepEndedOutcomeFailed && !strings.Contains(end.Said, "builder_reserve_done")) {
if end.Outcome != c.want || !strings.Contains(end.Said, c.said) || (c.want == meshbuilds.StepEndedOutcomeFailed && strings.Contains(end.Said, "builder_reserve_done") == c.takenOver) {
t.Errorf("end = %+v, want %s saying %q", end, c.want, c.said)
}
})
@@ -0,0 +1,205 @@
package versions
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
)
// **The reservations' index by digest and by version** (design 47 H H-A-269 (1), H-A-271): beside the reservations, in the same
// state, the builder keeps which reservation keys hold a built digest and which stand at a version, so a registry seat's published
// finds its reservation in one read and a published whose digest no reservation holds is judged from its version's keys in one
// more, never a scan of every reservation. The index keys have three tokens, so the reservations' filter (two tokens) never lists
// them. It is maintained at reserve, at a digest recorded, at release and at drop; an entry left behind by a write that ended
// halfway is harmless (every read checks the reservation it names), and a missing one is written again by Reindex at the start.
const (
indexDigestPrefix = "index.digest."
indexVersionPrefix = "index.version."
indexUnbuiltPrefix = "index.unbuilt."
// IndexDigestFilter and IndexVersionFilter are the filters of the index's two halves.
IndexDigestFilter = indexDigestPrefix + "*"
IndexVersionFilter = indexVersionPrefix + "*"
// IndexUnbuiltFilter is the filter of the reservations at a version whose build has not recorded every digest (H-A-280).
IndexUnbuiltFilter = indexUnbuiltPrefix + "*"
)
// IndexDigestKey is the index key of the reservations holding digest as a built artifact's.
func IndexDigestKey(digest string) string { return indexDigestPrefix + EscapeToken(digest) }
// IndexVersionKey is the index key of the reservations standing at version (MAJOR.MINOR.PATCH).
func IndexVersionKey(version string) string { return indexVersionPrefix + EscapeToken(version) }
// IndexUnbuiltKey is the index key of the reservations at version whose build has not recorded every digest (design 47 H
// H-A-280): the only ones a published whose digest no reservation holds is kept on (A-415), kept apart from the built.
func IndexUnbuiltKey(version string) string { return indexUnbuiltPrefix + EscapeToken(version) }
// IndexEntry is an index key's value: the reservation keys, sorted.
type IndexEntry struct {
Keys []string `json:"keys"`
}
// indexed reads an index key's reservation keys.
func (r *Reserver) indexed(ctx context.Context, key string) ([]string, error) {
raw, _, found, err := r.Store.Get(ctx, key)
if err != nil || !found {
return nil, err
}
var e IndexEntry
if err := json.Unmarshal(raw, &e); err != nil {
return nil, fmt.Errorf("%s holds what is no index entry: %w", key, err)
}
return e.Keys, nil
}
// index adds (or, with add false, removes) reserved under the index key key, by compare-and-set; an entry left empty is deleted.
func (r *Reserver) index(ctx context.Context, key, reserved string, add bool) error {
for {
raw, rev, found, err := r.Store.Get(ctx, key)
if err != nil {
return err
}
var e IndexEntry
if found {
_ = json.Unmarshal(raw, &e)
}
has := slices.Contains(e.Keys, reserved)
if has == add {
return nil
}
if add {
e.Keys = append(e.Keys, reserved)
sort.Strings(e.Keys)
} else {
e.Keys = slices.DeleteFunc(e.Keys, func(k string) bool { return k == reserved })
}
if len(e.Keys) == 0 {
return r.Store.Delete(ctx, key)
}
out, _ := json.Marshal(e)
if found {
_, err = r.Store.Update(ctx, key, out, rev)
} else {
_, err = r.Store.Create(ctx, key, out)
}
if errors.Is(err, ErrConflict) {
continue
}
return err
}
}
// indexKeys are the index keys a reservation stands under: its version's and each built digest's.
func indexKeys(v ReservedVersion) []string {
out := []string{}
if v.Version != "" {
out = append(out, IndexVersionKey(v.Version))
if v.BuiltAt.IsZero() {
out = append(out, IndexUnbuiltKey(v.Version))
}
}
for _, d := range v.Artifacts {
if d != "" && !slices.Contains(out, IndexDigestKey(d)) {
out = append(out, IndexDigestKey(d))
}
}
return out
}
// reindex moves reservation key from the index keys of was (its value before, zero when there was none) to those of now (zero when
// it is gone).
func (r *Reserver) reindex(ctx context.Context, key string, was, now ReservedVersion) error {
add, drop := indexKeys(now), indexKeys(was)
var errs []error
for _, k := range add {
if err := r.index(ctx, k, key, true); err != nil {
errs = append(errs, err)
}
}
for _, k := range drop {
if !slices.Contains(add, k) {
if err := r.index(ctx, k, key, false); err != nil {
errs = append(errs, err)
}
}
}
if err := errors.Join(errs...); err != nil {
return fmt.Errorf("the index of %s is not written (Reindex writes it at the start): %w", key, err)
}
return nil
}
// IndexedAt are the keys of the reservations standing at version, by the index by version alone (H-A-280): no reservation read.
func (r *Reserver) IndexedAt(ctx context.Context, version string) ([]string, error) {
return r.indexed(ctx, IndexVersionKey(version))
}
// AtVersion are the reservations standing at version, read through the index by version (H-A-271): one read of the index and
// one of each reservation it names.
func (r *Reserver) AtVersion(ctx context.Context, version string) ([]ReservedVersion, error) {
keys, err := r.indexed(ctx, IndexVersionKey(version))
if err != nil {
return nil, err
}
var out []ReservedVersion
for _, k := range keys {
cur, _, found, err := r.get(ctx, k)
if err != nil {
return out, err
}
if found && cur.Version == version {
out = append(out, cur)
}
}
return out, nil
}
// Reindex writes the index again from the reservations, once at the builder's start (H-A-269 (1), H-A-271): each reservation is
// added under its keys, and a reservation an index key names that does not stand under it (read again) is removed from it. Both
// are the same compare-and-set as every other index write, so a reservation written meanwhile keeps its entries. It answers how
// many reservations it read.
func (r *Reserver) Reindex(ctx context.Context) (int, error) {
all, err := r.All(ctx)
if err != nil {
return 0, err
}
var errs []error
for _, v := range all {
key := ReservedKey(v.Module, v.Series)
for _, k := range indexKeys(v) {
if err := r.index(ctx, k, key, true); err != nil {
errs = append(errs, err)
}
}
}
for _, filter := range []string{IndexDigestFilter, IndexVersionFilter, IndexUnbuiltFilter} {
keys, err := r.Store.KeysOf(ctx, filter)
if err != nil {
return len(all), fmt.Errorf("the index cannot be listed: %w", err)
}
for _, k := range keys {
named, err := r.indexed(ctx, k)
if err != nil {
errs = append(errs, err)
continue
}
for _, rk := range named {
cur, _, found, err := r.get(ctx, rk)
if err != nil {
errs = append(errs, err)
continue
}
if found && slices.Contains(indexKeys(cur), k) {
continue
}
if err := r.index(ctx, k, rk, false); err != nil {
errs = append(errs, err)
}
}
}
}
return len(all), errors.Join(errs...)
}
@@ -0,0 +1,144 @@
package versions
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
)
// countingReserve counts the listings of the reservations and the reads of a reservation key (two tokens), so a test sees that
// a published reads its reservation and no other.
type countingReserve struct {
*reserveMem
lists, reads atomic.Int64
}
func (c *countingReserve) KeysOf(ctx context.Context, filter string) ([]string, error) {
c.lists.Add(1)
return c.reserveMem.KeysOf(ctx, filter)
}
func (c *countingReserve) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
if !strings.HasPrefix(key, "index.") {
c.reads.Add(1)
}
return c.reserveMem.Get(ctx, key)
}
// TestTheIndexFindsAPublishedReservationInOneRead: design 47 H H-A-269 (1), H-A-271 — a registry seat's published finds its
// reservation through the index by digest, reading that reservation alone and listing none; a published whose digest no
// reservation holds is judged from its version's keys (kept on a build that recorded no digest yet, A-415), never a scan; the
// index goes with a release and a drop, and is written again at the start from reservations written before it existed.
func TestTheIndexFindsAPublishedReservationInOneRead(t *testing.T) {
ctx := context.Background()
fill := func(t *testing.T) (*Reserver, *countingReserve) {
store := &countingReserve{reserveMem: &reserveMem{mem: newMem()}}
r := &Reserver{Store: store, ReleasedOf: func(context.Context, string) ([]string, error) { return nil, nil }}
for i := range 40 {
m := fmt.Sprintf("m%02d", i)
if _, err := r.Reserve(ctx, m, "0.1", []string{m + ":a"}, Source{Fingerprint: "fp" + m}); err != nil {
t.Fatal(err)
}
if i%2 == 0 { // half are built, each with its own digest
if _, err := r.Built(ctx, m, "0.1.0", m+":a", "sha256:"+m, nil); err != nil {
t.Fatal(err)
}
}
}
store.lists.Store(0)
store.reads.Store(0)
return r, store
}
t.Run("a published by digest reads its one reservation", func(t *testing.T) {
r, store := fill(t)
marked, _, err := r.Published(ctx, "m10-a", "v0.1.0", "sha256:m10")
if err != nil || len(marked) != 1 || marked[0].Reserved.Module != "m10" || marked[0].Artifact != "m10:a" {
t.Fatalf("the published marked %+v (%v), want m10:a", marked, err)
}
if l, n := store.lists.Load(), store.reads.Load(); l != 0 || n != 1 {
t.Errorf("the published listed %d times and read %d reservations, want none listed and one read", l, n)
}
})
t.Run("a digest no reservation holds is judged from its version's unbuilt keys", func(t *testing.T) {
r, store := fill(t)
marked, kept, err := r.Published(ctx, "m11-a", "0.1.0", "sha256:early")
if err != nil || len(marked) != 0 {
t.Fatalf("an unbuilt digest marked %v (%v)", marked, err)
}
// The 20 unbuilt reservations at 0.1.0 keep it as heard (A-415); the 20 built ones are not read at all (H-A-280).
if len(kept) != 20 || store.lists.Load() != 0 || store.reads.Load() != 20 {
t.Errorf("kept on %d (want 20), listed %d (want 0), read %d (want only the 20 unbuilt at 0.1.0)", len(kept), store.lists.Load(), store.reads.Load())
}
at, err := r.AtVersion(ctx, "0.9.9")
if err != nil || len(at) != 0 || store.lists.Load() != 0 {
t.Errorf("a version no reservation stands at answered %v (%v), listed %d", at, err, store.lists.Load())
}
})
t.Run("a published at a version many share reads each reservation once and hands them on", func(t *testing.T) {
r, store := fill(t)
// A stale digest entry naming unbuilt reservations too: the second pass takes them as the first read them.
stale, _ := json.Marshal(IndexEntry{Keys: []string{ReservedKey("m01", "0.1"), ReservedKey("m03", "0.1")}})
if _, err := store.Create(ctx, IndexDigestKey("sha256:stale"), stale); err != nil {
t.Fatal(err)
}
slow := 0
out, err := r.PublishedRead(ctx, "x", "0.1.0", "sha256:stale", func() { slow++ })
if err != nil || len(out.Kept) != 20 || len(out.Read) != 20 || slow != 1 {
t.Fatalf("kept %d, read %d, told slow %d (%v): want 20 kept, 20 read and handed on, slow told once", len(out.Kept), len(out.Read), slow, err)
}
if n := store.reads.Load(); n != 20 {
t.Errorf("%d reservation reads, want each of the 20 read once", n)
}
})
t.Run("a release and a drop take the reservation out of the index", func(t *testing.T) {
r, store := fill(t)
if _, err := r.Released(ctx, "m10", "0.1.0"); err != nil {
t.Fatal(err)
}
if _, err := r.Drop(ctx, "m12", "0.1"); err != nil {
t.Fatal(err)
}
for _, k := range []string{IndexDigestKey("sha256:m10"), IndexDigestKey("sha256:m12")} {
if _, _, found, _ := store.Get(ctx, k); found {
t.Errorf("%s stands after its reservation went", k)
}
}
keys, _ := r.indexed(ctx, IndexVersionKey("0.1.0"))
for _, k := range keys {
if k == ReservedKey("m10", "0.1") || k == ReservedKey("m12", "0.1") {
t.Errorf("the version's index still names %s", k)
}
}
})
t.Run("the index is written again at the start from the reservations", func(t *testing.T) {
store := &countingReserve{reserveMem: &reserveMem{mem: newMem()}}
r := &Reserver{Store: store}
// A reservation written before the index existed, and an index entry naming a reservation that is gone.
raw, _ := json.Marshal(ReservedVersion{V: 1, Module: "old", Series: "0.2", Version: "0.2.4", Artifacts: map[string]string{"old:a": "sha256:old"}})
if _, err := store.Create(ctx, ReservedKey("old", "0.2"), raw); err != nil {
t.Fatal(err)
}
stale, _ := json.Marshal(IndexEntry{Keys: []string{ReservedKey("gone", "0.1")}})
if _, err := store.Create(ctx, IndexDigestKey("sha256:gone"), stale); err != nil {
t.Fatal(err)
}
if n, err := r.Reindex(ctx); err != nil || n != 1 {
t.Fatalf("the reindex read %d reservations (%v), want 1", n, err)
}
if _, _, found, _ := store.Get(ctx, IndexDigestKey("sha256:gone")); found {
t.Error("the index entry of a reservation that is gone stands after the reindex")
}
if keys, _ := store.KeysOf(ctx, AllFilter); len(keys) != 1 {
t.Errorf("the reservations list %v: the index keys are listed among them", keys)
}
store.lists.Store(0)
store.reads.Store(0)
marked, _, err := r.Published(ctx, "old-a", "0.2.4", "sha256:old")
if err != nil || len(marked) != 1 || store.lists.Load() != 0 || store.reads.Load() != 1 {
t.Errorf("after the reindex the published marked %v (%v), listed %d, read %d", marked, err, store.lists.Load(), store.reads.Load())
}
})
}
@@ -5,7 +5,9 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -33,8 +35,19 @@ const (
QueueFull = "queue full"
// NotStoredWords opens what a refused artifact is said as.
NotStoredWords = "not stored: "
// VerbPublish is the registry seats' verb a publish again sends.
VerbPublish = "publish"
// VerbPublish is the registry seats' verb a publish again sends, VerbVersions the one whose record says what a seat holds
// (H-A-272).
VerbPublish = "publish"
VerbVersions = "versions"
// PublishClassLive is the class of a publish whose build ended in this run, which a pipeline's build step waits on;
// PublishClassStart that of a publish asked again at a start, of a build that ended before it. A holder's queue sends every
// live one before any of the start, each class oldest first (design 47 H H-A-290 (1)).
PublishClassLive = "build-ended-this-run"
PublishClassStart = "asked-again-at-start"
// PublishSendsAtOnce bounds the sends of one holder's queue that run at once (H-A-290 (3)).
PublishSendsAtOnce = 4
// PublishQueueListCap caps the queued versions and the publish items builder_publish_queue lists (H-A-290 (5)).
PublishQueueListCap = 50
)
// RepublishTransfer is a publish again's transfer name, the builder's own (the registry part's publish: the caller names its
@@ -75,8 +88,31 @@ type QueuedPublish struct {
Artifacts map[string]string `json:"artifacts"`
Queued time.Time `json:"queued"`
Begun bool `json:"begun,omitempty"`
// Class is PublishClassLive or PublishClassStart (H-A-290 (1)); a value written before it had one is of the start.
Class string `json:"class,omitempty"`
}
// rank is the place of q's class in a holder's queue: the live first.
func (q QueuedPublish) rank() int {
if q.Class == PublishClassLive {
return 0
}
return 1
}
// sortQueued sorts a holder's queued versions in the order they are sent: the live class first, each class oldest first.
func sortQueued(qs []QueuedPublish) {
sort.SliceStable(qs, func(i, j int) bool {
if a, b := qs[i].rank(), qs[j].rank(); a != b {
return a < b
}
return qs[i].Queued.Before(qs[j].Queued)
})
}
// QueueFilter is the filter of every queued version: a queue key has three tokens, which AllFilter (two) never lists on the bus.
const QueueFilter = "*.*.*"
// QueueKey is the key of holder's queued version of module.
func QueueKey(holder, module, version string) string {
return EscapeToken(holder) + "." + EscapeToken(module) + "." + EscapeToken(version)
@@ -115,9 +151,18 @@ type Republisher struct {
// Holders answers a registry seat's holders: the catalog's SeatHolders, the part's one holder function that its FollowSeat
// of the assignments takes too, read from the builder's start (design 47 G A-416, A-416a); nil answers none.
Holders func(seat string) []string
// Started is when this run of the builder started: a version whose build ended at or after it is of the live class, one
// that ended before it of the start's (H-A-290 (1)). Zero makes every version live.
Started time.Time
// Ended ends the publish item of a version that never publishes, its reservation withdrawn or taken over (H-A-290 (2));
// nil ends none.
Ended func(ctx context.Context, module, version string)
mu sync.Mutex
busy map[string]bool
// relist marks a holder whose pass lists again before its next start-class send: a live publish was queued meanwhile
// (H-A-291 (2)).
relist map[string]bool
// lastQueued is the newest time a version was queued at: a clock that has not moved between two versions (a coarse clock,
// or a fixed one) must not make the queue's order the order of its keys, so each is queued after the last.
lastQueued time.Time
@@ -149,6 +194,127 @@ func (r *Republisher) logf(f string, a ...any) {
}
}
// classOf is the class of v's publish: live when its build ended in this run (H-A-290 (1)).
func (r *Republisher) classOf(v ReservedVersion) string {
if v.BuiltAt.Before(r.Started) {
return PublishClassStart
}
return PublishClassLive
}
// NotStanding says why the reservation of a version, as Of answers it (found), never publishes: it stands no more at the
// version (released, dropped or taken over by a newer build), or it was withdrawn (design 47 H H-A-225, H-A-279). Empty when it
// stands (H-A-290 (2)).
func NotStanding(v ReservedVersion, found bool) string {
switch {
case !found:
return "its reservation stands no more at the version (released, dropped or taken over)"
case hasKey(v.Withdrawn, v.Version):
return "its reservation was withdrawn"
}
return ""
}
// newerVersion says whether MAJOR.MINOR.PATCH a is above b, number by number.
func newerVersion(a, b string) bool {
pa, pb := strings.Split(a, "."), strings.Split(b, ".")
if len(pa) != 3 || len(pb) != 3 {
return false
}
for i := range 3 {
x, errx := strconv.Atoi(pa[i])
y, erry := strconv.Atoi(pb[i])
if errx != nil || erry != nil {
return false
}
if x != y {
return x > y
}
}
return false
}
// Retired are the done builds without a release among all that a newer reservation or a newer release of the same module and
// series (MAJOR.MINOR) supersedes, each with why (design 47 H H-A-291 (1), H-A-293; H-A-275: the older digest is dead): keyed
// `<module> <version>`. A standing newer series never retires an older one's build. The releases are the builder's own record
// of them (Releases), read once per module.
func (r *Reserver) Retired(ctx context.Context, all []ReservedVersion) (map[string]string, error) {
seriesOf := func(v ReservedVersion) string {
s, _, err := SplitVersion(v.Version)
if err != nil {
return ""
}
return v.Module + " " + s
}
newest := map[string]string{} // module and series -> its newest reserved version
for _, v := range all {
k := seriesOf(v)
if n, ok := newest[k]; k != "" && (!ok || newerVersion(v.Version, n)) {
newest[k] = v.Version
}
}
kept := map[string][]string{} // module -> the releases the record keeps
read := map[string]bool{}
out := map[string]string{}
for _, v := range all {
k := seriesOf(v)
if v.BuiltAt.IsZero() || k == "" {
continue
}
if n := newest[k]; newerVersion(n, v.Version) {
out[v.Module+" "+v.Version] = "the newer reservation " + n + " of " + v.Module + " supersedes it"
continue
}
if !read[v.Module] && r.Releases != nil {
read[v.Module] = true
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(v.Module))
if err != nil {
return out, fmt.Errorf("the releases of %s cannot be read: %w", v.Module, err)
}
var rec ReleasedRecord
if found && json.Unmarshal(raw, &rec) == nil {
for _, rb := range rec.Kept {
kept[v.Module] = append(kept[v.Module], rb.Version)
}
}
}
series, _, _ := SplitVersion(v.Version)
rel := ""
for _, kv := range kept[v.Module] {
if s, _, err := SplitVersion(kv); err == nil && s == series && newerVersion(kv, v.Version) && (rel == "" || newerVersion(kv, rel)) {
rel = kv
}
}
if rel != "" {
out[v.Module+" "+v.Version] = "the newer release " + rel + " of " + v.Module + " supersedes it"
}
}
return out, nil
}
// drop deletes every queued key of module's version and ends its publish item: it never publishes (H-A-290 (2)), said with why.
func (r *Republisher) drop(ctx context.Context, module, version, why string) {
ctx = context.WithoutCancel(ctx)
keys, err := r.Queue.KeysOf(ctx, "*."+EscapeToken(module)+"."+EscapeToken(version))
if err != nil {
r.logf("publish again: %s %s never publishes (%s), and its queued keys cannot be listed: %v", module, version, why, err)
}
for _, k := range keys {
if err := r.Queue.Delete(ctx, k); err != nil {
r.logf("%s cannot be deleted from %s: %v", k, PublishQueueState, err)
}
}
r.end(ctx, module, version, why)
}
// end ends module's version's publish item, said with why.
func (r *Republisher) end(ctx context.Context, module, version, why string) {
r.logf("publish again: %s %s is not published: %s; its queued key is dropped and its publish item ended", module, version, why)
if r.Ended != nil {
r.Ended(context.WithoutCancel(ctx), module, version)
}
}
// ActedKey is the acted key of a subject.
func ActedKey(subject string) string { return EscapeToken(subject) }
@@ -183,6 +349,10 @@ func (r *Republisher) queue(ctx context.Context, vs []ReservedVersion, holdersOf
if max > 0 && len(queued) >= max {
break
}
if why := NotStanding(v, true); why != "" {
r.drop(ctx, v.Module, v.Version, why)
continue
}
if v.Fingerprint == "" || v.BuiltAt.IsZero() {
r.logf("publish again: %s %s is not queued: it holds no fingerprint or its build has not ended with every digest", v.Module, v.Version)
continue
@@ -245,7 +415,7 @@ func (r *Republisher) enqueue(ctx context.Context, v ReservedVersion, holder str
r.logf("publish again: %s %s is queued at %s already", v.Module, v.Version, holder)
return false, nil
}
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: holder, Module: v.Module, Version: v.Version, Artifacts: arts, Queued: r.queuedAt()})
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: holder, Module: v.Module, Version: v.Version, Artifacts: arts, Queued: r.queuedAt(), Class: r.classOf(v)})
used, err := r.Queue.Bytes(ctx)
if err != nil {
return false, fmt.Errorf("the size of the state %s cannot be read: %w", PublishQueueState, err)
@@ -262,7 +432,15 @@ func (r *Republisher) enqueue(ctx context.Context, v ReservedVersion, holder str
}
return false, fmt.Errorf("%s cannot be queued: %w", key, err)
}
r.logf("publish again: %s %s is queued to publish at %s: %d artifacts", v.Module, v.Version, holder, len(arts))
r.logf("publish again: %s %s is queued to publish at %s: %d artifacts, class %s", v.Module, v.Version, holder, len(arts), r.classOf(v))
if r.classOf(v) == PublishClassLive { // a pass sending a start list lists again, so this one goes next (H-A-291 (2))
r.mu.Lock()
if r.relist == nil {
r.relist = map[string]bool{}
}
r.relist[holder] = true
r.mu.Unlock()
}
return true, nil
}
@@ -299,13 +477,43 @@ func (r *Republisher) Republish(ctx context.Context, module, version string) ([]
return r.queue(ctx, vs, r.holdersFn, 0)
}
// Queued are the queued versions, per holder in the order they are sent, and the acted keys with their sequences:
// builder_publish_queue.
// Queued is builder_publish_queue's answer, small at any size of the queue (design 47 H H-A-290 (5)): the queued versions'
// count per holder and class with the oldest and newest queue time, the first PublishQueueListCap of them in the order they are
// sent (each with its class), the acted keys with their sequences, and the publish items, counted and capped alike.
type Queued struct {
Queue []QueuedPublish `json:"queue"`
Acted map[string]uint64 `json:"acted"`
// Items are the publish items (design 47 G A-457a), each with its tries and its next try; an item that ended says why.
Items []PublishItem `json:"items"`
Total int `json:"total"`
Counts []QueueCount `json:"counts"`
// Queue are the first queued versions, per holder in the order they are sent (the live class first, each class oldest first);
// with a module named, only that module's.
Queue []QueuedPublish `json:"queue"`
// Listed says how many of Total the answer lists.
Listed int `json:"listed"`
Acted map[string]uint64 `json:"acted"`
// ItemCount is how many publish items stand; Items the first of them by their next try (design 47 G A-457a), each with its
// tries and its next try; an item that ended says why.
ItemCount int `json:"item-count"`
Items []PublishItem `json:"items"`
// Capped says a state held more than the one page read of it (H-A-293); Read names each such state, the keys read and its
// size. Every count above is then of the keys read.
Capped bool `json:"capped,omitempty"`
Read []StateCapped `json:"read,omitempty"`
}
// capped notes a state read only in part.
func (q *Queued) capped(c *StateCapped) {
if c != nil {
q.Capped = true
q.Read = append(q.Read, *c)
}
}
// QueueCount is how many versions one holder's queue holds of one class, and when the oldest and the newest were queued.
type QueueCount struct {
Holder string `json:"holder"`
Class string `json:"class"`
Count int `json:"count"`
Oldest time.Time `json:"oldest"`
Newest time.Time `json:"newest"`
}
// PublishItemKeyPrefix opens the key of a publish item in the builder's re-ask state.
@@ -329,69 +537,113 @@ func PublishItemKey(module, version string) string {
return PublishItemKeyPrefix + EscapeToken(module) + "." + EscapeToken(version)
}
// AnswerQueue is builder_publish_queue over the two sections as the tools bundle reads them.
func AnswerQueue(queue, acted, reasks Reader) (Queued, error) {
out := Queued{Queue: []QueuedPublish{}, Acted: map[string]uint64{}, Items: []PublishItem{}}
keys, err := queue.Keys()
// AnswerQueue is builder_publish_queue over the three states as the tools bundle reads them (H-A-290 (5)): each state read in
// bulk (Reader.All), the answer counted and capped at PublishQueueListCap whatever the queue holds; module (and version) narrow the
// listed queue to one module's (one version's), so an item's class can be read.
func AnswerQueue(queue, acted, reasks Reader, module, version string) (Queued, error) {
out := Queued{Counts: []QueueCount{}, Queue: []QueuedPublish{}, Acted: map[string]uint64{}, Items: []PublishItem{}}
vals, c, err := queue.values(PublishQueueState)
out.capped(c)
if err != nil {
return out, fmt.Errorf("the state %s cannot be listed: %w", PublishQueueState, err)
return out, fmt.Errorf("the state %s cannot be read: %w", PublishQueueState, err)
}
for _, k := range keys {
raw, ok, err := queue.Get(k)
if err != nil || !ok {
counts := map[[2]string]*QueueCount{}
var listed []QueuedPublish
for _, raw := range vals {
var q QueuedPublish
if json.Unmarshal(raw, &q) != nil {
continue
}
var q QueuedPublish
if json.Unmarshal(raw, &q) == nil {
if q.Class == "" {
q.Class = PublishClassStart
}
out.Total++
c := counts[[2]string{q.Holder, q.Class}]
if c == nil {
c = &QueueCount{Holder: q.Holder, Class: q.Class, Oldest: q.Queued, Newest: q.Queued}
counts[[2]string{q.Holder, q.Class}] = c
}
c.Count++
if q.Queued.Before(c.Oldest) {
c.Oldest = q.Queued
}
if q.Queued.After(c.Newest) {
c.Newest = q.Queued
}
if (module == "" || q.Module == module) && (version == "" || q.Version == version) {
listed = append(listed, q)
}
}
for _, c := range counts {
out.Counts = append(out.Counts, *c)
}
sort.Slice(out.Counts, func(i, j int) bool {
if out.Counts[i].Holder != out.Counts[j].Holder {
return out.Counts[i].Holder < out.Counts[j].Holder
}
return out.Counts[i].Class < out.Counts[j].Class
})
sort.SliceStable(listed, func(i, j int) bool { return listed[i].Holder < listed[j].Holder })
byHolder := map[string][]QueuedPublish{}
var holders []string
for _, q := range listed {
if byHolder[q.Holder] == nil {
holders = append(holders, q.Holder)
}
byHolder[q.Holder] = append(byHolder[q.Holder], q)
}
for _, h := range holders {
qs := byHolder[h]
sortQueued(qs)
for _, q := range qs {
if len(out.Queue) == PublishQueueListCap {
break
}
out.Queue = append(out.Queue, q)
}
}
sort.SliceStable(out.Queue, func(i, j int) bool {
if out.Queue[i].Holder != out.Queue[j].Holder {
return out.Queue[i].Holder < out.Queue[j].Holder
}
return out.Queue[i].Queued.Before(out.Queue[j].Queued)
})
akeys, err := acted.Keys()
out.Listed = len(out.Queue)
avals, c, err := acted.values(ActedState)
out.capped(c)
if err != nil {
return out, fmt.Errorf("the state %s cannot be listed: %w", ActedState, err)
return out, fmt.Errorf("the state %s cannot be read: %w", ActedState, err)
}
for _, k := range akeys {
raw, ok, err := acted.Get(k)
if err != nil || !ok {
continue
}
for k, raw := range avals {
var a Acted
if json.Unmarshal(raw, &a) == nil {
out.Acted[k] = a.Sequence
}
}
rkeys, err := reasks.Keys()
rvals, c, err := reasks.values("reasks")
out.capped(c)
if err != nil {
return out, fmt.Errorf("the re-ask state cannot be listed: %w", err)
return out, fmt.Errorf("the re-ask state cannot be read: %w", err)
}
for _, k := range rkeys {
for k, raw := range rvals {
if !strings.HasPrefix(k, PublishItemKeyPrefix) {
continue
}
raw, ok, err := reasks.Get(k)
if err != nil || !ok {
continue
}
var it PublishItem
if json.Unmarshal(raw, &it) == nil {
out.Items = append(out.Items, it)
}
}
out.ItemCount = len(out.Items)
sort.SliceStable(out.Items, func(i, j int) bool { return out.Items[i].NextTry.Before(out.Items[j].NextTry) })
if len(out.Items) > PublishQueueListCap {
out.Items = out.Items[:PublishQueueListCap]
}
return out, nil
}
// PublishQueueDescription is builder_publish_queue's description.
const PublishQueueDescription = "The versions the builder will publish again (its state " + PublishQueueState + ", design 47 G section 588): per registry " +
"seat holder, in the order they are sent, each with its artifacts' digests, when it was queued and whether its publish has begun; and the " +
"keys of the state " + ActedState + ", the last stream sequence the builder acted on for each subject; and the publish items (design 47 G " +
"A-457a), each version's tries and next try, and why one ended after its tries. Read-only."
const PublishQueueDescription = "The versions the builder will publish again (its state " + PublishQueueState + ", design 47 G section 588, H H-A-290): " +
"how many each registry seat holder's queue holds per class (build-ended-this-run, sent first, or asked-again-at-start) with the oldest and newest " +
"queue time; the first 50 in the order they are sent, each with its class, its artifacts' digests, when it was queued and whether its publish has " +
"begun (module, and version, list only that module's, so one item's class can be read); the keys of the state " + ActedState + ", the last stream " +
"sequence the builder acted on for each subject; and the publish items (design 47 G A-457a), counted, the first 50 by next try, each version's " +
"tries and next try, and why one ended after its tries. Each state is read in one page; a state past it is answered capped, naming the " +
"keys read and the state's size (H-A-293). Read-only."
// PublishAgainDescription is builder_publish_again's description.
const PublishAgainDescription = "Publish reserved versions again (design 47 G section 588), which builds nothing: with module and version, that reserved " +
@@ -400,22 +652,57 @@ const PublishAgainDescription = "Publish reserved versions again (design 47 G se
"with the versions it queued; each outcome is said as package-published, or left in version-unreleased with the holder's words. A hand act, " +
"recorded in the hand-act log."
// Resume takes up, at the builder's start, every queued version again, a begun one sent anew (§588).
// Resume takes up, at the builder's start, every queued version again, a begun one sent anew (§588): a version whose reservation
// was withdrawn or taken over is dropped with its publish item ended (H-A-290 (2)), and one of the live class whose build ended
// before this run is of the start's now (H-A-290 (1)). It reads the reservations once.
func (r *Republisher) Resume(ctx context.Context) error {
keys, err := r.Queue.KeysOf(ctx, AllFilter)
keys, err := r.Queue.KeysOf(ctx, QueueFilter)
if err != nil {
return fmt.Errorf("the state %s cannot be listed: %w", PublishQueueState, err)
}
all, err := r.Reserved.All(ctx)
if err != nil {
return fmt.Errorf("the queued publishes are not taken up: %w", err)
}
reserved := make(map[string]ReservedVersion, len(all))
for _, v := range all {
reserved[v.Module+" "+v.Version] = v
}
retired, err := r.Reserved.Retired(ctx, all)
if err != nil {
r.logf("publish again: the superseded builds are not retired at the start: %v", err)
}
holders := map[string]bool{}
for _, k := range keys {
raw, _, found, err := r.Queue.Get(ctx, k)
raw, rev, found, err := r.Queue.Get(ctx, k)
if err != nil || !found {
continue
}
var q QueuedPublish
if json.Unmarshal(raw, &q) == nil {
holders[q.Holder] = true
if json.Unmarshal(raw, &q) != nil {
continue
}
v, ok := reserved[q.Module+" "+q.Version]
why := NotStanding(v, ok)
if why == "" {
why = retired[q.Module+" "+q.Version]
}
if why != "" {
if err := r.Queue.Delete(ctx, k); err != nil {
r.logf("%s cannot be deleted from %s: %v", k, PublishQueueState, err)
}
r.end(ctx, q.Module, q.Version, why)
continue
}
if class := r.classOf(v); q.Class != class {
q.Class = class
if raw, err := json.Marshal(q); err == nil {
if _, err := r.Queue.Update(ctx, k, raw, rev); err != nil {
r.logf("%s keeps its class %s at the start: %v", k, q.Class, err)
}
}
}
holders[q.Holder] = true
}
for h := range holders {
r.drainLater(ctx, h)
@@ -445,16 +732,22 @@ func (r *Republisher) drainLater(ctx context.Context, holder string) {
go run()
}
// Drain sends holder's queued versions one at a time, oldest first, each bounded by UnreleasedBound; a key is deleted when
// its outcome is said.
// Drain sends holder's queued versions in passes (H-A-290 (3)): a pass lists the holder's keys once, reads each once, and sends
// them in their order (the live class first, each class oldest first), PublishSendsAtOnce at a time; the keys are listed again
// only when the pass is spent, or when a live publish was queued while it sends a start list: then the sends in flight end and
// the keys are listed again, so the live one goes next (H-A-291 (2)); a start-class one queued meanwhile waits for the pass's end.
// Each send is bounded by UnreleasedBound; a key is deleted when its outcome is said.
func (r *Republisher) Drain(ctx context.Context, holder string) {
for ctx.Err() == nil {
r.mu.Lock()
delete(r.relist, holder)
r.mu.Unlock()
keys, err := r.Queue.KeysOf(ctx, EscapeToken(holder)+".*.*")
if err != nil {
r.logf("the queue of %s cannot be listed: %v", holder, err)
return
}
var items []QueuedPublish
items := make([]QueuedPublish, 0, len(keys))
for _, k := range keys {
if raw, _, found, err := r.Queue.Get(ctx, k); err == nil && found {
var q QueuedPublish
@@ -466,12 +759,35 @@ func (r *Republisher) Drain(ctx context.Context, holder string) {
if len(items) == 0 {
return
}
sort.SliceStable(items, func(i, j int) bool { return items[i].Queued.Before(items[j].Queued) })
r.send(ctx, items[0])
sortQueued(items)
sem := make(chan struct{}, PublishSendsAtOnce)
var wg sync.WaitGroup
for _, q := range items {
if ctx.Err() != nil {
break
}
sem <- struct{}{}
r.mu.Lock()
again := q.Class != PublishClassLive && r.relist[holder]
r.mu.Unlock()
if again {
<-sem
r.logf("publish again: %s's pass lists again after its sends in flight: a live publish was queued", holder)
break
}
wg.Add(1)
go func() {
defer func() { <-sem; wg.Done() }()
r.send(ctx, q)
}()
}
wg.Wait()
}
}
// send publishes one queued version's artifacts and says the outcome, then deletes its key.
// send publishes one queued version's artifacts and says the outcome, then deletes its key. Every call it makes honours the one
// bound it is given, UnreleasedBound (a context's bound, no timer, ADR 0437 §8), and a send that returns without sending says why
// (H-A-290 (4)).
func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
key := QueueKey(q.Holder, q.Module, q.Version)
defer func() {
@@ -479,18 +795,44 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
r.logf("%s cannot be deleted from %s: %v", key, PublishQueueState, err)
}
}()
bound, cancel := context.WithTimeout(ctx, UnreleasedBound)
defer cancel()
if !q.Begun {
q.Begun = true
raw, _ := json.Marshal(q)
if _, rev, found, err := r.Queue.Get(ctx, key); err == nil && found {
_, _ = r.Queue.Update(ctx, key, raw, rev)
if _, rev, found, err := r.Queue.Get(bound, key); err == nil && found {
_, _ = r.Queue.Update(bound, key, raw, rev)
}
}
v, found, err := r.Reserved.Of(ctx, q.Module, q.Version)
if err != nil || !found {
return // released meanwhile, or dropped
all, err := r.Reserved.All(bound)
if err != nil {
r.logf("publish again: %s %s is not sent to %s: its reservation cannot be read: %v; its publish item asks it again", q.Module, q.Version, q.Holder, err)
return
}
e, err := r.Entry(ctx, v.Fingerprint)
var v ReservedVersion
found := false
var mine []ReservedVersion
for _, w := range all {
if w.Module == q.Module {
mine = append(mine, w)
if w.Version == q.Version {
v, found = w, true
}
}
}
why := NotStanding(v, found)
if why == "" {
retired, err := r.Reserved.Retired(bound, mine)
if err != nil {
r.logf("publish again: %s %s: whether a newer version supersedes it cannot be read, so it is sent: %v", q.Module, q.Version, err)
}
why = retired[q.Module+" "+q.Version]
}
if why != "" {
r.end(ctx, q.Module, q.Version, why)
return
}
e, err := r.Entry(bound, v.Fingerprint)
if err != nil {
r.say(ctx, v, fmt.Sprintf("the catalog entry %s cannot be read: %v", v.Fingerprint, err))
return
@@ -501,29 +843,13 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
}
sort.Strings(refs)
for _, ref := range refs {
var art *meshcatalog.Artifact
for i := range e.Artifacts {
if e.Artifacts[i].Ref == ref {
art = &e.Artifacts[i]
}
}
seat, ok := "", false
if art != nil {
seat, ok = SeatOfArtifact(*art)
}
seat, name, kind, ok := publishTarget(e, q.Module, ref)
if !ok {
r.say(ctx, v, ref+": the catalog entry names no registry seat for it")
return
}
name := NameOf(q.Module, strings.TrimPrefix(ref, q.Module+":"))
if art.Package != nil && art.Package.Name != "" {
name = art.Package.Name
}
kind := art.Kind
bound, cancel := context.WithTimeout(ctx, UnreleasedBound) // a context's bound, no timer (ADR 0437 §8)
err := r.Send(bound, Publish{Seat: seat, Kind: kind, Name: name, Version: v.Version, Digest: q.Artifacts[ref], Fingerprint: v.Fingerprint,
Commit: v.Commit, Transfer: RepublishTransfer(q.Module, v.Version, strings.TrimPrefix(ref, q.Module+":"))})
cancel()
if err == nil {
r.logf("publish again: %s %s: %s sent to %s as %s %s, digest %s; the package-published is said when the seat says published", q.Module, q.Version, ref, seat, name, v.Version, q.Artifacts[ref])
}
@@ -531,7 +857,7 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
r.logf("publish again: %s %s: %s refused by %s: %v", q.Module, q.Version, ref, seat, err)
words := err.Error()
if strings.Contains(words, strings.TrimSuffix(NotStoredWords, ": ")) {
if ok, merr := r.Reserved.MarkUnstored(ctx, q.Module, q.Version, ref, words); merr != nil {
if ok, merr := r.Reserved.MarkUnstored(bound, q.Module, q.Version, ref, words); merr != nil {
r.logf("%s %s: the unstored %s is not kept: %v", q.Module, q.Version, ref, merr)
} else if ok {
r.say(ctx, v, ref+" "+NotStoredWords+words)
@@ -543,6 +869,28 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
}
}
// publishTarget is the registry seat, the name and the kind module's artifact ref is published as, from its catalog entry; ok is
// false when the entry names no registry seat for it.
func publishTarget(e meshcatalog.CatalogEntry, module, ref string) (seat, name, kind string, ok bool) {
var art *meshcatalog.Artifact
for i := range e.Artifacts {
if e.Artifacts[i].Ref == ref {
art = &e.Artifacts[i]
}
}
if art == nil {
return "", "", "", false
}
if seat, ok = SeatOfArtifact(*art); !ok {
return "", "", "", false
}
name = NameOf(module, strings.TrimPrefix(ref, module+":"))
if art.Package != nil && art.Package.Name != "" {
name = art.Package.Name
}
return seat, name, art.Kind, true
}
func (r *Republisher) say(ctx context.Context, v ReservedVersion, why string) {
r.logf("%s %s is not published again: %s", v.Module, v.Version, why)
if r.Unreleased != nil {
@@ -579,3 +927,105 @@ func (r *Reserver) MarkUnstored(ctx context.Context, module, version, artifact,
return true, nil
}
}
// UnheardBound is how many messages the builder's durable on the events stream may hold pending and unacknowledged while a
// publish is asked again (design 47 H H-A-269 (2), H-A-272): at or above it a published of the version may still be unheard, so
// the try is skipped. A constant of the builder, no setting.
const UnheardBound = 100
// HeldVersion is one version a registry seat's verb versions answers of a name, with its digest.
type HeldVersion struct {
Version, Digest string
}
// HeldArtifact is an artifact of a reservation as a registry seat's record holds it at the reservation's version: its reference,
// the seat and the name it is published as, the version and the digest the seat holds.
type HeldArtifact struct {
Ref, Seat, Name, Version, Digest string
}
// Judged is what the registries' record says of a reserved version's missing artifacts (design 47 H H-A-272, H-A-274, H-A-275):
// those held at the version with the built digest (Held), those held at the version with another digest (Elsewhere), and, when an
// artifact is held at neither, the first one lacking (Lacks). All says every missing artifact is held with its built digest.
type Judged struct {
Held, Elsewhere []HeldArtifact
Lacks string
All bool
}
// Held judges v's missing artifacts against the registries' record (design 47 H H-A-269 (2), H-A-272, H-A-274, H-A-275): each
// artifact's registry seat and name as a publish again sends them, its record read through the seat's verb versions (versionsOf).
// A version with no missing artifact is All.
func (r *Republisher) Held(ctx context.Context, v ReservedVersion, versionsOf func(ctx context.Context, seat, name string) ([]HeldVersion, error)) (Judged, error) {
missing := v.Missing()
if len(missing) == 0 {
return Judged{All: true}, nil
}
e, err := r.Entry(ctx, v.Fingerprint)
if err != nil {
return Judged{}, fmt.Errorf("the catalog entry %s of %s %s cannot be read: %w", v.Fingerprint, v.Module, v.Version, err)
}
var out Judged
for _, ref := range missing {
seat, name, _, ok := publishTarget(e, v.Module, ref)
if !ok {
if out.Lacks == "" {
out.Lacks = ref + ": the catalog entry names no registry seat for it"
}
continue
}
held, err := versionsOf(ctx, seat, name)
if err != nil {
return Judged{}, fmt.Errorf("%s's versions of %s cannot be read: %w", seat, name, err)
}
at, found := HeldArtifact{}, false
for _, h := range held {
if strings.TrimPrefix(h.Version, "v") != v.Version {
continue
}
at, found = HeldArtifact{Ref: ref, Seat: seat, Name: name, Version: h.Version, Digest: h.Digest}, true
if h.Digest == v.Artifacts[ref] {
break
}
}
switch {
case !found:
if out.Lacks == "" {
out.Lacks = fmt.Sprintf("%s holds no %s %s", seat, name, v.Version)
}
case at.Digest == v.Artifacts[ref]:
out.Held = append(out.Held, at)
default:
out.Elsewhere = append(out.Elsewhere, at)
}
}
out.All = len(out.Held) == len(missing)
return out, nil
}
// HeldNumbers are the versions (MAJOR.MINOR.PATCH) a registry seat's record holds of any of module's artifacts refs, as the catalog
// entry of fingerprint names their seats and names (design 47 H H-A-275): a number one holds is never reserved. An artifact the
// entry names no registry seat for is read nowhere.
func (r *Republisher) HeldNumbers(ctx context.Context, module, fingerprint string, refs []string, versionsOf func(ctx context.Context, seat, name string) ([]HeldVersion, error)) ([]string, error) {
e, err := r.Entry(ctx, fingerprint)
if err != nil {
return nil, fmt.Errorf("the catalog entry %s of %s cannot be read: %w", fingerprint, module, err)
}
var out []string
for _, ref := range refs {
seat, name, _, ok := publishTarget(e, module, ref)
if !ok {
continue
}
held, err := versionsOf(ctx, seat, name)
if err != nil {
return out, fmt.Errorf("%s's versions of %s cannot be read: %w", seat, name, err)
}
for _, h := range held {
if v := strings.TrimPrefix(h.Version, "v"); v != "" && !slices.Contains(out, v) {
out = append(out, v)
}
}
}
return out, nil
}
@@ -5,8 +5,13 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
@@ -15,10 +20,19 @@ import (
type queueMem struct {
*reserveMem
bytes uint64
// listings counts the listings of one holder's keys (a filter `<holder>.*.*`), what a drain's pass lists.
listings atomic.Int32
}
func (q *queueMem) Bytes(context.Context) (uint64, error) { return q.bytes, nil }
func (q *queueMem) KeysOf(ctx context.Context, filter string) ([]string, error) {
if strings.HasSuffix(filter, ".*.*") && filter != QueueFilter {
q.listings.Add(1)
}
return q.reserveMem.KeysOf(ctx, filter)
}
// republishBed is a Republisher over memory states: reservations of module m (a bundle by the oci seat and a Go package by
// the Go seat), one holder module `oci` that claims the oci seat, and a Send and a Unreleased that record.
type republishBed struct {
@@ -26,8 +40,10 @@ type republishBed struct {
reserved *reserveMem
queue *queueMem
now time.Time
mu sync.Mutex // sent and said, which a holder's sends at once write
sent []Publish
said []string
ended []string
refuse string // the words Send fails with, empty for none
// holders are the registry seats' holders the seat-holders view names (design 47 G A-416).
holders map[string][]string
@@ -48,6 +64,8 @@ func newRepublishBed(t *testing.T) *republishBed {
{Ref: m + ":lib", Kind: "package", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "git.novox.be/novox/" + m + "/go"}}}}, nil
},
Send: func(_ context.Context, p Publish) error {
b.mu.Lock()
defer b.mu.Unlock()
b.sent = append(b.sent, p)
if b.refuse != "" {
return errors.New(b.refuse)
@@ -55,8 +73,15 @@ func newRepublishBed(t *testing.T) *republishBed {
return nil
},
Unreleased: func(_ context.Context, v ReservedVersion, why string) {
b.mu.Lock()
defer b.mu.Unlock()
b.said = append(b.said, v.Module+" "+v.Version+": "+why)
},
Ended: func(_ context.Context, module, version string) {
b.mu.Lock()
defer b.mu.Unlock()
b.ended = append(b.ended, module+" "+version)
},
Logf: t.Logf}
b.Holders = func(seat string) []string { return b.holders[seat] }
return b
@@ -190,9 +215,324 @@ func TestRepublish(t *testing.T) {
return json.RawMessage(`{"mode":"publish","module":"m","version":"0.1.0","tries":3,"next-try":"2026-11-17T01:00:00Z","why":"w"}`), true, nil
}}
none := Reader{Keys: func() ([]string, error) { return nil, nil }}
answer, err := AnswerQueue(none, none, items)
answer, err := AnswerQueue(none, none, items, "", "")
if err != nil || len(answer.Items) != 1 || answer.Items[0].Tries != 3 || answer.Items[0].Version != "0.1.0" {
t.Errorf("builder_publish_queue items = %+v (%v), want the one publish item with its tries", answer.Items, err)
}
})
}
// TestTheRegistriesRecordSaysWhetherAVersionIsHeld: design 47 H H-A-272, H-A-274, H-A-275 — a missing artifact its registry seat
// holds at the version (the Go form too) with the built digest is held; at the version with another digest it is held elsewhere;
// a version the seat lacks is said by name. The numbers a registry holds of the module's artifacts are what reserving skips.
func TestTheRegistriesRecordSaysWhetherAVersionIsHeld(t *testing.T) {
ctx := context.Background()
entry := meshcatalog.CatalogEntry{Artifacts: []meshcatalog.Artifact{{Ref: "m:lib", Kind: "package", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "git.example/m"}}}}
r := &Republisher{Entry: func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }}
v := ReservedVersion{Module: "m", Version: "0.1.37", Fingerprint: "fp", Artifacts: map[string]string{"m:lib": "sha256:a"}}
for _, c := range []struct {
name string
holds []HeldVersion
all, elsewhere, lacks bool
}{
{name: "held in the Go form with the built digest", holds: []HeldVersion{{Version: "v0.1.36", Digest: "sha256:z"}, {Version: "v0.1.37", Digest: "sha256:a"}}, all: true},
{name: "another digest at the version", holds: []HeldVersion{{Version: "0.1.37", Digest: "sha256:b"}}, elsewhere: true},
{name: "the version lacked", holds: []HeldVersion{{Version: "0.1.36", Digest: "sha256:a"}}, lacks: true},
} {
t.Run(c.name, func(t *testing.T) {
var seat, name string
versionsOf := func(_ context.Context, s, n string) ([]HeldVersion, error) { seat, name = s, n; return c.holds, nil }
j, err := r.Held(ctx, v, versionsOf)
if err != nil || j.All != c.all || (len(j.Elsewhere) == 1) != c.elsewhere || (j.Lacks != "") != c.lacks || seat != SeatGoPackages || name != "git.example/m" {
t.Fatalf("judged %+v (%v) asking %s for %s, want all %v, elsewhere %v, lacks %v asking %s for git.example/m", j, err, seat, name, c.all, c.elsewhere, c.lacks, SeatGoPackages)
}
if c.lacks && !strings.Contains(j.Lacks, "0.1.37") {
t.Errorf("what the registry lacks is said as %q", j.Lacks)
}
nums, err := r.HeldNumbers(ctx, "m", "fp", []string{"m:lib"}, versionsOf)
if err != nil || len(nums) != len(c.holds) {
t.Errorf("the held numbers are %v (%v), want one per version the seat holds", nums, err)
}
})
}
v.Published = map[string]bool{"m:lib": true}
if j, err := r.Held(ctx, v, nil); err != nil || !j.All {
t.Errorf("a version with no missing artifact is judged %+v (%v), want all held without a read", j, err)
}
}
// queueAt writes a queued version of module m at holder oci directly, as a start finds it.
func (b *republishBed) queueAt(t *testing.T, v ReservedVersion, class string, at time.Time) {
t.Helper()
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: "oci", Module: v.Module, Version: v.Version, Artifacts: map[string]string{v.Module + ":prog": "sha256:aa"}, Queued: at, Class: class})
if _, err := b.queue.Create(context.Background(), QueueKey("oci", v.Module, v.Version), raw); err != nil {
t.Fatal(err)
}
}
// TestAHoldersQueueSendsLiveFirstInPasses: design 47 H H-A-290 (1)-(3) — a publish whose build ended in this run is sent before
// any of a build that ended before the start, each class oldest first; a pass lists the holder's keys once and sends them, at most
// four at a time; a build whose reservation was withdrawn or taken over never publishes, its key dropped and its publish item ended,
// at enqueue, at the pass and at the start.
func TestAHoldersQueueSendsLiveFirstInPasses(t *testing.T) {
ctx := context.Background()
t.Run("the live class first, at most four sends at once", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
b := newRepublishBed(t)
var old []ReservedVersion
for _, m := range []string{"o1", "o2", "o3", "o4", "o5"} {
old = append(old, b.reserve(t, m))
}
b.Started = b.now
b.reserve(t, "live")
for i, v := range old {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.now = b.now.Add(time.Hour) // the live one is queued last, after every old one
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
release := make(chan struct{})
var arrived []string
var inflight, most int
send := b.Send
b.Send = func(ctx context.Context, p Publish) error {
b.mu.Lock()
arrived = append(arrived, p.Name)
inflight++
most = max(most, inflight)
b.mu.Unlock()
<-release
b.mu.Lock()
inflight--
b.mu.Unlock()
return send(ctx, p)
}
b.Go = func(f func()) { go f() }
if q, err := b.Republish(ctx, "live", ""); err != nil || len(q) != 1 {
t.Fatalf("queued %v (%v), want the live version", q, err)
}
synctest.Wait() // every send the bound lets run waits on release
b.mu.Lock()
first := slices.Clone(arrived)
b.mu.Unlock()
if len(first) != PublishSendsAtOnce || !slices.Contains(first, "live/prog") {
t.Errorf("the sends running at once are %v: want %d, the live one among them", first, PublishSendsAtOnce)
}
close(release)
synctest.Wait()
if len(b.sent) != 6 || most != PublishSendsAtOnce {
t.Errorf("sent %d, at most %d at once: want all six, %d at once", len(b.sent), most, PublishSendsAtOnce)
}
})
})
t.Run("a pass lists the keys once for all its sends", func(t *testing.T) {
b := newRepublishBed(t)
var vs []ReservedVersion
for i := range 6 {
vs = append(vs, b.reserve(t, fmt.Sprintf("m%d", i)))
}
for i, v := range vs {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.Drain(ctx, "oci")
if n := b.queue.listings.Load(); len(b.sent) != 6 || n != 2 {
t.Errorf("sent %d with %d listings, want six sends from one pass and one listing that finds it spent", len(b.sent), n)
}
})
t.Run("a start makes a live key of a build before it a start key", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.Started = b.now
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {} // the start's drain is not run here
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
raw, _, _, _ := b.queue.Get(ctx, QueueKey("oci", "m", v.Version))
var q QueuedPublish
if json.Unmarshal(raw, &q) != nil || q.Class != PublishClassStart {
t.Errorf("the key after the start is %s, want the class %s: its build did not end in this run", raw, PublishClassStart)
}
})
for _, at := range []string{"enqueue", "pass", "start"} {
t.Run("a withdrawn build never publishes, at the "+at, func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
if _, ok, err := b.Reserved.Withdraw(ctx, "m", v.Version); err != nil || !ok {
t.Fatalf("withdrawn %v (%v)", ok, err)
}
switch at {
case "enqueue":
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {} // a key queued before the withdrawal is dropped by the enqueue itself
if q, _ := b.Republish(ctx, "m", v.Version); len(q) != 0 {
t.Errorf("queued %v", q)
}
case "pass":
b.queueAt(t, v, PublishClassLive, b.now)
b.Drain(ctx, "oci")
case "start":
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {}
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
}
keys, _ := b.queue.KeysOf(ctx, QueueFilter)
if len(b.sent) != 0 || len(keys) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("sent %v, keys %v, items ended %v: want nothing sent, the key dropped and m %s's item ended", b.sent, keys, b.ended, v.Version)
}
})
}
t.Run("a live publish queued mid-pass goes after the sends in flight, before the rest of the start list", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
b := newRepublishBed(t)
var old []ReservedVersion
for _, m := range []string{"o1", "o2", "o3", "o4", "o5", "o6"} {
old = append(old, b.reserve(t, m))
}
b.Started = b.now
b.reserve(t, "live")
for i, v := range old {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.now = b.now.Add(time.Hour)
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
first, then := make(chan struct{}), make(chan struct{}) // the four in flight wait on first, every later send on then
var arrived []string
send := b.Send
b.Send = func(ctx context.Context, p Publish) error {
b.mu.Lock()
arrived = append(arrived, p.Name)
gate := map[bool]chan struct{}{true: first, false: then}[len(arrived) <= PublishSendsAtOnce]
b.mu.Unlock()
<-gate
return send(ctx, p)
}
b.Go = func(f func()) { go f() }
b.drainLater(ctx, "oci")
synctest.Wait() // four start sends in flight, the pass holding o5 and o6
if q, err := b.Republish(ctx, "live", ""); err != nil || len(q) != 1 {
t.Fatalf("queued %v (%v)", q, err)
}
close(first)
synctest.Wait() // the sends after the four: with the live one among them, or o5 and o6 alone when the pass goes on
b.mu.Lock()
next := slices.Clone(arrived[PublishSendsAtOnce:])
b.mu.Unlock()
if !slices.Contains(next, "live/prog") {
t.Errorf("after the four in flight the sends are %v, want the live one among them, the pass listed again", next)
}
close(then)
synctest.Wait()
if len(arrived) != 7 {
t.Errorf("sent %v, want all seven", arrived)
}
})
})
t.Run("a build a newer patch or release of its series supersedes is retired at the pass and at the start, never by a newer series", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
if _, err := b.Reserved.Reserve(ctx, "m", "0.2", []string{"m:prog"}, Source{Fingerprint: "src2:m"}); err != nil {
t.Fatal(err)
}
b.queueAt(t, v, PublishClassStart, b.now)
b.Drain(ctx, "oci")
if len(b.sent) != 1 || len(b.ended) != 0 {
t.Errorf("with a standing 0.2: sent %v, ended %v, want m %s sent, its series not superseded (H-A-293)", b.sent, b.ended, v.Version)
}
rel := &reserveMem{mem: newMem()}
raw, _ := json.Marshal(ReleasedRecord{Kept: []ReleasedBuild{{Fingerprint: "f", Version: "0.2.3"}, {Fingerprint: "g", Version: "0.1.9"}}})
_, _ = rel.Create(ctx, ReleasedKey("m"), raw)
b.Reserved.Releases = rel
b.sent = nil
b.queueAt(t, v, PublishClassStart, b.now)
b.Drain(ctx, "oci")
if len(b.sent) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("at the pass: sent %v, ended %v, want m %s retired by the release 0.1.9", b.sent, b.ended, v.Version)
}
b.ended = nil
b.queueAt(t, v, PublishClassStart, b.now)
b.Go = func(func()) {}
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
if keys, _ := b.queue.KeysOf(ctx, QueueFilter); len(keys) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("at the start: keys %v, ended %v, want m %s retired", keys, b.ended, v.Version)
}
only := &reserveMem{mem: newMem()}
raw, _ = json.Marshal(ReleasedRecord{Kept: []ReleasedBuild{{Fingerprint: "f", Version: "0.2.0"}}})
_, _ = only.Create(ctx, ReleasedKey("n"), raw)
got, err := (&Reserver{Releases: only}).Retired(ctx, []ReservedVersion{{Module: "n", Version: "0.1.4", BuiltAt: b.now}, {Module: "n", Version: "0.1.6"},
{Module: "n", Version: "0.3.0", BuiltAt: b.now}, {Module: "p", Version: "0.1.2", BuiltAt: b.now}})
if err != nil || len(got) != 1 || !strings.Contains(got["n 0.1.4"], "reservation 0.1.6") {
t.Errorf("retired %v (%v), want only n 0.1.4, by the newer patch 0.1.6; n 0.3.0 and p 0.1.2 kept, a release of 0.2 retiring neither", got, err)
}
})
t.Run("a version taken over is not sent, and its item ended", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.queueAt(t, v, PublishClassStart, b.now)
if _, err := b.Reserved.Reserve(ctx, "m", "0.1", []string{"m:prog"}, Source{Fingerprint: "src2:other"}); err != nil {
t.Fatal(err)
}
b.Drain(ctx, "oci")
if len(b.sent) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("sent %v, ended %v: want nothing sent and the item ended", b.sent, b.ended)
}
})
}
// TestThePublishQueueToolAnswersAtAnySize: design 47 H H-A-290 (5) — builder_publish_queue over two thousand queued versions answers
// within a second and small: counts per holder and class with the oldest and newest queue time, a list capped at fifty in the
// order they are sent, and one module's item with its class; each state read in bulk where the reader can.
func TestThePublishQueueToolAnswersAtAnySize(t *testing.T) {
at := time.Date(2026, 11, 20, 17, 0, 0, 0, time.UTC)
vals := map[string]json.RawMessage{}
for i := range 2000 {
holder, class := "oci", PublishClassStart
if i%2 == 1 {
holder = "go"
}
if i >= 1990 {
class = PublishClassLive
}
q := QueuedPublish{V: 1, Holder: holder, Module: fmt.Sprintf("m%d", i), Version: "0.1.0", Artifacts: map[string]string{"a": "sha256:aa"}, Queued: at.Add(time.Duration(i) * time.Second), Class: class}
vals[QueueKey(holder, q.Module, q.Version)], _ = json.Marshal(q)
}
gets := 0
queue := Reader{All: func() (StatePage, error) { return StatePage{Values: vals}, nil },
Keys: func() ([]string, error) { return slices.Collect(maps.Keys(vals)), nil },
Get: func(k string) (json.RawMessage, bool, error) { gets++; v, ok := vals[k]; return v, ok, nil }}
none := Reader{Keys: func() ([]string, error) { return nil, nil }}
start := time.Now()
a, err := AnswerQueue(queue, none, none, "", "")
took := time.Since(start)
raw, _ := json.Marshal(a)
if err != nil || took > time.Second || a.Total != 2000 || len(a.Queue) != PublishQueueListCap || a.Listed != PublishQueueListCap || len(raw) > 64<<10 || gets != 0 {
t.Fatalf("answered %d of %d in %s, %d bytes, %d single reads (%v): want all counted, fifty listed, within a second, small, in bulk", len(a.Queue), a.Total, took, len(raw), gets, err)
}
if fmt.Sprintln(a.Counts[1].Holder, a.Counts[1].Class, a.Counts[1].Count, a.Counts[1].Oldest.Equal(at.Add(1991*time.Second)), a.Counts[1].Newest.Equal(at.Add(1999*time.Second))) != "go "+PublishClassLive+" 5 true true\n" || len(a.Counts) != 4 {
t.Errorf("counts %+v, want per holder and class with their oldest and newest", a.Counts)
}
if a.Queue[0].Class != PublishClassLive || a.Queue[0].Holder != "go" {
t.Errorf("the list opens with %+v, want the go holder's live class, sent first", a.Queue[0])
}
one, err := AnswerQueue(queue, none, none, "m1995", "")
if err != nil || len(one.Queue) != 1 || one.Queue[0].Class != PublishClassLive || one.Total != 2000 {
t.Errorf("one module's answer %+v (%v), want its item with its class", one.Queue, err)
}
if a.Capped || len(a.Read) != 0 {
t.Errorf("a state one page holds is answered capped: %+v", a.Read)
}
queue.All = func() (StatePage, error) { return StatePage{Values: vals, More: true, Keys: 5000, Bytes: 3 << 20}, nil }
if c, err := AnswerQueue(queue, none, none, "", ""); err != nil || !c.Capped || len(c.Read) != 1 || c.Read[0].State != PublishQueueState ||
c.Read[0].KeysRead != 2000 || c.Read[0].StateKeys != 5000 || c.Read[0].StateBytes != 3<<20 || c.Total != 2000 {
t.Errorf("a state past its one page answered capped %v, read %+v (%v): want capped, 2000 keys read of 5000, its size said (H-A-293)", c.Capped, c.Read, err)
}
queue.All = func() (StatePage, error) { return StatePage{}, errors.ErrUnsupported }
if b, err := AnswerQueue(queue, none, none, "", ""); err != nil || b.Total != 2000 || gets != 2000 {
t.Errorf("a reader with no bulk read answered %d (%v) with %d single reads, want every key read one by one", b.Total, err, gets)
}
}
@@ -37,6 +37,9 @@ const (
UnreleasedBound = 10 * time.Minute
// VersionUnreleased is that finding.
VersionUnreleased = "version-unreleased"
// VersionHeldElsewhere is the finding of a number a registry seat already holds, skipped at reserving or moved from (design 47
// H H-A-275), scope module.
VersionHeldElsewhere = "version-held-elsewhere"
)
// ReservedVersion is the value of a reservation key.
@@ -135,6 +138,50 @@ type Reserver struct {
// DefaultReleasedKept.
ReleasedKept int
Now func() time.Time
// Logf says an index write that failed (H-A-269 (1)): the reservation stands, and Reindex writes its index at the next
// start. Nil says nothing.
Logf func(string, ...any)
// HeldNumbers are the versions a registry seat's record holds of any of module's artifacts (design 47 H H-A-275), read before a
// number is reserved: a number one holds is skipped. Nil reads none; a read that fails is said and the reserve proceeds.
HeldNumbers func(ctx context.Context, module, fingerprint string, refs []string) ([]string, error)
// HeldElsewhere is told the numbers of module a reserve skipped because a registry holds them (H-A-275); nil is told none.
HeldElsewhere func(ctx context.Context, module string, skipped []string)
}
// heldNumbers are the patches of series a registry holds of module's artifacts (H-A-275); a read that fails is said, and none.
func (r *Reserver) heldNumbers(ctx context.Context, module, series, fingerprint string, refs []string) map[int]bool {
if r.HeldNumbers == nil {
return nil
}
vs, err := r.HeldNumbers(ctx, module, fingerprint, refs)
if err != nil {
r.said(fmt.Errorf("%s: the registries' versions are not read before reserving, so a number they hold is not skipped: %w", module, err))
}
out := map[int]bool{}
for _, v := range vs {
if s, p, err := SplitVersion(v); err == nil && s == series {
out[p] = true
}
}
return out
}
// nextFree is the first patch after after that no registry holds, and the held numbers it skipped (H-A-275).
func nextFree(series string, after int, held map[int]bool) (int, []string) {
var skipped []string
p := after + 1
for held[p] {
skipped = append(skipped, fmt.Sprintf("%s.%d", series, p))
p++
}
return p, skipped
}
// said is told the outcome of a reindex: a failure is said, the reservation's own write stands.
func (r *Reserver) said(err error) {
if err != nil && r.Logf != nil {
r.Logf("%v", err)
}
}
func (r *Reserver) now() time.Time {
@@ -178,23 +225,26 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
}
}
key := ReservedKey(module, series)
held := r.heldNumbers(ctx, module, series, src.Fingerprint, artifacts)
for {
cur, rev, found, err := r.get(ctx, key)
if err != nil {
return ReservedVersion{}, err
}
if found && cur.Fingerprint == src.Fingerprint && cur.Fingerprint != "" && !slices.Contains(released, cur.Version) {
withdrawn := found && hasKey(cur.Withdrawn, cur.Version) // a number a registry holds elsewhere (H-A-279)
if found && !withdrawn && cur.Fingerprint == src.Fingerprint && cur.Fingerprint != "" && !slices.Contains(released, cur.Version) {
if _, p, err := SplitVersion(cur.Version); err == nil && p > high {
return cur, nil // kept until released (H-A-225); one at or below the catalog's highest can never be released
}
}
next := high
if found && !(src.Rebuild && cur.Fingerprint == src.Fingerprint) {
if found && (withdrawn || !(src.Rebuild && cur.Fingerprint == src.Fingerprint)) { // a withdrawn number is never taken again (H-A-226)
if _, p, err := SplitVersion(cur.Version); err == nil && p > next {
next = p
}
}
out := ReservedVersion{V: 1, Module: module, Series: series, Version: fmt.Sprintf("%s.%d", series, next+1), Artifacts: map[string]string{}, At: r.now(),
patch, skipped := nextFree(series, next, held)
out := ReservedVersion{V: 1, Module: module, Series: series, Version: fmt.Sprintf("%s.%d", series, patch), Artifacts: map[string]string{}, At: r.now(),
Repository: src.Repository, Commit: src.Commit, Fingerprint: src.Fingerprint, Branch: src.Branch, Source: src.Revision, Revision: src.CatalogRevision}
if src.GoPackage {
out.Tag = GoTagOf(out.Version)
@@ -231,6 +281,10 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
if err != nil {
return ReservedVersion{}, fmt.Errorf("%s cannot be reserved: %w", key, err)
}
r.said(r.reindex(ctx, key, cur, out)) // cur is zero when there was none
if len(skipped) > 0 && r.HeldElsewhere != nil {
r.HeldElsewhere(ctx, module, skipped)
}
if found && r.Took != nil {
r.Took(ctx, cur, out)
}
@@ -250,6 +304,38 @@ func (r *Reserver) get(ctx context.Context, key string) (ReservedVersion, uint64
return v, rev, true, nil
}
// Withdraw withdraws module's reservation of version for the build of its fingerprint, whose number a registry holds at another
// digest (design 47 H H-A-275, H-A-279): the number is kept in Withdrawn and stays a gap (H-A-226), so the build again of that
// fingerprint reserves the next number no registry holds, with fresh digests. It answers the reservation as withdrawn, and false
// when none stands at version.
func (r *Reserver) Withdraw(ctx context.Context, module, version string) (ReservedVersion, bool, error) {
series, _, err := SplitVersion(version)
if err != nil {
return ReservedVersion{}, false, err
}
key := ReservedKey(module, series)
for {
cur, rev, found, err := r.get(ctx, key)
if err != nil || !found || cur.Version != version {
return cur, false, err
}
if hasKey(cur.Withdrawn, version) {
return cur, true, nil
}
if cur.Withdrawn == nil {
cur.Withdrawn = map[string]string{}
}
cur.Withdrawn[version] = cur.Fingerprint
raw, _ := json.Marshal(cur)
if _, err := r.Store.Update(ctx, key, raw, rev); errors.Is(err, ErrConflict) {
continue
} else if err != nil {
return cur, false, fmt.Errorf("%s cannot be withdrawn: %w", key, err)
}
return cur, true, nil
}
}
// Built records artifact's digest under the reserved version of module's series; once every artifact has one the build is
// ended (BuiltAt). A reservation moved on to another version (a newer build took the next) is left alone and answers false.
func (r *Reserver) Built(ctx context.Context, module, version, artifact, digest string, deps []meshbuilds.Dependency) (bool, error) {
@@ -263,6 +349,8 @@ func (r *Reserver) Built(ctx context.Context, module, version, artifact, digest
if err != nil || !found || cur.Version != version {
return false, err
}
was := cur
was.Artifacts = maps.Clone(cur.Artifacts)
cur.Artifacts[artifact] = digest
if len(deps) > 0 {
if cur.Dependencies == nil {
@@ -279,6 +367,7 @@ func (r *Reserver) Built(ctx context.Context, module, version, artifact, digest
} else if err != nil {
return false, fmt.Errorf("%s cannot be written: %w", key, err)
}
r.said(r.reindex(ctx, key, was, cur))
return true, nil
}
}
@@ -397,6 +486,7 @@ func (r *Reserver) Released(ctx context.Context, module, version string) ([]stri
if err := r.Store.Delete(ctx, k); err != nil {
return gone, fmt.Errorf("%s cannot be deleted: %w", k, err)
}
r.said(r.reindex(ctx, k, cur, ReservedVersion{}))
gone = append(gone, k)
}
sort.Strings(gone)
@@ -460,15 +550,39 @@ func (r *Reserver) All(ctx context.Context) ([]ReservedVersion, error) {
return out, nil
}
// TakenOver is the reservation of module's series whose withdrawn numbers hold one of the build of fingerprint (design 47 H
// H-A-226), and whether there is one: the series moved to that reservation's fingerprint and version (H-A-244).
func (r *Reserver) TakenOver(ctx context.Context, module, fingerprint string) (ReservedVersion, bool, error) {
all, err := r.All(ctx)
if err != nil {
return ReservedVersion{}, false, err
}
for _, v := range all {
if v.Module != module || v.Fingerprint == fingerprint {
continue
}
for _, fp := range v.Withdrawn {
if fp == fingerprint {
return v, true, nil
}
}
}
return ReservedVersion{}, false, nil
}
// Drop deletes module's reservation of series on a person's word (builder_reserved_drop); the caller writes the hand-act
// log. It answers whether there was one.
func (r *Reserver) Drop(ctx context.Context, module, series string) (bool, error) {
key := ReservedKey(module, series)
_, _, found, err := r.get(ctx, key)
cur, _, found, err := r.get(ctx, key)
if err != nil || !found {
return false, err
}
return true, r.Store.Delete(ctx, key)
if err := r.Store.Delete(ctx, key); err != nil {
return true, err
}
r.said(r.reindex(ctx, key, cur, ReservedVersion{}))
return true, nil
}
// Marked is an artifact of a reservation a registry seat's published newly marked: the reservation as it stands then.
@@ -487,69 +601,138 @@ func (m Marked) PackagePublished(digest string, at time.Time) meshbuilds.Package
Version: m.Reserved.Version, Digest: digest, Dependencies: m.Reserved.Dependencies[m.Artifact]}
}
// Published takes a registry seat's published in: the artifacts of a reservation at version (a leading v is the Go form)
// whose built digest it names are marked published. It answers each artifact it newly marked, with the reservation as marked.
// A published that marks no artifact of a reservation at version whose build has not recorded every digest is kept on that
// reservation as heard (name, version, digest), and the keys it is kept on are answered too (design 47 G A-415): Kept marks
// it when the build records its digests.
// Published takes a registry seat's published in, as PublishedRead does, answering the artifacts newly marked and the keys it is
// kept on.
func (r *Reserver) Published(ctx context.Context, name, version, digest string) ([]Marked, []string, error) {
out, err := r.PublishedRead(ctx, name, version, digest, nil)
return out.Marked, out.Kept, err
}
// ReadReservation is a reservation a published read, as it stood after the published was taken in, and whether one of its built
// digests is the published's (design 47 H H-A-280).
type ReadReservation struct {
Key string
Reserved ReservedVersion
Matched bool
}
// PublishedOutcome is what a published did: the artifacts it newly marked, the keys it is kept on (A-415), and every reservation it
// read, each once, which the journal line is made from (H-A-280).
type PublishedOutcome struct {
Marked []Marked
Kept []string
Read []ReadReservation
}
// PublishedRead takes a registry seat's published in: the artifacts of a reservation at version (a leading v is the Go form)
// whose built digest it names are marked published. A published that marks no artifact of a reservation at version whose build
// has not recorded every digest is kept on that reservation as heard (name, version, digest), and the keys it is kept on are
// answered (design 47 G A-415): Kept marks it when the build records its digests. The reservation is found through the index
// (H-A-269 (1), H-A-271, H-A-280): the reservations holding the digest in one read, and only when none at version holds it, those
// at the version whose build has not recorded every digest in one more; never a scan of every reservation, and each reservation
// read at most once (again only on a write that met another). slow is told before the second pass, which may read many, so the
// caller can say in progress to the bus; nil is told nothing.
func (r *Reserver) PublishedRead(ctx context.Context, name, version, digest string, slow func()) (PublishedOutcome, error) {
version = strings.TrimPrefix(version, "v")
var out PublishedOutcome
if version == "" || digest == "" {
return nil, nil, nil
return out, nil
}
keys, err := r.Store.KeysOf(ctx, AllFilter)
keys, err := r.indexed(ctx, IndexDigestKey(digest))
if err != nil {
return nil, nil, fmt.Errorf("the reservations cannot be listed: %w", err)
return out, fmt.Errorf("the index of digest %s cannot be read: %w", digest, err)
}
sort.Strings(keys)
var marked []Marked
var kept []string
read := map[string]readOne{}
matched, err := r.publishedOn(ctx, keys, name, version, digest, false, read, &out)
if err != nil || matched {
return out, err
}
keys, err = r.indexed(ctx, IndexUnbuiltKey(version))
if err != nil {
return out, fmt.Errorf("the index of the unbuilt at version %s cannot be read: %w", version, err)
}
if len(keys) > 0 && slow != nil {
slow()
}
_, err = r.publishedOn(ctx, keys, name, version, digest, true, read, &out)
return out, err
}
// readOne is a reservation as one pass read it, kept so the next pass reads it no more.
type readOne struct {
v ReservedVersion
rev uint64
ok bool
}
// publishedOn takes a published in on the reservations under keys: an artifact whose built digest it names is marked; with keep,
// a reservation at version whose build has not recorded every digest and that it matches nothing of keeps it (A-415). A
// reservation read by an earlier pass (read) is taken as read; each read is added to out.Read once. It answers whether a
// reservation at version holds the digest.
func (r *Reserver) publishedOn(ctx context.Context, keys []string, name, version, digest string, keep bool, read map[string]readOne, out *PublishedOutcome) (bool, error) {
anyMatched := false
for _, k := range keys {
again:
cur, rev, found, err := r.get(ctx, k)
if err != nil {
return marked, kept, err
}
if !found || cur.Version != version {
continue
}
changed, matched, keep := false, false, false
var arts []string
for a, d := range cur.Artifacts {
if d != digest {
continue
}
matched = true
if !cur.Published[a] {
if cur.Published == nil {
cur.Published = map[string]bool{}
one, seen := read[k]
for {
if !seen {
cur, rev, found, err := r.get(ctx, k)
if err != nil {
return anyMatched, err
}
cur.Published[a], changed = true, true
arts = append(arts, a)
one = readOne{v: cur, rev: rev, ok: found}
}
}
sort.Strings(arts)
if !matched && cur.BuiltAt.IsZero() && !cur.keeps(name, digest) {
cur.Kept = append(cur.Kept, KeptPublished{Name: name, Version: version, Digest: digest})
changed, keep = true, true
}
if !changed {
continue
}
raw, _ := json.Marshal(cur)
if _, err := r.Store.Update(ctx, k, raw, rev); errors.Is(err, ErrConflict) {
goto again
} else if err != nil {
return marked, kept, fmt.Errorf("%s cannot be written: %w", k, err)
}
for _, a := range arts {
marked = append(marked, Marked{Reserved: cur, Artifact: a})
}
if keep {
kept = append(kept, k)
seen = false // a write that meets another reads again
cur := one.v
if !one.ok || cur.Version != version {
read[k] = one
break
}
changed, matched, keepIt := false, false, false
var arts []string
for a, d := range cur.Artifacts {
if d != digest {
continue
}
matched = true
if !cur.Published[a] {
cur.Published = maps.Clone(cur.Published)
if cur.Published == nil {
cur.Published = map[string]bool{}
}
cur.Published[a], changed = true, true
arts = append(arts, a)
}
}
sort.Strings(arts)
if keep && !matched && cur.BuiltAt.IsZero() && !cur.keeps(name, digest) {
cur.Kept = append(slices.Clone(cur.Kept), KeptPublished{Name: name, Version: version, Digest: digest})
changed, keepIt = true, true
}
rev := one.rev
if changed {
raw, _ := json.Marshal(cur)
var err error
if rev, err = r.Store.Update(ctx, k, raw, one.rev); errors.Is(err, ErrConflict) {
continue
} else if err != nil {
return anyMatched, fmt.Errorf("%s cannot be written: %w", k, err)
}
for _, a := range arts {
out.Marked = append(out.Marked, Marked{Reserved: cur, Artifact: a})
}
if keepIt {
out.Kept = append(out.Kept, k)
}
}
anyMatched = anyMatched || matched
if _, was := read[k]; !was {
out.Read = append(out.Read, ReadReservation{Key: k, Reserved: cur, Matched: matched})
}
read[k] = readOne{v: cur, rev: rev, ok: true}
break
}
}
return marked, kept, nil
return anyMatched, nil
}
func (v ReservedVersion) keeps(name, digest string) bool {
@@ -221,7 +221,7 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
}
}
// TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226 — a dropped number stays a gap, kept in
// TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226, H-A-244 — a dropped number stays a gap, kept in
// the series key's Withdrawn while it is above the catalog's highest released; at each write those at or below it go, so the
// map never grows without end.
func TestTheWithdrawnVersionsAreDroppedOnceReleasedPast(t *testing.T) {
@@ -239,6 +239,13 @@ func TestTheWithdrawnVersionsAreDroppedOnceReleasedPast(t *testing.T) {
if cur.Version != "0.1.6" || len(cur.Withdrawn) != 2 || cur.Withdrawn["0.1.4"] != "src2:a" || cur.Withdrawn["0.1.5"] != "src2:b" {
t.Fatalf("reservation %+v; want 0.1.6 with 0.1.4 and 0.1.5 withdrawn", cur)
}
// H-A-244: the build whose number was withdrawn finds the reservation its series moved to; the one standing finds none.
if by, took, err := r.TakenOver(ctx, "m", "src2:a"); err != nil || !took || by.Fingerprint != "src2:c" || by.Version != "0.1.6" {
t.Errorf("taken over from src2:a by %+v (%v, %v); want src2:c 0.1.6", by, took, err)
}
if _, took, err := r.TakenOver(ctx, "m", "src2:c"); err != nil || took {
t.Errorf("the standing build src2:c is said taken over (%v, %v)", took, err)
}
highest = "0.1.5" // the catalog released past both withdrawn numbers but the newest
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "src2:d"})
if err != nil || got.Version != "0.1.7" || len(got.Withdrawn) != 1 || got.Withdrawn["0.1.6"] != "src2:c" {
@@ -290,3 +297,52 @@ func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *
})
}
}
// TestANumberARegistryHoldsIsNeverReserved: design 47 H H-A-275 (the rule of H-A-245 for a registry's record) — a reserve skips
// every number a registry seat holds of the module's artifacts and says the skipped ones; a read that fails lets it proceed; a
// done build whose number a registry holds at another digest has it withdrawn as a gap (H-A-226), and its build again reserves the
// next free number with fresh digests (H-A-279), the index following it.
func TestANumberARegistryHoldsIsNeverReserved(t *testing.T) {
ctx := context.Background()
store := &reserveMem{mem: newMem()}
held := []string{"0.1.37", "0.1.38", "0.2.0"}
var readErr error
var skipped []string
r := &Reserver{Store: store, ReleasedOf: func(context.Context, string) ([]string, error) { return []string{"0.1.36"}, nil },
HeldNumbers: func(_ context.Context, module, fp string, refs []string) ([]string, error) {
if module != "node-tools" || fp != "fp1" || len(refs) != 1 {
t.Errorf("the registries are read for %s %s %v", module, fp, refs)
}
return held, readErr
},
HeldElsewhere: func(_ context.Context, _ string, s []string) { skipped = append(skipped, s...) }}
v, err := r.Reserve(ctx, "node-tools", "0.1", []string{"node-tools:program"}, Source{Fingerprint: "fp1", GoPackage: true})
if err != nil || v.Version != "0.1.39" || len(skipped) != 2 {
t.Fatalf("reserved %q (%v), skipped %v: want 0.1.39 past the two numbers the registry holds", v.Version, err, skipped)
}
// The build ends; the registry holds 0.1.39 at another digest meanwhile: the number is withdrawn, and the build again of the same
// fingerprint reserves the next number no registry holds, with no digest yet (H-A-279).
if _, err := r.Built(ctx, "node-tools", "0.1.39", "node-tools:program", "sha256:7c9b10", nil); err != nil {
t.Fatal(err)
}
if w, ok, err := r.Withdraw(ctx, "node-tools", "0.1.39"); err != nil || !ok || w.Withdrawn["0.1.39"] != "fp1" {
t.Fatalf("withdrew %+v (%v %v), want 0.1.39 withdrawn for fp1", w, ok, err)
}
held = []string{"0.1.37", "0.1.38", "0.1.39", "0.1.40", "0.2.0"}
// The build again of a done key reserves as a rebuild (A-490b): a withdrawn number is never its floor's to reuse.
again, err := r.Reserve(ctx, "node-tools", "0.1", []string{"node-tools:program"}, Source{Fingerprint: "fp1", GoPackage: true, Rebuild: true})
if err != nil || again.Version != "0.1.41" || again.Artifacts["node-tools:program"] != "" || again.Withdrawn["0.1.39"] != "fp1" || again.Tag != GoTagOf("0.1.41") {
t.Fatalf("the build again reserved %+v (%v), want 0.1.41 with no digest and 0.1.39 kept withdrawn", again, err)
}
if at, _ := r.AtVersion(ctx, "0.1.39"); len(at) != 0 {
t.Errorf("the index by version still names the withdrawn number: %v", at)
}
if _, ok, _ := r.Withdraw(ctx, "node-tools", "0.1.39"); ok {
t.Error("a number no reservation stands at any more was withdrawn")
}
// A registry that cannot be read lets the reserve proceed.
readErr, skipped = errors.New("registry down"), nil
if v, err := r.Reserve(ctx, "node-tools", "0.2", []string{"node-tools:program"}, Source{Fingerprint: "fp1"}); err != nil || v.Version != "0.2.1" {
t.Errorf("with the registry down reserved %q (%v), want 0.2.1 (what it answered still skipped)", v.Version, err)
}
}
@@ -4,12 +4,28 @@ import (
"context"
"errors"
"fmt"
"strings"
"time"
)
// AllFilter is the key filter of every version of every name: two tokens.
const AllFilter = "*.*"
// FilterMatches says whether key is under filter as the bus reads a key filter: token by token, a `*` any one token, so AllFilter
// lists the two-token keys alone and never the index's three-token keys (H-A-269 (1)).
func FilterMatches(filter, key string) bool {
ft, kt := strings.Split(filter, "."), strings.Split(key, ".")
if len(ft) != len(kt) {
return false
}
for i := range ft {
if ft[i] != "*" && ft[i] != kt[i] {
return false
}
}
return true
}
// ErrConflict is what a Store answers when a Create finds the key or an Update finds another revision.
var ErrConflict = errors.New("the key exists or changed")
@@ -21,10 +21,59 @@ const (
OldFormClosedEvent = "version.old-form-closed"
)
// Reader reads the state as the tools bundle may: every key, and one.
// Reader reads the state as the tools bundle may: every key, and one; and, where All is set, the state's values in one page,
// one request for many (design 47 H H-A-290 (5), H-A-292).
type Reader struct {
Keys func() ([]string, error)
Get func(key string) (json.RawMessage, bool, error)
// All is one page of the state's values; nil, or an error satisfying errors.Is(err, errors.ErrUnsupported), reads key by key.
All func() (StatePage, error)
}
// StatePage is one page of a state (H-A-292, H-A-293): its values, and, when the state holds more than the page, More with the
// state's size as its info says it (Keys, Bytes; InfoWords when the info cannot be read). No second page is asked.
type StatePage struct {
Values map[string]json.RawMessage
More bool
Keys, Bytes uint64
InfoWords string
}
// StateCapped is what an answer says of a state it read only the first page of (H-A-293): how many keys it read, and the
// state's size from its info.
type StateCapped struct {
State string `json:"state"`
KeysRead int `json:"keys-read"`
StateKeys uint64 `json:"state-keys"`
StateBytes uint64 `json:"state-bytes"`
InfoWords string `json:"info,omitempty"`
}
// values is the state's values: one page where the reader can (capped says what a page past the state's end left out), else
// key by key.
func (r Reader) values(state string) (map[string]json.RawMessage, *StateCapped, error) {
if r.All != nil {
pg, err := r.All()
if !errors.Is(err, errors.ErrUnsupported) {
if err != nil || !pg.More {
return pg.Values, nil, err
}
return pg.Values, &StateCapped{State: state, KeysRead: len(pg.Values), StateKeys: pg.Keys, StateBytes: pg.Bytes, InfoWords: pg.InfoWords}, nil
}
}
keys, err := r.Keys()
if err != nil {
return nil, nil, err
}
out := make(map[string]json.RawMessage, len(keys))
for _, k := range keys {
raw, ok, err := r.Get(k)
if err != nil || !ok {
continue
}
out[k] = raw
}
return out, nil, nil
}
// readStore is a Store over a Reader that refuses every write: the tools bundle only reads versions.
@@ -45,13 +94,9 @@ func (s readStore) KeysOf(_ context.Context, filter string) ([]string, error) {
if err != nil {
return nil, err
}
if filter == AllFilter {
return all, nil
}
prefix := strings.TrimSuffix(filter, "*")
var out []string
for _, k := range all {
if strings.HasPrefix(k, prefix) && !strings.Contains(k[len(prefix):], ".") {
if FilterMatches(filter, k) {
out = append(out, k)
}
}
@@ -66,21 +66,9 @@ func (m *mem) KeysOf(_ context.Context, filter string) ([]string, error) {
if m.down != nil {
return nil, m.down
}
prefix := strings.TrimSuffix(filter, "*")
var out []string
for k := range m.vals {
if filter == AllFilter {
out = append(out, k)
continue
}
if strings.Count(filter, ".") == 2 { // a three-token filter, `a.*.*` as the bus reads it
ft, kt := strings.Split(filter, "."), strings.Split(k, ".")
if len(kt) == 3 && kt[0] == ft[0] {
out = append(out, k)
}
continue
}
if strings.HasPrefix(k, prefix) && !strings.Contains(k[len(prefix):], ".") {
if FilterMatches(filter, k) { // token by token, as the bus reads a filter
out = append(out, k)
}
}
@@ -17,6 +17,7 @@ import (
"log"
"os"
"os/signal"
"strings"
"sync/atomic"
"syscall"
"time"
@@ -289,7 +290,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The indexer's findings are the seat mesh-conditions' generated finding, written through mesh-sdk's held finding writer
// (design 47 G §689, A-29): it holds what is written before the state is bound, and starts once the state is.
writer, err := views.NewHeldWriter(views.WriterConfig[mcseat.Finding, mcseat.FindingCleared, mcseat.FindingsStated, mcseat.FindingsAskedEvent]{
Module: cred.Module, State: calls.State(index.FindingsState), Node: cred.Node, Asked: mcseat.SubjectFindingsAsked, JS: js, Conn: conn, Logf: logger.Printf})
Module: cred.Module, State: calls.State(index.FindingsState), Node: cred.Node, Asked: mcseat.SubjectFindingsAsked, JS: js, Conn: conn, Logf: logger.Printf,
// its own kinds only (G §580, H-A-242, H-A-247): the tool runner's writer states its kinds over the same state
Kinds: index.FindingKinds, Retired: index.RetiredFindingKinds})
if err != nil {
return fmt.Errorf("the indexer's finding writer cannot be made: %w", err)
}
@@ -378,7 +381,16 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
logger.Printf("start: %s", name)
return func() { logger.Printf("start: %s done in %s", name, time.Since(began).Round(time.Millisecond)) }
}
done := step("reading the build log")
// A module of the current trunk graph with no registration is registered once at the start, as a kept trunk revision
// registers one entering the graph (design 47 H H-A-262, H-A-264 (1)); said loudly, whatever it answers.
done := step("registering the trunk graph's modules that have no registration")
if written, err := ix.RegisterTrunkGraph(ctx); err != nil {
logger.Printf("start: the trunk graph's modules with no registration are not all registered (%d were): %v", len(written), err)
} else {
logger.Printf("start: %d modules of the trunk graph had no registration and are registered now: %s", len(written), strings.Join(written, ", "))
}
done()
done = step("reading the build log")
// The log's entries at the start are read before any fingerprint is computed (below).
var started atomic.Bool
var startEnds []stdio.LogEntry // the log's entries read at the start, whose imports are kept once the packages are published
@@ -5,6 +5,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"path"
"slices"
@@ -115,10 +116,15 @@ func trunkVia(call toolCaller) func(ctx context.Context, repository, sha string)
}
// filesVia asks the forge for every blob under each directory at a commit (the seat git's trees with files, ADR 0593 §2),
// by directory, then path relative to it. An answer that lists no directory asked is an error naming it.
// by directory, then path relative to it. An answer that lists no directory asked is an error naming it. The seat answers
// a directory the commit does not hold with the kind not-found naming it (the contract's files field; design 47 H H-A-232):
// that is a failure of the tree too (index.ErrNotInTree), so the indexer reads each directory alone and leaves the gone one out.
func filesVia(call toolCaller) func(ctx context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
return func(_ context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
a, err := treesOf(call, repository, git.TreesInput{Commit: sha, Files: dirs})
if errors.Is(err, index.ErrNotFound) {
return nil, fmt.Errorf("%w: %w", err, index.ErrNotInTree)
}
if err != nil {
return nil, err
}
@@ -75,6 +75,20 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
if _, err := filesVia(noDir)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); !errors.Is(err, index.ErrNotInTree) {
t.Errorf("a missing directory: %v, want ErrNotInTree", err)
}
// H-A-232 (2): the seat's not-found naming a directory the commit lacks is a failure of the tree too, so the
// one-directory fallback runs; a failed listing of another kind is not.
gone := func(string, any) (json.RawMessage, error) {
return json.RawMessage(`{"error":"the directory modules/mesh-build-go/buildstep is not in novox/mesh-catalog at x","error-kind":"not-found"}`), nil
}
if _, err := filesVia(gone)(context.Background(), "novox/mesh-catalog", strings.Repeat("a", 40), []string{"modules/mesh-build-go/buildstep"}); !errors.Is(err, index.ErrNotInTree) {
t.Errorf("a directory the seat says not-found: %v, want ErrNotInTree", err)
}
failed := func(string, any) (json.RawMessage, error) {
return json.RawMessage(`{"error":"the forge did not answer","error-kind":"failed"}`), nil
}
if _, err := filesVia(failed)(context.Background(), "novox/mesh-catalog", strings.Repeat("a", 40), []string{"modules/a"}); err == nil || errors.Is(err, index.ErrNotInTree) {
t.Errorf("a listing that failed: %v, want an error that is not ErrNotInTree", err)
}
down := func(string, any) (json.RawMessage, error) { return nil, errors.New("down") }
if _, err := filesVia(down)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); err == nil || errors.Is(err, index.ErrNotInTree) {
t.Errorf("a failed call: %v, want an error that is not ErrNotInTree", err)
@@ -156,6 +156,7 @@ type Indexer struct {
filling atomic.Bool // a FillTaken runs in the background (KickFill)
fillAgain atomic.Bool // a kick arrived while it ran
underSeats bool // a dry run inside judge.WithSeats: its judge passes do not hold the seat list (verdicts.go)
isDry bool // a dry run (dryrun.go): it shares the live packages state, so it drops no kept package key (packages.go)
judgeMu sync.Mutex // one judge pass at a time (verdicts.go)
keepMu sync.Mutex // one catalog graph revision kept at a time (keepEntries)
// Pins is the state pins (pins.go); nil keeps none. PinsPending says how many messages the indexer's durable consumer of the
@@ -446,6 +446,11 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
ix.sayFill(ctx, meshcatalog.EventFilling, fmt.Sprintf("%d", rev.Revision), meshcatalog.Fill{Bytes: after, Cap: capacity})
}
ix.logf("catalog graph: revision %d of source revision %d kept (%s), %d entries, %d changed", rev.Revision, src, call.Call, len(rev.Entries), len(changed))
// A module entering the trunk's graph is registered before the revision is said, at every kind of kept trunk revision
// (design 47 H H-A-262, H-A-264).
if isTrunkBranch(branch) {
ix.registerEntering(ctx, slices.Collect(maps.Values(entries)))
}
if ix.SayCatalogRevision != nil {
if err := ix.SayCatalogRevision(ctx, meshcatalog.CatalogRevisionEvent{V: CatalogEntryLayout, Revision: rev.Revision, Kind: rev.Kind,
Branch: keys.EscapeName(branch), Source: src, Call: call.Call, CallKind: call.Kind, Module: call.Module, Version: call.Version,
@@ -35,7 +35,7 @@ func (ix *Indexer) dry() *dryRun {
// requirements, findings, seat definitions, holds and their bounds are not overlaid, nothing is said, and no hold is
// weighed (Holds and Counted nil hold nothing); ReadFile only reads.
Requirements: nil, Findings: nil, SeatDefinitions: nil, ReadFile: ix.ReadFile, SaySeatDefinition: nil,
Heard: nil, underSeats: true,
Heard: nil, underSeats: true, isDry: true,
// A dry run keeps no catalog graph revision (catalog_revisions.go): its revision is the take-in's to keep.
Revisions: nil, SayCatalogRevision: nil, SayFill: nil, EntryIndex: nil, NamedSets: nil, Versions: nil, SayVersionReleased: nil, SayVersionsFilled: nil, Inputs: nil, InputObjects: nil, Pins: nil, PinsPending: nil,
// A dry run writes no registration and asks no one.
@@ -0,0 +1,206 @@
package index
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient/heard"
"git.novox.be/novox/mesh-sdk/go/views"
)
// Design 47 H H-A-240, H-A-242, H-A-247, G §580: every finding kind the daemon's code raises or clears is one its writer owns
// (FindingKinds, and a views reader's views.ViewKinds), so the writer never refuses a kind of the indexer's own. The test
// reads the daemon's code (this package and cmd/mesh-delivery-indexer) for every place a kind is given to the writer: the
// kind argument of observed, finding, clearFinding, clearHeld and a writer's Clear, the Kind of a Finding literal, and the
// kinds of an SDK part handed the writer (partKinds). mcc.KindCallUnanswered stays readable, so a return to the tool runner's
// kind fails the test. A
// kind there is a string, a constant of the package or a constant named in imported below; any other form fails the test,
// naming its place, but for the pass-through parameter `kind`, a held finding's `.Kind` and the kind cut from a key a views
// reader names, which carry a kind given elsewhere.
func TestFindingKindsHoldEveryKindTheCodeRaises(t *testing.T) {
imported := map[string]string{"mcc.KindCallUnanswered": mcc.KindCallUnanswered, "views.ViewWaitsKind": views.ViewWaitsKind}
// kindArg is the place of the kind among a raising call's arguments.
kindArg := map[string]int{"observed": 2, "finding": 2, "clearFinding": 3, "clearHeld": 4, "Clear": 3}
// partKinds are the kinds each SDK part raises through the writer it is handed (H-A-247 (1)); handed says a literal's field
// gives it the writer.
partKinds := map[string][]string{"views.ProcessReader": {views.EventUnhandledKind}, "views.Emitter": {views.UnsaidKind, views.TooLargeKind},
"heard.Writer": {heard.Kind}}
handed := func(e ast.Expr) bool {
kv, ok := e.(*ast.KeyValueExpr)
if !ok {
return false
}
id, ok := kv.Key.(*ast.Ident)
return ok && (id.Name == "Findings" || id.Name == "Put")
}
fset := token.NewFileSet()
var files []*ast.File
consts := map[string]string{}
for _, dir := range []string{".", filepath.Join("..", "..", "cmd", "mesh-delivery-indexer")} {
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
if !strings.HasSuffix(e.Name(), ".go") || strings.HasSuffix(e.Name(), "_test.go") {
continue
}
f, err := parser.ParseFile(fset, filepath.Join(dir, e.Name()), nil, 0)
if err != nil {
t.Fatal(err)
}
files = append(files, f)
if dir != "." {
continue
}
for _, d := range f.Decls {
g, ok := d.(*ast.GenDecl)
if !ok || g.Tok != token.CONST {
continue
}
for _, s := range g.Specs {
v := s.(*ast.ValueSpec)
for i, n := range v.Names {
if i < len(v.Values) {
if l, ok := v.Values[i].(*ast.BasicLit); ok && l.Kind == token.STRING {
consts[n.Name], _ = strconv.Unquote(l.Value)
}
}
}
}
}
}
}
declared := append(slices.Clone(FindingKinds), views.ViewKinds...)
raised := map[string]string{} // kind -> a place it is raised
kindOf := func(e ast.Expr) (string, bool) {
switch x := e.(type) {
case *ast.BasicLit:
s, err := strconv.Unquote(x.Value)
return s, err == nil
case *ast.Ident:
if x.Name == "kind" {
return "", true // a pass-through parameter: its callers give the kind
}
s, ok := consts[x.Name]
if !ok {
s, ok = imported["index."+x.Name]
}
return s, ok
case *ast.SliceExpr:
return "", true // the kind of a key a views reader names: its own kinds (views.ViewKinds)
case *ast.SelectorExpr:
if x.Sel.Name == "Kind" {
return "", true // a held finding's kind, given when it was raised
}
if p, ok := x.X.(*ast.Ident); ok {
if p.Name == "index" {
s, ok := consts[x.Sel.Name]
return s, ok
}
s, ok := imported[p.Name+"."+x.Sel.Name]
return s, ok
}
}
return "", false
}
note := func(e ast.Expr) {
k, ok := kindOf(e)
if !ok {
t.Errorf("%s: a finding kind the test cannot read; give it as a constant (or name it in imported)", fset.Position(e.Pos()))
return
}
if k != "" {
raised[k] = fset.Position(e.Pos()).String()
}
}
for _, f := range files {
ast.Inspect(f, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.CallExpr:
var name string
switch fn := x.Fun.(type) {
case *ast.Ident:
name = fn.Name
case *ast.SelectorExpr:
name = fn.Sel.Name
}
if i, ok := kindArg[name]; ok && len(x.Args) > i {
if name == "Clear" && len(x.Args) != 5 {
return true // not a finding writer's Clear(ctx, scope, id, kind, why)
}
note(x.Args[i])
}
case *ast.CompositeLit:
// an SDK part handed the writer raises its own kinds through it (H-A-247 (1))
if s, ok := x.Type.(*ast.SelectorExpr); ok && slices.ContainsFunc(x.Elts, handed) {
if p, ok := s.X.(*ast.Ident); ok && p.Name != "index" { // the indexer's own code is read above
part := p.Name + "." + s.Sel.Name
if kinds, ok := partKinds[part]; ok {
for _, k := range kinds {
raised[k] = fset.Position(x.Pos()).String() + " (" + part + ")"
}
} else {
t.Errorf("%s: %s is handed the finding writer, and the test does not know its kinds: name them in partKinds",
fset.Position(x.Pos()), part)
}
}
}
if s, ok := x.Type.(*ast.SelectorExpr); ok && s.Sel.Name == "Finding" {
for _, el := range x.Elts {
if kv, ok := el.(*ast.KeyValueExpr); ok {
if id, ok := kv.Key.(*ast.Ident); ok && id.Name == "Kind" {
note(kv.Value)
}
}
}
}
}
return true
})
}
for _, k := range []string{PinMissingKind, OutOfStepKind, PackagesNotPublishedKind, CallUnansweredKind, views.EventUnhandledKind, heard.Kind} {
if _, ok := raised[k]; !ok {
t.Errorf("the test no longer finds the kind %s the code raises: it reads the code wrongly", k)
}
}
for k, at := range raised {
if !slices.Contains(declared, k) {
t.Errorf("%s: the code raises the finding kind %q, which the writer's list FindingKinds does not declare", at, k)
}
}
// a retired kind is one no code raises and the writer does not own (H-A-247 (3))
for _, r := range RetiredFindingKinds {
if at, ok := raised[r.Kind]; ok || slices.Contains(declared, r.Kind) {
t.Errorf("the retired kind %q is still declared or raised (%s)", r.Kind, at)
}
}
// H-A-249: the retired call-unanswered names the ids the indexer's unanswered calls are written under, as in a key
covers := func(scope, id string) bool {
for _, r := range RetiredFindingKinds {
if r.Kind == mcc.KindCallUnanswered && (r.Scope == "" || r.Scope == scope) && (slices.Contains(r.IDs, id) ||
slices.ContainsFunc(r.Prefixes, func(p string) bool { return strings.HasPrefix(id, p) })) {
return true
}
}
return false
}
for _, id := range []string{keyID("node-build-go." + StepsVerb), keyID("node-build-image." + StepsVerb), keyID(floorKey("mesh-tools")),
keyID(BuilderVersionsCall)} {
if !covers(mcc.ScopeCall, id) {
t.Errorf("the retired call-unanswered does not cover the indexer's id %s", id)
}
}
// the approval gate's findings: scope machine, id <node>=2e<ask>
if covers("machine", "novox=2eask-1") || covers("machine", "node-build-go.steps") || covers(mcc.ScopeCall, "novox=2eask-1") {
t.Error("the retired call-unanswered covers an id of the tool runner's")
}
}
@@ -4,7 +4,9 @@ import (
"context"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient/heard"
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-sdk/go/views"
)
// FindingsState is the indexer's state of its findings (design 47 G §383, §687).
@@ -12,6 +14,26 @@ const (
FindingsState = "findings"
)
// CallUnansweredKind is the indexer's finding of a call nobody answered within its bound (scope call, its id the call): a kind
// of its own, since the tool runner's writer over the same state owns call-unanswered (design 47 H H-A-247 (2)).
const CallUnansweredKind = "indexer-call-unanswered"
// FindingKinds are the finding kinds the indexer's daemon raises, the kinds its writer owns (design 47 G §580, H H-A-240,
// H-A-242, H-A-247): its own, the process reader's event-unhandled and the family-not-heard of the heard writer it is handed
// (H-A-247 (1)); a views reader's and the emitter's (views.ViewKinds) are always the writer's besides. The tool runner's
// writer over the same state `findings` owns its own. TestFindingKindsHoldEveryKindTheCodeRaises holds the code to this list.
var FindingKinds = []string{PinMissingKind, OutOfStepKind, PackagesNotPublishedKind, HistoryFillingKind, SeedWaitsKind,
SeedLeftOutKind, SeatDefinitionUnparsedKind, SeatDefinedTwiceKind, SeatOutsideItsDomainKind, CallUnansweredKind,
views.EventUnhandledKind, heard.Kind}
// RetiredFindingKinds are kinds no code of the indexer raises any more, which its writer clears once at its start (H-A-247
// (3)): merge-untaken went with H-A-82; call-unanswered became indexer-call-unanswered (H-A-247 (2)), and since the tool
// runner's writer owns call-unanswered in the same state, only the ids the indexer wrote under it, in scope call, are cleared (H-A-249):
// the seed's <build seat>.steps (every build seat is named node-build-<kind>), the floor's mesh-catalog.versions-floor.<module>
// and the versions fill's mesh-delivery-builder.builder_versions.
var RetiredFindingKinds = []views.RetiredKind{{Kind: "merge-untaken"}, {Kind: mcc.KindCallUnanswered, Scope: mcc.ScopeCall,
IDs: []string{keyID(BuilderVersionsCall)}, Prefixes: []string{"node-build-", keyID(FloorCall) + "."}}}
// FindingWriter is the indexer's finding writer, mesh-sdk's held writer of the seat mesh-conditions' generated finding (design
// 47 G A-29, A-38): Write raises a finding, Clear clears one, Held answers the findings it holds, so the indexer reads what
// stands from the writer.
@@ -0,0 +1,246 @@
package index
import (
"context"
"fmt"
"path"
"slices"
"strings"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// seedSteps are the seed's module-wide statuses (§8, design 47 H §745): the statuses the check judged until then, lint, unit
// tests, integration tests, composition and validate required, code review optional, approval bypass.
func seedSteps() meshcatalog.StepStatuses {
return meshcatalog.StepStatuses{Lint: "required", Unit: "required", Integration: "required",
Composition: "required", Review: "optional", Approval: "bypass", Validate: "required"}
}
// GraphModule is a module of a trunk revision's module graph as its registration is composed when it enters the graph
// (design 47 H H-A-262): its repository, the path of its manifest and the names of its artifacts.
type GraphModule struct {
Module string
Origin string
Path string
Artifacts []string
// LeftOut says why the seed left the module out (its finding seed-left-out, A-324): such a module is not registered by
// the graph (H-A-265 (2)). Empty: it is registered.
LeftOut string
}
// GraphModulesOf are the modules of a trunk revision's entries, sorted by module.
func GraphModulesOf(entries []meshcatalog.CatalogEntry) []GraphModule {
out := make([]GraphModule, 0, len(entries))
for _, e := range entries {
m := GraphModule{Module: e.Module, Origin: e.Repository, Path: path.Join(e.Path, ModuleManifest)}
for _, a := range e.Artifacts {
m.Artifacts = append(m.Artifacts, artifactName(a.Ref))
}
out = append(out, m)
}
slices.SortFunc(out, func(a, b GraphModule) int { return strings.Compare(a.Module, b.Module) })
return out
}
// enteringPolicy is the policy of a module entering the graph (H-A-262, H-A-263): the module-wide steps every other
// registered module of its repository holds alike, else the seed's; a per-artifact status only for an artifact name the module
// has that every sibling naming it holds identically; nothing else.
func enteringPolicy(m GraphModule, siblings []meshcatalog.Registration) meshcatalog.Policy {
p := meshcatalog.Policy{Steps: seedSteps()}
if len(siblings) > 0 {
alike := true
for _, s := range siblings[1:] {
alike = alike && s.Policy.Steps == siblings[0].Policy.Steps
}
if alike {
p.Steps = siblings[0].Policy.Steps
}
}
for _, a := range m.Artifacts {
var held []meshcatalog.StepStatuses
for _, s := range siblings {
if st, ok := s.Policy.Artifacts[a]; ok {
held = append(held, st)
}
}
if len(held) == 0 || slices.ContainsFunc(held[1:], func(st meshcatalog.StepStatuses) bool { return st != held[0] }) {
continue
}
if p.Artifacts == nil {
p.Artifacts = map[string]meshcatalog.StepStatuses{}
}
p.Artifacts[a] = held[0]
}
return p
}
// RegisterEntering registers every module of mods that has no registration (design 47 H H-A-262 to H-A-266; the operator,
// 2026-11-14: a module in the module graph is a registration): origin its repository, path its manifest's, the branch its
// repository's registered modules watch first alike, else the repository's default branch read through the seat git
// (enteringBranch), the policy of its repository's other registered modules (enteringPolicy), version 1 with no ask, said
// registration-changed. A module already registered is left as it is, and none is ever removed here. Skipped, each said in the
// journal: a module with a register proposal pending (its ask stands, H-A-265 (1)), one the seed left out (LeftOut, H-A-265
// (2)), and one whose default branch is not read (registered at the next kept revision, H-A-266). While no registration is
// held it registers none: the seed writes the first (H-A-264 (2)). It answers the modules it registered, sorted.
func (r *Registry) RegisterEntering(ctx context.Context, mods []GraphModule, by string) ([]string, error) {
r.mu.Lock()
defer r.mu.Unlock()
// One listing finds the modules with no registration; the registrations themselves are read only when one has none.
keys, err := r.Registrations.Keys(ctx)
if err != nil {
return nil, fmt.Errorf("the registrations are not listed: %w", err)
}
if len(keys) == 0 {
// With none held, the seed is the first and only writer: its policy reads the build seats' steps (H-A-264 (2)).
r.logf("registrations: none is held, so no module entering the module graph is registered here: the seed writes the first (H-A-264)")
return nil, nil
}
listed := map[string]bool{}
for _, k := range keys {
listed[k] = true
}
var entering []GraphModule
for _, m := range mods {
if m.Module != "" && !listed[RegistrationKey(m.Module)] {
entering = append(entering, m)
}
}
if len(entering) == 0 {
return nil, nil
}
held, err := r.RegistrationsAnswer(ctx, meshcatalog.RegistrationsInput{})
if err != nil {
return nil, fmt.Errorf("the registrations are not read: %w", err)
}
have := map[string]bool{}
byOrigin := map[string][]meshcatalog.Registration{}
for _, reg := range held.Registrations {
have[reg.Module] = true
byOrigin[reg.Origin] = append(byOrigin[reg.Origin], reg)
}
var written []string
var errs []string
for _, m := range entering {
if have[m.Module] {
continue
}
if m.LeftOut != "" {
r.logf("registrations: %s entered the module graph and is not registered: the seed left it out (%s), so it keeps seed-left-out (H-A-265)", m.Module, m.LeftOut)
continue
}
if p, _, found, err := r.load(ctx, ProposalKey(verbRegister, m.Module, "")); err != nil {
errs = append(errs, fmt.Sprintf("%s: its register proposal is not read: %v", m.Module, err))
continue
} else if found && p.State == ProposalOpen {
r.logf("registrations: %s entered the module graph with a register proposal pending (ask %s): it is not registered here, the operator's Approve registers it (H-A-265)", m.Module, p.Ask)
continue
}
branch, err := r.enteringBranch(m.Origin, byOrigin[m.Origin])
if err != nil {
r.logf("registrations: %s entered the module graph and is not registered now: %v; the next kept trunk revision registers it (H-A-266)", m.Module, err)
continue
}
reg := meshcatalog.Registration{Module: m.Module, Origin: m.Origin, Path: m.Path, Branches: []string{branch},
Policy: enteringPolicy(m, byOrigin[m.Origin])}
p := Proposal{V: 1, Verb: verbRegister, Module: m.Module, Caller: by, At: r.now(), State: ProposalOpen,
Lines: []string{fmt.Sprintf("%s: registered as it entered the module graph (%s, %s)", m.Module, m.Origin, m.Path)}}
if err := r.write(ctx, "", p, reg, oc.Warrant{Ask: "module-graph"}); err != nil {
errs = append(errs, fmt.Sprintf("%s: %v", m.Module, err))
continue
}
have[m.Module] = true
written = append(written, m.Module)
r.logf("registrations: %s entered the module graph of %s with no registration: registered as version 1 from %s (%s), watching %s (H-A-262)",
m.Module, TrunkBranch, m.Origin, m.Path, branch)
}
if len(errs) > 0 {
return written, fmt.Errorf("the registrations of modules entering the module graph are not written: %s", strings.Join(errs, "; "))
}
return written, nil
}
// enteringBranch is the branch a module entering the graph watches (H-A-265 (3), H-A-266): the first watched branch every
// registered module of its repository holds alike, else the repository's default branch as the seed reads it from the seat
// git; an error when that is not read.
func (r *Registry) enteringBranch(origin string, siblings []meshcatalog.Registration) (string, error) {
first := ""
alike := len(siblings) > 0
for _, s := range siblings {
if len(s.Branches) == 0 || (first != "" && s.Branches[0] != first) {
alike = false
break
}
first = s.Branches[0]
}
if alike && first != "" {
return first, nil
}
if r.Branch == nil {
return "", fmt.Errorf("the default branch of %s is not asked of the seat git", origin)
}
b, err := r.Branch(origin, "")
if err != nil {
return "", fmt.Errorf("the seat git does not answer the default branch of %s: %w", origin, err)
}
if b.Error != "" || b.Branch == "" {
return "", fmt.Errorf("the default branch of %s is not read: %s", origin, b.Error)
}
return b.Branch, nil
}
// registerEntering registers each module of a kept trunk revision's entries that has no registration (H-A-262, H-A-264 (1),
// (3)): a module entering the trunk's graph, at any kind of kept trunk revision. A dry run, or an indexer with no
// registrations, registers none. A failure is said in the journal: the revision is kept whatever it answers.
func (ix *Indexer) registerEntering(ctx context.Context, entries []meshcatalog.CatalogEntry) {
if ix.isDry || ix.Registry == nil || ix.Registry.Registrations == nil {
return
}
written, err := ix.Registry.RegisterEntering(ctx, ix.withLeftOut(GraphModulesOf(entries)), ix.Registry.Module)
if len(written) > 0 {
ix.logf("registrations: %d modules entering the module graph of %s registered: %s (H-A-262)", len(written), TrunkBranch, strings.Join(written, ", "))
}
if err != nil {
ix.logf("registrations: %v (H-A-262)", err)
}
}
// RegisterTrunkGraph registers, once at the indexer's start, each module of the current trunk graph that has no registration
// (H-A-264 (1)), as a kept trunk revision does; it answers the modules it registered.
func (ix *Indexer) RegisterTrunkGraph(ctx context.Context) ([]string, error) {
if ix.isDry || ix.Registry == nil || ix.Registry.Registrations == nil || ix.Revisions == nil {
return nil, nil
}
_, entries, found, err := readNewestOf(ctx, ix.Revisions, TrunkBranch)
if err != nil {
return nil, fmt.Errorf("the newest trunk revision is not read: %w", err)
}
if !found {
return nil, nil
}
return ix.Registry.RegisterEntering(ctx, ix.withLeftOut(GraphModulesOf(entries)), ix.Registry.Module)
}
// withLeftOut marks each module the indexer holds the finding seed-left-out of (A-324, H-A-265 (2)).
func (ix *Indexer) withLeftOut(mods []GraphModule) []GraphModule {
if ix.Findings == nil {
return mods
}
left := map[string]string{}
for _, f := range ix.Findings.Held() {
if f.Kind == SeedLeftOutKind && f.Scope == mcc.ScopeModule {
left[f.ID] = f.Summary
}
}
for i := range mods {
if why, ok := left[keyID(mods[i].Module)]; ok {
if why == "" {
why = "seed-left-out"
}
mods[i].LeftOut = why
}
}
return mods
}
@@ -0,0 +1,164 @@
package index
import (
"context"
"encoding/json"
"errors"
"slices"
"testing"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// Design 47 H H-A-262, H-A-263, H-A-264 (ADR 0666 §2 read with the operator's rule of 2026-11-14): a module that enters a
// watched trunk's module graph is registered as the revision is kept, before it is said, with no ask: origin its repository,
// path its manifest's, the branch its repository's modules watch first alike (else its default branch), the module-wide steps its repository's other modules hold alike (else the
// seed's) and a per-artifact status only where every sibling naming that artifact holds it identically, said
// registration-changed. A module registered already is untouched; one with a register proposal pending, one the seed left out
// and one whose default branch is not read are skipped (H-A-265, H-A-266); a branch that is not the trunk registers none, an indexer
// holding no registration leaves the first to the seed, and the start registers the current trunk graph's unregistered ones.
func TestAModuleEnteringTheTrunkGraphIsRegistered(t *testing.T) {
ctx := context.Background()
g := newRegRig()
var order []string
say := g.r.Say
g.r.Say = func(ctx context.Context, e meshcatalog.RegistrationChangedEvent) error {
order = append(order, "registration-changed "+e.Module)
return say(ctx, e)
}
ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Findings: newFindings(), Now: g.r.Now}
// The seat git's default branches: novox/new's is develop, novox/mixed's trunk, novox/broken's not read (H-A-266).
g.r.Branch = func(repo, branch string) (gitseat.BranchAnswer, error) {
switch repo {
case "novox/new":
return gitseat.BranchAnswer{Branch: "develop", Repository: repo}, nil
case "novox/mixed":
return gitseat.BranchAnswer{Branch: "trunk", Repository: repo}, nil
case "novox/broken":
return gitseat.BranchAnswer{}, errors.New("the forge does not answer")
}
return gitseat.BranchAnswer{Branch: "main", Repository: repo}, nil
}
ix.SayCatalogRevision = func(_ context.Context, e meshcatalog.CatalogRevisionEvent) error {
order = append(order, "catalog-revision "+e.Branch)
return nil
}
entry := func(module, repo, dir string, artifacts ...string) meshcatalog.CatalogEntry {
e := meshcatalog.CatalogEntry{Module: module, Repository: repo, Path: dir, Commit: "c1", Manifest: map[string]any{}}
for _, a := range artifacts {
e.Artifacts = append(e.Artifacts, meshcatalog.Artifact{Ref: module + ":" + a})
}
return e
}
keep := func(branch string, src int64, entries ...meshcatalog.CatalogEntry) {
t.Helper()
m := map[string]meshcatalog.CatalogEntry{}
for _, e := range entries {
m[e.Module] = e
}
if err := ix.KeepBranchEntries(ctx, branch, src, m, nil); err != nil {
t.Fatal(err)
}
}
// No registration held: the seed is the first writer, so a kept trunk revision registers nothing (H-A-264 (2)).
ix.Registry = g.r
keep("main", 1, entry("core", "novox/c", "go/core", "lib"))
if regs, _ := g.r.RegistrationsAnswer(ctx, meshcatalog.RegistrationsInput{}); len(regs.Registrations) != 0 {
t.Fatalf("with no registration held a trunk revision registered %v, want none: the seed writes the first", regs.Registrations)
}
steps := meshcatalog.StepStatuses{Lint: "required", Unit: "required", Integration: "required", Composition: "required", Review: "optional", Approval: "bypass"}
other := steps
other.Lint = "optional"
held := []meshcatalog.Registration{
{Module: "core", Origin: "novox/c", Path: "go/core/module.json", Branches: []string{"stable"}, Policy: meshcatalog.Policy{Steps: steps,
Artifacts: map[string]meshcatalog.StepStatuses{"lib": {Integration: "bypass"}, "files": {Lint: "bypass", Unit: "bypass"}}}},
{Module: "tools", Origin: "novox/c", Path: "go/tools/module.json", Branches: []string{"stable"}, Policy: meshcatalog.Policy{Steps: steps,
Artifacts: map[string]meshcatalog.StepStatuses{"lib": {Integration: "bypass"}, "files": {Unit: "bypass"}}}},
{Module: "x1", Origin: "novox/mixed", Path: "x1/module.json", Branches: []string{"main"}, Policy: meshcatalog.Policy{Steps: steps}},
{Module: "x2", Origin: "novox/mixed", Path: "x2/module.json", Branches: []string{"beta"}, Policy: meshcatalog.Policy{Steps: other}},
}
for _, reg := range held {
if err := g.r.write(ctx, "", Proposal{V: 1, Verb: verbRegister, Module: reg.Module, Caller: "test", State: ProposalOpen}, reg, oc.Warrant{Ask: "test"}); err != nil {
t.Fatal(err)
}
}
g.events, order = nil, nil
// pend has a register proposal pending (H-A-265 (1)); left was left out by the seed (A-324, H-A-265 (2)).
pending, _ := json.Marshal(Proposal{V: 1, Verb: verbRegister, Module: "pend", Ask: "reg-1", State: ProposalOpen})
if err := g.r.Proposals.Create(ctx, ProposalKey(verbRegister, "pend", ""), pending); err != nil {
t.Fatal(err)
}
if err := ix.Findings.Write(ctx, mcseat.Finding{Scope: mcc.ScopeModule, ID: keyID("left"), Kind: SeedLeftOutKind, Summary: "no build seat builds left:app"}); err != nil {
t.Fatal(err)
}
// A take-in that lists new modules beside a registered one.
keep("main", 2, entry("core", "novox/c", "go/core", "lib"), entry("lib2", "novox/c", "go/lib2", "lib", "files", "bin"),
entry("solo", "novox/new", "solo", "app"), entry("x3", "novox/mixed", "x3"), entry("pend", "novox/c", "go/pend"),
entry("left", "novox/c", "go/left", "app"), entry("lost", "novox/broken", "lost"))
got := func(module string) meshcatalog.Registration {
t.Helper()
r, found, err := g.r.Registration(ctx, module)
if err != nil || !found {
t.Fatalf("%s is not registered (%v)", module, err)
}
return r
}
lib2 := got("lib2")
if lib2.Version != 1 || lib2.Origin != "novox/c" || lib2.Path != "go/lib2/module.json" || !slices.Equal(lib2.Branches, []string{"stable"}) {
t.Errorf("lib2 registered as %+v, want version 1 from novox/c, go/lib2/module.json, watching stable, the branch its siblings watch first alike (H-A-266), not its default branch main", lib2)
}
if lib2.Policy.Steps != steps {
t.Errorf("lib2's steps %+v, want its repository's modules' alike steps %+v (H-A-263)", lib2.Policy.Steps, steps)
}
if a := lib2.Policy.Artifacts; len(a) != 1 || a["lib"] != (meshcatalog.StepStatuses{Integration: "bypass"}) {
t.Errorf("lib2's artifacts %+v, want lib alone, the one status its siblings hold identically (files differs, bin none)", a)
}
if solo := got("solo"); solo.Policy.Steps != seedSteps() || len(solo.Policy.Artifacts) != 0 || !slices.Equal(solo.Branches, []string{"develop"}) {
t.Errorf("solo, alone in its repository, has %+v watching %v, want the seed's default, watching its default branch develop", solo.Policy, solo.Branches)
}
if x3 := got("x3"); x3.Policy.Steps != seedSteps() || !slices.Equal(x3.Branches, []string{"trunk"}) {
t.Errorf("x3, whose siblings' steps and branches differ, has %+v watching %v, want the seed's default, watching the default branch trunk", x3.Policy.Steps, x3.Branches)
}
for _, m := range []string{"pend", "left", "lost"} {
if _, found, _ := g.r.Registration(ctx, m); found {
t.Errorf("%s was registered, want it skipped: a pending proposal, the seed's left-out, a default branch not read", m)
}
}
if core := got("core"); core.Version != 1 || core.By != "test" {
t.Errorf("core, registered already, became %+v, want it untouched", core)
}
if len(g.asks) != 0 {
t.Errorf("registering modules entering the graph asked %v, want no ask", g.asks)
}
wantOrder := []string{"registration-changed lib2", "registration-changed solo", "registration-changed x3", "catalog-revision main"}
if !slices.Equal(order, wantOrder) {
t.Errorf("said %v, want %v: each registration said before the revision", order, wantOrder)
}
// A branch that is not the trunk registers none (H-A-264 (3)); a module that leaves the graph keeps its registration.
keep("feature", 3, entry("feat", "novox/c", "go/feat", "lib"))
keep("main", 4, entry("core", "novox/c", "go/core", "lib"))
if _, found, _ := g.r.Registration(ctx, "feat"); found {
t.Error("a revision of the branch feature registered feat, want only the trunk's")
}
got("lib2")
// At the start, a module of the current trunk graph with no registration is registered (H-A-264 (1)); again, none.
ix.Registry = nil
keep("main", 5, entry("core", "novox/c", "go/core", "lib"), entry("late", "novox/c", "go/late"))
ix.Registry = g.r
if written, err := ix.RegisterTrunkGraph(ctx); err != nil || !slices.Equal(written, []string{"late"}) {
t.Fatalf("the start registered %v (%v), want late", written, err)
}
if written, err := ix.RegisterTrunkGraph(ctx); err != nil || len(written) != 0 {
t.Errorf("a second start registered %v (%v), want none", written, err)
}
}
@@ -68,7 +68,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
}
mods.put("mesh-sdk", sdkModule)
held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false); err != nil {
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
pub := ix.publishedAll(ctx)[meshSDK]
@@ -99,7 +99,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
// mesh-sdk merges a change of buildstep only: avahi does not import it.
held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c2, held, false); err != nil {
if err := ix.publish(ctx, "novox/mesh-sdk", c2, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
@@ -111,7 +111,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
// mesh-sdk merges a change of announce: avahi moves, to the import form, pinned to the new version.
held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false); err != nil {
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
@@ -195,7 +195,7 @@ func aSiblingDirectoryAloneMovesItsImporter(t *testing.T) {
fpAt := func(commit string) meshcatalog.ModulesByModuleValue {
t.Helper()
held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, catalogueRepo, commit, held, false); err != nil {
if err := ix.publish(ctx, catalogueRepo, commit, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
@@ -274,7 +274,7 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
trunk := func(commit string) func() error {
return func() error {
held, _ := ix.heldModules(ctx)
return ix.publish(ctx, "novox/mesh-sdk", commit, held, false)
return ix.publish(ctx, "novox/mesh-sdk", commit, held, false, TrunkBranch)
}
}
hear := func(version, commit string) func() error {
@@ -328,7 +328,7 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
}
mods.put("mesh-sdk", raw)
held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false); err != nil {
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
@@ -339,7 +339,7 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
t.Fatal(err)
}
held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false); err != nil {
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false, TrunkBranch); err != nil {
t.Fatal(err)
}
if err := ix.Recompute(ctx); err != nil {
@@ -156,7 +156,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
// 0593 §3). A forge that cannot list them leaves the state as it was, said: the importers move at the next merge.
reach := maps.Clone(held)
maps.Copy(reach, overlay)
if err := ix.publish(ctx, f.Repository, f.Merge, reach, false); err != nil {
if err := ix.publish(ctx, f.Repository, f.Merge, reach, false, f.Branch); err != nil {
ix.logf("%s at %s: its packages are not published: %v", f.Repository, f.Merge, err)
}
// What this merge moved reaches the modules standing on it (a base's tree, build-agent's fingerprint), in the graph's order.
@@ -14,6 +14,8 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
@@ -138,8 +140,14 @@ func reached(held map[string]meshcatalog.ModulesByModuleValue) (modules, imports
// directory, else by the Go module whose go.mod is at its directory or the nearest above it (asked of the forge
// directory by directory). A module the listing has no go.mod for is said and left out.
func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, modules, imports []string) (map[string]meshcatalog.PackagesByModuleValue, error) {
out, _, err := ix.packagesIn(ctx, repository, commit, modules, imports)
return out, err
}
// packagesIn is packagesAt, with the modules whose directory the tree at the commit does not hold (gone), by module path.
func (ix *Indexer) packagesIn(ctx context.Context, repository, commit string, modules, imports []string) (map[string]meshcatalog.PackagesByModuleValue, []string, error) {
if ix.Files == nil || commit == "" {
return nil, nil
return nil, nil, nil
}
dirOf := map[string]string{} // module path to its directory in the repository
for _, m := range modules {
@@ -182,7 +190,7 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
_, dir, _ := trunk.PackageRepository(imp)
for d := dir; ; d = parent(d) {
if err := list([]string{d}); err != nil {
return nil, err
return nil, nil, err
}
if _, ok := listings[d]["go.mod"]; ok {
m := strings.TrimSuffix(trunk.ForgeModulePrefix+strings.TrimPrefix(repository, "novox/")+"/"+d, "/")
@@ -201,27 +209,40 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
absent := map[string]bool{}
dirs := slices.Sorted(maps.Values(dirOf))
if err := list(dirs); err != nil {
if !errors.Is(err, ErrNotInTree) || len(dirs) < 2 {
return nil, err
if !errors.Is(err, ErrNotInTree) {
return nil, nil, err
}
whole := err
for _, d := range dirs {
if err := list([]string{d}); err != nil {
if len(dirs) > 1 {
err = list([]string{d})
}
if err != nil {
if !errors.Is(err, ErrNotInTree) {
return nil, err
return nil, nil, err
}
ix.logf("%s at %s: the directory %q is not in the tree, so the module there is not read: %v", repository, commit, d, err)
absent[d] = true
}
}
// Every directory asked is missing: gone only when the commit holds a tree at all (the seat's trees of the commit, no
// files), else the tree itself fails (H-A-232, H-A-233).
if len(absent) == len(dirs) {
return nil, whole // the tree itself fails
if ix.Trunk == nil {
return nil, nil, whole
}
if _, err := ix.Trunk(ctx, repository, commit); err != nil {
return nil, nil, fmt.Errorf("%w (and the commit's trees: %v)", whole, err)
}
ix.logf("%s at %s: the commit holds a tree, so every directory asked (%s) is gone", repository, commit, strings.Join(dirs, ", "))
}
}
out := map[string]meshcatalog.PackagesByModuleValue{}
var gone []string
for _, m := range slices.Sorted(maps.Keys(dirOf)) {
dir := dirOf[m]
if absent[dir] {
gone = append(gone, m)
continue
}
requires, pkgs, err := fingerprint.Packages(m, listings[dir])
@@ -236,7 +257,7 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
out[m] = meshcatalog.PackagesByModuleValue{Module: m, Repository: trunk.RepositoryName(repository), Dir: d, Commit: commit,
Requires: requires, Packages: pkgs}
}
return out, nil
return out, gone, nil
}
// parent is the directory above d, "" for the repository's root.
@@ -252,8 +273,10 @@ func parent(d string) string {
}
// publish writes the packages state for the Go modules of repository the held modules (and the merge's overlay) reach,
// as the merge commit holds them: a key whose commit, requirements and digests are what it holds already is left.
func (ix *Indexer) publish(ctx context.Context, repository, commit string, held map[string]meshcatalog.ModulesByModuleValue, onlyMissing bool) error {
// as the merge commit holds them: a key whose commit, requirements and digests are what it holds already is left. The kept
// key of a package whose directory the commit does not hold is dropped at the trunk's merge on the real indexer only, never
// at a branch's take-in (branch) or in a dry run, which share the live packages state (H-A-232 (1)).
func (ix *Indexer) publish(ctx context.Context, repository, commit string, held map[string]meshcatalog.ModulesByModuleValue, onlyMissing bool, branch string) error {
if ix.Packages == nil || ix.Files == nil {
return nil
}
@@ -263,6 +286,7 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
return nil
}
have := ix.publishedAll(ctx)
drop := !ix.isDry && isTrunkBranch(branch)
if onlyMissing {
// Imports a held module already resolves are not asked again.
var ms, is []string
@@ -287,10 +311,28 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
modules[repo] = append(modules[repo], m)
}
}
got, err := ix.packagesAt(ctx, repo, commit, modules[repo], imports[repo])
got, gone, err := ix.packagesIn(ctx, repo, commit, modules[repo], imports[repo])
if err != nil {
ix.notPublished(ctx, repo, commit, err)
return err
}
ix.published(ctx, repo, commit)
// A repository's packages are those in the merge's tree (design 47 H H-A-232): the kept key of a package whose directory
// the trunk's merge no longer holds is dropped, said, so the forge is never asked for it again.
for _, m := range gone {
if _, kept := have[m]; !kept || !drop {
continue
}
key, err := PackagesKey(m)
if err != nil {
ix.logf("%s: %v", m, err)
continue
}
if err := write(ctx, ix.Packages, key, func(_ []byte, found bool) ([]byte, bool, error) { return nil, found, nil }); err != nil {
return fmt.Errorf("packages.%s: %w", key, err)
}
ix.logf("package %s of %s: its directory is gone at %s: its kept key dropped", m, repo, commit)
}
for _, m := range slices.Sorted(maps.Keys(got)) {
next := got[m]
key, err := PackagesKey(m)
@@ -316,6 +358,36 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
return nil
}
// PackagesNotPublishedKind is the finding of a repository whose packages a take-in's publish could not list, for a reason
// that is not a gone directory (design 47 H H-A-233, H-A-234): scope merge, id the repository, with the forge's words;
// raised at once and cleared at the repository's next publish that lists.
const PackagesNotPublishedKind = "packages-not-published"
// notPublished raises the finding packages-not-published of repository.
func (ix *Indexer) notPublished(ctx context.Context, repository, commit string, cause error) {
if ix.Findings == nil {
return
}
f := ix.observed(mcc.ScopeMerge, repository, PackagesNotPublishedKind,
fmt.Sprintf("the packages of %s are not published at %.12s", repository, commit),
"A repository's packages are not published",
fmt.Sprintf("The take-in of %s at %s could not list its packages (%v), so their digests stay at their last published commit and a change to one does not move its importers.", repository, commit, cause),
"indexer."+PackagesNotPublishedKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("%s: the finding %s is not written: %v", repository, PackagesNotPublishedKind, err)
}
}
// published clears the finding packages-not-published of repository: its publish listed.
func (ix *Indexer) published(ctx context.Context, repository, commit string) {
if ix.Findings == nil {
return
}
if err := ix.clearFinding(ctx, mcc.ScopeMerge, repository, PackagesNotPublishedKind, "its packages are listed at "+commit); err != nil {
ix.logf("%s: the finding %s is not cleared: %v", repository, PackagesNotPublishedKind, err)
}
}
// FillPackages publishes, at the repository's taken merge (the trunk the modules say), each reached Go module the
// packages state does not hold yet: the state of a new indexer, or of a module a build newly reported an import of.
// A repository with no taken merge is left for its next merge.
@@ -349,7 +421,7 @@ func (ix *Indexer) FillPackages(ctx context.Context) error {
continue
}
began := time.Now()
if err := ix.publish(ctx, repo, t.Merge, held, true); err != nil {
if err := ix.publish(ctx, repo, t.Merge, held, true, TrunkBranch); err != nil {
ix.logf("%s: its packages are not published at %s: %v", repo, t.Merge, err)
}
ix.logf("publishing the packages at start: %s at %.12s took %s", repo, t.Merge, time.Since(began).Round(time.Millisecond))
@@ -0,0 +1,166 @@
package index
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// Design 47 H H-A-232 (1), H-A-233, H-A-234: a repository's packages are those in the merge's tree. The kept key of a
// package whose directory the trunk's merge no longer holds is dropped, said, and the repository's other packages are
// published all the same (issue 1585's one-directory fallback); a branch's take-in and a dry run, which share the live
// packages state, drop nothing. A repository whose every asked directory is gone, at a commit that holds a tree, drops them
// and raises nothing; a listing that fails for another reason, or a commit with no tree, raises the finding
// packages-not-published (scope merge, id the repository), cleared at the next publish that lists.
func TestARepositorysPackagesAreThoseInTheMergesTree(t *testing.T) {
const (
live = "git.novox.be/novox/mesh-catalog/modules/disk-load/diskreadings"
gone = "git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep"
liveDir = "modules/disk-load/diskreadings"
goneDir = "modules/mesh-build-go/buildstep"
)
ctx := context.Background()
old, merge := strings.Repeat("d", 40), strings.Repeat("e", 40)
reaching := func(m string) map[string]meshcatalog.ModulesByModuleValue {
return map[string]meshcatalog.ModulesByModuleValue{
"user": {Repository: catalogueRepo, Inputs2: &meshcatalog.ModulesByModuleValueInputs2{Packages: []string{m}}},
}
}
type setup struct {
ix *Indexer
findings *memFindings
said *[]string
down *error // the forge's listing fails for a reason that is not a gone directory
noTree *error // the seat's trees of the commit, without files, fail
}
newSetup := func(t *testing.T, kept ...string) setup {
t.Helper()
var down, noTree error
said := []string{}
files := func(_ context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
if down != nil {
return nil, down
}
out := map[string]map[string]string{}
for _, d := range dirs {
if d != liveDir {
// as filesVia says the seat's not-found naming the directory
return nil, fmt.Errorf("the directory %s is not in %s at %s: %w", d, repository, sha, ErrNotInTree)
}
out[d] = map[string]string{"go.mod": "m1", "readings.go": "r-" + sha[:1]}
}
return out, nil
}
trees := func(context.Context, string, string) ([]string, error) { return []string{"modules/disk-load"}, noTree }
findings := newFindings()
ix := &Indexer{Module: testSelf, Packages: newMem(), Files: files, Trunk: trees, Findings: findings,
Logf: func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }}
dirOf := map[string]string{live: liveDir, gone: goneDir}
for _, m := range kept {
key, err := PackagesKey(m)
if err != nil {
t.Fatal(err)
}
raw, err := EncodePublishedPackages(meshcatalog.PackagesByModuleValue{Module: m, Repository: catalogueRepo, Dir: dirOf[m], Commit: old,
Requires: "req", At: ptr(time.Unix(0, 0).UTC()), Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: m, Digest: "x"}}})
if err != nil {
t.Fatal(err)
}
ix.Packages.(*memStore).put(key, raw)
}
return setup{ix: ix, findings: findings, said: &said, down: &down, noTree: &noTree}
}
notPublished := func(s setup) bool {
for _, f := range s.findings.Held() {
if f.Kind == PackagesNotPublishedKind && f.Scope == mcc.ScopeMerge && f.ID == keyID(catalogueRepo) {
return true
}
}
return false
}
dropped := "package " + gone + " of " + catalogueRepo + ": its directory is gone at " + merge + ": its kept key dropped"
t.Run("the trunk's merge drops a gone package's key and publishes the others; a failed listing is a finding until one lists", func(t *testing.T) {
s := newSetup(t, live, gone)
*s.down = errors.New("seat:git.trees (failed): the forge did not answer")
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(live), false, TrunkBranch); err == nil {
t.Fatal("a listing that failed published")
}
if !notPublished(s) {
t.Fatalf("a listing that failed raised no finding %s: %v", PackagesNotPublishedKind, s.findings.Held())
}
*s.down = nil
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(live), false, TrunkBranch); err != nil {
t.Fatal(err)
}
have := s.ix.publishedAll(ctx)
if have[live].Commit != merge {
t.Errorf("packages of %s at %q, want the merge %s: a gone sibling kept it from being published", live, have[live].Commit, merge)
}
if _, kept := have[gone]; kept {
t.Errorf("the kept key of %s, whose directory the merge does not hold, stays", gone)
}
if !strings.Contains(strings.Join(*s.said, "\n"), dropped) {
t.Errorf("the journal does not say %q:\n%s", dropped, strings.Join(*s.said, "\n"))
}
if notPublished(s) {
t.Error("the finding packages-not-published stays after a publish that listed")
}
})
for _, row := range []struct {
name string
dry bool
branch string
}{
{"a branch's take-in keeps the key: only the trunk's merge says a directory is gone", false, "feat/x"},
{"a dry run keeps the key: it shares the live packages state", true, TrunkBranch},
} {
t.Run(row.name, func(t *testing.T) {
s := newSetup(t, live, gone)
ix := s.ix
if row.dry {
ix = s.ix.dry().ix
}
if err := ix.publish(ctx, catalogueRepo, merge, reaching(live), false, row.branch); err != nil {
t.Fatal(err)
}
if _, kept := s.ix.publishedAll(ctx)[gone]; !kept {
t.Errorf("the kept key of %s was dropped", gone)
}
})
}
t.Run("a one-key repository whose directory is gone drops it and raises nothing, when the commit holds a tree", func(t *testing.T) {
s := newSetup(t, gone)
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(gone), false, TrunkBranch); err != nil {
t.Fatal(err)
}
if _, kept := s.ix.publishedAll(ctx)[gone]; kept {
t.Errorf("the kept key of %s, the repository's one package, whose directory is gone, stays", gone)
}
if notPublished(s) {
t.Error("a gone directory raised packages-not-published")
}
})
t.Run("every asked directory missing at a commit with no tree is the tree's failure: the keys stay and it is a finding", func(t *testing.T) {
s := newSetup(t, gone)
*s.noTree = errors.New("seat:git.trees (not-found): Gitea API GET: 404 Not Found")
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(gone), false, TrunkBranch); err == nil {
t.Fatal("a commit with no tree published")
}
if _, kept := s.ix.publishedAll(ctx)[gone]; !kept {
t.Errorf("the kept key of %s was dropped at a commit with no tree", gone)
}
if !notPublished(s) {
t.Error("a commit with no tree raised no finding")
}
})
}
@@ -74,9 +74,7 @@ func (r *Registry) ComposeSeed(modules []SeedModule) (SeedProposal, error) {
return out, fmt.Errorf("the default branch of %s is not read: %s", m.Origin, b.Error)
}
reg := meshcatalog.Registration{Module: m.Module, Origin: m.Origin, Path: m.Path, Branches: []string{b.Branch},
Policy: meshcatalog.Policy{Steps: meshcatalog.StepStatuses{Lint: "required", Unit: "required", Integration: "required",
Composition: "required", Review: "optional", Approval: "bypass",
Validate: "required"}}} // design 47 H §745: the step validate is required in every policy by default
Policy: meshcatalog.Policy{Steps: seedSteps()}} // design 47 H §745: the step validate is required in every policy by default
for _, a := range m.Artifacts {
var s meshcatalog.StepStatuses
if !slices.Contains(a.Steps, "lint") {
@@ -124,7 +122,7 @@ func (r *Registry) ProposeSeed(ctx context.Context, modules []SeedModule, again
return "", err
}
if len(seed.Unanswered) > 0 {
return "", fmt.Errorf("the build seats %v have not answered steps: the seed waits for them (call-unanswered after five minutes)", seed.Unanswered)
return "", fmt.Errorf("the build seats %v have not answered steps: the seed waits for them (indexer-call-unanswered after five minutes)", seed.Unanswered)
}
regs, left = seed.Registrations, seed.Left
}
@@ -57,7 +57,7 @@ func buildSeatOf(defs map[string]seatdef.Definition, a meshcatalog.Artifact) (st
// SeedModulesFrom reads the modules the newest trunk revision holds for the seed (design 47 G §704, §8): for each artifact of
// each module the seat list's build seat answers steps with {module, artifact, fingerprint, fetch} from the module's trunk
// entry, the fetch its repository, commit, directory and the artifact's named paths. A seat that does not answer leaves
// its artifact unanswered, which holds the seed; its call-unanswered finding (scope call, id <seat>.steps) is written once
// its artifact unanswered, which holds the seed; its indexer-call-unanswered finding (scope call, id <seat>.steps) is written once
// it is unanswered five minutes and cleared when it answers.
func (ix *Indexer) SeedModulesFrom(ctx context.Context, defs map[string]seatdef.Definition, steps StepsCaller) ([]SeedModule, error) {
if ix.Revisions == nil {
@@ -202,7 +202,7 @@ func artifactName(ref string) string {
return ref
}
// stepsFindings writes call-unanswered for each seat unanswered for five minutes and clears it for each that answered.
// stepsFindings writes indexer-call-unanswered for each seat unanswered for five minutes and clears it for each that answered.
func (ix *Indexer) stepsFindings(ctx context.Context, answered map[string]bool, unanswered map[string]error) {
if ix.Findings == nil {
return
@@ -223,11 +223,11 @@ func (ix *Indexer) stepsFindings(ctx context.Context, answered map[string]bool,
}
for _, seat := range slices.Sorted(slices.Values(raise)) {
id := seat + "." + StepsVerb
f := ix.observed(mcc.ScopeCall, id, mcc.KindCallUnanswered,
f := ix.observed(mcc.ScopeCall, id, CallUnansweredKind,
fmt.Sprintf("the call %s is unanswered since %s", id, since[seat].UTC().Format(time.RFC3339)),
"A call whose answer is kept is not answered",
fmt.Sprintf("The seed asks %s of every artifact its build seat builds; it is not answered since %s (%v), so the first registrations wait for it.", id, since[seat].UTC().Format(time.RFC3339), unanswered[seat]),
"indexer."+mcc.KindCallUnanswered)
"indexer."+CallUnansweredKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("seed: the finding %s is not written: %v", id, err)
}
@@ -236,7 +236,7 @@ func (ix *Indexer) stepsFindings(ctx context.Context, answered map[string]bool,
if _, bad := unanswered[seat]; bad {
continue
}
if err := ix.clearFinding(ctx, mcc.ScopeCall, seat+"."+StepsVerb, mcc.KindCallUnanswered, "the seat answers steps"); err != nil {
if err := ix.clearFinding(ctx, mcc.ScopeCall, seat+"."+StepsVerb, CallUnansweredKind, "the seat answers steps"); err != nil {
ix.logf("seed: the finding %s.%s is not cleared: %v", seat, StepsVerb, err)
}
}
@@ -15,7 +15,7 @@ import (
// Design 47 G §704: the seed asks `steps` of the build seat whose `build` rule matches each artifact, with {module,
// artifact, fingerprint, fetch} from the newest trunk entry; a seat that does not answer holds the seed and, after five
// minutes, is the finding call-unanswered (scope call, id <seat>.steps), cleared when it answers.
// minutes, is the finding indexer-call-unanswered (scope call, id <seat>.steps), cleared when it answers.
func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
ctx := context.Background()
now := time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC)
@@ -85,7 +85,7 @@ func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
if _, err := ix.SeedModulesFrom(ctx, defs, call); err != nil {
t.Fatal(err)
}
if h := w.Held(); len(h) != 1 || h[0].Scope != mcc.ScopeCall || h[0].ID != "node-build-go.steps" || h[0].Kind != mcc.KindCallUnanswered {
if h := w.Held(); len(h) != 1 || h[0].Scope != mcc.ScopeCall || h[0].ID != "node-build-go.steps" || h[0].Kind != CallUnansweredKind {
t.Fatalf("the findings after five minutes: %+v", h)
}
goUp = true
@@ -22,7 +22,7 @@ import (
// filled per module whenever the indexer starts and at each catalog revision kept, not once (A-490d, A-490f): the key
// versions-filled closes only the builder's half of the fill.
// FloorCall is the id of the finding call-unanswered when a floor read (the forge's tags or a package holder's versions) is
// FloorCall is the id of the finding indexer-call-unanswered when a floor read (the forge's tags or a package holder's versions) is
// unanswered five minutes after the first (A-490h).
const FloorCall = "mesh-catalog.versions-floor"
@@ -91,7 +91,7 @@ func (ix *Indexer) FloorModules(ctx context.Context) ([]string, error) {
// FillFloor fills one module's floor, the highest version among its packages' tags and holders' versions (A-490g). Its reads are
// kept calls made through mesh-sdk's form, which asks again through every transport fault; a read it gives up on (its own
// pause spent) is asked again at once, never given up, so the read has no bound (A-490i). The first unanswered try starts the
// module's clock: five minutes later the finding call-unanswered is written, and cleared when a read answers. A read the
// module's clock: five minutes later the finding indexer-call-unanswered is written, and cleared when a read answers. A read the
// callee answered with a failure ends the fill, left for the next start or revision. One fill of a module runs at a time.
func (ix *Indexer) FillFloor(ctx context.Context, module string, src FloorSources) error {
if _, running := ix.floors.LoadOrStore(module, true); running {
@@ -198,7 +198,7 @@ func FloorSinceKey(module string) string { return floorSinceKey(module) }
func floorSinceKey(module string) string { return "call-since." + keys.EscapeName(floorKey(module)) }
// floorUnanswered writes call-unanswered once the module's floor read is unanswered five minutes (the time kept in the own state).
// floorUnanswered writes indexer-call-unanswered once the module's floor read is unanswered five minutes (the time kept in the own state).
func (ix *Indexer) floorUnanswered(ctx context.Context, module string, first time.Time, cause error) {
now := ix.now()
since := ix.unansweredSince(ctx, floorSinceKey(module), first)
@@ -206,11 +206,11 @@ func (ix *Indexer) floorUnanswered(ctx context.Context, module string, first tim
return
}
at := since.UTC().Format(time.RFC3339)
f := ix.observed(mcc.ScopeCall, floorKey(module), mcc.KindCallUnanswered,
f := ix.observed(mcc.ScopeCall, floorKey(module), CallUnansweredKind,
fmt.Sprintf("the floor of %s's versions is unanswered since %s", module, at),
"A call whose answer is kept is not answered",
fmt.Sprintf("The floor of %s's versions reads the forge's tags and the package holders' versions; a read is not answered since %s (%v), so it has no version.", module, at, cause),
"indexer."+mcc.KindCallUnanswered)
"indexer."+CallUnansweredKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("floor: the finding %s is not written: %v", floorKey(module), err)
}
@@ -219,7 +219,7 @@ func (ix *Indexer) floorUnanswered(ctx context.Context, module string, first tim
func (ix *Indexer) floorAnswered(ctx context.Context, module string) {
ix.answered(ctx, floorSinceKey(module))
if ix.Findings != nil {
if err := ix.clearFinding(ctx, mcc.ScopeCall, floorKey(module), mcc.KindCallUnanswered, "the floor's reads answer"); err != nil {
if err := ix.clearFinding(ctx, mcc.ScopeCall, floorKey(module), CallUnansweredKind, "the floor's reads answer"); err != nil {
ix.logf("floor: the finding %s is not cleared: %v", floorKey(module), err)
}
}
@@ -63,7 +63,7 @@ func versionNameOf(module string, a meshcatalog.Artifact) string {
// FillVersions fills the state versions once from builder_versions and says versions-filled (§372). A fill already made
// is not made again; a builder_versions that does not answer holds the fill and, after five minutes, is the finding
// call-unanswered (scope call, id mesh-delivery-builder.builder_versions), cleared when it answers.
// indexer-call-unanswered (scope call, id mesh-delivery-builder.builder_versions), cleared when it answers.
func (ix *Indexer) FillVersions(ctx context.Context, ask VersionsAsker) error {
if ix.Versions == nil || ix.Heard == nil {
return errors.New("the state versions is not kept here")
@@ -127,7 +127,7 @@ func (ix *Indexer) FillVersions(ctx context.Context, ask VersionsAsker) error {
}
ix.answered(ctx, versionsFillSince)
if ix.Findings != nil {
if err := ix.clearFinding(ctx, mcc.ScopeCall, BuilderVersionsCall, mcc.KindCallUnanswered, "builder_versions answers"); err != nil {
if err := ix.clearFinding(ctx, mcc.ScopeCall, BuilderVersionsCall, CallUnansweredKind, "builder_versions answers"); err != nil {
ix.logf("versions fill: the finding %s is not cleared: %v", BuilderVersionsCall, err)
}
}
@@ -184,18 +184,18 @@ func (ix *Indexer) FillVersions(ctx context.Context, ask VersionsAsker) error {
return nil
}
// fillUnanswered writes call-unanswered once builder_versions is unanswered five minutes (the time kept in the own state).
// fillUnanswered writes indexer-call-unanswered once builder_versions is unanswered five minutes (the time kept in the own state).
func (ix *Indexer) fillUnanswered(ctx context.Context, cause error) {
now := ix.now()
since := ix.unansweredSince(ctx, versionsFillSince, now)
if ix.Findings == nil || now.Sub(since) < mcc.CallUnansweredAfter {
return
}
f := ix.observed(mcc.ScopeCall, BuilderVersionsCall, mcc.KindCallUnanswered,
f := ix.observed(mcc.ScopeCall, BuilderVersionsCall, CallUnansweredKind,
fmt.Sprintf("the call %s is unanswered since %s", BuilderVersionsCall, since.UTC().Format(time.RFC3339)),
"A call whose answer is kept is not answered",
fmt.Sprintf("The first fill of the catalog's versions asks %s of every artifact; it is not answered since %s (%v), so no version is released and the builder reserves nothing.", BuilderVersionsCall, since.UTC().Format(time.RFC3339), cause),
"indexer."+mcc.KindCallUnanswered)
"indexer."+CallUnansweredKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("versions fill: the finding %s is not written: %v", BuilderVersionsCall, err)
}
@@ -87,7 +87,7 @@ func TestAVersionIsReleasedOnceEveryArtifactIsPublished(t *testing.T) {
// Design 47 G §372, A-350: the first fill records, once, each module's highest version the builder gave a number, numbered or
// published, over its artifacts as released and says versions-filled once; a builder_versions that does not answer holds it and, after five minutes, is the finding
// call-unanswered; a fill made is not made again.
// indexer-call-unanswered; a fill made is not made again.
func TestTheFirstFillOfTheVersionsComesFromTheBuildersNumbers(t *testing.T) {
ctx := context.Background()
now := time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC)
@@ -501,7 +501,7 @@ func TestAnInputIsDeletedAtTheStepEndedOfItsAsk(t *testing.T) {
// Design 47 G §412 as A-490c, A-490d and A-490e amend it: a module the versions hold none of is filled from the highest tag of its
// Go package in the repository it was taken in from, per module and not once; a forge that does not answer fills nothing and
// raises call-unanswered after five minutes.
// raises indexer-call-unanswered after five minutes.
func TestTheTagFloorFillsAModuleWithNoVersionFromItsGoTags(t *testing.T) {
ctx := context.Background()
now := time.Date(2026, 11, 19, 17, 0, 0, 0, time.UTC)
@@ -623,7 +623,7 @@ func TestAFloorEntryIsReplacedByThePublishHeardAtItsVersion(t *testing.T) {
}
}
// ADR 0668 §3, design 47 G A-490i: the finding call-unanswered of a module's floor clears, with its call-since key, on every way the
// ADR 0668 §3, design 47 G A-490i: the finding indexer-call-unanswered of a module's floor clears, with its call-since key, on every way the
// fill ends but the end of its context: a release gave the module a version, the module left the trunk, or the callee answered with a
// failure (FloorModules never picks the module again to clear it).
func TestTheFloorFindingClearsOnEveryExitOfTheFill(t *testing.T) {
@@ -0,0 +1,155 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/nats-io/nats.go/jetstream"
nodeengine "git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"mesh-delivery-pipeline/internal/pipeline"
)
// envAskUnansweredBound names the file the manifest writes the setting ask-unanswered-bound into, `${config:<key>}`
// (design 47 H H-A-238 (1)).
const envAskUnansweredBound = "MESH_RUNNER_ASK_UNANSWERED_BOUND_FILE"
// envAskSilentBound names the file of the setting ask-silent-bound (H-A-243 (2)).
const envAskSilentBound = "MESH_RUNNER_ASK_SILENT_BOUND_FILE"
// askUnansweredBound reads the setting ask-unanswered-bound at the start, a Go duration (H-A-235, H-A-238 (1)): its default
// ten minutes when the file is missing or empty, an error naming the file for a content that is no positive duration.
func askUnansweredBound(getenv func(string) string) (time.Duration, error) {
return durationSetting(getenv, envAskUnansweredBound, pipeline.DefaultAskUnansweredBound)
}
// askSilentBound reads the setting ask-silent-bound at the start, a Go duration, forty-five minutes by default (H-A-243 (2)).
func askSilentBound(getenv func(string) string) (time.Duration, error) {
return durationSetting(getenv, envAskSilentBound, pipeline.DefaultAskSilentBound)
}
func durationSetting(getenv func(string) string, env string, def time.Duration) (time.Duration, error) {
v, err := settingText(getenv(env))
if err != nil || v == "" {
return def, err
}
d, err := time.ParseDuration(v)
if err != nil || d <= 0 {
return 0, fmt.Errorf("%s holds %q, which is not a positive duration", getenv(env), v)
}
return d, nil
}
// askClock is the bound of every ask but a test run's over the state bounds (H-A-238 (5)): a key armed with its own time to
// live before the ask is put, deleted at its holder's first word, its expiry heard by watchBounds; the finding call-unanswered,
// scope call, id the ask, raised at the second bound and cleared when the ask is answered after all (H-A-238 (2)).
func askClock(ctx context.Context, bounds jetstream.KeyValue, bound, silent time.Duration, w callFindings, module string, now func() time.Time) *pipeline.AskClock {
return &pipeline.AskClock{Bound: bound, Silent: silent,
Arm: func(key string, ttl time.Duration) error {
_, err := bounds.Create(ctx, key, []byte(`{"ask":"`+key+`"}`), jetstream.KeyTTL(ttl))
if errors.Is(err, jetstream.ErrKeyExists) {
return nil // armed already: the same ask put again at a start keeps its first bound
}
return err
},
// A first word is heard for every ask, most of them unbounded or answered already: a key not there is deleted of
// nothing and writes no marker.
Unbound: func(key string) error {
if _, err := bounds.Get(ctx, key); errors.Is(err, jetstream.ErrKeyNotFound) {
return nil
} else if err != nil {
return err
}
if err := bounds.Delete(ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return err
}
return nil
},
Raise: func(onSeat, ask string, within time.Duration) error {
if w == nil {
return errors.New("the runner has no finding writer")
}
return w.Write(ctx, condseat.Finding{Scope: mcc.ScopeCall, ID: ask, Kind: mcc.KindCallUnanswered, Severity: condseat.FindingSeverityWarning,
Summary: fmt.Sprintf("the ask %s of the seat %s is unanswered", ask, onSeat),
Headline: "A step's ask is not answered",
Explanation: fmt.Sprintf("The runner put the ask %s on the seat %s and heard no word of its holder about it within %s, so its step ended could not run and its pipeline went on (H-A-235, H-A-243).", ask, onSeat, within),
Source: module + "." + mcc.KindCallUnanswered, Observer: module, At: now().UTC(), Subject: condseat.Subject{Scope: mcc.ScopeCall, ID: ask}})
},
Clear: func(_, ask string) error {
if w == nil {
return nil
}
return w.Clear(ctx, mcc.ScopeCall, ask, mcc.KindCallUnanswered, "the ask is answered")
}}
}
// answered deletes the bounds of the ask on onSeat at its holder's first word, said when it fails: the bound then passes and
// its expiry finds the step answered or asks it once more.
func answered(c *pipeline.AskClock, onSeat, ask string, logf func(string, ...any)) {
if err := c.Answered(onSeat, ask); err != nil {
logf("the bounds of ask %s on %s are not deleted at its holder's word: %v", ask, onSeat, err)
}
}
// hearWantsProgress takes a step progress of the seat mesh-wants as its holder's first word on the ask it names (H-A-238 (3)).
func hearWantsProgress(hub handlers, c *pipeline.AskClock, logf func(string, ...any)) error {
return hub.Handle(wants.SubjectStepProgressAll, func(m jetstream.Msg) {
var ev wants.StepProgressEvent
if err := json.Unmarshal(m.Data(), &ev); err != nil || ev.Ask == "" {
logf("%s is no step-progress: %v", m.Subject(), err)
_ = m.Term()
return
}
answered(c, wants.Seat, ev.Ask, logf)
_ = m.Ack()
})
}
// holdersNotReadDelay is how long a running heard before the seat-holders view is read waits before it is heard again.
const holdersNotReadDelay = 2 * time.Second
// hearHolderStarted puts again the open asks of each silent seat whose holder the node-engine says running (H-A-243 (1)).
func hearHolderStarted(hub handlers, started func(module string) (int, error), logf func(string, ...any)) error {
return hub.Handle(nodeengine.SubjectRunningAll, func(m jetstream.Msg) {
var ev nodeengine.Running
if err := json.Unmarshal(m.Data(), &ev); err != nil || ev.Module == "" {
logf("%s is no running: %v", m.Subject(), err)
_ = m.Term()
return
}
if ev.Stopped {
_ = m.Ack()
return
}
n, err := started(ev.Module)
switch {
case errors.Is(err, pipeline.ErrHoldersNotRead):
_ = m.NakWithDelay(holdersNotReadDelay) // heard again once the view is read, without spinning until then
return
case err != nil:
logf("the asks of the seats %s holds, put again as it runs on %s: %d, and one is not: %v", ev.Module, ev.Node, n, err)
}
_ = m.Ack()
})
}
// askBoundExpired takes the expiry of an ask's bound (H-A-235): the first asks once more, the second ends its step.
func askBoundExpired(unanswered func(onSeat, ask string, again bool) error, logf func(string, ...any)) func(key string) bool {
return func(key string) bool {
s, ask, again, ok := pipeline.ParseAskBound(key)
if !ok {
return false
}
if err := unanswered(s, ask, again); err != nil {
logf("the bound of ask %s on %s passed and is not taken: %v", ask, s, err)
}
return true
}
}
@@ -72,8 +72,8 @@ func dropUnanswered(ctx context.Context, w callFindings, module string, now func
}
}
// boundExpired takes the expiry of a key of the state bounds: a drop's says the drop unanswered (A-222); another key is none
// of the runner's, an ask having no bound of its own (§535, §647 as A-466y amends them).
// boundExpired takes the expiry of a drop's key of the state bounds: it says the drop unanswered (A-222); an ask's key is
// taken by askBoundExpired (H-A-235).
func boundExpired(unanswered func(buildSeat, ask string) error, logf func(string, ...any)) func(key string) {
return func(key string) {
if s, ask, ok := pipeline.ParseDropBound(key); ok {
@@ -2,9 +2,12 @@ package main
import (
"context"
"slices"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
@@ -50,3 +53,25 @@ func TestADropBoundsExpirySaysCallUnanswered(t *testing.T) {
t.Errorf("an answered drop: %v, cleared %v, unbound %v", err, w.cleared, unbound)
}
}
// H-A-238 (5), review: at a start, a first bound's expiry is taken only when the first read holds no entry for its second
// bound (put again and taken: a delete marker; put again: a live put; both passed: an expiry, the second taken alone); a
// second bound's expiry is taken; a delete marker or a drop's expiry is none.
func TestMissedBoundsAtTheStart(t *testing.T) {
const s = "node-build-go"
key := func(a string) string { return pipeline.AskBoundKey(s, a) }
again := func(a string) string { return pipeline.AskAgainBoundKey(s, pipeline.AgainID(a)) }
last := map[string]jetstream.KeyValueOp{
key("lostaaaa000001"): jetstream.KeyValuePurge, // passed while down, never asked again: taken
key("takenaaa000002"): jetstream.KeyValuePurge, again("takenaaa000002"): jetstream.KeyValueDelete, // asked again and taken
key("askedaaa000003"): jetstream.KeyValuePurge, again("askedaaa000003"): jetstream.KeyValuePut, // asked again, bound live
key("bothaaaa000004"): jetstream.KeyValuePurge, again("bothaaaa000004"): jetstream.KeyValuePurge, // both passed
key("wordaaaa000005"): jetstream.KeyValueDelete, // a first word
pipeline.DropBoundKey(s, "dropaaaa000006"): jetstream.KeyValuePurge,
}
want := []string{key("lostaaaa000001"), again("bothaaaa000004")}
slices.Sort(want)
if got := missedBounds(last); !slices.Equal(got, want) {
t.Errorf("missedBounds = %v, want %v", got, want)
}
}
@@ -67,8 +67,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
// The bound of each superseded step's drop: a key with its own time to live, its expiry heard (A-222, A-77).
bounds, err := process.EssentialBucket(ctx, js, "pipeline runner", ownBucket(cred.Module, "bounds"), "bound a drop no holder answers")
// The bound of each superseded step's drop and of each ask: a key with its own time to live, its expiry heard (A-222, A-77;
// H-A-238 (5)).
bounds, err := process.EssentialBucket(ctx, js, "pipeline runner", ownBucket(cred.Module, "bounds"), "bound a drop or an ask no holder answers")
if err != nil {
return err
}
@@ -149,7 +150,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
logger.Printf("widens: each read bound %s, %d catalog revisions kept", readBound, revisionsKept)
widener := widenerOf(calls.Call, readBound, revisionsKept, logger.Printf)
hearer.Widens = widener.Widens
hearer.Widens = widener.Reads
hearer.LogLast = stepLog{ctx: ctx, js: js, module: cred.Module}.Last
hearer.Where = routerRecordOf(cred.Module, calls.Call, calls.Ask)
hearer.Failures = &pipeline.Failures{Findings: rv.Findings, Observer: cred.Module, Author: commitAuthor(calls.Call), Logf: logger.Printf}
@@ -158,9 +159,26 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
starter := &pipeline.Starter{Pipelines: store, Logf: logger.Printf, Say: say,
Registration: rv.Registrations.Registration, Steps: busStepAsker{ctx: ctx, js: js, cred: cred}, Name: cred.Module,
Kept: keptReader(ctx, kept), Toolchain: toolchainReader(ctx, toolchains),
Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing}
Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing,
Findings: rv.Findings, Observer: cred.Module} // a mover waiting on its registration is registration-missing (H-A-262)
hearer.AskSteps = starter.AskSteps
hearer.RegistrationWaits = func() { starter.JudgeRegistrationMissing(store) } // at every end (H-A-265 (4))
// Every ask but a test run's is bounded until its holder's first word (H-A-235, H-A-238), its bounds keys of the state
// bounds.
askBound, err := askUnansweredBound(getenv)
if err != nil {
return err
}
silentBound, err := askSilentBound(getenv)
if err != nil {
return err
}
logger.Printf("asks: an ask unanswered bound %s, asked once more, then its step ends could not run; an ask of a seat that says nothing before its end bound %s", askBound, silentBound)
clock := askClock(ctx, bounds, askBound, silentBound, rv.Findings, cred.Module, time.Now)
starter.Asks, hearer.Asks = clock, clock
starter.Supersede = func(newer seat.Pipeline) { hearer.SupersedeRunning(newer) }
starter.Left = hearer.LeftInPlace // a same-source re-pin's superseded steps leave their runs (H-A-284)
starter.Revision = widener.Revision // a kept mover is placed from its kept revision (H-A-286)
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.AskPinHeld = func(key, module string, pins map[string]string) {
if _, err := starter.AskPinHeld(hearer.Store, key, module, pins, widener.Revision); err != nil {
@@ -256,6 +274,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := placeLostRuns(starter.UnplacedRuns, calls.CallKept, placeHalf, logger.Printf); err != nil {
logger.Printf("the runs not placed at the start: %v", err)
}
// A running pipeline nothing of which is still open ends now, its turn freed, before the queue is placed (H-A-288 (c)).
if n, err := hearer.EndAtStart(); err != nil || n > 0 {
logger.Printf("the pipelines with nothing open at the start: %d ended (%v)", n, err)
}
// The queue is placed at the start from its own state (§748).
if n, err := starter.StartQueue(rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the queue at the start: %d waiting pipelines placed (%v)", n, err)
@@ -297,10 +319,26 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
return hearer.RunTaken(ask, node)
}
if err := hearAskTaken(heard, taken, logger.Printf); err != nil {
if err := hearAskTaken(heard, taken, clock, logger.Printf); err != nil {
return err
}
expired := boundExpired(dropUnanswered(ctx, rv.Findings, cred.Module, time.Now), logger.Printf)
if err := hearWantsProgress(heard, clock, logger.Printf); err != nil {
return err
}
// The seat's write verbs are answered by the process that holds the pipelines (A-107); made before the bounds are heard,
// since an ask's expired bound asks it once more through them (H-A-235).
verbs := writeVerbs(hearer, starter, func(branch string) (pipeline.Half, bool) { return rv.Catalog.View(branch) }, calls)
verbs.Runs = pipeline.RunsRead{Runs: runsKept{ctx: ctx, kv: runsKV}, Object: runObjs.Get, Fill: starter.TestRunsFill}
if err := hearHolderStarted(heard, verbs.HolderStarted, logger.Printf); err != nil {
return err
}
dropExpired := boundExpired(dropUnanswered(ctx, rv.Findings, cred.Module, time.Now), logger.Printf)
askExpired := askBoundExpired(verbs.AskUnanswered, logger.Printf)
expired := func(key string) {
if !askExpired(key) {
dropExpired(key)
}
}
if err := watchBounds(ctx, bounds, expired, logger.Printf); err != nil {
return err
}
@@ -336,9 +374,6 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The steps' ends are kept seat events, each on its own stream (step-ended of the build seats, mesh-builds and
// mesh-wants), never on EVENTS: bound here, or no step of any pipeline is ever heard ending (G-1 §24).
logger.Printf("the kept seat events heard: %d streams bound", hub.StartKept(ctx, js, cred))
// The seat's write verbs are answered here, by the process that holds the pipelines (A-107).
verbs := writeVerbs(hearer, starter, func(branch string) (pipeline.Half, bool) { return rv.Catalog.View(branch) }, calls)
verbs.Runs = pipeline.RunsRead{Runs: runsKept{ctx: ctx, kv: runsKV}, Object: runObjs.Get, Fill: starter.TestRunsFill}
stopVerbs, err := answerVerbs(calls, seats, verbs)
if err != nil {
return err
@@ -107,6 +107,17 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
const router = "operator-router"
answer("mesh.seat."+seatlist.HolderSeat+".call."+seatlist.HolderVerb+".*", map[string]any{"seats": []any{
map[string]any{"seat": oc.Seat, "holders": []any{map[string]any{"module": router}}}}})
// Each mesh-builds step ask as its holder hears it, on its plain subscription.
var buildAsksMu sync.Mutex
buildAsks := map[string]int{}
if _, err := nc.Subscribe(builds.AcceptSubjectStep, func(m *nats.Msg) {
buildAsksMu.Lock()
buildAsks[m.Header.Get(nats.MsgIdHdr)]++
buildAsksMu.Unlock()
}); err != nil {
t.Fatal(err)
}
buildAsked := func(id string) int { buildAsksMu.Lock(); defer buildAsksMu.Unlock(); return buildAsks[id] }
records, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: router + "_" + oc.StateAsks, History: 1})
if err != nil {
t.Fatal(err)
@@ -153,6 +164,8 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
if err := os.WriteFile(waitingCap, []byte("1\n"), 0o600); err != nil {
t.Fatal(err)
}
askBound := filepath.Join(dir, "ask-unanswered-bound")
silentBound := filepath.Join(dir, "ask-silent-bound")
logs := &syncBuffer{}
start := func() func() {
runCtx, stop := context.WithCancel(ctx)
@@ -164,6 +177,10 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
return broker
case envPipelinesWaitingCap:
return waitingCap
case envAskUnansweredBound:
return askBound
case envAskSilentBound:
return silentBound
}
return ""
}, logs)
@@ -359,9 +376,16 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
}
// At the runner's next start the waiting main.10 is placed from its own state, saying wait-ended, which clears
// pipelines-stalled; the start begins no pipeline of its own.
// pipelines-stalled; the start begins no pipeline of its own. That start's ask-unanswered-bound is one second, its
// ask-silent-bound four.
stopRunner()
setMaxMsgSize(t, ctx, js, "KV_"+module+"_"+seat.StateQueue, -1)
if err := os.WriteFile(askBound, []byte("1s\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(silentBound, []byte("4s\n"), 0o600); err != nil {
t.Fatal(err)
}
before := pipelineKeys(t, ctx, pipelines)
stopRunner = start()
outcome("main.10", seat.PipelineOutcomeRunning)
@@ -370,6 +394,50 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
if after := pipelineKeys(t, ctx, pipelines); !slices.Equal(before, after) {
t.Fatalf("the start wrote pipelines %v, want %v", after, before)
}
// H-A-243: main.10's build ask on mesh-builds, whose holder says nothing before its end, is never asked again for
// ask-unanswered-bound (a build running four of those bounds is not touched); its one bound,
// ask-silent-bound, ends its step could not run "its ask <id> had no end", the pipeline ending failed, its turn freed and
// the finding call-unanswered naming the ask.
build := stepFor(get("main.10"), "app", seat.StepResultStepBuild)
if build.Ask == "" || build.Started == nil {
t.Fatalf("main.10's build = %+v, want it asked", build)
}
// Review of H-A-238 (5): the bound expires while the runner is down; at its next start the expiry marker it left is
// read and taken at once, never a fresh bound's full length later.
boundsKV, err := js.KeyValue(ctx, module+"_bounds")
if err != nil {
t.Fatal(err)
}
waitFor(t, ctx, logs, func() bool {
_, err := boundsKV.Get(ctx, pipeline.AskBoundKey(builds.Seat, build.Ask))
return err == nil
})
stopRunner()
waitFor(t, ctx, logs, func() bool {
_, err := boundsKV.Get(ctx, pipeline.AskBoundKey(builds.Seat, build.Ask))
return errors.Is(err, jetstream.ErrKeyNotFound)
})
restarted := time.Now()
stopRunner = start()
outcome("main.10", seat.PipelineOutcomeFailed)
if e := stepFor(get("main.10"), "app", seat.StepResultStepBuild).Ended; e == nil || e.Sub(restarted) >= 3*time.Second {
t.Fatalf("main.10's build ended %v, the runner restarted at %v; want its missed bound taken at the start", e, restarted)
}
if !strings.Contains(logs.String(), "expired while the runner was away: taken now") {
t.Fatal("the start did not say the bound it took")
}
s := stepFor(get("main.10"), "app", seat.StepResultStepBuild)
if s.Outcome != seat.StepResultOutcomeCouldNotRun || s.Why != "its ask "+build.Ask+" had no end" || s.Ended == nil || s.Ended.Sub(*build.Started) < 4*time.Second {
t.Fatalf("main.10's build = %+v, want could not run, its ask with no end, at its silent bound of 4s", s)
}
if strings.Contains(logs.String(), ": asked once more as ") || buildAsked(pipeline.AgainID(build.Ask)) != 0 {
t.Fatalf("the silent build's ask was asked again for ask-unanswered-bound (put %d times, again %d)", buildAsked(build.Ask), buildAsked(pipeline.AgainID(build.Ask)))
}
waitFor(t, ctx, logs, func() bool {
e, err := findings.Get(ctx, "call."+build.Ask+".call-unanswered")
return err == nil && strings.Contains(string(e.Value()), build.Ask) && len(running()) == 0
})
}
func stepFor(p seat.Pipeline, module, step string) seat.StepResult {
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"github.com/nats-io/nats.go/jetstream"
@@ -141,6 +142,8 @@ func hearStepEnded(hub handlers, h *pipeline.Hearer, logf func(string, ...any))
logf("the step-ended of ask %s is not taken yet, heard again: %v", ev.ID, err)
_ = m.Nak()
default:
// The step's end is its holder's word on the ask, the first for mesh-builds (H-A-238 (3)).
answered(h.Asks, bs.Seat, ev.ID, logf)
_ = m.Ack()
}
})
@@ -153,8 +156,9 @@ func hearStepEnded(hub handlers, h *pipeline.Hearer, logf func(string, ...any))
// hearAskTaken keeps on its step the node whose build seat holder took an ask (§602, §647; A-182), so a drop of it is asked of
// that node's holder, and begins the clock of a test run's ask (§725).
func hearAskTaken(hub handlers, taken func(ask, node string) error, logf func(string, ...any)) error {
func hearAskTaken(hub handlers, taken func(ask, node string) error, c *pipeline.AskClock, logf func(string, ...any)) error {
for _, ts := range pipeline.TakenSeats {
ts := ts
err := hub.Handle(ts.Subject, func(m jetstream.Msg) {
var ev nodebuildgo.AskTaken
if err := json.Unmarshal(m.Data(), &ev); err != nil || ev.Ask == "" {
@@ -167,6 +171,8 @@ func hearAskTaken(hub handlers, taken func(ask, node string) error, logf func(st
_ = m.Nak()
return
}
// The seat's ask-taken is its holder's first word on the ask (H-A-238 (3)).
answered(c, ts.Seat, ev.Ask, logf)
_ = m.Ack()
})
if err != nil {
@@ -176,14 +182,17 @@ func hearAskTaken(hub handlers, taken func(ask, node string) error, logf func(st
return nil
}
// watchBounds hears the end of each drop's bound, the bus's marker of the key's expiry, and says a drop no holder answered
// unanswered (A-222, A-77): no timer in the process.
// watchBounds hears the end of each drop's and each ask's bound, the bus's marker of the key's expiry (A-222, A-77; H-A-235,
// H-A-238 (5)): no timer in the process. An ask's bound that expired while the runner was down left its marker before the
// watch began: the watch's first entries, each key's last, are kept to their end, and the ask bounds missedBounds names are
// taken as expired then. Then every expiry is heard as it comes.
func watchBounds(ctx context.Context, bounds jetstream.KeyValue, expired func(key string), logf func(string, ...any)) error {
w, err := bounds.WatchAll(ctx, jetstream.UpdatesOnly())
w, err := bounds.WatchAll(ctx)
if err != nil {
return err
}
go func() {
last := map[string]jetstream.KeyValueOp{} // each key's last entry, until the first read ends
for {
select {
case <-ctx.Done():
@@ -193,16 +202,49 @@ func watchBounds(ctx context.Context, bounds jetstream.KeyValue, expired func(ke
if !ok {
return
}
if e == nil || e.Operation() != jetstream.KeyValuePurge {
if e == nil {
for _, key := range missedBounds(last) {
logf("the bound %s expired while the runner was away: taken now", key)
expired(key)
}
last = nil // the first read is over: what follows is new
continue
}
expired(e.Key())
if last != nil {
last[e.Key()] = e.Operation()
continue
}
if e.Operation() == jetstream.KeyValuePurge {
expired(e.Key())
}
}
}
}()
return nil
}
// missedBounds are the ask bounds the first read found expired that are still to be taken (H-A-238 (5)): a second bound's
// expiry always; a first bound's only when the read holds no entry at all for its second bound's key, since a live second
// bound, its expiry or its delete (the holder's ask-taken) says the ask was asked again already. A delete marker is a first
// word, never an expiry; a drop's bound is not taken here.
func missedBounds(last map[string]jetstream.KeyValueOp) []string {
var out []string
for key, op := range last {
if op != jetstream.KeyValuePurge {
continue
}
onSeat, ask, again, ok := pipeline.ParseAskBound(key)
if !ok {
continue
}
if _, asked := last[pipeline.AskAgainBoundKey(onSeat, pipeline.AgainID(ask))]; again || !asked {
out = append(out, key)
}
}
sort.Strings(out)
return out
}
// keptReader reads the kept results of the packages named (§25, §416): the entries of the state kept whose package is a
// name or lies beneath it, in one pass over the bucket.
func keptReader(ctx context.Context, kept jetstream.KeyValue) func([]string) (map[string][]nodebuildgo.KeptResult, error) {
@@ -16,6 +16,7 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
nodeengine "git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -99,6 +100,19 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
if _, err := holders.Put(ctx, "node-build-go", []byte(`{"holders":[{"module":"mesh-build-go"}]}`)); err != nil {
t.Fatal(err)
}
// mesh-builds is held by the builder, which takes its asks on a plain subscription and says nothing before an end.
if _, err := holders.Put(ctx, builds.Seat, []byte(`{"holders":[{"module":"mesh-delivery-builder"}]}`)); err != nil {
t.Fatal(err)
}
var buildAsksMu sync.Mutex
buildAsks := map[string]int{}
if _, err := nc.Subscribe(builds.AcceptSubjectStep, func(m *nats.Msg) {
buildAsksMu.Lock()
buildAsks[m.Header.Get(nats.MsgIdHdr)]++
buildAsksMu.Unlock()
}); err != nil {
t.Fatal(err)
}
// The registrations: app tests its program and tools bundle from one directory, its unit step required and its
// integration step optional; web is built by node-build-node.
@@ -331,6 +345,17 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
const p3 = "main.1"
takeIn("app", "c1", "src2:aa", 3, 1)
placed(p3) // started by the take-in alone: the runner starts no pipeline by sweep (design 47 H §747)
// H-A-243 (1): the node-engine's running of the builder puts its open build ask again, under its own id: an ask put while
// it was down is lost; a running of a module holding no silent seat puts nothing again.
firstBuild := step(read(p3), "", seat.StepResultStepBuild).Ask
publish(nodeengine.SubjectRunning("n1", "mesh-build-go"), nodeengine.Running{Module: "mesh-build-go", Node: "n1", At: time.Now().UTC()})
publish(nodeengine.SubjectRunning("n1", "mesh-delivery-builder"), nodeengine.Running{Module: "mesh-delivery-builder", Node: "n1", At: time.Now().UTC()})
waitFor(t, ctx, logs, func() bool {
buildAsksMu.Lock()
n := buildAsks[firstBuild]
buildAsksMu.Unlock()
return n == 2 && strings.Count(logs.String(), "the ask "+firstBuild+" of the build step of "+p3+" is put again on mesh-builds") == 1
})
passBuild(p3, "app", "c1", "src2:aa")
unitRun, unit := runOfStep(p3, "app:program", "unit")
integRun, integ := runOfStep(p3, "app:program", "integration")
@@ -350,7 +375,8 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
if s := step(read(p3), "app:tools", "unit"); s.Outcome != seat.StepResultOutcomeWaitingForABuildNode || s.Ask != "" || s.Shared[0].Ask != unit.Ask {
t.Fatalf("the tools' unit step = %+v; want it waiting for a build node, reading the program's run", s)
}
// Its wait has no bound (§647 as A-466y amends it): the state bounds holds no key for a queued ask, only drops' keys.
// A test run's ask keeps §725's bounds alone (H-A-238 (4)): the state bounds holds no key naming it, only drops' and other
// asks' keys.
boundsKV, err := js.KeyValue(ctx, module+"_bounds")
if err != nil {
t.Fatal(err)
@@ -359,8 +385,9 @@ func TestTestRunsAreAskedOncePerLevelAndMoveEveryReader(t *testing.T) {
t.Fatal(err)
} else {
for _, k := range ks {
if !strings.HasPrefix(k, pipeline.DropBoundPrefix) {
t.Errorf("the state bounds holds %s after the asks were put; want no bound of a queued ask", k)
_, ask, _, isAsk := pipeline.ParseAskBound(k)
if (!isAsk && !strings.HasPrefix(k, pipeline.DropBoundPrefix)) || ask == unit.Ask || ask == integ.Ask {
t.Errorf("the state bounds holds %s after the asks were put; want no bound of a test run's ask", k)
}
}
}
+1 -1
View File
@@ -10,7 +10,7 @@ require (
require (
git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-00010101000000-000000000000 // indirect
git.novox.be/novox/mesh-contracts/go/seats/nodeengine v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient v0.0.0-00010101000000-000000000000
git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions v0.0.0-00010101000000-000000000000
@@ -206,6 +206,7 @@ func (h *Hearer) OnDecided(event oc.Warrant) (seat.Pipeline, error) {
return seat.Pipeline{}, err
}
for _, key := range found {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
@@ -219,7 +220,7 @@ func (h *Hearer) OnDecided(event oc.Warrant) (seat.Pipeline, error) {
if err != nil || !changed {
return p, err
}
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
i := stepOfAsk(p, w.Ask)
if i >= 0 {
p.Unlogged = append(p.Unlogged, endEntry(key, p.Steps[i], now))
@@ -0,0 +1,350 @@
package pipeline
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"strings"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
)
// DefaultAskUnansweredBound is the setting ask-unanswered-bound's default (design 47 H H-A-235, H-A-238 (1)): ten minutes
// from an ask until its holder's first word.
const DefaultAskUnansweredBound = 10 * time.Minute
// DefaultAskSilentBound is the setting ask-silent-bound's default (H-A-243 (2)): forty-five minutes from an ask on a seat whose
// holder says no word before the step's end (mesh-builds, mesh-wants) until that end.
const DefaultAskSilentBound = 45 * time.Minute
// Silent says whether the holder of a seat the runner asks says nothing of an ask before its step's end: every seat but the
// build seats, which say ask-taken (H-A-241, H-A-243).
func Silent(onSeat string) bool { return !BoundSeat(onSeat) }
// AgainID is the id an ask is put again under (H-A-243 (3)): the first id's time part and six hex digits of the sha256 of the
// first id and ".2", the form the holders accept, so no duplicate window is relied on.
func AgainID(id string) string {
if len(id) <= 6 {
return id
}
sum := sha256.Sum256([]byte(id + ".2"))
return id[:len(id)-6] + hex.EncodeToString(sum[:3])
}
// The keys of the state bounds that time an ask the runner put (H-A-238 (5)): `ask.<seat>.<ask>` armed before the ask is put,
// `ask2.<seat>.<ask>` armed when it is asked once more, each with the setting as its own time to live.
const (
AskBoundPrefix = "ask."
AskAgainBoundPrefix = "ask2."
)
// AskBoundKey is the key of the first bound of the ask on onSeat; AskAgainBoundKey that of its second.
func AskBoundKey(onSeat, ask string) string { return AskBoundPrefix + onSeat + "." + ask }
func AskAgainBoundKey(onSeat, ask string) string { return AskAgainBoundPrefix + onSeat + "." + ask }
// ParseAskBound reads a key of the state bounds as an ask's bound: the seat, the ask and whether it is the second bound.
func ParseAskBound(key string) (onSeat, ask string, again, ok bool) {
rest, found := strings.CutPrefix(key, AskAgainBoundPrefix)
again = found
if !found {
if rest, found = strings.CutPrefix(key, AskBoundPrefix); !found {
return "", "", false, false
}
}
onSeat, ask, found = strings.Cut(rest, ".")
return onSeat, ask, again, found && onSeat != "" && ask != "" && !strings.Contains(ask, ".")
}
// UnansweredWhy is the why of a step whose ask was not answered within its two bounds (H-A-235).
func UnansweredWhy(ask string) string { return "its ask " + ask + " was not answered" }
// SilentWhy is the why of a step on a silent seat whose ask had no end within ask-silent-bound (H-A-243 (2)).
func SilentWhy(ask string) string { return "its ask " + ask + " had no end" }
// AskClock bounds every ask the runner puts on a seat but a test run's, which keeps §725's bounds (H-A-235, H-A-238 (4)):
// Arm writes a key of the state bounds with Bound as its own time to live, Unbound deletes one; Raise raises the finding
// call-unanswered of an ask that ended its step, Clear clears it once the ask is answered after all (H-A-238 (2)).
type AskClock struct {
// Bound is ask-unanswered-bound, for a seat that says ask-taken; Silent is ask-silent-bound, for a silent seat.
Bound time.Duration
Silent time.Duration
Arm func(key string, ttl time.Duration) error
Unbound func(key string) error
Raise func(onSeat, ask string, bound time.Duration) error
Clear func(onSeat, ask string) error
}
// arm arms the first bound of a, before it is put (H-A-238 (5)); a test run's ask, and a nil clock, arm none.
func (c *AskClock) arm(a Asked) error {
if c == nil || c.Arm == nil || a.Run != "" {
return nil
}
if err := c.Arm(AskBoundKey(a.Seat, a.ID), c.boundOf(a.Seat)); err != nil {
return fmt.Errorf("the bound of ask %s is not armed: %w", a.ID, err)
}
return nil
}
// boundOf is the bound of an ask on onSeat: ask-silent-bound for a silent seat (H-A-243 (2)), else ask-unanswered-bound.
func (c *AskClock) boundOf(onSeat string) time.Duration {
if !Silent(onSeat) {
return c.bound()
}
if c.Silent > 0 {
return c.Silent
}
return DefaultAskSilentBound
}
func (c *AskClock) bound() time.Duration {
if c.Bound > 0 {
return c.Bound
}
return DefaultAskUnansweredBound
}
// Answered deletes the bounds of the ask heard on onSeat, its holder's first word: an ask-taken, a step progress or, for a
// silent seat, the step's end (H-A-238 (3), H-A-241). Heard under a first id, the second bound of the ask put again under
// AgainID goes too.
func (c *AskClock) Answered(onSeat, ask string) error {
if c == nil || c.Unbound == nil || ask == "" {
return nil
}
return errors.Join(c.Unbound(AskBoundKey(onSeat, ask)), c.Unbound(AskAgainBoundKey(onSeat, ask)),
c.Unbound(AskAgainBoundKey(onSeat, AgainID(ask))))
}
// answeredLate clears the finding of the ask of step s, which ended unanswered or with no end, its holder heard after all
// (H-A-238 (2)).
func (c *AskClock) answeredLate(s seat.StepResult, logf func(string, ...any)) {
if c == nil || c.Clear == nil || s.Ask == "" || (s.Why != UnansweredWhy(s.Ask) && s.Why != SilentWhy(s.Ask)) {
return
}
if err := c.Clear(s.Seat, s.Ask); err != nil {
logf("the finding of ask %s, answered after its step ended, is not cleared: %v", s.Ask, err)
}
}
// AskUnanswered takes the expiry of a bound of the ask on onSeat (H-A-235, H-A-243). On a silent seat its one bound,
// ask-silent-bound, ends the step could not run "its ask <id> had no end", never asking again (that would double a running
// build). On a seat that says ask-taken the first bound puts the ask once more under AgainID, said in the journal, its
// second bound armed under that id; the second ends the step could not run "its ask <id> was not answered". An ended step
// raises the finding call-unanswered, so the pipeline ends and its turn frees. A bound whose ask no running step names any
// longer asks nothing.
func (v Verbs) AskUnanswered(onSeat, ask string, again bool) error {
h, c := v.Hearer, v.Starter.Asks
if c == nil {
return nil
}
key, p, i, err := v.runningAsk(onSeat, ask)
if err != nil || key == "" {
return err
}
s := p.Steps[i]
if Silent(onSeat) || again {
why, within := UnansweredWhy(s.Ask), 2*c.bound()
if Silent(onSeat) {
why, within = SilentWhy(s.Ask), c.boundOf(onSeat)
}
h.logf("the ask %s of the %s step of %s on %s had no word from its holder within %s: the step ends could not run (H-A-235, H-A-243)",
s.Ask, s.Step, key, onSeat, within)
if err := h.endUnanswered(key, s.Ask, why); err != nil {
return err
}
if c.Raise != nil {
return c.Raise(onSeat, s.Ask, within)
}
return nil
}
a, err := v.askAgain(p, s, AgainID(s.Ask), h.now())
if err != nil {
h.logf("the ask %s of the %s step of %s on %s had no word from its holder within %s and cannot be put again: %v", s.Ask, s.Step, key, onSeat, c.bound(), err)
} else {
h.logf("the ask %s of the %s step of %s on %s had no word from its holder within %s: asked once more as %s (H-A-235, H-A-243)", s.Ask, s.Step, key, onSeat, c.bound(), a.ID)
}
if c.Arm != nil {
if aerr := c.Arm(AskAgainBoundKey(onSeat, AgainID(s.Ask)), c.bound()); aerr != nil {
return fmt.Errorf("the second bound of ask %s is not armed, so it is not asked again: %w", s.Ask, aerr)
}
}
if err != nil {
return nil // its second bound ends the step
}
v.Starter.carry(key, []Asked{*a})
return askAll(v.Starter.Steps, key, []Asked{*a}, func(commit string) bool { return v.Starter.recovered(key, commit) }, nil)
}
// HolderStarted puts again every open ask of each silent seat module holds (H-A-243 (1)): its holder takes its asks on a plain
// subscription, so an ask put while it was down is lost; heard on the node-engine's running of module. Each ask is put under
// its own id, a rerun's marks kept, said in the journal; its bound stands as armed.
func (v Verbs) HolderStarted(module string) (int, error) {
st := v.Starter
if st == nil || st.Holders == nil {
return 0, nil
}
var silent []string
for onSeat := range acceptStep {
if !Silent(onSeat) {
continue
}
holders, read := st.Holders(onSeat)
if !read {
return 0, ErrHoldersNotRead
}
if containsString(holders, module) {
silent = append(silent, onSeat)
}
}
if len(silent) == 0 {
return 0, nil
}
pkeys, err := allKeys(v.Hearer.Store)
if err != nil {
return 0, fmt.Errorf("the pipelines are not listed: %w", err)
}
now, asked := v.Hearer.now(), 0
var first error
for _, key := range pkeys {
p, _, err := v.Hearer.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning {
continue
}
seen := map[string]bool{}
var due []Asked
for _, s := range p.Steps {
if !open(s) || s.Ask == "" || seen[s.Ask] || !containsString(silent, s.Seat) {
continue
}
seen[s.Ask] = true
a, err := v.askAgain(p, s, s.Ask, now)
if err != nil {
v.Hearer.logf("the ask %s of the %s step of %s on %s is not put again as %s starts: %v", s.Ask, s.Step, key, s.Seat, module, err)
continue
}
v.Hearer.logf("the ask %s of the %s step of %s is put again on %s: its holder %s started, and takes no ask put while it was down (H-A-243)",
s.Ask, s.Step, key, s.Seat, module)
due = append(due, *a)
}
if len(due) == 0 {
continue
}
st.carry(key, due)
if err := askAll(st.Steps, key, due, func(commit string) bool { return st.recovered(key, commit) }, nil); err != nil && first == nil {
first = err
}
asked += len(due)
}
return asked, first
}
// ErrHoldersNotRead is HolderStarted's answer before the seat-holders view's first view is read: the running is heard again.
var ErrHoldersNotRead = errors.New("the seat-holders view is not read yet")
func containsString(list []string, s string) bool {
for _, x := range list {
if x == s {
return true
}
}
return false
}
// runningAsk is the pipeline and the index of its running step asked under ask, or put again under it, of onSeat; an empty
// key when none is.
func (v Verbs) runningAsk(onSeat, ask string) (string, seat.Pipeline, int, error) {
pkeys, err := allKeys(v.Hearer.Store)
if err != nil {
return "", seat.Pipeline{}, -1, fmt.Errorf("the pipelines are not listed: %w", err)
}
for _, key := range pkeys {
p, _, err := v.Hearer.Store.Get(key)
if err != nil {
continue
}
if i := stepOfAsk(p, ask); i >= 0 && open(p.Steps[i]) && p.Steps[i].Seat == onSeat {
return key, p, i, nil
}
}
return "", seat.Pipeline{}, -1, nil
}
// askAgain is the ask of running step s as it was put, under id: a rerun's marks kept.
func (v Verbs) askAgain(p seat.Pipeline, s seat.StepResult, id string, now time.Time) (*Asked, error) {
a, err := v.placedAsk(p, s, id, now)
if err != nil {
return nil, err
}
if s.Rerun {
markRerun(a, s.By)
}
return a, nil
}
// markRerun marks a's body a rerun by caller.
func markRerun(a *Asked, caller string) {
switch b := a.Ask.(type) {
case nodebuildgo.StepAsk:
if b.Step != nil {
step := *b.Step
step.Rerun, step.By = true, caller
b.Step = &step
}
a.Ask = b
case builds.StepRun:
b.Rerun, b.By = true, caller
a.Ask = b
case wants.StepRun:
b.Rerun, b.By = true, caller
a.Ask = b
}
}
// endUnanswered ends the running step of pipeline key asked under ask could not run, saying why (H-A-235, H-A-243): every
// other mover's validate reading the ask with it, a composition its validate held ended not asked, the pipeline ended when
// nothing else is open, said and logged as any step's end.
func (h *Hearer) endUnanswered(key, ask, why string) error {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
return nil
}
i := stepOfAsk(p, ask)
if i < 0 || !open(p.Steps[i]) {
return nil
}
now := h.now()
s := &p.Steps[i]
s.Outcome, s.Why, s.Ended = seat.StepResultOutcomeCouldNotRun, why, &now
p.Unlogged = append(p.Unlogged, endEntry(key, *s, now))
var readers []int
if s.Step == seat.StepResultStepValidate {
readers = sameValidate(&p, i, now)
_, notAsked := releaseCompositions(&p, now, h.kept(key))
readers = append(readers, notAsked...)
}
asked, err := askApproval(key, &p, now, h.Asker)
if err != nil {
return err
}
ended := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
return fmt.Errorf("pipeline %s cannot be written: %w", key, err)
}
h.Failures.StepEnded(h.Store, p, *s)
for _, j := range append([]int{i}, readers...) {
if err := h.say(seat.EventStepEnded, p, &p.Steps[j]); err != nil {
return fmt.Errorf("pipeline %s's %s is written but not said: %w", key, seat.EventStepEnded, err)
}
}
h.appendUnlogged(key)
return h.said(p, asked, ended)
}
}
@@ -0,0 +1,197 @@
package pipeline
import (
"slices"
"strings"
"testing"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
)
// H-A-235, H-A-238: every ask but a test run's is bounded from its put until its holder's first word. A bound that passes asks
// once more under the ask's own id, its second bound armed; past the second the step ends could not run "its ask <id> was not
// answered", the finding raised, and the pipeline ends; a first word deletes both bounds, and a late one clears the finding.
func TestAnAskNobodyAnswersIsAskedOnceMoreThenEndsItsStep(t *testing.T) {
v, store, said, asks, _, _ := verbsFixture()
var armed, unbound, raised, cleared []string
logs := []string{}
v.Hearer.Logf = func(f string, a ...any) { logs = append(logs, f) }
c := &AskClock{Bound: 10 * time.Minute,
Arm: func(key string, ttl time.Duration) error { armed = append(armed, key+" "+ttl.String()); return nil },
Unbound: func(key string) error { unbound = append(unbound, key); return nil },
Raise: func(s, ask string, _ time.Duration) error { raised = append(raised, s+" "+ask); return nil },
Clear: func(s, ask string) error { cleared = append(cleared, s+" "+ask); return nil }}
v.Starter.Asks, v.Hearer.Asks = c, c
// A test run's ask keeps §725's bounds; a nil clock arms none.
if err := c.arm(Asked{Seat: nodebuildgo.Seat, ID: "r1", Run: "run"}); err != nil || len(armed) != 0 {
t.Fatalf("a test run's ask armed %v (%v)", armed, err)
}
if err := (*AskClock)(nil).arm(Asked{Seat: nodebuildgo.Seat, ID: "x"}); err != nil {
t.Fatal(err)
}
// Every step but app:program's lint has ended; that lint failed, and its rerun's ask is armed before it is put.
p, rev, _ := store.Get(pkey)
for i := range p.Steps {
if open(p.Steps[i]) {
p.Steps[i].Outcome = seat.StepResultOutcomePassed
}
}
p.Steps[stepIndex(p, "lint", "app:program")].Outcome = seat.StepResultOutcomeFailed
store.Update(pkey, p, rev)
r, err := v.Rerun("operator", seat.RerunInput{Pipeline: pkey, Step: "lint", Artifact: "app:program", Why: "retry"})
if err != nil {
t.Fatal(err)
}
ask := r.Result.Ask
if want := []string{AskBoundKey(nodebuildgo.Seat, ask) + " 10m0s"}; !slices.Equal(armed, want) || len(asks.ids) != 1 {
t.Fatalf("armed %v asking %v, want %v before one ask", armed, asks.ids, want)
}
// The first bound passes: asked once more under the id derived from the first, in the form the holders accept, a rerun
// still, the second bound armed under it and said (H-A-243 (3)).
again := AgainID(ask)
if err := sdkstep.CheckAskID(again); err != nil || again == ask || again[:len(again)-6] != ask[:len(ask)-6] {
t.Fatalf("AgainID(%s) = %s (%v), want the first's time part and other hex digits in the ask id form", ask, again, err)
}
if err := v.AskUnanswered(nodebuildgo.Seat, ask, false); err != nil {
t.Fatal(err)
}
if len(asks.ids) != 2 || asks.ids[1] != again || armed[1] != AskAgainBoundKey(nodebuildgo.Seat, again)+" 10m0s" {
t.Fatalf("asked %v, armed %v, want %s once more and its second bound", asks.ids, armed, again)
}
if b, ok := asks.asks[1].(nodebuildgo.StepAsk); !ok || b.ID != again || b.Step == nil || !b.Step.Rerun || b.Step.By != "operator" {
t.Errorf("the ask put again = %+v", asks.asks[1])
}
if !slices.ContainsFunc(logs, func(l string) bool { return strings.Contains(l, "asked once more") }) {
t.Errorf("no journal line names the ask asked again: %v", logs)
}
if got, _, _ := store.Get(pkey); got.Steps[stepIndex(got, "lint", "app:program")].Outcome != seat.StepResultOutcomeRunning {
t.Fatal("the step ended at the first bound")
}
// The second bound passes: the step ends could not run, the finding is raised, the pipeline ends failed.
if err := v.AskUnanswered(nodebuildgo.Seat, again, true); err != nil {
t.Fatal(err)
}
got, _, _ := store.Get(pkey)
s := got.Steps[stepIndex(got, "lint", "app:program")]
if s.Outcome != seat.StepResultOutcomeCouldNotRun || s.Why != "its ask "+ask+" was not answered" || s.Ended == nil {
t.Errorf("the step = %+v", s)
}
if got.Outcome != seat.PipelineOutcomeFailed || len(asks.ids) != 2 {
t.Errorf("the pipeline is %s (%s), asked %d, want failed and nothing more asked", got.Outcome, got.Why, len(asks.ids))
}
if !slices.Equal(raised, []string{nodebuildgo.Seat + " " + ask}) || !slices.Contains(*said, "pipeline-ended.main.5") {
t.Errorf("raised %v, said %v", raised, *said)
}
// A bound whose step ended asks nothing; a first word deletes both bounds; a late end clears the finding.
if err := v.AskUnanswered(nodebuildgo.Seat, ask, false); err != nil || len(asks.ids) != 2 {
t.Errorf("a bound of an ended step asked %v (%v)", asks.ids, err)
}
if err := c.Answered(nodebuildgo.Seat, ask); err != nil || !slices.Equal(unbound, []string{AskBoundKey(nodebuildgo.Seat, ask), AskAgainBoundKey(nodebuildgo.Seat, ask), AskAgainBoundKey(nodebuildgo.Seat, again)}) {
t.Errorf("a first word deleted %v (%v)", unbound, err)
}
if err := v.Hearer.Taken(again, "n1"); err != nil || !slices.Equal(cleared, []string{nodebuildgo.Seat + " " + ask}) {
t.Errorf("a late ask-taken cleared %v (%v)", cleared, err)
}
for _, k := range []string{AskBoundKey("mesh-wants", "a1"), AskAgainBoundKey("mesh-wants", "a1")} {
if s, a, again, ok := ParseAskBound(k); !ok || s != "mesh-wants" || a != "a1" || again != strings.HasPrefix(k, AskAgainBoundPrefix) {
t.Errorf("ParseAskBound(%s) = %s %s %v %v", k, s, a, again, ok)
}
}
if _, _, _, ok := ParseAskBound(DropBoundKey("node-build-go", "a1")); ok {
t.Error("a drop's bound read as an ask's")
}
}
// H-A-243: an ask on a seat whose holder says nothing before the step's end (mesh-builds) is never asked again for a bound:
// its one bound, ask-silent-bound, ends the step could not run "its ask <id> had no end", the finding raised. The node-engine's
// running of the seat's holder puts every open ask of the seat again under its own id, a module holding no such seat none.
func TestASilentSeatsAskIsPutAgainWhenItsHolderStartsAndEndsAtItsSilentBound(t *testing.T) {
v, store, _, asks, _, _ := verbsFixture()
var armed, raised []string
c := &AskClock{Bound: time.Minute, Silent: 45 * time.Minute,
Arm: func(key string, ttl time.Duration) error { armed = append(armed, key+" "+ttl.String()); return nil },
Raise: func(s, ask string, within time.Duration) error {
raised = append(raised, s+" "+ask+" "+within.String())
return nil
}}
v.Starter.Asks, v.Hearer.Asks = c, c
v.Starter.Holders = func(s string) ([]string, bool) {
return map[string][]string{builds.Seat: {"mesh-delivery-builder"}}[s], true
}
p, rev, _ := store.Get(pkey)
// Every step has ended but app's build, asked of mesh-builds.
b := stepIndex(p, seat.StepResultStepBuild, "")
for i := range p.Steps {
if open(p.Steps[i]) && i != b {
p.Steps[i].Outcome = seat.StepResultOutcomePassed
}
}
if p.Steps[b].Seat != builds.Seat || !open(p.Steps[b]) {
t.Fatalf("the build step = %+v, want running on mesh-builds", p.Steps[b])
}
store.Update(pkey, p, rev)
build := p.Steps[b].Ask
if err := c.arm(Asked{Seat: builds.Seat, ID: build}); err != nil || !slices.Equal(armed, []string{AskBoundKey(builds.Seat, build) + " 45m0s"}) {
t.Fatalf("armed %v (%v), want the silent bound", armed, err)
}
// A module holding no silent seat puts nothing again; the builder's running puts the build again under its own id.
if n, err := v.HolderStarted("mesh-build-go"); err != nil || n != 0 || len(asks.ids) != 0 {
t.Fatalf("another holder's start put %d (%v)", n, err)
}
if n, err := v.HolderStarted("mesh-delivery-builder"); err != nil || n != 1 || len(asks.ids) != 1 || asks.ids[0] != build {
t.Fatalf("the builder's start put %d asks %v (%v), want %s again", n, asks.ids, err, build)
}
if _, ok := asks.asks[0].(builds.StepRun); !ok || asks.at[0] != builds.AcceptSubjectStep {
t.Errorf("the ask put again = %+v on %s", asks.asks[0], asks.at[0])
}
// The silent bound passes: no ask again, the step ends could not run, the finding is raised, the pipeline ends.
if err := v.AskUnanswered(builds.Seat, build, false); err != nil {
t.Fatal(err)
}
got, _, _ := store.Get(pkey)
s := got.Steps[b]
if len(asks.ids) != 1 || s.Outcome != seat.StepResultOutcomeCouldNotRun || s.Why != "its ask "+build+" had no end" || got.Outcome != seat.PipelineOutcomeFailed {
t.Errorf("asked %v; the build %+v; the pipeline %s", asks.ids, s, got.Outcome)
}
if !slices.Equal(raised, []string{builds.Seat + " " + build + " 45m0s"}) {
t.Errorf("raised %v", raised)
}
}
// H-A-252: the pipeline's one validate ask names no mover (H-A-51), and its holder's end names the mover whose body it took:
// it is matched by its ask alone and ends the validate step of every mover under it, whichever mover it names.
func TestAValidateEndNamingOneMoverEndsEveryMoversValidate(t *testing.T) {
h, store, _, _, _, _ := fixture()
p, rev, _ := store.Get(pkey)
p.Movers = []seat.Mover{{Module: "app", Commit: "c1", Fingerprint: "src2:aa"}, {Module: "lib", Commit: "c2", Fingerprint: "src2:bb"}}
p.Steps = []seat.StepResult{
{Module: "app", Step: seat.StepResultStepValidate, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomeRunning, Ask: "v1", Seat: wants.Seat},
{Module: "lib", Step: seat.StepResultStepValidate, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomeRunning, Ask: "v1", Seat: wants.Seat}}
store.Update(pkey, p, rev)
var bs StepSeat
for _, x := range StepSeats {
if x.Seat == wants.Seat {
bs = x
}
}
ev := StepEnded{ID: "v1", Node: "n1", Pipeline: pkey, Module: "lib", Commit: "c2", Fingerprint: "src2:bb", Step: seat.StepResultStepValidate, Outcome: "passed"}
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", ev); err != nil {
t.Fatal(err)
}
got, _, _ := store.Get(pkey)
for _, s := range got.Steps {
if s.Outcome != seat.StepResultOutcomePassed {
t.Errorf("%s's validate = %s, want passed by the one end", s.Module, s.Outcome)
}
}
if got.Outcome != seat.PipelineOutcomePassed {
t.Errorf("the pipeline = %s (%s), want passed", got.Outcome, got.Why)
}
}
@@ -11,8 +11,8 @@ import (
)
// DropBound is how long a superseded step's drop may go unanswered before it is said (design 47 G A-222, A-77): five minutes,
// a key of the state `bounds` with its own time to live. An ask waiting for its holder has no such bound (§535, §647 as
// A-466y amends them): it is timed by §86's build-capacity finding and, for a test run, by run-untaken-bound (§725).
// a key of the state `bounds` with its own time to live. An ask waiting for its holder is bounded by ask-unanswered-bound
// (H-A-235, H-A-238; AskClock), a test run's by run-untaken-bound (§725).
const DropBound = 5 * time.Minute
// TakenSeat is a build seat whose holder says `ask-taken` when it takes an ask (design 47 G §602): where the runner hears it.
@@ -0,0 +1,129 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/identity-and-access/go/seats/meshgrants"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
)
// refusedOnce is a store whose first write of a pipeline is refused as changed, another write having come between (the
// pipeline's key moving while its end is judged, as main.117's did).
type refusedOnce struct {
memStore
refused map[string]bool
}
func (r *refusedOnce) Update(key string, p seat.Pipeline, rev uint64) (uint64, error) {
if !r.refused[key] {
r.refused[key] = true
e := r.memStore[key]
e.rev++
r.memStore[key] = e
return 0, ErrChanged
}
return r.memStore.Update(key, p, rev)
}
// H-A-288, H-A-289 (met with main.117 and main.76): a pipeline ends, and the next waiting is placed, the moment nothing of it is
// still open: when its last mover leaves, when its last open step ends by the silent bound, and at the runner's start pass;
// its outcome is said in the journal. The reads its end needs are made once, before the compare-and-set, shared by every mover
// and every retry of the write: a write refused once is written at the retry without asking grants-for again.
func TestAPipelineWithNothingOpenEndsAtOnce(t *testing.T) {
t0 := time.Date(2026, 11, 21, 17, 45, 19, 0, time.UTC)
movers := []seat.Mover{{Module: "a", Commit: "c1", Fingerprint: "src2:a1"}, {Module: "b", Commit: "c1", Fingerprint: "src2:b1"},
{Module: "c", Commit: "c1", Fingerprint: "src2:c1"}}
step := func(module, outcome, ask string) seat.StepResult {
return seat.StepResult{Module: module, Step: seat.StepResultStepBuild, Status: seat.StepResultStatusRequired, Outcome: outcome, Ask: ask, Seat: builds.Seat}
}
rows := []struct {
name string
steps []seat.StepResult
end func(t *testing.T, h *Hearer)
outcome string
}{
{"its last mover leaves for a newer pipeline (H-A-223)",
[]seat.StepResult{step("a", seat.StepResultOutcomeRunning, "x1"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
h.SupersedeRunning(seat.Pipeline{Pipeline: "main.120", Branch: "main", Revision: 120, Movers: []seat.Mover{
{Module: "a", Commit: "c2", Fingerprint: "src2:a2"}, {Module: "b", Commit: "c2", Fingerprint: "src2:b2"}, {Module: "c", Commit: "c2", Fingerprint: "src2:c2"}}})
}, seat.PipelineOutcomeSuperseded},
{"its last open step ends could not run on the silent bound (H-A-243)",
[]seat.StepResult{step("a", seat.StepResultOutcomeRunning, "mwql5pio3390aa"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
if err := h.endUnanswered("main.117", "mwql5pio3390aa", SilentWhy("mwql5pio3390aa")); err != nil {
t.Fatal(err)
}
}, seat.PipelineOutcomeFailed},
{"the runner's start pass finds it with nothing open",
[]seat.StepResult{step("a", seat.StepResultOutcomePassed, "x1"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
if n, err := h.EndAtStart(); n != 1 || err != nil {
t.Fatalf("the start pass ended %d (%v), want 1", n, err)
}
}, seat.PipelineOutcomePassed},
}
for _, r := range rows {
t.Run(r.name, func(t *testing.T) {
store := &refusedOnce{memStore: memStore{}, refused: map[string]bool{}}
store.memStore["main.117"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.117", Branch: "main", Source: 117, Revision: 117, Outcome: seat.PipelineOutcomeRunning,
Movers: slices.Clone(movers), Steps: slices.Clone(r.steps)}, 1}
// Another running pipeline with a step open, and one waiting its turn: neither ends.
store.memStore["main.118"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.118", Branch: "other", Source: 118, Revision: 118, Outcome: seat.PipelineOutcomeRunning,
Movers: movers[:1], Steps: []seat.StepResult{step("a", seat.StepResultOutcomeRunning, "y1")}}, 1}
store.memStore["main.119"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.119", Branch: "other", Source: 119, Revision: 119, Outcome: seat.PipelineOutcomeWaiting, Movers: movers[:1]}, 1}
store.refused["main.118"], store.refused["main.119"] = true, true
grants, revisions := 0, 0
w := &Widener{
Revision: func(string, int64) (Half, error) { revisions++; return Half{}, nil },
Holder: func(s string) (string, error) { return s, nil },
GrantsFor: func(string, int64) (meshgrants.GrantsFor, error) {
grants++
return meshgrants.GrantsFor{}, fmt.Errorf("cut after 5s")
},
}
var turned, journal []string
h := &Hearer{Store: store, Say: func(string, any) error { return nil }, Now: func() time.Time { return t0 }, Widens: w.Reads,
Turn: func(key string) { turned = append(turned, key) },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) }}
r.end(t, h)
p, _, _ := store.Get("main.117")
if p.Outcome != r.outcome {
t.Fatalf("main.117 = %s (%s), want %s", p.Outcome, p.Why, r.outcome)
}
if !slices.Equal(turned, []string{"main.117"}) {
t.Errorf("the turns freed = %v, want main.117's alone, so the next waiting is placed", turned)
}
if !slices.ContainsFunc(journal, func(l string) bool { return strings.HasPrefix(l, "main.117 ended "+r.outcome) }) {
t.Errorf("the journal does not say main.117's end %s: %q", r.outcome, journal)
}
if grants != 1 || revisions != 2 {
t.Errorf("the end asked grants-for %d times and a revision %d times over 3 movers and a refused write, want 1 and 2", grants, revisions)
}
for _, m := range p.Movers {
if len(m.Widens) != 1 || m.Widens[0].Name != "mesh-grants" {
t.Errorf("%s's widens = %+v, want the cut grants-for said", m.Module, m.Widens)
}
}
for _, k := range []string{"main.118", "main.119"} {
if q, _, _ := store.Get(k); q.Outcome == seat.PipelineOutcomeSuperseded || q.Outcome == seat.PipelineOutcomePassed || q.Outcome == seat.PipelineOutcomeFailed {
t.Errorf("%s ended %s with a step open or waiting its turn", k, q.Outcome)
}
}
})
}
}
@@ -288,6 +288,65 @@ func (s *MoverStore) Get(key string) (seat.Pipeline, uint64, error) {
// when the record or a mover key was written since the read revision gave. A record carrying its steps inline has its
// mover keys written before the record, so its steps are never lost between the two.
func (s *MoverStore) Update(key string, p seat.Pipeline, revision uint64) (uint64, error) {
return s.update(key, p, revision, nil)
}
// UpdateKept is Update with kept laid over the record's kept-revisions (H-A-285): the catalog revision each kept mover's
// fingerprint is from (H-A-283, H-A-284 (2)).
func (s *MoverStore) UpdateKept(key string, p seat.Pipeline, revision uint64, kept map[string]int64) (uint64, error) {
return s.update(key, p, revision, kept)
}
// KeptRevisions are pipeline key's kept-revisions, the catalog revision each mover a same-source re-pin kept in it is from
// (H-A-283, H-A-284 (2), H-A-285); none for a pipeline that kept none.
func (s *MoverStore) KeptRevisions(key string) (map[string]int64, error) {
defer s.lock(key)()
raw, _, err := s.Raw.Get(key)
if err != nil {
return nil, err
}
if raw == nil {
return nil, fmt.Errorf("%w: %s", ErrNoPipeline, key)
}
return keptOf(raw)
}
// keptRevisionsField is the pipeline record's property kept-revisions, beside its movers (H-A-285): written and read by the
// runner alone, never part of the seat's Go form of a pipeline, which drops it on every decode.
const keptRevisionsField = "kept-revisions"
// keptOf is the kept-revisions a record carries; none when it carries none.
func keptOf(raw []byte) (map[string]int64, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return nil, fmt.Errorf("the record is no object: %w", err)
}
f, ok := fields[keptRevisionsField]
if !ok {
return nil, nil
}
var kept map[string]int64
if err := json.Unmarshal(f, &kept); err != nil {
return nil, fmt.Errorf("the record's %s is no map of module to revision: %w", keptRevisionsField, err)
}
return kept, nil
}
// withKept is record with kept as its kept-revisions.
func withKept(record []byte, kept map[string]int64) ([]byte, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(record, &fields); err != nil {
return nil, err
}
raw, err := json.Marshal(kept)
if err != nil {
return nil, err
}
fields[keptRevisionsField] = raw
return json.Marshal(fields)
}
func (s *MoverStore) update(key string, p seat.Pipeline, revision uint64, kept map[string]int64) (uint64, error) {
defer s.lock(key)()
record, want, err := split(key, p)
if err != nil {
@@ -303,6 +362,20 @@ func (s *MoverStore) Update(key string, p seat.Pipeline, revision uint64) (uint6
if cur != revision {
return 0, ErrChanged
}
// The record's kept-revisions ride on through every write, kept laid over them (H-A-285).
merged, err := keptOf(raw)
if err != nil {
return 0, fmt.Errorf("the pipeline %s: %w", key, err)
}
if len(kept) > 0 && merged == nil {
merged = map[string]int64{}
}
maps.Copy(merged, kept)
if len(merged) > 0 {
if record, err = withKept(record, merged); err != nil {
return 0, err
}
}
vals, revs, err := s.moverKeys(key)
if err != nil {
return 0, err
@@ -61,7 +61,18 @@ var moduleSteps = []string{seat.StepResultStepValidate, seat.StepResultStepCompo
// placement makes the same id at every start and the bus dedupes on it, and each mover's step its own id (design 47 H
// §746); started is the pipeline's start time, or a rerun's own (design 47 G A-297).
func AskID(started time.Time, pipeline, module, artifact, step string) string {
sum := sha256.Sum256([]byte(pipeline + "\n" + module + "\n" + artifact + "\n" + step))
return keptAskID(started, pipeline, module, artifact, step, 0)
}
// keptAskID is AskID of a step of a mover a same-source re-pin kept at catalog revision kept (H-A-286): the kept revision
// is mixed into the id's sum, so no id equals one the pipeline asked for the mover's old fingerprint, and every placing,
// pin-held ask, composition, rerun and restart of the kept mover finds the same ids again; kept 0 is AskID's own form.
func keptAskID(started time.Time, pipeline, module, artifact, step string, kept int64) string {
in := pipeline + "\n" + module + "\n" + artifact + "\n" + step
if kept > 0 {
in += "\nkept " + strconv.FormatInt(kept, 10)
}
sum := sha256.Sum256([]byte(in))
return strconv.FormatInt(started.UnixMilli(), 36) + hex.EncodeToString(sum[:3])
}
@@ -80,14 +91,16 @@ func startOf(p seat.Pipeline, now time.Time) time.Time {
// could not run naming the missing seat; the module's steps wait. It returns the steps and the asks to put on the seats'
// work queues, one per running step.
func Place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[string]catalog.CatalogEntry, now time.Time, asker string) ([]seat.StepResult, []Asked) {
return place(p, m, reg, entries, now, asker, true)
return place(p, m, reg, entries, now, asker, true, 0)
}
// place is Place; share false places each build step its own ask, as a step asked before the test runs is (§730, A-468k).
// With share, the build steps of one candidate, level and share digest are asked once (§724): the first artifact's step, in
// reference order, asks the run and names it in Shared; every other step of the key reads it, keeping no ask of its own and
// naming the run's ask and the step that asked it (A-469m, A-469q).
func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[string]catalog.CatalogEntry, now time.Time, asker string, share bool) ([]seat.StepResult, []Asked) {
// kept is the catalog revision a same-source re-pin kept m at, 0 for a mover not kept: its every ask id mixes it in
// (H-A-286).
func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[string]catalog.CatalogEntry, now time.Time, asker string, share bool, kept int64) ([]seat.StepResult, []Asked) {
entry := entries[m.Module]
candidate, named := CandidateOf(p)
share = share && named
@@ -99,16 +112,9 @@ func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[
// The build step is every mover's first, required whatever the policy names, asked of the seat mesh-builds for the
// mover's own entry only (A-99, A-97): each mover is built by its own step (§746).
build := seat.StepResult{Step: seat.StepResultStepBuild, Module: m.Module, Status: seat.StepResultStatusRequired, Fingerprint: m.Fingerprint,
Seat: builds.Seat, Outcome: seat.StepResultOutcomeRunning, Started: &now, Ask: AskID(startOf(p, now), p.Pipeline, m.Module, "", seat.StepResultStepBuild)}
Seat: builds.Seat, Outcome: seat.StepResultOutcomeRunning, Started: &now, Ask: keptAskID(startOf(p, now), p.Pipeline, m.Module, "", seat.StepResultStepBuild, kept)}
steps = append(steps, build)
// A run on request's build step ask names no source (the seat mesh-builds' contract; H-A-162): its candidate's
// fingerprints are never in the builder's picture of the branch, so it waits for no revision there.
source := p.Source
if p.Run != "" {
source = 0
}
asks = append(asks, Asked{Seat: builds.Seat, ID: build.Ask, Commit: m.Commit, Ask: builds.StepRun{Pipeline: p.Pipeline, Step: builds.StepRunStepBuild, Branch: p.Branch,
Revision: p.Revision, Source: source, Entries: []builds.BuildEntry{{Module: m.Module, Commit: m.Commit, Fingerprint: m.Fingerprint}}}})
asks = append(asks, buildAsk(p, m, build.Ask))
for _, a := range arts {
name := a.Ref
if _, after, ok := strings.Cut(a.Ref, ":"); ok {
@@ -151,7 +157,7 @@ func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[
continue
}
s.Seat, s.Outcome, s.Started = bs, seat.StepResultOutcomeRunning, &now
s.Ask = AskID(startOf(p, now), p.Pipeline, m.Module, a.Ref, step)
s.Ask = keptAskID(startOf(p, now), p.Pipeline, m.Module, a.Ref, step, kept)
body := stepAsk(p, m, entry, a, language, step, s.Ask, asker, now)
ask := Asked{Seat: bs, ID: s.Ask, Ask: body, Toolchain: a.Language, Commit: m.Commit}
if share && BoundSeat(bs) {
@@ -196,7 +202,7 @@ func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[
// validate is the assigner's, an accepted kind step of the seat mesh-wants, run once per take-in revision: every
// mover's validate step reads the one ask, its id naming no mover (§745, H-A-51).
s.Seat, s.Outcome, s.Started, s.Fingerprint = wants.Seat, seat.StepResultOutcomeRunning, &now, m.Fingerprint
s.Ask = ValidateAskID(p, now)
s.Ask = keptAskID(startOf(p, now), p.Pipeline, "", "", seat.StepResultStepValidate, kept)
asks = append(asks, wantsAsk(p, m, s.Ask, wants.StepRunStepValidate))
validating = true
case seat.StepResultStepComposition:
@@ -204,7 +210,7 @@ func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[
// runner's seat as its asker (H-A-88). It is asked once the mover's validate passed, at once when it has none
// (§745).
s.Seat, s.Fingerprint = wants.Seat, m.Fingerprint
s.Ask = AskID(startOf(p, now), p.Pipeline, m.Module, "", step)
s.Ask = keptAskID(startOf(p, now), p.Pipeline, m.Module, "", step, kept)
if validating {
s.Ask, s.Why = "", "asked once validate passed"
break
@@ -309,12 +315,13 @@ func (s *Starter) AskPinHeld(store Updater, key, module string, pins map[string]
return 0, nil
}
var entry catalog.CatalogEntry
unread := ""
switch half, err := readRevision(revision, p.Branch, p.Revision); {
at, kept, whose, unread := moverRevision(store, p, module)
switch half, err := readRevision(revision, p.Branch, at); {
case unread != "":
case err != nil:
unread = fmt.Sprintf("revision %d of %s, the pipeline's, cannot be read for its entry: %v", p.Revision, p.Branch, err)
unread = fmt.Sprintf("revision %d of %s, %s, cannot be read for its entry: %v", at, p.Branch, whose, err)
case half.Entries[module].Fp != m.Fingerprint:
unread = fmt.Sprintf("revision %d of %s holds %s at %q, not the mover's %s", p.Revision, p.Branch, module, half.Entries[module].Fp, m.Fingerprint)
unread = fmt.Sprintf("revision %d of %s holds %s at %q, not the mover's %s", at, p.Branch, module, half.Entries[module].Fp, m.Fingerprint)
default:
entry = half.Entries[module]
}
@@ -332,7 +339,7 @@ func (s *Starter) AskPinHeld(store Updater, key, module string, pins map[string]
if unread != "" || j < 0 || err != nil {
why := unread
if why == "" {
why = fmt.Sprintf("revision %d of %s names no artifact %s of %s to ask it of", p.Revision, p.Branch, st.Artifact, module)
why = fmt.Sprintf("revision %d of %s names no artifact %s of %s to ask it of", at, p.Branch, st.Artifact, module)
}
st.Outcome, st.Ended = seat.StepResultOutcomeCouldNotRun, &now
st.Why = "its build passed with its pins, but " + why + " (H-A-207, H-A-205)"
@@ -342,7 +349,7 @@ func (s *Starter) AskPinHeld(store Updater, key, module string, pins map[string]
}
a := entry.Artifacts[j]
st.Outcome, st.Why, st.Started = seat.StepResultOutcomeRunning, "", &now
st.Ask = AskID(startOf(p, now), p.Pipeline, module, a.Ref, st.Step)
st.Ask = keptAskID(startOf(p, now), p.Pipeline, module, a.Ref, st.Step, kept)
ask := Asked{Seat: st.Seat, ID: st.Ask, Ask: stepAsk(p, m, entry, a, language, st.Step, st.Ask, s.Name, now), Toolchain: a.Language, Commit: m.Commit}
if a.Package != nil {
ask.Package = a.Package.Name
@@ -363,6 +370,57 @@ func (s *Starter) AskPinHeld(store Updater, key, module string, pins map[string]
}
}
// keptRevisions is the state pipelines' kept-revisions (MoverStore, H-A-285): the catalog revision each mover a same-source
// re-pin kept in a pipeline is from (H-A-283, H-A-284 (2)).
type keptRevisions interface {
KeptRevisions(key string) (map[string]int64, error)
UpdateKept(key string, p seat.Pipeline, revision uint64, kept map[string]int64) (uint64, error)
}
// moverRevision is the catalog revision mover module's entry of p is read at (H-A-284 (2)): the one its pipeline kept it at
// when a same-source re-pin kept it, else the pipeline's; whose says which, and unread why it is not read when its record
// cannot be.
// It answers that revision, the kept revision (0 for a mover not kept), whose it is, and why it is not read.
func moverRevision(store Updater, p seat.Pipeline, module string) (int64, int64, string, string) {
kept, err := keptOfStore(store, p.Pipeline)
if err != nil {
return p.Revision, 0, "the pipeline's", fmt.Sprintf("the revision %s was kept at in %s cannot be read: %v", module, p.Pipeline, err)
}
if r := kept[module]; r > 0 {
return r, r, "the one the pipeline kept " + module + " at (H-A-284)", ""
}
return p.Revision, 0, "the pipeline's", ""
}
// keptOfStore is pipeline key's kept-revisions in store (H-A-285); none for a store that keeps none.
func keptOfStore(store any, key string) (map[string]int64, error) {
ks, ok := store.(keptRevisions)
if !ok || key == "" {
return nil, nil
}
kept, err := ks.KeptRevisions(key)
if errors.Is(err, ErrNoPipeline) {
return nil, nil // a pipeline not written yet keeps none
}
return kept, err
}
// keptEntries are the entries of branch's catalog revision rev, the one a same-source re-pin kept a mover at, read by its
// number (H-A-286): a kept mover is placed and pin-checked from them, never from the branch's held view; false while they
// cannot be read, and the mover waits.
func (s *Starter) keptEntries(branch string, rev int64) (map[string]catalog.CatalogEntry, bool) {
if s.Revision == nil {
s.logf("the runner reads no catalog revision, so a mover kept at revision %d of %s waits", rev, branch)
return nil, false
}
half, err := s.Revision(branch, rev)
if err != nil || half.Revision != rev {
s.logf("revision %d of %s, which a mover is kept at, is not read (%v): the mover waits for it (H-A-286)", rev, branch, err)
return nil, false
}
return half.Entries, true
}
func stepAsk(p seat.Pipeline, m seat.Mover, e catalog.CatalogEntry, a catalog.Artifact, language, step, id, asker string, now time.Time) nodebuildgo.StepAsk {
// A step is for its module alone (principle 10, A-447): the fetch always names the module's directory, and the
// artifact's named paths in it when the entry has them.
@@ -387,17 +445,23 @@ func stepAsk(p seat.Pipeline, m seat.Mover, e catalog.CatalogEntry, a catalog.Ar
Module: m.Module, Pipeline: p.Pipeline, Revision: p.Revision, Source: p.Source, Step: step}}
}
// askAll puts each ask on its seat's work queue; the first refusal is named, the rest are still put. An ask waits for its
// holder with no bound of the runner's own (§535, §647 as A-466y amends them).
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool) error {
// askAll puts each ask on its seat's work queue; the first refusal is named, the rest are still put. arm, when not nil, arms
// each ask's bound before it is put (H-A-235, H-A-238 (5)); an ask whose bound is not armed is not put.
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool, arm func(Asked) error) error {
var first error
for _, a := range asks {
accept, ok := acceptStep[a.Seat]
if to, named := acceptRecovery[a.Seat]; named && recovery != nil && recovery(a.Commit) {
accept = to
}
err := fmt.Errorf("the seat %s is no build seat the runner asks", a.Seat)
if ok {
var err error
switch {
case !ok:
err = fmt.Errorf("the seat %s is no build seat the runner asks", a.Seat)
case arm != nil:
err = arm(a)
}
if err == nil {
err = asker.AskStep(accept, a.ID, a.Ask)
}
if err != nil && first == nil {
@@ -434,7 +498,8 @@ func wantsAsk(p seat.Pipeline, m seat.Mover, id, step string) Asked {
// releaseCompositions asks the composition of each mover whose validate ended passed, and ends `not asked` the composition
// of each whose validate did not pass, so a manifest the validator refuses is never composed (design 47 H §745). It answers
// the asks to put and the indexes of the steps it ended.
func releaseCompositions(p *seat.Pipeline, now time.Time) ([]Asked, []int) {
// kept is p's kept-revisions: a kept mover's composition id mixes its kept revision in (H-A-286).
func releaseCompositions(p *seat.Pipeline, now time.Time, kept map[string]int64) ([]Asked, []int) {
var asks []Asked
var ended []int
for _, m := range p.Movers {
@@ -460,7 +525,7 @@ func releaseCompositions(p *seat.Pipeline, now time.Time) ([]Asked, []int) {
ended = append(ended, c)
continue
}
s.Ask = AskID(startOf(*p, now), p.Pipeline, m.Module, "", seat.StepResultStepComposition)
s.Ask = keptAskID(startOf(*p, now), p.Pipeline, m.Module, "", seat.StepResultStepComposition, kept[m.Module])
s.Outcome, s.Started, s.Why = seat.StepResultOutcomeRunning, &now, ""
asks = append(asks, wantsAsk(*p, m, s.Ask, wants.StepRunStepComposition))
}
@@ -176,6 +176,52 @@ func TestThePipelinesWaitTheirTurn(t *testing.T) {
}
}
// H-A-244 (1), H-A-223: a mover a newer revision moves leaves an older waiting pipeline also when the newer one is placed at
// once, not only when it waits: the waiting pipeline keeps the movers still its own, and with none left ends superseded,
// never placed.
func TestANewerPlacedPipelineSupersedesAnOlderWaitingOne(t *testing.T) {
s, mem, q, said, _, _, view, _ := queueFixture()
take := takeIn(t, s, view)
take(4, "app")
take(5, "lib", "tool")
if p, _, _ := mem.Get("main.5"); p.Outcome != seat.PipelineOutcomeWaiting {
t.Fatalf("main.5 = %s, want waiting", p.Outcome)
}
s.Queue.AtOnce = 3
take(6, "lib")
if p, _, _ := mem.Get("main.6"); p.Outcome != seat.PipelineOutcomeRunning {
t.Fatalf("main.6 = %s, want placed at once", p.Outcome)
}
p5, _, _ := mem.Get("main.5")
if p5.Outcome != seat.PipelineOutcomeWaiting || len(p5.Movers) != 1 || p5.Movers[0].Module != "tool" || !slices.Equal(p5.Superseded, []string{"lib"}) {
t.Fatalf("main.5 = %s moving %+v, superseded %v; want waiting with tool alone, lib superseded", p5.Outcome, p5.Movers, p5.Superseded)
}
// At its turn it is placed with tool alone.
p4, rev, _ := mem.Get("main.4")
p4.Outcome = seat.PipelineOutcomePassed
mem.Update("main.4", p4, rev)
if _, err := s.Turn("main.4", view); err != nil {
t.Fatal(err)
}
if p, _, _ := mem.Get("main.5"); p.Outcome != seat.PipelineOutcomeRunning || len(stepsOf(p, "lib")) != 0 || len(stepsOf(p, "tool")) == 0 {
t.Fatalf("main.5 at its turn = %s with steps %+v, want running with tool's steps alone", p.Outcome, p.Steps)
}
// A waiting pipeline whose one mover a newer placed pipeline moves ends superseded, never placed.
s, mem, q, said, _, _, view, _ = queueFixture()
take = takeIn(t, s, view)
take(4, "app")
take(5, "tool")
s.Queue.AtOnce = 3
take(7, "tool")
if p, _, _ := mem.Get("main.5"); p.Outcome != seat.PipelineOutcomeSuperseded || len(p.Steps) != 0 || p.Ended == nil {
t.Errorf("main.5 = %s with %d steps, want superseded and never placed", p.Outcome, len(p.Steps))
}
if _, ok := q["waiting.00000000000000000001"]; ok || !slices.Contains(*said, "pipeline-ended.main.5") {
t.Errorf("the queue %v, said %v; want main.5's waiting key gone and its pipeline-ended said", q, *said)
}
}
// R13/AC1 (design 47 H §748, ADR 0682 §7): a newer waiting pipeline of the same branch takes the movers it also moves out of
// an older waiting one, which ends superseded, saying wait-ended and pipeline-ended, once no mover is left; a running
// pipeline moving the same module runs to its end.
@@ -0,0 +1,111 @@
package pipeline
import (
"context"
"fmt"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// RegistrationMissingKind is the runner's finding of a mover that waits on its module's registration (design 47 H H-A-262,
// G A-327): scope module, id the module, raised while a mover of a running pipeline has no registration to place its steps
// from, cleared when the registration arrives or no running pipeline still waits on it (H-A-265 (4)).
const RegistrationMissingKind = "registration-missing"
// registrationMissingID is the finding's id: the module, escaped as a key's name is.
func registrationMissingID(module string) string { return keys.EscapeName(module) }
// registrationMissing raises registration-missing of module, waited on by the pipeline key (H-A-262).
func (s *Starter) registrationMissing(module, key string) {
if s.Findings == nil {
return
}
id := registrationMissingID(module)
err := s.Findings.Write(context.Background(), condseat.Finding{Scope: mcc.ScopeModule, ID: id, Kind: RegistrationMissingKind,
Severity: condseat.FindingSeverityWarning,
Summary: fmt.Sprintf("%s has no registration: the pipeline %s waits on it for its steps", module, key),
Headline: "A pipeline waits on a module's registration",
Explanation: fmt.Sprintf("The pipeline %s moves %s, which has no registration, so none of its steps is placed (A-327). The indexer registers a module the take-in first lists (H-A-262); this is cleared when the registration of %s arrives.",
key, module, module),
Needs: "the registration of " + module,
Source: "pipeline-runner." + RegistrationMissingKind, Observer: s.Observer, At: s.now(),
Subject: condseat.Subject{Scope: mcc.ScopeModule, ID: id}})
if err != nil {
s.logf("the finding %s of %s is not written: %v", RegistrationMissingKind, module, err)
}
}
// registrationArrived clears registration-missing of module, saying why, when the findings hold it (H-A-262, H-A-265 (4)).
func (s *Starter) registrationArrived(module, why string) {
if s.Findings == nil {
return
}
id := registrationMissingID(module)
for _, f := range s.Findings.Held() {
if f.Scope != mcc.ScopeModule || f.ID != id || f.Kind != RegistrationMissingKind {
continue
}
if err := s.Findings.Clear(context.Background(), mcc.ScopeModule, id, RegistrationMissingKind, why); err != nil {
s.logf("the finding %s of %s is not cleared: %v", RegistrationMissingKind, module, err)
}
return
}
}
// JudgeRegistrationMissing clears registration-missing of every module the registrations held now name, and of every module
// no running pipeline still has an unplaced mover of (H-A-265 (4)): a pipeline that ended, failed (H-A-163) or cancelled,
// with the mover still waiting leaves no finding behind. It is judged at a pipeline's end and in Waiting.
func (s *Starter) JudgeRegistrationMissing(store Updater) {
if s.Findings == nil {
return
}
var modules []string
for _, f := range s.Findings.Held() {
if f.Scope != mcc.ScopeModule || f.Kind != RegistrationMissingKind {
continue
}
module, err := keys.UnescapeName(f.ID)
if err != nil {
module = f.ID
}
if s.Registration != nil {
if _, ok := s.Registration(module); ok {
s.registrationArrived(module, "the registration of "+module+" arrived")
continue
}
}
modules = append(modules, module)
}
if len(modules) == 0 || store == nil {
return
}
pkeys, err := allKeys(store)
if err != nil {
s.logf("the finding %s is not judged: the pipelines are not listed: %v", RegistrationMissingKind, err)
return
}
waiting := map[string]bool{}
for _, key := range pkeys {
p, _, err := store.Get(key)
if err != nil {
s.logf("the finding %s is not judged: the pipeline %s is not read: %v", RegistrationMissingKind, key, err)
return
}
if p.Outcome != seat.PipelineOutcomeRunning {
continue
}
for _, m := range modules {
if Moves(p, m) && !placed(p, m) {
waiting[m] = true
}
}
}
for _, m := range modules {
if !waiting[m] {
s.registrationArrived(m, "no running pipeline still waits on the registration of "+m)
}
}
}
@@ -0,0 +1,213 @@
package pipeline
import (
"errors"
"fmt"
"slices"
"strings"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
)
// keepSameSource keeps, in the open pipeline of the same source revision that holds it, each mover of the revision half a
// re-pin made (design 47 H H-A-283): a revision of the same source moves no mover out of a running or waiting pipeline (as
// H-A-223 and H-A-244 (1) move one for a revision of a new source); the pipeline keeps the mover and takes its new
// fingerprint, and the revision's own pipeline starts without it. It answers the movers no open pipeline of the source
// holds, which the revision's pipeline carries.
func (s *Starter) keepSameSource(branch string, half Half, movers []seat.Mover) []seat.Mover {
if s.Store == nil || len(movers) == 0 {
return movers
}
pkeys, err := branchKeys(s.Store, branch)
if err != nil {
s.logf("the pipelines of %s are not listed, so no open pipeline of source %d keeps the movers of revision %d: %v", branch, half.Source, half.Revision, err)
return movers
}
left := movers
for _, key := range pkeys {
if len(left) == 0 {
break
}
left = s.keepIn(key, half, left)
}
return left
}
// holdsOpen says whether the pipeline p, not ended and of no run on request, is of revision half's source, older than it, and
// still holds mover module as its own.
func holdsOpen(p seat.Pipeline, half Half, module string) bool {
return (p.Outcome == seat.PipelineOutcomeRunning || p.Outcome == seat.PipelineOutcomeWaiting) && p.Run == "" &&
p.Source == half.Source && p.Revision < half.Revision && Moves(p, module) && !slices.Contains(p.Superseded, module)
}
// keepIn keeps in pipeline key each of movers it holds (H-A-283), answering the movers it does not hold. A kept mover whose
// fingerprint the re-pin moved takes its new fingerprint and commit, and the pipeline keeps the revision it is from in its
// kept-revisions (H-A-285); when its steps were placed, every one of them, open or ended, ends superseded and its steps are
// placed again from the revision's entry, the build first (H-A-284 (1)), each under an ask id its kept revision is mixed
// into (H-A-286), so no ask of the old fingerprint is kept for the new. One not placed yet is placed when it is due, from
// the entries of the revision it is kept at.
func (s *Starter) keepIn(key string, half Half, movers []seat.Mover) []seat.Mover {
for {
p, rev, err := s.Store.Get(key)
if err != nil {
return movers
}
var kept, left []seat.Mover
for _, m := range movers {
if holdsOpen(p, half, m.Module) {
kept = append(kept, m)
} else {
left = append(left, m)
}
}
if len(kept) == 0 {
return movers
}
now := s.now()
why := fmt.Sprintf("superseded: the re-pin revision %d of the pipeline's own source %d moved its mover's fingerprint; its steps are placed again (H-A-283, H-A-284)", half.Revision, half.Source)
revisions := map[string]int64{}
olds := slices.Clone(p.Movers) // the superseded steps are said at the fingerprint they ran for
var moved, unregistered []string
var ended []seat.StepResult
var leave, drop []seat.StepResult
var asks []Asked
var notAsked []int
for _, m := range kept {
i := slices.IndexFunc(p.Movers, func(o seat.Mover) bool { return o.Module == m.Module })
old := p.Movers[i]
if old.Fingerprint == m.Fingerprint && old.Commit == m.Commit {
continue
}
p.Movers[i].Fingerprint, p.Movers[i].Commit = m.Fingerprint, m.Commit
revisions[m.Module] = half.Revision
moved = append(moved, fmt.Sprintf("%s %s to %s", m.Module, old.Fingerprint, m.Fingerprint))
if p.Outcome != seat.PipelineOutcomeRunning || !placed(p, m.Module) {
continue
}
rest := make([]seat.StepResult, 0, len(p.Steps))
for _, st := range p.Steps {
if st.Module != m.Module {
rest = append(rest, st)
continue
}
live := open(st) || st.Outcome == seat.StepResultOutcomeWaiting
l, d := supersedeStep(&p, &st, why, now)
if live {
leave, drop = append(leave, l...), append(drop, d...)
}
ended = append(ended, st)
}
p.Steps = rest
a, n, unreg := s.placeAgain(&p, p.Movers[i], half, now)
asks, notAsked = append(asks, a...), append(notAsked, n...)
if unreg {
unregistered = append(unregistered, m.Module)
}
}
if len(moved) == 0 {
s.logf("%s keeps %d movers of revision %d, of its own source %d, at the fingerprints it holds (H-A-283)", key, len(kept), half.Revision, half.Source)
return left
}
var holderEnded []int
if len(asks) > 0 {
asks, holderEnded = s.holderless(&p, asks, now)
}
ks, keeps := s.Store.(keptRevisions)
if keeps {
_, err = ks.UpdateKept(key, p, rev, revisions)
} else {
_, err = s.Store.Update(key, p, rev)
}
if errors.Is(err, ErrChanged) {
continue
} else if err != nil {
s.logf("%s is not written, so the movers revision %d re-pinned stay its own at their old fingerprints: %v", key, half.Revision, err)
return left
}
if !keeps {
s.logf("%s keeps no kept-revisions, so the pin check of its re-pinned movers reads the pipeline's revision (H-A-285)", key)
}
s.logf("%s keeps the movers revision %d of its own source %d re-pinned, %s: %d steps ended superseded, %d asked again (H-A-283, H-A-284)",
key, half.Revision, half.Source, strings.Join(moved, ", "), len(ended), len(asks))
for _, st := range ended {
if s.Say == nil {
break
}
was := p
was.Movers = olds
if err := s.Say(stepEvent(seat.EventStepEnded, was, st, now)); err != nil {
s.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, st.Module, st.Step, err)
}
}
if s.Left != nil {
s.Left(key, leave, drop, why)
}
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
s.sayNotAsked(p, notAsked)
if len(holderEnded) > 0 {
if err := s.endedAtPlacing(p, holderEnded); err != nil {
s.logf("%v", err)
}
return left
}
if s.Steps != nil && len(asks) > 0 {
if err := s.askSteps(key, asks); err != nil {
s.logf("%s: the steps placed again are not all put, they are asked at the next start: %v", key, err)
}
}
return left
}
}
// placeAgain places the steps of kept mover m of p from the re-pin revision half's entry (H-A-284 (1)), as a mover is placed
// (placeMovers), under ask ids its kept revision is mixed into (H-A-286): it answers the asks to put, the steps placed not asked because a
// dependency did not build, and whether m has no registration. A mover not due yet (it waits on a publish) is left
// unplaced, placed when the publish is heard.
func (s *Starter) placeAgain(p *seat.Pipeline, m seat.Mover, half Half, now time.Time) ([]Asked, []int, bool) {
if s.Registration == nil {
return nil, nil, false
}
r, ok := s.Registration(m.Module)
if !ok {
return nil, nil, true
}
due, waiting, failed := ready(*p, m.Module, half.Entries)
if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
return nil, nil, false
}
steps, asks := place(*p, m, r, half.Entries, now, s.Name, true, half.Revision)
supersededWhy(*p, m.Module, half.Entries, steps)
var notAsked []int
if failed != "" {
why := "its dependency " + failed + " did not build"
for k := range steps {
st := &steps[k]
if st.Outcome == seat.StepResultOutcomeBypassed {
continue
}
st.Outcome, st.Why, st.Ask, st.Shared, st.Ended = seat.StepResultOutcomeNotAsked, why, "", nil, &now
p.Unlogged = append(p.Unlogged, endEntry(p.Pipeline, *st, now))
notAsked = append(notAsked, len(p.Steps)+k)
}
asks = nil
}
p.Steps = append(p.Steps, steps...)
return asks, notAsked, false
}
// buildAsk is mover m's build step ask under id, of the seat mesh-builds for the mover's own entry only (A-99, A-97): a run on
// request's names no source (H-A-162).
func buildAsk(p seat.Pipeline, m seat.Mover, id string) Asked {
source := p.Source
if p.Run != "" {
source = 0
}
return Asked{Seat: builds.Seat, ID: id, Commit: m.Commit, Ask: builds.StepRun{Pipeline: p.Pipeline, Step: builds.StepRunStepBuild, Branch: p.Branch,
Revision: p.Revision, Source: source, Entries: []builds.BuildEntry{{Module: m.Module, Commit: m.Commit, Fingerprint: m.Fingerprint}}}}
}
@@ -0,0 +1,191 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// ticking is a clock that moves a second at every read, so a start and a later re-pin never share a millisecond.
func ticking() func() time.Time {
at := time.Date(2026, 11, 21, 17, 0, 0, 0, time.UTC)
return func() time.Time { at = at.Add(time.Second); return at }
}
// repinFixture is a pipeline main.10 of source 41 moving app at src2:aa, started, with the re-pin revision 11 of the same
// source moving app to src2:ab (an entry naming a package with no version yet), and a revision 12 of a new source 42.
func repinFixture() (catalog.Registration, map[string]catalog.CatalogEntry, map[string]catalog.CatalogEntry, func(string, int64) (Half, error)) {
reg, entries := placeFixture()
repinned := map[string]catalog.CatalogEntry{}
for k, e := range entries {
repinned[k] = e
}
app := entries["app"]
app.Fp = "src2:aa"
entries["app"] = app
re := app
re.Fp, re.Packages = "src2:ab", map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.26", "git.novox.be/novox/x/go": ""}
repinned["app"] = re
read := func(branch string, revision int64) (Half, error) {
switch revision {
case 10:
return Half{Revision: 10, Source: 41, Entries: entries}, nil
case 11:
return Half{Revision: 11, Source: 41, Entries: repinned, Call: "release of mesh-sdk 0.1.26",
Movers: []catalog.TakeInEntry{{Module: "app", Commit: "c1", Fingerprint: "src2:ab", Repository: "o/app"}}}, nil
case 12:
return Half{Revision: 12, Source: 42, Entries: repinned, Call: "take-in of 42",
Movers: []catalog.TakeInEntry{{Module: "app", Commit: "c2", Fingerprint: "src2:ac", Repository: "o/app"}}}, nil
}
return Half{}, fmt.Errorf("no revision %d", revision)
}
return reg, entries, repinned, read
}
func revisionEvent(n int64) catalog.CatalogRevisionEvent {
return catalog.CatalogRevisionEvent{Kind: "trunk", Branch: keys.EscapeName("main"), Revision: n, CallKind: "release"}
}
// H-A-283, H-A-284, H-A-285, H-A-286: a re-pin revision of the same source moves no mover out of the running pipeline of that source:
// the pipeline keeps it at its new fingerprint, every step of it ends superseded (said at the old fingerprint, its asks
// dropped) and is placed again from the re-pin revision's entry under new asks, the build first, still naming the
// pipeline's revision; the revision the mover is kept at rides in the record's kept-revisions, where the pin check reads
// its entry; the re-pin starts no pipeline. A revision of a new source still starts its own and moves the mover out.
func TestASameSourceRepinKeepsTheMoverInItsPipeline(t *testing.T) {
reg, entries, _, read := repinFixture()
store := &MoverStore{Raw: newMemRaw()}
asks := &stepAsks{}
var said []seat.StepEvent
var left [][]seat.StepResult
s := &Starter{Pipelines: store, Store: store, Steps: asks, Name: "the-runner", Now: ticking(), Revision: read,
Registration: func(m string) (catalog.Registration, bool) { return reg, m == "app" },
Say: func(_ string, body any) error {
if ev, ok := body.(seat.StepEvent); ok && ev.Result.Outcome == seat.StepResultOutcomeSuperseded {
said = append(said, ev)
}
return nil
},
Left: func(_ string, leave, drop []seat.StepResult, _ string) { left = append(left, append(leave, drop...)) }}
h := &Hearer{Store: store, Say: func(string, any) error { return nil }}
s.Supersede = func(newer seat.Pipeline) { h.SupersedeRunning(newer) }
key := Key("main", 10)
if ok, err := s.Start(Begin{Branch: "main", Source: 41, Revision: 10, Entries: entries,
Movers: []seat.Mover{{Module: "app", Commit: "c1", Fingerprint: "src2:aa", Repository: "o/app"}}}); !ok || err != nil {
t.Fatalf("start = %v, %v", ok, err)
}
before, _, _ := store.Get(key)
oldAsks := map[string]bool{}
for _, st := range before.Steps {
if st.Ask != "" {
oldAsks[st.Ask] = true
}
}
asks.at, asks.ids, asks.asks = nil, nil, nil
ev := revisionEvent
if n, err := s.Revised(ev(11), read); n != 0 || err != nil {
t.Fatalf("the same-source re-pin started %d pipelines (%v), want none", n, err)
}
if _, _, err := store.Get(Key("main", 11)); err == nil {
t.Fatal("the same-source re-pin wrote a pipeline of its own")
}
p, _, err := store.Get(key)
if err != nil {
t.Fatal(err)
}
if m, _ := MoverOf(p, "app"); m.Fingerprint != "src2:ab" || slices.Contains(p.Superseded, "app") || p.Outcome != seat.PipelineOutcomeRunning {
t.Fatalf("the kept mover = %+v, superseded %v, outcome %s: want it kept at src2:ab, the pipeline running", m, p.Superseded, p.Outcome)
}
if len(p.Steps) != len(before.Steps) || p.Steps[0].Step != seat.StepResultStepBuild || p.Steps[0].Outcome != seat.StepResultOutcomeRunning ||
p.Steps[0].Fingerprint != "src2:ab" || oldAsks[p.Steps[0].Ask] {
t.Fatalf("the steps placed again = %+v, want the build first, running at src2:ab under a new ask", p.Steps)
}
for _, st := range p.Steps {
if st.Outcome == seat.StepResultOutcomeSuperseded || (st.Ask != "" && oldAsks[st.Ask]) {
t.Errorf("the step %s %s stands for the old fingerprint: %+v", st.Artifact, st.Step, st)
}
}
// Every old step ended superseded, said at the fingerprint it ran for; the ones open on a build node are dropped.
if len(said) != len(before.Steps) || said[0].Fingerprint != "src2:aa" || len(left) != 1 || len(left[0]) == 0 {
t.Errorf("said %d superseded steps (of %d), the first at %+v; left %v", len(said), len(before.Steps), said, left)
}
b, ok := asks.asks[0].(builds.StepRun)
if !ok || asks.ids[0] != p.Steps[0].Ask || b.Revision != 10 || b.Source != 41 || len(b.Entries) != 1 || b.Entries[0].Fingerprint != "src2:ab" {
t.Fatalf("the build asked again = %+v (%s), want the new fingerprint at the pipeline's revision", asks.asks[0], asks.ids[0])
}
if kept, err := store.KeptRevisions(key); err != nil || kept["app"] != 11 {
t.Errorf("kept-revisions = %v (%v), want app at 11", kept, err)
}
// The pin check reads the kept mover's entry at the revision it was kept at, never the pipeline's.
if n, err := s.AskPinHeld(store, key, "app", map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.26", "git.novox.be/novox/x/go": "v0.1.1"}, read); n == 0 || err != nil {
after, _, _ := store.Get(key)
t.Fatalf("the held steps asked %d (%v): %+v", n, err, after.Steps)
}
last := asks.asks[len(asks.asks)-1].(nodebuildgo.StepAsk)
if last.Step.Fingerprint != "src2:ab" || last.Step.Fetch.Packages["git.novox.be/novox/x/go"] != "v0.1.1" {
t.Errorf("a held step asked = %+v", last.Step)
}
// A revision of a new source starts its own pipeline and moves the mover out (H-A-223).
if n, err := s.Revised(ev(12), read); n != 1 || err != nil {
t.Fatalf("a new source's revision started %d (%v)", n, err)
}
if p, _, _ := store.Get(key); !slices.Contains(p.Superseded, "app") {
t.Errorf("the new source's revision left app in %s: %v", key, p.Superseded)
}
}
// H-A-286: a kept mover not placed at its keep (its registration arrived later) is placed from the revision it is kept at
// while the branch's held view still holds the older revision, under ask ids its kept revision is mixed into; and a restart
// (Resume) finds its running steps by those ids and asks them again.
func TestAKeptMoverIsPlacedLateAndResumedAtItsKeptRevision(t *testing.T) {
reg, entries, _, read := repinFixture()
store := &MoverStore{Raw: newMemRaw()}
asks := &stepAsks{}
registered := false
s := &Starter{Pipelines: store, Store: store, Steps: asks, Name: "the-runner", Now: ticking(), Revision: read,
Say: func(string, any) error { return nil },
Registration: func(m string) (catalog.Registration, bool) { return reg, registered && m == "app" }}
key := Key("main", 10)
if ok, err := s.Start(Begin{Branch: "main", Source: 41, Revision: 10, Entries: entries,
Movers: []seat.Mover{{Module: "app", Commit: "c1", Fingerprint: "src2:aa", Repository: "o/app"}}}); !ok || err != nil {
t.Fatalf("start = %v, %v", ok, err)
}
if n, err := s.Revised(revisionEvent(11), read); n != 0 || err != nil {
t.Fatalf("the re-pin started %d (%v)", n, err)
}
// The registration arrives while the runner's view still holds revision 10.
registered = true
lagging := func(string) (Half, bool) { return Half{Revision: 10, Source: 41, Entries: entries}, true }
if n, err := s.Registered(store, reg, lagging); n != 1 || err != nil {
t.Fatalf("the late registration placed in %d pipelines (%v)", n, err)
}
p, _, err := store.Get(key)
if err != nil {
t.Fatal(err)
}
var build seat.StepResult
held := 0
for _, st := range p.Steps {
switch {
case st.Step == seat.StepResultStepBuild:
build = st
case pinHeld(st) && strings.HasSuffix(st.Why, "git.novox.be/novox/x/go"):
held++
}
}
if want := keptAskID(*p.Started, key, "app", "", seat.StepResultStepBuild, 11); build.Ask != want || build.Fingerprint != "src2:ab" || held == 0 {
t.Fatalf("the late placing: build %+v (want ask %s), %d steps held on revision 11's unpinned package: %+v", build, want, held, p.Steps)
}
// A restart asks the kept mover's running steps again under the same ids.
asks.at, asks.ids, asks.asks = nil, nil, nil
if n, err := s.Resume(store, "main", Half{Revision: 10, Source: 41, Entries: entries}); n == 0 || err != nil || !slices.Contains(asks.ids, build.Ask) {
t.Fatalf("Resume asked %d (%v), ids %v: want the kept build %s again", n, err, asks.ids, build.Ask)
}
}
@@ -165,6 +165,13 @@ type Starter struct {
// Supersede takes out of every older running pipeline of newer's branch each mover whose fingerprint newer moved
// (H-A-223; Hearer.SupersedeRunning); nil leaves running pipelines as they are.
Supersede func(newer seat.Pipeline)
// Left takes the steps a same-source re-pin superseded in a pipeline that kept their mover out of the test runs they
// read and drops their own asks (H-A-283, H-A-284 (1); Hearer.LeftInPlace); nil leaves them.
Left func(key string, leave, drop []seat.StepResult, why string)
// Revision reads a catalog revision of a branch by its number through the catalog seat's revision verb: a mover a
// same-source re-pin kept is placed from the entries of the revision it is kept at (H-A-286); nil leaves such a mover
// waiting.
Revision func(branch string, revision int64) (Half, error)
// Fills refuses a start while a store of the runner is full (§366); nil refuses none.
Fills *Fills
// Recovered says whether a commit is named a recovery (§81, §86): its steps are asked on the build seats' stream delivery.
@@ -183,9 +190,15 @@ type Starter struct {
// Queue bounds the pipelines asked at once to pipelines-at-once, the others waiting in order (design 47 H §748); nil
// places every start at once.
Queue *Queue
// Asks bounds every ask but a test run's until its holder's first word (H-A-235, H-A-238); nil bounds none.
Asks *AskClock
// RunTurn places a run on request whose turn in the queue came, from its candidate's newest revision (A-222); nil places
// it at the runner's next start.
RunTurn func(UnplacedRun)
// Findings is the runner's finding writer, through which a mover waiting on its registration raises
// registration-missing (H-A-262); Observer is the runner's own module name. nil raises none.
Findings Findings
Observer string
}
// holderless ends at once, as it is placed, a pipeline with a build step whose seat the seat mesh-module-assigner's view
@@ -252,7 +265,7 @@ func (s *Starter) askSteps(key string, asks []Asked) error {
var err error
if len(asks) > 0 {
s.carry(key, asks)
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) })
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) }, s.Asks.arm)
}
s.sayStarted(key, placed)
return err
@@ -328,6 +341,11 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
var unregistered []string
var notAsked []int
var waits []seat.Mover
kept, err := keptOfStore(s.Store, p.Pipeline)
if err != nil {
s.logf("the kept-revisions of %s are not read, so its movers are placed as movers not kept: %v", p.Pipeline, err)
}
all := entries
for _, m := range p.Movers {
if placed(*p, m.Module) {
continue
@@ -337,6 +355,13 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
unregistered = append(unregistered, m.Module)
continue
}
// A mover a same-source re-pin kept is placed from the revision it is kept at, its ids its own (H-A-286).
entries := all
if rev := kept[m.Module]; rev > 0 {
if entries, ok = s.keptEntries(p.Branch, rev); !ok {
continue
}
}
// A mover's steps wait for the publish of each mover it builds on (§746, H-A-83 (2)): placed once each is heard,
// ended not asked when one's build ended without passing.
due, waiting, failed := ready(*p, m.Module, entries)
@@ -345,7 +370,7 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
waits = append(waits, m)
continue
}
steps, a := Place(*p, m, r, entries, now, s.Name)
steps, a := place(*p, m, r, entries, now, s.Name, true, kept[m.Module])
supersededWhy(*p, m.Module, entries, steps)
for _, st := range steps {
if pinHeld(st) {
@@ -384,13 +409,13 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
}
}
}
if steps, ended := cycled(p, waits, unregistered, reg, entries, now, s.Name); ended {
if steps, ended := cycled(p, waits, unregistered, reg, all, now, s.Name); ended {
return nil, unregistered, steps, notAsked
} else {
notAsked = append(notAsked, steps...)
}
// A mover placed after the validate ended has its composition asked, or not asked, at once (§745).
more, _ := releaseCompositions(p, now)
more, _ := releaseCompositions(p, now, kept)
asks = append(asks, more...)
asks, ended := s.holderless(p, asks, now)
return asks, unregistered, ended, notAsked
@@ -526,6 +551,7 @@ func (s *Starter) Start(b Begin) (bool, error) {
asks, unregistered, ended, notAsked = s.placeMovers(&p, s.Registration, b.Entries, now)
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
}
if err := s.create(key, p); err != nil {
@@ -545,11 +571,12 @@ func (s *Starter) Start(b Begin) (bool, error) {
if s.Supersede != nil {
s.Supersede(p)
}
// ... and every older waiting one, whether this one waits or is placed at once (H-A-244 (1)).
if _, err := s.supersedeWaiting(p); err != nil {
s.logf("the older waiting pipelines of %s are not superseded by %s: %v", p.Branch, key, err)
}
if !place {
s.sayQueueWait(seat.EventWaitHeld, p, now, "")
if _, err := s.supersedeWaiting(p); err != nil {
s.logf("the older waiting pipelines of %s are not superseded by %s: %v", p.Branch, key, err)
}
s.Queue.overCap(key, s.logf)
return true, nil
}
@@ -603,13 +630,24 @@ func (s *Starter) Resume(store Updater, branch string, half Half) (int, error) {
}
}
var due []Asked
kept, err := keptOfStore(store, key)
if err != nil {
s.logf("the kept-revisions of %s are not read, so its kept movers' steps are not asked again: %v", key, err)
}
for _, m := range p.Movers {
reg, ok := s.Registration(m.Module)
if !ok {
continue
}
_, shared := Place(p, m, reg, half.Entries, now, s.Name)
_, unshared := place(p, m, reg, half.Entries, now, s.Name, false)
// A kept mover's steps are found by the ids its kept revision gives, from that revision's entries (H-A-286).
entries, rev := half.Entries, kept[m.Module]
if rev > 0 {
if entries, ok = s.keptEntries(p.Branch, rev); !ok {
continue
}
}
_, shared := place(p, m, reg, entries, now, s.Name, true, rev)
_, unshared := place(p, m, reg, entries, now, s.Name, false, rev)
for _, a := range shared {
if running[a.ID] && !own[a.ID] {
due = append(due, a)
@@ -664,6 +702,7 @@ func (s *Starter) Registered(store Updater, reg catalog.Registration, view func(
if reg.Module == "" {
return 0, nil
}
defer s.registrationArrived(reg.Module, "the registration of "+reg.Module+" arrived") // the finding of a mover that waited on it goes (H-A-262)
return s.placeWaiting(store, func(p seat.Pipeline) bool { return Moves(p, reg.Module) && !placed(p, reg.Module) },
func(module string) (catalog.Registration, bool) { return reg, module == reg.Module },
view, "the registration of "+reg.Module+", which arrived after the pipeline started")
@@ -677,6 +716,7 @@ func (s *Starter) Waiting(store Updater, branch string, view func(branch string)
if s.Registration == nil {
return 0, nil
}
defer s.JudgeRegistrationMissing(store) // the finding of a mover registered now, or no longer waited on, goes (H-A-262, H-A-265 (4))
return s.placeWaiting(store, func(p seat.Pipeline) bool {
if branch != "" && p.Branch != branch {
return false
@@ -729,6 +769,7 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
if len(p.Steps) == before {
continue
@@ -1012,6 +1053,12 @@ func (s *Starter) Revised(ev catalog.CatalogRevisionEvent, read func(branch stri
for _, e := range half.Movers {
movers = append(movers, seat.Mover{Module: e.Module, Commit: e.Commit, Fingerprint: e.Fingerprint, Repository: e.Repository})
}
// A revision of the same source moves no mover out of the open pipeline of that source holding it: that pipeline keeps
// it, and this revision starts no pipeline of its own for it (H-A-283); only a revision of a new source moves movers.
if movers = s.keepSameSource(branch, half, movers); len(movers) == 0 {
s.logf("the revision %d of %s (%s) moves only movers the open pipelines of its source %d keep: it starts no pipeline (H-A-283)", ev.Revision, branch, ev.Call, half.Source)
return 0, nil
}
movers = Tiers(movers, half.Entries, s.logf)
call := half.Call
if call == "" {
@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
condseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"slices"
"strings"
"testing"
@@ -87,6 +88,9 @@ func TestARevisionOfAReleaseStartsOnePipeline(t *testing.T) {
return Half{}, fmt.Errorf("branch %q", branch)
}
r := Half{Revision: revision, Source: 41, Entries: entries, Call: "release of mesh-sdk 0.1.17"}
if revision > 10 {
r.Source = 41 + revision // a later source's revision: one of the same source keeps its movers in 10's (H-A-283)
}
if revision != 12 {
r.Movers = movers
}
@@ -203,6 +207,8 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
var said []string
s := &Starter{Pipelines: store, Steps: asks, Name: "the-runner", Say: func(subject string, _ any) error { said = append(said, subject); return nil },
Registration: func(m string) (catalog.Registration, bool) { return reg, registered && m == "app" }}
findings := newHeld()
s.Findings, s.Observer = findings, "the-runner"
view := func(string) (Half, bool) { return Half{Entries: entries}, true }
taken := catalog.TakeInTaken{Branch: "main", Source: 4, Entries: []catalog.TakeInEntry{{Module: "app", Commit: "c1", Fingerprint: "src2:aa"}}}
if n, err := s.TakenIn(taken, view); n != 1 || err != nil {
@@ -211,8 +217,15 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
if p, _, _ := store.Get("main.4"); len(p.Steps) != 0 || len(asks.asks) != 0 {
t.Fatalf("before its registration the pipeline has %d steps and %d asks, want none", len(p.Steps), len(asks.asks))
}
// H-A-262: the mover waiting on its registration is the finding registration-missing, scope module, id the module.
if f, ok := findings.held["module.app."+RegistrationMissingKind]; !ok || f.Observer != "the-runner" || !strings.Contains(f.Summary, "main.4") {
t.Fatalf("the mover waiting on its registration raised %v, want registration-missing of app naming main.4", findings.held)
}
registered = true
n, err := s.Registered(store.memStore, reg, view)
if _, ok := findings.held["module.app."+RegistrationMissingKind]; ok {
t.Errorf("the registration of app arrived and its registration-missing is still held")
}
p, _, _ := store.Get("main.4")
if n != 1 || err != nil || len(p.Steps) == 0 || len(asks.asks) == 0 || p.Outcome != seat.PipelineOutcomeRunning {
t.Fatalf("the registration placed %d (%v): %d steps, %d asks, want the pipeline's steps placed and asked", n, err, len(p.Steps), len(asks.asks))
@@ -303,6 +316,41 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
if !slices.ContainsFunc(logged, func(l string) bool { return strings.Contains(l, "next.8: contracts has no registration yet") }) {
t.Errorf("placing next.8 said %q, want that contracts waits on its registration (A-327)", logged)
}
// H-A-262: the mover still waiting is raised from the placing too, and cleared once the registrations held name it.
if _, ok := findings.held["module.contracts."+RegistrationMissingKind]; !ok {
t.Errorf("placing next.8 raised %v, want registration-missing of contracts", findings.held)
}
// H-A-265 (4): a pipeline that ends with the mover still waiting (failed, H-A-163) leaves no finding: judged at its end
// (the hearer's said) and in Waiting, it clears once no running pipeline has an unplaced mover of the module.
next8, rev8, _ := store.Get("next.8")
next8.Outcome = seat.PipelineOutcomeFailed
if _, err := store.Update("next.8", next8, rev8); err != nil {
t.Fatal(err)
}
h := &Hearer{Store: store.memStore, Say: func(string, any) error { return nil }, RegistrationWaits: func() { s.JudgeRegistrationMissing(store.memStore) }}
if err := h.said(next8, nil, true); err != nil {
t.Fatal(err)
}
if _, ok := findings.held["module.contracts."+RegistrationMissingKind]; ok {
t.Errorf("next.8 ended with contracts unplaced and no other pipeline waits on it, yet its registration-missing is held")
}
// Still waited on by a running pipeline, it stays at an end; the registrations held naming it clear it in Waiting.
findings.held["module.contracts."+RegistrationMissingKind] = condseat.Finding{Scope: "module", ID: "contracts", Kind: RegistrationMissingKind}
taken = catalog.TakeInTaken{Branch: "next", Source: 9, Entries: []catalog.TakeInEntry{{Module: "contracts", Commit: "c5", Fingerprint: "src2:ff"}}}
if n, err := s.TakenIn(taken, late); n != 1 || err != nil {
t.Fatalf("the take-in of contracts started %d (%v)", n, err)
}
s.JudgeRegistrationMissing(store.memStore)
if _, ok := findings.held["module.contracts."+RegistrationMissingKind]; !ok {
t.Errorf("next.9 still waits on contracts' registration, yet its registration-missing was cleared")
}
s.Registration = func(string) (catalog.Registration, bool) { return reg, true }
if _, err := s.Waiting(store.memStore, "", late); err != nil {
t.Fatal(err)
}
if _, ok := findings.held["module.contracts."+RegistrationMissingKind]; ok {
t.Errorf("the registrations held name contracts and its registration-missing is still held")
}
}
// R1/AC1 (design 47 H §747): a start names an event, never a sweep. A take-in starts the pipelines of its own entries
@@ -147,12 +147,16 @@ type Hearer struct {
Now func() time.Time
// Failures says a failed head as the finding pipeline-failed (§33); nil says none.
Failures *Failures
// RegistrationWaits judges, at every pipeline's end, the finding registration-missing of the modules no running pipeline
// still waits on (Starter.JudgeRegistrationMissing, H-A-265 (4)); nil judges none.
RegistrationWaits func()
// Ager ages the runner's own kept data at each step's end (§330); nil ages none.
Ager *Ager
// Fills says the stores' fill at each step's end (§366); nil says none.
Fills *Fills
// Widens computes a mover's widens, kept in its value at the pipeline's end (§71, A-207; H-A-51); nil keeps none.
Widens func(p seat.Pipeline, module string) []seat.Widening
// Widens makes, for one end, the reader of each mover's widens, kept in its value at the pipeline's end (§71, A-207;
// H-A-51): made before the end's compare-and-set, its reads shared by every mover and every retry (H-A-289); nil keeps none.
Widens func() func(p seat.Pipeline, module string) []seat.Widening
// TestRuns is the state test-runs and Clock its bound keys (§725); Runs the settings of its runs; DropRun drops a
// run's ask when its run ends unfinished (§119). A nil TestRuns keeps no test run.
TestRuns TestRuns
@@ -176,6 +180,9 @@ type Hearer struct {
// Unplaced says why a mover of a pipeline has no step placed, for the why of its end (H-A-171; Starter.Unplaced): it
// waits on its registration or on a publish; nil says only that it has no step placed.
Unplaced func(p seat.Pipeline, module string) string
// Asks clears the finding of an ask that ended its step unanswered once it is answered after all (H-A-238 (2)); nil
// clears none.
Asks *AskClock
// Rejoin keys a pipeline running again once its approval is answered and it still runs (H-A-106 (b); Starter.Rejoin);
// nil keys none.
Rejoin func(key string)
@@ -193,6 +200,15 @@ func (h *Hearer) EndReaders(run string, readers []seat.TestRunReader, r seat.Tes
return first
}
// kept is pipeline key's kept-revisions (H-A-285, H-A-286); none when they cannot be read, said.
func (h *Hearer) kept(key string) map[string]int64 {
kept, err := keptOfStore(h.Store, key)
if err != nil {
h.logf("the kept-revisions of %s are not read, so its kept movers' asks take the ids of movers not kept: %v", key, err)
}
return kept
}
func (h *Hearer) now() time.Time {
if h.Now != nil {
return h.Now().UTC()
@@ -230,12 +246,19 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
if ev.Asker != "" && ev.Asker != h.Name {
return seat.Pipeline{}, nil
}
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(ev.Pipeline)
if err != nil {
return seat.Pipeline{}, fmt.Errorf("%w: ask %s names pipeline %q, which the runner cannot read: %v", ErrRefused, ev.ID, ev.Pipeline, err)
}
i := stepOfAsk(p, ev.ID)
// The validate ask is the pipeline's one, naming no mover (H-A-51): its end, naming the mover whose body the holder
// took, is matched by its ask alone and ends every mover's validate step under it (H-A-252).
oneValidate := i >= 0 && p.Steps[i].Step == seat.StepResultStepValidate && p.Steps[i].Seat == wants.Seat
if oneValidate {
i = validateStepOfAsk(p, ev.ID, ev.Module)
}
var m seat.Mover
if i >= 0 {
m, _ = MoverOf(p, p.Steps[i].Module)
@@ -245,10 +268,11 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
return p, fmt.Errorf("%w: pipeline %s asked no step under ask %s", ErrRefused, ev.Pipeline, ev.ID)
case p.Steps[i].Seat != bs.Seat:
return p, fmt.Errorf("%w: ask %s was asked of %s, not %s", ErrRefused, ev.ID, p.Steps[i].Seat, bs.Seat)
case ev.Commit != m.Commit || ev.Fingerprint != m.Fingerprint || ev.Module != m.Module || ev.Step != p.Steps[i].Step:
case ev.Step != p.Steps[i].Step || (!oneValidate && (ev.Commit != m.Commit || ev.Fingerprint != m.Fingerprint || ev.Module != m.Module)):
return p, fmt.Errorf("%w: ask %s ended %s of %s at %s (%s), asked as %s of %s at %s (%s)", ErrRefused, ev.ID, ev.Step,
ev.Module, ev.Commit, ev.Fingerprint, p.Steps[i].Step, p.Steps[i].Module, m.Commit, m.Fingerprint)
case !open(p.Steps[i]):
h.Asks.answeredLate(p.Steps[i], h.logf)
if run := sharedRun(p.Steps[i]); run != "" && h.TestRuns != nil {
if r, _, err := h.TestRuns.Get(run); err == nil && r.Ended == nil && r.Ask == ev.ID {
// The asking step's end was written and its run's was not (a write failed): the run ends now, heard again.
@@ -301,7 +325,7 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
if s.Step == seat.StepResultStepValidate {
readers = append(readers, sameValidate(&p, i, now)...)
var notAsked []int
due, notAsked = releaseCompositions(&p, now)
due, notAsked = releaseCompositions(&p, now, h.kept(ev.Pipeline))
readers = append(readers, notAsked...)
}
from := now
@@ -315,7 +339,7 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
// A mover's build passed is what a kind-3 edge waits for (§746, H-A-83 (2)); one that did not pass ends its
// dependents' steps not asked.
release := s.Step == seat.StepResultStepBuild && (!stepPassed(*s) || built(&p, s.Module))
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(ev.Pipeline, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
@@ -455,7 +479,16 @@ func (h *Hearer) said(p seat.Pipeline, asked []int, ended bool) error {
h.Turn(p.Pipeline)
}
if ended {
// The journal says which end it was (H-A-288).
if p.Why != "" {
h.logf("%s ended %s: %s", p.Pipeline, p.Outcome, p.Why)
} else {
h.logf("%s ended %s", p.Pipeline, p.Outcome)
}
h.Failures.Ended(p)
if h.RegistrationWaits != nil {
h.RegistrationWaits()
}
err := h.say(seat.EventPipelineEnded, p, nil)
// Every end, a cancel's alike, places the next waiting (§748, H-A-53), said or not.
if h.Turn != nil {
@@ -468,6 +501,51 @@ func (h *Hearer) said(p seat.Pipeline, asked []int, ended bool) error {
return nil
}
// EndAtStart judges, at the runner's start pass, the end of every running pipeline nothing of which is still open (H-A-288
// (c)): one whose last mover left or whose last open step ended while no runner wrote its end ends as at a step's end,
// superseded, failed or passed by the end judgement, said, and the next waiting placed. It answers how many it ended.
func (h *Hearer) EndAtStart() (int, error) {
pkeys, err := allKeys(h.Store)
if err != nil {
return 0, fmt.Errorf("the pipelines are not listed: %w", err)
}
n := 0
var first error
for _, key := range pkeys {
ended, err := h.endIfNothingOpen(key)
if err != nil && first == nil {
first = err
}
if ended {
n++
}
}
return n, first
}
// endIfNothingOpen ends the running pipeline key when nothing of it is still open (H-A-288), its reads made before the
// compare-and-set (H-A-289); true when it ended.
func (h *Hearer) endIfNothingOpen(key string) (bool, error) {
wr := h.widensReads()
for {
p, rev, err := h.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning {
return false, nil
}
now := h.now()
if !h.endPipeline(&p, now, wr) {
return false, nil
}
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
return false, fmt.Errorf("pipeline %s cannot be written: %w", key, err)
}
h.logf("%s had nothing open at the runner's start: its end is written now (H-A-288)", key)
return true, h.said(p, nil, true)
}
}
// endEntry is the step log's entry of a step's end, with the keys of the kept results it wrote or carried.
func endEntry(key string, s seat.StepResult, at time.Time, kept ...string) seat.LogEntry {
return seat.LogEntry{Pipeline: key, Module: s.Module, Step: s.Step, Artifact: s.Artifact, What: seat.LogEntryWhatEnded, Outcome: s.Outcome,
@@ -583,9 +661,36 @@ func endOutcome(o string) bool {
return false
}
// validateStepOfAsk is the validate step the end of the shared validate ask id ends first (H-A-252): the open one of the mover
// the end names, else any open one under the ask (sameValidate ends the rest), else the first under it.
func validateStepOfAsk(p seat.Pipeline, id, module string) int {
first, anyOpen := -1, -1
for i, s := range p.Steps {
if s.Step != seat.StepResultStepValidate || s.Ask == "" || (s.Ask != id && AgainID(s.Ask) != id) {
continue
}
if first < 0 {
first = i
}
if open(s) {
if s.Module == module {
return i
}
if anyOpen < 0 {
anyOpen = i
}
}
}
if anyOpen >= 0 {
return anyOpen
}
return first
}
func stepOfAsk(p seat.Pipeline, id string) int {
for i, s := range p.Steps {
if id != "" && s.Ask == id {
// An ask put again is put under AgainID of the step's ask (H-A-243 (3)): its holder's answer under either is the step's.
if id != "" && s.Ask != "" && (s.Ask == id || AgainID(s.Ask) == id) {
return i
}
}
@@ -677,19 +782,28 @@ func endPipeline(p *seat.Pipeline, now time.Time, unplaced func(seat.Pipeline, s
return true
}
// endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51).
func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time) bool {
// endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51), by wr, the
// reads the caller made before its compare-and-set (H-A-289); nil computes none.
func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time, wr func(seat.Pipeline, string) []seat.Widening) bool {
if !endPipeline(p, now, h.Unplaced) {
return false
}
if h.Widens != nil {
if wr != nil {
for i := range p.Movers {
p.Movers[i].Widens = h.Widens(*p, p.Movers[i].Module)
p.Movers[i].Widens = wr(*p, p.Movers[i].Module)
}
}
return true
}
// widensReads is the reader of one end's widens, made before its compare-and-set and kept for every retry (H-A-289).
func (h *Hearer) widensReads() func(seat.Pipeline, string) []seat.Widening {
if h.Widens == nil {
return nil
}
return h.Widens()
}
// EndedAtPlacing says a pipeline its starter ended at placing, a seat with no holder (A-470g, A-470h): the step-ended of each
// step of steps, its step log entries appended, the head's finding pipeline-failed (§33), then its pipeline-ended.
func (h *Hearer) EndedAtPlacing(p seat.Pipeline, steps []int) error {
@@ -363,7 +363,7 @@ func TestARecoverysStepsAreAskedOnTheStreamDelivery(t *testing.T) {
}{{true, "mesh.seat.node-build-go.accept.delivery.mesh-pipeline-runner," + wants.AcceptSubjectStep}, {false, "mesh.seat.node-build-go.accept.check.mesh-pipeline-runner," + wants.AcceptSubjectStep}} {
var to []string
asker := askerFunc(func(accept, _ string, _ any) error { to = append(to, accept); return nil })
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }); err != nil || strings.Join(to, ",") != c.want {
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }, nil); err != nil || strings.Join(to, ",") != c.want {
t.Errorf("recovery %v: asked on %v (%v), want %s", c.recovery, to, err, c.want)
}
}
@@ -34,6 +34,9 @@ func (h *Hearer) Taken(ask, node string) error {
break
}
i := stepOfAsk(p, ask)
if i >= 0 && !open(p.Steps[i]) {
h.Asks.answeredLate(p.Steps[i], h.logf)
}
if i < 0 || !open(p.Steps[i]) || p.Steps[i].Node == node {
break
}
@@ -88,6 +91,7 @@ func (h *Hearer) SupersedeRunning(newer seat.Pipeline) []string {
}
func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning || p.Branch != newer.Branch || p.Run != "" || newer.Run != "" ||
@@ -115,18 +119,12 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
if !slices.Contains(moved, st.Module) || (!open(*st) && st.Outcome != seat.StepResultOutcomeWaiting) {
continue
}
switch {
case sharedRun(*st) != "" && st.Seat != wants.Seat:
leave = append(leave, *st)
case st.Ask != "" && BoundSeat(st.Seat):
drop = append(drop, *st) // its own ask on a build node: the seat's drop cancels it
}
st.Outcome, st.Why, st.Ended = seat.StepResultOutcomeSuperseded, why, &now
p.Unlogged = append(p.Unlogged, endEntry(key, *st, now))
l, d := supersedeStep(&p, st, why, now)
leave, drop = append(leave, l...), append(drop, d...)
ended = append(ended, i)
}
p.Superseded = append(p.Superseded, moved...)
done := h.endPipeline(&p, now)
done := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
@@ -139,20 +137,7 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
h.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, p.Steps[i].Module, p.Steps[i].Step, err)
}
}
for _, st := range leave {
h.leaveRun(key, st, "the mover "+st.Module+" left "+key+" for the newer "+newer.Pipeline)
}
// Its running asks on the build nodes are cancelled through the seat's drop (H-A-223). The seat mesh-builds serves no
// drop, so a build ask there runs to its end, which the runner ignores: the step is no longer open.
for _, st := range drop {
if h.DropRun == nil {
h.logf("%s: the ask %s of %s's %s is not dropped: the runner has no drop", key, st.Ask, st.Module, st.Step)
continue
}
if err := h.DropRun(st.Seat, st.Node, st.Ask, why); err != nil {
h.logf("%s: the drop of the ask %s of %s's %s is not asked of %s: %v", key, st.Ask, st.Module, st.Step, st.Seat, err)
}
}
h.leaveSteps(key, leave, drop, "the mover %s left "+key+" for the newer "+newer.Pipeline, why)
if err := h.said(p, nil, done); err != nil {
h.logf("%v", err)
}
@@ -161,12 +146,52 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
}
}
// supersedeStep ends the open or waiting step st of p superseded, naming why, with its step log entry: a step reading a test
// run is answered to leave it, and one with its own ask on a build node to drop it (H-A-223).
func supersedeStep(p *seat.Pipeline, st *seat.StepResult, why string, now time.Time) (leave, drop []seat.StepResult) {
switch {
case sharedRun(*st) != "" && st.Seat != wants.Seat:
leave = append(leave, *st)
case st.Ask != "" && BoundSeat(st.Seat):
drop = append(drop, *st) // its own ask on a build node: the seat's drop cancels it
}
st.Outcome, st.Why, st.Ended = seat.StepResultOutcomeSuperseded, why, &now
p.Unlogged = append(p.Unlogged, endEntry(p.Pipeline, *st, now))
return leave, drop
}
// leaveSteps takes the superseded steps of pipeline key out of the test runs they read (leaving, its %s the step's mover)
// and drops their own asks on the build nodes, naming why (H-A-223). The seat mesh-builds serves no drop, so a build ask
// there runs to its end, which the runner ignores: the step is no longer open.
func (h *Hearer) leaveSteps(key string, leave, drop []seat.StepResult, leaving, why string) {
for _, st := range leave {
h.leaveRun(key, st, fmt.Sprintf(leaving, st.Module))
}
for _, st := range drop {
if h.DropRun == nil {
h.logf("%s: the ask %s of %s's %s is not dropped: the runner has no drop", key, st.Ask, st.Module, st.Step)
continue
}
if err := h.DropRun(st.Seat, st.Node, st.Ask, why); err != nil {
h.logf("%s: the drop of the ask %s of %s's %s is not asked of %s: %v", key, st.Ask, st.Module, st.Step, st.Seat, err)
}
}
}
// LeftInPlace takes the steps a same-source re-pin superseded in pipeline key, whose mover it kept (H-A-283, H-A-284 (1)),
// out of the test runs they read and drops their own asks, naming why, and appends their step log entries.
func (h *Hearer) LeftInPlace(key string, leave, drop []seat.StepResult, why string) {
h.leaveSteps(key, leave, drop, "the mover %s of "+key+" was re-pinned and its steps placed again (H-A-284)", why)
h.appendUnlogged(key)
}
// supersedeWaiting takes, in each older waiting pipeline of newer's branch, the movers newer also moves out of it (design 47 H
// §748, ADR 0682 §7): a waiting pipeline left with no mover ends superseded, its waiting key deleted, saying wait-ended and its
// pipeline-ended. newer is a pipeline just written waiting; a running pipeline's movers leave it by SupersedeRunning
// (H-A-223). It answers the pipelines it ended.
// pipeline-ended. newer is a pipeline just written, waiting or placed: a mover a newer revision moves leaves every pipeline not
// ended, a waiting one as a running one (H-A-223, H-A-244 (1)), so a waiting pipeline is placed with only the movers still its
// own; a running pipeline's movers leave it by SupersedeRunning. It answers the pipelines it ended.
func (s *Starter) supersedeWaiting(newer seat.Pipeline) ([]string, error) {
if s.Queue == nil || newer.Outcome != seat.PipelineOutcomeWaiting {
if s.Queue == nil || (newer.Outcome != seat.PipelineOutcomeWaiting && newer.Outcome != seat.PipelineOutcomeRunning) {
return nil, nil
}
wkeys, pipelines, err := s.Queue.Waiting()
@@ -193,7 +218,7 @@ func (s *Starter) supersedeWaiting(newer seat.Pipeline) ([]string, error) {
func (s *Starter) supersedeOne(wkey, key string, newer seat.Pipeline) (bool, error) {
for {
p, rev, err := s.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeWaiting || p.Branch != newer.Branch || p.Seq >= newer.Seq {
if err != nil || p.Outcome != seat.PipelineOutcomeWaiting || p.Branch != newer.Branch || !olderWaiting(p, newer) {
return false, nil
}
left := slices.DeleteFunc(slices.Clone(p.Movers), func(m seat.Mover) bool { return Moves(newer, m.Module) })
@@ -232,6 +257,15 @@ func (s *Starter) supersedeOne(wkey, key string, newer seat.Pipeline) (bool, err
}
}
// olderWaiting says whether the waiting pipeline p is older than newer: by its sequence in the queue when newer waits too,
// by its catalog revision when newer was placed at once (H-A-244 (1)); a run on request's pipelines are never compared.
func olderWaiting(p, newer seat.Pipeline) bool {
if newer.Outcome == seat.PipelineOutcomeWaiting {
return p.Seq < newer.Seq
}
return p.Run == "" && newer.Run == "" && p.Revision < newer.Revision
}
// supersededWhy writes, into mover module's build steps, that a dependency it builds on was taken out of p by a newer
// pipeline, so it builds against the newest published package (design 47 H §748; H-A-53, H-A-107).
func supersededWhy(p seat.Pipeline, module string, entries map[string]catalog.CatalogEntry, steps []seat.StepResult) {
@@ -3,10 +3,10 @@ package pipeline
import (
"errors"
"fmt"
"slices"
"strings"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
@@ -52,6 +52,7 @@ func (v Verbs) record(verb, what, why string) error {
func (v Verbs) change(key, event string, record func() error, mutate func(p *seat.Pipeline, now time.Time) (int, error)) (seat.Pipeline, int, error) {
h := v.Hearer
recorded := false
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
@@ -73,7 +74,7 @@ func (v Verbs) change(key, event string, record func() error, mutate func(p *sea
if err != nil {
return p, i, err
}
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
if i >= 0 && event == seat.EventStepEnded {
p.Unlogged = append(p.Unlogged, endEntry(key, p.Steps[i], now))
}
@@ -297,11 +298,15 @@ func (v Verbs) Rerun(caller string, in seat.RerunInput) (seat.RerunAnswer, error
return -1, err
}
}
oldAsk := s.Ask
*s = seat.StepResult{Step: s.Step, Module: s.Module, Artifact: s.Artifact, Status: s.Status, Fingerprint: s.Fingerprint, Seat: s.Seat,
Outcome: seat.StepResultOutcomeWaiting, By: caller, Rerun: true}
if ask != nil {
s.Outcome, s.Started, s.Ask = seat.StepResultOutcomeRunning, &now, ask.ID
}
if ask != nil && s.Step == seat.StepResultStepValidate && s.Seat == wants.Seat {
rerunValidate(p, i, oldAsk, caller, now)
}
// The pipeline runs again; an approval never asked waits again, to be asked when the step passes.
if p.Outcome != seat.PipelineOutcomeRunning {
p.Outcome, p.Why, p.Ended = seat.PipelineOutcomeRunning, "", nil
@@ -321,7 +326,7 @@ func (v Verbs) Rerun(caller string, in seat.RerunInput) (seat.RerunAnswer, error
}
if ask != nil {
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -468,7 +473,7 @@ func (v Verbs) rerunTest(caller string, in seat.RerunInput, st seat.StepResult)
break
}
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -480,6 +485,29 @@ func (v Verbs) rerunTest(caller string, in seat.RerunInput, st seat.StepResult)
return seat.RerunAnswer{Pipeline: in.Pipeline, Result: &res}, nil
}
// rerunValidate puts, with the rerun of validate step i, every other mover's validate step under the same ask back to running
// under its one new id, a mover that left the pipeline skipped, so the pipeline keeps one validate ask (H-A-51); and each
// composition its validate held ended not asked waits again, asked once validate passed as at the placing (§745).
func rerunValidate(p *seat.Pipeline, i int, oldAsk, caller string, now time.Time) {
id := p.Steps[i].Ask
for j := range p.Steps {
st := &p.Steps[j]
if slices.Contains(p.Superseded, st.Module) {
continue
}
switch {
case j != i && st.Step == seat.StepResultStepValidate && st.Seat == wants.Seat && oldAsk != "" &&
(st.Ask == oldAsk || st.Ask == AgainID(oldAsk)) && st.Outcome != seat.StepResultOutcomeSuperseded && st.Outcome != seat.StepResultOutcomeCancelled:
*st = seat.StepResult{Step: st.Step, Module: st.Module, Artifact: st.Artifact, Status: st.Status, Fingerprint: st.Fingerprint, Seat: st.Seat,
Outcome: seat.StepResultOutcomeRunning, Started: &now, Ask: id, By: caller, Rerun: true}
case st.Step == seat.StepResultStepComposition && st.Seat == wants.Seat && st.Outcome == seat.StepResultOutcomeNotAsked &&
strings.HasPrefix(st.Why, "validate ended "):
*st = seat.StepResult{Step: st.Step, Module: st.Module, Artifact: st.Artifact, Status: st.Status, Fingerprint: st.Fingerprint, Seat: st.Seat,
Outcome: seat.StepResultOutcomeWaiting, Why: "asked once validate passed"}
}
}
}
// passedReader says whether reader's step of run passed, or a person accepted it (§30): a step that moves with no new run.
func (v Verbs) passedReader(reader RunReader, run string) bool {
p, _, err := v.Hearer.Store.Get(reader.Pipeline)
@@ -500,8 +528,26 @@ func (v Verbs) passedReader(reader RunReader, run string) bool {
}
// askOf is the ask to put on the serving seat for step s again: the one Place makes for it, under a new id (an ask put twice
// under one id is kept once), saying it is a rerun and by whom.
// under one id is kept once), saying it is a rerun and by whom. The validate ask is the pipeline's one, naming no mover
// (H-A-51): its rerun asks that one ask again; a composition's rerun builds the mover's composition ask (H-A-236).
func (v Verbs) askOf(p seat.Pipeline, s seat.StepResult, caller string, now time.Time) (*Asked, error) {
// A kept mover's rerun mixes its kept revision into the id (H-A-286).
rev := v.keptOf(p.Pipeline)[s.Module]
id := keptAskID(now, p.Pipeline, s.Module, s.Artifact, s.Step, rev)
if s.Seat == wants.Seat && s.Step == seat.StepResultStepValidate {
id = keptAskID(now, p.Pipeline, "", "", seat.StepResultStepValidate, rev)
}
a, err := v.placedAsk(p, s, id, now)
if err != nil {
return nil, err
}
markRerun(a, caller)
return a, nil
}
// placedAsk is the ask of step s under id: a validate or composition step of the seat mesh-wants built directly for its
// mover (H-A-236), any other the one Place makes for it.
func (v Verbs) placedAsk(p seat.Pipeline, s seat.StepResult, id string, now time.Time) (*Asked, error) {
if v.Starter == nil || v.Starter.Registration == nil || v.Starter.Steps == nil || (v.View == nil && p.Run == "") {
return nil, errors.New("the runner cannot ask a step: it holds no registration or no ask")
}
@@ -509,6 +555,16 @@ func (v Verbs) askOf(p seat.Pipeline, s seat.StepResult, caller string, now time
if !ok {
return nil, fmt.Errorf("%s moves no module %q", p.Pipeline, s.Module)
}
if s.Seat == wants.Seat {
switch s.Step {
case seat.StepResultStepValidate:
a := wantsAsk(p, m, id, wants.StepRunStepValidate)
return &a, nil
case seat.StepResultStepComposition:
a := wantsAsk(p, m, id, wants.StepRunStepComposition)
return &a, nil
}
}
reg, ok := v.Starter.Registration(m.Module)
if !ok {
return nil, fmt.Errorf("%s has no registration", m.Module)
@@ -527,27 +583,22 @@ func (v Verbs) askOf(p seat.Pipeline, s seat.StepResult, caller string, now time
} else {
return nil, fmt.Errorf("the catalog half of %s is not held yet: the step is asked when it is", p.Branch)
}
_, placed := place(p, m, reg, half.Entries, now, v.Starter.Name, false)
base := AskID(startOf(p, now), p.Pipeline, m.Module, s.Artifact, s.Step)
// A kept mover's step is the one placed from the revision it is kept at, under its kept ids (H-A-286).
entries, rev := half.Entries, v.keptOf(p.Pipeline)[m.Module]
if rev > 0 {
if entries, ok = v.Starter.keptEntries(p.Branch, rev); !ok {
return nil, fmt.Errorf("revision %d of %s, which %s is kept at, is not read: the step is asked when it is", rev, p.Branch, m.Module)
}
}
_, placed := place(p, m, reg, entries, now, v.Starter.Name, false, rev)
base := keptAskID(startOf(p, now), p.Pipeline, m.Module, s.Artifact, s.Step, rev)
for _, a := range placed {
if a.ID != base {
continue
}
a.ID = AskID(now, p.Pipeline, m.Module, s.Artifact, s.Step)
switch b := a.Ask.(type) {
case nodebuildgo.StepAsk:
b.ID = a.ID
if b.Step != nil {
step := *b.Step
step.Rerun, step.By = true, caller
b.Step = &step
}
a.Ask = b
case builds.StepRun:
b.Rerun, b.By = true, caller
a.Ask = b
case wants.StepRun:
b.Rerun, b.By = true, caller
a.ID = id
if b, ok := a.Ask.(nodebuildgo.StepAsk); ok {
b.ID = id
a.Ask = b
}
return &a, nil
@@ -555,6 +606,14 @@ func (v Verbs) askOf(p seat.Pipeline, s seat.StepResult, caller string, now time
return nil, fmt.Errorf("the %s step of %s is placed no ask now", s.Step, p.Pipeline)
}
// keptOf is pipeline key's kept-revisions (H-A-285); none when they cannot be read, said.
func (v Verbs) keptOf(key string) map[string]int64 {
if v.Hearer == nil {
return nil
}
return v.Hearer.kept(key)
}
// Run starts a run on request for the caller the bus proved (design 47 G §64; Starter.Run): its pipelines written, its
// run-started said, its steps placed when its candidate's revision is heard.
func (v Verbs) Run(caller string, in seat.RunInput) (seat.RunAnswer, error) {
@@ -8,6 +8,7 @@ import (
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-sdk/go/views"
)
@@ -252,6 +253,47 @@ func TestRerunOfARunAsksWithItsCandidatesHalf(t *testing.T) {
}
}
// H-A-236: a rerun of a failed pipeline's validate step asks the pipeline's one validate ask again (H-A-51), naming no
// mover in its id; a rerun of a composition step builds the mover's composition ask directly. Neither is refused "placed no
// ask now".
func TestRerunOfValidateOrCompositionAsksTheSharedAsk(t *testing.T) {
v, store, _, asks, _, _ := verbsFixture()
reg, entries := placeFixture()
reg.Policy.Steps.Validate, reg.Policy.Steps.Composition = "required", "required"
v.Starter.Registration = func(string) (catalog.Registration, bool) { return reg, true }
p, rev, _ := store.Get(pkey)
p.Steps, _ = Place(p, p.Movers[0], reg, entries, time.Unix(1, 0), "the-runner")
for i := range p.Steps {
switch p.Steps[i].Step {
case seat.StepResultStepValidate:
p.Steps[i].Outcome = seat.StepResultOutcomeFailed
case seat.StepResultStepComposition:
p.Steps[i].Outcome = seat.StepResultOutcomeNotAsked
}
}
p.Outcome = seat.PipelineOutcomeFailed
store.Update(pkey, p, rev)
now := v.now()
for _, c := range []struct{ step, module string }{{seat.StepResultStepValidate, ""}, {seat.StepResultStepComposition, "app"}} {
n := len(asks.asks)
r, err := v.Rerun("operator", seat.RerunInput{Pipeline: pkey, Module: "app", Step: c.step, Why: "the holder restarted"})
want := AskID(now, pkey, c.module, "", c.step)
if err != nil || r.Result == nil || r.Result.Ask != want || r.Result.Outcome != seat.StepResultOutcomeRunning {
t.Fatalf("a rerun of %s = %+v, %v, want running under %s", c.step, r.Result, err, want)
}
if len(asks.asks) != n+1 || asks.ids[n] != want {
t.Fatalf("a rerun of %s asked %v, want one ask under %s", c.step, asks.ids[n:], want)
}
b, ok := asks.asks[n].(wants.StepRun)
if !ok || b.Step != c.step || b.Module != "app" || b.Pipeline != pkey || !b.Rerun || b.By != "operator" || b.Asker != seat.Seat {
t.Errorf("the %s ask = %+v", c.step, asks.asks[n])
}
p, rev, _ = store.Get(pkey)
p.Steps[stepIndex(p, c.step, "")].Outcome = seat.StepResultOutcomeFailed
store.Update(pkey, p, rev)
}
}
func stepIndex(p seat.Pipeline, step, artifact string) int {
for i, s := range p.Steps {
if s.Step == step && s.Artifact == artifact {
@@ -340,3 +382,44 @@ func TestStartAndAskAgain(t *testing.T) {
t.Error("an approval running was asked again")
}
}
// Review of H-A-236 (H-A-51): a rerun of validate puts every mover's validate step under the old ask back to running under
// the one new id, a mover that left skipped, and the compositions its failed validate ended not asked wait again; the new
// ask's end ends both validates and asks both compositions.
func TestAValidateRerunMovesEveryMoversValidateAndReopensItsCompositions(t *testing.T) {
v, store, _, asks, _, _ := verbsFixture()
p, rev, _ := store.Get(pkey)
p.Outcome = seat.PipelineOutcomeFailed
p.Movers = append(p.Movers, seat.Mover{Module: "lib", Commit: "c2", Fingerprint: "src2:bb"}, seat.Mover{Module: "gone", Commit: "c3", Fingerprint: "src2:cc"})
p.Superseded = []string{"gone"}
notAsked := "validate ended failed: nothing is composed"
p.Steps = nil
for _, m := range []string{"app", "lib", "gone"} {
p.Steps = append(p.Steps,
seat.StepResult{Module: m, Step: seat.StepResultStepValidate, Status: seat.StepResultStatusRequired, Seat: wants.Seat, Ask: "old", Outcome: seat.StepResultOutcomeFailed},
seat.StepResult{Module: m, Step: seat.StepResultStepComposition, Status: seat.StepResultStatusRequired, Seat: wants.Seat, Outcome: seat.StepResultOutcomeNotAsked, Why: notAsked})
}
p.Steps[4].Outcome, p.Steps[5].Outcome = seat.StepResultOutcomeSuperseded, seat.StepResultOutcomeSuperseded
store.Update(pkey, p, rev)
r, err := v.Rerun("operator", seat.RerunInput{Pipeline: pkey, Module: "app", Step: seat.StepResultStepValidate, Why: "again"})
if err != nil || len(asks.ids) != 1 || r.Result.Ask != asks.ids[0] {
t.Fatalf("the rerun = %+v (%v), asked %v; want one validate ask", r.Result, err, asks.ids)
}
got, _, _ := store.Get(pkey)
for i, want := range []string{"running " + r.Result.Ask, "waiting ", "running " + r.Result.Ask, "waiting ", "superseded old", "superseded "} {
if s := got.Steps[i]; s.Outcome+" "+s.Ask != want {
t.Errorf("%s's %s = %s %q, want %s", s.Module, s.Step, s.Outcome, s.Ask, want)
}
}
bs := StepSeats[len(StepSeats)-1]
ev := StepEnded{ID: r.Result.Ask, Node: "n1", Pipeline: pkey, Module: "lib", Commit: "c2", Fingerprint: "src2:bb", Step: seat.StepResultStepValidate, Outcome: "passed"}
if _, err := v.Hearer.OnStepEnded(bs, bs.Prefix+"n1", ev); err != nil {
t.Fatal(err)
}
got, _, _ = store.Get(pkey)
for _, i := range []int{1, 3} {
if s := got.Steps[i]; s.Outcome != seat.StepResultOutcomeRunning || s.Ask == "" {
t.Errorf("%s's composition = %s %q, want asked once validate passed", s.Module, s.Outcome, s.Ask)
}
}
}
@@ -31,20 +31,87 @@ func (w *Widener) Widens(p seat.Pipeline, module string) []seat.Widening {
if w == nil || w.Revision == nil {
return nil
}
return w.Reads()(p, module)
}
// Reads is Widens over reads made once and kept (H-A-289): the two revisions, each serving seat's holder and the grants part's
// grants-for are each read at their first need and shared by every mover of an end and every retry of its write, so a retry
// rereads only the pipeline's key. One end takes one Reads, made before its compare-and-set.
func (w *Widener) Reads() func(p seat.Pipeline, module string) []seat.Widening {
if w == nil || w.Revision == nil {
return func(seat.Pipeline, string) []seat.Widening { return nil }
}
r := &endReads{w: w, revisions: map[string]revisionRead{}, holders: map[string]holderRead{}, grants: map[string]grantsRead{}}
return r.widens
}
type revisionRead struct {
half Half
err error
}
type holderRead struct {
holder string
err error
}
type grantsRead struct {
g meshgrants.GrantsFor
err error
}
// endReads are the reads of one end, each made at its first need and kept.
type endReads struct {
w *Widener
revisions map[string]revisionRead
holders map[string]holderRead
grants map[string]grantsRead
}
func (r *endReads) revision(branch string, rev int64) (Half, error) {
k := fmt.Sprintf("%s.%d", branch, rev)
got, ok := r.revisions[k]
if !ok {
got.half, got.err = r.w.Revision(branch, rev)
r.revisions[k] = got
}
return got.half, got.err
}
func (r *endReads) holder(s string) (string, error) {
got, ok := r.holders[s]
if !ok {
got.holder, got.err = r.w.Holder(s)
r.holders[s] = got
}
return got.holder, got.err
}
func (r *endReads) grantsFor(branch string, source int64) (meshgrants.GrantsFor, error) {
k := fmt.Sprintf("%s.%d", branch, source)
got, ok := r.grants[k]
if !ok {
got.g, got.err = r.w.GrantsFor(branch, source)
r.grants[k] = got
}
return got.g, got.err
}
func (r *endReads) widens(p seat.Pipeline, module string) []seat.Widening {
could := func(name, said string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatCouldNotRun, Name: name, Said: said}}
}
cand, err := w.Revision(p.Branch, p.Revision)
cand, err := r.revision(p.Branch, p.Revision)
if err != nil {
return could("mesh-catalog", fmt.Sprintf("the candidate's revision %d is not read: %v", p.Revision, err))
}
newest, err := w.Revision(p.Branch, 0)
newest, err := r.revision(p.Branch, 0)
if err != nil {
return could("mesh-catalog", fmt.Sprintf("the branch's newest revision is not read: %v", err))
}
var out []seat.Widening
for _, s := range servingSeats(p) {
holder, err := w.Holder(s)
holder, err := r.holder(s)
if err != nil {
out = append(out, seat.Widening{What: seat.WideningWhatCouldNotRun, Name: s, Said: fmt.Sprintf("the holder of %s is not read: %v", s, err)})
continue
@@ -57,8 +124,8 @@ func (w *Widener) Widens(p seat.Pipeline, module string) []seat.Widening {
}
out = append(out, ruleWidens(module, newest.Entries[module], cand.Entries[module])...)
out = append(out, credentialWidens(module, newest.Entries[module], cand.Entries[module])...)
if w.GrantsFor != nil {
g, err := w.GrantsFor(p.Branch, p.Source)
if r.w.GrantsFor != nil {
g, err := r.grantsFor(p.Branch, p.Source)
switch {
case err != nil:
out = append(out, seat.Widening{What: seat.WideningWhatCouldNotRun, Name: "mesh-grants", Said: err.Error()})
@@ -88,17 +88,19 @@ func TestWidensIsTheCandidateAgainstTheNewest(t *testing.T) {
// A pipeline's end keeps each mover's widens in its value, whether or not anything widens (design 47 G §71, A-207; H H-A-51).
func TestAnEndKeepsEachMoversWidens(t *testing.T) {
h := &Hearer{Widens: func(p seat.Pipeline, module string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatGrant, Name: module + " s v"}}
h := &Hearer{Widens: func() func(seat.Pipeline, string) []seat.Widening {
return func(p seat.Pipeline, module string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatGrant, Name: module + " s v"}}
}
}}
p := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Movers: []seat.Mover{{Module: "app"}, {Module: "lib"}},
Steps: []seat.StepResult{{Module: "app", Step: "build", Outcome: "passed"}, {Module: "lib", Step: "build", Outcome: "passed"}}}
if !h.endPipeline(&p, time.Unix(1, 0)) || len(p.Movers[0].Widens) != 1 || p.Movers[0].Widens[0].Name != "app s v" || p.Movers[1].Widens[0].Name != "lib s v" {
if !h.endPipeline(&p, time.Unix(1, 0), h.widensReads()) || len(p.Movers[0].Widens) != 1 || p.Movers[0].Widens[0].Name != "app s v" || p.Movers[1].Widens[0].Name != "lib s v" {
t.Errorf("the ended movers = %+v", p.Movers)
}
h.Widens = nil
q := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Movers: []seat.Mover{{Module: "app"}}, Steps: []seat.StepResult{{Module: "app", Step: "build", Outcome: "passed"}}}
if !h.endPipeline(&q, time.Unix(1, 0)) || len(q.Movers[0].Widens) != 0 {
if !h.endPipeline(&q, time.Unix(1, 0), h.widensReads()) || len(q.Movers[0].Widens) != 0 {
t.Errorf("a runner with no widener kept %v", q.Movers[0].Widens)
}
}
@@ -84,7 +84,7 @@ func TestManifest(t *testing.T) {
"mesh-delivery-indexer.take-in.taken", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-builds.step-ended",
"operator-channel.decided", "node-build-go.step-ended", "node-build-node.step-ended",
"node-build-image.step-ended", "node-build-go.ask-taken", "node-build-node.ask-taken", "node-build-image.ask-taken",
"node-build-go.step-progress", "node-build-node.step-progress", "node-build-image.step-progress", "mesh-wants.step-ended", "mesh-catalog.registration-changed",
"node-build-go.step-progress", "node-build-node.step-progress", "node-build-image.step-progress", "mesh-wants.step-ended", "mesh-wants.step-progress", "node-engine.running", "mesh-catalog.registration-changed",
"mesh-catalog.catalog-revision", "mesh-catalog.version-released", "mesh-conditions.findings-asked", "mesh-delivery.recovery-merged",
// A-186: the seat list its write verbs' callers are checked against, heard as it changes.
"mesh-catalog.seat-definition-changed", "mesh-catalog.seat-definition-removed",
@@ -199,7 +199,7 @@ func runSettingsOf(raw []byte) bool {
if json.Unmarshal(raw, &m) != nil {
return false
}
want := map[string]string{"test-runs-cap-mib": "number", "test-runs-kept": "string", "test-step-bound": "string", "run-untaken-bound": "string", "widens-read-bound": "string", "revisions-kept": "number",
want := map[string]string{"test-runs-cap-mib": "number", "test-runs-kept": "string", "test-step-bound": "string", "run-untaken-bound": "string", "ask-unanswered-bound": "string", "ask-silent-bound": "string", "widens-read-bound": "string", "revisions-kept": "number",
"pipelines-at-once": "number", "pipelines-waiting-cap": "number"}
if len(m.Config) != len(want) || m.Config["test-runs-cap-mib"]["maximum"] != float64(64) || m.Config["test-runs-cap-mib"]["default"] != float64(16) ||
m.Config["pipelines-at-once"]["minimum"] != float64(1) || m.Config["pipelines-at-once"]["maximum"] != float64(8) ||
@@ -58,6 +58,8 @@
"node-build-node.step-progress",
"node-build-image.step-progress",
"mesh-wants.step-ended",
"mesh-wants.step-progress",
"node-engine.running",
"mesh-catalog.registration-changed",
"mesh-catalog.catalog-revision",
"mesh-catalog.version-released",
@@ -151,6 +153,18 @@
"description": "how long a test run's ask may wait untaken after the run's start before the run ends could not run, a Go duration such as 24h (design 47 G section 725, A-466s, A-469x).",
"trusted": false,
"default": "24h"
},
"ask-unanswered-bound": {
"type": "string",
"description": "how long an ask the runner puts on a seat, but a test run's, may go without its holder's first word (an ask-taken, a step progress or the step's end) before it is asked once more, and as long again before its step ends could not run, a Go duration such as 10m (design 47 H H-A-235, H-A-238).",
"trusted": false,
"default": "10m"
},
"ask-silent-bound": {
"type": "string",
"description": "how long an ask the runner puts on a seat whose holder says nothing before the step's end (mesh-builds, mesh-wants) may go without that end before its step ends could not run, a Go duration such as 45m (design 47 H H-A-243).",
"trusted": false,
"default": "45m"
}
},
"state": [
@@ -278,6 +292,22 @@
"mode": "0600",
"content": "${config:run-untaken-bound}"
},
{
"id": "ask-unanswered-bound",
"type": "file",
"path": "${dir:state}/ask-unanswered-bound",
"owner": "mesh-delivery-pipeline",
"mode": "0600",
"content": "${config:ask-unanswered-bound}"
},
{
"id": "ask-silent-bound",
"type": "file",
"path": "${dir:state}/ask-silent-bound",
"owner": "mesh-delivery-pipeline",
"mode": "0600",
"content": "${config:ask-silent-bound}"
},
{
"id": "widens-read-bound",
"type": "file",
@@ -326,6 +356,8 @@
"MESH_RUNNER_WIDENS_READ_BOUND_FILE": "${dir:state}/widens-read-bound",
"MESH_RUNNER_REVISIONS_KEPT_FILE": "${dir:state}/revisions-kept",
"MESH_RUNNER_RUN_UNTAKEN_BOUND_FILE": "${dir:state}/run-untaken-bound",
"MESH_RUNNER_ASK_UNANSWERED_BOUND_FILE": "${dir:state}/ask-unanswered-bound",
"MESH_RUNNER_ASK_SILENT_BOUND_FILE": "${dir:state}/ask-silent-bound",
"MESH_RUNNER_PIPELINES_AT_ONCE_FILE": "${dir:state}/pipelines-at-once",
"MESH_RUNNER_PIPELINES_WAITING_CAP_FILE": "${dir:state}/pipelines-waiting-cap"
},