Compare commits

..
28 Commits
Author SHA1 Message Date
mesh-tools 7e597a623e Merge pull request 'The runner and the builder write a journal line per step end, ask placed and ask answered (H-A-320, H-A-333)' (#320) from fix/47-H2-journal-lines into main 2026-11-21 21:01:55 +00:00
jschoubben 6a847a8d9c H-A-333: the ended line at every said end, the answered line at every answer
The runner's ended line was written only for a heard step-ended and its run's
readers; the unanswered-bound end, the end at placing, a supersede, a re-pin,
the approval's end, the start pass and the verbs said their ends silently. It
is now written in Hearer.say, the one place that says step-ended, and at the
starter's two sayers that call stepEvent directly (repin, sayNotAsked). The
builder's answered line sat on four of buildEntry's returns; the ask's answer
is now one line in BuildStep's finish, naming the pipeline, the movers, the
step and the outcome, the per-entry existing/built-now line kept beside it.

Builds: design 47 H H-A-320 (1), H-A-333
2026-11-21 21:58:36 +01:00
jschoubben dc83ef3e27 H-A-320 (1): a journal line per step end, step ask and build answer
A working 36-mover pipeline was read as stalled twice: the runner wrote no
line when a step ended or was asked, and the builder none when it answered a
step ask. The runner now writes "<pipeline> <mover> <step> ended <outcome>
(<node>, <duration>)" at each step end it writes (the asking step, every
reader of its test run, and each reader EndRun ends), and "<pipeline> <mover>
<step> asked on <seat>" for each ask put (the re-ask paths keep their own
line). The builder writes "<module>@<fingerprint> build step answered
<outcome> for <pipeline> (existing build / built now)".

Builds: design 47 H H-A-320 (1)
2026-11-21 21:50:48 +01:00
mesh-tools 600c13b9e9 Merge pull request 'Design 47 H: the runner redoes a cut placing, places on a late end, names a stalled pipeline, and reads a prior publish from the build seat (H-A-311, H-A-313, H-A-326, H-A-328, H-A-331)' (#319) from fix/47-H1-runner-placing-redone into main 2026-11-21 20:48:59 +00:00
mesh-tools 5396dc9b11 Merge pull request 'The teller's want-of-room test bed answers the seat-holder ask; one taker of registration-changed (H-A-295)' (#318) from fix/47-H1-teller-test-want-of-room into main 2026-11-21 20:48:56 +00:00
mesh-tools 425dbaef99 Merge pull request 'Design 47 H: a passed pipeline whose movers' versions arrive late still writes its wants (H-A-270, H-A-273, H-A-277)' (#317) from fix/47-H1-end-taken-when-versioned into main 2026-11-21 20:48:49 +00:00
jschoubben 714c2c1ccc H-A-331: the published answer carries the publish time and digest where kept; a withdrawn number is never published
The verb published answered no time, and after a release no digest,
while the seat form says both come when published. The builder now
keeps when it heard or confirmed each publish on the mark, and each
published digest on a kept release; marks and releases written before
answer without them. A number a take-over withdrew is not published.
The runner drops a pipeline's kept answers at its end.

Builds: design 47 H H-A-331, H-A-328
2026-11-21 21:38:50 +01:00
jschoubben e583dce741 H-A-326 (1), (3): a mover's wait on a package is a read first, an event second
main.137 stood twenty minutes with nats and mesh-vault waiting on
mesh-grants packages published at 20:00, before the wait was written,
so no publish event came. The runner now asks the build seat's
published when it writes the wait and places the mover when the
package is published, saying so in the journal apart from an event.

Builds: design 47 H H-A-326 (1), H-A-326 (3), H-A-328
2026-11-21 21:25:03 +01:00
jschoubben 5e76fdf725 H-A-328: the seat mesh-builds answers published; the builder serves it
A package published before a pipeline existed is in no record of the
runner, so the builder, which owns publishes, answers whether an
artifact built at a fingerprint is published, from its reservations
(marked from the registries' record) and its releases.

Builds: design 47 H H-A-328, H-A-326 (1), H-A-274
2026-11-21 21:25:03 +01:00
jschoubben 438c8740a6 H-A-326 (2): a pipeline waiting only on packages is named stalled
A step held on a package's publish counted as open, so a pipeline whose
only movers waited on publishes that had come before their wait was
never named by runner_health (main.137, twenty minutes).

Builds: design 47 H H-A-326 (2), H-A-311 (3)
2026-11-21 21:18:32 +01:00
jschoubben 05d9f90bdb H-A-311 (3) review: the health reads records keeping revisions; a required step waiting is open
runner_health, show, list and stands refused every pipeline record
carrying the runner's own kept-revisions ("unknown field
kept-revisions" on main.131 and main.137): the field is now dropped
before the seat's strict decode. A required step waiting (an approval,
a pin-held step) now counts as open, as the end's judgement holds it.

Builds: design 47 H H-A-311 (3), H-A-313, H-A-285, H-A-309, H-A-322
2026-11-21 20:53:04 +01:00
jschoubben f22945c26a H-A-311 (3): runner_health names a running pipeline with nothing open and nothing placed for five minutes
main.137 stood seventeen minutes with nothing open while runner_health
said healthy. The tools program says each such pipeline in its journal
once per key (H-A-313).

Builds: design 47 H H-A-311 (3), H-A-313
2026-11-21 20:44:43 +01:00
jschoubben e137df7189 H-A-311 (2): a step-progress or step-end heard late still places what its end unlocks
A late progress or end moved nothing, so movers waiting on what the
step's end gave stayed unplaced: main.137 placed nothing after its
lint's late progress at 20:17:55.

Builds: design 47 H H-A-311 (2), H-A-286, H-A-288
2026-11-21 20:43:02 +01:00
jschoubben 6c66bf9a5a H-A-311 (1): a placing cut by a concurrent write is done again from the record read again
A placing whose compare-and-set was refused because the pipeline was
written meanwhile was dropped and never done again: main.137 stood
seventeen minutes with eleven steps placed until a restart.

Builds: design 47 H H-A-311 (1), H-A-288
2026-11-21 20:41:29 +01:00
jschoubben b79fd19c84 Teller: hear registration-changed on the views reader's one subscription
Since mesh-sdk heard a seat's event on its bare subject too (H-A-166), the teller's
own hub handler of registration-changed overlapped the one the views reader makes
for the catalog half's followers: the hub refused the reader's, said "a message has
one taker", and registered none of the reader's adopted events after it, so the
catalog half's registrations and source branches were their first views' alone.
The widenings now hear the event through ProcessReader.AdoptEvent beside the
followers; a failed handling is given back for the bus to ask again, as before.
The carry test now fails when any of the reader's events is refused.

issue 1960 (H1), test stability.

Builds: design 47 G A-199, A-222, §532, §335
2026-11-21 19:24:22 +01:00
jschoubben 70bb8f7f51 Teller test bed: answer who holds operator-channel, so a catch-up does not hold the asks' watch
The want-of-room test failed most runs since 20 Nov: the teller's picture of the
assigner's asks hands the first ask on twice (its first view, then again), and the
second sight of an open asking catches up with the router's record, which first asks
the seat mesh-module-assigner's assignments who holds operator-channel. The bed
neither granted nor answered that call, so it waited out its call timeout on the
watch's goroutine, and the second ask, written meanwhile, was not raised within the
test's 10 s. The bed now grants and answers the call as the live seat does, naming
ops-router, so the catch-up reads ops-router_asks at once (20 of 20 runs pass, were
0 of 20 locally).

issue 1960 (H1), test stability.

Builds: design 47 G §570 (issue 1610 AC1, the refused ask asked again), §717
2026-11-21 19:21:24 +01:00
mesh-tools e196064e51 Merge pull request 'Design 47 H: the builder's publish queue: live asks first, newest per series, one listing per pass (H-A-290..293)' (#316) from fix/47-H1-builder-publish-queue into main 2026-11-21 18:00:25 +00:00
jschoubben 2eaa41a420 H-A-292, H-A-293: builder_publish_queue reads one page; a build is retired only within its series
The review found two faults. The tool paged on through a state of any size, one replay of the bucket per page; it now
reads one page of each of its own states and, when the state holds more, answers that page capped, with the keys read
and the state's size from its info, never asking a second page and never refusing. And the retiring of superseded done
builds keyed the newest reservation and release by module alone, so a standing 0.2 would retire a done 0.1.x; both are
now found per module and series (MAJOR.MINOR).

Builds: design 47 H H-A-292, H-A-293, H-A-291 (1), H-A-290 (5)
2026-11-21 18:59:58 +01:00
jschoubben 90ff867e7a lock the test fakes' maps
Two tests failed under -race on main too: the re-ask state fake's maps were written by a try on the watch's goroutine while
the test read and deleted them, and the operator-ask test read the build store unlocked and waited on the builder while
the build its answer starts could still be counting itself in. The fake's maps are locked and read through it, the store
is read under its lock, and the builder is waited on only once that build has claimed its key.

Builds: design 47 H H-A-290 (the module's tests pass with -race, as its checks ask)
2026-11-21 18:59:58 +01:00
jschoubben 77fb6c4ab1 H-A-291: a start publishes only each module's newest done build; a live enqueue cuts a start list short
A start asked a publish again for every unreleased done build, older ones whose digests a newer reservation or release
of the same module made dead included, and a live publish queued during a long start pass waited for its end. A done
build a newer reservation or release (the builder's own record) supersedes is now retired at the start and at the pass:
its key dropped, its publish item ended, said in the journal; the start asks only each module's newest. A live enqueue
marks its holder, and a pass about to send a start-class item stops there, lets the sends in flight end and lists again,
so the live one goes next; a start-class enqueue never does.

Builds: design 47 H H-A-291 (1), (2), H-A-290, H-A-275
2026-11-21 18:59:58 +01:00
jschoubben bda61ae64c H-A-290: the builder's publish queue sends live publishes first, in passes, never a withdrawn build
The tool runner's release waited forty minutes behind old versions: every start asked a publish again for every done
build, and each send relisted and reread the whole queue. A publish whose build ended in this run now goes before any
asked again at a start (the class kept on the key's value, a live key of a build before the start made a start key at
the start); a pass lists a holder's keys once and sends them four at a time; a withdrawn or taken-over build is dropped
with its publish item ended at enqueue, at the pass and at the start; a send that returns unsent says why, and every call
it makes honours its one bound; builder_publish_queue reads its states in pages and answers counts per holder and class,
the oldest and newest queue time, a list capped at fifty and one module's items with their class.

Builds: design 47 H H-A-290 (1)-(5), H-A-269, H-A-271, H-A-275
2026-11-21 18:59:58 +01:00
mesh-tools 5547fd028e Merge pull request 'Design 47 H: a pipeline with nothing open ends at once; the end's reads are made once (H-A-288, H-A-289)' (#315) from fix/47-H1-runner-end-when-empty into main 2026-11-21 17:22:44 +00:00
jschoubben d51d389c1b Runner: a pipeline with nothing open ends at once (47-H1-runner-end-when-empty)
main.117 judged its end at 17:45:19 and never wrote it: every mover's widens
was read inside the end's compare-and-set loop, 126 movers each cut on
grants-for, so each try took about eleven minutes and every write was refused
as changed until the restart, whose start pass judged no end.

The reads an end needs are made once, before the compare-and-set, shared by
every mover and every retry; the runner's start pass ends each running
pipeline with nothing open before the queue is placed; every end says its
outcome in the journal.

Builds: design 47 H H-A-288, H-A-289, H-A-223, H-A-243
2026-11-21 18:18:32 +01:00
jschoubben 070f60404e Assigner: a passed mover with no version yet holds its tier
The tier progress started at the lowest tier of the versioned movers, so every
lower tier counted as run and a mover whose version came late was wanted after
its consumer. The progress now holds every passed mover, and a tier with one
whose version is not written does not run until it is.

Builds: design 47 H H-A-270, H-A-273, H-A-277 (1)
2026-11-21 15:55:30 +01:00
jschoubben 7f855f0e52 Runner: say the end again only for a passed pipeline, and again on a version heard again
A say that failed after the version was written was lost: the redelivery found
the version held and said nothing. A failed, superseded or cancelled end writes
no want, so it is not said again.

Builds: design 47 H H-A-270 (2), H-A-277 (2), (3)
2026-11-21 15:52:18 +01:00
jschoubben 3dfc18023a Assigner: a mover whose version came late joins the tier progress at its tier
The tier progress was written once, from the movers the first end named, so
a mover taken from the end said again was wanted out of tier order: a late
tier-2 consumer and the tier above it were wanted together. It now joins
the progress at its tier, and is wanted at once only when that tier ran.

Builds: design 47 H H-A-270, H-A-273
2026-11-21 15:44:58 +01:00
jschoubben 1489493431 Assigner: say a passed mover with no version yet; one registration-changed handler
The holder skipped such a mover silently, so a passed pipeline wrote no want
and nothing said why; the runner now says the end again once the version is
written, and the holder takes it then. The pipeline ends took registrations
from a second handler that the hub refused at every start (a message has one
taker); they are told by the catalog half's one follower instead.

Builds: design 47 H H-A-270 (1), (3)
2026-11-21 15:41:03 +01:00
jschoubben e927bb8971 Runner: a version written after a pipeline's end says pipeline-ended again
The seat mesh-wants' holder skips a passed mover with no version yet; the
version-released arrives late, so the end it took named none and no want was
written. Saying the same pipeline-ended again, naming the version, lets the
holder take the mover it skipped.

Builds: design 47 H H-A-270 (2)
2026-11-21 15:34:34 +01:00
68 changed files with 2577 additions and 295 deletions
@@ -37,6 +37,10 @@ type halves struct {
// settled is told once when the first watched branch's first view is answered, for the first pass to wait on.
settled chan<- struct{}
told bool
// regMu orders the registrations told to registered: the ones held when it is set, then each change (H-A-270 (3)).
regMu sync.Mutex
registered func(reg meshcatalog.Registration, gone bool)
}
// follow makes the branch follower on the reader; the reader's start asks the first views.
@@ -50,6 +54,15 @@ func (h *halves) follow() error {
h.changed(branch, module, entry, deleted)
}
b.Adopted = h.adopted
// The one handler of registration-changed is the follower's (H-A-270 (3)): the pipeline ends are told each change from it,
// never from a second handler on the hub.
reconcile := b.Registrations.Follower.Changed
b.Registrations.Follower.Changed = func(module string, value json.RawMessage, gone bool, at time.Time) {
if reconcile != nil {
reconcile(module, value, gone, at)
}
h.tellRegistration(module, value, gone)
}
h.mu.Lock()
h.branches = b
h.mu.Unlock()
@@ -215,3 +228,39 @@ func (h *halves) watchedOf(module string) []string {
slices.SortStableFunc(out, func(x, y string) int { return strings.Compare(keys.EscapeName(x), keys.EscapeName(y)) })
return out
}
// tellRegistration tells registered of one registration the follower holds now, or of one removed.
func (h *halves) tellRegistration(module string, value json.RawMessage, gone bool) {
h.regMu.Lock()
defer h.regMu.Unlock()
if h.registered == nil {
return
}
reg := meshcatalog.Registration{Module: module}
if !gone {
if err := json.Unmarshal(value, &reg); err != nil {
h.logf("the registration of %s does not read: %v", module, err)
return
}
}
h.registered(reg, gone)
}
// followRegistrations tells registered of every registration the branch follower holds and then of each change it takes
// (design 47 G §353): the registrations are followed once, by the follower's one handler of registration-changed (H-A-270
// (3)). It says false when no follower is made.
func (h *halves) followRegistrations(registered func(meshcatalog.Registration, bool)) bool {
h.mu.Lock()
b := h.branches
h.mu.Unlock()
if b == nil || b.Registrations == nil {
return false
}
h.regMu.Lock()
defer h.regMu.Unlock()
h.registered = registered
for _, reg := range b.Registrations.All() {
registered(reg, false)
}
return true
}
@@ -53,6 +53,7 @@ func TestAMeshThatNamesNoBranchSettlesTheAssignersFirstPass(t *testing.T) {
// Design 47 H-A-143, H-A-144, G §532: a module's watched branches are its registration's, in its order; once registrations are held a
// module none names has none; before the first registrations the gate's source's branches stand in, in the source's order.
// H-A-270 (3): the pipeline ends are told the registrations by the same follower, whether set before its first view or after.
func TestAModulesWatchedBranchesAreItsRegistrations(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
@@ -81,11 +82,23 @@ func TestAModulesWatchedBranchesAreItsRegistrations(t *testing.T) {
if err := h.follow(); err != nil {
t.Fatal(err)
}
var live, held []string
if !h.followRegistrations(func(reg seat.Registration, gone bool) { live = append(live, reg.Module) }) {
t.Fatal("no follower of the registrations")
}
for _, f := range reader.Followers()[:2] { // the registrations and the gate's source
if err := f.Start(ctx); err != nil {
t.Fatal(err)
}
}
h.followRegistrations(func(reg seat.Registration, gone bool) { held = append(held, reg.Module) })
var regs []string
for _, r := range tc.regs {
regs = append(regs, r.Module)
}
if !slices.Equal(live, regs) || !slices.Equal(held, regs) {
t.Errorf("the ends were told %v at the first view and %v held, want %v", live, held, regs)
}
if got := h.watchedOf(tc.module); !slices.Equal(got, tc.want) {
t.Errorf("the watched branches of %s are %v, want %v", tc.module, got, tc.want)
}
@@ -880,6 +880,14 @@ func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
if _, err := b.buckets[engineReadsBucket()].Get(b.ctx, "shanks"); err == nil {
t.Error("the state engine-reads holds a key of shanks, whose engine said nothing")
}
// H-A-270 (3): registration-changed has one handler, the catalog half's follower, so the start's hub refuses no overlap.
b.journal.await(t, 0, "the kept seat events heard", "the hub started", time.Second)
b.journal.mu.Lock()
overlap := strings.Contains(b.journal.text.String(), "a message has one taker")
b.journal.mu.Unlock()
if overlap {
t.Error("the start's hub refused a handler that overlaps another: a message has one taker")
}
}
// subscribed returns once the server holds every subscription nc made: nats.go sends a SUB from its flusher, after
@@ -649,12 +649,12 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
p.ends = newPipelineEnds(kvEnds{ctx, kv}, wake, logger.Printf)
p.ends.trunk = w.trunkName
p.ends.runEnded = func(group string) error { return p.dropCandidate(ctx, group) }
err := hearEnds(hub, p.ends)
if err == nil && bus != nil {
err = followRegistrations(ctx, hub, bus.Call, p.ends)
if err := hearEnds(hub, p.ends); err != nil {
logger.Printf("a pipeline's end is not heard, so the wants keep the trunk-only form until the assigner starts again: %v", err)
}
if err != nil {
logger.Printf("a pipeline's end or a registration is not heard, so the wants keep the trunk-only form until the assigner starts again: %v", err)
// The registrations come from the catalog half's one follower, its one handler of registration-changed (H-A-270 (3)).
if iv == nil || iv.h == nil || !iv.h.followRegistrations(p.ends.registered) {
logger.Printf("the registrations are not followed, so the wants keep the trunk-only form until the assigner starts again: the catalog half is not followed")
}
}
hearReleases(ctx, js, w, wake, findings, logger.Printf) // version-released, machine-away and machine-back
@@ -14,7 +14,6 @@ import (
"github.com/nats-io/nats.go/jetstream"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
mprc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshpipelinerunnerclient"
mpcat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
mpr "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -161,22 +160,33 @@ func newPipelineEnds(st endStore, wake func(), logf func(string, ...any)) *pipel
// take keeps, for each mover of a passed pipeline that released a version, that version as the newest of its module on the
// pipeline's branch, by compare-and-set on a newer version (design 47 H §750): an older version, or the same one again, writes
// nothing. It says whether any key changed.
// nothing. A passed mover with no version yet is said in the journal and taken when the runner says the end again naming it
// (H-A-270); it holds its tier in the pipeline's tier progress meanwhile (H-A-277 (1)). It says whether any key changed.
func (e *pipelineEnds) take(p mpr.PipelineEnded) (bool, error) {
if p.Outcome != mpr.PipelineEndedOutcomePassed || p.Run != "" {
return false, nil
}
changed := false
var kept []tierMover
versioned := map[string]bool{}
defer func() {
if err := e.startTiers(p.Pipeline, p.Source, kept); err != nil {
if err := e.startTiers(p.Pipeline, p.Source, kept, versioned); err != nil {
e.logf("the tier progress of %s is not written: %v", p.Pipeline, err)
}
}()
for _, m := range p.Movers {
if m.Version == "" || m.Fingerprint == "" || !moverPassed(m) {
if m.Fingerprint == "" || !moverPassed(m) {
continue
}
if m.Version == "" {
// The runner writes a mover's version at its release's version-released, and says this end again once it does
// (H-A-270 (1), (2)): the skip is said, never silent.
e.logf("%s: %s passed with no version yet: taken again when its version is written", p.Pipeline, m.Module)
// It holds its tier in the progress until its version is written (H-A-277 (1)).
kept = append(kept, tierMover{Module: m.Module, Fp: m.Fingerprint, Tier: max(m.Tier, 1), NoVersion: true})
continue
}
versioned[m.Module] = true
key := endKey(m.Module, p.Branch)
for {
raw, rev, err := e.st.get(key)
@@ -372,32 +382,3 @@ func (e *pipelineEnds) registered(reg mpcat.Registration, gone bool) {
}
e.wake()
}
// followRegistrations follows the registrations through the seat mesh-catalog (design 47 G §353): a first view of its verb
// and each registration-changed on the hub. It returns when the hub's handler is registered; the first view is asked on its own
// goroutine.
func followRegistrations(ctx context.Context, hub *process.EventHub, call mcc.Call, e *pipelineEnds) error {
regs := mcc.NewRegistrations(call)
regs.Follower.Logf = e.logf
regs.Follower.Changed = func(module string, value json.RawMessage, gone bool, _ time.Time) {
reg := mpcat.Registration{Module: module}
if !gone {
_ = json.Unmarshal(value, &reg)
}
e.registered(reg, gone)
}
if err := hub.Handle(mpcat.SubjectRegistrationChanged, func(m jetstream.Msg) {
if err := regs.Hear(ctx, m.Data()); err != nil {
e.logf("%s: %v", m.Subject(), err)
}
_ = m.Ack()
}); err != nil {
return err
}
go func() {
if err := regs.Start(ctx); err != nil && ctx.Err() == nil {
e.logf("the first view of the registrations is not answered: %v", err)
}
}()
return nil
}
@@ -141,7 +141,8 @@ func TestThePassedVersionIsKept(t *testing.T) {
// A6/AC1, A5/AC1 (design 47 H §750): a pass writes, on every node the module is assigned to, the want naming the newest version
// released on its branch whose pipeline passed, with the pipeline and the rank; with none passed it writes none. No grant,
// canary, provider or composition is asked for.
// canary, provider or composition is asked for. H-A-270 (1): a passed end whose mover names no version yet writes no want and
// says so in the journal; the same end said again naming the version writes it.
func TestAPassWritesTheWantOfThePassedVersion(t *testing.T) {
fp := meshcatalogclient.SourcePrefix2 + strings.Repeat("a", 64)
for _, passed := range []bool{false, true} {
@@ -154,9 +155,15 @@ func TestAPassWritesTheWantOfThePassedVersion(t *testing.T) {
p.w.manifests[mdigest] = parsed{module: "web", manifest: catalogue.Manifest{Module: "web", Requires: []string{"database"}}}
placeOn(t, p.w, "g14", "web")
placeOn(t, p.w, "ace", "web")
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
var journal []string
p.ends = newPipelineEnds(newMemEnds(), func() {}, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
p.ends.trunk = p.w.trunkName
if passed {
_, _ = p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("web", fp, "", 2, mpr.StepResultOutcomePassed)))
p.wants(context.Background())
if len(wantsKV.values) != 0 || !slices.Contains(journal, trunkBranch+".5: web passed with no version yet: taken again when its version is written") {
t.Errorf("an end naming no version wrote %d wants and said %q", len(wantsKV.values), journal)
}
_, _ = p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("web", fp, "0.1.2", 2, mpr.StepResultOutcomePassed)))
}
p.wants(context.Background())
@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
@@ -40,6 +41,8 @@ type tierMover struct {
Module string `json:"module"`
Fp string `json:"fp"`
Tier int64 `json:"tier"`
// NoVersion is a passed mover whose version is not yet written: its tier does not run until it is (H-A-277 (1)).
NoVersion bool `json:"no-version,omitempty"`
}
// tierProgress is a pipeline's tier progress (H-A-56): the tier whose wants are written, since when, the movers, and the source
@@ -61,24 +64,53 @@ func isTierKey(key string) bool {
return strings.HasPrefix(key, tierPrefix) && strings.Count(key, ".") == 2
}
// startTiers writes the tier progress of a passed pipeline whose movers' versions are now kept, its first tier the lowest of
// theirs, since now. A pipeline heard again writes nothing.
func (e *pipelineEnds) startTiers(pipeline string, source int64, movers []tierMover) error {
if len(movers) == 0 {
return nil
}
first := movers[0].Tier
for _, m := range movers {
first = min(first, m.Tier)
}
value, err := json.Marshal(tierProgress{Tier: first, Since: e.clock().UTC(), Movers: movers, Source: source})
if err != nil {
// startTiers writes the tier progress of a passed pipeline, holding every passed mover, versioned or not, its first tier the
// lowest of theirs, since now (H-A-277 (1)). A pipeline heard again adds to the progress that stands each mover it does not hold,
// at the mover's tier, and marks versioned each held mover whose version is now written (H-A-270, H-A-273): its want is written
// when the progress reaches its tier, or at once when that tier ran already, never out of tier order.
func (e *pipelineEnds) startTiers(pipeline string, source int64, movers []tierMover, versioned map[string]bool) error {
for {
t, rev, ok := e.tiersOf(pipeline)
if !ok {
if len(movers) == 0 {
return nil
}
first := movers[0].Tier
for _, m := range movers {
first = min(first, m.Tier)
}
t = tierProgress{Tier: first, Since: e.clock().UTC(), Source: source}
}
moved := false
for i := range t.Movers {
if t.Movers[i].NoVersion && versioned[t.Movers[i].Module] {
t.Movers[i].NoVersion, moved = false, true
}
}
for _, m := range movers {
if !slices.ContainsFunc(t.Movers, func(h tierMover) bool { return h.Module == m.Module }) {
t.Movers, moved = append(t.Movers, m), true
}
}
if !moved {
return nil
}
value, err := json.Marshal(t)
if err != nil {
return err
}
err = e.st.swap(tierKey(pipeline), value, rev)
if errors.Is(err, errSwapped) {
if _, _, held := e.tiersOf(pipeline); !ok && !held {
return nil // a key that stands but does not read is left as it is
}
continue
}
if err == nil && ok {
e.logf("pipeline %s: a mover whose version came after its end is taken into the tier progress", pipeline)
}
return err
}
if err := e.st.swap(tierKey(pipeline), value, 0); err != nil && !errors.Is(err, errSwapped) {
return err
}
return nil
}
func (e *pipelineEnds) clock() time.Time {
@@ -196,6 +228,18 @@ func (p *passer) notRunning(n assign.Nodes, placedOn func(module, branch string)
if m.Tier != tier {
continue
}
if m.NoVersion {
// A mover whose version is not yet written holds its tier, under the tier bound (H-A-277 (1)).
machines := []string{}
for _, node := range placedOn(m.Module, branch) {
if online, known := n.Online[node]; !known || online {
machines = append(machines, node)
}
}
sort.Strings(machines)
out = append(out, meshwants.NotRunning{Module: m.Module, Machines: machines})
continue
}
if end, ok := p.ends.endOf(m.Module, branch); !ok || end.Pipeline != pipeline {
continue // a later pipeline moved it
}
@@ -125,3 +125,86 @@ func TestAPipelinesWantsAreWrittenTierByTier(t *testing.T) {
}
}
}
// H-A-270, H-A-273, H-A-277 (1): a pipeline's tier progress holds every passed mover, versioned or not, and a tier with a mover
// whose version is not yet written does not run until it is: no later tier is wanted out of order. When the lowest tier's version
// comes last, nothing is wanted before it; with db (1) and api (3) versioned and web (2) late, api is wanted only once web runs.
func TestALateMoverHoldsItsTier(t *testing.T) {
type step struct {
versions map[string]string // a pipeline-ended taken, each mover's version ("": none yet)
runs string // or: this module runs on g14
wanted string // then: the modules wanted, in order of the list
}
for _, c := range []struct {
name string
steps []step
}{
{"the lowest tier's version comes last", []step{
{versions: map[string]string{"db": "", "web": "0.1.2", "api": "0.3.0"}, wanted: ""},
{versions: map[string]string{"db": "0.2.0", "web": "0.1.2", "api": "0.3.0"}, wanted: "db"},
{runs: "db", wanted: "db,web"},
{runs: "web", wanted: "db,web,api"},
}},
{"a middle tier's version comes late", []step{
{versions: map[string]string{"db": "0.2.0", "web": "", "api": "0.3.0"}, wanted: "db"},
{runs: "db", wanted: "db"},
{versions: map[string]string{"db": "0.2.0", "web": "0.1.2", "api": "0.3.0"}, wanted: "db,web"},
{runs: "web", wanted: "db,web,api"},
}},
} {
t.Run(c.name, func(t *testing.T) {
fps := map[string]string{}
tiers := map[string]int64{"db": 1, "web": 2, "api": 3}
for i, m := range []string{"db", "web", "api"} {
fps[m] = meshcatalogclient.SourcePrefix2 + strings.Repeat(string(rune('a'+i)), 64)
}
start := time.Date(2026, 11, 21, 4, 0, 0, 0, time.UTC)
now := start
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(kvSet{buckets.Wants(): wantsKV, buckets.Asks(): &memKV{values: map[string][]byte{}}}, start)
p.now = func() time.Time { return now }
p.say = func(oc.Condition) error { return nil }
p.publishWants = func(string, any, string) error { return nil }
for i, m := range []string{"db", "web", "api"} {
digest := strings.Repeat(string(rune('1'+i)), 64)
mod, _ := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Fp2: fps[m], Manifest: digest, Repository: "novox/" + m, Commit: strings.Repeat("1", 40), Path: meshcatalogclient.RootPath})
putModuleRaw(p.w, trunkBranch, m, mod)
p.w.manifests[digest] = parsed{module: m, manifest: catalogue.Manifest{Module: m}}
}
placeOn(t, p.w, "g14", "db", "web", "api")
p.w.mu.Lock()
p.w.runningHeard = true
p.w.mu.Unlock()
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
p.ends.trunk, p.ends.now = p.w.trunkName, func() time.Time { return now }
for i, s := range c.steps {
if s.versions != nil {
var movers []mpr.MoverEnded
for _, m := range []string{"db", "web", "api"} {
movers = append(movers, mover(m, fps[m], s.versions[m], tiers[m], mpr.StepResultOutcomePassed))
}
if _, err := p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, movers...)); err != nil {
t.Fatal(err)
}
} else {
raw, _ := json.Marshal(nodeengine.Running{Node: "g14", Module: s.runs, Fingerprint: fps[s.runs]})
if err := p.w.hearRunning("running.g14."+s.runs, raw); err != nil {
t.Fatal(err)
}
}
now = now.Add(time.Minute)
p.wants(context.Background())
var got []string
for _, m := range []string{"db", "web", "api"} {
key, _ := mwc.WantKey("g14", m)
if _, ok := wantsKV.values[key]; ok {
got = append(got, m)
}
}
if strings.Join(got, ",") != s.wanted {
t.Fatalf("step %d: wanted %v, want %q", i+1, got, s.wanted)
}
}
})
}
}
@@ -14,7 +14,8 @@ import (
"mesh-delivery-builder/internal/versions"
)
// readerOf is a kept state as the versions package's reader.
// readerOf is a kept state as the versions package's reader: its values read in one page, one request for many (design 47 H
// H-A-290 (5), H-A-292), where a key-by-key read of a queue of two thousand keys outlasts the tool's call.
func readerOf(s stdio.KeptState) versions.Reader {
return versions.Reader{Keys: s.Keys, Get: func(key string) (json.RawMessage, bool, error) {
e, err := s.Get(key)
@@ -22,7 +23,34 @@ func readerOf(s stdio.KeptState) versions.Reader {
return nil, false, err
}
return e.Value, true, nil
}}
}, All: func() (versions.StatePage, error) { return onePage(s.Page, s.Info) }}
}
// statePageBytes and statePageBudget bound the one page of a state read, so its answer fits one bus message and its time.
const (
statePageBytes = 768 * 1024
statePageBudget = 10 * time.Second
)
// onePage is the first page of a state's values, and never a second (H-A-292, H-A-293): when the state holds more, the page says
// so with the state's size from its info. A runtime that answers no page says errors.ErrUnsupported.
func onePage(page func(after string, maxBytes int, budget time.Duration) (stdio.StatePage, error), info func() (*stdio.StateInfo, error)) (versions.StatePage, error) {
pg, err := page("", statePageBytes, statePageBudget)
if err != nil {
return versions.StatePage{}, err
}
out := versions.StatePage{Values: make(map[string]json.RawMessage, len(pg.Entries)), More: pg.Next != ""}
for _, e := range pg.Entries {
out.Values[e.Key] = e.Value
}
if out.More {
if in, err := info(); err != nil || in == nil {
out.InfoWords = fmt.Sprintf("the state's info cannot be read: %v", err)
} else {
out.Keys, out.Bytes = in.Keys, in.Bytes
}
}
return out, nil
}
// republishTools are builder_publish_again and builder_publish_queue (design 47 G §588). The publish again is a hand act:
@@ -59,7 +87,11 @@ func republishTools(queue, acted, reasks versions.Reader, ask queueAsker, now fu
return map[string]any{"key": key, "queued": a.Queued, "answer": a.Answer}, nil
}},
{Name: versions.PublishQueueTool, Description: versions.PublishQueueDescription,
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return versions.AnswerQueue(queue, acted, reasks) }},
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "list only this module's queued versions, each with its class"},
"version": map[string]any{"type": "string", "description": "with module: only this version, MAJOR.MINOR.PATCH"}}},
Run: func(args map[string]any) (any, error) {
return versions.AnswerQueue(queue, acted, reasks, str(args, "module"), str(args, "version"))
}},
}
}
@@ -0,0 +1,44 @@
package main
import (
"encoding/json"
"errors"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// TestAStateIsReadInOnePage: design 47 H H-A-292, H-A-293 — builder_publish_queue reads each of its own states in one page and
// never asks a second: a state past the page answers that page with More and the state's size from its info; one the page holds
// answers whole without reading the info.
func TestAStateIsReadInOnePage(t *testing.T) {
for _, c := range []struct {
name string
next string
}{{"past one page", "k1"}, {"held by one page", ""}} {
t.Run(c.name, func(t *testing.T) {
var asked []string
infos := 0
page := func(after string, maxBytes int, budget time.Duration) (stdio.StatePage, error) {
asked = append(asked, after)
if after != "" {
return stdio.StatePage{Entries: []stdio.StateEntry{{Key: "k2", Value: json.RawMessage(`2`)}}}, nil
}
return stdio.StatePage{Entries: []stdio.StateEntry{{Key: "k1", Value: json.RawMessage(`1`)}}, Next: c.next}, nil
}
info := func() (*stdio.StateInfo, error) { infos++; return &stdio.StateInfo{Keys: 2, Bytes: 640}, nil }
pg, err := onePage(page, info)
more := c.next != ""
if err != nil || len(asked) != 1 || len(pg.Values) != 1 || pg.More != more || (more && (pg.Keys != 2 || pg.Bytes != 640 || infos != 1)) || (!more && infos != 0) {
t.Errorf("asked the pages after %q, read %v, more %v, size %d keys %d bytes, info read %d times (%v): want one page only, its rest said with the state's size", asked, pg.Values, pg.More, pg.Keys, pg.Bytes, infos, err)
}
})
}
unsupported := func(string, int, time.Duration) (stdio.StatePage, error) {
return stdio.StatePage{}, errors.ErrUnsupported
}
if _, err := onePage(unsupported, nil); !errors.Is(err, errors.ErrUnsupported) {
t.Errorf("a runtime with no pages answered %v, want ErrUnsupported so the reader reads key by key", err)
}
}
@@ -261,7 +261,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
republish := &versions.Republisher{Reserved: reserved.r, Queue: awaitedQueue{awaitedReserved{awaitedVersions{b: queueKV, bucket: own.queue}}},
Entry: entryVia(calls.Call), Send: sendVia(calls.Call),
Unreleased: reserved.unreleasedWords, Logf: logger.Printf,
Holders: holders} // a registry seat's holder from the seat-holders view, from the start (design 47 G A-416, A-416a)
Holders: holders, // a registry seat's holder from the seat-holders view, from the start (design 47 G A-416, A-416a)
Started: time.Now()} // a build ended from now on is of the live class, sent first (H-A-290 (1))
// A number a registry seat already holds is never reserved (H-A-275): read before reserving, through the seats' verb versions.
reserved.r.HeldNumbers = func(ctx context.Context, module, fp string, refs []string) ([]string, error) {
return republish.HeldNumbers(ctx, module, fp, refs, registryVersionsVia(calls.Call))
@@ -303,10 +304,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return republish.Held(ctx, v, registryVersionsVia(calls.Call))
},
markHeld: reserved.markHeld, moveElsewhere: reserved.movedElsewhere,
allDone: func(ctx context.Context) ([]versions.ReservedVersion, error) {
all, err := reserved.r.All(ctx)
return builtOf(all), err
}}
allDone: func(ctx context.Context) ([]versions.ReservedVersion, error) { return reserved.r.All(ctx) },
retired: reserved.r.Retired} // only each module's newest done build is asked at the start (H-A-291 (1))
// A version that never publishes (withdrawn or taken over) has its publish item ended by the queue too (H-A-290 (2)).
republish.Ended = items.Done
reasksKV.Ready(func(jetstream.KeyValue) { items.AskAllDone(ctx) })
wirePublishItems(reserved, items, awaitedReserved{awaitedVersions{b: actedKV, bucket: own.acted}}, releasedKept)
acts.Republish = republish.Republish
@@ -472,13 +473,15 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// modules again (design 47 G A-82, §204).
// The runner's build step ask, the builder's one entry point for a candidate (design 47 G A-97).
b.StepEntry = newEntryWaits(branches, half.manifests).entry
// The seat mesh-builds' verb reserved, answered here (A-115, A-127): the reservations, the catalog half's carriers and the seat
// mesh-catalog's versions.
// The seat mesh-builds' verbs reserved, answered here (A-115, A-127): the reservations, the catalog half's carriers and the seat
// mesh-catalog's versions; and published, from the reservations and releases (H-A-328).
stopReserved, err := calls.AnswerSeat(meshbuilds.Seat, seatlist.Definition(seatGate.Reader, meshbuilds.Seat), map[string]process.SeatHandler{
meshbuilds.VerbReserved: reservedVerb(ctx, reservedReader(ctx, reserved.r.Store), versions.CarriersFrom(versions.EntriesOf(branches)), catalogReleased(calls.Call),
func(name string) {
logger.Printf("reserved: no entry carries the name %s; the entries looked at: %s", name, versions.LookedAt(versions.EntriesOf(branches)))
})})
}),
// published (H-A-328): whether a package built at a fingerprint is published, asked by the runner writing a wait on it.
meshbuilds.VerbPublished: publishedVerb(ctx, reserved.r.PublishedOf)})
if err != nil {
return fmt.Errorf("answering the verb reserved: %w", err)
}
@@ -46,7 +46,11 @@ type publishItems struct {
reserved func(ctx context.Context, module, version string) (versions.ReservedVersion, bool, error)
republish func(ctx context.Context, module, version string) ([]string, error)
endedWith func(ctx context.Context, v versions.ReservedVersion, why string) // raises the finding version-unreleased
allDone func(ctx context.Context) ([]versions.ReservedVersion, error)
// allDone are the reservations, done or not: those whose build is done are asked at the start (builtOf).
allDone func(ctx context.Context) ([]versions.ReservedVersion, error)
// retired are the done builds a newer reservation or release of the same module supersedes, with why (H-A-291 (1)); nil
// retires none.
retired func(ctx context.Context, all []versions.ReservedVersion) (map[string]string, error)
// unheard is how many messages the builder's durable on the events stream holds pending and unacknowledged, read before each
// try (design 47 H H-A-269 (2), H-A-272); nil reads none.
unheard func(ctx context.Context) (int, error)
@@ -98,12 +102,28 @@ func (p *publishItems) Ask(ctx context.Context, module, version string) error {
// AskAllDone asks the publish item of every reservation whose build is done and that has no release: at the builder's start, so
// what stood before this change moves too, whatever its artifacts.
func (p *publishItems) AskAllDone(ctx context.Context) {
vs, err := p.allDone(ctx)
all, err := p.allDone(ctx)
if err != nil {
p.logf("the reservations of done builds are not asked to publish at the start: %v", err)
return
}
for _, v := range vs {
var retired map[string]string
if p.retired != nil {
if retired, err = p.retired(ctx, all); err != nil {
p.logf("the superseded builds are not retired at the start: %v", err)
}
}
for _, v := range builtOf(all) {
if why := retired[v.Module+" "+v.Version]; why != "" { // only each module's newest is asked (H-A-291 (1), H-A-275)
p.logf("%s %s is not asked to publish at the start: %s; its publish item is ended", v.Module, v.Version, why)
p.Done(ctx, v.Module, v.Version)
continue
}
if why := versions.NotStanding(v, true); why != "" { // withdrawn: it never publishes (H-A-290 (2))
p.logf("%s %s is not asked to publish at the start: %s; its publish item is ended", v.Module, v.Version, why)
p.Done(ctx, v.Module, v.Version)
continue
}
if err := p.Ask(ctx, v.Module, v.Version); err != nil {
p.logf("%v", err)
}
@@ -137,7 +157,8 @@ func (p *publishItems) Try(ctx context.Context, kv jetstream.KeyValue, e jetstre
p.logf("%s %s: its reservation cannot be read, the publish item waits: %v", r.Module, r.Version, err)
return
}
if !held { // dropped, taken over or released: never because its artifacts are published (A-456b)
if why := versions.NotStanding(v, held); why != "" { // dropped, taken over, released or withdrawn: never because its artifacts are published (A-456b, H-A-290 (2))
p.logf("%s %s: the publish item is ended: %s", r.Module, r.Version, why)
_ = kv.Delete(ctx, e.Key())
return
}
@@ -4,9 +4,12 @@ import (
"context"
"encoding/json"
"errors"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
@@ -19,9 +22,10 @@ import (
"mesh-delivery-builder/internal/versions"
)
// itemKV is the re-ask state in memory: one revision per key.
// itemKV is the re-ask state in memory: one revision per key, its maps locked (a try writes from the re-ask watch's goroutine).
type itemKV struct {
jetstream.KeyValue
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
rev uint64
@@ -42,13 +46,25 @@ func (e itemEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValueP
func (k *itemKV) entry(key string) jetstream.KeyValueEntry {
return itemEntry{key: key, value: k.vals[key], rev: k.revs[key]}
}
// has says whether key holds a value.
func (k *itemKV) has(key string) bool {
k.mu.Lock()
defer k.mu.Unlock()
_, ok := k.vals[key]
return ok
}
func (k *itemKV) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; !ok {
return nil, jetstream.ErrKeyNotFound
}
return k.entry(key), nil
}
func (k *itemKV) Create(_ context.Context, key string, v []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; ok {
return 0, jetstream.ErrKeyExists
}
@@ -57,6 +73,8 @@ func (k *itemKV) Create(_ context.Context, key string, v []byte, _ ...jetstream.
return k.rev, nil
}
func (k *itemKV) Update(_ context.Context, key string, v []byte, rev uint64) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if k.revs[key] != rev {
return 0, jetstream.ErrKeyExists
}
@@ -65,6 +83,8 @@ func (k *itemKV) Update(_ context.Context, key string, v []byte, rev uint64) (ui
return k.rev, nil
}
func (k *itemKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
k.mu.Lock()
defer k.mu.Unlock()
delete(k.vals, key)
return nil
}
@@ -140,21 +160,49 @@ func TestAPublishItemIsTriedWithAGrowingPauseAndEndsLoudly(t *testing.T) {
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
p.allDone = func(context.Context) ([]versions.ReservedVersion, error) {
published := versions.ReservedVersion{Module: "n", Version: "0.2.0", Artifacts: map[string]string{"n:a": "sha256:a"}, Published: map[string]bool{"n:a": true}, BuiltAt: now}
return builtOf([]versions.ReservedVersion{v, published, {Module: "o", Version: "0.1.0"}}), nil // the last build has not ended
withdrawn := versions.ReservedVersion{Module: "w", Version: "0.4.0", BuiltAt: now, Withdrawn: map[string]string{"0.4.0": "fp"}}
return builtOf([]versions.ReservedVersion{v, published, withdrawn, {Module: "o", Version: "0.1.0"}}), nil // the last build has not ended
}
_ = p.Ask(ctx, "w", "0.4.0") // asked before its reservation was withdrawn
p.AskAllDone(ctx)
if len(kv.vals) != 2 {
t.Errorf("the start asked %d items, want one per done unreleased reservation", len(kv.vals))
if _, ok := kv.vals[versions.PublishItemKey("w", "0.4.0")]; len(kv.vals) != 2 || ok {
t.Errorf("the start left %d items (withdrawn's kept: %v), want one per done unreleased reservation and none for a withdrawn one (H-A-290 (2))", len(kv.vals), ok)
}
// A start with three unreleased done versions of one module asks only the newest and retires the two (H-A-291 (1), H-A-275).
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
_ = p.Ask(ctx, "s", "0.1.3") // asked in an earlier run
p.allDone = func(context.Context) ([]versions.ReservedVersion, error) {
var out []versions.ReservedVersion
for _, ver := range []string{"0.1.3", "0.1.10", "0.1.2"} { // one series (H-A-293)
out = append(out, versions.ReservedVersion{Module: "s", Version: ver, Artifacts: map[string]string{"s:a": "sha256:a"}, BuiltAt: now})
}
return out, nil
}
p.retired = (&versions.Reserver{}).Retired
p.AskAllDone(ctx)
if _, ok := kv.vals[versions.PublishItemKey("s", "0.1.10")]; len(kv.vals) != 1 || !ok {
t.Errorf("the start left the items %v, want only the newest, s 0.1.10, asked and the two older retired", slices.Collect(maps.Keys(kv.vals)))
}
kv.vals, kv.revs = map[string][]byte{}, map[string]uint64{}
// A released version's item is deleted; a version no longer reserved has it deleted at its next try.
p.Done(ctx, "m", "0.1.2")
p.Done(ctx, "n", "0.2.0")
if len(kv.vals) != 0 {
t.Errorf("the release left %v", kv.vals)
}
// A withdrawn reservation has its item ended at its try, nothing queued (H-A-290 (2)).
queued = nil
_ = p.Ask(ctx, "m", "0.1.2")
v.Withdrawn = map[string]string{"0.1.2": "fp"}
e, _ := kv.Get(ctx, key)
p.Try(ctx, kv, e, decodeItem(t, e.Value()))
if len(kv.vals) != 0 || len(queued) != 0 {
t.Errorf("a withdrawn reservation left its item %v, queued %v", kv.vals, queued)
}
v.Withdrawn = nil
_ = p.Ask(ctx, "m", "0.1.2")
reserved = false
e, _ := kv.Get(ctx, key)
e, _ = kv.Get(ctx, key)
p.Try(ctx, kv, e, decodeItem(t, e.Value()))
if len(kv.vals) != 0 {
t.Errorf("a dropped reservation left its item %v", kv.vals)
@@ -220,7 +268,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
}
first := done("src2:one")
key := versions.PublishItemKey("m", first.Version)
if _, ok := kv.vals[key]; !ok {
if !kv.has(key) {
t.Fatalf("a done build has no publish item: %v", kv.vals)
}
// A build whose artifacts are all published still gets its item, until its release is written (A-456b).
@@ -231,7 +279,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if err := rv.Built(ctx, "m", first.Version, map[string]string{"m:a": "sha256:a"}, nil); err != nil {
t.Fatal(err)
}
if _, ok := kv.vals[key]; !ok {
if !kv.has(key) {
t.Fatalf("a build with every artifact published has no item: %v", kv.vals)
}
// The watch hands the item to its try, which queues the publish.
@@ -247,11 +295,11 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
}
// A takeover by another build deletes the older's item and asks the newer's.
second := done("src2:two")
if _, ok := kv.vals[key]; ok {
if kv.has(key) {
t.Errorf("the taken-over reservation kept its item")
}
secondKey := versions.PublishItemKey("m", second.Version)
if _, ok := kv.vals[secondKey]; !ok {
if !kv.has(secondKey) {
t.Fatalf("the newer build has no item: %v", kv.vals)
}
// version-released deletes the item and keeps the release.
@@ -259,7 +307,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if !rv.heard(ctx, msg) || !msg.acked {
t.Fatal("version-released was not taken in")
}
if _, ok := kv.vals[secondKey]; ok {
if kv.has(secondKey) {
t.Errorf("a released version kept its publish item")
}
if ok, _ := rv.ReleasedFor(ctx, "m", "src2:two"); !ok {
@@ -270,7 +318,7 @@ func TestThePublishItemsAreWiredToTheReservationsAndTheReAskWatch(t *testing.T)
if _, err := rv.dropped(ctx, "m", "0.1", "op", "never", time.Unix(0, 0)); err != nil {
t.Fatal(err)
}
if _, ok := kv.vals[versions.PublishItemKey("m", third.Version)]; ok {
if kv.has(versions.PublishItemKey("m", third.Version)) {
t.Errorf("a dropped reservation kept its item")
}
}
@@ -33,8 +33,8 @@ func (s awaitedQueue) Bytes(ctx context.Context) (uint64, error) {
// entryVia reads a catalog graph entry by its fingerprint through the seat mesh-catalog's verb entry.
func entryVia(call toolCaller) func(ctx context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
return func(_ context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
raw, err := call("seat:"+meshcatalog.Seat+"."+meshcatalog.VerbEntry, meshcatalog.EntryInput{Fingerprint: fingerprint})
return func(ctx context.Context, fingerprint string) (meshcatalog.CatalogEntry, error) {
raw, err := callWithin(ctx, call, "seat:"+meshcatalog.Seat+"."+meshcatalog.VerbEntry, meshcatalog.EntryInput{Fingerprint: fingerprint})
if err != nil {
return meshcatalog.CatalogEntry{}, err
}
@@ -50,8 +50,8 @@ func entryVia(call toolCaller) func(ctx context.Context, fingerprint string) (me
// artifact store, addressed by digest. The four seats' publish take one wire form; an answer that lacks parts, or a fault of
// the call, is the holder's refusal, said in its words (not stored when the holder lacks the bytes).
func sendVia(call toolCaller) func(ctx context.Context, p versions.Publish) error {
return func(_ context.Context, p versions.Publish) error {
raw, err := call("seat:"+p.Seat+"."+versions.VerbPublish, publishBody(p))
return func(ctx context.Context, p versions.Publish) error {
raw, err := callWithin(ctx, call, "seat:"+p.Seat+"."+versions.VerbPublish, publishBody(p))
if err != nil {
return fmt.Errorf("%s's publish of %s %s: %w", p.Seat, p.Name, p.Version, err)
}
@@ -66,6 +66,29 @@ func sendVia(call toolCaller) func(ctx context.Context, p versions.Publish) erro
}
}
// callWithin is call within ctx's bound (design 47 H H-A-290 (4)): its answer, or, once the bound ends first, that it was not
// answered within it. The call itself ends by its own bound (the SDK's call timeout), its answer then dropped.
func callWithin(ctx context.Context, call toolCaller, key string, body any) (json.RawMessage, error) {
if err := ctx.Err(); err != nil {
return nil, fmt.Errorf("%s is not called: %w", key, context.Cause(ctx))
}
type answer struct {
raw json.RawMessage
err error
}
done := make(chan answer, 1)
go func() {
raw, err := call(key, body)
done <- answer{raw, err}
}()
select {
case a := <-done:
return a.raw, a.err
case <-ctx.Done():
return nil, fmt.Errorf("%s is not answered within its bound: %w", key, context.Cause(ctx))
}
}
// publishBody is the seat's publish input for p: the digest, its kind and name and version, no parts.
func publishBody(p versions.Publish) gopackageregistry.PublishInput {
return gopackageregistry.PublishInput{Commit: p.Commit, Digest: p.Digest, Fingerprint: p.Fingerprint, Kind: p.Kind, Name: p.Name, Transfer: p.Transfer, Version: p.Version}
@@ -57,6 +57,26 @@ func TestRepublishSend(t *testing.T) {
t.Errorf("%s: asked %q, error %v, want %q", name, asked, err, c.want)
}
}
// A send and its entry read honour the bound they are given (design 47 H H-A-290 (4)): a call still unanswered when it ends
// is said as not answered within it, and the send goes on.
release := make(chan struct{})
defer close(release)
called := make(chan struct{}, 2)
slow := func(string, any) (json.RawMessage, error) {
called <- struct{}{}
<-release
return json.RawMessage(`{}`), nil
}
for name, call := range map[string]func(context.Context) error{
"the publish": func(ctx context.Context) error { return sendVia(slow)(ctx, p) },
"the entry": func(ctx context.Context) error { _, err := entryVia(slow)(ctx, "fp"); return err },
} {
ctx, cancel := context.WithCancel(context.Background())
go func() { <-called; cancel() }()
if err := call(ctx); err == nil || !strings.Contains(err.Error(), "within its bound") {
t.Errorf("%s past its bound answered %v, want it not answered within its bound", name, err)
}
}
if _, _, err := build.NewRepublish("", "0.1.3", "why", "me", versionsNow()); err == nil {
t.Error("a version with no module is accepted")
}
@@ -49,6 +49,18 @@ func reservedVerb(ctx context.Context, rr versions.Reader, carriers func(context
}
}
// publishedVerb is the handler of the verb published (design 47 H-A-328): whether an artifact built at a fingerprint is
// published, from the reservations and the releases the builder keeps.
func publishedVerb(ctx context.Context, of func(ctx context.Context, module, artifact, fingerprint string) (meshbuilds.PublishedAnswer, error)) process.SeatHandler {
return func(_ string, body json.RawMessage) (any, error) {
var in meshbuilds.PublishedInput
if err := json.Unmarshal(body, &in); err != nil {
return nil, fmt.Errorf("the arguments do not fit the verb published: %w", err)
}
return of(ctx, in.Module, in.Artifact, in.Fingerprint)
}
}
// catalogReleased are a module's released versions, asked of the seat mesh-catalog's verb versions through call.
func catalogReleased(call toolCaller) versions.CatalogVersions {
return func(_ context.Context, module string) ([]meshcatalog.VersionEntry, error) {
@@ -102,7 +102,10 @@ func seed(t *testing.T, st *memStore, b builds.BuilderBuild) string {
func valueOf(t *testing.T, st *memStore, key string) builds.BuilderBuild {
t.Helper()
b, err := builds.DecodeBuilderBuild(st.vals[key])
st.mu.Lock()
raw := st.vals[key]
st.mu.Unlock()
b, err := builds.DecodeBuilderBuild(raw)
if err != nil {
t.Fatal(err)
}
@@ -95,10 +95,14 @@ func theOperatorAnswers(t *testing.T, option string) {
if err != nil || h != occ.Acted {
t.Fatalf("%s: %v, %v", option, h, err)
}
for i := 0; i < 50 && option == OptionBuildOnceMore; i++ { // the build the answer starts runs on its own goroutine
r.b.Wait()
if valueOf(t, r.st, r.key).State == builds.BuildDone {
// The build the answer starts runs on its own goroutine: the builder is waited on only once that build has claimed the key
// (its start counted before its claim, read through the store's lock), never while its start may still be counted.
for i := 0; i < 50 && option == OptionBuildOnceMore; i++ {
if st := valueOf(t, r.st, r.key).State; st == builds.BuildDone {
break
} else if st != builds.BuildFailed {
r.b.Wait()
continue
}
time.Sleep(time.Millisecond)
}
@@ -24,9 +24,24 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
b.init()
started := b.now()
end := meshbuilds.StepEnded{V: 1, Pipeline: run.Pipeline, Step: meshbuilds.StepEndedStepBuild, Rerun: run.Rerun, By: run.By, Entries: run.Entries, Node: b.Node, Started: &started}
// finish is the one place the ask is answered: its line names the pipeline, the movers, the step and the outcome, with
// the words of an end that did not pass (H-A-320 (1), H-A-333).
finish := func(outcome, said string) meshbuilds.StepEnded {
at := b.now()
end.Ended, end.Outcome, end.Said = &at, outcome, said
movers := make([]string, 0, len(run.Entries))
for _, e := range run.Entries {
movers = append(movers, e.Module)
}
mv := strings.Join(movers, ",")
if mv == "" {
mv = "-"
}
if said != "" {
b.logf("%s %s %s step answered %s: %s", run.Pipeline, mv, end.Step, outcome, said)
} else {
b.logf("%s %s %s step answered %s", run.Pipeline, mv, end.Step, outcome)
}
return end
}
if len(run.Entries) == 0 {
@@ -43,7 +58,7 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
var failed []string
for _, e := range run.Entries {
var pins map[string]string
outcome, said := b.buildEntry(ctx, e, &pins)
outcome, said := b.buildEntry(ctx, run.Pipeline, e, &pins)
// The end reports the pins each entry was built with, so the runner asks its held test steps with them (H-A-207).
for path, version := range pins {
if version == "" {
@@ -88,9 +103,10 @@ func (b *Builder) askedRevision(run meshbuilds.StepRun) {
}
}
// buildEntry builds one entry at its fingerprint and waits for its end; pins is set, when it is done, to the pins its
// done key keeps it was built with, the registry's included (H-A-207).
func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins *map[string]string) (outcome, said string) {
// buildEntry builds one entry at its fingerprint for pipeline and waits for its end; pins is set, when it is done, to the pins
// its done key keeps it was built with, the registry's included (H-A-207). The answer from a build is one journal line,
// naming the entry, its outcome, the pipeline and whether the build existed or was built now (H-A-320 (1)).
func (b *Builder) buildEntry(ctx context.Context, pipeline string, e meshbuilds.BuildEntry, pins *map[string]string) (outcome, said string) {
entry, err := b.StepEntry(ctx, e.Fingerprint)
if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry %s at %s cannot be read: %v", e.Module, e.Fingerprint, err)
@@ -110,13 +126,18 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins
}
ended := b.awaitEnd(key)
defer b.dropEnd(key, ended)
answered := func(outcome, said, how string) (string, string) {
b.logf("%s@%s build step answered %s for %s (%s)", entry.Module, entry.Fp, outcome, pipeline, how)
return outcome, said
}
if done, failedWords, found := b.keyEnd(ctx, key); found {
if done {
// A key done before reports the pins it was built with, kept with it, never the ones a publish gives now.
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry)
outcome, said := b.passedWithItsVersion(ctx, entry)
return answered(outcome, said, "existing build")
}
return meshbuilds.StepEndedOutcomeFailed, failedWords
return answered(meshbuilds.StepEndedOutcomeFailed, failedWords, "existing build")
}
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalogclient.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
@@ -125,9 +146,10 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins
case log := <-ended:
if log.Outcome == builds.BuildDone {
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry)
outcome, said := b.passedWithItsVersion(ctx, entry)
return answered(outcome, said, "built now")
}
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error)
return answered(meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error), "built now")
case <-ctx.Done():
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the builder stopped before its build ended", entry.Module, entry.Fp)
}
@@ -7,6 +7,7 @@ import (
"reflect"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -36,12 +37,23 @@ func TestABuildStepAnswersItsEnd(t *testing.T) {
ask := func(entries ...meshbuilds.BuildEntry) meshbuilds.StepEnded {
return r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.1.m", Step: meshbuilds.StepRunStepBuild, Entries: entries})
}
// Every answer is one journal line naming the pipeline, the movers, the step and the outcome (H-A-320 (1), H-A-333).
var answers []string
r.b.Logf = func(f string, a ...any) {
if l := fmt.Sprintf(f, a...); strings.Contains(l, " step answered ") {
answers = append(answers, l)
}
}
if end := ask(); end.Outcome != meshbuilds.StepEndedOutcomeNothingToRun {
t.Errorf("an ask naming no entry: %+v", end)
}
if end := ask(meshbuilds.BuildEntry{Module: "m", Fingerprint: "src2:" + strings.Repeat("2", 64)}); end.Outcome != meshbuilds.StepEndedOutcomeCouldNotRun || !strings.Contains(end.Said, "cannot be read") {
t.Errorf("an entry the seat does not hold: %+v", end)
}
if len(answers) != 2 || answers[0] != "main.1.m - build step answered nothing to run: the ask names no entry" ||
!strings.HasPrefix(answers[1], "main.1.m m build step answered could not run: the entry m at ") {
t.Errorf("the answers journaled %q, want one line per answer", answers)
}
if end := ask(meshbuilds.BuildEntry{Module: "m", Fingerprint: fp1, Commit: "other"}); end.Outcome != meshbuilds.StepEndedOutcomeCouldNotRun {
t.Errorf("an entry at another commit: %+v", end)
}
@@ -171,6 +183,18 @@ func awaitWaiting[T any](t *testing.T, waiting <-chan struct{}, ended <-chan T)
// another series and the module's release number pin nothing.
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
r, _, waiting := unpinnedRig(t)
// Each entry's answer from a build is one journal line, built now or existing, beside the ask's answer (H-A-320 (1), H-A-333).
var mu sync.Mutex
var answers []string
logf := r.b.Logf
r.b.Logf = func(format string, args ...any) {
logf(format, args...)
if l := fmt.Sprintf(format, args...); strings.HasPrefix(l, "m@") && strings.Contains(l, "build step answered") {
mu.Lock()
answers = append(answers, l)
mu.Unlock()
}
}
r.b.PackagePublished(unpinnedX, "v0.2.9") // another series than the 0.1 the module accepts: no pin
r.b.PackagePublished(unpinnedX, "0.1.4-feat-x.2") // a group's tagged version: no pin
ended := make(chan meshbuilds.StepEnded, 1)
@@ -206,6 +230,12 @@ func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
if again.Outcome != meshbuilds.StepEndedOutcomePassed || again.Packages[unpinnedX] != "v0.1.3" {
t.Errorf("the done key's end = %+v; want passed reporting x at v0.1.3, as built", again)
}
mu.Lock()
want := []string{"m@" + fp1 + " build step answered passed for main.9.m (built now)", "m@" + fp1 + " build step answered passed for main.9.m (existing build)"}
if !reflect.DeepEqual(answers, want) {
t.Errorf("the answers journaled %q, want %q", answers, want)
}
mu.Unlock()
// A done key that keeps no pins reports none, so the runner ends the waiting steps loud.
v, err := builds.DecodeBuilderBuild(r.st.vals[r.key])
if err != nil {
@@ -7,6 +7,7 @@ import (
"fmt"
"slices"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -38,6 +39,15 @@ const (
// (H-A-272).
VerbPublish = "publish"
VerbVersions = "versions"
// PublishClassLive is the class of a publish whose build ended in this run, which a pipeline's build step waits on;
// PublishClassStart that of a publish asked again at a start, of a build that ended before it. A holder's queue sends every
// live one before any of the start, each class oldest first (design 47 H H-A-290 (1)).
PublishClassLive = "build-ended-this-run"
PublishClassStart = "asked-again-at-start"
// PublishSendsAtOnce bounds the sends of one holder's queue that run at once (H-A-290 (3)).
PublishSendsAtOnce = 4
// PublishQueueListCap caps the queued versions and the publish items builder_publish_queue lists (H-A-290 (5)).
PublishQueueListCap = 50
)
// RepublishTransfer is a publish again's transfer name, the builder's own (the registry part's publish: the caller names its
@@ -78,6 +88,26 @@ type QueuedPublish struct {
Artifacts map[string]string `json:"artifacts"`
Queued time.Time `json:"queued"`
Begun bool `json:"begun,omitempty"`
// Class is PublishClassLive or PublishClassStart (H-A-290 (1)); a value written before it had one is of the start.
Class string `json:"class,omitempty"`
}
// rank is the place of q's class in a holder's queue: the live first.
func (q QueuedPublish) rank() int {
if q.Class == PublishClassLive {
return 0
}
return 1
}
// sortQueued sorts a holder's queued versions in the order they are sent: the live class first, each class oldest first.
func sortQueued(qs []QueuedPublish) {
sort.SliceStable(qs, func(i, j int) bool {
if a, b := qs[i].rank(), qs[j].rank(); a != b {
return a < b
}
return qs[i].Queued.Before(qs[j].Queued)
})
}
// QueueFilter is the filter of every queued version: a queue key has three tokens, which AllFilter (two) never lists on the bus.
@@ -121,9 +151,18 @@ type Republisher struct {
// Holders answers a registry seat's holders: the catalog's SeatHolders, the part's one holder function that its FollowSeat
// of the assignments takes too, read from the builder's start (design 47 G A-416, A-416a); nil answers none.
Holders func(seat string) []string
// Started is when this run of the builder started: a version whose build ended at or after it is of the live class, one
// that ended before it of the start's (H-A-290 (1)). Zero makes every version live.
Started time.Time
// Ended ends the publish item of a version that never publishes, its reservation withdrawn or taken over (H-A-290 (2));
// nil ends none.
Ended func(ctx context.Context, module, version string)
mu sync.Mutex
busy map[string]bool
// relist marks a holder whose pass lists again before its next start-class send: a live publish was queued meanwhile
// (H-A-291 (2)).
relist map[string]bool
// lastQueued is the newest time a version was queued at: a clock that has not moved between two versions (a coarse clock,
// or a fixed one) must not make the queue's order the order of its keys, so each is queued after the last.
lastQueued time.Time
@@ -155,6 +194,127 @@ func (r *Republisher) logf(f string, a ...any) {
}
}
// classOf is the class of v's publish: live when its build ended in this run (H-A-290 (1)).
func (r *Republisher) classOf(v ReservedVersion) string {
if v.BuiltAt.Before(r.Started) {
return PublishClassStart
}
return PublishClassLive
}
// NotStanding says why the reservation of a version, as Of answers it (found), never publishes: it stands no more at the
// version (released, dropped or taken over by a newer build), or it was withdrawn (design 47 H H-A-225, H-A-279). Empty when it
// stands (H-A-290 (2)).
func NotStanding(v ReservedVersion, found bool) string {
switch {
case !found:
return "its reservation stands no more at the version (released, dropped or taken over)"
case hasKey(v.Withdrawn, v.Version):
return "its reservation was withdrawn"
}
return ""
}
// newerVersion says whether MAJOR.MINOR.PATCH a is above b, number by number.
func newerVersion(a, b string) bool {
pa, pb := strings.Split(a, "."), strings.Split(b, ".")
if len(pa) != 3 || len(pb) != 3 {
return false
}
for i := range 3 {
x, errx := strconv.Atoi(pa[i])
y, erry := strconv.Atoi(pb[i])
if errx != nil || erry != nil {
return false
}
if x != y {
return x > y
}
}
return false
}
// Retired are the done builds without a release among all that a newer reservation or a newer release of the same module and
// series (MAJOR.MINOR) supersedes, each with why (design 47 H H-A-291 (1), H-A-293; H-A-275: the older digest is dead): keyed
// `<module> <version>`. A standing newer series never retires an older one's build. The releases are the builder's own record
// of them (Releases), read once per module.
func (r *Reserver) Retired(ctx context.Context, all []ReservedVersion) (map[string]string, error) {
seriesOf := func(v ReservedVersion) string {
s, _, err := SplitVersion(v.Version)
if err != nil {
return ""
}
return v.Module + " " + s
}
newest := map[string]string{} // module and series -> its newest reserved version
for _, v := range all {
k := seriesOf(v)
if n, ok := newest[k]; k != "" && (!ok || newerVersion(v.Version, n)) {
newest[k] = v.Version
}
}
kept := map[string][]string{} // module -> the releases the record keeps
read := map[string]bool{}
out := map[string]string{}
for _, v := range all {
k := seriesOf(v)
if v.BuiltAt.IsZero() || k == "" {
continue
}
if n := newest[k]; newerVersion(n, v.Version) {
out[v.Module+" "+v.Version] = "the newer reservation " + n + " of " + v.Module + " supersedes it"
continue
}
if !read[v.Module] && r.Releases != nil {
read[v.Module] = true
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(v.Module))
if err != nil {
return out, fmt.Errorf("the releases of %s cannot be read: %w", v.Module, err)
}
var rec ReleasedRecord
if found && json.Unmarshal(raw, &rec) == nil {
for _, rb := range rec.Kept {
kept[v.Module] = append(kept[v.Module], rb.Version)
}
}
}
series, _, _ := SplitVersion(v.Version)
rel := ""
for _, kv := range kept[v.Module] {
if s, _, err := SplitVersion(kv); err == nil && s == series && newerVersion(kv, v.Version) && (rel == "" || newerVersion(kv, rel)) {
rel = kv
}
}
if rel != "" {
out[v.Module+" "+v.Version] = "the newer release " + rel + " of " + v.Module + " supersedes it"
}
}
return out, nil
}
// drop deletes every queued key of module's version and ends its publish item: it never publishes (H-A-290 (2)), said with why.
func (r *Republisher) drop(ctx context.Context, module, version, why string) {
ctx = context.WithoutCancel(ctx)
keys, err := r.Queue.KeysOf(ctx, "*."+EscapeToken(module)+"."+EscapeToken(version))
if err != nil {
r.logf("publish again: %s %s never publishes (%s), and its queued keys cannot be listed: %v", module, version, why, err)
}
for _, k := range keys {
if err := r.Queue.Delete(ctx, k); err != nil {
r.logf("%s cannot be deleted from %s: %v", k, PublishQueueState, err)
}
}
r.end(ctx, module, version, why)
}
// end ends module's version's publish item, said with why.
func (r *Republisher) end(ctx context.Context, module, version, why string) {
r.logf("publish again: %s %s is not published: %s; its queued key is dropped and its publish item ended", module, version, why)
if r.Ended != nil {
r.Ended(context.WithoutCancel(ctx), module, version)
}
}
// ActedKey is the acted key of a subject.
func ActedKey(subject string) string { return EscapeToken(subject) }
@@ -189,6 +349,10 @@ func (r *Republisher) queue(ctx context.Context, vs []ReservedVersion, holdersOf
if max > 0 && len(queued) >= max {
break
}
if why := NotStanding(v, true); why != "" {
r.drop(ctx, v.Module, v.Version, why)
continue
}
if v.Fingerprint == "" || v.BuiltAt.IsZero() {
r.logf("publish again: %s %s is not queued: it holds no fingerprint or its build has not ended with every digest", v.Module, v.Version)
continue
@@ -251,7 +415,7 @@ func (r *Republisher) enqueue(ctx context.Context, v ReservedVersion, holder str
r.logf("publish again: %s %s is queued at %s already", v.Module, v.Version, holder)
return false, nil
}
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: holder, Module: v.Module, Version: v.Version, Artifacts: arts, Queued: r.queuedAt()})
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: holder, Module: v.Module, Version: v.Version, Artifacts: arts, Queued: r.queuedAt(), Class: r.classOf(v)})
used, err := r.Queue.Bytes(ctx)
if err != nil {
return false, fmt.Errorf("the size of the state %s cannot be read: %w", PublishQueueState, err)
@@ -268,7 +432,15 @@ func (r *Republisher) enqueue(ctx context.Context, v ReservedVersion, holder str
}
return false, fmt.Errorf("%s cannot be queued: %w", key, err)
}
r.logf("publish again: %s %s is queued to publish at %s: %d artifacts", v.Module, v.Version, holder, len(arts))
r.logf("publish again: %s %s is queued to publish at %s: %d artifacts, class %s", v.Module, v.Version, holder, len(arts), r.classOf(v))
if r.classOf(v) == PublishClassLive { // a pass sending a start list lists again, so this one goes next (H-A-291 (2))
r.mu.Lock()
if r.relist == nil {
r.relist = map[string]bool{}
}
r.relist[holder] = true
r.mu.Unlock()
}
return true, nil
}
@@ -305,13 +477,43 @@ func (r *Republisher) Republish(ctx context.Context, module, version string) ([]
return r.queue(ctx, vs, r.holdersFn, 0)
}
// Queued are the queued versions, per holder in the order they are sent, and the acted keys with their sequences:
// builder_publish_queue.
// Queued is builder_publish_queue's answer, small at any size of the queue (design 47 H H-A-290 (5)): the queued versions'
// count per holder and class with the oldest and newest queue time, the first PublishQueueListCap of them in the order they are
// sent (each with its class), the acted keys with their sequences, and the publish items, counted and capped alike.
type Queued struct {
Queue []QueuedPublish `json:"queue"`
Acted map[string]uint64 `json:"acted"`
// Items are the publish items (design 47 G A-457a), each with its tries and its next try; an item that ended says why.
Items []PublishItem `json:"items"`
Total int `json:"total"`
Counts []QueueCount `json:"counts"`
// Queue are the first queued versions, per holder in the order they are sent (the live class first, each class oldest first);
// with a module named, only that module's.
Queue []QueuedPublish `json:"queue"`
// Listed says how many of Total the answer lists.
Listed int `json:"listed"`
Acted map[string]uint64 `json:"acted"`
// ItemCount is how many publish items stand; Items the first of them by their next try (design 47 G A-457a), each with its
// tries and its next try; an item that ended says why.
ItemCount int `json:"item-count"`
Items []PublishItem `json:"items"`
// Capped says a state held more than the one page read of it (H-A-293); Read names each such state, the keys read and its
// size. Every count above is then of the keys read.
Capped bool `json:"capped,omitempty"`
Read []StateCapped `json:"read,omitempty"`
}
// capped notes a state read only in part.
func (q *Queued) capped(c *StateCapped) {
if c != nil {
q.Capped = true
q.Read = append(q.Read, *c)
}
}
// QueueCount is how many versions one holder's queue holds of one class, and when the oldest and the newest were queued.
type QueueCount struct {
Holder string `json:"holder"`
Class string `json:"class"`
Count int `json:"count"`
Oldest time.Time `json:"oldest"`
Newest time.Time `json:"newest"`
}
// PublishItemKeyPrefix opens the key of a publish item in the builder's re-ask state.
@@ -335,69 +537,113 @@ func PublishItemKey(module, version string) string {
return PublishItemKeyPrefix + EscapeToken(module) + "." + EscapeToken(version)
}
// AnswerQueue is builder_publish_queue over the two sections as the tools bundle reads them.
func AnswerQueue(queue, acted, reasks Reader) (Queued, error) {
out := Queued{Queue: []QueuedPublish{}, Acted: map[string]uint64{}, Items: []PublishItem{}}
keys, err := queue.Keys()
// AnswerQueue is builder_publish_queue over the three states as the tools bundle reads them (H-A-290 (5)): each state read in
// bulk (Reader.All), the answer counted and capped at PublishQueueListCap whatever the queue holds; module (and version) narrow the
// listed queue to one module's (one version's), so an item's class can be read.
func AnswerQueue(queue, acted, reasks Reader, module, version string) (Queued, error) {
out := Queued{Counts: []QueueCount{}, Queue: []QueuedPublish{}, Acted: map[string]uint64{}, Items: []PublishItem{}}
vals, c, err := queue.values(PublishQueueState)
out.capped(c)
if err != nil {
return out, fmt.Errorf("the state %s cannot be listed: %w", PublishQueueState, err)
return out, fmt.Errorf("the state %s cannot be read: %w", PublishQueueState, err)
}
for _, k := range keys {
raw, ok, err := queue.Get(k)
if err != nil || !ok {
counts := map[[2]string]*QueueCount{}
var listed []QueuedPublish
for _, raw := range vals {
var q QueuedPublish
if json.Unmarshal(raw, &q) != nil {
continue
}
var q QueuedPublish
if json.Unmarshal(raw, &q) == nil {
if q.Class == "" {
q.Class = PublishClassStart
}
out.Total++
c := counts[[2]string{q.Holder, q.Class}]
if c == nil {
c = &QueueCount{Holder: q.Holder, Class: q.Class, Oldest: q.Queued, Newest: q.Queued}
counts[[2]string{q.Holder, q.Class}] = c
}
c.Count++
if q.Queued.Before(c.Oldest) {
c.Oldest = q.Queued
}
if q.Queued.After(c.Newest) {
c.Newest = q.Queued
}
if (module == "" || q.Module == module) && (version == "" || q.Version == version) {
listed = append(listed, q)
}
}
for _, c := range counts {
out.Counts = append(out.Counts, *c)
}
sort.Slice(out.Counts, func(i, j int) bool {
if out.Counts[i].Holder != out.Counts[j].Holder {
return out.Counts[i].Holder < out.Counts[j].Holder
}
return out.Counts[i].Class < out.Counts[j].Class
})
sort.SliceStable(listed, func(i, j int) bool { return listed[i].Holder < listed[j].Holder })
byHolder := map[string][]QueuedPublish{}
var holders []string
for _, q := range listed {
if byHolder[q.Holder] == nil {
holders = append(holders, q.Holder)
}
byHolder[q.Holder] = append(byHolder[q.Holder], q)
}
for _, h := range holders {
qs := byHolder[h]
sortQueued(qs)
for _, q := range qs {
if len(out.Queue) == PublishQueueListCap {
break
}
out.Queue = append(out.Queue, q)
}
}
sort.SliceStable(out.Queue, func(i, j int) bool {
if out.Queue[i].Holder != out.Queue[j].Holder {
return out.Queue[i].Holder < out.Queue[j].Holder
}
return out.Queue[i].Queued.Before(out.Queue[j].Queued)
})
akeys, err := acted.Keys()
out.Listed = len(out.Queue)
avals, c, err := acted.values(ActedState)
out.capped(c)
if err != nil {
return out, fmt.Errorf("the state %s cannot be listed: %w", ActedState, err)
return out, fmt.Errorf("the state %s cannot be read: %w", ActedState, err)
}
for _, k := range akeys {
raw, ok, err := acted.Get(k)
if err != nil || !ok {
continue
}
for k, raw := range avals {
var a Acted
if json.Unmarshal(raw, &a) == nil {
out.Acted[k] = a.Sequence
}
}
rkeys, err := reasks.Keys()
rvals, c, err := reasks.values("reasks")
out.capped(c)
if err != nil {
return out, fmt.Errorf("the re-ask state cannot be listed: %w", err)
return out, fmt.Errorf("the re-ask state cannot be read: %w", err)
}
for _, k := range rkeys {
for k, raw := range rvals {
if !strings.HasPrefix(k, PublishItemKeyPrefix) {
continue
}
raw, ok, err := reasks.Get(k)
if err != nil || !ok {
continue
}
var it PublishItem
if json.Unmarshal(raw, &it) == nil {
out.Items = append(out.Items, it)
}
}
out.ItemCount = len(out.Items)
sort.SliceStable(out.Items, func(i, j int) bool { return out.Items[i].NextTry.Before(out.Items[j].NextTry) })
if len(out.Items) > PublishQueueListCap {
out.Items = out.Items[:PublishQueueListCap]
}
return out, nil
}
// PublishQueueDescription is builder_publish_queue's description.
const PublishQueueDescription = "The versions the builder will publish again (its state " + PublishQueueState + ", design 47 G section 588): per registry " +
"seat holder, in the order they are sent, each with its artifacts' digests, when it was queued and whether its publish has begun; and the " +
"keys of the state " + ActedState + ", the last stream sequence the builder acted on for each subject; and the publish items (design 47 G " +
"A-457a), each version's tries and next try, and why one ended after its tries. Read-only."
const PublishQueueDescription = "The versions the builder will publish again (its state " + PublishQueueState + ", design 47 G section 588, H H-A-290): " +
"how many each registry seat holder's queue holds per class (build-ended-this-run, sent first, or asked-again-at-start) with the oldest and newest " +
"queue time; the first 50 in the order they are sent, each with its class, its artifacts' digests, when it was queued and whether its publish has " +
"begun (module, and version, list only that module's, so one item's class can be read); the keys of the state " + ActedState + ", the last stream " +
"sequence the builder acted on for each subject; and the publish items (design 47 G A-457a), counted, the first 50 by next try, each version's " +
"tries and next try, and why one ended after its tries. Each state is read in one page; a state past it is answered capped, naming the " +
"keys read and the state's size (H-A-293). Read-only."
// PublishAgainDescription is builder_publish_again's description.
const PublishAgainDescription = "Publish reserved versions again (design 47 G section 588), which builds nothing: with module and version, that reserved " +
@@ -406,22 +652,57 @@ const PublishAgainDescription = "Publish reserved versions again (design 47 G se
"with the versions it queued; each outcome is said as package-published, or left in version-unreleased with the holder's words. A hand act, " +
"recorded in the hand-act log."
// Resume takes up, at the builder's start, every queued version again, a begun one sent anew (§588).
// Resume takes up, at the builder's start, every queued version again, a begun one sent anew (§588): a version whose reservation
// was withdrawn or taken over is dropped with its publish item ended (H-A-290 (2)), and one of the live class whose build ended
// before this run is of the start's now (H-A-290 (1)). It reads the reservations once.
func (r *Republisher) Resume(ctx context.Context) error {
keys, err := r.Queue.KeysOf(ctx, QueueFilter)
if err != nil {
return fmt.Errorf("the state %s cannot be listed: %w", PublishQueueState, err)
}
all, err := r.Reserved.All(ctx)
if err != nil {
return fmt.Errorf("the queued publishes are not taken up: %w", err)
}
reserved := make(map[string]ReservedVersion, len(all))
for _, v := range all {
reserved[v.Module+" "+v.Version] = v
}
retired, err := r.Reserved.Retired(ctx, all)
if err != nil {
r.logf("publish again: the superseded builds are not retired at the start: %v", err)
}
holders := map[string]bool{}
for _, k := range keys {
raw, _, found, err := r.Queue.Get(ctx, k)
raw, rev, found, err := r.Queue.Get(ctx, k)
if err != nil || !found {
continue
}
var q QueuedPublish
if json.Unmarshal(raw, &q) == nil {
holders[q.Holder] = true
if json.Unmarshal(raw, &q) != nil {
continue
}
v, ok := reserved[q.Module+" "+q.Version]
why := NotStanding(v, ok)
if why == "" {
why = retired[q.Module+" "+q.Version]
}
if why != "" {
if err := r.Queue.Delete(ctx, k); err != nil {
r.logf("%s cannot be deleted from %s: %v", k, PublishQueueState, err)
}
r.end(ctx, q.Module, q.Version, why)
continue
}
if class := r.classOf(v); q.Class != class {
q.Class = class
if raw, err := json.Marshal(q); err == nil {
if _, err := r.Queue.Update(ctx, k, raw, rev); err != nil {
r.logf("%s keeps its class %s at the start: %v", k, q.Class, err)
}
}
}
holders[q.Holder] = true
}
for h := range holders {
r.drainLater(ctx, h)
@@ -451,16 +732,22 @@ func (r *Republisher) drainLater(ctx context.Context, holder string) {
go run()
}
// Drain sends holder's queued versions one at a time, oldest first, each bounded by UnreleasedBound; a key is deleted when
// its outcome is said.
// Drain sends holder's queued versions in passes (H-A-290 (3)): a pass lists the holder's keys once, reads each once, and sends
// them in their order (the live class first, each class oldest first), PublishSendsAtOnce at a time; the keys are listed again
// only when the pass is spent, or when a live publish was queued while it sends a start list: then the sends in flight end and
// the keys are listed again, so the live one goes next (H-A-291 (2)); a start-class one queued meanwhile waits for the pass's end.
// Each send is bounded by UnreleasedBound; a key is deleted when its outcome is said.
func (r *Republisher) Drain(ctx context.Context, holder string) {
for ctx.Err() == nil {
r.mu.Lock()
delete(r.relist, holder)
r.mu.Unlock()
keys, err := r.Queue.KeysOf(ctx, EscapeToken(holder)+".*.*")
if err != nil {
r.logf("the queue of %s cannot be listed: %v", holder, err)
return
}
var items []QueuedPublish
items := make([]QueuedPublish, 0, len(keys))
for _, k := range keys {
if raw, _, found, err := r.Queue.Get(ctx, k); err == nil && found {
var q QueuedPublish
@@ -472,12 +759,35 @@ func (r *Republisher) Drain(ctx context.Context, holder string) {
if len(items) == 0 {
return
}
sort.SliceStable(items, func(i, j int) bool { return items[i].Queued.Before(items[j].Queued) })
r.send(ctx, items[0])
sortQueued(items)
sem := make(chan struct{}, PublishSendsAtOnce)
var wg sync.WaitGroup
for _, q := range items {
if ctx.Err() != nil {
break
}
sem <- struct{}{}
r.mu.Lock()
again := q.Class != PublishClassLive && r.relist[holder]
r.mu.Unlock()
if again {
<-sem
r.logf("publish again: %s's pass lists again after its sends in flight: a live publish was queued", holder)
break
}
wg.Add(1)
go func() {
defer func() { <-sem; wg.Done() }()
r.send(ctx, q)
}()
}
wg.Wait()
}
}
// send publishes one queued version's artifacts and says the outcome, then deletes its key.
// send publishes one queued version's artifacts and says the outcome, then deletes its key. Every call it makes honours the one
// bound it is given, UnreleasedBound (a context's bound, no timer, ADR 0437 §8), and a send that returns without sending says why
// (H-A-290 (4)).
func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
key := QueueKey(q.Holder, q.Module, q.Version)
defer func() {
@@ -485,18 +795,44 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
r.logf("%s cannot be deleted from %s: %v", key, PublishQueueState, err)
}
}()
bound, cancel := context.WithTimeout(ctx, UnreleasedBound)
defer cancel()
if !q.Begun {
q.Begun = true
raw, _ := json.Marshal(q)
if _, rev, found, err := r.Queue.Get(ctx, key); err == nil && found {
_, _ = r.Queue.Update(ctx, key, raw, rev)
if _, rev, found, err := r.Queue.Get(bound, key); err == nil && found {
_, _ = r.Queue.Update(bound, key, raw, rev)
}
}
v, found, err := r.Reserved.Of(ctx, q.Module, q.Version)
if err != nil || !found {
return // released meanwhile, or dropped
all, err := r.Reserved.All(bound)
if err != nil {
r.logf("publish again: %s %s is not sent to %s: its reservation cannot be read: %v; its publish item asks it again", q.Module, q.Version, q.Holder, err)
return
}
e, err := r.Entry(ctx, v.Fingerprint)
var v ReservedVersion
found := false
var mine []ReservedVersion
for _, w := range all {
if w.Module == q.Module {
mine = append(mine, w)
if w.Version == q.Version {
v, found = w, true
}
}
}
why := NotStanding(v, found)
if why == "" {
retired, err := r.Reserved.Retired(bound, mine)
if err != nil {
r.logf("publish again: %s %s: whether a newer version supersedes it cannot be read, so it is sent: %v", q.Module, q.Version, err)
}
why = retired[q.Module+" "+q.Version]
}
if why != "" {
r.end(ctx, q.Module, q.Version, why)
return
}
e, err := r.Entry(bound, v.Fingerprint)
if err != nil {
r.say(ctx, v, fmt.Sprintf("the catalog entry %s cannot be read: %v", v.Fingerprint, err))
return
@@ -512,10 +848,8 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
r.say(ctx, v, ref+": the catalog entry names no registry seat for it")
return
}
bound, cancel := context.WithTimeout(ctx, UnreleasedBound) // a context's bound, no timer (ADR 0437 §8)
err := r.Send(bound, Publish{Seat: seat, Kind: kind, Name: name, Version: v.Version, Digest: q.Artifacts[ref], Fingerprint: v.Fingerprint,
Commit: v.Commit, Transfer: RepublishTransfer(q.Module, v.Version, strings.TrimPrefix(ref, q.Module+":"))})
cancel()
if err == nil {
r.logf("publish again: %s %s: %s sent to %s as %s %s, digest %s; the package-published is said when the seat says published", q.Module, q.Version, ref, seat, name, v.Version, q.Artifacts[ref])
}
@@ -523,7 +857,7 @@ func (r *Republisher) send(ctx context.Context, q QueuedPublish) {
r.logf("publish again: %s %s: %s refused by %s: %v", q.Module, q.Version, ref, seat, err)
words := err.Error()
if strings.Contains(words, strings.TrimSuffix(NotStoredWords, ": ")) {
if ok, merr := r.Reserved.MarkUnstored(ctx, q.Module, q.Version, ref, words); merr != nil {
if ok, merr := r.Reserved.MarkUnstored(bound, q.Module, q.Version, ref, words); merr != nil {
r.logf("%s %s: the unstored %s is not kept: %v", q.Module, q.Version, ref, merr)
} else if ok {
r.say(ctx, v, ref+" "+NotStoredWords+words)
@@ -5,8 +5,13 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
@@ -15,10 +20,19 @@ import (
type queueMem struct {
*reserveMem
bytes uint64
// listings counts the listings of one holder's keys (a filter `<holder>.*.*`), what a drain's pass lists.
listings atomic.Int32
}
func (q *queueMem) Bytes(context.Context) (uint64, error) { return q.bytes, nil }
func (q *queueMem) KeysOf(ctx context.Context, filter string) ([]string, error) {
if strings.HasSuffix(filter, ".*.*") && filter != QueueFilter {
q.listings.Add(1)
}
return q.reserveMem.KeysOf(ctx, filter)
}
// republishBed is a Republisher over memory states: reservations of module m (a bundle by the oci seat and a Go package by
// the Go seat), one holder module `oci` that claims the oci seat, and a Send and a Unreleased that record.
type republishBed struct {
@@ -26,8 +40,10 @@ type republishBed struct {
reserved *reserveMem
queue *queueMem
now time.Time
mu sync.Mutex // sent and said, which a holder's sends at once write
sent []Publish
said []string
ended []string
refuse string // the words Send fails with, empty for none
// holders are the registry seats' holders the seat-holders view names (design 47 G A-416).
holders map[string][]string
@@ -48,6 +64,8 @@ func newRepublishBed(t *testing.T) *republishBed {
{Ref: m + ":lib", Kind: "package", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "git.novox.be/novox/" + m + "/go"}}}}, nil
},
Send: func(_ context.Context, p Publish) error {
b.mu.Lock()
defer b.mu.Unlock()
b.sent = append(b.sent, p)
if b.refuse != "" {
return errors.New(b.refuse)
@@ -55,8 +73,15 @@ func newRepublishBed(t *testing.T) *republishBed {
return nil
},
Unreleased: func(_ context.Context, v ReservedVersion, why string) {
b.mu.Lock()
defer b.mu.Unlock()
b.said = append(b.said, v.Module+" "+v.Version+": "+why)
},
Ended: func(_ context.Context, module, version string) {
b.mu.Lock()
defer b.mu.Unlock()
b.ended = append(b.ended, module+" "+version)
},
Logf: t.Logf}
b.Holders = func(seat string) []string { return b.holders[seat] }
return b
@@ -190,7 +215,7 @@ func TestRepublish(t *testing.T) {
return json.RawMessage(`{"mode":"publish","module":"m","version":"0.1.0","tries":3,"next-try":"2026-11-17T01:00:00Z","why":"w"}`), true, nil
}}
none := Reader{Keys: func() ([]string, error) { return nil, nil }}
answer, err := AnswerQueue(none, none, items)
answer, err := AnswerQueue(none, none, items, "", "")
if err != nil || len(answer.Items) != 1 || answer.Items[0].Tries != 3 || answer.Items[0].Version != "0.1.0" {
t.Errorf("builder_publish_queue items = %+v (%v), want the one publish item with its tries", answer.Items, err)
}
@@ -235,3 +260,279 @@ func TestTheRegistriesRecordSaysWhetherAVersionIsHeld(t *testing.T) {
t.Errorf("a version with no missing artifact is judged %+v (%v), want all held without a read", j, err)
}
}
// queueAt writes a queued version of module m at holder oci directly, as a start finds it.
func (b *republishBed) queueAt(t *testing.T, v ReservedVersion, class string, at time.Time) {
t.Helper()
raw, _ := json.Marshal(QueuedPublish{V: 1, Holder: "oci", Module: v.Module, Version: v.Version, Artifacts: map[string]string{v.Module + ":prog": "sha256:aa"}, Queued: at, Class: class})
if _, err := b.queue.Create(context.Background(), QueueKey("oci", v.Module, v.Version), raw); err != nil {
t.Fatal(err)
}
}
// TestAHoldersQueueSendsLiveFirstInPasses: design 47 H H-A-290 (1)-(3) — a publish whose build ended in this run is sent before
// any of a build that ended before the start, each class oldest first; a pass lists the holder's keys once and sends them, at most
// four at a time; a build whose reservation was withdrawn or taken over never publishes, its key dropped and its publish item ended,
// at enqueue, at the pass and at the start.
func TestAHoldersQueueSendsLiveFirstInPasses(t *testing.T) {
ctx := context.Background()
t.Run("the live class first, at most four sends at once", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
b := newRepublishBed(t)
var old []ReservedVersion
for _, m := range []string{"o1", "o2", "o3", "o4", "o5"} {
old = append(old, b.reserve(t, m))
}
b.Started = b.now
b.reserve(t, "live")
for i, v := range old {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.now = b.now.Add(time.Hour) // the live one is queued last, after every old one
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
release := make(chan struct{})
var arrived []string
var inflight, most int
send := b.Send
b.Send = func(ctx context.Context, p Publish) error {
b.mu.Lock()
arrived = append(arrived, p.Name)
inflight++
most = max(most, inflight)
b.mu.Unlock()
<-release
b.mu.Lock()
inflight--
b.mu.Unlock()
return send(ctx, p)
}
b.Go = func(f func()) { go f() }
if q, err := b.Republish(ctx, "live", ""); err != nil || len(q) != 1 {
t.Fatalf("queued %v (%v), want the live version", q, err)
}
synctest.Wait() // every send the bound lets run waits on release
b.mu.Lock()
first := slices.Clone(arrived)
b.mu.Unlock()
if len(first) != PublishSendsAtOnce || !slices.Contains(first, "live/prog") {
t.Errorf("the sends running at once are %v: want %d, the live one among them", first, PublishSendsAtOnce)
}
close(release)
synctest.Wait()
if len(b.sent) != 6 || most != PublishSendsAtOnce {
t.Errorf("sent %d, at most %d at once: want all six, %d at once", len(b.sent), most, PublishSendsAtOnce)
}
})
})
t.Run("a pass lists the keys once for all its sends", func(t *testing.T) {
b := newRepublishBed(t)
var vs []ReservedVersion
for i := range 6 {
vs = append(vs, b.reserve(t, fmt.Sprintf("m%d", i)))
}
for i, v := range vs {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.Drain(ctx, "oci")
if n := b.queue.listings.Load(); len(b.sent) != 6 || n != 2 {
t.Errorf("sent %d with %d listings, want six sends from one pass and one listing that finds it spent", len(b.sent), n)
}
})
t.Run("a start makes a live key of a build before it a start key", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.Started = b.now
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {} // the start's drain is not run here
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
raw, _, _, _ := b.queue.Get(ctx, QueueKey("oci", "m", v.Version))
var q QueuedPublish
if json.Unmarshal(raw, &q) != nil || q.Class != PublishClassStart {
t.Errorf("the key after the start is %s, want the class %s: its build did not end in this run", raw, PublishClassStart)
}
})
for _, at := range []string{"enqueue", "pass", "start"} {
t.Run("a withdrawn build never publishes, at the "+at, func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
if _, ok, err := b.Reserved.Withdraw(ctx, "m", v.Version); err != nil || !ok {
t.Fatalf("withdrawn %v (%v)", ok, err)
}
switch at {
case "enqueue":
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {} // a key queued before the withdrawal is dropped by the enqueue itself
if q, _ := b.Republish(ctx, "m", v.Version); len(q) != 0 {
t.Errorf("queued %v", q)
}
case "pass":
b.queueAt(t, v, PublishClassLive, b.now)
b.Drain(ctx, "oci")
case "start":
b.queueAt(t, v, PublishClassLive, b.now)
b.Go = func(func()) {}
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
}
keys, _ := b.queue.KeysOf(ctx, QueueFilter)
if len(b.sent) != 0 || len(keys) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("sent %v, keys %v, items ended %v: want nothing sent, the key dropped and m %s's item ended", b.sent, keys, b.ended, v.Version)
}
})
}
t.Run("a live publish queued mid-pass goes after the sends in flight, before the rest of the start list", func(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
b := newRepublishBed(t)
var old []ReservedVersion
for _, m := range []string{"o1", "o2", "o3", "o4", "o5", "o6"} {
old = append(old, b.reserve(t, m))
}
b.Started = b.now
b.reserve(t, "live")
for i, v := range old {
b.queueAt(t, v, PublishClassStart, b.now.Add(time.Duration(i)*time.Second))
}
b.now = b.now.Add(time.Hour)
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
first, then := make(chan struct{}), make(chan struct{}) // the four in flight wait on first, every later send on then
var arrived []string
send := b.Send
b.Send = func(ctx context.Context, p Publish) error {
b.mu.Lock()
arrived = append(arrived, p.Name)
gate := map[bool]chan struct{}{true: first, false: then}[len(arrived) <= PublishSendsAtOnce]
b.mu.Unlock()
<-gate
return send(ctx, p)
}
b.Go = func(f func()) { go f() }
b.drainLater(ctx, "oci")
synctest.Wait() // four start sends in flight, the pass holding o5 and o6
if q, err := b.Republish(ctx, "live", ""); err != nil || len(q) != 1 {
t.Fatalf("queued %v (%v)", q, err)
}
close(first)
synctest.Wait() // the sends after the four: with the live one among them, or o5 and o6 alone when the pass goes on
b.mu.Lock()
next := slices.Clone(arrived[PublishSendsAtOnce:])
b.mu.Unlock()
if !slices.Contains(next, "live/prog") {
t.Errorf("after the four in flight the sends are %v, want the live one among them, the pass listed again", next)
}
close(then)
synctest.Wait()
if len(arrived) != 7 {
t.Errorf("sent %v, want all seven", arrived)
}
})
})
t.Run("a build a newer patch or release of its series supersedes is retired at the pass and at the start, never by a newer series", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
if _, err := b.Reserved.Reserve(ctx, "m", "0.2", []string{"m:prog"}, Source{Fingerprint: "src2:m"}); err != nil {
t.Fatal(err)
}
b.queueAt(t, v, PublishClassStart, b.now)
b.Drain(ctx, "oci")
if len(b.sent) != 1 || len(b.ended) != 0 {
t.Errorf("with a standing 0.2: sent %v, ended %v, want m %s sent, its series not superseded (H-A-293)", b.sent, b.ended, v.Version)
}
rel := &reserveMem{mem: newMem()}
raw, _ := json.Marshal(ReleasedRecord{Kept: []ReleasedBuild{{Fingerprint: "f", Version: "0.2.3"}, {Fingerprint: "g", Version: "0.1.9"}}})
_, _ = rel.Create(ctx, ReleasedKey("m"), raw)
b.Reserved.Releases = rel
b.sent = nil
b.queueAt(t, v, PublishClassStart, b.now)
b.Drain(ctx, "oci")
if len(b.sent) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("at the pass: sent %v, ended %v, want m %s retired by the release 0.1.9", b.sent, b.ended, v.Version)
}
b.ended = nil
b.queueAt(t, v, PublishClassStart, b.now)
b.Go = func(func()) {}
if err := b.Resume(ctx); err != nil {
t.Fatal(err)
}
if keys, _ := b.queue.KeysOf(ctx, QueueFilter); len(keys) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("at the start: keys %v, ended %v, want m %s retired", keys, b.ended, v.Version)
}
only := &reserveMem{mem: newMem()}
raw, _ = json.Marshal(ReleasedRecord{Kept: []ReleasedBuild{{Fingerprint: "f", Version: "0.2.0"}}})
_, _ = only.Create(ctx, ReleasedKey("n"), raw)
got, err := (&Reserver{Releases: only}).Retired(ctx, []ReservedVersion{{Module: "n", Version: "0.1.4", BuiltAt: b.now}, {Module: "n", Version: "0.1.6"},
{Module: "n", Version: "0.3.0", BuiltAt: b.now}, {Module: "p", Version: "0.1.2", BuiltAt: b.now}})
if err != nil || len(got) != 1 || !strings.Contains(got["n 0.1.4"], "reservation 0.1.6") {
t.Errorf("retired %v (%v), want only n 0.1.4, by the newer patch 0.1.6; n 0.3.0 and p 0.1.2 kept, a release of 0.2 retiring neither", got, err)
}
})
t.Run("a version taken over is not sent, and its item ended", func(t *testing.T) {
b := newRepublishBed(t)
v := b.reserve(t, "m")
b.queueAt(t, v, PublishClassStart, b.now)
if _, err := b.Reserved.Reserve(ctx, "m", "0.1", []string{"m:prog"}, Source{Fingerprint: "src2:other"}); err != nil {
t.Fatal(err)
}
b.Drain(ctx, "oci")
if len(b.sent) != 0 || fmt.Sprint(b.ended) != "[m "+v.Version+"]" {
t.Errorf("sent %v, ended %v: want nothing sent and the item ended", b.sent, b.ended)
}
})
}
// TestThePublishQueueToolAnswersAtAnySize: design 47 H H-A-290 (5) — builder_publish_queue over two thousand queued versions answers
// within a second and small: counts per holder and class with the oldest and newest queue time, a list capped at fifty in the
// order they are sent, and one module's item with its class; each state read in bulk where the reader can.
func TestThePublishQueueToolAnswersAtAnySize(t *testing.T) {
at := time.Date(2026, 11, 20, 17, 0, 0, 0, time.UTC)
vals := map[string]json.RawMessage{}
for i := range 2000 {
holder, class := "oci", PublishClassStart
if i%2 == 1 {
holder = "go"
}
if i >= 1990 {
class = PublishClassLive
}
q := QueuedPublish{V: 1, Holder: holder, Module: fmt.Sprintf("m%d", i), Version: "0.1.0", Artifacts: map[string]string{"a": "sha256:aa"}, Queued: at.Add(time.Duration(i) * time.Second), Class: class}
vals[QueueKey(holder, q.Module, q.Version)], _ = json.Marshal(q)
}
gets := 0
queue := Reader{All: func() (StatePage, error) { return StatePage{Values: vals}, nil },
Keys: func() ([]string, error) { return slices.Collect(maps.Keys(vals)), nil },
Get: func(k string) (json.RawMessage, bool, error) { gets++; v, ok := vals[k]; return v, ok, nil }}
none := Reader{Keys: func() ([]string, error) { return nil, nil }}
start := time.Now()
a, err := AnswerQueue(queue, none, none, "", "")
took := time.Since(start)
raw, _ := json.Marshal(a)
if err != nil || took > time.Second || a.Total != 2000 || len(a.Queue) != PublishQueueListCap || a.Listed != PublishQueueListCap || len(raw) > 64<<10 || gets != 0 {
t.Fatalf("answered %d of %d in %s, %d bytes, %d single reads (%v): want all counted, fifty listed, within a second, small, in bulk", len(a.Queue), a.Total, took, len(raw), gets, err)
}
if fmt.Sprintln(a.Counts[1].Holder, a.Counts[1].Class, a.Counts[1].Count, a.Counts[1].Oldest.Equal(at.Add(1991*time.Second)), a.Counts[1].Newest.Equal(at.Add(1999*time.Second))) != "go "+PublishClassLive+" 5 true true\n" || len(a.Counts) != 4 {
t.Errorf("counts %+v, want per holder and class with their oldest and newest", a.Counts)
}
if a.Queue[0].Class != PublishClassLive || a.Queue[0].Holder != "go" {
t.Errorf("the list opens with %+v, want the go holder's live class, sent first", a.Queue[0])
}
one, err := AnswerQueue(queue, none, none, "m1995", "")
if err != nil || len(one.Queue) != 1 || one.Queue[0].Class != PublishClassLive || one.Total != 2000 {
t.Errorf("one module's answer %+v (%v), want its item with its class", one.Queue, err)
}
if a.Capped || len(a.Read) != 0 {
t.Errorf("a state one page holds is answered capped: %+v", a.Read)
}
queue.All = func() (StatePage, error) { return StatePage{Values: vals, More: true, Keys: 5000, Bytes: 3 << 20}, nil }
if c, err := AnswerQueue(queue, none, none, "", ""); err != nil || !c.Capped || len(c.Read) != 1 || c.Read[0].State != PublishQueueState ||
c.Read[0].KeysRead != 2000 || c.Read[0].StateKeys != 5000 || c.Read[0].StateBytes != 3<<20 || c.Total != 2000 {
t.Errorf("a state past its one page answered capped %v, read %+v (%v): want capped, 2000 keys read of 5000, its size said (H-A-293)", c.Capped, c.Read, err)
}
queue.All = func() (StatePage, error) { return StatePage{}, errors.ErrUnsupported }
if b, err := AnswerQueue(queue, none, none, "", ""); err != nil || b.Total != 2000 || gets != 2000 {
t.Errorf("a reader with no bulk read answered %d (%v) with %d single reads, want every key read one by one", b.Total, err, gets)
}
}
@@ -54,7 +54,10 @@ type ReservedVersion struct {
BuiltAt time.Time `json:"built-at,omitzero"`
// Published are the artifacts a registry seat said published at this version with the built digest.
Published map[string]bool `json:"published,omitempty"`
At time.Time `json:"at"`
// PublishedAt is when the builder heard or confirmed each artifact's publish (H-A-331); absent for a mark written before it
// was kept.
PublishedAt map[string]time.Time `json:"published-at,omitempty"`
At time.Time `json:"at"`
// Repository and Commit are what the build was made from, and Tag the git tag its release is written as (a Go
// package's, GoTagOf): empty for a module with no Go package (design 47 G §370).
Repository string `json:"repository,omitempty"`
@@ -84,6 +87,17 @@ type ReservedVersion struct {
Withdrawn map[string]string `json:"withdrawn,omitempty"`
}
// markPublished marks artifact a published at at, the moment the builder heard or confirmed its publish (H-A-331).
func (v *ReservedVersion) markPublished(a string, at time.Time) {
if v.Published == nil {
v.Published = map[string]bool{}
}
if v.PublishedAt == nil {
v.PublishedAt = map[string]time.Time{}
}
v.Published[a], v.PublishedAt[a] = true, at.UTC()
}
// KeptPublished is a registry seat's published kept on a reservation, as heard (design 47 G A-415).
type KeptPublished struct {
Name string `json:"name"`
@@ -396,6 +410,8 @@ type ReleasedRecord struct {
type ReleasedBuild struct {
Fingerprint string `json:"fingerprint"`
Version string `json:"version"`
// Digests are the published digest of each artifact, by reference (H-A-331); absent for a release kept before.
Digests map[string]string `json:"digests,omitempty"`
}
// ReleasedFor says whether the builder saw a version released for the build of fingerprint among the module's last releases.
@@ -419,8 +435,54 @@ func (r *Reserver) ReleasedFor(ctx context.Context, module, fingerprint string)
return false, nil
}
// PublishedOf answers the seat mesh-builds' verb published (design 47 H-A-328): whether artifact of module built at fingerprint
// is published. A reservation of that fingerprint whose artifact a registry seat said published at the built digest (marked
// from the registry's record too, H-A-274), its number not withdrawn, answers its version, digest and, where kept, the
// publish time (H-A-331); a release of that fingerprint among the module's
// last releases answers published at its version, every artifact of a release being published; otherwise not published.
func (r *Reserver) PublishedOf(ctx context.Context, module, artifact, fingerprint string) (meshbuilds.PublishedAnswer, error) {
module, artifact, fingerprint = strings.TrimSpace(module), strings.TrimSpace(artifact), strings.TrimSpace(fingerprint)
if module == "" || artifact == "" || fingerprint == "" {
return meshbuilds.PublishedAnswer{}, errors.New("module, artifact and fingerprint are required")
}
keys, err := r.Store.KeysOf(ctx, NameFilter(module))
if err != nil {
return meshbuilds.PublishedAnswer{}, fmt.Errorf("the reservations of %s cannot be listed: %w", module, err)
}
sort.Strings(keys)
for _, k := range keys {
cur, _, found, err := r.get(ctx, k)
if err != nil {
return meshbuilds.PublishedAnswer{}, err
}
// A number withdrawn by a take-over is never answered published (H-A-331).
if found && cur.Module == module && cur.Fingerprint == fingerprint && cur.Published[artifact] && !hasKey(cur.Withdrawn, cur.Version) {
a := meshbuilds.PublishedAnswer{Published: true, Version: cur.Version, Digest: cur.Artifacts[artifact]}
if at, ok := cur.PublishedAt[artifact]; ok {
a.At = &at
}
return a, nil
}
}
if r.Releases != nil {
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(module))
if err != nil {
return meshbuilds.PublishedAnswer{}, fmt.Errorf("the releases of %s cannot be read: %w", module, err)
}
var rec ReleasedRecord
if found && json.Unmarshal(raw, &rec) == nil {
for i := len(rec.Kept) - 1; i >= 0; i-- {
if rec.Kept[i].Fingerprint == fingerprint {
return meshbuilds.PublishedAnswer{Published: true, Version: rec.Kept[i].Version, Digest: rec.Kept[i].Digests[artifact]}, nil
}
}
}
}
return meshbuilds.PublishedAnswer{Published: false}, nil
}
// keepRelease adds a release to the module's record, dropping the oldest past ReleasedKept.
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string) error {
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string, digests map[string]string) error {
n := r.ReleasedKept
if n < 1 {
n = DefaultReleasedKept
@@ -440,7 +502,7 @@ func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version
kept = append(kept, k)
}
}
kept = append(kept, ReleasedBuild{Fingerprint: fingerprint, Version: version})
kept = append(kept, ReleasedBuild{Fingerprint: fingerprint, Version: version, Digests: digests})
if len(kept) > n {
kept = kept[len(kept)-n:]
}
@@ -479,7 +541,7 @@ func (r *Reserver) Released(ctx context.Context, module, version string) ([]stri
continue
}
if cur.Version == version && cur.Fingerprint != "" && r.Releases != nil {
if err := r.keepRelease(ctx, module, cur.Fingerprint, version); err != nil {
if err := r.keepRelease(ctx, module, cur.Fingerprint, version, cur.Artifacts); err != nil {
return gone, fmt.Errorf("the release of %s %s cannot be recorded: %w", module, version, err)
}
}
@@ -695,11 +757,9 @@ func (r *Reserver) publishedOn(ctx context.Context, keys []string, name, version
}
matched = true
if !cur.Published[a] {
cur.Published = maps.Clone(cur.Published)
if cur.Published == nil {
cur.Published = map[string]bool{}
}
cur.Published[a], changed = true, true
cur.Published, cur.PublishedAt = maps.Clone(cur.Published), maps.Clone(cur.PublishedAt)
cur.markPublished(a, r.now())
changed = true
arts = append(arts, a)
}
}
@@ -768,10 +828,7 @@ func (r *Reserver) Kept(ctx context.Context, module, version string) ([]Marked,
}
matched = true
if !cur.Published[a] {
if cur.Published == nil {
cur.Published = map[string]bool{}
}
cur.Published[a] = true
cur.markPublished(a, r.now())
arts = append(arts, a)
}
}
@@ -221,6 +221,87 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
}
}
// TestPublishedAnswersByTheBuildFingerprint: design 47 H-A-328 — the verb published answers published for a reservation of the
// fingerprint whose artifact a registry's record marked published, with its version and digest, and for a release of it; an
// artifact not yet marked, and another fingerprint, are not published.
func TestPublishedAnswersByTheBuildFingerprint(t *testing.T) {
ctx := context.Background()
heard := time.Date(2026, 11, 21, 19, 0, 0, 0, time.UTC)
store := &reserveMem{mem: newMem()}
r := &Reserver{Store: store, Releases: &reserveMem{mem: newMem()}, ReleasedKept: 2, Filled: func() bool { return true },
ReleasedOf: func(context.Context, string) ([]string, error) { return nil, nil }, Now: func() time.Time { return heard }}
v, err := r.Reserve(ctx, "mesh-grants", "0.1", []string{"mesh-grants:busgrants", "mesh-grants:grantsstate"}, Source{Fingerprint: "src2:aa"})
if err != nil {
t.Fatal(err)
}
if _, err := r.Built(ctx, "mesh-grants", v.Version, "mesh-grants:busgrants", "sha256:bb", nil); err != nil {
t.Fatal(err)
}
ask := func(artifact, fp string) meshbuilds.PublishedAnswer {
t.Helper()
a, err := r.PublishedOf(ctx, "mesh-grants", artifact, fp)
if err != nil {
t.Fatal(err)
}
return a
}
if a := ask("mesh-grants:busgrants", "src2:aa"); a.Published {
t.Errorf("a built artifact no registry holds is answered %+v, want not published", a)
}
if marked, _, err := r.Published(ctx, "mesh-grants-busgrants", v.Version, "sha256:bb"); err != nil || len(marked) != 1 {
t.Fatalf("the registry's published marked %v (%v)", marked, err)
}
// H-A-331: the mark keeps when the publish was heard, answered as at.
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.Version != v.Version || a.Digest != "sha256:bb" || a.At == nil || !a.At.Equal(heard) {
t.Errorf("published answered %+v, want %s at sha256:bb, heard %s", a, v.Version, heard)
}
// A mark written before the publish time was kept answers no time, never an invented one; a withdrawn number is never
// answered published.
key := ReservedKey("mesh-grants", "0.1")
cur, rev, _, _ := r.get(ctx, key)
older := cur
older.PublishedAt = nil
raw, _ := json.Marshal(older)
if rev, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.At != nil {
t.Errorf("an older mark answered %+v, want published with no at", a)
}
withdrawn := older
withdrawn.Withdrawn = map[string]string{v.Version: "src2:aa"}
raw, _ = json.Marshal(withdrawn)
if rev, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:busgrants", "src2:aa"); a.Published {
t.Errorf("a withdrawn number is answered %+v, want not published", a)
}
raw, _ = json.Marshal(cur)
if _, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:grantsstate", "src2:aa"); a.Published {
t.Errorf("an artifact not marked is answered %+v", a)
}
if a := ask("mesh-grants:busgrants", "src2:other"); a.Published {
t.Errorf("another fingerprint is answered %+v", a)
}
if _, err := r.Released(ctx, "mesh-grants", v.Version); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:grantsstate", "src2:aa"); !a.Published || a.Version != v.Version {
t.Errorf("after its release the fingerprint's artifact is answered %+v, want published at %s", a, v.Version)
}
// H-A-331: the kept release keeps each published digest with its version.
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.Digest != "sha256:bb" {
t.Errorf("after its release busgrants is answered %+v, want its digest sha256:bb", a)
}
if _, err := r.PublishedOf(ctx, "mesh-grants", "", "src2:aa"); err == nil {
t.Error("an ask naming no artifact is answered")
}
}
// TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226, H-A-244 — a dropped number stays a gap, kept in
// the series key's Withdrawn while it is above the catalog's highest released; at each write those at or below it go, so the
// map never grows without end.
@@ -21,10 +21,59 @@ const (
OldFormClosedEvent = "version.old-form-closed"
)
// Reader reads the state as the tools bundle may: every key, and one.
// Reader reads the state as the tools bundle may: every key, and one; and, where All is set, the state's values in one page,
// one request for many (design 47 H H-A-290 (5), H-A-292).
type Reader struct {
Keys func() ([]string, error)
Get func(key string) (json.RawMessage, bool, error)
// All is one page of the state's values; nil, or an error satisfying errors.Is(err, errors.ErrUnsupported), reads key by key.
All func() (StatePage, error)
}
// StatePage is one page of a state (H-A-292, H-A-293): its values, and, when the state holds more than the page, More with the
// state's size as its info says it (Keys, Bytes; InfoWords when the info cannot be read). No second page is asked.
type StatePage struct {
Values map[string]json.RawMessage
More bool
Keys, Bytes uint64
InfoWords string
}
// StateCapped is what an answer says of a state it read only the first page of (H-A-293): how many keys it read, and the
// state's size from its info.
type StateCapped struct {
State string `json:"state"`
KeysRead int `json:"keys-read"`
StateKeys uint64 `json:"state-keys"`
StateBytes uint64 `json:"state-bytes"`
InfoWords string `json:"info,omitempty"`
}
// values is the state's values: one page where the reader can (capped says what a page past the state's end left out), else
// key by key.
func (r Reader) values(state string) (map[string]json.RawMessage, *StateCapped, error) {
if r.All != nil {
pg, err := r.All()
if !errors.Is(err, errors.ErrUnsupported) {
if err != nil || !pg.More {
return pg.Values, nil, err
}
return pg.Values, &StateCapped{State: state, KeysRead: len(pg.Values), StateKeys: pg.Keys, StateBytes: pg.Bytes, InfoWords: pg.InfoWords}, nil
}
}
keys, err := r.Keys()
if err != nil {
return nil, nil, err
}
out := make(map[string]json.RawMessage, len(keys))
for _, k := range keys {
raw, ok, err := r.Get(k)
if err != nil || !ok {
continue
}
out[k] = raw
}
return out, nil, nil
}
// readStore is a Store over a Reader that refuses every write: the tools bundle only reads versions.
@@ -198,9 +198,9 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if len(m.Logs) != 1 || m.Logs[0].Name != builds.BuilderLog || m.Logs[0].MaxAgeDays != builds.BuilderLogDays || m.Logs[0].MaxMiB != builds.BuilderLogMiB {
t.Errorf("logs = %+v, want %s kept %d days in %d MiB", m.Logs, builds.BuilderLog, builds.BuilderLogDays, builds.BuilderLogMiB)
}
// Design 47 G §52, §369: the seat mesh-builds, held once for the mesh, serving reserved.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-builds" || m.Claims[0].Scope != "mesh" || !reflect.DeepEqual(m.Claims[0].Serves, []string{"reserved"}) {
t.Errorf("claims = %+v, want mesh-builds, scope mesh, serving reserved", m.Claims)
// Design 47 G §52, §369, H-A-328: the seat mesh-builds, held once for the mesh, serving reserved and published.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-builds" || m.Claims[0].Scope != "mesh" || !reflect.DeepEqual(m.Claims[0].Serves, []string{"reserved", "published"}) {
t.Errorf("claims = %+v, want mesh-builds, scope mesh, serving reserved and published", m.Claims)
}
// A-115: the process answers the verb itself and says so in `serves` beside its process, with no `bus`; the bundle serves no tool named so.
var resources []map[string]any
@@ -208,8 +208,8 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
t.Fatal(err)
}
for _, r := range resources {
if r["type"] == "process" && !reflect.DeepEqual(r["serves"], []any{"seat:mesh-builds.reserved"}) {
t.Errorf("the process serves %v, want seat:mesh-builds.reserved (A-115)", r["serves"])
if r["type"] == "process" && !reflect.DeepEqual(r["serves"], []any{"seat:mesh-builds.reserved", "seat:mesh-builds.published"}) {
t.Errorf("the process serves %v, want seat:mesh-builds.reserved and published (A-115, H-A-328)", r["serves"])
}
}
if slices.Contains(m.Tools, "reserved") {
+4 -2
View File
@@ -7,7 +7,8 @@
"name": "mesh-builds",
"scope": "mesh",
"serves": [
"reserved"
"reserved",
"published"
]
}
],
@@ -293,7 +294,8 @@
"name": "mesh-delivery-builder",
"artifact": "program",
"serves": [
"seat:mesh-builds.reserved"
"seat:mesh-builds.reserved",
"seat:mesh-builds.published"
],
"run": [
"./mesh-delivery-builder"
@@ -9,7 +9,8 @@ import "time"
const Seat = "mesh-builds"
const (
VerbReserved = "reserved"
VerbReserved = "reserved"
VerbPublished = "published"
)
const (
@@ -24,7 +25,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string {
return []string{VerbReserved}
return []string{VerbReserved, VerbPublished}
}
// Events is every event published on the seat's subject: a new copy each call.
@@ -217,6 +218,28 @@ type ReservedAnswer struct {
Version string `json:"version,omitempty"`
}
// PublishedInput is what published takes.
type PublishedInput struct {
// the package's artifact reference, <module>:<artifact>
Artifact string `json:"artifact"`
// the build fingerprint the package is built at, which the answer is matched by
Fingerprint string `json:"fingerprint"`
// the module that builds the package
Module string `json:"module"`
}
// PublishedAnswer is what published answers.
type PublishedAnswer struct {
// when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331)
At *time.Time `json:"at,omitempty"`
// the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331)
Digest string `json:"digest,omitempty"`
// true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331)
Published bool `json:"published"`
// the version it is published at, MAJOR.MINOR.PATCH, when published
Version string `json:"version,omitempty"`
}
// StepAccept is the body of step.
type StepAccept = StepRun
@@ -35,6 +35,12 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v ReservedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "reserved answers an error alone, naming the name or the version", subject: "", given: "{\"error\":\"no entry carries the name example.org/none\"}", wire: "{\"error\":\"no entry carries the name example.org/none\"}",
decode: func(b []byte) (any, error) { var v ReservedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published asks an artifact at its build fingerprint", subject: "", given: "{\"artifact\":\"mesh-grants:busgrants\",\"fingerprint\":\"src2:aa\",\"module\":\"mesh-grants\"}", wire: "{\"artifact\":\"mesh-grants:busgrants\",\"fingerprint\":\"src2:aa\",\"module\":\"mesh-grants\"}",
decode: func(b []byte) (any, error) { var v PublishedInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published answers the version, digest and time of a published artifact", subject: "", given: "{\"at\":\"2026-11-21T19:00:00Z\",\"digest\":\"sha256:aa\",\"published\":true,\"version\":\"0.1.4\"}", wire: "{\"at\":\"2026-11-21T19:00:00Z\",\"digest\":\"sha256:aa\",\"published\":true,\"version\":\"0.1.4\"}",
decode: func(b []byte) (any, error) { var v PublishedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published answers an artifact not published", subject: "", given: "{\"published\":false}", wire: "{\"published\":false}",
decode: func(b []byte) (any, error) { var v PublishedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
}
for _, c := range cases {
v, err := c.decode([]byte(c.given))
@@ -52,5 +52,31 @@
"given": {
"error": "no entry carries the name example.org/none"
}
},
{
"name": "published asks an artifact at its build fingerprint",
"of": "verb:published.input",
"given": {
"module": "mesh-grants",
"artifact": "mesh-grants:busgrants",
"fingerprint": "src2:aa"
}
},
{
"name": "published answers the version, digest and time of a published artifact",
"of": "verb:published.answer",
"given": {
"published": true,
"version": "0.1.4",
"digest": "sha256:aa",
"at": "2026-11-21T19:00:00Z"
}
},
{
"name": "published answers an artifact not published",
"of": "verb:published.answer",
"given": {
"published": false
}
}
]
@@ -70,6 +70,58 @@
},
"description": "the reservation or release, or the error; module, version and artifacts are present whenever error is not (design 47 G 679)"
}
},
{
"name": "published",
"class": "read",
"description": "Whether an artifact built at a build fingerprint is published, from the builder's reservations and the registries' record, for the pipeline runner writing a mover's wait on a package: a package published before the wait was written is read, not waited for (design 47 H-A-326, H-A-328, H-A-274).",
"input": {
"type": "object",
"properties": {
"module": {
"type": "string",
"description": "the module that builds the package"
},
"artifact": {
"type": "string",
"description": "the package's artifact reference, <module>:<artifact>"
},
"fingerprint": {
"type": "string",
"description": "the build fingerprint the package is built at, which the answer is matched by"
}
},
"required": [
"module",
"artifact",
"fingerprint"
]
},
"answer": {
"type": "object",
"properties": {
"published": {
"type": "boolean",
"description": "true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331)"
},
"version": {
"type": "string",
"description": "the version it is published at, MAJOR.MINOR.PATCH, when published"
},
"digest": {
"type": "string",
"description": "the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331)"
},
"at": {
"type": "string",
"format": "date-time",
"description": "when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331)"
}
},
"required": [
"published"
]
}
}
],
"accepts": [
@@ -6,6 +6,7 @@ export declare const seat: "mesh-builds";
export declare const verbs: {
readonly reserved: "reserved";
readonly published: "published";
};
export declare const accepts: {
@@ -171,6 +172,28 @@ export interface ReservedAnswer {
"version"?: string;
}
/** is what published takes. */
export interface PublishedInput {
/** the package's artifact reference, <module>:<artifact> */
"artifact": string;
/** the build fingerprint the package is built at, which the answer is matched by */
"fingerprint": string;
/** the module that builds the package */
"module": string;
}
/** is what published answers. */
export interface PublishedAnswer {
/** when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331) */
"at"?: string;
/** the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331) */
"digest"?: string;
/** true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331) */
"published": boolean;
/** the version it is published at, MAJOR.MINOR.PATCH, when published */
"version"?: string;
}
/** is the body of step. */
export type StepAccept = StepRun;
@@ -6,6 +6,7 @@ export const seat = "mesh-builds";
export const verbs = {
reserved: "reserved",
published: "published",
};
export const accepts = {
@@ -7,6 +7,7 @@ import (
"encoding/json"
"fmt"
"os"
"sync"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
@@ -98,13 +99,29 @@ func FirstViewTools(module string, st stdio.KeptState, acts views.HandActs, now
}
// HealthTool is the runner's health check, run outside its process (design 47 G §76, §468, §495; A-164): it reads the
// runner's state pipelines as show does and the ages its process writes in the state consumers, and asks no seat verb.
func HealthTool(pipelines, consumers pipeline.Kept, now func() time.Time) stdio.Tool {
// runner's state pipelines as show does and the ages its process writes in the state consumers, and asks no seat verb. A
// running pipeline it first names stalled is said in this program's journal through logf, once per key for the
// program's life (H-A-311 (3), H-A-313).
func HealthTool(pipelines, consumers pipeline.Kept, now func() time.Time, logf func(string, ...any)) stdio.Tool {
var mu sync.Mutex
said := map[string]bool{}
return stdio.Tool{Name: Prefix + "_health",
Description: "The pipeline runner's health: healthy when it answers show and each of its durable consumers' oldest " +
"unacknowledged message waits less than five minutes; otherwise it names each consumer behind.",
Description: "The pipeline runner's health: healthy when it answers show, each of its durable consumers' oldest " +
"unacknowledged message waits less than five minutes and no running pipeline has had nothing open and nothing " +
"newly placed for five minutes; otherwise it names each consumer behind and each such pipeline by its key.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return pipeline.Health(pipelines, consumers, now()), nil }}
Run: func(map[string]any) (any, error) {
a, stalled := pipeline.HealthStalled(pipelines, consumers, now())
mu.Lock()
defer mu.Unlock()
for _, key := range stalled {
if !said[key] && logf != nil {
said[key] = true
logf("the running pipeline %s has had nothing open and nothing newly placed for %s or more: runner_health names it unhealthy (H-A-311)", key, pipeline.StalledAfter)
}
}
return a, nil
}}
}
// StoresFill is each of the runner's stores' fill through the bundle's calls (design 47 G §366, §623, A-214): the states
@@ -154,7 +171,8 @@ func main() {
tools = append(tools, CancelTool(r.List, stdio.Ask), RerunTool(stdio.Ask))
tools = append(tools, VerbTools(stdio.Ask)...)
tools = append(tools, TestRunsTools(stdio.Ask)...)
tools = append(tools, HealthTool(r.Pipelines, kept{stdio.State(pipeline.StateConsumers)}, time.Now))
tools = append(tools, HealthTool(r.Pipelines, kept{stdio.State(pipeline.StateConsumers)}, time.Now,
func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-pipeline-tools] "+f+"\n", a...) }))
tools = append(tools, WaitingTool(kept{stdio.State(seat.StateQueue)}))
if err := stdio.Serve("", tools); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-pipeline-tools] %v\n", err)
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
@@ -37,7 +38,7 @@ func TestTools(t *testing.T) {
names = append(names, tl.Name)
}
// RN25 (§468, §76; A-164): the health check is the bundle's runner_health.
names = append(names, HealthTool(none{}, none{}, nil).Name, WaitingTool(none{}).Name)
names = append(names, HealthTool(none{}, none{}, nil, nil).Name, WaitingTool(none{}).Name)
// §606, §549: the unlogged entries are listed and appended by hand.
for _, tl := range UnloggedTools(stdio.State("pipelines"), stdio.Log("step-log"), nil, nil) {
names = append(names, tl.Name)
@@ -53,6 +54,22 @@ func TestTools(t *testing.T) {
if got, _ := json.Marshal(out); err != nil || string(got) != `[{"sequence":1,"pipeline":"main.5"},{"sequence":3,"pipeline":"main.7"}]` {
t.Errorf("runner_waiting = %s (%v)", got, err)
}
// H-A-311 (3), H-A-313: runner_health names a running pipeline quiet five minutes, and this program's journal says it
// once per key, however often the health is asked.
now := time.Date(2026, 11, 21, 20, 35, 0, 0, time.UTC)
quiet := queueOf{"main.137": `{"pipeline":"main.137","branch":"main","outcome":"running","source":137,"started":"2026-11-21T20:15:00Z",` +
`"movers":[{"module":"app"}],"steps":[{"module":"app","step":"build","outcome":"passed","status":"required","started":"2026-11-21T20:15:52Z","ended":"2026-11-21T20:17:55Z"}]}`}
var journal []string
health := HealthTool(quiet, none{}, func() time.Time { return now }, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
for range 2 {
out, err := health.Run(nil)
if got, _ := json.Marshal(out); err != nil || !strings.Contains(string(got), "main.137 (nothing open and nothing placed for 17m5s)") {
t.Errorf("runner_health = %s (%v), want main.137 named stalled", got, err)
}
}
if len(journal) != 1 || !strings.Contains(journal[0], "main.137") {
t.Errorf("the journal said %q, want main.137 said once", journal)
}
}
// queueOf is a state queue in memory, its keys in no order.
@@ -150,7 +150,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
logger.Printf("widens: each read bound %s, %d catalog revisions kept", readBound, revisionsKept)
widener := widenerOf(calls.Call, readBound, revisionsKept, logger.Printf)
hearer.Widens = widener.Widens
hearer.Widens = widener.Reads
hearer.LogLast = stepLog{ctx: ctx, js: js, module: cred.Module}.Last
hearer.Where = routerRecordOf(cred.Module, calls.Call, calls.Ask)
hearer.Failures = &pipeline.Failures{Findings: rv.Findings, Observer: cred.Module, Author: commitAuthor(calls.Call), Logf: logger.Printf}
@@ -179,6 +179,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
starter.Supersede = func(newer seat.Pipeline) { hearer.SupersedeRunning(newer) }
starter.Left = hearer.LeftInPlace // a same-source re-pin's superseded steps leave their runs (H-A-284)
starter.Revision = widener.Revision // a kept mover is placed from its kept revision (H-A-286)
// A mover's wait on a package is read first: a package published before the wait places it at once (H-A-326, H-A-328).
starter.AskPublished = publishedOf(boundedCalls(calls.Call, readBound, time.After, logger.Printf))
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.AskPinHeld = func(key, module string, pins map[string]string) {
if _, err := starter.AskPinHeld(hearer.Store, key, module, pins, widener.Revision); err != nil {
@@ -274,6 +276,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := placeLostRuns(starter.UnplacedRuns, calls.CallKept, placeHalf, logger.Printf); err != nil {
logger.Printf("the runs not placed at the start: %v", err)
}
// A running pipeline nothing of which is still open ends now, its turn freed, before the queue is placed (H-A-288 (c)).
if n, err := hearer.EndAtStart(); err != nil || n > 0 {
logger.Printf("the pipelines with nothing open at the start: %d ended (%v)", n, err)
}
// The queue is placed at the start from its own state (§748).
if n, err := starter.StartQueue(rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the queue at the start: %d waiting pipelines placed (%v)", n, err)
@@ -0,0 +1,29 @@
package main
import (
"encoding/json"
"fmt"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// publishedOf asks the seat mesh-builds' verb published through call whether a package built at a fingerprint is published
// (design 47 H-A-326 (1), H-A-328), asked when a mover's wait on it is written.
func publishedOf(call func(string, any) (json.RawMessage, error)) func(builds.PublishedInput) (builds.PublishedAnswer, error) {
return func(in builds.PublishedInput) (builds.PublishedAnswer, error) {
raw, err := call("seat:"+builds.Seat+"."+builds.VerbPublished, in)
if err != nil {
return builds.PublishedAnswer{}, err
}
answer, said, ok := buscalls.Unwrap(raw)
if !ok {
return builds.PublishedAnswer{}, fmt.Errorf("%s answered: %s", builds.VerbPublished, said)
}
var a builds.PublishedAnswer
if err := json.Unmarshal(answer, &a); err != nil {
return builds.PublishedAnswer{}, fmt.Errorf("%s answered what does not read: %w", builds.VerbPublished, err)
}
return a, nil
}
}
@@ -206,6 +206,7 @@ func (h *Hearer) OnDecided(event oc.Warrant) (seat.Pipeline, error) {
return seat.Pipeline{}, err
}
for _, key := range found {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
@@ -219,7 +220,7 @@ func (h *Hearer) OnDecided(event oc.Warrant) (seat.Pipeline, error) {
if err != nil || !changed {
return p, err
}
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
i := stepOfAsk(p, w.Ask)
if i >= 0 {
p.Unlogged = append(p.Unlogged, endEntry(key, p.Steps[i], now))
@@ -176,7 +176,7 @@ func (v Verbs) AskUnanswered(onSeat, ask string, again bool) error {
return nil // its second bound ends the step
}
v.Starter.carry(key, []Asked{*a})
return askAll(v.Starter.Steps, key, []Asked{*a}, func(commit string) bool { return v.Starter.recovered(key, commit) }, nil)
return askAll(v.Starter.Steps, key, []Asked{*a}, func(commit string) bool { return v.Starter.recovered(key, commit) }, nil, nil)
}
// HolderStarted puts again every open ask of each silent seat module holds (H-A-243 (1)): its holder takes its asks on a plain
@@ -234,7 +234,7 @@ func (v Verbs) HolderStarted(module string) (int, error) {
continue
}
st.carry(key, due)
if err := askAll(st.Steps, key, due, func(commit string) bool { return st.recovered(key, commit) }, nil); err != nil && first == nil {
if err := askAll(st.Steps, key, due, func(commit string) bool { return st.recovered(key, commit) }, nil, nil); err != nil && first == nil {
first = err
}
asked += len(due)
@@ -308,6 +308,7 @@ func markRerun(a *Asked, caller string) {
// other mover's validate reading the ask with it, a composition its validate held ended not asked, the pipeline ended when
// nothing else is open, said and logged as any step's end.
func (h *Hearer) endUnanswered(key, ask, why string) error {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
@@ -331,7 +332,7 @@ func (h *Hearer) endUnanswered(key, ask, why string) error {
if err != nil {
return err
}
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
@@ -1,6 +1,7 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
@@ -20,7 +21,7 @@ func TestAnAskNobodyAnswersIsAskedOnceMoreThenEndsItsStep(t *testing.T) {
v, store, said, asks, _, _ := verbsFixture()
var armed, unbound, raised, cleared []string
logs := []string{}
v.Hearer.Logf = func(f string, a ...any) { logs = append(logs, f) }
v.Hearer.Logf = func(f string, a ...any) { logs = append(logs, fmt.Sprintf(f, a...)) }
c := &AskClock{Bound: 10 * time.Minute,
Arm: func(key string, ttl time.Duration) error { armed = append(armed, key+" "+ttl.String()); return nil },
Unbound: func(key string) error { unbound = append(unbound, key); return nil },
@@ -83,6 +84,10 @@ func TestAnAskNobodyAnswersIsAskedOnceMoreThenEndsItsStep(t *testing.T) {
if s.Outcome != seat.StepResultOutcomeCouldNotRun || s.Why != "its ask "+ask+" was not answered" || s.Ended == nil {
t.Errorf("the step = %+v", s)
}
// The unanswered-bound end is one journal line, as every said end (H-A-320 (1), H-A-333).
if n := countLines(logs, pkey+" app lint ended could not run (no node, "); n != 1 {
t.Errorf("the unanswered end journaled %d times in %q, want once", n, logs)
}
if got.Outcome != seat.PipelineOutcomeFailed || len(asks.ids) != 2 {
t.Errorf("the pipeline is %s (%s), asked %d, want failed and nothing more asked", got.Outcome, got.Why, len(asks.ids))
}
@@ -7,6 +7,7 @@ import (
"strings"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
)
@@ -19,13 +20,83 @@ const StateConsumers = process.ConsumersState
// behind (§76).
const BehindAfter = 5 * time.Minute
// Health is the runner's health as its tool runner_health answers it (§468, §76, §495; A-164): healthy when its state
// pipelines answers, as show reads it, and each durable consumer's oldest unacknowledged message waits less than
// BehindAfter; otherwise unhealthy naming each consumer behind and how long.
func Health(pipelines, consumers Kept, now time.Time) health.Answer {
if _, err := pipelines.Keys(); err != nil {
return health.Unhealthy(fmt.Sprintf("The runner does not answer show: its state pipelines cannot be listed (%v).", err))
// StalledAfter is how long a running pipeline may have nothing open and nothing newly placed before the runner's health
// names it stalled (H-A-311 (3)): the mesh's own rule for nothing moving (nothing-moved-in-five-minutes), not a setting
// (H-A-322).
const StalledAfter = 5 * time.Minute
// Stalled is the key of each running pipeline, as show reads it, that has had nothing open (no step asked and not ended,
// no required step waiting but on a package's publish) and nothing newly placed, started or ended for StalledAfter at now (H-A-311 (3)), each with
// how long, in key order.
func Stalled(pipelines Kept, now time.Time) ([]string, error) {
all, err := Reads{Pipelines: pipelines}.pipelines()
if err != nil {
return nil, err
}
var out []string
for _, p := range all {
if p.Outcome != seat.PipelineOutcomeRunning {
continue
}
last, busy := startedOf(p), false
for _, st := range p.Steps {
// A required step waiting (an approval, a composition on its validate) is open, as the end's judgement holds it;
// an optional one waiting holds nothing (A-104), and neither does one waiting on a package's publish, as a mover
// with no steps waiting on one does not (H-A-326 (2)).
if open(st) || (st.Outcome == seat.StepResultOutcomeWaiting && st.Status == seat.StepResultStatusRequired && !pinHeld(st)) {
busy = true
break
}
for _, at := range []*time.Time{st.Started, st.Ended} {
if at != nil && at.After(last) {
last = *at
}
}
}
if busy || last.IsZero() || now.Sub(last) < StalledAfter {
continue
}
out = append(out, fmt.Sprintf("%s (nothing open and nothing placed for %s)", p.Pipeline, now.Sub(last).Round(time.Second)))
}
return out, nil
}
// Health is the runner's health as its tool runner_health answers it (§468, §76, §495; A-164): healthy when its state
// pipelines answers, as show reads it, each durable consumer's oldest unacknowledged message waits less than
// BehindAfter, and no running pipeline is stalled (H-A-311 (3)); otherwise unhealthy naming each consumer behind and
// how long, and each stalled pipeline by its key.
func Health(pipelines, consumers Kept, now time.Time) health.Answer {
a, _ := HealthStalled(pipelines, consumers, now)
return a
}
// HealthStalled is Health with the keys of the running pipelines it names stalled, so the tools program says each in its
// journal once (H-A-313).
func HealthStalled(pipelines, consumers Kept, now time.Time) (health.Answer, []string) {
if _, err := pipelines.Keys(); err != nil {
return health.Unhealthy(fmt.Sprintf("The runner does not answer show: its state pipelines cannot be listed (%v).", err)), nil
}
stalled, err := Stalled(pipelines, now)
if err != nil {
return health.Unhealthy(fmt.Sprintf("The runner's running pipelines cannot be read as show reads them (%v).", err)), nil
}
var keys []string
for _, s := range stalled {
keys = append(keys, strings.SplitN(s, " ", 2)[0])
}
a := consumersHealth(consumers, now)
if len(stalled) == 0 {
return a, nil
}
why := fmt.Sprintf("The runner's running pipelines with nothing open and nothing newly placed for %s or more: %s.", StalledAfter, strings.Join(stalled, ", "))
if !a.Healthy {
why = a.Why + " " + why
}
return health.Unhealthy(why), keys
}
// consumersHealth is the part of Health its consumers' ages give.
func consumersHealth(consumers Kept, now time.Time) health.Answer {
keys, err := consumers.Keys()
if err != nil {
return health.Unhealthy(fmt.Sprintf("The ages of the runner's consumers cannot be read from its state %s (%v).", StateConsumers, err))
@@ -7,6 +7,7 @@ import (
"testing"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/process"
)
@@ -50,6 +51,19 @@ func TestHealth(t *testing.T) {
{"six minutes names the consumer", mapKept{}, age("n_runner", 6*time.Minute), false, "n_runner (its oldest unacknowledged message waits 6m0s)"},
{"show does not answer", failingKept{}, age("n_runner", 0), false, "does not answer show"},
{"no event heard yet", mapKept{}, mapKept{}, true, "has heard no event yet"},
// H-A-311 (3): a running pipeline with nothing open and nothing newly placed for five minutes is named by its key.
{"a pipeline quiet six minutes is named", piped(now, "main.137", -6*time.Minute, seat.StepResultOutcomePassed), age("n_runner", 0), false,
"main.137 (nothing open and nothing placed for 6m0s)"},
{"a pipeline quiet four minutes is not", piped(now, "main.137", -4*time.Minute, seat.StepResultOutcomePassed), age("n_runner", 0), true, ""},
{"a pipeline with a step open is not", piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeRunning), age("n_runner", 0), true, ""},
// H-A-285, H-A-309: a record carrying the runner's kept-revisions is read, not refused.
{"a quiet pipeline keeping revisions is named", kept(piped(now, "main.131", -6*time.Minute, seat.StepResultOutcomePassed), "main.131"), age("n_runner", 0), false,
"main.131 (nothing open and nothing placed for 6m0s)"},
// A required step waiting (an approval on the operator) is open, as the end's judgement holds it.
{"a pipeline whose approval waits is not", piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeWaiting), age("n_runner", 0), true, ""},
// H-A-326 (2): a pipeline whose only movers wait on packages (a step held on a publish, a mover with no steps) is named.
{"a pipeline waiting only on packages is named", heldOnPublish(piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeWaiting), "main.137"), age("n_runner", 0), false,
"main.137 (nothing open and nothing placed for 20m0s)"},
}
for _, r := range rows {
a := Health(r.pipelines, r.consumers, now)
@@ -58,3 +72,41 @@ func TestHealth(t *testing.T) {
}
}
}
// piped is the state pipelines holding one running pipeline key whose one step was placed and last moved at now+moved,
// with outcome.
func piped(now time.Time, key string, moved time.Duration, outcome string) mapKept {
at, started := now.Add(moved), now.Add(-time.Hour)
st := seat.StepResult{Module: "app", Step: seat.StepResultStepBuild, Outcome: outcome, Status: seat.StepResultStatusRequired, Started: &at}
if outcome == seat.StepResultOutcomeWaiting {
st.Step, st.On = seat.StepResultStepApproval, seat.WaitOnPerson
} else if outcome != seat.StepResultOutcomeRunning {
st.Ended = &at
}
raw, _ := json.Marshal(seat.Pipeline{Pipeline: key, Branch: "main", Outcome: seat.PipelineOutcomeRunning, Started: &started,
Movers: []seat.Mover{{Module: "app"}}, Steps: []seat.StepResult{st}})
return mapKept{key: raw}
}
// kept is m with key's record carrying the runner's kept-revisions (H-A-285).
func kept(m mapKept, key string) mapKept {
raw, err := withKept(m[key], map[string]int64{"app": 130})
if err != nil {
panic(err)
}
m[key] = raw
return m
}
// heldOnPublish is m with key's one step held on a package's publish (H-A-206) and a second mover with no steps, waiting on one.
func heldOnPublish(m mapKept, key string) mapKept {
var p seat.Pipeline
if err := json.Unmarshal(m[key], &p); err != nil {
panic(err)
}
p.Steps[0].Step, p.Steps[0].On, p.Steps[0].Why = seat.StepResultStepBuild, "", PinWait+"mesh-grants:busgrants"
p.Movers = append(p.Movers, seat.Mover{Module: "nats"})
raw, _ := json.Marshal(p)
m[key] = raw
return m
}
@@ -1,8 +1,10 @@
package pipeline
import (
"errors"
"strings"
"testing"
"time"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -108,8 +110,11 @@ func TestThePipelinePassesOnNothingToRunAndFailsOnOneMover(t *testing.T) {
// R8 (design 47 H §746, H-A-52, H-A-62): a version-released writes its version into the mover of that module in the pipeline
// of the source revision built, on the branch it was released for, so pipeline-ended names it; another branch's pipeline,
// and a version heard again, write nothing.
//
// H-A-270 (2), H-A-277 (2), (3): a version written into a pipeline that ended passed says its pipeline-ended again, naming the
// version; one still running, or ended failed, says nothing. A say that failed is said when the version is heard again.
func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
h, store, _, _ := twoMovers(t)
h, store, bodies, order := twoMovers(t)
other := store[pkey]
other.p.Pipeline, other.p.Branch, other.p.Movers = "beta.5", "beta", append([]seat.Mover(nil), other.p.Movers...)
store["beta.5"] = other
@@ -120,9 +125,29 @@ func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
again.p.Pipeline, again.p.Source, again.p.Movers = key, source, append([]seat.Mover(nil), again.p.Movers...)
store[key] = again
}
at := time.Unix(1, 0)
for key, outcome := range map[string]string{pkey: seat.PipelineOutcomePassed, "beta.5": seat.PipelineOutcomeFailed} {
done := store[key]
done.p.Ended, done.p.Outcome = &at, outcome
store[key] = done
}
say := h.Say
fail := true
h.Say = func(subject string, body any) error {
if fail {
fail = false
return errors.New("the bus is away")
}
return say(subject, body)
}
ev := catalog.VersionReleasedEvent{Module: "app", Version: "0.4.1", Source: 5}
if n, err := h.Released(ev, "main"); n != 2 || err != nil {
t.Fatalf("released wrote %d (%v), want app's mover of main.5 and main.8", n, err)
first, err := h.Released(ev, "main")
if err == nil || len(*order) != 0 {
t.Fatalf("released wrote %d (%v) and said %v, want the failed say returned", first, err, *order)
}
// Heard again (the nak's redelivery), main.5's version is held already and its end is said (H-A-277 (3)).
if n, err := h.Released(ev, "main"); first+n != 2 || err != nil {
t.Fatalf("released wrote %d, then %d again (%v), want app's mover of main.5 and main.8", first, n, err)
}
if m, _ := MoverOf(store["main.6"].p, "app"); m.Version != "" {
t.Errorf("main.6's mover = %+v, want no version: another source", m)
@@ -133,8 +158,19 @@ func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
if m, _ := MoverOf(store["beta.5"].p, "app"); m.Version != "" {
t.Errorf("beta's mover = %+v, want no version: released for main", m)
}
if n, _ := h.Released(ev, "main"); n != 0 {
t.Errorf("the version heard again wrote %d", n)
pe, ok := bodies["pipeline-ended.main.5"].(seat.PipelineEnded)
if !ok || len(pe.Movers) != 2 || pe.Movers[1].Module != "app" || pe.Movers[1].Version != "0.4.1" {
t.Errorf("pipeline-ended.main.5 = %+v, want said again naming app at 0.4.1 (said %v)", bodies["pipeline-ended.main.5"], *order)
}
if len(*order) != 1 {
t.Errorf("said %v, want only main.5's end again: main.8 has not ended", *order)
}
if n, _ := h.Released(ev, "main"); n != 0 || len(*order) != 2 {
t.Errorf("the version heard once more wrote %d and said %v, want main.5's end said again", n, *order)
}
// beta's failed pipeline, its version written, is never said again (H-A-277 (2)).
if n, _ := h.Released(ev, ""); n != 1 || len(*order) != 3 {
t.Errorf("released for every branch wrote %d and said %v, want beta's mover written and main.5's end alone said", n, *order)
}
if got := pipelineEnded(store[pkey].p).Movers[1]; got.Module != "app" || got.Version != "0.4.1" {
t.Errorf("pipeline-ended names %+v, want app at 0.4.1", got)
@@ -0,0 +1,129 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/identity-and-access/go/seats/meshgrants"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
)
// refusedOnce is a store whose first write of a pipeline is refused as changed, another write having come between (the
// pipeline's key moving while its end is judged, as main.117's did).
type refusedOnce struct {
memStore
refused map[string]bool
}
func (r *refusedOnce) Update(key string, p seat.Pipeline, rev uint64) (uint64, error) {
if !r.refused[key] {
r.refused[key] = true
e := r.memStore[key]
e.rev++
r.memStore[key] = e
return 0, ErrChanged
}
return r.memStore.Update(key, p, rev)
}
// H-A-288, H-A-289 (met with main.117 and main.76): a pipeline ends, and the next waiting is placed, the moment nothing of it is
// still open: when its last mover leaves, when its last open step ends by the silent bound, and at the runner's start pass;
// its outcome is said in the journal. The reads its end needs are made once, before the compare-and-set, shared by every mover
// and every retry of the write: a write refused once is written at the retry without asking grants-for again.
func TestAPipelineWithNothingOpenEndsAtOnce(t *testing.T) {
t0 := time.Date(2026, 11, 21, 17, 45, 19, 0, time.UTC)
movers := []seat.Mover{{Module: "a", Commit: "c1", Fingerprint: "src2:a1"}, {Module: "b", Commit: "c1", Fingerprint: "src2:b1"},
{Module: "c", Commit: "c1", Fingerprint: "src2:c1"}}
step := func(module, outcome, ask string) seat.StepResult {
return seat.StepResult{Module: module, Step: seat.StepResultStepBuild, Status: seat.StepResultStatusRequired, Outcome: outcome, Ask: ask, Seat: builds.Seat}
}
rows := []struct {
name string
steps []seat.StepResult
end func(t *testing.T, h *Hearer)
outcome string
}{
{"its last mover leaves for a newer pipeline (H-A-223)",
[]seat.StepResult{step("a", seat.StepResultOutcomeRunning, "x1"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
h.SupersedeRunning(seat.Pipeline{Pipeline: "main.120", Branch: "main", Revision: 120, Movers: []seat.Mover{
{Module: "a", Commit: "c2", Fingerprint: "src2:a2"}, {Module: "b", Commit: "c2", Fingerprint: "src2:b2"}, {Module: "c", Commit: "c2", Fingerprint: "src2:c2"}}})
}, seat.PipelineOutcomeSuperseded},
{"its last open step ends could not run on the silent bound (H-A-243)",
[]seat.StepResult{step("a", seat.StepResultOutcomeRunning, "mwql5pio3390aa"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
if err := h.endUnanswered("main.117", "mwql5pio3390aa", SilentWhy("mwql5pio3390aa")); err != nil {
t.Fatal(err)
}
}, seat.PipelineOutcomeFailed},
{"the runner's start pass finds it with nothing open",
[]seat.StepResult{step("a", seat.StepResultOutcomePassed, "x1"), step("b", seat.StepResultOutcomePassed, "x2"), step("c", seat.StepResultOutcomePassed, "x3")},
func(t *testing.T, h *Hearer) {
if n, err := h.EndAtStart(); n != 1 || err != nil {
t.Fatalf("the start pass ended %d (%v), want 1", n, err)
}
}, seat.PipelineOutcomePassed},
}
for _, r := range rows {
t.Run(r.name, func(t *testing.T) {
store := &refusedOnce{memStore: memStore{}, refused: map[string]bool{}}
store.memStore["main.117"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.117", Branch: "main", Source: 117, Revision: 117, Outcome: seat.PipelineOutcomeRunning,
Movers: slices.Clone(movers), Steps: slices.Clone(r.steps)}, 1}
// Another running pipeline with a step open, and one waiting its turn: neither ends.
store.memStore["main.118"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.118", Branch: "other", Source: 118, Revision: 118, Outcome: seat.PipelineOutcomeRunning,
Movers: movers[:1], Steps: []seat.StepResult{step("a", seat.StepResultOutcomeRunning, "y1")}}, 1}
store.memStore["main.119"] = struct {
p seat.Pipeline
rev uint64
}{seat.Pipeline{V: 1, Pipeline: "main.119", Branch: "other", Source: 119, Revision: 119, Outcome: seat.PipelineOutcomeWaiting, Movers: movers[:1]}, 1}
store.refused["main.118"], store.refused["main.119"] = true, true
grants, revisions := 0, 0
w := &Widener{
Revision: func(string, int64) (Half, error) { revisions++; return Half{}, nil },
Holder: func(s string) (string, error) { return s, nil },
GrantsFor: func(string, int64) (meshgrants.GrantsFor, error) {
grants++
return meshgrants.GrantsFor{}, fmt.Errorf("cut after 5s")
},
}
var turned, journal []string
h := &Hearer{Store: store, Say: func(string, any) error { return nil }, Now: func() time.Time { return t0 }, Widens: w.Reads,
Turn: func(key string) { turned = append(turned, key) },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) }}
r.end(t, h)
p, _, _ := store.Get("main.117")
if p.Outcome != r.outcome {
t.Fatalf("main.117 = %s (%s), want %s", p.Outcome, p.Why, r.outcome)
}
if !slices.Equal(turned, []string{"main.117"}) {
t.Errorf("the turns freed = %v, want main.117's alone, so the next waiting is placed", turned)
}
if !slices.ContainsFunc(journal, func(l string) bool { return strings.HasPrefix(l, "main.117 ended "+r.outcome) }) {
t.Errorf("the journal does not say main.117's end %s: %q", r.outcome, journal)
}
if grants != 1 || revisions != 2 {
t.Errorf("the end asked grants-for %d times and a revision %d times over 3 movers and a refused write, want 1 and 2", grants, revisions)
}
for _, m := range p.Movers {
if len(m.Widens) != 1 || m.Widens[0].Name != "mesh-grants" {
t.Errorf("%s's widens = %+v, want the cut grants-for said", m.Module, m.Widens)
}
}
for _, k := range []string{"main.118", "main.119"} {
if q, _, _ := store.Get(k); q.Outcome == seat.PipelineOutcomeSuperseded || q.Outcome == seat.PipelineOutcomePassed || q.Outcome == seat.PipelineOutcomeFailed {
t.Errorf("%s ended %s with a step open or waiting its turn", k, q.Outcome)
}
}
})
}
}
@@ -332,6 +332,20 @@ func keptOf(raw []byte) (map[string]int64, error) {
return kept, nil
}
// withoutKept is record without its kept-revisions, the runner's own property and never part of the seat's Go form, so the
// seat's strict decode of the record the runner's reads make refuses no record carrying it (H-A-285, H-A-309).
func withoutKept(record []byte) ([]byte, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(record, &fields); err != nil {
return nil, fmt.Errorf("the record is no object: %w", err)
}
if _, ok := fields[keptRevisionsField]; !ok {
return record, nil
}
delete(fields, keptRevisionsField)
return json.Marshal(fields)
}
// withKept is record with kept as its kept-revisions.
func withKept(record []byte, kept map[string]int64) ([]byte, error) {
var fields map[string]json.RawMessage
@@ -1,6 +1,7 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
@@ -94,6 +95,23 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
return e
}
before := mem["main.7"]
// H-A-311 (2): the build's end written while its placing was cut (heard by a hearer that places nothing): the same end
// heard again, late, moves nothing of the build and still places tool, which that end unlocked.
cut := *h
cut.Release = nil
if _, err := cut.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
if p, _, _ = mem.Get("main.7"); placed(p, "tool") {
t.Fatalf("the cut placing placed tool: %+v", stepsOf(p, "tool"))
}
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
if p, _, _ = mem.Get("main.7"); !placed(p, "tool") || stepsOf(p, "tool")[0].Outcome != seat.StepResultOutcomeRunning {
t.Errorf("after builder's build passed, heard late, tool's steps = %+v, want placed and asked", stepsOf(p, "tool"))
}
mem["main.7"] = before
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
@@ -124,6 +142,43 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
t.Errorf("tool's step-ended = %+v, want not asked naming builder", ev.Result)
}
}
// H-A-326 (1), (3), H-A-328: a package published before its importer's wait was written (by an earlier pipeline) is read of
// the build seat when the wait is written, matched by its mover's fingerprint, and the importer is placed with no event;
// the journal says the wait was resolved from the build seat's published, and a retried placing asks it once.
mem2 := memStore{}
var asked []builds.PublishedInput
var journal []string
s2 := &Starter{Pipelines: memCreated{mem2}, Store: mem2, Say: func(string, any) error { return nil }, Steps: &stepAsks{}, Name: "the-runner",
Registration: func(string) (catalog.Registration, bool) { return reg, true },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) },
AskPublished: func(in builds.PublishedInput) (builds.PublishedAnswer, error) {
asked = append(asked, in)
return builds.PublishedAnswer{Published: in.Artifact == "mesh-sdk:go" && in.Fingerprint == "src2:00", Version: "0.2.0"}, nil
}}
if n, err := s2.TakenIn(ev, view); n != 1 || err != nil {
t.Fatalf("the take-in started %d (%v)", n, err)
}
p, _, _ = mem2.Get("main.7")
if sdk, _ := MoverOf(p, "mesh-sdk"); !placed(p, "app") || placed(p, "tool") || !slices.Contains(sdk.Published, "mesh-sdk:go") {
t.Errorf("with mesh-sdk:go published before the wait: app placed %v, tool placed %v, mesh-sdk %+v; want app placed with no event", placed(p, "app"), placed(p, "tool"), sdk)
}
if !slices.ContainsFunc(journal, func(l string) bool {
return strings.Contains(l, "app's wait on mesh-sdk:go") && strings.Contains(l, "placed from the published record")
}) {
t.Errorf("the journal %q does not say app's wait was resolved from the build seat's published", journal)
}
before2 := len(asked)
if _, err := s2.Waiting(mem2, "main", view); err != nil {
t.Fatal(err)
}
if len(asked) != before2 {
t.Errorf("a placing again asked published %d more times, want each package asked once", len(asked)-before2)
}
// At the pipeline's end its kept answers go.
if _, err := s2.Turn("main.7", view); err != nil || len(s2.publishedRead) != 0 {
t.Errorf("after main.7's end %d answers are kept (%v), want none", len(s2.publishedRead), err)
}
}
// Design 47 G §270, A-326 (H §746, H-A-83 (2)): what a build module's new version is built from is built by its running
@@ -446,8 +446,10 @@ func stepAsk(p seat.Pipeline, m seat.Mover, e catalog.CatalogEntry, a catalog.Ar
}
// askAll puts each ask on its seat's work queue; the first refusal is named, the rest are still put. arm, when not nil, arms
// each ask's bound before it is put (H-A-235, H-A-238 (5)); an ask whose bound is not armed is not put.
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool, arm func(Asked) error) error {
// each ask's bound before it is put (H-A-235, H-A-238 (5)); an ask whose bound is not armed is not put. logf, when not nil,
// writes one journal line per ask put, naming the pipeline, the mover, the step and the seat (H-A-320 (1)); a caller that
// says the ask in a line of its own (an ask put again) gives none.
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool, arm func(Asked) error, logf func(string, ...any)) error {
var first error
for _, a := range asks {
accept, ok := acceptStep[a.Seat]
@@ -464,6 +466,10 @@ func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit stri
if err == nil {
err = asker.AskStep(accept, a.ID, a.Ask)
}
if err == nil && logf != nil {
mover, step := askedStep(a)
logf("%s %s %s asked on %s", key, mover, step, a.Seat)
}
if err != nil && first == nil {
first = fmt.Errorf("the step %s of %s is not asked of %s: %w", a.ID, key, a.Seat, err)
}
@@ -471,6 +477,31 @@ func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit stri
return first
}
// askedStep is the mover and the step an ask names, read from its body; "-" where it names none (the validate ask names no
// mover, H-A-51).
func askedStep(a Asked) (mover, step string) {
switch b := a.Ask.(type) {
case nodebuildgo.StepAsk:
if b.Step != nil {
mover, step = b.Step.Module, b.Step.Step
}
case builds.StepRun:
step = b.Step
if len(b.Entries) == 1 {
mover = b.Entries[0].Module
}
case wants.StepRun:
mover, step = b.Module, b.Step
}
if mover == "" {
mover = "-"
}
if step == "" {
step = "-"
}
return mover, step
}
// artifactPaths are the named paths the entry keeps for the artifact, by its reference or its name.
func artifactPaths(e catalog.CatalogEntry, ref string) []string {
if e.Inputs == nil {
@@ -236,6 +236,7 @@ func (s *Starter) StartQueue(view func(branch string) (Half, bool)) (int, error)
// Turn takes the end of pipeline key (its pipeline-ended, a cancel alike, H-A-53): its running key is deleted and the
// queue placed (PlaceQueued).
func (s *Starter) Turn(key string, view func(branch string) (Half, bool)) (int, error) {
s.forgetPublished(key) // the ended pipeline's published answers go with it (H-A-328)
if s.Queue == nil {
return 0, nil
}
@@ -206,6 +206,10 @@ func (r Reads) pipelineOf(key string, all []string) (seat.Pipeline, bool, error)
if raw == nil {
return seat.Pipeline{}, false, nil
}
raw, err = withoutKept(raw)
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s: %w", seat.StatePipelines, key, err)
}
p, err := seat.DecodePipelinesByBranchRevisionValue(raw)
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s: %w", seat.StatePipelines, key, err)
@@ -53,6 +53,12 @@ func TestReads(t *testing.T) {
// A pipeline kept as H-A-187 keeps it: its record's steps empty, each mover's under steps.<pipeline>.<mover>.
put(t, pipes, "next.8", seat.Pipeline{Pipeline: "next.8", Branch: "next", Source: 8, Outcome: seat.PipelineOutcomeRunning,
Started: &t0, Steps: []seat.StepResult{}, Movers: []seat.Mover{{Module: "delta", Commit: "n8"}, {Module: "epsilon", Commit: "n8"}}})
// H-A-285, H-A-309: the record carries the runner's own kept-revisions, which show, list and stands read past.
withRevs, err := withKept(pipes["next.8"], map[string]int64{"delta": 7})
if err != nil {
t.Fatal(err)
}
pipes["next.8"] = withRevs
put(t, pipes, "steps.next.8.epsilon", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "epsilon",
Step: seat.StepResultStepBuild, Status: "required", Outcome: seat.StepResultOutcomePassed, Output: "next.8.build.epsilon:program"}}})
put(t, pipes, "steps.next.8.delta", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "delta",
@@ -5,13 +5,15 @@ import (
"fmt"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// Released writes a version-released's version into the entry of the mover it names (design 47 H §746, H-A-52): the mover of
// that module in each pipeline of the source revision built, a run's aside, so its pipeline-ended names the released version.
// branch, when not empty, is the one watched branch the version was released for (H-A-62); empty matches every branch's
// pipeline of that source. It says how many movers it wrote.
// pipeline of that source. A pipeline that ended passed says its end again, naming the version, and again whenever the version
// is heard again (H-A-270 (2), H-A-277). It says how many movers it wrote.
func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int, error) {
if ev.Module == "" || ev.Version == "" || ev.Source <= 0 {
return 0, fmt.Errorf("a version-released names its module, version and source revision: %+v", ev)
@@ -46,7 +48,15 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
i = j
}
}
if i < 0 || p.Movers[i].Version == ev.Version {
if i < 0 {
break
}
if p.Movers[i].Version == ev.Version {
// A version heard again says the end of an ended passed pipeline again (H-A-277 (3)): a say that failed after
// the version was written is said now; taking it again is a no-op for the holder.
if err := h.sayEndedAgain(key, p, ev); err != nil {
return wrote, err
}
break
}
p.Movers[i].Version = ev.Version
@@ -56,6 +66,11 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
return wrote, fmt.Errorf("the pipeline %s cannot be written: %w", key, err)
}
wrote++
// A version written after the pipeline's end was said says that end again (H-A-270 (2)): the same pipeline-ended,
// now naming the version, so the seat mesh-wants' holder takes the mover it skipped; taking a mover again is a no-op.
if err := h.sayEndedAgain(key, p, ev); err != nil {
return wrote, err
}
break
}
}
@@ -65,6 +80,20 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
return wrote, nil
}
// sayEndedAgain says p's pipeline-ended again when p ended passed (H-A-270 (2), H-A-277 (2)): only a passed end writes a want,
// so a superseded, cancelled or failed one is not said again.
func (h *Hearer) sayEndedAgain(key string, p seat.Pipeline, ev catalog.VersionReleasedEvent) error {
if p.Ended == nil || p.Outcome != seat.PipelineOutcomePassed {
return nil
}
if err := h.say(seat.EventPipelineEnded, p, nil); err != nil {
return fmt.Errorf("the version %s of %s is in the ended pipeline %s but its %s is not said again: %w",
ev.Version, ev.Module, key, seat.EventPipelineEnded, err)
}
h.logf("%s: the version %s of %s is written after its end: %s said again", key, ev.Version, ev.Module, seat.EventPipelineEnded)
return nil
}
// ReleasedBranch is the watched branch a module's versions are released for, its registration's first (H-A-62); empty when
// the registration is not held.
func ReleasedBranch(reg func(string) (catalog.Registration, bool), module string) string {
@@ -137,6 +137,7 @@ func (s *Starter) keepIn(key string, half Half, movers []seat.Mover) []seat.Move
}
was := p
was.Movers = olds
logEnded(s.logf, key, st)
if err := s.Say(stepEvent(seat.EventStepEnded, was, st, now)); err != nil {
s.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, st.Module, st.Step, err)
}
@@ -177,6 +178,9 @@ func (s *Starter) placeAgain(p *seat.Pipeline, m seat.Mover, half Half, now time
return nil, nil, true
}
due, waiting, failed := ready(*p, m.Module, half.Entries)
if !due && failed == "" && s.readPublished(p, m.Module, waiting) {
due, waiting, failed = ready(*p, m.Module, half.Entries)
}
if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
return nil, nil, false
@@ -75,12 +75,18 @@ func IsRunBound(key string) bool {
// runOfAsk is the test run asked under ask: the run its asking step names in Shared, read from the state pipelines.
func (h *Hearer) runOfAsk(ask string) (string, bool) {
run, _, ok := h.runAndPipelineOfAsk(ask)
return run, ok
}
// runAndPipelineOfAsk is runOfAsk with the key of the pipeline whose step asked it.
func (h *Hearer) runAndPipelineOfAsk(ask string) (string, string, bool) {
if h.TestRuns == nil || ask == "" {
return "", false
return "", "", false
}
pkeys, err := allKeys(h.Store)
if err != nil {
return "", false
return "", "", false
}
for _, key := range pkeys {
p, _, err := h.Store.Get(key)
@@ -89,10 +95,26 @@ func (h *Hearer) runOfAsk(ask string) (string, bool) {
}
if i := stepOfAsk(p, ask); i >= 0 {
run := sharedRun(p.Steps[i])
return run, run != ""
return run, key, run != ""
}
}
return "", "", false
}
// releaseLate places what the end of a step heard late unlocks (H-A-311 (2)): a step-progress or step-end that arrives
// after its ask ended moves nothing of the step itself, yet the movers waiting on what that end gave (the dependents'
// builds) are placed, in each pipeline named, as the end heard in time would have placed them (Release).
func (h *Hearer) releaseLate(pipelines ...string) {
if h.Release == nil {
return
}
seen := map[string]bool{}
for _, key := range pipelines {
if key != "" && !seen[key] {
seen[key] = true
h.Release(key)
}
}
return "", false
}
// RunTaken takes the ask-taken of a test run's ask (§725, A-470): the node that took it is kept on the run, its untaken
@@ -158,13 +180,18 @@ func armBound(clock RunClock, runs TestRuns, key, run, ask string, ttl time.Dura
// RunProgress takes a build seat's step-progress of a test run's ask (§725, A-466p): its clock begins again, so a run that
// keeps saying its progress is never ended by it. An ask of no running, taken test run takes nothing.
func (h *Hearer) RunProgress(ask string) error {
run, ok := h.runOfAsk(ask)
run, key, ok := h.runAndPipelineOfAsk(ask)
if !ok {
return nil
}
r, _, err := h.TestRuns.Get(run)
if errors.Is(err, ErrNoRun) || (err == nil && (r.Ask != ask || r.Ended != nil)) {
h.logf("the step-progress of ask %s is late: the test run %s names ask %q (ended %v), so it moves nothing (A-470n)", ask, run, r.Ask, r.Ended != nil)
h.logf("the step-progress of ask %s is late: the test run %s names ask %q (ended %v), so it moves nothing of its own; what its end unlocks is placed (A-470n, H-A-311)", ask, run, r.Ask, r.Ended != nil)
late := []string{key}
for _, reader := range r.Readers {
late = append(late, reader.Pipeline)
}
h.releaseLate(late...)
return nil
}
if err != nil {
@@ -8,6 +8,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
runnerclient "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshpipelinerunnerclient"
@@ -199,6 +200,67 @@ type Starter struct {
// registration-missing (H-A-262); Observer is the runner's own module name. nil raises none.
Findings Findings
Observer string
// AskPublished asks the seat mesh-builds' verb published whether a package built at a fingerprint is published (H-A-326
// (1), H-A-328): read when a mover's wait on it is written, so a publish made before the wait places the mover without
// any event; nil reads nothing, the publish event alone placing it.
AskPublished func(in builds.PublishedInput) (builds.PublishedAnswer, error)
publishedMu sync.Mutex
publishedRead map[string]builds.PublishedAnswer
}
// readPublished asks, once per pipeline, package and fingerprint, the seat mesh-builds whether each package mover waits on
// in p is published already (H-A-326 (1), H-A-328): a wait is a read first and an event second. Each package answered
// published goes into its From mover's published in p, written with the placing, and the journal says the wait resolved
// from the build seat's published, apart from one a package-published places (H-A-326 (3)). The answer is kept for the
// placing's compare-and-set retries, so no retry asks again. It says whether it added any.
func (s *Starter) readPublished(p *seat.Pipeline, mover string, waiting []string) bool {
if s.AskPublished == nil {
return false
}
added := false
for _, ref := range waiting {
i := slices.IndexFunc(p.Movers, func(m seat.Mover) bool { return m.Module == moduleOf(ref) })
if i < 0 || slices.Contains(p.Movers[i].Published, ref) {
continue
}
in := builds.PublishedInput{Module: p.Movers[i].Module, Artifact: ref, Fingerprint: p.Movers[i].Fingerprint}
key := p.Pipeline + "\x00" + ref + "\x00" + in.Fingerprint
s.publishedMu.Lock()
a, asked := s.publishedRead[key]
s.publishedMu.Unlock()
if !asked {
var err error
if a, err = s.AskPublished(in); err != nil {
s.logf("%s: whether %s at %s is published is not read, so %s waits for its publish event: %v", p.Pipeline, ref, in.Fingerprint, mover, err)
continue
}
s.publishedMu.Lock()
if s.publishedRead == nil {
s.publishedRead = map[string]builds.PublishedAnswer{}
}
s.publishedRead[key] = a
s.publishedMu.Unlock()
}
if !a.Published {
continue
}
p.Movers[i].Published = append(p.Movers[i].Published, ref)
added = true
s.logf("%s: %s's wait on %s is resolved from the build seat's published (at %s), not from an event: placed from the published record (H-A-326)", p.Pipeline, mover, ref, a.Version)
}
return added
}
// forgetPublished drops the published answers readPublished kept for pipeline key, at its end (H-A-328).
func (s *Starter) forgetPublished(key string) {
s.publishedMu.Lock()
defer s.publishedMu.Unlock()
for k := range s.publishedRead {
if strings.HasPrefix(k, key+"\x00") {
delete(s.publishedRead, k)
}
}
}
// holderless ends at once, as it is placed, a pipeline with a build step whose seat the seat mesh-module-assigner's view
@@ -265,7 +327,7 @@ func (s *Starter) askSteps(key string, asks []Asked) error {
var err error
if len(asks) > 0 {
s.carry(key, asks)
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) }, s.Asks.arm)
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) }, s.Asks.arm, s.logf)
}
s.sayStarted(key, placed)
return err
@@ -365,6 +427,9 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
// A mover's steps wait for the publish of each mover it builds on (§746, H-A-83 (2)): placed once each is heard,
// ended not asked when one's build ended without passing.
due, waiting, failed := ready(*p, m.Module, entries)
if !due && failed == "" && s.readPublished(p, m.Module, waiting) {
due, waiting, failed = ready(*p, m.Module, entries)
}
if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
waits = append(waits, m)
@@ -495,6 +560,7 @@ func (s *Starter) sayNotAsked(p seat.Pipeline, steps []int) {
}
at := s.now()
for _, i := range steps {
logEnded(s.logf, p.Pipeline, p.Steps[i])
subject, ev := stepEvent(seat.EventStepEnded, p, p.Steps[i], at)
if err := s.Say(subject, ev); err != nil {
s.logf("step-ended of %s's %s %s, not asked, is not said: %v", p.Pipeline, p.Steps[i].Module, p.Steps[i].Step, err)
@@ -746,40 +812,15 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
}
}
for _, key := range pkeys {
p, rev, err := store.Get(key)
p, asks, ended, notAsked, n, err := s.placeOne(store, key, due, reg, view, now)
if err != nil {
note(fmt.Errorf("the pipeline %s is not read: %w", key, err))
continue
note(err)
}
if p.Outcome != seat.PipelineOutcomeRunning || p.Run != "" || !due(p) {
continue
}
half, ok := view(p.Branch)
if !ok {
s.logf("the catalog half of %s is not held yet: %s waits on it for its steps, placed once it is held", p.Branch, key)
continue
}
if p.Revision > 0 && half.Revision < p.Revision {
s.logf("the catalog half of %s holds revision %d: %s waits for revision %d, which its take-in made, to place its steps (§200)", p.Branch, half.Revision, key, p.Revision)
continue
}
before := len(p.Steps)
asks, unregistered, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now)
// Until its registration arrives, the pipeline says it waits on it (A-327): a mover with no registration has no step,
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
if len(p.Steps) == before {
continue
}
if _, err := store.Update(key, p, rev); err != nil {
s.logf("%s is written meanwhile, so nothing is placed in it now: %v", key, err)
if n == 0 {
continue
}
count++
s.logf("%s: %d steps placed from %s (A-327)", key, len(p.Steps)-before, from)
s.logf("%s: %d steps placed from %s (A-327)", key, n, from)
// A pipeline whose steps were never asked is started by this placing (A-446a); the pipeline-started of its start,
// said already, is said again harmlessly.
s.sayStartedPipeline(p, now)
@@ -799,6 +840,50 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
return count, first
}
// placeOne places, in pipeline key, the steps of each mover with none, when the pipeline is running, not a run on request
// and due names it (placeWaiting). Its write is by compare-and-set: a write refused because the pipeline was written
// meanwhile is done again at once from the record as read again, until it lands or the pipeline no longer waits on a
// placing (it ended, or every mover due is placed), so a concurrent write never cuts a placing for good (H-A-311 (1)). It
// answers the pipeline as written, its asks, the steps ended or not asked at the placing, and how many steps it placed.
func (s *Starter) placeOne(store Updater, key string, due func(seat.Pipeline) bool, reg func(module string) (catalog.Registration, bool), view func(branch string) (Half, bool), now time.Time) (seat.Pipeline, []Asked, []int, []int, int, error) {
for {
p, rev, err := store.Get(key)
if err != nil {
return p, nil, nil, nil, 0, fmt.Errorf("the pipeline %s is not read: %w", key, err)
}
if p.Outcome != seat.PipelineOutcomeRunning || p.Run != "" || !due(p) {
return p, nil, nil, nil, 0, nil
}
half, ok := view(p.Branch)
if !ok {
s.logf("the catalog half of %s is not held yet: %s waits on it for its steps, placed once it is held", p.Branch, key)
return p, nil, nil, nil, 0, nil
}
if p.Revision > 0 && half.Revision < p.Revision {
s.logf("the catalog half of %s holds revision %d: %s waits for revision %d, which its take-in made, to place its steps (§200)", p.Branch, half.Revision, key, p.Revision)
return p, nil, nil, nil, 0, nil
}
before := len(p.Steps)
asks, unregistered, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now)
// Until its registration arrives, the pipeline says it waits on it (A-327): a mover with no registration has no step,
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
if len(p.Steps) == before {
return p, nil, nil, nil, 0, nil
}
if _, err := store.Update(key, p, rev); errors.Is(err, ErrChanged) {
s.logf("%s is written meanwhile, so its placing is done again from the record read again (H-A-311): %v", key, err)
continue
} else if err != nil {
return p, nil, nil, nil, 0, fmt.Errorf("the placing of %s is not written: %w", key, err)
}
return p, asks, ended, notAsked, len(p.Steps) - before, nil
}
}
// recovered says whether the commit of pipeline key is named a recovery; a read that fails asks on the stream check, said, since
// a step is asked either way (a recovery's only goes ahead of the first round).
func (s *Starter) recovered(key, commit string) bool {
@@ -300,6 +300,25 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
t.Errorf("the pipeline a runner before this one set aside: placed %d (%v), %d steps, %d asks more, want it started as a newly started one", n, err, len(p.Steps), len(asks.asks)-before)
}
// H-A-311 (1): a placing whose write is refused because the pipeline was written meanwhile (a step's end, a publish) is
// done again at once from the record read again, never left unplaced until a restart.
held = false
taken = catalog.TakeInTaken{Branch: "next", Source: 7, Entries: []catalog.TakeInEntry{{Module: "app", Commit: "c4", Fingerprint: "src2:dd"}}}
if n, err := s.TakenIn(taken, late); n != 1 || err != nil {
t.Fatalf("the take-in of next.7 started %d (%v)", n, err)
}
held = true
raced := &racingStore{memStore: store.memStore, before: func(m memStore) {
e := m["next.7"]
e.p.Superseded, e.rev = append(e.p.Superseded, "other"), e.rev+1 // a concurrent write the placing did not read
m["next.7"] = e
}}
n, err = s.Waiting(raced, "next", late)
p, _, _ = store.Get("next.7")
if n != 1 || err != nil || len(p.Steps) == 0 || !slices.Contains(p.Superseded, "other") {
t.Errorf("a placing cut by a concurrent write placed %d (%v): %d steps, superseded %v, want its steps placed on the record read again (H-A-311)", n, err, len(p.Steps), p.Superseded)
}
// A-327: a mover with no registration, in a pipeline placed from the half held later (the queue's turn, the half held),
// is said to wait on its registration, so the pipeline that cannot end before it is placed says why.
var logged []string
@@ -411,3 +430,17 @@ func TestAPublishIsTakenIntoThePipelineOfItsRevision(t *testing.T) {
t.Errorf("a publish naming no revision (%v): main.7 %v, main.8 %v; want main.7's, the source's lowest", err, published("main.7"), published("main.8"))
}
}
// racingStore is memStore whose first Update is preceded by before, a write another part makes between a read and its write.
type racingStore struct {
memStore
before func(memStore)
}
func (r *racingStore) Update(key string, p seat.Pipeline, rev uint64) (uint64, error) {
if b := r.before; b != nil {
r.before = nil
b(r.memStore)
}
return r.memStore.Update(key, p, rev)
}
@@ -154,8 +154,9 @@ type Hearer struct {
Ager *Ager
// Fills says the stores' fill at each step's end (§366); nil says none.
Fills *Fills
// Widens computes a mover's widens, kept in its value at the pipeline's end (§71, A-207; H-A-51); nil keeps none.
Widens func(p seat.Pipeline, module string) []seat.Widening
// Widens makes, for one end, the reader of each mover's widens, kept in its value at the pipeline's end (§71, A-207;
// H-A-51): made before the end's compare-and-set, its reads shared by every mover and every retry (H-A-289); nil keeps none.
Widens func() func(p seat.Pipeline, module string) []seat.Widening
// TestRuns is the state test-runs and Clock its bound keys (§725); Runs the settings of its runs; DropRun drops a
// run's ask when its run ends unfinished (§119). A nil TestRuns keeps no test run.
TestRuns TestRuns
@@ -245,6 +246,7 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
if ev.Asker != "" && ev.Asker != h.Name {
return seat.Pipeline{}, nil
}
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(ev.Pipeline)
if err != nil {
@@ -271,6 +273,8 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
ev.Module, ev.Commit, ev.Fingerprint, p.Steps[i].Step, p.Steps[i].Module, m.Commit, m.Fingerprint)
case !open(p.Steps[i]):
h.Asks.answeredLate(p.Steps[i], h.logf)
// An end heard late moves nothing of its step, and still places what that step's end unlocks (H-A-311 (2)).
defer h.releaseLate(ev.Pipeline)
if run := sharedRun(p.Steps[i]); run != "" && h.TestRuns != nil {
if r, _, err := h.TestRuns.Get(run); err == nil && r.Ended == nil && r.Ask == ev.ID {
// The asking step's end was written and its run's was not (a write failed): the run ends now, heard again.
@@ -337,7 +341,7 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
// A mover's build passed is what a kind-3 edge waits for (§746, H-A-83 (2)); one that did not pass ends its
// dependents' steps not asked.
release := s.Step == seat.StepResultStepBuild && (!stepPassed(*s) || built(&p, s.Module))
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(ev.Pipeline, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
@@ -477,6 +481,12 @@ func (h *Hearer) said(p seat.Pipeline, asked []int, ended bool) error {
h.Turn(p.Pipeline)
}
if ended {
// The journal says which end it was (H-A-288).
if p.Why != "" {
h.logf("%s ended %s: %s", p.Pipeline, p.Outcome, p.Why)
} else {
h.logf("%s ended %s", p.Pipeline, p.Outcome)
}
h.Failures.Ended(p)
if h.RegistrationWaits != nil {
h.RegistrationWaits()
@@ -493,12 +503,77 @@ func (h *Hearer) said(p seat.Pipeline, asked []int, ended bool) error {
return nil
}
// EndAtStart judges, at the runner's start pass, the end of every running pipeline nothing of which is still open (H-A-288
// (c)): one whose last mover left or whose last open step ended while no runner wrote its end ends as at a step's end,
// superseded, failed or passed by the end judgement, said, and the next waiting placed. It answers how many it ended.
func (h *Hearer) EndAtStart() (int, error) {
pkeys, err := allKeys(h.Store)
if err != nil {
return 0, fmt.Errorf("the pipelines are not listed: %w", err)
}
n := 0
var first error
for _, key := range pkeys {
ended, err := h.endIfNothingOpen(key)
if err != nil && first == nil {
first = err
}
if ended {
n++
}
}
return n, first
}
// endIfNothingOpen ends the running pipeline key when nothing of it is still open (H-A-288), its reads made before the
// compare-and-set (H-A-289); true when it ended.
func (h *Hearer) endIfNothingOpen(key string) (bool, error) {
wr := h.widensReads()
for {
p, rev, err := h.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning {
return false, nil
}
now := h.now()
if !h.endPipeline(&p, now, wr) {
return false, nil
}
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
return false, fmt.Errorf("pipeline %s cannot be written: %w", key, err)
}
h.logf("%s had nothing open at the runner's start: its end is written now (H-A-288)", key)
return true, h.said(p, nil, true)
}
}
// endEntry is the step log's entry of a step's end, with the keys of the kept results it wrote or carried.
func endEntry(key string, s seat.StepResult, at time.Time, kept ...string) seat.LogEntry {
return seat.LogEntry{Pipeline: key, Module: s.Module, Step: s.Step, Artifact: s.Artifact, What: seat.LogEntryWhatEnded, Outcome: s.Outcome,
Output: s.Output, Kept: kept, At: at}
}
// logEnded writes the journal line of a step's end said (H-A-320 (1), H-A-333): the pipeline, the mover, the step, its
// outcome, the node that ended it and how long it ran from its start. It is written where step-ended is said, Hearer.say and
// the starter's two sayers, so every said end has its one line.
func logEnded(logf func(string, ...any), key string, s seat.StepResult) {
if logf == nil {
return
}
mover, node, took := s.Module, s.Node, "no start"
if mover == "" {
mover = "-"
}
if node == "" {
node = "no node"
}
if s.Started != nil && s.Ended != nil {
took = s.Ended.Sub(*s.Started).Round(time.Millisecond).String()
}
logf("%s %s %s ended %s (%s, %s)", key, mover, s.Step, s.Outcome, node, took)
}
func (h *Hearer) logf(format string, args ...any) {
if h.Logf != nil {
h.Logf(format, args...)
@@ -512,6 +587,9 @@ func (h *Hearer) say(event string, p seat.Pipeline, step *seat.StepResult) error
at := h.now()
switch event {
case seat.EventStepEnded, seat.EventStepStarted:
if event == seat.EventStepEnded {
logEnded(h.logf, p.Pipeline, *step)
}
subject, ev := stepEvent(event, p, *step, at)
return h.Say(subject, ev)
case seat.EventPipelineEnded:
@@ -729,19 +807,28 @@ func endPipeline(p *seat.Pipeline, now time.Time, unplaced func(seat.Pipeline, s
return true
}
// endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51).
func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time) bool {
// endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51), by wr, the
// reads the caller made before its compare-and-set (H-A-289); nil computes none.
func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time, wr func(seat.Pipeline, string) []seat.Widening) bool {
if !endPipeline(p, now, h.Unplaced) {
return false
}
if h.Widens != nil {
if wr != nil {
for i := range p.Movers {
p.Movers[i].Widens = h.Widens(*p, p.Movers[i].Module)
p.Movers[i].Widens = wr(*p, p.Movers[i].Module)
}
}
return true
}
// widensReads is the reader of one end's widens, made before its compare-and-set and kept for every retry (H-A-289).
func (h *Hearer) widensReads() func(seat.Pipeline, string) []seat.Widening {
if h.Widens == nil {
return nil
}
return h.Widens()
}
// EndedAtPlacing says a pipeline its starter ended at placing, a seat with no holder (A-470g, A-470h): the step-ended of each
// step of steps, its step log entries appended, the head's finding pipeline-failed (§33), then its pipeline-ended.
func (h *Hearer) EndedAtPlacing(p seat.Pipeline, steps []int) error {
@@ -10,6 +10,7 @@ import (
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
@@ -363,10 +364,21 @@ func TestARecoverysStepsAreAskedOnTheStreamDelivery(t *testing.T) {
}{{true, "mesh.seat.node-build-go.accept.delivery.mesh-pipeline-runner," + wants.AcceptSubjectStep}, {false, "mesh.seat.node-build-go.accept.check.mesh-pipeline-runner," + wants.AcceptSubjectStep}} {
var to []string
asker := askerFunc(func(accept, _ string, _ any) error { to = append(to, accept); return nil })
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }, nil); err != nil || strings.Join(to, ",") != c.want {
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }, nil, nil); err != nil || strings.Join(to, ",") != c.want {
t.Errorf("recovery %v: asked on %v (%v), want %s", c.recovery, to, err, c.want)
}
}
// Each ask put is one journal line naming the pipeline, the mover, the step and the seat: the build ask on mesh-builds and a
// check ask alike (H-A-320 (1)); an ask refused writes none.
var lines []string
put := []Asked{{Seat: builds.Seat, ID: "b1", Ask: builds.StepRun{Step: builds.StepRunStepBuild, Entries: []builds.BuildEntry{{Module: "app"}}}},
{Seat: nodebuildgo.Seat, ID: "u1", Ask: nodebuildgo.StepAsk{Step: &nodebuildgo.StepRun{Module: "app", Step: "unit"}}},
{Seat: "no-seat", ID: "x1", Ask: 3}}
_ = askAll(askerFunc(func(string, string, any) error { return nil }), "main.4", put, nil, nil,
func(f string, a ...any) { lines = append(lines, fmt.Sprintf(f, a...)) })
if want := "main.4 app build asked on mesh-builds\nmain.4 app unit asked on node-build-go"; strings.Join(lines, "\n") != want {
t.Errorf("journal %q, want %q", lines, want)
}
// Starter.recovered reads the commit; a failed read asks on check and says so.
var said []string
s := &Starter{Logf: func(f string, a ...any) { said = append(said, f) }, Recovered: func(c string) (bool, error) {
@@ -91,6 +91,7 @@ func (h *Hearer) SupersedeRunning(newer seat.Pipeline) []string {
}
func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil || p.Outcome != seat.PipelineOutcomeRunning || p.Branch != newer.Branch || p.Run != "" || newer.Run != "" ||
@@ -123,7 +124,7 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
ended = append(ended, i)
}
p.Superseded = append(p.Superseded, moved...)
done := h.endPipeline(&p, now)
done := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
} else if err != nil {
@@ -2,6 +2,7 @@ package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
"time"
@@ -278,6 +279,12 @@ func TestATestRunsEndMovesEveryReaderAndIsCountedOnce(t *testing.T) {
a := f.start(t, 4, 9)
b := f.start(t, 6, 9) // the same candidate: it reads a's runs
unit := stepNamed(a, "app:program", "unit")
// One journal line per ask put, naming the pipeline, the mover, the step and the seat (H-A-320 (1)).
if n := countLines(f.logged, a.Pipeline+" app unit asked on node-build-go"); n != 1 || len(f.asks.ids) == 0 ||
countLines(f.logged, " asked on ") != len(f.asks.ids) {
t.Errorf("journal %q for %d asks put; want one asked-on line per ask, a's unit ask once", f.logged, len(f.asks.ids))
}
f.logged = nil
ended := time.Now()
if _, err := f.h.OnStepEnded(stepSeatOf(t, "node-build-go"), "mesh.seat.node-build-go.event.step-ended.ace", StepEnded{ID: unit.Ask,
Pipeline: a.Pipeline, Module: "app", Commit: "c1", Fingerprint: "src2:aa", Step: "unit", Node: "ace", Outcome: "failed",
@@ -308,6 +315,13 @@ func TestATestRunsEndMovesEveryReaderAndIsCountedOnce(t *testing.T) {
if saidReaders != 4 {
t.Errorf("said %d unit step-ended, want one per reading step", saidReaders)
}
// One journal line per step's end, the asking step's and each reader's, naming its pipeline, mover, step, outcome and node
// (H-A-320 (1)).
for _, key := range []string{a.Pipeline, b.Pipeline} {
if n := countLines(f.logged, key+" app unit ended failed (ace, "); n != 2 {
t.Errorf("%s's unit ends journaled %d times in %q, want once per artifact", key, n, f.logged)
}
}
// a, cancelled, leaves the lint run to b's readers; b, cancelled after, leaves none: the ask is dropped and the run ends.
lint := stepNamed(a, "app:program", "lint")
v := Verbs{Hearer: f.h, Acts: &actLog{}}
@@ -599,6 +613,8 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
run := unit.Shared[0].Run
asked(f, run)
delete(f.clock.bound, UntakenPrefix+run)
var released []string
f.h.Release = func(key string) { released = append(released, key) }
_ = f.h.RunTaken(unit.Ask, "ace")
_ = f.h.RunProgress(unit.Ask)
late := 0
@@ -610,6 +626,10 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
if _, ok := f.clock.bound[BoundPrefix+run]; ok || late != 2 {
t.Errorf("bounds %v, journal %q; want no clock and both late signals journaled", f.clock.bound, f.logged)
}
// H-A-311 (2): the late step-progress still places what its step's end unlocks, in the pipeline that asked it.
if !slices.Contains(released, p.Pipeline) {
t.Errorf("the late step-progress placed in %v, want %s's waiting movers placed", released, p.Pipeline)
}
})
// A run's end is written as its step-ended is heard, before the finding the step says (A-470o).
t.Run("the run ends before the step's finding is written", func(t *testing.T) {
@@ -671,3 +691,14 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
}
})
}
// countLines is how many of the journal's lines hold part.
func countLines(lines []string, part string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, part) {
n++
}
}
return n
}
@@ -52,6 +52,7 @@ func (v Verbs) record(verb, what, why string) error {
func (v Verbs) change(key, event string, record func() error, mutate func(p *seat.Pipeline, now time.Time) (int, error)) (seat.Pipeline, int, error) {
h := v.Hearer
recorded := false
wr := h.widensReads() // made before the compare-and-set, kept for its retries (H-A-289)
for {
p, rev, err := h.Store.Get(key)
if err != nil {
@@ -73,7 +74,7 @@ func (v Verbs) change(key, event string, record func() error, mutate func(p *sea
if err != nil {
return p, i, err
}
ended := h.endPipeline(&p, now)
ended := h.endPipeline(&p, now, wr)
if i >= 0 && event == seat.EventStepEnded {
p.Unlogged = append(p.Unlogged, endEntry(key, p.Steps[i], now))
}
@@ -325,7 +326,7 @@ func (v Verbs) Rerun(caller string, in seat.RerunInput) (seat.RerunAnswer, error
}
if ask != nil {
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm, v.Starter.logf); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -472,7 +473,7 @@ func (v Verbs) rerunTest(caller string, in seat.RerunInput, st seat.StepResult)
break
}
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm, v.Starter.logf); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -31,20 +31,87 @@ func (w *Widener) Widens(p seat.Pipeline, module string) []seat.Widening {
if w == nil || w.Revision == nil {
return nil
}
return w.Reads()(p, module)
}
// Reads is Widens over reads made once and kept (H-A-289): the two revisions, each serving seat's holder and the grants part's
// grants-for are each read at their first need and shared by every mover of an end and every retry of its write, so a retry
// rereads only the pipeline's key. One end takes one Reads, made before its compare-and-set.
func (w *Widener) Reads() func(p seat.Pipeline, module string) []seat.Widening {
if w == nil || w.Revision == nil {
return func(seat.Pipeline, string) []seat.Widening { return nil }
}
r := &endReads{w: w, revisions: map[string]revisionRead{}, holders: map[string]holderRead{}, grants: map[string]grantsRead{}}
return r.widens
}
type revisionRead struct {
half Half
err error
}
type holderRead struct {
holder string
err error
}
type grantsRead struct {
g meshgrants.GrantsFor
err error
}
// endReads are the reads of one end, each made at its first need and kept.
type endReads struct {
w *Widener
revisions map[string]revisionRead
holders map[string]holderRead
grants map[string]grantsRead
}
func (r *endReads) revision(branch string, rev int64) (Half, error) {
k := fmt.Sprintf("%s.%d", branch, rev)
got, ok := r.revisions[k]
if !ok {
got.half, got.err = r.w.Revision(branch, rev)
r.revisions[k] = got
}
return got.half, got.err
}
func (r *endReads) holder(s string) (string, error) {
got, ok := r.holders[s]
if !ok {
got.holder, got.err = r.w.Holder(s)
r.holders[s] = got
}
return got.holder, got.err
}
func (r *endReads) grantsFor(branch string, source int64) (meshgrants.GrantsFor, error) {
k := fmt.Sprintf("%s.%d", branch, source)
got, ok := r.grants[k]
if !ok {
got.g, got.err = r.w.GrantsFor(branch, source)
r.grants[k] = got
}
return got.g, got.err
}
func (r *endReads) widens(p seat.Pipeline, module string) []seat.Widening {
could := func(name, said string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatCouldNotRun, Name: name, Said: said}}
}
cand, err := w.Revision(p.Branch, p.Revision)
cand, err := r.revision(p.Branch, p.Revision)
if err != nil {
return could("mesh-catalog", fmt.Sprintf("the candidate's revision %d is not read: %v", p.Revision, err))
}
newest, err := w.Revision(p.Branch, 0)
newest, err := r.revision(p.Branch, 0)
if err != nil {
return could("mesh-catalog", fmt.Sprintf("the branch's newest revision is not read: %v", err))
}
var out []seat.Widening
for _, s := range servingSeats(p) {
holder, err := w.Holder(s)
holder, err := r.holder(s)
if err != nil {
out = append(out, seat.Widening{What: seat.WideningWhatCouldNotRun, Name: s, Said: fmt.Sprintf("the holder of %s is not read: %v", s, err)})
continue
@@ -57,8 +124,8 @@ func (w *Widener) Widens(p seat.Pipeline, module string) []seat.Widening {
}
out = append(out, ruleWidens(module, newest.Entries[module], cand.Entries[module])...)
out = append(out, credentialWidens(module, newest.Entries[module], cand.Entries[module])...)
if w.GrantsFor != nil {
g, err := w.GrantsFor(p.Branch, p.Source)
if r.w.GrantsFor != nil {
g, err := r.grantsFor(p.Branch, p.Source)
switch {
case err != nil:
out = append(out, seat.Widening{What: seat.WideningWhatCouldNotRun, Name: "mesh-grants", Said: err.Error()})
@@ -88,17 +88,19 @@ func TestWidensIsTheCandidateAgainstTheNewest(t *testing.T) {
// A pipeline's end keeps each mover's widens in its value, whether or not anything widens (design 47 G §71, A-207; H H-A-51).
func TestAnEndKeepsEachMoversWidens(t *testing.T) {
h := &Hearer{Widens: func(p seat.Pipeline, module string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatGrant, Name: module + " s v"}}
h := &Hearer{Widens: func() func(seat.Pipeline, string) []seat.Widening {
return func(p seat.Pipeline, module string) []seat.Widening {
return []seat.Widening{{What: seat.WideningWhatGrant, Name: module + " s v"}}
}
}}
p := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Movers: []seat.Mover{{Module: "app"}, {Module: "lib"}},
Steps: []seat.StepResult{{Module: "app", Step: "build", Outcome: "passed"}, {Module: "lib", Step: "build", Outcome: "passed"}}}
if !h.endPipeline(&p, time.Unix(1, 0)) || len(p.Movers[0].Widens) != 1 || p.Movers[0].Widens[0].Name != "app s v" || p.Movers[1].Widens[0].Name != "lib s v" {
if !h.endPipeline(&p, time.Unix(1, 0), h.widensReads()) || len(p.Movers[0].Widens) != 1 || p.Movers[0].Widens[0].Name != "app s v" || p.Movers[1].Widens[0].Name != "lib s v" {
t.Errorf("the ended movers = %+v", p.Movers)
}
h.Widens = nil
q := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Movers: []seat.Mover{{Module: "app"}}, Steps: []seat.StepResult{{Module: "app", Step: "build", Outcome: "passed"}}}
if !h.endPipeline(&q, time.Unix(1, 0)) || len(q.Movers[0].Widens) != 0 {
if !h.endPipeline(&q, time.Unix(1, 0), h.widensReads()) || len(q.Movers[0].Widens) != 0 {
t.Errorf("a runner with no widener kept %v", q.Movers[0].Widens)
}
}
@@ -106,6 +106,8 @@ func TestManifest(t *testing.T) {
// A-102 (§22, §24): the accepted kinds the runner puts its steps on.
"seat:mesh-builds.step", "seat:mesh-wants.step", "seat:node-build-go.check.step", "seat:node-build-node.check.step",
"seat:node-build-image.check.step",
// H-A-328: whether a package a mover waits on is published, read when the wait is written.
"seat:mesh-builds.published",
// RN22 (§86, §415, A-207): a recovery's steps on the build seats' stream delivery.
"seat:node-build-go.delivery.step", "seat:node-build-node.delivery.step", "seat:node-build-image.delivery.step",
// A-231: the queues of the build seats it asks, check for the steps and delivery for a recovery's.
@@ -78,6 +78,7 @@
"seat:mesh-module-assigner.assignments",
"seat:mesh-grants.grants-for",
"seat:mesh-builds.step",
"seat:mesh-builds.published",
"seat:mesh-wants.step",
"seat:node-build-go.check.step",
"seat:node-build-node.check.step",
@@ -120,6 +120,10 @@ func fullConf(dir string) string {
teller = append(append(teller, viewstest.Grants(assignerModule, assignerModule+".seat-holders", tellerConsumer)...),
"mesh.seat.mesh-catalog.call.revision", "mesh.seat.mesh-catalog.call.revision.>", "mesh.seat.mesh-catalog.call.registrations",
"mesh.seat.mesh-catalog.call.registrations.>", "mesh.seat.mesh-delivery.call.source", "mesh.seat.mesh-delivery.call.source.>")
// Who holds operator-channel, asked of the seat mesh-module-assigner's assignments (manifest invokes) before the router's
// record of an open ask is read: a bed that refused the call held the teller's watch of the asks for the call's whole
// timeout at each catch-up, and every ask written meanwhile waited (issue 1960, H1).
teller = append(teller, "mesh.seat.mesh-module-assigner.call.assignments", "mesh.seat.mesh-module-assigner.call.assignments.>")
teller = append(teller, keepsOf(bucketsOf(tellerModule).findings)...)
view := append(append(readOf(tellerAnswers), readOf(assignerAsks)...), keepsOf(viewAnswers)...)
assigner := keepsOf(assignerAsks)
@@ -224,7 +228,9 @@ func (b *fullBed) place(t *testing.T) {
}
// serveCatalogHalf answers the catalog half's three first views (design 47 G §614): the gate's one source branch, the
// assigner registered on it, and its trunk revision whose entry of the assigner claims mesh-module-assigner.
// assigner registered on it, and its trunk revision whose entry of the assigner claims mesh-module-assigner. It also answers
// the seat mesh-module-assigner's assignments as the live seat does, naming ops-router the holder of operator-channel, so a
// catch-up reads the router's record (ops-router_asks) at once.
func serveCatalogHalf(t *testing.T, nc *nats.Conn) {
t.Helper()
respond := func(a any) nats.MsgHandler {
@@ -235,8 +241,10 @@ func serveCatalogHalf(t *testing.T, nc *nats.Conn) {
Entries: []meshcatalog.CatalogEntry{{Module: assignerModule, ManifestDigest: "d1", Manifest: map[string]any{"module": assignerModule,
"claims": []any{map[string]any{"name": "mesh-module-assigner", "scope": "mesh"}}}}}}
registrations := meshcatalog.RegistrationsAnswer{Registrations: []meshcatalog.Registration{{Module: assignerModule, Version: 1, Branches: []string{"release"}}}}
assignments := json.RawMessage(`{"seats":[{"seat":"operator-channel","scope":"mesh","holders":[{"module":"ops-router","node":"n2"}]}]}`)
for subject, handle := range map[string]nats.MsgHandler{"mesh.seat.mesh-catalog.call.revision": respond(revision),
"mesh.seat.mesh-catalog.call.registrations": respond(registrations), "mesh.seat.mesh-delivery.call.source": respond(map[string]any{"branches": []string{"release"}})} {
"mesh.seat.mesh-module-assigner.call.assignments": respond(assignments),
"mesh.seat.mesh-catalog.call.registrations": respond(registrations), "mesh.seat.mesh-delivery.call.source": respond(map[string]any{"branches": []string{"release"}})} {
for _, s := range []string{subject, subject + ".>"} {
sub, err := nc.Subscribe(s, handle)
if err != nil {
@@ -375,6 +383,11 @@ func TestTellerCarriesAnAskAndWritesTheAnswer(t *testing.T) { // 1343/AC1, AC2
}
// The program hears warrants before it reads the asks; it has its subscription before the ask is written.
b.listening(t)
// Every event the views reader adopted is heard once the hub pulls: registration-changed, which the teller's widenings and the
// catalog half both hear, is one subscription (A-199, §532), so no handler of the teller takes the reader's away.
if l := logs.String(); strings.Contains(l, "a message has one taker") {
t.Errorf("an event of the views reader is not heard: two handlers took one subject\n%s", l)
}
// Design 47 G §717: the teller started before any module held mesh-wants; once the holder is placed it follows it, and
// raises the ask the holder then writes.
b.place(t)
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-contracts/go/seats/meshbroker"
occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdeliveryview"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"io"
@@ -91,9 +92,6 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := hub.Handle(pipelineEndedSubject, widens.HearEnded); err != nil {
return fmt.Errorf("hearing %s: %w", pipelineEndedSubject, err)
}
if err := hub.Handle(registrationChangedSubject, widens.HearRegistration); err != nil {
return fmt.Errorf("hearing %s: %w", registrationChangedSubject, err)
}
// A merge named a recovery is told on every channel (design 47 G §81, §514); a merge made at the forge is no record of its
// own (H §763, H-A-119).
if err := hub.Handle(recoveryMergedSubject, widens.HearRecovery); err != nil {
@@ -108,6 +106,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return calls.Emit(answeredEvent, Answered{Key: key, Ask: askID, Answer: answer}, stdio.EmitOptions{MessageID: answeredID(key, askID)})
}
reader := &views.ProcessReader{Module: cred.Module, Call: calls.Call, Logf: logger.Printf, Hub: hub}
// registration-changed is heard on the reader's one subscription of it, which the catalog client's followers share
// (A-199, §532): the hub gives a message one taker.
if err := reader.AdoptEvent(meshcatalog.Seat, meshcatalog.EventRegistrationChanged, widens.HearRegistration); err != nil {
return fmt.Errorf("hearing %s: %w", meshcatalog.SubjectRegistrationChanged, err)
}
// The teller's finding stream is its `finding-events` entry, asked of the seat mesh-broker's show (design 47 G A-274,
// §529, §577): the writer says its events under that binding's subjects.
binding := process.NewBinding(meshbroker.Seat, cred.Module, calls.CallKept, logger.Printf, views.FindingEventsLocal)
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -33,9 +34,8 @@ const (
// The events the teller hears for it (manifest consumes).
const (
pipelineEndedSubject = "mesh.seat." + mpr.Seat + ".event." + mpr.EventPipelineEnded + ".>"
registrationChangedSubject = meshcatalog.SubjectRegistrationChanged
recoveryMergedSubject = meshdelivery.SubjectRecoveryMerged
pipelineEndedSubject = "mesh.seat." + mpr.Seat + ".event." + mpr.EventPipelineEnded + ".>"
recoveryMergedSubject = meshdelivery.SubjectRecoveryMerged
)
// widensState is the state widens-said as the teller keeps it.
@@ -151,32 +151,39 @@ func (w *Widens) HearEnded(m jetstream.Msg) {
}
// HearRegistration is the handler of registration-changed: it asks the registration of the module and drops the branches it
// no longer watches.
func (w *Widens) HearRegistration(m jetstream.Msg) {
// no longer watches. It is heard on the one subscription the views reader's followers of the event share (ProcessReader's
// AdoptEvent, design 47 G A-199, §532), beside the catalog client's own handler: the event hub gives a message one taker,
// so a second handler of the subject on the hub would take the reader's subscription away. A failure leaves the message
// unacknowledged, and the bus asks again (A-222); a message that names no module is passed over.
func (w *Widens) HearRegistration(_ context.Context, data []byte) error {
var ev meshcatalog.RegistrationChangedEvent
if err := json.Unmarshal(m.Data(), &ev); err != nil || ev.Module == "" {
w.logf("%s names no module and is left: %v", m.Subject(), err)
_ = m.Ack()
return
if err := json.Unmarshal(data, &ev); err != nil || ev.Module == "" {
w.logf("a registration-changed that names no module is left: %v", err)
return nil
}
raw, err := w.Call(registrationsKey, map[string]any{"module": ev.Module})
if err != nil {
w.finish(m, fmt.Errorf("the registration of %s cannot be read: %w", ev.Module, err))
return
return w.heardAgain(fmt.Errorf("the registration of %s cannot be read: %w", ev.Module, err))
}
var ans meshcatalog.RegistrationsAnswer
if err := json.Unmarshal(raw, &ans); err != nil {
w.finish(m, fmt.Errorf("the registration of %s is not read: %w", ev.Module, err))
return
return w.heardAgain(fmt.Errorf("the registration of %s is not read: %w", ev.Module, err))
}
for _, r := range ans.Registrations {
if r.Module == ev.Module {
w.finish(m, w.Unwatched(ev.Module, r.Branches))
return
return w.heardAgain(w.Unwatched(ev.Module, r.Branches))
}
}
// A module with no registration watches no branch.
w.finish(m, w.Unwatched(ev.Module, nil))
return w.heardAgain(w.Unwatched(ev.Module, nil))
}
// heardAgain says err, when not nil, as a message heard again, and gives it back.
func (w *Widens) heardAgain(err error) error {
if err != nil {
w.logf("%v; the message is heard again", err)
}
return err
}
// finish acknowledges the message when err is nil and asks it again when it is not.
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"errors"
"slices"
@@ -155,15 +156,13 @@ func TestARegistrationChangedDropsTheBranchesNoLongerWatched(t *testing.T) {
st.put(k, "c")
}
w := &Widens{State: st, Call: c.call}
m := &fakeMsg{data: []byte(`{"module":"nats","moved":[],"version":2,"warrant":"w"}`)}
w.HearRegistration(m)
if err := w.HearRegistration(context.Background(), []byte(`{"module":"nats","moved":[],"version":2,"warrant":"w"}`)); err != nil {
t.Errorf("the registration-changed is given back: %v", err)
}
keys, _ := st.Keys()
if want := []string{"nats.main", "natsx.next", "other.next"}; !slices.Equal(keys, want) {
t.Errorf("keys %v, want %v", keys, want)
}
if !m.acked || m.naked {
t.Errorf("acked %v naked %v", m.acked, m.naked)
}
}
// A pipeline-ended whose say fails is given back, and one that is told is acknowledged.