Compare commits

..
Author SHA1 Message Date
mesh-tools d1d092980a Merge pull request 'Indexer: the take-in's manifest read sets an unknown top-level word aside (H-A-465)' (#323) from fix/takein-unknown-word into main 2026-11-22 03:15:38 +00:00
jschoubben 241cfd581a The take-in sets an unknown top-level manifest word aside (H-A-465)
The running indexer read every manifest strictly for its fingerprint
inputs, so a manifest carrying a word its SDK does not know yet (class)
got no fingerprint and nothing of the revision delivered. The take-in's
read (ParseManifestForInputs) now sets an unknown top-level word aside,
as the SDK's decoder does, and takes the inputs from the words it knows;
a word inside a known one is still refused. The judgement's read
(judge.Manifest without a predefined list) stays strict.

Builds: design 47 H2 H-A-465
2026-11-22 04:11:49 +01:00
mesh-tools fe5fa6ac94 Merge pull request 'The runner takes out the stayers' steps not asked on a mover that left and places them again at the hand-off (H-A-361)' (#321) from fix/47-H2-end-after-handoff into main 2026-11-21 22:38:18 +00:00
jschoubben 99b25a7db4 H-A-361: a hand-off places again the stayers not asked on a mover that left
On 21 Nov the order was the reverse of the first commit's: mesh-contracts'
old build was withdrawn first (21:50:16), so the stayers that awaited it
were placed not asked, and only then did the hand-offs take it away. A
mover that left is still a dependency of those stayers: their not-asked
steps held the pipeline's end as failures. At the hand-off the steps of
each stayer that ended not asked on a dependency that left, directly or
through a mover itself taken out, are taken out, so the placing that
follows places them again against the newest published package; a
dependency that truly did not build is named again at that placing. The
journal says which movers are placed again.

Builds: design 47 H H-A-361, H-A-53, H-A-223
2026-11-21 23:34:24 +01:00
jschoubben 7606d515fd H-A-361: a hand-off places the movers that stay; a mover that left is not awaited
main.158 and main.163 ended failed seconds after newer pipelines took
movers away: the movers that stay still awaited the movers that left,
whose builds had ended superseded, so the next placing ended them "not
asked: its dependency did not build" and the end judgement counted
them as failures. A mover that left is no longer awaited (the stayer
builds against the newest published package, H-A-53), is never placed
in the older pipeline, and the hand-off places what stays at once and
says in the journal which movers stay.

Builds: design 47 H H-A-361, H-A-53, H-A-223
2026-11-21 23:17:59 +01:00
mesh-tools 7e597a623e Merge pull request 'The runner and the builder write a journal line per step end, ask placed and ask answered (H-A-320, H-A-333)' (#320) from fix/47-H2-journal-lines into main 2026-11-21 21:01:55 +00:00
jschoubben 6a847a8d9c H-A-333: the ended line at every said end, the answered line at every answer
The runner's ended line was written only for a heard step-ended and its run's
readers; the unanswered-bound end, the end at placing, a supersede, a re-pin,
the approval's end, the start pass and the verbs said their ends silently. It
is now written in Hearer.say, the one place that says step-ended, and at the
starter's two sayers that call stepEvent directly (repin, sayNotAsked). The
builder's answered line sat on four of buildEntry's returns; the ask's answer
is now one line in BuildStep's finish, naming the pipeline, the movers, the
step and the outcome, the per-entry existing/built-now line kept beside it.

Builds: design 47 H H-A-320 (1), H-A-333
2026-11-21 21:58:36 +01:00
jschoubben dc83ef3e27 H-A-320 (1): a journal line per step end, step ask and build answer
A working 36-mover pipeline was read as stalled twice: the runner wrote no
line when a step ended or was asked, and the builder none when it answered a
step ask. The runner now writes "<pipeline> <mover> <step> ended <outcome>
(<node>, <duration>)" at each step end it writes (the asking step, every
reader of its test run, and each reader EndRun ends), and "<pipeline> <mover>
<step> asked on <seat>" for each ask put (the re-ask paths keep their own
line). The builder writes "<module>@<fingerprint> build step answered
<outcome> for <pipeline> (existing build / built now)".

Builds: design 47 H H-A-320 (1)
2026-11-21 21:50:48 +01:00
mesh-tools 600c13b9e9 Merge pull request 'Design 47 H: the runner redoes a cut placing, places on a late end, names a stalled pipeline, and reads a prior publish from the build seat (H-A-311, H-A-313, H-A-326, H-A-328, H-A-331)' (#319) from fix/47-H1-runner-placing-redone into main 2026-11-21 20:48:59 +00:00
mesh-tools 5396dc9b11 Merge pull request 'The teller's want-of-room test bed answers the seat-holder ask; one taker of registration-changed (H-A-295)' (#318) from fix/47-H1-teller-test-want-of-room into main 2026-11-21 20:48:56 +00:00
mesh-tools 425dbaef99 Merge pull request 'Design 47 H: a passed pipeline whose movers' versions arrive late still writes its wants (H-A-270, H-A-273, H-A-277)' (#317) from fix/47-H1-end-taken-when-versioned into main 2026-11-21 20:48:49 +00:00
jschoubben 714c2c1ccc H-A-331: the published answer carries the publish time and digest where kept; a withdrawn number is never published
The verb published answered no time, and after a release no digest,
while the seat form says both come when published. The builder now
keeps when it heard or confirmed each publish on the mark, and each
published digest on a kept release; marks and releases written before
answer without them. A number a take-over withdrew is not published.
The runner drops a pipeline's kept answers at its end.

Builds: design 47 H H-A-331, H-A-328
2026-11-21 21:38:50 +01:00
jschoubben e583dce741 H-A-326 (1), (3): a mover's wait on a package is a read first, an event second
main.137 stood twenty minutes with nats and mesh-vault waiting on
mesh-grants packages published at 20:00, before the wait was written,
so no publish event came. The runner now asks the build seat's
published when it writes the wait and places the mover when the
package is published, saying so in the journal apart from an event.

Builds: design 47 H H-A-326 (1), H-A-326 (3), H-A-328
2026-11-21 21:25:03 +01:00
jschoubben 5e76fdf725 H-A-328: the seat mesh-builds answers published; the builder serves it
A package published before a pipeline existed is in no record of the
runner, so the builder, which owns publishes, answers whether an
artifact built at a fingerprint is published, from its reservations
(marked from the registries' record) and its releases.

Builds: design 47 H H-A-328, H-A-326 (1), H-A-274
2026-11-21 21:25:03 +01:00
jschoubben 438c8740a6 H-A-326 (2): a pipeline waiting only on packages is named stalled
A step held on a package's publish counted as open, so a pipeline whose
only movers waited on publishes that had come before their wait was
never named by runner_health (main.137, twenty minutes).

Builds: design 47 H H-A-326 (2), H-A-311 (3)
2026-11-21 21:18:32 +01:00
jschoubben 05d9f90bdb H-A-311 (3) review: the health reads records keeping revisions; a required step waiting is open
runner_health, show, list and stands refused every pipeline record
carrying the runner's own kept-revisions ("unknown field
kept-revisions" on main.131 and main.137): the field is now dropped
before the seat's strict decode. A required step waiting (an approval,
a pin-held step) now counts as open, as the end's judgement holds it.

Builds: design 47 H H-A-311 (3), H-A-313, H-A-285, H-A-309, H-A-322
2026-11-21 20:53:04 +01:00
jschoubben f22945c26a H-A-311 (3): runner_health names a running pipeline with nothing open and nothing placed for five minutes
main.137 stood seventeen minutes with nothing open while runner_health
said healthy. The tools program says each such pipeline in its journal
once per key (H-A-313).

Builds: design 47 H H-A-311 (3), H-A-313
2026-11-21 20:44:43 +01:00
jschoubben e137df7189 H-A-311 (2): a step-progress or step-end heard late still places what its end unlocks
A late progress or end moved nothing, so movers waiting on what the
step's end gave stayed unplaced: main.137 placed nothing after its
lint's late progress at 20:17:55.

Builds: design 47 H H-A-311 (2), H-A-286, H-A-288
2026-11-21 20:43:02 +01:00
jschoubben 6c66bf9a5a H-A-311 (1): a placing cut by a concurrent write is done again from the record read again
A placing whose compare-and-set was refused because the pipeline was
written meanwhile was dropped and never done again: main.137 stood
seventeen minutes with eleven steps placed until a restart.

Builds: design 47 H H-A-311 (1), H-A-288
2026-11-21 20:41:29 +01:00
jschoubben b79fd19c84 Teller: hear registration-changed on the views reader's one subscription
Since mesh-sdk heard a seat's event on its bare subject too (H-A-166), the teller's
own hub handler of registration-changed overlapped the one the views reader makes
for the catalog half's followers: the hub refused the reader's, said "a message has
one taker", and registered none of the reader's adopted events after it, so the
catalog half's registrations and source branches were their first views' alone.
The widenings now hear the event through ProcessReader.AdoptEvent beside the
followers; a failed handling is given back for the bus to ask again, as before.
The carry test now fails when any of the reader's events is refused.

issue 1960 (H1), test stability.

Builds: design 47 G A-199, A-222, §532, §335
2026-11-21 19:24:22 +01:00
jschoubben 70bb8f7f51 Teller test bed: answer who holds operator-channel, so a catch-up does not hold the asks' watch
The want-of-room test failed most runs since 20 Nov: the teller's picture of the
assigner's asks hands the first ask on twice (its first view, then again), and the
second sight of an open asking catches up with the router's record, which first asks
the seat mesh-module-assigner's assignments who holds operator-channel. The bed
neither granted nor answered that call, so it waited out its call timeout on the
watch's goroutine, and the second ask, written meanwhile, was not raised within the
test's 10 s. The bed now grants and answers the call as the live seat does, naming
ops-router, so the catch-up reads ops-router_asks at once (20 of 20 runs pass, were
0 of 20 locally).

issue 1960 (H1), test stability.

Builds: design 47 G §570 (issue 1610 AC1, the refused ask asked again), §717
2026-11-21 19:21:24 +01:00
jschoubben 070f60404e Assigner: a passed mover with no version yet holds its tier
The tier progress started at the lowest tier of the versioned movers, so every
lower tier counted as run and a mover whose version came late was wanted after
its consumer. The progress now holds every passed mover, and a tier with one
whose version is not written does not run until it is.

Builds: design 47 H H-A-270, H-A-273, H-A-277 (1)
2026-11-21 15:55:30 +01:00
jschoubben 7f855f0e52 Runner: say the end again only for a passed pipeline, and again on a version heard again
A say that failed after the version was written was lost: the redelivery found
the version held and said nothing. A failed, superseded or cancelled end writes
no want, so it is not said again.

Builds: design 47 H H-A-270 (2), H-A-277 (2), (3)
2026-11-21 15:52:18 +01:00
jschoubben 3dfc18023a Assigner: a mover whose version came late joins the tier progress at its tier
The tier progress was written once, from the movers the first end named, so
a mover taken from the end said again was wanted out of tier order: a late
tier-2 consumer and the tier above it were wanted together. It now joins
the progress at its tier, and is wanted at once only when that tier ran.

Builds: design 47 H H-A-270, H-A-273
2026-11-21 15:44:58 +01:00
jschoubben 1489493431 Assigner: say a passed mover with no version yet; one registration-changed handler
The holder skipped such a mover silently, so a passed pipeline wrote no want
and nothing said why; the runner now says the end again once the version is
written, and the holder takes it then. The pipeline ends took registrations
from a second handler that the hub refused at every start (a message has one
taker); they are told by the catalog half's one follower instead.

Builds: design 47 H H-A-270 (1), (3)
2026-11-21 15:41:03 +01:00
jschoubben e927bb8971 Runner: a version written after a pipeline's end says pipeline-ended again
The seat mesh-wants' holder skips a passed mover with no version yet; the
version-released arrives late, so the end it took named none and no want was
written. Saying the same pipeline-ended again, naming the version, lets the
holder take the mover it skipped.

Builds: design 47 H H-A-270 (2)
2026-11-21 15:34:34 +01:00
59 changed files with 1599 additions and 193 deletions
@@ -37,6 +37,10 @@ type halves struct {
// settled is told once when the first watched branch's first view is answered, for the first pass to wait on.
settled chan<- struct{}
told bool
// regMu orders the registrations told to registered: the ones held when it is set, then each change (H-A-270 (3)).
regMu sync.Mutex
registered func(reg meshcatalog.Registration, gone bool)
}
// follow makes the branch follower on the reader; the reader's start asks the first views.
@@ -50,6 +54,15 @@ func (h *halves) follow() error {
h.changed(branch, module, entry, deleted)
}
b.Adopted = h.adopted
// The one handler of registration-changed is the follower's (H-A-270 (3)): the pipeline ends are told each change from it,
// never from a second handler on the hub.
reconcile := b.Registrations.Follower.Changed
b.Registrations.Follower.Changed = func(module string, value json.RawMessage, gone bool, at time.Time) {
if reconcile != nil {
reconcile(module, value, gone, at)
}
h.tellRegistration(module, value, gone)
}
h.mu.Lock()
h.branches = b
h.mu.Unlock()
@@ -215,3 +228,39 @@ func (h *halves) watchedOf(module string) []string {
slices.SortStableFunc(out, func(x, y string) int { return strings.Compare(keys.EscapeName(x), keys.EscapeName(y)) })
return out
}
// tellRegistration tells registered of one registration the follower holds now, or of one removed.
func (h *halves) tellRegistration(module string, value json.RawMessage, gone bool) {
h.regMu.Lock()
defer h.regMu.Unlock()
if h.registered == nil {
return
}
reg := meshcatalog.Registration{Module: module}
if !gone {
if err := json.Unmarshal(value, &reg); err != nil {
h.logf("the registration of %s does not read: %v", module, err)
return
}
}
h.registered(reg, gone)
}
// followRegistrations tells registered of every registration the branch follower holds and then of each change it takes
// (design 47 G §353): the registrations are followed once, by the follower's one handler of registration-changed (H-A-270
// (3)). It says false when no follower is made.
func (h *halves) followRegistrations(registered func(meshcatalog.Registration, bool)) bool {
h.mu.Lock()
b := h.branches
h.mu.Unlock()
if b == nil || b.Registrations == nil {
return false
}
h.regMu.Lock()
defer h.regMu.Unlock()
h.registered = registered
for _, reg := range b.Registrations.All() {
registered(reg, false)
}
return true
}
@@ -53,6 +53,7 @@ func TestAMeshThatNamesNoBranchSettlesTheAssignersFirstPass(t *testing.T) {
// Design 47 H-A-143, H-A-144, G §532: a module's watched branches are its registration's, in its order; once registrations are held a
// module none names has none; before the first registrations the gate's source's branches stand in, in the source's order.
// H-A-270 (3): the pipeline ends are told the registrations by the same follower, whether set before its first view or after.
func TestAModulesWatchedBranchesAreItsRegistrations(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
@@ -81,11 +82,23 @@ func TestAModulesWatchedBranchesAreItsRegistrations(t *testing.T) {
if err := h.follow(); err != nil {
t.Fatal(err)
}
var live, held []string
if !h.followRegistrations(func(reg seat.Registration, gone bool) { live = append(live, reg.Module) }) {
t.Fatal("no follower of the registrations")
}
for _, f := range reader.Followers()[:2] { // the registrations and the gate's source
if err := f.Start(ctx); err != nil {
t.Fatal(err)
}
}
h.followRegistrations(func(reg seat.Registration, gone bool) { held = append(held, reg.Module) })
var regs []string
for _, r := range tc.regs {
regs = append(regs, r.Module)
}
if !slices.Equal(live, regs) || !slices.Equal(held, regs) {
t.Errorf("the ends were told %v at the first view and %v held, want %v", live, held, regs)
}
if got := h.watchedOf(tc.module); !slices.Equal(got, tc.want) {
t.Errorf("the watched branches of %s are %v, want %v", tc.module, got, tc.want)
}
@@ -880,6 +880,14 @@ func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
if _, err := b.buckets[engineReadsBucket()].Get(b.ctx, "shanks"); err == nil {
t.Error("the state engine-reads holds a key of shanks, whose engine said nothing")
}
// H-A-270 (3): registration-changed has one handler, the catalog half's follower, so the start's hub refuses no overlap.
b.journal.await(t, 0, "the kept seat events heard", "the hub started", time.Second)
b.journal.mu.Lock()
overlap := strings.Contains(b.journal.text.String(), "a message has one taker")
b.journal.mu.Unlock()
if overlap {
t.Error("the start's hub refused a handler that overlaps another: a message has one taker")
}
}
// subscribed returns once the server holds every subscription nc made: nats.go sends a SUB from its flusher, after
@@ -649,12 +649,12 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
p.ends = newPipelineEnds(kvEnds{ctx, kv}, wake, logger.Printf)
p.ends.trunk = w.trunkName
p.ends.runEnded = func(group string) error { return p.dropCandidate(ctx, group) }
err := hearEnds(hub, p.ends)
if err == nil && bus != nil {
err = followRegistrations(ctx, hub, bus.Call, p.ends)
if err := hearEnds(hub, p.ends); err != nil {
logger.Printf("a pipeline's end is not heard, so the wants keep the trunk-only form until the assigner starts again: %v", err)
}
if err != nil {
logger.Printf("a pipeline's end or a registration is not heard, so the wants keep the trunk-only form until the assigner starts again: %v", err)
// The registrations come from the catalog half's one follower, its one handler of registration-changed (H-A-270 (3)).
if iv == nil || iv.h == nil || !iv.h.followRegistrations(p.ends.registered) {
logger.Printf("the registrations are not followed, so the wants keep the trunk-only form until the assigner starts again: the catalog half is not followed")
}
}
hearReleases(ctx, js, w, wake, findings, logger.Printf) // version-released, machine-away and machine-back
@@ -14,7 +14,6 @@ import (
"github.com/nats-io/nats.go/jetstream"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
mprc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshpipelinerunnerclient"
mpcat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
mpr "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -161,22 +160,33 @@ func newPipelineEnds(st endStore, wake func(), logf func(string, ...any)) *pipel
// take keeps, for each mover of a passed pipeline that released a version, that version as the newest of its module on the
// pipeline's branch, by compare-and-set on a newer version (design 47 H §750): an older version, or the same one again, writes
// nothing. It says whether any key changed.
// nothing. A passed mover with no version yet is said in the journal and taken when the runner says the end again naming it
// (H-A-270); it holds its tier in the pipeline's tier progress meanwhile (H-A-277 (1)). It says whether any key changed.
func (e *pipelineEnds) take(p mpr.PipelineEnded) (bool, error) {
if p.Outcome != mpr.PipelineEndedOutcomePassed || p.Run != "" {
return false, nil
}
changed := false
var kept []tierMover
versioned := map[string]bool{}
defer func() {
if err := e.startTiers(p.Pipeline, p.Source, kept); err != nil {
if err := e.startTiers(p.Pipeline, p.Source, kept, versioned); err != nil {
e.logf("the tier progress of %s is not written: %v", p.Pipeline, err)
}
}()
for _, m := range p.Movers {
if m.Version == "" || m.Fingerprint == "" || !moverPassed(m) {
if m.Fingerprint == "" || !moverPassed(m) {
continue
}
if m.Version == "" {
// The runner writes a mover's version at its release's version-released, and says this end again once it does
// (H-A-270 (1), (2)): the skip is said, never silent.
e.logf("%s: %s passed with no version yet: taken again when its version is written", p.Pipeline, m.Module)
// It holds its tier in the progress until its version is written (H-A-277 (1)).
kept = append(kept, tierMover{Module: m.Module, Fp: m.Fingerprint, Tier: max(m.Tier, 1), NoVersion: true})
continue
}
versioned[m.Module] = true
key := endKey(m.Module, p.Branch)
for {
raw, rev, err := e.st.get(key)
@@ -372,32 +382,3 @@ func (e *pipelineEnds) registered(reg mpcat.Registration, gone bool) {
}
e.wake()
}
// followRegistrations follows the registrations through the seat mesh-catalog (design 47 G §353): a first view of its verb
// and each registration-changed on the hub. It returns when the hub's handler is registered; the first view is asked on its own
// goroutine.
func followRegistrations(ctx context.Context, hub *process.EventHub, call mcc.Call, e *pipelineEnds) error {
regs := mcc.NewRegistrations(call)
regs.Follower.Logf = e.logf
regs.Follower.Changed = func(module string, value json.RawMessage, gone bool, _ time.Time) {
reg := mpcat.Registration{Module: module}
if !gone {
_ = json.Unmarshal(value, &reg)
}
e.registered(reg, gone)
}
if err := hub.Handle(mpcat.SubjectRegistrationChanged, func(m jetstream.Msg) {
if err := regs.Hear(ctx, m.Data()); err != nil {
e.logf("%s: %v", m.Subject(), err)
}
_ = m.Ack()
}); err != nil {
return err
}
go func() {
if err := regs.Start(ctx); err != nil && ctx.Err() == nil {
e.logf("the first view of the registrations is not answered: %v", err)
}
}()
return nil
}
@@ -141,7 +141,8 @@ func TestThePassedVersionIsKept(t *testing.T) {
// A6/AC1, A5/AC1 (design 47 H §750): a pass writes, on every node the module is assigned to, the want naming the newest version
// released on its branch whose pipeline passed, with the pipeline and the rank; with none passed it writes none. No grant,
// canary, provider or composition is asked for.
// canary, provider or composition is asked for. H-A-270 (1): a passed end whose mover names no version yet writes no want and
// says so in the journal; the same end said again naming the version writes it.
func TestAPassWritesTheWantOfThePassedVersion(t *testing.T) {
fp := meshcatalogclient.SourcePrefix2 + strings.Repeat("a", 64)
for _, passed := range []bool{false, true} {
@@ -154,9 +155,15 @@ func TestAPassWritesTheWantOfThePassedVersion(t *testing.T) {
p.w.manifests[mdigest] = parsed{module: "web", manifest: catalogue.Manifest{Module: "web", Requires: []string{"database"}}}
placeOn(t, p.w, "g14", "web")
placeOn(t, p.w, "ace", "web")
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
var journal []string
p.ends = newPipelineEnds(newMemEnds(), func() {}, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
p.ends.trunk = p.w.trunkName
if passed {
_, _ = p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("web", fp, "", 2, mpr.StepResultOutcomePassed)))
p.wants(context.Background())
if len(wantsKV.values) != 0 || !slices.Contains(journal, trunkBranch+".5: web passed with no version yet: taken again when its version is written") {
t.Errorf("an end naming no version wrote %d wants and said %q", len(wantsKV.values), journal)
}
_, _ = p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("web", fp, "0.1.2", 2, mpr.StepResultOutcomePassed)))
}
p.wants(context.Background())
@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
@@ -40,6 +41,8 @@ type tierMover struct {
Module string `json:"module"`
Fp string `json:"fp"`
Tier int64 `json:"tier"`
// NoVersion is a passed mover whose version is not yet written: its tier does not run until it is (H-A-277 (1)).
NoVersion bool `json:"no-version,omitempty"`
}
// tierProgress is a pipeline's tier progress (H-A-56): the tier whose wants are written, since when, the movers, and the source
@@ -61,24 +64,53 @@ func isTierKey(key string) bool {
return strings.HasPrefix(key, tierPrefix) && strings.Count(key, ".") == 2
}
// startTiers writes the tier progress of a passed pipeline whose movers' versions are now kept, its first tier the lowest of
// theirs, since now. A pipeline heard again writes nothing.
func (e *pipelineEnds) startTiers(pipeline string, source int64, movers []tierMover) error {
if len(movers) == 0 {
return nil
}
first := movers[0].Tier
for _, m := range movers {
first = min(first, m.Tier)
}
value, err := json.Marshal(tierProgress{Tier: first, Since: e.clock().UTC(), Movers: movers, Source: source})
if err != nil {
// startTiers writes the tier progress of a passed pipeline, holding every passed mover, versioned or not, its first tier the
// lowest of theirs, since now (H-A-277 (1)). A pipeline heard again adds to the progress that stands each mover it does not hold,
// at the mover's tier, and marks versioned each held mover whose version is now written (H-A-270, H-A-273): its want is written
// when the progress reaches its tier, or at once when that tier ran already, never out of tier order.
func (e *pipelineEnds) startTiers(pipeline string, source int64, movers []tierMover, versioned map[string]bool) error {
for {
t, rev, ok := e.tiersOf(pipeline)
if !ok {
if len(movers) == 0 {
return nil
}
first := movers[0].Tier
for _, m := range movers {
first = min(first, m.Tier)
}
t = tierProgress{Tier: first, Since: e.clock().UTC(), Source: source}
}
moved := false
for i := range t.Movers {
if t.Movers[i].NoVersion && versioned[t.Movers[i].Module] {
t.Movers[i].NoVersion, moved = false, true
}
}
for _, m := range movers {
if !slices.ContainsFunc(t.Movers, func(h tierMover) bool { return h.Module == m.Module }) {
t.Movers, moved = append(t.Movers, m), true
}
}
if !moved {
return nil
}
value, err := json.Marshal(t)
if err != nil {
return err
}
err = e.st.swap(tierKey(pipeline), value, rev)
if errors.Is(err, errSwapped) {
if _, _, held := e.tiersOf(pipeline); !ok && !held {
return nil // a key that stands but does not read is left as it is
}
continue
}
if err == nil && ok {
e.logf("pipeline %s: a mover whose version came after its end is taken into the tier progress", pipeline)
}
return err
}
if err := e.st.swap(tierKey(pipeline), value, 0); err != nil && !errors.Is(err, errSwapped) {
return err
}
return nil
}
func (e *pipelineEnds) clock() time.Time {
@@ -196,6 +228,18 @@ func (p *passer) notRunning(n assign.Nodes, placedOn func(module, branch string)
if m.Tier != tier {
continue
}
if m.NoVersion {
// A mover whose version is not yet written holds its tier, under the tier bound (H-A-277 (1)).
machines := []string{}
for _, node := range placedOn(m.Module, branch) {
if online, known := n.Online[node]; !known || online {
machines = append(machines, node)
}
}
sort.Strings(machines)
out = append(out, meshwants.NotRunning{Module: m.Module, Machines: machines})
continue
}
if end, ok := p.ends.endOf(m.Module, branch); !ok || end.Pipeline != pipeline {
continue // a later pipeline moved it
}
@@ -125,3 +125,86 @@ func TestAPipelinesWantsAreWrittenTierByTier(t *testing.T) {
}
}
}
// H-A-270, H-A-273, H-A-277 (1): a pipeline's tier progress holds every passed mover, versioned or not, and a tier with a mover
// whose version is not yet written does not run until it is: no later tier is wanted out of order. When the lowest tier's version
// comes last, nothing is wanted before it; with db (1) and api (3) versioned and web (2) late, api is wanted only once web runs.
func TestALateMoverHoldsItsTier(t *testing.T) {
type step struct {
versions map[string]string // a pipeline-ended taken, each mover's version ("": none yet)
runs string // or: this module runs on g14
wanted string // then: the modules wanted, in order of the list
}
for _, c := range []struct {
name string
steps []step
}{
{"the lowest tier's version comes last", []step{
{versions: map[string]string{"db": "", "web": "0.1.2", "api": "0.3.0"}, wanted: ""},
{versions: map[string]string{"db": "0.2.0", "web": "0.1.2", "api": "0.3.0"}, wanted: "db"},
{runs: "db", wanted: "db,web"},
{runs: "web", wanted: "db,web,api"},
}},
{"a middle tier's version comes late", []step{
{versions: map[string]string{"db": "0.2.0", "web": "", "api": "0.3.0"}, wanted: "db"},
{runs: "db", wanted: "db"},
{versions: map[string]string{"db": "0.2.0", "web": "0.1.2", "api": "0.3.0"}, wanted: "db,web"},
{runs: "web", wanted: "db,web,api"},
}},
} {
t.Run(c.name, func(t *testing.T) {
fps := map[string]string{}
tiers := map[string]int64{"db": 1, "web": 2, "api": 3}
for i, m := range []string{"db", "web", "api"} {
fps[m] = meshcatalogclient.SourcePrefix2 + strings.Repeat(string(rune('a'+i)), 64)
}
start := time.Date(2026, 11, 21, 4, 0, 0, 0, time.UTC)
now := start
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(kvSet{buckets.Wants(): wantsKV, buckets.Asks(): &memKV{values: map[string][]byte{}}}, start)
p.now = func() time.Time { return now }
p.say = func(oc.Condition) error { return nil }
p.publishWants = func(string, any, string) error { return nil }
for i, m := range []string{"db", "web", "api"} {
digest := strings.Repeat(string(rune('1'+i)), 64)
mod, _ := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Fp2: fps[m], Manifest: digest, Repository: "novox/" + m, Commit: strings.Repeat("1", 40), Path: meshcatalogclient.RootPath})
putModuleRaw(p.w, trunkBranch, m, mod)
p.w.manifests[digest] = parsed{module: m, manifest: catalogue.Manifest{Module: m}}
}
placeOn(t, p.w, "g14", "db", "web", "api")
p.w.mu.Lock()
p.w.runningHeard = true
p.w.mu.Unlock()
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
p.ends.trunk, p.ends.now = p.w.trunkName, func() time.Time { return now }
for i, s := range c.steps {
if s.versions != nil {
var movers []mpr.MoverEnded
for _, m := range []string{"db", "web", "api"} {
movers = append(movers, mover(m, fps[m], s.versions[m], tiers[m], mpr.StepResultOutcomePassed))
}
if _, err := p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, movers...)); err != nil {
t.Fatal(err)
}
} else {
raw, _ := json.Marshal(nodeengine.Running{Node: "g14", Module: s.runs, Fingerprint: fps[s.runs]})
if err := p.w.hearRunning("running.g14."+s.runs, raw); err != nil {
t.Fatal(err)
}
}
now = now.Add(time.Minute)
p.wants(context.Background())
var got []string
for _, m := range []string{"db", "web", "api"} {
key, _ := mwc.WantKey("g14", m)
if _, ok := wantsKV.values[key]; ok {
got = append(got, m)
}
}
if strings.Join(got, ",") != s.wanted {
t.Fatalf("step %d: wanted %v, want %q", i+1, got, s.wanted)
}
}
})
}
}
@@ -473,13 +473,15 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// modules again (design 47 G A-82, §204).
// The runner's build step ask, the builder's one entry point for a candidate (design 47 G A-97).
b.StepEntry = newEntryWaits(branches, half.manifests).entry
// The seat mesh-builds' verb reserved, answered here (A-115, A-127): the reservations, the catalog half's carriers and the seat
// mesh-catalog's versions.
// The seat mesh-builds' verbs reserved, answered here (A-115, A-127): the reservations, the catalog half's carriers and the seat
// mesh-catalog's versions; and published, from the reservations and releases (H-A-328).
stopReserved, err := calls.AnswerSeat(meshbuilds.Seat, seatlist.Definition(seatGate.Reader, meshbuilds.Seat), map[string]process.SeatHandler{
meshbuilds.VerbReserved: reservedVerb(ctx, reservedReader(ctx, reserved.r.Store), versions.CarriersFrom(versions.EntriesOf(branches)), catalogReleased(calls.Call),
func(name string) {
logger.Printf("reserved: no entry carries the name %s; the entries looked at: %s", name, versions.LookedAt(versions.EntriesOf(branches)))
})})
}),
// published (H-A-328): whether a package built at a fingerprint is published, asked by the runner writing a wait on it.
meshbuilds.VerbPublished: publishedVerb(ctx, reserved.r.PublishedOf)})
if err != nil {
return fmt.Errorf("answering the verb reserved: %w", err)
}
@@ -49,6 +49,18 @@ func reservedVerb(ctx context.Context, rr versions.Reader, carriers func(context
}
}
// publishedVerb is the handler of the verb published (design 47 H-A-328): whether an artifact built at a fingerprint is
// published, from the reservations and the releases the builder keeps.
func publishedVerb(ctx context.Context, of func(ctx context.Context, module, artifact, fingerprint string) (meshbuilds.PublishedAnswer, error)) process.SeatHandler {
return func(_ string, body json.RawMessage) (any, error) {
var in meshbuilds.PublishedInput
if err := json.Unmarshal(body, &in); err != nil {
return nil, fmt.Errorf("the arguments do not fit the verb published: %w", err)
}
return of(ctx, in.Module, in.Artifact, in.Fingerprint)
}
}
// catalogReleased are a module's released versions, asked of the seat mesh-catalog's verb versions through call.
func catalogReleased(call toolCaller) versions.CatalogVersions {
return func(_ context.Context, module string) ([]meshcatalog.VersionEntry, error) {
@@ -24,9 +24,24 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
b.init()
started := b.now()
end := meshbuilds.StepEnded{V: 1, Pipeline: run.Pipeline, Step: meshbuilds.StepEndedStepBuild, Rerun: run.Rerun, By: run.By, Entries: run.Entries, Node: b.Node, Started: &started}
// finish is the one place the ask is answered: its line names the pipeline, the movers, the step and the outcome, with
// the words of an end that did not pass (H-A-320 (1), H-A-333).
finish := func(outcome, said string) meshbuilds.StepEnded {
at := b.now()
end.Ended, end.Outcome, end.Said = &at, outcome, said
movers := make([]string, 0, len(run.Entries))
for _, e := range run.Entries {
movers = append(movers, e.Module)
}
mv := strings.Join(movers, ",")
if mv == "" {
mv = "-"
}
if said != "" {
b.logf("%s %s %s step answered %s: %s", run.Pipeline, mv, end.Step, outcome, said)
} else {
b.logf("%s %s %s step answered %s", run.Pipeline, mv, end.Step, outcome)
}
return end
}
if len(run.Entries) == 0 {
@@ -43,7 +58,7 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
var failed []string
for _, e := range run.Entries {
var pins map[string]string
outcome, said := b.buildEntry(ctx, e, &pins)
outcome, said := b.buildEntry(ctx, run.Pipeline, e, &pins)
// The end reports the pins each entry was built with, so the runner asks its held test steps with them (H-A-207).
for path, version := range pins {
if version == "" {
@@ -88,9 +103,10 @@ func (b *Builder) askedRevision(run meshbuilds.StepRun) {
}
}
// buildEntry builds one entry at its fingerprint and waits for its end; pins is set, when it is done, to the pins its
// done key keeps it was built with, the registry's included (H-A-207).
func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins *map[string]string) (outcome, said string) {
// buildEntry builds one entry at its fingerprint for pipeline and waits for its end; pins is set, when it is done, to the pins
// its done key keeps it was built with, the registry's included (H-A-207). The answer from a build is one journal line,
// naming the entry, its outcome, the pipeline and whether the build existed or was built now (H-A-320 (1)).
func (b *Builder) buildEntry(ctx context.Context, pipeline string, e meshbuilds.BuildEntry, pins *map[string]string) (outcome, said string) {
entry, err := b.StepEntry(ctx, e.Fingerprint)
if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry %s at %s cannot be read: %v", e.Module, e.Fingerprint, err)
@@ -110,13 +126,18 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins
}
ended := b.awaitEnd(key)
defer b.dropEnd(key, ended)
answered := func(outcome, said, how string) (string, string) {
b.logf("%s@%s build step answered %s for %s (%s)", entry.Module, entry.Fp, outcome, pipeline, how)
return outcome, said
}
if done, failedWords, found := b.keyEnd(ctx, key); found {
if done {
// A key done before reports the pins it was built with, kept with it, never the ones a publish gives now.
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry)
outcome, said := b.passedWithItsVersion(ctx, entry)
return answered(outcome, said, "existing build")
}
return meshbuilds.StepEndedOutcomeFailed, failedWords
return answered(meshbuilds.StepEndedOutcomeFailed, failedWords, "existing build")
}
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalogclient.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
@@ -125,9 +146,10 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins
case log := <-ended:
if log.Outcome == builds.BuildDone {
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry)
outcome, said := b.passedWithItsVersion(ctx, entry)
return answered(outcome, said, "built now")
}
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error)
return answered(meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error), "built now")
case <-ctx.Done():
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the builder stopped before its build ended", entry.Module, entry.Fp)
}
@@ -7,6 +7,7 @@ import (
"reflect"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -36,12 +37,23 @@ func TestABuildStepAnswersItsEnd(t *testing.T) {
ask := func(entries ...meshbuilds.BuildEntry) meshbuilds.StepEnded {
return r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.1.m", Step: meshbuilds.StepRunStepBuild, Entries: entries})
}
// Every answer is one journal line naming the pipeline, the movers, the step and the outcome (H-A-320 (1), H-A-333).
var answers []string
r.b.Logf = func(f string, a ...any) {
if l := fmt.Sprintf(f, a...); strings.Contains(l, " step answered ") {
answers = append(answers, l)
}
}
if end := ask(); end.Outcome != meshbuilds.StepEndedOutcomeNothingToRun {
t.Errorf("an ask naming no entry: %+v", end)
}
if end := ask(meshbuilds.BuildEntry{Module: "m", Fingerprint: "src2:" + strings.Repeat("2", 64)}); end.Outcome != meshbuilds.StepEndedOutcomeCouldNotRun || !strings.Contains(end.Said, "cannot be read") {
t.Errorf("an entry the seat does not hold: %+v", end)
}
if len(answers) != 2 || answers[0] != "main.1.m - build step answered nothing to run: the ask names no entry" ||
!strings.HasPrefix(answers[1], "main.1.m m build step answered could not run: the entry m at ") {
t.Errorf("the answers journaled %q, want one line per answer", answers)
}
if end := ask(meshbuilds.BuildEntry{Module: "m", Fingerprint: fp1, Commit: "other"}); end.Outcome != meshbuilds.StepEndedOutcomeCouldNotRun {
t.Errorf("an entry at another commit: %+v", end)
}
@@ -171,6 +183,18 @@ func awaitWaiting[T any](t *testing.T, waiting <-chan struct{}, ended <-chan T)
// another series and the module's release number pin nothing.
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
r, _, waiting := unpinnedRig(t)
// Each entry's answer from a build is one journal line, built now or existing, beside the ask's answer (H-A-320 (1), H-A-333).
var mu sync.Mutex
var answers []string
logf := r.b.Logf
r.b.Logf = func(format string, args ...any) {
logf(format, args...)
if l := fmt.Sprintf(format, args...); strings.HasPrefix(l, "m@") && strings.Contains(l, "build step answered") {
mu.Lock()
answers = append(answers, l)
mu.Unlock()
}
}
r.b.PackagePublished(unpinnedX, "v0.2.9") // another series than the 0.1 the module accepts: no pin
r.b.PackagePublished(unpinnedX, "0.1.4-feat-x.2") // a group's tagged version: no pin
ended := make(chan meshbuilds.StepEnded, 1)
@@ -206,6 +230,12 @@ func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
if again.Outcome != meshbuilds.StepEndedOutcomePassed || again.Packages[unpinnedX] != "v0.1.3" {
t.Errorf("the done key's end = %+v; want passed reporting x at v0.1.3, as built", again)
}
mu.Lock()
want := []string{"m@" + fp1 + " build step answered passed for main.9.m (built now)", "m@" + fp1 + " build step answered passed for main.9.m (existing build)"}
if !reflect.DeepEqual(answers, want) {
t.Errorf("the answers journaled %q, want %q", answers, want)
}
mu.Unlock()
// A done key that keeps no pins reports none, so the runner ends the waiting steps loud.
v, err := builds.DecodeBuilderBuild(r.st.vals[r.key])
if err != nil {
@@ -54,7 +54,10 @@ type ReservedVersion struct {
BuiltAt time.Time `json:"built-at,omitzero"`
// Published are the artifacts a registry seat said published at this version with the built digest.
Published map[string]bool `json:"published,omitempty"`
At time.Time `json:"at"`
// PublishedAt is when the builder heard or confirmed each artifact's publish (H-A-331); absent for a mark written before it
// was kept.
PublishedAt map[string]time.Time `json:"published-at,omitempty"`
At time.Time `json:"at"`
// Repository and Commit are what the build was made from, and Tag the git tag its release is written as (a Go
// package's, GoTagOf): empty for a module with no Go package (design 47 G §370).
Repository string `json:"repository,omitempty"`
@@ -84,6 +87,17 @@ type ReservedVersion struct {
Withdrawn map[string]string `json:"withdrawn,omitempty"`
}
// markPublished marks artifact a published at at, the moment the builder heard or confirmed its publish (H-A-331).
func (v *ReservedVersion) markPublished(a string, at time.Time) {
if v.Published == nil {
v.Published = map[string]bool{}
}
if v.PublishedAt == nil {
v.PublishedAt = map[string]time.Time{}
}
v.Published[a], v.PublishedAt[a] = true, at.UTC()
}
// KeptPublished is a registry seat's published kept on a reservation, as heard (design 47 G A-415).
type KeptPublished struct {
Name string `json:"name"`
@@ -396,6 +410,8 @@ type ReleasedRecord struct {
type ReleasedBuild struct {
Fingerprint string `json:"fingerprint"`
Version string `json:"version"`
// Digests are the published digest of each artifact, by reference (H-A-331); absent for a release kept before.
Digests map[string]string `json:"digests,omitempty"`
}
// ReleasedFor says whether the builder saw a version released for the build of fingerprint among the module's last releases.
@@ -419,8 +435,54 @@ func (r *Reserver) ReleasedFor(ctx context.Context, module, fingerprint string)
return false, nil
}
// PublishedOf answers the seat mesh-builds' verb published (design 47 H-A-328): whether artifact of module built at fingerprint
// is published. A reservation of that fingerprint whose artifact a registry seat said published at the built digest (marked
// from the registry's record too, H-A-274), its number not withdrawn, answers its version, digest and, where kept, the
// publish time (H-A-331); a release of that fingerprint among the module's
// last releases answers published at its version, every artifact of a release being published; otherwise not published.
func (r *Reserver) PublishedOf(ctx context.Context, module, artifact, fingerprint string) (meshbuilds.PublishedAnswer, error) {
module, artifact, fingerprint = strings.TrimSpace(module), strings.TrimSpace(artifact), strings.TrimSpace(fingerprint)
if module == "" || artifact == "" || fingerprint == "" {
return meshbuilds.PublishedAnswer{}, errors.New("module, artifact and fingerprint are required")
}
keys, err := r.Store.KeysOf(ctx, NameFilter(module))
if err != nil {
return meshbuilds.PublishedAnswer{}, fmt.Errorf("the reservations of %s cannot be listed: %w", module, err)
}
sort.Strings(keys)
for _, k := range keys {
cur, _, found, err := r.get(ctx, k)
if err != nil {
return meshbuilds.PublishedAnswer{}, err
}
// A number withdrawn by a take-over is never answered published (H-A-331).
if found && cur.Module == module && cur.Fingerprint == fingerprint && cur.Published[artifact] && !hasKey(cur.Withdrawn, cur.Version) {
a := meshbuilds.PublishedAnswer{Published: true, Version: cur.Version, Digest: cur.Artifacts[artifact]}
if at, ok := cur.PublishedAt[artifact]; ok {
a.At = &at
}
return a, nil
}
}
if r.Releases != nil {
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(module))
if err != nil {
return meshbuilds.PublishedAnswer{}, fmt.Errorf("the releases of %s cannot be read: %w", module, err)
}
var rec ReleasedRecord
if found && json.Unmarshal(raw, &rec) == nil {
for i := len(rec.Kept) - 1; i >= 0; i-- {
if rec.Kept[i].Fingerprint == fingerprint {
return meshbuilds.PublishedAnswer{Published: true, Version: rec.Kept[i].Version, Digest: rec.Kept[i].Digests[artifact]}, nil
}
}
}
}
return meshbuilds.PublishedAnswer{Published: false}, nil
}
// keepRelease adds a release to the module's record, dropping the oldest past ReleasedKept.
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string) error {
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string, digests map[string]string) error {
n := r.ReleasedKept
if n < 1 {
n = DefaultReleasedKept
@@ -440,7 +502,7 @@ func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version
kept = append(kept, k)
}
}
kept = append(kept, ReleasedBuild{Fingerprint: fingerprint, Version: version})
kept = append(kept, ReleasedBuild{Fingerprint: fingerprint, Version: version, Digests: digests})
if len(kept) > n {
kept = kept[len(kept)-n:]
}
@@ -479,7 +541,7 @@ func (r *Reserver) Released(ctx context.Context, module, version string) ([]stri
continue
}
if cur.Version == version && cur.Fingerprint != "" && r.Releases != nil {
if err := r.keepRelease(ctx, module, cur.Fingerprint, version); err != nil {
if err := r.keepRelease(ctx, module, cur.Fingerprint, version, cur.Artifacts); err != nil {
return gone, fmt.Errorf("the release of %s %s cannot be recorded: %w", module, version, err)
}
}
@@ -695,11 +757,9 @@ func (r *Reserver) publishedOn(ctx context.Context, keys []string, name, version
}
matched = true
if !cur.Published[a] {
cur.Published = maps.Clone(cur.Published)
if cur.Published == nil {
cur.Published = map[string]bool{}
}
cur.Published[a], changed = true, true
cur.Published, cur.PublishedAt = maps.Clone(cur.Published), maps.Clone(cur.PublishedAt)
cur.markPublished(a, r.now())
changed = true
arts = append(arts, a)
}
}
@@ -768,10 +828,7 @@ func (r *Reserver) Kept(ctx context.Context, module, version string) ([]Marked,
}
matched = true
if !cur.Published[a] {
if cur.Published == nil {
cur.Published = map[string]bool{}
}
cur.Published[a] = true
cur.markPublished(a, r.now())
arts = append(arts, a)
}
}
@@ -221,6 +221,87 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
}
}
// TestPublishedAnswersByTheBuildFingerprint: design 47 H-A-328 — the verb published answers published for a reservation of the
// fingerprint whose artifact a registry's record marked published, with its version and digest, and for a release of it; an
// artifact not yet marked, and another fingerprint, are not published.
func TestPublishedAnswersByTheBuildFingerprint(t *testing.T) {
ctx := context.Background()
heard := time.Date(2026, 11, 21, 19, 0, 0, 0, time.UTC)
store := &reserveMem{mem: newMem()}
r := &Reserver{Store: store, Releases: &reserveMem{mem: newMem()}, ReleasedKept: 2, Filled: func() bool { return true },
ReleasedOf: func(context.Context, string) ([]string, error) { return nil, nil }, Now: func() time.Time { return heard }}
v, err := r.Reserve(ctx, "mesh-grants", "0.1", []string{"mesh-grants:busgrants", "mesh-grants:grantsstate"}, Source{Fingerprint: "src2:aa"})
if err != nil {
t.Fatal(err)
}
if _, err := r.Built(ctx, "mesh-grants", v.Version, "mesh-grants:busgrants", "sha256:bb", nil); err != nil {
t.Fatal(err)
}
ask := func(artifact, fp string) meshbuilds.PublishedAnswer {
t.Helper()
a, err := r.PublishedOf(ctx, "mesh-grants", artifact, fp)
if err != nil {
t.Fatal(err)
}
return a
}
if a := ask("mesh-grants:busgrants", "src2:aa"); a.Published {
t.Errorf("a built artifact no registry holds is answered %+v, want not published", a)
}
if marked, _, err := r.Published(ctx, "mesh-grants-busgrants", v.Version, "sha256:bb"); err != nil || len(marked) != 1 {
t.Fatalf("the registry's published marked %v (%v)", marked, err)
}
// H-A-331: the mark keeps when the publish was heard, answered as at.
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.Version != v.Version || a.Digest != "sha256:bb" || a.At == nil || !a.At.Equal(heard) {
t.Errorf("published answered %+v, want %s at sha256:bb, heard %s", a, v.Version, heard)
}
// A mark written before the publish time was kept answers no time, never an invented one; a withdrawn number is never
// answered published.
key := ReservedKey("mesh-grants", "0.1")
cur, rev, _, _ := r.get(ctx, key)
older := cur
older.PublishedAt = nil
raw, _ := json.Marshal(older)
if rev, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.At != nil {
t.Errorf("an older mark answered %+v, want published with no at", a)
}
withdrawn := older
withdrawn.Withdrawn = map[string]string{v.Version: "src2:aa"}
raw, _ = json.Marshal(withdrawn)
if rev, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:busgrants", "src2:aa"); a.Published {
t.Errorf("a withdrawn number is answered %+v, want not published", a)
}
raw, _ = json.Marshal(cur)
if _, err = store.Update(ctx, key, raw, rev); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:grantsstate", "src2:aa"); a.Published {
t.Errorf("an artifact not marked is answered %+v", a)
}
if a := ask("mesh-grants:busgrants", "src2:other"); a.Published {
t.Errorf("another fingerprint is answered %+v", a)
}
if _, err := r.Released(ctx, "mesh-grants", v.Version); err != nil {
t.Fatal(err)
}
if a := ask("mesh-grants:grantsstate", "src2:aa"); !a.Published || a.Version != v.Version {
t.Errorf("after its release the fingerprint's artifact is answered %+v, want published at %s", a, v.Version)
}
// H-A-331: the kept release keeps each published digest with its version.
if a := ask("mesh-grants:busgrants", "src2:aa"); !a.Published || a.Digest != "sha256:bb" {
t.Errorf("after its release busgrants is answered %+v, want its digest sha256:bb", a)
}
if _, err := r.PublishedOf(ctx, "mesh-grants", "", "src2:aa"); err == nil {
t.Error("an ask naming no artifact is answered")
}
}
// TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226, H-A-244 — a dropped number stays a gap, kept in
// the series key's Withdrawn while it is above the catalog's highest released; at each write those at or below it go, so the
// map never grows without end.
@@ -198,9 +198,9 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if len(m.Logs) != 1 || m.Logs[0].Name != builds.BuilderLog || m.Logs[0].MaxAgeDays != builds.BuilderLogDays || m.Logs[0].MaxMiB != builds.BuilderLogMiB {
t.Errorf("logs = %+v, want %s kept %d days in %d MiB", m.Logs, builds.BuilderLog, builds.BuilderLogDays, builds.BuilderLogMiB)
}
// Design 47 G §52, §369: the seat mesh-builds, held once for the mesh, serving reserved.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-builds" || m.Claims[0].Scope != "mesh" || !reflect.DeepEqual(m.Claims[0].Serves, []string{"reserved"}) {
t.Errorf("claims = %+v, want mesh-builds, scope mesh, serving reserved", m.Claims)
// Design 47 G §52, §369, H-A-328: the seat mesh-builds, held once for the mesh, serving reserved and published.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-builds" || m.Claims[0].Scope != "mesh" || !reflect.DeepEqual(m.Claims[0].Serves, []string{"reserved", "published"}) {
t.Errorf("claims = %+v, want mesh-builds, scope mesh, serving reserved and published", m.Claims)
}
// A-115: the process answers the verb itself and says so in `serves` beside its process, with no `bus`; the bundle serves no tool named so.
var resources []map[string]any
@@ -208,8 +208,8 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
t.Fatal(err)
}
for _, r := range resources {
if r["type"] == "process" && !reflect.DeepEqual(r["serves"], []any{"seat:mesh-builds.reserved"}) {
t.Errorf("the process serves %v, want seat:mesh-builds.reserved (A-115)", r["serves"])
if r["type"] == "process" && !reflect.DeepEqual(r["serves"], []any{"seat:mesh-builds.reserved", "seat:mesh-builds.published"}) {
t.Errorf("the process serves %v, want seat:mesh-builds.reserved and published (A-115, H-A-328)", r["serves"])
}
}
if slices.Contains(m.Tools, "reserved") {
+4 -2
View File
@@ -7,7 +7,8 @@
"name": "mesh-builds",
"scope": "mesh",
"serves": [
"reserved"
"reserved",
"published"
]
}
],
@@ -293,7 +294,8 @@
"name": "mesh-delivery-builder",
"artifact": "program",
"serves": [
"seat:mesh-builds.reserved"
"seat:mesh-builds.reserved",
"seat:mesh-builds.published"
],
"run": [
"./mesh-delivery-builder"
@@ -9,7 +9,8 @@ import "time"
const Seat = "mesh-builds"
const (
VerbReserved = "reserved"
VerbReserved = "reserved"
VerbPublished = "published"
)
const (
@@ -24,7 +25,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string {
return []string{VerbReserved}
return []string{VerbReserved, VerbPublished}
}
// Events is every event published on the seat's subject: a new copy each call.
@@ -217,6 +218,28 @@ type ReservedAnswer struct {
Version string `json:"version,omitempty"`
}
// PublishedInput is what published takes.
type PublishedInput struct {
// the package's artifact reference, <module>:<artifact>
Artifact string `json:"artifact"`
// the build fingerprint the package is built at, which the answer is matched by
Fingerprint string `json:"fingerprint"`
// the module that builds the package
Module string `json:"module"`
}
// PublishedAnswer is what published answers.
type PublishedAnswer struct {
// when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331)
At *time.Time `json:"at,omitempty"`
// the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331)
Digest string `json:"digest,omitempty"`
// true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331)
Published bool `json:"published"`
// the version it is published at, MAJOR.MINOR.PATCH, when published
Version string `json:"version,omitempty"`
}
// StepAccept is the body of step.
type StepAccept = StepRun
@@ -35,6 +35,12 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v ReservedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "reserved answers an error alone, naming the name or the version", subject: "", given: "{\"error\":\"no entry carries the name example.org/none\"}", wire: "{\"error\":\"no entry carries the name example.org/none\"}",
decode: func(b []byte) (any, error) { var v ReservedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published asks an artifact at its build fingerprint", subject: "", given: "{\"artifact\":\"mesh-grants:busgrants\",\"fingerprint\":\"src2:aa\",\"module\":\"mesh-grants\"}", wire: "{\"artifact\":\"mesh-grants:busgrants\",\"fingerprint\":\"src2:aa\",\"module\":\"mesh-grants\"}",
decode: func(b []byte) (any, error) { var v PublishedInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published answers the version, digest and time of a published artifact", subject: "", given: "{\"at\":\"2026-11-21T19:00:00Z\",\"digest\":\"sha256:aa\",\"published\":true,\"version\":\"0.1.4\"}", wire: "{\"at\":\"2026-11-21T19:00:00Z\",\"digest\":\"sha256:aa\",\"published\":true,\"version\":\"0.1.4\"}",
decode: func(b []byte) (any, error) { var v PublishedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "published answers an artifact not published", subject: "", given: "{\"published\":false}", wire: "{\"published\":false}",
decode: func(b []byte) (any, error) { var v PublishedAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
}
for _, c := range cases {
v, err := c.decode([]byte(c.given))
@@ -52,5 +52,31 @@
"given": {
"error": "no entry carries the name example.org/none"
}
},
{
"name": "published asks an artifact at its build fingerprint",
"of": "verb:published.input",
"given": {
"module": "mesh-grants",
"artifact": "mesh-grants:busgrants",
"fingerprint": "src2:aa"
}
},
{
"name": "published answers the version, digest and time of a published artifact",
"of": "verb:published.answer",
"given": {
"published": true,
"version": "0.1.4",
"digest": "sha256:aa",
"at": "2026-11-21T19:00:00Z"
}
},
{
"name": "published answers an artifact not published",
"of": "verb:published.answer",
"given": {
"published": false
}
}
]
@@ -70,6 +70,58 @@
},
"description": "the reservation or release, or the error; module, version and artifacts are present whenever error is not (design 47 G 679)"
}
},
{
"name": "published",
"class": "read",
"description": "Whether an artifact built at a build fingerprint is published, from the builder's reservations and the registries' record, for the pipeline runner writing a mover's wait on a package: a package published before the wait was written is read, not waited for (design 47 H-A-326, H-A-328, H-A-274).",
"input": {
"type": "object",
"properties": {
"module": {
"type": "string",
"description": "the module that builds the package"
},
"artifact": {
"type": "string",
"description": "the package's artifact reference, <module>:<artifact>"
},
"fingerprint": {
"type": "string",
"description": "the build fingerprint the package is built at, which the answer is matched by"
}
},
"required": [
"module",
"artifact",
"fingerprint"
]
},
"answer": {
"type": "object",
"properties": {
"published": {
"type": "boolean",
"description": "true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331)"
},
"version": {
"type": "string",
"description": "the version it is published at, MAJOR.MINOR.PATCH, when published"
},
"digest": {
"type": "string",
"description": "the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331)"
},
"at": {
"type": "string",
"format": "date-time",
"description": "when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331)"
}
},
"required": [
"published"
]
}
}
],
"accepts": [
@@ -6,6 +6,7 @@ export declare const seat: "mesh-builds";
export declare const verbs: {
readonly reserved: "reserved";
readonly published: "published";
};
export declare const accepts: {
@@ -171,6 +172,28 @@ export interface ReservedAnswer {
"version"?: string;
}
/** is what published takes. */
export interface PublishedInput {
/** the package's artifact reference, <module>:<artifact> */
"artifact": string;
/** the build fingerprint the package is built at, which the answer is matched by */
"fingerprint": string;
/** the module that builds the package */
"module": string;
}
/** is what published answers. */
export interface PublishedAnswer {
/** when the builder heard or confirmed the publish, when published; absent for a mark written before the publish time was kept (design 47 H-A-331) */
"at"?: string;
/** the published artifact's digest, when published; absent for a release kept before the digest was kept (design 47 H-A-331) */
"digest"?: string;
/** true when the artifact built at the fingerprint is published; a number a take-over withdrew is never published (design 47 H-A-331) */
"published": boolean;
/** the version it is published at, MAJOR.MINOR.PATCH, when published */
"version"?: string;
}
/** is the body of step. */
export type StepAccept = StepRun;
@@ -6,6 +6,7 @@ export const seat = "mesh-builds";
export const verbs = {
reserved: "reserved",
published: "published",
};
export const accepts = {
@@ -143,8 +143,9 @@ func TestAnArtifactOfNoToolchainHasNoLanguageAndNoLock(t *testing.T) { // 1450/A
func TestAManifestTheParserRefusesLeavesNoInputsAndSaysWhy(t *testing.T) { // 1450/AC5
for name, raw := range map[string]string{
"not json": "not json",
"unknown field": strings.Replace(catalogueManifest(t, "avahi"), `"version": "1",`, `"version": "1", "scope": "everything",`, 1),
"not json": "not json",
// A word inside a known one is still refused; a top-level word is set aside (below).
"unknown field": strings.Replace(catalogueManifest(t, "avahi"), `"name": "tools",`, `"name": "tools", "a-field-from-later": "everything",`, 1),
} {
in := inputsOf("avahi", raw, "t", nil, noBases)
if in.Kept != nil || !strings.HasPrefix(in.None2, "the manifest cannot be read: ") || len(in.None2) <= len("the manifest cannot be read: ") {
@@ -154,6 +155,15 @@ func TestAManifestTheParserRefusesLeavesNoInputsAndSaysWhy(t *testing.T) { // 14
if in := inputsOf("someone-else", catalogueManifest(t, "avahi"), "t", nil, noBases); in.Kept != nil || !strings.Contains(in.None2, "avahi") {
t.Errorf("a manifest of another module: kept %+v none2 %q", in.Kept, in.None2)
}
// H-A-465: a top-level word the running indexer's SDK does not know yet (`class`, which a newer SDK adds) is set
// aside, and the inputs are those of the same manifest without it.
without := inputsOf("avahi", catalogueManifest(t, "avahi"), "t", map[string]string{"go.sum": avahiGoSum}, noBases)
with := inputsOf("avahi", strings.Replace(catalogueManifest(t, "avahi"), `"version": "1",`, `"version": "1", "class": "core",`, 1),
"t", map[string]string{"go.sum": avahiGoSum}, noBases)
if with.Kept == nil || without.Kept == nil || !reflect.DeepEqual(*with.Kept, *without.Kept) || with.None2 != without.None2 ||
!reflect.DeepEqual(with.Languages, without.Languages) || !reflect.DeepEqual(with.ArtifactPaths, without.ArtifactPaths) {
t.Errorf("with an unknown top-level word: %+v (kept %+v), want the inputs without it %+v (kept %+v)", with, with.Kept, without, without.Kept)
}
}
func TestAContextIsNotPinned(t *testing.T) { // 1450 (the rule 1444 keeps: context not pinned, as today's form)
@@ -7,12 +7,12 @@ import (
)
// unreadableAvahi is avahi's manifest with a field the running parser refuses: the shape of a manifest an older SDK could
// not read.
// not read. The field is inside an artifact, since a top-level word the take-in does not know is set aside (H-A-465).
func unreadableAvahi(t *testing.T) string {
t.Helper()
raw := strings.Replace(catalogueManifest(t, "avahi"), `"version": "1",`, `"version": "1", "scope": "everything",`, 1)
raw := strings.Replace(catalogueManifest(t, "avahi"), `"name": "tools",`, `"name": "tools", "a-field-from-later": "everything",`, 1)
if raw == catalogueManifest(t, "avahi") {
t.Fatal("the fixture has no version line")
t.Fatal("the fixture has no artifact named tools")
}
return raw
}
+1 -1
View File
@@ -40,7 +40,7 @@ func Manifest(raw []byte, predefined catalogue.PredefinedList) (m catalogue.Mani
if predefined != nil {
m, err = ParseManifestWith(raw, predefined)
} else {
m, err = ParseManifestForInputs(raw)
m, err = parseManifestFallingBack(raw)
}
if err != nil {
return catalogue.Manifest{}, problemsOf(err)
@@ -27,6 +27,9 @@ func TestTheJudge(t *testing.T) {
want: map[string]string{"a": `secret-`}},
{name: "an unknown field", raws: map[string]string{"a": `{"module":"a","version":"1.0","a-field-from-later":1}`},
want: map[string]string{"a": `"a-field-from-later"`}},
// H-A-465: the take-in sets a top-level word it does not know aside (ForInputs); the judgement still refuses it.
{name: "a word a newer SDK adds", raws: map[string]string{"a": `{"module":"a","version":"1.0","class":"core"}`},
want: map[string]string{"a": `"class"`}},
// Design 47 G §23, §28: a step's status is kept in the registration and never in a manifest.
{name: "a manifest naming a step's status", raws: map[string]string{"a": `{"module":"a","version":"1.0","policy":{"lint":"bypass"}}`},
want: map[string]string{"a": `"policy"`}},
+66 -1
View File
@@ -3995,13 +3995,23 @@ var protectedRoots = []string{"/", "/bin", "/boot", "/dev", "/etc", "/home", "/l
"/var/spool"}
// ParseManifestForInputs reads a manifest for the inputs of its build and its fingerprint: the indexer's and the
// builder's read (novox/hq issue 1751). It is never a judge of config values. A value falling back to a predefined key
// builder's read (novox/hq issue 1751). It is never a judge of config values, and never of words: a top-level word the
// SDK's manifest does not know (a word a newer SDK adds, such as `class`) is set aside, as the SDK's decoder sets it
// aside, and the inputs are taken from the words it knows (design 47 H, H-A-465), so a revision carrying a new word
// still gets its fingerprint from the running indexer. The judgement's read (Manifest, ParseManifest) stays strict: the
// validator built from the revision's own indexer refuses what it does not know. A value falling back to a predefined key
// (ADR 0348 §7) is judged against the list the mesh published when this process holds one (PublishedPredefined);
// where it holds none, as in the indexer and the builder, each key a value names is taken as the value declares it (its
// type, trusted when it is), since the list is read where a manifest is judged: the merge gate from the tree's root,
// the controller from what it published. No input of a fingerprint or a build comes from the list. Everything else
// ParseManifest refuses is refused here too.
func ParseManifestForInputs(raw []byte) (catalogue.Manifest, error) {
return parseManifestFallingBack(withoutUnknownWords(raw))
}
// parseManifestFallingBack is the strict read judging each predefined key a value falls back to against the list the
// mesh published, or, where this process holds none, as the value declares it (fallbacksOf).
func parseManifestFallingBack(raw []byte) (catalogue.Manifest, error) {
list := catalogue.PublishedPredefined()
if list == nil {
list = fallbacksOf(raw)
@@ -4009,6 +4019,61 @@ func ParseManifestForInputs(raw []byte) (catalogue.Manifest, error) {
return parseManifest(raw, list)
}
// withoutUnknownWords is raw without each top-level word the SDK's manifest does not know, the others in their order
// and their values as written; raw itself when it knows every word or is no JSON object (the strict read then says
// what is wrong with it). A word inside a known one is not set aside here: the strict read refuses it.
func withoutUnknownWords(raw []byte) []byte {
decoder := json.NewDecoder(bytes.NewReader(raw))
if t, err := decoder.Token(); err != nil || t != json.Delim('{') {
return raw
}
var out bytes.Buffer
out.WriteByte('{')
kept, setAside := 0, false
for decoder.More() {
t, err := decoder.Token()
if err != nil {
return raw
}
word, _ := t.(string)
var value json.RawMessage
if err := decoder.Decode(&value); err != nil {
return raw
}
if !knownWord(word) {
setAside = true
continue
}
if kept > 0 {
out.WriteByte(',')
}
quoted, _ := json.Marshal(word)
out.Write(quoted)
out.WriteByte(':')
out.Write(value)
kept++
}
if !setAside {
return raw
}
out.WriteByte('}')
return out.Bytes()
}
// knownWord is whether the SDK's manifest knows a top-level word: a manifest of that word alone, valued null, is read
// with nothing set aside. A word the SDK reads by a rule of its own that refuses null is known.
func knownWord(word string) bool {
probe, err := json.Marshal(map[string]json.RawMessage{word: json.RawMessage("null")})
if err != nil {
return true
}
var m catalogue.Manifest
if err := json.Unmarshal(probe, &m); err != nil {
return true
}
return m.SetAside() == ""
}
// fallbacksOf is the list of the predefined keys a manifest's config values fall back to, each as the value declares
// it. A manifest that is not JSON gives an empty list, and the parser says what is wrong with it.
func fallbacksOf(raw []byte) catalogue.PredefinedList {
@@ -7,6 +7,7 @@ import (
"encoding/json"
"fmt"
"os"
"sync"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
@@ -98,13 +99,29 @@ func FirstViewTools(module string, st stdio.KeptState, acts views.HandActs, now
}
// HealthTool is the runner's health check, run outside its process (design 47 G §76, §468, §495; A-164): it reads the
// runner's state pipelines as show does and the ages its process writes in the state consumers, and asks no seat verb.
func HealthTool(pipelines, consumers pipeline.Kept, now func() time.Time) stdio.Tool {
// runner's state pipelines as show does and the ages its process writes in the state consumers, and asks no seat verb. A
// running pipeline it first names stalled is said in this program's journal through logf, once per key for the
// program's life (H-A-311 (3), H-A-313).
func HealthTool(pipelines, consumers pipeline.Kept, now func() time.Time, logf func(string, ...any)) stdio.Tool {
var mu sync.Mutex
said := map[string]bool{}
return stdio.Tool{Name: Prefix + "_health",
Description: "The pipeline runner's health: healthy when it answers show and each of its durable consumers' oldest " +
"unacknowledged message waits less than five minutes; otherwise it names each consumer behind.",
Description: "The pipeline runner's health: healthy when it answers show, each of its durable consumers' oldest " +
"unacknowledged message waits less than five minutes and no running pipeline has had nothing open and nothing " +
"newly placed for five minutes; otherwise it names each consumer behind and each such pipeline by its key.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return pipeline.Health(pipelines, consumers, now()), nil }}
Run: func(map[string]any) (any, error) {
a, stalled := pipeline.HealthStalled(pipelines, consumers, now())
mu.Lock()
defer mu.Unlock()
for _, key := range stalled {
if !said[key] && logf != nil {
said[key] = true
logf("the running pipeline %s has had nothing open and nothing newly placed for %s or more: runner_health names it unhealthy (H-A-311)", key, pipeline.StalledAfter)
}
}
return a, nil
}}
}
// StoresFill is each of the runner's stores' fill through the bundle's calls (design 47 G §366, §623, A-214): the states
@@ -154,7 +171,8 @@ func main() {
tools = append(tools, CancelTool(r.List, stdio.Ask), RerunTool(stdio.Ask))
tools = append(tools, VerbTools(stdio.Ask)...)
tools = append(tools, TestRunsTools(stdio.Ask)...)
tools = append(tools, HealthTool(r.Pipelines, kept{stdio.State(pipeline.StateConsumers)}, time.Now))
tools = append(tools, HealthTool(r.Pipelines, kept{stdio.State(pipeline.StateConsumers)}, time.Now,
func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-pipeline-tools] "+f+"\n", a...) }))
tools = append(tools, WaitingTool(kept{stdio.State(seat.StateQueue)}))
if err := stdio.Serve("", tools); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-pipeline-tools] %v\n", err)
@@ -3,6 +3,7 @@ package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
@@ -37,7 +38,7 @@ func TestTools(t *testing.T) {
names = append(names, tl.Name)
}
// RN25 (§468, §76; A-164): the health check is the bundle's runner_health.
names = append(names, HealthTool(none{}, none{}, nil).Name, WaitingTool(none{}).Name)
names = append(names, HealthTool(none{}, none{}, nil, nil).Name, WaitingTool(none{}).Name)
// §606, §549: the unlogged entries are listed and appended by hand.
for _, tl := range UnloggedTools(stdio.State("pipelines"), stdio.Log("step-log"), nil, nil) {
names = append(names, tl.Name)
@@ -53,6 +54,22 @@ func TestTools(t *testing.T) {
if got, _ := json.Marshal(out); err != nil || string(got) != `[{"sequence":1,"pipeline":"main.5"},{"sequence":3,"pipeline":"main.7"}]` {
t.Errorf("runner_waiting = %s (%v)", got, err)
}
// H-A-311 (3), H-A-313: runner_health names a running pipeline quiet five minutes, and this program's journal says it
// once per key, however often the health is asked.
now := time.Date(2026, 11, 21, 20, 35, 0, 0, time.UTC)
quiet := queueOf{"main.137": `{"pipeline":"main.137","branch":"main","outcome":"running","source":137,"started":"2026-11-21T20:15:00Z",` +
`"movers":[{"module":"app"}],"steps":[{"module":"app","step":"build","outcome":"passed","status":"required","started":"2026-11-21T20:15:52Z","ended":"2026-11-21T20:17:55Z"}]}`}
var journal []string
health := HealthTool(quiet, none{}, func() time.Time { return now }, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
for range 2 {
out, err := health.Run(nil)
if got, _ := json.Marshal(out); err != nil || !strings.Contains(string(got), "main.137 (nothing open and nothing placed for 17m5s)") {
t.Errorf("runner_health = %s (%v), want main.137 named stalled", got, err)
}
}
if len(journal) != 1 || !strings.Contains(journal[0], "main.137") {
t.Errorf("the journal said %q, want main.137 said once", journal)
}
}
// queueOf is a state queue in memory, its keys in no order.
@@ -179,6 +179,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
starter.Supersede = func(newer seat.Pipeline) { hearer.SupersedeRunning(newer) }
starter.Left = hearer.LeftInPlace // a same-source re-pin's superseded steps leave their runs (H-A-284)
starter.Revision = widener.Revision // a kept mover is placed from its kept revision (H-A-286)
// A mover's wait on a package is read first: a package published before the wait places it at once (H-A-326, H-A-328).
starter.AskPublished = publishedOf(boundedCalls(calls.Call, readBound, time.After, logger.Printf))
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.AskPinHeld = func(key, module string, pins map[string]string) {
if _, err := starter.AskPinHeld(hearer.Store, key, module, pins, widener.Revision); err != nil {
@@ -0,0 +1,29 @@
package main
import (
"encoding/json"
"fmt"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// publishedOf asks the seat mesh-builds' verb published through call whether a package built at a fingerprint is published
// (design 47 H-A-326 (1), H-A-328), asked when a mover's wait on it is written.
func publishedOf(call func(string, any) (json.RawMessage, error)) func(builds.PublishedInput) (builds.PublishedAnswer, error) {
return func(in builds.PublishedInput) (builds.PublishedAnswer, error) {
raw, err := call("seat:"+builds.Seat+"."+builds.VerbPublished, in)
if err != nil {
return builds.PublishedAnswer{}, err
}
answer, said, ok := buscalls.Unwrap(raw)
if !ok {
return builds.PublishedAnswer{}, fmt.Errorf("%s answered: %s", builds.VerbPublished, said)
}
var a builds.PublishedAnswer
if err := json.Unmarshal(answer, &a); err != nil {
return builds.PublishedAnswer{}, fmt.Errorf("%s answered what does not read: %w", builds.VerbPublished, err)
}
return a, nil
}
}
@@ -176,7 +176,7 @@ func (v Verbs) AskUnanswered(onSeat, ask string, again bool) error {
return nil // its second bound ends the step
}
v.Starter.carry(key, []Asked{*a})
return askAll(v.Starter.Steps, key, []Asked{*a}, func(commit string) bool { return v.Starter.recovered(key, commit) }, nil)
return askAll(v.Starter.Steps, key, []Asked{*a}, func(commit string) bool { return v.Starter.recovered(key, commit) }, nil, nil)
}
// HolderStarted puts again every open ask of each silent seat module holds (H-A-243 (1)): its holder takes its asks on a plain
@@ -234,7 +234,7 @@ func (v Verbs) HolderStarted(module string) (int, error) {
continue
}
st.carry(key, due)
if err := askAll(st.Steps, key, due, func(commit string) bool { return st.recovered(key, commit) }, nil); err != nil && first == nil {
if err := askAll(st.Steps, key, due, func(commit string) bool { return st.recovered(key, commit) }, nil, nil); err != nil && first == nil {
first = err
}
asked += len(due)
@@ -1,6 +1,7 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
@@ -20,7 +21,7 @@ func TestAnAskNobodyAnswersIsAskedOnceMoreThenEndsItsStep(t *testing.T) {
v, store, said, asks, _, _ := verbsFixture()
var armed, unbound, raised, cleared []string
logs := []string{}
v.Hearer.Logf = func(f string, a ...any) { logs = append(logs, f) }
v.Hearer.Logf = func(f string, a ...any) { logs = append(logs, fmt.Sprintf(f, a...)) }
c := &AskClock{Bound: 10 * time.Minute,
Arm: func(key string, ttl time.Duration) error { armed = append(armed, key+" "+ttl.String()); return nil },
Unbound: func(key string) error { unbound = append(unbound, key); return nil },
@@ -83,6 +84,10 @@ func TestAnAskNobodyAnswersIsAskedOnceMoreThenEndsItsStep(t *testing.T) {
if s.Outcome != seat.StepResultOutcomeCouldNotRun || s.Why != "its ask "+ask+" was not answered" || s.Ended == nil {
t.Errorf("the step = %+v", s)
}
// The unanswered-bound end is one journal line, as every said end (H-A-320 (1), H-A-333).
if n := countLines(logs, pkey+" app lint ended could not run (no node, "); n != 1 {
t.Errorf("the unanswered end journaled %d times in %q, want once", n, logs)
}
if got.Outcome != seat.PipelineOutcomeFailed || len(asks.ids) != 2 {
t.Errorf("the pipeline is %s (%s), asked %d, want failed and nothing more asked", got.Outcome, got.Why, len(asks.ids))
}
@@ -7,6 +7,7 @@ import (
"strings"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
)
@@ -19,13 +20,83 @@ const StateConsumers = process.ConsumersState
// behind (§76).
const BehindAfter = 5 * time.Minute
// Health is the runner's health as its tool runner_health answers it (§468, §76, §495; A-164): healthy when its state
// pipelines answers, as show reads it, and each durable consumer's oldest unacknowledged message waits less than
// BehindAfter; otherwise unhealthy naming each consumer behind and how long.
func Health(pipelines, consumers Kept, now time.Time) health.Answer {
if _, err := pipelines.Keys(); err != nil {
return health.Unhealthy(fmt.Sprintf("The runner does not answer show: its state pipelines cannot be listed (%v).", err))
// StalledAfter is how long a running pipeline may have nothing open and nothing newly placed before the runner's health
// names it stalled (H-A-311 (3)): the mesh's own rule for nothing moving (nothing-moved-in-five-minutes), not a setting
// (H-A-322).
const StalledAfter = 5 * time.Minute
// Stalled is the key of each running pipeline, as show reads it, that has had nothing open (no step asked and not ended,
// no required step waiting but on a package's publish) and nothing newly placed, started or ended for StalledAfter at now (H-A-311 (3)), each with
// how long, in key order.
func Stalled(pipelines Kept, now time.Time) ([]string, error) {
all, err := Reads{Pipelines: pipelines}.pipelines()
if err != nil {
return nil, err
}
var out []string
for _, p := range all {
if p.Outcome != seat.PipelineOutcomeRunning {
continue
}
last, busy := startedOf(p), false
for _, st := range p.Steps {
// A required step waiting (an approval, a composition on its validate) is open, as the end's judgement holds it;
// an optional one waiting holds nothing (A-104), and neither does one waiting on a package's publish, as a mover
// with no steps waiting on one does not (H-A-326 (2)).
if open(st) || (st.Outcome == seat.StepResultOutcomeWaiting && st.Status == seat.StepResultStatusRequired && !pinHeld(st)) {
busy = true
break
}
for _, at := range []*time.Time{st.Started, st.Ended} {
if at != nil && at.After(last) {
last = *at
}
}
}
if busy || last.IsZero() || now.Sub(last) < StalledAfter {
continue
}
out = append(out, fmt.Sprintf("%s (nothing open and nothing placed for %s)", p.Pipeline, now.Sub(last).Round(time.Second)))
}
return out, nil
}
// Health is the runner's health as its tool runner_health answers it (§468, §76, §495; A-164): healthy when its state
// pipelines answers, as show reads it, each durable consumer's oldest unacknowledged message waits less than
// BehindAfter, and no running pipeline is stalled (H-A-311 (3)); otherwise unhealthy naming each consumer behind and
// how long, and each stalled pipeline by its key.
func Health(pipelines, consumers Kept, now time.Time) health.Answer {
a, _ := HealthStalled(pipelines, consumers, now)
return a
}
// HealthStalled is Health with the keys of the running pipelines it names stalled, so the tools program says each in its
// journal once (H-A-313).
func HealthStalled(pipelines, consumers Kept, now time.Time) (health.Answer, []string) {
if _, err := pipelines.Keys(); err != nil {
return health.Unhealthy(fmt.Sprintf("The runner does not answer show: its state pipelines cannot be listed (%v).", err)), nil
}
stalled, err := Stalled(pipelines, now)
if err != nil {
return health.Unhealthy(fmt.Sprintf("The runner's running pipelines cannot be read as show reads them (%v).", err)), nil
}
var keys []string
for _, s := range stalled {
keys = append(keys, strings.SplitN(s, " ", 2)[0])
}
a := consumersHealth(consumers, now)
if len(stalled) == 0 {
return a, nil
}
why := fmt.Sprintf("The runner's running pipelines with nothing open and nothing newly placed for %s or more: %s.", StalledAfter, strings.Join(stalled, ", "))
if !a.Healthy {
why = a.Why + " " + why
}
return health.Unhealthy(why), keys
}
// consumersHealth is the part of Health its consumers' ages give.
func consumersHealth(consumers Kept, now time.Time) health.Answer {
keys, err := consumers.Keys()
if err != nil {
return health.Unhealthy(fmt.Sprintf("The ages of the runner's consumers cannot be read from its state %s (%v).", StateConsumers, err))
@@ -7,6 +7,7 @@ import (
"testing"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/process"
)
@@ -50,6 +51,19 @@ func TestHealth(t *testing.T) {
{"six minutes names the consumer", mapKept{}, age("n_runner", 6*time.Minute), false, "n_runner (its oldest unacknowledged message waits 6m0s)"},
{"show does not answer", failingKept{}, age("n_runner", 0), false, "does not answer show"},
{"no event heard yet", mapKept{}, mapKept{}, true, "has heard no event yet"},
// H-A-311 (3): a running pipeline with nothing open and nothing newly placed for five minutes is named by its key.
{"a pipeline quiet six minutes is named", piped(now, "main.137", -6*time.Minute, seat.StepResultOutcomePassed), age("n_runner", 0), false,
"main.137 (nothing open and nothing placed for 6m0s)"},
{"a pipeline quiet four minutes is not", piped(now, "main.137", -4*time.Minute, seat.StepResultOutcomePassed), age("n_runner", 0), true, ""},
{"a pipeline with a step open is not", piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeRunning), age("n_runner", 0), true, ""},
// H-A-285, H-A-309: a record carrying the runner's kept-revisions is read, not refused.
{"a quiet pipeline keeping revisions is named", kept(piped(now, "main.131", -6*time.Minute, seat.StepResultOutcomePassed), "main.131"), age("n_runner", 0), false,
"main.131 (nothing open and nothing placed for 6m0s)"},
// A required step waiting (an approval on the operator) is open, as the end's judgement holds it.
{"a pipeline whose approval waits is not", piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeWaiting), age("n_runner", 0), true, ""},
// H-A-326 (2): a pipeline whose only movers wait on packages (a step held on a publish, a mover with no steps) is named.
{"a pipeline waiting only on packages is named", heldOnPublish(piped(now, "main.137", -20*time.Minute, seat.StepResultOutcomeWaiting), "main.137"), age("n_runner", 0), false,
"main.137 (nothing open and nothing placed for 20m0s)"},
}
for _, r := range rows {
a := Health(r.pipelines, r.consumers, now)
@@ -58,3 +72,41 @@ func TestHealth(t *testing.T) {
}
}
}
// piped is the state pipelines holding one running pipeline key whose one step was placed and last moved at now+moved,
// with outcome.
func piped(now time.Time, key string, moved time.Duration, outcome string) mapKept {
at, started := now.Add(moved), now.Add(-time.Hour)
st := seat.StepResult{Module: "app", Step: seat.StepResultStepBuild, Outcome: outcome, Status: seat.StepResultStatusRequired, Started: &at}
if outcome == seat.StepResultOutcomeWaiting {
st.Step, st.On = seat.StepResultStepApproval, seat.WaitOnPerson
} else if outcome != seat.StepResultOutcomeRunning {
st.Ended = &at
}
raw, _ := json.Marshal(seat.Pipeline{Pipeline: key, Branch: "main", Outcome: seat.PipelineOutcomeRunning, Started: &started,
Movers: []seat.Mover{{Module: "app"}}, Steps: []seat.StepResult{st}})
return mapKept{key: raw}
}
// kept is m with key's record carrying the runner's kept-revisions (H-A-285).
func kept(m mapKept, key string) mapKept {
raw, err := withKept(m[key], map[string]int64{"app": 130})
if err != nil {
panic(err)
}
m[key] = raw
return m
}
// heldOnPublish is m with key's one step held on a package's publish (H-A-206) and a second mover with no steps, waiting on one.
func heldOnPublish(m mapKept, key string) mapKept {
var p seat.Pipeline
if err := json.Unmarshal(m[key], &p); err != nil {
panic(err)
}
p.Steps[0].Step, p.Steps[0].On, p.Steps[0].Why = seat.StepResultStepBuild, "", PinWait+"mesh-grants:busgrants"
p.Movers = append(p.Movers, seat.Mover{Module: "nats"})
raw, _ := json.Marshal(p)
m[key] = raw
return m
}
@@ -1,8 +1,10 @@
package pipeline
import (
"errors"
"strings"
"testing"
"time"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -108,8 +110,11 @@ func TestThePipelinePassesOnNothingToRunAndFailsOnOneMover(t *testing.T) {
// R8 (design 47 H §746, H-A-52, H-A-62): a version-released writes its version into the mover of that module in the pipeline
// of the source revision built, on the branch it was released for, so pipeline-ended names it; another branch's pipeline,
// and a version heard again, write nothing.
//
// H-A-270 (2), H-A-277 (2), (3): a version written into a pipeline that ended passed says its pipeline-ended again, naming the
// version; one still running, or ended failed, says nothing. A say that failed is said when the version is heard again.
func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
h, store, _, _ := twoMovers(t)
h, store, bodies, order := twoMovers(t)
other := store[pkey]
other.p.Pipeline, other.p.Branch, other.p.Movers = "beta.5", "beta", append([]seat.Mover(nil), other.p.Movers...)
store["beta.5"] = other
@@ -120,9 +125,29 @@ func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
again.p.Pipeline, again.p.Source, again.p.Movers = key, source, append([]seat.Mover(nil), again.p.Movers...)
store[key] = again
}
at := time.Unix(1, 0)
for key, outcome := range map[string]string{pkey: seat.PipelineOutcomePassed, "beta.5": seat.PipelineOutcomeFailed} {
done := store[key]
done.p.Ended, done.p.Outcome = &at, outcome
store[key] = done
}
say := h.Say
fail := true
h.Say = func(subject string, body any) error {
if fail {
fail = false
return errors.New("the bus is away")
}
return say(subject, body)
}
ev := catalog.VersionReleasedEvent{Module: "app", Version: "0.4.1", Source: 5}
if n, err := h.Released(ev, "main"); n != 2 || err != nil {
t.Fatalf("released wrote %d (%v), want app's mover of main.5 and main.8", n, err)
first, err := h.Released(ev, "main")
if err == nil || len(*order) != 0 {
t.Fatalf("released wrote %d (%v) and said %v, want the failed say returned", first, err, *order)
}
// Heard again (the nak's redelivery), main.5's version is held already and its end is said (H-A-277 (3)).
if n, err := h.Released(ev, "main"); first+n != 2 || err != nil {
t.Fatalf("released wrote %d, then %d again (%v), want app's mover of main.5 and main.8", first, n, err)
}
if m, _ := MoverOf(store["main.6"].p, "app"); m.Version != "" {
t.Errorf("main.6's mover = %+v, want no version: another source", m)
@@ -133,8 +158,19 @@ func TestAReleasedVersionIsTheMoversVersion(t *testing.T) {
if m, _ := MoverOf(store["beta.5"].p, "app"); m.Version != "" {
t.Errorf("beta's mover = %+v, want no version: released for main", m)
}
if n, _ := h.Released(ev, "main"); n != 0 {
t.Errorf("the version heard again wrote %d", n)
pe, ok := bodies["pipeline-ended.main.5"].(seat.PipelineEnded)
if !ok || len(pe.Movers) != 2 || pe.Movers[1].Module != "app" || pe.Movers[1].Version != "0.4.1" {
t.Errorf("pipeline-ended.main.5 = %+v, want said again naming app at 0.4.1 (said %v)", bodies["pipeline-ended.main.5"], *order)
}
if len(*order) != 1 {
t.Errorf("said %v, want only main.5's end again: main.8 has not ended", *order)
}
if n, _ := h.Released(ev, "main"); n != 0 || len(*order) != 2 {
t.Errorf("the version heard once more wrote %d and said %v, want main.5's end said again", n, *order)
}
// beta's failed pipeline, its version written, is never said again (H-A-277 (2)).
if n, _ := h.Released(ev, ""); n != 1 || len(*order) != 3 {
t.Errorf("released for every branch wrote %d and said %v, want beta's mover written and main.5's end alone said", n, *order)
}
if got := pipelineEnded(store[pkey].p).Movers[1]; got.Module != "app" || got.Version != "0.4.1" {
t.Errorf("pipeline-ended names %+v, want app at 0.4.1", got)
@@ -332,6 +332,20 @@ func keptOf(raw []byte) (map[string]int64, error) {
return kept, nil
}
// withoutKept is record without its kept-revisions, the runner's own property and never part of the seat's Go form, so the
// seat's strict decode of the record the runner's reads make refuses no record carrying it (H-A-285, H-A-309).
func withoutKept(record []byte) ([]byte, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(record, &fields); err != nil {
return nil, fmt.Errorf("the record is no object: %w", err)
}
if _, ok := fields[keptRevisionsField]; !ok {
return record, nil
}
delete(fields, keptRevisionsField)
return json.Marshal(fields)
}
// withKept is record with kept as its kept-revisions.
func withKept(record []byte, kept map[string]int64) ([]byte, error) {
var fields map[string]json.RawMessage
@@ -139,13 +139,16 @@ var buildKinds = []int64{3, 1}
// awaited are the artifact references mover m's steps wait for in p (§746, H-A-51): the From of each edge of a build kind
// into m from another mover of p, toolchain edges first, each once. A toolchain edge (kind 3) from a build module whose
// own new version is built from m is not awaited: what a build module's new version is built from is built by its
// running version, never waited for (G §270, A-326), so the running build module builds m and the new one the rest.
// running version, never waited for (G §270, A-326), so the running build module builds m and the new one the rest. A
// mover that left p for a newer pipeline is not awaited either: m builds against the newest published package (H-A-53),
// and its build ended superseded is no dependency that did not build (H-A-361).
func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) []catalog.Edge {
var out []catalog.Edge
for _, kind := range buildKinds {
for _, x := range entries[m].Edges {
from := moduleOf(x.From)
if x.Kind != kind || from == m || !Moves(p, from) || slices.ContainsFunc(out, func(e catalog.Edge) bool { return e.From == x.From }) {
if x.Kind != kind || from == m || !Moves(p, from) || slices.Contains(p.Superseded, from) ||
slices.ContainsFunc(out, func(e catalog.Edge) bool { return e.From == x.From }) {
continue
}
if kind == 3 && builtFrom(p, from, entries)[m] {
@@ -1,6 +1,7 @@
package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
@@ -94,6 +95,23 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
return e
}
before := mem["main.7"]
// H-A-311 (2): the build's end written while its placing was cut (heard by a hearer that places nothing): the same end
// heard again, late, moves nothing of the build and still places tool, which that end unlocked.
cut := *h
cut.Release = nil
if _, err := cut.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
if p, _, _ = mem.Get("main.7"); placed(p, "tool") {
t.Fatalf("the cut placing placed tool: %+v", stepsOf(p, "tool"))
}
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
if p, _, _ = mem.Get("main.7"); !placed(p, "tool") || stepsOf(p, "tool")[0].Outcome != seat.StepResultOutcomeRunning {
t.Errorf("after builder's build passed, heard late, tool's steps = %+v, want placed and asked", stepsOf(p, "tool"))
}
mem["main.7"] = before
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", end("passed")); err != nil {
t.Fatal(err)
}
@@ -124,6 +142,172 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
t.Errorf("tool's step-ended = %+v, want not asked naming builder", ev.Result)
}
}
// H-A-326 (1), (3), H-A-328: a package published before its importer's wait was written (by an earlier pipeline) is read of
// the build seat when the wait is written, matched by its mover's fingerprint, and the importer is placed with no event;
// the journal says the wait was resolved from the build seat's published, and a retried placing asks it once.
mem2 := memStore{}
var asked []builds.PublishedInput
var journal []string
s2 := &Starter{Pipelines: memCreated{mem2}, Store: mem2, Say: func(string, any) error { return nil }, Steps: &stepAsks{}, Name: "the-runner",
Registration: func(string) (catalog.Registration, bool) { return reg, true },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) },
AskPublished: func(in builds.PublishedInput) (builds.PublishedAnswer, error) {
asked = append(asked, in)
return builds.PublishedAnswer{Published: in.Artifact == "mesh-sdk:go" && in.Fingerprint == "src2:00", Version: "0.2.0"}, nil
}}
if n, err := s2.TakenIn(ev, view); n != 1 || err != nil {
t.Fatalf("the take-in started %d (%v)", n, err)
}
p, _, _ = mem2.Get("main.7")
if sdk, _ := MoverOf(p, "mesh-sdk"); !placed(p, "app") || placed(p, "tool") || !slices.Contains(sdk.Published, "mesh-sdk:go") {
t.Errorf("with mesh-sdk:go published before the wait: app placed %v, tool placed %v, mesh-sdk %+v; want app placed with no event", placed(p, "app"), placed(p, "tool"), sdk)
}
if !slices.ContainsFunc(journal, func(l string) bool {
return strings.Contains(l, "app's wait on mesh-sdk:go") && strings.Contains(l, "placed from the published record")
}) {
t.Errorf("the journal %q does not say app's wait was resolved from the build seat's published", journal)
}
before2 := len(asked)
if _, err := s2.Waiting(mem2, "main", view); err != nil {
t.Fatal(err)
}
if len(asked) != before2 {
t.Errorf("a placing again asked published %d more times, want each package asked once", len(asked)-before2)
}
// At the pipeline's end its kept answers go.
if _, err := s2.Turn("main.7", view); err != nil || len(s2.publishedRead) != 0 {
t.Errorf("after main.7's end %d answers are kept (%v), want none", len(s2.publishedRead), err)
}
// H-A-361 (main.158 at 22:51, main.163 at 23:00 of 2026-11-21), H-A-53, H-A-223: a newer pipeline takes mesh-sdk and
// builder, whose builds run, and web, not placed yet; app and tool, which awaited them, stay. They are placed and asked
// at the hand-off, building against the newest published package, never ended not asked as if their dependency had
// failed; web is not placed here; the journal says the hand-off and what stays; a later placing and the end judgement
// leave the pipeline running until its kept steps end.
t.Run("a hand-off places the kept movers that awaited a mover that left", func(t *testing.T) {
web := entries["app"]
web.Module, web.Edges = "web", []catalog.Edge{{From: "mesh-sdk:go", To: "web:program", Kind: 1}}
entries["web"] = web
mem := memStore{}
s := &Starter{Pipelines: memCreated{mem}, Store: mem, Say: func(string, any) error { return nil }, Steps: &stepAsks{}, Name: "the-runner",
Registration: func(string) (catalog.Registration, bool) { return reg, true }}
ev := ev
ev.Entries = append(slices.Clone(ev.Entries), catalog.TakeInEntry{Module: "web", Commit: "c4", Fingerprint: "src2:dd"})
if n, err := s.TakenIn(ev, view); n != 1 || err != nil {
t.Fatalf("the take-in started %d (%v)", n, err)
}
var journal []string
h := &Hearer{Store: mem, Name: "the-runner", Say: func(string, any) error { return nil },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) },
Release: func(key string) { _, _ = s.Release(mem, key, view) }}
h.SupersedeRunning(seat.Pipeline{Pipeline: "main.9", Branch: "main", Revision: 9, Movers: []seat.Mover{
{Module: "mesh-sdk", Commit: "c5", Fingerprint: "src2:01"}, {Module: "builder", Commit: "c5", Fingerprint: "src2:b2"},
{Module: "web", Commit: "c5", Fingerprint: "src2:d2"}}})
check := func(when string) {
p, _, _ := mem.Get("main.7")
if p.Outcome != seat.PipelineOutcomeRunning {
t.Fatalf("%s: main.7 = %s (%s), want running until its kept steps end", when, p.Outcome, p.Why)
}
for _, m := range []string{"app", "tool"} {
st := stepsOf(p, m)
if len(st) == 0 || st[0].Step != seat.StepResultStepBuild || st[0].Outcome != seat.StepResultOutcomeRunning || st[0].Ask == "" {
t.Errorf("%s: %s's steps = %+v, want its build placed and asked", when, m, st)
}
}
if st := stepsOf(p, "app"); len(st) > 0 && !strings.Contains(st[0].Why, "mesh-sdk") {
t.Errorf("%s: app's build why %q does not say mesh-sdk was taken out (H-A-53)", when, st[0].Why)
}
if placed(p, "web") {
t.Errorf("%s: web, which left, is placed in main.7: %+v", when, stepsOf(p, "web"))
}
}
check("at the hand-off")
if !slices.ContainsFunc(journal, func(l string) bool {
return strings.Contains(l, "main.7: ") && strings.Contains(l, "leave it") && strings.Contains(l, "stay: app, tool")
}) {
t.Errorf("the journal %q does not say the hand-off and that app and tool stay", journal)
}
// A placing heard later (a publish, a step end) and the end judgement again: still running.
if _, err := s.Release(mem, "main.7", view); err != nil {
t.Fatal(err)
}
p, rev, _ := mem.Get("main.7")
if endPipeline(&p, time.Now(), nil) {
_, _ = mem.Update("main.7", p, rev)
}
check("after a later placing and the end judgement")
})
// H-A-361, the reverse order (21:50:16 of 2026-11-21, mesh-contracts' old build withdrawn): builder's build ends without
// passing first, so tool, awaiting it, and ext, awaiting tool, are placed not asked; only then does a newer pipeline take
// builder away. At the hand-off tool and ext, not asked on a dependency that left (ext through tool), are taken out and
// placed again: tool is asked, building against the newest published package, ext waits for tool's build; the journal
// says it, and a later placing and the end judgement leave the pipeline running.
t.Run("a build that did not pass, then a hand-off, places again the movers not asked on the mover that left", func(t *testing.T) {
ext := entries["app"]
ext.Module, ext.Edges = "ext", []catalog.Edge{{From: "tool:program", To: "ext:program", Kind: 1}}
entries["ext"] = ext
mem := memStore{}
s := &Starter{Pipelines: memCreated{mem}, Store: mem, Say: func(string, any) error { return nil }, Steps: &stepAsks{}, Name: "the-runner",
Registration: func(string) (catalog.Registration, bool) { return reg, true }}
ev := ev
ev.Entries = append(slices.Clone(ev.Entries), catalog.TakeInEntry{Module: "ext", Commit: "c6", Fingerprint: "src2:ee"})
if n, err := s.TakenIn(ev, view); n != 1 || err != nil {
t.Fatalf("the take-in started %d (%v)", n, err)
}
var journal []string
h := &Hearer{Store: mem, Name: "the-runner", Say: func(string, any) error { return nil },
Logf: func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) },
Release: func(key string) { _, _ = s.Release(mem, key, view) }}
p, _, _ := mem.Get("main.7")
b := stepsOf(p, "builder")[0]
e, err := bs.Decode([]byte(`{"id":"` + b.Ask + `","pipeline":"main.7","step":"build","node":"n1","outcome":"failed","entries":[{"module":"builder","commit":"c2","fingerprint":"src2:bb"}]}`))
if err != nil {
t.Fatal(err)
}
if _, err := h.OnStepEnded(bs, bs.Prefix+"n1", e); err != nil {
t.Fatal(err)
}
// A later placing places ext, whose dependency tool was not asked, not asked too.
if _, err := s.Release(mem, "main.7", view); err != nil {
t.Fatal(err)
}
p, _, _ = mem.Get("main.7")
for _, m := range []string{"tool", "ext"} {
if st := stepsOf(p, m); len(st) == 0 || st[0].Outcome != seat.StepResultOutcomeNotAsked {
t.Fatalf("after builder's build failed, %s's steps = %+v, want placed not asked", m, st)
}
}
h.SupersedeRunning(seat.Pipeline{Pipeline: "main.9", Branch: "main", Revision: 9, Movers: []seat.Mover{
{Module: "builder", Commit: "c5", Fingerprint: "src2:b2"}}})
check := func(when string) {
p, _, _ := mem.Get("main.7")
if p.Outcome != seat.PipelineOutcomeRunning {
t.Fatalf("%s: main.7 = %s (%s), want running until its kept steps end", when, p.Outcome, p.Why)
}
if st := stepsOf(p, "tool"); len(st) == 0 || st[0].Step != seat.StepResultStepBuild || st[0].Outcome != seat.StepResultOutcomeRunning || st[0].Ask == "" {
t.Errorf("%s: tool's steps = %+v, want its build placed again and asked", when, st)
}
if placed(p, "ext") {
t.Errorf("%s: ext's steps = %+v, want it waiting for tool's build, not placed", when, stepsOf(p, "ext"))
}
}
check("at the hand-off")
if !slices.ContainsFunc(journal, func(l string) bool {
return strings.Contains(l, "main.7: tool, ext, not asked on a dependency that left, placed again")
}) {
t.Errorf("the journal %q does not say tool and ext are placed again", journal)
}
if _, err := s.Release(mem, "main.7", view); err != nil {
t.Fatal(err)
}
p, rev, _ := mem.Get("main.7")
if endPipeline(&p, time.Now(), nil) {
_, _ = mem.Update("main.7", p, rev)
}
check("after a later placing and the end judgement")
})
}
// Design 47 G §270, A-326 (H §746, H-A-83 (2)): what a build module's new version is built from is built by its running
@@ -446,8 +446,10 @@ func stepAsk(p seat.Pipeline, m seat.Mover, e catalog.CatalogEntry, a catalog.Ar
}
// askAll puts each ask on its seat's work queue; the first refusal is named, the rest are still put. arm, when not nil, arms
// each ask's bound before it is put (H-A-235, H-A-238 (5)); an ask whose bound is not armed is not put.
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool, arm func(Asked) error) error {
// each ask's bound before it is put (H-A-235, H-A-238 (5)); an ask whose bound is not armed is not put. logf, when not nil,
// writes one journal line per ask put, naming the pipeline, the mover, the step and the seat (H-A-320 (1)); a caller that
// says the ask in a line of its own (an ask put again) gives none.
func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit string) bool, arm func(Asked) error, logf func(string, ...any)) error {
var first error
for _, a := range asks {
accept, ok := acceptStep[a.Seat]
@@ -464,6 +466,10 @@ func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit stri
if err == nil {
err = asker.AskStep(accept, a.ID, a.Ask)
}
if err == nil && logf != nil {
mover, step := askedStep(a)
logf("%s %s %s asked on %s", key, mover, step, a.Seat)
}
if err != nil && first == nil {
first = fmt.Errorf("the step %s of %s is not asked of %s: %w", a.ID, key, a.Seat, err)
}
@@ -471,6 +477,31 @@ func askAll(asker StepAsker, key string, asks []Asked, recovery func(commit stri
return first
}
// askedStep is the mover and the step an ask names, read from its body; "-" where it names none (the validate ask names no
// mover, H-A-51).
func askedStep(a Asked) (mover, step string) {
switch b := a.Ask.(type) {
case nodebuildgo.StepAsk:
if b.Step != nil {
mover, step = b.Step.Module, b.Step.Step
}
case builds.StepRun:
step = b.Step
if len(b.Entries) == 1 {
mover = b.Entries[0].Module
}
case wants.StepRun:
mover, step = b.Module, b.Step
}
if mover == "" {
mover = "-"
}
if step == "" {
step = "-"
}
return mover, step
}
// artifactPaths are the named paths the entry keeps for the artifact, by its reference or its name.
func artifactPaths(e catalog.CatalogEntry, ref string) []string {
if e.Inputs == nil {
@@ -236,6 +236,7 @@ func (s *Starter) StartQueue(view func(branch string) (Half, bool)) (int, error)
// Turn takes the end of pipeline key (its pipeline-ended, a cancel alike, H-A-53): its running key is deleted and the
// queue placed (PlaceQueued).
func (s *Starter) Turn(key string, view func(branch string) (Half, bool)) (int, error) {
s.forgetPublished(key) // the ended pipeline's published answers go with it (H-A-328)
if s.Queue == nil {
return 0, nil
}
@@ -206,6 +206,10 @@ func (r Reads) pipelineOf(key string, all []string) (seat.Pipeline, bool, error)
if raw == nil {
return seat.Pipeline{}, false, nil
}
raw, err = withoutKept(raw)
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s: %w", seat.StatePipelines, key, err)
}
p, err := seat.DecodePipelinesByBranchRevisionValue(raw)
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s: %w", seat.StatePipelines, key, err)
@@ -53,6 +53,12 @@ func TestReads(t *testing.T) {
// A pipeline kept as H-A-187 keeps it: its record's steps empty, each mover's under steps.<pipeline>.<mover>.
put(t, pipes, "next.8", seat.Pipeline{Pipeline: "next.8", Branch: "next", Source: 8, Outcome: seat.PipelineOutcomeRunning,
Started: &t0, Steps: []seat.StepResult{}, Movers: []seat.Mover{{Module: "delta", Commit: "n8"}, {Module: "epsilon", Commit: "n8"}}})
// H-A-285, H-A-309: the record carries the runner's own kept-revisions, which show, list and stands read past.
withRevs, err := withKept(pipes["next.8"], map[string]int64{"delta": 7})
if err != nil {
t.Fatal(err)
}
pipes["next.8"] = withRevs
put(t, pipes, "steps.next.8.epsilon", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "epsilon",
Step: seat.StepResultStepBuild, Status: "required", Outcome: seat.StepResultOutcomePassed, Output: "next.8.build.epsilon:program"}}})
put(t, pipes, "steps.next.8.delta", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "delta",
@@ -5,13 +5,15 @@ import (
"fmt"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// Released writes a version-released's version into the entry of the mover it names (design 47 H §746, H-A-52): the mover of
// that module in each pipeline of the source revision built, a run's aside, so its pipeline-ended names the released version.
// branch, when not empty, is the one watched branch the version was released for (H-A-62); empty matches every branch's
// pipeline of that source. It says how many movers it wrote.
// pipeline of that source. A pipeline that ended passed says its end again, naming the version, and again whenever the version
// is heard again (H-A-270 (2), H-A-277). It says how many movers it wrote.
func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int, error) {
if ev.Module == "" || ev.Version == "" || ev.Source <= 0 {
return 0, fmt.Errorf("a version-released names its module, version and source revision: %+v", ev)
@@ -46,7 +48,15 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
i = j
}
}
if i < 0 || p.Movers[i].Version == ev.Version {
if i < 0 {
break
}
if p.Movers[i].Version == ev.Version {
// A version heard again says the end of an ended passed pipeline again (H-A-277 (3)): a say that failed after
// the version was written is said now; taking it again is a no-op for the holder.
if err := h.sayEndedAgain(key, p, ev); err != nil {
return wrote, err
}
break
}
p.Movers[i].Version = ev.Version
@@ -56,6 +66,11 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
return wrote, fmt.Errorf("the pipeline %s cannot be written: %w", key, err)
}
wrote++
// A version written after the pipeline's end was said says that end again (H-A-270 (2)): the same pipeline-ended,
// now naming the version, so the seat mesh-wants' holder takes the mover it skipped; taking a mover again is a no-op.
if err := h.sayEndedAgain(key, p, ev); err != nil {
return wrote, err
}
break
}
}
@@ -65,6 +80,20 @@ func (h *Hearer) Released(ev catalog.VersionReleasedEvent, branch string) (int,
return wrote, nil
}
// sayEndedAgain says p's pipeline-ended again when p ended passed (H-A-270 (2), H-A-277 (2)): only a passed end writes a want,
// so a superseded, cancelled or failed one is not said again.
func (h *Hearer) sayEndedAgain(key string, p seat.Pipeline, ev catalog.VersionReleasedEvent) error {
if p.Ended == nil || p.Outcome != seat.PipelineOutcomePassed {
return nil
}
if err := h.say(seat.EventPipelineEnded, p, nil); err != nil {
return fmt.Errorf("the version %s of %s is in the ended pipeline %s but its %s is not said again: %w",
ev.Version, ev.Module, key, seat.EventPipelineEnded, err)
}
h.logf("%s: the version %s of %s is written after its end: %s said again", key, ev.Version, ev.Module, seat.EventPipelineEnded)
return nil
}
// ReleasedBranch is the watched branch a module's versions are released for, its registration's first (H-A-62); empty when
// the registration is not held.
func ReleasedBranch(reg func(string) (catalog.Registration, bool), module string) string {
@@ -137,6 +137,7 @@ func (s *Starter) keepIn(key string, half Half, movers []seat.Mover) []seat.Move
}
was := p
was.Movers = olds
logEnded(s.logf, key, st)
if err := s.Say(stepEvent(seat.EventStepEnded, was, st, now)); err != nil {
s.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, st.Module, st.Step, err)
}
@@ -177,6 +178,9 @@ func (s *Starter) placeAgain(p *seat.Pipeline, m seat.Mover, half Half, now time
return nil, nil, true
}
due, waiting, failed := ready(*p, m.Module, half.Entries)
if !due && failed == "" && s.readPublished(p, m.Module, waiting) {
due, waiting, failed = ready(*p, m.Module, half.Entries)
}
if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
return nil, nil, false
@@ -185,7 +189,7 @@ func (s *Starter) placeAgain(p *seat.Pipeline, m seat.Mover, half Half, now time
supersededWhy(*p, m.Module, half.Entries, steps)
var notAsked []int
if failed != "" {
why := "its dependency " + failed + " did not build"
why := notBuiltWhy(failed)
for k := range steps {
st := &steps[k]
if st.Outcome == seat.StepResultOutcomeBypassed {
@@ -75,12 +75,18 @@ func IsRunBound(key string) bool {
// runOfAsk is the test run asked under ask: the run its asking step names in Shared, read from the state pipelines.
func (h *Hearer) runOfAsk(ask string) (string, bool) {
run, _, ok := h.runAndPipelineOfAsk(ask)
return run, ok
}
// runAndPipelineOfAsk is runOfAsk with the key of the pipeline whose step asked it.
func (h *Hearer) runAndPipelineOfAsk(ask string) (string, string, bool) {
if h.TestRuns == nil || ask == "" {
return "", false
return "", "", false
}
pkeys, err := allKeys(h.Store)
if err != nil {
return "", false
return "", "", false
}
for _, key := range pkeys {
p, _, err := h.Store.Get(key)
@@ -89,10 +95,26 @@ func (h *Hearer) runOfAsk(ask string) (string, bool) {
}
if i := stepOfAsk(p, ask); i >= 0 {
run := sharedRun(p.Steps[i])
return run, run != ""
return run, key, run != ""
}
}
return "", "", false
}
// releaseLate places what the end of a step heard late unlocks (H-A-311 (2)): a step-progress or step-end that arrives
// after its ask ended moves nothing of the step itself, yet the movers waiting on what that end gave (the dependents'
// builds) are placed, in each pipeline named, as the end heard in time would have placed them (Release).
func (h *Hearer) releaseLate(pipelines ...string) {
if h.Release == nil {
return
}
seen := map[string]bool{}
for _, key := range pipelines {
if key != "" && !seen[key] {
seen[key] = true
h.Release(key)
}
}
return "", false
}
// RunTaken takes the ask-taken of a test run's ask (§725, A-470): the node that took it is kept on the run, its untaken
@@ -158,13 +180,18 @@ func armBound(clock RunClock, runs TestRuns, key, run, ask string, ttl time.Dura
// RunProgress takes a build seat's step-progress of a test run's ask (§725, A-466p): its clock begins again, so a run that
// keeps saying its progress is never ended by it. An ask of no running, taken test run takes nothing.
func (h *Hearer) RunProgress(ask string) error {
run, ok := h.runOfAsk(ask)
run, key, ok := h.runAndPipelineOfAsk(ask)
if !ok {
return nil
}
r, _, err := h.TestRuns.Get(run)
if errors.Is(err, ErrNoRun) || (err == nil && (r.Ask != ask || r.Ended != nil)) {
h.logf("the step-progress of ask %s is late: the test run %s names ask %q (ended %v), so it moves nothing (A-470n)", ask, run, r.Ask, r.Ended != nil)
h.logf("the step-progress of ask %s is late: the test run %s names ask %q (ended %v), so it moves nothing of its own; what its end unlocks is placed (A-470n, H-A-311)", ask, run, r.Ask, r.Ended != nil)
late := []string{key}
for _, reader := range r.Readers {
late = append(late, reader.Pipeline)
}
h.releaseLate(late...)
return nil
}
if err != nil {
@@ -8,6 +8,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
runnerclient "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshpipelinerunnerclient"
@@ -133,6 +134,22 @@ func placed(p seat.Pipeline, module string) bool {
return false
}
// notBuiltWhy is the why of a step placed not asked because dependency failed did not build (H-A-101 (2)).
func notBuiltWhy(failed string) string { return "its dependency " + failed + " did not build" }
// notBuiltDependency answers the dependency a step placed not asked names as not built (notBuiltWhy), if it is one.
func notBuiltDependency(st seat.StepResult) (string, bool) {
if st.Outcome != seat.StepResultOutcomeNotAsked {
return "", false
}
d, ok := strings.CutPrefix(st.Why, "its dependency ")
if !ok {
return "", false
}
d, ok = strings.CutSuffix(d, " did not build")
return d, ok && d != ""
}
// Starter starts pipelines (design 47 G §54, §103): it writes a pipeline's key by compare-and-set on its absence, so one
// source revision and module has one pipeline whichever event starts it first, and says pipeline-started for the one it
// wrote.
@@ -199,6 +216,67 @@ type Starter struct {
// registration-missing (H-A-262); Observer is the runner's own module name. nil raises none.
Findings Findings
Observer string
// AskPublished asks the seat mesh-builds' verb published whether a package built at a fingerprint is published (H-A-326
// (1), H-A-328): read when a mover's wait on it is written, so a publish made before the wait places the mover without
// any event; nil reads nothing, the publish event alone placing it.
AskPublished func(in builds.PublishedInput) (builds.PublishedAnswer, error)
publishedMu sync.Mutex
publishedRead map[string]builds.PublishedAnswer
}
// readPublished asks, once per pipeline, package and fingerprint, the seat mesh-builds whether each package mover waits on
// in p is published already (H-A-326 (1), H-A-328): a wait is a read first and an event second. Each package answered
// published goes into its From mover's published in p, written with the placing, and the journal says the wait resolved
// from the build seat's published, apart from one a package-published places (H-A-326 (3)). The answer is kept for the
// placing's compare-and-set retries, so no retry asks again. It says whether it added any.
func (s *Starter) readPublished(p *seat.Pipeline, mover string, waiting []string) bool {
if s.AskPublished == nil {
return false
}
added := false
for _, ref := range waiting {
i := slices.IndexFunc(p.Movers, func(m seat.Mover) bool { return m.Module == moduleOf(ref) })
if i < 0 || slices.Contains(p.Movers[i].Published, ref) {
continue
}
in := builds.PublishedInput{Module: p.Movers[i].Module, Artifact: ref, Fingerprint: p.Movers[i].Fingerprint}
key := p.Pipeline + "\x00" + ref + "\x00" + in.Fingerprint
s.publishedMu.Lock()
a, asked := s.publishedRead[key]
s.publishedMu.Unlock()
if !asked {
var err error
if a, err = s.AskPublished(in); err != nil {
s.logf("%s: whether %s at %s is published is not read, so %s waits for its publish event: %v", p.Pipeline, ref, in.Fingerprint, mover, err)
continue
}
s.publishedMu.Lock()
if s.publishedRead == nil {
s.publishedRead = map[string]builds.PublishedAnswer{}
}
s.publishedRead[key] = a
s.publishedMu.Unlock()
}
if !a.Published {
continue
}
p.Movers[i].Published = append(p.Movers[i].Published, ref)
added = true
s.logf("%s: %s's wait on %s is resolved from the build seat's published (at %s), not from an event: placed from the published record (H-A-326)", p.Pipeline, mover, ref, a.Version)
}
return added
}
// forgetPublished drops the published answers readPublished kept for pipeline key, at its end (H-A-328).
func (s *Starter) forgetPublished(key string) {
s.publishedMu.Lock()
defer s.publishedMu.Unlock()
for k := range s.publishedRead {
if strings.HasPrefix(k, key+"\x00") {
delete(s.publishedRead, k)
}
}
}
// holderless ends at once, as it is placed, a pipeline with a build step whose seat the seat mesh-module-assigner's view
@@ -265,7 +343,7 @@ func (s *Starter) askSteps(key string, asks []Asked) error {
var err error
if len(asks) > 0 {
s.carry(key, asks)
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) }, s.Asks.arm)
err = askAll(s.Steps, key, asks, func(commit string) bool { return s.recovered(key, commit) }, s.Asks.arm, s.logf)
}
s.sayStarted(key, placed)
return err
@@ -347,7 +425,8 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
}
all := entries
for _, m := range p.Movers {
if placed(*p, m.Module) {
// A mover that left for a newer pipeline is placed there, never here (H-A-223, H-A-361).
if placed(*p, m.Module) || slices.Contains(p.Superseded, m.Module) {
continue
}
r, ok := reg(m.Module)
@@ -365,6 +444,9 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
// A mover's steps wait for the publish of each mover it builds on (§746, H-A-83 (2)): placed once each is heard,
// ended not asked when one's build ended without passing.
due, waiting, failed := ready(*p, m.Module, entries)
if !due && failed == "" && s.readPublished(p, m.Module, waiting) {
due, waiting, failed = ready(*p, m.Module, entries)
}
if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
waits = append(waits, m)
@@ -378,7 +460,7 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
}
}
if failed != "" {
why := "its dependency " + failed + " did not build"
why := notBuiltWhy(failed)
for k := range steps {
st := &steps[k]
if st.Outcome == seat.StepResultOutcomeBypassed {
@@ -495,6 +577,7 @@ func (s *Starter) sayNotAsked(p seat.Pipeline, steps []int) {
}
at := s.now()
for _, i := range steps {
logEnded(s.logf, p.Pipeline, p.Steps[i])
subject, ev := stepEvent(seat.EventStepEnded, p, p.Steps[i], at)
if err := s.Say(subject, ev); err != nil {
s.logf("step-ended of %s's %s %s, not asked, is not said: %v", p.Pipeline, p.Steps[i].Module, p.Steps[i].Step, err)
@@ -746,40 +829,15 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
}
}
for _, key := range pkeys {
p, rev, err := store.Get(key)
p, asks, ended, notAsked, n, err := s.placeOne(store, key, due, reg, view, now)
if err != nil {
note(fmt.Errorf("the pipeline %s is not read: %w", key, err))
continue
note(err)
}
if p.Outcome != seat.PipelineOutcomeRunning || p.Run != "" || !due(p) {
continue
}
half, ok := view(p.Branch)
if !ok {
s.logf("the catalog half of %s is not held yet: %s waits on it for its steps, placed once it is held", p.Branch, key)
continue
}
if p.Revision > 0 && half.Revision < p.Revision {
s.logf("the catalog half of %s holds revision %d: %s waits for revision %d, which its take-in made, to place its steps (§200)", p.Branch, half.Revision, key, p.Revision)
continue
}
before := len(p.Steps)
asks, unregistered, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now)
// Until its registration arrives, the pipeline says it waits on it (A-327): a mover with no registration has no step,
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
if len(p.Steps) == before {
continue
}
if _, err := store.Update(key, p, rev); err != nil {
s.logf("%s is written meanwhile, so nothing is placed in it now: %v", key, err)
if n == 0 {
continue
}
count++
s.logf("%s: %d steps placed from %s (A-327)", key, len(p.Steps)-before, from)
s.logf("%s: %d steps placed from %s (A-327)", key, n, from)
// A pipeline whose steps were never asked is started by this placing (A-446a); the pipeline-started of its start,
// said already, is said again harmlessly.
s.sayStartedPipeline(p, now)
@@ -799,6 +857,50 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
return count, first
}
// placeOne places, in pipeline key, the steps of each mover with none, when the pipeline is running, not a run on request
// and due names it (placeWaiting). Its write is by compare-and-set: a write refused because the pipeline was written
// meanwhile is done again at once from the record as read again, until it lands or the pipeline no longer waits on a
// placing (it ended, or every mover due is placed), so a concurrent write never cuts a placing for good (H-A-311 (1)). It
// answers the pipeline as written, its asks, the steps ended or not asked at the placing, and how many steps it placed.
func (s *Starter) placeOne(store Updater, key string, due func(seat.Pipeline) bool, reg func(module string) (catalog.Registration, bool), view func(branch string) (Half, bool), now time.Time) (seat.Pipeline, []Asked, []int, []int, int, error) {
for {
p, rev, err := store.Get(key)
if err != nil {
return p, nil, nil, nil, 0, fmt.Errorf("the pipeline %s is not read: %w", key, err)
}
if p.Outcome != seat.PipelineOutcomeRunning || p.Run != "" || !due(p) {
return p, nil, nil, nil, 0, nil
}
half, ok := view(p.Branch)
if !ok {
s.logf("the catalog half of %s is not held yet: %s waits on it for its steps, placed once it is held", p.Branch, key)
return p, nil, nil, nil, 0, nil
}
if p.Revision > 0 && half.Revision < p.Revision {
s.logf("the catalog half of %s holds revision %d: %s waits for revision %d, which its take-in made, to place its steps (§200)", p.Branch, half.Revision, key, p.Revision)
return p, nil, nil, nil, 0, nil
}
before := len(p.Steps)
asks, unregistered, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now)
// Until its registration arrives, the pipeline says it waits on it (A-327): a mover with no registration has no step,
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
s.registrationMissing(m, key)
}
if len(p.Steps) == before {
return p, nil, nil, nil, 0, nil
}
if _, err := store.Update(key, p, rev); errors.Is(err, ErrChanged) {
s.logf("%s is written meanwhile, so its placing is done again from the record read again (H-A-311): %v", key, err)
continue
} else if err != nil {
return p, nil, nil, nil, 0, fmt.Errorf("the placing of %s is not written: %w", key, err)
}
return p, asks, ended, notAsked, len(p.Steps) - before, nil
}
}
// recovered says whether the commit of pipeline key is named a recovery; a read that fails asks on the stream check, said, since
// a step is asked either way (a recovery's only goes ahead of the first round).
func (s *Starter) recovered(key, commit string) bool {
@@ -939,7 +1041,7 @@ func (s *Starter) Release(store Updater, key string, view func(branch string) (H
return false
}
for _, m := range p.Movers {
if !placed(p, m.Module) {
if !placed(p, m.Module) && !slices.Contains(p.Superseded, m.Module) {
return true
}
}
@@ -300,6 +300,25 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
t.Errorf("the pipeline a runner before this one set aside: placed %d (%v), %d steps, %d asks more, want it started as a newly started one", n, err, len(p.Steps), len(asks.asks)-before)
}
// H-A-311 (1): a placing whose write is refused because the pipeline was written meanwhile (a step's end, a publish) is
// done again at once from the record read again, never left unplaced until a restart.
held = false
taken = catalog.TakeInTaken{Branch: "next", Source: 7, Entries: []catalog.TakeInEntry{{Module: "app", Commit: "c4", Fingerprint: "src2:dd"}}}
if n, err := s.TakenIn(taken, late); n != 1 || err != nil {
t.Fatalf("the take-in of next.7 started %d (%v)", n, err)
}
held = true
raced := &racingStore{memStore: store.memStore, before: func(m memStore) {
e := m["next.7"]
e.p.Superseded, e.rev = append(e.p.Superseded, "other"), e.rev+1 // a concurrent write the placing did not read
m["next.7"] = e
}}
n, err = s.Waiting(raced, "next", late)
p, _, _ = store.Get("next.7")
if n != 1 || err != nil || len(p.Steps) == 0 || !slices.Contains(p.Superseded, "other") {
t.Errorf("a placing cut by a concurrent write placed %d (%v): %d steps, superseded %v, want its steps placed on the record read again (H-A-311)", n, err, len(p.Steps), p.Superseded)
}
// A-327: a mover with no registration, in a pipeline placed from the half held later (the queue's turn, the half held),
// is said to wait on its registration, so the pipeline that cannot end before it is placed says why.
var logged []string
@@ -411,3 +430,17 @@ func TestAPublishIsTakenIntoThePipelineOfItsRevision(t *testing.T) {
t.Errorf("a publish naming no revision (%v): main.7 %v, main.8 %v; want main.7's, the source's lowest", err, published("main.7"), published("main.8"))
}
}
// racingStore is memStore whose first Update is preceded by before, a write another part makes between a read and its write.
type racingStore struct {
memStore
before func(memStore)
}
func (r *racingStore) Update(key string, p seat.Pipeline, rev uint64) (uint64, error) {
if b := r.before; b != nil {
r.before = nil
b(r.memStore)
}
return r.memStore.Update(key, p, rev)
}
@@ -273,6 +273,8 @@ func (h *Hearer) OnStepEndedPinned(bs StepSeat, subject string, ev StepEnded, sh
ev.Module, ev.Commit, ev.Fingerprint, p.Steps[i].Step, p.Steps[i].Module, m.Commit, m.Fingerprint)
case !open(p.Steps[i]):
h.Asks.answeredLate(p.Steps[i], h.logf)
// An end heard late moves nothing of its step, and still places what that step's end unlocks (H-A-311 (2)).
defer h.releaseLate(ev.Pipeline)
if run := sharedRun(p.Steps[i]); run != "" && h.TestRuns != nil {
if r, _, err := h.TestRuns.Get(run); err == nil && r.Ended == nil && r.Ask == ev.ID {
// The asking step's end was written and its run's was not (a write failed): the run ends now, heard again.
@@ -552,6 +554,26 @@ func endEntry(key string, s seat.StepResult, at time.Time, kept ...string) seat.
Output: s.Output, Kept: kept, At: at}
}
// logEnded writes the journal line of a step's end said (H-A-320 (1), H-A-333): the pipeline, the mover, the step, its
// outcome, the node that ended it and how long it ran from its start. It is written where step-ended is said, Hearer.say and
// the starter's two sayers, so every said end has its one line.
func logEnded(logf func(string, ...any), key string, s seat.StepResult) {
if logf == nil {
return
}
mover, node, took := s.Module, s.Node, "no start"
if mover == "" {
mover = "-"
}
if node == "" {
node = "no node"
}
if s.Started != nil && s.Ended != nil {
took = s.Ended.Sub(*s.Started).Round(time.Millisecond).String()
}
logf("%s %s %s ended %s (%s, %s)", key, mover, s.Step, s.Outcome, node, took)
}
func (h *Hearer) logf(format string, args ...any) {
if h.Logf != nil {
h.Logf(format, args...)
@@ -565,6 +587,9 @@ func (h *Hearer) say(event string, p seat.Pipeline, step *seat.StepResult) error
at := h.now()
switch event {
case seat.EventStepEnded, seat.EventStepStarted:
if event == seat.EventStepEnded {
logEnded(h.logf, p.Pipeline, *step)
}
subject, ev := stepEvent(event, p, *step, at)
return h.Say(subject, ev)
case seat.EventPipelineEnded:
@@ -10,6 +10,7 @@ import (
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
@@ -363,10 +364,21 @@ func TestARecoverysStepsAreAskedOnTheStreamDelivery(t *testing.T) {
}{{true, "mesh.seat.node-build-go.accept.delivery.mesh-pipeline-runner," + wants.AcceptSubjectStep}, {false, "mesh.seat.node-build-go.accept.check.mesh-pipeline-runner," + wants.AcceptSubjectStep}} {
var to []string
asker := askerFunc(func(accept, _ string, _ any) error { to = append(to, accept); return nil })
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }, nil); err != nil || strings.Join(to, ",") != c.want {
if err := askAll(asker, "k", asks, func(string) bool { return c.recovery }, nil, nil); err != nil || strings.Join(to, ",") != c.want {
t.Errorf("recovery %v: asked on %v (%v), want %s", c.recovery, to, err, c.want)
}
}
// Each ask put is one journal line naming the pipeline, the mover, the step and the seat: the build ask on mesh-builds and a
// check ask alike (H-A-320 (1)); an ask refused writes none.
var lines []string
put := []Asked{{Seat: builds.Seat, ID: "b1", Ask: builds.StepRun{Step: builds.StepRunStepBuild, Entries: []builds.BuildEntry{{Module: "app"}}}},
{Seat: nodebuildgo.Seat, ID: "u1", Ask: nodebuildgo.StepAsk{Step: &nodebuildgo.StepRun{Module: "app", Step: "unit"}}},
{Seat: "no-seat", ID: "x1", Ask: 3}}
_ = askAll(askerFunc(func(string, string, any) error { return nil }), "main.4", put, nil, nil,
func(f string, a ...any) { lines = append(lines, fmt.Sprintf(f, a...)) })
if want := "main.4 app build asked on mesh-builds\nmain.4 app unit asked on node-build-go"; strings.Join(lines, "\n") != want {
t.Errorf("journal %q, want %q", lines, want)
}
// Starter.recovered reads the commit; a failed read asks on check and says so.
var said []string
s := &Starter{Logf: func(f string, a ...any) { said = append(said, f) }, Recovered: func(c string) (bool, error) {
@@ -124,6 +124,7 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
ended = append(ended, i)
}
p.Superseded = append(p.Superseded, moved...)
again := takeOutNotAsked(&p)
done := h.endPipeline(&p, now, wr)
if _, err := h.Store.Update(key, p, rev); errors.Is(err, ErrChanged) {
continue
@@ -131,7 +132,17 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
h.logf("the running pipeline %s is not written, so its movers moved by %s stay: %v", key, newer.Pipeline, err)
return false
}
h.logf("%s: %s moved by the newer %s leave it, %d of its open steps ended superseded (H-A-223)", key, strings.Join(moved, ", "), newer.Pipeline, len(ended))
var stay []string
for _, m := range p.Movers {
if !slices.Contains(p.Superseded, m.Module) {
stay = append(stay, m.Module)
}
}
h.logf("%s: %s moved by the newer %s leave it, %d of its open steps ended superseded (H-A-223); %d stay: %s (H-A-361)", key,
strings.Join(moved, ", "), newer.Pipeline, len(ended), len(stay), strings.Join(stay, ", "))
if len(again) > 0 {
h.logf("%s: %s, not asked on a dependency that left, placed again (H-A-361)", key, strings.Join(again, ", "))
}
for _, i := range ended {
if err := h.say(seat.EventStepEnded, p, &p.Steps[i]); err != nil {
h.logf("step-ended of %s's %s %s, superseded, is not said: %v", key, p.Steps[i].Module, p.Steps[i].Step, err)
@@ -142,10 +153,44 @@ func (h *Hearer) supersedeRunningOne(key string, newer seat.Pipeline) bool {
h.logf("%v", err)
}
h.appendUnlogged(key)
// The movers that stay and waited on a mover that left are placed now, building against the newest published package
// (H-A-53, H-A-361): no event of the mover that left comes to this pipeline again.
if !done && h.Release != nil {
h.Release(key)
}
return true
}
}
// takeOutNotAsked takes out of p the steps of each mover that stays whose steps ended not asked because a dependency did not
// build, that dependency a mover that left p, or a mover itself taken out here (H-A-361): a left mover is no dependency of
// the movers that stay (H-A-53), so they are placed again, and a dependency that truly did not build is named again at that
// placing. It answers the movers taken out, in p's order.
func takeOutNotAsked(p *seat.Pipeline) []string {
var out []string
for changed := true; changed; {
changed = false
for _, m := range p.Movers {
if slices.Contains(p.Superseded, m.Module) || slices.Contains(out, m.Module) {
continue
}
dep := ""
for _, st := range stepsOf(*p, m.Module) {
if d, ok := notBuiltDependency(st); ok {
dep = d
break
}
}
if dep == "" || (!slices.Contains(p.Superseded, dep) && !slices.Contains(out, dep)) {
continue
}
p.Steps = slices.DeleteFunc(p.Steps, func(st seat.StepResult) bool { return st.Module == m.Module })
out, changed = append(out, m.Module), true
}
}
return out
}
// supersedeStep ends the open or waiting step st of p superseded, naming why, with its step log entry: a step reading a test
// run is answered to leave it, and one with its own ask on a build node to drop it (H-A-223).
func supersedeStep(p *seat.Pipeline, st *seat.StepResult, why string, now time.Time) (leave, drop []seat.StepResult) {
@@ -2,6 +2,7 @@ package pipeline
import (
"fmt"
"slices"
"strings"
"testing"
"time"
@@ -278,6 +279,12 @@ func TestATestRunsEndMovesEveryReaderAndIsCountedOnce(t *testing.T) {
a := f.start(t, 4, 9)
b := f.start(t, 6, 9) // the same candidate: it reads a's runs
unit := stepNamed(a, "app:program", "unit")
// One journal line per ask put, naming the pipeline, the mover, the step and the seat (H-A-320 (1)).
if n := countLines(f.logged, a.Pipeline+" app unit asked on node-build-go"); n != 1 || len(f.asks.ids) == 0 ||
countLines(f.logged, " asked on ") != len(f.asks.ids) {
t.Errorf("journal %q for %d asks put; want one asked-on line per ask, a's unit ask once", f.logged, len(f.asks.ids))
}
f.logged = nil
ended := time.Now()
if _, err := f.h.OnStepEnded(stepSeatOf(t, "node-build-go"), "mesh.seat.node-build-go.event.step-ended.ace", StepEnded{ID: unit.Ask,
Pipeline: a.Pipeline, Module: "app", Commit: "c1", Fingerprint: "src2:aa", Step: "unit", Node: "ace", Outcome: "failed",
@@ -308,6 +315,13 @@ func TestATestRunsEndMovesEveryReaderAndIsCountedOnce(t *testing.T) {
if saidReaders != 4 {
t.Errorf("said %d unit step-ended, want one per reading step", saidReaders)
}
// One journal line per step's end, the asking step's and each reader's, naming its pipeline, mover, step, outcome and node
// (H-A-320 (1)).
for _, key := range []string{a.Pipeline, b.Pipeline} {
if n := countLines(f.logged, key+" app unit ended failed (ace, "); n != 2 {
t.Errorf("%s's unit ends journaled %d times in %q, want once per artifact", key, n, f.logged)
}
}
// a, cancelled, leaves the lint run to b's readers; b, cancelled after, leaves none: the ask is dropped and the run ends.
lint := stepNamed(a, "app:program", "lint")
v := Verbs{Hearer: f.h, Acts: &actLog{}}
@@ -599,6 +613,8 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
run := unit.Shared[0].Run
asked(f, run)
delete(f.clock.bound, UntakenPrefix+run)
var released []string
f.h.Release = func(key string) { released = append(released, key) }
_ = f.h.RunTaken(unit.Ask, "ace")
_ = f.h.RunProgress(unit.Ask)
late := 0
@@ -610,6 +626,10 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
if _, ok := f.clock.bound[BoundPrefix+run]; ok || late != 2 {
t.Errorf("bounds %v, journal %q; want no clock and both late signals journaled", f.clock.bound, f.logged)
}
// H-A-311 (2): the late step-progress still places what its step's end unlocks, in the pipeline that asked it.
if !slices.Contains(released, p.Pipeline) {
t.Errorf("the late step-progress placed in %v, want %s's waiting movers placed", released, p.Pipeline)
}
})
// A run's end is written as its step-ended is heard, before the finding the step says (A-470o).
t.Run("the run ends before the step's finding is written", func(t *testing.T) {
@@ -671,3 +691,14 @@ func TestNoWriteOfATestRunActsOnAReadBeforeItsLatestChange(t *testing.T) {
}
})
}
// countLines is how many of the journal's lines hold part.
func countLines(lines []string, part string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, part) {
n++
}
}
return n
}
@@ -326,7 +326,7 @@ func (v Verbs) Rerun(caller string, in seat.RerunInput) (seat.RerunAnswer, error
}
if ask != nil {
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm, v.Starter.logf); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -473,7 +473,7 @@ func (v Verbs) rerunTest(caller string, in seat.RerunInput, st seat.StepResult)
break
}
v.Starter.carry(in.Pipeline, []Asked{*ask})
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm); err != nil {
if err := askAll(v.Starter.Steps, in.Pipeline, []Asked{*ask}, func(commit string) bool { return v.Starter.recovered(in.Pipeline, commit) }, v.Starter.Asks.arm, v.Starter.logf); err != nil {
return seat.RerunAnswer{}, fmt.Errorf("the step is written running but not asked, it is asked again at the next start: %w", err)
}
}
@@ -106,6 +106,8 @@ func TestManifest(t *testing.T) {
// A-102 (§22, §24): the accepted kinds the runner puts its steps on.
"seat:mesh-builds.step", "seat:mesh-wants.step", "seat:node-build-go.check.step", "seat:node-build-node.check.step",
"seat:node-build-image.check.step",
// H-A-328: whether a package a mover waits on is published, read when the wait is written.
"seat:mesh-builds.published",
// RN22 (§86, §415, A-207): a recovery's steps on the build seats' stream delivery.
"seat:node-build-go.delivery.step", "seat:node-build-node.delivery.step", "seat:node-build-image.delivery.step",
// A-231: the queues of the build seats it asks, check for the steps and delivery for a recovery's.
@@ -78,6 +78,7 @@
"seat:mesh-module-assigner.assignments",
"seat:mesh-grants.grants-for",
"seat:mesh-builds.step",
"seat:mesh-builds.published",
"seat:mesh-wants.step",
"seat:node-build-go.check.step",
"seat:node-build-node.check.step",
@@ -120,6 +120,10 @@ func fullConf(dir string) string {
teller = append(append(teller, viewstest.Grants(assignerModule, assignerModule+".seat-holders", tellerConsumer)...),
"mesh.seat.mesh-catalog.call.revision", "mesh.seat.mesh-catalog.call.revision.>", "mesh.seat.mesh-catalog.call.registrations",
"mesh.seat.mesh-catalog.call.registrations.>", "mesh.seat.mesh-delivery.call.source", "mesh.seat.mesh-delivery.call.source.>")
// Who holds operator-channel, asked of the seat mesh-module-assigner's assignments (manifest invokes) before the router's
// record of an open ask is read: a bed that refused the call held the teller's watch of the asks for the call's whole
// timeout at each catch-up, and every ask written meanwhile waited (issue 1960, H1).
teller = append(teller, "mesh.seat.mesh-module-assigner.call.assignments", "mesh.seat.mesh-module-assigner.call.assignments.>")
teller = append(teller, keepsOf(bucketsOf(tellerModule).findings)...)
view := append(append(readOf(tellerAnswers), readOf(assignerAsks)...), keepsOf(viewAnswers)...)
assigner := keepsOf(assignerAsks)
@@ -224,7 +228,9 @@ func (b *fullBed) place(t *testing.T) {
}
// serveCatalogHalf answers the catalog half's three first views (design 47 G §614): the gate's one source branch, the
// assigner registered on it, and its trunk revision whose entry of the assigner claims mesh-module-assigner.
// assigner registered on it, and its trunk revision whose entry of the assigner claims mesh-module-assigner. It also answers
// the seat mesh-module-assigner's assignments as the live seat does, naming ops-router the holder of operator-channel, so a
// catch-up reads the router's record (ops-router_asks) at once.
func serveCatalogHalf(t *testing.T, nc *nats.Conn) {
t.Helper()
respond := func(a any) nats.MsgHandler {
@@ -235,8 +241,10 @@ func serveCatalogHalf(t *testing.T, nc *nats.Conn) {
Entries: []meshcatalog.CatalogEntry{{Module: assignerModule, ManifestDigest: "d1", Manifest: map[string]any{"module": assignerModule,
"claims": []any{map[string]any{"name": "mesh-module-assigner", "scope": "mesh"}}}}}}
registrations := meshcatalog.RegistrationsAnswer{Registrations: []meshcatalog.Registration{{Module: assignerModule, Version: 1, Branches: []string{"release"}}}}
assignments := json.RawMessage(`{"seats":[{"seat":"operator-channel","scope":"mesh","holders":[{"module":"ops-router","node":"n2"}]}]}`)
for subject, handle := range map[string]nats.MsgHandler{"mesh.seat.mesh-catalog.call.revision": respond(revision),
"mesh.seat.mesh-catalog.call.registrations": respond(registrations), "mesh.seat.mesh-delivery.call.source": respond(map[string]any{"branches": []string{"release"}})} {
"mesh.seat.mesh-module-assigner.call.assignments": respond(assignments),
"mesh.seat.mesh-catalog.call.registrations": respond(registrations), "mesh.seat.mesh-delivery.call.source": respond(map[string]any{"branches": []string{"release"}})} {
for _, s := range []string{subject, subject + ".>"} {
sub, err := nc.Subscribe(s, handle)
if err != nil {
@@ -375,6 +383,11 @@ func TestTellerCarriesAnAskAndWritesTheAnswer(t *testing.T) { // 1343/AC1, AC2
}
// The program hears warrants before it reads the asks; it has its subscription before the ask is written.
b.listening(t)
// Every event the views reader adopted is heard once the hub pulls: registration-changed, which the teller's widenings and the
// catalog half both hear, is one subscription (A-199, §532), so no handler of the teller takes the reader's away.
if l := logs.String(); strings.Contains(l, "a message has one taker") {
t.Errorf("an event of the views reader is not heard: two handlers took one subject\n%s", l)
}
// Design 47 G §717: the teller started before any module held mesh-wants; once the holder is placed it follows it, and
// raises the ask the holder then writes.
b.place(t)
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-contracts/go/seats/meshbroker"
occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdeliveryview"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"io"
@@ -91,9 +92,6 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := hub.Handle(pipelineEndedSubject, widens.HearEnded); err != nil {
return fmt.Errorf("hearing %s: %w", pipelineEndedSubject, err)
}
if err := hub.Handle(registrationChangedSubject, widens.HearRegistration); err != nil {
return fmt.Errorf("hearing %s: %w", registrationChangedSubject, err)
}
// A merge named a recovery is told on every channel (design 47 G §81, §514); a merge made at the forge is no record of its
// own (H §763, H-A-119).
if err := hub.Handle(recoveryMergedSubject, widens.HearRecovery); err != nil {
@@ -108,6 +106,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return calls.Emit(answeredEvent, Answered{Key: key, Ask: askID, Answer: answer}, stdio.EmitOptions{MessageID: answeredID(key, askID)})
}
reader := &views.ProcessReader{Module: cred.Module, Call: calls.Call, Logf: logger.Printf, Hub: hub}
// registration-changed is heard on the reader's one subscription of it, which the catalog client's followers share
// (A-199, §532): the hub gives a message one taker.
if err := reader.AdoptEvent(meshcatalog.Seat, meshcatalog.EventRegistrationChanged, widens.HearRegistration); err != nil {
return fmt.Errorf("hearing %s: %w", meshcatalog.SubjectRegistrationChanged, err)
}
// The teller's finding stream is its `finding-events` entry, asked of the seat mesh-broker's show (design 47 G A-274,
// §529, §577): the writer says its events under that binding's subjects.
binding := process.NewBinding(meshbroker.Seat, cred.Module, calls.CallKept, logger.Printf, views.FindingEventsLocal)
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -33,9 +34,8 @@ const (
// The events the teller hears for it (manifest consumes).
const (
pipelineEndedSubject = "mesh.seat." + mpr.Seat + ".event." + mpr.EventPipelineEnded + ".>"
registrationChangedSubject = meshcatalog.SubjectRegistrationChanged
recoveryMergedSubject = meshdelivery.SubjectRecoveryMerged
pipelineEndedSubject = "mesh.seat." + mpr.Seat + ".event." + mpr.EventPipelineEnded + ".>"
recoveryMergedSubject = meshdelivery.SubjectRecoveryMerged
)
// widensState is the state widens-said as the teller keeps it.
@@ -151,32 +151,39 @@ func (w *Widens) HearEnded(m jetstream.Msg) {
}
// HearRegistration is the handler of registration-changed: it asks the registration of the module and drops the branches it
// no longer watches.
func (w *Widens) HearRegistration(m jetstream.Msg) {
// no longer watches. It is heard on the one subscription the views reader's followers of the event share (ProcessReader's
// AdoptEvent, design 47 G A-199, §532), beside the catalog client's own handler: the event hub gives a message one taker,
// so a second handler of the subject on the hub would take the reader's subscription away. A failure leaves the message
// unacknowledged, and the bus asks again (A-222); a message that names no module is passed over.
func (w *Widens) HearRegistration(_ context.Context, data []byte) error {
var ev meshcatalog.RegistrationChangedEvent
if err := json.Unmarshal(m.Data(), &ev); err != nil || ev.Module == "" {
w.logf("%s names no module and is left: %v", m.Subject(), err)
_ = m.Ack()
return
if err := json.Unmarshal(data, &ev); err != nil || ev.Module == "" {
w.logf("a registration-changed that names no module is left: %v", err)
return nil
}
raw, err := w.Call(registrationsKey, map[string]any{"module": ev.Module})
if err != nil {
w.finish(m, fmt.Errorf("the registration of %s cannot be read: %w", ev.Module, err))
return
return w.heardAgain(fmt.Errorf("the registration of %s cannot be read: %w", ev.Module, err))
}
var ans meshcatalog.RegistrationsAnswer
if err := json.Unmarshal(raw, &ans); err != nil {
w.finish(m, fmt.Errorf("the registration of %s is not read: %w", ev.Module, err))
return
return w.heardAgain(fmt.Errorf("the registration of %s is not read: %w", ev.Module, err))
}
for _, r := range ans.Registrations {
if r.Module == ev.Module {
w.finish(m, w.Unwatched(ev.Module, r.Branches))
return
return w.heardAgain(w.Unwatched(ev.Module, r.Branches))
}
}
// A module with no registration watches no branch.
w.finish(m, w.Unwatched(ev.Module, nil))
return w.heardAgain(w.Unwatched(ev.Module, nil))
}
// heardAgain says err, when not nil, as a message heard again, and gives it back.
func (w *Widens) heardAgain(err error) error {
if err != nil {
w.logf("%v; the message is heard again", err)
}
return err
}
// finish acknowledges the message when err is nil and asks it again when it is not.
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"errors"
"slices"
@@ -155,15 +156,13 @@ func TestARegistrationChangedDropsTheBranchesNoLongerWatched(t *testing.T) {
st.put(k, "c")
}
w := &Widens{State: st, Call: c.call}
m := &fakeMsg{data: []byte(`{"module":"nats","moved":[],"version":2,"warrant":"w"}`)}
w.HearRegistration(m)
if err := w.HearRegistration(context.Background(), []byte(`{"module":"nats","moved":[],"version":2,"warrant":"w"}`)); err != nil {
t.Errorf("the registration-changed is given back: %v", err)
}
keys, _ := st.Keys()
if want := []string{"nats.main", "natsx.next", "other.next"}; !slices.Equal(keys, want) {
t.Errorf("keys %v, want %v", keys, want)
}
if !m.acked || m.naked {
t.Errorf("acked %v naked %v", m.acked, m.naked)
}
}
// A pipeline-ended whose say fails is given back, and one that is told is acknowledged.