Compare commits

...
72 Commits
Author SHA1 Message Date
mesh-tools b9cc97470b Merge pull request 'Design 47 H1: the entry pins every requirement of the module graph; same-repository directories move the fingerprint (H-A-201, H-A-202)' (#296) from fix/47-H1-package-tree-paths into main 2026-11-20 21:37:24 +00:00
mesh-tools ae91f91e88 Merge pull request 'Design 47 H1: the catalog seat's hand-written digest helpers live in its client (H-A-200)' (#295) from fix/47-H1-contracts-steps into main 2026-11-20 21:37:22 +00:00
jschoubben 4664eb1b00 Design 47 H1: the entry pins every requirement of the module graph
The Go build module borrowed a sibling's version for a mesh package the
entry did not pin (a transitive requirement), and given no pin it now
refuses. The indexer now pins, in the entry's packages, every mesh package
of another repository that the go.mod of the module's directory, of each
Go package artifact's directory and of every same-repository module they
require transitively names. A module of the module's own repository is
never pinned, whatever a build reported: it is built from the tree.

Builds: design 47 H H-A-201, H-A-183, H-A-157, H-A-202
2026-11-20 22:30:45 +01:00
jschoubben d9a3458927 Design 47 H1: a Go build's same-repository modules move its fingerprint
A Go build resolves the same-repository modules its go.mod files require at
the zero version, transitively, from the fetched tree; a change in one of
them must move the fingerprint, whatever the module's form. The indexer
walks the go.mod graph of the module's directory and its Go package
artifacts at the take-in and appends a line per derived directory and its
git object to the tree input; the entry's paths keep the manifest's form,
and the Go build module derives the same directories itself.

Builds: design 47 H H-A-202, H-A-172, H-A-157, H-A-198
2026-11-20 22:29:15 +01:00
jschoubben b058389d46 Contracts: the catalog seat's manifest-digest helpers move to its client, out of the generated root (H-A-200)
The seat generator's check failed every unit step of the contracts mover (pipeline main.43): manifestdigest.go was
written by hand under go/seats/meshcatalog (598125a1), a generated root every file of which is the generator's. The two
helpers move to the seat's hand-written client meshcatalogclient and their callers in the builder and the indexer
follow; the generator then writes the contract as the module holds it. steprun's generator test now also holds this
repository's own contracts module in step under both roots.

Builds: design 47 G §370, G §578, H §774, H-A-200 (ADR 0608 §5)
2026-11-20 22:18:00 +01:00
jschoubben 6c65e82926 Design 47 H1: a step's named path may leave the module's directory, never the repository
A package artifact's tree names same-repository siblings its go.mod requires
as "../<module>/<dir>"; the fetch request's check and the narrowing refused
them as leaving the module. They are now accepted when their join with the
module's directory stays inside the repository, narrowed from the root, and
refused when they leave the repository.

Builds: design 47 H H-A-198, H-A-199, H-A-157, H-A-172
2026-11-20 22:14:53 +01:00
mesh-tools d40f314e52 Merge pull request 'Design 47 H1: the grants holder keeps the catalog revisions it read and learns the newest from the event (H-A-193, H-A-195)' (#294) from fix/47-H1-grants-for-cache into main 2026-11-20 20:56:03 +00:00
mesh-tools fc85f7b936 Merge pull request 'Design 47 H1: the assigner's test helper holds its map writes under a lock (H §774)' (#293) from fix/47-H1-assigner-test-race into main 2026-11-20 20:52:56 +00:00
mesh-tools 37a8f7a83e Merge pull request 'Design 47 H1: a pipeline's steps and unlogged entries live under one key per mover (H-A-187, H-A-189, H-A-190, H-A-191)' (#292) from fix/47-H1-runner-steps-per-mover into main 2026-11-20 20:52:51 +00:00
mesh-tools 9ef830d224 Merge pull request 'Design 47 H1: the builder fetches a module whole and never computes a pin (H-A-186, H-A-192, H-A-194)' (#291) from 47-H1-builder-program-paths into main 2026-11-20 20:52:45 +00:00
jschoubben 120a0554d6 mesh-grants: grants-for keeps a bounded cache of the followed branches
The kept catalog revisions had no design line and their bound stood in
the code. They are now bounded per branch by the module's setting
grants-for-revisions-per-branch (default 4, oldest dropped, a run's
candidate counted within its branch's), read at the start from the file
the manifest writes, and kept only for the watched branches the part's
branch follower follows: a branch that leaves them is freed at the next
grants-for or catalog-revision event. A kept revision is a cache of the
catalog seat's own truth, never a state of the mesh.

Builds: design 47 G §441, §401; H H-A-195, H-A-193
2026-11-20 21:51:43 +01:00
jschoubben d8a900dac0 Read what the registry seat holds before a build waits for an unpinned package (H-A-194)
The publishes the builder heard lived only in its memory, so a build waiting for a package published before a
restart waited the whole bound and failed although the registry held the version. Before it waits, the build asks
the Go package registry seat's versions once for each unpinned path and takes the newest in the accepted series
through PackagePublished; a failed read is said and the wait goes on by the published events.

Builds: design 47 H H-A-194, H-A-192, H-A-165
2026-11-20 21:48:40 +01:00
jschoubben a62f7b40c1 mesh-grants: grants-for learns the newest revision from the event
A grants-for call still read the first page of each revision (up to
768 KB) only to learn its number. The grants holder already hears the
seat mesh-catalog's catalog-revision event, which names each branch's
newest trunk revision and the source revision it was taken in from; the
part now records both from the event and asks the revision verb only for
a revision it does not hold, by its number. Before the first event after
a start the first ask's answer names the newest once. An event below the
newest held is the numbering started again: what is kept of the branch
is dropped.

Builds: design 47 G §441, §401; H H-A-193
2026-11-20 21:44:06 +01:00
jschoubben 429f03d4cf Pin an unpinned package only at the version its registry publish carries (H-A-192)
A step ask waiting for a package the entry names with no pin took the release number of the module holding it
("v" + release), and the lane of an entry taken in dropped the empty pin and asked its build without one, so the Go
build module took another package's version: both built against versions never published (main.41, v0.1.25,
v0.1.97, v0.1.103 of the builder's builds). The builder now hears each package registry seat's published (the
package path and the version it carries) and keeps the newest per series; a build, the step's and the lane's alike,
waits for one in the series its module's manifest accepts (packages, else the default) up to package-wait-minutes,
and past it fails naming the package. The release-number pin (VersionReleased, NewestReleased) is gone.

Builds: design 47 H H-A-192, H-A-183, H-A-184, H-A-185, H-A-165
2026-11-20 21:42:33 +01:00
jschoubben 20d59ff4f0 mesh-grants: grants-for keeps the catalog revisions it read
grants-for read two whole catalog revisions page by page on every call
(about 1.4 MB each) and answered in 3.4 to 5.0 s, so the pipeline
runner's 5 s read bound cut it and every widening touching the vault
said "could not run". A revision never changes, so the part now keeps
each revision it read whole, by branch, kind, group, heads and number,
the newest four per branch for the process's life (as the runner keeps
its revisionCache); once a first page names a kept revision the pages
after it are not read. The branch's newest is read on by the number its
first page names, and a revision whose pages name another is not kept.

Quick fix (not chosen): raise the runner's widens read bound, a catalog
merge that hides the cost. Thorough fix (chosen): the cache here. The
compose of every account twice is kept whole: narrowing it to the
candidate's modules would stop a refused principal elsewhere from
failing grants-for, which is not obviously safe.

Builds: design 47 G §441, §401; H §774
2026-11-20 21:40:15 +01:00
jschoubben 12b18721b9 Build a module with a program naming no paths from its whole directory (H-A-186)
Since #290 the builder's manifest declares its packages builds and buildrun with paths beside the program and
tools, which name none. ModuleOf took the entry's named paths of any artifact as the module's, so the lane's one
fetch was narrowed to builds/ and buildrun/: the module's go.mod and cmd/ were removed and the program's build
failed with "go: cannot find main module" (main.42, 21:10 and 21:13 CET). An artifact without paths keeps its
module's directory as its build source (ADR 0499 clause 1); only a module whose every artifact names its paths is
built from those alone (clauses 2 and 4), as the indexer's own value already has it.

Builds: design 47 H H-A-186, H-A-188; ADR 0499 clauses 1, 2, 4
2026-11-20 21:28:39 +01:00
jschoubben 3183e0e056 Design 47 H1: a mover's unlogged entries live under its mover key, each entry names its mover, and list answers records with counts (H-A-189, H-A-190, H-A-191)
The record still kept every ended step's unlogged step-log entry, so a step
log outage across a large pipeline grew the record as its steps had. A
mover key now holds its steps and its unlogged entries, written in the same
compare-and-set; the record keeps neither. An entry of the step log names
its mover's module, set where an entry is made, so two movers' approvals
ending at one moment are filed under their own keys and told apart when
read back; an older entry naming none goes under the one mover whose step it
matches, else stays in the record. list answers each pipeline's record with
its movers and steps counted and its steps empty; show keeps the steps. The
seat's definition changes: its Go and TypeScript contracts are generated
again.

Builds: design 47 H H-A-187, H-A-189, H-A-190, H-A-191
2026-11-20 21:22:09 +01:00
jschoubben 30ccad4edc Assigner tests: hold putInto's writes one at a time
The composition step starts its machines' runs side by side, and each may
raise assignment-unlisted into the test's map: unguarded, the unit suite
crashed now and then with concurrent map writes. Only the test raced; the
running module writes the bus's findings state.

Builds: design 47 H §774
2026-11-20 21:15:50 +01:00
jschoubben 4e585f02b1 Design 47 H1: the runner keeps each mover's steps under its own key, so a record never grows with the catalog (H-A-187)
The runner's state pipelines held a pipeline's every step in its record, and
a record of 112 or 129 movers exceeded the bus's 1 MiB message limit (ADR
0486), so main.26 and main.30 placed no step. One MoverStore per process now
reads and writes the state: the record keeps the movers, tiers, call, source
and outcome with its own steps empty, each mover's steps are written under
steps.<pipeline>.<mover> by compare-and-set as the record is, and nothing is
written when the record or a mover key changed since the read. Every read
and judgement of steps goes through it, so the placing, the step ends, the
end judgement, the hand acts and the ageing see one composed pipeline; show,
list, stands and why-carried compose from the mover keys, and show naming a
mover answers its steps alone. A record still carrying its steps inline is
read whole, said once in the journal, and split at its next write, its
mover keys written before the record.

Builds: design 47 H H-A-187
2026-11-20 21:11:59 +01:00
jschoubben f2d3172eac Design 47 H1: the seat mesh-pipeline-runner keeps a mover's steps under its own key, and show names a mover (H-A-187)
A pipeline record of 112 or 129 movers exceeded the bus's message limit once
its steps were placed (main.26, main.30), so its steps could never be
written. The state pipelines gains the key form steps.<pipeline>.<mover>,
one mover's steps, and show takes an optional mover whose steps it answers
alone. The seat's definition changes: its Go and TypeScript contracts are
generated again.

Builds: design 47 H H-A-187
2026-11-20 21:11:59 +01:00
mesh-tools e4781edcd6 Merge pull request 'Design 47 H1: the Go packages other modules import are declared artifacts of their modules (H-A-186, H-A-188)' (#290) from fix/47-H1-declare-package-artifacts into main 2026-11-20 20:08:40 +00:00
jschoubben 1db38fcc72 Declare asks in the assigner's version series, and have the teller and the view accept it there (H-A-188)
The builder numbers every artifact of a release in its module's version, so the assigner's package asks is
published in 0.3, not the 0.1 eba4309b declared: asks now names series 0.3, and the teller and the view, which
import it, accept it at 0.3 in their manifests' packages. The other eight packages' modules are all at 0.1, so
their series stand. The gate in mesh-catalog, which imports asks too, follows with the catalog's next change.

Builds: design 47 G §370, H-A-186, H-A-188
2026-11-20 21:05:39 +01:00
jschoubben eba4309b19 Declare every Go package another module imports as a package artifact of its module (H-A-186)
mesh-build-go and other modules import packages of this repository that no manifest declared, so no release
published them and the registry holds no version: their importers stay unpinned and fail past the bound. Each
module now declares its imported packages in the form they had before issue 1966's bootstrap took them out:
asks (assigner), builds and buildrun (builder), forgemsg (forge bridge), takein (indexer), tellerevents
(teller), viewstate (view), grantsstate and busgrants (grants), each a Go package of series 0.1. Each
module's manifest test asserts its packages again.

Builds: design 47 G §370, H-A-186
2026-11-20 20:58:06 +01:00
mesh-tools a7fc37561e Merge pull request 'Design 47 H1: a pin is the version the registry published for the package path within the module's accepted series (H-A-183, H-A-184, H-A-185)' (#287) from fix/47-H1-pins-by-package-path into main 2026-11-20 19:15:04 +00:00
jschoubben e93575ffe7 mesh-delivery-indexer: the accepted series is the manifest's, contents or none
The pin fell back to the path's newest publish while no series value
held contents, which live none does, so a publish in a series a module
does not accept pinned it. The series a module accepts is the one its
manifest names, or the default; the pin is the newest version published
for the package path within it.

Builds: design 47 H H-A-185, H-A-184
2026-11-20 20:13:15 +01:00
jschoubben 859f88b2f0 mesh-delivery-indexer: a pin stays within the series the module accepts
A pin by the package path across every series moved a module to a
series it does not accept while its contents came from the accepted
one. The pin is the accepted series' newest published version where
that series gives the contents; only where none does yet is it the
newest published of the path. A publish in another series pins nothing
and moves nothing.

Builds: design 47 H H-A-184, H-A-183
2026-11-20 20:07:46 +01:00
jschoubben 517a4b3cf1 mesh-delivery-indexer: a pin is the version the registry published for the package's path
At main.39 every importer of a nested package was pinned to the release
number of the module holding it (v0.1.97 of the builder for its package
builds), and the Go seat found no tag <package dir>/v0.1.97. A Go
package's version is its tag: the pin is now the newest version the
registry published for that package path (the seats' published, kept in
the package's series), never a module's release; a package the
registry never published stays unpinned.

Builds: design 47 H H-A-183, H-A-167, G §370
2026-11-20 20:05:52 +01:00
mesh-tools 7f88916863 Merge pull request 'Design 47 H1: the forge bridge keeps only an act's outcome and pull request; the act stays in the gate's stream (H-A-181, H-A-182)' (#286) from fix/47-H1-bridge-performed-cap into main 2026-11-20 18:46:57 +00:00
jschoubben 5dee4cb263 mesh-delivery-forge-bridge: owed again refuses a performed value of the old form
A value kept before this change names no pull request, so bridge_owed_again read the subject owed..0 and said the
stream no longer held the act while it did. Every act the bridge decodes names its repository, so a value without
one is the old form: refused with what is true, never read again; it expires with the state's TTL.

Builds: design 47 G §163; H-A-182
2026-11-20 19:45:38 +01:00
jschoubben 77b683c1c1 mesh-delivery-forge-bridge: performed keeps the outcome and the pull request, not the act
The state performed filled to 77% of its template's 64 MiB (19,530 keys, 2.6 KB each): the bridge kept every owed
event whole beside its outcome. performed's value is now the outcome said as owed.ended, with when, and the
repository and number the act names (groups.Performed); bridge_owed_again reads the act from the gate's stream owed
on its pull request's subject through an ordered consumer, the stream being the act's one copy; bridge_owed reads
one act or one pull request's acts, never the whole state, whose answer had grown past what the bus carries.

Builds: design 47 G §163, §162, §168; H-A-181, H-A-182
2026-11-20 19:38:34 +01:00
mesh-tools 60e32772a5 Merge pull request 'Design 47 H1: the composition step of a module no machine holds ends nothing to run (H-A-179)' (#285) from fix/47-H1-composition-nothing-to-run into main 2026-11-20 18:08:15 +00:00
jschoubben fd2ef6c7e8 assigner: a mover no machine holds has nothing to compose (H-A-179)
A package module with a required composition step and no node placing
it ended could not run, so its pipeline could never pass. A step with
nothing to judge ends nothing to run, which counts as passed, saying why.

Builds: design 47 H H-A-179
2026-11-20 19:05:49 +01:00
mesh-tools 71371df160 Merge pull request 'Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7)' (#284) from proof/47-H1-two-modules-2 into main 2026-11-20 17:55:44 +00:00
jschoubben b2d5644c34 Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7) 2026-11-20 18:55:12 +01:00
mesh-tools 9598b92c39 Merge pull request 'Design 47 H1: the runner's integration tests make the module's findings stream (H §743)' (#283) from fix/47-H1-runner-queue-test into main 2026-11-20 17:37:22 +00:00
mesh-tools f00d34673f Merge pull request 'Design 47 H1: the indexer's runs-candidate test is folded into TestIndexerWhole and waits for the run-started's acknowledgement (H §743)' (#282) from fix/47-H1-indexer-runs-test into main 2026-11-20 17:35:17 +00:00
jschoubben 87357d84bd mesh-delivery-pipeline: the queue and run tests make the module's findings stream
Built against mesh-sdk at its pin (H-A-166, H-A-169), the runner counts every
event no handler takes as a finding; with no stream for its finding events each
write waited its 3 s for the bus, holding the event loop, so the queue test ran
into its 90 s bound. The tests now make the stream the manifest asks for.

Builds: design 47 H H-A-176, H-A-175 (the tests of the runner's placing), H-A-166, H-A-169
2026-11-20 18:31:59 +01:00
jschoubben 08dde53e33 mesh-delivery-indexer: the run on request's bed lives in TestIndexerWhole, its restart after the run-started's acknowledgement
TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded failed on the build node
under load (pipelines main.29 and main.31). It restarted the indexer as soon as the candidate was
written, before the run-started was acknowledged (the write comes first), and it polled both
waits with sleeps: the same kind as the toolchains bed's restart fixed in 20a7bd0f. A stop
between the write and the acknowledgement leaves the run-started for redelivery after its
acknowledgement wait, which writes the candidate again once its run has ended.

The run is now one more step of TestIndexerWhole around its restart: run-started of run 9 writes
candidates.run-9 and the run's key (watched, not polled), the restart waits for the run-started's
acknowledgement beside the source-moved's, and the member's pipeline-ended after the restart drops
both. The separate test is removed: the module's integration tests go from 541 to 540; no test
added.

Builds: design 47 H §743
2026-11-20 18:30:27 +01:00
mesh-tools c772eb0683 Merge pull request 'Design 47 H1: a build module's inputs do not wait for its toolchain; a cycle among unplaced movers ends loud (H-A-175, H-A-176)' (#281) from fix/47-H1-runner-order-cycle into main 2026-11-20 17:17:28 +00:00
jschoubben a7b2fb0ffa mesh-delivery-pipeline: a cycle ends only its own steps; the end judgement ends the pipeline
A cycle ended every open step of the pipeline at once and named edges outside
it. It now ends not asked only the steps of the movers without steps, names
the edges inside a strongly connected group alone, and leaves the pipeline's
end to the end judgement, so running steps run to their end.

Builds: design 47 H H-A-176, H-A-175, H-A-163, G §270, G-22 A-326
2026-11-20 18:15:47 +01:00
jschoubben ef086e0b75 mesh-delivery-pipeline: a cycle among unplaced movers ends the pipeline failed
A cycle the running build module does not break left a pipeline running with
journal lines only, since validate is asked only for a placed mover. The
placing now ends it failed, its why naming each artifact and edge kind.

Builds: design 47 H H-A-175, G §270, G-22 A-326, H-A-101 (2)
2026-11-20 18:12:04 +01:00
jschoubben a8adda1a8c mesh-delivery-pipeline: a build module's own inputs await not its toolchain edge
A pipeline moving the Go build module with a package it imports (mesh-sdk,
mesh-delivery-contracts) placed nothing: the package's build awaited the new
build module's files, the build module awaited the package. What a build
module's new version is built from is built by its running version.

Builds: design 47 G §270, G-22 A-326, H §746, H-A-83 (2)
2026-11-20 18:09:07 +01:00
mesh-tools e1f0c1a1cc Merge pull request 'Design 47 H1: a release pins the revision its entry lives in; freed by the module's next release (H-A-168, H-A-170, H-A-173, H-A-174)' (#280) from fix/47-H1-release-pins into main 2026-11-20 17:03:27 +00:00
jschoubben bf24bb9740 mesh-delivery-indexer: a tier-late pipeline-wanted frees no release
A tier-late pipeline's later-tier movers released a version, so the newer revision holds their
release record, yet their wants were not written: freeing their older release on the tier-late
event would remove the revision their want names. HeardPipelineWanted now reads the event's outcome:
tier-late frees nothing, and the same pipeline's later pipeline-wanted with outcome running frees as
before. The pins test's freeing sequence says a tier-late event first (nothing freed), then the
running one (the older release freed); the events test's row hands the outcome on. No test added.

Builds: design 47 H-A-174 (hq design/47-H1-answers-59 84e2514d), H-A-173
2026-11-20 18:01:56 +01:00
jschoubben 814244fd5a mesh-delivery-indexer: a release pin only for a released mover, freed only by a newer release of it
A passed pipeline's movers include ones the assigner never takes (superseded, no version released,
a tier-late pipeline's later tiers); freeing their older release on the newer pipeline-wanted would
remove the revision their want still names, and a mover releasing nothing would pin a revision per
pipeline for good. The release record is now written only for a mover that released a version in
the pipeline, and pipeline-wanted frees a module's older record only where the newer pipeline's own
revision holds that module's release record.

Tests: a pins table row (a passed mover with no version pins nothing), the freeing sequence (a
passed pipeline releasing no version frees no older release, the next one that releases does), and
TestIndexerWhole now says a pipeline-wanted on the bus after main.12's passed end and waits for
main.11's release to be freed and main.11 removed, so the indexer's hearing of pipeline-wanted is
tested end to end (review round 1). No test added.

Builds: design 47 H-A-173 (hq design/47-H1-answers-59 21297155), H-A-170, H-A-168
2026-11-20 17:58:45 +01:00
mesh-tools 442b03f61d Merge pull request 'Design 47 H1: package edges from go.mod; a pin is the newest released version, in every form's fingerprint; a release re-pins its importers (H-A-157, H-A-161, H-A-165, H-A-167, H-A-172)' (#279) from fix/47-H1-package-edges-from-gomod into main 2026-11-20 16:53:25 +00:00
mesh-tools fb23967de3 Merge pull request 'Design 47 H1: a pipeline ends failed as soon as it can no longer pass; end-again judges it by hand (H-A-163, H-A-164, H-A-171)' (#278) from fix/47-H1-runner-ends-multi-mover into main 2026-11-20 16:51:19 +00:00
jschoubben 9083287d57 mesh-delivery-indexer: a passed pipeline's release pins its revision until a newer one is the want
The pin judge removed main.23 and its entries when main.24 became current, while the assigner had
just written main.23's builder as the want: the node-engine asked its entry and was told none is
kept, holding the want for good. A passed pipeline-ended now pins the pipeline's kept revision
under a record release.<branch>.<module>, one per mover, beside the pipeline's own pin; the seat
mesh-wants' pipeline-wanted of a newer pipeline on that branch (its wants written to the end,
tier-late included) frees that record from every revision below the newer one, for each module the
newer revision's movers name. The indexer consumes pipeline-wanted (its manifest says so). A
revision a release pin holds keeps its entries, as removeUnlisted already does for any kept revision.

Tests: rows in the pins table (a passed pipeline's revision kept once no longer current; a failed
one's goes), the freeing in the pins test (a release past a failed pipeline, freed by the next
passed one's pipeline-wanted, twice changing nothing), and a pipeline-wanted row in the events
test. TestIndexerWhole now reads main.11's release beside its validate base, and waits for the held
source-moved's acknowledgement before it reads the consumer (the write comes first). No test added.

Builds: design 47 H-A-168, H-A-170 (hq design/47-H1-answers-59 87b43ec8), H-A-147, H §749
2026-11-20 17:50:08 +01:00
jschoubben 07dd8ea842 Design 47 H1: the end names why each mover has no step, its registration or a publish (H-A-171)
The end judgement took every mover with no step as waiting on its
registration; one may wait for the publish of a mover it builds on. The
registration lookup tells them apart, and the why names each with its own
reason.

Builds: design 47 H §746, H-A-163, H-A-171, G A-327
2026-11-20 17:48:10 +01:00
jschoubben d0436bb38c mesh-delivery-indexer: pins are in every form's fingerprint
The import form's fingerprint left the pins out, so an importer whose
trunk build reported its imports never moved on a re-pin and was never
built again against the release. Every form now takes its sorted pins;
a build that stands keeps its fingerprint while its pins are the ones
it was built with, and a fingerprint with no pin stays as it was.

Builds: design 47 H H-A-172, H-A-167
2026-11-20 17:46:53 +01:00
jschoubben fe51379ee2 mesh-delivery-indexer: a release re-pins or is not acknowledged; pins enter the version line's fingerprint
A re-pin that left the fingerprint as it was left the build made
against the older pin as the one the mesh runs. The version line's
fingerprint now takes the pins, so a re-pin moves the importer, which
becomes a mover built against the release. An accepted series' older
version no longer stands over a newer release. A re-pin that fails
returns its error before the heard artifacts are forgotten, and a
package-published heard again for a released version re-pins, so the
redelivery recovers it. The release path and the seed's pins carry
tests that fail without them (review round 2).

Builds: design 47 H H-A-165, H-A-167, H-A-161
2026-11-20 17:46:53 +01:00
jschoubben 037cb9555e mesh-delivery-indexer: a version-released re-pins the modules holding the package
Pins moved to a new released version only at the next merge, toolchain
report or hand recompute. The release now starts the pass of H-A-161
for the modules whose packages hold a package of the released module
alone; every other module stands as held.

Builds: design 47 H H-A-165 (the indexer's part), H-A-161
2026-11-20 17:46:53 +01:00
jschoubben 5c5136d6d1 mesh-delivery-indexer: the requirements state reads the go.sum whatever packages the reader carried
With the go.mod's mesh requires carried at every take-in, requirementsOf
returned early for every module requiring a mesh package: its go.sum was
never read and keepRequirements never cleared an old version it named
(indexer_requirements for mesh-sdk still names v0.1.13). The go.sum is
read whenever its hash moved or this process has not kept it yet.

Builds: design 47 H H-A-157 (review round 1)
2026-11-20 17:46:53 +01:00
jschoubben c43e7a5d1c mesh-delivery-indexer, mesh-delivery-builder: a package is pinned by its released version, never by a commit
The indexer filled a module's packages with the commit of a module value
it holds, which Go turned into a pseudo-version of whatever that tree
was (mesh-delivery-sdk at an 11-19 commit, nodeengine pre-H1). A pin is
now the newest released version of the module holding the package (the
versions state version-released says), an accepted series' version
stands, and a package with none released is kept with no version: its
kind-1 edge stands and the builder asks no pin for it. A commit of
another repository moves a module only through a package of it the
module imports (A-476a), now that pins name no commit.

Builds: design 47 H H-A-161, G §370, H §746, H-A-157 (kind-1 edges)
2026-11-20 17:46:53 +01:00
jschoubben e04321acae mesh-delivery-indexer: a build's reported imports refine the go.mod's packages, never empty them
An import form whose imports name no package of another repository, or a
build that stands with versions held empty from the go.sum era, wrote an
empty package set while the go.mod requires a mesh package: no kind-1
edge. The version line's versions of the go.mod's set stand then.

Builds: design 47 H H-A-157, G §47 (kind 1)
2026-11-20 17:46:53 +01:00
jschoubben 4c2914194b mesh-delivery-indexer: a module's mesh packages are what its go.mod requires
At main's revision 21, 64 of 82 Go modules carried no package edge: the
version line read the mesh packages go.sum names, and go.sum names none
required at the zero version (ADR 0573 §2). The take-in now reads them
from the go.mod's require lines, at any version, with no build needed;
the go.sum is read only for the requirements state. The sibling rule
(replaces by a directory) stands.

Builds: design 47 H H-A-157, H-A-136 (sibling rule), G §47 (kind 1)
2026-11-20 17:46:53 +01:00
mesh-tools 277c53067e Merge pull request 'Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7)' (#273) from proof/47-H1-two-modules into main 2026-11-20 16:44:51 +00:00
mesh-tools 227f6e68ce Merge pull request 'Design 47 H1: a build with an unpinned package waits for its release (H-A-165)' (#277) from fix/47-H1-builder-waits-package into main 2026-11-20 16:41:47 +00:00
jschoubben 521cbf089e Design 47 H1: the seat mesh-pipeline-runner gains end-again, a hand act judging a pipeline's end once more (H-A-164)
A pipeline the end judgement before H-A-163 left running had no way to end
but a cancel. The runner's process answers end-again, so the end takes its
turn in the queue; runner_end_again asks it. The seat's definition changes:
its Go and TypeScript contracts are generated again.

Builds: design 47 H §746, H-A-163, H-A-164
2026-11-20 17:38:20 +01:00
jschoubben 39a2c6a944 Design 47 H1: a pipeline ends failed once it can no longer pass, a mover waiting on its registration named (H-A-163)
main.20 and main.22 never ended: a mover with no step placed (no registration)
held the end even after a placed mover's required step failed and nothing was
open. It now holds the end only while the pipeline could still pass.

Builds: design 47 H §746, H-A-163, G A-327
2026-11-20 17:38:20 +01:00
jschoubben 577ec05205 Design 47 H1: the runner says a mover waits on its registration when placed from the half or the queue (A-327)
A pipeline placed at its turn in the queue, or once its branch's half was held,
dropped its movers with no registration without a word: main.20 and main.22 sat
running with every step ended and nothing said why (their mover
mesh-delivery-contracts has no registration, so it is never placed and the
pipeline never ends).

Builds: design 47 G A-327, design 47 H §746
2026-11-20 17:38:20 +01:00
jschoubben 744204c5ab Builder: releases heard wake the package wait; its setting package-wait-minutes (H-A-165)
The process tells the builder each version-released it takes in and
gives it the newest release its record keeps, so an unpinned package's
step ask wakes on the event and pins the release; the wait's bound is
the setting package-wait-minutes (default 10), carried as a file under
the state dir like revision-wait-minutes.

Builds: design 47 H H-A-165, H-A-156, H-A-155
2026-11-20 17:37:25 +01:00
jschoubben 0f41b405ba Builder: a step ask naming a package with no pin waits for its release (H-A-165)
An entry keeps a mesh package with no released version unpinned
(H-A-161). Its step ask now waits, in the wait of H-A-155, for the
release of the module holding the package, heard or kept in the
builder's record of releases, and builds with that release as the pin;
past the bound the step fails naming the package and builds nothing.

Builds: design 47 H H-A-165, H-A-155, H-A-161
2026-11-20 17:37:25 +01:00
mesh-tools 20f0e9baaf Merge pull request 'Design 47 H1: a build waits for its revision and never builds without a version; a run's ask names no source (H-A-155, H-A-156, H-A-162)' (#276) from fix/47-H1-builder-waits-revision into main 2026-11-20 16:35:46 +00:00
mesh-tools 75c636a5ec Merge pull request 'Design 47 H1: the builder numbers nothing of its own; a published of other bytes leaves a kept entry (H-A-159)' (#275) from fix/47-H1-builder-versions-record into main 2026-11-20 16:32:12 +00:00
jschoubben 0cbabe9860 Runner and builder: a run's build step ask names no source; a held revision wakes the builder's wait (H-A-162)
A run on request copied its pipeline's source onto its build step ask,
so the builder waited for the run's candidate fingerprints in its
picture of the branch, where they never come, and failed every run past
the bound. The seat mesh-builds' contract leaves a run's source absent:
the runner now does. A revision the builder's view takes in whole that
changes no entry now also wakes a waiting ask, told by the catalog
client's new Held.

Builds: design 47 H H-A-162, H-A-155
2026-11-20 17:30:56 +01:00
jschoubben d2c9dc7de6 builder: builder_floor_again refuses; it asks no floor any more
With nothing numbered of the builder's own, the tool answered that a floor
would be asked at the next claim and looked again, doing nothing.

Builds: design 47 H H-A-159, G §369
2026-11-20 17:30:49 +01:00
jschoubben 53199f16ce Builder: the revision wait reads the picture and its setting revision-wait-minutes (H-A-155, H-A-156)
The process gives the builder its picture of each watched branch (the
source revision the branch's view holds) and the bound of a step ask's
wait for its revision, read from the setting revision-wait-minutes
(default 2) carried as a file under the state dir like
toolchain-wait-minutes.

Builds: design 47 H H-A-155, H-A-156, H-A-141
2026-11-20 17:27:37 +01:00
jschoubben 256eb34a0d Builder: a build step ask waits for its revision before it builds (H-A-155)
A step ask reached the builder before its picture of main's catalog
revisions held the revision the pipeline builds; the fingerprint was
built as no first branch's build, reserved no version, and the step
failed on builder_reserve_done's words. The ask now waits, woken by each
entry the picture takes in, until the picture holds its source revision
and the asked entry from its branch; past the bound it fails naming the
revision and builds nothing.

Builds: design 47 H H-A-155, H-A-141
2026-11-20 17:27:37 +01:00
jschoubben a47c8e7922 builder: number nothing of its own; the reservation numbers
Every look numbered one more patch in the state versions (claims carry no
source revision since H-A-48): nodeengine 0.1.5 to 0.1.36, beside the
reservation's numbers.

Builds: design 47 H H-A-159, G §348, §349, §369
2026-11-20 17:24:48 +01:00
jschoubben 16f9ad2950 builder: a published of other bytes leaves a kept versions entry as it is
The reservation's published of nodeengine 0.1.4 (9724da7a) was taken into the
state versions' entry 0.1.4, numbered for an older fingerprint, which turned
published and kept the old digest c0ef9a3d.

Builds: design 47 H H-A-159, G §369, A-415; ADR 0605 §4
2026-11-20 17:23:19 +01:00
jschoubben e312999843 Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7) 2026-11-20 17:06:04 +01:00
119 changed files with 4519 additions and 573 deletions
@@ -95,15 +95,17 @@ type machineEnd struct {
// run runs one composition step and answers what it came to: per machine not away it joins or starts the candidate's // run runs one composition step and answers what it came to: per machine not away it joins or starts the candidate's
// composition run (§718), waits for each run's end in its key, and judges each machine from it; failed when a machine does not // composition run (§718), waits for each run's end in its key, and judges each machine from it; failed when a machine does not
// compose, could not run when a machine could not be composed or was not judged, else passed with a warning per machine away. // compose, could not run when a machine could not be composed or was not judged, nothing to run when no node places the module on
// the branch (H-A-179), else passed with a warning per machine away.
func (r *compositionRun) run(ctx context.Context, id string, step meshwants.StepRun) stepOutcome { func (r *compositionRun) run(ctx context.Context, id string, step meshwants.StepRun) stepOutcome {
candidate, err := candidateKeyOf(step) candidate, err := candidateKeyOf(step)
if err != nil { if err != nil {
return couldNotRun(err.Error()) return couldNotRun(err.Error())
} }
nodes := r.inputs.machines(step.Module, step.Branch) nodes := r.inputs.machines(step.Module, step.Branch)
if len(nodes) == 0 { if len(nodes) == 0 { // a mover no machine holds on the branch has nothing to judge: nothing to run, passed (H-A-179, ADR 0682 §8)
return couldNotRun(fmt.Sprintf("no node places %s on the branch %q, so the composition step has no machine to compose", step.Module, step.Branch)) return stepOutcome{Outcome: meshwants.StepEndedOutcomeNothingToRun,
Said: fmt.Sprintf("no node places %s on the branch %q, so the composition step has no machine to compose", step.Module, step.Branch)}
} }
reader := readerOf(id, step) reader := readerOf(id, step)
prep := &runPrep{r: r, step: step} prep := &runPrep{r: r, step: step}
@@ -259,7 +259,11 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
} }
none := newRunStand(t, "src2:e1", "src2:e1", "d1") none := newRunStand(t, "src2:e1", "src2:e1", "d1")
none.r.inputs.machines = func(string, string) []string { return machines } none.r.inputs.machines = func(string, string) []string { return machines }
if out := none.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeCouldNotRun || outcome := meshwants.StepEndedOutcomeCouldNotRun
if len(machines) == 0 { // a mover no machine holds: nothing to run, saying why (H-A-179)
outcome = meshwants.StepEndedOutcomeNothingToRun
}
if out := none.r.run(context.Background(), "ask-step", step); out.Outcome != outcome ||
!strings.Contains(out.Said, want) || (len(machines) > 0 && !strings.Contains(out.Said, "shanks is away")) || len(none.asks) != 0 { !strings.Contains(out.Said, want) || (len(machines) > 0 && !strings.Contains(out.Said, "shanks is away")) || len(none.asks) != 0 {
t.Errorf("%s came to %+v, %d asks", name, out, len(none.asks)) t.Errorf("%s came to %+v, %d asks", name, out, len(none.asks))
} }
@@ -19,6 +19,7 @@ import (
"runtime" "runtime"
"slices" "slices"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
@@ -203,11 +204,17 @@ func (w *watchKV) Keys(context.Context, ...jetstream.WatchOpt) ([]string, error)
return nil, jetstream.ErrNoKeysFound return nil, jetstream.ErrNoKeysFound
} }
// putIntoMu holds putInto's writes one at a time: the composition step starts its machines' runs side by side, and each may
// raise a finding into the same map.
var putIntoMu sync.Mutex
// putInto is an assign.Findings Put that keeps each finding in into, as the finding writer would, by its key and its words. // putInto is an assign.Findings Put that keeps each finding in into, as the finding writer would, by its key and its words.
func putInto(into map[string][]byte) func(mcseat.Finding) error { func putInto(into map[string][]byte) func(mcseat.Finding) error {
return func(f mcseat.Finding) error { return func(f mcseat.Finding) error {
raw, err := json.Marshal(f) raw, err := json.Marshal(f)
putIntoMu.Lock()
into[views.FindingKey(f.Scope, f.ID, f.Kind)] = raw into[views.FindingKey(f.Scope, f.ID, f.Kind)] = raw
putIntoMu.Unlock()
return err return err
} }
} }
@@ -119,6 +119,13 @@ func TestManifest(t *testing.T) {
} }
} }
} }
// H-A-186 (design 47 G §370): asks, which the gate, the teller and the view import, is a declared Go package of the
// module, so every release publishes it; H-A-188: in the module's version series, 0.3.
if !slices.ContainsFunc(m.Build.Artifacts, func(a catalogue.Artifact) bool {
return a.Name == "asks" && a.Kind == catalogue.ArtifactPackage && a.Language == "go" && slices.Equal(a.Paths, []string{"asks"}) && a.Series == m.Version && m.Version == "0.3"
}) {
t.Errorf("artifacts = %+v, want the Go package asks of series 0.3, the module's", m.Build.Artifacts)
}
// ADR 0635 §5: its views family is its ordinary events, `views.<state>` among its emits, on no stream of its own. // ADR 0635 §5: its views family is its ordinary events, `views.<state>` among its emits, on no stream of its own.
// The seat's four events are said on the seat's own subject (the seat's family), so the module keeps no stream for them // The seat's four events are said on the seat's own subject (the seat's family), so the module keeps no stream for them
// and lists none among its emits. // and lists none among its emits.
@@ -370,6 +370,15 @@
"loads": [ "loads": [
"mesh-delivery-assigner" "mesh-delivery-assigner"
] ]
},
{
"name": "asks",
"kind": "package",
"language": "go",
"paths": [
"asks"
],
"series": "0.3"
} }
] ]
} }
@@ -45,7 +45,7 @@ func (m *entryManifests) keep(e meshcatalog.CatalogEntry) error {
if m.byFp == nil { if m.byFp == nil {
m.byFp = map[string]catalogue.Manifest{} m.byFp = map[string]catalogue.Manifest{}
} }
m.byFp[meshcatalog.BareManifestDigest(e.ManifestDigest)] = mf m.byFp[meshcatalogclient.BareManifestDigest(e.ManifestDigest)] = mf
return nil return nil
} }
@@ -132,6 +132,20 @@ func (c *catalogHalf) Changed(ctx context.Context) func(branch, module string, e
} }
} }
// picture is build.Builder.Picture: the source revision the view of branch holds (design 47 H H-A-155), false while the branch
// is not followed or its view holds none.
func (c *catalogHalf) picture(branch string) (int64, bool) {
cat, ok := c.branches.Catalog(branch)
if !ok {
return 0, false
}
half, ok := cat.View()
if !ok {
return 0, false
}
return half.Revision.Source, true
}
// namesOf are the package names an entry's artifacts carry, in one line. // namesOf are the package names an entry's artifacts carry, in one line.
func namesOf(e meshcatalog.CatalogEntry) string { func namesOf(e meshcatalog.CatalogEntry) string {
var names []string var names []string
@@ -33,8 +33,13 @@ type catalogHolder struct {
mu sync.Mutex mu sync.Mutex
entries map[string]meshcatalog.CatalogEntry entries map[string]meshcatalog.CatalogEntry
revision int64 revision int64
// source is the source revision each trunk revision is taken in from; 0 is 1.
source int64
} }
// sourceLocked is the source revision of the revisions put now. The caller holds mu.
func (h *catalogHolder) sourceLocked() int64 { return max(h.source, 1) }
func newCatalogHolder(t *testing.T, ctx context.Context, js jetstream.JetStream, nc *nats.Conn) *catalogHolder { func newCatalogHolder(t *testing.T, ctx context.Context, js jetstream.JetStream, nc *nats.Conn) *catalogHolder {
t.Helper() t.Helper()
h := &catalogHolder{ctx: ctx, js: js, entries: map[string]meshcatalog.CatalogEntry{}} h := &catalogHolder{ctx: ctx, js: js, entries: map[string]meshcatalog.CatalogEntry{}}
@@ -84,7 +89,7 @@ func (h *catalogHolder) whole() meshcatalog.RevisionAnswer {
all = append(all, e) all = append(all, e)
} }
at := time.Now() at := time.Now()
return meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: h.revision, Kind: "trunk", Branch: "main", Entries: refs, Source: 1, At: &at}, Entries: all} return meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: h.revision, Kind: "trunk", Branch: "main", Entries: refs, Source: h.sourceLocked(), At: &at}, Entries: all}
} }
// put sets one module's entry in a new trunk revision of main and says catalog-revision. // put sets one module's entry in a new trunk revision of main and says catalog-revision.
@@ -100,7 +105,7 @@ func (h *catalogHolder) putAll(t *testing.T, es ...meshcatalog.CatalogEntry) {
h.entries[e.Module] = e h.entries[e.Module] = e
changed = append(changed, e.Module) changed = append(changed, e.Module)
} }
ev, _ := json.Marshal(meshcatalog.CatalogRevisionEvent{Branch: "main", Kind: "trunk", Revision: h.revision, Source: 1, Entries: int64(len(h.entries)), Changed: changed, V: 1}) ev, _ := json.Marshal(meshcatalog.CatalogRevisionEvent{Branch: "main", Kind: "trunk", Revision: h.revision, Source: h.sourceLocked(), Entries: int64(len(h.entries)), Changed: changed, V: 1})
h.mu.Unlock() h.mu.Unlock()
if _, err := h.js.Publish(h.ctx, meshcatalog.SubjectCatalogRevision("main"), ev); err != nil { if _, err := h.js.Publish(h.ctx, meshcatalog.SubjectCatalogRevision("main"), ev); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -522,9 +522,9 @@ func isState(state string) func(builds.BuilderBuild) bool {
// the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4). // the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4).
func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7 func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
b := startBed(t) b := startBed(t)
// DLV-B10 (design 51, round 2): the builder's versions, findings and the floor's three sources are on the bus; the seat // The builder's versions, findings and the floor's three sources are on the bus, the seat answering 0.1.4; two merges close
// answers 0.1.4, so netcheck's bundles are numbered from 0.1.5. Two merges close together, the first replaced before // together, the first replaced before its claim. The builder numbers neither in its state versions (design 47 H H-A-159,
// its claim: the second is numbered at once, and the first never. // G §348, §369): the reservation is the one place a version is numbered.
reg := b.registry(t, "n1", "0.1.4") reg := b.registry(t, "n1", "0.1.4")
const netcheck = "catalogue-a67f0196-netcheck.module.json" const netcheck = "catalogue-a67f0196-netcheck.module.json"
first := "src2:" + strings.Repeat("1", 64) first := "src2:" + strings.Repeat("1", 64)
@@ -534,8 +534,8 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
v := b.await(t, "netcheck", isState(builds.BuildDone)) v := b.await(t, "netcheck", isState(builds.BuildDone))
for _, a := range []string{"tools-go", "tools-typescript"} { for _, a := range []string{"tools-go", "tools-typescript"} {
es := reg.of(t, "netcheck/"+a) es := reg.of(t, "netcheck/"+a)
if len(es) != 1 || es[0].Version != "0.1.5" || es[0].Fingerprint != fp || es[0].State != versions.Numbered || es[0].Digest != digestOf(a) { if len(es) != 0 {
t.Errorf("versions of netcheck/%s = %+v, want the second merge numbered 0.1.5 at once with its digest, the first never", a, es) t.Errorf("versions of netcheck/%s = %+v, want none: the builder numbers nothing of its own (H-A-159)", a, es)
} }
} }
if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 { if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 {
@@ -565,13 +565,13 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
} }
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a // 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made. // reader with the log's read reads them, the end naming the manifest and what was made.
// DLV-B10: between them the build log holds the build's narrations (build.said), each version numbered, and why. // H-A-159: no version is numbered of the builder's own, so the build log holds no narration of one (build.said).
var ended []builds.BuildLogEntry var ended []builds.BuildLogEntry
for _, e := range b.logged(t, "netcheck") { for _, e := range b.logged(t, "netcheck") {
if e.Event != builds.BuildSaidEvent { if e.Event != builds.BuildSaidEvent {
ended = append(ended, e) ended = append(ended, e)
} else if e.Why != versions.SaidNumbered || !strings.Contains(e.Line, "numbered 0.1.5") || e.FP != fp { } else if e.Why == versions.SaidNumbered || e.Why == versions.SaidKept {
t.Errorf("said = %+v, want netcheck's versions numbered", e) t.Errorf("said = %+v, want no version of the builder's own numbered (H-A-159)", e)
} }
} }
if len(ended) != 2 || ended[0].Event != builds.BuildStartedEvent || ended[1].Event != builds.BuildEndedEvent || if len(ended) != 2 || ended[0].Event != builds.BuildStartedEvent || ended[1].Event != builds.BuildEndedEvent ||
@@ -596,7 +596,7 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
} }
} }
// Design 32, design 47 G §348: a package's name the catalog half's entry of the module gives later reaches the builder by the // Design 32, design 47 G §348: a package's name the catalog half's entry of the module gives later reaches the builder by the
// branch follower's view (never the indexer's state), and looks at the module again; the done bundles are kept, not numbered twice. // branch follower's view (never the indexer's state), and looks at the module again.
named := b.entry("netcheck", "modules/netcheck", commit, fp, digest, nil, bothToolchains, 0) named := b.entry("netcheck", "modules/netcheck", commit, fp, digest, nil, bothToolchains, 0)
named.Manifest = b.manifestOf(t, "catalogue-a67f0196-netcheck.module.json") named.Manifest = b.manifestOf(t, "catalogue-a67f0196-netcheck.module.json")
named.Artifacts = []meshcatalog.Artifact{{Ref: "netcheck:lib", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "netcheck"}}} named.Artifacts = []meshcatalog.Artifact{{Ref: "netcheck:lib", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "netcheck"}}}
@@ -606,11 +606,12 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
if got := reg.numberedOf("netcheck/tools-go"); len(got) != 0 { if got := reg.numberedOf("netcheck/tools-go"); len(got) != 0 {
t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got) t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got)
} }
for _, a := range []string{"tools-go", "tools-typescript"} { // Versions numbered before G, which the state keeps until G §372 removes it: one a holder removes below, one it refuses.
reg.published(t, "netcheck/"+a, "0.1.5", digestOf(a)) kept, _ := json.Marshal(versions.Entry{Name: "netcheck/tools-typescript", Kind: versions.KindBundle, Series: "0.1", Version: "0.1.5", Module: "netcheck",
Fingerprint: fp, Digest: digestOf("tools-typescript"), Seat: ociartifactregistry.Seat, State: versions.Published, At: time.Now().UTC()})
if _, err := reg.versions.Put(b.ctx, versions.Key("netcheck/tools-typescript", "0.1.5"), kept); err != nil {
t.Fatal(err)
} }
reg.awaitOf(t, "netcheck/tools-go", func(es []versions.Entry) bool { return len(es) == 1 && es[0].State == versions.Published })
// A numbered version a holder will refuse below, written to the state as the builder numbers one.
refusable, _ := json.Marshal(versions.Entry{Name: "netcheck/tools-go", Kind: versions.KindBundle, Series: "0.1", Version: "0.1.6", Module: "netcheck", refusable, _ := json.Marshal(versions.Entry{Name: "netcheck/tools-go", Kind: versions.KindBundle, Series: "0.1", Version: "0.1.6", Module: "netcheck",
Fingerprint: "src2:" + strings.Repeat("6", 64), Seat: ociartifactregistry.Seat, State: versions.Numbered, At: time.Now().UTC()}) Fingerprint: "src2:" + strings.Repeat("6", 64), Seat: ociartifactregistry.Seat, State: versions.Numbered, At: time.Now().UTC()})
if _, err := reg.versions.Put(b.ctx, versions.Key("netcheck/tools-go", "0.1.6"), refusable); err != nil { if _, err := reg.versions.Put(b.ctx, versions.Key("netcheck/tools-go", "0.1.6"), refusable); err != nil {
@@ -643,7 +644,7 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got) t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got)
} }
// DLV-B10: a build module's bound running out fails the trunk build, which is the builder's finding build-failed; the // DLV-B10: a build module's bound running out fails the trunk build, which is the builder's finding build-failed; the
// fingerprint built again keeps its number, said with why, and its done build clears the finding. // fingerprint built again clears it with its done build, numbering nothing of the builder's own (H-A-159).
release := make(chan struct{}) release := make(chan struct{})
b.fake.hang.Store(&release) b.fake.hang.Store(&release)
third := "src2:" + strings.Repeat("4", 64) third := "src2:" + strings.Repeat("4", 64)
@@ -666,12 +667,6 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
close(release) close(release)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, "89abcdef0123456789abcdef0123456789abcdef", third, 3) // the entry moves to a later commit, same fingerprint b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, "89abcdef0123456789abcdef0123456789abcdef", third, 3) // the entry moves to a later commit, same fingerprint
b.awaitFP(t, "netcheck", third, isState(builds.BuildDone)) b.awaitFP(t, "netcheck", third, isState(builds.BuildDone))
b.awaitSaid(t, "kept: fingerprint "+third+" is numbered under it")
if !slices.ContainsFunc(b.logged(t, "netcheck"), func(e builds.BuildLogEntry) bool {
return e.Event == builds.BuildSaidEvent && e.Why == versions.SaidKept && e.FP == third && strings.Contains(e.Line, "kept: fingerprint "+third)
}) {
t.Errorf("the build log of netcheck says no version kept at %s", third)
}
reg.awaitFinding(t, failed, false) reg.awaitFinding(t, failed, false)
// DLV-B4: the seats' refused and removed, heard on the builder's durable consumer, move a numbered version: a retention // DLV-B4: the seats' refused and removed, heard on the builder's durable consumer, move a numbered version: a retention
// removal turns the entry removed with its reason, a refusal refused with the holder's reason and the bytes it refused. // removal turns the entry removed with its reason, a refusal refused with the holder's reason and the bytes it refused.
@@ -35,6 +35,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
sweep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshartifactsweep" sweep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshartifactsweep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/operatorask" "git.novox.be/novox/mesh-sdk/go/operatorask"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
@@ -156,9 +157,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") }) conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs // The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done, on its calls connection. // (the seat verb held, novox/hq issue 1937) before it leaves the key done, on its calls connection.
// The builder numbers each trunk build at its claim (novox/hq design 51 *The versions*): its state versions, bound once // The state versions, kept from before G (design 51 *The versions*) and read through builder_versions until G §372 removes
// the bus holds it; the floor's three sources asked as its own account. // it, bound once the bus holds it; the builder numbers nothing in it (design 47 H H-A-159).
versionsKV := process.AwaitBucket(ctx, js, conn, own.versions, "no trunk build is numbered: each is built unnumbered and published by its digest", logger.Printf) versionsKV := process.AwaitBucket(ctx, js, conn, own.versions, "builder_versions answers nothing of the versions numbered before G", logger.Printf)
found := &findings{module: cred.Module, w: fw, logf: logger.Printf, now: time.Now} found := &findings{module: cred.Module, w: fw, logf: logger.Printf, now: time.Now}
floorWaits := &versions.FloorWaits{Holder: versions.HolderOf(branches), Logf: logger.Printf, Said: found.floorSaid(context.WithoutCancel(ctx))} floorWaits := &versions.FloorWaits{Holder: versions.HolderOf(branches), Logf: logger.Printf, Said: found.floorSaid(context.WithoutCancel(ctx))}
// The names waiting on their floor are kept in the state floor-waits (DLV-B4): a restarted builder takes them in, so // The names waiting on their floor are kept in the state floor-waits (DLV-B4): a restarted builder takes them in, so
@@ -205,13 +206,30 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
unput := &unputs{bounds: pubBoundsKV.KV, store: b.Claimer.Store, looking: b.Looking, findings: found, unput := &unputs{bounds: pubBoundsKV.KV, store: b.Claimer.Store, looking: b.Looking, findings: found,
bound: meshconditionsclient.CallUnansweredAfter, now: time.Now, logf: logger.Printf} bound: meshconditionsclient.CallUnansweredAfter, now: time.Now, logf: logger.Printf}
b.Left, b.Ended = unput.left, unput.ended b.Left, b.Ended = unput.left, unput.ended
b.Numbers = build.VersionNumbers{N: numberer} // The builder numbers nothing of its own (design 47 H H-A-159, G §348, §349, §369): b.Numbers stays nil, so no trunk build
// is numbered in the state versions, waits on a floor or arms a publish-unanswered bound of it; the reservation is the one
// place a version is numbered. The state, kept from before G, is still read (builder_versions) until G §372 removes it.
// The builder's toolchain waits (design 47 G §672): its state waits, and wait-held and wait-ended on its stream waits; its // The builder's toolchain waits (design 47 G §672): its state waits, and wait-held and wait-ended on its stream waits; its
// bound is its setting toolchain-wait-minutes. // bound is its setting toolchain-wait-minutes.
waitBound, err := toolchainWaitOf(getenv("MESH_BUILDER_TOOLCHAIN_WAIT_FILE")) waitBound, err := toolchainWaitOf(getenv("MESH_BUILDER_TOOLCHAIN_WAIT_FILE"))
if err != nil { if err != nil {
return err return err
} }
// A build step ask waits for the builder's picture of its branch to hold its source revision (design 47 H H-A-155); its
// bound is its setting revision-wait-minutes (H-A-156).
revisionWait, err := numberFile(getenv("MESH_BUILDER_REVISION_WAIT_FILE"), int(build.DefaultRevisionWait/time.Minute))
if err != nil {
return err
}
b.RevisionWait = time.Duration(revisionWait) * time.Minute
// A step ask naming a package with no pin waits for its release (H-A-165); its bound is package-wait-minutes.
packageWait, err := numberFile(getenv("MESH_BUILDER_PACKAGE_WAIT_FILE"), int(build.DefaultPackageWait/time.Minute))
if err != nil {
return err
}
b.PackageWait = time.Duration(packageWait) * time.Minute
// Before it waits for an unpinned package, the build reads what the Go package registry seat holds of it (H-A-194).
b.PublishedVersions = publishedVersionsVia(calls.Call)
waitsBucket := process.AwaitBucket(ctx, js, conn, builds.StateBucket(cred.Module, build.WaitsState), "the toolchain waits are kept in memory only, and builder_waits shows none", logger.Printf) waitsBucket := process.AwaitBucket(ctx, js, conn, builds.StateBucket(cred.Module, build.WaitsState), "the toolchain waits are kept in memory only, and builder_waits shows none", logger.Printf)
b.Waits = newWaitsKept(waitsBucket, func(ctx context.Context, event, id string, body any) error { b.Waits = newWaitsKept(waitsBucket, func(ctx context.Context, event, id string, body any) error {
return emitEvent(context.WithoutCancel(ctx), js, cred, event, id, body) return emitEvent(context.WithoutCancel(ctx), js, cred, event, id, body)
@@ -450,6 +468,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// fingerprint no branch carries is closed (issue 1660) once the first view of every followed branch is in (never at the start: // fingerprint no branch carries is closed (issue 1660) once the first view of every followed branch is in (never at the start:
// a holder of the seat that is down holds no other part of it). // a holder of the seat that is down holds no other part of it).
half.b = b half.b = b
b.Picture = half.picture
branches.Held = func(string, meshcatalog.CatalogRevision) { b.PictureHeld() } // a revision held wakes a step ask's wait (H-A-162)
half.ready = func() { half.ready = func() {
if err := b.Sweep(ctx, half.pictured()...); err != nil { if err := b.Sweep(ctx, half.pictured()...); err != nil {
errs <- err errs <- err
@@ -105,6 +105,10 @@ func (h *seatHearer) heard(ctx context.Context, m jetstream.Msg) bool {
if h.res != nil && se.event == versions.HeardPublished { if h.res != nil && se.event == versions.HeardPublished {
h.res.published(ctx, o) h.res.published(ctx, o)
} }
if se.event == versions.HeardPublished && o.Kind == versions.KindPackage && h.b != nil {
// A package registry seat's published pins a package an entry names with none, at the version it carries (H-A-192).
h.b.PackagePublished(o.Name, o.Version)
}
h.ack(m) h.ack(m)
if len(wake) > 0 { if len(wake) > 0 {
h.logf("the floor of %s is asked again: %s said %s", strings.Join(wake, ", "), se.seat, se.event) h.logf("the floor of %s is asked again: %s said %s", strings.Join(wake, ", "), se.seat, se.event)
@@ -4,6 +4,7 @@ package main
import ( import (
"encoding/json" "encoding/json"
"strings"
"testing" "testing"
"time" "time"
@@ -49,4 +50,29 @@ func TestTheRunnersBuildStepAskIsAnsweredAsStepEnded(t *testing.T) {
case <-time.After(20 * time.Second): case <-time.After(20 * time.Second):
t.Fatal("no step-ended within 20 s") t.Fatal("no step-ended within 20 s")
} }
// H-A-155: an ask naming a source revision the builder's picture of main does not hold yet waits for it, said in the
// journal, and is answered once a revision taken in from it arrives.
run.Pipeline, run.Revision, run.Source = "main.2.netcheck", 2, 2
data, _ = json.Marshal(run)
msg = nats.NewMsg(meshbuilds.AcceptSubjectStep)
msg.Data = data
msg.Header.Set("x-event-id", "ask-2")
if err := sub.PublishMsg(msg); err != nil {
t.Fatal(err)
}
b.awaitSaid(t, "main.2.netcheck: revision 2 of main not in the builder's picture yet")
b.catalog.mu.Lock()
b.catalog.source = 2
b.catalog.mu.Unlock()
// Revision 2, taken in from source 2, moves another module: netcheck's entry is the one the picture holds already.
b.moduleFP(t, "ca-trust", "catalogue-a67f0196-ca-trust.module.json", "modules/ca-trust", nil, commit, "src2:"+strings.Repeat("4", 64), 0)
select {
case m := <-ended:
var end meshbuilds.StepEnded
if err := json.Unmarshal(m.Data, &end); err != nil || end.ID != "ask-2" || end.Outcome != meshbuilds.StepEndedOutcomePassed {
t.Fatalf("step-ended = %+v, %v; want the ask ask-2 passed once the picture held source revision 2", end, err)
}
case <-time.After(20 * time.Second):
t.Fatal("no step-ended for ask-2 within 20 s")
}
} }
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions" "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git" gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build" "mesh-delivery-builder/internal/build"
@@ -96,6 +97,28 @@ func (s awaitedVersions) KeysOf(ctx context.Context, filter string) ([]string, e
return out, nil return out, nil
} }
// publishedVersionsVia asks the Go package registry seat's verb versions for one package path (design 47 H H-A-194): its
// versions, newest first.
func publishedVersionsVia(call toolCaller) func(context.Context, string) ([]string, error) {
return func(_ context.Context, path string) ([]string, error) {
raw, err := call("seat:"+gopackageregistry.Seat+"."+gopackageregistry.VerbVersions, gopackageregistry.VersionsInput{Name: path})
if err != nil {
return nil, err
}
var a gopackageregistry.VersionsAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s's versions answered what is no list of versions: %w", gopackageregistry.Seat, err)
}
var out []string
for _, v := range a.Versions {
if v.Version != "" {
out = append(out, v.Version)
}
}
return out, nil
}
}
// floorVia is the floor's three sources as calls of the builder's own account (design 51 *The floor*): the seat's verb // floorVia is the floor's three sources as calls of the builder's own account (design 51 *The floor*): the seat's verb
// versions, the seat git's tags and the requirements the catalog half's entries name (A-119). A call that fails is that source // versions, the seat git's tags and the requirements the catalog half's entries name (A-119). A call that fails is that source
// silent. // silent.
@@ -13,6 +13,7 @@ import (
func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) { func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) {
e := meshcatalog.CatalogEntry{Module: "netcheck", Repository: "novox/mesh-catalog", Path: "modules/netcheck", Commit: "abc", Fp: "src2:f", e := meshcatalog.CatalogEntry{Module: "netcheck", Repository: "novox/mesh-catalog", Path: "modules/netcheck", Commit: "abc", Fp: "src2:f",
ManifestDigest: "sha256:m", Bases: []string{"base"}, Packages: map[string]string{"p": "1"}, Needs: []string{"n"}, ManifestDigest: "sha256:m", Bases: []string{"base"}, Packages: map[string]string{"p": "1"}, Needs: []string{"n"},
Artifacts: []meshcatalog.Artifact{{Ref: "netcheck:lib", Kind: "package", Language: "go"}},
Inputs: &meshcatalog.EntryInputs{Toolchains: map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1", Recipe: "1"}}, Inputs: &meshcatalog.EntryInputs{Toolchains: map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1", Recipe: "1"}},
Paths: map[string][]string{"lib": {"a/b", "c"}}}} Paths: map[string][]string{"lib": {"a/b", "c"}}}}
got := ModuleOf(e) got := ModuleOf(e)
@@ -30,4 +31,11 @@ func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) {
if bare := ModuleOf(meshcatalog.CatalogEntry{Module: "m"}); bare.Inputs2 != nil || bare.Fp2 != "" { if bare := ModuleOf(meshcatalog.CatalogEntry{Module: "m"}); bare.Inputs2 != nil || bare.Fp2 != "" {
t.Errorf("ModuleOf of an entry with none = %+v, want nothing set", bare) t.Errorf("ModuleOf of an entry with none = %+v, want nothing set", bare)
} }
// A module with an artifact that names no paths (a Go program beside its packages) is built from its directory whole
// (ADR 0499 clauses 1 and 4): its fetch names no paths, so the program's go.mod and main package stay in the tree.
mixed := e
mixed.Artifacts = append([]meshcatalog.Artifact{{Ref: "netcheck:program", Kind: "bundle", Language: "go"}}, e.Artifacts...)
if got := ModuleOf(mixed); got.Inputs2 == nil || len(got.Inputs2.Paths) != 0 {
t.Errorf("ModuleOf of a module with a program naming no paths = %+v, want bases and no named paths", got.Inputs2)
}
} }
@@ -0,0 +1,275 @@
package build
import (
"context"
"fmt"
"maps"
"slices"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
)
// **A build waits for its revision; a build with no version is never made** (design 47 H H-A-155, H-A-156; H-A-141): a step
// ask names its branch and source revision, and the builder reserves the version before it builds. When the builder's picture
// of that branch does not hold that source revision yet, or holds it but has not taken in the entry of the asked fingerprint
// from it, the ask waits for the picture to take it in, woken by each entry the picture takes in (take, EntryGone) and each
// revision its view holds (PictureHeld, H-A-162), never a
// timer or a poll, up to RevisionWait (the setting revision-wait-minutes); past it the ask fails, naming the revision and the
// branch, and nothing is built. A step ask with no source (a run on request) does not wait.
// DefaultRevisionWait is the bound of a step ask's wait for its revision, its setting revision-wait-minutes' default (H-A-156).
const DefaultRevisionWait = 2 * time.Minute
func (b *Builder) revisionWait() time.Duration {
if b.RevisionWait > 0 {
return b.RevisionWait
}
return DefaultRevisionWait
}
// PictureHeld is told each revision a branch's view takes in whole, whether or not an entry changed (H-A-162): every step
// ask waiting for its revision looks again.
func (b *Builder) PictureHeld() {
b.init()
b.pictureMoved()
}
// pictureMoved wakes every step ask waiting for the picture: an entry was taken in or went, or a revision was held.
func (b *Builder) pictureMoved() {
b.mu.Lock()
moved := b.moved
b.moved = make(chan struct{})
b.mu.Unlock()
close(moved)
}
// holdsRevision says whether the builder's picture of branch holds the source revision and has taken in every entry of the
// ask from that branch.
func (b *Builder) holdsRevision(branch string, source int64, entries []meshbuilds.BuildEntry) bool {
held, ok := b.Picture(branch)
if !ok || held < source {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
for _, e := range entries {
key, err := builds.BuilderBuildKey(e.Module, e.Fingerprint)
if err != nil {
return true // the build key is refused where the entry is built, with why
}
if b.origin[key].Branch != branch && b.heard[branch+"\x00"+e.Module].Fp2 != e.Fingerprint {
return false
}
}
return true
}
// awaitRevision returns no outcome once the builder's picture of the ask's branch holds its source revision; past the bound the
// step's outcome failed, at the builder's stop could not run, each with its words.
func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (outcome, said string) {
if b.Picture == nil || run.Source <= 0 || run.Branch == "" {
return "", ""
}
bound := b.revisionWait()
notHeld := fmt.Sprintf("revision %d of %s not in the builder's picture", run.Source, run.Branch)
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
defer stop()
told := false
for {
b.mu.Lock()
moved := b.moved
b.mu.Unlock()
if b.holdsRevision(run.Branch, run.Source, run.Entries) {
if told {
b.logf("%s: revision %d of %s is in the builder's picture: the build goes on", run.Pipeline, run.Source, run.Branch)
}
return "", ""
}
if !told {
b.logf("%s: %s yet: the ask waits for it, at most %s", run.Pipeline, notHeld, bound)
told = true
}
select {
case <-moved:
case <-wait.Done():
if ctx.Err() != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, notHeld + ": the builder stopped before it was"
}
b.logf("%s: %s within %s: the step fails, nothing is built", run.Pipeline, notHeld, bound)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notHeld, bound)
}
}
}
// **A package with no pin waits for its publish** (design 47 H H-A-192, H-A-165, H-A-183, H-A-184, H-A-185): an entry
// keeps a mesh package the registry has not published within the series its module accepts with an empty version. Its
// build, a step ask's or a lane's, first reads the versions the registry seat holds of the path (its verb versions,
// H-A-194), then waits for the registry seat's published of that package path in that series (the
// series the module's manifest names in `packages`, else the default; catalogue.Manifest.AcceptedSeries), up to
// PackageWait (the setting package-wait-minutes), and pins the version that event carries; the builder never derives a
// pin (no release number of the module that holds the package). Past the bound the build fails, naming the package, and
// nothing is built.
// DefaultPackageWait is the bound of a build's wait for a package's publish, its setting package-wait-minutes' default.
const DefaultPackageWait = 10 * time.Minute
func (b *Builder) packageWait() time.Duration {
if b.PackageWait > 0 {
return b.PackageWait
}
return DefaultPackageWait
}
// PackagePublished takes the Go package registry seat's published of one package path in (its name and the version the
// event carries, H-A-192): the newest published in the version's series is kept, and a build waiting for the package
// wakes. A version that is no MAJOR.MINOR.PATCH (a group's tagged version) pins nothing.
func (b *Builder) PackagePublished(path, version string) {
v := strings.TrimPrefix(version, "v")
series, ok := seriesOfVersion(v)
if path == "" || !ok {
return
}
b.init()
b.mu.Lock()
in := b.publishedPins[path]
if in == nil {
in = map[string]string{}
b.publishedPins[path] = in
}
if old := in[series]; old == "" || compareVersions(v, old) > 0 {
in[series] = v
}
b.mu.Unlock()
b.pictureMoved()
}
// publishedPin is the pin the registry seat's published gave the package path in the series, as Go names it; "" for none.
func (b *Builder) publishedPin(path, series string) string {
b.mu.Lock()
v := b.publishedPins[path][series]
b.mu.Unlock()
if v == "" {
return ""
}
return "v" + v
}
// awaitPublished fills each package of packages with no pin from the published of its path within the series the manifest
// with this digest accepts, waiting for one (H-A-192, H-A-165). It answers the packages with every pin, or an outcome and
// its words: failed past the bound naming the package, could not run when the manifest cannot be read or the builder stops.
func (b *Builder) awaitPublished(ctx context.Context, module, fp, digest string, packages map[string]string) (map[string]string, string, string) {
var unpinned []string
for p, v := range packages {
if v == "" {
unpinned = append(unpinned, p)
}
}
if len(unpinned) == 0 {
return packages, "", ""
}
slices.Sort(unpinned)
if b.Manifests == nil {
return nil, meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the series %s accepts of %s cannot be read: no manifests", module, unpinned[0])
}
man, err := b.Manifests.Manifest(ctx, digest)
if err != nil {
return nil, meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the series %s accepts of %s cannot be read: its manifest %s: %v", module, unpinned[0], digest, err)
}
// What the registry seat holds is read once before the wait (H-A-194): the publishes heard are a cache of it.
for _, p := range unpinned {
if b.PublishedVersions == nil || b.publishedPin(p, man.AcceptedSeries(p)) != "" {
continue
}
vs, err := b.PublishedVersions(ctx, p)
if err != nil {
b.logf("%s at %s: the versions the registry seat holds of %s cannot be read, so the build waits for its publish: %v", module, fp, p, err)
continue
}
for _, v := range vs {
b.PackagePublished(p, v)
}
}
bound := b.packageWait()
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
defer stop()
told := false
for {
b.mu.Lock()
moved := b.moved
b.mu.Unlock()
pins := map[string]string{}
missing, missingSeries := "", ""
for _, p := range unpinned {
series := man.AcceptedSeries(p)
pins[p] = b.publishedPin(p, series)
if pins[p] == "" && missing == "" {
missing, missingSeries = p, series
}
}
if missing == "" {
out := maps.Clone(packages)
maps.Copy(out, pins)
if told {
b.logf("%s at %s: every package it names is published: built with %v", module, fp, pins)
}
return out, "", ""
}
notReleased := fmt.Sprintf("package %s has no released version in the series %s", missing, missingSeries)
if !told {
b.logf("%s at %s: %s yet: the build waits for its publish, at most %s", module, fp, notReleased, bound)
told = true
}
select {
case <-moved:
case <-wait.Done():
if ctx.Err() != nil {
return nil, meshbuilds.StepEndedOutcomeCouldNotRun, notReleased + ": the builder stopped before it was"
}
b.logf("%s at %s: %s within %s: the build fails, nothing is built", module, fp, notReleased, bound)
return nil, meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notReleased, bound)
}
}
}
// seriesOfVersion is a MAJOR.MINOR.PATCH version's series, MAJOR.MINOR; false for any other form.
func seriesOfVersion(v string) (string, bool) {
if _, ok := versionParts(v); !ok {
return "", false
}
major, rest, _ := strings.Cut(v, ".")
minor, _, _ := strings.Cut(rest, ".")
return major + "." + minor, true
}
// versionParts are a MAJOR.MINOR.PATCH version's numbers; false for any other form.
func versionParts(v string) ([3]int, bool) {
var out [3]int
parts := strings.Split(v, ".")
if len(parts) != 3 {
return out, false
}
for i, p := range parts {
n, err := strconv.Atoi(p)
if err != nil || n < 0 {
return out, false
}
out[i] = n
}
return out, true
}
// compareVersions orders two MAJOR.MINOR.PATCH versions numerically.
func compareVersions(a, b string) int {
x, _ := versionParts(a)
y, _ := versionParts(b)
for i := range x {
if x[i] != y[i] {
return x[i] - y[i]
}
}
return 0
}
@@ -103,10 +103,23 @@ type Builder struct {
Now func() time.Time Now func() time.Time
Node string Node string
// Numbers numbers each trunk build at its claim and records what it was built with (design 51 *The versions*); nil // Numbers numbers each trunk build at its claim and records what it was built with (design 51 *The versions*); nil
// numbers nothing. // numbers nothing, as the builder's program leaves it (design 47 H H-A-159: the reservation numbers, G §369).
Numbers Numbers Numbers Numbers
// Reserved reserves the version of a trunk build at its start (design 47 G §369); nil reserves nothing. // Reserved reserves the version of a trunk build at its start (design 47 G §369); nil reserves nothing.
Reserved Reservations Reserved Reservations
// Picture answers the source revision the builder's picture of a watched branch holds, false while it holds none (design 47
// H H-A-155, H-A-141): a step ask naming a newer one waits for it. Nil waits for nothing.
Picture func(branch string) (source int64, held bool)
// RevisionWait bounds a step ask's wait for its revision, its setting revision-wait-minutes (H-A-156); DefaultRevisionWait
// when 0.
RevisionWait time.Duration
// PackageWait bounds a build's wait for the publish of a package its entry names with no pin, its setting
// package-wait-minutes (H-A-165); DefaultPackageWait when 0. Only a registry seat's published heard (PackagePublished)
// gives a pin (H-A-192).
PackageWait time.Duration
// PublishedVersions asks the Go package registry seat's verb versions for one package path, its versions newest first
// (H-A-194): read once before a build waits for that package. Nil reads none.
PublishedVersions func(ctx context.Context, path string) ([]string, error)
// Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine // Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine
// answered when last asked, and counted, every node the assignments names, present or away, a node with no statement // answered when last asked, and counted, every node the assignments names, present or away, a node with no statement
// stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in // stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in
@@ -149,6 +162,11 @@ type Builder struct {
// builds are the ends that withdraw the asks of a key's build in flight, by build key (ADR 0664 §4): a newer fingerprint of // builds are the ends that withdraw the asks of a key's build in flight, by build key (ADR 0664 §4): a newer fingerprint of
// the module takes its older one's build out, and the asker drops that build's asks. // the module takes its older one's build out, and the asker drops that build's asks.
builds map[string]context.CancelCauseFunc builds map[string]context.CancelCauseFunc
// moved is closed and made again at each entry the picture takes in or drops: the step asks waiting for a revision wake.
moved chan struct{}
// publishedPins are the newest version a registry seat said published, by package path and series (PackagePublished):
// a package with no pin waits for one (H-A-192).
publishedPins map[string]map[string]string
} }
type planned struct { type planned struct {
@@ -178,6 +196,8 @@ func (b *Builder) init() {
b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{} b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{}
b.hands, b.events = map[string]*handWatch{}, map[string]bool{} b.hands, b.events = map[string]*handWatch{}, map[string]bool{}
b.builds = map[string]context.CancelCauseFunc{} b.builds = map[string]context.CancelCauseFunc{}
b.moved = make(chan struct{})
b.publishedPins = map[string]map[string]string{}
}) })
} }
@@ -230,6 +250,7 @@ func (b *Builder) EntryGone(branch string, first bool, module string) {
b.mu.Lock() b.mu.Lock()
delete(b.heard, branch+"\x00"+module) delete(b.heard, branch+"\x00"+module)
b.mu.Unlock() b.mu.Unlock()
b.pictureMoved()
if first { if first {
b.Forget(module) b.Forget(module)
} }
@@ -256,6 +277,7 @@ func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.
b.edges[bk] = e b.edges[bk] = e
} }
b.mu.Unlock() b.mu.Unlock()
b.pictureMoved()
if superseded != nil { if superseded != nil {
// A newer fingerprint of the module: the build of the older one in flight is taken out, and its asks are withdrawn // A newer fingerprint of the module: the build of the older one in flight is taken out, and its asks are withdrawn
// from the build seats (ADR 0664 §4). // from the build seats (ADR 0664 §4).
@@ -266,16 +288,19 @@ func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.
// ModuleOf is a catalog entry in the form the builder builds by: its fingerprint, manifest digest, source (repository, // ModuleOf is a catalog entry in the form the builder builds by: its fingerprint, manifest digest, source (repository,
// directory, commit; no source stamp, design 47 H H-A-48), the toolchains its fingerprint was computed with, the bases it is // directory, commit; no source stamp, design 47 H H-A-48), the toolchains its fingerprint was computed with, the bases it is
// built against, its packages and needs, and the named paths of its artifacts. // built against, its packages and needs, and the named paths of its artifacts when every artifact names them.
func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue { func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
m := meshcatalog.ModulesByModuleValue{Repository: e.Repository, Path: e.Path, Commit: e.Commit, Fp2: e.Fp, None2: e.None, m := meshcatalog.ModulesByModuleValue{Repository: e.Repository, Path: e.Path, Commit: e.Commit, Fp2: e.Fp, None2: e.None,
Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs} Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs}
if len(e.Bases) > 0 { if len(e.Bases) > 0 {
m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{Bases: e.Bases} m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{Bases: e.Bases}
} }
if in := e.Inputs; in != nil { if in := e.Inputs; in != nil {
m.Toolchains, m.Problems = in.Toolchains, in.Problems m.Toolchains, m.Problems = in.Toolchains, in.Problems
if len(in.Paths) > 0 { // Only a module whose every artifact names its paths is built from those alone (ADR 0499 clauses 1, 2 and 4): an
// artifact without paths, such as a Go program, keeps its module's directory as its build source, so a module
// with one names no paths and its fetch keeps the directory whole.
if len(in.Paths) > 0 && everyArtifactNamesPaths(e) {
if m.Inputs2 == nil { if m.Inputs2 == nil {
m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{} m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{}
} }
@@ -290,6 +315,20 @@ func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
return m return m
} }
// everyArtifactNamesPaths is whether the entry names artifacts and the named paths of each, by its reference or its name.
func everyArtifactNamesPaths(e meshcatalog.CatalogEntry) bool {
if len(e.Artifacts) == 0 || e.Inputs == nil {
return false
}
for _, a := range e.Artifacts {
_, name, _ := strings.Cut(a.Ref, ":")
if len(e.Inputs.Paths[a.Ref]) == 0 && len(e.Inputs.Paths[name]) == 0 {
return false
}
}
return true
}
// originOf is the catalog revision the entry of a build key came from; a key no entry named is of the first branch, as a build // originOf is the catalog revision the entry of a build key came from; a key no entry named is of the first branch, as a build
// of the trunk's value was before branches were watched. // of the trunk's value was before branches were watched.
func (b *Builder) originOf(key string) Origin { func (b *Builder) originOf(key string) Origin {
@@ -1093,8 +1132,15 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m mes
ask := sdkstep.StepAsk{ID: sdkstep.NewID(), Kind: kind, Language: language, Asker: b.OwnModule, At: time.Now(), Version: l.version, Build: job} ask := sdkstep.StepAsk{ID: sdkstep.NewID(), Kind: kind, Language: language, Asker: b.OwnModule, At: time.Now(), Version: l.version, Build: job}
if language == sdkstep.AskGo && job.Fetch != nil && len(m.Packages) > 0 { if language == sdkstep.AskGo && job.Fetch != nil && len(m.Packages) > 0 {
// The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes // The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes
// them into the workspace, whatever the repository's go.mod names. // them into the workspace, whatever the repository's go.mod names. A package the entry names with no pin waits for
ask.Packages = maps.Clone(m.Packages) // its publish and is built with the version that publish carries; past the bound the lane fails naming it, and no
// ask is put (H-A-192, H-A-165).
pinned, outcome, said := b.awaitPublished(ctx, module, m.Fp2, m.Manifest, m.Packages)
if outcome != "" {
l.err = errors.New(said)
return
}
ask.Packages = maps.Clone(pinned)
for _, fb := range req.Builds { for _, fb := range req.Builds {
delete(ask.Packages, fb.Package) // a package `builds` names is named in neither packages nor package (§465) delete(ask.Packages, fb.Package) // a package `builds` names is named in neither packages nor package (§465)
} }
@@ -332,10 +332,14 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) { func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
r := newRig(t, goBundle) r := newRig(t, goBundle)
r.asks.Answer = nil // no node has pulled it yet r.asks.Answer = nil // no node has pulled it yet
// Issue 1585 (ADR 0573 §2): the ask carries the package versions the fingerprint was computed with. // Issue 1585 (ADR 0573 §2): the ask carries the module's pins; a package the entry keeps with no version (none
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": hex40('5')} // published, design 47 H-A-161) is pinned at the version the registry seat's published carries (H-A-192), never
// asked without it.
r.b.PackagePublished("git.novox.be/novox/mesh-contracts/go/seats/x", "v0.1.7")
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": "v0.1.7"}
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20",
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains), Packages: sdk}) Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains),
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": ""}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -6,6 +6,7 @@ import (
"strings" "strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
) )
@@ -34,6 +35,10 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
if b.StepEntry == nil { if b.StepEntry == nil {
return finish(meshbuilds.StepEndedOutcomeCouldNotRun, "the builder reads no entry of the seat mesh-catalog") return finish(meshbuilds.StepEndedOutcomeCouldNotRun, "the builder reads no entry of the seat mesh-catalog")
} }
// The version is reserved before the build (H-A-155): the ask waits for the builder's picture to hold its revision.
if outcome, said := b.awaitRevision(ctx, run); outcome != "" {
return finish(outcome, said)
}
var failed []string var failed []string
for _, e := range run.Entries { for _, e := range run.Entries {
outcome, said := b.buildEntry(ctx, e) outcome, said := b.buildEntry(ctx, e)
@@ -59,6 +64,12 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
if entry.Module != e.Module || (e.Commit != "" && entry.Commit != e.Commit) { if entry.Module != e.Module || (e.Commit != "" && entry.Commit != e.Commit) {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit) return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit)
} }
// A package the entry names with no pin is built against its publish, waited for (H-A-192, H-A-165).
pinned, outcome, said := b.awaitPublished(ctx, entry.Module, entry.Fp, meshcatalogclient.BareManifestDigest(entry.ManifestDigest), entry.Packages)
if outcome != "" {
return outcome, said
}
entry.Packages = pinned
key, err := builds.BuilderBuildKey(entry.Module, entry.Fp) key, err := builds.BuilderBuildKey(entry.Module, entry.Fp)
if err != nil { if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, err.Error() return meshbuilds.StepEndedOutcomeCouldNotRun, err.Error()
@@ -71,8 +82,8 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
} }
return meshbuilds.StepEndedOutcomeFailed, failedWords return meshbuilds.StepEndedOutcomeFailed, failedWords
} }
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalog.BareManifestDigest(entry.ManifestDigest), Path: entry.Path, m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalogclient.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages} Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
b.consider(ctx, entry.Module, m) b.consider(ctx, entry.Module, m)
select { select {
case log := <-ended: case log := <-ended:
@@ -103,7 +114,7 @@ func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.Ca
if !reserving { if !reserving {
return meshbuilds.StepEndedOutcomePassed, "" return meshbuilds.StepEndedOutcomePassed, ""
} }
plan, planErr, _ := b.planOf(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest)) plan, planErr, _ := b.planOf(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))
switch { switch {
case planErr != nil: case planErr != nil:
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the plan cannot be read to judge its version: %v", entry.Module, entry.Fp, planErr) return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the plan cannot be read to judge its version: %v", entry.Module, entry.Fp, planErr)
@@ -3,13 +3,18 @@ package build
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"reflect"
"slices"
"strings" "strings"
"sync/atomic"
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-builder/internal/versions" "mesh-delivery-builder/internal/versions"
@@ -112,6 +117,263 @@ func TestABuildStepPassesAReservingBuildOnlyWithItsReservedVersion(t *testing.T)
} }
}) })
} }
t.Run("H-A-155: an ask waits for the picture to take its revision in, then builds with its version reserved", aStepAskWaitsForItsRevision)
t.Run("H-A-155: past the bound the ask fails naming its revision, and nothing is built", aStepAskPastItsRevisionBoundFails)
t.Run("H-A-162: a revision held that changes no entry wakes the ask", aRevisionHeldWakesTheAsk)
t.Run("H-A-165: a package with no pin waits for its release and is built with it", aPackageWithNoPinWaitsForItsRelease)
t.Run("H-A-165: past its bound a package with no release fails the step", aPackageWithNoReleaseFailsPastItsBound)
t.Run("H-A-192: a lane waits for an unpinned package's publish and fails past its bound", aLaneWaitsForAnUnpinnedPackage)
t.Run("H-A-194: what the registry seat holds pins the package before any wait", theSeatsVersionsPinBeforeTheWait)
}
// unpinnedRig is a rig whose step entry m names the package x with no version. Its manifest names no `packages`, so it
// accepts x in the default series 0.1. waiting is told when a build starts its wait for a package's publish.
func unpinnedRig(t *testing.T) (*rig, meshcatalog.CatalogEntry, <-chan struct{}) {
var held atomic.Int64
r, _, entry := revisionRig(t, &held, make(chan struct{}, 1))
r.b.Picture = nil // no revision to wait for here
entry.Packages = map[string]string{unpinnedX: "", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}
r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }
waiting := make(chan struct{}, 1)
r.b.Logf = func(format string, args ...any) {
t.Logf(format, args...)
if strings.Contains(fmt.Sprintf(format, args...), "waits for its publish") {
select {
case waiting <- struct{}{}:
default:
}
}
}
return r, entry, waiting
}
const unpinnedX = "git.novox.be/novox/mesh-contracts/go/seats/x"
// awaitWaiting fails the test unless the build started its wait before ended said anything.
func awaitWaiting[T any](t *testing.T, waiting <-chan struct{}, ended <-chan T) {
t.Helper()
select {
case <-waiting:
case end := <-ended:
t.Fatalf("the build ended before the package was published: %+v", end)
case <-time.After(5 * time.Second):
t.Fatal("the build never waited for the package's publish")
}
}
// H-A-192: the pin is the version the registry seat's published carries for the path in the accepted series; a publish in
// another series and the module's release number pin nothing.
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
r, _, waiting := unpinnedRig(t)
r.b.PackagePublished(unpinnedX, "v0.2.9") // another series than the 0.1 the module accepts: no pin
r.b.PackagePublished(unpinnedX, "0.1.4-feat-x.2") // a group's tagged version: no pin
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
awaitWaiting(t, waiting, ended)
if _, ok := r.st.vals[r.key]; ok {
t.Fatal("the build was made before the package was published")
}
r.b.PackagePublished(unpinnedX, "v0.1.3")
var end meshbuilds.StepEnded
select {
case end = <-ended:
case <-time.After(5 * time.Second):
t.Fatal("the publish did not wake the ask")
}
r.b.Wait()
asks := r.asks.Asks()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 ||
!reflect.DeepEqual(asks[0].Packages, buildstep.PackageVersions{unpinnedX: "v0.1.3", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}) {
t.Errorf("end = %+v, asks %+v; want passed, built with x at v0.1.3", end, asks)
}
}
func aPackageWithNoReleaseFailsPastItsBound(t *testing.T) {
r, _, _ := unpinnedRig(t)
r.b.PackageWait = time.Nanosecond
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || built ||
!strings.Contains(end.Said, "package "+unpinnedX+" has no released version in the series 0.1") {
t.Errorf("end = %+v, built %v; want failed naming the package, nothing built", end, built)
}
// A publish heard before gives the pin at once.
r.b.PackagePublished(unpinnedX, "v0.1.2")
if end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}); strings.Contains(end.Said, "no released version") {
t.Errorf("a publish heard did not pin the package: %+v", end)
}
}
// H-A-194: the build reads the registry seat's versions of the path once before it waits: a version in the accepted series
// pins the package with no event heard; one only in another series leaves the build waiting for a publish.
func theSeatsVersionsPinBeforeTheWait(t *testing.T) {
r, _, _ := unpinnedRig(t)
r.b.PackageWait = time.Nanosecond // no wait: the seat's answer alone gives the pin
var read []string
r.b.PublishedVersions = func(_ context.Context, path string) ([]string, error) {
read = append(read, path)
return []string{"0.2.1", "0.1.9", "0.1.8"}, nil // newest first
}
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
asks := r.asks.Asks()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 || asks[0].Packages[unpinnedX] != "v0.1.9" || !slices.Equal(read, []string{unpinnedX}) {
t.Errorf("end = %+v, asks %+v, read %v; want passed, built with x at v0.1.9 read once from the seat", end, asks, read)
}
r2, _, waiting := unpinnedRig(t)
r2.b.PublishedVersions = func(context.Context, string) ([]string, error) { return []string{"0.2.1"}, nil }
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r2.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
awaitWaiting(t, waiting, ended)
r2.b.PackagePublished(unpinnedX, "0.1.10")
if end := <-ended; end.Outcome != meshbuilds.StepEndedOutcomePassed {
t.Errorf("end = %+v, want passed once x was published in its series", end)
}
r2.b.Wait()
}
// H-A-192, H-A-165: the lane of an entry taken in waits the same way, never asking a build without the pin, and fails the
// key past the bound naming the package.
func aLaneWaitsForAnUnpinnedPackage(t *testing.T) {
r, entry, waiting := unpinnedRig(t)
done := make(chan struct{})
go func() {
r.b.Entry(t.Context(), Origin{Branch: "main", Source: 41, First: true}, entry)
r.b.Wait()
close(done)
}()
awaitWaiting(t, waiting, done)
if asks := r.asks.Asks(); len(asks) != 0 {
t.Fatalf("a build was asked without the package's pin: %+v", asks)
}
r.b.PackagePublished(unpinnedX, "0.1.5")
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("the publish did not wake the lane")
}
if asks := r.asks.Asks(); len(asks) == 0 || asks[0].Packages[unpinnedX] != "v0.1.5" {
t.Errorf("asks %+v; want one asked with x at v0.1.5", asks)
}
// Past its bound the key fails naming the package, and nothing is asked.
r2, entry2, _ := unpinnedRig(t)
r2.b.PackageWait = time.Nanosecond
r2.b.Entry(t.Context(), Origin{Branch: "main", Source: 41, First: true}, entry2)
r2.b.Wait()
if asks := r2.asks.Asks(); len(asks) != 0 || !strings.Contains(string(r2.st.vals[r2.key]), "package "+unpinnedX+" has no released version") {
t.Errorf("asks %+v, key %s; want none asked and the key failed naming the package", asks, r2.st.vals[r2.key])
}
}
func aRevisionHeldWakesTheAsk(t *testing.T) {
var held atomic.Int64
held.Store(40)
asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked)
r.b.Entry(context.Background(), Origin{Branch: "main", Source: 40, First: true}, entry) // the entry, unchanged since 40
r.b.Wait()
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.8.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 8, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
select {
case <-asked:
case end := <-ended:
t.Fatalf("the ask ended before the picture held its revision: %+v", end)
case <-time.After(5 * time.Second):
t.Fatal("the ask never read the builder's picture")
}
held.Store(41)
r.b.PictureHeld() // revision 8, from source 41, changes no entry
select {
case end := <-ended:
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 {
t.Errorf("end = %+v, reserved from %+v; want passed with its version reserved", end, res.sources)
}
case <-time.After(5 * time.Second):
t.Fatal("a revision held did not wake the ask")
}
}
// revisionRig is a rig whose builder reserves, its step entry m at fp1, its picture of main holding the source revision held.
func revisionRig(t *testing.T, held *atomic.Int64, asked chan struct{}) (*rig, *fakeReservations, meshcatalog.CatalogEntry) {
r := newRig(t, goBundle)
r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: "0.1", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
res := &fakeReservations{filled: true}
r.b.Reserved = res
entry := meshcatalog.CatalogEntry{Module: "m", Repository: "novox/mesh-catalog", Path: "modules/m", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Fp: fp1,
ManifestDigest: mdigest('a'), Inputs: &meshcatalog.EntryInputs{Toolchains: CatalogToolchains(agreedToolchains)}}
r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }
r.b.Picture = func(branch string) (int64, bool) {
select {
case asked <- struct{}{}:
default:
}
return held.Load(), branch == "main" && held.Load() > 0
}
return r, res, entry
}
func aStepAskWaitsForItsRevision(t *testing.T) {
var held atomic.Int64
held.Store(40)
asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked)
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
select {
case <-asked: // the ask read the picture, which holds source revision 40
case end := <-ended:
t.Fatalf("the ask ended before the picture held its revision: %+v, reserved from %+v", end, res.sources)
case <-time.After(5 * time.Second):
t.Fatal("the ask never read the builder's picture")
}
if _, ok := r.st.vals[r.key]; ok {
t.Fatal("the ask built before the picture held its revision")
}
held.Store(41)
r.b.Entry(context.Background(), Origin{Branch: "main", Source: 41, First: true}, entry) // the picture takes revision 41 in
var end meshbuilds.StepEnded
select {
case end = <-ended:
case <-time.After(5 * time.Second):
t.Fatal("the ask was not woken by the entry the picture took in")
}
r.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 {
t.Errorf("end = %+v, reserved from %+v; want passed with one version reserved from revision 41", end, res.sources)
}
}
func aStepAskPastItsRevisionBoundFails(t *testing.T) {
var held atomic.Int64
held.Store(40)
r, res, _ := revisionRig(t, &held, make(chan struct{}, 1))
r.b.RevisionWait = time.Nanosecond
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, "revision 41 of main not in the builder's picture") || built || len(res.sources) != 0 {
t.Errorf("end = %+v, built %v, reserved %+v; want failed naming revision 41 of main, nothing built", end, built, res.sources)
}
// A step ask with no source (a run on request) does not wait (H-A-156).
end = r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "run.1.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
if strings.Contains(end.Said, "not in the builder's picture") {
t.Errorf("an ask with no source waited: %+v", end)
}
} }
// digestOf is the manifest digest a step's entry names: the rig's manifest, or one no store holds, whose plan cannot be read. // digestOf is the manifest digest a step's entry names: the rig's manifest, or one no store holds, whose plan cannot be read.
@@ -3,16 +3,14 @@ package versions
import ( import (
"context" "context"
"fmt" "fmt"
"strings"
"time" "time"
) )
// The tools' descriptions. // The tools' descriptions.
const ( const (
FloorAgainDescription = "Ask the floor of one name again now, by hand (design 51 *The floor*): the seat's versions, the seat git's " + FloorAgainDescription = "Refused for every name: the builder numbers nothing of its own and asks no floor (design 47 H H-A-159); " +
"git.tags and the indexer's indexer_requirements, as the builder asks them by itself when a silent source says something. " + "a version is numbered by the reservation (design 47 G §369, builder_reserved). Kept until design 47 G §372 removes it with " +
"Answers which sources were silent and whether the floor is now known. A name the builder's versions already hold needs no floor: " + "the state versions."
"refused, saying so. A hand act."
) )
// Acts are the builder's hand acts on its versions, answered by its program (the tools bundle writes them to reasks). // Acts are the builder's hand acts on its versions, answered by its program (the tools bundle writes them to reasks).
@@ -31,24 +29,9 @@ type Acts struct {
DropReserved func(ctx context.Context, module, series, by, why string, at time.Time) (string, error) DropReserved func(ctx context.Context, module, series, by, why string, at time.Time) (string, error)
} }
// FloorAgain is builder_floor_again of name: the floor counts only for a name the versions hold nothing of. // FloorAgain is builder_floor_again of name: refused, since the builder numbers nothing of its own and no claim asks a floor
func (a Acts) FloorAgain(ctx context.Context, name string) (string, error) { // (design 47 H H-A-159; the reservation numbers, G §369). The tool stays until G §372 removes it with the state versions.
entries, err := a.Numberer.Versions.Of(ctx, name) func (a Acts) FloorAgain(_ context.Context, name string) (string, error) {
if err != nil { return "", fmt.Errorf("the floor of %s is not asked: the builder numbers nothing of its own and asks no floor (design 47 H H-A-159); "+
return "", err "a version is numbered by the reservation (G §369, builder_reserved)", name)
}
if len(entries) > 0 {
return "", fmt.Errorf("%s holds %d versions: its floor counts only for its first number, so nothing is asked", name, len(entries))
}
answer := name + " waits on no floor in this builder: the floor is asked at its next claim"
if a.Numberer.Waits != nil {
for _, w := range a.Numberer.Waits.Waiting() {
if w.Name == name {
answer = fmt.Sprintf("the floor of %s is asked again (%s were silent)", name, strings.Join(w.Silent, ", "))
}
}
a.Numberer.Waits.Wake(WakeByHand, "", "", name)
}
a.Reconsider(ctx)
return answer + "; builder_versions shows its version once numbered", nil
} }
@@ -43,6 +43,11 @@ func (n *Numberer) Heard(ctx context.Context, seat, event, id string, o Outcome)
var change func(*Entry) var change func(*Entry)
switch event { switch event {
case HeardPublished: case HeardPublished:
// A published of other bytes than an entry records under its number is no publish of that entry: the reservation
// numbered it and takes it in (design 47 H H-A-159, G §369, A-415; ADR 0605 §4), and the entry stays as it is.
if e, ok, _, err := n.Versions.Recorded(ctx, o.Name, o.Version, o.Digest); err == nil && !ok && e.Digest != "" && o.Digest != "" && e.Digest != o.Digest {
return wake, fmt.Errorf("%s@%s is recorded with digest %s, so %s's published of %s is %w", o.Name, o.Version, e.Digest, seat, o.Digest, ErrNotNumbered)
}
change = func(e *Entry) { change = func(e *Entry) {
e.State, e.Reason = Published, "" e.State, e.Reason = Published, ""
if e.Digest == "" { if e.Digest == "" {
@@ -440,6 +440,13 @@ func TestTheBuildersVersions(t *testing.T) {
if wake, _ := n.Heard(ctx, SeatGoPackages, HeardPublished, "e2", Outcome{Name: "m/x", Digest: "sha256:dd"}); !reflect.DeepEqual(wake, []string{"m/lib"}) { if wake, _ := n.Heard(ctx, SeatGoPackages, HeardPublished, "e2", Outcome{Name: "m/x", Digest: "sha256:dd"}); !reflect.DeepEqual(wake, []string{"m/lib"}) {
t.Fatalf("a published on the Go seat woke %v, want m/lib, whose seat was silent", wake) t.Fatalf("a published on the Go seat woke %v, want m/lib, whose seat was silent", wake)
} }
// Design 47 G §369, A-415; ADR 0605 §4: a published of other bytes under the number is the reservation's, never this entry's.
if _, err := n.Heard(ctx, SeatArtifacts, HeardPublished, "e5", Outcome{Name: "m/program", Version: "0.1.14", Digest: "sha256:new"}); !errors.Is(err, ErrNotNumbered) {
t.Fatalf("a published of other bytes under 0.1.14: %v, want ErrNotNumbered", err)
}
if all, _ := n.Versions.Of(ctx, "m/program"); all[0].State != Numbered || all[0].Digest != "sha256:aa" {
t.Fatalf("after a published of other bytes: %+v, want it numbered with sha256:aa", all[0])
}
for _, h := range []struct{ event, state, reason string }{{HeardRefused, Refused, "bytes differ"}, {HeardPublished, Published, ""}, {HeardRefused, Published, ""}, {HeardRemoved, Removed, "retention"}} { for _, h := range []struct{ event, state, reason string }{{HeardRefused, Refused, "bytes differ"}, {HeardPublished, Published, ""}, {HeardRefused, Published, ""}, {HeardRemoved, Removed, "retention"}} {
if _, err := n.Heard(ctx, SeatArtifacts, h.event, "e3", Outcome{Name: "m/program", Version: "0.1.14", Digest: "sha256:aa", Reason: h.reason}); err != nil { if _, err := n.Heard(ctx, SeatArtifacts, h.event, "e3", Outcome{Name: "m/program", Version: "0.1.14", Digest: "sha256:aa", Reason: h.reason}); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -581,11 +588,13 @@ func TestTheBuildersVersions(t *testing.T) {
} }
_, _ = v.Mark(ctx, "m/p", "0.1.4", func(e *Entry) { e.State = Published }) _, _ = v.Mark(ctx, "m/p", "0.1.4", func(e *Entry) { e.State = Published })
acts := Acts{Numberer: &Numberer{Versions: v, Waits: &FloorWaits{}}, Reconsider: func(context.Context) {}} acts := Acts{Numberer: &Numberer{Versions: v, Waits: &FloorWaits{}}, Reconsider: func(context.Context) {}}
if _, err := acts.FloorAgain(ctx, "m/p"); err == nil { reconsidered := false
t.Error("a name with versions asks its floor again") acts.Reconsider = func(context.Context) { reconsidered = true }
} // H-A-159: the builder numbers nothing of its own, so no claim asks a floor: the tool refuses, saying so, and does nothing.
if ans, err := acts.FloorAgain(ctx, "m/new"); err != nil || ans == "" { for _, name := range []string{"m/p", "m/new"} {
t.Errorf("floor again of a new name: %q %v", ans, err) if ans, err := acts.FloorAgain(ctx, name); err == nil || !strings.Contains(err.Error(), "numbers nothing of its own") || reconsidered {
t.Errorf("floor again of %s: %q %v, looked again %v; want refused naming H-A-159, nothing looked at", name, ans, err, reconsidered)
}
} }
}) })
} }
+13 -7
View File
@@ -186,9 +186,9 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
m.Streams[2].Name != "mesh-delivery-builder-findings" || !reflect.DeepEqual(m.Streams[2].Events, []string{"finding-written.*", "finding-cleared.*", "findings-stated.*"}) { m.Streams[2].Name != "mesh-delivery-builder-findings" || !reflect.DeepEqual(m.Streams[2].Events, []string{"finding-written.*", "finding-cleared.*", "findings-stated.*"}) {
t.Errorf("streams = %+v, want waits with wait-held.** and wait-ended.**, 7 days, 16 MiB, the builds' stream (build.ended.**, 7 days, kept last per subject: design 47 G A-181) and the findings' stream", m.Streams) t.Errorf("streams = %+v, want waits with wait-held.** and wait-ended.**, 7 days, 16 MiB, the builds' stream (build.ended.**, 7 days, kept last per subject: design 47 G A-181) and the findings' stream", m.Streams)
} }
if c, ok := m.Config["toolchain-wait-minutes"]; !ok || c.Default != float64(60) || len(m.Config) != 5 || if c, ok := m.Config["toolchain-wait-minutes"]; !ok || c.Default != float64(60) || len(m.Config) != 7 || m.Config["revision-wait-minutes"].Default != float64(2) || m.Config["package-wait-minutes"].Default != float64(10) ||
m.Config["publish-pause"].Default != float64(30) || m.Config["publish-pause-max"].Default != float64(30) || m.Config["publish-tries"].Default != float64(8) { m.Config["publish-pause"].Default != float64(30) || m.Config["publish-pause-max"].Default != float64(30) || m.Config["publish-tries"].Default != float64(8) {
t.Errorf("config = %+v, want toolchain-wait-minutes of 60, the publish settings 30 s, 30 min and 8 tries (design 47 G §588, §633, A-457a)", m.Config) t.Errorf("config = %+v, want toolchain-wait-minutes of 60, revision-wait-minutes of 2 (H-A-156), package-wait-minutes of 10 (H-A-165), the publish settings 30 s, 30 min and 8 tries (design 47 G §588, §633, A-457a)", m.Config)
} }
// Design 47 G §349: version.numbered is no longer said; old-form-closed on the family version, apart from the builds' ends. // Design 47 G §349: version.numbered is no longer said; old-form-closed on the family version, apart from the builds' ends.
if want := []string{"build.asked-again", "version.old-form-closed"}; !reflect.DeepEqual(m.Emits, want) { if want := []string{"build.asked-again", "version.old-form-closed"}; !reflect.DeepEqual(m.Emits, want) {
@@ -241,7 +241,7 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
t.Errorf("process = %v", p) t.Errorf("process = %v", p)
} }
env, _ := p["env"].(map[string]any) env, _ := p["env"].(map[string]any)
if len(env) != 6 || env["MESH_BUILDER_PUBLISH_TRIES_FILE"] != "${dir:state}/publish-tries" || env["MESH_BROKER_FILE"] != "${dir:state}/broker" || env["MESH_BUILDER_TOOLCHAIN_WAIT_FILE"] != "${dir:state}/toolchain-wait-minutes" { if len(env) != 8 || env["MESH_BUILDER_REVISION_WAIT_FILE"] != "${dir:state}/revision-wait-minutes" || env["MESH_BUILDER_PACKAGE_WAIT_FILE"] != "${dir:state}/package-wait-minutes" || env["MESH_BUILDER_PUBLISH_TRIES_FILE"] != "${dir:state}/publish-tries" || env["MESH_BROKER_FILE"] != "${dir:state}/broker" || env["MESH_BUILDER_TOOLCHAIN_WAIT_FILE"] != "${dir:state}/toolchain-wait-minutes" {
t.Errorf("env = %v, want the broker file and the path of the toolchain wait's file, never the setting itself (design 47 G §672, A-71)", p["env"]) t.Errorf("env = %v, want the broker file and the path of the toolchain wait's file, never the setting itself (design 47 G §672, A-71)", p["env"])
} }
for k, v := range env { for k, v := range env {
@@ -266,10 +266,16 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
if _, bus := p["bus"]; bus { if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential") t.Errorf("the process declares a bus: it runs on its own broker credential")
} }
// Issue 1966's bootstrap: the package artifact builds is out of the manifest until the new builder runs; the follow-up // H-A-186, design 47 G §370: a Go package another module imports is a declared package artifact of its module, so every
// puts it back with its assertion (ADR 0634 §1, ADR 0606 §4: three artifacts, the third the package builds). // release publishes it: builds (ADR 0634 §1, ADR 0606 §4), imported by the build modules and the node-engine, and
if len(m.Build.Artifacts) != 2 { // buildrun, imported by the indexer, the view and the watchman.
t.Fatalf("artifacts = %+v, want the program and the tools", m.Build.Artifacts) if len(m.Build.Artifacts) != 4 {
t.Fatalf("artifacts = %+v, want the program, the tools and the packages builds and buildrun", m.Build.Artifacts)
}
for i, name := range []string{"builds", "buildrun"} {
if a := m.Build.Artifacts[2+i]; a.Name != name || a.Kind != "package" || a.Language != "go" || !reflect.DeepEqual(a.Paths, []string{name}) || a.Series != "0.1" {
t.Errorf("package artifact = %+v, want the Go package %s of series 0.1", a, name)
}
} }
if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-builder-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-builder-tools"}) { if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-builder-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-builder-tools"}) {
t.Errorf("tools artifact = %+v", a) // 1945 t.Errorf("tools artifact = %+v", a) // 1945
+52
View File
@@ -147,6 +147,22 @@
"trusted": false, "trusted": false,
"default": 60 "default": 60
}, },
"revision-wait-minutes": {
"type": "number",
"description": "minutes a build step ask waits for the builder's picture of its branch to hold its source revision before it fails, building nothing (design 47 H H-A-155, H-A-156).",
"minimum": 1,
"maximum": 60,
"trusted": false,
"default": 2
},
"package-wait-minutes": {
"type": "number",
"description": "minutes a build step ask waits for the release of a mesh package its entry names with no pin before it fails, building nothing (design 47 H H-A-165).",
"minimum": 1,
"maximum": 120,
"trusted": false,
"default": 10
},
"publish-pause": { "publish-pause": {
"type": "number", "type": "number",
"description": "seconds before a version's second publish try, doubled per try up to publish-pause-max (design 47 G A-457a).", "description": "seconds before a version's second publish try, doubled per try up to publish-pause-max (design 47 G A-457a).",
@@ -223,6 +239,22 @@
"mode": "0600", "mode": "0600",
"content": "${config:toolchain-wait-minutes}" "content": "${config:toolchain-wait-minutes}"
}, },
{
"id": "revision-wait",
"type": "file",
"path": "${dir:state}/revision-wait-minutes",
"owner": "mesh-delivery-builder",
"mode": "0600",
"content": "${config:revision-wait-minutes}"
},
{
"id": "package-wait",
"type": "file",
"path": "${dir:state}/package-wait-minutes",
"owner": "mesh-delivery-builder",
"mode": "0600",
"content": "${config:package-wait-minutes}"
},
{ {
"id": "publish-pause", "id": "publish-pause",
"type": "file", "type": "file",
@@ -269,6 +301,8 @@
"env": { "env": {
"MESH_BROKER_FILE": "${dir:state}/broker", "MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_BUILDER_TOOLCHAIN_WAIT_FILE": "${dir:state}/toolchain-wait-minutes", "MESH_BUILDER_TOOLCHAIN_WAIT_FILE": "${dir:state}/toolchain-wait-minutes",
"MESH_BUILDER_REVISION_WAIT_FILE": "${dir:state}/revision-wait-minutes",
"MESH_BUILDER_PACKAGE_WAIT_FILE": "${dir:state}/package-wait-minutes",
"MESH_BUILDER_PUBLISH_PAUSE_FILE": "${dir:state}/publish-pause", "MESH_BUILDER_PUBLISH_PAUSE_FILE": "${dir:state}/publish-pause",
"MESH_BUILDER_PUBLISH_PAUSE_MAX_FILE": "${dir:state}/publish-pause-max", "MESH_BUILDER_PUBLISH_PAUSE_MAX_FILE": "${dir:state}/publish-pause-max",
"MESH_BUILDER_PUBLISH_TRIES_FILE": "${dir:state}/publish-tries", "MESH_BUILDER_PUBLISH_TRIES_FILE": "${dir:state}/publish-tries",
@@ -300,6 +334,24 @@
"loads": [ "loads": [
"mesh-delivery-builder-tools" "mesh-delivery-builder-tools"
] ]
},
{
"name": "builds",
"kind": "package",
"language": "go",
"paths": [
"builds"
],
"series": "0.1"
},
{
"name": "buildrun",
"kind": "package",
"language": "go",
"paths": [
"buildrun"
],
"series": "0.1"
} }
] ]
} }
@@ -74,6 +74,8 @@ type Branches struct {
holders atomic.Pointer[seatHolders] holders atomic.Pointer[seatHolders]
// Changed, when not nil, is told of each change to a branch's view (module, its entry, deleted), set before Wire. // Changed, when not nil, is told of each change to a branch's view (module, its entry, deleted), set before Wire.
Changed func(branch, module string, entry json.RawMessage, deleted bool, at time.Time) Changed func(branch, module string, entry json.RawMessage, deleted bool, at time.Time)
// Held, when not nil, is told of each revision a branch's view takes in whole (Catalog.Held; H-A-162), set before Wire.
Held func(branch string, revision seat.CatalogRevision)
// Adopted, when not nil, is told on a goroutine of its own of each branch whose view was adopted, with that view (a part that waits for // Adopted, when not nil, is told on a goroutine of its own of each branch whose view was adopted, with that view (a part that waits for
// its first branch's first view, design 47 G §507), set before Wire. // its first branch's first view, design 47 G §507), set before Wire.
Adopted func(branch string, c *Catalog) Adopted func(branch string, c *Catalog)
@@ -253,6 +255,11 @@ func (b *Branches) reconcile() {
b.Changed(branch, module, entry, deleted, at) b.Changed(branch, module, entry, deleted, at)
} }
} }
c.Held = func(r seat.CatalogRevision) {
if b.Held != nil {
b.Held(branch, r)
}
}
// A catalog-revision also asks again each entry whose ask failed in transport (design 47 H, H-A-60). // A catalog-revision also asks again each entry whose ask failed in transport (design 47 H, H-A-60).
hear := func(ctx context.Context, data []byte) error { hear := func(ctx context.Context, data []byte) error {
err := c.Hear(ctx, data) err := c.Hear(ctx, data)
@@ -59,6 +59,9 @@ type Catalog struct {
waits int64 waits int64
// made is when the view's numbering last started again (H-A-141), the follower's Created; zero is catalogCreated. // made is when the view's numbering last started again (H-A-141), the follower's Created; zero is catalogCreated.
made time.Time made time.Time
// Held, when not nil, is told each revision the view takes in whole, once it holds it, whether or not an entry
// changed (design 47 H H-A-162): a part waiting for a revision wakes on it. Set before Start.
Held func(seat.CatalogRevision)
} }
// created is madeLocked, taking mu. // created is madeLocked, taking mu.
@@ -232,7 +235,9 @@ func (c *Catalog) ask(_ context.Context, after string) (views.Answer, error) {
} }
rev := a.Revision.Revision rev := a.Revision.Revision
holds := false holds := false
if a.Next == "" { whole := a.Next == ""
held := *a.Revision
if whole {
c.held, c.paging = c.paging, nil c.held, c.paging = c.paging, nil
if c.waits <= rev { if c.waits <= rev {
c.waits, holds = 0, true c.waits, holds = 0, true
@@ -242,6 +247,9 @@ func (c *Catalog) ask(_ context.Context, after string) (views.Answer, error) {
if holds { if holds {
c.Follower.Holds(ViewName(rev)) c.Follower.Holds(ViewName(rev))
} }
if whole && c.Held != nil {
c.Held(held)
}
return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Revision: uint64(rev), Created: c.created(), Values: values, Next: a.Next, At: time.Now()}, nil return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Revision: uint64(rev), Created: c.created(), Values: values, Next: a.Next, At: time.Now()}, nil
} }
@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"slices"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -83,6 +84,14 @@ func TestCatalogIsAViewPerWatchedBranch(t *testing.T) {
c.Follower.Wait = time.Nanosecond c.Follower.Wait = time.Nanosecond
c.Follower.Logf = logf c.Follower.Logf = logf
} }
// H-A-162: each revision the view takes in whole is told to Held.
var heldMu sync.Mutex
var held []int64
main.Held = func(r seat.CatalogRevision) {
heldMu.Lock()
defer heldMu.Unlock()
held = append(held, r.Revision)
}
if main.Follower.State == rel.Follower.State || main.Follower.State != "catalog-revisions.main" || rel.Follower.State != views.BranchState(seat.StateCatalogRevisions, "release/1") { if main.Follower.State == rel.Follower.State || main.Follower.State != "catalog-revisions.main" || rel.Follower.State != views.BranchState(seat.StateCatalogRevisions, "release/1") {
t.Fatalf("views are named %q and %q", main.Follower.State, rel.Follower.State) t.Fatalf("views are named %q and %q", main.Follower.State, rel.Follower.State)
} }
@@ -116,17 +125,25 @@ func TestCatalogIsAViewPerWatchedBranch(t *testing.T) {
} }
deadline := time.Now().Add(5 * time.Second) deadline := time.Now().Add(5 * time.Second)
for { for {
if h, _ := main.View(); h.Revision.Revision == 4 { heldMu.Lock()
told := len(held)
heldMu.Unlock()
if h, _ := main.View(); h.Revision.Revision == 4 && told == 2 {
break break
} }
if time.Now().After(deadline) { if time.Now().After(deadline) {
t.Fatal("main never held revision 4") t.Fatal("main never held revision 4, told to Held")
} }
time.Sleep(time.Millisecond) time.Sleep(time.Millisecond)
} }
if n := f.asked("release/1"); n != 1 { if n := f.asked("release/1"); n != 1 {
t.Errorf("release/1 asked %d times after main's revision", n) t.Errorf("release/1 asked %d times after main's revision", n)
} }
heldMu.Lock()
if !slices.Equal(held, []int64{3, 4}) {
t.Errorf("Held was told %v, want 3 then 4", held)
}
heldMu.Unlock()
logMu.Lock() logMu.Lock()
said := false said := false
for _, l := range logged { for _, l := range logged {
@@ -1,4 +1,7 @@
package meshcatalog package meshcatalogclient
// The two helpers below are hand-written, so they live in the seat's client, never under its contract's generated
// root (design 47 H, H-A-200; ADR 0608 §5).
import "strings" import "strings"
@@ -0,0 +1,22 @@
package meshcatalogclient
import "testing"
// An entry's manifest-digest carries sha256: and the module's value and the key of `manifests` the 64 digits bare, one
// digest in two forms (design 47 G §578); the two helpers live in the seat's client, outside its contract's generated
// root (H-A-200).
func TestAManifestDigestIsPrefixedForTheEntryAndBareForTheKey(t *testing.T) {
const bare = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
for _, tc := range []struct{ in, entry, key string }{
{bare, "sha256:" + bare, bare},
{"sha256:" + bare, "sha256:" + bare, bare},
{"", "", ""},
} {
if got := EntryManifestDigest(tc.in); got != tc.entry {
t.Errorf("EntryManifestDigest(%q) = %q, want %q", tc.in, got, tc.entry)
}
if got := BareManifestDigest(tc.in); got != tc.key {
t.Errorf("BareManifestDigest(%q) = %q, want %q", tc.in, got, tc.key)
}
}
}
@@ -26,6 +26,7 @@ const (
VerbRerun = "rerun" VerbRerun = "rerun"
VerbAccept = "accept" VerbAccept = "accept"
VerbCancel = "cancel" VerbCancel = "cancel"
VerbEndAgain = "end-again"
VerbPlace = "place" VerbPlace = "place"
VerbAskAgain = "ask-again" VerbAskAgain = "ask-again"
VerbTestRuns = "test-runs" VerbTestRuns = "test-runs"
@@ -47,7 +48,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call. // Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string { func Verbs() []string {
return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbPlace, VerbAskAgain, VerbTestRuns, VerbTestRunsDrop} return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbEndAgain, VerbPlace, VerbAskAgain, VerbTestRuns, VerbTestRunsDrop}
} }
// Events is every event published on the seat's subject: a new copy each call. // Events is every event published on the seat's subject: a new copy each call.
@@ -275,6 +276,48 @@ func DecodePipelinesByBranchRunRunValue(raw []byte) (PipelinesByBranchRunRunValu
return v, nil return v, nil
} }
// PipelinesByStepsPipelineMoverKey is the key of one value of the state pipelines under the form "steps.<pipeline>.<mover>".
func PipelinesByStepsPipelineMoverKey(pipeline, mover string) string {
return "steps." + pipeline + "." + mover
}
// ParsePipelinesByStepsPipelineMoverKey reads the parts of a key of the form "steps.<pipeline>.<mover>", in the order of the form; the last result is false for a key of another form.
func ParsePipelinesByStepsPipelineMoverKey(key string) (string, string, bool) {
rest := key
if !strings.HasPrefix(rest, "steps.") {
return "", "", false
}
rest = rest[6:]
v1, after1, found1 := strings.Cut(rest, ".")
if !found1 || v1 == "" {
return "", "", false
}
rest = after1
if rest == "" {
return "", "", false
}
v3 := rest
rest = ""
if rest != "" {
return "", "", false
}
return v1, v3, true
}
// DecodePipelinesByStepsPipelineMoverValue reads one value of the state pipelines under the form "steps.<pipeline>.<mover>", refusing a field PipelinesByStepsPipelineMoverValue does not know and any text after the value.
func DecodePipelinesByStepsPipelineMoverValue(raw []byte) (PipelinesByStepsPipelineMoverValue, error) {
var v PipelinesByStepsPipelineMoverValue
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&v); err != nil {
return v, err
}
if dec.More() {
return v, errors.New("text after the value")
}
return v, nil
}
// QueueByQueueNextKey is the key of one value of the state queue under the form "queue.next". // QueueByQueueNextKey is the key of one value of the state queue under the form "queue.next".
func QueueByQueueNextKey() string { return "queue.next" } func QueueByQueueNextKey() string { return "queue.next" }
@@ -522,8 +565,10 @@ type LogEntry struct {
Artifact string `json:"artifact,omitempty"` Artifact string `json:"artifact,omitempty"`
At time.Time `json:"at"` At time.Time `json:"at"`
// the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) // the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470)
Kept []string `json:"kept,omitempty"` Kept []string `json:"kept,omitempty"`
Outcome string `json:"outcome,omitempty"` // the module of the mover whose step the entry is of (H-A-190)
Module string `json:"module,omitempty"`
Outcome string `json:"outcome,omitempty"`
// the output's name in step-outputs // the output's name in step-outputs
Output string `json:"output,omitempty"` Output string `json:"output,omitempty"`
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
@@ -620,6 +665,8 @@ type Pipeline struct {
// how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) // how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325)
EndCount int64 `json:"end-count,omitempty"` EndCount int64 `json:"end-count,omitempty"`
Ended *time.Time `json:"ended,omitempty"` Ended *time.Time `json:"ended,omitempty"`
// how many movers the pipeline has, set by list alone (H-A-191)
MoverCount int64 `json:"mover-count,omitempty"`
// the modules the pipeline moves, its take-in's entries (design 47 H §746) // the modules the pipeline moves, its take-in's entries (design 47 H §746)
Movers []Mover `json:"movers"` Movers []Mover `json:"movers"`
Outcome string `json:"outcome"` Outcome string `json:"outcome"`
@@ -636,11 +683,13 @@ type Pipeline struct {
// the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) // the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9)
Source int64 `json:"source"` Source int64 `json:"source"`
Started *time.Time `json:"started,omitempty"` Started *time.Time `json:"started,omitempty"`
// every mover's steps, each naming its mover, approval last // how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191)
StepCount int64 `json:"step-count,omitempty"`
// every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187)
Steps []StepResult `json:"steps"` Steps []StepResult `json:"steps"`
// the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) // the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107)
Superseded []string `json:"superseded,omitempty"` Superseded []string `json:"superseded,omitempty"`
// the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606) // the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189)
Unlogged []LogEntry `json:"unlogged,omitempty"` Unlogged []LogEntry `json:"unlogged,omitempty"`
// the layout version // the layout version
V int64 `json:"v,omitempty"` V int64 `json:"v,omitempty"`
@@ -1025,6 +1074,8 @@ type StartAnswer struct {
// ShowInput is what show takes. // ShowInput is what show takes.
type ShowInput struct { type ShowInput struct {
// a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187)
Mover string `json:"mover,omitempty"`
// the pipeline's key // the pipeline's key
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
} }
@@ -1195,6 +1246,19 @@ type CancelAnswer struct {
Pipeline Pipeline `json:"pipeline"` Pipeline Pipeline `json:"pipeline"`
} }
// EndAgainInput is what end-again takes.
type EndAgainInput struct {
// the pipeline's key
Pipeline string `json:"pipeline"`
// why, written with the caller the runtime proves in the hand-act log
Why string `json:"why"`
}
// EndAgainAnswer is what end-again answers.
type EndAgainAnswer struct {
Pipeline Pipeline `json:"pipeline"`
}
// PlaceInput is what place takes. // PlaceInput is what place takes.
type PlaceInput struct { type PlaceInput struct {
// the pipeline's key // the pipeline's key
@@ -1300,6 +1364,14 @@ type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline
// PipelinesByBranchRunRunValue is one value of the state pipelines under the key <branch>.run.<run>. // PipelinesByBranchRunRunValue is one value of the state pipelines under the key <branch>.run.<run>.
type PipelinesByBranchRunRunValue = Pipeline type PipelinesByBranchRunRunValue = Pipeline
// PipelinesByStepsPipelineMoverValue is one value of the state pipelines under the key steps.<pipeline>.<mover>.
type PipelinesByStepsPipelineMoverValue struct {
// the mover's steps
Steps []StepResult `json:"steps"`
// the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189)
Unlogged []LogEntry `json:"unlogged,omitempty"`
}
// QueueByQueueNextValue is one value of the state queue under the key queue.next. // QueueByQueueNextValue is one value of the state queue under the key queue.next.
type QueueByQueueNextValue = int64 type QueueByQueueNextValue = int64
@@ -119,6 +119,8 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v TestRunsDropAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil}, decode: func(b []byte) (any, error) { var v TestRunsDropAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "place asks a waiting pipeline ahead of its turn, with why", subject: "", given: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}", wire: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}", {name: "place asks a waiting pipeline ahead of its turn, with why", subject: "", given: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}", wire: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}",
decode: func(b []byte) (any, error) { var v PlaceInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil}, decode: func(b []byte) (any, error) { var v PlaceInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "end-again asks a running pipeline's end judged once more, with why", subject: "", given: "{\"pipeline\":\"main.20\",\"why\":\"a mover waits on its registration and another failed\"}", wire: "{\"pipeline\":\"main.20\",\"why\":\"a mover waits on its registration and another failed\"}",
decode: func(b []byte) (any, error) { var v EndAgainInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
} }
for _, c := range cases { for _, c := range cases {
v, err := c.decode([]byte(c.given)) v, err := c.decode([]byte(c.given))
@@ -336,5 +336,13 @@
"pipeline": "main.3590", "pipeline": "main.3590",
"why": "the operator wants the fix out first" "why": "the operator wants the fix out first"
} }
},
{
"name": "end-again asks a running pipeline's end judged once more, with why",
"of": "verb:end-again.input",
"given": {
"pipeline": "main.20",
"why": "a mover waits on its registration and another failed"
}
} }
] ]
@@ -86,13 +86,17 @@
}, },
{ {
"name": "show", "name": "show",
"description": "One pipeline whole: its steps, its step log and its outputs, with the stores' fill. Reads only.", "description": "One pipeline whole: its record with every mover's steps, or one mover's steps alone when it names the mover, its step log and its outputs, with the stores' fill. Reads only.",
"input": { "input": {
"type": "object", "type": "object",
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key" "description": "the pipeline's key"
},
"mover": {
"type": "string",
"description": "a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187)"
} }
}, },
"required": [ "required": [
@@ -133,7 +137,7 @@
}, },
{ {
"name": "list", "name": "list",
"description": "The pipelines and runs of a module, a branch or a commit, newest first, with the store's fill. Reads only.", "description": "The pipelines and runs of a module, a branch or a commit, newest first, with the store's fill: each pipeline's record with its movers and steps counted, never its steps (show gives those). Reads only.",
"input": { "input": {
"type": "object", "type": "object",
"properties": { "properties": {
@@ -474,6 +478,39 @@
}, },
"class": "act" "class": "act"
}, },
{
"name": "end-again",
"description": "A running pipeline's end judged once more, as at a step's end: it ends when it can, failed as soon as it can no longer pass (design 47 H §746, H-A-163, H-A-164). Written in the hand-act log.",
"input": {
"type": "object",
"properties": {
"pipeline": {
"type": "string",
"description": "the pipeline's key"
},
"why": {
"type": "string",
"description": "why, written with the caller the runtime proves in the hand-act log"
}
},
"required": [
"pipeline",
"why"
]
},
"answer": {
"type": "object",
"properties": {
"pipeline": {
"$ref": "#/defs/Pipeline"
}
},
"required": [
"pipeline"
]
},
"class": "act"
},
{ {
"name": "place", "name": "place",
"description": "A waiting pipeline placed ahead of its turn, its steps asked now; written in the hand-act log (design 47 H §748, H-A-53).", "description": "A waiting pipeline placed ahead of its turn, its steps asked now; written in the hand-act log (design 47 H §748, H-A-53).",
@@ -738,7 +775,7 @@
"name": "pipelines", "name": "pipelines",
"template": "state", "template": "state",
"history": 1, "history": 1,
"description": "one key per pipeline, one per take-in, written by compare-and-set on its absence (design 47 H §746, §747)", "description": "one key per pipeline, one per take-in, written by compare-and-set on its absence (design 47 H §746, §747), and one key per mover of a pipeline holding that mover's steps (H-A-187)",
"key-forms": [ "key-forms": [
{ {
"key": "<branch>.<source>", "key": "<branch>.<source>",
@@ -760,6 +797,32 @@
"$ref": "#/defs/Pipeline" "$ref": "#/defs/Pipeline"
}, },
"description": "a run on request's pipeline, one per run, its movers the named commits' modules; it delivers nothing (H-A-83)" "description": "a run on request's pipeline, one per run, its movers the named commits' modules; it delivers nothing (H-A-83)"
},
{
"key": "steps.<pipeline>.<mover>",
"value": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"$ref": "#/defs/StepResult"
},
"description": "the mover's steps"
},
"unlogged": {
"type": "array",
"items": {
"$ref": "#/defs/LogEntry"
},
"description": "the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189)"
}
},
"required": [
"steps"
]
},
"description": "one mover's steps and unlogged step-log entries in a pipeline, its key the pipeline's and the mover's module escaped as a key's name, written by compare-and-set as the pipeline's record is; the record's own steps stay empty, so a record never grows with the catalog (H-A-187)"
} }
] ]
}, },
@@ -1205,7 +1268,7 @@
"items": { "items": {
"$ref": "#/defs/StepResult" "$ref": "#/defs/StepResult"
}, },
"description": "every mover's steps, each naming its mover, approval last" "description": "every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187)"
}, },
"started": { "started": {
"type": "string", "type": "string",
@@ -1220,7 +1283,15 @@
"items": { "items": {
"$ref": "#/defs/LogEntry" "$ref": "#/defs/LogEntry"
}, },
"description": "the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606)" "description": "the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189)"
},
"mover-count": {
"type": "integer",
"description": "how many movers the pipeline has, set by list alone (H-A-191)"
},
"step-count": {
"type": "integer",
"description": "how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191)"
}, },
"published": { "published": {
"$ref": "#/defs/Published", "$ref": "#/defs/Published",
@@ -1580,6 +1651,10 @@
"pipeline": { "pipeline": {
"type": "string" "type": "string"
}, },
"module": {
"type": "string",
"description": "the module of the mover whose step the entry is of (H-A-190)"
},
"step": { "step": {
"type": "string", "type": "string",
"enum": [ "enum": [
@@ -16,6 +16,7 @@ export declare const verbs: {
readonly rerun: "rerun"; readonly rerun: "rerun";
readonly accept: "accept"; readonly accept: "accept";
readonly cancel: "cancel"; readonly cancel: "cancel";
readonly endAgain: "end-again";
readonly place: "place"; readonly place: "place";
readonly askAgain: "ask-again"; readonly askAgain: "ask-again";
readonly testRuns: "test-runs"; readonly testRuns: "test-runs";
@@ -111,6 +112,12 @@ export declare function pipelinesByBranchRunRunKey(branch: string, run: string):
/** The parts of a key of the form "<branch>.run.<run>", or undefined for a key of another form. */ /** The parts of a key of the form "<branch>.run.<run>", or undefined for a key of another form. */
export declare function parsePipelinesByBranchRunRunKey(_key: string): { branch: string; run: string } | undefined; export declare function parsePipelinesByBranchRunRunKey(_key: string): { branch: string; run: string } | undefined;
/** The key of one value of the state pipelines under the form "steps.<pipeline>.<mover>". */
export declare function pipelinesByStepsPipelineMoverKey(pipeline: string, mover: string): string;
/** The parts of a key of the form "steps.<pipeline>.<mover>", or undefined for a key of another form. */
export declare function parsePipelinesByStepsPipelineMoverKey(_key: string): { pipeline: string; mover: string } | undefined;
/** The key of one value of the state queue under the form "queue.next". */ /** The key of one value of the state queue under the form "queue.next". */
export declare function queueByQueueNextKey(): string; export declare function queueByQueueNextKey(): string;
@@ -185,6 +192,8 @@ export interface LogEntry {
"at": string; "at": string;
/** the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) */ /** the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) */
"kept"?: string[]; "kept"?: string[];
/** the module of the mover whose step the entry is of (H-A-190) */
"module"?: string;
"outcome"?: "running" | "waiting" | "passed" | "failed" | "nothing to run" | "bypassed" | "carried" | "could not run" | "waiting for a build node" | "did not build" | "accepted" | "declined" | "superseded" | "cancelled"; "outcome"?: "running" | "waiting" | "passed" | "failed" | "nothing to run" | "bypassed" | "carried" | "could not run" | "waiting for a build node" | "did not build" | "accepted" | "declined" | "superseded" | "cancelled";
/** the output's name in step-outputs */ /** the output's name in step-outputs */
"output"?: string; "output"?: string;
@@ -247,6 +256,8 @@ export interface Pipeline {
/** how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) */ /** how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) */
"end-count"?: number; "end-count"?: number;
"ended"?: string; "ended"?: string;
/** how many movers the pipeline has, set by list alone (H-A-191) */
"mover-count"?: number;
/** the modules the pipeline moves, its take-in's entries (design 47 H §746) */ /** the modules the pipeline moves, its take-in's entries (design 47 H §746) */
"movers": Mover[]; "movers": Mover[];
"outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled"; "outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled";
@@ -263,11 +274,13 @@ export interface Pipeline {
/** the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) */ /** the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) */
"source": number; "source": number;
"started"?: string; "started"?: string;
/** every mover's steps, each naming its mover, approval last */ /** how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191) */
"step-count"?: number;
/** every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187) */
"steps": StepResult[]; "steps": StepResult[];
/** the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) */ /** the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) */
"superseded"?: string[]; "superseded"?: string[];
/** the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606) */ /** the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189) */
"unlogged"?: LogEntry[]; "unlogged"?: LogEntry[];
/** the layout version */ /** the layout version */
"v"?: number; "v"?: number;
@@ -555,6 +568,8 @@ export interface StartAnswer {
/** is what show takes. */ /** is what show takes. */
export interface ShowInput { export interface ShowInput {
/** a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187) */
"mover"?: string;
/** the pipeline's key */ /** the pipeline's key */
"pipeline": string; "pipeline": string;
} }
@@ -683,6 +698,19 @@ export interface CancelAnswer {
"pipeline": Pipeline; "pipeline": Pipeline;
} }
/** is what end-again takes. */
export interface EndAgainInput {
/** the pipeline's key */
"pipeline": string;
/** why, written with the caller the runtime proves in the hand-act log */
"why": string;
}
/** is what end-again answers. */
export interface EndAgainAnswer {
"pipeline": Pipeline;
}
/** is what place takes. */ /** is what place takes. */
export interface PlaceInput { export interface PlaceInput {
/** the pipeline's key */ /** the pipeline's key */
@@ -781,6 +809,14 @@ export type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline;
/** is one value of the state pipelines under the key <branch>.run.<run>. */ /** is one value of the state pipelines under the key <branch>.run.<run>. */
export type PipelinesByBranchRunRunValue = Pipeline; export type PipelinesByBranchRunRunValue = Pipeline;
/** is one value of the state pipelines under the key steps.<pipeline>.<mover>. */
export interface PipelinesByStepsPipelineMoverValue {
/** the mover's steps */
"steps": StepResult[];
/** the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189) */
"unlogged"?: LogEntry[];
}
/** is one value of the state queue under the key queue.next. */ /** is one value of the state queue under the key queue.next. */
export type QueueByQueueNextValue = number; export type QueueByQueueNextValue = number;
@@ -16,6 +16,7 @@ export const verbs = {
rerun: "rerun", rerun: "rerun",
accept: "accept", accept: "accept",
cancel: "cancel", cancel: "cancel",
endAgain: "end-again",
place: "place", place: "place",
askAgain: "ask-again", askAgain: "ask-again",
testRuns: "test-runs", testRuns: "test-runs",
@@ -172,6 +173,27 @@ export function parsePipelinesByBranchRunRunKey(_key) {
return { branch, run }; return { branch, run };
} }
/** The key of one value of the state pipelines under the form "steps.<pipeline>.<mover>". */
export function pipelinesByStepsPipelineMoverKey(pipeline, mover) {
return "steps." + pipeline + "." + mover;
}
/** The parts of a key of the form "steps.<pipeline>.<mover>", or undefined for a key of another form. */
export function parsePipelinesByStepsPipelineMoverKey(_key) {
let _rest = _key;
if (!_rest.startsWith("steps.")) return undefined;
_rest = _rest.slice(6);
const _at1 = _rest.indexOf(".");
if (_at1 <= 0) return undefined;
const pipeline = _rest.slice(0, _at1);
_rest = _rest.slice(_at1 + 1);
if (_rest === "") return undefined;
const mover = _rest;
_rest = "";
if (_rest !== "") return undefined;
return { pipeline, mover };
}
/** The key of one value of the state queue under the form "queue.next". */ /** The key of one value of the state queue under the form "queue.next". */
export function queueByQueueNextKey() { export function queueByQueueNextKey() {
return "queue.next"; return "queue.next";
@@ -141,7 +141,7 @@ func main() {
tools = append(tools, OwedTools(askState(owedAsks, groups.OwedAsksBucket(served), 25*time.Second, func(v []byte) bool { tools = append(tools, OwedTools(askState(owedAsks, groups.OwedAsksBucket(served), 25*time.Second, func(v []byte) bool {
a, err := groups.DecodeOwedAsk(v) a, err := groups.DecodeOwedAsk(v)
return err == nil && a.Done() return err == nil && a.Done()
}), func() ([]OwedRow, error) { return performedRows(performed) }, func(what, why string) error { }), func(f OwedFilter) ([]OwedRow, error) { return performedRows(performed, f) }, func(what, why string) error {
_, err := recorder.Record(meshhandactsclient.Request{What: what, Why: why, Cause: "owed-dropped"}) _, err := recorder.Record(meshhandactsclient.Request{What: what, Why: why, Cause: "owed-dropped"})
return err return err
}, time.Now)...) }, time.Now)...)
@@ -94,17 +94,55 @@ func TestTheBridgeTools(t *testing.T) {
} }
return json.Marshal(a) return json.Marshal(a)
} }
rows := []OwedRow{{Of: seat.Seat, ID: "a", Outcome: forgemsg.OwedPerformed, At: at}} rows := []OwedRow{
{OwedEnded: forgemsg.OwedEnded{Of: seat.Seat, ID: "a", Outcome: forgemsg.OwedPerformed, At: at}, Repository: "novox/mesh-sdk", Number: 178},
{OwedEnded: forgemsg.OwedEnded{Of: seat.Seat, ID: "b", Outcome: forgemsg.OwedPerformed, At: at}, Repository: "novox/mesh-sdk", Number: 179}}
var filters []OwedFilter
performed := func(f OwedFilter) ([]OwedRow, error) {
filters = append(filters, f)
var out []OwedRow
for _, r := range rows {
if f.holds(r) {
out = append(out, r)
}
}
return out, nil
}
var recorded []string var recorded []string
recordErr := error(nil) recordErr := error(nil)
record := func(what, why string) error { recorded = append(recorded, what+": "+why); return recordErr } record := func(what, why string) error { recorded = append(recorded, what+": "+why); return recordErr }
tools := map[string]func(map[string]any) (any, error){} tools := map[string]func(map[string]any) (any, error){}
for _, tl := range OwedTools(ask, func() ([]OwedRow, error) { return rows, nil }, record, func() time.Time { return at }) { for _, tl := range OwedTools(ask, performed, record, func() time.Time { return at }) {
tools[tl.Name] = tl.Run tools[tl.Name] = tl.Run
} }
out, err := tools[OwedTool](map[string]any{"of": seat.Seat}) // bridge_owed reads one act or one pull request's acts, never the whole state (H-A-181).
if m, _ := out.(map[string]any); err != nil || len(m["performed"].([]OwedRow)) != 1 || len(m["backlog"].([]groups.Backlog)) != 1 { for _, c := range []struct {
t.Errorf("bridge_owed: %v %v", out, err) name string
args map[string]any
ids []string
}{
{"one pull request", map[string]any{"repository": "novox/mesh-sdk", "number": float64(179)}, []string{"b"}},
{"one act", map[string]any{"of": seat.Seat, "id": "a"}, []string{"a"}},
{"the whole state", map[string]any{"of": seat.Seat}, nil},
{"a repository with no number", map[string]any{"repository": "novox/mesh-sdk"}, nil},
} {
n := len(filters)
out, err := tools[OwedTool](c.args)
if c.ids == nil {
if err == nil || len(filters) != n {
t.Errorf("bridge_owed of %s: %v %v, read %d; want refused, nothing read", c.name, out, err, len(filters)-n)
}
continue
}
m, _ := out.(map[string]any)
got, _ := m["performed"].([]OwedRow)
var ids []string
for _, r := range got {
ids = append(ids, r.ID)
}
if err != nil || !reflect.DeepEqual(ids, c.ids) || len(m["backlog"].([]groups.Backlog)) != 1 {
t.Errorf("bridge_owed of %s: %v %v, want %v", c.name, out, err, c.ids)
}
} }
if out, err := tools[OwedAgainTool](map[string]any{"of": seat.Seat, "id": "a"}); err != nil || out.(map[string]any)["outcome"] != forgemsg.OwedPerformed || if out, err := tools[OwedAgainTool](map[string]any{"of": seat.Seat, "id": "a"}); err != nil || out.(map[string]any)["outcome"] != forgemsg.OwedPerformed ||
asked[len(asked)-1].Kind != groups.OwedAgain { asked[len(asked)-1].Kind != groups.OwedAgain {
@@ -21,26 +21,50 @@ const (
OwedDropTool = "bridge_owed_drop" OwedDropTool = "bridge_owed_drop"
) )
// OwedRow is one act the bridge performed, as the state performed keeps it: the outcome said as owed.ended. // OwedRow is one act the bridge performed, as the state performed keeps it: the outcome said as owed.ended, and the pull
type OwedRow = forgemsg.OwedEnded // request the act names (groups.Performed).
type OwedRow struct {
forgemsg.OwedEnded
Repository string `json:"repository,omitempty"`
Number int `json:"number,omitempty"`
}
// performedRows reads every act the state performed holds; a key that is not an act (a group end's bookkeeping) is left. // OwedFilter is what bridge_owed reads of performed: one act (its id, of narrowing it to one key) or one pull request's
func performedRows(s stdio.KeptState) ([]OwedRow, error) { // acts (repository and number), never the whole state (novox/hq design 47 H-A-181).
keys, err := s.Keys() type OwedFilter struct {
if err != nil { Of, ID, Repository string
return nil, fmt.Errorf("the bridge's performed state cannot be listed: %w", err) Number int
}
func (f OwedFilter) holds(r OwedRow) bool {
return (f.Of == "" || r.Of == f.Of) && (f.ID == "" || r.ID == f.ID) &&
(f.Repository == "" || (r.Repository == f.Repository && r.Number == f.Number))
}
// performedRows reads the acts of performed f names, newest first: one key when f names the seat and the id, else each
// key's value held to f. A key that is not an act (a group end's bookkeeping) is left.
func performedRows(s stdio.KeptState, f OwedFilter) ([]OwedRow, error) {
var keys []string
if f.Of != "" && f.ID != "" {
keys = []string{groups.PerformedKey(f.Of, f.ID)}
} else {
all, err := s.Keys()
if err != nil {
return nil, fmt.Errorf("the bridge's performed state cannot be listed: %w", err)
}
keys = all
} }
var rows []OwedRow rows := []OwedRow{}
for _, k := range keys { for _, k := range keys {
e, err := s.Get(k) e, err := s.Get(k)
if err != nil || e == nil { if err != nil || e == nil {
continue continue
} }
var p struct { var p groups.Performed
Ended OwedRow `json:"ended"`
}
if json.Unmarshal(e.Value, &p) == nil && p.Ended.Of != "" { if json.Unmarshal(e.Value, &p) == nil && p.Ended.Of != "" {
rows = append(rows, p.Ended) if r := (OwedRow{OwedEnded: p.Ended, Repository: p.Repository, Number: p.Number}); f.holds(r) {
rows = append(rows, r)
}
} }
} }
sort.Slice(rows, func(i, j int) bool { return rows[i].At.After(rows[j].At) }) sort.Slice(rows, func(i, j int) bool { return rows[i].At.After(rows[j].At) })
@@ -50,7 +74,7 @@ func performedRows(s stdio.KeptState) ([]OwedRow, error) {
// OwedTools are bridge_owed, bridge_owed_again and bridge_owed_drop: ask writes an owed-asks value and waits for the // OwedTools are bridge_owed, bridge_owed_again and bridge_owed_drop: ask writes an owed-asks value and waits for the
// bridge's answer; performed reads the state performed; record writes a hand act in the seat mesh-hand-acts' log before // bridge's answer; performed reads the state performed; record writes a hand act in the seat mesh-hand-acts' log before
// a drop is asked, so a drop is never made without its record. // a drop is asked, so a drop is never made without its record.
func OwedTools(ask func(key string, value []byte) ([]byte, error), performed func() ([]OwedRow, error), record func(what, why string) error, now func() time.Time) []stdio.Tool { func OwedTools(ask func(key string, value []byte) ([]byte, error), performed func(OwedFilter) ([]OwedRow, error), record func(what, why string) error, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) } str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
act := map[string]any{ act := map[string]any{
"of": map[string]any{"type": "string", "description": "the seat that said the act: mesh-delivery (the gate)"}, "of": map[string]any{"type": "string", "description": "the seat that said the act: mesh-delivery (the gate)"},
@@ -66,6 +90,12 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
} }
return groups.DecodeOwedAsk(v) return groups.DecodeOwedAsk(v)
} }
read := map[string]any{
"repository": map[string]any{"type": "string", "description": "the pull request's repository, as owner/name; with number"},
"number": map[string]any{"type": "integer", "description": "the pull request's number; with repository"}}
for k, v := range act {
read[k] = v
}
withWhy := map[string]any{"why": map[string]any{"type": "string", "description": "why it is dropped, in plain words"}} withWhy := map[string]any{"why": map[string]any{"type": "string", "description": "why it is dropped, in plain words"}}
for k, v := range act { for k, v := range act {
withWhy[k] = v withWhy[k] = v
@@ -74,22 +104,24 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
{Name: OwedTool, {Name: OwedTool,
Description: "What the forge bridge performed of what the gate and the checker owe the forge (design 47): each act it said as " + Description: "What the forge bridge performed of what the gate and the checker owe the forge (design 47): each act it said as " +
"owed.ended, newest first, with its seat, id, pull request, outcome (performed, refused, could not be performed, " + "owed.ended, newest first, with its seat, id, pull request, outcome (performed, refused, could not be performed, " +
"undecodable, dropped), the forge's words and when, from its state performed (kept 7 days); and the backlog of its " + "undecodable, dropped), the forge's words and when, from its state performed (kept 7 days), of one act (id, of " +
"narrowing it) or of one pull request (repository and number), never the whole state; and the backlog of its " +
"durable on each seat's stream owed: the last event applied, the events waiting and held, the age of the oldest. " + "durable on each seat's stream owed: the last event applied, the events waiting and held, the age of the oldest. " +
"of or id narrow it. Read-only.", "Read-only.",
Input: map[string]any{"type": "object", "properties": act}, Input: map[string]any{"type": "object", "properties": read},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
rows, err := performed() f := OwedFilter{Of: str(args, "of"), ID: str(args, "id"), Repository: str(args, "repository")}
if n, ok := args["number"].(float64); ok {
f.Number = int(n)
}
if f.ID == "" && (f.Repository == "" || f.Number <= 0) {
return nil, fmt.Errorf("bridge_owed reads one act (id) or one pull request's acts (repository and number), never the whole state")
}
rows, err := performed(f)
if err != nil { if err != nil {
return nil, err return nil, err
} }
kept := []OwedRow{} out := map[string]any{"performed": rows}
for _, r := range rows {
if (str(args, "of") == "" || r.Of == str(args, "of")) && (str(args, "id") == "" || r.ID == str(args, "id")) {
kept = append(kept, r)
}
}
out := map[string]any{"performed": kept}
if a, err := asked(groups.OwedBacklog, args); err != nil { if a, err := asked(groups.OwedBacklog, args); err != nil {
out["backlog-error"] = err.Error() out["backlog-error"] = err.Error()
} else { } else {
@@ -100,8 +132,8 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
{Name: OwedAgainTool, {Name: OwedAgainTool,
Description: "Perform one act owed to the forge again now, by hand, whatever its last outcome (a refusal the forge no longer " + Description: "Perform one act owed to the forge again now, by hand, whatever its last outcome (a refusal the forge no longer " +
"makes, an act that could not be performed): the bridge calls the seat git's verb of the act under its id (safe to " + "makes, an act that could not be performed): the bridge calls the seat git's verb of the act under its id (safe to " +
"repeat) and says its outcome as owed.ended. The act is the one the bridge kept, or, for the gate's, the seat " + "repeat) and says its outcome as owed.ended. The act is read from the stream owed of the seat that said it, on the " +
"mesh-delivery's groups. Answers the outcome.", "subject of the pull request the bridge's state performed names (kept 7 days there). Answers the outcome.",
Input: map[string]any{"type": "object", "required": []string{"of", "id"}, "properties": act}, Input: map[string]any{"type": "object", "required": []string{"of", "id"}, "properties": act},
Run: func(args map[string]any) (any, error) { Run: func(args map[string]any) (any, error) {
a, err := asked(groups.OwedAgain, args) a, err := asked(groups.OwedAgain, args)
@@ -0,0 +1,89 @@
package main
import (
"context"
"fmt"
"strconv"
"github.com/nats-io/nats.go/jetstream"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// The act bridge_owed_again performs again is read from the stream owed of the seat that said it, on the subject of the
// pull request performed names, the stream being the act's one copy (novox/hq design 47 G §162, §163, H-A-181).
// actReader reads the act the seat of said under id from its stream owed, on the subject of the pull request
// (repository and number) the act names, for decode to say each message's id; found false when the stream no longer
// holds it.
type actReader interface {
act(ctx context.Context, of, repository string, number int, id string, idOf func([]byte) string) (data []byte, found bool, err error)
}
// actOf is the act of to perform again, read from its seat's stream owed on the pull request p names.
func (o *owing) actOf(ctx context.Context, of string, p Performed, id string) ([]byte, error) {
read, err := readerOf(of)
if err != nil {
return nil, err
}
// A value of the form before H-A-182 names no pull request: every act the bridge decodes names its repository (a
// status may name no number, said on `.0` as the gate says it), so one with none is the old form, never read again.
if p.Repository == "" {
return nil, fmt.Errorf("performed names no pull request for the act %s of %s: kept in its old form, it expires with the state's TTL", id, of)
}
if o.acts == nil {
return nil, fmt.Errorf("the bridge reads no stream owed, so the act %s of %s cannot be performed again", id, of)
}
data, found, err := o.acts.act(ctx, of, p.Repository, p.Number, id, func(m []byte) string { c, _ := read(m); return c.ID })
if err != nil {
return nil, fmt.Errorf("the act %s of %s cannot be read from %s's stream owed: %w", id, of, of, err)
}
if !found {
return nil, fmt.Errorf("%s's stream owed no longer holds the act %s of %s#%d (kept 7 days)", of, id, p.Repository, p.Number)
}
return data, nil
}
// streamActs reads an act from the gate's stream owed (seat mesh-delivery's StreamOwed) by an ordered consumer filtered
// on the pull request's subject, as the gate says it (SubjectOwed with the repository escaped): every message the
// stream keeps there, the newest of the id kept.
type streamActs struct{ js jetstream.JetStream }
func (s streamActs) act(ctx context.Context, of, repository string, number int, id string, idOf func([]byte) string) ([]byte, bool, error) {
if of != seat.Seat {
return nil, false, fmt.Errorf("the seat %q keeps no stream owed the bridge reads", of)
}
subject := seat.SubjectOwed(keys.EscapeName(repository), strconv.Itoa(number))
st, err := s.js.Stream(ctx, seat.StreamOwed)
if err != nil {
return nil, false, err
}
info, err := st.Info(ctx, jetstream.WithSubjectFilter(subject))
if err != nil {
return nil, false, err
}
held := info.State.Subjects[subject]
if held == 0 {
return nil, false, nil
}
c, err := st.OrderedConsumer(ctx, jetstream.OrderedConsumerConfig{FilterSubjects: []string{subject}})
if err != nil {
return nil, false, err
}
var found []byte
for range held {
m, err := c.Next(jetstream.FetchContext(ctx))
if err != nil {
return nil, false, err
}
if idOf(m.Data()) == id {
found = m.Data()
}
if md, err := m.Metadata(); err == nil && md.NumPending == 0 {
break
}
}
return found, found != nil, nil
}
@@ -5,6 +5,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"log" "log"
"strings" "strings"
"testing" "testing"
@@ -38,6 +39,24 @@ func (m memPerformed) Get(_ context.Context, key string) ([]byte, bool, error) {
} }
func (m memPerformed) Put(_ context.Context, key string, v []byte) error { m[key] = v; return nil } func (m memPerformed) Put(_ context.Context, key string, v []byte) error { m[key] = v; return nil }
// memActs is a seat's stream owed by pull request subject (`<repository>#<number>`); asked records what was read.
type memActs struct {
held map[string][][]byte
asked []string
}
func (m *memActs) act(_ context.Context, of, repository string, number int, id string, idOf func([]byte) string) ([]byte, bool, error) {
pr := fmt.Sprintf("%s %s#%d", of, repository, number)
m.asked = append(m.asked, pr)
var found []byte
for _, d := range m.held[pr] {
if idOf(d) == id {
found = d
}
}
return found, found != nil, nil
}
// seatFake answers each seat verb from answers, by its key; one absent is not answered (an error, as a bound passed). // seatFake answers each seat verb from answers, by its key; one absent is not answered (an error, as a bound passed).
type seatFake struct { type seatFake struct {
calls []string calls []string
@@ -109,9 +128,12 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
if v, ok := kept[performedKey(c.of, e.ID)]; ok != c.kept { if v, ok := kept[performedKey(c.of, e.ID)]; ok != c.kept {
t.Errorf("%s: kept %v (%s), want %v", c.name, ok, v, c.kept) t.Errorf("%s: kept %v (%s), want %v", c.name, ok, v, c.kept)
} else if ok { } else if ok {
// The outcome and when, and the pull request the act names, nothing more (§163, H-A-181, H-A-182).
var p Performed var p Performed
if err := json.Unmarshal(v, &p); err != nil || p.Ended.Outcome != c.outcome || !json.Valid(p.Owed) { var fields map[string]json.RawMessage
t.Errorf("%s: kept %s, want the outcome and the owed event", c.name, v) if err := json.Unmarshal(v, &p); err != nil || p.Ended.Outcome != c.outcome || p.Ended.At.IsZero() ||
p.Repository != "novox/mesh-sdk" || p.Number != 178 || json.Unmarshal(v, &fields) != nil || len(fields) != 3 {
t.Errorf("%s: kept %s, want the outcome, when, and the pull request novox/mesh-sdk#178, nothing more", c.name, v)
} }
} }
} }
@@ -142,6 +164,9 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
t.Run("asked again, dropped, and its backlog read by hand", func(t *testing.T) { t.Run("asked again, dropped, and its backlog read by hand", func(t *testing.T) {
f := &seatFake{answers: statusOK} f := &seatFake{answers: statusOK}
o, pub, kept := newOwing(f) o, pub, kept := newOwing(f)
// The act is read again from the gate's stream owed on its pull request's subject, never from performed (H-A-181).
acts := &memActs{held: map[string][][]byte{"mesh-delivery novox/mesh-sdk#178": {[]byte(`{`), gate("owed is said, a status")}}}
o.acts = acts
status := "status.novox/mesh-sdk#178.aaaaaaaaaaaa.mesh/merge-gate" status := "status.novox/mesh-sdk#178.aaaaaaaaaaaa.mesh/merge-gate"
if _, err := o.heard(context.Background(), seat.Seat, gate("owed is said, a status")); err != nil { if _, err := o.heard(context.Background(), seat.Seat, gate("owed is said, a status")); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -153,11 +178,20 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
calls, said int calls, said int
outcome, answers string outcome, answers string
}{ }{
{"again, from performed", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 2, forgemsg.OwedPerformed, "performed"}, {"again, from the gate's stream owed", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 2, forgemsg.OwedPerformed, "performed"},
{"dropped", groups.OwedAsk{Kind: groups.OwedDrop, Of: seat.Seat, ID: status, Why: "set by hand", By: "lead@shanks"}, 2, 3, forgemsg.OwedDropped, "set by hand"}, {"dropped", groups.OwedAsk{Kind: groups.OwedDrop, Of: seat.Seat, ID: status, Why: "set by hand", By: "lead@shanks"}, 2, 3, forgemsg.OwedDropped, "set by hand"},
{"again, an act nobody holds", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: "x"}, 2, 3, "", "holds no owed act"}, {"again, an act nobody holds", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: "x"}, 2, 3, "", "holds no owed act"},
{"the backlog", groups.OwedAsk{Kind: groups.OwedBacklog}, 2, 3, "", "backlog"}, {"the backlog", groups.OwedAsk{Kind: groups.OwedBacklog}, 2, 3, "", "backlog"},
{"again, the stream no longer holding it", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 3, "", "no longer holds"},
{"again, a value of the old form", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: "old"}, 2, 3, "", "names no pull request"},
} { } {
switch c.name {
case "again, the stream no longer holding it":
acts.held = nil
case "again, a value of the old form":
// Kept before H-A-182: the outcome and the whole act, no pull request; the stream is not read for it.
kept[performedKey(seat.Seat, "old")] = []byte(`{"ended":{"v":1,"of":"mesh-delivery","id":"old","outcome":"performed","at":"2026-11-09T21:00:00Z"},"owed":{"id":"old"}}`)
}
a := o.answer(context.Background(), c.ask, backlog) a := o.answer(context.Background(), c.ask, backlog)
if !a.Done() || a.Outcome != c.outcome || !strings.Contains(a.Answer, c.answers) || len(f.calls) != c.calls || len(pub.data) != c.said { if !a.Done() || a.Outcome != c.outcome || !strings.Contains(a.Answer, c.answers) || len(f.calls) != c.calls || len(pub.data) != c.said {
t.Errorf("%s: %+v, called %v, said %d; want %q naming %q, %d calls, %d said", c.name, a, f.calls, len(pub.data), c.outcome, c.answers, c.calls, c.said) t.Errorf("%s: %+v, called %v, said %d; want %q naming %q, %d calls, %d said", c.name, a, f.calls, len(pub.data), c.outcome, c.answers, c.calls, c.said)
@@ -167,8 +201,12 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
} }
} }
var p Performed var p Performed
if err := json.Unmarshal(kept[performedKey(seat.Seat, status)], &p); err != nil || p.Ended.Outcome != forgemsg.OwedDropped { if err := json.Unmarshal(kept[performedKey(seat.Seat, status)], &p); err != nil || p.Ended.Outcome != forgemsg.OwedDropped ||
t.Errorf("performed after the drop: %+v %v", p, err) p.Repository != "novox/mesh-sdk" || p.Number != 178 {
t.Errorf("performed after the drop: %+v %v; want dropped, its pull request kept", p, err)
}
if len(acts.asked) != 2 || acts.asked[0] != "mesh-delivery novox/mesh-sdk#178" {
t.Errorf("the stream owed was read on %v, want twice on the pull request performed names", acts.asked)
} }
}) })
} }
@@ -22,6 +22,7 @@ import (
git "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git" git "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-forge-bridge/internal/gitfixture" "mesh-delivery-forge-bridge/internal/gitfixture"
@@ -33,7 +34,8 @@ import (
// no others, so a permission the bridge lacks fails a test here and not on the mesh (issue 1279). // no others, so a permission the bridge lacks fails a test here and not on the mesh (issue 1279).
// //
// From module.json: the registrations and the gate's source-moved heard through its durable on EVENTS; branch-moved and // From module.json: the registrations and the gate's source-moved heard through its durable on EVENTS; branch-moved and
// what the gate owes heard through its durable on each seat's own stream (SEAT_<seat>_<event>); the seat git's verbs, the // what the gate owes heard through its durable on each seat's own stream (SEAT_<seat>_<event>), and each seat's kept
// event also read through an ordered consumer of its own (mesh-grants heardGrants, A-233; bridge_owed_again, H-A-181); the seat git's verbs, the
// seat mesh-catalog's registrations, the seat mesh-delivery's source, the seat mesh-broker's show and the seat // seat mesh-catalog's registrations, the seat mesh-delivery's source, the seat mesh-broker's show and the seat
// mesh-hand-acts' hand-act called on mesh.seat.<seat>.call.<verb>.<token> (or its plain fallback); its own states heads // mesh-hand-acts' hand-act called on mesh.seat.<seat>.call.<verb>.<token> (or its plain fallback); its own states heads
// and owed-asks, and performed under the name its binding gives; its stream forge. `**` is written `>` on the wire. The // and owed-asks, and performed under the name its binding gives; its stream forge. `**` is written `>` on the wire. The
@@ -82,6 +84,11 @@ accounts {
"$JS.ACK.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge.>" "$JS.ACK.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge.>"
"$JS.API.CONSUMER.INFO.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge" "$JS.API.CONSUMER.INFO.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge"
"$JS.API.CONSUMER.MSG.NEXT.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge" "$JS.API.CONSUMER.MSG.NEXT.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge"
"$JS.API.CONSUMER.CREATE.SEAT_mesh-delivery_owed"
"$JS.API.CONSUMER.CREATE.SEAT_mesh-delivery_owed.>"
"$JS.API.CONSUMER.MSG.NEXT.SEAT_mesh-delivery_owed.*"
"$JS.API.CONSUMER.INFO.SEAT_mesh-delivery_owed.*"
"$JS.API.CONSUMER.DELETE.SEAT_mesh-delivery_owed.*"
"mesh.seat.mesh-broker.call.show.n1~mesh-delivery-forge-bridge" "mesh.seat.mesh-broker.call.show.n1~mesh-delivery-forge-bridge"
"mesh.seat.mesh-broker.tool.show" "mesh.seat.mesh-broker.tool.show"
"mesh.seat.mesh-catalog.call.registrations.n1~mesh-delivery-forge-bridge" "mesh.seat.mesh-catalog.call.registrations.n1~mesh-delivery-forge-bridge"
@@ -518,7 +525,9 @@ func TestBridgePerformsWhatIsOwedOnce(t *testing.T) {
} }
} }
gateOwed := gitfixture.Seat(t, seat.Seat, "events", "owed is said, a status", "event:owed")[0] gateOwed := gitfixture.Seat(t, seat.Seat, "events", "owed is said, a status", "event:owed")[0]
subject := seat.SubjectOwed("novox~mesh-sdk", "178") // On the pull request's subject as the gate says it (its repository escaped as a key part), where bridge_owed_again
// reads the act again (H-A-181).
subject := seat.SubjectOwed(keys.EscapeName("novox/mesh-sdk"), "178")
for range 2 { for range 2 {
if _, err := b.js.Publish(b.ctx, subject, gateOwed); err != nil { if _, err := b.js.Publish(b.ctx, subject, gateOwed); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -597,12 +606,22 @@ func TestBridgePerformsWhatIsOwedOnce(t *testing.T) {
if a, err := groupkeys.DecodeOwedAsk(e.Value()); err == nil && a.Done() { if a, err := groupkeys.DecodeOwedAsk(e.Value()); err == nil && a.Done() {
answered = true answered = true
if a.Outcome != forgemsg.OwedPerformed || count("git.status") != 3 { if a.Outcome != forgemsg.OwedPerformed || count("git.status") != 3 {
t.Errorf("bridge_owed_again: %+v, status asked %d times; want performed again", a, count("git.status")) t.Errorf("bridge_owed_again: %+v, status asked %d times; want performed again, read from the gate's stream owed", a, count("git.status"))
} }
} }
case <-bound: case <-bound:
t.Fatal("bridge_owed_again was not answered within 20 s") t.Fatal("bridge_owed_again was not answered within 20 s")
} }
} }
// performed keeps the outcome, when and the pull request, never the act (§163, H-A-181, H-A-182).
kept, err := b.js.KeyValue(b.ctx, boundPerformed)
if err != nil {
t.Fatal(err)
}
if e, err := kept.Get(b.ctx, performedKey(seat.Seat, g.ID)); err != nil {
t.Errorf("performed of %s: %v", g.ID, err)
} else if fields := (map[string]json.RawMessage{}); json.Unmarshal(e.Value(), &fields) != nil || len(fields) != 3 || fields["owed"] != nil {
t.Errorf("performed of %s keeps %s, want the outcome, when and the pull request alone", g.ID, e.Value())
}
refusedRead(t, b, "mesh-delivery-gate_owed") refusedRead(t, b, "mesh-delivery-gate_owed")
} }
@@ -108,7 +108,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
} }
performedKV := &boundBucket{binding: binding, local: performedLocal, js: js} performedKV := &boundBucket{binding: binding, local: performedLocal, js: js}
// What the gate owes the forge, performed through the seat git (design 47 *What the gate owes the forge*). // What the gate owes the forge, performed through the seat git (design 47 *What the gate owes the forge*).
owe := &owing{call: call, pub: js, performed: kvPerformed{performedKV}, waits: binding.Waiting, now: time.Now, logger: logger} owe := &owing{call: call, pub: js, performed: kvPerformed{performedKV}, acts: streamActs{js: js}, waits: binding.Waiting, now: time.Now, logger: logger}
keptCall := keptCallOf(ctx, call) keptCall := keptCallOf(ctx, call)
reader := &views.ProcessReader{Module: forgeBridgeModule(), Call: keptCall, Logf: logger.Printf, Hub: hub} reader := &views.ProcessReader{Module: forgeBridgeModule(), Call: keptCall, Logf: logger.Printf, Hub: hub}
src := sourcebranches.New(sourcebranches.Call(keptCall)) src := sourcebranches.New(sourcebranches.Call(keptCall))
@@ -2,7 +2,6 @@ package main
import ( import (
"context" "context"
"encoding/base64"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
@@ -22,22 +21,14 @@ import (
) )
// The state performed (manifest `requires`, the template state, opened by its binding: bound.go) keeps one key per owed act it // The state performed (manifest `requires`, the template state, opened by its binding: bound.go) keeps one key per owed act it
// performed, the act's outcome and when, with the owed event as heard (novox/hq design 47 *What the gate owes the // performed, in the form groups.Performed says (novox/hq design 47 G §163, H-A-181, H-A-182). It is a saving and a read,
// forge*). It is a saving and a read, never what keeps an act from happening twice: every act is safe to repeat at the // never what keeps an act from happening twice: every act is safe to repeat at the forge.
// forge.
// performedKey is the key of one act in performed: the seat that said it and its id, the id in unpadded base64url, since // performedKey is the key of one act in performed (groups.PerformedKey).
// an id may hold what a key may not (`#`, `/`). func performedKey(of, id string) string { return groups.PerformedKey(of, id) }
func performedKey(of, id string) string {
return of + "." + base64.RawURLEncoding.EncodeToString([]byte(id))
}
// Performed is one value of the state performed: the outcome said as owed.ended, and the owed event it answers, so // Performed is one value of the state performed.
// bridge_owed_again performs it again from here. type Performed = groups.Performed
type Performed struct {
Ended forgemsg.OwedEnded `json:"ended"`
Owed json.RawMessage `json:"owed,omitempty"`
}
// performedStore is the state performed as the owing reads and writes it. // performedStore is the state performed as the owing reads and writes it.
type performedStore interface { type performedStore interface {
@@ -70,6 +61,8 @@ type owing struct {
call caller call caller
pub publisher pub publisher
performed performedStore performed performedStore
// acts reads an act again from the stream owed of the seat that said it (H-A-181), for bridge_owed_again.
acts actReader
// waits, when set, says why the state performed cannot be used yet (its binding, design 47 G §577); empty when it can. // waits, when set, says why the state performed cannot be used yet (its binding, design 47 G §577); empty when it can.
waits func() string waits func() string
now func() time.Time now func() time.Time
@@ -90,12 +83,12 @@ func readerOf(of string) (func([]byte) (bridge.OwedCall, error), error) {
func (o *owing) heard(ctx context.Context, of string, data []byte) (forgemsg.OwedEnded, error) { func (o *owing) heard(ctx context.Context, of string, data []byte) (forgemsg.OwedEnded, error) {
read, err := readerOf(of) read, err := readerOf(of)
if err != nil { if err != nil {
return o.end(ctx, forgemsg.OwedEnded{Of: of, Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, data) return o.end(ctx, forgemsg.OwedEnded{Of: of, Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, "", 0)
} }
c, err := read(data) c, err := read(data)
if err != nil { if err != nil {
return o.end(ctx, forgemsg.OwedEnded{Of: of, ID: c.ID, return o.end(ctx, forgemsg.OwedEnded{Of: of, ID: c.ID,
Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, data) Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, c.Repository, c.Number)
} }
if w := o.waiting(); w != "" { if w := o.waiting(); w != "" {
return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, fmt.Errorf("the bridge %s", w) return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, fmt.Errorf("the bridge %s", w)
@@ -104,7 +97,7 @@ func (o *owing) heard(ctx context.Context, of string, data []byte) (forgemsg.Owe
// said twice, or said again while its outcome was on its way: performed once, the outcome said once // said twice, or said again while its outcome was on its way: performed once, the outcome said once
return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, nil return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, nil
} }
return o.perform(ctx, c, data) return o.perform(ctx, c)
} }
// hear is one message of a seat's stream owed: performed and acknowledged, or, when its outcome could not be said, // hear is one message of a seat's stream owed: performed and acknowledged, or, when its outcome could not be said,
@@ -127,19 +120,20 @@ func (o *owing) waiting() string {
} }
// perform calls the seat git's verb of one act and says what it came to. // perform calls the seat git's verb of one act and says what it came to.
func (o *owing) perform(ctx context.Context, c bridge.OwedCall, data []byte) (forgemsg.OwedEnded, error) { func (o *owing) perform(ctx context.Context, c bridge.OwedCall) (forgemsg.OwedEnded, error) {
answer, callErr := o.call(ctx, "seat:"+git.Seat+"."+c.Verb, c.Input) answer, callErr := o.call(ctx, "seat:"+git.Seat+"."+c.Verb, c.Input)
outcome, words := bridge.Outcome(answer, callErr) outcome, words := bridge.Outcome(answer, callErr)
if outcome != forgemsg.OwedPerformed { if outcome != forgemsg.OwedPerformed {
o.logger.Printf("the owed %s %s of %s#%d (said by %s) is %s: %s", c.Act, c.ID, c.Repository, c.Number, c.Of, outcome, words) o.logger.Printf("the owed %s %s of %s#%d (said by %s) is %s: %s", c.Act, c.ID, c.Repository, c.Number, c.Of, outcome, words)
} }
return o.end(ctx, forgemsg.OwedEnded{Of: c.Of, ID: c.ID, return o.end(ctx, forgemsg.OwedEnded{Of: c.Of, ID: c.ID,
Outcome: outcome, Words: words}, data) Outcome: outcome, Words: words}, c.Repository, c.Number)
} }
// end says an act's outcome as owed.ended on the stream forge, then keeps it in performed. An outcome not kept is // end says an act's outcome as owed.ended on the stream forge, then keeps it in performed with the pull request the act
// logged: the act may be performed again, which changes nothing at the forge. // names (repository and number). An outcome not kept is logged: the act may be performed again, which changes nothing at
func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, owed []byte) (forgemsg.OwedEnded, error) { // the forge.
func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, repository string, number int) (forgemsg.OwedEnded, error) {
e.At = o.now().UTC() e.At = o.now().UTC()
data, err := forgemsg.EncodeOwedEnded(e) data, err := forgemsg.EncodeOwedEnded(e)
if err != nil { if err != nil {
@@ -151,10 +145,7 @@ func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, owed []byte) (for
if e.ID == "" { if e.ID == "" {
return e, nil // an undecodable act with no id is said, and has no key to keep return e, nil // an undecodable act with no id is said, and has no key to keep
} }
if !json.Valid(owed) { v, err := json.Marshal(Performed{Ended: e, Repository: repository, Number: number})
owed = nil
}
v, err := json.Marshal(Performed{Ended: e, Owed: owed})
if err == nil { if err == nil {
err = o.performed.Put(ctx, performedKey(e.Of, e.ID), v) err = o.performed.Put(ctx, performedKey(e.Of, e.ID), v)
} }
@@ -175,19 +166,17 @@ func (o *owing) held(ctx context.Context, of, id string) bool {
return found return found
} }
// owedOf is the owed event of an act to perform again, as performed keeps it. // kept is what performed holds of an act: its outcome and its pull request.
func (o *owing) owedOf(ctx context.Context, of, id string) ([]byte, error) { func (o *owing) kept(ctx context.Context, of, id string) (Performed, error) {
v, found, err := o.performed.Get(ctx, performedKey(of, id)) v, found, err := o.performed.Get(ctx, performedKey(of, id))
if err != nil { if err != nil {
return nil, fmt.Errorf("performed cannot be read: %w", err) return Performed{}, fmt.Errorf("performed cannot be read: %w", err)
} }
if found { var p Performed
var p Performed if !found || json.Unmarshal(v, &p) != nil {
if err := json.Unmarshal(v, &p); err == nil && len(p.Owed) > 0 { return Performed{}, fmt.Errorf("the bridge holds no owed act %s of %s in its state performed", id, of)
return p.Owed, nil
}
} }
return nil, fmt.Errorf("the bridge holds no owed act %s of %s in its state performed", id, of) return p, nil
} }
// answer answers one owed ask: an act performed again now, whatever performed holds of it; an act dropped, said as // answer answers one owed ask: an act performed again now, whatever performed holds of it; an act dropped, said as
@@ -199,20 +188,22 @@ func (o *owing) answer(ctx context.Context, a groups.OwedAsk, backlog func(conte
a.Backlog, a.Answer = backlog(ctx), "the backlog of each seat's stream owed" a.Backlog, a.Answer = backlog(ctx), "the backlog of each seat's stream owed"
return a return a
} }
data, err := o.owedOf(ctx, a.Of, a.ID) p, err := o.kept(ctx, a.Of, a.ID)
if err != nil { if err != nil {
a.Answer = err.Error() a.Answer = err.Error()
return a return a
} }
var e forgemsg.OwedEnded var e forgemsg.OwedEnded
if a.Kind == groups.OwedDrop { if a.Kind == groups.OwedDrop {
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedDropped, Words: a.Why + " (dropped by " + a.By + ")"}, data) e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedDropped, Words: a.Why + " (dropped by " + a.By + ")"}, p.Repository, p.Number)
} else if read, rerr := readerOf(a.Of); rerr != nil { } else if read, rerr := readerOf(a.Of); rerr != nil {
err = rerr err = rerr
} else if data, rerr := o.actOf(ctx, a.Of, p, a.ID); rerr != nil {
err = rerr
} else if c, rerr := read(data); rerr != nil { } else if c, rerr := read(data); rerr != nil {
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedUndecodable, Words: rerr.Error()}, data) e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedUndecodable, Words: rerr.Error()}, p.Repository, p.Number)
} else { } else {
e, err = o.perform(ctx, c, data) e, err = o.perform(ctx, c)
} }
if err != nil { if err != nil {
a.Answer = err.Error() a.Answer = err.Error()
@@ -0,0 +1,26 @@
package groups
import (
"encoding/base64"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
)
// The state performed keeps one key per owed act the bridge performed, its outcome and when, and the pull request the
// act names, nothing more (novox/hq design 47 G §163, H-A-181, H-A-182): the act itself stays in the stream owed of the
// seat that said it, its one copy, where bridge_owed_again reads it again. The bridge writes it; its tools read it.
// PerformedKey is the key of one act in performed: the seat that said it and its id, the id in unpadded base64url, since
// an id may hold what a key may not (`#`, `/`).
func PerformedKey(of, id string) string {
return of + "." + base64.RawURLEncoding.EncodeToString([]byte(id))
}
// Performed is one value of the state performed (§163, H-A-182): the outcome said as owed.ended, with when, and the
// repository and number of the pull request the act names, so bridge_owed reads one pull request's acts and
// bridge_owed_again finds the act on its pull request's subject of the stream owed (H-A-181).
type Performed struct {
Ended forgemsg.OwedEnded `json:"ended"`
Repository string `json:"repository,omitempty"`
Number int `json:"number,omitempty"`
}
@@ -187,10 +187,13 @@ func TestManifestProcess(t *testing.T) { // 1235/AC1
if _, bus := p["bus"]; bus { if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential") t.Errorf("the process declares a bus: it runs on its own broker credential")
} }
// Issue 1966's bootstrap: the package artifact forgemsg is out of the manifest until the new builder runs; the follow-up if len(m.Build.Artifacts) != 3 || m.Build.Artifacts[1].Name != "tools" || m.Build.Artifacts[1].From != "cmd/mesh-delivery-forge-bridge-tools" {
// puts it back with its assertion (ADR 0573 §1: three artifacts, the third the package forgemsg). t.Fatalf("artifacts = %+v, want the program, the tools bundle and the package forgemsg", m.Build.Artifacts)
if len(m.Build.Artifacts) != 2 || m.Build.Artifacts[1].Name != "tools" || m.Build.Artifacts[1].From != "cmd/mesh-delivery-forge-bridge-tools" { }
t.Fatalf("artifacts = %+v, want the program and the tools bundle", m.Build.Artifacts) // ADR 0573 §1, H-A-186 (design 47 G §370): the gate, the builder and the indexer import forgemsg, so the module declares
// it as a package and every release publishes it.
if a := m.Build.Artifacts[2]; a.Name != "forgemsg" || a.Kind != "package" || a.Language != "go" || len(a.Paths) != 1 || a.Paths[0] != "forgemsg" || a.Series != "0.1" {
t.Errorf("package artifact = %+v", a)
} }
a := m.Build.Artifacts[0] a := m.Build.Artifacts[0]
if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-forge-bridge" || !strings.HasSuffix(a.From, a.Binary) { if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-forge-bridge" || !strings.HasSuffix(a.From, a.Binary) {
@@ -101,6 +101,15 @@
"loads": [ "loads": [
"mesh-delivery-forge-bridge-tools" "mesh-delivery-forge-bridge-tools"
] ]
},
{
"name": "forgemsg",
"kind": "package",
"language": "go",
"paths": [
"forgemsg"
],
"series": "0.1"
} }
] ]
} }
@@ -31,7 +31,6 @@ import (
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine" "git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -303,8 +302,9 @@ func startIndexerRun(t *testing.T, put func(context.Context, buildLog), before .
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
t.Cleanup(cancel) t.Cleanup(cancel)
// The gate's source-moved on its kept stream, with the indexer's durable of each watched branch the beds register // The gate's source-moved on its kept stream, with the indexer's durable of each watched branch the beds register
// (watchBranch), and the streams of the seat events the indexer consumes on durables of its own: its registration-changed // (watchBranch), and the streams of the seat events the indexer consumes on durables of its own: its registration-changed,
// and the runner's run-started, pipeline-started and pipeline-ended (design 47 H §743, §749), as the grants part makes them. // the runner's run-started, pipeline-started and pipeline-ended (design 47 H §743, §749) and the seat mesh-wants' pipeline-wanted
// (H-A-170), as the grants part makes them.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: meshdelivery.StreamSourceMoved, Subjects: []string{meshdelivery.SubjectSourceMovedAll}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: meshdelivery.StreamSourceMoved, Subjects: []string{meshdelivery.SubjectSourceMovedAll}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -313,7 +313,7 @@ func startIndexerRun(t *testing.T, put func(context.Context, buildLog), before .
} }
for stream, subject := range map[string]string{meshcatalog.StreamRegistrationChanged: meshcatalog.SubjectRegistrationChanged, for stream, subject := range map[string]string{meshcatalog.StreamRegistrationChanged: meshcatalog.SubjectRegistrationChanged,
meshpipelinerunner.StreamRunStarted: meshpipelinerunner.SubjectRunStartedAll, meshpipelinerunner.StreamPipelineStarted: meshpipelinerunner.SubjectPipelineStartedAll, meshpipelinerunner.StreamRunStarted: meshpipelinerunner.SubjectRunStartedAll, meshpipelinerunner.StreamPipelineStarted: meshpipelinerunner.SubjectPipelineStartedAll,
meshpipelinerunner.StreamPipelineEnded: meshpipelinerunner.SubjectPipelineEndedAll} { meshpipelinerunner.StreamPipelineEnded: meshpipelinerunner.SubjectPipelineEndedAll, meshwants.StreamPipelineWanted: meshwants.SubjectPipelineWantedAll} {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{subject}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{subject}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -646,91 +646,6 @@ func (l buildLog) end(t *testing.T, repo, module, path string) {
} }
} }
// TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded (design 47 G §64, §67 test 64, §190; A-160, A-174):
// the indexer starts with the runner's streams run-started and pipeline-ended and its durables on them, its state runs and a
// registration of avahi; the runner says run-started of run 9 with avahi at a commit on a branch no registration watches;
// within 20 s candidates.run-9 holds one pull of number 0 at that commit, avahi's manifest read through the seat git; the
// indexer is restarted, the runner says avahi's pipeline of run 9 ended, and the candidate and the run's key go.
func TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded(t *testing.T) {
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil {
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const commit = "0123456789abcdef0123456789abcdef01234567"
var runs jetstream.KeyValue
js, ctx, candidates, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
var err error
if runs, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.RunsState), History: 1}); err != nil {
t.Fatal(err)
}
regs, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.RegistrationsState), History: 1})
if err != nil {
t.Fatal(err)
}
reg, _ := json.Marshal(meshcatalog.Registration{V: 1, Module: "avahi", Origin: "novox/mesh-catalog", Path: "modules/avahi", Branches: []string{"main"}})
if _, err := regs.Create(ctx, index.RegistrationKey("avahi"), reg); err != nil {
t.Fatal(err)
}
seedBuildModules(t, ctx, js, nil)
// The seat git as the forge module serves it: avahi's directory and its manifest at the run's commit.
reply := func(m *nats.Msg, result any) {
raw, _ := json.Marshal(map[string]any{"result": result, "node": "n0"})
_ = m.Respond(raw)
}
if _, err := js.Conn().Subscribe("mesh.seat.git.tool.trees", func(m *nats.Msg) {
reply(m, git.TreesAnswer{Root: "r0", Commit: commit, Trees: map[string]string{"modules/avahi": "newtree"}})
}); err != nil {
t.Fatal(err)
}
if _, err := js.Conn().Subscribe("mesh.seat.git.tool.file", func(m *nats.Msg) {
var a git.FileInput
_ = json.Unmarshal(m.Data, &a)
switch a.Path {
case "modules/avahi/module.json":
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Content: string(manifest)})
case "modules/avahi/go.sum":
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Content: goSum})
default:
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Absent: true})
}
}); err != nil {
t.Fatal(err)
}
})
started, _ := json.Marshal(meshpipelinerunner.RunStarted{V: 1, Run: "9", By: "lead@n1", Members: []meshpipelinerunner.Member{{Module: "avahi", Commit: commit, Branch: "feature/1966-x"}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectRunStarted("avahi", "feature-1966-x", commit), started); err != nil {
t.Fatal(err)
}
key, _ := meshcatalogclient.CandidatesKey("run-9")
var c meshcatalog.CandidatesByGroupValue
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := candidates.Get(ctx, key); err == nil {
if c, err = index.DecodeCandidate(e.Value()); err == nil {
break
}
}
}
if len(c.Pulls) != 1 || c.Pulls[0].Number != 0 || c.Pulls[0].Head != commit || c.Pulls[0].Fps["avahi"].Manifest != meshcatalogclient.ManifestDigest(manifest) {
t.Fatalf("candidates.run-9 = %+v within 20 s of run-started, want avahi at the run's commit with its manifest", c)
}
restart()
ended, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: "feature-1966-x.0", Branch: "feature/1966-x", Run: "9",
Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}}}, Outcome: meshpipelinerunner.PipelineEndedOutcomePassed})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded("feature-1966-x", "0"), ended); err != nil {
t.Fatal(err)
}
gone := false
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && !gone; time.Sleep(50 * time.Millisecond) {
_, cerr := candidates.Get(ctx, key)
_, rerr := runs.Get(ctx, "run-9")
gone = errors.Is(cerr, jetstream.ErrKeyNotFound) && errors.Is(rerr, jetstream.ErrKeyNotFound)
}
if !gone {
t.Error("the run's candidate or its runs key stays 20 s after its member's pipeline ended across a restart")
}
}
// TestIndexerCountsOnlyThePlacedBuildNodesToolchains (issue 1940, issue 774): the toolchains bucket holds the report of // TestIndexerCountsOnlyThePlacedBuildNodesToolchains (issue 1940, issue 774): the toolchains bucket holds the report of
// n2, where the catalogue no longer places mesh-build-go, at another version than n1's, as ace's v26.12.0 stayed beside // n2, where the catalogue no longer places mesh-build-go, at another version than n1's, as ace's v26.12.0 stayed beside
// the v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's // the v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's
@@ -182,9 +182,9 @@ func containsVia(call toolCaller) func(ctx context.Context, repository, commit,
var lockFiles = []string{"go.sum", "package-lock.json"} var lockFiles = []string{"go.sum", "package-lock.json"}
// readVia reads a repository at a commit as the forge bridge does for a merged message: the seat git's trees, then the // readVia reads a repository at a commit as the forge bridge does for a merged message: the seat git's trees, then the
// module.json and lock files of the one directory dir by its file (a repository can hold hundreds). A manifest the forge // module.json and lock files of the one directory dir by its file (a repository can hold hundreds), and the go.mod beside a
// does not have is an error; a directory with no module.json in the trees is answered with no manifest; a lock file it does // go.sum for its mesh packages (Packages). A manifest the forge does not have is an error; a directory with no module.json in
// not have is left out. // the trees is answered with no manifest; a lock file it does not have is left out.
func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir string) (index.RepoRead, error) { func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir string) (index.RepoRead, error) {
return func(_ context.Context, repository, sha, dir string) (index.RepoRead, error) { return func(_ context.Context, repository, sha, dir string) (index.RepoRead, error) {
a, err := treesOf(call, repository, git.TreesInput{Commit: sha}) a, err := treesOf(call, repository, git.TreesInput{Commit: sha})
@@ -222,13 +222,20 @@ func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir str
out.Locks[dir] = map[string]string{} out.Locks[dir] = map[string]string{}
} }
out.Locks[dir][name] = hex.EncodeToString(sum[:]) out.Locks[dir][name] = hex.EncodeToString(sum[:])
if name == "go.sum" { if name != "go.sum" {
if p := fingerprint.MeshPackagesOf([]byte(c)); len(p) > 0 { continue
if out.Packages == nil { }
out.Packages = map[string][]string{} // A Go module's mesh packages are what its go.mod requires, at any version (design 47 H-A-157): its go.sum
} // names none required at the zero version (ADR 0573 §2).
out.Packages[dir] = p // the mesh packages it uses, as the bridge reads them (ADR 0573 §2) mod, absent, err := file(dir, "go.mod")
if err != nil {
return index.RepoRead{}, fmt.Errorf("%s: %w", path.Join(dir, "go.mod"), err)
}
if p := fingerprint.MeshRequiresOf([]byte(mod)); !absent && len(p) > 0 {
if out.Packages == nil {
out.Packages = map[string][]string{}
} }
out.Packages[dir] = p
} }
} }
} }
@@ -83,22 +83,29 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
// or a run that can never be taken in, is terminated, any other failure is left for redelivery, and a taken one is acked. // or a run that can never be taken in, is terminated, any other failure is left for redelivery, and a taken one is acked.
for _, row := range []struct { for _, row := range []struct {
name, body string name, body string
ended bool ended, wanted bool
err error err error
acked, termd int acked, termd int
started, endedOf string started, endedOf string
}{ }{
{name: "a run started", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}],"by":"s"}`, acked: 1, started: "7"}, {name: "a run started", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}],"by":"s"}`, acked: 1, started: "7"},
{name: "a run's pipeline ended", body: `{"v":1,"pipeline":"p","module":"a","branch":"b","commit":"c","outcome":"passed","steps":[],"run":"7"}`, ended: true, acked: 1, endedOf: "7"}, {name: "a run's pipeline ended", body: `{"v":1,"pipeline":"p","module":"a","branch":"b","commit":"c","outcome":"passed","steps":[],"run":"7"}`, ended: true, acked: 1, endedOf: "7"},
{name: "a pipeline's wants written to the end", body: `{"v":1,"pipeline":"main.12","branch":"main","source":12,"outcome":"tier-late"}`, wanted: true, acked: 1},
{name: "not json", body: `nope`, termd: 1}, {name: "not json", body: `nope`, termd: 1},
{name: "a run never taken in", body: `{"v":1,"run":"7","members":[]}`, err: fmt.Errorf("%w: no member", index.ErrUnreadable), termd: 1, started: "7"}, {name: "a run never taken in", body: `{"v":1,"run":"7","members":[]}`, err: fmt.Errorf("%w: no member", index.ErrUnreadable), termd: 1, started: "7"},
{name: "a bus fault", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}]}`, err: errors.New("the bus did not answer"), started: "7"}, {name: "a bus fault", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}]}`, err: errors.New("the bus did not answer"), started: "7"},
} { } {
ix := &fakeRuns{err: row.err} ix := &fakeRuns{err: row.err}
m := &fakeMsg{subject: "x", data: []byte(row.body)} m := &fakeMsg{subject: "x", data: []byte(row.body)}
if row.ended { switch {
case row.wanted:
heardPipelineWanted(context.Background(), ix, m, t.Logf)
if ix.wanted != "main.12 tier-late" {
t.Errorf("%s: was handed %q, want main.12 tier-late", row.name, ix.wanted)
}
case row.ended:
heardPipelineEnded(context.Background(), ix, m, t.Logf) heardPipelineEnded(context.Background(), ix, m, t.Logf)
} else { default:
heardRunStarted(context.Background(), ix, m, t.Logf) heardRunStarted(context.Background(), ix, m, t.Logf)
} }
if m.acked != row.acked || m.termd != row.termd || ix.started != row.started || ix.ended != row.endedOf { if m.acked != row.acked || m.termd != row.termd || ix.started != row.started || ix.ended != row.endedOf {
@@ -120,6 +127,8 @@ func TestReadViaReadsManifestsAndLocks(t *testing.T) {
return json.RawMessage(`{"path":"module.json","content":"{\"module\":\"snake\"}"}`), nil return json.RawMessage(`{"path":"module.json","content":"{\"module\":\"snake\"}"}`), nil
case "go.sum": case "go.sum":
return json.RawMessage(`{"path":"go.sum","content":"x"}`), nil return json.RawMessage(`{"path":"go.sum","content":"x"}`), nil
case "go.mod": // a mesh package required at the zero version, which no go.sum line names (design 47 H-A-157)
return json.RawMessage(`{"path":"go.mod","content":"module snake\n\nrequire (\n\tgit.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000\n)\n"}`), nil
} }
return json.RawMessage(`{"path":"x","absent":true}`), nil return json.RawMessage(`{"path":"x","absent":true}`), nil
} }
@@ -129,18 +138,21 @@ func TestReadViaReadsManifestsAndLocks(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if len(got.Manifests) != 1 || len(files) != 3 { // the one directory asked: its manifest and two lock files, never the others if len(got.Manifests) != 1 || len(files) != 4 { // the one directory asked: its manifest, two lock files and the go.mod, never the others
t.Fatalf("manifests = %v, files asked = %v", got.Manifests, files) t.Fatalf("manifests = %v, files asked = %v", got.Manifests, files)
} }
if got.Root != "r0" || got.Manifests[""] != `{"module":"snake"}` || len(got.Locks[""]) != 1 || got.Locks[""]["go.sum"] == "" { if got.Root != "r0" || got.Manifests[""] != `{"module":"snake"}` || len(got.Locks[""]) != 1 || got.Locks[""]["go.sum"] == "" {
t.Fatalf("read = %+v", got) t.Fatalf("read = %+v", got)
} }
if p := got.Packages[""]; len(p) != 1 || p[0] != "git.novox.be/novox/mesh-sdk/go" {
t.Fatalf("mesh packages %v, want mesh-sdk, which the go.mod requires", got.Packages)
}
} }
// fakeRuns records the run events handed to it and answers err. // fakeRuns records the run events handed to it and answers err.
type fakeRuns struct { type fakeRuns struct {
err error err error
started, ended string started, ended, wanted string
} }
func (f *fakeRuns) StartRun(_ context.Context, e meshpipelinerunner.RunStarted) error { func (f *fakeRuns) StartRun(_ context.Context, e meshpipelinerunner.RunStarted) error {
@@ -157,4 +169,9 @@ func (f *fakeRuns) HeardPins(context.Context, *meshpipelinerunner.PipelineStarte
return nil return nil
} }
func (f *fakeRuns) HeardPipelineWanted(_ context.Context, branch string, source int64, outcome string) error {
f.wanted = fmt.Sprintf("%s.%d %s", branch, source, outcome)
return f.err
}
func (f *fakeRuns) JudgePins(context.Context) {} func (f *fakeRuns) JudgePins(context.Context) {}
@@ -58,7 +58,8 @@ func untilModule(t *testing.T, ctx context.Context, js jetstream.JetStream, name
} }
// TestASourceRevisionMovingMeshSDKNamesItAndItsImporters (design 47 H §743; A-476a; ADR 0593 §2 to §4): the indexer whole against // TestASourceRevisionMovingMeshSDKNamesItAndItsImporters (design 47 H §743; A-476a; ADR 0593 §2 to §4): the indexer whole against
// the bus. Two Go modules of the catalogue, importer, whose go.sum requires mesh-sdk's Go module, and plain, which requires // the bus. Two Go modules of the catalogue, importer, whose go.mod requires mesh-sdk's Go module at the zero version (no
// go.sum line names it, ADR 0573 §2; design 47 H-A-157), and plain, which requires
// nothing of it, are taken in with mesh-sdk as main's first source revision. A source revision moving mesh-sdk to a commit // nothing of it, are taken in with mesh-sdk as main's first source revision. A source revision moving mesh-sdk to a commit
// changing its buildstep package is taken in as one revision whose movers are mesh-sdk and importer, of another repository, // changing its buildstep package is taken in as one revision whose movers are mesh-sdk and importer, of another repository,
// and not plain; the packages key of mesh-sdk/go names that commit with a digest per package. // and not plain; the packages key of mesh-sdk/go names that commit with a digest per package.
@@ -67,7 +68,7 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
sdkGo = "git.novox.be/novox/mesh-sdk/go" sdkGo = "git.novox.be/novox/mesh-sdk/go"
m1, m2 = "1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222" m1, m2 = "1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222"
catalogMerge = "3333333333333333333333333333333333333333" catalogMerge = "3333333333333333333333333333333333333333"
goSumContent = "git.novox.be/novox/mesh-sdk/go v0.1.14 h1:x=\ngit.novox.be/novox/mesh-sdk/go v0.1.14/go.mod h1:y=\n" goModContent = "module importer\n\nrequire git.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000\n"
sdkRoot1, sdk2 = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" sdkRoot1, sdk2 = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
) )
avahi, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json") avahi, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
@@ -119,7 +120,8 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
forge.put(m2, sdkAt(sdk2, "f6", "7777777777777777777777777777777777777777")) forge.put(m2, sdkAt(sdk2, "f6", "7777777777777777777777777777777777777777"))
forge.put(catalogMerge, repoAt{trees: map[string]string{"modules/importer": "t1", "modules/plain": "t2"}, forge.put(catalogMerge, repoAt{trees: map[string]string{"modules/importer": "t1", "modules/plain": "t2"},
files: map[string]string{"modules/importer/module.json": named("importer"), "modules/plain/module.json": named("plain"), files: map[string]string{"modules/importer/module.json": named("importer"), "modules/plain/module.json": named("plain"),
"modules/importer/go.sum": goSumContent, "modules/plain/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n"}}) "modules/importer/go.mod": goModContent, "modules/importer/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n",
"modules/plain/go.mod": "module plain\n\nrequire golang.org/x/sys v0.1.0\n", "modules/plain/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n"}})
regs := registerAvahi(t, ctx, js, "main") regs := registerAvahi(t, ctx, js, "main")
for _, r := range []meshcatalog.Registration{{V: 1, Module: "importer", Origin: "novox/mesh-catalog", Path: "modules/importer", Branches: []string{"main"}}, for _, r := range []meshcatalog.Registration{{V: 1, Module: "importer", Origin: "novox/mesh-catalog", Path: "modules/importer", Branches: []string{"main"}},
{V: 1, Module: "plain", Origin: "novox/mesh-catalog", Path: "modules/plain", Branches: []string{"main"}}, {V: 1, Module: "plain", Origin: "novox/mesh-catalog", Path: "modules/plain", Branches: []string{"main"}},
@@ -142,7 +144,11 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
say(10, true, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-sdk", Commit: m1}, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-catalog", Commit: catalogMerge}) say(10, true, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-sdk", Commit: m1}, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-catalog", Commit: catalogMerge})
heard.next(t, "main", 10) heard.next(t, "main", 10)
heard.acked(t, "main") heard.acked(t, "main")
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != "" && m.Packages[sdkGo] == m1 }) // mesh-sdk has released no version: importer keeps the package for its kind-1 edge, with no pin (H-A-161).
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool {
pin, kept := m.Packages[sdkGo]
return m.Fp2 != "" && kept && pin == ""
})
plain := untilModule(t, ctx, js, "plain", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != "" }) plain := untilModule(t, ctx, js, "plain", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != "" })
if _, ok := plain.Packages[sdkGo]; ok { if _, ok := plain.Packages[sdkGo]; ok {
t.Fatalf("plain requires %s: %+v", sdkGo, plain.Packages) t.Fatalf("plain requires %s: %+v", sdkGo, plain.Packages)
@@ -6,15 +6,17 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
) )
// The indexer pins a pipeline's revision on the runner's pipeline-started and frees it on its pipeline-ended, each heard on // The indexer pins a pipeline's revision on the runner's pipeline-started and frees it on its pipeline-ended, a passed one's release
// a durable consumer of its own on that event's stream (design 47 H §749, runs.go): an event is applied to the state pins, // pinned there and freed on the seat mesh-wants' pipeline-wanted of a newer pipeline (H-A-168, H-A-170), each heard on a durable
// and only then acknowledged, so one said while the indexer was down is applied when it comes back. // consumer of its own on that event's stream (design 47 H §749, runs.go): an event is applied to the state pins, and only then
// acknowledged, so one said while the indexer was down is applied when it comes back.
// pinsStreams are the streams of the runner's events the indexer's pins follow. // pinsStreams are the streams of the events the indexer's pins follow.
var pinsStreams = []string{meshpipelinerunner.StreamPipelineStarted, meshpipelinerunner.StreamPipelineEnded} var pinsStreams = []string{meshpipelinerunner.StreamPipelineStarted, meshpipelinerunner.StreamPipelineEnded, meshwants.StreamPipelineWanted}
// pendingOf says how many messages of the indexer's durable consumers of the pins' streams have not been given to it yet. // pendingOf says how many messages of the indexer's durable consumers of the pins' streams have not been given to it yet.
func pendingOf(js jetstream.JetStream, cred process.Account) func(context.Context) (uint64, error) { func pendingOf(js jetstream.JetStream, cred process.Account) func(context.Context) (uint64, error) {
@@ -8,17 +8,20 @@ import (
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/internal/index" "mesh-delivery-indexer/internal/index"
) )
// runTaker is what the indexer does with the runner's events: a run on request's (index.Indexer.StartRun, EndRunPipeline), // runTaker is what the indexer does with the runner's events: a run on request's (index.Indexer.StartRun, EndRunPipeline),
// and a pipeline's pins (HeardPins, design 47 H §749), the kept revisions judged after each pin freed (JudgePins). // and a pipeline's pins (HeardPins, design 47 H §749), a release's freed on the seat mesh-wants' pipeline-wanted (HeardPipelineWanted,
// H-A-168, H-A-170), the kept revisions judged after each pin freed (JudgePins).
type runTaker interface { type runTaker interface {
StartRun(ctx context.Context, e meshpipelinerunner.RunStarted) error StartRun(ctx context.Context, e meshpipelinerunner.RunStarted) error
EndRunPipeline(ctx context.Context, e meshpipelinerunner.PipelineEnded) error EndRunPipeline(ctx context.Context, e meshpipelinerunner.PipelineEnded) error
HeardPins(ctx context.Context, started *meshpipelinerunner.PipelineStarted, ended *meshpipelinerunner.PipelineEnded) error HeardPins(ctx context.Context, started *meshpipelinerunner.PipelineStarted, ended *meshpipelinerunner.PipelineEnded) error
HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error
JudgePins(ctx context.Context) JudgePins(ctx context.Context)
} }
@@ -33,6 +36,7 @@ func hearRuns(ctx context.Context, js jetstream.JetStream, cred process.Account,
{meshpipelinerunner.StreamRunStarted, func(m jetstream.Msg) { heardRunStarted(ctx, ix, m, logf) }}, {meshpipelinerunner.StreamRunStarted, func(m jetstream.Msg) { heardRunStarted(ctx, ix, m, logf) }},
{meshpipelinerunner.StreamPipelineStarted, func(m jetstream.Msg) { heardPipelineStarted(ctx, ix, m, logf) }}, {meshpipelinerunner.StreamPipelineStarted, func(m jetstream.Msg) { heardPipelineStarted(ctx, ix, m, logf) }},
{meshpipelinerunner.StreamPipelineEnded, func(m jetstream.Msg) { heardPipelineEnded(ctx, ix, m, logf) }}, {meshpipelinerunner.StreamPipelineEnded, func(m jetstream.Msg) { heardPipelineEnded(ctx, ix, m, logf) }},
{meshwants.StreamPipelineWanted, func(m jetstream.Msg) { heardPipelineWanted(ctx, ix, m, logf) }},
} { } {
_ = notHeard.Bind(s.stream, "no run on request's candidate is written or dropped until the indexer starts again", func() error { _ = notHeard.Bind(s.stream, "no run on request's candidate is written or dropped until the indexer starts again", func() error {
cc, err := cred.HearSeatEvents(ctx, js, s.stream, s.each, logf) cc, err := cred.HearSeatEvents(ctx, js, s.stream, s.each, logf)
@@ -85,6 +89,23 @@ func heardPipelineEnded(ctx context.Context, ix runTaker, m delivered, logf func
ix.JudgePins(ctx) ix.JudgePins(ctx)
} }
// heardPipelineWanted takes one pipeline-wanted of the seat mesh-wants in: the pipeline's wants are written to the end, tier-late
// included, so the older releases of the modules it moves free their revisions, and the kept revisions are judged (H-A-168,
// H-A-170).
func heardPipelineWanted(ctx context.Context, ix runTaker, m delivered, logf func(string, ...any)) {
var e meshwants.PipelineWanted
if err := json.Unmarshal(m.Data(), &e); err != nil || e.Branch == "" || e.Source <= 0 {
logf("%s: not a pipeline-wanted naming a branch and a source, dropped: %v", m.Subject(), err)
_ = m.Term()
return
}
err := ix.HeardPipelineWanted(ctx, e.Branch, e.Source, e.Outcome)
settleRun(m, err, "pipeline-wanted of "+e.Pipeline, logf)
if err == nil {
ix.JudgePins(ctx)
}
}
func settleRun(m delivered, err error, what string, logf func(string, ...any)) { func settleRun(m delivered, err error, what string, logf func(string, ...any)) {
switch { switch {
case err == nil: case err == nil:
@@ -7,6 +7,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"os" "os"
"slices"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@@ -16,10 +17,12 @@ import (
"github.com/nats-io/nats.go" "github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/keys" "git.novox.be/novox/mesh-sdk/go/keys"
@@ -185,12 +188,14 @@ func sayMoved(t *testing.T, ctx context.Context, js jetstream.JetStream, branch
type takeIns struct { type takeIns struct {
taken chan meshcatalog.TakeInTaken taken chan meshcatalog.TakeInTaken
pending chan meshcatalog.TakeInPending pending chan meshcatalog.TakeInPending
acks chan string // the escaped branch whose source-moved the indexer acknowledged acks chan string // the escaped branch whose source-moved the indexer acknowledged
runAcks chan struct{} // a run-started the indexer acknowledged
} }
func hearTakeIns(t *testing.T, nc *nats.Conn) *takeIns { func hearTakeIns(t *testing.T, nc *nats.Conn) *takeIns {
t.Helper() t.Helper()
h := &takeIns{taken: make(chan meshcatalog.TakeInTaken, 64), pending: make(chan meshcatalog.TakeInPending, 64), acks: make(chan string, 64)} h := &takeIns{taken: make(chan meshcatalog.TakeInTaken, 64), pending: make(chan meshcatalog.TakeInPending, 64), acks: make(chan string, 64),
runAcks: make(chan struct{}, 64)}
prefix := "$JS.ACK." + meshdelivery.StreamSourceMoved + "." prefix := "$JS.ACK." + meshdelivery.StreamSourceMoved + "."
acks, err := nc.Subscribe(prefix+"*.>", func(m *nats.Msg) { acks, err := nc.Subscribe(prefix+"*.>", func(m *nats.Msg) {
consumer, _, _ := strings.Cut(strings.TrimPrefix(m.Subject, prefix), ".") consumer, _, _ := strings.Cut(strings.TrimPrefix(m.Subject, prefix), ".")
@@ -202,6 +207,15 @@ func hearTakeIns(t *testing.T, nc *nats.Conn) *takeIns {
t.Fatal(err) t.Fatal(err)
} }
t.Cleanup(func() { _ = acks.Unsubscribe() }) t.Cleanup(func() { _ = acks.Unsubscribe() })
runAcks, err := nc.Subscribe("$JS.ACK."+meshpipelinerunner.StreamRunStarted+".n1_mesh-delivery-indexer.>", func(m *nats.Msg) {
if string(m.Data) == "+ACK" || len(m.Data) == 0 {
h.runAcks <- struct{}{}
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = runAcks.Unsubscribe() })
sub, err := nc.Subscribe("mesh.mod."+testIndexer+".event.take-in.*", func(m *nats.Msg) { sub, err := nc.Subscribe("mesh.mod."+testIndexer+".event.take-in.*", func(m *nats.Msg) {
switch m.Subject { switch m.Subject {
case "mesh.mod." + testIndexer + ".event.take-in.taken": case "mesh.mod." + testIndexer + ".event.take-in.taken":
@@ -251,6 +265,16 @@ func (h *takeIns) acked(t *testing.T, branch string) {
} }
} }
// runAcked waits for the indexer's acknowledgement of a run-started.
func (h *takeIns) runAcked(t *testing.T) {
t.Helper()
select {
case <-h.runAcks:
case <-time.After(waitBound):
t.Fatalf("no run-started acknowledged within %s", waitBound)
}
}
// nextPending is the next take-in.pending said, which must be of that branch and source. // nextPending is the next take-in.pending said, which must be of that branch and source.
func (h *takeIns) nextPending(t *testing.T, branch string, source int64) meshcatalog.TakeInPending { func (h *takeIns) nextPending(t *testing.T, branch string, source int64) meshcatalog.TakeInPending {
t.Helper() t.Helper()
@@ -339,17 +363,22 @@ func sayPipeline(t *testing.T, ctx context.Context, js jetstream.JetStream, ende
return return
} }
raw, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: pipeline, Branch: branch, Source: source, Revision: source, raw, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: pipeline, Branch: branch, Source: source, Revision: source,
Outcome: meshpipelinerunner.PipelineEndedOutcomePassed, Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}, Tier: 1}}}) Outcome: meshpipelinerunner.PipelineEndedOutcomePassed, Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}, Tier: 1,
Version: "0.1." + s}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded(b, s), raw); err != nil { if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded(b, s), raw); err != nil {
t.Fatal(err) t.Fatal(err)
} }
} }
// pinned is whether a pins value names the record. // pinned is whether a pins value names exactly the records, in any order.
func pinned(record string) func([]byte) bool { func pinned(records ...string) func([]byte) bool {
return func(v []byte) bool { return func(v []byte) bool {
var p index.PinValue var p index.PinValue
return v != nil && json.Unmarshal(v, &p) == nil && len(p.Records) == 1 && p.Records[0] == record if v == nil || json.Unmarshal(v, &p) != nil {
return false
}
slices.Sort(p.Records)
return slices.Equal(p.Records, slices.Sorted(slices.Values(records)))
} }
} }
@@ -364,10 +393,14 @@ func absent(v []byte) bool { return v == nil }
// given again says the same from the kept movers and keeps no second revision; main.10, the branch's previous revision, is // given again says the same from the kept movers and keeps no second revision; main.10, the branch's previous revision, is
// pinned pipeline.main.11 in the same write as validate's base (H-A-147); // pinned pipeline.main.11 in the same write as validate's base (H-A-147);
// - pipeline-started of main.12 pins it, a restart keeps main.11 and main.10 while main.11's pipeline runs; // - pipeline-started of main.12 pins it, a restart keeps main.11 and main.10 while main.11's pipeline runs;
// - a run on request's run-started (run 9, avahi at main 12's commit on a branch no registration watches) writes
// candidates.run-9, one pull of number 0 at that commit with avahi's manifest read through the seat git, and the run's key; across
// the restart, the runner's pipeline-ended of its one member drops both (design 47 G §64, §190; A-160, A-174);
// - with the store full, main.13 is held, said once as take-in.pending, written whole in the state held and acknowledged // - with the store full, main.13 is held, said once as take-in.pending, written whole in the state held and acknowledged
// (H-A-114, H-A-118); nothing takes it in on a // (H-A-114, H-A-118); nothing takes it in on a
// timer: main.11's pipeline-ended frees both its pins, main.10 goes (main.11 stays, main.12's base), the indexer says its // timer: main.11's pipeline-ended frees both its pins, main.10 goes (main.11 stays, main.12's base and its passed release), the
// room and takes main.13 in on it; // indexer says its room and takes main.13 in on it; main.12's passed end and its pipeline-wanted free main.11's release, and
// main.11 goes (H-A-168, H-A-170, H-A-173);
// - beta's take-in whose commit the forge cannot read is held while main's next goes on; the hand skip, with why, is recorded in // - beta's take-in whose commit the forge cannot read is held while main's next goes on; the hand skip, with why, is recorded in
// the hand-act log before it is asked, and take-in.taken says it skipped; // the hand-act log before it is asked, and take-in.taken says it skipped;
// - a branch watched later is bound on registration-changed and its first revision taken in. // - a branch watched later is bound on registration-changed and its first revision taken in.
@@ -378,8 +411,8 @@ func TestIndexerWhole(t *testing.T) {
) )
var forge *forgeCommits var forge *forgeCommits
var heard *takeIns var heard *takeIns
var pins, held, takeinsKV, regs jetstream.KeyValue var pins, held, takeinsKV, regs, runs jetstream.KeyValue
js, ctx, _, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) { js, ctx, candidates, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
heard = hearTakeIns(t, js.Conn()) heard = hearTakeIns(t, js.Conn())
forge = serveCommits(t, js.Conn()) forge = serveCommits(t, js.Conn())
for commit, tree := range map[string]string{c10: "t10", c11: "t11", c12: "t12", c13: "t13"} { for commit, tree := range map[string]string{c10: "t10", c11: "t11", c12: "t12", c13: "t13"} {
@@ -389,7 +422,7 @@ func TestIndexerWhole(t *testing.T) {
for _, b := range []struct { for _, b := range []struct {
state string state string
kv *jetstream.KeyValue kv *jetstream.KeyValue
}{{index.HeldState, &held}, {index.TakeInsState, &takeinsKV}} { }{{index.HeldState, &held}, {index.TakeInsState, &takeinsKV}, {index.RunsState, &runs}} {
if *b.kv, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, b.state), History: 1}); err != nil { if *b.kv, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, b.state), History: 1}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -457,6 +490,25 @@ func TestIndexerWhole(t *testing.T) {
} }
sayPipeline(t, ctx, js, false, "main", 12) sayPipeline(t, ctx, js, false, "main", 12)
untilKey(t, ctx, pins, index.PinKey("main", 12), pinned("pipeline.main.12")) untilKey(t, ctx, pins, index.PinKey("main", 12), pinned("pipeline.main.12"))
// A run on request: its candidate and its key written on run-started, from avahi's files at the run's commit (A-160, A-174).
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil {
t.Fatal(err)
}
started, _ := json.Marshal(meshpipelinerunner.RunStarted{V: 1, Run: "9", By: "lead@n1", Members: []meshpipelinerunner.Member{{Module: "avahi", Commit: c12, Branch: "feature/1966-x"}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectRunStarted("avahi", "feature-1966-x", c12), started); err != nil {
t.Fatal(err)
}
runKey, _ := meshcatalogclient.CandidatesKey("run-9")
untilKey(t, ctx, candidates, runKey, func(v []byte) bool {
c, err := index.DecodeCandidate(v)
return v != nil && err == nil && len(c.Pulls) == 1 && c.Pulls[0].Number == 0 && c.Pulls[0].Head == c12 &&
c.Pulls[0].Fps["avahi"].Manifest == meshcatalogclient.ManifestDigest(manifest)
})
untilKey(t, ctx, runs, "run-9", func(v []byte) bool { return v != nil })
// The candidate is written before its run-started is acknowledged: the restart waits for that too, so the stop leaves it
// unacknowledged for no redelivery to write the candidate again after its run ended.
heard.runAcked(t)
// The restart waits for the take-ins' ends, their source-moved acknowledged (design 47 H §743: main 10, 11, 11 given // The restart waits for the take-ins' ends, their source-moved acknowledged (design 47 H §743: main 10, 11, 11 given
// again and 12), so the stop cancels none and leaves none unacknowledged ahead of main 13. // again and 12), so the stop cancels none and leaves none unacknowledged ahead of main 13.
for range 4 { for range 4 {
@@ -468,6 +520,14 @@ func TestIndexerWhole(t *testing.T) {
t.Fatalf("%s is not kept across a restart while main.11's pipeline runs: %v", name, err) t.Fatalf("%s is not kept across a restart while main.11's pipeline runs: %v", name, err)
} }
} }
// The run's member's pipeline ends across the restart: its candidate and its key go (A-174).
ended, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: "feature-1966-x.0", Branch: "feature/1966-x", Run: "9",
Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}}}, Outcome: meshpipelinerunner.PipelineEndedOutcomePassed})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded("feature-1966-x", "0"), ended); err != nil {
t.Fatal(err)
}
untilKey(t, ctx, candidates, runKey, absent)
untilKey(t, ctx, runs, "run-9", absent)
// The store full: held, said once, in the state held; taken in on the indexer's own room once a freed pin makes it. // The store full: held, said once, in the state held; taken in on the indexer's own room once a freed pin makes it.
used, _, err := objects.Fill(ctx) used, _, err := objects.Fill(ctx)
@@ -484,7 +544,9 @@ func TestIndexerWhole(t *testing.T) {
return v != nil && json.Unmarshal(v, &hs) == nil && len(hs) == 1 && hs[0].Source == 13 && hs[0].Why != "" && !hs[0].Since.IsZero() && return v != nil && json.Unmarshal(v, &hs) == nil && len(hs) == 1 && hs[0].Source == 13 && hs[0].Why != "" && !hs[0].Since.IsZero() &&
hs[0].Call != "" && len(hs[0].Heads) == 1 && hs[0].Heads[0].Commit == c13 hs[0].Call != "" && len(hs[0].Heads) == 1 && hs[0].Heads[0].Commit == c13
}) })
// H-A-114: the held source-moved is acknowledged on the bus, kept only in the state held. // H-A-114: the held source-moved is acknowledged on the bus, kept only in the state held; its acknowledgement follows the
// write, so it is waited for before the consumer is read.
heard.acked(t, "main")
if cons, err := testAccount.NamedEventConsumer(ctx, js, meshdelivery.StreamSourceMoved, keys.EscapeName("main")); err != nil { if cons, err := testAccount.NamedEventConsumer(ctx, js, meshdelivery.StreamSourceMoved, keys.EscapeName("main")); err != nil {
t.Fatal(err) t.Fatal(err)
} else if info, err := cons.Info(ctx); err != nil || info.NumAckPending != 0 { } else if info, err := cons.Info(ctx); err != nil || info.NumAckPending != 0 {
@@ -496,8 +558,22 @@ func TestIndexerWhole(t *testing.T) {
t.Errorf("main 13 taken in on room said %+v, want avahi at %s", got, c13) t.Errorf("main 13 taken in on room said %+v, want avahi at %s", got, c13)
} }
untilKey(t, ctx, held, keys.EscapeName("main"), absent) untilKey(t, ctx, held, keys.EscapeName("main"), absent)
untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.12")) // main.12's validate base stays (H-A-147) // main.12's validate base stays (H-A-147), and main.11's passed pipeline's release pins it (H-A-168, H-A-170).
untilRevisionGone(t, ctx, js, "main.10") // the end freed both of main.11's pins untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.12", index.ReleaseRecord("main", "avahi")))
untilRevisionGone(t, ctx, js, "main.10") // the end freed both of main.11's pins
// main.12's passed pipeline releases avahi again; its pipeline-wanted, the seat mesh-wants' (its wants written to the end),
// frees main.11's older release, and main.11 goes (H-A-168, H-A-170, H-A-173).
sayPipeline(t, ctx, js, true, "main", 12)
untilKey(t, ctx, pins, index.PinKey("main", 12), func(v []byte) bool {
var p index.PinValue
return v != nil && json.Unmarshal(v, &p) == nil && slices.Contains(p.Records, index.ReleaseRecord("main", "avahi"))
})
wantedRaw, _ := json.Marshal(meshwants.PipelineWanted{V: 1, Pipeline: "main.12", Branch: "main", Source: 12, Outcome: meshwants.PipelineWantedOutcomeRunning})
if _, err := js.Publish(ctx, meshwants.SubjectPipelineWanted("main", "12"), wantedRaw); err != nil {
t.Fatal(err)
}
untilKey(t, ctx, pins, index.PinKey("main", 11), absent)
untilRevisionGone(t, ctx, js, "main.11")
if n := len(heard.pending); n != 0 { if n := len(heard.pending); n != 0 {
t.Errorf("take-in.pending said %d more times for one held source revision, want once", n) t.Errorf("take-in.pending said %d more times for one held source revision, want once", n)
} }
@@ -135,14 +135,32 @@ func NamedTree(manifest string, paths map[string]string) (string, error) {
return NamedTreePrefix + hex.EncodeToString(sum[:]), nil return NamedTreePrefix + hex.EncodeToString(sum[:]), nil
} }
// MeshPackagesOf are the mesh Go modules a go.sum names (the module path of each line under // MeshRequiresOf are the mesh Go modules a go.mod requires (the module path of each `require` line under
// trunk.ForgeModulePrefix), sorted and each once; nil for none. They are what a module with no reported import set // trunk.ForgeModulePrefix, alone or in a block, at any version, the zero version and `// indirect` included), sorted and
// takes the version line of (ADR 0593 §2). // each once; nil for none. They are a module's mesh packages from the take-in, with no build needed (design 47 H-A-157):
func MeshPackagesOf(goSum []byte) []string { // a go.sum names none required at the zero version (ADR 0573 §2), so it is never read for them.
func MeshRequiresOf(goMod []byte) []string {
seen := map[string]bool{} seen := map[string]bool{}
for _, line := range strings.Split(string(goSum), "\n") { inBlock := false
for _, line := range strings.Split(string(goMod), "\n") {
line = strings.TrimSpace(line)
if i := strings.Index(line, "//"); i >= 0 {
line = strings.TrimSpace(line[:i])
}
switch {
case line == "require (":
inBlock = true
continue
case inBlock && line == ")":
inBlock = false
continue
case strings.HasPrefix(line, "require "):
line = strings.TrimSpace(strings.TrimPrefix(line, "require "))
case !inBlock:
continue
}
f := strings.Fields(line) f := strings.Fields(line)
if len(f) < 2 || !strings.HasPrefix(f[0], trunk.ForgeModulePrefix) { if len(f) != 2 || !strings.HasPrefix(f[0], trunk.ForgeModulePrefix) {
continue continue
} }
if _, _, ok := trunk.PackageRepository(f[0]); ok { if _, _, ok := trunk.PackageRepository(f[0]); ok {
@@ -405,41 +423,49 @@ func sortedKeys[V any](m map[string]V) []string { return slices.Sorted(maps.Keys
// commit or no directory, is in missing, sorted; versions is nil when packages is empty. // commit or no directory, is in missing, sorted; versions is nil when packages is empty.
func HeldVersionsOf(packages []string, modules map[string]meshcatalog.ModulesByModuleValue) (versions sdkstep.PackageVersions, missing []string) { func HeldVersionsOf(packages []string, modules map[string]meshcatalog.ModulesByModuleValue) (versions sdkstep.PackageVersions, missing []string) {
for _, pkg := range packages { for _, pkg := range packages {
repository, dir, ok := trunk.PackageRepository(pkg) key := HolderOf(pkg, modules)
commit, depth := "", -1 if key == "" {
if ok {
for _, key := range slices.Sorted(maps.Keys(modules)) {
m := modules[key]
if !trunk.SameRepository(m.Repository, repository) || m.Commit == "" {
continue
}
at := strings.Trim(m.Path, "/")
switch {
case at == "":
continue // a value that names no directory says nothing of where the package lies
case at == meshcatalogclient.RootPath:
at = ""
case dir != at && !strings.HasPrefix(dir, at+"/"):
continue
}
if d := len(at); d > depth {
commit, depth = m.Commit, d
}
}
}
if commit == "" {
missing = append(missing, pkg) missing = append(missing, pkg)
continue continue
} }
if versions == nil { if versions == nil {
versions = sdkstep.PackageVersions{} versions = sdkstep.PackageVersions{}
} }
versions[pkg] = commit versions[pkg] = modules[key].Commit
} }
slices.Sort(missing) slices.Sort(missing)
return versions, missing return versions, missing
} }
// HolderOf is the key of the held module of the package's repository whose directory holds the package's directory (the
// deepest, a module at the repository's root holding every directory) and that names a commit; "" for none.
func HolderOf(pkg string, modules map[string]meshcatalog.ModulesByModuleValue) string {
repository, dir, ok := trunk.PackageRepository(pkg)
if !ok {
return ""
}
holder, depth := "", -1
for _, key := range slices.Sorted(maps.Keys(modules)) {
m := modules[key]
if !trunk.SameRepository(m.Repository, repository) || m.Commit == "" {
continue
}
at := strings.Trim(m.Path, "/")
switch {
case at == "":
continue // a value that names no directory says nothing of where the package lies
case at == meshcatalogclient.RootPath:
at = ""
case dir != at && !strings.HasPrefix(dir, at+"/"):
continue
}
if d := len(at); d > depth {
holder, depth = key, d
}
}
return holder
}
// digestOf is the content digest of the package at import path pkg in a published module, and whether it holds it // digestOf is the content digest of the package at import path pkg in a published module, and whether it holds it
// (its packages are sorted by import path). // (its packages are sorted by import path).
func digestOf(p meshcatalog.PackagesByModuleValue, pkg string) (string, bool) { func digestOf(p meshcatalog.PackagesByModuleValue, pkg string) (string, bool) {
@@ -455,3 +481,21 @@ func digestOf(p meshcatalog.PackagesByModuleValue, pkg string) (string, bool) {
// NamedTreePrefix opens a module's tree when it is computed over the paths its artifacts name, not the directory's own // NamedTreePrefix opens a module's tree when it is computed over the paths its artifacts name, not the directory's own
// git tree (novox/hq ADR 0499 clause 2): the indexer alone computes it (NamedTree). // git tree (novox/hq ADR 0499 clause 2): the indexer alone computes it (NamedTree).
const NamedTreePrefix = "named:" const NamedTreePrefix = "named:"
// WithPins is the fingerprint fp with the pins a build of it is made against (design 47 H-A-167): fp itself when no
// package is pinned, else `src2:` and the hex sha256 of fp and, sorted, the lines "pin <package> <version>" of each pin,
// joined by a newline. A package kept with no version (none released) is no pin.
func WithPins(fp string, pins sdkstep.PackageVersions) string {
var lines []string
for p, v := range pins {
if v != "" {
lines = append(lines, "pin "+p+" "+v)
}
}
if fp == "" || len(lines) == 0 {
return fp
}
sort.Strings(lines)
sum := sha256.Sum256([]byte(fp + "\n" + strings.Join(lines, "\n")))
return meshcatalogclient.SourcePrefix2 + hex.EncodeToString(sum[:])
}
@@ -31,6 +31,20 @@ func versionLineTakesASiblingDirectoryByItsContent(t *testing.T) {
if got := ReplacedSiblings(goMod, repo); len(got) != 1 || got[0] != buildrun { if got := ReplacedSiblings(goMod, repo); len(got) != 1 || got[0] != buildrun {
t.Fatalf("siblings %v, want buildrun alone (mesh-sdk is another repository, replaced by a version)", got) t.Fatalf("siblings %v, want buildrun alone (mesh-sdk is another repository, replaced by a version)", got)
} }
// Its mesh packages are every mesh package its go.mod requires, at any version, the zero version included (design 47
// H-A-157); a replace line or a module outside the forge is none.
for _, c := range []struct {
goMod string
want []string
}{
{string(goMod), []string{buildrun, sdk}},
{"module x\n\nrequire " + sdk + " v0.0.0-00010101000000-000000000000 // indirect\nrequire github.com/nats-io/nats.go v1.37.0\n", []string{sdk}},
{"module x\n\nreplace " + sdk + " => ../sdk\n", nil},
} {
if got := MeshRequiresOf([]byte(c.goMod)); strings.Join(got, ",") != strings.Join(c.want, ",") {
t.Errorf("mesh packages of %q: %v, want %v", c.goMod, got, c.want)
}
}
published := func(digest string) map[string]meshcatalog.PackagesByModuleValue { published := func(digest string) map[string]meshcatalog.PackagesByModuleValue {
return map[string]meshcatalog.PackagesByModuleValue{buildrun: {Module: buildrun, Repository: repo, Requires: "r1", return map[string]meshcatalog.PackagesByModuleValue{buildrun: {Module: buildrun, Repository: repo, Requires: "r1",
Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: buildrun, Digest: digest}}}} Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: buildrun, Digest: digest}}}}
@@ -9,6 +9,7 @@ import (
"reflect" "reflect"
"slices" "slices"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -30,7 +31,7 @@ const CatalogEntryLayout = 1
// ModuleValueOf gives the module's value back from the entry alone. The acting rules are the revision's to add. // ModuleValueOf gives the module's value back from the entry alone. The acting rules are the revision's to add.
func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catalogue.Manifest, raw string, n graph.Node) meshcatalog.CatalogEntry { func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catalogue.Manifest, raw string, n graph.Node) meshcatalog.CatalogEntry {
e := meshcatalog.CatalogEntry{V: CatalogEntryLayout, Module: module, Repository: cur.Repository, Path: cur.Path, e := meshcatalog.CatalogEntry{V: CatalogEntryLayout, Module: module, Repository: cur.Repository, Path: cur.Path,
Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalog.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases, Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalogclient.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases,
Packages: maps.Clone(cur.Packages), Provides: n.Provides, Requires: n.Requires, Awaits: n.Awaits, Invokes: n.Invokes, Packages: maps.Clone(cur.Packages), Provides: n.Provides, Requires: n.Requires, Awaits: n.Awaits, Invokes: n.Invokes,
Serves: n.Serves, LeftOut: n.LeftOut, Main: MainOf(m), Contributes: contributesOf(m)} Serves: n.Serves, LeftOut: n.LeftOut, Main: MainOf(m), Contributes: contributesOf(m)}
// requires names every requirement as catalogue.Manifest.Wants lists them, optional ones and those only contributes names // requires names every requirement as catalogue.Manifest.Wants lists them, optional ones and those only contributes names
@@ -67,7 +68,7 @@ func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catal
// ruling on milestone 47/G finding 1; ADR 0656 §6: one form of the fact). The old line's fields (the old-form // ruling on milestone 47/G finding 1; ADR 0656 §6: one form of the fact). The old line's fields (the old-form
// fingerprint, the reader needs) are not kept. // fingerprint, the reader needs) are not kept.
func ModuleValueOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue { func ModuleValueOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest), v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest),
Verdict: e.Verdict, Fp2: e.Fp, None2: e.None, Packages: maps.Clone(e.Packages)} Verdict: e.Verdict, Fp2: e.Fp, None2: e.None, Packages: maps.Clone(e.Packages)}
if in := e.Inputs; in != nil { if in := e.Inputs; in != nil {
v.Toolchains, v.Paths, v.Problems, v.Warnings, v.Unknown = maps.Clone(in.Toolchains), maps.Clone(in.Paths), in.Problems, in.Warnings, in.Unknown v.Toolchains, v.Paths, v.Problems, v.Warnings, v.Unknown = maps.Clone(in.Toolchains), maps.Clone(in.Paths), in.Problems, in.Warnings, in.Unknown
@@ -544,7 +544,7 @@ func (k *KeptTrunk) reread(ctx context.Context, o Objects) error {
} }
manifests := map[string]string{} manifests := map[string]string{}
for _, e := range entries { for _, e := range entries {
if d := meshcatalog.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil { if d := meshcatalogclient.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil {
if raw, err := json.Marshal(e.Manifest); err == nil { if raw, err := json.Marshal(e.Manifest); err == nil {
manifests[d] = string(raw) manifests[d] = string(raw)
} }
@@ -17,6 +17,11 @@ import (
type form struct { type form struct {
fp, none string fp, none string
versions sdkstep.PackageVersions versions sdkstep.PackageVersions
// stands: fp is the held fingerprint of the build that reported the imports, made with the pins built (its held
// packages) from the import form raw; pinnedOf keeps it while the pins are those (H-A-172).
stands bool
raw string
built map[string]string
} }
// formOf is the module's fingerprint from its kept inputs (bases already its bases' fingerprints) and toolchains, in the // formOf is the module's fingerprint from its kept inputs (bases already its bases' fingerprints) and toolchains, in the
@@ -24,15 +29,16 @@ type form struct {
// //
// - **The import form**, for a module whose last trunk build reported its imports (kept.Imports) when every import is // - **The import form**, for a module whose last trunk build reported its imports (kept.Imports) when every import is
// published (the packages state): the content digest of each imported package and the requirements' digest of each // published (the packages state): the content digest of each imported package and the requirements' digest of each
// mesh Go module holding one. Its versions are each holding module's published commit (another repository's only). // mesh Go module holding one. Its versions are each holding module's published commit (another repository's only); an
// import set naming none of another repository while the go.mod requires one keeps the version line's (refined).
// - **A change of form never rebuilds by itself** (§4): kept.ImportForm is the import-form fingerprint of the inputs // - **A change of form never rebuilds by itself** (§4): kept.ImportForm is the import-form fingerprint of the inputs
// the build that reported the imports was built from. While the import form computes to it, nothing the build was // the build that reported the imports was built from. While the import form computes to it, nothing the build was
// built from changed: the module keeps the fingerprint it holds (prev), in whatever form, and its versions. // built from changed: the module keeps the fingerprint it holds (prev), in whatever form, and its versions.
// Otherwise the import form is written, and ImportForm moves with it. // Otherwise the import form is written, and ImportForm moves with it.
// - **The version line** otherwise (no import set, or an import none publishes): the version of each mesh Go module of // - **The version line** otherwise (no import set, or an import none publishes): the version of each mesh Go module of
// another repository its go.sum names, read as the indexer always read it (fingerprint.HeldVersionsOf), and the // another repository its go.mod requires, at any version (design 47 H-A-157), read as the indexer always read it
// content of each mesh Go module of its own repository its go.mod replaces by a directory (fingerprint.VersionLine), // (fingerprint.HeldVersionsOf), and the content of each mesh Go module of its own repository its go.mod replaces by a
// so a change of that directory alone moves it (review finding 2). Its versions are the other repositories' alone. // directory (fingerprint.VersionLine), so a change of that directory alone moves it (review finding 2). Its versions are the other repositories' alone.
// A package of another repository none of them publishes takes the commit the gate's source names for its repository // A package of another repository none of them publishes takes the commit the gate's source names for its repository
// (heads, design 47 G A-365). // (heads, design 47 G A-365).
// - **An accepted series** (DLV-I6): published and pins come from acceptedSeries, so a mesh Go module of another // - **An accepted series** (DLV-I6): published and pins come from acceptedSeries, so a mesh Go module of another
@@ -51,10 +57,12 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
case fp == "": case fp == "":
return form{none: none} return form{none: none}
case kept.ImportForm == fp && prev.Fp2 != "": case kept.ImportForm == fp && prev.Fp2 != "":
return form{fp: prev.Fp2, versions: prev.Packages} // the build that reported the imports stands // The build that reported the imports stands, and so do its versions.
return form{fp: prev.Fp2, versions: refined(sdkstep.PackageVersions(prev.Packages), r.Versions, module, repository, kept, held, pins, heads),
stands: true, raw: fp, built: prev.Packages}
} }
kept.ImportForm = fp kept.ImportForm = fp
return form{fp: fp, versions: r.Versions} return form{fp: fp, versions: refined(r.Versions, nil, module, repository, kept, held, pins, heads)}
} }
if logf != nil { if logf != nil {
logf("%s: no published package holds %s, so its fingerprint takes the version line", module, strings.Join(r.Missing, ", ")) logf("%s: no published package holds %s, so its fingerprint takes the version line", module, strings.Join(r.Missing, ", "))
@@ -63,6 +71,34 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
// The fingerprint written here is the version line's, not the import form's: no import form is held for the inputs it was // The fingerprint written here is the version line's, not the import form's: no import form is held for the inputs it was
// computed from, so a later pass whose import form computes the same does not take this one for the build's (issue 1966). // computed from, so a later pass whose import form computes the same does not take this one for the build's (issue 1966).
kept.ImportForm = "" kept.ImportForm = ""
versions := lineVersions(module, repository, kept, held, pins, heads, logf)
in.Packages = fingerprint.VersionLine(repository, kept.Packages, versions, published)
fp, none := fingerprint.Of(in)
if fp == "" {
return form{none: none}
}
return form{fp: fp, versions: versions}
}
// refined are an import form's versions: the first of sets that names a package, else, while the module's go.mod
// requires a mesh package of another repository (kept.Packages), the version line's, so a build's reported imports
// refine the go.mod's set and never empty it (design 47 H-A-157); nil when nothing names one.
func refined(versions, then sdkstep.PackageVersions, module, repository string, kept *meshcatalog.ModulesByModuleValueInputs2, held map[string]meshcatalog.ModulesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string) sdkstep.PackageVersions {
switch {
case len(versions) > 0:
return versions
case len(then) > 0:
return then
case len(fingerprint.OtherPackages(kept.Packages, repository)) > 0:
return lineVersions(module, repository, kept, held, pins, heads, nil)
}
return versions
}
// lineVersions are the version line's versions of the mesh Go modules of other repositories the module's go.mod requires
// (kept.Packages): each as the indexer has always read it, an accepted series' pin over it, else the commit the gate's
// source names for its repository.
func lineVersions(module, repository string, kept *meshcatalog.ModulesByModuleValueInputs2, held map[string]meshcatalog.ModulesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string, logf func(string, ...any)) sdkstep.PackageVersions {
others := fingerprint.OtherPackages(kept.Packages, repository) others := fingerprint.OtherPackages(kept.Packages, repository)
versions, unpublished := fingerprint.HeldVersionsOf(others, held) versions, unpublished := fingerprint.HeldVersionsOf(others, held)
for p, v := range pins { // an accepted series' newest published version (acceptedSeries, DLV-I6) for p, v := range pins { // an accepted series' newest published version (acceptedSeries, DLV-I6)
@@ -90,12 +126,7 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
if len(unpublished) > 0 && logf != nil { if len(unpublished) > 0 && logf != nil {
logf("%s: no module the indexer holds publishes %s, so its fingerprint takes no version of it", module, strings.Join(unpublished, ", ")) logf("%s: no module the indexer holds publishes %s, so its fingerprint takes no version of it", module, strings.Join(unpublished, ", "))
} }
in.Packages = fingerprint.VersionLine(repository, kept.Packages, versions, published) return versions
fp, none := fingerprint.Of(in)
if fp == "" {
return form{none: none}
}
return form{fp: fp, versions: versions}
} }
// inputsFrom is the fingerprint's inputs of module m from its kept inputs and the given toolchains, no package of // inputsFrom is the fingerprint's inputs of module m from its kept inputs and the given toolchains, no package of
@@ -118,10 +118,39 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
t.Fatal(err) t.Fatal(err)
} }
moved, _ := moduleOf(t, mods, "avahi") moved, _ := moduleOf(t, mods, "avahi")
if moved.Fp2 == before.Fp2 || moved.Fp2 != moved.Inputs2.ImportForm || moved.Packages[meshSDK] != c3 { if pin, kept := moved.Packages[meshSDK]; moved.Fp2 == before.Fp2 || moved.Fp2 != moved.Inputs2.ImportForm || !kept || pin != "" {
t.Fatalf("a change of an imported package: fp2 %q (was %q), import form %q, versions %v; want the import form at c3", t.Fatalf("a change of an imported package: fp2 %q (was %q), import form %q, versions %v; want the import form, mesh-sdk kept unpinned (none released, H-A-161)",
moved.Fp2, before.Fp2, moved.Inputs2.ImportForm, moved.Packages) moved.Fp2, before.Fp2, moved.Inputs2.ImportForm, moved.Packages)
} }
// mesh-sdk's module releases 0.1.97 while the registry has published nothing of its Go package: the module's release
// number is never the package's pin (H-A-183), so avahi stays unpinned and where it was.
ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.97"), []byte(`{}`))
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "avahi"); m.Fp2 != moved.Fp2 || m.Packages[meshSDK] != "" {
t.Fatalf("after mesh-sdk's module release alone: fp2 %q (was %q), versions %v; want it unpinned where it was", m.Fp2, moved.Fp2, m.Packages)
}
// The registry publishes its Go package at 0.1.20, another version than the module's release: the import form's
// fingerprint takes that pin and moves, once (H-A-172, H-A-183).
if _, err := ix.HearVersion(ctx, HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.1.20", Commit: c3, GoModule: meshSDK, Published: true}); err != nil {
t.Fatal(err)
}
for range 2 {
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
}
if m, _ := moduleOf(t, mods, "avahi"); m.Fp2 == moved.Fp2 || m.Fp2 == "" || m.Packages[meshSDK] != "v0.1.20" || m.Inputs2.ImportForm != moved.Inputs2.ImportForm {
t.Fatalf("after mesh-sdk's release: fp2 %q (was %q), versions %v, import form %q; want a new fingerprint pinned to v0.1.20, the import form kept",
m.Fp2, moved.Fp2, m.Packages, m.Inputs2.ImportForm)
} else if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
} else if again, _ := moduleOf(t, mods, "avahi"); again.Fp2 != m.Fp2 {
t.Fatalf("a pass with nothing moved moved the pinned fingerprint %q to %q", m.Fp2, again.Fp2)
}
} }
// A build of a fingerprint the module no longer holds says nothing of its inputs now: its imports are not kept. // A build of a fingerprint the module no longer holds says nothing of its inputs now: its imports are not kept.
@@ -256,8 +285,8 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
} }
} }
before := step("trunk at c1", trunk(c1)) before := step("trunk at c1", trunk(c1))
if before.Packages[meshSDK] != c1 { if pin, kept := before.Packages[meshSDK]; !kept || pin != "" {
t.Fatalf("with no series published, versions %v, want the commit form c1", before.Packages) t.Fatalf("with no series published, versions %v, want mesh-sdk kept with no pin, never the commit (H-A-161)", before.Packages)
} }
step("the build reports its imports", func() error { step("the build reports its imports", func() error {
return ix.ApplyBuildEnd(ctx, buildEnd(t, "avahi", before.Fp2, meshSDK, meshSDK+"/announce")) return ix.ApplyBuildEnd(ctx, buildEnd(t, "avahi", before.Fp2, meshSDK, meshSDK+"/announce"))
@@ -266,12 +295,19 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
if pinned.Fp2 == before.Fp2 || pinned.Packages[meshSDK] != "v0.1.15" { if pinned.Fp2 == before.Fp2 || pinned.Packages[meshSDK] != "v0.1.15" {
t.Fatalf("after 0.1.15: fp2 %q (was %q), versions %v; want moved by c3's announce, pinned to v0.1.15", pinned.Fp2, before.Fp2, pinned.Packages) t.Fatalf("after 0.1.15: fp2 %q (was %q), versions %v; want moved by c3's announce, pinned to v0.1.15", pinned.Fp2, before.Fp2, pinned.Packages)
} }
// A publish in a series avahi does not accept pins nothing and moves nothing (H-A-184).
if m := step("0.2.0 published from c1", hear("0.2.0", c1)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" { if m := step("0.2.0 published from c1", hear("0.2.0", c1)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a publish of a series avahi does not accept moved it: %q %v", m.Fp2, m.Packages) t.Errorf("a publish of a series avahi does not accept moved it: %q %v", m.Fp2, m.Packages)
} }
if m := step("trunk at c2", trunk(c2)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" { if m := step("trunk at c2", trunk(c2)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages) t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages)
} }
// mesh-sdk's module releases 0.3.0: a module's release number is never a package's pin (H-A-183).
ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.3.0"), []byte(`{}`))
if m := step("0.3.0 released", func() error { return nil }); m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a module's release pinned its package: %v", m.Packages)
}
} }
// Issue 1966 (the indexer started while the bus's storage was down): a read of the packages state that failed, or that // Issue 1966 (the indexer started while the bus's storage was down): a read of the packages state that failed, or that
@@ -370,22 +406,36 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
head := strings.Repeat("c", 40) head := strings.Repeat("c", 40)
contracts := "git.novox.be/novox/mesh-contracts/go/seats/findings" contracts := "git.novox.be/novox/mesh-contracts/go/seats/findings"
held := map[string]meshcatalog.ModulesByModuleValue{"other": {Repository: "novox/mesh-sdk", Commit: strings.Repeat("d", 40), Path: "go"}} held := map[string]meshcatalog.ModulesByModuleValue{"other": {Repository: "novox/mesh-sdk", Commit: strings.Repeat("d", 40), Path: "go"}}
// A build's imports of its own repository alone (design 47 H-A-157): they refine the go.mod's set and never empty it.
own := "git.novox.be/novox/mesh-catalog/modules/avahi/lib"
published := map[string]meshcatalog.PackagesByModuleValue{own: {Module: own, Repository: "novox/mesh-catalog", Commit: head, Requires: "r1",
Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: own, Digest: "d1"}}}}
for _, c := range []struct { for _, c := range []struct {
name string name string
heads map[string]string heads map[string]string
held map[string]meshcatalog.ModulesByModuleValue held map[string]meshcatalog.ModulesByModuleValue
want string imports []string
want string
}{ }{
{"the source names its repository", map[string]string{"novox/mesh-contracts": head}, held, head}, {"the source names its repository", map[string]string{"novox/mesh-contracts": head}, held, nil, head},
{"the source names another repository only", map[string]string{"novox/mesh-sdk": head}, held, ""}, {"the source names another repository only", map[string]string{"novox/mesh-sdk": head}, held, nil, ""},
{"a module the indexer holds publishes it", map[string]string{"novox/mesh-contracts": head}, {"a module the indexer holds publishes it", map[string]string{"novox/mesh-contracts": head},
map[string]meshcatalog.ModulesByModuleValue{"c": {Repository: "novox/mesh-contracts", Commit: strings.Repeat("e", 40), Path: "go/seats/findings"}}, strings.Repeat("e", 40)}, map[string]meshcatalog.ModulesByModuleValue{"c": {Repository: "novox/mesh-contracts", Commit: strings.Repeat("e", 40), Path: "go/seats/findings"}}, nil, strings.Repeat("e", 40)},
{"a build's imports naming none of another repository keep the go.mod's", map[string]string{"novox/mesh-contracts": head}, held, []string{own}, head},
} { } {
kept := meshcatalog.ModulesByModuleValueInputs2{Tree: "t", Packages: []string{contracts}} kept := meshcatalog.ModulesByModuleValueInputs2{Tree: "t", Packages: []string{contracts}, Imports: c.imports}
f := formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{}, c.held, nil, nil, c.heads, nil) f := formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{}, c.held, published, nil, c.heads, nil)
if got := f.versions[contracts]; got != c.want { if got := f.versions[contracts]; got != c.want {
t.Errorf("%s: version of %s = %q, want %q", c.name, contracts, got, c.want) t.Errorf("%s: version of %s = %q, want %q", c.name, contracts, got, c.want)
} }
if c.imports == nil {
continue
}
// The build that reported them stands, its held versions empty (a version line read from go.sum).
f = formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{Fp2: "src2:" + strings.Repeat("b", 64)}, c.held, published, nil, c.heads, nil)
if got := f.versions[contracts]; got != c.want || f.fp != "src2:"+strings.Repeat("b", 64) {
t.Errorf("%s, the build standing: version of %s = %q (fp %q), want %q", c.name, contracts, got, f.fp, c.want)
}
} }
}) })
t.Run("formOf clears the import form when it writes the version line", func(t *testing.T) { t.Run("formOf clears the import form when it writes the version line", func(t *testing.T) {
@@ -10,6 +10,7 @@ import (
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/views" "git.novox.be/novox/mesh-sdk/go/views"
) )
@@ -94,7 +95,7 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
} }
// A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held. // A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held.
listed = objects.n.Load() listed = objects.n.Load()
if raw, err := kept.Manifest(meshcatalog.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` { if raw, err := kept.Manifest(meshcatalogclient.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err) t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err)
} }
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 { if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 {
@@ -63,6 +63,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
return w, err return w, err
} }
objects, namedErr := ix.namedObjects(ctx, f) objects, namedErr := ix.namedObjects(ctx, f)
trees := ix.goModTree(ctx, f)
overlay := map[string]meshcatalog.ModulesByModuleValue{} overlay := map[string]meshcatalog.ModulesByModuleValue{}
dirOf := map[string]string{} dirOf := map[string]string{}
named, every := map[string]bool{}, true // every module the message's manifests name, and whether each names one named, every := map[string]bool{}, true // every module the message's manifests name, and whether each names one
@@ -108,8 +109,24 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
if next.Manifest == "" { if next.Manifest == "" {
next.None2 = "the manifest cannot be read: it cannot be kept as a value" next.None2 = "the manifest cannot be read: it cannot be kept as a value"
} else { } else {
packages := ix.goSumPackages(ctx, f, dir, cur, has) packages := ix.goPackages(ctx, f, dir, cur, has)
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
// The entry pins every mesh package of another repository the module's build requires, transitively across
// the same-repository modules of its graph (design 47 H-A-201).
packages = unionSorted(packages, trees.otherRequired(goModuleDirs(m, dir)))
}
in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx)) in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx))
if in.Kept != nil {
// A Go build resolves the same-repository modules its go.mod files require, transitively, from the tree: their
// directories' objects join the tree input of every form, never the entry's paths (design 47 H-A-202, H-A-172).
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
if tree, none := ix.derivedTree(ctx, f, trees, m, dir, in.Kept.Tree); none != "" {
in.Kept, in.None2 = nil, none
} else {
in.Kept.Tree = tree
}
}
}
next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths
if in.Kept != nil && has && cur.Inputs2 != nil { if in.Kept != nil && has && cur.Inputs2 != nil {
// The imports its last trunk build reported stand until its next build reports again (ADR 0593 §2, §4). // The imports its last trunk build reported stand until its next build reports again (ADR 0593 §2, §4).
@@ -365,79 +382,72 @@ func unreadInputs(tree string, locks, paths map[string]string, packages []string
return in return in
} }
// goSumPackages are the version line's packages of the merge's directory dir (ADR 0573 §2, ADR 0593 §2): the mesh Go // goPackages are the version line's packages of the merge's directory dir (ADR 0593 §2; design 47 H-A-157): the mesh Go
// modules of other repositories its go.sum names, and those of its own repository its go.mod replaces by a directory // modules of other repositories its go.mod requires, at any version, and those of its own repository its go.mod replaces
// (siblingPackages), which no go.sum line names. A directory with no go.sum is no Go module and has none. // by a directory (siblingPackages, H-A-136's sibling rule). A directory with no go.sum is no Go module and has none.
func (ix *Indexer) goSumPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string { func (ix *Indexer) goPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string {
others := ix.otherPackages(ctx, f, dir, cur, has)
if f.Locks[dir]["go.sum"] == "" && len(f.Packages[dir]) == 0 { if f.Locks[dir]["go.sum"] == "" && len(f.Packages[dir]) == 0 {
return others return nil
}
ix.requirementsOf(ctx, f, dir, cur, has)
goMod, read := ix.goModOf(ctx, f, dir)
others := fingerprint.OtherPackages(f.Packages[dir], f.Repository) // carried: what the go.mod requires, read with the tree
switch {
case len(others) > 0:
case read:
others = fingerprint.OtherPackages(fingerprint.MeshRequiresOf(goMod), f.Repository)
case has && cur.Inputs2 != nil:
others = fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
var siblings []string
if read {
siblings = fingerprint.ReplacedSiblings(goMod, f.Repository)
} else if has && cur.Inputs2 != nil {
for _, p := range cur.Inputs2.Packages {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, f.Repository) {
siblings = append(siblings, p)
}
}
} }
siblings := ix.siblingPackages(ctx, f, dir, cur, has)
if len(siblings) == 0 { if len(siblings) == 0 {
return others return others
} }
return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(others), siblings...)))) return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(others), siblings...))))
} }
// siblingPackages are the mesh Go modules of the merge's own repository the go.mod of dir replaces by a directory, read // goModOf is the go.mod of the merge's directory dir, read from the forge (GoMod), and whether it was read; where it
// from the forge (GoMod); where it cannot be read, the held module's own-repository packages stand, said. // cannot be, the held module's packages stand, said.
func (ix *Indexer) siblingPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string { func (ix *Indexer) goModOf(ctx context.Context, f whole, dir string) ([]byte, bool) {
held := func() []string {
if !has || cur.Inputs2 == nil {
return nil
}
var out []string
for _, p := range cur.Inputs2.Packages {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, f.Repository) {
out = append(out, p)
}
}
return out
}
if ix.GoMod == nil { if ix.GoMod == nil {
return held() return nil, false
} }
raw, err := ix.GoMod(ctx, f.Repository, f.Merge, dir) raw, err := ix.GoMod(ctx, f.Repository, f.Merge, dir)
if err != nil { if err != nil {
ix.logf("%s at %s: the go.mod of %q cannot be read, so its module keeps the directory replaces it held: %v", f.Repository, f.Merge, dir, err) ix.logf("%s at %s: the go.mod of %q cannot be read, so its module keeps the packages it held: %v", f.Repository, f.Merge, dir, err)
return held() return nil, false
} }
return fingerprint.ReplacedSiblings(raw, f.Repository) return raw, true
} }
// otherPackages are the mesh Go modules of other repositories the go.sum of the merge's directory dir names: those the // requirementsOf keeps what the go.sum of the merge's directory dir requires in the requirements state
// message carries (an older bridge), else the held module's while its go.sum's hash is unchanged, else the go.sum read // (keepRequirements): read from the forge (GoSum) when its hash moved, or this process has not kept it yet, whatever
// from the forge (GoSum). // packages the reader carried. It gives no package: those are the go.mod's (H-A-157).
func (ix *Indexer) otherPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string { func (ix *Indexer) requirementsOf(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) {
if carried := f.Packages[dir]; len(carried) > 0 {
return fingerprint.OtherPackages(carried, f.Repository) // its own repository's come with its tree
}
hash := f.Locks[dir]["go.sum"] hash := f.Locks[dir]["go.sum"]
if hash == "" { if hash == "" || ix.GoSum == nil {
return nil return
} }
if has && cur.Inputs2 != nil && trunk.SameRepository(cur.Repository, f.Repository) && ix.requirementsKept(f.Repository, dir) { if has && cur.Inputs2 != nil && trunk.SameRepository(cur.Repository, f.Repository) && ix.requirementsKept(f.Repository, dir) {
for _, h := range cur.Inputs2.Locks { for _, h := range cur.Inputs2.Locks {
if h == hash { if h == hash {
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository) return
} }
} }
} }
if ix.GoSum == nil {
if has && cur.Inputs2 != nil {
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
return nil
}
raw, err := ix.GoSum(ctx, f.Repository, f.Merge, dir) raw, err := ix.GoSum(ctx, f.Repository, f.Merge, dir)
if err != nil { if err != nil {
ix.logf("%s at %s: the go.sum of %q cannot be read, so its module keeps the packages it held: %v", f.Repository, f.Merge, dir, err) ix.logf("%s at %s: the go.sum of %q cannot be read, so what it requires is not kept: %v", f.Repository, f.Merge, dir, err)
if has && cur.Inputs2 != nil { return
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
return nil
} }
ix.keepRequirements(ctx, f.Repository, dir, raw) ix.keepRequirements(ctx, f.Repository, dir, raw)
return fingerprint.OtherPackages(fingerprint.MeshPackagesOf(raw), f.Repository)
} }
@@ -401,8 +401,36 @@ func TestARootModuleHasTheDirectoryDot(t *testing.T) { // 1615/AC3: empty means
} }
func TestAMergeAddingAModuleDirectoryWritesItsValueWithNoBuildRecordAnywhere(t *testing.T) { // I1: issue 1851, ADR 0501 clause 6 func TestAMergeAddingAModuleDirectoryWritesItsValueWithNoBuildRecordAnywhere(t *testing.T) { // I1: issue 1851, ADR 0501 clause 6
ix, mods, _ := manifestsIndexer(t) // no build record of any module
avahi := catalogueManifest(t, "avahi") avahi := catalogueManifest(t, "avahi")
// Its go.mod requires mesh-sdk at the zero version (ADR 0573 §2): the go.mod's require lines are its mesh packages, with
// no build, carried by the reader or read through GoMod (design 47 H-A-157). Its go.sum is still read for the
// requirements state, carried packages or not, so an older version it names is kept there (and cleared when gone).
for _, carried := range [][]string{nil, {meshSDK}} {
ix, mods, _ := manifestsIndexer(t)
ix.Requirements = newMem()
ix.GoMod = func(context.Context, string, string, string) ([]byte, error) {
return []byte("module avahi\n\nrequire " + meshSDK + " v0.0.0-00010101000000-000000000000\n"), nil
}
ix.GoSum = func(context.Context, string, string, string) ([]byte, error) {
return []byte(meshSDK + " v0.1.14 h1:x=\n"), nil
}
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi},
map[string]map[string]string{"modules/avahi": {"go.sum": avahiGoSum}})
if carried != nil {
f.Packages = map[string][]string{"modules/avahi": carried}
}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "avahi")
if m.Inputs2 == nil || !slices.Equal(m.Inputs2.Packages, []string{meshSDK}) {
t.Errorf("carried %v: mesh packages %+v, want mesh-sdk, which its go.mod requires at the zero version", carried, m.Inputs2)
}
if _, kept := ix.Requirements.(*memStore).vals[meshSDK]; !kept {
t.Errorf("carried %v: the requirements state holds no %s, which the go.sum names", carried, meshSDK)
}
}
ix, mods, _ := manifestsIndexer(t) // no build record of any module
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi},
map[string]map[string]string{"modules/avahi": {"go.sum": avahiGoSum}}) map[string]map[string]string{"modules/avahi": {"go.sum": avahiGoSum}})
if err := ix.Apply(context.Background(), f); err != nil { if err := ix.Apply(context.Background(), f); err != nil {
@@ -0,0 +1,196 @@
package index
import (
"context"
"fmt"
"path"
"path/filepath"
"slices"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// goModTree reads the go.mod files of one merge's tree, each directory once (Indexer.GoMod), for the walk of the module
// graph a Go build resolves from the tree (design 47 H-A-202, H-A-157).
type goModTree struct {
ix *Indexer
ctx context.Context
f whole
read map[string][]byte // by directory from the repository's root; nil for none
}
func (ix *Indexer) goModTree(ctx context.Context, f whole) *goModTree {
return &goModTree{ix: ix, ctx: ctx, f: f, read: map[string][]byte{}}
}
// goMod is the go.mod of the directory dir (from the root, "" for the root) at the merge commit; nil when there is none
// or it cannot be read, said.
func (t *goModTree) goMod(dir string) []byte {
if raw, ok := t.read[dir]; ok {
return raw
}
var raw []byte
if t.ix.GoMod != nil {
var err error
if raw, err = t.ix.GoMod(t.ctx, t.f.Repository, t.f.Merge, dir); err != nil {
t.ix.logf("%s at %s: the go.mod of %q cannot be read, so no module of the tree is found there: %v", t.f.Repository, t.f.Merge, dir, err)
raw = nil
}
}
t.read[dir] = raw
return raw
}
// sameRepositoryDir is the directory of the merge's tree that holds the Go module required: the required path starts with
// the repository's module prefix, and the directory it names holds a go.mod declaring that path (H-A-198, H-A-202). false
// for a module of another repository, or one the tree holds no go.mod of.
func (t *goModTree) sameRepositoryDir(required string) (string, bool) {
repo, dir, ok := trunk.PackageRepository(required)
if !ok || dir == "" || !trunk.SameRepository(repo, t.f.Repository) {
return "", false
}
if goModModule(t.goMod(dir)) != required {
return "", false
}
return dir, true
}
// sameRepositoryRequired are the directories (from the root, sorted, each once) of every same-repository Go module the
// go.mod of each start directory requires, transitively through those modules' own go.mod files (H-A-202, H-A-157). The
// start directories themselves are not among them.
func (t *goModTree) sameRepositoryRequired(start []string) []string {
seen := map[string]bool{}
for _, s := range start {
seen[s] = true
}
queue := slices.Clone(start)
var out []string
for len(queue) > 0 {
dir := queue[0]
queue = queue[1:]
for _, req := range fingerprint.MeshRequiresOf(t.goMod(dir)) {
d, ok := t.sameRepositoryDir(req)
if !ok || seen[d] {
continue
}
seen[d] = true
out = append(out, d)
queue = append(queue, d)
}
}
slices.Sort(out)
return out
}
// goModuleDirs are the directories (from the root) a module's Go build starts from: its own directory and the directory
// of each path of each Go package artifact of its manifest m.
func goModuleDirs(m catalogue.Manifest, dir string) []string {
out := []string{dir}
if m.Build == nil {
return out
}
for _, a := range m.Build.Artifacts {
if a.Kind != catalogue.ArtifactPackage || a.Language != "go" {
continue
}
for _, p := range a.Paths {
if d := path.Join(dir, strings.TrimSuffix(filepath.ToSlash(p), "/")); !slices.Contains(out, d) {
out = append(out, d)
}
}
}
return out
}
// otherRequired are the mesh packages of other repositories (sorted, each once) that the go.mod of each start directory,
// and of every same-repository module those require transitively (sameRepositoryRequired), requires at any version, the
// zero version included: every one the module's build resolves, which the entry pins (design 47 H-A-201, H-A-183,
// H-A-157). A same-repository module is none: it is built from the tree (H-A-198, H-A-202). nil for none.
func (t *goModTree) otherRequired(start []string) []string {
var out []string
for _, d := range append(slices.Clone(start), t.sameRepositoryRequired(start)...) {
out = append(out, fingerprint.OtherPackages(fingerprint.MeshRequiresOf(t.goMod(d)), t.f.Repository)...)
}
if len(out) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(out)))
}
// unionSorted is a and b together, sorted and each once; nil for none.
func unionSorted(a, b []string) []string {
if len(a)+len(b) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(a), b...))))
}
// derivedDirs are the same-repository module directories (from the root, sorted) the module's Go build resolves from the
// tree beyond its own tree input (H-A-202): every one its go.mod files require, transitively, that its tree input does
// not hold already, which is the module's directory for the directory's tree, and its named paths for the named tree.
func (t *goModTree) derivedDirs(m catalogue.Manifest, dir string) []string {
var held []string
if paths, every := m.NamedPaths(); every {
for _, p := range paths {
held = append(held, path.Join(dir, filepath.ToSlash(p)))
}
} else {
held = []string{dir}
}
var out []string
for _, d := range t.sameRepositoryRequired(goModuleDirs(m, dir)) {
if !slices.ContainsFunc(held, func(h string) bool { return h == "" || h == "." || d == h || strings.HasPrefix(d, h+"/") }) {
out = append(out, d)
}
}
return out
}
// DerivedLine opens each line of the tree input naming a derived directory and its git object (H-A-202, H-A-172).
const DerivedLine = "derived "
// derivedTree is the module's tree input with a line per derived directory and its object appended, sorted by directory
// (H-A-202, H-A-172: what a build reads moves its fingerprint), so a change of such a directory moves the fingerprint of
// every form. The objects are asked of the forge at the merge commit by their paths from the root (Named); an answer that
// lacks one gives the reason the module has no inputs, naming them. A module with no derived directory keeps its tree.
func (ix *Indexer) derivedTree(ctx context.Context, f whole, t *goModTree, m catalogue.Manifest, dir, tree string) (string, string) {
dirs := t.derivedDirs(m, dir)
if len(dirs) == 0 {
return tree, ""
}
if ix.Named == nil {
return "", "the forge cannot be asked the git objects of the derived directories " + strings.Join(dirs, ", ")
}
got, err := ix.Named(ctx, f.Repository, f.Merge, map[string][]string{"": dirs})
if err != nil {
return "", fmt.Sprintf("the git objects of the derived directories %s: %v", strings.Join(dirs, ", "), err)
}
lines := []string{tree}
var missing []string
for _, d := range dirs {
o := got[""][d]
if o == "" {
missing = append(missing, d)
continue
}
lines = append(lines, DerivedLine+d+" "+o)
}
if len(missing) > 0 {
return "", "the forge names no git object of the derived directories " + strings.Join(missing, ", ")
}
return strings.Join(lines, "\n"), ""
}
// goModModule is the module path a go.mod declares; "" for none.
func goModModule(raw []byte) string {
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) == 2 && f[0] == "module" {
return strings.Trim(f[1], `"`)
}
}
return ""
}
@@ -0,0 +1,144 @@
package index
import (
"context"
"strings"
"testing"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// A fixture repository novox/mesh-delivery: the Go package state of modules/g requires the same-repository module
// modules/b/builds at the zero version, which requires modules/c/client in turn; mesh-sdk is of another repository, and
// modules/b/gone names a directory the tree holds no go.mod of.
const ptpZero = " v0.0.0-00010101000000-000000000000"
var ptpGoMods = map[string]string{
"modules/g/state": "module git.novox.be/novox/mesh-delivery/modules/g/state\n\nrequire (\n\tgit.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero +
"\n\tgit.novox.be/novox/mesh-sdk/go" + ptpZero + "\n\tgit.novox.be/novox/mesh-delivery/modules/b/gone" + ptpZero + "\n)\n",
"modules/b/builds": "module git.novox.be/novox/mesh-delivery/modules/b/builds\n\nrequire git.novox.be/novox/mesh-delivery/modules/c/client" + ptpZero + " // indirect\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n",
}
// ptpProgram is modules/g's manifest: a program naming no paths beside the package state.
const ptpProgram = `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"program","kind":"bundle","language":"go","system":"arch","from":"cmd/g","binary":"g","loads":["g"]},` +
`{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
// ptpIndexer answers the fixture's go.mod files and, for a path asked from the root, the object objects names for it.
func ptpIndexer(t *testing.T, objects map[string]string) (*Indexer, *memStore, *[]map[string][]string) {
ix, mods, _ := manifestsIndexer(t)
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := ptpGoMods[dir]; ok {
return []byte(c), nil
}
return nil, nil
}
var asked []map[string][]string
ix.Named = func(_ context.Context, _, _ string, paths map[string][]string) (map[string]map[string]string, error) {
asked = append(asked, paths)
out := map[string]map[string]string{}
for dir, ps := range paths {
out[dir] = map[string]string{}
for _, p := range ps {
out[dir][p] = objects[p]
}
}
return out, nil
}
return ix, mods, &asked
}
// Design 47 H-A-202, H-A-172: the same-repository modules a Go build resolves from the tree, transitively, join the tree
// input of every form with their objects, so a change of one moves the fingerprint of a module whose program names no
// paths; the entry's paths stay the manifest's.
func TestADerivedDirectoryMovesTheFingerprintOfEveryForm(t *testing.T) {
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
tree := func(objects map[string]string) string {
ix, mods, asked := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram}, nil)
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if got := strings.Join(m.Paths["state"], ","); got != "state" {
t.Errorf("the entry's paths of state are %q, want the manifest's", got)
}
if len(*asked) != 1 || strings.Join((*asked)[0][""], ",") != "modules/b/builds,modules/c/client" {
t.Fatalf("asked the forge %v, want the derived directories from the root once", *asked)
}
if m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, "gtree\n") {
t.Fatalf("inputs = %+v (%s), want the directory's tree with the derived lines", m.Inputs2, m.None2)
}
return m.Inputs2.Tree
}
before := tree(objects)
if want := "gtree\nderived modules/b/builds " + objects["modules/b/builds"] + "\nderived modules/c/client " + objects["modules/c/client"]; before != want {
t.Errorf("the tree input is %q, want %q", before, want)
}
objects["modules/c/client"] = strings.Repeat("d", 40)
after := tree(objects)
fp := func(tree string) string {
v, _ := fingerprint.Of(fingerprint.Inputs{Module: "g", Tree: tree})
return v
}
if fp(before) == fp(after) {
t.Errorf("a change of modules/c/client left the fingerprint %s", fp(after))
}
// The named tree takes them too: a module whose every artifact names its paths.
named := `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
ix, mods, _ := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": named}, nil)
f.Paths = map[string]map[string]string{"modules/g": {"state": strings.Repeat("1", 40)}}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, fingerprint.NamedTreePrefix) ||
!strings.HasSuffix(m.Inputs2.Tree, "\nderived modules/c/client "+objects["modules/c/client"]) || strings.Join(m.Paths["state"], ",") != "state" {
t.Errorf("module = %+v, want the named tree with the derived lines and the manifest's paths", m)
}
// A forge that names no object of a derived directory gives the module none, naming it.
ix, mods, _ = ptpIndexer(t, map[string]string{"modules/b/builds": strings.Repeat("b", 40)})
if err := ix.Apply(context.Background(), withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}),
map[string]string{"modules/g": ptpProgram}, nil)); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Fp2 != "" || !strings.Contains(m.None2, "modules/c/client") {
t.Errorf("module = %+v, want none naming the directory the forge gave no object of", m)
}
}
// Design 47 H-A-201: the entry pins every mesh package of another repository the module's build requires, across the
// same-repository modules of its graph: here the one modules/c/client requires, which modules/g's go.mod never names.
// A module of the module's own repository is never pinned, whatever a build reported (H-A-198, H-A-202).
func TestTheEntryPinsEveryRequirementOfTheModuleGraph(t *testing.T) {
const conditions = "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
ix, mods, _ := ptpIndexer(t, objects)
mod := map[string]string{
"modules/g": "module g\n\nrequire git.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero + "\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n\nrequire " + conditions + ptpZero + "\n",
}
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := mod[dir]; ok {
return []byte(c), nil
}
return []byte(ptpGoMods[dir]), nil
}
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram},
map[string]map[string]string{"modules/g": {"go.sum": avahiGoSum}})
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if m.Inputs2 == nil || strings.Join(m.Inputs2.Packages, ",") != conditions+",git.novox.be/novox/mesh-sdk/go" {
t.Errorf("packages = %+v, want the client's requirement and state's mesh-sdk, and no module of the repository", m.Inputs2)
}
builds := "git.novox.be/novox/mesh-delivery/modules/b/builds"
pins := releasedPins(sdkstep.PackageVersions{builds: "v0.1.106", conditions: "v0.1.4"}, nil, catalogue.Manifest{}, "novox/mesh-delivery")
if _, pinned := pins[builds]; pinned || len(pins) != 1 {
t.Errorf("pins = %v, want %s only, never a module of the repository", pins, conditions)
}
}
@@ -14,14 +14,15 @@ import (
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient" mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/keys" "git.novox.be/novox/mesh-sdk/go/keys"
) )
// The indexer keeps the records pinning each catalog graph revision and removes a revision only on an event (design 47 G §153, // The indexer keeps the records pinning each catalog graph revision and removes a revision only on an event (design 47 G §153,
// §178, §189, §215, §216, §579, A-169, A-171): its state pins holds one key per revision, `<branch>.<revision>` (the watched branch // §178, §189, §215, §216, §579, A-169, A-171): its state pins holds one key per revision, `<branch>.<revision>` (the watched branch
// and the revision's number, the branch escaped as a name is), whose value lists the records that pin it: a take-in's and each // and the revision's number, the branch escaped as a name is), whose value lists the records that pin it: a take-in's, each
// running pipeline's (design 47 H §743, §749). // running pipeline's (design 47 H §743, §749) and each passed pipeline's release until a newer one is the want (H-A-168, H-A-170).
const ( const (
// PinsState is the state's name; PinsCapMiB is its cap in the manifest (about 1,800 keys of about 0.2 KiB). // PinsState is the state's name; PinsCapMiB is its cap in the manifest (about 1,800 keys of about 0.2 KiB).
PinsState = "pins" PinsState = "pins"
@@ -84,6 +85,19 @@ func (ix *Indexer) HeardPins(ctx context.Context, started *runner.PipelineStarte
case started != nil && started.Revision > 0 && started.Pipeline != "": case started != nil && started.Revision > 0 && started.Pipeline != "":
return ix.pin(ctx, PinKey(started.Branch, started.Revision), PipelineRecord(started.Pipeline)) return ix.pin(ctx, PinKey(started.Branch, started.Revision), PipelineRecord(started.Pipeline))
case ended != nil && ended.Revision > 0 && ended.Pipeline != "": case ended != nil && ended.Revision > 0 && ended.Pipeline != "":
// A passed pipeline's release pins the revision its movers' entries live in, before the pipeline's own pin is freed,
// one record per mover that released a version in it (H-A-168, H-A-170, H-A-173); a mover that released none, and a run
// on request's pipeline, which delivers nothing, pin none.
if ended.Outcome == runner.PipelineEndedOutcomePassed && ended.Run == "" {
for _, m := range ended.Movers {
if m.Version == "" {
continue
}
if err := ix.pin(ctx, PinKey(ended.Branch, ended.Revision), ReleaseRecord(ended.Branch, m.Module)); err != nil {
return err
}
}
}
record := PipelineRecord(ended.Pipeline) record := PipelineRecord(ended.Pipeline)
if err := ix.unpin(ctx, PinKey(ended.Branch, ended.Revision), record); err != nil { if err := ix.unpin(ctx, PinKey(ended.Branch, ended.Revision), record); err != nil {
return err return err
@@ -93,6 +107,77 @@ func (ix *Indexer) HeardPins(ctx context.Context, started *runner.PipelineStarte
return nil return nil
} }
// ReleaseRecord is the record a passed pipeline's release pins its revision under, one per mover: `release.<branch>.<module>`,
// each name escaped as a key's name is (design 47 H-A-168, H-A-170).
func ReleaseRecord(branch, module string) string {
return "release." + keys.EscapeName(branch) + "." + keys.EscapeName(module)
}
// HeardPipelineWanted applies one pipeline-wanted of the seat mesh-wants, said when a pipeline's wants are written to the end
// (design 47 H §751, H-A-168, H-A-170, H-A-173, H-A-174). Outcome running: a module whose release record the pipeline's own
// kept revision holds released its newer version there, and that is the want everywhere, so the module's record is freed from
// every revision of the branch below it; a mover the revision holds no release of (superseded, no version released) keeps its
// older release's revision. Outcome tier-late frees nothing: its later tiers' wants are not written, though their release
// records are held, and the same pipeline's later running frees them. A pipeline whose revision is not kept frees nothing and
// says so; one applied twice changes nothing.
func (ix *Indexer) HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error {
if ix.Pins == nil || ix.Revisions == nil {
return errors.New("the indexer keeps no state " + PinsState)
}
if outcome != meshwants.PipelineWantedOutcomeRunning {
ix.logf("pins: pipeline-wanted of %s %d is %s: it frees no release until its wants are written everywhere (H-A-174)", branch, source, outcome)
return nil
}
name, err := revisionOfSource(ctx, ix.Revisions, branch, source)
if err != nil {
ix.logf("pins: pipeline-wanted of %s %d frees no release: its revision is not kept: %v", branch, source, err)
return nil
}
rev, _, err := ReadRevision(ctx, ix.Revisions, name)
if err != nil {
return fmt.Errorf("%s: %w", name, err)
}
own, _, found, err := ix.Pins.Get(ctx, PinKey(branch, rev.Revision))
if err != nil {
return err
}
var held PinValue
if found {
_ = json.Unmarshal(own, &held)
}
var released []meshcatalog.TakeInEntry
for _, m := range rev.Movers {
if slices.Contains(held.Records, ReleaseRecord(branch, m.Module)) {
released = append(released, m)
}
}
pinKeys, err := ix.Pins.Keys(ctx)
if err != nil {
return err
}
for _, k := range pinKeys {
b, r, ok := ParsePinKey(k)
if !ok || b != branch || r >= rev.Revision {
continue
}
for _, m := range released {
v, _, found, err := ix.Pins.Get(ctx, k)
if err != nil {
return err
}
var pv PinValue
record := ReleaseRecord(branch, m.Module)
if !found || json.Unmarshal(v, &pv) != nil || !slices.Contains(pv.Records, record) {
continue
}
if err := ix.unpin(ctx, k, record); err != nil {
return err
}
}
}
return nil
}
// unpinElsewhere frees a record on every other key that lists it: the branch's previous revision the take-in pinned for the // unpinElsewhere frees a record on every other key that lists it: the branch's previous revision the take-in pinned for the
// same pipeline (H-A-147), which the pipeline-ended does not name. // same pipeline (H-A-147), which the pipeline-ended does not name.
func (ix *Indexer) unpinElsewhere(ctx context.Context, record string) error { func (ix *Indexer) unpinElsewhere(ctx context.Context, record string) error {
@@ -10,6 +10,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
) )
// Design 47 H §749 (G §579, A-169, A-171): the indexer pins a revision on a pipeline-started that names it and frees it on its // Design 47 H §749 (G §579, A-169, A-171): the indexer pins a revision on a pipeline-started that names it and frees it on its
@@ -92,6 +93,55 @@ func TestTheIndexerRemovesARevisionOnlyWhenNoRecordPinsItAndItIsNotCurrent(t *te
if keys, _ := ix.Pins.Keys(ctx); slices.Contains(keys, "main.1") { if keys, _ := ix.Pins.Keys(ctx); slices.Contains(keys, "main.1") {
t.Errorf("the state holds %v after the pin drop, want no main.1", keys) t.Errorf("the state holds %v after the pin drop, want no main.1", keys)
} }
// H-A-168, H-A-170, H-A-173: a passed pipeline's release keeps its revision past a later failed pipeline; a passed one whose
// mover released no version pins nothing and its pipeline-wanted frees nothing; the next one that released a version frees
// the older release on its pipeline-wanted running (its wants written to the end), not on a tier-late one (H-A-174), and the
// revision goes.
mover := []meshcatalog.TakeInEntry{{Module: "reader", Commit: "c", Fingerprint: "f", Repository: "novox/r"}}
take := func(source int64) {
t.Helper()
e := reader
e.Commit = fmt.Sprintf("r%d", source)
if err := ix.KeepBranchEntries(ctx, "main", source, map[string]meshcatalog.CatalogEntry{"reader": e}, mover); err != nil {
t.Fatal(err)
}
}
end := func(source int64, outcome, version string) {
t.Helper()
e := runner.PipelineEnded{Branch: "main", Pipeline: fmt.Sprintf("main.%d", source), Revision: source, Source: source, Outcome: outcome,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: version}}}
if err := ix.HeardPins(ctx, nil, &e); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
}
wanted := func(source int64, outcome string) {
t.Helper()
if err := ix.HeardPipelineWanted(ctx, "main", source, outcome); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
}
take(10)
end(10, runner.PipelineEndedOutcomePassed, "0.1.10")
take(11)
end(11, runner.PipelineEndedOutcomeFailed, "")
want("a passed release past a later failed pipeline", "main.10", "main.11")
take(12)
end(12, runner.PipelineEndedOutcomePassed, "")
wanted(12, meshwants.PipelineWantedOutcomeRunning)
want("a passed pipeline releasing no version frees no older release", "main.10", "main.12")
take(13)
end(13, runner.PipelineEndedOutcomePassed, "0.1.13")
want("two releases, the unreleased one between them gone", "main.10", "main.13")
wanted(13, meshwants.PipelineWantedOutcomeTierLate) // H-A-174: its later tiers' wants are not written, so it frees nothing
want("a tier-late pipeline-wanted", "main.10", "main.13")
wanted(13, meshwants.PipelineWantedOutcomeRunning)
wanted(13, meshwants.PipelineWantedOutcomeRunning) // heard twice: changes nothing more
want("the newer release written everywhere", "main.13")
if v, _, _, _ := ix.Pins.Get(ctx, "main.13"); !slices.Equal(mustRecords(t, v), []string{ReleaseRecord("main", "reader")}) {
t.Errorf("the pin of main.13 holds %s, want its release alone", v)
}
} }
func mustRecords(t *testing.T, v []byte) []string { func mustRecords(t *testing.T, v []byte) []string {
@@ -107,7 +157,8 @@ func mustRecords(t *testing.T, v []byte) []string {
// H-A-147 (§743, §749, H-A-108 (3)): a take-in with a mover pins, in its write, the branch's previous revision (the newest kept // H-A-147 (§743, §749, H-A-108 (3)): a take-in with a mover pins, in its write, the branch's previous revision (the newest kept
// before the write) for the same pipeline record as its own, so the validate step's case before can read it while the pipeline // before the write) for the same pipeline record as its own, so the validate step's case before can read it while the pipeline
// runs, and names it as its base on the revision it writes (H-A-151, H-A-154); a take-in with no revision before it pins only // runs, and names it as its base on the revision it writes (H-A-151, H-A-154); a take-in with no revision before it pins only
// its own and names no base; the pipeline's end frees both. // its own and names no base; the pipeline's end frees both. A passed pipeline's end pins its revision under its mover's release
// record instead (H-A-168, H-A-170), so it stays when a later take-in makes it no longer current.
func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) { func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
ctx := context.Background() ctx := context.Background()
reader := meshcatalog.CatalogEntry{Module: "reader", Repository: "novox/r", Manifest: map[string]any{}} reader := meshcatalog.CatalogEntry{Module: "reader", Repository: "novox/r", Manifest: map[string]any{}}
@@ -117,11 +168,21 @@ func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
before []int64 // sources taken in with no mover before the take-in of source 5 before []int64 // sources taken in with no mover before the take-in of source 5
whileRuns []string whileRuns []string
afterTheEnd []string afterTheEnd []string
base int64 // the base main.5 names (H-A-154) base int64 // the base main.5 names (H-A-154)
passed bool // main.5's pipeline passed: its release pins main.5
unreleased bool // its mover released no version there, so nothing pins main.5 (H-A-173)
then []int64 // sources taken in with no mover after the end
released []string
}{ }{
{name: "the previous revision kept and pinned", before: []int64{3, 4}, whileRuns: []string{"main.4", "main.5"}, afterTheEnd: []string{"main.5"}, base: 4}, {name: "the previous revision kept and pinned", before: []int64{3, 4}, whileRuns: []string{"main.4", "main.5"}, afterTheEnd: []string{"main.5"}, base: 4},
{name: "the newest kept before, whatever its number", before: []int64{2}, whileRuns: []string{"main.2", "main.5"}, afterTheEnd: []string{"main.5"}, base: 2}, {name: "the newest kept before, whatever its number", before: []int64{2}, whileRuns: []string{"main.2", "main.5"}, afterTheEnd: []string{"main.5"}, base: 2},
{name: "no revision before it pins only its own", whileRuns: []string{"main.5"}, afterTheEnd: []string{"main.5"}}, {name: "no revision before it pins only its own", whileRuns: []string{"main.5"}, afterTheEnd: []string{"main.5"}},
{name: "a passed pipeline's release keeps its revision once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, passed: true, then: []int64{6}, afterTheEnd: []string{"main.5", "main.6"}, released: []string{"main.5"}},
{name: "a passed pipeline whose mover released no version pins nothing", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, passed: true, unreleased: true, then: []int64{6}, afterTheEnd: []string{"main.6"}},
{name: "a failed pipeline's revision goes once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, then: []int64{6}, afterTheEnd: []string{"main.6"}},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Pins: newMem(), Findings: newFindings(), ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Pins: newMem(), Findings: newFindings(),
@@ -156,17 +217,32 @@ func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
t.Errorf("the pin of %s holds %s, want the pipeline's record", k, v) t.Errorf("the pin of %s holds %s, want the pipeline's record", k, v)
} }
} }
if err := ix.HeardPins(ctx, nil, &runner.PipelineEnded{Branch: "main", Pipeline: "main.5", Revision: 5}); err != nil { end := runner.PipelineEnded{Branch: "main", Pipeline: "main.5", Revision: 5, Outcome: runner.PipelineEndedOutcomeFailed,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: "0.1.5"}}}
if tc.unreleased {
end.Movers[0].Version = ""
}
if tc.passed {
end.Outcome = runner.PipelineEndedOutcomePassed
}
if err := ix.HeardPins(ctx, nil, &end); err != nil {
t.Fatal(err) t.Fatal(err)
} }
ix.JudgePins(ctx) ix.JudgePins(ctx)
for _, s := range tc.then {
keep(s, nil)
}
if got := kept(); !slices.Equal(got, tc.afterTheEnd) { if got := kept(); !slices.Equal(got, tc.afterTheEnd) {
t.Errorf("after the pipeline's end: kept %v, want %v", got, tc.afterTheEnd) t.Errorf("after the pipeline's end: kept %v, want %v", got, tc.afterTheEnd)
} }
if keys, _ := ix.Pins.Keys(ctx); len(keys) > 0 { if keys, _ := ix.Pins.Keys(ctx); len(keys) > 0 {
for _, k := range keys { for _, k := range keys {
if v, _, _, _ := ix.Pins.Get(ctx, k); len(mustRecords(t, v)) > 0 { want := []string{}
t.Errorf("after the pipeline's end the pin of %s holds %s, want none", k, v) if slices.Contains(tc.released, k) {
want = []string{ReleaseRecord("main", "reader")}
}
if v, _, _, _ := ix.Pins.Get(ctx, k); !slices.Equal(append([]string{}, mustRecords(t, v)...), want) {
t.Errorf("after the pipeline's end the pin of %s holds %s, want %v", k, v, want)
} }
} }
} }
@@ -41,7 +41,7 @@ func (r Requirement) Versions() []string {
return out return out
} }
// RequiredOf is, per mesh Go module a go.sum names (as fingerprint.MeshPackagesOf takes them), the highest version it // RequiredOf is, per mesh Go module a go.sum names (a line under trunk.ForgeModulePrefix whose path names a repository), the highest version it
// names, its module line or its go.mod line alike. // names, its module line or its go.mod line alike.
func RequiredOf(goSum []byte) map[string]string { func RequiredOf(goSum []byte) map[string]string {
out := map[string]string{} out := map[string]string{}
@@ -108,7 +108,7 @@ func (ix *Indexer) SeedModulesFrom(ctx context.Context, defs map[string]seatdef.
seat := seatOf[a.Ref] seat := seatOf[a.Ref]
sa := SeedArtifact{Name: name, Seat: seat} sa := SeedArtifact{Name: name, Seat: seat}
in := nodebuildgo.StepsInput{Module: e.Module, Artifact: nodebuildgo.DetectedStepsByModuleArtifactKey(e.Module, name), Fingerprint: e.Fp, in := nodebuildgo.StepsInput{Module: e.Module, Artifact: nodebuildgo.DetectedStepsByModuleArtifactKey(e.Module, name), Fingerprint: e.Fp,
Fetch: &nodebuildgo.Fetch{Repository: e.Repository, Commit: e.Commit, Packages: maps.Clone(e.Packages)}} Fetch: &nodebuildgo.Fetch{Repository: e.Repository, Commit: e.Commit, Packages: PinsOf(e.Packages)}}
// The module's directory and its named paths go together, only when the entry names its paths (ADR 0499 clause // The module's directory and its named paths go together, only when the entry names its paths (ADR 0499 clause
// 4); a request without them builds from the module's directory (sdk/buildstep FetchRequest). // 4); a request without them builds from the module's directory (sdk/buildstep FetchRequest).
if paths := namedPaths(e, name); len(paths) > 0 { if paths := namedPaths(e, name); len(paths) > 0 {
@@ -24,7 +24,7 @@ func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
fp := "src2:" + strings.Repeat("a", 64) fp := "src2:" + strings.Repeat("a", 64)
entries := map[string]meshcatalog.CatalogEntry{"avahi": {Module: "avahi", Repository: "novox/r", Commit: "c1", Path: "mods/avahi", Fp: fp, Manifest: map[string]any{}, entries := map[string]meshcatalog.CatalogEntry{"avahi": {Module: "avahi", Repository: "novox/r", Commit: "c1", Path: "mods/avahi", Fp: fp, Manifest: map[string]any{},
Artifacts: []meshcatalog.Artifact{{Ref: "avahi:program", Language: "go"}, {Ref: "avahi:image", Language: "go", Kind: "image"}}, Artifacts: []meshcatalog.Artifact{{Ref: "avahi:program", Language: "go"}, {Ref: "avahi:image", Language: "go", Kind: "image"}},
Packages: map[string]string{"git.novox.be/novox/mesh-sdk": "c9"}, Packages: map[string]string{"git.novox.be/novox/mesh-sdk": "v0.1.9", "git.novox.be/novox/mesh-contracts/go": ""}, // the second none released (H-A-161)
Inputs: &meshcatalog.EntryInputs{Paths: map[string][]string{"program": {"cmd"}}}}} Inputs: &meshcatalog.EntryInputs{Paths: map[string][]string{"program": {"cmd"}}}}}
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, entries) rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, entries)
if err != nil { if err != nil {
@@ -71,9 +71,12 @@ func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
t.Errorf("the image artifact is %+v", a) t.Errorf("the image artifact is %+v", a)
} }
in := got[0] in := got[0]
if in.Module != "avahi" || in.Artifact != "avahi.program" || in.Fingerprint != fp || in.Fetch == nil || in.Fetch.Repository != "novox/r" || in.Fetch.Commit != "c1" || in.Fetch.Dir != "mods/avahi" || len(in.Fetch.Paths) != 1 || in.Fetch.Packages["git.novox.be/novox/mesh-sdk"] != "c9" { if in.Module != "avahi" || in.Artifact != "avahi.program" || in.Fingerprint != fp || in.Fetch == nil || in.Fetch.Repository != "novox/r" || in.Fetch.Commit != "c1" || in.Fetch.Dir != "mods/avahi" || len(in.Fetch.Paths) != 1 || in.Fetch.Packages["git.novox.be/novox/mesh-sdk"] != "v0.1.9" {
t.Errorf("the steps input is %+v", in) t.Errorf("the steps input is %+v", in)
} }
if _, pinned := in.Fetch.Packages["git.novox.be/novox/mesh-contracts/go"]; pinned {
t.Errorf("a package kept with no version is pinned in the fetch: %v", in.Fetch.Packages)
}
if len(w.Held()) != 0 { if len(w.Held()) != 0 {
t.Fatal("a call unanswered for no time wrote its finding") t.Fatal("a call unanswered for no time wrote its finding")
} }
@@ -9,7 +9,7 @@ type RepoRead struct {
Trees map[string]string Trees map[string]string
Manifests map[string]string Manifests map[string]string
Locks map[string]map[string]string Locks map[string]map[string]string
// Packages are the mesh packages each directory's go.sum names (fingerprint.MeshPackagesOf, novox/hq ADR 0573 §2). // Packages are the mesh packages each directory's go.mod requires (fingerprint.MeshRequiresOf, design 47 H-A-157).
Packages map[string][]string Packages map[string][]string
} }
@@ -16,6 +16,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
"mesh-delivery-indexer/judge" "mesh-delivery-indexer/judge"
) )
@@ -152,11 +153,13 @@ func (ix *Indexer) recomputeModules(ctx context.Context, overlay map[string]mesh
} }
ix.recompute.Lock() ix.recompute.Lock()
defer ix.recompute.Unlock() defer ix.recompute.Unlock()
return ix.recomputeLocked(ctx, overlay, w, repository) return ix.recomputeLocked(ctx, overlay, w, repository, nil)
} }
// recomputeLocked is recomputeModules with ix.recompute held by the caller (a hold that moves, holds.go). // recomputeLocked is recomputeModules with ix.recompute held by the caller (a hold that moves, holds.go).
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string) error { //
// only, when not nil, names the modules the pass moves (a re-pin, H-A-165): every other stands as held.
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string, only map[string]bool) error {
if ix.Manifests == nil { if ix.Manifests == nil {
return ix.writeOverlay(ctx, overlay, nil, w) return ix.writeOverlay(ctx, overlay, nil, w)
} }
@@ -287,18 +290,20 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
r.none = strings.Join(missing, "; ") r.none = strings.Join(missing, "; ")
case noToolchain != "": case noToolchain != "":
r.none = noToolchain r.none = noToolchain
if fp, pkgs, ok := stillComputed(module, kept, k, toolchains, held, pub, pins, ix.sourceHeads()); ok { if fp, pkgs, ok := stillComputed(module, kept, k, toolchains, held, pub, pins, ix.sourceHeads(), series); ok {
r = result{fp: fp, tcs: k.held.Toolchains, pkgs: pkgs} r = result{fp: fp, tcs: k.held.Toolchains, pkgs: pkgs}
} }
default: default:
f := formOf(module, k.cur.Repository, &kept, toolchains, k.held, held, pub, pins, ix.sourceHeads(), ix.logf) f := formOf(module, k.cur.Repository, &kept, toolchains, k.held, held, pub, pins, ix.sourceHeads(), ix.logf)
r.fp, r.none = f.fp, f.none r.fp, r.none = f.fp, f.none
if r.fp != "" { if r.fp != "" {
r.tcs, r.pkgs = toolchainsUsed(toolchains), f.versions // a pin is a released version, never a commit (H-A-161), and the version line's fingerprint takes it (H-A-167)
r.fp, r.pkgs = pinnedOf(f, series, k.manifest, k.cur.Repository)
r.tcs = toolchainsUsed(toolchains)
} }
} }
if _, merged := overlay[module]; !merged && repository != "" && !trunk.SameRepository(k.cur.Repository, repository) && if _, merged := overlay[module]; !merged && (repository != "" && !trunk.SameRepository(k.cur.Repository, repository) &&
maps.Equal(k.held.Packages, map[string]string(r.pkgs)) { !importsOf(r.pkgs, repository) || only != nil && !only[module]) {
// A module registered for another repository is moved by this repository's commit only when it imports a package the // A module registered for another repository is moved by this repository's commit only when it imports a package the
// commit moved (A-476a, ADR 0593 §3); any other stands as held (A-476 c), and so it is for a base of this repository's. // commit moved (A-476a, ADR 0593 §3); any other stands as held (A-476 c), and so it is for a base of this repository's.
r = result{fp: k.held.Fp2, none: k.held.None2, tcs: k.held.Toolchains, pkgs: sdkstep.PackageVersions(k.held.Packages)} r = result{fp: k.held.Fp2, none: k.held.None2, tcs: k.held.Toolchains, pkgs: sdkstep.PackageVersions(k.held.Packages)}
@@ -492,7 +497,7 @@ func hasContext(m catalogue.Manifest) bool {
// with the toolchains it was computed with in place of the languages agreed on none, give it again, in its form. Any // with the toolchains it was computed with in place of the languages agreed on none, give it again, in its form. Any
// other input that moved (a merge, a base, an agreed toolchain, an imported package) gives another fingerprint, and the // other input that moved (a merge, a base, an agreed toolchain, an imported package) gives another fingerprint, and the
// module has none until they agree. // module has none until they agree.
func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2, k *known, agreed map[string]meshcatalogclient.Toolchain2, held map[string]meshcatalog.ModulesByModuleValue, published map[string]meshcatalog.PackagesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string) (string, sdkstep.PackageVersions, bool) { func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2, k *known, agreed map[string]meshcatalogclient.Toolchain2, held map[string]meshcatalog.ModulesByModuleValue, published map[string]meshcatalog.PackagesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string, series map[string]SeriesPackage) (string, sdkstep.PackageVersions, bool) {
if k.held.Fp2 == "" { if k.held.Fp2 == "" {
return "", nil, false return "", nil, false
} }
@@ -509,7 +514,11 @@ func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2,
toolchains[artifact] = meshcatalogclient.Toolchain2{Language: language, Version: used.Version, Recipe: used.Recipe} toolchains[artifact] = meshcatalogclient.Toolchain2{Language: language, Version: used.Version, Recipe: used.Recipe}
} }
f := formOf(module, k.held.Repository, &kept, toolchains, k.held, held, published, pins, heads, nil) f := formOf(module, k.held.Repository, &kept, toolchains, k.held, held, published, pins, heads, nil)
return f.fp, f.versions, f.none == "" && f.fp == k.held.Fp2 if f.none != "" || f.fp == "" {
return "", nil, false
}
fp, pkgs := pinnedOf(f, series, k.manifest, k.held.Repository)
return fp, pkgs, fp == k.held.Fp2
} }
// toolchainsUsed are the toolchains a fingerprint was computed with, by language (an artifact's toolchain is its language's). // toolchainsUsed are the toolchains a fingerprint was computed with, by language (an artifact's toolchain is its language's).
@@ -577,3 +586,41 @@ func (ix *Indexer) fillPaths(ctx context.Context) error {
} }
return nil return nil
} }
// importsOf says whether pkgs name a package of repository: a pin is a released version (H-A-161), so a commit of
// repository moves a module of another only through a package of it the module imports (A-476a).
func importsOf(pkgs sdkstep.PackageVersions, repository string) bool {
for p := range pkgs {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, repository) {
return true
}
}
return false
}
// Repin moves the pins of every module whose packages hold a package of module (a held module's directory holds it) to
// module's newest released version, the pass of H-A-161 for those modules alone: a version-released of module starts it
// (design 47 H-A-165), so pins move on the event and never wait for a merge or a hand act. It answers how many it named.
func (ix *Indexer) Repin(ctx context.Context, module string) (int, error) {
if ix.Modules == nil || ix.Manifests == nil {
return 0, nil
}
held, err := ix.heldModules(ctx)
if err != nil {
return 0, err
}
only := map[string]bool{}
for key, m := range held {
for p := range m.Packages {
if moduleOfKey(fingerprint.HolderOf(p, held)) == module {
only[key] = true
}
}
}
if len(only) == 0 {
return 0, nil
}
ix.recompute.Lock()
defer ix.recompute.Unlock()
return len(only), ix.recomputeLocked(ctx, nil, nil, "", only)
}
@@ -17,6 +17,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep" sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -757,7 +758,7 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Errorf("no published mesh-sdk: fp2 %q none2 %q", u.Fp2, u.None2) t.Errorf("no published mesh-sdk: fp2 %q none2 %q", u.Fp2, u.None2)
} }
tools, _ := moduleOf(t, mods, "mesh-tools") tools, _ := moduleOf(t, mods, "mesh-tools")
sdkMerge := func(commit, tree string) meshcatalog.ModulesByModuleValue { sdkMerge := func(commit, tree, pin string) meshcatalog.ModulesByModuleValue {
t.Helper() t.Helper()
f := whole{Repository: "novox/mesh-sdk", Branch: "feat/x", Head: h1, f := whole{Repository: "novox/mesh-sdk", Branch: "feat/x", Head: h1,
Merge: commit, Root: tree, Trees: map[string]string{"": tree}, Manifests: map[string]string{"": `{"module":"mesh-sdk","version":"1.0"}`}} Merge: commit, Root: tree, Trees: map[string]string{"": tree}, Manifests: map[string]string{"": `{"module":"mesh-sdk","version":"1.0"}`}}
@@ -765,9 +766,10 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// A-476a: a commit of mesh-sdk moves avahi, of another repository, which imports its package, and leaves mesh-tools, // A-476a: a commit of mesh-sdk moves avahi, of another repository, which imports its package, and leaves mesh-tools,
// which imports nothing it moved. // which imports nothing it moved. Its pin is mesh-sdk's newest released version, never the commit; with none
// released it has no pin, the package kept for its kind-1 edge (H-A-161).
a, _ := moduleOf(t, mods, "avahi") a, _ := moduleOf(t, mods, "avahi")
if a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: commit}) { if a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: pin}) {
t.Fatalf("after mesh-sdk's merge %.8s: fp2 %q none2 %q packages %v", commit, a.Fp2, a.None2, a.Packages) t.Fatalf("after mesh-sdk's merge %.8s: fp2 %q none2 %q packages %v", commit, a.Fp2, a.None2, a.Packages)
} }
if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 != tools.Fp2 || o.Packages != nil { if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 != tools.Fp2 || o.Packages != nil {
@@ -781,8 +783,56 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
stale.Fp2 = "src2:" + strings.Repeat("5", 64) stale.Fp2 = "src2:" + strings.Repeat("5", 64)
mods.put("mesh-tools", mustEncode(t, stale)) mods.put("mesh-tools", mustEncode(t, stale))
tools = stale tools = stale
first := sdkMerge(strings.Repeat("1", 40), "tree-sdk-1") first := sdkMerge(strings.Repeat("1", 40), "tree-sdk-1", "")
second := sdkMerge(strings.Repeat("2", 40), "tree-sdk-2") // mesh-sdk releases 0.1.20 (0.1.9 before it): the release re-pins avahi with no merge, its fingerprint moving with the
// pin so it is built again against the release, and leaves mesh-tools, which holds none of it (H-A-165, H-A-167). A
// re-pin that fails is no release acknowledged: the package-published delivered again re-pins.
ix.Versions, ix.Heard = newMem(), newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.9"), []byte(`{}`))
revisions := ix.Revisions
ix.Revisions = newMemObjects(1 << 20)
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, map[string]meshcatalog.CatalogEntry{
"mesh-sdk": {Module: "mesh-sdk", Manifest: map[string]any{}, Artifacts: []meshcatalog.Artifact{{Ref: "mesh-sdk:go"}}}})
if err != nil {
t.Fatal(err)
}
if err := KeepRevision(context.Background(), ix.Revisions, rev, objects); err != nil {
t.Fatal(err)
}
release := meshbuilds.PackagePublished{Module: "mesh-sdk", Artifact: "mesh-sdk:go", Digest: "sha256:g", Version: "0.1.20", Source: 7, Branch: "main"}
mods.keysErr = errors.New("the bus did not answer")
if err := ix.HearPackagePublished(context.Background(), release); err == nil {
t.Fatal("a release whose re-pin failed was acknowledged")
}
mods.keysErr = nil
// The registry published mesh-sdk's Go package at 0.1.18, not at the module's release number 0.1.20, in the series
// avahi accepts (the default 0.1), with no contents read: the pin is that version (H-A-183, H-A-185).
ix.Packages = newMem()
if _, err := ix.HearVersion(context.Background(), HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.1.18", Commit: strings.Repeat("1", 40), GoModule: sdk, Published: true}); err != nil {
t.Fatal(err)
}
if err := ix.HearPackagePublished(context.Background(), release); err != nil {
t.Fatal(err)
}
ix.Revisions = revisions
if a, _ := moduleOf(t, mods, "avahi"); a.Fp2 == first.Fp2 || a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: "v0.1.18"}) {
t.Fatalf("after mesh-sdk's release: fp2 %q (was %q) packages %v, want a new fingerprint pinned to v0.1.18", a.Fp2, first.Fp2, a.Packages)
} else {
// A 0.2.0 publish, of a series avahi does not accept, leaves it alone, contents or none (H-A-184, H-A-185).
if _, err := ix.HearVersion(context.Background(), HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.2.0", Commit: strings.Repeat("1", 40), GoModule: sdk, Published: true}); err != nil {
t.Fatal(err)
}
if _, err := ix.Repin(context.Background(), "mesh-sdk"); err != nil { // the pass for avahi alone: mesh-tools is held stale here
t.Fatal(err)
}
if b, _ := moduleOf(t, mods, "avahi"); b.Fp2 != a.Fp2 || b.Packages[sdk] != "v0.1.18" {
t.Fatalf("a publish of another series moved avahi: fp2 %q (was %q) packages %v", b.Fp2, a.Fp2, b.Packages)
}
}
if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 != tools.Fp2 {
t.Errorf("a re-pin moved mesh-tools, which holds no package of mesh-sdk: %q (was %q)", o.Fp2, tools.Fp2)
}
second := sdkMerge(strings.Repeat("2", 40), "tree-sdk-2", "v0.1.18")
if second.Fp2 == first.Fp2 { if second.Fp2 == first.Fp2 {
t.Errorf("a mesh-sdk trunk merge kept avahi's fingerprint %q", first.Fp2) t.Errorf("a mesh-sdk trunk merge kept avahi's fingerprint %q", first.Fp2)
} }
@@ -6,14 +6,21 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"maps"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
stdio "git.novox.be/novox/mesh-sdk/go" stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/keys" "git.novox.be/novox/mesh-sdk/go/keys"
"mesh-delivery-indexer/internal/fingerprint"
) )
// The catalog's released versions (design 47 G §349, §372, §374): the indexer keeps, for the seat mesh-catalog, each module's // The catalog's released versions (design 47 G §349, §372, §374): the indexer keeps, for the seat mesh-catalog, each module's
@@ -123,8 +130,9 @@ func (ix *Indexer) HearPackagePublished(ctx context.Context, p meshbuilds.Packag
if raw, _, found, err := ix.Versions.Get(ctx, meshcatalog.VersionsByModuleVersionKey(p.Module, p.Version)); err != nil { if raw, _, found, err := ix.Versions.Get(ctx, meshcatalog.VersionsByModuleVersionKey(p.Module, p.Version)); err != nil {
return err return err
} else if found && !isFloor(raw) { } else if found && !isFloor(raw) {
ix.logf("versions: %s %s is released already, so the package-published of %s is heard again and changes nothing", p.Module, p.Version, p.Artifact) // A package-published delivered again after a re-pin failed (H-A-167): the re-pin is made again, nothing else.
return nil ix.logf("versions: %s %s is released already, so the package-published of %s is heard again and only re-pins", p.Module, p.Version, p.Artifact)
return ix.repin(ctx, p.Module, p.Version)
} }
entry, err := ix.entryOf(ctx, p.Module) entry, err := ix.entryOf(ctx, p.Module)
if err != nil { if err != nil {
@@ -234,6 +242,13 @@ func (ix *Indexer) releaseVersion(ctx context.Context, p meshbuilds.PackagePubli
} }
ix.logf("versions: %s %s is released and version-released is said (%d artifacts, source %d)", p.Module, p.Version, len(v.Artifacts), v.Source) ix.logf("versions: %s %s is released and version-released is said (%d artifacts, source %d)", p.Module, p.Version, len(v.Artifacts), v.Source)
} }
if !late {
// The release re-pins every module whose packages hold one of it (design 47 H-A-165); one that fails is not
// acknowledged, so the package-published is delivered again (H-A-167).
if err := ix.repin(ctx, p.Module, p.Version); err != nil {
return err
}
}
if err := ix.trimVersions(ctx, p.Module); err != nil { if err := ix.trimVersions(ctx, p.Module); err != nil {
ix.logf("versions: %s's oldest versions are not deleted: %v", p.Module, err) ix.logf("versions: %s's oldest versions are not deleted: %v", p.Module, err)
} }
@@ -352,3 +367,88 @@ func isFloor(raw []byte) bool {
var v meshcatalog.VersionEntry var v meshcatalog.VersionEntry
return json.Unmarshal(raw, &v) == nil && len(v.Artifacts) == 0 return json.Unmarshal(raw, &v) == nil && len(v.Artifacts) == 0
} }
// releasedPins are a module's pins (its catalog entry's packages, which the builder builds it against) from the versions its
// fingerprint was computed with (design 47 H-A-161, H-A-183, H-A-184, H-A-185, G §370): each package is pinned to the
// newest version the registry published for its path (the seats' published, kept in the package's series by HearVersion)
// within the series the module accepts, the one its manifest m names or the default (Manifest.AcceptedSeries), whether
// or not that series holds contents yet; a publish in another series pins nothing and moves nothing, and a module's
// release number is never a pin. A package with none published in that series is kept with no version, so its kind-1
// edge stands (H §746) and its build waits for the publish (H-A-165). A module of the module's own repository is never
// pinned, whatever a build reported: it is built from the tree (design 47 H-A-198, H-A-201, H-A-202). nil for none.
func releasedPins(versions sdkstep.PackageVersions, series map[string]SeriesPackage, m catalogue.Manifest, repository string) sdkstep.PackageVersions {
if len(versions) == 0 {
return nil
}
out := sdkstep.PackageVersions{}
for p := range versions {
if repo, _, ok := trunk.PackageRepository(p); ok && repository != "" && trunk.SameRepository(repo, repository) {
continue
}
out[p] = ""
for _, s := range series {
if s.GoModule != p || s.Newest == nil || s.Series != m.AcceptedSeries(s.Name) {
continue
}
v := strings.TrimPrefix(s.Newest.Version, "v")
if _, ok := versionParts(v); ok && (out[p] == "" || compareVersions(v, strings.TrimPrefix(out[p], "v")) > 0) {
out[p] = "v" + v
}
}
}
return out
}
// pinnedOf is a form's fingerprint and its pins (releasedPins): the pins are part of what a build is made of, so every
// form's fingerprint takes them (design 47 H-A-167, H-A-172; ADR 0573 §4) and a re-pin moves it, the module a mover
// built again against the release; a fingerprint with no pin is the form's as it is. The held fingerprint of a build that
// stands is kept while its pins are the ones it was built with, else its import form takes the new pins.
func pinnedOf(f form, series map[string]SeriesPackage, m catalogue.Manifest, repository string) (string, sdkstep.PackageVersions) {
pins := releasedPins(f.versions, series, m, repository)
if f.stands {
if maps.Equal(f.built, map[string]string(pins)) {
return f.fp, pins
}
return fingerprint.WithPins(f.raw, pins), pins
}
return fingerprint.WithPins(f.fp, pins), pins
}
// moduleOfKey is the module a modules key names; "" for none.
func moduleOfKey(key string) string {
if key == "" {
return ""
}
if m, err := meshcatalogclient.ParseModulesKey(key); err == nil {
return m
}
return key
}
// PinsOf are the pins a build is asked with from a catalog entry's packages: each package with a version; one kept with
// none (no released version, H-A-161) is no pin. nil for none.
func PinsOf(packages map[string]string) sdkstep.PackageVersions {
var out sdkstep.PackageVersions
for p, v := range packages {
if v == "" {
continue
}
if out == nil {
out = sdkstep.PackageVersions{}
}
out[p] = v
}
return out
}
// repin is Repin of a release of module at version, said in the journal; its failure is the caller's error.
func (ix *Indexer) repin(ctx context.Context, module, version string) error {
n, err := ix.Repin(ctx, module)
if err != nil {
return fmt.Errorf("the modules pinned to %s are not re-pinned to %s: %w", module, version, err)
}
if n > 0 {
ix.logf("versions: %d modules re-pinned to %s %s", n, module, version)
}
return nil
}
@@ -42,6 +42,8 @@ type artifact struct {
Binary string `json:"binary"` Binary string `json:"binary"`
Loads []string `json:"loads"` Loads []string `json:"loads"`
Env map[string]string `json:"env"` Env map[string]string `json:"env"`
Paths []string `json:"paths"`
Series string `json:"series"`
} }
type stream struct { type stream struct {
@@ -170,7 +172,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// internal/broker/state.go:62-93). // internal/broker/state.go:62-93).
// The registry seats' published fill each (§413: no consume of version.numbered) package's series (DLV-I5, design 51 *The versions*). // The registry seats' published fill each (§413: no consume of version.numbered) package's series (DLV-I5, design 51 *The versions*).
if want := []string{"go-package-registry.published", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-catalog.room", "mesh-conditions.findings-asked", if want := []string{"go-package-registry.published", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-catalog.room", "mesh-conditions.findings-asked",
"mesh-delivery.source-moved", "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.pipeline-started", "mesh-pipeline-runner.run-started", "mesh-wants.wait-ended", "mesh-wants.wanted", "node-build-go.step-ended", "node-engine.running", "npm-package-registry.published", "operator-channel.decided", "seat:mesh-module-assigner.views.assignments", "seat:mesh-module-assigner.views.seat-holders"}; !reflect.DeepEqual(sorted(m.Consumes), want) { "mesh-delivery.source-moved", "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.pipeline-started", "mesh-pipeline-runner.run-started", "mesh-wants.pipeline-wanted", "mesh-wants.wait-ended", "mesh-wants.wanted", "node-build-go.step-ended", "node-engine.running", "npm-package-registry.published", "operator-channel.decided", "seat:mesh-module-assigner.views.assignments", "seat:mesh-module-assigner.views.seat-holders"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want) t.Errorf("consumes = %v, want %v", m.Consumes, want)
} }
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" { if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
@@ -344,8 +346,12 @@ func TestManifestProcess(t *testing.T) { // 1238/AC1
if _, bus := p["bus"]; bus { if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential") t.Errorf("the process declares a bus: it runs on its own broker credential")
} }
if len(m.Build.Artifacts) != 2 { if len(m.Build.Artifacts) != 3 {
t.Fatalf("artifacts = %+v, want the program and the tools", m.Build.Artifacts) t.Fatalf("artifacts = %+v, want the program, the tools and the package takein", m.Build.Artifacts)
}
// H-A-186 (design 47 G §370): takein, which the gate, the view and the watchman import, is a declared package.
if a := m.Build.Artifacts[2]; a.Name != "takein" || a.Kind != "package" || a.Language != "go" || !reflect.DeepEqual(a.Paths, []string{"takein"}) || a.Series != "0.1" {
t.Errorf("package artifact = %+v", a)
} }
if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-indexer-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-indexer-tools"}) { if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-indexer-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-indexer-tools"}) {
t.Errorf("tools artifact = %+v", a) // 1945 t.Errorf("tools artifact = %+v", a) // 1945
+10
View File
@@ -89,6 +89,7 @@
"mesh-pipeline-runner.run-started", "mesh-pipeline-runner.run-started",
"mesh-pipeline-runner.pipeline-started", "mesh-pipeline-runner.pipeline-started",
"mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.pipeline-ended",
"mesh-wants.pipeline-wanted",
"node-engine.running", "node-engine.running",
"operator-channel.decided", "operator-channel.decided",
"npm-package-registry.published" "npm-package-registry.published"
@@ -222,6 +223,15 @@
"loads": [ "loads": [
"mesh-delivery-indexer-tools" "mesh-delivery-indexer-tools"
] ]
},
{
"name": "takein",
"kind": "package",
"language": "go",
"paths": [
"takein"
],
"series": "0.1"
} }
] ]
} }
@@ -42,7 +42,7 @@ func TestTools(t *testing.T) {
for _, tl := range UnloggedTools(stdio.State("pipelines"), stdio.Log("step-log"), nil, nil) { for _, tl := range UnloggedTools(stdio.State("pipelines"), stdio.Log("step-log"), nil, nil) {
names = append(names, tl.Name) names = append(names, tl.Name)
} }
want := "mesh-pipeline-runner.show mesh-pipeline-runner.list mesh-pipeline-runner.stands mesh-pipeline-runner.why-carried mesh-pipeline-runner.kept runner_first_views runner_first_view_again runner_cancel runner_rerun runner_start runner_run runner_accept runner_place runner_ask_again runner_health runner_waiting runner_unlogged runner_unlogged_append" want := "mesh-pipeline-runner.show mesh-pipeline-runner.list mesh-pipeline-runner.stands mesh-pipeline-runner.why-carried mesh-pipeline-runner.kept runner_first_views runner_first_view_again runner_cancel runner_rerun runner_start runner_run runner_accept runner_place runner_end_again runner_ask_again runner_health runner_waiting runner_unlogged runner_unlogged_append"
if got := strings.Join(names, " "); got != want { if got := strings.Join(names, " "); got != want {
t.Errorf("tools %q, want %q", got, want) t.Errorf("tools %q, want %q", got, want)
} }
@@ -19,7 +19,7 @@ import (
// §549): runner_unlogged lists them, runner_unlogged_append appends them by hand, written in the hand-act log; it is of class // §549): runner_unlogged lists them, runner_unlogged_append appends them by hand, written in the hand-act log; it is of class
// act and takes §391's default callers. // act and takes §391's default callers.
func UnloggedTools(pipelines stdio.KeptState, log stdio.KeptLog, acts views.HandActs, logf func(string, ...any)) []stdio.Tool { func UnloggedTools(pipelines stdio.KeptState, log stdio.KeptLog, acts views.HandActs, logf func(string, ...any)) []stdio.Tool {
h := &pipeline.Hearer{Store: stdioPipelines{pipelines}, Logf: logf, h := &pipeline.Hearer{Store: &pipeline.MoverStore{Raw: stdioPipelines{pipelines}, Logf: logf}, Logf: logf,
Log: func(e seat.LogEntry) error { Log: func(e seat.LogEntry) error {
_, err := log.Append(keys.EscapeName(e.Pipeline), e) _, err := log.Append(keys.EscapeName(e.Pipeline), e)
return err return err
@@ -67,26 +67,36 @@ func UnloggedTools(pipelines stdio.KeptState, log stdio.KeptLog, acts views.Hand
} }
} }
// stdioPipelines is the state pipelines as the runtime serves it, for the by-hand append. // stdioPipelines is the state pipelines as bytes, as the runtime serves it, for the by-hand append: read and written through
// a pipeline.MoverStore, which keeps each mover's steps under its own key (H-A-187).
type stdioPipelines struct{ s stdio.KeptState } type stdioPipelines struct{ s stdio.KeptState }
func (s stdioPipelines) Get(key string) (seat.Pipeline, uint64, error) { func (s stdioPipelines) Get(key string) ([]byte, uint64, error) {
e, err := s.s.Get(key) e, err := s.s.Get(key)
if err != nil || e == nil { if err != nil || e == nil {
return seat.Pipeline{}, 0, errors.Join(err, errors.New("no such pipeline")) return nil, 0, err
} }
var p seat.Pipeline return e.Value, e.Revision, nil
return p, e.Revision, json.Unmarshal(e.Value, &p)
} }
func (s stdioPipelines) Update(key string, p seat.Pipeline, rev uint64) (uint64, error) { func (s stdioPipelines) Create(key string, value []byte) (uint64, error) {
r, err := s.s.Update(key, p, rev) r, err := s.s.Create(key, json.RawMessage(value))
if err != nil && strings.Contains(err.Error(), "exists") {
return 0, pipeline.ErrExists
}
return r, err
}
func (s stdioPipelines) Update(key string, value []byte, rev uint64) (uint64, error) {
r, err := s.s.Update(key, json.RawMessage(value), rev)
if err != nil && (strings.Contains(err.Error(), "wrong last sequence") || strings.Contains(err.Error(), "revision")) { if err != nil && (strings.Contains(err.Error(), "wrong last sequence") || strings.Contains(err.Error(), "revision")) {
return 0, pipeline.ErrChanged return 0, pipeline.ErrChanged
} }
return r, err return r, err
} }
func (s stdioPipelines) Delete(key string) error { return s.s.Delete(key) }
func (s stdioPipelines) Keys(filter string) ([]string, error) { func (s stdioPipelines) Keys(filter string) ([]string, error) {
all, err := s.s.Keys() all, err := s.s.Keys()
var out []string var out []string
@@ -54,6 +54,15 @@ func VerbTools(ask func(key string, body any) (json.RawMessage, error)) []stdio.
} }
return whyOf(in.Why) return whyOf(in.Why)
}), }),
verbTool[seat.EndAgainInput](ask, "end_again", seat.VerbEndAgain,
"Judge a running pipeline's end once more by the seat verb end-again (design 47 H H-A-163, H-A-164): it ends as at a step's end, failed as soon as it can no longer pass (a mover with no registration named), and says pipeline-ended; one that still cannot end is answered running. The pipeline's key and why; written in the hand-act log. A pipeline not running is refused by the verb.",
obj([]string{"pipeline", "why"}, map[string]any{"pipeline": str, "why": str}),
func(in seat.EndAgainInput) string {
if in.Pipeline == "" {
return "end-again names the pipeline"
}
return whyOf(in.Why)
}),
verbTool[seat.AskAgainInput](ask, "ask_again", seat.VerbAskAgain, verbTool[seat.AskAgainInput](ask, "ask_again", seat.VerbAskAgain,
"Ask a declined approval step of a pipeline once more by the seat verb ask-again: the pipeline's key, the mover's module (needed when the pipeline moves several) and why.", "Ask a declined approval step of a pipeline once more by the seat verb ask-again: the pipeline's key, the mover's module (needed when the pipeline moves several) and why.",
obj([]string{"pipeline", "why"}, map[string]any{"pipeline": str, "module": str, "why": str}), obj([]string{"pipeline", "why"}, map[string]any{"pipeline": str, "module": str, "why": str}),
@@ -32,6 +32,10 @@ func TestVerbTools(t *testing.T) {
{"place", "runner_place", "seat:mesh-pipeline-runner.place", "", map[string]any{"pipeline": "main.6", "why": "urgent"}, {"place", "runner_place", "seat:mesh-pipeline-runner.place", "", map[string]any{"pipeline": "main.6", "why": "urgent"},
seat.PlaceInput{Pipeline: "main.6", Why: "urgent"}}, seat.PlaceInput{Pipeline: "main.6", Why: "urgent"}},
{"place no why", "runner_place", "", "names why", map[string]any{"pipeline": "main.6"}, nil}, {"place no why", "runner_place", "", "names why", map[string]any{"pipeline": "main.6"}, nil},
// H-A-164: runner_end_again asks the verb end-again with the pipeline and why.
{"end-again", "runner_end_again", "seat:mesh-pipeline-runner.end-again", "", map[string]any{"pipeline": "main.20", "why": "left running"},
seat.EndAgainInput{Pipeline: "main.20", Why: "left running"}},
{"end-again no why", "runner_end_again", "", "names why", map[string]any{"pipeline": "main.20"}, nil},
{"ask-again no why", "runner_ask_again", "", "names why", map[string]any{"pipeline": "p"}, nil}, {"ask-again no why", "runner_ask_again", "", "names why", map[string]any{"pipeline": "p"}, nil},
} { } {
t.Run(c.name, func(t *testing.T) { t.Run(c.name, func(t *testing.T) {
@@ -136,9 +136,9 @@ func (k keptStore) Get(key string) (seat.KeptResult, bool, error) {
func (k keptStore) Delete(key string) error { return k.kv.Delete(k.ctx, key) } func (k keptStore) Delete(key string) error { return k.kv.Delete(k.ctx, key) }
// newAger is the ageing over the runner's own stores. // newAger is the ageing over the runner's own stores.
func newAger(ctx context.Context, js jetstream.JetStream, cred process.Account, pipelines, kept, aged jetstream.KeyValue, logf func(string, ...any)) *pipeline.Ager { func newAger(ctx context.Context, js jetstream.JetStream, cred process.Account, pipelines *pipeline.MoverStore, kept, aged jetstream.KeyValue, logf func(string, ...any)) *pipeline.Ager {
return &pipeline.Ager{Log: stepLog{ctx: ctx, js: js, module: cred.Module}.Read, Aged: agedState{ctx: ctx, kv: aged}, return &pipeline.Ager{Log: stepLog{ctx: ctx, js: js, module: cred.Module}.Read, Aged: agedState{ctx: ctx, kv: aged},
Pipelines: pipelineDeleter{pipelineStore{ctx: ctx, kv: pipelines}}, Kept: keptStore{ctx: ctx, kv: kept}, Logf: logf, Pipelines: pipelines, Kept: keptStore{ctx: ctx, kv: kept}, Logf: logf,
Outputs: func(name string) error { Outputs: func(name string) error {
o, err := js.ObjectStore(ctx, catalogue.ObjectsBucket(cred.Module, seat.StateStepOutputs)) o, err := js.ObjectStore(ctx, catalogue.ObjectsBucket(cred.Module, seat.StateStepOutputs))
if err != nil { if err != nil {
@@ -151,11 +151,6 @@ func newAger(ctx context.Context, js jetstream.JetStream, cred process.Account,
}} }}
} }
// pipelineDeleter is the pipelines store with the delete only its owner makes (§330).
type pipelineDeleter struct{ pipelineStore }
func (d pipelineDeleter) Delete(key string) error { return d.kv.Delete(d.ctx, key) }
// storeFills reads the fill of the runner's stores against their caps: pipelines and kept (their key-value streams), step-outputs // storeFills reads the fill of the runner's stores against their caps: pipelines and kept (their key-value streams), step-outputs
// (its object store's) and step-log (its log's), as the bus's own streams say it (design 47 G §366, §486). // (its object store's) and step-log (its log's), as the bus's own streams say it (design 47 G §366, §486).
func storeFills(ctx context.Context, js jetstream.JetStream, cred process.Account, now func() time.Time) func() ([]seat.Fill, error) { func storeFills(ctx context.Context, js jetstream.JetStream, cred process.Account, now func() time.Time) func() ([]seat.Fill, error) {
@@ -128,9 +128,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
} }
return err return err
} }
hearer := newHearer(ctx, js, cred, pipelines, kept, say, logger.Printf) store := newPipelineStore(ctx, pipelines, logger.Printf)
hearer := newHearer(ctx, js, cred, store, kept, say, logger.Printf)
hearer.Asker = busAsker{conn: conn, cred: cred} hearer.Asker = busAsker{conn: conn, cred: cred}
hearer.Ager = newAger(ctx, js, cred, pipelines, kept, aged, logger.Printf) hearer.Ager = newAger(ctx, js, cred, store, kept, aged, logger.Printf)
// The stores' fill is said before a store is full, and a full one refuses a start (§366). // The stores' fill is said before a store is full, and a full one refuses a start (§366).
fills := &pipeline.Fills{Read: storeFills(ctx, js, cred, time.Now), Logf: logger.Printf, fills := &pipeline.Fills{Read: storeFills(ctx, js, cred, time.Now), Logf: logger.Printf,
Say: func(event string, f seat.Fill) error { return say("mesh.seat."+seat.Seat+".event."+event, f) }} Say: func(event string, f seat.Fill) error { return say("mesh.seat."+seat.Seat+".event."+event, f) }}
@@ -147,11 +148,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
hearer.Failures = &pipeline.Failures{Findings: rv.Findings, Observer: cred.Module, Author: commitAuthor(calls.Call), Logf: logger.Printf} hearer.Failures = &pipeline.Failures{Findings: rv.Findings, Observer: cred.Module, Author: commitAuthor(calls.Call), Logf: logger.Printf}
// Its asking half (§22, §24): a pipeline's steps are placed from the module's registration and each build seat is asked // Its asking half (§22, §24): a pipeline's steps are placed from the module's registration and each build seat is asked
// for its step. // for its step.
starter := &pipeline.Starter{Pipelines: createdKV{ctx: ctx, kv: pipelines}, Logf: logger.Printf, Say: say, starter := &pipeline.Starter{Pipelines: store, Logf: logger.Printf, Say: say,
Registration: rv.Registrations.Registration, Steps: busStepAsker{ctx: ctx, js: js, cred: cred}, Name: cred.Module, Registration: rv.Registrations.Registration, Steps: busStepAsker{ctx: ctx, js: js, cred: cred}, Name: cred.Module,
Kept: keptReader(ctx, kept), Toolchain: toolchainReader(ctx, toolchains), Kept: keptReader(ctx, kept), Toolchain: toolchainReader(ctx, toolchains),
Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing} Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing}
hearer.AskSteps = starter.AskSteps hearer.AskSteps = starter.AskSteps
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.Release = func(key string) { hearer.Release = func(key string) {
if _, err := starter.Release(hearer.Store, key, rv.Catalog.View); err != nil { if _, err := starter.Release(hearer.Store, key, rv.Catalog.View); err != nil {
logger.Printf("the movers of %s waiting for a build are not placed: %v", key, err) logger.Printf("the movers of %s waiting for a build are not placed: %v", key, err)
@@ -169,10 +171,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return return
} }
// A pipeline of the branch whose registration was heard before this half was held gets its steps now (A-327). // A pipeline of the branch whose registration was heard before this half was held gets its steps now (A-327).
if n, err := starter.Waiting(pipelineStore{ctx: ctx, kv: pipelines}, branch, rv.Catalog.View); err != nil || n > 0 { if n, err := starter.Waiting(store, branch, rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the pipelines of %s that waited on its half for their steps: %d placed (%v)", branch, n, err) logger.Printf("the pipelines of %s that waited on its half for their steps: %d placed (%v)", branch, n, err)
} }
n, err := starter.Resume(pipelineStore{ctx: ctx, kv: pipelines}, branch, half) n, err := starter.Resume(store, branch, half)
if err != nil { if err != nil {
logger.Printf("the steps placed on %s and not yet asked: %d asked, then %v", branch, n, err) logger.Printf("the steps placed on %s and not yet asked: %d asked, then %v", branch, n, err)
return return
@@ -183,19 +185,19 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The view reaching a newer revision places the steps of the pipelines that waited for it, and asks nothing again (A-448, §200). // The view reaching a newer revision places the steps of the pipelines that waited for it, and asks nothing again (A-448, §200).
advance := func(branch string) { advance := func(branch string) {
rv.Holders.refresh() rv.Holders.refresh()
if n, err := starter.Waiting(pipelineStore{ctx: ctx, kv: pipelines}, branch, rv.Catalog.View); err != nil || n > 0 { if n, err := starter.Waiting(store, branch, rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the pipelines of %s that waited for a newer revision of its half: %d placed (%v)", branch, n, err) logger.Printf("the pipelines of %s that waited for a newer revision of its half: %d placed (%v)", branch, n, err)
} }
} }
advancer.Store(&advance) advancer.Store(&advance)
// A registration-changed places the steps of its module's pipelines started before it, from it (A-327). // A registration-changed places the steps of its module's pipelines started before it, from it (A-327).
register := func(reg catalog.Registration) { register := func(reg catalog.Registration) {
if n, err := starter.Registered(pipelineStore{ctx: ctx, kv: pipelines}, reg, rv.Catalog.View); err != nil || n > 0 { if n, err := starter.Registered(store, reg, rv.Catalog.View); err != nil || n > 0 {
logger.Printf("the pipelines of %s started before its registration: %d placed (%v)", reg.Module, n, err) logger.Printf("the pipelines of %s started before its registration: %d placed (%v)", reg.Module, n, err)
} }
} }
registrar.Store(&register) registrar.Store(&register)
starter.Store = pipelineStore{ctx: ctx, kv: pipelines} starter.Store = store
// The queue (design 47 H §748): at most pipelines-at-once pipelines have their steps asked, the others wait in order in // The queue (design 47 H §748): at most pipelines-at-once pipelines have their steps asked, the others wait in order in
// the state queue, placed at the start from it and on every end. // the state queue, placed at the start from it and on every end.
atOnce, waitingCap, err := queueSettings(getenv) atOnce, waitingCap, err := queueSettings(getenv)
@@ -67,6 +67,7 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
findingsStream(t, ctx, js, module)
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}}) events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -400,6 +401,8 @@ func pipelineKeys(t *testing.T, ctx context.Context, kv jetstream.KeyValue) []st
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// A mover's steps are a key of their own (H-A-187), placed with its pipeline: no pipeline of their own.
ks = slices.DeleteFunc(ks, pipeline.IsStepsKey)
slices.Sort(ks) slices.Sort(ks)
return ks return ks
} }
@@ -7,6 +7,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -75,6 +76,7 @@ func TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview(t *tes
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil { if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil {
t.Fatal(err) t.Fatal(err)
} }
findingsStream(t, ctx, js, module)
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}}) events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -220,6 +222,34 @@ func TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview(t *tes
waitFor(t, ctx, logs, func() bool { return readPipeline(ctx, pipelines, key, &p) && p.Outcome == seat.PipelineOutcomePassed }) waitFor(t, ctx, logs, func() bool { return readPipeline(ctx, pipelines, key, &p) && p.Outcome == seat.PipelineOutcomePassed })
// H-A-187 over the bus: show composes the pipeline from its record and mover keys, naming a mover its steps alone, and a
// record written before it, its steps inline, is read whole and split at its next write.
shown, err := pipeline.Reads{Pipelines: kvKept{ctx, pipelines}}.Show(seat.ShowInput{Pipeline: key, Mover: "app"})
if err != nil || len(shown.Pipeline.Steps) != len(p.Steps) || shown.Pipeline.Steps[0].Module != "app" {
t.Fatalf("show of %s naming app = %d steps (%v), want its %d", key, len(shown.Pipeline.Steps), err, len(p.Steps))
}
inline := p
inline.Pipeline = "main.2"
inlineRaw, _ := json.Marshal(inline)
if _, err := pipelines.Put(ctx, "main.2", inlineRaw); err != nil {
t.Fatal(err)
}
store := newPipelineStore(ctx, pipelines, func(f string, a ...any) { fmt.Fprintf(logs, f+"\n", a...) })
old, oldRev, err := store.Get("main.2")
if err != nil || len(old.Steps) != len(p.Steps) {
t.Fatalf("the inline record read %d steps (%v), want %d", len(old.Steps), err, len(p.Steps))
}
if _, err := store.Update("main.2", old, oldRev); err != nil {
t.Fatal(err)
}
var split seat.Pipeline
if !readPipeline(ctx, pipelines, "main.2", &split) || len(split.Steps) != len(p.Steps) {
t.Fatalf("the record split reads %d steps, want %d", len(split.Steps), len(p.Steps))
}
if _, err := pipelines.Get(ctx, pipeline.StepsKey("main.2", "app")); err != nil || !strings.Contains(logs.String(), "split into 1 mover keys") {
t.Fatalf("the mover key of main.2's app: %v; the journal says the split: %t", err, strings.Contains(logs.String(), "split into 1 mover keys"))
}
// §470, §330: the step log is the log's stream, read onward by sequence the way the ageing reads it. // §470, §330: the step log is the log's stream, read onward by sequence the way the ageing reads it.
logged, err := stepLog{ctx: ctx, js: js, module: module}.Read(1, 100) logged, err := stepLog{ctx: ctx, js: js, module: module}.Read(1, 100)
if err != nil || len(logged) == 0 || logged[0].Entry.Pipeline != key { if err != nil || len(logged) == 0 || logged[0].Entry.Pipeline != key {
@@ -273,16 +303,33 @@ func stepOf(p seat.Pipeline, step string) seat.StepResult {
return seat.StepResult{} return seat.StepResult{}
} }
// readPipeline reads pipeline key as the runner does, its steps composed from its mover keys, and holds its record to
// H-A-187: a record the runner wrote carries no steps of its own, so it never grows with the catalog.
func readPipeline(ctx context.Context, kv jetstream.KeyValue, key string, p *seat.Pipeline) bool { func readPipeline(ctx context.Context, kv jetstream.KeyValue, key string, p *seat.Pipeline) bool {
e, err := kv.Get(ctx, key) e, err := kv.Get(ctx, key)
if err != nil { if err != nil {
return false return false
} }
*p = seat.Pipeline{} var record seat.Pipeline
return json.Unmarshal(e.Value(), p) == nil if json.Unmarshal(e.Value(), &record) != nil || len(record.Steps) > 0 {
panic(fmt.Sprintf("the record of %s carries %d steps of its own; each mover's steps are its own key (H-A-187)", key, len(record.Steps)))
}
got, _, err := newPipelineStore(ctx, kv, nil).Get(key)
*p = got
return err == nil
} }
// waitFor asks cond until it holds or the test's context ends, the module's journal in the failure. // waitFor asks cond until it holds or the test's context ends, the module's journal in the failure.
// findingsStream makes the stream of the module's own finding events, as its manifest's streams ask the bus for
// (`mesh.stream.<module>.event.>`), so a finding the runner writes, such as the unhandled-event count of the SDK's view
// reader (H-A-166, H-A-169), is acknowledged at once and holds no event behind its 3 s wait for the bus.
func findingsStream(t *testing.T, ctx context.Context, js jetstream.JetStream, module string) {
t.Helper()
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: module + "-findings", Subjects: []string{"mesh.stream." + module + ".event.>"}}); err != nil {
t.Fatal(err)
}
}
func waitFor(t *testing.T, ctx context.Context, logs *syncBuffer, cond func() bool) { func waitFor(t *testing.T, ctx context.Context, logs *syncBuffer, cond func() bool) {
t.Helper() t.Helper()
for ctx.Err() == nil { for ctx.Err() == nil {
@@ -1,9 +1,7 @@
package main package main
import ( import (
"context"
"encoding/json" "encoding/json"
"errors"
"github.com/nats-io/nats.go/jetstream" "github.com/nats-io/nats.go/jetstream"
@@ -13,20 +11,6 @@ import (
"mesh-delivery-pipeline/internal/pipeline" "mesh-delivery-pipeline/internal/pipeline"
) )
// createdKV is the state pipelines written by compare-and-set on a key's absence (§54).
type createdKV struct {
ctx context.Context
kv jetstream.KeyValue
}
func (c createdKV) Create(key string, value []byte) error {
_, err := c.kv.Create(c.ctx, key, value)
if errors.Is(err, jetstream.ErrKeyExists) {
return pipeline.ErrExists
}
return err
}
// hearPublished hears each package-published (design 47 H §746, §747): in its source's running pipeline, or starting one // hearPublished hears each package-published (design 47 H §746, §747): in its source's running pipeline, or starting one
// pipeline of the package's kind-1 dependents that pipeline does not move. The message is acknowledged once that is // pipeline of the package's kind-1 dependents that pipeline does not move. The message is acknowledged once that is
// written, on a goroutine of its own, so a slow write holds no other event; the durable consumer delivers a publish heard // written, on a goroutine of its own, so a slow write holds no other event; the durable consumer delivers a publish heard
@@ -19,64 +19,74 @@ import (
"mesh-delivery-pipeline/internal/pipeline" "mesh-delivery-pipeline/internal/pipeline"
) )
// pipelineStore is the state pipelines as the hearer reads and writes it: a write by compare-and-set on the revision read (§27). // kvState is the state pipelines as bytes on the bus, by compare-and-set (§27); the runner reads and writes pipelines
type pipelineStore struct { // through one pipeline.MoverStore over it, which keeps each mover's steps under its own key (H-A-187).
type kvState struct {
ctx context.Context ctx context.Context
kv jetstream.KeyValue kv jetstream.KeyValue
} }
func (s pipelineStore) Get(key string) (seat.Pipeline, uint64, error) { // newPipelineStore is the process's one store of the state pipelines.
e, err := s.kv.Get(s.ctx, key) func newPipelineStore(ctx context.Context, kv jetstream.KeyValue, logf func(string, ...any)) *pipeline.MoverStore {
if err != nil { return &pipeline.MoverStore{Raw: kvState{ctx: ctx, kv: kv}, Logf: logf}
return seat.Pipeline{}, 0, err
}
var p seat.Pipeline
if err := json.Unmarshal(e.Value(), &p); err != nil {
return seat.Pipeline{}, 0, fmt.Errorf("the pipeline %s is no pipeline: %w", key, err)
}
return p, e.Revision(), nil
} }
func (s pipelineStore) Update(key string, p seat.Pipeline, revision uint64) (uint64, error) { func (s kvState) Get(key string) ([]byte, uint64, error) {
raw, err := json.Marshal(p) e, err := s.kv.Get(s.ctx, key)
if err != nil { if errors.Is(err, jetstream.ErrKeyNotFound) {
return 0, err return nil, 0, nil
} }
rev, err := s.kv.Update(s.ctx, key, raw, revision) if err != nil {
return nil, 0, err
}
return e.Value(), e.Revision(), nil
}
func (s kvState) Create(key string, value []byte) (uint64, error) {
rev, err := s.kv.Create(s.ctx, key, value)
if errors.Is(err, jetstream.ErrKeyExists) {
return 0, pipeline.ErrExists
}
return rev, err
}
func (s kvState) Update(key string, value []byte, revision uint64) (uint64, error) {
rev, err := s.kv.Update(s.ctx, key, value, revision)
if err != nil && (errors.Is(err, jetstream.ErrKeyExists) || strings.Contains(err.Error(), "wrong last sequence")) { if err != nil && (errors.Is(err, jetstream.ErrKeyExists) || strings.Contains(err.Error(), "wrong last sequence")) {
return 0, pipeline.ErrChanged return 0, pipeline.ErrChanged
} }
return rev, err return rev, err
} }
func (s pipelineStore) Keys(filter string) ([]string, error) { func (s kvState) Delete(key string) error {
all, err := s.kv.Keys(s.ctx) if err := s.kv.Delete(s.ctx, key); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
if errors.Is(err, jetstream.ErrNoKeysFound) { return err
return nil, nil
} }
return nil
}
// Keys are the keys matching filter, `*` one key part, as the bus filters a key-value's subjects.
func (s kvState) Keys(filter string) ([]string, error) {
l, err := s.kv.ListKeysFiltered(s.ctx, filter)
if err != nil {
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return nil, err
}
// The lister ends its own watch once the current keys are listed.
var out []string var out []string
want := strings.Split(filter, ".") for k := range l.Keys() {
for _, k := range all { out = append(out, k)
got := strings.Split(k, ".")
if len(got) != len(want) {
continue
}
ok := true
for i := range want {
ok = ok && (want[i] == "*" || want[i] == got[i])
}
if ok {
out = append(out, k)
}
} }
return out, err return out, nil
} }
// newHearer is the hearer over the runner's own states: pipelines, kept, the step log on its stream and step-outputs on its // newHearer is the hearer over the runner's own states: pipelines, kept, the step log on its stream and step-outputs on its
// objects (§27). Each store is opened where it is used, so one the grants part has not made yet fails that write loudly. // objects (§27). Each store is opened where it is used, so one the grants part has not made yet fails that write loudly.
func newHearer(ctx context.Context, js jetstream.JetStream, cred process.Account, pipelines, kept jetstream.KeyValue, func newHearer(ctx context.Context, js jetstream.JetStream, cred process.Account, pipelines pipeline.Updater, kept jetstream.KeyValue,
say func(string, any) error, logf func(string, ...any)) *pipeline.Hearer { say func(string, any) error, logf func(string, ...any)) *pipeline.Hearer {
return &pipeline.Hearer{Store: pipelineStore{ctx: ctx, kv: pipelines}, Say: say, Logf: logf, Name: cred.Module, return &pipeline.Hearer{Store: pipelines, Say: say, Logf: logf, Name: cred.Module,
Log: stepLog{ctx: ctx, js: js, module: cred.Module}.Append, Log: stepLog{ctx: ctx, js: js, module: cred.Module}.Append,
Keep: func(key string, k seat.KeptResult) error { Keep: func(key string, k seat.KeptResult) error {
raw, err := json.Marshal(k) raw, err := json.Marshal(k)
@@ -38,7 +38,7 @@ func answerVerbs(conn *process.Conn, seats *seatlist.Part, v pipeline.Verbs) (fu
return conn.AnswerSeat(seat.Seat, seatlist.Definition(seats.Reader, seat.Seat), writeHandlers(v)) return conn.AnswerSeat(seat.Seat, seatlist.Definition(seats.Reader, seat.Seat), writeHandlers(v))
} }
// writeHandlers are the seat's write verbs the process answers: run, review, rerun, accept, cancel, start, and ask-again; and // writeHandlers are the seat's write verbs the process answers: run, review, rerun, accept, cancel, end-again, start, and ask-again; and
// the test runs' verbs test-runs and test-runs-drop (A-469g, A-470i). // the test runs' verbs test-runs and test-runs-drop (A-469g, A-470i).
func writeHandlers(v pipeline.Verbs) map[string]process.SeatHandler { func writeHandlers(v pipeline.Verbs) map[string]process.SeatHandler {
return map[string]process.SeatHandler{ return map[string]process.SeatHandler{
@@ -47,6 +47,7 @@ func writeHandlers(v pipeline.Verbs) map[string]process.SeatHandler {
seat.VerbRerun: verbOf(func(c string, in seat.RerunInput) (any, error) { return v.Rerun(c, in) }), seat.VerbRerun: verbOf(func(c string, in seat.RerunInput) (any, error) { return v.Rerun(c, in) }),
seat.VerbAccept: verbOf(func(c string, in seat.AcceptInput) (any, error) { return v.Accept(c, in) }), seat.VerbAccept: verbOf(func(c string, in seat.AcceptInput) (any, error) { return v.Accept(c, in) }),
seat.VerbCancel: verbOf(func(c string, in seat.CancelInput) (any, error) { return v.Cancel(c, in) }), seat.VerbCancel: verbOf(func(c string, in seat.CancelInput) (any, error) { return v.Cancel(c, in) }),
seat.VerbEndAgain: verbOf(func(c string, in seat.EndAgainInput) (any, error) { return v.EndAgain(c, in) }),
seat.VerbPlace: verbOf(func(c string, in seat.PlaceInput) (any, error) { return v.Place(c, in) }), seat.VerbPlace: verbOf(func(c string, in seat.PlaceInput) (any, error) { return v.Place(c, in) }),
seat.VerbStart: verbOf(func(c string, in seat.StartInput) (any, error) { return v.Start(c, in) }), seat.VerbStart: verbOf(func(c string, in seat.StartInput) (any, error) { return v.Start(c, in) }),
seat.VerbAskAgain: verbOf(func(c string, in seat.AskAgainInput) (any, error) { return v.AskAgain(c, in) }), seat.VerbAskAgain: verbOf(func(c string, in seat.AskAgainInput) (any, error) { return v.AskAgain(c, in) }),
@@ -23,7 +23,7 @@ func TestEveryWriteVerbIsServed(t *testing.T) {
got = append(got, v) got = append(got, v)
} }
slices.Sort(got) slices.Sort(got)
want := []string{seat.VerbAccept, seat.VerbAskAgain, seat.VerbCancel, seat.VerbPlace, seat.VerbRerun, seat.VerbReview, seat.VerbStart, seat.VerbRun, seat.VerbTestRuns, seat.VerbTestRunsDrop} want := []string{seat.VerbAccept, seat.VerbAskAgain, seat.VerbCancel, seat.VerbEndAgain, seat.VerbPlace, seat.VerbRerun, seat.VerbReview, seat.VerbStart, seat.VerbRun, seat.VerbTestRuns, seat.VerbTestRunsDrop}
slices.Sort(want) slices.Sort(want)
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Errorf("the process answers %v, want %v", got, want) t.Errorf("the process answers %v, want %v", got, want)
@@ -0,0 +1,396 @@
package pipeline
import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"sort"
"strings"
"sync"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// A pipeline's steps live one key per mover (design 47 H-A-187): the state pipelines keeps the pipeline's record (its
// movers, tiers, call, source and outcome) under the pipeline's key with its own steps empty, and each mover's steps under
// `steps.<pipeline>.<mover>`, written by compare-and-set as the record is, so a record never grows with the catalog and no
// value nears the bus's message limit (ADR 0486). Every read of a pipeline composes its steps from the mover keys it names.
// StepsKey is the key of one mover's steps in the state pipelines, `steps.<pipeline>.<mover>`, the mover's module escaped
// as a key's name is (H-A-187).
func StepsKey(pipeline, mover string) string {
return seat.PipelinesByStepsPipelineMoverKey(pipeline, keys.EscapeName(mover))
}
// stepsPrefix is the start of every mover key of one pipeline.
func stepsPrefix(pipeline string) string { return StepsKey(pipeline, "") }
// IsStepsKey says whether a key of the state pipelines is a mover's steps rather than a pipeline's record: `steps.` and a
// pipeline key of a take-in's (4 parts), a run's (5 parts, its third `run`) or a publish's (7 parts) and one mover.
func IsStepsKey(key string) bool {
parts := strings.Split(key, ".")
if parts[0] != "steps" {
return false
}
switch len(parts) {
case 4, 7:
return true
case 5:
return parts[2] == runToken
}
return false
}
// RawState is the state pipelines as bytes: a key's value and revision (nil, 0 and no error when it has none), a create by
// compare-and-set on absence (ErrExists), an update on the revision read (ErrChanged), a delete, and the keys matching a
// filter, `*` one key part.
type RawState interface {
Get(key string) ([]byte, uint64, error)
Create(key string, value []byte) (uint64, error)
Update(key string, value []byte, revision uint64) (uint64, error)
Delete(key string) error
Keys(filter string) ([]string, error)
}
// ErrNoPipeline is a read of a pipeline key the state pipelines does not hold.
var ErrNoPipeline = errors.New("no such pipeline")
// MoverStore is the state pipelines as the runner reads and writes it (§27, H-A-187): Get composes a pipeline from its
// record and its mover keys, Update writes the record and each mover key that changed, each by compare-and-set on the
// revision read, and nothing when any of them was written since. One MoverStore serves a process: a write and a read of one
// pipeline take its lock, so the process never reads a record and mover keys of two writes.
type MoverStore struct {
Raw RawState
Logf func(string, ...any)
mu sync.Mutex
locks map[string]*sync.Mutex
read map[string]readAt
inline map[string]bool
}
// readAt is what the last read of a pipeline found: its record's revision and each mover key's revision.
type readAt struct {
rev uint64
movers map[string]uint64
}
func (s *MoverStore) lock(key string) func() {
s.mu.Lock()
if s.locks == nil {
s.locks, s.read, s.inline = map[string]*sync.Mutex{}, map[string]readAt{}, map[string]bool{}
}
l, ok := s.locks[key]
if !ok {
l = &sync.Mutex{}
s.locks[key] = l
}
s.mu.Unlock()
l.Lock()
return l.Unlock
}
func (s *MoverStore) logf(f string, a ...any) {
if s.Logf != nil {
s.Logf(f, a...)
}
}
// moverKeys are the mover keys of pipeline key with their values and revisions.
func (s *MoverStore) moverKeys(key string) (map[string][]byte, map[string]uint64, error) {
ks, err := s.Raw.Keys(stepsPrefix(key) + "*")
if err != nil {
return nil, nil, fmt.Errorf("the mover keys of %s are not listed: %w", key, err)
}
vals, revs := map[string][]byte{}, map[string]uint64{}
for _, k := range ks {
v, rev, err := s.Raw.Get(k)
if err != nil {
return nil, nil, fmt.Errorf("%s is not read: %w", k, err)
}
if v != nil {
vals[k], revs[k] = v, rev
}
}
return vals, revs, nil
}
// composeSteps sets p's steps from its mover keys, in the order of p's movers, a key no mover names last in key order, and
// its unlogged step-log entries: the record's own (an entry naming no mover, written before H-A-190) and each mover's, in
// the order they ended (H-A-189).
func composeSteps(p *seat.Pipeline, key string, vals map[string][]byte) error {
var names []string
for k := range vals {
names = append(names, k)
}
sort.Strings(names)
order := make([]string, 0, len(names))
for _, m := range p.Movers {
if k := StepsKey(key, m.Module); vals[k] != nil && !slices.Contains(order, k) {
order = append(order, k)
}
}
for _, k := range names {
if !slices.Contains(order, k) {
order = append(order, k)
}
}
steps := []seat.StepResult{}
unlogged := append([]seat.LogEntry(nil), p.Unlogged...)
for _, k := range order {
v, err := seat.DecodePipelinesByStepsPipelineMoverValue(vals[k])
if err != nil {
return fmt.Errorf("%s holds no mover's steps: %w", k, err)
}
steps = append(steps, v.Steps...)
unlogged = append(unlogged, v.Unlogged...)
}
sort.SliceStable(unlogged, func(i, j int) bool { return unlogged[i].At.Before(unlogged[j].At) })
p.Steps, p.Unlogged = steps, unlogged
return nil
}
// split is p's record, its own steps and unlogged entries empty, and each mover's steps and unlogged step-log entries by
// its mover key (H-A-187, H-A-189). An entry naming no mover (one written before H-A-190) goes under the mover whose step
// it alone matches, else it stays in the record.
func split(key string, p seat.Pipeline) ([]byte, map[string][]byte, error) {
by := map[string]*seat.PipelinesByStepsPipelineMoverValue{}
at := func(module string) *seat.PipelinesByStepsPipelineMoverValue {
k := StepsKey(key, module)
if by[k] == nil {
by[k] = &seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{}}
}
return by[k]
}
for _, st := range p.Steps {
if st.Module == "" {
return nil, nil, fmt.Errorf("a step %s of %s names no mover, so it has no mover key", st.Step, key)
}
v := at(st.Module)
v.Steps = append(v.Steps, st)
}
var kept []seat.LogEntry
for _, e := range p.Unlogged {
module := e.Module
if module == "" {
module = moverOfEntry(p, e)
}
if module == "" {
kept = append(kept, e)
continue
}
v := at(module)
v.Unlogged = append(v.Unlogged, e)
}
movers := map[string][]byte{}
for k, v := range by {
raw, err := json.Marshal(v)
if err != nil {
return nil, nil, err
}
movers[k] = raw
}
p.Steps, p.Unlogged = []seat.StepResult{}, kept
record, err := json.Marshal(p)
return record, movers, err
}
// moverOfEntry is the mover of the one step an entry naming no mover matches by step and artifact, empty when none or
// more than one does.
func moverOfEntry(p seat.Pipeline, e seat.LogEntry) string {
found := ""
for _, st := range p.Steps {
if st.Step != e.Step || st.Artifact != e.Artifact {
continue
}
if found != "" && found != st.Module {
return ""
}
found = st.Module
}
return found
}
// Get is pipeline key with every mover's steps and the record's revision. A record that still carries its steps inline (one
// written before H-A-187) is read whole, said once in the journal, and split at its next write.
func (s *MoverStore) Get(key string) (seat.Pipeline, uint64, error) {
defer s.lock(key)()
raw, rev, err := s.Raw.Get(key)
if err != nil {
return seat.Pipeline{}, 0, err
}
if raw == nil {
return seat.Pipeline{}, 0, fmt.Errorf("%w: %s", ErrNoPipeline, key)
}
var p seat.Pipeline
if err := json.Unmarshal(raw, &p); err != nil {
return seat.Pipeline{}, 0, fmt.Errorf("the pipeline %s is no pipeline: %w", key, err)
}
vals, revs, err := s.moverKeys(key)
if err != nil {
return seat.Pipeline{}, 0, err
}
s.read[key] = readAt{rev: rev, movers: revs}
if len(p.Steps) > 0 {
if !s.inline[key] {
s.inline[key] = true
s.logf("the pipeline %s carries its %d steps inline in its record: they are read whole and split into one key per mover at its next write (H-A-187)", key, len(p.Steps))
}
return p, rev, nil
}
if p.Steps == nil {
p.Steps = []seat.StepResult{}
}
return p, rev, composeSteps(&p, key, vals)
}
// Update writes p as pipeline key's record and its mover keys, each by compare-and-set: nothing is written, ErrChanged,
// when the record or a mover key was written since the read revision gave. A record carrying its steps inline has its
// mover keys written before the record, so its steps are never lost between the two.
func (s *MoverStore) Update(key string, p seat.Pipeline, revision uint64) (uint64, error) {
defer s.lock(key)()
record, want, err := split(key, p)
if err != nil {
return 0, err
}
raw, cur, err := s.Raw.Get(key)
if err != nil {
return 0, err
}
if raw == nil {
return 0, fmt.Errorf("%w: %s", ErrNoPipeline, key)
}
if cur != revision {
return 0, ErrChanged
}
vals, revs, err := s.moverKeys(key)
if err != nil {
return 0, err
}
// A mover key written since the read is another write between: the read named its revision.
if seen, ok := s.read[key]; ok && seen.rev == revision {
for k, r := range revs {
if seen.movers[k] != r {
return 0, ErrChanged
}
}
for k := range seen.movers {
if _, ok := revs[k]; !ok {
return 0, ErrChanged
}
}
}
var was seat.Pipeline
inline := json.Unmarshal(raw, &was) == nil && len(was.Steps) > 0
var after map[string]uint64
if inline {
if after, err = s.writeMovers(key, want, vals, revs); err != nil {
return 0, err
}
}
rev, err := s.Raw.Update(key, record, revision)
if err != nil {
return 0, err
}
if !inline {
if after, err = s.writeMovers(key, want, vals, revs); err != nil {
return 0, err
}
} else {
s.logf("the pipeline %s's steps are split into %d mover keys and its record no longer carries them (H-A-187)", key, len(want))
delete(s.inline, key)
}
s.read[key] = readAt{rev: rev, movers: after}
return rev, nil
}
// writeMovers writes each mover key whose steps changed and deletes each that p no longer names, by compare-and-set on the
// revisions read, and answers every mover key's revision after it.
func (s *MoverStore) writeMovers(key string, want, vals map[string][]byte, revs map[string]uint64) (map[string]uint64, error) {
after := map[string]uint64{}
for _, k := range slices.Sorted(maps.Keys(want)) {
v := want[k]
old, ok := vals[k]
var rev uint64
var err error
switch {
case ok && string(old) == string(v):
after[k] = revs[k]
continue
case ok:
rev, err = s.Raw.Update(k, v, revs[k])
default:
rev, err = s.Raw.Create(k, v)
}
if err != nil {
return nil, fmt.Errorf("the steps of %s are not written: %w", k, err)
}
after[k] = rev
}
for k := range vals {
if _, ok := want[k]; !ok {
if err := s.Raw.Delete(k); err != nil {
return nil, fmt.Errorf("the steps of %s are not deleted: %w", k, err)
}
}
}
return after, nil
}
// Create writes a new pipeline, its record by compare-and-set on its absence (ErrExists: the second start writes nothing,
// §54), then its mover keys; a mover key an earlier pipeline of the key left is written over, said in the journal.
func (s *MoverStore) Create(key string, value []byte) error {
var p seat.Pipeline
if err := json.Unmarshal(value, &p); err != nil {
return fmt.Errorf("the pipeline %s is no pipeline: %w", key, err)
}
defer s.lock(key)()
record, want, err := split(key, p)
if err != nil {
return err
}
rev, err := s.Raw.Create(key, record)
if err != nil {
return err
}
vals, revs, err := s.moverKeys(key)
if err != nil {
return err
}
for k := range vals {
s.logf("%s was left by an earlier pipeline of the key: it is written over (H-A-187)", k)
}
after, err := s.writeMovers(key, want, vals, revs)
if err != nil {
return err
}
s.read[key] = readAt{rev: rev, movers: after}
return nil
}
// Delete removes pipeline key's mover keys, then its record (§330): a delete stopped between leaves the record, which the
// next ageing deletes again.
func (s *MoverStore) Delete(key string) error {
unlock := s.lock(key)
vals, _, err := s.moverKeys(key)
if err == nil {
for k := range vals {
if err = s.Raw.Delete(k); err != nil {
break
}
}
}
if err == nil {
err = s.Raw.Delete(key)
}
delete(s.read, key)
delete(s.inline, key)
unlock()
return err
}
// Keys are the keys of the state pipelines matching filter, `*` one key part.
func (s *MoverStore) Keys(filter string) ([]string, error) { return s.Raw.Keys(filter) }
@@ -0,0 +1,254 @@
package pipeline
import (
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
)
// memRaw is the state pipelines as bytes, one revision sequence for the bucket as a key-value's stream has.
type memRaw struct {
vals map[string][]byte
revs map[string]uint64
seq uint64
}
func newMemRaw() *memRaw { return &memRaw{vals: map[string][]byte{}, revs: map[string]uint64{}} }
func (m *memRaw) Get(key string) ([]byte, uint64, error) {
v, ok := m.vals[key]
if !ok {
return nil, 0, nil
}
return append([]byte(nil), v...), m.revs[key], nil
}
func (m *memRaw) put(key string, value []byte) uint64 {
m.seq++
m.vals[key], m.revs[key] = append([]byte(nil), value...), m.seq
return m.seq
}
func (m *memRaw) Create(key string, value []byte) (uint64, error) {
if _, ok := m.vals[key]; ok {
return 0, ErrExists
}
return m.put(key, value), nil
}
func (m *memRaw) Update(key string, value []byte, rev uint64) (uint64, error) {
if m.revs[key] != rev {
return 0, ErrChanged
}
return m.put(key, value), nil
}
func (m *memRaw) Delete(key string) error {
delete(m.vals, key)
delete(m.revs, key)
return nil
}
func (m *memRaw) Keys(filter string) ([]string, error) {
var out []string
for k := range m.vals {
if ok, _ := matchKey(filter, k); ok {
out = append(out, k)
}
}
return out, nil
}
// recordSteps is the steps the raw record of key carries itself.
func recordSteps(t *testing.T, m *memRaw, key string) int {
t.Helper()
var p seat.Pipeline
if err := json.Unmarshal(m.vals[key], &p); err != nil {
t.Fatal(err)
}
return len(p.Steps)
}
// TestMoverKeys holds the state pipelines' shape to H-A-187: a pipeline's record keeps its movers and outcome, each mover's
// steps their own key written by compare-and-set as the record is, a read composing them, a record carrying its steps inline
// read whole and split at its next write, one row per case.
func TestMoverKeys(t *testing.T) {
const key = "main.7"
two := func() seat.Pipeline {
return seat.Pipeline{V: 1, Pipeline: key, Branch: "main", Source: 7, Outcome: seat.PipelineOutcomeRunning,
Movers: []seat.Mover{{Module: "beta"}, {Module: "alpha"}},
Steps: []seat.StepResult{{Module: "alpha", Step: seat.StepResultStepBuild, Outcome: seat.StepResultOutcomeRunning},
{Module: "beta", Step: seat.StepResultStepBuild, Outcome: seat.StepResultOutcomeRunning},
{Module: "beta", Step: seat.StepResultStepApproval, Outcome: seat.StepResultOutcomeNotAsked}}}
}
created := func(t *testing.T) (*memRaw, *MoverStore, *[]string) {
m, logs := newMemRaw(), &[]string{}
s := &MoverStore{Raw: m, Logf: func(f string, a ...any) { *logs = append(*logs, fmt.Sprintf(f, a...)) }}
raw, _ := json.Marshal(two())
if err := s.Create(key, raw); err != nil {
t.Fatal(err)
}
return m, s, logs
}
steps := func(p seat.Pipeline) string {
var out []string
for _, st := range p.Steps {
out = append(out, st.Module+" "+st.Step+" "+st.Outcome)
}
return strings.Join(out, "; ")
}
rows := []struct {
name string
run func(t *testing.T) string
want string
}{
{"a write keeps the record's steps empty and each mover's under its own key; a read composes them in the movers' order", func(t *testing.T) string {
m, s, _ := created(t)
p, _, err := s.Get(key)
return fmt.Sprintf("record %d, alpha %t, beta %t: %s %v", recordSteps(t, m, key), m.vals["steps.main.7.alpha"] != nil,
m.vals["steps.main.7.beta"] != nil, steps(p), err)
}, "record 0, alpha true, beta true: beta build running; beta approval not asked; alpha build running <nil>"},
{"an update writes the mover whose steps changed and leaves the other's key as it was", func(t *testing.T) string {
m, s, _ := created(t)
p, rev, _ := s.Get(key)
beta := m.revs["steps.main.7.beta"]
p.Steps[2].Outcome = seat.StepResultOutcomePassed
_, err := s.Update(key, p, rev)
q, _, _ := s.Get(key)
return fmt.Sprintf("beta kept %t: %s %v", m.revs["steps.main.7.beta"] == beta, steps(q), err)
}, "beta kept true: beta build running; beta approval not asked; alpha build passed <nil>"},
{"a mover key written since the read refuses the update, nothing written", func(t *testing.T) string {
m, s, _ := created(t)
p, rev, _ := s.Get(key)
m.put("steps.main.7.alpha", m.vals["steps.main.7.alpha"])
p.Outcome = seat.PipelineOutcomePassed
_, err := s.Update(key, p, rev)
return fmt.Sprintf("%t, record at %t", errors.Is(err, ErrChanged), m.revs[key] == rev)
}, "true, record at true"},
{"a record written since the read refuses the update", func(t *testing.T) string {
_, s, _ := created(t)
p, rev, _ := s.Get(key)
if _, err := s.Update(key, p, rev); err != nil {
return err.Error()
}
_, err := s.Update(key, p, rev)
return fmt.Sprint(errors.Is(err, ErrChanged))
}, "true"},
{"a record carrying its steps inline is read whole, said once, and split at its next write", func(t *testing.T) string {
m, logs := newMemRaw(), &[]string{}
s := &MoverStore{Raw: m, Logf: func(f string, a ...any) { *logs = append(*logs, fmt.Sprintf(f, a...)) }}
raw, _ := json.Marshal(two())
m.put(key, raw)
p, rev, _ := s.Get(key)
_, _, _ = s.Get(key)
read := fmt.Sprintf("read %d steps, %d lines", len(p.Steps), len(*logs))
_, err := s.Update(key, p, rev)
q, _, _ := s.Get(key)
return fmt.Sprintf("%s; written: record %d, %s, split said %t %v", read, recordSteps(t, m, key), steps(q),
strings.Contains(strings.Join(*logs, "\n"), "split into 2 mover keys"), err)
}, "read 3 steps, 1 lines; written: record 0, beta build running; beta approval not asked; alpha build running, split said true <nil>"},
{"a second create writes nothing", func(t *testing.T) string {
m, s, _ := created(t)
p := two()
p.Steps[0].Outcome = seat.StepResultOutcomeFailed
raw, _ := json.Marshal(p)
err := s.Create(key, raw)
q, _, _ := s.Get(key)
return fmt.Sprintf("%t %d: %s", errors.Is(err, ErrExists), len(m.vals), steps(q))
}, "true 3: beta build running; beta approval not asked; alpha build running"},
{"a delete removes the mover keys and the record", func(t *testing.T) string {
m, s, _ := created(t)
err := s.Delete(key)
return fmt.Sprintf("%d %v", len(m.vals), err)
}, "0 <nil>"},
{"a record of 129 movers never grows with their steps (main.26)", func(t *testing.T) string {
m, s, _ := created(t)
p, rev, _ := s.Get(key)
p.Movers, p.Steps = nil, nil
for i := range 129 {
mod := fmt.Sprintf("module-%03d", i)
p.Movers = append(p.Movers, seat.Mover{Module: mod, Commit: strings.Repeat("c", 40), Fingerprint: "fp-" + mod})
for _, step := range []string{seat.StepResultStepBuild, seat.StepResultStepLint, seat.StepResultStepUnit,
seat.StepResultStepIntegration, seat.StepResultStepValidate, seat.StepResultStepComposition,
seat.StepResultStepReview, seat.StepResultStepApproval} {
p.Steps = append(p.Steps, seat.StepResult{Module: mod, Step: step, Artifact: mod + ":program",
Outcome: seat.StepResultOutcomeFailed, Why: strings.Repeat("why ", 250)})
}
}
if _, err := s.Update(key, p, rev); err != nil {
return err.Error()
}
largest := 0
for k, v := range m.vals {
if IsStepsKey(k) && len(v) > largest {
largest = len(v)
}
}
q, _, _ := s.Get(key)
return fmt.Sprintf("record under 64 KiB %t, a mover key under 16 KiB %t, %d steps read", len(m.vals[key]) < 64<<10,
largest < 16<<10, len(q.Steps))
}, "record under 64 KiB true, a mover key under 16 KiB true, 1032 steps read"},
{"a mover's unlogged entries live under its mover key, each filed by the mover it names (H-A-189, H-A-190)", func(t *testing.T) string {
m, s, _ := created(t)
p, rev, _ := s.Get(key)
at := time.Date(2026, 11, 20, 20, 0, 0, 0, time.UTC)
// Two movers' approvals end alike at one moment; each entry names its mover.
p.Unlogged = []seat.LogEntry{{Pipeline: key, Module: "beta", Step: seat.StepResultStepApproval, What: seat.LogEntryWhatEnded, At: at},
{Pipeline: key, Module: "alpha", Step: seat.StepResultStepApproval, What: seat.LogEntryWhatEnded, At: at.Add(time.Second)}}
if _, err := s.Update(key, p, rev); err != nil {
return err.Error()
}
var rec seat.Pipeline
_ = json.Unmarshal(m.vals[key], &rec)
beta, _ := seat.DecodePipelinesByStepsPipelineMoverValue(m.vals["steps.main.7.beta"])
q, _, _ := s.Get(key)
var got []string
for _, e := range q.Unlogged {
got = append(got, e.Module)
}
return fmt.Sprintf("record %d unlogged, beta's key %d (%s), read %s", len(rec.Unlogged), len(beta.Unlogged),
beta.Unlogged[0].Module, strings.Join(got, " "))
}, "record 0 unlogged, beta's key 1 (beta), read beta alpha"},
{"a 129-mover pipeline with 1,000 unlogged entries keeps its record small (H-A-189)", func(t *testing.T) string {
m, s, _ := created(t)
p, rev, _ := s.Get(key)
p.Movers, p.Steps, p.Unlogged = nil, nil, nil
at := time.Date(2026, 11, 20, 20, 0, 0, 0, time.UTC)
for i := range 129 {
mod := fmt.Sprintf("module-%03d", i)
p.Movers = append(p.Movers, seat.Mover{Module: mod, Commit: strings.Repeat("c", 40), Fingerprint: "fp-" + mod})
p.Steps = append(p.Steps, seat.StepResult{Module: mod, Step: seat.StepResultStepBuild, Outcome: seat.StepResultOutcomeNotAsked})
}
for i := range 1000 {
mod := fmt.Sprintf("module-%03d", i%129)
p.Unlogged = append(p.Unlogged, seat.LogEntry{Pipeline: key, Module: mod, Step: seat.StepResultStepBuild, Artifact: mod + ":program",
What: seat.LogEntryWhatEnded, Outcome: seat.StepResultOutcomeNotAsked, Output: strings.Repeat("o", 200), At: at.Add(time.Duration(i) * time.Millisecond)})
}
if _, err := s.Update(key, p, rev); err != nil {
return err.Error()
}
q, _, _ := s.Get(key)
return fmt.Sprintf("record under 64 KiB %t, %d unlogged read", len(m.vals[key]) < 64<<10, len(q.Unlogged))
}, "record under 64 KiB true, 1000 unlogged read"},
{"mover keys are told from records of every key form", func(t *testing.T) string {
var out []string
for _, k := range []string{"steps.main.7.alpha", "steps.main.run.ab12.alpha", "steps.main.7.published.p.v.alpha",
"main.7", "steps.7", "steps.run.ab12", "steps.7.published.p.v"} {
out = append(out, fmt.Sprint(IsStepsKey(k)))
}
return strings.Join(out, " ")
}, "true true true false false false false"},
}
for _, r := range rows {
t.Run(r.name, func(t *testing.T) {
if got := r.run(t); got != r.want {
t.Errorf("got %s\nwant %s", got, r.want)
}
})
}
}
@@ -137,7 +137,9 @@ func components(movers []seat.Mover, after map[string][]string) map[string]int {
var buildKinds = []int64{3, 1} var buildKinds = []int64{3, 1}
// awaited are the artifact references mover m's steps wait for in p (§746, H-A-51): the From of each edge of a build kind // awaited are the artifact references mover m's steps wait for in p (§746, H-A-51): the From of each edge of a build kind
// into m from another mover of p, toolchain edges first, each once. // into m from another mover of p, toolchain edges first, each once. A toolchain edge (kind 3) from a build module whose
// own new version is built from m is not awaited: what a build module's new version is built from is built by its
// running version, never waited for (G §270, A-326), so the running build module builds m and the new one the rest.
func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) []catalog.Edge { func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) []catalog.Edge {
var out []catalog.Edge var out []catalog.Edge
for _, kind := range buildKinds { for _, kind := range buildKinds {
@@ -146,12 +148,35 @@ func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry)
if x.Kind != kind || from == m || !Moves(p, from) || slices.ContainsFunc(out, func(e catalog.Edge) bool { return e.From == x.From }) { if x.Kind != kind || from == m || !Moves(p, from) || slices.ContainsFunc(out, func(e catalog.Edge) bool { return e.From == x.From }) {
continue continue
} }
if kind == 3 && builtFrom(p, from, entries)[m] {
continue
}
out = append(out, x) out = append(out, x)
} }
} }
return out return out
} }
// builtFrom are the movers of p that module's new version is built from (G §270): the From mover of each edge of a build
// kind into it, and what those are built from in turn.
func builtFrom(p seat.Pipeline, module string, entries map[string]catalog.CatalogEntry) map[string]bool {
seen := map[string]bool{}
next := []string{module}
for len(next) > 0 {
cur := next[0]
next = next[1:]
for _, x := range entries[cur].Edges {
from := moduleOf(x.From)
if !slices.Contains(buildKinds, x.Kind) || from == cur || seen[from] || !Moves(p, from) {
continue
}
seen[from] = true
next = append(next, from)
}
}
return seen
}
// ready says whether mover m of p may be placed (§746, H-A-84): every artifact it awaits is in its From mover's published. // ready says whether mover m of p may be placed (§746, H-A-84): every artifact it awaits is in its From mover's published.
// It answers the references still awaited, and the first From mover whose build ended without passing, so m never builds. // It answers the references still awaited, and the first From mover whose build ended without passing, so m never builds.
func ready(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) (bool, []string, string) { func ready(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) (bool, []string, string) {
@@ -4,6 +4,7 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
@@ -124,3 +125,115 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
} }
} }
} }
// Design 47 G §270, A-326 (H §746, H-A-83 (2)): what a build module's new version is built from is built by its running
// version, never waited for: a mover the moving build module imports, directly or through another mover, does not await
// that build module's toolchain edge, so a pipeline moving mesh-sdk or the contracts with the Go build module places them
// at once (main.28, main.30 of 2026-11-20 waited on each other forever); every other mover still awaits the new one.
func TestAToolchainEdgeIsNotAwaitedByTheBuildModulesOwnInputs(t *testing.T) {
e := func(from, to string, kind int64) catalog.Edge { return catalog.Edge{From: from, To: to, Kind: kind} }
entries := map[string]catalog.CatalogEntry{
"mesh-sdk": {Edges: []catalog.Edge{e("mesh-build-go:files", "mesh-sdk:go", 3)}},
"contracts": {Edges: []catalog.Edge{e("mesh-build-go:files", "contracts:go-a", 3), e("mesh-build-go:files", "contracts:go-b", 3)}},
"delivery-sdk": {Edges: []catalog.Edge{e("mesh-build-go:files", "delivery-sdk:go", 3), e("mesh-sdk:go", "delivery-sdk:go", 1)}},
"mesh-build-go": {Edges: []catalog.Edge{e("contracts:go-a", "mesh-build-go:tools", 1), e("delivery-sdk:go", "mesh-build-go:tools", 1), e("nats:tools", "mesh-build-go:files", 4)}},
"zfs": {Edges: []catalog.Edge{e("mesh-build-go:files", "zfs:tools", 3), e("contracts:go-a", "zfs:tools", 1)}},
"tool": {Edges: []catalog.Edge{e("builder:program", "tool:program", 3)}},
"builder": {},
}
pipe := func(modules ...string) seat.Pipeline {
var p seat.Pipeline
for _, m := range modules {
p.Movers = append(p.Movers, seat.Mover{Module: m})
}
return p
}
all := pipe("mesh-sdk", "contracts", "delivery-sdk", "mesh-build-go", "zfs")
for _, c := range []struct {
name string
p seat.Pipeline
mover string
want []string
}{
{"the build module's own package input awaits no toolchain edge", all, "contracts", nil},
{"an input through another mover awaits no toolchain edge", all, "mesh-sdk", nil},
{"an input that imports another input awaits only that package", all, "delivery-sdk", []string{"mesh-sdk:go"}},
{"the build module awaits its inputs' packages", all, "mesh-build-go", []string{"contracts:go-a", "delivery-sdk:go"}},
{"any other mover awaits the new build module first", all, "zfs", []string{"mesh-build-go:files", "contracts:go-a"}},
{"a build module that is not built from the mover is awaited", pipe("tool", "builder"), "tool", []string{"builder:program"}},
{"a build module that does not move is not awaited", pipe("tool"), "tool", nil},
} {
var got []string
for _, x := range awaited(c.p, c.mover, entries) {
got = append(got, x.From)
}
if !slices.Equal(got, c.want) {
t.Errorf("%s: %s awaits %v, want %v", c.name, c.mover, got, c.want)
}
}
}
// Design 47 H H-A-175, H-A-176 (G §270, A-326; H-A-101 (2), H-A-163): when every mover without steps waits only on movers
// without steps, their steps are placed ended not asked, the why `cycle` naming the edges inside the cycle alone; a step
// of another mover is asked as ever, and the pipeline ends by the end judgement, at once when no required step is open; a
// cycle through a build module's own inputs is broken (G §270) and placed; a mover waiting on a placed mover only waits.
func TestACycleAmongUnplacedMoversEndsThePipelineFailed(t *testing.T) {
reg, base := placeFixture()
reg.Policy = catalog.Policy{Steps: catalog.StepStatuses{Lint: "bypass", Unit: "bypass", Integration: "bypass", Validate: "bypass", Composition: "bypass"}}
entry := func(m string, edges ...catalog.Edge) catalog.CatalogEntry {
return catalog.CatalogEntry{Module: m, Artifacts: []catalog.Artifact{{Ref: m + ":go", Language: "go"}}, Edges: edges}
}
e := func(from, to string, kind int64) catalog.Edge { return catalog.Edge{From: from, To: to, Kind: kind} }
cycle := []catalog.CatalogEntry{entry("a", e("b:go", "a:go", 1)), entry("b", e("a:go", "b:go", 1))}
for _, c := range []struct {
name string
entries []catalog.CatalogEntry
outcome string
asked []string
notAsked []string
}{
{"a cycle G §270 breaks is placed", []catalog.CatalogEntry{
entry("mesh-sdk", e("mesh-build-go:files", "mesh-sdk:go", 3)),
entry("mesh-build-go", e("mesh-sdk:go", "mesh-build-go:go", 1))}, seat.PipelineOutcomeRunning, []string{"mesh-sdk"}, nil},
{"a cycle beside a free mover ends its own steps and the free one is asked",
append(slices.Clone(cycle), entry("c"), entry("d", e("a:go", "d:go", 1))), seat.PipelineOutcomeRunning, []string{"c"}, []string{"a", "b", "d"}},
{"a cycle alone ends the pipeline failed at once", cycle, seat.PipelineOutcomeFailed, nil, []string{"a", "b"}},
{"a mover waiting on a placed mover waits", []catalog.CatalogEntry{
entry("a", e("b:go", "a:go", 1)), entry("b")}, seat.PipelineOutcomeRunning, []string{"b"}, nil},
} {
entries := map[string]catalog.CatalogEntry{"mesh-build-go": base["mesh-build-go"]}
p := seat.Pipeline{Pipeline: "main.4", Branch: "main", Source: 4, Outcome: seat.PipelineOutcomeRunning}
for _, en := range c.entries {
entries[en.Module] = en
p.Movers = append(p.Movers, seat.Mover{Module: en.Module, Commit: "c", Fingerprint: "src2:" + en.Module})
}
s := &Starter{Name: "the-runner"}
asks, _, ended, notAsked := s.placeMovers(&p, func(string) (catalog.Registration, bool) { return reg, true }, entries, time.Unix(1, 0))
if p.Outcome != c.outcome || (c.outcome == seat.PipelineOutcomeFailed) != (len(ended) > 0) || (len(c.asked) > 0) != (len(asks) > 0) {
t.Errorf("%s: pipeline %s (%s), %d asks, %d ended; want %s", c.name, p.Outcome, p.Why, len(asks), len(ended), c.outcome)
}
said := map[int]bool{}
for _, i := range append(ended, notAsked...) {
said[i] = true
}
var asked, ends []string
for i, st := range p.Steps {
switch {
case st.Outcome == seat.StepResultOutcomeNotAsked && strings.HasPrefix(st.Why, "cycle: ") && said[i]:
if !slices.Contains(ends, st.Module) {
ends = append(ends, st.Module)
}
if st.Why != "cycle: a:go before b:go (kind 1); b:go before a:go (kind 1)" {
t.Errorf("%s: %s's %s why %q, want the cycle's two edges alone", c.name, st.Module, st.Step, st.Why)
}
case open(st) || st.Outcome == seat.StepResultOutcomeWaiting:
if !slices.Contains(asked, st.Module) {
asked = append(asked, st.Module)
}
}
}
if !slices.Equal(asked, c.asked) || !slices.Equal(ends, c.notAsked) {
t.Errorf("%s: open %v, ended not asked %v; want %v and %v", c.name, asked, ends, c.asked, c.notAsked)
}
}
}
@@ -98,8 +98,14 @@ func place(p seat.Pipeline, m seat.Mover, reg catalog.Registration, entries map[
build := seat.StepResult{Step: seat.StepResultStepBuild, Module: m.Module, Status: seat.StepResultStatusRequired, Fingerprint: m.Fingerprint, build := seat.StepResult{Step: seat.StepResultStepBuild, Module: m.Module, Status: seat.StepResultStatusRequired, Fingerprint: m.Fingerprint,
Seat: builds.Seat, Outcome: seat.StepResultOutcomeRunning, Started: &now, Ask: AskID(startOf(p, now), p.Pipeline, m.Module, "", seat.StepResultStepBuild)} Seat: builds.Seat, Outcome: seat.StepResultOutcomeRunning, Started: &now, Ask: AskID(startOf(p, now), p.Pipeline, m.Module, "", seat.StepResultStepBuild)}
steps = append(steps, build) steps = append(steps, build)
// A run on request's build step ask names no source (the seat mesh-builds' contract; H-A-162): its candidate's
// fingerprints are never in the builder's picture of the branch, so it waits for no revision there.
source := p.Source
if p.Run != "" {
source = 0
}
asks = append(asks, Asked{Seat: builds.Seat, ID: build.Ask, Commit: m.Commit, Ask: builds.StepRun{Pipeline: p.Pipeline, Step: builds.StepRunStepBuild, Branch: p.Branch, asks = append(asks, Asked{Seat: builds.Seat, ID: build.Ask, Commit: m.Commit, Ask: builds.StepRun{Pipeline: p.Pipeline, Step: builds.StepRunStepBuild, Branch: p.Branch,
Revision: p.Revision, Source: p.Source, Entries: []builds.BuildEntry{{Module: m.Module, Commit: m.Commit, Fingerprint: m.Fingerprint}}}}) Revision: p.Revision, Source: source, Entries: []builds.BuildEntry{{Module: m.Module, Commit: m.Commit, Fingerprint: m.Fingerprint}}}})
for _, a := range arts { for _, a := range arts {
name := a.Ref name := a.Ref
if _, after, ok := strings.Cut(a.Ref, ":"); ok { if _, after, ok := strings.Cut(a.Ref, ":"); ok {
@@ -39,7 +39,8 @@ func (r Reads) fills() ([]seat.Fill, error) {
return f, nil return f, nil
} }
// Show is one pipeline whole, with its steps and the names of their outputs in step-outputs (§83). // Show is one pipeline whole, with its steps and the names of their outputs in step-outputs (§83), or, naming a mover, the
// pipeline's record with that mover's steps alone (H-A-187).
func (r Reads) Show(in seat.ShowInput) (seat.ShowAnswer, error) { func (r Reads) Show(in seat.ShowInput) (seat.ShowAnswer, error) {
p, ok, err := r.pipeline(in.Pipeline) p, ok, err := r.pipeline(in.Pipeline)
if err != nil { if err != nil {
@@ -52,6 +53,13 @@ func (r Reads) Show(in seat.ShowInput) (seat.ShowAnswer, error) {
if err != nil { if err != nil {
return seat.ShowAnswer{}, err return seat.ShowAnswer{}, err
} }
// Named, a mover is answered by its steps alone (H-A-187): one mover's of a pipeline of a hundred movers.
if m := strings.TrimSpace(in.Mover); m != "" {
if !Moves(p, m) {
return seat.ShowAnswer{}, fmt.Errorf("the pipeline %s does not move %s", in.Pipeline, m)
}
p.Steps = stepsOf(p, m)
}
a := seat.ShowAnswer{Pipeline: p, Fill: fills} a := seat.ShowAnswer{Pipeline: p, Fill: fills}
for _, s := range p.Steps { for _, s := range p.Steps {
if s.Output != "" { if s.Output != "" {
@@ -62,7 +70,7 @@ func (r Reads) Show(in seat.ShowInput) (seat.ShowAnswer, error) {
} }
// List is the pipelines and runs of a module, a branch, a commit or a run, newest started first, at most Limit when it // List is the pipelines and runs of a module, a branch, a commit or a run, newest started first, at most Limit when it
// names one (§83). // names one (§83): each one's record with its movers and steps counted, its steps empty (H-A-191).
func (r Reads) List(in seat.ListInput) (seat.ListAnswer, error) { func (r Reads) List(in seat.ListInput) (seat.ListAnswer, error) {
all, err := r.pipelines() all, err := r.pipelines()
if err != nil { if err != nil {
@@ -72,6 +80,9 @@ func (r Reads) List(in seat.ListInput) (seat.ListAnswer, error) {
for _, p := range all { for _, p := range all {
if (in.Module == "" || Moves(p, in.Module)) && (in.Branch == "" || p.Branch == in.Branch) && if (in.Module == "" || Moves(p, in.Module)) && (in.Branch == "" || p.Branch == in.Branch) &&
(in.Commit == "" || movesCommit(p, in.Commit)) && (in.Run == "" || p.Run == in.Run) { (in.Commit == "" || movesCommit(p, in.Commit)) && (in.Run == "" || p.Run == in.Run) {
// Each pipeline's record with its movers and steps counted, never its steps: show gives those (H-A-191).
p.MoverCount, p.StepCount = int64(len(p.Movers)), int64(len(p.Steps))
p.Steps, p.Unlogged = []seat.StepResult{}, nil
out = append(out, p) out = append(out, p)
} }
} }
@@ -178,6 +189,16 @@ func (r Reads) pipeline(key string) (seat.Pipeline, bool, error) {
if key == "" { if key == "" {
return seat.Pipeline{}, false, fmt.Errorf("name the pipeline's key, <branch>.<revision>.<module>") return seat.Pipeline{}, false, fmt.Errorf("name the pipeline's key, <branch>.<revision>.<module>")
} }
all, err := r.Pipelines.Keys()
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s cannot be read: %w", seat.StatePipelines, err)
}
return r.pipelineOf(key, all)
}
// pipelineOf is pipeline key, its steps composed from the mover keys among all that it names (H-A-187); a record that
// still carries its steps inline is read whole.
func (r Reads) pipelineOf(key string, all []string) (seat.Pipeline, bool, error) {
raw, err := r.Pipelines.Value(key) raw, err := r.Pipelines.Value(key)
if err != nil { if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s cannot be read: %w", seat.StatePipelines, key, err) return seat.Pipeline{}, false, fmt.Errorf("%s %s cannot be read: %w", seat.StatePipelines, key, err)
@@ -192,7 +213,24 @@ func (r Reads) pipeline(key string) (seat.Pipeline, bool, error) {
if p.Pipeline == "" { if p.Pipeline == "" {
p.Pipeline = key p.Pipeline = key
} }
return p, true, nil if len(p.Steps) > 0 {
return p, true, nil
}
prefix := stepsPrefix(key)
vals := map[string][]byte{}
for _, k := range all {
if !strings.HasPrefix(k, prefix) || strings.Contains(k[len(prefix):], ".") {
continue
}
v, err := r.Pipelines.Value(k)
if err != nil {
return seat.Pipeline{}, false, fmt.Errorf("%s %s cannot be read: %w", seat.StatePipelines, k, err)
}
if v != nil {
vals[k] = v
}
}
return p, true, composeSteps(&p, key, vals)
} }
func (r Reads) pipelines() ([]seat.Pipeline, error) { func (r Reads) pipelines() ([]seat.Pipeline, error) {
@@ -203,7 +241,10 @@ func (r Reads) pipelines() ([]seat.Pipeline, error) {
sort.Strings(keys) sort.Strings(keys)
var out []seat.Pipeline var out []seat.Pipeline
for _, k := range keys { for _, k := range keys {
p, ok, err := r.pipeline(k) if IsStepsKey(k) {
continue
}
p, ok, err := r.pipelineOf(k, keys)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -3,6 +3,7 @@ package pipeline
import ( import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -49,6 +50,15 @@ func TestReads(t *testing.T) {
Outcome: seat.PipelineOutcomePassed, Started: &t2, Movers: []seat.Mover{{Module: "alpha", Commit: "b3"}}}) Outcome: seat.PipelineOutcomePassed, Started: &t2, Movers: []seat.Mover{{Module: "alpha", Commit: "b3"}}})
put(t, pipes, "main.7.gamma", seat.Pipeline{Pipeline: "main.7.gamma", Branch: "main", put(t, pipes, "main.7.gamma", seat.Pipeline{Pipeline: "main.7.gamma", Branch: "main",
Outcome: seat.PipelineOutcomeRunning, Started: &t1, Movers: []seat.Mover{{Module: "gamma", Commit: "c7"}}}) Outcome: seat.PipelineOutcomeRunning, Started: &t1, Movers: []seat.Mover{{Module: "gamma", Commit: "c7"}}})
// A pipeline kept as H-A-187 keeps it: its record's steps empty, each mover's under steps.<pipeline>.<mover>.
put(t, pipes, "next.8", seat.Pipeline{Pipeline: "next.8", Branch: "next", Source: 8, Outcome: seat.PipelineOutcomeRunning,
Started: &t0, Steps: []seat.StepResult{}, Movers: []seat.Mover{{Module: "delta", Commit: "n8"}, {Module: "epsilon", Commit: "n8"}}})
put(t, pipes, "steps.next.8.epsilon", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "epsilon",
Step: seat.StepResultStepBuild, Status: "required", Outcome: seat.StepResultOutcomePassed, Output: "next.8.build.epsilon:program"}}})
put(t, pipes, "steps.next.8.delta", seat.PipelinesByStepsPipelineMoverValue{Steps: []seat.StepResult{{Module: "delta",
Step: seat.StepResultStepBuild, Status: "required", Outcome: seat.StepResultOutcomePassed}, {Module: "delta",
Step: seat.StepResultStepApproval, Status: "required", Outcome: seat.StepResultOutcomeWaiting, On: seat.WaitOnPerson, Started: &t2}},
Unlogged: []seat.LogEntry{{Pipeline: "next.8", Module: "delta", Step: seat.StepResultStepBuild, What: seat.LogEntryWhatEnded, At: t1}}})
put(t, kept, "unit.sha256-aa", seat.KeptResult{Level: "unit", Digest: "sha256-aa", Package: "alpha/x"}) put(t, kept, "unit.sha256-aa", seat.KeptResult{Level: "unit", Digest: "sha256-aa", Package: "alpha/x"})
put(t, kept, "integration.sha256-bb", seat.KeptResult{Level: "integration", Digest: "sha256-bb", Package: "alpha/y"}) put(t, kept, "integration.sha256-bb", seat.KeptResult{Level: "integration", Digest: "sha256-bb", Package: "alpha/y"})
r := Reads{Pipelines: pipes, Kept: kept} r := Reads{Pipelines: pipes, Kept: kept}
@@ -73,6 +83,35 @@ func TestReads(t *testing.T) {
_, err := r.Show(seat.ShowInput{Pipeline: "main.9.alpha"}) _, err := r.Show(seat.ShowInput{Pipeline: "main.9.alpha"})
return "", err return "", err
}, `error: no pipeline "main.9.alpha" in pipelines`}, }, `error: no pipeline "main.9.alpha" in pipelines`},
{"show composes a pipeline's steps from its mover keys, in its movers' order (H-A-187)", func() (string, error) {
a, err := r.Show(seat.ShowInput{Pipeline: "next.8"})
var s []string
for _, st := range a.Pipeline.Steps {
s = append(s, st.Module+" "+st.Step)
}
return strings.Join(s, ", ") + " | " + strings.Join(a.Outputs, ","), err
}, "delta build, delta approval, epsilon build | next.8.build.epsilon:program"},
{"show naming a mover answers its steps alone", func() (string, error) {
a, err := r.Show(seat.ShowInput{Pipeline: "next.8", Mover: "epsilon"})
var s []string
for _, st := range a.Pipeline.Steps {
s = append(s, st.Module+" "+st.Step)
}
return fmt.Sprintf("%s, %d movers", strings.Join(s, ", "), len(a.Pipeline.Movers)), err
}, "epsilon build, 2 movers"},
{"show refuses a mover the pipeline does not move", func() (string, error) {
_, err := r.Show(seat.ShowInput{Pipeline: "next.8", Mover: "alpha"})
return "", err
}, "error: the pipeline next.8 does not move alpha"},
{"list names a pipeline, never a mover key, its record with its movers and steps counted and no step (H-A-191)", func() (string, error) {
a, err := r.List(seat.ListInput{Branch: "next"})
var s []string
for _, p := range a.Pipelines {
s = append(s, fmt.Sprintf("%s %d movers %d steps, %d steps and %d unlogged answered", p.Pipeline, p.MoverCount, p.StepCount,
len(p.Steps), len(p.Unlogged)))
}
return strings.Join(s, "; "), err
}, "next.8 2 movers 3 steps, 0 steps and 0 unlogged answered"},
{"list by module, newest first", func() (string, error) { {"list by module, newest first", func() (string, error) {
a, err := r.List(seat.ListInput{Module: "alpha"}) a, err := r.List(seat.ListInput{Module: "alpha"})
return names(a.Pipelines), err return names(a.Pipelines), err
@@ -92,7 +131,7 @@ func TestReads(t *testing.T) {
s = append(s, w.Pipeline+" "+w.Step+" "+w.On+" "+w.Since.Format("15:04")) s = append(s, w.Pipeline+" "+w.Step+" "+w.On+" "+w.Since.Format("15:04"))
} }
return strings.Join(s, ";"), err return strings.Join(s, ";"), err
}, "main.7.alpha approval person 10:01"}, }, "main.7.alpha approval person 10:01;next.8 approval person 10:02"},
{"why-carried names where a carried step came from", func() (string, error) { {"why-carried names where a carried step came from", func() (string, error) {
a, err := r.WhyCarried(seat.WhyCarriedInput{Pipeline: "main.7.alpha", Step: "unit", Artifact: "alpha:program"}) a, err := r.WhyCarried(seat.WhyCarriedInput{Pipeline: "main.7.alpha", Step: "unit", Artifact: "alpha:program"})
return a.CarriedFrom, err return a.CarriedFrom, err
@@ -5,6 +5,7 @@ import (
"strings" "strings"
"testing" "testing"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
) )
@@ -97,6 +98,19 @@ func TestRunOnRequest(t *testing.T) {
if ids, _ := s.UnplacedRuns(); len(ids) != 0 { if ids, _ := s.UnplacedRuns(); len(ids) != 0 {
t.Errorf("a placed run is still unplaced: %v", ids) t.Errorf("a placed run is still unplaced: %v", ids)
} }
// H-A-162: a run's build step asks name no source, as the seat mesh-builds' contract says of a run on request.
builtAsks := 0
for _, x := range asks.asks {
if r, ok := x.(builds.StepRun); ok {
builtAsks++
if r.Source != 0 {
t.Errorf("the run's build step ask names source %d, want none", r.Source)
}
}
}
if builtAsks == 0 {
t.Errorf("the run asked no build step: %v", asks.asks)
}
if n, _ := s.PlaceRun(ev, Half{Revision: 12, Entries: cand}, base); n != 0 { if n, _ := s.PlaceRun(ev, Half{Revision: 12, Entries: cand}, base); n != 0 {
t.Errorf("the revision heard again placed %d", n) t.Errorf("the revision heard again placed %d", n)
} }
@@ -366,7 +366,7 @@ func (h *Hearer) endReader(reader RunReader, run string, r Run) error {
if err != nil { if err != nil {
return err return err
} }
ended := endPipeline(&p, now) ended := endPipeline(&p, now, h.Unplaced)
if _, err := h.Store.Update(reader.Pipeline, p, rev); errors.Is(err, ErrChanged) { if _, err := h.Store.Update(reader.Pipeline, p, rev); errors.Is(err, ErrChanged) {
continue continue
} else if err != nil { } else if err != nil {
@@ -317,6 +317,7 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
var asks []Asked var asks []Asked
var unregistered []string var unregistered []string
var notAsked []int var notAsked []int
var waits []seat.Mover
for _, m := range p.Movers { for _, m := range p.Movers {
if placed(*p, m.Module) { if placed(*p, m.Module) {
continue continue
@@ -331,6 +332,7 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
due, waiting, failed := ready(*p, m.Module, entries) due, waiting, failed := ready(*p, m.Module, entries)
if !due && failed == "" { if !due && failed == "" {
s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", ")) s.logf("%s: %s waits for %s before its build", p.Pipeline, m.Module, strings.Join(waiting, ", "))
waits = append(waits, m)
continue continue
} }
steps, a := Place(*p, m, r, entries, now, s.Name) steps, a := Place(*p, m, r, entries, now, s.Name)
@@ -367,6 +369,11 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
} }
} }
} }
if steps, ended := cycled(p, waits, unregistered, reg, entries, now, s.Name); ended {
return nil, unregistered, steps, notAsked
} else {
notAsked = append(notAsked, steps...)
}
// A mover placed after the validate ended has its composition asked, or not asked, at once (§745). // A mover placed after the validate ended has its composition asked, or not asked, at once (§745).
more, _ := releaseCompositions(p, now) more, _ := releaseCompositions(p, now)
asks = append(asks, more...) asks = append(asks, more...)
@@ -374,6 +381,72 @@ func (s *Starter) placeMovers(p *seat.Pipeline, reg func(module string) (catalog
return asks, unregistered, ended, notAsked return asks, unregistered, ended, notAsked
} }
// cycled ends not asked, at p's placing, the steps of every mover still without steps when each waits only on movers
// without steps (H-A-175, H-A-176; G §270 and A-326 break a build module's own inputs before this): a cycle no build can
// start, and the movers that wait only on it. Their steps are placed ended not asked (as H-A-101 (2) does), the why
// `cycle` naming each artifact and edge kind whose two movers share a strongly connected group; no step already asked is
// touched. The pipeline then ends by the end judgement of H-A-163 (endPipeline), failed once no required step is open,
// its running steps running to their own end. It answers the steps it ended not asked, or, when the pipeline ended, every
// step it ended, and whether the pipeline ended.
func cycled(p *seat.Pipeline, waits []seat.Mover, unregistered []string, reg func(module string) (catalog.Registration, bool), entries map[string]catalog.CatalogEntry, now time.Time, name string) ([]int, bool) {
if len(waits) == 0 || len(unregistered) > 0 {
return nil, false
}
after := map[string][]string{}
edges := map[string][]catalog.Edge{}
for _, m := range waits {
for _, x := range awaited(*p, m.Module, entries) {
from := moduleOf(x.From)
if placed(*p, from) {
return nil, false
}
after[m.Module] = append(after[m.Module], from)
edges[m.Module] = append(edges[m.Module], x)
}
}
comp := components(waits, after)
var on []string
for _, m := range waits {
for _, x := range edges[m.Module] {
if comp[moduleOf(x.From)] == comp[m.Module] {
on = append(on, fmt.Sprintf("%s before %s (kind %d)", x.From, x.To, x.Kind))
}
}
}
sort.Strings(on)
why := "cycle: " + strings.Join(on, "; ")
var ended []int
for _, m := range waits {
r, _ := reg(m.Module)
steps, _ := Place(*p, m, r, entries, now, name)
for k := range steps {
st := &steps[k]
if st.Outcome == seat.StepResultOutcomeBypassed {
continue
}
st.Outcome, st.Why, st.Ask, st.Shared, st.Ended = seat.StepResultOutcomeNotAsked, why, "", nil, &now
p.Unlogged = append(p.Unlogged, endEntry(p.Pipeline, *st, now))
ended = append(ended, len(p.Steps)+k)
}
p.Steps = append(p.Steps, steps...)
}
var approvals []int
for i, st := range p.Steps {
if st.Step == seat.StepResultStepApproval && st.Outcome == seat.StepResultOutcomeWaiting {
approvals = append(approvals, i)
}
}
if !endPipeline(p, now, nil) {
return ended, false
}
for _, i := range approvals {
if p.Steps[i].Outcome == seat.StepResultOutcomeNotAsked {
ended = append(ended, i)
}
}
return ended, true
}
// sayNotAsked says the step-ended of each step of p placed not asked because its mover's dependency did not build, naming // sayNotAsked says the step-ended of each step of p placed not asked because its mover's dependency did not build, naming
// the dependency in its why, so no step ends silently (H-A-101 (2)). // the dependency in its why, so no step ends silently (H-A-101 (2)).
func (s *Starter) sayNotAsked(p seat.Pipeline, steps []int) { func (s *Starter) sayNotAsked(p seat.Pipeline, steps []int) {
@@ -541,6 +614,28 @@ func (s *Starter) Resume(store Updater, branch string, half Half) (int, error) {
return asked, first return asked, first
} }
// Unplaced says why mover module of p has no step placed (H-A-171): "<module> has no registration" when the registrations
// do not name it (A-327), else "<module> waits on the publish of <package>, …", the packages it awaits not yet published
// as the catalog half of p's branch holds its edges (§746).
func (s *Starter) Unplaced(p seat.Pipeline, module string, view func(branch string) (Half, bool)) string {
if s.Registration == nil {
return module + " has no registration"
}
if _, ok := s.Registration(module); !ok {
return module + " has no registration"
}
var waiting []string
if view != nil {
if half, ok := view(p.Branch); ok {
_, waiting, _ = ready(p, module, half.Entries)
}
}
if len(waiting) == 0 {
return module + " waits on the publish of the movers it builds on"
}
return module + " waits on the publish of " + strings.Join(waiting, ", ")
}
// Registered places, on a registration-changed of reg's module, the steps of that module's mover in every running pipeline // Registered places, on a registration-changed of reg's module, the steps of that module's mover in every running pipeline
// that moves it with none placed, from the registration (design 47 G §21, §23, A-327): a pipeline started before its module // that moves it with none placed, from the registration (design 47 G §21, §23, A-327): a pipeline started before its module
// had a registration. Each is written by compare-and-set, so it is placed once, its steps asked, and said; a run on request // had a registration. Each is written by compare-and-set, so it is placed once, its steps asked, and said; a run on request
@@ -610,7 +705,12 @@ func (s *Starter) placeWaiting(store Updater, due func(seat.Pipeline) bool, reg
continue continue
} }
before := len(p.Steps) before := len(p.Steps)
asks, _, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now) asks, unregistered, ended, notAsked := s.placeMovers(&p, reg, half.Entries, now)
// Until its registration arrives, the pipeline says it waits on it (A-327): a mover with no registration has no step,
// and it holds the pipeline's end only while every placed mover's required steps passed (H-A-163, H-A-171).
for _, m := range unregistered {
s.logf("%s: %s has no registration yet: its steps wait on its registration, which places them (A-327)", key, m)
}
if len(p.Steps) == before { if len(p.Steps) == before {
continue continue
} }
@@ -3,6 +3,8 @@ package pipeline
import ( import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"slices"
"strings" "strings"
"testing" "testing"
@@ -280,6 +282,23 @@ func TestAPipelineStartedBeforeItsRegistrationIsPlacedWhenItArrives(t *testing.T
if n != 1 || err != nil || len(p.Steps) == 0 || len(asks.asks) == before { if n != 1 || err != nil || len(p.Steps) == 0 || len(asks.asks) == before {
t.Errorf("the pipeline a runner before this one set aside: placed %d (%v), %d steps, %d asks more, want it started as a newly started one", n, err, len(p.Steps), len(asks.asks)-before) t.Errorf("the pipeline a runner before this one set aside: placed %d (%v), %d steps, %d asks more, want it started as a newly started one", n, err, len(p.Steps), len(asks.asks)-before)
} }
// A-327: a mover with no registration, in a pipeline placed from the half held later (the queue's turn, the half held),
// is said to wait on its registration, so the pipeline that cannot end before it is placed says why.
var logged []string
s.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
held = false
taken = catalog.TakeInTaken{Branch: "next", Source: 8, Entries: []catalog.TakeInEntry{{Module: "app", Commit: "c4", Fingerprint: "src2:dd"}, {Module: "contracts", Commit: "c4", Fingerprint: "src2:ee"}}}
if n, err := s.TakenIn(taken, late); n != 1 || err != nil {
t.Fatalf("the take-in of two movers started %d (%v)", n, err)
}
held, logged = true, nil
if n, err := s.Waiting(store.memStore, "next", late); n != 1 || err != nil {
t.Fatalf("the half of next placed %d (%v), want next.8's registered mover", n, err)
}
if !slices.ContainsFunc(logged, func(l string) bool { return strings.Contains(l, "next.8: contracts has no registration yet") }) {
t.Errorf("placing next.8 said %q, want that contracts waits on its registration (A-327)", logged)
}
} }
// R1/AC1 (design 47 H §747): a start names an event, never a sweep. A take-in starts the pipelines of its own entries // R1/AC1 (design 47 H §747): a start names an event, never a sweep. A take-in starts the pipelines of its own entries
@@ -158,6 +158,9 @@ type Hearer struct {
// Turn takes a pipeline's end into the queue, its running key deleted and the next placed (design 47 H §748; // Turn takes a pipeline's end into the queue, its running key deleted and the next placed (design 47 H §748;
// Starter.Turn); nil takes none. // Starter.Turn); nil takes none.
Turn func(key string) Turn func(key string)
// Unplaced says why a mover of a pipeline has no step placed, for the why of its end (H-A-171; Starter.Unplaced): it
// waits on its registration or on a publish; nil says only that it has no step placed.
Unplaced func(p seat.Pipeline, module string) string
// Rejoin keys a pipeline running again once its approval is answered and it still runs (H-A-106 (b); Starter.Rejoin); // Rejoin keys a pipeline running again once its approval is answered and it still runs (H-A-106 (b); Starter.Rejoin);
// nil keys none. // nil keys none.
Rejoin func(key string) Rejoin func(key string)
@@ -438,7 +441,7 @@ func (h *Hearer) said(p seat.Pipeline, asked []int, ended bool) error {
// endEntry is the step log's entry of a step's end, with the keys of the kept results it wrote or carried. // endEntry is the step log's entry of a step's end, with the keys of the kept results it wrote or carried.
func endEntry(key string, s seat.StepResult, at time.Time, kept ...string) seat.LogEntry { func endEntry(key string, s seat.StepResult, at time.Time, kept ...string) seat.LogEntry {
return seat.LogEntry{Pipeline: key, Step: s.Step, Artifact: s.Artifact, What: seat.LogEntryWhatEnded, Outcome: s.Outcome, return seat.LogEntry{Pipeline: key, Module: s.Module, Step: s.Step, Artifact: s.Artifact, What: seat.LogEntryWhatEnded, Outcome: s.Outcome,
Output: s.Output, Kept: kept, At: at} Output: s.Output, Kept: kept, At: at}
} }
@@ -568,16 +571,22 @@ func stepPassed(s seat.StepResult) bool {
// endPipeline ends a pipeline whose every mover is placed and every step ended: passed when every mover's required steps // endPipeline ends a pipeline whose every mover is placed and every step ended: passed when every mover's required steps
// passed, were carried or had nothing to run, else failed, naming each required step that held it (ADR 0666 §7, §9; // passed, were carried or had nothing to run, else failed, naming each required step that held it (ADR 0666 §7, §9;
// design 47 H §746). A failed optional step, and an optional step still waiting, hold nothing. An approval step still // design 47 H §746). A mover with no step placed, waiting on its registration (A-327) or on the publish of a mover it
// waiting when a required step did not pass is `not asked` (§26). It says whether the pipeline ended now, its end count one // builds on (§746), holds the end only while the pipeline could still pass: once a placed mover's required step did not
// higher. // pass and none is open, it ends failed, the why naming each unplaced mover with the reason unplaced gives (H-A-163,
func endPipeline(p *seat.Pipeline, now time.Time) bool { // H-A-171; nil says only that it has no step placed). A failed optional step, and an optional step still waiting, hold
// nothing. An approval step still waiting when a required step did not pass is `not asked` (§26). It says whether the
// pipeline ended now, its end count one higher.
func endPipeline(p *seat.Pipeline, now time.Time, unplaced func(seat.Pipeline, string) string) bool {
if p.Outcome != seat.PipelineOutcomeRunning { if p.Outcome != seat.PipelineOutcomeRunning {
return false return false
} }
// A mover with no step placed waits on its registration or a publish: it holds the end only while every placed mover's
// required steps passed, since once one failed or could not run the pipeline can no longer pass (H-A-163, H-A-171).
var waiting []string
for _, m := range p.Movers { for _, m := range p.Movers {
if !placed(*p, m.Module) { if !placed(*p, m.Module) {
return false waiting = append(waiting, m.Module)
} }
} }
var held []string var held []string
@@ -598,6 +607,9 @@ func endPipeline(p *seat.Pipeline, now time.Time) bool {
held = append(held, strings.TrimSpace(s.Module+" "+s.Artifact+" "+s.Step)+" "+s.Outcome) held = append(held, strings.TrimSpace(s.Module+" "+s.Artifact+" "+s.Step)+" "+s.Outcome)
} }
} }
if len(waiting) > 0 && len(held) == 0 {
return false
}
if len(notAsked) > 0 { if len(notAsked) > 0 {
if len(held) == 0 { if len(held) == 0 {
return false return false
@@ -611,6 +623,13 @@ func endPipeline(p *seat.Pipeline, now time.Time) bool {
p.Outcome, p.Why = seat.PipelineOutcomePassed, "" p.Outcome, p.Why = seat.PipelineOutcomePassed, ""
if len(held) > 0 { if len(held) > 0 {
p.Outcome, p.Why = seat.PipelineOutcomeFailed, "required steps not passed: "+strings.Join(held, ", ") p.Outcome, p.Why = seat.PipelineOutcomeFailed, "required steps not passed: "+strings.Join(held, ", ")
for _, m := range waiting {
why := m + " has no step placed"
if unplaced != nil {
why = unplaced(*p, m)
}
p.Why += "; " + why
}
} }
for i := range p.Movers { for i := range p.Movers {
m := &p.Movers[i] m := &p.Movers[i]
@@ -622,7 +641,7 @@ func endPipeline(p *seat.Pipeline, now time.Time) bool {
// endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51). // endPipeline is endPipeline with each mover's widens computed into its value at the end (§71, A-207; H-A-51).
func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time) bool { func (h *Hearer) endPipeline(p *seat.Pipeline, now time.Time) bool {
if !endPipeline(p, now) { if !endPipeline(p, now, h.Unplaced) {
return false return false
} }
if h.Widens != nil { if h.Widens != nil {
@@ -8,6 +8,7 @@ import (
"git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient" "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel" oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants" wants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo" nodebuildgo "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
@@ -291,23 +292,57 @@ func TestApprovalEnds(t *testing.T) {
} }
// An optional step still waiting (a code review nobody has given) does not hold a pipeline's end; a required one does // An optional step still waiting (a code review nobody has given) does not hold a pipeline's end; a required one does
// (design 47 G A-104). // (design 47 G A-104). A mover with no step placed, waiting on its registration (A-327) or on the publish of a mover it
// builds on (§746), holds the end while every placed mover's required steps passed or one is open, and once a placed one
// did not pass the pipeline ends failed, naming each with its own reason (H-A-163, H-A-171).
func TestAnOptionalStepStillWaitingHoldsNoEnd(t *testing.T) { func TestAnOptionalStepStillWaitingHoldsNoEnd(t *testing.T) {
now := time.Date(2026, 11, 11, 12, 0, 0, 0, time.UTC) now := time.Date(2026, 11, 11, 12, 0, 0, 0, time.UTC)
// lib is registered and awaits core's package, which core's passed build has not published yet; contracts has no
// registration.
starter := &Starter{Registration: func(m string) (catalog.Registration, bool) { return catalog.Registration{Module: m}, m != "contracts" }}
view := func(string) (Half, bool) {
return Half{Entries: map[string]catalog.CatalogEntry{"lib": {Edges: []catalog.Edge{{From: "core:go", To: "lib:program", Kind: 1}}}}}, true
}
unplaced := func(p seat.Pipeline, m string) string { return starter.Unplaced(p, m, view) }
for name, c := range map[string]struct { for name, c := range map[string]struct {
review string review, build, unit string
ends bool waiting []string
}{"optional review waiting": {seat.StepResultStatusOptional, true}, "required review waiting": {seat.StepResultStatusRequired, false}} { ends bool
p := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Steps: []seat.StepResult{ outcome string
{Step: seat.StepResultStepBuild, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomePassed}, why []string
{Step: seat.StepResultStepReview, Status: c.review, Outcome: seat.StepResultOutcomeWaiting}, }{
{Step: seat.StepResultStepApproval, Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomePassed}, "optional review waiting": {seat.StepResultStatusOptional, seat.StepResultOutcomePassed, seat.StepResultOutcomePassed, nil, true, seat.PipelineOutcomePassed, nil},
"required review waiting": {seat.StepResultStatusRequired, seat.StepResultOutcomePassed, seat.StepResultOutcomePassed, nil, false, "", nil},
"unregistered mover, the placed one passed": {seat.StepResultStatusOptional, seat.StepResultOutcomePassed, seat.StepResultOutcomePassed, []string{"contracts"}, false, "", nil},
"unregistered mover, the placed one failed": {seat.StepResultStatusOptional, seat.StepResultOutcomeFailed, seat.StepResultOutcomePassed, []string{"contracts"}, true, seat.PipelineOutcomeFailed, []string{"contracts has no registration"}},
"unregistered mover, failed and a step open": {seat.StepResultStatusOptional, seat.StepResultOutcomeFailed, seat.StepResultOutcomeRunning, []string{"contracts"}, false, "", nil},
"unregistered mover, the placed one could not run": {seat.StepResultStatusOptional, seat.StepResultOutcomePassed, seat.StepResultOutcomeCouldNotRun, []string{"contracts"}, true, seat.PipelineOutcomeFailed, []string{"contracts has no registration"}},
"a mover waiting on a publish, the placed one failed": {seat.StepResultStatusOptional, seat.StepResultOutcomeFailed, seat.StepResultOutcomePassed, []string{"contracts", "lib"}, true, seat.PipelineOutcomeFailed,
[]string{"contracts has no registration", "lib waits on the publish of core:go"}},
} {
p := seat.Pipeline{Outcome: seat.PipelineOutcomeRunning, Movers: []seat.Mover{{Module: "app"}, {Module: "core"}}, Steps: []seat.StepResult{
{Step: seat.StepResultStepBuild, Module: "app", Status: seat.StepResultStatusRequired, Outcome: c.build},
{Step: seat.StepResultStepUnit, Module: "app", Status: seat.StepResultStatusRequired, Outcome: c.unit},
{Step: seat.StepResultStepReview, Module: "app", Status: c.review, Outcome: seat.StepResultOutcomeWaiting},
{Step: seat.StepResultStepApproval, Module: "app", Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomePassed},
{Step: seat.StepResultStepBuild, Module: "core", Status: seat.StepResultStatusRequired, Outcome: seat.StepResultOutcomePassed},
}} }}
if got := endPipeline(&p, now); got != c.ends { for _, m := range c.waiting {
p.Movers = append(p.Movers, seat.Mover{Module: m})
}
if got := endPipeline(&p, now, unplaced); got != c.ends {
t.Errorf("%s: ended = %v, want %v", name, got, c.ends) t.Errorf("%s: ended = %v, want %v", name, got, c.ends)
} }
if c.ends && p.Outcome != seat.PipelineOutcomePassed { if c.ends && p.Outcome != c.outcome {
t.Errorf("%s: outcome %s", name, p.Outcome) t.Errorf("%s: outcome %s, want %s", name, p.Outcome, c.outcome)
}
for _, w := range c.why {
if !strings.Contains(p.Why, w) {
t.Errorf("%s: why %q, want it to name %q", name, p.Why, w)
}
}
if strings.Contains(p.Why, "lib has no registration") {
t.Errorf("%s: why %q names a registered mover as having no registration", name, p.Why)
} }
} }
} }
@@ -7,9 +7,10 @@ import (
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner" seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
) )
// sameEntry says whether two step-log entries are one: the same pipeline, step, artifact, end or start and time. // sameEntry says whether two step-log entries are one: the same pipeline, mover (H-A-190), step, artifact, end or start and
// time.
func sameEntry(a, b seat.LogEntry) bool { func sameEntry(a, b seat.LogEntry) bool {
return a.Pipeline == b.Pipeline && a.Step == b.Step && a.Artifact == b.Artifact && a.What == b.What && a.At.Equal(b.At) return a.Pipeline == b.Pipeline && a.Module == b.Module && a.Step == b.Step && a.Artifact == b.Artifact && a.What == b.What && a.At.Equal(b.At)
} }
// appendUnlogged appends the unlogged entries of pipeline key to the step log in their order and clears each appended one in a // appendUnlogged appends the unlogged entries of pipeline key to the step log in their order and clears each appended one in a

Some files were not shown because too many files have changed in this diff Show More