Compare commits

..
87 Commits
Author SHA1 Message Date
mesh-tools fe6c5961ae Merge pull request 'Design 47 H1: a repository's packages are those in the merge's tree (H-A-232)' (#306) from fix/47-H1-indexer-gone-packages into main 2026-11-21 02:09:12 +00:00
jschoubben 588a6ff8d5 Design 47 H1: only the trunk's merge drops a gone package; a commit's tree tells an all-gone repository (H-A-232, H-A-233)
Review round 1. A branch's take-in runs through the dry indexer, which
shares the live packages state, so a branch cut before main added a
module would have deleted main's key: publish drops a kept key only on
the real indexer at the trunk's merge. And a repository whose every
asked directory is gone (one key, or all gone) returned the whole error
and raised packages-not-published at every take-in: the seat's trees of
the commit now tell a gone directory from a commit with no tree.

Builds: design 47 H H-A-232, H-A-233, H-A-234; issue 1585
2026-11-21 03:07:02 +01:00
jschoubben ced27928d9 Design 47 H1: the indexer publishes a repository's packages in the merge's tree (H-A-232, H-A-233, H-A-234)
A Go module mesh-catalog removed stayed in the packages state, so every
publish of mesh-catalog asked the forge for its directory, the seat git
answered not-found, and the one-directory fallback (issue 1585) ran only
on an answer without the directory: nothing of the repository was
published and its packages' digests froze unsaid.

filesVia takes the seat's not-found as a failure of the tree, so the
fallback runs; publish drops the kept key of a package whose directory
the trunk's merge no longer holds, said in the journal; a listing that
fails for another reason raises the finding packages-not-published
(scope merge, id the repository), cleared at the next publish that lists.

Builds: design 47 H H-A-232, H-A-233, H-A-234; issue 1585
2026-11-21 02:54:58 +01:00
mesh-tools 2cb6971194 Merge pull request 'Design 47 H1: a receiver reads its store's fill once per transfer (H-A-227, H-A-228)' (#305) from fix/47-H1-receiver-fill-once into main 2026-11-21 01:12:32 +00:00
jschoubben 6a730c33e6 H-A-227, H-A-228: the receiver reads its objects' fill once per transfer
A fill read on every chunk made the tool runner list the registry's
incoming store per chunk. The kept transfer now holds the fill read at its
open and the bytes taken since; each chunk tells their sum, written with
the put already made per chunk, and the store is read again only at the
finish, a drop, or when a part's chunks are dropped.

Builds: design 47 H H-A-227, H-A-228
2026-11-21 02:11:39 +01:00
mesh-tools d1aab1d08d Merge pull request 'Design 47 H1: the validate and composition run asks carry their entry's pins (H-A-203)' (#304) from fix/47-H1-runner-validate-pins into main 2026-11-21 01:09:03 +00:00
jschoubben c6b4b6054b Design 47 H1: the validator's and the composer's run asks are built against their entry's pins (H-A-203)
The validate step's run ask built the manifest validator from the indexer's
tree with no pins, so the Go build module refused it for a requirement of
another repository (main.74's feh validate, main.67's mesh-sdk validate);
the composition step's run ask built the composer from the node-engine's
tree the same way. The seat mesh-wants' holder builds both asks: each now
carries the pins of the entry its program is built from, the indexer's
entry at the step's own revision and the node-engine entry chosen per
machine, those with a version, in the form the entry keeps and the
builder sends.

Builds: design 47 H H-A-203, H-A-192, H-A-205
2026-11-21 02:03:18 +01:00
mesh-tools 4df45a3793 Merge pull request 'Design 47 H1: the builder keeps a reservation for its fingerprint and withdraws a taken-over build (H-A-225, H-A-226)' (#303) from fix/47-H1-builder-reservation-kept into main 2026-11-21 00:41:48 +00:00
jschoubben e2e13ddd29 Drop withdrawn versions the catalog released past; a dropped number stays a gap (H-A-226)
The series key's Withdrawn grew at every take-over and was never pruned. At each write it now keeps only the
withdrawn versions above the catalog's highest released (those at or below it are refused by the plain rule anyway),
carried across a renumbering of the same fingerprint too; a dropped number is never reused (H-A-226 withdraws
H-A-225's freeing). The comment of the rebuild's reservation now states A-490b as H-A-225 amends it.

Builds: design 47 H H-A-225, H-A-226
2026-11-21 01:40:18 +01:00
jschoubben dcf5b7fa8f Keep a reservation for its fingerprint until released; withdraw a taken-over build (H-A-225)
After a builder restart, the run already building a module carried its reserved number while a re-run of the same
fingerprint reserved the next, so its publish was refused "has no version reserved or released" (main.61: the builder
0.1.111 vs 0.1.112, mesh-grants 0.1.52 vs 0.1.54, and others reserved twice within seconds). A take-over by another
fingerprint dropped the older reservation but let the older build run on and publish a number that no longer existed.

Reserve now answers the standing reservation of the same fingerprint while it is unreleased and above the catalog's
highest, a build again or a re-run alike (A-490b amended so). A take-over keeps the dropped version in Withdrawn and
withdraws the older build (ended superseded, ADR 0664 §4); a late publish of it is refused as withdrawn, naming the
fingerprint that took it over.

Builds: design 47 H H-A-225; G A-490b as amended; ADR 0664 §4
2026-11-21 01:36:32 +01:00
mesh-tools 1fe6ade359 Merge pull request 'Design 47 H1: a same-repository module named only by a replace is a requirement of the module graph (H-A-224)' (#302) from fix/47-H1-indexer-replace-required into main 2026-11-21 00:30:50 +00:00
jschoubben 9e09767a4b Indexer: a same-repository module named only by a replace is a requirement
The watchman's go.mod names buildrun in a replace directive with no require
line, and the walk of the module graph followed require lines only, so
buildrun's requirement of the module assigner's client was never pinned and
every test step of the watchman failed "no pin". The walk now takes the
same-repository modules a go.mod replaces by a directory as required, with
the same same-repository test, so their go.mod files are read and their
other-repository requirements pinned, transitively.

Builds: design 47 H-A-224, H-A-201
2026-11-21 01:26:29 +01:00
mesh-tools 59443ab3f7 Merge pull request 'Design 47 H1: a mover a newer revision moved leaves a running pipeline too (H-A-223)' (#301) from fix/47-H1-runner-moved-leaves into main 2026-11-21 00:24:57 +00:00
mesh-tools 92d0ab8602 Merge pull request 'Design 47 H1: the indexer's test fakes answer what the SDK's pull guard reads (H-A-196)' (#300) from fix/47-H1-indexer-fake-consumer into main 2026-11-21 00:24:55 +00:00
jschoubben 3a31331ad3 Design 47 H1: a mover leaving a running pipeline has its build-node asks dropped (H-A-223)
A moved mover's steps ended superseded, but their asks kept running on the
build nodes. Each such step holding its own ask on a node-build seat, one
reading no shared run, now has the seat's drop asked with the newer
pipeline's why, through the drop the runner already bounds and keeps. The
seat mesh-builds serves no drop, so a build ask there runs to its end,
which the runner ignores: the step is no longer open.

Builds: design 47 H H-A-223
2026-11-21 01:22:10 +01:00
jschoubben d1581666d2 Design 47 H1: the start of a newer pipeline is held to take moved movers out, and reads its branch alone (H-A-223)
No test failed when Start stopped calling the supersede of running
pipelines, the path the live runner takes. Two rows now start a newer
pipeline through Start, a take-in's and a release revision's, with the
supersede wired to a hearer over the same store, and hold that the older
running pipeline's moved mover left it. The supersede lists only the keys
of the newer pipeline's branch instead of every pipeline's.

Builds: design 47 H H-A-223
2026-11-21 01:18:55 +01:00
jschoubben d4d7895392 Indexer: the branch consumer fake answers its CachedInfo and ends when stopped
With the SDK's guard of one pull per durable (pullOnce reads the consumer's
CachedInfo and releases on the consume's Closed), the fake of a branch's
durable embedded a nil consumer and a nil consume context, so
TestABranchDurableNotThereIsMadeAndHeard panicked. The fake now answers a
ConsumerInfo with its stream, name and a created time of its own, and its
consume context closes when stopped. No other fake in mesh-delivery embeds
jetstream.Consumer; the assigner's, runner's, builder's, watchman's and
grants' unit suites pass with the new SDK unchanged.

Builds: design 47 H-A-196, H §774
2026-11-21 01:18:22 +01:00
mesh-tools d6c209a10c Merge pull request 'Design 47 H1: the catalog seat's revision verb answers the store; the builder's picture holds a kept revision by number (H-A-222)' (#299) from fix/47-H1-revision-verb-store into main 2026-11-21 00:14:44 +00:00
mesh-tools 51ac838732 Merge pull request 'Design 47 H1: the indexer decodes every hand tool's ask (H-A-208, H-A-210)' (#298) from fix/47-H1-indexer-again-decode into main 2026-11-21 00:14:40 +00:00
jschoubben 06a52a10e0 Design 47 H1: a moved mover leaves a running pipeline too (H-A-223)
A release re-pinned node-tools while main.54 still built it at its old
fingerprint; the build failed and failed the pipeline for its other 119
movers, because supersession took movers out of waiting pipelines only.
Every start of a newer revision's pipeline now also walks the older running
pipelines of its branch: each mover whose fingerprint it moved leaves, its
open steps ending superseded with a why naming the newer pipeline, a step
reading a test run leaving it as a cancel's does, and the mover kept in the
pipeline's superseded. The end judgement counts a left mover's steps as
neither holding nor failed, so the pipeline ends by its other movers alone;
one every mover left ends superseded. A run on request's pipeline is never
touched.

Builds: design 47 H H-A-223, H-A-163, H-A-213
2026-11-21 01:13:57 +01:00
jschoubben 773e2e33a1 Builder: a kept revision of another source or fingerprint fails the ask at once
Review round 1: an ask names the revision and the source of one revision, so a
kept revision read by number whose source differs from the ask's is not the
ask's; the check is restored, naming both sources. And a kept revision naming
another source or other fingerprints is a mismatch no wait can mend: the step
fails at once, loud, instead of waiting out revision-wait-minutes; only a
revision the catalog does not answer is waited for.

Builds: design 47 H-A-222 (2), H-A-218
2026-11-21 01:10:37 +01:00
jschoubben d817d85fca Builder: an ask naming a revision is held by that revision read by number
An ask of the very source the picture holds, whose entries the picture no
longer names (a re-pin keeps a later revision of the same source), waited for
the picture and failed past its bound, because the kept revision was read by
number only for a source below the picture's. Now an ask naming a revision is
read by number whenever the picture does not name its entries, and is built
when the catalog keeps it and it names every fingerprint asked, whatever the
source says; the source is compared only for an ask naming no revision.

Builds: design 47 H-A-222 (2), H-A-205, H-A-213
2026-11-21 01:07:18 +01:00
jschoubben 6d5000871d Indexer: the revision verb answers the newest kept revision from the store
The tools bundle answered the catalog seat's revision verb from its view of the
trunk in memory, which ran one or more revisions behind what the indexer kept
until it heard the event, so the runner and the builder waited on revisions the
bus held all along. The verb now reads which revision is newest from the store
at the time of the ask (a listing and one object) and takes the view's entries
only when the view holds that very revision.

Builds: design 47 H-A-222 (1), H-A-205, H-A-213
2026-11-21 01:05:55 +01:00
jschoubben 5c7648744d Indexer: read a takeins value's hand acts from TakeInAsk's tags, not a hand-kept list
indexer_take_in_again wrote TakeInAsk{Again} and DecodeTakeIn refused it: its
hand-kept list of act fields lacked Again, so the handler never reached its
Again branch. Every field of TakeInAsk now says by its tag takein whether it is
an act, an argument or of the ask; DecodeTakeIn reads the acts from the tags,
and a table test builds every hand tool's value by its own Run path, decodes it
and acts on it through takenIn, and refuses a field left unclassed.

Builds: design 47 H §774, H-A-208, H-A-210
2026-11-21 01:03:32 +01:00
mesh-tools 8802f16019 Merge pull request 'Design 47 H1: the pin rule whole: revisions numbered by the indexer with their call, a re-pin or recompute keeps a revision, pins carried in every ask, pipelines keyed by revision (H-A-203 to H-A-221)' (#297) from fix/47-H1-pin-kind-join into main 2026-11-20 23:49:57 +00:00
jschoubben 6087306dfd Design 47 H1: the pipelines store guards its shared maps, so the runner's start no longer crashes (H-A-187)
The runner crashed at every start on a live bus with "concurrent map
writes" in MoverStore.Get: the waiting pipelines of each branch are placed
from goroutines of their own, and the store's last reads and inline marks
were written under each pipeline's own lock alone. The store's mutex now
guards the lock table, the last reads and the inline marks; behaviour is
unchanged. The run test starts the runner over six kept pipelines in the
old form, one of 437 steps, so the start path runs under the race detector.

Builds: design 47 H H-A-187, H §774
2026-11-21 00:43:16 +01:00
jschoubben 667aa87611 Join fix/47-H1-builder-revision-picture 2026-11-21 00:27:10 +01:00
jschoubben 0f2fd1a171 Join fix/47-H1-runner-published-revision
# Conflicts:
#	modules/mesh-delivery-indexer/internal/index/catalog_revisions.go
#	modules/mesh-delivery-pipeline/internal/pipeline/start.go
2026-11-21 00:27:04 +01:00
jschoubben e1ce5cc8ed Join fix/47-H1-indexer-take-in-again 2026-11-21 00:26:46 +01:00
jschoubben 7794ee9d2d Join fix/47-H1-sdk-empty-pin 2026-11-21 00:26:46 +01:00
jschoubben 396893dcac Design 47 H1: the pipeline-runner seat names a pipeline's key and its subjects by revision
The seat's key form of a pipeline is <branch>.<revision>, and its four per-pipeline
subjects are said on <branch>.<revision> (the definition and generated contracts the
same as the runner branch's). The indexer pins a revision for the pipeline keyed
PipelinesByBranchRevisionKey, and this branch's runner copy follows the renamed
generated functions; the wire form is unchanged.

Builds: design 47 H H-A-213 (4), H-A-219, H-A-221
2026-11-21 00:25:58 +01:00
jschoubben 626a6fb358 Design 47 H1: the pipeline-runner seat names a pipeline's key and its subjects by revision
A pipeline is keyed by its catalog revision's number, so the seat's key form is
<branch>.<revision> (the generated PipelinesByBranchRevisionKey and its siblings; the
fixtures follow), and pipeline-started, pipeline-ended, wait-held and wait-ended are
said on <branch>.<revision>, carrying the same number as the key; every reader takes
the body. The client's SourceSubject is RevisionSubject. The published key form, under
which pipelines were kept before, stays.

Builds: design 47 H H-A-213 (4), H-A-219, H-A-221
2026-11-21 00:25:08 +01:00
jschoubben 20ee1cae7b Merge the runner branch into the builder branch, so both merge into main as one
The builder's step end reports the pins it built with (H-A-207, from the runner
branch) and names the run's catalog revision on its origin (H-A-218, H-A-220); step.go
keeps both. The seat mesh-builds carries packages on step-ended and revision on
package-published once each; seatgen -check is clean.

Builds: design 47 H H-A-207, H-A-218, H-A-220
2026-11-21 00:22:19 +01:00
jschoubben 671dc5a963 Design 47 H1 (5c): the runner takes a publish into the pipeline of the revision it names
One source now has several catalog revisions, so the publishing pipeline is found by
the branch and revision the package-published names. A publish naming no revision,
from a reservation made before the field, goes to its source's lowest-numbered
revision, the take-in's. The seat mesh-builds' package-published gains its optional
revision here too, identical to the builder branch's, so this branch compiles alone.

Builds: design 47 H H-A-218, H-A-220
2026-11-21 00:17:55 +01:00
jschoubben 3f62410c22 Design 47 H1 (5c): a package-published names the catalog revision its build was made from
One source now has several catalog revisions, so a publish naming branch and source
alone does not say which pipeline published it. The seat mesh-builds'
package-published gains an optional revision (seatgen regenerated). The version
reserved for a build keeps the catalog revision number: from the picture's half when
the picture starts the build (the same revision its source comes from), from the step
ask's run when the ask reaches the build first. Its package-published says it. A
reservation from before the field says none.

Builds: design 47 H H-A-218, H-A-220
2026-11-21 00:17:17 +01:00
jschoubben 4af1ba99cf Design 47 H1 (5a): the runner's integration tests key a pipeline by its catalog revision
The queue's and the test runs' integration tests took several sources in at one
catalog revision, which now are one key. Each take-in has its own revision; the
test runs' test also checks that a test ask names revision and source. A replaced
reader of another pipeline's run stays the unit tests' (one revision is one pipeline).

Builds: design 47 H H-A-213 (4), H-A-219
2026-11-21 00:12:36 +01:00
jschoubben 91b5e8d086 Design 47 H1 (5b): a release, take-in again or recompute revision starts one pipeline, read by its number
A re-pin on a module's release, a take-in again and a recompute that moves a
fingerprint each keep a catalog revision of their own. The runner hears a trunk
catalog-revision of those kinds, reads that revision by its number through the catalog
seat's revision verb, and starts one pipeline keyed <branch>.<revision> with the
revision's movers. A package-published no longer starts per-artifact pipelines: it
only places the movers waiting for it. A branch's newest pipeline for a mover is the
one of the highest revision number.

Builds: design 47 H H-A-212, H-A-213 (2) and (4), H-A-217
2026-11-21 00:07:55 +01:00
jschoubben 189d75ad40 Design 47 H1 (5a): a pipeline is keyed by its catalog revision, and every step ask names revision and source
One source now has several catalog revisions, so a pipeline keyed by its source could
not tell them apart. A pipeline is keyed <branch>.<revision>; one of a source (a
publish's, a version-released's) is found by the source it names, never by its key, so
pipelines kept under their source are read as they are. The Go, Node and image step
asks name the catalog revision in revision and the gate's source in a new optional
source (the build-step part of the seats node-build-go, node-build-node and
node-build-image, seatgen regenerated).

Builds: design 47 H H-A-211, H-A-213 (4), H-A-219
2026-11-21 00:02:48 +01:00
jschoubben 595510ef38 Design 47 H1 (9): a recompute that moves a fingerprint keeps a revision of kind recompute
Every whole recompute of the modules state (a toolchain reported, the toolchains read, the assignments of a build module, the indexer's start, a hand act) moved fingerprints with no kept catalog revision. It now runs as a re-pin does: when at least one fingerprint moved against the trunk's newest revision, it keeps the trunk's next revision of the same source, its movers the moved modules, its call-kind recompute (the seat mesh-catalog's enumeration gains it, seatgen regenerated) and its call naming the reason; one that moves none keeps nothing.

Builds: design 47 H H-A-211, H-A-213 (2), H-A-217
2026-11-20 23:50:50 +01:00
jschoubben 04f54d7773 Design 47 H1 (7): a registry's published recomputes no module
The indexer kept a registry's published in its series and then recomputed every module, moving fingerprints with no kept catalog revision beside the re-pin a release's package-published starts. It keeps the version in its series and recomputes nothing: a fingerprint moves only with a kept revision, one per module release, kept by the re-pin. The seat mesh-wants' pipeline-wanted subject names its last token revision, the number of the runner's key (its form unchanged).

Builds: design 47 H H-A-211, H-A-213 (2), H-A-213 (4) (the subject's token), H-pins-spec-and-gaps leftover L3
2026-11-20 23:48:08 +01:00
jschoubben 07dc14bf53 Design 47 H1 (6): pipeline-wanted names the pipeline's revision, and the indexer frees that revision's pins
The seat mesh-wants' pipeline-wanted gains revision, the pipeline's catalog revision number (seatgen regenerated, additive). The assigner keeps the source the pipeline-ended names in its tier progress and says it beside the revision, the number of the pipeline's key; a progress kept before the field says the key's number as both. The indexer frees the pins of the revision the event names, not of another revision of the same source, and for an event naming none takes the source's lowest-numbered revision, the take-in's.

Builds: design 47 H H-A-213 (4), H-A-214, H-A-216
2026-11-20 23:46:25 +01:00
jschoubben 9b3a728f9e Design 47 H1 (8): a catalog revision's call names its kind, one of three constants
The seat mesh-catalog's CatalogRevision and catalog-revision gain call-kind, an enumeration the seat's schema validates (take-in, release, take-in-again), with module and version for a release and repository for a take-in again, beside call, the words for a person (regenerated by seatgen, additive). The indexer's three writers set them: a take-in's from the gate's call, a re-pin's on a module's release, a take-in again's; no reader parses the words.

Builds: design 47 H H-A-213 (3), H-A-214, H-A-215
2026-11-20 23:43:33 +01:00
jschoubben 83e4470757 Design 47 H1 (1), review round 1: a source answers its take-in's revision
The revision verb's source input, and the pipeline-wanted that frees pins by it, answer the lowest-numbered revision of a source (the take-in's), never a later re-pin's of the same source. TestIndexerWhole asks the exact first number: the bed seeds main.1 of source 0, so source 10's is 2 (no source is kept twice).

Builds: design 47 H-A-213 (1), (4), H-A-211
2026-11-20 23:39:53 +01:00
jschoubben 1b1fd9a711 Design 47 H1 (2): a take-in again keeps the trunk's next catalog revision of the same source
The take-in again keeps what it recomputed as a re-pin does: the trunk's next revision of the newest's source, its call 'take-in again of <repository>', its movers every module whose fingerprint moved, pinned for its pipeline, said as catalog-revision.

Builds: design 47 H-A-208, H-A-210 as H-A-211 and H-A-212 amend it, H-A-213 (1), (3)
2026-11-20 23:36:39 +01:00
jschoubben 47ea130332 Design 47 H1 (1): a release re-pin keeps the next catalog revision of the same source on every watched branch
A branch's revision number is the indexer's own count (the newest + 1); the gate's source is the revision's field source, which the take-in's dedupe and source-to-take read. A catalog revision carries its call (the seat mesh-catalog's CatalogRevision and catalog-revision gain call, regenerated by seatgen). A re-pin on a module's release runs on every watched branch holding a module that names its package (the first on the modules state, another on its newest revision's picture), and keeps, where a fingerprint moved, the branch's next revision of the same source with the call 'release of <module> <version>' and the moved modules as its movers, pinned for the release's pipeline, and says catalog-revision; no take-in.taken.

Builds: design 47 H S11, S12, S13, S14 (H-pins-spec-and-gaps item 1), H-A-211, H-A-212, H-A-213 (1)-(4: the pin of every revision a pipeline runs on), H-A-147
2026-11-20 23:36:39 +01:00
jschoubben 52f7a5b6f6 Design 47 H1 (4): a candidate's entry may name a package with no version yet
An entry names an unpublished package with an empty version, never left out, so a run on request's candidate checks its packages as an entry's (CheckEntry), not as an ask's. Built against the SDK head aee7e4c2 (fix/47-H1-sdk-empty-pin), which goes in the same merge call.

Builds: design 47 H S8 (H-pins-spec-and-gaps item 4, L10), H-A-204
2026-11-20 23:25:24 +01:00
jschoubben aee7e4c26c mesh-delivery-sdk: an entry's packages may name an empty version (H item 4, the SDK's half)
The indexer names a package of another repository with no version published yet by an empty
version, which waits for its publish; PackageVersions.Check refused it, so a run on request of such a
module failed its candidate check. CheckEntry is the entry form, allowing an empty version, while
Check (an ask's pins) still refuses one.

Builds: design 47 H S8, S9, L10 (H-pins-spec-and-gaps section 8 item 4); H-A-204, H-A-165
2026-11-20 23:24:16 +01:00
jschoubben 64d5490ea2 Design 47 H1 (3): a publish whose recompute fails is not acknowledged
The registry's published is offered again when the recompute of its users fails, as a failed series write already was.

Builds: design 47 H S15 (H-pins-spec-and-gaps item 3), H-A-167
2026-11-20 23:21:23 +01:00
jschoubben 7199824e9d Design 47 H1: indexer_take_in_again re-reads a repository at its kept merge
A repository with no merge since the walk of the module graph keeps its
modules' direct requirements only, and no tool took it in again. The tool
asks the indexer, through the state takeins, to read the repository's trees
at the merge its taken key keeps and apply them as a take-in does, whose
recompute of the modules state is the one a publish re-pin runs; kept
requirements and pins gain the transitive packages. An unknown repository,
or one never taken in, is refused in the answer and as the tool's error.
It keeps no catalog revision yet: H-A-211's revision follows with the
re-pin's.

Builds: design 47 H H-A-208, H-A-210, H-A-201 (hq a7b876c1, d7d18ef0)
2026-11-20 23:16:25 +01:00
jschoubben c418f49038 Design 47 H1: held test steps are asked from the pipeline's own catalog revision (H-A-205, H-A-207)
The runner built the asks of the steps that waited for their pins from the
branch's newest catalog half, so a newer revision that moved the module
changed what a passed build's tests fetched. The entry is now read at the
pipeline's own revision through the runner's revision reader, which keeps
what it read; a revision that cannot be read, or whose entry is of another
fingerprint than the mover's, ends the held steps could not run, naming
why.

Builds: design 47 H H-A-205, H-A-207, H-A-203
2026-11-20 23:13:47 +01:00
jschoubben e29aa8aca9 Design 47 H1: a done build key keeps the pins it was built with, and its step's end reports those (H-A-207)
A build step whose key was done before reported the pins it resolved at
that moment, so a publish since would have the runner ask the waiting
test steps against a version the build was never built with. The done
key's record keeps the versions its Go ask was built with, beside its
digests, and the step's end reports the record's; a done key keeping none
reports none, so the runner ends those steps could not run, loud.

Builds: design 47 H H-A-207, H-A-203
2026-11-20 23:12:29 +01:00
jschoubben 64cba95a10 Design 47 H1: a held test step is asked with the pins its build passed with (H-A-207)
When the build of an entry with an empty pin passed, its held lint, unit
and integration steps had no pin to run with and nothing ended them, so a
pipeline stood running and, one pipeline at a time, the queue stopped. The
runner reads the pins the build's end names and, when the build passed,
asks each held step with them, its fetch the entry's as placed; a passed
build whose end names no pins ends the held steps could not run, naming
why, as does an artifact the branch's half no longer names.

Builds: design 47 H H-A-207, H-A-206, H-A-203
2026-11-20 23:07:33 +01:00
jschoubben cb26645614 Design 47 H1: the build step's end reports the pins it was built with (H-A-207)
A test step held for an entry's empty pin had nothing to run with once the
build passed: the builder took the pin from the registry and kept it to
itself. The seat mesh-builds' step end gains packages, and the builder sets
it from the pins each entry was built with, the registry's included, so the
runner asks the held steps with what the build was built against. The
seat's definition changes: its Go and TypeScript contracts are generated
again.

Builds: design 47 H H-A-207
2026-11-20 23:05:08 +01:00
jschoubben b0d2d22002 Design 47 H1: indexer_entry reads one entry's pins and paths at a revision
A reader twice could not read one entry's pins because the whole revision
(1.3 MB) exceeds what the bus carries; the tool answers one module's
packages, paths per artifact, fingerprint and manifest digest, at the
newest trunk revision (read as entryOf reads it) or at a given one from
its kept list and that one entry object.

Builds: design 47 H H-A-204 (hq f977c0e2); tools-are-how-a-module-is-spoken-to
2026-11-20 23:04:07 +01:00
jschoubben 014f7b7c2f Reserve the version of an ask admitted by its kept revision (H-A-205)
An ask of an older kept revision was admitted, but after a builder restart its fingerprint had never been heard on
the branch, so its build neither reserved a version nor was checked against one and passed silently. Each asked
fingerprint is now recorded as the branch's (its source revision, and first as the module's registration says),
without moving what the picture heard, so the newer fingerprint's build is not superseded.

Builds: design 47 H H-A-205, H-A-155
2026-11-20 23:03:45 +01:00
jschoubben 0fcc614e25 Build an ask of an older kept revision by reading it by number, never waiting (H-A-205)
Two pipelines of consecutive revisions ran at once (main.45, 22:5x CET): the picture held source revision 46, whose
entries had moved on, so the older pipeline's builds waited for a picture that only moves forward and failed "revision
45 of main not in the builder's picture within 2m0s". An ask of a source revision below the newest the picture holds
now reads its catalog revision by number through the catalog seat's verb revision (kept while its pipeline runs,
H-A-147) and is built when that revision is of its source and names every entry it asks; only a revision above the
newest is waited for (H-A-155).

Builds: design 47 H H-A-205, H-A-147, H-A-155
2026-11-20 22:59:31 +01:00
jschoubben e3fed77515 Design 47 H1: a Go step waits for its entry's pins as the build does (H-A-206)
An entry may name a package with no version yet, and the runner cannot pin
it from a publish it does not hear, so a step asked then is refused by the
Go build module. The placing now asks a Go artifact's lint, unit and
integration steps only when every pin of the entry has a version; until
then each stands waiting, its why "waits on the publish of <path>" as the
build's wait reads, said in the journal and read by show. When the mover's
build ends without passing, the held steps end as it ended; the publish
re-pins the entry and moves the mover, whose pipeline runs them. A held
step still counts in the judgement of a seat with no holder.

Builds: design 47 H H-A-206, H-A-203, H-A-165
2026-11-20 22:58:30 +01:00
jschoubben 94202fb651 Design 47 H1: a release re-pins every entry naming its package
A release re-pinned only the entries whose pins already named one of its
packages, so an entry whose module did not change and whose pins left the
package out was never re-pinned. Every kept entry whose pins or kept
requirements name a package of the released module is now re-pinned, of
any repository and changed or not; its fingerprint moves with the pin.

Builds: design 47 H H-A-204, H-A-165, H-A-167, H-A-201
2026-11-20 22:58:06 +01:00
jschoubben 0418a2b94b Design 47 H1: an entry's packages never leave out a requirement
Build asks were refused for a requirement with no pin: the import form's
versions name only what the last build imported, and the version line
leaves out a package none publishes, so the pins kept only those. The pins
now name every other-repository package of the module's go.mod graph (its
kept packages), at the newest published version of its accepted series, or
with no version while none is published, so the builder waits for it.

Builds: design 47 H H-A-204, H-A-201, H-A-165, H-A-167
2026-11-20 22:57:26 +01:00
jschoubben a926a40eed Design 47 H1: every step ask fetches with the entry's pins, as the build does (H-A-203)
The runner's lint, unit and integration asks carried no pins, only the
build ask did, so the Go build module, which refuses a requirement of
another repository with no pin, could not run twelve test steps of the
contracts mover (main.45). stepAsk now puts the entry's packages into the
fetch in the form the entry keeps and the builder sends, so a test runs
against what the build was built against; a package the entry names with
no pin yet is left out.

Builds: design 47 H H-A-203, H-A-192, H-A-183
2026-11-20 22:55:36 +01:00
mesh-tools b9cc97470b Merge pull request 'Design 47 H1: the entry pins every requirement of the module graph; same-repository directories move the fingerprint (H-A-201, H-A-202)' (#296) from fix/47-H1-package-tree-paths into main 2026-11-20 21:37:24 +00:00
mesh-tools ae91f91e88 Merge pull request 'Design 47 H1: the catalog seat's hand-written digest helpers live in its client (H-A-200)' (#295) from fix/47-H1-contracts-steps into main 2026-11-20 21:37:22 +00:00
jschoubben 4664eb1b00 Design 47 H1: the entry pins every requirement of the module graph
The Go build module borrowed a sibling's version for a mesh package the
entry did not pin (a transitive requirement), and given no pin it now
refuses. The indexer now pins, in the entry's packages, every mesh package
of another repository that the go.mod of the module's directory, of each
Go package artifact's directory and of every same-repository module they
require transitively names. A module of the module's own repository is
never pinned, whatever a build reported: it is built from the tree.

Builds: design 47 H H-A-201, H-A-183, H-A-157, H-A-202
2026-11-20 22:30:45 +01:00
jschoubben d9a3458927 Design 47 H1: a Go build's same-repository modules move its fingerprint
A Go build resolves the same-repository modules its go.mod files require at
the zero version, transitively, from the fetched tree; a change in one of
them must move the fingerprint, whatever the module's form. The indexer
walks the go.mod graph of the module's directory and its Go package
artifacts at the take-in and appends a line per derived directory and its
git object to the tree input; the entry's paths keep the manifest's form,
and the Go build module derives the same directories itself.

Builds: design 47 H H-A-202, H-A-172, H-A-157, H-A-198
2026-11-20 22:29:15 +01:00
jschoubben b058389d46 Contracts: the catalog seat's manifest-digest helpers move to its client, out of the generated root (H-A-200)
The seat generator's check failed every unit step of the contracts mover (pipeline main.43): manifestdigest.go was
written by hand under go/seats/meshcatalog (598125a1), a generated root every file of which is the generator's. The two
helpers move to the seat's hand-written client meshcatalogclient and their callers in the builder and the indexer
follow; the generator then writes the contract as the module holds it. steprun's generator test now also holds this
repository's own contracts module in step under both roots.

Builds: design 47 G §370, G §578, H §774, H-A-200 (ADR 0608 §5)
2026-11-20 22:18:00 +01:00
jschoubben 6c65e82926 Design 47 H1: a step's named path may leave the module's directory, never the repository
A package artifact's tree names same-repository siblings its go.mod requires
as "../<module>/<dir>"; the fetch request's check and the narrowing refused
them as leaving the module. They are now accepted when their join with the
module's directory stays inside the repository, narrowed from the root, and
refused when they leave the repository.

Builds: design 47 H H-A-198, H-A-199, H-A-157, H-A-172
2026-11-20 22:14:53 +01:00
mesh-tools d40f314e52 Merge pull request 'Design 47 H1: the grants holder keeps the catalog revisions it read and learns the newest from the event (H-A-193, H-A-195)' (#294) from fix/47-H1-grants-for-cache into main 2026-11-20 20:56:03 +00:00
mesh-tools fc85f7b936 Merge pull request 'Design 47 H1: the assigner's test helper holds its map writes under a lock (H §774)' (#293) from fix/47-H1-assigner-test-race into main 2026-11-20 20:52:56 +00:00
mesh-tools 37a8f7a83e Merge pull request 'Design 47 H1: a pipeline's steps and unlogged entries live under one key per mover (H-A-187, H-A-189, H-A-190, H-A-191)' (#292) from fix/47-H1-runner-steps-per-mover into main 2026-11-20 20:52:51 +00:00
mesh-tools 9ef830d224 Merge pull request 'Design 47 H1: the builder fetches a module whole and never computes a pin (H-A-186, H-A-192, H-A-194)' (#291) from 47-H1-builder-program-paths into main 2026-11-20 20:52:45 +00:00
jschoubben 120a0554d6 mesh-grants: grants-for keeps a bounded cache of the followed branches
The kept catalog revisions had no design line and their bound stood in
the code. They are now bounded per branch by the module's setting
grants-for-revisions-per-branch (default 4, oldest dropped, a run's
candidate counted within its branch's), read at the start from the file
the manifest writes, and kept only for the watched branches the part's
branch follower follows: a branch that leaves them is freed at the next
grants-for or catalog-revision event. A kept revision is a cache of the
catalog seat's own truth, never a state of the mesh.

Builds: design 47 G §441, §401; H H-A-195, H-A-193
2026-11-20 21:51:43 +01:00
jschoubben d8a900dac0 Read what the registry seat holds before a build waits for an unpinned package (H-A-194)
The publishes the builder heard lived only in its memory, so a build waiting for a package published before a
restart waited the whole bound and failed although the registry held the version. Before it waits, the build asks
the Go package registry seat's versions once for each unpinned path and takes the newest in the accepted series
through PackagePublished; a failed read is said and the wait goes on by the published events.

Builds: design 47 H H-A-194, H-A-192, H-A-165
2026-11-20 21:48:40 +01:00
jschoubben a62f7b40c1 mesh-grants: grants-for learns the newest revision from the event
A grants-for call still read the first page of each revision (up to
768 KB) only to learn its number. The grants holder already hears the
seat mesh-catalog's catalog-revision event, which names each branch's
newest trunk revision and the source revision it was taken in from; the
part now records both from the event and asks the revision verb only for
a revision it does not hold, by its number. Before the first event after
a start the first ask's answer names the newest once. An event below the
newest held is the numbering started again: what is kept of the branch
is dropped.

Builds: design 47 G §441, §401; H H-A-193
2026-11-20 21:44:06 +01:00
jschoubben 429f03d4cf Pin an unpinned package only at the version its registry publish carries (H-A-192)
A step ask waiting for a package the entry names with no pin took the release number of the module holding it
("v" + release), and the lane of an entry taken in dropped the empty pin and asked its build without one, so the Go
build module took another package's version: both built against versions never published (main.41, v0.1.25,
v0.1.97, v0.1.103 of the builder's builds). The builder now hears each package registry seat's published (the
package path and the version it carries) and keeps the newest per series; a build, the step's and the lane's alike,
waits for one in the series its module's manifest accepts (packages, else the default) up to package-wait-minutes,
and past it fails naming the package. The release-number pin (VersionReleased, NewestReleased) is gone.

Builds: design 47 H H-A-192, H-A-183, H-A-184, H-A-185, H-A-165
2026-11-20 21:42:33 +01:00
jschoubben 20d59ff4f0 mesh-grants: grants-for keeps the catalog revisions it read
grants-for read two whole catalog revisions page by page on every call
(about 1.4 MB each) and answered in 3.4 to 5.0 s, so the pipeline
runner's 5 s read bound cut it and every widening touching the vault
said "could not run". A revision never changes, so the part now keeps
each revision it read whole, by branch, kind, group, heads and number,
the newest four per branch for the process's life (as the runner keeps
its revisionCache); once a first page names a kept revision the pages
after it are not read. The branch's newest is read on by the number its
first page names, and a revision whose pages name another is not kept.

Quick fix (not chosen): raise the runner's widens read bound, a catalog
merge that hides the cost. Thorough fix (chosen): the cache here. The
compose of every account twice is kept whole: narrowing it to the
candidate's modules would stop a refused principal elsewhere from
failing grants-for, which is not obviously safe.

Builds: design 47 G §441, §401; H §774
2026-11-20 21:40:15 +01:00
jschoubben 12b18721b9 Build a module with a program naming no paths from its whole directory (H-A-186)
Since #290 the builder's manifest declares its packages builds and buildrun with paths beside the program and
tools, which name none. ModuleOf took the entry's named paths of any artifact as the module's, so the lane's one
fetch was narrowed to builds/ and buildrun/: the module's go.mod and cmd/ were removed and the program's build
failed with "go: cannot find main module" (main.42, 21:10 and 21:13 CET). An artifact without paths keeps its
module's directory as its build source (ADR 0499 clause 1); only a module whose every artifact names its paths is
built from those alone (clauses 2 and 4), as the indexer's own value already has it.

Builds: design 47 H H-A-186, H-A-188; ADR 0499 clauses 1, 2, 4
2026-11-20 21:28:39 +01:00
jschoubben 3183e0e056 Design 47 H1: a mover's unlogged entries live under its mover key, each entry names its mover, and list answers records with counts (H-A-189, H-A-190, H-A-191)
The record still kept every ended step's unlogged step-log entry, so a step
log outage across a large pipeline grew the record as its steps had. A
mover key now holds its steps and its unlogged entries, written in the same
compare-and-set; the record keeps neither. An entry of the step log names
its mover's module, set where an entry is made, so two movers' approvals
ending at one moment are filed under their own keys and told apart when
read back; an older entry naming none goes under the one mover whose step it
matches, else stays in the record. list answers each pipeline's record with
its movers and steps counted and its steps empty; show keeps the steps. The
seat's definition changes: its Go and TypeScript contracts are generated
again.

Builds: design 47 H H-A-187, H-A-189, H-A-190, H-A-191
2026-11-20 21:22:09 +01:00
jschoubben 30ccad4edc Assigner tests: hold putInto's writes one at a time
The composition step starts its machines' runs side by side, and each may
raise assignment-unlisted into the test's map: unguarded, the unit suite
crashed now and then with concurrent map writes. Only the test raced; the
running module writes the bus's findings state.

Builds: design 47 H §774
2026-11-20 21:15:50 +01:00
jschoubben 4e585f02b1 Design 47 H1: the runner keeps each mover's steps under its own key, so a record never grows with the catalog (H-A-187)
The runner's state pipelines held a pipeline's every step in its record, and
a record of 112 or 129 movers exceeded the bus's 1 MiB message limit (ADR
0486), so main.26 and main.30 placed no step. One MoverStore per process now
reads and writes the state: the record keeps the movers, tiers, call, source
and outcome with its own steps empty, each mover's steps are written under
steps.<pipeline>.<mover> by compare-and-set as the record is, and nothing is
written when the record or a mover key changed since the read. Every read
and judgement of steps goes through it, so the placing, the step ends, the
end judgement, the hand acts and the ageing see one composed pipeline; show,
list, stands and why-carried compose from the mover keys, and show naming a
mover answers its steps alone. A record still carrying its steps inline is
read whole, said once in the journal, and split at its next write, its
mover keys written before the record.

Builds: design 47 H H-A-187
2026-11-20 21:11:59 +01:00
jschoubben f2d3172eac Design 47 H1: the seat mesh-pipeline-runner keeps a mover's steps under its own key, and show names a mover (H-A-187)
A pipeline record of 112 or 129 movers exceeded the bus's message limit once
its steps were placed (main.26, main.30), so its steps could never be
written. The state pipelines gains the key form steps.<pipeline>.<mover>,
one mover's steps, and show takes an optional mover whose steps it answers
alone. The seat's definition changes: its Go and TypeScript contracts are
generated again.

Builds: design 47 H H-A-187
2026-11-20 21:11:59 +01:00
mesh-tools e4781edcd6 Merge pull request 'Design 47 H1: the Go packages other modules import are declared artifacts of their modules (H-A-186, H-A-188)' (#290) from fix/47-H1-declare-package-artifacts into main 2026-11-20 20:08:40 +00:00
jschoubben 1db38fcc72 Declare asks in the assigner's version series, and have the teller and the view accept it there (H-A-188)
The builder numbers every artifact of a release in its module's version, so the assigner's package asks is
published in 0.3, not the 0.1 eba4309b declared: asks now names series 0.3, and the teller and the view, which
import it, accept it at 0.3 in their manifests' packages. The other eight packages' modules are all at 0.1, so
their series stand. The gate in mesh-catalog, which imports asks too, follows with the catalog's next change.

Builds: design 47 G §370, H-A-186, H-A-188
2026-11-20 21:05:39 +01:00
jschoubben eba4309b19 Declare every Go package another module imports as a package artifact of its module (H-A-186)
mesh-build-go and other modules import packages of this repository that no manifest declared, so no release
published them and the registry holds no version: their importers stay unpinned and fail past the bound. Each
module now declares its imported packages in the form they had before issue 1966's bootstrap took them out:
asks (assigner), builds and buildrun (builder), forgemsg (forge bridge), takein (indexer), tellerevents
(teller), viewstate (view), grantsstate and busgrants (grants), each a Go package of series 0.1. Each
module's manifest test asserts its packages again.

Builds: design 47 G §370, H-A-186
2026-11-20 20:58:06 +01:00
mesh-tools a7fc37561e Merge pull request 'Design 47 H1: a pin is the version the registry published for the package path within the module's accepted series (H-A-183, H-A-184, H-A-185)' (#287) from fix/47-H1-pins-by-package-path into main 2026-11-20 19:15:04 +00:00
jschoubben e93575ffe7 mesh-delivery-indexer: the accepted series is the manifest's, contents or none
The pin fell back to the path's newest publish while no series value
held contents, which live none does, so a publish in a series a module
does not accept pinned it. The series a module accepts is the one its
manifest names, or the default; the pin is the newest version published
for the package path within it.

Builds: design 47 H H-A-185, H-A-184
2026-11-20 20:13:15 +01:00
jschoubben 859f88b2f0 mesh-delivery-indexer: a pin stays within the series the module accepts
A pin by the package path across every series moved a module to a
series it does not accept while its contents came from the accepted
one. The pin is the accepted series' newest published version where
that series gives the contents; only where none does yet is it the
newest published of the path. A publish in another series pins nothing
and moves nothing.

Builds: design 47 H H-A-184, H-A-183
2026-11-20 20:07:46 +01:00
jschoubben 517a4b3cf1 mesh-delivery-indexer: a pin is the version the registry published for the package's path
At main.39 every importer of a nested package was pinned to the release
number of the module holding it (v0.1.97 of the builder for its package
builds), and the Go seat found no tag <package dir>/v0.1.97. A Go
package's version is its tag: the pin is now the newest version the
registry published for that package path (the seats' published, kept in
the package's series), never a module's release; a package the
registry never published stays unpinned.

Builds: design 47 H H-A-183, H-A-167, G §370
2026-11-20 20:05:52 +01:00
166 changed files with 6129 additions and 1004 deletions
@@ -216,7 +216,7 @@ type runPrep struct {
seats []json.RawMessage seats []json.RawMessage
manifests map[string]composer.Manifest manifests map[string]composer.Manifest
none map[string]string none map[string]string
programOf func(node string) (buildstep.RunProgram, string, error) programOf func(node string) (buildstep.RunProgram, string, map[string]string, error)
err error err error
} }
@@ -274,7 +274,7 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
end(node + " gave no composition input: " + in.Why) end(node + " gave no composition input: " + in.Why)
return return
} }
program, engineFp, err := prep.programOf(node) program, engineFp, pins, err := prep.programOf(node)
if err != nil { if err != nil {
end(node + ": " + err.Error()) end(node + ": " + err.Error())
return return
@@ -317,7 +317,8 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
return return
} }
a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(), a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(),
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the composition of " + node + " by the node-engine at " + program.Commit}} Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the composition of " + node + " by the node-engine at " + program.Commit},
Packages: pins} // the composer is built against its entry's pins, as every build (H-A-203, H-A-192)
if err := r.put(ctx, a); err != nil { if err := r.put(ctx, a); err != nil {
end(node + ": its run ask " + ask + " is not put: " + err.Error()) end(node + ": its run ask " + ask + " is not put: " + err.Error())
} }
@@ -732,53 +733,76 @@ func programOfEntry(e meshcatalog.CatalogEntry) (buildstep.RunProgram, error) {
return buildstep.RunProgram{Repository: e.Repository, Commit: e.Commit, Dir: dir, Package: composer.Package}, nil return buildstep.RunProgram{Repository: e.Repository, Commit: e.Commit, Dir: dir, Package: composer.Package}, nil
} }
// pinsOfEntry are the mesh packages' versions an entry pins, the ones it names with a version (H-A-203, H-A-183): the pins
// a run's program is built against, in the form the entry keeps and the builder sends. nil when it pins none.
func pinsOfEntry(e meshcatalog.CatalogEntry) map[string]string {
var out map[string]string
for path, version := range e.Packages {
if version == "" {
continue
}
if out == nil {
out = map[string]string{}
}
out[path] = version
}
return out
}
// programs answers, per machine, the composer its run builds and the node-engine fingerprint it was chosen from (A-226, A-243, // programs answers, per machine, the composer its run builds and the node-engine fingerprint it was chosen from (A-226, A-243,
// A-252, §719): where the revision's node-engine entry differs from the trunk's, that build for every machine; else the build the // A-252, §719): where the revision's node-engine entry differs from the trunk's, that build for every machine; else the build the
// machine's node-engine runs, as its `running` says, its entry read through mesh-catalog's `entry`; a machine whose node-engine's // machine's node-engine runs, as its `running` says, its entry read through mesh-catalog's `entry`; a machine whose node-engine's
// `running` is not heard gets the revision's entry, said (A-451). // `running` is not heard gets the revision's entry, said (A-451).
func (r *compositionRun) programs(ctx context.Context, half mcc.Half) (func(node string) (buildstep.RunProgram, string, error), error) { func (r *compositionRun) programs(ctx context.Context, half mcc.Half) (func(node string) (buildstep.RunProgram, string, map[string]string, error), error) {
engine, ok := nodeEngineEntry(half.Entries) engine, ok := nodeEngineEntry(half.Entries)
if !ok { if !ok {
return nil, fmt.Errorf("the catalog revision %d holds no module claiming the seat %s", half.Revision.Revision, ne.Seat) return nil, fmt.Errorf("the catalog revision %d holds no module claiming the seat %s", half.Revision.Revision, ne.Seat)
} }
if trunk, ok := r.trunk(); !ok || func() bool { t, ok := nodeEngineEntry(trunk.Entries); return !ok || t.Fp != engine.Fp }() { if trunk, ok := r.trunk(); !ok || func() bool { t, ok := nodeEngineEntry(trunk.Entries); return !ok || t.Fp != engine.Fp }() {
p, err := programOfEntry(engine) p, err := programOfEntry(engine)
return func(string) (buildstep.RunProgram, string, error) { return p, engine.Fp, err }, nil return func(string) (buildstep.RunProgram, string, map[string]string, error) {
return p, engine.Fp, pinsOfEntry(engine), err
}, nil
} }
var mu sync.Mutex var mu sync.Mutex
byFp := map[string]buildstep.RunProgram{} type built struct {
return func(node string) (buildstep.RunProgram, string, error) { program buildstep.RunProgram
pins map[string]string
}
byFp := map[string]built{}
return func(node string) (buildstep.RunProgram, string, map[string]string, error) {
running, ok := r.running(node, engine.Module) running, ok := r.running(node, engine.Module)
if !ok || running.Fingerprint == "" { if !ok || running.Fingerprint == "" {
r.logf("%s: no build of %s is heard running there, so its composer is the catalog revision's entry %s (A-451)", node, r.logf("%s: no build of %s is heard running there, so its composer is the catalog revision's entry %s (A-451)", node,
engine.Module, engine.Fp) engine.Module, engine.Fp)
p, err := programOfEntry(engine) p, err := programOfEntry(engine)
return p, engine.Fp, err return p, engine.Fp, pinsOfEntry(engine), err
} }
fp := running.Fingerprint fp := running.Fingerprint
mu.Lock() mu.Lock()
p, ok := byFp[fp] b, ok := byFp[fp]
mu.Unlock() mu.Unlock()
if ok { if ok {
return p, fp, nil return b.program, fp, b.pins, nil
} }
what := "the entry of the node-engine's build " + fp + " is not read" what := "the entry of the node-engine's build " + fp + " is not read"
body, said, err := r.callRetry(ctx, entryVerbKey, meshcatalog.EntryInput{Fingerprint: fp}) body, said, err := r.callRetry(ctx, entryVerbKey, meshcatalog.EntryInput{Fingerprint: fp})
if err != nil || body == nil { if err != nil || body == nil {
return buildstep.RunProgram{}, fp, unreadErr(what, err, said) return buildstep.RunProgram{}, fp, nil, unreadErr(what, err, said)
} }
var a meshcatalog.EntryAnswer var a meshcatalog.EntryAnswer
if err := json.Unmarshal(body, &a); err != nil { if err := json.Unmarshal(body, &a); err != nil {
return buildstep.RunProgram{}, fp, fmt.Errorf("%s: %w", what, err) return buildstep.RunProgram{}, fp, nil, fmt.Errorf("%s: %w", what, err)
} }
p, err = programOfEntry(a.Entry) p, err := programOfEntry(a.Entry)
if err != nil { if err != nil {
return p, fp, err return p, fp, nil, err
} }
pins := pinsOfEntry(a.Entry)
mu.Lock() mu.Lock()
byFp[fp] = p byFp[fp] = built{program: p, pins: pins}
mu.Unlock() mu.Unlock()
return p, fp, nil return p, fp, pins, nil
}, nil }, nil
} }
@@ -53,7 +53,7 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
s := &runStand{asks: map[string]buildstep.StepAsk{}, requests: map[string]composer.Request{}} s := &runStand{asks: map[string]buildstep.StepAsk{}, requests: map[string]composer.Request{}}
engine := func(fp, commit string) meshcatalog.CatalogEntry { engine := func(fp, commit string) meshcatalog.CatalogEntry {
return meshcatalog.CatalogEntry{Module: "engine", Fp: fp, Repository: "novox/mesh-host", Commit: commit, Path: ".", Serves: []string{ne.Seat + ".composition"}, return meshcatalog.CatalogEntry{Module: "engine", Fp: fp, Repository: "novox/mesh-host", Commit: commit, Path: ".", Serves: []string{ne.Seat + ".composition"},
Manifest: map[string]any{"module": "engine"}} Manifest: map[string]any{"module": "engine"}, Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.25-" + fp, "git.novox.be/novox/x/go": ""}}
} }
revisionCommit := commitE1 revisionCommit := commitE1
if revisionEngine != "src2:e1" { if revisionEngine != "src2:e1" {
@@ -186,6 +186,13 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
if s.asks["g14"].Run.Program.Commit != commitE2 { if s.asks["g14"].Run.Program.Commit != commitE2 {
t.Errorf("g14's composer is built at %s, want the build its node-engine runs", s.asks["g14"].Run.Program.Commit) t.Errorf("g14's composer is built at %s, want the build its node-engine runs", s.asks["g14"].Run.Program.Commit)
} }
// Each composer is built against the pins of the node-engine entry it is built from, a pin with no version left out
// (H-A-203, H-A-192).
for node, fp := range map[string]string{"ace": "src2:e1", "g14": "src2:e2"} {
if got := s.asks[node].Packages; len(got) != 1 || got["git.novox.be/novox/mesh-sdk/go"] != "v0.1.25-"+fp {
t.Errorf("%s's composer is built against the pins %v; want its entry's", node, got)
}
}
if req.Depth != composer.DepthCompose || req.BlameTries != 20 || len(req.Machines) != 1 || req.Machines[0] != "ace" || len(req.Cases) != 1 || req.Cases[0].Name != changeCase || if req.Depth != composer.DepthCompose || req.BlameTries != 20 || len(req.Machines) != 1 || req.Machines[0] != "ace" || len(req.Cases) != 1 || req.Cases[0].Name != changeCase ||
string(req.Input) != `{"node":"ace"}` || len(req.Seats) != 1 || string(req.Seats[0]) != `{"seat":"s1"}` { string(req.Input) != `{"node":"ace"}` || len(req.Seats) != 1 || string(req.Seats[0]) != `{"seat":"s1"}` {
t.Errorf("the request of ace %+v", req) t.Errorf("the request of ace %+v", req)
@@ -19,6 +19,7 @@ import (
"runtime" "runtime"
"slices" "slices"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
@@ -203,11 +204,17 @@ func (w *watchKV) Keys(context.Context, ...jetstream.WatchOpt) ([]string, error)
return nil, jetstream.ErrNoKeysFound return nil, jetstream.ErrNoKeysFound
} }
// putIntoMu holds putInto's writes one at a time: the composition step starts its machines' runs side by side, and each may
// raise a finding into the same map.
var putIntoMu sync.Mutex
// putInto is an assign.Findings Put that keeps each finding in into, as the finding writer would, by its key and its words. // putInto is an assign.Findings Put that keeps each finding in into, as the finding writer would, by its key and its words.
func putInto(into map[string][]byte) func(mcseat.Finding) error { func putInto(into map[string][]byte) func(mcseat.Finding) error {
return func(f mcseat.Finding) error { return func(f mcseat.Finding) error {
raw, err := json.Marshal(f) raw, err := json.Marshal(f)
putIntoMu.Lock()
into[views.FindingKey(f.Scope, f.ID, f.Kind)] = raw into[views.FindingKey(f.Scope, f.ID, f.Kind)] = raw
putIntoMu.Unlock()
return err return err
} }
} }
@@ -169,7 +169,7 @@ func (e *pipelineEnds) take(p mpr.PipelineEnded) (bool, error) {
changed := false changed := false
var kept []tierMover var kept []tierMover
defer func() { defer func() {
if err := e.startTiers(p.Pipeline, kept); err != nil { if err := e.startTiers(p.Pipeline, p.Source, kept); err != nil {
e.logf("the tier progress of %s is not written: %v", p.Pipeline, err) e.logf("the tier progress of %s is not written: %v", p.Pipeline, err)
} }
}() }()
@@ -42,11 +42,14 @@ type tierMover struct {
Tier int64 `json:"tier"` Tier int64 `json:"tier"`
} }
// tierProgress is a pipeline's tier progress (H-A-56): the tier whose wants are written, since when, and the movers. // tierProgress is a pipeline's tier progress (H-A-56): the tier whose wants are written, since when, the movers, and the source
// revision its pipeline-ended names, said in pipeline-wanted beside the key's revision (design 47 H-A-214, H-A-216); zero in a
// value kept before the field, read as the key's number.
type tierProgress struct { type tierProgress struct {
Tier int64 `json:"tier"` Tier int64 `json:"tier"`
Since time.Time `json:"since"` Since time.Time `json:"since"`
Movers []tierMover `json:"movers"` Movers []tierMover `json:"movers"`
Source int64 `json:"source,omitempty"`
} }
// tierKey is the key of pipeline's tier progress; pipeline is the runner's key `<branch>.<source>`, its branch escaped. // tierKey is the key of pipeline's tier progress; pipeline is the runner's key `<branch>.<source>`, its branch escaped.
@@ -60,7 +63,7 @@ func isTierKey(key string) bool {
// startTiers writes the tier progress of a passed pipeline whose movers' versions are now kept, its first tier the lowest of // startTiers writes the tier progress of a passed pipeline whose movers' versions are now kept, its first tier the lowest of
// theirs, since now. A pipeline heard again writes nothing. // theirs, since now. A pipeline heard again writes nothing.
func (e *pipelineEnds) startTiers(pipeline string, movers []tierMover) error { func (e *pipelineEnds) startTiers(pipeline string, source int64, movers []tierMover) error {
if len(movers) == 0 { if len(movers) == 0 {
return nil return nil
} }
@@ -68,7 +71,7 @@ func (e *pipelineEnds) startTiers(pipeline string, movers []tierMover) error {
for _, m := range movers { for _, m := range movers {
first = min(first, m.Tier) first = min(first, m.Tier)
} }
value, err := json.Marshal(tierProgress{Tier: first, Since: e.clock().UTC(), Movers: movers}) value, err := json.Marshal(tierProgress{Tier: first, Since: e.clock().UTC(), Movers: movers, Source: source})
if err != nil { if err != nil {
return err return err
} }
@@ -131,7 +134,7 @@ func (p *passer) tiers(n assign.Nodes, placedOn func(module, branch string) []st
continue continue
} }
pipeline := strings.TrimPrefix(key, tierPrefix) pipeline := strings.TrimPrefix(key, tierPrefix)
branchEsc, source, _ := strings.Cut(pipeline, ".") branchEsc, number, _ := strings.Cut(pipeline, ".")
branch, err := keys.UnescapeName(branchEsc) branch, err := keys.UnescapeName(branchEsc)
if err != nil { if err != nil {
continue continue
@@ -144,7 +147,7 @@ func (p *passer) tiers(n assign.Nodes, placedOn func(module, branch string) []st
notRunning := p.notRunning(n, placedOn, pipeline, branch, t, t.Tier) notRunning := p.notRunning(n, placedOn, pipeline, branch, t, t.Tier)
late := !now.Before(t.Since.Add(bound)) late := !now.Before(t.Since.Add(bound))
if late { if late {
p.tierLate(pipeline, branch, source, t, notRunning) p.tierLate(pipeline, branch, number, t, notRunning)
break break
} }
if len(notRunning) > 0 { if len(notRunning) > 0 {
@@ -165,10 +168,10 @@ func (p *passer) tiers(n assign.Nodes, placedOn func(module, branch string) []st
p.logf("the tier progress %s is not deleted: %v", key, err) p.logf("the tier progress %s is not deleted: %v", key, err)
break break
} }
p.sayPipelineWanted(pipeline, branch, source, meshwants.PipelineWantedOutcomeRunning, nil) p.sayPipelineWanted(pipeline, branch, number, t.Source, meshwants.PipelineWantedOutcomeRunning, nil)
break break
} }
value, err := json.Marshal(tierProgress{Tier: later, Since: now, Movers: t.Movers}) value, err := json.Marshal(tierProgress{Tier: later, Since: now, Movers: t.Movers, Source: t.Source})
if err == nil { if err == nil {
err = p.ends.st.swap(key, value, rev) err = p.ends.st.swap(key, value, rev)
} }
@@ -217,7 +220,7 @@ func (p *passer) notRunning(n assign.Nodes, placedOn func(module, branch string)
// tierLate ends a pipeline's delivery at a tier past tier-bound: the finding tier-late is raised and pipeline-wanted says tier-late, // tierLate ends a pipeline's delivery at a tier past tier-bound: the finding tier-late is raised and pipeline-wanted says tier-late,
// once each; the finding clears once the late movers run or a later pipeline moved them. The progress stays, so no later tier is // once each; the finding clears once the late movers run or a later pipeline moved them. The progress stays, so no later tier is
// written, until no later tier's mover is kept at this pipeline any more. // written, until no later tier's mover is kept at this pipeline any more.
func (p *passer) tierLate(pipeline, branch, source string, t tierProgress, notRunning []meshwants.NotRunning) { func (p *passer) tierLate(pipeline, branch, number string, t tierProgress, notRunning []meshwants.NotRunning) {
o := tierLateFinding(pipeline, t.Tier, notRunning) o := tierLateFinding(pipeline, t.Tier, notRunning)
key := views.FindingKey(o.Scope, assign.KeyID(o.ID), o.Kind) key := views.FindingKey(o.Scope, assign.KeyID(o.ID), o.Kind)
if len(notRunning) > 0 { if len(notRunning) > 0 {
@@ -231,7 +234,7 @@ func (p *passer) tierLate(pipeline, branch, source string, t tierProgress, notRu
p.said[key+".raised"] = true p.said[key+".raised"] = true
delete(p.said, key+".cleared") delete(p.said, key+".cleared")
p.logf("pipeline %s: tier %d is late (tier-bound %s): %s; no later tier is written", pipeline, t.Tier, p.tierBoundOf(), o.Summary) p.logf("pipeline %s: tier %d is late (tier-bound %s): %s; no later tier is written", pipeline, t.Tier, p.tierBoundOf(), o.Summary)
p.sayPipelineWanted(pipeline, branch, source, meshwants.PipelineWantedOutcomeTierLate, notRunning) p.sayPipelineWanted(pipeline, branch, number, t.Source, meshwants.PipelineWantedOutcomeTierLate, notRunning)
} }
} else if !p.said[key+".cleared"] { } else if !p.said[key+".cleared"] {
if p.findings.Delete != nil { if p.findings.Delete != nil {
@@ -271,11 +274,16 @@ func tierLateFinding(pipeline string, tier int64, notRunning []meshwants.NotRunn
Source: "assigner." + kindTierLate} Source: "assigner." + kindTierLate}
} }
// sayPipelineWanted says the seat mesh-wants' pipeline-wanted of pipeline, once (its id the pipeline and the outcome). // sayPipelineWanted says the seat mesh-wants' pipeline-wanted of pipeline, once (its id the pipeline and the outcome): its
func (p *passer) sayPipelineWanted(pipeline, branch, source, outcome string, notRunning []meshwants.NotRunning) { // revision the number of the pipeline's key `<branch>.<revision>`, its source the one the pipeline-ended named (design 47
var src int64 // H-A-214, H-A-216), the key's number for a progress kept before it held one.
_, _ = fmt.Sscan(source, &src) func (p *passer) sayPipelineWanted(pipeline, branch, number string, source int64, outcome string, notRunning []meshwants.NotRunning) {
body := meshwants.PipelineWantedEvent{V: keys.Version, Pipeline: pipeline, Branch: branch, Source: src, Outcome: outcome, NotRunning: notRunning} var revision int64
_, _ = fmt.Sscan(number, &revision)
if source <= 0 {
source = revision
}
body := meshwants.PipelineWantedEvent{V: keys.Version, Pipeline: pipeline, Branch: branch, Source: source, Revision: revision, Outcome: outcome, NotRunning: notRunning}
event := meshwants.EventPipelineWanted + "." + pipeline event := meshwants.EventPipelineWanted + "." + pipeline
if err := p.wantsSay(event, body, "pipeline-wanted."+pipeline+"."+outcome); err != nil { if err := p.wantsSay(event, body, "pipeline-wanted."+pipeline+"."+outcome); err != nil {
p.logf("pipeline-wanted of %s is not said: %v", pipeline, err) p.logf("pipeline-wanted of %s is not said: %v", pipeline, err)
@@ -62,8 +62,11 @@ func TestAPipelinesWantsAreWrittenTierByTier(t *testing.T) {
p.w.mu.Unlock() p.w.mu.Unlock()
p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf) p.ends = newPipelineEnds(newMemEnds(), func() {}, t.Logf)
p.ends.trunk, p.ends.now = p.w.trunkName, func() time.Time { return now } p.ends.trunk, p.ends.now = p.w.trunkName, func() time.Time { return now }
if _, err := p.ends.take(ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("db", db, "0.2.0", 1, mpr.StepResultOutcomePassed), // The pipeline of revision 5 (its key's number) of source 4: a re-pin's revision (H-A-213, H-A-216).
mover("web", web, "0.1.2", 2, mpr.StepResultOutcomePassed))); err != nil { e := ended(trunkBranch, 5, mpr.PipelineEndedOutcomePassed, mover("db", db, "0.2.0", 1, mpr.StepResultOutcomePassed),
mover("web", web, "0.1.2", 2, mpr.StepResultOutcomePassed))
e.Source = 4
if _, err := p.ends.take(e); err != nil {
t.Fatal(err) t.Fatal(err)
} }
wanted := func(node, module string) bool { wanted := func(node, module string) bool {
@@ -88,7 +91,7 @@ func TestAPipelinesWantsAreWrittenTierByTier(t *testing.T) {
raw, _ = json.Marshal(nodeengine.Running{Node: "g14", Module: "web", Fingerprint: web}) raw, _ = json.Marshal(nodeengine.Running{Node: "g14", Module: "web", Fingerprint: web})
_ = p.w.hearRunning("running.g14.web", raw) _ = p.w.hearRunning("running.g14.web", raw)
p.wants(context.Background()) p.wants(context.Background())
if len(said) != 1 || said[0].Outcome != meshwants.PipelineWantedOutcomeRunning || len(findings) != 0 { if len(said) != 1 || said[0].Outcome != meshwants.PipelineWantedOutcomeRunning || said[0].Revision != 5 || said[0].Source != 4 || len(findings) != 0 {
t.Errorf("every tier runs: said %+v, findings %v", said, findings) t.Errorf("every tier runs: said %+v, findings %v", said, findings)
} }
if _, _, ok := p.ends.tiersOf(trunkBranch + ".5"); ok { if _, _, ok := p.ends.tiersOf(trunkBranch + ".5"); ok {
@@ -108,8 +111,14 @@ func TestAPipelinesWantsAreWrittenTierByTier(t *testing.T) {
if !ok || !strings.Contains(f.Summary, "db on g14") || strings.Contains(f.Summary, "ace") { if !ok || !strings.Contains(f.Summary, "db on g14") || strings.Contains(f.Summary, "ace") {
t.Errorf("tier-late is not raised naming db on g14 alone: %v", findings) t.Errorf("tier-late is not raised naming db on g14 alone: %v", findings)
} }
if len(said) != 1 || said[0].Outcome != meshwants.PipelineWantedOutcomeTierLate || len(said[0].NotRunning) != 1 || said[0].NotRunning[0].Module != "db" { if len(said) != 1 || said[0].Outcome != meshwants.PipelineWantedOutcomeTierLate || len(said[0].NotRunning) != 1 || said[0].NotRunning[0].Module != "db" ||
t.Errorf("pipeline-wanted tier-late is not said once: %+v", said) said[0].Revision != 5 || said[0].Source != 4 {
t.Errorf("pipeline-wanted tier-late is not said once, naming revision 5 of source 4: %+v", said)
}
// A tier progress kept before it held its source says the key's number as both (H-A-216).
p.sayPipelineWanted(trunkBranch+".5", trunkBranch, "5", 0, meshwants.PipelineWantedOutcomeRunning, nil)
if len(said) != 2 || said[1].Revision != 5 || said[1].Source != 5 {
t.Errorf("a progress with no source: said %+v, want revision and source 5", said)
} }
if wanted("g14", "web") || wanted("ace", "web") { if wanted("g14", "web") || wanted("ace", "web") {
t.Error("a consumer's want is written after its provider's tier was late") t.Error("a consumer's want is written after its provider's tier was late")
@@ -345,7 +345,8 @@ func (r *compositionRun) startValidate(ctx context.Context, key string, run mesh
return return
} }
a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(), a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(),
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the manifest validator of " + step.Pipeline + " at " + program.Commit}} Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the manifest validator of " + step.Pipeline + " at " + program.Commit},
Packages: pinsOfEntry(entry)} // the validator is built against its entry's pins at the step's revision (H-A-203, H-A-192, H-A-205)
if err := r.put(ctx, a); err != nil { if err := r.put(ctx, a); err != nil {
couldNot("its run ask " + ask + " is not put: " + err.Error()) couldNot("its run ask " + ask + " is not put: " + err.Error())
} }
@@ -25,7 +25,8 @@ import (
// is judged against an empty case before, said in its end, and no revision below it is asked. // is judged against an empty case before, said in its end, and no revision below it is asked.
func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) { func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
indexer := meshcatalog.CatalogEntry{Module: "indexer", Repository: "novox/mesh-delivery", Commit: commitE3, Path: "modules/mesh-delivery-indexer", indexer := meshcatalog.CatalogEntry{Module: "indexer", Repository: "novox/mesh-delivery", Commit: commitE3, Path: "modules/mesh-delivery-indexer",
Fp: "src2:ix", Serves: []string{"seat:" + meshcatalog.Seat + ".revision"}, Manifest: map[string]any{"module": "indexer"}, ManifestDigest: "d-ix"} Fp: "src2:ix", Serves: []string{"seat:" + meshcatalog.Seat + ".revision"}, Manifest: map[string]any{"module": "indexer"}, ManifestDigest: "d-ix",
Packages: map[string]string{"git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient": "v0.1.4"}}
entry := func(module, digest string, edges ...meshcatalog.Edge) meshcatalog.CatalogEntry { entry := func(module, digest string, edges ...meshcatalog.Edge) meshcatalog.CatalogEntry {
return meshcatalog.CatalogEntry{Module: module, Manifest: map[string]any{"module": module, "d": digest}, ManifestDigest: digest, Edges: edges} return meshcatalog.CatalogEntry{Module: module, Manifest: map[string]any{"module": module, "d": digest}, ManifestDigest: digest, Edges: edges}
} }
@@ -124,6 +125,10 @@ func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
if p := asks[0].Run.Program; p.Commit != commitE3 || p.Dir != "modules/mesh-delivery-indexer" || p.Package != validatorPackage || len(asks[0].Run.Heads) != 0 { if p := asks[0].Run.Program; p.Commit != commitE3 || p.Dir != "modules/mesh-delivery-indexer" || p.Package != validatorPackage || len(asks[0].Run.Heads) != 0 {
t.Errorf("the validator is built from %+v", asks[0].Run) t.Errorf("the validator is built from %+v", asks[0].Run)
} }
// The validator is built against the indexer entry's pins at the step's revision (H-A-203, H-A-192, H-A-205).
if got := asks[0].Packages; len(got) != 1 || got["git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"] != "v0.1.4" {
t.Errorf("the validator is built against the pins %v; want its entry's", got)
}
if c := requests[0].Cases; len(c) != 2 || c[0].Name != changeCase || c[1].Name != beforeCase || len(c[1].Manifests) != 3 || requests[0].Predefined != nil || requests[0].Packages != nil { if c := requests[0].Cases; len(c) != 2 || c[0].Name != changeCase || c[1].Name != beforeCase || len(c[1].Manifests) != 3 || requests[0].Predefined != nil || requests[0].Packages != nil {
t.Errorf("the validator's request is %+v", requests[0]) t.Errorf("the validator's request is %+v", requests[0])
} }
@@ -119,6 +119,13 @@ func TestManifest(t *testing.T) {
} }
} }
} }
// H-A-186 (design 47 G §370): asks, which the gate, the teller and the view import, is a declared Go package of the
// module, so every release publishes it; H-A-188: in the module's version series, 0.3.
if !slices.ContainsFunc(m.Build.Artifacts, func(a catalogue.Artifact) bool {
return a.Name == "asks" && a.Kind == catalogue.ArtifactPackage && a.Language == "go" && slices.Equal(a.Paths, []string{"asks"}) && a.Series == m.Version && m.Version == "0.3"
}) {
t.Errorf("artifacts = %+v, want the Go package asks of series 0.3, the module's", m.Build.Artifacts)
}
// ADR 0635 §5: its views family is its ordinary events, `views.<state>` among its emits, on no stream of its own. // ADR 0635 §5: its views family is its ordinary events, `views.<state>` among its emits, on no stream of its own.
// The seat's four events are said on the seat's own subject (the seat's family), so the module keeps no stream for them // The seat's four events are said on the seat's own subject (the seat's family), so the module keeps no stream for them
// and lists none among its emits. // and lists none among its emits.
@@ -370,6 +370,15 @@
"loads": [ "loads": [
"mesh-delivery-assigner" "mesh-delivery-assigner"
] ]
},
{
"name": "asks",
"kind": "package",
"language": "go",
"paths": [
"asks"
],
"series": "0.3"
} }
] ]
} }
@@ -242,6 +242,11 @@ type BuilderBuild struct {
// Imports are, on a done key, the mesh packages the built programs import, as the build module answered them // Imports are, on a done key, the mesh packages the built programs import, as the build module answered them
// (BuildAnswer.Imports; novox/hq ADR 0593 §2): the builder carries them onto the build's end (BuildLogEntry.Imports). // (BuildAnswer.Imports; novox/hq ADR 0593 §2): the builder carries them onto the build's end (BuildLogEntry.Imports).
Imports []string `json:"imports,omitempty"` Imports []string `json:"imports,omitempty"`
// Packages are, on a done key only, the mesh packages' versions its Go build was built with, by module path, the ones
// the registry gave a package the entry named with none included (H-A-207): the build step's end reports them, so the
// runner asks the steps that waited for the pins with what the build was built against. A reader that does not know
// the field reads the build as reporting none.
Packages map[string]string `json:"packages,omitempty"`
// Needs are, on a done key only, the reader capabilities (`form:<name>`) the build's reference forms need of a // Needs are, on a done key only, the reader capabilities (`form:<name>`) the build's reference forms need of a
// node-engine, beside the ones its manifest needs (novox/hq ADR 0618 §1). A reader that does not know the field // node-engine, beside the ones its manifest needs (novox/hq ADR 0618 §1). A reader that does not know the field
// reads the build as needing none. // reads the build as needing none.
@@ -45,7 +45,7 @@ func (m *entryManifests) keep(e meshcatalog.CatalogEntry) error {
if m.byFp == nil { if m.byFp == nil {
m.byFp = map[string]catalogue.Manifest{} m.byFp = map[string]catalogue.Manifest{}
} }
m.byFp[meshcatalog.BareManifestDigest(e.ManifestDigest)] = mf m.byFp[meshcatalogclient.BareManifestDigest(e.ManifestDigest)] = mf
return nil return nil
} }
@@ -101,9 +101,7 @@ func (c *catalogHalf) Changed(ctx context.Context) func(branch, module string, e
} }
origin := build.Origin{Branch: branch, First: first} origin := build.Origin{Branch: branch, First: first}
if cat, ok := c.branches.Catalog(branch); ok { if cat, ok := c.branches.Catalog(branch); ok {
if half, ok := cat.View(); ok { origin = pictureOrigin(origin, cat.View)
origin.Source = half.Revision.Source
}
} }
var e meshcatalog.CatalogEntry var e meshcatalog.CatalogEntry
if err := json.Unmarshal(entry, &e); err != nil || e.Module != module { if err := json.Unmarshal(entry, &e); err != nil || e.Module != module {
@@ -132,6 +130,16 @@ func (c *catalogHalf) Changed(ctx context.Context) func(branch, module string, e
} }
} }
// pictureOrigin is the origin of an entry the picture takes in: the source revision and the catalog revision number of the
// half the view holds, one revision for both (design 47 G §371, H-A-218, H-A-220), so a build the picture starts before any
// step ask reserves a version that names its revision; none while the view holds no half.
func pictureOrigin(o build.Origin, view func() (meshcatalogclient.Half, bool)) build.Origin {
if half, ok := view(); ok {
o.Source, o.Revision = half.Revision.Source, half.Revision.Revision
}
return o
}
// picture is build.Builder.Picture: the source revision the view of branch holds (design 47 H H-A-155), false while the branch // picture is build.Builder.Picture: the source revision the view of branch holds (design 47 H H-A-155), false while the branch
// is not followed or its view holds none. // is not followed or its view holds none.
func (c *catalogHalf) picture(branch string) (int64, bool) { func (c *catalogHalf) picture(branch string) (int64, bool) {
@@ -4,7 +4,9 @@ import (
"context" "context"
"testing" "testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"mesh-delivery-builder/internal/build"
) )
// TestTheManifestOfAnEntryIsReadByItsDigest: the builder reads a manifest from the entry that holds it, by the entry's manifest // TestTheManifestOfAnEntryIsReadByItsDigest: the builder reads a manifest from the entry that holds it, by the entry's manifest
@@ -25,3 +27,18 @@ func TestTheManifestOfAnEntryIsReadByItsDigest(t *testing.T) {
t.Errorf("an entry with no manifest kept one: %v, %v", err, m.byFp) t.Errorf("an entry with no manifest kept one: %v, %v", err, m.byFp)
} }
} }
// H-A-220: an entry the picture takes in has the source revision and the catalog revision number of the half its view holds,
// one revision for both; a view holding no half names neither.
func TestAnEntryThePictureTakesInNamesItsHalfsRevision(t *testing.T) {
o := build.Origin{Branch: "main", First: true}
got := pictureOrigin(o, func() (meshcatalogclient.Half, bool) {
return meshcatalogclient.Half{Revision: meshcatalog.CatalogRevision{Revision: 47, Source: 45}}, true
})
if got != (build.Origin{Branch: "main", First: true, Source: 45, Revision: 47}) {
t.Errorf("origin = %+v, want source 45 at revision 47", got)
}
if got := pictureOrigin(o, func() (meshcatalogclient.Half, bool) { return meshcatalogclient.Half{}, false }); got != o {
t.Errorf("origin with no half = %+v, want %+v", got, o)
}
}
@@ -0,0 +1,47 @@
package main
import (
"context"
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// keptRevisionVia reads one kept revision of a branch by its number through the catalog seat's verb revision, page by page
// (design 47 H H-A-205, H-A-147): the source revision it was taken in from and the fingerprint of each module it names.
func keptRevisionVia(call toolCaller) func(context.Context, string, int64) (int64, map[string]string, error) {
return func(_ context.Context, branch string, revision int64) (int64, map[string]string, error) {
fps := map[string]string{}
var source int64
after := ""
for {
raw, err := call(meshcatalogclient.RevisionVerb, meshcatalog.RevisionInput{Branch: keys.EscapeName(branch), Revision: revision, After: after})
if err != nil {
return 0, nil, err
}
answer, said, ok := buscalls.Unwrap(raw)
if !ok {
return 0, nil, fmt.Errorf("%s answered: %s", meshcatalogclient.RevisionVerb, said)
}
var a meshcatalog.RevisionAnswer
if err := json.Unmarshal(answer, &a); err != nil {
return 0, nil, fmt.Errorf("%s answered no revision: %w", meshcatalogclient.RevisionVerb, err)
}
if a.NoneYet || a.Revision == nil || a.Revision.Revision != revision {
return 0, nil, fmt.Errorf("%s answered no revision %d of %s", meshcatalogclient.RevisionVerb, revision, branch)
}
source = a.Revision.Source
for _, e := range a.Entries {
fps[e.Module] = e.Fp
}
if a.Next == "" {
return source, fps, nil
}
after = a.Next
}
}
}
@@ -228,6 +228,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err return err
} }
b.PackageWait = time.Duration(packageWait) * time.Minute b.PackageWait = time.Duration(packageWait) * time.Minute
// Before it waits for an unpinned package, the build reads what the Go package registry seat holds of it (H-A-194).
b.PublishedVersions = publishedVersionsVia(calls.Call)
waitsBucket := process.AwaitBucket(ctx, js, conn, builds.StateBucket(cred.Module, build.WaitsState), "the toolchain waits are kept in memory only, and builder_waits shows none", logger.Printf) waitsBucket := process.AwaitBucket(ctx, js, conn, builds.StateBucket(cred.Module, build.WaitsState), "the toolchain waits are kept in memory only, and builder_waits shows none", logger.Printf)
b.Waits = newWaitsKept(waitsBucket, func(ctx context.Context, event, id string, body any) error { b.Waits = newWaitsKept(waitsBucket, func(ctx context.Context, event, id string, body any) error {
return emitEvent(context.WithoutCancel(ctx), js, cred, event, id, body) return emitEvent(context.WithoutCancel(ctx), js, cred, event, id, body)
@@ -242,9 +244,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return emitSeatEvent(context.WithoutCancel(ctx), js, cred, meshbuilds.SubjectPackagePublished, "package-published-"+p.Module+"-"+p.Version+"-"+p.Artifact, p) return emitSeatEvent(context.WithoutCancel(ctx), js, cred, meshbuilds.SubjectPackagePublished, "package-published-"+p.Module+"-"+p.Version+"-"+p.Artifact, p)
} }
b.Reserved = reserved b.Reserved = reserved
// A release heard, or the newest the builder's record keeps, pins a package an entry names with none (H-A-165). // A take-over of a series' reservation by another fingerprint withdraws the older build (H-A-225).
reserved.heardReleased = b.VersionReleased reserved.withdraw = b.Withdraw
b.NewestReleased = reserved.r.NewestReleased
// The last versions-filled the stream keeps is read at the start (design 47 G §369, §372): a restart does not lose it. // The last versions-filled the stream keeps is read at the start (design 47 G §369, §372): a restart does not lose it.
reserved.readFilled(ctx, lastKeptOf(js, cred, logger.Printf), sleeps) reserved.readFilled(ctx, lastKeptOf(js, cred, logger.Printf), sleeps)
// A state bound later numbers the builds looked at without it. // A state bound later numbers the builds looked at without it.
@@ -470,6 +471,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// a holder of the seat that is down holds no other part of it). // a holder of the seat that is down holds no other part of it).
half.b = b half.b = b
b.Picture = half.picture b.Picture = half.picture
// An ask of a revision below the newest the picture holds is read by number from the catalog seat (H-A-205).
b.KeptRevision = keptRevisionVia(calls.Call)
b.FirstBranch = half.first
branches.Held = func(string, meshcatalog.CatalogRevision) { b.PictureHeld() } // a revision held wakes a step ask's wait (H-A-162) branches.Held = func(string, meshcatalog.CatalogRevision) { b.PictureHeld() } // a revision held wakes a step ask's wait (H-A-162)
half.ready = func() { half.ready = func() {
if err := b.Sweep(ctx, half.pictured()...); err != nil { if err := b.Sweep(ctx, half.pictured()...); err != nil {
@@ -44,15 +44,14 @@ type reservations struct {
// built asks a done build's publish (design 47 G A-457a), released ends its item; nil are none. // built asks a done build's publish (design 47 G A-457a), released ends its item; nil are none.
built func(ctx context.Context, module, version string) error built func(ctx context.Context, module, version string) error
released func(ctx context.Context, module, version string) released func(ctx context.Context, module, version string)
// heardReleased is told each version-released taken in (design 47 H H-A-165): a step ask waiting for a package of the filled atomic.Bool
// module wakes. Nil tells nobody. bounds func() jetstream.KeyValue
heardReleased func(module, version string) found *findings
filled atomic.Bool bound time.Duration
bounds func() jetstream.KeyValue wake func(context.Context)
found *findings logf func(string, ...any)
bound time.Duration // withdraw ends the running build of a module's fingerprint, superseded, with why (H-A-225); nil ends none.
wake func(context.Context) withdraw func(module, fingerprint, why string)
logf func(string, ...any)
// call asks a seat's verb: the git seat's tag (design 47 G §370). // call asks a seat's verb: the git seat's tag (design 47 G §370).
call toolCaller call toolCaller
// say says package-published on the seat mesh-builds; nil says none. // say says package-published on the seat mesh-builds; nil says none.
@@ -95,6 +94,11 @@ func (rv *reservations) took(ctx context.Context, older, newer versions.Reserved
} }
rv.dropBound(ctx, versions.ReservedKey(older.Module, older.Series)) rv.dropBound(ctx, versions.ReservedKey(older.Module, older.Series))
rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, older.Module) rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, older.Module)
if older.Fingerprint != newer.Fingerprint && rv.withdraw != nil {
// The older build is ended superseded, as the picture's supersession ends one (H-A-225, ADR 0664 §4): nothing publishes
// under the dropped number.
rv.withdraw(older.Module, older.Fingerprint, "its reservation "+older.Version+" was taken over by the build of "+newer.Fingerprint+", which reserves "+newer.Version)
}
} }
var _ build.Reservations = (*reservations)(nil) var _ build.Reservations = (*reservations)(nil)
@@ -255,9 +259,6 @@ func (rv *reservations) heard(ctx context.Context, m jetstream.Msg) bool {
if rv.released != nil { if rv.released != nil {
rv.released(ctx, ev.Module, ev.Version) rv.released(ctx, ev.Module, ev.Version)
} }
if rv.heardReleased != nil {
rv.heardReleased(ev.Module, ev.Version)
}
rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, ev.Module) rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, ev.Module)
default: default:
return false return false
@@ -134,15 +134,10 @@ func TestTheCatalogsWordsMoveTheReservations(t *testing.T) {
if err != nil || !strings.Contains(got.Summary, "m:b unpublished") || strings.Contains(got.Summary, "m:a unpublished") || !strings.Contains(got.Summary, "1.4.3") { if err != nil || !strings.Contains(got.Summary, "m:b unpublished") || strings.Contains(got.Summary, "m:a unpublished") || !strings.Contains(got.Summary, "1.4.3") {
t.Fatalf("finding = %+v (%v), want version-unreleased naming m:b of 1.4.3", got, err) t.Fatalf("finding = %+v (%v), want version-unreleased naming m:b of 1.4.3", got, err)
} }
var heardReleases []string // H-A-165: a release heard wakes a step ask waiting for a package of the module
rv.heardReleased = func(m, v string) { heardReleases = append(heardReleases, m+" "+v) }
released := &wakeMsg{subject: meshcatalog.SubjectVersionReleased, data: []byte(`{"v":1,"module":"m","version":"1.4.3","source":1,"artifacts":[]}`)} released := &wakeMsg{subject: meshcatalog.SubjectVersionReleased, data: []byte(`{"v":1,"module":"m","version":"1.4.3","source":1,"artifacts":[]}`)}
if !rv.heard(ctx, released) || !released.acked { if !rv.heard(ctx, released) || !released.acked {
t.Fatalf("version-released was not taken in (acked %v)", released.acked) t.Fatalf("version-released was not taken in (acked %v)", released.acked)
} }
if strings.Join(heardReleases, ", ") != "m 1.4.3" {
t.Errorf("the release heard was told %v, want m 1.4.3 (H-A-165)", heardReleases)
}
if strings.Join(items, ", ") != "asked m 1.4.3, ended m 1.4.3" { if strings.Join(items, ", ") != "asked m 1.4.3, ended m 1.4.3" {
t.Errorf("publish items %v, want the build's end to ask the publish and the release to end it", items) t.Errorf("publish items %v, want the build's end to ask the publish and the release to end it", items)
} }
@@ -271,7 +266,8 @@ func TestAPublishedHeardBeforeTheDigestsIsSaidWhenTheBuildRecordsThem(t *testing
} }
// TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo: design 47 G A-455 — a build that reserves while the series is // TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo: design 47 G A-455 — a build that reserves while the series is
// reserved by an older build never released drops the older one's bound and finding and says so in the builder's journal. // reserved by an older build never released drops the older one's bound and finding and says so in the builder's journal; and
// H-A-225 — the older build is withdrawn, so nothing publishes under the dropped number, while a re-run of the newer keeps it.
func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T) { func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T) {
ctx := context.Background() ctx := context.Background()
st := &memFindings{vals: map[string][]byte{}} st := &memFindings{vals: map[string][]byte{}}
@@ -284,6 +280,8 @@ func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T)
rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, call, func() jetstream.KeyValue { return bounds }, found, rv := newReservations(&memReserve{vals: map[string][]byte{}, revs: map[string]uint64{}}, call, func() jetstream.KeyValue { return bounds }, found,
nil, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) }) nil, func(f string, a ...any) { journal = append(journal, fmt.Sprintf(f, a...)) })
rv.filled.Store(true) rv.filled.Store(true)
var withdrawn []string
rv.withdraw = func(module, fp, why string) { withdrawn = append(withdrawn, module+" "+fp+": "+why) }
older, err := rv.Reserve(ctx, "m", "0.1", []string{"m:a"}, versions.Source{Fingerprint: "fp-old"}) older, err := rv.Reserve(ctx, "m", "0.1", []string{"m:a"}, versions.Source{Fingerprint: "fp-old"})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -310,6 +308,12 @@ func TestABuildTakingOverAnOlderReservationDropsItsBoundAndSaysSo(t *testing.T)
if got, ok, _ := rv.ReservedFor(ctx, "m", "fp-new"); !ok || got.Version != "0.1.1" { if got, ok, _ := rv.ReservedFor(ctx, "m", "fp-new"); !ok || got.Version != "0.1.1" {
t.Errorf("reserved for fp-new: %+v %v", got, ok) t.Errorf("reserved for fp-new: %+v %v", got, ok)
} }
if len(withdrawn) != 1 || withdrawn[0] != "m fp-old: its reservation 0.1.0 was taken over by the build of fp-new, which reserves 0.1.1" {
t.Errorf("withdrawn %q; want the build of fp-old withdrawn once", withdrawn)
}
if again, err := rv.Reserve(ctx, "m", "0.1", []string{"m:a"}, versions.Source{Fingerprint: "fp-new"}); err != nil || again.Version != "0.1.1" || len(withdrawn) != 1 {
t.Errorf("a re-run of fp-new reserved %q (%v), withdrew %q; want 0.1.1 kept, nothing withdrawn", again.Version, err, withdrawn)
}
} }
// ansKV records the value versionActed writes back to a reasks key. // ansKV records the value versionActed writes back to a reasks key.
@@ -105,6 +105,10 @@ func (h *seatHearer) heard(ctx context.Context, m jetstream.Msg) bool {
if h.res != nil && se.event == versions.HeardPublished { if h.res != nil && se.event == versions.HeardPublished {
h.res.published(ctx, o) h.res.published(ctx, o)
} }
if se.event == versions.HeardPublished && o.Kind == versions.KindPackage && h.b != nil {
// A package registry seat's published pins a package an entry names with none, at the version it carries (H-A-192).
h.b.PackagePublished(o.Name, o.Version)
}
h.ack(m) h.ack(m)
if len(wake) > 0 { if len(wake) > 0 {
h.logf("the floor of %s is asked again: %s said %s", strings.Join(wake, ", "), se.seat, se.event) h.logf("the floor of %s is asked again: %s said %s", strings.Join(wake, ", "), se.seat, se.event)
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions" "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git" gitseat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/gopackageregistry"
"git.novox.be/novox/mesh-sdk/go/process" "git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build" "mesh-delivery-builder/internal/build"
@@ -96,6 +97,28 @@ func (s awaitedVersions) KeysOf(ctx context.Context, filter string) ([]string, e
return out, nil return out, nil
} }
// publishedVersionsVia asks the Go package registry seat's verb versions for one package path (design 47 H H-A-194): its
// versions, newest first.
func publishedVersionsVia(call toolCaller) func(context.Context, string) ([]string, error) {
return func(_ context.Context, path string) ([]string, error) {
raw, err := call("seat:"+gopackageregistry.Seat+"."+gopackageregistry.VerbVersions, gopackageregistry.VersionsInput{Name: path})
if err != nil {
return nil, err
}
var a gopackageregistry.VersionsAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("%s's versions answered what is no list of versions: %w", gopackageregistry.Seat, err)
}
var out []string
for _, v := range a.Versions {
if v.Version != "" {
out = append(out, v.Version)
}
}
return out, nil
}
}
// floorVia is the floor's three sources as calls of the builder's own account (design 51 *The floor*): the seat's verb // floorVia is the floor's three sources as calls of the builder's own account (design 51 *The floor*): the seat's verb
// versions, the seat git's tags and the requirements the catalog half's entries name (A-119). A call that fails is that source // versions, the seat git's tags and the requirements the catalog half's entries name (A-119). A call that fails is that source
// silent. // silent.
@@ -13,6 +13,7 @@ import (
func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) { func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) {
e := meshcatalog.CatalogEntry{Module: "netcheck", Repository: "novox/mesh-catalog", Path: "modules/netcheck", Commit: "abc", Fp: "src2:f", e := meshcatalog.CatalogEntry{Module: "netcheck", Repository: "novox/mesh-catalog", Path: "modules/netcheck", Commit: "abc", Fp: "src2:f",
ManifestDigest: "sha256:m", Bases: []string{"base"}, Packages: map[string]string{"p": "1"}, Needs: []string{"n"}, ManifestDigest: "sha256:m", Bases: []string{"base"}, Packages: map[string]string{"p": "1"}, Needs: []string{"n"},
Artifacts: []meshcatalog.Artifact{{Ref: "netcheck:lib", Kind: "package", Language: "go"}},
Inputs: &meshcatalog.EntryInputs{Toolchains: map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1", Recipe: "1"}}, Inputs: &meshcatalog.EntryInputs{Toolchains: map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1", Recipe: "1"}},
Paths: map[string][]string{"lib": {"a/b", "c"}}}} Paths: map[string][]string{"lib": {"a/b", "c"}}}}
got := ModuleOf(e) got := ModuleOf(e)
@@ -30,4 +31,11 @@ func TestAnEntryOfTheCatalogHalfIsTheFormTheBuilderBuildsBy(t *testing.T) {
if bare := ModuleOf(meshcatalog.CatalogEntry{Module: "m"}); bare.Inputs2 != nil || bare.Fp2 != "" { if bare := ModuleOf(meshcatalog.CatalogEntry{Module: "m"}); bare.Inputs2 != nil || bare.Fp2 != "" {
t.Errorf("ModuleOf of an entry with none = %+v, want nothing set", bare) t.Errorf("ModuleOf of an entry with none = %+v, want nothing set", bare)
} }
// A module with an artifact that names no paths (a Go program beside its packages) is built from its directory whole
// (ADR 0499 clauses 1 and 4): its fetch names no paths, so the program's go.mod and main package stay in the tree.
mixed := e
mixed.Artifacts = append([]meshcatalog.Artifact{{Ref: "netcheck:program", Kind: "bundle", Language: "go"}}, e.Artifacts...)
if got := ModuleOf(mixed); got.Inputs2 == nil || len(got.Inputs2.Paths) != 0 {
t.Errorf("ModuleOf of a module with a program naming no paths = %+v, want bases and no named paths", got.Inputs2)
}
} }
@@ -194,7 +194,7 @@ func (b *Builder) reserve(ctx context.Context, module string, m meshcatalog.Modu
for _, t := range plan.Tasks { for _, t := range plan.Tasks {
refs = append(refs, module+":"+t.Artifact) refs = append(refs, module+":"+t.Artifact)
} }
src := versions.Source{Repository: m.Repository, Commit: m.Commit, Fingerprint: m.Fp2, Branch: o.Branch, Revision: o.Source, Rebuild: rebuild} src := versions.Source{Repository: m.Repository, Commit: m.Commit, Fingerprint: m.Fp2, Branch: o.Branch, Revision: o.Source, CatalogRevision: o.Revision, Rebuild: rebuild}
for _, t := range plan.Tasks { for _, t := range plan.Tasks {
src.GoPackage = src.GoPackage || (t.Kind == catalogue.ArtifactPackage && t.Language == "go") src.GoPackage = src.GoPackage || (t.Kind == catalogue.ArtifactPackage && t.Language == "go")
} }
@@ -362,10 +362,10 @@ func (m reserveMap) Delete(_ context.Context, key string) error {
return nil return nil
} }
// TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber: design 47 G A-490b, at the build, through the real // TestABuildAgainKeepsItsOwnStandingNumber: design 47 G A-490b as H-A-225 amends it, at the build, through the real reserver —
// reserver — a done key whose own fingerprint has an unreleased 0.1.30 standing, the catalog's highest 0.1.16, is built again // a done key whose own fingerprint has an unreleased 0.1.30 standing, the catalog's highest 0.1.16, is built again under 0.1.30
// under 0.1.17 (never 0.1.31), the standing reservation taken over. // (never 0.1.31, never a new number): the run building carries the standing number until it is released.
func TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber(t *testing.T) { func TestABuildAgainKeepsItsOwnStandingNumber(t *testing.T) {
r := newRig(t, goBundle) r := newRig(t, goBundle)
r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: "0.1", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}} r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: "0.1", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
if v := r.consider(t, "modules/m"); v.State != builds.BuildDone { if v := r.consider(t, "modules/m"); v.State != builds.BuildDone {
@@ -381,8 +381,8 @@ func TestABuildAgainReservesAboveTheCatalogsHighestNotItsOwnStandingNumber(t *te
r.b.Reserved = real r.b.Reserved = real
got := r.b.BuildAgain(context.Background(), "m", fp1, "its publish was refused", "shanks.lead", 2*time.Second) got := r.b.BuildAgain(context.Background(), "m", fp1, "its publish was refused", "shanks.lead", 2*time.Second)
r.b.Wait() r.b.Wait()
cur, ok, err := real.Of(context.Background(), "m", "0.1.17") cur, ok, err := real.Of(context.Background(), "m", "")
if got.Refused != "" || err != nil || !ok || cur.Fingerprint != fp1 { if got.Refused != "" || err != nil || !ok || cur.Fingerprint != fp1 || cur.Version != "0.1.30" {
t.Fatalf("answer %+v; reservation of 0.1.17 = %+v (%v, %v); want built again under 0.1.17, the catalog's highest 0.1.16 plus one", got, cur, ok, err) t.Fatalf("answer %+v; reservation = %+v (%v, %v); want built again under its standing 0.1.30", got, cur, ok, err)
} }
} }
@@ -5,11 +5,12 @@ import (
"fmt" "fmt"
"maps" "maps"
"slices" "slices"
"strconv"
"strings"
"time" "time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
) )
// **A build waits for its revision; a build with no version is never made** (design 47 H H-A-155, H-A-156; H-A-141): a step // **A build waits for its revision; a build with no version is never made** (design 47 H H-A-155, H-A-156; H-A-141): a step
@@ -19,6 +20,17 @@ import (
// revision its view holds (PictureHeld, H-A-162), never a // revision its view holds (PictureHeld, H-A-162), never a
// timer or a poll, up to RevisionWait (the setting revision-wait-minutes); past it the ask fails, naming the revision and the // timer or a poll, up to RevisionWait (the setting revision-wait-minutes); past it the ask fails, naming the revision and the
// branch, and nothing is built. A step ask with no source (a run on request) does not wait. // branch, and nothing is built. A step ask with no source (a run on request) does not wait.
//
// **The picture holds every kept revision, not only the newest** (design 47 H H-A-205, H-A-147): an ask of a source revision
// below the newest the picture holds, whose entries the picture no longer names, asks the catalog seat's verb revision for
// its catalog revision by number, once, and is built when that kept revision is of its source and names every entry it
// asks; it never waits, since the picture only moves on. Only a revision above the newest is waited for.
//
// **An ask naming a revision is held by its number** (design 47 H H-A-222): when the picture does not name the ask's
// entries, the ask's catalog revision is read by number, and the ask is built when the catalog keeps it and it names every
// fingerprint asked, whatever the source the picture holds says (a re-pin keeps a later revision of the same source, H-A-211,
// H-A-213, so the source alone cannot say it). The source is compared only for an ask naming no revision; an ask whose
// revision the catalog does not answer and whose source is not below the picture's waits as before.
// DefaultRevisionWait is the bound of a step ask's wait for its revision, its setting revision-wait-minutes' default (H-A-156). // DefaultRevisionWait is the bound of a step ask's wait for its revision, its setting revision-wait-minutes' default (H-A-156).
const DefaultRevisionWait = 2 * time.Minute const DefaultRevisionWait = 2 * time.Minute
@@ -67,12 +79,68 @@ func (b *Builder) holdsRevision(branch string, source int64, entries []meshbuild
return true return true
} }
// keptRevisionHolds answers no outcome when the catalog keeps the ask's revision (its catalog revision by number) and that
// revision names every entry of the ask (H-A-205, H-A-222); else failed or could not run, with its words.
func (b *Builder) keptRevisionHolds(ctx context.Context, run meshbuilds.StepRun, held int64) (outcome, said string) {
below := fmt.Sprintf("revision %d of %s is not named by the builder's picture, which holds source revision %d", run.Source, run.Branch, held)
if run.Source < held {
below = fmt.Sprintf("revision %d of %s is below the %d the builder's picture holds", run.Source, run.Branch, held)
}
if b.KeptRevision == nil || run.Revision <= 0 {
return meshbuilds.StepEndedOutcomeFailed, below + ", and the ask names no catalog revision to read it by"
}
source, fps, err := b.KeptRevision(ctx, run.Branch, run.Revision)
if err != nil {
b.logf("%s: %s, and the catalog's revision %d cannot be read: %v", run.Pipeline, below, run.Revision, err)
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s, and the catalog's revision %d cannot be read: %v", below, run.Revision, err)
}
// An ask names the revision and the source of one revision (H-A-218, H-A-219): a kept revision of another source is
// not the ask's.
if source != run.Source {
b.logf("%s: %s, and the catalog's revision %d is of source revision %d, the ask names %d: the step fails, nothing is built", run.Pipeline, below, run.Revision, source, run.Source)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s, and the catalog's revision %d is of source revision %d, the ask names %d", below, run.Revision, source, run.Source)
}
for _, e := range run.Entries {
if fps[e.Module] != e.Fingerprint {
b.logf("%s: %s, and the catalog's revision %d names %s at %q, not %s: the step fails, nothing is built", run.Pipeline, below, run.Revision, e.Module, fps[e.Module], e.Fingerprint)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s, and the catalog's revision %d names %s at %q, not %s", below, run.Revision, e.Module, fps[e.Module], e.Fingerprint)
}
}
// Each asked fingerprint is recorded as the branch's, so its build reserves its version (H-A-155) as one the picture
// took in does; what the picture heard stays as it is, so the newer fingerprint's build is not superseded.
b.mu.Lock()
for _, e := range run.Entries {
key, err := builds.BuilderBuildKey(e.Module, e.Fingerprint)
if err != nil || b.origin[key].Branch != "" {
continue
}
first := b.FirstBranch == nil || b.FirstBranch(run.Branch, e.Module)
b.trunkFP[key] = true
b.origin[key] = Origin{Branch: run.Branch, Source: run.Source, Revision: run.Revision, First: first}
}
b.mu.Unlock()
b.logf("%s: %s; the catalog keeps revision %d with its entries: the build goes on", run.Pipeline, below, run.Revision)
return "", ""
}
// awaitRevision returns no outcome once the builder's picture of the ask's branch holds its source revision; past the bound the // awaitRevision returns no outcome once the builder's picture of the ask's branch holds its source revision; past the bound the
// step's outcome failed, at the builder's stop could not run, each with its words. // step's outcome failed, at the builder's stop could not run, each with its words.
func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (outcome, said string) { func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (outcome, said string) {
if b.Picture == nil || run.Source <= 0 || run.Branch == "" { if b.Picture == nil || run.Source <= 0 || run.Branch == "" {
return "", "" return "", ""
} }
if held, ok := b.Picture(run.Branch); !b.holdsRevision(run.Branch, run.Source, run.Entries) {
below := ok && run.Source < held
if (run.Revision > 0 && b.KeptRevision != nil) || below {
// By its number (H-A-222), whatever the source the picture holds; for an ask naming none, only below the
// picture (H-A-205). A kept revision of another source or naming other fingerprints is a mismatch the wait
// cannot mend: it fails at once; only a revision the catalog does not answer is waited for.
outcome, said := b.keptRevisionHolds(ctx, run, held)
if outcome == "" || below || outcome == meshbuilds.StepEndedOutcomeFailed {
return outcome, said
}
}
}
bound := b.revisionWait() bound := b.revisionWait()
notHeld := fmt.Sprintf("revision %d of %s not in the builder's picture", run.Source, run.Branch) notHeld := fmt.Sprintf("revision %d of %s not in the builder's picture", run.Source, run.Branch)
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8) wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
@@ -104,13 +172,16 @@ func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (ou
} }
} }
// **A package with no pin waits for its release** (design 47 H H-A-165, H-A-161): an entry keeps a mesh package none of // **A package with no pin waits for its publish** (design 47 H H-A-192, H-A-165, H-A-183, H-A-184, H-A-185): an entry
// whose versions is released with an empty version. Its step ask waits, as for its revision, for the release of the module // keeps a mesh package the registry has not published within the series its module accepts with an empty version. Its
// that holds the package (the catalog's version-released, VersionReleased), up to PackageWait (the setting // build, a step ask's or a lane's, first reads the versions the registry seat holds of the path (its verb versions,
// package-wait-minutes), then builds with that release as the package's pin (v<version>); past the bound the step fails, // H-A-194), then waits for the registry seat's published of that package path in that series (the
// naming the package, and nothing is built. // series the module's manifest names in `packages`, else the default; catalogue.Manifest.AcceptedSeries), up to
// PackageWait (the setting package-wait-minutes), and pins the version that event carries; the builder never derives a
// pin (no release number of the module that holds the package). Past the bound the build fails, naming the package, and
// nothing is built.
// DefaultPackageWait is the bound of a step ask's wait for a package's release, its setting package-wait-minutes' default. // DefaultPackageWait is the bound of a build's wait for a package's publish, its setting package-wait-minutes' default.
const DefaultPackageWait = 10 * time.Minute const DefaultPackageWait = 10 * time.Minute
func (b *Builder) packageWait() time.Duration { func (b *Builder) packageWait() time.Duration {
@@ -120,91 +191,152 @@ func (b *Builder) packageWait() time.Duration {
return DefaultPackageWait return DefaultPackageWait
} }
// VersionReleased takes the catalog's version-released of module in: a step ask waiting for a package of it wakes. // PackagePublished takes the Go package registry seat's published of one package path in (its name and the version the
func (b *Builder) VersionReleased(module, version string) { // event carries, H-A-192): the newest published in the version's series is kept, and a build waiting for the package
// wakes. A version that is no MAJOR.MINOR.PATCH (a group's tagged version) pins nothing.
func (b *Builder) PackagePublished(path, version string) {
v := strings.TrimPrefix(version, "v")
series, ok := seriesOfVersion(v)
if path == "" || !ok {
return
}
b.init() b.init()
b.mu.Lock() b.mu.Lock()
b.releases[module] = version in := b.publishedPins[path]
if in == nil {
in = map[string]string{}
b.publishedPins[path] = in
}
if old := in[series]; old == "" || compareVersions(v, old) > 0 {
in[series] = v
}
b.mu.Unlock() b.mu.Unlock()
b.pictureMoved() b.pictureMoved()
} }
// releaseOf is the pin a package is built against: the release of the module holding it, as Go names it; "" for none yet. // publishedPin is the pin the registry seat's published gave the package path in the series, as Go names it; "" for none.
func (b *Builder) releaseOf(ctx context.Context, pkg string) string { func (b *Builder) publishedPin(path, series string) string {
if b.Members == nil {
return ""
}
module, _, found, err := b.Members(ctx, pkg)
if err != nil || !found {
return ""
}
b.mu.Lock() b.mu.Lock()
v := b.releases[module] v := b.publishedPins[path][series]
b.mu.Unlock() b.mu.Unlock()
if v == "" && b.NewestReleased != nil {
if kept, ok, err := b.NewestReleased(ctx, module); err == nil && ok {
v = kept
}
}
if v == "" { if v == "" {
return "" return ""
} }
return "v" + v return "v" + v
} }
// awaitPins fills each package entry names with no pin from its release, waiting for one; no outcome once every package has // awaitPublished fills each package of packages with no pin from the published of its path within the series the manifest
// a pin, failed past the bound, could not run at the builder's stop. // with this digest accepts, waiting for one (H-A-192, H-A-165). It answers the packages with every pin, or an outcome and
func (b *Builder) awaitPins(ctx context.Context, entry *meshcatalog.CatalogEntry) (outcome, said string) { // its words: failed past the bound naming the package, could not run when the manifest cannot be read or the builder stops.
func (b *Builder) awaitPublished(ctx context.Context, module, fp, digest string, packages map[string]string) (map[string]string, string, string) {
var unpinned []string var unpinned []string
for p, v := range entry.Packages { for p, v := range packages {
if v == "" { if v == "" {
unpinned = append(unpinned, p) unpinned = append(unpinned, p)
} }
} }
if len(unpinned) == 0 { if len(unpinned) == 0 {
return "", "" return packages, "", ""
} }
slices.Sort(unpinned) slices.Sort(unpinned)
if b.Manifests == nil {
return nil, meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the series %s accepts of %s cannot be read: no manifests", module, unpinned[0])
}
man, err := b.Manifests.Manifest(ctx, digest)
if err != nil {
return nil, meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the series %s accepts of %s cannot be read: its manifest %s: %v", module, unpinned[0], digest, err)
}
// What the registry seat holds is read once before the wait (H-A-194): the publishes heard are a cache of it.
for _, p := range unpinned {
if b.PublishedVersions == nil || b.publishedPin(p, man.AcceptedSeries(p)) != "" {
continue
}
vs, err := b.PublishedVersions(ctx, p)
if err != nil {
b.logf("%s at %s: the versions the registry seat holds of %s cannot be read, so the build waits for its publish: %v", module, fp, p, err)
continue
}
for _, v := range vs {
b.PackagePublished(p, v)
}
}
bound := b.packageWait() bound := b.packageWait()
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8) wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
defer stop() defer stop()
pins := map[string]string{}
told := false told := false
for { for {
b.mu.Lock() b.mu.Lock()
moved := b.moved moved := b.moved
b.mu.Unlock() b.mu.Unlock()
missing := "" pins := map[string]string{}
missing, missingSeries := "", ""
for _, p := range unpinned { for _, p := range unpinned {
if pins[p] == "" { series := man.AcceptedSeries(p)
pins[p] = b.releaseOf(ctx, p) pins[p] = b.publishedPin(p, series)
}
if pins[p] == "" && missing == "" { if pins[p] == "" && missing == "" {
missing = p missing, missingSeries = p, series
} }
} }
if missing == "" { if missing == "" {
packages := maps.Clone(entry.Packages) out := maps.Clone(packages)
maps.Copy(packages, pins) maps.Copy(out, pins)
entry.Packages = packages
if told { if told {
b.logf("%s at %s: every package it names has a released version: built with %v", entry.Module, entry.Fp, pins) b.logf("%s at %s: every package it names is published: built with %v", module, fp, pins)
} }
return "", "" return out, "", ""
} }
notReleased := fmt.Sprintf("package %s has no released version", missing) notReleased := fmt.Sprintf("package %s has no released version in the series %s", missing, missingSeries)
if !told { if !told {
b.logf("%s at %s: %s yet: the ask waits for its release, at most %s", entry.Module, entry.Fp, notReleased, bound) b.logf("%s at %s: %s yet: the build waits for its publish, at most %s", module, fp, notReleased, bound)
told = true told = true
} }
select { select {
case <-moved: case <-moved:
case <-wait.Done(): case <-wait.Done():
if ctx.Err() != nil { if ctx.Err() != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, notReleased + ": the builder stopped before it was" return nil, meshbuilds.StepEndedOutcomeCouldNotRun, notReleased + ": the builder stopped before it was"
} }
b.logf("%s at %s: %s within %s: the step fails, nothing is built", entry.Module, entry.Fp, notReleased, bound) b.logf("%s at %s: %s within %s: the build fails, nothing is built", module, fp, notReleased, bound)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notReleased, bound) return nil, meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notReleased, bound)
} }
} }
} }
// seriesOfVersion is a MAJOR.MINOR.PATCH version's series, MAJOR.MINOR; false for any other form.
func seriesOfVersion(v string) (string, bool) {
if _, ok := versionParts(v); !ok {
return "", false
}
major, rest, _ := strings.Cut(v, ".")
minor, _, _ := strings.Cut(rest, ".")
return major + "." + minor, true
}
// versionParts are a MAJOR.MINOR.PATCH version's numbers; false for any other form.
func versionParts(v string) ([3]int, bool) {
var out [3]int
parts := strings.Split(v, ".")
if len(parts) != 3 {
return out, false
}
for i, p := range parts {
n, err := strconv.Atoi(p)
if err != nil || n < 0 {
return out, false
}
out[i] = n
}
return out, true
}
// compareVersions orders two MAJOR.MINOR.PATCH versions numerically.
func compareVersions(a, b string) int {
x, _ := versionParts(a)
y, _ := versionParts(b)
for i := range x {
if x[i] != y[i] {
return x[i] - y[i]
}
}
return 0
}
@@ -110,14 +110,24 @@ type Builder struct {
// Picture answers the source revision the builder's picture of a watched branch holds, false while it holds none (design 47 // Picture answers the source revision the builder's picture of a watched branch holds, false while it holds none (design 47
// H H-A-155, H-A-141): a step ask naming a newer one waits for it. Nil waits for nothing. // H H-A-155, H-A-141): a step ask naming a newer one waits for it. Nil waits for nothing.
Picture func(branch string) (source int64, held bool) Picture func(branch string) (source int64, held bool)
// KeptRevision asks the catalog seat's verb revision for one kept revision of a branch by its number (design 47 H
// H-A-205, H-A-147): the source revision it was taken in from and the fingerprint of each module it names. A step ask of
// a source revision below the newest the picture holds is built when that kept revision names its entries. Nil reads none.
KeptRevision func(ctx context.Context, branch string, revision int64) (source int64, fingerprints map[string]string, err error)
// FirstBranch says whether branch is the first watched branch of the module's registration, the one whose builds reserve
// a version (design 47 G §371): an ask admitted by its kept revision is recorded as that branch's (H-A-205, H-A-155). Nil
// takes every branch as the first, as an origin never heard is.
FirstBranch func(branch, module string) bool
// RevisionWait bounds a step ask's wait for its revision, its setting revision-wait-minutes (H-A-156); DefaultRevisionWait // RevisionWait bounds a step ask's wait for its revision, its setting revision-wait-minutes (H-A-156); DefaultRevisionWait
// when 0. // when 0.
RevisionWait time.Duration RevisionWait time.Duration
// PackageWait bounds a step ask's wait for the release of a package its entry names with no pin, its setting // PackageWait bounds a build's wait for the publish of a package its entry names with no pin, its setting
// package-wait-minutes (H-A-165); DefaultPackageWait when 0. NewestReleased reads the newest version the builder's // package-wait-minutes (H-A-165); DefaultPackageWait when 0. Only a registry seat's published heard (PackagePublished)
// record of releases keeps for a module; nil reads none, and only a release heard (VersionReleased) gives a pin. // gives a pin (H-A-192).
PackageWait time.Duration PackageWait time.Duration
NewestReleased func(ctx context.Context, module string) (version string, found bool, err error) // PublishedVersions asks the Go package registry seat's verb versions for one package path, its versions newest first
// (H-A-194): read once before a build waits for that package. Nil reads none.
PublishedVersions func(ctx context.Context, path string) ([]string, error)
// Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine // Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine
// answered when last asked, and counted, every node the assignments names, present or away, a node with no statement // answered when last asked, and counted, every node the assignments names, present or away, a node with no statement
// stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in // stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in
@@ -162,8 +172,9 @@ type Builder struct {
builds map[string]context.CancelCauseFunc builds map[string]context.CancelCauseFunc
// moved is closed and made again at each entry the picture takes in or drops: the step asks waiting for a revision wake. // moved is closed and made again at each entry the picture takes in or drops: the step asks waiting for a revision wake.
moved chan struct{} moved chan struct{}
// releases are the versions heard released, by module (VersionReleased): a package with no pin waits for one. // publishedPins are the newest version a registry seat said published, by package path and series (PackagePublished):
releases map[string]string // a package with no pin waits for one (H-A-192).
publishedPins map[string]map[string]string
} }
type planned struct { type planned struct {
@@ -194,7 +205,7 @@ func (b *Builder) init() {
b.hands, b.events = map[string]*handWatch{}, map[string]bool{} b.hands, b.events = map[string]*handWatch{}, map[string]bool{}
b.builds = map[string]context.CancelCauseFunc{} b.builds = map[string]context.CancelCauseFunc{}
b.moved = make(chan struct{}) b.moved = make(chan struct{})
b.releases = map[string]string{} b.publishedPins = map[string]map[string]string{}
}) })
} }
@@ -226,7 +237,9 @@ func (b *Builder) Module(ctx context.Context, key string, value []byte) {
type Origin struct { type Origin struct {
Branch string Branch string
Source int64 Source int64
First bool // Revision is the catalog revision number a step ask of the build named (design 47 H H-A-218), 0 until one did.
Revision int64
First bool
} }
// Entry takes the catalog graph entry of one module on one watched branch in (design 47 G §348, §614) and builds its // Entry takes the catalog graph entry of one module on one watched branch in (design 47 G §348, §614) and builds its
@@ -253,6 +266,23 @@ func (b *Builder) EntryGone(branch string, first bool, module string) {
} }
} }
// Withdraw ends the running build of module's fingerprint, superseded, with why (design 47 H H-A-225, ADR 0664 §4): its key
// fails "withdrawn" and its asks are withdrawn from the build seats. A fingerprint with no build running is left.
func (b *Builder) Withdraw(module, fingerprint, why string) {
key, err := builds.BuilderBuildKey(module, fingerprint)
if err != nil {
return
}
b.init()
b.mu.Lock()
withdraw := b.builds[key]
b.mu.Unlock()
if withdraw != nil {
b.logf("%s at %s: the build is withdrawn: %s", module, fingerprint, why)
withdraw(buildrun.Withdrawn{Why: why})
}
}
func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.ModulesByModuleValue, e []meshcatalog.Edge) { func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.ModulesByModuleValue, e []meshcatalog.Edge) {
b.mu.Lock() b.mu.Lock()
hk := o.Branch + "\x00" + key hk := o.Branch + "\x00" + key
@@ -285,16 +315,19 @@ func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.
// ModuleOf is a catalog entry in the form the builder builds by: its fingerprint, manifest digest, source (repository, // ModuleOf is a catalog entry in the form the builder builds by: its fingerprint, manifest digest, source (repository,
// directory, commit; no source stamp, design 47 H H-A-48), the toolchains its fingerprint was computed with, the bases it is // directory, commit; no source stamp, design 47 H H-A-48), the toolchains its fingerprint was computed with, the bases it is
// built against, its packages and needs, and the named paths of its artifacts. // built against, its packages and needs, and the named paths of its artifacts when every artifact names them.
func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue { func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
m := meshcatalog.ModulesByModuleValue{Repository: e.Repository, Path: e.Path, Commit: e.Commit, Fp2: e.Fp, None2: e.None, m := meshcatalog.ModulesByModuleValue{Repository: e.Repository, Path: e.Path, Commit: e.Commit, Fp2: e.Fp, None2: e.None,
Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs} Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest), Packages: e.Packages, Needs: e.Needs}
if len(e.Bases) > 0 { if len(e.Bases) > 0 {
m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{Bases: e.Bases} m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{Bases: e.Bases}
} }
if in := e.Inputs; in != nil { if in := e.Inputs; in != nil {
m.Toolchains, m.Problems = in.Toolchains, in.Problems m.Toolchains, m.Problems = in.Toolchains, in.Problems
if len(in.Paths) > 0 { // Only a module whose every artifact names its paths is built from those alone (ADR 0499 clauses 1, 2 and 4): an
// artifact without paths, such as a Go program, keeps its module's directory as its build source, so a module
// with one names no paths and its fetch keeps the directory whole.
if len(in.Paths) > 0 && everyArtifactNamesPaths(e) {
if m.Inputs2 == nil { if m.Inputs2 == nil {
m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{} m.Inputs2 = &meshcatalog.ModulesByModuleValueInputs2{}
} }
@@ -309,6 +342,20 @@ func ModuleOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
return m return m
} }
// everyArtifactNamesPaths is whether the entry names artifacts and the named paths of each, by its reference or its name.
func everyArtifactNamesPaths(e meshcatalog.CatalogEntry) bool {
if len(e.Artifacts) == 0 || e.Inputs == nil {
return false
}
for _, a := range e.Artifacts {
_, name, _ := strings.Cut(a.Ref, ":")
if len(e.Inputs.Paths[a.Ref]) == 0 && len(e.Inputs.Paths[name]) == 0 {
return false
}
}
return true
}
// originOf is the catalog revision the entry of a build key came from; a key no entry named is of the first branch, as a build // originOf is the catalog revision the entry of a build key came from; a key no entry named is of the first branch, as a build
// of the trunk's value was before branches were watched. // of the trunk's value was before branches were watched.
func (b *Builder) originOf(key string) Origin { func (b *Builder) originOf(key string) Origin {
@@ -677,8 +724,9 @@ func (b *Builder) try(ctx context.Context, module string, m meshcatalog.ModulesB
var reserved versions.ReservedVersion var reserved versions.ReservedVersion
hasReserved := false hasReserved := false
if rebuild && cl.Build.State == builds.BuildBuilding { if rebuild && cl.Build.State == builds.BuildBuilding {
// Reserved afresh, above the highest the catalog's versions answers, taking over the older unreleased reservation of this // Reserved as A-490b amended by H-A-225 has it: this fingerprint's standing reservation is kept while it is unreleased
// fingerprint (A-455), never its standing number kept (A-490b); the build's own publish then runs. // and above the highest the catalog's versions answers, else the next above that highest is taken (A-455); the build's
// own publish then runs.
if b.Reserved.Filled() { if b.Reserved.Filled() {
reserved, hasReserved = b.reserve(ctx, module, m, plan, b.originOf(key), true) reserved, hasReserved = b.reserve(ctx, module, m, plan, b.originOf(key), true)
} }
@@ -813,6 +861,7 @@ type lane struct {
tcVersion string // and the version it ran tcVersion string // and the version it ran
kept string // the digest of the artifact the build node kept instead of storing it (BuildAnswer.Kept), else empty kept string // the digest of the artifact the build node kept instead of storing it (BuildAnswer.Kept), else empty
outOfStep []string // the generated files its build module's compare found out of step, passed or failed outOfStep []string // the generated files its build module's compare found out of step, passed or failed
pins map[string]string // the mesh packages' versions its Go ask was built with (H-A-207)
err error err error
} }
@@ -962,7 +1011,7 @@ func (b *Builder) build(ctx context.Context, module string, m meshcatalog.Module
used = nil used = nil
} }
done := builds.BuilderBuild{State: builds.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used, done := builds.BuilderBuild{State: builds.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used,
Imports: importsOf(lanes), Needs: forms.Current.Needs(artifacts)} // the forms it is written in (ADR 0618 §1) Imports: importsOf(lanes), Needs: forms.Current.Needs(artifacts), Packages: pinsOf(lanes)} // the forms it is written in (ADR 0618 §1)
for _, l := range lanes { for _, l := range lanes {
if l.kept != "" { if l.kept != "" {
done.KeptOn = l.node // design 47 G §671: the build node that keeps the build done.KeptOn = l.node // design 47 G §671: the build node that keeps the build
@@ -1112,12 +1161,19 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m mes
ask := sdkstep.StepAsk{ID: sdkstep.NewID(), Kind: kind, Language: language, Asker: b.OwnModule, At: time.Now(), Version: l.version, Build: job} ask := sdkstep.StepAsk{ID: sdkstep.NewID(), Kind: kind, Language: language, Asker: b.OwnModule, At: time.Now(), Version: l.version, Build: job}
if language == sdkstep.AskGo && job.Fetch != nil && len(m.Packages) > 0 { if language == sdkstep.AskGo && job.Fetch != nil && len(m.Packages) > 0 {
// The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes // The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes
// them into the workspace, whatever the repository's go.mod names. // them into the workspace, whatever the repository's go.mod names. A package the entry names with no pin waits for
ask.Packages = maps.Clone(m.Packages) // its publish and is built with the version that publish carries; past the bound the lane fails naming it, and no
maps.DeleteFunc(ask.Packages, func(_, v string) bool { return v == "" }) // a package with no released version is no pin (H-A-161) // ask is put (H-A-192, H-A-165).
pinned, outcome, said := b.awaitPublished(ctx, module, m.Fp2, m.Manifest, m.Packages)
if outcome != "" {
l.err = errors.New(said)
return
}
ask.Packages = maps.Clone(pinned)
for _, fb := range req.Builds { for _, fb := range req.Builds {
delete(ask.Packages, fb.Package) // a package `builds` names is named in neither packages nor package (§465) delete(ask.Packages, fb.Package) // a package `builds` names is named in neither packages nor package (§465)
} }
l.pins = maps.Clone(ask.Packages) // kept with the done key, so its step's end reports them (H-A-207)
} }
if id := asked.resume(language); id != "" { if id := asked.resume(language); id != "" {
ask.ID = id // put before this builder started again: the stream holds it, or a node already ran it ask.ID = id // put before this builder started again: the stream holds it, or a node already ran it
@@ -1300,6 +1356,20 @@ func (b *Builder) takeOutOfStep(key string) []string {
// importsOf are the mesh packages the lanes' programs import, as their build modules answered them (novox/hq ADR 0593 // importsOf are the mesh packages the lanes' programs import, as their build modules answered them (novox/hq ADR 0593
// §2): sorted and each once; nil when none answered any. A lane kept from a prior build answered none, so a build that // §2): sorted and each once; nil when none answered any. A lane kept from a prior build answered none, so a build that
// kept one reports only what it built. // kept one reports only what it built.
// pinsOf is the mesh packages' versions the lanes were built with (H-A-207), nil when none was pinned.
func pinsOf(lanes []*lane) map[string]string {
var out map[string]string
for _, l := range lanes {
for p, v := range l.pins {
if out == nil {
out = map[string]string{}
}
out[p] = v
}
}
return out
}
func importsOf(lanes []*lane) []string { func importsOf(lanes []*lane) []string {
seen := map[string]bool{} seen := map[string]bool{}
for _, l := range lanes { for _, l := range lanes {
@@ -327,14 +327,47 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
} }
} }
// H-A-225 (ADR 0664 §4): a build withdrawn while it runs (its reservation taken over by another fingerprint) ends failed
// "withdrawn" with why, so it never publishes under the dropped number; a fingerprint with no build running is left.
func TestAWithdrawnBuildEndsWithdrawn(t *testing.T) {
r := newRig(t, goBundle)
asked, release := make(chan struct{}), make(chan struct{})
answer := r.answer
r.asks.Answer = func(a buildstep.StepAsk) buildstep.Step {
close(asked)
<-release
return answer(a)
}
r.b.Withdraw("m", fp1, "nothing runs yet") // left: no build of it runs
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20",
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains)})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
select {
case <-asked:
case <-time.After(5 * time.Second):
t.Fatal("no ask was put")
}
r.b.Withdraw("m", fp1, "its reservation 0.1.3 was taken over by the build of src2:b, which reserves 0.1.4")
close(release)
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != builds.BuildFailed || !strings.Contains(v.Error, "withdrawn: its reservation 0.1.3 was taken over") {
t.Fatalf("the key = %+v, want failed withdrawn naming the take-over", v)
}
}
// 1887/AC1, AC2: a trunk fingerprint with no key is one ask on the Go seat's delivery stream, named in the building key // 1887/AC1, AC2: a trunk fingerprint with no key is one ask on the Go seat's delivery stream, named in the building key
// before it is put, and the builder calls no node; the step a node wrote for it makes the key done naming its reference. // before it is put, and the builder calls no node; the step a node wrote for it makes the key done naming its reference.
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) { func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
r := newRig(t, goBundle) r := newRig(t, goBundle)
r.asks.Answer = nil // no node has pulled it yet r.asks.Answer = nil // no node has pulled it yet
// Issue 1585 (ADR 0573 §2): the ask carries the module's pins; a package the entry keeps with no version (none // Issue 1585 (ADR 0573 §2): the ask carries the module's pins; a package the entry keeps with no version (none
// released, design 47 H-A-161) is no pin. // published, design 47 H-A-161) is pinned at the version the registry seat's published carries (H-A-192), never
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": "v0.1.20"} // asked without it.
r.b.PackagePublished("git.novox.be/novox/mesh-contracts/go/seats/x", "v0.1.7")
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": "v0.1.7"}
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20",
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains), Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains),
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": ""}}) Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": ""}})
@@ -6,6 +6,7 @@ import (
"strings" "strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
) )
@@ -38,9 +39,21 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
if outcome, said := b.awaitRevision(ctx, run); outcome != "" { if outcome, said := b.awaitRevision(ctx, run); outcome != "" {
return finish(outcome, said) return finish(outcome, said)
} }
b.askedRevision(run)
var failed []string var failed []string
for _, e := range run.Entries { for _, e := range run.Entries {
outcome, said := b.buildEntry(ctx, e) var pins map[string]string
outcome, said := b.buildEntry(ctx, e, &pins)
// The end reports the pins each entry was built with, so the runner asks its held test steps with them (H-A-207).
for path, version := range pins {
if version == "" {
continue
}
if end.Packages == nil {
end.Packages = map[string]string{}
}
end.Packages[path] = version
}
switch outcome { switch outcome {
case meshbuilds.StepEndedOutcomeCouldNotRun: case meshbuilds.StepEndedOutcomeCouldNotRun:
return finish(outcome, said) return finish(outcome, said)
@@ -54,8 +67,30 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
return finish(meshbuilds.StepEndedOutcomePassed, "") return finish(meshbuilds.StepEndedOutcomePassed, "")
} }
// buildEntry builds one entry at its fingerprint and waits for its end. // askedRevision records on the origin of each entry of run the catalog revision the ask names (design 47 H H-A-218), where
func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outcome, said string) { // the origin is of the ask's branch and source revision: the version reserved for its build names it, and so does its
// package-published, so the runner finds the publishing pipeline by branch and revision.
func (b *Builder) askedRevision(run meshbuilds.StepRun) {
if run.Revision <= 0 {
return
}
b.mu.Lock()
defer b.mu.Unlock()
for _, e := range run.Entries {
key, err := builds.BuilderBuildKey(e.Module, e.Fingerprint)
if err != nil {
continue
}
if o, ok := b.origin[key]; ok && o.Branch == run.Branch && o.Source == run.Source {
o.Revision = run.Revision
b.origin[key] = o
}
}
}
// buildEntry builds one entry at its fingerprint and waits for its end; pins is set, when it is done, to the pins its
// done key keeps it was built with, the registry's included (H-A-207).
func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry, pins *map[string]string) (outcome, said string) {
entry, err := b.StepEntry(ctx, e.Fingerprint) entry, err := b.StepEntry(ctx, e.Fingerprint)
if err != nil { if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry %s at %s cannot be read: %v", e.Module, e.Fingerprint, err) return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry %s at %s cannot be read: %v", e.Module, e.Fingerprint, err)
@@ -63,10 +98,12 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
if entry.Module != e.Module || (e.Commit != "" && entry.Commit != e.Commit) { if entry.Module != e.Module || (e.Commit != "" && entry.Commit != e.Commit) {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit) return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit)
} }
// A package the entry names with no pin is built against its release, waited for (H-A-165). // A package the entry names with no pin is built against its publish, waited for (H-A-192, H-A-165).
if outcome, said := b.awaitPins(ctx, &entry); outcome != "" { pinned, outcome, said := b.awaitPublished(ctx, entry.Module, entry.Fp, meshcatalogclient.BareManifestDigest(entry.ManifestDigest), entry.Packages)
if outcome != "" {
return outcome, said return outcome, said
} }
entry.Packages = pinned
key, err := builds.BuilderBuildKey(entry.Module, entry.Fp) key, err := builds.BuilderBuildKey(entry.Module, entry.Fp)
if err != nil { if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, err.Error() return meshbuilds.StepEndedOutcomeCouldNotRun, err.Error()
@@ -75,16 +112,19 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
defer b.dropEnd(key, ended) defer b.dropEnd(key, ended)
if done, failedWords, found := b.keyEnd(ctx, key); found { if done, failedWords, found := b.keyEnd(ctx, key); found {
if done { if done {
// A key done before reports the pins it was built with, kept with it, never the ones a publish gives now.
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry) return b.passedWithItsVersion(ctx, entry)
} }
return meshbuilds.StepEndedOutcomeFailed, failedWords return meshbuilds.StepEndedOutcomeFailed, failedWords
} }
m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalog.BareManifestDigest(entry.ManifestDigest), Path: entry.Path, m := meshcatalog.ModulesByModuleValue{Repository: entry.Repository, Commit: entry.Commit, Fp2: entry.Fp, Manifest: meshcatalogclient.BareManifestDigest(entry.ManifestDigest), Path: entry.Path,
Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages} Toolchains: CatalogToolchains(b.agreedFor(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))), Packages: entry.Packages}
b.consider(ctx, entry.Module, m) b.consider(ctx, entry.Module, m)
select { select {
case log := <-ended: case log := <-ended:
if log.Outcome == builds.BuildDone { if log.Outcome == builds.BuildDone {
*pins = b.keptPins(ctx, key)
return b.passedWithItsVersion(ctx, entry) return b.passedWithItsVersion(ctx, entry)
} }
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error) return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s at %s: %s %s", log.Module, log.FP, log.Outcome, log.Error)
@@ -111,7 +151,7 @@ func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.Ca
if !reserving { if !reserving {
return meshbuilds.StepEndedOutcomePassed, "" return meshbuilds.StepEndedOutcomePassed, ""
} }
plan, planErr, _ := b.planOf(ctx, meshcatalog.BareManifestDigest(entry.ManifestDigest)) plan, planErr, _ := b.planOf(ctx, meshcatalogclient.BareManifestDigest(entry.ManifestDigest))
switch { switch {
case planErr != nil: case planErr != nil:
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the plan cannot be read to judge its version: %v", entry.Module, entry.Fp, planErr) return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("%s at %s: the plan cannot be read to judge its version: %v", entry.Module, entry.Fp, planErr)
@@ -133,6 +173,20 @@ func (b *Builder) passedWithItsVersion(ctx context.Context, entry meshcatalog.Ca
return meshbuilds.StepEndedOutcomePassed, "" return meshbuilds.StepEndedOutcomePassed, ""
} }
// keptPins are the pins a done key keeps it was built with (H-A-207); none when it keeps none or cannot be read, so the
// runner ends the steps that waited for them could not run, loud.
func (b *Builder) keptPins(ctx context.Context, key string) map[string]string {
raw, _, ok, err := b.Claimer.Store.Get(ctx, key)
if err != nil || !ok {
return nil
}
v, err := builds.DecodeBuilderBuild(raw)
if err != nil || v.State != builds.BuildDone {
return nil
}
return v.Packages
}
// keyEnd reads the build key: whether it is done, and its error when it failed for good; found only for a key that has ended. // keyEnd reads the build key: whether it is done, and its error when it failed for good; found only for a key that has ended.
func (b *Builder) keyEnd(ctx context.Context, key string) (done bool, failed string, found bool) { func (b *Builder) keyEnd(ctx context.Context, key string) (done bool, failed string, found bool) {
raw, _, ok, err := b.Claimer.Store.Get(ctx, key) raw, _, ok, err := b.Claimer.Store.Get(ctx, key)
@@ -3,9 +3,10 @@ package build
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"reflect" "reflect"
"slices"
"strings" "strings"
"sync"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time" "time"
@@ -118,79 +119,185 @@ func TestABuildStepPassesAReservingBuildOnlyWithItsReservedVersion(t *testing.T)
} }
t.Run("H-A-155: an ask waits for the picture to take its revision in, then builds with its version reserved", aStepAskWaitsForItsRevision) t.Run("H-A-155: an ask waits for the picture to take its revision in, then builds with its version reserved", aStepAskWaitsForItsRevision)
t.Run("H-A-155: past the bound the ask fails naming its revision, and nothing is built", aStepAskPastItsRevisionBoundFails) t.Run("H-A-155: past the bound the ask fails naming its revision, and nothing is built", aStepAskPastItsRevisionBoundFails)
t.Run("H-A-205: an ask of a revision below the picture's newest is read by number and built", anOlderKeptRevisionIsReadByNumber)
t.Run("H-A-162: a revision held that changes no entry wakes the ask", aRevisionHeldWakesTheAsk) t.Run("H-A-162: a revision held that changes no entry wakes the ask", aRevisionHeldWakesTheAsk)
t.Run("H-A-165: a package with no pin waits for its release and is built with it", aPackageWithNoPinWaitsForItsRelease) t.Run("H-A-165: a package with no pin waits for its release and is built with it", aPackageWithNoPinWaitsForItsRelease)
t.Run("H-A-165: past its bound a package with no release fails the step", aPackageWithNoReleaseFailsPastItsBound) t.Run("H-A-165: past its bound a package with no release fails the step", aPackageWithNoReleaseFailsPastItsBound)
t.Run("H-A-192: a lane waits for an unpinned package's publish and fails past its bound", aLaneWaitsForAnUnpinnedPackage)
t.Run("H-A-194: what the registry seat holds pins the package before any wait", theSeatsVersionsPinBeforeTheWait)
} }
// unpinnedRig is a rig whose step entry m names the package x with no version, built by the module contracts. // unpinnedRig is a rig whose step entry m names the package x with no version. Its manifest names no `packages`, so it
func unpinnedRig(t *testing.T) (*rig, *fakeReservations) { // accepts x in the default series 0.1. waiting is told when a build starts its wait for a package's publish.
func unpinnedRig(t *testing.T) (*rig, meshcatalog.CatalogEntry, <-chan struct{}) {
var held atomic.Int64 var held atomic.Int64
r, res, entry := revisionRig(t, &held, make(chan struct{}, 1)) r, _, entry := revisionRig(t, &held, make(chan struct{}, 1))
r.b.Picture = nil // no revision to wait for here r.b.Picture = nil // no revision to wait for here
entry.Packages = map[string]string{"git.novox.be/novox/mesh-contracts/go/seats/x": "", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"} entry.Packages = map[string]string{unpinnedX: "", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}
r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil } r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }
r.b.Members = func(_ context.Context, pkg string) (string, string, bool, error) { waiting := make(chan struct{}, 1)
return "contracts", "contracts:x", pkg == "git.novox.be/novox/mesh-contracts/go/seats/x", nil r.b.Logf = func(format string, args ...any) {
t.Logf(format, args...)
if strings.Contains(fmt.Sprintf(format, args...), "waits for its publish") {
select {
case waiting <- struct{}{}:
default:
}
}
} }
return r, res return r, entry, waiting
} }
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) { const unpinnedX = "git.novox.be/novox/mesh-contracts/go/seats/x"
r, _ := unpinnedRig(t)
var once sync.Once // awaitWaiting fails the test unless the build started its wait before ended said anything.
looked := make(chan struct{}) func awaitWaiting[T any](t *testing.T, waiting <-chan struct{}, ended <-chan T) {
r.b.NewestReleased = func(context.Context, string) (string, bool, error) { // the record holds none yet t.Helper()
once.Do(func() { close(looked) }) select {
return "", false, nil case <-waiting:
case end := <-ended:
t.Fatalf("the build ended before the package was published: %+v", end)
case <-time.After(5 * time.Second):
t.Fatal("the build never waited for the package's publish")
} }
}
// H-A-192: the pin is the version the registry seat's published carries for the path in the accepted series; a publish in
// another series and the module's release number pin nothing.
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
r, _, waiting := unpinnedRig(t)
r.b.PackagePublished(unpinnedX, "v0.2.9") // another series than the 0.1 the module accepts: no pin
r.b.PackagePublished(unpinnedX, "0.1.4-feat-x.2") // a group's tagged version: no pin
ended := make(chan meshbuilds.StepEnded, 1) ended := make(chan meshbuilds.StepEnded, 1)
go func() { go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41, ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}) Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}() }()
select { awaitWaiting(t, waiting, ended)
case <-looked:
case end := <-ended:
t.Fatalf("the ask ended before the package was released: %+v, asks %+v", end, r.asks.Asks())
case <-time.After(5 * time.Second):
t.Fatal("the ask never looked for the package's release")
}
if _, ok := r.st.vals[r.key]; ok { if _, ok := r.st.vals[r.key]; ok {
t.Fatal("the build was made before the package had a release") t.Fatal("the build was made before the package was published")
} }
r.b.VersionReleased("contracts", "0.1.3") r.b.PackagePublished(unpinnedX, "v0.1.3")
var end meshbuilds.StepEnded var end meshbuilds.StepEnded
select { select {
case end = <-ended: case end = <-ended:
case <-time.After(5 * time.Second): case <-time.After(5 * time.Second):
t.Fatal("the release did not wake the ask") t.Fatal("the publish did not wake the ask")
} }
r.b.Wait() r.b.Wait()
asks := r.asks.Asks() asks := r.asks.Asks()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 || if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 ||
!reflect.DeepEqual(asks[0].Packages, buildstep.PackageVersions{"git.novox.be/novox/mesh-contracts/go/seats/x": "v0.1.3", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}) { !reflect.DeepEqual(asks[0].Packages, buildstep.PackageVersions{unpinnedX: "v0.1.3", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}) {
t.Errorf("end = %+v, asks %+v; want passed, built with x at v0.1.3", end, asks) t.Errorf("end = %+v, asks %+v; want passed, built with x at v0.1.3", end, asks)
} }
// The end reports the pins the build was built with, the one the publish gave included (H-A-207).
if !reflect.DeepEqual(end.Packages, map[string]string{unpinnedX: "v0.1.3", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}) {
t.Errorf("the end reports the pins %v; want the ones it was built with", end.Packages)
}
// A done key reports the pins it keeps it was built with, never the newer one a publish gives now (H-A-207).
r.b.PackagePublished(unpinnedX, "v0.1.5")
again := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
if again.Outcome != meshbuilds.StepEndedOutcomePassed || again.Packages[unpinnedX] != "v0.1.3" {
t.Errorf("the done key's end = %+v; want passed reporting x at v0.1.3, as built", again)
}
// A done key that keeps no pins reports none, so the runner ends the waiting steps loud.
v, err := builds.DecodeBuilderBuild(r.st.vals[r.key])
if err != nil {
t.Fatal(err)
}
v.Packages = nil
raw, err := builds.EncodeBuilderBuild(v)
if err != nil {
t.Fatal(err)
}
r.st.vals[r.key] = raw
if bare := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}); len(bare.Packages) != 0 {
t.Errorf("a done key keeping no pins reported %v; want none", bare.Packages)
}
} }
func aPackageWithNoReleaseFailsPastItsBound(t *testing.T) { func aPackageWithNoReleaseFailsPastItsBound(t *testing.T) {
r, _ := unpinnedRig(t) r, _, _ := unpinnedRig(t)
r.b.PackageWait = time.Nanosecond r.b.PackageWait = time.Nanosecond
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41, end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}) Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait() r.b.Wait()
if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || built || if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || built ||
!strings.Contains(end.Said, "package git.novox.be/novox/mesh-contracts/go/seats/x has no released version") { !strings.Contains(end.Said, "package "+unpinnedX+" has no released version in the series 0.1") {
t.Errorf("end = %+v, built %v; want failed naming the package, nothing built", end, built) t.Errorf("end = %+v, built %v; want failed naming the package, nothing built", end, built)
} }
// A release the builder's own record keeps gives the pin at once. // A publish heard before gives the pin at once.
r.b.NewestReleased = func(_ context.Context, module string) (string, bool, error) { r.b.PackagePublished(unpinnedX, "v0.1.2")
return "0.1.2", module == "contracts", nil
}
if end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41, if end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}); strings.Contains(end.Said, "no released version") { Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}); strings.Contains(end.Said, "no released version") {
t.Errorf("a release the record keeps did not pin the package: %+v", end) t.Errorf("a publish heard did not pin the package: %+v", end)
}
}
// H-A-194: the build reads the registry seat's versions of the path once before it waits: a version in the accepted series
// pins the package with no event heard; one only in another series leaves the build waiting for a publish.
func theSeatsVersionsPinBeforeTheWait(t *testing.T) {
r, _, _ := unpinnedRig(t)
r.b.PackageWait = time.Nanosecond // no wait: the seat's answer alone gives the pin
var read []string
r.b.PublishedVersions = func(_ context.Context, path string) ([]string, error) {
read = append(read, path)
return []string{"0.2.1", "0.1.9", "0.1.8"}, nil // newest first
}
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
asks := r.asks.Asks()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 || asks[0].Packages[unpinnedX] != "v0.1.9" || !slices.Equal(read, []string{unpinnedX}) {
t.Errorf("end = %+v, asks %+v, read %v; want passed, built with x at v0.1.9 read once from the seat", end, asks, read)
}
r2, _, waiting := unpinnedRig(t)
r2.b.PublishedVersions = func(context.Context, string) ([]string, error) { return []string{"0.2.1"}, nil }
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r2.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
awaitWaiting(t, waiting, ended)
r2.b.PackagePublished(unpinnedX, "0.1.10")
if end := <-ended; end.Outcome != meshbuilds.StepEndedOutcomePassed {
t.Errorf("end = %+v, want passed once x was published in its series", end)
}
r2.b.Wait()
}
// H-A-192, H-A-165: the lane of an entry taken in waits the same way, never asking a build without the pin, and fails the
// key past the bound naming the package.
func aLaneWaitsForAnUnpinnedPackage(t *testing.T) {
r, entry, waiting := unpinnedRig(t)
done := make(chan struct{})
go func() {
r.b.Entry(t.Context(), Origin{Branch: "main", Source: 41, First: true}, entry)
r.b.Wait()
close(done)
}()
awaitWaiting(t, waiting, done)
if asks := r.asks.Asks(); len(asks) != 0 {
t.Fatalf("a build was asked without the package's pin: %+v", asks)
}
r.b.PackagePublished(unpinnedX, "0.1.5")
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("the publish did not wake the lane")
}
if asks := r.asks.Asks(); len(asks) == 0 || asks[0].Packages[unpinnedX] != "v0.1.5" {
t.Errorf("asks %+v; want one asked with x at v0.1.5", asks)
}
// Past its bound the key fails naming the package, and nothing is asked.
r2, entry2, _ := unpinnedRig(t)
r2.b.PackageWait = time.Nanosecond
r2.b.Entry(t.Context(), Origin{Branch: "main", Source: 41, First: true}, entry2)
r2.b.Wait()
if asks := r2.asks.Asks(); len(asks) != 0 || !strings.Contains(string(r2.st.vals[r2.key]), "package "+unpinnedX+" has no released version") {
t.Errorf("asks %+v, key %s; want none asked and the key failed naming the package", asks, r2.st.vals[r2.key])
} }
} }
@@ -249,6 +356,11 @@ func aStepAskWaitsForItsRevision(t *testing.T) {
held.Store(40) held.Store(40)
asked := make(chan struct{}, 1) asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked) r, res, entry := revisionRig(t, &held, asked)
var keptReads atomic.Int64 // H-A-205, H-A-222: a revision above the newest is read by number once; not answered, it is waited for
r.b.KeptRevision = func(context.Context, string, int64) (int64, map[string]string, error) {
keptReads.Add(1)
return 0, nil, errors.New("not asked")
}
ended := make(chan meshbuilds.StepEnded, 1) ended := make(chan meshbuilds.StepEnded, 1)
go func() { go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41, ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41,
@@ -273,8 +385,84 @@ func aStepAskWaitsForItsRevision(t *testing.T) {
t.Fatal("the ask was not woken by the entry the picture took in") t.Fatal("the ask was not woken by the entry the picture took in")
} }
r.b.Wait() r.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 { if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 || keptReads.Load() != 1 {
t.Errorf("end = %+v, reserved from %+v; want passed with one version reserved from revision 41", end, res.sources) t.Errorf("end = %+v, reserved from %+v, kept revisions read %d; want passed with one version reserved from revision 41, read by number once", end, res.sources, keptReads.Load())
}
}
// H-A-205: two pipelines of consecutive revisions run at once; the picture holds source revision 46, whose entry of m moved
// on, while the older pipeline asks the build of revision 45. Its kept catalog revision is read by number and the build runs;
// a kept revision naming another fingerprint fails the step, naming it.
func anOlderKeptRevisionIsReadByNumber(t *testing.T) {
var held atomic.Int64
held.Store(46)
r, res, _ := revisionRig(t, &held, make(chan struct{}, 1)) // a picture at 46 that never heard 45 (a restart)
r.b.RevisionWait = time.Nanosecond // a wait would fail at once: the older revision is never waited for
r.b.FirstBranch = func(branch, module string) bool { return branch == "main" && module == "m" }
var read []string
r.b.KeptRevision = func(_ context.Context, branch string, revision int64) (int64, map[string]string, error) {
read = append(read, fmt.Sprintf("%s %d", branch, revision))
return 45, map[string]string{"m": fp1}, nil
}
run := meshbuilds.StepRun{Pipeline: "main.47.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 47, Source: 45,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}
end := r.b.BuildStep(t.Context(), run)
r.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || !slices.Equal(read, []string{"main 47"}) {
t.Errorf("end = %+v, kept revisions read %v; want passed, revision 47 of main read once", end, read)
}
// Its fingerprint is recorded as the branch's: its build reserves its version from source revision 45 (H-A-155).
// The version reserved names the catalog revision the ask named, which its package-published says (H-A-218).
if len(res.sources) != 1 || res.sources[0].Revision != 45 || res.sources[0].CatalogRevision != 47 {
t.Errorf("reserved from %+v; want one version reserved from source revision 45 at catalog revision 47 (H-A-218)", res.sources)
}
r2, _, _ := revisionRig(t, &held, make(chan struct{}, 1))
r2.b.RevisionWait = time.Nanosecond
r2.b.KeptRevision = func(context.Context, string, int64) (int64, map[string]string, error) {
return 45, map[string]string{"m": "src2:other"}, nil
}
if end := r2.b.BuildStep(t.Context(), run); end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, "names m at \"src2:other\"") {
t.Errorf("end = %+v; want failed naming the fingerprint the kept revision names", end)
}
// H-A-222: an ask of the very source the picture holds, whose entries the picture no longer names (a re-pin kept revision 48
// of the same source 46 and the picture took its entries in), is held by its revision read by number, never waited for.
r3, res3, _ := revisionRig(t, &held, make(chan struct{}, 1))
r3.b.RevisionWait = time.Nanosecond
r3.b.FirstBranch = func(branch, module string) bool { return branch == "main" && module == "m" }
var read3 []string
r3.b.KeptRevision = func(_ context.Context, branch string, revision int64) (int64, map[string]string, error) {
read3 = append(read3, fmt.Sprintf("%s %d", branch, revision))
return 46, map[string]string{"m": fp1}, nil
}
same := meshbuilds.StepRun{Pipeline: "main.48.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 48, Source: 46,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}
end3 := r3.b.BuildStep(t.Context(), same)
r3.b.Wait()
if end3.Outcome != meshbuilds.StepEndedOutcomePassed || !slices.Equal(read3, []string{"main 48"}) || len(res3.sources) != 1 || res3.sources[0].CatalogRevision != 48 {
t.Errorf("end = %+v, kept revisions read %v, reserved %+v; want passed, revision 48 of main read once and reserved at it", end3, read3, res3.sources)
}
// H-A-218, H-A-222: a kept revision of another source than the ask names, or naming another fingerprint, is a mismatch:
// the step fails at once naming it, never waiting out its bound (the bound here is long, the ask is not below the picture).
for _, c := range []struct {
name string
source int64
fp string
said string
}{
{"another source", 45, fp1, "the catalog's revision 48 is of source revision 45, the ask names 46"},
{"another fingerprint", 46, "src2:other", "names m at \"src2:other\""},
} {
rc, resc, _ := revisionRig(t, &held, make(chan struct{}, 1))
rc.b.RevisionWait = time.Hour
rc.b.KeptRevision = func(context.Context, string, int64) (int64, map[string]string, error) {
return c.source, map[string]string{"m": c.fp}, nil
}
end := rc.b.BuildStep(t.Context(), same)
rc.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, c.said) || len(resc.sources) != 0 {
t.Errorf("%s: end = %+v, reserved %+v; want failed at once saying %q, nothing reserved", c.name, end, resc.sources, c.said)
}
} }
} }
@@ -5,6 +5,8 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"maps"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -65,12 +67,18 @@ type ReservedVersion struct {
// (design 47 G §371, A-57): what `package-published` says of the publication. // (design 47 G §371, A-57): what `package-published` says of the publication.
Branch string `json:"branch,omitempty"` Branch string `json:"branch,omitempty"`
Source int64 `json:"source,omitempty"` Source int64 `json:"source,omitempty"`
// Revision is the catalog revision number the build's step ask named, the indexer's count (design 47 H H-A-218): what
// `package-published` names beside the source, so the runner finds the publishing pipeline; 0 when no step ask named one.
Revision int64 `json:"revision,omitempty"`
// Dependencies are, by artifact reference, what the step's result says each artifact resolved (kind 1) and was built with // Dependencies are, by artifact reference, what the step's result says each artifact resolved (kind 1) and was built with
// (kind 3), kept for `package-published` (design 47 G §466, §673, A-57). // (kind 3), kept for `package-published` (design 47 G §466, §673, A-57).
Dependencies map[string][]meshbuilds.Dependency `json:"dependencies,omitempty"` Dependencies map[string][]meshbuilds.Dependency `json:"dependencies,omitempty"`
// Kept are the registry seats' published heard while the build had not yet recorded every digest and whose digest marked // Kept are the registry seats' published heard while the build had not yet recorded every digest and whose digest marked
// no artifact then: kept as heard, they mark their artifact when the build records its digests (design 47 G A-415). // no artifact then: kept as heard, they mark their artifact when the build records its digests (design 47 G A-415).
Kept []KeptPublished `json:"kept-published,omitempty"` Kept []KeptPublished `json:"kept-published,omitempty"`
// Withdrawn are the versions of the series a take-over by another fingerprint withdrew, each with the fingerprint whose
// build it was (design 47 H H-A-225): a late publish of one is refused as withdrawn. Kept while the series' key stands.
Withdrawn map[string]string `json:"withdrawn,omitempty"`
} }
// KeptPublished is a registry seat's published kept on a reservation, as heard (design 47 G A-415). // KeptPublished is a registry seat's published kept on a reservation, as heard (design 47 G A-415).
@@ -91,6 +99,8 @@ type Source struct {
// that catalog revision (design 47 G §371, A-57), kept on the reservation for the publication. // that catalog revision (design 47 G §371, A-57), kept on the reservation for the publication.
Branch string Branch string
Revision int64 Revision int64
// CatalogRevision is the catalog revision number the build's step ask named (H-A-218), 0 when none did.
CatalogRevision int64
// Rebuild marks a build again of a done key whose version is not released: the standing reservation of Fingerprint is replaced // Rebuild marks a build again of a done key whose version is not released: the standing reservation of Fingerprint is replaced
// and is no floor, so the number is the next above the catalog's highest alone (design 47 G A-490b). // and is no floor, so the number is the next above the catalog's highest alone (design 47 G A-490b).
Rebuild bool Rebuild bool
@@ -149,7 +159,10 @@ func SplitVersion(version string) (series string, patch int, err error) {
// Reserve reserves the next version of module's series for a build that starts, for artifacts (their names), and answers // Reserve reserves the next version of module's series for a build that starts, for artifacts (their names), and answers
// it. The next patch is the one after the higher of the highest released and the one reserved, 0 when there is neither. A // it. The next patch is the one after the higher of the highest released and the one reserved, 0 when there is neither. A
// conflicting write (another build reserved first) is read again and the next taken. // conflicting write (another build reserved first) is read again and the next taken. A reservation standing for the same
// fingerprint whose version is not released and is above the catalog's highest is kept and answered as it stands, a build again or a re-run after a restart
// alike (design 47 H H-A-225, which amends G A-490b): the run already building carries that number. A reservation another
// fingerprint takes over is withdrawn: its version is kept in Withdrawn, and Took is told so the older build is ended.
func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts []string, src Source) (ReservedVersion, error) { func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts []string, src Source) (ReservedVersion, error) {
if r.Filled != nil && !r.Filled() { if r.Filled != nil && !r.Filled() {
return ReservedVersion{}, ErrNotFilled return ReservedVersion{}, ErrNotFilled
@@ -170,6 +183,11 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
if err != nil { if err != nil {
return ReservedVersion{}, err return ReservedVersion{}, err
} }
if found && cur.Fingerprint == src.Fingerprint && cur.Fingerprint != "" && !slices.Contains(released, cur.Version) {
if _, p, err := SplitVersion(cur.Version); err == nil && p > high {
return cur, nil // kept until released (H-A-225); one at or below the catalog's highest can never be released
}
}
next := high next := high
if found && !(src.Rebuild && cur.Fingerprint == src.Fingerprint) { if found && !(src.Rebuild && cur.Fingerprint == src.Fingerprint) {
if _, p, err := SplitVersion(cur.Version); err == nil && p > next { if _, p, err := SplitVersion(cur.Version); err == nil && p > next {
@@ -177,13 +195,30 @@ func (r *Reserver) Reserve(ctx context.Context, module, series string, artifacts
} }
} }
out := ReservedVersion{V: 1, Module: module, Series: series, Version: fmt.Sprintf("%s.%d", series, next+1), Artifacts: map[string]string{}, At: r.now(), out := ReservedVersion{V: 1, Module: module, Series: series, Version: fmt.Sprintf("%s.%d", series, next+1), Artifacts: map[string]string{}, At: r.now(),
Repository: src.Repository, Commit: src.Commit, Fingerprint: src.Fingerprint, Branch: src.Branch, Source: src.Revision} Repository: src.Repository, Commit: src.Commit, Fingerprint: src.Fingerprint, Branch: src.Branch, Source: src.Revision, Revision: src.CatalogRevision}
if src.GoPackage { if src.GoPackage {
out.Tag = GoTagOf(out.Version) out.Tag = GoTagOf(out.Version)
} }
for _, a := range artifacts { for _, a := range artifacts {
out.Artifacts[a] = "" out.Artifacts[a] = ""
} }
if found {
out.Withdrawn = maps.Clone(cur.Withdrawn)
if cur.Fingerprint != src.Fingerprint {
if out.Withdrawn == nil {
out.Withdrawn = map[string]string{}
}
out.Withdrawn[cur.Version] = cur.Fingerprint // a dropped number stays a gap (H-A-226)
}
// One at or below the catalog's highest released is refused by the plain rule anyway: the map never grows without end.
maps.DeleteFunc(out.Withdrawn, func(v, _ string) bool {
_, p, err := SplitVersion(v)
return err != nil || p <= high
})
if len(out.Withdrawn) == 0 {
out.Withdrawn = nil
}
}
raw, _ := json.Marshal(out) raw, _ := json.Marshal(out)
if found { if found {
_, err = r.Store.Update(ctx, key, raw, rev) _, err = r.Store.Update(ctx, key, raw, rev)
@@ -295,23 +330,6 @@ func (r *Reserver) ReleasedFor(ctx context.Context, module, fingerprint string)
return false, nil return false, nil
} }
// NewestReleased is the newest version the builder's own record of releases keeps for module (design 47 H H-A-165): a
// package with no pin is built against it; false when the record holds none.
func (r *Reserver) NewestReleased(ctx context.Context, module string) (string, bool, error) {
if r.Releases == nil {
return "", false, nil
}
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(module))
if err != nil || !found {
return "", false, err
}
var rec ReleasedRecord
if json.Unmarshal(raw, &rec) != nil || len(rec.Kept) == 0 {
return "", false, nil
}
return rec.Kept[len(rec.Kept)-1].Version, true, nil
}
// keepRelease adds a release to the module's record, dropping the oldest past ReleasedKept. // keepRelease adds a release to the module's record, dropping the oldest past ReleasedKept.
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string) error { func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string) error {
n := r.ReleasedKept n := r.ReleasedKept
@@ -460,11 +478,12 @@ type Marked struct {
} }
// PackagePublished is the seat mesh-builds' package-published of one artifact the holder said published at the reservation's // PackagePublished is the seat mesh-builds' package-published of one artifact the holder said published at the reservation's
// version (design 47 G §52, §334, §673, A-57): the artifact reference, the version and the digest published, the watched branch and // version (design 47 G §52, §334, §673, A-57): the artifact reference, the version and the digest published, the watched branch,
// the source revision the build was made from, and the dependencies its step result named. // the source revision the build was made from and the catalog revision its step ask named (H-A-218), and the dependencies its
// step result named.
func (m Marked) PackagePublished(digest string, at time.Time) meshbuilds.PackagePublished { func (m Marked) PackagePublished(digest string, at time.Time) meshbuilds.PackagePublished {
at = at.UTC() at = at.UTC()
return meshbuilds.PackagePublished{V: 1, Artifact: m.Artifact, At: &at, Branch: m.Reserved.Branch, Source: m.Reserved.Source, Module: m.Reserved.Module, return meshbuilds.PackagePublished{V: 1, Artifact: m.Artifact, At: &at, Branch: m.Reserved.Branch, Source: m.Reserved.Source, Revision: m.Reserved.Revision, Module: m.Reserved.Module,
Version: m.Reserved.Version, Digest: digest, Dependencies: m.Reserved.Dependencies[m.Artifact]} Version: m.Reserved.Version, Digest: digest, Dependencies: m.Reserved.Dependencies[m.Artifact]}
} }
@@ -722,11 +741,24 @@ func AnswerReservedByName(ctx context.Context, r Reader, carriers func(context.C
} }
return out, nil return out, nil
} }
if series, _, err := SplitVersion(version); err == nil {
if cur, _, found, err := rs.get(ctx, ReservedKey(module, series)); err == nil && found && hasKey(cur.Withdrawn, version) {
// A late publish of a build a take-over withdrew (H-A-225): refused as what it is.
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) version %s was withdrawn: the build of %s it was reserved for was taken over by %s, which reserves %s",
name, module, version, cur.Withdrawn[version], cur.Fingerprint, cur.Version)}, nil
}
}
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no version %s reserved or released", name, module, version)}, nil return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no version %s reserved or released", name, module, version)}, nil
} }
return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no reserved version", name, module)}, nil return meshbuilds.ReservedAnswer{Error: fmt.Sprintf("%s (%s) has no reserved version", name, module)}, nil
} }
// hasKey says whether m holds k.
func hasKey(m map[string]string, k string) bool {
_, ok := m[k]
return ok
}
// NoEntryCarries opens the answer of reserved given a name no entry carries. // NoEntryCarries opens the answer of reserved given a name no entry carries.
const NoEntryCarries = "no entry carries the name " const NoEntryCarries = "no entry carries the name "
@@ -125,7 +125,7 @@ func TestReservationIsReadAndMarkedPublished(t *testing.T) {
ctx := context.Background() ctx := context.Background()
store := &reserveMem{mem: newMem()} store := &reserveMem{mem: newMem()}
r := &Reserver{Store: store, ReleasedOf: func(context.Context, string) ([]string, error) { return []string{"1.4.2"}, nil }} r := &Reserver{Store: store, ReleasedOf: func(context.Context, string) ([]string, error) { return []string{"1.4.2"}, nil }}
v, err := r.Reserve(ctx, "m", "1.4", []string{"m:a", "m:b"}, Source{Branch: "main", Revision: 41}) v, err := r.Reserve(ctx, "m", "1.4", []string{"m:a", "m:b"}, Source{Branch: "main", Revision: 41, CatalogRevision: 44})
if err != nil || v.Version != "1.4.3" { if err != nil || v.Version != "1.4.3" {
t.Fatalf("reserved %q (%v), want 1.4.3", v.Version, err) t.Fatalf("reserved %q (%v), want 1.4.3", v.Version, err)
} }
@@ -154,12 +154,12 @@ func TestReservationIsReadAndMarkedPublished(t *testing.T) {
if err != nil || len(marked) != 1 { if err != nil || len(marked) != 1 {
t.Fatalf("a published of 1.4.3 with a built digest marked %v (%v)", marked, err) t.Fatalf("a published of 1.4.3 with a built digest marked %v (%v)", marked, err)
} }
// Design 47 G §52, §673, A-57: the package-published of that artifact carries the digest published, the branch and the source // Design 47 G §52, §673, A-57: the package-published of that artifact carries the digest published, the branch, the source
// revision the reservation was made for, and the dependencies its step result named. // revision and the catalog revision (H-A-218) the reservation was made for, and the dependencies its step result named.
at := time.Unix(100, 0) at := time.Unix(100, 0)
said := marked[0].PackagePublished("sha256:aa", at) said := marked[0].PackagePublished("sha256:aa", at)
atUTC := at.UTC() atUTC := at.UTC()
if wantSaid := (meshbuilds.PackagePublished{V: 1, Artifact: "m:a", At: &atUTC, Branch: "main", Source: 41, Module: "m", Version: "1.4.3", Digest: "sha256:aa", Dependencies: deps}); !reflect.DeepEqual(said, wantSaid) { if wantSaid := (meshbuilds.PackagePublished{V: 1, Artifact: "m:a", At: &atUTC, Branch: "main", Source: 41, Revision: 44, Module: "m", Version: "1.4.3", Digest: "sha256:aa", Dependencies: deps}); !reflect.DeepEqual(said, wantSaid) {
t.Errorf("package-published = %+v, want %+v", said, wantSaid) t.Errorf("package-published = %+v, want %+v", said, wantSaid)
} }
if marked, _, _ := r.Published(ctx, "m-a", "1.4.3", "sha256:other"); len(marked) != 0 { if marked, _, _ := r.Published(ctx, "m-a", "1.4.3", "sha256:other"); len(marked) != 0 {
@@ -201,13 +201,11 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
t.Errorf("reservations left: %v", all) t.Errorf("reservations left: %v", all)
} }
// One key per module holds the last releases, the oldest dropped (A-456b). // One key per module holds the last releases, the oldest dropped (A-456b).
newest := ""
for _, fp := range []string{"src2:b1", "src2:b2"} { for _, fp := range []string{"src2:b1", "src2:b2"} {
n, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: fp}) n, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: fp})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
newest = n.Version
if _, err := r.Released(ctx, "m", n.Version); err != nil { if _, err := r.Released(ctx, "m", n.Version); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -221,27 +219,48 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
if ok, _ := r.ReleasedFor(ctx, "m", "src2:b2"); !ok { if ok, _ := r.ReleasedFor(ctx, "m", "src2:b2"); !ok {
t.Error("the newest release is not kept") t.Error("the newest release is not kept")
} }
// H-A-165: the newest version the record keeps is the pin of a package with none. }
if v, ok, err := r.NewestReleased(ctx, "m"); !ok || err != nil || v != newest {
t.Errorf("newest released of m = %q, %v (%v), want %s", v, ok, err, newest) // TestTheWithdrawnVersionsAreDroppedOnceReleasedPast: design 47 H H-A-225, H-A-226 — a dropped number stays a gap, kept in
// the series key's Withdrawn while it is above the catalog's highest released; at each write those at or below it go, so the
// map never grows without end.
func TestTheWithdrawnVersionsAreDroppedOnceReleasedPast(t *testing.T) {
ctx := context.Background()
store := &reserveMem{mem: newMem()}
highest := "0.1.3"
r := &Reserver{Store: store, Filled: func() bool { return true },
ReleasedOf: func(context.Context, string) ([]string, error) { return []string{highest}, nil }}
for _, fp := range []string{"src2:a", "src2:b", "src2:c"} {
if _, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: fp}); err != nil {
t.Fatal(err)
}
} }
if _, ok, _ := r.NewestReleased(ctx, "other"); ok { cur, _, _, _ := r.get(ctx, ReservedKey("m", "0.1"))
t.Error("a module never released has a newest release") if cur.Version != "0.1.6" || len(cur.Withdrawn) != 2 || cur.Withdrawn["0.1.4"] != "src2:a" || cur.Withdrawn["0.1.5"] != "src2:b" {
t.Fatalf("reservation %+v; want 0.1.6 with 0.1.4 and 0.1.5 withdrawn", cur)
}
highest = "0.1.5" // the catalog released past both withdrawn numbers but the newest
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "src2:d"})
if err != nil || got.Version != "0.1.7" || len(got.Withdrawn) != 1 || got.Withdrawn["0.1.6"] != "src2:c" {
t.Errorf("reserved %+v (%v); want 0.1.7 keeping only 0.1.6 withdrawn", got, err)
} }
} }
// TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas: design 47 G A-490b — a build again of a done key replaces // TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas: design 47 G A-490b as H-A-225 amends it — a build again
// its fingerprint's standing reservation (and tells Took) and numbers above the catalog's highest alone, whether the standing number // of a done key, or a re-run after a restart, keeps its fingerprint's standing reservation while it is unreleased and above the
// is below it or above it; another fingerprint's standing reservation stays a floor. // catalog's highest; one at or below it is replaced (Took told) by the next above the catalog's highest alone; another
// fingerprint's standing reservation stays a floor and is withdrawn.
func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *testing.T) { func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *testing.T) {
ctx := context.Background() ctx := context.Background()
for _, c := range []struct { for _, c := range []struct {
name, standing, standingFp string name, standing, standingFp string
want string want string
rebuild, kept bool
}{ }{
{name: "standing below the catalog's highest", standing: "0.1.0", standingFp: "fp", want: "0.1.17"}, {name: "standing below the catalog's highest", standing: "0.1.0", standingFp: "fp", want: "0.1.17", rebuild: true},
{name: "standing above the catalog's highest", standing: "0.1.30", standingFp: "fp", want: "0.1.17"}, {name: "H-A-225: standing above the catalog's highest is kept by a build again", standing: "0.1.30", standingFp: "fp", want: "0.1.30", rebuild: true, kept: true},
{name: "another fingerprint's standing number stays a floor", standing: "0.1.30", standingFp: "other", want: "0.1.31"}, {name: "H-A-225: a re-run after a restart keeps its standing number", standing: "0.1.111", standingFp: "fp", want: "0.1.111", kept: true},
{name: "another fingerprint's standing number stays a floor", standing: "0.1.30", standingFp: "other", want: "0.1.31", rebuild: true},
} { } {
t.Run(c.name, func(t *testing.T) { t.Run(c.name, func(t *testing.T) {
store := &reserveMem{mem: newMem()} store := &reserveMem{mem: newMem()}
@@ -255,10 +274,19 @@ func TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas(t *
r.Took = func(_ context.Context, older, newer ReservedVersion) { r.Took = func(_ context.Context, older, newer ReservedVersion) {
took = append(took, older.Version+">"+newer.Version) took = append(took, older.Version+">"+newer.Version)
} }
got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "fp", Rebuild: true}) got, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: "fp", Rebuild: c.rebuild})
if c.kept {
if err != nil || got.Version != c.want || len(took) != 0 || got.Artifacts["m:a"] != "d" {
t.Fatalf("reserved %+v (%v), took %v; want the standing %s kept as it stands", got, err, took, c.want)
}
return
}
if err != nil || got.Version != c.want || len(took) != 1 || took[0] != c.standing+">"+c.want { if err != nil || got.Version != c.want || len(took) != 1 || took[0] != c.standing+">"+c.want {
t.Fatalf("reserved %q (%v), took %v; want %s replacing %s", got.Version, err, took, c.want, c.standing) t.Fatalf("reserved %q (%v), took %v; want %s replacing %s", got.Version, err, took, c.want, c.standing)
} }
if w := got.Withdrawn[c.standing]; (c.standingFp != "fp") != (w == c.standingFp) {
t.Errorf("withdrawn %v; want %s withdrawn only when another fingerprint stood", got.Withdrawn, c.standing)
}
}) })
} }
} }
@@ -67,6 +67,20 @@ func TestReservedByNameAnswersFromTheViewAndTheCatalog(t *testing.T) {
t.Errorf("%s = %+v, want error naming %q alone", name, got, c.want) t.Errorf("%s = %+v, want error naming %q alone", name, got, c.want)
} }
} }
// H-A-225: another fingerprint taking the series over withdraws the older's version; a late publish of it is refused as
// withdrawn, naming the fingerprint that took it over, never as a version that was never reserved.
if _, err := r.Reserve(ctx, "mesh-sdk", "0.1", []string{"mesh-sdk:ts"}, Source{Fingerprint: "src2:a"}); err != nil {
t.Fatal(err)
}
if _, err := r.Reserve(ctx, "mesh-sdk", "0.1", []string{"mesh-sdk:ts"}, Source{Fingerprint: "src2:b"}); err != nil {
t.Fatal(err)
}
if got := ask("@novox/mesh-sdk", "0.1.5"); !strings.Contains(got.Error, "version 0.1.5 was withdrawn: the build of src2:a it was reserved for was taken over by src2:b, which reserves 0.1.6") {
t.Errorf("a late publish of a withdrawn version = %+v, want refused as withdrawn", got)
}
if got := ask("@novox/mesh-sdk", "9.9.9"); !strings.Contains(got.Error, "reserved or released") {
t.Errorf("a version never reserved = %+v, want refused as neither reserved nor released", got)
}
if _, err := AnswerReservedByName(ctx, rd, Carriers(packagesMem(nil)), released, "x", ""); err == nil { if _, err := AnswerReservedByName(ctx, rd, Carriers(packagesMem(nil)), released, "x", ""); err == nil {
t.Error("an unreadable view was answered as if no entry carried the name") t.Error("an unreadable view was answered as if no entry carried the name")
} }
+10 -4
View File
@@ -266,10 +266,16 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
if _, bus := p["bus"]; bus { if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential") t.Errorf("the process declares a bus: it runs on its own broker credential")
} }
// Issue 1966's bootstrap: the package artifact builds is out of the manifest until the new builder runs; the follow-up // H-A-186, design 47 G §370: a Go package another module imports is a declared package artifact of its module, so every
// puts it back with its assertion (ADR 0634 §1, ADR 0606 §4: three artifacts, the third the package builds). // release publishes it: builds (ADR 0634 §1, ADR 0606 §4), imported by the build modules and the node-engine, and
if len(m.Build.Artifacts) != 2 { // buildrun, imported by the indexer, the view and the watchman.
t.Fatalf("artifacts = %+v, want the program and the tools", m.Build.Artifacts) if len(m.Build.Artifacts) != 4 {
t.Fatalf("artifacts = %+v, want the program, the tools and the packages builds and buildrun", m.Build.Artifacts)
}
for i, name := range []string{"builds", "buildrun"} {
if a := m.Build.Artifacts[2+i]; a.Name != name || a.Kind != "package" || a.Language != "go" || !reflect.DeepEqual(a.Paths, []string{name}) || a.Series != "0.1" {
t.Errorf("package artifact = %+v, want the Go package %s of series 0.1", a, name)
}
} }
if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-builder-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-builder-tools"}) { if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-builder-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-builder-tools"}) {
t.Errorf("tools artifact = %+v", a) // 1945 t.Errorf("tools artifact = %+v", a) // 1945
+18
View File
@@ -334,6 +334,24 @@
"loads": [ "loads": [
"mesh-delivery-builder-tools" "mesh-delivery-builder-tools"
] ]
},
{
"name": "builds",
"kind": "package",
"language": "go",
"paths": [
"builds"
],
"series": "0.1"
},
{
"name": "buildrun",
"kind": "package",
"language": "go",
"paths": [
"buildrun"
],
"series": "0.1"
} }
] ]
} }
@@ -1,4 +1,7 @@
package meshcatalog package meshcatalogclient
// The two helpers below are hand-written, so they live in the seat's client, never under its contract's generated
// root (design 47 H, H-A-200; ADR 0608 §5).
import "strings" import "strings"
@@ -0,0 +1,22 @@
package meshcatalogclient
import "testing"
// An entry's manifest-digest carries sha256: and the module's value and the key of `manifests` the 64 digits bare, one
// digest in two forms (design 47 G §578); the two helpers live in the seat's client, outside its contract's generated
// root (H-A-200).
func TestAManifestDigestIsPrefixedForTheEntryAndBareForTheKey(t *testing.T) {
const bare = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
for _, tc := range []struct{ in, entry, key string }{
{bare, "sha256:" + bare, bare},
{"sha256:" + bare, "sha256:" + bare, bare},
{"", "", ""},
} {
if got := EntryManifestDigest(tc.in); got != tc.entry {
t.Errorf("EntryManifestDigest(%q) = %q, want %q", tc.in, got, tc.entry)
}
if got := BareManifestDigest(tc.in); got != tc.key {
t.Errorf("BareManifestDigest(%q) = %q, want %q", tc.in, got, tc.key)
}
}
}
@@ -17,8 +17,9 @@ import (
// eventPrefix starts every event subject of the seat. // eventPrefix starts every event subject of the seat.
const eventPrefix = "mesh.seat." + seat.Seat + ".event." const eventPrefix = "mesh.seat." + seat.Seat + ".event."
// perSource are the events said per source revision, on <event>.<branch>.<source> (design 47 H §746, H-A-72 (3)). // perRevision are the events said per pipeline, on <event>.<branch>.<revision>, its catalog revision's number as its key
var perSource = []string{seat.EventPipelineStarted, seat.EventPipelineEnded, seat.EventWaitHeld, seat.EventWaitEnded} // names it (design 47 H §746, H-A-72 (3), H-A-221).
var perRevision = []string{seat.EventPipelineStarted, seat.EventPipelineEnded, seat.EventWaitHeld, seat.EventWaitEnded}
// Subject is where the runner says a step's event or a run's start: mesh.seat.mesh-pipeline-runner.event.<event>. // Subject is where the runner says a step's event or a run's start: mesh.seat.mesh-pipeline-runner.event.<event>.
// <module>.<branch>.<commit>, each name escaped as a key's name is, so a branch holding a dot is one token (design 47 H // <module>.<branch>.<commit>, each name escaped as a key's name is, so a branch holding a dot is one token (design 47 H
@@ -27,41 +28,42 @@ func Subject(event, module, branch, commit string) string {
return eventPrefix + event + "." + keys.EscapeName(module) + "." + keys.EscapeName(branch) + "." + keys.EscapeName(commit) return eventPrefix + event + "." + keys.EscapeName(module) + "." + keys.EscapeName(branch) + "." + keys.EscapeName(commit)
} }
// SourceSubject is where the runner says a pipeline's start and end and its waits: mesh.seat.mesh-pipeline-runner. // RevisionSubject is where the runner says a pipeline's start and end and its waits: mesh.seat.mesh-pipeline-runner.
// event.<event>.<branch>.<source>, the branch escaped as a key's name is (design 47 H §746, H-A-72 (3)). // event.<event>.<branch>.<revision>, the pipeline's catalog revision number, the branch escaped as a key's name is (design
func SourceSubject(event, branch string, source int64) string { // 47 H §746, H-A-72 (3), H-A-221).
return eventPrefix + event + "." + keys.EscapeName(branch) + "." + strconv.FormatInt(source, 10) func RevisionSubject(event, branch string, revision int64) string {
return eventPrefix + event + "." + keys.EscapeName(branch) + "." + strconv.FormatInt(revision, 10)
} }
// Of is what an event's subject names: a step's event or a run's start names the module, the branch and the commit; an // Of is what an event's subject names: a step's event or a run's start names the module, the branch and the commit; an
// event per source revision names the branch and the source. // event per pipeline names the branch and the pipeline's catalog revision (H-A-221).
type Of struct { type Of struct {
Event, Module, Branch, Commit string Event, Module, Branch, Commit string
Source int64 Revision int64
} }
// Parse reads a subject Subject or SourceSubject built: an event of the seat's, and its names unescaped. // Parse reads a subject Subject or RevisionSubject built: an event of the seat's, and its names unescaped.
func Parse(subject string) (Of, error) { func Parse(subject string) (Of, error) {
rest, ok := strings.CutPrefix(subject, eventPrefix) rest, ok := strings.CutPrefix(subject, eventPrefix)
toks := strings.Split(rest, ".") toks := strings.Split(rest, ".")
bySource := len(toks) > 0 && slices.Contains(perSource, toks[0]) byRevision := len(toks) > 0 && slices.Contains(perRevision, toks[0])
want := 4 want := 4
if bySource { if byRevision {
want = 3 want = 3
} }
if !ok || len(toks) != want || !slices.Contains(seat.Events(), toks[0]) { if !ok || len(toks) != want || !slices.Contains(seat.Events(), toks[0]) {
return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q is no event subject of the seat %s", subject, seat.Seat) return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q is no event subject of the seat %s", subject, seat.Seat)
} }
if bySource { if byRevision {
branch, err := keys.UnescapeName(toks[1]) branch, err := keys.UnescapeName(toks[1])
if err != nil || branch == "" { if err != nil || branch == "" {
return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q: branch: %v", subject, err) return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q: branch: %v", subject, err)
} }
source, err := strconv.ParseInt(toks[2], 10, 64) revision, err := strconv.ParseInt(toks[2], 10, 64)
if err != nil || source < 1 { if err != nil || revision < 1 {
return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q: no source revision", subject) return Of{}, fmt.Errorf("meshpipelinerunnerclient: %q: no revision", subject)
} }
return Of{Event: toks[0], Branch: branch, Source: source}, nil return Of{Event: toks[0], Branch: branch, Revision: revision}, nil
} }
names := make([]string, 0, 3) names := make([]string, 0, 3)
for i, t := range toks[1:] { for i, t := range toks[1:] {
@@ -17,14 +17,14 @@ func TestSubjectsAndBodies(t *testing.T) {
if of, err := Parse(s); err != nil || of != (Of{Event: seat.EventStepEnded, Module: "nats", Branch: "release.1", Commit: "c1"}) { if of, err := Parse(s); err != nil || of != (Of{Event: seat.EventStepEnded, Module: "nats", Branch: "release.1", Commit: "c1"}) {
t.Errorf("parse %s: %+v, %v", s, of, err) t.Errorf("parse %s: %+v, %v", s, of, err)
} }
ps := SourceSubject(seat.EventPipelineEnded, "release.1", 3589) ps := RevisionSubject(seat.EventPipelineEnded, "release.1", 3589)
if ps != "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.release=2e1.3589" { if ps != "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.release=2e1.3589" {
t.Errorf("pipeline-ended subject %s", ps) t.Errorf("pipeline-ended subject %s", ps)
} }
if of, err := Parse(ps); err != nil || of != (Of{Event: seat.EventPipelineEnded, Branch: "release.1", Source: 3589}) { if of, err := Parse(ps); err != nil || of != (Of{Event: seat.EventPipelineEnded, Branch: "release.1", Revision: 3589}) {
t.Errorf("parse %s: %+v, %v", ps, of, err) t.Errorf("parse %s: %+v, %v", ps, of, err)
} }
if of, err := Parse(SourceSubject(seat.EventWaitHeld, "main", 7)); err != nil || of.Source != 7 || of.Module != "" { if of, err := Parse(RevisionSubject(seat.EventWaitHeld, "main", 7)); err != nil || of.Revision != 7 || of.Module != "" {
t.Errorf("wait-held: %+v, %v", of, err) t.Errorf("wait-held: %+v, %v", of, err)
} }
for _, bad := range []string{"not.an.event.subject", for _, bad := range []string{"not.an.event.subject",
@@ -95,6 +95,8 @@ type PackagePublished struct {
Digest string `json:"digest"` Digest string `json:"digest"`
// the module // the module
Module string `json:"module"` Module string `json:"module"`
// the catalog revision number of that branch the build was made from, the indexer's count (design 47 H-A-218, H-A-220); absent from a reservation made before the field
Revision int64 `json:"revision,omitempty"`
// the source revision of that branch the build was made from // the source revision of that branch the build was made from
Source int64 `json:"source"` Source int64 `json:"source"`
V int64 `json:"v"` V int64 `json:"v"`
@@ -114,9 +116,11 @@ type StepEnded struct {
// the node the builder ran it on, as in the subject // the node the builder ran it on, as in the subject
Node string `json:"node"` Node string `json:"node"`
// never passed for a build that built nothing // never passed for a build that built nothing
Outcome string `json:"outcome"` Outcome string `json:"outcome"`
Pipeline string `json:"pipeline"` // the pins the step's entries were built with, each mesh package's version by its module path, the ones the builder took from the registry for a package the entry named with none included; the runner asks the entry's held test steps with them (H-A-207)
Rerun bool `json:"rerun,omitempty"` Packages map[string]string `json:"packages,omitempty"`
Pipeline string `json:"pipeline"`
Rerun bool `json:"rerun,omitempty"`
// why it failed or could not run: the entry not built, the part not heard // why it failed or could not run: the entry not built, the part not heard
Said string `json:"said,omitempty"` Said string `json:"said,omitempty"`
Started *time.Time `json:"started,omitempty"` Started *time.Time `json:"started,omitempty"`
@@ -715,6 +715,10 @@ type CatalogRevision struct {
Base int64 `json:"base,omitempty"` Base int64 `json:"base,omitempty"`
// the watched branch the revision is of // the watched branch the revision is of
Branch string `json:"branch,omitempty"` Branch string `json:"branch,omitempty"`
// what made the revision (design 47 H-A-212, H-A-213): a take-in's the gate's call its source revision names, a re-pin's release of <module> <version>, a take-in again's take-in again of <repository>; absent for a candidate's
Call string `json:"call,omitempty"`
// the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217): a take-in's, a re-pin's on a module's release, a take-in again's, a recompute's that moved a fingerprint (its call naming the reason); absent for a candidate's
CallKind string `json:"call-kind,omitempty"`
// the seat definitions it was judged with, each by its digest, read whole through seat-definitions // the seat definitions it was judged with, each by its digest, read whole through seat-definitions
Definitions []CatalogRevisionDefinitionsItem `json:"definitions,omitempty"` Definitions []CatalogRevisionDefinitionsItem `json:"definitions,omitempty"`
// one per module, by name // one per module, by name
@@ -725,8 +729,12 @@ type CatalogRevision struct {
Heads string `json:"heads,omitempty"` Heads string `json:"heads,omitempty"`
// trunk or candidate // trunk or candidate
Kind string `json:"kind"` Kind string `json:"kind"`
// for call-kind release, the module released (H-A-215)
Module string `json:"module,omitempty"`
// the movers of the take-in that wrote it, so a redelivered source-moved whose revision is kept says take-in.taken again from them, recomputing nothing (design 47 H §743, H-A-92); absent for one with no mover // the movers of the take-in that wrote it, so a redelivered source-moved whose revision is kept says take-in.taken again from them, recomputing nothing (design 47 H §743, H-A-92); absent for one with no mover
Movers []TakeInEntry `json:"movers,omitempty"` Movers []TakeInEntry `json:"movers,omitempty"`
// for call-kind take-in-again, the repository taken in again (H-A-215)
Repository string `json:"repository,omitempty"`
// the catalog graph revision's number, counted by its holder // the catalog graph revision's number, counted by its holder
Revision int64 `json:"revision"` Revision int64 `json:"revision"`
// the source revision it was taken in from; for a candidate, the source revision its heads are merged onto // the source revision it was taken in from; for a candidate, the source revision its heads are merged onto
@@ -735,8 +743,18 @@ type CatalogRevision struct {
Trunk []Head `json:"trunk,omitempty"` Trunk []Head `json:"trunk,omitempty"`
// the layout version // the layout version
V int64 `json:"v,omitempty"` V int64 `json:"v,omitempty"`
// for call-kind release, the version released (H-A-215)
Version string `json:"version,omitempty"`
} }
// The values of CatalogRevision's call-kind.
const (
CatalogRevisionCallKindTakeIn = "take-in"
CatalogRevisionCallKindRelease = "release"
CatalogRevisionCallKindTakeInAgain = "take-in-again"
CatalogRevisionCallKindRecompute = "recompute"
)
// Edge is one edge of the catalog graph between artifact references, B before A (ADR 0666 section 14): kind 1 B published before A builds (a package A imports), 2 B published before A delivers (fetched by reference at its start), 3 B delivered before A builds (its build module and toolchain), 4 B delivered before A delivers (a provision, bus state, a format's reader), 5 B has acted for A's new version (an acting rule), 6 B runs a version that serves A (a verb or a manifest word A uses). // Edge is one edge of the catalog graph between artifact references, B before A (ADR 0666 section 14): kind 1 B published before A builds (a package A imports), 2 B published before A delivers (fetched by reference at its start), 3 B delivered before A builds (its build module and toolchain), 4 B delivered before A delivers (a provision, bus state, a format's reader), 5 B has acted for A's new version (an acting rule), 6 B runs a version that serves A (a verb or a manifest word A uses).
type Edge struct { type Edge struct {
// declared in the artifact entry's needs of A's manifest, where nothing else says it // declared in the artifact entry's needs of A's manifest, where nothing else says it
@@ -1301,6 +1319,10 @@ type CatalogRevisionEvent struct {
At *time.Time `json:"at,omitempty"` At *time.Time `json:"at,omitempty"`
// the watched branch the revision is of, escaped as a key name is // the watched branch the revision is of, escaped as a key name is
Branch string `json:"branch,omitempty"` Branch string `json:"branch,omitempty"`
// what made the revision, as the revision's call names it (design 47 H-A-212, H-A-213); absent for a candidate's
Call string `json:"call,omitempty"`
// the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217); absent for a candidate's
CallKind string `json:"call-kind,omitempty"`
// the modules whose entries differ from the revision before it of the same kind // the modules whose entries differ from the revision before it of the same kind
Changed []string `json:"changed,omitempty"` Changed []string `json:"changed,omitempty"`
// how many entries it lists // how many entries it lists
@@ -1308,12 +1330,26 @@ type CatalogRevisionEvent struct {
Group string `json:"group,omitempty"` Group string `json:"group,omitempty"`
Heads string `json:"heads,omitempty"` Heads string `json:"heads,omitempty"`
// trunk or candidate: a candidate's moves no running part // trunk or candidate: a candidate's moves no running part
Kind string `json:"kind"` Kind string `json:"kind"`
Revision int64 `json:"revision"` // for call-kind release, the module released (H-A-215)
Source int64 `json:"source"` Module string `json:"module,omitempty"`
V int64 `json:"v"` // for call-kind take-in-again, the repository taken in again (H-A-215)
Repository string `json:"repository,omitempty"`
Revision int64 `json:"revision"`
Source int64 `json:"source"`
V int64 `json:"v"`
// for call-kind release, the version released (H-A-215)
Version string `json:"version,omitempty"`
} }
// The values of CatalogRevisionEvent's call-kind.
const (
CatalogRevisionEventCallKindTakeIn = "take-in"
CatalogRevisionEventCallKindRelease = "release"
CatalogRevisionEventCallKindTakeInAgain = "take-in-again"
CatalogRevisionEventCallKindRecompute = "recompute"
)
// FillingEvent is the body of the event filling. // FillingEvent is the body of the event filling.
type FillingEvent = Fill type FillingEvent = Fill
@@ -17,7 +17,7 @@ func TestFixtures(t *testing.T) {
decode func([]byte) (any, error) decode func([]byte) (any, error)
subjectIs *regexp.Regexp subjectIs *regexp.Regexp
}{ }{
{name: "a trunk revision kept, on the seat's subject", subject: "mesh.seat.mesh-catalog.event.catalog-revision.main", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"changed\":[\"xclip\"],\"entries\":2,\"kind\":\"trunk\",\"revision\":12,\"source\":3589,\"v\":1}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"changed\":[\"xclip\"],\"entries\":2,\"kind\":\"trunk\",\"revision\":12,\"source\":3589,\"v\":1}", {name: "a trunk revision kept, on the seat's subject", subject: "mesh.seat.mesh-catalog.event.catalog-revision.main", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"call\":\"release of mesh-sdk 0.1.27\",\"call-kind\":\"release\",\"changed\":[\"xclip\"],\"entries\":2,\"kind\":\"trunk\",\"module\":\"mesh-sdk\",\"revision\":12,\"source\":3589,\"v\":1,\"version\":\"0.1.27\"}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"call\":\"release of mesh-sdk 0.1.27\",\"call-kind\":\"release\",\"changed\":[\"xclip\"],\"entries\":2,\"kind\":\"trunk\",\"module\":\"mesh-sdk\",\"revision\":12,\"source\":3589,\"v\":1,\"version\":\"0.1.27\"}",
decode: func(b []byte) (any, error) { var v CatalogRevisionEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-catalog\\.event\\.catalog-revision\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v CatalogRevisionEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-catalog\\.event\\.catalog-revision\\.[^.*> ]+$")},
{name: "a candidate's revision kept", subject: "mesh.seat.mesh-catalog.event.catalog-revision.main", given: "{\"branch\":\"main\",\"entries\":2,\"group\":\"feat/47-G\",\"heads\":\"0123456789ab\",\"kind\":\"candidate\",\"revision\":13,\"source\":3589,\"v\":1}", wire: "{\"branch\":\"main\",\"entries\":2,\"group\":\"feat/47-G\",\"heads\":\"0123456789ab\",\"kind\":\"candidate\",\"revision\":13,\"source\":3589,\"v\":1}", {name: "a candidate's revision kept", subject: "mesh.seat.mesh-catalog.event.catalog-revision.main", given: "{\"branch\":\"main\",\"entries\":2,\"group\":\"feat/47-G\",\"heads\":\"0123456789ab\",\"kind\":\"candidate\",\"revision\":13,\"source\":3589,\"v\":1}", wire: "{\"branch\":\"main\",\"entries\":2,\"group\":\"feat/47-G\",\"heads\":\"0123456789ab\",\"kind\":\"candidate\",\"revision\":13,\"source\":3589,\"v\":1}",
decode: func(b []byte) (any, error) { var v CatalogRevisionEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-catalog\\.event\\.catalog-revision\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v CatalogRevisionEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-catalog\\.event\\.catalog-revision\\.[^.*> ]+$")},
@@ -59,7 +59,7 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v ModulesAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil}, decode: func(b []byte) (any, error) { var v ModulesAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "the module verb (captured: xclip)", subject: "", given: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}", wire: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}", {name: "the module verb (captured: xclip)", subject: "", given: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}", wire: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}",
decode: func(b []byte) (any, error) { var v ModuleAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil}, decode: func(b []byte) (any, error) { var v ModuleAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "a watched branch's catalog graph revision", subject: "", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}", {name: "a watched branch's catalog graph revision", subject: "", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"call\":\"release of mesh-sdk 0.1.27\",\"call-kind\":\"release\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"module\":\"mesh-sdk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1,\"version\":\"0.1.27\"}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"call\":\"release of mesh-sdk 0.1.27\",\"call-kind\":\"release\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"module\":\"mesh-sdk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1,\"version\":\"0.1.27\"}",
decode: func(b []byte) (any, error) { decode: func(b []byte) (any, error) {
var v CatalogRevisionsByBranchRevisionValue var v CatalogRevisionsByBranchRevisionValue
err := json.Unmarshal(b, &v) err := json.Unmarshal(b, &v)
@@ -65,9 +65,9 @@ const (
// StreamPipelineStarted is the stream pipeline-started is kept in. // StreamPipelineStarted is the stream pipeline-started is kept in.
const StreamPipelineStarted = "SEAT_mesh-pipeline-runner_pipeline-started" const StreamPipelineStarted = "SEAT_mesh-pipeline-runner_pipeline-started"
// SubjectPipelineStarted is where pipeline-started is published, on mesh.seat.mesh-pipeline-runner.event.pipeline-started.<branch>.<source>. // SubjectPipelineStarted is where pipeline-started is published, on mesh.seat.mesh-pipeline-runner.event.pipeline-started.<branch>.<revision>.
func SubjectPipelineStarted(branch, source string) string { func SubjectPipelineStarted(branch, revision string) string {
return "mesh.seat.mesh-pipeline-runner.event.pipeline-started." + branch + "." + source return "mesh.seat.mesh-pipeline-runner.event.pipeline-started." + branch + "." + revision
} }
// SubjectPipelineStartedAll hears every pipeline-started. // SubjectPipelineStartedAll hears every pipeline-started.
@@ -76,9 +76,9 @@ const SubjectPipelineStartedAll = "mesh.seat.mesh-pipeline-runner.event.pipeline
// StreamPipelineEnded is the stream pipeline-ended is kept in. // StreamPipelineEnded is the stream pipeline-ended is kept in.
const StreamPipelineEnded = "SEAT_mesh-pipeline-runner_pipeline-ended" const StreamPipelineEnded = "SEAT_mesh-pipeline-runner_pipeline-ended"
// SubjectPipelineEnded is where pipeline-ended is published, on mesh.seat.mesh-pipeline-runner.event.pipeline-ended.<branch>.<source>. // SubjectPipelineEnded is where pipeline-ended is published, on mesh.seat.mesh-pipeline-runner.event.pipeline-ended.<branch>.<revision>.
func SubjectPipelineEnded(branch, source string) string { func SubjectPipelineEnded(branch, revision string) string {
return "mesh.seat.mesh-pipeline-runner.event.pipeline-ended." + branch + "." + source return "mesh.seat.mesh-pipeline-runner.event.pipeline-ended." + branch + "." + revision
} }
// SubjectPipelineEndedAll hears every pipeline-ended. // SubjectPipelineEndedAll hears every pipeline-ended.
@@ -120,9 +120,9 @@ const SubjectStepEndedAll = "mesh.seat.mesh-pipeline-runner.event.step-ended.*.*
// StreamWaitHeld is the stream wait-held is kept in. // StreamWaitHeld is the stream wait-held is kept in.
const StreamWaitHeld = "SEAT_mesh-pipeline-runner_wait-held" const StreamWaitHeld = "SEAT_mesh-pipeline-runner_wait-held"
// SubjectWaitHeld is where wait-held is published, on mesh.seat.mesh-pipeline-runner.event.wait-held.<branch>.<source>. // SubjectWaitHeld is where wait-held is published, on mesh.seat.mesh-pipeline-runner.event.wait-held.<branch>.<revision>.
func SubjectWaitHeld(branch, source string) string { func SubjectWaitHeld(branch, revision string) string {
return "mesh.seat.mesh-pipeline-runner.event.wait-held." + branch + "." + source return "mesh.seat.mesh-pipeline-runner.event.wait-held." + branch + "." + revision
} }
// SubjectWaitHeldAll hears every wait-held. // SubjectWaitHeldAll hears every wait-held.
@@ -131,9 +131,9 @@ const SubjectWaitHeldAll = "mesh.seat.mesh-pipeline-runner.event.wait-held.*.*"
// StreamWaitEnded is the stream wait-ended is kept in. // StreamWaitEnded is the stream wait-ended is kept in.
const StreamWaitEnded = "SEAT_mesh-pipeline-runner_wait-ended" const StreamWaitEnded = "SEAT_mesh-pipeline-runner_wait-ended"
// SubjectWaitEnded is where wait-ended is published, on mesh.seat.mesh-pipeline-runner.event.wait-ended.<branch>.<source>. // SubjectWaitEnded is where wait-ended is published, on mesh.seat.mesh-pipeline-runner.event.wait-ended.<branch>.<revision>.
func SubjectWaitEnded(branch, source string) string { func SubjectWaitEnded(branch, revision string) string {
return "mesh.seat.mesh-pipeline-runner.event.wait-ended." + branch + "." + source return "mesh.seat.mesh-pipeline-runner.event.wait-ended." + branch + "." + revision
} }
// SubjectWaitEndedAll hears every wait-ended. // SubjectWaitEndedAll hears every wait-ended.
@@ -156,11 +156,11 @@ const (
StateStepOutputs = "step-outputs" StateStepOutputs = "step-outputs"
) )
// PipelinesByBranchSourceKey is the key of one value of the state pipelines under the form "<branch>.<source>". // PipelinesByBranchRevisionKey is the key of one value of the state pipelines under the form "<branch>.<revision>".
func PipelinesByBranchSourceKey(branch, source string) string { return branch + "." + source } func PipelinesByBranchRevisionKey(branch, revision string) string { return branch + "." + revision }
// ParsePipelinesByBranchSourceKey reads the parts of a key of the form "<branch>.<source>", in the order of the form; the last result is false for a key of another form. // ParsePipelinesByBranchRevisionKey reads the parts of a key of the form "<branch>.<revision>", in the order of the form; the last result is false for a key of another form.
func ParsePipelinesByBranchSourceKey(key string) (string, string, bool) { func ParsePipelinesByBranchRevisionKey(key string) (string, string, bool) {
rest := key rest := key
v0, after0, found0 := strings.Cut(rest, ".") v0, after0, found0 := strings.Cut(rest, ".")
if !found0 || v0 == "" { if !found0 || v0 == "" {
@@ -178,9 +178,9 @@ func ParsePipelinesByBranchSourceKey(key string) (string, string, bool) {
return v0, v2, true return v0, v2, true
} }
// DecodePipelinesByBranchSourceValue reads one value of the state pipelines under the form "<branch>.<source>", refusing a field PipelinesByBranchSourceValue does not know and any text after the value. // DecodePipelinesByBranchRevisionValue reads one value of the state pipelines under the form "<branch>.<revision>", refusing a field PipelinesByBranchRevisionValue does not know and any text after the value.
func DecodePipelinesByBranchSourceValue(raw []byte) (PipelinesByBranchSourceValue, error) { func DecodePipelinesByBranchRevisionValue(raw []byte) (PipelinesByBranchRevisionValue, error) {
var v PipelinesByBranchSourceValue var v PipelinesByBranchRevisionValue
dec := json.NewDecoder(bytes.NewReader(raw)) dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields() dec.DisallowUnknownFields()
if err := dec.Decode(&v); err != nil { if err := dec.Decode(&v); err != nil {
@@ -276,6 +276,48 @@ func DecodePipelinesByBranchRunRunValue(raw []byte) (PipelinesByBranchRunRunValu
return v, nil return v, nil
} }
// PipelinesByStepsPipelineMoverKey is the key of one value of the state pipelines under the form "steps.<pipeline>.<mover>".
func PipelinesByStepsPipelineMoverKey(pipeline, mover string) string {
return "steps." + pipeline + "." + mover
}
// ParsePipelinesByStepsPipelineMoverKey reads the parts of a key of the form "steps.<pipeline>.<mover>", in the order of the form; the last result is false for a key of another form.
func ParsePipelinesByStepsPipelineMoverKey(key string) (string, string, bool) {
rest := key
if !strings.HasPrefix(rest, "steps.") {
return "", "", false
}
rest = rest[6:]
v1, after1, found1 := strings.Cut(rest, ".")
if !found1 || v1 == "" {
return "", "", false
}
rest = after1
if rest == "" {
return "", "", false
}
v3 := rest
rest = ""
if rest != "" {
return "", "", false
}
return v1, v3, true
}
// DecodePipelinesByStepsPipelineMoverValue reads one value of the state pipelines under the form "steps.<pipeline>.<mover>", refusing a field PipelinesByStepsPipelineMoverValue does not know and any text after the value.
func DecodePipelinesByStepsPipelineMoverValue(raw []byte) (PipelinesByStepsPipelineMoverValue, error) {
var v PipelinesByStepsPipelineMoverValue
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&v); err != nil {
return v, err
}
if dec.More() {
return v, errors.New("text after the value")
}
return v, nil
}
// QueueByQueueNextKey is the key of one value of the state queue under the form "queue.next". // QueueByQueueNextKey is the key of one value of the state queue under the form "queue.next".
func QueueByQueueNextKey() string { return "queue.next" } func QueueByQueueNextKey() string { return "queue.next" }
@@ -523,8 +565,10 @@ type LogEntry struct {
Artifact string `json:"artifact,omitempty"` Artifact string `json:"artifact,omitempty"`
At time.Time `json:"at"` At time.Time `json:"at"`
// the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) // the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470)
Kept []string `json:"kept,omitempty"` Kept []string `json:"kept,omitempty"`
Outcome string `json:"outcome,omitempty"` // the module of the mover whose step the entry is of (H-A-190)
Module string `json:"module,omitempty"`
Outcome string `json:"outcome,omitempty"`
// the output's name in step-outputs // the output's name in step-outputs
Output string `json:"output,omitempty"` Output string `json:"output,omitempty"`
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
@@ -621,10 +665,12 @@ type Pipeline struct {
// how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) // how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325)
EndCount int64 `json:"end-count,omitempty"` EndCount int64 `json:"end-count,omitempty"`
Ended *time.Time `json:"ended,omitempty"` Ended *time.Time `json:"ended,omitempty"`
// how many movers the pipeline has, set by list alone (H-A-191)
MoverCount int64 `json:"mover-count,omitempty"`
// the modules the pipeline moves, its take-in's entries (design 47 H §746) // the modules the pipeline moves, its take-in's entries (design 47 H §746)
Movers []Mover `json:"movers"` Movers []Mover `json:"movers"`
Outcome string `json:"outcome"` Outcome string `json:"outcome"`
// the pipeline's key, <branch>.<source>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83) // the pipeline's key, <branch>.<revision>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// for a pipeline a package's publish started, that publish (A-202) // for a pipeline a package's publish started, that publish (A-202)
Published *Published `json:"published,omitempty"` Published *Published `json:"published,omitempty"`
@@ -637,11 +683,13 @@ type Pipeline struct {
// the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) // the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9)
Source int64 `json:"source"` Source int64 `json:"source"`
Started *time.Time `json:"started,omitempty"` Started *time.Time `json:"started,omitempty"`
// every mover's steps, each naming its mover, approval last // how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191)
StepCount int64 `json:"step-count,omitempty"`
// every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187)
Steps []StepResult `json:"steps"` Steps []StepResult `json:"steps"`
// the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) // the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107)
Superseded []string `json:"superseded,omitempty"` Superseded []string `json:"superseded,omitempty"`
// the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606) // the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189)
Unlogged []LogEntry `json:"unlogged,omitempty"` Unlogged []LogEntry `json:"unlogged,omitempty"`
// the layout version // the layout version
V int64 `json:"v,omitempty"` V int64 `json:"v,omitempty"`
@@ -672,7 +720,7 @@ type PipelineEnded struct {
// per mover its fingerprint, released version, step results and tier, in the delivery order (design 47 H §746, H-A-2) // per mover its fingerprint, released version, step results and tier, in the delivery order (design 47 H §746, H-A-2)
Movers []MoverEnded `json:"movers"` Movers []MoverEnded `json:"movers"`
Outcome string `json:"outcome"` Outcome string `json:"outcome"`
// the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) // the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// the catalog graph revision of the branch the pipeline runs for // the catalog graph revision of the branch the pipeline runs for
Revision int64 `json:"revision,omitempty"` Revision int64 `json:"revision,omitempty"`
@@ -705,7 +753,7 @@ type PipelineStarted struct {
Call string `json:"call,omitempty"` Call string `json:"call,omitempty"`
// the pipeline's movers // the pipeline's movers
Movers []Mover `json:"movers"` Movers []Mover `json:"movers"`
// the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) // the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// for a pipeline a package's publish started, that publish (A-202) // for a pipeline a package's publish started, that publish (A-202)
Published *Published `json:"published,omitempty"` Published *Published `json:"published,omitempty"`
@@ -766,7 +814,7 @@ type StepEvent struct {
// the module's build-input fingerprint at the commit // the module's build-input fingerprint at the commit
Fingerprint string `json:"fingerprint,omitempty"` Fingerprint string `json:"fingerprint,omitempty"`
Module string `json:"module"` Module string `json:"module"`
// the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) // the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
Result StepResult `json:"result"` Result StepResult `json:"result"`
// the catalog graph revision of the branch the pipeline runs for // the catalog graph revision of the branch the pipeline runs for
@@ -918,7 +966,7 @@ type ToolchainUsed struct {
Version string `json:"version"` Version string `json:"version"`
} }
// Wait is a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<source> (design 47 H-A-72). // Wait is a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<revision> (design 47 H-A-72, H-A-221).
type Wait struct { type Wait struct {
Artifact string `json:"artifact,omitempty"` Artifact string `json:"artifact,omitempty"`
// the step's own ask, where it has one (design 47 G §731) // the step's own ask, where it has one (design 47 G §731)
@@ -935,7 +983,7 @@ type Wait struct {
Module string `json:"module,omitempty"` Module string `json:"module,omitempty"`
// what the step waits on (design 47 *Health and going back*) // what the step waits on (design 47 *Health and going back*)
On string `json:"on"` On string `json:"on"`
// the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) // the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// the catalog graph revision of the branch the pipeline runs for // the catalog graph revision of the branch the pipeline runs for
Revision int64 `json:"revision,omitempty"` Revision int64 `json:"revision,omitempty"`
@@ -976,7 +1024,7 @@ const (
// Waiting is a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72). // Waiting is a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72).
type Waiting struct { type Waiting struct {
// the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) // the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
} }
@@ -1026,6 +1074,8 @@ type StartAnswer struct {
// ShowInput is what show takes. // ShowInput is what show takes.
type ShowInput struct { type ShowInput struct {
// a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187)
Mover string `json:"mover,omitempty"`
// the pipeline's key // the pipeline's key
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
} }
@@ -1305,8 +1355,8 @@ type FullEvent = Fill
// RoomEvent is the body of the event room. // RoomEvent is the body of the event room.
type RoomEvent = Fill type RoomEvent = Fill
// PipelinesByBranchSourceValue is one value of the state pipelines under the key <branch>.<source>. // PipelinesByBranchRevisionValue is one value of the state pipelines under the key <branch>.<revision>.
type PipelinesByBranchSourceValue = Pipeline type PipelinesByBranchRevisionValue = Pipeline
// PipelinesByBranchSourcePublishedPackageVersionValue is one value of the state pipelines under the key <branch>.<source>.published.<package>.<version>. // PipelinesByBranchSourcePublishedPackageVersionValue is one value of the state pipelines under the key <branch>.<source>.published.<package>.<version>.
type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline
@@ -1314,6 +1364,14 @@ type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline
// PipelinesByBranchRunRunValue is one value of the state pipelines under the key <branch>.run.<run>. // PipelinesByBranchRunRunValue is one value of the state pipelines under the key <branch>.run.<run>.
type PipelinesByBranchRunRunValue = Pipeline type PipelinesByBranchRunRunValue = Pipeline
// PipelinesByStepsPipelineMoverValue is one value of the state pipelines under the key steps.<pipeline>.<mover>.
type PipelinesByStepsPipelineMoverValue struct {
// the mover's steps
Steps []StepResult `json:"steps"`
// the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189)
Unlogged []LogEntry `json:"unlogged,omitempty"`
}
// QueueByQueueNextValue is one value of the state queue under the key queue.next. // QueueByQueueNextValue is one value of the state queue under the key queue.next.
type QueueByQueueNextValue = int64 type QueueByQueueNextValue = int64
@@ -45,7 +45,7 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v StepEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.step-ended\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v StepEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.step-ended\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "a take-in's pipeline kept, its steps naming their mover", subject: "", given: "{\"branch\":\"main\",\"call\":\"0123456789abcdef\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"passed\",\"pipeline\":\"main.3589\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.3588\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"module\":\"nats\",\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"0123456789abcdef\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"passed\",\"pipeline\":\"main.3589\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.3588\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"module\":\"nats\",\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}", {name: "a take-in's pipeline kept, its steps naming their mover", subject: "", given: "{\"branch\":\"main\",\"call\":\"0123456789abcdef\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"passed\",\"pipeline\":\"main.3589\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.3588\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"module\":\"nats\",\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"0123456789abcdef\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"passed\",\"pipeline\":\"main.3589\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.3588\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"module\":\"nats\",\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"module\":\"nats\",\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}",
decode: func(b []byte) (any, error) { decode: func(b []byte) (any, error) {
var v PipelinesByBranchSourceValue var v PipelinesByBranchRevisionValue
err := json.Unmarshal(b, &v) err := json.Unmarshal(b, &v)
return v, err return v, err
}, subjectIs: nil}, }, subjectIs: nil},
@@ -73,13 +73,13 @@ func TestFixtures(t *testing.T) {
}, subjectIs: nil}, }, subjectIs: nil},
{name: "a pipeline waiting its turn, its seq given", subject: "", given: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"v\":1}", {name: "a pipeline waiting its turn, its seq given", subject: "", given: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"v\":1}",
decode: func(b []byte) (any, error) { decode: func(b []byte) (any, error) {
var v PipelinesByBranchSourceValue var v PipelinesByBranchRevisionValue
err := json.Unmarshal(b, &v) err := json.Unmarshal(b, &v)
return v, err return v, err
}, subjectIs: nil}, }, subjectIs: nil},
{name: "a waiting pipeline names the movers a newer take-in took out of it", subject: "", given: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"superseded\":[\"mesh-config\"],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"superseded\":[\"mesh-config\"],\"v\":1}", {name: "a waiting pipeline names the movers a newer take-in took out of it", subject: "", given: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"superseded\":[\"mesh-config\"],\"v\":1}", wire: "{\"branch\":\"main\",\"call\":\"fedcba9876543210\",\"movers\":[{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0124\",\"module\":\"nats\",\"repository\":\"mesh-networking\",\"tier\":1}],\"outcome\":\"waiting\",\"pipeline\":\"main.3590\",\"revision\":13,\"seq\":2,\"source\":3590,\"steps\":[],\"superseded\":[\"mesh-config\"],\"v\":1}",
decode: func(b []byte) (any, error) { decode: func(b []byte) (any, error) {
var v PipelinesByBranchSourceValue var v PipelinesByBranchRevisionValue
err := json.Unmarshal(b, &v) err := json.Unmarshal(b, &v)
return v, err return v, err
}, subjectIs: nil}, }, subjectIs: nil},
@@ -83,9 +83,9 @@ const SubjectWantedAll = "mesh.seat.mesh-wants.event.wanted.*.*"
// StreamPipelineWanted is the stream pipeline-wanted is kept in. // StreamPipelineWanted is the stream pipeline-wanted is kept in.
const StreamPipelineWanted = "SEAT_mesh-wants_pipeline-wanted" const StreamPipelineWanted = "SEAT_mesh-wants_pipeline-wanted"
// SubjectPipelineWanted is where pipeline-wanted is published, on mesh.seat.mesh-wants.event.pipeline-wanted.<branch>.<source>. // SubjectPipelineWanted is where pipeline-wanted is published, on mesh.seat.mesh-wants.event.pipeline-wanted.<branch>.<revision>.
func SubjectPipelineWanted(branch, source string) string { func SubjectPipelineWanted(branch, revision string) string {
return "mesh.seat.mesh-wants.event.pipeline-wanted." + branch + "." + source return "mesh.seat.mesh-wants.event.pipeline-wanted." + branch + "." + revision
} }
// SubjectPipelineWantedAll hears every pipeline-wanted. // SubjectPipelineWantedAll hears every pipeline-wanted.
@@ -1001,10 +1001,12 @@ type PipelineWanted struct {
// for tier-late, the late tier's movers not running and the machines they do not run on // for tier-late, the late tier's movers not running and the machines they do not run on
NotRunning []NotRunning `json:"not-running,omitempty"` NotRunning []NotRunning `json:"not-running,omitempty"`
Outcome string `json:"outcome"` Outcome string `json:"outcome"`
// the runner's pipeline key, <branch>.<source>, a branch escaped as a key's name is (design 47 H §746) // the runner's pipeline key, <branch>.<revision>, a branch escaped as a key's name is (design 47 H §746, H-A-213 (4))
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
Source int64 `json:"source"` // the pipeline's catalog revision number, the number of its key; the indexer frees the pins of that revision, not of another revision of the same source (design 47 H-A-214); absent from an older event
V int64 `json:"v"` Revision int64 `json:"revision,omitempty"`
Source int64 `json:"source"`
V int64 `json:"v"`
} }
// The values of PipelineWanted's outcome. // The values of PipelineWanted's outcome.
@@ -29,7 +29,7 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v WantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wanted\\.[^.*> ]+\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v WantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wanted\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "a want removed, its node left or its module unplaced", subject: "mesh.seat.mesh-wants.event.wanted.n1.xclip", given: "{\"at\":\"2026-11-09T19:05:00Z\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"module\":\"xclip\",\"node\":\"n1\",\"removed\":true,\"revision\":41,\"v\":1}", wire: "{\"at\":\"2026-11-09T19:05:00Z\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"module\":\"xclip\",\"node\":\"n1\",\"removed\":true,\"revision\":41,\"v\":1}", {name: "a want removed, its node left or its module unplaced", subject: "mesh.seat.mesh-wants.event.wanted.n1.xclip", given: "{\"at\":\"2026-11-09T19:05:00Z\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"module\":\"xclip\",\"node\":\"n1\",\"removed\":true,\"revision\":41,\"v\":1}", wire: "{\"at\":\"2026-11-09T19:05:00Z\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"module\":\"xclip\",\"node\":\"n1\",\"removed\":true,\"revision\":41,\"v\":1}",
decode: func(b []byte) (any, error) { var v WantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wanted\\.[^.*> ]+\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v WantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wanted\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "a pipeline's wants all run", subject: "mesh.seat.mesh-wants.event.pipeline-wanted.main.3589", given: "{\"branch\":\"main\",\"outcome\":\"running\",\"pipeline\":\"main.3589\",\"source\":3589,\"v\":1}", wire: "{\"branch\":\"main\",\"outcome\":\"running\",\"pipeline\":\"main.3589\",\"source\":3589,\"v\":1}", {name: "a pipeline's wants all run", subject: "mesh.seat.mesh-wants.event.pipeline-wanted.main.3590", given: "{\"branch\":\"main\",\"outcome\":\"running\",\"pipeline\":\"main.3590\",\"revision\":3590,\"source\":3589,\"v\":1}", wire: "{\"branch\":\"main\",\"outcome\":\"running\",\"pipeline\":\"main.3590\",\"revision\":3590,\"source\":3589,\"v\":1}",
decode: func(b []byte) (any, error) { var v PipelineWantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.pipeline-wanted\\.[^.*> ]+\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v PipelineWantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.pipeline-wanted\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "a pipeline's tier late, naming the movers and machines not running", subject: "mesh.seat.mesh-wants.event.pipeline-wanted.main.3590", given: "{\"branch\":\"main\",\"not-running\":[{\"machines\":[\"anchor\",\"laptop\"],\"module\":\"nats\"}],\"outcome\":\"tier-late\",\"pipeline\":\"main.3590\",\"source\":3590,\"v\":1}", wire: "{\"branch\":\"main\",\"not-running\":[{\"machines\":[\"anchor\",\"laptop\"],\"module\":\"nats\"}],\"outcome\":\"tier-late\",\"pipeline\":\"main.3590\",\"source\":3590,\"v\":1}", {name: "a pipeline's tier late, naming the movers and machines not running", subject: "mesh.seat.mesh-wants.event.pipeline-wanted.main.3590", given: "{\"branch\":\"main\",\"not-running\":[{\"machines\":[\"anchor\",\"laptop\"],\"module\":\"nats\"}],\"outcome\":\"tier-late\",\"pipeline\":\"main.3590\",\"source\":3590,\"v\":1}", wire: "{\"branch\":\"main\",\"not-running\":[{\"machines\":[\"anchor\",\"laptop\"],\"module\":\"nats\"}],\"outcome\":\"tier-late\",\"pipeline\":\"main.3590\",\"source\":3590,\"v\":1}",
decode: func(b []byte) (any, error) { var v PipelineWantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.pipeline-wanted\\.[^.*> ]+\\.[^.*> ]+$")}, decode: func(b []byte) (any, error) { var v PipelineWantedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.pipeline-wanted\\.[^.*> ]+\\.[^.*> ]+$")},
@@ -905,8 +905,10 @@ type StepRun struct {
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// the step is asked again by the runner's rerun // the step is asked again by the runner's rerun
Rerun bool `json:"rerun,omitempty"` Rerun bool `json:"rerun,omitempty"`
// the source revision of the watched branch the pipeline runs for; absent for a run on request // the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request
Revision int64 `json:"revision,omitempty"` Revision int64 `json:"revision,omitempty"`
// the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request
Source int64 `json:"source,omitempty"`
// the pipeline step // the pipeline step
Step string `json:"step"` Step string `json:"step"`
Toolchain *ToolchainUsed `json:"toolchain,omitempty"` Toolchain *ToolchainUsed `json:"toolchain,omitempty"`
@@ -903,8 +903,10 @@ type StepRun struct {
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// the step is asked again by the runner's rerun // the step is asked again by the runner's rerun
Rerun bool `json:"rerun,omitempty"` Rerun bool `json:"rerun,omitempty"`
// the source revision of the watched branch the pipeline runs for; absent for a run on request // the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request
Revision int64 `json:"revision,omitempty"` Revision int64 `json:"revision,omitempty"`
// the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request
Source int64 `json:"source,omitempty"`
// the pipeline step // the pipeline step
Step string `json:"step"` Step string `json:"step"`
Toolchain *ToolchainUsed `json:"toolchain,omitempty"` Toolchain *ToolchainUsed `json:"toolchain,omitempty"`
@@ -903,8 +903,10 @@ type StepRun struct {
Pipeline string `json:"pipeline"` Pipeline string `json:"pipeline"`
// the step is asked again by the runner's rerun // the step is asked again by the runner's rerun
Rerun bool `json:"rerun,omitempty"` Rerun bool `json:"rerun,omitempty"`
// the source revision of the watched branch the pipeline runs for; absent for a run on request // the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request
Revision int64 `json:"revision,omitempty"` Revision int64 `json:"revision,omitempty"`
// the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request
Source int64 `json:"source,omitempty"`
// the pipeline step // the pipeline step
Step string `json:"step"` Step string `json:"step"`
Toolchain *ToolchainUsed `json:"toolchain,omitempty"` Toolchain *ToolchainUsed `json:"toolchain,omitempty"`
@@ -1521,7 +1521,11 @@
}, },
"revision": { "revision": {
"type": "integer", "type": "integer",
"description": "the source revision of the watched branch the pipeline runs for; absent for a run on request" "description": "the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request"
},
"source": {
"type": "integer",
"description": "the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request"
}, },
"fingerprint": { "fingerprint": {
"type": "string", "type": "string",
@@ -180,6 +180,10 @@
"type": "integer", "type": "integer",
"description": "the source revision of that branch the build was made from" "description": "the source revision of that branch the build was made from"
}, },
"revision": {
"type": "integer",
"description": "the catalog revision number of that branch the build was made from, the indexer's count (design 47 H-A-218, H-A-220); absent from a reservation made before the field"
},
"dependencies": { "dependencies": {
"type": "array", "type": "array",
"items": { "items": {
@@ -363,6 +367,13 @@
"$ref": "#/defs/BuildEntry" "$ref": "#/defs/BuildEntry"
}, },
"description": "each entry the step built, by module and fingerprint" "description": "each entry the step built, by module and fingerprint"
},
"packages": {
"type": "object",
"additionalProperties": {
"type": "string"
},
"description": "the pins the step's entries were built with, each mesh package's version by its module path, the ones the builder took from the registry for a package the entry named with none included; the runner asks the entry's held test steps with them (H-A-207)"
} }
}, },
"required": [ "required": [
@@ -8,6 +8,10 @@
"revision": 12, "revision": 12,
"kind": "trunk", "kind": "trunk",
"source": 3589, "source": 3589,
"call": "release of mesh-sdk 0.1.27",
"call-kind": "release",
"module": "mesh-sdk",
"version": "0.1.27",
"entries": 2, "entries": 2,
"changed": [ "changed": [
"xclip" "xclip"
@@ -209,7 +209,11 @@
"repository": "novox/mesh-catalog" "repository": "novox/mesh-catalog"
} }
], ],
"base": 9 "base": 9,
"call": "release of mesh-sdk 0.1.27",
"call-kind": "release",
"module": "mesh-sdk",
"version": "0.1.27"
} }
}, },
{ {
@@ -657,6 +657,32 @@
"source": { "source": {
"type": "integer" "type": "integer"
}, },
"call": {
"type": "string",
"description": "what made the revision, as the revision's call names it (design 47 H-A-212, H-A-213); absent for a candidate's"
},
"call-kind": {
"type": "string",
"enum": [
"take-in",
"release",
"take-in-again",
"recompute"
],
"description": "the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217); absent for a candidate's"
},
"module": {
"type": "string",
"description": "for call-kind release, the module released (H-A-215)"
},
"version": {
"type": "string",
"description": "for call-kind release, the version released (H-A-215)"
},
"repository": {
"type": "string",
"description": "for call-kind take-in-again, the repository taken in again (H-A-215)"
},
"group": { "group": {
"type": "string" "type": "string"
}, },
@@ -1871,6 +1897,32 @@
"type": "integer", "type": "integer",
"description": "the source revision it was taken in from; for a candidate, the source revision its heads are merged onto" "description": "the source revision it was taken in from; for a candidate, the source revision its heads are merged onto"
}, },
"call": {
"type": "string",
"description": "what made the revision (design 47 H-A-212, H-A-213): a take-in's the gate's call its source revision names, a re-pin's release of <module> <version>, a take-in again's take-in again of <repository>; absent for a candidate's"
},
"call-kind": {
"type": "string",
"enum": [
"take-in",
"release",
"take-in-again",
"recompute"
],
"description": "the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217): a take-in's, a re-pin's on a module's release, a take-in again's, a recompute's that moved a fingerprint (its call naming the reason); absent for a candidate's"
},
"module": {
"type": "string",
"description": "for call-kind release, the module released (H-A-215)"
},
"version": {
"type": "string",
"description": "for call-kind release, the version released (H-A-215)"
},
"repository": {
"type": "string",
"description": "for call-kind take-in-again, the repository taken in again (H-A-215)"
},
"group": { "group": {
"type": "string", "type": "string",
"description": "a candidate's group" "description": "a candidate's group"
@@ -1,7 +1,7 @@
[ [
{ {
"name": "a take-in's pipeline kept, its steps naming their mover", "name": "a take-in's pipeline kept, its steps naming their mover",
"of": "state:pipelines/<branch>.<source>", "of": "state:pipelines/<branch>.<revision>",
"given": { "given": {
"pipeline": "main.3589", "pipeline": "main.3589",
"branch": "main", "branch": "main",
@@ -200,7 +200,7 @@
}, },
{ {
"name": "a pipeline waiting its turn, its seq given", "name": "a pipeline waiting its turn, its seq given",
"of": "state:pipelines/<branch>.<source>", "of": "state:pipelines/<branch>.<revision>",
"given": { "given": {
"pipeline": "main.3590", "pipeline": "main.3590",
"branch": "main", "branch": "main",
@@ -224,7 +224,7 @@
}, },
{ {
"name": "a waiting pipeline names the movers a newer take-in took out of it", "name": "a waiting pipeline names the movers a newer take-in took out of it",
"of": "state:pipelines/<branch>.<source>", "of": "state:pipelines/<branch>.<revision>",
"given": { "given": {
"pipeline": "main.3590", "pipeline": "main.3590",
"branch": "main", "branch": "main",
@@ -86,13 +86,17 @@
}, },
{ {
"name": "show", "name": "show",
"description": "One pipeline whole: its steps, its step log and its outputs, with the stores' fill. Reads only.", "description": "One pipeline whole: its record with every mover's steps, or one mover's steps alone when it names the mover, its step log and its outputs, with the stores' fill. Reads only.",
"input": { "input": {
"type": "object", "type": "object",
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key" "description": "the pipeline's key"
},
"mover": {
"type": "string",
"description": "a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187)"
} }
}, },
"required": [ "required": [
@@ -133,7 +137,7 @@
}, },
{ {
"name": "list", "name": "list",
"description": "The pipelines and runs of a module, a branch or a commit, newest first, with the store's fill. Reads only.", "description": "The pipelines and runs of a module, a branch or a commit, newest first, with the store's fill: each pipeline's record with its movers and steps counted, never its steps (show gives those). Reads only.",
"input": { "input": {
"type": "object", "type": "object",
"properties": { "properties": {
@@ -668,7 +672,7 @@
"events": [ "events": [
{ {
"name": "pipeline-started", "name": "pipeline-started",
"subject": "pipeline-started.<branch>.<source>", "subject": "pipeline-started.<branch>.<revision>",
"body": { "body": {
"$ref": "#/defs/PipelineStarted" "$ref": "#/defs/PipelineStarted"
}, },
@@ -679,7 +683,7 @@
}, },
{ {
"name": "pipeline-ended", "name": "pipeline-ended",
"subject": "pipeline-ended.<branch>.<source>", "subject": "pipeline-ended.<branch>.<revision>",
"body": { "body": {
"$ref": "#/defs/PipelineEnded" "$ref": "#/defs/PipelineEnded"
}, },
@@ -723,7 +727,7 @@
}, },
{ {
"name": "wait-held", "name": "wait-held",
"subject": "wait-held.<branch>.<source>", "subject": "wait-held.<branch>.<revision>",
"body": { "body": {
"$ref": "#/defs/Wait" "$ref": "#/defs/Wait"
}, },
@@ -734,7 +738,7 @@
}, },
{ {
"name": "wait-ended", "name": "wait-ended",
"subject": "wait-ended.<branch>.<source>", "subject": "wait-ended.<branch>.<revision>",
"body": { "body": {
"$ref": "#/defs/Wait" "$ref": "#/defs/Wait"
}, },
@@ -771,14 +775,14 @@
"name": "pipelines", "name": "pipelines",
"template": "state", "template": "state",
"history": 1, "history": 1,
"description": "one key per pipeline, one per take-in, written by compare-and-set on its absence (design 47 H §746, §747)", "description": "one key per pipeline, one per take-in, written by compare-and-set on its absence (design 47 H §746, §747), and one key per mover of a pipeline holding that mover's steps (H-A-187)",
"key-forms": [ "key-forms": [
{ {
"key": "<branch>.<source>", "key": "<branch>.<revision>",
"value": { "value": {
"$ref": "#/defs/Pipeline" "$ref": "#/defs/Pipeline"
}, },
"description": "a take-in's pipeline" "description": "a pipeline of one catalog revision, keyed by the revision's number (design 47 H-A-213 (4), H-A-219); one kept from before under its source has the same form"
}, },
{ {
"key": "<branch>.<source>.published.<package>.<version>", "key": "<branch>.<source>.published.<package>.<version>",
@@ -793,6 +797,32 @@
"$ref": "#/defs/Pipeline" "$ref": "#/defs/Pipeline"
}, },
"description": "a run on request's pipeline, one per run, its movers the named commits' modules; it delivers nothing (H-A-83)" "description": "a run on request's pipeline, one per run, its movers the named commits' modules; it delivers nothing (H-A-83)"
},
{
"key": "steps.<pipeline>.<mover>",
"value": {
"type": "object",
"properties": {
"steps": {
"type": "array",
"items": {
"$ref": "#/defs/StepResult"
},
"description": "the mover's steps"
},
"unlogged": {
"type": "array",
"items": {
"$ref": "#/defs/LogEntry"
},
"description": "the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189)"
}
},
"required": [
"steps"
]
},
"description": "one mover's steps and unlogged step-log entries in a pipeline, its key the pipeline's and the mover's module escaped as a key's name, written by compare-and-set as the pipeline's record is; the record's own steps stay empty, so a record never grows with the catalog (H-A-187)"
} }
] ]
}, },
@@ -1169,7 +1199,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83)" "description": "the pipeline's key, <branch>.<revision>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83)"
}, },
"branch": { "branch": {
"type": "string", "type": "string",
@@ -1238,7 +1268,7 @@
"items": { "items": {
"$ref": "#/defs/StepResult" "$ref": "#/defs/StepResult"
}, },
"description": "every mover's steps, each naming its mover, approval last" "description": "every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187)"
}, },
"started": { "started": {
"type": "string", "type": "string",
@@ -1253,7 +1283,15 @@
"items": { "items": {
"$ref": "#/defs/LogEntry" "$ref": "#/defs/LogEntry"
}, },
"description": "the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606)" "description": "the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189)"
},
"mover-count": {
"type": "integer",
"description": "how many movers the pipeline has, set by list alone (H-A-191)"
},
"step-count": {
"type": "integer",
"description": "how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191)"
}, },
"published": { "published": {
"$ref": "#/defs/Published", "$ref": "#/defs/Published",
@@ -1275,7 +1313,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)" "description": "the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)"
}, },
"branch": { "branch": {
"type": "string", "type": "string",
@@ -1350,7 +1388,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)" "description": "the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)"
}, },
"branch": { "branch": {
"type": "string", "type": "string",
@@ -1457,7 +1495,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)" "description": "the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)"
}, },
"module": { "module": {
"type": "string" "type": "string"
@@ -1512,7 +1550,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)" "description": "the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)"
}, },
"module": { "module": {
"type": "string", "type": "string",
@@ -1597,7 +1635,7 @@
"description": "the run the step waits on, where the runner knows it: a test step's run key and ask (design 47 G §731, A-469v)" "description": "the run the step waits on, where the runner knows it: a test step's run key and ask (design 47 G §731, A-469v)"
} }
}, },
"description": "a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<source> (design 47 H-A-72)", "description": "a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<revision> (design 47 H-A-72, H-A-221)",
"required": [ "required": [
"v", "v",
"pipeline", "pipeline",
@@ -1613,6 +1651,10 @@
"pipeline": { "pipeline": {
"type": "string" "type": "string"
}, },
"module": {
"type": "string",
"description": "the module of the mover whose step the entry is of (H-A-190)"
},
"step": { "step": {
"type": "string", "type": "string",
"enum": [ "enum": [
@@ -1936,7 +1978,7 @@
"properties": { "properties": {
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)" "description": "the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747)"
} }
}, },
"description": "a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72)", "description": "a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72)",
@@ -65,12 +65,13 @@
{ {
"name": "a pipeline's wants all run", "name": "a pipeline's wants all run",
"of": "event:pipeline-wanted", "of": "event:pipeline-wanted",
"subject": "mesh.seat.mesh-wants.event.pipeline-wanted.main.3589", "subject": "mesh.seat.mesh-wants.event.pipeline-wanted.main.3590",
"given": { "given": {
"v": 1, "v": 1,
"pipeline": "main.3589", "pipeline": "main.3590",
"branch": "main", "branch": "main",
"source": 3589, "source": 3589,
"revision": 3590,
"outcome": "running" "outcome": "running"
} }
}, },
@@ -434,7 +434,7 @@
}, },
{ {
"name": "pipeline-wanted", "name": "pipeline-wanted",
"subject": "pipeline-wanted.<branch>.<source>", "subject": "pipeline-wanted.<branch>.<revision>",
"keep": { "keep": {
"max-age-days": 30, "max-age-days": 30,
"max-mib": 64 "max-mib": 64
@@ -1725,7 +1725,7 @@
}, },
"pipeline": { "pipeline": {
"type": "string", "type": "string",
"description": "the runner's pipeline key, <branch>.<source>, a branch escaped as a key's name is (design 47 H §746)" "description": "the runner's pipeline key, <branch>.<revision>, a branch escaped as a key's name is (design 47 H §746, H-A-213 (4))"
}, },
"branch": { "branch": {
"type": "string" "type": "string"
@@ -1733,6 +1733,10 @@
"source": { "source": {
"type": "integer" "type": "integer"
}, },
"revision": {
"type": "integer",
"description": "the pipeline's catalog revision number, the number of its key; the indexer frees the pins of that revision, not of another revision of the same source (design 47 H-A-214); absent from an older event"
},
"outcome": { "outcome": {
"type": "string", "type": "string",
"enum": [ "enum": [
@@ -69,6 +69,8 @@ export interface PackagePublished {
"digest": string; "digest": string;
/** the module */ /** the module */
"module": string; "module": string;
/** the catalog revision number of that branch the build was made from, the indexer's count (design 47 H-A-218, H-A-220); absent from a reservation made before the field */
"revision"?: number;
/** the source revision of that branch the build was made from */ /** the source revision of that branch the build was made from */
"source": number; "source": number;
"v": number; "v": number;
@@ -89,6 +91,8 @@ export interface StepEnded {
"node": string; "node": string;
/** never passed for a build that built nothing */ /** never passed for a build that built nothing */
"outcome": "passed" | "failed" | "nothing to run" | "could not run"; "outcome": "passed" | "failed" | "nothing to run" | "could not run";
/** the pins the step's entries were built with, each mesh package's version by its module path, the ones the builder took from the registry for a package the entry named with none included; the runner asks the entry's held test steps with them (H-A-207) */
"packages"?: Record<string, string>;
"pipeline": string; "pipeline": string;
"rerun"?: boolean; "rerun"?: boolean;
/** why it failed or could not run: the entry not built, the part not heard */ /** why it failed or could not run: the entry not built, the part not heard */
@@ -277,6 +277,10 @@ export interface CatalogRevision {
"base"?: number; "base"?: number;
/** the watched branch the revision is of */ /** the watched branch the revision is of */
"branch"?: string; "branch"?: string;
/** what made the revision (design 47 H-A-212, H-A-213): a take-in's the gate's call its source revision names, a re-pin's release of <module> <version>, a take-in again's take-in again of <repository>; absent for a candidate's */
"call"?: string;
/** the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217): a take-in's, a re-pin's on a module's release, a take-in again's, a recompute's that moved a fingerprint (its call naming the reason); absent for a candidate's */
"call-kind"?: "take-in" | "release" | "take-in-again" | "recompute";
/** the seat definitions it was judged with, each by its digest, read whole through seat-definitions */ /** the seat definitions it was judged with, each by its digest, read whole through seat-definitions */
"definitions"?: CatalogRevisionDefinitionsItem[]; "definitions"?: CatalogRevisionDefinitionsItem[];
/** one per module, by name */ /** one per module, by name */
@@ -287,8 +291,12 @@ export interface CatalogRevision {
"heads"?: string; "heads"?: string;
/** trunk or candidate */ /** trunk or candidate */
"kind": string; "kind": string;
/** for call-kind release, the module released (H-A-215) */
"module"?: string;
/** the movers of the take-in that wrote it, so a redelivered source-moved whose revision is kept says take-in.taken again from them, recomputing nothing (design 47 H §743, H-A-92); absent for one with no mover */ /** the movers of the take-in that wrote it, so a redelivered source-moved whose revision is kept says take-in.taken again from them, recomputing nothing (design 47 H §743, H-A-92); absent for one with no mover */
"movers"?: TakeInEntry[]; "movers"?: TakeInEntry[];
/** for call-kind take-in-again, the repository taken in again (H-A-215) */
"repository"?: string;
/** the catalog graph revision's number, counted by its holder */ /** the catalog graph revision's number, counted by its holder */
"revision": number; "revision": number;
/** the source revision it was taken in from; for a candidate, the source revision its heads are merged onto */ /** the source revision it was taken in from; for a candidate, the source revision its heads are merged onto */
@@ -297,6 +305,8 @@ export interface CatalogRevision {
"trunk"?: Head[]; "trunk"?: Head[];
/** the layout version */ /** the layout version */
"v"?: number; "v"?: number;
/** for call-kind release, the version released (H-A-215) */
"version"?: string;
} }
/** one edge of the catalog graph between artifact references, B before A (ADR 0666 section 14): kind 1 B published before A builds (a package A imports), 2 B published before A delivers (fetched by reference at its start), 3 B delivered before A builds (its build module and toolchain), 4 B delivered before A delivers (a provision, bus state, a format's reader), 5 B has acted for A's new version (an acting rule), 6 B runs a version that serves A (a verb or a manifest word A uses) */ /** one edge of the catalog graph between artifact references, B before A (ADR 0666 section 14): kind 1 B published before A builds (a package A imports), 2 B published before A delivers (fetched by reference at its start), 3 B delivered before A builds (its build module and toolchain), 4 B delivered before A delivers (a provision, bus state, a format's reader), 5 B has acted for A's new version (an acting rule), 6 B runs a version that serves A (a verb or a manifest word A uses) */
@@ -796,6 +806,10 @@ export interface CatalogRevisionEvent {
"at"?: string; "at"?: string;
/** the watched branch the revision is of, escaped as a key name is */ /** the watched branch the revision is of, escaped as a key name is */
"branch"?: string; "branch"?: string;
/** what made the revision, as the revision's call names it (design 47 H-A-212, H-A-213); absent for a candidate's */
"call"?: string;
/** the kind of what made the revision, which a reader reads instead of call's words (design 47 H-A-213 (3), H-A-214, H-A-215, H-A-217); absent for a candidate's */
"call-kind"?: "take-in" | "release" | "take-in-again" | "recompute";
/** the modules whose entries differ from the revision before it of the same kind */ /** the modules whose entries differ from the revision before it of the same kind */
"changed"?: string[]; "changed"?: string[];
/** how many entries it lists */ /** how many entries it lists */
@@ -804,9 +818,15 @@ export interface CatalogRevisionEvent {
"heads"?: string; "heads"?: string;
/** trunk or candidate: a candidate's moves no running part */ /** trunk or candidate: a candidate's moves no running part */
"kind": string; "kind": string;
/** for call-kind release, the module released (H-A-215) */
"module"?: string;
/** for call-kind take-in-again, the repository taken in again (H-A-215) */
"repository"?: string;
"revision": number; "revision": number;
"source": number; "source": number;
"v": number; "v": number;
/** for call-kind release, the version released (H-A-215) */
"version"?: string;
} }
/** is the body of the event filling. */ /** is the body of the event filling. */
@@ -40,13 +40,13 @@ export declare const family: { readonly name: "mesh-pipeline-runner.events"; rea
export declare const pipelineStartedStream: "SEAT_mesh-pipeline-runner_pipeline-started"; export declare const pipelineStartedStream: "SEAT_mesh-pipeline-runner_pipeline-started";
export declare function pipelineStartedSubject(branch: string, source: string): string; export declare function pipelineStartedSubject(branch: string, revision: string): string;
export declare const pipelineStartedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.pipeline-started.*.*"; export declare const pipelineStartedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.pipeline-started.*.*";
export declare const pipelineEndedStream: "SEAT_mesh-pipeline-runner_pipeline-ended"; export declare const pipelineEndedStream: "SEAT_mesh-pipeline-runner_pipeline-ended";
export declare function pipelineEndedSubject(branch: string, source: string): string; export declare function pipelineEndedSubject(branch: string, revision: string): string;
export declare const pipelineEndedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.*.*"; export declare const pipelineEndedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.*.*";
@@ -70,13 +70,13 @@ export declare const stepEndedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.
export declare const waitHeldStream: "SEAT_mesh-pipeline-runner_wait-held"; export declare const waitHeldStream: "SEAT_mesh-pipeline-runner_wait-held";
export declare function waitHeldSubject(branch: string, source: string): string; export declare function waitHeldSubject(branch: string, revision: string): string;
export declare const waitHeldSubjectAll: "mesh.seat.mesh-pipeline-runner.event.wait-held.*.*"; export declare const waitHeldSubjectAll: "mesh.seat.mesh-pipeline-runner.event.wait-held.*.*";
export declare const waitEndedStream: "SEAT_mesh-pipeline-runner_wait-ended"; export declare const waitEndedStream: "SEAT_mesh-pipeline-runner_wait-ended";
export declare function waitEndedSubject(branch: string, source: string): string; export declare function waitEndedSubject(branch: string, revision: string): string;
export declare const waitEndedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.wait-ended.*.*"; export declare const waitEndedSubjectAll: "mesh.seat.mesh-pipeline-runner.event.wait-ended.*.*";
@@ -94,11 +94,11 @@ export declare const states: {
readonly stepOutputs: "step-outputs"; readonly stepOutputs: "step-outputs";
}; };
/** The key of one value of the state pipelines under the form "<branch>.<source>". */ /** The key of one value of the state pipelines under the form "<branch>.<revision>". */
export declare function pipelinesByBranchSourceKey(branch: string, source: string): string; export declare function pipelinesByBranchRevisionKey(branch: string, revision: string): string;
/** The parts of a key of the form "<branch>.<source>", or undefined for a key of another form. */ /** The parts of a key of the form "<branch>.<revision>", or undefined for a key of another form. */
export declare function parsePipelinesByBranchSourceKey(_key: string): { branch: string; source: string } | undefined; export declare function parsePipelinesByBranchRevisionKey(_key: string): { branch: string; revision: string } | undefined;
/** The key of one value of the state pipelines under the form "<branch>.<source>.published.<package>.<version>". */ /** The key of one value of the state pipelines under the form "<branch>.<source>.published.<package>.<version>". */
export declare function pipelinesByBranchSourcePublishedPackageVersionKey(branch: string, source: string, packagePart: string, version: string): string; export declare function pipelinesByBranchSourcePublishedPackageVersionKey(branch: string, source: string, packagePart: string, version: string): string;
@@ -112,6 +112,12 @@ export declare function pipelinesByBranchRunRunKey(branch: string, run: string):
/** The parts of a key of the form "<branch>.run.<run>", or undefined for a key of another form. */ /** The parts of a key of the form "<branch>.run.<run>", or undefined for a key of another form. */
export declare function parsePipelinesByBranchRunRunKey(_key: string): { branch: string; run: string } | undefined; export declare function parsePipelinesByBranchRunRunKey(_key: string): { branch: string; run: string } | undefined;
/** The key of one value of the state pipelines under the form "steps.<pipeline>.<mover>". */
export declare function pipelinesByStepsPipelineMoverKey(pipeline: string, mover: string): string;
/** The parts of a key of the form "steps.<pipeline>.<mover>", or undefined for a key of another form. */
export declare function parsePipelinesByStepsPipelineMoverKey(_key: string): { pipeline: string; mover: string } | undefined;
/** The key of one value of the state queue under the form "queue.next". */ /** The key of one value of the state queue under the form "queue.next". */
export declare function queueByQueueNextKey(): string; export declare function queueByQueueNextKey(): string;
@@ -186,6 +192,8 @@ export interface LogEntry {
"at": string; "at": string;
/** the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) */ /** the keys of the kept results the step wrote or carried, <level>.<digest>, which the runner ages by their last-read (design 47 G sections 330, 470) */
"kept"?: string[]; "kept"?: string[];
/** the module of the mover whose step the entry is of (H-A-190) */
"module"?: string;
"outcome"?: "running" | "waiting" | "passed" | "failed" | "nothing to run" | "bypassed" | "carried" | "could not run" | "waiting for a build node" | "did not build" | "accepted" | "declined" | "superseded" | "cancelled"; "outcome"?: "running" | "waiting" | "passed" | "failed" | "nothing to run" | "bypassed" | "carried" | "could not run" | "waiting for a build node" | "did not build" | "accepted" | "declined" | "superseded" | "cancelled";
/** the output's name in step-outputs */ /** the output's name in step-outputs */
"output"?: string; "output"?: string;
@@ -248,10 +256,12 @@ export interface Pipeline {
/** how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) */ /** how often the pipeline has ended: 1 at its first end, one higher at each end again after a passing rerun, so the assigner takes a later end of the same source revision (design 47 G sections 29, 325) */
"end-count"?: number; "end-count"?: number;
"ended"?: string; "ended"?: string;
/** how many movers the pipeline has, set by list alone (H-A-191) */
"mover-count"?: number;
/** the modules the pipeline moves, its take-in's entries (design 47 H §746) */ /** the modules the pipeline moves, its take-in's entries (design 47 H §746) */
"movers": Mover[]; "movers": Mover[];
"outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled"; "outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled";
/** the pipeline's key, <branch>.<source>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83) */ /** the pipeline's key, <branch>.<revision>, <branch>.<source>.published.<package>.<version> for a pipeline a publish started, or <branch>.run.<run> for a run on request; a branch escaped as a key's name is (design 47 H §746, §747; H-A-83) */
"pipeline": string; "pipeline": string;
/** for a pipeline a package's publish started, that publish (A-202) */ /** for a pipeline a package's publish started, that publish (A-202) */
"published"?: Published; "published"?: Published;
@@ -264,11 +274,13 @@ export interface Pipeline {
/** the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) */ /** the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) */
"source": number; "source": number;
"started"?: string; "started"?: string;
/** every mover's steps, each naming its mover, approval last */ /** how many steps the pipeline's movers have, set by list alone, which answers the record with its steps empty (H-A-191) */
"step-count"?: number;
/** every mover's steps, each naming its mover, approval last; in the state pipelines a record's own are empty, each mover's kept under its own key (H-A-187) */
"steps": StepResult[]; "steps": StepResult[];
/** the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) */ /** the modules a newer waiting take-in of the same branch took out of the pipeline, written with the movers by compare-and-set; at its placing the runner writes a mover's build-step why from it (design 47 H §748, H-A-107) */
"superseded"?: string[]; "superseded"?: string[];
/** the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606) */ /** the step-log entries of the steps that ended and are not appended to the step log yet, in order; written with the step's end in one write and cleared when appended (design 47 G sections 486, 606); in the state pipelines a record's own are empty, each mover's kept under its own key beside its steps (H-A-189) */
"unlogged"?: LogEntry[]; "unlogged"?: LogEntry[];
/** the layout version */ /** the layout version */
"v"?: number; "v"?: number;
@@ -288,7 +300,7 @@ export interface PipelineEnded {
/** per mover its fingerprint, released version, step results and tier, in the delivery order (design 47 H §746, H-A-2) */ /** per mover its fingerprint, released version, step results and tier, in the delivery order (design 47 H §746, H-A-2) */
"movers": MoverEnded[]; "movers": MoverEnded[];
"outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled"; "outcome": "waiting" | "running" | "passed" | "failed" | "refused" | "superseded" | "cancelled";
/** the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */ /** the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */
"pipeline": string; "pipeline": string;
/** the catalog graph revision of the branch the pipeline runs for */ /** the catalog graph revision of the branch the pipeline runs for */
"revision"?: number; "revision"?: number;
@@ -310,7 +322,7 @@ export interface PipelineStarted {
"call"?: string; "call"?: string;
/** the pipeline's movers */ /** the pipeline's movers */
"movers": Mover[]; "movers": Mover[];
/** the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */ /** the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */
"pipeline": string; "pipeline": string;
/** for a pipeline a package's publish started, that publish (A-202) */ /** for a pipeline a package's publish started, that publish (A-202) */
"published"?: Published; "published"?: Published;
@@ -371,7 +383,7 @@ export interface StepEvent {
/** the module's build-input fingerprint at the commit */ /** the module's build-input fingerprint at the commit */
"fingerprint"?: string; "fingerprint"?: string;
"module": string; "module": string;
/** the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */ /** the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */
"pipeline": string; "pipeline": string;
"result": StepResult; "result": StepResult;
/** the catalog graph revision of the branch the pipeline runs for */ /** the catalog graph revision of the branch the pipeline runs for */
@@ -478,7 +490,7 @@ export interface ToolchainUsed {
"version": string; "version": string;
} }
/** a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<source> (design 47 H-A-72) */ /** a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283); said on <branch>.<revision> (design 47 H-A-72, H-A-221) */
export interface Wait { export interface Wait {
"artifact"?: string; "artifact"?: string;
/** the step's own ask, where it has one (design 47 G §731) */ /** the step's own ask, where it has one (design 47 G §731) */
@@ -495,7 +507,7 @@ export interface Wait {
"module"?: string; "module"?: string;
/** what the step waits on (design 47 *Health and going back*) */ /** what the step waits on (design 47 *Health and going back*) */
"on": "person" | "build node" | "revision view" | "pipelines-at-once"; "on": "person" | "build node" | "revision view" | "pipelines-at-once";
/** the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */ /** the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */
"pipeline": string; "pipeline": string;
/** the catalog graph revision of the branch the pipeline runs for */ /** the catalog graph revision of the branch the pipeline runs for */
"revision"?: number; "revision"?: number;
@@ -515,7 +527,7 @@ export interface Wait {
/** a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72) */ /** a pipeline waiting its turn under pipelines-at-once (design 47 H §748, H-A-72) */
export interface Waiting { export interface Waiting {
/** the pipeline's key, <branch>.<source>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */ /** the pipeline's key, <branch>.<revision>, or <branch>.<source>.published.<package>.<version> for a pipeline a publish started; a branch escaped as a key's name is (design 47 H §746, §747) */
"pipeline": string; "pipeline": string;
} }
@@ -556,6 +568,8 @@ export interface StartAnswer {
/** is what show takes. */ /** is what show takes. */
export interface ShowInput { export interface ShowInput {
/** a mover's module: the answer's pipeline then holds that mover's steps alone (H-A-187) */
"mover"?: string;
/** the pipeline's key */ /** the pipeline's key */
"pipeline": string; "pipeline": string;
} }
@@ -786,8 +800,8 @@ export type FullEvent = Fill;
/** is the body of the event room. */ /** is the body of the event room. */
export type RoomEvent = Fill; export type RoomEvent = Fill;
/** is one value of the state pipelines under the key <branch>.<source>. */ /** is one value of the state pipelines under the key <branch>.<revision>. */
export type PipelinesByBranchSourceValue = Pipeline; export type PipelinesByBranchRevisionValue = Pipeline;
/** is one value of the state pipelines under the key <branch>.<source>.published.<package>.<version>. */ /** is one value of the state pipelines under the key <branch>.<source>.published.<package>.<version>. */
export type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline; export type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline;
@@ -795,6 +809,14 @@ export type PipelinesByBranchSourcePublishedPackageVersionValue = Pipeline;
/** is one value of the state pipelines under the key <branch>.run.<run>. */ /** is one value of the state pipelines under the key <branch>.run.<run>. */
export type PipelinesByBranchRunRunValue = Pipeline; export type PipelinesByBranchRunRunValue = Pipeline;
/** is one value of the state pipelines under the key steps.<pipeline>.<mover>. */
export interface PipelinesByStepsPipelineMoverValue {
/** the mover's steps */
"steps": StepResult[];
/** the step-log entries of the mover's steps that ended and are not appended to the step log yet, in order (H-A-189) */
"unlogged"?: LogEntry[];
}
/** is one value of the state queue under the key queue.next. */ /** is one value of the state queue under the key queue.next. */
export type QueueByQueueNextValue = number; export type QueueByQueueNextValue = number;
@@ -40,16 +40,16 @@ export const family = { name: "mesh-pipeline-runner.events", maxAgeDays: 7 };
export const pipelineStartedStream = "SEAT_mesh-pipeline-runner_pipeline-started"; export const pipelineStartedStream = "SEAT_mesh-pipeline-runner_pipeline-started";
export function pipelineStartedSubject(branch, source) { export function pipelineStartedSubject(branch, revision) {
return "mesh.seat.mesh-pipeline-runner.event.pipeline-started." + branch + "." + source; return "mesh.seat.mesh-pipeline-runner.event.pipeline-started." + branch + "." + revision;
} }
export const pipelineStartedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.pipeline-started.*.*"; export const pipelineStartedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.pipeline-started.*.*";
export const pipelineEndedStream = "SEAT_mesh-pipeline-runner_pipeline-ended"; export const pipelineEndedStream = "SEAT_mesh-pipeline-runner_pipeline-ended";
export function pipelineEndedSubject(branch, source) { export function pipelineEndedSubject(branch, revision) {
return "mesh.seat.mesh-pipeline-runner.event.pipeline-ended." + branch + "." + source; return "mesh.seat.mesh-pipeline-runner.event.pipeline-ended." + branch + "." + revision;
} }
export const pipelineEndedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.*.*"; export const pipelineEndedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.pipeline-ended.*.*";
@@ -80,16 +80,16 @@ export const stepEndedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.step-en
export const waitHeldStream = "SEAT_mesh-pipeline-runner_wait-held"; export const waitHeldStream = "SEAT_mesh-pipeline-runner_wait-held";
export function waitHeldSubject(branch, source) { export function waitHeldSubject(branch, revision) {
return "mesh.seat.mesh-pipeline-runner.event.wait-held." + branch + "." + source; return "mesh.seat.mesh-pipeline-runner.event.wait-held." + branch + "." + revision;
} }
export const waitHeldSubjectAll = "mesh.seat.mesh-pipeline-runner.event.wait-held.*.*"; export const waitHeldSubjectAll = "mesh.seat.mesh-pipeline-runner.event.wait-held.*.*";
export const waitEndedStream = "SEAT_mesh-pipeline-runner_wait-ended"; export const waitEndedStream = "SEAT_mesh-pipeline-runner_wait-ended";
export function waitEndedSubject(branch, source) { export function waitEndedSubject(branch, revision) {
return "mesh.seat.mesh-pipeline-runner.event.wait-ended." + branch + "." + source; return "mesh.seat.mesh-pipeline-runner.event.wait-ended." + branch + "." + revision;
} }
export const waitEndedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.wait-ended.*.*"; export const waitEndedSubjectAll = "mesh.seat.mesh-pipeline-runner.event.wait-ended.*.*";
@@ -108,23 +108,23 @@ export const states = {
stepOutputs: "step-outputs", stepOutputs: "step-outputs",
}; };
/** The key of one value of the state pipelines under the form "<branch>.<source>". */ /** The key of one value of the state pipelines under the form "<branch>.<revision>". */
export function pipelinesByBranchSourceKey(branch, source) { export function pipelinesByBranchRevisionKey(branch, revision) {
return branch + "." + source; return branch + "." + revision;
} }
/** The parts of a key of the form "<branch>.<source>", or undefined for a key of another form. */ /** The parts of a key of the form "<branch>.<revision>", or undefined for a key of another form. */
export function parsePipelinesByBranchSourceKey(_key) { export function parsePipelinesByBranchRevisionKey(_key) {
let _rest = _key; let _rest = _key;
const _at0 = _rest.indexOf("."); const _at0 = _rest.indexOf(".");
if (_at0 <= 0) return undefined; if (_at0 <= 0) return undefined;
const branch = _rest.slice(0, _at0); const branch = _rest.slice(0, _at0);
_rest = _rest.slice(_at0 + 1); _rest = _rest.slice(_at0 + 1);
if (_rest === "") return undefined; if (_rest === "") return undefined;
const source = _rest; const revision = _rest;
_rest = ""; _rest = "";
if (_rest !== "") return undefined; if (_rest !== "") return undefined;
return { branch, source }; return { branch, revision };
} }
/** The key of one value of the state pipelines under the form "<branch>.<source>.published.<package>.<version>". */ /** The key of one value of the state pipelines under the form "<branch>.<source>.published.<package>.<version>". */
@@ -173,6 +173,27 @@ export function parsePipelinesByBranchRunRunKey(_key) {
return { branch, run }; return { branch, run };
} }
/** The key of one value of the state pipelines under the form "steps.<pipeline>.<mover>". */
export function pipelinesByStepsPipelineMoverKey(pipeline, mover) {
return "steps." + pipeline + "." + mover;
}
/** The parts of a key of the form "steps.<pipeline>.<mover>", or undefined for a key of another form. */
export function parsePipelinesByStepsPipelineMoverKey(_key) {
let _rest = _key;
if (!_rest.startsWith("steps.")) return undefined;
_rest = _rest.slice(6);
const _at1 = _rest.indexOf(".");
if (_at1 <= 0) return undefined;
const pipeline = _rest.slice(0, _at1);
_rest = _rest.slice(_at1 + 1);
if (_rest === "") return undefined;
const mover = _rest;
_rest = "";
if (_rest !== "") return undefined;
return { pipeline, mover };
}
/** The key of one value of the state queue under the form "queue.next". */ /** The key of one value of the state queue under the form "queue.next". */
export function queueByQueueNextKey() { export function queueByQueueNextKey() {
return "queue.next"; return "queue.next";
@@ -50,7 +50,7 @@ export declare const wantedSubjectAll: "mesh.seat.mesh-wants.event.wanted.*.*";
export declare const pipelineWantedStream: "SEAT_mesh-wants_pipeline-wanted"; export declare const pipelineWantedStream: "SEAT_mesh-wants_pipeline-wanted";
export declare function pipelineWantedSubject(branch: string, source: string): string; export declare function pipelineWantedSubject(branch: string, revision: string): string;
export declare const pipelineWantedSubjectAll: "mesh.seat.mesh-wants.event.pipeline-wanted.*.*"; export declare const pipelineWantedSubjectAll: "mesh.seat.mesh-wants.event.pipeline-wanted.*.*";
@@ -412,8 +412,10 @@ export interface PipelineWanted {
/** for tier-late, the late tier's movers not running and the machines they do not run on */ /** for tier-late, the late tier's movers not running and the machines they do not run on */
"not-running"?: NotRunning[]; "not-running"?: NotRunning[];
"outcome": "running" | "tier-late"; "outcome": "running" | "tier-late";
/** the runner's pipeline key, <branch>.<source>, a branch escaped as a key's name is (design 47 H §746) */ /** the runner's pipeline key, <branch>.<revision>, a branch escaped as a key's name is (design 47 H §746, H-A-213 (4)) */
"pipeline": string; "pipeline": string;
/** the pipeline's catalog revision number, the number of its key; the indexer frees the pins of that revision, not of another revision of the same source (design 47 H-A-214); absent from an older event */
"revision"?: number;
"source": number; "source": number;
"v": number; "v": number;
} }
@@ -52,8 +52,8 @@ export const wantedSubjectAll = "mesh.seat.mesh-wants.event.wanted.*.*";
export const pipelineWantedStream = "SEAT_mesh-wants_pipeline-wanted"; export const pipelineWantedStream = "SEAT_mesh-wants_pipeline-wanted";
export function pipelineWantedSubject(branch, source) { export function pipelineWantedSubject(branch, revision) {
return "mesh.seat.mesh-wants.event.pipeline-wanted." + branch + "." + source; return "mesh.seat.mesh-wants.event.pipeline-wanted." + branch + "." + revision;
} }
export const pipelineWantedSubjectAll = "mesh.seat.mesh-wants.event.pipeline-wanted.*.*"; export const pipelineWantedSubjectAll = "mesh.seat.mesh-wants.event.pipeline-wanted.*.*";
@@ -510,8 +510,10 @@ export interface StepRun {
"pipeline": string; "pipeline": string;
/** the step is asked again by the runner's rerun */ /** the step is asked again by the runner's rerun */
"rerun"?: boolean; "rerun"?: boolean;
/** the source revision of the watched branch the pipeline runs for; absent for a run on request */ /** the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request */
"revision"?: number; "revision"?: number;
/** the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request */
"source"?: number;
/** the pipeline step */ /** the pipeline step */
"step": "lint" | "unit" | "integration"; "step": "lint" | "unit" | "integration";
"toolchain"?: ToolchainUsed; "toolchain"?: ToolchainUsed;
@@ -510,8 +510,10 @@ export interface StepRun {
"pipeline": string; "pipeline": string;
/** the step is asked again by the runner's rerun */ /** the step is asked again by the runner's rerun */
"rerun"?: boolean; "rerun"?: boolean;
/** the source revision of the watched branch the pipeline runs for; absent for a run on request */ /** the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request */
"revision"?: number; "revision"?: number;
/** the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request */
"source"?: number;
/** the pipeline step */ /** the pipeline step */
"step": "lint" | "unit" | "integration"; "step": "lint" | "unit" | "integration";
"toolchain"?: ToolchainUsed; "toolchain"?: ToolchainUsed;
@@ -510,8 +510,10 @@ export interface StepRun {
"pipeline": string; "pipeline": string;
/** the step is asked again by the runner's rerun */ /** the step is asked again by the runner's rerun */
"rerun"?: boolean; "rerun"?: boolean;
/** the source revision of the watched branch the pipeline runs for; absent for a run on request */ /** the catalog revision number of the watched branch the pipeline runs on, the indexer's count, which a build module reads a revision by (design 47 H-A-219); absent for a run on request */
"revision"?: number; "revision"?: number;
/** the gate's source revision the pipeline's catalog revision was taken in from (design 47 H-A-219); absent for a run on request */
"source"?: number;
/** the pipeline step */ /** the pipeline step */
"step": "lint" | "unit" | "integration"; "step": "lint" | "unit" | "integration";
"toolchain"?: ToolchainUsed; "toolchain"?: ToolchainUsed;
@@ -187,10 +187,13 @@ func TestManifestProcess(t *testing.T) { // 1235/AC1
if _, bus := p["bus"]; bus { if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential") t.Errorf("the process declares a bus: it runs on its own broker credential")
} }
// Issue 1966's bootstrap: the package artifact forgemsg is out of the manifest until the new builder runs; the follow-up if len(m.Build.Artifacts) != 3 || m.Build.Artifacts[1].Name != "tools" || m.Build.Artifacts[1].From != "cmd/mesh-delivery-forge-bridge-tools" {
// puts it back with its assertion (ADR 0573 §1: three artifacts, the third the package forgemsg). t.Fatalf("artifacts = %+v, want the program, the tools bundle and the package forgemsg", m.Build.Artifacts)
if len(m.Build.Artifacts) != 2 || m.Build.Artifacts[1].Name != "tools" || m.Build.Artifacts[1].From != "cmd/mesh-delivery-forge-bridge-tools" { }
t.Fatalf("artifacts = %+v, want the program and the tools bundle", m.Build.Artifacts) // ADR 0573 §1, H-A-186 (design 47 G §370): the gate, the builder and the indexer import forgemsg, so the module declares
// it as a package and every release publishes it.
if a := m.Build.Artifacts[2]; a.Name != "forgemsg" || a.Kind != "package" || a.Language != "go" || len(a.Paths) != 1 || a.Paths[0] != "forgemsg" || a.Series != "0.1" {
t.Errorf("package artifact = %+v", a)
} }
a := m.Build.Artifacts[0] a := m.Build.Artifacts[0]
if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-forge-bridge" || !strings.HasSuffix(a.From, a.Binary) { if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-forge-bridge" || !strings.HasSuffix(a.From, a.Binary) {
@@ -101,6 +101,15 @@
"loads": [ "loads": [
"mesh-delivery-forge-bridge-tools" "mesh-delivery-forge-bridge-tools"
] ]
},
{
"name": "forgemsg",
"kind": "package",
"language": "go",
"paths": [
"forgemsg"
],
"series": "0.1"
} }
] ]
} }
@@ -0,0 +1,85 @@
package main
import (
"context"
"fmt"
"strconv"
"strings"
stdio "git.novox.be/novox/mesh-sdk/go"
"mesh-delivery-indexer/internal/index"
)
// entryAnswer is what indexer_entry answers: the trunk revision read, the entry's packages (the pins: each other-repository
// package with the version it builds against, empty while none is published), its named paths by artifact, its fingerprint
// and its manifest's digest.
type entryAnswer struct {
Module string `json:"module"`
Revision int64 `json:"revision"`
Packages map[string]string `json:"packages"`
Paths map[string][]string `json:"paths"`
Fp string `json:"fp,omitempty"`
ManifestDigest string `json:"manifest-digest,omitempty"`
}
// entryReadTool is indexer_entry (H-A-204; tools-are-how-a-module-is-spoken-to) over the indexer's catalog-revisions objects:
// one module's entry pins and paths at a trunk revision, small, so a pin can be read without the revision whole (1.3 MB,
// more than the bus carries).
func entryReadTool(o index.Objects) stdio.Tool {
return stdio.Tool{Name: index.EntryPinsTool,
Description: "Read one module's catalog graph entry pins: the packages it builds against with the version each is pinned to " +
"(an empty version while none is published), the paths each artifact is built from, its fingerprint and its manifest " +
"digest, at the newest trunk revision or at a trunk revision given; a small answer, for when the revision whole is too " +
"large for the bus (H-A-204).",
Input: map[string]any{"type": "object", "required": []string{"module"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module whose entry is read"},
"revision": map[string]any{"type": "integer", "description": "the trunk revision; none reads the newest"}}},
Run: func(args map[string]any) (any, error) {
module, _ := args["module"].(string)
module = strings.TrimSpace(module)
if module == "" {
return nil, fmt.Errorf("%s names the module", index.EntryPinsTool)
}
revision, err := revisionArg(args["revision"])
if err != nil {
return nil, err
}
rev, e, err := index.TrunkEntryAt(context.Background(), o, module, revision)
if err != nil {
return nil, err
}
out := entryAnswer{Module: e.Module, Revision: rev.Revision, Packages: e.Packages, Paths: map[string][]string{},
Fp: e.Fp, ManifestDigest: e.ManifestDigest}
if out.Packages == nil {
out.Packages = map[string]string{}
}
if e.Inputs != nil && e.Inputs.Paths != nil {
out.Paths = e.Inputs.Paths
}
return out, nil
}}
}
// revisionArg is the revision argument as a number: absent is 0 (the newest); a number or its digits as a string.
func revisionArg(v any) (int64, error) {
switch r := v.(type) {
case nil:
return 0, nil
case float64:
if r < 0 || r != float64(int64(r)) {
return 0, fmt.Errorf("revision %v is not a trunk revision's number", r)
}
return int64(r), nil
case string:
if strings.TrimSpace(r) == "" {
return 0, nil
}
n, err := strconv.ParseInt(strings.TrimSpace(r), 10, 64)
if err != nil || n < 0 {
return 0, fmt.Errorf("revision %q is not a trunk revision's number", r)
}
return n, nil
}
return 0, fmt.Errorf("revision %v is not a trunk revision's number", v)
}
@@ -0,0 +1,100 @@
package main
import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"mesh-delivery-indexer/internal/index"
)
// memObjects is catalog-revisions in memory.
type memObjects map[string][]byte
func (m memObjects) Put(_ context.Context, name string, data []byte) error {
m[name] = data
return nil
}
func (m memObjects) Get(_ context.Context, name string) ([]byte, error) {
d, ok := m[name]
if !ok {
return nil, index.ErrNoObject
}
return d, nil
}
func (m memObjects) Delete(_ context.Context, name string) error { delete(m, name); return nil }
func (m memObjects) Names(context.Context) ([]string, error) {
var out []string
for n := range m {
out = append(out, n)
}
return out, nil
}
func (memObjects) Fill(context.Context) (int64, int64, error) { return 0, 0, nil }
// TestEntryTool: indexer_entry answers one entry's pins, paths, fingerprint and manifest digest, small, at the newest trunk
// revision and at a given one read from its kept object; an unknown module or revision is an error (H-A-204).
func TestEntryTool(t *testing.T) {
o := memObjects{}
keep := func(revision int64, entries ...meshcatalog.CatalogEntry) {
rev := meshcatalog.CatalogRevision{Branch: index.TrunkBranch, Revision: revision}
for _, e := range entries {
e.Manifest = map[string]any{"module": e.Module, "big": strings.Repeat("x", 64<<10)} // the manifest whole is never answered
data, _ := json.Marshal(e)
digest := e.Module + "-" + e.Fp
o[index.EntryObject(digest)] = data
rev.Entries = append(rev.Entries, meshcatalog.EntryRef{Module: e.Module, Digest: digest})
}
data, _ := json.Marshal(rev)
o[index.BranchObject(index.TrunkBranch, revision)] = data
}
paths := &meshcatalog.EntryInputs{Paths: map[string][]string{"mesh-grants:program": {"modules/mesh-grants", "sdk"}}}
keep(7, meshcatalog.CatalogEntry{Module: "mesh-grants", Fp: "src2:old", ManifestDigest: "sha256:a", Inputs: paths,
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.26"}}, meshcatalog.CatalogEntry{Module: "other", Fp: "src2:o"})
keep(9, meshcatalog.CatalogEntry{Module: "mesh-grants", Fp: "src2:new", ManifestDigest: "sha256:b", Inputs: paths,
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.27", "busgrants": ""}})
tool := entryReadTool(o)
if tool.Name != "indexer_entry" {
t.Fatalf("name %s", tool.Name)
}
for _, c := range []struct {
name string
args map[string]any
want entryAnswer
err string
}{
{"newest", map[string]any{"module": "mesh-grants"}, entryAnswer{Module: "mesh-grants", Revision: 9, Fp: "src2:new", ManifestDigest: "sha256:b",
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.27", "busgrants": ""}, Paths: paths.Paths}, ""},
{"given", map[string]any{"module": "mesh-grants", "revision": float64(7)}, entryAnswer{Module: "mesh-grants", Revision: 7, Fp: "src2:old", ManifestDigest: "sha256:a",
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.26"}, Paths: paths.Paths}, ""},
{"given as digits, no paths", map[string]any{"module": "other", "revision": "7"}, entryAnswer{Module: "other", Revision: 7, Fp: "src2:o",
Packages: map[string]string{}, Paths: map[string][]string{}}, ""},
{"not in the newest", map[string]any{"module": "other"}, entryAnswer{}, "holds no module other"},
{"revision not kept", map[string]any{"module": "mesh-grants", "revision": float64(8)}, entryAnswer{}, "revision 8 is not kept"},
{"no module", map[string]any{}, entryAnswer{}, "names the module"},
} {
t.Run(c.name, func(t *testing.T) {
got, err := tool.Run(c.args)
if c.err != "" {
if err == nil || !strings.Contains(err.Error(), c.err) {
t.Fatalf("error %v, want %q", err, c.err)
}
return
}
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, c.want) {
t.Fatalf("answer %+v, want %+v", got, c.want)
}
data, _ := json.Marshal(got)
if len(data) >= 8<<10 {
t.Fatalf("answer of %d bytes, want under 8 KB", len(data))
}
})
}
}
@@ -193,7 +193,7 @@ func main() {
tools = append(tools, catalogTools(trunk.State, kept, watched)...) tools = append(tools, catalogTools(trunk.State, kept, watched)...)
// The seat mesh-catalog's verb revision, a first view of the catalog half answered in this runtime of its own to exactly // The seat mesh-catalog's verb revision, a first view of the catalog half answered in this runtime of its own to exactly
// the parts whose entry declares it (novox/hq ADR 0663). // the parts whose entry declares it (novox/hq ADR 0663).
tools = append(tools, index.RevisionTool(revisions, kept), index.EntryTool(revisions, keptRead{stdio.State(index.EntryIndexState)}), index.EntriesTool(revisions, keptRead{stdio.State(index.EntryIndexState)}, keptRead{stdio.State(index.NamedState)}), index.VersionsTool(keptRead{stdio.State(index.VersionsState)}), tools = append(tools, index.RevisionTool(revisions, kept), index.EntryTool(revisions, keptRead{stdio.State(index.EntryIndexState)}), index.EntriesTool(revisions, keptRead{stdio.State(index.EntryIndexState)}, keptRead{stdio.State(index.NamedState)}), entryReadTool(revisions), index.VersionsTool(keptRead{stdio.State(index.VersionsState)}),
index.OutdatedTool(&index.Indexer{Versions: keptRead{stdio.State(index.VersionsState)}, Heard: keptRead{stdio.State(index.HeardState)}, Revisions: revisions}), index.OutdatedTool(&index.Indexer{Versions: keptRead{stdio.State(index.VersionsState)}, Heard: keptRead{stdio.State(index.HeardState)}, Revisions: revisions}),
index.InputsReadTool(keptRead{stdio.State(index.InputsState)}, stateobjects.ForState(index.InputsState).Objects)) index.InputsReadTool(keptRead{stdio.State(index.InputsState)}, stateobjects.ForState(index.InputsState).Objects))
tools = append(tools, index.CatalogRevisionTools(revisions, asks, time.Now, func(what, why string) error { tools = append(tools, index.CatalogRevisionTools(revisions, asks, time.Now, func(what, why string) error {
@@ -433,7 +433,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
done() done()
done = step("following the assignments") done = step("following the assignments")
followAssignments(ctx, reader, seatHolders.holders(), process.AwaitBucket(ctx, js, conn, index.Bucket(cred.Module, views.StatusState), "indexer_first_views reads nothing and indexer_first_view_again is not heard", logger.Printf), cred, js, gate, func(ctx context.Context) { followAssignments(ctx, reader, seatHolders.holders(), process.AwaitBucket(ctx, js, conn, index.Bucket(cred.Module, views.StatusState), "indexer_first_views reads nothing and indexer_first_view_again is not heard", logger.Printf), cred, js, gate, func(ctx context.Context) {
if err := ix.Recompute(ctx); err != nil { if err := ix.RecomputeKept(ctx, "the assignments of a build module"); err != nil {
logger.Printf("writing the modules again after the assignments of a build module changed: %v", err) logger.Printf("writing the modules again after the assignments of a build module changed: %v", err)
} }
}, logger.Printf) }, logger.Printf)
@@ -449,7 +449,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// the toolchain. // the toolchain.
return ix.ToolchainReported(ctx) return ix.ToolchainReported(ctx)
}, },
recompute: ix.Recompute}, func() []string { return buildModuleNames(ix.BuildModules()) }, logger) recompute: func(ctx context.Context) error { return ix.RecomputeKept(ctx, "the toolchains read") }}, func() []string { return buildModuleNames(ix.BuildModules()) }, logger)
if err := tools.start(); err != nil { if err := tools.start(); err != nil {
return err return err
} }
@@ -5,6 +5,7 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"path" "path"
"slices" "slices"
@@ -115,10 +116,15 @@ func trunkVia(call toolCaller) func(ctx context.Context, repository, sha string)
} }
// filesVia asks the forge for every blob under each directory at a commit (the seat git's trees with files, ADR 0593 §2), // filesVia asks the forge for every blob under each directory at a commit (the seat git's trees with files, ADR 0593 §2),
// by directory, then path relative to it. An answer that lists no directory asked is an error naming it. // by directory, then path relative to it. An answer that lists no directory asked is an error naming it. The seat answers
// a directory the commit does not hold with the kind not-found naming it (the contract's files field; design 47 H H-A-232):
// that is a failure of the tree too (index.ErrNotInTree), so the indexer reads each directory alone and leaves the gone one out.
func filesVia(call toolCaller) func(ctx context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) { func filesVia(call toolCaller) func(ctx context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
return func(_ context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) { return func(_ context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
a, err := treesOf(call, repository, git.TreesInput{Commit: sha, Files: dirs}) a, err := treesOf(call, repository, git.TreesInput{Commit: sha, Files: dirs})
if errors.Is(err, index.ErrNotFound) {
return nil, fmt.Errorf("%w: %w", err, index.ErrNotInTree)
}
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -75,6 +75,20 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
if _, err := filesVia(noDir)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); !errors.Is(err, index.ErrNotInTree) { if _, err := filesVia(noDir)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); !errors.Is(err, index.ErrNotInTree) {
t.Errorf("a missing directory: %v, want ErrNotInTree", err) t.Errorf("a missing directory: %v, want ErrNotInTree", err)
} }
// H-A-232 (2): the seat's not-found naming a directory the commit lacks is a failure of the tree too, so the
// one-directory fallback runs; a failed listing of another kind is not.
gone := func(string, any) (json.RawMessage, error) {
return json.RawMessage(`{"error":"the directory modules/mesh-build-go/buildstep is not in novox/mesh-catalog at x","error-kind":"not-found"}`), nil
}
if _, err := filesVia(gone)(context.Background(), "novox/mesh-catalog", strings.Repeat("a", 40), []string{"modules/mesh-build-go/buildstep"}); !errors.Is(err, index.ErrNotInTree) {
t.Errorf("a directory the seat says not-found: %v, want ErrNotInTree", err)
}
failed := func(string, any) (json.RawMessage, error) {
return json.RawMessage(`{"error":"the forge did not answer","error-kind":"failed"}`), nil
}
if _, err := filesVia(failed)(context.Background(), "novox/mesh-catalog", strings.Repeat("a", 40), []string{"modules/a"}); err == nil || errors.Is(err, index.ErrNotInTree) {
t.Errorf("a listing that failed: %v, want an error that is not ErrNotInTree", err)
}
down := func(string, any) (json.RawMessage, error) { return nil, errors.New("down") } down := func(string, any) (json.RawMessage, error) { return nil, errors.New("down") }
if _, err := filesVia(down)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); err == nil || errors.Is(err, index.ErrNotInTree) { if _, err := filesVia(down)(context.Background(), "novox/mesh-sdk", strings.Repeat("a", 40), []string{"go"}); err == nil || errors.Is(err, index.ErrNotInTree) {
t.Errorf("a failed call: %v, want an error that is not ErrNotInTree", err) t.Errorf("a failed call: %v, want an error that is not ErrNotInTree", err)
@@ -90,7 +104,7 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
}{ }{
{name: "a run started", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}],"by":"s"}`, acked: 1, started: "7"}, {name: "a run started", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}],"by":"s"}`, acked: 1, started: "7"},
{name: "a run's pipeline ended", body: `{"v":1,"pipeline":"p","module":"a","branch":"b","commit":"c","outcome":"passed","steps":[],"run":"7"}`, ended: true, acked: 1, endedOf: "7"}, {name: "a run's pipeline ended", body: `{"v":1,"pipeline":"p","module":"a","branch":"b","commit":"c","outcome":"passed","steps":[],"run":"7"}`, ended: true, acked: 1, endedOf: "7"},
{name: "a pipeline's wants written to the end", body: `{"v":1,"pipeline":"main.12","branch":"main","source":12,"outcome":"tier-late"}`, wanted: true, acked: 1}, {name: "a pipeline's wants written to the end", body: `{"v":1,"pipeline":"main.13","branch":"main","source":12,"revision":13,"outcome":"tier-late"}`, wanted: true, acked: 1},
{name: "not json", body: `nope`, termd: 1}, {name: "not json", body: `nope`, termd: 1},
{name: "a run never taken in", body: `{"v":1,"run":"7","members":[]}`, err: fmt.Errorf("%w: no member", index.ErrUnreadable), termd: 1, started: "7"}, {name: "a run never taken in", body: `{"v":1,"run":"7","members":[]}`, err: fmt.Errorf("%w: no member", index.ErrUnreadable), termd: 1, started: "7"},
{name: "a bus fault", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}]}`, err: errors.New("the bus did not answer"), started: "7"}, {name: "a bus fault", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}]}`, err: errors.New("the bus did not answer"), started: "7"},
@@ -100,8 +114,8 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
switch { switch {
case row.wanted: case row.wanted:
heardPipelineWanted(context.Background(), ix, m, t.Logf) heardPipelineWanted(context.Background(), ix, m, t.Logf)
if ix.wanted != "main.12 tier-late" { if ix.wanted != "main.12.13 tier-late" { // its revision handed beside its source (H-A-214)
t.Errorf("%s: was handed %q, want main.12 tier-late", row.name, ix.wanted) t.Errorf("%s: was handed %q, want main.12.13 tier-late", row.name, ix.wanted)
} }
case row.ended: case row.ended:
heardPipelineEnded(context.Background(), ix, m, t.Logf) heardPipelineEnded(context.Background(), ix, m, t.Logf)
@@ -169,8 +183,11 @@ func (f *fakeRuns) HeardPins(context.Context, *meshpipelinerunner.PipelineStarte
return nil return nil
} }
func (f *fakeRuns) HeardPipelineWanted(_ context.Context, branch string, source int64, outcome string) error { func (f *fakeRuns) HeardPipelineWanted(_ context.Context, branch string, source, revision int64, outcome string) error {
f.wanted = fmt.Sprintf("%s.%d %s", branch, source, outcome) f.wanted = fmt.Sprintf("%s.%d %s", branch, source, outcome)
if revision > 0 {
f.wanted = fmt.Sprintf("%s.%d.%d %s", branch, source, revision, outcome)
}
return f.err return f.err
} }
@@ -21,7 +21,7 @@ type runTaker interface {
StartRun(ctx context.Context, e meshpipelinerunner.RunStarted) error StartRun(ctx context.Context, e meshpipelinerunner.RunStarted) error
EndRunPipeline(ctx context.Context, e meshpipelinerunner.PipelineEnded) error EndRunPipeline(ctx context.Context, e meshpipelinerunner.PipelineEnded) error
HeardPins(ctx context.Context, started *meshpipelinerunner.PipelineStarted, ended *meshpipelinerunner.PipelineEnded) error HeardPins(ctx context.Context, started *meshpipelinerunner.PipelineStarted, ended *meshpipelinerunner.PipelineEnded) error
HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error HeardPipelineWanted(ctx context.Context, branch string, source, revision int64, outcome string) error
JudgePins(ctx context.Context) JudgePins(ctx context.Context)
} }
@@ -99,7 +99,7 @@ func heardPipelineWanted(ctx context.Context, ix runTaker, m delivered, logf fun
_ = m.Term() _ = m.Term()
return return
} }
err := ix.HeardPipelineWanted(ctx, e.Branch, e.Source, e.Outcome) err := ix.HeardPipelineWanted(ctx, e.Branch, e.Source, e.Revision, e.Outcome)
settleRun(m, err, "pipeline-wanted of "+e.Pipeline, logf) settleRun(m, err, "pipeline-wanted of "+e.Pipeline, logf)
if err == nil { if err == nil {
ix.JudgePins(ctx) ix.JudgePins(ctx)
@@ -22,8 +22,9 @@ var (
) )
// heardVersion reads one message of the consumer as a version, false for any other subject. A package's is kept in its // heardVersion reads one message of the consumer as a version, false for any other subject. A package's is kept in its
// series; one that does not decode is said and acknowledged; a write that fails is left unacknowledged, so the bus gives // series; one that does not decode is said and acknowledged; a write that fails is left unacknowledged, so the bus gives it
// it again. // again. It recomputes no module: a fingerprint moves only with a kept catalog revision, a release's by the re-pin its
// package-published starts, one revision per module release (design 47 H-A-211, H-A-213 (2)).
func heardVersion(ctx context.Context, ix *index.Indexer, m jetstream.Msg, logf func(string, ...any)) bool { func heardVersion(ctx context.Context, ix *index.Indexer, m jetstream.Msg, logf func(string, ...any)) bool {
var h index.HeardVersion var h index.HeardVersion
switch seat, ok := publishedSeats[m.Subject()]; { switch seat, ok := publishedSeats[m.Subject()]; {
@@ -51,12 +52,6 @@ func heardVersion(ctx context.Context, ix *index.Indexer, m jetstream.Msg, logf
if kept { if kept {
logf("packages: %s %s kept in its series (published %v)", h.Package, h.Version, h.Published) logf("packages: %s %s kept in its series (published %v)", h.Package, h.Version, h.Published)
} }
if kept && h.Published {
// A series' newest moved: its users' fingerprints take it (DLV-I6), in the same turn as the publish is heard.
if err := ix.Recompute(ctx); err != nil {
logf("writing the modules again after %s %s was published: %v", h.Package, h.Version, err)
}
}
_ = m.Ack() _ = m.Ack()
return true return true
} }
@@ -0,0 +1,52 @@
package main
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/gopackageregistry"
"mesh-delivery-indexer/internal/index"
)
// ackedMsg is a message that counts its acknowledgements.
type ackedMsg struct {
jetstream.Msg
subject string
data []byte
acked int
}
func (a *ackedMsg) Subject() string { return a.subject }
func (a *ackedMsg) Data() []byte { return a.data }
func (a *ackedMsg) Ack() error { a.acked++; return nil }
// unlistedStore is a store whose listing fails: a recompute over it fails.
type unlistedStore struct{ memStore }
func (unlistedStore) Keys(context.Context) ([]string, error) {
return nil, errors.New("the bus does not answer")
}
// TestHeardVersionMovesNoFingerprint: a registry's published is kept in its series and acknowledged, and recomputes no
// module: a fingerprint moves only with a kept revision, a release's by its re-pin on the package-published (design 47
// H-A-211, H-A-213 (2); the pin specification's leftover L3). The modules state here fails any recompute.
func TestHeardVersionMovesNoFingerprint(t *testing.T) {
data, err := json.Marshal(gopackageregistry.PublishedEvent{Kind: "package", Name: "git.example/novox/lib", Version: "0.1.4"})
if err != nil {
t.Fatal(err)
}
ix := &index.Indexer{Packages: &memStore{vals: map[string][]byte{}}, Modules: &unlistedStore{memStore{vals: map[string][]byte{}}},
Manifests: &memStore{vals: map[string][]byte{}}}
m := &ackedMsg{subject: gopackageregistry.SubjectPublished, data: data}
if !heardVersion(context.Background(), ix, m, t.Logf) {
t.Fatal("the registry's published is not taken as a version")
}
if m.acked != 1 {
t.Errorf("acknowledged %d times, want once: a publish recomputes nothing that could fail", m.acked)
}
}
@@ -286,6 +286,14 @@ func (a actor) HeldAgain(ctx context.Context, branch string) index.TakenIn {
return r return r
} }
// TakeInAgain takes a repository in again at its kept merge commit (indexer_take_in_again, H-A-208, H-A-210), one take-in
// again at a time beside the held ones.
func (a actor) TakeInAgain(ctx context.Context, repository string) index.TakenIn {
a.s.again.Lock()
defer a.s.again.Unlock()
return a.Indexer.TakeInAgain(ctx, repository)
}
// HeldSkip ends a branch's held source revision unwritten (indexer_take_in_skip): take-in.taken says it skipped. // HeldSkip ends a branch's held source revision unwritten (indexer_take_in_skip): take-in.taken says it skipped.
func (a actor) HeldSkip(ctx context.Context, branch string, source int64, by, why string) index.TakenIn { func (a actor) HeldSkip(ctx context.Context, branch string, source int64, by, why string) index.TakenIn {
r := index.TakenIn{Module: branch} r := index.TakenIn{Module: branch}
@@ -7,6 +7,8 @@ import (
"reflect" "reflect"
"slices" "slices"
"strings" "strings"
"sync"
"sync/atomic"
"testing" "testing"
"time" "time"
@@ -172,25 +174,43 @@ func (f *fakeBranchJS) Consumer(_ context.Context, stream, name string) (jetstre
func (f *fakeBranchJS) CreateConsumer(_ context.Context, stream string, cfg jetstream.ConsumerConfig) (jetstream.Consumer, error) { func (f *fakeBranchJS) CreateConsumer(_ context.Context, stream string, cfg jetstream.ConsumerConfig) (jetstream.Consumer, error) {
f.made = append(f.made, cfg) f.made = append(f.made, cfg)
c := &fakeBranchConsumer{} c := newFakeBranchConsumer(stream, cfg.Durable)
f.held[stream+"/"+cfg.Durable] = c f.held[stream+"/"+cfg.Durable] = c
return c, nil return c, nil
} }
// fakeBranchConsumer hands one source-moved to the hearing as soon as it is consumed. // fakeBranchConsumer hands one source-moved to the hearing as soon as it is consumed. It answers its CachedInfo as the bus
// does, with its stream, name and created time, which the SDK's guard of one pull per durable keys by (design 47 H-A-196).
type fakeBranchConsumer struct { type fakeBranchConsumer struct {
jetstream.Consumer jetstream.Consumer
info jetstream.ConsumerInfo
heard int heard int
} }
type fakeConsumeContext struct{ jetstream.ConsumeContext } // fakeConsumersMade numbers the fakes, so each is created at a time of its own, as each durable the bus makes is.
var fakeConsumersMade atomic.Int64
func (fakeConsumeContext) Stop() {} func newFakeBranchConsumer(stream, durable string) *fakeBranchConsumer {
return &fakeBranchConsumer{info: jetstream.ConsumerInfo{Stream: stream, Name: durable, Config: jetstream.ConsumerConfig{Durable: durable},
Created: time.Unix(0, fakeConsumersMade.Add(1)).UTC()}}
}
func (c *fakeBranchConsumer) CachedInfo() *jetstream.ConsumerInfo { return &c.info }
// fakeConsumeContext is a consume that ends when stopped, as the guard's release waits for (H-A-196).
type fakeConsumeContext struct {
jetstream.ConsumeContext
stop sync.Once
closed chan struct{}
}
func (c *fakeConsumeContext) Stop() { c.stop.Do(func() { close(c.closed) }) }
func (c *fakeConsumeContext) Closed() <-chan struct{} { return c.closed }
func (c *fakeBranchConsumer) Consume(handle jetstream.MessageHandler, _ ...jetstream.PullConsumeOpt) (jetstream.ConsumeContext, error) { func (c *fakeBranchConsumer) Consume(handle jetstream.MessageHandler, _ ...jetstream.PullConsumeOpt) (jetstream.ConsumeContext, error) {
c.heard++ c.heard++
handle(msgOf{f: &fakeMsg{subject: "mesh.seat.mesh-delivery.event.source-moved.main.2"}}) handle(msgOf{f: &fakeMsg{subject: "mesh.seat.mesh-delivery.event.source-moved.main.2"}})
return fakeConsumeContext{}, nil return &fakeConsumeContext{closed: make(chan struct{})}, nil
} }
// Design 47 H §743, H-A-134: a branch's durable the bus does not hold the indexer makes, as a module's seat-event consumer is // Design 47 H §743, H-A-134: a branch's durable the bus does not hold the indexer makes, as a module's seat-event consumer is
@@ -207,7 +227,7 @@ func TestABranchDurableNotThereIsMadeAndHeard(t *testing.T) {
name := "n1_mesh-delivery-indexer_" + keys.EscapeName("main") name := "n1_mesh-delivery-indexer_" + keys.EscapeName("main")
var existing *fakeBranchConsumer var existing *fakeBranchConsumer
if tc.held { if tc.held {
existing = &fakeBranchConsumer{} existing = newFakeBranchConsumer(meshdelivery.StreamSourceMoved, name)
js.held[meshdelivery.StreamSourceMoved+"/"+name] = existing js.held[meshdelivery.StreamSourceMoved+"/"+name] = existing
} }
var heard []string var heard []string
@@ -24,6 +24,8 @@ type takerIn interface {
// HeldAgain and HeldSkip act on a watched branch's held source-moved (design 47 H §744). // HeldAgain and HeldSkip act on a watched branch's held source-moved (design 47 H §744).
HeldAgain(ctx context.Context, branch string) index.TakenIn HeldAgain(ctx context.Context, branch string) index.TakenIn
HeldSkip(ctx context.Context, branch string, source int64, by, why string) index.TakenIn HeldSkip(ctx context.Context, branch string, source int64, by, why string) index.TakenIn
// TakeInAgain takes a repository in again at its kept merge commit (H-A-208, H-A-210).
TakeInAgain(ctx context.Context, repository string) index.TakenIn
} }
// takenIn answers one value of the takeins state (a hand act, issue 1945): a value not yet answered is acted on now and // takenIn answers one value of the takeins state (a hand act, issue 1945): a value not yet answered is acted on now and
@@ -53,6 +55,9 @@ func takenIn(ctx context.Context, ix takerIn, kv jetstream.KeyValue, e jetstream
} else if a.Held != "" { } else if a.Held != "" {
asked, r = "take-in again of a held take-in", ix.HeldAgain(ctx, a.Held) asked, r = "take-in again of a held take-in", ix.HeldAgain(ctx, a.Held)
a.Module = a.Held a.Module = a.Held
} else if a.Again != "" {
asked, r = "take-in again of a repository", ix.TakeInAgain(ctx, a.Again)
a.Module = a.Again
} else if a.Registry != nil { } else if a.Registry != nil {
asked, r = "registrations' act "+a.Registry.Verb, ix.RegistryAct(ctx, a) asked, r = "registrations' act "+a.Registry.Verb, ix.RegistryAct(ctx, a)
} else if a.Recompute { } else if a.Recompute {
@@ -6,6 +6,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"os" "os"
"slices" "slices"
"strconv" "strconv"
@@ -447,37 +448,42 @@ func TestIndexerWhole(t *testing.T) {
// The first revision: no mover, no pin; its entries are the base. // The first revision: no mover, no pin; its entries are the base.
sayMoved(t, ctx, js, "main", 10, c10, true) sayMoved(t, ctx, js, "main", 10, c10, true)
if first := heard.next(t, "main", 10); len(first.Entries) != 0 || first.Revision != 10 { // A revision's number is the indexer's own count, the branch's newest + 1 (design 47 H-A-213 (1)): the later take-ins of
t.Fatalf("the first revision said %+v, want revision 10 with no entries", first) // sources 11 to 13 are counted on from the number of source 10's.
first := heard.next(t, "main", 10)
r := func(source int64) int64 { return first.Revision + source - 10 }
rn := func(source int64) string { return fmt.Sprintf("main.%d", r(source)) }
if len(first.Entries) != 0 || first.Revision != 2 || first.Source != 10 {
t.Fatalf("the first revision said %+v, want revision 2 (the bed's seed is main.1, of source 0) of source 10 with no entries", first)
} }
untilKey(t, ctx, pins, index.PinKey("main", 10), absent) untilKey(t, ctx, pins, index.PinKey("main", r(10)), absent)
// One take-in, one revision carrying its movers, pinned for its pipeline; said again from the kept movers. // One take-in, one revision carrying its movers, pinned for its pipeline; said again from the kept movers.
sayMoved(t, ctx, js, "main", 11, c11, false) sayMoved(t, ctx, js, "main", 11, c11, false)
taken := heard.next(t, "main", 11) taken := heard.next(t, "main", 11)
if len(taken.Entries) != 1 || taken.Entries[0].Module != "avahi" || taken.Entries[0].Commit != c11 || taken.Entries[0].Fingerprint == "" || if len(taken.Entries) != 1 || taken.Entries[0].Module != "avahi" || taken.Entries[0].Commit != c11 || taken.Entries[0].Fingerprint == "" ||
taken.Entries[0].Repository != "novox/mesh-catalog" || taken.Revision != 11 || taken.Call == "" { taken.Entries[0].Repository != "novox/mesh-catalog" || taken.Revision != r(11) || taken.Call == "" {
t.Fatalf("take-in.taken of main 11 = %+v, want avahi at %s as its one mover", taken, c11) t.Fatalf("take-in.taken of main 11 = %+v, want avahi at %s as its one mover", taken, c11)
} }
rev, _, err := index.ReadRevision(ctx, objects, "main.11") rev, _, err := index.ReadRevision(ctx, objects, rn(11))
if err != nil || len(rev.Movers) != 1 || rev.Movers[0] != taken.Entries[0] { if err != nil || len(rev.Movers) != 1 || rev.Movers[0] != taken.Entries[0] {
t.Fatalf("main.11 = %+v (%v), want it kept carrying the movers said", rev, err) t.Fatalf("main.11 = %+v (%v), want it kept carrying the movers said", rev, err)
} }
untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.11")) untilKey(t, ctx, pins, index.PinKey("main", r(11)), pinned("pipeline."+rn(11)))
untilKey(t, ctx, pins, index.PinKey("main", 10), pinned("pipeline.main.11")) // validate's base, H-A-147 untilKey(t, ctx, pins, index.PinKey("main", r(10)), pinned("pipeline."+rn(11))) // validate's base, H-A-147
info, err := objects.open(ctx) info, err := objects.open(ctx)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
before, err := info.GetInfo(ctx, "main.11") before, err := info.GetInfo(ctx, rn(11))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
sayMoved(t, ctx, js, "main", 11, c11, false) sayMoved(t, ctx, js, "main", 11, c11, false)
if again := heard.next(t, "main", 11); len(again.Entries) != 1 || again.Entries[0] != taken.Entries[0] || again.Revision != 11 { if again := heard.next(t, "main", 11); len(again.Entries) != 1 || again.Entries[0] != taken.Entries[0] || again.Revision != r(11) {
t.Errorf("the source-moved given again said %+v, want the kept movers %+v", again, taken.Entries) t.Errorf("the source-moved given again said %+v, want the kept movers %+v", again, taken.Entries)
} }
if after, err := info.GetInfo(ctx, "main.11"); err != nil || after.NUID != before.NUID { if after, err := info.GetInfo(ctx, rn(11)); err != nil || after.NUID != before.NUID {
t.Errorf("main.11 was written again (%v): a source-moved given again keeps no second revision", err) t.Errorf("main.11 was written again (%v): a source-moved given again keeps no second revision", err)
} }
@@ -485,11 +491,11 @@ func TestIndexerWhole(t *testing.T) {
// (H-A-147), which the take-in pinned for it. // (H-A-147), which the take-in pinned for it.
sayMoved(t, ctx, js, "main", 12, c12, false) sayMoved(t, ctx, js, "main", 12, c12, false)
heard.next(t, "main", 12) heard.next(t, "main", 12)
if err := pins.Delete(ctx, index.PinKey("main", 12)); err != nil { // the take-in's own pin, so pipeline-started alone writes it below if err := pins.Delete(ctx, index.PinKey("main", r(12))); err != nil { // the take-in's own pin, so pipeline-started alone writes it below
t.Fatal(err) t.Fatal(err)
} }
sayPipeline(t, ctx, js, false, "main", 12) sayPipeline(t, ctx, js, false, "main", r(12))
untilKey(t, ctx, pins, index.PinKey("main", 12), pinned("pipeline.main.12")) untilKey(t, ctx, pins, index.PinKey("main", r(12)), pinned("pipeline."+rn(12)))
// A run on request: its candidate and its key written on run-started, from avahi's files at the run's commit (A-160, A-174). // A run on request: its candidate and its key written on run-started, from avahi's files at the run's commit (A-160, A-174).
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json") manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil { if err != nil {
@@ -515,7 +521,7 @@ func TestIndexerWhole(t *testing.T) {
heard.acked(t, "main") heard.acked(t, "main")
} }
restart() restart()
for _, name := range []string{"main.10", "main.11"} { for _, name := range []string{rn(10), rn(11)} {
if _, _, err := index.ReadRevision(ctx, objects, name); err != nil { if _, _, err := index.ReadRevision(ctx, objects, name); err != nil {
t.Fatalf("%s is not kept across a restart while main.11's pipeline runs: %v", name, err) t.Fatalf("%s is not kept across a restart while main.11's pipeline runs: %v", name, err)
} }
@@ -553,27 +559,27 @@ func TestIndexerWhole(t *testing.T) {
t.Errorf("the held source-moved's consumer %+v (%v), want nothing unacknowledged", info, err) t.Errorf("the held source-moved's consumer %+v (%v), want nothing unacknowledged", info, err)
} }
setCap(t, ctx, js, 1<<30) // a larger cap alone wakes nothing: no timer takes a held take-in in setCap(t, ctx, js, 1<<30) // a larger cap alone wakes nothing: no timer takes a held take-in in
sayPipeline(t, ctx, js, true, "main", 11) sayPipeline(t, ctx, js, true, "main", r(11))
if got := heard.next(t, "main", 13); len(got.Entries) != 1 || got.Entries[0].Commit != c13 { if got := heard.next(t, "main", 13); len(got.Entries) != 1 || got.Entries[0].Commit != c13 {
t.Errorf("main 13 taken in on room said %+v, want avahi at %s", got, c13) t.Errorf("main 13 taken in on room said %+v, want avahi at %s", got, c13)
} }
untilKey(t, ctx, held, keys.EscapeName("main"), absent) untilKey(t, ctx, held, keys.EscapeName("main"), absent)
// main.12's validate base stays (H-A-147), and main.11's passed pipeline's release pins it (H-A-168, H-A-170). // main.12's validate base stays (H-A-147), and main.11's passed pipeline's release pins it (H-A-168, H-A-170).
untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.12", index.ReleaseRecord("main", "avahi"))) untilKey(t, ctx, pins, index.PinKey("main", r(11)), pinned("pipeline."+rn(12), index.ReleaseRecord("main", "avahi")))
untilRevisionGone(t, ctx, js, "main.10") // the end freed both of main.11's pins untilRevisionGone(t, ctx, js, rn(10)) // the end freed both of main.11's pins
// main.12's passed pipeline releases avahi again; its pipeline-wanted, the seat mesh-wants' (its wants written to the end), // main.12's passed pipeline releases avahi again; its pipeline-wanted, the seat mesh-wants' (its wants written to the end),
// frees main.11's older release, and main.11 goes (H-A-168, H-A-170, H-A-173). // frees main.11's older release, and main.11 goes (H-A-168, H-A-170, H-A-173).
sayPipeline(t, ctx, js, true, "main", 12) sayPipeline(t, ctx, js, true, "main", r(12))
untilKey(t, ctx, pins, index.PinKey("main", 12), func(v []byte) bool { untilKey(t, ctx, pins, index.PinKey("main", r(12)), func(v []byte) bool {
var p index.PinValue var p index.PinValue
return v != nil && json.Unmarshal(v, &p) == nil && slices.Contains(p.Records, index.ReleaseRecord("main", "avahi")) return v != nil && json.Unmarshal(v, &p) == nil && slices.Contains(p.Records, index.ReleaseRecord("main", "avahi"))
}) })
wantedRaw, _ := json.Marshal(meshwants.PipelineWanted{V: 1, Pipeline: "main.12", Branch: "main", Source: 12, Outcome: meshwants.PipelineWantedOutcomeRunning}) wantedRaw, _ := json.Marshal(meshwants.PipelineWanted{V: 1, Pipeline: rn(12), Branch: "main", Source: 12, Outcome: meshwants.PipelineWantedOutcomeRunning})
if _, err := js.Publish(ctx, meshwants.SubjectPipelineWanted("main", "12"), wantedRaw); err != nil { if _, err := js.Publish(ctx, meshwants.SubjectPipelineWanted("main", "12"), wantedRaw); err != nil {
t.Fatal(err) t.Fatal(err)
} }
untilKey(t, ctx, pins, index.PinKey("main", 11), absent) untilKey(t, ctx, pins, index.PinKey("main", r(11)), absent)
untilRevisionGone(t, ctx, js, "main.11") untilRevisionGone(t, ctx, js, rn(11))
if n := len(heard.pending); n != 0 { if n := len(heard.pending); n != 0 {
t.Errorf("take-in.pending said %d more times for one held source revision, want once", n) t.Errorf("take-in.pending said %d more times for one held source revision, want once", n)
} }
@@ -0,0 +1,147 @@
package main
import (
"context"
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-indexer/internal/index"
)
// askedBy is a takerIn that names the hand act it was asked.
type askedBy struct{ asked []string }
func (s *askedBy) did(act string) index.TakenIn {
s.asked = append(s.asked, act)
return index.TakenIn{Module: "m", Stored: true, Answer: act}
}
func (s *askedBy) SeatDefinitionsAgain(context.Context, string) index.TakenIn {
return s.did("seat-definitions")
}
func (s *askedBy) FingerprintsAgain(context.Context, string) index.TakenIn {
return s.did("fingerprints")
}
func (s *askedBy) RecomputeByHand(context.Context, string, string) index.TakenIn {
return s.did("recompute")
}
func (s *askedBy) DropCandidateGraph(context.Context, string, string, string, string) index.TakenIn {
return s.did("candidate-group")
}
func (s *askedBy) NamedDrop(context.Context, string, string, string, string) index.TakenIn {
return s.did("named-fingerprint")
}
func (s *askedBy) PinDrop(context.Context, string, string, string) index.TakenIn {
return s.did("pin-drop")
}
func (s *askedBy) DropRunByHand(context.Context, string, string, string) index.TakenIn {
return s.did("run-drop")
}
func (s *askedBy) RegistryAct(context.Context, index.TakeInAsk) index.TakenIn {
return s.did("registry")
}
func (s *askedBy) HeldAgain(context.Context, string) index.TakenIn { return s.did("held") }
func (s *askedBy) HeldSkip(context.Context, string, int64, string, string) index.TakenIn {
return s.did("held-skip")
}
func (s *askedBy) TakeInAgain(context.Context, string) index.TakenIn { return s.did("again") }
// answeredKV keeps the answer takenIn writes back.
type answeredKV struct {
jetstream.KeyValue
written []byte
}
func (k *answeredKV) Update(_ context.Context, _ string, value []byte, _ uint64) (uint64, error) {
k.written = value
return 2, nil
}
type askEntry struct {
jetstream.KeyValueEntry
key string
value []byte
}
func (e askEntry) Key() string { return e.key }
func (e askEntry) Value() []byte { return e.value }
func (e askEntry) Revision() uint64 { return 1 }
func (e askEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
// Design 47 H-A-208, H-A-210, H §774: every hand tool's takeins value, built by the tool's own Run path (or the constructor it
// calls), is read by DecodeTakeIn and acted on by takenIn as the act it names; indexer_take_in_again's was refused, its Again
// missing from a hand-kept list. Every field of TakeInAsk says by its tag whether it is an act, so an act added is read.
func TestEveryHandToolsAskIsReadAndActedOn(t *testing.T) {
t.Run("every field is classed", func(t *testing.T) {
ty := reflect.TypeFor[index.TakeInAsk]()
for i := range ty.NumField() {
if c := ty.Field(i).Tag.Get("takein"); !slices.Contains([]string{"act", "argument", "ask"}, c) {
t.Errorf("TakeInAsk.%s has takein %q, want act, argument or ask", ty.Field(i).Name, c)
}
}
})
var values [][]byte
ask := func(_ string, value []byte) (index.TakeInAsk, error) {
values = append(values, value)
return index.TakeInAsk{V: 1, Result: &index.TakenIn{Stored: true}, Answered: time.Now()}, nil
}
record := func(string, string) error { return nil }
tools := map[string]func(map[string]any) (any, error){}
for _, tl := range slices.Concat(index.HeldTools(nil, ask, time.Now, record), index.PinsTools(nil, nil, ask, time.Now, record),
index.RunsTools(nil, nil, ask, time.Now, record), index.NamedTools(ask, time.Now, record),
index.FingerprintsTools(ask, time.Now, record, time.Minute), index.CatalogRevisionTools(nil, ask, time.Now, record),
index.RegistrationTools(nil, nil, ask, time.Now, func(string, string, string, string) error { return nil })) {
tools[tl.Name] = tl.Run
}
tools[index.SeatDefinitionsAgainTool] = func(args map[string]any) (any, error) { // its tool lives in the tools bundle
_, value, err := index.NewSeatDefinitionsAgain(args["module"].(string), "why", "by", time.Now())
values = append(values, value)
return nil, err
}
for _, c := range []struct {
tool string
args map[string]any
act string
}{
{index.TakeInAgainTool, map[string]any{"repository": "novox/mesh-host", "why": "rule changed"}, "again"},
{index.TakeInPendingTool, map[string]any{"branch": "main"}, "held"},
{index.TakeInSkipTool, map[string]any{"branch": "main", "source": float64(4), "why": "stale"}, "held-skip"},
{index.PinDropTool, map[string]any{"key": "main.7", "why": "no pipeline"}, "pin-drop"},
{index.RunDropTool, map[string]any{"run": "run-12", "why": "never ends"}, "run-drop"},
{index.NamedDropTool, map[string]any{"fingerprint": "abc", "member": "ace.feh", "why": "gone"}, "named-fingerprint"},
{index.FingerprintsAgainTool, map[string]any{"module": "feh", "why": "unread packages"}, "fingerprints"},
{index.RecomputeTool, map[string]any{"why": "check"}, "recompute"},
{index.DropCandidateGraphTool, map[string]any{"group": "g", "heads": "h", "why": "no gate event"}, "candidate-group"},
{index.SeedAgainTool, map[string]any{"why": "empty"}, "registry"},
{index.SeatDefinitionsAgainTool, map[string]any{"module": "mesh-contracts"}, "seat-definitions"},
} {
t.Run(c.tool, func(t *testing.T) {
run, ok := tools[c.tool]
if !ok {
t.Fatalf("no tool %s", c.tool)
}
values = nil
if _, err := run(c.args); err != nil || len(values) != 1 {
t.Fatalf("%s wrote %d values (%v)", c.tool, len(values), err)
}
if _, err := index.DecodeTakeIn(values[0]); err != nil {
t.Fatalf("DecodeTakeIn refuses %s: %v", values[0], err)
}
ix, kv := &askedBy{}, &answeredKV{}
var said []string
takenIn(t.Context(), ix, kv, askEntry{key: "k", value: values[0]}, func(f string, a ...any) { said = append(said, f) })
if !slices.Equal(ix.asked, []string{c.act}) {
t.Errorf("takenIn asked %v of %s, want %s (said %s)", ix.asked, values[0], c.act, strings.Join(said, "; "))
}
var answered index.TakeInAsk
if err := json.Unmarshal(kv.written, &answered); err != nil || !answered.Done() {
t.Errorf("takenIn wrote back %s (%v), want the answered ask", kv.written, err)
}
})
}
}
@@ -39,7 +39,9 @@ func checkCandidateModule(m meshcatalog.CandidatesByGroupValuePullsItemFpsValue)
return fmt.Errorf("index: the named path %q names no git object", p) return fmt.Errorf("index: the named path %q names no git object", p)
} }
} }
if err := sdkstep.PackageVersions(m.Packages).Check(); err != nil { // An entry names a package with no published version yet with an empty version (design 47 H S8, H-A-204): it is checked
// as an entry's packages, not as an ask's.
if err := sdkstep.PackageVersions(m.Packages).CheckEntry(); err != nil {
return err return err
} }
if m.Fp2 != "" { if m.Fp2 != "" {
@@ -0,0 +1,30 @@
package index
import (
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// TestCheckCandidateModulePins: an entry's package with no published version (an empty pin) passes a run on request's
// candidate; a pin that is no version is refused (design 47 H S8, the pin specification's item 4).
func TestCheckCandidateModulePins(t *testing.T) {
for _, c := range []struct {
name string
packages map[string]string
ok bool
}{
{"an empty pin", map[string]string{"git.example/novox/lib/go": ""}, true},
{"a version", map[string]string{"git.example/novox/lib/go": "v0.1.4"}, true},
{"no version", map[string]string{"git.example/novox/lib/go": "latest"}, false},
} {
t.Run(c.name, func(t *testing.T) {
m := meshcatalog.CandidatesByGroupValuePullsItemFpsValue{Fp2: "src2:" + strings.Repeat("ab", 32), Packages: c.packages}
err := checkCandidateModule(m)
if (err == nil) != c.ok {
t.Errorf("checkCandidateModule(%v) = %v, want ok %v", c.packages, err, c.ok)
}
})
}
}
@@ -156,6 +156,7 @@ type Indexer struct {
filling atomic.Bool // a FillTaken runs in the background (KickFill) filling atomic.Bool // a FillTaken runs in the background (KickFill)
fillAgain atomic.Bool // a kick arrived while it ran fillAgain atomic.Bool // a kick arrived while it ran
underSeats bool // a dry run inside judge.WithSeats: its judge passes do not hold the seat list (verdicts.go) underSeats bool // a dry run inside judge.WithSeats: its judge passes do not hold the seat list (verdicts.go)
isDry bool // a dry run (dryrun.go): it shares the live packages state, so it drops no kept package key (packages.go)
judgeMu sync.Mutex // one judge pass at a time (verdicts.go) judgeMu sync.Mutex // one judge pass at a time (verdicts.go)
keepMu sync.Mutex // one catalog graph revision kept at a time (keepEntries) keepMu sync.Mutex // one catalog graph revision kept at a time (keepEntries)
// Pins is the state pins (pins.go); nil keeps none. PinsPending says how many messages the indexer's durable consumer of the // Pins is the state pins (pins.go); nil keeps none. PinsPending says how many messages the indexer's durable consumer of the
@@ -36,13 +36,13 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
for _, n := range names { for _, n := range names {
have[n] = true have[n] = true
} }
for _, want := range []string{"main.5", "feature/x=2ey.7", "feature/x=2ey.8"} { for _, want := range []string{"main.1", "feature/x=2ey.1", "feature/x=2ey.2"} {
if !have[want] { if !have[want] {
t.Errorf("the objects hold %v, want %s", names, want) t.Errorf("the objects hold %v, want %s", names, want)
} }
} }
if len(said) != 3 || said[0].Branch != "main" || said[1].Branch != "feature/x=2ey" || said[2].Revision != 8 { if len(said) != 3 || said[0].Branch != "main" || said[1].Branch != "feature/x=2ey" || said[2].Revision != 2 || said[2].Source != 8 {
t.Errorf("said %+v, want main 5, then the escaped branch 7 and 8", said) t.Errorf("said %+v, want main 1 of source 5, then the escaped branch 1 and 2 of sources 7 and 8", said)
} }
answer := func(branch string, revision int64) meshcatalog.RevisionAnswer { answer := func(branch string, revision int64) meshcatalog.RevisionAnswer {
a, err := Revision(ctx, ix.Revisions, nil, meshcatalog.RevisionInput{Branch: branch, Revision: revision}) a, err := Revision(ctx, ix.Revisions, nil, meshcatalog.RevisionInput{Branch: branch, Revision: revision})
@@ -51,14 +51,14 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
} }
return a return a
} }
if a := answer("main", 0); a.Revision.Revision != 5 || a.Entries[0].Commit != "c1" { if a := answer("main", 0); a.Revision.Revision != 1 || a.Revision.Source != 5 || a.Entries[0].Commit != "c1" {
t.Errorf("main answered %+v", a) t.Errorf("main answered %+v", a)
} }
if a := answer("feature/x=2ey", 0); a.Revision.Revision != 8 || a.Revision.Branch != "feature/x.y" || a.Entries[0].Commit != "c3" { if a := answer("feature/x=2ey", 0); a.Revision.Revision != 2 || a.Revision.Branch != "feature/x.y" || a.Entries[0].Commit != "c3" {
t.Errorf("the branch's newest answered %+v, want its revision 8 at c3", a.Revision) t.Errorf("the branch's newest answered %+v, want its revision 2 at c3", a.Revision)
} }
if a := answer("feature/x=2ey", 7); a.Revision.Revision != 7 || a.Entries[0].Commit != "c2" { if a := answer("feature/x=2ey", 1); a.Revision.Revision != 1 || a.Entries[0].Commit != "c2" {
t.Errorf("the branch's revision 7 answered %+v", a.Revision) t.Errorf("the branch's revision 1 answered %+v", a.Revision)
} }
if a := answer("beta", 0); !a.NoneYet { if a := answer("beta", 0); !a.NoneYet {
t.Errorf("a branch with no revision answered %+v, want none-yet", a) t.Errorf("a branch with no revision answered %+v, want none-yet", a)
@@ -68,15 +68,22 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
bySource := func(branch string, source int64) (meshcatalog.RevisionAnswer, error) { bySource := func(branch string, source int64) (meshcatalog.RevisionAnswer, error) {
return Revision(ctx, ix.Revisions, nil, meshcatalog.RevisionInput{Branch: branch, Source: source}) return Revision(ctx, ix.Revisions, nil, meshcatalog.RevisionInput{Branch: branch, Source: source})
} }
if a, err := bySource("main", 5); err != nil || a.Revision.Revision != 5 || a.Entries[0].Commit != "c1" { if a, err := bySource("main", 5); err != nil || a.Revision.Revision != 1 || a.Revision.Source != 5 || a.Entries[0].Commit != "c1" {
t.Errorf("main's source 5 answered %+v (%v)", a.Revision, err) t.Errorf("main's source 5 answered %+v (%v)", a.Revision, err)
} }
if a, err := bySource("feature/x=2ey", 7); err != nil || a.Revision.Revision != 7 || a.Entries[0].Commit != "c2" { if a, err := bySource("feature/x=2ey", 7); err != nil || a.Revision.Revision != 1 || a.Entries[0].Commit != "c2" {
t.Errorf("the branch's source 7 answered %+v (%v), want its revision 7 at c2", a.Revision, err) t.Errorf("the branch's source 7 answered %+v (%v), want its revision 1 at c2", a.Revision, err)
} }
if a, err := bySource("feature/x=2ey", 8); err != nil || a.Revision.Revision != 8 { if a, err := bySource("feature/x=2ey", 8); err != nil || a.Revision.Revision != 2 {
t.Errorf("the branch's source 8 answered %+v (%v)", a.Revision, err) t.Errorf("the branch's source 8 answered %+v (%v)", a.Revision, err)
} }
// H-A-213: a re-pin's revision of source 8 answers below the take-in's: source 8 answers revision 2.
if _, err := ix.keepBranchEntries(ctx, "feature/x.y", 8, releaseCall("mesh-sdk", "0.1.27"), tip("c4"), nil); err != nil {
t.Fatal(err)
}
if a, err := bySource("feature/x=2ey", 8); err != nil || a.Revision.Revision != 2 || a.Entries[0].Commit != "c3" {
t.Errorf("the branch's source 8 after a re-pin answered %+v (%v), want the take-in's revision 2", a.Revision, err)
}
if _, err := bySource("feature/x=2ey", 9); err == nil { if _, err := bySource("feature/x=2ey", 9); err == nil {
t.Error("a source no revision was taken in from was answered") t.Error("a source no revision was taken in from was answered")
} }
@@ -9,6 +9,7 @@ import (
"reflect" "reflect"
"slices" "slices"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph"
"git.novox.be/novox/mesh-sdk/go/catalogue" "git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -30,7 +31,7 @@ const CatalogEntryLayout = 1
// ModuleValueOf gives the module's value back from the entry alone. The acting rules are the revision's to add. // ModuleValueOf gives the module's value back from the entry alone. The acting rules are the revision's to add.
func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catalogue.Manifest, raw string, n graph.Node) meshcatalog.CatalogEntry { func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catalogue.Manifest, raw string, n graph.Node) meshcatalog.CatalogEntry {
e := meshcatalog.CatalogEntry{V: CatalogEntryLayout, Module: module, Repository: cur.Repository, Path: cur.Path, e := meshcatalog.CatalogEntry{V: CatalogEntryLayout, Module: module, Repository: cur.Repository, Path: cur.Path,
Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalog.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases, Commit: cur.Commit, Manifest: manifestWhole(raw), ManifestDigest: meshcatalogclient.EntryManifestDigest(cur.Manifest), Verdict: cur.Verdict, Fp: n.FP, None: n.None, Bases: n.Bases,
Packages: maps.Clone(cur.Packages), Provides: n.Provides, Requires: n.Requires, Awaits: n.Awaits, Invokes: n.Invokes, Packages: maps.Clone(cur.Packages), Provides: n.Provides, Requires: n.Requires, Awaits: n.Awaits, Invokes: n.Invokes,
Serves: n.Serves, LeftOut: n.LeftOut, Main: MainOf(m), Contributes: contributesOf(m)} Serves: n.Serves, LeftOut: n.LeftOut, Main: MainOf(m), Contributes: contributesOf(m)}
// requires names every requirement as catalogue.Manifest.Wants lists them, optional ones and those only contributes names // requires names every requirement as catalogue.Manifest.Wants lists them, optional ones and those only contributes names
@@ -67,7 +68,7 @@ func CatalogEntryOf(module string, cur meshcatalog.ModulesByModuleValue, m catal
// ruling on milestone 47/G finding 1; ADR 0656 §6: one form of the fact). The old line's fields (the old-form // ruling on milestone 47/G finding 1; ADR 0656 §6: one form of the fact). The old line's fields (the old-form
// fingerprint, the reader needs) are not kept. // fingerprint, the reader needs) are not kept.
func ModuleValueOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue { func ModuleValueOf(e meshcatalog.CatalogEntry) meshcatalog.ModulesByModuleValue {
v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalog.BareManifestDigest(e.ManifestDigest), v := meshcatalog.ModulesByModuleValue{V: keys.Version, Repository: e.Repository, Path: e.Path, Commit: e.Commit, Manifest: meshcatalogclient.BareManifestDigest(e.ManifestDigest),
Verdict: e.Verdict, Fp2: e.Fp, None2: e.None, Packages: maps.Clone(e.Packages)} Verdict: e.Verdict, Fp2: e.Fp, None2: e.None, Packages: maps.Clone(e.Packages)}
if in := e.Inputs; in != nil { if in := e.Inputs; in != nil {
v.Toolchains, v.Paths, v.Problems, v.Warnings, v.Unknown = maps.Clone(in.Toolchains), maps.Clone(in.Paths), in.Problems, in.Warnings, in.Unknown v.Toolchains, v.Paths, v.Problems, v.Warnings, v.Unknown = maps.Clone(in.Toolchains), maps.Clone(in.Paths), in.Problems, in.Warnings, in.Unknown
@@ -5,6 +5,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"maps"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -87,6 +88,37 @@ func BranchRevisionOf(branch, name string) (int64, bool) {
return r, err == nil && r > 0 return r, err == nil && r > 0
} }
// watchedBranches are the watched branches the bucket keeps a revision of, the first included when it keeps one, sorted:
// each name <branch>.<n> but an entry's, a candidate's and the earlier forms' (earlierForm).
func (ix *Indexer) watchedBranches(ctx context.Context) ([]string, error) {
if ix.Revisions == nil {
return nil, nil
}
names, err := ix.Revisions.Names(ctx)
if err != nil {
return nil, fmt.Errorf("the objects %s: %w", CatalogRevisionsObjects, err)
}
seen := map[string]bool{}
for _, n := range names {
if strings.HasPrefix(n, "entry.") || strings.HasPrefix(n, "candidate.") || earlierForm(n) {
continue
}
i := strings.LastIndexByte(n, '.')
if i <= 0 {
continue
}
if r, err := strconv.ParseInt(n[i+1:], 10, 64); err != nil || r <= 0 {
continue
}
branch, err := keys.UnescapeName(n[:i])
if err != nil {
continue
}
seen[branch] = true
}
return slices.Sorted(maps.Keys(seen)), nil
}
// TrunkRevisionOf is the revision number a trunk object's name holds, and whether it is one. // TrunkRevisionOf is the revision number a trunk object's name holds, and whether it is one.
func TrunkRevisionOf(name string) (int64, bool) { return BranchRevisionOf(TrunkBranch, name) } func TrunkRevisionOf(name string) (int64, bool) { return BranchRevisionOf(TrunkBranch, name) }
@@ -210,6 +242,54 @@ func readNewestOf(ctx context.Context, o Objects, branch string) (meshcatalog.Ca
} }
} }
// TrunkEntryAt is a module's entry in a trunk revision, with that revision's list (H-A-204): for revision 0 the newest trunk
// revision read as entryOf reads it (readNewestOf); else the kept list of that trunk revision and only the module's entry
// object, so one entry is read without the revision whole.
func TrunkEntryAt(ctx context.Context, o Objects, module string, revision int64) (meshcatalog.CatalogRevision, meshcatalog.CatalogEntry, error) {
if revision == 0 {
rev, entries, found, err := readNewestOf(ctx, o, TrunkBranch)
if err != nil {
return rev, meshcatalog.CatalogEntry{}, err
}
if !found {
return rev, meshcatalog.CatalogEntry{}, errors.New("the indexer keeps no trunk revision yet")
}
for _, e := range entries {
if e.Module == module {
return rev, e, nil
}
}
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("the trunk revision %d holds no module %s", rev.Revision, module)
}
name := BranchObject(TrunkBranch, revision)
var rev meshcatalog.CatalogRevision
data, err := o.Get(ctx, name)
if errors.Is(err, ErrNoObject) {
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("the trunk revision %d is not kept", revision)
}
if err != nil {
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("revision %s: %w", name, err)
}
if err := json.Unmarshal(data, &rev); err != nil {
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("revision %s: %w", name, err)
}
for _, ref := range rev.Entries {
if ref.Module != module {
continue
}
data, err := o.Get(ctx, EntryObject(ref.Digest))
if err != nil {
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("revision %s, the entry of %s (%s): %w", name, module, ref.Digest, err)
}
var e meshcatalog.CatalogEntry
if err := json.Unmarshal(data, &e); err != nil {
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("revision %s, the entry of %s: %w", name, module, err)
}
return rev, e, nil
}
return rev, meshcatalog.CatalogEntry{}, fmt.Errorf("the trunk revision %d holds no module %s", revision, module)
}
// changedEntries are the modules whose entry differs between two revisions' lists, one gone or added included, by name. // changedEntries are the modules whose entry differs between two revisions' lists, one gone or added included, by name.
func changedEntries(before, after []meshcatalog.EntryRef) []string { func changedEntries(before, after []meshcatalog.EntryRef) []string {
was := map[string]string{} was := map[string]string{}
@@ -266,17 +346,22 @@ func (ix *Indexer) viewEntries(ctx context.Context) (map[string]meshcatalog.Cata
} }
// KeepBranchEntries keeps entries as the catalog revision of a watched branch taken in from its source revision src (design // KeepBranchEntries keeps entries as the catalog revision of a watched branch taken in from its source revision src (design
// 47 H §743, H-A-32): <branch>.<src>, the trunk's included, numbered by the source, and says catalog-revision naming the // 47 H §743, H-A-32), as keepBranchEntries does with no call.
// branch. movers are the take-in's: kept on the revision (H-A-92), and the revision pinned for the take-in's pipeline,
// pipeline.<branch>.<src>, in the same write when there is one (A-449a), with the branch's previous revision (H-A-147). A revision already kept under that name is not written
// again (read before write, under the branch's one consumer); ErrObjectsFull is a store too full to keep it.
func (ix *Indexer) KeepBranchEntries(ctx context.Context, branch string, src int64, entries map[string]meshcatalog.CatalogEntry, movers []meshcatalog.TakeInEntry) error { func (ix *Indexer) KeepBranchEntries(ctx context.Context, branch string, src int64, entries map[string]meshcatalog.CatalogEntry, movers []meshcatalog.TakeInEntry) error {
return ix.keepBranchEntries(ctx, branch, src, entries, movers) _, err := ix.keepBranchEntries(ctx, branch, src, revisionCall{}, entries, movers)
return err
} }
func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int64, entries map[string]meshcatalog.CatalogEntry, movers []meshcatalog.TakeInEntry) error { // keepBranchEntries keeps entries as the next catalog revision of a watched branch, <branch>.<n>, the trunk's included: n is
// the indexer's own count, the branch's newest revision + 1, and src the gate's source revision it was read at, its field
// source (design 47 H-A-213 (1)); call is what made it (H-A-212, H-A-213 (3)): a take-in's the gate's call, a re-pin's
// "release of <module> <version>", a take-in again's "take-in again of <repository>". It says catalog-revision naming the
// branch, its source and its call. movers are kept on the revision (H-A-92), and the revision pinned for its one pipeline,
// pipeline.<branch>.<n>, in the same write when there is one (A-449a, H-A-213 (4)), with the branch's previous revision
// (H-A-147). It answers the number it kept; ErrObjectsFull is a store too full to keep it.
func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int64, call revisionCall, entries map[string]meshcatalog.CatalogEntry, movers []meshcatalog.TakeInEntry) (int64, error) {
if ix.Revisions == nil { if ix.Revisions == nil {
return nil return 0, nil
} }
if branch == "" { if branch == "" {
branch = TrunkBranch branch = TrunkBranch
@@ -286,28 +371,24 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
newest, _, err := newestOf(ctx, ix.Revisions, branch) newest, _, err := newestOf(ctx, ix.Revisions, branch)
if errors.Is(err, process.ErrNotYet) { if errors.Is(err, process.ErrNotYet) {
ix.logf("catalog graph: no revision is kept: its objects %s are not on the bus yet", CatalogRevisionsObjects) ix.logf("catalog graph: no revision is kept: its objects %s are not on the bus yet", CatalogRevisionsObjects)
return err return 0, err
} }
if err != nil { if err != nil {
ix.logf("catalog graph: no revision is kept: the newest is not read: %v", err) ix.logf("catalog graph: no revision is kept: the newest is not read: %v", err)
return fmt.Errorf("the newest catalog graph revision: %w", err) return 0, fmt.Errorf("the newest catalog graph revision: %w", err)
}
if _, err := ix.Revisions.Get(ctx, BranchObject(branch, src)); err == nil {
ix.logf("catalog graph: revision %s is kept already: not written again", BranchObject(branch, src))
return nil
} else if !errors.Is(err, ErrNoObject) {
return fmt.Errorf("revision %s: %w", BranchObject(branch, src), err)
} }
number := newest.Revision + 1
entries = WithEdges(ix.withActs(ctx, entries)) // the acting rules and the edges between artifact references are written here, once, for every reader (A-208) entries = WithEdges(ix.withActs(ctx, entries)) // the acting rules and the edges between artifact references are written here, once, for every reader (A-208)
at := ix.now().UTC() at := ix.now().UTC()
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: src, Kind: "trunk", Branch: branch, Source: src, At: &at}, entries) rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: number, Kind: "trunk", Branch: branch, Source: src, Call: call.Call,
CallKind: call.Kind, Module: call.Module, Version: call.Version, Repository: call.Repository, At: &at}, entries)
if err != nil { if err != nil {
return err return 0, err
} }
rev.Movers = movers rev.Movers = movers
// The base of the validate step's case before: the newest revision kept before this write, which the take-in pins for its // The base of the validate step's case before: the newest revision kept before this write, which the take-in pins for its
// pipeline below, named on the revision in this same write (H-A-147, H-A-151, H-A-154); none on the branch's first take-in. // pipeline below, named on the revision in this same write (H-A-147, H-A-151, H-A-154); none on the branch's first take-in.
pinsBase := len(movers) > 0 && ix.Pins != nil && newest.Revision > 0 && newest.Revision != src pinsBase := len(movers) > 0 && ix.Pins != nil && newest.Revision > 0
if pinsBase { if pinsBase {
rev.Base = newest.Revision rev.Base = newest.Revision
} }
@@ -319,32 +400,32 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
refused := fmt.Sprintf("the next revision; source revision %d held pending with the reason %s", src, HeldObjectsFull) refused := fmt.Sprintf("the next revision; source revision %d held pending with the reason %s", src, HeldObjectsFull)
ix.logf("catalog graph: revision %d of source revision %d is not kept: %s is full (%d of %d bytes)", rev.Revision, src, CatalogRevisionsObjects, used, capacity) ix.logf("catalog graph: revision %d of source revision %d is not kept: %s is full (%d of %d bytes)", rev.Revision, src, CatalogRevisionsObjects, used, capacity)
ix.sayFill(ctx, meshcatalog.EventFull, fmt.Sprintf("%d", src), meshcatalog.Fill{Bytes: used, Cap: capacity, Refused: refused}) ix.sayFill(ctx, meshcatalog.EventFull, fmt.Sprintf("%d", src), meshcatalog.Fill{Bytes: used, Cap: capacity, Refused: refused})
return fmt.Errorf("revision %d of source revision %d: %w", rev.Revision, src, ErrObjectsFull) return 0, fmt.Errorf("revision %d of source revision %d: %w", rev.Revision, src, ErrObjectsFull)
} }
if err := ix.indexEntries(ctx, rev, entries); err != nil { if err := ix.indexEntries(ctx, rev, entries); err != nil {
ix.logf("catalog graph: revision %d of source revision %d is not kept: %v", rev.Revision, src, err) ix.logf("catalog graph: revision %d of source revision %d is not kept: %v", rev.Revision, src, err)
return err return 0, err
} }
if err := KeepRevision(ctx, ix.Revisions, rev, objects); err != nil { if err := KeepRevision(ctx, ix.Revisions, rev, objects); err != nil {
ix.logf("catalog graph: revision %d of source revision %d is not kept: %v", rev.Revision, src, err) ix.logf("catalog graph: revision %d of source revision %d is not kept: %v", rev.Revision, src, err)
return err return 0, err
} }
// The take-in's revision pinned for its one pipeline before any judge can remove it (§743, §749, A-449a). // The take-in's revision pinned for its one pipeline before any judge can remove it (§743, §749, A-449a).
if len(movers) > 0 && ix.Pins != nil { if len(movers) > 0 && ix.Pins != nil {
record := PipelineRecord(runner.PipelinesByBranchSourceKey(keys.EscapeName(branch), strconv.FormatInt(src, 10))) record := PipelineRecord(runner.PipelinesByBranchRevisionKey(keys.EscapeName(branch), strconv.FormatInt(number, 10)))
if err := ix.pin(ctx, PinKey(branch, src), record); err != nil { if err := ix.pin(ctx, PinKey(branch, number), record); err != nil {
ix.logf("pins: the revision %s is not pinned for %s: %v", BranchObject(branch, src), record, err) ix.logf("pins: the revision %s is not pinned for %s: %v", BranchObject(branch, number), record, err)
return fmt.Errorf("the pin of %s: %w", BranchObject(branch, src), err) return 0, fmt.Errorf("the pin of %s: %w", BranchObject(branch, number), err)
} }
// The branch's previous revision, the newest kept before this write, pinned for the same pipeline: it is the base of // The branch's previous revision, the newest kept before this write, pinned for the same pipeline: it is the base of
// the validate step's case before (H-A-108 (3), H-A-147), kept as long as the pipeline that needs it. // the validate step's case before (H-A-108 (3), H-A-147), kept as long as the pipeline that needs it.
if pinsBase { if pinsBase {
if err := ix.pin(ctx, PinKey(branch, rev.Base), record); err != nil { if err := ix.pin(ctx, PinKey(branch, rev.Base), record); err != nil {
ix.logf("pins: the previous revision %s is not pinned for %s: %v", BranchObject(branch, newest.Revision), record, err) ix.logf("pins: the previous revision %s is not pinned for %s: %v", BranchObject(branch, newest.Revision), record, err)
return fmt.Errorf("the pin of %s: %w", BranchObject(branch, newest.Revision), err) return 0, fmt.Errorf("the pin of %s: %w", BranchObject(branch, newest.Revision), err)
} }
} else { } else {
ix.logf("pins: %s has no revision before it on its branch: none is pinned for %s as its validate base", BranchObject(branch, src), record) ix.logf("pins: %s has no revision before it on its branch: none is pinned for %s as its validate base", BranchObject(branch, number), record)
} }
} }
// The revision before this one in its branch ceases to be current (§579): it goes unless a record pins it. // The revision before this one in its branch ceases to be current (§579): it goes unless a record pins it.
@@ -364,16 +445,48 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
if after, capacity, err := ix.Revisions.Fill(ctx); fillErr == nil && err == nil && capacity > 0 && used*5 < capacity*4 && after*5 >= capacity*4 { if after, capacity, err := ix.Revisions.Fill(ctx); fillErr == nil && err == nil && capacity > 0 && used*5 < capacity*4 && after*5 >= capacity*4 {
ix.sayFill(ctx, meshcatalog.EventFilling, fmt.Sprintf("%d", rev.Revision), meshcatalog.Fill{Bytes: after, Cap: capacity}) ix.sayFill(ctx, meshcatalog.EventFilling, fmt.Sprintf("%d", rev.Revision), meshcatalog.Fill{Bytes: after, Cap: capacity})
} }
ix.logf("catalog graph: revision %d of source revision %d kept, %d entries, %d changed", rev.Revision, src, len(rev.Entries), len(changed)) ix.logf("catalog graph: revision %d of source revision %d kept (%s), %d entries, %d changed", rev.Revision, src, call.Call, len(rev.Entries), len(changed))
if ix.SayCatalogRevision != nil { if ix.SayCatalogRevision != nil {
if err := ix.SayCatalogRevision(ctx, meshcatalog.CatalogRevisionEvent{V: CatalogEntryLayout, Revision: rev.Revision, Kind: rev.Kind, if err := ix.SayCatalogRevision(ctx, meshcatalog.CatalogRevisionEvent{V: CatalogEntryLayout, Revision: rev.Revision, Kind: rev.Kind,
Branch: keys.EscapeName(branch), Source: src, At: &at, Changed: changed, Entries: int64(len(rev.Entries))}); err != nil { Branch: keys.EscapeName(branch), Source: src, Call: call.Call, CallKind: call.Kind, Module: call.Module, Version: call.Version,
Repository: call.Repository, At: &at, Changed: changed, Entries: int64(len(rev.Entries))}); err != nil {
ix.logf("catalog graph: revision %d is not said: %v", rev.Revision, err) ix.logf("catalog graph: revision %d is not said: %v", rev.Revision, err)
} }
} }
return nil return number, nil
} }
// revisionCall is what made a catalog revision (design 47 H-A-213 (3), H-A-215): call, the words for a person; Kind, one of the
// seat mesh-catalog's three call kinds, which a reader reads instead of the words; Module and Version for a release, Repository
// for a take-in again. The zero value names no call (a revision kept by hand or a test).
type revisionCall struct {
Call, Kind, Module, Version, Repository string
}
// takeInCall is a take-in's: the gate's call its source revision names.
func takeInCall(call string) revisionCall {
return revisionCall{Call: call, Kind: meshcatalog.CatalogRevisionCallKindTakeIn}
}
// releaseCall is a re-pin's on the release of module at version (H-A-213 (2)).
func releaseCall(module, version string) revisionCall {
return revisionCall{Call: ReleaseCall(module, version), Kind: meshcatalog.CatalogRevisionCallKindRelease, Module: module, Version: version}
}
// takeInAgainCall is a take-in again's of repository (H-A-212).
func takeInAgainCall(repository string) revisionCall {
return revisionCall{Call: TakeInAgainCall(repository), Kind: meshcatalog.CatalogRevisionCallKindTakeInAgain, Repository: repository}
}
// recomputeCall is a recompute's that moved a fingerprint, its call naming the reason (H-A-217).
func recomputeCall(reason string) revisionCall {
return revisionCall{Call: RecomputeCall(reason), Kind: meshcatalog.CatalogRevisionCallKindRecompute}
}
// RecomputeCall is the call of the catalog revision a recompute keeps: its reason, such as the assignments of a build module,
// a toolchain or a hand act (design 47 H-A-217).
func RecomputeCall(reason string) string { return "recompute on " + reason }
// HeldObjectsFull is the reason a pending merge is held while the catalog graph revisions are full (Pending.Held). // HeldObjectsFull is the reason a pending merge is held while the catalog graph revisions are full (Pending.Held).
const HeldObjectsFull = "objects full" const HeldObjectsFull = "objects full"
@@ -544,7 +657,7 @@ func (k *KeptTrunk) reread(ctx context.Context, o Objects) error {
} }
manifests := map[string]string{} manifests := map[string]string{}
for _, e := range entries { for _, e := range entries {
if d := meshcatalog.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil { if d := meshcatalogclient.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil {
if raw, err := json.Marshal(e.Manifest); err == nil { if raw, err := json.Marshal(e.Manifest); err == nil {
manifests[d] = string(raw) manifests[d] = string(raw)
} }
@@ -35,7 +35,7 @@ func (ix *Indexer) dry() *dryRun {
// requirements, findings, seat definitions, holds and their bounds are not overlaid, nothing is said, and no hold is // requirements, findings, seat definitions, holds and their bounds are not overlaid, nothing is said, and no hold is
// weighed (Holds and Counted nil hold nothing); ReadFile only reads. // weighed (Holds and Counted nil hold nothing); ReadFile only reads.
Requirements: nil, Findings: nil, SeatDefinitions: nil, ReadFile: ix.ReadFile, SaySeatDefinition: nil, Requirements: nil, Findings: nil, SeatDefinitions: nil, ReadFile: ix.ReadFile, SaySeatDefinition: nil,
Heard: nil, underSeats: true, Heard: nil, underSeats: true, isDry: true,
// A dry run keeps no catalog graph revision (catalog_revisions.go): its revision is the take-in's to keep. // A dry run keeps no catalog graph revision (catalog_revisions.go): its revision is the take-in's to keep.
Revisions: nil, SayCatalogRevision: nil, SayFill: nil, EntryIndex: nil, NamedSets: nil, Versions: nil, SayVersionReleased: nil, SayVersionsFilled: nil, Inputs: nil, InputObjects: nil, Pins: nil, PinsPending: nil, Revisions: nil, SayCatalogRevision: nil, SayFill: nil, EntryIndex: nil, NamedSets: nil, Versions: nil, SayVersionReleased: nil, SayVersionsFilled: nil, Inputs: nil, InputObjects: nil, Pins: nil, PinsPending: nil,
// A dry run writes no registration and asks no one. // A dry run writes no registration and asks no one.
@@ -21,6 +21,8 @@ const (
EntryIndexState = "entry-index" EntryIndexState = "entry-index"
// EntryIndexTool is the tool that reads a fingerprint's key and its object's digest (class read). // EntryIndexTool is the tool that reads a fingerprint's key and its object's digest (class read).
EntryIndexTool = "indexer_entries" EntryIndexTool = "indexer_entries"
// EntryPinsTool is the tool that reads one module's entry pins and paths at a trunk revision (class read; H-A-204).
EntryPinsTool = "indexer_entry"
) )
// EntryIndexValue is one key of entry-index: the digest of the entry object of the fingerprint the key names. // EntryIndexValue is one key of entry-index: the digest of the entry object of the fingerprint the key names.
@@ -49,7 +49,7 @@ func TestAKeptRevisionIndexesItsEntries(t *testing.T) {
ctx := context.Background() ctx := context.Background()
fp := "src2:" + strings.Repeat("d", 64) fp := "src2:" + strings.Repeat("d", 64)
ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), EntryIndex: newMem(), Now: func() time.Time { return time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC) }} ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), EntryIndex: newMem(), Now: func() time.Time { return time.Date(2026, 11, 1, 3, 0, 0, 0, time.UTC) }}
if err := ix.keepBranchEntries(ctx, TrunkBranch, 1, map[string]meshcatalog.CatalogEntry{"avahi": {Module: "avahi", Repository: "novox/r", Commit: "c", Fp: fp, Manifest: map[string]any{}}}, nil); err != nil { if err := ix.KeepBranchEntries(ctx, TrunkBranch, 1, map[string]meshcatalog.CatalogEntry{"avahi": {Module: "avahi", Repository: "novox/r", Commit: "c", Fp: fp, Manifest: map[string]any{}}}, nil); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if got, err := Entry(ctx, ix.Revisions, ix.EntryIndex, meshcatalog.EntryInput{Fingerprint: fp}); err != nil || got.Entry.Module != "avahi" { if got, err := Entry(ctx, ix.Revisions, ix.EntryIndex, meshcatalog.EntryInput{Fingerprint: fp}); err != nil || got.Entry.Module != "avahi" {
@@ -260,6 +260,32 @@ func HeldTools(held Store, ask func(key string, value []byte) (TakeInAsk, error)
} }
return a.Result, nil return a.Result, nil
}}, }},
{Name: TakeInAgainTool,
Description: "Take a repository in again now at the merge commit the indexer keeps of it (design 47 H-A-208, H-A-210): its " +
"module graph is read again (each module's requirements, transitively) and the modules state recomputed as a " +
"package's publish re-pins it, so a changed take-in rule reaches a repository with no merge since; pins and " +
"fingerprints move, no catalog revision is kept and the gate is not asked. An unknown repository is refused.",
Input: map[string]any{"type": "object", "required": []string{"repository"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "the repository as owner/name, such as novox/mesh-host"},
"why": map[string]any{"type": "string", "description": "why it is taken in again now"}}},
Run: func(args map[string]any) (any, error) {
repository := str(args, "repository")
if repository == "" {
return nil, errors.New("name the repository (repository) as owner/name")
}
data, err := json.Marshal(TakeInAsk{V: 1, Again: repository, Why: str(args, "why"), By: stdio.CallerOf(args), At: now().UTC()})
if err != nil {
return nil, err
}
a, err := ask("again."+keys.EscapeName(repository), data)
if err != nil {
return nil, err
}
if a.Result != nil && !a.Result.Stored {
return a.Result, fmt.Errorf("%s is not taken in again: %s", repository, a.Result.Answer)
}
return a.Result, nil
}},
{Name: TakeInSkipTool, {Name: TakeInSkipTool,
Description: "End a watched branch's held source-moved unwritten (design 47 H §744): it is acknowledged, take-in.taken is said " + Description: "End a watched branch's held source-moved unwritten (design 47 H §744): it is acknowledged, take-in.taken is said " +
"with no entries and skipped, and the hand act is recorded through mesh-hand-acts before it is asked. indexer_pending " + "with no entries and skipped, and the hand act is recorded through mesh-hand-acts before it is asked. indexer_pending " +
@@ -68,7 +68,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
} }
mods.put("mesh-sdk", sdkModule) mods.put("mesh-sdk", sdkModule)
held, _ := ix.heldModules(ctx) held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false); err != nil { if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
pub := ix.publishedAll(ctx)[meshSDK] pub := ix.publishedAll(ctx)[meshSDK]
@@ -99,7 +99,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
// mesh-sdk merges a change of buildstep only: avahi does not import it. // mesh-sdk merges a change of buildstep only: avahi does not import it.
held, _ = ix.heldModules(ctx) held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c2, held, false); err != nil { if err := ix.publish(ctx, "novox/mesh-sdk", c2, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
@@ -111,7 +111,7 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
// mesh-sdk merges a change of announce: avahi moves, to the import form, pinned to the new version. // mesh-sdk merges a change of announce: avahi moves, to the import form, pinned to the new version.
held, _ = ix.heldModules(ctx) held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false); err != nil { if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
@@ -123,9 +123,21 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
moved.Fp2, before.Fp2, moved.Inputs2.ImportForm, moved.Packages) moved.Fp2, before.Fp2, moved.Inputs2.ImportForm, moved.Packages)
} }
// mesh-sdk releases 0.1.20: the import form's fingerprint takes the pin too and moves, once (H-A-172). // mesh-sdk's module releases 0.1.97 while the registry has published nothing of its Go package: the module's release
// number is never the package's pin (H-A-183), so avahi stays unpinned and where it was.
ix.Versions = newMem() ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.20"), []byte(`{}`)) ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.97"), []byte(`{}`))
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "avahi"); m.Fp2 != moved.Fp2 || m.Packages[meshSDK] != "" {
t.Fatalf("after mesh-sdk's module release alone: fp2 %q (was %q), versions %v; want it unpinned where it was", m.Fp2, moved.Fp2, m.Packages)
}
// The registry publishes its Go package at 0.1.20, another version than the module's release: the import form's
// fingerprint takes that pin and moves, once (H-A-172, H-A-183).
if _, err := ix.HearVersion(ctx, HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.1.20", Commit: c3, GoModule: meshSDK, Published: true}); err != nil {
t.Fatal(err)
}
for range 2 { for range 2 {
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -183,7 +195,7 @@ func aSiblingDirectoryAloneMovesItsImporter(t *testing.T) {
fpAt := func(commit string) meshcatalog.ModulesByModuleValue { fpAt := func(commit string) meshcatalog.ModulesByModuleValue {
t.Helper() t.Helper()
held, _ := ix.heldModules(ctx) held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, catalogueRepo, commit, held, false); err != nil { if err := ix.publish(ctx, catalogueRepo, commit, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
@@ -262,7 +274,7 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
trunk := func(commit string) func() error { trunk := func(commit string) func() error {
return func() error { return func() error {
held, _ := ix.heldModules(ctx) held, _ := ix.heldModules(ctx)
return ix.publish(ctx, "novox/mesh-sdk", commit, held, false) return ix.publish(ctx, "novox/mesh-sdk", commit, held, false, TrunkBranch)
} }
} }
hear := func(version, commit string) func() error { hear := func(version, commit string) func() error {
@@ -283,17 +295,18 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
if pinned.Fp2 == before.Fp2 || pinned.Packages[meshSDK] != "v0.1.15" { if pinned.Fp2 == before.Fp2 || pinned.Packages[meshSDK] != "v0.1.15" {
t.Fatalf("after 0.1.15: fp2 %q (was %q), versions %v; want moved by c3's announce, pinned to v0.1.15", pinned.Fp2, before.Fp2, pinned.Packages) t.Fatalf("after 0.1.15: fp2 %q (was %q), versions %v; want moved by c3's announce, pinned to v0.1.15", pinned.Fp2, before.Fp2, pinned.Packages)
} }
// A publish in a series avahi does not accept pins nothing and moves nothing (H-A-184).
if m := step("0.2.0 published from c1", hear("0.2.0", c1)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" { if m := step("0.2.0 published from c1", hear("0.2.0", c1)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a publish of a series avahi does not accept moved it: %q %v", m.Fp2, m.Packages) t.Errorf("a publish of a series avahi does not accept moved it: %q %v", m.Fp2, m.Packages)
} }
if m := step("trunk at c2", trunk(c2)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" { if m := step("trunk at c2", trunk(c2)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages) t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages)
} }
// mesh-sdk releases 0.1.20: an accepted series' older version never stands over a newer release (H-A-167). // mesh-sdk's module releases 0.3.0: a module's release number is never a package's pin (H-A-183).
ix.Versions = newMem() ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.20"), []byte(`{}`)) ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.3.0"), []byte(`{}`))
if m := step("0.1.20 released", func() error { return nil }); m.Packages[meshSDK] != "v0.1.20" { if m := step("0.3.0 released", func() error { return nil }); m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a newer release left avahi on its series' older version: %v", m.Packages) t.Errorf("a module's release pinned its package: %v", m.Packages)
} }
} }
@@ -315,7 +328,7 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
} }
mods.put("mesh-sdk", raw) mods.put("mesh-sdk", raw)
held, _ := ix.heldModules(ctx) held, _ := ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false); err != nil { if err := ix.publish(ctx, "novox/mesh-sdk", c1, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
@@ -326,7 +339,7 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
t.Fatal(err) t.Fatal(err)
} }
held, _ = ix.heldModules(ctx) held, _ = ix.heldModules(ctx)
if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false); err != nil { if err := ix.publish(ctx, "novox/mesh-sdk", c3, held, false, TrunkBranch); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := ix.Recompute(ctx); err != nil { if err := ix.Recompute(ctx); err != nil {
@@ -10,6 +10,7 @@ import (
"testing" "testing"
"time" "time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/views" "git.novox.be/novox/mesh-sdk/go/views"
) )
@@ -92,16 +93,20 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
if e, ok, err := modules.Get("ca-trust"); err != nil || !ok || e.Revision != 2 { if e, ok, err := modules.Get("ca-trust"); err != nil || !ok || e.Revision != 2 {
t.Errorf("after the event ca-trust = %+v %v (%v), want it at revision 2", e, ok, err) t.Errorf("after the event ca-trust = %+v %v (%v), want it at revision 2", e, ok, err)
} }
// A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held. // A-487c: the manifests by digest answer from the view held; the verb revision asked of the trunk's newest lists the bucket
// once for which revision is newest (H-A-222) and takes the entries of the view held at that very revision.
listed = objects.n.Load() listed = objects.n.Load()
if raw, err := kept.Manifest(meshcatalog.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` { if raw, err := kept.Manifest(meshcatalogclient.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err) t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err)
} }
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("the manifest listed the bucket %d times, want none", got)
}
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 { if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 {
t.Errorf("revision of the newest answered %+v (%v), want revision 2 whole", a, err) t.Errorf("revision of the newest answered %+v (%v), want revision 2 whole", a, err)
} }
if got := objects.n.Load() - listed; got != 0 { if got := objects.n.Load() - listed; got != 1 {
t.Errorf("the manifest and the verb revision listed the bucket %d times, want none", got) t.Errorf("the verb revision listed the bucket %d times, want once, for the newest", got)
} }
if len(said) != 2 || !strings.Contains(said[1], "read revision 2 (was 1), 2 modules, in ") { if len(said) != 2 || !strings.Contains(said[1], "read revision 2 (was 1), 2 modules, in ") {
t.Errorf("said %q, want each read with its revision, module count and time", said) t.Errorf("said %q, want each read with its revision, module count and time", said)
@@ -125,6 +130,16 @@ func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
if keys, _ := modules.Keys(); len(keys) != 3 { if keys, _ := modules.Keys(); len(keys) != 3 {
t.Errorf("the next heard event left the view at %v, want revision 3's three modules", keys) t.Errorf("the next heard event left the view at %v, want revision 3's three modules", keys)
} }
// H-A-222: two revisions kept that the view has not heard are answered by the verb revision from the store, whole; the
// view held stays a cache at its own revision.
keep(4, "avahi", "ca-trust", "dbus")
keep(5, "avahi", "ca-trust", "dbus", "feh")
if a, err := Revision(context.Background(), objects, kept, meshcatalog.RevisionInput{}); err != nil || a.Revision == nil || a.Revision.Revision != 5 || len(a.Entries) != 4 {
t.Errorf("revision of the newest answered %+v (%v) while the view held 3, want revision 5 whole from the store", a, err)
}
if keys, _ := modules.Keys(); len(keys) != 3 {
t.Errorf("the verb moved the view held to %v, want it at revision 3 until it hears the event", keys)
}
} }
// A-487: the verb module reads the one module it is asked. // A-487: the verb module reads the one module it is asked.
@@ -63,6 +63,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
return w, err return w, err
} }
objects, namedErr := ix.namedObjects(ctx, f) objects, namedErr := ix.namedObjects(ctx, f)
trees := ix.goModTree(ctx, f)
overlay := map[string]meshcatalog.ModulesByModuleValue{} overlay := map[string]meshcatalog.ModulesByModuleValue{}
dirOf := map[string]string{} dirOf := map[string]string{}
named, every := map[string]bool{}, true // every module the message's manifests name, and whether each names one named, every := map[string]bool{}, true // every module the message's manifests name, and whether each names one
@@ -109,7 +110,23 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
next.None2 = "the manifest cannot be read: it cannot be kept as a value" next.None2 = "the manifest cannot be read: it cannot be kept as a value"
} else { } else {
packages := ix.goPackages(ctx, f, dir, cur, has) packages := ix.goPackages(ctx, f, dir, cur, has)
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
// The entry pins every mesh package of another repository the module's build requires, transitively across
// the same-repository modules of its graph (design 47 H-A-201).
packages = unionSorted(packages, trees.otherRequired(goModuleDirs(m, dir)))
}
in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx)) in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx))
if in.Kept != nil {
// A Go build resolves the same-repository modules its go.mod files require, transitively, from the tree: their
// directories' objects join the tree input of every form, never the entry's paths (design 47 H-A-202, H-A-172).
if m, err := judge.ForInputs([]byte(raw)); err == nil && usesGo(languagesOf(m)) {
if tree, none := ix.derivedTree(ctx, f, trees, m, dir, in.Kept.Tree); none != "" {
in.Kept, in.None2 = nil, none
} else {
in.Kept.Tree = tree
}
}
}
next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths
if in.Kept != nil && has && cur.Inputs2 != nil { if in.Kept != nil && has && cur.Inputs2 != nil {
// The imports its last trunk build reported stand until its next build reports again (ADR 0593 §2, §4). // The imports its last trunk build reported stand until its next build reports again (ADR 0593 §2, §4).
@@ -139,7 +156,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
// 0593 §3). A forge that cannot list them leaves the state as it was, said: the importers move at the next merge. // 0593 §3). A forge that cannot list them leaves the state as it was, said: the importers move at the next merge.
reach := maps.Clone(held) reach := maps.Clone(held)
maps.Copy(reach, overlay) maps.Copy(reach, overlay)
if err := ix.publish(ctx, f.Repository, f.Merge, reach, false); err != nil { if err := ix.publish(ctx, f.Repository, f.Merge, reach, false, f.Branch); err != nil {
ix.logf("%s at %s: its packages are not published: %v", f.Repository, f.Merge, err) ix.logf("%s at %s: its packages are not published: %v", f.Repository, f.Merge, err)
} }
// What this merge moved reaches the modules standing on it (a base's tree, build-agent's fingerprint), in the graph's order. // What this merge moved reaches the modules standing on it (a base's tree, build-agent's fingerprint), in the graph's order.
@@ -0,0 +1,198 @@
package index
import (
"context"
"fmt"
"path"
"path/filepath"
"slices"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// goModTree reads the go.mod files of one merge's tree, each directory once (Indexer.GoMod), for the walk of the module
// graph a Go build resolves from the tree (design 47 H-A-202, H-A-157).
type goModTree struct {
ix *Indexer
ctx context.Context
f whole
read map[string][]byte // by directory from the repository's root; nil for none
}
func (ix *Indexer) goModTree(ctx context.Context, f whole) *goModTree {
return &goModTree{ix: ix, ctx: ctx, f: f, read: map[string][]byte{}}
}
// goMod is the go.mod of the directory dir (from the root, "" for the root) at the merge commit; nil when there is none
// or it cannot be read, said.
func (t *goModTree) goMod(dir string) []byte {
if raw, ok := t.read[dir]; ok {
return raw
}
var raw []byte
if t.ix.GoMod != nil {
var err error
if raw, err = t.ix.GoMod(t.ctx, t.f.Repository, t.f.Merge, dir); err != nil {
t.ix.logf("%s at %s: the go.mod of %q cannot be read, so no module of the tree is found there: %v", t.f.Repository, t.f.Merge, dir, err)
raw = nil
}
}
t.read[dir] = raw
return raw
}
// sameRepositoryDir is the directory of the merge's tree that holds the Go module required: the required path starts with
// the repository's module prefix, and the directory it names holds a go.mod declaring that path (H-A-198, H-A-202). false
// for a module of another repository, or one the tree holds no go.mod of.
func (t *goModTree) sameRepositoryDir(required string) (string, bool) {
repo, dir, ok := trunk.PackageRepository(required)
if !ok || dir == "" || !trunk.SameRepository(repo, t.f.Repository) {
return "", false
}
if goModModule(t.goMod(dir)) != required {
return "", false
}
return dir, true
}
// sameRepositoryRequired are the directories (from the root, sorted, each once) of every same-repository Go module the
// go.mod of each start directory requires, transitively through those modules' own go.mod files (H-A-202, H-A-157). A
// same-repository module a go.mod names in a replace directive counts as required, whether or not a require line names it
// (H-A-224). The start directories themselves are not among them.
func (t *goModTree) sameRepositoryRequired(start []string) []string {
seen := map[string]bool{}
for _, s := range start {
seen[s] = true
}
queue := slices.Clone(start)
var out []string
for len(queue) > 0 {
dir := queue[0]
queue = queue[1:]
raw := t.goMod(dir)
for _, req := range append(fingerprint.MeshRequiresOf(raw), fingerprint.ReplacedSiblings(raw, t.f.Repository)...) {
d, ok := t.sameRepositoryDir(req)
if !ok || seen[d] {
continue
}
seen[d] = true
out = append(out, d)
queue = append(queue, d)
}
}
slices.Sort(out)
return out
}
// goModuleDirs are the directories (from the root) a module's Go build starts from: its own directory and the directory
// of each path of each Go package artifact of its manifest m.
func goModuleDirs(m catalogue.Manifest, dir string) []string {
out := []string{dir}
if m.Build == nil {
return out
}
for _, a := range m.Build.Artifacts {
if a.Kind != catalogue.ArtifactPackage || a.Language != "go" {
continue
}
for _, p := range a.Paths {
if d := path.Join(dir, strings.TrimSuffix(filepath.ToSlash(p), "/")); !slices.Contains(out, d) {
out = append(out, d)
}
}
}
return out
}
// otherRequired are the mesh packages of other repositories (sorted, each once) that the go.mod of each start directory,
// and of every same-repository module those require transitively (sameRepositoryRequired), requires at any version, the
// zero version included: every one the module's build resolves, which the entry pins (design 47 H-A-201, H-A-183,
// H-A-157). A same-repository module is none: it is built from the tree (H-A-198, H-A-202). nil for none.
func (t *goModTree) otherRequired(start []string) []string {
var out []string
for _, d := range append(slices.Clone(start), t.sameRepositoryRequired(start)...) {
out = append(out, fingerprint.OtherPackages(fingerprint.MeshRequiresOf(t.goMod(d)), t.f.Repository)...)
}
if len(out) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(out)))
}
// unionSorted is a and b together, sorted and each once; nil for none.
func unionSorted(a, b []string) []string {
if len(a)+len(b) == 0 {
return nil
}
return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(a), b...))))
}
// derivedDirs are the same-repository module directories (from the root, sorted) the module's Go build resolves from the
// tree beyond its own tree input (H-A-202): every one its go.mod files require, transitively, that its tree input does
// not hold already, which is the module's directory for the directory's tree, and its named paths for the named tree.
func (t *goModTree) derivedDirs(m catalogue.Manifest, dir string) []string {
var held []string
if paths, every := m.NamedPaths(); every {
for _, p := range paths {
held = append(held, path.Join(dir, filepath.ToSlash(p)))
}
} else {
held = []string{dir}
}
var out []string
for _, d := range t.sameRepositoryRequired(goModuleDirs(m, dir)) {
if !slices.ContainsFunc(held, func(h string) bool { return h == "" || h == "." || d == h || strings.HasPrefix(d, h+"/") }) {
out = append(out, d)
}
}
return out
}
// DerivedLine opens each line of the tree input naming a derived directory and its git object (H-A-202, H-A-172).
const DerivedLine = "derived "
// derivedTree is the module's tree input with a line per derived directory and its object appended, sorted by directory
// (H-A-202, H-A-172: what a build reads moves its fingerprint), so a change of such a directory moves the fingerprint of
// every form. The objects are asked of the forge at the merge commit by their paths from the root (Named); an answer that
// lacks one gives the reason the module has no inputs, naming them. A module with no derived directory keeps its tree.
func (ix *Indexer) derivedTree(ctx context.Context, f whole, t *goModTree, m catalogue.Manifest, dir, tree string) (string, string) {
dirs := t.derivedDirs(m, dir)
if len(dirs) == 0 {
return tree, ""
}
if ix.Named == nil {
return "", "the forge cannot be asked the git objects of the derived directories " + strings.Join(dirs, ", ")
}
got, err := ix.Named(ctx, f.Repository, f.Merge, map[string][]string{"": dirs})
if err != nil {
return "", fmt.Sprintf("the git objects of the derived directories %s: %v", strings.Join(dirs, ", "), err)
}
lines := []string{tree}
var missing []string
for _, d := range dirs {
o := got[""][d]
if o == "" {
missing = append(missing, d)
continue
}
lines = append(lines, DerivedLine+d+" "+o)
}
if len(missing) > 0 {
return "", "the forge names no git object of the derived directories " + strings.Join(missing, ", ")
}
return strings.Join(lines, "\n"), ""
}
// goModModule is the module path a go.mod declares; "" for none.
func goModModule(raw []byte) string {
for _, line := range strings.Split(string(raw), "\n") {
if f := strings.Fields(line); len(f) == 2 && f[0] == "module" {
return strings.Trim(f[1], `"`)
}
}
return ""
}
@@ -0,0 +1,164 @@
package index
import (
"context"
"slices"
"strings"
"testing"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
)
// A fixture repository novox/mesh-delivery: the Go package state of modules/g requires the same-repository module
// modules/b/builds at the zero version, which requires modules/c/client in turn; mesh-sdk is of another repository, and
// modules/b/gone names a directory the tree holds no go.mod of.
const ptpZero = " v0.0.0-00010101000000-000000000000"
var ptpGoMods = map[string]string{
"modules/g/state": "module git.novox.be/novox/mesh-delivery/modules/g/state\n\nrequire (\n\tgit.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero +
"\n\tgit.novox.be/novox/mesh-sdk/go" + ptpZero + "\n\tgit.novox.be/novox/mesh-delivery/modules/b/gone" + ptpZero + "\n)\n",
"modules/b/builds": "module git.novox.be/novox/mesh-delivery/modules/b/builds\n\nrequire git.novox.be/novox/mesh-delivery/modules/c/client" + ptpZero + " // indirect\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n",
}
// ptpProgram is modules/g's manifest: a program naming no paths beside the package state.
const ptpProgram = `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"program","kind":"bundle","language":"go","system":"arch","from":"cmd/g","binary":"g","loads":["g"]},` +
`{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
// ptpIndexer answers the fixture's go.mod files and, for a path asked from the root, the object objects names for it.
func ptpIndexer(t *testing.T, objects map[string]string) (*Indexer, *memStore, *[]map[string][]string) {
ix, mods, _ := manifestsIndexer(t)
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := ptpGoMods[dir]; ok {
return []byte(c), nil
}
return nil, nil
}
var asked []map[string][]string
ix.Named = func(_ context.Context, _, _ string, paths map[string][]string) (map[string]map[string]string, error) {
asked = append(asked, paths)
out := map[string]map[string]string{}
for dir, ps := range paths {
out[dir] = map[string]string{}
for _, p := range ps {
out[dir][p] = objects[p]
}
}
return out, nil
}
return ix, mods, &asked
}
// Design 47 H-A-202, H-A-172: the same-repository modules a Go build resolves from the tree, transitively, join the tree
// input of every form with their objects, so a change of one moves the fingerprint of a module whose program names no
// paths; the entry's paths stay the manifest's.
func TestADerivedDirectoryMovesTheFingerprintOfEveryForm(t *testing.T) {
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
tree := func(objects map[string]string) string {
ix, mods, asked := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram}, nil)
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if got := strings.Join(m.Paths["state"], ","); got != "state" {
t.Errorf("the entry's paths of state are %q, want the manifest's", got)
}
if len(*asked) != 1 || strings.Join((*asked)[0][""], ",") != "modules/b/builds,modules/c/client" {
t.Fatalf("asked the forge %v, want the derived directories from the root once", *asked)
}
if m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, "gtree\n") {
t.Fatalf("inputs = %+v (%s), want the directory's tree with the derived lines", m.Inputs2, m.None2)
}
return m.Inputs2.Tree
}
before := tree(objects)
if want := "gtree\nderived modules/b/builds " + objects["modules/b/builds"] + "\nderived modules/c/client " + objects["modules/c/client"]; before != want {
t.Errorf("the tree input is %q, want %q", before, want)
}
objects["modules/c/client"] = strings.Repeat("d", 40)
after := tree(objects)
fp := func(tree string) string {
v, _ := fingerprint.Of(fingerprint.Inputs{Module: "g", Tree: tree})
return v
}
if fp(before) == fp(after) {
t.Errorf("a change of modules/c/client left the fingerprint %s", fp(after))
}
// The named tree takes them too: a module whose every artifact names its paths.
named := `{"module":"g","version":"1.0","build":{"artifacts":[{"name":"state","kind":"package","language":"go","paths":["state"],"series":"0.1"}]}}`
ix, mods, _ := ptpIndexer(t, objects)
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": named}, nil)
f.Paths = map[string]map[string]string{"modules/g": {"state": strings.Repeat("1", 40)}}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Inputs2 == nil || !strings.HasPrefix(m.Inputs2.Tree, fingerprint.NamedTreePrefix) ||
!strings.HasSuffix(m.Inputs2.Tree, "\nderived modules/c/client "+objects["modules/c/client"]) || strings.Join(m.Paths["state"], ",") != "state" {
t.Errorf("module = %+v, want the named tree with the derived lines and the manifest's paths", m)
}
// A forge that names no object of a derived directory gives the module none, naming it.
ix, mods, _ = ptpIndexer(t, map[string]string{"modules/b/builds": strings.Repeat("b", 40)})
if err := ix.Apply(context.Background(), withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}),
map[string]string{"modules/g": ptpProgram}, nil)); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Fp2 != "" || !strings.Contains(m.None2, "modules/c/client") {
t.Errorf("module = %+v, want none naming the directory the forge gave no object of", m)
}
}
// Design 47 H-A-201: the entry pins every mesh package of another repository the module's build requires, across the
// same-repository modules of its graph: here the one modules/c/client requires, which modules/g's go.mod never names.
// A module of the module's own repository is never pinned, whatever a build reported (H-A-198, H-A-202).
func TestTheEntryPinsEveryRequirementOfTheModuleGraph(t *testing.T) {
const conditions = "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
ix, mods, _ := ptpIndexer(t, objects)
mod := map[string]string{
"modules/g": "module g\n\nrequire git.novox.be/novox/mesh-delivery/modules/b/builds" + ptpZero + "\n",
"modules/c/client": "module git.novox.be/novox/mesh-delivery/modules/c/client\n\nrequire " + conditions + ptpZero + "\n",
}
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := mod[dir]; ok {
return []byte(c), nil
}
return []byte(ptpGoMods[dir]), nil
}
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram},
map[string]map[string]string{"modules/g": {"go.sum": avahiGoSum}})
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "g")
if m.Inputs2 == nil || strings.Join(m.Inputs2.Packages, ",") != conditions+",git.novox.be/novox/mesh-sdk/go" {
t.Errorf("packages = %+v, want the client's requirement and state's mesh-sdk, and no module of the repository", m.Inputs2)
}
builds := "git.novox.be/novox/mesh-delivery/modules/b/builds"
pins := releasedPins(sdkstep.PackageVersions{builds: "v0.1.106", conditions: "v0.1.4"}, nil, catalogue.Manifest{}, "novox/mesh-delivery", nil)
if _, pinned := pins[builds]; pinned || len(pins) != 1 {
t.Errorf("pins = %v, want %s only, never a module of the repository", pins, conditions)
}
// H-A-224: a same-repository module the go.mod names only in a replace directive (the watchman's buildrun) counts as
// required: its own requirement of another repository's client is pinned too.
const assigner = "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
objects["modules/d/buildrun"] = strings.Repeat("d", 40)
ix, mods, _ = ptpIndexer(t, objects)
mod["modules/g"] += "\nreplace git.novox.be/novox/mesh-delivery/modules/d/buildrun => ../d/buildrun\n"
mod["modules/d/buildrun"] = "module git.novox.be/novox/mesh-delivery/modules/d/buildrun\n\nrequire " + assigner + ptpZero + "\n"
ix.GoMod = func(_ context.Context, _, _, dir string) ([]byte, error) {
if c, ok := mod[dir]; ok {
return []byte(c), nil
}
return []byte(ptpGoMods[dir]), nil
}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "g"); m.Inputs2 == nil || !slices.Contains(m.Inputs2.Packages, assigner) || !slices.Contains(m.Inputs2.Packages, conditions) {
t.Errorf("packages = %+v, want the replaced sibling's requirement %s pinned beside the client's", m.Inputs2, assigner)
}
}
@@ -14,6 +14,8 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk" "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
@@ -138,8 +140,14 @@ func reached(held map[string]meshcatalog.ModulesByModuleValue) (modules, imports
// directory, else by the Go module whose go.mod is at its directory or the nearest above it (asked of the forge // directory, else by the Go module whose go.mod is at its directory or the nearest above it (asked of the forge
// directory by directory). A module the listing has no go.mod for is said and left out. // directory by directory). A module the listing has no go.mod for is said and left out.
func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, modules, imports []string) (map[string]meshcatalog.PackagesByModuleValue, error) { func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, modules, imports []string) (map[string]meshcatalog.PackagesByModuleValue, error) {
out, _, err := ix.packagesIn(ctx, repository, commit, modules, imports)
return out, err
}
// packagesIn is packagesAt, with the modules whose directory the tree at the commit does not hold (gone), by module path.
func (ix *Indexer) packagesIn(ctx context.Context, repository, commit string, modules, imports []string) (map[string]meshcatalog.PackagesByModuleValue, []string, error) {
if ix.Files == nil || commit == "" { if ix.Files == nil || commit == "" {
return nil, nil return nil, nil, nil
} }
dirOf := map[string]string{} // module path to its directory in the repository dirOf := map[string]string{} // module path to its directory in the repository
for _, m := range modules { for _, m := range modules {
@@ -182,7 +190,7 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
_, dir, _ := trunk.PackageRepository(imp) _, dir, _ := trunk.PackageRepository(imp)
for d := dir; ; d = parent(d) { for d := dir; ; d = parent(d) {
if err := list([]string{d}); err != nil { if err := list([]string{d}); err != nil {
return nil, err return nil, nil, err
} }
if _, ok := listings[d]["go.mod"]; ok { if _, ok := listings[d]["go.mod"]; ok {
m := strings.TrimSuffix(trunk.ForgeModulePrefix+strings.TrimPrefix(repository, "novox/")+"/"+d, "/") m := strings.TrimSuffix(trunk.ForgeModulePrefix+strings.TrimPrefix(repository, "novox/")+"/"+d, "/")
@@ -201,27 +209,40 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
absent := map[string]bool{} absent := map[string]bool{}
dirs := slices.Sorted(maps.Values(dirOf)) dirs := slices.Sorted(maps.Values(dirOf))
if err := list(dirs); err != nil { if err := list(dirs); err != nil {
if !errors.Is(err, ErrNotInTree) || len(dirs) < 2 { if !errors.Is(err, ErrNotInTree) {
return nil, err return nil, nil, err
} }
whole := err whole := err
for _, d := range dirs { for _, d := range dirs {
if err := list([]string{d}); err != nil { if len(dirs) > 1 {
err = list([]string{d})
}
if err != nil {
if !errors.Is(err, ErrNotInTree) { if !errors.Is(err, ErrNotInTree) {
return nil, err return nil, nil, err
} }
ix.logf("%s at %s: the directory %q is not in the tree, so the module there is not read: %v", repository, commit, d, err) ix.logf("%s at %s: the directory %q is not in the tree, so the module there is not read: %v", repository, commit, d, err)
absent[d] = true absent[d] = true
} }
} }
// Every directory asked is missing: gone only when the commit holds a tree at all (the seat's trees of the commit, no
// files), else the tree itself fails (H-A-232, H-A-233).
if len(absent) == len(dirs) { if len(absent) == len(dirs) {
return nil, whole // the tree itself fails if ix.Trunk == nil {
return nil, nil, whole
}
if _, err := ix.Trunk(ctx, repository, commit); err != nil {
return nil, nil, fmt.Errorf("%w (and the commit's trees: %v)", whole, err)
}
ix.logf("%s at %s: the commit holds a tree, so every directory asked (%s) is gone", repository, commit, strings.Join(dirs, ", "))
} }
} }
out := map[string]meshcatalog.PackagesByModuleValue{} out := map[string]meshcatalog.PackagesByModuleValue{}
var gone []string
for _, m := range slices.Sorted(maps.Keys(dirOf)) { for _, m := range slices.Sorted(maps.Keys(dirOf)) {
dir := dirOf[m] dir := dirOf[m]
if absent[dir] { if absent[dir] {
gone = append(gone, m)
continue continue
} }
requires, pkgs, err := fingerprint.Packages(m, listings[dir]) requires, pkgs, err := fingerprint.Packages(m, listings[dir])
@@ -236,7 +257,7 @@ func (ix *Indexer) packagesAt(ctx context.Context, repository, commit string, mo
out[m] = meshcatalog.PackagesByModuleValue{Module: m, Repository: trunk.RepositoryName(repository), Dir: d, Commit: commit, out[m] = meshcatalog.PackagesByModuleValue{Module: m, Repository: trunk.RepositoryName(repository), Dir: d, Commit: commit,
Requires: requires, Packages: pkgs} Requires: requires, Packages: pkgs}
} }
return out, nil return out, gone, nil
} }
// parent is the directory above d, "" for the repository's root. // parent is the directory above d, "" for the repository's root.
@@ -252,8 +273,10 @@ func parent(d string) string {
} }
// publish writes the packages state for the Go modules of repository the held modules (and the merge's overlay) reach, // publish writes the packages state for the Go modules of repository the held modules (and the merge's overlay) reach,
// as the merge commit holds them: a key whose commit, requirements and digests are what it holds already is left. // as the merge commit holds them: a key whose commit, requirements and digests are what it holds already is left. The kept
func (ix *Indexer) publish(ctx context.Context, repository, commit string, held map[string]meshcatalog.ModulesByModuleValue, onlyMissing bool) error { // key of a package whose directory the commit does not hold is dropped at the trunk's merge on the real indexer only, never
// at a branch's take-in (branch) or in a dry run, which share the live packages state (H-A-232 (1)).
func (ix *Indexer) publish(ctx context.Context, repository, commit string, held map[string]meshcatalog.ModulesByModuleValue, onlyMissing bool, branch string) error {
if ix.Packages == nil || ix.Files == nil { if ix.Packages == nil || ix.Files == nil {
return nil return nil
} }
@@ -263,6 +286,7 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
return nil return nil
} }
have := ix.publishedAll(ctx) have := ix.publishedAll(ctx)
drop := !ix.isDry && isTrunkBranch(branch)
if onlyMissing { if onlyMissing {
// Imports a held module already resolves are not asked again. // Imports a held module already resolves are not asked again.
var ms, is []string var ms, is []string
@@ -287,10 +311,28 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
modules[repo] = append(modules[repo], m) modules[repo] = append(modules[repo], m)
} }
} }
got, err := ix.packagesAt(ctx, repo, commit, modules[repo], imports[repo]) got, gone, err := ix.packagesIn(ctx, repo, commit, modules[repo], imports[repo])
if err != nil { if err != nil {
ix.notPublished(ctx, repo, commit, err)
return err return err
} }
ix.published(ctx, repo, commit)
// A repository's packages are those in the merge's tree (design 47 H H-A-232): the kept key of a package whose directory
// the trunk's merge no longer holds is dropped, said, so the forge is never asked for it again.
for _, m := range gone {
if _, kept := have[m]; !kept || !drop {
continue
}
key, err := PackagesKey(m)
if err != nil {
ix.logf("%s: %v", m, err)
continue
}
if err := write(ctx, ix.Packages, key, func(_ []byte, found bool) ([]byte, bool, error) { return nil, found, nil }); err != nil {
return fmt.Errorf("packages.%s: %w", key, err)
}
ix.logf("package %s of %s: its directory is gone at %s: its kept key dropped", m, repo, commit)
}
for _, m := range slices.Sorted(maps.Keys(got)) { for _, m := range slices.Sorted(maps.Keys(got)) {
next := got[m] next := got[m]
key, err := PackagesKey(m) key, err := PackagesKey(m)
@@ -316,6 +358,36 @@ func (ix *Indexer) publish(ctx context.Context, repository, commit string, held
return nil return nil
} }
// PackagesNotPublishedKind is the finding of a repository whose packages a take-in's publish could not list, for a reason
// that is not a gone directory (design 47 H H-A-233, H-A-234): scope merge, id the repository, with the forge's words;
// raised at once and cleared at the repository's next publish that lists.
const PackagesNotPublishedKind = "packages-not-published"
// notPublished raises the finding packages-not-published of repository.
func (ix *Indexer) notPublished(ctx context.Context, repository, commit string, cause error) {
if ix.Findings == nil {
return
}
f := ix.observed(mcc.ScopeMerge, repository, PackagesNotPublishedKind,
fmt.Sprintf("the packages of %s are not published at %.12s", repository, commit),
"A repository's packages are not published",
fmt.Sprintf("The take-in of %s at %s could not list its packages (%v), so their digests stay at their last published commit and a change to one does not move its importers.", repository, commit, cause),
"indexer."+PackagesNotPublishedKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("%s: the finding %s is not written: %v", repository, PackagesNotPublishedKind, err)
}
}
// published clears the finding packages-not-published of repository: its publish listed.
func (ix *Indexer) published(ctx context.Context, repository, commit string) {
if ix.Findings == nil {
return
}
if err := ix.clearFinding(ctx, mcc.ScopeMerge, repository, PackagesNotPublishedKind, "its packages are listed at "+commit); err != nil {
ix.logf("%s: the finding %s is not cleared: %v", repository, PackagesNotPublishedKind, err)
}
}
// FillPackages publishes, at the repository's taken merge (the trunk the modules say), each reached Go module the // FillPackages publishes, at the repository's taken merge (the trunk the modules say), each reached Go module the
// packages state does not hold yet: the state of a new indexer, or of a module a build newly reported an import of. // packages state does not hold yet: the state of a new indexer, or of a module a build newly reported an import of.
// A repository with no taken merge is left for its next merge. // A repository with no taken merge is left for its next merge.
@@ -349,7 +421,7 @@ func (ix *Indexer) FillPackages(ctx context.Context) error {
continue continue
} }
began := time.Now() began := time.Now()
if err := ix.publish(ctx, repo, t.Merge, held, true); err != nil { if err := ix.publish(ctx, repo, t.Merge, held, true, TrunkBranch); err != nil {
ix.logf("%s: its packages are not published at %s: %v", repo, t.Merge, err) ix.logf("%s: its packages are not published at %s: %v", repo, t.Merge, err)
} }
ix.logf("publishing the packages at start: %s at %.12s took %s", repo, t.Merge, time.Since(began).Round(time.Millisecond)) ix.logf("publishing the packages at start: %s at %.12s took %s", repo, t.Merge, time.Since(began).Round(time.Millisecond))
@@ -0,0 +1,166 @@
package index
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// Design 47 H H-A-232 (1), H-A-233, H-A-234: a repository's packages are those in the merge's tree. The kept key of a
// package whose directory the trunk's merge no longer holds is dropped, said, and the repository's other packages are
// published all the same (issue 1585's one-directory fallback); a branch's take-in and a dry run, which share the live
// packages state, drop nothing. A repository whose every asked directory is gone, at a commit that holds a tree, drops them
// and raises nothing; a listing that fails for another reason, or a commit with no tree, raises the finding
// packages-not-published (scope merge, id the repository), cleared at the next publish that lists.
func TestARepositorysPackagesAreThoseInTheMergesTree(t *testing.T) {
const (
live = "git.novox.be/novox/mesh-catalog/modules/disk-load/diskreadings"
gone = "git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep"
liveDir = "modules/disk-load/diskreadings"
goneDir = "modules/mesh-build-go/buildstep"
)
ctx := context.Background()
old, merge := strings.Repeat("d", 40), strings.Repeat("e", 40)
reaching := func(m string) map[string]meshcatalog.ModulesByModuleValue {
return map[string]meshcatalog.ModulesByModuleValue{
"user": {Repository: catalogueRepo, Inputs2: &meshcatalog.ModulesByModuleValueInputs2{Packages: []string{m}}},
}
}
type setup struct {
ix *Indexer
findings *memFindings
said *[]string
down *error // the forge's listing fails for a reason that is not a gone directory
noTree *error // the seat's trees of the commit, without files, fail
}
newSetup := func(t *testing.T, kept ...string) setup {
t.Helper()
var down, noTree error
said := []string{}
files := func(_ context.Context, repository, sha string, dirs []string) (map[string]map[string]string, error) {
if down != nil {
return nil, down
}
out := map[string]map[string]string{}
for _, d := range dirs {
if d != liveDir {
// as filesVia says the seat's not-found naming the directory
return nil, fmt.Errorf("the directory %s is not in %s at %s: %w", d, repository, sha, ErrNotInTree)
}
out[d] = map[string]string{"go.mod": "m1", "readings.go": "r-" + sha[:1]}
}
return out, nil
}
trees := func(context.Context, string, string) ([]string, error) { return []string{"modules/disk-load"}, noTree }
findings := newFindings()
ix := &Indexer{Module: testSelf, Packages: newMem(), Files: files, Trunk: trees, Findings: findings,
Logf: func(format string, args ...any) { said = append(said, fmt.Sprintf(format, args...)) }}
dirOf := map[string]string{live: liveDir, gone: goneDir}
for _, m := range kept {
key, err := PackagesKey(m)
if err != nil {
t.Fatal(err)
}
raw, err := EncodePublishedPackages(meshcatalog.PackagesByModuleValue{Module: m, Repository: catalogueRepo, Dir: dirOf[m], Commit: old,
Requires: "req", At: ptr(time.Unix(0, 0).UTC()), Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: m, Digest: "x"}}})
if err != nil {
t.Fatal(err)
}
ix.Packages.(*memStore).put(key, raw)
}
return setup{ix: ix, findings: findings, said: &said, down: &down, noTree: &noTree}
}
notPublished := func(s setup) bool {
for _, f := range s.findings.Held() {
if f.Kind == PackagesNotPublishedKind && f.Scope == mcc.ScopeMerge && f.ID == keyID(catalogueRepo) {
return true
}
}
return false
}
dropped := "package " + gone + " of " + catalogueRepo + ": its directory is gone at " + merge + ": its kept key dropped"
t.Run("the trunk's merge drops a gone package's key and publishes the others; a failed listing is a finding until one lists", func(t *testing.T) {
s := newSetup(t, live, gone)
*s.down = errors.New("seat:git.trees (failed): the forge did not answer")
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(live), false, TrunkBranch); err == nil {
t.Fatal("a listing that failed published")
}
if !notPublished(s) {
t.Fatalf("a listing that failed raised no finding %s: %v", PackagesNotPublishedKind, s.findings.Held())
}
*s.down = nil
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(live), false, TrunkBranch); err != nil {
t.Fatal(err)
}
have := s.ix.publishedAll(ctx)
if have[live].Commit != merge {
t.Errorf("packages of %s at %q, want the merge %s: a gone sibling kept it from being published", live, have[live].Commit, merge)
}
if _, kept := have[gone]; kept {
t.Errorf("the kept key of %s, whose directory the merge does not hold, stays", gone)
}
if !strings.Contains(strings.Join(*s.said, "\n"), dropped) {
t.Errorf("the journal does not say %q:\n%s", dropped, strings.Join(*s.said, "\n"))
}
if notPublished(s) {
t.Error("the finding packages-not-published stays after a publish that listed")
}
})
for _, row := range []struct {
name string
dry bool
branch string
}{
{"a branch's take-in keeps the key: only the trunk's merge says a directory is gone", false, "feat/x"},
{"a dry run keeps the key: it shares the live packages state", true, TrunkBranch},
} {
t.Run(row.name, func(t *testing.T) {
s := newSetup(t, live, gone)
ix := s.ix
if row.dry {
ix = s.ix.dry().ix
}
if err := ix.publish(ctx, catalogueRepo, merge, reaching(live), false, row.branch); err != nil {
t.Fatal(err)
}
if _, kept := s.ix.publishedAll(ctx)[gone]; !kept {
t.Errorf("the kept key of %s was dropped", gone)
}
})
}
t.Run("a one-key repository whose directory is gone drops it and raises nothing, when the commit holds a tree", func(t *testing.T) {
s := newSetup(t, gone)
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(gone), false, TrunkBranch); err != nil {
t.Fatal(err)
}
if _, kept := s.ix.publishedAll(ctx)[gone]; kept {
t.Errorf("the kept key of %s, the repository's one package, whose directory is gone, stays", gone)
}
if notPublished(s) {
t.Error("a gone directory raised packages-not-published")
}
})
t.Run("every asked directory missing at a commit with no tree is the tree's failure: the keys stay and it is a finding", func(t *testing.T) {
s := newSetup(t, gone)
*s.noTree = errors.New("seat:git.trees (not-found): Gitea API GET: 404 Not Found")
if err := s.ix.publish(ctx, catalogueRepo, merge, reaching(gone), false, TrunkBranch); err == nil {
t.Fatal("a commit with no tree published")
}
if _, kept := s.ix.publishedAll(ctx)[gone]; !kept {
t.Errorf("the kept key of %s was dropped at a commit with no tree", gone)
}
if !notPublished(s) {
t.Error("a commit with no tree raised no finding")
}
})
}
@@ -119,8 +119,9 @@ func ReleaseRecord(branch, module string) string {
// every revision of the branch below it; a mover the revision holds no release of (superseded, no version released) keeps its // every revision of the branch below it; a mover the revision holds no release of (superseded, no version released) keeps its
// older release's revision. Outcome tier-late frees nothing: its later tiers' wants are not written, though their release // older release's revision. Outcome tier-late frees nothing: its later tiers' wants are not written, though their release
// records are held, and the same pipeline's later running frees them. A pipeline whose revision is not kept frees nothing and // records are held, and the same pipeline's later running frees them. A pipeline whose revision is not kept frees nothing and
// says so; one applied twice changes nothing. // says so; one applied twice changes nothing. The pipeline's revision is the one the event names (design 47 H-A-214); an event
func (ix *Indexer) HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error { // naming none (revision zero, said before the field) is of the source's lowest-numbered revision, the take-in's.
func (ix *Indexer) HeardPipelineWanted(ctx context.Context, branch string, source, revision int64, outcome string) error {
if ix.Pins == nil || ix.Revisions == nil { if ix.Pins == nil || ix.Revisions == nil {
return errors.New("the indexer keeps no state " + PinsState) return errors.New("the indexer keeps no state " + PinsState)
} }
@@ -128,12 +129,21 @@ func (ix *Indexer) HeardPipelineWanted(ctx context.Context, branch string, sourc
ix.logf("pins: pipeline-wanted of %s %d is %s: it frees no release until its wants are written everywhere (H-A-174)", branch, source, outcome) ix.logf("pins: pipeline-wanted of %s %d is %s: it frees no release until its wants are written everywhere (H-A-174)", branch, source, outcome)
return nil return nil
} }
name, err := revisionOfSource(ctx, ix.Revisions, branch, source) var name string
if err != nil { if revision > 0 {
ix.logf("pins: pipeline-wanted of %s %d frees no release: its revision is not kept: %v", branch, source, err) name = RevisionObject(meshcatalog.CatalogRevision{Branch: branch, Revision: revision})
return nil } else {
var err error
if name, err = revisionOfSource(ctx, ix.Revisions, branch, source); err != nil {
ix.logf("pins: pipeline-wanted of %s %d frees no release: its revision is not kept: %v", branch, source, err)
return nil
}
} }
rev, _, err := ReadRevision(ctx, ix.Revisions, name) rev, _, err := ReadRevision(ctx, ix.Revisions, name)
if revision > 0 && errors.Is(err, ErrNoObject) {
ix.logf("pins: pipeline-wanted of %s revision %d frees no release: %s is not kept", branch, revision, name)
return nil
}
if err != nil { if err != nil {
return fmt.Errorf("%s: %w", name, err) return fmt.Errorf("%s: %w", name, err)
} }
@@ -106,42 +106,54 @@ func TestTheIndexerRemovesARevisionOnlyWhenNoRecordPinsItAndItIsNotCurrent(t *te
t.Fatal(err) t.Fatal(err)
} }
} }
end := func(source int64, outcome, version string) { endOf := func(source, revision int64, outcome, version string) {
t.Helper() t.Helper()
e := runner.PipelineEnded{Branch: "main", Pipeline: fmt.Sprintf("main.%d", source), Revision: source, Source: source, Outcome: outcome, e := runner.PipelineEnded{Branch: "main", Pipeline: fmt.Sprintf("main.%d", revision), Revision: revision, Source: source, Outcome: outcome,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: version}}} Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: version}}}
if err := ix.HeardPins(ctx, nil, &e); err != nil { if err := ix.HeardPins(ctx, nil, &e); err != nil {
t.Fatal(err) t.Fatal(err)
} }
ix.JudgePins(ctx) ix.JudgePins(ctx)
} }
end := func(source int64, outcome, version string) { t.Helper(); endOf(source, source, outcome, version) }
// An event naming no revision (said before the field) is of the source's lowest-numbered revision, the take-in's (H-A-214).
wanted := func(source int64, outcome string) { wanted := func(source int64, outcome string) {
t.Helper() t.Helper()
if err := ix.HeardPipelineWanted(ctx, "main", source, outcome); err != nil { if err := ix.HeardPipelineWanted(ctx, "main", source, 0, outcome); err != nil {
t.Fatal(err) t.Fatal(err)
} }
ix.JudgePins(ctx) ix.JudgePins(ctx)
} }
take(10) take(4)
end(10, runner.PipelineEndedOutcomePassed, "0.1.10") end(4, runner.PipelineEndedOutcomePassed, "0.1.10")
take(11) take(5)
end(11, runner.PipelineEndedOutcomeFailed, "") end(5, runner.PipelineEndedOutcomeFailed, "")
want("a passed release past a later failed pipeline", "main.10", "main.11") want("a passed release past a later failed pipeline", "main.4", "main.5")
take(12) take(6)
end(12, runner.PipelineEndedOutcomePassed, "") end(6, runner.PipelineEndedOutcomePassed, "")
wanted(12, meshwants.PipelineWantedOutcomeRunning) wanted(6, meshwants.PipelineWantedOutcomeRunning)
want("a passed pipeline releasing no version frees no older release", "main.10", "main.12") want("a passed pipeline releasing no version frees no older release", "main.4", "main.6")
take(13) take(7)
end(13, runner.PipelineEndedOutcomePassed, "0.1.13") end(7, runner.PipelineEndedOutcomePassed, "0.1.13")
want("two releases, the unreleased one between them gone", "main.10", "main.13") want("two releases, the unreleased one between them gone", "main.4", "main.7")
wanted(13, meshwants.PipelineWantedOutcomeTierLate) // H-A-174: its later tiers' wants are not written, so it frees nothing wanted(7, meshwants.PipelineWantedOutcomeTierLate) // H-A-174: its later tiers' wants are not written, so it frees nothing
want("a tier-late pipeline-wanted", "main.10", "main.13") want("a tier-late pipeline-wanted", "main.4", "main.7")
wanted(13, meshwants.PipelineWantedOutcomeRunning) wanted(7, meshwants.PipelineWantedOutcomeRunning)
wanted(13, meshwants.PipelineWantedOutcomeRunning) // heard twice: changes nothing more wanted(7, meshwants.PipelineWantedOutcomeRunning) // heard twice: changes nothing more
want("the newer release written everywhere", "main.13") want("the newer release written everywhere", "main.7")
if v, _, _, _ := ix.Pins.Get(ctx, "main.13"); !slices.Equal(mustRecords(t, v), []string{ReleaseRecord("main", "reader")}) { if v, _, _, _ := ix.Pins.Get(ctx, "main.7"); !slices.Equal(mustRecords(t, v), []string{ReleaseRecord("main", "reader")}) {
t.Errorf("the pin of main.13 holds %s, want its release alone", v) t.Errorf("the pin of main.7 holds %s, want its release alone", v)
} }
// H-A-214: a re-pin's revision 8 of the same source 7 releases 0.1.14; its pipeline-wanted names revision 8, so the release
// on revision 8 frees the older one on revision 7, which the source's lowest-numbered revision would not.
take(7)
endOf(7, 8, runner.PipelineEndedOutcomePassed, "0.1.14")
want("a re-pin's release beside the take-in's", "main.7", "main.8")
if err := ix.HeardPipelineWanted(ctx, "main", 7, 8, meshwants.PipelineWantedOutcomeRunning); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
want("the re-pin's release written everywhere", "main.8")
} }
func mustRecords(t *testing.T, v []byte) []string { func mustRecords(t *testing.T, v []byte) []string {
@@ -168,21 +180,22 @@ func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
before []int64 // sources taken in with no mover before the take-in of source 5 before []int64 // sources taken in with no mover before the take-in of source 5
whileRuns []string whileRuns []string
afterTheEnd []string afterTheEnd []string
base int64 // the base main.5 names (H-A-154) at int64 // the number the take-in of source 5 keeps: the branch's newest + 1 (H-A-213 (1))
base int64 // the base that revision names (H-A-154)
passed bool // main.5's pipeline passed: its release pins main.5 passed bool // main.5's pipeline passed: its release pins main.5
unreleased bool // its mover released no version there, so nothing pins main.5 (H-A-173) unreleased bool // its mover released no version there, so nothing pins main.5 (H-A-173)
then []int64 // sources taken in with no mover after the end then []int64 // sources taken in with no mover after the end
released []string released []string
}{ }{
{name: "the previous revision kept and pinned", before: []int64{3, 4}, whileRuns: []string{"main.4", "main.5"}, afterTheEnd: []string{"main.5"}, base: 4}, {name: "the previous revision kept and pinned", before: []int64{3, 4}, at: 3, whileRuns: []string{"main.2", "main.3"}, afterTheEnd: []string{"main.3"}, base: 2},
{name: "the newest kept before, whatever its number", before: []int64{2}, whileRuns: []string{"main.2", "main.5"}, afterTheEnd: []string{"main.5"}, base: 2}, {name: "the newest kept before, whatever its source", before: []int64{2}, at: 2, whileRuns: []string{"main.1", "main.2"}, afterTheEnd: []string{"main.2"}, base: 1},
{name: "no revision before it pins only its own", whileRuns: []string{"main.5"}, afterTheEnd: []string{"main.5"}}, {name: "no revision before it pins only its own", at: 1, whileRuns: []string{"main.1"}, afterTheEnd: []string{"main.1"}},
{name: "a passed pipeline's release keeps its revision once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"}, {name: "a passed pipeline's release keeps its revision once no longer current", before: []int64{4}, at: 2, whileRuns: []string{"main.1", "main.2"},
base: 4, passed: true, then: []int64{6}, afterTheEnd: []string{"main.5", "main.6"}, released: []string{"main.5"}}, base: 1, passed: true, then: []int64{6}, afterTheEnd: []string{"main.2", "main.3"}, released: []string{"main.2"}},
{name: "a passed pipeline whose mover released no version pins nothing", before: []int64{4}, whileRuns: []string{"main.4", "main.5"}, {name: "a passed pipeline whose mover released no version pins nothing", before: []int64{4}, at: 2, whileRuns: []string{"main.1", "main.2"},
base: 4, passed: true, unreleased: true, then: []int64{6}, afterTheEnd: []string{"main.6"}}, base: 1, passed: true, unreleased: true, then: []int64{6}, afterTheEnd: []string{"main.3"}},
{name: "a failed pipeline's revision goes once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"}, {name: "a failed pipeline's revision goes once no longer current", before: []int64{4}, at: 2, whileRuns: []string{"main.1", "main.2"},
base: 4, then: []int64{6}, afterTheEnd: []string{"main.6"}}, base: 1, then: []int64{6}, afterTheEnd: []string{"main.3"}},
} { } {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Pins: newMem(), Findings: newFindings(), ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Pins: newMem(), Findings: newFindings(),
@@ -206,18 +219,19 @@ func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
keep(s, nil) keep(s, nil)
} }
keep(5, mover) keep(5, mover)
if rev, _, err := ReadRevision(ctx, ix.Revisions, "main.5"); err != nil || rev.Base != tc.base { at := fmt.Sprintf("main.%d", tc.at)
t.Errorf("main.5 names the base %d (%v), want %d, the revision its take-in pinned (H-A-154)", rev.Base, err, tc.base) if rev, _, err := ReadRevision(ctx, ix.Revisions, at); err != nil || rev.Base != tc.base || rev.Source != 5 {
t.Errorf("%s names the source %d and the base %d (%v), want 5 and %d, the revision its take-in pinned (H-A-154)", at, rev.Source, rev.Base, err, tc.base)
} }
if got := kept(); !slices.Equal(got, tc.whileRuns) { if got := kept(); !slices.Equal(got, tc.whileRuns) {
t.Errorf("while the pipeline runs: kept %v, want %v", got, tc.whileRuns) t.Errorf("while the pipeline runs: kept %v, want %v", got, tc.whileRuns)
} }
for _, k := range tc.whileRuns { for _, k := range tc.whileRuns {
if v, _, _, _ := ix.Pins.Get(ctx, k); !slices.Equal(mustRecords(t, v), []string{"pipeline.main.5"}) { if v, _, _, _ := ix.Pins.Get(ctx, k); !slices.Equal(mustRecords(t, v), []string{"pipeline." + at}) {
t.Errorf("the pin of %s holds %s, want the pipeline's record", k, v) t.Errorf("the pin of %s holds %s, want the pipeline's record", k, v)
} }
} }
end := runner.PipelineEnded{Branch: "main", Pipeline: "main.5", Revision: 5, Outcome: runner.PipelineEndedOutcomeFailed, end := runner.PipelineEnded{Branch: "main", Pipeline: at, Revision: tc.at, Outcome: runner.PipelineEndedOutcomeFailed,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: "0.1.5"}}} Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: "0.1.5"}}}
if tc.unreleased { if tc.unreleased {
end.Movers[0].Version = "" end.Movers[0].Version = ""
@@ -0,0 +1,100 @@
package index
import (
"context"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Design 47 H-A-204, H-A-201, H-A-165: an entry's packages name every requirement of its module graph of another
// repository: one the form's versions leave out (an import form naming only what the build imported) is pinned by its
// series' newest published version, and one none publishes yet is kept with no version, never left out.
func TestAnEntryNamesEveryRequirementPinnedOrWaiting(t *testing.T) {
const (
wireguard = "git.novox.be/novox/mesh-networking/modules/mesh-wireguard/go"
busgrants = "git.novox.be/novox/mesh-delivery/modules/mesh-grants/busgrants"
sdk = "git.novox.be/novox/mesh-sdk/go"
)
series := map[string]SeriesPackage{"wg": {Name: "mesh-wireguard/go", GoModule: wireguard, Series: catalogue.DefaultSeries,
Newest: &meshcatalog.PackagesByPackageSeriesValueNewest{Version: "0.1.47"}}}
pins := releasedPins(sdkstep.PackageVersions{sdk: "c0ffee"}, series, catalogue.Manifest{}, "novox/mesh-catalog", []string{busgrants, sdk, wireguard})
want := sdkstep.PackageVersions{sdk: "", wireguard: "v0.1.47", busgrants: ""}
if len(pins) != len(want) {
t.Fatalf("pins = %v, want %v", pins, want)
}
for p, v := range want {
if got, named := pins[p]; !named || got != v {
t.Errorf("the pin of %s is %q (named %v), want %q", p, got, named, v)
}
}
}
// Design 47 H-A-204, H-A-165: a release re-pins every kept entry naming one of its packages, of any repository and
// changed or not, by its pins or by its kept requirements alone: here g, of another repository, holds no pin of mesh-sdk.
// H-A-211, H-A-212, H-A-213: on every watched branch holding g, the re-pin keeps the branch's next revision, of the same
// source as the one before it, with the call "release of mesh-sdk 0.1.27" and g as its mover at its new fingerprint, and
// says catalog-revision with that call.
func TestAReleaseRepinsAnEntryNamingItsPackageWithNoPin(t *testing.T) {
ctx := context.Background()
objects := map[string]string{"modules/b/builds": strings.Repeat("b", 40), "modules/c/client": strings.Repeat("c", 40)}
ix, mods, _ := ptpIndexer(t, objects)
ix.Revisions, ix.Packages = newMemObjects(1<<20), newMem()
reportToolchain(t, ix, "mesh-build-go", "n1", "go", "1.26.0")
var said []meshcatalog.CatalogRevisionEvent
ix.SayCatalogRevision = func(_ context.Context, e meshcatalog.CatalogRevisionEvent) error { said = append(said, e); return nil }
f := withManifests(mergedMsg("novox/mesh-delivery", map[string]string{"modules/g": "gtree"}), map[string]string{"modules/g": ptpProgram}, nil)
if err := ix.Apply(ctx, f); err != nil {
t.Fatal(err)
}
g, _ := moduleOf(t, mods, "g")
if g.Inputs2 == nil || !slices.Contains(g.Inputs2.Packages, "git.novox.be/novox/mesh-sdk/go") {
t.Fatalf("g = %+v, want mesh-sdk among its kept requirements", g.Inputs2)
}
g.Packages = nil
mods.put("g", mustEncode(t, g))
mods.put("mesh-sdk", mustEncode(t, meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-sdk", Commit: strings.Repeat("1", 40),
Path: meshcatalogclient.RootPath, None2: None2Pending}))
entries, err := ix.viewEntries(ctx)
if err != nil {
t.Fatal(err)
}
sources := map[string]int64{"main": 5, "feature/x": 9}
for branch, src := range sources {
if err := ix.KeepBranchEntries(ctx, branch, src, entries, nil); err != nil {
t.Fatal(err)
}
}
said = nil
if _, err := ix.HearVersion(ctx, HeardVersion{Package: "mesh-sdk/go", GoModule: "git.novox.be/novox/mesh-sdk/go", Version: "0.1.27", Published: true}); err != nil {
t.Fatal(err)
}
if n, err := ix.Repin(ctx, "mesh-sdk", "0.1.27"); err != nil || n != 2 {
t.Errorf("a release of mesh-sdk re-pinned %d modules (%v), want g on each of the two branches, which requires it with no pin", n, err)
}
now, _ := moduleOf(t, mods, "g")
if now.Fp2 == "" || now.Fp2 == g.Fp2 || now.Packages["git.novox.be/novox/mesh-sdk/go"] != "v0.1.27" {
t.Fatalf("g after the re-pin = %s pinned %v, want a new fingerprint pinned to v0.1.27 (was %s)", now.Fp2, now.Packages, g.Fp2)
}
for branch, src := range sources {
rev, kept, found, err := readNewestOf(ctx, ix.Revisions, branch)
if err != nil || !found || rev.Revision != 2 || rev.Source != src || rev.Call != "release of mesh-sdk 0.1.27" ||
rev.CallKind != meshcatalog.CatalogRevisionCallKindRelease || rev.Module != "mesh-sdk" || rev.Version != "0.1.27" || // H-A-215
len(rev.Movers) != 1 || rev.Movers[0].Module != "g" || rev.Movers[0].Fingerprint != now.Fp2 {
t.Errorf("%s's newest = %+v (%v), want revision 2 of source %d, its call the release, g its mover at %s", branch, rev, err, src, now.Fp2)
continue
}
if i := slices.IndexFunc(kept, func(e meshcatalog.CatalogEntry) bool { return e.Module == "g" }); i < 0 || kept[i].Fp != now.Fp2 {
t.Errorf("%s's newest holds no entry of g at %s", branch, now.Fp2)
}
}
if len(said) != 2 || said[0].Call != "release of mesh-sdk 0.1.27" || said[1].Call != said[0].Call ||
said[0].CallKind != meshcatalog.CatalogRevisionEventCallKindRelease || said[1].Module != "mesh-sdk" || said[1].Version != "0.1.27" {
t.Errorf("said %+v, want catalog-revision of each branch with the release's call", said)
}
}

Some files were not shown because too many files have changed in this diff Show More