Compare commits

...
Author SHA1 Message Date
jschoubben c062d24dfb Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7) 2026-11-20 20:25:02 +01:00
mesh-tools a7fc37561e Merge pull request 'Design 47 H1: a pin is the version the registry published for the package path within the module's accepted series (H-A-183, H-A-184, H-A-185)' (#287) from fix/47-H1-pins-by-package-path into main 2026-11-20 19:15:04 +00:00
jschoubben e93575ffe7 mesh-delivery-indexer: the accepted series is the manifest's, contents or none
The pin fell back to the path's newest publish while no series value
held contents, which live none does, so a publish in a series a module
does not accept pinned it. The series a module accepts is the one its
manifest names, or the default; the pin is the newest version published
for the package path within it.

Builds: design 47 H H-A-185, H-A-184
2026-11-20 20:13:15 +01:00
jschoubben 859f88b2f0 mesh-delivery-indexer: a pin stays within the series the module accepts
A pin by the package path across every series moved a module to a
series it does not accept while its contents came from the accepted
one. The pin is the accepted series' newest published version where
that series gives the contents; only where none does yet is it the
newest published of the path. A publish in another series pins nothing
and moves nothing.

Builds: design 47 H H-A-184, H-A-183
2026-11-20 20:07:46 +01:00
jschoubben 517a4b3cf1 mesh-delivery-indexer: a pin is the version the registry published for the package's path
At main.39 every importer of a nested package was pinned to the release
number of the module holding it (v0.1.97 of the builder for its package
builds), and the Go seat found no tag <package dir>/v0.1.97. A Go
package's version is its tag: the pin is now the newest version the
registry published for that package path (the seats' published, kept in
the package's series), never a module's release; a package the
registry never published stays unpinned.

Builds: design 47 H H-A-183, H-A-167, G §370
2026-11-20 20:05:52 +01:00
mesh-tools 7f88916863 Merge pull request 'Design 47 H1: the forge bridge keeps only an act's outcome and pull request; the act stays in the gate's stream (H-A-181, H-A-182)' (#286) from fix/47-H1-bridge-performed-cap into main 2026-11-20 18:46:57 +00:00
jschoubben 5dee4cb263 mesh-delivery-forge-bridge: owed again refuses a performed value of the old form
A value kept before this change names no pull request, so bridge_owed_again read the subject owed..0 and said the
stream no longer held the act while it did. Every act the bridge decodes names its repository, so a value without
one is the old form: refused with what is true, never read again; it expires with the state's TTL.

Builds: design 47 G §163; H-A-182
2026-11-20 19:45:38 +01:00
jschoubben 77b683c1c1 mesh-delivery-forge-bridge: performed keeps the outcome and the pull request, not the act
The state performed filled to 77% of its template's 64 MiB (19,530 keys, 2.6 KB each): the bridge kept every owed
event whole beside its outcome. performed's value is now the outcome said as owed.ended, with when, and the
repository and number the act names (groups.Performed); bridge_owed_again reads the act from the gate's stream owed
on its pull request's subject through an ordered consumer, the stream being the act's one copy; bridge_owed reads
one act or one pull request's acts, never the whole state, whose answer had grown past what the bus carries.

Builds: design 47 G §163, §162, §168; H-A-181, H-A-182
2026-11-20 19:38:34 +01:00
mesh-tools 60e32772a5 Merge pull request 'Design 47 H1: the composition step of a module no machine holds ends nothing to run (H-A-179)' (#285) from fix/47-H1-composition-nothing-to-run into main 2026-11-20 18:08:15 +00:00
jschoubben fd2ef6c7e8 assigner: a mover no machine holds has nothing to compose (H-A-179)
A package module with a required composition step and no node placing
it ended could not run, so its pipeline could never pass. A step with
nothing to judge ends nothing to run, which counts as passed, saying why.

Builds: design 47 H H-A-179
2026-11-20 19:05:49 +01:00
mesh-tools 71371df160 Merge pull request 'Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7)' (#284) from proof/47-H1-two-modules-2 into main 2026-11-20 17:55:44 +00:00
jschoubben b2d5644c34 Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7) 2026-11-20 18:55:12 +01:00
mesh-tools 9598b92c39 Merge pull request 'Design 47 H1: the runner's integration tests make the module's findings stream (H §743)' (#283) from fix/47-H1-runner-queue-test into main 2026-11-20 17:37:22 +00:00
mesh-tools f00d34673f Merge pull request 'Design 47 H1: the indexer's runs-candidate test is folded into TestIndexerWhole and waits for the run-started's acknowledgement (H §743)' (#282) from fix/47-H1-indexer-runs-test into main 2026-11-20 17:35:17 +00:00
jschoubben 87357d84bd mesh-delivery-pipeline: the queue and run tests make the module's findings stream
Built against mesh-sdk at its pin (H-A-166, H-A-169), the runner counts every
event no handler takes as a finding; with no stream for its finding events each
write waited its 3 s for the bus, holding the event loop, so the queue test ran
into its 90 s bound. The tests now make the stream the manifest asks for.

Builds: design 47 H H-A-176, H-A-175 (the tests of the runner's placing), H-A-166, H-A-169
2026-11-20 18:31:59 +01:00
jschoubben 08dde53e33 mesh-delivery-indexer: the run on request's bed lives in TestIndexerWhole, its restart after the run-started's acknowledgement
TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded failed on the build node
under load (pipelines main.29 and main.31). It restarted the indexer as soon as the candidate was
written, before the run-started was acknowledged (the write comes first), and it polled both
waits with sleeps: the same kind as the toolchains bed's restart fixed in 20a7bd0f. A stop
between the write and the acknowledgement leaves the run-started for redelivery after its
acknowledgement wait, which writes the candidate again once its run has ended.

The run is now one more step of TestIndexerWhole around its restart: run-started of run 9 writes
candidates.run-9 and the run's key (watched, not polled), the restart waits for the run-started's
acknowledgement beside the source-moved's, and the member's pipeline-ended after the restart drops
both. The separate test is removed: the module's integration tests go from 541 to 540; no test
added.

Builds: design 47 H §743
2026-11-20 18:30:27 +01:00
mesh-tools c772eb0683 Merge pull request 'Design 47 H1: a build module's inputs do not wait for its toolchain; a cycle among unplaced movers ends loud (H-A-175, H-A-176)' (#281) from fix/47-H1-runner-order-cycle into main 2026-11-20 17:17:28 +00:00
jschoubben a7b2fb0ffa mesh-delivery-pipeline: a cycle ends only its own steps; the end judgement ends the pipeline
A cycle ended every open step of the pipeline at once and named edges outside
it. It now ends not asked only the steps of the movers without steps, names
the edges inside a strongly connected group alone, and leaves the pipeline's
end to the end judgement, so running steps run to their end.

Builds: design 47 H H-A-176, H-A-175, H-A-163, G §270, G-22 A-326
2026-11-20 18:15:47 +01:00
jschoubben ef086e0b75 mesh-delivery-pipeline: a cycle among unplaced movers ends the pipeline failed
A cycle the running build module does not break left a pipeline running with
journal lines only, since validate is asked only for a placed mover. The
placing now ends it failed, its why naming each artifact and edge kind.

Builds: design 47 H H-A-175, G §270, G-22 A-326, H-A-101 (2)
2026-11-20 18:12:04 +01:00
jschoubben a8adda1a8c mesh-delivery-pipeline: a build module's own inputs await not its toolchain edge
A pipeline moving the Go build module with a package it imports (mesh-sdk,
mesh-delivery-contracts) placed nothing: the package's build awaited the new
build module's files, the build module awaited the package. What a build
module's new version is built from is built by its running version.

Builds: design 47 G §270, G-22 A-326, H §746, H-A-83 (2)
2026-11-20 18:09:07 +01:00
mesh-tools e1f0c1a1cc Merge pull request 'Design 47 H1: a release pins the revision its entry lives in; freed by the module's next release (H-A-168, H-A-170, H-A-173, H-A-174)' (#280) from fix/47-H1-release-pins into main 2026-11-20 17:03:27 +00:00
jschoubben bf24bb9740 mesh-delivery-indexer: a tier-late pipeline-wanted frees no release
A tier-late pipeline's later-tier movers released a version, so the newer revision holds their
release record, yet their wants were not written: freeing their older release on the tier-late
event would remove the revision their want names. HeardPipelineWanted now reads the event's outcome:
tier-late frees nothing, and the same pipeline's later pipeline-wanted with outcome running frees as
before. The pins test's freeing sequence says a tier-late event first (nothing freed), then the
running one (the older release freed); the events test's row hands the outcome on. No test added.

Builds: design 47 H-A-174 (hq design/47-H1-answers-59 84e2514d), H-A-173
2026-11-20 18:01:56 +01:00
jschoubben 814244fd5a mesh-delivery-indexer: a release pin only for a released mover, freed only by a newer release of it
A passed pipeline's movers include ones the assigner never takes (superseded, no version released,
a tier-late pipeline's later tiers); freeing their older release on the newer pipeline-wanted would
remove the revision their want still names, and a mover releasing nothing would pin a revision per
pipeline for good. The release record is now written only for a mover that released a version in
the pipeline, and pipeline-wanted frees a module's older record only where the newer pipeline's own
revision holds that module's release record.

Tests: a pins table row (a passed mover with no version pins nothing), the freeing sequence (a
passed pipeline releasing no version frees no older release, the next one that releases does), and
TestIndexerWhole now says a pipeline-wanted on the bus after main.12's passed end and waits for
main.11's release to be freed and main.11 removed, so the indexer's hearing of pipeline-wanted is
tested end to end (review round 1). No test added.

Builds: design 47 H-A-173 (hq design/47-H1-answers-59 21297155), H-A-170, H-A-168
2026-11-20 17:58:45 +01:00
mesh-tools 442b03f61d Merge pull request 'Design 47 H1: package edges from go.mod; a pin is the newest released version, in every form's fingerprint; a release re-pins its importers (H-A-157, H-A-161, H-A-165, H-A-167, H-A-172)' (#279) from fix/47-H1-package-edges-from-gomod into main 2026-11-20 16:53:25 +00:00
mesh-tools fb23967de3 Merge pull request 'Design 47 H1: a pipeline ends failed as soon as it can no longer pass; end-again judges it by hand (H-A-163, H-A-164, H-A-171)' (#278) from fix/47-H1-runner-ends-multi-mover into main 2026-11-20 16:51:19 +00:00
jschoubben 9083287d57 mesh-delivery-indexer: a passed pipeline's release pins its revision until a newer one is the want
The pin judge removed main.23 and its entries when main.24 became current, while the assigner had
just written main.23's builder as the want: the node-engine asked its entry and was told none is
kept, holding the want for good. A passed pipeline-ended now pins the pipeline's kept revision
under a record release.<branch>.<module>, one per mover, beside the pipeline's own pin; the seat
mesh-wants' pipeline-wanted of a newer pipeline on that branch (its wants written to the end,
tier-late included) frees that record from every revision below the newer one, for each module the
newer revision's movers name. The indexer consumes pipeline-wanted (its manifest says so). A
revision a release pin holds keeps its entries, as removeUnlisted already does for any kept revision.

Tests: rows in the pins table (a passed pipeline's revision kept once no longer current; a failed
one's goes), the freeing in the pins test (a release past a failed pipeline, freed by the next
passed one's pipeline-wanted, twice changing nothing), and a pipeline-wanted row in the events
test. TestIndexerWhole now reads main.11's release beside its validate base, and waits for the held
source-moved's acknowledgement before it reads the consumer (the write comes first). No test added.

Builds: design 47 H-A-168, H-A-170 (hq design/47-H1-answers-59 87b43ec8), H-A-147, H §749
2026-11-20 17:50:08 +01:00
jschoubben 07dd8ea842 Design 47 H1: the end names why each mover has no step, its registration or a publish (H-A-171)
The end judgement took every mover with no step as waiting on its
registration; one may wait for the publish of a mover it builds on. The
registration lookup tells them apart, and the why names each with its own
reason.

Builds: design 47 H §746, H-A-163, H-A-171, G A-327
2026-11-20 17:48:10 +01:00
jschoubben d0436bb38c mesh-delivery-indexer: pins are in every form's fingerprint
The import form's fingerprint left the pins out, so an importer whose
trunk build reported its imports never moved on a re-pin and was never
built again against the release. Every form now takes its sorted pins;
a build that stands keeps its fingerprint while its pins are the ones
it was built with, and a fingerprint with no pin stays as it was.

Builds: design 47 H H-A-172, H-A-167
2026-11-20 17:46:53 +01:00
jschoubben fe51379ee2 mesh-delivery-indexer: a release re-pins or is not acknowledged; pins enter the version line's fingerprint
A re-pin that left the fingerprint as it was left the build made
against the older pin as the one the mesh runs. The version line's
fingerprint now takes the pins, so a re-pin moves the importer, which
becomes a mover built against the release. An accepted series' older
version no longer stands over a newer release. A re-pin that fails
returns its error before the heard artifacts are forgotten, and a
package-published heard again for a released version re-pins, so the
redelivery recovers it. The release path and the seed's pins carry
tests that fail without them (review round 2).

Builds: design 47 H H-A-165, H-A-167, H-A-161
2026-11-20 17:46:53 +01:00
jschoubben 037cb9555e mesh-delivery-indexer: a version-released re-pins the modules holding the package
Pins moved to a new released version only at the next merge, toolchain
report or hand recompute. The release now starts the pass of H-A-161
for the modules whose packages hold a package of the released module
alone; every other module stands as held.

Builds: design 47 H H-A-165 (the indexer's part), H-A-161
2026-11-20 17:46:53 +01:00
jschoubben 5c5136d6d1 mesh-delivery-indexer: the requirements state reads the go.sum whatever packages the reader carried
With the go.mod's mesh requires carried at every take-in, requirementsOf
returned early for every module requiring a mesh package: its go.sum was
never read and keepRequirements never cleared an old version it named
(indexer_requirements for mesh-sdk still names v0.1.13). The go.sum is
read whenever its hash moved or this process has not kept it yet.

Builds: design 47 H H-A-157 (review round 1)
2026-11-20 17:46:53 +01:00
jschoubben c43e7a5d1c mesh-delivery-indexer, mesh-delivery-builder: a package is pinned by its released version, never by a commit
The indexer filled a module's packages with the commit of a module value
it holds, which Go turned into a pseudo-version of whatever that tree
was (mesh-delivery-sdk at an 11-19 commit, nodeengine pre-H1). A pin is
now the newest released version of the module holding the package (the
versions state version-released says), an accepted series' version
stands, and a package with none released is kept with no version: its
kind-1 edge stands and the builder asks no pin for it. A commit of
another repository moves a module only through a package of it the
module imports (A-476a), now that pins name no commit.

Builds: design 47 H H-A-161, G §370, H §746, H-A-157 (kind-1 edges)
2026-11-20 17:46:53 +01:00
jschoubben e04321acae mesh-delivery-indexer: a build's reported imports refine the go.mod's packages, never empty them
An import form whose imports name no package of another repository, or a
build that stands with versions held empty from the go.sum era, wrote an
empty package set while the go.mod requires a mesh package: no kind-1
edge. The version line's versions of the go.mod's set stand then.

Builds: design 47 H H-A-157, G §47 (kind 1)
2026-11-20 17:46:53 +01:00
jschoubben 4c2914194b mesh-delivery-indexer: a module's mesh packages are what its go.mod requires
At main's revision 21, 64 of 82 Go modules carried no package edge: the
version line read the mesh packages go.sum names, and go.sum names none
required at the zero version (ADR 0573 §2). The take-in now reads them
from the go.mod's require lines, at any version, with no build needed;
the go.sum is read only for the requirements state. The sibling rule
(replaces by a directory) stands.

Builds: design 47 H H-A-157, H-A-136 (sibling rule), G §47 (kind 1)
2026-11-20 17:46:53 +01:00
mesh-tools 277c53067e Merge pull request 'Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7)' (#273) from proof/47-H1-two-modules into main 2026-11-20 16:44:51 +00:00
mesh-tools 227f6e68ce Merge pull request 'Design 47 H1: a build with an unpinned package waits for its release (H-A-165)' (#277) from fix/47-H1-builder-waits-package into main 2026-11-20 16:41:47 +00:00
jschoubben 521cbf089e Design 47 H1: the seat mesh-pipeline-runner gains end-again, a hand act judging a pipeline's end once more (H-A-164)
A pipeline the end judgement before H-A-163 left running had no way to end
but a cancel. The runner's process answers end-again, so the end takes its
turn in the queue; runner_end_again asks it. The seat's definition changes:
its Go and TypeScript contracts are generated again.

Builds: design 47 H §746, H-A-163, H-A-164
2026-11-20 17:38:20 +01:00
jschoubben 39a2c6a944 Design 47 H1: a pipeline ends failed once it can no longer pass, a mover waiting on its registration named (H-A-163)
main.20 and main.22 never ended: a mover with no step placed (no registration)
held the end even after a placed mover's required step failed and nothing was
open. It now holds the end only while the pipeline could still pass.

Builds: design 47 H §746, H-A-163, G A-327
2026-11-20 17:38:20 +01:00
jschoubben 577ec05205 Design 47 H1: the runner says a mover waits on its registration when placed from the half or the queue (A-327)
A pipeline placed at its turn in the queue, or once its branch's half was held,
dropped its movers with no registration without a word: main.20 and main.22 sat
running with every step ended and nothing said why (their mover
mesh-delivery-contracts has no registration, so it is never placed and the
pipeline never ends).

Builds: design 47 G A-327, design 47 H §746
2026-11-20 17:38:20 +01:00
jschoubben 744204c5ab Builder: releases heard wake the package wait; its setting package-wait-minutes (H-A-165)
The process tells the builder each version-released it takes in and
gives it the newest release its record keeps, so an unpinned package's
step ask wakes on the event and pins the release; the wait's bound is
the setting package-wait-minutes (default 10), carried as a file under
the state dir like revision-wait-minutes.

Builds: design 47 H H-A-165, H-A-156, H-A-155
2026-11-20 17:37:25 +01:00
jschoubben 0f41b405ba Builder: a step ask naming a package with no pin waits for its release (H-A-165)
An entry keeps a mesh package with no released version unpinned
(H-A-161). Its step ask now waits, in the wait of H-A-155, for the
release of the module holding the package, heard or kept in the
builder's record of releases, and builds with that release as the pin;
past the bound the step fails naming the package and builds nothing.

Builds: design 47 H H-A-165, H-A-155, H-A-161
2026-11-20 17:37:25 +01:00
mesh-tools 20f0e9baaf Merge pull request 'Design 47 H1: a build waits for its revision and never builds without a version; a run's ask names no source (H-A-155, H-A-156, H-A-162)' (#276) from fix/47-H1-builder-waits-revision into main 2026-11-20 16:35:46 +00:00
mesh-tools 75c636a5ec Merge pull request 'Design 47 H1: the builder numbers nothing of its own; a published of other bytes leaves a kept entry (H-A-159)' (#275) from fix/47-H1-builder-versions-record into main 2026-11-20 16:32:12 +00:00
jschoubben 0cbabe9860 Runner and builder: a run's build step ask names no source; a held revision wakes the builder's wait (H-A-162)
A run on request copied its pipeline's source onto its build step ask,
so the builder waited for the run's candidate fingerprints in its
picture of the branch, where they never come, and failed every run past
the bound. The seat mesh-builds' contract leaves a run's source absent:
the runner now does. A revision the builder's view takes in whole that
changes no entry now also wakes a waiting ask, told by the catalog
client's new Held.

Builds: design 47 H H-A-162, H-A-155
2026-11-20 17:30:56 +01:00
jschoubben d2c9dc7de6 builder: builder_floor_again refuses; it asks no floor any more
With nothing numbered of the builder's own, the tool answered that a floor
would be asked at the next claim and looked again, doing nothing.

Builds: design 47 H H-A-159, G §369
2026-11-20 17:30:49 +01:00
jschoubben 53199f16ce Builder: the revision wait reads the picture and its setting revision-wait-minutes (H-A-155, H-A-156)
The process gives the builder its picture of each watched branch (the
source revision the branch's view holds) and the bound of a step ask's
wait for its revision, read from the setting revision-wait-minutes
(default 2) carried as a file under the state dir like
toolchain-wait-minutes.

Builds: design 47 H H-A-155, H-A-156, H-A-141
2026-11-20 17:27:37 +01:00
jschoubben 256eb34a0d Builder: a build step ask waits for its revision before it builds (H-A-155)
A step ask reached the builder before its picture of main's catalog
revisions held the revision the pipeline builds; the fingerprint was
built as no first branch's build, reserved no version, and the step
failed on builder_reserve_done's words. The ask now waits, woken by each
entry the picture takes in, until the picture holds its source revision
and the asked entry from its branch; past the bound it fails naming the
revision and builds nothing.

Builds: design 47 H H-A-155, H-A-141
2026-11-20 17:27:37 +01:00
jschoubben a47c8e7922 builder: number nothing of its own; the reservation numbers
Every look numbered one more patch in the state versions (claims carry no
source revision since H-A-48): nodeengine 0.1.5 to 0.1.36, beside the
reservation's numbers.

Builds: design 47 H H-A-159, G §348, §349, §369
2026-11-20 17:24:48 +01:00
jschoubben 16f9ad2950 builder: a published of other bytes leaves a kept versions entry as it is
The reservation's published of nodeengine 0.1.4 (9724da7a) was taken into the
state versions' entry 0.1.4, numbered for an older fingerprint, which turned
published and kept the old digest c0ef9a3d.

Builds: design 47 H H-A-159, G §369, A-415; ADR 0605 §4
2026-11-20 17:23:19 +01:00
mesh-tools b1f25f91d5 Merge pull request 'Design 47 H1: validate reads the base the take-in names on the catalog revision; the indexer's restart tests wait for the bus (H-A-151, H-A-154, H §743)' (#274) from fix/47-H1-indexer-toolchains-test into main 2026-11-20 16:09:50 +00:00
jschoubben d281faeeb7 mesh-delivery-assigner: the validate test judges a revision whose base is below its number minus one
Every revision the validate test judged with a base had it at its number minus one, so the test
passed with the case before read as the number before (review round 1). Revision 12 now names base
7 with 8 to 11 never kept: it ends manifest-refused against base 7's manifests, and could not run
when the case before is read as the number minus one. No test added.

Builds: design 47 H-A-154 (hq design/47-H1-answers-59 01842bbe), H-A-151
2026-11-20 17:08:23 +01:00
jschoubben e312999843 Design 47 H1: proof 2, a two-module change along a dependency edge delivered by its pipeline (H §5 step 7) 2026-11-20 17:06:04 +01:00
jschoubben 5dba7fe68b mesh-catalog's revision names its base; validate's case before reads it, not the number before
A take-in that moves nothing keeps no revision for its source number, so validate's case before,
read as the step's revision minus one, named a revision never kept and could not run. The take-in
now writes, on the catalog revision it keeps, the field base: the newest kept revision below it,
the one it pins for its pipeline, in the same write; absent on the branch's first take-in. The
assigner's validate reads the base from the revision it already asks for; where the revision names
none the case before is empty and its end says so, never a could-not-run.

- mesh-delivery-contracts: the seat mesh-catalog's CatalogRevision gains base (schema, fixture,
  generated Go and TypeScript by seatgen).
- mesh-delivery-indexer: keepBranchEntries names the base it pins; the pin test's rows check it.
- mesh-delivery-assigner: before reads the base; the validate test asks a revision with none, and
  both stand-ins name the base as the take-in does.

Builds: design 47 H-A-151, H-A-154 (hq design/47-H1-answers-59 01842bbe), H-A-147, H-A-108 (3), design 47 H §743
2026-11-20 17:00:55 +01:00
mesh-tools f0f28c56a4 Merge pull request 'Design 47 H1: an assignment no watched branch lists is left out of a machine's composition, said and raised as a finding (H-A-148, H-A-149, H-A-152, H-A-153)' (#272) from fix/47-H1-unlisted-assignment-left-out into main 2026-11-20 15:54:36 +00:00
jschoubben 20a7bd0f3c mesh-delivery-indexer: the restart tests wait for the take-in's end and the bus's close
TestIndexerCountsOnlyThePlacedBuildNodesToolchains failed on the build node under load
(pipeline main.18) in two ways, both of the test, not of the indexer's count of the
placed build nodes' toolchains:

- it restarted the indexer as soon as the module was written, before its take-in
  ended: the stop cancelled the take-in, its source-moved stayed unacknowledged, and
  the branch's durable (one message at a time) held the next one behind it for its
  acknowledgement wait, past the test's 20 s bound. Each restart now waits for the
  source-moved's acknowledgement; TestIndexerWhole's restart the same way.
- a source-moved said right after a restart could reach the stopped program's pull,
  still counted until the bus closed its connection. restart now returns only once
  the bus has closed every connection of the stopped program, heard on the system
  account's disconnect events, with no sleep.

Under load (8 runs side by side, GOMAXPROCS=1) the test failed 5 of 80 runs before,
2 of 240 with the first change alone, and 0 of 600 with both; TestIndexerWhole 0 of 600.
No test added.

Builds: design 47 H §743
2026-11-20 16:54:26 +01:00
jschoubben 0b1ce08985 assigner: one set of unlisted assignments, carried on the run (H-A-152, H-A-153)
The leaving-out took every assigned module the case's manifests lacked,
while the judgement said only those the watched branches' newest
revisions list nowhere, read again at judgement time. Now the set is read
once from the newest revisions when the run starts, written on the run,
and used for the leaving-out, the finding and every step's judgement and
the resume. The candidate never enters it: a module the change drops
while still assigned stays in and fails the composition as its own
fault. While no newest revision is followed nothing is left out.

Builds: design 47 H H-A-152, H-A-153
2026-11-20 16:51:28 +01:00
jschoubben f053e3b041 mesh-wants: a composition run carries its unlisted assignments (H-A-153)
The judgement of every step reading a run, and the resume after a
restart, read only the stored run; the set of assignments left out is
written there when the run starts, so it is not kept in a step's memory.

Builds: design 47 H H-A-153
2026-11-20 16:47:57 +01:00
jschoubben d428650ce4 assigner: raise assignment-unlisted where a step leaves one out, clear it in the pass (H-A-149)
A stale assignment left out of a machine's composition is now a finding the
condition keeper carries: kind assignment-unlisted, scope delivery, id
<node>.<module>, a warning, raised by the composition step that leaves it
out. The pass over the assignments clears it when the assignment is gone or a
watched branch of the module lists it again, so the clear never waits for the
next step on that machine.

Builds: design 47 H H-A-149, H-A-148
2026-11-20 16:31:50 +01:00
mesh-tools 351eda0867 Merge pull request 'indexer: a take-in pins the branch's previous revision for its pipeline, so validate's base stays (H-A-147)' (#271) from fix/47-H1-validate-previous-revision into main 2026-11-20 15:31:05 +00:00
jschoubben 5e3d92ae33 assigner: leave a stale assignment out of a machine's composition (H-A-148)
The retired checker's assignment on the control-node made every composition
there refuse "nothing provides", so every composition step on that machine
ended could not run. A module assigned to a machine that the case's manifests
do not hold (no watched branch lists it) is now left out of the case's
assignments as well, said in the journal and at the judgement as unlisted, and
the machine is judged on what remains: a stale assignment is a fault of the
assignments, not of the change judged.

Builds: design 47 H H-A-148, H-A-143; ADR 0578 §2
2026-11-20 16:29:16 +01:00
jschoubben e5e5c50e67 mesh-delivery-indexer: a take-in pins the branch's previous revision for its pipeline, so validate's base stays
The pin judge removed the branch's previous revision in the very write that took the next one in
(pipelines main.13 and main.15 on 2026-11-20), so the validate step's case before could not be read
and the composition was never asked. The take-in now pins the newest revision it kept before its
write under the same pipeline record as its own, and the pipeline's end frees the record on every
key it pins; a take-in with no revision before it pins only its own and says so.

Builds: H-A-147 (hq design/47-H1-answers-59 310f8955ba16a1ca8ae9661d8196cab16e3d9597), design 47 H §743, §749, H-A-108 (3)
2026-11-20 16:27:21 +01:00
mesh-tools 8167d825ed Merge pull request 'mesh-grants: the marks subtest calls on the caller's granted token and waits for its refusals; the test's nats_streams stand-in skips a consumer gone' (#270) from fix/47-H1-grants-marks-test into main 2026-11-20 15:19:12 +00:00
jschoubben ac95e5628c mesh-grants: the test's stand-in nats_streams skips a consumer gone between its listing and its read
Under load (8 copies side by side) TestTheGrantsPartWhole panicked at
integration_test.go:151 at abb7cfcf too: st.Consumer's error was dropped
and a consumer deleted after ConsumerNames listed it gave a nil one.

Builds: design 47 H §746 (the pipeline's integration step)
2026-11-20 16:11:14 +01:00
jschoubben bed9b61d5a mesh-grants: the marks subtest calls on the caller's granted token and waits for its refusals
The subtest "a runtime denied the marks is delivered no call on a marked
subject" published the caller-named call on mesh.mod.shell.call.execute.asker~a,
a token the composer never grants (buscompose.CallerToken gives a~asker
for the user a.asker), so the server refused it every time; the test
passed only because it read the refusals with a non-blocking select
right after Flush, before the client's error callback had run. Under
load the callback won the race and the pipeline's integration step
failed (main.12, 15:50). The call now uses CallerToken, and a probe on
another caller's token is published last: the server refuses in order,
so waiting for the probe's refusal waits on every refusal asserted.

Builds: design 47 H §746 (the pipeline's integration step), G A-203 (AM-D4)
2026-11-20 16:09:35 +01:00
mesh-tools 67ef449230 Merge pull request 'mesh-grants: bring a held kept stream to its definition's source filter (H-A-145)' (#269) from fix/47-H1-kept-stream-source-filter into main 2026-11-20 14:47:59 +00:00
mesh-tools 8a52b6f801 Merge pull request 'forge bridge: adopt only an added branch; its integration tests wait on what they assert' (#268) from fix/47-H1-forge-bridge-integration into main 2026-11-20 14:46:50 +00:00
jschoubben 380d4781e6 mesh-grants: bring a held kept stream to its definition's source filter
The kept stream of the runner's pipeline-ended was made at G's
definition, sourced by a three-part subject; H1 moved the event to
<branch>.<source>, so the stream took nothing in since 08:13 and the
assigner, which hears a kept event from its kept stream, heard no end.
createKept left every held kept stream as it was. It now brings a held
one's source filter from the events stream to the definition's in place
(a stream update, its messages and other settings kept, never made
again) and says the held and the composed filter in the journal and the
pass; the same holds for pipeline-started and wait-held.

Builds: design 47 H H-A-145 (hq design/47-H1-answers-59 0787704b), §746
2026-11-20 15:42:24 +01:00
jschoubben 7534bf5c07 forge bridge: adopt only an added branch; its integration tests wait on what they assert
Pipeline main.8 failed TestBridgeSaysOneHeadMovedPerMove and TestBridgePerformsWhatIsOwedOnce on
shanks; both fail the same way at 403fb306, before the catalog client change, under load.

- The bridge read every head again, start true, whenever the registrations' first view said its
  values after the start had begun, so a head moved just after the start was said as a start.
  Only a registration adding a watched branch is adopted now (Watched.Set says so), decided under
  one lock with the start's read.
- The head-moved test moved the head before the restart's read was answered, and read the state
  heads before the bridge wrote it; it now waits for the stand-in's answer and watches heads.
- The owed test stopped the bridge while an outcome's publish was on its way (heard again after
  the start), and an act said while the bridge stops can be handed to the stopped bridge's pull,
  which the bus hands again only after the consumer's acknowledgement wait (30 s) past the
  test's 20 s bound. It now hears the bridge's acknowledgements before the stop and waits the
  consumer's acknowledgement wait for the act said meanwhile.

Builds: design 47 H §739; design 47 G §532; design 47 H-A-48
2026-11-20 15:41:10 +01:00
mesh-tools 08fe40f233 Merge pull request 'assigner: a machine's composition across the watched branches (H-A-143, H-A-144)' (#267) from fix/47-H1-compose-across-branches into main 2026-11-20 14:38:28 +00:00
jschoubben 27e96a2a1b assigner: say an unlisted module on every path; no registration, no branches (H-A-144)
Review round 1: an unlisted module was lost where the composer blamed a module.
It is now said in the failed or could-not-run words where one is set, else as a
warning. Once registrations are held, a module none names has no watched
branches and is unlisted, never composed from the branches the client follows;
before the first registrations the gate's source's branches stand in, in its
order. A module is judged unlisted by its own watched branches.

Builds: design 47 H H-A-143, H-A-144, G §532
2026-11-20 15:35:59 +01:00
jschoubben f98d5ec748 assigner: compose a machine across every watched branch (H-A-143)
The composition step judged a machine at the step's one branch revision, so an
app registered on another watched branch was missing and the machine failed
"nothing provides", ending every mover there "could not run". Each module's
entry now comes from the newest revision of the first of its watched branches,
in its registration's order, that lists it; the candidate's revision replaces
its branch's newest for the modules it moves. A module assigned to the machine
that no watched branch lists is said in the machine's reason.

Builds: design 47 H H-A-143, G §532
2026-11-20 15:28:29 +01:00
mesh-tools abb7cfcf5d Merge pull request 'meshcatalogclient: a revision below the held one asks the first view again (H-A-141)' (#266) from fix/47-H1-catalog-client-numbering-restart into main 2026-11-20 14:09:37 +00:00
jschoubben 3956c0cf94 meshcatalogclient: a revision below the held one asks the first view again
A reader whose picture held the earlier model's revision 528 skipped
revisions 1 to 7 of the new numbering in silence: the builder built
every pipeline step as a candidate's, unversioned, so nothing was
released and no want moved. An event below the held revision is now
the state made again: said with both revisions, the view's made time
moved on (so the follower forgets the old numbering's revisions heard
and is not left behind for good), and the first view asked again. An
event equal to the held revision is still dropped.

Builds: design 47 H H-A-141 (hq design/47-H1-answers-59 3d722934), §750, §754
2026-11-20 15:05:06 +01:00
mesh-tools 403fb3068b Merge pull request 'The indexer makes its own per-branch durables and drops the earlier model's revisions at a take-in (design 47 H, H-A-134, H-A-135)' (#265) from fix/47-H1-indexer-durables into main 2026-11-20 11:21:50 +00:00
jschoubben e54bf7f346 indexer: a branch's durable it makes waits 30 s, delivers 5 times, starts from now
As the grants part gives a module's consumer on a seat event's stream kept by age:
without a start from now a made durable would replay every past source-moved of the
branch past the start's guard; the current revision is read at the start already.

Builds: design 47 H §743, H-A-134 (hq 8d30cc42), H-A-129, H-A-130

Claude-Session: https://claude.ai/code/session_01JG73KHUnBze48rdua8u2sy
2026-11-20 12:19:50 +01:00
jschoubben f95a164941 indexer: drop the earlier model's trunk and branch revisions at a take-in
catalog-revisions kept trunk.<n> and branch.<branch>.<n> objects beside the
branch-keyed revisions, the newest trunk one marked current, so a reader heard
revision 528 while its picture held 1. A branch's take-in now drops every such
object no record pins, says the count in the journal, and the revisions tool marks
each branch's newest revision current.

Builds: design 47 H §743, H-A-135

Claude-Session: https://claude.ai/code/session_01JG73KHUnBze48rdua8u2sy
2026-11-20 12:12:23 +01:00
jschoubben 0cf419619e indexer: make a watched branch's source-moved durable when the bus holds none
The grants part composes no per-branch durable on the source-moved stream, so the
indexer never heard a source revision after its start. Its bind now makes the
durable itself, as a module's seat-event consumer is made (SeatEventConsumerConfig,
named <node>_<indexer>_<branch>, filtered to the branch, one message at a time),
and binds an existing one as it stands.

Builds: design 47 H §743, H-A-134

Claude-Session: https://claude.ai/code/session_01JG73KHUnBze48rdua8u2sy
2026-11-20 12:10:46 +01:00
mesh-tools 8b6f6b22d4 Merge pull request 'The indexer reads a branch's source by the verb when its consumer is not bound (design 47 H, H-A-129, H-A-130)' (#264) from fix/47-H1-indexer-bootstrap into main 2026-11-20 09:46:08 +00:00
jschoubben 9cddb5921a indexer: take a watched branch's source revision in from the gate's source when its consumer cannot be bound
At a clean state the grants part makes the per-branch consumers of source-moved only from the modules view, which the
indexer answers only after its first take-in, so nothing ever took the trunk in. At its start, and again at each
failed bind, the indexer now reads the branch's current source revision through mesh-delivery.source and takes it in
as a heard source-moved of it would, when its view keeps no revision of the branch at or above it: start true exactly
when it keeps none, every repository the answer names a head. No timer; binding on registration-changed is unchanged.

Builds: design 47 H §743, §768, H-A-129, H-A-130

Claude-Session: https://claude.ai/code/session_01JG73KHUnBze48rdua8u2sy
2026-11-20 10:39:45 +01:00
122 changed files with 4125 additions and 572 deletions
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"sync"
@@ -12,6 +13,8 @@ import (
"github.com/nats-io/nats.go/jetstream"
mmac "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
mma "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
ne "git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
@@ -19,6 +22,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"git.novox.be/novox/mesh-sdk/go/keys"
"mesh-delivery-assigner/internal/assign"
)
// The run of a composition step (design 47 G §354, §63, §64, §718; A-226, A-238, A-240, A-252, A-254, A-261): the step reads one
@@ -53,6 +57,18 @@ type compositionRun struct {
call func(key string, body any) (json.RawMessage, error)
// trunk is the trunk's catalog half as the assigner holds it.
trunk func() (mcc.Half, bool)
// newest is the newest catalog half the assigner's views hold of each watched branch (halves.views), and watched a module's
// watched branches in its registration's order (halves.watchedOf): a machine's composition takes each module's manifest across
// them (design 47 H-A-143, G §532). Nil: the step's revision alone.
newest func() map[string]mcc.Half
watched func(module string) []string
// assigned is the modules the assignments place on a machine (world.assignments), for the modules no watched branch lists;
// assignment is the machine's assignments whole, which the case composes with those modules left out (H-A-148).
assigned func(node string) []string
assignment func(node string) (mma.Assignments, bool)
// findings is where the step raises the finding assignment-unlisted of each assignment it leaves out (H-A-149); the pass over
// the assignments clears it.
findings assign.Findings
// running is the last `running` heard of a module on a node (world.runningOf), never the node-engine's bucket.
running func(node, module string) (ne.Running, bool)
// away says the watchman holds machine away (machine-away, not back since): the one presence the assigner keeps (world.awayNow),
@@ -79,15 +95,17 @@ type machineEnd struct {
// run runs one composition step and answers what it came to: per machine not away it joins or starts the candidate's
// composition run (§718), waits for each run's end in its key, and judges each machine from it; failed when a machine does not
// compose, could not run when a machine could not be composed or was not judged, else passed with a warning per machine away.
// compose, could not run when a machine could not be composed or was not judged, nothing to run when no node places the module on
// the branch (H-A-179), else passed with a warning per machine away.
func (r *compositionRun) run(ctx context.Context, id string, step meshwants.StepRun) stepOutcome {
candidate, err := candidateKeyOf(step)
if err != nil {
return couldNotRun(err.Error())
}
nodes := r.inputs.machines(step.Module, step.Branch)
if len(nodes) == 0 {
return couldNotRun(fmt.Sprintf("no node places %s on the branch %q, so the composition step has no machine to compose", step.Module, step.Branch))
if len(nodes) == 0 { // a mover no machine holds on the branch has nothing to judge: nothing to run, passed (H-A-179, ADR 0682 §8)
return stepOutcome{Outcome: meshwants.StepEndedOutcomeNothingToRun,
Said: fmt.Sprintf("no node places %s on the branch %q, so the composition step has no machine to compose", step.Module, step.Branch)}
}
reader := readerOf(id, step)
prep := &runPrep{r: r, step: step}
@@ -209,6 +227,9 @@ func (p *runPrep) get(ctx context.Context) error {
p.err = errors.New("the step's catalog revision is not read: " + err.Error())
return
}
if p.r.newest != nil && p.r.watched != nil {
half = acrossBranches(p.step, half, p.r.newest(), p.r.watched)
}
if p.seats, p.err = p.r.seats(ctx, half.Revision.Definitions); p.err != nil {
return
}
@@ -229,6 +250,21 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
r.logf("the composition run %s is not ended: %v", key, err)
}
}
// The one set of unlisted assignments, read once at the run's start and carried on the run to the leaving-out, the finding and
// every step's judgement, the resume's too (design 47 H-A-152, H-A-153).
unlisted := r.unlistedOn(node)
if len(unlisted) > 0 {
if _, _, err := r.runs.change(key, func(c *meshwants.CompositionRun) bool {
if c.Ask != ask || c.Ended != nil {
return false
}
c.Unlisted = unlisted
return true
}); err != nil {
end(node + ": the run's unlisted assignments are not written: " + err.Error())
return
}
}
if err := prep.get(ctx); err != nil {
end(node + ": " + err.Error())
return
@@ -243,8 +279,15 @@ func (r *compositionRun) start(ctx context.Context, key string, run meshwants.Co
end(node + ": " + err.Error())
return
}
cs := composer.Case{Name: changeCase, Manifests: prep.manifests, None: prep.none}
if left, err := r.leftOut(node, unlisted); err != nil {
end(node + ": " + err.Error())
return
} else if left != nil {
cs.Assignments = left
}
req := composer.Request{V: composer.Version, Input: in.Body, Depth: composer.DepthCompose, Machines: []string{node},
Cases: []composer.Case{{Name: changeCase, Manifests: prep.manifests, None: prep.none}}, Seats: prep.seats,
Cases: []composer.Case{cs}, Seats: prep.seats,
BlameTries: r.runs.set.blameTries} // the composer's bound of its search for culprits (§721, A-466r)
raw, err := json.Marshal(req)
if err != nil {
@@ -289,7 +332,29 @@ const notTried = "not tried: the search reached its bound"
// could not run where the module was not tried (A-470b); a machine that does not compose and blames no module fails every step on
// it, on the node's own account; a verdict with no `blamed` (a composer before §721) is judged as before, with the warning "the
// composer names no culprits"; otherwise the machine passes, with a warning for each other module blamed there.
//
// The run's unlisted are the modules the machine is assigned that no watched branch's newest revision lists, left out of its
// composition, as the run's start read them (H-A-152, H-A-153): each is said on every path, in the machine's failed or
// could-not-run words where one is set, else as a warning, never the machine's failure and never dropped in silence (H-A-143,
// H-A-144, H-A-148).
func judgeRun(module string, run meshwants.CompositionRun) machineEnd {
end := judgeVerdict(module, run)
for _, u := range run.Unlisted {
said := run.Machine + " is assigned " + u + ", which no watched branch lists (H-A-143)"
switch {
case end.failed != "":
end.failed += "; " + said
case end.couldNot != "":
end.couldNot += "; " + said
default:
end.warnings = append(end.warnings, said)
}
}
return end
}
// judgeVerdict is judgeRun before the unlisted modules are said.
func judgeVerdict(module string, run meshwants.CompositionRun) machineEnd {
node := run.Machine
if run.Ended == nil {
return machineEnd{couldNot: node + ": the step stopped before its composition run " + runKeyOf(run.Candidate, node) + " ended"}
@@ -503,6 +568,135 @@ func caseOf(half mcc.Half) (map[string]composer.Manifest, map[string]string, err
return manifests, none, nil
}
// acrossBranches is the catalog half a machine's composition reads (design 47 H-A-143, H-A-144, G §532): each module's entry from
// the newest revision of the first of its watched branches, in its registration's order, that lists it, none for a module with no
// watched branches; the candidate's revision replaces its
// branch's newest for the modules it moves (its movers and the step's module), and the whole of it where it is that branch's newest
// or the branch's is not held. The half keeps the candidate's revision, its definitions with it.
func acrossBranches(step meshwants.StepRun, candidate mcc.Half, newest map[string]mcc.Half, watched func(module string) []string) mcc.Half {
halves := make(map[string]mcc.Half, len(newest)+1)
for b, h := range newest {
halves[b] = h
}
if own, held := newest[step.Branch]; !held || candidate.Revision.Revision >= own.Revision.Revision {
halves[step.Branch] = candidate
} else {
mix := mcc.Half{Revision: own.Revision, Entries: maps.Clone(own.Entries)}
moved := []string{step.Module}
for _, m := range candidate.Revision.Movers {
moved = append(moved, m.Module)
}
for _, m := range moved {
if e, ok := candidate.Entries[m]; ok {
mix.Entries[m] = e
} else {
delete(mix.Entries, m)
}
}
halves[step.Branch] = mix
}
out := mcc.Half{Revision: candidate.Revision, Entries: map[string]meshcatalog.CatalogEntry{}}
for _, h := range halves {
for name := range h.Entries {
if _, done := out.Entries[name]; done {
continue
}
for _, b := range watched(name) { // none: no watched branch lists it, it is unlisted (H-A-144)
if e, ok := halves[b].Entries[name]; ok {
out.Entries[name] = e
break
}
}
}
}
return out
}
// unlistedOn is the one set of unlisted assignments of node (H-A-152): the modules the assignments place there that the newest
// revision of none of their watched branches lists, a module with no watched branches among them (H-A-143, H-A-144), by name.
// The candidate never enters it, so a module the change drops while still assigned stays in the composition and fails it as the
// change's own fault (ADR 0578 §2). Where the newest revisions are not followed yet nothing is unlisted: none while no branch's
// newest is held, and not a module one of whose watched branches' newest is not held yet.
func (r *compositionRun) unlistedOn(node string) []string {
if r.assigned == nil || r.newest == nil || r.watched == nil {
return nil
}
newest := r.newest()
if len(newest) == 0 {
return nil
}
var out []string
for _, m := range r.assigned(node) {
watched := r.watched(m)
if !listedOn(newest, watched, m) && !slices.ContainsFunc(watched, func(b string) bool { _, held := newest[b]; return !held }) {
out = append(out, m)
}
}
slices.Sort(out)
return slices.Compact(out)
}
// leftOut is the case's assignments of node with each module of the run's unlisted set left out, or nil where none is (the case
// then composes the input's): a stale assignment is a fault of the assignments, not of the change judged, so it is left out of the
// composition, said in the journal, raised as assignment-unlisted and, at the judgement, said as unlisted from the same set
// (design 47 H-A-148, H-A-149, H-A-152, H-A-143; ADR 0578 §2).
func (r *compositionRun) leftOut(node string, set []string) (map[string]json.RawMessage, error) {
if r.assignment == nil || len(set) == 0 {
return nil, nil
}
a, ok := r.assignment(node)
if !ok {
return nil, nil
}
var unlisted []string
for _, m := range a.Modules {
if slices.Contains(set, m.Module) {
unlisted = append(unlisted, m.Module)
}
}
if len(unlisted) == 0 {
return nil, nil
}
a.Modules = slices.DeleteFunc(slices.Clone(a.Modules), func(m mma.PlacedModule) bool { return slices.Contains(unlisted, m.Module) })
raw, err := mmac.EncodeAssignments(a)
if err != nil {
return nil, fmt.Errorf("its assignments with %s left out cannot be encoded: %w", strings.Join(unlisted, ", "), err)
}
r.logf("%s: the composition leaves out %s, assigned there and listed on no watched branch (H-A-148)", node, strings.Join(unlisted, ", "))
for _, m := range unlisted {
r.raiseUnlisted(node, m)
}
return map[string]json.RawMessage{node: raw}, nil
}
// raiseUnlisted writes the finding assignment-unlisted of module on node (H-A-149); one not written is said in the journal.
func (r *compositionRun) raiseUnlisted(node, module string) {
if r.findings.Put == nil {
return
}
now := r.now
if now == nil {
now = time.Now
}
f, err := assign.UnlistedFinding(node, module, now())
if err == nil {
err = r.findings.Put(f)
}
if err != nil {
r.logf("the finding %s is not written: %v", assign.UnlistedKey(node, module), err)
}
}
// listedOn says whether the newest revision of one of module's watched branches lists it (H-A-143, H-A-144).
func listedOn(newest map[string]mcc.Half, watched []string, module string) bool {
for _, b := range watched {
if _, ok := newest[b].Entries[module]; ok {
return true
}
}
return false
}
// nodeEngineEntry is the entry of the module that claims the seat node-engine, or false.
func nodeEngineEntry(entries map[string]meshcatalog.CatalogEntry) (meshcatalog.CatalogEntry, bool) {
for _, name := range slices.Sorted(mapsKeys(entries)) {
@@ -616,3 +810,32 @@ func runAskPublisher(js jetstream.JetStream) func(ctx context.Context, a buildst
return err
}
}
// newestOf is the newest catalog half the views hold of each watched branch, or nil while the catalog half is not followed.
func (iv *catalogViews) newestOf() func() map[string]mcc.Half {
if iv == nil {
return nil
}
return func() map[string]mcc.Half {
if iv.h == nil {
return map[string]mcc.Half{}
}
return iv.h.views()
}
}
// listed says whether one of module's watched branches lists it in the newest revision the views hold (H-A-149's clear).
func (iv *catalogViews) listed(module string) bool {
if iv == nil || iv.h == nil {
return false
}
return listedOn(iv.h.views(), iv.h.watchedOf(module), module)
}
// watchedOf is a module's watched branches in its registration's order (halves.watchedOf); none while the half is not followed.
func (iv *catalogViews) watchedOf(module string) []string {
if iv == nil || iv.h == nil {
return nil
}
return iv.h.watchedOf(module)
}
@@ -4,18 +4,24 @@ import (
"context"
"encoding/json"
"errors"
"maps"
"slices"
"strings"
"sync"
"testing"
"time"
mmac "git.novox.be/novox/mesh-contracts/go/clients/meshmoduleassignerclient"
mma "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
ne "git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
meshcatalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"git.novox.be/novox/mesh-sdk/go/keys"
"mesh-delivery-assigner/internal/assign"
)
const (
@@ -253,12 +259,114 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
}
none := newRunStand(t, "src2:e1", "src2:e1", "d1")
none.r.inputs.machines = func(string, string) []string { return machines }
if out := none.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeCouldNotRun ||
outcome := meshwants.StepEndedOutcomeCouldNotRun
if len(machines) == 0 { // a mover no machine holds: nothing to run, saying why (H-A-179)
outcome = meshwants.StepEndedOutcomeNothingToRun
}
if out := none.r.run(context.Background(), "ask-step", step); out.Outcome != outcome ||
!strings.Contains(out.Said, want) || (len(machines) > 0 && !strings.Contains(out.Said, "shanks is away")) || len(none.asks) != 0 {
t.Errorf("%s came to %+v, %d asks", name, out, len(none.asks))
}
}
// Another watched branch's app is in every machine's case, and a module assigned there that no watched branch lists is said in
// the machine's reason (H-A-143, §532).
across := newRunStand(t, "src2:e1", "src2:e1", "d1")
across.r.newest = func() map[string]mcc.Half {
return map[string]mcc.Half{"master": {Revision: meshcatalog.CatalogRevision{Branch: "master", Revision: 2},
Entries: map[string]meshcatalog.CatalogEntry{"f1-circuits": {Module: "f1-circuits", Fp: "f1", Manifest: map[string]any{"module": "f1-circuits"}}}}}
}
across.r.watched = func(module string) []string {
if module == "checker" || module == "lidarr" { // no registration names them; lidarr is on main's revision all the same (H-A-144)
return nil
}
return []string{"main", "master"}
}
across.r.assigned = func(node string) []string { return []string{"m", "f1-circuits", "checker", "lidarr"} }
if out := across.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeFailed ||
!strings.Contains(out.Said, "g14 is assigned checker, which no watched branch lists (H-A-143)") ||
!strings.Contains(out.Said, "g14 is assigned lidarr, which no watched branch lists (H-A-143)") {
t.Errorf("the step across the watched branches came to %+v", out)
}
if c := across.requests["ace"].Cases[0]; string(c.Manifests["f1-circuits"].Raw) != `{"module":"f1-circuits"}` || len(c.Manifests) != 4 ||
c.Manifests["lidarr"].Raw != nil {
t.Errorf("the case across the watched branches holds %v", slices.Sorted(maps.Keys(c.Manifests)))
}
if c := across.requests["ace"].Cases[0]; c.Assignments != nil {
t.Errorf("with no assignments held the case composes the input's, it carries %v", c.Assignments)
}
// A stale assignment is left out of the case's assignments as well, said, and the machine composes; a listed module stays in
// (H-A-148, ADR 0578 §2). The one set is the watched branches' newest revisions', carried on the run (H-A-152, H-A-153): a module
// the candidate drops while the newest still lists it stays in, the change's own fault, and while no newest revision is followed
// nothing is left out.
dropsWeb := func() map[string]mcc.Half {
h := across.r.newest()
h["main"] = mcc.Half{Revision: meshcatalog.CatalogRevision{Branch: "main", Revision: 6},
Entries: map[string]meshcatalog.CatalogEntry{"m": {Module: "m", Manifest: map[string]any{"module": "m"}},
"web": {Module: "web", Fp: "w1", Manifest: map[string]any{"module": "web"}}}}
return h
}
for _, tc := range []struct {
name string
assigned []string
newest func() map[string]mcc.Half // nil: across's
want []string // the case's modules of ace; nil: the input's assignments
}{
{"a stale assignment left out and said, the machine composes", []string{"m", "f1-circuits", "checker"}, nil, []string{"m", "f1-circuits"}},
{"listed modules alone: the input's assignments", []string{"m", "f1-circuits"}, nil, nil},
{"a module the candidate drops, the newest lists: kept", []string{"m", "f1-circuits", "web"}, dropsWeb, nil},
{"no newest revision followed: nothing left out", []string{"m", "checker"}, func() map[string]mcc.Half { return map[string]mcc.Half{} }, nil},
} {
left := newRunStand(t, "src2:e1", "src2:e1", "d1")
left.r.newest, left.r.watched = across.r.newest, across.r.watched
if tc.newest != nil {
left.r.newest = tc.newest
}
left.r.assigned = func(string) []string { return tc.assigned }
left.r.assignment = func(node string) (mma.Assignments, bool) {
a := mma.Assignments{Node: node}
for _, m := range tc.assigned {
a.Modules = append(a.Modules, mma.PlacedModule{Module: m})
}
return a, true
}
left.verdict = func(string) (composer.Machine, bool) { return composer.Machine{Composes: true}, true }
raised := map[string][]byte{}
left.r.findings = assign.Findings{Put: putInto(raised)}
out := left.r.run(context.Background(), "ask-step", step)
// H-A-149: the step that leaves an assignment out raises assignment-unlisted, scope delivery, id <node>.<module>, a warning.
var f mcseat.Finding
_ = json.Unmarshal(raised[assign.UnlistedKey("ace", "checker")], &f)
if want := tc.want != nil; want != (f.Kind == assign.KindAssignmentUnlisted && f.Scope == "delivery" && f.ID == "ace.checker" &&
f.Severity == mcseat.FindingSeverityWarning) || (!want && len(raised) != 0) {
t.Errorf("%s: raised %v", tc.name, raised)
}
said := "ace is assigned checker, which no watched branch lists (H-A-143)"
if out.Outcome != meshwants.StepEndedOutcomePassed || (tc.want != nil) != slices.Contains(out.Warnings, said) {
t.Errorf("%s: the step came to %+v", tc.name, out)
}
if run, _, _, _ := left.runs.read(runKeyOf("main.7", "ace")); (tc.want != nil) != slices.Equal(run.Unlisted, []string{"checker"}) ||
(tc.want == nil && run.Unlisted != nil) {
t.Errorf("%s: the run carries the unlisted %v", tc.name, run.Unlisted)
}
raw, ok := left.requests["ace"].Cases[0].Assignments["ace"]
if tc.want == nil {
if ok {
t.Errorf("%s: the case carries assignments %s", tc.name, raw)
}
continue
}
a, err := mmac.DecodeAssignments(raw)
var got []string
for _, m := range a.Modules {
got = append(got, m.Module)
}
if err != nil || !slices.Equal(got, tc.want) {
t.Errorf("%s: the case composes ace with %v (%v), want %v", tc.name, got, err, tc.want)
}
}
// At the inputs state's cap the ask is refused, naming the store (§354).
full := newRunStand(t, "src2:e1", "src2:e1", "d1")
full.r.keeper.used = func() (int64, error) { return inputsCap, nil }
@@ -438,21 +546,35 @@ func TestAMachineIsJudgedByTheModulesItBlames(t *testing.T) {
run meshwants.CompositionRun
failed, couldNo string
warnings []string
unlisted []string
}{
{"blamed module fails", "a", runOf(blamesA), "ace holds a: /etc/x is a's and c's", "", nil},
{"another module passes with a warning", "b", runOf(blamesA), "", "", []string{"ace holds a: /etc/x is a's and c's"}},
{"blames no module fails both", "b", runOf(`{"composes":false,"reason":"the node's own composition","blamed":[]}`), "ace does not compose and blames no module: the node's own composition", "", nil},
{"an old composer", "b", runOf(`{"composes":false,"reason":"x"}`), "ace does not compose: x", "", []string{"ace: the composer names no culprits"}},
{"composes", "b", runOf(`{"composes":true,"blamed":[]}`), "", "", nil},
{"blamed module fails", "a", runOf(blamesA), "ace holds a: /etc/x is a's and c's", "", nil, nil},
{"another module passes with a warning", "b", runOf(blamesA), "", "", []string{"ace holds a: /etc/x is a's and c's"}, nil},
{"blames no module fails both", "b", runOf(`{"composes":false,"reason":"the node's own composition","blamed":[]}`), "ace does not compose and blames no module: the node's own composition", "", nil, nil},
{"an old composer", "b", runOf(`{"composes":false,"reason":"x"}`), "ace does not compose: x", "", []string{"ace: the composer names no culprits"}, nil},
{"composes", "b", runOf(`{"composes":true,"blamed":[]}`), "", "", nil, nil},
{"own module not tried", "a", runOf(`{"composes":false,"blamed":[{"module":"a","problem":"not tried: the search reached its bound"}]}`), "",
"ace holds a untried: the composer's search for culprits reached its bound of blame_tries (A-470b)", nil},
"ace holds a untried: the composer's search for culprits reached its bound of blame_tries (A-470b)", nil, nil},
{"another not tried", "b", runOf(`{"composes":false,"blamed":[{"module":"a","problem":"not tried: the search reached its bound"}]}`), "", "",
[]string{"ace holds a: not tried: the search reached its bound"}},
{"not judged", "a", runOf(`{"composes":false,"not_judged":"no seat list","blamed":[]}`), "", "ace was not judged: no seat list", nil},
{"could not run", "a", meshwants.CompositionRun{Machine: "ace", Ended: &at, Outcome: meshwants.CompositionRunOutcomeCouldNotRun, Said: "ace: gone"}, "", "ace: gone", nil},
[]string{"ace holds a: not tried: the search reached its bound"}, nil},
{"not judged", "a", runOf(`{"composes":false,"not_judged":"no seat list","blamed":[]}`), "", "ace was not judged: no seat list", nil, nil},
{"could not run", "a", meshwants.CompositionRun{Machine: "ace", Ended: &at, Outcome: meshwants.CompositionRunOutcomeCouldNotRun, Said: "ace: gone"}, "", "ace: gone", nil, nil},
// H-A-143: a module assigned there that no watched branch lists is said in the reason, or as a warning where it composes.
{"an unlisted module in the reason", "b", runOf(`{"composes":false,"reason":"nothing provides \"photos\"","blamed":[]}`),
"ace does not compose and blames no module: nothing provides \"photos\"; ace is assigned photos, which no watched branch lists (H-A-143)", "", nil, []string{"photos"}},
{"does not compose, blames a module, an unlisted module", "a", runOf(`{"composes":false,"reason":"nothing provides \"photos\"","blamed":[{"module":"a","problem":"nothing provides \"photos\""}]}`),
"ace holds a: nothing provides \"photos\"; ace is assigned photos, which no watched branch lists (H-A-143)", "", nil, []string{"photos"}},
{"another module blamed, an unlisted module", "b", runOf(`{"composes":false,"blamed":[{"module":"a","problem":"x"}]}`), "", "",
[]string{"ace holds a: x", "ace is assigned photos, which no watched branch lists (H-A-143)"}, []string{"photos"}},
{"could not run, an unlisted module", "a", meshwants.CompositionRun{Machine: "ace", Ended: &at, Outcome: meshwants.CompositionRunOutcomeCouldNotRun, Said: "ace: gone"},
"", "ace: gone; ace is assigned photos, which no watched branch lists (H-A-143)", nil, []string{"photos"}},
{"an unlisted module where it composes", "b", runOf(`{"composes":true,"blamed":[]}`), "", "",
[]string{"ace is assigned photos, which no watched branch lists (H-A-143)"}, []string{"photos"}},
} {
t.Run(tc.name, func(t *testing.T) {
e := judgeRun(tc.module, tc.run)
run := tc.run
run.Unlisted = tc.unlisted // the set the run's start wrote (H-A-153)
e := judgeRun(tc.module, run)
if e.failed != tc.failed || e.couldNot != tc.couldNo || !slices.Equal(e.warnings, tc.warnings) {
t.Errorf("judged %+v, want failed %q, could not run %q, warnings %q", e, tc.failed, tc.couldNo, tc.warnings)
}
@@ -572,3 +694,106 @@ func TestARerunMovesEveryReaderThatDidNotPass(t *testing.T) {
}
}
}
// Design 47 H-A-143, G §532: a machine's composition takes each module's entry from the newest revision of the first of its
// watched branches, in its registration's order, that lists it; the candidate's revision replaces its branch's newest for the
// modules it moves, and whole where it is that branch's newest; the half keeps the candidate's revision.
func TestACompositionTakesEachModuleAcrossTheWatchedBranches(t *testing.T) {
entry := func(module, fp string) meshcatalog.CatalogEntry {
return meshcatalog.CatalogEntry{Module: module, Fp: fp, Manifest: map[string]any{"module": module}}
}
half := func(branch string, revision int64, movers []string, entries ...meshcatalog.CatalogEntry) mcc.Half {
h := mcc.Half{Revision: meshcatalog.CatalogRevision{Branch: branch, Revision: revision, Definitions: []meshcatalog.CatalogRevisionDefinitionsItem{{Seat: "s1", Digest: branch}}},
Entries: map[string]meshcatalog.CatalogEntry{}}
for _, m := range movers {
h.Revision.Movers = append(h.Revision.Movers, meshcatalog.TakeInEntry{Module: m})
}
for _, e := range entries {
h.Entries[e.Module] = e
}
return h
}
newest := map[string]mcc.Half{
"main": half("main", 4, nil, entry("engine", "e1"), entry("a", "a1"), entry("both", "main")),
"master": half("master", 9, nil, entry("f1-circuits", "f1")),
"nox-mesh": half("nox-mesh", 3, nil, entry("link2pay", "l1"), entry("photos", "p1"), entry("both", "nox")),
}
registrations := map[string][]string{"both": {"nox-mesh", "main"}, "f1-circuits": {"master"}, "link2pay": {"nox-mesh"}, "photos": {"nox-mesh"}}
registrations["engine"], registrations["a"] = []string{"main"}, []string{"main"}
watched := func(module string) []string { return registrations[module] } // a module no registration names has none (H-A-144)
for _, tc := range []struct {
name string
step meshwants.StepRun
candidate mcc.Half
newest map[string]mcc.Half
want map[string]string // module -> fingerprint
}{
{"the newest of main, the others' apps beside it", meshwants.StepRun{Branch: "main", Module: "a", Revision: 5},
half("main", 5, []string{"a"}, entry("engine", "e1"), entry("a", "a2"), entry("both", "main")), newest,
map[string]string{"engine": "e1", "a": "a2", "both": "nox", "f1-circuits": "f1", "link2pay": "l1", "photos": "p1"}},
{"an older candidate moves only its movers", meshwants.StepRun{Branch: "nox-mesh", Module: "link2pay", Revision: 2},
half("nox-mesh", 2, []string{"photos"}, entry("link2pay", "l0"), entry("both", "nox0")), newest,
map[string]string{"engine": "e1", "a": "a1", "both": "nox", "f1-circuits": "f1", "link2pay": "l0"}},
{"a branch not yet held is the candidate's", meshwants.StepRun{Branch: "master", Module: "f1-circuits", Revision: 1},
half("master", 1, nil, entry("f1-circuits", "f0")), map[string]mcc.Half{"main": newest["main"]},
map[string]string{"engine": "e1", "a": "a1", "both": "main", "f1-circuits": "f0"}},
{"no view held yet: the candidate alone", meshwants.StepRun{Branch: "main", Module: "a", Revision: 5},
half("main", 5, nil, entry("a", "a2")), nil, map[string]string{"a": "a2"}},
{"a module no registration names is not composed", meshwants.StepRun{Branch: "main", Module: "a", Revision: 5},
half("main", 5, nil, entry("a", "a2"), entry("checker", "c1")), nil, map[string]string{"a": "a2"}},
} {
t.Run(tc.name, func(t *testing.T) {
got := acrossBranches(tc.step, tc.candidate, tc.newest, watched)
fps := map[string]string{}
for name, e := range got.Entries {
fps[name] = e.Fp
}
if !maps.Equal(fps, tc.want) {
t.Errorf("composed from %v, want %v", fps, tc.want)
}
if got.Revision.Revision != tc.candidate.Revision.Revision || len(got.Revision.Definitions) != 1 || got.Revision.Definitions[0].Digest != tc.step.Branch {
t.Errorf("the half keeps revision %d with definitions %v, want the candidate's %d with its own", got.Revision.Revision, got.Revision.Definitions,
tc.candidate.Revision.Revision)
}
})
}
}
// Design 47 H-A-149: the pass over the assignments clears assignment-unlisted when the assignment is gone or one of the module's
// watched branches lists it again, and keeps it while it stands; another finding is left alone.
func TestThePassClearsAnUnlistedAssignmentsFinding(t *testing.T) {
for _, tc := range []struct {
name string
assigned []string // novox's modules
listed bool
kept bool
}{
{"still assigned and listed nowhere: kept", []string{"checker"}, false, true},
{"the assignment gone: cleared", []string{"web"}, false, false},
{"listed again: cleared", []string{"checker"}, true, false},
} {
t.Run(tc.name, func(t *testing.T) {
w := newWorld()
a := mma.Assignments{V: keys.Version, Node: "novox"}
for _, m := range tc.assigned {
a.Modules = append(a.Modules, mma.PlacedModule{Module: m})
}
putAssignments(w, a)
findings := map[string][]byte{}
f, _ := assign.UnlistedFinding("novox", "checker", time.Now())
_ = putInto(findings)(f)
other, _ := assign.UnplacedFinding("g14", time.Now())
_ = putInto(findings)(other)
p := &passer{w: w, logf: t.Logf, listed: func(string) bool { return tc.listed }, findings: assign.Findings{Put: putInto(findings),
Delete: func(k string) error { delete(findings, k); return nil },
Keys: func() ([]string, error) { return slices.Collect(maps.Keys(findings)), nil }}}
p.clearUnlisted()
if _, ok := findings[assign.UnlistedKey("novox", "checker")]; ok != tc.kept {
t.Errorf("the finding kept %v, want %v", ok, tc.kept)
}
if _, ok := findings[assign.UnplacedKey("g14")]; !ok {
t.Error("another finding is cleared")
}
})
}
}
@@ -818,7 +818,7 @@ func (r *compositionRuns) leave(pipeline string, stop func(ids []string)) ([]str
// passedOn is whether the step of module passed on the machine of an ended run (§722): neither failed nor could not run there.
func passedOn(module string, run meshwants.CompositionRun) bool {
e := judgeRun(module, run)
e := judgeVerdict(module, run) // the unlisted modules change only the words, never whether it passed
return e.failed == "" && e.couldNot == ""
}
@@ -6,6 +6,7 @@ import (
"encoding/hex"
"encoding/json"
"slices"
"strings"
"sync"
"time"
@@ -13,6 +14,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/views"
)
@@ -191,3 +193,25 @@ func (h *halves) views() map[string]mcc.Half {
}
return out
}
// watchedOf is module's watched branches in its registration's order (the branch `register` named first, then each branch in the
// order `watch` added it). Once registrations are held, a module none names has none: it is unlisted (H-A-144). Only before the
// first registrations are held do the gate's source's branches stand in (the branches followed then), in the order the source
// gives them, by their names escaped as its keys are (design 47 G §532, H-A-143, H-A-144).
func (h *halves) watchedOf(module string) []string {
h.mu.Lock()
b := h.branches
h.mu.Unlock()
if b == nil {
return nil
}
if b.Registrations != nil && len(b.Registrations.All()) > 0 {
if reg, ok := b.Registrations.Registration(module); ok {
return slices.Clone(reg.Branches)
}
return nil
}
out := b.Followed()
slices.SortStableFunc(out, func(x, y string) int { return strings.Compare(keys.EscapeName(x), keys.EscapeName(y)) })
return out
}
@@ -3,6 +3,8 @@ package main
import (
"context"
"encoding/json"
"errors"
"slices"
"testing"
"time"
@@ -48,3 +50,45 @@ func TestAMeshThatNamesNoBranchSettlesTheAssignersFirstPass(t *testing.T) {
}
}
}
// Design 47 H-A-143, H-A-144, G §532: a module's watched branches are its registration's, in its order; once registrations are held a
// module none names has none; before the first registrations the gate's source's branches stand in, in the source's order.
func TestAModulesWatchedBranchesAreItsRegistrations(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
for _, tc := range []struct {
name string
regs []seat.Registration
module string
want []string
}{
{"its registration's order", []seat.Registration{{Module: "link2pay", Branches: []string{"nox-mesh", "main"}}}, "link2pay", []string{"nox-mesh", "main"}},
{"no registration names it", []seat.Registration{{Module: "link2pay", Branches: []string{"nox-mesh", "main"}}}, "checker", nil},
{"before the first registrations, the source's", []seat.Registration{}, "checker", []string{"main", "master"}},
} {
t.Run(tc.name, func(t *testing.T) {
call := func(key string, _ any) (json.RawMessage, error) {
switch key {
case mcc.RegistrationsVerb:
return json.Marshal(seat.RegistrationsAnswer{Registrations: tc.regs})
case mcc.SourceVerb:
return json.Marshal(map[string][]string{"branches": {"master", "main"}})
}
return nil, errors.New("not answered here")
}
reader := &views.ProcessReader{Module: "mesh-delivery-assigner", Call: call, Logf: t.Logf}
h := &halves{w: newWorld(), call: call, reader: reader, wake: func() {}, logf: t.Logf, ctx: ctx}
if err := h.follow(); err != nil {
t.Fatal(err)
}
for _, f := range reader.Followers()[:2] { // the registrations and the gate's source
if err := f.Start(ctx); err != nil {
t.Fatal(err)
}
}
if got := h.watchedOf(tc.module); !slices.Equal(got, tc.want) {
t.Errorf("the watched branches of %s are %v, want %v", tc.module, got, tc.want)
}
})
}
}
@@ -1769,6 +1769,11 @@ func serveCatalog(t *testing.T, b *bed, nc *nats.Conn) {
}
if kept, ok := b.cat.at[branch][in.Revision]; ok {
rev := meshcatalog.CatalogRevision{Kind: "trunk", Branch: in.Branch, Revision: in.Revision}
for k := range b.cat.at[branch] { // its base, the newest kept below it, as the take-in names it (H-A-154)
if k < in.Revision && k > rev.Base {
rev.Base = k
}
}
for _, e := range kept {
rev.Entries = append(rev.Entries, meshcatalog.EntryRef{Module: e.Module, Digest: e.ManifestDigest})
}
@@ -581,7 +581,7 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{},
reads: reads, seats: gate, askForFile: getenv("MESH_ASSIGNER_ASK_OPERATOR_FOR_FILE")}
reads: reads, seats: gate, askForFile: getenv("MESH_ASSIGNER_ASK_OPERATOR_FOR_FILE"), listed: iv.listed}
p.askOperatorFor() // the setting ask-operator-for said once at the start (design 47 G A-425a)
// The setting tier-bound (design 47 H §751) and the one timer that wakes the pass at a tier's deadline (H-A-56).
tierBound, err := tierBoundSetting(getenv)
@@ -665,7 +665,7 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
// The composition step (design 47 G §22, §354): the seat mesh-wants' accepted kind step, run by one run ask per machine.
if bus != nil && js != nil && inputs != nil && p.publishWants != nil && runs != nil {
run := &compositionRun{runs: runs, inputs: &compositionInputs{machines: w.machinesOf, callFile: bus.CallFile}, keeper: inputs, reads: &compositionReads{}, call: bus.CallKept,
trunk: trunkOf(iv, w.trunkName), running: w.runningOf, away: func(machine string) bool { return w.awayNow()[machine] }, put: runAskPublisher(js), ask: bus.CallFile, now: time.Now, logf: logger.Printf}
trunk: trunkOf(iv, w.trunkName), newest: iv.newestOf(), watched: iv.watchedOf, assigned: w.assignedOn, assignment: w.assignmentOf, findings: findings, running: w.runningOf, away: func(machine string) bool { return w.awayNow()[machine] }, put: runAskPublisher(js), ask: bus.CallFile, now: time.Now, logf: logger.Printf}
steps := &compositionSteps{node: thisNode, say: p.publishWants, run: run.run, resume: run.resume,
validate: run.validate, resumeValidate: run.resumeValidate, ended: runs.stepEnded, now: time.Now, logf: logger.Printf}
move := steps.move(ctx) // the readers a rerun carried (§727, A-470d)
@@ -56,6 +56,8 @@ type passer struct {
findings assign.Findings
// unplaced are the nodes whose unplaced finding this process wrote.
unplaced map[string]bool
// listed says whether a module's watched branches list it (catalogViews.listed), for the clear of assignment-unlisted.
listed func(module string) bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64
// now is the clock; nil is time.Now.
@@ -208,6 +210,7 @@ func (p *passer) wants(ctx context.Context) {
}))
chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket()))
defer p.unplacedFindings()
defer p.clearUnlisted()
defer func() { p.seatHolders(ctx, viewOf(p.w.trunkName()).manifests) }()
p.clearStaleCalls()
placed := p.w.placedNodes()
@@ -655,6 +658,33 @@ func (p *passer) unplacedFindings() {
}
}
// clearUnlisted clears, in the pass over the assignments, each finding assignment-unlisted whose assignment is gone or whose module
// one of its watched branches lists again, so the clear never waits for the next composition step on that machine (H-A-149).
func (p *passer) clearUnlisted() {
if p.findings.Keys == nil || p.findings.Delete == nil || p.listed == nil {
return
}
keys, err := p.findings.Keys()
if err != nil {
p.logf("the findings assignment-unlisted are not read, so none is cleared in this pass: %v", err)
return
}
assignments := p.w.assignments()
for _, key := range keys {
node, module, ok := assign.UnlistedOfKey(key)
if !ok {
continue
}
assigned := slices.ContainsFunc(assignments[node].Modules, func(m mma.PlacedModule) bool { return m.Module == module })
if assigned && !p.listed(module) {
continue
}
if err := p.findings.Delete(key); err != nil {
p.logf("the finding %s is not cleared: %v", key, err)
}
}
}
// seatHolders writes the seat mesh-module-assigner's state seat-holders from every node's assignments (ADR 0571 §4): one key per
// seat held once, `<seat>`, and one per node of a node seat, `<seat>.<node>`, written only where it differs, and a key no
// assignments holds any more deleted. A pass follows each assignments change and each manifest the indexer takes in, so no
@@ -22,7 +22,7 @@ import (
// `<pipeline>.validate` of the state composition-runs in the CompositionRun form with no machine, which every mover's step
// reads: the cycle the revision's entries make (graph.RefusedCycle) ends it at once; else the manifest validator built from
// the revision's own entry of the module holding the seat mesh-catalog, on the Go build seat as the composer is, runs in the
// sandbox over the revision's manifests and the branch's previous revision's, its input kept and served on mesh-wants' input.
// sandbox over the revision's manifests and its base's (H-A-154), its input kept and served on mesh-wants' input.
// Every mover's step ends the same: failed `cycle` naming the cycle, failed `manifest-refused` naming every problem of a
// manifest the revision alters or between manifests one it alters takes part in, or passed.
@@ -32,7 +32,7 @@ const (
validatorPackage = "mesh-delivery-indexer/cmd/mesh-judge"
// validateKeySuffix ends the key of a revision's validate run, `<pipeline>.validate` (H-A-108).
validateKeySuffix = "." + meshwants.StepRunStepValidate
// beforeCase is the validator's case of the branch's previous revision, beside changeCase (H-A-108 (3)).
// beforeCase is the validator's case of the revision's base, beside changeCase (H-A-108 (3), H-A-151, H-A-154).
beforeCase = "before"
// The reasons a validate step fails with, the first word of its said (§745).
reasonManifestRefused = "manifest-refused"
@@ -112,16 +112,29 @@ func (r *compositionRun) judgeValidate(ctx context.Context, step meshwants.StepR
if err != nil {
return couldNotRun("the step's catalog revision is not read: " + err.Error())
}
before, err := r.before(ctx, step)
before, err := r.before(ctx, step, half)
if err != nil {
return couldNotRun("the branch's previous catalog revision is not read: " + err.Error())
return couldNotRun("the base of the case before is not read: " + err.Error())
}
// A revision that names no base (the branch's first take-in) is judged against an empty case before, and says so (H-A-151).
empty := ""
if half.Revision.Base <= 0 {
empty = emptyBeforeWords
}
if problems := causedProblems(v.Change, v.Before, altered(half, before)); len(problems) > 0 {
return stepOutcome{Outcome: meshwants.StepEndedOutcomeFailed, Said: reasonManifestRefused + ": " + strings.Join(problems, "; ")}
said := reasonManifestRefused + ": " + strings.Join(problems, "; ")
if empty != "" {
said += " (" + empty + ")"
}
return stepOutcome{Outcome: meshwants.StepEndedOutcomeFailed, Said: said}
}
return stepOutcome{Outcome: meshwants.StepEndedOutcomePassed}
return stepOutcome{Outcome: meshwants.StepEndedOutcomePassed, Said: empty}
}
// emptyBeforeWords say, in a validate step's end, that its revision names no base: the branch's first take-in, judged against
// an empty case before (H-A-151, H-A-154).
const emptyBeforeWords = "the case before is empty: the revision names no base, the branch's first take-in"
// validateVerdict is a validate run's verdict as the run keeps it: the cycle the revision makes, or the validator's two cases.
type validateVerdict struct {
Cycle []meshcatalog.ArtifactRef `json:"cycle,omitempty"`
@@ -148,15 +161,16 @@ func cycleWords(cycle []meshcatalog.ArtifactRef) string {
return strings.Join(words, " → ")
}
// before is the branch's catalog revision before the step's, read through the seat mesh-catalog's verb revision (H-A-108 (3));
// none before the first.
func (r *compositionRun) before(ctx context.Context, step meshwants.StepRun) (mcc.Half, error) {
if step.Revision <= 1 {
// before is the case before of the step's revision half: the revision its base names, the newest kept revision below it that
// its take-in pinned, read through the seat mesh-catalog's verb revision (H-A-108 (3), H-A-151, H-A-154); empty where the
// revision names no base, the branch's first take-in.
func (r *compositionRun) before(ctx context.Context, step meshwants.StepRun, half mcc.Half) (mcc.Half, error) {
if half.Revision.Base <= 0 {
return mcc.Half{Entries: map[string]meshcatalog.CatalogEntry{}}, nil
}
prev := step
prev.Revision--
return r.revision(ctx, prev)
base := step
base.Revision = half.Revision.Base
return r.revision(ctx, base)
}
// altered are the modules whose manifest the revision adds or changes against the one before it.
@@ -270,9 +284,9 @@ func (r *compositionRun) startValidate(ctx context.Context, key string, run mesh
end(meshwants.CompositionRunOutcomePassed, "", verdict)
return
}
before, err := r.before(ctx, step)
before, err := r.before(ctx, step, half)
if err != nil {
couldNot("the branch's previous catalog revision is not read: " + err.Error())
couldNot("the base of the case before is not read: " + err.Error())
return
}
entry, ok := catalogEntry(half.Entries)
@@ -20,7 +20,9 @@ import (
// run, `<pipeline>.validate` of the state composition-runs; a manifest the revision alters that names an unknown field fails
// it `manifest-refused` naming the problem, a problem of a manifest it leaves as it was does not, and the validator is the
// revision's own entry of the module holding the seat mesh-catalog, on the Go build seat; a revision whose entries make a cycle
// fails it `cycle` naming the cycle, with no validator asked.
// fails it `cycle` naming the cycle, with no validator asked. The case before is the revision its base names (H-A-151, H-A-154:
// the newest kept below it, which the stand-in names as the indexer does); a revision naming no base, the branch's first take-in,
// is judged against an empty case before, said in its end, and no revision below it is asked.
func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
indexer := meshcatalog.CatalogEntry{Module: "indexer", Repository: "novox/mesh-delivery", Commit: commitE3, Path: "modules/mesh-delivery-indexer",
Fp: "src2:ix", Serves: []string{"seat:" + meshcatalog.Seat + ".revision"}, Manifest: map[string]any{"module": "indexer"}, ManifestDigest: "d-ix"}
@@ -28,9 +30,24 @@ func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
return meshcatalog.CatalogEntry{Module: module, Manifest: map[string]any{"module": module, "d": digest}, ManifestDigest: digest, Edges: edges}
}
revisions := map[int64][]meshcatalog.CatalogEntry{
6: {indexer, entry("web", "d-web-1"), entry("db", "d-db")},
7: {indexer, entry("web", "d-web-2"), entry("db", "d-db")},
9: {indexer, entry("a", "d-a", meshcatalog.Edge{From: "a:x", To: "b:y", Kind: 3}, meshcatalog.Edge{From: "b:y", To: "a:x", Kind: 3}), entry("b", "d-b")},
6: {indexer, entry("web", "d-web-1"), entry("db", "d-db")},
7: {indexer, entry("web", "d-web-2"), entry("db", "d-db")},
9: {indexer, entry("a", "d-a", meshcatalog.Edge{From: "a:x", To: "b:y", Kind: 3}, meshcatalog.Edge{From: "b:y", To: "a:x", Kind: 3}), entry("b", "d-b")},
3: {indexer},
12: {indexer, entry("web", "d-web-2"), entry("db", "d-db")},
}
// base is the newest kept revision below n, as the take-in names it; none below the first. Revision 12 names 7 with 8 to 11
// never kept, so a case before read as the number minus one could not run (H-A-154).
base := func(n int64) (b int64) {
if n == 12 {
return 7
}
for k := range revisions {
if k < n && k > b {
b = k
}
}
return b
}
call := func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
@@ -42,7 +59,7 @@ func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
if !ok || in.Branch != "main" {
return nil, errors.New("asked another revision")
}
return json.Marshal(meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: in.Revision,
return json.Marshal(meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: in.Revision, Base: base(in.Revision),
Definitions: []meshcatalog.CatalogRevisionDefinitionsItem{{Seat: "s1", Digest: "sd"}}}, Entries: entries})
case seatDefinitionsKey:
return json.Marshal(meshcatalog.SeatDefinitionsAnswer{Found: true, Entry: &meshcatalog.SeatDefinitionsAnswerEntry{Seat: "s1", Digest: "sd", Definition: map[string]any{"seat": "s1"}}})
@@ -117,4 +134,17 @@ func TestTheValidateStepRefusesAManifestProblemAndACycle(t *testing.T) {
if o.Outcome != meshwants.StepEndedOutcomeFailed || o.Said != reasonCycle+": a:x → b:y → a:x" || len(asks) != 1 {
t.Errorf("a revision making a cycle ended %+v (%d validator asks)", o, len(asks))
}
// The stand-in validator answers its one verdict: judged against the empty case before, web's problem is the revision's.
if o := r.validate(ctx, "ask-first", step("indexer", 3)); o.Outcome != meshwants.StepEndedOutcomeFailed ||
o.Said != reasonManifestRefused+`: web: unknown field "colour" (`+emptyBeforeWords+")" || len(asks) != 2 || len(requests[1].Cases[1].Manifests) != 0 {
t.Errorf("a revision naming no base ended %+v (%d validator asks), want it judged against an empty case before, said", o, len(asks))
}
// Revision 12 holds revision 7's manifests and names base 7: judged against it, web's problem is the revision's, db's old one
// is not, and the case before holds base 7's manifests.
if o := r.validate(ctx, "ask-base", step("web", 12)); o.Outcome != meshwants.StepEndedOutcomeFailed ||
o.Said != reasonManifestRefused+`: web: unknown field "colour"` || len(asks) != 3 {
t.Errorf("a revision naming base 7 below its number ended %+v (%d validator asks), want it judged against base 7", o, len(asks))
} else if web := requests[2].Cases[1].Manifests["web"]; len(requests[2].Cases[1].Manifests) != 3 || !strings.Contains(string(web.Raw), "d-web-2") {
t.Errorf("the case before of revision 12 is %+v, want base 7's manifests", requests[2].Cases[1].Manifests)
}
}
@@ -553,3 +553,18 @@ func (w *world) placedBranches() map[string]map[string]string {
}
return out
}
// assignedOn is the modules the assignments place on node.
func (w *world) assignedOn(node string) []string {
var out []string
for _, m := range w.assignments()[node].Modules {
out = append(out, m.Module)
}
return out
}
// assignmentOf is node's assignments, or false where none is held.
func (w *world) assignmentOf(node string) (mma.Assignments, bool) {
a, ok := w.assignments()[node]
return a, ok
}
@@ -0,0 +1,46 @@
package assign
import (
"fmt"
"strings"
"time"
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-sdk/go/views"
)
// KindAssignmentUnlisted is a module assigned to a node that no watched branch lists, left out of that node's composition
// (design 47 H-A-148, H-A-149): scope delivery, id `<node>.<module>`, severity warning.
const (
KindAssignmentUnlisted = "assignment-unlisted"
ScopeAssignmentUnlisted = "delivery"
)
// UnlistedKey is its finding's key, `delivery.<node>.<module>.assignment-unlisted`.
func UnlistedKey(node, module string) string {
return views.FindingKey(ScopeAssignmentUnlisted, KeyID(node+"."+module), KindAssignmentUnlisted)
}
// UnlistedOfKey is the node and module of an assignment-unlisted finding's key, or false for another finding's.
func UnlistedOfKey(key string) (node, module string, ok bool) {
rest, ok := strings.CutPrefix(key, ScopeAssignmentUnlisted+".")
if !ok {
return "", "", false
}
if rest, ok = strings.CutSuffix(rest, "."+KindAssignmentUnlisted); !ok {
return "", "", false
}
node, module, ok = strings.Cut(rest, ".")
return node, module, ok && node != "" && module != "" && !strings.Contains(module, ".")
}
// UnlistedFinding is the finding of module assigned to node and listed on none of its watched branches, raised by the composition
// step that leaves the assignment out (H-A-149).
func UnlistedFinding(node, module string, at time.Time) (mcseat.Finding, error) {
summary := fmt.Sprintf("%s is assigned %s, which no watched branch lists, so its composition leaves the assignment out", node, module)
return encode(Observed{Scope: ScopeAssignmentUnlisted, ID: node + "." + module, Kind: KindAssignmentUnlisted, Machine: node,
Severity: mcseat.FindingSeverityWarning, Summary: summary, Headline: clipHeadline(module + " on " + node + " is listed nowhere"),
Explanation: summary + ": a stale assignment is a fault of the assignments, not of the change judged (ADR 0578 §2). " +
"Take it off, or register the module on a watched branch.",
Source: "assigner." + KindAssignmentUnlisted}, at)
}
@@ -132,6 +132,20 @@ func (c *catalogHalf) Changed(ctx context.Context) func(branch, module string, e
}
}
// picture is build.Builder.Picture: the source revision the view of branch holds (design 47 H H-A-155), false while the branch
// is not followed or its view holds none.
func (c *catalogHalf) picture(branch string) (int64, bool) {
cat, ok := c.branches.Catalog(branch)
if !ok {
return 0, false
}
half, ok := cat.View()
if !ok {
return 0, false
}
return half.Revision.Source, true
}
// namesOf are the package names an entry's artifacts carry, in one line.
func namesOf(e meshcatalog.CatalogEntry) string {
var names []string
@@ -33,8 +33,13 @@ type catalogHolder struct {
mu sync.Mutex
entries map[string]meshcatalog.CatalogEntry
revision int64
// source is the source revision each trunk revision is taken in from; 0 is 1.
source int64
}
// sourceLocked is the source revision of the revisions put now. The caller holds mu.
func (h *catalogHolder) sourceLocked() int64 { return max(h.source, 1) }
func newCatalogHolder(t *testing.T, ctx context.Context, js jetstream.JetStream, nc *nats.Conn) *catalogHolder {
t.Helper()
h := &catalogHolder{ctx: ctx, js: js, entries: map[string]meshcatalog.CatalogEntry{}}
@@ -84,7 +89,7 @@ func (h *catalogHolder) whole() meshcatalog.RevisionAnswer {
all = append(all, e)
}
at := time.Now()
return meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: h.revision, Kind: "trunk", Branch: "main", Entries: refs, Source: 1, At: &at}, Entries: all}
return meshcatalog.RevisionAnswer{Revision: &meshcatalog.CatalogRevision{Revision: h.revision, Kind: "trunk", Branch: "main", Entries: refs, Source: h.sourceLocked(), At: &at}, Entries: all}
}
// put sets one module's entry in a new trunk revision of main and says catalog-revision.
@@ -100,7 +105,7 @@ func (h *catalogHolder) putAll(t *testing.T, es ...meshcatalog.CatalogEntry) {
h.entries[e.Module] = e
changed = append(changed, e.Module)
}
ev, _ := json.Marshal(meshcatalog.CatalogRevisionEvent{Branch: "main", Kind: "trunk", Revision: h.revision, Source: 1, Entries: int64(len(h.entries)), Changed: changed, V: 1})
ev, _ := json.Marshal(meshcatalog.CatalogRevisionEvent{Branch: "main", Kind: "trunk", Revision: h.revision, Source: h.sourceLocked(), Entries: int64(len(h.entries)), Changed: changed, V: 1})
h.mu.Unlock()
if _, err := h.js.Publish(h.ctx, meshcatalog.SubjectCatalogRevision("main"), ev); err != nil {
t.Fatal(err)
@@ -522,9 +522,9 @@ func isState(state string) func(builds.BuilderBuild) bool {
// the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4).
func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
b := startBed(t)
// DLV-B10 (design 51, round 2): the builder's versions, findings and the floor's three sources are on the bus; the seat
// answers 0.1.4, so netcheck's bundles are numbered from 0.1.5. Two merges close together, the first replaced before
// its claim: the second is numbered at once, and the first never.
// The builder's versions, findings and the floor's three sources are on the bus, the seat answering 0.1.4; two merges close
// together, the first replaced before its claim. The builder numbers neither in its state versions (design 47 H H-A-159,
// G §348, §369): the reservation is the one place a version is numbered.
reg := b.registry(t, "n1", "0.1.4")
const netcheck = "catalogue-a67f0196-netcheck.module.json"
first := "src2:" + strings.Repeat("1", 64)
@@ -534,8 +534,8 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
v := b.await(t, "netcheck", isState(builds.BuildDone))
for _, a := range []string{"tools-go", "tools-typescript"} {
es := reg.of(t, "netcheck/"+a)
if len(es) != 1 || es[0].Version != "0.1.5" || es[0].Fingerprint != fp || es[0].State != versions.Numbered || es[0].Digest != digestOf(a) {
t.Errorf("versions of netcheck/%s = %+v, want the second merge numbered 0.1.5 at once with its digest, the first never", a, es)
if len(es) != 0 {
t.Errorf("versions of netcheck/%s = %+v, want none: the builder numbers nothing of its own (H-A-159)", a, es)
}
}
if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 {
@@ -565,13 +565,13 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
}
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made.
// DLV-B10: between them the build log holds the build's narrations (build.said), each version numbered, and why.
// H-A-159: no version is numbered of the builder's own, so the build log holds no narration of one (build.said).
var ended []builds.BuildLogEntry
for _, e := range b.logged(t, "netcheck") {
if e.Event != builds.BuildSaidEvent {
ended = append(ended, e)
} else if e.Why != versions.SaidNumbered || !strings.Contains(e.Line, "numbered 0.1.5") || e.FP != fp {
t.Errorf("said = %+v, want netcheck's versions numbered", e)
} else if e.Why == versions.SaidNumbered || e.Why == versions.SaidKept {
t.Errorf("said = %+v, want no version of the builder's own numbered (H-A-159)", e)
}
}
if len(ended) != 2 || ended[0].Event != builds.BuildStartedEvent || ended[1].Event != builds.BuildEndedEvent ||
@@ -596,7 +596,7 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
}
}
// Design 32, design 47 G §348: a package's name the catalog half's entry of the module gives later reaches the builder by the
// branch follower's view (never the indexer's state), and looks at the module again; the done bundles are kept, not numbered twice.
// branch follower's view (never the indexer's state), and looks at the module again.
named := b.entry("netcheck", "modules/netcheck", commit, fp, digest, nil, bothToolchains, 0)
named.Manifest = b.manifestOf(t, "catalogue-a67f0196-netcheck.module.json")
named.Artifacts = []meshcatalog.Artifact{{Ref: "netcheck:lib", Language: "go", Package: &meshcatalog.ArtifactPackage{Ecosystem: "go", Name: "netcheck"}}}
@@ -606,11 +606,12 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
if got := reg.numberedOf("netcheck/tools-go"); len(got) != 0 {
t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got)
}
for _, a := range []string{"tools-go", "tools-typescript"} {
reg.published(t, "netcheck/"+a, "0.1.5", digestOf(a))
// Versions numbered before G, which the state keeps until G §372 removes it: one a holder removes below, one it refuses.
kept, _ := json.Marshal(versions.Entry{Name: "netcheck/tools-typescript", Kind: versions.KindBundle, Series: "0.1", Version: "0.1.5", Module: "netcheck",
Fingerprint: fp, Digest: digestOf("tools-typescript"), Seat: ociartifactregistry.Seat, State: versions.Published, At: time.Now().UTC()})
if _, err := reg.versions.Put(b.ctx, versions.Key("netcheck/tools-typescript", "0.1.5"), kept); err != nil {
t.Fatal(err)
}
reg.awaitOf(t, "netcheck/tools-go", func(es []versions.Entry) bool { return len(es) == 1 && es[0].State == versions.Published })
// A numbered version a holder will refuse below, written to the state as the builder numbers one.
refusable, _ := json.Marshal(versions.Entry{Name: "netcheck/tools-go", Kind: versions.KindBundle, Series: "0.1", Version: "0.1.6", Module: "netcheck",
Fingerprint: "src2:" + strings.Repeat("6", 64), Seat: ociartifactregistry.Seat, State: versions.Numbered, At: time.Now().UTC()})
if _, err := reg.versions.Put(b.ctx, versions.Key("netcheck/tools-go", "0.1.6"), refusable); err != nil {
@@ -643,7 +644,7 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
t.Errorf("version.numbered of netcheck/tools-go said %v, want none", got)
}
// DLV-B10: a build module's bound running out fails the trunk build, which is the builder's finding build-failed; the
// fingerprint built again keeps its number, said with why, and its done build clears the finding.
// fingerprint built again clears it with its done build, numbering nothing of the builder's own (H-A-159).
release := make(chan struct{})
b.fake.hang.Store(&release)
third := "src2:" + strings.Repeat("4", 64)
@@ -666,12 +667,6 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
close(release)
b.moduleFP(t, "netcheck", netcheck, "modules/netcheck", bothToolchains, "89abcdef0123456789abcdef0123456789abcdef", third, 3) // the entry moves to a later commit, same fingerprint
b.awaitFP(t, "netcheck", third, isState(builds.BuildDone))
b.awaitSaid(t, "kept: fingerprint "+third+" is numbered under it")
if !slices.ContainsFunc(b.logged(t, "netcheck"), func(e builds.BuildLogEntry) bool {
return e.Event == builds.BuildSaidEvent && e.Why == versions.SaidKept && e.FP == third && strings.Contains(e.Line, "kept: fingerprint "+third)
}) {
t.Errorf("the build log of netcheck says no version kept at %s", third)
}
reg.awaitFinding(t, failed, false)
// DLV-B4: the seats' refused and removed, heard on the builder's durable consumer, move a numbered version: a retention
// removal turns the entry removed with its reason, a refusal refused with the holder's reason and the bytes it refused.
@@ -2,6 +2,7 @@
// new-form fingerprint the indexer writes, dark, into its own build keys (builds.<module>.<fingerprint>), through the build
// modules' fetch and build tools. Nothing reads its keys to deliver until release R4e. It serves no request subject and
// holds no timer: every reaction is a watch or a bus event (ADR 0437 §8).
// Design 47 H1, proof 2 once more: the importer half of the edge mesh-delivery-sdk to mesh-delivery-builder.
package main
import (
@@ -35,6 +36,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
sweep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshartifactsweep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/operatorask"
"git.novox.be/novox/mesh-sdk/go/process"
@@ -156,9 +158,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done, on its calls connection.
// The builder numbers each trunk build at its claim (novox/hq design 51 *The versions*): its state versions, bound once
// the bus holds it; the floor's three sources asked as its own account.
versionsKV := process.AwaitBucket(ctx, js, conn, own.versions, "no trunk build is numbered: each is built unnumbered and published by its digest", logger.Printf)
// The state versions, kept from before G (design 51 *The versions*) and read through builder_versions until G §372 removes
// it, bound once the bus holds it; the builder numbers nothing in it (design 47 H H-A-159).
versionsKV := process.AwaitBucket(ctx, js, conn, own.versions, "builder_versions answers nothing of the versions numbered before G", logger.Printf)
found := &findings{module: cred.Module, w: fw, logf: logger.Printf, now: time.Now}
floorWaits := &versions.FloorWaits{Holder: versions.HolderOf(branches), Logf: logger.Printf, Said: found.floorSaid(context.WithoutCancel(ctx))}
// The names waiting on their floor are kept in the state floor-waits (DLV-B4): a restarted builder takes them in, so
@@ -205,13 +207,28 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
unput := &unputs{bounds: pubBoundsKV.KV, store: b.Claimer.Store, looking: b.Looking, findings: found,
bound: meshconditionsclient.CallUnansweredAfter, now: time.Now, logf: logger.Printf}
b.Left, b.Ended = unput.left, unput.ended
b.Numbers = build.VersionNumbers{N: numberer}
// The builder numbers nothing of its own (design 47 H H-A-159, G §348, §349, §369): b.Numbers stays nil, so no trunk build
// is numbered in the state versions, waits on a floor or arms a publish-unanswered bound of it; the reservation is the one
// place a version is numbered. The state, kept from before G, is still read (builder_versions) until G §372 removes it.
// The builder's toolchain waits (design 47 G §672): its state waits, and wait-held and wait-ended on its stream waits; its
// bound is its setting toolchain-wait-minutes.
waitBound, err := toolchainWaitOf(getenv("MESH_BUILDER_TOOLCHAIN_WAIT_FILE"))
if err != nil {
return err
}
// A build step ask waits for the builder's picture of its branch to hold its source revision (design 47 H H-A-155); its
// bound is its setting revision-wait-minutes (H-A-156).
revisionWait, err := numberFile(getenv("MESH_BUILDER_REVISION_WAIT_FILE"), int(build.DefaultRevisionWait/time.Minute))
if err != nil {
return err
}
b.RevisionWait = time.Duration(revisionWait) * time.Minute
// A step ask naming a package with no pin waits for its release (H-A-165); its bound is package-wait-minutes.
packageWait, err := numberFile(getenv("MESH_BUILDER_PACKAGE_WAIT_FILE"), int(build.DefaultPackageWait/time.Minute))
if err != nil {
return err
}
b.PackageWait = time.Duration(packageWait) * time.Minute
waitsBucket := process.AwaitBucket(ctx, js, conn, builds.StateBucket(cred.Module, build.WaitsState), "the toolchain waits are kept in memory only, and builder_waits shows none", logger.Printf)
b.Waits = newWaitsKept(waitsBucket, func(ctx context.Context, event, id string, body any) error {
return emitEvent(context.WithoutCancel(ctx), js, cred, event, id, body)
@@ -226,6 +243,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return emitSeatEvent(context.WithoutCancel(ctx), js, cred, meshbuilds.SubjectPackagePublished, "package-published-"+p.Module+"-"+p.Version+"-"+p.Artifact, p)
}
b.Reserved = reserved
// A release heard, or the newest the builder's record keeps, pins a package an entry names with none (H-A-165).
reserved.heardReleased = b.VersionReleased
b.NewestReleased = reserved.r.NewestReleased
// The last versions-filled the stream keeps is read at the start (design 47 G §369, §372): a restart does not lose it.
reserved.readFilled(ctx, lastKeptOf(js, cred, logger.Printf), sleeps)
// A state bound later numbers the builds looked at without it.
@@ -450,6 +470,8 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// fingerprint no branch carries is closed (issue 1660) once the first view of every followed branch is in (never at the start:
// a holder of the seat that is down holds no other part of it).
half.b = b
b.Picture = half.picture
branches.Held = func(string, meshcatalog.CatalogRevision) { b.PictureHeld() } // a revision held wakes a step ask's wait (H-A-162)
half.ready = func() {
if err := b.Sweep(ctx, half.pictured()...); err != nil {
errs <- err
@@ -44,12 +44,15 @@ type reservations struct {
// built asks a done build's publish (design 47 G A-457a), released ends its item; nil are none.
built func(ctx context.Context, module, version string) error
released func(ctx context.Context, module, version string)
filled atomic.Bool
bounds func() jetstream.KeyValue
found *findings
bound time.Duration
wake func(context.Context)
logf func(string, ...any)
// heardReleased is told each version-released taken in (design 47 H H-A-165): a step ask waiting for a package of the
// module wakes. Nil tells nobody.
heardReleased func(module, version string)
filled atomic.Bool
bounds func() jetstream.KeyValue
found *findings
bound time.Duration
wake func(context.Context)
logf func(string, ...any)
// call asks a seat's verb: the git seat's tag (design 47 G §370).
call toolCaller
// say says package-published on the seat mesh-builds; nil says none.
@@ -252,6 +255,9 @@ func (rv *reservations) heard(ctx context.Context, m jetstream.Msg) bool {
if rv.released != nil {
rv.released(ctx, ev.Module, ev.Version)
}
if rv.heardReleased != nil {
rv.heardReleased(ev.Module, ev.Version)
}
rv.found.clear(ctx, versions.VersionUnreleased, meshconditionsclient.ScopeModule, ev.Module)
default:
return false
@@ -134,10 +134,15 @@ func TestTheCatalogsWordsMoveTheReservations(t *testing.T) {
if err != nil || !strings.Contains(got.Summary, "m:b unpublished") || strings.Contains(got.Summary, "m:a unpublished") || !strings.Contains(got.Summary, "1.4.3") {
t.Fatalf("finding = %+v (%v), want version-unreleased naming m:b of 1.4.3", got, err)
}
var heardReleases []string // H-A-165: a release heard wakes a step ask waiting for a package of the module
rv.heardReleased = func(m, v string) { heardReleases = append(heardReleases, m+" "+v) }
released := &wakeMsg{subject: meshcatalog.SubjectVersionReleased, data: []byte(`{"v":1,"module":"m","version":"1.4.3","source":1,"artifacts":[]}`)}
if !rv.heard(ctx, released) || !released.acked {
t.Fatalf("version-released was not taken in (acked %v)", released.acked)
}
if strings.Join(heardReleases, ", ") != "m 1.4.3" {
t.Errorf("the release heard was told %v, want m 1.4.3 (H-A-165)", heardReleases)
}
if strings.Join(items, ", ") != "asked m 1.4.3, ended m 1.4.3" {
t.Errorf("publish items %v, want the build's end to ask the publish and the release to end it", items)
}
@@ -4,6 +4,7 @@ package main
import (
"encoding/json"
"strings"
"testing"
"time"
@@ -49,4 +50,29 @@ func TestTheRunnersBuildStepAskIsAnsweredAsStepEnded(t *testing.T) {
case <-time.After(20 * time.Second):
t.Fatal("no step-ended within 20 s")
}
// H-A-155: an ask naming a source revision the builder's picture of main does not hold yet waits for it, said in the
// journal, and is answered once a revision taken in from it arrives.
run.Pipeline, run.Revision, run.Source = "main.2.netcheck", 2, 2
data, _ = json.Marshal(run)
msg = nats.NewMsg(meshbuilds.AcceptSubjectStep)
msg.Data = data
msg.Header.Set("x-event-id", "ask-2")
if err := sub.PublishMsg(msg); err != nil {
t.Fatal(err)
}
b.awaitSaid(t, "main.2.netcheck: revision 2 of main not in the builder's picture yet")
b.catalog.mu.Lock()
b.catalog.source = 2
b.catalog.mu.Unlock()
// Revision 2, taken in from source 2, moves another module: netcheck's entry is the one the picture holds already.
b.moduleFP(t, "ca-trust", "catalogue-a67f0196-ca-trust.module.json", "modules/ca-trust", nil, commit, "src2:"+strings.Repeat("4", 64), 0)
select {
case m := <-ended:
var end meshbuilds.StepEnded
if err := json.Unmarshal(m.Data, &end); err != nil || end.ID != "ask-2" || end.Outcome != meshbuilds.StepEndedOutcomePassed {
t.Fatalf("step-ended = %+v, %v; want the ask ask-2 passed once the picture held source revision 2", end, err)
}
case <-time.After(20 * time.Second):
t.Fatal("no step-ended for ask-2 within 20 s")
}
}
@@ -0,0 +1,210 @@
package build
import (
"context"
"fmt"
"maps"
"slices"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
)
// **A build waits for its revision; a build with no version is never made** (design 47 H H-A-155, H-A-156; H-A-141): a step
// ask names its branch and source revision, and the builder reserves the version before it builds. When the builder's picture
// of that branch does not hold that source revision yet, or holds it but has not taken in the entry of the asked fingerprint
// from it, the ask waits for the picture to take it in, woken by each entry the picture takes in (take, EntryGone) and each
// revision its view holds (PictureHeld, H-A-162), never a
// timer or a poll, up to RevisionWait (the setting revision-wait-minutes); past it the ask fails, naming the revision and the
// branch, and nothing is built. A step ask with no source (a run on request) does not wait.
// DefaultRevisionWait is the bound of a step ask's wait for its revision, its setting revision-wait-minutes' default (H-A-156).
const DefaultRevisionWait = 2 * time.Minute
func (b *Builder) revisionWait() time.Duration {
if b.RevisionWait > 0 {
return b.RevisionWait
}
return DefaultRevisionWait
}
// PictureHeld is told each revision a branch's view takes in whole, whether or not an entry changed (H-A-162): every step
// ask waiting for its revision looks again.
func (b *Builder) PictureHeld() {
b.init()
b.pictureMoved()
}
// pictureMoved wakes every step ask waiting for the picture: an entry was taken in or went, or a revision was held.
func (b *Builder) pictureMoved() {
b.mu.Lock()
moved := b.moved
b.moved = make(chan struct{})
b.mu.Unlock()
close(moved)
}
// holdsRevision says whether the builder's picture of branch holds the source revision and has taken in every entry of the
// ask from that branch.
func (b *Builder) holdsRevision(branch string, source int64, entries []meshbuilds.BuildEntry) bool {
held, ok := b.Picture(branch)
if !ok || held < source {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
for _, e := range entries {
key, err := builds.BuilderBuildKey(e.Module, e.Fingerprint)
if err != nil {
return true // the build key is refused where the entry is built, with why
}
if b.origin[key].Branch != branch && b.heard[branch+"\x00"+e.Module].Fp2 != e.Fingerprint {
return false
}
}
return true
}
// awaitRevision returns no outcome once the builder's picture of the ask's branch holds its source revision; past the bound the
// step's outcome failed, at the builder's stop could not run, each with its words.
func (b *Builder) awaitRevision(ctx context.Context, run meshbuilds.StepRun) (outcome, said string) {
if b.Picture == nil || run.Source <= 0 || run.Branch == "" {
return "", ""
}
bound := b.revisionWait()
notHeld := fmt.Sprintf("revision %d of %s not in the builder's picture", run.Source, run.Branch)
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
defer stop()
told := false
for {
b.mu.Lock()
moved := b.moved
b.mu.Unlock()
if b.holdsRevision(run.Branch, run.Source, run.Entries) {
if told {
b.logf("%s: revision %d of %s is in the builder's picture: the build goes on", run.Pipeline, run.Source, run.Branch)
}
return "", ""
}
if !told {
b.logf("%s: %s yet: the ask waits for it, at most %s", run.Pipeline, notHeld, bound)
told = true
}
select {
case <-moved:
case <-wait.Done():
if ctx.Err() != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, notHeld + ": the builder stopped before it was"
}
b.logf("%s: %s within %s: the step fails, nothing is built", run.Pipeline, notHeld, bound)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notHeld, bound)
}
}
}
// **A package with no pin waits for its release** (design 47 H H-A-165, H-A-161): an entry keeps a mesh package none of
// whose versions is released with an empty version. Its step ask waits, as for its revision, for the release of the module
// that holds the package (the catalog's version-released, VersionReleased), up to PackageWait (the setting
// package-wait-minutes), then builds with that release as the package's pin (v<version>); past the bound the step fails,
// naming the package, and nothing is built.
// DefaultPackageWait is the bound of a step ask's wait for a package's release, its setting package-wait-minutes' default.
const DefaultPackageWait = 10 * time.Minute
func (b *Builder) packageWait() time.Duration {
if b.PackageWait > 0 {
return b.PackageWait
}
return DefaultPackageWait
}
// VersionReleased takes the catalog's version-released of module in: a step ask waiting for a package of it wakes.
func (b *Builder) VersionReleased(module, version string) {
b.init()
b.mu.Lock()
b.releases[module] = version
b.mu.Unlock()
b.pictureMoved()
}
// releaseOf is the pin a package is built against: the release of the module holding it, as Go names it; "" for none yet.
func (b *Builder) releaseOf(ctx context.Context, pkg string) string {
if b.Members == nil {
return ""
}
module, _, found, err := b.Members(ctx, pkg)
if err != nil || !found {
return ""
}
b.mu.Lock()
v := b.releases[module]
b.mu.Unlock()
if v == "" && b.NewestReleased != nil {
if kept, ok, err := b.NewestReleased(ctx, module); err == nil && ok {
v = kept
}
}
if v == "" {
return ""
}
return "v" + v
}
// awaitPins fills each package entry names with no pin from its release, waiting for one; no outcome once every package has
// a pin, failed past the bound, could not run at the builder's stop.
func (b *Builder) awaitPins(ctx context.Context, entry *meshcatalog.CatalogEntry) (outcome, said string) {
var unpinned []string
for p, v := range entry.Packages {
if v == "" {
unpinned = append(unpinned, p)
}
}
if len(unpinned) == 0 {
return "", ""
}
slices.Sort(unpinned)
bound := b.packageWait()
wait, stop := context.WithTimeout(ctx, bound) // a context's bound, no timer (ADR 0437 §8)
defer stop()
pins := map[string]string{}
told := false
for {
b.mu.Lock()
moved := b.moved
b.mu.Unlock()
missing := ""
for _, p := range unpinned {
if pins[p] == "" {
pins[p] = b.releaseOf(ctx, p)
}
if pins[p] == "" && missing == "" {
missing = p
}
}
if missing == "" {
packages := maps.Clone(entry.Packages)
maps.Copy(packages, pins)
entry.Packages = packages
if told {
b.logf("%s at %s: every package it names has a released version: built with %v", entry.Module, entry.Fp, pins)
}
return "", ""
}
notReleased := fmt.Sprintf("package %s has no released version", missing)
if !told {
b.logf("%s at %s: %s yet: the ask waits for its release, at most %s", entry.Module, entry.Fp, notReleased, bound)
told = true
}
select {
case <-moved:
case <-wait.Done():
if ctx.Err() != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, notReleased + ": the builder stopped before it was"
}
b.logf("%s at %s: %s within %s: the step fails, nothing is built", entry.Module, entry.Fp, notReleased, bound)
return meshbuilds.StepEndedOutcomeFailed, fmt.Sprintf("%s within %s", notReleased, bound)
}
}
}
@@ -103,10 +103,21 @@ type Builder struct {
Now func() time.Time
Node string
// Numbers numbers each trunk build at its claim and records what it was built with (design 51 *The versions*); nil
// numbers nothing.
// numbers nothing, as the builder's program leaves it (design 47 H H-A-159: the reservation numbers, G §369).
Numbers Numbers
// Reserved reserves the version of a trunk build at its start (design 47 G §369); nil reserves nothing.
Reserved Reservations
// Picture answers the source revision the builder's picture of a watched branch holds, false while it holds none (design 47
// H H-A-155, H-A-141): a step ask naming a newer one waits for it. Nil waits for nothing.
Picture func(branch string) (source int64, held bool)
// RevisionWait bounds a step ask's wait for its revision, its setting revision-wait-minutes (H-A-156); DefaultRevisionWait
// when 0.
RevisionWait time.Duration
// PackageWait bounds a step ask's wait for the release of a package its entry names with no pin, its setting
// package-wait-minutes (H-A-165); DefaultPackageWait when 0. NewestReleased reads the newest version the builder's
// record of releases keeps for a module; nil reads none, and only a release heard (VersionReleased) gives a pin.
PackageWait time.Duration
NewestReleased func(ctx context.Context, module string) (version string, found bool, err error)
// Forms answers what each node's engine states it reads (ADR 0618 §2, ADR 0624 §1): present, the nodes whose engine
// answered when last asked, and counted, every node the assignments names, present or away, a node with no statement
// stating nothing (the baseline's first forms). A done key names each artifact in the forms these keep, the oldest in
@@ -149,6 +160,10 @@ type Builder struct {
// builds are the ends that withdraw the asks of a key's build in flight, by build key (ADR 0664 §4): a newer fingerprint of
// the module takes its older one's build out, and the asker drops that build's asks.
builds map[string]context.CancelCauseFunc
// moved is closed and made again at each entry the picture takes in or drops: the step asks waiting for a revision wake.
moved chan struct{}
// releases are the versions heard released, by module (VersionReleased): a package with no pin waits for one.
releases map[string]string
}
type planned struct {
@@ -178,6 +193,8 @@ func (b *Builder) init() {
b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{}
b.hands, b.events = map[string]*handWatch{}, map[string]bool{}
b.builds = map[string]context.CancelCauseFunc{}
b.moved = make(chan struct{})
b.releases = map[string]string{}
})
}
@@ -230,6 +247,7 @@ func (b *Builder) EntryGone(branch string, first bool, module string) {
b.mu.Lock()
delete(b.heard, branch+"\x00"+module)
b.mu.Unlock()
b.pictureMoved()
if first {
b.Forget(module)
}
@@ -256,6 +274,7 @@ func (b *Builder) take(ctx context.Context, o Origin, key string, m meshcatalog.
b.edges[bk] = e
}
b.mu.Unlock()
b.pictureMoved()
if superseded != nil {
// A newer fingerprint of the module: the build of the older one in flight is taken out, and its asks are withdrawn
// from the build seats (ADR 0664 §4).
@@ -1095,6 +1114,7 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m mes
// The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes
// them into the workspace, whatever the repository's go.mod names.
ask.Packages = maps.Clone(m.Packages)
maps.DeleteFunc(ask.Packages, func(_, v string) bool { return v == "" }) // a package with no released version is no pin (H-A-161)
for _, fb := range req.Builds {
delete(ask.Packages, fb.Package) // a package `builds` names is named in neither packages nor package (§465)
}
@@ -332,10 +332,12 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
r := newRig(t, goBundle)
r.asks.Answer = nil // no node has pulled it yet
// Issue 1585 (ADR 0573 §2): the ask carries the package versions the fingerprint was computed with.
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": hex40('5')}
// Issue 1585 (ADR 0573 §2): the ask carries the module's pins; a package the entry keeps with no version (none
// released, design 47 H-A-161) is no pin.
sdk := buildstep.PackageVersions{"git.novox.be/novox/mesh-sdk/go": "v0.1.20"}
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20",
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains), Packages: sdk})
Fp2: fp1, Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(agreedToolchains),
Packages: map[string]string{"git.novox.be/novox/mesh-sdk/go": "v0.1.20", "git.novox.be/novox/mesh-contracts/go/seats/x": ""}})
if err != nil {
t.Fatal(err)
}
@@ -34,6 +34,10 @@ func (b *Builder) BuildStep(ctx context.Context, run meshbuilds.StepRun) meshbui
if b.StepEntry == nil {
return finish(meshbuilds.StepEndedOutcomeCouldNotRun, "the builder reads no entry of the seat mesh-catalog")
}
// The version is reserved before the build (H-A-155): the ask waits for the builder's picture to hold its revision.
if outcome, said := b.awaitRevision(ctx, run); outcome != "" {
return finish(outcome, said)
}
var failed []string
for _, e := range run.Entries {
outcome, said := b.buildEntry(ctx, e)
@@ -59,6 +63,10 @@ func (b *Builder) buildEntry(ctx context.Context, e meshbuilds.BuildEntry) (outc
if entry.Module != e.Module || (e.Commit != "" && entry.Commit != e.Commit) {
return meshbuilds.StepEndedOutcomeCouldNotRun, fmt.Sprintf("the entry of %s at %s is of %s at %s", e.Module, e.Fingerprint, entry.Module, entry.Commit)
}
// A package the entry names with no pin is built against its release, waited for (H-A-165).
if outcome, said := b.awaitPins(ctx, &entry); outcome != "" {
return outcome, said
}
key, err := builds.BuilderBuildKey(entry.Module, entry.Fp)
if err != nil {
return meshbuilds.StepEndedOutcomeCouldNotRun, err.Error()
@@ -3,13 +3,17 @@ package build
import (
"context"
"errors"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-builder/internal/versions"
@@ -112,6 +116,184 @@ func TestABuildStepPassesAReservingBuildOnlyWithItsReservedVersion(t *testing.T)
}
})
}
t.Run("H-A-155: an ask waits for the picture to take its revision in, then builds with its version reserved", aStepAskWaitsForItsRevision)
t.Run("H-A-155: past the bound the ask fails naming its revision, and nothing is built", aStepAskPastItsRevisionBoundFails)
t.Run("H-A-162: a revision held that changes no entry wakes the ask", aRevisionHeldWakesTheAsk)
t.Run("H-A-165: a package with no pin waits for its release and is built with it", aPackageWithNoPinWaitsForItsRelease)
t.Run("H-A-165: past its bound a package with no release fails the step", aPackageWithNoReleaseFailsPastItsBound)
}
// unpinnedRig is a rig whose step entry m names the package x with no version, built by the module contracts.
func unpinnedRig(t *testing.T) (*rig, *fakeReservations) {
var held atomic.Int64
r, res, entry := revisionRig(t, &held, make(chan struct{}, 1))
r.b.Picture = nil // no revision to wait for here
entry.Packages = map[string]string{"git.novox.be/novox/mesh-contracts/go/seats/x": "", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}
r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }
r.b.Members = func(_ context.Context, pkg string) (string, string, bool, error) {
return "contracts", "contracts:x", pkg == "git.novox.be/novox/mesh-contracts/go/seats/x", nil
}
return r, res
}
func aPackageWithNoPinWaitsForItsRelease(t *testing.T) {
r, _ := unpinnedRig(t)
var once sync.Once
looked := make(chan struct{})
r.b.NewestReleased = func(context.Context, string) (string, bool, error) { // the record holds none yet
once.Do(func() { close(looked) })
return "", false, nil
}
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
select {
case <-looked:
case end := <-ended:
t.Fatalf("the ask ended before the package was released: %+v, asks %+v", end, r.asks.Asks())
case <-time.After(5 * time.Second):
t.Fatal("the ask never looked for the package's release")
}
if _, ok := r.st.vals[r.key]; ok {
t.Fatal("the build was made before the package had a release")
}
r.b.VersionReleased("contracts", "0.1.3")
var end meshbuilds.StepEnded
select {
case end = <-ended:
case <-time.After(5 * time.Second):
t.Fatal("the release did not wake the ask")
}
r.b.Wait()
asks := r.asks.Asks()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(asks) == 0 ||
!reflect.DeepEqual(asks[0].Packages, buildstep.PackageVersions{"git.novox.be/novox/mesh-contracts/go/seats/x": "v0.1.3", "git.novox.be/novox/mesh-sdk/go": "v0.1.20"}) {
t.Errorf("end = %+v, asks %+v; want passed, built with x at v0.1.3", end, asks)
}
}
func aPackageWithNoReleaseFailsPastItsBound(t *testing.T) {
r, _ := unpinnedRig(t)
r.b.PackageWait = time.Nanosecond
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || built ||
!strings.Contains(end.Said, "package git.novox.be/novox/mesh-contracts/go/seats/x has no released version") {
t.Errorf("end = %+v, built %v; want failed naming the package, nothing built", end, built)
}
// A release the builder's own record keeps gives the pin at once.
r.b.NewestReleased = func(_ context.Context, module string) (string, bool, error) {
return "0.1.2", module == "contracts", nil
}
if end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.9.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}}); strings.Contains(end.Said, "no released version") {
t.Errorf("a release the record keeps did not pin the package: %+v", end)
}
}
func aRevisionHeldWakesTheAsk(t *testing.T) {
var held atomic.Int64
held.Store(40)
asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked)
r.b.Entry(context.Background(), Origin{Branch: "main", Source: 40, First: true}, entry) // the entry, unchanged since 40
r.b.Wait()
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.8.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 8, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
select {
case <-asked:
case end := <-ended:
t.Fatalf("the ask ended before the picture held its revision: %+v", end)
case <-time.After(5 * time.Second):
t.Fatal("the ask never read the builder's picture")
}
held.Store(41)
r.b.PictureHeld() // revision 8, from source 41, changes no entry
select {
case end := <-ended:
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 {
t.Errorf("end = %+v, reserved from %+v; want passed with its version reserved", end, res.sources)
}
case <-time.After(5 * time.Second):
t.Fatal("a revision held did not wake the ask")
}
}
// revisionRig is a rig whose builder reserves, its step entry m at fp1, its picture of main holding the source revision held.
func revisionRig(t *testing.T, held *atomic.Int64, asked chan struct{}) (*rig, *fakeReservations, meshcatalog.CatalogEntry) {
r := newRig(t, goBundle)
r.mf.byDigest[mdigest('a')] = catalogue.Manifest{Module: "m", Version: "0.1", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
res := &fakeReservations{filled: true}
r.b.Reserved = res
entry := meshcatalog.CatalogEntry{Module: "m", Repository: "novox/mesh-catalog", Path: "modules/m", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Fp: fp1,
ManifestDigest: mdigest('a'), Inputs: &meshcatalog.EntryInputs{Toolchains: CatalogToolchains(agreedToolchains)}}
r.b.StepEntry = func(context.Context, string) (meshcatalog.CatalogEntry, error) { return entry, nil }
r.b.Picture = func(branch string) (int64, bool) {
select {
case asked <- struct{}{}:
default:
}
return held.Load(), branch == "main" && held.Load() > 0
}
return r, res, entry
}
func aStepAskWaitsForItsRevision(t *testing.T) {
var held atomic.Int64
held.Store(40)
asked := make(chan struct{}, 1)
r, res, entry := revisionRig(t, &held, asked)
ended := make(chan meshbuilds.StepEnded, 1)
go func() {
ended <- r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
}()
select {
case <-asked: // the ask read the picture, which holds source revision 40
case end := <-ended:
t.Fatalf("the ask ended before the picture held its revision: %+v, reserved from %+v", end, res.sources)
case <-time.After(5 * time.Second):
t.Fatal("the ask never read the builder's picture")
}
if _, ok := r.st.vals[r.key]; ok {
t.Fatal("the ask built before the picture held its revision")
}
held.Store(41)
r.b.Entry(context.Background(), Origin{Branch: "main", Source: 41, First: true}, entry) // the picture takes revision 41 in
var end meshbuilds.StepEnded
select {
case end = <-ended:
case <-time.After(5 * time.Second):
t.Fatal("the ask was not woken by the entry the picture took in")
}
r.b.Wait()
if end.Outcome != meshbuilds.StepEndedOutcomePassed || len(res.sources) != 1 || res.sources[0].Revision != 41 {
t.Errorf("end = %+v, reserved from %+v; want passed with one version reserved from revision 41", end, res.sources)
}
}
func aStepAskPastItsRevisionBoundFails(t *testing.T) {
var held atomic.Int64
held.Store(40)
r, res, _ := revisionRig(t, &held, make(chan struct{}, 1))
r.b.RevisionWait = time.Nanosecond
end := r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "main.7.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Revision: 7, Source: 41,
Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
r.b.Wait()
if _, built := r.st.vals[r.key]; end.Outcome != meshbuilds.StepEndedOutcomeFailed || !strings.Contains(end.Said, "revision 41 of main not in the builder's picture") || built || len(res.sources) != 0 {
t.Errorf("end = %+v, built %v, reserved %+v; want failed naming revision 41 of main, nothing built", end, built, res.sources)
}
// A step ask with no source (a run on request) does not wait (H-A-156).
end = r.b.BuildStep(t.Context(), meshbuilds.StepRun{Pipeline: "run.1.m", Step: meshbuilds.StepRunStepBuild, Branch: "main", Entries: []meshbuilds.BuildEntry{{Module: "m", Fingerprint: fp1}}})
if strings.Contains(end.Said, "not in the builder's picture") {
t.Errorf("an ask with no source waited: %+v", end)
}
}
// digestOf is the manifest digest a step's entry names: the rig's manifest, or one no store holds, whose plan cannot be read.
@@ -3,16 +3,14 @@ package versions
import (
"context"
"fmt"
"strings"
"time"
)
// The tools' descriptions.
const (
FloorAgainDescription = "Ask the floor of one name again now, by hand (design 51 *The floor*): the seat's versions, the seat git's " +
"git.tags and the indexer's indexer_requirements, as the builder asks them by itself when a silent source says something. " +
"Answers which sources were silent and whether the floor is now known. A name the builder's versions already hold needs no floor: " +
"refused, saying so. A hand act."
FloorAgainDescription = "Refused for every name: the builder numbers nothing of its own and asks no floor (design 47 H H-A-159); " +
"a version is numbered by the reservation (design 47 G §369, builder_reserved). Kept until design 47 G §372 removes it with " +
"the state versions."
)
// Acts are the builder's hand acts on its versions, answered by its program (the tools bundle writes them to reasks).
@@ -31,24 +29,9 @@ type Acts struct {
DropReserved func(ctx context.Context, module, series, by, why string, at time.Time) (string, error)
}
// FloorAgain is builder_floor_again of name: the floor counts only for a name the versions hold nothing of.
func (a Acts) FloorAgain(ctx context.Context, name string) (string, error) {
entries, err := a.Numberer.Versions.Of(ctx, name)
if err != nil {
return "", err
}
if len(entries) > 0 {
return "", fmt.Errorf("%s holds %d versions: its floor counts only for its first number, so nothing is asked", name, len(entries))
}
answer := name + " waits on no floor in this builder: the floor is asked at its next claim"
if a.Numberer.Waits != nil {
for _, w := range a.Numberer.Waits.Waiting() {
if w.Name == name {
answer = fmt.Sprintf("the floor of %s is asked again (%s were silent)", name, strings.Join(w.Silent, ", "))
}
}
a.Numberer.Waits.Wake(WakeByHand, "", "", name)
}
a.Reconsider(ctx)
return answer + "; builder_versions shows its version once numbered", nil
// FloorAgain is builder_floor_again of name: refused, since the builder numbers nothing of its own and no claim asks a floor
// (design 47 H H-A-159; the reservation numbers, G §369). The tool stays until G §372 removes it with the state versions.
func (a Acts) FloorAgain(_ context.Context, name string) (string, error) {
return "", fmt.Errorf("the floor of %s is not asked: the builder numbers nothing of its own and asks no floor (design 47 H H-A-159); "+
"a version is numbered by the reservation (G §369, builder_reserved)", name)
}
@@ -43,6 +43,11 @@ func (n *Numberer) Heard(ctx context.Context, seat, event, id string, o Outcome)
var change func(*Entry)
switch event {
case HeardPublished:
// A published of other bytes than an entry records under its number is no publish of that entry: the reservation
// numbered it and takes it in (design 47 H H-A-159, G §369, A-415; ADR 0605 §4), and the entry stays as it is.
if e, ok, _, err := n.Versions.Recorded(ctx, o.Name, o.Version, o.Digest); err == nil && !ok && e.Digest != "" && o.Digest != "" && e.Digest != o.Digest {
return wake, fmt.Errorf("%s@%s is recorded with digest %s, so %s's published of %s is %w", o.Name, o.Version, e.Digest, seat, o.Digest, ErrNotNumbered)
}
change = func(e *Entry) {
e.State, e.Reason = Published, ""
if e.Digest == "" {
@@ -295,6 +295,23 @@ func (r *Reserver) ReleasedFor(ctx context.Context, module, fingerprint string)
return false, nil
}
// NewestReleased is the newest version the builder's own record of releases keeps for module (design 47 H H-A-165): a
// package with no pin is built against it; false when the record holds none.
func (r *Reserver) NewestReleased(ctx context.Context, module string) (string, bool, error) {
if r.Releases == nil {
return "", false, nil
}
raw, _, found, err := r.Releases.Get(ctx, ReleasedKey(module))
if err != nil || !found {
return "", false, err
}
var rec ReleasedRecord
if json.Unmarshal(raw, &rec) != nil || len(rec.Kept) == 0 {
return "", false, nil
}
return rec.Kept[len(rec.Kept)-1].Version, true, nil
}
// keepRelease adds a release to the module's record, dropping the oldest past ReleasedKept.
func (r *Reserver) keepRelease(ctx context.Context, module, fingerprint, version string) error {
n := r.ReleasedKept
@@ -201,11 +201,13 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
t.Errorf("reservations left: %v", all)
}
// One key per module holds the last releases, the oldest dropped (A-456b).
newest := ""
for _, fp := range []string{"src2:b1", "src2:b2"} {
n, err := r.Reserve(ctx, "m", "0.1", []string{"m:a"}, Source{Fingerprint: fp})
if err != nil {
t.Fatal(err)
}
newest = n.Version
if _, err := r.Released(ctx, "m", n.Version); err != nil {
t.Fatal(err)
}
@@ -219,6 +221,13 @@ func TestAReleaseIsRecordedByFingerprintBeforeItsReservationGoes(t *testing.T) {
if ok, _ := r.ReleasedFor(ctx, "m", "src2:b2"); !ok {
t.Error("the newest release is not kept")
}
// H-A-165: the newest version the record keeps is the pin of a package with none.
if v, ok, err := r.NewestReleased(ctx, "m"); !ok || err != nil || v != newest {
t.Errorf("newest released of m = %q, %v (%v), want %s", v, ok, err, newest)
}
if _, ok, _ := r.NewestReleased(ctx, "other"); ok {
t.Error("a module never released has a newest release")
}
}
// TestARebuildReservesAboveTheCatalogsHighestWhateverItsStandingNumberWas: design 47 G A-490b — a build again of a done key replaces
@@ -440,6 +440,13 @@ func TestTheBuildersVersions(t *testing.T) {
if wake, _ := n.Heard(ctx, SeatGoPackages, HeardPublished, "e2", Outcome{Name: "m/x", Digest: "sha256:dd"}); !reflect.DeepEqual(wake, []string{"m/lib"}) {
t.Fatalf("a published on the Go seat woke %v, want m/lib, whose seat was silent", wake)
}
// Design 47 G §369, A-415; ADR 0605 §4: a published of other bytes under the number is the reservation's, never this entry's.
if _, err := n.Heard(ctx, SeatArtifacts, HeardPublished, "e5", Outcome{Name: "m/program", Version: "0.1.14", Digest: "sha256:new"}); !errors.Is(err, ErrNotNumbered) {
t.Fatalf("a published of other bytes under 0.1.14: %v, want ErrNotNumbered", err)
}
if all, _ := n.Versions.Of(ctx, "m/program"); all[0].State != Numbered || all[0].Digest != "sha256:aa" {
t.Fatalf("after a published of other bytes: %+v, want it numbered with sha256:aa", all[0])
}
for _, h := range []struct{ event, state, reason string }{{HeardRefused, Refused, "bytes differ"}, {HeardPublished, Published, ""}, {HeardRefused, Published, ""}, {HeardRemoved, Removed, "retention"}} {
if _, err := n.Heard(ctx, SeatArtifacts, h.event, "e3", Outcome{Name: "m/program", Version: "0.1.14", Digest: "sha256:aa", Reason: h.reason}); err != nil {
t.Fatal(err)
@@ -581,11 +588,13 @@ func TestTheBuildersVersions(t *testing.T) {
}
_, _ = v.Mark(ctx, "m/p", "0.1.4", func(e *Entry) { e.State = Published })
acts := Acts{Numberer: &Numberer{Versions: v, Waits: &FloorWaits{}}, Reconsider: func(context.Context) {}}
if _, err := acts.FloorAgain(ctx, "m/p"); err == nil {
t.Error("a name with versions asks its floor again")
}
if ans, err := acts.FloorAgain(ctx, "m/new"); err != nil || ans == "" {
t.Errorf("floor again of a new name: %q %v", ans, err)
reconsidered := false
acts.Reconsider = func(context.Context) { reconsidered = true }
// H-A-159: the builder numbers nothing of its own, so no claim asks a floor: the tool refuses, saying so, and does nothing.
for _, name := range []string{"m/p", "m/new"} {
if ans, err := acts.FloorAgain(ctx, name); err == nil || !strings.Contains(err.Error(), "numbers nothing of its own") || reconsidered {
t.Errorf("floor again of %s: %q %v, looked again %v; want refused naming H-A-159, nothing looked at", name, ans, err, reconsidered)
}
}
})
}
@@ -186,9 +186,9 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
m.Streams[2].Name != "mesh-delivery-builder-findings" || !reflect.DeepEqual(m.Streams[2].Events, []string{"finding-written.*", "finding-cleared.*", "findings-stated.*"}) {
t.Errorf("streams = %+v, want waits with wait-held.** and wait-ended.**, 7 days, 16 MiB, the builds' stream (build.ended.**, 7 days, kept last per subject: design 47 G A-181) and the findings' stream", m.Streams)
}
if c, ok := m.Config["toolchain-wait-minutes"]; !ok || c.Default != float64(60) || len(m.Config) != 5 ||
if c, ok := m.Config["toolchain-wait-minutes"]; !ok || c.Default != float64(60) || len(m.Config) != 7 || m.Config["revision-wait-minutes"].Default != float64(2) || m.Config["package-wait-minutes"].Default != float64(10) ||
m.Config["publish-pause"].Default != float64(30) || m.Config["publish-pause-max"].Default != float64(30) || m.Config["publish-tries"].Default != float64(8) {
t.Errorf("config = %+v, want toolchain-wait-minutes of 60, the publish settings 30 s, 30 min and 8 tries (design 47 G §588, §633, A-457a)", m.Config)
t.Errorf("config = %+v, want toolchain-wait-minutes of 60, revision-wait-minutes of 2 (H-A-156), package-wait-minutes of 10 (H-A-165), the publish settings 30 s, 30 min and 8 tries (design 47 G §588, §633, A-457a)", m.Config)
}
// Design 47 G §349: version.numbered is no longer said; old-form-closed on the family version, apart from the builds' ends.
if want := []string{"build.asked-again", "version.old-form-closed"}; !reflect.DeepEqual(m.Emits, want) {
@@ -241,7 +241,7 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
t.Errorf("process = %v", p)
}
env, _ := p["env"].(map[string]any)
if len(env) != 6 || env["MESH_BUILDER_PUBLISH_TRIES_FILE"] != "${dir:state}/publish-tries" || env["MESH_BROKER_FILE"] != "${dir:state}/broker" || env["MESH_BUILDER_TOOLCHAIN_WAIT_FILE"] != "${dir:state}/toolchain-wait-minutes" {
if len(env) != 8 || env["MESH_BUILDER_REVISION_WAIT_FILE"] != "${dir:state}/revision-wait-minutes" || env["MESH_BUILDER_PACKAGE_WAIT_FILE"] != "${dir:state}/package-wait-minutes" || env["MESH_BUILDER_PUBLISH_TRIES_FILE"] != "${dir:state}/publish-tries" || env["MESH_BROKER_FILE"] != "${dir:state}/broker" || env["MESH_BUILDER_TOOLCHAIN_WAIT_FILE"] != "${dir:state}/toolchain-wait-minutes" {
t.Errorf("env = %v, want the broker file and the path of the toolchain wait's file, never the setting itself (design 47 G §672, A-71)", p["env"])
}
for k, v := range env {
+34
View File
@@ -147,6 +147,22 @@
"trusted": false,
"default": 60
},
"revision-wait-minutes": {
"type": "number",
"description": "minutes a build step ask waits for the builder's picture of its branch to hold its source revision before it fails, building nothing (design 47 H H-A-155, H-A-156).",
"minimum": 1,
"maximum": 60,
"trusted": false,
"default": 2
},
"package-wait-minutes": {
"type": "number",
"description": "minutes a build step ask waits for the release of a mesh package its entry names with no pin before it fails, building nothing (design 47 H H-A-165).",
"minimum": 1,
"maximum": 120,
"trusted": false,
"default": 10
},
"publish-pause": {
"type": "number",
"description": "seconds before a version's second publish try, doubled per try up to publish-pause-max (design 47 G A-457a).",
@@ -223,6 +239,22 @@
"mode": "0600",
"content": "${config:toolchain-wait-minutes}"
},
{
"id": "revision-wait",
"type": "file",
"path": "${dir:state}/revision-wait-minutes",
"owner": "mesh-delivery-builder",
"mode": "0600",
"content": "${config:revision-wait-minutes}"
},
{
"id": "package-wait",
"type": "file",
"path": "${dir:state}/package-wait-minutes",
"owner": "mesh-delivery-builder",
"mode": "0600",
"content": "${config:package-wait-minutes}"
},
{
"id": "publish-pause",
"type": "file",
@@ -269,6 +301,8 @@
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_BUILDER_TOOLCHAIN_WAIT_FILE": "${dir:state}/toolchain-wait-minutes",
"MESH_BUILDER_REVISION_WAIT_FILE": "${dir:state}/revision-wait-minutes",
"MESH_BUILDER_PACKAGE_WAIT_FILE": "${dir:state}/package-wait-minutes",
"MESH_BUILDER_PUBLISH_PAUSE_FILE": "${dir:state}/publish-pause",
"MESH_BUILDER_PUBLISH_PAUSE_MAX_FILE": "${dir:state}/publish-pause-max",
"MESH_BUILDER_PUBLISH_TRIES_FILE": "${dir:state}/publish-tries",
@@ -74,6 +74,8 @@ type Branches struct {
holders atomic.Pointer[seatHolders]
// Changed, when not nil, is told of each change to a branch's view (module, its entry, deleted), set before Wire.
Changed func(branch, module string, entry json.RawMessage, deleted bool, at time.Time)
// Held, when not nil, is told of each revision a branch's view takes in whole (Catalog.Held; H-A-162), set before Wire.
Held func(branch string, revision seat.CatalogRevision)
// Adopted, when not nil, is told on a goroutine of its own of each branch whose view was adopted, with that view (a part that waits for
// its first branch's first view, design 47 G §507), set before Wire.
Adopted func(branch string, c *Catalog)
@@ -253,6 +255,11 @@ func (b *Branches) reconcile() {
b.Changed(branch, module, entry, deleted, at)
}
}
c.Held = func(r seat.CatalogRevision) {
if b.Held != nil {
b.Held(branch, r)
}
}
// A catalog-revision also asks again each entry whose ask failed in transport (design 47 H, H-A-60).
hear := func(ctx context.Context, data []byte) error {
err := c.Hear(ctx, data)
@@ -34,7 +34,7 @@ type Half struct {
// ViewName is how the catalog half's revision is named in a part's first-view status (views.Status.View, WaitsFor).
func ViewName(revision int64) string { return strconv.FormatInt(revision, 10) }
// catalogCreated stands for the view's Created in its follower: a revision view is never made again, so one fixed time,
// catalogCreated stands for the view's Created in its follower until its numbering starts again (H-A-141): one fixed time,
// never zero, lets the follower tell a revision heard from one it holds.
var catalogCreated = time.Unix(0, 0).UTC()
@@ -57,6 +57,26 @@ type Catalog struct {
paging *Half
// waits is the revision waited for, 0 when none.
waits int64
// made is when the view's numbering last started again (H-A-141), the follower's Created; zero is catalogCreated.
made time.Time
// Held, when not nil, is told each revision the view takes in whole, once it holds it, whether or not an entry
// changed (design 47 H H-A-162): a part waiting for a revision wakes on it. Set before Start.
Held func(seat.CatalogRevision)
}
// created is madeLocked, taking mu.
func (c *Catalog) created() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.madeLocked()
}
// madeLocked is the view's Created in its follower: catalogCreated until a numbering started again. The caller holds mu.
func (c *Catalog) madeLocked() time.Time {
if c.made.IsZero() {
return catalogCreated
}
return c.made
}
// NewCatalog is a Catalog of branch asking the seat through call. Nothing is asked before Start.
@@ -146,21 +166,34 @@ func (c *Catalog) Hear(ctx context.Context, data []byte) error {
if c.held != nil {
held = c.held.Revision.Revision
}
if ev.Revision > c.waits {
c.waits = ev.Revision
}
c.mu.Unlock()
if ev.Revision <= held {
if ev.Revision == held {
c.mu.Unlock()
return nil
}
restarted := ev.Revision < held
if restarted {
// A revision below the one the picture holds is the state made again: its numbering started anew (design 47 H
// H-A-141). The view's made time moves on, so the follower forgets the revisions heard of the earlier numbering,
// and what it waits for is this revision.
c.made = time.Now().UTC()
c.waits = ev.Revision
} else if ev.Revision > c.waits {
c.waits = ev.Revision
}
made := c.madeLocked()
c.mu.Unlock()
// The follower keeps the newest revision heard, so an ask that was running when it was heard is asked once more.
c.Follower.Apply(views.Event{Owner: seat.Seat, State: c.Follower.State, Revision: uint64(ev.Revision), Created: catalogCreated, Gap: true})
c.Follower.Apply(views.Event{Owner: seat.Seat, State: c.Follower.State, Revision: uint64(ev.Revision), Created: made, Gap: true})
c.Follower.WaitFor(ctx, ViewName(ev.Revision))
why := fmt.Sprintf("catalog revision %d of %s said, the view holds %d", ev.Revision, c.Branch, held)
if restarted {
why = fmt.Sprintf("catalog revision %d of %s said, below the %d the view holds: the numbering started again", ev.Revision, c.Branch, held)
}
if c.Follower.Logf != nil {
c.Follower.Logf("meshcatalogclient: catalog revision %d of %s said, the view holds %d: %s is asked once more", ev.Revision, c.Branch, held, RevisionVerb)
c.Follower.Logf("meshcatalogclient: %s: %s is asked once more", why, RevisionVerb)
}
go func() {
_ = c.Follower.Gap(ctx, fmt.Sprintf("catalog revision %d of %s said, the view holds %d", ev.Revision, c.Branch, held))
_ = c.Follower.Gap(ctx, why)
}()
return nil
}
@@ -177,7 +210,7 @@ func (c *Catalog) ask(_ context.Context, after string) (views.Answer, error) {
a, err := askRevision(c.call, in)
if errors.Is(err, errNoneYet) {
// A branch with no revision yet is ready and empty, and nothing waits for it (§479, §495).
return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Created: catalogCreated, Values: map[string]json.RawMessage{}, At: time.Now()}, nil
return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Created: c.created(), Values: map[string]json.RawMessage{}, At: time.Now()}, nil
}
if err != nil {
return views.Answer{}, err
@@ -202,7 +235,9 @@ func (c *Catalog) ask(_ context.Context, after string) (views.Answer, error) {
}
rev := a.Revision.Revision
holds := false
if a.Next == "" {
whole := a.Next == ""
held := *a.Revision
if whole {
c.held, c.paging = c.paging, nil
if c.waits <= rev {
c.waits, holds = 0, true
@@ -212,7 +247,10 @@ func (c *Catalog) ask(_ context.Context, after string) (views.Answer, error) {
if holds {
c.Follower.Holds(ViewName(rev))
}
return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Revision: uint64(rev), Created: catalogCreated, Values: values, Next: a.Next, At: time.Now()}, nil
if whole && c.Held != nil {
c.Held(held)
}
return views.Answer{V: 1, Owner: seat.Seat, State: c.Follower.State, Revision: uint64(rev), Created: c.created(), Values: values, Next: a.Next, At: time.Now()}, nil
}
// errNoneYet is the seat's answer `none-yet`: the branch has no revision yet (design 47 G §530).
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"slices"
"strings"
"sync"
"testing"
@@ -83,6 +84,14 @@ func TestCatalogIsAViewPerWatchedBranch(t *testing.T) {
c.Follower.Wait = time.Nanosecond
c.Follower.Logf = logf
}
// H-A-162: each revision the view takes in whole is told to Held.
var heldMu sync.Mutex
var held []int64
main.Held = func(r seat.CatalogRevision) {
heldMu.Lock()
defer heldMu.Unlock()
held = append(held, r.Revision)
}
if main.Follower.State == rel.Follower.State || main.Follower.State != "catalog-revisions.main" || rel.Follower.State != views.BranchState(seat.StateCatalogRevisions, "release/1") {
t.Fatalf("views are named %q and %q", main.Follower.State, rel.Follower.State)
}
@@ -116,17 +125,25 @@ func TestCatalogIsAViewPerWatchedBranch(t *testing.T) {
}
deadline := time.Now().Add(5 * time.Second)
for {
if h, _ := main.View(); h.Revision.Revision == 4 {
heldMu.Lock()
told := len(held)
heldMu.Unlock()
if h, _ := main.View(); h.Revision.Revision == 4 && told == 2 {
break
}
if time.Now().After(deadline) {
t.Fatal("main never held revision 4")
t.Fatal("main never held revision 4, told to Held")
}
time.Sleep(time.Millisecond)
}
if n := f.asked("release/1"); n != 1 {
t.Errorf("release/1 asked %d times after main's revision", n)
}
heldMu.Lock()
if !slices.Equal(held, []int64{3, 4}) {
t.Errorf("Held was told %v, want 3 then 4", held)
}
heldMu.Unlock()
logMu.Lock()
said := false
for _, l := range logged {
@@ -188,3 +205,89 @@ func TestCandidatesReachTheHook(t *testing.T) {
t.Errorf("the hook was handed %+v, want the candidate's revision 4 alone", hooked)
}
}
// H-A-141 (design 47 H §750, §754): a catalog-revision below the revision the view holds is the state made again, its
// numbering started anew: said in the log with both revisions, and the first view asked again, so the view follows the new
// numbering and is behind nothing; one equal to the held revision asks nothing; one above is taken in as before.
func TestCatalogFollowsANumberingStartedAgain(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
f := &fakeRevisions{newest: map[string]int64{"main": 528}, asks: map[string]int{}}
var logMu sync.Mutex
var logged []string
c, err := NewCatalog(f.call, "main")
if err != nil {
t.Fatal(err)
}
c.Follower.Wait = time.Nanosecond
c.Follower.Logf = func(format string, args ...any) {
logMu.Lock()
defer logMu.Unlock()
logged = append(logged, fmt.Sprintf(format, args...))
}
came := make(chan string, 16) // the module of each entry that came, one per revision answered
c.Follower.Changed = func(module string, _ json.RawMessage, deleted bool, _ time.Time) {
if !deleted {
came <- module
}
}
if err := c.Start(ctx); err != nil {
t.Fatal(err)
}
<-c.Ready()
<-came // the first view's entry
for _, row := range []struct {
name string
newest int64 // what the seat answers from now
heard int64
asks int // the asks of the first view after the event
holds int64 // the revision the view holds after it
said string // a line the log holds after it; "" none new
restarting bool
}{
{name: "equal to the held revision: dropped", newest: 528, heard: 528, asks: 1, holds: 528},
{name: "above: taken in", newest: 529, heard: 529, asks: 2, holds: 529, said: "catalog revision 529 of main said, the view holds 528"},
{name: "below: the numbering started again", newest: 7, heard: 7, asks: 3, holds: 7, said: "catalog revision 7 of main said, below the 529 the view holds: the numbering started again", restarting: true},
{name: "equal after the restart: dropped", newest: 7, heard: 7, asks: 3, holds: 7},
{name: "above after the restart: taken in", newest: 8, heard: 8, asks: 4, holds: 8, said: "catalog revision 8 of main said, the view holds 7"},
} {
t.Run(row.name, func(t *testing.T) {
logMu.Lock()
before := len(logged)
logMu.Unlock()
f.set("main", row.newest)
if err := c.Hear(ctx, revisionEvent("main", KindTrunk, row.heard)); err != nil {
t.Fatal(err)
}
if row.said != "" {
select {
case m := <-came:
if want := fmt.Sprintf("m-of-main-%d", row.holds); m != want {
t.Fatalf("the entry %s came, want %s", m, want)
}
case <-ctx.Done():
t.Fatalf("the view never took revision %d in", row.holds)
}
}
if h, ok := c.View(); !ok || h.Revision.Revision != row.holds {
t.Fatalf("the view holds %+v, want revision %d", h.Revision, row.holds)
}
if n := f.asked("main"); n != row.asks {
t.Errorf("the first view was asked %d times, want %d", n, row.asks)
}
st := c.Follower.Status()
if st.Behind != "" || st.WaitsFor != "" {
t.Errorf("the view is behind %q, waits for %q, holding %d", st.Behind, st.WaitsFor, st.Revision)
}
logMu.Lock()
fresh := strings.Join(logged[before:], "\n")
logMu.Unlock()
if row.said == "" && strings.Contains(fresh, "meshcatalogclient:") {
t.Errorf("an event the view holds was said: %s", fresh)
}
if row.said != "" && !strings.Contains(fresh, row.said) {
t.Errorf("the log does not say %q: %s", row.said, fresh)
}
})
}
}
@@ -711,6 +711,8 @@ type CatalogRevisionDefinitionsItem struct {
type CatalogRevision struct {
// when (RFC 3339 time)
At *time.Time `json:"at,omitempty"`
// the newest kept revision of its branch below it, which the take-in that wrote it pinned for its pipeline: the base of the validate step's case before (design 47 H-A-147, H-A-151, H-A-154); absent on the branch's first take-in
Base int64 `json:"base,omitempty"`
// the watched branch the revision is of
Branch string `json:"branch,omitempty"`
// the seat definitions it was judged with, each by its digest, read whole through seat-definitions
@@ -59,7 +59,7 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v ModulesAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "the module verb (captured: xclip)", subject: "", given: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}", wire: "{\"module\":{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fp\":\"src2:b427bdde1f100e2f52490cfa9df0c5112dd3700226b4892349f0c14a15a4af56\",\"manifest\":{\"build\":{\"artifacts\":[{\"binary\":\"xclip-tools\",\"from\":\"cmd/xclip-tools\",\"kind\":\"bundle\",\"language\":\"go\",\"loads\":[\"xclip-tools\"],\"name\":\"tools\",\"system\":\"arch\"}]},\"capabilities\":[\"package-manager\"],\"module\":\"xclip\",\"resources\":[{\"id\":\"package\",\"package\":\"xclip\",\"type\":\"package\"}],\"tools\":[\"xclip_copy\",\"xclip_paste\",\"xclip_targets\",\"xclip_session\"],\"version\":\"1\"},\"module\":\"xclip\",\"path\":\"modules/xclip\",\"repository\":\"novox/mesh-catalog\"}}",
decode: func(b []byte) (any, error) { var v ModuleAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "a watched branch's catalog graph revision", subject: "", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}",
{name: "a watched branch's catalog graph revision", subject: "", given: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}", wire: "{\"at\":\"2026-11-09T19:00:00Z\",\"base\":9,\"branch\":\"main\",\"definitions\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"seat\":\"node-build-go\"}],\"entries\":[{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"mesh-build-go\"},{\"digest\":\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\",\"module\":\"xclip\"}],\"kind\":\"trunk\",\"movers\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"fingerprint\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\",\"module\":\"xclip\",\"repository\":\"novox/mesh-catalog\"}],\"revision\":12,\"source\":3589,\"trunk\":[{\"commit\":\"f469eb387a3015569d9b16c66ed418e9dd098f64\",\"repository\":\"novox/mesh-catalog\"}],\"v\":1}",
decode: func(b []byte) (any, error) {
var v CatalogRevisionsByBranchRevisionValue
err := json.Unmarshal(b, &v)
@@ -26,6 +26,7 @@ const (
VerbRerun = "rerun"
VerbAccept = "accept"
VerbCancel = "cancel"
VerbEndAgain = "end-again"
VerbPlace = "place"
VerbAskAgain = "ask-again"
VerbTestRuns = "test-runs"
@@ -47,7 +48,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string {
return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbPlace, VerbAskAgain, VerbTestRuns, VerbTestRunsDrop}
return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbEndAgain, VerbPlace, VerbAskAgain, VerbTestRuns, VerbTestRunsDrop}
}
// Events is every event published on the seat's subject: a new copy each call.
@@ -1195,6 +1196,19 @@ type CancelAnswer struct {
Pipeline Pipeline `json:"pipeline"`
}
// EndAgainInput is what end-again takes.
type EndAgainInput struct {
// the pipeline's key
Pipeline string `json:"pipeline"`
// why, written with the caller the runtime proves in the hand-act log
Why string `json:"why"`
}
// EndAgainAnswer is what end-again answers.
type EndAgainAnswer struct {
Pipeline Pipeline `json:"pipeline"`
}
// PlaceInput is what place takes.
type PlaceInput struct {
// the pipeline's key
@@ -119,6 +119,8 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v TestRunsDropAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "place asks a waiting pipeline ahead of its turn, with why", subject: "", given: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}", wire: "{\"pipeline\":\"main.3590\",\"why\":\"the operator wants the fix out first\"}",
decode: func(b []byte) (any, error) { var v PlaceInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "end-again asks a running pipeline's end judged once more, with why", subject: "", given: "{\"pipeline\":\"main.20\",\"why\":\"a mover waits on its registration and another failed\"}", wire: "{\"pipeline\":\"main.20\",\"why\":\"a mover waits on its registration and another failed\"}",
decode: func(b []byte) (any, error) { var v EndAgainInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
}
for _, c := range cases {
v, err := c.decode([]byte(c.given))
@@ -899,6 +899,8 @@ type CompositionRun struct {
// the outcome's words
Said string `json:"said,omitempty"`
Started *time.Time `json:"started,omitempty"`
// the module names of the assignments left out of the machine's composition, those no watched branch's newest revision lists, written when the run starts and read by every step's judgement and by the resume (design 47 H-A-152, H-A-153)
Unlisted []string `json:"unlisted,omitempty"`
// that machine's composer verdict (design 47 G §721)
Verdict map[string]any `json:"verdict,omitempty"`
}
@@ -208,7 +208,8 @@
"fingerprint": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"repository": "novox/mesh-catalog"
}
]
],
"base": 9
}
},
{
@@ -1927,6 +1927,10 @@
"$ref": "#/defs/TakeInEntry"
},
"description": "the movers of the take-in that wrote it, so a redelivered source-moved whose revision is kept says take-in.taken again from them, recomputing nothing (design 47 H §743, H-A-92); absent for one with no mover"
},
"base": {
"type": "integer",
"description": "the newest kept revision of its branch below it, which the take-in that wrote it pinned for its pipeline: the base of the validate step's case before (design 47 H-A-147, H-A-151, H-A-154); absent on the branch's first take-in"
}
},
"required": [
@@ -336,5 +336,13 @@
"pipeline": "main.3590",
"why": "the operator wants the fix out first"
}
},
{
"name": "end-again asks a running pipeline's end judged once more, with why",
"of": "verb:end-again.input",
"given": {
"pipeline": "main.20",
"why": "a mover waits on its registration and another failed"
}
}
]
@@ -474,6 +474,39 @@
},
"class": "act"
},
{
"name": "end-again",
"description": "A running pipeline's end judged once more, as at a step's end: it ends when it can, failed as soon as it can no longer pass (design 47 H §746, H-A-163, H-A-164). Written in the hand-act log.",
"input": {
"type": "object",
"properties": {
"pipeline": {
"type": "string",
"description": "the pipeline's key"
},
"why": {
"type": "string",
"description": "why, written with the caller the runtime proves in the hand-act log"
}
},
"required": [
"pipeline",
"why"
]
},
"answer": {
"type": "object",
"properties": {
"pipeline": {
"$ref": "#/defs/Pipeline"
}
},
"required": [
"pipeline"
]
},
"class": "act"
},
{
"name": "place",
"description": "A waiting pipeline placed ahead of its turn, its steps asked now; written in the hand-act log (design 47 H §748, H-A-53).",
@@ -1701,6 +1701,13 @@
"verdict": {
"type": "object",
"description": "that machine's composer verdict (design 47 G §721)"
},
"unlisted": {
"type": "array",
"items": {
"type": "string"
},
"description": "the module names of the assignments left out of the machine's composition, those no watched branch's newest revision lists, written when the run starts and read by every step's judgement and by the resume (design 47 H-A-152, H-A-153)"
}
},
"required": [
@@ -273,6 +273,8 @@ export interface CatalogRevisionDefinitionsItem {
export interface CatalogRevision {
/** when (RFC 3339 time) */
"at"?: string;
/** the newest kept revision of its branch below it, which the take-in that wrote it pinned for its pipeline: the base of the validate step's case before (design 47 H-A-147, H-A-151, H-A-154); absent on the branch's first take-in */
"base"?: number;
/** the watched branch the revision is of */
"branch"?: string;
/** the seat definitions it was judged with, each by its digest, read whole through seat-definitions */
@@ -16,6 +16,7 @@ export declare const verbs: {
readonly rerun: "rerun";
readonly accept: "accept";
readonly cancel: "cancel";
readonly endAgain: "end-again";
readonly place: "place";
readonly askAgain: "ask-again";
readonly testRuns: "test-runs";
@@ -683,6 +684,19 @@ export interface CancelAnswer {
"pipeline": Pipeline;
}
/** is what end-again takes. */
export interface EndAgainInput {
/** the pipeline's key */
"pipeline": string;
/** why, written with the caller the runtime proves in the hand-act log */
"why": string;
}
/** is what end-again answers. */
export interface EndAgainAnswer {
"pipeline": Pipeline;
}
/** is what place takes. */
export interface PlaceInput {
/** the pipeline's key */
@@ -16,6 +16,7 @@ export const verbs = {
rerun: "rerun",
accept: "accept",
cancel: "cancel",
endAgain: "end-again",
place: "place",
askAgain: "ask-again",
testRuns: "test-runs",
@@ -319,6 +319,8 @@ export interface CompositionRun {
/** the outcome's words */
"said"?: string;
"started"?: string;
/** the module names of the assignments left out of the machine's composition, those no watched branch's newest revision lists, written when the run starts and read by every step's judgement and by the resume (design 47 H-A-152, H-A-153) */
"unlisted"?: string[];
/** that machine's composer verdict (design 47 G §721) */
"verdict"?: Record<string, unknown>;
}
@@ -141,7 +141,7 @@ func main() {
tools = append(tools, OwedTools(askState(owedAsks, groups.OwedAsksBucket(served), 25*time.Second, func(v []byte) bool {
a, err := groups.DecodeOwedAsk(v)
return err == nil && a.Done()
}), func() ([]OwedRow, error) { return performedRows(performed) }, func(what, why string) error {
}), func(f OwedFilter) ([]OwedRow, error) { return performedRows(performed, f) }, func(what, why string) error {
_, err := recorder.Record(meshhandactsclient.Request{What: what, Why: why, Cause: "owed-dropped"})
return err
}, time.Now)...)
@@ -94,17 +94,55 @@ func TestTheBridgeTools(t *testing.T) {
}
return json.Marshal(a)
}
rows := []OwedRow{{Of: seat.Seat, ID: "a", Outcome: forgemsg.OwedPerformed, At: at}}
rows := []OwedRow{
{OwedEnded: forgemsg.OwedEnded{Of: seat.Seat, ID: "a", Outcome: forgemsg.OwedPerformed, At: at}, Repository: "novox/mesh-sdk", Number: 178},
{OwedEnded: forgemsg.OwedEnded{Of: seat.Seat, ID: "b", Outcome: forgemsg.OwedPerformed, At: at}, Repository: "novox/mesh-sdk", Number: 179}}
var filters []OwedFilter
performed := func(f OwedFilter) ([]OwedRow, error) {
filters = append(filters, f)
var out []OwedRow
for _, r := range rows {
if f.holds(r) {
out = append(out, r)
}
}
return out, nil
}
var recorded []string
recordErr := error(nil)
record := func(what, why string) error { recorded = append(recorded, what+": "+why); return recordErr }
tools := map[string]func(map[string]any) (any, error){}
for _, tl := range OwedTools(ask, func() ([]OwedRow, error) { return rows, nil }, record, func() time.Time { return at }) {
for _, tl := range OwedTools(ask, performed, record, func() time.Time { return at }) {
tools[tl.Name] = tl.Run
}
out, err := tools[OwedTool](map[string]any{"of": seat.Seat})
if m, _ := out.(map[string]any); err != nil || len(m["performed"].([]OwedRow)) != 1 || len(m["backlog"].([]groups.Backlog)) != 1 {
t.Errorf("bridge_owed: %v %v", out, err)
// bridge_owed reads one act or one pull request's acts, never the whole state (H-A-181).
for _, c := range []struct {
name string
args map[string]any
ids []string
}{
{"one pull request", map[string]any{"repository": "novox/mesh-sdk", "number": float64(179)}, []string{"b"}},
{"one act", map[string]any{"of": seat.Seat, "id": "a"}, []string{"a"}},
{"the whole state", map[string]any{"of": seat.Seat}, nil},
{"a repository with no number", map[string]any{"repository": "novox/mesh-sdk"}, nil},
} {
n := len(filters)
out, err := tools[OwedTool](c.args)
if c.ids == nil {
if err == nil || len(filters) != n {
t.Errorf("bridge_owed of %s: %v %v, read %d; want refused, nothing read", c.name, out, err, len(filters)-n)
}
continue
}
m, _ := out.(map[string]any)
got, _ := m["performed"].([]OwedRow)
var ids []string
for _, r := range got {
ids = append(ids, r.ID)
}
if err != nil || !reflect.DeepEqual(ids, c.ids) || len(m["backlog"].([]groups.Backlog)) != 1 {
t.Errorf("bridge_owed of %s: %v %v, want %v", c.name, out, err, c.ids)
}
}
if out, err := tools[OwedAgainTool](map[string]any{"of": seat.Seat, "id": "a"}); err != nil || out.(map[string]any)["outcome"] != forgemsg.OwedPerformed ||
asked[len(asked)-1].Kind != groups.OwedAgain {
@@ -21,26 +21,50 @@ const (
OwedDropTool = "bridge_owed_drop"
)
// OwedRow is one act the bridge performed, as the state performed keeps it: the outcome said as owed.ended.
type OwedRow = forgemsg.OwedEnded
// OwedRow is one act the bridge performed, as the state performed keeps it: the outcome said as owed.ended, and the pull
// request the act names (groups.Performed).
type OwedRow struct {
forgemsg.OwedEnded
Repository string `json:"repository,omitempty"`
Number int `json:"number,omitempty"`
}
// performedRows reads every act the state performed holds; a key that is not an act (a group end's bookkeeping) is left.
func performedRows(s stdio.KeptState) ([]OwedRow, error) {
keys, err := s.Keys()
if err != nil {
return nil, fmt.Errorf("the bridge's performed state cannot be listed: %w", err)
// OwedFilter is what bridge_owed reads of performed: one act (its id, of narrowing it to one key) or one pull request's
// acts (repository and number), never the whole state (novox/hq design 47 H-A-181).
type OwedFilter struct {
Of, ID, Repository string
Number int
}
func (f OwedFilter) holds(r OwedRow) bool {
return (f.Of == "" || r.Of == f.Of) && (f.ID == "" || r.ID == f.ID) &&
(f.Repository == "" || (r.Repository == f.Repository && r.Number == f.Number))
}
// performedRows reads the acts of performed f names, newest first: one key when f names the seat and the id, else each
// key's value held to f. A key that is not an act (a group end's bookkeeping) is left.
func performedRows(s stdio.KeptState, f OwedFilter) ([]OwedRow, error) {
var keys []string
if f.Of != "" && f.ID != "" {
keys = []string{groups.PerformedKey(f.Of, f.ID)}
} else {
all, err := s.Keys()
if err != nil {
return nil, fmt.Errorf("the bridge's performed state cannot be listed: %w", err)
}
keys = all
}
var rows []OwedRow
rows := []OwedRow{}
for _, k := range keys {
e, err := s.Get(k)
if err != nil || e == nil {
continue
}
var p struct {
Ended OwedRow `json:"ended"`
}
var p groups.Performed
if json.Unmarshal(e.Value, &p) == nil && p.Ended.Of != "" {
rows = append(rows, p.Ended)
if r := (OwedRow{OwedEnded: p.Ended, Repository: p.Repository, Number: p.Number}); f.holds(r) {
rows = append(rows, r)
}
}
}
sort.Slice(rows, func(i, j int) bool { return rows[i].At.After(rows[j].At) })
@@ -50,7 +74,7 @@ func performedRows(s stdio.KeptState) ([]OwedRow, error) {
// OwedTools are bridge_owed, bridge_owed_again and bridge_owed_drop: ask writes an owed-asks value and waits for the
// bridge's answer; performed reads the state performed; record writes a hand act in the seat mesh-hand-acts' log before
// a drop is asked, so a drop is never made without its record.
func OwedTools(ask func(key string, value []byte) ([]byte, error), performed func() ([]OwedRow, error), record func(what, why string) error, now func() time.Time) []stdio.Tool {
func OwedTools(ask func(key string, value []byte) ([]byte, error), performed func(OwedFilter) ([]OwedRow, error), record func(what, why string) error, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
act := map[string]any{
"of": map[string]any{"type": "string", "description": "the seat that said the act: mesh-delivery (the gate)"},
@@ -66,6 +90,12 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
}
return groups.DecodeOwedAsk(v)
}
read := map[string]any{
"repository": map[string]any{"type": "string", "description": "the pull request's repository, as owner/name; with number"},
"number": map[string]any{"type": "integer", "description": "the pull request's number; with repository"}}
for k, v := range act {
read[k] = v
}
withWhy := map[string]any{"why": map[string]any{"type": "string", "description": "why it is dropped, in plain words"}}
for k, v := range act {
withWhy[k] = v
@@ -74,22 +104,24 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
{Name: OwedTool,
Description: "What the forge bridge performed of what the gate and the checker owe the forge (design 47): each act it said as " +
"owed.ended, newest first, with its seat, id, pull request, outcome (performed, refused, could not be performed, " +
"undecodable, dropped), the forge's words and when, from its state performed (kept 7 days); and the backlog of its " +
"undecodable, dropped), the forge's words and when, from its state performed (kept 7 days), of one act (id, of " +
"narrowing it) or of one pull request (repository and number), never the whole state; and the backlog of its " +
"durable on each seat's stream owed: the last event applied, the events waiting and held, the age of the oldest. " +
"of or id narrow it. Read-only.",
Input: map[string]any{"type": "object", "properties": act},
"Read-only.",
Input: map[string]any{"type": "object", "properties": read},
Run: func(args map[string]any) (any, error) {
rows, err := performed()
f := OwedFilter{Of: str(args, "of"), ID: str(args, "id"), Repository: str(args, "repository")}
if n, ok := args["number"].(float64); ok {
f.Number = int(n)
}
if f.ID == "" && (f.Repository == "" || f.Number <= 0) {
return nil, fmt.Errorf("bridge_owed reads one act (id) or one pull request's acts (repository and number), never the whole state")
}
rows, err := performed(f)
if err != nil {
return nil, err
}
kept := []OwedRow{}
for _, r := range rows {
if (str(args, "of") == "" || r.Of == str(args, "of")) && (str(args, "id") == "" || r.ID == str(args, "id")) {
kept = append(kept, r)
}
}
out := map[string]any{"performed": kept}
out := map[string]any{"performed": rows}
if a, err := asked(groups.OwedBacklog, args); err != nil {
out["backlog-error"] = err.Error()
} else {
@@ -100,8 +132,8 @@ func OwedTools(ask func(key string, value []byte) ([]byte, error), performed fun
{Name: OwedAgainTool,
Description: "Perform one act owed to the forge again now, by hand, whatever its last outcome (a refusal the forge no longer " +
"makes, an act that could not be performed): the bridge calls the seat git's verb of the act under its id (safe to " +
"repeat) and says its outcome as owed.ended. The act is the one the bridge kept, or, for the gate's, the seat " +
"mesh-delivery's groups. Answers the outcome.",
"repeat) and says its outcome as owed.ended. The act is read from the stream owed of the seat that said it, on the " +
"subject of the pull request the bridge's state performed names (kept 7 days there). Answers the outcome.",
Input: map[string]any{"type": "object", "required": []string{"of", "id"}, "properties": act},
Run: func(args map[string]any) (any, error) {
a, err := asked(groups.OwedAgain, args)
@@ -0,0 +1,89 @@
package main
import (
"context"
"fmt"
"strconv"
"github.com/nats-io/nats.go/jetstream"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// The act bridge_owed_again performs again is read from the stream owed of the seat that said it, on the subject of the
// pull request performed names, the stream being the act's one copy (novox/hq design 47 G §162, §163, H-A-181).
// actReader reads the act the seat of said under id from its stream owed, on the subject of the pull request
// (repository and number) the act names, for decode to say each message's id; found false when the stream no longer
// holds it.
type actReader interface {
act(ctx context.Context, of, repository string, number int, id string, idOf func([]byte) string) (data []byte, found bool, err error)
}
// actOf is the act of to perform again, read from its seat's stream owed on the pull request p names.
func (o *owing) actOf(ctx context.Context, of string, p Performed, id string) ([]byte, error) {
read, err := readerOf(of)
if err != nil {
return nil, err
}
// A value of the form before H-A-182 names no pull request: every act the bridge decodes names its repository (a
// status may name no number, said on `.0` as the gate says it), so one with none is the old form, never read again.
if p.Repository == "" {
return nil, fmt.Errorf("performed names no pull request for the act %s of %s: kept in its old form, it expires with the state's TTL", id, of)
}
if o.acts == nil {
return nil, fmt.Errorf("the bridge reads no stream owed, so the act %s of %s cannot be performed again", id, of)
}
data, found, err := o.acts.act(ctx, of, p.Repository, p.Number, id, func(m []byte) string { c, _ := read(m); return c.ID })
if err != nil {
return nil, fmt.Errorf("the act %s of %s cannot be read from %s's stream owed: %w", id, of, of, err)
}
if !found {
return nil, fmt.Errorf("%s's stream owed no longer holds the act %s of %s#%d (kept 7 days)", of, id, p.Repository, p.Number)
}
return data, nil
}
// streamActs reads an act from the gate's stream owed (seat mesh-delivery's StreamOwed) by an ordered consumer filtered
// on the pull request's subject, as the gate says it (SubjectOwed with the repository escaped): every message the
// stream keeps there, the newest of the id kept.
type streamActs struct{ js jetstream.JetStream }
func (s streamActs) act(ctx context.Context, of, repository string, number int, id string, idOf func([]byte) string) ([]byte, bool, error) {
if of != seat.Seat {
return nil, false, fmt.Errorf("the seat %q keeps no stream owed the bridge reads", of)
}
subject := seat.SubjectOwed(keys.EscapeName(repository), strconv.Itoa(number))
st, err := s.js.Stream(ctx, seat.StreamOwed)
if err != nil {
return nil, false, err
}
info, err := st.Info(ctx, jetstream.WithSubjectFilter(subject))
if err != nil {
return nil, false, err
}
held := info.State.Subjects[subject]
if held == 0 {
return nil, false, nil
}
c, err := st.OrderedConsumer(ctx, jetstream.OrderedConsumerConfig{FilterSubjects: []string{subject}})
if err != nil {
return nil, false, err
}
var found []byte
for range held {
m, err := c.Next(jetstream.FetchContext(ctx))
if err != nil {
return nil, false, err
}
if idOf(m.Data()) == id {
found = m.Data()
}
if md, err := m.Metadata(); err == nil && md.NumPending == 0 {
break
}
}
return found, found != nil, nil
}
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"testing"
@@ -38,6 +39,24 @@ func (m memPerformed) Get(_ context.Context, key string) ([]byte, bool, error) {
}
func (m memPerformed) Put(_ context.Context, key string, v []byte) error { m[key] = v; return nil }
// memActs is a seat's stream owed by pull request subject (`<repository>#<number>`); asked records what was read.
type memActs struct {
held map[string][][]byte
asked []string
}
func (m *memActs) act(_ context.Context, of, repository string, number int, id string, idOf func([]byte) string) ([]byte, bool, error) {
pr := fmt.Sprintf("%s %s#%d", of, repository, number)
m.asked = append(m.asked, pr)
var found []byte
for _, d := range m.held[pr] {
if idOf(d) == id {
found = d
}
}
return found, found != nil, nil
}
// seatFake answers each seat verb from answers, by its key; one absent is not answered (an error, as a bound passed).
type seatFake struct {
calls []string
@@ -109,9 +128,12 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
if v, ok := kept[performedKey(c.of, e.ID)]; ok != c.kept {
t.Errorf("%s: kept %v (%s), want %v", c.name, ok, v, c.kept)
} else if ok {
// The outcome and when, and the pull request the act names, nothing more (§163, H-A-181, H-A-182).
var p Performed
if err := json.Unmarshal(v, &p); err != nil || p.Ended.Outcome != c.outcome || !json.Valid(p.Owed) {
t.Errorf("%s: kept %s, want the outcome and the owed event", c.name, v)
var fields map[string]json.RawMessage
if err := json.Unmarshal(v, &p); err != nil || p.Ended.Outcome != c.outcome || p.Ended.At.IsZero() ||
p.Repository != "novox/mesh-sdk" || p.Number != 178 || json.Unmarshal(v, &fields) != nil || len(fields) != 3 {
t.Errorf("%s: kept %s, want the outcome, when, and the pull request novox/mesh-sdk#178, nothing more", c.name, v)
}
}
}
@@ -142,6 +164,9 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
t.Run("asked again, dropped, and its backlog read by hand", func(t *testing.T) {
f := &seatFake{answers: statusOK}
o, pub, kept := newOwing(f)
// The act is read again from the gate's stream owed on its pull request's subject, never from performed (H-A-181).
acts := &memActs{held: map[string][][]byte{"mesh-delivery novox/mesh-sdk#178": {[]byte(`{`), gate("owed is said, a status")}}}
o.acts = acts
status := "status.novox/mesh-sdk#178.aaaaaaaaaaaa.mesh/merge-gate"
if _, err := o.heard(context.Background(), seat.Seat, gate("owed is said, a status")); err != nil {
t.Fatal(err)
@@ -153,11 +178,20 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
calls, said int
outcome, answers string
}{
{"again, from performed", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 2, forgemsg.OwedPerformed, "performed"},
{"again, from the gate's stream owed", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 2, forgemsg.OwedPerformed, "performed"},
{"dropped", groups.OwedAsk{Kind: groups.OwedDrop, Of: seat.Seat, ID: status, Why: "set by hand", By: "lead@shanks"}, 2, 3, forgemsg.OwedDropped, "set by hand"},
{"again, an act nobody holds", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: "x"}, 2, 3, "", "holds no owed act"},
{"the backlog", groups.OwedAsk{Kind: groups.OwedBacklog}, 2, 3, "", "backlog"},
{"again, the stream no longer holding it", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: status}, 2, 3, "", "no longer holds"},
{"again, a value of the old form", groups.OwedAsk{Kind: groups.OwedAgain, Of: seat.Seat, ID: "old"}, 2, 3, "", "names no pull request"},
} {
switch c.name {
case "again, the stream no longer holding it":
acts.held = nil
case "again, a value of the old form":
// Kept before H-A-182: the outcome and the whole act, no pull request; the stream is not read for it.
kept[performedKey(seat.Seat, "old")] = []byte(`{"ended":{"v":1,"of":"mesh-delivery","id":"old","outcome":"performed","at":"2026-11-09T21:00:00Z"},"owed":{"id":"old"}}`)
}
a := o.answer(context.Background(), c.ask, backlog)
if !a.Done() || a.Outcome != c.outcome || !strings.Contains(a.Answer, c.answers) || len(f.calls) != c.calls || len(pub.data) != c.said {
t.Errorf("%s: %+v, called %v, said %d; want %q naming %q, %d calls, %d said", c.name, a, f.calls, len(pub.data), c.outcome, c.answers, c.calls, c.said)
@@ -167,8 +201,12 @@ func TestWhatIsOwedIsPerformedSaidAndKept(t *testing.T) {
}
}
var p Performed
if err := json.Unmarshal(kept[performedKey(seat.Seat, status)], &p); err != nil || p.Ended.Outcome != forgemsg.OwedDropped {
t.Errorf("performed after the drop: %+v %v", p, err)
if err := json.Unmarshal(kept[performedKey(seat.Seat, status)], &p); err != nil || p.Ended.Outcome != forgemsg.OwedDropped ||
p.Repository != "novox/mesh-sdk" || p.Number != 178 {
t.Errorf("performed after the drop: %+v %v; want dropped, its pull request kept", p, err)
}
if len(acts.asked) != 2 || acts.asked[0] != "mesh-delivery novox/mesh-sdk#178" {
t.Errorf("the stream owed was read on %v, want twice on the pull request performed names", acts.asked)
}
})
}
@@ -22,6 +22,7 @@ import (
git "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-forge-bridge/internal/gitfixture"
@@ -33,7 +34,8 @@ import (
// no others, so a permission the bridge lacks fails a test here and not on the mesh (issue 1279).
//
// From module.json: the registrations and the gate's source-moved heard through its durable on EVENTS; branch-moved and
// what the gate owes heard through its durable on each seat's own stream (SEAT_<seat>_<event>); the seat git's verbs, the
// what the gate owes heard through its durable on each seat's own stream (SEAT_<seat>_<event>), and each seat's kept
// event also read through an ordered consumer of its own (mesh-grants heardGrants, A-233; bridge_owed_again, H-A-181); the seat git's verbs, the
// seat mesh-catalog's registrations, the seat mesh-delivery's source, the seat mesh-broker's show and the seat
// mesh-hand-acts' hand-act called on mesh.seat.<seat>.call.<verb>.<token> (or its plain fallback); its own states heads
// and owed-asks, and performed under the name its binding gives; its stream forge. `**` is written `>` on the wire. The
@@ -82,6 +84,11 @@ accounts {
"$JS.ACK.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge.>"
"$JS.API.CONSUMER.INFO.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge"
"$JS.API.CONSUMER.MSG.NEXT.SEAT_mesh-delivery_owed.n1_mesh-delivery-forge-bridge"
"$JS.API.CONSUMER.CREATE.SEAT_mesh-delivery_owed"
"$JS.API.CONSUMER.CREATE.SEAT_mesh-delivery_owed.>"
"$JS.API.CONSUMER.MSG.NEXT.SEAT_mesh-delivery_owed.*"
"$JS.API.CONSUMER.INFO.SEAT_mesh-delivery_owed.*"
"$JS.API.CONSUMER.DELETE.SEAT_mesh-delivery_owed.*"
"mesh.seat.mesh-broker.call.show.n1~mesh-delivery-forge-bridge"
"mesh.seat.mesh-broker.tool.show"
"mesh.seat.mesh-catalog.call.registrations.n1~mesh-delivery-forge-bridge"
@@ -162,6 +169,8 @@ type bed struct {
answer func(verb string, answer string)
// calls are the verbs the stand-in was called with, in order, as `<seat>.<verb> <input>`.
calls func() []string
// called is signalled after each call the stand-in answered, its answer chosen.
called <-chan struct{}
// restart stops the bridge, runs between, and starts it again on the same bus.
restart func(between func())
}
@@ -237,6 +246,7 @@ func startBed(t *testing.T, head string) *bed {
answers["git."+verb] = string(gitfixture.Given(t, verb+" answers", "verb:"+verb+".answer"))
}
var called []string
signal := make(chan struct{}, 1)
for _, subject := range []string{"mesh.seat.*.call.*.*", "mesh.seat.*.tool.*"} {
sub, err := ctl.Subscribe(subject, func(m *nats.Msg) {
parts := strings.Split(m.Subject, ".")
@@ -249,6 +259,10 @@ func startBed(t *testing.T, head string) *bed {
answer = answers[verb+" "+in.Repository]
}
mu.Unlock()
select {
case signal <- struct{}{}:
default:
}
if answer == "" {
answer = `{"error":"the stand-in holds no answer"}`
}
@@ -279,7 +293,7 @@ func startBed(t *testing.T, head string) *bed {
start()
t.Cleanup(func() { stop() })
set := func(verb, answer string) { mu.Lock(); answers[verb] = answer; mu.Unlock() }
return &bed{srv: srv, ctl: ctl, js: js, ctx: ctx, done: done, answer: set,
return &bed{srv: srv, ctl: ctl, js: js, ctx: ctx, done: done, answer: set, called: signal,
calls: func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), called...) },
restart: func(between func()) {
stop()
@@ -289,6 +303,30 @@ func startBed(t *testing.T, head string) *bed {
}}
}
// callsOf is how many calls of the stand-in start with prefix.
func callsOf(b *bed, prefix string) int {
n := 0
for _, c := range b.calls() {
if strings.HasPrefix(c, prefix) {
n++
}
}
return n
}
// calledTimes waits until the stand-in answered n calls starting with prefix: a call's answer is chosen when it is counted.
func calledTimes(t *testing.T, b *bed, prefix string, n int) {
t.Helper()
bound := time.After(20 * time.Second)
for callsOf(b, prefix) < n {
select {
case <-b.called:
case <-bound:
t.Fatalf("%d calls %s within 20 s, want %d", callsOf(b, prefix), prefix, n)
}
}
}
// headAt sets the stand-in seat git so main of the catalog is at head, protected or not.
func headAt(b *bed, head string, protected bool) {
p := "true"
@@ -363,7 +401,10 @@ func TestBridgeSaysOneHeadMovedPerMove(t *testing.T) {
b.restart(func() { headAt(b, h4, true) })
said("a restart after a move", forgemsg.HeadMoved{Head: h4, Before: h3, Start: true})
held(t, b, h4)
branchReads := callsOf(b, "git.branch ")
b.restart(func() {}) // at the head held: nothing
// The head moves only once the start has read the head held, so that read is not the move's (§739: start true).
calledTimes(t, b, "git.branch ", branchReads+1)
headAt(b, h5, true)
move(t, b, "main")
said("the next move, the first message after the restart at the head held", forgemsg.HeadMoved{Head: h5, Before: h4})
@@ -372,13 +413,11 @@ func TestBridgeSaysOneHeadMovedPerMove(t *testing.T) {
t.Errorf("the bridge asked %s; it asks the seat git nothing but branch", c)
}
}
heads, err := b.js.KeyValue(b.ctx, headsBucket(forgeBridgeModule()))
if err != nil {
t.Fatal(err)
}
if e, err := heads.Get(b.ctx, "branch.novox.mesh-catalog.main"); err != nil || string(e.Value()) != h5 {
t.Errorf("the state heads holds %v (%v), want %s", e, err, h5)
// One read per start and per move of main: the first view's registrations, said after a start, adopt no branch (§532).
if n := callsOf(b, "git.branch "); n != 3+4 {
t.Errorf("the bridge read the head %d times, want 7: once at each of 3 starts and once per move of main (4)", n)
}
held(t, b, h5) // kept after the message is written, so watched for, not read at once
}
func heard(t *testing.T, b *bed) <-chan *nats.Msg {
@@ -395,10 +434,16 @@ func heard(t *testing.T, b *bed) <-chan *nats.Msg {
return ch
}
// next is the next message heard on subject within a bound; the others are passed by.
// next is the next message heard on subject within 20 s; the others are passed by.
func next(t *testing.T, ch <-chan *nats.Msg, subject string) []byte {
t.Helper()
bound := time.After(20 * time.Second)
return nextWithin(t, ch, subject, 20*time.Second)
}
// nextWithin is next within bound.
func nextWithin(t *testing.T, ch <-chan *nats.Msg, subject string, within time.Duration) []byte {
t.Helper()
bound := time.After(within)
for {
select {
case m := <-ch:
@@ -406,7 +451,7 @@ func next(t *testing.T, ch <-chan *nats.Msg, subject string) []byte {
return m.Data
}
case <-bound:
t.Fatalf("nothing on %s within 20 s", subject)
t.Fatalf("nothing on %s within %s", subject, within)
return nil
}
}
@@ -455,8 +500,34 @@ func TestBridgePerformsWhatIsOwedOnce(t *testing.T) {
}
return n
}
// The bridge's acknowledgements on its durable of the stream owed, heard as the bus hears them.
acks := make(chan *nats.Msg, 16)
ackSub, err := b.ctl.ChanSubscribe("$JS.ACK."+seat.StreamOwed+".n1_mesh-delivery-forge-bridge.>", acks)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = ackSub.Unsubscribe() })
if err := b.ctl.Flush(); err != nil {
t.Fatal(err)
}
acked := func(n int) {
t.Helper()
bound := time.After(20 * time.Second)
for n > 0 {
select {
case m := <-acks:
if string(m.Data) == "+ACK" {
n--
}
case <-bound:
t.Fatalf("the bridge did not acknowledge what it heard on %s within 20 s", seat.StreamOwed)
}
}
}
gateOwed := gitfixture.Seat(t, seat.Seat, "events", "owed is said, a status", "event:owed")[0]
subject := seat.SubjectOwed("novox~mesh-sdk", "178")
// On the pull request's subject as the gate says it (its repository escaped as a key part), where bridge_owed_again
// reads the act again (H-A-181).
subject := seat.SubjectOwed(keys.EscapeName("novox/mesh-sdk"), "178")
for range 2 {
if _, err := b.js.Publish(b.ctx, subject, gateOwed); err != nil {
t.Fatal(err)
@@ -479,12 +550,26 @@ func TestBridgePerformsWhatIsOwedOnce(t *testing.T) {
_ = json.Unmarshal(gateOwed, &other)
other["id"] = "status.novox/mesh-sdk#178.aaaaaaaaaaaa.mesh/other"
otherOwed, _ := json.Marshal(other)
// The three acts heard are acknowledged before the stop: one stopped while its outcome is on its way is heard again
// after the start, by design, which is not what this part of the test is about.
acked(3)
// An act said while the bridge stops may be handed to the stopped bridge's pull before the bus has seen it go: the bus
// hands it again after its consumer's acknowledgement wait, its redelivery the only clock (H-A-48).
owedConsumer, err := b.js.Consumer(b.ctx, seat.StreamOwed, "n1_mesh-delivery-forge-bridge")
if err != nil {
t.Fatal(err)
}
ackWait := owedConsumer.CachedInfo().Config.AckWait
b.restart(func() {
if _, err := b.js.Publish(b.ctx, subject, otherOwed); err != nil {
t.Fatal(err)
}
})
if e := ended(); e.Of != seat.Seat || e.Outcome != forgemsg.OwedPerformed {
afterRestart, err := forgemsg.DecodeOwedEnded(nextWithin(t, ch, forgeSubject(forgemsg.OwedEndedEvent), ackWait+20*time.Second))
if err != nil {
t.Fatal(err)
}
if e := afterRestart; e.Of != seat.Seat || e.Outcome != forgemsg.OwedPerformed {
t.Errorf("the gate's status after the restart: %+v, want performed", e)
}
if n := count("git.status"); n != 2 {
@@ -521,12 +606,22 @@ func TestBridgePerformsWhatIsOwedOnce(t *testing.T) {
if a, err := groupkeys.DecodeOwedAsk(e.Value()); err == nil && a.Done() {
answered = true
if a.Outcome != forgemsg.OwedPerformed || count("git.status") != 3 {
t.Errorf("bridge_owed_again: %+v, status asked %d times; want performed again", a, count("git.status"))
t.Errorf("bridge_owed_again: %+v, status asked %d times; want performed again, read from the gate's stream owed", a, count("git.status"))
}
}
case <-bound:
t.Fatal("bridge_owed_again was not answered within 20 s")
}
}
// performed keeps the outcome, when and the pull request, never the act (§163, H-A-181, H-A-182).
kept, err := b.js.KeyValue(b.ctx, boundPerformed)
if err != nil {
t.Fatal(err)
}
if e, err := kept.Get(b.ctx, performedKey(seat.Seat, g.ID)); err != nil {
t.Errorf("performed of %s: %v", g.ID, err)
} else if fields := (map[string]json.RawMessage{}); json.Unmarshal(e.Value(), &fields) != nil || len(fields) != 3 || fields["owed"] != nil {
t.Errorf("performed of %s keeps %s, want the outcome, when and the pull request alone", g.ID, e.Value())
}
refusedRead(t, b, "mesh-delivery-gate_owed")
}
@@ -108,31 +108,38 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
performedKV := &boundBucket{binding: binding, local: performedLocal, js: js}
// What the gate owes the forge, performed through the seat git (design 47 *What the gate owes the forge*).
owe := &owing{call: call, pub: js, performed: kvPerformed{performedKV}, waits: binding.Waiting, now: time.Now, logger: logger}
owe := &owing{call: call, pub: js, performed: kvPerformed{performedKV}, acts: streamActs{js: js}, waits: binding.Waiting, now: time.Now, logger: logger}
keptCall := keptCallOf(ctx, call)
reader := &views.ProcessReader{Module: forgeBridgeModule(), Call: keptCall, Logf: logger.Printf, Hub: hub}
src := sourcebranches.New(sourcebranches.Call(keptCall))
var regs *meshcatalogclient.Registrations
turn := &sync.Mutex{}
mv := &mover{call: call, pub: js, heads: heads, watched: watched, turn: turn, logger: logger}
recompute := func() {
// begun says the start's read of the heads covers the watched branches from now on; it and what is watched are set
// under setting, so a branch added before begun is read by the start and one added after it is adopted.
var setting sync.Mutex
begun := false
recompute := func() (added bool) {
if regs != nil {
watched.Set(registeredOf(regs.All()), unescaped(src.Branches()))
added = watched.Set(registeredOf(regs.All()), unescaped(src.Branches()))
}
return added
}
started := make(chan struct{}) // closed once the watched branches are read
regs, _, err = meshcatalogclient.WireRegistrationsProcess(reader, keptCall, func(mcseat.Registration, bool) {
recompute()
select {
case <-started: // a branch registered while the bridge runs is adopted: its head read (§532)
setting.Lock()
adopt := recompute() && begun
setting.Unlock()
// Only a registration adding a watched branch while the bridge runs is adopted, its head read (§532); the first
// view's registrations, said after the start, add none, so the start reads their heads once, with start true (§739).
if adopt {
go mv.readHeads(ctx)
default:
}
})
if err != nil {
return err
}
src.Follower.Changed = func(string, json.RawMessage, bool, time.Time) { recompute() }
src.Follower.Changed = func(string, json.RawMessage, bool, time.Time) { setting.Lock(); recompute(); setting.Unlock() }
if _, err := reader.Adopt(src.Follower, sourcebranches.Seat, sourcebranches.EventSourceMoved, src.Hear); err != nil {
return err
}
@@ -167,7 +174,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
}
}
setting.Lock()
recompute()
begun = true
setting.Unlock()
close(started)
// What the gate says it owes, on its seat's stream owed through this module's durable there (made from its consumes).
// A stream not there yet is said, and the bridge goes on: its other work does not wait on the gate.
@@ -2,7 +2,6 @@ package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
@@ -22,22 +21,14 @@ import (
)
// The state performed (manifest `requires`, the template state, opened by its binding: bound.go) keeps one key per owed act it
// performed, the act's outcome and when, with the owed event as heard (novox/hq design 47 *What the gate owes the
// forge*). It is a saving and a read, never what keeps an act from happening twice: every act is safe to repeat at the
// forge.
// performed, in the form groups.Performed says (novox/hq design 47 G §163, H-A-181, H-A-182). It is a saving and a read,
// never what keeps an act from happening twice: every act is safe to repeat at the forge.
// performedKey is the key of one act in performed: the seat that said it and its id, the id in unpadded base64url, since
// an id may hold what a key may not (`#`, `/`).
func performedKey(of, id string) string {
return of + "." + base64.RawURLEncoding.EncodeToString([]byte(id))
}
// performedKey is the key of one act in performed (groups.PerformedKey).
func performedKey(of, id string) string { return groups.PerformedKey(of, id) }
// Performed is one value of the state performed: the outcome said as owed.ended, and the owed event it answers, so
// bridge_owed_again performs it again from here.
type Performed struct {
Ended forgemsg.OwedEnded `json:"ended"`
Owed json.RawMessage `json:"owed,omitempty"`
}
// Performed is one value of the state performed.
type Performed = groups.Performed
// performedStore is the state performed as the owing reads and writes it.
type performedStore interface {
@@ -70,6 +61,8 @@ type owing struct {
call caller
pub publisher
performed performedStore
// acts reads an act again from the stream owed of the seat that said it (H-A-181), for bridge_owed_again.
acts actReader
// waits, when set, says why the state performed cannot be used yet (its binding, design 47 G §577); empty when it can.
waits func() string
now func() time.Time
@@ -90,12 +83,12 @@ func readerOf(of string) (func([]byte) (bridge.OwedCall, error), error) {
func (o *owing) heard(ctx context.Context, of string, data []byte) (forgemsg.OwedEnded, error) {
read, err := readerOf(of)
if err != nil {
return o.end(ctx, forgemsg.OwedEnded{Of: of, Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, data)
return o.end(ctx, forgemsg.OwedEnded{Of: of, Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, "", 0)
}
c, err := read(data)
if err != nil {
return o.end(ctx, forgemsg.OwedEnded{Of: of, ID: c.ID,
Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, data)
Outcome: forgemsg.OwedUndecodable, Words: err.Error()}, c.Repository, c.Number)
}
if w := o.waiting(); w != "" {
return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, fmt.Errorf("the bridge %s", w)
@@ -104,7 +97,7 @@ func (o *owing) heard(ctx context.Context, of string, data []byte) (forgemsg.Owe
// said twice, or said again while its outcome was on its way: performed once, the outcome said once
return forgemsg.OwedEnded{Of: c.Of, ID: c.ID}, nil
}
return o.perform(ctx, c, data)
return o.perform(ctx, c)
}
// hear is one message of a seat's stream owed: performed and acknowledged, or, when its outcome could not be said,
@@ -127,19 +120,20 @@ func (o *owing) waiting() string {
}
// perform calls the seat git's verb of one act and says what it came to.
func (o *owing) perform(ctx context.Context, c bridge.OwedCall, data []byte) (forgemsg.OwedEnded, error) {
func (o *owing) perform(ctx context.Context, c bridge.OwedCall) (forgemsg.OwedEnded, error) {
answer, callErr := o.call(ctx, "seat:"+git.Seat+"."+c.Verb, c.Input)
outcome, words := bridge.Outcome(answer, callErr)
if outcome != forgemsg.OwedPerformed {
o.logger.Printf("the owed %s %s of %s#%d (said by %s) is %s: %s", c.Act, c.ID, c.Repository, c.Number, c.Of, outcome, words)
}
return o.end(ctx, forgemsg.OwedEnded{Of: c.Of, ID: c.ID,
Outcome: outcome, Words: words}, data)
Outcome: outcome, Words: words}, c.Repository, c.Number)
}
// end says an act's outcome as owed.ended on the stream forge, then keeps it in performed. An outcome not kept is
// logged: the act may be performed again, which changes nothing at the forge.
func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, owed []byte) (forgemsg.OwedEnded, error) {
// end says an act's outcome as owed.ended on the stream forge, then keeps it in performed with the pull request the act
// names (repository and number). An outcome not kept is logged: the act may be performed again, which changes nothing at
// the forge.
func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, repository string, number int) (forgemsg.OwedEnded, error) {
e.At = o.now().UTC()
data, err := forgemsg.EncodeOwedEnded(e)
if err != nil {
@@ -151,10 +145,7 @@ func (o *owing) end(ctx context.Context, e forgemsg.OwedEnded, owed []byte) (for
if e.ID == "" {
return e, nil // an undecodable act with no id is said, and has no key to keep
}
if !json.Valid(owed) {
owed = nil
}
v, err := json.Marshal(Performed{Ended: e, Owed: owed})
v, err := json.Marshal(Performed{Ended: e, Repository: repository, Number: number})
if err == nil {
err = o.performed.Put(ctx, performedKey(e.Of, e.ID), v)
}
@@ -175,19 +166,17 @@ func (o *owing) held(ctx context.Context, of, id string) bool {
return found
}
// owedOf is the owed event of an act to perform again, as performed keeps it.
func (o *owing) owedOf(ctx context.Context, of, id string) ([]byte, error) {
// kept is what performed holds of an act: its outcome and its pull request.
func (o *owing) kept(ctx context.Context, of, id string) (Performed, error) {
v, found, err := o.performed.Get(ctx, performedKey(of, id))
if err != nil {
return nil, fmt.Errorf("performed cannot be read: %w", err)
return Performed{}, fmt.Errorf("performed cannot be read: %w", err)
}
if found {
var p Performed
if err := json.Unmarshal(v, &p); err == nil && len(p.Owed) > 0 {
return p.Owed, nil
}
var p Performed
if !found || json.Unmarshal(v, &p) != nil {
return Performed{}, fmt.Errorf("the bridge holds no owed act %s of %s in its state performed", id, of)
}
return nil, fmt.Errorf("the bridge holds no owed act %s of %s in its state performed", id, of)
return p, nil
}
// answer answers one owed ask: an act performed again now, whatever performed holds of it; an act dropped, said as
@@ -199,20 +188,22 @@ func (o *owing) answer(ctx context.Context, a groups.OwedAsk, backlog func(conte
a.Backlog, a.Answer = backlog(ctx), "the backlog of each seat's stream owed"
return a
}
data, err := o.owedOf(ctx, a.Of, a.ID)
p, err := o.kept(ctx, a.Of, a.ID)
if err != nil {
a.Answer = err.Error()
return a
}
var e forgemsg.OwedEnded
if a.Kind == groups.OwedDrop {
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedDropped, Words: a.Why + " (dropped by " + a.By + ")"}, data)
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedDropped, Words: a.Why + " (dropped by " + a.By + ")"}, p.Repository, p.Number)
} else if read, rerr := readerOf(a.Of); rerr != nil {
err = rerr
} else if data, rerr := o.actOf(ctx, a.Of, p, a.ID); rerr != nil {
err = rerr
} else if c, rerr := read(data); rerr != nil {
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedUndecodable, Words: rerr.Error()}, data)
e, err = o.end(ctx, forgemsg.OwedEnded{Of: a.Of, ID: a.ID, Outcome: forgemsg.OwedUndecodable, Words: rerr.Error()}, p.Repository, p.Number)
} else {
e, err = o.perform(ctx, c, data)
e, err = o.perform(ctx, c)
}
if err != nil {
a.Answer = err.Error()
@@ -27,6 +27,23 @@ func TestAMoveOfAWatchedBranchIsOneHeadMoved(t *testing.T) {
if !watchedOf("main").Watches("novox/other", "main") || watchedOf("main").Watches("novox/other", "beta") {
t.Errorf("before the first registration the gate's source's branches are watched, of every repository")
}
// §532: only a registration adding a watched branch is adopted; the same registrations read again (a first view's
// values said after the start) add none.
regs := []Registered{{Origin: "novox/mesh-sdk", Branches: []string{"main", "beta"}}}
for _, c := range []struct {
name string
regs []Registered
want bool
}{
{"the same registrations again", regs, false},
{"a branch fewer", []Registered{{Origin: "novox/mesh-sdk", Branches: []string{"main"}}}, false},
{"a branch registered", regs, true},
{"a repository registered", append(regs, Registered{Origin: "novox/other", Branches: []string{"main"}}), true},
} {
if got := w.Set(c.regs, []string{"main"}); got != c.want {
t.Errorf("%s: added %v, want %v", c.name, got, c.want)
}
}
const head, before = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "cccccccccccccccccccccccccccccccccccccccc"
moved := Moved{Owner: "novox", Repo: "mesh-sdk", Branch: "feat/47-H"}
@@ -21,8 +21,10 @@ type Registered struct {
Branches []string
}
// Set takes what the bridge read: the registrations (none before the first) and the gate's source's branches by name.
func (w *Watched) Set(regs []Registered, source []string) {
// Set takes what the bridge read: the registrations (none before the first) and the gate's source's branches by name. It
// says whether a branch is watched now that was not (§532: a registration adding a watched branch is adopted); the same
// registrations read again add none.
func (w *Watched) Set(regs []Registered, source []string) (added bool) {
by := map[string][]string{}
for _, r := range regs {
for _, b := range r.Branches {
@@ -33,8 +35,33 @@ func (w *Watched) Set(regs []Registered, source []string) {
}
w.mu.Lock()
defer w.mu.Unlock()
before := w.pairsLocked()
w.byRepo, w.registered = by, len(regs) > 0
w.source = slices.Clone(source)
for p := range w.pairsLocked() {
if !before[p] {
added = true
}
}
return added
}
// pairsLocked is every watched branch as `<repository> <branch>`, the repository empty before the first registration
// (the gate's source's branch, of every repository). The caller holds mu.
func (w *Watched) pairsLocked() map[string]bool {
out := map[string]bool{}
if !w.registered {
for _, b := range w.source {
out[" "+b] = true
}
return out
}
for repo, branches := range w.byRepo {
for _, b := range branches {
out[repo+" "+b] = true
}
}
return out
}
// Watches says whether a move of branch in repository is one the bridge writes.
@@ -0,0 +1,26 @@
package groups
import (
"encoding/base64"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
)
// The state performed keeps one key per owed act the bridge performed, its outcome and when, and the pull request the
// act names, nothing more (novox/hq design 47 G §163, H-A-181, H-A-182): the act itself stays in the stream owed of the
// seat that said it, its one copy, where bridge_owed_again reads it again. The bridge writes it; its tools read it.
// PerformedKey is the key of one act in performed: the seat that said it and its id, the id in unpadded base64url, since
// an id may hold what a key may not (`#`, `/`).
func PerformedKey(of, id string) string {
return of + "." + base64.RawURLEncoding.EncodeToString([]byte(id))
}
// Performed is one value of the state performed (§163, H-A-182): the outcome said as owed.ended, with when, and the
// repository and number of the pull request the act names, so bridge_owed reads one pull request's acts and
// bridge_owed_again finds the act on its pull request's subject of the stream owed (H-A-181).
type Performed struct {
Ended forgemsg.OwedEnded `json:"ended"`
Repository string `json:"repository,omitempty"`
Number int `json:"number,omitempty"`
}
@@ -31,7 +31,6 @@ import (
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
@@ -49,10 +48,12 @@ import (
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
// controller's user and the indexer's own principal <node>.<module>.
// controller's user and the indexer's own principal <node>.<module>; and the bus's system account, whose disconnect events
// tell a restart when the stopped indexer's connections are gone (restartWaits).
const busConf = `
listen: 127.0.0.1:-1
jetstream { store_dir: %q }
system_account: SYS
accounts {
MESH {
jetstream: enabled
@@ -61,6 +62,11 @@ accounts {
{ user: "n1.mesh-delivery-indexer", password: "pw" }
]
}
SYS {
users = [
{ user: "sys", password: "pw" }
]
}
}
`
@@ -296,8 +302,9 @@ func startIndexerRun(t *testing.T, put func(context.Context, buildLog), before .
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
t.Cleanup(cancel)
// The gate's source-moved on its kept stream, with the indexer's durable of each watched branch the beds register
// (watchBranch), and the streams of the seat events the indexer consumes on durables of its own: its registration-changed
// and the runner's run-started, pipeline-started and pipeline-ended (design 47 H §743, §749), as the grants part makes them.
// (watchBranch), and the streams of the seat events the indexer consumes on durables of its own: its registration-changed,
// the runner's run-started, pipeline-started and pipeline-ended (design 47 H §743, §749) and the seat mesh-wants' pipeline-wanted
// (H-A-170), as the grants part makes them.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: meshdelivery.StreamSourceMoved, Subjects: []string{meshdelivery.SubjectSourceMovedAll}}); err != nil {
t.Fatal(err)
}
@@ -306,7 +313,7 @@ func startIndexerRun(t *testing.T, put func(context.Context, buildLog), before .
}
for stream, subject := range map[string]string{meshcatalog.StreamRegistrationChanged: meshcatalog.SubjectRegistrationChanged,
meshpipelinerunner.StreamRunStarted: meshpipelinerunner.SubjectRunStartedAll, meshpipelinerunner.StreamPipelineStarted: meshpipelinerunner.SubjectPipelineStartedAll,
meshpipelinerunner.StreamPipelineEnded: meshpipelinerunner.SubjectPipelineEndedAll} {
meshpipelinerunner.StreamPipelineEnded: meshpipelinerunner.SubjectPipelineEndedAll, meshwants.StreamPipelineWanted: meshwants.SubjectPipelineWantedAll} {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{subject}}); err != nil {
t.Fatal(err)
}
@@ -420,12 +427,54 @@ func startIndexerRun(t *testing.T, put func(context.Context, buildLog), before .
}
start()
t.Cleanup(func() { stop() })
// restart stops the program and starts it again on the same bus, as the node-engine does with a new build.
restart := func() { stop(); start() }
// restart stops the program and starts it again on the same bus, as the node-engine does with a new build, once the bus has
// closed every connection of the stopped one (restartWaits).
gone := restartWaits(t, srv)
restart := func() { gone(stop); start() }
return js, ctx, candidates, modules, restart
}
// restartWaits gives a stop that returns only once the bus has closed every connection the indexer held, heard on the system
// account's disconnect events, never by sleeping. A program's stop closes its connections on its side alone; until the bus
// has closed them its pulls on a durable still count, and a source-moved said in between is delivered to the stopped
// program, unacknowledged for the durable's whole acknowledgement wait, while the branch's durable delivers one message
// at a time (design 47 H §743).
func restartWaits(t *testing.T, srv *server.Server) func(stop func()) {
t.Helper()
sys, err := nats.Connect(srv.ClientURL(), nats.UserInfo("sys", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(sys.Close)
return func(stop func()) {
t.Helper()
disconnects, err := sys.SubscribeSync("$SYS.ACCOUNT.MESH.DISCONNECT")
if err != nil {
t.Fatal(err)
}
defer func() { _ = disconnects.Unsubscribe() }()
if err := sys.Flush(); err != nil {
t.Fatal(err)
}
open, err := srv.Connz(&server.ConnzOptions{User: "n1.mesh-delivery-indexer"})
if err != nil {
t.Fatal(err)
}
stop()
for left := open.NumConns; left > 0; {
m, err := disconnects.NextMsg(waitBound)
if err != nil {
t.Fatalf("the bus closed %d of the stopped indexer's %d connections within %s: %v", open.NumConns-left, open.NumConns, waitBound, err)
}
var ev server.DisconnectEventMsg
if json.Unmarshal(m.Data, &ev) == nil && ev.Client.User == "n1.mesh-delivery-indexer" {
left--
}
}
}
}
// The build seats and their holders as the mesh has them (design 47:482-485 and 1583: a language's build module is the
// module claiming the seat whose `acts` is step build on that language): the seat definitions as modules/mesh-delivery-contracts/seats/<seat>/
// seat.json has them (acts included) and the manifests as the catalog's modules/mesh-build-go and mesh-build-node claim them.
@@ -597,91 +646,6 @@ func (l buildLog) end(t *testing.T, repo, module, path string) {
}
}
// TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded (design 47 G §64, §67 test 64, §190; A-160, A-174):
// the indexer starts with the runner's streams run-started and pipeline-ended and its durables on them, its state runs and a
// registration of avahi; the runner says run-started of run 9 with avahi at a commit on a branch no registration watches;
// within 20 s candidates.run-9 holds one pull of number 0 at that commit, avahi's manifest read through the seat git; the
// indexer is restarted, the runner says avahi's pipeline of run 9 ended, and the candidate and the run's key go.
func TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded(t *testing.T) {
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil {
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const commit = "0123456789abcdef0123456789abcdef01234567"
var runs jetstream.KeyValue
js, ctx, candidates, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
var err error
if runs, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.RunsState), History: 1}); err != nil {
t.Fatal(err)
}
regs, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.RegistrationsState), History: 1})
if err != nil {
t.Fatal(err)
}
reg, _ := json.Marshal(meshcatalog.Registration{V: 1, Module: "avahi", Origin: "novox/mesh-catalog", Path: "modules/avahi", Branches: []string{"main"}})
if _, err := regs.Create(ctx, index.RegistrationKey("avahi"), reg); err != nil {
t.Fatal(err)
}
seedBuildModules(t, ctx, js, nil)
// The seat git as the forge module serves it: avahi's directory and its manifest at the run's commit.
reply := func(m *nats.Msg, result any) {
raw, _ := json.Marshal(map[string]any{"result": result, "node": "n0"})
_ = m.Respond(raw)
}
if _, err := js.Conn().Subscribe("mesh.seat.git.tool.trees", func(m *nats.Msg) {
reply(m, git.TreesAnswer{Root: "r0", Commit: commit, Trees: map[string]string{"modules/avahi": "newtree"}})
}); err != nil {
t.Fatal(err)
}
if _, err := js.Conn().Subscribe("mesh.seat.git.tool.file", func(m *nats.Msg) {
var a git.FileInput
_ = json.Unmarshal(m.Data, &a)
switch a.Path {
case "modules/avahi/module.json":
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Content: string(manifest)})
case "modules/avahi/go.sum":
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Content: goSum})
default:
reply(m, git.FileAnswer{Path: a.Path, Ref: a.Ref, Absent: true})
}
}); err != nil {
t.Fatal(err)
}
})
started, _ := json.Marshal(meshpipelinerunner.RunStarted{V: 1, Run: "9", By: "lead@n1", Members: []meshpipelinerunner.Member{{Module: "avahi", Commit: commit, Branch: "feature/1966-x"}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectRunStarted("avahi", "feature-1966-x", commit), started); err != nil {
t.Fatal(err)
}
key, _ := meshcatalogclient.CandidatesKey("run-9")
var c meshcatalog.CandidatesByGroupValue
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := candidates.Get(ctx, key); err == nil {
if c, err = index.DecodeCandidate(e.Value()); err == nil {
break
}
}
}
if len(c.Pulls) != 1 || c.Pulls[0].Number != 0 || c.Pulls[0].Head != commit || c.Pulls[0].Fps["avahi"].Manifest != meshcatalogclient.ManifestDigest(manifest) {
t.Fatalf("candidates.run-9 = %+v within 20 s of run-started, want avahi at the run's commit with its manifest", c)
}
restart()
ended, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: "feature-1966-x.0", Branch: "feature/1966-x", Run: "9",
Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}}}, Outcome: meshpipelinerunner.PipelineEndedOutcomePassed})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded("feature-1966-x", "0"), ended); err != nil {
t.Fatal(err)
}
gone := false
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && !gone; time.Sleep(50 * time.Millisecond) {
_, cerr := candidates.Get(ctx, key)
_, rerr := runs.Get(ctx, "run-9")
gone = errors.Is(cerr, jetstream.ErrKeyNotFound) && errors.Is(rerr, jetstream.ErrKeyNotFound)
}
if !gone {
t.Error("the run's candidate or its runs key stays 20 s after its member's pipeline ended across a restart")
}
}
// TestIndexerCountsOnlyThePlacedBuildNodesToolchains (issue 1940, issue 774): the toolchains bucket holds the report of
// n2, where the catalogue no longer places mesh-build-go, at another version than n1's, as ace's v26.12.0 stayed beside
// the v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's
@@ -691,6 +655,7 @@ func TestIndexerBuildsARunsCandidateAndDropsItWhenEveryMembersPipelineEnded(t *t
func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
var assignments, toolchains jetstream.KeyValue
var heard *takeIns
place := func(ctx context.Context, t *testing.T, node string, modules ...string) {
p := meshmoduleassigner.Assignments{Node: node}
for _, m := range modules {
@@ -712,6 +677,7 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
assignerLate = up
t.Cleanup(func() { assignerLate = nil })
js, ctx, _, modules, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
heard = hearTakeIns(t, js.Conn())
registerAvahi(t, ctx, js, "main")
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.ToolchainsBucket("mesh-build-go"), History: 1})
if err != nil {
@@ -762,6 +728,10 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
if !agreed(held) {
t.Fatalf("within 20 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27 alone", held.Fp2, held.None2, held.Toolchains)
}
// The module is written before its take-in ends: each restart below waits for the source-moved's acknowledgement, the
// take-in's end (design 47 H §743), so the stop cancels no take-in and leaves none unacknowledged ahead of the next one
// on the branch's durable, which delivers one message at a time.
heard.acked(t, "main")
place(ctx, t, "n2", "mesh-build-go")
key, _ := index.ToolchainKey("n2", "go")
value, err := index.EncodeToolchain(nodebuildgo.Toolchain{Language: "go", Version: "go1.28.1", Recipe: "1", At: ptr(time.Now().Add(time.Second))})
@@ -789,6 +759,7 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
if m := read(disagree); !disagree(m) {
t.Fatalf("within 20 s of a merge after n2's assignments avahi has fp2 %q, none2 %q: want the builders to disagree", m.Fp2, m.None2)
}
heard.acked(t, "main")
// The view moves only at a take-in (design 47 H-A-49, H-A-103): with n2's assignments taken away, the next take-in agrees.
place(ctx, t, "n2")
restart()
@@ -357,7 +357,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
registry.Pruned = ix.PruneBranchBuilds
ix.Registry = registry
// The gate's source revisions, on one durable per watched branch (design 47 H §743, H-A-90), bound below.
sources := &sourceMoved{ctx: ctx, ix: ix, branches: watchedBranches(registry), bind: bindBranch(js, cred, logger.Printf), logf: logger.Printf}
sources := &sourceMoved{ctx: ctx, ix: ix, branches: watchedBranches(registry), bind: bindBranch(js, cred, logger.Printf), source: sourceVia(calls.Call), logf: logger.Printf}
defer sources.stop()
named.Changed = ix.RemoveUnlisted
// The first fill of the versions (design 47 G §372): at the start, and again at each catalog revision kept until it is made.
@@ -182,9 +182,9 @@ func containsVia(call toolCaller) func(ctx context.Context, repository, commit,
var lockFiles = []string{"go.sum", "package-lock.json"}
// readVia reads a repository at a commit as the forge bridge does for a merged message: the seat git's trees, then the
// module.json and lock files of the one directory dir by its file (a repository can hold hundreds). A manifest the forge
// does not have is an error; a directory with no module.json in the trees is answered with no manifest; a lock file it does
// not have is left out.
// module.json and lock files of the one directory dir by its file (a repository can hold hundreds), and the go.mod beside a
// go.sum for its mesh packages (Packages). A manifest the forge does not have is an error; a directory with no module.json in
// the trees is answered with no manifest; a lock file it does not have is left out.
func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir string) (index.RepoRead, error) {
return func(_ context.Context, repository, sha, dir string) (index.RepoRead, error) {
a, err := treesOf(call, repository, git.TreesInput{Commit: sha})
@@ -222,13 +222,20 @@ func readVia(call toolCaller) func(ctx context.Context, repository, sha, dir str
out.Locks[dir] = map[string]string{}
}
out.Locks[dir][name] = hex.EncodeToString(sum[:])
if name == "go.sum" {
if p := fingerprint.MeshPackagesOf([]byte(c)); len(p) > 0 {
if out.Packages == nil {
out.Packages = map[string][]string{}
}
out.Packages[dir] = p // the mesh packages it uses, as the bridge reads them (ADR 0573 §2)
if name != "go.sum" {
continue
}
// A Go module's mesh packages are what its go.mod requires, at any version (design 47 H-A-157): its go.sum
// names none required at the zero version (ADR 0573 §2).
mod, absent, err := file(dir, "go.mod")
if err != nil {
return index.RepoRead{}, fmt.Errorf("%s: %w", path.Join(dir, "go.mod"), err)
}
if p := fingerprint.MeshRequiresOf([]byte(mod)); !absent && len(p) > 0 {
if out.Packages == nil {
out.Packages = map[string][]string{}
}
out.Packages[dir] = p
}
}
}
@@ -83,22 +83,29 @@ func TestTheTrunkIsTheDirectoriesTheForgeListsTrees(t *testing.T) { // 1900/AC3
// or a run that can never be taken in, is terminated, any other failure is left for redelivery, and a taken one is acked.
for _, row := range []struct {
name, body string
ended bool
ended, wanted bool
err error
acked, termd int
started, endedOf string
}{
{name: "a run started", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}],"by":"s"}`, acked: 1, started: "7"},
{name: "a run's pipeline ended", body: `{"v":1,"pipeline":"p","module":"a","branch":"b","commit":"c","outcome":"passed","steps":[],"run":"7"}`, ended: true, acked: 1, endedOf: "7"},
{name: "a pipeline's wants written to the end", body: `{"v":1,"pipeline":"main.12","branch":"main","source":12,"outcome":"tier-late"}`, wanted: true, acked: 1},
{name: "not json", body: `nope`, termd: 1},
{name: "a run never taken in", body: `{"v":1,"run":"7","members":[]}`, err: fmt.Errorf("%w: no member", index.ErrUnreadable), termd: 1, started: "7"},
{name: "a bus fault", body: `{"v":1,"run":"7","members":[{"module":"a","commit":"c"}]}`, err: errors.New("the bus did not answer"), started: "7"},
} {
ix := &fakeRuns{err: row.err}
m := &fakeMsg{subject: "x", data: []byte(row.body)}
if row.ended {
switch {
case row.wanted:
heardPipelineWanted(context.Background(), ix, m, t.Logf)
if ix.wanted != "main.12 tier-late" {
t.Errorf("%s: was handed %q, want main.12 tier-late", row.name, ix.wanted)
}
case row.ended:
heardPipelineEnded(context.Background(), ix, m, t.Logf)
} else {
default:
heardRunStarted(context.Background(), ix, m, t.Logf)
}
if m.acked != row.acked || m.termd != row.termd || ix.started != row.started || ix.ended != row.endedOf {
@@ -120,6 +127,8 @@ func TestReadViaReadsManifestsAndLocks(t *testing.T) {
return json.RawMessage(`{"path":"module.json","content":"{\"module\":\"snake\"}"}`), nil
case "go.sum":
return json.RawMessage(`{"path":"go.sum","content":"x"}`), nil
case "go.mod": // a mesh package required at the zero version, which no go.sum line names (design 47 H-A-157)
return json.RawMessage(`{"path":"go.mod","content":"module snake\n\nrequire (\n\tgit.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000\n)\n"}`), nil
}
return json.RawMessage(`{"path":"x","absent":true}`), nil
}
@@ -129,18 +138,21 @@ func TestReadViaReadsManifestsAndLocks(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(got.Manifests) != 1 || len(files) != 3 { // the one directory asked: its manifest and two lock files, never the others
if len(got.Manifests) != 1 || len(files) != 4 { // the one directory asked: its manifest, two lock files and the go.mod, never the others
t.Fatalf("manifests = %v, files asked = %v", got.Manifests, files)
}
if got.Root != "r0" || got.Manifests[""] != `{"module":"snake"}` || len(got.Locks[""]) != 1 || got.Locks[""]["go.sum"] == "" {
t.Fatalf("read = %+v", got)
}
if p := got.Packages[""]; len(p) != 1 || p[0] != "git.novox.be/novox/mesh-sdk/go" {
t.Fatalf("mesh packages %v, want mesh-sdk, which the go.mod requires", got.Packages)
}
}
// fakeRuns records the run events handed to it and answers err.
type fakeRuns struct {
err error
started, ended string
err error
started, ended, wanted string
}
func (f *fakeRuns) StartRun(_ context.Context, e meshpipelinerunner.RunStarted) error {
@@ -157,4 +169,9 @@ func (f *fakeRuns) HeardPins(context.Context, *meshpipelinerunner.PipelineStarte
return nil
}
func (f *fakeRuns) HeardPipelineWanted(_ context.Context, branch string, source int64, outcome string) error {
f.wanted = fmt.Sprintf("%s.%d %s", branch, source, outcome)
return f.err
}
func (f *fakeRuns) JudgePins(context.Context) {}
@@ -58,7 +58,8 @@ func untilModule(t *testing.T, ctx context.Context, js jetstream.JetStream, name
}
// TestASourceRevisionMovingMeshSDKNamesItAndItsImporters (design 47 H §743; A-476a; ADR 0593 §2 to §4): the indexer whole against
// the bus. Two Go modules of the catalogue, importer, whose go.sum requires mesh-sdk's Go module, and plain, which requires
// the bus. Two Go modules of the catalogue, importer, whose go.mod requires mesh-sdk's Go module at the zero version (no
// go.sum line names it, ADR 0573 §2; design 47 H-A-157), and plain, which requires
// nothing of it, are taken in with mesh-sdk as main's first source revision. A source revision moving mesh-sdk to a commit
// changing its buildstep package is taken in as one revision whose movers are mesh-sdk and importer, of another repository,
// and not plain; the packages key of mesh-sdk/go names that commit with a digest per package.
@@ -67,7 +68,7 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
sdkGo = "git.novox.be/novox/mesh-sdk/go"
m1, m2 = "1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222"
catalogMerge = "3333333333333333333333333333333333333333"
goSumContent = "git.novox.be/novox/mesh-sdk/go v0.1.14 h1:x=\ngit.novox.be/novox/mesh-sdk/go v0.1.14/go.mod h1:y=\n"
goModContent = "module importer\n\nrequire git.novox.be/novox/mesh-sdk/go v0.0.0-00010101000000-000000000000\n"
sdkRoot1, sdk2 = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
)
avahi, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
@@ -119,7 +120,8 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
forge.put(m2, sdkAt(sdk2, "f6", "7777777777777777777777777777777777777777"))
forge.put(catalogMerge, repoAt{trees: map[string]string{"modules/importer": "t1", "modules/plain": "t2"},
files: map[string]string{"modules/importer/module.json": named("importer"), "modules/plain/module.json": named("plain"),
"modules/importer/go.sum": goSumContent, "modules/plain/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n"}})
"modules/importer/go.mod": goModContent, "modules/importer/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n",
"modules/plain/go.mod": "module plain\n\nrequire golang.org/x/sys v0.1.0\n", "modules/plain/go.sum": "golang.org/x/sys v0.1.0 h1:x=\n"}})
regs := registerAvahi(t, ctx, js, "main")
for _, r := range []meshcatalog.Registration{{V: 1, Module: "importer", Origin: "novox/mesh-catalog", Path: "modules/importer", Branches: []string{"main"}},
{V: 1, Module: "plain", Origin: "novox/mesh-catalog", Path: "modules/plain", Branches: []string{"main"}},
@@ -142,7 +144,11 @@ func TestASourceRevisionMovingMeshSDKNamesItAndItsImporters(t *testing.T) {
say(10, true, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-sdk", Commit: m1}, meshdelivery.SourceMovedEventHeadsItem{Repository: "novox/mesh-catalog", Commit: catalogMerge})
heard.next(t, "main", 10)
heard.acked(t, "main")
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != "" && m.Packages[sdkGo] == m1 })
// mesh-sdk has released no version: importer keeps the package for its kind-1 edge, with no pin (H-A-161).
untilModule(t, ctx, js, "importer", func(m meshcatalog.ModulesByModuleValue) bool {
pin, kept := m.Packages[sdkGo]
return m.Fp2 != "" && kept && pin == ""
})
plain := untilModule(t, ctx, js, "plain", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != "" })
if _, ok := plain.Packages[sdkGo]; ok {
t.Fatalf("plain requires %s: %+v", sdkGo, plain.Packages)
@@ -6,15 +6,17 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The indexer pins a pipeline's revision on the runner's pipeline-started and frees it on its pipeline-ended, each heard on
// a durable consumer of its own on that event's stream (design 47 H §749, runs.go): an event is applied to the state pins,
// and only then acknowledged, so one said while the indexer was down is applied when it comes back.
// The indexer pins a pipeline's revision on the runner's pipeline-started and frees it on its pipeline-ended, a passed one's release
// pinned there and freed on the seat mesh-wants' pipeline-wanted of a newer pipeline (H-A-168, H-A-170), each heard on a durable
// consumer of its own on that event's stream (design 47 H §749, runs.go): an event is applied to the state pins, and only then
// acknowledged, so one said while the indexer was down is applied when it comes back.
// pinsStreams are the streams of the runner's events the indexer's pins follow.
var pinsStreams = []string{meshpipelinerunner.StreamPipelineStarted, meshpipelinerunner.StreamPipelineEnded}
// pinsStreams are the streams of the events the indexer's pins follow.
var pinsStreams = []string{meshpipelinerunner.StreamPipelineStarted, meshpipelinerunner.StreamPipelineEnded, meshwants.StreamPipelineWanted}
// pendingOf says how many messages of the indexer's durable consumers of the pins' streams have not been given to it yet.
func pendingOf(js jetstream.JetStream, cred process.Account) func(context.Context) (uint64, error) {
@@ -8,17 +8,20 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/internal/index"
)
// runTaker is what the indexer does with the runner's events: a run on request's (index.Indexer.StartRun, EndRunPipeline),
// and a pipeline's pins (HeardPins, design 47 H §749), the kept revisions judged after each pin freed (JudgePins).
// and a pipeline's pins (HeardPins, design 47 H §749), a release's freed on the seat mesh-wants' pipeline-wanted (HeardPipelineWanted,
// H-A-168, H-A-170), the kept revisions judged after each pin freed (JudgePins).
type runTaker interface {
StartRun(ctx context.Context, e meshpipelinerunner.RunStarted) error
EndRunPipeline(ctx context.Context, e meshpipelinerunner.PipelineEnded) error
HeardPins(ctx context.Context, started *meshpipelinerunner.PipelineStarted, ended *meshpipelinerunner.PipelineEnded) error
HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error
JudgePins(ctx context.Context)
}
@@ -33,6 +36,7 @@ func hearRuns(ctx context.Context, js jetstream.JetStream, cred process.Account,
{meshpipelinerunner.StreamRunStarted, func(m jetstream.Msg) { heardRunStarted(ctx, ix, m, logf) }},
{meshpipelinerunner.StreamPipelineStarted, func(m jetstream.Msg) { heardPipelineStarted(ctx, ix, m, logf) }},
{meshpipelinerunner.StreamPipelineEnded, func(m jetstream.Msg) { heardPipelineEnded(ctx, ix, m, logf) }},
{meshwants.StreamPipelineWanted, func(m jetstream.Msg) { heardPipelineWanted(ctx, ix, m, logf) }},
} {
_ = notHeard.Bind(s.stream, "no run on request's candidate is written or dropped until the indexer starts again", func() error {
cc, err := cred.HearSeatEvents(ctx, js, s.stream, s.each, logf)
@@ -85,6 +89,23 @@ func heardPipelineEnded(ctx context.Context, ix runTaker, m delivered, logf func
ix.JudgePins(ctx)
}
// heardPipelineWanted takes one pipeline-wanted of the seat mesh-wants in: the pipeline's wants are written to the end, tier-late
// included, so the older releases of the modules it moves free their revisions, and the kept revisions are judged (H-A-168,
// H-A-170).
func heardPipelineWanted(ctx context.Context, ix runTaker, m delivered, logf func(string, ...any)) {
var e meshwants.PipelineWanted
if err := json.Unmarshal(m.Data(), &e); err != nil || e.Branch == "" || e.Source <= 0 {
logf("%s: not a pipeline-wanted naming a branch and a source, dropped: %v", m.Subject(), err)
_ = m.Term()
return
}
err := ix.HeardPipelineWanted(ctx, e.Branch, e.Source, e.Outcome)
settleRun(m, err, "pipeline-wanted of "+e.Pipeline, logf)
if err == nil {
ix.JudgePins(ctx)
}
}
func settleRun(m delivered, err error, what string, logf func(string, ...any)) {
switch {
case err == nil:
@@ -7,6 +7,7 @@ import (
"fmt"
"slices"
"sync"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
@@ -20,17 +21,22 @@ import (
// the event's stream, named <node>_<indexer>_<branch>, the branch escaped as a key's name is (H-A-90), each delivering one message
// at a time (the grants part makes it so, H-A-30), acknowledged once its catalog revision and take-in.taken are written. The
// watched branches are the registrations'; a newly watched branch's consumer is bound on the indexer's own
// registration-changed, not only at its start.
// registration-changed, not only at its start. At its start, and again whenever a branch's consumer cannot be bound, the
// branch's current source revision is read through the seat mesh-delivery's verb source and taken in as a heard source-moved
// of it would be, when the view keeps no revision of the branch at or above it (H-A-129, H-A-130): at a clean state the
// consumers are made only once the indexer answers its modules view, which it does only after its first take-in.
type sourceMoved struct {
ctx context.Context
ix *index.Indexer
branches func(context.Context) ([]string, error) // the watched branches the registrations name
bind func(ctx context.Context, branch string, each func(jetstream.Msg)) (stop func(), err error)
source func(ctx context.Context, branch string) (meshdelivery.SourceAnswer, error) // the gate's source revision of a branch
logf func(string, ...any)
mu sync.Mutex
bound map[string]func() // by branch: the consumer's stop
again sync.Mutex // one take-in again at a time
mu sync.Mutex
bound map[string]func() // by branch: the consumer's stop
started bool // the start's refresh ran: every branch's source was read once
again sync.Mutex // one take-in again at a time
}
// watchedBranches are the branches every registration watches, each once, sorted.
@@ -53,10 +59,40 @@ func watchedBranches(r *index.Registry) func(context.Context) ([]string, error)
}
}
// bindBranch binds the indexer's durable of one watched branch on the stream of source-moved through mesh-sdk.
// sourceVia reads a watched branch's current source revision through the seat mesh-delivery's verb source, as the
// indexer's own account (H-A-129): a call where an answer is needed. An answer naming why it could not be read is that
// failure. The branch is given by its name, as the gate reads it (it escapes the name for its key itself).
func sourceVia(call toolCaller) func(context.Context, string) (meshdelivery.SourceAnswer, error) {
verb := "seat:" + meshdelivery.Seat + "." + meshdelivery.VerbSource
return func(_ context.Context, branch string) (meshdelivery.SourceAnswer, error) {
var a meshdelivery.SourceAnswer
raw, err := call(verb, meshdelivery.SourceInput{Branch: branch})
if err != nil {
return a, err
}
if err := json.Unmarshal(raw, &a); err != nil {
return a, fmt.Errorf("the answer of %s is not its seat's answer: %w", verb, err)
}
if a.Error != "" {
return a, fmt.Errorf("%s: %s", verb, a.Error)
}
return a, nil
}
}
// bindBranch binds the indexer's durable of one watched branch on the stream of source-moved through mesh-sdk. One the bus does
// not hold the indexer makes itself, as its own kept consumer, and then binds (design 47 H §743, H-A-134: the grants part
// composes none until H4b); one the bus holds is bound as it stands, never made again.
func bindBranch(js jetstream.JetStream, cred process.Account, logf func(string, ...any)) func(context.Context, string, func(jetstream.Msg)) (func(), error) {
return func(ctx context.Context, branch string, each func(jetstream.Msg)) (func(), error) {
cons, err := cred.NamedEventConsumer(ctx, js, meshdelivery.StreamSourceMoved, keys.EscapeName(branch))
if errors.Is(err, jetstream.ErrConsumerNotFound) {
cfg := branchConsumerConfig(cred, branch)
if cons, err = js.CreateConsumer(ctx, meshdelivery.StreamSourceMoved, cfg); err != nil {
return nil, fmt.Errorf("the consumer %s on %s is not there and is not made: %w", cfg.Durable, meshdelivery.StreamSourceMoved, err)
}
logf("source-moved: the consumer %s on %s was not there: the indexer made it (H-A-134)", cfg.Durable, meshdelivery.StreamSourceMoved)
}
if err != nil {
return nil, err
}
@@ -68,31 +104,107 @@ func bindBranch(js jetstream.JetStream, cred process.Account, logf func(string,
}
}
// branchConsumerConfig is the indexer's durable of one watched branch as the mesh gives a module's seat-event consumer
// (mesh-sdk's SeatEventConsumerConfig), named <node>_<indexer>_<branch> as NamedEventConsumer binds it, filtered to the
// branch's source-moved and delivering one message at a time, with the ack wait of 30 s, the at most 5 deliveries and the
// start from now the grants part gives a module's consumer on a seat event's stream kept by age (mesh-grants buscompose
// heardConsumer): the branch's current source revision is read at the start through the gate's source (H-A-129, H-A-130), so
// its past source-moved are never taken in again (design 47 H §743, H-A-90, H-A-134).
func branchConsumerConfig(cred process.Account, branch string) jetstream.ConsumerConfig {
esc := keys.EscapeName(branch)
cfg := cred.SeatEventConsumerConfig([]string{meshdelivery.SubjectSourceMoved(esc, "*")}, false)
cfg.Durable = cred.EventConsumerName() + "_" + esc
cfg.MaxAckPending = 1
cfg.AckWait, cfg.MaxDeliver = branchAckWait, branchMaxDeliver
cfg.DeliverPolicy = jetstream.DeliverNewPolicy
return cfg
}
// The ack wait and the deliveries of a branch's durable, as the grants part gives a module's consumers (H-A-134, H-A-133).
const (
branchAckWait = 30 * time.Second
branchMaxDeliver = 5
)
// refresh binds the consumer of every watched branch not bound yet: at the start and on each registration-changed. A consumer
// not there yet is said and bound at the next registration-changed or start.
// not there yet is said and bound at the next registration-changed or start. The start's refresh reads every watched branch's
// source revision, a later one each branch whose consumer could not be bound (H-A-129), after the binds.
func (s *sourceMoved) refresh() {
branches, err := s.branches(s.ctx)
if err != nil {
s.logf("source-moved: the watched branches are not read, so no new branch's consumer is bound: %v", err)
return
}
for _, branch := range s.bindAll(branches) {
s.readSource(branch)
}
}
// bindAll binds the branches not bound yet and answers those whose source revision is to be read now: at the start every
// watched branch, later each one whose consumer could not be bound.
func (s *sourceMoved) bindAll(branches []string) (read []string) {
s.mu.Lock()
defer s.mu.Unlock()
if s.bound == nil {
s.bound = map[string]func(){}
}
start := !s.started
s.started = true
for _, b := range branches {
if _, ok := s.bound[b]; ok {
if start {
read = append(read, b)
}
continue
}
branch := b
stop, err := s.bind(s.ctx, branch, func(m jetstream.Msg) { s.heard(branch, m) })
if err != nil {
s.logf("source-moved: the consumer of the branch %s is not bound, so its source revisions are not taken in until the next registration-changed or start: %v", branch, err)
s.logf("source-moved: the consumer of the branch %s is not bound, so its later source revisions are not heard until the next registration-changed or start; its current one is read through the gate's source: %v", branch, err)
read = append(read, branch)
continue
}
s.bound[branch] = stop
s.logf("source-moved: the consumer of the branch %s is bound", branch)
if start {
read = append(read, branch)
}
}
return read
}
// readSource reads a branch's current source revision through the gate's verb source and takes it in as a heard
// source-moved of that revision would be (takeIn), when the view keeps no revision of the branch at or above it (H-A-129,
// H-A-130: index.SourceToTake).
func (s *sourceMoved) readSource(branch string) {
if s.source == nil {
return
}
a, err := s.source(s.ctx, branch)
if err != nil {
s.logf("source-moved: the gate's source revision of %s is not read, so it is taken in only once its source-moved is heard: %v", branch, err)
return
}
if a.Revision <= 0 {
s.logf("source-moved: the gate keeps no source revision of %s yet", branch)
return
}
s.again.Lock()
defer s.again.Unlock()
e, take, err := s.ix.SourceToTake(s.ctx, branch, a)
if err != nil {
s.logf("source-moved: the catalog revisions of %s are not read, so the gate's source revision %d of it is not taken in now: %v", branch, a.Revision, err)
return
}
if !take {
return
}
s.logf("source-moved: source revision %d of %s is read through the gate's source and taken in now (start %t)", a.Revision, branch, e.Start)
switch err := s.takeIn(branch, e); {
case errors.Is(err, index.ErrOldForm):
s.logf("source-moved: the gate's source revision %d of %s is refused, the old form: %v", e.Source, branch, err)
case err != nil:
s.logf("source-moved: the gate's source revision %d of %s is not taken in now: %v", e.Source, branch, err)
}
}
@@ -109,10 +221,7 @@ func (s *sourceMoved) heard(branch string, m jetstream.Msg) {
}
s.again.Lock()
defer s.again.Unlock()
behind, err := s.ix.HoldBehind(s.ctx, e)
if !behind && err == nil {
_, err = s.ix.TakeInSource(s.ctx, e)
}
err := s.takeIn(branch, e)
switch {
case errors.Is(err, index.ErrOldForm):
s.logf("source-moved of %s on %s is refused, the old form: %v", branch, m.Subject(), err)
@@ -123,14 +232,25 @@ func (s *sourceMoved) heard(branch string, m jetstream.Msg) {
return
case err != nil:
s.logf("source-moved: source revision %d of %s is held, not taken in: %v", e.Source, branch, err)
case behind:
s.logf("source-moved: source revision %d of %s is held behind the branch's earlier held ones", e.Source, branch)
}
if err := m.Ack(); err != nil {
s.logf("source-moved: source revision %d of %s is taken in or held and its acknowledgement failed: %v", e.Source, branch, err)
}
}
// takeIn takes one source-moved of a branch in, or holds it behind the branch's earlier held ones, so its source revisions
// are taken in in their order; the caller holds s.again. A heard one and one read through the gate's source go this one way.
func (s *sourceMoved) takeIn(branch string, e meshdelivery.SourceMovedEvent) error {
behind, err := s.ix.HoldBehind(s.ctx, e)
if !behind && err == nil {
_, err = s.ix.TakeInSource(s.ctx, e)
}
if behind && err == nil {
s.logf("source-moved: source revision %d of %s is held behind the branch's earlier held ones", e.Source, branch)
}
return err
}
// takeAgain takes every held source revision in again from the state held, each branch in its order, on the indexer's own room
// (H-A-50) and at its start (H-A-114).
func (s *sourceMoved) takeAgain() {
@@ -2,14 +2,20 @@ package main
import (
"context"
"errors"
"fmt"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/internal/index"
)
@@ -88,3 +94,143 @@ func TestAHeldSourceMovedIsSkippedByHand(t *testing.T) {
t.Errorf("held %+v after the skip, want source 5 alone", hs)
}
}
// revisionObjects is the catalog revisions' object bucket in memory.
type revisionObjects map[string][]byte
func (o revisionObjects) Put(_ context.Context, name string, data []byte) error {
o[name] = data
return nil
}
func (o revisionObjects) Get(_ context.Context, name string) ([]byte, error) {
if d, ok := o[name]; ok {
return d, nil
}
return nil, index.ErrNoObject
}
func (o revisionObjects) Delete(_ context.Context, name string) error { delete(o, name); return nil }
func (o revisionObjects) Names(context.Context) ([]string, error) {
var out []string
for n := range o {
out = append(out, n)
}
return out, nil
}
func (o revisionObjects) Fill(context.Context) (int64, int64, error) { return 0, 1 << 30, nil }
// Design 47 H §743, §768, H-A-129: at a clean state no branch's consumer is there to bind; the start reads each watched
// branch's source revision through the gate's verb source and takes it in, and a failed bind at a registration-changed reads
// it again without taking it in twice. A start whose view keeps the revision reads it and takes nothing in.
func TestAStartWithNoConsumerTakesTheGatesSourceRevisionIn(t *testing.T) {
ctx := context.Background()
var taken []string
objects := revisionObjects{}
ix := &index.Indexer{Module: testIndexer, Revisions: objects, Modules: index.MemoryStore(), Graph: index.MemoryStore(),
Held: &memStore{vals: map[string][]byte{}}, Logf: func(string, ...any) {},
SayTakeIn: func(_ context.Context, event, id string, _ any) error {
taken = append(taken, event+" "+id)
return nil
}}
var asked []string
newSources := func() *sourceMoved {
return &sourceMoved{ctx: ctx, ix: ix,
branches: func(context.Context) ([]string, error) { return []string{"main"}, nil },
bind: func(context.Context, string, func(jetstream.Msg)) (func(), error) {
return nil, errors.New("nats: API error: code=404 err_code=10014 description=consumer not found")
},
source: func(_ context.Context, branch string) (meshdelivery.SourceAnswer, error) {
asked = append(asked, branch)
return meshdelivery.SourceAnswer{Branch: branch, Revision: 1, Call: "0123456789abcdef"}, nil
},
logf: func(string, ...any) {}}
}
s := newSources()
s.refresh() // the start
if _, ok := objects[index.BranchObject("main", 1)]; !ok || !reflect.DeepEqual(taken, []string{"take-in.taken " + index.TakenID("main", 1)}) {
t.Fatalf("the start kept %v and said %v; want main.1 taken in from the gate's source", objects, taken)
}
s.refresh() // a registration-changed, the consumer still not there
newSources().refresh() // a start whose view keeps the revision
if !reflect.DeepEqual(asked, []string{"main", "main", "main"}) || len(taken) != 1 {
t.Errorf("asked the gate for %v and said %v; want main asked at each and taken in once", asked, taken)
}
}
// fakeBranchJS is the bus as bindBranch uses it: the source-moved stream's consumers by name, and the consumers made.
type fakeBranchJS struct {
jetstream.JetStream
held map[string]*fakeBranchConsumer
made []jetstream.ConsumerConfig
}
func (f *fakeBranchJS) Consumer(_ context.Context, stream, name string) (jetstream.Consumer, error) {
if c, ok := f.held[stream+"/"+name]; ok {
return c, nil
}
return nil, jetstream.ErrConsumerNotFound
}
func (f *fakeBranchJS) CreateConsumer(_ context.Context, stream string, cfg jetstream.ConsumerConfig) (jetstream.Consumer, error) {
f.made = append(f.made, cfg)
c := &fakeBranchConsumer{}
f.held[stream+"/"+cfg.Durable] = c
return c, nil
}
// fakeBranchConsumer hands one source-moved to the hearing as soon as it is consumed.
type fakeBranchConsumer struct {
jetstream.Consumer
heard int
}
type fakeConsumeContext struct{ jetstream.ConsumeContext }
func (fakeConsumeContext) Stop() {}
func (c *fakeBranchConsumer) Consume(handle jetstream.MessageHandler, _ ...jetstream.PullConsumeOpt) (jetstream.ConsumeContext, error) {
c.heard++
handle(msgOf{f: &fakeMsg{subject: "mesh.seat.mesh-delivery.event.source-moved.main.2"}})
return fakeConsumeContext{}, nil
}
// Design 47 H §743, H-A-134: a branch's durable the bus does not hold the indexer makes, as a module's seat-event consumer is
// made (SeatEventConsumerConfig), named <node>_<indexer>_<branch>, filtered to the branch, one message at a time, and then
// hears on it; one the bus holds is bound, never made again.
func TestABranchDurableNotThereIsMadeAndHeard(t *testing.T) {
cred := process.Account{Node: "n1", Module: "mesh-delivery-indexer"}
for _, tc := range []struct {
name string
held bool
}{{"not there: made", false}, {"there: bound", true}} {
t.Run(tc.name, func(t *testing.T) {
js := &fakeBranchJS{held: map[string]*fakeBranchConsumer{}}
name := "n1_mesh-delivery-indexer_" + keys.EscapeName("main")
var existing *fakeBranchConsumer
if tc.held {
existing = &fakeBranchConsumer{}
js.held[meshdelivery.StreamSourceMoved+"/"+name] = existing
}
var heard []string
stop, err := bindBranch(js, cred, func(string, ...any) {})(context.Background(), "main", func(m jetstream.Msg) { heard = append(heard, m.Subject()) })
if err != nil {
t.Fatal(err)
}
stop()
if len(heard) != 1 {
t.Fatalf("heard %v, want the one source-moved the consumer hands", heard)
}
if tc.held {
if len(js.made) != 0 || existing.heard != 1 {
t.Fatalf("made %v and heard the held one %d times; want it bound, not made again", js.made, existing.heard)
}
return
}
want := cred.SeatEventConsumerConfig([]string{meshdelivery.SubjectSourceMoved(keys.EscapeName("main"), "*")}, false)
want.Durable, want.MaxAckPending = name, 1
want.AckWait, want.MaxDeliver, want.DeliverPolicy = 30*time.Second, 5, jetstream.DeliverNewPolicy
if len(js.made) != 1 || !reflect.DeepEqual(js.made[0], want) {
t.Fatalf("made %+v, want once %+v", js.made, want)
}
})
}
}
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"os"
"slices"
"strconv"
"strings"
"sync"
@@ -16,10 +17,12 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/git"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshdelivery"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/keys"
@@ -185,12 +188,14 @@ func sayMoved(t *testing.T, ctx context.Context, js jetstream.JetStream, branch
type takeIns struct {
taken chan meshcatalog.TakeInTaken
pending chan meshcatalog.TakeInPending
acks chan string // the escaped branch whose source-moved the indexer acknowledged
acks chan string // the escaped branch whose source-moved the indexer acknowledged
runAcks chan struct{} // a run-started the indexer acknowledged
}
func hearTakeIns(t *testing.T, nc *nats.Conn) *takeIns {
t.Helper()
h := &takeIns{taken: make(chan meshcatalog.TakeInTaken, 64), pending: make(chan meshcatalog.TakeInPending, 64), acks: make(chan string, 64)}
h := &takeIns{taken: make(chan meshcatalog.TakeInTaken, 64), pending: make(chan meshcatalog.TakeInPending, 64), acks: make(chan string, 64),
runAcks: make(chan struct{}, 64)}
prefix := "$JS.ACK." + meshdelivery.StreamSourceMoved + "."
acks, err := nc.Subscribe(prefix+"*.>", func(m *nats.Msg) {
consumer, _, _ := strings.Cut(strings.TrimPrefix(m.Subject, prefix), ".")
@@ -202,6 +207,15 @@ func hearTakeIns(t *testing.T, nc *nats.Conn) *takeIns {
t.Fatal(err)
}
t.Cleanup(func() { _ = acks.Unsubscribe() })
runAcks, err := nc.Subscribe("$JS.ACK."+meshpipelinerunner.StreamRunStarted+".n1_mesh-delivery-indexer.>", func(m *nats.Msg) {
if string(m.Data) == "+ACK" || len(m.Data) == 0 {
h.runAcks <- struct{}{}
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = runAcks.Unsubscribe() })
sub, err := nc.Subscribe("mesh.mod."+testIndexer+".event.take-in.*", func(m *nats.Msg) {
switch m.Subject {
case "mesh.mod." + testIndexer + ".event.take-in.taken":
@@ -251,6 +265,16 @@ func (h *takeIns) acked(t *testing.T, branch string) {
}
}
// runAcked waits for the indexer's acknowledgement of a run-started.
func (h *takeIns) runAcked(t *testing.T) {
t.Helper()
select {
case <-h.runAcks:
case <-time.After(waitBound):
t.Fatalf("no run-started acknowledged within %s", waitBound)
}
}
// nextPending is the next take-in.pending said, which must be of that branch and source.
func (h *takeIns) nextPending(t *testing.T, branch string, source int64) meshcatalog.TakeInPending {
t.Helper()
@@ -339,17 +363,22 @@ func sayPipeline(t *testing.T, ctx context.Context, js jetstream.JetStream, ende
return
}
raw, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: pipeline, Branch: branch, Source: source, Revision: source,
Outcome: meshpipelinerunner.PipelineEndedOutcomePassed, Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}, Tier: 1}}})
Outcome: meshpipelinerunner.PipelineEndedOutcomePassed, Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}, Tier: 1,
Version: "0.1." + s}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded(b, s), raw); err != nil {
t.Fatal(err)
}
}
// pinned is whether a pins value names the record.
func pinned(record string) func([]byte) bool {
// pinned is whether a pins value names exactly the records, in any order.
func pinned(records ...string) func([]byte) bool {
return func(v []byte) bool {
var p index.PinValue
return v != nil && json.Unmarshal(v, &p) == nil && len(p.Records) == 1 && p.Records[0] == record
if v == nil || json.Unmarshal(v, &p) != nil {
return false
}
slices.Sort(p.Records)
return slices.Equal(p.Records, slices.Sorted(slices.Values(records)))
}
}
@@ -361,11 +390,17 @@ func absent(v []byte) bool { return v == nil }
//
// - main's first source revision (start true) is taken in with no mover and pins nothing; the next, moving avahi's tree, is one
// revision main.11 carrying avahi as its one mover, pinned pipeline.main.11, said once as take-in.taken; the same source-moved
// given again says the same from the kept movers and keeps no second revision;
// - pipeline-started of main.12 pins it, a restart keeps main.11 while its pipeline runs and removes main.10;
// given again says the same from the kept movers and keeps no second revision; main.10, the branch's previous revision, is
// pinned pipeline.main.11 in the same write as validate's base (H-A-147);
// - pipeline-started of main.12 pins it, a restart keeps main.11 and main.10 while main.11's pipeline runs;
// - a run on request's run-started (run 9, avahi at main 12's commit on a branch no registration watches) writes
// candidates.run-9, one pull of number 0 at that commit with avahi's manifest read through the seat git, and the run's key; across
// the restart, the runner's pipeline-ended of its one member drops both (design 47 G §64, §190; A-160, A-174);
// - with the store full, main.13 is held, said once as take-in.pending, written whole in the state held and acknowledged
// (H-A-114, H-A-118); nothing takes it in on a
// timer: main.11's pipeline-ended frees its pin, the revision goes, the indexer says its room and takes main.13 in on it;
// timer: main.11's pipeline-ended frees both its pins, main.10 goes (main.11 stays, main.12's base and its passed release), the
// indexer says its room and takes main.13 in on it; main.12's passed end and its pipeline-wanted free main.11's release, and
// main.11 goes (H-A-168, H-A-170, H-A-173);
// - beta's take-in whose commit the forge cannot read is held while main's next goes on; the hand skip, with why, is recorded in
// the hand-act log before it is asked, and take-in.taken says it skipped;
// - a branch watched later is bound on registration-changed and its first revision taken in.
@@ -376,8 +411,8 @@ func TestIndexerWhole(t *testing.T) {
)
var forge *forgeCommits
var heard *takeIns
var pins, held, takeinsKV, regs jetstream.KeyValue
js, ctx, _, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
var pins, held, takeinsKV, regs, runs jetstream.KeyValue
js, ctx, candidates, _, restart := startIndexerRun(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
heard = hearTakeIns(t, js.Conn())
forge = serveCommits(t, js.Conn())
for commit, tree := range map[string]string{c10: "t10", c11: "t11", c12: "t12", c13: "t13"} {
@@ -387,7 +422,7 @@ func TestIndexerWhole(t *testing.T) {
for _, b := range []struct {
state string
kv *jetstream.KeyValue
}{{index.HeldState, &held}, {index.TakeInsState, &takeinsKV}} {
}{{index.HeldState, &held}, {index.TakeInsState, &takeinsKV}, {index.RunsState, &runs}} {
if *b.kv, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, b.state), History: 1}); err != nil {
t.Fatal(err)
}
@@ -429,6 +464,7 @@ func TestIndexerWhole(t *testing.T) {
t.Fatalf("main.11 = %+v (%v), want it kept carrying the movers said", rev, err)
}
untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.11"))
untilKey(t, ctx, pins, index.PinKey("main", 10), pinned("pipeline.main.11")) // validate's base, H-A-147
info, err := objects.open(ctx)
if err != nil {
t.Fatal(err)
@@ -445,8 +481,8 @@ func TestIndexerWhole(t *testing.T) {
t.Errorf("main.11 was written again (%v): a source-moved given again keeps no second revision", err)
}
// pipeline-started pins its revision; a restart while main.11's pipeline runs keeps it, and the judge at the start removes
// main.10, which nothing pins and which is no longer current.
// pipeline-started pins its revision; a restart while main.11's pipeline runs keeps it and main.10, its validate base
// (H-A-147), which the take-in pinned for it.
sayMoved(t, ctx, js, "main", 12, c12, false)
heard.next(t, "main", 12)
if err := pins.Delete(ctx, index.PinKey("main", 12)); err != nil { // the take-in's own pin, so pipeline-started alone writes it below
@@ -454,11 +490,44 @@ func TestIndexerWhole(t *testing.T) {
}
sayPipeline(t, ctx, js, false, "main", 12)
untilKey(t, ctx, pins, index.PinKey("main", 12), pinned("pipeline.main.12"))
restart()
untilRevisionGone(t, ctx, js, "main.10")
if _, _, err := index.ReadRevision(ctx, objects, "main.11"); err != nil {
t.Fatalf("main.11 is not kept across a restart while its pipeline runs: %v", err)
// A run on request: its candidate and its key written on run-started, from avahi's files at the run's commit (A-160, A-174).
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil {
t.Fatal(err)
}
started, _ := json.Marshal(meshpipelinerunner.RunStarted{V: 1, Run: "9", By: "lead@n1", Members: []meshpipelinerunner.Member{{Module: "avahi", Commit: c12, Branch: "feature/1966-x"}}})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectRunStarted("avahi", "feature-1966-x", c12), started); err != nil {
t.Fatal(err)
}
runKey, _ := meshcatalogclient.CandidatesKey("run-9")
untilKey(t, ctx, candidates, runKey, func(v []byte) bool {
c, err := index.DecodeCandidate(v)
return v != nil && err == nil && len(c.Pulls) == 1 && c.Pulls[0].Number == 0 && c.Pulls[0].Head == c12 &&
c.Pulls[0].Fps["avahi"].Manifest == meshcatalogclient.ManifestDigest(manifest)
})
untilKey(t, ctx, runs, "run-9", func(v []byte) bool { return v != nil })
// The candidate is written before its run-started is acknowledged: the restart waits for that too, so the stop leaves it
// unacknowledged for no redelivery to write the candidate again after its run ended.
heard.runAcked(t)
// The restart waits for the take-ins' ends, their source-moved acknowledged (design 47 H §743: main 10, 11, 11 given
// again and 12), so the stop cancels none and leaves none unacknowledged ahead of main 13.
for range 4 {
heard.acked(t, "main")
}
restart()
for _, name := range []string{"main.10", "main.11"} {
if _, _, err := index.ReadRevision(ctx, objects, name); err != nil {
t.Fatalf("%s is not kept across a restart while main.11's pipeline runs: %v", name, err)
}
}
// The run's member's pipeline ends across the restart: its candidate and its key go (A-174).
ended, _ := json.Marshal(meshpipelinerunner.PipelineEnded{V: 1, Pipeline: "feature-1966-x.0", Branch: "feature/1966-x", Run: "9",
Movers: []meshpipelinerunner.MoverEnded{{Module: "avahi", Steps: []meshpipelinerunner.StepResult{}}}, Outcome: meshpipelinerunner.PipelineEndedOutcomePassed})
if _, err := js.Publish(ctx, meshpipelinerunner.SubjectPipelineEnded("feature-1966-x", "0"), ended); err != nil {
t.Fatal(err)
}
untilKey(t, ctx, candidates, runKey, absent)
untilKey(t, ctx, runs, "run-9", absent)
// The store full: held, said once, in the state held; taken in on the indexer's own room once a freed pin makes it.
used, _, err := objects.Fill(ctx)
@@ -475,7 +544,9 @@ func TestIndexerWhole(t *testing.T) {
return v != nil && json.Unmarshal(v, &hs) == nil && len(hs) == 1 && hs[0].Source == 13 && hs[0].Why != "" && !hs[0].Since.IsZero() &&
hs[0].Call != "" && len(hs[0].Heads) == 1 && hs[0].Heads[0].Commit == c13
})
// H-A-114: the held source-moved is acknowledged on the bus, kept only in the state held.
// H-A-114: the held source-moved is acknowledged on the bus, kept only in the state held; its acknowledgement follows the
// write, so it is waited for before the consumer is read.
heard.acked(t, "main")
if cons, err := testAccount.NamedEventConsumer(ctx, js, meshdelivery.StreamSourceMoved, keys.EscapeName("main")); err != nil {
t.Fatal(err)
} else if info, err := cons.Info(ctx); err != nil || info.NumAckPending != 0 {
@@ -487,7 +558,22 @@ func TestIndexerWhole(t *testing.T) {
t.Errorf("main 13 taken in on room said %+v, want avahi at %s", got, c13)
}
untilKey(t, ctx, held, keys.EscapeName("main"), absent)
// main.12's validate base stays (H-A-147), and main.11's passed pipeline's release pins it (H-A-168, H-A-170).
untilKey(t, ctx, pins, index.PinKey("main", 11), pinned("pipeline.main.12", index.ReleaseRecord("main", "avahi")))
untilRevisionGone(t, ctx, js, "main.10") // the end freed both of main.11's pins
// main.12's passed pipeline releases avahi again; its pipeline-wanted, the seat mesh-wants' (its wants written to the end),
// frees main.11's older release, and main.11 goes (H-A-168, H-A-170, H-A-173).
sayPipeline(t, ctx, js, true, "main", 12)
untilKey(t, ctx, pins, index.PinKey("main", 12), func(v []byte) bool {
var p index.PinValue
return v != nil && json.Unmarshal(v, &p) == nil && slices.Contains(p.Records, index.ReleaseRecord("main", "avahi"))
})
wantedRaw, _ := json.Marshal(meshwants.PipelineWanted{V: 1, Pipeline: "main.12", Branch: "main", Source: 12, Outcome: meshwants.PipelineWantedOutcomeRunning})
if _, err := js.Publish(ctx, meshwants.SubjectPipelineWanted("main", "12"), wantedRaw); err != nil {
t.Fatal(err)
}
untilKey(t, ctx, pins, index.PinKey("main", 11), absent)
untilRevisionGone(t, ctx, js, "main.11")
if n := len(heard.pending); n != 0 {
t.Errorf("take-in.pending said %d more times for one held source revision, want once", n)
}
@@ -135,14 +135,32 @@ func NamedTree(manifest string, paths map[string]string) (string, error) {
return NamedTreePrefix + hex.EncodeToString(sum[:]), nil
}
// MeshPackagesOf are the mesh Go modules a go.sum names (the module path of each line under
// trunk.ForgeModulePrefix), sorted and each once; nil for none. They are what a module with no reported import set
// takes the version line of (ADR 0593 §2).
func MeshPackagesOf(goSum []byte) []string {
// MeshRequiresOf are the mesh Go modules a go.mod requires (the module path of each `require` line under
// trunk.ForgeModulePrefix, alone or in a block, at any version, the zero version and `// indirect` included), sorted and
// each once; nil for none. They are a module's mesh packages from the take-in, with no build needed (design 47 H-A-157):
// a go.sum names none required at the zero version (ADR 0573 §2), so it is never read for them.
func MeshRequiresOf(goMod []byte) []string {
seen := map[string]bool{}
for _, line := range strings.Split(string(goSum), "\n") {
inBlock := false
for _, line := range strings.Split(string(goMod), "\n") {
line = strings.TrimSpace(line)
if i := strings.Index(line, "//"); i >= 0 {
line = strings.TrimSpace(line[:i])
}
switch {
case line == "require (":
inBlock = true
continue
case inBlock && line == ")":
inBlock = false
continue
case strings.HasPrefix(line, "require "):
line = strings.TrimSpace(strings.TrimPrefix(line, "require "))
case !inBlock:
continue
}
f := strings.Fields(line)
if len(f) < 2 || !strings.HasPrefix(f[0], trunk.ForgeModulePrefix) {
if len(f) != 2 || !strings.HasPrefix(f[0], trunk.ForgeModulePrefix) {
continue
}
if _, _, ok := trunk.PackageRepository(f[0]); ok {
@@ -405,41 +423,49 @@ func sortedKeys[V any](m map[string]V) []string { return slices.Sorted(maps.Keys
// commit or no directory, is in missing, sorted; versions is nil when packages is empty.
func HeldVersionsOf(packages []string, modules map[string]meshcatalog.ModulesByModuleValue) (versions sdkstep.PackageVersions, missing []string) {
for _, pkg := range packages {
repository, dir, ok := trunk.PackageRepository(pkg)
commit, depth := "", -1
if ok {
for _, key := range slices.Sorted(maps.Keys(modules)) {
m := modules[key]
if !trunk.SameRepository(m.Repository, repository) || m.Commit == "" {
continue
}
at := strings.Trim(m.Path, "/")
switch {
case at == "":
continue // a value that names no directory says nothing of where the package lies
case at == meshcatalogclient.RootPath:
at = ""
case dir != at && !strings.HasPrefix(dir, at+"/"):
continue
}
if d := len(at); d > depth {
commit, depth = m.Commit, d
}
}
}
if commit == "" {
key := HolderOf(pkg, modules)
if key == "" {
missing = append(missing, pkg)
continue
}
if versions == nil {
versions = sdkstep.PackageVersions{}
}
versions[pkg] = commit
versions[pkg] = modules[key].Commit
}
slices.Sort(missing)
return versions, missing
}
// HolderOf is the key of the held module of the package's repository whose directory holds the package's directory (the
// deepest, a module at the repository's root holding every directory) and that names a commit; "" for none.
func HolderOf(pkg string, modules map[string]meshcatalog.ModulesByModuleValue) string {
repository, dir, ok := trunk.PackageRepository(pkg)
if !ok {
return ""
}
holder, depth := "", -1
for _, key := range slices.Sorted(maps.Keys(modules)) {
m := modules[key]
if !trunk.SameRepository(m.Repository, repository) || m.Commit == "" {
continue
}
at := strings.Trim(m.Path, "/")
switch {
case at == "":
continue // a value that names no directory says nothing of where the package lies
case at == meshcatalogclient.RootPath:
at = ""
case dir != at && !strings.HasPrefix(dir, at+"/"):
continue
}
if d := len(at); d > depth {
holder, depth = key, d
}
}
return holder
}
// digestOf is the content digest of the package at import path pkg in a published module, and whether it holds it
// (its packages are sorted by import path).
func digestOf(p meshcatalog.PackagesByModuleValue, pkg string) (string, bool) {
@@ -455,3 +481,21 @@ func digestOf(p meshcatalog.PackagesByModuleValue, pkg string) (string, bool) {
// NamedTreePrefix opens a module's tree when it is computed over the paths its artifacts name, not the directory's own
// git tree (novox/hq ADR 0499 clause 2): the indexer alone computes it (NamedTree).
const NamedTreePrefix = "named:"
// WithPins is the fingerprint fp with the pins a build of it is made against (design 47 H-A-167): fp itself when no
// package is pinned, else `src2:` and the hex sha256 of fp and, sorted, the lines "pin <package> <version>" of each pin,
// joined by a newline. A package kept with no version (none released) is no pin.
func WithPins(fp string, pins sdkstep.PackageVersions) string {
var lines []string
for p, v := range pins {
if v != "" {
lines = append(lines, "pin "+p+" "+v)
}
}
if fp == "" || len(lines) == 0 {
return fp
}
sort.Strings(lines)
sum := sha256.Sum256([]byte(fp + "\n" + strings.Join(lines, "\n")))
return meshcatalogclient.SourcePrefix2 + hex.EncodeToString(sum[:])
}
@@ -31,6 +31,20 @@ func versionLineTakesASiblingDirectoryByItsContent(t *testing.T) {
if got := ReplacedSiblings(goMod, repo); len(got) != 1 || got[0] != buildrun {
t.Fatalf("siblings %v, want buildrun alone (mesh-sdk is another repository, replaced by a version)", got)
}
// Its mesh packages are every mesh package its go.mod requires, at any version, the zero version included (design 47
// H-A-157); a replace line or a module outside the forge is none.
for _, c := range []struct {
goMod string
want []string
}{
{string(goMod), []string{buildrun, sdk}},
{"module x\n\nrequire " + sdk + " v0.0.0-00010101000000-000000000000 // indirect\nrequire github.com/nats-io/nats.go v1.37.0\n", []string{sdk}},
{"module x\n\nreplace " + sdk + " => ../sdk\n", nil},
} {
if got := MeshRequiresOf([]byte(c.goMod)); strings.Join(got, ",") != strings.Join(c.want, ",") {
t.Errorf("mesh packages of %q: %v, want %v", c.goMod, got, c.want)
}
}
published := func(digest string) map[string]meshcatalog.PackagesByModuleValue {
return map[string]meshcatalog.PackagesByModuleValue{buildrun: {Module: buildrun, Repository: repo, Requires: "r1",
Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: buildrun, Digest: digest}}}}
@@ -268,7 +268,7 @@ func (ix *Indexer) viewEntries(ctx context.Context) (map[string]meshcatalog.Cata
// KeepBranchEntries keeps entries as the catalog revision of a watched branch taken in from its source revision src (design
// 47 H §743, H-A-32): <branch>.<src>, the trunk's included, numbered by the source, and says catalog-revision naming the
// branch. movers are the take-in's: kept on the revision (H-A-92), and the revision pinned for the take-in's pipeline,
// pipeline.<branch>.<src>, in the same write when there is one (A-449a). A revision already kept under that name is not written
// pipeline.<branch>.<src>, in the same write when there is one (A-449a), with the branch's previous revision (H-A-147). A revision already kept under that name is not written
// again (read before write, under the branch's one consumer); ErrObjectsFull is a store too full to keep it.
func (ix *Indexer) KeepBranchEntries(ctx context.Context, branch string, src int64, entries map[string]meshcatalog.CatalogEntry, movers []meshcatalog.TakeInEntry) error {
return ix.keepBranchEntries(ctx, branch, src, entries, movers)
@@ -305,6 +305,12 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
return err
}
rev.Movers = movers
// The base of the validate step's case before: the newest revision kept before this write, which the take-in pins for its
// pipeline below, named on the revision in this same write (H-A-147, H-A-151, H-A-154); none on the branch's first take-in.
pinsBase := len(movers) > 0 && ix.Pins != nil && newest.Revision > 0 && newest.Revision != src
if pinsBase {
rev.Base = newest.Revision
}
changed := changedEntries(newest.Entries, rev.Entries)
used, capacity, fillErr := ix.Revisions.Fill(ctx)
if fillErr == nil && capacity > 0 && used+newBytes(ctx, ix.Revisions, rev, objects) > capacity {
@@ -330,6 +336,16 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
ix.logf("pins: the revision %s is not pinned for %s: %v", BranchObject(branch, src), record, err)
return fmt.Errorf("the pin of %s: %w", BranchObject(branch, src), err)
}
// The branch's previous revision, the newest kept before this write, pinned for the same pipeline: it is the base of
// the validate step's case before (H-A-108 (3), H-A-147), kept as long as the pipeline that needs it.
if pinsBase {
if err := ix.pin(ctx, PinKey(branch, rev.Base), record); err != nil {
ix.logf("pins: the previous revision %s is not pinned for %s: %v", BranchObject(branch, newest.Revision), record, err)
return fmt.Errorf("the pin of %s: %w", BranchObject(branch, newest.Revision), err)
}
} else {
ix.logf("pins: %s has no revision before it on its branch: none is pinned for %s as its validate base", BranchObject(branch, src), record)
}
}
// The revision before this one in its branch ceases to be current (§579): it goes unless a record pins it.
if removed, err := ix.judgePinsLocked(ctx); err != nil {
@@ -337,6 +353,11 @@ func (ix *Indexer) keepBranchEntries(ctx context.Context, branch string, src int
} else if len(removed) > 0 {
ix.logf("pins: %d revisions removed (%s), no record pins them and none is current", len(removed), strings.Join(removed, ", "))
}
if dropped, err := ix.dropEarlierFormsLocked(ctx); err != nil {
ix.logf("catalog graph: the revisions of the earlier forms (trunk.*, branch.*) are not dropped: %v", err)
} else if len(dropped) > 0 {
ix.logf("catalog graph: %d revisions of the earlier forms dropped (%s), no record pins them; each branch's newest revision is its current one", len(dropped), strings.Join(dropped, ", "))
}
if _, err := ix.removeUnlisted(ctx); err != nil {
ix.logf("catalog graph: the entries nothing lists or names any more are not removed: %v", err) // at the first revision after none does (§675)
}
@@ -17,6 +17,11 @@ import (
type form struct {
fp, none string
versions sdkstep.PackageVersions
// stands: fp is the held fingerprint of the build that reported the imports, made with the pins built (its held
// packages) from the import form raw; pinnedOf keeps it while the pins are those (H-A-172).
stands bool
raw string
built map[string]string
}
// formOf is the module's fingerprint from its kept inputs (bases already its bases' fingerprints) and toolchains, in the
@@ -24,15 +29,16 @@ type form struct {
//
// - **The import form**, for a module whose last trunk build reported its imports (kept.Imports) when every import is
// published (the packages state): the content digest of each imported package and the requirements' digest of each
// mesh Go module holding one. Its versions are each holding module's published commit (another repository's only).
// mesh Go module holding one. Its versions are each holding module's published commit (another repository's only); an
// import set naming none of another repository while the go.mod requires one keeps the version line's (refined).
// - **A change of form never rebuilds by itself** (§4): kept.ImportForm is the import-form fingerprint of the inputs
// the build that reported the imports was built from. While the import form computes to it, nothing the build was
// built from changed: the module keeps the fingerprint it holds (prev), in whatever form, and its versions.
// Otherwise the import form is written, and ImportForm moves with it.
// - **The version line** otherwise (no import set, or an import none publishes): the version of each mesh Go module of
// another repository its go.sum names, read as the indexer always read it (fingerprint.HeldVersionsOf), and the
// content of each mesh Go module of its own repository its go.mod replaces by a directory (fingerprint.VersionLine),
// so a change of that directory alone moves it (review finding 2). Its versions are the other repositories' alone.
// another repository its go.mod requires, at any version (design 47 H-A-157), read as the indexer always read it
// (fingerprint.HeldVersionsOf), and the content of each mesh Go module of its own repository its go.mod replaces by a
// directory (fingerprint.VersionLine), so a change of that directory alone moves it (review finding 2). Its versions are the other repositories' alone.
// A package of another repository none of them publishes takes the commit the gate's source names for its repository
// (heads, design 47 G A-365).
// - **An accepted series** (DLV-I6): published and pins come from acceptedSeries, so a mesh Go module of another
@@ -51,10 +57,12 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
case fp == "":
return form{none: none}
case kept.ImportForm == fp && prev.Fp2 != "":
return form{fp: prev.Fp2, versions: prev.Packages} // the build that reported the imports stands
// The build that reported the imports stands, and so do its versions.
return form{fp: prev.Fp2, versions: refined(sdkstep.PackageVersions(prev.Packages), r.Versions, module, repository, kept, held, pins, heads),
stands: true, raw: fp, built: prev.Packages}
}
kept.ImportForm = fp
return form{fp: fp, versions: r.Versions}
return form{fp: fp, versions: refined(r.Versions, nil, module, repository, kept, held, pins, heads)}
}
if logf != nil {
logf("%s: no published package holds %s, so its fingerprint takes the version line", module, strings.Join(r.Missing, ", "))
@@ -63,6 +71,34 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
// The fingerprint written here is the version line's, not the import form's: no import form is held for the inputs it was
// computed from, so a later pass whose import form computes the same does not take this one for the build's (issue 1966).
kept.ImportForm = ""
versions := lineVersions(module, repository, kept, held, pins, heads, logf)
in.Packages = fingerprint.VersionLine(repository, kept.Packages, versions, published)
fp, none := fingerprint.Of(in)
if fp == "" {
return form{none: none}
}
return form{fp: fp, versions: versions}
}
// refined are an import form's versions: the first of sets that names a package, else, while the module's go.mod
// requires a mesh package of another repository (kept.Packages), the version line's, so a build's reported imports
// refine the go.mod's set and never empty it (design 47 H-A-157); nil when nothing names one.
func refined(versions, then sdkstep.PackageVersions, module, repository string, kept *meshcatalog.ModulesByModuleValueInputs2, held map[string]meshcatalog.ModulesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string) sdkstep.PackageVersions {
switch {
case len(versions) > 0:
return versions
case len(then) > 0:
return then
case len(fingerprint.OtherPackages(kept.Packages, repository)) > 0:
return lineVersions(module, repository, kept, held, pins, heads, nil)
}
return versions
}
// lineVersions are the version line's versions of the mesh Go modules of other repositories the module's go.mod requires
// (kept.Packages): each as the indexer has always read it, an accepted series' pin over it, else the commit the gate's
// source names for its repository.
func lineVersions(module, repository string, kept *meshcatalog.ModulesByModuleValueInputs2, held map[string]meshcatalog.ModulesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string, logf func(string, ...any)) sdkstep.PackageVersions {
others := fingerprint.OtherPackages(kept.Packages, repository)
versions, unpublished := fingerprint.HeldVersionsOf(others, held)
for p, v := range pins { // an accepted series' newest published version (acceptedSeries, DLV-I6)
@@ -90,12 +126,7 @@ func formOf(module, repository string, kept *meshcatalog.ModulesByModuleValueInp
if len(unpublished) > 0 && logf != nil {
logf("%s: no module the indexer holds publishes %s, so its fingerprint takes no version of it", module, strings.Join(unpublished, ", "))
}
in.Packages = fingerprint.VersionLine(repository, kept.Packages, versions, published)
fp, none := fingerprint.Of(in)
if fp == "" {
return form{none: none}
}
return form{fp: fp, versions: versions}
return versions
}
// inputsFrom is the fingerprint's inputs of module m from its kept inputs and the given toolchains, no package of
@@ -118,10 +118,39 @@ func TestImportsReportedMoveTheFingerprintOnlyWithAnImportedPackage(t *testing.T
t.Fatal(err)
}
moved, _ := moduleOf(t, mods, "avahi")
if moved.Fp2 == before.Fp2 || moved.Fp2 != moved.Inputs2.ImportForm || moved.Packages[meshSDK] != c3 {
t.Fatalf("a change of an imported package: fp2 %q (was %q), import form %q, versions %v; want the import form at c3",
if pin, kept := moved.Packages[meshSDK]; moved.Fp2 == before.Fp2 || moved.Fp2 != moved.Inputs2.ImportForm || !kept || pin != "" {
t.Fatalf("a change of an imported package: fp2 %q (was %q), import form %q, versions %v; want the import form, mesh-sdk kept unpinned (none released, H-A-161)",
moved.Fp2, before.Fp2, moved.Inputs2.ImportForm, moved.Packages)
}
// mesh-sdk's module releases 0.1.97 while the registry has published nothing of its Go package: the module's release
// number is never the package's pin (H-A-183), so avahi stays unpinned and where it was.
ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.97"), []byte(`{}`))
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
if m, _ := moduleOf(t, mods, "avahi"); m.Fp2 != moved.Fp2 || m.Packages[meshSDK] != "" {
t.Fatalf("after mesh-sdk's module release alone: fp2 %q (was %q), versions %v; want it unpinned where it was", m.Fp2, moved.Fp2, m.Packages)
}
// The registry publishes its Go package at 0.1.20, another version than the module's release: the import form's
// fingerprint takes that pin and moves, once (H-A-172, H-A-183).
if _, err := ix.HearVersion(ctx, HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.1.20", Commit: c3, GoModule: meshSDK, Published: true}); err != nil {
t.Fatal(err)
}
for range 2 {
if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
}
}
if m, _ := moduleOf(t, mods, "avahi"); m.Fp2 == moved.Fp2 || m.Fp2 == "" || m.Packages[meshSDK] != "v0.1.20" || m.Inputs2.ImportForm != moved.Inputs2.ImportForm {
t.Fatalf("after mesh-sdk's release: fp2 %q (was %q), versions %v, import form %q; want a new fingerprint pinned to v0.1.20, the import form kept",
m.Fp2, moved.Fp2, m.Packages, m.Inputs2.ImportForm)
} else if err := ix.Recompute(ctx); err != nil {
t.Fatal(err)
} else if again, _ := moduleOf(t, mods, "avahi"); again.Fp2 != m.Fp2 {
t.Fatalf("a pass with nothing moved moved the pinned fingerprint %q to %q", m.Fp2, again.Fp2)
}
}
// A build of a fingerprint the module no longer holds says nothing of its inputs now: its imports are not kept.
@@ -256,8 +285,8 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
}
}
before := step("trunk at c1", trunk(c1))
if before.Packages[meshSDK] != c1 {
t.Fatalf("with no series published, versions %v, want the commit form c1", before.Packages)
if pin, kept := before.Packages[meshSDK]; !kept || pin != "" {
t.Fatalf("with no series published, versions %v, want mesh-sdk kept with no pin, never the commit (H-A-161)", before.Packages)
}
step("the build reports its imports", func() error {
return ix.ApplyBuildEnd(ctx, buildEnd(t, "avahi", before.Fp2, meshSDK, meshSDK+"/announce"))
@@ -266,12 +295,19 @@ func anAcceptedSeriesPinsItsNewest(t *testing.T) {
if pinned.Fp2 == before.Fp2 || pinned.Packages[meshSDK] != "v0.1.15" {
t.Fatalf("after 0.1.15: fp2 %q (was %q), versions %v; want moved by c3's announce, pinned to v0.1.15", pinned.Fp2, before.Fp2, pinned.Packages)
}
// A publish in a series avahi does not accept pins nothing and moves nothing (H-A-184).
if m := step("0.2.0 published from c1", hear("0.2.0", c1)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a publish of a series avahi does not accept moved it: %q %v", m.Fp2, m.Packages)
}
if m := step("trunk at c2", trunk(c2)); m.Fp2 != pinned.Fp2 || m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a trunk merge moved avahi off its accepted series' newest: %q %v", m.Fp2, m.Packages)
}
// mesh-sdk's module releases 0.3.0: a module's release number is never a package's pin (H-A-183).
ix.Versions = newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.3.0"), []byte(`{}`))
if m := step("0.3.0 released", func() error { return nil }); m.Packages[meshSDK] != "v0.1.15" {
t.Errorf("a module's release pinned its package: %v", m.Packages)
}
}
// Issue 1966 (the indexer started while the bus's storage was down): a read of the packages state that failed, or that
@@ -370,22 +406,36 @@ func anUnreadPackagesStateWritesNoFingerprintAndFingerprintsAreComputedAgain(t *
head := strings.Repeat("c", 40)
contracts := "git.novox.be/novox/mesh-contracts/go/seats/findings"
held := map[string]meshcatalog.ModulesByModuleValue{"other": {Repository: "novox/mesh-sdk", Commit: strings.Repeat("d", 40), Path: "go"}}
// A build's imports of its own repository alone (design 47 H-A-157): they refine the go.mod's set and never empty it.
own := "git.novox.be/novox/mesh-catalog/modules/avahi/lib"
published := map[string]meshcatalog.PackagesByModuleValue{own: {Module: own, Repository: "novox/mesh-catalog", Commit: head, Requires: "r1",
Packages: []meshcatalog.PackagesByModuleValuePackagesItem{{Package: own, Digest: "d1"}}}}
for _, c := range []struct {
name string
heads map[string]string
held map[string]meshcatalog.ModulesByModuleValue
want string
name string
heads map[string]string
held map[string]meshcatalog.ModulesByModuleValue
imports []string
want string
}{
{"the source names its repository", map[string]string{"novox/mesh-contracts": head}, held, head},
{"the source names another repository only", map[string]string{"novox/mesh-sdk": head}, held, ""},
{"the source names its repository", map[string]string{"novox/mesh-contracts": head}, held, nil, head},
{"the source names another repository only", map[string]string{"novox/mesh-sdk": head}, held, nil, ""},
{"a module the indexer holds publishes it", map[string]string{"novox/mesh-contracts": head},
map[string]meshcatalog.ModulesByModuleValue{"c": {Repository: "novox/mesh-contracts", Commit: strings.Repeat("e", 40), Path: "go/seats/findings"}}, strings.Repeat("e", 40)},
map[string]meshcatalog.ModulesByModuleValue{"c": {Repository: "novox/mesh-contracts", Commit: strings.Repeat("e", 40), Path: "go/seats/findings"}}, nil, strings.Repeat("e", 40)},
{"a build's imports naming none of another repository keep the go.mod's", map[string]string{"novox/mesh-contracts": head}, held, []string{own}, head},
} {
kept := meshcatalog.ModulesByModuleValueInputs2{Tree: "t", Packages: []string{contracts}}
f := formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{}, c.held, nil, nil, c.heads, nil)
kept := meshcatalog.ModulesByModuleValueInputs2{Tree: "t", Packages: []string{contracts}, Imports: c.imports}
f := formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{}, c.held, published, nil, c.heads, nil)
if got := f.versions[contracts]; got != c.want {
t.Errorf("%s: version of %s = %q, want %q", c.name, contracts, got, c.want)
}
if c.imports == nil {
continue
}
// The build that reported them stands, its held versions empty (a version line read from go.sum).
f = formOf("avahi", "novox/mesh-catalog", &kept, nil, meshcatalog.ModulesByModuleValue{Fp2: "src2:" + strings.Repeat("b", 64)}, c.held, published, nil, c.heads, nil)
if got := f.versions[contracts]; got != c.want || f.fp != "src2:"+strings.Repeat("b", 64) {
t.Errorf("%s, the build standing: version of %s = %q (fp %q), want %q", c.name, contracts, got, f.fp, c.want)
}
}
})
t.Run("formOf clears the import form when it writes the version line", func(t *testing.T) {
@@ -108,7 +108,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f whole, drop bool, moves ma
if next.Manifest == "" {
next.None2 = "the manifest cannot be read: it cannot be kept as a value"
} else {
packages := ix.goSumPackages(ctx, f, dir, cur, has)
packages := ix.goPackages(ctx, f, dir, cur, has)
in := inputsOfNamed(name, raw, f.Trees[dir], f.Locks[dir], objects[dir], packages, ix.baseFP(ctx))
next.Inputs2, next.None2, next.Paths = in.Kept, in.None2, in.ArtifactPaths
if in.Kept != nil && has && cur.Inputs2 != nil {
@@ -365,79 +365,72 @@ func unreadInputs(tree string, locks, paths map[string]string, packages []string
return in
}
// goSumPackages are the version line's packages of the merge's directory dir (ADR 0573 §2, ADR 0593 §2): the mesh Go
// modules of other repositories its go.sum names, and those of its own repository its go.mod replaces by a directory
// (siblingPackages), which no go.sum line names. A directory with no go.sum is no Go module and has none.
func (ix *Indexer) goSumPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string {
others := ix.otherPackages(ctx, f, dir, cur, has)
// goPackages are the version line's packages of the merge's directory dir (ADR 0593 §2; design 47 H-A-157): the mesh Go
// modules of other repositories its go.mod requires, at any version, and those of its own repository its go.mod replaces
// by a directory (siblingPackages, H-A-136's sibling rule). A directory with no go.sum is no Go module and has none.
func (ix *Indexer) goPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string {
if f.Locks[dir]["go.sum"] == "" && len(f.Packages[dir]) == 0 {
return others
return nil
}
ix.requirementsOf(ctx, f, dir, cur, has)
goMod, read := ix.goModOf(ctx, f, dir)
others := fingerprint.OtherPackages(f.Packages[dir], f.Repository) // carried: what the go.mod requires, read with the tree
switch {
case len(others) > 0:
case read:
others = fingerprint.OtherPackages(fingerprint.MeshRequiresOf(goMod), f.Repository)
case has && cur.Inputs2 != nil:
others = fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
var siblings []string
if read {
siblings = fingerprint.ReplacedSiblings(goMod, f.Repository)
} else if has && cur.Inputs2 != nil {
for _, p := range cur.Inputs2.Packages {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, f.Repository) {
siblings = append(siblings, p)
}
}
}
siblings := ix.siblingPackages(ctx, f, dir, cur, has)
if len(siblings) == 0 {
return others
}
return slices.Compact(slices.Sorted(slices.Values(append(slices.Clone(others), siblings...))))
}
// siblingPackages are the mesh Go modules of the merge's own repository the go.mod of dir replaces by a directory, read
// from the forge (GoMod); where it cannot be read, the held module's own-repository packages stand, said.
func (ix *Indexer) siblingPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string {
held := func() []string {
if !has || cur.Inputs2 == nil {
return nil
}
var out []string
for _, p := range cur.Inputs2.Packages {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, f.Repository) {
out = append(out, p)
}
}
return out
}
// goModOf is the go.mod of the merge's directory dir, read from the forge (GoMod), and whether it was read; where it
// cannot be, the held module's packages stand, said.
func (ix *Indexer) goModOf(ctx context.Context, f whole, dir string) ([]byte, bool) {
if ix.GoMod == nil {
return held()
return nil, false
}
raw, err := ix.GoMod(ctx, f.Repository, f.Merge, dir)
if err != nil {
ix.logf("%s at %s: the go.mod of %q cannot be read, so its module keeps the directory replaces it held: %v", f.Repository, f.Merge, dir, err)
return held()
ix.logf("%s at %s: the go.mod of %q cannot be read, so its module keeps the packages it held: %v", f.Repository, f.Merge, dir, err)
return nil, false
}
return fingerprint.ReplacedSiblings(raw, f.Repository)
return raw, true
}
// otherPackages are the mesh Go modules of other repositories the go.sum of the merge's directory dir names: those the
// message carries (an older bridge), else the held module's while its go.sum's hash is unchanged, else the go.sum read
// from the forge (GoSum).
func (ix *Indexer) otherPackages(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) []string {
if carried := f.Packages[dir]; len(carried) > 0 {
return fingerprint.OtherPackages(carried, f.Repository) // its own repository's come with its tree
}
// requirementsOf keeps what the go.sum of the merge's directory dir requires in the requirements state
// (keepRequirements): read from the forge (GoSum) when its hash moved, or this process has not kept it yet, whatever
// packages the reader carried. It gives no package: those are the go.mod's (H-A-157).
func (ix *Indexer) requirementsOf(ctx context.Context, f whole, dir string, cur meshcatalog.ModulesByModuleValue, has bool) {
hash := f.Locks[dir]["go.sum"]
if hash == "" {
return nil
if hash == "" || ix.GoSum == nil {
return
}
if has && cur.Inputs2 != nil && trunk.SameRepository(cur.Repository, f.Repository) && ix.requirementsKept(f.Repository, dir) {
for _, h := range cur.Inputs2.Locks {
if h == hash {
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
return
}
}
}
if ix.GoSum == nil {
if has && cur.Inputs2 != nil {
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
return nil
}
raw, err := ix.GoSum(ctx, f.Repository, f.Merge, dir)
if err != nil {
ix.logf("%s at %s: the go.sum of %q cannot be read, so its module keeps the packages it held: %v", f.Repository, f.Merge, dir, err)
if has && cur.Inputs2 != nil {
return fingerprint.OtherPackages(cur.Inputs2.Packages, f.Repository)
}
return nil
ix.logf("%s at %s: the go.sum of %q cannot be read, so what it requires is not kept: %v", f.Repository, f.Merge, dir, err)
return
}
ix.keepRequirements(ctx, f.Repository, dir, raw)
return fingerprint.OtherPackages(fingerprint.MeshPackagesOf(raw), f.Repository)
}
@@ -401,8 +401,36 @@ func TestARootModuleHasTheDirectoryDot(t *testing.T) { // 1615/AC3: empty means
}
func TestAMergeAddingAModuleDirectoryWritesItsValueWithNoBuildRecordAnywhere(t *testing.T) { // I1: issue 1851, ADR 0501 clause 6
ix, mods, _ := manifestsIndexer(t) // no build record of any module
avahi := catalogueManifest(t, "avahi")
// Its go.mod requires mesh-sdk at the zero version (ADR 0573 §2): the go.mod's require lines are its mesh packages, with
// no build, carried by the reader or read through GoMod (design 47 H-A-157). Its go.sum is still read for the
// requirements state, carried packages or not, so an older version it names is kept there (and cleared when gone).
for _, carried := range [][]string{nil, {meshSDK}} {
ix, mods, _ := manifestsIndexer(t)
ix.Requirements = newMem()
ix.GoMod = func(context.Context, string, string, string) ([]byte, error) {
return []byte("module avahi\n\nrequire " + meshSDK + " v0.0.0-00010101000000-000000000000\n"), nil
}
ix.GoSum = func(context.Context, string, string, string) ([]byte, error) {
return []byte(meshSDK + " v0.1.14 h1:x=\n"), nil
}
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi},
map[string]map[string]string{"modules/avahi": {"go.sum": avahiGoSum}})
if carried != nil {
f.Packages = map[string][]string{"modules/avahi": carried}
}
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "avahi")
if m.Inputs2 == nil || !slices.Equal(m.Inputs2.Packages, []string{meshSDK}) {
t.Errorf("carried %v: mesh packages %+v, want mesh-sdk, which its go.mod requires at the zero version", carried, m.Inputs2)
}
if _, kept := ix.Requirements.(*memStore).vals[meshSDK]; !kept {
t.Errorf("carried %v: the requirements state holds no %s, which the go.sum names", carried, meshSDK)
}
}
ix, mods, _ := manifestsIndexer(t) // no build record of any module
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi},
map[string]map[string]string{"modules/avahi": {"go.sum": avahiGoSum}})
if err := ix.Apply(context.Background(), f); err != nil {
@@ -14,14 +14,15 @@ import (
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// The indexer keeps the records pinning each catalog graph revision and removes a revision only on an event (design 47 G §153,
// §178, §189, §215, §216, §579, A-169, A-171): its state pins holds one key per revision, `<branch>.<revision>` (the watched branch
// and the revision's number, the branch escaped as a name is), whose value lists the records that pin it: a take-in's and each
// running pipeline's (design 47 H §743, §749).
// and the revision's number, the branch escaped as a name is), whose value lists the records that pin it: a take-in's, each
// running pipeline's (design 47 H §743, §749) and each passed pipeline's release until a newer one is the want (H-A-168, H-A-170).
const (
// PinsState is the state's name; PinsCapMiB is its cap in the manifest (about 1,800 keys of about 0.2 KiB).
PinsState = "pins"
@@ -72,7 +73,8 @@ func pinObject(key string) (string, bool) {
}
// HeardPins applies one event of the seat mesh-pipeline-runner the indexer's durable consumers gave it (design 47 H §749): a
// pipeline-started pins the revision it names under the record `pipeline.<pipeline>`, its pipeline-ended frees it. A body
// pipeline-started pins the revision it names under the record `pipeline.<pipeline>`, its pipeline-ended frees it and every
// other revision the record pins (H-A-147). A body
// that is no such event, or names no revision, changes nothing; an error leaves the event unacknowledged, so the bus gives it
// again; one applied twice changes nothing.
func (ix *Indexer) HeardPins(ctx context.Context, started *runner.PipelineStarted, ended *runner.PipelineEnded) error {
@@ -83,7 +85,118 @@ func (ix *Indexer) HeardPins(ctx context.Context, started *runner.PipelineStarte
case started != nil && started.Revision > 0 && started.Pipeline != "":
return ix.pin(ctx, PinKey(started.Branch, started.Revision), PipelineRecord(started.Pipeline))
case ended != nil && ended.Revision > 0 && ended.Pipeline != "":
return ix.unpin(ctx, PinKey(ended.Branch, ended.Revision), PipelineRecord(ended.Pipeline))
// A passed pipeline's release pins the revision its movers' entries live in, before the pipeline's own pin is freed,
// one record per mover that released a version in it (H-A-168, H-A-170, H-A-173); a mover that released none, and a run
// on request's pipeline, which delivers nothing, pin none.
if ended.Outcome == runner.PipelineEndedOutcomePassed && ended.Run == "" {
for _, m := range ended.Movers {
if m.Version == "" {
continue
}
if err := ix.pin(ctx, PinKey(ended.Branch, ended.Revision), ReleaseRecord(ended.Branch, m.Module)); err != nil {
return err
}
}
}
record := PipelineRecord(ended.Pipeline)
if err := ix.unpin(ctx, PinKey(ended.Branch, ended.Revision), record); err != nil {
return err
}
return ix.unpinElsewhere(ctx, record)
}
return nil
}
// ReleaseRecord is the record a passed pipeline's release pins its revision under, one per mover: `release.<branch>.<module>`,
// each name escaped as a key's name is (design 47 H-A-168, H-A-170).
func ReleaseRecord(branch, module string) string {
return "release." + keys.EscapeName(branch) + "." + keys.EscapeName(module)
}
// HeardPipelineWanted applies one pipeline-wanted of the seat mesh-wants, said when a pipeline's wants are written to the end
// (design 47 H §751, H-A-168, H-A-170, H-A-173, H-A-174). Outcome running: a module whose release record the pipeline's own
// kept revision holds released its newer version there, and that is the want everywhere, so the module's record is freed from
// every revision of the branch below it; a mover the revision holds no release of (superseded, no version released) keeps its
// older release's revision. Outcome tier-late frees nothing: its later tiers' wants are not written, though their release
// records are held, and the same pipeline's later running frees them. A pipeline whose revision is not kept frees nothing and
// says so; one applied twice changes nothing.
func (ix *Indexer) HeardPipelineWanted(ctx context.Context, branch string, source int64, outcome string) error {
if ix.Pins == nil || ix.Revisions == nil {
return errors.New("the indexer keeps no state " + PinsState)
}
if outcome != meshwants.PipelineWantedOutcomeRunning {
ix.logf("pins: pipeline-wanted of %s %d is %s: it frees no release until its wants are written everywhere (H-A-174)", branch, source, outcome)
return nil
}
name, err := revisionOfSource(ctx, ix.Revisions, branch, source)
if err != nil {
ix.logf("pins: pipeline-wanted of %s %d frees no release: its revision is not kept: %v", branch, source, err)
return nil
}
rev, _, err := ReadRevision(ctx, ix.Revisions, name)
if err != nil {
return fmt.Errorf("%s: %w", name, err)
}
own, _, found, err := ix.Pins.Get(ctx, PinKey(branch, rev.Revision))
if err != nil {
return err
}
var held PinValue
if found {
_ = json.Unmarshal(own, &held)
}
var released []meshcatalog.TakeInEntry
for _, m := range rev.Movers {
if slices.Contains(held.Records, ReleaseRecord(branch, m.Module)) {
released = append(released, m)
}
}
pinKeys, err := ix.Pins.Keys(ctx)
if err != nil {
return err
}
for _, k := range pinKeys {
b, r, ok := ParsePinKey(k)
if !ok || b != branch || r >= rev.Revision {
continue
}
for _, m := range released {
v, _, found, err := ix.Pins.Get(ctx, k)
if err != nil {
return err
}
var pv PinValue
record := ReleaseRecord(branch, m.Module)
if !found || json.Unmarshal(v, &pv) != nil || !slices.Contains(pv.Records, record) {
continue
}
if err := ix.unpin(ctx, k, record); err != nil {
return err
}
}
}
return nil
}
// unpinElsewhere frees a record on every other key that lists it: the branch's previous revision the take-in pinned for the
// same pipeline (H-A-147), which the pipeline-ended does not name.
func (ix *Indexer) unpinElsewhere(ctx context.Context, record string) error {
pinKeys, err := ix.Pins.Keys(ctx)
if err != nil {
return err
}
for _, k := range pinKeys {
v, _, found, err := ix.Pins.Get(ctx, k)
if err != nil {
return err
}
var pv PinValue
if !found || json.Unmarshal(v, &pv) != nil || !slices.Contains(pv.Records, record) {
continue
}
if err := ix.unpin(ctx, k, record); err != nil {
return err
}
}
return nil
}
@@ -253,25 +366,10 @@ func (ix *Indexer) judgePinsLocked(ctx context.Context) ([]string, error) {
return nil, err
}
current := CurrentRevisions(names)
pinned := map[string]bool{}
pinKeys, err := ix.Pins.Keys(ctx)
pinKeys, pinned, err := ix.pinnedObjects(ctx)
if err != nil {
return nil, err
}
for _, k := range pinKeys {
object, ok := pinObject(k)
if !ok {
continue
}
v, _, found, err := ix.Pins.Get(ctx, k)
if err != nil {
return nil, err
}
var pv PinValue
if found && json.Unmarshal(v, &pv) == nil && len(pv.Records) > 0 {
pinned[object] = true
}
}
var removed []string
for _, n := range names {
if _, _, ok := RevisionGroup(n); !ok || pinned[n] {
@@ -301,6 +399,72 @@ func (ix *Indexer) judgePinsLocked(ctx context.Context) ([]string, error) {
return removed, nil
}
// pinnedObjects are the state pins' keys and the revision objects a record pins.
func (ix *Indexer) pinnedObjects(ctx context.Context) ([]string, map[string]bool, error) {
pinned := map[string]bool{}
pinKeys, err := ix.Pins.Keys(ctx)
if err != nil {
return nil, nil, err
}
for _, k := range pinKeys {
object, ok := pinObject(k)
if !ok {
continue
}
v, _, found, err := ix.Pins.Get(ctx, k)
if err != nil {
return nil, nil, err
}
var pv PinValue
if found && json.Unmarshal(v, &pv) == nil && len(pv.Records) > 0 {
pinned[object] = true
}
}
return pinKeys, pinned, nil
}
// earlierForm says whether a revision object is of the earlier model's forms, trunk.<n> or branch.<branch>.<n>, beside the
// branch-keyed <branch>.<n> of design 47 H §743 (H-A-135).
func earlierForm(name string) bool {
if _, _, ok := RevisionGroup(name); !ok {
return false
}
return strings.HasPrefix(name, "trunk.") || strings.HasPrefix(name, "branch.")
}
// dropEarlierFormsLocked drops every revision object of the earlier forms (earlierForm) no record pins, at a branch's take-in
// (design 47 H §743, H-A-135): the branch's newest revision is its current one, and an earlier trunk revision kept beside it
// would be the newest trunk revision a reader hears. It drops none while the pins do not count yet; the caller holds keepMu.
func (ix *Indexer) dropEarlierFormsLocked(ctx context.Context) ([]string, error) {
if ix.PinsPending != nil {
if pending, err := ix.PinsPending(ctx); err != nil || pending != 0 {
return nil, err
}
}
names, err := ix.Revisions.Names(ctx)
if err != nil {
return nil, err
}
pinned := map[string]bool{}
if ix.Pins != nil {
if _, pinned, err = ix.pinnedObjects(ctx); err != nil {
return nil, err
}
}
var dropped []string
for _, n := range names {
if !earlierForm(n) || pinned[n] {
continue
}
if err := ix.Revisions.Delete(ctx, n); err != nil {
return dropped, fmt.Errorf("%s: %w", n, err)
}
dropped = append(dropped, n)
}
sort.Strings(dropped)
return dropped, nil
}
// PinDrop drops one pin of the state pins by hand, by its key <branch>.<revision> (indexer_pin_drop, design 47 H §765, H-A-57):
// every record's, for a pin that names no running pipeline; the finding pin-missing of the key is cleared, and the removal
// rule then applies.
@@ -3,12 +3,14 @@ package index
import (
"context"
"encoding/json"
"fmt"
"slices"
"testing"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
runner "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
)
// Design 47 H §749 (G §579, A-169, A-171): the indexer pins a revision on a pipeline-started that names it and frees it on its
@@ -91,6 +93,55 @@ func TestTheIndexerRemovesARevisionOnlyWhenNoRecordPinsItAndItIsNotCurrent(t *te
if keys, _ := ix.Pins.Keys(ctx); slices.Contains(keys, "main.1") {
t.Errorf("the state holds %v after the pin drop, want no main.1", keys)
}
// H-A-168, H-A-170, H-A-173: a passed pipeline's release keeps its revision past a later failed pipeline; a passed one whose
// mover released no version pins nothing and its pipeline-wanted frees nothing; the next one that released a version frees
// the older release on its pipeline-wanted running (its wants written to the end), not on a tier-late one (H-A-174), and the
// revision goes.
mover := []meshcatalog.TakeInEntry{{Module: "reader", Commit: "c", Fingerprint: "f", Repository: "novox/r"}}
take := func(source int64) {
t.Helper()
e := reader
e.Commit = fmt.Sprintf("r%d", source)
if err := ix.KeepBranchEntries(ctx, "main", source, map[string]meshcatalog.CatalogEntry{"reader": e}, mover); err != nil {
t.Fatal(err)
}
}
end := func(source int64, outcome, version string) {
t.Helper()
e := runner.PipelineEnded{Branch: "main", Pipeline: fmt.Sprintf("main.%d", source), Revision: source, Source: source, Outcome: outcome,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: version}}}
if err := ix.HeardPins(ctx, nil, &e); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
}
wanted := func(source int64, outcome string) {
t.Helper()
if err := ix.HeardPipelineWanted(ctx, "main", source, outcome); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
}
take(10)
end(10, runner.PipelineEndedOutcomePassed, "0.1.10")
take(11)
end(11, runner.PipelineEndedOutcomeFailed, "")
want("a passed release past a later failed pipeline", "main.10", "main.11")
take(12)
end(12, runner.PipelineEndedOutcomePassed, "")
wanted(12, meshwants.PipelineWantedOutcomeRunning)
want("a passed pipeline releasing no version frees no older release", "main.10", "main.12")
take(13)
end(13, runner.PipelineEndedOutcomePassed, "0.1.13")
want("two releases, the unreleased one between them gone", "main.10", "main.13")
wanted(13, meshwants.PipelineWantedOutcomeTierLate) // H-A-174: its later tiers' wants are not written, so it frees nothing
want("a tier-late pipeline-wanted", "main.10", "main.13")
wanted(13, meshwants.PipelineWantedOutcomeRunning)
wanted(13, meshwants.PipelineWantedOutcomeRunning) // heard twice: changes nothing more
want("the newer release written everywhere", "main.13")
if v, _, _, _ := ix.Pins.Get(ctx, "main.13"); !slices.Equal(mustRecords(t, v), []string{ReleaseRecord("main", "reader")}) {
t.Errorf("the pin of main.13 holds %s, want its release alone", v)
}
}
func mustRecords(t *testing.T, v []byte) []string {
@@ -102,3 +153,99 @@ func mustRecords(t *testing.T, v []byte) []string {
slices.Sort(pv.Records)
return pv.Records
}
// H-A-147 (§743, §749, H-A-108 (3)): a take-in with a mover pins, in its write, the branch's previous revision (the newest kept
// before the write) for the same pipeline record as its own, so the validate step's case before can read it while the pipeline
// runs, and names it as its base on the revision it writes (H-A-151, H-A-154); a take-in with no revision before it pins only
// its own and names no base; the pipeline's end frees both. A passed pipeline's end pins its revision under its mover's release
// record instead (H-A-168, H-A-170), so it stays when a later take-in makes it no longer current.
func TestATakeInPinsTheBranchsPreviousRevisionForItsPipeline(t *testing.T) {
ctx := context.Background()
reader := meshcatalog.CatalogEntry{Module: "reader", Repository: "novox/r", Manifest: map[string]any{}}
mover := []meshcatalog.TakeInEntry{{Module: "reader", Commit: "c", Fingerprint: "f", Repository: "novox/r"}}
for _, tc := range []struct {
name string
before []int64 // sources taken in with no mover before the take-in of source 5
whileRuns []string
afterTheEnd []string
base int64 // the base main.5 names (H-A-154)
passed bool // main.5's pipeline passed: its release pins main.5
unreleased bool // its mover released no version there, so nothing pins main.5 (H-A-173)
then []int64 // sources taken in with no mover after the end
released []string
}{
{name: "the previous revision kept and pinned", before: []int64{3, 4}, whileRuns: []string{"main.4", "main.5"}, afterTheEnd: []string{"main.5"}, base: 4},
{name: "the newest kept before, whatever its number", before: []int64{2}, whileRuns: []string{"main.2", "main.5"}, afterTheEnd: []string{"main.5"}, base: 2},
{name: "no revision before it pins only its own", whileRuns: []string{"main.5"}, afterTheEnd: []string{"main.5"}},
{name: "a passed pipeline's release keeps its revision once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, passed: true, then: []int64{6}, afterTheEnd: []string{"main.5", "main.6"}, released: []string{"main.5"}},
{name: "a passed pipeline whose mover released no version pins nothing", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, passed: true, unreleased: true, then: []int64{6}, afterTheEnd: []string{"main.6"}},
{name: "a failed pipeline's revision goes once no longer current", before: []int64{4}, whileRuns: []string{"main.4", "main.5"},
base: 4, then: []int64{6}, afterTheEnd: []string{"main.6"}},
} {
t.Run(tc.name, func(t *testing.T) {
ix := &Indexer{Module: testSelf, Revisions: newMemObjects(1 << 20), Pins: newMem(), Findings: newFindings(),
Now: func() time.Time { return time.Date(2026, 11, 20, 16, 0, 0, 0, time.UTC) },
PinsPending: func(context.Context) (uint64, error) { return 0, nil }}
keep := func(source int64, movers []meshcatalog.TakeInEntry) {
t.Helper()
e := reader
e.Commit = fmt.Sprintf("c%d", source)
if err := ix.KeepBranchEntries(ctx, "main", source, map[string]meshcatalog.CatalogEntry{"reader": e}, movers); err != nil {
t.Fatal(err)
}
}
kept := func() []string {
n, _ := ix.Revisions.Names(ctx)
n = slices.DeleteFunc(n, func(s string) bool { _, _, ok := RevisionGroup(s); return !ok })
slices.Sort(n)
return n
}
for _, s := range tc.before {
keep(s, nil)
}
keep(5, mover)
if rev, _, err := ReadRevision(ctx, ix.Revisions, "main.5"); err != nil || rev.Base != tc.base {
t.Errorf("main.5 names the base %d (%v), want %d, the revision its take-in pinned (H-A-154)", rev.Base, err, tc.base)
}
if got := kept(); !slices.Equal(got, tc.whileRuns) {
t.Errorf("while the pipeline runs: kept %v, want %v", got, tc.whileRuns)
}
for _, k := range tc.whileRuns {
if v, _, _, _ := ix.Pins.Get(ctx, k); !slices.Equal(mustRecords(t, v), []string{"pipeline.main.5"}) {
t.Errorf("the pin of %s holds %s, want the pipeline's record", k, v)
}
}
end := runner.PipelineEnded{Branch: "main", Pipeline: "main.5", Revision: 5, Outcome: runner.PipelineEndedOutcomeFailed,
Movers: []runner.MoverEnded{{Module: "reader", Fingerprint: "f", Version: "0.1.5"}}}
if tc.unreleased {
end.Movers[0].Version = ""
}
if tc.passed {
end.Outcome = runner.PipelineEndedOutcomePassed
}
if err := ix.HeardPins(ctx, nil, &end); err != nil {
t.Fatal(err)
}
ix.JudgePins(ctx)
for _, s := range tc.then {
keep(s, nil)
}
if got := kept(); !slices.Equal(got, tc.afterTheEnd) {
t.Errorf("after the pipeline's end: kept %v, want %v", got, tc.afterTheEnd)
}
if keys, _ := ix.Pins.Keys(ctx); len(keys) > 0 {
for _, k := range keys {
want := []string{}
if slices.Contains(tc.released, k) {
want = []string{ReleaseRecord("main", "reader")}
}
if v, _, _, _ := ix.Pins.Get(ctx, k); !slices.Equal(append([]string{}, mustRecords(t, v)...), want) {
t.Errorf("after the pipeline's end the pin of %s holds %s, want %v", k, v, want)
}
}
}
})
}
}
@@ -41,7 +41,7 @@ func (r Requirement) Versions() []string {
return out
}
// RequiredOf is, per mesh Go module a go.sum names (as fingerprint.MeshPackagesOf takes them), the highest version it
// RequiredOf is, per mesh Go module a go.sum names (a line under trunk.ForgeModulePrefix whose path names a repository), the highest version it
// names, its module line or its go.mod line alike.
func RequiredOf(goSum []byte) map[string]string {
out := map[string]string{}
@@ -61,7 +61,8 @@ type RevisionSummary struct {
Heads string `json:"heads,omitempty"`
At *time.Time `json:"at,omitempty"`
Entries int `json:"entries"`
// Current says it is the trunk's newest revision, which is kept whatever pins it.
// Current says it is the newest revision of its watched branch (CurrentRevisions, §579; H §743, H-A-135: no longer the
// newest trunk-kind revision over every branch, and never one of the earlier forms), which is kept whatever pins it.
Current bool `json:"current,omitempty"`
}
@@ -84,7 +85,6 @@ func ListRevisions(ctx context.Context, o Objects, candidates bool) (RevisionsAn
return out, fmt.Errorf("the objects %s cannot be listed: %w", CatalogRevisionsObjects, err)
}
listed := map[string]bool{}
var newest int64
entries := []string{}
for _, n := range names {
if strings.HasPrefix(n, "entry.") {
@@ -103,17 +103,16 @@ func ListRevisions(ctx context.Context, o Objects, candidates bool) (RevisionsAn
for _, ref := range rev.Entries {
listed[EntryObject(ref.Digest)] = true
}
if rev.Kind == "trunk" && rev.Revision > newest {
newest = rev.Revision
}
if candidates && rev.Kind != "candidate" {
continue
}
out.Revisions = append(out.Revisions, RevisionSummary{Name: n, Kind: rev.Kind, Revision: rev.Revision, Source: rev.Source,
Group: rev.Group, Heads: rev.Heads, At: rev.At, Entries: len(rev.Entries)})
}
current := CurrentRevisions(names)
for i := range out.Revisions {
out.Revisions[i].Current = out.Revisions[i].Kind == "trunk" && out.Revisions[i].Revision == newest
_, cur := current[out.Revisions[i].Name]
out.Revisions[i].Current = cur && out.Revisions[i].Kind == "trunk" && !earlierForm(out.Revisions[i].Name)
}
slices.SortFunc(out.Revisions, func(a, b RevisionSummary) int { return strings.Compare(a.Name, b.Name) })
out.Entries = len(entries)
@@ -108,7 +108,7 @@ func (ix *Indexer) SeedModulesFrom(ctx context.Context, defs map[string]seatdef.
seat := seatOf[a.Ref]
sa := SeedArtifact{Name: name, Seat: seat}
in := nodebuildgo.StepsInput{Module: e.Module, Artifact: nodebuildgo.DetectedStepsByModuleArtifactKey(e.Module, name), Fingerprint: e.Fp,
Fetch: &nodebuildgo.Fetch{Repository: e.Repository, Commit: e.Commit, Packages: maps.Clone(e.Packages)}}
Fetch: &nodebuildgo.Fetch{Repository: e.Repository, Commit: e.Commit, Packages: PinsOf(e.Packages)}}
// The module's directory and its named paths go together, only when the entry names its paths (ADR 0499 clause
// 4); a request without them builds from the module's directory (sdk/buildstep FetchRequest).
if paths := namedPaths(e, name); len(paths) > 0 {
@@ -24,7 +24,7 @@ func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
fp := "src2:" + strings.Repeat("a", 64)
entries := map[string]meshcatalog.CatalogEntry{"avahi": {Module: "avahi", Repository: "novox/r", Commit: "c1", Path: "mods/avahi", Fp: fp, Manifest: map[string]any{},
Artifacts: []meshcatalog.Artifact{{Ref: "avahi:program", Language: "go"}, {Ref: "avahi:image", Language: "go", Kind: "image"}},
Packages: map[string]string{"git.novox.be/novox/mesh-sdk": "c9"},
Packages: map[string]string{"git.novox.be/novox/mesh-sdk": "v0.1.9", "git.novox.be/novox/mesh-contracts/go": ""}, // the second none released (H-A-161)
Inputs: &meshcatalog.EntryInputs{Paths: map[string][]string{"program": {"cmd"}}}}}
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, entries)
if err != nil {
@@ -71,9 +71,12 @@ func TestTheSeedAsksStepsOfTheBuildSeatsOfTheTrunkEntries(t *testing.T) {
t.Errorf("the image artifact is %+v", a)
}
in := got[0]
if in.Module != "avahi" || in.Artifact != "avahi.program" || in.Fingerprint != fp || in.Fetch == nil || in.Fetch.Repository != "novox/r" || in.Fetch.Commit != "c1" || in.Fetch.Dir != "mods/avahi" || len(in.Fetch.Paths) != 1 || in.Fetch.Packages["git.novox.be/novox/mesh-sdk"] != "c9" {
if in.Module != "avahi" || in.Artifact != "avahi.program" || in.Fingerprint != fp || in.Fetch == nil || in.Fetch.Repository != "novox/r" || in.Fetch.Commit != "c1" || in.Fetch.Dir != "mods/avahi" || len(in.Fetch.Paths) != 1 || in.Fetch.Packages["git.novox.be/novox/mesh-sdk"] != "v0.1.9" {
t.Errorf("the steps input is %+v", in)
}
if _, pinned := in.Fetch.Packages["git.novox.be/novox/mesh-contracts/go"]; pinned {
t.Errorf("a package kept with no version is pinned in the fetch: %v", in.Fetch.Packages)
}
if len(w.Held()) != 0 {
t.Fatal("a call unanswered for no time wrote its finding")
}
@@ -142,3 +142,37 @@ func (ix *Indexer) sayTaken(ctx context.Context, t meshcatalog.TakeInTaken) erro
func PendingID(branch string, source int64) string {
return takein.TakeInPendingEvent + "." + keys.EscapeName(branch) + "." + strconv.FormatInt(source, 10)
}
// SourceToTake is the source-moved a read of the gate's source revision of a branch, through the seat mesh-delivery's verb
// source, takes in as a heard one of that revision would be (design 47 H H-A-129), and false when the view keeps a revision of
// the branch at or above it, so a revision is never taken in twice. It is the branch's first take-in, start true, exactly
// when the view keeps no revision of the branch; otherwise its movers are judged against the newest kept (H-A-130). Every
// repository the answer names is a head: the movers are the fingerprint changes alone.
func (ix *Indexer) SourceToTake(ctx context.Context, branch string, a meshdelivery.SourceAnswer) (meshdelivery.SourceMovedEvent, bool, error) {
e := meshdelivery.SourceMovedEvent{V: 1, Branch: branch, Source: a.Revision, Call: a.Call,
Heads: make([]meshdelivery.SourceMovedEventHeadsItem, 0, len(a.Repositories))}
if a.Revision <= 0 {
return e, false, nil
}
if ix.Revisions == nil {
return e, false, errors.New("the indexer keeps no catalog revisions")
}
names, err := ix.Revisions.Names(ctx)
if err != nil {
return e, false, fmt.Errorf("the objects %s: %w", CatalogRevisionsObjects, err)
}
var kept int64
for _, n := range names {
if r, ok := BranchRevisionOf(branch, n); ok && r > kept {
kept = r
}
}
if kept >= a.Revision {
return e, false, nil
}
e.Start = kept == 0
for _, r := range a.Repositories {
e.Heads = append(e.Heads, meshdelivery.SourceMovedEventHeadsItem{Repository: r.Repository, Commit: r.Head})
}
return e, true, nil
}
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"testing"
@@ -121,6 +122,39 @@ func TestAFirstSourceRevisionIsTakenInWithNoMover(t *testing.T) {
}
}
// Design 47 H §743, H-A-135: at a branch's take-in the earlier model's revisions (trunk.<n>, branch.<branch>.<n>) no record
// pins are dropped, one a record pins stays, and the branch's newest revision is the current one, no longer the newest trunk.
func TestATakeInDropsTheEarlierFormsNoRecordPins(t *testing.T) {
ctx := context.Background()
ix, objects, _, f := takeInIndexer(t)
for _, n := range []string{"trunk.527", "trunk.528", "branch.master.1"} {
_, rev, _ := RevisionGroup(n)
raw, _ := json.Marshal(meshcatalog.CatalogRevision{Revision: rev, Kind: "trunk", Source: 1000 + rev})
if err := objects.Put(ctx, n, raw); err != nil {
t.Fatal(err)
}
}
if err := ix.pin(ctx, "trunk.527", "pipeline.old"); err != nil {
t.Fatal(err)
}
start := meshdelivery.SourceMovedEvent{V: 1, Branch: "main", Source: 1, Call: "seed.operator.20261120", Start: true,
Heads: []meshdelivery.SourceMovedEventHeadsItem{{Repository: f.Repository, Commit: f.Merge}}}
if _, err := ix.TakeInSource(ctx, start); err != nil {
t.Fatal(err)
}
a, err := ListRevisions(ctx, objects, false)
if err != nil {
t.Fatal(err)
}
var got []string
for _, r := range a.Revisions {
got = append(got, fmt.Sprintf("%s current=%t", r.Name, r.Current))
}
if want := []string{"main.1 current=true", "trunk.527 current=false"}; !reflect.DeepEqual(got, want) {
t.Errorf("kept %v, want %v: the unpinned earlier forms dropped, the pinned one kept, main.1 current", got, want)
}
}
// Design 47 H §769: the take-in of a source revision moving the catalogue repository reads `predefined` from that commit's
// trees, and one changing a predefined key rewrites it.
func TestATakeInOfTheCatalogsCommitRewritesPredefined(t *testing.T) {
@@ -222,3 +256,35 @@ func TestATakeInOnAFullStoreIsHeldAndSaidOnce(t *testing.T) {
t.Errorf("held = %+v after the skip, want none", held)
}
}
// Design 47 H §743, §768, H-A-129, H-A-130: the gate's source revision read through its verb source is taken in as a heard
// source-moved of it would be: start true when the view keeps no revision of the branch, every repository it names a head;
// one the view keeps at or above is not taken again; a later one is judged against the newest kept, start false.
func TestASourceRevisionReadFromTheGateIsTakenInOnce(t *testing.T) {
ctx := context.Background()
ix, _, said, f := takeInIndexer(t)
answer := meshdelivery.SourceAnswer{Branch: "main", Revision: 1, Call: "0123456789abcdef",
Repositories: []meshdelivery.SourceHead{{Repository: f.Repository, Trunk: "main", Head: f.Merge}}}
e, take, err := ix.SourceToTake(ctx, "main", answer)
want := meshdelivery.SourceMovedEvent{V: 1, Branch: "main", Source: 1, Call: answer.Call, Start: true,
Heads: []meshdelivery.SourceMovedEventHeadsItem{{Repository: f.Repository, Commit: f.Merge}}}
if err != nil || !take || !reflect.DeepEqual(e, want) {
t.Fatalf("a branch the view keeps none of answered %+v, %t (%v); want %+v taken", e, take, err, want)
}
if taken, err := ix.TakeInSource(ctx, e); err != nil || len(taken.Entries) != 0 || taken.Revision != 1 {
t.Fatalf("the first revision read from the gate said %+v (%v), want revision 1 with no mover", taken, err)
}
if _, take, err := ix.SourceToTake(ctx, "main", answer); take || err != nil {
t.Errorf("a revision the view keeps is taken again (%v)", err)
}
if _, take, _ := ix.SourceToTake(ctx, "nox-mesh", meshdelivery.SourceAnswer{Branch: "nox-mesh"}); take {
t.Error("a branch the gate keeps no revision of is taken in")
}
answer.Revision, answer.Call = 2, "1123456789abcdef"
if e, take, err := ix.SourceToTake(ctx, "main", answer); err != nil || !take || e.Start || e.Source != 2 {
t.Errorf("a later revision answered %+v, %t (%v); want revision 2 taken, start false", e, take, err)
}
if len(*said) != 1 {
t.Errorf("said %d take-in.taken, want 1", len(*said))
}
}
@@ -9,7 +9,7 @@ type RepoRead struct {
Trees map[string]string
Manifests map[string]string
Locks map[string]map[string]string
// Packages are the mesh packages each directory's go.sum names (fingerprint.MeshPackagesOf, novox/hq ADR 0573 §2).
// Packages are the mesh packages each directory's go.mod requires (fingerprint.MeshRequiresOf, design 47 H-A-157).
Packages map[string][]string
}
@@ -16,6 +16,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/internal/fingerprint"
"mesh-delivery-indexer/judge"
)
@@ -152,11 +153,13 @@ func (ix *Indexer) recomputeModules(ctx context.Context, overlay map[string]mesh
}
ix.recompute.Lock()
defer ix.recompute.Unlock()
return ix.recomputeLocked(ctx, overlay, w, repository)
return ix.recomputeLocked(ctx, overlay, w, repository, nil)
}
// recomputeLocked is recomputeModules with ix.recompute held by the caller (a hold that moves, holds.go).
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string) error {
//
// only, when not nil, names the modules the pass moves (a re-pin, H-A-165): every other stands as held.
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string, only map[string]bool) error {
if ix.Manifests == nil {
return ix.writeOverlay(ctx, overlay, nil, w)
}
@@ -287,18 +290,20 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
r.none = strings.Join(missing, "; ")
case noToolchain != "":
r.none = noToolchain
if fp, pkgs, ok := stillComputed(module, kept, k, toolchains, held, pub, pins, ix.sourceHeads()); ok {
if fp, pkgs, ok := stillComputed(module, kept, k, toolchains, held, pub, pins, ix.sourceHeads(), series); ok {
r = result{fp: fp, tcs: k.held.Toolchains, pkgs: pkgs}
}
default:
f := formOf(module, k.cur.Repository, &kept, toolchains, k.held, held, pub, pins, ix.sourceHeads(), ix.logf)
r.fp, r.none = f.fp, f.none
if r.fp != "" {
r.tcs, r.pkgs = toolchainsUsed(toolchains), f.versions
// a pin is a released version, never a commit (H-A-161), and the version line's fingerprint takes it (H-A-167)
r.fp, r.pkgs = pinnedOf(f, series, k.manifest)
r.tcs = toolchainsUsed(toolchains)
}
}
if _, merged := overlay[module]; !merged && repository != "" && !trunk.SameRepository(k.cur.Repository, repository) &&
maps.Equal(k.held.Packages, map[string]string(r.pkgs)) {
if _, merged := overlay[module]; !merged && (repository != "" && !trunk.SameRepository(k.cur.Repository, repository) &&
!importsOf(r.pkgs, repository) || only != nil && !only[module]) {
// A module registered for another repository is moved by this repository's commit only when it imports a package the
// commit moved (A-476a, ADR 0593 §3); any other stands as held (A-476 c), and so it is for a base of this repository's.
r = result{fp: k.held.Fp2, none: k.held.None2, tcs: k.held.Toolchains, pkgs: sdkstep.PackageVersions(k.held.Packages)}
@@ -492,7 +497,7 @@ func hasContext(m catalogue.Manifest) bool {
// with the toolchains it was computed with in place of the languages agreed on none, give it again, in its form. Any
// other input that moved (a merge, a base, an agreed toolchain, an imported package) gives another fingerprint, and the
// module has none until they agree.
func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2, k *known, agreed map[string]meshcatalogclient.Toolchain2, held map[string]meshcatalog.ModulesByModuleValue, published map[string]meshcatalog.PackagesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string) (string, sdkstep.PackageVersions, bool) {
func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2, k *known, agreed map[string]meshcatalogclient.Toolchain2, held map[string]meshcatalog.ModulesByModuleValue, published map[string]meshcatalog.PackagesByModuleValue, pins sdkstep.PackageVersions, heads map[string]string, series map[string]SeriesPackage) (string, sdkstep.PackageVersions, bool) {
if k.held.Fp2 == "" {
return "", nil, false
}
@@ -509,7 +514,11 @@ func stillComputed(module string, kept meshcatalog.ModulesByModuleValueInputs2,
toolchains[artifact] = meshcatalogclient.Toolchain2{Language: language, Version: used.Version, Recipe: used.Recipe}
}
f := formOf(module, k.held.Repository, &kept, toolchains, k.held, held, published, pins, heads, nil)
return f.fp, f.versions, f.none == "" && f.fp == k.held.Fp2
if f.none != "" || f.fp == "" {
return "", nil, false
}
fp, pkgs := pinnedOf(f, series, k.manifest)
return fp, pkgs, fp == k.held.Fp2
}
// toolchainsUsed are the toolchains a fingerprint was computed with, by language (an artifact's toolchain is its language's).
@@ -577,3 +586,41 @@ func (ix *Indexer) fillPaths(ctx context.Context) error {
}
return nil
}
// importsOf says whether pkgs name a package of repository: a pin is a released version (H-A-161), so a commit of
// repository moves a module of another only through a package of it the module imports (A-476a).
func importsOf(pkgs sdkstep.PackageVersions, repository string) bool {
for p := range pkgs {
if repo, _, ok := trunk.PackageRepository(p); ok && trunk.SameRepository(repo, repository) {
return true
}
}
return false
}
// Repin moves the pins of every module whose packages hold a package of module (a held module's directory holds it) to
// module's newest released version, the pass of H-A-161 for those modules alone: a version-released of module starts it
// (design 47 H-A-165), so pins move on the event and never wait for a merge or a hand act. It answers how many it named.
func (ix *Indexer) Repin(ctx context.Context, module string) (int, error) {
if ix.Modules == nil || ix.Manifests == nil {
return 0, nil
}
held, err := ix.heldModules(ctx)
if err != nil {
return 0, err
}
only := map[string]bool{}
for key, m := range held {
for p := range m.Packages {
if moduleOfKey(fingerprint.HolderOf(p, held)) == module {
only[key] = true
}
}
}
if len(only) == 0 {
return 0, nil
}
ix.recompute.Lock()
defer ix.recompute.Unlock()
return len(only), ix.recomputeLocked(ctx, nil, nil, "", only)
}
@@ -17,6 +17,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/builds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -757,7 +758,7 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Errorf("no published mesh-sdk: fp2 %q none2 %q", u.Fp2, u.None2)
}
tools, _ := moduleOf(t, mods, "mesh-tools")
sdkMerge := func(commit, tree string) meshcatalog.ModulesByModuleValue {
sdkMerge := func(commit, tree, pin string) meshcatalog.ModulesByModuleValue {
t.Helper()
f := whole{Repository: "novox/mesh-sdk", Branch: "feat/x", Head: h1,
Merge: commit, Root: tree, Trees: map[string]string{"": tree}, Manifests: map[string]string{"": `{"module":"mesh-sdk","version":"1.0"}`}}
@@ -765,9 +766,10 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Fatal(err)
}
// A-476a: a commit of mesh-sdk moves avahi, of another repository, which imports its package, and leaves mesh-tools,
// which imports nothing it moved.
// which imports nothing it moved. Its pin is mesh-sdk's newest released version, never the commit; with none
// released it has no pin, the package kept for its kind-1 edge (H-A-161).
a, _ := moduleOf(t, mods, "avahi")
if a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: commit}) {
if a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: pin}) {
t.Fatalf("after mesh-sdk's merge %.8s: fp2 %q none2 %q packages %v", commit, a.Fp2, a.None2, a.Packages)
}
if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 != tools.Fp2 || o.Packages != nil {
@@ -781,8 +783,56 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
stale.Fp2 = "src2:" + strings.Repeat("5", 64)
mods.put("mesh-tools", mustEncode(t, stale))
tools = stale
first := sdkMerge(strings.Repeat("1", 40), "tree-sdk-1")
second := sdkMerge(strings.Repeat("2", 40), "tree-sdk-2")
first := sdkMerge(strings.Repeat("1", 40), "tree-sdk-1", "")
// mesh-sdk releases 0.1.20 (0.1.9 before it): the release re-pins avahi with no merge, its fingerprint moving with the
// pin so it is built again against the release, and leaves mesh-tools, which holds none of it (H-A-165, H-A-167). A
// re-pin that fails is no release acknowledged: the package-published delivered again re-pins.
ix.Versions, ix.Heard = newMem(), newMem()
ix.Versions.(*memStore).put(meshcatalog.VersionsByModuleVersionKey("mesh-sdk", "0.1.9"), []byte(`{}`))
revisions := ix.Revisions
ix.Revisions = newMemObjects(1 << 20)
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, map[string]meshcatalog.CatalogEntry{
"mesh-sdk": {Module: "mesh-sdk", Manifest: map[string]any{}, Artifacts: []meshcatalog.Artifact{{Ref: "mesh-sdk:go"}}}})
if err != nil {
t.Fatal(err)
}
if err := KeepRevision(context.Background(), ix.Revisions, rev, objects); err != nil {
t.Fatal(err)
}
release := meshbuilds.PackagePublished{Module: "mesh-sdk", Artifact: "mesh-sdk:go", Digest: "sha256:g", Version: "0.1.20", Source: 7, Branch: "main"}
mods.keysErr = errors.New("the bus did not answer")
if err := ix.HearPackagePublished(context.Background(), release); err == nil {
t.Fatal("a release whose re-pin failed was acknowledged")
}
mods.keysErr = nil
// The registry published mesh-sdk's Go package at 0.1.18, not at the module's release number 0.1.20, in the series
// avahi accepts (the default 0.1), with no contents read: the pin is that version (H-A-183, H-A-185).
ix.Packages = newMem()
if _, err := ix.HearVersion(context.Background(), HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.1.18", Commit: strings.Repeat("1", 40), GoModule: sdk, Published: true}); err != nil {
t.Fatal(err)
}
if err := ix.HearPackagePublished(context.Background(), release); err != nil {
t.Fatal(err)
}
ix.Revisions = revisions
if a, _ := moduleOf(t, mods, "avahi"); a.Fp2 == first.Fp2 || a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: "v0.1.18"}) {
t.Fatalf("after mesh-sdk's release: fp2 %q (was %q) packages %v, want a new fingerprint pinned to v0.1.18", a.Fp2, first.Fp2, a.Packages)
} else {
// A 0.2.0 publish, of a series avahi does not accept, leaves it alone, contents or none (H-A-184, H-A-185).
if _, err := ix.HearVersion(context.Background(), HeardVersion{Package: "mesh-sdk/go", Seat: "go-package-registry", Version: "0.2.0", Commit: strings.Repeat("1", 40), GoModule: sdk, Published: true}); err != nil {
t.Fatal(err)
}
if _, err := ix.Repin(context.Background(), "mesh-sdk"); err != nil { // the pass for avahi alone: mesh-tools is held stale here
t.Fatal(err)
}
if b, _ := moduleOf(t, mods, "avahi"); b.Fp2 != a.Fp2 || b.Packages[sdk] != "v0.1.18" {
t.Fatalf("a publish of another series moved avahi: fp2 %q (was %q) packages %v", b.Fp2, a.Fp2, b.Packages)
}
}
if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 != tools.Fp2 {
t.Errorf("a re-pin moved mesh-tools, which holds no package of mesh-sdk: %q (was %q)", o.Fp2, tools.Fp2)
}
second := sdkMerge(strings.Repeat("2", 40), "tree-sdk-2", "v0.1.18")
if second.Fp2 == first.Fp2 {
t.Errorf("a mesh-sdk trunk merge kept avahi's fingerprint %q", first.Fp2)
}
@@ -6,14 +6,20 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strconv"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/keys"
"mesh-delivery-indexer/internal/fingerprint"
)
// The catalog's released versions (design 47 G §349, §372, §374): the indexer keeps, for the seat mesh-catalog, each module's
@@ -123,8 +129,9 @@ func (ix *Indexer) HearPackagePublished(ctx context.Context, p meshbuilds.Packag
if raw, _, found, err := ix.Versions.Get(ctx, meshcatalog.VersionsByModuleVersionKey(p.Module, p.Version)); err != nil {
return err
} else if found && !isFloor(raw) {
ix.logf("versions: %s %s is released already, so the package-published of %s is heard again and changes nothing", p.Module, p.Version, p.Artifact)
return nil
// A package-published delivered again after a re-pin failed (H-A-167): the re-pin is made again, nothing else.
ix.logf("versions: %s %s is released already, so the package-published of %s is heard again and only re-pins", p.Module, p.Version, p.Artifact)
return ix.repin(ctx, p.Module, p.Version)
}
entry, err := ix.entryOf(ctx, p.Module)
if err != nil {
@@ -234,6 +241,13 @@ func (ix *Indexer) releaseVersion(ctx context.Context, p meshbuilds.PackagePubli
}
ix.logf("versions: %s %s is released and version-released is said (%d artifacts, source %d)", p.Module, p.Version, len(v.Artifacts), v.Source)
}
if !late {
// The release re-pins every module whose packages hold one of it (design 47 H-A-165); one that fails is not
// acknowledged, so the package-published is delivered again (H-A-167).
if err := ix.repin(ctx, p.Module, p.Version); err != nil {
return err
}
}
if err := ix.trimVersions(ctx, p.Module); err != nil {
ix.logf("versions: %s's oldest versions are not deleted: %v", p.Module, err)
}
@@ -352,3 +366,84 @@ func isFloor(raw []byte) bool {
var v meshcatalog.VersionEntry
return json.Unmarshal(raw, &v) == nil && len(v.Artifacts) == 0
}
// releasedPins are a module's pins (its catalog entry's packages, which the builder builds it against) from the versions its
// fingerprint was computed with (design 47 H-A-161, H-A-183, H-A-184, H-A-185, G §370): each package is pinned to the
// newest version the registry published for its path (the seats' published, kept in the package's series by HearVersion)
// within the series the module accepts, the one its manifest m names or the default (Manifest.AcceptedSeries), whether
// or not that series holds contents yet; a publish in another series pins nothing and moves nothing, and a module's
// release number is never a pin. A package with none published in that series is kept with no version, so its kind-1
// edge stands (H §746) and its build waits for the publish (H-A-165). nil for none.
func releasedPins(versions sdkstep.PackageVersions, series map[string]SeriesPackage, m catalogue.Manifest) sdkstep.PackageVersions {
if len(versions) == 0 {
return nil
}
out := sdkstep.PackageVersions{}
for p := range versions {
out[p] = ""
for _, s := range series {
if s.GoModule != p || s.Newest == nil || s.Series != m.AcceptedSeries(s.Name) {
continue
}
v := strings.TrimPrefix(s.Newest.Version, "v")
if _, ok := versionParts(v); ok && (out[p] == "" || compareVersions(v, strings.TrimPrefix(out[p], "v")) > 0) {
out[p] = "v" + v
}
}
}
return out
}
// pinnedOf is a form's fingerprint and its pins (releasedPins): the pins are part of what a build is made of, so every
// form's fingerprint takes them (design 47 H-A-167, H-A-172; ADR 0573 §4) and a re-pin moves it, the module a mover
// built again against the release; a fingerprint with no pin is the form's as it is. The held fingerprint of a build that
// stands is kept while its pins are the ones it was built with, else its import form takes the new pins.
func pinnedOf(f form, series map[string]SeriesPackage, m catalogue.Manifest) (string, sdkstep.PackageVersions) {
pins := releasedPins(f.versions, series, m)
if f.stands {
if maps.Equal(f.built, map[string]string(pins)) {
return f.fp, pins
}
return fingerprint.WithPins(f.raw, pins), pins
}
return fingerprint.WithPins(f.fp, pins), pins
}
// moduleOfKey is the module a modules key names; "" for none.
func moduleOfKey(key string) string {
if key == "" {
return ""
}
if m, err := meshcatalogclient.ParseModulesKey(key); err == nil {
return m
}
return key
}
// PinsOf are the pins a build is asked with from a catalog entry's packages: each package with a version; one kept with
// none (no released version, H-A-161) is no pin. nil for none.
func PinsOf(packages map[string]string) sdkstep.PackageVersions {
var out sdkstep.PackageVersions
for p, v := range packages {
if v == "" {
continue
}
if out == nil {
out = sdkstep.PackageVersions{}
}
out[p] = v
}
return out
}
// repin is Repin of a release of module at version, said in the journal; its failure is the caller's error.
func (ix *Indexer) repin(ctx context.Context, module, version string) error {
n, err := ix.Repin(ctx, module)
if err != nil {
return fmt.Errorf("the modules pinned to %s are not re-pinned to %s: %w", module, version, err)
}
if n > 0 {
ix.logf("versions: %d modules re-pinned to %s %s", n, module, version)
}
return nil
}
@@ -170,7 +170,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// internal/broker/state.go:62-93).
// The registry seats' published fill each (§413: no consume of version.numbered) package's series (DLV-I5, design 51 *The versions*).
if want := []string{"go-package-registry.published", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-catalog.room", "mesh-conditions.findings-asked",
"mesh-delivery.source-moved", "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.pipeline-started", "mesh-pipeline-runner.run-started", "mesh-wants.wait-ended", "mesh-wants.wanted", "node-build-go.step-ended", "node-engine.running", "npm-package-registry.published", "operator-channel.decided", "seat:mesh-module-assigner.views.assignments", "seat:mesh-module-assigner.views.seat-holders"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
"mesh-delivery.source-moved", "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.pipeline-started", "mesh-pipeline-runner.run-started", "mesh-wants.pipeline-wanted", "mesh-wants.wait-ended", "mesh-wants.wanted", "node-build-go.step-ended", "node-engine.running", "npm-package-registry.published", "operator-channel.decided", "seat:mesh-module-assigner.views.assignments", "seat:mesh-module-assigner.views.seat-holders"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want)
}
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
@@ -182,7 +182,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// Three invokes: the file of a registered module's manifest and lock files (registered.go), the forge's trees (the named paths' objects, since the indexer is the one reader of a manifest's paths:
// ADR 0499 clauses 2 and 3, ADR 0495 clause 1; and the module directories of the trunk, issue 1900) and whether a
// commit is in another (issue 1900).
if want := []string{"seat:git.trees", "seat:git.tags", "seat:go-package-registry.versions", "seat:npm-package-registry.versions", "seat:git.file", "seat:git.contains", "seat:git.branch", "seat:mesh-module-assigner.assignments", "seat:mesh-module-assigner.placement", "seat:mesh-hand-acts.hand-act", "seat:mesh-watchdogs.presence", "seat:node-build-go.steps",
if want := []string{"seat:git.trees", "seat:git.tags", "seat:go-package-registry.versions", "seat:npm-package-registry.versions", "seat:git.file", "seat:git.contains", "seat:git.branch", "seat:mesh-delivery.source", "seat:mesh-module-assigner.assignments", "seat:mesh-module-assigner.placement", "seat:mesh-hand-acts.hand-act", "seat:mesh-watchdogs.presence", "seat:node-build-go.steps",
"seat:node-build-node.steps", "seat:node-build-image.steps", "mesh-delivery-builder.builder_versions"}; !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
@@ -89,6 +89,7 @@
"mesh-pipeline-runner.run-started",
"mesh-pipeline-runner.pipeline-started",
"mesh-pipeline-runner.pipeline-ended",
"mesh-wants.pipeline-wanted",
"node-engine.running",
"operator-channel.decided",
"npm-package-registry.published"
@@ -152,6 +153,7 @@
"seat:git.file",
"seat:git.contains",
"seat:git.branch",
"seat:mesh-delivery.source",
"seat:mesh-module-assigner.assignments",
"seat:mesh-module-assigner.placement",
"seat:mesh-hand-acts.hand-act",
@@ -42,7 +42,7 @@ func TestTools(t *testing.T) {
for _, tl := range UnloggedTools(stdio.State("pipelines"), stdio.Log("step-log"), nil, nil) {
names = append(names, tl.Name)
}
want := "mesh-pipeline-runner.show mesh-pipeline-runner.list mesh-pipeline-runner.stands mesh-pipeline-runner.why-carried mesh-pipeline-runner.kept runner_first_views runner_first_view_again runner_cancel runner_rerun runner_start runner_run runner_accept runner_place runner_ask_again runner_health runner_waiting runner_unlogged runner_unlogged_append"
want := "mesh-pipeline-runner.show mesh-pipeline-runner.list mesh-pipeline-runner.stands mesh-pipeline-runner.why-carried mesh-pipeline-runner.kept runner_first_views runner_first_view_again runner_cancel runner_rerun runner_start runner_run runner_accept runner_place runner_end_again runner_ask_again runner_health runner_waiting runner_unlogged runner_unlogged_append"
if got := strings.Join(names, " "); got != want {
t.Errorf("tools %q, want %q", got, want)
}
@@ -54,6 +54,15 @@ func VerbTools(ask func(key string, body any) (json.RawMessage, error)) []stdio.
}
return whyOf(in.Why)
}),
verbTool[seat.EndAgainInput](ask, "end_again", seat.VerbEndAgain,
"Judge a running pipeline's end once more by the seat verb end-again (design 47 H H-A-163, H-A-164): it ends as at a step's end, failed as soon as it can no longer pass (a mover with no registration named), and says pipeline-ended; one that still cannot end is answered running. The pipeline's key and why; written in the hand-act log. A pipeline not running is refused by the verb.",
obj([]string{"pipeline", "why"}, map[string]any{"pipeline": str, "why": str}),
func(in seat.EndAgainInput) string {
if in.Pipeline == "" {
return "end-again names the pipeline"
}
return whyOf(in.Why)
}),
verbTool[seat.AskAgainInput](ask, "ask_again", seat.VerbAskAgain,
"Ask a declined approval step of a pipeline once more by the seat verb ask-again: the pipeline's key, the mover's module (needed when the pipeline moves several) and why.",
obj([]string{"pipeline", "why"}, map[string]any{"pipeline": str, "module": str, "why": str}),
@@ -32,6 +32,10 @@ func TestVerbTools(t *testing.T) {
{"place", "runner_place", "seat:mesh-pipeline-runner.place", "", map[string]any{"pipeline": "main.6", "why": "urgent"},
seat.PlaceInput{Pipeline: "main.6", Why: "urgent"}},
{"place no why", "runner_place", "", "names why", map[string]any{"pipeline": "main.6"}, nil},
// H-A-164: runner_end_again asks the verb end-again with the pipeline and why.
{"end-again", "runner_end_again", "seat:mesh-pipeline-runner.end-again", "", map[string]any{"pipeline": "main.20", "why": "left running"},
seat.EndAgainInput{Pipeline: "main.20", Why: "left running"}},
{"end-again no why", "runner_end_again", "", "names why", map[string]any{"pipeline": "main.20"}, nil},
{"ask-again no why", "runner_ask_again", "", "names why", map[string]any{"pipeline": "p"}, nil},
} {
t.Run(c.name, func(t *testing.T) {
@@ -152,6 +152,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Kept: keptReader(ctx, kept), Toolchain: toolchainReader(ctx, toolchains),
Fills: fills, Recovered: kvRecoveries{ctx, recovered}.Has, Holders: rv.Holders.Holders, Ended: hearer.EndedAtPlacing}
hearer.AskSteps = starter.AskSteps
hearer.Unplaced = func(p seat.Pipeline, module string) string { return starter.Unplaced(p, module, rv.Catalog.View) }
hearer.Release = func(key string) {
if _, err := starter.Release(hearer.Store, key, rv.Catalog.View); err != nil {
logger.Printf("the movers of %s waiting for a build are not placed: %v", key, err)
@@ -67,6 +67,7 @@ func TestTheQueueRunsPipelinesInTurn(t *testing.T) {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil {
t.Fatal(err)
}
findingsStream(t, ctx, js, module)
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil {
t.Fatal(err)
@@ -75,6 +75,7 @@ func TestTheRunnerPlacesAPipelineAsksItsBuildTakesItsEndAndAnswersAReview(t *tes
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(module, seat.StateStepLog), Subjects: []string{buscompose.LogSubject(module, seat.StateStepLog) + ".>"}}); err != nil {
t.Fatal(err)
}
findingsStream(t, ctx, js, module)
events, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: process.EventsStream, Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"}})
if err != nil {
t.Fatal(err)
@@ -283,6 +284,16 @@ func readPipeline(ctx context.Context, kv jetstream.KeyValue, key string, p *sea
}
// waitFor asks cond until it holds or the test's context ends, the module's journal in the failure.
// findingsStream makes the stream of the module's own finding events, as its manifest's streams ask the bus for
// (`mesh.stream.<module>.event.>`), so a finding the runner writes, such as the unhandled-event count of the SDK's view
// reader (H-A-166, H-A-169), is acknowledged at once and holds no event behind its 3 s wait for the bus.
func findingsStream(t *testing.T, ctx context.Context, js jetstream.JetStream, module string) {
t.Helper()
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: module + "-findings", Subjects: []string{"mesh.stream." + module + ".event.>"}}); err != nil {
t.Fatal(err)
}
}
func waitFor(t *testing.T, ctx context.Context, logs *syncBuffer, cond func() bool) {
t.Helper()
for ctx.Err() == nil {
@@ -38,7 +38,7 @@ func answerVerbs(conn *process.Conn, seats *seatlist.Part, v pipeline.Verbs) (fu
return conn.AnswerSeat(seat.Seat, seatlist.Definition(seats.Reader, seat.Seat), writeHandlers(v))
}
// writeHandlers are the seat's write verbs the process answers: run, review, rerun, accept, cancel, start, and ask-again; and
// writeHandlers are the seat's write verbs the process answers: run, review, rerun, accept, cancel, end-again, start, and ask-again; and
// the test runs' verbs test-runs and test-runs-drop (A-469g, A-470i).
func writeHandlers(v pipeline.Verbs) map[string]process.SeatHandler {
return map[string]process.SeatHandler{
@@ -47,6 +47,7 @@ func writeHandlers(v pipeline.Verbs) map[string]process.SeatHandler {
seat.VerbRerun: verbOf(func(c string, in seat.RerunInput) (any, error) { return v.Rerun(c, in) }),
seat.VerbAccept: verbOf(func(c string, in seat.AcceptInput) (any, error) { return v.Accept(c, in) }),
seat.VerbCancel: verbOf(func(c string, in seat.CancelInput) (any, error) { return v.Cancel(c, in) }),
seat.VerbEndAgain: verbOf(func(c string, in seat.EndAgainInput) (any, error) { return v.EndAgain(c, in) }),
seat.VerbPlace: verbOf(func(c string, in seat.PlaceInput) (any, error) { return v.Place(c, in) }),
seat.VerbStart: verbOf(func(c string, in seat.StartInput) (any, error) { return v.Start(c, in) }),
seat.VerbAskAgain: verbOf(func(c string, in seat.AskAgainInput) (any, error) { return v.AskAgain(c, in) }),
@@ -23,7 +23,7 @@ func TestEveryWriteVerbIsServed(t *testing.T) {
got = append(got, v)
}
slices.Sort(got)
want := []string{seat.VerbAccept, seat.VerbAskAgain, seat.VerbCancel, seat.VerbPlace, seat.VerbRerun, seat.VerbReview, seat.VerbStart, seat.VerbRun, seat.VerbTestRuns, seat.VerbTestRunsDrop}
want := []string{seat.VerbAccept, seat.VerbAskAgain, seat.VerbCancel, seat.VerbEndAgain, seat.VerbPlace, seat.VerbRerun, seat.VerbReview, seat.VerbStart, seat.VerbRun, seat.VerbTestRuns, seat.VerbTestRunsDrop}
slices.Sort(want)
if !slices.Equal(got, want) {
t.Errorf("the process answers %v, want %v", got, want)
@@ -137,7 +137,9 @@ func components(movers []seat.Mover, after map[string][]string) map[string]int {
var buildKinds = []int64{3, 1}
// awaited are the artifact references mover m's steps wait for in p (§746, H-A-51): the From of each edge of a build kind
// into m from another mover of p, toolchain edges first, each once.
// into m from another mover of p, toolchain edges first, each once. A toolchain edge (kind 3) from a build module whose
// own new version is built from m is not awaited: what a build module's new version is built from is built by its
// running version, never waited for (G §270, A-326), so the running build module builds m and the new one the rest.
func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) []catalog.Edge {
var out []catalog.Edge
for _, kind := range buildKinds {
@@ -146,12 +148,35 @@ func awaited(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry)
if x.Kind != kind || from == m || !Moves(p, from) || slices.ContainsFunc(out, func(e catalog.Edge) bool { return e.From == x.From }) {
continue
}
if kind == 3 && builtFrom(p, from, entries)[m] {
continue
}
out = append(out, x)
}
}
return out
}
// builtFrom are the movers of p that module's new version is built from (G §270): the From mover of each edge of a build
// kind into it, and what those are built from in turn.
func builtFrom(p seat.Pipeline, module string, entries map[string]catalog.CatalogEntry) map[string]bool {
seen := map[string]bool{}
next := []string{module}
for len(next) > 0 {
cur := next[0]
next = next[1:]
for _, x := range entries[cur].Edges {
from := moduleOf(x.From)
if !slices.Contains(buildKinds, x.Kind) || from == cur || seen[from] || !Moves(p, from) {
continue
}
seen[from] = true
next = append(next, from)
}
}
return seen
}
// ready says whether mover m of p may be placed (§746, H-A-84): every artifact it awaits is in its From mover's published.
// It answers the references still awaited, and the first From mover whose build ended without passing, so m never builds.
func ready(p seat.Pipeline, m string, entries map[string]catalog.CatalogEntry) (bool, []string, string) {
@@ -4,6 +4,7 @@ import (
"slices"
"strings"
"testing"
"time"
builds "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshbuilds"
catalog "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
@@ -124,3 +125,115 @@ func TestAMoversBuildWaitsForItsDependenciesInThePipeline(t *testing.T) {
}
}
}
// Design 47 G §270, A-326 (H §746, H-A-83 (2)): what a build module's new version is built from is built by its running
// version, never waited for: a mover the moving build module imports, directly or through another mover, does not await
// that build module's toolchain edge, so a pipeline moving mesh-sdk or the contracts with the Go build module places them
// at once (main.28, main.30 of 2026-11-20 waited on each other forever); every other mover still awaits the new one.
func TestAToolchainEdgeIsNotAwaitedByTheBuildModulesOwnInputs(t *testing.T) {
e := func(from, to string, kind int64) catalog.Edge { return catalog.Edge{From: from, To: to, Kind: kind} }
entries := map[string]catalog.CatalogEntry{
"mesh-sdk": {Edges: []catalog.Edge{e("mesh-build-go:files", "mesh-sdk:go", 3)}},
"contracts": {Edges: []catalog.Edge{e("mesh-build-go:files", "contracts:go-a", 3), e("mesh-build-go:files", "contracts:go-b", 3)}},
"delivery-sdk": {Edges: []catalog.Edge{e("mesh-build-go:files", "delivery-sdk:go", 3), e("mesh-sdk:go", "delivery-sdk:go", 1)}},
"mesh-build-go": {Edges: []catalog.Edge{e("contracts:go-a", "mesh-build-go:tools", 1), e("delivery-sdk:go", "mesh-build-go:tools", 1), e("nats:tools", "mesh-build-go:files", 4)}},
"zfs": {Edges: []catalog.Edge{e("mesh-build-go:files", "zfs:tools", 3), e("contracts:go-a", "zfs:tools", 1)}},
"tool": {Edges: []catalog.Edge{e("builder:program", "tool:program", 3)}},
"builder": {},
}
pipe := func(modules ...string) seat.Pipeline {
var p seat.Pipeline
for _, m := range modules {
p.Movers = append(p.Movers, seat.Mover{Module: m})
}
return p
}
all := pipe("mesh-sdk", "contracts", "delivery-sdk", "mesh-build-go", "zfs")
for _, c := range []struct {
name string
p seat.Pipeline
mover string
want []string
}{
{"the build module's own package input awaits no toolchain edge", all, "contracts", nil},
{"an input through another mover awaits no toolchain edge", all, "mesh-sdk", nil},
{"an input that imports another input awaits only that package", all, "delivery-sdk", []string{"mesh-sdk:go"}},
{"the build module awaits its inputs' packages", all, "mesh-build-go", []string{"contracts:go-a", "delivery-sdk:go"}},
{"any other mover awaits the new build module first", all, "zfs", []string{"mesh-build-go:files", "contracts:go-a"}},
{"a build module that is not built from the mover is awaited", pipe("tool", "builder"), "tool", []string{"builder:program"}},
{"a build module that does not move is not awaited", pipe("tool"), "tool", nil},
} {
var got []string
for _, x := range awaited(c.p, c.mover, entries) {
got = append(got, x.From)
}
if !slices.Equal(got, c.want) {
t.Errorf("%s: %s awaits %v, want %v", c.name, c.mover, got, c.want)
}
}
}
// Design 47 H H-A-175, H-A-176 (G §270, A-326; H-A-101 (2), H-A-163): when every mover without steps waits only on movers
// without steps, their steps are placed ended not asked, the why `cycle` naming the edges inside the cycle alone; a step
// of another mover is asked as ever, and the pipeline ends by the end judgement, at once when no required step is open; a
// cycle through a build module's own inputs is broken (G §270) and placed; a mover waiting on a placed mover only waits.
func TestACycleAmongUnplacedMoversEndsThePipelineFailed(t *testing.T) {
reg, base := placeFixture()
reg.Policy = catalog.Policy{Steps: catalog.StepStatuses{Lint: "bypass", Unit: "bypass", Integration: "bypass", Validate: "bypass", Composition: "bypass"}}
entry := func(m string, edges ...catalog.Edge) catalog.CatalogEntry {
return catalog.CatalogEntry{Module: m, Artifacts: []catalog.Artifact{{Ref: m + ":go", Language: "go"}}, Edges: edges}
}
e := func(from, to string, kind int64) catalog.Edge { return catalog.Edge{From: from, To: to, Kind: kind} }
cycle := []catalog.CatalogEntry{entry("a", e("b:go", "a:go", 1)), entry("b", e("a:go", "b:go", 1))}
for _, c := range []struct {
name string
entries []catalog.CatalogEntry
outcome string
asked []string
notAsked []string
}{
{"a cycle G §270 breaks is placed", []catalog.CatalogEntry{
entry("mesh-sdk", e("mesh-build-go:files", "mesh-sdk:go", 3)),
entry("mesh-build-go", e("mesh-sdk:go", "mesh-build-go:go", 1))}, seat.PipelineOutcomeRunning, []string{"mesh-sdk"}, nil},
{"a cycle beside a free mover ends its own steps and the free one is asked",
append(slices.Clone(cycle), entry("c"), entry("d", e("a:go", "d:go", 1))), seat.PipelineOutcomeRunning, []string{"c"}, []string{"a", "b", "d"}},
{"a cycle alone ends the pipeline failed at once", cycle, seat.PipelineOutcomeFailed, nil, []string{"a", "b"}},
{"a mover waiting on a placed mover waits", []catalog.CatalogEntry{
entry("a", e("b:go", "a:go", 1)), entry("b")}, seat.PipelineOutcomeRunning, []string{"b"}, nil},
} {
entries := map[string]catalog.CatalogEntry{"mesh-build-go": base["mesh-build-go"]}
p := seat.Pipeline{Pipeline: "main.4", Branch: "main", Source: 4, Outcome: seat.PipelineOutcomeRunning}
for _, en := range c.entries {
entries[en.Module] = en
p.Movers = append(p.Movers, seat.Mover{Module: en.Module, Commit: "c", Fingerprint: "src2:" + en.Module})
}
s := &Starter{Name: "the-runner"}
asks, _, ended, notAsked := s.placeMovers(&p, func(string) (catalog.Registration, bool) { return reg, true }, entries, time.Unix(1, 0))
if p.Outcome != c.outcome || (c.outcome == seat.PipelineOutcomeFailed) != (len(ended) > 0) || (len(c.asked) > 0) != (len(asks) > 0) {
t.Errorf("%s: pipeline %s (%s), %d asks, %d ended; want %s", c.name, p.Outcome, p.Why, len(asks), len(ended), c.outcome)
}
said := map[int]bool{}
for _, i := range append(ended, notAsked...) {
said[i] = true
}
var asked, ends []string
for i, st := range p.Steps {
switch {
case st.Outcome == seat.StepResultOutcomeNotAsked && strings.HasPrefix(st.Why, "cycle: ") && said[i]:
if !slices.Contains(ends, st.Module) {
ends = append(ends, st.Module)
}
if st.Why != "cycle: a:go before b:go (kind 1); b:go before a:go (kind 1)" {
t.Errorf("%s: %s's %s why %q, want the cycle's two edges alone", c.name, st.Module, st.Step, st.Why)
}
case open(st) || st.Outcome == seat.StepResultOutcomeWaiting:
if !slices.Contains(asked, st.Module) {
asked = append(asked, st.Module)
}
}
}
if !slices.Equal(asked, c.asked) || !slices.Equal(ends, c.notAsked) {
t.Errorf("%s: open %v, ended not asked %v; want %v and %v", c.name, asked, ends, c.asked, c.notAsked)
}
}
}

Some files were not shown because too many files have changed in this diff Show More