Compare commits

..
Author SHA1 Message Date
mesh-tools 7ba0eddd92 Merge pull request 'Catalog answers from its newest trunk revision in memory (A-479, A-487, A-487b, A-487c)' (#251) from fix/catalog-answers-from-memory into main 2026-11-19 10:48:58 +00:00
jschoubben 37dd3dac13 Merge fix/a479-catalog-answers-in-time (ea106ea8) into the catalog-answers-from-memory branch
Builds: design 47 G A-487..A-487c, A-479

# Conflicts:
#	modules/mesh-delivery-indexer/cmd/mesh-delivery-indexer-tools/catalog.go
#	modules/mesh-delivery-indexer/cmd/mesh-delivery-indexer-tools/main.go
2026-11-19 01:24:39 +01:00
jschoubben 533a9e4ca9 Revision, module and provides answer from the held trunk revision; a failed re-read is said as behind
KeptTrunk also holds the newest revision and its entries, and the manifests the entries carry by digest.
The verb revision asked of the trunk's newest and the manifests the verbs module and provides read come
from it, with no read of the bucket or the manifests state; a candidate, a branch and an older number
still read the bucket. A failed re-read says the view is BEHIND and the next heard catalog-revision
reads again. The follower's state is named catalog-half.

Builds: design 47 G §421, lead answers A-487, A-487b, A-487c (03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md)
2026-11-19 01:19:32 +01:00
jschoubben efd8c7b34f Catalog half reads its own objects at the start and on the heard catalog-revision; no invoke of its own seat
KeptTrunk reads the newest trunk revision from catalog-revisions once at the start (a follower of the
bundle's views.Reader, adopted on the seat's event catalog-revision) and re-reads once for each heard
event of a newer trunk revision; reads list nothing. The invoke seat:mesh-catalog.revision and the
meshcatalogclient follower are removed; the consume stays.

Builds: design 47 G §421, lead answers A-487 and A-487b (03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md)
2026-11-19 01:04:33 +01:00
jschoubben 80cfaa175f Catalog answers from the newest trunk revision held in memory, and module reads one module
The tools bundle's view of the trunk (KeptTrunk) is made from the half of the catalog its own follower
of the seat holds (meshcatalogclient.WireCatalog on a views.Reader: the newest revision once at the
start, again on the seat's event catalog-revision), once per revision, and says each read with its
revision, module count and time. A read no longer lists catalog-revisions or fetches the newest
revision object. The verb module reads the one module it is asked (CatalogRead.Module). The manifest
consumes mesh-catalog.catalog-revision and invokes seat:mesh-catalog.revision for the follower.

Builds: design 47 G §421, lead answer A-487 (03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md)
2026-11-19 01:02:43 +01:00
mesh-tools 16b795af70 Merge pull request 'Pipeline runner: a tool for every seat verb no tool reaches (A-486)' (#250) from fix/runner-verb-tools into main 2026-11-18 21:37:19 +00:00
jschoubben 59ca36e55b G1: tools runner_start, runner_run, runner_accept, runner_ask_again over the runner's seat verbs
Each tool asks its served verb with the same arguments and answers its answer; the verb records the hand act, as for runner_rerun. Declared in the manifest.
The seat has no verb release: no tool for it.
Lead answer A-486 (G-22-lead-answers-2.md): tools are how a module is spoken to.

Builds: design 47 G §83, §84, A-486
2026-11-18 02:22:54 +01:00
mesh-tools 09be0bc27f Merge pull request 'Judge: a process's own secrets belong to the account it runs as (ADR 0680)' (#249) from fix/adr0680-credential-owner into main 2026-11-18 01:18:50 +00:00
jschoubben 3a47cccbee Judge: a process's own secrets belong to the account it runs as, whatever that account (ADR 0680)
The refusals of root and the operator account as an own secret's owner and as a bus process's account end; the
secrets-owner is held to the accounts an own secret's owner is; a process naming an own secret (a word equal to its
path) it runs neither as the owner of nor as root is refused, container modules excepted (lead answer A-485).
Restic's captured manifest is re-captured from mesh-catalog cd3b8fe8 by CAPTURED's recipe.

Builds: ADR 0680 §2, §3, §4; lead answer A-485
2026-11-18 02:11:47 +01:00
jschoubben ea106ea8b9 A-479: the seat mesh-catalog's verbs answer from memory, one read of the trunk view per call
The tools bundle's verbs module, modules, provides and dependents read the modules and the module graph key by key from
the newest trunk revision, and each key listed the whole catalog-revisions bucket again (6 s per call at 11 000 objects,
35 s with 12 callers); seat-definitions read its state once per seat. The view is read once per call (SourceValues), the
seat definitions come from a watched copy, a manifest is read once by its digest.

Builds: design 47 G (the seat mesh-catalog's verbs served by the indexer's tools bundle; ADR 0604 §2, ADR 0580 clause 1)
2026-11-17 22:43:08 +01:00
jschoubben d006132c2a A-476 (b): the watchman and the view no longer name an indexer build a revision waits for
A pending revision now holds a lost or unread message or a refusal; the watchman's finding and the view's wait say so, the take-in
event no longer requires WaitsFor (old values still decode), and the hand take-in comment loses the build.

Builds: design 47 G A-476 (b)
2026-11-17 20:37:32 +01:00
jschoubben 763d6a107a A-476, principle 4: indexer_take_in_pending takes a pending revision in again, and a group
A revision whose message is unread or lost, or held for room, has a hand recovery; its description says nothing of an indexer build.

Builds: design 47 G A-476
2026-11-17 20:34:49 +01:00
jschoubben a79b811113 A-476 r1: no merge waits for the build the indexer runs; leftovers of the validation and the hold go; a commit's scope is decided inside the recompute
ErrOwnBuildUnknown, its say and the own-build-unknown wait, TakeIn.Unknown and Failed go; the build only names the writer.
indexer_take_in_pending takes only a group in. The node-build-go accept.check and output invokes go. A module of another
repository stays as held inside of(), so a base of this repository's module is not given a value that is never written
(A-476 c, A-476a); one case in the toolchains test proves it.

Builds: design 47 G A-476, A-476a
2026-11-17 20:33:52 +01:00
jschoubben 40b6ced375 A-476a: a commit moves its own repository's modules and the importers of what it moved; A-471 toolchain test data
The recompute of a merge leaves a module of another repository as held unless its imported package versions changed
(ADR 0593 §3 stands). TestIndexerCountsOnlyThePlacedBuildNodesToolchains reports language versions (A-471).

Builds: design 47 G A-476a, A-471
2026-11-17 20:22:55 +01:00
jschoubben 9768d0b67d A-476: the take-in asks no validation, holds for no indexer build, and moves only its repository's modules
(a) validation_wait.go and validate.go go with the validation ask, its end consumer, indexer_validate_again and
Pending.Validation; (b) evaluate always takes the revision in with the build the program runs (holdFor and the build
comparison go); (c) a merge's recompute writes only the modules registered for its repository.

Builds: design 47 G A-476 (a), (b), (c)
2026-11-17 20:20:45 +01:00
mesh-tools 0adcfbbbdb Merge pull request 'A-471: a build toolchain is agreed by its language version, not its patch' (#248) from fix/a471-toolchain-language-version into main 2026-11-17 14:56:46 +00:00
jschoubben 109a3e1e49 A-471: buildrun and refusal texts name the language version
buildrun tests expect the agreed toolchain as ToolchainAgreement gives it; the guard's
refusal and a decline name the language version alone.

Builds: design 47 G A-471 (G-22-lead-answers-2.md)
2026-11-17 15:53:05 +01:00
jschoubben a66583236a A-471: a toolchain is agreed, reported and named by its language version alone
A build node whose installed patch differs from the ask's takes the ask: one shared
comparison in mesh-delivery-sdk buildstep (LanguageVersion, SameToolchainVersion),
used by the agreement (indexer, builder, checker), a build node's step guard and ask
check, and the builder's digest list. A fingerprint names the language version, so
every module's fingerprint changes once.

Builds: design 47 G A-471 (G-22-lead-answers-2.md)
2026-11-17 15:53:05 +01:00
mesh-tools 80a7061efa Merge pull request 'G-24 tests on main: the assigner hears presence once; grants composes each module's consumer from its own subjects' (#247) from fix/g24-tests-on-main into main 2026-11-17 14:35:31 +00:00
jschoubben 10d87dcc77 mesh-grants: a module's consumer is composed from its own subjects, not from its account's wide grant
ConsumerFor drew its filters from PermissionsFor, which since every account may subscribe to every subject (e1476cfd) is ">": no filter
named an event, so no module got an account consumer on EVENTS. The scoped permissions give them again. The integration subtests that
read each account's own subjects run with the wide grant off, as the unit tests do.

Builds: design 47 G A-434 (G-22-lead-answers-2.md: every account may do everything, the mechanisms stay whole), the grants part's account consumers (ADR 0561 §6)
2026-11-17 15:31:00 +01:00
mesh-tools 1cd984a4c7 Merge pull request 'A-473: the builder asks the tagged publish of a reserved version when its build ends' (#246) from fix/a473-builder-publishes-at-build-end into main 2026-11-17 14:09:34 +00:00
jschoubben b31924a2f5 A-473: the builder asks the tagged publish of a reserved version when its build ends
Reserved build ends (built-at set, every digest known) now queue that version at each
artifact's registry seat holder and drain it, as a holder's running does.

Builds: design 47 G A-473 (design 51 Publishing, A-417)
2026-11-17 15:06:38 +01:00
mesh-tools 7654838182 Merge pull request 'A-472: a pipeline's end waits on no other part' (#245) from fix/a472-runner-end-waits-on-nothing into main 2026-11-17 14:00:25 +00:00
jschoubben 3f51d83147 Runner: widens-read-bound and revisions-kept are the module's settings (A-472a)
Declared in module.json's config and written to state files as test-step-bound is; runner_cancel keeps its note.

Builds: design 47 G A-472a
2026-11-17 14:59:25 +01:00
jschoubben 0be2b3d01a assigner: the watchman's machine-away and machine-back have one hearer, the world's, so a durable on their streams no longer splits them
The composition step's own hub handler for machine-away and machine-back shared the durable of hearDelivery on the kept streams, so
each event went to one of the two and a machine away was often not known away to the delivery judge. The step now asks the world.

Builds: design 47 G §389, §64
2026-11-17 14:57:12 +01:00
jschoubben 871f75acd3 Runner: a pipeline's end waits on no other part (A-472)
The runner keeps each catalog revision it read, bounds every read widens makes to 5 s and says each read's duration, failure or cut in the journal and in could-not-run; runner_cancel reads the pipeline after an ask that did not answer.

Builds: design 47 G A-472
2026-11-17 14:55:27 +01:00
mesh-tools 4b75791d3d Merge pull request 'G-24: one composition run per candidate and machine, one test run per level, every reading step timed' (#244) from fix/47-G-shared-runs into main 2026-11-17 12:15:35 +00:00
jschoubben 4205f63786 Merge branch 'fix/47-G-shared-runs-watchman' into fix/47-G-shared-runs 2026-11-17 13:14:44 +01:00
jschoubben 2438f4a108 Merge branch 'fix/47-G-shared-runs-runner' into fix/47-G-shared-runs 2026-11-17 13:14:44 +01:00
jschoubben 9081966682 assigner: a case holding that a take-up says an end again under the open run's ask, a first end under the ask's id (A-470q)
Builds: design 47 G §719; A-470q.
2026-11-17 13:12:57 +01:00
jschoubben 16cb874ccc watchman: the integration test proves the end of a run's ask begins a test step's clock again (W1)
Builds: design 47 G-24 §732, §733 (watchman case)
2026-11-17 13:10:25 +01:00
jschoubben 20d6a68bfb assigner: a reader that leaves a run not yet taken, or whose place a rerun takes, has its wait ended superseded at once (A-470u)
Builds: design 47 G §727, §723; A-469y, A-470u.
2026-11-17 13:09:18 +01:00
jschoubben 21bef6c044 assigner: a drop ends each reader whose end is owed, once, and moves no reader (A-470r)
A carried reader that ended passed keeps its verdict; a reader marked rerun is ended by the drop alone, not again by the run's
move.

Builds: design 47 G §728; A-470r, A-470m.
2026-11-17 13:07:21 +01:00
jschoubben ae105b4631 runner: test 109 reads a step's end said again from EVENTS through a durable consumer
The runner keeps no message id on its step events (A-470t), so the bus's deduplication drops no end
said again (A-470q); test 109 now reads the tools' unit step-ended from the stream through a durable
consumer, its first end failed and the end said again by the move passed, both kept.

Builds: design 47 G §727; A-470d, A-470q, A-470t
2026-11-17 13:06:46 +01:00
jschoubben b58a8b2e7f assigner: an end said again is published under a message id naming the run that makes it say again, so the bus never drops it (A-470q)
A move names the moved run's ask, a take-up of a reader that ended the open run's, a drop the dropped run's; a first end keeps
the ask's id. Test 108 reads the ends through a durable consumer, as the runner hears them.

Builds: design 47 G §727; A-470d, A-470q.
2026-11-17 13:06:36 +01:00
jschoubben 533a076eb3 runner: test 109 asserts a queued ask has no bound in the state bounds
The case of a test run waiting with its holder present was asserted just after placing, so a
five-minute bound put back on the ask failed no test; test 109 now reads the state bounds after the
asks are put and holds only drops' keys there.

Builds: design 47 G §647 (as A-466y amends it), §535
2026-11-17 13:05:11 +01:00
jschoubben 4e33533bd0 assigner: a case holding that an end names its run's ask, so a replaced run's late end ends nothing (A-470n)
Builds: design 47 G §719; A-470n.
2026-11-17 13:04:19 +01:00
jschoubben 1f344bd47c assigner: a step asked again that joins a run not yet taken says its wait held, as join does (A-469y)
Builds: design 47 G §727; A-469y.
2026-11-17 13:04:02 +01:00
jschoubben 5b8fa23a8d watchman: the integration test seeds steps through the runner's stands verb; its findings name the runner's verbs (W1)
Builds: design 47 G-24 §731, §732, §733 (watchman case); A-469p
2026-11-17 13:03:57 +01:00
jschoubben ed78003b23 assigner: a composition reader names when its step said its end, so a take-up says no end again for a carried reader that passed (A-470l, A-470m)
The step's end, once said, is written on its reader in each run of its candidate naming it, by compare-and-set; a rerun's run
carries it with the reader; the take-up at the start skips a reader that ended and is not marked rerun.

Builds: design 47 G §719, §727; A-470l, A-470m.
2026-11-17 12:50:38 +01:00
jschoubben 20e966b9cd contracts: a composition reader names when its step ended, so a take-up tells a reader that passed from one that joined later (A-470m)
Builds: design 47 G §719; A-470l, A-470m. Generated by seatgen from the mesh-wants definition.
2026-11-17 12:47:39 +01:00
jschoubben 7697ef5988 assigner: a run's bound holds its ask, armed by compare-and-set, and its end comes before the input's delete (A-470n, A-470o, A-470p)
A bound was armed by a delete and a create with no revision, holding nothing, so a late ask-taken or
step-progress of a replaced run armed a clock that later ended the run asked next, and an expiry heard
after the bound was armed again ended a run that kept saying progress. A bound key now holds its run's
ask, published at the key's latest revision with its own time to live, and is deleted at its own
revision when the run read again no longer names that ask; a late signal moves nothing and is
journaled; an expiry ends the run only while its bound is still absent when read again and the run is
taken (bound.) or untaken (untaken.), the untaken expiry's readers said superseded as the key is gone.
A step-ended's run end is written before the kept input is deleted; a failed delete asks the event
again, whose redelivery finds the run ended and still deletes the input.

Builds: design 47 G §720, §719; A-470n, A-470o, A-470p
2026-11-17 12:39:12 +01:00
jschoubben 768213b816 Merge branch 'fix/47-G-shared-runs-watchman' into fix/47-G-shared-runs 2026-11-17 12:30:47 +01:00
jschoubben f9dd20e86d Merge branch 'fix/47-G-shared-runs-runner' into fix/47-G-shared-runs 2026-11-17 12:30:47 +01:00
jschoubben a96f873e21 runner: test 109, the test runs whole against the real bus
The runner whole against an embedded bus: one unit and one integration run per level read by both
artifacts, a failing run counted once, a rerun moving the other reader and not the accepted one, a
restart's take-up, progress past the bound, a replaced reader, a silent ask ending could not run with
its readers and its ask dropped, a second revision's own runs, a seat with no holder failing its
pipeline at once with its other steps not asked, test-runs and test-runs-drop over the bus, and the cap.
The finding writer's stream is hosted, as on the mesh.

Builds: design 47 G §733, §724, §725, §726, §727, §728, §535, §647; A-468f, A-470f, A-470g, A-470h, A-470i, A-470n
2026-11-17 12:30:19 +01:00
jschoubben ab9d9604b4 runner: a run's bound holds its ask, armed by compare-and-set, and its end is written before the step's finding (A-470n, A-470o)
A bound was armed by a delete and a create with no revision, so a late ask-taken or step-progress of a
replaced run could arm a clock for the run asked next, an expiry heard after the bound was armed again
ended the run, and a step-ended's run end waited behind its finding's event (3 s in test 109, past its
bound). A bound key now holds the run's ask and is published at the key's latest revision with its own
time to live, read against the run after it is written and deleted at its own revision when the run no
longer names that ask; a late signal moves nothing and is journaled; an expiry ends the run only while
its bound is still absent when read again; the run's end is written as its step-ended is heard, before
the finding.

Builds: design 47 G §720, §725, §726; A-470n, A-470o
2026-11-17 12:28:26 +01:00
jschoubben 2a90323af3 runner: no write of a test run acts on a read made before its latest change
The age and test-runs-drop deleted a run's key whatever was written after their read, an expiry or the
last reader's leave ended whatever run the key held by then, a rerun answered a write between its read
and its write as an error, and a step-progress heard as its run ended wrote a clock that later ended
the run asked next. A delete of test-runs is now a compare-and-set at the revision read, retried from
the latest read; a run's end ends only the run under the ask its caller judged; the last reader ends
the run in its own leave; a rerun writes its new run from the latest read; a clock is read against the
run after it is written, and a clock of a run not taken ends nothing.

Builds: design 47 G §725, §726, §727, §728; A-469l, A-469z, A-470d
2026-11-17 12:23:40 +01:00
jschoubben 00e1588ef9 assigner: no delete of composition-runs lands on a value newer than the one it judged
Every value write of the state was already a compare-and-set from the latest revision, retried on
conflict; the deletes were not. The sweep and the drop deleted a run's key whatever it held, so a
rerun's run replacing the key between the read and the delete was lost; the end of a run, the
ask-taken and a rerun deleted bound keys a newer run of the key may have armed. Each delete is now
at the revision just read, while the key holds the run it was judged from, judged again on a
conflict (the kind the runner's test 109 met in test-runs).

Builds: design 47 G §719 (readers added and removed by compare-and-set; the key deleted after composition-runs-kept), §720 (the bound keys)
2026-11-17 12:23:38 +01:00
jschoubben a439b8da47 runner: a carried reader with rerun false says no step-ended again at a take-up (A-470l)
The runner's take-up of a rerun's test run already ends only the readers marked rerun and the open
ones (endReader); this table case holds it there, so a reader that passed is never said again.

Builds: design 47 G §725 (take-up), §727; A-470d, A-470l
2026-11-17 12:16:01 +01:00
jschoubben a2f4e8b499 assigner: the running builds whose manifest is not read are said once per change of their set
The assigner said the same list of about 100 modules every two or three seconds, once per
pass (live on the stable node since 2026-11-17 11:02 CET, the second meeting); it now says
the list once each time it changes. Why their manifests cannot be read is the builder's
stream of build ends, which holds no end of a build made before 2026-11-11, said apart.

Builds: design 47 G G-23 (the assigner's long line, met a second time)
2026-11-17 12:11:38 +01:00
jschoubben 8e95c2486f assigner: a composition run's request names the revision's modules with no fingerprint (A-470j)
A module whose build can never be made left its machine not judged, so every step of the
candidate could not run; the case now carries none, the revision's words for each module it
gives no fingerprint, for the composer to blame that module alone.

Builds: design 47 G §721, §722; A-470j, A-470k
2026-11-17 12:08:44 +01:00
jschoubben 1878607208 assigner: test 108, the composition runs started whole against the bus
One test of the whole assigner: one run per candidate and machine read by every step, a
module's fault failing only its own steps, the bound of owed progress, the untaken ask, a
step-ended after the bound ignored, a rerun moving the reader that did not pass, a
superseded step leaving its run, a restart taking a run up by its ask, the drop verb and
the cap.

Builds: design 47 G §733 (test 108); A-470f
2026-11-17 12:07:56 +01:00
jschoubben 930a407a3a assigner: a composition run is found by the keys its state's watch handed, beside the listing
A bucket's listing misses a key that is being written at that moment (97 of 300 listings on
an embedded bus while one key was updated), so a run's end, its ask-taken or a moved reader
could miss the run being written. Every lookup of composition-runs now takes the keys the
state's own watch handed beside the listing.

Builds: design 47 G §719 (the assigner's own watch of its state), §720, §727; A-468, A-470d
2026-11-17 12:07:56 +01:00
jschoubben eadfcd4cf8 composer: a case names the modules the revision gives no fingerprint, with its none words (G-24 H3)
The assigner fills it from the revision; the node-engine's composer reads it alone to tell a build that can
never be made, and judges the machine with that module held instead of leaving it not judged.

Builds: design 47 G §721, §722, A-470j, A-470k
2026-11-17 12:01:12 +01:00
jschoubben 19fbc076d6 runner: test-runs is answered by the process, its fill said against test-runs-cap-mib (A-470g, A-470i)
The bundle said test-runs' fill against the bucket's 64 MiB ceiling, a cap that refuses nothing; the
process, which holds test-runs-cap-mib and judges it, now answers the verb test-runs, its fill's cap
being that setting, and runner_test_runs calls the verb.

Builds: design 47 G §728, §725 (the cap); A-469d, A-470g, A-470i
2026-11-17 12:00:35 +01:00
jschoubben 1a924f2ca7 runner: a seat with no holder ends its pipeline failed at once, its other steps not asked (A-470g, A-470h)
A pipeline with a step whose seat the seat-holders view names no holder for was failed only once its
other steps ended, after their asks ran for nothing; it now ends failed at placing, naming the seat,
with every other step not asked and said, and none of its steps asked.

Builds: design 47 G §32, §535, §647 (as A-466y and A-466z amend them); A-470g, A-470h
2026-11-17 11:58:25 +01:00
jschoubben ebe49b9bdd runner: a test run's steps are said waiting for a build node until the holder takes the run's ask (A-469b)
A test step queued on a build seat was said running, so the watchman's build-capacity waits never saw
it; every step of a test run now waits for a build node from the run's ask, and at the holder's
ask-taken each is said started again, running.

Builds: design 47 G §725, §732, §86; A-469b
2026-11-17 11:48:28 +01:00
jschoubben 7923ab63b1 runner: the test runs are read and dropped by the seat's verbs test-runs and test-runs-drop (R7)
A test run had no tool to read or recover it; test-runs answers each run with its packages from its
output object, or says the output gone, with the state's fill, and test-runs-drop, answered by the
process, records the hand act first, ends the readers naming the drop and deletes the key and its
object; runner_test_runs and runner_test_runs_drop call them.

Builds: design 47 G §728, §725 (the output object), §733 (unit); A-466v, A-469f, A-469g, A-469u, A-469v
2026-11-17 11:47:05 +01:00
jschoubben 09224d2cdf assigner: the composition runs are read and dropped through mesh-wants' verbs (A7)
A composition run must be seen and recoverable without a hand act on the bus: the seat
mesh-wants' verbs compositions and compositions-drop are answered by the process, the drop
recorded in the hand-act log first, its readers ended could not run naming it; the tools
assigner_compositions and assigner_compositions_drop call them.

Builds: design 47 G §728 (the assigner's two); A-469f, A-469u, A-469v
2026-11-17 11:46:29 +01:00
jschoubben 167a9f2818 assigner: a rerun of a composition step moves every reader that did not pass (A6)
Steps asked again together start one run per machine they did not pass: a rerun keeps the
verdict of a machine it passed, joins a running run, reads one it did not read, or starts
a new run carrying every former reader, those that did not pass marked rerun with the
caller; at the new run's end each marked reader says its end again, naming the new ask,
and a reader that passed does not move.

Builds: design 47 G §727 (composition part); A-468g, A-469l, A-470d, A-470e
2026-11-17 11:44:43 +01:00
jschoubben ad40520e89 runner: stands names a waiting step's ask and its run, and every step asked says step-started (R6)
The watchman could not time a step reading a run: stands named no ask, and step-started was said
only on rerun and ask-again. A waiting step now stands with its own ask and, for a test step, the
run's key and ask; each step asked at a start, a resume, a placing or a run says step-started,
naming its run in shared with no asker.

Builds: design 47 G §731, §729, §726 (step-started of a reading step); A-468e, A-469a, A-469q, A-469r, A-469v
2026-11-17 11:43:52 +01:00
jschoubben 2b3a9e4a6a runner: asking a test run again moves every reader that did not pass (R5)
A rerun of one artifact's failed test step left the other artifact's step failed on a result no
longer true; the rerun now joins, reads or starts the key's run, carrying the former readers
marked in the key, and the new run's end moves the marked ones, said again as a rerun by its
caller and naming the new ask, the pipeline reopened; an accepted step does not move.

Builds: design 47 G §727 (test part); A-469l, A-470d, A-470e
2026-11-17 11:42:09 +01:00
jschoubben 72ebb4deb6 runner: no holder is read from the seat-holders view, and an ask waits for its holder with no five-minute bound (R4)
A step queued for a present holder was ended failed after five minutes; the five-minute ask bound
goes, and a seat ends its steps could not run only when the seat mesh-module-assigner's view
seat-holders, followed from its first view on, names no holder for it.

Builds: design 47 G §535, §647 (as A-466y and A-466z amend them), §717 (the follower)
2026-11-17 11:38:22 +01:00
jschoubben d957a60286 assigner: a reading step names the runs it read, and a superseded step leaves them (A5)
A step's end and its progress name per machine the composition run and its ask, so a
result is traced to the one ask that made it; a superseded or cancelled pipeline's
composition step leaves the runs it reads, the run going on for its other readers, and a
run left with no reader drops its ask and ends could not run.

Builds: design 47 G §723, §729 (mesh-wants' step-ended and step-progress shared); A-469n, A-469t, A-469z
2026-11-17 11:38:10 +01:00
jschoubben ba11ff58dd assigner: a composition run's bound times owed progress, its untaken ask its own (A4)
A run whose build node stops saying progress, or whose ask no build node takes, would hold
its readers for ever: ask-taken arms bound.<run key> and each step-progress arms it again;
untaken.<run key> times the queue from the start, the readers said waiting for a build node
meanwhile; an expiry drops the ask and ends the run could not run naming the bound, and a
step-ended heard after it is ignored and journaled. Readers of a run left open at a restart
are taken up from the keys.

Builds: design 47 G §719 (take-up at start), §720 (the bound, when the clock ends, a run never taken, what the assigner hears); A-466q, A-466s, A-468, A-468b, A-469b, A-469s, A-469y, A-470
2026-11-17 11:36:15 +01:00
jschoubben 97f68b04ea runner: a test run's end moves every step reading it, counted once, dropped only with its last reader (R3)
The run's ask's end now ends each reading step with the run's outcome, counts and shared, its
output kept in test-runs-objects; a pipeline counts a run's tests once; a superseded or cancelled
step leaves the run, the last to leave dropping its ask and ending it; a composition step keeps
the runs mesh-wants names.

Builds: design 47 G §726, §725 (output object), §724 (an ended run read), §729; A-466i, A-468d, A-469m, A-469t, A-469z
2026-11-17 11:33:54 +01:00
jschoubben 61a11f8c15 assigner: a composition step judges each machine by the modules its verdict blames (A3)
One module's fault fails only its own pipelines: a machine blaming module A fails A's step
and passes B's with a warning naming A; a machine that blames none fails every step on it;
a composer before blamed is judged as before, saying it names no culprits; a module the
search did not try is could not run for its own step. Each request carries blame_tries.

Builds: design 47 G §721 (an old composer, its bound), §722; A-466r, A-468a, A-469h, A-470b
2026-11-17 11:32:19 +01:00
jschoubben cd08f1e1c2 runner: a test run is kept in the state test-runs with one clock, a cap and an age (R2)
A test run is now one key of test-runs: its untaken bound written at its start, its clock at
ask-taken and each step-progress, an expiry ending the run and every reader could not run and
dropping its ask, a run past its bound while the runner was away ended at its start, the cap
judged by the process, and an ended run aged with its output object in the step log's order.

Builds: design 47 G §725, §720 (When the clock ends, A run never taken); A-466n, A-466p, A-466s, A-468b, A-468c, A-469d, A-469e, A-469w, A-469x, A-470, A-470a
2026-11-17 11:31:26 +01:00
jschoubben afa95fd0df assigner: one composition run per candidate and machine, every step reading it (A2)
The steps of one candidate share a machine's run, so a revision of 24 pipelines composes
each machine once: the step that starts a run asks the node's input, keeps one request,
chooses and records the composer program, and puts one run ask; every other step reads
the run's verdict from its key, its end written there from the build seat's step-ended.

Builds: design 47 G §718, §719, §720 (within its candidate only, could not run is never read); A-468, A-468a
2026-11-17 11:31:05 +01:00
jschoubben ef5e5d13e0 assigner: the state composition-runs, one key per candidate and machine (A1)
A composition run is kept once per candidate and machine, so steps of one candidate share
it: the key is written create-only, a reader is the step's StepRun with its ask id, a run
that could not run is replaced, a run past composition-runs-kept goes at the next start,
and at the cap no run starts, naming the fill. The settings reach the process through
resource files.

Builds: design 47 G §718, §719, §720 (could not run is never read); A-468, A-468b, A-468c, A-469d, A-469w, A-469x, A-470
2026-11-17 11:26:51 +01:00
jschoubben 5388bbce43 runner: a test step is asked once per candidate, level and share digest, the others read it (R1)
Two artifacts of one directory ran the same tests twice and failed them twice; the first step of
a test-runs key now asks the run, written create-only, and every other step of the candidate reads it.

Builds: design 47 G §724, §725 (the key), §730; A-468k, A-469c, A-469m, A-469q
2026-11-17 11:22:55 +01:00
jschoubben 185fe0ae1d watchman: times every step that reads a run, seeded from the runner's stands (W1)
The watchman keeps each started step under <pipeline>/<step>/<artifact>, begins
its clock at step-started and at each mesh-wants step-progress, build seat
step-progress or step-ended of the step's ask or an ask its shared names, raises
pipeline-stands past five minutes, clears it at the step's step-ended, owes none
to a step on a person or queued for a build node, and seeds from stands at start.

Builds: design 47 G-24 §731, §732, §733 (watchman case); A-468j, A-469, A-469o, A-469p, A-469t, A-469r
2026-11-17 11:20:22 +01:00
jschoubben 8fa1b8aee7 contracts: shared on a step's end and progress, ask and shared on stands, the four run verbs (C1)
A step that read a composition run or a test run names it in shared, so the result is
traced to the one ask that made it; stands names what a reading step waits on; the
runs' read and drop are seat verbs with their class (seatgen regenerated, no hand edit).

Builds: design 47 G §728, §729, §731; A-468d, A-468e, A-469f, A-469q, A-469r, A-469t, A-469u, A-469v
2026-11-17 11:15:55 +01:00
jschoubben c17ccd3070 composer: a request carries blame_tries, a machine's verdict names the modules it blames (K1)
The assigner bounds the composer's culprit search through the request, since the composer program has no settings; a
verdict always writes blamed, an empty list when it blames none, so an old composer's absent field stays tellable.

Builds: design 47 G §721, §729; A-469h, A-469i
2026-11-17 11:10:46 +01:00
jschoubben 9849c49920 assigner: the key form of the state inputs is the assigner's own; an ask's key holds digest and time (A-465b)
The state inputs leaves the seat mesh-wants' definition (seatgen regenerated, no hand
edit). An ask's key holds {digest, at}; the sweep deletes an ask's key a day after its
time and an input with its object once no ask's key names it; keys of the old form
<time>.<digest> go by the time they carry, never earlier.

Builds: design 47 G A-465a, A-465b
2026-11-17 11:05:12 +01:00
jschoubben 4cdf14680e assigner: a composition input is kept once per content, by its sha256 (A-465a)
The key of the state inputs is the digest alone (seat mesh-wants' key form <digest>);
each ask's name holds the digest, the input and its object end with the last ask that
names it or a day after its last write (at.<digest>); the contract files of the seat are
edited by hand (generator not found).

Builds: design 47 G A-465a
2026-11-17 11:05:12 +01:00
mesh-tools bf5f794553 Merge pull request 'assigner: a composition step with no machine judged never passes (A-467a)' (#243) from fix/47-G-composition-none-judged into main 2026-11-17 09:49:30 +00:00
jschoubben 34da4fe001 assigner: a composition step with no machine judged ends could not run (A-467a)
A step whose every machine is away never passes: it ends could not run, naming each machine and why. An away machine is still
not asked and still said as a warning when another machine is judged.

Builds: design 47 G A-467a, A-467
2026-11-17 10:49:05 +01:00
mesh-tools 2406bb4000 Merge pull request 'assigner: a composition step never passes on a machine it did not judge (A-467)' (#242) from fix/47-G-composition-judges into main 2026-11-17 09:47:26 +00:00
jschoubben db62653472 assigner: a composition step never passes on a machine it did not judge, and hands the composer the trunk seat list (A-467)
A machine the composer says `not judged` ends the step could not run, naming why (was a warning). A revision that names no seat
definition (A-234's stand-in) hands the composer every definition of the trunk seat list, read through mesh-catalog's
seat-definitions; a list or definition not read ends the step could not run, naming it (was an empty list, no error). The
integration bed makes the inputs object store, which a request carrying the seat list needs.

Builds: design 47 G A-467, A-234, A-252, A-261
2026-11-17 10:45:20 +01:00
mesh-tools 6bb6452603 Merge pull request 'pipeline runner: a step-log read never waits for a batch it cannot fill' (#241) from fix/47-G-runner-ends into main 2026-11-17 09:40:11 +00:00
jschoubben 8d93c3ff9f Pipeline runner: the ageing reads the step log without waiting for a batch the log cannot fill
Every step's end waited 2 s per retention (6 s) in the step log's Fetch when fewer than 20 entries lay past the aged pointer, so ends were taken one in 6 s serially.

Builds: design 47 G §330 (ageing of the runner's own data at each step's end)
2026-11-17 10:38:13 +01:00
mesh-tools f93e8b9750 Merge pull request 'indexer test: seed a want and a running build before an entry may be removed' (#240) from fix/47-G-integration-tests into main 2026-11-17 08:44:59 +00:00
jschoubben 9a8609f770 Indexer test: seed a want and a running build, since no entry is removed before both streams read non-empty
Builds: design 47 G §675 (kept while named)
2026-11-17 09:43:05 +01:00
mesh-tools 39162a4435 Merge pull request 'assigner: composition inputs end with their objects; assigner_inputs_drop (A-465)' (#239) from fix/47-G-inputs-objects into main 2026-11-17 08:23:36 +00:00
jschoubben 1ccde6fc3e assigner: assigner_inputs says when the key-gone objects cannot be listed; the start's drop is asserted (A-465)
Builds: design 47 G-22 A-465
2026-11-17 09:22:14 +01:00
jschoubben 45f7bf8e01 assigner: an input the assigner ends ends its object too; the objects of deleted keys go at the start and by assigner_inputs_drop (A-465)
Builds: design 47 G-22 A-465
2026-11-17 09:02:34 +01:00
mesh-tools eed0833a29 Merge pull request 'indexer: take in a merge it never took in, and read takeins (A-463, A-463a, A-464a)' (#238) from fix/47-G-untaken-merge into main 2026-11-17 01:13:06 +00:00
jschoubben 223b096295 indexer: a head the pending holds is not behind; a revision heard during a catch-up is compared after it (A-463a)
Builds: design 47 G-22 A-463a, A-463
2026-11-17 02:10:17 +01:00
jschoubben 1987a3915e indexer: correct the comment on movedPackages, which A-464 made untrue
Builds: design 47 G-22 A-464
2026-11-17 02:02:12 +01:00
jschoubben 250a0f1b12 indexer: take in a merge it never took in, and say so while it is behind (A-463)
At its start and on every source revision the indexer compares each repository's taken merge with the head the gate's source names; a repository behind is taken in as its merged message would have been, with the finding merge-untaken raised until it is.

Builds: design 47 G-22 A-463
2026-11-17 01:55:46 +01:00
jschoubben 61843f823d indexer: the tool indexer_read_takeins reads the state takeins (A-464a)
Builds: design 47 G-22 A-464a
2026-11-17 01:55:46 +01:00
mesh-tools 44bdbbcbed Merge pull request 'assigner: a backup ask only for a failed backup (A-453, A-453a)' (#237) from fix/47-G-backup-ask-only-for-failed-backup into main 2026-11-16 23:51:25 +00:00
jschoubben 482a618d1e assigner: a held have names the build it keeps running; only a have not held at another fingerprint is past the hold (A-453a)
Builds: design 47 G A-453, A-453a, ADR 0539 §16
2026-11-17 00:49:26 +01:00
jschoubben 1a187445a5 assigner: a backup ask stands while its have is mid-move; it goes only past the hold (A-453a)
Builds: design 47 G A-453, A-453a, ADR 0539 §16
2026-11-17 00:46:47 +01:00
jschoubben 8ab75bb727 assigner: the backup question is asked only for a held have with no wait (A-453)
A held have that carries a wait is a hold of the composition, not a failed backup; a backup ask whose have now carries a wait or is no longer held is withdrawn with its condition.

Builds: design 47 G A-453, ADR 0539 §16, §541
2026-11-17 00:41:00 +01:00
mesh-tools 8c4489fb3d Merge pull request 'assigner: its calls wait 3 minutes, so a node composing a candidate is answered' (#236) from fix/assigner-waits-for-composing into main 2026-11-16 22:41:19 +00:00
jschoubben a2bcb23f0e assigner: its calls wait 3 minutes, so a node composing a candidate is answered
compose-candidate composes a whole revision in shadow on each node and
answered in about 34 s on g14 and shanks (2026-11-16 23:27), past the
SDK's 30 s call bound, so node-tools' composition step could not run. A
responder that is absent still answers at once. Quick fix; the thorough
one is a bound per verb from the seat's definition (after G).
2026-11-16 23:41:01 +01:00
mesh-tools 43050bf20a Merge pull request 'Pipeline runner tools: runner_rerun, the runner's rerun is reachable as a tool' (#235) from feat/runner-rerun-tool into main 2026-11-16 22:34:16 +00:00
jschoubben 8710148ce0 Pipeline runner tools: runner_rerun, the runner's rerun is reachable as a tool
main.981.node-tools passed build, lint, unit and integration; only its
composition step could not run (g14 and shanks composed in about 34 s,
past the assigner's 30 s, while shanks ran the pipeline's own tests).
The seat verb rerun asks one step again, but no tool reached it.

Builds: design 47 G §482
2026-11-16 23:33:57 +01:00
mesh-tools 6618925841 Merge pull request 'Hand acts name their cause: the runner's write verbs and the checker's requests drop' (#234) from fix/hand-acts-name-their-cause into main 2026-11-16 22:13:04 +00:00
jschoubben 1eb090d407 hand acts name their cause: the runner's write verbs and the checker's requests drop
mesh-hand-acts 0.1.13 refuses a hand act with no cause; the pipeline
runner's record (cancel, rerun, accept, run, ask-again, review, start) and
the checker tools' requests drop gave none, so every one was refused and
nothing was done (runner_cancel, 2026-11-16 23:12: 96 of 96 refused). The
cause is the verb's own word, the one a second act for the same reason uses.
2026-11-16 23:12:51 +01:00
mesh-tools f373cf4ae6 Merge pull request 'Pipeline runner tools: runner_cancel, the runner's cancel is reachable as a tool' (#233) from fix/pipeline-runner-ends-stale into main 2026-11-16 22:05:02 +00:00
jschoubben e040cc5fc2 Pipeline runner tools: runner_cancel, the runner's cancel is reachable as a tool
Builds: design 47 G §482 (cancel)
2026-11-16 23:04:07 +01:00
mesh-tools 38f1c2f10a Merge pull request 'assigner: a composition step reads its catalog revision once and again on a timeout' (#232) from fix/composition-reads-revision-once into main 2026-11-16 20:32:07 +00:00
jschoubben bece2421f1 assigner: a composition step reads its catalog revision once and again on a timeout
Builds: design 47 G §354, A-252, A-261
2026-11-16 21:31:34 +01:00
187 changed files with 13114 additions and 3646 deletions
@@ -0,0 +1,375 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// A composition run's bound (design 47 G §720; A-466n, A-466p, A-466q, A-466s, A-468b, A-469b, A-469s, A-469y, A-470): a run
// has one clock, the key `bound.<run key>` of composition-runs with its own time to live composition-step-bound, written at the
// build seat's ask-taken of the run's ask and written again at each step-progress of that ask; an ask still waiting in the queue
// has no clock but `untaken.<run key>`, written at the run's start with the time to live run-untaken-bound. Either's expiry,
// heard as the bus's event, ends the run: its ask dropped through the build seat's drop, the run ended could not run naming the
// ask and the bound, so every reader ends could not run with it. While the ask waits for a build node each reader is said waiting
// by a wait-held on mesh-wants (`on: build node`), ended `ran` at ask-taken or `superseded` when the run ends untaken.
// waitOnBuildNode is what a composition run's waiting reader waits on (A-469b, A-469y).
const waitOnBuildNode = "build node"
// dropVerbKey is the Go build seat's drop, asked as a kept call (§119, A-182).
const dropVerbKey = "seat:" + nodebuildgo.Seat + "." + nodebuildgo.VerbDrop
// dropOver asks the Go build seat's drop of a run ask as a kept call, of any holder, which records it (A-469s).
func dropOver(call func(key string, body any) (json.RawMessage, error)) func(ask, why string) error {
return func(ask, why string) error {
raw, err := call(dropVerbKey, nodebuildgo.DropInput{ID: ask, Why: why, By: runAsker})
if err != nil {
return err
}
if _, said, ok := buscalls.Unwrap(raw); !ok {
return fmt.Errorf("%s answered: %s", dropVerbKey, said)
}
return nil
}
}
// waitOf is a reader's wait for a build node to take its run's ask (A-469y).
func waitOf(run meshwants.CompositionRun, reader meshwants.CompositionReader) meshwants.Wait {
w := meshwants.Wait{Step: "pipeline", Owner: meshwants.Seat, Module: reader.Module, Node: run.Machine, On: waitOnBuildNode, Fp: reader.Fingerprint,
Words: fmt.Sprintf("the composition step of %s waits for a build node to take the composition run %s of %s (ask %s)", reader.Pipeline,
runKeyOf(run.Candidate, run.Machine), run.Machine, run.Ask)}
if run.Started != nil {
w.Since = *run.Started
}
return w
}
// sayWaits says, for each reader, its wait held or ended (how: ran, superseded); a wait that cannot be said is journaled.
func (r *compositionRuns) sayWaits(run meshwants.CompositionRun, readers []meshwants.CompositionReader, how string) {
if r.say == nil {
return
}
for _, c := range readers {
w := waitOf(run, c)
event, id := meshwants.EventWaitHeld, "composition."+runKeyOf(run.Candidate, run.Machine)+"."+run.Ask+"."+c.ID+".held"
if how != "" {
w.How, event, id = how, meshwants.EventWaitEnded, strings.TrimSuffix(id, ".held")+".ended"
}
if err := r.say(event, w, id); err != nil {
r.logf("the %s of %s on the composition run of %s is not said: %v", event, c.Pipeline, run.Machine, err)
}
}
}
// untaken is whether the run of ask has not been taken yet: its untaken key stands holding that ask (A-470n).
func (r *compositionRuns) untaken(key, ask string) bool {
held, _, standing, err := r.boundOf(runUntakenPrefix + key)
return err == nil && standing && held == ask
}
// boundOf reads a bound key: the ask it holds and its revision; standing false when it is not written.
func (r *compositionRuns) boundOf(boundKey string) (string, uint64, bool, error) {
raw, rev, err := r.kv.get(boundKey)
if err != nil || raw == nil {
return "", 0, false, err
}
var v boundValue
_ = json.Unmarshal(raw, &v)
return v.Ask, rev, true, nil
}
// armBound writes the bound key prefix+key holding ask, judged from a read of the run, then reads the run again (A-470n): when
// it no longer names ask, or has ended, the bound just written is deleted at its revision, so no bound written from the earlier
// read ends the run after it, and a bound another run wrote since is kept. It answers whether the bound stands for the run.
func (r *compositionRuns) armBound(prefix, key, ask string, ttl time.Duration) (bool, error) {
rev, err := r.kv.arm(prefix+key, ask, ttl)
if err != nil {
return false, err
}
run, _, ok, err := r.read(key)
if err != nil {
return false, err
}
if ok && run.Ended == nil && run.Ask == ask {
return true, nil
}
if err := r.kv.delAt(prefix+key, rev); err != nil && !errors.Is(err, errRunChanged) {
return false, err
}
return false, nil
}
// heardAskTaken begins a run's clock at its ask's ask-taken: the bound key armed holding the ask, the untaken key of that ask
// deleted, each reader's wait ended ran. An ask no running run names is another asker's, or a late signal of a run replaced or
// ended, which moves nothing and is journaled (A-470n).
func (r *compositionRuns) heardAskTaken(e nodebuildgo.AskTaken) error {
key, run, err := r.runOfAsk(e.Asker, e.Ask)
if err != nil {
return err
}
if key == "" {
r.late("ask-taken", e.Asker, e.Ask)
return nil
}
armed, err := r.armBound(runBoundPrefix, key, e.Ask, r.set.stepBound)
if err != nil {
return fmt.Errorf("the bound of the composition run %s is not armed: %w", key, err)
}
if !armed {
r.late("ask-taken", e.Asker, e.Ask)
return nil
}
if r.untaken(key, e.Ask) {
if err := r.clearBounds(key, e.Ask, runUntakenPrefix); err != nil {
return err
}
r.sayWaits(run, run.Readers, "ran")
}
return nil
}
// heardProgress begins a running run's clock again at each step-progress of its ask (§720: owed progress, not run length); a
// late one moves nothing and is journaled (A-470n).
func (r *compositionRuns) heardProgress(e nodebuildgo.StepProgress) error {
key, _, err := r.runOfAsk(e.Asker, e.Ask)
if err != nil {
return err
}
armed := false
if key != "" {
if armed, err = r.armBound(runBoundPrefix, key, e.Ask, r.set.stepBound); err != nil {
return err
}
}
if !armed {
r.late("step-progress", e.Asker, e.Ask)
}
return nil
}
// late journals a signal of an ask the assigner put whose run no key names running any more (A-470n).
func (r *compositionRuns) late(signal, asker, ask string) {
if asker == runAsker && ask != "" {
r.logf("the %s of the run ask %s is late: no composition run names it running, so it moves nothing (A-470n)", signal, ask)
}
}
// runOfAsk is the running run of an ask the assigner put, or none.
func (r *compositionRuns) runOfAsk(asker, ask string) (string, meshwants.CompositionRun, error) {
if asker != runAsker || ask == "" {
return "", meshwants.CompositionRun{}, nil
}
key, run, err := r.keyOfAsk(ask)
if err != nil || key == "" || run.Ended != nil {
return "", meshwants.CompositionRun{}, err
}
return key, run, nil
}
// expired ends the run whose bound key's time to live ended (§720): its ask dropped, the run ended could not run naming the
// ask and the bound. A run that ended already, or a key of none, takes nothing.
func (r *compositionRuns) expired(boundKey string) {
key, bound, setting := "", time.Duration(0), ""
switch {
case strings.HasPrefix(boundKey, runBoundPrefix):
key, bound, setting = strings.TrimPrefix(boundKey, runBoundPrefix), r.set.stepBound, "composition-step-bound"
case strings.HasPrefix(boundKey, runUntakenPrefix):
key, bound, setting = strings.TrimPrefix(boundKey, runUntakenPrefix), r.set.untaken, "run-untaken-bound"
default:
return
}
run, _, ok, err := r.read(key)
if err != nil || !ok || run.Ended != nil {
return
}
// The marker of an expiry holds no value (A-470o): the run ends only while its bound is still absent when read again, and
// the run is taken for bound., untaken for untaken., so an expiry heard after a bound was armed again, or of a run replaced
// since, ends nothing.
other := runUntakenPrefix
if setting == "run-untaken-bound" {
other = runBoundPrefix
}
_, _, armed, err := r.boundOf(boundKey)
if err != nil {
r.logf("the bound %s is not read again, so its expiry ends nothing: %v", boundKey, err)
return
}
held, _, standing, err := r.boundOf(other + key)
if err != nil {
r.logf("the bound %s is not read, so the expiry of %s ends nothing: %v", other+key, boundKey, err)
return
}
if armed || (standing && held == run.Ask) {
r.logf("the expiry of %s ends nothing: the bound is armed again, or the run of ask %s is not where it bounds (A-470o)", boundKey, run.Ask)
return
}
said := fmt.Sprintf("%s: its run ask %s said no progress within %s (%s)", run.Machine, run.Ask, bound, setting)
if setting == "run-untaken-bound" {
said = fmt.Sprintf("%s: its run ask %s was taken by no build node within %s (%s)", run.Machine, run.Ask, bound, setting)
}
if r.drop != nil {
if err := r.drop(run.Ask, said); err != nil {
r.logf("the drop of the run ask %s of the composition run %s is not answered: %v", run.Ask, key, err)
}
}
// The untaken key is gone with its expiry, so the run ended by it is said untaken here: its readers' waits end superseded.
if _, err := r.endRunOf(key, run.Ask, meshwants.CompositionRunOutcomeCouldNotRun, said, nil, setting == "run-untaken-bound"); err != nil {
r.logf("the composition run %s past its bound is not ended: %v", key, err)
}
}
// endRun ends a run (end) and clears its bounds: the untaken key's readers said superseded, both bound keys deleted.
func (r *compositionRuns) endRun(key, ask, outcome, said string, verdict map[string]any) (bool, error) {
return r.endRunOf(key, ask, outcome, said, verdict, r.untaken(key, ask))
}
// endRunOf is endRun of a run whose ask is untaken, or not, as its caller judged.
func (r *compositionRuns) endRunOf(key, ask, outcome, said string, verdict map[string]any, untaken bool) (bool, error) {
return r.endRunMoving(key, ask, outcome, said, verdict, untaken, true)
}
// endRunMoving is endRunOf, moving the readers a rerun carried or not: a drop says its readers' ends itself (A-470r).
func (r *compositionRuns) endRunMoving(key, ask, outcome, said string, verdict map[string]any, untaken, move bool) (bool, error) {
// The readers a rerun carried move with the run's end (§727, A-470d): chosen before the end is written, while the rerun's own
// step still waits on it, so it alone says its own end; each mover waits for the end in the key.
if moved := r.moved.Load(); moved != nil && move {
if run, _, ok, err := r.read(key); err == nil && ok && run.Ended == nil && run.Ask == ask {
(*moved)(run)
}
}
ended, err := r.end(key, ask, outcome, said, verdict)
if err != nil || !ended {
return ended, err
}
if untaken {
if run, _, ok, err := r.read(key); err == nil && ok {
r.sayWaits(run, run.Readers, "superseded")
}
}
if err := r.clearBounds(key, ask, runBoundPrefix, runUntakenPrefix); err != nil {
r.logf("the bound keys of the ended composition run %s are not deleted: %v", key, err)
}
return true, nil
}
// readersOfOpen are the readers of every run with no `ended`, each with its ask's id once: what the assigner takes up again at
// its start (§719), each reader's end said from the keys alone (A-468). A reader whose step said its end already, carried into a
// rerun's run, is taken up only when it is marked `rerun` (A-470l, A-470m); each with the ask of the open run it is taken up from.
func (r *compositionRuns) readersOfOpen() ([]openReader, error) {
all, err := r.keys()
if err != nil {
return nil, err
}
seen := map[string]bool{}
var out []openReader
for _, key := range all {
if !isRunKey(key) {
continue
}
run, _, ok, err := r.read(key)
if err != nil || !ok || run.Ended != nil {
continue
}
for _, c := range run.Readers {
if c.Ended != nil && !c.Rerun {
continue // it ended passed already: only a reader marked rerun says its end again
}
if !seen[c.ID] {
seen[c.ID] = true
out = append(out, openReader{CompositionReader: c, ask: run.Ask})
}
}
}
return out, nil
}
// openReader is a reader taken up at the start and the ask of the open run it is taken up from.
type openReader struct {
meshwants.CompositionReader
ask string
}
// keysOf are the keys of the candidate's runs that the reader reads.
func (r *compositionRuns) keysOf(candidate, id string) ([]string, error) {
all, err := r.keys()
if err != nil {
return nil, err
}
var out []string
prefix := keysPart(candidate) + "."
for _, key := range all {
if !isRunKey(key) || !strings.HasPrefix(key, prefix) {
continue
}
if run, _, ok, err := r.read(key); err == nil && ok && hasReader(run.Readers, id) {
out = append(out, key)
}
}
slices.Sort(out) // by machine, as a step names its machines
return out, nil
}
// hearRunBounds hears the build seats' ask-taken and step-progress on the assigner's durable consumer (§720, A-466q): each
// handed to the runs; one that cannot be written is given again by the bus.
func hearRunBounds(hub interface {
Handle(string, func(jetstream.Msg)) error
}, r *compositionRuns, logf func(string, ...any)) error {
if err := hub.Handle(nodebuildgo.SubjectAskTakenAll, func(m jetstream.Msg) {
var e nodebuildgo.AskTaken
if err := json.Unmarshal(m.Data(), &e); err != nil {
logf("%s holds what is no ask-taken, left: %v", m.Subject(), err)
_ = m.Ack()
return
}
ackOrNak(m, r.heardAskTaken(e), logf)
}); err != nil {
return err
}
return hub.Handle(nodebuildgo.SubjectStepProgressAll, func(m jetstream.Msg) {
var e nodebuildgo.StepProgress
if err := json.Unmarshal(m.Data(), &e); err != nil {
logf("%s holds what is no step-progress, left: %v", m.Subject(), err)
_ = m.Ack()
return
}
ackOrNak(m, r.heardProgress(e), logf)
})
}
func ackOrNak(m jetstream.Msg, err error, logf func(string, ...any)) {
if err != nil {
logf("%s is not taken, so the bus gives it again: %v", m.Subject(), err)
_ = m.Nak()
return
}
_ = m.Ack()
}
// takeUp says, at the assigner's start, the end of each reader of a run with no `ended` once its runs end (§719, A-468): the
// step's goroutine is gone, so its end is judged from the candidate's keys naming it.
func (c *compositionSteps) takeUp(ctx context.Context, r *compositionRuns) {
readers, err := r.readersOfOpen()
if err != nil {
c.logf("composition-runs: the runs standing at the start are not read, so their readers are not taken up: %v", err)
return
}
for _, reader := range readers {
c.logf("the composition step %s of %s is taken up again by its runs (ask %s)", reader.Pipeline, reader.Module, reader.ID)
again := "" // a reader whose step said its end already says it again under the open run's ask (A-470q)
if reader.Ended != nil {
again = reader.ask
}
go c.say1(ctx, reader.ID, stepRunOf(reader.CompositionReader), c.resume, again)
}
}
@@ -10,7 +10,6 @@ import (
"errors"
"fmt"
"slices"
"sync"
"time"
nec "git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient"
@@ -18,8 +17,8 @@ import (
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
)
// The inputs of a composition step (design 47 G §354, A-243, A-252): one request per machine, each carrying that machine's input,
// asked of the machine's node-engine by the seat's verb compose-candidate.
// The inputs of a composition run (design 47 G §354, §718, A-243, A-252): one request per candidate and machine, carrying that
// machine's input, asked of the machine's node-engine by the seat's verb compose-candidate when the run starts.
// composeCandidateKey is the seat verb's call key, asked of one node.
const composeCandidateKey = "seat:" + ne.Seat + "." + ne.VerbComposeCandidate
@@ -80,37 +79,14 @@ type compositionInputs struct {
bound time.Duration
}
// gather asks each machine of the step's module and branch for its input with the step's candidate key, and answers every machine in
// name order: one away (away) is not asked; a node that is silent, unready or answers no input has Why set; no machine at all is an
// error. The run keeps each request built from an input under its run ask (compositionRun.machine).
func (c *compositionInputs) gather(ctx context.Context, run meshwants.StepRun, away func(node string) bool) ([]nodeInput, error) {
key, err := candidateKeyOf(run)
if err != nil {
return nil, err
}
nodes := c.machines(run.Module, run.Branch)
if len(nodes) == 0 {
return nil, fmt.Errorf("no node places %s on the branch %q, so the composition step has no machine to compose", run.Module, run.Branch)
}
// of asks the machine node for its input with the candidate's key (§718): asked only by the composition step that starts the
// machine's composition run, once per candidate and machine. A node that is silent, unready or answers no input has Why set.
func (c *compositionInputs) of(ctx context.Context, node, candidate string) nodeInput {
bound := c.bound
if bound <= 0 {
bound = nec.AnswerBound
}
out := make([]nodeInput, len(nodes))
var wg sync.WaitGroup
for i, node := range nodes {
if away != nil && away(node) {
out[i] = nodeInput{Node: node, Away: true}
continue
}
wg.Add(1)
go func() {
defer wg.Done()
out[i] = c.one(ctx, bound, node, key)
}()
}
wg.Wait()
return out, nil
return c.one(ctx, bound, node, candidate)
}
// one asks one node for its input.
@@ -68,7 +68,7 @@ func TestTheMachinesOfAStep(t *testing.T) {
// §354, A-243, A-252, §64: each machine is asked compose-candidate with the step's key and input true, its numbers as written; a
// silent or unready node says why, and a machine away is not asked.
func TestAStepsInputsAreAskedOfEachMachine(t *testing.T) {
// gather asks the machines side by side, so the record of what each was asked is guarded.
// the machines are asked side by side, so the record of what each was asked is guarded.
var mu sync.Mutex
asked := map[string]string{}
call := func(key string, body any) (json.RawMessage, error) {
@@ -86,7 +86,7 @@ func TestAStepsInputsAreAskedOfEachMachine(t *testing.T) {
return json.RawMessage(`{"candidate":"run-r1","node":"` + node + `","verdict":"composed","input":{"n":12345678901234567890,"node":"` + node + `"},"modules":[],"own":{},"took":1,"v":1}`), nil
}
c := &compositionInputs{machines: func(module, branch string) []string { return []string{"ace", "g14", "novox", "shanks"} }, callFile: call, bound: time.Second}
got, err := c.gather(context.Background(), meshwants.StepRun{Module: "m", Branch: "main", Run: "r1"}, func(node string) bool { return node == "shanks" })
got, err := gatherAll(c, meshwants.StepRun{Module: "m", Branch: "main", Run: "r1"}, func(node string) bool { return node == "shanks" })
if err != nil || len(got) != 4 {
t.Fatalf("%+v %v", got, err)
}
@@ -105,7 +105,7 @@ func TestAStepsInputsAreAskedOfEachMachine(t *testing.T) {
t.Errorf("the machine away is asked or not said away: %+v", got[3])
}
none := &compositionInputs{machines: func(string, string) []string { return nil }, callFile: call}
if _, err := none.gather(context.Background(), meshwants.StepRun{Module: "m", Run: "r1"}, nil); err == nil {
if _, err := gatherAll(none, meshwants.StepRun{Module: "m", Run: "r1"}, nil); err == nil {
t.Error("a step with no machine is gathered")
}
}
@@ -124,7 +124,7 @@ func TestAStepsInputIsTheFileOfTheNodesChunks(t *testing.T) {
return json.RawMessage(answer + `,"file":"` + file + `"}`), nil
}
c := &compositionInputs{machines: func(module, branch string) []string { return []string{"ace", "g14"} }, callFile: call, bound: time.Second}
got, err := c.gather(context.Background(), meshwants.StepRun{Module: "m", Branch: "main", Run: "r1"}, func(string) bool { return false })
got, err := gatherAll(c, meshwants.StepRun{Module: "m", Branch: "main", Run: "r1"}, func(string) bool { return false })
if err != nil || len(got) != 2 {
t.Fatalf("%+v %v", got, err)
}
@@ -135,3 +135,28 @@ func TestAStepsInputIsTheFileOfTheNodesChunks(t *testing.T) {
t.Errorf("g14: %+v", got[1])
}
}
// gatherAll asks each machine of a step for its input as each run's start does (compositionInputs.of), side by side; a machine away
// is not asked; no machine is an error.
func gatherAll(c *compositionInputs, run meshwants.StepRun, away func(string) bool) ([]nodeInput, error) {
key, err := candidateKeyOf(run)
if err != nil {
return nil, err
}
nodes := c.machines(run.Module, run.Branch)
if len(nodes) == 0 {
return nil, errors.New("no machine")
}
out := make([]nodeInput, len(nodes))
var wg sync.WaitGroup
for i, node := range nodes {
if away != nil && away(node) {
out[i] = nodeInput{Node: node, Away: true}
continue
}
wg.Add(1)
go func() { defer wg.Done(); out[i] = c.of(context.Background(), node, key) }()
}
wg.Wait()
return out, nil
}
@@ -0,0 +1,181 @@
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
mcc "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-sdk/go/buscalls"
)
// The reads of a composition step (design 47 G §354, A-252, A-261): the entries of one catalog revision are read once and shared by
// every composition step of that revision, since an entry is content-addressed and a revision never changes; a read that times out
// or fails in transport is made again with a growing wait for a bounded time, said in the journal, and only then ends the step as
// could not run.
const (
// readsBound is how long a read is made again before the step could not run.
readsBound = 3 * time.Minute
// readsFirstWait and readsMaxWait bound the wait between two tries.
readsFirstWait = time.Second
readsMaxWait = 15 * time.Second
// readsKept is how many catalog revisions' halves are kept; the oldest read goes first.
readsKept = 8
)
// readKey names one revision of one watched branch.
type readKey struct {
branch string
revision int
}
// readSlot is one revision's read: done closes when half or err is set.
type readSlot struct {
done chan struct{}
half mcc.Half
err error
}
// compositionReads shares the reads of composition steps.
type compositionReads struct {
mu sync.Mutex
revs map[readKey]*readSlot
order []readKey
seats map[string]json.RawMessage // by seat and digest: the definition, raw
}
// transient says whether a read's fault is of the transport or a timeout, so that it is worth reading again.
func transient(err error, said string) bool {
text := said
if err != nil {
text = err.Error() + " " + said
}
text = strings.ToLower(text)
for _, w := range []string{"deadline exceeded", "timeout", "timed out", "no responders", "connection", "i/o", "eof", "broken pipe", "unavailable"} {
if strings.Contains(text, w) {
return true
}
}
return false
}
// callRetry calls the seat verb key, again with a growing wait while its fault is transient, for readsBound; it answers the body of
// a good answer, or the last fault.
func (r *compositionRun) callRetry(ctx context.Context, key string, body any) (json.RawMessage, string, error) {
start := r.now()
wait := readsFirstWait
for try := 1; ; try++ {
raw, err := r.call(key, body)
var inner json.RawMessage
var said string
if err == nil {
var ok bool
if inner, said, ok = buscalls.Unwrap(raw); ok {
return inner, "", nil
}
}
if !transient(err, said) {
return nil, said, err
}
if r.now().Sub(start)+wait > readsBound {
return nil, said, err
}
r.logf("composition step: %s is not read (try %d, %v %s), read again in %s", key, try, err, said, wait)
if serr := r.waitFor(ctx, wait); serr != nil {
return nil, said, serr
}
if wait *= 2; wait > readsMaxWait {
wait = readsMaxWait
}
}
}
// waitFor waits d, or until ctx ends.
func (r *compositionRun) waitFor(ctx context.Context, d time.Duration) error {
if r.sleep != nil {
return r.sleep(ctx, d)
}
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
return nil
}
}
// half is the revision's half, read once for every step of it: a step that finds it being read waits for that read, and a read that
// failed is not kept, so the next step reads again.
func (r *compositionRun) half(ctx context.Context, key readKey, read func() (mcc.Half, error)) (mcc.Half, error) {
c := r.reads
c.mu.Lock()
if c.revs == nil {
c.revs = map[readKey]*readSlot{}
}
if s, ok := c.revs[key]; ok {
c.mu.Unlock()
select {
case <-s.done:
if s.err == nil {
return s.half, nil
}
case <-ctx.Done():
return mcc.Half{}, ctx.Err()
}
// The read this step waited for failed: read it here (the failed slot is already dropped).
return r.half(ctx, key, read)
}
s := &readSlot{done: make(chan struct{})}
c.revs[key] = s
c.order = append(c.order, key)
for len(c.order) > readsKept {
delete(c.revs, c.order[0])
c.order = c.order[1:]
}
c.mu.Unlock()
s.half, s.err = read()
if s.err != nil {
c.mu.Lock()
if c.revs[key] == s {
delete(c.revs, key)
}
for i, k := range c.order {
if k == key {
c.order = append(c.order[:i], c.order[i+1:]...)
break
}
}
c.mu.Unlock()
}
close(s.done)
return s.half, s.err
}
// seatKept is the definition kept of a seat at a digest.
func (c *compositionReads) seatKept(seat, digest string) (json.RawMessage, bool) {
c.mu.Lock()
defer c.mu.Unlock()
d, ok := c.seats[seat+"@"+digest]
return d, ok
}
func (c *compositionReads) keepSeat(seat, digest string, def json.RawMessage) {
c.mu.Lock()
defer c.mu.Unlock()
if c.seats == nil {
c.seats = map[string]json.RawMessage{}
}
c.seats[seat+"@"+digest] = def
}
func unreadErr(what string, err error, said string) error {
if err != nil {
return fmt.Errorf("%s: %w", what, err)
}
return fmt.Errorf("%s: %s", what, said)
}
@@ -18,16 +18,17 @@ import (
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/keys"
)
// The run of a composition step (design 47 G §354, §63, §64; A-226, A-238, A-240, A-243, A-252, A-254, A-261): one
// composer.Request per machine, depth compose, the one machine and one case `change`, its manifests, verdicts and seats from the
// step's catalog revision read through mesh-catalog's `revision`; one run ask per machine of the seat node-build-go, naming no
// head, its program the node-engine build's catalog entry with the composer's package; the run's end heard on the build seat's
// step-ended of that ask, its answer (a composer.Verdict) read inline from `output`. A machine the watchman said away
// (machine-away) is not composed and passes with a warning (§64).
// The run of a composition step (design 47 G §354, §63, §64, §718; A-226, A-238, A-240, A-252, A-254, A-261): the step reads one
// composition run per machine of its candidate (compositionruns.go), and the step that starts a machine's run composes it: one
// compose-candidate ask of the node, one composer.Request, depth compose, the one machine and one case `change`, its manifests,
// verdicts and seats from the step's catalog revision read through mesh-catalog's `revision`, kept once in `inputs`; one run ask
// of the seat node-build-go, naming no head, its program the node-engine build's catalog entry with the composer's package,
// chosen once and recorded in the run. The run's end is heard on the build seat's step-ended of that ask, its answer (a
// composer.Verdict) read from `output`, and written in the run's key, whose change every reading step hears. A machine the
// watchman said away (machine-away) is not composed and passes with a warning (§64).
const (
// runAsker is the assigner's name as an asker of a build seat, its seat's own name: the last token of the subject its run
@@ -42,74 +43,10 @@ const (
seatDefinitionsKey = "seat:" + meshcatalog.Seat + "." + meshcatalog.VerbSeatDefinitions
)
// runEnds hands each build seat's step-ended of an ask to the composition run that waits for it (A-240), by the ask's id.
type runEnds struct {
mu sync.Mutex
waits map[string]chan buildstep.StepEnded
}
// expect is the end of the ask id, once heard; called before the ask is put, so no end is missed.
func (r *runEnds) expect(id string) <-chan buildstep.StepEnded {
r.mu.Lock()
defer r.mu.Unlock()
if r.waits == nil {
r.waits = map[string]chan buildstep.StepEnded{}
}
ch := make(chan buildstep.StepEnded, 1)
r.waits[id] = ch
return ch
}
// forget stops waiting for the ask id.
func (r *runEnds) forget(id string) {
r.mu.Lock()
delete(r.waits, id)
r.mu.Unlock()
}
// heard hands an end to the run that waits for its ask; one nobody waits for is another asker's, left.
func (r *runEnds) heard(e buildstep.StepEnded) {
r.mu.Lock()
ch := r.waits[e.Ask]
delete(r.waits, e.Ask)
r.mu.Unlock()
if ch != nil {
ch <- e
}
}
// awayNodes are the machines the watchman said away (machine-away of the seat mesh-watchdogs, §389) and not back since.
type awayNodes struct {
mu sync.Mutex
away map[string]bool
}
// set keeps machine away (true) or back (false).
func (a *awayNodes) set(machine string, away bool) {
a.mu.Lock()
defer a.mu.Unlock()
if a.away == nil {
a.away = map[string]bool{}
}
if away {
a.away[machine] = true
} else {
delete(a.away, machine)
}
}
// is says machine is away.
func (a *awayNodes) is(machine string) bool {
if a == nil {
return false
}
a.mu.Lock()
defer a.mu.Unlock()
return a.away[machine]
}
// compositionRun runs a composition step.
type compositionRun struct {
// runs are the composition runs of the state composition-runs (compositionruns.go).
runs *compositionRuns
inputs *compositionInputs
keeper *inputKeeper
// call asks a seat verb held once for the mesh (process.Conn.CallKept).
@@ -118,10 +55,14 @@ type compositionRun struct {
trunk func() (mcc.Half, bool)
// running is the last `running` heard of a module on a node (world.runningOf), never the node-engine's bucket.
running func(node, module string) (ne.Running, bool)
away *awayNodes
// away says the watchman holds machine away (machine-away, not back since): the one presence the assigner keeps (world.awayNow),
// so a durable on the watchman's streams has one hearer.
away func(machine string) bool
// put puts a run ask on the build seat's check queue under the assigner's name.
put func(ctx context.Context, a buildstep.StepAsk) error
ends *runEnds
put func(ctx context.Context, a buildstep.StepAsk) error
// reads shares the catalog reads among steps (compositionreads.go); sleep waits between two tries of a read (nil: a timer).
reads *compositionReads
sleep func(ctx context.Context, d time.Duration) error
// ask asks a seat verb on one node, for the run's answer kept as the build seat's object (buildstep.StepEnded.ReadOutput).
ask buildstep.AskVerb
now func() time.Time
@@ -130,56 +71,105 @@ type compositionRun struct {
// machineEnd is what one machine came to.
type machineEnd struct {
failed, couldNot, warning string
failed, couldNot string
warnings []string
// judged is set where the composer judged the machine, composing or not (A-467a).
judged bool
}
// run runs one composition step and answers what it came to: failed when a machine does not compose, could not run when a machine
// could not be composed, else passed with a warning per machine away or not judged.
func (r *compositionRun) run(ctx context.Context, _ string, run meshwants.StepRun) stepOutcome {
half, err := r.revision(run)
if err != nil {
return couldNotRun("the step's catalog revision is not read: " + err.Error())
}
seats, err := r.seats(half.Revision.Definitions)
// run runs one composition step and answers what it came to: per machine not away it joins or starts the candidate's
// composition run (§718), waits for each run's end in its key, and judges each machine from it; failed when a machine does not
// compose, could not run when a machine could not be composed or was not judged, else passed with a warning per machine away.
func (r *compositionRun) run(ctx context.Context, id string, step meshwants.StepRun) stepOutcome {
candidate, err := candidateKeyOf(step)
if err != nil {
return couldNotRun(err.Error())
}
manifests, verdicts, err := caseOf(half)
if err != nil {
return couldNotRun(err.Error())
nodes := r.inputs.machines(step.Module, step.Branch)
if len(nodes) == 0 {
return couldNotRun(fmt.Sprintf("no node places %s on the branch %q, so the composition step has no machine to compose", step.Module, step.Branch))
}
programOf, err := r.programs(half)
if err != nil {
return couldNotRun(err.Error())
}
inputs, err := r.inputs.gather(ctx, run, r.away.is)
if err != nil {
return couldNotRun(err.Error())
reader := readerOf(id, step)
prep := &runPrep{r: r, step: step}
var ends []machineEnd
var keys []string
for _, node := range nodes {
if r.away != nil && r.away(node) {
ends = append(ends, machineEnd{warnings: []string{node + " is away (the watchman's machine-away), so it is not composed"}})
continue
}
join := r.runs.join
if step.Rerun { // a step asked again (§727)
join = r.runs.rejoin
}
j, err := join(reader, candidate, node, buildstep.NewID())
if err != nil {
ends = append(ends, machineEnd{couldNot: node + ": " + err.Error()})
continue
}
if j.started {
go r.start(ctx, j.key, j.run, prep)
}
keys = append(keys, j.key)
}
return r.read(ctx, step.Module, keys, ends)
}
// read waits for the end of each run of keys and judges the step of module from them, naming in its progress and its end the
// runs it reads and their asks (§723, §729).
func (r *compositionRun) read(ctx context.Context, module string, keys []string, ends []machineEnd) stepOutcome {
setShared(ctx, r.runs.sharedOf(keys, nil))
setPhase(ctx, meshwants.StepProgressEventPhaseRun)
ends := make([]machineEnd, len(inputs))
var wg sync.WaitGroup
for i, in := range inputs {
wg.Add(1)
go func() {
defer wg.Done()
ends[i] = r.machine(ctx, in, buildstep.NewID(), programOf, manifests, verdicts, seats)
}()
}
wg.Wait()
runs := r.runs.await(ctx, keys)
setPhase(ctx, meshwants.StepProgressEventPhaseReport)
for _, key := range keys {
ends = append(ends, judgeRun(module, runs[key]))
}
out := outcomeOf(ends)
out.Shared = r.runs.sharedOf(keys, runs)
return out
}
// resume judges a step taken up at the assigner's start (§719, A-468): the runs of its candidate naming it, once each has ended.
func (r *compositionRun) resume(ctx context.Context, id string, step meshwants.StepRun) stepOutcome {
candidate, err := candidateKeyOf(step)
if err != nil {
return couldNotRun(err.Error())
}
keys, err := r.runs.keysOf(candidate, id)
if err != nil {
return couldNotRun("the step's composition runs are not read: " + err.Error())
}
return r.read(ctx, step.Module, keys, nil)
}
// outcomeOf is a step's outcome from its machines: failed before could not run before passed (§354); a step that judged no
// machine never passes (A-467a), each machine not judged named with why.
func outcomeOf(ends []machineEnd) stepOutcome {
var out stepOutcome
var failed, couldNot []string
judged := 0
for _, e := range ends {
if e.judged {
judged++
}
failed = appendSome(failed, e.failed)
couldNot = appendSome(couldNot, e.couldNot)
out.Warnings = appendSome(out.Warnings, e.warning)
out.Warnings = append(out.Warnings, e.warnings...)
}
switch {
case len(failed) > 0:
out.Outcome, out.Said = meshwants.StepEndedOutcomeFailed, strings.Join(failed, "; ")
case len(couldNot) > 0:
out.Outcome, out.Said = meshwants.StepEndedOutcomeCouldNotRun, strings.Join(couldNot, "; ")
case judged == 0:
said := "no machine was judged"
if len(out.Warnings) == 0 {
said += ": the step has no machine"
} else {
said += ": " + strings.Join(out.Warnings, "; ")
}
out.Outcome, out.Said = meshwants.StepEndedOutcomeCouldNotRun, said
default:
out.Outcome = meshwants.StepEndedOutcomePassed
}
@@ -197,98 +187,207 @@ func couldNotRun(said string) stepOutcome {
return stepOutcome{Outcome: meshwants.StepEndedOutcomeCouldNotRun, Said: said}
}
// machine composes one machine: its request kept under its ask, the ask put, its end heard and its verdict read.
func (r *compositionRun) machine(ctx context.Context, in nodeInput, ask string, programOf func(node string) (buildstep.RunProgram, error),
manifests map[string]composer.Manifest, verdicts map[string]json.RawMessage, seats []json.RawMessage) machineEnd {
node := in.Node
switch {
case in.Away:
return machineEnd{warning: node + " is away (the watchman's machine-away), so it is not composed"}
case in.Why != "":
return machineEnd{couldNot: node + " gave no composition input: " + in.Why}
// runPrep is what the step that starts a run reads of its catalog revision, once for every machine it starts: the seats, the
// case's manifests and verdicts, and the composer program per machine.
type runPrep struct {
r *compositionRun
step meshwants.StepRun
once sync.Once
// read once
seats []json.RawMessage
manifests map[string]composer.Manifest
verdicts map[string]json.RawMessage
none map[string]string
programOf func(node string) (buildstep.RunProgram, string, error)
err error
}
func (p *runPrep) get(ctx context.Context) error {
p.once.Do(func() {
half, err := p.r.revision(ctx, p.step)
if err != nil {
p.err = errors.New("the step's catalog revision is not read: " + err.Error())
return
}
if p.seats, p.err = p.r.seats(ctx, half.Revision.Definitions); p.err != nil {
return
}
if p.manifests, p.verdicts, p.none, p.err = caseOf(half); p.err != nil {
return
}
p.programOf, p.err = p.r.programs(ctx, half)
})
return p.err
}
// start composes the machine of the run it started (§718): the node's input of this moment, its request kept once in `inputs`,
// the composer program chosen and recorded with the request's digest, and one run ask put. A fault ends the run could not run.
func (r *compositionRun) start(ctx context.Context, key string, run meshwants.CompositionRun, prep *runPrep) {
node, ask := run.Machine, run.Ask
end := func(said string) {
if _, err := r.runs.endRun(key, ask, meshwants.CompositionRunOutcomeCouldNotRun, said, nil); err != nil {
r.logf("the composition run %s is not ended: %v", key, err)
}
}
program, err := programOf(node)
if err := prep.get(ctx); err != nil {
end(node + ": " + err.Error())
return
}
in := r.inputs.of(ctx, node, run.Candidate)
if in.Why != "" {
end(node + " gave no composition input: " + in.Why)
return
}
program, engineFp, err := prep.programOf(node)
if err != nil {
return machineEnd{couldNot: node + ": " + err.Error()}
end(node + ": " + err.Error())
return
}
req := composer.Request{V: composer.Version, Input: in.Body, Depth: composer.DepthCompose, Machines: []string{node},
Cases: []composer.Case{{Name: changeCase, Manifests: manifests, Verdicts: verdicts}}, Seats: seats}
Cases: []composer.Case{{Name: changeCase, Manifests: prep.manifests, Verdicts: prep.verdicts, None: prep.none}}, Seats: prep.seats,
BlameTries: r.runs.set.blameTries} // the composer's bound of its search for culprits (§721, A-466r)
raw, err := json.Marshal(req)
if err != nil {
return machineEnd{couldNot: node + ": its request cannot be encoded: " + err.Error()}
end(node + ": its request cannot be encoded: " + err.Error())
return
}
if _, err := r.keeper.keep(ask, raw); err != nil {
digest, err := r.keeper.keep(ask, raw)
if err != nil {
r.logf("the composition request of %s is refused: %v", node, err)
return machineEnd{couldNot: node + ": its request is not kept: " + err.Error()}
end(node + ": its request is not kept: " + err.Error())
return
}
ref, err := buildstep.RunInputOf(inputVerbKey, raw)
if err != nil {
return machineEnd{couldNot: node + ": " + err.Error()}
end(node + ": " + err.Error())
return
}
if _, _, err := r.runs.change(key, func(c *meshwants.CompositionRun) bool {
if c.Ask != ask || c.Ended != nil {
return false
}
c.Program = &meshwants.CompositionRunProgram{Repository: program.Repository, Commit: program.Commit, Directory: program.Dir, Fingerprint: engineFp}
c.Input = digest
return true
}); err != nil {
end(node + ": the run's program and input are not written: " + err.Error())
return
}
a := buildstep.StepAsk{ID: ask, Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: runAsker, At: r.now().UTC(),
Run: &buildstep.RunJob{Program: program, Heads: []buildstep.RunHead{}, Input: ref, Reader: "the composition of " + node + " by the node-engine at " + program.Commit}}
ended := r.ends.expect(ask)
defer r.ends.forget(ask)
if err := r.put(ctx, a); err != nil {
return machineEnd{couldNot: node + ": its run ask " + ask + " is not put: " + err.Error()}
end(node + ": its run ask " + ask + " is not put: " + err.Error())
}
var e buildstep.StepEnded
select {
case <-ctx.Done():
return machineEnd{couldNot: node + ": the step stopped before its run ask " + ask + " ended"}
case e = <-ended:
}
if e.Outcome != buildstep.OutcomePassed {
return machineEnd{couldNot: fmt.Sprintf("%s: its run ask %s ended %s on %s: %s", node, ask, e.Outcome, e.Node, e.Said)}
}
answer, err := e.ReadOutput(buildstep.AskGo, r.ask) // the program's last line of standard output (A-254)
if err != nil {
return machineEnd{couldNot: node + ": " + err.Error()}
}
return verdictOf(node, answer)
}
// verdictOf reads the composer's verdict of one machine: not composing is failed, naming why; not judged passes with a warning.
func verdictOf(node string, answer []byte) machineEnd {
// notTried is the problem the composer names a module with that its search for culprits did not try, past the request's
// blame_tries (design 47 G §721; the composer's composer.NotTried in mesh-host, compared by its words, A-470b).
const notTried = "not tried: the search reached its bound"
// judgeRun judges one machine of the step of module from the run it read (§721, §722): could not run with the run's words; else
// from the machine's composer verdict: not judged is could not run (A-467); a verdict blaming the step's module fails it, or is
// could not run where the module was not tried (A-470b); a machine that does not compose and blames no module fails every step on
// it, on the node's own account; a verdict with no `blamed` (a composer before §721) is judged as before, with the warning "the
// composer names no culprits"; otherwise the machine passes, with a warning for each other module blamed there.
func judgeRun(module string, run meshwants.CompositionRun) machineEnd {
node := run.Machine
if run.Ended == nil {
return machineEnd{couldNot: node + ": the step stopped before its composition run " + runKeyOf(run.Candidate, node) + " ended"}
}
if run.Outcome != meshwants.CompositionRunOutcomePassed {
return machineEnd{couldNot: run.Said}
}
m, err := machineOf(run.Verdict)
if err != nil {
return machineEnd{couldNot: node + ": the run's verdict does not read: " + err.Error()}
}
if m.NotJudged != "" { // a step never passes on a machine it did not judge (A-467)
return machineEnd{couldNot: node + " was not judged: " + m.NotJudged}
}
why := strings.Join(appendSome(append(slices.Clone(m.Problems), m.Refusals...), m.Reason), "; ")
switch {
case m.Blamed == nil && !m.Composes: // an old composer (§721)
return machineEnd{judged: true, failed: node + " does not compose: " + why, warnings: []string{node + ": the composer names no culprits"}}
case len(m.Blamed) == 0 && !m.Composes: // the node's own account (A-452)
return machineEnd{judged: true, failed: node + " does not compose and blames no module: " + why}
}
end := machineEnd{judged: true}
var own, others []string
for _, b := range m.Blamed {
switch {
case b.Module == module && b.Problem == notTried:
end.couldNot = fmt.Sprintf("%s holds %s untried: the composer's search for culprits reached its bound of blame_tries (A-470b)", node, module)
case b.Module == module:
own = append(own, b.Problem)
default:
others = append(others, node+" holds "+b.Module+": "+b.Problem)
}
}
if len(own) > 0 {
end.failed, end.couldNot = node+" holds "+module+": "+strings.Join(own, "; "), ""
}
end.warnings = others
return end
}
// machineOf is the composer's verdict of one machine as a run keeps it.
func machineOf(v map[string]any) (composer.Machine, error) {
var m composer.Machine
raw, err := json.Marshal(v)
if err == nil {
err = json.Unmarshal(raw, &m)
}
return m, err
}
// runVerdict reads the composer's answer of one machine: its verdict as the run keeps it, or why it is none.
func runVerdict(node string, answer []byte) (map[string]any, string) {
var v composer.Verdict
if err := json.Unmarshal(answer, &v); err != nil {
return machineEnd{couldNot: node + ": the composer answered no verdict: " + err.Error()}
return nil, node + ": the composer answered no verdict: " + err.Error()
}
if v.V != composer.Version {
return machineEnd{couldNot: fmt.Sprintf("%s: the composer answered a verdict of version %d, the assigner reads %d", node, v.V, composer.Version)}
return nil, fmt.Sprintf("%s: the composer answered a verdict of version %d, the assigner reads %d", node, v.V, composer.Version)
}
for _, c := range v.Cases {
if c.Name != changeCase {
continue
}
m, ok := c.Machines[node]
switch {
case !ok:
return machineEnd{couldNot: node + ": the composer's verdict names no such machine"}
case m.NotJudged != "":
return machineEnd{warning: node + " was not judged: " + m.NotJudged}
case !m.Composes:
return machineEnd{failed: node + " does not compose: " + strings.Join(appendSome(append(slices.Clone(m.Problems), m.Refusals...), m.Reason), "; ")}
if !ok {
return nil, node + ": the composer's verdict names no such machine"
}
return machineEnd{}
raw, err := json.Marshal(m)
var out map[string]any
if err == nil {
err = json.Unmarshal(raw, &out)
}
if err != nil {
return nil, node + ": the composer's verdict cannot be kept: " + err.Error()
}
return out, ""
}
return machineEnd{couldNot: node + ": the composer's verdict has no case " + changeCase}
return nil, node + ": the composer's verdict has no case " + changeCase
}
// revision reads the step's catalog revision whole, page by page, through mesh-catalog's `revision` (A-252).
func (r *compositionRun) revision(run meshwants.StepRun) (mcc.Half, error) {
// revision reads the step's catalog revision whole, page by page, through mesh-catalog's `revision` (A-252), once for every step of
// it (compositionreads.go).
func (r *compositionRun) revision(ctx context.Context, run meshwants.StepRun) (mcc.Half, error) {
if run.Revision <= 0 {
return mcc.Half{}, errors.New("the step names no catalog revision")
}
return r.half(ctx, readKey{run.Branch, int(run.Revision)}, func() (mcc.Half, error) { return r.readRevision(ctx, run) })
}
func (r *compositionRun) readRevision(ctx context.Context, run meshwants.StepRun) (mcc.Half, error) {
half := mcc.Half{Entries: map[string]meshcatalog.CatalogEntry{}}
in := meshcatalog.RevisionInput{Branch: keys.EscapeName(run.Branch), Revision: run.Revision}
for {
raw, err := r.call(mcc.RevisionVerb, in)
body, said, err := r.callRetry(ctx, mcc.RevisionVerb, in)
if err != nil {
return mcc.Half{}, err
}
body, said, ok := buscalls.Unwrap(raw)
if !ok {
if body == nil {
return mcc.Half{}, fmt.Errorf("%s answered: %s", mcc.RevisionVerb, said)
}
var a meshcatalog.RevisionAnswer
@@ -311,28 +410,17 @@ func (r *compositionRun) revision(run meshwants.StepRun) (mcc.Half, error) {
}
}
// seats are the definitions of the revision's (seat, digest) pairs, raw (A-261): the revision's definitions carry no definition
// yet (A-234), so each is mesh-catalog's seat-definitions answer for the seat where the trunk's digest is the revision's; a digest
// neither holds is an error naming the seat and the digest.
func (r *compositionRun) seats(defs []meshcatalog.CatalogRevisionDefinitionsItem) ([]json.RawMessage, error) {
// seats are the definitions the composer is handed, raw. A revision that names definitions (A-261): the definition of each (seat, digest)
// pair, from mesh-catalog's seat-definitions where the trunk's digest is the revision's; a digest neither holds is an error naming the seat
// and the digest. A revision that names none (A-234's stand-in, A-467): every definition of the trunk seat list, read through
// seat-definitions; a list or a definition that cannot be read is an error naming it, never an empty list.
func (r *compositionRun) seats(ctx context.Context, defs []meshcatalog.CatalogRevisionDefinitionsItem) ([]json.RawMessage, error) {
if len(defs) == 0 {
return r.trunkSeats(ctx)
}
out := make([]json.RawMessage, 0, len(defs))
for _, d := range defs {
raw, err := r.call(seatDefinitionsKey, meshcatalog.SeatDefinitionsInput{Seat: d.Seat})
if err != nil {
return nil, fmt.Errorf("the definition of the seat %s is not read: %w", d.Seat, err)
}
body, said, ok := buscalls.Unwrap(raw)
if !ok {
return nil, fmt.Errorf("the definition of the seat %s is not read: %s", d.Seat, said)
}
var a meshcatalog.SeatDefinitionsAnswer
if err := json.Unmarshal(body, &a); err != nil {
return nil, fmt.Errorf("the definition of the seat %s is not read: %w", d.Seat, err)
}
if a.Entry == nil || a.Entry.Digest != d.Digest || a.Entry.Definition == nil {
return nil, fmt.Errorf("no definition of the seat %s with the digest %s is held: neither the revision nor the trunk holds it", d.Seat, d.Digest)
}
def, err := json.Marshal(a.Entry.Definition)
def, err := r.seatAt(ctx, d.Seat, d.Digest)
if err != nil {
return nil, err
}
@@ -341,20 +429,78 @@ func (r *compositionRun) seats(defs []meshcatalog.CatalogRevisionDefinitionsItem
return out, nil
}
// caseOf is the case's manifests and verdicts from the revision's entries: each manifest whole, and the verdict of each one not
// accepted (a module without one is accepted).
func caseOf(half mcc.Half) (map[string]composer.Manifest, map[string]json.RawMessage, error) {
// trunkSeats are the definitions of every seat of the trunk's seat list (seat-definitions with no seat names them with their digests).
func (r *compositionRun) trunkSeats(ctx context.Context) ([]json.RawMessage, error) {
what := "the trunk's seat list is not read"
body, said, err := r.callRetry(ctx, seatDefinitionsKey, meshcatalog.SeatDefinitionsInput{})
if err != nil || body == nil {
return nil, unreadErr(what, err, said)
}
var list meshcatalog.SeatDefinitionsAnswer
if err := json.Unmarshal(body, &list); err != nil {
return nil, fmt.Errorf("%s: %w", what, err)
}
if len(list.Seats) == 0 {
return nil, errors.New(what + ": mesh-catalog's seat-definitions names no seat, so the composer has no seat list to judge by")
}
out := make([]json.RawMessage, 0, len(list.Seats))
for _, s := range list.Seats {
def, err := r.seatAt(ctx, s.Seat, s.Digest)
if err != nil {
return nil, err
}
out = append(out, def)
}
return out, nil
}
// seatAt is the definition of a seat at a digest, kept once read (a definition at a digest never changes).
func (r *compositionRun) seatAt(ctx context.Context, seat, digest string) (json.RawMessage, error) {
if def, ok := r.reads.seatKept(seat, digest); ok {
return def, nil
}
what := "the definition of the seat " + seat + " is not read"
body, said, err := r.callRetry(ctx, seatDefinitionsKey, meshcatalog.SeatDefinitionsInput{Seat: seat})
if err != nil || body == nil {
return nil, unreadErr(what, err, said)
}
var a meshcatalog.SeatDefinitionsAnswer
if err := json.Unmarshal(body, &a); err != nil {
return nil, fmt.Errorf("%s: %w", what, err)
}
if a.Entry == nil || a.Entry.Digest != digest || a.Entry.Definition == nil {
return nil, fmt.Errorf("no definition of the seat %s with the digest %s is held: neither the revision nor the trunk holds it", seat, digest)
}
def, err := json.Marshal(a.Entry.Definition)
if err != nil {
return nil, err
}
r.reads.keepSeat(seat, digest, def)
return def, nil
}
// caseOf is the case's manifests, verdicts and none from the revision's entries: each manifest whole, the verdict of each one not
// accepted (a module without one is accepted), and the revision's none words of each module it gives no fingerprint, whose
// build can never be made (design 47 G A-470j, A-470k).
func caseOf(half mcc.Half) (map[string]composer.Manifest, map[string]json.RawMessage, map[string]string, error) {
manifests := map[string]composer.Manifest{}
verdicts := map[string]json.RawMessage{}
var none map[string]string
for name, e := range half.Entries {
if len(e.Manifest) == 0 {
continue
}
raw, err := json.Marshal(e.Manifest)
if err != nil {
return nil, nil, fmt.Errorf("the manifest of %s cannot be encoded: %w", name, err)
return nil, nil, nil, fmt.Errorf("the manifest of %s cannot be encoded: %w", name, err)
}
manifests[name] = composer.Manifest{Raw: raw}
if e.Fp == "" && e.None != "" {
if none == nil {
none = map[string]string{}
}
none[name] = e.None
}
if e.Verdict == "" || e.Verdict == mcc.VerdictAccepted {
continue
}
@@ -367,11 +513,11 @@ func caseOf(half mcc.Half) (map[string]composer.Manifest, map[string]json.RawMes
}
raw, err = json.Marshal(v)
if err != nil {
return nil, nil, err
return nil, nil, nil, err
}
verdicts[name] = raw
}
return manifests, verdicts, nil
return manifests, verdicts, none, nil
}
// nodeEngineEntry is the entry of the module that claims the seat node-engine, or false.
@@ -409,76 +555,56 @@ func programOfEntry(e meshcatalog.CatalogEntry) (buildstep.RunProgram, error) {
return buildstep.RunProgram{Repository: e.Repository, Commit: e.Commit, Dir: dir, Package: composer.Package}, nil
}
// programs answers, per machine, the composer its run builds (A-226, A-243, A-252): where the revision's node-engine entry differs
// from the trunk's, that build for every machine; else the build the machine's node-engine runs, as its `running` says, its entry
// read through mesh-catalog's `entry`; a machine whose node-engine's `running` is not heard gets the revision's entry, said (A-451).
func (r *compositionRun) programs(half mcc.Half) (func(node string) (buildstep.RunProgram, error), error) {
// programs answers, per machine, the composer its run builds and the node-engine fingerprint it was chosen from (A-226, A-243,
// A-252, §719): where the revision's node-engine entry differs from the trunk's, that build for every machine; else the build the
// machine's node-engine runs, as its `running` says, its entry read through mesh-catalog's `entry`; a machine whose node-engine's
// `running` is not heard gets the revision's entry, said (A-451).
func (r *compositionRun) programs(ctx context.Context, half mcc.Half) (func(node string) (buildstep.RunProgram, string, error), error) {
engine, ok := nodeEngineEntry(half.Entries)
if !ok {
return nil, fmt.Errorf("the catalog revision %d holds no module claiming the seat %s", half.Revision.Revision, ne.Seat)
}
if trunk, ok := r.trunk(); !ok || func() bool { t, ok := nodeEngineEntry(trunk.Entries); return !ok || t.Fp != engine.Fp }() {
p, err := programOfEntry(engine)
return func(string) (buildstep.RunProgram, error) { return p, err }, nil
return func(string) (buildstep.RunProgram, string, error) { return p, engine.Fp, err }, nil
}
var mu sync.Mutex
byFp := map[string]buildstep.RunProgram{}
return func(node string) (buildstep.RunProgram, error) {
return func(node string) (buildstep.RunProgram, string, error) {
running, ok := r.running(node, engine.Module)
if !ok || running.Fingerprint == "" {
r.logf("%s: no build of %s is heard running there, so its composer is the catalog revision's entry %s (A-451)", node,
engine.Module, engine.Fp)
return programOfEntry(engine)
p, err := programOfEntry(engine)
return p, engine.Fp, err
}
fp := running.Fingerprint
mu.Lock()
p, ok := byFp[running.Fingerprint]
p, ok := byFp[fp]
mu.Unlock()
if ok {
return p, nil
return p, fp, nil
}
raw, err := r.call(entryVerbKey, meshcatalog.EntryInput{Fingerprint: running.Fingerprint})
if err != nil {
return buildstep.RunProgram{}, fmt.Errorf("the entry of the node-engine's build %s is not read: %w", running.Fingerprint, err)
}
body, said, ok := buscalls.Unwrap(raw)
if !ok {
return buildstep.RunProgram{}, fmt.Errorf("the entry of the node-engine's build %s is not read: %s", running.Fingerprint, said)
what := "the entry of the node-engine's build " + fp + " is not read"
body, said, err := r.callRetry(ctx, entryVerbKey, meshcatalog.EntryInput{Fingerprint: fp})
if err != nil || body == nil {
return buildstep.RunProgram{}, fp, unreadErr(what, err, said)
}
var a meshcatalog.EntryAnswer
if err := json.Unmarshal(body, &a); err != nil {
return buildstep.RunProgram{}, fmt.Errorf("the entry of the node-engine's build %s is not read: %w", running.Fingerprint, err)
return buildstep.RunProgram{}, fp, fmt.Errorf("%s: %w", what, err)
}
p, err = programOfEntry(a.Entry)
if err != nil {
return p, err
return p, fp, err
}
mu.Lock()
byFp[running.Fingerprint] = p
byFp[fp] = p
mu.Unlock()
return p, nil
return p, fp, nil
}, nil
}
// heardPresence keeps a machine-away (away true) or machine-back of the seat mesh-watchdogs (§389, §436): the machine it names.
func heardPresence(a *awayNodes, away bool, m jetstream.Msg, logf func(string, ...any)) {
var e struct {
Machine string `json:"machine"`
}
if err := json.Unmarshal(m.Data(), &e); err != nil || e.Machine == "" {
logf("%s holds no machine, left: %v", m.Subject(), err)
} else {
a.set(e.Machine, away)
if away {
logf("%s is away (machine-away): a composition step does not compose it until its machine-back", e.Machine)
} else {
logf("%s is back (machine-back): a composition step composes it again", e.Machine)
}
}
if err := m.Ack(); err != nil {
logf("acknowledging %s: %v", m.Subject(), err)
}
}
// trunkOf is the trunk's catalog half the assigner's views hold, trunk naming the first watched branch, or false while none is
// held.
func trunkOf(iv *catalogViews, trunk func() string) func() (mcc.Half, bool) {
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"slices"
"strings"
"sync"
"testing"
@@ -30,6 +31,16 @@ type runStand struct {
requests map[string]composer.Request // by node
entries []string // the fingerprints `entry` was asked
r *compositionRun
runs *compositionRuns
puts int // the run asks put
// noDefinitions: the revision names no seat definition (A-234's stand-in); trunkSeats: the trunk's seat list as (seat, digest)
// pairs; unjudged: the machine the composer does not judge; listFault: the trunk's seat list cannot be read.
noDefinitions bool
trunkSeats []meshcatalog.SeatDefinitionsAnswerSeatsItem
unjudged string
listFault error
// verdict, when set, is the composer's verdict of a machine (by node), else the stand's own.
verdict func(node string) (composer.Machine, bool)
}
func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *runStand {
@@ -46,7 +57,8 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
"": {Revision: &meshcatalog.CatalogRevision{Revision: 7, Branch: "main", Definitions: []meshcatalog.CatalogRevisionDefinitionsItem{{Seat: "s1", Digest: "d1"}}},
Entries: []meshcatalog.CatalogEntry{engine(revisionEngine, revisionCommit), {Module: "bad", Verdict: mcc.VerdictRefused, Inputs: &meshcatalog.EntryInputs{Problems: []string{"no seat x"}},
Manifest: map[string]any{"module": "bad"}}}, Next: "bad"},
"bad": {Revision: &meshcatalog.CatalogRevision{Revision: 7}, Entries: []meshcatalog.CatalogEntry{{Module: "m", Verdict: mcc.VerdictAccepted, Manifest: map[string]any{"module": "m"}}}},
"bad": {Revision: &meshcatalog.CatalogRevision{Revision: 7}, Entries: []meshcatalog.CatalogEntry{{Module: "m", Verdict: mcc.VerdictAccepted, Manifest: map[string]any{"module": "m"}},
{Module: "lidarr", None: "builders disagree: node", Manifest: map[string]any{"module": "lidarr"}}}}, // no fingerprint (A-470j)
}
call := func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
@@ -57,8 +69,22 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
if in.Revision != 7 || in.Branch != "main" {
return nil, errors.New("asked another revision")
}
return json.Marshal(pages[in.After])
page := pages[in.After]
if s.noDefinitions && page.Revision != nil {
rev := *page.Revision
rev.Definitions = nil
page.Revision = &rev
}
return json.Marshal(page)
case seatDefinitionsKey:
var sin meshcatalog.SeatDefinitionsInput
_ = json.Unmarshal(raw, &sin)
if sin.Seat == "" {
if s.listFault != nil {
return nil, s.listFault
}
return json.Marshal(meshcatalog.SeatDefinitionsAnswer{Seats: s.trunkSeats})
}
return json.Marshal(meshcatalog.SeatDefinitionsAnswer{Found: true, Entry: &meshcatalog.SeatDefinitionsAnswerEntry{Seat: "s1", Digest: seatDigest, Definition: map[string]any{"seat": "s1"}}})
case entryVerbKey:
var in meshcatalog.EntryInput
@@ -76,13 +102,12 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
return json.RawMessage(`{"candidate":"main.7","node":"` + node + `","verdict":"composed","input":{"node":"` + node + `"},"modules":[],"own":{},"took":1,"v":1}`), nil
}
running := map[string]string{"ace": "src2:e1", "g14": "src2:e2"}
away := &awayNodes{}
away.set("shanks", true)
away := func(machine string) bool { return machine == "shanks" }
keeper := testKeeper(&objectsMem{objs: map[string][]byte{}}, time.Now)
ends := &runEnds{}
s.r = &compositionRun{
s.runs = newCompositionRuns(newRunsMem(), defaultRunSettings, time.Now, t.Logf)
s.r = &compositionRun{runs: s.runs,
inputs: &compositionInputs{machines: func(string, string) []string { return []string{"ace", "g14", "shanks"} }, callFile: engines, bound: time.Second},
keeper: keeper, call: call, ends: ends, away: away, now: time.Now, logf: t.Logf,
keeper: keeper, reads: &compositionReads{}, call: call, away: away, now: time.Now, logf: t.Logf,
trunk: func() (mcc.Half, bool) {
return mcc.Half{Entries: map[string]meshcatalog.CatalogEntry{"engine": engine(trunkEngine, commitE1)}}, true
},
@@ -104,11 +129,27 @@ func newRunStand(t *testing.T, revisionEngine, trunkEngine, seatDigest string) *
s.asks[node], s.requests[node] = a, req
s.mu.Unlock()
m := composer.Machine{Composes: true}
if node == "g14" {
override, set := composer.Machine{}, false
if s.verdict != nil {
override, set = s.verdict(node)
}
if set {
m = override
} else if node == s.unjudged {
m = composer.Machine{NotJudged: "the composer was handed no seat list"}
} else if node == "g14" {
m = composer.Machine{Reason: "two modules own one path", Problems: []string{"/etc/x is m's and bad's"}}
}
v, _ := json.Marshal(composer.Verdict{V: composer.Version, Cases: []composer.CaseVerdict{{Name: changeCase, Machines: map[string]composer.Machine{node: m}}}})
go ends.heard(buildstep.StepEnded{Ask: a.ID, Asker: runAsker, Node: "builder1", Outcome: buildstep.OutcomePassed, Output: string(v)})
s.mu.Lock()
s.puts++
s.mu.Unlock()
go func() {
if err := s.runs.heardEnd(buildstep.StepEnded{Ask: a.ID, Asker: runAsker, Node: "builder1", Outcome: buildstep.OutcomePassed, Output: string(v)},
func(e buildstep.StepEnded) ([]byte, error) { return []byte(e.Output), nil }); err != nil {
t.Error(err)
}
}()
return nil
},
}
@@ -124,7 +165,8 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
s := newRunStand(t, "src2:e1", "src2:e1", "d1")
out := s.r.run(context.Background(), "ask-step", step)
if out.Outcome != meshwants.StepEndedOutcomeFailed || !strings.Contains(out.Said, "g14 does not compose") || !strings.Contains(out.Said, "/etc/x") ||
strings.Contains(out.Said, "ace") || len(out.Warnings) != 1 || !strings.Contains(out.Warnings[0], "shanks is away") {
strings.Contains(out.Said, "ace") || len(out.Warnings) != 2 || !strings.Contains(out.Warnings[0], "shanks is away") ||
out.Warnings[1] != "g14: the composer names no culprits" {
t.Fatalf("the step came to %+v", out)
}
if len(s.asks) != 2 {
@@ -138,12 +180,12 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
if s.asks["g14"].Run.Program.Commit != commitE2 {
t.Errorf("g14's composer is built at %s, want the build its node-engine runs", s.asks["g14"].Run.Program.Commit)
}
if req.Depth != composer.DepthCompose || len(req.Machines) != 1 || req.Machines[0] != "ace" || len(req.Cases) != 1 || req.Cases[0].Name != changeCase ||
if req.Depth != composer.DepthCompose || req.BlameTries != 20 || len(req.Machines) != 1 || req.Machines[0] != "ace" || len(req.Cases) != 1 || req.Cases[0].Name != changeCase ||
string(req.Input) != `{"node":"ace"}` || len(req.Seats) != 1 || string(req.Seats[0]) != `{"seat":"s1"}` {
t.Errorf("the request of ace %+v", req)
}
c := req.Cases[0]
if string(c.Manifests["m"].Raw) != `{"module":"m"}` || len(c.Manifests) != 3 || len(c.Verdicts) != 1 || !strings.Contains(string(c.Verdicts["bad"]), `"refused"`) ||
if string(c.Manifests["m"].Raw) != `{"module":"m"}` || len(c.Manifests) != 4 || len(c.None) != 1 || c.None["lidarr"] != "builders disagree: node" || len(c.Verdicts) != 1 || !strings.Contains(string(c.Verdicts["bad"]), `"refused"`) ||
!strings.Contains(string(c.Verdicts["bad"]), "no seat x") {
t.Errorf("the case %+v %s", c.Manifests, c.Verdicts)
}
@@ -179,6 +221,45 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
t.Errorf("a seat digest not held came to %+v, %d asks", out, len(other.asks))
}
// A revision naming no definition (A-234's stand-in): the composer is handed the trunk seat list's definitions, read through
// seat-definitions (A-467); a list that cannot be read, or names no seat, ends the step could not run, naming it, with no ask put.
bare := newRunStand(t, "src2:e1", "src2:e1", "d1")
bare.noDefinitions, bare.trunkSeats = true, []meshcatalog.SeatDefinitionsAnswerSeatsItem{{Seat: "s1", Digest: "d1"}}
if out := bare.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeFailed ||
len(bare.requests["ace"].Seats) != 1 || string(bare.requests["ace"].Seats[0]) != `{"seat":"s1"}` {
t.Errorf("a revision with no definitions came to %+v, seats %v", out, bare.requests["ace"].Seats)
}
for name, fault := range map[string]error{"unreadable": errors.New("no responder"), "empty": nil} {
s := newRunStand(t, "src2:e1", "src2:e1", "d1")
s.noDefinitions, s.listFault = true, fault
if out := s.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeCouldNotRun ||
!strings.Contains(out.Said, "seat list is not read") || len(s.asks) != 0 {
t.Errorf("the %s seat list came to %+v, %d asks", name, out, len(s.asks))
}
}
// A machine the composer did not judge is could not run, naming why: a step that judged nothing never passes (A-467).
nj := newRunStand(t, "src2:e1", "src2:e1", "d1")
nj.unjudged = "g14"
if out := nj.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeCouldNotRun ||
!strings.Contains(out.Said, "g14 was not judged: the composer was handed no seat list") {
t.Errorf("a machine not judged came to %+v", out)
}
// No machine judged: every machine away, or no machine, never passes; each is named with why (A-467a).
for name, machines := range map[string][]string{"every machine away": {"shanks"}, "no machine": {}} { // none placed: the inputs say so already
want := "no machine was judged"
if len(machines) == 0 {
want = "has no machine to compose"
}
none := newRunStand(t, "src2:e1", "src2:e1", "d1")
none.r.inputs.machines = func(string, string) []string { return machines }
if out := none.r.run(context.Background(), "ask-step", step); out.Outcome != meshwants.StepEndedOutcomeCouldNotRun ||
!strings.Contains(out.Said, want) || (len(machines) > 0 && !strings.Contains(out.Said, "shanks is away")) || len(none.asks) != 0 {
t.Errorf("%s came to %+v, %d asks", name, out, len(none.asks))
}
}
// At the inputs state's cap the ask is refused, naming the store (§354).
full := newRunStand(t, "src2:e1", "src2:e1", "d1")
full.r.keeper.used = func() (int64, error) { return inputsCap, nil }
@@ -187,3 +268,308 @@ func TestTheCompositionStepRunsOneAskPerMachine(t *testing.T) {
t.Errorf("an ask at the cap came to %+v, %d asks", out, len(full.asks))
}
}
// Design 47 G §354, A-252: the entries of one catalog revision are read once for every step of it, and a read that times out is
// made again with a growing wait for a bounded time, said in the journal, before the step could not run.
func TestTheCompositionStepReadsTheRevisionOnceAndAgainOnATimeout(t *testing.T) {
step := meshwants.StepRun{Module: "m", Branch: "main", Revision: 7, Step: meshwants.StepRunStepComposition}
for _, tc := range []struct {
name string
faults int // consecutive faults of the `revision` read before it answers
fault error // the fault
steps int
want string // the outcome of the last step
revisionCalls int
waits int
}{
{"two steps of one revision read it once", 0, nil, 2, meshwants.StepEndedOutcomeFailed, 2, 0},
{"a timeout is read again", 3, errors.New("context deadline exceeded"), 1, meshwants.StepEndedOutcomeFailed, 5, 3},
{"a fault that stays ends could not run", 1000, errors.New("nats: timeout"), 1, meshwants.StepEndedOutcomeCouldNotRun, -1, -1},
{"a fault of the answer is not read again", 1, errors.New("no such revision"), 1, meshwants.StepEndedOutcomeCouldNotRun, 1, 0},
} {
t.Run(tc.name, func(t *testing.T) {
s := newRunStand(t, "src2:e1", "src2:e1", "d1")
inner := s.r.call
var mu sync.Mutex
calls, faults := 0, tc.faults
s.r.call = func(key string, body any) (json.RawMessage, error) {
if key == mcc.RevisionVerb {
mu.Lock()
calls++
f := faults > 0
faults--
mu.Unlock()
if f {
return nil, tc.fault
}
}
return inner(key, body)
}
clock := time.Unix(1000, 0)
var waits []time.Duration
s.r.now = func() time.Time { mu.Lock(); defer mu.Unlock(); return clock }
s.r.sleep = func(_ context.Context, d time.Duration) error {
mu.Lock()
defer mu.Unlock()
waits = append(waits, d)
clock = clock.Add(d)
return nil
}
var wg sync.WaitGroup
outs := make([]stepOutcome, tc.steps)
for i := range outs {
wg.Add(1)
go func() { defer wg.Done(); outs[i] = s.r.run(context.Background(), "ask-step", step) }()
}
wg.Wait()
for _, o := range outs {
if o.Outcome != tc.want {
t.Fatalf("a step came to %+v, want %s", o, tc.want)
}
}
if tc.waits == -1 { // never answers: bounded, growing waits
if len(waits) < 3 || waits[0] != readsFirstWait || waits[1] != 2*readsFirstWait || waits[len(waits)-1] != readsMaxWait {
t.Errorf("the waits %v do not grow to %s", waits, readsMaxWait)
}
var total time.Duration
for _, w := range waits {
total += w
}
if total > readsBound {
t.Errorf("read again for %s, bound %s", total, readsBound)
}
if !strings.Contains(outs[0].Said, "catalog revision is not read") || !strings.Contains(outs[0].Said, "timeout") {
t.Errorf("the step said %q", outs[0].Said)
}
return
}
if calls != tc.revisionCalls || len(waits) != tc.waits {
t.Errorf("`revision` asked %d times with %d waits, want %d and %d", calls, len(waits), tc.revisionCalls, tc.waits)
}
})
}
}
// Design 47 G §718, §719, §720: three composition steps of one revision on two machines start two runs, one request and one run
// ask per machine, and each step reads both verdicts; the run records its program with the node-engine fingerprint it was chosen
// from and the digest of its kept request, which names no module of the asking pipeline; a step of another candidate starts its
// own runs, and a run that could not run is never read again.
func TestTheStepsOfOneCandidateReadOneRunPerMachine(t *testing.T) {
s := newRunStand(t, "src2:e1", "src2:e1", "d1")
s.r.inputs.machines = func(string, string) []string { return []string{"ace", "g14"} }
var wg sync.WaitGroup
outs := make([]stepOutcome, 3)
for i, m := range []string{"m", "m2", "m3"} {
wg.Add(1)
go func() {
defer wg.Done()
outs[i] = s.r.run(context.Background(), "ask-"+m, meshwants.StepRun{Pipeline: "main.7." + m, Module: m, Branch: "main", Revision: 7, Step: meshwants.StepRunStepComposition})
}()
}
wg.Wait()
for i, o := range outs {
if len(o.Shared) != 2 || o.Shared[0].Machine != "ace" || o.Shared[0].Run != runKeyOf("main.7", "ace") || o.Shared[0].Ask == "" ||
o.Shared[1].Machine != "g14" || o.Shared[1].Ask == o.Shared[0].Ask { // §723, §729: per machine the run read and its ask
t.Errorf("step %d names the runs %+v", i, o.Shared)
}
if o.Outcome != meshwants.StepEndedOutcomeFailed || !strings.Contains(o.Said, "g14 does not compose") {
t.Errorf("step %d came to %+v, want both machines read", i, o)
}
}
if s.puts != 2 {
t.Fatalf("%d run asks put, want one per machine", s.puts)
}
ace, _, _, _ := s.runs.read(runKeyOf("main.7", "ace"))
g14, _, _, _ := s.runs.read(runKeyOf("main.7", "g14"))
if len(ace.Readers) != 3 || len(g14.Readers) != 3 || ace.Program == nil || ace.Program.Fingerprint != "src2:e1" || ace.Program.Commit != commitE1 ||
g14.Program == nil || g14.Program.Fingerprint != "src2:e2" || ace.Input == "" || ace.Input != digestOf(mustJSON(t, s.requests["ace"])) ||
ace.Outcome != meshwants.CompositionRunOutcomePassed || g14.Outcome != meshwants.CompositionRunOutcomePassed || g14.Verdict["composes"] != false {
t.Errorf("the runs: ace %+v, g14 %+v", ace, g14)
}
if req := s.requests["ace"]; strings.Contains(string(mustJSON(t, req)), "main.7.m") || string(req.Input) != `{"node":"ace"}` {
t.Errorf("the request names the asking pipeline or not the node's input: %s", mustJSON(t, req))
}
// Another candidate (a run on request) starts its own runs.
if o := s.r.run(context.Background(), "ask-r", meshwants.StepRun{Pipeline: "run-r1.m", Module: "m", Branch: "main", Revision: 7, Run: "r1", Step: meshwants.StepRunStepComposition}); o.Outcome != meshwants.StepEndedOutcomeFailed || s.puts != 4 {
t.Errorf("a step of another candidate came to %+v with %d asks, want its own two runs", o, s.puts)
}
// A run whose ask could not be put could not run; the next step of the candidate starts it again.
f := newRunStand(t, "src2:e1", "src2:e1", "d1")
f.r.inputs.machines = func(string, string) []string { return []string{"ace"} }
put, refused := f.r.put, 0
f.r.put = func(ctx context.Context, a buildstep.StepAsk) error {
if refused == 0 {
refused++
return errors.New("no stream")
}
return put(ctx, a)
}
first := f.r.run(context.Background(), "ask-1", meshwants.StepRun{Pipeline: "main.7.m", Module: "m", Branch: "main", Revision: 7, Step: meshwants.StepRunStepComposition})
second := f.r.run(context.Background(), "ask-2", meshwants.StepRun{Pipeline: "main.7.m2", Module: "m2", Branch: "main", Revision: 7, Step: meshwants.StepRunStepComposition})
if first.Outcome != meshwants.StepEndedOutcomeCouldNotRun || !strings.Contains(first.Said, "is not put") || second.Outcome != meshwants.StepEndedOutcomePassed || f.puts != 1 {
t.Errorf("after a run that could not run: %+v then %+v, %d asks", first, second, f.puts)
}
}
func mustJSON(t *testing.T, v any) []byte {
t.Helper()
raw, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return raw
}
// Design 47 G §721, §722, A-470b: a step judges each machine from its verdict's `blamed`: a verdict blaming the step's module fails
// it and passes another module's step with a warning naming the module; a machine that does not compose and blames no module fails
// every step; a verdict with no `blamed` is judged as before and says "the composer names no culprits"; a module the search did not
// try is could not run for its own step; a run that could not run is could not run.
func TestAMachineIsJudgedByTheModulesItBlames(t *testing.T) {
at := time.Now()
runOf := func(verdict string) meshwants.CompositionRun {
var v map[string]any
if err := json.Unmarshal([]byte(verdict), &v); err != nil {
t.Fatal(err)
}
return meshwants.CompositionRun{Candidate: "main.7", Machine: "ace", Ended: &at, Outcome: meshwants.CompositionRunOutcomePassed, Verdict: v}
}
blamesA := `{"composes":false,"reason":"two modules own one path","blamed":[{"module":"a","problem":"/etc/x is a's and c's"}]}`
for _, tc := range []struct {
name, module string
run meshwants.CompositionRun
failed, couldNo string
warnings []string
}{
{"blamed module fails", "a", runOf(blamesA), "ace holds a: /etc/x is a's and c's", "", nil},
{"another module passes with a warning", "b", runOf(blamesA), "", "", []string{"ace holds a: /etc/x is a's and c's"}},
{"blames no module fails both", "b", runOf(`{"composes":false,"reason":"the node's own composition","blamed":[]}`), "ace does not compose and blames no module: the node's own composition", "", nil},
{"an old composer", "b", runOf(`{"composes":false,"reason":"x"}`), "ace does not compose: x", "", []string{"ace: the composer names no culprits"}},
{"composes", "b", runOf(`{"composes":true,"blamed":[]}`), "", "", nil},
{"own module not tried", "a", runOf(`{"composes":false,"blamed":[{"module":"a","problem":"not tried: the search reached its bound"}]}`), "",
"ace holds a untried: the composer's search for culprits reached its bound of blame_tries (A-470b)", nil},
{"another not tried", "b", runOf(`{"composes":false,"blamed":[{"module":"a","problem":"not tried: the search reached its bound"}]}`), "", "",
[]string{"ace holds a: not tried: the search reached its bound"}},
{"not judged", "a", runOf(`{"composes":false,"not_judged":"no seat list","blamed":[]}`), "", "ace was not judged: no seat list", nil},
{"could not run", "a", meshwants.CompositionRun{Machine: "ace", Ended: &at, Outcome: meshwants.CompositionRunOutcomeCouldNotRun, Said: "ace: gone"}, "", "ace: gone", nil},
} {
t.Run(tc.name, func(t *testing.T) {
e := judgeRun(tc.module, tc.run)
if e.failed != tc.failed || e.couldNot != tc.couldNo || !slices.Equal(e.warnings, tc.warnings) {
t.Errorf("judged %+v, want failed %q, could not run %q, warnings %q", e, tc.failed, tc.couldNo, tc.warnings)
}
})
}
}
// Design 47 G §727, A-468g, A-470d: a rerun of a composition step starts a new run only on the machine it did not pass, replacing
// that key and carrying every former reader, the one that did not pass marked rerun with the rerun's caller; at the new run's end
// that reader says its end again, marked rerun, naming the new run's ask; a reader that passed does not move; the machine the
// step passed keeps its verdict.
func TestARerunMovesEveryReaderThatDidNotPass(t *testing.T) {
s := newRunStand(t, "src2:e1", "src2:e1", "d1")
s.r.inputs.machines = func(string, string) []string { return []string{"ace", "g14"} }
var mu sync.Mutex
fixed := false
s.verdict = func(node string) (composer.Machine, bool) {
mu.Lock()
defer mu.Unlock()
m := composer.Machine{Composes: true}
m.Blamed = []struct {
Module string `json:"module"`
Problem string `json:"problem"`
}{}
if node == "g14" && !fixed {
m.Composes, m.Reason = false, "two modules own one path"
m.Blamed = append(m.Blamed, struct {
Module string `json:"module"`
Problem string `json:"problem"`
}{"a", "/etc/x"}, struct {
Module string `json:"module"`
Problem string `json:"problem"`
}{"d", "/etc/x"})
}
return m, true
}
ends := make(chan meshwants.StepEnded, 16)
msgIDs := map[string]string{} // the message id of each step's latest end
steps := &compositionSteps{node: "n1", now: time.Now, logf: t.Logf, run: s.r.run, resume: s.r.resume, ended: s.runs.stepEnded,
say: func(event string, body any, msgID string) error {
if e, ok := body.(meshwants.StepEnded); ok {
mu.Lock()
msgIDs[e.ID] = msgID
mu.Unlock()
ends <- e
}
return nil
}}
move := steps.move(context.Background())
s.runs.moved.Store(&move)
step := func(m string, rerun bool) []byte {
raw, _ := json.Marshal(meshwants.StepRun{Pipeline: "main.7." + m, Module: m, Branch: "main", Revision: 7, Step: meshwants.StepRunStepComposition, Rerun: rerun, By: map[bool]string{true: "op@n1"}[rerun]})
return raw
}
got := map[string]meshwants.StepEnded{}
for _, m := range []string{"a", "b", "d"} {
steps.take(context.Background(), "ask-"+m, step(m, false))
e := <-ends
got[e.Module] = e
}
if got["a"].Outcome != meshwants.StepEndedOutcomeFailed || got["d"].Outcome != meshwants.StepEndedOutcomeFailed || got["b"].Outcome != meshwants.StepEndedOutcomePassed {
t.Fatalf("the first ends: %+v", got)
}
former := got["d"].Shared[1].Ask
mu.Lock()
fixed = true
mu.Unlock()
steps.take(context.Background(), "ask-a2", step("a", true))
moved := map[string]meshwants.StepEnded{}
for range 2 {
select {
case e := <-ends:
moved[e.ID] = e
case <-time.After(5 * time.Second):
t.Fatalf("the ends after the rerun: %+v", moved)
}
}
a, d := moved["ask-a2"], moved["ask-d"]
if a.Outcome != meshwants.StepEndedOutcomePassed || d.Outcome != meshwants.StepEndedOutcomePassed || !d.Rerun || d.By != "op@n1" ||
len(d.Shared) != 2 || d.Shared[1].Machine != "g14" || d.Shared[1].Ask == former || d.Shared[1].Ask != a.Shared[1].Ask {
t.Fatalf("after the rerun: a %+v, d %+v", a, d)
}
mu.Lock()
// A-470q: the first end keeps the ask's id; the end said again names the moved run's ask, so the bus does not drop it.
if msgIDs["ask-a2"] != "ask-a2" || msgIDs["ask-d"] != "ask-d."+a.Shared[1].Ask || msgIDs["ask-b"] != "ask-b" {
t.Errorf("the message ids of the ends: %v", msgIDs)
}
mu.Unlock()
if s.puts != 3 {
t.Errorf("%d run asks, want one more for the machine a did not pass", s.puts)
}
select {
case e := <-ends:
t.Errorf("a reader that passed moved: %+v", e)
case <-time.After(50 * time.Millisecond):
}
run, _, _, _ := s.runs.read(runKeyOf("main.7", "g14"))
marks := map[string]bool{}
for _, c := range run.Readers {
marks[c.ID] = c.Rerun
}
if len(run.Readers) != 3 || !marks["ask-d"] || marks["ask-b"] || !marks["ask-a2"] {
t.Errorf("the new run's readers %+v", run.Readers)
}
// A-470m: each reader's `ended` is written once its step said its end, and ask-b's carried into the new run.
deadline := time.After(5 * time.Second)
for {
wake := s.runs.wake()
run, _, _, _ = s.runs.read(runKeyOf("main.7", "g14"))
if !slices.ContainsFunc(run.Readers, func(c meshwants.CompositionReader) bool { return c.Ended == nil }) {
break
}
select {
case <-wake:
case <-deadline:
t.Fatalf("a reader of the new run names no end: %+v", run.Readers)
}
}
}
@@ -0,0 +1,896 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"slices"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go/jetstream"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/keys"
"mesh-delivery-assigner/internal/buckets"
)
// The composition runs (design 47 G §718, §719; A-468, A-469d, A-469w, A-469x): the assigner composes each machine once per
// candidate, whatever number of composition steps name it. Each run is one key of its old state `composition-runs`,
// `<candidate key>.<machine>`, each part escaped as a key's name; the step that starts a run writes the key create-only,
// so two steps of one candidate starting at once start one run and the second reads it. A reader is the composition step's
// StepRun with its ask's id (A-468), added and removed by compare-and-set. A key is deleted once composition-runs-kept has
// passed since its `ended`, judged at each new run over at most 20 of the oldest ended keys, as the state inputs is
// (A-465b, A-468b); at the cap (composition-runs-cap-mib, the manifest's max-mib its ceiling, A-469d) no run is started.
const (
// runsSwept is how many of the oldest ended keys each new run judges.
runsSwept = 20
// runBoundPrefix and runUntakenPrefix open a run's two bound keys, each with its own time to live (§720, A-468b,
// A-470): `bound.<run key>` from the ask's ask-taken, `untaken.<run key>` from the run's start.
runBoundPrefix = "bound."
runUntakenPrefix = "untaken."
)
// runKeyOf is the key of a candidate's composition run of a machine (§719).
func runKeyOf(candidate, machine string) string {
return keysPart(candidate) + "." + keys.EscapeName(machine)
}
// keysPart is a name as a key's part.
func keysPart(name string) string { return keys.EscapeName(name) }
// isRunKey is whether a key of the state is a run's own, not one of its bound keys.
func isRunKey(key string) bool {
return !strings.HasPrefix(key, runBoundPrefix) && !strings.HasPrefix(key, runUntakenPrefix)
}
// runSettings are the settings of the composition runs, each read at the start from the file the manifest declares.
type runSettings struct {
// capBytes is composition-runs-cap-mib in bytes.
capBytes int64
// kept is composition-runs-kept, stepBound composition-step-bound, untaken run-untaken-bound (A-469x).
kept, stepBound, untaken time.Duration
// blameTries is blame-tries, carried in each run's request (§721).
blameTries int
}
// defaultRunSettings are the settings' defaults, as the manifest gives them.
var defaultRunSettings = runSettings{capBytes: 16 << 20, kept: 24 * time.Hour, stepBound: 30 * time.Minute, untaken: 24 * time.Hour, blameTries: 20}
// runSettingsOf reads the settings composition-runs-cap-mib, composition-runs-kept, composition-step-bound,
// run-untaken-bound and blame-tries from the files the environment names (A-469x, as outsideSettings): a missing or empty
// file is its default; a content that is no positive whole number or Go duration is an error naming the file.
func runSettingsOf(getenv func(string) string) (runSettings, error) {
s := defaultRunSettings
mib, err := settingFile(getenv("MESH_ASSIGNER_COMPOSITION_RUNS_CAP_MIB_FILE"))
if err != nil {
return s, err
}
if mib > 0 {
s.capBytes = int64(mib) << 20
}
tries, err := settingFile(getenv("MESH_ASSIGNER_BLAME_TRIES_FILE"))
if err != nil {
return s, err
}
if tries > 0 {
s.blameTries = tries
}
for env, to := range map[string]*time.Duration{"MESH_ASSIGNER_COMPOSITION_RUNS_KEPT_FILE": &s.kept,
"MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE": &s.stepBound, "MESH_ASSIGNER_RUN_UNTAKEN_BOUND_FILE": &s.untaken} {
d, err := durationFile(getenv(env))
if err != nil {
return s, err
}
if d > 0 {
*to = d
}
}
return s, nil
}
// durationFile reads one setting's file holding a Go duration: 0 for no path, no file or an empty one.
func durationFile(path string) (time.Duration, error) {
if path == "" {
return 0, nil
}
raw, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
return 0, nil
}
if err != nil {
return 0, err
}
v := strings.TrimSpace(string(raw))
if v == "" {
return 0, nil
}
d, err := time.ParseDuration(v)
if err != nil || d <= 0 {
return 0, fmt.Errorf("%s holds %q, which is not a positive Go duration such as 30m", path, v)
}
return d, nil
}
// errRunChanged is a compare-and-set of a run's key that met another write.
var errRunChanged = errors.New("the composition run's key changed")
// runsKV is the state composition-runs: the bus bucket, or a test's.
type runsKV interface {
// get is a key's value and revision; nil for a key not there.
get(key string) ([]byte, uint64, error)
// create writes a key that does not stand, update one at its revision: errRunChanged when it does not stand so.
create(key string, value []byte) (uint64, error)
update(key string, value []byte, revision uint64) (uint64, error)
// delAt deletes a key at its revision: errRunChanged when it no longer stands so, nothing for a key not there. No write
// of the state is made from a value read before its latest change.
delAt(key string, revision uint64) error
keys() ([]string, error)
// arm writes a bound key afresh holding the ask of the run it bounds, with its own time to live (§720, A-470n): at the key's
// latest revision, a delete marker counted, read again when another write came between; its expiry is the bus's event. It
// answers the revision written.
arm(key, ask string, ttl time.Duration) (uint64, error)
// used is how many bytes the state holds.
used() (int64, error)
}
// busRuns is runsKV over the bus bucket.
type busRuns struct {
ctx context.Context
js jetstream.JetStream
kv jetstream.KeyValue
}
func (b busRuns) get(key string) ([]byte, uint64, error) {
e, err := b.kv.Get(b.ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, nil
}
if err != nil {
return nil, 0, err
}
return e.Value(), e.Revision(), nil
}
func (b busRuns) create(key string, value []byte) (uint64, error) {
rev, err := b.kv.Create(b.ctx, key, value)
if errors.Is(err, jetstream.ErrKeyExists) {
return 0, errRunChanged
}
return rev, err
}
func (b busRuns) update(key string, value []byte, revision uint64) (uint64, error) {
rev, err := b.kv.Update(b.ctx, key, value, revision)
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return 0, errRunChanged
}
return rev, err
}
func (b busRuns) delAt(key string, revision uint64) error {
err := b.kv.Delete(b.ctx, key, jetstream.LastRevision(revision))
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return errRunChanged
}
return err
}
func (b busRuns) keys() ([]string, error) {
l, err := b.kv.ListKeys(b.ctx)
if err != nil {
return nil, err
}
var out []string
for k := range l.Keys() {
out = append(out, k)
}
return out, nil
}
// boundValue is a bound key's value: the key and the ask of the run it bounds (A-470n).
type boundValue struct {
Bound string `json:"bound"`
Ask string `json:"ask"`
}
// arm publishes the bound key holding ask with ttl as its own time to live, expecting the key's latest message (a delete
// marker counted), so a write between is read again and never overwritten unseen (A-470n); the earlier message, replaced,
// takes its time to live with it, so the expiry is heard only ttl after the last write.
func (b busRuns) arm(key, ask string, ttl time.Duration) (uint64, error) {
if b.js == nil {
return 0, errors.New("no bus to arm a bound on")
}
raw, err := json.Marshal(boundValue{Bound: key, Ask: ask})
if err != nil {
return 0, err
}
subject := "$KV." + b.kv.Bucket() + "." + key
for {
st, err := b.js.Stream(b.ctx, "KV_"+b.kv.Bucket())
if err != nil {
return 0, err
}
var last uint64
m, err := st.GetLastMsgForSubject(b.ctx, subject)
switch {
case err == nil:
last = m.Sequence
case !errors.Is(err, jetstream.ErrMsgNotFound):
return 0, err
}
ack, err := b.js.Publish(b.ctx, subject, raw, jetstream.WithMsgTTL(ttl), jetstream.WithExpectLastSequencePerSubject(last))
if err != nil && strings.Contains(err.Error(), "wrong last sequence") {
continue
}
if err != nil {
return 0, err
}
return ack.Sequence, nil
}
}
func (b busRuns) used() (int64, error) {
st, err := b.kv.Status(b.ctx)
if err != nil {
return 0, err
}
return int64(st.Bytes()), nil
}
// compositionRuns keeps the composition runs in the state composition-runs.
type compositionRuns struct {
kv runsKV
set runSettings
now func() time.Time
logf func(string, ...any)
mu sync.Mutex
// changed is closed and replaced at each change of the state heard (heard): a step waiting on a run reads its key again.
changed chan struct{}
// seen are the keys the state's watch handed and has not seen deleted: a listing of the bucket misses a key that is
// being written at that moment, so every lookup takes these beside it (keys).
seen map[string]bool
// say says an event of the seat mesh-wants (a reader's wait, A-469y); drop asks the build seat's drop of a run ask (§119);
// nil says or drops nothing.
say publisher
drop func(ask, why string) error
// moved is told each run that ended, for the readers a rerun carried to say their end again (compositionSteps.move).
// Set once the steps are served, after the hub starts.
moved atomic.Pointer[func(run meshwants.CompositionRun)]
}
func newCompositionRuns(kv runsKV, set runSettings, now func() time.Time, logf func(string, ...any)) *compositionRuns {
return &compositionRuns{kv: kv, set: set, now: now, logf: logf, changed: make(chan struct{})}
}
// heard takes a change of the state from its watch (a module watches its own state, principle 1).
func (r *compositionRuns) heard() {
r.mu.Lock()
close(r.changed)
r.changed = make(chan struct{})
r.mu.Unlock()
}
// saw keeps a key the state's watch handed: standing, or deleted or expired.
func (r *compositionRuns) saw(key string, standing bool) {
r.mu.Lock()
defer r.mu.Unlock()
if r.seen == nil {
r.seen = map[string]bool{}
}
if standing {
r.seen[key] = true
} else {
delete(r.seen, key)
}
}
// keys are the state's keys: the bucket's listing with the keys its watch handed beside it, sorted. A key read gone later
// is skipped by its reader.
func (r *compositionRuns) keys() ([]string, error) {
all, err := r.kv.keys()
if err != nil {
return nil, err
}
r.mu.Lock()
for key := range r.seen {
all = append(all, key)
}
r.mu.Unlock()
slices.Sort(all)
return slices.Compact(all), nil
}
// wake is closed at the next change heard.
func (r *compositionRuns) wake() <-chan struct{} {
r.mu.Lock()
defer r.mu.Unlock()
return r.changed
}
// read is a run's value and revision, or false.
func (r *compositionRuns) read(key string) (meshwants.CompositionRun, uint64, bool, error) {
raw, rev, err := r.kv.get(key)
if err != nil || raw == nil {
return meshwants.CompositionRun{}, 0, false, err
}
var run meshwants.CompositionRun
if err := json.Unmarshal(raw, &run); err != nil {
return meshwants.CompositionRun{}, 0, false, fmt.Errorf("the composition run %s does not read: %w", key, err)
}
return run, rev, true, nil
}
// change writes a run's key by compare-and-set: f changes the value read and says whether to write it; false leaves it.
// It answers the value written or left, and false for a key not there.
func (r *compositionRuns) change(key string, f func(*meshwants.CompositionRun) bool) (meshwants.CompositionRun, bool, error) {
for {
run, rev, ok, err := r.read(key)
if err != nil || !ok {
return run, ok, err
}
if !f(&run) {
return run, true, nil
}
raw, err := json.Marshal(run)
if err != nil {
return run, true, err
}
if _, err := r.kv.update(key, raw, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return run, true, err
}
r.heard()
return run, true, nil
}
}
// fill is the state's fill against its cap.
func (r *compositionRuns) fill() (meshwants.Fill, error) {
at := r.now().UTC()
used, err := r.kv.used()
return meshwants.Fill{Store: buckets.CompositionRunsState, Bytes: used, Cap: r.set.capBytes, At: &at}, err
}
// joined is a reader's place in a run: the run's key, whether this reader started it (so it composes the machine), and the
// value as it stood.
type joined struct {
key string
started bool
run meshwants.CompositionRun
}
// join finds or starts the candidate's run of the machine for reader (§718, §719, §720): a run that stands and has not
// ended, or that ended other than could not run, is read, the reader added; none, or one that could not run, is started:
// written create-only (or replacing the ended key by compare-and-set), the reader its first, ask its run ask's id. At the
// cap no run is started, an error naming the fill.
func (r *compositionRuns) join(reader meshwants.CompositionReader, candidate, machine, ask string) (joined, error) {
key := runKeyOf(candidate, machine)
for {
run, rev, ok, err := r.read(key)
if err != nil {
return joined{}, err
}
if ok && (run.Ended == nil || run.Outcome != meshwants.CompositionRunOutcomeCouldNotRun) {
if hasReader(run.Readers, reader.ID) {
return joined{key: key, run: run}, nil
}
run.Readers = append(run.Readers, reader)
raw, err := json.Marshal(run)
if err != nil {
return joined{}, err
}
if _, err := r.kv.update(key, raw, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return joined{}, err
}
r.heard()
if run.Ended == nil && r.untaken(key, run.Ask) {
r.sayWaits(run, []meshwants.CompositionReader{reader}, "")
}
return joined{key: key, run: run}, nil
}
if err := r.room(); err != nil {
return joined{}, err
}
started := r.now().UTC()
fresh := meshwants.CompositionRun{Candidate: candidate, Machine: machine, Ask: ask, Started: &started,
Readers: []meshwants.CompositionReader{reader}}
raw, err := json.Marshal(fresh)
if err != nil {
return joined{}, err
}
if ok {
_, err = r.kv.update(key, raw, rev) // could not run is never read: a new run replaces it (§720)
} else {
_, err = r.kv.create(key, raw)
}
if errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return joined{}, err
}
r.heard()
// An ask waiting in the queue has no clock but run-untaken-bound from its start (A-466s, A-468b, A-470, A-470n).
if _, err := r.armBound(runUntakenPrefix, key, ask, r.set.untaken); err != nil {
r.logf("the untaken bound of the composition run %s is not armed: %v", key, err)
}
r.sayWaits(fresh, fresh.Readers, "")
return joined{key: key, started: true, run: fresh}, nil
}
}
// room sweeps the state and says whether a new run may start: an error naming the fill at the cap (as A-465a).
func (r *compositionRuns) room() error {
if err := r.sweep(); err != nil {
r.logf("composition-runs: the keys past composition-runs-kept are not all deleted: %v", err)
}
f, err := r.fill()
if err != nil {
return fmt.Errorf("the fill of the state %s is not read, so no composition run is started: %w", buckets.CompositionRunsState, err)
}
if f.Bytes >= f.Cap {
return fmt.Errorf("the state %s is at its cap of %d MiB (%d bytes kept), so no composition run is started", buckets.CompositionRunsState,
f.Cap>>20, f.Bytes)
}
return nil
}
// sweep deletes, among the 20 oldest ended runs, those ended more than composition-runs-kept ago, with their bound keys.
func (r *compositionRuns) sweep() error {
all, err := r.keys()
if err != nil {
return err
}
type endedAt struct {
key string
at time.Time
}
var ended []endedAt
for _, key := range all {
if !isRunKey(key) {
continue
}
run, _, ok, err := r.read(key)
if err != nil || !ok || run.Ended == nil {
continue
}
ended = append(ended, endedAt{key, *run.Ended})
}
slices.SortFunc(ended, func(a, b endedAt) int { return a.at.Compare(b.at) })
if len(ended) > runsSwept {
ended = ended[:runsSwept]
}
var errs []error
for _, e := range ended {
if r.now().Sub(e.at) >= r.set.kept {
errs = append(errs, r.deleteRun(e.key, func(run meshwants.CompositionRun) bool {
return run.Ended != nil && r.now().Sub(*run.Ended) >= r.set.kept
}))
}
}
return errors.Join(errs...)
}
// deleteRun deletes a run's key with its bound keys while the value it holds is still one gone judges gone, each delete at the
// revision just read and judged again on a conflict: a run that replaced the key meanwhile, and its bounds, stand.
func (r *compositionRuns) deleteRun(key string, gone func(meshwants.CompositionRun) bool) error {
for {
run, rev, ok, err := r.read(key)
if err != nil || !ok || !gone(run) {
return err
}
if err := r.clearBounds(key, run.Ask, runBoundPrefix, runUntakenPrefix); err != nil {
return err
}
if err := r.kv.delAt(key, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return err
}
r.heard()
return nil
}
}
// clearBounds deletes the bound keys of the run of ask while key holds that run (or none) and the bound holds that ask: each at
// the revision it was read at, read again on a conflict, so a bound a newer run of the key armed is never deleted.
func (r *compositionRuns) clearBounds(key, ask string, prefixes ...string) error {
var errs []error
for _, prefix := range prefixes {
for {
run, _, ok, err := r.read(key)
if err != nil {
errs = append(errs, err)
break
}
if ok && run.Ask != ask {
break // a newer run holds the key: its bounds are its own
}
held, rev, standing, err := r.boundOf(prefix + key)
if err != nil || !standing {
errs = append(errs, err)
break
}
if held != ask {
break // the bound of another ask (A-470n): not this run's to delete
}
if err := r.kv.delAt(prefix+key, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
errs = append(errs, err)
}
break
}
}
return errors.Join(errs...)
}
// stepEnded writes at as the `ended` of the reader asked under id in each run of its step's candidate naming it, by
// compare-and-set (A-470m): the time its step said its end, carried with it into a rerun's run.
func (r *compositionRuns) stepEnded(id string, step meshwants.StepRun, at time.Time) {
candidate, err := candidateKeyOf(step)
if err != nil {
r.logf("the end of the composition step %s is not written on its runs: %v", id, err)
return
}
keys, err := r.keysOf(candidate, id)
if err != nil {
r.logf("the end of the composition step %s is not written on its runs: %v", id, err)
return
}
at = at.UTC()
for _, key := range keys {
if _, _, err := r.change(key, func(c *meshwants.CompositionRun) bool {
i := slices.IndexFunc(c.Readers, func(c meshwants.CompositionReader) bool { return c.ID == id })
if i < 0 {
return false
}
c.Readers[i].Ended = &at
return true
}); err != nil {
r.logf("the end of the composition step %s is not written on the composition run %s: %v", id, key, err)
}
}
}
// hasReader is whether readers holds the step asked under id.
func hasReader(readers []meshwants.CompositionReader, id string) bool {
return slices.ContainsFunc(readers, func(c meshwants.CompositionReader) bool { return c.ID == id })
}
// readerOf is a composition step as a run's reader: its StepRun with its ask's id (A-468).
func readerOf(id string, s meshwants.StepRun) meshwants.CompositionReader {
return meshwants.CompositionReader{ID: id, Pipeline: s.Pipeline, Step: s.Step, Module: s.Module, Branch: s.Branch, Commit: s.Commit,
Revision: s.Revision, Source: s.Source, Fingerprint: s.Fingerprint, Run: s.Run, Rerun: s.Rerun, By: s.By}
}
// stepRunOf is a reader's StepRun again, for its step-ended.
func stepRunOf(c meshwants.CompositionReader) meshwants.StepRun {
return meshwants.StepRun{Pipeline: c.Pipeline, Step: c.Step, Module: c.Module, Branch: c.Branch, Commit: c.Commit, Revision: c.Revision,
Source: c.Source, Fingerprint: c.Fingerprint, Run: c.Run, Rerun: c.Rerun, By: c.By}
}
// watchRuns hears the changes of the state composition-runs (principle 1): each change wakes the steps waiting on a run, and
// the expiry of a bound key, the bus's marker, is handed to expired (§720). After the values that stand, ready is called.
func watchRuns(ctx context.Context, kv jetstream.KeyValue, r *compositionRuns, expired func(key string), ready func()) error {
w, err := kv.WatchAll(ctx)
if err != nil {
return err
}
go func() {
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
return
}
if e == nil {
if ready != nil {
ready()
ready = nil
}
continue
}
r.saw(e.Key(), e.Operation() == jetstream.KeyValuePut)
if e.Operation() == jetstream.KeyValuePurge && !isRunKey(e.Key()) && expired != nil {
expired(e.Key())
}
r.heard()
}
}
}()
return nil
}
// end ends the run of ask in its key with an outcome, its words and the machine's verdict, by compare-and-set; it answers
// false when the key holds another run or the run has ended already.
func (r *compositionRuns) end(key, ask, outcome, said string, verdict map[string]any) (bool, error) {
ended := false
_, _, err := r.change(key, func(c *meshwants.CompositionRun) bool {
if c.Ask != ask || c.Ended != nil {
return false
}
at := r.now().UTC()
c.Ended, c.Outcome, c.Said, c.Verdict = &at, outcome, said, verdict
ended = true
return true
})
return ended, err
}
// keyOfAsk is the key of the run whose ask is ask, or none.
func (r *compositionRuns) keyOfAsk(ask string) (string, meshwants.CompositionRun, error) {
all, err := r.keys()
if err != nil {
return "", meshwants.CompositionRun{}, err
}
for _, key := range all {
if !isRunKey(key) {
continue
}
run, _, ok, err := r.read(key)
if err == nil && ok && run.Ask == ask {
return key, run, nil
}
}
return "", meshwants.CompositionRun{}, nil
}
// heardEnd ends the run of the ask a build seat's step-ended names (§719, A-240): a run ask that ended other than passed
// could not run; a passed one's answer, read with read, is the machine's verdict, or why it is none. An end of an ask no run
// names is another asker's, or of a run gone, and changes nothing.
func (r *compositionRuns) heardEnd(e buildstep.StepEnded, read func(buildstep.StepEnded) ([]byte, error)) error {
if e.Asker != runAsker || e.Ask == "" {
return nil
}
key, run, err := r.keyOfAsk(e.Ask)
if err != nil || key == "" {
return err
}
if run.Ended != nil { // ended by its bound or a drop: the end heard later is ignored (§720)
r.logf("the step-ended of the run ask %s is ignored: its composition run %s ended before it (%s)", e.Ask, key, run.Said)
return nil
}
node := run.Machine
outcome, said := meshwants.CompositionRunOutcomeCouldNotRun, ""
var verdict map[string]any
if e.Outcome != buildstep.OutcomePassed {
said = fmt.Sprintf("%s: its run ask %s ended %s on %s: %s", node, e.Ask, e.Outcome, e.Node, e.Said)
} else if answer, err := read(e); err != nil { // the program's last line of standard output (A-254)
said = node + ": " + err.Error()
} else if verdict, said = runVerdict(node, answer); said == "" {
outcome = meshwants.CompositionRunOutcomePassed
}
_, err = r.endRun(key, e.Ask, outcome, said, verdict)
return err
}
// await waits until each run of keys has ended, through the state's changes heard, or ctx ends: it answers each run as it
// stands then; a key gone could not run.
func (r *compositionRuns) await(ctx context.Context, keys []string) map[string]meshwants.CompositionRun {
out := map[string]meshwants.CompositionRun{}
for {
wake := r.wake()
open := false
for _, key := range keys {
run, _, ok, err := r.read(key)
switch {
case err != nil:
open = true
continue
case !ok:
at := r.now().UTC()
candidate, machine, _ := strings.Cut(key, ".")
run = meshwants.CompositionRun{Candidate: unescape(candidate), Machine: unescape(machine), Ended: &at,
Outcome: meshwants.CompositionRunOutcomeCouldNotRun, Said: unescape(machine) + ": its composition run " + key + " is gone"}
case run.Ended == nil:
open = true
}
out[key] = run
}
if !open {
return out
}
select {
case <-ctx.Done():
return out
case <-wake:
}
}
}
// unescape is a key's part as a name again; a part that does not read stays as it is.
func unescape(part string) string {
if name, err := keys.UnescapeName(part); err == nil {
return name
}
return part
}
// sharedOf names per machine the runs of keys and their asks (§729): from runs where given, else as each key stands.
func (r *compositionRuns) sharedOf(keys []string, runs map[string]meshwants.CompositionRun) []meshwants.Shared {
out := []meshwants.Shared{}
for _, key := range keys {
run, ok := runs[key]
if !ok {
if run, _, ok, _ = r.read(key); !ok {
continue
}
}
out = append(out, meshwants.Shared{Machine: run.Machine, Run: key, Ask: run.Ask})
}
return out
}
// leave removes the composition steps of a superseded or cancelled pipeline from every run's readers (§723, A-469n), and answers
// their asks' ids, each told to stop before a run is ended; a running run left with no reader has its ask dropped and ends could not run, "no reader is left" (A-469z),
// so a later step of the candidate starts a new run. A run other readers read goes on for them.
func (r *compositionRuns) leave(pipeline string, stop func(ids []string)) ([]string, error) {
all, err := r.keys()
if err != nil {
return nil, err
}
var ids []string
var errs []error
for _, key := range all {
if !isRunKey(key) {
continue
}
var left []string
var leaving []meshwants.CompositionReader
run, ok, err := r.change(key, func(c *meshwants.CompositionRun) bool {
left, leaving = nil, nil
keep := c.Readers[:0:0]
for _, reader := range c.Readers {
if reader.Pipeline == pipeline && reader.Step == meshwants.CompositionReaderStepComposition {
left = append(left, reader.ID)
leaving = append(leaving, reader)
continue
}
keep = append(keep, reader)
}
c.Readers = keep
return len(left) > 0
})
if err != nil {
errs = append(errs, err)
continue
}
if !ok || len(left) == 0 {
continue
}
ids = append(ids, left...)
if run.Ended == nil && r.untaken(key, run.Ask) { // a reader leaving a run not yet taken: its wait ends at once (A-470u)
r.sayWaits(run, leaving, "superseded")
}
if stop != nil {
stop(left) // the steps that left say no end, the run's included
}
if run.Ended != nil || len(run.Readers) > 0 {
continue
}
why := "no reader is left: the pipeline " + pipeline + " is superseded or cancelled"
if r.drop != nil {
if err := r.drop(run.Ask, why); err != nil {
r.logf("the drop of the run ask %s of the composition run %s is not answered: %v", run.Ask, key, err)
}
}
if _, err := r.endRun(key, run.Ask, meshwants.CompositionRunOutcomeCouldNotRun, run.Machine+": "+why, nil); err != nil {
errs = append(errs, err)
}
}
return ids, errors.Join(errs...)
}
// passedOn is whether the step of module passed on the machine of an ended run (§722): neither failed nor could not run there.
func passedOn(module string, run meshwants.CompositionRun) bool {
e := judgeRun(module, run)
return e.failed == "" && e.couldNot == ""
}
// formerOf is the place in readers of a step's former ask: the reader of the same pipeline's composition step, or -1.
func formerOf(readers []meshwants.CompositionReader, pipeline string) int {
return slices.IndexFunc(readers, func(c meshwants.CompositionReader) bool {
return c.Pipeline == pipeline && c.Step == meshwants.CompositionReaderStepComposition
})
}
// rejoin is join for a step asked again (§727, A-468g, A-470d): on a machine whose ended run it passed it keeps that verdict,
// reading the run under its new ask; a running run it joins; an ended run it did not read and that could run it reads, as one
// started after its last end; otherwise it starts a new run, replacing the ended key by compare-and-set and carrying every former
// reader into the new value: each that did not pass on the former run marked `rerun` with by the rerun's caller, each that passed
// not (A-470d).
func (r *compositionRuns) rejoin(reader meshwants.CompositionReader, candidate, machine, ask string) (joined, error) {
key := runKeyOf(candidate, machine)
for {
run, rev, ok, err := r.read(key)
if err != nil {
return joined{}, err
}
if !ok {
return r.join(reader, candidate, machine, ask)
}
at := formerOf(run.Readers, reader.Pipeline)
ended := run.Ended != nil
keep := ended && at >= 0 && passedOn(reader.Module, run)
read := !ended || (at < 0 && run.Outcome != meshwants.CompositionRunOutcomeCouldNotRun)
if keep || read {
if at >= 0 && run.Readers[at].ID == reader.ID {
return joined{key: key, run: run}, nil
}
var former []meshwants.CompositionReader
if at >= 0 {
former = []meshwants.CompositionReader{run.Readers[at]}
run.Readers[at] = reader
} else {
run.Readers = append(run.Readers, reader)
}
if err := r.write(key, run, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return joined{}, err
}
if run.Ended == nil && r.untaken(key, run.Ask) { // as join: a reader of a run not yet taken waits for a build node (A-469y)
r.sayWaits(run, former, "superseded") // the place it takes: that wait ends at once (A-470u)
r.sayWaits(run, []meshwants.CompositionReader{reader}, "")
}
return joined{key: key, run: run}, nil
}
if err := r.room(); err != nil {
return joined{}, err
}
started := r.now().UTC()
fresh := meshwants.CompositionRun{Candidate: candidate, Machine: machine, Ask: ask, Started: &started}
for i, c := range run.Readers {
if i == at {
continue
}
c.Rerun, c.By = !passedOn(c.Module, run), ""
if c.Rerun {
c.By = reader.By
}
fresh.Readers = append(fresh.Readers, c)
}
fresh.Readers = append(fresh.Readers, reader)
if err := r.clearBounds(key, run.Ask, runBoundPrefix, runUntakenPrefix); err != nil { // the former run's bounds go with it
r.logf("the bound keys of the former composition run %s are not deleted: %v", key, err)
}
if err := r.write(key, fresh, rev); errors.Is(err, errRunChanged) {
continue
} else if err != nil {
return joined{}, err
}
if _, err := r.armBound(runUntakenPrefix, key, ask, r.set.untaken); err != nil {
r.logf("the untaken bound of the composition run %s is not armed: %v", key, err)
}
r.sayWaits(fresh, fresh.Readers, "")
return joined{key: key, started: true, run: fresh}, nil
}
}
// write writes a run's value at its revision.
func (r *compositionRuns) write(key string, run meshwants.CompositionRun, rev uint64) error {
raw, err := json.Marshal(run)
if err != nil {
return err
}
if _, err := r.kv.update(key, raw, rev); err != nil {
return err
}
r.heard()
return nil
}
@@ -0,0 +1,404 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient/enginereads"
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
mpr "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"mesh-delivery-assigner/internal/buckets"
)
// Test 108 (design 47 G §733; §718 to §723, §727, §728; A-468 to A-470f): the assigner started whole against the bus composes
// each machine once per candidate. Two or three composition steps of one candidate on two machines make one key, one request and
// one run ask per machine, and each step reads both verdicts; a machine blaming a and b fails their steps and passes c's with a
// warning; a verdict with no `blamed` is judged as before, "the composer names no culprits", and a machine blaming none fails
// every step; a step of another candidate starts its own runs. A taken ask that keeps saying progress is not ended by the bound,
// one silent past it ends could not run with its ask dropped, a step-ended after that is ignored and journaled, and an ask never
// taken ends at run-untaken-bound; a bound holds its run's ask (A-470n). A rerun of a starts one run on the machine it did not
// pass and moves b; a superseded step leaves
// a run running for its other reader; an assigner restarted during a run takes it up by its ask; compositions-drop ends the
// readers and deletes the key; at the cap a step ends could not run, naming the fill.
func TestOneCompositionRunPerCandidateAndMachine(t *testing.T) {
b := startBed(t, "")
b.settings = map[string]string{"MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE": "2s", "MESH_ASSIGNER_RUN_UNTAKEN_BOUND_FILE": "15s", // past a restart
"MESH_ASSIGNER_COMPOSITION_RUNS_CAP_MIB_FILE": "1"}
nc, err := nats.Connect(b.srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
const commit = "1111111111111111111111111111111111111111"
if _, err := b.js.CreateOrUpdateConsumer(b.ctx, "EVENTS", jetstream.ConsumerConfig{Durable: "n1_mesh-delivery-assigner",
FilterSubjects: []string{enginereads.SubjectPrefix + "*", "mesh.mod." + buckets.IndexerModule + ".event.views.>", meshcatalog.SubjectCatalogRevisionAll,
nodebuildgo.SubjectStepEndedAll, nodebuildgo.SubjectAskTakenAll, nodebuildgo.SubjectStepProgressAll, pipelineEndedSubject},
AckPolicy: jetstream.AckExplicitPolicy, AckWait: 2 * time.Second}); err != nil { // an event handed to the stopped assigner's pull comes again soon
t.Fatal(err)
}
for _, node := range []string{"n1", "n2"} {
raw, _ := json.Marshal(nodeengine.Running{V: 1, Node: node, Module: "engine", Fingerprint: "src2:e1", At: time.Now().UTC()})
if _, err := b.js.Publish(b.ctx, nodeengine.SubjectRunning(node, "engine"), raw); err != nil {
t.Fatal(err)
}
}
awaitKept(t, b, nodeengine.StreamRunning, 2)
engine := meshcatalog.CatalogEntry{Module: "engine", Fp: "src2:e1", Repository: "novox/mesh-host", Commit: commit, Path: ".",
Serves: []string{nodeengine.Seat + ".composition"}, Manifest: map[string]any{"module": "engine", "version": "0.1"}}
b.cat.mu.Lock()
b.cat.entries[trunkBranch] = map[string]meshcatalog.CatalogEntry{"engine": engine}
for _, m := range []string{"a", "b", "c"} {
b.cat.entries[trunkBranch][m] = meshcatalog.CatalogEntry{Module: m, Fp: "src2:" + m, Repository: "novox/" + m, Commit: commit, Path: ".", Manifest: map[string]any{"module": m}}
}
b.cat.rev[trunkBranch] = 3
b.cat.mu.Unlock()
for _, node := range []string{"n1", "n2"} {
for _, m := range []string{"a", "b", "c"} {
b.place(t, node, m)
}
}
var mu sync.Mutex
inputsAsked := map[string]int{} // by candidate and node
var drops []string
respond := func(subject string, f func(m *nats.Msg) any) {
sub, err := nc.Subscribe(subject, func(m *nats.Msg) {
raw, _ := json.Marshal(map[string]any{"result": f(m), "node": "n0"})
_ = m.Respond(raw)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
respond("mesh.seat.node-engine.call.compose-candidate.*.*", func(m *nats.Msg) any {
node := strings.Split(m.Subject, ".")[5]
var in nodeengine.ComposeCandidateInput
_ = json.Unmarshal(m.Data, &in)
mu.Lock()
inputsAsked[in.Candidate+" "+node]++
mu.Unlock()
return map[string]any{"candidate": in.Candidate, "node": node, "verdict": "composed", "input": map[string]any{"node": node}, "modules": []any{}, "own": map[string]any{}, "took": 1, "v": 1}
})
respond("mesh.seat.mesh-catalog.call.entry.*", func(*nats.Msg) any { return meshcatalog.EntryAnswer{Entry: engine} })
respond("mesh.seat.node-build-go.call.drop.*", func(m *nats.Msg) any {
var in nodebuildgo.DropInput
_ = json.Unmarshal(m.Data, &in)
mu.Lock()
drops = append(drops, in.ID)
mu.Unlock()
return map[string]any{}
})
respond("mesh.seat.mesh-hand-acts.call.hand-act.*", func(*nats.Msg) any { return map[string]any{"log": "hand-act-1"} })
// The ends are read from the stream through a durable consumer, as the runner hears them: an end the bus drops as a
// repeated message id is never heard (A-470q).
ended := make(chan meshwants.StepEnded, 32)
runner, err := b.js.CreateOrUpdateConsumer(b.ctx, "EVENTS", jetstream.ConsumerConfig{Durable: "runner-stand-in",
FilterSubject: "mesh.seat.mesh-wants.event.step-ended.n1", AckPolicy: jetstream.AckExplicitPolicy, DeliverPolicy: jetstream.DeliverNewPolicy})
if err != nil {
t.Fatal(err)
}
heard, err := runner.Consume(func(m jetstream.Msg) {
var e meshwants.StepEnded
_ = json.Unmarshal(m.Data(), &e)
_ = m.Ack()
ended <- e
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(heard.Stop)
subscribed(t, nc)
from := b.journal.mark()
_, stop := b.startStoppable(t)
b.journal.await(t, from, "the composition step is heard on", "the composition step served", 30*time.Second)
select {
case <-b.seatListRead:
case <-time.After(30 * time.Second):
t.Fatal("the assigner did not read the seat list within 30 s")
}
queue, _ := buildstep.AskStream(buildstep.AskGo)
worker, err := b.js.CreateOrUpdateConsumer(b.ctx, queue, jetstream.ConsumerConfig{Durable: "stand-in", FilterSubject: "mesh.seat.node-build-go.accept.check.mesh-wants",
AckPolicy: jetstream.AckExplicitPolicy})
if err != nil {
t.Fatal(err)
}
step := func(id, module, run string, rerun bool) {
t.Helper()
s := meshwants.StepRun{Pipeline: "main.3." + module, Step: meshwants.StepRunStepComposition, Module: module, Branch: trunkBranch, Revision: 3,
Commit: commit, Fingerprint: "src2:" + module, Run: run, Rerun: rerun}
if run != "" {
s.Pipeline = "run-" + run + "." + module
}
if rerun {
s.By = "op@n1"
}
raw, _ := json.Marshal(s)
msg := nats.NewMsg(meshwants.AcceptSubjectStep)
msg.Header.Set("x-event-id", id)
msg.Data = raw
if err := nc.PublishMsg(msg); err != nil {
t.Fatal(err)
}
}
type asked struct {
ask buildstep.StepAsk
machine string
}
// asks takes n run asks of the stand-in build seat, each with the machine its request composes.
asks := func(n int) []asked {
t.Helper()
var out []asked
for range n {
got, err := worker.Next(jetstream.FetchMaxWait(20 * time.Second))
if err != nil {
t.Fatalf("%d of %d run asks put: %v", len(out), n, err)
}
_ = got.Ack()
a, err := buildstep.DecodeStepAsk(got.Data())
if err != nil || a.Run == nil {
t.Fatalf("the run ask %s does not read: %v", got.Data(), err)
}
req, err := composer.Decode(readInput(t, nc, a.Run.Input))
if err != nil || len(req.Machines) != 1 || req.BlameTries != 20 {
t.Fatalf("the request of %s: %+v %v", a.ID, req, err)
}
out = append(out, asked{a, req.Machines[0]})
}
return out
}
noMoreAsks := func(what string) {
t.Helper()
if got, err := worker.Next(jetstream.FetchMaxWait(300 * time.Millisecond)); err == nil {
t.Fatalf("%s: one more run ask is put: %s", what, got.Data())
}
}
say := func(event string, node string, body any) {
t.Helper()
raw, _ := json.Marshal(body)
if _, err := b.js.Publish(b.ctx, "mesh.seat.node-build-go.event."+event+"."+node, raw); err != nil {
t.Fatal(err)
}
}
answer := func(a asked, machine string) {
t.Helper()
v := fmt.Sprintf(`{"v":%d,"cases":[{"name":"change","manifests":{},"machines":{%q:%s}}]}`, composer.Version, a.machine, machine)
say("step-ended", "b1", buildstep.StepEnded{V: 1, ID: a.ask.ID, Ask: a.ask.ID, Asker: runAsker, Node: "b1", Outcome: buildstep.OutcomePassed, Output: v, At: time.Now().UTC()})
}
ends := func(n int, what string) map[string]meshwants.StepEnded {
t.Helper()
out := map[string]meshwants.StepEnded{}
for len(out) < n {
select {
case e := <-ended:
out[e.ID] = e
case <-time.After(20 * time.Second):
t.Fatalf("%s: %d of %d steps ended: %+v", what, len(out), n, out)
}
}
return out
}
runKeys := func(candidate string) []string {
all, _ := b.buckets[buckets.CompositionRuns()].Keys(b.ctx)
var out []string
for _, k := range all {
if isRunKey(k) && strings.HasPrefix(k, keysPart(candidate)+".") {
out = append(out, k)
}
}
return out
}
// readers waits, through a watch of the key, until the run of key has n readers.
readers := func(key string, n int) {
t.Helper()
w, err := b.buckets[buckets.CompositionRuns()].Watch(b.ctx, key)
if err != nil {
t.Fatal(err)
}
defer func() { _ = w.Stop() }()
deadline := time.After(10 * time.Second)
for {
select {
case e := <-w.Updates():
var run meshwants.CompositionRun
if e != nil && json.Unmarshal(e.Value(), &run) == nil && len(run.Readers) == n {
return
}
case <-deadline:
t.Fatalf("the run %s has not %d readers", key, n)
}
}
}
blames := `{"composes":false,"reason":"two modules own one path","blamed":[{"module":"a","problem":"/etc/x is a's and b's"},{"module":"b","problem":"/etc/x is a's and b's"}]}`
composes := `{"composes":true,"blamed":[]}`
// One candidate, three steps, two machines: two runs.
for _, m := range []string{"a", "b", "c"} {
step("step-"+m, m, "", false)
}
first := asks(2)
noMoreAsks("three steps of one candidate")
for _, a := range first {
answer(a, map[string]string{"n1": composes, "n2": blames}[a.machine])
}
got := ends(3, "the three steps")
if a, bb, c := got["step-a"], got["step-b"], got["step-c"]; a.Outcome != meshwants.StepEndedOutcomeFailed || !strings.Contains(a.Said, "n2 holds a") ||
bb.Outcome != meshwants.StepEndedOutcomeFailed || c.Outcome != meshwants.StepEndedOutcomePassed || len(c.Warnings) != 2 ||
!strings.Contains(c.Warnings[0], "n2 holds a") || len(c.Shared) != 2 || c.Shared[0].Ask == c.Shared[1].Ask {
t.Fatalf("the steps of one candidate: %+v", got)
}
if keys := runKeys("main.3"); len(keys) != 2 || inputsAsked["main.3 n1"] != 1 || inputsAsked["main.3 n2"] != 1 {
t.Fatalf("keys %v, inputs asked %v: want one run and one input per machine", keys, inputsAsked)
}
// Another candidate (a run on request): its own runs; an old composer and a machine blaming none fail.
step("step-r1", "a", "r1", false)
for _, a := range asks(2) {
answer(a, map[string]string{"n1": `{"composes":false,"reason":"x"}`, "n2": `{"composes":false,"reason":"own","blamed":[]}`}[a.machine])
}
if e := ends(1, "another candidate")["step-r1"]; e.Outcome != meshwants.StepEndedOutcomeFailed || !strings.Contains(e.Said, "n2 does not compose and blames no module") ||
len(e.Warnings) != 1 || e.Warnings[0] != "n1: the composer names no culprits" {
t.Fatalf("a step of another candidate: %+v", e)
}
// The bound: n1's ask taken and saying progress past the bound, then silent; n2's never taken.
from = b.journal.mark()
step("step-r2", "a", "r2", false)
bound := asks(2)
var n1 asked
for _, a := range bound {
if a.machine == "n1" {
n1 = a
}
}
say("ask-taken", "b1", nodebuildgo.AskTaken{Ask: n1.ask.ID, Asker: runAsker, At: time.Now().UTC(), Node: "b1"})
for i := range 3 { // past the 2 s bound, each beat within it
select {
case e := <-ended:
t.Fatalf("a run saying its progress ended: %+v", e)
case <-time.After(time.Second):
}
if i == 0 { // the bound holds its run's ask, published at the key's latest revision with its own time to live (A-470n)
runs := b.buckets[buckets.CompositionRuns()]
all, _ := runs.Keys(b.ctx)
held := ""
for _, k := range all {
if strings.HasPrefix(k, runBoundPrefix) {
if e, err := runs.Get(b.ctx, k); err == nil {
var v boundValue
_ = json.Unmarshal(e.Value(), &v)
held += v.Ask
}
}
}
if held != n1.ask.ID {
t.Fatalf("the bound keys hold the asks %q, want n1's %s alone", held, n1.ask.ID)
}
}
say("step-progress", "b1", nodebuildgo.StepProgress{Ask: n1.ask.ID, Asker: runAsker, At: time.Now().UTC(), Phase: "run"})
}
e := ends(1, "the bounds")["step-r2"]
if e.Outcome != meshwants.StepEndedOutcomeCouldNotRun || !strings.Contains(e.Said, "said no progress within 2s (composition-step-bound)") ||
!strings.Contains(e.Said, "taken by no build node within 15s (run-untaken-bound)") {
t.Fatalf("past the bounds: %+v", e)
}
mu.Lock()
dropped := len(drops)
mu.Unlock()
if dropped != 2 {
t.Fatalf("%d asks dropped past their bounds, want 2", dropped)
}
answer(n1, composes) // heard after the bound: ignored and journaled
b.journal.await(t, from, "the step-ended of the run ask "+n1.ask.ID+" is ignored", "a late step-ended", 10*time.Second)
// A rerun of a: n1 it passed keeps its verdict; n2 it failed starts one run carrying b, which moves.
step("step-a2", "a", "", true)
again := asks(1)
noMoreAsks("a rerun of a")
if again[0].machine != "n2" {
t.Fatalf("the rerun composes %s, want only the machine a did not pass", again[0].machine)
}
answer(again[0], composes)
moved := ends(2, "the rerun")
if a2, bb := moved["step-a2"], moved["step-b"]; a2.Outcome != meshwants.StepEndedOutcomePassed || bb.Outcome != meshwants.StepEndedOutcomePassed ||
!bb.Rerun || bb.By != "op@n1" || len(bb.Shared) != 2 || bb.Shared[1].Ask != again[0].ask.ID {
t.Fatalf("after the rerun (asked again %s): a2 %+v; b %+v", again[0].ask.ID, moved["step-a2"].Shared, moved["step-b"])
}
// A superseded step leaves the runs; its other reader goes on.
step("step-r3a", "a", "r3", false)
step("step-r3b", "b", "r3", false)
sup := asks(2)
readers(runKeyOf("run-r3", "n1"), 2)
readers(runKeyOf("run-r3", "n2"), 2)
pe, _ := json.Marshal(mpr.PipelineEnded{V: 1, Pipeline: "run-r3.a", Module: "a", Branch: trunkBranch, Commit: commit, Outcome: mpr.PipelineEndedOutcomeSuperseded, Run: "r3", Steps: []mpr.StepResult{}})
if _, err := b.js.Publish(b.ctx, "mesh.seat."+mpr.Seat+".event."+mpr.EventPipelineEnded+".n1", pe); err != nil {
t.Fatal(err)
}
for _, a := range sup {
answer(a, composes)
}
if got := ends(1, "the superseded run"); got["step-r3b"].Outcome != meshwants.StepEndedOutcomePassed {
t.Fatalf("the other reader: %+v", got)
}
select {
case e := <-ended:
t.Fatalf("a superseded step said its end: %+v", e)
case <-time.After(200 * time.Millisecond):
}
// Restarted during a run: the new assigner takes it up by its ask.
step("step-r4", "c", "r4", false)
restart := asks(2)
stop()
for _, a := range restart {
answer(a, composes)
}
from = b.journal.mark()
b.startStoppable(t)
b.journal.await(t, from, "is taken up again by its runs (ask step-r4)", "the take-up", 30*time.Second)
if e := ends(1, "the take-up")["step-r4"]; e.Outcome != meshwants.StepEndedOutcomePassed || len(e.Shared) != 2 {
t.Fatalf("the step taken up: %+v", e)
}
// compositions-drop: the readers ended naming the drop, the key deleted.
step("step-r5", "a", "r5", false)
asks(2)
body, _ := json.Marshal(meshwants.CompositionsDropInput{Candidate: "run-r5", Machine: "n1", Why: "stuck"})
reply, err := nc.Request("mesh.seat.mesh-wants.call.compositions-drop.n1~mesh-delivery-pipeline", body, 10*time.Second)
if err != nil || !strings.Contains(string(reply.Data), `"ended":[`) {
t.Fatalf("compositions-drop answered %s %v", reply.Data, err)
}
if e := ends(1, "the drop")["step-r5"]; e.Outcome != meshwants.StepEndedOutcomeCouldNotRun || !strings.Contains(e.Said, "dropped by hand: stuck") {
t.Fatalf("a reader of the dropped run: %+v", e)
}
if keys := runKeys("run-r5"); len(keys) != 1 {
t.Errorf("the keys of run-r5 after the drop: %v, want n2's alone", keys)
}
// At the cap of 1 MiB no run starts: the step ends could not run, naming the fill.
for i := range 2 {
if _, err := b.buckets[buckets.CompositionRuns()].Put(b.ctx, fmt.Sprintf("fill%d", i), make([]byte, 700<<10)); err != nil {
t.Fatal(err)
}
}
step("step-r6", "a", "r6", false)
if e := ends(1, "the cap")["step-r6"]; e.Outcome != meshwants.StepEndedOutcomeCouldNotRun || !strings.Contains(e.Said, "is at its cap of 1 MiB") {
t.Fatalf("at the cap: %+v", e)
}
}
@@ -0,0 +1,642 @@
package main
import (
"context"
"encoding/json"
"fmt"
"maps"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
)
// runsMem is the state composition-runs in memory, its bound keys' times to live kept beside.
type runsMem struct {
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
ttl map[string]time.Duration
bytes int64
n uint64
// beforeDel, when set, runs once before the next delete: another write landing between a read and the delete.
beforeDel func()
// beforeArm, when set, runs once before the next arm: another write landing between a read and the arm.
beforeArm func()
}
func (k *runsMem) another() {
k.mu.Lock()
f := k.beforeDel
k.beforeDel = nil
k.mu.Unlock()
if f != nil {
f()
}
}
func newRunsMem() *runsMem {
return &runsMem{vals: map[string][]byte{}, revs: map[string]uint64{}, ttl: map[string]time.Duration{}}
}
func (k *runsMem) get(key string) ([]byte, uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
return k.vals[key], k.revs[key], nil
}
func (k *runsMem) create(key string, v []byte) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; ok {
return 0, errRunChanged
}
k.n++
k.vals[key], k.revs[key] = v, k.n
return k.n, nil
}
func (k *runsMem) update(key string, v []byte, rev uint64) (uint64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if k.revs[key] != rev {
return 0, errRunChanged
}
k.n++
k.vals[key], k.revs[key] = v, k.n
return k.n, nil
}
func (k *runsMem) delAt(key string, rev uint64) error {
k.another()
k.mu.Lock()
defer k.mu.Unlock()
if _, ok := k.vals[key]; !ok {
return nil
}
if k.revs[key] != rev {
return errRunChanged
}
delete(k.vals, key)
delete(k.revs, key)
delete(k.ttl, key)
return nil
}
func (k *runsMem) keys() ([]string, error) {
k.mu.Lock()
defer k.mu.Unlock()
var out []string
for key := range k.vals {
out = append(out, key)
}
slices.Sort(out)
return out, nil
}
func (k *runsMem) arm(key, ask string, ttl time.Duration) (uint64, error) {
k.mu.Lock()
f := k.beforeArm
k.beforeArm = nil
k.mu.Unlock()
if f != nil {
f()
}
raw, _ := json.Marshal(boundValue{Bound: key, Ask: ask})
k.mu.Lock()
defer k.mu.Unlock()
k.n++
k.vals[key], k.revs[key], k.ttl[key] = raw, k.n, ttl
return k.n, nil
}
// expire ends a bound key's time to live: the key gone, as the bus's marker leaves it.
func (k *runsMem) expire(key string) {
k.mu.Lock()
defer k.mu.Unlock()
delete(k.vals, key)
delete(k.revs, key)
delete(k.ttl, key)
}
func (k *runsMem) used() (int64, error) {
k.mu.Lock()
defer k.mu.Unlock()
if k.bytes > 0 {
return k.bytes, nil
}
var n int64
for _, v := range k.vals {
n += int64(len(v))
}
return n, nil
}
// Design 47 G §718, §719, §720, A-468, A-469d, A-469x: a run's key is `<candidate key>.<machine>`, each part escaped; two
// steps of one candidate joining at once start one run and the second reads it; a run that could not run is replaced; at the
// cap no run starts, naming the fill; a run is deleted composition-runs-kept after its end; the settings are read from their files.
func TestACompositionRunIsOneKeyPerCandidateAndMachine(t *testing.T) {
now := time.Date(2026, 11, 17, 12, 0, 0, 0, time.UTC)
step := func(id, module string) meshwants.CompositionReader {
return readerOf(id, meshwants.StepRun{Pipeline: "main.3." + module, Step: meshwants.StepRunStepComposition, Module: module, Branch: "main", Revision: 3})
}
t.Run("key", func(t *testing.T) {
if k := runKeyOf("main.3", "n1"); k != "main=2e3.n1" {
t.Errorf("the key is %q, want each part escaped as a key's name", k)
}
if isRunKey(runBoundPrefix+"main=2e3.n1") || isRunKey(runUntakenPrefix+"main=2e3.n1") || !isRunKey("main=2e3.n1") {
t.Error("a bound key is taken for a run's or a run's for a bound key")
}
})
t.Run("one run for two steps at once", func(t *testing.T) {
kv := newRunsMem()
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
var wg sync.WaitGroup
got := make([]joined, 2)
for i, m := range []string{"a", "b"} {
wg.Add(1)
go func() {
defer wg.Done()
j, err := r.join(step("ask-"+m, m), "main.3", "n1", "run-"+m)
if err != nil {
t.Error(err)
}
got[i] = j
}()
}
wg.Wait()
runKeys := func() []string {
all, _ := kv.keys()
return slices.DeleteFunc(all, func(k string) bool { return !isRunKey(k) })
}
if got[0].started == got[1].started {
t.Fatalf("started %v and %v: want exactly one step to start the run", got[0].started, got[1].started)
}
keys := runKeys()
run, _, _, _ := r.read(runKeyOf("main.3", "n1"))
if len(keys) != 1 || len(run.Readers) != 2 || run.Candidate != "main.3" || run.Machine != "n1" || run.Started == nil {
t.Errorf("keys %v, run %+v: want one key read by both", keys, run)
}
if j, _ := r.join(step("ask-a", "a"), "main.3", "n1", "run-c"); j.started || len(j.run.Readers) != 2 {
t.Errorf("a reader joining again is added again or starts a run: %+v", j)
}
})
t.Run("could not run is never read", func(t *testing.T) {
kv := newRunsMem()
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
if _, err := r.join(step("ask-a", "a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
key := runKeyOf("main.3", "n1")
_, _, _ = r.change(key, func(c *meshwants.CompositionRun) bool {
c.Ended, c.Outcome = &now, meshwants.CompositionRunOutcomeCouldNotRun
return true
})
j, err := r.join(step("ask-b", "b"), "main.3", "n1", "run-2")
if err != nil || !j.started || j.run.Ask != "run-2" || len(j.run.Readers) != 1 || j.run.Ended != nil {
t.Errorf("after a could not run: %+v %v, want a new run read by its starter alone", j, err)
}
_, _, _ = r.change(key, func(c *meshwants.CompositionRun) bool {
c.Ended, c.Outcome = &now, meshwants.CompositionRunOutcomePassed
return true
})
if j, _ := r.join(step("ask-c", "c"), "main.3", "n1", "run-3"); j.started || j.run.Ask != "run-2" {
t.Errorf("a passed run is not read: %+v", j)
}
if j, _ := r.join(step("ask-d", "d"), "main.4", "n1", "run-4"); !j.started {
t.Errorf("a step of another candidate reads the first's run: %+v", j)
}
})
t.Run("cap", func(t *testing.T) {
kv := newRunsMem()
kv.bytes = 16 << 20
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
_, err := r.join(step("ask-a", "a"), "main.3", "n1", "run-1")
if err == nil || !strings.Contains(err.Error(), "at its cap of 16 MiB (16777216 bytes kept)") {
t.Errorf("at the cap: %v, want the fill named", err)
}
})
t.Run("kept", func(t *testing.T) {
kv := newRunsMem()
at := now
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return at }, t.Logf)
for _, c := range []string{"main.1", "main.2"} {
if _, err := r.join(step("ask-"+c, "a"), c, "n1", "run-"+c); err != nil {
t.Fatal(err)
}
}
old := now.Add(-25 * time.Hour)
_, _, _ = r.change(runKeyOf("main.1", "n1"), func(c *meshwants.CompositionRun) bool { c.Ended, c.Outcome = &old, "passed"; return true })
_, _ = kv.arm(runBoundPrefix+runKeyOf("main.1", "n1"), "run-main.1", time.Minute)
recent := now.Add(-time.Hour)
_, _, _ = r.change(runKeyOf("main.2", "n1"), func(c *meshwants.CompositionRun) bool { c.Ended, c.Outcome = &recent, "passed"; return true })
if _, err := r.join(step("ask-x", "a"), "main.9", "n1", "run-9"); err != nil {
t.Fatal(err)
}
keys, _ := kv.keys()
for _, k := range []string{runBoundPrefix, runUntakenPrefix} {
if v, _, _ := kv.get(k + runKeyOf("main.1", "n1")); v != nil {
t.Errorf("the key %s stays past its run", k+runKeyOf("main.1", "n1"))
}
}
keys = slices.DeleteFunc(keys, func(k string) bool { return !isRunKey(k) })
if want := []string{runKeyOf("main.2", "n1"), runKeyOf("main.9", "n1")}; !slices.Equal(keys, want) {
t.Errorf("keys %v, want %v: a run ended more than a day ago goes with its bound key at the next run", keys, want)
}
})
t.Run("a listing that misses a key being written", func(t *testing.T) {
kv := &missingRuns{runsMem: newRunsMem()}
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
if _, err := r.join(step("ask-a", "a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
kv.hide = runKeyOf("main.3", "n1")
if key, _, _ := r.keyOfAsk("run-1"); key != "" {
t.Fatalf("the stand's listing does not hide the key")
}
r.saw(kv.hide, true) // as the state's watch hands it
if key, _, _ := r.keyOfAsk("run-1"); key != kv.hide {
t.Errorf("the run of ask run-1 is not found by the keys the watch handed: %q", key)
}
r.saw(kv.hide, false)
if keys, _ := r.keys(); slices.Contains(keys, kv.hide) {
t.Errorf("a key the watch saw deleted is still listed: %v", keys)
}
})
// No delete lands on a value newer than the one it was judged from (the kind test 109 met in test-runs): a rerun's run
// replacing the key, and its untaken bound, stand.
for _, c := range []struct {
name string
ended time.Time
act func(r *compositionRuns, key string) error
}{
{"the sweep of a run replaced meanwhile", now.Add(-25 * time.Hour), func(r *compositionRuns, _ string) error { return r.sweep() }},
{"the end of a run replaced meanwhile", time.Time{}, func(r *compositionRuns, key string) error {
_, err := r.endRun(key, "run-1", meshwants.CompositionRunOutcomePassed, "", nil)
return err
}},
} {
t.Run(c.name, func(t *testing.T) {
kv := newRunsMem()
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
if _, err := r.join(step("ask-a", "a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
key := runKeyOf("main.3", "n1")
if !c.ended.IsZero() {
_, _, _ = r.change(key, func(run *meshwants.CompositionRun) bool { run.Ended, run.Outcome = &c.ended, "failed"; return true })
}
_, _ = kv.arm(runBoundPrefix+key, "run-1", time.Minute)
kv.beforeDel = func() { // a rerun replaces the key and arms its untaken bound
for {
raw, rev, _ := kv.get(key)
var run meshwants.CompositionRun
_ = json.Unmarshal(raw, &run)
if run.Ended == nil {
at := now
run.Ended, run.Outcome = &at, "failed"
}
fresh, _ := json.Marshal(meshwants.CompositionRun{Candidate: "main.3", Machine: "n1", Ask: "run-2", Started: &now, Readers: run.Readers})
if _, err := kv.update(key, fresh, rev); err == nil {
break
}
}
_, _ = kv.arm(runUntakenPrefix+key, "run-2", time.Hour)
}
if err := c.act(r, key); err != nil {
t.Fatal(err)
}
if run, _, ok, _ := r.read(key); !ok || run.Ask != "run-2" || run.Ended != nil {
t.Errorf("the key holds %+v (stands %v), want the rerun's running run-2", run, ok)
}
if v, _, _ := kv.get(runUntakenPrefix + key); v == nil {
t.Error("the untaken bound the rerun's run armed is deleted")
}
})
}
t.Run("settings", func(t *testing.T) {
dir := t.TempDir()
files := map[string]string{"MESH_ASSIGNER_COMPOSITION_RUNS_CAP_MIB_FILE": "8", "MESH_ASSIGNER_COMPOSITION_RUNS_KEPT_FILE": "2h",
"MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE": "2s", "MESH_ASSIGNER_RUN_UNTAKEN_BOUND_FILE": "", "MESH_ASSIGNER_BLAME_TRIES_FILE": "5"}
env := map[string]string{}
for name, content := range files {
p := filepath.Join(dir, name)
if err := os.WriteFile(p, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
env[name] = p
}
s, err := runSettingsOf(func(k string) string { return env[k] })
if err != nil || s.capBytes != 8<<20 || s.kept != 2*time.Hour || s.stepBound != 2*time.Second || s.untaken != 24*time.Hour || s.blameTries != 5 {
t.Errorf("settings %+v %v", s, err)
}
_ = os.WriteFile(env["MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE"], []byte("30"), 0o600)
if _, err := runSettingsOf(func(k string) string { return env[k] }); err == nil || !strings.Contains(err.Error(), "not a positive Go duration") {
t.Errorf("a bound of no unit: %v", err)
}
if s, err := runSettingsOf(func(string) string { return "" }); err != nil || s != defaultRunSettings {
t.Errorf("no files: %+v %v, want the defaults", s, err)
}
})
raw, _ := json.Marshal(step("x", "a"))
if !strings.Contains(string(raw), `"id":"x"`) {
t.Errorf("a reader names no ask id: %s", raw)
}
}
// Design 47 G §720, A-466s, A-468b, A-469y, A-470: a run's untaken bound is armed at its start and each reader said waiting for a
// build node; ask-taken arms the bound of composition-step-bound and ends the waits ran; each step-progress arms it again; the
// bound's expiry drops the ask and ends the run could not run naming the ask and the bound, a step-ended heard later ignored and
// journaled; the untaken bound's expiry ends it naming run-untaken-bound, the waits superseded; the readers of a run with no end
// are taken up at the start.
func TestACompositionRunsBoundTimesOwedProgress(t *testing.T) {
now := time.Date(2026, 11, 17, 12, 0, 0, 0, time.UTC)
reader := func(id string) meshwants.CompositionReader {
return readerOf(id, meshwants.StepRun{Pipeline: "main.3." + id, Step: meshwants.StepRunStepComposition, Module: id, Branch: "main", Revision: 3})
}
type said struct{ event, how, module string }
setup := func(t *testing.T) (*runsMem, *compositionRuns, *[]said, *[]string, *[]string) {
kv := newRunsMem()
var saids []said
var drops, journal []string
var mu sync.Mutex
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, func(f string, a ...any) {
mu.Lock()
journal = append(journal, fmt.Sprintf(f, a...))
mu.Unlock()
})
r.say = func(event string, body any, _ string) error {
w := body.(meshwants.Wait)
if w.On != waitOnBuildNode || w.Step != "pipeline" || w.Owner != meshwants.Seat || w.Node == "" {
t.Errorf("the wait said %+v", w)
}
saids = append(saids, said{event, w.How, w.Module})
return nil
}
r.drop = func(ask, why string) error { drops = append(drops, ask+": "+why); return nil }
return kv, r, &saids, &drops, &journal
}
key := runKeyOf("main.3", "n1")
t.Run("taken, progress, bound", func(t *testing.T) {
kv, r, saids, drops, journal := setup(t)
if _, err := r.join(reader("a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
if _, err := r.join(reader("b"), "main.3", "n1", "run-2"); err != nil {
t.Fatal(err)
}
if kv.ttl[runUntakenPrefix+key] != 24*time.Hour || !slices.Equal(*saids, []said{{meshwants.EventWaitHeld, "", "a"}, {meshwants.EventWaitHeld, "", "b"}}) {
t.Fatalf("at the start: untaken %v, said %v", kv.ttl, *saids)
}
if err := r.heardAskTaken(nodebuildgo.AskTaken{Ask: "run-1", Asker: "someone-else"}); err != nil || kv.ttl[runBoundPrefix+key] != 0 {
t.Fatalf("another asker's ask-taken arms a bound: %v %v", kv.ttl, err)
}
if err := r.heardAskTaken(nodebuildgo.AskTaken{Ask: "run-1", Asker: runAsker, Node: "b1"}); err != nil {
t.Fatal(err)
}
_, untaken, _ := kv.get(runUntakenPrefix + key)
if kv.ttl[runBoundPrefix+key] != 30*time.Minute || untaken != 0 || len(*saids) != 4 || (*saids)[2].how != "ran" || (*saids)[3].how != "ran" {
t.Fatalf("at ask-taken: %v, said %v", kv.ttl, *saids)
}
before := kv.revs[runBoundPrefix+key]
if err := r.heardProgress(nodebuildgo.StepProgress{Ask: "run-1", Asker: runAsker}); err != nil || kv.revs[runBoundPrefix+key] == before {
t.Fatalf("a step-progress does not arm the bound again: %v", err)
}
kv.expire(runBoundPrefix + key)
r.expired(runBoundPrefix + key)
run, _, _, _ := r.read(key)
if run.Outcome != meshwants.CompositionRunOutcomeCouldNotRun || run.Said != "n1: its run ask run-1 said no progress within 30m0s (composition-step-bound)" ||
len(*drops) != 1 || !strings.HasPrefix((*drops)[0], "run-1: ") {
t.Fatalf("past the bound: %+v, drops %v", run, *drops)
}
if got := judgeRun("a", run); got.couldNot != run.Said {
t.Errorf("a reader of the run past its bound: %+v", got)
}
late := buildstep.StepEnded{Ask: "run-1", Asker: runAsker, Outcome: buildstep.OutcomePassed, Output: "{}"}
if err := r.heardEnd(late, func(e buildstep.StepEnded) ([]byte, error) { return []byte(e.Output), nil }); err != nil {
t.Fatal(err)
}
if again, _, _, _ := r.read(key); again.Said != run.Said || !slices.ContainsFunc(*journal, func(l string) bool {
return strings.Contains(l, "step-ended of the run ask run-1 is ignored") && strings.Contains(l, key)
}) {
t.Errorf("a step-ended after the bound: %+v, journal %v", again, *journal)
}
if keys, _ := kv.keys(); !slices.Equal(keys, []string{key}) {
t.Errorf("the bound keys stay past the run's end: %v", keys)
}
})
t.Run("untaken", func(t *testing.T) {
kv, r, saids, drops, _ := setup(t)
if _, err := r.join(reader("a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
kv.expire(runUntakenPrefix + key)
r.expired(runUntakenPrefix + key)
run, _, _, _ := r.read(key)
if run.Outcome != meshwants.CompositionRunOutcomeCouldNotRun || run.Said != "n1: its run ask run-1 was taken by no build node within 24h0m0s (run-untaken-bound)" ||
len(*drops) != 1 || !slices.Equal(*saids, []said{{meshwants.EventWaitHeld, "", "a"}, {meshwants.EventWaitEnded, "superseded", "a"}}) {
t.Errorf("untaken past its bound: %+v, drops %v, said %v", run, *drops, *saids)
}
r.expired(runUntakenPrefix + key) // a run ended takes nothing
if len(*drops) != 1 {
t.Errorf("an ended run is dropped again: %v", *drops)
}
})
// A-470n, A-470o: a bound holds its run's ask, armed for it alone; a late signal of a run replaced moves nothing and is
// journaled; an expiry ends the run only while its bound is still absent and the run taken (bound.) or untaken (untaken.).
rerun := func(kv *runsMem, r *compositionRuns) func() { // a rerun replaces run-1 by run-2, its bounds gone, run-2's untaken armed
return func() {
_, _, _ = r.change(key, func(c *meshwants.CompositionRun) bool { c.Ask = "run-2"; return true })
kv.expire(runBoundPrefix + key)
_, _ = kv.arm(runUntakenPrefix+key, "run-2", time.Hour)
}
}
taken := func(t *testing.T, r *compositionRuns) {
if err := r.heardAskTaken(nodebuildgo.AskTaken{Ask: "run-1", Asker: runAsker, Node: "b1"}); err != nil {
t.Fatal(err)
}
}
for _, c := range []struct {
name string
act func(t *testing.T, kv *runsMem, r *compositionRuns)
ended bool
bound string // the ask the bound key holds, "" for none
untaken string
journal string
}{
{"a late ask-taken of a run replaced as it is heard", func(t *testing.T, kv *runsMem, r *compositionRuns) {
kv.beforeArm = rerun(kv, r)
taken(t, r)
}, false, "", "run-2", "the ask-taken of the run ask run-1 is late"},
{"a late step-progress of a run replaced as it is heard", func(t *testing.T, kv *runsMem, r *compositionRuns) {
taken(t, r)
kv.beforeArm = rerun(kv, r)
if err := r.heardProgress(nodebuildgo.StepProgress{Ask: "run-1", Asker: runAsker}); err != nil {
t.Fatal(err)
}
kv.expire(runBoundPrefix + key) // a bound left standing would end run-2 here
r.expired(runBoundPrefix + key)
}, false, "", "run-2", "the step-progress of the run ask run-1 is late"},
{"a step-progress of a run gone", func(t *testing.T, kv *runsMem, r *compositionRuns) {
taken(t, r)
rerun(kv, r)()
if err := r.heardProgress(nodebuildgo.StepProgress{Ask: "run-1", Asker: runAsker}); err != nil {
t.Fatal(err)
}
}, false, "", "run-2", "the step-progress of the run ask run-1 is late"},
{"an expiry heard after the bound was armed again", func(t *testing.T, kv *runsMem, r *compositionRuns) {
taken(t, r)
r.expired(runBoundPrefix + key)
}, false, "run-1", "", "the expiry of " + runBoundPrefix + key + " ends nothing"},
{"the bound's expiry of a run not taken", func(t *testing.T, kv *runsMem, r *compositionRuns) {
_, _ = kv.arm(runBoundPrefix+key, "run-1", time.Minute)
kv.expire(runBoundPrefix + key)
r.expired(runBoundPrefix + key)
}, false, "", "run-1", "the expiry of " + runBoundPrefix + key + " ends nothing"},
{"the untaken expiry of a run taken", func(t *testing.T, kv *runsMem, r *compositionRuns) {
_, _ = kv.arm(runBoundPrefix+key, "run-1", time.Minute)
kv.expire(runUntakenPrefix + key)
r.expired(runUntakenPrefix + key)
}, false, "run-1", "", "the expiry of " + runUntakenPrefix + key + " ends nothing"},
{"an end of another ask", func(t *testing.T, kv *runsMem, r *compositionRuns) {
if ended, err := r.end(key, "run-0", meshwants.CompositionRunOutcomePassed, "", nil); ended || err != nil {
t.Errorf("the end of run-0 ended the run of run-1: %v %v", ended, err)
}
}, false, "", "run-1", ""},
{"the ask-taken of the run's own ask", func(t *testing.T, kv *runsMem, r *compositionRuns) {
taken(t, r)
}, false, "run-1", "", ""},
} {
t.Run(c.name, func(t *testing.T) {
kv, r, _, _, journal := setup(t)
if _, err := r.join(reader("a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
c.act(t, kv, r)
run, _, _, _ := r.read(key)
if (run.Ended != nil) != c.ended {
t.Errorf("the run ended %v (%s), want %v", run.Ended != nil, run.Said, c.ended)
}
for prefix, want := range map[string]string{runBoundPrefix: c.bound, runUntakenPrefix: c.untaken} {
if held, _, _, _ := r.boundOf(prefix + key); held != want {
t.Errorf("%s holds the ask %q, want %q", prefix+key, held, want)
}
}
if c.journal != "" && !slices.ContainsFunc(*journal, func(l string) bool { return strings.Contains(l, c.journal) }) {
t.Errorf("the journal %v does not say %q", *journal, c.journal)
}
})
}
// A-469y, A-470u: a step asked again that joins a running run not yet taken, beside a reader or in its pipeline's former
// reader's place, says its own wait held, as join does; the former reader's wait, and that of a reader leaving, ends at once
// superseded.
held := func(m string) said { return said{meshwants.EventWaitHeld, "", m} }
gone := func(m string) said { return said{meshwants.EventWaitEnded, "superseded", m} }
for _, c := range []struct {
name string
act func(t *testing.T, r *compositionRuns)
want []said
}{
{"a rerun joins", func(t *testing.T, r *compositionRuns) {
again := reader("b")
again.Rerun = true
if _, err := r.rejoin(again, "main.3", "n1", "run-2"); err != nil {
t.Fatal(err)
}
}, []said{held("a"), held("b")}},
{"a rerun takes its former place", func(t *testing.T, r *compositionRuns) {
again := reader("a2")
again.Pipeline, again.Module, again.Rerun = "main.3.a", "a2", true
if _, err := r.rejoin(again, "main.3", "n1", "run-2"); err != nil {
t.Fatal(err)
}
}, []said{held("a"), gone("a"), held("a2")}},
{"a reader leaves", func(t *testing.T, r *compositionRuns) {
if _, err := r.join(reader("b"), "main.3", "n1", "run-2"); err != nil {
t.Fatal(err)
}
if _, err := r.leave("main.3.a", nil); err != nil {
t.Fatal(err)
}
}, []said{held("a"), held("b"), gone("a")}},
} {
t.Run(c.name, func(t *testing.T) {
_, r, saids, _, _ := setup(t)
if _, err := r.join(reader("a"), "main.3", "n1", "run-1"); err != nil {
t.Fatal(err)
}
c.act(t, r)
if !slices.Equal(*saids, c.want) {
t.Errorf("said %v, want %v", *saids, c.want)
}
})
}
t.Run("taken up at the start", func(t *testing.T) {
_, r, _, _, _ := setup(t)
for _, c := range []struct{ id, machine string }{{"a", "n1"}, {"a", "n2"}, {"b", "n1"}} {
if _, err := r.join(reader(c.id), "main.3", c.machine, "run-"+c.machine); err != nil {
t.Fatal(err)
}
}
ended := now
_, _, _ = r.change(runKeyOf("main.3", "n2"), func(c *meshwants.CompositionRun) bool { c.Ended, c.Outcome = &ended, "passed"; return true })
// A rerun's run carrying readers whose steps said their end (A-470l, A-470m): c passed and is not taken up, d is marked rerun.
_, _, _ = r.change(runKeyOf("main.3", "n1"), func(run *meshwants.CompositionRun) bool {
for _, id := range []string{"c", "d"} {
c := reader(id)
c.Ended, c.Rerun = &ended, id == "d"
run.Readers = append(run.Readers, c)
}
return true
})
readers, err := r.readersOfOpen()
if err != nil || len(readers) != 3 || readers[0].ID != "a" || readers[1].ID != "b" || readers[2].ID != "d" {
t.Fatalf("the readers taken up: %+v %v", readers, err)
}
if keys, _ := r.keysOf("main.3", "a"); !slices.Equal(keys, []string{runKeyOf("main.3", "n1"), runKeyOf("main.3", "n2")}) {
t.Errorf("the runs of reader a: %v", keys)
}
// A-470q: a reader with no `ended` says its first end under its ask's id; d, marked rerun, says its end again under
// `<id>.<the open run's ask>`.
ids := make(chan [2]string, 3)
steps := &compositionSteps{node: "n1", now: func() time.Time { return now }, logf: t.Logf,
resume: func(context.Context, string, meshwants.StepRun) stepOutcome {
return stepOutcome{Outcome: meshwants.StepEndedOutcomePassed}
},
say: func(_ string, body any, id string) error {
ids <- [2]string{body.(meshwants.StepEnded).ID, id}
return nil
}}
steps.takeUp(context.Background(), r)
got := map[string]string{}
for range 3 {
select {
case e := <-ids:
got[e[0]] = e[1]
case <-time.After(5 * time.Second):
t.Fatalf("the ends of the take-up: %v", got)
}
}
if want := map[string]string{"a": "a", "b": "b", "d": "d.run-n1"}; !maps.Equal(got, want) {
t.Errorf("the message ids of the take-up's ends: %v, want %v", got, want)
}
})
}
// missingRuns is runsMem whose listing misses one key, as a bucket's listing misses a key being written at that moment.
type missingRuns struct {
*runsMem
hide string
}
func (k *missingRuns) keys() ([]string, error) {
all, err := k.runsMem.keys()
return slices.DeleteFunc(all, func(key string) bool { return key == k.hide }), err
}
@@ -0,0 +1,162 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The composition runs' read and drop (design 47 G §728; A-469f, A-469u, A-469v): the seat mesh-wants' verbs `compositions`
// (class read) and `compositions-drop` (class act, §391's default callers), answered by the assigner's process, which keeps the
// runs and says the readers' ends; the module's tools `assigner_compositions` and `assigner_compositions_drop` call them.
// The tools that call the two verbs.
const (
compositionsTool = "assigner_compositions"
compositionsDropTool = "assigner_compositions_drop"
)
// compositions answers each composition run the state keeps, under §719's field names plus its key, with the candidate and the
// machine named only those, and the state's fill against its cap.
func (r *compositionRuns) compositions(in meshwants.CompositionsInput) (meshwants.CompositionsAnswer, error) {
all, err := r.keys()
if err != nil {
return meshwants.CompositionsAnswer{}, fmt.Errorf("the state %s is not listed: %w", "composition-runs", err)
}
slices.Sort(all)
out := meshwants.CompositionsAnswer{Runs: []meshwants.CompositionRun{}}
for _, key := range all {
if !isRunKey(key) {
continue
}
run, _, ok, err := r.read(key)
if err != nil || !ok || (in.Candidate != "" && run.Candidate != in.Candidate) || (in.Machine != "" && run.Machine != in.Machine) {
continue
}
run.Key = key
out.Runs = append(out.Runs, run)
}
out.Fill, err = r.fill()
return out, err
}
// drop drops one composition run by hand (§728): the act and why written in the hand-act log first (record), then a running run's
// ask dropped, each of its readers' waits stopped and, for each reader whose end is owed (no `ended`, or marked rerun), its end
// said could not run naming the drop, once (A-470r), and the key deleted with its bound keys. It answers the entry deleted and
// the readers it ended.
func (c *compositionSteps) drop(r *compositionRuns, record func(meshhandactsclient.Request) error, caller string, in meshwants.CompositionsDropInput) (meshwants.CompositionsDropAnswer, error) {
why := strings.TrimSpace(in.Why)
if in.Candidate == "" || in.Machine == "" || why == "" {
return meshwants.CompositionsDropAnswer{}, errors.New("name the candidate, the machine and why the composition run is dropped")
}
key := runKeyOf(in.Candidate, in.Machine)
run, _, ok, err := r.read(key)
if err != nil {
return meshwants.CompositionsDropAnswer{}, err
}
if !ok {
return meshwants.CompositionsDropAnswer{}, fmt.Errorf("no composition run of %s on %s is kept", in.Candidate, in.Machine)
}
if err := record(meshhandactsclient.Request{What: meshwants.VerbCompositionsDrop + " " + key, Why: why + " (" + caller + ")", Cause: "compositions-drop"}); err != nil {
return meshwants.CompositionsDropAnswer{}, fmt.Errorf("the hand act was not recorded, so nothing was dropped: %w", err)
}
said := fmt.Sprintf("%s: the composition run %s was dropped by hand: %s", in.Machine, key, why)
answer := meshwants.CompositionsDropAnswer{Run: run}
answer.Run.Key = key
if run.Ended == nil {
if r.drop != nil {
if err := r.drop(run.Ask, said); err != nil {
r.logf("the drop of the run ask %s of the composition run %s is not answered: %v", run.Ask, key, err)
}
}
ids := make([]string, 0, len(run.Readers))
for _, reader := range run.Readers {
ids = append(ids, reader.ID)
}
c.leave(ids) // their waits say no end of their own
// The run ends without moving its readers: the drop says each owed end itself, once (A-470r).
if _, err := r.endRunMoving(key, run.Ask, meshwants.CompositionRunOutcomeCouldNotRun, said, nil, r.untaken(key, run.Ask), false); err != nil {
return answer, err
}
shared := []meshwants.Shared{{Machine: run.Machine, Run: key, Ask: run.Ask}}
for _, reader := range run.Readers {
if reader.Ended != nil && !reader.Rerun {
continue // it ended passed already and keeps its verdict (A-470r)
}
again := "" // a reader whose step said its end already says it again under the dropped run's ask (A-470q)
if reader.Ended != nil {
again = run.Ask
}
c.sayEnd(reader.ID, stepRunOf(reader), nil, stepOutcome{Outcome: meshwants.StepEndedOutcomeCouldNotRun, Said: said, Shared: shared}, again)
answer.Ended = append(answer.Ended, reader)
}
}
// The dropped run's key goes while it holds that run: a run started on it since stands.
err = r.deleteRun(key, func(c meshwants.CompositionRun) bool { return c.Ask == run.Ask })
r.heard()
return answer, err
}
// compositionVerbs are the process's handlers of the seat mesh-wants' verbs compositions and compositions-drop.
func compositionVerbs(r *compositionRuns, c *compositionSteps, record func(meshhandactsclient.Request) error) map[string]process.SeatHandler {
return map[string]process.SeatHandler{
meshwants.VerbCompositions: func(_ string, raw json.RawMessage) (any, error) {
var in meshwants.CompositionsInput
if len(raw) > 0 {
if err := json.Unmarshal(raw, &in); err != nil {
return nil, fmt.Errorf("the arguments are not JSON: %w", err)
}
}
return r.compositions(in)
},
meshwants.VerbCompositionsDrop: func(caller string, raw json.RawMessage) (any, error) {
var in meshwants.CompositionsDropInput
if err := json.Unmarshal(raw, &in); err != nil {
return nil, fmt.Errorf("the arguments are not JSON: %w", err)
}
return c.drop(r, record, caller, in)
},
}
}
// compositionsTools are the module's tools of §728's names, each calling its verb (A-469f).
func compositionsTools(ask func(string, any) (json.RawMessage, error)) []stdio.Tool {
call := func(verb string, args map[string]any, names ...string) (any, error) {
in := map[string]any{}
for _, n := range names {
if v := arg(args, n); v != "" {
in[n] = v
}
}
raw, err := ask("seat:"+meshwants.Seat+"."+verb, in)
if err != nil {
return nil, err
}
var out any
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("%s answered what is not JSON: %w", verb, err)
}
return out, nil
}
return []stdio.Tool{
{Name: compositionsTool, Description: "The composition runs the assigner keeps, one per candidate and machine (design 47 G §719, §728): each with " +
"its program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap; with candidate or machine, only those. Read-only.",
Input: schema(map[string]string{"candidate": "the candidate's key: <branch>.<revision>, or run-<id>", "machine": "a machine"}, nil),
Run: func(args map[string]any) (any, error) {
return call(meshwants.VerbCompositions, args, "candidate", "machine")
}},
{Name: compositionsDropTool, Description: "Drop one composition run by hand (design 47 G §728): written in the hand-act log first, then its ask " +
"is dropped, each reader ended could not run naming the drop, and the key deleted.",
Input: schema(map[string]string{"candidate": "the candidate's key", "machine": "the machine", "why": "why the run is dropped"}, []string{"candidate", "machine", "why"}),
Run: func(args map[string]any) (any, error) {
return call(meshwants.VerbCompositionsDrop, args, "candidate", "machine", "why")
}},
}
}
@@ -0,0 +1,91 @@
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
)
// Design 47 G §728, A-469f, A-469u, A-469v: compositions answers each run with its key and the fill, only those of a candidate or
// machine when named; compositions-drop writes the hand act first (nothing is dropped when it is not recorded), drops a running
// run's ask, ends each reader could not run naming the drop and deletes the key with its bound keys; the tools call the verbs.
func TestTheCompositionRunsAreReadAndDroppedByTheirVerbs(t *testing.T) {
now := time.Date(2026, 11, 17, 12, 0, 0, 0, time.UTC)
kv := newRunsMem()
r := newCompositionRuns(kv, defaultRunSettings, func() time.Time { return now }, t.Logf)
var drops []string
r.drop = func(ask, why string) error { drops = append(drops, ask); return nil }
ends := newSaidEvents()
var endIDs []string // the message id of each end said
c := &compositionSteps{node: "n1", now: func() time.Time { return now }, logf: t.Logf, say: func(event string, body any, id string) error {
endIDs = append(endIDs, id)
return ends.say(event, body, id)
}}
moves := 0
move := func(meshwants.CompositionRun) { moves++ }
r.moved.Store(&move)
for _, m := range []string{"n1", "n2"} {
if _, err := r.join(readerOf("ask-a", meshwants.StepRun{Pipeline: "main.3.a", Step: meshwants.StepRunStepComposition, Module: "a"}), "main.3", m, "run-"+m); err != nil {
t.Fatal(err)
}
}
// A-470r: n2's run carries a reader that ended passed (p) and one marked rerun (q); only q's end is owed, said once.
_, _, _ = r.change(runKeyOf("main.3", "n2"), func(run *meshwants.CompositionRun) bool {
for _, id := range []string{"ask-p", "ask-q"} {
c := readerOf(id, meshwants.StepRun{Pipeline: "main.3." + id, Step: meshwants.StepRunStepComposition, Module: id})
c.Ended, c.Rerun = &now, id == "ask-q"
run.Readers = append(run.Readers, c)
}
return true
})
var recorded []meshhandactsclient.Request
record := func(q meshhandactsclient.Request) error { recorded = append(recorded, q); return nil }
verbs := compositionVerbs(r, c, record)
out, err := verbs[meshwants.VerbCompositions]("op", json.RawMessage(`{"machine":"n2"}`))
read, _ := out.(meshwants.CompositionsAnswer)
if err != nil || len(read.Runs) != 1 || read.Runs[0].Key != runKeyOf("main.3", "n2") || read.Runs[0].Ask != "run-n2" || len(read.Runs[0].Readers) != 3 ||
read.Fill.Store != "composition-runs" || read.Fill.Cap != 16<<20 || read.Fill.Bytes == 0 {
t.Fatalf("compositions = %+v %v", out, err)
}
refuse := compositionVerbs(r, c, func(meshhandactsclient.Request) error { return errors.New("no holder") })
if _, err := refuse[meshwants.VerbCompositionsDrop]("op", json.RawMessage(`{"candidate":"main.3","machine":"n2","why":"stuck"}`)); err == nil ||
!strings.Contains(err.Error(), "nothing was dropped") || len(drops) != 0 {
t.Fatalf("a drop not recorded: %v, drops %v", err, drops)
}
out, err = verbs[meshwants.VerbCompositionsDrop]("op@n1", json.RawMessage(`{"candidate":"main.3","machine":"n2","why":"stuck"}`))
dropped, _ := out.(meshwants.CompositionsDropAnswer)
if err != nil || dropped.Run.Key != runKeyOf("main.3", "n2") || len(dropped.Ended) != 2 || dropped.Ended[0].ID != "ask-a" || dropped.Ended[1].ID != "ask-q" ||
moves != 0 || len(endIDs) != 2 || endIDs[0] != "ask-a" || endIDs[1] != "ask-q.run-n2" ||
len(recorded) != 1 || recorded[0].Why != "stuck (op@n1)" || len(drops) != 1 || drops[0] != "run-n2" {
t.Fatalf("compositions-drop = %+v %v, recorded %+v, drops %v, moves %d, ends %v", out, err, recorded, drops, moves, endIDs)
}
end, _ := ends.said[meshwants.EventStepEnded+".n1"].(meshwants.StepEnded)
if end.ID != "ask-q" || end.Outcome != meshwants.StepEndedOutcomeCouldNotRun || !strings.Contains(end.Said, "dropped by hand: stuck") ||
len(end.Shared) != 1 || end.Shared[0].Ask != "run-n2" {
t.Errorf("the reader's end %+v", end)
}
if keys, _ := kv.keys(); len(keys) != 2 || keys[0] != runKeyOf("main.3", "n1") {
t.Errorf("keys after the drop %v, want n1's run and its untaken key", keys)
}
// The tools call the verbs with what they were given.
var asked []string
tools := compositionsTools(func(key string, body any) (json.RawMessage, error) {
raw, _ := json.Marshal(body)
asked = append(asked, key+" "+string(raw))
return json.RawMessage(`{"runs":[],"fill":{"store":"composition-runs","bytes":0,"cap":1}}`), nil
})
for _, tool := range tools {
if _, err := tool.Run(map[string]any{"candidate": "main.3", "machine": "n1", "why": "x"}); err != nil {
t.Fatal(err)
}
}
if len(tools) != 2 || tools[0].Name != compositionsTool || tools[1].Name != compositionsDropTool ||
asked[0] != `seat:mesh-wants.compositions {"candidate":"main.3","machine":"n1"}` || asked[1] != `seat:mesh-wants.compositions-drop {"candidate":"main.3","machine":"n1","why":"x"}` {
t.Errorf("the tools asked %v", asked)
}
}
@@ -23,6 +23,8 @@ type stepOutcome struct {
Outcome string
Said string
Warnings []string
// Shared names per machine composed the composition run the step read and its ask (design 47 G §723, §729).
Shared []meshwants.Shared
}
// compositionSteps takes the composition steps and says their ends.
@@ -30,14 +32,21 @@ type compositionSteps struct {
node string
// say says one event of the seat mesh-wants (wantsPublisher).
say publisher
// run runs one step and answers what it came to.
run func(ctx context.Context, id string, run meshwants.StepRun) stepOutcome
now func() time.Time
logf func(string, ...any)
// run runs one step and answers what it came to; resume judges a step taken up at the start from its runs (takeUp).
run func(ctx context.Context, id string, run meshwants.StepRun) stepOutcome
resume func(ctx context.Context, id string, run meshwants.StepRun) stepOutcome
// ended writes, once the step asked under id said its end, that time on its reader in each run naming it (A-470m); nil
// writes none.
ended func(id string, run meshwants.StepRun, at time.Time)
now func() time.Time
logf func(string, ...any)
// asker is the name the step's progress names as its asker, read when it is said (stepAskerOf).
asker func() string
// ticks is a ticker of every d, its channel and its stop (time.NewTicker); nil is the clock's. A test hands its own.
ticks func(d time.Duration) (<-chan time.Time, func())
mu sync.Mutex
// stops ends the wait of a running step by its ask's id, when its pipeline is superseded or cancelled (leave).
stops map[string]context.CancelFunc
}
// stepAskerOf is the asker a composition step's progress names: the one caller the seat mesh-wants' definition lists for its
@@ -69,10 +78,27 @@ func stepAskerOf(definition func() (seatdef.Definition, bool)) func() string {
// progressEvery is how often a running composition step says its progress (design 47 G §496, A-262).
const progressEvery = time.Minute
// stepPhase is the phase a running step is in: fetch, build, run or report (§496), set by the run through its context.
// stepPhase is the phase a running step is in: fetch, build, run or report (§496), set by the run through its context, and the
// composition runs it waits on (§723, §729).
type stepPhase struct {
mu sync.Mutex
phase string
mu sync.Mutex
phase string
shared []meshwants.Shared
}
// setShared names the runs the step ctx runs waits on, when it runs one.
func setShared(ctx context.Context, shared []meshwants.Shared) {
if p, ok := ctx.Value(phaseKey{}).(*stepPhase); ok {
p.mu.Lock()
p.shared = shared
p.mu.Unlock()
}
}
func (p *stepPhase) waitsOn() []meshwants.Shared {
p.mu.Lock()
defer p.mu.Unlock()
return p.shared
}
type phaseKey struct{}
@@ -100,23 +126,97 @@ func (c *compositionSteps) take(ctx context.Context, id string, data []byte) boo
c.logf("a step ask that is no composition step with an id is left: %v", err)
return false
}
go c.runOne(ctx, id, run)
go c.say1(ctx, id, run, c.run, "")
return true
}
// runOne runs one step and says its end.
func (c *compositionSteps) runOne(ctx context.Context, id string, run meshwants.StepRun) {
// say1 runs one step with how and says its end, naming in `shared` the runs it read (§723); again is, for an end said again, the
// ask of the run whose end makes it say again (sayEnd). A step whose wait was ended by its pipeline's supersede or cancel (leave)
// says nothing: the runner ended it already.
func (c *compositionSteps) say1(ctx context.Context, id string, run meshwants.StepRun, how func(context.Context, string, meshwants.StepRun) stepOutcome, again string) {
started := c.now().UTC()
ctx, stopWait := context.WithCancel(ctx)
c.mu.Lock()
if c.stops == nil {
c.stops = map[string]context.CancelFunc{}
}
if c.stops[id] != nil { // the step asked under id is running already: it says its own end
c.mu.Unlock()
stopWait()
return
}
c.stops[id] = stopWait
c.mu.Unlock()
defer func() {
c.mu.Lock()
delete(c.stops, id)
c.mu.Unlock()
stopWait()
}()
phase := &stepPhase{phase: meshwants.StepProgressEventPhaseFetch}
stop := c.sayProgress(id, phase)
o := c.run(context.WithValue(ctx, phaseKey{}, phase), id, run)
o := how(context.WithValue(ctx, phaseKey{}, phase), id, run)
stop()
if ctx.Err() != nil {
c.logf("the composition step %s of %s says no end: its wait ended (%v)", id, run.Pipeline, context.Cause(ctx))
return
}
c.sayEnd(id, run, &started, o, again)
}
// sayEnd says the end of the step asked under id as the seat's step-ended.<node>, its id the ask's; once said, its time is
// written as its reader's `ended` (A-470m). Its first end's message id is the ask's id; an end said again is published under
// `<ask id>.<again>`, again the ask of the run whose end makes it say again, so the bus's deduplication never drops it (A-470q).
func (c *compositionSteps) sayEnd(id string, run meshwants.StepRun, started *time.Time, o stepOutcome, again string) {
ended := c.now().UTC()
if started == nil {
started = &ended
}
end := meshwants.StepEnded{V: 1, ID: id, Pipeline: run.Pipeline, Step: meshwants.StepEndedStepComposition, Module: run.Module,
Commit: run.Commit, Fingerprint: run.Fingerprint, Node: c.node, Outcome: o.Outcome, Said: o.Said, Warnings: o.Warnings,
Rerun: run.Rerun, By: run.By, Started: &started, Ended: &ended}
if err := c.say(meshwants.EventStepEnded+"."+c.node, end, id); err != nil {
Rerun: run.Rerun, By: run.By, Started: started, Ended: &ended, Shared: o.Shared}
msgID := id
if again != "" {
msgID = id + "." + again
}
if err := c.say(meshwants.EventStepEnded+"."+c.node, end, msgID); err != nil {
c.logf("saying the end of the composition step %s: %v", id, err)
return
}
if c.ended != nil {
c.ended(id, run, ended)
}
}
// move says the end again of each reader a rerun carried into an ended run, marked rerun (§727, A-470d): its machines judged
// from the candidate's keys naming it, the run's new ask in `shared`. A reader running here says its own end.
func (c *compositionSteps) move(ctx context.Context) func(run meshwants.CompositionRun) {
return func(run meshwants.CompositionRun) {
for _, reader := range run.Readers {
if !reader.Rerun || c.running(reader.ID) {
continue
}
c.logf("the composition step %s of %s moves with the composition run of %s (ask %s)", reader.ID, reader.Pipeline, run.Machine, run.Ask)
go c.say1(ctx, reader.ID, stepRunOf(reader), c.resume, run.Ask) // said again under the moved run's ask (A-470q)
}
}
}
// running is whether the step asked under id runs here.
func (c *compositionSteps) running(id string) bool {
c.mu.Lock()
defer c.mu.Unlock()
return c.stops[id] != nil
}
// leave ends the waits of the steps asked under ids (§723: a superseded or cancelled step leaves its runs).
func (c *compositionSteps) leave(ids []string) {
c.mu.Lock()
defer c.mu.Unlock()
for _, id := range ids {
if stop := c.stops[id]; stop != nil {
stop()
}
}
}
@@ -139,7 +239,7 @@ func (c *compositionSteps) sayProgress(id string, phase *stepPhase) (stop func()
return
case <-tick:
at := c.now().UTC()
p := meshwants.StepProgressEvent{Ask: id, Asker: c.asker(), Phase: phase.get(), At: at}
p := meshwants.StepProgressEvent{Ask: id, Asker: c.asker(), Phase: phase.get(), At: at, Shared: phase.waitsOn()}
if err := c.say(meshwants.EventStepProgress+"."+c.node, p, id+".progress."+at.Format(time.RFC3339Nano)); err != nil {
c.logf("saying the progress of the composition step %s: %v", id, err)
}
@@ -146,6 +146,9 @@ func TestTheCompositionStepComposesEachMachineThroughTheBuildSeat(t *testing.T)
if err != nil || len(req.Machines) != 1 {
t.Fatalf("the request of %s: %+v %v", a.ID, req, err)
}
if len(req.Seats) == 0 { // the revision names no definition: the request carries the trunk seat list's (A-467)
t.Errorf("the request of %s carries no seat definition", a.ID)
}
node := req.Machines[0]
programs[node] = a.Run.Program.Commit
m := composer.Machine{Composes: true}
@@ -165,7 +168,8 @@ func TestTheCompositionStepComposesEachMachineThroughTheBuildSeat(t *testing.T)
t.Fatal("the composition step said no end")
}
if e.ID != "step-ask-1" || e.Outcome != meshwants.StepEndedOutcomeFailed || !strings.Contains(e.Said, "n2 does not compose") ||
strings.Contains(e.Said, "n1") || len(e.Warnings) != 1 || !strings.Contains(e.Warnings[0], "n3 is away") {
strings.Contains(e.Said, "n1") || len(e.Warnings) != 2 || !strings.Contains(e.Warnings[0], "n3 is away") ||
e.Warnings[1] != "n2: the composer names no culprits" {
t.Fatalf("the step ended %+v", e)
}
if programs["n1"] != engineCommit || programs["n2"] != engineCommit {
@@ -177,9 +181,24 @@ func TestTheCompositionStepComposesEachMachineThroughTheBuildSeat(t *testing.T)
t.Error("the machine away was asked for its input")
}
}
keys, _ := b.buckets[buckets.Inputs()].Keys(b.ctx)
if len(keys) != 0 {
t.Errorf("the inputs stay kept past their step-ended: %v", keys)
// The run's end comes before the input's delete (A-470o, A-470p): the inputs go after the step's end, heard by their watch.
inputs := b.buckets[buckets.Inputs()]
w, err := inputs.WatchAll(b.ctx)
if err != nil {
t.Fatal(err)
}
defer func() { _ = w.Stop() }()
deadline := time.After(10 * time.Second)
for {
keys, _ := inputs.Keys(b.ctx)
if len(keys) == 0 {
break
}
select {
case <-w.Updates():
case <-deadline:
t.Fatalf("the inputs stay kept past their step-ended: %v", keys)
}
}
}
@@ -4,6 +4,8 @@ import (
"context"
"encoding/json"
"git.novox.be/novox/mesh-sdk/go/seatdef"
"slices"
"strings"
"sync"
"testing"
"time"
@@ -40,7 +42,8 @@ func TestTheCompositionStepIsTakenAndItsEndSaid(t *testing.T) {
c := &compositionSteps{node: "n1", say: said.say, now: func() time.Time { return at }, logf: t.Logf,
run: func(_ context.Context, id string, run meshwants.StepRun) stepOutcome {
asked = run
return stepOutcome{Outcome: meshwants.StepEndedOutcomeFailed, Said: "a and b own /etc/x", Warnings: []string{"n3 asleep"}}
return stepOutcome{Outcome: meshwants.StepEndedOutcomeFailed, Said: "a and b own /etc/x", Warnings: []string{"n3 asleep"},
Shared: []meshwants.Shared{{Machine: "n1", Run: "main=2e4.n1", Ask: "run-1"}}}
}}
other, _ := json.Marshal(map[string]any{"pipeline": "main.4.m", "step": "build", "module": "m"})
if c.take(context.Background(), "ask-0", other) {
@@ -67,7 +70,8 @@ func TestTheCompositionStepIsTakenAndItsEndSaid(t *testing.T) {
if asked.Module != "m" || end.ID != "ask-1" || end.Pipeline != "main.4.m" || end.Step != meshwants.StepEndedStepComposition ||
end.Module != "m" || end.Commit != "c0ffee" || end.Fingerprint != "fp1" || end.Node != "n1" ||
end.Outcome != meshwants.StepEndedOutcomeFailed || end.Said != "a and b own /etc/x" || len(end.Warnings) != 1 ||
!end.Rerun || end.By != "lead@n1" || end.Started == nil || end.Ended == nil {
!end.Rerun || end.By != "lead@n1" || end.Started == nil || end.Ended == nil ||
len(end.Shared) != 1 || end.Shared[0] != (meshwants.Shared{Machine: "n1", Run: "main=2e4.n1", Ask: "run-1"}) { // §723, §729
t.Fatalf("step-ended %+v", end)
}
}
@@ -105,6 +109,7 @@ func TestARunningCompositionStepSaysItsProgress(t *testing.T) {
},
run: func(ctx context.Context, _ string, _ meshwants.StepRun) stepOutcome {
setPhase(ctx, meshwants.StepProgressEventPhaseRun)
setShared(ctx, []meshwants.Shared{{Machine: "n1", Run: "main=2e4.n1", Ask: "run-1"}})
<-release
return stepOutcome{Outcome: meshwants.StepEndedOutcomePassed}
}}
@@ -122,7 +127,8 @@ func TestARunningCompositionStepSaysItsProgress(t *testing.T) {
said.mu.Lock()
p, ok := said.said[progress].(meshwants.StepProgressEvent)
said.mu.Unlock()
if !ok || p.Ask != "ask-1" || p.Asker != "mesh-pipeline-runner" || p.Phase != meshwants.StepProgressEventPhaseRun || !p.At.Equal(at) {
if !ok || p.Ask != "ask-1" || p.Asker != "mesh-pipeline-runner" || p.Phase != meshwants.StepProgressEventPhaseRun || !p.At.Equal(at) ||
len(p.Shared) != 1 || p.Shared[0].Ask != "run-1" { // the runs the step waits on (§723)
t.Fatalf("step-progress %+v", p)
}
close(release)
@@ -135,3 +141,70 @@ func TestARunningCompositionStepSaysItsProgress(t *testing.T) {
t.Fatal("the ticker is not stopped when the step has ended")
}
}
// Design 47 G §723, A-469n, A-469z: a superseded or cancelled pipeline's composition step leaves the runs it reads and says no end;
// a run another pipeline's step reads goes on for it; a run left with no reader has its ask dropped and ends could not run, "no
// reader is left", so a later step starts a new run.
func TestASupersededStepLeavesItsRuns(t *testing.T) {
now := time.Date(2026, 11, 17, 12, 0, 0, 0, time.UTC)
r := newCompositionRuns(newRunsMem(), defaultRunSettings, func() time.Time { return now }, t.Logf)
var drops []string
r.drop = func(ask, why string) error { drops = append(drops, ask+": "+why); return nil }
step := func(pipeline string) meshwants.StepRun {
return meshwants.StepRun{Pipeline: pipeline, Step: meshwants.StepRunStepComposition, Module: pipeline, Branch: "main", Revision: 3}
}
said := newSaidEvents()
c := &compositionSteps{node: "n1", say: said.say, now: func() time.Time { return now }, logf: t.Logf,
run: func(ctx context.Context, id string, s meshwants.StepRun) stepOutcome {
j, err := r.join(readerOf(id, s), "main.3", "n1", "run-"+id)
if err != nil {
t.Error(err)
}
r.await(ctx, []string{j.key})
return stepOutcome{Outcome: meshwants.StepEndedOutcomePassed}
}}
body := func(s meshwants.StepRun) []byte { raw, _ := json.Marshal(s); return raw }
for _, p := range []string{"main.3.a", "main.3.b"} {
if !c.take(context.Background(), "ask-"+p, body(step(p))) {
t.Fatal("not taken")
}
}
key := runKeyOf("main.3", "n1")
for {
wake := r.wake()
if run, _, _, _ := r.read(key); len(run.Readers) == 2 {
break
}
<-wake
}
ids, err := r.leave("main.3.a", c.leave)
run, _, _, _ := r.read(key)
if err != nil || !slices.Equal(ids, []string{"ask-main.3.a"}) || run.Ended != nil || len(run.Readers) != 1 || len(drops) != 0 {
t.Fatalf("after a's supersede: %v %v, run %+v, drops %v: want the run going on for b", ids, err, run, drops)
}
_, err = r.leave("main.3.b", c.leave)
run, _, _, _ = r.read(key)
if err != nil || run.Outcome != meshwants.CompositionRunOutcomeCouldNotRun || !strings.Contains(run.Said, "no reader is left") || len(drops) != 1 ||
!strings.HasPrefix(drops[0], run.Ask+": no reader is left") {
t.Fatalf("after b's supersede: %v, run %+v, drops %v", err, run, drops)
}
deadline := time.After(5 * time.Second)
for {
c.mu.Lock()
n := len(c.stops)
c.mu.Unlock()
if n == 0 {
break
}
select {
case <-deadline:
t.Fatal("the steps that left still wait")
case <-time.After(time.Millisecond):
}
}
said.mu.Lock()
defer said.mu.Unlock()
if _, ok := said.said[meshwants.EventStepEnded+".n1"]; ok {
t.Errorf("a step that left says its end: %v", said.said)
}
}
@@ -123,6 +123,7 @@ func hearDelivery(ctx context.Context, js jetstream.JetStream, w *world, wake fu
m, at, err := presenceOf(data)
if err == nil {
w.heardPresence(m, true, at)
logf("%s is away (machine-away): a composition step does not compose it until its machine-back", m)
}
return err
}},
@@ -130,6 +131,7 @@ func hearDelivery(ctx context.Context, js jetstream.JetStream, w *world, wake fu
m, at, err := presenceOf(data)
if err == nil {
w.heardPresence(m, false, at)
logf("%s is back (machine-back): a composition step composes it again", m)
}
return err
}},
@@ -16,6 +16,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -24,14 +25,17 @@ import (
"git.novox.be/novox/mesh-sdk/go/keys"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
"mesh-delivery-assigner/internal/buckets"
)
// A step's input by its digest (design 47 G §354, §712, A-237, A-243): the assigner keeps the input of each composition step
// request it asked a build seat to run in its state `inputs`, one key `<time>.<digest>` per input, the value the input's bytes in
// request it asked a build seat to run in its state `inputs`, one key per input, its sha256 alone (A-465a), the value the input's bytes in
// base64 with the state's object store `inputs-objects` holding one over a key's limit (mesh-sdk stateobjects), and the seat
// mesh-wants' verb `input` answers one by its digest in chunks, to the holders of the build seats alone (the verb's callers, §338).
// An input is deleted when the build seat's step-ended of its ask is heard, or a day after its write, whichever is first, the day
// judged at each new ask over at most 20 of the oldest keys. `assigner_inputs` reads them. This mirrors the indexer's
// An input is kept once per content: every ask that names its digest shares it (A-465a). It is deleted, with its object, when the
// step-ended of the last ask that names it is heard, or a day after its last write, whichever is first, the day judged at each new
// ask over at most 20 of the oldest inputs. `assigner_inputs` reads them. This mirrors the indexer's
// (mesh-delivery-indexer internal/index/run_inputs.go): the two are separate modules' code, owed to mesh-sdk.
// inputsTool is the tool that reads the kept inputs.
@@ -44,10 +48,39 @@ const (
inputsSwept = 20
inputsTime = "20060102T150405Z"
// inputsAskPrefix opens the keys that name an input by the id of the ask it was kept for: ask.<id>, the id escaped as a name,
// holding the input's own key, so the build seat's step-ended of that ask finds it.
// holding the digest of the input the ask uses, so the build seat's step-ended of that ask finds it.
inputsAskPrefix = "ask."
)
// inputAsk is what an ask's key holds (A-465b): the digest of the input the ask uses and the time the ask wrote it, in inputsTime.
type inputAsk struct {
Digest string `json:"digest"`
At string `json:"at"`
}
// readAsk reads an ask's key: the JSON form, or the bare key of the old form `<time>.<digest>` an earlier assigner wrote, which
// carries its time in the key itself.
func readAsk(raw []byte) (digest string, at time.Time) {
var a inputAsk
if json.Unmarshal(raw, &a) == nil && a.Digest != "" {
at, _ = time.Parse(inputsTime, a.At)
return a.Digest, at
}
digest = string(raw)
t, _, _ := strings.Cut(digest, ".")
at, _ = time.Parse(inputsTime, t)
return digest, at
}
// isDigest is whether a key is an input's own key: 64 hex digits.
func isDigest(key string) bool {
if len(key) != 64 {
return false
}
_, err := hex.DecodeString(key)
return err == nil
}
// digestOf is an input's digest as the ask names it: its sha256 in hex.
func digestOf(b []byte) string { sum := sha256.Sum256(b); return hex.EncodeToString(sum[:]) }
@@ -63,8 +96,9 @@ type inputKeeper struct {
used func() (int64, error)
}
// keep keeps the exact bytes of the input of the ask under their write time and sha256, after sweeping the keys more than a day old
// among the 20 oldest, and names it by the ask's id; it answers the digest. The same bytes written again in the same second are one key.
// keep keeps the exact bytes of the input of the ask under their sha256, after sweeping, and names the digest and the time in the
// ask's key; it answers the digest. The same bytes asked again are the same one input (A-465a, A-465b). The ask's key is written
// first, so a sweep never finds the input unnamed.
func (k *inputKeeper) keep(ask string, body []byte) (string, error) {
if ask == "" {
return "", errors.New("name the ask the input is kept for")
@@ -72,29 +106,43 @@ func (k *inputKeeper) keep(ask string, body []byte) (string, error) {
if err := k.sweep(); err != nil {
k.logf("inputs: the keys older than a day are not all deleted: %v", err)
}
if k.used != nil {
used, err := k.used()
if err != nil {
return "", fmt.Errorf("the fill of the state %s is not read, so no input is kept: %w", meshwants.StateInputs, err)
}
if used+int64(len(body)) > inputsCap {
return "", fmt.Errorf("the state %s is at its cap of %d MiB (%d bytes kept), so the ask's input of %d bytes is refused",
meshwants.StateInputs, inputsCap>>20, used, len(body))
}
}
digest := digestOf(body)
key := meshwants.InputsByTimeDigestKey(k.now().UTC().Format(inputsTime), digest)
if err := k.vals.Put(key, base64.StdEncoding.EncodeToString(body)); err != nil && !errors.Is(err, stateobjects.ErrChanged) {
raw, err := json.Marshal(inputAsk{Digest: digest, At: k.now().UTC().Format(inputsTime)})
if err != nil {
return "", err
}
if err := k.vals.KV.Create(inputsAskPrefix+keys.EscapeName(ask), []byte(key)); err != nil && !errors.Is(err, stateobjects.ErrChanged) {
askKey := inputsAskPrefix + keys.EscapeName(ask)
if err := k.vals.KV.Create(askKey, raw); err != nil && !errors.Is(err, stateobjects.ErrChanged) {
return "", err
}
have, err := k.vals.KV.Get(digest)
if err != nil {
return "", err
}
if have != nil {
return digest, nil
}
if k.used != nil {
used, err := k.used()
if err == nil && used+int64(len(body)) > inputsCap {
err = fmt.Errorf("the state %s is at its cap of %d MiB (%d bytes kept), so the ask's input of %d bytes is refused",
buckets.InputsState, inputsCap>>20, used, len(body))
} else if err != nil {
err = fmt.Errorf("the fill of the state %s is not read, so no input is kept: %w", buckets.InputsState, err)
}
if err != nil {
_ = k.vals.KV.Delete(askKey)
return "", err
}
}
if err := k.vals.Put(digest, base64.StdEncoding.EncodeToString(body)); err != nil && !errors.Is(err, stateobjects.ErrChanged) {
return "", err
}
return digest, nil
}
// ended deletes the input kept for an ask when the build seat's step-ended of that ask is heard; an ask that kept none, or whose
// input a sweep deleted, changes nothing.
// ended removes the ask's key when the build seat's step-ended of that ask is heard, and deletes the input it named, with its
// object, when no other ask's key names it (A-465a); an ask that kept none changes nothing.
func (k *inputKeeper) ended(ask string) error {
if ask == "" {
return nil
@@ -104,66 +152,165 @@ func (k *inputKeeper) ended(ask string) error {
if err != nil || e == nil {
return err
}
if err := k.vals.KV.Delete(string(e.Value)); err != nil {
digest, _ := readAsk(e.Value)
if err := k.vals.KV.Delete(askKey); err != nil {
return err
}
return k.vals.KV.Delete(askKey)
named, err := k.named()
if err != nil {
return err
}
if named[digest] {
return nil // another running ask still names it
}
return k.deleteInput(digest)
}
// sweep deletes, among the 20 oldest keys, those written more than a day ago, and a name by ask whose input is gone.
// named is the inputs some ask's key names now.
func (k *inputKeeper) named() (map[string]bool, error) {
all, err := k.vals.KV.Keys()
if err != nil {
return nil, err
}
named := map[string]bool{}
for _, key := range all {
if !strings.HasPrefix(key, inputsAskPrefix) {
continue
}
o, err := k.vals.KV.Get(key)
if err != nil {
return nil, err
}
if o != nil {
d, _ := readAsk(o.Value)
named[d] = true
}
}
return named, nil
}
// deleteInput deletes an input's key and the object that key names, when it names one (A-465): an owner that ends its own key
// ends its object. A key already gone leaves nothing to remove here; the start and assigner_inputs_drop remove what a key left.
func (k *inputKeeper) deleteInput(key string) error {
e, err := k.vals.KV.Get(key)
if err != nil {
return err
}
if err := k.vals.KV.Delete(key); err != nil {
return err
}
if e == nil {
return nil
}
if ref, ok := stateobjects.ReadRef(e.Value); ok {
if err := k.vals.Objects.Delete(ref.Object); err != nil {
return fmt.Errorf("the object %s of %s: %w", ref.Object, key, err)
}
}
return nil
}
// dropKeyGone removes every object of the state whose key is gone and answers their count and bytes (A-465).
func (k *inputKeeper) dropKeyGone() (dropped []stateobjects.Kept, bytes int64, err error) {
kept, err := k.vals.Kept()
if err != nil {
return nil, 0, err
}
names, err := k.vals.DropKeyGone()
gone := map[string]bool{}
for _, n := range names {
gone[n] = true
}
for _, o := range kept {
if gone[o.Name] {
dropped = append(dropped, o)
bytes += o.Size
}
}
return dropped, bytes, err
}
// dropAtStart removes the objects whose key is gone at the program's start, said with their count and bytes (A-465). An object
// store not on the bus yet is said and left.
func (k *inputKeeper) dropAtStart() {
dropped, bytes, err := k.dropKeyGone()
if err != nil {
k.logf("inputs: the objects of deleted keys are not all removed at the start (%d objects, %d bytes removed): %v", len(dropped), bytes, err)
return
}
k.logf("inputs: %d objects of deleted keys removed at the start, %d bytes", len(dropped), bytes)
}
// sweep deletes, among the 20 oldest ask keys, those written more than a day ago; every input of the digest form no ask's key names
// now, with its object; a key of the old form `<time>.<digest>` a day after the time it carries; and an old-form ask key whose
// input is gone (A-465b).
func (k *inputKeeper) sweep() error {
all, err := k.vals.KV.Keys()
if err != nil {
return err
}
var errs []error
var inputs, asks []string
type askAt struct {
key string
at time.Time
}
var asks []askAt
var inputs []string
for _, key := range all {
if strings.HasPrefix(key, inputsAskPrefix) {
asks = append(asks, key)
} else {
inputs = append(inputs, key)
}
}
slices.Sort(inputs) // the write time leads the key, so the oldest come first
if len(inputs) > inputsSwept {
inputs = inputs[:inputsSwept]
}
for _, key := range inputs {
t, _, ok := meshwants.ParseInputsByTimeDigestKey(key)
at, perr := time.Parse(inputsTime, t)
if !ok || perr != nil || k.now().Sub(at) < inputsKeptFor {
continue
}
if err := k.vals.KV.Delete(key); err != nil {
errs = append(errs, err)
}
}
for _, key := range asks {
if e, err := k.vals.KV.Get(key); err == nil && e != nil {
if gone, err := k.vals.KV.Get(string(e.Value)); err == nil && gone == nil {
if err := k.vals.KV.Delete(key); err != nil {
errs = append(errs, err)
e, err := k.vals.KV.Get(key)
if err != nil || e == nil {
continue
}
d, at := readAsk(e.Value)
if !isDigest(d) {
if gone, err := k.vals.KV.Get(d); err == nil && gone == nil {
errs = append(errs, k.vals.KV.Delete(key))
}
}
asks = append(asks, askAt{key, at})
continue
}
if !isDigest(key) {
t, _, _ := strings.Cut(key, ".")
if at, err := time.Parse(inputsTime, t); err == nil && k.now().Sub(at) >= inputsKeptFor {
errs = append(errs, k.deleteInput(key))
}
continue
}
inputs = append(inputs, key)
}
slices.SortFunc(asks, func(a, b askAt) int { return a.at.Compare(b.at) })
if len(asks) > inputsSwept {
asks = asks[:inputsSwept]
}
for _, a := range asks {
if !a.at.IsZero() && k.now().Sub(a.at) >= inputsKeptFor {
errs = append(errs, k.vals.KV.Delete(a.key))
}
}
named, err := k.named()
if err != nil {
return errors.Join(append(errs, err)...)
}
for _, key := range inputs {
if !named[key] {
errs = append(errs, k.deleteInput(key))
}
}
return errors.Join(errs...)
}
// keyOf is the key of the input of the digest, or none.
// keyOf is the key of the input of the digest, or none: the digest itself.
func (k *inputKeeper) keyOf(digest string) (string, error) {
all, err := k.vals.KV.Keys()
if err != nil {
if !isDigest(digest) {
return "", nil
}
e, err := k.vals.KV.Get(digest)
if err != nil || e == nil {
return "", err
}
slices.Sort(all)
for _, key := range all {
if _, d, ok := meshwants.ParseInputsByTimeDigestKey(key); ok && d == digest && !strings.HasPrefix(key, inputsAskPrefix) {
return key, nil
}
}
return "", nil
return digest, nil
}
// body is the exact bytes of the input of the digest, checked against it, whatever its size.
@@ -173,7 +320,7 @@ func (k *inputKeeper) body(digest string) ([]byte, error) {
}
key, err := k.keyOf(digest)
if err != nil {
return nil, fmt.Errorf("the state %s cannot be read: %w", meshwants.StateInputs, err)
return nil, fmt.Errorf("the state %s cannot be read: %w", buckets.InputsState, err)
}
if key == "" {
return nil, fmt.Errorf("the assigner keeps no input with the digest %s", digest)
@@ -241,7 +388,7 @@ func (o *inputObjects) open() (stateobjects.Objects, error) {
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
store, err := o.js.ObjectStore(ctx, catalogue.BusName(o.module, catalogue.StateObjectsName(meshwants.StateInputs)))
store, err := o.js.ObjectStore(ctx, catalogue.BusName(o.module, catalogue.StateObjectsName(buckets.InputsState)))
if errors.Is(err, jetstream.ErrBucketNotFound) || errors.Is(err, jetstream.ErrStreamNotFound) {
return nil, fmt.Errorf("the object store of the state inputs is not on the bus yet: %w", process.ErrNotYet)
}
@@ -292,7 +439,7 @@ func inputsOver(kv jetstream.KeyValue, js jetstream.JetStream, module string, lo
return nil
}
objects := &inputObjects{js: js, module: module}
v := stateobjects.Values{KV: stateobjects.BusKV{KV: kv}, Objects: objects, Store: catalogue.StateObjectsName(meshwants.StateInputs)}
v := stateobjects.Values{KV: stateobjects.BusKV{KV: kv}, Objects: objects, Store: catalogue.StateObjectsName(buckets.InputsState)}
used := func() (int64, error) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
@@ -333,12 +480,21 @@ func inputsRead(k *inputKeeper) stdio.Tool {
return nil, fmt.Errorf("the inputs cannot be listed: %w", err)
}
slices.Sort(all)
lastAsk := map[string]string{} // the latest write time an ask's key names for each input
for _, key := range all {
if e, err := k.vals.KV.Get(key); err == nil && e != nil && strings.HasPrefix(key, inputsAskPrefix) {
if d, at := readAsk(e.Value); !at.IsZero() && at.Format(inputsTime) > lastAsk[d] {
lastAsk[d] = at.Format(inputsTime)
}
}
}
out := []inputSaid{}
for _, key := range all {
t, d, ok := meshwants.ParseInputsByTimeDigestKey(key)
if !ok || strings.HasPrefix(key, inputsAskPrefix) || (want != "" && d != want) {
d := key
if !isDigest(key) || (want != "" && d != want) {
continue
}
t := lastAsk[key]
var text string
if found, err := k.vals.Get(key, &text); err != nil || !found {
continue
@@ -352,26 +508,74 @@ func inputsRead(k *inputKeeper) stdio.Tool {
}
out = append(out, said)
}
return map[string]any{"inputs": out}, nil
gone := []map[string]any{}
var goneBytes int64
answer := map[string]any{"inputs": out}
kept, err := k.vals.Kept()
if err != nil {
// never an empty list for objects that could not be read (A-465)
answer["key-gone-error"] = fmt.Sprintf("the objects of the state inputs cannot be listed, so key-gone objects are not known: %v", err)
return answer, nil
}
for _, o := range kept {
if o.KeyGone {
gone = append(gone, map[string]any{"object": o.Name, "key": o.Key, "size": o.Size})
goneBytes += o.Size
}
}
answer["key-gone"], answer["key-gone-bytes"] = gone, goneBytes
return answer, nil
}}
}
// heardStepEnded deletes the input kept for the ask a build seat's step-ended names (design 47 G §354), and hands the end to the
// composition run that waits for that ask (A-240); one whose input cannot be deleted is given again by the bus.
func heardStepEnded(k *inputKeeper, ends *runEnds, m jetstream.Msg, logf func(string, ...any)) {
// inputsDropTool is the tool that removes the objects whose key is gone.
const inputsDropTool = "assigner_inputs_drop"
// inputsDrop is the tool assigner_inputs_drop (A-465): a hand act, recorded first, that removes the objects of the state inputs
// whose key is gone now and answers what it removed.
func inputsDrop(k *inputKeeper, ask func(string, any) (json.RawMessage, error)) stdio.Tool {
return stdio.Tool{Name: inputsDropTool, Description: "Remove the objects of the assigner's state inputs whose key is gone now (design 47 G A-465), " +
"and answer which and how many bytes. An object a key names is never removed. Recorded as a hand act; assigner_inputs lists them as key-gone.",
Input: schema(map[string]string{"why": "why the objects of deleted keys are dropped now"}, []string{"why"}), Run: func(args map[string]any) (any, error) {
why := strings.TrimSpace(arg(args, "why"))
if why == "" {
return nil, errors.New("say why the objects of deleted keys are dropped")
}
taken, err := meshhandactsclient.Recorder{Ask: ask}.Record(meshhandactsclient.Request{What: inputsDropTool, Why: why + " (" + stdio.CallerOf(args) + ")", Cause: "inputs-drop"})
if err != nil {
return nil, fmt.Errorf("the hand act was not recorded, so nothing was removed: %w", err)
}
dropped, bytes, err := k.dropKeyGone()
out := []map[string]any{}
for _, o := range dropped {
out = append(out, map[string]any{"object": o.Name, "key": o.Key, "size": o.Size})
}
return map[string]any{"dropped": out, "count": len(out), "bytes": bytes, "hand-act": taken.Log}, err
}}
}
// heardStepEnded hands the end a build seat's step-ended names to the composition run of that ask (A-240, §719), which writes it
// in the run's key, and then deletes the input kept for the ask (design 47 G §354): the run's end first, so no other act delays
// it past its bound (A-470o, A-470p). One whose run's end cannot be written, or whose input cannot be deleted, is given again by
// the bus; its redelivery finds the run ended, said as ignored for the run, and still deletes the input.
func heardStepEnded(k *inputKeeper, runEnded func(buildstep.StepEnded) error, m jetstream.Msg, logf func(string, ...any)) {
var e buildstep.StepEnded
if err := json.Unmarshal(m.Data(), &e); err != nil {
logf("%s holds what is no step-ended, left: %v", m.Subject(), err)
_ = m.Ack()
return
}
if runEnded != nil {
if err := runEnded(e); err != nil {
logf("the composition run of the ask %s is not ended, so the bus gives its step-ended again: %v", e.Ask, err)
_ = m.Nak()
return
}
}
if err := k.ended(e.Ask); err != nil {
logf("inputs: the input of the ask %s could not be deleted, so the bus gives it again: %v", e.Ask, err)
_ = m.Nak()
return
}
if ends != nil {
ends.heard(e)
}
_ = m.Ack()
}
@@ -3,8 +3,10 @@ package main
import (
"bytes"
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
@@ -14,6 +16,7 @@ import (
"github.com/nats-io/nats.go"
meshwants "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
)
@@ -22,6 +25,8 @@ type kvMem struct {
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
// delErr, when set, fails the next Delete once.
delErr error
}
func newKVMem() *kvMem { return &kvMem{vals: map[string][]byte{}, revs: map[string]uint64{}} }
@@ -56,6 +61,10 @@ func (k *kvMem) Update(key string, v []byte, rev uint64) error {
func (k *kvMem) Delete(key string) error {
k.mu.Lock()
defer k.mu.Unlock()
if err := k.delErr; err != nil {
k.delErr = nil
return err
}
delete(k.vals, key)
return nil
}
@@ -71,8 +80,9 @@ func (k *kvMem) Keys() ([]string, error) {
}
type objectsMem struct {
mu sync.Mutex
objs map[string][]byte
mu sync.Mutex
objs map[string][]byte
listErr error // List fails with it
}
func (*objectsMem) MostBytes() int { return 0 }
@@ -91,8 +101,24 @@ func (o *objectsMem) Get(name string) ([]byte, error) {
}
return d, nil
}
func (o *objectsMem) Delete(string) error { return nil }
func (o *objectsMem) List() ([]stateobjects.Object, error) { return nil, nil }
func (o *objectsMem) Delete(name string) error {
o.mu.Lock()
defer o.mu.Unlock()
delete(o.objs, name)
return nil
}
func (o *objectsMem) List() ([]stateobjects.Object, error) {
o.mu.Lock()
defer o.mu.Unlock()
if o.listErr != nil {
return nil, o.listErr
}
var out []stateobjects.Object
for n, d := range o.objs {
out = append(out, stateobjects.Object{Name: n, Size: int64(len(d))})
}
return out, nil
}
// lineMem is a core bus in memory: a chunk published is kept, and its acknowledgement heard at once.
type lineMem struct {
@@ -228,4 +254,250 @@ func TestAnInputIsDeletedAtItsStepEndedOrAfterADay(t *testing.T) {
if _, err := k.body(d2); err == nil {
t.Error("an input a day old is still kept at the next ask")
}
// A-470o, A-470p: the step-ended's run end is written before the input is deleted; a delete that fails asks the event again,
// whose redelivery finds the run ended and still deletes the input; a run end not written asks it again with the input kept.
for _, c := range []struct {
name string
delErr error
runErr error
naked bool
runs int
deleted bool
}{
{name: "the run's end before the input's delete", runs: 1, deleted: true},
{name: "an input not deleted asks the event again", delErr: errors.New("the bus is away"), naked: true, runs: 2, deleted: true},
{name: "a run's end not written asks the event again", runErr: errors.New("the run is not read"), naked: true, runs: 1},
} {
t.Run(c.name, func(t *testing.T) {
k := testKeeper(&objectsMem{objs: map[string][]byte{}}, time.Now)
d, _ := k.keep("ask/run", []byte(`{"heads":"abc"}`))
k.vals.KV.(*kvMem).delErr = c.delErr
runs := 0
runEnded := func(e buildstep.StepEnded) error {
runs++
if _, err := k.body(d); err != nil {
t.Errorf("the run's end is written after the input's delete (%d)", runs)
}
return c.runErr
}
raw, _ := json.Marshal(buildstep.StepEnded{Ask: "ask/run"})
m := &heardMsg{data: raw, subject: "step-ended"}
heardStepEnded(k, runEnded, m, t.Logf)
if m.naked != c.naked || m.acked == c.naked {
t.Fatalf("acked %v, naked %v, want naked %v", m.acked, m.naked, c.naked)
}
if m.naked && c.delErr != nil { // the redelivery
m = &heardMsg{data: raw, subject: "step-ended"}
heardStepEnded(k, func(e buildstep.StepEnded) error { runs++; return nil }, m, t.Logf)
}
if _, err := k.body(d); runs != c.runs || (err != nil) != c.deleted {
t.Errorf("the run's end heard %d times (want %d), the input deleted %v (want %v)", runs, c.runs, err != nil, c.deleted)
}
})
}
}
// Design 47 G A-465: an input the assigner ends itself (the step-ended of its ask, the sweep of a day-old key) ends its object
// too; the objects of keys another part deleted are removed by dropKeyGone and by the tool assigner_inputs_drop, which lists
// them as key-gone first, and never removes an object a key names.
func TestAnEndedInputEndsItsObjectAndTheDropToolRemovesWhatAKeyLeft(t *testing.T) {
now := time.Date(2026, 11, 17, 10, 0, 0, 0, time.UTC)
objs := &objectsMem{objs: map[string][]byte{}}
k := testKeeper(objs, func() time.Time { return now })
big := func(b byte) []byte { return bytes.Repeat([]byte{b}, 2<<20) }
if _, err := k.keep("ask-one", big('a')); err != nil {
t.Fatal(err)
}
if _, err := k.keep("ask-two", big('b')); err != nil {
t.Fatal(err)
}
if len(objs.objs) != 2 {
t.Fatalf("objects = %d, want 2", len(objs.objs))
}
if err := k.ended("ask-one"); err != nil || len(objs.objs) != 1 {
t.Fatalf("the step-ended left %d objects (%v), want 1", len(objs.objs), err)
}
now = now.Add(25 * time.Hour)
if _, err := k.keep("ask-three", big('c')); err != nil {
t.Fatal(err)
}
if len(objs.objs) != 1 {
t.Fatalf("after the sweep of the day-old input %d objects, want only the new one", len(objs.objs))
}
// An object whose key another part deleted is listed as key-gone and removed by the tool; a live one stays.
keys, _ := k.vals.KV.Keys()
for _, key := range keys {
if !strings.HasPrefix(key, inputsAskPrefix) {
if err := k.vals.KV.Delete(key); err != nil {
t.Fatal(err)
}
}
}
live := []byte(`{"heads":"x"}`)
if _, err := k.keep("ask-four", append(live, make([]byte, 2<<20)...)); err != nil {
t.Fatal(err)
}
list, err := inputsRead(k).Run(map[string]any{})
if err != nil || len(list.(map[string]any)["key-gone"].([]map[string]any)) != 1 {
t.Fatalf("assigner_inputs key-gone = %+v %v, want the one object of the deleted key", list, err)
}
ask := func(string, any) (json.RawMessage, error) {
return json.RawMessage(`{"answer":{"log":"hand-acts"}}`), nil
}
if _, err := inputsDrop(k, ask).Run(map[string]any{}); err == nil {
t.Error("dropped with no reason")
}
out, err := inputsDrop(k, ask).Run(map[string]any{"why": "a key left its object"})
if err != nil || out.(map[string]any)["count"] != 1 || len(objs.objs) != 1 {
t.Fatalf("assigner_inputs_drop = %+v %v, %d objects left: want one removed and the live one kept", out, err, len(objs.objs))
}
// The start drops what a key left, said with its count and bytes.
if _, err := k.keep("ask-five", live); err != nil {
t.Fatal(err)
}
for _, key := range keysOf(t, k) {
if !strings.HasPrefix(key, inputsAskPrefix) {
_ = k.vals.KV.Delete(key)
}
}
var logged []string
k.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
before := len(objs.objs)
k.dropAtStart()
if len(objs.objs) != 0 || before == 0 || len(logged) != 1 || !strings.Contains(logged[0], fmt.Sprintf("%d objects of deleted keys removed at the start", before)) || !strings.Contains(logged[0], " bytes") {
t.Errorf("the start left %d of %d objects and said %q: want every key-gone object removed and its count and bytes said", len(objs.objs), before, logged)
}
}
func keysOf(t *testing.T, k *inputKeeper) []string {
t.Helper()
keys, err := k.vals.KV.Keys()
if err != nil {
t.Fatal(err)
}
return keys
}
// Design 47 G A-465: assigner_inputs says why key-gone objects are not known when the objects cannot be listed, and never
// answers them as none.
func TestAssignerInputsSaysWhenTheKeyGoneObjectsCannotBeListed(t *testing.T) {
objs := &objectsMem{objs: map[string][]byte{}, listErr: errors.New("the object store does not answer")}
k := testKeeper(objs, time.Now)
out, err := inputsRead(k).Run(map[string]any{})
if err != nil {
t.Fatal(err)
}
m := out.(map[string]any)
if msg, _ := m["key-gone-error"].(string); !strings.Contains(msg, "does not answer") {
t.Errorf("answer %+v: want key-gone-error naming the failure", m)
}
if _, has := m["key-gone"]; has {
t.Errorf("answer %+v: an unreadable store is not answered as no key-gone objects", m)
}
}
// Design 47 G A-465a: an input is kept once per content under its sha256 alone; every ask names the digest it uses, and the input
// goes, with its object, at the step-ended of the last ask that names it, or a day after its last write; a request the cap still
// refuses says the fill; an input of the old form (a key that is no digest) is ended with its object.
func TestAnInputIsKeptOncePerContentAndEndsWithItsLastAsk(t *testing.T) {
now := time.Date(2026, 11, 17, 10, 0, 0, 0, time.UTC)
objs := &objectsMem{objs: map[string][]byte{}}
k := testKeeper(objs, func() time.Time { return now })
body := bytes.Repeat([]byte{'n'}, 2<<20)
var digests []string
for _, ask := range []string{"ask-1", "ask-2", "ask-3"} {
d, err := k.keep(ask, body)
if err != nil {
t.Fatal(err)
}
digests = append(digests, d)
}
if digests[0] != digestOf(body) || digests[1] != digests[0] || digests[2] != digests[0] {
t.Fatalf("digests = %v, want the sha256 of the bytes each time", digests)
}
if _, err := k.vals.KV.Get(digests[0]); err != nil {
t.Fatal(err)
}
if e, _ := k.vals.KV.Get(digests[0]); e == nil || len(objs.objs) != 1 {
t.Fatalf("one key by the digest and one object wanted, got entry %v and %d objects", e, len(objs.objs))
}
for _, key := range keysOf(t, k) {
if !strings.HasPrefix(key, inputsAskPrefix) && key != digests[0] {
t.Errorf("a key %q beside the one input", key)
}
}
if e, _ := k.vals.KV.Get(inputsAskPrefix + "ask-2"); e == nil || !strings.Contains(string(e.Value), digests[0]) || !strings.Contains(string(e.Value), "20261117T100000Z") {
t.Errorf("the ask does not name the digest it uses: %v", e)
}
for _, ask := range []string{"ask-1", "ask-2"} {
if err := k.ended(ask); err != nil {
t.Fatal(err)
}
if _, err := k.body(digests[0]); err != nil {
t.Fatalf("the input went at the end of %s while ask-3 still names it: %v", ask, err)
}
}
if err := k.ended("ask-3"); err != nil {
t.Fatal(err)
}
if _, err := k.body(digests[0]); err == nil || len(objs.objs) != 0 || len(keysOf(t, k)) != 0 {
t.Errorf("the last step-ended left the input (%v), %d objects, keys %v", err, len(objs.objs), keysOf(t, k))
}
// a day after its last write, a renewed write counted
if _, err := k.keep("ask-4", body); err != nil {
t.Fatal(err)
}
now = now.Add(20 * time.Hour)
if _, err := k.keep("ask-5", body); err != nil {
t.Fatal(err)
}
now = now.Add(20 * time.Hour)
if _, err := k.keep("ask-6", []byte("other")); err != nil {
t.Fatal(err)
}
if _, err := k.body(digests[0]); err != nil {
t.Errorf("an input written again 20 hours ago was swept: %v", err)
}
now = now.Add(25 * time.Hour)
if _, err := k.keep("ask-7", []byte("another")); err != nil {
t.Fatal(err)
}
if _, err := k.body(digests[0]); err == nil || len(objs.objs) != 0 {
t.Errorf("an input a day after its last write is kept (%v), %d objects", err, len(objs.objs))
}
// an input of the old form goes a day after the time its key carries, with its object, never earlier
old := "20261117T090000Z." + digestOf([]byte("old"))
if err := k.vals.Put(old, base64.StdEncoding.EncodeToString(bytes.Repeat([]byte{'o'}, 2<<20))); err != nil {
t.Fatal(err)
}
recent := now.Add(-time.Hour).Format(inputsTime) + "." + digestOf([]byte("recent"))
if err := k.vals.Put(recent, "cmVjZW50"); err != nil {
t.Fatal(err)
}
if err := k.sweep(); err != nil {
t.Fatal(err)
}
if e, _ := k.vals.KV.Get(old); e != nil || len(objs.objs) != 0 {
t.Errorf("the input of the old form is kept (%v) with %d objects", e, len(objs.objs))
}
if e, _ := k.vals.KV.Get(recent); e == nil {
t.Error("an input of the old form under a day old was deleted early")
}
// a new content the cap refuses says the fill; the same content, already kept, needs no room
full := newTestKeeperWithFill(objs, now, inputsCap-10)
if _, err := full.keep("ask-8", []byte("fits")); err != nil {
t.Fatalf("a content that fits is refused: %v", err)
}
if _, err := full.keep("ask-9", []byte("does not fit at all")); err == nil || !strings.Contains(err.Error(), "bytes kept") {
t.Errorf("the refusal does not name the fill: %v", err)
}
if _, err := full.keep("ask-10", []byte("fits")); err != nil {
t.Errorf("the content already kept is refused at the cap: %v", err)
}
}
func newTestKeeperWithFill(objs *objectsMem, now time.Time, used int64) *inputKeeper {
k := testKeeper(objs, func() time.Time { return now })
k.used = func() (int64, error) { return used, nil }
return k
}
@@ -69,11 +69,13 @@ import (
type bed struct {
// seatListRead is closed once the assigner read the seat list (serveSeatList).
seatListRead <-chan struct{}
srv *server.Server
js jetstream.JetStream
ctx context.Context
dir string
buckets map[string]jetstream.KeyValue
// settings are the files of settings a test sets, by the variable naming each (as the manifest's resources fill them).
settings map[string]string
srv *server.Server
js jetstream.JetStream
ctx context.Context
dir string
buckets map[string]jetstream.KeyValue
// placed is, by node, the modules the bed's assignments names on it (place).
placed map[string][]string
// journal is what the programs the bed starts write in their journal.
@@ -169,7 +171,7 @@ func startBed(t *testing.T, leaveOut string) *bed {
continue
}
cfg := jetstream.KeyValueConfig{Bucket: name, History: 5}
if name == plc.PulsesBucket() {
if name == plc.PulsesBucket() || name == buckets.CompositionRuns() {
cfg.History, cfg.LimitMarkerTTL = 1, time.Minute // a time to live per key, as the controller makes it (key-ttl)
}
kv, err := js.CreateKeyValue(ctx, cfg)
@@ -178,6 +180,11 @@ func startBed(t *testing.T, leaveOut string) *bed {
}
b.buckets[name] = kv
}
// The object store beside the state inputs, which the grants part makes from the manifest's objects: a composition request that
// carries the whole seat list (A-467) is too big for the state's value and is kept there.
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: catalogue.BusName("mesh-delivery-assigner", catalogue.StateObjectsName(buckets.InputsState))}); err != nil {
t.Fatal(err)
}
// The builder's module log, as the runtime raises it (ADR 0297): the running commits are read from it (ADR 0571 clause 1).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(builds.BuilderModule, builds.BuilderLog),
Subjects: []string{buscompose.LogSubject(builds.BuilderModule, builds.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
@@ -296,6 +303,13 @@ func (b *bed) startStoppable(t *testing.T) (<-chan error, func()) {
t.Fatal(err)
}
env := map[string]string{"MESH_BROKER_FILE": broker, "MESH_ASSIGNER_OUTSIDE_SETTLE_FILE": settle}
for name, value := range b.settings {
file := filepath.Join(b.dir, name)
if err := os.WriteFile(file, []byte(value), 0o600); err != nil {
t.Fatal(err)
}
env[name] = file
}
ctx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() {
@@ -1182,9 +1196,10 @@ func TestABackupAnswerIsWrittenIntoTheNodesWant(t *testing.T) {
b := startBed(t, "")
fNew := meshcatalogclient.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
fRun := meshcatalogclient.SourcePrefix2 + strings.Repeat("a", 64) // a held have names the build it keeps running
b.compose(t, "ace", 'b')
b.putWant(t, "ace", "web", mwc.Want{Fp: fNew, Self: true}) // a placed node's want names the build alone
held, _ := nodeengineclient.EncodeHave(nodeengine.HaveByNodeModuleValue{Fp: fNew, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits, Detail: "no space left"})
held, _ := nodeengineclient.EncodeHave(nodeengine.HaveByNodeModuleValue{Fp: fRun, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits, Detail: "no space left"})
b.putJSON(t, buckets.Have, "ace.web", held)
b.start(t)
akey, _ := asks.BackupAskKey("ace", "web", fNew)
@@ -1203,8 +1218,20 @@ func TestABackupAnswerIsWrittenIntoTheNodesWant(t *testing.T) {
if w1.Fp != fNew || w1.Composed != "" || !w1.Self {
t.Errorf("the want = %+v", w1)
}
// the node retries and the backup fails again: a new have value, asked again under the next number
again, _ := nodeengineclient.EncodeHave(nodeengine.HaveByNodeModuleValue{Fp: fNew, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits, Detail: "no space left, again"})
// the node retries: its have is backing up, not held. The ask stands and its condition is not cleared (A-453a).
backing, _ := nodeengineclient.EncodeHave(nodeengine.HaveByNodeModuleValue{Fp: fNew, Phase: nodeengineclient.PhaseBackingUp, Health: nodeengineclient.HealthWaits})
b.putJSON(t, buckets.Have, "ace.web", backing)
time.Sleep(time.Second)
if !b.hasKey(buckets.Asks(), akey) {
t.Fatal("the backup ask was taken away while the node backs up again")
}
for k, c := range b.conditionsSaid(t) {
if strings.Contains(k, "backup") && c.State == occ.StateCleared {
t.Fatalf("the condition %s was cleared while the node backs up again", k)
}
}
// the backup fails again: a new have value, asked again under the next number
again, _ := nodeengineclient.EncodeHave(nodeengine.HaveByNodeModuleValue{Fp: fRun, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits, Detail: "no space left, again"})
b.putJSON(t, buckets.Have, "ace.web", again)
a2 := b.askNow(t, akey, 2)
if a2.Ask.ID == a.Ask.ID {
@@ -9,8 +9,10 @@ package main
import (
"context"
"fmt"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshwants"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscalls"
"io"
"log"
@@ -30,7 +32,6 @@ import (
mma "git.novox.be/novox/mesh-contracts/go/seats/meshmoduleassigner"
mcseat "git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshconditions"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/seats/meshwatchdogs"
occ "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/clients/operatorchannelclient"
oc "git.novox.be/novox/mesh-contracts/operator-and-conversation/go/seats/operatorchannel"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
@@ -46,6 +47,9 @@ import (
plc "mesh-delivery-assigner/internal/assignment"
)
// composeCallBound is how long the program's calls wait for an answer (main).
const composeCallBound = 3 * time.Minute
func main() {
if os.Getenv(process.BrokerFileEnv) == "" {
// Started by the tool runner, with no credential of its own: the put's tools (ADR 0546), served through the runtime,
@@ -56,6 +60,10 @@ func main() {
}
return
}
// A call waits up to composeCallBound for its answer: compose-candidate composes a whole revision in shadow on each node and
// answered in about 34 s on g14 and shanks (2026-11-16 23:27), past the SDK's 30 s, so every composition step could not run.
// A responder that is absent still answers at once. To be replaced by a bound per verb from the seat's definition.
process.CallTimeout = composeCallBound
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
@@ -82,7 +90,7 @@ func ownBuckets() []string {
return []string{buckets.Asks(), buckets.Differences(),
buckets.Puts(), buckets.Wants(), plc.AssignmentsBucket(), plc.ProposalsBucket(),
plc.CandidatesBucket(), plc.PulsesBucket(), assign.FindingsBucket(), assign.PoliciesBucket(), plc.SeatHoldersBucket(),
engineReadsBucket(), retiredBucket(), copiesBucket(), writtenBucket(), buckets.Waits(), buckets.PutWaits(), buckets.DeliveryWaits(), buckets.DeliveredAgain(), buckets.Inputs(), pinsBucket()}
engineReadsBucket(), retiredBucket(), copiesBucket(), writtenBucket(), buckets.Waits(), buckets.PutWaits(), buckets.DeliveryWaits(), buckets.DeliveredAgain(), buckets.Inputs(), buckets.CompositionRuns(), pinsBucket()}
}
// engineReadsState is the assigner's state of what each node's engine states it reads (ADR 0618 §3), one key `<node>`, as
@@ -126,11 +134,12 @@ var keptStates = []announce.State{
{Name: copiesState, Readable: true},
{Name: writtenState, Readable: true},
{Name: buckets.WaitsState, Readable: true},
{Name: buckets.PutWaitsState, Readable: true}, // the put waits said (putwaits.go, design 47 G A-211)
{Name: buckets.DeliveryWaitsState, Readable: true}, // the delivery waits said (delivered.go, design 47 G A-246)
{Name: buckets.DeliveredAgainState, Readable: true}, // the asks of assigner_delivered_again (delivered.go, design 47 G A-250)
{Name: buckets.InputsState, Readable: true}, // read by the tool process alone: the verb input and assigner_inputs (design 47 G §354)
{Name: pinsState, Readable: true}, // the records pinning each assignment graph revision (design 47 G §579)
{Name: buckets.PutWaitsState, Readable: true}, // the put waits said (putwaits.go, design 47 G A-211)
{Name: buckets.DeliveryWaitsState, Readable: true}, // the delivery waits said (delivered.go, design 47 G A-246)
{Name: buckets.DeliveredAgainState, Readable: true}, // the asks of assigner_delivered_again (delivered.go, design 47 G A-250)
{Name: buckets.InputsState, Readable: true}, // read by the tool process alone: the verb input and assigner_inputs (design 47 G §354)
{Name: buckets.CompositionRunsState, Readable: true}, // the composition runs, read by the process and its tools (design 47 G §719, A-468c)
{Name: pinsState, Readable: true}, // the records pinning each assignment graph revision (design 47 G §579)
}
// kvSet is every bucket the assigner opened, by name.
@@ -298,6 +307,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// against the seat's definition it holds (A-186).
gate := seatlist.NewPart("assigner", seatListAgainCallers)
if inputs != nil {
inputs.dropAtStart() // the objects of keys another part deleted (A-465)
stopInput, err := said.AnswerSeat(meshwants.Seat, seatlist.Definition(gate.Reader, meshwants.Seat), map[string]process.SeatHandler{meshwants.VerbInput: inputs.verb(conn, buscalls.SeatAckWithin)})
if err != nil {
return fmt.Errorf("answering the verb input: %w", err)
@@ -516,17 +526,35 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
gate.Follow(seatlist.ProcessBus(bus.Call, hub), wake, logger.Printf)
}
// The build seat's step-ended of an ask deletes the input kept for it (design 47 G §354) and ends the composition run that waits
// for it (A-240); the watchman's machine-away and machine-back say which machines a composition step does not compose (§64,
// §389). Registered before the hub starts.
ends, away := &runEnds{}, &awayNodes{}
if inputs != nil {
if err := hub.Handle(nodebuildgo.SubjectStepEndedAll, func(m jetstream.Msg) { heardStepEnded(inputs, ends, m, logger.Printf) }); err != nil {
// for it (A-240). The machines a composition step does not compose are the watchman's machine-away, heard once, in the world (§64, §389).
// The composition runs (design 47 G §718, §719), watched in their own state; a build seat's step-ended of a run's ask ends
// the run in its key.
var runs *compositionRuns
if kv := kvs[buckets.CompositionRuns()]; kv != nil {
set, err := runSettingsOf(getenv)
if err != nil {
return err
}
runs = newCompositionRuns(busRuns{ctx, js, kv}, set, time.Now, logger.Printf)
if js != nil {
runs.say = wantsPublisher(ctx, js, thisNode)
}
if bus != nil {
runs.drop = dropOver(bus.CallKept)
}
// The build seats' ask-taken and step-progress begin a run's clock (§720, A-466q).
if err := hearRunBounds(hub, runs, logger.Printf); err != nil {
logger.Printf("the build seats' ask-taken and step-progress are not heard, so a composition run is ended by no bound: %v", err)
}
}
for subject, isAway := range map[string]bool{meshwatchdogs.SubjectMachineAwayAll: true, meshwatchdogs.SubjectMachineBackAll: false} {
if err := hub.Handle(subject, func(m jetstream.Msg) { heardPresence(away, isAway, m, logger.Printf) }); err != nil {
logger.Printf("%s is not heard, so a composition step asks a machine that sleeps: %v", subject, err)
if inputs != nil {
var runEnded func(buildstep.StepEnded) error
if runs != nil && bus != nil {
read := func(e buildstep.StepEnded) ([]byte, error) { return e.ReadOutput(buildstep.AskGo, bus.CallFile) }
runEnded = func(e buildstep.StepEnded) error { return runs.heardEnd(e, read) }
}
if err := hub.Handle(nodebuildgo.SubjectStepEndedAll, func(m jetstream.Msg) { heardStepEnded(inputs, runEnded, m, logger.Printf) }); err != nil {
return err
}
}
// The indexer's manifests, modules and module graph, through its first views and views family, never its buckets
@@ -674,11 +702,40 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
p.acts = newPutActs(bus, wake, logger.Printf)
}
// The composition step (design 47 G §22, §354): the seat mesh-wants' accepted kind step, run by one run ask per machine.
if bus != nil && js != nil && inputs != nil && p.publishWants != nil {
run := &compositionRun{inputs: &compositionInputs{machines: w.machinesOf, callFile: bus.CallFile}, keeper: inputs, call: bus.CallKept,
trunk: trunkOf(iv, w.trunkName), running: w.runningOf, away: away, put: runAskPublisher(js), ends: ends, ask: bus.CallFile, now: time.Now, logf: logger.Printf}
steps := &compositionSteps{node: thisNode, say: p.publishWants, run: run.run, now: time.Now, logf: logger.Printf,
if bus != nil && js != nil && inputs != nil && p.publishWants != nil && runs != nil {
run := &compositionRun{runs: runs, inputs: &compositionInputs{machines: w.machinesOf, callFile: bus.CallFile}, keeper: inputs, reads: &compositionReads{}, call: bus.CallKept,
trunk: trunkOf(iv, w.trunkName), running: w.runningOf, away: func(machine string) bool { return w.awayNow()[machine] }, put: runAskPublisher(js), ask: bus.CallFile, now: time.Now, logf: logger.Printf}
steps := &compositionSteps{node: thisNode, say: p.publishWants, run: run.run, resume: run.resume, ended: runs.stepEnded, now: time.Now, logf: logger.Printf,
asker: stepAskerOf(seatlist.Definition(gate.Reader, meshwants.Seat))}
move := steps.move(ctx) // the readers a rerun carried (§727, A-470d)
runs.moved.Store(&move)
// The state's watch wakes the steps waiting on a run and hears a bound's expiry; the runs standing at the start are taken up
// once the values that stand are read (§719, §720).
if err := watchRuns(ctx, kvs[buckets.CompositionRuns()], runs, runs.expired, func() { steps.takeUp(ctx, runs) }); err != nil {
return fmt.Errorf("the state %s is not watched: %w", buckets.CompositionRunsState, err)
}
// The runs' read and drop, the seat mesh-wants' verbs compositions and compositions-drop (§728, A-469f), answered here
// where the runs are kept and the readers' ends said; the drop's hand act recorded through the seat mesh-hand-acts first.
if e, ok := bus.(emitting); ok {
record := func(q meshhandactsclient.Request) error {
_, err := meshhandactsclient.Recorder{Ask: bus.Call}.Record(q)
return err
}
stopVerbs, err := e.AnswerSeat(meshwants.Seat, seatlist.Definition(gate.Reader, meshwants.Seat), compositionVerbs(runs, steps, record))
if err != nil {
logger.Printf("the verbs %s and %s are not answered until the assigner starts again: %v", meshwants.VerbCompositions, meshwants.VerbCompositionsDrop, err)
} else {
go func() { <-ctx.Done(); stopVerbs() }()
}
}
// A superseded or cancelled pipeline's composition step leaves the runs it reads (§723, A-469n).
if p.ends != nil {
left := func(pipeline string) error {
_, err := runs.leave(pipeline, steps.leave)
return err
}
p.ends.left.Store(&left)
}
if err := serveCompositionSteps(ctx, nc, steps); err != nil {
logger.Printf("the composition step is not heard, so no composition step runs until the assigner starts again: %v", err)
} else {
@@ -45,7 +45,9 @@ type passer struct {
kvs kvSet
w *world
logf func(string, ...any)
sdk string
// unread is the set of modules whose running build's manifest served last could not read, said once per change of it.
unread saidSet
sdk string
// say publishes a condition to the operator; said is, by condition key and state, what this process has said, so a
// pass says a condition once and a restart says it again (the router keeps one per key).
say func(oc.Condition) error
@@ -839,6 +841,24 @@ func (p *passer) staleAsk(key string, trunk map[string]meshcatalog.ModulesByModu
if w.Fp != k.FP {
return "its module's want on " + k.Node + " names " + w.Fp + " now"
}
// A backup ask is about a failed backup: a have that is no longer held, or that now carries a wait (a hold of the
// composition, not a failed backup), takes it away (design 47 G A-453). A have not read yet is not stale.
if k.Kind == asks.AskKindBackup {
if hkey, err := nodeengineclient.HaveKey(k.Node, k.Module); err == nil {
if raw := p.w.value(buckets.Have, hkey); raw != nil {
if h, err := nodeengineclient.DecodeHave(raw); err == nil {
// Past the hold is running, removed, or (a have not held) at another fingerprint, a held have naming the build it keeps running; a have mid-move (backing up again after an answer,
// applying, checking) is not (design 47 G A-453a): the ask stays, and asks again under the next number.
if h.Phase == nodeengineclient.PhaseRunning || h.Phase == nodeengineclient.PhaseRemoved || (h.Phase != nodeengineclient.PhaseHeld && h.Fp != k.FP) {
return "its module's have on " + k.Node + " has moved past the hold"
}
if h.Wait != nil {
return "its module's have on " + k.Node + " carries a wait (" + h.Wait.Reason + "), not a failed backup"
}
}
}
}
}
case asks.AskKindHold, asks.AskKindRestore:
m, on := trunk[k.Module]
if !on {
@@ -1065,6 +1085,8 @@ func (p *passer) recordAsk(ctx context.Context, module, fp string, need assign.R
// after "back up again" and is answered "again" again changes the key and the node retries. That the backup failed again is seen
// in a new have value after the answer was written.
//
// Only a held have with no wait is a failed backup (design 47 G A-453): the node-engine's other holds carry a wait.
//
// A have that holds a revert (reverts, which answers its `<node>.<module>` in reverting) is left to the revert ask.
func (p *passer) backups(ctx context.Context, reverting map[string]bool) {
wanted := p.w.wantsOf()
@@ -1077,6 +1099,9 @@ func (p *passer) backups(ctx context.Context, reverting map[string]bool) {
if err != nil || have.Phase != nodeengineclient.PhaseHeld || reverting[node+"."+module] {
continue
}
if have.Wait != nil {
continue // a hold of the composition with its wait, not a failed backup (design 47 G A-453)
}
want, ok := wanted[module][node]
if !ok || want.Removed {
continue
@@ -160,16 +160,25 @@ func TestAHoldAskedAgainAfterALiftAndARestartHasANewID(t *testing.T) {
for _, c := range []struct {
name string
want *mwc.Want
have *nodeengine.HaveByNodeModuleValue // A-453: the module's have on the node
stale bool
}{
{"its want names another fingerprint", &mwc.Want{Fp: fpOther, Self: true}, true},
{"its want is removed", &mwc.Want{Removed: true}, true},
{"its want names the fingerprint in another composition", &mwc.Want{Fp: fpTest, Composed: "sha256:" + strings.Repeat("e", 64)}, false},
{"no want yet", nil, false},
{"its want names another fingerprint", &mwc.Want{Fp: fpOther, Self: true}, nil, true},
{"its want is removed", &mwc.Want{Removed: true}, nil, true},
{"its want names the fingerprint in another composition", &mwc.Want{Fp: fpTest, Composed: "sha256:" + strings.Repeat("e", 64)}, nil, false},
{"no want yet", nil, nil, false},
// A-453: only a held have with no wait is a failed backup; a hold that carries its wait, or a have no longer held, takes the ask away.
{"its have is held with no wait", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpTest, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits}, false},
{"its have is held with a wait", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpTest, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits,
Wait: &nodeengine.Wait{Step: nodeengine.WaitStepCompose, Reason: "entry-missing", Since: now}}, true},
{"its have runs", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpTest, Phase: nodeengineclient.PhaseRunning, Health: nodeengineclient.HealthOK}, true},
{"its have backs up again (A-453a)", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpTest, Phase: nodeengineclient.PhaseBackingUp, Health: nodeengineclient.HealthWaits}, false},
{"its have is held, naming the build it keeps running", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpOther, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits}, false},
{"its have applies at another fingerprint", &mwc.Want{Fp: fpTest, Self: true}, &nodeengine.HaveByNodeModuleValue{Fp: fpOther, Phase: nodeengineclient.PhaseApplying, Health: nodeengineclient.HealthWaits}, true},
} {
t.Run("ended backup ask: "+c.name, func(t *testing.T) {
bkv := &memKV{values: map[string][]byte{}}
pb := newPasser(kvSet{buckets.Asks(): bkv, buckets.Wants(): &memKV{values: map[string][]byte{}}}, now)
pb := newPasser(kvSet{buckets.Asks(): bkv, buckets.Wants(): &memKV{values: map[string][]byte{}}, buckets.Have: &memKV{values: map[string][]byte{}}}, now)
ba := assign.BackupAsk(backKey, "ace", "web", fpTest, "sha256:"+strings.Repeat("d", 64), "w", 2, now)
bv, err := assign.EncodeAsk(backKey, ba)
if err != nil {
@@ -181,6 +190,14 @@ func TestAHoldAskedAgainAfterALiftAndARestartHasANewID(t *testing.T) {
raw, _ := mwc.EncodeWant(*c.want)
pb.w.put(buckets.Wants(), wkey, raw)
}
if c.have != nil {
hkey, _ := nodeengineclient.HaveKey("ace", "web")
raw, err := nodeengineclient.EncodeHave(*c.have)
if err != nil {
t.Fatal(err)
}
pb.w.put(buckets.Have, hkey, raw)
}
_, bask, _ := assign.DecodeAsk(backKey, bv)
body, err := tellerevents.EncodeEnded(tellerevents.Ended{Key: backKey, Ask: bask.ID, Outcome: occ.OutcomeExpired})
if err != nil {
@@ -1027,7 +1044,7 @@ func TestAsksOfAFingerprintTheTrunkLeftBehindAreTakenAway(t *testing.T) {
{"the trunk moved on", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld}, false, nil},
{"the backup's want names another fingerprint", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld, backOld}, false, &mwc.Want{Fp: fpNew, Self: true}},
{"the backup's want is removed", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld, backOld}, false, &mwc.Want{Removed: true}},
{"the backup's want names the fingerprint in another composition", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld}, false, &mwc.Want{Fp: fpOld, Composed: digNew}},
{"the backup's want names the fingerprint in another composition (its have waits for a yes: a mid-move have is not past the hold, A-453a)", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld}, false, &mwc.Want{Fp: fpOld, Composed: digNew}},
{"the trunk is at the old fingerprint", trunkOf(fpOld), []string{holdNew, holdGone, restNew, recOld}, false, nil},
{"the trunk's fingerprint is not read yet", trunkOf(""), []string{holdGone}, false, nil},
{"the composition is not read yet", trunkOf(fpNew), []string{holdOld, holdGone, restOld}, true, nil},
@@ -1417,6 +1434,17 @@ func aWantWaitsOnTheVerbEdgesTheGraphKeeps(t *testing.T) {
t.Errorf("served(%s) = %v, want %v: the running server's manifest serves w alone, late's is not read", v, served(v), want)
}
}
// The modules whose running build's manifest is not read are said once per change of their set (G-23).
var said []string
p.logf = func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }
lateOnly := assign.Nodes{Haves: map[string]map[string]nodeengine.HaveByNodeModuleValue{"g14": {"server": at(fp("f")), "late": at(fp("2"))}}}
p.served(g, lateOnly)
p.served(g, lateOnly)
p.served(g, assign.Nodes{Haves: map[string]map[string]nodeengine.HaveByNodeModuleValue{"g14": {"late": at(fp("2")), "loop": at(fp("1"))}}})
p.served(g, lateOnly)
if len(said) != 2 || !strings.Contains(said[0], "[loop late]") && !strings.Contains(said[0], "[late loop]") || !strings.Contains(said[1], "[late]") {
t.Errorf("the unread manifests were said %q, want once per change of the set", said)
}
// A-426a: a healthy build held (no catalog entry of it) or waiting for a yes serves what it serves, said once; one held unhealthy,
// or whose running statement says it stopped, does not.
held := at(fp("2"))
@@ -153,6 +153,9 @@ type pipelineEnds struct {
firsts map[string]string
// runEnded is told a run on request's group-ended, its group `run-<id>`, to drop the run's candidate (A-235); nil drops nothing.
runEnded func(group string) error
// left is told the key of a pipeline ended superseded or cancelled, whose composition steps leave the runs they read
// (design 47 G §723, A-469n); none tells nothing. Set once the composition steps are served, after the hub starts.
left atomic.Pointer[func(pipeline string) error]
// onBranches is told the branches a registration names, for the catalog half of each to be followed (design 47 G §14, §195).
onBranches func([]string)
// trunk names the first watched branch, the trunk's (design 47 G §507), the branch of a module registered with none.
@@ -285,6 +288,13 @@ func (e *pipelineEnds) branchOf(module, placed string) string {
// §29, §64).
func (e *pipelineEnds) hearPipeline(m jetstream.Msg) {
p, err := mprc.PipelineEnded(m.Data())
if left := e.left.Load(); err == nil && left != nil && (p.Outcome == mpr.PipelineEndedOutcomeSuperseded || p.Outcome == mpr.PipelineEndedOutcomeCancelled) {
if err := (*left)(p.Pipeline); err != nil {
e.logf("%s: the composition steps of %s do not all leave their runs: %v; it is heard again", m.Subject(), p.Pipeline, err)
_ = m.Nak()
return
}
}
switch {
case err != nil:
e.logf("%s is not taken: %v", m.Subject(), err)
@@ -133,4 +133,22 @@ func TestAHeldRevertIsAskedAndALiveAnswerIsWrittenIntoTheWant(t *testing.T) {
if _, raised := findings[fkey]; raised || p.w.has(buckets.Asks(), rkey) {
t.Fatalf("a newer want left the ask (%v) or the finding (%v)", p.w.has(buckets.Asks(), rkey), raised)
}
// A-453: only a held have with no wait is a failed backup and asked; the node-engine's other holds carry a wait.
ukey, _ := asks.BackupAskKey("ace", "web", fpTest)
for _, withWait := range []bool{true, false} {
h := nodeengine.HaveByNodeModuleValue{Fp: fpTest, Phase: nodeengineclient.PhaseHeld, Health: nodeengineclient.HealthWaits, Detail: "no space left"}
if withWait {
h.Wait = &nodeengine.Wait{Step: nodeengine.WaitStepCompose, Reason: "entry-missing", Since: time.Now()}
}
raw, err := nodeengineclient.EncodeHave(h)
if err != nil {
t.Fatal(err)
}
p.w.put(buckets.Have, hkey, raw)
pass()
if got := p.w.has(buckets.Asks(), ukey); got == withWait {
t.Fatalf("a held have with a wait = %v: backup ask asked = %v", withWait, got)
}
}
}
@@ -421,7 +421,9 @@ func serveTools() error {
gate.Holders(seatlist.HoldersIn(func() (map[string][]byte, error) { return wants.watched.Values(mma.StateSeatHolders) }))
wants.revisions = runtimeObjects{busstate.ObjectsOf(assigngraph.RevisionsObjects)}
tools = append(tools, wants.tools()...)
tools = append(tools, inputsRead(&inputKeeper{vals: stateobjects.ForState(buckets.InputsState), now: time.Now, logf: func(string, ...any) {}}))
keeper := &inputKeeper{vals: stateobjects.ForState(buckets.InputsState), now: time.Now, logf: func(string, ...any) {}}
tools = append(tools, inputsRead(keeper), inputsDrop(keeper, stdio.Ask))
tools = append(tools, compositionsTools(stdio.Ask)...) // the composition runs' read and drop, through their verbs (§728)
tools = append(tools, viewTools(os.Getenv, keptSource, time.Now)...)
// What each node's engine reads, as the assigner uses it (ADR 0618 §3): its own state engine-reads, asked again through it.
tools = append(tools, enginereads.Tool(engineReadsTool, "the assigner", enginereads.KeptState(stdio.State(engineReadsState)), false, time.Now))
@@ -2,6 +2,8 @@ package main
import (
"slices"
"strings"
"sync"
"git.novox.be/novox/mesh-contracts/go/clients/nodeengineclient"
"git.novox.be/novox/mesh-contracts/go/seats/nodeengine"
@@ -53,12 +55,31 @@ func (p *passer) served(g graph.Graph, n assign.Nodes) func(string) bool {
}
served, unread := graph.Served(servingHaves(n, p.countedServing), func(module, fp string) ([]string, bool) { return p.servesOf(module, fp) },
func(module string) []string { return g[module].Serves })
if len(unread) > 0 {
if p.unread.changed(unread) && len(unread) > 0 {
p.logf("the manifest of the running build of %v cannot be read, so what the module graph says they serve is counted as served", unread)
}
return served
}
// saidSet is the last set of names said, so a pass says a set again only when it changed.
type saidSet struct {
mu sync.Mutex
last string
}
// changed keeps names as the set said last and says whether it differs from the one before.
func (s *saidSet) changed(names []string) bool {
sorted := slices.Sorted(slices.Values(names))
key := strings.Join(sorted, " ")
s.mu.Lock()
defer s.mu.Unlock()
if key == s.last {
return false
}
s.last = key
return true
}
// offering is, for this pass, whether a build whose manifest offers a provision runs healthy on a node present (ADR 0603
// §5, ADR 0614 §5): every running healthy have on an online node, its build's manifest read as servesOf reads it. A have
// waiting for a person's yes on a newer build counts too: it names the build that still runs and that build's health, and
@@ -29,6 +29,9 @@ const (
PutWaitsState = "put-waits"
// InputsState holds the inputs of the composition steps, `<time>.<digest>`, which the seat mesh-wants' verb input answers (design 47 G §354).
InputsState = "inputs"
// CompositionRunsState holds the composition runs, one key `<candidate key>.<machine>` per run and its bound keys
// `bound.<run key>` and `untaken.<run key>` (design 47 G §719, §720, A-470).
CompositionRunsState = "composition-runs"
// PoliciesState is the state of the policies a person set for a module's upgrade.
PoliciesState = "policies"
)
@@ -37,16 +40,17 @@ const (
// constant of its name (design 47 G A-296, §428).
func own(state string) string { return plc.Module() + "_" + state }
func Wants() string { return own(wants.StateWants) }
func Differences() string { return own(wants.StateDifferences) }
func Asks() string { return own("asks") }
func Puts() string { return own("puts") }
func Waits() string { return own(WaitsState) }
func DeliveredAgain() string { return own(DeliveredAgainState) }
func DeliveryWaits() string { return own(DeliveryWaitsState) }
func PutWaits() string { return own(PutWaitsState) }
func Inputs() string { return own(InputsState) }
func Policies() string { return own(PoliciesState) }
func Wants() string { return own(wants.StateWants) }
func Differences() string { return own(wants.StateDifferences) }
func Asks() string { return own("asks") }
func Puts() string { return own("puts") }
func Waits() string { return own(WaitsState) }
func DeliveredAgain() string { return own(DeliveredAgainState) }
func DeliveryWaits() string { return own(DeliveryWaitsState) }
func PutWaits() string { return own(PutWaitsState) }
func Inputs() string { return own(InputsState) }
func CompositionRuns() string { return own(CompositionRunsState) }
func Policies() string { return own(PoliciesState) }
// AsksHistory is the revisions the bucket of asks keeps per key.
const AsksHistory = 5
@@ -31,7 +31,8 @@ func TestTheAssignersOwnBucketsTakeTheModuleItRunsAs(t *testing.T) {
"Wants": {Wants(), "some-assigner_wants"}, "Differences": {Differences(), "some-assigner_differences"},
"Asks": {Asks(), "some-assigner_asks"}, "Puts": {Puts(), "some-assigner_puts"}, "Policies": {Policies(), "some-assigner_policies"},
"Waits": {Waits(), "some-assigner_waits"}, "Inputs": {Inputs(), "some-assigner_inputs"},
"Assignments": {plc.AssignmentsBucket(), "some-assigner_assignments"}, "SeatHolders": {plc.SeatHoldersBucket(), "some-assigner_seat-holders"},
"CompositionRuns": {CompositionRuns(), "some-assigner_composition-runs"},
"Assignments": {plc.AssignmentsBucket(), "some-assigner_assignments"}, "SeatHolders": {plc.SeatHoldersBucket(), "some-assigner_seat-holders"},
} {
if c[0] != c[1] {
t.Errorf("%s = %q, want %q", name, c[0], c[1])
@@ -70,7 +70,7 @@ func TestManifest(t *testing.T) {
}
if want := map[string]int{"asks": 5, "differences": 64, "puts": 20, "wants": 20,
"assignments": 64, "placement": 64, "copies": 1, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5, views.MarksState: 1, views.StatusState: 1,
"engine-reads": 1, "written": 5, "waits": 5, "put-waits": 1, "delivery-waits": 1, "delivered-again": 1, "inputs": 1, "pins": 5}; !reflect.DeepEqual(states, want) {
"engine-reads": 1, "written": 5, "waits": 5, "put-waits": 1, "delivery-waits": 1, "delivered-again": 1, "inputs": 1, "composition-runs": 1, "pins": 5}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -93,7 +93,7 @@ func TestManifest(t *testing.T) {
// revision, entry and difference from its revisions kept as objects in the old section, form 1, capped at 512 MiB.
if len(m.Claims) != 2 || m.Claims[0].Name != meshmoduleassigner.Seat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"assignments", "proposals", "propose", "withdraw", "again", "placement"}) ||
m.Claims[1].Name != meshwants.Seat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus", "revision", "entry", "difference", "input"}) {
m.Claims[1].Name != meshwants.Seat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus", "revision", "entry", "difference", "input", "compositions", "compositions-drop"}) {
t.Errorf("the assigner claims %+v, want mesh-module-assigner with its five verbs and mesh-wants with its five", m.Claims)
}
if len(m.Objects) != 1 || m.Objects[0].Name != "assignment-revisions" || m.Objects[0].MaxMiB != 512 || m.Objects[0].Form != 1 {
@@ -103,7 +103,7 @@ func TestManifest(t *testing.T) {
// ADR 0621 §1: the first-view tools assignments, seat-holders, wants and asks (§717) and design 25 § 12.2's six read tools, of the
// program's bundle, which runs in a runtime of its own on the assigner's account; ADR 0618 §3: and assigner_engine_reads,
// which lists and asks again what each node's engine reads.
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts", "assigner_waits", "assigner_inputs", "assignments", "seat-holders", "wants", "asks",
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts", "assigner_waits", "assigner_inputs", "assigner_inputs_drop", "assigner_compositions", "assigner_compositions_drop", "assignments", "seat-holders", "wants", "asks",
"assigner_read_assignments", "assigner_read_seat_holders", "assigner_read_wants", "assigner_read_findings", "assigner_read_differences", "assigner_read_copies",
"assigner_first_views", "assigner_first_view_again", "assigner_engine_reads", "assigner_seat_list", "assigner_seat_list_again", "assigner_ask_again", "assigner_written", "assigner_why", "assigner_pins", "assigner_unpin", "assigner_delivered_again"}) {
t.Errorf("the assigner serves tools %v, want its three put tools, three first views, six read tools, assigner_engine_reads, the seat list's two (ADR 0636 §7) assigner_ask_again (ADR 0668 §5) and assigner_why (design 47 G A-439)", m.Tools)
@@ -137,7 +137,7 @@ func TestManifest(t *testing.T) {
// ADR 0618 §4: and it hears each node's engine say its reads changed; ADR 0636 §4 (issue 1966): and the seat list's two events;
// issue 1966 note 151: and the teller's ask-ended, its subject the one the assigner hears; §376, §389: its own version-delivered and
// the watchman's machine-away and machine-back, read whole at its start.
if !slices.Equal(m.Consumes, []string{"mesh-conditions.findings-asked", "mesh-conditions.condition-raised", "mesh-conditions.condition-changed", "mesh-conditions.condition-cleared", "mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-delivery.source-moved", "mesh-catalog.version-released", "mesh-wants.version-delivered", "mesh-watchdogs.machine-away", "mesh-watchdogs.machine-back", "mesh-catalog.seat-definition-changed", "mesh-catalog.seat-definition-removed", builds.BuilderModule + "." + builds.BuildsStreamEvents, buckets.IndexerModule + ".take-in.taken", buckets.TellerModule + "." + tellerevents.AskEnded, buckets.TellerModule + ".views." + buckets.AnswersState, "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.group-ended", "node-build-go.step-ended", "node-engine.manifest-reads-changed", "node-engine.running", "mesh-delivery.pinned", "mesh-delivery.freed", "mesh-delivery.heads-replaced", "mesh-delivery.pins-stated"}) {
if !slices.Equal(m.Consumes, []string{"mesh-conditions.findings-asked", "mesh-conditions.condition-raised", "mesh-conditions.condition-changed", "mesh-conditions.condition-cleared", "mesh-catalog.catalog-revision", "mesh-catalog.registration-changed", "mesh-delivery.source-moved", "mesh-catalog.version-released", "mesh-wants.version-delivered", "mesh-watchdogs.machine-away", "mesh-watchdogs.machine-back", "mesh-catalog.seat-definition-changed", "mesh-catalog.seat-definition-removed", builds.BuilderModule + "." + builds.BuildsStreamEvents, buckets.IndexerModule + ".take-in.taken", buckets.TellerModule + "." + tellerevents.AskEnded, buckets.TellerModule + ".views." + buckets.AnswersState, "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.group-ended", "node-build-go.step-ended", "node-build-go.ask-taken", "node-build-go.step-progress", "node-engine.manifest-reads-changed", "node-engine.running", "mesh-delivery.pinned", "mesh-delivery.freed", "mesh-delivery.heads-replaced", "mesh-delivery.pins-stated"}) {
t.Errorf("consumes = %v, want the catalog half's revisions (design 47 G §195), the teller's ask-ended, the engines' reads changed and what they run", m.Consumes)
}
if want := []string{"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules",
@@ -147,7 +147,8 @@ func TestManifest(t *testing.T) {
"seat:mesh-delivery.source", // §507: the branches the gate keeps a source revision of, before the first registrations
"seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-acts", "seat:mesh-hand-acts.warranted", // ADR 0565 §9, §11: the seat alone
"seat:node-backup.backed-up", "seat:node-backup.now",
"seat:node-build-go.accept.check", // A-231: the composition run's ask on the Go build seat's queue check (accept.check.<asker>)
"seat:node-build-go.accept.check", // A-231
"seat:node-build-go.drop", // §720, A-469s: a composition run past its bound drops its ask: the composition run's ask on the Go build seat's queue check (accept.check.<asker>)
"seat:node-build-go.output", // A-254: a composition run's answer kept as the build seat's object
"seat:node-engine.compose-candidate", // §354, A-226: each machine's composition input
"seat:node-engine.manifest-reads", // ADR 0618 §3
@@ -167,13 +168,21 @@ func TestManifest(t *testing.T) {
}
file, _ := os.ReadFile("module.json")
if err := json.Unmarshal(file, &raw); err != nil || raw.Config["outside-wait-minutes"].Default != float64(30) || raw.Config["outside-settle-seconds"].Default != float64(90) ||
raw.Config["ask-operator-for"].Default != "take-off,remove-node,stops" || len(raw.Config) != 3 {
raw.Config["ask-operator-for"].Default != "take-off,remove-node,stops" || len(raw.Config) != 8 ||
raw.Config["composition-runs-cap-mib"].Default != float64(16) || raw.Config["composition-runs-kept"].Default != "24h" ||
raw.Config["composition-step-bound"].Default != "30m" || raw.Config["run-untaken-bound"].Default != "24h" || raw.Config["blame-tries"].Default != float64(20) {
t.Errorf("config = %+v (%v), want outside-wait-minutes of 30, outside-settle-seconds of 90 and ask-operator-for of take-off,remove-node,stops (A-425a)", raw.Config, err)
}
for _, f := range []struct{ env, path, key string }{
{"MESH_ASSIGNER_OUTSIDE_WAIT_FILE", "${dir:state}/outside-wait-minutes", "outside-wait-minutes"},
{"MESH_ASSIGNER_OUTSIDE_SETTLE_FILE", "${dir:state}/outside-settle-seconds", "outside-settle-seconds"},
{"MESH_ASSIGNER_ASK_OPERATOR_FOR_FILE", "${dir:state}/ask-operator-for", "ask-operator-for"},
// design 47 G §719, §720, §721, A-469x: the composition runs' settings
{"MESH_ASSIGNER_COMPOSITION_RUNS_CAP_MIB_FILE", "${dir:state}/composition-runs-cap-mib", "composition-runs-cap-mib"},
{"MESH_ASSIGNER_COMPOSITION_RUNS_KEPT_FILE", "${dir:state}/composition-runs-kept", "composition-runs-kept"},
{"MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE", "${dir:state}/composition-step-bound", "composition-step-bound"},
{"MESH_ASSIGNER_RUN_UNTAKEN_BOUND_FILE", "${dir:state}/run-untaken-bound", "run-untaken-bound"},
{"MESH_ASSIGNER_BLAME_TRIES_FILE", "${dir:state}/blame-tries", "blame-tries"},
} {
if !strings.Contains(string(file), `"`+f.env+`": "`+f.path+`"`) || !strings.Contains(string(file), `"path": "`+f.path+`"`) ||
!strings.Contains(string(file), `"content": "${config:`+f.key+`}"`) {
+90 -2
View File
@@ -25,7 +25,9 @@
"revision",
"entry",
"difference",
"input"
"input",
"compositions",
"compositions-drop"
]
}
],
@@ -59,6 +61,8 @@
"mesh-pipeline-runner.pipeline-ended",
"mesh-pipeline-runner.group-ended",
"node-build-go.step-ended",
"node-build-go.ask-taken",
"node-build-go.step-progress",
"node-engine.manifest-reads-changed",
"node-engine.running",
"mesh-delivery.pinned",
@@ -82,6 +86,7 @@
"seat:node-backup.backed-up",
"seat:node-backup.now",
"seat:node-build-go.output",
"seat:node-build-go.drop",
"seat:node-engine.compose-candidate",
"seat:node-engine.manifest-reads",
"seat:operator-channel.asks",
@@ -112,6 +117,9 @@
"assigner_puts",
"assigner_waits",
"assigner_inputs",
"assigner_inputs_drop",
"assigner_compositions",
"assigner_compositions_drop",
"assignments",
"seat-holders",
"wants",
@@ -157,6 +165,7 @@
{ "name": "delivery-waits", "history": 1 },
{ "name": "delivered-again", "history": 1 },
{ "name": "inputs", "history": 1, "max-mib": 64 },
{ "name": "composition-runs", "history": 1, "key-ttl": true, "max-mib": 64 },
{ "name": "pins", "history": 5, "max-mib": 4 }
],
"config": {
@@ -181,6 +190,40 @@
"description": "the change kinds of a proposal the assigner asks the operator before it writes it, separated by commas: place, take-off, pin, unpin, hold, unhold, decide, config, add-node, remove-node, and stops for a verdict whose answers say a node would stop running a module (design 47 G A-425, A-425a). A proposal none of whose changes is of a kind named is applied at once when its verdict passes, said in the log and by assignments-changed. The default is take-off,remove-node,stops; empty asks for nothing.",
"trusted": false,
"default": "take-off,remove-node,stops"
},
"composition-runs-cap-mib": {
"type": "number",
"description": "the cap of the state composition-runs in MiB, the process refusing a new composition run at it with its steps ending could not run naming the fill (design 47 G section 719, A-469d); its maximum is the manifest's max-mib.",
"minimum": 1,
"maximum": 64,
"trusted": false,
"default": 16
},
"composition-runs-kept": {
"type": "string",
"description": "how long a composition run's key is kept after it ended, a Go duration such as 24h (design 47 G section 719, A-469x).",
"trusted": false,
"default": "24h"
},
"composition-step-bound": {
"type": "string",
"description": "how long a taken composition run's ask may go without saying its progress before the run ends could not run, a Go duration such as 30m; written again at each step-progress (design 47 G section 720, A-469x).",
"trusted": false,
"default": "30m"
},
"run-untaken-bound": {
"type": "string",
"description": "how long a composition run's ask may wait for a build node to take it before the run ends could not run, a Go duration such as 24h (design 47 G section 720, A-466s, A-469x).",
"trusted": false,
"default": "24h"
},
"blame-tries": {
"type": "number",
"description": "the most compositions a machine's composer makes to find the modules it blames, carried in the run's request as blame_tries (design 47 G section 721, A-466r).",
"minimum": 1,
"maximum": 1000,
"trusted": false,
"default": 20
}
},
"objects": [
@@ -230,6 +273,46 @@
"mode": "0600",
"content": "${config:ask-operator-for}"
},
{
"id": "composition-runs-cap-mib",
"type": "file",
"path": "${dir:state}/composition-runs-cap-mib",
"owner": "mesh-delivery-assigner",
"mode": "0600",
"content": "${config:composition-runs-cap-mib}"
},
{
"id": "composition-runs-kept",
"type": "file",
"path": "${dir:state}/composition-runs-kept",
"owner": "mesh-delivery-assigner",
"mode": "0600",
"content": "${config:composition-runs-kept}"
},
{
"id": "composition-step-bound",
"type": "file",
"path": "${dir:state}/composition-step-bound",
"owner": "mesh-delivery-assigner",
"mode": "0600",
"content": "${config:composition-step-bound}"
},
{
"id": "run-untaken-bound",
"type": "file",
"path": "${dir:state}/run-untaken-bound",
"owner": "mesh-delivery-assigner",
"mode": "0600",
"content": "${config:run-untaken-bound}"
},
{
"id": "blame-tries",
"type": "file",
"path": "${dir:state}/blame-tries",
"owner": "mesh-delivery-assigner",
"mode": "0600",
"content": "${config:blame-tries}"
},
{
"id": "assigner",
"type": "process",
@@ -242,7 +325,12 @@
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_ASSIGNER_OUTSIDE_WAIT_FILE": "${dir:state}/outside-wait-minutes",
"MESH_ASSIGNER_OUTSIDE_SETTLE_FILE": "${dir:state}/outside-settle-seconds",
"MESH_ASSIGNER_ASK_OPERATOR_FOR_FILE": "${dir:state}/ask-operator-for"
"MESH_ASSIGNER_ASK_OPERATOR_FOR_FILE": "${dir:state}/ask-operator-for",
"MESH_ASSIGNER_COMPOSITION_RUNS_CAP_MIB_FILE": "${dir:state}/composition-runs-cap-mib",
"MESH_ASSIGNER_COMPOSITION_RUNS_KEPT_FILE": "${dir:state}/composition-runs-kept",
"MESH_ASSIGNER_COMPOSITION_STEP_BOUND_FILE": "${dir:state}/composition-step-bound",
"MESH_ASSIGNER_RUN_UNTAKEN_BOUND_FILE": "${dir:state}/run-untaken-bound",
"MESH_ASSIGNER_BLAME_TRIES_FILE": "${dir:state}/blame-tries"
},
"user": "mesh-delivery-assigner",
"health": {
@@ -20,7 +20,7 @@ func tcs(language, version string, nodes ...string) []buildstep.NodeToolchain {
}
func chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { // 1607/AC4, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "novox", "ace", "g14", "shanks")
entries := tcs("go", "go1.27", "novox", "ace", "g14", "shanks")
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 3, "ace": 1, "g14": 1, "shanks": 0}), nil, 2)
if err != nil {
t.Fatal(err)
@@ -28,13 +28,13 @@ func chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { /
if len(nodes) != 2 || nodes[0] != "shanks" || nodes[1] != "ace" {
t.Errorf("nodes = %v, want [shanks ace]: fewest steps first, a tie by name", nodes)
}
if tc.Version != "go1.27.1" || tc.Recipe != "1" || tc.Language != "go" {
if tc.Version != "go1.27" || tc.Recipe != "1" || tc.Language != "go" {
t.Errorf("toolchain = %+v", tc)
}
}
func chooseWithOneNodeAnswersOneNode(t *testing.T) { // 1607/AC4
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "novox"), nil, nil, 2)
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27", "novox"), nil, nil, 2)
if err != nil || len(nodes) != 1 || nodes[0] != "novox" {
t.Errorf("nodes = %v, err %v, want [novox]", nodes, err)
}
@@ -45,9 +45,9 @@ func chooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies(t *testing.T)
if err == nil || !strings.Contains(err.Error(), "typescript") || !strings.Contains(err.Error(), buildstep.ToolchainNoReport) {
t.Errorf("no report: error = %v, want it to name typescript and %q", err, buildstep.ToolchainNoReport)
}
entries := append(tcs("go", "go1.27.1", "novox"), tcs("go", "go1.26.0", "ace")...)
entries := append(tcs("go", "go1.27", "novox"), tcs("go", "go1.26.0", "ace")...)
_, _, err = ChooseNodes("go", entries, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), "go1.26.0") || !strings.Contains(err.Error(), "go") {
if err == nil || !strings.Contains(err.Error(), "go1.27") || !strings.Contains(err.Error(), "go1.26") || !strings.Contains(err.Error(), "go") {
t.Errorf("disagreeing nodes: error = %v, want both versions named", err)
}
}
@@ -67,17 +67,17 @@ func agreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer(t *testing.T) { // 16
}
func chooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs(t *testing.T) { // 1607 review 3, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "ace", "g14", "novox")
entries := tcs("go", "go1.27", "ace", "g14", "novox")
online := func(n string) bool { return n != "g14" } // a sleeping laptop
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 5}), online, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27.1" {
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27" {
t.Errorf("nodes = %v, toolchain %+v, err %v, want [ace novox] with g14 left out although it runs nothing", nodes, tc, err)
}
_, tc, err = ChooseNodes("go", entries, nil, func(string) bool { return false }, 2)
if err == nil || !strings.Contains(err.Error(), "online") || !strings.Contains(err.Error(), "g14") || !strings.Contains(err.Error(), "go") {
t.Errorf("none online: error = %v, want it to name the nodes that report and that none is online", err)
}
if tc.Version != "go1.27.1" {
if tc.Version != "go1.27" {
t.Errorf("none online: the agreed toolchain %+v is still answered", tc)
}
}
@@ -88,7 +88,7 @@ func placesRunning(running map[string]int) *Places {
}
func choosePrefersNodesBelowTheirCapacity(t *testing.T) { // round 3
entries := tcs("go", "go1.27.1", "ace", "novox", "shanks")
entries := tcs("go", "go1.27", "ace", "novox", "shanks")
// shanks has capacity 3 with 2 running: room; ace has the default 1 with 1 running: none; novox has none running.
p := placesRunning(map[string]int{"shanks": 2, "ace": 1})
p.Capacity = map[string]int{"shanks": 3}
@@ -58,7 +58,7 @@ func TestToolchainGate(t *testing.T) {
t.Run("not placed on ace: its report is no builder's", func(t *testing.T) {
gate, set, said := start(t, true)
key, value := gateReport(t, "ace", "v26.12.0", old)
key, value := gateReport(t, "ace", "v26.11.0", old)
gate.Toolchain(testNode, key, value, false)
if a := buildstep.ToolchainAgreement(set.Entries("typescript")); len(a.SittingOut) != 1 || a.SittingOut[0].Node != "ace" { // ADR 0584: the majority is agreed, ace's report counts as the one that sits out
t.Fatalf("before the assignments is settled every report counts; got %+v", a)
@@ -68,7 +68,7 @@ func TestToolchainGate(t *testing.T) {
if !gate.Settled() {
t.Fatal("settling left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
if v, none := agreedOf(set); v != "v26.12" || none != "" {
t.Fatalf("the placed nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "does not place mesh-build-node on ace") {
@@ -80,14 +80,14 @@ func TestToolchainGate(t *testing.T) {
// Placed on ace with a report written since: it counts, and the builders really disagree.
k, v = placing(t, "ace", testGo, testNode)
gate.Assignments(k, v, false, old.Add(-time.Minute))
if a := buildstep.ToolchainAgreement(set.Entries("typescript")); a.Toolchain.Version != "v26.12.1" || len(a.SittingOut) != 1 || a.SittingOut[0].Version != "v26.12.0" {
if a := buildstep.ToolchainAgreement(set.Entries("typescript")); a.Toolchain.Version != "v26.12" || len(a.SittingOut) != 1 || a.SittingOut[0].Version != "v26.11" {
t.Fatalf("builders that run different versions: the majority is agreed and ace sits out; got %+v", a)
}
// Taken off ace again (its key deleted): its report goes again.
if !gate.Assignments(k, nil, true, placed) {
t.Fatal("taking ace's assignments away passed nothing on")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
if v, none := agreedOf(set); v != "v26.12" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
// A report deleted from the bucket goes whether its node is placed or not.
@@ -131,7 +131,7 @@ func TestToolchainGate(t *testing.T) {
if gate.Assignments(k, v, false, placed) {
t.Fatal("assignments with no report passed something on")
}
key, value := gateReport(t, "ace", "v26.12.0", c.at)
key, value := gateReport(t, "ace", "v26.11.0", c.at)
if got := gate.Toolchain(testNode, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
@@ -143,7 +143,7 @@ func TestToolchainGate(t *testing.T) {
if c.counts {
return
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
if v, none := agreedOf(set); v != "v26.12" || none != "" {
t.Fatalf("ace's left report is no builder's; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "before mesh-build-node was placed on ace at 2026-10-25T19:56:45Z") {
@@ -158,7 +158,7 @@ func TestToolchainGate(t *testing.T) {
t.Run("no assignments held: every report counts", func(t *testing.T) {
gate, set, _ := start(t, false)
key, value := gateReport(t, "ace", "v26.12.0", old)
key, value := gateReport(t, "ace", "v26.11.0", old)
gate.Toolchain(testNode, key, value, false)
gate.Settled()
if n := len(set.Entries("typescript")); n != 4 {
@@ -183,7 +183,7 @@ func TestToolchainGate(t *testing.T) {
for _, node := range append([]string{"ace"}, three...) {
key, value := gateReport(t, node, "v26.12.1", placed.Add(time.Minute))
if node == "ace" {
key, value = gateReport(t, node, "v26.12.0", old)
key, value = gateReport(t, node, "v26.11.0", old)
}
gate.Toolchain(testNode, key, value, false)
}
@@ -69,7 +69,7 @@ type AgreedDigest struct {
func Digests(entries []buildstep.NodeToolchain, agreed buildstep.Toolchain) []AgreedDigest {
by := map[string][]string{}
for _, e := range entries {
if e.Version == agreed.Version && e.Recipe == agreed.Recipe && e.Digest != "" {
if buildstep.LanguageVersion(e.Version) == agreed.Version && e.Recipe == agreed.Recipe && e.Digest != "" {
by[e.Digest] = append(by[e.Digest], e.Node)
}
}
@@ -41,7 +41,7 @@ func TestDigestsOfTheAgreedVersion(t *testing.T) {
{Node: "novox", Toolchain: tc("go", "1.26.0", "sha256:b")}, {Node: "ace", Toolchain: tc("go", "1.26.0", "sha256:a")},
{Node: "g14", Toolchain: tc("go", "1.26.0", "sha256:a")}, {Node: "shanks", Toolchain: tc("go", "1.25.0", "sha256:c")},
{Node: "old", Toolchain: tc("go", "1.26.0", "")}}
got := Digests(entries, tc("go", "1.26.0", ""))
got := Digests(entries, buildstep.ToolchainAgreement(entries[:4]).Toolchain)
want := []AgreedDigest{{Digest: "sha256:a", Nodes: []string{"ace", "g14"}}, {Digest: "sha256:b", Nodes: []string{"novox"}}}
if !reflect.DeepEqual(got, want) {
t.Errorf("digests = %+v, want %+v", got, want)
@@ -369,7 +369,7 @@ func (b *bed) module(t *testing.T, name, fixture, path string) string {
}
// bothToolchains are the toolchains the bed's nodes report, as a module's fingerprint names them.
var bothToolchains = map[string]builds.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
var bothToolchains = map[string]builds.ToolchainUsed{"go": {Version: "go1.27", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
// moduleWith is module with the toolchains its fingerprint was computed with.
func (b *bed) moduleWith(t *testing.T, name, fixture, path string, toolchains map[string]builds.ToolchainUsed) string {
@@ -546,7 +546,7 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
t.Errorf("artifact = %+v, want the digest every node built and the reference its push answered (ADR 0498 clause 1)", a)
}
}
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
if v.Toolchains["go"].Version != "go1.27" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
// A-115: the builder's process answers the seat mesh-builds' verb reserved over the bus; the reservation of a module it never
@@ -750,7 +750,7 @@ func TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFing
}
}
// TestAToolchainChangeDoesNotBuildTheOldFingerprint: the fingerprint names go1.27.1, the nodes then agree on go1.27.2 (a
// TestAToolchainChangeDoesNotBuildTheOldFingerprint: the fingerprint names go1.27.1, the nodes then agree on go1.28.0 (a
// report of both nodes), and the builder writes no key for the old fingerprint although it was woken by the report; a
// module value naming go1.27.2 (the new fingerprint, computed by the indexer) is built.
func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/AC4
@@ -765,15 +765,15 @@ func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/A
if err := st.Purge(b.ctx, jetstream.WithPurgeSubject("mesh.seat.node-build-go.event.toolchain-reported.>")); err != nil {
t.Fatal(err)
}
b.report(t, "mesh-build-go", "n1", "go", "go1.27.2", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.2", "1")
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // names go1.27.1
b.report(t, "mesh-build-go", "n1", "go", "go1.28.0", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.28.0", "1")
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // names go1.27
b.start(t, "n1", nil)
// Another module, whose fingerprint names the new toolchain, is built; by then the old one has been looked at.
b.moduleCommit(t, "netcheck2", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck",
map[string]builds.ToolchainUsed{"go": {Version: "go1.27.2", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}, commit)
map[string]builds.ToolchainUsed{"go": {Version: "go1.28", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}, commit)
b.await(t, "netcheck2", isState(builds.BuildDone))
// The builder never agreed on go1.27.1, not for a moment: that is what would build the old fingerprint.
// The builder never agreed on go1.27, not for a moment: that is what would build the old fingerprint.
info, err := st.Info(b.ctx)
if err != nil {
t.Fatal(err)
@@ -786,12 +786,12 @@ func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/A
if err != nil {
t.Fatal(err)
}
if strings.HasSuffix(m.Subject, ".toolchain-agreed") && strings.Contains(string(m.Data), `"version":"go1.27.1"`) {
t.Errorf("the builder said the nodes agree on go1.27.1: %s", m.Data)
if strings.HasSuffix(m.Subject, ".toolchain-agreed") && strings.Contains(string(m.Data), `"version":"go1.27"`) {
t.Errorf("the builder said the nodes agree on go1.27: %s", m.Data)
}
}
if _, err := b.builds.Get(b.ctx, mustKey(t, "netcheck")); err == nil {
t.Error("the old fingerprint, naming go1.27.1, was built with go1.27.2; the log: " + b.out.String())
t.Error("the old fingerprint, naming go1.27, was built with go1.28.0; the log: " + b.out.String())
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("build_go_build was called %d times, want 1 (the new fingerprint on one node only)", n)
@@ -245,6 +245,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Unreleased: reserved.unreleasedWords, Max: republishMax, Logf: logger.Printf,
Holders: holders} // a registry seat's holder from the seat-holders view, from the start (design 47 G A-416, A-416a)
acts.Republish = republish.Republish
reserved.publishAtEnd = republish.Ended
again := &republishing{rp: republish}
// The operator is asked once a key's event builds are spent (design 47 G §568): the builder's asker on the seat operator-channel.
decided := &decisions{subject: oc.SubjectDecided(cred.Module)}
@@ -51,6 +51,8 @@ type reservations struct {
call toolCaller
// say says package-published on the seat mesh-builds; nil says none.
say func(context.Context, meshbuilds.PackagePublished) error
// publishAtEnd asks the tagged publish of a reservation whose build has ended (design 47 G A-473); nil asks none.
publishAtEnd func(ctx context.Context, module, version string) ([]string, error)
}
// newReservations makes the reserving over store; the released versions of a module are asked of the seat mesh-catalog's
@@ -107,6 +109,12 @@ func (rv *reservations) Built(ctx context.Context, module, version string, diges
for _, m := range marked {
rv.sayPublished(ctx, m, m.Reserved.Artifacts[m.Artifact])
}
// The build's end asks the tagged publish itself, at the holder of each artifact's registry seat (design 47 G A-473).
if rv.publishAtEnd != nil {
if _, err := rv.publishAtEnd(ctx, module, version); err != nil {
rv.logf("the publish of %s %s is not asked at the build's end: %v", module, version, err)
}
}
return nil
}
@@ -102,9 +102,17 @@ func TestTheCatalogsWordsMoveTheReservations(t *testing.T) {
if err != nil || res.Version != "1.4.3" {
t.Fatalf("reserved %q (%v), want 1.4.3 after the catalog's 1.4.2", res.Version, err)
}
var asked []string
rv.publishAtEnd = func(_ context.Context, module, version string) ([]string, error) {
asked = append(asked, module+" "+version)
return nil, nil
}
if err := rv.Built(ctx, "m", "1.4.3", map[string]string{"m:a": "sha256:a", "m:b": "sha256:b"}, nil); err != nil {
t.Fatal(err)
}
if len(asked) != 1 || asked[0] != "m 1.4.3" {
t.Fatalf("the build's end asked the publish of %v, want m 1.4.3 once (A-473)", asked)
}
boundKey := versions.ReservedBoundKey("m", "1.4")
if !bounds.keys[boundKey] {
t.Fatalf("the publish's bound is not armed: %v", bounds.keys)
@@ -111,7 +111,7 @@ func TestTheBuilderKeepsTheToolchainsItHeardAndForgetsThemOnStopped(t *testing.T
if len(keys) != 3 || keys[0] != "ace.go" {
t.Fatalf("the state holds %v, want the three nodes' keys <node>.go", keys)
}
if len(r.said) != 1 || r.said[0].Version != "1.26.0" || r.said[0].Language != "go" {
if len(r.said) != 1 || r.said[0].Version != "1.26" || r.said[0].Language != "go" {
t.Fatalf("toolchain-agreed said %+v, want one for go 1.26.0 (said only when the version changes)", r.said)
}
// The first report agreed it with one digest; a later digest of the same version is not said again.
@@ -120,13 +120,13 @@ func TestTheBuilderKeepsTheToolchainsItHeardAndForgetsThemOnStopped(t *testing.T
}
// A restarted builder reads the keys back and agrees the same version with no new report; the digests it says are all three nodes'.
again := newHeardRig(t, state)
if len(again.said) != 1 || again.said[0].Version != "1.26.0" || len(again.said[0].Digests) != 2 || again.said[0].Digests[0].Nodes[1] != "g14" {
if len(again.said) != 1 || again.said[0].Version != "1.26" || len(again.said[0].Digests) != 2 || again.said[0].Digests[0].Nodes[1] != "g14" {
t.Fatalf("a restarted builder said %+v, want go 1.26.0 with the digests of its three nodes", again.said)
}
// A version change is said; the stopped build module of ace takes ace's key out, a placement taking nothing.
again.report(t, "node-build-go", "ace", "go", "1.27.0", da)
again.report(t, "node-build-go", "g14", "go", "1.27.0", da)
if len(again.said) != 2 || again.said[1].Version != "1.27.0" {
if len(again.said) != 2 || again.said[1].Version != "1.27" {
t.Fatalf("a changed agreed version was not said: %+v", again.said)
}
at := time.Now()
@@ -197,7 +197,7 @@ func newRig(t *testing.T, artifacts ...catalogue.Artifact) *rig {
}
// agreedToolchains are what the rig's nodes report, and so what a fingerprint computed now names.
var agreedToolchains = map[string]builds.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
var agreedToolchains = map[string]builds.ToolchainUsed{"go": {Version: "go1.27", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
func moduleValue(t *testing.T, path string) []byte { return moduleValueWith(t, path, agreedToolchains) }
@@ -301,7 +301,7 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
if !slices.Equal(v.Needs, []string{"form:artifact-store"}) || v.Artifacts[0].References != nil {
t.Errorf("needs = %v, references = %v", v.Needs, v.Artifacts[0].References)
}
if got := v.Toolchains["go"]; got.Version != "go1.27.1" || got.Recipe != "1" {
if got := v.Toolchains["go"]; got.Version != "go1.27" || got.Recipe != "1" {
t.Errorf("toolchains = %+v, want go and typescript named", v.Toolchains)
}
if got := v.Toolchains["typescript"]; got.Version != "tsc5" {
@@ -684,32 +684,32 @@ func (d *seatDrops) get() []string {
func TestAKeyIsBuiltOnlyWithTheToolchainItsFingerprintNamesAndNothingIsWrittenOtherwise(t *testing.T) { // 1615/AC4
r := newRig(t, goBundle)
old := map[string]builds.ToolchainUsed{"go": {Version: "go1.26.0", Recipe: "1"}}
old := map[string]builds.ToolchainUsed{"go": {Version: "go1.26", Recipe: "1"}}
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", old))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Run("a recipe that moves builds the module again under its new fingerprint", aRecipeThatMovesBuildsTheModuleAgainUnderItsNewFingerprint)
t.Errorf("keys %v and %d calls for a fingerprint of go1.26.0 while the nodes agree on go1.27.1, want neither", r.st.vals, len(r.bm.calls))
t.Errorf("keys %v and %d calls for a fingerprint of go1.26 while the nodes agree on go1.27, want neither", r.st.vals, len(r.bm.calls))
}
// A recipe that differs is a different toolchain too, and a value that names none (written before) is not built.
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]builds.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "0"}}))
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]builds.ToolchainUsed{"go": {Version: "go1.27", Recipe: "0"}}))
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", nil))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Errorf("keys %v and %d calls for a recipe that differs or a value that names no toolchain", r.st.vals, len(r.bm.calls))
}
// The toolchain moves under a key that is not built yet: the old fingerprint stays unbuilt, the new one builds.
r.tools.Set("ace", tcOf("go", "go1.27.2"))
r.tools.Set("novox", tcOf("go", "go1.27.2"))
r.tools.Set("ace", tcOf("go", "go1.28.0"))
r.tools.Set("novox", tcOf("go", "go1.28.0"))
r.b.Reconsider(context.Background())
r.b.Wait()
if len(r.st.vals) != 0 {
t.Errorf("keys %v: the old fingerprint was built with the new toolchain", r.st.vals)
}
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]builds.ToolchainUsed{"go": {Version: "go1.27.2", Recipe: "1"}}))
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]builds.ToolchainUsed{"go": {Version: "go1.28", Recipe: "1"}}))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != builds.BuildDone || v.Toolchains["go"].Version != "go1.27.2" {
t.Errorf("build = %+v, want done with go1.27.2", v)
if v := valueOf(t, r.st, r.key); v.State != builds.BuildDone || v.Toolchains["go"].Version != "go1.28" {
t.Errorf("build = %+v, want done with go1.28", v)
}
}
@@ -730,7 +730,7 @@ func aRecipeThatMovesBuildsTheModuleAgainUnderItsNewFingerprint(t *testing.T) {
t.Errorf("%d calls, was %d: the first fingerprint was built again for the new recipe", len(r.bm.calls), calls)
}
data, err := meshcatalogclient.EncodeModule(meshcatalog.ModulesByModuleValue{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Fp: "src1:" + hex64('f'), Fp2: fp2,
Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(map[string]builds.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "2"}})})
Manifest: mdigest('a'), Path: "modules/m", Toolchains: CatalogToolchains(map[string]builds.ToolchainUsed{"go": {Version: "go1.27", Recipe: "2"}})})
if err != nil {
t.Fatal(err)
}
@@ -41,7 +41,7 @@ func (w *memWaits) End(_ context.Context, module, fp, how string) bool {
// built and ended `ran` when the agreed toolchain moves to that version; and one held is built with the running build
// module by Release and ended `released`.
func TestAnEntryWhoseFingerprintNamesAnotherToolchainWaitsAndIsBuiltWhenItIsAgreedOrReleased(t *testing.T) {
old := map[string]builds.ToolchainUsed{"go": {Version: "go1.26.0", Recipe: "1"}}
old := map[string]builds.ToolchainUsed{"go": {Version: "go1.26", Recipe: "1"}}
t.Run("ran", func(t *testing.T) {
r := newRig(t, goBundle)
w := &memWaits{}
@@ -52,7 +52,7 @@ func TestAnEntryWhoseFingerprintNamesAnotherToolchainWaitsAndIsBuiltWhenItIsAgre
t.Fatalf("held %v, keys %v: want one wait and nothing written", w.held, r.st.vals)
}
for _, h := range w.held {
if h.On() != "toolchain:go@go1.26.0" {
if h.On() != "toolchain:go@go1.26" {
t.Errorf("the wait is on %s", h.On())
}
}
@@ -75,7 +75,7 @@ func TestAnEntryWhoseFingerprintNamesAnotherToolchainWaitsAndIsBuiltWhenItIsAgre
t.Fatalf("release refused: %s", why)
}
r.b.Wait()
if got := valueOf(t, r.st, r.key); got.State != builds.BuildDone || got.Toolchains["go"].Version != "go1.27.1" || len(w.ended) != 1 || w.ended[0] != WaitReleased {
if got := valueOf(t, r.st, r.key); got.State != builds.BuildDone || got.Toolchains["go"].Version != "go1.27" || len(w.ended) != 1 || w.ended[0] != WaitReleased {
t.Errorf("build %+v, ended %v: want done with the running toolchain and a wait ended released", got, w.ended)
}
if why := r.b.Release(context.Background(), "m", fp1, "again", "operator"); why == "" {
@@ -444,6 +444,18 @@ func (r *Republisher) Republish(ctx context.Context, module, version string) ([]
return r.queue(ctx, vs, r.holdersFn, 0)
}
// Ended queues a reserved version whose build has just ended (built-at set, every artifact's digest known) at the holder of
// each artifact's registry seat, as a holder's running does, and drains it (design 47 G A-473). A version with no built-at, or
// whose artifacts are all published, queues nothing. A holder that refuses or does not answer leaves the version in
// version-unreleased with its words (send says it); a seat with no holder leaves it for the holder's running.
func (r *Republisher) Ended(ctx context.Context, module, version string) ([]string, error) {
v, ok, err := r.Reserved.Of(ctx, module, version)
if err != nil || !ok || v.BuiltAt.IsZero() || len(v.Missing()) == 0 {
return nil, err
}
return r.queue(ctx, []ReservedVersion{v}, r.holdersFn, 0)
}
// Queued are the queued versions, per holder in the order they are sent, and the acted keys with their sequences:
// builder_publish_queue.
type Queued struct {
@@ -119,6 +119,26 @@ func TestRepublish(t *testing.T) {
t.Errorf("a key stays in the queue after its outcome: %v", keys)
}
})
t.Run("a build's end queues its version at the holders, without any event (A-473)", func(t *testing.T) {
b := newRepublishBed(t, 50)
b.holders = map[string][]string{SeatArtifacts: {"oci"}}
b.now = b.now.Add(time.Minute)
v, err := b.Reserved.Reserve(ctx, "m", "0.1", []string{"m:prog", "m:lib"}, Source{Fingerprint: "src2:m"})
if err != nil {
t.Fatal(err)
}
if _, err := b.Ended(ctx, "m", v.Version); err != nil || len(b.sent) != 0 {
t.Fatalf("a build not ended sent %d (%v)", len(b.sent), err)
}
for ref, d := range map[string]string{"m:prog": "sha256:aa", "m:lib": "sha256:bb"} {
if _, err := b.Reserved.Built(ctx, "m", v.Version, ref, d, nil); err != nil {
t.Fatal(err)
}
}
if queued, err := b.Ended(ctx, "m", v.Version); err != nil || len(queued) != 1 || len(b.sent) != 1 || b.sent[0].Name != "m/prog" || b.sent[0].Digest != "sha256:aa" {
t.Fatalf("the ended build queued %v, sent %+v (%v), want the bundle at the oci holder", queued, b.sent, err)
}
})
t.Run("returned acts at the seat's holder, started does not", func(t *testing.T) {
b := newRepublishBed(t, 50)
b.reserve(t, "m")
@@ -422,7 +422,7 @@ func RequestDropTools(s check.StateDrop, acts views.HandActs) []stdio.Tool {
key, _ := args["check"].(string)
why, _ := args["why"].(string)
return check.DropRequests(s, key, why, func(what string) error {
return acts.RecordHandAct(views.HandAct{What: what, Why: why, Verb: check.RequestsDropTool})
return acts.RecordHandAct(views.HandAct{What: what, Why: why, Cause: "requests-drop", Verb: check.RequestsDropTool})
})
}}}
}
@@ -555,8 +555,8 @@ func TestOnlyThePackageWithoutAKeptVerdictRuns(t *testing.T) { // 1288/AC1
t.Fatalf("verdict %+v, asks %+v: want a pass of three asks, the first naming no need", out.Verdict, asks)
}
for _, a := range asks { // ADR 0584: every check ask names the agreed toolchain, so a node off it declines
if a.Check.Toolchain == nil || a.Check.Toolchain.Version != "go1.27.1" || a.Check.Toolchain.Recipe != "7" {
t.Errorf("a check ask names toolchain %+v, want go1.27.1 recipe 7", a.Check.Toolchain)
if a.Check.Toolchain == nil || a.Check.Toolchain.Version != "go1.27" || a.Check.Toolchain.Recipe != "7" {
t.Errorf("a check ask names toolchain %+v, want go1.27 recipe 7", a.Check.Toolchain)
}
}
for _, a := range asks[1:] {
@@ -1191,7 +1191,7 @@ func TestACheckIsOneAskAndAReJudgeGoesOnTheRecheckStream(t *testing.T) {
t.Fatalf("the job = %+v", asks[0].Check)
}
// B2g: the job carries the toolchain the checker agreed (ADR 0584), with its language, so a node on another declines it.
if tc := asks[0].Check.Toolchain; tc == nil || *tc != (buildstep.ToolchainUsed{Language: "go", Version: "go1.27.1", Recipe: "7"}) {
if tc := asks[0].Check.Toolchain; tc == nil || *tc != (buildstep.ToolchainUsed{Language: "go", Version: "go1.27", Recipe: "7"}) {
t.Errorf("the job's toolchain = %+v, want the agreed go toolchain", tc)
}
cs := caseOf(t, pullOf(repoA, 7, head1))
@@ -15,18 +15,20 @@ import (
const Seat = "mesh-pipeline-runner"
const (
VerbRun = "run"
VerbStart = "start"
VerbShow = "show"
VerbList = "list"
VerbStands = "stands"
VerbWhyCarried = "why-carried"
VerbKept = "kept"
VerbReview = "review"
VerbRerun = "rerun"
VerbAccept = "accept"
VerbCancel = "cancel"
VerbAskAgain = "ask-again"
VerbRun = "run"
VerbStart = "start"
VerbShow = "show"
VerbList = "list"
VerbStands = "stands"
VerbWhyCarried = "why-carried"
VerbKept = "kept"
VerbReview = "review"
VerbRerun = "rerun"
VerbAccept = "accept"
VerbCancel = "cancel"
VerbAskAgain = "ask-again"
VerbTestRuns = "test-runs"
VerbTestRunsDrop = "test-runs-drop"
)
const (
@@ -45,7 +47,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string {
return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbAskAgain}
return []string{VerbRun, VerbStart, VerbShow, VerbList, VerbStands, VerbWhyCarried, VerbKept, VerbReview, VerbRerun, VerbAccept, VerbCancel, VerbAskAgain, VerbTestRuns, VerbTestRunsDrop}
}
// Events is every event published on the seat's subject: a new copy each call.
@@ -596,6 +598,26 @@ type RunStarted struct {
V int64 `json:"v"`
}
// SharedAskedBy is the step that asked the run, for a test step reading another's run.
type SharedAskedBy struct {
// its artifact, <module>:<artifact>
Artifact string `json:"artifact"`
// its pipeline's key
Pipeline string `json:"pipeline"`
}
// Shared is a run whose result a step took or waits on, asked once for every step that reads it (design 47 G §729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself.
type Shared struct {
// the id of the run's ask of the build seat
Ask string `json:"ask"`
// the step that asked the run, for a test step reading another's run
AskedBy *SharedAskedBy `json:"asked-by,omitempty"`
// the machine composed, for a composition run
Machine string `json:"machine,omitempty"`
// the run's key: in the assigner's composition-runs, or the runner's test-runs
Run string `json:"run"`
}
// StepEvent is a step's start or end: the step, its status, its outcome, its fingerprint, its test counts, what a carried result was carried from, and for a rerun that it is one and its proved caller; a bypassed step too.
type StepEvent struct {
At *time.Time `json:"at,omitempty"`
@@ -645,7 +667,9 @@ type StepResult struct {
// the step was asked again by the runner's rerun
Rerun bool `json:"rerun,omitempty"`
// the seat that serves the step, whose holder alone may end it (ADR 0666 section 21)
Seat string `json:"seat,omitempty"`
Seat string `json:"seat,omitempty"`
// the run the step read or asked for every step that reads it: a composition step's per machine, as mesh-wants' step-ended names them; a test step's one, with asked-by when another step asked it (design 47 G §726, §729; A-468d, A-468e, A-469t)
Shared []Shared `json:"shared,omitempty"`
Started *time.Time `json:"started,omitempty"`
// the step's status in the module's policy
Status string `json:"status"`
@@ -698,6 +722,56 @@ type TestCounts struct {
Unit int64 `json:"unit,omitempty"`
}
// TestRun is one test run of a candidate, level and tested packages, a key of the runner's state test-runs (design 47 G §725, §728; A-469u).
type TestRun struct {
// the id of the build seat's ask
Ask string `json:"ask,omitempty"`
// the share digest (design 47 G §724)
Digest string `json:"digest"`
Ended *time.Time `json:"ended,omitempty"`
// why the packages are not named: the run's output is gone
Gone string `json:"gone,omitempty"`
// the key, <candidate key>.<level>.<digest>, each part escaped as a key's name
Key string `json:"key"`
Level string `json:"level"`
// the node whose holder took the ask
Node string `json:"node,omitempty"`
Outcome string `json:"outcome,omitempty"`
// the run's output object, by its name in test-runs-objects
Output string `json:"output,omitempty"`
// the packages it ran, read from its output object
Packages []TestRunPackage `json:"packages,omitempty"`
// the steps reading the run
Readers []TestRunReader `json:"readers,omitempty"`
// the outcome's words
Said string `json:"said,omitempty"`
// the build seat asked
Seat string `json:"seat,omitempty"`
Started *time.Time `json:"started,omitempty"`
Tests *TestCounts `json:"tests,omitempty"`
}
// The values of TestRun's level.
const (
TestRunLevelLint = "lint"
TestRunLevelUnit = "unit"
TestRunLevelIntegration = "integration"
)
// TestRunPackage is one package a test run ran, with its digest (design 47 G §724).
type TestRunPackage struct {
Digest string `json:"digest"`
Package string `json:"package"`
}
// TestRunReader is a test step reading a test run (design 47 G §725).
type TestRunReader struct {
// its artifact, <module>:<artifact>
Artifact string `json:"artifact"`
// its pipeline's key
Pipeline string `json:"pipeline"`
}
// ToolchainUsed is the toolchain a build was made with: the language, the installed version and the build module's recipe.
type ToolchainUsed struct {
Language string `json:"language"`
@@ -708,6 +782,8 @@ type ToolchainUsed struct {
// Wait is a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283).
type Wait struct {
Artifact string `json:"artifact,omitempty"`
// the step's own ask, where it has one (design 47 G §731)
Ask string `json:"ask,omitempty"`
// the watched branch, or the branch of a run on request
Branch string `json:"branch"`
// the commit the module's entry was last changed at
@@ -726,8 +802,10 @@ type Wait struct {
// the catalog graph revision of the branch the pipeline runs for
Revision int64 `json:"revision,omitempty"`
// the run on request's id, for a pipeline of a run; such a pipeline delivers nothing
Run string `json:"run,omitempty"`
Since time.Time `json:"since"`
Run string `json:"run,omitempty"`
// the run the step waits on, where the runner knows it: a test step's run key and ask (design 47 G §731, A-469v)
Shared []Shared `json:"shared,omitempty"`
Since time.Time `json:"since"`
// the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9)
Source int64 `json:"source,omitempty"`
// a pipeline step a wait inside a pipeline is in
@@ -977,6 +1055,44 @@ type AskAgainAnswer struct {
Result StepResult `json:"result"`
}
// TestRunsInput is what test-runs takes.
type TestRunsInput struct {
// the candidate's key
Candidate string `json:"candidate,omitempty"`
Digest string `json:"digest,omitempty"`
Level string `json:"level,omitempty"`
Module string `json:"module,omitempty"`
}
// The values of TestRunsInput's level.
const (
TestRunsInputLevelLint = "lint"
TestRunsInputLevelUnit = "unit"
TestRunsInputLevelIntegration = "integration"
)
// TestRunsAnswer is what test-runs answers.
type TestRunsAnswer struct {
Fill Fill `json:"fill"`
Runs []TestRun `json:"runs"`
}
// TestRunsDropInput is what test-runs-drop takes.
type TestRunsDropInput struct {
// the run's key in test-runs
Key string `json:"key"`
// why, written with the caller the runtime proves in the hand-act log
Why string `json:"why"`
}
// TestRunsDropAnswer is what test-runs-drop answers.
type TestRunsDropAnswer struct {
// the readers it ended
Ended []TestRunReader `json:"ended,omitempty"`
// the entry deleted
Run TestRun `json:"run"`
}
// PipelineStartedEvent is the body of the event pipeline-started.
type PipelineStartedEvent = PipelineStarted
@@ -37,6 +37,12 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v WaitHeldEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.wait-held\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "the wait ended answered", subject: "mesh.seat.mesh-pipeline-runner.event.wait-ended.nats.main.0123456789abcdef0123456789abcdef01234567", given: "{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"how\":\"answered\",\"module\":\"nats\",\"on\":\"person\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"since\":\"2026-11-10T10:10:00Z\",\"source\":3589,\"step\":\"approval\",\"v\":1,\"who\":\"the operator\"}", wire: "{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"how\":\"answered\",\"module\":\"nats\",\"on\":\"person\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"since\":\"2026-11-10T10:10:00Z\",\"source\":3589,\"step\":\"approval\",\"v\":1,\"who\":\"the operator\"}",
decode: func(b []byte) (any, error) { var v WaitEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.wait-ended\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "C1/AC2 a test step started reading a test run, naming its run and ask", subject: "mesh.seat.mesh-pipeline-runner.event.step-started.nats.main.0123456789abcdef0123456789abcdef01234567", given: "{\"at\":\"2026-11-17T10:00:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"artifact\":\"nats:tools\",\"fingerprint\":\"fp-0123\",\"outcome\":\"running\",\"seat\":\"node-build-go\",\"shared\":[{\"ask\":\"t1a2\",\"run\":\"main.12.unit.d1\"}],\"status\":\"required\",\"step\":\"unit\"},\"revision\":12,\"source\":3589,\"v\":1}", wire: "{\"at\":\"2026-11-17T10:00:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"artifact\":\"nats:tools\",\"fingerprint\":\"fp-0123\",\"outcome\":\"running\",\"seat\":\"node-build-go\",\"shared\":[{\"ask\":\"t1a2\",\"run\":\"main.12.unit.d1\"}],\"status\":\"required\",\"step\":\"unit\"},\"revision\":12,\"source\":3589,\"v\":1}",
decode: func(b []byte) (any, error) { var v StepStartedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.step-started\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "C1/AC2 a test step ended with the run another step asked", subject: "mesh.seat.mesh-pipeline-runner.event.step-ended.nats.main.0123456789abcdef0123456789abcdef01234567", given: "{\"at\":\"2026-11-17T10:05:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"artifact\":\"nats:tools\",\"fingerprint\":\"fp-0123\",\"node\":\"anchor\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"shared\":[{\"ask\":\"t1a2\",\"asked-by\":{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},\"run\":\"main.12.unit.d1\"}],\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},\"revision\":12,\"source\":3589,\"v\":1}", wire: "{\"at\":\"2026-11-17T10:05:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"artifact\":\"nats:tools\",\"fingerprint\":\"fp-0123\",\"node\":\"anchor\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"shared\":[{\"ask\":\"t1a2\",\"asked-by\":{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},\"run\":\"main.12.unit.d1\"}],\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},\"revision\":12,\"source\":3589,\"v\":1}",
decode: func(b []byte) (any, error) { var v StepEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.step-ended\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "C1 a composition step ended with mesh-wants' runs as they came", subject: "mesh.seat.mesh-pipeline-runner.event.step-ended.nats.main.0123456789abcdef0123456789abcdef01234567", given: "{\"at\":\"2026-11-17T10:05:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"fingerprint\":\"fp-0123\",\"node\":\"laptop\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"}],\"status\":\"required\",\"step\":\"composition\"},\"revision\":12,\"source\":3589,\"v\":1}", wire: "{\"at\":\"2026-11-17T10:05:00Z\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"result\":{\"fingerprint\":\"fp-0123\",\"node\":\"laptop\",\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"}],\"status\":\"required\",\"step\":\"composition\"},\"revision\":12,\"source\":3589,\"v\":1}",
decode: func(b []byte) (any, error) { var v StepEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-pipeline-runner\\.event\\.step-ended\\.[^.*> ]+\\.[^.*> ]+\\.[^.*> ]+$")},
{name: "a pipeline kept", subject: "", given: "{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"passed\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.11.nats\",\"fingerprint\":\"fp-0123\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}", wire: "{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"end-count\":1,\"ended\":\"2026-11-10T10:20:00Z\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"outcome\":\"passed\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"source\":3589,\"started\":\"2026-11-10T10:00:00Z\",\"steps\":[{\"artifact\":\"nats:program\",\"ask\":\"m2kx9a1b3c4d5e\",\"fingerprint\":\"fp-0123\",\"node\":\"anchor\",\"outcome\":\"passed\",\"seat\":\"node-build-go\",\"status\":\"required\",\"step\":\"lint\"},{\"artifact\":\"nats:program\",\"carried-from\":\"main.11.nats\",\"fingerprint\":\"fp-0123\",\"outcome\":\"carried\",\"status\":\"required\",\"step\":\"unit\",\"tests\":{\"unit\":42}},{\"artifact\":\"nats:program\",\"fingerprint\":\"fp-0123\",\"node\":\"home-server\",\"outcome\":\"failed\",\"seat\":\"node-build-go\",\"status\":\"optional\",\"step\":\"integration\",\"tests\":{\"integration\":3},\"why\":\"TestTheBusWhole failed\"},{\"outcome\":\"passed\",\"seat\":\"mesh-wants\",\"status\":\"required\",\"step\":\"composition\"},{\"outcome\":\"bypassed\",\"status\":\"bypass\",\"step\":\"review\"},{\"outcome\":\"passed\",\"status\":\"required\",\"step\":\"approval\"}],\"v\":1}",
decode: func(b []byte) (any, error) {
var v PipelinesByBranchRevisionModuleValue
@@ -67,6 +73,12 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v StandsAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "kept answers by digest", subject: "", given: "{\"fill\":{\"bytes\":300,\"cap\":134217728,\"store\":\"kept\"},\"kept\":[{\"digest\":\"sha256:aa\",\"level\":\"unit\",\"package\":\"git.novox.be/novox/nats\",\"pipeline\":\"main.12.nats\"}]}", wire: "{\"fill\":{\"bytes\":300,\"cap\":134217728,\"store\":\"kept\"},\"kept\":[{\"digest\":\"sha256:aa\",\"level\":\"unit\",\"package\":\"git.novox.be/novox/nats\",\"pipeline\":\"main.12.nats\"}]}",
decode: func(b []byte) (any, error) { var v KeptAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1/AC3 stands answers a waiting test step with its ask and the run it waits on", subject: "", given: "{\"stands\":[{\"artifact\":\"nats:tools\",\"ask\":\"t1a2\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"on\":\"build node\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"shared\":[{\"ask\":\"t1a2\",\"run\":\"main.12.unit.d1\"}],\"since\":\"2026-11-17T10:00:00Z\",\"source\":3589,\"step\":\"unit\",\"v\":1}]}", wire: "{\"stands\":[{\"artifact\":\"nats:tools\",\"ask\":\"t1a2\",\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"module\":\"nats\",\"on\":\"build node\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"shared\":[{\"ask\":\"t1a2\",\"run\":\"main.12.unit.d1\"}],\"since\":\"2026-11-17T10:00:00Z\",\"source\":3589,\"step\":\"unit\",\"v\":1}]}",
decode: func(b []byte) (any, error) { var v StandsAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1 test-runs answers each run with its packages, readers and the fill", subject: "", given: "{\"fill\":{\"bytes\":2048,\"cap\":16777216,\"store\":\"test-runs\"},\"runs\":[{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"packages\":[{\"digest\":\"sha256:aa\",\"package\":\"git.novox.be/novox/nats/internal/x\"}],\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}},{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"gone\":\"the run's output is gone\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}}]}", wire: "{\"fill\":{\"bytes\":2048,\"cap\":16777216,\"store\":\"test-runs\"},\"runs\":[{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"packages\":[{\"digest\":\"sha256:aa\",\"package\":\"git.novox.be/novox/nats/internal/x\"}],\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}},{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"gone\":\"the run's output is gone\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}}]}",
decode: func(b []byte) (any, error) { var v TestRunsAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1 test-runs-drop answers the run it deleted and the readers it ended", subject: "", given: "{\"ended\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"run\":{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"packages\":[{\"digest\":\"sha256:aa\",\"package\":\"git.novox.be/novox/nats/internal/x\"}],\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}}}", wire: "{\"ended\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"run\":{\"ask\":\"t1a2\",\"digest\":\"d1\",\"ended\":\"2026-11-17T10:05:00Z\",\"key\":\"main.12.unit.d1\",\"level\":\"unit\",\"node\":\"anchor\",\"outcome\":\"failed\",\"output\":\"t1a2\",\"packages\":[{\"digest\":\"sha256:aa\",\"package\":\"git.novox.be/novox/nats/internal/x\"}],\"readers\":[{\"artifact\":\"nats:program\",\"pipeline\":\"main.12.nats\"},{\"artifact\":\"nats:tools\",\"pipeline\":\"main.12.nats\"}],\"said\":\"1 test failed\",\"seat\":\"node-build-go\",\"started\":\"2026-11-17T10:00:00Z\",\"tests\":{\"unit\":42}}}",
decode: func(b []byte) (any, error) { var v TestRunsDropAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
}
for _, c := range cases {
v, err := c.decode([]byte(c.given))
@@ -15,12 +15,14 @@ import (
const Seat = "mesh-wants"
const (
VerbUpgrade = "upgrade"
VerbBus = "bus"
VerbRevision = "revision"
VerbEntry = "entry"
VerbDifference = "difference"
VerbInput = "input"
VerbUpgrade = "upgrade"
VerbBus = "bus"
VerbRevision = "revision"
VerbEntry = "entry"
VerbDifference = "difference"
VerbInput = "input"
VerbCompositions = "compositions"
VerbCompositionsDrop = "compositions-drop"
)
const (
@@ -44,7 +46,7 @@ const (
// Verbs is every verb the seat serves, in its definition's order: a new copy each call.
func Verbs() []string {
return []string{VerbUpgrade, VerbBus, VerbRevision, VerbEntry, VerbDifference, VerbInput}
return []string{VerbUpgrade, VerbBus, VerbRevision, VerbEntry, VerbDifference, VerbInput, VerbCompositions, VerbCompositionsDrop}
}
// Events is every event published on the seat's subject: a new copy each call.
@@ -125,7 +127,6 @@ const (
StateDifferences = "differences"
StateCompositions = "compositions"
StateAssignmentRevisions = "assignment-revisions"
StateInputs = "inputs"
)
// WantsByWantNodeModuleKey is the key of one value of the state wants under the form "want.<node>.<module>".
@@ -741,42 +742,6 @@ func DecodeAssignmentRevisionsByEntryDigestValue(raw []byte) (AssignmentRevision
return v, nil
}
// InputsByTimeDigestKey is the key of one value of the state inputs under the form "<time>.<digest>".
func InputsByTimeDigestKey(time, digest string) string { return time + "." + digest }
// ParseInputsByTimeDigestKey reads the parts of a key of the form "<time>.<digest>", in the order of the form; the last result is false for a key of another form.
func ParseInputsByTimeDigestKey(key string) (string, string, bool) {
rest := key
v0, after0, found0 := strings.Cut(rest, ".")
if !found0 || v0 == "" {
return "", "", false
}
rest = after0
if rest == "" {
return "", "", false
}
v2 := rest
rest = ""
if rest != "" {
return "", "", false
}
return v0, v2, true
}
// DecodeInputsByTimeDigestValue reads one value of the state inputs under the form "<time>.<digest>", refusing a field InputsByTimeDigestValue does not know and any text after the value.
func DecodeInputsByTimeDigestValue(raw []byte) (InputsByTimeDigestValue, error) {
var v InputsByTimeDigestValue
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&v); err != nil {
return v, err
}
if dec.More() {
return v, errors.New("text after the value")
}
return v, nil
}
// Ask is an ask the assigner puts to a person: the router's ask whole, as the assigner composes it, and how many times it was written at its key.
type Ask struct {
// the seat operator-channel's ask, whole
@@ -867,6 +832,79 @@ type Change struct {
To string `json:"to,omitempty"`
}
// CompositionReader is a composition step reading a composition run: its step, as the runner asked it, with its ask's id (design 47 G §719, A-468).
type CompositionReader struct {
// the watched branch, or the branch of a run on request
Branch string `json:"branch"`
// for a rerun, the caller the runtime proved
By string `json:"by,omitempty"`
// the commit the step runs at
Commit string `json:"commit"`
// when the reader's step said its end; carried into a rerun's run, so a take-up tells a reader that passed from one that joined later (A-470m)
Ended *time.Time `json:"ended,omitempty"`
// the module's build-input fingerprint at the commit
Fingerprint string `json:"fingerprint"`
// the step's ask's id
ID string `json:"id"`
Module string `json:"module"`
// the runner's pipeline key, <branch>.<revision>.<module>, a branch escaped as a key's name is
Pipeline string `json:"pipeline"`
// the step is asked again by the runner's rerun
Rerun bool `json:"rerun,omitempty"`
// the catalog graph revision of the branch the pipeline runs for
Revision int64 `json:"revision,omitempty"`
// the run on request's id: its composition writes no want (ADR 0666 section 18)
Run string `json:"run,omitempty"`
// the source revision of the watched branch; absent for a run on request
Source int64 `json:"source,omitempty"`
Step string `json:"step"`
}
// The values of CompositionReader's step.
const (
CompositionReaderStepComposition = "composition"
)
// CompositionRunProgram is the composer program, chosen once when the run started.
type CompositionRunProgram struct {
Commit string `json:"commit,omitempty"`
Directory string `json:"directory,omitempty"`
// the node-engine fingerprint it was chosen from
Fingerprint string `json:"fingerprint,omitempty"`
Repository string `json:"repository,omitempty"`
}
// CompositionRun is one composition run of a candidate and a machine, a key of the assigner's state composition-runs (design 47 G §719, §728; A-469u).
type CompositionRun struct {
// the id of the build seat's run ask
Ask string `json:"ask,omitempty"`
// the candidate's key: <branch>.<revision>, or run-<id> for a run on request
Candidate string `json:"candidate"`
Ended *time.Time `json:"ended,omitempty"`
// the digest of the kept request
Input string `json:"input,omitempty"`
// the key, <candidate key>.<machine>, each part escaped as a key's name
Key string `json:"key"`
Machine string `json:"machine"`
Outcome string `json:"outcome,omitempty"`
// the composer program, chosen once when the run started
Program *CompositionRunProgram `json:"program,omitempty"`
// the composition steps reading the run
Readers []CompositionReader `json:"readers,omitempty"`
// the outcome's words
Said string `json:"said,omitempty"`
Started *time.Time `json:"started,omitempty"`
// that machine's composer verdict (design 47 G §721)
Verdict map[string]any `json:"verdict,omitempty"`
}
// The values of CompositionRun's outcome.
const (
CompositionRunOutcomePassed = "passed"
CompositionRunOutcomeFailed = "failed"
CompositionRunOutcomeCouldNotRun = "could not run"
)
// DifferenceRefusedItem is a type of the contract.
type DifferenceRefusedItem struct {
Module string `json:"module"`
@@ -944,6 +982,26 @@ type NodeEntryRef struct {
Node string `json:"node"`
}
// SharedAskedBy is the step that asked the run, for a test step reading another's run.
type SharedAskedBy struct {
// its artifact, <module>:<artifact>
Artifact string `json:"artifact"`
// its pipeline's key
Pipeline string `json:"pipeline"`
}
// Shared is a run whose result a step took or waits on, asked once for every step that reads it (design 47 G §729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself.
type Shared struct {
// the id of the run's ask of the build seat
Ask string `json:"ask"`
// the step that asked the run, for a test step reading another's run
AskedBy *SharedAskedBy `json:"asked-by,omitempty"`
// the machine composed, for a composition run
Machine string `json:"machine,omitempty"`
// the run's key: in the assigner's composition-runs, or the runner's test-runs
Run string `json:"run"`
}
// StepEnded is the composition step's end, said by the assigner on the node it runs on (ADR 0666 sections 7, 21): the runner takes it only from that holder's account, naming the ask, the commit and the fingerprint it was asked at.
type StepEnded struct {
// for a rerun, the caller the runtime proved
@@ -961,7 +1019,9 @@ type StepEnded struct {
Pipeline string `json:"pipeline"`
Rerun bool `json:"rerun,omitempty"`
// why it failed or could not run: the two modules owning one path, the part not heard
Said string `json:"said,omitempty"`
Said string `json:"said,omitempty"`
// per machine composed, the composition run the step read, its ask and so where its verdict was taken (design 47 G §723, §729)
Shared []Shared `json:"shared,omitempty"`
Started *time.Time `json:"started,omitempty"`
Step string `json:"step"`
// the layout version
@@ -1151,6 +1211,36 @@ type InputAnswer struct {
Digest string `json:"digest"`
}
// CompositionsInput is what compositions takes.
type CompositionsInput struct {
// the candidate's key
Candidate string `json:"candidate,omitempty"`
Machine string `json:"machine,omitempty"`
}
// CompositionsAnswer is what compositions answers.
type CompositionsAnswer struct {
Fill Fill `json:"fill"`
Runs []CompositionRun `json:"runs"`
}
// CompositionsDropInput is what compositions-drop takes.
type CompositionsDropInput struct {
// the candidate's key
Candidate string `json:"candidate"`
Machine string `json:"machine"`
// why, written with the caller the runtime proves in the hand-act log
Why string `json:"why"`
}
// CompositionsDropAnswer is what compositions-drop answers.
type CompositionsDropAnswer struct {
// the readers it ended
Ended []CompositionReader `json:"ended,omitempty"`
// the entry deleted
Run CompositionRun `json:"run"`
}
// StepAccept is the body of step.
type StepAccept = StepRun
@@ -1234,6 +1324,8 @@ type StepProgressEvent struct {
Asker string `json:"asker"`
At time.Time `json:"at"`
Phase string `json:"phase"`
// the composition runs the step waits on, by machine, each with its ask (design 47 G §723, §729)
Shared []Shared `json:"shared,omitempty"`
}
// The values of StepProgressEvent's phase.
@@ -1355,6 +1447,3 @@ type AssignmentRevisionsByCandidateGroupHeadsValue = AssignmentRevision
// AssignmentRevisionsByEntryDigestValue is one value of the state assignment-revisions under the key entry.<digest>.
type AssignmentRevisionsByEntryDigestValue = AssignmentEntry
// InputsByTimeDigestValue is one value of the state inputs under the key <time>.<digest>.
type InputsByTimeDigestValue = string
@@ -59,6 +59,10 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v WaitHeldEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wait-held$")},
{name: "an outside wait ended by the operator's Go on", subject: "mesh.seat.mesh-wants.event.wait-ended", given: "{\"fingerprint\":\"src2:aaaa\",\"how\":\"go-on\",\"module\":\"nats\",\"node\":\"anchor\",\"on\":\"silent:home-server\",\"owner\":\"mesh-delivery-assigner\",\"since\":\"2026-11-11T10:00:00Z\",\"step\":\"outside\"}", wire: "{\"fingerprint\":\"src2:aaaa\",\"how\":\"go-on\",\"module\":\"nats\",\"node\":\"anchor\",\"on\":\"silent:home-server\",\"owner\":\"mesh-delivery-assigner\",\"since\":\"2026-11-11T10:00:00Z\",\"step\":\"outside\"}",
decode: func(b []byte) (any, error) { var v WaitEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.wait-ended$")},
{name: "C1/AC1 a reading composition step ended, naming per machine the run it read and its ask", subject: "mesh.seat.mesh-wants.event.step-ended.laptop", given: "{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"node\":\"laptop\",\"outcome\":\"passed\",\"pipeline\":\"main.12.nats\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"},{\"ask\":\"r4d5e6\",\"machine\":\"home-server\",\"run\":\"main.12.home-server\"}],\"step\":\"composition\",\"v\":1,\"warnings\":[\"anchor holds xclip: xclip and demo both own /etc/x\"]}", wire: "{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"node\":\"laptop\",\"outcome\":\"passed\",\"pipeline\":\"main.12.nats\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"},{\"ask\":\"r4d5e6\",\"machine\":\"home-server\",\"run\":\"main.12.home-server\"}],\"step\":\"composition\",\"v\":1,\"warnings\":[\"anchor holds xclip: xclip and demo both own /etc/x\"]}",
decode: func(b []byte) (any, error) { var v StepEndedEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.step-ended\\.[^.*> ]+$")},
{name: "C1/AC1 a composition step's progress, naming the runs it waits on", subject: "mesh.seat.mesh-wants.event.step-progress.anchor", given: "{\"ask\":\"a1b2\",\"asker\":\"assigner\",\"at\":\"2026-11-17T10:01:00Z\",\"phase\":\"run\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"}]}", wire: "{\"ask\":\"a1b2\",\"asker\":\"assigner\",\"at\":\"2026-11-17T10:01:00Z\",\"phase\":\"run\",\"shared\":[{\"ask\":\"r1a2b3\",\"machine\":\"anchor\",\"run\":\"main.12.anchor\"}]}",
decode: func(b []byte) (any, error) { var v StepProgressEvent; err := json.Unmarshal(b, &v); return v, err }, subjectIs: regexp.MustCompile("^mesh\\.seat\\.mesh-wants\\.event\\.step-progress\\.[^.*> ]+$")},
{name: "a want", subject: "", given: "{\"fp\":\"ab12\",\"rule\":3,\"v\":1}", wire: "{\"fp\":\"ab12\",\"rule\":3,\"v\":1}",
decode: func(b []byte) (any, error) {
var v WantsByWantNodeModuleValue
@@ -141,6 +145,12 @@ func TestFixtures(t *testing.T) {
decode: func(b []byte) (any, error) { var v InputInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "input answers how many chunks carry the exact bytes", subject: "", given: "{\"chunks\":3,\"digest\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"}", wire: "{\"chunks\":3,\"digest\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"}",
decode: func(b []byte) (any, error) { var v InputAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1 compositions answers each run with its readers, verdict and the fill", subject: "", given: "{\"fill\":{\"bytes\":4096,\"cap\":16777216,\"store\":\"composition-runs\"},\"runs\":[{\"ask\":\"r1a2b3\",\"candidate\":\"main.12\",\"ended\":\"2026-11-17T10:03:00Z\",\"input\":\"sha256:0123\",\"key\":\"main.12.anchor\",\"machine\":\"anchor\",\"outcome\":\"passed\",\"program\":{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"directory\":\"cmd/mesh-compose\",\"fingerprint\":\"src2:aaaa\",\"repository\":\"novox/mesh-host\"},\"readers\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"started\":\"2026-11-17T10:00:00Z\",\"verdict\":{\"blamed\":[],\"composes\":true}}]}", wire: "{\"fill\":{\"bytes\":4096,\"cap\":16777216,\"store\":\"composition-runs\"},\"runs\":[{\"ask\":\"r1a2b3\",\"candidate\":\"main.12\",\"ended\":\"2026-11-17T10:03:00Z\",\"input\":\"sha256:0123\",\"key\":\"main.12.anchor\",\"machine\":\"anchor\",\"outcome\":\"passed\",\"program\":{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"directory\":\"cmd/mesh-compose\",\"fingerprint\":\"src2:aaaa\",\"repository\":\"novox/mesh-host\"},\"readers\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"started\":\"2026-11-17T10:00:00Z\",\"verdict\":{\"blamed\":[],\"composes\":true}}]}",
decode: func(b []byte) (any, error) { var v CompositionsAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1 compositions-drop takes a candidate, a machine and why", subject: "", given: "{\"candidate\":\"main.12\",\"machine\":\"anchor\",\"why\":\"its ask never came back\"}", wire: "{\"candidate\":\"main.12\",\"machine\":\"anchor\",\"why\":\"its ask never came back\"}",
decode: func(b []byte) (any, error) { var v CompositionsDropInput; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
{name: "C1 compositions-drop answers the run it deleted and the readers it ended", subject: "", given: "{\"ended\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"run\":{\"ask\":\"r1a2b3\",\"candidate\":\"main.12\",\"ended\":\"2026-11-17T10:03:00Z\",\"input\":\"sha256:0123\",\"key\":\"main.12.anchor\",\"machine\":\"anchor\",\"outcome\":\"passed\",\"program\":{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"directory\":\"cmd/mesh-compose\",\"fingerprint\":\"src2:aaaa\",\"repository\":\"novox/mesh-host\"},\"readers\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"started\":\"2026-11-17T10:00:00Z\",\"verdict\":{\"blamed\":[],\"composes\":true}}}", wire: "{\"ended\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"run\":{\"ask\":\"r1a2b3\",\"candidate\":\"main.12\",\"ended\":\"2026-11-17T10:03:00Z\",\"input\":\"sha256:0123\",\"key\":\"main.12.anchor\",\"machine\":\"anchor\",\"outcome\":\"passed\",\"program\":{\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"directory\":\"cmd/mesh-compose\",\"fingerprint\":\"src2:aaaa\",\"repository\":\"novox/mesh-host\"},\"readers\":[{\"branch\":\"main\",\"commit\":\"0123456789abcdef0123456789abcdef01234567\",\"fingerprint\":\"fp-0123\",\"id\":\"m2kx9a1b3c4d6c\",\"module\":\"nats\",\"pipeline\":\"main.12.nats\",\"revision\":12,\"step\":\"composition\"}],\"started\":\"2026-11-17T10:00:00Z\",\"verdict\":{\"blamed\":[],\"composes\":true}}}",
decode: func(b []byte) (any, error) { var v CompositionsDropAnswer; err := json.Unmarshal(b, &v); return v, err }, subjectIs: nil},
}
for _, c := range cases {
v, err := c.decode([]byte(c.given))
@@ -313,5 +313,104 @@
"how": "answered",
"fingerprint": "fp-0123"
}
},
{
"name": "C1/AC2 a test step started reading a test run, naming its run and ask",
"of": "event:step-started",
"subject": "mesh.seat.mesh-pipeline-runner.event.step-started.nats.main.0123456789abcdef0123456789abcdef01234567",
"given": {
"pipeline": "main.12.nats",
"module": "nats",
"branch": "main",
"revision": 12,
"source": 3589,
"commit": "0123456789abcdef0123456789abcdef01234567",
"v": 1,
"fingerprint": "fp-0123",
"result": {
"step": "unit",
"artifact": "nats:tools",
"status": "required",
"outcome": "running",
"fingerprint": "fp-0123",
"seat": "node-build-go",
"shared": [
{
"run": "main.12.unit.d1",
"ask": "t1a2"
}
]
},
"at": "2026-11-17T10:00:00Z"
}
},
{
"name": "C1/AC2 a test step ended with the run another step asked",
"of": "event:step-ended",
"subject": "mesh.seat.mesh-pipeline-runner.event.step-ended.nats.main.0123456789abcdef0123456789abcdef01234567",
"given": {
"pipeline": "main.12.nats",
"module": "nats",
"branch": "main",
"revision": 12,
"source": 3589,
"commit": "0123456789abcdef0123456789abcdef01234567",
"v": 1,
"fingerprint": "fp-0123",
"result": {
"step": "unit",
"artifact": "nats:tools",
"status": "required",
"outcome": "failed",
"fingerprint": "fp-0123",
"seat": "node-build-go",
"node": "anchor",
"tests": {
"unit": 42
},
"shared": [
{
"run": "main.12.unit.d1",
"ask": "t1a2",
"asked-by": {
"pipeline": "main.12.nats",
"artifact": "nats:program"
}
}
]
},
"at": "2026-11-17T10:05:00Z"
}
},
{
"name": "C1 a composition step ended with mesh-wants' runs as they came",
"of": "event:step-ended",
"subject": "mesh.seat.mesh-pipeline-runner.event.step-ended.nats.main.0123456789abcdef0123456789abcdef01234567",
"given": {
"pipeline": "main.12.nats",
"module": "nats",
"branch": "main",
"revision": 12,
"source": 3589,
"commit": "0123456789abcdef0123456789abcdef01234567",
"v": 1,
"fingerprint": "fp-0123",
"result": {
"step": "composition",
"status": "required",
"outcome": "passed",
"fingerprint": "fp-0123",
"seat": "mesh-wants",
"node": "laptop",
"shared": [
{
"machine": "anchor",
"run": "main.12.anchor",
"ask": "r1a2b3"
}
]
},
"at": "2026-11-17T10:05:00Z"
}
}
]
@@ -143,5 +143,154 @@
"cap": 134217728
}
}
},
{
"name": "C1/AC3 stands answers a waiting test step with its ask and the run it waits on",
"of": "verb:stands.answer",
"given": {
"stands": [
{
"pipeline": "main.12.nats",
"module": "nats",
"branch": "main",
"revision": 12,
"source": 3589,
"commit": "0123456789abcdef0123456789abcdef01234567",
"v": 1,
"fingerprint": "fp-0123",
"step": "unit",
"artifact": "nats:tools",
"on": "build node",
"since": "2026-11-17T10:00:00Z",
"ask": "t1a2",
"shared": [
{
"run": "main.12.unit.d1",
"ask": "t1a2"
}
]
}
]
}
},
{
"name": "C1 test-runs answers each run with its packages, readers and the fill",
"of": "verb:test-runs.answer",
"given": {
"runs": [
{
"key": "main.12.unit.d1",
"level": "unit",
"digest": "d1",
"seat": "node-build-go",
"ask": "t1a2",
"node": "anchor",
"started": "2026-11-17T10:00:00Z",
"readers": [
{
"pipeline": "main.12.nats",
"artifact": "nats:program"
},
{
"pipeline": "main.12.nats",
"artifact": "nats:tools"
}
],
"ended": "2026-11-17T10:05:00Z",
"outcome": "failed",
"said": "1 test failed",
"tests": {
"unit": 42
},
"output": "t1a2",
"packages": [
{
"package": "git.novox.be/novox/nats/internal/x",
"digest": "sha256:aa"
}
]
},
{
"key": "main.12.unit.d1",
"level": "unit",
"digest": "d1",
"seat": "node-build-go",
"ask": "t1a2",
"node": "anchor",
"started": "2026-11-17T10:00:00Z",
"readers": [
{
"pipeline": "main.12.nats",
"artifact": "nats:program"
},
{
"pipeline": "main.12.nats",
"artifact": "nats:tools"
}
],
"ended": "2026-11-17T10:05:00Z",
"outcome": "failed",
"said": "1 test failed",
"tests": {
"unit": 42
},
"output": "t1a2",
"gone": "the run's output is gone"
}
],
"fill": {
"store": "test-runs",
"bytes": 2048,
"cap": 16777216
}
}
},
{
"name": "C1 test-runs-drop answers the run it deleted and the readers it ended",
"of": "verb:test-runs-drop.answer",
"given": {
"run": {
"key": "main.12.unit.d1",
"level": "unit",
"digest": "d1",
"seat": "node-build-go",
"ask": "t1a2",
"node": "anchor",
"started": "2026-11-17T10:00:00Z",
"readers": [
{
"pipeline": "main.12.nats",
"artifact": "nats:program"
},
{
"pipeline": "main.12.nats",
"artifact": "nats:tools"
}
],
"ended": "2026-11-17T10:05:00Z",
"outcome": "failed",
"said": "1 test failed",
"tests": {
"unit": 42
},
"output": "t1a2",
"packages": [
{
"package": "git.novox.be/novox/nats/internal/x",
"digest": "sha256:aa"
}
]
},
"ended": [
{
"pipeline": "main.12.nats",
"artifact": "nats:program"
},
{
"pipeline": "main.12.nats",
"artifact": "nats:tools"
}
]
}
}
]
@@ -495,6 +495,93 @@
]
},
"class": "act"
},
{
"name": "test-runs",
"class": "read",
"description": "Each test run the runner keeps (design 47 G §728): its packages read from its output object, ask, node, readers, outcome and counts, and the state's fill against its cap; with candidate, level, digest or module, only those.",
"input": {
"type": "object",
"properties": {
"candidate": {
"type": "string",
"description": "the candidate's key"
},
"level": {
"type": "string",
"enum": [
"lint",
"unit",
"integration"
]
},
"digest": {
"type": "string"
},
"module": {
"type": "string"
}
}
},
"answer": {
"type": "object",
"properties": {
"runs": {
"type": "array",
"items": {
"$ref": "#/defs/TestRun"
}
},
"fill": {
"$ref": "#/defs/Fill"
}
},
"required": [
"runs",
"fill"
]
}
},
{
"name": "test-runs-drop",
"class": "act",
"description": "Drop one test run by hand (design 47 G §728): the act and why are written in the hand-act log first, then the run's ask is dropped, each reader ended could not run naming the drop, and the key deleted with the run's output object.",
"input": {
"type": "object",
"properties": {
"key": {
"type": "string",
"description": "the run's key in test-runs"
},
"why": {
"type": "string",
"description": "why, written with the caller the runtime proves in the hand-act log"
}
},
"required": [
"key",
"why"
]
},
"answer": {
"type": "object",
"properties": {
"run": {
"$ref": "#/defs/TestRun",
"description": "the entry deleted"
},
"ended": {
"type": "array",
"items": {
"$ref": "#/defs/TestRunReader"
},
"description": "the readers it ended"
}
},
"required": [
"run"
]
}
}
],
"events": [
@@ -887,6 +974,13 @@
"ended": {
"type": "string",
"format": "date-time"
},
"shared": {
"type": "array",
"items": {
"$ref": "#/defs/Shared"
},
"description": "the run the step read or asked for every step that reads it: a composition step's per machine, as mesh-wants' step-ended names them; a test step's one, with asked-by when another step asked it (design 47 G §726, §729; A-468d, A-468e, A-469t)"
}
},
"description": "one step of a pipeline: its status in the policy and its outcome (design 47 *Pipeline steps and the policy*)",
@@ -1494,6 +1588,17 @@
"how": {
"type": "string",
"description": "for its end, how it ended: answered, ran, superseded"
},
"ask": {
"type": "string",
"description": "the step's own ask, where it has one (design 47 G §731)"
},
"shared": {
"type": "array",
"items": {
"$ref": "#/defs/Shared"
},
"description": "the run the step waits on, where the runner knows it: a test step's run key and ask (design 47 G §731, A-469v)"
}
},
"description": "a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283)",
@@ -1577,6 +1682,159 @@
"what",
"at"
]
},
"Shared": {
"type": "object",
"description": "a run whose result a step took or waits on, asked once for every step that reads it (design 47 G §729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself",
"properties": {
"machine": {
"type": "string",
"description": "the machine composed, for a composition run"
},
"run": {
"type": "string",
"description": "the run's key: in the assigner's composition-runs, or the runner's test-runs"
},
"ask": {
"type": "string",
"description": "the id of the run's ask of the build seat"
},
"asked-by": {
"type": "object",
"description": "the step that asked the run, for a test step reading another's run",
"properties": {
"pipeline": {
"type": "string",
"description": "its pipeline's key"
},
"artifact": {
"type": "string",
"description": "its artifact, <module>:<artifact>"
}
},
"required": [
"pipeline",
"artifact"
]
}
},
"required": [
"run",
"ask"
]
},
"TestRunReader": {
"type": "object",
"description": "a test step reading a test run (design 47 G §725)",
"properties": {
"pipeline": {
"type": "string",
"description": "its pipeline's key"
},
"artifact": {
"type": "string",
"description": "its artifact, <module>:<artifact>"
}
},
"required": [
"pipeline",
"artifact"
]
},
"TestRunPackage": {
"type": "object",
"description": "one package a test run ran, with its digest (design 47 G §724)",
"properties": {
"package": {
"type": "string"
},
"digest": {
"type": "string"
}
},
"required": [
"package",
"digest"
]
},
"TestRun": {
"type": "object",
"description": "one test run of a candidate, level and tested packages, a key of the runner's state test-runs (design 47 G §725, §728; A-469u)",
"properties": {
"key": {
"type": "string",
"description": "the key, <candidate key>.<level>.<digest>, each part escaped as a key's name"
},
"level": {
"type": "string",
"enum": [
"lint",
"unit",
"integration"
]
},
"digest": {
"type": "string",
"description": "the share digest (design 47 G §724)"
},
"seat": {
"type": "string",
"description": "the build seat asked"
},
"ask": {
"type": "string",
"description": "the id of the build seat's ask"
},
"node": {
"type": "string",
"description": "the node whose holder took the ask"
},
"started": {
"type": "string",
"format": "date-time"
},
"readers": {
"type": "array",
"items": {
"$ref": "#/defs/TestRunReader"
},
"description": "the steps reading the run"
},
"ended": {
"type": "string",
"format": "date-time"
},
"outcome": {
"type": "string"
},
"said": {
"type": "string",
"description": "the outcome's words"
},
"tests": {
"$ref": "#/defs/TestCounts"
},
"output": {
"type": "string",
"description": "the run's output object, by its name in test-runs-objects"
},
"packages": {
"type": "array",
"items": {
"$ref": "#/defs/TestRunPackage"
},
"description": "the packages it ran, read from its output object"
},
"gone": {
"type": "string",
"description": "why the packages are not named: the run's output is gone"
}
},
"required": [
"key",
"level",
"digest"
]
}
}
}
@@ -284,5 +284,54 @@
"since": "2026-11-11T10:00:00Z",
"how": "go-on"
}
},
{
"name": "C1/AC1 a reading composition step ended, naming per machine the run it read and its ask",
"of": "event:step-ended",
"subject": "mesh.seat.mesh-wants.event.step-ended.laptop",
"given": {
"v": 1,
"id": "m2kx9a1b3c4d6c",
"pipeline": "main.12.nats",
"step": "composition",
"module": "nats",
"commit": "0123456789abcdef0123456789abcdef01234567",
"fingerprint": "fp-0123",
"node": "laptop",
"outcome": "passed",
"warnings": [
"anchor holds xclip: xclip and demo both own /etc/x"
],
"shared": [
{
"machine": "anchor",
"run": "main.12.anchor",
"ask": "r1a2b3"
},
{
"machine": "home-server",
"run": "main.12.home-server",
"ask": "r4d5e6"
}
]
}
},
{
"name": "C1/AC1 a composition step's progress, naming the runs it waits on",
"of": "event:step-progress",
"subject": "mesh.seat.mesh-wants.event.step-progress.anchor",
"given": {
"asker": "assigner",
"ask": "a1b2",
"phase": "run",
"at": "2026-11-17T10:01:00Z",
"shared": [
{
"machine": "anchor",
"run": "main.12.anchor",
"ask": "r1a2b3"
}
]
}
}
]
@@ -175,5 +175,109 @@
"chunks": 3,
"digest": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
}
},
{
"name": "C1 compositions answers each run with its readers, verdict and the fill",
"of": "verb:compositions.answer",
"given": {
"runs": [
{
"key": "main.12.anchor",
"candidate": "main.12",
"machine": "anchor",
"program": {
"repository": "novox/mesh-host",
"commit": "0123456789abcdef0123456789abcdef01234567",
"directory": "cmd/mesh-compose",
"fingerprint": "src2:aaaa"
},
"input": "sha256:0123",
"ask": "r1a2b3",
"started": "2026-11-17T10:00:00Z",
"readers": [
{
"id": "m2kx9a1b3c4d6c",
"pipeline": "main.12.nats",
"step": "composition",
"module": "nats",
"branch": "main",
"commit": "0123456789abcdef0123456789abcdef01234567",
"revision": 12,
"fingerprint": "fp-0123"
}
],
"ended": "2026-11-17T10:03:00Z",
"outcome": "passed",
"verdict": {
"composes": true,
"blamed": []
}
}
],
"fill": {
"store": "composition-runs",
"bytes": 4096,
"cap": 16777216
}
}
},
{
"name": "C1 compositions-drop takes a candidate, a machine and why",
"of": "verb:compositions-drop.input",
"given": {
"candidate": "main.12",
"machine": "anchor",
"why": "its ask never came back"
}
},
{
"name": "C1 compositions-drop answers the run it deleted and the readers it ended",
"of": "verb:compositions-drop.answer",
"given": {
"run": {
"key": "main.12.anchor",
"candidate": "main.12",
"machine": "anchor",
"program": {
"repository": "novox/mesh-host",
"commit": "0123456789abcdef0123456789abcdef01234567",
"directory": "cmd/mesh-compose",
"fingerprint": "src2:aaaa"
},
"input": "sha256:0123",
"ask": "r1a2b3",
"started": "2026-11-17T10:00:00Z",
"readers": [
{
"id": "m2kx9a1b3c4d6c",
"pipeline": "main.12.nats",
"step": "composition",
"module": "nats",
"branch": "main",
"commit": "0123456789abcdef0123456789abcdef01234567",
"revision": 12,
"fingerprint": "fp-0123"
}
],
"ended": "2026-11-17T10:03:00Z",
"outcome": "passed",
"verdict": {
"composes": true,
"blamed": []
}
},
"ended": [
{
"id": "m2kx9a1b3c4d6c",
"pipeline": "main.12.nats",
"step": "composition",
"module": "nats",
"branch": "main",
"commit": "0123456789abcdef0123456789abcdef01234567",
"revision": 12,
"fingerprint": "fp-0123"
}
]
}
}
]
@@ -202,6 +202,86 @@
]
},
"chunks": true
},
{
"name": "compositions",
"class": "read",
"description": "Each composition run the assigner keeps (design 47 G \u00a7728): its program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap; with candidate or machine, only those.",
"input": {
"type": "object",
"properties": {
"candidate": {
"type": "string",
"description": "the candidate's key"
},
"machine": {
"type": "string"
}
}
},
"answer": {
"type": "object",
"properties": {
"runs": {
"type": "array",
"items": {
"$ref": "#/defs/CompositionRun"
}
},
"fill": {
"$ref": "#/defs/Fill"
}
},
"required": [
"runs",
"fill"
]
}
},
{
"name": "compositions-drop",
"class": "act",
"description": "Drop one composition run by hand (design 47 G \u00a7728): the act and why are written in the hand-act log first, then the run's ask is dropped, each reader ended could not run naming the drop, and the key deleted.",
"input": {
"type": "object",
"properties": {
"candidate": {
"type": "string",
"description": "the candidate's key"
},
"machine": {
"type": "string"
},
"why": {
"type": "string",
"description": "why, written with the caller the runtime proves in the hand-act log"
}
},
"required": [
"candidate",
"machine",
"why"
]
},
"answer": {
"type": "object",
"properties": {
"run": {
"$ref": "#/defs/CompositionRun",
"description": "the entry deleted"
},
"ended": {
"type": "array",
"items": {
"$ref": "#/defs/CompositionReader"
},
"description": "the readers it ended"
}
},
"required": [
"run"
]
}
}
],
"accepts": [
@@ -470,6 +550,13 @@
"at": {
"type": "string",
"format": "date-time"
},
"shared": {
"type": "array",
"items": {
"$ref": "#/defs/Shared"
},
"description": "the composition runs the step waits on, by machine, each with its ask (design 47 G \u00a7723, \u00a7729)"
}
},
"required": [
@@ -808,22 +895,6 @@
"description": "an entry, written once, removed when no kept revision lists it"
}
]
},
{
"name": "inputs",
"template": "state",
"history": 1,
"description": "a step's input, named by its write time and its sha256; a value over one chunk is held in the state's objects store (design 47 G sections 354, 712)",
"key-forms": [
{
"key": "<time>.<digest>",
"value": {
"type": "string",
"description": "the input's bytes, base64"
},
"description": "one input"
}
]
}
],
"defs": {
@@ -1434,6 +1505,13 @@
"ended": {
"type": "string",
"format": "date-time"
},
"shared": {
"type": "array",
"items": {
"$ref": "#/defs/Shared"
},
"description": "per machine composed, the composition run the step read, its ask and so where its verdict was taken (design 47 G \u00a7723, \u00a7729)"
}
},
"required": [
@@ -1488,6 +1566,195 @@
"required": [
"ask"
]
},
"Shared": {
"type": "object",
"description": "a run whose result a step took or waits on, asked once for every step that reads it (design 47 G \u00a7729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself",
"properties": {
"machine": {
"type": "string",
"description": "the machine composed, for a composition run"
},
"run": {
"type": "string",
"description": "the run's key: in the assigner's composition-runs, or the runner's test-runs"
},
"ask": {
"type": "string",
"description": "the id of the run's ask of the build seat"
},
"asked-by": {
"type": "object",
"description": "the step that asked the run, for a test step reading another's run",
"properties": {
"pipeline": {
"type": "string",
"description": "its pipeline's key"
},
"artifact": {
"type": "string",
"description": "its artifact, <module>:<artifact>"
}
},
"required": [
"pipeline",
"artifact"
]
}
},
"required": [
"run",
"ask"
]
},
"CompositionReader": {
"type": "object",
"description": "a composition step reading a composition run: its step, as the runner asked it, with its ask's id (design 47 G \u00a7719, A-468)",
"properties": {
"id": {
"type": "string",
"description": "the step's ask's id"
},
"pipeline": {
"type": "string",
"description": "the runner's pipeline key, <branch>.<revision>.<module>, a branch escaped as a key's name is"
},
"step": {
"type": "string",
"enum": [
"composition"
]
},
"module": {
"type": "string"
},
"branch": {
"type": "string",
"description": "the watched branch, or the branch of a run on request"
},
"commit": {
"type": "string",
"description": "the commit the step runs at"
},
"revision": {
"type": "integer",
"description": "the catalog graph revision of the branch the pipeline runs for"
},
"source": {
"type": "integer",
"description": "the source revision of the watched branch; absent for a run on request"
},
"fingerprint": {
"type": "string",
"description": "the module's build-input fingerprint at the commit"
},
"run": {
"type": "string",
"description": "the run on request's id: its composition writes no want (ADR 0666 section 18)"
},
"rerun": {
"type": "boolean",
"description": "the step is asked again by the runner's rerun"
},
"by": {
"type": "string",
"description": "for a rerun, the caller the runtime proved"
},
"ended": {
"type": "string",
"format": "date-time",
"description": "when the reader's step said its end; carried into a rerun's run, so a take-up tells a reader that passed from one that joined later (A-470m)"
}
},
"required": [
"id",
"pipeline",
"step",
"module",
"branch",
"commit",
"fingerprint"
]
},
"CompositionRun": {
"type": "object",
"description": "one composition run of a candidate and a machine, a key of the assigner's state composition-runs (design 47 G \u00a7719, \u00a7728; A-469u)",
"properties": {
"key": {
"type": "string",
"description": "the key, <candidate key>.<machine>, each part escaped as a key's name"
},
"candidate": {
"type": "string",
"description": "the candidate's key: <branch>.<revision>, or run-<id> for a run on request"
},
"machine": {
"type": "string"
},
"program": {
"type": "object",
"description": "the composer program, chosen once when the run started",
"properties": {
"repository": {
"type": "string"
},
"commit": {
"type": "string"
},
"directory": {
"type": "string"
},
"fingerprint": {
"type": "string",
"description": "the node-engine fingerprint it was chosen from"
}
}
},
"input": {
"type": "string",
"description": "the digest of the kept request"
},
"ask": {
"type": "string",
"description": "the id of the build seat's run ask"
},
"started": {
"type": "string",
"format": "date-time"
},
"readers": {
"type": "array",
"items": {
"$ref": "#/defs/CompositionReader"
},
"description": "the composition steps reading the run"
},
"ended": {
"type": "string",
"format": "date-time"
},
"outcome": {
"type": "string",
"enum": [
"passed",
"failed",
"could not run"
]
},
"said": {
"type": "string",
"description": "the outcome's words"
},
"verdict": {
"type": "object",
"description": "that machine's composer verdict (design 47 G \u00a7721)"
}
},
"required": [
"key",
"candidate",
"machine"
]
}
},
"acts": [
@@ -17,6 +17,8 @@ export declare const verbs: {
readonly accept: "accept";
readonly cancel: "cancel";
readonly askAgain: "ask-again";
readonly testRuns: "test-runs";
readonly testRunsDrop: "test-runs-drop";
};
export declare const events: {
@@ -316,6 +318,26 @@ export interface RunStarted {
"v": number;
}
/** the step that asked the run, for a test step reading another's run */
export interface SharedAskedBy {
/** its artifact, <module>:<artifact> */
"artifact": string;
/** its pipeline's key */
"pipeline": string;
}
/** a run whose result a step took or waits on, asked once for every step that reads it (design 47 G §729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself */
export interface Shared {
/** the id of the run's ask of the build seat */
"ask": string;
/** the step that asked the run, for a test step reading another's run */
"asked-by"?: SharedAskedBy;
/** the machine composed, for a composition run */
"machine"?: string;
/** the run's key: in the assigner's composition-runs, or the runner's test-runs */
"run": string;
}
/** a step's start or end: the step, its status, its outcome, its fingerprint, its test counts, what a carried result was carried from, and for a rerun that it is one and its proved caller; a bypassed step too */
export interface StepEvent {
"at"?: string;
@@ -366,6 +388,8 @@ export interface StepResult {
"rerun"?: boolean;
/** the seat that serves the step, whose holder alone may end it (ADR 0666 section 21) */
"seat"?: string;
/** the run the step read or asked for every step that reads it: a composition step's per machine, as mesh-wants' step-ended names them; a test step's one, with asked-by when another step asked it (design 47 G §726, §729; A-468d, A-468e, A-469t) */
"shared"?: Shared[];
"started"?: string;
/** the step's status in the module's policy */
"status": "required" | "optional" | "bypass";
@@ -381,6 +405,49 @@ export interface TestCounts {
"unit"?: number;
}
/** one test run of a candidate, level and tested packages, a key of the runner's state test-runs (design 47 G §725, §728; A-469u) */
export interface TestRun {
/** the id of the build seat's ask */
"ask"?: string;
/** the share digest (design 47 G §724) */
"digest": string;
"ended"?: string;
/** why the packages are not named: the run's output is gone */
"gone"?: string;
/** the key, <candidate key>.<level>.<digest>, each part escaped as a key's name */
"key": string;
"level": "lint" | "unit" | "integration";
/** the node whose holder took the ask */
"node"?: string;
"outcome"?: string;
/** the run's output object, by its name in test-runs-objects */
"output"?: string;
/** the packages it ran, read from its output object */
"packages"?: TestRunPackage[];
/** the steps reading the run */
"readers"?: TestRunReader[];
/** the outcome's words */
"said"?: string;
/** the build seat asked */
"seat"?: string;
"started"?: string;
"tests"?: TestCounts;
}
/** one package a test run ran, with its digest (design 47 G §724) */
export interface TestRunPackage {
"digest": string;
"package": string;
}
/** a test step reading a test run (design 47 G §725) */
export interface TestRunReader {
/** its artifact, <module>:<artifact> */
"artifact": string;
/** its pipeline's key */
"pipeline": string;
}
/** the toolchain a build was made with: the language, the installed version and the build module's recipe */
export interface ToolchainUsed {
"language": string;
@@ -391,6 +458,8 @@ export interface ToolchainUsed {
/** a step's wait, held and ended; a wait on a person is said as waiting on that person, never a stall (ADR 0283) */
export interface Wait {
"artifact"?: string;
/** the step's own ask, where it has one (design 47 G §731) */
"ask"?: string;
/** the watched branch, or the branch of a run on request */
"branch": string;
/** the commit the module's entry was last changed at */
@@ -410,6 +479,8 @@ export interface Wait {
"revision"?: number;
/** the run on request's id, for a pipeline of a run; such a pipeline delivers nothing */
"run"?: string;
/** the run the step waits on, where the runner knows it: a test step's run key and ask (design 47 G §731, A-469v) */
"shared"?: Shared[];
"since": string;
/** the source revision of the watched branch; the pipeline is keyed by it (ADR 0666 section 9) */
"source"?: number;
@@ -594,6 +665,37 @@ export interface AskAgainAnswer {
"result": StepResult;
}
/** is what test-runs takes. */
export interface TestRunsInput {
/** the candidate's key */
"candidate"?: string;
"digest"?: string;
"level"?: "lint" | "unit" | "integration";
"module"?: string;
}
/** is what test-runs answers. */
export interface TestRunsAnswer {
"fill": Fill;
"runs": TestRun[];
}
/** is what test-runs-drop takes. */
export interface TestRunsDropInput {
/** the run's key in test-runs */
"key": string;
/** why, written with the caller the runtime proves in the hand-act log */
"why": string;
}
/** is what test-runs-drop answers. */
export interface TestRunsDropAnswer {
/** the readers it ended */
"ended"?: TestRunReader[];
/** the entry deleted */
"run": TestRun;
}
/** is the body of the event pipeline-started. */
export type PipelineStartedEvent = PipelineStarted;
@@ -17,6 +17,8 @@ export const verbs = {
accept: "accept",
cancel: "cancel",
askAgain: "ask-again",
testRuns: "test-runs",
testRunsDrop: "test-runs-drop",
};
export const events = {
+117 -10
View File
@@ -11,6 +11,8 @@ export declare const verbs: {
readonly entry: "entry";
readonly difference: "difference";
readonly input: "input";
readonly compositions: "compositions";
readonly compositionsDrop: "compositions-drop";
};
export declare const accepts: {
@@ -78,7 +80,6 @@ export declare const states: {
readonly differences: "differences";
readonly compositions: "compositions";
readonly assignmentRevisions: "assignment-revisions";
readonly inputs: "inputs";
};
/** The key of one value of the state wants under the form "want.<node>.<module>". */
@@ -171,12 +172,6 @@ export declare function assignmentRevisionsByEntryDigestKey(digest: string): str
/** The parts of a key of the form "entry.<digest>", or undefined for a key of another form. */
export declare function parseAssignmentRevisionsByEntryDigestKey(_key: string): { digest: string } | undefined;
/** The key of one value of the state inputs under the form "<time>.<digest>". */
export declare function inputsByTimeDigestKey(time: string, digest: string): string;
/** The parts of a key of the form "<time>.<digest>", or undefined for a key of another form. */
export declare function parseInputsByTimeDigestKey(_key: string): { time: string; digest: string } | undefined;
/** an ask the assigner puts to a person: the router's ask whole, as the assigner composes it, and how many times it was written at its key */
export interface Ask {
/** the seat operator-channel's ask, whole */
@@ -265,6 +260,67 @@ export interface Change {
"to"?: string;
}
/** a composition step reading a composition run: its step, as the runner asked it, with its ask's id (design 47 G §719, A-468) */
export interface CompositionReader {
/** the watched branch, or the branch of a run on request */
"branch": string;
/** for a rerun, the caller the runtime proved */
"by"?: string;
/** the commit the step runs at */
"commit": string;
/** when the reader's step said its end; carried into a rerun's run, so a take-up tells a reader that passed from one that joined later (A-470m) */
"ended"?: string;
/** the module's build-input fingerprint at the commit */
"fingerprint": string;
/** the step's ask's id */
"id": string;
"module": string;
/** the runner's pipeline key, <branch>.<revision>.<module>, a branch escaped as a key's name is */
"pipeline": string;
/** the step is asked again by the runner's rerun */
"rerun"?: boolean;
/** the catalog graph revision of the branch the pipeline runs for */
"revision"?: number;
/** the run on request's id: its composition writes no want (ADR 0666 section 18) */
"run"?: string;
/** the source revision of the watched branch; absent for a run on request */
"source"?: number;
"step": "composition";
}
/** the composer program, chosen once when the run started */
export interface CompositionRunProgram {
"commit"?: string;
"directory"?: string;
/** the node-engine fingerprint it was chosen from */
"fingerprint"?: string;
"repository"?: string;
}
/** one composition run of a candidate and a machine, a key of the assigner's state composition-runs (design 47 G §719, §728; A-469u) */
export interface CompositionRun {
/** the id of the build seat's run ask */
"ask"?: string;
/** the candidate's key: <branch>.<revision>, or run-<id> for a run on request */
"candidate": string;
"ended"?: string;
/** the digest of the kept request */
"input"?: string;
/** the key, <candidate key>.<machine>, each part escaped as a key's name */
"key": string;
"machine": string;
"outcome"?: "passed" | "failed" | "could not run";
/** the composer program, chosen once when the run started */
"program"?: CompositionRunProgram;
/** the composition steps reading the run */
"readers"?: CompositionReader[];
/** the outcome's words */
"said"?: string;
"started"?: string;
/** that machine's composer verdict (design 47 G §721) */
"verdict"?: Record<string, unknown>;
}
export interface DifferenceRefusedItem {
"module": string;
"why": string;
@@ -340,6 +396,26 @@ export interface NodeEntryRef {
"node": string;
}
/** the step that asked the run, for a test step reading another's run */
export interface SharedAskedBy {
/** its artifact, <module>:<artifact> */
"artifact": string;
/** its pipeline's key */
"pipeline": string;
}
/** a run whose result a step took or waits on, asked once for every step that reads it (design 47 G §729, A-469t): for a composition step one per machine composed, `run` the `composition-runs` key and `ask` its run ask's id; for a test step one with no machine, `run` the `test-runs` key, `ask` its build seat ask's id and `asked-by` the step that asked the run, left out for that step itself */
export interface Shared {
/** the id of the run's ask of the build seat */
"ask": string;
/** the step that asked the run, for a test step reading another's run */
"asked-by"?: SharedAskedBy;
/** the machine composed, for a composition run */
"machine"?: string;
/** the run's key: in the assigner's composition-runs, or the runner's test-runs */
"run": string;
}
/** the composition step's end, said by the assigner on the node it runs on (ADR 0666 sections 7, 21): the runner takes it only from that holder's account, naming the ask, the commit and the fingerprint it was asked at */
export interface StepEnded {
/** for a rerun, the caller the runtime proved */
@@ -358,6 +434,8 @@ export interface StepEnded {
"rerun"?: boolean;
/** why it failed or could not run: the two modules owning one path, the part not heard */
"said"?: string;
/** per machine composed, the composition run the step read, its ask and so where its verdict was taken (design 47 G §723, §729) */
"shared"?: Shared[];
"started"?: string;
"step": "composition";
/** the layout version */
@@ -506,6 +584,36 @@ export interface InputAnswer {
"digest": string;
}
/** is what compositions takes. */
export interface CompositionsInput {
/** the candidate's key */
"candidate"?: string;
"machine"?: string;
}
/** is what compositions answers. */
export interface CompositionsAnswer {
"fill": Fill;
"runs": CompositionRun[];
}
/** is what compositions-drop takes. */
export interface CompositionsDropInput {
/** the candidate's key */
"candidate": string;
"machine": string;
/** why, written with the caller the runtime proves in the hand-act log */
"why": string;
}
/** is what compositions-drop answers. */
export interface CompositionsDropAnswer {
/** the readers it ended */
"ended"?: CompositionReader[];
/** the entry deleted */
"run": CompositionRun;
}
/** is the body of step. */
export type StepAccept = StepRun;
@@ -589,6 +697,8 @@ export interface StepProgressEvent {
"asker": string;
"at": string;
"phase": "fetch" | "build" | "run" | "report";
/** the composition runs the step waits on, by machine, each with its ask (design 47 G §723, §729) */
"shared"?: Shared[];
}
/** is one value of the state wants under the key want.<node>.<module>. */
@@ -692,6 +802,3 @@ export type AssignmentRevisionsByCandidateGroupHeadsValue = AssignmentRevision;
/** is one value of the state assignment-revisions under the key entry.<digest>. */
export type AssignmentRevisionsByEntryDigestValue = AssignmentEntry;
/** is one value of the state inputs under the key <time>.<digest>. */
export type InputsByTimeDigestValue = string;
@@ -11,6 +11,8 @@ export const verbs = {
entry: "entry",
difference: "difference",
input: "input",
compositions: "compositions",
compositionsDrop: "compositions-drop",
};
export const accepts = {
@@ -84,7 +86,6 @@ export const states = {
differences: "differences",
compositions: "compositions",
assignmentRevisions: "assignment-revisions",
inputs: "inputs",
};
/** The key of one value of the state wants under the form "want.<node>.<module>". */
@@ -403,22 +404,3 @@ export function parseAssignmentRevisionsByEntryDigestKey(_key) {
return { digest };
}
/** The key of one value of the state inputs under the form "<time>.<digest>". */
export function inputsByTimeDigestKey(time, digest) {
return time + "." + digest;
}
/** The parts of a key of the form "<time>.<digest>", or undefined for a key of another form. */
export function parseInputsByTimeDigestKey(_key) {
let _rest = _key;
const _at0 = _rest.indexOf(".");
if (_at0 <= 0) return undefined;
const time = _rest.slice(0, _at0);
_rest = _rest.slice(_at0 + 1);
if (_rest === "") return undefined;
const digest = _rest;
_rest = "";
if (_rest !== "") return undefined;
return { time, digest };
}
@@ -1,8 +1,6 @@
package main
import (
"fmt"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -15,59 +13,43 @@ import (
// catalogRead reads the indexer's view of the trunk (its modules and module graph, from the newest catalog graph
// revision) and its manifests state for the seat's verbs.
func catalogRead(modules views.Source, manifests stdio.KeptState, nodes views.Source, seatDefinitions, marks views.Source) index.CatalogRead {
func catalogRead(modules views.Source, manifest func(digest string) (string, error), nodes views.Source, seatDefinitions, marks views.Source) index.CatalogRead {
return index.CatalogRead{
SeatDefinitions: seatDefinitions, Marks: marks,
Modules: func() (map[string]meshcatalog.ModulesByModuleValue, error) {
keys, err := modules.Keys()
values, err := index.SourceValues(modules)
if err != nil {
return nil, err
}
out := map[string]meshcatalog.ModulesByModuleValue{}
for _, k := range keys {
e, ok, err := modules.Get(k)
if err != nil {
return nil, fmt.Errorf("modules.%s: %w", k, err)
}
if !ok {
continue
}
if m, err := meshcatalogclient.DecodeModule(e.Value); err == nil {
for k, v := range values {
if m, err := meshcatalogclient.DecodeModule(v); err == nil {
out[k] = m
}
}
return out, nil
},
Manifest: func(digest string) (string, error) {
key, err := meshcatalogclient.ManifestsKey(digest)
Module: func(name string) (meshcatalog.ModulesByModuleValue, bool, error) {
key, err := meshcatalogclient.ModulesKey(name)
if err != nil {
return "", err
return meshcatalog.ModulesByModuleValue{}, false, nil // a name no key holds is a module the trunk holds none of
}
e, err := manifests.Get(key)
if err != nil {
return "", err
e, ok, err := modules.Get(key)
if err != nil || !ok {
return meshcatalog.ModulesByModuleValue{}, false, err
}
if e == nil {
return "", fmt.Errorf("the manifests state holds no %s", digest)
}
kept, err := meshcatalogclient.DecodeManifest(e.Value)
return kept.Manifest, err
m, err := meshcatalogclient.DecodeModule(e.Value)
return m, err == nil, nil
},
Manifest: manifest,
Graph: func() (graph.Graph, error) {
keys, err := nodes.Keys()
values, err := index.SourceValues(nodes)
if err != nil {
return nil, err
}
g := graph.Graph{}
for _, k := range keys {
e, ok, err := nodes.Get(k)
if err != nil {
return nil, fmt.Errorf("graph.%s: %w", k, err)
}
if !ok {
continue
}
if n, err := graph.Decode(e.Value); err == nil {
for k, v := range values {
if n, err := graph.Decode(v); err == nil {
g[k] = n
}
}
@@ -77,7 +59,27 @@ func catalogRead(modules views.Source, manifests stdio.KeptState, nodes views.So
}
// catalogTools are the seat's verbs over the indexer's own states, served beside the indexer's own tools, since the indexer's manifest claims the seat (ADR 0604 §2).
func catalogTools(state func(string) views.Source) []stdio.Tool {
return index.CatalogTools(catalogRead(state("modules"), stdio.State("manifests"), state(graph.State),
keptSource(index.SeatDefinitionsState), keptSource(views.MarksState)))
// Every state a verb reads whole is answered from memory (the view of the trunk held, or a watched copy), so a call costs no
// round trip per key: four node-engines ask every module at once.
func catalogTools(state func(string) views.Source, kept *index.KeptTrunk, watched *stdio.WatchedStates) []stdio.Tool {
return index.CatalogTools(catalogRead(state("modules"), kept.Manifest, state(graph.State),
watchedSource{watched, index.SeatDefinitionsState}, keptSource(views.MarksState)))
}
// watchedSource is a state as its watched copy holds it (stdio.WatchedStates), as a Source; it offers every value in one read.
type watchedSource struct {
w *stdio.WatchedStates
state string
}
func (s watchedSource) Keys() ([]string, error) { return s.w.Keys(s.state) }
func (s watchedSource) Get(key string) (*views.Entry, bool, error) {
v, ok, err := s.w.Get(s.state, key)
if err != nil || !ok {
return nil, false, err
}
return &views.Entry{Value: v}, true, nil
}
func (s watchedSource) Values() (map[string][]byte, error) { return s.w.Values(s.state) }
@@ -17,9 +17,11 @@ import (
"time"
"git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshhandactsclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/busstate"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/views"
"mesh-delivery-indexer/internal/index"
)
@@ -286,13 +288,30 @@ func main() {
tools := Tools(asks, time.Now, stdio.State(index.VerdictsState), heldCounts(stdio.State(index.OldSectionsState)), acts)
// The modules and the module graph are the newest catalog graph revision's (milestone 47/G): revision.go.
revisions := bundleObjects{busstate.ObjectsOf(index.CatalogRevisionsObjects)}
trunk := index.TrunkStates{Objects: revisions, Under: keptSource, Kept: &index.KeptTrunk{Say: func(format string, a ...any) {
// The view of the trunk is read from the indexer's own objects once when the runtime starts and again each time the seat's
// event catalog-revision is heard (design 47 G A-487b), through the bundle's view path.
say := func(format string, a ...any) {
fmt.Fprintf(os.Stderr, "[mesh-delivery-indexer-tools] "+format+"\n", a...)
}}}
tools = append(tools, catalogTools(trunk.State)...)
}
kept := &index.KeptTrunk{Say: say}
trunk := index.TrunkStates{Objects: revisions, Under: keptSource, Kept: kept}
follow := &views.Reader{Module: os.Getenv(stdio.ServedAs), Logf: say, Acts: acts}
if err := follow.Adopt(trunkFollower(kept, revisions), meshcatalog.Seat, meshcatalog.EventCatalogRevision, func(ctx context.Context, data []byte) error {
var ev meshcatalog.CatalogRevisionEvent
if err := json.Unmarshal(data, &ev); err != nil {
return nil // not a revision event: offering it again changes nothing
}
return kept.Heard(ctx, revisions, ev)
}); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-indexer-tools] %v\n", err)
os.Exit(1)
}
follow.Start(context.Background())
watched := stdio.NewWatchedStates()
tools = append(tools, catalogTools(trunk.State, kept, watched)...)
// The seat mesh-catalog's verb revision, a first view of the catalog half answered in this runtime of its own to exactly
// the parts whose entry declares it (novox/hq ADR 0663).
tools = append(tools, index.RevisionTool(revisions, os.Getenv(stdio.ServedAs)), index.EntryTool(revisions, keptRead{stdio.State(index.EntryIndexState)}), index.EntriesTool(revisions, keptRead{stdio.State(index.EntryIndexState)}, keptRead{stdio.State(index.NamedState)}), index.VersionsTool(keptRead{stdio.State(index.VersionsState)}),
tools = append(tools, index.RevisionTool(revisions, kept, os.Getenv(stdio.ServedAs)), index.EntryTool(revisions, keptRead{stdio.State(index.EntryIndexState)}), index.EntriesTool(revisions, keptRead{stdio.State(index.EntryIndexState)}, keptRead{stdio.State(index.NamedState)}), index.VersionsTool(keptRead{stdio.State(index.VersionsState)}),
index.OutdatedTool(&index.Indexer{Versions: keptRead{stdio.State(index.VersionsState)}, Heard: keptRead{stdio.State(index.HeardState)}, Revisions: revisions}),
index.InputsReadTool(keptRead{stdio.State(index.InputsState)}, stateobjects.ForState(index.InputsState).Objects))
tools = append(tools, index.CatalogRevisionTools(revisions, asks, time.Now, func(what, why string) error {
@@ -318,18 +337,26 @@ func main() {
})...)
tools = append(tools, viewTools(os.Getenv, trunk.State, time.Now)...)
tools = append(tools, index.PendingTools(keptRead{stdio.State(index.PendingState)}, keptRead{stdio.State(index.OwnBuildStateName)}, asks, time.Now)...)
tools = append(tools, index.ValidationTools(asks, time.Now, func(what, why string) error {
_, err := acts.Record(meshhandactsclient.Request{What: what, Why: why, Cause: index.ValidateAgainTool, Verb: "validate-again"})
return err
})...)
tools = append(tools, index.FingerprintsTools(asks, time.Now, func(what, why string) error {
_, err := acts.Record(meshhandactsclient.Request{What: what, Why: why, Cause: index.FingerprintsAgainTool, Verb: "fingerprints-again"})
return err
}, 10*time.Second)...)
tools = append(tools, configReadersTool(stdio.NewWatchedStates(), trunk.State("modules")), busStateTool(trunk.State), requirementsTool(keptSource),
tools = append(tools, configReadersTool(watched, trunk.State("modules")), busStateTool(trunk.State), requirementsTool(keptSource),
packageUsersTool(trunk.State), seatDefinitionTool(keptSource), outOfStepTool(os.Getenv(stdio.ServedAs), keptSource))
if err := stdio.Serve("", tools); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-indexer-tools] %v\n", err)
os.Exit(1)
}
}
// trunkFollower is the bundle's view of the trunk as a follower of the reader: its first view is the one read of the
// indexer's objects the start makes, and answers no value (the picture is KeptTrunk's, not the reader's).
func trunkFollower(kept *index.KeptTrunk, revisions index.Objects) *views.Follower {
return &views.Follower{Owner: meshcatalog.Seat, State: "catalog-half", Tool: "catalog-revisions",
Ask: func(ctx context.Context, _ string) (views.Answer, error) {
if err := kept.Reread(ctx, revisions); err != nil {
return views.Answer{}, err
}
return views.Answer{V: views.Version, Owner: meshcatalog.Seat, State: "catalog-half", Created: time.Unix(1, 0).UTC(), Values: map[string]json.RawMessage{}, At: time.Now()}, nil
}}
}
@@ -22,7 +22,6 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
@@ -97,11 +96,7 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
// The forge bridge's tool bridge_merged_message (issue 1585): answers the merged message published for a merge, by
// repository@merge, as the bridge does; a restarted indexer holds none in memory and asks it.
var bridgeMessages sync.Map
// The build seat node-build-go's stand-in (design 47 G §712, §713): it hears the indexer's validation asks and ends each with
// the manifest validator's verdict; refusing names, by the commit the job runs at, the manifest a revision's change refuses.
var validations *standIn
var refusing sync.Map
js, ctx, candidates, _, restart := startIndexerRun(t, func(ctx context.Context, builds buildLog) {
js, ctx, candidates, _, _ := startIndexerRun(t, func(ctx context.Context, builds buildLog) {
// The builder's log entries of two builds' ends (ADR 0571 clause 1), as the builder appends them.
builds.end(t, "novox/mesh-delivery", "a", "modules/a")
builds.end(t, "novox/mesh-catalog", "c", "modules/c")
@@ -128,9 +123,8 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
if _, err := tc.Put(ctx, key, value); err != nil {
t.Fatal(err)
}
// The trunk holds the indexer's entry, taken in at an earlier commit with its own manifest; the validation names no Go module (A-268).
// The trunk holds the indexer's entry, taken in at an earlier commit with its own manifest; the take-in asks no validation (A-476 a).
seedBuildModules(t, ctx, js, map[string]meshcatalog.CatalogEntry{testIndexer: trunkIndexerEntry(t, ctx, js)})
validations = standInBuildSeat(t, ctx, js, &refusing)
// The node-engine's composition on n1 (process.Conn.Call's subjects): the build of the indexer that runs there.
answer := func(m *nats.Msg) {
raw, _ := json.Marshal(map[string]any{"result": map[string]any{"runs": map[string]string{"fp": *running.Load()}}, "node": "n1"})
@@ -281,17 +275,13 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
}
return takein.TakeIn{}, false
}
held, ok := said(takein.TakeInPending, m1Key)
if !ok || held.What != takein.TakeInRevision || held.WaitsFor == "" || held.WaitsFor == first || len(held.Merges) != 1 {
t.Fatalf("take-in.pending.%s = %+v (said %v) within 20 s, want the revision waiting for another build than %s", m1Key, held, ok, first)
// A-476 a, b: the revision changing the indexer's own tree is taken in at once, with the build this program runs and no
// validation ask: it is never said pending, and the build the revision names is not waited for.
if taken, ok := said(takein.TakeInTaken, m1Key); !ok || len(taken.Merges) != 1 || taken.Merges[0].Merge != m1.Merge {
t.Fatalf("take-in.taken.%s = %+v (said %v) within 20 s, want the revision taken in at once", m1Key, taken, ok)
}
named := held.WaitsFor
// IX5 (§712): the revision was validated first, by one run ask whose job names the indexer entry's repository, path and
// the revision's head, and no heads; accepted, it is never asked again while it waits for the named indexer.
if asks := validations.asked(); len(asks) != 1 || asks[0].Run == nil || asks[0].Kind != sdkstep.AskCheck ||
asks[0].Run.Program.Repository != m1.Repository || asks[0].Run.Program.Dir != "modules/"+ix || asks[0].Run.Program.Commit != m1.Merge ||
len(asks[0].Run.Heads) != 0 || asks[0].Run.Program.Package != "./cmd/mesh-judge" {
t.Fatalf("validation asks = %+v, want one run of the indexer's manifest validator at %s", asks, m1.Merge)
if held := saidAll(takein.TakeInPending, m1Key); len(held) != 0 {
t.Errorf("the revision was said pending: %+v", held)
}
get := func(bucket, key string) ([]byte, bool) {
t.Helper()
@@ -312,11 +302,10 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
}
return e.Value(), true
}
// Only what building and moving the named indexer needs is written.
raw, ok := get(index.ModulesState, ix)
own, err := meshcatalogclient.DecodeModule(raw)
if !ok || err != nil || own.Fp2 != named || own.Commit != m1.Merge {
t.Fatalf("modules.%s = %+v (%v), want the named build %s at the merge", ix, own, err, named)
if !ok || err != nil || own.Commit != m1.Merge {
t.Fatalf("modules.%s = %+v (%v), want the indexer at the merge", ix, own, err)
}
if _, ok := get(index.Bucket(testIndexer, index.ManifestsState), own.Manifest); !ok {
t.Errorf("manifests.%s is not written", own.Manifest)
@@ -325,16 +314,11 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
t.Errorf("the verdict on the indexer's manifest %s is not written", own.Manifest)
}
repoKey, _ := trunk.RepositoryKey(m1.Repository)
if _, ok := get(index.ModulesState, "a"); ok {
t.Errorf("modules.a is written while the revision is pending")
if raw, ok := get(index.ModulesState, "a"); !ok || !strings.Contains(string(raw), m1.Merge) {
t.Errorf("modules.a = %s, want it at the merge %s", raw, m1.Merge)
}
if _, ok := get(index.Bucket(testIndexer, index.TakenState), repoKey); ok {
t.Errorf("taken.%s is written while the revision is pending", repoKey)
}
if e, err := candidates.Get(ctx, "feat/x"); err != nil {
t.Errorf("the candidate feat/x is gone while the merge of #12 is pending: %v", err)
} else if c, err := index.DecodeCandidate(e.Value()); err != nil || len(c.Pulls) != 2 {
t.Errorf("the candidate feat/x = %+v while the merge of #12 is pending, want its two pull requests", c)
if _, ok := get(index.Bucket(testIndexer, index.TakenState), repoKey); !ok {
t.Errorf("taken.%s is not written", repoKey)
}
// A group's first member arrives meanwhile; its end is still to come (the knowledge base's remark: it stays across the restart).
@@ -365,34 +349,13 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
}
var w []takein.PendingWait
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if w = waits(); len(w) == 2 && w[0].WaitsFor == named && w[1].Said != "" {
if w = waits(); len(w) == 1 && w[0].Said != "" {
break
}
}
if len(w) != 2 || w[0].Key != m1Key || w[0].WaitsFor != named || !slices.Contains(w[0].Holds, "a") || !slices.Contains(w[0].Holds, ix) ||
w[1].Key != groupKey || w[1].What != takein.TakeInGroup || len(w[1].Merges) != 1 || w[1].Said != takein.TakeInPendingEvent {
t.Fatalf("indexer_pending = %+v, want the revision %s waiting for %s holding a and the indexer, then the group gathered", w, m1Key, named)
}
// The named indexer starts (the node-engine now runs that build): it takes the revision in, and leaves the group.
running.Store(&named)
restart()
if taken, ok := said(takein.TakeInTaken, m1Key); !ok || len(taken.Merges) != 1 || taken.Merges[0].Merge != m1.Merge {
t.Fatalf("take-in.taken.%s = %+v (said %v) within 20 s of the start of %s", m1Key, taken, ok, named)
}
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if w = waits(); len(w) == 1 {
break
}
}
if raw, ok := get(index.ModulesState, "a"); !ok || !strings.Contains(string(raw), m1.Merge) {
t.Errorf("modules.a = %s after the take-in at the start, want it at the merge %s", raw, m1.Merge)
}
if _, ok := get(index.Bucket(testIndexer, index.TakenState), repoKey); !ok {
t.Errorf("taken.%s is not written by the take-in at the start", repoKey)
}
if len(w) != 1 || w[0].Key != groupKey || len(w[0].Keys) != 1 || w[0].Keys[0] != g1Key {
t.Fatalf("indexer_pending after the start = %+v, want only the group gathered, its first member %s", w, g1Key)
if len(w) != 1 || w[0].Key != groupKey || w[0].What != takein.TakeInGroup || len(w[0].Merges) != 1 || len(w[0].Keys) != 1 || w[0].Keys[0] != g1Key ||
w[0].Said != takein.TakeInPendingEvent {
t.Fatalf("indexer_pending = %+v, want only the group gathered, its first member %s", w, g1Key)
}
if _, ok := get(index.ModulesState, "c"); ok {
t.Errorf("modules.c is written while its group is gathered")
@@ -425,23 +388,6 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10; 1585/T14, T15
t.Errorf("the group's first member %s was said pending as a revision of its own", g1Key)
}
// IX5 (§712, §713): a merge whose changed manifest gains a problem the trunk case lacks stays pending, refused naming it,
// said in take-in.pending; nothing of it is taken in.
m3 := forgemsg.Forge{Kind: forgemsg.ForgeMerged, Repository: "novox/mesh-delivery", Branch: "feat/y", Number: 13, Head: h1,
Merge: "3333333333333333333333333333333333333333", Root: "r0", Trees: map[string]string{"modules/a": "tree-a-3"},
Manifests: map[string]string{"modules/a": `{"module":"a","version":"2.0"}`}}
refusing.Store(m3.Merge, "a")
m3Key := takein.PendingKey(merged(m3))
if refused, ok := said(takein.TakeInPending, m3Key); !ok || !strings.Contains(refused.Refused, "a: the stand-in's problem") || refused.Failed != "" {
t.Fatalf("take-in.pending.%s = %+v (said %v) within 20 s, want the revision refused naming a's problem", m3Key, refused, ok)
}
if w = waits(); len(w) != 1 || w[0].Key != m3Key || w[0].Validation == nil || w[0].Validation.Accepted != 0 || len(validations.asked()) != 3 {
t.Fatalf("indexer_pending = %+v, asks %d: want the refused revision pending with its validation, asked once after m1 and the group", w, len(validations.asked()))
}
if raw, ok := get(index.ModulesState, "a"); !ok || strings.Contains(string(raw), m3.Merge) {
t.Errorf("modules.a = %s: the refused revision was taken in", raw)
}
}
// trunkIndexerEntry keeps the indexer's own manifest by its digest, accepted, and answers its trunk entry at an earlier commit.
@@ -475,63 +421,6 @@ func trunkIndexerEntry(t *testing.T, ctx context.Context, js jetstream.JetStream
return e
}
// standIn is the build seat node-build-go as the indexer's validation meets it: the asks it heard, oldest first.
type standIn struct {
mu sync.Mutex
asks []sdkstep.StepAsk
}
func (s *standIn) asked() []sdkstep.StepAsk {
s.mu.Lock()
defer s.mu.Unlock()
return slices.Clone(s.asks)
}
// standInBuildSeat keeps the seat's check queue as a stream and hears the indexer's asks on it (accept.check.indexer): each is
// ended at once with step-ended on the seat's family, its answer inline (A-254), the program's last line: the manifest validator's verdict over both cases,
// the change case refusing the manifest refusing names for the job's commit.
func standInBuildSeat(t *testing.T, ctx context.Context, js jetstream.JetStream, refusing *sync.Map) *standIn {
t.Helper()
subject, err := sdkstep.AskSubjectBy(sdkstep.AskGo, sdkstep.AskCheck, index.ValidationAsker)
if err != nil || subject != "mesh.seat.node-build-go.accept.check.mesh-catalog" {
t.Fatalf("the validation's subject %q %v, want the indexer's seat name its asker (A-163, A-232)", subject, err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "SEAT_NODE_BUILD_GO_CHECK", Subjects: []string{"mesh.seat.node-build-go.accept.check.>"}}); err != nil {
t.Fatal(err)
}
// The indexer's state inputs and its objects, where the validation's request is kept for the build seat's holder (§354).
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.InputsState), History: 1}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{Bucket: catalogue.BusName(testIndexer, catalogue.StateObjectsName(index.InputsState))}); err != nil {
t.Fatal(err)
}
s := &standIn{}
if _, err := js.Conn().Subscribe(subject, func(m *nats.Msg) {
a, err := sdkstep.DecodeStepAsk(m.Data)
if err != nil || a.Run == nil {
t.Errorf("the stand-in build seat heard what is no run ask: %v", err)
return
}
s.mu.Lock()
s.asks = append(s.asks, a)
s.mu.Unlock()
change := composer.CaseVerdict{Name: "change", Manifests: map[string]composer.ManifestRead{}}
if module, ok := refusing.Load(a.Run.Program.Commit); ok {
change.Manifests[module.(string)] = composer.ManifestRead{Refused: "the stand-in's problem"}
}
verdict, _ := json.Marshal(composer.Verdict{V: 1, Cases: []composer.CaseVerdict{change, {Name: judge.TrunkCase, Manifests: map[string]composer.ManifestRead{}}}})
end, _ := json.Marshal(sdkstep.StepEnded{V: 1, ID: a.ID, Ask: a.ID, Asker: index.ValidationAsker, Node: "n1", Outcome: sdkstep.OutcomePassed,
Output: string(verdict), At: time.Now().UTC()})
if _, err := js.Publish(ctx, nodebuildgo.SubjectStepEnded("n1"), end); err != nil {
t.Errorf("the stand-in build seat could not say step-ended: %v", err)
}
}); err != nil {
t.Fatal(err)
}
return s
}
// jsSource is a bucket as a views.Source.
type jsSource struct {
ctx context.Context
@@ -1027,6 +916,17 @@ func TestIndexerWritesModulesAtAMerge(t *testing.T) { // 1239/AC9; I1, I3: the m
if asks, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: index.Bucket(testIndexer, index.TakeInsState), History: 1}); err != nil {
t.Fatal(err)
}
// A want and a running build said before the start: the indexer removes no entry until both streams were read whole and
// not empty (design 47 G §675, 5188d36a), so the drop below removes entry.dx only with them.
fp := "src2:" + strings.Repeat("b", 64)
at := time.Now().UTC()
want, _ := json.Marshal(meshwants.WantedEvent{V: 1, Node: "n1", Module: "avahi", Fp: fp, Revision: 1, At: &at})
run, _ := json.Marshal(nodeengine.Running{V: 1, Node: "n2", Module: "avahi", Fingerprint: fp, At: at})
for subject, data := range map[string][]byte{meshwants.SubjectWanted("n1", "avahi"): want, nodeengine.SubjectRunning("n2", "avahi"): run} {
if _, err := js.Publish(ctx, subject, data); err != nil {
t.Fatal(err)
}
}
})
const merge = "89abcdef0123456789abcdef0123456789abcdef"
data, err := forgemsg.EncodeForge(forgemsg.Forge{Kind: forgemsg.ForgeMerged, Repository: "novox/mesh-delivery", Branch: "feat/x", Number: 12,
@@ -1518,7 +1418,8 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
t.Fatal(err)
}
toolchains = kv
for node, version := range map[string]string{"n1": "go1.27.3", "n2": "go1.27.2"} {
// A-471: a toolchain is agreed by its language version (go1.27), not its patch; n2 reports another language version.
for node, version := range map[string]string{"n1": "go1.27", "n2": "go1.28.1"} {
key, _ := index.ToolchainKey(node, "go")
value, err := index.EncodeToolchain(nodebuildgo.Toolchain{Language: "go", Version: version, Recipe: "1", At: ptr(time.Now())})
if err != nil {
@@ -1562,15 +1463,15 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
return got
}
agreed := func(m meshcatalog.ModulesByModuleValue) bool {
return m.Fp2 != "" && m.Toolchains["go"].Version == "go1.27.3"
return m.Fp2 != "" && m.Toolchains["go"].Version == "go1.27"
}
held := read(agreed)
if !agreed(held) {
t.Fatalf("within 20 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27.3 alone", held.Fp2, held.None2, held.Toolchains)
t.Fatalf("within 20 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27 alone", held.Fp2, held.None2, held.Toolchains)
}
place(ctx, t, "n2", "mesh-build-go")
key, _ := index.ToolchainKey("n2", "go")
value, err := index.EncodeToolchain(nodebuildgo.Toolchain{Language: "go", Version: "go1.27.2", Recipe: "1", At: ptr(time.Now().Add(time.Second))})
value, err := index.EncodeToolchain(nodebuildgo.Toolchain{Language: "go", Version: "go1.28.1", Recipe: "1", At: ptr(time.Now().Add(time.Second))})
if err != nil {
t.Fatal(err)
}
@@ -1593,7 +1494,7 @@ func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
t.Fatal(err)
}
disagree := func(m meshcatalog.ModulesByModuleValue) bool {
return m.Fp2 == "" && strings.HasPrefix(m.None2, "builders disagree: go1.27.2 recipe 1 on n2")
return m.Fp2 == "" && strings.HasPrefix(m.None2, "builders disagree: go1.27 recipe 1 on n1; go1.28 recipe 1 on n2")
}
if m := read(disagree); !disagree(m) {
t.Fatalf("within 20 s of a merge after n2's assignments avahi has fp2 %q, none2 %q: want the builders to disagree", m.Fp2, m.None2)
@@ -479,11 +479,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The take-in step (ADR 0633): its state pending and its stream take-in, both this delivery's, bound when they appear;
// until then every merge is taken in at once, as before.
Pending: awaitedStore{b: process.AwaitBucket(ctx, js, conn, index.Bucket(cred.Module, index.PendingState), "no merge is held for the indexer its revision names: each is taken in at once", logger.Printf)},
SayTakeIn: sayTakeIn(js, cred),
// The validation's run ask (design 47 G §713, A-247, A-248), put under the indexer's own asker.
ValidationAsks: buildrun.Asker{Name: index.ValidationAsker, Publisher: buildrun.JetStreamAsks{JS: js}},
// The build seat's verb output, for a validation's answer kept as an object (A-254).
AskVerb: calls.CallFile}
SayTakeIn: sayTakeIn(js, cred)}
// The registrations and the changes that wait for their warrant (design 47 G §2, §667): their states are bound when they
// appear; a warrant is heard on the router's decided event for this module, below.
registry := newRegistry(js, conn, cred, calls.Call,
@@ -206,6 +206,7 @@ func TestAMeshSDKMergeMovesOnlyTheModulesImportingAChangedPackage(t *testing.T)
if pub.Commit != m2 || len(pub.Packages) != 3 {
t.Fatalf("packages.%s = %+v within 20 s, want the merge %s with a digest of go, announce and buildstep", key, pub, m2)
}
// A-476a: mesh-sdk's merge moves importer, of another repository, which imports a package it moved.
moved := module("importer", func(m meshcatalog.ModulesByModuleValue) bool { return m.Fp2 != importer.Fp2 })
if moved.Packages[sdkGo] != m2 {
t.Errorf("importer moved to %+v, want it built against %s", moved.Packages, m2)
@@ -28,7 +28,6 @@ type takerIn interface {
NamedDrop(ctx context.Context, fingerprint, member, by, why string) index.TakenIn
Unpin(ctx context.Context, revision, record, by, why string) index.TakenIn
DropRunByHand(ctx context.Context, run, by, why string) index.TakenIn
ValidateAgainByHand(ctx context.Context, key, by, why string) index.TakenIn
RegistryAct(ctx context.Context, a index.TakeInAsk) index.TakenIn
}
@@ -75,9 +74,6 @@ func takenIn(ctx context.Context, ix takerIn, kv jetstream.KeyValue, e jetstream
} else if a.Merge != "" {
asked, r = "hand take-in of a merge", ix.TakeInMerge(ctx, a.MergeRepository, a.Merge, a.Sequence, a.By, a.Why)
a.Module = r.Module
} else if a.Pending != "" && a.ValidateAgain {
asked, r = "validation asked again", ix.ValidateAgainByHand(ctx, a.Pending, a.By, a.Why)
a.Module = a.Pending
} else if a.Pending != "" && a.Drop {
asked, r = "drop of a pending merge", ix.DropPending(ctx, a.Pending, a.By, a.Why, a.Abandon)
a.Module = a.Pending
@@ -7,8 +7,6 @@ import (
"sync/atomic"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
@@ -155,15 +153,6 @@ func heardStepEnded(ctx context.Context, ix *index.Indexer, m jetstream.Msg, log
_ = m.Ack()
return
}
// The end of a validation ask resumes its pending revision (design 47 G §713); its own fault is given again by the bus.
var end buildstep.StepEnded
if err := json.Unmarshal(m.Data(), &end); err == nil {
if err := ix.HearValidationEnded(ctx, end); err != nil {
logf("take-in: the end of the validation ask %s is not taken in, so the bus gives it again: %v", end.Ask, err)
leftForBus(m, "the end of the validation ask "+end.Ask, err, logf)
return
}
}
if err := ix.HearInputEnded(ctx, e); err != nil {
logf("inputs: the input of the ask %s could not be deleted, so the bus gives it again: %v", e.Ask, err)
leftForBus(m, "the step-ended of the ask "+e.Ask, err, logf)
@@ -0,0 +1,190 @@
package index
import (
"context"
"errors"
"fmt"
"maps"
"slices"
mcc "git.novox.be/novox/mesh-contracts/health-and-repair/go/clients/meshconditionsclient"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
)
// A merge the indexer never took in (design 47 G A-463, amended by A-464): at its start and on every source revision the
// gate's source names, the indexer compares each repository's `taken` merge with the head that source names. A repository
// whose head is newer than its taken merge (the taken merge is that head's ancestor, or no merge is taken) is taken in as
// its merged message would have been (the catch-up of indexer_take_in, for that repository), said in the log, with the
// finding merge-untaken raised while it stays behind and cleared once it is taken in. A head the taken merge contains (the
// source not yet moved to a merge just taken in) is left, and a head the taken merge neither contains nor is in is said.
// UntakenMergeKind is the finding of a repository whose head the gate's source names but whose merge the indexer never took in.
const UntakenMergeKind = "merge-untaken"
func untakenFindingID(own, key string) string { return own + ".untaken." + key }
// kickBehind runs the comparison in the background, one at a time: a source revision that arrives while one runs makes it
// run once more, on the latest heads. It runs after the first start's catch-up, which takes every head in itself.
func (ix *Indexer) kickBehind(ctx context.Context) {
if ix.Taken == nil || ix.Message == nil || ix.Modules == nil {
return
}
if ix.catchingUp.Load() {
ix.behindWaiting.Store(true) // compared again when that catch-up ends (A-463a)
return
}
if ix.behindRunning.Swap(true) {
ix.behindAgain.Store(true)
return
}
ix.catchUpWG.Add(1)
go func() {
defer ix.catchUpWG.Done()
for {
ix.behindAgain.Store(false)
ix.TakeInBehind(ctx)
if !ix.behindAgain.Load() {
ix.behindRunning.Store(false)
if !ix.behindAgain.Load() || ix.behindRunning.Swap(true) {
return
}
}
}
}()
}
// TakeInBehind compares each repository's taken merge with the head the latest source names, takes in each repository
// that is behind and answers the repositories it took in.
func (ix *Indexer) TakeInBehind(ctx context.Context) []string {
heads := ix.sourceHeads()
if ix.Taken == nil || ix.Message == nil || ix.Modules == nil || len(heads) == 0 {
return nil
}
behind := map[string]string{}
taken := map[string]string{}
held := ix.pendingHeads(ctx)
for _, repository := range slices.Sorted(maps.Keys(heads)) {
head := heads[repository]
key, err := trunk.RepositoryKey(repository)
if head == "" || err != nil {
continue
}
value, _, found, err := ix.Taken.Get(ctx, key)
if errors.Is(err, process.ErrNotYet) {
return nil // the state is not on the bus yet: the next source revision compares again
}
if err != nil {
ix.logf("merges: taken.%s cannot be read, so %s is not compared with its head: %v", key, repository, err)
continue
}
merge := ""
if found {
t, err := takein.DecodeTaken(value)
if err != nil {
ix.logf("merges: taken.%s holds what is not a taken merge, left: %v", key, err)
continue
}
merge = t.Merge
}
switch {
case held[head]:
// A head the pending holds (a member waiting for its group's end) is taken in by that path, whole with
// its group: it is not behind (A-463a).
ix.clearUntaken(ctx, key, "the head of "+repository+" waits in the indexer's pending")
case merge == head:
ix.clearUntaken(ctx, key, "the indexer took in the head of "+repository)
case merge == "":
behind[repository], taken[repository] = head, ""
case ix.Contains == nil:
ix.logf("merges: %s is taken in at %.12s and its head is %.12s, but the indexer reads no ancestry of the forge, so which is newer is not said", repository, merge, head)
default:
inHead, err := ix.Contains(ctx, repository, merge, head)
if err != nil {
ix.logf("merges: whether %.12s is in %.12s of %s cannot be read: %v", merge, head, repository, err)
continue
}
if inHead {
behind[repository], taken[repository] = head, merge
continue
}
ix.clearUntaken(ctx, key, "the head of "+repository+" is in the merge taken in")
if headIn, err := ix.Contains(ctx, repository, head, merge); err == nil && !headIn {
ix.logf("merges: %s is taken in at %.12s and its head is %.12s, and neither holds the other: left", repository, merge, head)
}
}
}
for _, repository := range slices.Sorted(maps.Keys(behind)) {
ix.sayUntaken(ctx, repository, taken[repository], behind[repository])
ix.logf("merges: %s is taken in at %.12s but the gate's source names %.12s: taken in now", repository, taken[repository], behind[repository])
}
if len(behind) == 0 {
return nil
}
unread := ix.takeInHeads(ctx, behind)
var done []string
for _, repository := range slices.Sorted(maps.Keys(behind)) {
if slices.Contains(unread, repository) {
continue
}
done = append(done, repository)
key, _ := trunk.RepositoryKey(repository)
ix.clearUntaken(ctx, key, fmt.Sprintf("%s taken in at %.12s", repository, behind[repository]))
ix.logf("merges: %s taken in at %.12s", repository, behind[repository])
}
if len(done) > 0 {
if err := ix.KeepView(ctx); err != nil {
ix.logf("merges: the view could not be kept after %v were taken in: %v", done, err)
}
}
return done
}
func (ix *Indexer) sayUntaken(ctx context.Context, repository, merge, head string) {
if ix.Findings == nil {
return
}
key, _ := trunk.RepositoryKey(repository)
taken := "no merge"
if merge != "" {
taken = fmt.Sprintf("%.12s", merge)
}
f := ix.observed(mcc.ScopeModule, untakenFindingID(ix.Module, key), UntakenMergeKind,
fmt.Sprintf("the indexer took in %s at %s, and the gate's source names the head %.12s", repository, taken, head),
"A merge the indexer never took in",
"A merge into a watched branch was never heard or applied, so no package or module of the repository follows its trunk. The indexer takes the head in itself and clears this; indexer_take_in with catch-up does it by hand.",
UntakenMergeKind)
if err := ix.Findings.Write(ctx, f); err != nil {
ix.logf("merges: the finding %s for %s is not written: %v", UntakenMergeKind, repository, err)
}
}
func (ix *Indexer) clearUntaken(ctx context.Context, key, why string) {
if ix.Findings == nil {
return
}
if err := ix.clearFinding(ctx, mcc.ScopeModule, untakenFindingID(ix.Module, key), UntakenMergeKind, why); err != nil {
ix.logf("merges: the finding %s for %s is not cleared: %v", UntakenMergeKind, key, err)
}
}
// pendingHeads are the merge commits the indexer's pending holds (A-463a).
func (ix *Indexer) pendingHeads(ctx context.Context) map[string]bool {
out := map[string]bool{}
if ix.Pending == nil || !ix.pendingOnBus(ctx) {
return out
}
entries, err := ReadPending(ctx, ix.Pending, ix.logf)
if err != nil {
ix.logf("merges: the pending state cannot be read, so every head is compared: %v", err)
return out
}
for _, e := range entries {
if e.Forge.Merge != "" {
out[e.Forge.Merge] = true
}
}
return out
}
@@ -0,0 +1,112 @@
package index
import (
"context"
"encoding/json"
"testing"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
)
// Design 47 G A-463: a merge the indexer never took in is noticed at a source revision and taken in as its merged message
// would have been, with a finding raised while it stays behind and cleared once it is taken in.
func TestAMergeTheIndexerNeverTookInIsTakenInAndSaidWhileItIsBehind(t *testing.T) {
ctx := context.Background()
b := newTakeInBed(t)
b.ix.Taken, b.ix.Findings = newMem(), newFindings()
key, _ := trunk.RepositoryKey(catalogueRepo)
old, head := sha('7'), sha('8')
data, _ := takein.EncodeTaken(takein.Taken{Repository: trunk.RepositoryName(catalogueRepo), Merge: old})
b.ix.Taken.(*memStore).put(key, data)
ancestry := map[[2]string]bool{{old, head}: true} // old is in head
b.ix.Contains = func(_ context.Context, _, commit, in string) (bool, error) {
return commit == in || ancestry[[2]string{commit, in}], nil
}
b.ix.source = sourceNow{rev: 3, heads: map[string]string{catalogueRepo: head}}
// The head has no merged message yet: nothing is taken in and the finding stays.
if done := b.ix.TakeInBehind(ctx); len(done) != 0 || len(b.ix.Findings.Held()) != 1 || b.ix.Findings.Held()[0].Kind != UntakenMergeKind {
t.Fatalf("a head whose message is not read: taken in %v, findings %+v: want none taken in and %s raised", done, b.ix.Findings.Held(), UntakenMergeKind)
}
// Its message is there: it is taken in, its taken key moves and the finding is cleared.
b.publish(t, b.merge(30, head, "tree-avahi-3", ""))
if done := b.ix.TakeInBehind(ctx); len(done) != 1 || len(b.ix.Findings.Held()) != 0 || b.commit(t, "avahi") != head {
t.Fatalf("a head with its message: taken in %v, findings %+v, avahi at %s: want it taken in at %s and the finding cleared", done, b.ix.Findings.Held(), b.commit(t, "avahi"), head)
}
if v, _, _, _ := b.ix.Taken.Get(ctx, key); string(v) == string(data) {
t.Errorf("taken.%s still names %s", key, old)
}
// A head the taken merge already holds (the source not yet moved to a merge just taken in) is left.
b.ix.source = sourceNow{rev: 4, heads: map[string]string{catalogueRepo: old}}
if done := b.ix.TakeInBehind(ctx); len(done) != 0 {
t.Errorf("a head older than the taken merge: taken in %v, want none", done)
}
// A source revision starts the comparison by itself (no timer): the taken key is put back to the old merge and the source
// names the head again.
b.ix.Taken.(*memStore).put(key, data)
if err := applyS(b.ix, ctx, 5, sourceOf(catalogueRepo, head)); err != nil {
t.Fatal(err)
}
b.ix.catchUpWG.Wait()
if v, _, _, _ := b.ix.Taken.Get(ctx, key); string(v) == string(data) {
t.Errorf("a source revision naming %s left taken.%s at %s", head, key, old)
}
}
// Design 47 G A-463a: a head the pending holds (a group member waiting for its group's end) is not behind: it raises no finding
// and is not taken in alone.
func TestAHeadThePendingHoldsIsNotBehind(t *testing.T) {
ctx := context.Background()
b := newTakeInBed(t)
b.ix.Taken, b.ix.Findings = newMem(), newFindings()
key, _ := trunk.RepositoryKey(catalogueRepo)
old, head := sha('7'), sha('8')
data, _ := takein.EncodeTaken(takein.Taken{Repository: trunk.RepositoryName(catalogueRepo), Merge: old})
b.ix.Taken.(*memStore).put(key, data)
b.ix.Contains = func(_ context.Context, _, commit, in string) (bool, error) { return commit == in || commit == old, nil }
b.publish(t, b.merge(30, head, "tree-avahi-3", ""))
// The head waits in pending as a member of a group not ended.
held := b.merge(30, head, "tree-avahi-3", "")
raw, _ := forgemsg.EncodeForge(held)
pv, _ := json.Marshal(Pending{V: 1, Forge: raw, Group: "feat", Since: time.Now(), LastWriter: "x"})
b.pending.put(takein.PendingKey(1), pv)
b.ix.source = sourceNow{rev: 3, heads: map[string]string{catalogueRepo: head}}
if done := b.ix.TakeInBehind(ctx); len(done) != 0 || len(b.ix.Findings.Held()) != 0 || b.commit(t, "avahi") == head {
t.Fatalf("a head the pending holds: taken in %v, findings %+v, avahi at %s: want none taken in and none raised", done, b.ix.Findings.Held(), b.commit(t, "avahi"))
}
}
// Design 47 G A-463a: a source revision heard while a catch-up runs is compared again when that catch-up ends.
func TestASourceRevisionHeardDuringACatchUpIsComparedWhenItEnds(t *testing.T) {
ctx := context.Background()
b := newTakeInBed(t)
b.ix.Taken, b.ix.Findings = newMem(), newFindings()
key, _ := trunk.RepositoryKey(catalogueRepo)
h1, h2 := sha('8'), sha('9')
b.ix.Contains = func(_ context.Context, _, commit, in string) (bool, error) {
return commit == in || (commit == h1 && in == h2), nil
}
b.publish(t, b.merge(30, h1, "tree-avahi-3", ""))
b.publish(t, b.merge(31, h2, "tree-avahi-4", ""))
b.ix.source = sourceNow{rev: 3, heads: map[string]string{catalogueRepo: h1}}
inner := b.ix.Message
heard := false
b.ix.Message = func(c context.Context, r, m string, n int) ([]byte, error) {
if !heard {
heard = true // the catch-up of h1 runs: a new source revision is heard now
if err := applyS(b.ix, ctx, 4, sourceOf(catalogueRepo, h2)); err != nil {
t.Error(err)
}
}
return inner(c, r, m, n)
}
b.ix.CatchUp(ctx)
b.ix.catchUpWG.Wait()
v, _, _, _ := b.ix.Taken.Get(ctx, key)
if got, _ := takein.DecodeTaken(v); got.Merge != h2 {
t.Errorf("taken after the catch-up = %s, want %s: the revision heard during it is compared afterwards\n%s", got.Merge, h2, b.logs)
}
}
@@ -6,7 +6,6 @@ import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-sdk/go/stateobjects"
"reflect"
"slices"
@@ -200,16 +199,19 @@ type Indexer struct {
keepMu sync.Mutex // one catalog graph revision kept at a time (keepEntries)
// Pins is the state pins (pins.go); nil keeps none. PinsPending says how many messages the indexer's durable consumer of the
// events stream has not been given: a revision is removed only while none is pending (§579).
Pins Store
PinsPending func(ctx context.Context) (uint64, error)
namesMu sync.Mutex // packageNames
packageNames map[string]string // the package names read through the forge, by repository, commit, path and artifact (artifacts.go)
catchingUp atomic.Bool // the first start's catch-up runs: its merges keep no revision until it ends (CatchUp)
catchUpWG sync.WaitGroup // the first start's catch-up in the background (kickCatchUp), for tests to wait on
ownFP string // the fingerprint of the build this program runs, fixed once at the start (FixOwnBuild)
ownFixed bool // FixOwnBuild asked already: it asks once
ownSince time.Time // when the build was asked and not known, with ownWhy
ownWhy string
Pins Store
PinsPending func(ctx context.Context) (uint64, error)
namesMu sync.Mutex // packageNames
packageNames map[string]string // the package names read through the forge, by repository, commit, path and artifact (artifacts.go)
catchingUp atomic.Bool // the first start's catch-up runs: its merges keep no revision until it ends (CatchUp)
behindRunning atomic.Bool // a comparison of the taken merges with the source's heads runs (kickBehind, A-463)
behindWaiting atomic.Bool // a source revision came while a catch-up ran (A-463a)
behindAgain atomic.Bool // a source revision came while it ran
catchUpWG sync.WaitGroup // the first start's catch-up in the background (kickCatchUp), for tests to wait on
ownFP string // the fingerprint of the build this program runs, fixed once at the start (FixOwnBuild)
ownFixed bool // FixOwnBuild asked already: it asks once
ownSince time.Time // when the build was asked and not known, with ownWhy
ownWhy string
// OwnBuildState keeps which build this process knows it runs, or since when and why not (FixOwnBuild); nil keeps none.
OwnBuildState Store
// Heard keeps what the indexer heard and compares later (A-80): the time a call is unanswered since (steps-since, call-since),
@@ -231,12 +233,6 @@ type Indexer struct {
Pending Store
// SayTakeIn publishes a take-in event (take-in.pending, take-in.taken; ADR 0633 §5); nil says none.
SayTakeIn func(ctx context.Context, t takein.TakeIn) error
// ValidationAsks puts the validation's run ask (design 47 G §713, A-247: buildrun.Asker, named as the indexer's asker,
// A-248); nil validates nothing, and a revision is taken in as before.
ValidationAsks ValidationAsks
// AskVerb asks a seat verb as the runtime calls it: the build seat's verb output, for a run's answer the seat keeps as an
// object (design 47 G A-254, buildstep.StepEnded.ReadOutput); nil reads none, and such an answer could not validate.
AskVerb buildstep.AskVerb
// Runs is the state runs: the runs on request the indexer holds a candidate for (runcandidates.go, A-174); nil keeps none.
Runs Store
runMu sync.Mutex // one run event at a time (runcandidates.go)
@@ -25,6 +25,9 @@ import (
type CatalogRead struct {
// Modules is every value of the modules state, by module.
Modules func() (map[string]meshcatalog.ModulesByModuleValue, error)
// Module is the one value of the modules state of that module, and whether the state holds it: the verb module reads
// the one module it is asked, never every module (design 47 G A-487).
Module func(name string) (meshcatalog.ModulesByModuleValue, bool, error)
// Manifest is the manifest kept under a digest in the manifests state, as its JSON.
Manifest func(digest string) (string, error)
// Graph is the module graph, by module.
@@ -51,17 +54,26 @@ func (c CatalogRead) CatalogModulesAnswer() (catalogue.CatalogModules, error) {
return catalogue.CatalogModules{Modules: out}, nil
}
// module is one module's value: through Module when set, else out of every module's.
func (c CatalogRead) module(name string) (meshcatalog.ModulesByModuleValue, bool, error) {
if c.Module != nil {
return c.Module(name)
}
mods, err := c.Modules()
m, ok := mods[name]
return m, ok, err
}
// CatalogModuleAnswer is the verb module: one module whole, its manifest included.
func (c CatalogRead) CatalogModuleAnswer(name string) (catalogue.CatalogOne, error) {
name = strings.TrimSpace(name)
if name == "" {
return catalogue.CatalogOne{}, fmt.Errorf("name the module")
}
mods, err := c.Modules()
m, ok, err := c.module(name)
if err != nil {
return catalogue.CatalogOne{}, fmt.Errorf("the indexer's modules cannot be read: %w", err)
}
m, ok := mods[name]
if !ok {
return catalogue.CatalogOne{}, fmt.Errorf("the trunk holds no module %s", name)
}
@@ -45,7 +45,7 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
t.Errorf("said %+v, want main 1, then the escaped branch 1 and 2", said)
}
answer := func(branch string, revision int64) meshcatalog.RevisionAnswer {
a, err := Revision(ctx, ix.Revisions, "indexer", "reader", meshcatalog.RevisionInput{Branch: branch, Revision: revision})
a, err := Revision(ctx, ix.Revisions, nil, "indexer", "reader", meshcatalog.RevisionInput{Branch: branch, Revision: revision})
if err != nil {
t.Fatalf("revision of %q: %v", branch, err)
}
@@ -66,7 +66,7 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
// A-205: `source` answers the branch's revision taken in from that source revision; a candidate's group alone answers its
// newest revision, whatever its heads.
bySource := func(branch string, source int64) (meshcatalog.RevisionAnswer, error) {
return Revision(ctx, ix.Revisions, "indexer", "reader", meshcatalog.RevisionInput{Branch: branch, Source: source})
return Revision(ctx, ix.Revisions, nil, "indexer", "reader", meshcatalog.RevisionInput{Branch: branch, Source: source})
}
if a, err := bySource("main", 5); err != nil || a.Revision.Revision != 1 || a.Entries[0].Commit != "c1" {
t.Errorf("main's source 5 answered %+v (%v)", a.Revision, err)
@@ -89,10 +89,10 @@ func TestACatalogRevisionIsKeptPerWatchedBranch(t *testing.T) {
t.Fatal(err)
}
}
if a, err := Revision(ctx, ix.Revisions, "indexer", "reader", meshcatalog.RevisionInput{Group: "run-ab"}); err != nil || a.Revision.Revision != 4 || a.Revision.Heads != "h2" || a.Entries[0].Commit != "c9" {
if a, err := Revision(ctx, ix.Revisions, nil, "indexer", "reader", meshcatalog.RevisionInput{Group: "run-ab"}); err != nil || a.Revision.Revision != 4 || a.Revision.Heads != "h2" || a.Entries[0].Commit != "c9" {
t.Errorf("a group alone answered %+v (%v), want its newest revision 4", a.Revision, err)
}
if _, err := Revision(ctx, ix.Revisions, "indexer", "reader", meshcatalog.RevisionInput{Group: "run-none"}); err == nil {
if _, err := Revision(ctx, ix.Revisions, nil, "indexer", "reader", meshcatalog.RevisionInput{Group: "run-none"}); err == nil {
t.Error("a candidate never kept was answered")
}
}
@@ -123,7 +123,7 @@ func TestTheNodeEngineReadsTheRevisionAsTheModuleHoldingItsSeat(t *testing.T) {
if err := ix.KeepBranchEntries(ctx, "main", 5, map[string]meshcatalog.CatalogEntry{"mesh-host": engine}); err != nil {
t.Fatal(err)
}
_, err := Revision(ctx, ix.Revisions, "indexer", c.caller, meshcatalog.RevisionInput{Branch: "main"})
_, err := Revision(ctx, ix.Revisions, nil, "indexer", c.caller, meshcatalog.RevisionInput{Branch: "main"})
if (err == nil) != c.want {
t.Errorf("caller %q: err %v, want answered %v", c.caller, err, c.want)
}
@@ -0,0 +1,72 @@
package index
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/views"
)
// listingObjects counts the listings of the bucket: each costs the whole bucket (56 ms at 11 000 objects, more through the runtime).
type listingObjects struct {
*memObjects
listings atomic.Int64
}
func (l *listingObjects) Names(ctx context.Context) ([]string, error) {
l.listings.Add(1)
return l.memObjects.Names(ctx)
}
// A seat verb that answers from the whole view of the trunk (modules, provides, dependents, module) reads the catalog-revisions
// bucket's listing once per call, never once per module: the node-engines ask `module` for every module at once, and 33 asks
// in 5 minutes passed the tool runner's 30 s (2026-11-17).
func TestTrunkViewIsReadOnceForAWholeAnswer(t *testing.T) {
objects := &listingObjects{memObjects: newMemObjects(0)}
const modules = 40
var refs []meshcatalog.EntryRef
for m := 0; m < modules; m++ {
name := fmt.Sprintf("mod%02d", m)
digest := fmt.Sprintf("%064x", m)
e := meshcatalog.CatalogEntry{Module: name, Repository: "o/r", Path: "m/" + name, Commit: "c", Fp: "src2:" + strings.Repeat("a", 64),
Manifest: map[string]any{"name": name}, ManifestDigest: "sha256:" + digest}
b, _ := json.Marshal(e)
if err := objects.Put(context.Background(), EntryObject(digest), b); err != nil {
t.Fatal(err)
}
refs = append(refs, meshcatalog.EntryRef{Module: name, Digest: digest})
}
b, _ := json.Marshal(meshcatalog.CatalogRevision{Kind: "trunk", Revision: 1, Entries: refs})
if err := objects.Put(context.Background(), "trunk.1", b); err != nil {
t.Fatal(err)
}
ts := TrunkStates{Objects: objects, Under: func(string) views.Source { return nil }, Kept: &KeptTrunk{}}
for _, state := range []string{"modules", "graph"} {
objects.listings.Store(0)
values, err := SourceValues(ts.State(state))
if err != nil || len(values) != modules {
t.Fatalf("the %s of the view = %d values (%v), want %d", state, len(values), err, modules)
}
if n := objects.listings.Load(); n > 2 {
t.Errorf("a whole read of the %s listed the bucket %d times, want at most 2 (once per call, not once per module)", state, n)
}
}
if _, err := meshcatalogclient.DecodeModule(mustValue(t, ts)); err != nil {
t.Errorf("a value of the view does not decode: %v", err)
}
}
func mustValue(t *testing.T, ts TrunkStates) []byte {
t.Helper()
v, err := SourceValues(ts.State("modules"))
if err != nil {
t.Fatal(err)
}
return v["mod00"]
}
@@ -438,15 +438,21 @@ func TrunkView(ctx context.Context, o Objects) (modules, nodes Store, at int64,
// BranchView is TrunkView of a watched branch: the view read from the newest revision the bucket keeps of that branch
// (design 47 G §291, A-93), empty and at 0 when it keeps none.
func BranchView(ctx context.Context, o Objects, branch string) (modules, nodes Store, at int64, err error) {
view, held := newOverlay(nil), newOverlay(nil)
newest, any, err := newestOf(ctx, o, branch)
if err != nil || !any {
return view, held, 0, err
return newOverlay(nil), newOverlay(nil), 0, err
}
_, entries, err := ReadRevision(ctx, o, RevisionObject(newest))
if err != nil {
return nil, nil, 0, err
}
return viewOfEntries(newest.Revision, entries)
}
// viewOfEntries is the view of the entries of one trunk revision, as BranchView reads them.
func viewOfEntries(revision int64, entries []meshcatalog.CatalogEntry) (modules, nodes Store, at int64, err error) {
view, held := newOverlay(nil), newOverlay(nil)
ctx, newest := context.Background(), meshcatalog.CatalogRevision{Revision: revision}
for _, e := range entries {
key, err := meshcatalogclient.ModulesKey(e.Module)
if err != nil {
@@ -496,14 +502,110 @@ type TrunkStates struct {
Kept *KeptTrunk
}
// KeptTrunk is the view of the trunk as last read, at its revision, shared by every read of a TrunkStates; Say, when set,
// says each read of a revision and how long it took.
// KeptTrunk is the view of the newest trunk revision, held in memory (design 47 G A-487, A-487b): it reads the newest
// revision from the indexer's own objects catalog-revisions once when the runtime starts (Reread) and again each time it
// hears the seat's event catalog-revision of a newer trunk revision (Heard); a read answers from the view held and never
// lists the bucket. Say, when set, says each read of a revision with its number, its module count and how long it took.
type KeptTrunk struct {
Say func(format string, a ...any)
mu sync.Mutex
at int64
mods Store
held Store
Say func(format string, a ...any)
read sync.Mutex // one re-read at a time; a read of the view never holds it
mu sync.Mutex
started bool
at int64
mods Store
held Store
rev meshcatalog.CatalogRevision
entries []meshcatalog.CatalogEntry
// manifests are the manifests the revision's entries carry whole, as JSON, by their digest as a module's value names it
// (A-487c: the verbs module and provides answer them from here, never from the manifests state).
manifests map[string]string
}
// Reread reads the newest trunk revision of the bucket and holds its view; a bucket keeping none holds an empty view at 0.
func (k *KeptTrunk) Reread(ctx context.Context, o Objects) error {
k.read.Lock()
defer k.read.Unlock()
return k.reread(ctx, o)
}
func (k *KeptTrunk) reread(ctx context.Context, o Objects) error {
began := time.Now()
fail := func(err error) error {
if k.Say != nil {
k.Say("the view of the trunk is BEHIND, its re-read failed after %s (the view held stays, the next heard catalog-revision reads again): %v", time.Since(began).Round(time.Millisecond), err)
}
return err
}
newest, any, err := newestTrunk(ctx, o)
if err != nil {
return fail(err)
}
var entries []meshcatalog.CatalogEntry
mods, held, at := Store(newOverlay(nil)), Store(newOverlay(nil)), int64(0)
if any {
if _, entries, err = ReadRevision(ctx, o, RevisionObject(newest)); err != nil {
return fail(err)
}
if mods, held, at, err = viewOfEntries(newest.Revision, entries); err != nil {
return fail(err)
}
}
manifests := map[string]string{}
for _, e := range entries {
if d := meshcatalog.BareManifestDigest(e.ManifestDigest); d != "" && e.Manifest != nil {
if raw, err := json.Marshal(e.Manifest); err == nil {
manifests[d] = string(raw)
}
}
}
k.mu.Lock()
was := k.at
k.mods, k.held, k.at, k.started, k.rev, k.entries, k.manifests = mods, held, at, true, newest, entries, manifests
k.mu.Unlock()
if k.Say != nil {
n, _ := mods.Keys(ctx)
k.Say("the view of the trunk: read revision %d (was %d), %d modules, in %s", at, was, len(n), time.Since(began).Round(time.Millisecond))
}
return nil
}
// Newest is the newest trunk revision held, with a copy of its entries; false when the bucket kept none at the last read.
func (k *KeptTrunk) Newest(ctx context.Context, o Objects) (meshcatalog.CatalogRevision, []meshcatalog.CatalogEntry, bool, error) {
if _, _, _, err := k.current(ctx, o); err != nil {
return meshcatalog.CatalogRevision{}, nil, false, err
}
k.mu.Lock()
defer k.mu.Unlock()
return k.rev, slices.Clone(k.entries), k.at != 0, nil
}
// Manifest is the manifest of that digest (as a module's value names it) the held revision's entries carry.
func (k *KeptTrunk) Manifest(digest string) (string, error) {
k.mu.Lock()
defer k.mu.Unlock()
raw, ok := k.manifests[digest]
if !ok {
return "", fmt.Errorf("the newest trunk revision's entries carry no manifest %s", digest)
}
return raw, nil
}
// Heard is the seat's event catalog-revision: one of a trunk revision newer than the one held wakes one re-read of the
// objects; any other event is no change. A failed re-read is an error, so the event is offered again.
func (k *KeptTrunk) Heard(ctx context.Context, o Objects, e meshcatalog.CatalogRevisionEvent) error {
if e.Kind != "trunk" || !isTrunkBranch(e.Branch) {
return nil
}
k.read.Lock()
defer k.read.Unlock()
k.mu.Lock()
held := k.started && e.Revision <= k.at
k.mu.Unlock()
if held {
return nil
}
return k.reread(ctx, o)
}
// read is the view's modules and nodes, and the revision they were read at; none kept yet is ErrNoRevision.
@@ -518,36 +620,37 @@ func (t TrunkStates) read(nodes bool) (Store, int64, error) {
return mods, at, err
}
// view is the view of the newest trunk revision: the one kept when it is still the newest, else read whole and kept.
// view is the view of the trunk: the one held while the follower's half is still at its revision, else made from the
// half and held.
func (t TrunkStates) view() (modules, nodes Store, at int64, err error) {
ctx := context.Background()
if t.Kept == nil {
return TrunkView(ctx, t.Objects)
return TrunkView(context.Background(), t.Objects)
}
k := t.Kept
newest, any, err := newestOf(ctx, t.Objects, TrunkBranch)
if err != nil || !any {
return newOverlay(nil), newOverlay(nil), 0, err
return t.Kept.current(context.Background(), t.Objects)
}
// current is the view held; before the start's read it reads the newest revision once.
func (k *KeptTrunk) current(ctx context.Context, o Objects) (Store, Store, int64, error) {
k.mu.Lock()
started := k.started
k.mu.Unlock()
if !started {
k.read.Lock()
k.mu.Lock()
started = k.started
k.mu.Unlock()
var err error
if !started {
err = k.reread(ctx, o)
}
k.read.Unlock()
if err != nil {
return nil, nil, 0, err
}
}
k.mu.Lock()
defer k.mu.Unlock()
if k.mods != nil && k.at == newest.Revision {
return k.mods, k.held, k.at, nil
}
started := time.Now()
mods, held, at, err := TrunkView(ctx, t.Objects)
if err != nil {
if k.Say != nil {
k.Say("the view of the trunk: revision %d cannot be read after %s: %v", newest.Revision, time.Since(started).Round(time.Millisecond), err)
}
return nil, nil, 0, err
}
if k.Say != nil {
n, _ := mods.Keys(ctx)
k.Say("the view of the trunk: read revision %d (was %d), %d modules, in %s", at, k.at, len(n), time.Since(started).Round(time.Millisecond))
}
k.mods, k.held, k.at = mods, held, at
return mods, held, at, nil
return k.mods, k.held, k.at, nil
}
// state is the source of a state by its name: modules and graph from the revision, their marks at its number, every
@@ -588,6 +691,57 @@ func (r revisionSource) Get(key string) (*views.Entry, bool, error) {
return &views.Entry{Value: value, Revision: uint64(at)}, true, nil
}
// Values is every value of the modules (or the module graph) of the newest trunk revision from one read of it: a seat verb
// that answers from the whole view asks which revision is newest once, never once per key (each ask lists the whole
// catalog-revisions bucket: 6 s for the 110 modules of a bucket of 11 000 objects, and 35 s with 12 callers at once).
func (r revisionSource) Values() (map[string][]byte, error) {
s, _, err := r.t.read(r.nodes)
if err != nil {
return nil, err
}
ctx := context.Background()
keys, err := s.Keys(ctx)
if err != nil {
return nil, err
}
out := make(map[string][]byte, len(keys))
for _, k := range keys {
value, _, found, err := s.Get(ctx, k)
if err != nil {
return nil, err
}
if found {
out[k] = value
}
}
return out, nil
}
// SourceValues is every key's value of a source: in one read where the source offers it (the view of the trunk, a
// watched copy of a state), else a get per key.
func SourceValues(src views.Source) (map[string][]byte, error) {
if all, ok := src.(interface {
Values() (map[string][]byte, error)
}); ok {
return all.Values()
}
keys, err := src.Keys()
if err != nil {
return nil, err
}
out := make(map[string][]byte, len(keys))
for _, k := range keys {
e, ok, err := src.Get(k)
if err != nil {
return nil, fmt.Errorf("%s: %w", k, err)
}
if ok && e != nil {
out[k] = e.Value
}
}
return out, nil
}
// Page is the values in key order after a key from one read of the revision, as many as fit maxBytes and at least one
// (views.Pager): a first view's page reads the revision once, never once per key.
func (r revisionSource) Page(after string, maxBytes int, _ time.Duration) ([]views.KeyedEntry, string, error) {
@@ -60,7 +60,7 @@ func (ix *Indexer) FingerprintsAgain(ctx context.Context, module string) TakenIn
ix.freshMu.Unlock()
// The path of a merge's recompute: the modules, then the candidates their fingerprints reach (recomputeWith), then the
// pending merges a changed fingerprint settles (PendingAgain, as after a toolchain report).
err = ix.recomputeWith(ctx, nil, nil)
err = ix.recomputeWith(ctx, nil, nil, "")
ix.freshMu.Lock()
ix.fresh = nil
ix.freshMu.Unlock()
@@ -0,0 +1,146 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-sdk/go/views"
)
// listedObjects counts the listings of its bucket.
type listedObjects struct {
*memObjects
n atomic.Int64
fail atomic.Bool
}
func (l *listedObjects) Names(ctx context.Context) ([]string, error) {
l.n.Add(1)
if l.fail.Load() {
return nil, errors.New("the bucket does not answer")
}
return l.memObjects.Names(ctx)
}
func keptDigest(name string) string {
return "sha256:" + fmt.Sprintf("%064x", len(name))
}
// A-487, A-487b, A-487c: the view of the trunk is read from the bucket once at the start and once for each heard event of a newer
// trunk revision; 100 reads between events list the bucket zero times.
func TestTheKeptTrunkIsReadAtTheStartAndOncePerHeardRevision(t *testing.T) {
objects := &listedObjects{memObjects: newMemObjects(0)}
keep := func(revision int64, modules ...string) {
t.Helper()
rev, entries := meshcatalog.CatalogRevision{Revision: revision, Kind: "trunk"}, map[string][]byte{}
for _, name := range modules {
data, _ := json.Marshal(meshcatalog.CatalogEntry{Module: name, Repository: catalogueRepo, None: "not built", ManifestDigest: keptDigest(name), Manifest: map[string]any{"module": name}})
digest := fmt.Sprintf("%d-%s", revision, name)
entries[digest] = data
rev.Entries = append(rev.Entries, meshcatalog.EntryRef{Module: name, Digest: digest})
}
if err := KeepRevision(context.Background(), objects, rev, entries); err != nil {
t.Fatal(err)
}
}
said := []string{}
kept := &KeptTrunk{Say: func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }}
modules := TrunkStates{Objects: objects, Under: func(string) views.Source { return nil }, Kept: kept}.State("modules")
keep(1, "avahi")
if err := kept.Reread(context.Background(), objects); err != nil { // the start's read
t.Fatal(err)
}
listed := objects.n.Load()
for i := 0; i < 50; i++ {
if _, _, err := modules.Get("avahi"); err != nil {
t.Fatal(err)
}
if _, err := modules.Keys(); err != nil {
t.Fatal(err)
}
}
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("100 reads listed the bucket %d times, want none", got)
}
// a newer revision kept is no reason to read: only the event says it
keep(2, "avahi", "ca-trust")
listed = objects.n.Load()
if keys, _ := modules.Keys(); len(keys) != 1 {
t.Errorf("the view holds %v before the event, want revision 1's one module", keys)
}
for _, e := range []meshcatalog.CatalogRevisionEvent{{Kind: "candidate", Revision: 9}, {Kind: "trunk", Branch: "other", Revision: 9}, {Kind: "trunk", Revision: 1}} {
if err := kept.Heard(context.Background(), objects, e); err != nil {
t.Errorf("%+v: %v", e, err)
}
}
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("events of a candidate, another branch and the revision held listed the bucket %d times, want none", got)
}
if err := kept.Heard(context.Background(), objects, meshcatalog.CatalogRevisionEvent{Kind: "trunk", Branch: "main", Revision: 2}); err != nil {
t.Fatal(err)
}
if got := objects.n.Load() - listed; got != 1 {
t.Errorf("a heard catalog-revision listed the bucket %d times, want exactly one re-read", got)
}
if e, ok, err := modules.Get("ca-trust"); err != nil || !ok || e.Revision != 2 {
t.Errorf("after the event ca-trust = %+v %v (%v), want it at revision 2", e, ok, err)
}
// A-487c: the manifests by digest and the verb revision asked of the trunk's newest answer from the view held.
listed = objects.n.Load()
if raw, err := kept.Manifest(meshcatalog.BareManifestDigest(keptDigest("ca-trust"))); err != nil || raw != `{"module":"ca-trust"}` {
t.Errorf("the held manifest = %q (%v), want the entry's manifest", raw, err)
}
if a, err := Revision(context.Background(), objects, kept, "avahi", "avahi", meshcatalog.RevisionInput{}); err != nil || a.Revision.Revision != 2 || len(a.Entries) != 2 {
t.Errorf("revision of the newest answered %+v (%v), want revision 2 whole", a, err)
}
if got := objects.n.Load() - listed; got != 0 {
t.Errorf("the manifest and the verb revision listed the bucket %d times, want none", got)
}
if len(said) != 2 || !strings.Contains(said[1], "read revision 2 (was 1), 2 modules, in ") {
t.Errorf("said %q, want each read with its revision, module count and time", said)
}
// A-487b: a re-read that fails is said loudly, the view held stays, and the next heard event reads again.
keep(3, "avahi", "ca-trust", "dbus")
objects.fail.Store(true)
if err := kept.Heard(context.Background(), objects, meshcatalog.CatalogRevisionEvent{Kind: "trunk", Revision: 3}); err == nil {
t.Error("a re-read that failed was taken")
}
if last := said[len(said)-1]; !strings.Contains(last, "BEHIND") {
t.Errorf("a failed re-read said %q, want it said loudly", last)
}
if keys, _ := modules.Keys(); len(keys) != 2 {
t.Errorf("the view held after a failed re-read has %v, want revision 2's two modules", keys)
}
objects.fail.Store(false)
if err := kept.Heard(context.Background(), objects, meshcatalog.CatalogRevisionEvent{Kind: "trunk", Revision: 3}); err != nil {
t.Fatal(err)
}
if keys, _ := modules.Keys(); len(keys) != 3 {
t.Errorf("the next heard event left the view at %v, want revision 3's three modules", keys)
}
}
// A-487: the verb module reads the one module it is asked.
func TestTheVerbModuleReadsOneModule(t *testing.T) {
c := CatalogRead{
Modules: func() (map[string]meshcatalog.ModulesByModuleValue, error) {
return nil, errors.New("every module was read")
},
Module: func(name string) (meshcatalog.ModulesByModuleValue, bool, error) {
return meshcatalog.ModulesByModuleValue{Manifest: "d1"}, name == "avahi", nil
},
Manifest: func(string) (string, error) { return `{"module":"avahi"}`, nil },
}
if one, err := c.CatalogModuleAnswer("avahi"); err != nil || one.Module.Module != "avahi" {
t.Errorf("module avahi = %+v (%v)", one, err)
}
if _, err := c.CatalogModuleAnswer("nope"); err == nil || !strings.Contains(err.Error(), "holds no module nope") {
t.Errorf("module nope = %v, want the trunk holds none", err)
}
}
@@ -144,7 +144,7 @@ func (ix *Indexer) applyMoving(ctx context.Context, f forgemsg.Forge, drop bool,
ix.logf("%s at %s: its packages are not published: %v", f.Repository, f.Merge, err)
}
// What this merge moved reaches the modules standing on it (a base's tree, build-agent's fingerprint), in the graph's order.
return w, ix.recomputeWith(ctx, overlay, w)
return w, ix.recomputeWith(ctx, overlay, w, f.Repository)
}
// baseFP answers the new-form fingerprint of a base module as modules.<base> holds it, or why it has none.
@@ -13,7 +13,6 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/nodebuildgo"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshdeliveryclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
@@ -23,13 +22,10 @@ import (
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/judge"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/graph"
)
// The take-in step (novox/hq ADR 0633): a source revision is taken in only by the indexer it names. A merged message is
// kept in the indexer's own state pending, one key per merge, until the take-in that ends it; a revision naming another
// build of the indexer writes only what building and moving that indexer needs, and the rest stays pending for it.
// The take-in step (novox/hq ADR 0633, A-476): a merged message is kept in the indexer's own state pending, one key per
// merge, until the take-in that ends it; a revision is taken in by the running indexer whatever build of the indexer it names.
const (
// PendingState is the indexer's state of held merges: the bucket mesh-delivery-indexer_pending, keyed
// takein.PendingKey, so the key order is the merge order. An old `state` entry (the indexer is foundation, ADR
@@ -48,10 +44,6 @@ var errClaimLost = errors.New("another take-in claimed the merges first, so this
// runs such a build, so the revision waits until a later merge gives it one or a hand take-in (indexer_take_in_pending).
const NoBuild = "no build: "
// ErrOwnBuildUnknown is what a merged message answers while the indexer cannot learn the build it runs (ADR 0633 §1):
// the message is left unacknowledged, so the bus delivers it again, and nothing is written.
var ErrOwnBuildUnknown = errors.New("the build of the indexer this program runs is not known")
// PendingRef is what a pending value keeps of its merged message instead of a copy (a merge of a repository with many
// modules is about 0.5 MB, over the state's 256 KiB value cap; issue 1585): the small fields the waits are worked out
// from. The message itself is read from the forge bridge's stream by Pending.Seq when a take-in needs it.
@@ -98,8 +90,6 @@ type Pending struct {
// Held is why a ready merge is not taken in although its revision names this build: objects full (HeldObjectsFull),
// until room is said (RoomAgain).
Held string `json:"held,omitempty"`
// Validation is the revision's validation (design 47 G §713, A-247), on the revision's newest value alone.
Validation *takein.Validation `json:"validation,omitempty"`
}
// PendingEntry is one key of the pending state as read: its key, revision, value and the merged message decoded. For a
@@ -389,26 +379,13 @@ func (ix *Indexer) pendingOnBus(ctx context.Context) bool {
}
// ownBuildNow is the build this program runs, as FixOwnBuild fixed it at the start: empty while unknown, and never asked again.
// It names only the writer of what is written (LastWriter); nothing waits for it (A-476 b).
func (ix *Indexer) ownBuildNow(context.Context) string {
ix.ownMu.Lock()
defer ix.ownMu.Unlock()
return ix.ownFP
}
// sayOwnBuildUnknown says, for a merge this process cannot take in because it does not know its own build (ADR 0633 §1), a
// take-in.pending that names why and since when, so the watchman's take-in-waits bound covers it. The merge is not written to
// pending; the stream delivers it again to the process that knows its build.
func (ix *Indexer) sayOwnBuildUnknown(ctx context.Context, seq uint64, f forgemsg.Forge) {
ix.ownMu.Lock()
since, why := ix.ownSince, ix.ownWhy
ix.ownMu.Unlock()
if since.IsZero() { // never fixed: the start did not run (no engine wired)
since, why = ix.now(), "the build this program runs was never asked"
}
ix.sayTakeIn(ctx, takein.TakeIn{Kind: takein.TakeInPending, Key: takein.PendingKey(seq), What: takein.TakeInRevision,
Merges: []takein.Merge{{Repository: f.Repository, Number: f.Number, Merge: f.Merge}}, Since: since, Unknown: why})
}
// sayTakeIn publishes a take-in event and keeps what it said on the entries, so indexer_pending reads the last event of
// each wait from the state. A publish that fails is said in the log: the next evaluation says it again.
func (ix *Indexer) sayTakeIn(ctx context.Context, t takein.TakeIn) bool {
@@ -456,10 +433,6 @@ func (ix *Indexer) Arrive(ctx context.Context, seq uint64, f forgemsg.Forge) (Ap
return a, "", err
}
own := ix.ownBuildNow(ctx)
if own == "" {
ix.sayOwnBuildUnknown(ctx, seq, f)
return Applied{}, "", ErrOwnBuildUnknown
}
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
key := takein.PendingKey(seq)
@@ -502,9 +475,6 @@ func (ix *Indexer) EndGroup(ctx context.Context, end forgemsg.ForgeGroupEnd) (Ap
return Applied{}, "no pending state: the group's merges were taken in one by one", nil
}
own := ix.ownBuildNow(ctx)
if own == "" {
return Applied{}, "", ErrOwnBuildUnknown
}
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
entries, err := ReadPending(ctx, ix.Pending, ix.logf)
@@ -677,21 +647,15 @@ func (ix *Indexer) takenAlready(ctx context.Context, m forgemsg.GroupMerge) (boo
}
// TakeInAtStart is the take-in at the start (ADR 0633 §2), before the consumer on the forge bridge's stream starts:
// it evaluates pending once with the build this program runs, taking in the whole revisions that name this build; a
// group still being gathered stays, and a revision naming yet another build is held for that one. A claim left by an
// indexer that ran before is void here. When the build is not known (the node-engine did not answer), it waits for
// nothing: it says so and the consumer starts all the same; a merged message is then left unacknowledged until the
// build is known (clause 1), and its redelivery, or a toolchain report (ToolchainReported), asks again. No timer.
// it evaluates pending once, taking in the whole revisions whatever build of the indexer they name (A-476 b); a group
// still being gathered stays. A claim left by an indexer that ran before is void here. The build this program runs, known
// or not, only names the writer. No timer.
func (ix *Indexer) TakeInAtStart(ctx context.Context) (Applied, string, error) {
if ix.Pending == nil || !ix.pendingOnBus(ctx) {
// Nothing can wait in a state not on the bus: the start asks no build and does not wait for one.
return Applied{}, "the pending state is not on the bus yet: nothing waits", nil
}
own := ix.ownBuildNow(ctx)
if own == "" {
ix.logf("take-in: the build this program runs is not known at its start, so nothing pending is taken in now; the forge consumer starts, and a merged message left unacknowledged asks it again")
return Applied{}, "the build this program runs is not known yet: nothing pending is taken in at the start", nil
}
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
a, words, err := ix.evaluate(ctx, own, true)
@@ -699,8 +663,8 @@ func (ix *Indexer) TakeInAtStart(ctx context.Context) (Applied, string, error) {
return a, "the pending state is not on the bus yet: nothing waits", nil
}
if err == nil && len(a.Written) == 0 {
// A start that takes nothing in says so, with the build it learnt (the wait's own build is in the line holdFor says).
ix.logf("take-in: at the start, this program runs build %s and took nothing in: %s", own, words)
// A start that takes nothing in says so.
ix.logf("take-in: at the start, this program (build %q) took nothing in: %s", own, words)
}
return a, words, err
}
@@ -731,16 +695,12 @@ func (ix *Indexer) UnreadAgain(ctx context.Context) error {
}
// PendingAgain evaluates pending again after something a fingerprint is computed from changed, a toolchain report
// (ADR 0633 §1, §3), or the verdicts state bound (issue 1966): a revision that now names the running build is taken in, and one that names another build is held
// for that one. It does nothing while the pending state is not on the bus or the build this program runs is not known.
// (ADR 0633 §3), or the verdicts state bound (issue 1966): what waits is taken in. It does nothing while the pending state is not on the bus.
func (ix *Indexer) PendingAgain(ctx context.Context) error {
if ix.Pending == nil || !ix.pendingOnBus(ctx) {
return nil
}
own := ix.ownBuildNow(ctx)
if own == "" {
return nil
}
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
_, words, err := ix.evaluate(ctx, own, false)
@@ -786,31 +746,37 @@ func (ix *Indexer) evaluate(ctx context.Context, own string, start bool) (Applie
if len(lost) > 0 {
return Applied{}, ix.waitingLine(w), ix.holdLost(ctx, w.Ready, lost, reasons, unread)
}
if wait, words, err := ix.validationGate(ctx, ready, own); wait || err != nil {
return Applied{}, words, err
}
names, dry, err := ix.computeOver(ctx, ready)
dry, err := ix.computeOver(ctx, ready)
if err != nil {
return Applied{}, "", err
}
if words, refused, err := ix.refuseCycle(ctx, ready, dry); refused || err != nil {
return Applied{}, words, err
}
if names == own {
a, err := ix.takeInWhole(ctx, ready, own, "")
if errors.Is(err, errClaimLost) {
return a, "claimed by another take-in", nil
}
if errors.Is(err, errVerdictsNotYet) {
return a, "the verdicts state is not on the bus yet: the revision waits for it, taken in when it binds", nil
}
if errors.Is(err, ErrObjectsFull) {
ix.holdFull(ctx, ready)
return a, ix.waitingLine(w) + "; held: " + HeldObjectsFull, nil
}
return a, ix.waitingLine(w), err
// The indexer takes in with the build it runs, whatever build of the indexer the revision names (A-476 b).
a, err := ix.takeInWhole(ctx, ready, own, "")
if errors.Is(err, errClaimLost) {
return a, "claimed by another take-in", nil
}
return Applied{}, ix.waitingLine(w), ix.holdFor(ctx, ready, names, own, dry)
if errors.Is(err, errVerdictsNotYet) {
return a, "the verdicts state is not on the bus yet: the revision waits for it, taken in when it binds", nil
}
if errors.Is(err, ErrObjectsFull) {
ix.holdFull(ctx, ready)
return a, ix.waitingLine(w) + "; held: " + HeldObjectsFull, nil
}
return a, ix.waitingLine(w), err
}
// newestOfRevision is the index in ready of the revision's newest merge.
func newestOfRevision(ready []PendingEntry) int {
n := 0
for i, e := range ready {
if e.Value.Seq > ready[n].Value.Seq {
n = i
}
}
return n
}
// refuseCycle refuses the revision whose catalog graph would hold a cycle with a build edge the take-in refuses (design 47 G
@@ -951,7 +917,6 @@ func (ix *Indexer) dry() *dryRun {
x := &Indexer{Module: ix.Module, Manifests: ix.Manifests, Packages: ix.Packages, JudgeID: ix.JudgeID, Files: ix.Files, GoSum: ix.GoSum,
GoMod: ix.GoMod, Now: ix.Now, Builds: ix.Builds, Named: ix.Named, Trunk: ix.Trunk, BranchHead: ix.BranchHead, Read: ix.Read, Contains: ix.Contains,
ReadOpen: nil, OwnBuild: nil, Message: ix.Message, SayRefused: nil, SayAccepted: nil, SayTakeIn: nil, Pending: nil,
ValidationAsks: nil, AskVerb: nil, // a dry run asks no validation (§713)
// The registry milestone's fields: what a take-in would write on the bus stays out of a dry run, so the states of the
// requirements, findings, seat definitions, holds and their bounds are not overlaid, nothing is said, and no hold is
// weighed (Holds and Counted nil hold nothing); ReadFile only reads.
@@ -991,16 +956,15 @@ func (ix *Indexer) dry() *dryRun {
}
// computeOver applies the ready merges, in merge order, to a dry indexer, as one overlay: each later merge's trees
// govern (ADR 0633 §3). It answers the fingerprint the indexer's own module has once they are applied (none is "",
// with its reason), and the dry run, whose overlays hold what the take-in would write. A merge that can never be read
// govern (ADR 0633 §3). It answers the dry run, whose overlays hold what the take-in would write. A merge that can never be read
// is left out, as its take-in will drop it.
func (ix *Indexer) computeOver(ctx context.Context, ready []PendingEntry) (string, *dryRun, error) {
func (ix *Indexer) computeOver(ctx context.Context, ready []PendingEntry) (*dryRun, error) {
d := ix.dry()
// The manifests are judged against the seat definitions the revision carries (seatsof.go, ADR 0633 §1), so the indexer
// it names is not refused for a verb or an event the built-in list lacks.
seats, err := ix.revisionSeats(ctx, ready)
if err != nil {
return "", d, fmt.Errorf("reading the seat definitions of %s: %w", ready[0].Key, err)
return d, fmt.Errorf("reading the seat definitions of %s: %w", ready[0].Key, err)
}
var applyErr error
judge.WithSeats(seats, func() {
@@ -1018,24 +982,9 @@ func (ix *Indexer) computeOver(ctx context.Context, ready []PendingEntry) (strin
}
})
if applyErr != nil {
return "", d, applyErr
return d, applyErr
}
key, _ := meshcatalogclient.ModulesKey(ix.Module)
value, _, found, err := d.ix.Modules.Get(ctx, key)
if err != nil {
return "", d, err
}
if !found {
return NoBuild + "the indexer holds no record of itself", d, nil
}
m, err := meshcatalogclient.DecodeModule(value)
if err != nil {
return "", d, nil
}
if m.Fp2 == "" {
return NoBuild + m.None2, d, nil
}
return m.Fp2, d, nil
return d, nil
}
// claim marks the entries as taken in by this take-in, by compare-and-set on the revisions read. A lost one (another
@@ -1238,8 +1187,7 @@ func (ix *Indexer) nextSource(branch string) int64 {
// takenWaits are the taken events of a take-in: every wait the entries were said under (saidWaits), every wait
// indexer_pending names the claimed entries by now (WaitsOf: a revision by its oldest merge's key, a group by its g. key) and
// the key of every claimed entry, since indexer_pending named it by that key earlier when it waited behind a group. So a wait
// the watchman learned of without the indexer marking it (seeded from indexer_pending, or said by the own-build-unknown
// path) ends too (design 47, Taken in by the indexer it names). No key is said twice.
// the watchman learned of without the indexer marking it (seeded from indexer_pending) ends too. No key is said twice.
func takenWaits(entries []PendingEntry) []takein.TakeIn {
out := saidWaits(entries)
said := map[string]bool{}
@@ -1298,83 +1246,6 @@ func saidWaits(entries []PendingEntry) []takein.TakeIn {
return out
}
// holdFor keeps the ready merges pending for the indexer build they name and writes only what building and moving
// that indexer needs (ADR 0633 §1): the indexer's own module entry, its key of the module graph and the verdict on its
// manifest, from the dry run; the manifests and the packages are written by the dry run itself. The revision is said
// pending under its oldest merge's key, and a wait it was said under before by another key is said taken.
func (ix *Indexer) holdFor(ctx context.Context, ready []PendingEntry, names, own string, d *dryRun) error {
key, _ := meshcatalogclient.ModulesKey(ix.Module)
if err := copyKey(ctx, d.modules, ix.Modules, key); err != nil {
return fmt.Errorf("the indexer's own module entry: %w", err)
}
if gk, err := graph.Key(ix.Module); err == nil {
if err := copyKey(ctx, d.graph, ix.Graph, gk); err != nil && !errors.Is(err, process.ErrNotYet) {
return fmt.Errorf("the indexer's key of the module graph: %w", err)
}
}
// Why the manifest of the indexer the revision names was refused, when it was (judged with the seat definitions the
// revision carries): said with the wait, as a merge the forge bridge could not give is (holdLost).
refused := ""
if value, _, found, err := d.ix.Modules.Get(ctx, key); err == nil && found {
if m, err := meshcatalogclient.DecodeModule(value); err == nil && m.Manifest != "" {
if vk, err := meshcatalogclient.VerdictsKey(m.Manifest); err == nil {
if err := copyKey(ctx, d.verdicts, ix.Verdicts, vk); err != nil && !errors.Is(err, process.ErrNotYet) {
return fmt.Errorf("the verdict on the indexer's manifest: %w", err)
}
if vv, _, ok, err := d.ix.Verdicts.Get(ctx, vk); err == nil && ok {
if v, err := meshcatalogclient.DecodeVerdict(vv); err == nil && meshcatalogclient.VerdictRefuses(v) {
refused = strings.Join(v.Problems, "; ")
}
}
}
}
}
// The indexer's own entry the held revision names is kept as the trunk revision after the newest, every other entry as
// it was (milestone 47/G W4.7: a merged revision naming a newer indexer replaces only the indexer's entry).
if err := ix.KeepView(ctx); err != nil {
return fmt.Errorf("the revision with the indexer's own entry: %w", err)
}
t := revisionWait(takein.TakeInPending, ready, names)
t.Refused = refused
var ended, unmarked []PendingEntry
said, changed := false, false
for _, e := range ready {
switch {
case e.Value.Said == t.Key && e.Value.SaidKind == takein.TakeInPending:
said = true
changed = changed || e.Value.WaitsFor != names || e.Value.Refused != refused
if e.Value.WaitsFor != names || e.Value.Refused != refused {
unmarked = append(unmarked, e)
}
case e.Value.Said != "" && e.Value.SaidKind == takein.TakeInPending:
ended = append(ended, e)
unmarked = append(unmarked, e)
default:
unmarked = append(unmarked, e)
}
}
for _, old := range saidWaits(ended) {
ix.sayTakeIn(ctx, old)
}
if said && !changed {
// Nothing to say on the bus, but the start decided nothing and a reader of the journal sees why: the build learnt, the build the wait names.
ix.logf("take-in: %d merges, oldest %s, still name the indexer build %s, this program runs %s: they wait, as already said", len(ready), t.Key, names, own)
}
if !said || changed {
ix.logf("take-in: %d merges, oldest %s, name the indexer build %s, not %s, which runs: they wait for it", len(ready), t.Key, names, own)
if refused != "" {
ix.logf("take-in: the manifest of the indexer they name is refused, judged with the seat definitions they carry, so no such indexer runs: %s", refused)
}
if !ix.sayTakeIn(ctx, t) {
ix.markSaid(ctx, unmarked, nil, func(p *Pending) { p.WaitsFor, p.LastWriter, p.Refused = names, own, refused })
return nil
}
unmarked = ready
}
ix.markSaid(ctx, unmarked, &t, func(p *Pending) { p.WaitsFor, p.LastWriter, p.Refused = names, own, refused })
return nil
}
// copyKey writes what the overlay holds of key to the store, when it wrote or deleted it.
func copyKey(ctx context.Context, from *overlayStore, to Store, key string) error {
if from == nil || to == nil {
@@ -1439,9 +1310,6 @@ func waitOf(entries []PendingEntry, what, behind string) takein.PendingWait {
if e.Value.Refused != "" {
pw.Refused = e.Value.Refused
}
if e.Value.Validation != nil {
pw.Validation = e.Value.Validation
}
if e.Value.Said != "" && e.Value.SaidAt.After(pw.SaidAt) {
pw.Said, pw.SaidAt = "take-in."+e.Value.SaidKind, e.Value.SaidAt
}
@@ -1450,9 +1318,9 @@ func waitOf(entries []PendingEntry, what, behind string) takein.PendingWait {
return pw
}
// TakeInPendingByHand is the hand take-in (indexer_take_in_pending, ADR 0633 §5): the wait named by its key, the
// ready revision (its m. key) or a group gathered so far (its g. key), is taken in with this indexer and its judge,
// whatever build it names, for when the named indexer can never run or the group's end never comes. by and why are
// TakeInPendingByHand is the hand take-in (indexer_take_in_pending, ADR 0633 §5, A-476, principle 4): the pending revision (its
// m. key) or the group gathered so far (its g. key) is taken in now with this indexer and its judge,
// for a revision whose message is unread or lost or held for room, or a group whose end never comes. by and why are
// said in the journal and on the take-in.taken event. What waited behind it is evaluated again after.
func (ix *Indexer) TakeInPendingByHand(ctx context.Context, key, by, why string) TakenIn {
r := TakenIn{Module: key}
@@ -106,6 +106,16 @@ func (b *takeInBed) merge(number int, c, avahi, indexer string) forgemsg.Forge {
return f
}
// arriveHeld arrives f while the verdicts state is not on the bus, so the take-in holds it (ADR 0668 §4): the way a merge is
// left pending now that no merge waits for the indexer build it names (A-476 b).
func (b *takeInBed) arriveHeld(t *testing.T, seq uint64, f forgemsg.Forge) {
t.Helper()
bound := b.ix.Verdicts
b.ix.Verdicts = &memStore{unbound: process.ErrNotYet}
defer func() { b.ix.Verdicts = bound }()
b.arrive(t, seq, f)
}
func (b *takeInBed) arrive(t *testing.T, seq uint64, f forgemsg.Forge) {
t.Helper()
b.publish(t, f)
@@ -176,63 +186,21 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
})
}
})
t.Run("T6 a merge while the own build is unknown writes nothing", func(t *testing.T) {
t.Run("T6 a merge while the own build is unknown is taken in all the same (A-476 b)", func(t *testing.T) {
b := newTakeInBed(t)
b.ix.ownFP, b.ix.ownFixed = "", false // this process is started again and asks the engine, which does not answer
b.ix.OwnBuild = func(context.Context) (string, error) { return "", errors.New("the engine does not answer") }
b.ix.FixOwnBuild(ctx)
b.said = nil
_, _, err := b.ix.Arrive(ctx, 1, b.merge(20, sha('b'), "tree-avahi-2", ""))
if !errors.Is(err, ErrOwnBuildUnknown) || len(b.keys()) != 0 || b.commit(t, "avahi") == sha('b') {
t.Errorf("err %v, pending %v, avahi at %s: want ErrOwnBuildUnknown and nothing written", err, b.keys(), b.commit(t, "avahi"))
if _, _, err := b.ix.Arrive(ctx, 1, b.merge(20, sha('b'), "tree-avahi-2", "tree-indexer-2")); err != nil || len(b.keys()) != 0 || b.commit(t, "avahi") != sha('b') {
t.Errorf("err %v, pending %v, avahi at %s: want the merge taken in at once", err, b.keys(), b.commit(t, "avahi"))
}
// The merge is said pending with why and since when (so the watchman's bound covers it), and indexer_pending says the
// build is unknown, from the own-build state.
if len(b.said) != 1 || b.said[0].Kind != takein.TakeInPending || b.said[0].Key != takein.PendingKey(1) || !strings.Contains(b.said[0].Unknown, "does not answer") ||
b.said[0].Since.IsZero() || b.said[0].WaitsFor != "" {
t.Errorf("said %+v, want one take-in.pending of m.1 saying why the own build is unknown", b.said)
} else if _, err := takein.EncodeTakeIn(b.said[0]); err != nil {
t.Errorf("the unknown-build event is refused by its own contract: %v", err)
if len(b.said) != 1 || b.said[0].Kind != takein.TakeInTaken {
t.Errorf("said %+v, want one take-in.taken and no wait said", b.said)
}
answer, err := PendingTools(b.pending, b.ix.OwnBuildState, nil, nil)[0].Run(map[string]any{})
if a, _ := answer.(takein.PendingAnswer); err != nil || !strings.Contains(a.OwnBuildUnknown, "does not answer") || a.OwnBuildSince.IsZero() {
t.Errorf("indexer_pending = %+v %v, want the build unknown with why and since", answer, err)
}
// Before the pending state is on the bus nothing waits: the start asks no build, and the merge is taken in at once,
// as before ADR 0633, though the build is unknown (TestIndexerWhole found the start waiting on it for ever).
b.ix.Pending = &memStore{unbound: process.ErrNotYet}
if _, _, err := b.ix.TakeInAtStart(ctx); err != nil {
t.Fatal(err)
}
if _, _, err := b.ix.Arrive(ctx, 1, b.merge(20, sha('b'), "tree-avahi-2", "")); err != nil || b.commit(t, "avahi") != sha('b') {
t.Errorf("no pending state: err %v, avahi at %s: want the merge taken in at once", err, b.commit(t, "avahi"))
}
})
t.Run("1966/AC195-2 AC195-3 a merge said pending with its build unknown and taken in on redelivery ends that wait under its key; a revision wait is said once", func(t *testing.T) {
b := newTakeInBed(t)
ownBuild := b.ix.OwnBuild
b.ix.ownFP, b.ix.ownFixed = "", false
b.ix.OwnBuild = func(context.Context) (string, error) { return "", errors.New("the engine does not answer") }
b.ix.FixOwnBuild(ctx)
b.said = nil
if _, _, err := b.ix.Arrive(ctx, 7, b.merge(30, sha('b'), "tree-avahi-2", "")); !errors.Is(err, ErrOwnBuildUnknown) {
t.Fatal(err)
}
b.ix.OwnBuild, b.ix.ownFP, b.ix.ownFixed = ownBuild, "", false
b.ix.FixOwnBuild(ctx)
b.arrive(t, 7, b.merge(30, sha('b'), "tree-avahi-2", "")) // the stream delivers it again
if want := []string{"pending " + takein.PendingKey(7), "taken " + takein.PendingKey(7)}; !slices.Equal(b.events(), want) || len(b.keys()) != 0 {
t.Fatalf("said %v, pending %v: want %v and nothing pending", b.events(), b.keys(), want)
}
// Two merges said pending under the oldest's key (waiting for a named indexer), then taken in together: one taken, under
// that key, and no entry's key said twice.
b.said = nil
b.arrive(t, 8, b.merge(31, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arrive(t, 9, b.merge(32, sha('d'), "tree-avahi-4", "tree-indexer-2"))
b.said = nil
b.arrive(t, 10, b.merge(33, sha('e'), "tree-avahi-5", b.base.Trees["modules/"+testSelf]))
if want := []string{"taken " + takein.PendingKey(8), "taken " + takein.PendingKey(9), "taken " + takein.PendingKey(10)}; !slices.Equal(b.events(), want) || len(b.keys()) != 0 {
t.Errorf("said %v, pending %v: want %v, the wait's key (m.8, said pending) once and each other entry's own", b.events(), b.keys(), want)
if a, _ := answer.(takein.PendingAnswer); err != nil || a.OwnBuild != "" || a.OwnBuildSince.IsZero() || len(a.Waits) != 0 {
t.Errorf("indexer_pending = %+v %v, want no wait and no known build", answer, err)
}
})
t.Run("T7 and T3 a merge alone, an indexer change, a merge behind it and a revert", func(t *testing.T) {
@@ -259,42 +227,28 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Fatalf("alone: avahi %s (was %s), pending %v, said %v: want taken in, said taken under its key (1966/AC195-1)", avahi1, avahi0, b.keys(), b.events())
}
b.said = nil
avahi1, verdicts := b.fp(t, "avahi"), len(b.ix.Verdicts.(*memStore).vals)
// T4: a change to the indexer writes only its entry (with its graph key and verdict); the rest is pending.
avahi1 := b.fp(t, "avahi")
// A-476 b: a merge naming another indexer build is taken in at once, with the build this program runs: the indexer's entry
// and avahi are written, nothing is held and said pending.
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
named := b.fp(t, testSelf)
if named == own || b.fp(t, "avahi") != avahi1 || b.commit(t, "avahi") != sha('b') {
t.Fatalf("indexer change: indexer %s (own %s), avahi %s at %s: want only the indexer written", named, own, b.fp(t, "avahi"), b.commit(t, "avahi"))
if named == own || b.fp(t, "avahi") == avahi1 || b.commit(t, "avahi") != sha('c') || len(b.keys()) != 0 {
t.Fatalf("indexer change: indexer %s (own %s), avahi %s at %s, pending %v: want the whole revision taken in at once", named, own, b.fp(t, "avahi"), b.commit(t, "avahi"), b.keys())
}
if g := graphOf(t, b.ix, testSelf); g.FP != named {
t.Errorf("graph of the indexer names %s, want %s", g.FP, named)
}
if n := len(b.ix.Verdicts.(*memStore).vals); n > verdicts+1 {
t.Errorf("verdicts %d, want at most %d: only the new indexer manifest's", n, verdicts+1)
if want := []string{"taken " + takein.PendingKey(2)}; !slices.Equal(b.events(), want) {
t.Errorf("said %v, want %v: nothing said pending", b.events(), want)
}
if want := []string{takein.PendingKey(2)}; !reflect.DeepEqual(b.keys(), want) || !slices.Equal(b.events(), []string{"pending " + takein.PendingKey(2)}) || b.said[0].WaitsFor != named {
t.Fatalf("pending %v, said %+v: want m.2 pending for %s", b.keys(), b.said, named)
}
// T5: the merge delivered again writes no second key and says nothing again.
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
if len(b.keys()) != 1 || len(b.said) != 1 {
t.Errorf("redelivered: pending %v, said %v", b.keys(), b.events())
}
// A later merge joins: the newest governs and still names the new indexer.
// A later merge is taken in at once too, and a revert to the first indexer.
b.arrive(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
if len(b.keys()) != 2 || b.commit(t, "avahi") != sha('b') || len(b.said) != 1 {
t.Fatalf("behind: pending %v, avahi at %s, said %v: want both pending and nothing said again", b.keys(), b.commit(t, "avahi"), b.events())
}
// A revert brings the indexer back to the running build: everything is taken in, oldest first.
b.arrive(t, 4, b.merge(23, sha('e'), "tree-avahi-5", b.base.Trees["modules/"+testSelf]))
if len(b.keys()) != 0 || b.fp(t, testSelf) != own || b.commit(t, "avahi") != sha('e') {
t.Fatalf("revert: pending %v, indexer %s, avahi at %s: want all taken in", b.keys(), b.fp(t, testSelf), b.commit(t, "avahi"))
}
if want := []string{"pending " + takein.PendingKey(2), "taken " + takein.PendingKey(2), "taken " + takein.PendingKey(3), "taken " + takein.PendingKey(4)}; !slices.Equal(b.events(), want) {
t.Errorf("said %v, want %v", b.events(), want)
}
})
t.Run("IX22 a merged revision naming a newer indexer is the previous catalog graph revision with only the indexer's entry replaced", func(t *testing.T) { // design 47 G §273, §237
t.Run("IX22 a merged revision naming a newer indexer is a catalog graph revision with its entry and the merge's others replaced", func(t *testing.T) { // design 47 G §273, §237
b := newTakeInBed(t)
store := newMemObjects(0)
b.ix.Revisions = store
@@ -305,14 +259,14 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
if err != nil || before.Revision != 1 {
t.Fatalf("the view's revision %+v (%v), want revision 1", before, err)
}
// The merge changes avahi and the indexer: only the indexer is written, the revision is made at once.
// The merge changes avahi and the indexer: both are taken in at once (A-476 b), in a revision after the first.
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
after, _, err := newestTrunk(ctx, store)
if err != nil || after.Revision != 2 {
t.Fatalf("after the merge naming a newer indexer the newest revision is %+v (%v), want revision 2", after, err)
if err != nil || after.Revision <= before.Revision {
t.Fatalf("after the merge naming a newer indexer the newest revision is %+v (%v), want one after revision %d", after, err, before.Revision)
}
if changed := changedEntries(before.Entries, after.Entries); !slices.Equal(changed, []string{testSelf}) {
t.Errorf("the revision differs from the previous in %v, want the indexer's entry alone", changed)
if changed := changedEntries(before.Entries, after.Entries); !slices.Equal(changed, []string{"avahi", testSelf}) && !slices.Equal(changed, []string{testSelf, "avahi"}) {
t.Errorf("the revision differs from the previous in %v, want avahi and the indexer", changed)
}
})
t.Run("P1 a revision that adds a Go module its own indexer uses is computed against that module's packages at its merge", func(t *testing.T) {
@@ -353,71 +307,13 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Fatal(err)
}
b.arrive(t, 2, f)
if len(b.said) != 1 || !strings.HasPrefix(b.said[0].WaitsFor, "src2:") {
t.Errorf("said %+v, want m.2 waiting for the indexer build its merge's packages give, not for a version no take-in has published", b.said)
if len(b.said) != 1 || b.said[0].Kind != takein.TakeInTaken || b.said[0].WaitsFor != "" || len(b.keys()) != 0 {
t.Errorf("said %+v, pending %v, want m.2 taken in at once", b.said, b.keys())
}
if _, ok := b.ix.publishedAll(ctx)[sibling]; !ok {
t.Errorf("the packages the revision publishes at its merge lack %s", sibling)
}
})
t.Run("K1 the indexer a revision names is judged with the seat definitions the revision carries", func(t *testing.T) {
// A successor's manifest invokes a verb of the node-engine's seat that the seat list lacks and the revision's
// definitions add (ADR 0633 §1, §6): refused without them, with its reason said on the wait and in indexer_pending;
// accepted with them, and the revision waits for the build it names.
const verb, contracts = "revision-verb", "modules/seat-contracts"
// The third row's definer is the Delivery domain's contracts module defining a Core seat: not taken (ADR 0636 §5).
for _, row := range []struct {
name string
defined bool
refused bool
module string
}{{"definitions the revision does not carry", false, true, ""}, {"definitions the revision carries", true, false, "seat-contracts"},
{"definitions of a seat outside the carrier's domain", true, true, "mesh-delivery-contracts"}} {
b := newTakeInBed(t)
f := b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2")
f.Manifests = map[string]string{}
for dir, raw := range b.base.Manifests {
f.Manifests[dir] = raw
}
dir := "modules/" + testSelf
f.Manifests[dir] = strings.Replace(f.Manifests[dir], `"module": "`+testSelf+`"`,
`"module": "`+testSelf+`", "invokes": ["seat:node-engine.`+verb+`"]`, 1)
if row.defined {
f.Trees[contracts] = "tree-seat-contracts"
f.Locks = map[string]map[string]string{}
for d, l := range b.base.Locks {
f.Locks[d] = l
}
f.Locks[contracts] = map[string]string{}
f.Manifests[contracts] = `{"module":"` + row.module + `","version":"1.0","build":{"artifacts":[{"name":"go-seats","kind":"package","language":"go","paths":["go"],"series":"0.1","generated-from":"."}]}}`
b.ix.Files = func(_ context.Context, repository, commit string, dirs []string) (map[string]map[string]string, error) {
if repository != f.Repository || commit != f.Merge || !slices.Equal(dirs, []string{contracts + "/seats"}) {
t.Errorf("listed %v of %s at %s", dirs, repository, commit)
}
return map[string]map[string]string{dirs[0]: {"node-engine/seat.json": "1", "node-engine/fixtures/a.json": "2"}}, nil
}
b.ix.ReadFile = func(_ context.Context, _, _, p string) ([]byte, error) {
switch p {
case contracts + "/seats/node-engine/seat.json":
return []byte(`{"format":1,"seat":"node-engine","scope":"node","grain":"seat-list","decision":"d","domain":"Core","verbs":[{"name":"` + verb + `"}]}`), nil
}
return nil, nil
}
}
b.arrive(t, 2, f)
waits := PendingWaits(readPending(t, b))
refused := ""
if len(b.said) > 0 {
refused = b.said[0].Refused
}
if got := strings.Contains(refused, `serves no verb "`+verb+`"`); got != row.refused || (len(waits) == 1 && waits[0].Refused != refused) {
t.Errorf("%s: said %q, indexer_pending %+v: want a refusal naming the verb %v", row.name, refused, waits, row.refused)
}
if _, ok := catalogue.SeatNamed("node-engine"); !ok || judgeSeatsHave("node-engine", verb) {
t.Errorf("%s: the seats are not put back after the dry run", row.name)
}
}
})
t.Run("A-326 a revision whose catalog graph holds a cycle with a build edge no build seat breaks is refused, naming it", func(t *testing.T) {
// avahi's artifact needs the indexer's (kind 2), and the indexer is built on avahi (kind 3): avahi holds no build seat, so
// no running version breaks the cycle, and the revision is refused at take-in naming every artifact on it (design 47 G §49).
@@ -443,15 +339,15 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Errorf("indexer_pending %+v, want the refusal naming the cycle", waits)
}
})
t.Run("T10 the named indexer takes the revision in at its start; a gathered group stays", func(t *testing.T) {
t.Run("T10 a start takes what waits in whatever its build; a gathered group stays", func(t *testing.T) {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
g := b.merge(30, sha('f'), "", "")
g.Repository, g.Trees, g.Manifests, g.Locks, g.Group = "novox/mesh-sdk", map[string]string{}, nil, nil, "feat/x"
g.Members = []meshdeliveryclient.Head{{Repository: "novox/mesh-sdk", Number: 30, Head: g.Head}, {Repository: "novox/mesh-host", Number: 4, Head: sha('9')}}
b.arrive(t, 5, g)
named := b.fp(t, testSelf)
// The new indexer starts; its build is not known (the engine does not answer at its start): a hand ask is answered
b.arriveHeld(t, 5, g)
own, _ := b.ix.OwnBuild(ctx)
// The indexer starts; its build is not known (the engine does not answer at its start): a hand ask is answered
// meanwhile (R5), and the start waits for nothing. It is never asked again, so it stays unknown until it is started again.
mk := func(ask func(context.Context) (string, error)) *Indexer {
n := &Indexer{Module: testSelf, Store: b.ix.Store, Modules: b.ix.Modules, Manifests: b.ix.Manifests, Graph: b.ix.Graph, Verdicts: b.ix.Verdicts,
@@ -462,39 +358,21 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
n.FixOwnBuild(ctx)
return n
}
// A start that decides nothing says so (issue 1921): the build it learnt, the build the held wait names, also when the
// wait was said before; and indexer_pending names the known build and since when.
oldBuild, _ := b.ix.OwnBuild(ctx)
oldStart := mk(func(context.Context) (string, error) { return oldBuild, nil })
b.logs.Reset()
if _, words, err := oldStart.TakeInAtStart(ctx); err != nil || len(b.keys()) != 2 || !strings.Contains(b.logs.String(), "still name the indexer build") ||
!strings.Contains(b.logs.String(), "at the start, this program runs build") {
t.Fatalf("a start that holds: %q %v, journal %q: want the build learnt and the build the wait names said", words, err, b.logs.String())
}
if answer, err := PendingTools(b.pending, b.ix.OwnBuildState, nil, nil)[0].Run(map[string]any{}); err != nil || answer.(takein.PendingAnswer).OwnBuild != oldBuild || answer.(takein.PendingAnswer).OwnBuildSince.IsZero() {
t.Errorf("indexer_pending = %+v %v, want the known build and since", answer, err)
}
asked := 0
next := mk(func(context.Context) (string, error) { asked++; return "", errors.New("not yet") })
// The build this program runs is not known (the engine does not answer at its start): nothing waits for it, so the start
// takes the revision in all the same (A-476 b), and a hand ask names what waits.
next := mk(func(context.Context) (string, error) { return "", errors.New("not yet") })
if r := next.TakeInPendingByHand(ctx, "m.nothing", "lead@shanks", "a test"); r.Stored || !strings.Contains(r.Answer, "nothing waits") {
t.Errorf("a hand ask while the own build is unknown: %+v", r)
t.Errorf("a hand ask for what does not wait: %+v", r)
}
if _, words, err := next.TakeInAtStart(ctx); err != nil || len(b.keys()) != 2 || !strings.Contains(words, "not known") {
t.Fatalf("a start with its build unknown: %q %v, pending %v: want it to answer at once and take nothing in", words, err, b.keys())
}
// The engine answers now, and the same process does not ask it again: it learnt nothing it did not know at its start.
next.OwnBuild = func(context.Context) (string, error) { asked++; return named, nil }
if _, words, err := next.TakeInAtStart(ctx); err != nil || asked != 1 || len(b.keys()) != 2 || !strings.Contains(words, "not known") {
t.Fatalf("the window of a replacement: %q %v, asked %d, pending %v: want the build fixed at the start, never learnt later", words, err, asked, b.keys())
}
// The module started again fixes the build at its start and takes the revision in.
next = mk(func(context.Context) (string, error) { return named, nil })
if _, _, err := next.TakeInAtStart(ctx); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(b.keys(), []string{takein.PendingKey(5)}) || b.commit(t, "avahi") != sha('c') {
t.Errorf("start: pending %v, avahi at %s: want the revision taken in, the group left", b.keys(), b.commit(t, "avahi"))
}
if answer, err := PendingTools(b.pending, b.ix.OwnBuildState, nil, nil)[0].Run(map[string]any{}); err != nil || answer.(takein.PendingAnswer).OwnBuild != own {
t.Errorf("indexer_pending = %+v %v, want the build the original program knows", answer, err)
}
})
t.Run("T8 T9 T11 a group gathered until its end, a merge behind it, one alone, an end again", func(t *testing.T) {
b := newTakeInBed(t)
@@ -544,7 +422,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
if _, words, err := b.ix.EndGroup(ctx, end); err != nil || !strings.Contains(words, "taken in already") {
t.Errorf("an end again: %q %v", words, err)
}
// T11 (the lead's row): a group whose revision changes the indexer: the group's wait is taken, then the revision pending.
// T11: a group whose revision changes the indexer: the group's wait is taken and the revision taken in at once (A-476 b).
b.said = nil
g := member(catalogueRepo, 40, sha('c'))
g.Group, g.Trees = "feat/two", b.merge(0, "", "", "tree-indexer-7").Trees
@@ -553,7 +431,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
if _, _, err := b.ix.EndGroup(ctx, forgemsg.ForgeGroupEnd{Group: "feat/two", Members: []forgemsg.GroupMerge{{Repository: catalogueRepo, Number: 40, Merge: sha('c')}}}); err != nil {
t.Fatal(err)
}
if want := []string{"pending g.feat/two", "taken g.feat/two", "pending " + takein.PendingKey(9)}; !slices.Equal(b.events(), want) {
if want := []string{"pending g.feat/two", "taken g.feat/two", "taken " + takein.PendingKey(9)}; !slices.Equal(b.events(), want) {
t.Errorf("a group naming another indexer: said %v, want %v", b.events(), want)
}
})
@@ -593,11 +471,8 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
}
held := func() *takeInBed {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arrive(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
b.ix.ownMu.Lock()
b.ix.ownFP = b.fp(t, testSelf)
b.ix.ownMu.Unlock()
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
return b
}
// A probe that fails keeps the group's wait and stops nothing else: an unrelated merge is still taken in.
@@ -719,7 +594,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
})
t.Run("T5 two take-ins of one pending revision: the one that loses the claim writes nothing", func(t *testing.T) {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
named := b.fp(t, testSelf)
b.ix.ownMu.Lock()
b.ix.ownFP = named
@@ -774,8 +649,8 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
{"a take-in that fails", func(b *takeInBed) { b.mods.failWrite = once(func(string) bool { return true }) }},
} {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arrive(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
named := b.fp(t, testSelf)
b.ix.ownMu.Lock()
b.ix.ownFP = named
@@ -794,7 +669,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
})
t.Run("W4.6 a full store holds the merge with the reason objects full, says full once, and room takes it in", func(t *testing.T) {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
named := b.fp(t, testSelf)
b.ix.ownMu.Lock()
b.ix.ownFP = named
@@ -826,7 +701,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
// The same take-in into a store it fills to 85 % of its cap says filling once, and is taken in.
kept := store.used()
b2 := newTakeInBed(t)
b2.arrive(t, 2, b2.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b2.arriveHeld(t, 2, b2.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b2.ix.ownMu.Lock()
b2.ix.ownFP = b2.fp(t, testSelf)
b2.ix.ownMu.Unlock()
@@ -845,7 +720,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Run("W4.12 a candidate's half dropped by hand frees its own entries, says room and takes the held merge in; the read tools list it; a recompute that moves nothing is equal", func(t *testing.T) {
bed := func(store *memObjects) *takeInBed {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.ix.ownMu.Lock()
b.ix.ownFP = b.fp(t, testSelf)
b.ix.ownMu.Unlock()
@@ -930,40 +805,10 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Errorf("indexer_recompute = %+v (%v): want equal to the kept revision 2", got, err)
}
})
t.Run("T7 a toolchain report evaluates pending again: the revision then names the running build and is taken in", func(t *testing.T) {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
entries, err := ReadPending(ctx, b.pending, nil)
if err != nil {
t.Fatal(err)
}
// The build the revision names under the next toolchain, worked out on a dry indexer; this one runs it.
x := b.ix.dry().ix
for _, node := range []string{"novox", "ace"} {
key, _ := ToolchainKey(node, "go")
x.PutToolchain("mesh-build-go", key, toolchainValue(t, "go", "go1.27.2", "1"), false)
}
ready, _, _, _ := b.ix.loadNewest(ctx, WaitsOf(entries).Ready)
next, _, err := x.computeOver(ctx, ready)
if err != nil || next == "" || next == b.said[0].WaitsFor {
t.Fatalf("the build under the next toolchain: %q %v (the revision names %s)", next, err, b.said[0].WaitsFor)
}
b.ix.ownMu.Lock()
b.ix.ownFP = next
b.ix.ownMu.Unlock()
reportToolchain(t, b.ix, "mesh-build-go", "novox", "go", "go1.27.2")
if len(b.keys()) != 1 {
t.Fatalf("one node's report: pending %v, want the revision still held (the nodes disagree)", b.keys())
}
reportToolchain(t, b.ix, "mesh-build-go", "ace", "go", "go1.27.2")
if len(b.keys()) != 0 || b.commit(t, "avahi") != sha('c') {
t.Errorf("after the toolchain report: pending %v, avahi at %s: want the revision taken in", b.keys(), b.commit(t, "avahi"))
}
})
t.Run("T12 T13 indexer_pending names the wait; a hand take-in and a drop say what they ended", func(t *testing.T) {
b := newTakeInBed(t)
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arrive(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
b.arriveHeld(t, 2, b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2"))
b.arriveHeld(t, 3, b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2"))
var answer takein.PendingAnswer
for _, tool := range PendingTools(b.pending, nil, nil, nil) {
if tool.Name == PendingTool {
@@ -975,7 +820,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
}
}
if len(answer.Waits) != 1 || answer.Waits[0].Key != takein.PendingKey(2) || !slices.Equal(answer.Waits[0].Holds, []string{testSelf, "avahi"}) ||
answer.Waits[0].Said != takein.TakeInPendingEvent || len(answer.Waits[0].Keys) != 2 {
len(answer.Waits[0].Keys) != 2 {
t.Fatalf("indexer_pending = %+v", answer)
}
// ADR 0668 §6: a later merge covers a pending one only when its commit has the pending merge's as an ancestor.
@@ -990,32 +835,47 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
}
history = "cd"
b.said = nil
bound := b.ix.Verdicts
b.ix.Verdicts = &memStore{unbound: process.ErrNotYet} // m.3 stays held while m.2 is dropped
if r := b.ix.DropPending(ctx, takein.PendingKey(2), "lead@shanks", "replaced by m.3", false); !r.Stored {
t.Fatalf("drop: %+v", r)
}
if len(b.said) < 2 || !b.said[0].Dropped || b.said[0].Key != takein.PendingKey(2) || b.said[0].By != "lead@shanks" || b.said[1].Key != takein.PendingKey(3) {
t.Fatalf("drop said %+v: want m.2 taken (dropped, by) and m.3 pending", b.said)
b.ix.Verdicts = bound
if len(b.keys()) != 1 || b.keys()[0] != takein.PendingKey(3) {
t.Fatalf("pending %v: want m.2 dropped and m.3 pending", b.keys())
}
// Issue 1966 (#650), ADR 0633 §5: the newest pending merge of its repository, which no taken merge covers, is not
// dropped; the answer names the read again. abandon drops it all the same (below).
// dropped; the answer names abandon, which drops it all the same (below).
if r := b.ix.DropPending(ctx, takein.PendingKey(3), "lead@shanks", "lost message", false); r.Stored ||
!strings.Contains(r.Answer, TakeInPendingTool) || len(b.keys()) != 1 {
!strings.Contains(r.Answer, "abandon") || len(b.keys()) != 1 {
t.Fatalf("a drop nothing covers: %+v, pending %v", r, b.keys())
}
// A pending revision is taken in by the tool, by its key (A-476, principle 4).
if _, _, err := NewPendingAsk(takein.PendingKey(3), "x", "lead@shanks", false, false, time.Unix(0, 0)); err != nil {
t.Fatalf("the ask to take a pending revision in: %v", err)
}
if r := b.ix.TakeInPendingByHand(ctx, takein.PendingKey(3), "lead@shanks", "its message was unread"); !r.Stored || len(b.keys()) != 0 || b.commit(t, "avahi") != sha('d') {
t.Fatalf("hand take-in of a revision: %+v, pending %v, avahi at %s", r, b.keys(), b.commit(t, "avahi"))
}
// A delivery group gathered is taken in by hand (A-476 b: only a group is).
b.ix.Pins = newMem()
b.ix.Revisions = newMemObjects(0) // A-448: the taken event names the catalog graph revision the take-in made
gm := b.merge(24, sha('f'), "tree-avahi-6", "")
gm.Group, gm.Head = "feat/h", sha('1')
gm.Members = []meshdeliveryclient.Head{{Repository: catalogueRepo, Number: 24, Head: sha('1')}, {Repository: "novox/mesh-sdk", Number: 25, Head: sha('2')}}
b.arrive(t, 5, gm)
b.said = nil
if r := b.ix.TakeInPendingByHand(ctx, takein.PendingKey(3), "lead@shanks", "the named indexer can never run"); !r.Stored {
if r := b.ix.TakeInPendingByHand(ctx, takein.GroupTakeInKey("feat/h"), "lead@shanks", "the group's end never comes"); !r.Stored {
t.Fatalf("hand take-in: %+v", r)
}
if len(b.keys()) != 0 || b.commit(t, "avahi") != sha('d') || len(b.said) != 1 || b.said[0].Kind != takein.TakeInTaken || b.said[0].By != "lead@shanks" {
if len(b.keys()) != 0 || b.commit(t, "avahi") != sha('f') || len(b.said) < 1 || b.said[0].Kind != takein.TakeInTaken || b.said[0].By != "lead@shanks" {
t.Errorf("hand take-in: pending %v, avahi at %s, said %+v", b.keys(), b.commit(t, "avahi"), b.said)
}
// §476, §493: a hand take-in's taken event carries the branch, the source revision it makes and its entries.
if len(b.said) == 1 {
if len(b.said) >= 1 {
got := b.said[0]
if got.Branch != TrunkBranch || got.Source <= 0 || len(got.Entries) != 1 || got.Entries[0].Module != "avahi" ||
got.Entries[0].Commit != sha('d') || got.Entries[0].Fingerprint == "" {
if got.Branch != TrunkBranch || got.Source <= 0 || len(got.Entries) != 2 || got.Entries[1].Module != "avahi" ||
got.Entries[1].Commit != sha('f') || got.Entries[1].Fingerprint == "" {
t.Errorf("hand take-in said branch %q source %d entries %+v: want main, a source revision and avahi at its commit", got.Branch, got.Source, got.Entries)
} else if newest, any, _ := newestTrunk(ctx, b.ix.Revisions); !any || got.Revision != newest.Revision {
t.Errorf("hand take-in said revision %d, want the revision it made, %d (A-448)", got.Revision, newest.Revision)
@@ -1025,7 +885,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
t.Errorf("the taken event is refused by its own contract: %v", err)
}
}
b.arrive(t, 4, b.merge(23, sha('e'), "tree-avahi-5", "tree-indexer-2"))
b.arriveHeld(t, 4, b.merge(23, sha('e'), "tree-avahi-5", "tree-indexer-2"))
if r := b.ix.DropPending(ctx, takein.PendingKey(4), "lead@shanks", "abandoned on the operator's word", true); !r.Stored || len(b.keys()) != 0 {
t.Errorf("an abandoned drop: %+v, pending %v", r, b.keys())
}
@@ -1062,8 +922,8 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
if _, _, found, _ := b.ix.Store.Get(ctx, key); !found {
t.Fatal("the group has no candidate")
}
b.arrive(t, 2, big(b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2")))
b.arrive(t, 3, big(b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2")))
b.arriveHeld(t, 2, big(b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2")))
b.arriveHeld(t, 3, big(b.merge(22, sha('d'), "tree-avahi-4", "tree-indexer-2")))
raw, _, _, _ := b.pending.Get(ctx, takein.PendingKey(2))
if len(b.keys()) != 2 || len(raw) > 4096 || b.commit(t, "avahi") != sha('b') {
t.Fatalf("a big merge held: pending %v, value %d bytes, avahi at %s: want two small values", b.keys(), len(raw), b.commit(t, "avahi"))
@@ -1139,7 +999,7 @@ func TestTakeIn(t *testing.T) { // 1585/T3-T13 (ADR 0633)
b := newTakeInBed(t)
f := b.merge(21, sha('c'), "tree-avahi-3", "tree-indexer-2")
f.Words, f.Branch = strings.Repeat("x", 300*1024), "feat/g"
b.arrive(t, 2, f) // held by reference, for an indexer build that is not this one
b.arriveHeld(t, 2, f) // held by reference, with the verdicts state not on the bus
b.ix.Taken = newMem()
b.ix.Contains = func(context.Context, string, string, string) (bool, error) { return c.behind, nil }
data, _ := takein.EncodeTaken(takein.Taken{Repository: trunk.RepositoryName(catalogueRepo), Merge: sha('d')})
@@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
@@ -44,15 +43,13 @@ func PendingTools(pending, ownBuild Store, ask func(key string, value []byte) (T
return []stdio.Tool{
{Name: PendingTool,
Description: "What waits for a take-in (ADR 0633): merges the indexer holds in its pending state instead of taking them in. " +
"A revision (what revision, its key its oldest merge's m.<sequence>) waits for the build of the indexer it names " +
"(waits-for): the running indexer wrote only what building and moving that indexer needs, and the new indexer takes " +
"it in at its start. A group (what group, key g.<group>) is a delivery group whose members are gathered until the " +
"A revision (what revision, its key its oldest merge's m.<sequence>) waits only while its message is unread or lost or the " +
"objects are full. A group (what group, key g.<group>) is a delivery group whose members are gathered until the " +
"forge bridge says its end; a merge in one of its members' repositories waits behind it (behind). Each names its " +
"merges, since when it waits, the modules it holds back (holds), the merges whose message the forge bridge's stream no " +
"longer holds (lost: they are not taken in; indexer_drop_pending drops them) and the last take-in event said of it. Empty waits: " +
"every merge is taken in. The watchman raises take-in-waits past 60 minutes for a revision, 30 for a group; " +
"indexer_take_in_pending and indexer_drop_pending steer a wait. own-build says which build the indexer knows it runs and own-build-since since when; own-build-unknown, when set, says the indexer cannot learn the build it " +
"runs (since when): it fixes it once at its start, so every merge waits (a take-in.pending that says so) until the module is started again.",
"indexer_take_in_pending and indexer_drop_pending steer a wait. own-build says which build the indexer knows it runs and own-build-since since when; it only names the writer, and no merge waits for it.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) {
var unread []string
@@ -63,7 +60,7 @@ func PendingTools(pending, ownBuild Store, ask func(key string, value []byte) (T
return nil, err
}
answer := takein.PendingAnswer{Waits: PendingWaits(entries), Unread: unread}
// Whether this indexer knows the build it runs: when it does not, every merge waits, and this says since when and why.
// The build this indexer knows it runs, and since when.
if ownBuild != nil {
if v, _, found, err := ownBuild.Get(context.Background(), OwnBuildKey); err == nil && found {
var o OwnBuild
@@ -71,8 +68,6 @@ func PendingTools(pending, ownBuild Store, ask func(key string, value []byte) (T
answer.OwnBuildSince = o.Since
if o.Known {
answer.OwnBuild = o.FP
} else {
answer.OwnBuildUnknown = o.Why
}
}
}
@@ -80,10 +75,10 @@ func PendingTools(pending, ownBuild Store, ask func(key string, value []byte) (T
return answer, nil
}},
{Name: TakeInPendingTool,
Description: "Take a wait in by hand with the running indexer and its judge (ADR 0633 §5): the revision under its m. key, " +
"whatever build of the indexer it names, or a delivery group under its g. key with the members gathered so far. For " +
"when the named indexer can never run, or a group's end never comes. A hand act: who and why are said in the " +
"indexer's journal and on the take-in.taken event, which clears take-in-waits. indexer_pending names the keys.",
Description: "Take in now a pending revision (its m.<sequence> key), for a merge whose message is unread or lost or that is held " +
"for room, or a delivery group (its g.<group> key) with the members gathered so far, for when its end never comes (ADR 0633 §5). " +
"A hand act: who and why are said in the indexer's journal and on the take-in.taken event, which clears take-in-waits. " +
"indexer_pending names the keys.",
Input: map[string]any{"type": "object", "required": []string{"key", "why"}, "properties": map[string]any{
"key": map[string]any{"type": "string", "description": "the wait's key, m.<sequence> or g.<group>, as indexer_pending names it"},
"why": map[string]any{"type": "string", "description": "why it is taken in by hand, in plain words"}}},
@@ -112,33 +107,3 @@ func PendingTools(pending, ownBuild Store, ask func(key string, value []byte) (T
Run: merge},
}
}
// ValidationTools is indexer_validate_again (class act, design 47 G §713): the tools bundle writes an ask in takeins and the
// indexer asks the revision's validation again; record says the hand act in the hand-act log before it is asked.
func ValidationTools(ask func(key string, value []byte) (TakeInAsk, error), now func() time.Time, record func(what, why string) error) []stdio.Tool {
return []stdio.Tool{{Name: ValidateAgainTool,
Description: "Ask again the validation of a pending revision that could not validate (design 47 G §712, §713): the manifest " +
"validator's run on the build seat node-build-go ended could not run, exited other than 0, or answered not both cases. " +
"The revision is asked under its next try; the answer says the validation's state (the ask, its try, accepted, refused, " +
"or why it could not validate). A revision accepted, refused or whose ask is out is asked nothing: a refusal is asked " +
"again only by a later merge of its branch. A hand act, recorded through mesh-hand-acts before it is asked. " +
"indexer_pending names the keys and shows the validation whole.",
Input: map[string]any{"type": "object", "required": []string{"key", "why"}, "properties": map[string]any{
"key": map[string]any{"type": "string", "description": "the revision's wait key, m.<sequence>, as indexer_pending names it"},
"why": map[string]any{"type": "string", "description": "why it is asked again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
str := func(k string) string { s, _ := args[k].(string); return s }
key, value, err := NewValidateAgainAsk(str("key"), str("why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
if err := record("asked the validation of the pending revision "+strings.TrimSpace(str("key"))+" again by hand", strings.TrimSpace(str("why"))); err != nil {
return nil, fmt.Errorf("the hand act was not recorded through mesh-hand-acts, so nothing was asked: %w", err)
}
a, err := ask(key, value)
if err != nil {
return nil, err
}
return a.Result, nil
}}}
}
@@ -65,9 +65,20 @@ const ToolRunner = "node-tools"
// answered only when the newest trunk revision's entry of its module declares the verb in invokes, and the holder is
// refused too. With no trunk revision kept yet the answer is `none-yet` alone (design 47 G §530), never an empty
// catalog and never an error.
func Revision(ctx context.Context, o Objects, holder, caller string, in meshcatalog.RevisionInput) (meshcatalog.RevisionAnswer, error) {
//
// kept, when not nil, is the trunk's newest revision held in memory: the trunk's newest answers from it with no read of
// the bucket (A-487c); a candidate, a branch and an older number read the bucket.
func Revision(ctx context.Context, o Objects, kept *KeptTrunk, holder, caller string, in meshcatalog.RevisionInput) (meshcatalog.RevisionAnswer, error) {
var out meshcatalog.RevisionAnswer
newest, ok, err := newestTrunk(ctx, o)
var newest meshcatalog.CatalogRevision
var ok bool
var trunkEntries []meshcatalog.CatalogEntry
var err error
if kept != nil {
newest, trunkEntries, ok, err = kept.Newest(ctx, o)
} else {
newest, ok, err = newestTrunk(ctx, o)
}
if err != nil {
return out, fmt.Errorf("the catalog graph revisions cannot be read: %w", err)
}
@@ -75,9 +86,10 @@ func Revision(ctx context.Context, o Objects, holder, caller string, in meshcata
// An answer is told by its fields (design 47 G §530): no trunk revision kept yet is `none-yet`, no error.
return meshcatalog.RevisionAnswer{NoneYet: true}, nil
}
_, trunkEntries, err := ReadRevision(ctx, o, RevisionObject(newest))
if err != nil {
return out, err
if kept == nil {
if _, trunkEntries, err = ReadRevision(ctx, o, RevisionObject(newest)); err != nil {
return out, err
}
}
caller = CallerModule(trunkEntries, caller)
if err := views.Refusal(meshcatalog.Seat+"."+meshcatalog.VerbRevision, RevisionCallers(trunkEntries, holder), caller); err != nil {
@@ -152,13 +164,13 @@ func Revision(ctx context.Context, o Objects, holder, caller string, in meshcata
// RevisionTool is the verb revision as the tool mesh-catalog.revision of the indexer's own runtime (ADR 0663 §1): the
// tool runner describes it with the seat's definition, so it carries no description or input of its own.
func RevisionTool(o Objects, holder string) stdio.Tool {
func RevisionTool(o Objects, kept *KeptTrunk, holder string) stdio.Tool {
return stdio.Tool{Name: meshcatalog.Seat + "." + meshcatalog.VerbRevision, Run: func(args map[string]any) (any, error) {
var in meshcatalog.RevisionInput
if raw, err := json.Marshal(args); err == nil {
_ = json.Unmarshal(raw, &in)
}
return Revision(context.Background(), o, holder, stdio.CallerOf(args), in)
return Revision(context.Background(), o, kept, holder, stdio.CallerOf(args), in)
}}
}
@@ -77,6 +77,8 @@ func (ix *Indexer) ApplySource(ctx context.Context, a meshdelivery.SourceAnswer)
if err := ix.kickCatchUp(ctx, s.Heads); err != nil {
return err
}
// A merge the indexer never took in is taken in at each source revision, and at the start (A-463).
ix.kickBehind(ctx)
// A registered module the forge could not read is read again at each new source revision (issue 1945), in the
// background: a forge that does not answer must not hold this revision. One taken in later is stamped when it is written
// (applyManifests reads srcOf).
@@ -122,7 +124,7 @@ func (ix *Indexer) ApplySource(ctx context.Context, a meshdelivery.SourceAnswer)
}
}
// A module built against the commit the source named as a repository's head (a package no module publishes, A-365) is
// computed again when that head moves: the gate moves the source only after the take-in.
// computed again when that head moves: the gate moves the source on the forge bridge's merged message (A-464), before the take-in, and at the take-in for a head no merged moved.
if movedPackages(held, before, s.Heads) {
if err := ix.Recompute(ctx); err != nil {
return err
@@ -176,6 +178,9 @@ func (ix *Indexer) kickCatchUp(ctx context.Context, heads map[string]string) err
defer ix.catchUpWG.Done()
unread := ix.takeInHeads(ctx, heads)
ix.catchingUp.Store(false)
if ix.behindWaiting.Swap(false) {
defer ix.kickBehind(ctx)
}
if len(unread) > 0 {
ix.logf("catalog graph: the first start's catch-up ended and no first revision is kept: %d repositories were not taken in (%s); taken in again at the next source revision",
len(unread), strings.Join(unread, ", "))
@@ -240,6 +245,9 @@ func (ix *Indexer) CatchUp(ctx context.Context) TakenIn {
}
unread := ix.takeInHeads(ctx, heads)
ix.catchingUp.Store(false)
if ix.behindWaiting.Swap(false) {
defer ix.kickBehind(ctx)
}
if err := ix.KeepView(ctx); err != nil {
out.Answer = "the repositories were taken in but the revision could not be kept: " + err.Error()
return out
@@ -51,9 +51,6 @@ type TakeInAsk struct {
// Abandon, with Drop, drops a pending merge no later merge covers (ADR 0633 §5 asks a later merge; issue 1966, ADR
// 0668 §5): the hand act says why it is abandoned all the same.
Abandon bool `json:"abandon,omitempty"`
// ValidateAgain, with Pending, asks the validation of the revision under that key again (indexer_validate_again, design 47
// G §713): one that could not validate.
ValidateAgain bool `json:"validate-again,omitempty"`
// Merge asks the merge of MergeRepository at this commit to be taken in by hand (indexer_take_in_merge): its merged message
// is asked of the forge bridge and run through the take-in as a delivered one, under its stream sequence Sequence.
Merge string `json:"merge,omitempty"`
@@ -167,20 +164,6 @@ func NewMergeAsk(repository, merge string, sequence uint64, why, by string, at t
return "merge." + merge, data, err
}
// NewValidateAgainAsk is the value indexer_validate_again writes: its key in takeins, `pending.validate-again.<key>`, and the
// value. key is a revision's wait key, m.<sequence>, as indexer_pending names it.
func NewValidateAgainAsk(key, why, by string, at time.Time) (string, []byte, error) {
key = strings.TrimSpace(key)
if !strings.HasPrefix(key, "m.") {
return "", nil, errors.New("name the revision whose validation is asked again by its key, m.<sequence>, as indexer_pending names it")
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why (why): a hand act is recorded through mesh-hand-acts")
}
data, err := json.Marshal(TakeInAsk{V: 1, Pending: key, ValidateAgain: true, Why: strings.TrimSpace(why), By: by, At: at.UTC()})
return "pending.validate-again." + key, data, err
}
// NewPendingAsk is the value indexer_take_in_pending (drop false) or indexer_drop_pending (drop true) writes: its key
// in takeins, `pending.<take-in|drop>.<key>`, and the value. key is a wait's key as indexer_pending names it, for a
// drop one pending merge's; abandon drops one no later merge covers.
@@ -7,14 +7,14 @@
},
"candidate": {
"base": 5,
"fpset": "01c840ec276b",
"fpset": "6ed3e0bd2a45",
"group": "feat/x",
"heads": "dfe9fb8f79e8",
"pulls": [
{
"fps": {
"avahi": {
"fp2": "src2:e8d595aea1b13f1f5578ef1f245dab0d28a351c28079ecdcd2276c63a932d7be",
"fp2": "src2:fc14d6ab7e13fd81e12df0ca9fcdcc536691cb4ea31d7efc24a25b5401efaf67",
"manifest": "8a3856b6c0566d76f2556fc4f46c5251dc7935517bb6bd15809ded7e5546a0d8"
}
},
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/clients/meshcatalogclient"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
sdkstep "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/judge"
@@ -127,24 +128,26 @@ type known struct {
// the order the bucket lists them in; modules that stand on each other have a cycle as their reason. A module is written
// only when its fingerprint or its reason changes, and only if its manifest is still the one computed from. A module with
// no kept inputs (a message that carried no manifest) is left as it is.
func (ix *Indexer) Recompute(ctx context.Context) error { return ix.recomputeWith(ctx, nil, nil) }
func (ix *Indexer) Recompute(ctx context.Context) error { return ix.recomputeWith(ctx, nil, nil, "") }
// recomputeWith is Recompute for a merge as well: the records the merge gives (overlay, by module key) stand in place
// of what the bucket holds, are computed with the others, and are written once, with their final fingerprint or reason
// (issue 1444: a merge writes each module once). A record of the overlay that has no kept inputs is written as it is. w is
// the merge's write (order.go), nil for none; it is told each module written.
func (ix *Indexer) recomputeWith(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite) error {
// the merge's write (order.go), nil for none; it is told each module written. A merge's recompute names its repository
// (A-476 c, A-476a): the modules registered for it are moved, and those of another repository only when they import a package it moved; "" moves every module (a toolchain
// report, a hand recompute).
func (ix *Indexer) recomputeWith(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string) error {
if err := ix.fillPaths(ctx); err != nil {
return err
}
if err := ix.recomputeModules(ctx, overlay, w); err != nil {
if err := ix.recomputeModules(ctx, overlay, w, repository); err != nil {
return err
}
// What the modules moved to reaches the candidates' fingerprints too (a base, the builder, a toolchain).
return ix.RecomputeCandidates(ctx)
}
func (ix *Indexer) recomputeModules(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite) error {
func (ix *Indexer) recomputeModules(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string) error {
if ix.Modules == nil {
return nil
}
@@ -153,11 +156,11 @@ func (ix *Indexer) recomputeModules(ctx context.Context, overlay map[string]mesh
}
ix.recompute.Lock()
defer ix.recompute.Unlock()
return ix.recomputeLocked(ctx, overlay, w)
return ix.recomputeLocked(ctx, overlay, w, repository)
}
// recomputeLocked is recomputeModules with ix.recompute held by the caller (a hold that moves, holds.go).
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite) error {
func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshcatalog.ModulesByModuleValue, w *mergeWrite, repository string) error {
if ix.Manifests == nil {
return ix.writeOverlay(ctx, overlay, nil, w)
}
@@ -298,6 +301,14 @@ func (ix *Indexer) recomputeLocked(ctx context.Context, overlay map[string]meshc
r.tcs, r.pkgs = toolchainsUsed(toolchains), f.versions
}
}
if _, merged := overlay[module]; !merged && repository != "" && !trunk.SameRepository(k.cur.Repository, repository) &&
maps.Equal(k.held.Packages, map[string]string(r.pkgs)) {
// A module registered for another repository is moved by this repository's commit only when it imports a package the
// commit moved (A-476a, ADR 0593 §3); any other stands as held (A-476 c), and so it is for a base of this repository's.
r = result{fp: k.held.Fp2, none: k.held.None2, tcs: k.held.Toolchains, pkgs: sdkstep.PackageVersions(k.held.Packages)}
done[module] = r
return r
}
done[module] = r
k.cur.Inputs2 = &kept
return r
@@ -170,13 +170,13 @@ func TestAgreedToolchainChangesAGoModulesReasonWithNoMerge(t *testing.T) { // 14
reportToolchain(t, ix, "mesh-build-go", "ace", "go", "go1.27.1")
m, _ := moduleOf(t, mods, "avahi")
// L3 (ADR 0603 §2, §3): the agreement moving rewrites the module's toolchain edge, and again when it moves on.
if g := graphOf(t, ix, "avahi"); g.FP != m.Fp2 || g.Toolchains["go"].Version != "go1.27.1" || g.Toolchains["go"].Module != "mesh-build-go" {
t.Errorf("after: graph.avahi = %+v, want the toolchain edge of go1.27.1", g)
if g := graphOf(t, ix, "avahi"); g.FP != m.Fp2 || g.Toolchains["go"].Version != "go1.27" || g.Toolchains["go"].Module != "mesh-build-go" {
t.Errorf("after: graph.avahi = %+v, want the toolchain edge of go1.27", g)
}
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.28.0")
reportToolchain(t, ix, "mesh-build-go", "ace", "go", "go1.28.0")
if g := graphOf(t, ix, "avahi"); g.Toolchains["go"].Version != "go1.28.0" {
t.Errorf("moved on: graph.avahi = %+v, want the toolchain edge of go1.28.0", g)
if g := graphOf(t, ix, "avahi"); g.Toolchains["go"].Version != "go1.28" {
t.Errorf("moved on: graph.avahi = %+v, want the toolchain edge of go1.28", g)
}
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.27.1")
reportToolchain(t, ix, "mesh-build-go", "ace", "go", "go1.27.1")
@@ -283,7 +283,7 @@ func TestNodesReportingDifferentVersionsLeaveTheReasonNamingBoth(t *testing.T) {
t.Fatal(err)
}
m, _ := moduleOf(t, mods, "avahi")
if !strings.Contains(m.None2, "builders disagree") || !strings.Contains(m.None2, "go1.27.1") || !strings.Contains(m.None2, "go1.26.0") {
if !strings.Contains(m.None2, "builders disagree") || !strings.Contains(m.None2, "go1.27") || !strings.Contains(m.None2, "go1.26") {
t.Errorf("none2 = %q, want 'builders disagree' naming both versions", m.None2)
}
}
@@ -304,7 +304,7 @@ func goInputs(t *testing.T, ix *Indexer, mods *memStore, module, builder string,
t.Helper()
return fingerprint.Inputs{Module: module, Tree: "tree-" + module, Builder: builder, Bases: bases,
Locks: map[string]string{"tools": avahiGoSum},
Toolchains: map[string]meshcatalogclient.Toolchain2{"tools": {Language: "go", Version: "go1.27.1", Recipe: "1"}}}
Toolchains: map[string]meshcatalogclient.Toolchain2{"tools": {Language: "go", Version: "go1.27", Recipe: "1"}}}
}
func fp2(t *testing.T, in fingerprint.Inputs) string {
@@ -333,7 +333,7 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Errorf("other: fp2 = %q none2 = %q, want none2 naming the base ts-base", m.Fp2, m.None2)
}
// L3 (ADR 0603 §2): the module graph holds each module's edges as this recompute computed them.
goTool := map[string]graph.Toolchain{"go": {Module: "mesh-build-go", Version: "go1.27.1", Recipe: "1"}}
goTool := map[string]graph.Toolchain{"go": {Module: "mesh-build-go", Version: "go1.27", Recipe: "1"}}
if g := graphOf(t, ix, "avahi"); g.FP != want["avahi"] || !reflect.DeepEqual(g.Bases, []string{"mesh-tools"}) ||
!reflect.DeepEqual(g.Toolchains, goTool) || g.Repository != catalogueRepo || g.Path != "modules/avahi" {
t.Errorf("graph.avahi = %+v", g)
@@ -359,7 +359,7 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Errorf("the indexer's main = %q (%v), want cmd/mesh-delivery-indexer", MainOf(m), err)
}
if e := entries["avahi"]; e.Fp != want["avahi"] || e.Commit == "" || e.ManifestDigest == "" || !reflect.DeepEqual(e.Bases, []string{"mesh-tools"}) ||
e.Toolchains["go"] != (meshcatalog.BuildEdge{Module: "mesh-build-go", Version: "go1.27.1", Recipe: "1"}) {
e.Toolchains["go"] != (meshcatalog.BuildEdge{Module: "mesh-build-go", Version: "go1.27", Recipe: "1"}) {
t.Errorf("the catalog entry of avahi = %+v", e)
}
// W3.1's inputs (the lead's ruling on finding 1): an entry carries what the next revision is computed from (the
@@ -372,7 +372,7 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Errorf("%s read back from its entry = %+v, want %+v", name, back, m)
}
}
if e := entries["avahi"]; e.Inputs == nil || e.Inputs.From == nil || e.Inputs.Toolchains["go"].Version != "go1.27.1" {
if e := entries["avahi"]; e.Inputs == nil || e.Inputs.From == nil || e.Inputs.Toolchains["go"].Version != "go1.27" {
t.Errorf("the catalog entry of avahi carries inputs %+v, want what its fingerprint is computed from", e.Inputs)
}
rev, objects, err := CatalogRevisionOf(meshcatalog.CatalogRevision{Revision: 1, Kind: "trunk", Source: 7}, entries)
@@ -645,22 +645,22 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Fatal(err)
}
}
if got, err := Revision(context.Background(), verbObjects, testSelf, "n1.node-tools", meshcatalog.RevisionInput{}); err != nil || !got.NoneYet || got.Revision != nil || len(got.Entries) != 0 {
if got, err := Revision(context.Background(), verbObjects, nil, testSelf, "n1.node-tools", meshcatalog.RevisionInput{}); err != nil || !got.NoneYet || got.Revision != nil || len(got.Entries) != 0 {
t.Errorf("revision with none kept answered %+v, %v, want none-yet alone with no error (design 47 G §530)", got, err)
}
keep(meshcatalog.CatalogRevision{Revision: 3, Kind: "trunk"}, map[string]bool{"node-tools": true, testSelf: true})
keep(meshcatalog.CatalogRevision{Revision: 4, Kind: "candidate", Group: "feat/x", Heads: "0123456789ab"}, map[string]bool{"node-tools": true, "mesh-grants": true})
for caller, answered := range map[string]bool{"n1.node-tools": true, "n1.avahi": true, "n1.mesh-grants": true, "n1." + testSelf: true, "person.someone": false, "unproven": false} {
got, err := Revision(context.Background(), verbObjects, testSelf, caller, meshcatalog.RevisionInput{})
got, err := Revision(context.Background(), verbObjects, nil, testSelf, caller, meshcatalog.RevisionInput{})
if (err == nil) != answered || (answered && (got.Revision.Revision != 3 || len(got.Entries) != 4 || got.Next != "")) {
t.Errorf("revision to %s answered %+v (%v), want answered %v with trunk revision 3 whole", caller, got, err, answered)
}
}
if got, err := Revision(context.Background(), verbObjects, testSelf, "n1.node-tools", meshcatalog.RevisionInput{Group: "feat/x", Heads: "0123456789ab", After: "mesh-grants"}); err != nil ||
if got, err := Revision(context.Background(), verbObjects, nil, testSelf, "n1.node-tools", meshcatalog.RevisionInput{Group: "feat/x", Heads: "0123456789ab", After: "mesh-grants"}); err != nil ||
got.Revision.Kind != "candidate" || len(got.Entries) != 1 || got.Entries[0].Module != "node-tools" {
t.Errorf("a candidate's revision after mesh-grants answered %+v (%v), want its one entry node-tools", got, err)
}
if _, err := Revision(context.Background(), verbObjects, testSelf, "n1.node-tools", meshcatalog.RevisionInput{Revision: 9}); err == nil {
if _, err := Revision(context.Background(), verbObjects, nil, testSelf, "n1.node-tools", meshcatalog.RevisionInput{Revision: 9}); err == nil {
t.Error("a trunk revision not kept was answered")
}
// ADR 0614 §2, §3: the graph names what each module invokes and serves, and leaves out, over the whole graph, the
@@ -799,7 +799,7 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
}
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.27.1")
m, _ := moduleOf(t, mods, "avahi")
want := map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}}
want := map[string]meshcatalog.ToolchainUsed{"go": {Version: "go1.27", Recipe: "1"}}
if m.Fp2 == "" || !reflect.DeepEqual(m.Toolchains, want) {
t.Fatalf("avahi: fp2 %q toolchains %+v, want %+v", m.Fp2, m.Toolchains, want)
}
@@ -807,9 +807,9 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Errorf("a module with no fingerprint carries toolchains %+v", o.Toolchains)
}
// An upgrade moves the fingerprint and the toolchains together, with no merge.
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.27.2")
reportToolchain(t, ix, "mesh-build-go", "novox", "go", "go1.28.0")
n, _ := moduleOf(t, mods, "avahi")
if n.Fp2 == m.Fp2 || n.Toolchains["go"].Version != "go1.27.2" {
if n.Fp2 == m.Fp2 || n.Toolchains["go"].Version != "go1.28" {
t.Errorf("after the upgrade: fp2 %q (was %q) toolchains %+v, want the new version with the new fingerprint", n.Fp2, m.Fp2, n.Toolchains)
}
// Issue 1585 (ADR 0573 §2): a module whose go.sum names mesh-sdk takes mesh-sdk's latest published version into its
@@ -832,6 +832,8 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
if err := ix.Apply(context.Background(), f); err != nil {
t.Fatal(err)
}
// A-476a: a commit of mesh-sdk moves avahi, of another repository, which imports its package, and leaves mesh-tools,
// which imports nothing it moved.
a, _ := moduleOf(t, mods, "avahi")
if a.Fp2 == "" || !reflect.DeepEqual(a.Packages, map[string]string{sdk: commit}) {
t.Fatalf("after mesh-sdk's merge %.8s: fp2 %q none2 %q packages %v", commit, a.Fp2, a.None2, a.Packages)
@@ -841,12 +843,24 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
}
return a
}
// A-476 c: mesh-tools, of another repository than mesh-sdk, holds a fingerprint a recompute would move (here a stale one); a
// commit of mesh-sdk, which moved nothing it imports, leaves it as held, and only a full recompute moves it.
stale := tools
stale.Fp2 = "src2:" + strings.Repeat("5", 64)
mods.put("mesh-tools", mustEncode(t, stale))
tools = stale
first := sdkMerge(strings.Repeat("1", 40), "tree-sdk-1")
second := sdkMerge(strings.Repeat("2", 40), "tree-sdk-2")
if second.Fp2 == first.Fp2 {
t.Errorf("a mesh-sdk trunk merge kept avahi's fingerprint %q", first.Fp2)
}
n = second
if err := ix.Recompute(context.Background()); err != nil {
t.Fatal(err)
}
if o, _ := moduleOf(t, mods, "mesh-tools"); o.Fp2 == stale.Fp2 {
t.Errorf("a full recompute left mesh-tools at %q: the check above proves nothing", o.Fp2)
}
n, _ = moduleOf(t, mods, "avahi") // moved by the recompute too: its base mesh-tools moved
// Nodes that disagree keep the fingerprint already computed, with its toolchains (issue 1940, 2026-10-25 20:56: a
// stale report blanked every module and the controller's take-in refused modules already built); only a new
// fingerprint needs the agreement: a merge that moves the module's tree gives it none, naming the disagreement.
@@ -863,7 +877,7 @@ func TestAModuleKeepsTheToolchainsItsFingerprintWasComputedWith(t *testing.T) {
t.Errorf("a new tree while nodes disagree: fp2 %q toolchains %+v none2 %q, want no fingerprint and the disagreement", d.Fp2, d.Toolchains, d.None2)
}
// They agree again: the new tree gets its fingerprint.
reportToolchain(t, ix, "mesh-build-go", "ace", "go", "go1.27.2")
reportToolchain(t, ix, "mesh-build-go", "ace", "go", "go1.28.0")
if a, _ := moduleOf(t, mods, "avahi"); a.Fp2 == "" || a.Fp2 == n.Fp2 {
t.Errorf("agreed again: fp2 %q (was %q), want a new fingerprint", a.Fp2, n.Fp2)
}
@@ -1,264 +0,0 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/trunk"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"mesh-delivery-indexer/judge"
)
// The validation before take-in (design 47 G §712, A-213, A-232): for a pending revision of a watched branch the running
// indexer asks the seat node-build-go a `run` of its own manifest validator over two cases, change and trunk, and refuses only
// what the revision causes (ADR 0579 §2). The ask rides the check queue under the indexer's name, its kind `run` in the body.
const (
// validationProgram is the manifest validator's program, by its directory under the indexer entry's Go module: the indexer's
// own code, so no repository or module name is fixed here (ADR 0666 §5).
validationProgram = "cmd/mesh-judge"
// ValidationAsker is the asker the ask is put under (buildrun.Asker.Name, A-248): the indexer's seat name, mesh-catalog
// (design 47 G A-163, A-232).
ValidationAsker = meshcatalog.Seat
// validationCatalogSeat is the seat the indexer entry claims.
validationCatalogSeat = "mesh-catalog"
// The cases of the request, as the manifest validator names them.
validationChange = "change"
validationTrunk = judge.TrunkCase
)
// validationInputVerb is where the build seat's holder asks the input: the seat mesh-catalog's verb input (§354).
var validationInputVerb = "seat:" + meshcatalog.Seat + "." + meshcatalog.VerbInput
// validationRequest is the manifest validator's request for the ready merges of branch over the branch's current revision
// (§712): case change, every module of the revision by its entry's manifest, each one the merges carry replaced by the merged
// message's bytes (the newest merge governs) and one added for a new directory; case trunk, every module as it stands; the
// seat definitions the merges' trees carry; the packages and the predefined list as the take-in's own validator has them;
// no machines. It answers the indexer entry of the change case too, the one that claims the seat mesh-catalog.
func (ix *Indexer) validationRequest(ctx context.Context, branch string, ready []PendingEntry) (composer.Request, meshcatalog.CatalogEntry, error) {
var indexer meshcatalog.CatalogEntry
if ix.Revisions == nil {
return composer.Request{}, indexer, errors.New("the catalog graph is not kept here: no revision to validate over")
}
rev, found, err := newestOf(ctx, ix.Revisions, branch)
if err != nil {
return composer.Request{}, indexer, err
}
var entries []meshcatalog.CatalogEntry
if found {
if _, entries, err = ReadRevision(ctx, ix.Revisions, RevisionObject(rev)); err != nil {
return composer.Request{}, indexer, err
}
}
trunkCase := composer.Case{Name: validationTrunk, Manifests: map[string]composer.Manifest{}}
byPlace := map[string]string{} // <repository> <path> -> module
of := map[string]meshcatalog.CatalogEntry{}
for _, e := range entries {
raw := ix.manifestBytes(ctx, e.ManifestDigest)
if raw == "" {
b, err := json.Marshal(e.Manifest)
if err != nil {
return composer.Request{}, indexer, fmt.Errorf("the manifest of %s: %w", e.Module, err)
}
raw = string(b)
}
trunkCase.Manifests[e.Module] = composer.Manifest{Raw: json.RawMessage(raw)}
byPlace[placeKey(e.Repository, e.Path)] = e.Module
of[e.Module] = e
}
change := composer.Case{Name: validationChange, Manifests: maps.Clone(trunkCase.Manifests)}
for _, e := range ready {
if e.Covered {
continue
}
for _, dir := range slices.Sorted(maps.Keys(e.Forge.Manifests)) {
raw := e.Forge.Manifests[dir]
module, known := byPlace[placeKey(e.Forge.Repository, dir)]
if !known {
m, err := judge.ForInputs([]byte(raw))
if err != nil || m.Module == "" {
module = e.Forge.Repository + "/" + dir // a new directory whose manifest names no module: judged under its place
} else {
module = m.Module
}
byPlace[placeKey(e.Forge.Repository, dir)] = module
of[module] = meshcatalog.CatalogEntry{Module: module, Repository: e.Forge.Repository, Path: dir}
}
change.Manifests[module] = composer.Manifest{Raw: json.RawMessage(raw)}
}
}
for _, module := range slices.Sorted(maps.Keys(change.Manifests)) {
m, err := judge.ForInputs(change.Manifests[module].Raw)
if err != nil || !slices.ContainsFunc(m.Claims, func(c catalogue.Claim) bool { return c.Name == validationCatalogSeat }) {
continue
}
indexer = of[module]
if indexer.Module == "" {
indexer.Module = module
}
break
}
seats, err := ix.revisionDefinitions(ctx, ready)
if err != nil {
return composer.Request{}, indexer, err
}
req := composer.Request{V: composer.Version, Input: json.RawMessage("{}"), Cases: []composer.Case{change, trunkCase}, Seats: seats,
Predefined: ix.predefinedList(ctx)}
if known := ix.knownPackages(ctx); known != nil {
req.Packages = slices.Sorted(maps.Keys(known))
}
return req, indexer, nil
}
func placeKey(repository, path string) string {
return strings.ToLower(trunk.RepositoryName(repository)) + " " + path
}
// revisionDefinitions are the validation's seat definitions, raw (§712 `seats`, A-241): the branch's revision's seat
// definitions with the definitions the merges' trees carry laid over them, so a revision that touches no definition is still
// judged against the whole seat list. A revision's own definitions are not kept yet (A-234), so the revision's are the
// trunk's, read from the state seat-definitions as the verb seat-definitions answers them.
func (ix *Indexer) revisionDefinitions(ctx context.Context, ready []PendingEntry) ([]json.RawMessage, error) {
defs, err := ix.trunkDefinitions(ctx)
if err != nil {
return nil, err
}
merged, err := ix.revisionSeatDefinitions(ctx, ready)
if err != nil {
return nil, err
}
for seat, d := range merged {
raw, err := json.Marshal(d)
if err != nil {
return nil, fmt.Errorf("the definition of %s: %w", seat, err)
}
defs[seat] = raw
}
var out []json.RawMessage
for _, seat := range slices.Sorted(maps.Keys(defs)) {
out = append(out, defs[seat])
}
return out, nil
}
// trunkDefinitions are the trunk's seat definitions of the state seat-definitions, raw, by seat: every `<seat>.trunk` entry
// that holds a definition; none where the indexer keeps no such state.
func (ix *Indexer) trunkDefinitions(ctx context.Context) (map[string]json.RawMessage, error) {
defs := map[string]json.RawMessage{}
if ix.SeatDefinitions == nil {
return defs, nil
}
keys, err := ix.SeatDefinitions.Keys(ctx)
if err != nil {
return nil, fmt.Errorf("the %s state cannot be listed: %w", SeatDefinitionsState, err)
}
for _, key := range keys {
seat, ok := strings.CutSuffix(key, "."+SourceTrunk)
if !ok || strings.HasPrefix(key, outOfStepPrefix) {
continue
}
v, _, found, err := ix.SeatDefinitions.Get(ctx, key)
if err != nil {
return nil, fmt.Errorf("%s.%s cannot be read: %w", SeatDefinitionsState, key, err)
}
var e SeatDefinitionEntry
if !found || json.Unmarshal(v, &e) != nil || e.Digest == "" || len(e.Definition) == 0 {
continue // an entry that holds no definition is no seat of the list
}
defs[seat] = e.Definition
}
return defs, nil
}
// validationJob is the run of the manifest validator for the request (§712): the program cmd/mesh-judge, its directory under
// the Go module the go.mod at the entry's path names (read by the build, A-268), at the entry's path, built at head, the revision's head of the entry's repository; no
// heads and no arguments; the input the request's bytes, served by the seat mesh-catalog's verb input.
func validationJob(indexer meshcatalog.CatalogEntry, head, branch string, seq uint64, input []byte) (buildstep.RunJob, error) {
switch {
case indexer.Repository == "":
return buildstep.RunJob{}, fmt.Errorf("no entry of the revision claims the seat %s", validationCatalogSeat)
case head == "":
return buildstep.RunJob{}, fmt.Errorf("the revision names no head of %s, the repository of the indexer entry %s", indexer.Repository, indexer.Module)
}
in, err := buildstep.RunInputOf(validationInputVerb, input)
if err != nil {
return buildstep.RunJob{}, err
}
dir := indexer.Path
if dir == "" {
dir = "."
}
return buildstep.RunJob{
Program: buildstep.RunProgram{Repository: indexer.Repository, Commit: head, Dir: dir, Package: "./" + validationProgram},
Input: in,
Reader: fmt.Sprintf("manifest validator of %s at %d", branchName(branch), seq),
}, nil
}
func branchName(branch string) string {
if isTrunkBranch(branch) {
return TrunkBranch
}
return branch
}
// ValidationRefusal is what a validation's verdict refuses (ADR 0579 §2, §712): of each manifest the change case refuses that
// the revision adds or changes (its bytes differ from the trunk case's), each problem the trunk case lacks; and each problem
// between manifests the trunk case lacks. A problem only in unchanged manifests refuses nothing. It is empty when the revision
// is accepted. An answer that holds not both cases is no verdict: it answers an error, so the run could not validate.
func ValidationRefusal(req composer.Request, v composer.Verdict) (map[string][]string, error) {
cases := map[string]composer.CaseVerdict{}
for _, c := range v.Cases {
cases[c.Name] = c
}
change, okChange := cases[validationChange]
trunkV, okTrunk := cases[validationTrunk]
if !okChange || !okTrunk {
return nil, fmt.Errorf("the manifest validator's answer holds not both cases %s and %s", validationChange, validationTrunk)
}
raws := map[string]map[string]string{}
for _, c := range req.Cases {
raws[c.Name] = map[string]string{}
for name, m := range c.Manifests {
raws[c.Name][name] = InputDigest(m.Raw)
}
}
refused := map[string][]string{}
for _, module := range slices.Sorted(maps.Keys(change.Manifests)) {
r := change.Manifests[module]
if r.Accepted || r.Refused == "" {
continue
}
before, had := raws[validationTrunk][module]
if module != judge.Unnamed && had && before == raws[validationChange][module] {
continue // unchanged: its problems refuse nothing, and its entry keeps its verdict (ADR 0579 §1)
}
old := map[string]bool{}
for _, p := range strings.Split(trunkV.Manifests[module].Refused, "\n") {
old[p] = true
}
for _, p := range strings.Split(r.Refused, "\n") {
if p != "" && !old[p] {
refused[module] = append(refused[module], p)
}
}
}
return refused, nil
}
// RefusedWords is a refusal as the pending value's `refused` says it: each manifest and its problems, one manifest a line.
func RefusedWords(refused map[string][]string) string {
var lines []string
for _, module := range slices.Sorted(maps.Keys(refused)) {
lines = append(lines, module+": "+strings.Join(refused[module], "; "))
}
return strings.Join(lines, "\n")
}
@@ -1,130 +0,0 @@
package index
import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshcatalog"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-forge-bridge/forgemsg"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"mesh-delivery-indexer/judge"
)
// The validation refuses only what the revision causes (design 47 G §712, ADR 0579 §2), and its run names the indexer entry's
// repository, path and the revision's head, and no heads.
func TestValidationRefusesWhatTheRevisionCauses(t *testing.T) {
manifests := func(m map[string]string) map[string]composer.Manifest {
out := map[string]composer.Manifest{}
for k, v := range m {
out[k] = composer.Manifest{Raw: json.RawMessage(v)}
}
return out
}
req := composer.Request{Cases: []composer.Case{
{Name: "change", Manifests: manifests(map[string]string{"a": `{"module":"a","v":2}`, "b": `{"module":"b"}`})},
{Name: "trunk", Manifests: manifests(map[string]string{"a": `{"module":"a"}`, "b": `{"module":"b"}`})},
}}
verdict := func(change, trunk map[string]string) composer.Verdict {
read := func(m map[string]string) map[string]composer.ManifestRead {
out := map[string]composer.ManifestRead{"a": {Accepted: true}, "b": {Accepted: true}}
for k, v := range m {
out[k] = composer.ManifestRead{Refused: v}
}
return out
}
return composer.Verdict{Cases: []composer.CaseVerdict{{Name: "change", Manifests: read(change)}, {Name: "trunk", Manifests: read(trunk)}}}
}
for _, c := range []struct {
name string
v composer.Verdict
want, wantErr string
}{
{"a changed manifest gains a problem", verdict(map[string]string{"a": "new\nold"}, map[string]string{"a": "old"}), "a: new", ""},
{"an unchanged manifest's problem refuses nothing", verdict(map[string]string{"b": "bad"}, nil), "", ""},
{"a problem between manifests the trunk lacks", verdict(map[string]string{judge.Unnamed: "x\ny"}, map[string]string{judge.Unnamed: "y"}), judge.Unnamed + ": x", ""},
{"an answer with one case is no verdict", composer.Verdict{Cases: verdict(nil, nil).Cases[:1]}, "", "not both cases"},
} {
got, err := ValidationRefusal(req, c.v)
if c.wantErr != "" {
if err == nil || !strings.Contains(err.Error(), c.wantErr) {
t.Errorf("%s: %v, want %q", c.name, err, c.wantErr)
}
continue
}
if err != nil || RefusedWords(got) != c.want {
t.Errorf("%s: %q, %v, want %q", c.name, RefusedWords(got), err, c.want)
}
}
head := strings.Repeat("a", 40)
entry := meshcatalog.CatalogEntry{Module: "mesh-delivery-indexer", Repository: "novox/mesh-delivery", Path: "modules/mesh-delivery-indexer",
Artifacts: []meshcatalog.Artifact{{Language: "go", Kind: "bundle"}}}
job, err := validationJob(entry, head, "main", 7, []byte(`{"v":1}`))
if err != nil {
t.Fatal(err)
}
want := buildstep.RunProgram{Repository: "novox/mesh-delivery", Commit: head, Dir: "modules/mesh-delivery-indexer", Package: "./cmd/mesh-judge"}
if job.Program != want || len(job.Heads) != 0 || len(job.Args) != 0 || job.Input.Verb != "seat:mesh-catalog.input" || job.Reader != "manifest validator of main at 7" {
t.Errorf("the job %+v", job)
}
if _, err := validationJob(meshcatalog.CatalogEntry{}, head, "main", 7, nil); err == nil || !strings.Contains(err.Error(), "claims the seat mesh-catalog") {
t.Errorf("no indexer entry: %v", err)
}
}
// The validation's seats are the branch's revision's seat definitions with the merges' changed ones laid over them (A-241): a
// revision that touches no definition is judged against the whole seat list, and a merge's definition replaces its seat's.
func TestValidationSeatsAreTheRevisionsWithTheMergesOver(t *testing.T) {
ctx := context.Background()
defs := newMem()
for seat, raw := range map[string]string{"node-engine": `{"seat":"node-engine","verbs":[{"name":"old"}]}`, "git": `{"seat":"git"}`} {
v, _ := json.Marshal(SeatDefinitionEntry{Seat: seat, Source: SourceTrunk, Digest: "d-" + seat, Definition: json.RawMessage(raw)})
defs.put(SeatDefinitionKey(seat, SourceTrunk, "", 0), v)
}
v, _ := json.Marshal(SeatDefinitionEntry{Seat: "pulled", Source: SourcePull, Pull: 3, Digest: "d", Definition: json.RawMessage(`{"seat":"pulled"}`)})
defs.put(SeatDefinitionKey("pulled", SourcePull, "r", 3), v)
ix := &Indexer{Module: testSelf, SeatDefinitions: defs}
seatsOf := func(raws []json.RawMessage) map[string]string {
out := map[string]string{}
for _, raw := range raws {
var d struct {
Seat string `json:"seat"`
Verbs []struct {
Name string `json:"name"`
} `json:"verbs"`
}
if err := json.Unmarshal(raw, &d); err != nil {
t.Fatal(err)
}
out[d.Seat] = ""
if len(d.Verbs) > 0 {
out[d.Seat] = d.Verbs[0].Name
}
}
return out
}
got, err := ix.revisionDefinitions(ctx, nil)
if err != nil || !reflect.DeepEqual(seatsOf(got), map[string]string{"node-engine": "old", "git": ""}) {
t.Errorf("a revision touching no definition: %v (%v), want the trunk's whole list and no pull request's", seatsOf(got), err)
}
const contracts = "modules/seat-contracts"
f := forgemsg.Forge{Repository: "r", Merge: strings.Repeat("c", 40), Manifests: map[string]string{
contracts: `{"module":"seat-contracts","version":"1.0","build":{"artifacts":[{"name":"go-seats","kind":"package","language":"go","paths":["go"],"series":"0.1","generated-from":"."}]}}`}}
ix.Files = func(_ context.Context, _, _ string, dirs []string) (map[string]map[string]string, error) {
return map[string]map[string]string{dirs[0]: {"node-engine/seat.json": "1"}}, nil
}
ix.ReadFile = func(_ context.Context, _, _, p string) ([]byte, error) {
if p == contracts+"/seats/node-engine/seat.json" {
return []byte(`{"format":1,"seat":"node-engine","scope":"node","grain":"seat-list","decision":"d","domain":"Core","verbs":[{"name":"new"}]}`), nil
}
return nil, nil
}
got, err = ix.revisionDefinitions(ctx, []PendingEntry{{Forge: f}})
if err != nil || !reflect.DeepEqual(seatsOf(got), map[string]string{"node-engine": "new", "git": ""}) {
t.Errorf("a merge changing a definition: %v (%v), want its definition over the trunk's and the rest kept", seatsOf(got), err)
}
}
@@ -1,329 +0,0 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
)
// The pending revision's wait for its validation (design 47 G §713, A-232, A-247): the take-in puts the run ask of §712, writes
// `validation` on the revision's newest pending value and ends there; the build seat's step-ended of that ask, heard through the
// indexer's own durable consumer, resumes it: accepted takes the revision in as before, refused keeps it pending with `refused`,
// and an end that could not validate keeps it pending with `failed`, asked again by a later merge of the branch or by hand.
// ValidationAsks puts a run ask on the build seat (buildrun.Asker).
type ValidationAsks interface {
Put(ctx context.Context, a buildstep.StepAsk) error
}
// newestOfRevision is the index in ready of the revision's newest merge, the value that carries `validation` (A-247).
func newestOfRevision(ready []PendingEntry) int {
n := 0
for i, e := range ready {
if e.Value.Seq > ready[n].Value.Seq {
n = i
}
}
return n
}
// branchOfRevision is the watched branch the revision's merges landed on, its newest merged message's base (design 47 G §714);
// a message that names none is the trunk's. A pending value's own branch is its pull request's, never the branch validated.
func branchOfRevision(ready []PendingEntry) string {
if base := ready[newestOfRevision(ready)].Forge.Base; base != "" {
return base
}
return TrunkBranch
}
// validationGate says whether the ready revision waits for its validation (§713): it goes on to the take-in once its newest
// value's validation was accepted at that value's sequence; it waits while its ask is out, while it is refused, and while it
// could not validate; otherwise it puts a new ask, the next try, and waits. Without an asker nothing is validated.
func (ix *Indexer) validationGate(ctx context.Context, ready []PendingEntry, own string) (bool, string, error) {
if ix.ValidationAsks == nil {
return false, "", nil
}
newest := ready[newestOfRevision(ready)]
seq := newest.Value.Seq
if v := newest.Value.Validation; v != nil && v.Seq == seq {
switch {
case v.Accepted == seq:
return false, "", nil
case v.Failed != "":
return true, fmt.Sprintf("the validation of the revision at %d could not validate: %s", seq, v.Failed), nil
case newest.Value.Refused != "":
return true, fmt.Sprintf("the revision at %d is refused: %s", seq, newest.Value.Refused), nil
case v.Ask != "":
return true, fmt.Sprintf("the revision at %d waits for its validation, the ask %s", seq, v.Ask), nil
}
}
return true, "", ix.askValidation(ctx, ready, own)
}
// askValidation puts the revision's validation ask (§712) and writes it on its newest value, the try after every earlier try
// of the revision; an older value's validation goes (A-247: on the newest alone). A request or job that cannot be made could not
// validate: it is written as `failed`, said, and no ask is put.
func (ix *Indexer) askValidation(ctx context.Context, ready []PendingEntry, own string) error {
n := newestOfRevision(ready)
newest := ready[n]
try := 0
for _, e := range ready {
if v := e.Value.Validation; v != nil && v.Try > try {
try = v.Try
}
}
v := &takein.Validation{Seq: newest.Value.Seq, Try: try + 1, Asked: ix.now()}
branch := branchOfRevision(ready)
ask := buildstep.StepAsk{ID: buildstep.NewID(), Kind: buildstep.AskCheck, Language: buildstep.AskGo, Asker: ValidationAsker, At: ix.now().UTC()}
req, entry, err := ix.validationRequest(ctx, branch, ready)
var raw []byte
if err == nil {
raw, err = json.Marshal(req)
}
var job buildstep.RunJob
if err == nil {
job, err = validationJob(entry, revisionHead(ready, entry.Repository, entry.Commit), branch, v.Seq, raw)
}
if err == nil {
v.Input, err = ix.KeepInput(ctx, ask.ID, raw)
}
if err != nil {
v.Failed = "the validation could not be asked: " + err.Error()
} else {
v.Ask, ask.Run = ask.ID, &job
}
if err := ix.writeValidation(ctx, ready, n, v, own); err != nil {
return err
}
if v.Failed != "" {
ix.sayValidation(ctx, ready, v)
return nil
}
if err := ix.ValidationAsks.Put(ctx, ask); err != nil {
return fmt.Errorf("the validation ask %s of the revision at %d could not be put: %w", ask.ID, v.Seq, err)
}
ix.logf("take-in: the revision at %d is validated by the ask %s (try %d); it waits for its end", v.Seq, ask.ID, v.Try)
return nil
}
// revisionHead is the revision's head of repository: the newest merge of it the revision covers, else the commit its entry stands at.
func revisionHead(ready []PendingEntry, repository, commit string) string {
var seq uint64
head := commit
for _, e := range ready {
if e.Forge.Merge != "" && strings.EqualFold(e.Forge.Repository, repository) && e.Value.Seq >= seq {
seq, head = e.Value.Seq, e.Forge.Merge
}
}
return head
}
// writeValidation writes v on ready[n] and takes it from every other value, each by compare-and-set; a new validation clears the
// value's refusal. A value lost under it is an error: the evaluation is made again.
func (ix *Indexer) writeValidation(ctx context.Context, ready []PendingEntry, n int, v *takein.Validation, own string) error {
for i, e := range ready {
if i != n && e.Value.Validation == nil {
continue
}
p := e.Value
p.Validation, p.LastWriter = nil, own
if i == n {
p.Validation, p.Refused = v, ""
}
data, err := encodePending(p)
if err != nil {
return err
}
if err := ix.Pending.Update(ctx, e.Key, data, e.Revision); err != nil {
return fmt.Errorf("%s.%s: %w", PendingState, e.Key, err)
}
}
return nil
}
// sayValidation says the revision's wait in take-in.pending with its refusal or why it could not validate.
func (ix *Indexer) sayValidation(ctx context.Context, ready []PendingEntry, v *takein.Validation) {
t := revisionWait(takein.TakeInPending, ready, "")
t.Failed = v.Failed
t.Refused = ready[newestOfRevision(ready)].Value.Refused
ix.sayTakeIn(ctx, t)
}
// errNoValidationAnswer is an end whose answer is no verdict of both cases.
var errNoValidationAnswer = errors.New("no answer")
// HearValidationEnded takes in the end of a validation ask (§713), heard on the build seat's step-ended through the indexer's own
// durable consumer: only an end of the indexer's asks outside a pipeline; an end whose ask no pending value names now is dropped,
// written in the journal. The answer is read from the event's own output or output object (A-247).
func (ix *Indexer) HearValidationEnded(ctx context.Context, e buildstep.StepEnded) error {
if e.Asker != ValidationAsker || e.Pipeline != "" || e.Ask == "" || ix.Pending == nil || !ix.pendingOnBus(ctx) {
return nil
}
own := ix.ownBuildNow(ctx)
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
entries, err := ReadPending(ctx, ix.Pending, ix.logf)
if err != nil {
return err
}
ready := WaitsOf(entries).Ready
n := -1
for i, p := range ready {
if p.Value.Validation != nil && p.Value.Validation.Ask == e.Ask {
n = i
}
}
if n < 0 {
ix.logf("take-in: the end of the validation ask %s (%s on %s) is dropped: no pending revision names that ask now", e.Ask, e.Outcome, e.Node)
return nil
}
ready, lost, _, _ := ix.loadNewest(ctx, ready)
if len(lost) > 0 {
return fmt.Errorf("the revision the validation ask %s ended for holds a merge that cannot be read", e.Ask)
}
n = newestOfRevision(ready)
v := *ready[n].Value.Validation
refused, failed := ix.validationAnswer(ctx, ready, e, v)
p := ready[n].Value
switch {
case failed != "":
v.Failed = failed
case refused != "":
p.Refused = refused
default:
v.Accepted = v.Seq
}
p.Validation, p.LastWriter = &v, own
data, err := encodePending(p)
if err != nil {
return err
}
if err := ix.Pending.Update(ctx, ready[n].Key, data, ready[n].Revision); err != nil {
return fmt.Errorf("%s.%s: %w", PendingState, ready[n].Key, err)
}
if v.Accepted == 0 {
ready[n].Value = p
ix.sayValidation(ctx, ready, &v)
ix.logf("take-in: the validation ask %s of the revision at %d ended: refused %q, could not validate %q", e.Ask, v.Seq, refused, failed)
return nil
}
ix.logf("take-in: the validation ask %s accepted the revision at %d", e.Ask, v.Seq)
if own == "" {
return nil // the take-in waits for the build this program runs, as every take-in does
}
_, _, err = ix.evaluate(ctx, own, false)
return err
}
// validationAnswer reads the end's answer: the refusal of the revision, or why it could not validate (the step did not pass, the
// program exited other than 0, its answer is no verdict of both cases, or its request is no longer the one asked).
func (ix *Indexer) validationAnswer(ctx context.Context, ready []PendingEntry, e buildstep.StepEnded, v takein.Validation) (string, string) {
if e.Outcome != buildstep.OutcomePassed {
return "", strings.TrimSpace(e.Outcome + ": " + e.Said)
}
// The answer is the program's last line of standard output, inline or kept by the seat as an object (A-254); a program that
// exited other than 0 ended the step failed, read above.
out, err := e.ReadOutput(buildstep.AskGo, ix.AskVerb)
if err != nil {
return "", "the run's answer could not be read: " + err.Error()
}
var verdict composer.Verdict
if err := json.Unmarshal(out, &verdict); err != nil {
return "", fmt.Sprintf("%v: the manifest validator's output is no verdict: %v", errNoValidationAnswer, err)
}
req, _, err := ix.validationRequest(ctx, branchOfRevision(ready), ready)
var raw []byte
if err == nil {
raw, err = json.Marshal(req)
}
if err != nil {
return "", "the request cannot be made again to read the verdict: " + err.Error()
}
if runDigest(raw) != v.Input {
return "", "the revision's request is no longer the one the ask " + v.Ask + " was asked with"
}
refused, err := ValidationRefusal(req, verdict)
if err != nil {
return "", err.Error()
}
return RefusedWords(refused), ""
}
// ValidateAgainTool asks the validation of a revision that could not validate again by hand (§713).
const ValidateAgainTool = "indexer_validate_again"
// validationWords is the validation's state as the hand ask answers it.
func validationWords(v *takein.Validation, refused string) string {
switch {
case v == nil:
return "no validation is asked yet"
case v.Accepted != 0:
return fmt.Sprintf("accepted at %d (try %d, the ask %s)", v.Accepted, v.Try, v.Ask)
case v.Failed != "":
return fmt.Sprintf("could not validate at %d (try %d): %s", v.Seq, v.Try, v.Failed)
case refused != "":
return fmt.Sprintf("refused at %d (try %d): %s", v.Seq, v.Try, refused)
default:
return fmt.Sprintf("the ask %s is out since %s (try %d at %d)", v.Ask, v.Asked.UTC().Format(time.RFC3339), v.Try, v.Seq)
}
}
// ValidateAgainByHand is indexer_validate_again answered (§713): the ready revision under key (its oldest merge's m. key, as
// indexer_pending names it) whose validation could not validate is asked again, the next try; its answer says the ask's state.
// A revision accepted, refused or whose ask is out is asked nothing: a refusal is asked again only by a later merge of its branch.
func (ix *Indexer) ValidateAgainByHand(ctx context.Context, key, by, why string) TakenIn {
r := TakenIn{Module: key}
if ix.Pending == nil || ix.ValidationAsks == nil {
r.Answer = "this indexer asks no validation: nothing asked"
return r
}
own := ix.ownBuildNow(ctx)
ix.pendingMu.Lock()
defer ix.pendingMu.Unlock()
entries, err := ReadPending(ctx, ix.Pending, ix.logf)
if err != nil {
r.Answer = err.Error()
return r
}
ready := WaitsOf(entries).Ready
if len(ready) == 0 || ready[0].Key != key {
r.Answer = fmt.Sprintf("no revision waits under %s: indexer_pending names the waits", key)
return r
}
ready, lost, reasons, _ := ix.loadNewest(ctx, ready)
if len(lost) > 0 {
r.Answer = fmt.Sprintf("nothing asked: the message of %s is not read (%s)", lost[0].Key, reasons[lost[0].Key])
return r
}
newest := ready[newestOfRevision(ready)]
v := newest.Value.Validation
if v == nil || v.Seq != newest.Value.Seq || v.Failed == "" {
r.Answer = "nothing asked: the validation is " + validationWords(v, newest.Value.Refused) + "; only one that could not validate is asked again by hand"
return r
}
ix.logf("take-in: the validation of %s is asked again by hand by %s: %s", key, by, why)
if err := ix.askValidation(ctx, ready, own); err != nil {
r.Answer = "the validation was not asked again: " + err.Error()
return r
}
after, err := ReadPending(ctx, ix.Pending, ix.logf)
if err == nil {
ready = WaitsOf(after).Ready
}
if err != nil || len(ready) == 0 {
r.Answer = "asked again; its state is not read back"
return r
}
n := ready[newestOfRevision(ready)]
r.Stored = n.Value.Validation != nil && n.Value.Validation.Failed == ""
r.Answer = fmt.Sprintf("asked again by %s (%s): the validation is %s", by, why, validationWords(n.Value.Validation, n.Value.Refused))
return r
}
@@ -1,113 +0,0 @@
package index
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-indexer/takein"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/buildstep"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-sdk/composer"
)
type askRecorder struct{ asks []buildstep.StepAsk }
func (r *askRecorder) Put(_ context.Context, a buildstep.StepAsk) error {
r.asks = append(r.asks, a)
return nil
}
// Design 47 G §713, A-247: a pending revision names its ask on its newest value and resumes on that ask's end; an end of a
// replaced ask is dropped; a run that could not validate is no refusal and never passed; the start asks nothing again while
// the ask is out.
func TestAPendingRevisionWaitsForItsValidation(t *testing.T) {
ctx := context.Background()
b := newTakeInBed(t)
asks := &askRecorder{}
b.ix.ValidationAsks, b.ix.Inputs, b.ix.Revisions = asks, newMem(), newMemObjects(0)
// No entry of the revision claims the seat mesh-catalog: the validation could not be asked, so it is failed, said, and
// nothing is taken in.
b.arrive(t, 2, b.merge(21, sha('c'), "tree-avahi-3", ""))
entries := readPending(t, b)
if len(entries) != 1 || entries[0].Value.Validation == nil || !strings.Contains(entries[0].Value.Validation.Failed, "claims the seat mesh-catalog") || len(asks.asks) != 0 {
t.Fatalf("pending %+v, asks %d: want m.2 pending with failed and no ask", entries, len(asks.asks))
}
if last := b.said[len(b.said)-1]; last.Kind != takein.TakeInPending || last.Failed == "" || last.Refused != "" {
t.Errorf("said %+v, want take-in.pending naming failed, never a refusal", last)
}
if w := PendingWaits(entries); len(w) != 1 || w[0].Validation == nil || w[0].Validation.Try != 1 {
t.Errorf("indexer_pending %+v, want the validation whole", w)
}
// An ask out: the newest value names it; the start asks nothing new while it is out.
loaded, _, _, _ := b.ix.loadNewest(ctx, entries)
req, _, err := b.ix.validationRequest(ctx, TrunkBranch, loaded)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(req)
setValidation := func(v takein.Validation) {
e := readPending(t, b)[0]
p := e.Value
p.Validation = &v
data, _ := encodePending(p)
if err := b.pending.Update(ctx, e.Key, data, e.Revision); err != nil {
t.Fatal(err)
}
}
setValidation(takein.Validation{Ask: "mvl0frv18d1957", Input: runDigest(raw), Seq: 2, Try: 2})
if _, _, err := b.ix.TakeInAtStart(ctx); err != nil || len(asks.asks) != 0 || len(b.keys()) != 1 {
t.Fatalf("the start: %v, asks %d, pending %v: want it waiting for its ask", err, len(asks.asks), b.keys())
}
end := func(ask, outcome, said string, answer any) buildstep.StepEnded {
out, _ := json.Marshal(answer)
return buildstep.StepEnded{Ask: ask, ID: ask, Asker: ValidationAsker, Node: "ace", Outcome: outcome, Said: said, Output: string(out)}
}
// The answer is the manifest validator's verdict itself, the program's last line (A-254).
accepted := composer.Verdict{Cases: []composer.CaseVerdict{{Name: "change"}, {Name: "trunk"}}}
// The end of a replaced ask is dropped, written in the journal.
if err := b.ix.HearValidationEnded(ctx, end("mvl0frv18d1950", buildstep.OutcomePassed, "", accepted)); err != nil || len(b.keys()) != 1 || !strings.Contains(b.logs.String(), "is dropped") {
t.Fatalf("a replaced ask's end: %v, pending %v", err, b.keys())
}
// could not run: pending with failed, never passed.
if err := b.ix.HearValidationEnded(ctx, end("mvl0frv18d1957", buildstep.OutcomeCouldNotRun, "no build node", nil)); err != nil {
t.Fatal(err)
}
if v := readPending(t, b)[0].Value.Validation; v == nil || v.Failed != "could not run: no build node" || v.Accepted != 0 {
t.Fatalf("after could not run: %+v", v)
}
// indexer_validate_again (§713): its hand act recorded first, then the next try asked; the answer says the ask's state.
var recorded []string
tools := ValidationTools(func(key string, value []byte) (TakeInAsk, error) {
a, err := DecodeTakeIn(value)
if err != nil || key != "pending.validate-again."+takein.PendingKey(2) || !a.ValidateAgain {
t.Fatalf("asked %s %+v %v", key, a, err)
}
r := b.ix.ValidateAgainByHand(ctx, a.Pending, a.By, a.Why)
a.Result = &r
return a, nil
}, b.ix.now, func(what, why string) error { recorded = append(recorded, what); return nil })
if len(tools) != 1 || tools[0].Name != ValidateAgainTool {
t.Fatalf("tools = %v", tools)
}
if _, err := tools[0].Run(map[string]any{"key": takein.PendingKey(2)}); err == nil || len(recorded) != 0 {
t.Fatalf("asked again with no why: %v, %v", err, recorded)
}
got, err := tools[0].Run(map[string]any{"key": takein.PendingKey(2), "why": "the build node is back"})
if v := readPending(t, b)[0].Value.Validation; err != nil || len(recorded) != 1 || v == nil || v.Try != 3 || v.Failed == "could not run: no build node" ||
!strings.Contains(got.(*TakenIn).Answer, "try 3") {
t.Fatalf("asked again: %v, recorded %v, validation %+v, answer %+v: want the try after 2 asked", err, recorded, v, got)
}
setValidation(takein.Validation{Ask: "mvl0frv18d1959", Input: runDigest(raw), Seq: 2, Try: 3})
if r := b.ix.ValidateAgainByHand(ctx, takein.PendingKey(2), "lead", "again"); r.Stored || !strings.Contains(r.Answer, "is out") || readPending(t, b)[0].Value.Validation.Try != 3 {
t.Fatalf("an ask out was asked again: %+v", r)
}
// Asked again (by hand or by a later merge) and accepted: the revision is taken in, and its keys go.
setValidation(takein.Validation{Ask: "mvl0frv18d1958", Input: runDigest(raw), Seq: 2, Try: 3})
if err := b.ix.HearValidationEnded(ctx, end("mvl0frv18d1958", buildstep.OutcomePassed, "", accepted)); err != nil {
t.Fatal(err)
}
if len(b.keys()) != 0 || b.commit(t, "avahi") != sha('c') {
t.Fatalf("accepted: pending %v %+v, avahi at %s: want it taken in\n%s", b.keys(), readPending(t, b)[0].Value, b.commit(t, "avahi"), b.logs.String())
}
}
@@ -506,8 +506,8 @@ func (ix *Indexer) writeVerdict(ctx context.Context, digest string, v meshcatalo
}
// OwnBuildState is the indexer's state of the build it runs (the bucket mesh-delivery-indexer_own-build, one key, OwnBuildKey):
// written once, at the process's start by FixOwnBuild, so indexer_pending says whether this process knows its build and, when it
// does not, since when and why.
// written once, at the process's start by FixOwnBuild, so indexer_pending says which build this process knows it runs, and since
// when. The build only names the writer of what is written (A-476 b).
const (
OwnBuildStateName = "own-build"
OwnBuildKey = "state"
@@ -526,8 +526,7 @@ type OwnBuild struct {
// seat (OwnBuild; never the trunk's value of the indexer, which names a build that may be merged and not delivered), before
// this process hears a merge. It is never asked again: while the node-engine replaces the indexer the seat already names the
// successor, and an old process that learnt it later would take revisions in with its old judge. Where the start's ask is not
// answered the build stays unknown for the life of this process: every merge waits, loudly (a take-in.pending that says so, the
// own-build state, indexer_pending), and the module is started again to learn it.
// answered the build stays unknown for the life of this process, said in the own-build state and the journal; nothing waits for it.
func (ix *Indexer) FixOwnBuild(ctx context.Context) {
ix.ownMu.Lock()
if ix.ownFixed || ix.OwnBuild == nil {
@@ -543,7 +542,7 @@ func (ix *Indexer) FixOwnBuild(ctx context.Context) {
st := OwnBuild{V: 1, Since: ix.now().UTC()}
if err != nil {
st.Why = err.Error()
ix.logf("judge: the build this program runs is not known (%v), so every merge waits and a manifest with a field it does not know waits and is not refused; it is learnt only at the start, so the indexer is started again", err)
ix.logf("judge: the build this program runs is not known (%v); merges are taken in all the same, and it is learnt only at the start", err)
} else {
st.Known, st.FP = true, fp
ix.logf("judge: this program runs the indexer build %s, fixed here at its start", fp)
@@ -31,6 +31,7 @@ var ReadTools = []ViewedTool{
{Tool: "indexer_read_graph", State: graph.State, Words: "the module graph: each module's build and provision edges."},
{Tool: "indexer_read_heard", State: HeardState, Words: "the forge messages the indexer heard and what it did with each."},
{Tool: "indexer_read_taken", State: "taken", Words: "each repository's merge the indexer took in last."},
{Tool: "indexer_read_takeins", State: TakeInsState, Words: "each take-in asked of the indexer: its ask, whether and when the indexer answered it and its answer (design 47 G A-464a)."},
{Tool: "indexer_read_findings", State: FindingsState, Words: "the indexer's findings, which the condition keeper raises and clears."},
{Tool: "indexer_read_seat_definitions", State: SeatDefinitionsState, Words: "the seat definitions the trunk's and each open pull request's contracts modules hold, resolved, or why one does not parse."},
{Tool: "indexer_read_unjudged", State: UnjudgedState, Words: "the manifests a take-in newly wrote that no judge pass judged yet, by digest, each with its module and merge; indexer_judge judges them."},
@@ -44,7 +45,7 @@ type ViewedTool struct {
// ViewedStates are the states the indexer's views family says: the ones its readers read.
var ViewedStates = map[string]bool{
ManifestsState: true, VerdictsState: true, PredefinedState: true, ConfigCandidatesState: true, CandidatesState: true,
PackagesState: true, HeardState: true, TakenState: true, FindingsState: true, SeatDefinitionsState: true, UnjudgedState: true,
PackagesState: true, HeardState: true, TakenState: true, TakeInsState: true, FindingsState: true, SeatDefinitionsState: true, UnjudgedState: true,
}
// ModulesViewsEvent is what the indexer says on its views family of modules when a new revision of the trunk branch
+87 -33
View File
@@ -2476,24 +2476,14 @@ func invokeProblems(m catalogue.Manifest) []string {
return problems
}
// ownSecretOwnerProblems is what is wrong with an own secret's declared owner (novox/hq ADR 0436 §1): naming
// an account the module makes — a `user` resource of that name — never root, which a secret file owned by it
// would make readable by anything, and never the operator account's placeholder (`${machine:account}`), which
// the mesh fills per machine and a manifest cannot say safely (novox/hq ADR 0436, settled by the lead).
// ownSecretOwnerProblems is what is wrong with an own secret's declared owner (novox/hq ADR 0680 §1, §2, which
// ends ADR 0436 §1's refusal of root and the operator account here): an account the module makes — a `user`
// resource of that name — the operator account's placeholder (`${machine:account}`) or root. Any other account
// the module does not make is still refused.
func ownSecretOwnerProblems(m catalogue.Manifest, name, owner string) []string {
if owner == "" {
if owner == "" || owner == "root" || owner == "${machine:account}" {
return nil
}
if owner == "root" {
return []string{fmt.Sprintf(
"%s names %q as the owner of its secret %q; an own secret's owner is an account the module "+
"makes, never root (novox/hq ADR 0436 §1)", m.Module, owner, name)}
}
if owner == "${machine:account}" {
return []string{fmt.Sprintf(
"%s names %q as the owner of its secret %q; an own secret's owner is an account the module "+
"makes, never the operator account (novox/hq ADR 0436 §1)", m.Module, owner, name)}
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "user" && fmt.Sprint(r["name"]) == owner {
return nil
@@ -2501,7 +2491,7 @@ func ownSecretOwnerProblems(m catalogue.Manifest, name, owner string) []string {
}
return []string{fmt.Sprintf(
"%s names %q as the owner of its secret %q, which it does not make: a user resource named %q "+
"(novox/hq ADR 0436 §1)", m.Module, owner, name, owner)}
"(novox/hq ADR 0680 §2)", m.Module, owner, name, owner)}
}
// ownSecretPromptProblems is what is wrong with an own secret's declared prompt, in the module check's words:
@@ -3218,6 +3208,9 @@ func parseManifest(raw []byte, predefined catalogue.PredefinedList) (catalogue.M
problems = append(problems, ownSecretPromptProblems(m, name, own.Prompt, catalogue.IsFamily(name))...)
problems = append(problems, ownSecretOwnerProblems(m, name, own.Owner)...)
}
// The module's secrets-owner is held to the same accounts as an own secret's owner (novox/hq ADR 0680 §2).
problems = append(problems, ownSecretOwnerProblems(m, "(its secrets-owner)", m.SecretsOwner)...)
problems = append(problems, ownSecretReadProblems(m)...)
localOf := map[string]string{}
for _, to := range m.SecretRequirements() {
if _, plain := m.Secrets[to]; plain {
@@ -4204,25 +4197,19 @@ func passableDir(m catalogue.Manifest, id string, depth int) bool {
return false
}
// processAccountProblems is what is wrong with one process's bus account (novox/hq ADR 0436 §1, §2): an
// account the module makes — a `user` resource of that name — never root and never the operator account,
// which a manifest cannot say safely (a node with no account has nothing to resolve it to, novox/hq ADR
// 0436, settled by the lead).
// processAccountProblems is what is wrong with one process's bus account (novox/hq ADR 0680 §1, §2, which ends ADR
// 0436 §1's refusal of root and the operator account here): an account the module makes — a `user` resource of
// that name — the operator account's placeholder (`${machine:account}`) or root. A process that declares bus
// and names no account, or one the module does not make, is refused.
func processAccountProblems(m catalogue.Manifest, processID, user string) []string {
switch {
case user == "":
switch user {
case "":
return []string{fmt.Sprintf(
"%s's process %s declares bus and names no account; a process account is a user resource the "+
"module makes, never root and never the operator account (novox/hq ADR 0436 §1, §2)",
"module makes, the operator account or root (novox/hq ADR 0680 §2)",
m.Module, processID)}
case user == "root":
return []string{fmt.Sprintf(
"%s's process %s runs as root; a process account is the module's own, never root (novox/hq ADR "+
"0436 §1, §2)", m.Module, processID)}
case user == "${machine:account}":
return []string{fmt.Sprintf(
"%s's process %s runs as the operator account; a process account is the module's own, never the "+
"operator's (novox/hq ADR 0436 §1, §2)", m.Module, processID)}
case "root", "${machine:account}":
return nil
}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "user" && fmt.Sprint(r["name"]) == user {
@@ -4230,8 +4217,75 @@ func processAccountProblems(m catalogue.Manifest, processID, user string) []stri
}
}
return []string{fmt.Sprintf(
"%s's process %s runs as %s, which it does not make: a user resource named %s (novox/hq ADR 0436 §1, "+
"§2)", m.Module, processID, user, user)}
"%s's process %s runs as %s, which it does not make: a user resource named %s (novox/hq ADR 0680 §2)",
m.Module, processID, user, user)}
}
// ownSecretReadProblems is what is wrong with a process that names an own secret it cannot read (novox/hq ADR
// 0680 §3): a process resource whose `run` or `env` names an own secret's path must run as that secret's
// resolved owner (its `owner`, else the module's `secrets-owner`, else root) or as root. A module with a
// container is excepted: a container reads its secrets as its own account. A process with no `user` runs as
// root, as the service manager runs it (lead answer A-485 a, b: a container module is any module with a
// `container` resource). A word names a secret only when it is the secret's path (a family's: begins with the path
// up to its `*`).
func ownSecretReadProblems(m catalogue.Manifest) []string {
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "container" {
return nil
}
}
var problems []string
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) != "process" {
continue
}
user, _ := r["user"].(string)
if user == "" || user == "root" {
continue
}
var words []string
if run, ok := r["run"].([]any); ok {
for _, w := range run {
words = append(words, fmt.Sprint(w))
}
}
if env, ok := r["env"].(map[string]any); ok {
for _, v := range env {
words = append(words, fmt.Sprint(v))
}
}
for name, own := range m.OwnSecrets {
path, _, _ := strings.Cut(own.Path, "*")
if path == "" {
continue
}
named := false
for _, w := range words {
if w == own.Path || (strings.Contains(own.Path, "*") && strings.HasPrefix(w, path)) {
named = true
}
}
owner := own.Owner
if owner == "" {
owner = m.SecretsOwner
}
if named && owner != user {
problems = append(problems, fmt.Sprintf(
"%s's process %v runs as %s and names its secret %q, which belongs to %s; a process runs as "+
"the owner of an own secret it names, or as root (novox/hq ADR 0680 §3)",
m.Module, r["id"], user, name, ownerWord(owner)))
}
}
}
sort.Strings(problems)
return problems
}
func ownerWord(owner string) string {
if owner == "" {
return "root"
}
return owner
}
// busKeeps says the process resource r's bus declares `keeps` (novox/hq ADR 0626 §1): read from the resource as
@@ -2113,19 +2113,79 @@ func caseAnOwnSecretSaysWhenAnOutsidePartyIssuesIt(t *testing.T) {
}
}
// 1170/AC2: an owner naming an account the module does not make, root, or the operator account's placeholder is
// refused, naming the secret and the account.
// ADR 0680 §2, §4: an own secret's owner of root or the operator account's placeholder is accepted; an owner naming an
// account the module does not make is refused, naming the secret and the account.
func caseAnOwnSecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes(t *testing.T) {
for name, c := range map[string]struct{ owner, says string }{
"root": {"root", "root"},
"the operator account": {"${machine:account}", "operator account"},
"an account not declared": {"not-made", "not-made"},
for name, c := range map[string]struct {
owner string
ok bool
}{
"root": {"root", true},
"the operator account": {"${machine:account}", true},
"an account the module makes": {"mesh-mail-send", true},
"an account not declared": {"not-made", false},
} {
_, err := ParseManifest([]byte(`{"module":"mesh-mail","version":"1.0","own-secrets":{` +
`"mail-credential-*":{"path":"/var/lib/mesh-mail/mail-credential-*.secret","issued-by":"outside","owner":"` +
c.owner + `"}},"resources":[{"id":"account","type":"user","name":"mesh-mail-send"}]}`))
if err == nil || !strings.Contains(err.Error(), "mail-credential-*") || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v; want a refusal naming the secret and %q", name, err, c.says)
switch {
case c.ok && err != nil:
t.Errorf("%s: refused: %v", name, err)
case !c.ok && (err == nil || !strings.Contains(err.Error(), "mail-credential-*") || !strings.Contains(err.Error(), c.owner)):
t.Errorf("%s: %v; want a refusal naming the secret and %q", name, err, c.owner)
}
}
}
// ADR 0680 §2, §4: the module's secrets-owner is held to the same accounts as an own secret's owner.
func caseASecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes(t *testing.T) {
for name, c := range map[string]struct {
owner string
ok bool
}{
"root": {"root", true},
"the operator account": {"${machine:account}", true},
"an account the module makes": {"mesh-mail-send", true},
"an account not declared": {"not-made", false},
} {
_, err := ParseManifest([]byte(`{"module":"mesh-mail","version":"1.0","secrets-owner":"` + c.owner +
`","resources":[{"id":"account","type":"user","name":"mesh-mail-send"}]}`))
switch {
case c.ok && err != nil:
t.Errorf("%s: refused: %v", name, err)
case !c.ok && (err == nil || !strings.Contains(err.Error(), c.owner)):
t.Errorf("%s: %v; want a refusal naming %q", name, err, c.owner)
}
}
}
// ADR 0680 §3, §4: a process naming an own secret runs as its resolved owner (its owner, else the module's
// secrets-owner) or as root; any other account is refused; a module with a container is excepted.
func caseAProcessNamingAnOwnSecretMustBeAbleToReadIt(t *testing.T) {
const users = `{"id":"u1","type":"user","name":"alice"},{"id":"u2","type":"user","name":"bob"}`
for name, c := range map[string]struct {
secret, extra, proc string
ok bool
}{
"the resolved owner": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"alice","run":["./x","--key","/var/lib/x/k"]`, true},
"the owner by secrets-owner": {`"path":"/var/lib/x/k"`, `"secrets-owner":"alice",`, `"user":"alice","env":{"KEY":"/var/lib/x/k"}`, true},
"root": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"root","run":["./x","/var/lib/x/k"]`, true},
"the operator account": {`"path":"/var/lib/x/k","owner":"${machine:account}"`, ``, `"user":"${machine:account}","run":["./x","/var/lib/x/k"]`, true},
"a secret it does not name": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"bob","run":["./x"]`, true},
"another account": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"bob","run":["./x","/var/lib/x/k"]`, false},
"another account by secrets-owner": {`"path":"/var/lib/x/k"`, `"secrets-owner":"alice",`, `"user":"bob","env":{"KEY":"/var/lib/x/k"}`, false},
"a secret owned by root": {`"path":"/var/lib/x/k"`, ``, `"user":"bob","run":["./x","/var/lib/x/k"]`, false},
"a longer word holding the path": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"bob","env":{"KEY":"/var/lib/x/keyring"}`, true},
"a container module": {`"path":"/var/lib/x/k","owner":"alice"`, ``, `"user":"bob","run":["./x","/var/lib/x/k"]},{"id":"c","type":"container","image":"x"`, true},
} {
raw := `{"module":"x","version":"1.0",` + c.extra + `"own-secrets":{"k":{` + c.secret + `}},"resources":[` + users +
`,{"id":"p","type":"process","artifact":"x",` + c.proc + `}]}`
_, err := ParseManifest([]byte(raw))
switch {
case c.ok && err != nil:
t.Errorf("%s: refused: %v", name, err)
case !c.ok && (err == nil || !strings.Contains(err.Error(), "ADR 0680 §3")):
t.Errorf("%s: %v; want the ADR 0680 §3 refusal", name, err)
}
}
}
@@ -2419,13 +2479,18 @@ func caseAPortWithNoSourceIsRefused(t *testing.T) {
}
}
// 1179/AC2: a bus on a process with no user, user root, or the operator account is refused naming the
// process and the account rule.
// ADR 0680 §2, §4: a bus on a process with no user, or a user the module does not make, is refused naming the
// process; root and the operator account are accepted.
func caseAProcessBusAccountMustBeTheModulesOwn(t *testing.T) {
for name, c := range map[string]struct{ user, says string }{
"no user": {"", "send"},
"user root": {"root", "root"},
"the operator account": {"${machine:account}", "operator"},
for name, c := range map[string]struct {
user string
ok bool
}{
"no user": {"", false},
"an account not declared": {"not-made", false},
"user root": {"root", true},
"the operator account": {"${machine:account}", true},
"the module's own": {"mesh-mail-send", true},
} {
raw := `{"module":"mesh-mail","version":"1.0","emits":["mail.send.sent"],"state":["sends"],` +
`"streams":[{"name":"mail-sends","events":["mail.send.approved"],"retention":"work-queue",` +
@@ -2434,10 +2499,13 @@ func caseAProcessBusAccountMustBeTheModulesOwn(t *testing.T) {
if c.user != "" {
raw += `,"user":"` + c.user + `"`
}
raw += `,"bus":{"emits":["mail.send.sent"]}}]}`
raw += `,"bus":{"emits":["mail.send.sent"],"queues":["mail-sends"]}}]}`
_, err := ParseManifest([]byte(raw))
if err == nil || !strings.Contains(err.Error(), "send") || !strings.Contains(err.Error(), c.says) {
t.Errorf("%s: %v; want a refusal naming the process and %q", name, err, c.says)
switch {
case c.ok && err != nil:
t.Errorf("%s: refused: %v", name, err)
case !c.ok && (err == nil || !strings.Contains(err.Error(), "send")):
t.Errorf("%s: %v; want a refusal naming the process", name, err)
}
}
}
@@ -6414,6 +6482,8 @@ func TestRulesSecrets(t *testing.T) {
t.Run("AnOffersIdentityIsParsedStrictly", caseAnOffersIdentityIsParsedStrictly)
t.Run("AnOwnSecretSaysWhenAnOutsidePartyIssuesIt", caseAnOwnSecretSaysWhenAnOutsidePartyIssuesIt)
t.Run("AnOwnSecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes", caseAnOwnSecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes)
t.Run("ASecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes", caseASecretsOwnerIsRefusedWhenItIsNotAnAccountTheModuleMakes)
t.Run("AProcessNamingAnOwnSecretMustBeAbleToReadIt", caseAProcessNamingAnOwnSecretMustBeAbleToReadIt)
t.Run("AnOwnSecretTakenSomeOtherWayIsRefused", caseAnOwnSecretTakenSomeOtherWayIsRefused)
t.Run("AnUnknownAccessReferenceIsRefusedAtParse", caseAnUnknownAccessReferenceIsRefusedAtParse)
t.Run("AnUnknownPlaceholderKeyIsRefused", caseAnUnknownPlaceholderKeyIsRefused)
@@ -165,7 +165,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// "Reads FORGE" is a consumes of the indexer's stream: the broker grants only buckets by reads (mesh-control
// internal/broker/state.go:62-93).
// The registry seats' published fill each (§413: no consume of version.numbered) package's series (DLV-I5, design 51 *The versions*).
if want := []string{"go-package-registry.published", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-conditions.findings-asked", "mesh-delivery-checker.verdict", "mesh-delivery-forge-bridge.forge.**",
if want := []string{"go-package-registry.published", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-builds.package-published", "mesh-builds.toolchain-agreed", "mesh-catalog.catalog-revision", "mesh-conditions.findings-asked", "mesh-delivery-checker.verdict", "mesh-delivery-forge-bridge.forge.**",
"mesh-delivery.freed", "mesh-delivery.heads-replaced", "mesh-delivery.pinned", "mesh-delivery.pins-stated", "mesh-delivery.source-moved", "mesh-pipeline-runner.pipeline-ended", "mesh-pipeline-runner.run-started", "mesh-wants.version-delivered", "mesh-wants.wait-ended", "mesh-wants.wanted", "node-build-go.step-ended", "node-engine.running", "npm-package-registry.published", "operator-channel.decided", "seat:mesh-module-assigner.views.assignments", "seat:mesh-module-assigner.views.seat-holders"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want)
}
@@ -179,8 +179,6 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// ADR 0499 clauses 2 and 3, ADR 0495 clause 1; and the module directories of the trunk, issue 1900) and whether a
// commit is in another (issue 1900).
if want := []string{"seat:git.trees", "seat:git.file", "seat:git.contains", "seat:git.open-pulls", "seat:git.branch", "mesh-delivery-forge-bridge.bridge_merged_message", "seat:node-engine.composition", "seat:mesh-delivery.source", "seat:mesh-module-assigner.assignments", "seat:mesh-module-assigner.placement", "seat:mesh-hand-acts.hand-act", "seat:mesh-watchdogs.presence", "seat:node-build-go.steps",
"seat:node-build-go.accept.check", // design 47 G A-231: the validation's run ask (§712) on the check queue
"seat:node-build-go.output", // design 47 G A-254: a validation's answer kept as an object
"seat:node-build-node.steps", "seat:node-build-image.steps", "mesh-delivery-builder.builder_versions"}; !reflect.DeepEqual(m.Invokes, want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
@@ -204,8 +202,7 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
// milestone 47/G W4.12 (design 47, the indexer's tools): the catalog graph revisions read and steered.
index.CatalogRevisionTool, index.CatalogRevisionsTool, index.CandidateGraphsTool, index.RecomputeTool, index.DropCandidateGraphTool,
index.EntryIndexTool, index.NamedDropTool, // design 47 G §675
index.ProposalsTool, index.ProposalDropTool, index.SeedAgainTool, index.InputsTool, index.PinsTool, index.UnpinTool, index.ConsumerTool, index.RunsTool, index.RunDropTool, // design 47 G §667, §8, §579, §64 (A-194)
index.ValidateAgainTool) // design 47 G §713
index.ProposalsTool, index.ProposalDropTool, index.SeedAgainTool, index.InputsTool, index.PinsTool, index.UnpinTool, index.ConsumerTool, index.RunsTool, index.RunDropTool) // design 47 G §667, §8, §579, §64 (A-194)
if !reflect.DeepEqual(m.Tools, want) || len(want) != 51 || !slices.Contains(m.Tools, "modules") {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
+3 -4
View File
@@ -43,6 +43,7 @@
"indexer_read_graph",
"indexer_read_heard",
"indexer_read_taken",
"indexer_read_takeins",
"indexer_read_findings",
"indexer_read_seat_definitions",
"indexer_read_unjudged",
@@ -76,8 +77,7 @@
"indexer_unpin",
"indexer_consumer",
"indexer_runs",
"indexer_run_drop",
"indexer_validate_again"
"indexer_run_drop"
],
"consumes": [
"go-package-registry.published",
@@ -93,6 +93,7 @@
"mesh-wants.wait-ended",
"mesh-delivery-checker.verdict",
"mesh-delivery-forge-bridge.forge.**",
"mesh-catalog.catalog-revision",
"mesh-delivery.source-moved",
"mesh-delivery.pinned",
"mesh-delivery.freed",
@@ -178,8 +179,6 @@
"seat:mesh-hand-acts.hand-act",
"seat:mesh-watchdogs.presence",
"seat:node-build-go.steps",
"seat:node-build-go.accept.check",
"seat:node-build-go.output",
"seat:node-build-node.steps",
"seat:node-build-image.steps",
"mesh-delivery-builder.builder_versions"
+11 -35
View File
@@ -1,5 +1,5 @@
// Package takein is the indexer's take-in contract (novox/hq ADR 0633 §1, §5): the events it publishes on its own stream
// `take-in` while a source revision waits for the indexer it names or a delivery group is gathered, their keys, and the
// `take-in` while a source revision waits (its message unread, lost or held for room) or a delivery group is gathered, their keys, and the
// answer of its tool indexer_pending. The indexer writes them; the watchman and the view read them. It is the indexer's
// own package: one module owns these types, so they live with it and not in mesh-sdk (the operator, 2026-11-04).
package takein
@@ -118,7 +118,7 @@ func checkTakeInKey(key string) error {
}
// TakeIn is the body of a take-in event. Kind is the event's kind and Key its key. What says what waits: a revision
// (Key its oldest merge's PendingKey, WaitsFor the indexer build it names) or a group being gathered (Key its
// (Key its oldest merge's PendingKey) or a group being gathered (Key its
// GroupTakeInKey, Members every approved member with its head, Merges the members held so far). Merges are the merges
// it holds, oldest first; Since is when the wait began. A `taken` event names the same once the wait ended; By and Why
// name a hand act (a hand take-in, or a drop, Dropped), and are carried by nothing else.
@@ -138,16 +138,9 @@ type TakeIn struct {
// Lost are the keys of merges whose message the forge bridge's stream no longer holds: they are not taken in, and the
// revision waits until indexer_drop_pending drops them (issue 1585).
Lost []string `json:"lost,omitempty"`
// Refused is why the manifest of the indexer the revision names was refused when the revision was worked out: with
// the seat definitions the revision carries (ADR 0633 §1), the manifest validator's problems. The revision waits for
// an indexer whose manifest is accepted, and says why here, as it says what it lost.
// Refused is why the revision is refused: the cycle with a build edge its catalog graph would hold (design 47 G §49). The
// revision waits, and says why here, as it says what it lost.
Refused string `json:"refused,omitempty"`
// Failed is why the revision's validation could not validate (design 47 G §713): the build seat's words, never a refusal
// and never passed; it is asked again by a later merge of the branch and by indexer_validate_again.
Failed string `json:"failed,omitempty"`
// Unknown is why the indexer holds a merge it does not take in although no indexer build is named: it cannot learn the build
// it runs itself (ADR 0633 §1), so Since is when it began not to know. A pending revision names WaitsFor or Unknown.
Unknown string `json:"own-build-unknown,omitempty"`
// Branch, Source and Entries are what a take-in took in (design 47 G §476, §493, §648): the watched branch, the source
// revision it makes of that branch, and one entry per module it wrote. Every taken event of one take-in carries the same;
// a taken event that ends a wait without a take-in (a group's end, a drop) carries none and starts no pipeline.
@@ -184,8 +177,8 @@ func (t TakeIn) check() error {
if !strings.HasPrefix(t.Key, "m.") {
return fmt.Errorf("delivery: a revision's take-in key is its oldest merge's m.<sequence>, not %q", t.Key)
}
if t.Kind == TakeInPending && t.WaitsFor == "" && t.Unknown == "" && len(t.Lost) == 0 && t.Refused == "" && t.Failed == "" {
return fmt.Errorf("delivery: a pending revision %s names the indexer build it waits for, why the indexer does not know its own, the merges it lost, its refusal or why it could not validate", t.Key)
if t.Kind == TakeInPending && len(t.Lost) == 0 && t.Refused == "" {
return fmt.Errorf("delivery: a pending revision %s names the merges it lost or its refusal", t.Key)
}
case TakeInGroup:
if t.Group == "" {
@@ -266,32 +259,15 @@ type PendingWait struct {
Unread []string `json:"unread,omitempty"`
// Refused is why the manifest of the indexer it names was refused (TakeIn.Refused).
Refused string `json:"refused,omitempty"`
// Validation is the revision's validation whole, as its newest pending value keeps it (design 47 G §713, A-247).
Validation *Validation `json:"validation,omitempty"`
}
// Validation is a pending revision's validation (design 47 G §712, §713, A-232, A-247), kept on the revision's newest pending
// value alone: the run ask's id, when it was asked, its input's digest, the sequence of the newest merge it covers and the try
// number for it, why it could not validate, and the sequence it was accepted at (never asked again once set).
type Validation struct {
Ask string `json:"ask,omitempty"`
Asked time.Time `json:"asked,omitzero"`
Input string `json:"input,omitempty"`
Seq uint64 `json:"seq"`
Try int `json:"try"`
Failed string `json:"failed,omitempty"`
Accepted uint64 `json:"accepted,omitempty"`
}
// PendingAnswer is indexer_pending's answer: every wait, oldest first, and the pending keys that do not read.
type PendingAnswer struct {
Waits []PendingWait `json:"waits"`
Unread []string `json:"unread,omitempty"`
// OwnBuildUnknown is why the running indexer does not know its own build, and since when: while it is set, every merge waits
// (a take-in.pending with the same words) and the indexer is started again to learn it. Empty when the build is known.
OwnBuildUnknown string `json:"own-build-unknown,omitempty"`
OwnBuildSince time.Time `json:"own-build-since,omitzero"`
// OwnBuild is the build the running indexer knows it runs, fixed at its start; OwnBuildSince is since when. Empty when unknown.
Waits []PendingWait `json:"waits"`
Unread []string `json:"unread,omitempty"`
OwnBuildSince time.Time `json:"own-build-since,omitzero"`
// OwnBuild is the build the running indexer knows it runs, fixed at its start; OwnBuildSince is since when. Empty when unknown:
// it names only the writer of what the indexer writes, and nothing waits for it.
OwnBuild string `json:"own-build,omitempty"`
}
@@ -21,10 +21,7 @@ func sampleTakeIns(since time.Time) []TakeIn {
Members: []meshdeliveryclient.Head{{Repository: "novox/mesh-delivery", Number: 9, Head: "ffff"}, {Repository: "novox/mesh-sdk", Number: 37, Head: "dddd"}}, Since: since},
{Kind: TakeInTaken, Key: PendingKey(12), What: TakeInRevision, Merges: m, WaitsFor: "src2:abc", Since: since,
By: "lead-47@shanks", Why: "the named indexer can never run"},
{Kind: TakeInPending, Key: PendingKey(13), What: TakeInRevision, Merges: m, Unknown: "the engine does not answer", Since: since}, // the indexer cannot learn its own build
// design 47 G §713: a revision refused by its validation, and one that could not validate.
{Kind: TakeInPending, Key: PendingKey(14), What: TakeInRevision, Merges: m, Refused: "a: names no version", Since: since},
{Kind: TakeInPending, Key: PendingKey(15), What: TakeInRevision, Merges: m, Failed: "could not run: no build node", Since: since},
{Kind: TakeInPending, Key: PendingKey(14), What: TakeInRevision, Merges: m, Refused: "a cycle with a build edge", Since: since},
}
}
@@ -15,6 +15,8 @@ mesh-catalog b5dbfbb784507e211372a7c9640d91f7d6dadd84 modules/*/module.json, m
mesh-host 233bb23bbcb1a829ce2a6164a31a5c5cc4f3d950 internal/apply/placement_guard.go
mesh-catalog 2d5358b8 (milestone 47/G's AM-K1) modules/zsh/module.json alone: the login shell uses operator-channel for
its seat's marked execute (design 46 approval mark §21), which the validator requires (§4)
mesh-catalog cd3b8fe8 (restic's secrets-owner, ADR 0680) modules/restic/module.json and modules/restic/files/** alone: its
processes run as the operator account, so the validator (ADR 0680 §3) needs the secrets-owner it names
To move them, from mesh-delivery's modules/mesh-delivery-indexer directory, with the two repositories checked out
beside mesh-delivery:
@@ -0,0 +1 @@
${config:put-off-bound}
@@ -1,6 +1,9 @@
{
"module": "restic",
"version": "1",
"version": "0.1",
"packages": {
"git.novox.be/novox/mesh-catalog/modules/disk-load/diskreadings": "0.1"
},
"moved": true,
"claims": [
{
@@ -13,13 +16,27 @@
]
}
],
"invokes": [
"seat:mesh-module-assigner.assignments",
"seat:mesh-broker.show"
],
"consumes": [
"mesh-recovery.restore-started",
"mesh-recovery.restored",
"mesh-conditions.findings-asked"
],
"emits": [
"backed-up"
],
"tools": [
"backup_health",
"backup_delete_retired",
"backup_deleted"
],
"secrets-owner": "${machine:account}",
"own-secrets": {
"repository": "${dir:state}/repository.secret"
"repository": "${dir:state}/repository.secret",
"broker": "${dir:state}/broker"
},
"data": {
"own": [
@@ -38,40 +55,48 @@
}
]
},
"config": {
"keep-daily": {
"type": "number",
"description": "How many nightly restore points are kept, a count from 1 to 9999: two weeks undo a mistake noticed within a fortnight. Read by the night, which thins nothing while it does not read (novox/hq issue 416); lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 14
},
"keep-weekly": {
"type": "number",
"description": "How many weekly restore points are kept behind the dailies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 8
},
"keep-monthly": {
"type": "number",
"description": "How many monthly restore points are kept behind the weeklies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 6
},
"keep-hand-taken-days": {
"type": "number",
"description": "How many days a restore point taken by hand (node-backup.now, or the mesh's before a module moves) is kept at least, a count from 1 to 9999; it is kept until a later good night of its module exists, and this many days in any case, so the next thinning never takes the one taken against a risky change (novox/hq issue 453). Lowering it lets go of the older ones taken by hand at the next thinning, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 7
}
"config": {
"keep-daily": {
"type": "number",
"description": "How many nightly restore points are kept, a count from 1 to 9999: two weeks undo a mistake noticed within a fortnight. Read by the night, which thins nothing while it does not read (novox/hq issue 416); lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 14
},
"keep-weekly": {
"type": "number",
"description": "How many weekly restore points are kept behind the dailies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 8
},
"keep-monthly": {
"type": "number",
"description": "How many monthly restore points are kept behind the weeklies, a count from 1 to 9999, refused by the night like keep-daily; lowering it deletes the restore points beyond the new count at the next night, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 6
},
"keep-hand-taken-days": {
"type": "number",
"description": "How many days a restore point taken by hand (node-backup.now, or the mesh's before a module moves) is kept at least, a count from 1 to 9999; it is kept until a later good night of its module exists, and this many days in any case, so the next thinning never takes the one taken against a risky change (novox/hq issue 453). Lowering it lets go of the older ones taken by hand at the next thinning, and that cannot be undone.",
"minimum": 1,
"maximum": 9999,
"trusted": true,
"default": 7
},
"put-off-bound": {
"type": "number",
"description": "minutes an item put off for a restore of the recovery module is waited for before the finding backup-waits is written and the wait ends, so the night never holds its lock without a bound; timed by a bound key of the state put-off (design 47 G sections 561, 557, A-155).",
"minimum": 1,
"maximum": 10080,
"trusted": false,
"default": 60
}
},
"resources": [
{
"id": "state",
@@ -108,6 +133,13 @@
"trusted": true,
"template": "files/retention.conf"
},
{
"id": "put-off-bound",
"type": "file",
"path": "${dir:state}/put-off-bound",
"mode": "0600",
"template": "files/put-off-bound"
},
{
"id": "tool",
"type": "package",
@@ -133,7 +165,32 @@
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
"MESH_BACKUP_DATA": "${dir:state}/data.conf",
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_BACKUP_PUT_OFF_BOUND_FILE": "${dir:state}/put-off-bound"
},
"health": {
"kind": "unit"
}
},
{
"id": "awake",
"type": "process",
"name": "restic-awake",
"artifact": "tools",
"run": [
"./restic-backups",
"awake"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf",
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_BACKUP_PUT_OFF_BOUND_FILE": "${dir:state}/put-off-bound"
},
"health": {
"kind": "tool",
@@ -143,27 +200,6 @@
"looks": 2,
"grace": "60s"
}
},
{
"id": "measure",
"type": "process",
"name": "restic-measure",
"artifact": "tools",
"run": [
"./restic-backups",
"measure"
],
"user": "${machine:account}",
"env": {
"MESH_BACKUP_DECLARED": "${dir:state}/backups.conf",
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
},
"health": {
"kind": "unit"
}
}
],
"build": {
@@ -183,7 +219,8 @@
"MESH_BACKUP_REPOSITORY": "${dir:repository}",
"MESH_BACKUP_PASSWORD_FILE": "${dir:state}/repository.secret",
"MESH_BACKUP_STATE": "${dir:state}",
"MESH_BACKUP_DATA": "${dir:state}/data.conf"
"MESH_BACKUP_DATA": "${dir:state}/data.conf",
"MESH_BACKUP_PUT_OFF_BOUND_FILE": "${dir:state}/put-off-bound"
}
},
{
@@ -192,5 +229,40 @@
"from": "files"
}
]
}
},
"reads": [
"disk-load.readings"
],
"state": [
"good"
],
"requires": [
{
"name": "put-off",
"storage": "key-value",
"template": "state",
"options": {
"key-ttl": true
}
},
{
"name": "findings",
"storage": "key-value",
"template": "state"
},
{
"name": "finding-events",
"storage": "stream",
"template": "events",
"options": {
"events": [
"finding-written.*",
"finding-cleared.*",
"findings-stated.*"
],
"max-age-days": 7,
"max-mib": 16
}
}
]
}
@@ -0,0 +1,140 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
seat "git.novox.be/novox/mesh-delivery/modules/mesh-delivery-contracts/go/seats/meshpipelinerunner"
stdio "git.novox.be/novox/mesh-sdk/go"
)
// CancelTool is runner_cancel: the runner's seat verb cancel, reachable as a tool (design 47 G §482). It lists the pipelines
// through the bundle's reader (the one the verb list uses), picks the running ones named, or the running ones of a branch
// whose source is below a bound, and asks the seat verb cancel for each over the bus; the verb records the hand act and says
// pipeline-ended. A pipeline that is not running is never asked.
func CancelTool(list func(seat.ListInput) (seat.ListAnswer, error), ask func(key string, body any) (json.RawMessage, error)) stdio.Tool {
return stdio.Tool{Name: Prefix + "_cancel",
Description: "Cancel running pipelines by the seat verb cancel: those named, or those of a branch (default main) whose source is below below_source; exactly one of the two. It names why, and dry_run lists what it would cancel. A pipeline that is not running is refused, never asked.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"pipelines": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
"below_source": map[string]any{"type": "integer"},
"branch": map[string]any{"type": "string"},
"why": map[string]any{"type": "string"},
"dry_run": map[string]any{"type": "boolean"},
}, "required": []string{"why"}},
Run: func(args map[string]any) (any, error) {
in, err := as[CancelInput](args)
if err != nil {
return nil, err
}
return in.run(list, ask)
}}
}
// CancelInput is what runner_cancel takes.
type CancelInput struct {
Pipelines []string `json:"pipelines,omitempty"`
BelowSource *int64 `json:"below_source,omitempty"`
Branch string `json:"branch,omitempty"`
Why string `json:"why"`
DryRun bool `json:"dry_run,omitempty"`
}
// CancelResult is one pipeline's end of runner_cancel: cancelled, would-cancel (dry run) or refused whole.
type CancelResult struct {
Pipeline string `json:"pipeline"`
Result string `json:"result"`
Refusal string `json:"refusal,omitempty"`
// Note says what an ask that did not answer left: the pipeline read again was cancelled (A-472).
Note string `json:"note,omitempty"`
}
func (in CancelInput) run(list func(seat.ListInput) (seat.ListAnswer, error), ask func(string, any) (json.RawMessage, error)) (any, error) {
if strings.TrimSpace(in.Why) == "" {
return nil, errors.New("cancel names why")
}
if (len(in.Pipelines) > 0) == (in.BelowSource != nil) {
return nil, errors.New("cancel takes exactly one of pipelines and below_source")
}
branch := in.Branch
if branch == "" {
branch = "main"
}
listed := seat.ListInput{}
if in.BelowSource != nil {
listed.Branch = branch
}
all, err := list(listed)
if err != nil {
return nil, fmt.Errorf("the pipelines are not read: %w", err)
}
byKey := map[string]seat.Pipeline{}
for _, p := range all.Pipelines {
byKey[p.Pipeline] = p
}
var results []CancelResult
var picked []string
if in.BelowSource != nil {
for _, p := range all.Pipelines {
if p.Outcome == seat.PipelineOutcomeRunning && p.Source > 0 && p.Source < *in.BelowSource {
picked = append(picked, p.Pipeline)
}
}
} else {
for _, name := range in.Pipelines {
p, ok := byKey[name]
switch {
case !ok:
results = append(results, CancelResult{Pipeline: name, Result: "refused", Refusal: "no such pipeline"})
case p.Outcome != seat.PipelineOutcomeRunning:
results = append(results, CancelResult{Pipeline: name, Result: "refused", Refusal: "the pipeline is " + p.Outcome + ", not running"})
default:
picked = append(picked, name)
}
}
}
counts := map[string]int{}
for _, name := range picked {
if in.DryRun {
results = append(results, CancelResult{Pipeline: name, Result: "would-cancel"})
} else if _, err := ask("seat:"+seat.Seat+"."+seat.VerbCancel, seat.CancelInput{Pipeline: name, Why: in.Why}); err != nil {
results = append(results, in.afterUnanswered(name, err, list))
} else {
results = append(results, CancelResult{Pipeline: name, Result: "cancelled"})
}
}
for _, r := range results {
counts[r.Result]++
}
return map[string]any{"results": results, "counts": counts}, nil
}
// afterUnanswered reads a pipeline's state after an ask that did not answer (A-472): the verb may have written the end and
// said it before the answer was cut, so a pipeline found cancelled is reported cancelled, with what the ask said; one still
// running, or not read, is refused with both.
func (in CancelInput) afterUnanswered(name string, asked error, list func(seat.ListInput) (seat.ListAnswer, error)) CancelResult {
branch := in.Branch
if branch == "" {
branch = "main"
}
listed := seat.ListInput{}
if in.BelowSource != nil {
listed.Branch = branch
}
all, err := list(listed)
if err != nil {
return CancelResult{Pipeline: name, Result: "refused", Refusal: asked.Error() + "; the pipeline is not read again: " + err.Error()}
}
for _, p := range all.Pipelines {
if p.Pipeline != name {
continue
}
if p.Outcome == seat.PipelineOutcomeCancelled {
return CancelResult{Pipeline: name, Result: "cancelled", Note: "the ask did not answer (" + asked.Error() + "), and the pipeline read again is cancelled"}
}
return CancelResult{Pipeline: name, Result: "refused", Refusal: asked.Error() + "; the pipeline read again is " + p.Outcome}
}
return CancelResult{Pipeline: name, Result: "refused", Refusal: asked.Error() + "; the pipeline is not found when read again"}
}

Some files were not shown because too many files have changed in this diff Show More