Compare commits

...
Author SHA1 Message Date
jschoubben c8a434b8c6 Assigner: write the bus's user-list revision for a node whose have is of the old form (issue 1945)
At the switch the bus node's have names an old-form build and no composition, so no
revision at the fingerprint it runs could ever exist, and §15 refused the first want:
the bus's user list on novox stood still from 11:23. The revision now names F when the
controller judged F's composition to differ from the declaration the node last applied
only in the user list (ADR 0539 §7, ADR 0540 §9). Any other difference stays the
operator's by §15, and the pass now says so on neither list instead of saying nothing.
2026-10-25 20:44:27 +01:00
jschoubben 5ba6881f9f Merge pull request 'Assigner rule 6 per node: a node is granted only when the grant names its own account (issue 1945 note 35)' (#46) from fix/1945-credential-when-row-without-seal into main
Reviewed-on: #46
2026-10-25 19:31:51 +00:00
jschoubben 61196c99e6 Hold a node's want until the grant names its account (issue 1945, rule 6 per node)
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 61196c99e6a4 on 5ba6881f9fd3 | unit 163 run, 421 kept | integration 173 run, 487 kept | composition passed on ace
Rule 6 read only whether a grant of the module's fingerprint existed. A grant
written for the module's other nodes then let a node whose account was never
issued or sealed be wanted: on 2026-10-25 at 20:04 ace's mesh-build-go was
wanted with no ace.mesh-build-go on the bus, and crash-looped 20 times for
its missing credential. A grant that names accounts now grants a node only
when it names that node's own or a process's account; one that names none
grants every node, as before.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 20:13:39 +01:00
jschoubben 250364a57b Merge pull request 'The assigner follows a kept composition's new digest, so a want held at an earlier build gets today's config (issue 1945, ADR 0545)' (#45) from fix/1945-kept-compositions-follow-config into main
Reviewed-on: #45
2026-10-25 18:38:03 +00:00
jschoubben 73779a5dd1 Let a want held back at an earlier fingerprint follow that fingerprint's new revision (issue 1945, ADR 0545)
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
The controller now composes a module's kept compositions again when its config
changes, so going back keeps today's config. A want held back at an earlier
fingerprint kept the digest it was first written with, and the node applied
the old revision from the bucket's history: on g14 a rollback of mesh-build-go
put its capacity back to 0. The want now names the new digest, asked through
the record hold as any move of a record module is. A want held at F by the
rules still waits for them, as a new fingerprint does (ADR 0539 §5).
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:24:44 +01:00
jschoubben ec76ab77c0 Merge pull request 'Assigner: a held module is on the held list, never on neither, and a neither from before a restart is cleared (issue 1945)' (#44) from fix/held-is-not-neither into main
Reviewed-on: #44
2026-10-25 16:11:33 +00:00
jschoubben 91d041b62a mesh-delivery-assigner: a held module is never on neither list (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 91d041b62a1a on ec76ab77c058 | unit 163 run, 421 kept | integration 173 run, 487 kept | over bound: integration (slowest integration m…
systemd-resolved, held on shanks by the operator under rule 2, was raised
as an urgent "on no list" condition and reminded every hour, although its
hold is already said and answered on the held list. A node now goes on
neither list only when it gets no want and the module is not held, by a
hold read at its fingerprint or one this pass raises. The first pass after
a start clears the neither condition of every assigned node not on neither
list, not only those with a want, so a neither raised before the restart
for a held module (or a move waiting for its record ask) is cleared too.
2026-10-25 17:07:50 +01:00
jschoubben 51444a8341 Merge pull request 'mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the runtime's line or say so (issue 1901)' (#43) from fix/1901-package-cause into main
Reviewed-on: #43
2026-10-25 15:07:47 +00:00
jschoubben dd9138310c mesh-delivery-checker: a package sent as its FAIL line alone is still worded, and the words keep their bound exactly (issue 1901, review round 1)
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass dd9138310c48 on 51444a834173 | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
The build module on main sends a timeout, exit or kill as only 'FAIL <pkg>':
dropping summary lines left empty words. They now say the package failed with
no cause sent. Causes match at column 0 only, summary lines only in go's exact
forms, and the words count the cause, the marker and the newlines within
MaxWords.
2026-10-25 15:58:09 +01:00
mesh-tools 653da54c0d Merge pull request 'Assigner: a running node gets its first want past the ordering rules, a module on neither list is raised, and a failed health holds only once a node is done moving (issue 1945)' (#42) from fix/r6-no-module-in-neither into main 2026-10-25 14:56:26 +00:00
jschoubben f72f57f6e2 mesh-delivery-checker: with no cause told apart, a race report is the cause (issue 1901)
An older build module's output for a race after the tests' PASS is now worded
by the race's heading, access and frame, not by go test's PASS or the report's
separator line (1901/AC5).
2026-10-25 15:52:21 +01:00
jschoubben 975f0ed654 Assigner: a hold lifts over haves not running, the first pass after a start clears stale conditions, and a first want after a removed want is asked (issue 1945, review round 2 of 3d7d65ed)
mesh/delivery merged: source.main moved to novox/mesh-delivery@653da54c0da3
mesh/check pass 975f0ed65460 on 653da54c0da3 | unit 154 run, 419 kept | integration 164 run, 485 kept | composition passed on ace
Phase guard, same kind as the failed-health hold. allRunHealthy counted
every have whatever its phase, so a node with a removed, held or still
moving have kept a hold up for ever: it counts only phase running (healthy
or not) and failed (not healthy). The failed-health hold already ignores
removed and held haves (phase running or failed only).

Stale conditions (B1). The neither and held conditions this process said
lived only in its memory, so one raised by a previous process stayed urgent
for ever once the module had its want or was no longer held. The first pass
after a start clears, once, the neither condition of every assigned module
with a want and not on neither list, and the held condition of every
module's newest fingerprint that is not held; a clear of a condition that is
not open is quiet at the router. A hold raised again forgets that it was
cleared, so a later release clears it again.

Record ask (review note 3). mustAsk wrote no ask when the want before was
removed, so the first-want path could write a recreates or moves-data
composition of a policy-record module with no ask. A removed want is now
no want, and the first want after it goes through the record ask.

Tests first, each seen failing: a hold lifted over removed, held and moving
haves; the record ask after a removed want; a fresh pass clearing what a
previous process raised, once.
2026-10-25 15:47:57 +01:00
jschoubben 706169aa8a Assigner: a failed health holds a fingerprint only once the node is done moving (phase running or failed)
On novox at 15:27:36 a have reading "health failed, phase applying" at the
controller's new fingerprint held its canary two seconds before the new
build was healthy: Wants held on a failed health whatever the phase. A
failed health while the node backs up, applies or checks is a step of the
move, not its outcome, so it holds nothing now.

Every other Health read in the assigner (rule 3's healthy, rule 5's
providedHealthy, the world's lastHealthy) already asks for phase running.
Test from the live row first, seen failing before the fix.
2026-10-25 15:39:45 +01:00
jschoubben 6d2d50b1e1 mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the panic line (issue 1901)
A try of mesh-controller's status read 'cmd/mesh-controller: PASS': the build
module sent a package failure whose output began with go test's PASS. The
checker now drops go test's summary lines, takes the first panic, fatal error
or signal line as the cause when the build module told none apart (read before
the words are cut), says the tests passed and the program failed after them
when nothing else is left, and cuts a cause to 1 KB so the words keep their
bound (1901/AC5).
2026-10-25 15:38:40 +01:00
jschoubben 3d7d65edce Assigner: a node running an old-form build gets its first want at F once the canary runs F healthy (issue 1945)
An old-form build (not src2:) has no composition, so no want can ever name
what such a node runs. Design limit, decided by the lead (lead-47): its first
want names the newest F, by every rule as usual - the canary (rule 3: this
node is the canary, or the canary runs F healthy), everything F needs (built,
not held, granted, composed) and the providers (rule 5). Otherwise nothing is
written and the neither condition stays. The canary stays in force, and
old-form modules (clipmenu and screen-lock on g14) leave neither list by
themselves, without a hand-written want.

The code already behaved so (an old-form have falls through to Holds); the
case is now named in Wants, and proven: a unit test for both sides of the
canary, the providers and the grant, and the integration test gives g14 F
and clears its neither condition once the canary ace runs F healthy.
2026-10-25 15:32:35 +01:00
mesh-tools 06fd53e5cb Merge pull request 'List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)' (#41) from fix/r6-hold-verbs into main 2026-10-25 14:30:12 +00:00
jschoubben 81182e9dcd Assigner: a running node's first want names the fingerprint it runs, never a newer one, and only a node that runs the module is on neither list (issue 1945, review of f7b23033)
A first want at the newest fingerprint passed over the canary for every module right after the switch. It now names what the node
runs: F when it runs F (by the rules without which there is nothing to want), or an older fingerprint of the new form whose
composition and grant exist, held back from F by the rule; otherwise nothing is written and the node stays on neither list. A new
assignment, with no have, waits for the rules and raises no condition (1945/AC19-AC21).
2026-10-25 15:26:03 +01:00
jschoubben f7b23033a2 Assigner: the integration test of wants on the bus proves a running node's first want and the neither condition; held-condition reads filter by key (issue 1945, try-1792937443523744996) 2026-10-25 15:18:07 +01:00
jschoubben cf26e95691 A view answer rewritten with the same option changes nothing; pin a unit test's clock against the integration tests' (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@06fd53e5cbf7
mesh/check pass cf26e956915c on 06fd53e5cbf7 | unit 154 run, 408 kept | integration 164 run, 474 kept | composition passed in 1ms on 2 machines
2026-10-25 15:15:40 +01:00
jschoubben 8daa5977f9 Assigner: a node running a module with no want gets its first want past the ordering rules, and a module on neither list is raised (issue 1945)
The node's own composition gives up every module with a fingerprint of the new form, so such a module is applied only through its
want. The assigner wrote no want while a rule held it back and the node had none, so clipmenu, screen-lock, i3status-rust and every
module requiring x11-display lay on neither list on g14 and shanks: xorg's health is unknown (rule 5) and the canary never ran them
healthy (rule 3), and their files were removed and never made again.

A node that runs a module and has no want for it now gets its first want by the rules without which there is nothing to want (built,
not held, granted, composed); the canary and providers rules order every move after it. Any module still on neither list is raised
as an urgent condition naming module, node and rule, written to the journal, and cleared once it has a want (1945/AC19-AC21).
2026-10-25 15:10:34 +01:00
jschoubben 30ca8a22bc List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)
A hold answered keep on the phone could not be changed: its ask ended with the
answer, nothing listed it, and a kept hold is never lifted by itself. The view
lists every hold with its node, cause and answer (mesh-delivery-cli holds) and
answers the standing hold ask with hold-release or hold-keep through its one
write; the teller takes a later view answer that changes a hold's option into
the answer of that asking, never an older one.
2026-10-25 15:09:40 +01:00
mesh-tools 97e605edb0 Merge pull request 'Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, and keep its cause (issue 1906)' (#40) from fix/r6-hold-asks into main 2026-10-25 13:27:29 +00:00
jschoubben 7ef796b04b Format the assigner's integration test (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@97e605edb0ad
mesh/check pass 7ef796b04b92 on 97e605edb0ad | unit 154 run, 404 kept | integration 164 run, 469 kept | composition passed on 2 machines
2026-10-25 14:16:58 +01:00
jschoubben ec93585736 Ask a hold again after a restart, leave a merge ask room, take an answer only for its asking (issue 1906, review)
A hold written again after a lift and a restart reused its first ask id, which
the router drops; the hold's time is now its asking. Seven holds could take the
router's three places for the teller and keep a merge Approve from being asked;
the teller keeps at most two assigner asks open. A release or keep answered for
one hold carried over to a later hold of the same fingerprint; an answer now
counts only for the asking its warrant names.
2026-10-25 14:15:44 +01:00
jschoubben 6b63f13da6 Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, keep the hold's cause (issue 1906)
Seven modules stayed held on every machine and the operator was never asked:
the teller only filled assigner asks from the view, a single failed check held
a fingerprint for good, and the held record kept no cause. The teller now asks
every assigner ask as it asks merge asks and writes the phone's answer under
the ask's key; the assigner lifts a hold whose fingerprint runs healthy on
every node that runs the module, says so once and takes its ask back; the held
record names health, phase, when and the node's words.
2026-10-25 14:06:58 +01:00
jschoubben 2268f94ef2 Merge pull request 'The view lists a node's own want among the assigner's wants (issue 1945)' (#39) from fix/r6-node-composition into main
Reviewed-on: #39
2026-10-25 00:18:11 +00:00
jschoubben 48aaecf556 List a node's own want among the assigner's wants, never as unreadable (issue 1945)
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking 48aaecf556c5 on 650a06ceab1d | novox/mesh-delivery#39 in group fix/r6-node-composition
2026-10-25 01:50:00 +02:00
jschoubben 650a06ceab Merge pull request 'R6: the assigner writes wants by rule 7 with the composition digest; the watchman and the view follow wants and haves (issue 1906)' (#38) from feat/r6 into main
Reviewed-on: #38
2026-10-24 23:17:46 +00:00
jschoubben 3e863884fd gofmt the passer's fields after main's merge
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/delivery delivering: 0 machine step(s) passed
mesh/check checking 3e863884fda8 on 650a06ceab1d | novox/mesh-delivery#38 in group feat/r6
2026-10-24 23:33:59 +02:00
jschoubben c21bed20a3 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943) 2026-10-24 23:06:02 +02:00
jschoubben aa31e85665 Merge main into feat/r6: keep R4g's composer at a group's mesh-sdk head beside R6's wants and asks
The assigner reads mesh-controller.builds again (the composer's runningCommits reads the build record of every open
assignment), puts its compose asks on node-build-go and says its conditions on operator-channel.
2026-10-24 23:03:42 +02:00
jschoubben 171bddacab Merge pull request 'R4g finish: the composer as a job on node-build-go's check stream, merge-check.sh gone, reads-outside markers (issues 1295, 1621-1623, 1935)' (#35) from feat/r4g-finish into main
Reviewed-on: #35
2026-10-24 20:57:15 +00:00
jschoubben 65496fc4f7 Merge main into feat/r4g-finish, keeping #37's same-branch mesh-sdk pin rule beside the node-engine check's controller placing (issue 1295)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking 65496fc4f7a6 on 171bddacabe2 | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 21:54:46 +02:00
jschoubben 36a1fd197f The switch exception skips the record hold only for a composition the controller judges unchanged; pin mesh-sdk d7a9b96 (issue 1906, ADR 0540 §9) 2026-10-24 21:18:48 +02:00
jschoubben 9a5b3d4881 Pin the assigner, watchman and view to mesh-sdk feat/r6 7f9cc0e (issue 1943) 2026-10-24 20:43:46 +02:00
jschoubben e263049551 Right after the switch a node running F with a have that names no composition is given its want at F with no record hold, the bus included (issue 1906, review round 2 N1) 2026-10-24 20:35:17 +02:00
jschoubben 0f7bc9f531 Pin the assigner, watchman and view to mesh-sdk feat/r6 e45bdc0 (issue 1943) 2026-10-24 20:21:57 +02:00
jschoubben da31740f00 The view's motion events carry the time they are about and call a held or past-bound module stopped, as the messenger reads them (issue 1906 D10) 2026-10-24 18:31:29 +02:00
jschoubben a3815051cc The view says modules in motion: moving, settled and stuck, once for a node, a module and a composition (issue 1906 D10, for the messenger) 2026-10-24 18:27:01 +02:00
jschoubben 8280506cee Start the world's change counter, which the pass of a held backup reads (issue 1906, review round 1) 2026-10-24 18:22:22 +02:00
jschoubben 3b00f3890d Review round 1: a user-list-only revision of the bus is written at once at the fingerprint it runs, before the record hold; pin mesh-sdk a523a8c (issue 1906) 2026-10-24 18:17:25 +02:00
jschoubben fcac03a4f2 Review round 1: the bus with no want is compared with its have; a backup ask is bound to its composition and number and each answer is a distinct want value; the view reads wants and haves, with dotted module names (issue 1906) 2026-10-24 18:15:13 +02:00
jschoubben 14c20b21ee Pin the assigner, watchman and view to the head of mesh-sdk feat/r6 (issue 1943) 2026-10-24 17:52:44 +02:00
jschoubben 333e0cd07d The checker reads a kept manifest as the indexer and the assigner do, so a value falling back to a predefined key is not logged unreadable (issue 1906, issues 1751 and 1921 class) 2026-10-24 17:42:24 +02:00
jschoubben 03e5951dc5 Merge remote-tracking branch 'origin/main' into feat/r6 2026-10-24 17:41:53 +02:00
mesh-tools 4c1d36c9cd Merge pull request 'A try accepts a mesh-sdk pin that is the head of mesh-sdk's branch of the same name, so a group's members can be tried before its pull requests (issue 1921)' (#37) from fix/1921-try-same-branch-pin into main 2026-10-24 15:37:07 +00:00
jschoubben 37d0ebba7a Let the view's manifest test read the emits key the delivered event adds 2026-10-24 16:47:33 +02:00
jschoubben 86fd0bbe96 The view says its event delivered once for each source revision whose wants and haves are all there (issue 1906 D9, for issue 1946) 2026-10-24 16:47:09 +02:00
jschoubben 37a5ab8f1c A node that runs a module of policy record with no want yet is asked like any other, not treated as one that never ran it (issue 1906 D4) 2026-10-24 16:43:01 +02:00
jschoubben 55f19283f4 Do not wait twice at the cleanup for an assigner that ended on its own, which made the refused-start test take ten seconds a bucket 2026-10-24 16:42:28 +02:00
jschoubben e7a3ff1e7e The record policy is the one the controller holds beside the composition, the operator's own choice included; pin mesh-sdk 64a502d (issue 1906 D4, ADR 0539 §14) 2026-10-24 16:41:58 +02:00
jschoubben 991589fdd2 A try of a branch accepts a mesh-sdk pin at the head of mesh-sdk's branch of the same name, so a milestone's repositories can be tried before their pull requests (issue 1921)
mesh/delivery delivered
mesh/check checking 991589fdd20a on 4c1d36c9cd7f
2026-10-24 16:41:39 +02:00
jschoubben f947070b7b The view shows the controller's judgement of the move beside each want (issue 1906 D7, ADR 0539 §14) 2026-10-24 16:38:24 +02:00
jschoubben 04c2c60f02 The assigner acts on answers: a release writes the held want again, a backup answer goes into the node's want, a record move waits for its yes, a restore answer sets a data module back (issue 1906 D4) 2026-10-24 16:37:37 +02:00
jschoubben 0df20b5d77 The assigner's seventh rule: a want names the digest of its composition and waits for it; a revision at the same fingerprint is a new want; the node's own want (issue 1906 D2, ADR 0539 §6) 2026-10-24 16:33:48 +02:00
jschoubben 806c8bf0e4 Pin the assigner, watchman and view to mesh-sdk's release 6 head (issue 1943 S3) and give their test wants a composition digest 2026-10-24 16:32:09 +02:00
jschoubben 12b01badb0 Prove the assigner replaced mid-pass resumes with nothing lost or written again (issue 1906 D8) 2026-10-24 16:27:34 +02:00
jschoubben 7b18696cd5 Give the data module's test manifest the directory and backup the catalogue asks of an irreplaceable item 2026-10-24 16:26:43 +02:00
jschoubben 204a59dab4 The assigner asks whether to restore a data module whose held fingerprint failed its health check, and raises the condition for a held fingerprint (issue 1906 D3) 2026-10-24 16:26:20 +02:00
jschoubben f048e2b443 The view shows each want beside the have of its node, and a source revision's derived state, in place of the comparison with sends (issue 1906 D7) 2026-10-24 16:23:51 +02:00
jschoubben 9d2e9dc6c9 The watchman mirrors each want of the assigner as a pulse owed by the node's agent until the have names its fingerprint (issue 1906 D6) 2026-10-24 16:22:02 +02:00
jschoubben 024a3787da End the assigner's comparison with the send: the controller sends nothing to compare with, so the differences bucket is kept read-only (issue 1906 D5, ADR 0539 §8) 2026-10-24 16:20:43 +02:00
jschoubben 4fd6b934c8 Read the assigner's wants, differences and the haves with tools, so the gap between wants and sends can be read live (issue 1906 D1) 2026-10-24 16:19:42 +02:00
mesh-tools fa46e7df7d Merge pull request 'The assigner reads a change's manifests as the indexer does, so a value falling back to a predefined key no longer fails every composition touching it' (#36) from fix/assigner-predefined-fallback into main 2026-10-24 14:09:58 +00:00
jschoubben 18b522184f Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds: a group touching pacman or xterm failed its composition on it (ADR 0348 §7, issue 1751)
mesh/delivery delivered
mesh/check pass 18b522184f73 on fa46e7df7dd1 | unit 128 run, 375 kept | integration 138 run, 426 kept | composition passed on ace
2026-10-24 15:03:14 +02:00
jschoubben 1d4dcdf41a Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds (ADR 0348 §7, issue 1751); pin mesh-sdk bfc3883a
mesh/delivery-group group feat/r4g-finish rejected: composed together: the checker's verdict at the members' heads (171a15384fa1): shanks
mesh/delivery superseded: a newer head of the same pull request
mesh/check checking 1d4dcdf41a22 on 4c1d36c9cd7f | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 14:57:36 +02:00
jschoubben f5fd74380c Pin to the group's new mesh-sdk head 19c1d359, whose composer names the reader's pinned SDK instead of a bare (devel) (ADR 0482 clause 5)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk 19c1d35939bc, which is neither on mesh-sdk's t…
2026-10-24 14:12:10 +02:00
jschoubben e78bbfbdee Pin the assigner to the group's mesh-sdk head after main's merge, so it composes with the SDK the group delivers
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's t…
2026-10-24 13:28:42 +02:00
jschoubben ce7dcccb1d Merge main into feat/r4g-finish (#33, #34) 2026-10-24 13:26:24 +02:00
jschoubben 42d880f075 Merge pull request 'Count a node's toolchain report only while its build module is assigned there, so a removed node no longer makes builders disagree (issue 1940)' (#34) from fix/1940-stale-toolchain into main
Reviewed-on: #34
2026-10-24 11:10:45 +00:00
jschoubben 3553cbaf08 Count only the toolchain reports of nodes a build module is assigned to, so a node it left does not make the builders disagree (issue 1940)
mesh/delivery delivered
mesh-build-node was unassigned from ace on 2026-10-23 and its ace.typescript key stayed in
mesh-build-node_toolchains at v26.12.0: nothing deletes it, so the indexer, the builder and the
checker took it for a builder and every Node fingerprint said 'builders disagree'. The three now
read mesh-controller_assignments and pass a toolchain report on only while its build module has an
open assignment on its node; builders that run different versions still disagree.
2026-10-24 12:48:41 +02:00
jschoubben d8b19cb332 Merge pull request 'Build a done key again when the store no longer holds its blob (issue 1937)' (#33) from fix/1937-done-key-blobs into main
Reviewed-on: #33
2026-10-24 10:35:21 +00:00
jschoubben 2259d98c5b Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:39:03 +02:00
jschoubben 4702c6bd77 Pin to the finish group's mesh-sdk head dd0d156b, so every member names one SDK (issue 1622) 2026-10-24 04:28:15 +02:00
jschoubben 23f4304d70 Merge remote-tracking branch 'origin/feat/r4g-finish-composer' into HEAD 2026-10-24 04:26:19 +02:00
jschoubben 28ee22ef27 Give the integration test's EVENTS stream the artifacts-gone subject alone, so it does not overlap the build seats' streams (issue 1937)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792840300970624489 (novox/mesh-delivery at 42d880f0 into main, d…
mesh/delivery-group group fix/1937-done-key-blobs closed: a member was closed before the group merged
mesh/check pass 28ee22ef27e2 on d8b19cb332b6 | unit 126 run, 370 kept | integration 136 run, 420 kept | composition passed on shanks
2026-10-24 04:25:19 +02:00
jschoubben f0c5be8c30 Say which artifacts the store check asks about: bundles and archives, whose blobs the collector deletes (issue 1937) 2026-10-24 04:19:17 +02:00
jschoubben 80a0183567 Build a done key again when the artifact store no longer holds its blob, and on the controller's artifacts-gone (issue 1937)
The store's nightly collector deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which
the controller had never taken in, and the builder kept them done: their next delivery would refuse. The builder now asks
the controller's held before it leaves a done trunk key done (once per key per process, at its start and on each module
value), claims a key whose blob is gone building as a new revision and says so in its log and the key's step, and hears
artifacts-gone, said when a take-in found a gone blob. No timer: a start, a value or an event.
2026-10-24 04:17:01 +02:00
jschoubben c256d732f4 Pin the assigner to mesh-sdk's head that reads a composer request from a file (issue 1622) 2026-10-24 04:14:33 +02:00
jschoubben 93427c08e7 Ask the Go build seat for the composer at a group's mesh-sdk head and at its readers' mesh-sdk, and follow each step (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben c58342786f Judge a group with a mesh-sdk member from the composer at its head, and fail a field its readers cannot read (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben 840e56eddb Fail the node-engine's check when the controller's trunk is unknown, so its step-nine test never skips (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 90fd07ac26 Merge pull request 'Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)' (#32) from fix/1933-rebuild-indexer into main
Reviewed-on: #32
2026-10-23 23:38:21 +00:00
jschoubben 90a79b1585 Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)
mesh/delivery delivered
mesh/check pass 90a79b15854b on 90fd07ac260a | unit 126 run, 366 kept | integration 136 run, 414 kept | composition passed on novox
2026-10-24 01:35:33 +02:00
jschoubben 45ab994777 Merge pull request 'The checker keeps a failed test's whole message with its cause first, and its status names the cause (issue 1901, with mesh-catalog)' (#30) from fix/1901-rebased into main
Reviewed-on: #30
2026-10-23 22:36:03 +00:00
jschoubben 55f486e040 Merge pull request 'The indexer keeps an unreadable manifest's inputs, so an SDK upgrade judges it again at start (issue 1923)' (#29) from fix/1923-stale-none2 into main
Reviewed-on: #29
2026-10-23 22:35:49 +00:00
jschoubben 7a88eb0a56 Merge pull request 'The checker reads mesh-sdk's list-form level digest, so kept verdicts hold for a check naming a package like token (issue 1930, with mesh-sdk and mesh-catalog)' (#31) from fix/1930-digest-shape into main
Reviewed-on: #31
2026-10-23 22:08:43 +00:00
jschoubben 9261347a8f Read a level's digest from mesh-sdk's list form, so kept verdicts hold for a check whose end names a package like token (issue 1930)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792794985391468127 (novox/mesh-delivery at 45ab9947 into main…
mesh/delivery-group group fix/1930-digest-shape merged: every member is merged
mesh/check checking 9261347a8f91 on 7a88eb0a5609
2026-10-23 23:37:49 +02:00
mesh-tools 263ff54a4e Merge pull request 'Integration tests that R4g never ran now pass: the assigner's fixture, the builder's restart test, four checker tests (issues 1927, 1928, 1929)' (#28) from fix/1927-assigner-fpset into main 2026-10-23 21:24:32 +00:00
jschoubben cd3e55e056 Bring the checker's integration tests to what R4g decided: Go-only repository, echoed beside, readable kept (issue 1929)
mesh/delivery delivered
mesh/check pass cd3e55e05605 on fae4d48e09a5 | unit 0 run 483 kept | int 0 run 540 kept | clone 17s | composition passed on 3 machines
2026-10-23 23:18:11 +02:00
jschoubben b303e32a41 Close the builder test's nodes from the bed's start, so no pull in hand takes the first ask (issue 1928) 2026-10-23 23:03:38 +02:00
jschoubben a31bd7d4a5 Give the assigner test's candidate the SDK's fingerprint set, which the SDK refuses without (issue 1927) 2026-10-23 22:57:05 +02:00
jschoubben 32542ebc14 Keep an unreadable manifest's inputs and judge its verdict again at start, so an upgraded SDK clears it (issue 1923)
mesh/delivery delivered
mesh/check pass 32542ebc14c3 on 7a88eb0a5609 | unit 122 run, 366 kept | integration 132 run, 414 kept | composition passed on novox
2026-10-23 22:17:09 +02:00
jschoubben a697ee86d4 Test 1901/AC3: an older pair of build module and checker still agree (issue 1901)
mesh/delivery delivered
mesh/delivery-group group fix/1901-rebased merged: every member is merged
mesh/check checking a697ee86d4af on 45ab99477701
2026-10-23 22:14:29 +02:00
jschoubben 0373a3b24e mesh-delivery-checker: a failed test's verdict keeps its last lines and the status line names its error (issue 1901) 2026-10-23 22:13:41 +02:00
jschoubben fae4d48e09 Merge pull request 'R4g+R5 (mesh-delivery): builder and checker put asks, the assigner and its composition check, the view's wants and backlog (design 47)' (#27) from feat/r4g into main
Reviewed-on: #27
2026-10-23 17:49:01 +00:00
jschoubben b2c545c40a Place mesh-controller beside the node-engine's tests, which read its manifest (issue 1295, R4g)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792777718506798022 (novox/mesh-delivery at fae4d48e into main…
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking b2c545c40aec on 85aca46348a0 | novox/mesh-delivery#27 in group feat/r4g
2026-10-23 18:36:13 +02:00
jschoubben ad26670996 Prove that a second check of the same heads keeps every package the first passed, the digest being taken inside the check ask as the build module answers it (issue 1917)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/apply.TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing: judge_directories_…
2026-10-23 17:18:37 +02:00
jschoubben fad96ae67b Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/config.TestTheUnmovedListHoldsNoModuleTheTrunksDoesNot: realtree_test.go:14: mesh-cata…
2026-10-23 17:06:09 +02:00
jschoubben b83af04f0c Read an ask's stream at take-over by its name, as the broker grants it, and wait for the ask when that read fails rather than fail a key whose build may still finish (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: novox: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group…
2026-10-23 16:59:37 +02:00
jschoubben c7bddbbeab Bound a running build step's pulse by the step's own bound from its start, since nothing rewrites the key while it runs (issue 1908) 2026-10-23 16:58:26 +02:00
jschoubben d3561923aa Say the unit level's gofmt and go vet time in the check's line, and pin the group's mesh-sdk head that carries it (issue 1887 D7-go, review round 2 item 5) 2026-10-23 16:51:55 +02:00
jschoubben d32ec6b7f8 Say what the builder, the indexer and the watchman do since R4g in their comments: no build-agent is asked, no build record moves a fingerprint, no checks-running bucket is mirrored (issue 1887, review round 2 item 6) 2026-10-23 16:47:36 +02:00
jschoubben 52d8b9b753 Say in the checker's status that a check without a verdict waits in a build seat's stream, not behind a plan key or a capacity, as A5b left it (issue 1887, review round 2 item 4) 2026-10-23 16:47:20 +02:00
jschoubben ce51457974 Prove on the real bus that a builder started again while its asks waited puts no second ask (issue 1887 AC3, review round 2 item 2) 2026-10-23 16:46:58 +02:00
jschoubben bac82bb91e Count the asks the builder puts and read their kind in its integration test, so 1887/AC1 is proven on the real bus (issue 1887, review round 2 item 3) 2026-10-23 16:46:57 +02:00
jschoubben 357e28f31d Merge remote-tracking branch 'origin/feat/r4g' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-builder/internal/build/run_test.go
2026-10-23 16:46:05 +02:00
jschoubben 5ddc221833 Bound a build or check ask's step from its start, and drop the checker's whole-check bound, so asks waiting in the streams during a large rebuild are never failed and asked twice (issue 1887, review round 2 item 1; ADR 0525 §3, §6) 2026-10-23 16:45:35 +02:00
jschoubben 37860d46a0 Stop watching the node-engine's have in the checker: a check's asks wait in the build seats' streams, which redeliver a lost node's ask, so no check fails for want of a node; presence stays for an unheard forge and the assigner's presence (issue 1912) 2026-10-23 16:41:34 +02:00
jschoubben 84b2c92ccf Put again, under its own id, an ask a build key names that neither its stream holds nor any node ran, so a builder that stopped between naming and putting it leaves no key waiting for an ask nobody put (issue 1910) 2026-10-23 16:39:56 +02:00
jschoubben c0f24f56b5 Drop the view's use of the build seats: it reads their workers and publishes no ask (issue 1887 A8a) 2026-10-23 16:35:05 +02:00
jschoubben 76bc810bda Test that an ask running when the watchman starts gets its pulse from the steps that stand (issue 1909) 2026-10-23 16:34:50 +02:00
jschoubben e927469460 Pulse a build ask only from when a node takes it, and end the pulse with its step, so a waiting ask is a backlog and not a stuck row (issue 1908) 2026-10-23 16:34:39 +02:00
jschoubben 97b42e3f0d Name in every composition, by machine, those that could run a moved or added module and are not assigned it and those whose grants alone the change alters, as not judged, as ADR 0537 clause 1 asks (issue 1297) 2026-10-23 16:29:57 +02:00
jschoubben 1b019f0353 Record the assigner's full requirements and mesh-sdk feat/r4g's checksums, so the new module builds without a workspace (R4g) 2026-10-23 16:27:05 +02:00
jschoubben 95f52afab2 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g) 2026-10-23 16:26:11 +02:00
jschoubben e4a40a1cb0 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, so every module r4g touches builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:02 +02:00
jschoubben dd8e05142d Merge branch 'feat/r4g-scripts' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 66be3dd0e4 Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 527f87f94c Merge branch 'feat/r4g-asks' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-checker/cmd/mesh-delivery-checker/judge.go
#	modules/mesh-delivery-checker/manifest_test.go
#	modules/mesh-delivery-checker/module.json
2026-10-23 16:20:17 +02:00
jschoubben 0463f21026 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 16:19:45 +02:00
jschoubben a4629bc649 Compute every module's fingerprint with no builder input and read no build-agent record for it, so a change of the builder rebuilds nothing (issue 1887 I1, ADR 0530 §1) 2026-10-23 16:14:19 +02:00
jschoubben 50836eb4d6 Format the watchman's integration test (issue 1887 A8b) 2026-10-23 16:14:03 +02:00
jschoubben 79ef768ea4 Show the backlog of build asks per language and kind: waiting, held by a node, and the oldest waiting ask's age (issue 1887 A8a) 2026-10-23 16:14:00 +02:00
jschoubben a8ba5c2fd5 Mirror a waiting build ask as a pulse until a node writes its step, and drop the read of the controller's checks-running (issue 1887 A8b) 2026-10-23 16:12:43 +02:00
jschoubben d0354574c6 Look the assigner's compositions bucket up at each check until it is there, so an assigner delivered after the checker started is read without a restart (issue 1297, review round 1 item 3) 2026-10-23 16:12:18 +02:00
jschoubben 69577f739d Judge hq as any repository, so its pull requests get a verdict and mesh/check from its Go tests and no script judges it (issue 1887 A10, ADR 0527 §2) 2026-10-23 16:11:43 +02:00
jschoubben d4977d9c9d Start the checker whole with the assigner's verdict on the bus and read its line, the level 1297/AC3 asks (issue 1297, review round 1 item 2) 2026-10-23 16:11:31 +02:00
jschoubben 179664898a Merge feat/r4g-asks into the view lane, keeping the checker's assigner compositions 2026-10-23 16:11:15 +02:00
jschoubben 77af82cfa2 Keep the asks state and write the ask about a held fingerprint, so the view's and the teller's read of mesh-delivery-assigner.asks is granted and has something to answer (issue 1297, review round 1 item 1; ADR 0528 clause 6) 2026-10-23 16:11:07 +02:00
jschoubben f37e5d5fa1 Delete merge-check.sh; the checker's levels and the build module's fetch beside run what it ran (issue 1295, S8; ADR 0529) 2026-10-23 16:10:46 +02:00
jschoubben d9d1e57e3a Delete the checker's hold, its own capacity and places, its node moves and its read of the controller's plans-building, so a check waits only in the build seats' streams (issue 1887 A5b, closes 1685; ADR 0525 §4, §7) 2026-10-23 16:09:34 +02:00
jschoubben e4b639c6de Judge each pull request's check as one ask on the build seats' check stream, a re-judge on recheck, so the checker chooses no node and a lost node's check is redelivered, not moved (issue 1887 A5a, ADR 0525 §1, §3, ADR 0536 §4) 2026-10-23 16:05:55 +02:00
jschoubben 271d1ca759 Pin mesh-sdk at feat/r4g's head c3915ffb, the merge of lane B's layouts, so every r4g module builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:01:15 +02:00
jschoubben 2227e7a81f Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 173178df86 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 9f27e2b1f8 Build each build module's part of a fingerprint as one ask on its seat's stream, named in the key before it is put, so the builder chooses no node, holds no capacity and never asks twice (issue 1887 A4a, A4b, A4c, ADR 0525 §1-§6) 2026-10-23 15:57:03 +02:00
jschoubben 85aca46348 Merge pull request 'Checks under 5 minutes (checker): a new head's check before trunk-move re-judges, and a build module's change judged by its own code (issues 1897, 1898; ADR 0535, 0536)' (#25) from feat/fastcheck into main
Reviewed-on: #25
2026-10-23 13:50:08 +00:00
jschoubben d5868c178f Take the words of a module the change removes from mesh-sdk in the indexer and the assigner, so the assigner's reading of a removal cannot drift from what the indexer writes (issue 1297, B3b) 2026-10-23 15:44:00 +02:00
jschoubben 7cbb1c4207 Wait for the assigner's composition only while the assigner is online, say at once that it is not judged when it is not, and take it in when the assigner writes it later, so no check waits its bound for a composition nobody makes (issue 1297, B3d) 2026-10-23 15:44:00 +02:00
jschoubben 4c98841d2a Judge a repository the mesh knows no module of with both build modules, so mesh-lab's nested Go modules and its npm test both run (issue 1887 A12) 2026-10-23 15:41:57 +02:00
jschoubben c9ee92a4ab Pin the assigner and the checker to the SDK commit that holds the assigner's ask value, so both build on the layouts the view reads (issue 1297, subtask B1c) 2026-10-23 15:39:49 +02:00
jschoubben 0f7091e4ac Place the change's own repository beside it at its trunk, so a shrink-only list test compares with the trunk's copy at $MESH_CHECK_BESIDE/<name> (issue 1887 A12, ADR 0529 §2) 2026-10-23 15:35:37 +02:00
jschoubben ed98a7519e Write every difference between a want and what the controller sends, and delete one that ended, so the view and acceptance test 17 read them on the bus (issue 1297, subtask B5b; ADR 0528 clause 4) 2026-10-23 15:31:11 +02:00
jschoubben d7ba5aa91e Compare each node's want with the fingerprint of the build the controller sends it, explaining a difference by the rule that holds the want back or saying it unexplained (issue 1297, subtask B5a; ADR 0528 clause 4) 2026-10-23 15:30:31 +02:00
jschoubben 016385751d Write each node's want by the six rules, the canary the first online node by name, and hold a fingerprint whose health failed with its nodes set back, so R5's wants can be compared with what the controller sends (issue 1297, subtasks B4a-B4b; ADR 0429 §6, ADR 0528 clauses 1 and 3, ADR 0435 §3) 2026-10-23 15:29:57 +02:00
jschoubben 5091cab652 Read the assigner's composition into the checker's verdict in place of 'not yet run', failing a check whose composition failed, naming its machine (issue 1297, subtask B3d; ADR 0528 clause 5) 2026-10-23 15:27:59 +02:00
jschoubben d8599855e1 Compose the machines a change touches, from the trunk's and the change's manifests and the controller's assignments on the bus, and write the verdict under the checker's key; what needs the facts snapshot is said as not judged (issue 1297, subtasks B3a-B3c; ADR 0446 clauses 1-3, ADR 0528 clause 5, ADR 0482 clause 3 as R4g delivers it) 2026-10-23 15:25:22 +02:00
jschoubben 57258b6fa4 Take the assigner's ask in the SDK's layout and name the asking in the view's answer, so a stale answer never fills a later asking (issue 1297 B9 and B10) 2026-10-23 15:22:32 +02:00
jschoubben c2b7b8867e Start the assigner whole on an embedded bus in an integration test: it says its three states and refuses to start without a bucket it reads (issue 1297, subtask B2b) 2026-10-23 15:19:49 +02:00
jschoubben a48279b31a Fill an assigner ask's answer from the view's answer under its view key, once and never for an ask that needs proof (issue 1297 B10) 2026-10-23 15:19:37 +02:00
jschoubben 034a6de9a7 Make the module mesh-delivery-assigner: its manifest's reads and states, its own credential and the states it announces, so the controller grants it what R5 reads (issue 1297, subtask B2a; ADR 0528, ADR 0429 §1) 2026-10-23 15:19:06 +02:00
jschoubben e7ed48b81a Give the command an answer verb, so the operator answers a hold or restore ask from a terminal as in the page (issue 1297 B9b) 2026-10-23 15:18:04 +02:00
jschoubben f7dd4ab586 Let the view answer the assigner's asks that need no proof, under the ask's view key, and refuse the merge's Approve on the phone's behalf (issue 1297 B9a) 2026-10-23 15:17:40 +02:00
jschoubben 57b0030258 Show every difference between the assigner's wants and the controller's sends with its rule, so the operator sees what holds a node back (issue 1297 B8) 2026-10-23 15:15:32 +02:00
mesh-tools db5d88f893 Merge pull request 'Drop the indexer's modules keys the catalogue's trunk no longer holds, at each merge and at start (issue 1900)' (#26) from fix/1900-indexer-drops-removed-modules into main 2026-10-23 12:56:31 +00:00
jschoubben 24753bfc2c Grant gitea_contains in the indexer's invokes and test every called forge tool is granted (issue 1900)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792794967663205383 (novox/mesh-delivery at 55f486e0 into main, d…
mesh/check pass 24753bfc2c02 on 75a4821eff9e | unit 431 run, 0 kept | integration 476 run, 0 kept | over bound: integration (slowest integration mesh-…
2026-10-23 14:48:41 +02:00
jschoubben 1dc9d80cfb Keep a key written ahead of the source head and a key with no path when reconciling (issue 1900) 2026-10-23 14:44:14 +02:00
jschoubben 26bfdbb637 Drop the indexer's modules keys the trunk no longer holds, at each merge and at start (issue 1900) 2026-10-23 14:31:55 +02:00
jschoubben 4ffe5f4d52 Start a never-judged head's check before the re-judges a trunk move asks, oldest first (issue 1897 S1)
mesh/delivery delivered
mesh/delivery-group group feat/fastcheck merged: every member is merged
mesh/check checking 4ffe5f4d5243 on 85aca46348a0 | novox/mesh-delivery#25 in group feat/fastcheck
A new pull request waited 25 to 45 minutes behind every open pull request's re-judge after a
merge. The semaphore becomes a ranked admission (ADR 0536 §4); checker_status names the
waiting checks by kind.
2026-10-23 14:09:42 +02:00
jschoubben 2bdb8d0e00 Ask the Go build module to run the candidate when a merge changes it, and say in the line which program judged (issue 1898 K3)
A change to mesh-build-go was judged by the installed module it replaces, so such pull
requests were merged by hand. ADR 0535 §1, §2, §4, §5.
2026-10-23 14:06:57 +02:00
jschoubben 75a4821eff Merge pull request 'Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)' (#24) from fix/1890-checker-stale-steps into main
Reviewed-on: #24
2026-10-23 11:01:10 +00:00
jschoubben e762028760 Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)
mesh/delivery delivered
mesh/check pass e76202876032 on b714eab995a5 | unit 422 run, 0 kept | integration 467 run, 0 kept | over bound: integration (slowest integration mesh-…
The checker's own places are released when a step ends or its wait passes the step bound; checker_status read every
'running' key in the build modules' steps buckets as a step in flight, so keys left by stopped, restarted or unassigned
build modules looked like full nodes. It now lists them under left_running with why (past the bound, module not on the
node, module not running), and leaves hq-only groups out of the unjudged candidates.
2026-10-23 09:03:45 +02:00
jschoubben 36bf3b7143 Merge pull request 'The builder says it writes references only once its build modules proved it (issue 1886, ADR 0522)' (#23) from fix/1886-builder-says-references into main
Reviewed-on: #23
2026-10-23 00:26:18 +00:00
jschoubben 9fad90ee89 Say that the builder writes references only once each build module answered one (issue 1886, ADR 0522 review)
mesh/merge-gate head 9fad90ee890a merged into trunk b714eab995a5 — pass: builds mesh-delivery-builder → novox; no bus step; the change touches no module of…
mesh/repo-check head 9fad90ee890a merged into trunk b714eab995a5 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/1886-builder-says-references merged: every member is merged
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The references come from the build modules' own pushes, so a new builder beside
an old build module writes none; saying it at start would send the controller
to keys that fail not stored.
2026-10-23 00:52:50 +02:00
jschoubben 4b8b1c047c Say in the builder's own bucket of instances that it writes references (issue 1886)
The controller delivers from the builder's keys only while a running builder
says its done keys name references; a builder from before R4e says nothing,
so R4e's own delivery is asked of build-agent instead of waiting for ever.
The say is put once started and taken away when stopped.
2026-10-23 00:48:28 +02:00
mesh-tools b714eab995 Merge pull request 'R4e (mesh-delivery): the builder pushes and keys artifacts by reference, the checker's verdict and try, the view, and the checker's owed state read by the forge' (#22) from feat/r4e into main 2026-10-22 22:13:05 +00:00
jschoubben efca168c55 Stop the forge bridge performing mesh/check: the forge module writes it (R2-1)
mesh/repo-check head efca168c5522 merged into trunk 0e372c87f1df beside mesh-catalog#457@896bd883 +1 more — pass: its merge-check.sh passed
mesh/merge-gate head efca168c5522 merged into trunk 0e372c87f1df — pass: group feat/r4e composed: every machine composes with the change as it did without …
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792707174627079192 (novox/mesh-delivery at b714eab9 into main, d…
mesh/delivery-group group feat/r4e closed: a member was closed before the group merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:989 | efca168c5522 on b714eab995a5 | unit 2092 run, 0 kept, 3 skipped | integration…
R4e round 2 finding R2-1, ADR 0521: nothing in the delivery plan orders the
forge module before the bridge, and the forge module's durable consumer keeps
its deliver policy when its filters change, so a check.status emitted before
it consumed the event was lost. The forge module now watches the checker's
owed state and writes mesh/check itself, once per revision.

The bridge emits no check.status, declares no emits, and neither reads the
checker's owed nor hears its stream; OwedAct still refuses mesh/check. The
integration test shows the bridge calls no tool and publishes nothing of its
own when the checker owes mesh/check.
2026-10-22 21:40:09 +02:00
jschoubben 19d06163de Announce the checker's states, owed readable, so the forge module may read it (R2-1)
R4e round 2 finding R2-1, ADR 0521: the forge module writes mesh/check from
the checker's owed state, read by a watch, because no event order between the
forge bridge and the forge module is kept by the delivery plan. A reader is
granted from the owner's announcement (ADR 0495 §2-§3), and the checker
announced nothing. It now says every state it keeps, with owed, previews and
verdicts readable: the forge module and the bridge read owed, the indexer
previews, the gate verdicts.
2026-10-22 21:38:53 +02:00
jschoubben 0997b071a3 Pin every R4e member module to mesh-sdk's final R4e head 104533e8, which holds review fixes S1 and S2, so the group pins one SDK 2026-10-22 21:29:20 +02:00
jschoubben bb4a4a82dc Emit mesh/check as the bridge's own check.status, never through a tool (K1)
Review finding K1 of R4e, issue 1301 S4 (ADR 0503 §3, ADR 0513 §4: the forge
requires mesh/check alone). gitea_check_status rested on "only the bridge
invokes it", but node-tools invokes every tool and a tool is told no caller,
so any session could set mesh/check and merge past the checker.

The bridge now emits check.status on mesh.mod.mesh-delivery-forge-bridge.event.
check.status, which the bus grants to its own account alone (emits), with the
owed status, its key and its revision; the event id check.<key>.<rev> is the
bus's dedup id. A status the bus did not take is not kept as done. OwedAct
refuses mesh/check, and gitea.gitea_check_status leaves the invokes. The
integration test shows the tool runner's account refused on that subject and
gitea's durable consumer handed statuses emitted before it fetched.
2026-10-22 21:23:27 +02:00
jschoubben 08bc672488 Pin every R4e member module to mesh-sdk's final R4e head 875c680d, which holds main's #68 and #69, so the group pins one SDK 2026-10-22 20:45:16 +02:00
jschoubben 90ac287e59 Drop the checker tools binary committed by mistake with the try verbs (7abe230): no module commits a build output 2026-10-22 20:29:53 +02:00
jschoubben 91e68ac13b Merge feat/r4e-builder into feat/r4e (R4e integration)
The checker's go.mod and go.sum conflicted on the mesh-sdk pin (checker lane 9cb4d59, builder lane e76a5a3): every
module touched is pinned to the sdk lane's newest head d1af87a until the final SDK head.
2026-10-22 20:29:53 +02:00
jschoubben 173becf35e Merge feat/r4e-checker into feat/r4e (R4e integration) 2026-10-22 20:29:10 +02:00
jschoubben 612658e788 Ask the forge for the objects of a merge's named paths in the indexer, the one reader of a manifest's paths, and narrow the module's tree from the answer (issue 1758, I6 reworked; ADR 0499 §2 §3, ADR 0495 §1, L8)
The manifest invokes gitea.gitea_commit_trees. A forge that cannot say gives the module none with its words.
2026-10-22 18:24:38 +02:00
jschoubben d801a6460e Read no manifest's paths in the forge bridge: the indexer, the one reader of paths (ADR 0495 §1, the lead's decision L8), asks the forge for their objects itself (issue 1758, I6 reworked)
Undoes 8ab8c5d's bridge change; the bridge keeps the mesh-sdk pin.
2026-10-22 18:23:29 +02:00
jschoubben 12cf0cc1f5 Format the builder's integration test 2026-10-22 18:21:04 +02:00
jschoubben 7cc97cd6b6 Name a candidate's named paths in its fetch, and prove on a real bus that claude-code's manifest is built (issues 1758 and 1751) 2026-10-22 18:20:59 +02:00
jschoubben ece328d0cc Give a candidate's module its named paths' objects from the preview, so the builder's fetch for it names them (issue 1758, ADR 0499 §4; mesh-sdk CandidateModule.Paths) 2026-10-22 18:20:37 +02:00
jschoubben b39090aab9 Read manifests with mesh-sdk's ParseManifestForInputs in the indexer and the builder, so claude-code, dunst and xterm are built too (issue 1751)
The indexer's own copy goes. Pins mesh-sdk feat/r4e-sdk e76a5a3 in every module of the group.
2026-10-22 18:20:37 +02:00
jschoubben 4bb04a47f9 Build each fingerprint on one node and drop the second-node retake, as ADR 0478 §4 has it from R4e (issue 1587, B9)
A key done on one node before R4e is not built again for a second; a key failed for differing digests stays as it is.
2026-10-22 18:18:59 +02:00
jschoubben 9a3f38629f Prove on a real bus that a merge of mesh-sdk changing only what its lib does not name leaves its tree input and new form unchanged (issue 1758, I9, ADR 0499 check 6 in miniature) 2026-10-22 18:16:54 +02:00
jschoubben 9bd84941ee Show a fingerprint of the new form against the builder's key of it, since the indexer no longer writes the old form and no build record names the new (issue 1851, I5)
The view watches the builder's builds; a value written before R4e keeps today's comparison. Pins mesh-sdk feat/r4e-sdk
9cb4d59.
2026-10-22 18:16:18 +02:00
jschoubben cc63123b78 Carry each named path's object from a fetch's module trees into the merge preview, so the indexer narrows a candidate's tree as the merge's (issue 1758, I8, ADR 0499 §2)
Pins mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:15:09 +02:00
jschoubben 23daa2c03c Take a module's tree input from the paths its artifacts name, by mesh-sdk's NamedTree, and keep their objects and each artifact's paths, so a module outside them moves for no change there (issue 1758, I7 and the indexer's I8, ADR 0499 §2 §3)
A merged message or a preview without the objects gives the module none, naming the paths. Every other module keeps its
directory's tree. Module.ArtifactPaths is filled for the checker (the lead's interface).
2026-10-22 18:14:30 +02:00
jschoubben 8ab8c5d04a Carry each named path's git object in the merged and pull messages of a module whose every artifact names its paths, read at its manifest's commit (issue 1758, I6, ADR 0499 §3)
Asks gitea_commit_trees with paths (the interface agreed with the lead); a forge that cannot say is a retry. Pins
mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:13:25 +02:00
jschoubben e81ef8bb63 Read a manifest whose config value falls back to a predefined key, so claude-code, dunst and xterm get a new-form fingerprint (issue 1751, ADR 0348 §7)
The indexer holds no published list of predefined keys; each key a value names is taken as the value declares it.
No fingerprint input comes from the list: the gate and the controller still judge the value.
2026-10-22 18:11:21 +02:00
jschoubben af4eaefb14 Read the controller's build records only for the builder's own input and to fill a key's missing directory once, so a key written before paths gets one with no merge (issue 1851 I3, issue 1614)
Both reads go in R4g with build-agent. A path a key names is never overwritten, and no fingerprint changes.
2026-10-22 18:09:56 +02:00
jschoubben 9f7e082688 List a repository's modules from the merged message's manifests and the indexer's own bucket, not the controller's build records, so a new module gets a value and a key with no build asked (issue 1851, I1 I2 I4, ADR 0501 §6, ADR 0453 §6)
I1: a merge writes each module its manifests name, at its directory. I2: a pull request's changed directory, a
preview's touched module and a source revision's modules come from the modules bucket (a module a pull request adds is
found in the preview by its manifest). I4: the value of today's form (fp, none) is no longer written, since it was
computed from the build records; mesh-sdk keeps it readable until R4g. One commit: the three share applyMerged and its
tests. Pins mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:08:48 +02:00
jschoubben 7d740be255 Fail a check whose forge read reached nobody at the forge, and judge it again when the nodes change (issue 1624, C7)
mesh-catalog #293 failed with 'go.mod cannot be read to check its mesh-sdk pin: nats: no responders', put at the
build node, and stayed failed: the forge's tool runner was not answering for a moment, which is not the change's
failure. The forge's reads now mark such a call unheard, as the build steps' calls are (issue 1820).
2026-10-22 18:07:22 +02:00
jschoubben 66ae4dc791 Format the judge (gofmt) 2026-10-22 18:06:11 +02:00
jschoubben 03eaf332ad Say the composition's time and the machines it composed in a check's line, once the assigner reports them (issue 1300, C10)
The line said only that the composition passed with an SDK commit, and its time sat among the parts that are dropped
first, so a forge status never said how long the composition took or which machines it covered.
2026-10-22 18:06:07 +02:00
jschoubben cbdcf6da0e Give a group's TypeScript members the mesh-sdk member's package when its merge changes the package's paths, and pin the SDK's R4e head (ADR 0500 §4-§5, issue 1309's checker half)
A TypeScript member of a group that changes mesh-sdk's package resolved the old version from the registry. Its Node
fetch now names the mesh-sdk member's head, which the Node build module packs and serves, publishing nothing. The paths
come from the indexer's modules value (lead decision L8), so a change of Go files or of the manifest alone resolves
the registry as before. A test holds that no step of a check carries publish.
2026-10-22 18:05:12 +02:00
jschoubben 8b2e8895c2 Prove on a real bus that every done key names its references, a refused push is built again, and a pinned upstream is copied (issue 1587, B8, ADR 0498 checks) 2026-10-22 18:03:39 +02:00
jschoubben 7abe230701 Judge a pushed head as a try on the checker, through checker_try and checker_try_read (ADR 0501 §5)
mesh-controller's try asked build-agent, which leaves the checking at R4e. A try is now a key in the checker's state
tries: the checker judges the head merged onto its trunk as it judges a pull request's head, through the same
judge, places and holds, writes its end in the key and owes the forge nothing (no status, no preview, no verdict).
One try per head, at most four at once; hq and a repository whose trunk the gate does not name are refused with
"no level of <repo> is checked by the checker".
2026-10-22 18:03:28 +02:00
jschoubben 7651e47cc6 Write a module with no artifact a done key with no artifact, so the controller has a done key to register it from (issue 1613, ADR 0478 §1, ADR 0501 §1)
Moves the mesh-sdk pin to feat/r4e-sdk 9cb4d59, which accepts such a key and names delivery.WaitsFor; the builder's
package wait now uses that shared prefix (B7).
2026-10-22 18:03:02 +02:00
jschoubben 957afa6926 Ask the fetch with mesh-sdk's FetchRequest, naming a module's directory and the paths its kept inputs hold, so a module naming its paths is built from those alone (issue 1758, ADR 0499 §4; the lead's fetch interface) 2026-10-22 18:02:08 +02:00
jschoubben 2e567b9b30 Let a key whose lock names an unpublished package version wait queued, naming it, and build it once the builder's own done key publishes it (issue 1587, B7, ADR 0500 §6)
The prefix 'waits for: ' is kept in the builder until mesh-sdk's delivery.WaitsFor lands on feat/r4e-sdk.
2026-10-22 18:01:33 +02:00
jschoubben 851811a318 Build a candidate's done package key again once the trunk names it, so the package is published from the trunk (issue 1587, B6, ADR 0498 §6) 2026-10-22 18:00:19 +02:00
jschoubben 2399a4f4cc Build a TypeScript package with the Node build module, published only for a trunk fingerprint; a version taken fails the key until the fingerprint changes (issue 1587, B5, ADR 0498 §5, ADR 0500 §4) 2026-10-22 17:59:43 +02:00
jschoubben 2855ae7ebf Set the owed mesh/check through gitea_check_status, the one tool that writes it (issue 1301's delivery part, option A)
The forge's gitea_commit_status refuses the context mesh/check from R4e, so that only the checker's account, through
this bridge, can make a pull request pass the one required status.
2026-10-22 17:59:30 +02:00
jschoubben 081f32d6c9 Copy a pinned upstream image with mesh-build-image's copy step, with no fetch, so its key names the registry's reference (issue 1587, B4, ADR 0498 §4) 2026-10-22 17:59:19 +02:00
jschoubben d9ace64681 Mark mesh/check pending on every open pull request into a moved trunk, and carry the gate's heads hash in each verdict (issue 1302's checker part, lead decision L4; issue 1300)
The gate decides on mesh/check alone and matches an Approve's members by the heads hash over lowercase repositories
and heads. A success judged on an old trunk stood on the forge while the next check waited for a place or a delivery,
and a candidate whose repository the forge named in capitals carried a hash the gate never finds. A check that ends
after its trunk moved now owes pending, not its verdict.
2026-10-22 17:58:49 +02:00
jschoubben c26f909152 Send a pinned upstream image to mesh-build-image's copy and fail an unpinned one 'not pinned:' until its fingerprint changes; refusals use mesh-sdk's RefusedKind (issue 1587, B3, ADR 0498 §4) 2026-10-22 17:58:33 +02:00
jschoubben 9871ed2d09 Build a key done before R4e again once, so every bundle and archive the controller delivers names its stored reference (issue 1587, B2, ADR 0498 §8) 2026-10-22 17:57:35 +02:00
jschoubben f26ff8f8bb Write a key done only when every bundle and archive names where it is stored, a candidate's too, so a merge builds nothing its check built (issue 1587, B1, ADR 0498 §1-§3)
Pins mesh-sdk feat/r4e-sdk a90f523 (R4e's group member). A trunk key's builds are asked to publish; a build answering
no reference fails the key 'not stored:', and a step's own 'not stored:', 'not pinned:', 'version taken:' or
'refused kind:' opens the key's error.
2026-10-22 17:57:01 +02:00
jschoubben f7d8622deb Keep a core repository's passing package under a digest that covers its siblings' commits, so core checks keep verdicts (issue 1287, C3a/C3b, lead decision L5)
The checker took and kept nothing for a workspace given siblings, and every core repository has siblings, so the
kept store stayed empty. The digest step now folds the siblings' commits in and echoes them; a verdict is kept or
taken only when the echo equals the siblings the fetch was given, so a digest that misses one never poisons another
check (ADR 0444, issue 482).
2026-10-22 17:56:59 +02:00
jschoubben d83156fb21 Add checker_previews and checker_status read tools so the live acceptance of issues 1680 and 1685 can be read (C1, C2) 2026-10-22 17:25:11 +02:00
jschoubben 16ca51dc1f Bound builder_builds by key prefix, limit and read budget, list unreadable keys, refuse fp2 with commit, add candidate reads (issue 1827 V1-V3, 1818, 1681) 2026-10-22 17:23:46 +02:00
mesh-tools 0e372c87f1 Merge pull request 'Read a go.mod the head does not hold at the trunk, so a branch older than a new Go module passes the pin check (issue 1845)' (#21) from fix/1845-pin-read-at-trunk into main 2026-10-22 13:44:08 +00:00
jschoubben 4a0c771059 Match the forge's 404 by its status, not by digits a path's sha may hold (issue 1845)
mesh/repo-check head 4a0c77105943 merged into trunk 2b635e2138ae — pass: its merge-check.sh passed
mesh/merge-gate head 4a0c77105943 merged into trunk 2b635e2138ae — pass: builds mesh-delivery-checker → novox; no bus step; every machine composes with the…
mesh/delivery delivered
mesh/check pass 4a0c77105943 on 2b635e2138ae | unit 371 run 0 kept | int 410 run 0 kept | clone 13s unit 206ms int 14s+45s | composition not yet run
2026-10-22 15:09:25 +02:00
jschoubben 14c78e485f Read a Go module's go.mod at the trunk when the head does not hold it, so a branch cut before the module was added passes the pin check (issue 1845) 2026-10-22 15:00:58 +02:00
mesh-tools 2b635e2138 Merge pull request 'Send image artifacts to mesh-build-image, one node, publishing only from the trunk (issue 1705)' (#20) from feat/r4c9-image-build into main 2026-10-22 08:57:21 +00:00
jschoubben db62e82e26 Write a failed publish of a merged image to the key; read mesh-build-image's captured steps (review r2)
mesh/repo-check head db62e82e26d7 merged into trunk 47ab8cd3cf45 beside mesh-catalog#435@7c5d11e5 — pass: its merge-check.sh passed
mesh/merge-gate head db62e82e26d7 merged into trunk 47ab8cd3cf45 — pass: group feat/r4c9-image-build composed: every machine composes with the change as it…
mesh/delivery delivered
mesh/delivery-group group feat/r4c9-image-build merged: every member is merged
mesh/check pass db62e82e26d7 on 2b635e2138ae | unit 366 run 0 kept | int 405 run 0 kept | clone 9.2s unit 191ms int 6.7s+44s | composition not yet run
2026-10-22 10:37:21 +02:00
mesh-tools 47ab8cd3cf Merge pull request 'The checker moves a step that no build node answers to another node (issue 1820)' (#19) from fix/1820-checker-heard-nodes into main 2026-10-22 08:36:34 +00:00
jschoubben 12f61998c5 Publish a merged candidate's image once; a retaken key builds only what is missing (review r1) 2026-10-22 10:33:52 +02:00
mesh-tools 5b09d1ccb5 Merge pull request 'Read tools for the builder's builds in the view (issue 1723)' (#18) from fix/1723-builder-builds-tool into main 2026-10-22 08:30:42 +00:00
jschoubben bbc690539b Build an image on one node through mesh-build-image; only a trunk key publishes (issue 1705) 2026-10-22 10:19:26 +02:00
jschoubben bbb42d704c The checker moves a step that no build node answers to another node and fails naming every node tried (issue 1820)
mesh/merge-gate head bbb42d704ce0 merged into trunk 5b09d1ccb52d — pass: builds mesh-delivery-checker → novox; no bus step; every machine composes with the…
mesh/repo-check head bbb42d704ce0 merged into trunk 5b09d1ccb52d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check pass bbb42d704ce0 on 5b09d1ccb52d | unit 354 run 0 kept | int 392 run 0 kept | clone 8.7s unit 199ms int 7.1s+44s | composition not yet run
2026-10-22 10:04:29 +02:00
jschoubben f2379dfeea Add builder_build and builder_builds read tools to the view, so the builder's keys can be checked live (issue 1723)
mesh/merge-gate head f2379dfeea2d merged into trunk 113f7aa93385 — pass: builds mesh-delivery-view → shanks; no bus step; every machine composes with the c…
mesh/repo-check head f2379dfeea2d merged into trunk 113f7aa93385 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: g14: fetch of novox/mesh-delivery#18: build_go_fetch on g14: nats: no responders available for request | f2379dfeea2d on 113f7aa933…
2026-10-22 09:54:46 +02:00
mesh-tools 113f7aa933 Merge pull request 'The checker runs beside today's check: the checker, the indexer's candidate fingerprints, the bridge's status' (#17) from feat/the-checker into main 2026-10-21 21:46:33 +00:00
jschoubben b519c8dde2 Keep nothing for a workspace given siblings, place the siblings where the tests state them, decide the hold from the listed plan keys (review D1-D3)
mesh/repo-check head b519c8dde2da merged into trunk a5aa69a4e815 beside mesh-catalog#423@12ff1cfb +1 more — pass: its merge-check.sh passed
mesh/merge-gate head b519c8dde2da merged into trunk a5aa69a4e815 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker merged: every member is merged
2026-10-21 21:52:41 +02:00
jschoubben 7214bf4368 Fail a candidate's build when its fetch answers files in conflict or no merge, so the trunk's build is not stored under the candidate's fingerprint (review D4) 2026-10-21 21:50:30 +02:00
jschoubben 284f21c4e3 Key the builder's candidate test as the indexer keys a group 2026-10-21 21:35:53 +02:00
jschoubben a20af30d54 A group's Go member naming a mesh-sdk commit that is neither on the trunk nor the group's head fails the check naming the member and the commit; a pin only a head carries names issue 1148 (issue 1624) 2026-10-21 21:34:35 +02:00
jschoubben 4cdba63b9b Read tools for the checker's verdicts, a pull request's report and the kept verdicts (issue 1686) 2026-10-21 21:32:08 +02:00
jschoubben ee994f57b9 Do not pin the integration test to one form of the line 2026-10-21 21:30:29 +02:00
jschoubben fecabfca8b Fit the status line in the forge's 140 characters by giving the times and counts in their compact form before dropping them (issue 1682) 2026-10-21 21:30:24 +02:00
jschoubben 4b063252e7 The checker's line says whether each candidate module's build is built, failed or waiting, read from the builder's keys (issue 1684) 2026-10-21 21:29:13 +02:00
jschoubben 66908d9bbc The builder builds the fingerprints a candidate carries, once each, from the head merged onto the trunk (issue 1684) 2026-10-21 21:28:38 +02:00
jschoubben 7433b0d047 Start no check while a delivery builds: the checker reads the controller's bucket of the plans in their build phase, absent as empty (issue 1685) 2026-10-21 21:26:46 +02:00
jschoubben a2a5167d72 Write a preview of the conflicts for a head that does not merge, and give the candidate's modules the reason; pin mesh-sdk to the R4c-5a head (issue 1445) 2026-10-21 21:25:45 +02:00
jschoubben e554094b74 The builder uses mesh-sdk's buildrun package for the caller, the steps, the places, the presence, toolchain and have sets and the choice of nodes, so the checker shares them (issue 1671) 2026-10-21 21:25:00 +02:00
jschoubben 64599db3fa Capture the fetch-with-base fixtures from mesh-catalog's code, and prove the checker whole against a real bus with fake build modules (issues 1679, 1287, 1288, 1289, 1678) 2026-10-21 21:23:22 +02:00
jschoubben 96ba6bd3dc Say the verdict in a line of at most 140 characters and owe the status mesh/check in the checker's bucket, which the forge bridge performs once per revision; the checker's stream checks wakes the readers of its buckets (issues 1682, 1683) 2026-10-21 21:20:42 +02:00
jschoubben 4242fb5b0b The indexer gives a candidate's touched module the fingerprint its merge preview gives it, and wakes on the checker's preview.written without a restart; pin mesh-sdk to the R4c-5a head (issues 1445, 1681) 2026-10-21 21:17:09 +02:00
jschoubben e175343837 Write the merge preview of each head the checker fetched, for the indexer's fingerprints (issue 1680) 2026-10-21 21:12:48 +02:00
jschoubben cfbd5b6d2d Judge a head merged onto its trunk through the build modules: run only packages without a kept verdict, the change's own tests first, stop at the first failure (issues 1679, 1288, 1289) 2026-10-21 21:11:43 +02:00
jschoubben 27072a01b5 Add the checker module: it watches the candidates, claims a verdict once, owes the status mesh/check and ends a check past its bound (issue 1678) 2026-10-21 21:07:14 +02:00
jschoubben a00df90c47 Keep a passing package's verdict once under its digest, so a second check can take it (issue 1287) 2026-10-21 21:02:54 +02:00
mesh-tools a5aa69a4e8 Merge pull request 'The builder takes build-agent's newest trunk build as its builder input (issue 1727)' (#16) from fix/1727-builder-input into main 2026-10-21 15:44:42 +00:00
mesh-tools 318064ce20 Merge pull request 'Ask again a merge ask the router refused for want of room, so the operator is asked (issue 1610)' (#15) from fix/1610-ask-reaches-channel into main 2026-10-21 15:39:20 +00:00
jschoubben 61fea8314f Compute from a build of build-agent only once a toolchain is read, so a restart drops no fingerprint; fixtures as live (issue 1727)
mesh/merge-gate head 61fea8314fe8 merged into trunk 318064ce206d — pass: builds mesh-delivery-indexer → novox; no bus step; every machine composes with the…
mesh/repo-check head 61fea8314fe8 merged into trunk 318064ce206d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 16:45:51 +02:00
jschoubben 037bf1ef36 Ask again at start a merge ask whose refusal the teller did not hear, since the router keeps no record of a refused ask (issue 1610, review round 1)
mesh/merge-gate head 037bf1ef361b merged into trunk b04ee764739a — pass: builds mesh-delivery-teller → novox; no bus step; every machine composes with the …
mesh/repo-check head 037bf1ef361b merged into trunk b04ee764739a — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 16:45:31 +02:00
jschoubben 38116fe00a Ask again a merge ask the router refused for want of room, once one of the teller's asks ends, so the operator is asked (issue 1610)
The router holds at most three open asks of one asker and refuses a fourth without keeping it; the teller only logged the refusal, so a merge ask in a burst never reached the operator while the gate read it as asked (issues 1610, 1620). Bounded at six refusals, then said as a condition.
2026-10-21 15:54:08 +02:00
jschoubben 5f9ed2b886 Take the builder's Builder input from build-agent's newest trunk build record, not its absent fingerprint, so modules get a new-form fingerprint (issue 1727) 2026-10-21 15:49:33 +02:00
mesh-tools b04ee76473 Merge pull request 'Make the builder's build capacity a setting per node, hold a failed second-node build until the nodes change, and close keys left building for a vanished fingerprint (issues 1661, 1664, 1660)' (#14) from fix/1661-builder-follow-ups into main 2026-10-21 12:26:03 +00:00
mesh-tools 2000c841bf Merge pull request 'Read the indexer's modules keys as stored from the view's tools, so the new-form fingerprint, path and toolchains can be checked live (issue 1562)' (#13) from fix/1562-indexer-reads into main 2026-10-21 12:04:07 +00:00
jschoubben 46702d1af3 Keep the builder off a node whose capacity setting is 0, and start the setting at shanks=3,ace=2,g14=0,novox=0 (issue 1661)
mesh/merge-gate head 46702d1af311 merged into trunk 2000c841bf70 — pass: builds mesh-delivery-builder, sent nowhere; no bus step; every machine composes wi…
mesh/repo-check head 46702d1af311 merged into trunk 2000c841bf70 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:46:14 +02:00
jschoubben 1239199b35 Name the view's tools binary as what its bundle loads, as the controller's module check requires of a bundle compiled to a binary (issue 1562)
mesh/merge-gate head 1239199b35dd merged into trunk d68d7ef3d9af — pass: builds mesh-delivery-view → shanks; no bus step; every machine composes with the c…
mesh/repo-check head 1239199b35dd merged into trunk d68d7ef3d9af — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:43:26 +02:00
jschoubben dcf62b0413 Make the builder's capacity a module setting, hold back a failed second-node build until the nodes change, and close at start a key left building for a fingerprint the module no longer has (issues 1661, 1664, 1660) 2026-10-21 13:34:39 +02:00
mesh-tools d68d7ef3d9 Merge pull request 'Add the builder that builds each fingerprint's bundles and archives on two build nodes beside build-agent, and have the indexer write a module's path and toolchains (R4c-4: issues 1606, 1607, 1615)' (#12) from feat/r4c4-builder into main 2026-10-21 11:31:05 +00:00
jschoubben 6c2b79acf1 Serve the indexer's read tools from the view, not the indexer (ADR 0437 4): indexer_module and indexer_modules read mesh-delivery-indexer.modules through the view's reads. Side effect: carrying the view's tools gives the tool runner read access to every bucket the view reads. (issue 1562)
mesh/merge-gate head 6c2b79acf106 merged into trunk d68d7ef3d9af — fail: builds mesh-delivery-view → shanks; no bus step; a manifest the change touches fai…
mesh/repo-check head 6c2b79acf106 merged into trunk d68d7ef3d9af — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 13:24:10 +02:00
jschoubben bfc64ac65e Build an archive with no toolchain, mark digests that differ and a one-node key so each is handled, reserve a place when a node is chosen, choose only nodes whose build module is healthy, and take over any key found building (issue 1607, review round 4)
mesh/merge-gate head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: builds mesh-delivery-indexer, new: modules/mesh-delivery-builder → novox; no bus s…
mesh/repo-check head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:15:28 +02:00
jschoubben e6ce4be4b4 Read the indexer's modules keys with two read-only tools, so the new-form fingerprint and the reason for none can be checked live (issue 1562) 2026-10-21 13:06:05 +02:00
jschoubben ebac167cd6 Move the indexer and builder to mesh-sdk main with the toolchains and root path (issue 1615) 2026-10-21 12:45:10 +02:00
jschoubben 27e4c3891f Run at most a node's build capacity of build steps on it at once, and hold a step back for a place with no timer (issue 1607, round 3) 2026-10-21 12:41:42 +02:00
jschoubben f555a05cfb Build a key only with the toolchain its fingerprint names, accept '.' as a root module's path, and choose only online nodes (issues 1615, 1607 review) 2026-10-21 11:36:32 +02:00
jschoubben 3bd2a065f9 The indexer writes '.' for a root module's path and the toolchains each fingerprint was computed with (issue 1615) 2026-10-21 11:33:49 +02:00
jschoubben 0b0cfb6e37 Test the builder whole over the real bus with fake build modules, and look again at a value that arrives while its key is built (issue 1607, AC6, AC7) 2026-10-21 11:25:13 +02:00
jschoubben 46c9f33d64 Build each new-form fingerprint through the build modules on two nodes and write its key done or failed (issue 1607, AC7, AC8) 2026-10-21 11:23:39 +02:00
jschoubben 71bb5a5fb3 Watch the build modules' toolchains and steps and keep the toolchains by node, with no timer (issue 1607, AC6) 2026-10-21 11:21:46 +02:00
jschoubben 3298ad58d3 Run a build module's long step: call it on its node, read the result at its steps key, bound the wait (issue 1607, AC5) 2026-10-21 11:21:14 +02:00
jschoubben 74d11f95a2 Choose build nodes by the agreed toolchain and fewest running steps, and agree the two nodes' digests (issue 1607, AC4) 2026-10-21 11:20:31 +02:00
jschoubben 1e6d00b4c7 Claim a build key by create and compare-and-set, never twice, retry a failed key except a refused kind (issue 1607, AC3) 2026-10-21 11:20:09 +02:00
jschoubben 87e550681f Plan a manifest's artifacts for the build modules and refuse image, upstream and package by kind (issue 1607, AC2) 2026-10-21 11:19:38 +02:00
jschoubben c3804b012f Add the builder part: its manifest, its credential and its bus connection (issue 1607, AC1) 2026-10-21 11:18:30 +02:00
jschoubben 07a28b03c4 The indexer writes each module's directory into modules.<m> and keeps it (issue 1606, ADR 0478 clause 8) 2026-10-21 11:16:16 +02:00
mesh-tools 26de6bc625 Merge pull request 'The indexer reads the build modules' toolchains, and an artifact with no lock carried has no lock entry (design 47 R4c-3)' (#11) from feat/r4c3-build-modules into main 2026-10-20 21:43:20 +00:00
mesh-tools a5da6079de Write each module once per merge and watch a build module's toolchains bucket on its toolchain.reported event, with no timer (issue 1444, review round 1)
mesh/repo-check head a5da6079de01 merged into trunk 340bf643883d beside mesh-catalog#389@bcee61ed — pass: its merge-check.sh passed
mesh/merge-gate head a5da6079de01 merged into trunk 340bf643883d — pass: group feat/r4c3-build-modules composed: every machine composes with the change as …
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792582266656254343 (novox/mesh-delivery at d68d7ef3 into main…
mesh/delivery-group group feat/r4c3-build-modules merged: every member is merged
2026-10-20 21:02:44 +00:00
mesh-tools 127ac64cce Leave an artifact without its lock file out of Locks (issue 1459) and read the build modules' toolchains (issue 1444), part 2 of 2 2026-10-20 20:48:30 +00:00
mesh-tools 74fef7dd5a Recompute the new-form fingerprints when a build module reports its toolchain (issue 1444), part 1 of 2 2026-10-20 20:47:35 +00:00
mesh-tools 340bf64388 Merge pull request 'The bridge carries each module's manifest and lock hashes, and the indexer writes manifests by digest and each module's new-form inputs (design 47 R4c-2, issues 1448, 1449, 1450, 1444, 1451, 1458)' (#10) from feat/r4c2-new-keys into main 2026-10-20 19:31:05 +00:00
mesh-tools c9f742d5f2 Read a merge's manifests and lock files at most eight at once, so a catalogue merge stays inside the bridge's bound (issues 1448, 1458, design 47 R4c-2)
mesh/repo-check head c9f742d5f20e merged into trunk 2520cc4a88f7 — pass: its merge-check.sh passed
mesh/merge-gate head c9f742d5f20e merged into trunk 2520cc4a88f7 — pass: group feat/r4c2-new-keys composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/r4c2-new-keys merged: every member is merged
2026-10-20 19:06:03 +00:00
mesh-tools f69a561a16 Test how a manifest and lock hashes become a module's new-form inputs (issue 1450, design 47 R4c-2) 2026-10-20 18:55:54 +00:00
mesh-tools 27bfd83fa0 Test the kept manifests, the new-form reasons and the candidate's manifests against real files (issues 1449, 1444, 1451, design 47 R4c-2) 2026-10-20 18:55:29 +00:00
mesh-tools a75d9cc851 Keep manifests and say why no new-form fingerprint exists, so the group check can start from facts (issues 1449, 1450, 1444, 1451, design 47 R4c-2) 2026-10-20 18:54:28 +00:00
mesh-tools ba7ce55887 Give the indexer a manifests bucket and the SDK that knows the new keys (issue 1449, design 47 R4c-2) 2026-10-20 18:53:40 +00:00
mesh-tools 3c1e7ad464 Test the bridge's manifest and lock file conversion (issue 1448, design 47 R4c-2) 2026-10-20 18:53:07 +00:00
mesh-tools 41225bf459 Test the forge bridge's file reads against the stand-in forge and the bus (issue 1448, design 47 R4c-2) 2026-10-20 18:52:27 +00:00
mesh-tools 543d91d626 Read each module's manifest and lock files in the forge bridge, so the indexer gets them (issue 1448, design 47 R4c-2) 2026-10-20 18:51:27 +00:00
mesh-tools d650a41ba6 Copy the real manifests the indexer's new-form tests read (issues 1450, 1444)
Fixtures are copied from the writer, never written by hand: four catalogue manifests at mesh-catalog a67f0196 and build-agent's at mesh-controller f2d7bc94.
2026-10-20 18:48:08 +00:00
mesh-tools 2520cc4a88 Merge pull request 'The parts serve the gate: the bridge performs owed acts and merges, the teller carries asks, the indexer reads source.main, the view shows asks (design 47 R3e, issues 1341-1345)' (#9) from feat/r3e-the-gate into main 2026-10-20 06:49:19 +00:00
mesh-tools c93954e53c Pin mesh-sdk to main's merge of #47 (88581d0a), the same tree as ba684e54 the review read
mesh/repo-check head c93954e53c8e merged into trunk 9c15112d935a beside mesh-catalog#380@cf5c0273 +1 more — pass: its merge-check.sh passed
mesh/merge-gate head c93954e53c8e merged into trunk 9c15112d935a — pass: group feat/r3e-the-gate composed: every machine composes with the change as it did…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792478961597980735 (novox/mesh-delivery at 2520cc4a into main…
mesh/delivery-group group feat/r3e-the-gate merged: every member is merged
2026-10-20 06:33:01 +00:00
mesh-tools bf59b5a27b An answer to an earlier asking no longer blocks the next asking: the teller and the view tell the asking by the answer's warrant id (review round 2, R2-1) 2026-10-20 01:35:26 +00:00
mesh-tools 14e9d56a22 Ask under each asking's own router id, spell the heads in lower case as the gate does, show a removed have plainly; pin mesh-sdk ba684e5 (review round 1: G2, T1, issue 1370) 2026-10-20 01:30:54 +00:00
mesh-tools 34f585106f Show the open asks in the delivery view: the gate's asks with no answer, in the table, the page and the command's JSON (issue 1345) 2026-10-20 00:34:28 +00:00
mesh-tools 05c180cf87 Add the indexer's source revision logic and its tests that the last commit named (issue 1344) 2026-10-20 00:31:19 +00:00
mesh-tools 409c41d4ae Stamp the modules with the gate's source revision, read from source.main (issue 1344) 2026-10-20 00:30:52 +00:00
mesh-tools 5ba1da5b82 Carry the gate's merge asks to the router and write the operator's answer back, a warrant that does not hold said as a condition (issue 1343) 2026-10-20 00:28:36 +00:00
mesh-tools 8ee56dbb5c Grant the bridge's test principal only a read of the gate's buckets, as the controller composes it (issues 1341, 1342) 2026-10-20 00:24:40 +00:00
mesh-tools be5c1b371e Merge the pull request a gate's merges key hands the bridge, with the Approve's heads, one pass stopping at a refusal (issue 1342) 2026-10-20 00:17:44 +00:00
mesh-tools eeb5561a23 Perform the gate's owed acts once per revision from the bridge, a refusal written on FORGE (issue 1341) 2026-10-20 00:16:43 +00:00
mesh-tools a0883a174b Test the bridge's two state buckets, heads and performed (issue 1341) 2026-10-20 00:15:54 +00:00
mesh-tools 7686cfc55d Turn an owed key into the forge call that performs it, and a refusal into a refused message (issue 1341) 2026-10-20 00:15:29 +00:00
mesh-tools 507fac41fe Declare the bridge's performed bucket (issue 1341) 2026-10-20 00:14:42 +00:00
mesh-tools 9c15112d93 Merge pull request 'Take the SDK that counts a build outcome as the controller writes it, so modules get their fingerprints (issue 1281)' (#8) from fix/1281-outcome-is-a-build into main 2026-10-19 23:02:00 +00:00
mesh-tools 3c655c5ab4 Write a failed build as the controller does, an outcome with failed set (issue 1281)
mesh/repo-check head 3c655c5ab449 merged into trunk 6a23627426c6 — pass: its merge-check.sh passed
mesh/merge-gate head 3c655c5ab449 merged into trunk 6a23627426c6 — pass: group fix/1281-outcome-is-a-build composed: every machine composes with the change…
mesh/delivery delivered
mesh/delivery-group group fix/1281-outcome-is-a-build merged: every member is merged
2026-10-19 22:47:41 +00:00
mesh-tools 8ffcf02a9c Take the SDK that counts a stateless build record as an outcome; fixtures in the controller's layout (issue 1281) 2026-10-19 22:44:21 +00:00
mesh-tools 6a23627426 Merge pull request 'Recognise a trunk build by repository identity and an empty ref in the indexer and the view (issue 1281)' (#6) from fix/1281-trunk-builds-by-repository into main 2026-10-19 20:58:07 +00:00
jschoubben 4b8e71370c Recognise a trunk build by repository identity and an empty ref in the indexer and the view (issue 1281)
mesh/repo-check head 4b8e71370cc2 merged into trunk fa4be27c1e40 — pass: its merge-check.sh passed
mesh/merge-gate head 4b8e71370cc2 merged into trunk fa4be27c1e40 — pass: group fix/1281-trunk-builds-by-repository composed: every machine composes with th…
mesh/delivery delivered
mesh/delivery-group group fix/1281-trunk-builds-by-repository merged: every member is merged
Build records name their repository by clone URL and a plan-asked build has no ref, so no module fingerprint was ever written or compared; both now use the SDK's tests.
2026-10-19 22:39:51 +02:00
249 changed files with 38237 additions and 619 deletions
-74
View File
@@ -1,74 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# This repository's own check, mesh/repo-check (novox/hq ADR 0238 §4), written with its first commit (ADR 0428 §4):
# every Go module in the tree is vetted, its unit tests run, and then its integration tests (-tags integration).
# A tree with no Go module has nothing to test, and the check says so and passes.
#
# No module commits its dependencies (no vendor directory: .gitignore). A module's dependencies are fetched, or a
# requirement of the mesh's own forge (the pinned mesh-sdk commit) is taken from its repository's clone beside the check
# (MESH_CHECK_BESIDE), through a go.mod of its own (-modfile), by mesh-catalog's helper checks/deps, built from the
# clone of mesh-catalog beside this check and not copied here: the build seat forgets the forge credential before the
# check runs, so a commit the seat's module cache does not hold cannot be fetched from the forge (novox/hq issue 385). A
# module whose dependencies cannot be had **fails the check**, by name and why: its tests could not run, and a test that
# could not run is a failure, never a note. Run by hand, without MESH_CHECK_BESIDE, the dependencies are fetched.
set -eu
cd "$(dirname "$0")"
# The toolchain the check runs in sets -mod=vendor; there is no vendor directory, so the modules are used as they are.
GOFLAGS="$(printf '%s' "${GOFLAGS:-}" | sed 's/-mod=[a-z]*//g') -mod=mod"
export GOFLAGS
export GOPRIVATE=git.novox.be
modules=$(find . -name go.mod -not -path '*/vendor/*' -not -path './.git/*' | sort)
if [ -z "$modules" ]; then
echo "This tree holds no Go module, so there are no tests to run; the check passes."
exit 0
fi
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
deps=""
helper="${MESH_CHECK_BESIDE:-}/mesh-catalog/checks/deps"
if [ -n "${MESH_CHECK_BESIDE:-}" ] && [ -f "$helper/main.go" ]; then
if ! (cd "$helper" && go build -o "$work/deps" .); then
echo "FAILED: the dependency helper of mesh-catalog beside this check could not be built"
exit 1
fi
deps="$work/deps"
elif [ -n "${MESH_CHECK_BESIDE:-}" ]; then
echo "NOT TAKEN FROM BESIDE: no mesh-catalog/checks/deps beside this check ($MESH_CHECK_BESIDE); the dependencies are fetched"
fi
untestable=""
for mod in $modules; do
dir=$(dirname "$mod")
modfile=""
if [ -n "$deps" ]; then
if ! modfile=$("$deps" -beside "$MESH_CHECK_BESIDE" -work "$work" "$dir"); then
echo "FAILED: $dir could not be tested: $modfile"
untestable="$untestable $dir"
continue
fi
[ -n "$modfile" ] && echo "$dir takes its forge dependencies from the clones beside the check"
elif ! fetched=$(cd "$dir" && go mod download 2>&1); then
echo "FAILED: $dir could not be tested: its dependencies cannot be fetched: $(printf '%s\n' "$fetched" | tail -n 1)"
untestable="$untestable $dir"
continue
fi
mf=""
[ -n "$modfile" ] && mf="-modfile=$modfile"
unformatted=$(cd "$dir" && gofmt -l .)
if [ -n "$unformatted" ]; then
echo "$dir is not gofmt'd: $unformatted"
exit 1
fi
echo "== $dir: go vet and the unit tests"
(cd "$dir" && go vet $mf ./... && go test $mf ./...)
echo "== $dir: the integration tests (-tags integration)"
(cd "$dir" && go vet $mf -tags integration ./... && go test $mf -tags integration ./...)
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
@@ -0,0 +1,161 @@
package main
import (
"context"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-assigner/internal/assign"
)
// AssignerModule is the asker the assigner's compose asks name.
const AssignerModule = "mesh-delivery-assigner"
// composeAsks are the compose asks the assigner put on the Go build seat's check stream (novox/hq ADR 0482 clause 4,
// issues 1622 and 1623), by what each composes: one ask per key, put once and followed by a watch of its step key until
// it ends, which wakes a pass. No timer: an ask that waits in the stream waits (its backlog is the view's and the
// watchman's to show). A restart forgets them and asks again; the build module keeps a composer built for a commit, so a
// second ask builds nothing.
type composeAsks struct {
ctx context.Context
asker buildrun.Asker
wake func()
logf func(string, ...any)
now func() time.Time
mu sync.Mutex
asks map[string]*composeAsk
}
type composeAsk struct {
id string
cancel context.CancelFunc
ran *assign.Ran
}
func newComposeAsks(ctx context.Context, asker buildrun.Asker, wake func(), logf func(string, ...any)) *composeAsks {
return &composeAsks{ctx: ctx, asker: asker, wake: wake, logf: logf, now: time.Now, asks: map[string]*composeAsk{}}
}
// get is the end of the ask that composes key, or nil and its id while it runs. The first get of a key puts the ask.
// A job its layout refuses ends at once, failed with why: never silent.
func (a *composeAsks) get(key string, job delivery.ComposeJob) (*assign.Ran, string) {
a.mu.Lock()
defer a.mu.Unlock()
if k, ok := a.asks[key]; ok {
return k.ran, k.id
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: AssignerModule,
At: a.now(), Compose: &job}
ctx, cancel := context.WithCancel(a.ctx)
k := &composeAsk{id: ask.ID, cancel: cancel}
a.asks[key] = k
if err := a.asker.Put(ctx, ask); err != nil {
a.logf("the compose ask %s for %s could not be put: %v", ask.ID, key, err)
k.ran = &assign.Ran{State: delivery.StepFailed, Reason: "the compose ask could not be put on the Go build seat: " + err.Error()}
cancel()
return k.ran, k.id
}
go func() {
step, err := a.asker.Wait(ctx, ask)
if ctx.Err() != nil {
return
}
ran := assign.RanOf(step)
if err != nil {
ran = assign.Ran{State: delivery.StepFailed, Reason: "the compose ask's step could not be followed: " + err.Error()}
}
a.mu.Lock()
k.ran = &ran
a.mu.Unlock()
a.wake()
}()
return nil, k.id
}
// sweep forgets, and stops following, every ask whose key no pass asked for this time: a candidate gone or moved on.
func (a *composeAsks) sweep(seen map[string]bool) {
a.mu.Lock()
defer a.mu.Unlock()
for key, k := range a.asks {
if !seen[key] {
k.cancel()
delete(a.asks, key)
}
}
}
// composeAtHead is the composition of a group with a mesh-sdk member: the composer at its head and at each reader's
// mesh-sdk, asked of the Go build seat, and the verdict once every ask ended; until then the verdict waits, naming the
// asks. The keys it used are marked in seen.
func (p *passer) composeAtHead(ctx context.Context, m assign.Mesh, ch assign.Change, seen map[string]bool, began time.Time) delivery.CompositionVerdict {
waiting := func(words string) delivery.CompositionVerdict {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionWaits, Words: words}
}
head, err := assign.HeadJob(m, ch)
if err != nil {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionFail, SDK: "mesh-sdk " + ch.SDKHead,
Words: "the compose ask at the group's mesh-sdk head cannot be made: " + err.Error()}
}
var pending []string
key := "head." + ch.Group + "." + ch.Heads + "." + ch.SDKHead
seen[key] = true
headRan, id := p.asks.get(key, head)
if headRan == nil {
pending = append(pending, "mesh-sdk "+ch.SDKHead+" (ask "+id+")")
}
var readers []assign.Reader
var notJudged []string
for _, module := range sortedNames(assign.ReaderModules) {
repository := assign.ReaderModules[module]
var commits []string
member := false
if h := ch.Members[repository]; h != "" {
commits, member = []string{h}, true
} else {
commits = p.w.runningCommits(ctx, kvGetter(p.kvs[delivery.BuildsBucket]), module)
}
if len(commits) == 0 {
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: no build of it is assigned with a "+
"commit on record, and the group holds no member of "+repository)
continue
}
for _, commit := range commits {
job, err := assign.ReaderJob(ch, module, repository, commit)
if err != nil {
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member,
Ran: assign.Ran{State: delivery.StepFailed, Reason: err.Error()}})
continue
}
rkey := "reader." + ch.Group + "." + ch.Heads + "." + module + "." + commit
seen[rkey] = true
ran, rid := p.asks.get(rkey, job)
if ran == nil {
pending = append(pending, module+" at "+commit+" (ask "+rid+")")
continue
}
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member, Ran: *ran})
}
}
if len(pending) > 0 {
return waiting("composing on the Go build seat with the mesh-sdk of " + strings.Join(pending, ", "))
}
v := assign.FromComposer(m, ch, *headRan, readers, time.Since(began))
v.Warnings = append(v.Warnings, notJudged...)
return v
}
func sortedNames(m map[string]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,180 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// holder is a test holder of the Go build seat: it takes every ask from the seat's check worker, as mesh-build-go does,
// and answers a compose ask by running mesh-sdk's composer over the request filled from the indexer's manifests (the
// linked composer standing for the one the build module builds at the asked commit), writing its step on its node's key.
type holder struct {
mu sync.Mutex
asks []delivery.StepAsk
}
func (h *holder) taken() []delivery.StepAsk {
h.mu.Lock()
defer h.mu.Unlock()
return append([]delivery.StepAsk(nil), h.asks...)
}
func (b *bed) hold(t *testing.T) *holder {
t.Helper()
seat, _ := delivery.AskStream(delivery.AskGo)
worker, _ := delivery.AskWorker(delivery.AskGo, delivery.AskCheck)
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskCheck)
cons, err := b.js.CreateOrUpdateConsumer(b.ctx, seat, jetstream.ConsumerConfig{Durable: worker, FilterSubject: subject, AckPolicy: jetstream.AckExplicitPolicy})
if err != nil {
t.Fatal(err)
}
steps := b.buckets[delivery.StepsBucket(goBuildModule)]
h := &holder{}
cc, err := cons.Consume(func(m jetstream.Msg) {
a, err := delivery.DecodeStepAsk(m.Data())
if err != nil || a.Compose == nil {
t.Errorf("the seat was put an ask that is not a compose: %v %s", err, m.Data())
m.Ack()
return
}
h.mu.Lock()
h.asks = append(h.asks, a)
h.mu.Unlock()
req, err := composer.Decode(a.Compose.Request)
if err == nil {
err = composer.Fill(&req, func(digest string) ([]byte, error) {
key, _ := delivery.ManifestsKey(digest)
e, err := b.buckets[delivery.ManifestsBucket].Get(b.ctx, key)
if err != nil {
return nil, err
}
man, err := delivery.DecodeManifest(e.Value())
return []byte(man.JSON), err
})
}
step := delivery.Step{ID: a.ID, Tool: "compose", Node: "ace", State: delivery.StepPassed, Started: time.Now().UTC(), Ended: time.Now().UTC()}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else if v, err := composer.Run(req); err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else {
vr, _ := json.Marshal(v)
step.Result, _ = json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0123456789abcdef", Verdict: vr})
}
raw, err := delivery.EncodeStep(step)
if err != nil {
t.Errorf("the step cannot be written: %v", err)
}
key, _ := delivery.AskStepKey("ace", a.ID)
if _, err := steps.Put(b.ctx, key, raw); err != nil {
t.Errorf("the step %s: %v", key, err)
}
m.Ack()
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(cc.Stop)
return h
}
// 1622/AC1-AC3 and 1623 on the bus: started whole, the assigner composes a group holding a mesh-sdk pull request by
// asking the Go build seat for the composer at the member's head, and for the composer at the mesh-sdk the running
// controller pins; it waits naming the asks while they run, writes the verdict from their answers naming the mesh-sdk the
// head's composer was built with, and asks nothing more on a later pass of the same heads.
func TestAGroupWithAnSDKMemberIsComposedByTheComposerAtItsHead(t *testing.T) {
b := startBed(t, "")
trunk := b.manifest(t, "web", `{"module":"web","version":"1"}`)
changed := b.manifest(t, "web", `{"module":"web","version":"2"}`)
mod, err := delivery.EncodeModule(delivery.Module{FP2: delivery.SourcePrefix2 + strings.Repeat("a", 64), Manifest: trunk, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ModulesBucket, "web", mod)
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
running := strings.Repeat("3", 40)
b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`))
b.putJSON(t, delivery.BuildsBucket, delivery.BuildKey("mesh-controller", "c-1"), []byte(`{"id":"c-1","repository":"novox/mesh-controller","commit":"`+running+`"}`))
sdkHead := strings.Repeat("4", 40)
heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
pulls := []delivery.CandidatePull{{Head: heads[0]}, {Head: heads[1],
Modules: []string{"web"}, Fps: map[string]delivery.CandidateModule{"web": {FP2: delivery.SourcePrefix2 + strings.Repeat("b", 64), Manifest: changed}}}}
cand, err := delivery.EncodeCandidate(delivery.Candidate{Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
b.putJSON(t, delivery.CandidatesBucket, ckey, cand)
b.start(t)
key, _ := delivery.CompositionKey("feat/x", hash)
// No holder takes the asks yet: the composition waits, naming them.
waits := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State == delivery.CompositionWaits })
if !strings.Contains(waits.Words, "mesh-sdk "+sdkHead) || !strings.Contains(waits.Words, "mesh-controller at "+running) {
t.Fatalf("the waiting composition says %q, want the head's and the running controller's asks", waits.Words)
}
h := b.hold(t)
v := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State != delivery.CompositionWaits })
if v.State != delivery.CompositionPass || len(v.Machines) != 1 || v.Machines[0] != "g14" || v.SDK == "" {
t.Fatalf("composition = %+v, want a pass on g14", v)
}
asks := h.taken()
if len(asks) != 2 {
t.Fatalf("the seat was asked %d composes, want two (the head, the running controller)", len(asks))
}
byRepo := map[string]delivery.StepAsk{}
for _, a := range asks {
byRepo[a.Compose.Fetch["repository"].(string)] = a
if a.Asker != AssignerModule || a.Kind != delivery.AskCheck || a.Language != delivery.AskGo {
t.Errorf("ask = %+v", a)
}
}
if a := byRepo["novox/mesh-sdk"]; a.Compose == nil || a.Compose.Fetch["commit"] != sdkHead || a.Compose.Dir != "go" {
t.Errorf("the head's ask = %+v", a.Compose)
}
if a := byRepo["novox/mesh-controller"]; a.Compose == nil || a.Compose.Fetch["commit"] != running || a.Compose.Dir != delivery.RootPath {
t.Errorf("the running controller's ask = %+v", a.Compose)
}
// A later pass of the same heads (woken by any change) asks nothing more.
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
time.Sleep(500 * time.Millisecond)
if n := len(h.taken()); n != 2 {
t.Errorf("a later pass of the same heads asked again: %d asks", n)
}
}
// verdict waits for the composition under key to satisfy ok, and answers it.
func (b *bed) verdict(t *testing.T, key string, ok func(delivery.CompositionVerdict) bool) delivery.CompositionVerdict {
t.Helper()
var last delivery.CompositionVerdict
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.buckets[delivery.AssignerCompositionsBucket].Get(b.ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
continue
}
if err != nil {
t.Fatal(err)
}
if last, err = delivery.DecodeCompositionVerdict(e.Value()); err != nil {
t.Fatal(err)
}
if ok(last) {
return last
}
}
t.Fatalf("no composition under %s as wanted after 10 s; last %+v", key, last)
return last
}
@@ -0,0 +1,896 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// bed is an embedded bus with JetStream, every bucket the assigner reads and keeps, and the stream the mesh's events
// are kept in, as the controller's broker raises them.
type bed struct {
srv *server.Server
js jetstream.JetStream
ctx context.Context
dir string
buckets map[string]jetstream.KeyValue
}
func startBed(t *testing.T, leaveOut string) *bed {
t.Helper()
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "assigner")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
srv, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: filepath.Join(dir, "js"), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
nc, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
t.Cleanup(cancel)
b := &bed{srv: srv, js: js, ctx: ctx, dir: dir, buckets: map[string]jetstream.KeyValue{}}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == leaveOut {
continue
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 5})
if err != nil {
t.Fatal(err)
}
b.buckets[name] = kv
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
// The operator-channel seat's stream of conditions, which the assigner says its conditions on.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "CONDITIONS", Subjects: []string{"mesh.seat.operator-channel.accept.condition.>"}}); err != nil {
t.Fatal(err)
}
// The Go build seat's work queue, as the broker derives it from the seat's accepted kinds (delivery.AskStream).
seat, _ := delivery.AskStream(delivery.AskGo)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: seat, Subjects: []string{"mesh.seat.node-build-go.accept.>"},
Retention: jetstream.WorkQueuePolicy}); err != nil {
t.Fatal(err)
}
return b
}
// start runs the program as the node does, through run with its environment and a broker file; it answers the
// channel run's error arrives on.
func (b *bed) start(t *testing.T) <-chan error {
t.Helper()
done, _ := b.startStoppable(t)
return done
}
// startStoppable is start, and answers the function that stops this one program (a replaced assigner is stopped, then
// another started).
func (b *bed) startStoppable(t *testing.T) (<-chan error, func()) {
t.Helper()
broker := filepath.Join(b.dir, "broker")
cred, _ := json.Marshal(process.Account{URL: b.srv.ClientURL(), Node: "n1", Module: "mesh-delivery-assigner", User: "n1.mesh-delivery-assigner", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
env := map[string]string{"MESH_BROKER_FILE": broker}
ctx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() {
done <- run(ctx, func(k string) string { return env[k] }, io.Discard)
close(done) // a program that ended on its own (a refused start) is not waited for again at the cleanup
}()
stopped := func() {
stop()
select {
case <-done:
case <-time.After(10 * time.Second):
}
}
t.Cleanup(stopped)
return done, stopped
}
// 1297/B2b: started whole on its credential, the assigner says on the bus the four states it keeps, each readable,
// having opened every bucket it names.
func TestTheAssignerStartsAndSaysWhatItKeeps(t *testing.T) {
b := startBed(t, "")
done := b.start(t)
events, err := b.js.Stream(b.ctx, "EVENTS")
if err != nil {
t.Fatal(err)
}
var said announce.States
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
select {
case err := <-done:
t.Fatalf("the assigner stopped: %v", err)
default:
}
m, err := events.GetLastMsgForSubject(b.ctx, announce.Subject("mesh-delivery-assigner"))
if err != nil {
continue
}
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
break
}
if said.Module != "mesh-delivery-assigner" || said.Node != "n1" || len(said.States) != 4 || !said.Readable("asks") ||
!said.Readable("wants") || !said.Readable("differences") || !said.Readable("compositions") {
t.Fatalf("the assigner said %+v, want its four states, each readable", said)
}
}
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets {
b := startBed(t, missing)
select {
case err := <-b.start(t):
if err == nil || !strings.Contains(err.Error(), missing) {
t.Errorf("without %s the assigner stopped with %v, want the bucket named", missing, err)
}
case <-time.After(15 * time.Second):
t.Errorf("without %s the assigner started", missing)
}
}
}
// putJSON writes a value into one of the bed's buckets.
func (b *bed) putJSON(t *testing.T, bucket, key string, value []byte) {
t.Helper()
if _, err := b.buckets[bucket].Put(b.ctx, key, value); err != nil {
t.Fatalf("%s %s: %v", bucket, key, err)
}
}
// compose says the controller composed module for node at fp, with the digest d (a letter repeated) and the judgement of the
// move to it, in mesh-controller_composed.
func (b *bed) compose(t *testing.T, node, module, fp string, d byte, judgement delivery.Judgement) string {
t.Helper()
key, err := delivery.ComposedKey(node, module, fp)
if err != nil {
t.Fatal(err)
}
digest := "sha256:" + strings.Repeat(string(d), 64)
cd := delivery.ComposedDigest{Digest: digest, Judgement: judgement}
if judgement != delivery.JudgedNeither {
cd.Against, cd.Words = "sha256:"+strings.Repeat("0", 64), "cannot be told"
}
v, err := delivery.EncodeComposedDigest(cd)
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ControllerComposedBucket, key, v)
return digest
}
// manifest keeps a manifest in the indexer's bucket by its digest and answers the digest.
func (b *bed) manifest(t *testing.T, module, body string) string {
t.Helper()
digest := delivery.ManifestDigest([]byte(body))
key, _ := delivery.ManifestsKey(digest)
value, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: body})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ManifestsBucket, key, value)
return digest
}
// 1297/AC1: against the real bus, a fixture change touching one module on one machine is composed and answered within
// seconds under the checker's key of the check, naming the machine composed and the mesh-sdk it composed with.
func TestAChangeIsComposedOnTheBus(t *testing.T) {
b := startBed(t, "")
trunk := b.manifest(t, "web", `{"module":"web","version":"1"}`)
changed := b.manifest(t, "web", `{"module":"web","version":"2"}`)
mod, err := delivery.EncodeModule(delivery.Module{FP2: delivery.SourcePrefix2 + strings.Repeat("a", 64), Manifest: trunk, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ModulesBucket, "web", mod)
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
b.putJSON(t, provider.AssignmentsBucket, "ace.other", []byte(`{"node":"ace","module":"other","build":"build-2"}`))
heads := []delivery.Head{{Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
// 1927/AC1: the candidate carries the fingerprint set the SDK computes, as the indexer writes it.
pulls := []delivery.CandidatePull{{Head: heads[0],
Modules: []string{"web"}, Fps: map[string]delivery.CandidateModule{"web": {FP2: delivery.SourcePrefix2 + strings.Repeat("b", 64), Manifest: changed}}}}
cand, err := delivery.EncodeCandidate(delivery.Candidate{Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
b.putJSON(t, delivery.CandidatesBucket, ckey, cand)
began := time.Now()
b.start(t)
key, _ := delivery.CompositionKey("feat/x", hash)
var v delivery.CompositionVerdict
for deadline := time.Now().Add(10 * time.Second); ; time.Sleep(50 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("no composition under %s after 10 s", key)
}
e, err := b.buckets[delivery.AssignerCompositionsBucket].Get(b.ctx, key)
if err != nil {
continue
}
if v, err = delivery.DecodeCompositionVerdict(e.Value()); err != nil {
t.Fatal(err)
}
break
}
if v.State != delivery.CompositionPass || len(v.Machines) != 1 || v.Machines[0] != "g14" || v.SDK == "" {
t.Fatalf("composition = %+v, want a pass on g14 alone naming its mesh-sdk", v)
}
if took := time.Since(began); took > 5*time.Second {
t.Errorf("the composition took %s", took)
}
// 1622/AC3: a group without a mesh-sdk member builds no composer: no ask is put on the Go build seat.
seat, _ := delivery.AskStream(delivery.AskGo)
if st, err := b.js.Stream(b.ctx, seat); err != nil {
t.Fatal(err)
} else if info, err := st.Info(b.ctx); err != nil || info.State.Msgs != 0 {
t.Errorf("a group without a mesh-sdk member put %d asks on the Go build seat (%v)", info.State.Msgs, err)
}
}
// get waits for a key of one of the bed's buckets and answers its value.
func (b *bed) get(t *testing.T, bucket, key string) []byte {
t.Helper()
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[bucket].Get(b.ctx, key); err == nil {
return e.Value()
}
}
t.Fatalf("no %s %s after 10 s", bucket, key)
return nil
}
// 1297/B4 on the bus, 1906/D5: the controller sends ace, g14 and shanks builds while the canary ace does not run F healthy yet.
// Started whole, the assigner wants F on the canary and writes no difference at all: there is no send to compare with (ADR 0539
// §8). 1945/AC19, AC21: shanks, which runs the older fingerprint (composed and granted) with no want, gets its first want at that
// fingerprint, never at F; g14, which runs a build of the old form, gets none, so web is on neither list there and the assigner
// raises the condition naming g14, web and the canary rule; once ace runs F healthy, g14's first want names F and the condition is
// cleared.
func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
built, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
if err != nil {
t.Fatal(err)
}
bkey, _ := delivery.BuilderBuildKey("web", fNew)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
for _, fp := range []string{fNew, fOld} {
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
}
runs := map[string]string{"ace": fOld, "g14": "build-9", "shanks": fOld}
for _, node := range []string{"ace", "g14", "shanks"} {
b.compose(t, node, "web", fNew, 'c', delivery.JudgedNeither)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, node, on)
have, _ := delivery.EncodeHave(delivery.Have{FP: runs[node], Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, node+".web", have)
b.putJSON(t, provider.AssignmentsBucket, node+".web", []byte(`{"node":"`+node+`","module":"web","build":"b-`+node+`"}`))
}
b.compose(t, "shanks", "web", fOld, 'a', delivery.JudgedNeither)
b.start(t)
wkey, _ := delivery.WantKey("ace", "web")
want, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey))
if err != nil || want.FP != fNew {
t.Fatalf("the canary ace's want = %+v, %v", want, err)
}
if w := b.wantNow(t, "shanks", "web", "shanks' first want at what it runs", func(w delivery.Want) bool { return w.FP != "" }); w.FP != fOld || w.Composed != "sha256:"+strings.Repeat("a", 64) || w.Newest != fNew {
t.Errorf("shanks' want = %+v, want the fingerprint it runs naming its composition, held back from F", w)
}
var neither asks.Condition
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && neither.Key == ""; time.Sleep(100 * time.Millisecond) {
for k, c := range b.conditionsSaid(t) {
if strings.Contains(k, ".neither.") && c.State == asks.StateRaised {
neither = c
}
}
}
if neither.Subject.Machine != "g14" || neither.Subject.ID != "web" || neither.Severity != asks.SeverityUrgent || !strings.Contains(neither.Explanation, "canary") {
t.Errorf("the condition of web on neither list on g14 = %+v", neither)
}
if keys, err := b.buckets[delivery.AssignerDifferencesBucket].Keys(b.ctx); err == nil && len(keys) != 0 {
t.Errorf("differences were written: %v", keys)
}
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, "want.g14.web"); err == nil {
t.Errorf("g14, which runs an old-form build and is held back by the canary, was given a want: %s", e.Value())
}
// 1945/AC21: once the canary ace runs F healthy, g14's first want names F by the rules as usual, and web leaves neither list
// there by itself: the condition is cleared.
healthyF, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthyF)
if w := b.wantNow(t, "g14", "web", "g14's first want at F once the canary runs it healthy", func(w delivery.Want) bool { return w.FP != "" }); w.FP != fNew || w.Composed != "sha256:"+strings.Repeat("c", 64) || w.Newest != "" {
t.Errorf("g14's want = %+v, want F with its composition", w)
}
cleared := false
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && !cleared; time.Sleep(100 * time.Millisecond) {
c, ok := b.conditionsSaid(t)[neither.Key]
cleared = ok && c.State == asks.StateCleared
}
if !cleared {
t.Errorf("the condition of web on neither list on g14 was not cleared once g14 got its want")
}
}
// 1297/B9 (the assigner's half, ADR 0528 clause 6): started whole, the assigner holds a fingerprint whose health check
// failed on a node and writes the ask about it in its own asks, which the view and the teller read; its asks bucket is
// the one its manifest declares (ownBuckets, held equal to the manifest by TestOpensWhatItsManifestNames).
func TestAHeldFingerprintIsAskedAbout(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("c", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putJSON(t, provider.AssignmentsBucket, "ace.web", []byte(`{"node":"ace","module":"web","build":"b-1"}`))
b.start(t)
hkey, _ := delivery.HeldKey("web", fNew)
// 1906/H3: the held record names the machine and keeps the health, when and the node's words.
if h, err := delivery.DecodeHeld(b.get(t, delivery.AssignerWantsBucket, hkey)); err != nil || h.Node != "ace" ||
!strings.HasPrefix(h.Words, "health failed, phase running, seen at ") || !strings.HasSuffix(h.Words, ": connection refused") {
t.Fatalf("held = %+v, %v", h, err)
}
akey, _ := delivery.HoldAskKey("web", fNew)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || a.Asking <= 0 || a.Ask.ID != delivery.AskingID(akey, a.Asking) || len(a.Ask.Options) != 2 {
t.Fatalf("the ask = %+v, %v", a, err)
}
}
// conditionsSaid reads what the assigner said on the operator channel's subject for conditions, by condition key.
func (b *bed) conditionsSaid(t *testing.T) map[string]asks.Condition {
t.Helper()
stream, err := b.js.Stream(b.ctx, "CONDITIONS")
if err != nil {
t.Fatal(err)
}
out := map[string]asks.Condition{}
cons, err := stream.CreateOrUpdateConsumer(b.ctx, jetstream.ConsumerConfig{AckPolicy: jetstream.AckNonePolicy})
if err != nil {
t.Fatal(err)
}
for {
batch, err := cons.FetchNoWait(100)
if err != nil {
t.Fatal(err)
}
n := 0
for m := range batch.Messages() {
n++
c, err := asks.DecodeCondition(m.Data())
if err != nil {
t.Fatal(err)
}
out[c.Key] = c
}
if n == 0 {
return out
}
}
}
// 1906/D3 (ADR 0539 §19), whole on the bus: a module holding data whose new fingerprint failed its health check on the
// canary is held, not set back, and asked about (restore it, or leave it stopped); the condition names the module, the node
// and the check's words.
func TestAFailedDataModuleIsAskedToRestore(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("d", 64)
digest := b.manifest(t, "mail", `{"module":"mail","version":"1","resources":[{"id":"data","type":"directory","path":"/var/lib/mail"}],"data":{"own":[{"id":"mail","path":"${dir:data}","class":"irreplaceable","backup":"copy"}]}}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/mail", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "mail", mod)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.mail", failed)
wantKey, _ := delivery.WantKey("ace", "mail")
want, _ := delivery.EncodeWant(delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("c", 64)})
b.putJSON(t, delivery.AssignerWantsBucket, wantKey, want)
b.putJSON(t, provider.AssignmentsBucket, "ace.mail", []byte(`{"node":"ace","module":"mail","build":"b-1"}`))
b.start(t)
akey, _ := delivery.RestoreAskKey("ace", "mail", fNew)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || len(a.Ask.Options) != 2 || a.Ask.Options[0].ID != "restore" || a.Ask.Options[1].ID != "leave-stopped" {
t.Fatalf("the restore ask = %+v, %v", a, err)
}
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wantKey)); err != nil || w.FP != fNew {
t.Errorf("the data module was set back: want = %+v, %v", w, err)
}
key := ""
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && key == ""; time.Sleep(100 * time.Millisecond) {
for k, c := range b.conditionsSaid(t) {
if c.State == asks.StateRaised && c.Subject.ID == "mail" && strings.Contains(k, ".held.") {
if c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") {
t.Fatalf("the condition = %+v, want it to name ace and the check's words", c)
}
key = k
}
}
}
if key == "" {
t.Fatal("no condition for the held fingerprint was said")
}
}
// 1906/D8 (scenario D of acceptance test 18, ADR 0539 §19), whole on the bus: the assigner is stopped while it writes the wants
// of forty modules and another is started; every module's want is written, and each exactly once, by revision: nothing lost
// and nothing written again.
func TestTheAssignerReplacedResumesWithNothingLostOrRepeated(t *testing.T) {
b := startBed(t, "")
const modules = 40
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
names := make([]string, modules)
for i := range names {
name := fmt.Sprintf("mod%02d", i)
names[i] = name
fp := delivery.SourcePrefix2 + strings.Repeat(string(rune('a'+i%6)), 63) + fmt.Sprintf("%x", i%16)
digest := b.manifest(t, name, fmt.Sprintf(`{"module":%q,"version":"1"}`, name))
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/" + name, Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, name, mod)
built, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
if err != nil {
t.Fatal(err)
}
bkey, _ := delivery.BuilderBuildKey(name, fp)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey(name, fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
b.compose(t, "ace", name, fp, 'c', delivery.JudgedNeither)
b.putJSON(t, provider.AssignmentsBucket, "ace."+name, []byte(fmt.Sprintf(`{"node":"ace","module":%q,"build":"b-%d"}`, name, i)))
}
count := func() int {
keys, err := b.buckets[delivery.AssignerWantsBucket].Keys(b.ctx)
if err != nil {
return 0
}
return len(keys)
}
_, stop := b.startStoppable(t)
for deadline := time.Now().Add(20 * time.Second); count() == 0 && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
stop() // replaced, wherever its pass had got to
b.start(t)
for deadline := time.Now().Add(30 * time.Second); count() < modules && time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
}
time.Sleep(time.Second) // and no pass of the second writes any again
for _, name := range names {
key, _ := delivery.WantKey("ace", name)
history, err := b.buckets[delivery.AssignerWantsBucket].History(b.ctx, key)
if err != nil {
t.Errorf("the want of %s was never written: %v", name, err)
continue
}
if len(history) != 1 {
t.Errorf("the want of %s was written %d times, want once", name, len(history))
}
}
}
// 1906/D2 (ADR 0539 §6), whole on the bus: a want waits for the composition of its module for the node, names its digest once it
// appears, and a new revision of the composition at the same fingerprint is written as a new want; the node's own want
// `want.<node>.node` is written once the node's composition exists, and again when its digest changes.
func TestAWantWaitsForItsComposition(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("e", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
bkey, _ := delivery.BuilderBuildKey("web", fNew)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey("web", fNew)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
b.putJSON(t, provider.AssignmentsBucket, "ace.web", []byte(`{"node":"ace","module":"web","build":"b-1"}`))
b.start(t)
wkey, _ := delivery.WantKey("ace", "web")
time.Sleep(time.Second)
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, wkey); err == nil {
t.Fatalf("a want was written before the composition existed: %s", e.Value())
}
first := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedNeither)
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey)); err != nil || w.FP != fNew || w.Composed != first {
t.Fatalf("the want = %+v, %v; want F naming %s", w, err, first)
}
second := b.compose(t, "ace", "web", fNew, 'd', delivery.JudgedNeither)
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, wkey); err == nil {
if w, err := delivery.DecodeWant(e.Value()); err == nil && w.Composed == second {
break
}
}
}
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey)); err != nil || w.Composed != second {
t.Fatalf("after a new revision at the same fingerprint the want = %+v, %v; want it naming %s", w, err, second)
}
// the node's own want
nkey, _ := delivery.WantNodeKey("ace")
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, nkey); err == nil {
t.Fatalf("the node's own want was written before its composition: %s", e.Value())
}
ownKey, _ := delivery.ComposedNodeKey("ace")
own, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("a", 64), Judgement: delivery.JudgedNeither})
b.putJSON(t, delivery.ControllerComposedBucket, ownKey, own)
if w, err := delivery.DecodeNodeWant(b.get(t, delivery.AssignerWantsBucket, nkey)); err != nil || w.Composed != "sha256:"+strings.Repeat("a", 64) {
t.Fatalf("the node's own want = %+v, %v", w, err)
}
}
// fixture is a module web (or the manifest given) at fp on the node ace, which is online: built, granted and assigned.
func (b *bed) fixture(t *testing.T, name, fp, manifest string) {
t.Helper()
digest := b.manifest(t, name, manifest)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/" + name, Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, name, mod)
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
bkey, _ := delivery.BuilderBuildKey(name, fp)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey(name, fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
b.putJSON(t, provider.AssignmentsBucket, "ace."+name, []byte(`{"node":"ace","module":"`+name+`","build":"b-1"}`))
}
func (b *bed) putWant(t *testing.T, node, module string, w delivery.Want) {
t.Helper()
key, _ := delivery.WantKey(node, module)
v, err := delivery.EncodeWant(w)
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.AssignerWantsBucket, key, v)
}
// wantNow waits until the want of module on node satisfies ok and answers it.
func (b *bed) wantNow(t *testing.T, node, module, what string, ok func(delivery.Want) bool) delivery.Want {
t.Helper()
key, _ := delivery.WantKey(node, module)
var last delivery.Want
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, key); err == nil {
if w, err := delivery.DecodeWant(e.Value()); err == nil {
last = w
if ok(w) {
return w
}
}
}
}
t.Fatalf("never saw: %s; the want is %+v", what, last)
return last
}
// answer says the teller wrote the operator's answer to the ask that stands at key (its warrant names that asking's id).
func (b *bed) answer(t *testing.T, key, option string) {
t.Helper()
a, err := delivery.DecodeAssignerAsk(key, b.get(t, delivery.AssignerAsksBucket, key))
if err != nil {
t.Fatal(err)
}
b.answerTo(t, key, option, a.Ask.ID)
}
// answerTo is answer to the ask with this id (the router's warrant names it).
func (b *bed) answerTo(t *testing.T, key, option, askID string) {
t.Helper()
v, err := delivery.EncodeAnswer(delivery.Answer{Option: option, Warrant: askID + "@2026-10-24T12:00:00Z", Words: "the operator"})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.AnswersBucket, key, v)
}
func (b *bed) hasKey(bucket, key string) bool {
_, err := b.buckets[bucket].Get(b.ctx, key)
return err == nil
}
// 1906/D4 (ADR 0539 §19), whole on the bus: the operator's release of a held fingerprint writes its want again, by the rules,
// and clears the condition.
func TestAReleaseAnswerWritesTheWantAgain(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
b.compose(t, "ace", "web", fOld, 'a', delivery.JudgedNeither)
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy) // it ran the older fingerprint healthy before
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
b.start(t)
w := b.wantNow(t, "ace", "web", "set back to the older fingerprint", func(w delivery.Want) bool { return w.FP == fOld })
if w.Newest != fNew {
t.Fatalf("the set-back want = %+v", w)
}
hold, _ := delivery.HoldAskKey("web", fNew)
b.get(t, delivery.AssignerAsksBucket, hold)
b.answer(t, hold, "release")
w = b.wantNow(t, "ace", "web", "the held fingerprint's want written again", func(w delivery.Want) bool { return w.FP == fNew })
if w.Composed != "sha256:"+strings.Repeat("b", 64) || w.Rule != 0 {
t.Errorf("the want = %+v", w)
}
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
cleared := false
for k, c := range b.conditionsSaid(t) {
cleared = cleared || (c.Subject.ID == "web" && c.State == asks.StateCleared && strings.Contains(k, ".held."))
}
if cleared {
return
}
}
t.Error("the condition of the held fingerprint was not cleared")
}
// askNow waits for the backup ask at key to be the one numbered asking and answers it.
func (b *bed) askNow(t *testing.T, key string, asking int) delivery.AssignerAsk {
t.Helper()
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerAsksBucket].Get(b.ctx, key); err == nil {
if a, err := delivery.DecodeAssignerAsk(key, e.Value()); err == nil && a.Asking == asking {
return a
}
}
}
t.Fatalf("the ask at %s never became asking %d", key, asking)
return delivery.AssignerAsk{}
}
// 1906/D4 (ADR 0539 §16), whole on the bus: a module held on a node because its backup failed is asked about, and the answer
// is written into the node's want; an answer "again" followed by another failure is asked again, and its answer is another want
// value, so the node retries; an old answer is never taken for a new revision at the same fingerprint.
func TestABackupAnswerIsWrittenIntoTheNodesWant(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
first := b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: first})
held, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseHeld, Health: delivery.HealthWaits, Detail: "no space left"})
b.putJSON(t, delivery.HaveBucket, "ace.web", held)
b.start(t)
akey, _ := delivery.BackupAskKey("ace", "web", fNew)
a := b.askNow(t, akey, 0)
if !strings.Contains(a.Ask.Explanation, "no space left") || !strings.Contains(a.Ask.Explanation, first) {
t.Fatalf("the backup ask = %+v", a)
}
// an answer to another ask is no answer to this one
b.answerTo(t, akey, delivery.BackupSkip, "someone-else")
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Backup != "" {
t.Fatalf("an answer to another ask was taken: %+v", w)
}
b.answerTo(t, akey, delivery.BackupAgain, a.Ask.ID)
w1 := b.wantNow(t, "ace", "web", "the answer in the want", func(w delivery.Want) bool { return w.Backup == delivery.BackupAgain })
if w1.FP != fNew || w1.Composed != first {
t.Errorf("the want = %+v", w1)
}
// the node retries and the backup fails again: a new have value, asked again under the next number
again, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseHeld, Health: delivery.HealthWaits, Detail: "no space left, again"})
b.putJSON(t, delivery.HaveBucket, "ace.web", again)
a2 := b.askNow(t, akey, 2)
if a2.Ask.ID == a.Ask.ID {
t.Fatalf("asked again under the same id")
}
b.answerTo(t, akey, delivery.BackupAgain, a2.Ask.ID)
w2 := b.wantNow(t, "ace", "web", "the second answer another want value", func(w delivery.Want) bool { return w.Backup == delivery.BackupAgain && w.Words != w1.Words })
if w2.Words == w1.Words {
t.Errorf("the second answer left the want as it was: %+v", w2)
}
// a new revision at the same fingerprint, whose backup fails as well, is asked, and the old answer is not taken for it
second := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedNeither)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: second})
a3 := b.askNow(t, akey, 3)
if !strings.Contains(a3.Ask.Explanation, second) {
t.Errorf("the ask for the new revision = %+v", a3)
}
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Backup != "" {
t.Errorf("the old answer was taken for the new revision: %+v", w)
}
}
const recorded = `{"module":"web","version":"1","upgrade":{"policy":"record","why":"a person takes each build"}}`
// 1906/D4 (ADR 0539 §14), whole on the bus: a move of a module of policy record waits for its ask's yes when judged to recreate
// a container, to move data or not to be judgeable, at a new fingerprint and at a new revision of the same alike; one judged
// neither is written at once.
func TestARecordMoveWaitsForTheYesAndANeitherMoveDoesNot(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "web", fNew, recorded)
old := "sha256:" + strings.Repeat("a", 64)
b.putWant(t, "ace", "web", delivery.Want{FP: fOld, Composed: old})
nw := b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedRecreates)
b.start(t)
akey, _ := delivery.RecordAskKey("ace", "web", nw)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || !strings.Contains(a.Ask.Explanation, "recreates a container") {
t.Fatalf("the record ask = %+v, %v", a, err)
}
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the old want", func(delivery.Want) bool { return true }); w.FP != fOld {
t.Fatalf("the move was written before the yes: %+v", w)
}
b.answer(t, akey, assign.OptionYes)
b.wantNow(t, "ace", "web", "the move after the yes", func(w delivery.Want) bool { return w.FP == fNew && w.Composed == nw })
// a new revision at the same fingerprint, judged to move data, waits as well
rev := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedMovesData)
rkey, _ := delivery.RecordAskKey("ace", "web", rev)
b.get(t, delivery.AssignerAsksBucket, rkey)
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Composed != nw {
t.Fatalf("the revision was written before the yes: %+v", w)
}
// and a revision judged neither is written at once
free := b.compose(t, "ace", "web", fNew, 'd', delivery.JudgedNeither)
b.wantNow(t, "ace", "web", "a move judged neither written at once", func(w delivery.Want) bool { return w.Composed == free })
}
// 1906/D4 (ADR 0539 §7, §15), whole on the bus: the bus's new fingerprint is not written, and a revision of its composition at the
// fingerprint it runs that differs only in its user list is written at once, with the shape the controller gives it: judged
// recreates (the file is mounted), policy record from the bus, user-list-only; no record ask is made.
func TestAUserListRevisionOfTheBusIsWrittenAtOnce(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "bus", fNew, `{"module":"bus","version":"1","provides":[{"name":"mesh-bus"}]}`)
running, _ := delivery.EncodeHave(delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.bus", running)
b.putWant(t, "ace", "bus", delivery.Want{FP: fOld, Composed: "sha256:" + strings.Repeat("a", 64)})
b.compose(t, "ace", "bus", fNew, 'b', delivery.JudgedNeither)
b.start(t)
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "bus", "the want", func(delivery.Want) bool { return true }); w.FP != fOld {
t.Fatalf("the assigner moved the bus to a new fingerprint: %+v", w)
}
// the controller composes the bus at the fingerprint it runs with a new user list
key, _ := delivery.ComposedKey("ace", "bus", fOld)
free := "sha256:" + strings.Repeat("e", 64)
v, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: free, Judgement: delivery.JudgedRecreates, Against: "sha256:" + strings.Repeat("a", 64),
Words: "the user list file is mounted", UserListOnly: true, Policy: delivery.PolicyRecord, PolicyFrom: "the bus"})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ControllerComposedBucket, key, v)
w := b.wantNow(t, "ace", "bus", "the user-list revision written at once", func(w delivery.Want) bool { return w.Composed == free })
if w.FP != fOld {
t.Errorf("the want = %+v, want it at the fingerprint the bus runs", w)
}
rkey, _ := delivery.RecordAskKey("ace", "bus", free)
if b.hasKey(delivery.AssignerAsksBucket, rkey) {
t.Errorf("the user-list revision was asked")
}
}
// 1906/H2, whole on the bus: given a fingerprint held after one failed check, when every node that runs the module then runs
// it healthy, the assigner lifts the hold by itself: the held record and its ask are taken away, the held condition is
// cleared saying why, and the want stays at the fingerprint.
func TestAHoldIsLiftedOnceEveryNodeRunsItHealthy(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
b.start(t)
hkey, _ := delivery.HeldKey("web", fNew)
akey, _ := delivery.HoldAskKey("web", fNew)
b.get(t, delivery.AssignerWantsBucket, hkey)
b.get(t, delivery.AssignerAsksBucket, akey)
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
if b.hasKey(delivery.AssignerWantsBucket, hkey) || b.hasKey(delivery.AssignerAsksBucket, akey) {
continue
}
for k, c := range b.conditionsSaid(t) {
if c.Subject.ID == "web" && c.State == asks.StateCleared && strings.Contains(k, ".held.") && strings.Contains(c.Resolved, "lifted its hold by itself") {
if w := b.wantNow(t, "ace", "web", "the want at the lifted fingerprint", func(w delivery.Want) bool { return w.FP == fNew }); w.Rule != 0 {
t.Errorf("the want = %+v", w)
}
return
}
}
}
t.Fatalf("the hold was not lifted: held %v, ask %v", b.hasKey(delivery.AssignerWantsBucket, hkey), b.hasKey(delivery.AssignerAsksBucket, akey))
}
// 1906/H2, whole on the bus: a hold lifted by itself and written again after the assigner was started again is asked under a
// new id, so the teller raises it; the answer to the first hold does not carry over (1906/H4).
func TestAHoldWrittenAgainAfterALiftAndARestartIsAskedAgain(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
_, stop := b.startStoppable(t)
hkey, _ := delivery.HeldKey("web", fNew)
akey, _ := delivery.HoldAskKey("web", fNew)
first, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); b.hasKey(delivery.AssignerWantsBucket, hkey); time.Sleep(100 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("the hold was not lifted")
}
}
stop()
time.Sleep(1100 * time.Millisecond) // the next hold is written in another second
b.answerTo(t, akey, "keep", first.Ask.ID) // an answer to the first hold, which the later hold must not take
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.start(t)
again, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil {
t.Fatal(err)
}
if again.Ask.ID == first.Ask.ID {
t.Fatalf("the hold written again is asked under the first id %s", first.Ask.ID)
}
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); b.hasKey(delivery.AssignerWantsBucket, hkey); time.Sleep(100 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("the stale \"keep\" of the first hold kept the second one held")
}
}
}
@@ -0,0 +1,254 @@
// Command mesh-delivery-assigner is delivery's assigner (novox/hq ADR 0429 §1 and §6, ADR 0446, ADR 0528, issue 1297): a
// process on its own bus credential that watches what the indexer, the builder, the controller, the node-engines and
// the watchman write, and writes what each node should run (`want.<node>.<module>`, by the six rules), the
// fingerprints it holds, every difference between a want and what the controller sends, and the composition check's
// verdict of each group the checker judges. In release 5 the wants are only compared: the controller still sends what
// each node runs (ADR 0429 §7). It holds no timer: every pass is started by a change it watched.
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"strings"
"syscall"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-assigner:", err)
os.Exit(1)
}
}
// readBuckets are the buckets of other parts the assigner reads, each named in its manifest's reads (the manifest test
// holds the two lists equal), so the controller's broker grants exactly these.
var readBuckets = []string{
provider.AssignmentsBucket,
delivery.BuildsBucket,
delivery.ControllerGrantsBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ManifestsBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.StepsBucket(goBuildModule),
}
// goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes
// their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go"
// ownBuckets are the assigner's own states, as its manifest declares them.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket, delivery.AssignerWantsBucket}
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// and the teller.
var keptStates = []announce.State{
{Name: "asks", Readable: true},
{Name: "compositions", Readable: true},
{Name: "differences", Readable: true},
{Name: "wants", Readable: true},
}
// buckets is every bucket the assigner opened, by name.
type buckets map[string]jetstream.KeyValue
// open opens every bucket the assigner reads and writes; a bucket it cannot open stops the start, named, since a pass
// without it would write wants from half the mesh.
func open(ctx context.Context, js jetstream.JetStream) (buckets, error) {
out := buckets{}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kv, err := js.KeyValue(ctx, name)
if err != nil {
return nil, fmt.Errorf("the bucket %s cannot be opened: %w", name, err)
}
out[name] = kv
}
return out, nil
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to read the bus with")
}
account, err := process.ReadAccount(path)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(account.Node + "." + account.Module),
nats.UserInfo(account.User, account.Password),
nats.CustomInboxPrefix(account.InboxPrefix()),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(account.Fingerprint) != "" {
opts = append(opts, nats.Secure(process.PinnedToFingerprint(account.Fingerprint)))
}
conn, err := nats.Connect(account.URL, opts...)
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
kvs, err := open(ctx, js)
if err != nil {
return err
}
// What the assigner keeps, said on the bus (ADR 0495 §2-§3): a reader is granted from it. Not fatal when it cannot
// be said now: process.Conn says it again at every reconnect.
said, err := process.Connect(account, nats.Name(account.Node+"."+account.Module+".keeps"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer said.Close()
if err := said.Keeps(keptStates...); err != nil {
logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err)
}
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, kvs, sayVia(ctx, js), logger)
}
// watched are the buckets whose every value a pass reads; the manifests are read by digest when a pass needs one, and
// watched only for keys, so a manifest that arrives wakes a composition that waits for it.
var watched = []string{
provider.AssignmentsBucket,
delivery.ControllerGrantsBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.AssignerAsksBucket,
delivery.AssignerCompositionsBucket,
delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket,
}
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(asks.Condition) error, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
select {
case dirty <- struct{}{}:
default:
}
}
errs := make(chan error, len(watched)+1)
settled := make(chan struct{}, len(watched)+1)
watch := func(name string, opts ...jetstream.WatchOpt) error {
kw, err := kvs[name].WatchAll(ctx, opts...)
if err != nil {
return fmt.Errorf("the bucket %s cannot be watched: %w", name, err)
}
go func() {
defer kw.Stop()
for {
select {
case <-ctx.Done():
errs <- nil
return
case e, ok := <-kw.Updates():
if !ok {
errs <- fmt.Errorf("the watch of %s ended", name)
return
}
if e == nil {
settled <- struct{}{}
continue
}
var value []byte
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
w.put(name, e.Key(), value)
wake()
}
}
}()
return nil
}
for _, name := range watched {
var opts []jetstream.WatchOpt
if name == delivery.HaveBucket {
opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back
}
if err := watch(name, opts...); err != nil {
return err
}
}
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err
}
for range len(watched) + 1 {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf)
wake()
for {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
case <-dirty:
p.pass(ctx)
}
}
}
// sayVia publishes a condition on the operator-channel seat's one subject for conditions, under the assigner's own name
// (ADR 0437 clause 9).
func sayVia(ctx context.Context, js jetstream.JetStream) func(asks.Condition) error {
return func(c asks.Condition) error {
subject, err := asks.ConditionSubject(delivery.AssignerModule)
if err != nil {
return err
}
data, err := asks.EncodeCondition(c)
if err != nil {
return err
}
_, err = js.Publish(ctx, subject, data)
return err
}
}
@@ -0,0 +1,59 @@
package main
import (
"context"
"io"
"os"
"path/filepath"
"reflect"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
// (`<module>.<state>` read as the bucket `<module>_<state>`), so it is granted what it reads and reads nothing else.
func TestOpensWhatItsManifestNames(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
var reads []string
for _, r := range m.Reads {
reads = append(reads, strings.Replace(r, ".", "_", 1))
}
var states []string
for _, s := range m.State {
states = append(states, m.Module+"_"+s.Name)
}
for _, pair := range []struct {
what string
code, want []string
}{{"reads", readBuckets, reads}, {"states", ownBuckets, states}} {
got, want := slices.Sorted(slices.Values(pair.code)), slices.Sorted(slices.Values(pair.want))
if !reflect.DeepEqual(got, want) {
t.Errorf("the program's %s are %v, the manifest's %v", pair.what, got, want)
}
}
var kept []string
for _, s := range keptStates {
kept = append(kept, m.Module+"_"+s.Name)
}
if got, want := slices.Sorted(slices.Values(kept)), slices.Sorted(slices.Values(ownBuckets)); !reflect.DeepEqual(got, want) {
t.Errorf("the program announces %v, keeps %v", got, want)
}
}
// 1297/B2a: without a credential the program does not start, and says why.
func TestNoCredential(t *testing.T) {
err := run(context.Background(), func(string) string { return "" }, io.Discard)
if err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
}
}
@@ -0,0 +1,536 @@
package main
import (
"context"
"errors"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/sdkversion"
"mesh-delivery-assigner/internal/assign"
)
// passer makes the assigner's passes: from what the world holds now, it writes each key whose value differs from the
// one written, and only those (the Same functions).
type passer struct {
kvs buckets
w *world
logf func(string, ...any)
sdk string
// say publishes a condition to the operator; said is, by condition key and state, what this process has said, so a
// pass says a condition once and a restart says it again (the router keeps one per key).
say func(asks.Condition) error
said map[string]bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64
asks *composeAsks
// now is the clock; nil is time.Now.
now func() time.Time
// neither is, by node and module, each module on neither list whose condition this process raised (issue 1945). Nil until the
// first pass after a start, which clears what a previous process may have raised (startClears).
neither map[string]neitherOne
}
func (p *passer) clock() time.Time {
if p.now != nil {
return p.now()
}
return time.Now()
}
func (p *passer) pass(ctx context.Context) {
p.compositions(ctx)
p.wants(ctx)
p.nodeWants(ctx)
p.backups(ctx)
}
// nodeWants writes each node's own want, `want.<node>.node`, naming the digest of the node's own composition, once the
// controller has composed it and again whenever that digest changes (ADR 0539 §6).
func (p *passer) nodeWants(ctx context.Context) {
written := p.w.read(delivery.AssignerWantsBucket)
for node, d := range p.w.nodeComposed() {
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
}
if before, err := delivery.DecodeNodeWant(written[key]); err == nil && before.Composed == d.Digest {
continue
}
want := delivery.NodeWant{Composed: d.Digest, Words: "the node's own composition"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
}
// wants writes, for every module the trunk holds a fingerprint of, each node's want by the six rules (ADR 0429 §6), and
// the hold of a fingerprint whose health check failed. The assigner compares its wants with nothing the controller
// sends: the differences bucket is kept read-only with its history (ADR 0539 §8).
func (p *passer) wants(ctx context.Context) {
get := kvGetter(p.kvs[delivery.ManifestsBucket])
assigned, _ := p.w.assigned()
trunk := p.w.trunk()
manifests := map[string]catalogue.Manifest{}
for name, mod := range trunk {
if mod.Manifest == "" {
continue
}
if got := p.w.manifest(ctx, get, mod.Manifest); got.err == "" {
manifests[name] = got.manifest
}
}
nodes := p.w.nodes(manifests, assigned)
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
nodesOf[m] = append(nodesOf[m], node)
}
}
wanted := p.w.wantsOf()
names := map[string]bool{}
for name := range trunk {
names[name] = true
}
for name := range wanted {
names[name] = true
}
neither := map[string]neitherOne{}
// The first pass after a start remembers nothing a previous process said: a neither or held condition it raised would stay
// raised for ever once the module has its want or is no longer held. That pass clears, once, the neither condition of every
// assigned module not on neither list and the held condition of every module not held; a clear of a condition that is not open is
// quiet at the router.
first := p.neither == nil
stale := map[string]neitherOne{}
defer func() { p.sayNeither(neither, stale) }()
for name := range names {
fp := trunk[name].FP2
name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(),
Composed: func(node, fp string) (delivery.ComposedDigest, bool) { return p.w.composed(node, name, fp) },
GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key)
},
Restored: func(node string) bool {
key, err := delivery.RestoreAskKey(node, name, fp)
return err == nil && p.answeredWith(key, assign.OptionRestore)
},
Allowed: func(node, digest string) bool {
key, err := delivery.RecordAskKey(node, name, digest)
return err == nil && p.answeredWith(key, assign.OptionYes)
}}
if fp != "" {
held, _ := delivery.HeldKey(name, fp)
grant, _ := delivery.ControllerGrantKey(name, fp)
hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(delivery.ControllerGrantsBucket, grant)
if raw := p.w.value(delivery.ControllerGrantsBucket, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
}
}
if m.Released {
p.clearHeld(name, fp)
}
}
last := map[string]string{}
for _, node := range m.Nodes {
last[node] = p.w.lastHealthy(node, name, fp)
}
out := assign.Wants(m, nodes, wanted[name], last)
gatherNeither(neither, name, out)
if first {
startClears(stale, m, out)
if fp != "" && !m.Held && out.Held == nil && !out.Lifted {
p.clearHeld(name, fp)
}
}
if out.Held != nil {
key, err := delivery.HeldKey(name, fp)
if err == nil {
held := *out.Held
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeHeld(held) })
}
}
if out.Lifted {
p.lift(ctx, name, fp)
}
if out.Held != nil || (m.Held && !out.Lifted) {
p.holdAsk(ctx, name, fp, out.Held)
p.heldCondition(ctx, name, fp, out.Held, len(out.Restore) > 0)
}
for _, node := range out.Restore {
p.restoreAsk(ctx, name, node, fp)
}
for _, need := range out.Record {
p.recordAsk(ctx, name, fp, need)
}
for node, want := range out.Wants {
key, err := delivery.WantKey(node, name)
if err != nil {
p.logf("no want of %s on %s is written: %v", name, node, err)
continue
}
want := want
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeWant(want) })
}
}
}
// neitherOne is a module assigned to a node that holds no want for it after a pass, and the rule holding its first want back.
type neitherOne struct {
node, module string
holding assign.Holding
}
// gatherNeither adds the nodes on which module is on neither list in its outcome, keyed by node and module.
func gatherNeither(into map[string]neitherOne, module string, out assign.Outcome) {
for node, h := range out.Neither {
into[node+"."+module] = neitherOne{node: node, module: module, holding: h}
}
}
// startClears adds, for the first pass after a start, every node of m that is not on neither list after the pass: a neither
// condition a previous process raised for it is cleared. That covers a node with a want and one with none that is not on neither
// list, such as a held module (issue 1945: the hold is said on its own, so a neither raised for it before the restart would stay up
// for ever) or a move waiting for its record ask. Clearing a condition that is not open is quiet at the router.
func startClears(into map[string]neitherOne, m assign.Module, out assign.Outcome) {
for _, node := range m.Nodes {
if _, on := out.Neither[node]; !on {
into[node+"."+m.Name] = neitherOne{node: node, module: m.Name}
}
}
}
// sayNeither raises, once, the condition of every module on neither list in this pass and writes it to the journal, and clears,
// once, the condition of each one said before that has a want again (issue 1945). A say that fails is made again at the next pass.
// stale is, on the first pass after a start, the modules whose condition a previous process may have raised and that have a want:
// each is cleared once.
func (p *passer) sayNeither(now, stale map[string]neitherOne) {
if p.neither == nil {
p.neither = map[string]neitherOne{}
for key, one := range stale {
if _, on := now[key]; !on {
p.neither[key] = one
}
}
}
for key, one := range now {
if _, said := p.neither[key]; said {
continue
}
c := assign.NeitherCondition(one.node, one.module, one.holding, true)
if p.say != nil {
if err := p.say(c); err != nil {
p.logf("the condition of %s on neither list on %s could not be said, and is said at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; rule %d holds its first want back: %s",
one.module, one.node, one.holding.Rule, one.holding.Words)
p.neither[key] = one
}
for key, one := range p.neither {
if _, still := now[key]; still {
continue
}
if p.say != nil {
if err := p.say(assign.NeitherCondition(one.node, one.module, assign.Holding{}, false)); err != nil {
p.logf("the condition of %s on neither list on %s could not be cleared, and is cleared at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s on %s has a want again", one.module, one.node)
delete(p.neither, key)
}
}
// compositions writes the composition check's verdict of every group the indexer holds a candidate for, under the
// checker's key of that check: waits, with why, while the indexer holds what it composes from only in part.
func (p *passer) compositions(ctx context.Context) {
if p.sdk == "" {
p.sdk = sdkversion.Version()
}
written := p.w.read(delivery.AssignerCompositionsBucket)
trunk := p.w.trunk()
var mesh *assign.Mesh
seen := map[string]bool{}
if p.asks != nil {
defer p.asks.sweep(seen)
}
for key, value := range p.w.read(delivery.CandidatesBucket) {
c, err := delivery.DecodeCandidate(value)
if err != nil {
p.logf("the candidate %s cannot be read, so it is not composed: %v", key, err)
continue
}
out, err := delivery.CompositionKey(c.Group, c.HeadsHash)
if err != nil {
p.logf("the candidate %s names no composition key: %v", key, err)
continue
}
began := time.Now()
ch, waits := p.w.change(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]), c, trunk)
var v delivery.CompositionVerdict
if waits != "" {
v = delivery.CompositionVerdict{Group: c.Group, Heads: c.HeadsHash, State: delivery.CompositionWaits, Words: waits}
} else {
if mesh == nil {
m := p.w.mesh(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]))
mesh = &m
}
if ch.SDKHead != "" && p.asks != nil {
// A group with a mesh-sdk member is composed by the composer built at its head (ADR 0482 clauses 3 to 5).
v = p.composeAtHead(ctx, *mesh, ch, seen, began)
} else {
v = assign.Compose(*mesh, ch, p.sdk, time.Since(began))
}
}
if before, err := delivery.DecodeCompositionVerdict(written[out]); err == nil {
before.Took, before.V = v.Took, 0
if delivery.SameCompositionVerdict(before, v) {
continue
}
}
p.put(ctx, delivery.AssignerCompositionsBucket, out, func() ([]byte, error) { return delivery.EncodeCompositionVerdict(v) })
}
}
// put writes one key of the assigner's own bucket; a write that fails is said and made again at the next pass.
func (p *passer) put(ctx context.Context, bucket, key string, encode func() ([]byte, error)) {
value, err := encode()
if err != nil {
p.logf("%s %s is not written: %v", bucket, key, err)
return
}
if _, err := p.kvs[bucket].Put(ctx, key, value); err != nil && !errors.Is(err, context.Canceled) {
p.logf("%s %s could not be written: %v", bucket, key, err)
return
}
p.w.put(bucket, key, value)
}
// holdAsk writes, once, the ask about a held fingerprint (ADR 0528 clause 6, ADR 0437 clause 10): the first ask that needs
// no proof of who answered, which the view may answer and the teller fills. In release 5 an answer changes nothing on a
// node (wants are only compared, ADR 0429 §7), and the options say so.
func (p *passer) holdAsk(ctx context.Context, module, fp string, fresh *delivery.Held) {
key, err := delivery.HoldAskKey(module, fp)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
// The asking is the hold's time in seconds: a hold of the same fingerprint written again after a lift is another asking, under
// an id of its own, also after a restart, since the router drops a second ask under an id it keeps (issue 1906).
now := p.clock()
asking := int(now.Unix())
h := p.heldHold(module, fp, fresh)
a := assign.HoldAsk(key, module, fp, h.Node, h.Words, asking, now)
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// lift takes away a hold the assigner lifted by itself, since every node that runs the module runs the held fingerprint healthy
// (issue 1906): the held record and its ask (the teller then takes the ask back from the operator), and the held condition is
// cleared saying why, once. A hold of the same fingerprint written later is said and asked again.
func (p *passer) lift(ctx context.Context, module, fp string) {
held, err := delivery.HeldKey(module, fp)
if err != nil {
return
}
ask, err := delivery.HoldAskKey(module, fp)
if err != nil {
return
}
c := assign.LiftedCondition(module, fp)
if !p.said[c.Key+".lifted"] {
if p.say != nil {
if err := p.say(c); err != nil {
// The hold stays written, so the next pass lifts it again and says so then.
p.logf("the condition of the lifted hold of %s could not be cleared, and is cleared at the next pass: %v", module, err)
return
}
}
p.logf("%s", assign.LiftedWords(module, fp))
p.said[c.Key+".lifted"] = true
delete(p.said, c.Key+".raised") // a hold written again is said again
}
p.remove(ctx, delivery.AssignerWantsBucket, held)
p.remove(ctx, delivery.AssignerAsksBucket, ask)
}
// remove deletes one key of the assigner's own bucket when it holds it; a delete that fails is said and made again at the next
// pass.
func (p *passer) remove(ctx context.Context, bucket, key string) {
if !p.w.has(bucket, key) {
return
}
if err := p.kvs[bucket].Delete(ctx, key); err != nil && !errors.Is(err, context.Canceled) {
p.logf("%s %s could not be taken away: %v", bucket, key, err)
return
}
p.w.put(bucket, key, nil)
}
// heldHold is the hold of fp as written: the node whose health check failed and its words.
func (p *passer) heldHold(module, fp string, fresh *delivery.Held) delivery.Held {
if fresh != nil {
return *fresh
}
key, err := delivery.HeldKey(module, fp)
if err != nil {
return delivery.Held{}
}
h, _ := delivery.DecodeHeld(p.w.read(delivery.AssignerWantsBucket)[key])
return h
}
// heldCondition raises, once, the condition for a fingerprint the assigner holds (ADR 0429 §6, ADR 0539 §19): the module, the
// node whose health check failed and the check's words. A module holding data needs the operator's answer to its restore ask.
func (p *passer) heldCondition(ctx context.Context, module, fp string, fresh *delivery.Held, data bool) {
h := p.heldHold(module, fp, fresh)
c := assign.HeldCondition(module, fp, h.Node, h.Words, true, data)
if p.say == nil || p.said[c.Key+".raised"] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the held %s could not be said, and is said at the next pass: %v", module, err)
return
}
p.said[c.Key+".raised"] = true
delete(p.said, c.Key+".lifted") // a hold raised again and lifted again is said lifted again
delete(p.said, c.Key+".cleared") // and released again is cleared again
}
// restoreAsk writes, once, the ask about a module holding data whose held fingerprint failed its health check on node.
func (p *passer) restoreAsk(ctx context.Context, module, node, fp string) {
key, err := delivery.RestoreAskKey(node, module, fp)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
words := p.heldHold(module, fp, nil).Words
a := assign.RestoreAsk(key, node, module, fp, words, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// answeredWith says whether the teller wrote an answer to the ask at key as it stands now and the option chosen is option. The
// answer is the operator's, given in the view or on the phone, and stays under its key (ADR 0539 §14, §16, §19); it counts only
// for the asking it answered (its warrant names that asking's id), so an answer to an ask taken away, or to an earlier hold of
// the same fingerprint, does not carry over to a later one (issue 1906).
func (p *passer) answeredWith(key, option string) bool {
a, ok := p.w.answer(key)
if !ok || a.Option != option {
return false
}
ask, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return false
}
id, _, _ := strings.Cut(a.Warrant, "@")
return id == ask.Ask.ID
}
// clearHeld clears, once, the condition of a hold the operator released.
func (p *passer) clearHeld(module, fp string) {
c := assign.HeldCondition(module, fp, "", "", false, false)
if p.say == nil || p.said[c.Key+".cleared"] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the released %s could not be cleared, and is cleared at the next pass: %v", module, err)
return
}
p.said[c.Key+".cleared"] = true
}
// recordAsk writes, once, the ask about a move of a module of policy record that waits for the operator's yes.
func (p *passer) recordAsk(ctx context.Context, module, fp string, need assign.RecordNeed) {
key, err := delivery.RecordAskKey(need.Node, module, need.Digest)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
a := assign.RecordAsk(key, need.Node, module, fp, need, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// backups sees to the modules held on a node because the backup before an apply failed (the have's phase held, ADR 0539 §16):
// it asks whether to back up again or go on without a backup, raises the condition naming node, module and the backup's words,
// and writes the answer into the node's want for the module, which the node agent reads from it and from nothing else.
//
// The ask is bound to the composition (it names the digest, and a new revision at the same fingerprint is asked again under a
// new number) and an answer counts only if the router's warrant names the id of the ask now at the key (the warrant's first part,
// before the `@`). Each answer is another want value (the asking's number is in the want's words), so a backup that fails again
// after "back up again" and is answered "again" again changes the key and the node retries. That the backup failed again is seen
// in a new have value after the answer was written.
func (p *passer) backups(ctx context.Context) {
wanted := p.w.wantsOf()
for key, value := range p.w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
continue
}
have, err := delivery.DecodeHave(value)
if err != nil || have.Phase != delivery.PhaseHeld {
continue
}
want, ok := wanted[module][node]
if !ok || want.Removed {
continue
}
akey, err := delivery.BackupAskKey(node, module, want.FP)
if err != nil {
continue
}
current, there := delivery.DecodeAssignerAsk(akey, p.w.read(delivery.AssignerAsksBucket)[akey])
if there != nil || assign.BackupAskDigest(current) != want.Composed {
after := -1
if there == nil {
after = current.Asking
}
p.backupAsk(ctx, akey, node, module, want, have.Detail, assign.NextBackupAsking(after))
continue
}
answer, answered := p.w.answer(akey)
if router, _, _ := strings.Cut(answer.Warrant, "@"); !answered || router != current.Ask.ID ||
(answer.Option != delivery.BackupAgain && answer.Option != delivery.BackupSkip) {
continue // not answered yet: the ask stands
}
words := assign.BackupAnswerWords(answer.Option, current.Asking)
haveChanges := p.w.changed(delivery.HaveBucket, key)
if want.Backup != answer.Option || want.Words != words {
want.Backup, want.Words = answer.Option, words
if wkey, err := delivery.WantKey(node, module); err == nil {
w := want
p.put(ctx, delivery.AssignerWantsBucket, wkey, func() ([]byte, error) { return delivery.EncodeWant(w) })
p.answeredAt[current.Ask.ID] = haveChanges
}
continue
}
if answer.Option != delivery.BackupAgain {
continue
}
// The answer is in the want. The node retried and the backup failed again when the have took another value since.
if seen, known := p.answeredAt[current.Ask.ID]; !known {
p.answeredAt[current.Ask.ID] = haveChanges // after a restart: wait for the next failure
} else if haveChanges > seen {
p.backupAsk(ctx, akey, node, module, want, have.Detail, assign.NextBackupAsking(current.Asking))
}
}
}
// backupAsk writes the backup ask at its asking and raises the condition of the failed backup, once.
func (p *passer) backupAsk(ctx context.Context, key, node, module string, want delivery.Want, words string, asking int) {
a := assign.BackupAsk(key, node, module, want.FP, want.Composed, words, asking, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
c := assign.BackupCondition(node, module, want.FP, words, true)
if p.say == nil || p.said[c.Key] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the failed backup of %s on %s could not be said: %v", module, node, err)
return
}
p.said[c.Key] = true
}
@@ -0,0 +1,238 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// memKV is the part of a bucket a pass writes: puts and deletes.
type memKV struct {
jetstream.KeyValue
values map[string][]byte
}
func (m *memKV) Put(_ context.Context, key string, value []byte) (uint64, error) {
m.values[key] = value
return uint64(len(m.values)), nil
}
func (m *memKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.values, key)
return nil
}
// newPasser is a pass on a fresh world, as after a start, writing to the buckets given, at the clock given.
func newPasser(kvs buckets, at time.Time) *passer {
return &passer{kvs: kvs, w: newWorld(), logf: func(string, ...any) {}, said: map[string]bool{}, answeredAt: map[string]uint64{},
now: func() time.Time { return at }}
}
var fpTest = delivery.SourcePrefix2 + strings.Repeat("b", 64)
// 1906/H2: given a hold lifted by itself, when the assigner is started again and the same fingerprint is held again, then the
// hold ask is written under another asking, with an id of its own, so the teller raises it (the router drops a second ask under
// an id it keeps).
func TestAHoldAskedAgainAfterALiftAndARestartHasANewID(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
kvs := buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}
key, _ := delivery.HoldAskKey("web", fpTest)
held := &delivery.Held{Node: "ace", Words: "health failed"}
first := newPasser(kvs, time.Date(2026, 10, 25, 1, 0, 0, 0, time.UTC))
first.holdAsk(context.Background(), "web", fpTest, held)
a1, err := delivery.DecodeAssignerAsk(key, asksKV.values[key])
if err != nil {
t.Fatal(err)
}
first.lift(context.Background(), "web", fpTest)
if _, there := asksKV.values[key]; there {
t.Fatal("the lift left the hold ask")
}
again := newPasser(kvs, time.Date(2026, 10, 25, 2, 0, 0, 0, time.UTC)) // started again: it remembers nothing
again.holdAsk(context.Background(), "web", fpTest, held)
a2, err := delivery.DecodeAssignerAsk(key, asksKV.values[key])
if err != nil {
t.Fatal(err)
}
if a2.Ask.ID == a1.Ask.ID {
t.Errorf("the ask written again after a lift and a restart has the id %s of the first", a1.Ask.ID)
}
}
// 1906/H4: an answer counts only for the asking it answered: an answer to an earlier hold of the same fingerprint, or to an ask
// taken away, is not taken for the hold that stands now.
func TestAnAnswerCountsOnlyForTheAskingItAnswered(t *testing.T) {
key, _ := delivery.HoldAskKey("web", fpTest)
p := newPasser(buckets{}, time.Now())
put := func(asking int) string {
a := assign.HoldAsk(key, "web", fpTest, "ace", "health failed", asking, time.Now())
v, err := delivery.EncodeAssignerAsk(key, a)
if err != nil {
t.Fatal(err)
}
p.w.put(delivery.AssignerAsksBucket, key, v)
return a.Ask.ID
}
first := put(100)
v, _ := delivery.EncodeAnswer(delivery.Answer{Option: "release", Warrant: first + "@2026-10-25T01:00:00Z", Words: "the operator"})
p.w.put(delivery.AnswersBucket, key, v)
if !p.answeredWith(key, "release") {
t.Fatal("the answer to the ask that stands is not taken")
}
if p.answeredWith(key, "keep") {
t.Error("an answer was taken for an option not chosen")
}
put(200) // the same fingerprint held again, after a lift
if p.answeredWith(key, "release") {
t.Error("the answer to an earlier hold carried over to a later one")
}
p.w.put(delivery.AssignerAsksBucket, key, nil) // the ask taken away
if p.answeredWith(key, "release") {
t.Error("an answer to an ask taken away was taken")
}
}
// 1945/AC21: given a module on neither list on a node, when passes follow, then the assigner raises its condition once, says it
// again after a say that failed, and clears it once when the module has a want again; a module on neither list is also written to
// the journal, so it is loud with no operator channel.
func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
p := newPasser(buckets{}, time.Date(2026, 10, 25, 15, 0, 0, 0, time.UTC))
var said []asks.Condition
failing := true
p.say = func(c asks.Condition) error {
if failing {
failing = false
return errors.New("the channel is away")
}
said = append(said, c)
return nil
}
var logged []string
p.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
one := map[string]neitherOne{"g14.screen-lock": {node: "g14", module: "screen-lock",
holding: assign.Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}}}
p.sayNeither(one, nil) // the say fails
p.sayNeither(one, nil)
p.sayNeither(one, nil)
if len(said) != 1 || said[0].State != asks.StateRaised || said[0].Subject.Machine != "g14" || said[0].Subject.ID != "screen-lock" {
t.Fatalf("said = %+v, want one raised condition of screen-lock on g14", said)
}
if !strings.Contains(strings.Join(logged, "\n"), "screen-lock") {
t.Errorf("the journal does not name the module on neither list: %q", logged)
}
p.sayNeither(map[string]neitherOne{}, nil)
p.sayNeither(map[string]neitherOne{}, nil)
if len(said) != 2 || said[1].State != asks.StateCleared || said[1].Key != said[0].Key {
t.Errorf("said = %+v, want the condition cleared once under its key", said)
}
}
// 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module.
func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
got := map[string]neitherOne{}
gatherNeither(got, "screen-lock", assign.Outcome{Neither: map[string]assign.Holding{"g14": {Rule: delivery.RuleGrant, Words: "w"}}})
gatherNeither(got, "rofi", assign.Outcome{})
if len(got) != 1 || got["g14.screen-lock"].node != "g14" || got["g14.screen-lock"].module != "screen-lock" || got["g14.screen-lock"].holding.Rule != delivery.RuleGrant {
t.Errorf("gathered = %+v", got)
}
}
// A neither or held condition a previous process raised is cleared by the first pass after a start: a fresh pass remembers
// nothing it said, so it clears, once, the neither condition of every assigned module not on neither list and the held condition of
// every module that is not held. Clearing a condition that is not open is quiet at the router.
func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerWantsBucket: wantsKV, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
var said []asks.Condition
p.say = func(c asks.Condition) error { said = append(said, c); return nil }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
p.w.put(delivery.ModulesBucket, "web", mod)
p.w.put(provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"b-1"}`))
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
p.w.put(delivery.PresenceBucket, "g14", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpTest, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
p.w.put(delivery.HaveBucket, "g14.web", have)
key, _ := delivery.WantKey("g14", "web")
want, _ := delivery.EncodeWant(delivery.Want{FP: fpTest, Composed: "sha256:" + strings.Repeat("c", 64)})
p.w.put(delivery.AssignerWantsBucket, key, want)
p.wants(context.Background())
cleared := map[string]bool{}
for _, c := range said {
if c.State != asks.StateCleared {
t.Errorf("the first pass raised %+v", c)
}
cleared[c.Key] = true
}
neither := assign.NeitherCondition("g14", "web", assign.Holding{}, false).Key
held := assign.HeldCondition("web", fpTest, "", "", false, false).Key
if !cleared[neither] || !cleared[held] {
t.Errorf("cleared = %v, want %s and %s", cleared, neither, held)
}
n := len(said)
p.wants(context.Background())
if len(said) != n {
t.Errorf("a second pass said again: %+v", said[n:])
}
}
// 1945/AC24: given a module held at F under rule 2 on a node that runs it with no want, when the assigner passes, then it raises no
// neither condition for it, a neither condition this process raised for it before is cleared, and so is one a previous process
// raised (the first pass after a start), each once.
func TestAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) {
setup := func() (*passer, *[]asks.Condition) {
p := newPasser(buckets{delivery.AssignerWantsBucket: &memKV{values: map[string][]byte{}}, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
said := &[]asks.Condition{}
p.say = func(c asks.Condition) error { *said = append(*said, c); return nil }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/resolved", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
p.w.put(delivery.ModulesBucket, "systemd-resolved", mod)
p.w.put(provider.AssignmentsBucket, "shanks.systemd-resolved", []byte(`{"node":"shanks","module":"systemd-resolved","build":"b-1"}`))
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
p.w.put(delivery.PresenceBucket, "shanks", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpTest, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed})
p.w.put(delivery.HaveBucket, "shanks.systemd-resolved", have)
hkey, _ := delivery.HeldKey("systemd-resolved", fpTest)
held, _ := delivery.EncodeHeld(delivery.Held{Node: "shanks", Words: "health failed"})
p.w.put(delivery.AssignerWantsBucket, hkey, held)
return p, said
}
key := assign.NeitherCondition("shanks", "systemd-resolved", assign.Holding{}, false).Key
neithers := func(said []asks.Condition) (raised, cleared int) {
for _, c := range said {
if c.Key != key {
continue
}
if c.State == asks.StateRaised {
raised++
} else {
cleared++
}
}
return
}
// a standing neither this process raised
p, said := setup()
p.neither = map[string]neitherOne{"shanks.systemd-resolved": {node: "shanks", module: "systemd-resolved", holding: assign.Holding{Rule: delivery.RuleNotHeld}}}
p.wants(context.Background())
p.wants(context.Background())
if r, c := neithers(*said); r != 0 || c != 1 {
t.Errorf("standing in this process: %d raised, %d cleared, want 0 and 1: %+v", r, c, *said)
}
// one a previous process raised: the first pass after a start clears it
p, said = setup()
p.wants(context.Background())
p.wants(context.Background())
if r, c := neithers(*said); r != 0 || c != 1 {
t.Errorf("after a start: %d raised, %d cleared, want 0 and 1: %+v", r, c, *said)
}
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,437 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// noModuleWords is the indexer's reason for a module of a candidate whose directory the merge preview holds no manifest
// for (mesh-sdk delivery.CandidateNoneNoModule, which the indexer writes): the change removes the module.
const noModuleWords = delivery.CandidateNoneNoModule
// assignment holds only the fields of the controller's Assignment the assigner reads (the view's own, mesh-delivery-view
// internal/view/view.go): an assignment stands while it has not ended and has no state.
type assignment struct {
Node string `json:"node"`
Module string `json:"module"`
Build string `json:"build"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
func (a assignment) open() bool { return a.Ended.IsZero() && a.State == "" }
// world is what the assigner watched, as it stands now: every value by bucket and key, and the manifests read by
// digest. The watches write it; a pass reads it.
type world struct {
mu sync.Mutex
values map[string]map[string][]byte
manifests map[string]parsed
// healthy is, by have key, every fingerprint the node ran healthy, oldest first, each once: what a set-back goes to.
healthy map[string][]string
// changes counts, by bucket and key, the values this process has seen the key take: a value written again is another.
changes map[string]uint64
}
// parsed is a manifest read by digest: its module and its manifest, or why it cannot be read.
type parsed struct {
module string
manifest catalogue.Manifest
err string
}
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{}}
}
// put takes a change of a watched bucket in: a value, or nil for a key deleted or purged.
func (w *world) put(bucket, key string, value []byte) {
w.mu.Lock()
defer w.mu.Unlock()
if w.values[bucket] == nil {
w.values[bucket] = map[string][]byte{}
}
if value == nil {
delete(w.values[bucket], key)
return
}
w.values[bucket][key] = value
w.changes[bucket+"|"+key]++
if bucket == delivery.HaveBucket {
if h, err := delivery.DecodeHave(value); err == nil && h.FP != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
list := slices.DeleteFunc(w.healthy[key], func(fp string) bool { return fp == h.FP })
w.healthy[key] = append(list, h.FP)
}
}
}
// lastHealthy is the newest fingerprint other than fp the node ran the module healthy at, or empty.
func (w *world) lastHealthy(node, module, fp string) string {
key, err := delivery.HaveKey(node, module)
if err != nil {
return ""
}
w.mu.Lock()
defer w.mu.Unlock()
list := w.healthy[key]
for i := len(list) - 1; i >= 0; i-- {
if list[i] != fp {
return list[i]
}
}
return ""
}
// read is a copy of what a bucket holds now.
func (w *world) read(bucket string) map[string][]byte {
w.mu.Lock()
defer w.mu.Unlock()
out := make(map[string][]byte, len(w.values[bucket]))
for k, v := range w.values[bucket] {
out[k] = v
}
return out
}
// getter reads one key's value; errNotThere when the key does not exist.
type getter func(ctx context.Context, key string) ([]byte, error)
var errNotThere = errors.New("no such key")
// kvGetter reads keys of a bucket.
func kvGetter(kv jetstream.KeyValue) getter {
return func(ctx context.Context, key string) ([]byte, error) {
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, errNotThere
}
if err != nil {
return nil, err
}
return e.Value(), nil
}
}
// manifest reads the manifest of a digest from the indexer's bucket once and keeps it: a digest's manifest never changes.
func (w *world) manifest(ctx context.Context, get getter, digest string) parsed {
w.mu.Lock()
p, ok := w.manifests[digest]
w.mu.Unlock()
if ok {
return p
}
key, err := delivery.ManifestsKey(digest)
if err != nil {
return parsed{err: err.Error()}
}
value, err := get(ctx, key)
if errors.Is(err, errNotThere) {
return parsed{err: "the indexer holds no manifest of digest " + digest + " yet"}
}
if err != nil {
return parsed{err: err.Error()}
}
m, err := delivery.DecodeManifest(value)
if err != nil {
p = parsed{err: err.Error()}
} else if man, err := catalogue.ParseManifestForInputs([]byte(m.JSON)); err != nil {
// Read as the indexer reads it (issue 1751): a value falling back to a predefined key is judged by the merge gate
// in the catalogue, against the predefined.json at its root that this process never holds (ADR 0348 §7).
p = parsed{module: m.Module, err: err.Error()}
} else {
p = parsed{module: m.Module, manifest: man}
}
w.mu.Lock()
w.manifests[digest] = p
w.mu.Unlock()
return p
}
// assigned is each node's modules with an open assignment, read from the controller's assignments, in name order.
func (w *world) assigned() (map[string][]string, map[string]assignment) {
out, all := map[string][]string{}, map[string]assignment{}
for _, value := range w.read(provider.AssignmentsBucket) {
var a assignment
if json.Unmarshal(value, &a) != nil || a.Node == "" || a.Module == "" || !a.open() {
continue
}
out[a.Node] = append(out[a.Node], a.Module)
all[a.Node+"."+a.Module] = a
}
for _, mods := range out {
sort.Strings(mods)
}
return out, all
}
// trunk is every module the indexer holds at the trunk, with its value.
func (w *world) trunk() map[string]delivery.Module {
out := map[string]delivery.Module{}
for key, value := range w.read(delivery.ModulesBucket) {
name, err := delivery.ParseModulesKey(key)
if err != nil {
continue
}
if m, err := delivery.DecodeModule(value); err == nil {
out[name] = m
}
}
return out
}
// mesh is the mesh as the trunk holds it, for the composition check.
func (w *world) mesh(ctx context.Context, manifests getter) assign.Mesh {
m := assign.Mesh{Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{}, Digests: map[string]string{}}
m.Assigned, _ = w.assigned()
for name, mod := range w.trunk() {
if mod.Manifest == "" {
continue
}
m.Digests[name] = mod.Manifest
p := w.manifest(ctx, manifests, mod.Manifest)
if p.err != "" {
m.Unreadable[name] = p.err
continue
}
m.Manifests[name] = p.manifest
}
return m
}
// change is a candidate's change for the composition check, or why the composition waits for it.
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}}
definedIn := map[string]map[string]bool{}
for _, p := range c.Pulls {
if p.Repository == assign.SDKRepository {
ch.SDKHead = p.Head.Head
}
ch.Members[p.Repository] = p.Head.Head
for _, name := range p.Modules {
fp, ok := p.Fps[name]
if !ok || fp.Manifest == "" {
if ok && fp.None2 == noModuleWords {
ch.Removed = append(ch.Removed, name)
continue
}
return ch, fmt.Sprintf("the indexer holds no manifest of %s for %s#%d yet", name, p.Repository, p.Number)
}
if fp.FP2 != "" && fp.FP2 != trunk[name].FP2 {
ch.Rebuilt++
}
got := w.manifest(ctx, manifests, fp.Manifest)
if strings.HasPrefix(got.err, "the indexer holds no manifest") {
return ch, got.err
}
ch.Digests[name] = fp.Manifest
if got.err != "" {
ch.Unreadable[name] = got.err
continue
}
ch.Manifests[got.module] = got.manifest
if definedIn[got.module] == nil {
definedIn[got.module] = map[string]bool{}
}
definedIn[got.module][p.Repository] = true
if t, ok := trunk[got.module]; ok && t.Repository != "" && t.Repository != p.Repository {
definedIn[got.module][t.Repository] = true
}
}
}
for name, repos := range definedIn {
if len(repos) > 1 {
for r := range repos {
ch.Twice[name] = append(ch.Twice[name], r)
}
sort.Strings(ch.Twice[name])
}
}
return ch, ""
}
// wantsOf is the wants the assigner wrote, by module then node.
func (w *world) wantsOf() map[string]map[string]delivery.Want {
out := map[string]map[string]delivery.Want{}
for key, value := range w.read(delivery.AssignerWantsBucket) {
node, module, err := delivery.ParseWantKey(key)
if err != nil {
continue
}
want, err := delivery.DecodeWant(value)
if err != nil {
continue
}
if out[module] == nil {
out[module] = map[string]delivery.Want{}
}
out[module][node] = want
}
return out
}
// nodes is what the six rules read of the nodes: presence, haves, and every provision's providers with their nodes.
func (w *world) nodes(manifests map[string]catalogue.Manifest, assigned map[string][]string) assign.Nodes {
n := assign.Nodes{Online: map[string]bool{}, Haves: map[string]map[string]delivery.Have{}, Providers: map[string]map[string][]string{}}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
continue
}
if p, err := delivery.DecodePresence(value); err == nil {
n.Online[node] = p.Online
}
}
for key, value := range w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
continue
}
if h, err := delivery.DecodeHave(value); err == nil {
if n.Haves[node] == nil {
n.Haves[node] = map[string]delivery.Have{}
}
n.Haves[node][module] = h
}
}
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
nodesOf[m] = append(nodesOf[m], node)
}
}
for name, man := range manifests {
for _, o := range man.Provides {
if n.Providers[o.Name] == nil {
n.Providers[o.Name] = map[string][]string{}
}
n.Providers[o.Name][name] = nodesOf[name]
}
}
return n
}
// value is what a bucket holds at a key now, nil when nothing.
func (w *world) value(bucket, key string) []byte {
w.mu.Lock()
defer w.mu.Unlock()
return w.values[bucket][key]
}
// has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool {
w.mu.Lock()
defer w.mu.Unlock()
_, ok := w.values[bucket][key]
return ok
}
// built says whether the builder's build of a module at a fingerprint is done.
func (w *world) built(module, fp string) bool {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return false
}
w.mu.Lock()
value, ok := w.values[delivery.BuilderBuildsBucket][key]
w.mu.Unlock()
if !ok {
return false
}
b, err := delivery.DecodeBuilderBuild(value)
return err == nil && b.State == delivery.BuildDone
}
// composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it
// (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read.
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
key, err := delivery.ComposedKey(node, module, fp)
if err != nil {
return delivery.ComposedDigest{}, false
}
w.mu.Lock()
value, ok := w.values[delivery.ControllerComposedBucket][key]
w.mu.Unlock()
if !ok {
return delivery.ComposedDigest{}, false
}
d, err := delivery.DecodeComposedDigest(value)
return d, err == nil
}
// nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`).
func (w *world) nodeComposed() map[string]delivery.ComposedDigest {
out := map[string]delivery.ComposedDigest{}
for key, value := range w.read(delivery.ControllerComposedBucket) {
node, _, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || !isNode {
continue
}
if d, err := delivery.DecodeComposedDigest(value); err == nil {
out[node] = d
}
}
return out
}
// answer is the teller's answer to the ask at key (mesh-delivery-teller_answers, the same key), and whether there is one.
func (w *world) answer(key string) (delivery.Answer, bool) {
w.mu.Lock()
value, ok := w.values[delivery.AnswersBucket][key]
w.mu.Unlock()
if !ok {
return delivery.Answer{}, false
}
a, err := delivery.DecodeAnswer(value)
return a, err == nil
}
// changed is how many values the key has taken in this process (see world.changes).
func (w *world) changed(bucket, key string) uint64 {
w.mu.Lock()
defer w.mu.Unlock()
return w.changes[bucket+"|"+key]
}
// runningCommits is, of every open assignment of module, the commit of the build it names, read from the controller's
// build record (`mesh-controller_builds` `<module>.build.<id>`), distinct and sorted: the commits a reader of the
// manifests runs at (novox/hq ADR 0482 clause 5, issue 1623). A build whose record names no full commit is left out.
func (w *world) runningCommits(ctx context.Context, get getter, module string) []string {
_, all := w.assigned()
seen := map[string]bool{}
for _, a := range all {
if a.Module != module || a.Build == "" {
continue
}
value, err := get(ctx, delivery.BuildKey(module, a.Build))
if err != nil {
continue
}
var rec delivery.BuildRecord
if json.Unmarshal(value, &rec) != nil || len(rec.Commit) != 40 {
continue
}
seen[rec.Commit] = true
}
out := make([]string, 0, len(seen))
for c := range seen {
out = append(out, c)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,127 @@
package main
import (
"context"
"os"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// fakeManifests is the indexer's manifests bucket in memory.
type fakeManifests map[string][]byte
func (f fakeManifests) add(t *testing.T, module, body string) string {
t.Helper()
digest := delivery.ManifestDigest([]byte(body))
value, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: body})
if err != nil {
t.Fatal(err)
}
f[digest] = value
return digest
}
func (f fakeManifests) get(_ context.Context, key string) ([]byte, error) {
if v, ok := f[key]; ok {
return v, nil
}
return nil, errNotThere
}
func fp(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
func pull(repo string, number int, modules map[string]delivery.CandidateModule) delivery.CandidatePull {
p := delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: strings.Repeat("3", 40)}, Fps: modules}
for m := range modules {
p.Modules = append(p.Modules, m)
}
slices.Sort(p.Modules)
return p
}
// 1297/B3b: a candidate's change reads each module's manifest by its digest, counts the modules rebuilt, takes a
// module with no manifest in the preview as removed, a mesh-sdk member's head, and a module the trunk holds from
// another repository as defined twice.
func TestTheChangeOfACandidate(t *testing.T) {
f := fakeManifests{}
web := f.add(t, "web", `{"module":"web","version":"2"}`)
mail := f.add(t, "mail", `{"module":"mail","version":"1"}`)
trunk := map[string]delivery.Module{"web": {FP2: fp('a'), Repository: "novox/web"}, "mail": {FP2: fp('c'), Repository: "novox/mail"}}
c := delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: web}, "old": {None2: noModuleWords}}),
pull("novox/other", 2, map[string]delivery.CandidateModule{"mail": {FP2: fp('c'), Manifest: mail}}),
pull(assign.SDKRepository, 3, nil),
}}
ch, waits := newWorld().change(context.Background(), f.get, c, trunk)
if waits != "" {
t.Fatalf("the change waits: %s", waits)
}
if ch.Manifests["web"].Version != "2" || ch.Rebuilt != 1 || !slices.Equal(ch.Removed, []string{"old"}) || ch.SDKHead != strings.Repeat("3", 40) {
t.Errorf("change = %+v", ch)
}
if !slices.Equal(ch.Twice["mail"], []string{"novox/mail", "novox/other"}) || len(ch.Twice) != 1 {
t.Errorf("twice = %v, want mail in novox/mail and novox/other", ch.Twice)
}
}
// 1297/B3b: a candidate whose module has no manifest digest yet, or whose manifest the indexer does not hold yet, waits,
// saying for what; an unreadable manifest is the change's, not a wait.
func TestAChangeWaitsForItsManifests(t *testing.T) {
f := fakeManifests{}
bad := f.add(t, "web", `{"module":"web","version":"2","scope":"x"}`)
for _, c := range []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {None2: "no preview yet"}}),
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: strings.Repeat("9", 64)}}),
} {
_, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{c}}, nil)
if waits == "" {
t.Errorf("a change without its manifest %+v does not wait", c.Fps)
}
}
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: bad}})}}, nil)
if waits != "" || ch.Unreadable["web"] == "" {
t.Errorf("an unreadable manifest: waits %q, change %+v", waits, ch)
}
}
// A manifest of the change whose config value falls back to a predefined key (novox/hq ADR 0348 §7) is read, not
// unreadable for the catalogue's predefined.json the assigner never holds: the merge gate judges the fallback in the
// catalogue, and the indexer reads it the same way (catalogue.ParseManifestForInputs, issue 1751). A group whose
// mesh-catalog member touched pacman and xterm failed its composition on this (group check 18df81380db3). The manifest is
// xterm's module.json from mesh-catalog at 835bf22b.
func TestAManifestFallingBackToAPredefinedKeyIsReadable(t *testing.T) {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"predefined"`) {
t.Fatal("the fixture names no predefined key")
}
f := fakeManifests{}
xterm := f.add(t, "xterm", string(raw))
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/mesh-catalog", 1, map[string]delivery.CandidateModule{"xterm": {FP2: fp('b'), Manifest: xterm}})}}, nil)
if waits != "" || ch.Unreadable["xterm"] != "" || ch.Manifests["xterm"].Module != "xterm" {
t.Errorf("xterm: waits %q, unreadable %q, want read", waits, ch.Unreadable["xterm"])
}
}
// Review round 1 (items 3 and 4): the world counts the values a key takes, so a have written again after an answer is seen.
func TestTheWorldCountsTheValuesAKeyTakes(t *testing.T) {
w := newWorld()
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.mail", []byte(`{"v":1}`))
if got := w.changed(delivery.HaveBucket, "ace.web"); got != 2 {
t.Errorf("changed = %d, want 2", got)
}
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
}
+22
View File
@@ -0,0 +1,22 @@
module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27
View File
@@ -0,0 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c h1:pY2YpHSSsfDPBrOEiWTNSIPn0zdnAayRmtmmyQ3hCHY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,206 @@
package assign
import (
"fmt"
"regexp"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HoldAsk is the ask about a module's fingerprint the assigner holds, at its key (delivery.HoldAskKey): keep it held or
// release it. Releasing writes the held fingerprint's want again by the rules (ADR 0539 §19). node and words are the held
// record's: the machine whose health check failed and the hold's words (issue 1906). asking numbers the writing: 0 the first, a
// larger number an ask written again after a hold lifted by itself, so the router takes it as new.
func HoldAsk(key, module, fp, node, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp)),
Explanation: fmt.Sprintf("The health check of %s failed on %s at %s (%s), so the assigner holds that version and wants "+
"no node to move to it. It lifts the hold by itself once every node that runs %s runs it healthy; releasing it "+
"lets the wants move to it now.", module, node, short(fp), words, module),
Options: []asks.Option{
{ID: "keep", Label: "Keep it held", Does: "the version stays held", Level: asks.Acknowledge},
{ID: "release", Label: "Release it", Does: "the version is released and the wants move to it by the rules again", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the version stays held",
About: key,
}}
}
// The options of the restore ask (ADR 0537 §3, ADR 0539 §19), and what each does on the node.
const (
OptionRestore = "restore"
OptionLeaveStopped = "leave-stopped"
)
// RestoreAsk is the ask about a module holding data whose fingerprint failed its health check on a node (ADR 0429 §6): it is
// not set back by itself, since a set-back would lose the writes made since. words is the check's words. Restore sets the
// node back to the last healthy fingerprint and loses the newer writes; leaving it stopped keeps the data as it is.
func RestoreAsk(key, node, module, fp, words string, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s failed its health check on %s", module, node),
Explanation: fmt.Sprintf("%s holds data, and its health check failed on %s at %s: %s. Setting it back to the version that ran healthy "+
"would lose the writes made since; leaving it stopped keeps the data as it is.", module, node, short(fp), words),
Options: []asks.Option{
{ID: OptionRestore, Label: "Restore it", Does: "the node is set back to the last healthy version and the newer writes are lost", Level: asks.Acknowledge},
{ID: OptionLeaveStopped, Label: "Leave it stopped", Does: "the module stays as it is on the node and the data is kept", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module stays as it is on the node",
About: key,
}}
}
// HeldCondition is the condition the assigner raises when it holds a fingerprint of a module (ADR 0429 §6, ADR 0539 §19), or
// clears (raised false) when the hold is released. It names the module, the node whose health check failed and the check's
// words, under the assigner's own name (ADR 0437 clause 9), keyed by the held fingerprint. needsYou is true for a module holding
// data, whose restore ask waits for the operator.
func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.Condition {
key := delivery.AssignerModule + ".held." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
c := asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: fmt.Sprintf("%s is held at %s", module, short(fp)),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp))}
what := fmt.Sprintf("The health check of %s failed on %s at %s: %s. The assigner holds that version and wants no node to move to it.", module, node, short(fp), words)
if needsYou {
c.Needs = "answer whether to restore " + module + " on " + node
c.Explanation = asks.NeedsYou + c.Needs + ". " + what
} else {
c.Explanation = asks.NothingToDo + " " + what + " Nodes that wanted it are set back to the version that ran healthy."
}
return c
}
// NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
// node and module, and cleared (raised false) when the node has a want for it again.
func NeitherCondition(node, module string, h Holding, raised bool) asks.Condition {
key := delivery.AssignerModule + ".neither." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s has a want again, so its own composition is applied there.", module, node)}
}
headline := fmt.Sprintf("%s on %s is on no list", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner's rule %d holds its first want back: %s. The assigner writes the want once the "+
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
c := HeldCondition(module, fp, "", "", false, false)
c.Resolved = LiftedWords(module, fp) + "."
return c
}
// The options of the record ask (ADR 0539 §14) and of the backup ask (§16).
const (
OptionYes = "yes"
OptionNo = "no"
OptionBackupAgain = delivery.BackupAgain
OptionBackupSkip = delivery.BackupSkip
)
// RecordAsk is the ask about a move of a module of policy record on node to the composition with the digest, judged by the
// controller to recreate a container, to move data, or not to be judgeable (ADR 0539 §14). The want names the composition only
// after the answer yes. The way the move was judged is in the ask.
func RecordAsk(key, node, module, fp string, need RecordNeed, now time.Time) delivery.AssignerAsk {
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + need.Words,
}[need.Judgement]
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s waits for your yes on %s", module, node),
Explanation: fmt.Sprintf("%s is recorded: a person takes each of its moves. The next one on %s (version %s) %s. "+
"The node is told to apply it only after you say yes.", module, node, short(fp), what),
Options: []asks.Option{
{ID: OptionYes, Label: "Yes, move it", Does: "the node is told to apply the new composition of " + module, Level: asks.Acknowledge},
{ID: OptionNo, Label: "No, not now", Does: "the node keeps what it runs", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the node keeps what it runs",
About: key,
}}
}
// NextBackupAsking is the number of the asking that follows the one numbered after (a negative number: none yet). Numbers 0 and 1
// give one id (delivery.AskingID), so the first asking is 0 and the next 2.
func NextBackupAsking(after int) int {
if after < 0 {
return 0
}
return max(after+1, 2)
}
var backupDigest = regexp.MustCompile(`\(composition (sha256:[0-9a-f]{64})\)`)
// BackupAskDigest is the digest of the composition a backup ask was made about, from its explanation; empty for an ask that names
// none. An answer is taken for the composition of the ask it answers, and a new revision at the same fingerprint is asked again
// under a new number (ADR 0539 §16).
func BackupAskDigest(a delivery.AssignerAsk) string {
if m := backupDigest.FindStringSubmatch(a.Ask.Explanation); m != nil {
return m[1]
}
return ""
}
// BackupAnswerWords are the words of a want carrying the answer to the asking-th ask: each asking's answer is another want
// value, so a backup that fails again after "back up again" and is answered "again" again changes the key and the node retries.
func BackupAnswerWords(option string, asking int) string {
return fmt.Sprintf("the operator answered %q to the backup ask (asking %d)", option, asking+1)
}
// BackupAsk is the ask about a backup of a module on node that failed before an apply (ADR 0539 §16): back up again, or go on
// without a backup. digest is the composition about to be applied and asking the number of the asking, 0 the first; words are the
// backup's own words, from the node's have.
func BackupAsk(key, node, module, fp, digest, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("The backup of %s failed on %s", module, node),
Explanation: fmt.Sprintf("The backup before applying %s at %s on %s failed: %s. %s keeps running what it has until you answer. (composition %s)",
module, short(fp), node, words, module, digest),
Options: []asks.Option{
{ID: OptionBackupAgain, Label: "Back up again", Does: "the node tries the backup again before it applies", Level: asks.Acknowledge},
{ID: OptionBackupSkip, Label: "Go on without a backup", Does: "the node applies the new version without a backup", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module keeps running what it has",
About: key,
}}
}
// BackupCondition is the condition raised when a backup before an apply fails and the module is held on the node (ADR 0539 §16),
// naming the node, the module and the backup's words; cleared under the same key.
func BackupCondition(node, module, fp, words string, raised bool) asks.Condition {
key := delivery.AssignerModule + ".backup." + node + "." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
headline := fmt.Sprintf("The backup of %s failed on %s", module, node)
needs := "answer whether to back up again or go on without a backup"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: asks.NeedsYou + needs + ". The backup before applying " + module + " at " + short(fp) + " on " + node + " failed: " + words + "."}
}
@@ -0,0 +1,165 @@
package assign
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at
// its key, of a kind the view may answer.
func TestTheHoldAskIsValid(t *testing.T) {
key, err := delivery.HoldAskKey("mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := HoldAsk(key, "mail", fNew, "ace", "health failed, phase running: connection refused", 0, now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
// 1906/H3: the ask names the machine whose check failed and the hold's words.
if !strings.Contains(a.Ask.Explanation, "on ace") || !strings.Contains(a.Ask.Explanation, "connection refused") {
t.Errorf("the ask does not name the node and the hold's words: %q", a.Ask.Explanation)
}
// 1906/H2: an ask written again after a lift is another asking, under an id of its own.
again := HoldAsk(key, "mail", fNew, "ace", "x", 7, now)
if again.Asking != 7 || again.Ask.ID != delivery.AskingID(key, 7) || again.Ask.ID == a.Ask.ID {
t.Errorf("the asking after a lift = %d %s", again.Asking, again.Ask.ID)
}
if _, err := delivery.EncodeAssignerAsk(key, again); err != nil {
t.Errorf("the ask asked again cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
}
// 1906/D3 (ADR 0539 §19): the ask about a data module held on a node is a valid ask of the router and of mesh-sdk's layout at
// its restore key, of a kind the view may answer; it offers restoring (losing the newer writes) and leaving it stopped.
func TestTheRestoreAskIsValid(t *testing.T) {
key, err := delivery.RestoreAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := RestoreAsk(key, "ace", "mail", fNew, "connection refused", now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
var ids []string
for _, o := range a.Ask.Options {
ids = append(ids, o.ID)
}
if len(ids) != 2 || ids[0] != OptionRestore || ids[1] != OptionLeaveStopped {
t.Errorf("options = %v", ids)
}
if !strings.Contains(a.Ask.Explanation, "connection refused") || !strings.Contains(a.Ask.Explanation, "ace") {
t.Errorf("the ask does not name the node and the check's words: %q", a.Ask.Explanation)
}
}
// 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own
// name (ADR 0437 clause 9), and is cleared under the same key.
func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Subject.ID != "mail" || c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") || !strings.Contains(c.Key, "mail") {
t.Errorf("condition = %+v", c)
}
if got := HeldCondition("mail", fNew, "", "", false, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateCleared || c.Key != HeldCondition("mail", fNew, "ace", "x", true, false).Key || !strings.Contains(c.Resolved, "lifted its hold by itself") {
t.Errorf("condition = %+v", c)
}
}
// 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in
// the view, and name the move and the node.
func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
now := time.Now()
rkey, err := delivery.RecordAskKey("ace", "mail", "sha256:"+strings.Repeat("b", 64))
if err != nil {
t.Fatal(err)
}
r := RecordAsk(rkey, "ace", "mail", fNew, RecordNeed{Node: "ace", Digest: "sha256:" + strings.Repeat("b", 64), Judgement: delivery.JudgedCannot, Words: "the composition it runs is gone"}, now)
bkey, err := delivery.BackupAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
dgst := "sha256:" + strings.Repeat("b", 64)
b := BackupAsk(bkey, "ace", "mail", fNew, dgst, "no space left", 0, now)
for key, a := range map[string]delivery.AssignerAsk{rkey: r, bkey: b} {
if err := a.Ask.Check(now); err != nil {
t.Errorf("%s: the router refuses the ask: %v", key, err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("%s: cannot be written: %v", key, err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("%s: the view may not answer it: %v", key, err)
}
}
if !strings.Contains(r.Ask.Explanation, "gone") || r.Ask.Options[0].ID != OptionYes {
t.Errorf("the record ask = %+v", r.Ask)
}
if !strings.Contains(b.Ask.Explanation, "no space left") || b.Ask.Options[0].ID != delivery.BackupAgain || b.Ask.Options[1].ID != delivery.BackupSkip {
t.Errorf("the backup ask = %+v", b.Ask)
}
// the id binds the ask to the composition and to its number (review round 1, items 3 and 4)
if again := BackupAsk(bkey, "ace", "mail", fNew, dgst, "x", NextBackupAsking(0), now); again.Ask.ID == b.Ask.ID {
t.Errorf("asked again under the same id")
}
if NextBackupAsking(-1) != 0 || NextBackupAsking(0) != 2 || NextBackupAsking(2) != 3 {
t.Errorf("the numbers of the askings: %d %d %d", NextBackupAsking(-1), NextBackupAsking(0), NextBackupAsking(2))
}
if BackupAskDigest(b) != dgst {
t.Errorf("the ask names composition %q, want %q", BackupAskDigest(b), dgst)
}
if BackupAnswerWords("again", 0) == BackupAnswerWords("again", 1) {
t.Errorf("two answers of the same option are one want value")
}
c := BackupCondition("ace", "mail", fNew, "no space left", true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Errorf("the backup condition is refused: %v", err)
}
}
// 1945/AC21: the condition of a module on neither list is urgent, names the module, the node and the rule holding its want back,
// under the assigner's own name keyed by node and module, and is cleared under the same key.
func TestTheNeitherConditionNamesModuleNodeAndRule(t *testing.T) {
c := NeitherCondition("g14", "i3status-rust", Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent || c.Subject.ID != "i3status-rust" || c.Subject.Machine != "g14" ||
!strings.Contains(c.Explanation, "x11-display") || !strings.Contains(c.Explanation, "rule 5") || c.Key != "mesh-delivery-assigner.neither.g14.i3status-rust" {
t.Errorf("condition = %+v", c)
}
if got := NeitherCondition("g14", "i3status-rust", Holding{}, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
@@ -0,0 +1,282 @@
// Package assign is the assigner's judgement, as functions over what it read from the bus: the composition check of a
// change (novox/hq ADR 0446 clauses 1 to 5, ADR 0528 clause 5, issue 1297), the six rules of what each node should
// run (ADR 0429 §6, ADR 0528 clause 3) and the comparison with what the controller sends (ADR 0528 clause 4). It reads
// and writes nothing itself: the program hands it what it watched and writes what it answers.
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// SDKRepository is the repository of mesh-sdk: a group with a member there changes the composition itself.
const SDKRepository = "novox/mesh-sdk"
// NotJudged names, in a composition's warnings, the failures of ADR 0446 clause 1 the assigner cannot judge from the
// bus, because what they need is only in the controller's facts snapshot, which is not on the bus yet (issue 1461;
// the design limit of R4g, issue 1297): said on every verdict, never passed silently.
const NotJudged = "not judged until the facts snapshot is on the bus (issue 1461): a consumer left out of a grant, a " +
"credential left bound elsewhere, a stored setting the changed definition cannot keep, and a new module the " +
"node-engine's validator would refuse"
// Mesh is the mesh as the trunk holds it: every module's manifest by name, the manifests that cannot be read with
// why, and each node's open assignments.
type Mesh struct {
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Assigned map[string][]string
// Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a
// composer at another mesh-sdk reads (issue 1622).
Digests map[string]string
}
// Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those
// it cannot read and why, the modules it removes, every module defined twice with where, the heads of a mesh-sdk
// member (empty when none), and how many modules it rebuilds.
type Change struct {
Group, Heads string
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Removed []string
Twice map[string][]string
SDKHead string
Rebuilt int
// Digests is the manifest digest of every module the change moves or adds, readable by the assigner or not, and
// Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623).
Digests map[string]string
Members map[string]string
}
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
// removes, those running a consumer of a provision such a module provides (before or after), and those running a
// provider of a provision such a module requires after the change. A module the change adds that is assigned nowhere
// touches no machine.
func Touched(m Mesh, c Change) []string {
moved := map[string]bool{}
for name := range c.Manifests {
moved[name] = true
}
for name := range c.Unreadable {
moved[name] = true
}
for _, name := range c.Removed {
moved[name] = true
}
provided, required := map[string]bool{}, map[string]bool{}
for name := range moved {
for _, man := range []catalogue.Manifest{m.Manifests[name], c.Manifests[name]} {
for _, o := range man.Provides {
provided[o.Name] = true
}
}
for _, r := range c.Manifests[name].Requires {
required[r] = true
}
}
var out []string
for node, modules := range m.Assigned {
for _, mod := range modules {
man := m.Manifests[mod]
if moved[mod] || slices.ContainsFunc(man.Requires, func(r string) bool { return provided[r] }) ||
slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return required[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// input is the composition's input for the mesh's catalogue and assignments: the nodes by name with what is assigned
// to them, and every seat the assigned modules claim as held there. What the bus does not carry (places on the
// private network, settings, credentials, licences) is left empty, so a trunk composition and a change's are made
// with the same gaps and compared (ADR 0446 clause 1: what was already broken is said and fails nothing).
func input(catalog map[string]catalogue.Manifest, assigned map[string][]string) compose.Input {
in := compose.Input{Catalogue: catalog}
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
for _, mod := range assigned[n] {
for _, cl := range catalog[mod].Claims {
scope := cl.Scope
if seat, ok := catalogue.SeatNamed(cl.Name); ok && scope == "" {
scope = seat.Scope
}
in.Holdings = append(in.Holdings, catalogue.Held{Claim: cl.Name, Scope: scope, Node: n, Module: mod})
}
}
}
return in
}
// Compose is the composition check of change c over the mesh m, composed with the mesh-sdk version sdk: every touched
// machine resolved as the trunk holds it and as the change leaves it. It fails on a manifest of the change that cannot
// be read, a module a machine runs that the change removes, a module defined twice, and a machine that resolves on the
// trunk and not with the change; a machine that does not resolve on either is said and fails nothing. It warns past
// CompositionRebuildWarning modules rebuilt, says what it does not judge (NotJudged), and for a group with a mesh-sdk
// member says it composed with its own mesh-sdk, not the member's head (issue 1621). took is how long it took.
func Compose(m Mesh, c Change, sdk string, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: sdk, Took: took.Milliseconds()}
var fails []string
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Unreadable) {
failAt("", fmt.Sprintf("the manifest of %s cannot be read: %s", name, c.Unreadable[name]))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
machines := Touched(m, c)
v.Machines = machines
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
}
trunk := input(m.Manifests, m.Assigned)
changed := map[string]catalogue.Manifest{}
for name, man := range m.Manifests {
changed[name] = man
}
for name, man := range c.Manifests {
changed[name] = man
}
for _, name := range c.Removed {
delete(changed, name)
}
after := input(changed, m.Assigned)
for _, node := range machines {
_, before := compose.Resolve(trunk, node)
_, now := compose.Resolve(after, node)
switch {
case now == nil:
case before != nil:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %v", node, before))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %v", node, now))
}
}
if c.SDKHead != "" {
v.Warnings = append(v.Warnings, fmt.Sprintf("composed with the assigner's own mesh-sdk %s, not the group's mesh-sdk head %s, "+
"and a field the running controller cannot read is not judged (issues 1621, 1622, 1623)", sdk, c.SDKHead))
}
return finish(v, m, c, machines, fails)
}
// finish adds what every composition says beside its own judgement (more modules rebuilt than CompositionRebuildWarning,
// what is not judged, the machines not composed, a change that touches none) and its state from fails.
func finish(v delivery.CompositionVerdict, m Mesh, c Change, machines []string, fails []string) delivery.CompositionVerdict {
if c.Rebuilt > delivery.CompositionRebuildWarning {
v.Warnings = append(v.Warnings, fmt.Sprintf("the merge rebuilds %d modules, more than %d", c.Rebuilt, delivery.CompositionRebuildWarning))
}
v.Warnings = append(v.Warnings, NotJudged)
if could := CouldRun(m, c, machines); len(could) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines that could run a moved or added module and are not assigned it: "+strings.Join(could, ", "))
}
if grants := GrantsAlone(m, c, machines); len(grants) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines whose grants alone the change alters: "+strings.Join(grants, ", "))
}
if len(machines) == 0 {
v.Warnings = append(v.Warnings, "the change touches no machine: no module it moves is assigned anywhere")
}
if len(fails) > 0 {
v.State, v.Words = delivery.CompositionFail, strings.Join(fails, "; ")
return v
}
v.State = delivery.CompositionPass
return v
}
// CouldRun is, by name, every machine of the mesh not composed that is not assigned a module the change moves or adds
// (ADR 0537 clause 1): which of them could run it, the bus does not say (capabilities, settings and places are only in
// the facts snapshot), so each is named as not judged.
func CouldRun(m Mesh, c Change, composed []string) []string {
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for name := range c.Manifests {
if !slices.Contains(modules, name) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// grantsOf is what of a manifest makes grants: its bus declarations and the provisions it requires and offers.
func grantsOf(man catalogue.Manifest) string {
var offers []string
for _, o := range man.Provides {
offers = append(offers, o.Name)
}
return fmt.Sprint(man.Requires, offers, man.Reads, man.State, man.Emits, man.Consumes, man.Invokes, man.Claims, man.Grants)
}
// GrantsAlone is, by name, every machine not composed whose grants alone the change alters (ADR 0537 clause 1): for a
// moved module whose grants differ from the trunk's, the machines assigned the bus's holder (whose user list carries
// every module's bus grants) and those assigned a provider of a provision it requires, before or after.
func GrantsAlone(m Mesh, c Change, composed []string) []string {
provisions := map[string]bool{}
altered := false
for name, man := range c.Manifests {
before, had := m.Manifests[name]
if had && grantsOf(before) == grantsOf(man) {
continue
}
altered = true
for _, r := range append(slices.Clone(man.Requires), before.Requires...) {
provisions[r] = true
}
}
if !altered {
return nil
}
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for _, mod := range modules {
man := m.Manifests[mod]
if man.ClaimsSeat("mesh-broker") || slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return provisions[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,141 @@
package assign
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func meshOf() Mesh {
return Mesh{
Manifests: map[string]catalogue.Manifest{
"mail": {Module: "mail", Version: "1", Requires: []string{"database"}},
"store": {Module: "store", Version: "1", Provides: []catalogue.Offer{{Name: "database"}}},
"web": {Module: "web", Version: "1"},
},
Assigned: map[string][]string{"ace": {"mail", "store"}, "g14": {"web"}},
}
}
func hasWarning(v delivery.CompositionVerdict, part string) bool {
return slices.ContainsFunc(v.Warnings, func(w string) bool { return strings.Contains(w, part) })
}
// 1297/AC1 (its unit half): a change touching one module on one machine composes that machine alone, passes, names
// the machine and the mesh-sdk it composed with, and says what it did not judge.
func TestAChangeOfOneModuleComposesItsMachine(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}}
v := Compose(meshOf(), c, "v0.1.14-x", 0)
if v.State != delivery.CompositionPass || !slices.Equal(v.Machines, []string{"g14"}) || v.SDK != "v0.1.14-x" {
t.Fatalf("composition = %+v, want a pass on g14 alone", v)
}
if !hasWarning(v, "issue 1461") {
t.Errorf("the verdict does not say what it did not judge: %v", v.Warnings)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
}
// 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module.
func TestRemovingAModuleAMachineRunsFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Removed: []string{"web"}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "g14" || !strings.Contains(v.Words, "g14 runs web") {
t.Fatalf("composition = %+v, want a fail naming g14 and web", v)
}
}
// 1297/B3c: a machine that composes on the trunk and not with the change fails it, naming the machine and why; the
// consumer's machine is touched when its provider changes.
func TestAMachineTheChangeBreaksFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"store": {Module: "store", Version: "2"}}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "ace" || !strings.Contains(v.Words, "database") {
t.Fatalf("composition = %+v, want a fail on ace naming the provision nothing provides", v)
}
}
// 1297/B3c: what was already broken on the trunk is said and fails nothing.
func TestWhatWasBrokenBeforeFailsNothing(t *testing.T) {
m := meshOf()
m.Manifests["store"] = catalogue.Manifest{Module: "store", Version: "1"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"mail": {Module: "mail", Version: "2", Requires: []string{"database"}}}}
v := Compose(m, c, "sdk", 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "on the trunk either") {
t.Fatalf("composition = %+v, want a pass saying ace was already broken", v)
}
}
// 1297/B3c: an unreadable manifest and a module defined twice fail; more than twelve rebuilt modules warn.
func TestUnreadableTwiceAndWide(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Unreadable: map[string]string{"web": "unknown field \"scope\""}, Rebuilt: 13}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web cannot be read") || !hasWarning(v, "rebuilds 13 modules") {
t.Fatalf("composition = %+v", v)
}
c = Change{Group: "g", Heads: "0123456789ab", Twice: map[string][]string{"web": {"novox/a", "novox/b"}}, Rebuilt: 12}
v = Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web is defined twice") || hasWarning(v, "rebuilds") {
t.Fatalf("composition = %+v", v)
}
}
// 1622's criterion as R4g delivers it: a group with a mesh-sdk member is composed with the assigner's own mesh-sdk, and
// the verdict says so, naming the member's head it did not use; a group without one says nothing of it.
func TestAGroupWithAnSDKMemberSaysWhichSDKComposedIt(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}, SDKHead: "f625d98713e4"}
v := Compose(meshOf(), c, "v0.1.14-own", 0)
if !hasWarning(v, "not the group's mesh-sdk head f625d98713e4") || !hasWarning(v, "v0.1.14-own") || !hasWarning(v, "1623") {
t.Errorf("the verdict does not say it composed with its own mesh-sdk: %v", v.Warnings)
}
c.SDKHead = ""
if v := Compose(meshOf(), c, "v0.1.14-own", 0); hasWarning(v, "mesh-sdk head") {
t.Errorf("a group without a mesh-sdk member says it has one: %v", v.Warnings)
}
}
// 1297/B3a: the machines touched are those running a moved module or a consumer of what it provides; an added module
// assigned nowhere touches none, and the verdict says so.
func TestTouched(t *testing.T) {
m := meshOf()
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace"}) {
t.Errorf("touched by store = %v", got)
}
m.Assigned["g14"] = append(m.Assigned["g14"], "mail")
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace", "g14"}) {
t.Errorf("touched by store with a consumer on g14 = %v", got)
}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"new": {Module: "new", Version: "1"}}}
if v := Compose(m, c, "sdk", 0); len(v.Machines) != 0 || !hasWarning(v, "touches no machine") || v.State != delivery.CompositionPass {
t.Errorf("an added module assigned nowhere: %+v", v)
}
}
// ADR 0537 clause 1: every verdict names, by machine, those that could run a moved or added module and are not assigned
// it, and those whose grants alone the change alters (the bus's machine, a provider of what the module requires), as not
// judged; shanks, providing what web now requires, is composed and so named in neither. A change that alters no grant
// names none of the second.
func TestTheMachinesNotJudgedAreNamed(t *testing.T) {
m := meshOf()
m.Manifests["bus"] = catalogue.Manifest{Module: "bus", Version: "1", Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: "mesh"}}}
m.Manifests["db"] = catalogue.Manifest{Module: "db", Version: "1", Provides: []catalogue.Offer{{Name: "cache"}}}
m.Assigned["novox"] = []string{"bus"}
m.Assigned["shanks"] = []string{"db"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{
"web": {Module: "web", Version: "2", Reads: []string{"mesh-controller.assignments"}, Requires: []string{"cache"}}}}
v := Compose(m, c, "sdk", 0)
if !hasWarning(v, "could run a moved or added module and are not assigned it: ace, novox") || !slices.Contains(v.Machines, "shanks") {
t.Errorf("the machines that could run web are not named: %v", v.Warnings)
}
if !hasWarning(v, "whose grants alone the change alters: novox") {
t.Errorf("the machines whose grants alone change are not named: %v", v.Warnings)
}
c.Manifests["web"] = catalogue.Manifest{Module: "web", Version: "3"}
if v := Compose(meshOf(), c, "sdk", 0); hasWarning(v, "grants alone") {
t.Errorf("a change altering no grant names machines whose grants change: %v", v.Warnings)
}
}
@@ -0,0 +1,241 @@
package assign
import (
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The cases of a group's composition at its mesh-sdk head: the trunk as it is, and the trunk with the change.
const (
CaseTrunk = "trunk"
CaseChange = "change"
)
// SDKDir is the directory of mesh-sdk's Go module in its repository, where the composer is built at a head; a reader's
// composer is built at its repository's root, at the mesh-sdk its go.mod pins.
const SDKDir = "go"
// The modules whose mesh-sdk reads the manifests a plan sends (ADR 0482 clause 5): the controller composes and sends
// them, the node-engine applies them. Each is judged at its repository.
var ReaderModules = map[string]string{"mesh-controller": "novox/mesh-controller", "mesh-host": "novox/mesh-host"}
// HeadJob is the compose job of a group with a mesh-sdk member (ADR 0482 clauses 3 and 4, issue 1622): the composer
// built at the member's head composes the machines the change touches, as the trunk holds them and as the change leaves
// them, every manifest named by its digest in the indexer's bucket, the seats held from the claims (the assigner's input).
func HeadJob(m Mesh, c Change) (delivery.ComposeJob, error) {
trunk, change := map[string]composer.Manifest{}, map[string]composer.Manifest{}
for name, digest := range m.Digests {
trunk[name] = composer.Manifest{Digest: digest}
change[name] = composer.Manifest{Digest: digest}
}
for name, digest := range c.Digests {
change[name] = composer.Manifest{Digest: digest}
}
for _, name := range c.Removed {
delete(change, name)
}
req := composer.Request{V: composer.Version, Input: plainInput(m.Assigned), Depth: composer.DepthResolve,
HoldingsFromClaims: true, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, {Name: CaseChange, Manifests: change}}}
return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group)
}
// ReaderJob is the compose job that reads the manifests a group changes with the mesh-sdk a reader module pins at
// commit (ADR 0482 clause 5, issue 1623): no machine is composed, every changed manifest is only read.
func ReaderJob(c Change, module, repository, commit string) (delivery.ComposeJob, error) {
changed := map[string]composer.Manifest{}
for name, digest := range c.Digests {
changed[name] = composer.Manifest{Digest: digest}
}
req := composer.Request{V: composer.Version, Cases: []composer.Case{{Name: CaseChange, Manifests: changed}}}
return job(repository, commit, delivery.RootPath, req, module+" at "+commit)
}
func job(repository, commit, dir string, req composer.Request, reader string) (delivery.ComposeJob, error) {
raw, err := json.Marshal(req)
if err != nil {
return delivery.ComposeJob{}, err
}
return delivery.ComposeJob{Fetch: map[string]any{"repository": repository, "commit": commit}, Dir: dir,
Package: composer.Package, Request: raw, Reader: reader}, nil
}
// plainInput is the composition's input without its catalogue: the nodes by name with what is assigned to them (the
// composer derives the seats held from the claims, as input does).
func plainInput(assigned map[string][]string) compose.Input {
var in compose.Input
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
}
return in
}
// Ran is one compose ask's end: its step's state and reason, and the composer's verdict when the step passed.
type Ran struct {
State string
Reason string
Verdict *composer.Verdict
}
// RanOf reads a compose ask's end from its step value; a step that passed with an answer that cannot be read is a
// failure, named.
func RanOf(step delivery.Step) Ran {
r := Ran{State: step.State, Reason: step.Reason}
if step.State != delivery.StepPassed {
return r
}
var a delivery.ComposeAnswer
var v composer.Verdict
if err := json.Unmarshal(step.Result, &a); err != nil {
return Ran{State: delivery.StepFailed, Reason: "the compose step's answer cannot be read: " + err.Error()}
}
if err := json.Unmarshal(a.Verdict, &v); err != nil || v.V != composer.Version {
return Ran{State: delivery.StepFailed, Reason: fmt.Sprintf("the composer's verdict cannot be read (version %d): %v", v.V, err)}
}
r.Verdict = &v
return r
}
func caseOf(v *composer.Verdict, name string) (composer.CaseVerdict, bool) {
if v == nil {
return composer.CaseVerdict{}, false
}
for _, c := range v.Cases {
if c.Name == name {
return c, true
}
}
return composer.CaseVerdict{}, false
}
// Reader is one module whose mesh-sdk reads the group's manifests, at the commit judged: the group's member head when
// the group holds one in its repository, else a build its node runs.
type Reader struct {
Module, Commit string
Member bool
Ran Ran
}
// FromComposer is the composition check of a group with a mesh-sdk member, judged from the composer built at its head
// (ADR 0482 clauses 3 to 5, issues 1622 and 1623): the same failures as Compose, read from the composer's two cases, the
// verdict naming the mesh-sdk the composer was built with; a trunk manifest the head's mesh-sdk refuses fails too (the
// group would deliver a reader that refuses it); and each reader that refuses a changed manifest the head reads fails
// it, naming the module and its words, unless the reader's mesh-sdk has no composer to ask (before issue 1621), which is
// said and judged nothing.
func FromComposer(m Mesh, c Change, head Ran, readers []Reader, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: "mesh-sdk " + c.SDKHead, Took: took.Milliseconds()}
machines := Touched(m, c)
v.Machines = machines
var fails []string
if head.State != delivery.StepPassed {
v.Words = fmt.Sprintf("the composer at mesh-sdk %s did not run: %s %s", c.SDKHead, head.State, head.Reason)
v.State = delivery.CompositionFail
return v
}
if head.Verdict.SDK != "" {
v.SDK = head.Verdict.SDK
}
trunk, okT := caseOf(head.Verdict, CaseTrunk)
change, okC := caseOf(head.Verdict, CaseChange)
if !okT || !okC {
v.State, v.Words = delivery.CompositionFail, "the composer's verdict holds no trunk or no change case"
return v
}
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Digests) {
if r := change.Manifests[name]; !r.Accepted {
failAt("", fmt.Sprintf("the manifest of %s cannot be read by mesh-sdk %s: %s", name, c.SDKHead, r.Refused))
}
}
for _, name := range sortedKeys(trunk.Manifests) {
if _, changed := c.Digests[name]; changed || slices.Contains(c.Removed, name) || trunk.Manifests[name].Accepted {
continue
}
if _, before := m.Unreadable[name]; before {
continue // already unreadable on the trunk: not the group's doing
}
failAt("", fmt.Sprintf("mesh-sdk %s refuses the trunk's manifest of %s, which the group does not change: %s", c.SDKHead, name, trunk.Manifests[name].Refused))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
before, now := trunk.Machines[node], change.Machines[node]
switch {
case now.Composes:
case !before.Composes:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %s", node, before.Reason))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %s", node, now.Reason))
}
}
readerFails, readerWarnings := judgeReaders(c, change, readers)
fails = append(fails, readerFails...)
v.Warnings = append(v.Warnings, readerWarnings...)
return finish(v, m, c, machines, fails)
}
// NoComposer says whether a compose step failed because the mesh-sdk it was built at has no composer command (a mesh-sdk
// from before issue 1621): Go's words for a package its module does not hold.
func NoComposer(reason string) bool {
for _, words := range []string{"cannot find package", "no required module provides package", "is not in std", "cannot find module providing package", "does not contain package"} {
if strings.Contains(reason, words) {
return true
}
}
return false
}
func judgeReaders(c Change, head composer.CaseVerdict, readers []Reader) (fails, warnings []string) {
for _, r := range readers {
at := r.Module + " at " + r.Commit
if r.Member {
at = r.Module + "'s member head " + r.Commit
}
if r.Ran.State != delivery.StepPassed {
if NoComposer(r.Ran.Reason) {
warnings = append(warnings, fmt.Sprintf("not judged whether %s reads the group's manifests: the mesh-sdk it pins has no "+
"composer command (before issue 1621)", at))
continue
}
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins did not run: %s %s", at, r.Ran.State, r.Ran.Reason))
continue
}
read, ok := caseOf(r.Ran.Verdict, CaseChange)
if !ok {
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins answered no change case", at))
continue
}
for _, name := range sortedKeys(c.Digests) {
if !head.Manifests[name].Accepted || read.Manifests[name].Accepted {
continue
}
fails = append(fails, fmt.Sprintf("the manifest of %s uses what %s (mesh-sdk %s) cannot read: %s; the group must move %s onto "+
"mesh-sdk %s", name, at, r.Ran.Verdict.SDK, read.Manifests[name].Refused, r.Module, c.SDKHead))
}
}
return fails, warnings
}
@@ -0,0 +1,184 @@
package assign
import (
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
var testTime = time.Date(2026, 10, 24, 3, 0, 0, 0, time.UTC)
const sdkHead = "1111111111111111111111111111111111111111"
// raws are the manifests of meshOf as their repositories hold them, by digest, as the indexer keeps them; `web2` is web
// with a field `shares` that the assigner's own mesh-sdk does not read (the shape of issue 1593).
var raws = map[string]string{
"d-mail": `{"module": "mail", "version": "1", "requires": ["postgres-database"]}`,
"d-store": `{"module": "store", "version": "1", "provides": [{"name": "postgres-database"}]}`,
"d-web": `{"module": "web", "version": "1"}`,
"d-web2": `{"module": "web", "version": "2", "shares": ["photos"]}`,
}
// asTheHeadReads is how mesh-sdk at the group's head reads each manifest: as the assigner's own reads it, but for the
// field it adds, which it accepts. The head's composer is built and run by the Go build module (ADR 0482 clause 4); here
// its verdict is made by mesh-sdk's composer.Run over the same request, with `web2` read as the head reads it.
var asTheHeadReads = map[string]string{"d-web2": `{"module": "web", "version": "2"}`}
func digestedMesh() Mesh {
m := meshOf()
m.Digests = map[string]string{"mail": "d-mail", "store": "d-store", "web": "d-web"}
return m
}
// webShares is a group of mesh-sdk and mesh-catalog whose catalogue member gives web the field only the SDK head reads:
// the assigner's own reader refuses it (Unreadable), as the indexer's manifest is read in world.change.
func webShares(members map[string]string) Change {
_, err := catalogue.ParseManifest([]byte(raws["d-web2"]))
return Change{Group: "g", Heads: "0123456789ab", SDKHead: sdkHead, Manifests: map[string]catalogue.Manifest{},
Unreadable: map[string]string{"web": err.Error()}, Digests: map[string]string{"web": "d-web2"}, Members: members}
}
// run is the composer's step for a job, as the build module answers it: the request filled from the indexer's digests
// (with read standing in for how that mesh-sdk reads a digest), run, and its verdict put in a passed step.
func run(t *testing.T, job delivery.ComposeJob, read map[string]string) Ran {
t.Helper()
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if err := composer.Fill(&req, func(d string) ([]byte, error) {
if r, ok := read[d]; ok {
return []byte(r), nil
}
return []byte(raws[d]), nil
}); err != nil {
t.Fatal(err)
}
v, err := composer.Run(req)
if err != nil {
t.Fatal(err)
}
v.SDK = "(devel) " + job.Fetch["commit"].(string)
vr, _ := json.Marshal(v)
ans, _ := json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0", Verdict: vr})
return RanOf(delivery.Step{State: delivery.StepPassed, Result: ans})
}
// 1622/AC1: a group holding a mesh-sdk pull request is composed by the composer built at the member's head: the job
// fetches mesh-sdk at that head, builds mesh-sdk's composer command in its Go module, and hands it the trunk and the
// change by digest, never the manifests' bytes, composing the machines the change touches.
func TestAGroupWithAnSDKMemberAsksTheComposerAtItsHead(t *testing.T) {
c := webShares(nil)
job, err := HeadJob(digestedMesh(), c)
if err != nil {
t.Fatal(err)
}
if job.Fetch["repository"] != SDKRepository || job.Fetch["commit"] != sdkHead || job.Dir != "go" || job.Package != composer.Package {
t.Fatalf("job = %+v", job)
}
ask := delivery.StepAsk{ID: "mh3k2x9qa1b2c5", Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: "mesh-delivery-assigner",
At: testTime, Compose: &job}
if _, err := delivery.EncodeStepAsk(ask); err != nil {
t.Fatalf("the ask is refused by its layout: %v", err)
}
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if len(req.Cases) != 2 || req.Cases[0].Name != CaseTrunk || req.Cases[1].Name != CaseChange {
t.Fatalf("cases = %+v", req.Cases)
}
if req.Cases[0].Manifests["web"].Digest != "d-web" || req.Cases[1].Manifests["web"].Digest != "d-web2" || len(req.Cases[1].Manifests["web"].Raw) != 0 {
t.Errorf("the cases do not name web by its trunk and its change digest: %+v", req.Cases)
}
if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims {
t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims)
}
}
// 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the
// verdict names the mesh-sdk the composer was built with; with the assigner's own mesh-sdk it fails.
func TestAFieldTheGroupAddsPassesWithTheHeadsComposer(t *testing.T) {
c := webShares(map[string]string{"novox/mesh-controller": "2222222222222222222222222222222222222222"})
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
v := FromComposer(digestedMesh(), c, head, nil, 0)
if v.State != delivery.CompositionPass || v.SDK != "(devel) "+sdkHead || !slices.Equal(v.Machines, []string{"g14"}) {
t.Fatalf("with the head's composer: %+v", v)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
own := Compose(digestedMesh(), c, "v0.1.14-own", 0)
if own.State != delivery.CompositionFail || !strings.Contains(own.Words, "shares") {
t.Fatalf("with the assigner's own mesh-sdk: %+v, want a fail naming the field", own)
}
}
// 1623/AC1, AC2: the group of mesh-sdk and mesh-catalog adding and using a field, without a controller member, fails
// naming the field and mesh-controller, whose running build pins an older mesh-sdk; with a controller member pinned to
// the SDK head (its composer reads what the head reads) it passes.
func TestAFieldTheRunningControllerCannotReadFailsTheGroup(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
running := "3333333333333333333333333333333333333333"
rjob, err := ReaderJob(c, "mesh-controller", "novox/mesh-controller", running)
if err != nil {
t.Fatal(err)
}
if rjob.Fetch["commit"] != running || rjob.Dir != delivery.RootPath {
t.Fatalf("reader job = %+v", rjob)
}
old := run(t, rjob, nil) // the running controller's mesh-sdk reads web2 as the assigner's own does: refused
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: running, Ran: old}}, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "shares") || !strings.Contains(v.Words, "move mesh-controller onto mesh-sdk "+sdkHead) {
t.Fatalf("without a controller member: %+v, want a fail naming the field and mesh-controller", v)
}
member := "2222222222222222222222222222222222222222"
pinned := run(t, rjob, asTheHeadReads)
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: member, Member: true, Ran: pinned}}, 0)
if v.State != delivery.CompositionPass {
t.Fatalf("with a controller member pinned to the head: %+v, want a pass", v)
}
}
// 1623: a reader whose mesh-sdk has no composer command (before issue 1621) is said, not failed; a reader's composer that
// failed for another cause fails the group, loud.
func TestAReaderWithNoComposerIsSaidAndAnotherFailureFails(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
none := Ran{State: delivery.StepFailed, Reason: "go build: no required module provides package git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose"}
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: none}}, 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "mesh-host at abc") || !hasWarning(v, "before issue 1621") {
t.Fatalf("a reader with no composer: %+v", v)
}
broke := Ran{State: delivery.StepFailed, Reason: "the fetch could not reach the forge"}
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: broke}}, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "could not reach the forge") {
t.Fatalf("a reader whose composer failed: %+v", v)
}
}
// 1622: the head's composer that did not run fails the composition naming why, never passes it; a trunk manifest the
// head's mesh-sdk refuses fails it too, since the group would deliver a reader that refuses it.
func TestTheHeadsComposerFailingOrRefusingTheTrunkFails(t *testing.T) {
c := webShares(nil)
v := FromComposer(digestedMesh(), c, Ran{State: delivery.StepRefused, Reason: "the guard refuses it"}, nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "the guard refuses it") || !strings.Contains(v.Words, sdkHead) {
t.Fatalf("a head composer that did not run: %+v", v)
}
job, _ := HeadJob(digestedMesh(), c)
strict := map[string]string{"d-web2": asTheHeadReads["d-web2"], "d-mail": `{"module": "mail", "version": "1", "gone": true}`}
v = FromComposer(digestedMesh(), c, run(t, job, strict), nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "refuses the trunk's manifest of mail") {
t.Fatalf("a head that refuses a trunk manifest: %+v", v)
}
}
@@ -0,0 +1,530 @@
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
// trunk (the indexer's modules), whether the builder's build of F is done, whether F is held, whether the controller
// issued its credential for F, its manifest at F, and the nodes it is assigned to.
type Module struct {
Name string
FP string
Built bool
Held bool
Granted bool
// GrantedAt says whether the controller issued the module's credential for another fingerprint than FP: the first want of a node
// that runs an older one names it (issue 1945). Nil: none.
GrantedAt func(fp string) bool
// GrantedOn says whether the controller's grant of FP names the module's accounts on a node, when the grant names any:
// a grant written for the module's other nodes says nothing of a node whose account is not issued and sealed (issue
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone.
GrantedOn func(node string) bool
Manifest catalogue.Manifest
Nodes []string
// Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its
// digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool)
// Allowed says whether the operator answered yes to the record ask about the move of this module on node to the composition
// with this digest (ADR 0539 §14). Nil: none did.
Allowed func(node, digest string) bool
// Released is a hold on FP the operator released (ADR 0539 §19): FP is not held again by a health check that failed.
Released bool
// Restored says whether the operator answered "restore" to the restore ask of this data module on node: it is then set back
// as any module is, losing the writes made since (ADR 0539 §19). Nil: none did.
Restored func(node string) bool
// Kept says the operator answered "keep" to the hold ask of FP: a hold kept so is not lifted by itself.
Kept bool
// Now is when this pass reads the haves: the held record names it as when the health check was seen to fail. Zero: not said.
Now time.Time
}
func (m Module) composition(node, fp string) (delivery.ComposedDigest, bool) {
if m.Composed == nil {
return delivery.ComposedDigest{}, false
}
return m.Composed(node, fp)
}
// Node is what the rules read of the nodes: whether each is online (the watchman's presence) and each module's have.
type Nodes struct {
Online map[string]bool
Haves map[string]map[string]delivery.Have
// Providers is, by provision, the modules that provide it, with the nodes each is assigned to.
Providers map[string]map[string][]string
}
// healthy says whether a have runs fp healthy.
func healthy(h delivery.Have, fp string) bool {
return h.FP == fp && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
// Canary is the module's canary: the first of its nodes, by name, that is online (ADR 0435 §3; one ring, ordered by
// name, ADR 0528 clause 3). Empty when none is online.
func Canary(nodes []string, online map[string]bool) string {
sorted := slices.Sorted(slices.Values(nodes))
for _, n := range sorted {
if online[n] {
return n
}
}
return ""
}
// Holds is whether every rule holds for moving node's want for m to m.FP, and if not, the first rule that does not,
// with why (ADR 0429 §6). Rule 4, at most `step` others ahead, always holds: the mesh keeps no step, and step 0 is no
// limit (ADR 0528 clause 3).
func Holds(m Module, node string, n Nodes) (bool, int, string) {
return holds(m, node, n, true)
}
// holds is Holds; with ordering false it leaves out the two rules that order a move among nodes and modules, the canary (3)
// and the providers (5), and keeps those without which there is nothing to want: the build done, not held, the grant and the
// composition.
func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
switch {
case !m.Built:
return false, delivery.RuleBuildDone, fmt.Sprintf("the build of %s at %s is not done", m.Name, short(m.FP))
case m.Held:
return false, delivery.RuleNotHeld, fmt.Sprintf("%s is held at %s", m.Name, short(m.FP))
}
if ordering {
canary := Canary(m.Nodes, n.Online)
switch {
case canary == "":
return false, delivery.RuleCanary, fmt.Sprintf("none of the nodes of %s is online, so it has no canary", m.Name)
case node != canary && !healthy(n.Haves[canary][m.Name], m.FP):
return false, delivery.RuleCanary, fmt.Sprintf("the canary %s does not run %s healthy yet", canary, short(m.FP))
}
for _, provision := range m.Manifest.Requires {
if ok, why := providedHealthy(provision, node, n); !ok {
return false, delivery.RuleProviders, why
}
}
}
if !m.Granted {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s for %s", m.Name, short(m.FP))
}
if m.GrantedOn != nil && !m.GrantedOn(node) {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s on %s for %s", m.Name, node,
short(m.FP))
}
if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP))
}
return true, 0, ""
}
// providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned
// elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node
// offers it) and holds.
func providedHealthy(provision, node string, n Nodes) (bool, string) {
providers := n.Providers[provision]
if len(providers) == 0 {
return true, ""
}
names := make([]string, 0, len(providers))
for p := range providers {
names = append(names, p)
}
sort.Strings(names)
for _, p := range names {
nodes := providers[p]
at := node
if !slices.Contains(nodes, node) {
at = Canary(nodes, n.Online)
}
if h, ok := n.Haves[at][p]; ok && at != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return true, ""
}
}
return false, fmt.Sprintf("no provider of %s runs healthy on %s or at its canary", provision, node)
}
// keepsData says whether a module holds data a set-back would lose: an own item of a class that keeps (irreplaceable,
// valuable or rebuildable; cache and none do not).
func keepsData(m catalogue.Manifest) bool {
for _, it := range m.DataItems() {
if it.Class == catalogue.ClassIrreplaceable || it.Class == catalogue.ClassValuable || it.Class == catalogue.ClassRebuildable {
return true
}
}
return false
}
// Outcome is what one module's pass decides: each node's want to write (only those that change), and a hold to write.
type Outcome struct {
Wants map[string]delivery.Want
// Held is the hold of m.FP to write, when a node's health check failed on it and it is not held yet.
Held *delivery.Held
// Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held
// record and its ask are taken away, and the wants move by the rules again in this pass.
Lifted bool
// Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14).
Record []RecordNeed
// Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set
// back: the operator is asked whether to restore each (ADR 0539 §19).
Restore []string
// Rules is, by node, the rule holding the node back from m.FP and why.
Rules map[string]Holding
// Neither is, by node, the nodes the module is assigned to that run it (or whose have names it), hold no want for it after this
// pass and are not held (a held module is on the held list), with the rule holding the first want back: the node's own composition does not hold a module with a fingerprint of the new form, so such a module is on
// neither list and nothing applies it on that node (issue 1945). A module with no fingerprint of the new form goes with the node's
// own composition and is never here.
Neither map[string]Holding
}
// RecordNeed is a move of a module of policy record on a node that waits for the operator's yes: the composition's digest, the
// controller's judgement of the move and its words.
type RecordNeed struct {
Node string
Digest string
Judgement delivery.Judgement
Words string
}
// Holding is a rule holding a node back, and why.
type Holding struct {
Rule int
Words string
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
// before (wants, by node); lastHealthy is, by node, the newest fingerprint other than m.FP the node ran healthy. A failed
// health check of m.FP on a node holds it: every node wanting m.FP is set back to its last healthy fingerprint, but a
// module holding data is not set back but asked about (Restore). A
// node that wants m but is no longer among its nodes gets its want written removed, never deleted (ADR 0408 §4).
func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[string]string) Outcome {
out := Outcome{Wants: map[string]delivery.Want{}, Rules: map[string]Holding{}, Neither: map[string]Holding{}}
held := m.Held
if held && !m.Kept && m.FP != "" && allRunHealthy(m, n) {
out.Lifted, held = true, false
}
if !held && !out.Lifted && !m.Released && m.FP != "" {
for _, node := range slices.Sorted(slices.Values(m.Nodes)) {
if h, ok := n.Haves[node][m.Name]; ok && h.FP == m.FP && failed(h) {
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
}
}
}
mm := m
mm.Held = held
for _, node := range m.Nodes {
if m.FP == "" {
break
}
before, had := wants[node]
var next delivery.Want
// A revision of the bus module's composition that differs only in its user list is written at once, at the fingerprint the
// node runs, outside the rules, the record hold and the bar on a new fingerprint: a new module's credential must reach the
// bus before the module can connect (ADR 0539 §7). The controller marks it; whatever it judges of the file's mount stands.
if w, ok := m.userListRevision(node, before, had, n); ok {
if !had || !delivery.SameWant(normal(before), normal(w)) {
out.Wants[node] = w
}
continue
}
ok, rule, words := Holds(mm, node, n)
target := m.FP
if have, ran := n.Haves[node][m.Name]; !ok && (!had || before.Removed) && (rule == delivery.RuleCanary || rule == delivery.RuleProviders) && runs(have) {
// A node that runs the module and has no want for it has no want the ordering rules could hold it back to: holding its first
// want back leaves the module on neither list, since the node's own composition gives up every module with a fingerprint of
// the new form (issue 1945: on g14 the bundles of clipmenu, screen-lock and every module requiring x11-display were removed
// and never made again, because xorg's health is unknown and screen-lock's canary never ran it healthy). Its first want names
// what it runs, never a newer fingerprint, so the canary is never passed over: F when it runs F (by the rules without which
// there is nothing to want), or an older fingerprint of the new form whose composition and grant exist. Rules 3 and 5 then
// order the move from it as from any want.
switch _, composed := m.composition(node, have.FP); {
case !ran:
case have.FP == m.FP:
ok, rule, words = holds(mm, node, n, false)
case !strings.HasPrefix(have.FP, delivery.SourcePrefix2):
// An old-form build (not src2:) has no composition, so no want can ever name it: its first want names F, and only by every
// rule as usual, the canary (3) and the providers (5) included, which here hold it back. Nothing is written and the module
// stays on neither list until the canary runs F healthy; then Holds passes and the want names F (issue 1945, a design
// limit set by the lead: the canary stays in force and old-form modules leave neither list by themselves).
case composed && m.GrantedAt != nil && m.GrantedAt(have.FP):
ok, target = true, have.FP
}
}
switch {
case ok:
d, _ := m.composition(node, target)
next = delivery.Want{FP: target, Composed: d.Digest}
at := m
if target != m.FP {
// the first want at the older fingerprint the node runs: held back from F by the rule, and judged as a want at it
next.Newest, next.Rule, next.Words = m.FP, rule, words
at.FP = target
}
if had && !before.Removed && before.FP == target && before.Composed == d.Digest {
next.Backup, next.Words = before.Backup, before.Words // the answer to a failed backup stays while its target stays (ADR 0539 §16)
}
running, ran := n.Haves[node][m.Name]
// A removed want is no want: a first want after it goes through the record ask as a first want does.
asked, hadLive := before, had
if before.Removed {
asked, hadLive = delivery.Want{}, false
}
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
} else if !hadLive && ran && runs(running) {
// The bus, which the assigner never moves (§15), runs here with no want: it is on neither list until the operator
// writes its want, and the pass says why (issue 1945).
out.Neither[node] = Holding{Rule: delivery.RuleComposed, Words: busMoveWords(m.Name, node, target, d)}
}
continue
}
case held && had && before.FP == m.FP:
out.Rules[node] = Holding{Rule: rule, Words: words}
last := lastHealthy[node]
switch {
case keepsData(m.Manifest) && (m.Restored == nil || !m.Restored(node)):
next = before
next.Words = fmt.Sprintf("%s holds data, so it is not set back from %s: the operator is asked whether to restore it", m.Name, short(m.FP))
out.Restore = append(out.Restore, node)
case last == "":
next = before
next.Words = fmt.Sprintf("%s is held at %s and %s ran no other fingerprint healthy to set it back to", m.Name, short(m.FP), node)
default:
d, there := m.composition(node, last)
if !there {
next = before
next.Words = fmt.Sprintf("%s is held at %s, and the composition of %s at %s is gone, so %s is not set back", m.Name, short(m.FP), m.Name, short(last), node)
break
}
next = delivery.Want{FP: last, Composed: d.Digest, Newest: m.FP, Rule: rule, Words: words}
}
default:
out.Rules[node] = Holding{Rule: rule, Words: words}
if !had || before.Removed {
// The node runs the module, or its have names it, and gets no want for it: the module is on neither list (issue 1945), and
// the pass says so. A new assignment, with no have, waits for the rules and is on no list yet by design. A held module (rule 2,
// held before or by this pass) is on the held list instead, whose hold is said and answered on its own: it is never on neither
// list (issue 1945: systemd-resolved, held on shanks by the operator, was said as urgent every hour).
if h, there := n.Haves[node][m.Name]; there && h.Phase != delivery.PhaseRemoved && !held {
out.Neither[node] = Holding{Rule: rule, Words: words}
}
continue
}
next = delivery.Want{FP: before.FP, Composed: before.Composed, Newest: m.FP, Rule: rule, Words: words}
if before.FP == m.FP {
// Held at F by the rules: a new revision of F's composition waits for them, as a new fingerprint does (ADR 0539 §5).
next = delivery.Want{FP: m.FP, Composed: before.Composed}
} else if d, there := m.composition(node, before.FP); there && d.Digest != "" && d.Digest != before.Composed {
// Held back at an earlier fingerprint whose kept composition the controller composed again with today's config
// (novox/hq ADR 0545): the want follows its new revision, so the node does not run that build with the config of
// the moment it was made (on g14 a capacity of 0 after a rollback). Asked as any move of a record module is.
at := m
at.FP = before.FP
running, ran := n.Haves[node][m.Name]
need, wait := at.mustAsk(node, before, true, ran, running.FP, running.Composed == "", d)
if need != nil {
out.Record = append(out.Record, *need)
}
if !wait {
next.Composed = d.Digest
}
}
}
if !had || !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
}
for node, before := range wants {
if !slices.Contains(m.Nodes, node) && !before.Removed {
out.Wants[node] = delivery.Want{Removed: true, Words: fmt.Sprintf("%s is no longer assigned to %s", m.Name, node)}
}
}
return out
}
// failed says whether a have's health check failed once the node is done moving: phase running or failed (never removed, held, or
// moving). A failed health while it
// backs up, applies or checks is a step of the move, not its outcome (on novox at 15:27:36 on 2026-10-25, "health failed, phase
// applying" held the controller's canary two seconds before the new build was healthy). Every healthy read (healthy, providedHealthy,
// the world's lastHealthy) already asks for phase running.
func failed(h delivery.Have) bool {
return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed)
}
// runs says whether a have is of a module the node runs: running, whatever its health.
func runs(h delivery.Have) bool {
return h.Phase == delivery.PhaseRunning
}
// allRunHealthy says whether every node of m that runs the module (its have's phase is running, or failed) runs m.FP healthy, and
// at least one does: a hold whose fingerprint runs healthy wherever the module runs holds nothing back any more (issue 1906: seven
// modules stayed held on every machine after one transient failed check). A node set back to another fingerprint does not run
// m.FP, and a failed one does not run it healthy, so the hold stays; a have that is removed, held for its backup or still moving
// (backing up, applying, checking) is not counted, or it would keep the hold up for ever.
func allRunHealthy(m Module, n Nodes) bool {
ran := 0
for _, node := range m.Nodes {
h, ok := n.Haves[node][m.Name]
if !ok || (h.Phase != delivery.PhaseRunning && h.Phase != delivery.PhaseFailed) {
continue
}
if !healthy(h, m.FP) {
return false
}
ran++
}
return ran > 0
}
// HeldWords are the words a hold keeps of its cause (issue 1906): the health and phase the node reported, when the assigner saw
// it, and the node's own words. The node is the held record's own field.
func HeldWords(h delivery.Have, at time.Time) string {
words := fmt.Sprintf("health %s, phase %s", h.Health, h.Phase)
if !at.IsZero() {
words += ", seen at " + at.UTC().Format(time.RFC3339)
}
if d := strings.TrimSpace(h.Detail); d != "" {
return words + ": " + d
}
return words + ": the node gave no words"
}
// LiftedWords say why a hold was lifted by itself.
func LiftedWords(module, fp string) string {
return fmt.Sprintf("every node that runs %s runs %s healthy, so the assigner lifted its hold by itself", module, short(fp))
}
// normal is a want without its version, as a decoded and a composed one are compared.
func normal(w delivery.Want) delivery.Want {
w.V = 0
return w
}
func short(fp string) string {
part, err := delivery.FingerprintKeyPart(fp)
if err != nil || len(part) < 12 {
return fp
}
return part[:12]
}
// mustAsk says whether the move to the composition d on node waits: for the operator's yes (need names the ask) or, for the bus,
// for the operator's hand (ADR 0539 §14, §15). A module the node never ran (it has no want and no have), a move judged neither, a module that is not of
// policy record and a want that stays are written as any want; so is a revision of the bus module's composition at the
// same fingerprint judged neither (a user list only, §7).
func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, running string, haveNamesNoComposition bool, d delivery.ComposedDigest) (*RecordNeed, bool) {
if (!had && !ran) || before.Removed || (before.FP == m.FP && before.Composed == d.Digest) {
return nil, false
}
// Right after the switch no node has a readable want, and a have from the old node-engine names a fingerprint and no composition
// (ADR 0539 §9, §11): a node running this fingerprint is given its want at it with no hold when the controller judges the composition
// unchanged, equal to the declaration the node last applied (ADR 0540 §9), since the node agent then applies nothing. A composition that
// differs from it (a changed setting or secret), or that the controller could not judge, waits for the ask. A new fingerprint still waits.
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
// The fingerprint the bus ran before: its want's, or with no readable want the one its have names (ADR 0539 §15 bars a new one).
prev := before.FP
if !had {
prev = running
}
if catalogue.ProvidesBus(m.Manifest) && prev != m.FP {
return nil, true // the bus is never moved by the assigner; the operator writes its want (§15)
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
if m.Allowed != nil && m.Allowed(node, d.Digest) {
return nil, false
}
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// busMoveWords say why the bus module's want at fp is not written on node: its composition differs from what the node runs in more
// than the bus's user list, so it is a move of the bus, which the operator writes (ADR 0539 §15).
func busMoveWords(module, node, fp string, d delivery.ComposedDigest) string {
why := string(d.Judgement)
if w := strings.TrimSpace(d.Words); w != "" {
why += ": " + w
}
return fmt.Sprintf("the composition of %s for %s at %s changes more than the bus's user list (judged %s), so it is a move of the bus, "+
"which the assigner never makes: the operator writes its want (ADR 0539 §15)", module, node, short(fp), why)
}
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
if d.Policy != "" {
return d.Policy == delivery.PolicyRecord
}
policy, _, _ := catalogue.DerivedUpgrade(m.Manifest)
return policy == catalogue.PolicyRecord
}
// userListRevision is the want that names the bus module's composition at the fingerprint the node runs when that composition
// differs from the one the node runs only in the bus's user list (ComposedDigest.UserListOnly), and whether there is one. A node
// whose want names another fingerprint than the one it runs, or a module that does not provide the bus, has none.
//
// A node whose have is of the old form (from the old node-engine: a build and no fingerprint of the new form, no composition) runs
// no fingerprint a composition is kept at, so its revision names F, the newest fingerprint, when the controller judged F's
// composition to differ from the declaration the node last applied only in the bus's user list (ADR 0539 §7, ADR 0540 §9). Any
// other difference is a move of the bus and stays the operator's (§15). Without this the bus never got its first want after the
// switch, and its user list was never written again (issue 1945: novox's nats from 11:23 on 2026-10-25).
func (m Module) userListRevision(node string, before delivery.Want, had bool, n Nodes) (delivery.Want, bool) {
if !catalogue.ProvidesBus(m.Manifest) {
return delivery.Want{}, false
}
running, ran := n.Haves[node][m.Name]
if ran && oldForm(running) && m.FP != "" && (!had || before.Removed || before.FP == m.FP) {
d, ok := m.composition(node, m.FP)
if !ok || !d.UserListOnly {
return delivery.Want{}, false
}
w := delivery.Want{FP: m.FP, Composed: d.Digest}
if had && !before.Removed && before.Composed == d.Digest {
w.Backup, w.Words = before.Backup, before.Words
}
return w, true
}
if !ran || running.FP == "" || (had && (before.Removed || before.FP != running.FP)) {
return delivery.Want{}, false
}
d, ok := m.composition(node, running.FP)
if !ok || !d.UserListOnly {
return delivery.Want{}, false
}
w := delivery.Want{FP: running.FP, Composed: d.Digest}
if had && before.Composed == d.Digest {
w.Backup, w.Words = before.Backup, before.Words
}
return w, true
}
// oldForm says whether a have is of the old form: the old node-engine's, naming a build and no fingerprint of the new form, and no
// composition (issue 1945: novox's nats, build-1792674595610176744).
func oldForm(h delivery.Have) bool {
return h.Phase != delivery.PhaseRemoved && h.Composed == "" && (h.FP != "" || h.Build != "") &&
!strings.HasPrefix(h.FP, delivery.SourcePrefix2)
}
// GrantNamesNode says whether a grant of module names its accounts on node: its own `<node>.<module>` or a process's
// `<node>.<module>.<process>`. A grant that names no account (a module that does not speak on the bus) names every node.
func GrantNamesNode(g delivery.ControllerGrant, node, module string) bool {
if len(g.Accounts) == 0 {
return true
}
own := node + "." + module
for _, a := range g.Accounts {
if a == own || strings.HasPrefix(a, own+".") {
return true
}
}
return false
}
@@ -0,0 +1,957 @@
package assign
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func fpOf(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
var (
fNew = fpOf('b')
fOld = fpOf('a')
)
// ready is a module mail at fNew on ace and g14 for which every rule holds: built, not held, granted, its provider
// store healthy on ace and the canary ace running fNew healthy.
func ready() (Module, Nodes) {
m := Module{Name: "mail", FP: fNew, Built: true, Granted: true, Composed: composedEverywhere, Nodes: []string{"g14", "ace"},
Manifest: catalogue.Manifest{Module: "mail", Requires: []string{"database"}}}
n := Nodes{
Online: map[string]bool{"ace": true, "g14": true},
Haves: map[string]map[string]delivery.Have{
"ace": {"mail": {FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}, "store": {FP: fpOf('s'), Phase: delivery.PhaseRunning, Health: delivery.HealthOK}},
"g14": {"mail": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}},
},
Providers: map[string]map[string][]string{"database": {"store": {"ace"}}},
}
return m, n
}
// dig is the digest of the composition at a fingerprint in these tests: the fingerprint's last character repeated.
func dig(fp string) string { return "sha256:" + strings.Repeat(fp[len(fp)-1:], 64) }
// composedEverywhere is a Module.Composed with a composition for every node at every fingerprint, judged neither.
func composedEverywhere(_, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedNeither}, true
}
func wantedOld() map[string]delivery.Want {
return map[string]delivery.Want{"ace": {FP: fOld, Composed: dig(fOld)}, "g14": {FP: fOld, Composed: dig(fOld)}}
}
// 1297/B4: with every rule holding, both nodes' wants move to the new fingerprint (the canary ace runs it healthy, so
// g14 follows at once: step 0, ADR 0528 clause 3).
func TestEveryRuleHoldsSoTheWantsMove(t *testing.T) {
m, n := ready()
out := Wants(m, n, wantedOld(), nil)
for _, node := range []string{"ace", "g14"} {
if out.Wants[node].FP != fNew || out.Wants[node].Rule != 0 || out.Wants[node].Composed != dig(fNew) {
t.Errorf("%s's want = %+v, want %s", node, out.Wants[node], fNew)
}
}
}
// 1297/B4, one test per rule: given every rule but N holds, no want moves to F, and the rule N is named.
func TestEachRuleHoldsTheWantBack(t *testing.T) {
for rule, breakIt := range map[int]func(*Module, *Nodes){
delivery.RuleBuildDone: func(m *Module, _ *Nodes) { m.Built = false },
delivery.RuleNotHeld: func(m *Module, _ *Nodes) { m.Held = true },
delivery.RuleCanary: func(_ *Module, n *Nodes) {
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
},
delivery.RuleProviders: func(_ *Module, n *Nodes) {
n.Haves["ace"]["store"] = delivery.Have{FP: fpOf('s'), Phase: delivery.PhaseRunning, Health: delivery.HealthWaits}
},
delivery.RuleGrant: func(m *Module, _ *Nodes) { m.Granted = false },
delivery.RuleComposed: func(m *Module, _ *Nodes) {
m.Composed = func(string, string) (delivery.ComposedDigest, bool) { return delivery.ComposedDigest{}, false }
},
} {
m, n := ready()
breakIt(&m, &n)
out := Wants(m, n, wantedOld(), nil)
if w, ok := out.Wants["g14"]; ok && w.FP == fNew {
t.Errorf("rule %d broken: g14's want moved to F", rule)
}
if got := out.Rules["g14"].Rule; got != rule {
t.Errorf("rule %d broken: g14 is held back by rule %d (%s)", rule, got, out.Rules["g14"].Words)
}
if w := out.Wants["g14"]; w.FP != "" && (w.FP != fOld || w.Newest != fNew || w.Rule != rule) {
t.Errorf("rule %d broken: g14's want is %+v", rule, w)
}
}
// Rule 6 per node: a grant written for the module's other nodes does not grant a node whose account it does not name
// (issue 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus).
m, n := ready()
grant := delivery.ControllerGrant{Accounts: []string{"ace.mail", "ace.mail.send"}}
m.GrantedOn = func(node string) bool { return GrantNamesNode(grant, node, "mail") }
out := Wants(m, n, wantedOld(), nil)
if w := out.Wants["ace"]; w.FP != fNew {
t.Errorf("ace, whose account the grant names, wants %+v", w)
}
if got := out.Rules["g14"]; got.Rule != delivery.RuleGrant || !strings.Contains(got.Words, "on g14") {
t.Errorf("g14, whose account the grant does not name, is held by %+v", got)
}
for _, c := range []struct {
accounts []string
node, module string
named bool
}{
{nil, "g14", "mail", true},
{[]string{"g14.mail.send"}, "g14", "mail", true},
{[]string{"g14.mailer"}, "g14", "mail", false},
{[]string{"ace.mail"}, "g14", "mail", false},
} {
if got := GrantNamesNode(delivery.ControllerGrant{Accounts: c.accounts}, c.node, c.module); got != c.named {
t.Errorf("%v names %s of %s: %v, want %v", c.accounts, c.module, c.node, got, c.named)
}
}
}
// 1297/B4, ADR 0435 §3: a canary that is offline is passed over: the next node by name is the canary, and moves first.
func TestAnOfflineCanaryIsPassedOver(t *testing.T) {
m, n := ready()
n.Online["ace"] = false
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
n.Providers["database"]["store"] = []string{"ace", "g14"}
n.Haves["g14"]["store"] = delivery.Have{FP: fpOf('s'), Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if c := Canary(m.Nodes, n.Online); c != "g14" {
t.Fatalf("canary = %q, want g14", c)
}
out := Wants(m, n, wantedOld(), nil)
if out.Wants["g14"].FP != fNew {
t.Errorf("the canary g14's want = %+v", out.Wants["g14"])
}
if out.Rules["ace"].Rule != delivery.RuleCanary {
t.Errorf("ace, behind the canary g14, is held back by %+v", out.Rules["ace"])
}
}
// 1297/B4: a failed health check of F holds F and sets the nodes that want it back to their last healthy fingerprint.
func TestAFailedHealthHoldsAndSetsBack(t *testing.T) {
m, n := ready()
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"}
m.Now = time.Date(2026, 10, 25, 1, 0, 0, 0, time.UTC)
out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}, "g14": {FP: fOld, Composed: dig(fOld)}}, map[string]string{"ace": fOld})
// 1906/H3: the held record keeps its cause in words: which machine (its node), which health, when, and the node's words.
if out.Held == nil || out.Held.Node != "ace" || out.Held.Words != "health failed, phase running, seen at 2026-10-25T01:00:00Z: connection refused" {
t.Fatalf("held = %+v", out.Held)
}
if w := out.Wants["ace"]; w.FP != fOld || w.Composed != dig(fOld) || w.Newest != fNew || w.Rule != delivery.RuleNotHeld {
t.Errorf("ace's want = %+v, want set back to the last healthy fingerprint", w)
}
if w, ok := out.Wants["g14"]; ok && w.FP == fNew {
t.Errorf("g14 moved to a held fingerprint: %+v", w)
}
}
// 1297/B4: a module holding data is not set back; its want stays and says why.
func TestADataModuleIsNotSetBack(t *testing.T) {
m, n := ready()
m.Manifest.Data = &catalogue.Data{Own: []catalogue.DataItem{{ID: "mail", Class: catalogue.ClassIrreplaceable}}}
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fNew}}, map[string]string{"ace": fOld})
if out.Held == nil {
t.Fatal("F was not held")
}
if w := out.Wants["ace"]; w.FP != fNew || !strings.Contains(w.Words, "holds data") {
t.Errorf("ace's want = %+v, want it left at F, saying it holds data", w)
}
// 1906/D3: and the node is to be asked whether to restore it
if !slices.Equal(out.Restore, []string{"ace"}) {
t.Errorf("restore asks = %v, want ace", out.Restore)
}
}
// 1906/D3: a module without data is set back and asked about nothing.
func TestAModuleWithoutDataIsNotAskedToRestore(t *testing.T) {
m, n := ready()
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fNew}}, map[string]string{"ace": fOld})
if len(out.Restore) != 0 {
t.Errorf("restore asks = %v", out.Restore)
}
}
// 1297/B4, ADR 0408 §4: a node that leaves the module's nodes gets its want written removed, never deleted; a want that
// is already as the rules give it is not written again.
func TestALeavingNodeIsRemovedAndNothingIsWrittenTwice(t *testing.T) {
m, n := ready()
m.Nodes = []string{"ace"}
out := Wants(m, n, map[string]delivery.Want{"ace": {V: 1, FP: fNew, Composed: dig(fNew)}, "g14": {FP: fOld, Composed: dig(fOld)}}, nil)
if w := out.Wants["g14"]; !w.Removed {
t.Errorf("g14's want = %+v, want removed", w)
}
if _, ok := out.Wants["ace"]; ok {
t.Errorf("ace's want, unchanged, was written again: %+v", out.Wants["ace"])
}
}
// 1906/D2 (ADR 0539 §6): the seventh rule. A want waits for the composition of the module for that node at F, and names its
// digest once it exists; a node with no want before still gets none until then.
func TestAWantWaitsForItsCompositionAndNamesItsDigest(t *testing.T) {
m, n := ready()
have := map[string]bool{}
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
if !have[node+"|"+fp] {
return delivery.ComposedDigest{}, false
}
return composedEverywhere(node, fp)
}
out := Wants(m, n, wantedOld(), nil)
for node, w := range out.Wants {
if w.FP == fNew {
t.Errorf("%s's want moved to F before its composition exists: %+v", node, w)
}
}
if r := out.Rules["ace"]; r.Rule != delivery.RuleComposed || !strings.Contains(r.Words, "ace") {
t.Errorf("ace is held back by %+v, want rule 7 naming the node", r)
}
// and a node that never had a want gets none
if out := Wants(m, n, nil, nil); len(out.Wants) != 0 {
t.Errorf("wants without a composition: %+v", out.Wants)
}
have["ace|"+fNew] = true
out = Wants(m, n, wantedOld(), nil)
if w := out.Wants["ace"]; w.FP != fNew || w.Composed != dig(fNew) {
t.Errorf("ace's want = %+v, want F naming the composition's digest", w)
}
if w, ok := out.Wants["g14"]; ok && w.FP == fNew {
t.Errorf("g14, whose composition does not exist, moved: %+v", w)
}
}
// 1906/D2 (ADR 0539 §5): a new revision of the composition at the same fingerprint reaches the node only through a new want,
// by the same rules: written when they hold, and not while one does not.
func TestARevisionAtTheSameFingerprintIsAWantByTheSameRules(t *testing.T) {
m, n := ready()
revised := "sha256:" + strings.Repeat("9", 64)
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: revised, Judgement: delivery.JudgedNeither}, true
}
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}, "g14": {FP: fNew, Composed: dig(fNew)}}
out := Wants(m, n, running, nil)
for _, node := range []string{"ace", "g14"} {
if w := out.Wants[node]; w.FP != fNew || w.Composed != revised {
t.Errorf("%s's want = %+v, want F naming the revised digest", node, w)
}
}
m.Granted = false
out = Wants(m, n, running, nil)
if w, ok := out.Wants["ace"]; ok && w.Composed == revised {
t.Errorf("a revision was written while a rule holds: %+v", w)
}
// A want held back at an earlier fingerprint follows the new revision of that fingerprint's kept composition, which the
// controller composed again with today's config (novox/hq ADR 0545: on g14 a capacity of 0 after a rollback), while
// the rules still hold F; for a record module whose revision recreates a container, only after the ask.
out = Wants(m, n, wantedOld(), nil)
if w := out.Wants["g14"]; w.FP != fOld || w.Composed != revised || w.Newest != fNew {
t.Errorf("g14, held back at its earlier fingerprint, wants %+v; want it at that fingerprint naming the revised digest", w)
}
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: revised, Judgement: delivery.JudgedRecreates, Policy: delivery.PolicyRecord,
Against: dig(fOld), Words: "recreates"}, true
}
out = Wants(m, n, wantedOld(), nil)
if w := out.Wants["g14"]; w.Composed == revised {
t.Errorf("a recorded module's revision that recreates a container was written with no ask: %+v", w)
}
if !slices.ContainsFunc(out.Record, func(r RecordNeed) bool { return r.Node == "g14" && r.Digest == revised }) {
t.Errorf("the revision was not asked: %+v", out.Record)
}
}
// 1906/D2: a set-back names the composition of the fingerprint it sets back to, and when that is gone the want stays and says so.
func TestASetBackNamesTheOlderCompositionOrStays(t *testing.T) {
m, n := ready()
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "x"}
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}}
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
if fp == fOld {
return delivery.ComposedDigest{}, false
}
return composedEverywhere(node, fp)
}
out := Wants(m, n, running, map[string]string{"ace": fOld})
if w := out.Wants["ace"]; w.FP == fOld {
t.Errorf("set back to a fingerprint whose composition is gone: %+v", w)
}
m.Composed = composedEverywhere
out = Wants(m, n, running, map[string]string{"ace": fOld})
if w := out.Wants["ace"]; w.FP != fOld || w.Composed != dig(fOld) {
t.Errorf("set-back want = %+v", w)
}
}
// movingTo is a module ready to move, whose composition at F is judged j against the one it runs, and whose digest is dig(F).
func movingTo(j delivery.Judgement) (Module, Nodes, map[string]delivery.Want) {
m, n := ready()
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
d := delivery.ComposedDigest{Digest: dig(fp), Judgement: j}
if j != delivery.JudgedNeither {
d.Against, d.Words = dig(fOld), "judged"
}
return d, true
}
return m, n, wantedOld()
}
func recording(m *Module) {
m.Manifest.Upgrade = &catalogue.UpgradePolicy{Policy: catalogue.PolicyRecord, Why: "a person takes each build"}
}
// 1906/D4 (ADR 0539 §14): a module of policy record whose move is judged to recreate a container, to move data, or cannot be
// judged, is not written until its ask is answered yes; the same at a new revision of the same fingerprint.
func TestARecordModuleWaitsForItsAskWhenTheMoveIsNotJudgedNeither(t *testing.T) {
for _, j := range []delivery.Judgement{delivery.JudgedRecreates, delivery.JudgedMovesData, delivery.JudgedCannot} {
m, n, wanted := movingTo(j)
recording(&m)
out := Wants(m, n, wanted, nil)
if w, ok := out.Wants["ace"]; ok {
t.Errorf("%s: ace's want was written without the ask's yes: %+v", j, w)
}
var ace *RecordNeed
for i := range out.Record {
if out.Record[i].Node == "ace" {
ace = &out.Record[i]
}
}
if ace == nil || ace.Digest != dig(fNew) || ace.Judgement != j {
t.Errorf("%s: the asks = %+v, want one for ace naming the digest and the judgement", j, out.Record)
}
// a new revision at the same fingerprint is asked too
same := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fOld)}}
if out := Wants(m, n, same, nil); len(out.Record) == 0 || len(out.Wants) != 0 && out.Wants["ace"].Composed == dig(fNew) {
t.Errorf("%s: a new revision at the same fingerprint: %+v", j, out)
}
m.Allowed = func(node, digest string) bool { return node == "ace" && digest == dig(fNew) }
out = Wants(m, n, wanted, nil)
if w := out.Wants["ace"]; w.FP != fNew || w.Composed != dig(fNew) {
t.Errorf("%s: after the yes ace's want = %+v", j, w)
}
}
}
// 1906/D4: a move judged neither is written at once, whatever the policy; so is a module the node never ran; and a module
// that rolls (not record) is never asked.
func TestAMoveJudgedNeitherOrNeverRunOrNotRecordedIsWrittenAtOnce(t *testing.T) {
m, n, wanted := movingTo(delivery.JudgedNeither)
recording(&m)
if out := Wants(m, n, wanted, nil); out.Wants["ace"].FP != fNew || len(out.Record) != 0 {
t.Errorf("neither: %+v", out)
}
m, n, _ = movingTo(delivery.JudgedRecreates)
recording(&m)
m.Nodes = []string{"ace"} // g14 runs it with no want, so its first want is asked (1945/AC19): this case is ace's alone
delete(n.Haves["ace"], "mail")
if out := Wants(m, n, nil, nil); out.Wants["ace"].FP != fNew || len(out.Record) != 0 {
t.Errorf("a node that never ran it: %+v", out)
}
// but one that runs an older fingerprint with no want yet is asked like any other (one running F is not: N1 below)
m, n, _ = movingTo(delivery.JudgedCannot)
recording(&m)
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if out := Wants(m, n, nil, nil); out.Wants["ace"].FP == fNew || len(out.Record) == 0 {
t.Errorf("a node that runs it with no want: %+v", out)
}
m, n, wanted = movingTo(delivery.JudgedRecreates)
if out := Wants(m, n, wanted, nil); out.Wants["ace"].FP != fNew || len(out.Record) != 0 {
t.Errorf("a rolled module: %+v", out)
}
}
// 1906/D4 (ADR 0539 §7, §15): the bus module's new fingerprint is never written by the assigner, but a revision of its
// composition at the same fingerprint that only changes the user list (judged neither) is written at once.
func TestTheBusIsNotMovedButItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
m, n, _ := movingTo(delivery.JudgedNeither)
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
if out := Wants(m, n, wantedOld(), nil); out.Wants["ace"].FP == fNew {
t.Errorf("the bus moved to a new fingerprint: %+v", out.Wants["ace"])
}
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fOld)}, "g14": {FP: fNew, Composed: dig(fOld)}}
out := Wants(m, n, running, nil)
if w := out.Wants["ace"]; w.FP != fNew || w.Composed != dig(fNew) {
t.Errorf("a user-list revision of the bus: ace's want = %+v", w)
}
if len(out.Record) != 0 {
t.Errorf("a user-list revision was asked: %+v", out.Record)
}
}
// 1906/D4 (ADR 0539 §16): the answer to a failed backup stays in the want while its target stays.
func TestTheBackupAnswerStaysInTheWantWhileItsTargetStays(t *testing.T) {
m, n := ready()
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew), Backup: delivery.BackupSkip, Words: BackupAnswerWords("skip", 0)}}
if w, ok := Wants(m, n, running, nil).Wants["ace"]; ok {
t.Errorf("the answer was erased: %+v", w)
}
}
// 1906/D4 (ADR 0539 §19): a hold released by the operator is not held again by a health check that failed before, and the
// wants move by the rules again.
func TestAReleasedHoldIsNotHeldAgain(t *testing.T) {
m, n := ready()
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "x"}
m.Released = true
out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fOld, Composed: dig(fOld), Newest: fNew, Rule: delivery.RuleNotHeld}}, map[string]string{"ace": fOld})
if out.Held != nil {
t.Errorf("a released fingerprint was held again: %+v", out.Held)
}
if w := out.Wants["ace"]; w.FP != fNew || w.Composed != dig(fNew) || w.Rule != 0 {
t.Errorf("ace's want = %+v, want the held fingerprint written again", w)
}
}
// 1906/D4: a data module whose restore ask the operator answered "restore" is set back like any other, losing the newer writes;
// answered "leave stopped" (or not answered) it stays.
func TestARestoreAnswerSetsADataModuleBack(t *testing.T) {
m, n := ready()
m.Manifest.Data = &catalogue.Data{Own: []catalogue.DataItem{{ID: "mail", Class: catalogue.ClassIrreplaceable}}}
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
running := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}}
m.Restored = func(node string) bool { return false }
if w, ok := Wants(m, n, running, map[string]string{"ace": fOld}).Wants["ace"]; ok && w.FP == fOld {
t.Errorf("set back without the operator's restore: %+v", w)
}
m.Restored = func(node string) bool { return node == "ace" }
if w := Wants(m, n, running, map[string]string{"ace": fOld}).Wants["ace"]; w.FP != fOld || w.Composed != dig(fOld) {
t.Errorf("after restore ace's want = %+v", w)
}
}
// 1906/D4 (ADR 0539 §14): the policy the controller holds beside the composition decides, the operator's own choice included;
// a composition with none (made by hand) falls back to what the manifest derives.
func TestTheRecordPolicyIsTheOneTheControllerHolds(t *testing.T) {
m, n, wanted := movingTo(delivery.JudgedMovesData)
with := func(policy, from string) {
inner := m.Composed
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
d, ok := inner(node, fp)
d.Policy, d.PolicyFrom = policy, from
return d, ok
}
}
with(delivery.PolicyRecord, "person") // a roll module the operator set to record
if out := Wants(m, n, wanted, nil); len(out.Record) == 0 || out.Wants["ace"].FP == fNew {
t.Errorf("the operator's record policy was not honoured: %+v", out)
}
recording(&m) // the manifest says record, the operator chose roll
with("roll", "person")
if out := Wants(m, n, wanted, nil); len(out.Record) != 0 || out.Wants["ace"].FP != fNew {
t.Errorf("the operator's roll policy was not honoured: %+v", out)
}
with("", "") // made by hand: the manifest decides
if out := Wants(m, n, wanted, nil); len(out.Record) == 0 {
t.Errorf("a hand-made composition fell through the manifest's record policy: %+v", out)
}
}
// Review round 1, item 2 (ADR 0539 §15): a node that runs the bus with no readable want compares the new fingerprint with the one
// it runs, from its have: the same fingerprint is not a new one (written by the rules), another is barred.
func TestTheBusWithNoWantIsComparedWithWhatItRuns(t *testing.T) {
m, n, _ := movingTo(delivery.JudgedNeither)
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if w, ok := Wants(m, n, nil, nil).Wants["ace"]; !ok || w.FP != fNew {
t.Errorf("the bus runs F with no want and was not given one: %+v %v", w, ok)
}
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if w, ok := Wants(m, n, nil, nil).Wants["ace"]; ok {
t.Errorf("the bus was moved to a new fingerprint with no want: %+v", w)
}
}
// Review round 1, item 1 (ADR 0539 §7): a revision of the bus module's composition that differs only in its user list is written
// at once, at the fingerprint the node runs, before the record hold, whatever the controller judges of the file's mount (it
// says recreates) and whatever policy the bus has (record, from the bus). The shape is the controller's: judged recreates against
// the composition the node runs, policy record, UserListOnly set.
func TestAUserListOnlyRevisionOfTheBusIsWrittenAtOnceBeforeTheRecordHold(t *testing.T) {
m, n, _ := movingTo(delivery.JudgedRecreates)
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
userList := "sha256:" + strings.Repeat("7", 64)
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
d := delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedRecreates, Against: dig(fOld), Words: "the user list file is mounted",
Policy: delivery.PolicyRecord, PolicyFrom: "the bus"}
if fp == fOld {
d.Digest, d.UserListOnly = userList, true // the composition at the fingerprint the node runs, with the new user list
}
return d, true
}
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
for name, wanted := range map[string]map[string]delivery.Want{
"a want at the fingerprint it runs": {"ace": {FP: fOld, Composed: dig(fOld)}},
"no readable want": nil,
} {
out := Wants(m, n, wanted, nil)
if w := out.Wants["ace"]; w.FP != fOld || w.Composed != userList {
t.Errorf("%s: ace's want = %+v, want the running fingerprint naming the user-list revision", name, w)
}
if len(out.Record) != 0 {
t.Errorf("%s: the user-list revision was asked: %+v", name, out.Record)
}
}
revision := m.Composed
// Issue 1945: a have of the old form (a build, no fingerprint of the new form, no composition) runs no fingerprint a composition is
// kept at. The revision then names F when the controller judged F's composition to differ from the declaration the node last
// applied only in the user list; any other difference is a move of the bus, refused by §15 and said on neither list.
for name, c := range map[string]struct {
userListOnly bool
wanted map[string]delivery.Want
want *delivery.Want // nil: no want written
neither bool
}{
"old form, user list only, no want": {userListOnly: true, want: &delivery.Want{FP: fNew, Composed: userList}},
"old form, user list only, a removed want": {userListOnly: true, wanted: map[string]delivery.Want{"ace": {Removed: true}}, want: &delivery.Want{FP: fNew, Composed: userList}},
"old form, user list only, written before": {userListOnly: true, wanted: map[string]delivery.Want{"ace": {FP: fNew, Composed: userList}}},
"old form, a real move of the bus": {neither: true},
} {
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
if fp != fNew {
return delivery.ComposedDigest{}, false
}
if c.userListOnly {
return delivery.ComposedDigest{Digest: userList, Judgement: delivery.JudgedNeither, UserListOnly: true, Against: dig(fOld),
Words: "only the bus's user list changes, against the declaration ace last applied", Policy: delivery.PolicyRecord}, true
}
return delivery.ComposedDigest{Digest: dig(fNew), Judgement: delivery.JudgedNeither, Words: "the node runs no composition of mail",
Policy: delivery.PolicyRecord}, true
}
n.Haves["ace"]["mail"] = delivery.Have{Build: "build-1792674595610176744", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out := Wants(m, n, c.wanted, nil)
w, ok := out.Wants["ace"]
switch {
case c.want == nil && ok:
t.Errorf("%s: a want was written: %+v", name, w)
case c.want != nil && (!ok || w.FP != c.want.FP || w.Composed != c.want.Composed || w.Removed):
t.Errorf("%s: ace's want = %+v (written %v), want %+v", name, w, ok, *c.want)
}
if len(out.Record) != 0 {
t.Errorf("%s: asked: %+v", name, out.Record)
}
h, on := out.Neither["ace"]
if on != c.neither || (on && (!strings.Contains(h.Words, "§15") || !strings.Contains(h.Words, "more than the bus's user list"))) {
t.Errorf("%s: on neither list %v (%+v), want %v with the words of §15", name, on, h, c.neither)
}
}
// the same flag on a module that is not the bus changes nothing
m.Manifest.Provides, m.Composed = nil, revision
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fOld, Composed: dig(fOld)}}, nil); out.Wants["ace"].Composed == userList {
t.Errorf("a module that is not the bus was written at once: %+v", out.Wants["ace"])
}
}
// Review round 2, N1 (ADR 0539 §9, §11): right after the switch no node has a readable want and every have comes from the old node-engine
// with a fingerprint and no composition, so every move is judged cannot-judge. A node running F with such a have gets its want at F
// with no record hold, for a record module and for the bus alike (the node agent applies nothing when the composition equals the
// declaration it kept); a new fingerprint still waits for its ask.
func TestAHaveFromBeforeTheSwitchGetsItsWantWithNoRecordHold(t *testing.T) {
for name, bus := range map[string]bool{"a record module": false, "the bus": true} {
// only a composition the controller judged unchanged (it equals the declaration the node last applied, ADR 0540 §9) skips the hold
unchanged := func(node, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedUnchanged}, true
}
m, n, _ := movingTo(delivery.JudgedCannot)
recording(&m)
m.Nodes = []string{"ace"}
if bus {
m.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
}
m.Composed = unchanged
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK} // no composition named
out := Wants(m, n, nil, nil)
if w := out.Wants["ace"]; w.FP != fNew || w.Composed != dig(fNew) {
t.Errorf("%s: ace's want = %+v, want F naming its composition", name, w)
}
if len(out.Record) != 0 {
t.Errorf("%s: asked at the switch: %+v", name, out.Record)
}
// a composition that differs from the declaration the node kept (a changed setting or secret), and one the controller did not judge,
// still wait for the ask
for what, j := range map[string]delivery.Judgement{"differing": delivery.JudgedRecreates, "unjudged": delivery.JudgedCannot, "judged neither": delivery.JudgedNeither} {
m2, n2, _ := movingTo(j)
recording(&m2)
m2.Nodes = []string{"ace"}
if bus {
m2.Manifest.Provides = []catalogue.Offer{{Name: "mesh-bus"}}
}
n2.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out := Wants(m2, n2, nil, nil)
if j == delivery.JudgedNeither {
if out.Wants["ace"].FP != fNew || len(out.Record) != 0 {
t.Errorf("%s, %s: %+v", name, what, out)
}
continue
}
if w, ok := out.Wants["ace"]; ok || len(out.Record) == 0 {
t.Errorf("%s, %s: written at the switch with no yes: %+v %+v", name, what, w, out.Record)
}
}
// a have that names a composition is judged as before
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedRecreates, Against: dig(fOld), Words: "judged"}, true
}
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Composed: dig(fOld), Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
if out := Wants(m, n, nil, nil); out.Wants["ace"].FP == fNew && len(out.Record) == 0 {
t.Errorf("%s: a have naming a composition skipped the hold: %+v", name, out)
}
// a new fingerprint still asks (the bus is barred instead)
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out = Wants(m, n, nil, nil)
if w, ok := out.Wants["ace"]; ok {
t.Errorf("%s: a new fingerprint was written with no yes: %+v", name, w)
}
if !bus && len(out.Record) == 0 {
t.Errorf("%s: a new fingerprint was not asked", name)
}
}
}
// 1906/H3: a failed check whose node gave no words is held saying so, never with empty words.
func TestAHoldOfAFailureWithoutWordsSaysSo(t *testing.T) {
m, n := ready()
n.Haves["ace"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
out := Wants(m, n, map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}}, map[string]string{"ace": fOld})
if out.Held == nil || out.Held.Words != "health failed, phase running: the node gave no words" {
t.Fatalf("held = %+v", out.Held)
}
}
// 1906/H2: given a held fingerprint that every node running the module now runs healthy, when the assigner passes, then the
// hold is lifted by itself and the wants move to it by the rules again. A node set back to another fingerprint, a node whose
// check fails, or a hold the operator answered "keep" leaves it held; a node that does not run the module at all does not count.
func TestAHoldIsLiftedOnceEveryNodeRunsItHealthy(t *testing.T) {
everywhere := func() (Module, Nodes) {
m, n := ready()
m.Held = true
n.Haves["g14"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
return m, n
}
m, n := everywhere()
out := Wants(m, n, wantedOld(), map[string]string{"ace": fOld, "g14": fOld})
if !out.Lifted || out.Held != nil {
t.Fatalf("lifted = %v, held = %+v: want the hold lifted", out.Lifted, out.Held)
}
for _, node := range []string{"ace", "g14"} {
if w := out.Wants[node]; w.FP != fNew || w.Rule != 0 {
t.Errorf("%s's want = %+v, want it moved to the lifted fingerprint", node, w)
}
}
m, n = everywhere()
delete(n.Haves["g14"], "mail")
if out := Wants(m, n, wantedOld(), nil); !out.Lifted {
t.Error("a node that does not run the module kept the hold")
}
for name, change := range map[string]func(*Module, *Nodes){
"set back": func(_ *Module, n *Nodes) {
n.Haves["g14"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
},
"failing": func(_ *Module, n *Nodes) {
n.Haves["g14"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
},
"kept": func(m *Module, _ *Nodes) { m.Kept = true },
"no haves": func(_ *Module, n *Nodes) { delete(n.Haves["g14"], "mail"); delete(n.Haves["ace"], "mail") },
"not held": func(m *Module, _ *Nodes) { m.Held = false },
} {
m, n := everywhere()
change(&m, &n)
if out := Wants(m, n, wantedOld(), map[string]string{"ace": fOld, "g14": fOld}); out.Lifted {
t.Errorf("%s: the hold was lifted", name)
}
}
}
// x11 is screen-lock as on g14 and shanks on 2026-10-25 (issue 1945): built, granted and composed at F and at the older fOld
// on both, requiring x11-display, whose provider xorg runs on each node with its health unknown; each node runs screen-lock at fOld
// (health unknown) and holds no want for it. The canary g14 does not run F.
func x11() (Module, Nodes) {
m := Module{Name: "screen-lock", FP: fNew, Built: true, Granted: true, Composed: composedEverywhere, Nodes: []string{"g14", "shanks"},
GrantedAt: func(fp string) bool { return fp == fNew || fp == fOld },
Manifest: catalogue.Manifest{Module: "screen-lock", Requires: []string{"x11-display"}}}
n := Nodes{
Online: map[string]bool{"g14": true, "shanks": true},
Haves: map[string]map[string]delivery.Have{
"g14": {"screen-lock": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown},
"xorg": {FP: fpOf('x'), Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}},
"shanks": {"screen-lock": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown},
"xorg": {FP: fpOf('x'), Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}},
},
Providers: map[string]map[string][]string{"x11-display": {"xorg": {"g14", "shanks"}}},
}
return m, n
}
// 1945/AC19: given a node that runs an older fingerprint of the new form, composed and granted, with no want, held back from F by
// the providers or the canary rule, when the assigner passes, then its first want names the fingerprint it runs with that
// composition, never the newest, and says the newest it is held back from; the canary is never passed over.
func TestANodeRunningAModuleWithNoWantGetsItsFirstWantAtWhatItRuns(t *testing.T) {
m, n := x11()
out := Wants(m, n, nil, nil)
for _, node := range []string{"g14", "shanks"} {
if w := out.Wants[node]; w.FP != fOld || w.Composed != dig(fOld) || w.Newest != fNew || w.Rule == 0 {
t.Errorf("%s's want = %+v, want its first want at the fingerprint it runs, held back from F", node, w)
}
}
if len(out.Neither) != 0 {
t.Errorf("left on neither list: %+v", out.Neither)
}
// with the provider healthy, only the canary holds shanks back: still at what it runs
m, n = x11()
n.Haves["g14"]["xorg"] = delivery.Have{FP: fpOf('x'), Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
n.Haves["shanks"]["xorg"] = n.Haves["g14"]["xorg"]
if w := Wants(m, n, nil, nil).Wants["shanks"]; w.FP != fOld || w.Rule != delivery.RuleCanary {
t.Errorf("shanks held back by the canary with no want: %+v, want at fOld by rule 3", w)
}
// a node already running F gets its want at F; a removed want is no want either
m, n = x11()
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
out = Wants(m, n, map[string]delivery.Want{"g14": {Removed: true}}, nil)
if w := out.Wants["g14"]; w.FP != fNew || w.Composed != dig(fNew) || w.Removed {
t.Errorf("g14 running F with a removed want = %+v, want F", w)
}
}
// 1945/AC19: a node running a fingerprint that is not of the new form, or one whose composition or grant does not exist, gets no
// want past the rules, and is named on neither list; the ordering rules still order every move from a want.
func TestNoFirstWantWithoutTheCompositionAndGrantOfWhatItRuns(t *testing.T) {
for what, change := range map[string]func(*Module, *Nodes){
"an old-form build": func(_ *Module, n *Nodes) {
n.Haves["shanks"]["screen-lock"] = delivery.Have{FP: "build-2", Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
},
"no grant": func(m *Module, _ *Nodes) { m.GrantedAt = func(fp string) bool { return fp == fNew } },
"no composition": func(m *Module, _ *Nodes) {
m.Composed = func(_, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedNeither}, fp == fNew
}
},
} {
m, n := x11()
change(&m, &n)
out := Wants(m, n, nil, nil)
if w, ok := out.Wants["shanks"]; ok {
t.Errorf("%s: shanks got a want: %+v", what, w)
}
if h, ok := out.Neither["shanks"]; !ok || h.Rule == 0 {
t.Errorf("%s: shanks on neither list = %+v (named %v)", what, h, ok)
}
}
m, n := x11()
before := map[string]delivery.Want{"g14": {FP: fOld, Composed: dig(fOld)}, "shanks": {FP: fOld, Composed: dig(fOld)}}
out := Wants(m, n, before, nil)
if len(out.Neither) != 0 {
t.Errorf("nodes holding a want named on neither list: %+v", out.Neither)
}
for node, w := range out.Wants {
if w.FP != fOld {
t.Errorf("%s's want moved from its want past the ordering rules: %+v", node, w)
}
}
}
// 1945/AC20: a new assignment (a node with no have of the module) waiting for the canary, the build or the grant gets no want and is
// never named on neither list: only a node that runs the module, or whose have names it, is; a module with no fingerprint of the new
// form goes with the node's own composition and is never named.
func TestANewAssignmentWaitingIsNotOnNeitherList(t *testing.T) {
m, n := x11()
delete(n.Haves["shanks"], "screen-lock")
out := Wants(m, n, nil, nil)
if w, ok := out.Wants["shanks"]; ok {
t.Errorf("a new assignment waiting for the canary got a want: %+v", w)
}
if _, ok := out.Neither["shanks"]; ok {
t.Errorf("a new assignment waiting for the canary is named on neither list: %+v", out.Neither)
}
m.Built = false
if out := Wants(m, n, nil, nil); len(out.Neither) != 1 || out.Neither["g14"].Rule != delivery.RuleBuildDone {
t.Errorf("with no build: neither = %+v, want g14 alone (it runs the module), by rule 1", out.Neither)
}
m.FP = ""
if out := Wants(m, n, nil, nil); len(out.Neither) != 0 {
t.Errorf("a module of the node's own composition named on neither list: %+v", out.Neither)
}
}
// 1945/AC21: a node whose have names an old-form build (no fingerprint of the new form, so no composition of it) can never be
// named in a want at what it runs; its first want names F, under the canary rule as usual and with everything F needs and the
// providers rule. Given the canary runs F healthy, shanks's first want names F; given it does not yet, shanks gets no want and
// stays on neither list. The old-form node that is itself the canary gets F once its providers run healthy.
func TestAnOldFormNodeGetsItsFirstWantAtFOnlyUnderTheCanary(t *testing.T) {
oldForm := func(n Nodes, node string) {
n.Haves[node]["screen-lock"] = delivery.Have{FP: "build-1792260876956849831", Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
}
providersHealthy := func(n Nodes) {
for _, node := range []string{"g14", "shanks"} {
n.Haves[node]["xorg"] = delivery.Have{FP: fpOf('x'), Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
}
}
// the canary g14 runs F healthy: shanks's first want names F
m, n := x11()
providersHealthy(n)
oldForm(n, "shanks")
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out := Wants(m, n, nil, nil)
if w := out.Wants["shanks"]; w.FP != fNew || w.Composed != dig(fNew) || w.Newest != "" || w.Rule != 0 {
t.Errorf("canary healthy on F: shanks's want = %+v, want F", w)
}
if _, ok := out.Neither["shanks"]; ok {
t.Errorf("canary healthy on F: shanks still on neither list: %+v", out.Neither)
}
// the canary does not run F healthy yet: no want, and shanks stays on neither list by rule 3
m, n = x11()
providersHealthy(n)
oldForm(n, "shanks")
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
out = Wants(m, n, nil, nil)
if w, ok := out.Wants["shanks"]; ok {
t.Errorf("canary not healthy on F: shanks got a want: %+v", w)
}
if h, ok := out.Neither["shanks"]; !ok || h.Rule != delivery.RuleCanary {
t.Errorf("canary not healthy on F: shanks on neither list = %+v (named %v), want rule 3", h, ok)
}
// the canary healthy on F but the providers not healthy on shanks: rule 5 still holds it back
m, n = x11()
oldForm(n, "shanks")
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out = Wants(m, n, nil, nil)
if w, ok := out.Wants["shanks"]; ok {
t.Errorf("providers not healthy: shanks got a want: %+v", w)
}
if h := out.Neither["shanks"]; h.Rule != delivery.RuleProviders {
t.Errorf("providers not healthy: shanks on neither list = %+v, want rule 5", h)
}
// F not granted: nothing, even with the canary healthy
m, n = x11()
providersHealthy(n)
oldForm(n, "shanks")
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
m.Granted = false
if w, ok := Wants(m, n, nil, nil).Wants["shanks"]; ok {
t.Errorf("F not granted: shanks got a want: %+v", w)
}
// the old-form node is the canary itself: F once its providers run healthy
m, n = x11()
providersHealthy(n)
oldForm(n, "g14")
if w := Wants(m, n, nil, nil).Wants["g14"]; w.FP != fNew || w.Composed != dig(fNew) {
t.Errorf("old-form canary: g14's want = %+v, want F", w)
}
}
// A failed health counts only once the node is done moving: phase running or failed, never backing-up, applying or checking.
// The live row on novox at 15:27:36 on 2026-10-25 ("health failed, phase applying" at the controller's new fingerprint) held the
// canary two seconds before the new build was healthy.
func TestAFailedHealthWhileTheNodeMovesHoldsNothing(t *testing.T) {
for _, phase := range []delivery.Phase{delivery.PhaseBackingUp, delivery.PhaseApplying, delivery.PhaseChecking} {
m := Module{Name: "mesh-controller", FP: fNew, Built: true, Granted: true, Composed: composedEverywhere, Nodes: []string{"novox"},
Manifest: catalogue.Manifest{Module: "mesh-controller"}}
n := Nodes{Online: map[string]bool{"novox": true}, Haves: map[string]map[string]delivery.Have{
"novox": {"mesh-controller": {FP: fNew, Phase: phase, Health: delivery.HealthFailed}}}}
before := map[string]delivery.Want{"novox": {FP: fNew, Composed: dig(fNew)}}
if out := Wants(m, n, before, map[string]string{"novox": fOld}); out.Held != nil {
t.Errorf("phase %s, health failed: held %+v, want no hold while the node moves", phase, out.Held)
}
}
for _, phase := range []delivery.Phase{delivery.PhaseRunning, delivery.PhaseFailed} {
m := Module{Name: "mesh-controller", FP: fNew, Built: true, Granted: true, Composed: composedEverywhere, Nodes: []string{"novox"},
Manifest: catalogue.Manifest{Module: "mesh-controller"}}
n := Nodes{Online: map[string]bool{"novox": true}, Haves: map[string]map[string]delivery.Have{
"novox": {"mesh-controller": {FP: fNew, Phase: phase, Health: delivery.HealthFailed}}}}
before := map[string]delivery.Want{"novox": {FP: fNew, Composed: dig(fNew)}}
if out := Wants(m, n, before, map[string]string{"novox": fOld}); out.Held == nil || out.Held.Node != "novox" {
t.Errorf("phase %s, health failed: held %+v, want a hold on novox", phase, out.Held)
}
}
}
// A hold is lifted by itself once every node that runs the module runs F healthy: a have of a node that is not running the module
// (removed, held for its backup, or still moving) is not counted, so it does not keep the hold up for ever; a node running another
// fingerprint, or failed, still does.
func TestAHoldIsLiftedOverHavesThatAreNotRunning(t *testing.T) {
setup := func(phase delivery.Phase, fp string, health delivery.Health) (Module, Nodes) {
m := Module{Name: "web", FP: fNew, Built: true, Granted: true, Held: true, Composed: composedEverywhere, Nodes: []string{"ace", "g14"}}
n := Nodes{Online: map[string]bool{"ace": true, "g14": true}, Haves: map[string]map[string]delivery.Have{
"ace": {"web": {FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}},
"g14": {"web": {FP: fp, Phase: phase, Health: health}}}}
return m, n
}
for _, phase := range []delivery.Phase{delivery.PhaseRemoved, delivery.PhaseHeld, delivery.PhaseApplying, delivery.PhaseBackingUp, delivery.PhaseChecking} {
m, n := setup(phase, fOld, delivery.HealthUnknown)
if out := Wants(m, n, nil, nil); !out.Lifted {
t.Errorf("g14's have in phase %s kept the hold up", phase)
}
}
for what, h := range map[string]delivery.Have{
"running another fingerprint": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK},
"failed": {FP: fNew, Phase: delivery.PhaseFailed, Health: delivery.HealthFailed},
} {
m, n := setup(h.Phase, h.FP, h.Health)
if out := Wants(m, n, nil, nil); out.Lifted {
t.Errorf("g14 %s: the hold was lifted", what)
}
}
// a failed health on a removed have holds nothing
m, n := setup(delivery.PhaseRemoved, fNew, delivery.HealthFailed)
m.Held = false
if out := Wants(m, n, nil, nil); out.Held != nil {
t.Errorf("a removed have with a failed health held F: %+v", out.Held)
}
}
// A module of policy record whose want was removed but which the node still runs gets its first want through the record ask: a
// removed want is no want, so a move the controller judges to recreate or move data waits for the operator's yes.
func TestAFirstWantAfterARemovedWantGoesThroughTheRecordAsk(t *testing.T) {
m, n := x11()
recording(&m)
m.Composed = func(_, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedRecreates, Policy: delivery.PolicyRecord}, true
}
out := Wants(m, n, map[string]delivery.Want{"shanks": {Removed: true}}, nil)
if w, ok := out.Wants["shanks"]; ok {
t.Errorf("shanks's first want after a removed want was written with no record ask: %+v", w)
}
asked := false
for _, r := range out.Record {
asked = asked || (r.Node == "shanks" && r.Digest == dig(fOld) && r.Judgement == delivery.JudgedRecreates)
}
if !asked {
t.Errorf("the record asks = %+v, want one for shanks at the composition it runs", out.Record)
}
}
// 1945/AC23: given a module held at F under rule 2 (the operator holds systemd-resolved on shanks, 2026-10-25), or held by the
// assigner in this pass after a failed health check of F, when a node that runs it holds no want for it, then the node is not on
// neither list: a held module is on the held list, whose hold is said and answered on its own.
func TestAHeldModuleWithNoWantIsNotOnNeitherList(t *testing.T) {
for what, change := range map[string]func(*Module, *Nodes){
"held at F": func(m *Module, _ *Nodes) { m.Held, m.Kept = true, true },
"held at F, running an old-form build": func(m *Module, n *Nodes) {
m.Held, m.Kept = true, true
n.Haves["shanks"]["screen-lock"] = delivery.Have{FP: "build-2", Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
},
"held by this pass": func(_ *Module, n *Nodes) {
n.Haves["g14"]["screen-lock"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed}
n.Haves["shanks"]["screen-lock"] = delivery.Have{FP: "build-2", Phase: delivery.PhaseRunning, Health: delivery.HealthUnknown}
},
} {
m, n := x11()
change(&m, &n)
out := Wants(m, n, nil, nil)
if len(out.Neither) != 0 {
t.Errorf("%s: a held module is on neither list: %+v", what, out.Neither)
}
if out.Rules["shanks"].Rule != delivery.RuleNotHeld {
t.Errorf("%s: shanks is held back by %+v, want rule 2", what, out.Rules["shanks"])
}
}
}
@@ -0,0 +1,62 @@
package manifest
import (
"os"
"reflect"
"slices"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
func load(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatalf("mesh-sdk's manifest reader refuses the manifest: %v", err)
}
return m
}
// 1297/B2a: the controller's reader (mesh-sdk catalogue, which the controller reads manifests with) takes the
// manifest; it reads exactly the buckets the assigner opens (the program's list, held equal in its own test), keeps
// its four states (the differences kept read-only, 1906/D5), and holds a credential of its own.
func TestManifest(t *testing.T) {
m := load(t)
if m.Module != "mesh-delivery-assigner" {
t.Errorf("module = %q", m.Module)
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
states := map[string]int{}
for _, s := range m.State {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "wants": 20}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
t.Errorf("own-secrets = %v, want exactly broker (or no credential is issued)", m.OwnSecrets)
}
if m.SecretsOwner != "mesh-delivery-assigner" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1906/D3: it says its conditions on the operator channel; 1622: it puts its compose asks on the Go build seat, as the
// checker puts its checks.
if !slices.Equal(m.Uses, []string{"node-build-go", "operator-channel"}) {
t.Errorf("uses = %v, want exactly node-build-go and operator-channel", m.Uses)
}
if len(m.Claims) != 0 || len(m.Tools) != 0 || len(m.Invokes) != 0 {
t.Errorf("the assigner claims, serves or calls something: claims %v, tools %v, invokes %v", m.Claims, m.Tools, m.Invokes)
}
}
@@ -0,0 +1,78 @@
{
"module": "mesh-delivery-assigner",
"version": "1",
"slug": "assigner",
"reads": [
"mesh-build-go.steps",
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-controller.composed",
"mesh-controller.grants",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"node-engine.have"
],
"uses": [
"node-build-go",
"operator-channel"
],
"state": [
{ "name": "asks", "history": 5 },
{ "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "wants", "history": 20 }
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-delivery-assigner",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-delivery-assigner",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-delivery-assigner"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-delivery-assigner",
"place": ".",
"unknown": "program"
},
{
"id": "assigner",
"type": "process",
"name": "mesh-delivery-assigner",
"artifact": "program",
"run": [
"./mesh-delivery-assigner"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker"
},
"user": "mesh-delivery-assigner",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "program",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-assigner",
"binary": "mesh-delivery-assigner"
}
]
}
}
@@ -0,0 +1,122 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// bucketOpener is the part of the bus the watches use.
type bucketOpener interface {
KeyValue(ctx context.Context, bucket string) (jetstream.KeyValue, error)
}
// kvStore is the builds bucket as the builder's store: create and compare-and-set.
type kvStore struct{ kv jetstream.KeyValue }
func conflict(err error) error {
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return build.ErrConflict
}
return err
}
func (s kvStore) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
e, err := s.kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, false, nil
}
if err != nil {
return nil, 0, false, err
}
return e.Value(), e.Revision(), true, nil
}
func (s kvStore) Create(ctx context.Context, key string, value []byte) (uint64, error) {
rev, err := s.kv.Create(ctx, key, value)
return rev, conflict(err)
}
func (s kvStore) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
rev, err := s.kv.Update(ctx, key, value, revision)
return rev, conflict(err)
}
func (s kvStore) Keys(ctx context.Context) ([]string, error) {
keys, err := s.kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
func opOf(e jetstream.KeyValueEntry) bool { return e.Operation() == jetstream.KeyValuePut }
// settle reads the entries a watcher holds now, up to its mark of the last values, and returns the watcher for the
// changes that follow.
func settle(ctx context.Context, kv jetstream.KeyValue, each func(jetstream.KeyValueEntry)) (jetstream.KeyWatcher, error) {
w, err := kv.WatchAll(ctx)
if err != nil {
return nil, err
}
for {
select {
case <-ctx.Done():
w.Stop()
return nil, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return nil, errors.New("the watch ended")
}
if e == nil {
return w, nil
}
each(e)
}
}
}
// follow hands each change of a watcher to each until ctx ends; a watch that ends is an error, and the process manager
// starts the program again.
func follow(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.KeyValueEntry)) error {
defer w.Stop()
for {
select {
case <-ctx.Done():
return nil
case e, ok := <-w.Updates():
if !ok {
return errors.New("a watch ended")
}
if e != nil {
each(e)
}
}
}
}
// manifestReader reads manifests.<digest> of the indexer and parses the manifest it holds.
type manifestReader struct{ kv jetstream.KeyValue }
func (m manifestReader) Manifest(ctx context.Context, digest string) (catalogue.Manifest, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return catalogue.Manifest{}, err
}
e, err := m.kv.Get(ctx, key)
if err != nil {
return catalogue.Manifest{}, fmt.Errorf("manifests.%s: %w", key, err)
}
kept, err := delivery.DecodeManifest(e.Value())
if err != nil {
return catalogue.Manifest{}, err
}
return catalogue.ParseManifestForInputs([]byte(kept.JSON))
}
@@ -0,0 +1,185 @@
//go:build integration
package main
import (
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again, by the builder whole against the bus
// (novox/hq issue 1937): on 2026-10-24 the store's 03:30 collection deleted the bare blobs of three done keys the
// controller had never taken in, and the builder kept them done.
// heldVerb answers the controller seat's verb held on the bus, as the serving controller does: {result: {output, ok,
// answer}}; a reference is gone while gone says so.
type heldVerb struct {
mu sync.Mutex
gone func(reference string) bool
asked []string
}
func (h *heldVerb) serve(t *testing.T, b *bed) {
t.Helper()
sub, err := b.js.Conn().Subscribe("mesh.seat.mesh-controller.call.held.>", func(m *nats.Msg) {
var args struct {
References []string `json:"references"`
}
_ = json.Unmarshal(m.Data, &args)
a := delivery.HeldAnswer{Store: "store.test:5100", Held: []string{}, Gone: []string{}}
h.mu.Lock()
for _, r := range args.References {
h.asked = append(h.asked, r)
if h.gone(r) {
a.Gone = append(a.Gone, r)
} else {
a.Held = append(a.Held, r)
}
}
h.mu.Unlock()
reply, _ := json.Marshal(map[string]any{"result": map[string]any{"output": "held", "ok": true, "answer": a}, "node": "ctl"})
_ = m.Respond(reply)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
func (h *heldVerb) asks() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.asked)
}
// eventsConsumer makes EVENTS and the builder's durable consumer of artifacts-gone on it, as the controller makes them
// from the manifest's consumes.
func eventsConsumer(t *testing.T, b *bed, node string) {
t.Helper()
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: eventsStream, Subjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
if _, err := b.js.CreateOrUpdateConsumer(b.ctx, eventsStream, jetstream.ConsumerConfig{Durable: node + "_mesh-delivery-builder",
AckPolicy: jetstream.AckExplicitPolicy, FilterSubjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
}
// doneKey writes builds.<module>.<fp> done, naming one archive at reference, and answers its revision.
func doneKey(t *testing.T, b *bed, module, manifest, reference, digest string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
data, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "n2.mesh-delivery-builder",
Manifest: manifest, Step: "built mesh-build-go on n2",
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Digest: digest, Reference: reference}}})
if err != nil {
t.Fatal(err)
}
rev, err := b.builds.Put(b.ctx, key, data)
if err != nil {
t.Fatal(err)
}
return rev
}
func revisionOf(t *testing.T, b *bed, module string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
e, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
return e.Revision()
}
// 1937/AC3 (integration): the builder, started whole, finds the done key of the trunk's fingerprint naming a blob held
// answers gone, builds it again as a new revision whose step says so, and leaves the rebuilt key done.
func TestADoneKeyWhoseBlobIsGoneIsBuiltAgainAtStart(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
old := digestOf("collected last night")
oldRef := "artifact-store://ca-trust/files/blobs/" + old
rev := doneKey(t, b, "ca-trust", manifest, oldRef, old)
h := &heldVerb{gone: func(r string) bool { return r == oldRef }}
h.serve(t, b)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && len(v.Artifacts) == 1 && v.Artifacts[0].Digest == digestOf("files")
})
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+oldRef) {
t.Errorf("the rebuilt key's step %q does not say why it was built again", v.Step)
}
if got := revisionOf(t, b, "ca-trust"); got <= rev {
t.Errorf("revision %d, was %d: want a new revision", got, rev)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("the archive was built %d times, want once", n)
}
if h.asks() == 0 {
t.Error("the builder never asked held")
}
}
// 1937/AC4 (integration): a done key whose blob held answers held at start stays done; when the controller says
// artifacts-gone for it, the builder asks again and builds it again.
func TestArtifactsGoneLeadsTheBuilderToBuildTheKeyAgain(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
d := digestOf("files")
ref := "artifact-store://ca-trust/files/blobs/" + d
rev := doneKey(t, b, "ca-trust", manifest, ref, d)
var mu sync.Mutex
collected := false
h := &heldVerb{gone: func(r string) bool {
mu.Lock()
defer mu.Unlock()
if r == ref && collected {
collected = false // gone once: the rebuild pushes it again
return true
}
return false
}}
h.serve(t, b)
b.start(t, "n1", nil)
for deadline := time.Now().Add(20 * time.Second); h.asks() == 0 && time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
}
if h.asks() == 0 {
t.Fatal("the builder never asked held at its start")
}
time.Sleep(300 * time.Millisecond)
if got := revisionOf(t, b, "ca-trust"); got != rev || b.fake.count("mesh-build-go.build_go_build") != 0 {
t.Fatalf("a key whose blob is held was built again (revision %d, was %d)", got, rev)
}
mu.Lock()
collected = true
mu.Unlock()
key, _ := delivery.BuilderBuildKey("ca-trust", fp)
body, err := delivery.EncodeArtifactsGone(delivery.ArtifactsGone{Module: "ca-trust", Key: key, Fingerprint: fp,
Gone: []string{ref}, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, delivery.ArtifactsGoneSubject, body); err != nil {
t.Fatal(err)
}
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && strings.Contains(v.Step, "built again")
})
if got := revisionOf(t, b, "ca-trust"); got <= rev || b.fake.count("mesh-build-go.build_go_build") != 1 {
t.Errorf("build = %+v at revision %d (was %d), %d builds: want built again once on the event", v, got, rev,
b.fake.count("mesh-build-go.build_go_build"))
}
}
@@ -0,0 +1,76 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// heldOnTheBus says whether an ask a build key names is on its build seat's stream, or a build node wrote a step for it
// (issue 1910): read once, at the take-over of a key at start, never polled.
type heldOnTheBus struct{ js jetstream.JetStream }
// Held is build.Builder's Held: a step key `<node>.<ask id>` in the build module's steps, or a message of the stream under
// the ask's subject whose Nats-Msg-Id is the ask's id.
func (h heldOnTheBus) Held(ctx context.Context, a delivery.StepAsk) (bool, error) {
module, err := delivery.BuildModuleOf(a.Language)
if err != nil {
return false, err
}
pattern, err := delivery.AskStepWatch(a.ID)
if err != nil {
return false, err
}
steps, err := h.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return false, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
keys, err := steps.ListKeysFiltered(ctx, pattern)
if err != nil {
return false, err
}
for range keys.Keys() {
keys.Stop()
return true, nil // a node took it
}
subject, err := delivery.AskSubject(a.Language, a.Kind)
if err != nil {
return false, err
}
// The seat's stream by its name (delivery.AskStream), as the broker grants the builder its information and its
// messages by sequence and nothing wider (issue 1910).
name, err := delivery.AskStream(a.Language)
if err != nil {
return false, err
}
stream, err := h.js.Stream(ctx, name)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return false, nil
}
if err != nil {
return false, err
}
// A seat's stream is a work queue, which takes no second consumer over its subjects: its messages are read by sequence
// instead, each a direct read that consumes nothing.
info, err := stream.Info(ctx)
if err != nil {
return false, err
}
for seq := info.State.FirstSeq; seq != 0 && seq <= info.State.LastSeq; seq++ {
m, err := stream.GetMsg(ctx, seq)
if errors.Is(err, jetstream.ErrMsgNotFound) {
continue // taken and acknowledged, or removed
}
if err != nil {
return false, err
}
if m.Subject == subject && m.Header.Get(jetstream.MsgIDHeader) == a.ID {
return true, nil
}
}
return false, nil
}
@@ -0,0 +1,823 @@
//go:build integration
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
// controller's user and the builder's own principals <node>.<module> (two instances, for the test of one build).
const busConf = `
listen: 127.0.0.1:-1
jetstream { store_dir: %q }
accounts {
MESH {
jetstream: enabled
users = [
{ user: "ctl", password: "pw" }
{ user: "n1.mesh-delivery-builder", password: "pw" }
{ user: "n2.mesh-delivery-builder", password: "pw" }
]
}
}
`
const (
commit = "7a74cee83f88e6585a5511c3cc53bdf0f5303d20"
fp = "src2:3333333333333333333333333333333333333333333333333333333333333333"
)
// fakeBuildModules are the build modules on the nodes as the real ones run since R4g (ADR 0525; issue 1887 A4): each pulls
// its build seat's asks on its nodes (buildstep.Puller over the seat's workers) and runs an ask's job with step bodies that
// answer as the real tools do, ending the step on <node>.<ask id> of its steps bucket.
type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
failBuild atomic.Bool
// refusePush makes every bundle and archive build fail as a push the artifact store refused (ADR 0498 clause 2).
refusePush atomic.Bool
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
fetches []map[string]any
builds []map[string]any // the arguments of every build call
// closed, while set, keeps every node's capacity at 0: no ask is pulled until open is called (a test of the asks
// waiting in their streams).
closed atomic.Bool
pullers []*buildstep.Puller
}
// open lets the nodes pull again and wakes their pullers.
func (f *fakeBuildModules) open() {
f.closed.Store(false)
f.mu.Lock()
ps := append([]*buildstep.Puller(nil), f.pullers...)
f.mu.Unlock()
for _, p := range ps {
p.Wake()
}
}
func (f *fakeBuildModules) count(tool string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[tool]
}
// digestOf is what every node builds an artifact to.
func digestOf(artifact string) string {
sum := sha256.Sum256([]byte(artifact))
return "sha256:" + hex.EncodeToString(sum[:])
}
var prefixOf = map[string]string{"mesh-build-go": "build_go_", "mesh-build-node": "build_node_", "mesh-build-image": "build_image_"}
// step is one tool's answer, as the real module's.
func (f *fakeBuildModules) step(module, name string, args map[string]any) (any, error) {
tool := prefixOf[module] + name
f.mu.Lock()
f.calls[module+"."+tool]++
if name == "fetch" {
f.fetches = append(f.fetches, args)
} else {
f.builds = append(f.builds, args)
}
f.mu.Unlock()
switch {
case name == "fetch":
merge := ""
if args["base"] != nil { // a fetch with a base answers the merge commit (mesh-build-go FetchResult)
merge = `,"merge":"` + strings.Repeat("9", 40) + `"`
}
return json.RawMessage(`{"workspace":"w` + buildstep.NewID() + `","commit":"` + commit + `"` + merge + `}`), nil
case f.failBuild.Load() && tool != "build_image_copy": // the build fails wherever it runs
return nil, fmt.Errorf("go build of %v: boom", args["artifact"])
case tool == "build_image_copy": // the upstream copy answers the digest and the registry's reference (ADR 0498 clause 4)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"artifact":"` + fmt.Sprint(args["artifact"]) + `","digest":"` + d + `","reference":"registry.example/` + fmt.Sprint(args["module"]) + `/` + fmt.Sprint(args["artifact"]) + `@` + d + `","pushed":true}`), nil
case tool != "build_image_build" && f.refusePush.Load():
return nil, fmt.Errorf("%s%v: the artifact store answered 507 Insufficient Storage", delivery.NotStored, args["artifact"])
case tool != "build_image_build": // a bundle or an archive is pushed and its reference answered (ADR 0498 clause 1)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"artifact-store://` + filepath.Base(fmt.Sprint(args["module"])) + `/` + fmt.Sprint(args["artifact"]) + `/blobs/` + d + `"}`), nil
case tool == "build_image_build" && args["publish"] == true: // a trunk image is pushed: the digest and the reference
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"registry.example/imaged/server@` + d + `"}`), nil
default:
return json.RawMessage(`{"digest":"` + digestOf(fmt.Sprint(args["artifact"])) + `"}`), nil
}
}
// serve makes module's steps bucket and its build seat's stream and workers, and pulls its asks on nodes.
func (f *fakeBuildModules) serve(t *testing.T, module string, nodes ...string) {
kv, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
language := strings.TrimPrefix(module, "mesh-build-")
if err := buildstep.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
call := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) { return f.step(module, name, args) }
}
steps := buildstep.AskSteps{Fetch: call("fetch"), Build: call("build"), Copy: call("copy")}
for _, node := range nodes {
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: node, Put: put,
Capacity: func() int {
if f.closed.Load() {
return 0
}
return 2
}, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
f.mu.Lock()
f.pullers = append(f.pullers, p)
f.mu.Unlock()
go func() { _ = p.Pull(f.ctx) }()
}
}
type bed struct {
ctx context.Context
js jetstream.JetStream
modules jetstream.KeyValue
candidates jetstream.KeyValue
builds jetstream.KeyValue
tools map[string]jetstream.KeyValue
instances jetstream.KeyValue
stop func() // stops the builder started last
fake *fakeBuildModules
srvURL string
dir string
}
// startBed starts the bus, the buckets the controller makes from the builder's and the build modules' manifests, the
// build modules' streams and the builder's durable consumers on them, two nodes' toolchains, the indexer's manifests, and
// the fake build modules. It runs no builder.
func startBed(t *testing.T) *bed { return startBedWith(t, false) }
// startBedWith is startBed with the nodes' capacity 0 from the start when closed: a puller that reads its capacity before a
// test could close the nodes may be in a pull already and takes the first ask put (issue 1928).
func startBedWith(t *testing.T, closed bool) *bed {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "b")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
conf := filepath.Join(dir, "bus.conf")
if err := os.WriteFile(conf, []byte(fmt.Sprintf(busConf, filepath.Join(dir, "js"))), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatal(err)
}
opts.NoLog, opts.NoSigs = true, true
srv, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
ctl, err := nats.Connect(srv.ClientURL(), nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(ctl.Close)
js, err := jetstream.New(ctl)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
t.Cleanup(cancel)
b := &bed{ctx: ctx, js: js, tools: map[string]jetstream.KeyValue{}, srvURL: srv.ClientURL(), dir: dir}
for module, stream := range toolchainStreams {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{"mesh.stream." + module + ".event.toolchain.reported"}}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"n1", "n2"} {
if _, err := js.CreateOrUpdateConsumer(ctx, stream, jetstream.ConsumerConfig{Durable: node + "_mesh-delivery-builder", AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
b.tools[module] = kv
}
mk := func(name string, history int) jetstream.KeyValue {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: uint8(history)})
if err != nil {
t.Fatal(err)
}
return kv
}
b.modules = mk(delivery.ModulesBucket, delivery.ModulesHistory)
b.candidates = mk(delivery.CandidatesBucket, delivery.CandidatesHistory)
mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
b.instances = mk(instancesBucket, 1)
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.closed.Store(closed)
b.fake.serve(t, "mesh-build-go", "n1", "n2")
b.fake.serve(t, "mesh-build-node", "n1", "n2")
b.fake.serve(t, "mesh-build-image", "n1") // the image build module runs on n1 only
b.report(t, "mesh-build-go", "n1", "go", "go1.27.1", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.1", "1")
b.report(t, "mesh-build-node", "n1", "typescript", "tsc5", "1")
b.report(t, "mesh-build-node", "n2", "typescript", "tsc5", "1")
return b
}
// report writes a node's toolchains key of a build module and says so on the module's stream, as the module does.
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper()
data, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: recipe, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.ToolchainKey(node, language)
if _, err := b.tools[module].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, "mesh.stream."+module+".event.toolchain.reported", []byte(`{}`)); err != nil {
t.Fatal(err)
}
}
// module writes modules.<name> as the indexer does for a module with a new-form fingerprint, and its manifest by digest.
func (b *bed) module(t *testing.T, name, fixture, path string) string {
return b.moduleWith(t, name, fixture, path, map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}})
}
// moduleWith is module with the toolchains its fingerprint was computed with.
func (b *bed) moduleWith(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed) string {
return b.moduleCommit(t, name, fixture, path, toolchains, commit)
}
func (b *bed) moduleCommit(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed, at string) string {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", fixture))
if err != nil {
t.Fatal(err)
}
digest := delivery.ManifestDigest(raw)
manifests, err := b.js.KeyValue(b.ctx, delivery.ManifestsBucket)
if err != nil {
t.Fatal(err)
}
mv, err := delivery.EncodeManifest(delivery.Manifest{Module: name, JSON: string(raw)})
if err != nil {
t.Fatal(err)
}
if _, err := manifests.Put(b.ctx, digest, mv); err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: at, FP: "src1:" + strings.Repeat("f", 64), FP2: fp,
Manifest: digest, Path: path, Toolchains: toolchains})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, name, data); err != nil {
t.Fatal(err)
}
return digest
}
// archive writes modules.<name> as the real indexer does for a module of an archive only: its fingerprint names no
// toolchain (mesh-delivery-indexer internal/index inputs.go languagesOf gives an archive no language), so Toolchains is empty.
func (b *bed) archive(t *testing.T, name, path, at string) string {
t.Helper()
return b.moduleCommit(t, name, "catalogue-a67f0196-ca-trust.module.json", path, nil, at)
}
// start runs the program as the builder of a node.
func (b *bed) start(t *testing.T, node string, env map[string]string) {
t.Helper()
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(credential{URL: b.srvURL, Node: node, Module: "mesh-delivery-builder", User: node + ".mesh-delivery-builder", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
e := map[string]string{"MESH_BROKER_FILE": broker}
for k, v := range env {
e[k] = v
}
runCtx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return e[k] }, io.Discard) }()
var once sync.Once
b.stop = func() { once.Do(func() { stop(); <-done }) }
t.Cleanup(b.stop)
}
// await reads the build key until pred holds of it, within 20 s.
func (b *bed) await(t *testing.T, module string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
t.Helper()
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
var last delivery.BuilderBuild
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.builds.Get(b.ctx, key)
if err != nil {
continue
}
if last, err = delivery.DecodeBuilderBuild(e.Value()); err == nil && pred(last) {
return last
}
}
t.Fatalf("builds.%s = %+v within 20 s", key, last)
return last
}
func isState(state string) func(delivery.BuilderBuild) bool {
return func(v delivery.BuilderBuild) bool { return v.State == state }
}
// TestBuilderMakesADoneKeyWithTheDigests: modules.netcheck (a Go and a TypeScript bundle) and its manifest are in the
// indexer's buckets, and two nodes report each build module's toolchain; the program writes builds.netcheck.<fp> done,
// the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4).
func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
b := startBed(t)
digest := b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
for _, a := range v.Artifacts {
if a.Digest != digestOf(a.Name) || a.Kind != "bundle" || a.Reference != "artifact-store://netcheck/"+a.Name+"/blobs/"+a.Digest {
t.Errorf("artifact = %+v, want the digest every node built and the reference its push answered (ADR 0498 clause 1)", a)
}
}
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_build": 1, "mesh-build-node.build_node_fetch": 1, "mesh-build-node.build_node_build": 1} {
if got := b.fake.count(tool); got != want {
t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want)
}
}
}
// TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey: two builders on two nodes start together on one module key of an
// archive only (no toolchain named, as the indexer writes it); the second takes over the key the first holds building
// (ADR 0478 clause 6), and the key ends done, written once.
func TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey(t *testing.T) { // 1607/AC7
b := startBed(t)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
b.start(t, "n2", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || v.Artifacts[0].Digest != digestOf("files") {
t.Fatalf("build = %+v, want the archive's digest", v)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n < 1 { // each instance that built, on one node
t.Errorf("build_go_build was called %d times, want one for each instance that built", n)
}
}
// TestAnArchiveOnlyModuleNamingNoToolchainIsBuilt: the indexer gives an archive no language, so modules.ca-trust has no
// toolchains; the builder builds it on the build module's nodes all the same (review round 4).
func TestAnArchiveOnlyModuleNamingNoToolchainIsBuilt(t *testing.T) {
b := startBed(t)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if len(v.Toolchains) != 0 || len(v.Artifacts) != 1 || strings.HasPrefix(v.Step, "one node (") {
t.Errorf("build = %+v, want done naming no toolchain", v)
}
}
// TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint: a build step fails, so the key is
// failed naming it; the indexer writing modules.<m> again with the same fingerprint wakes the builder, which builds
// the key again as a new revision of the same key, now done.
func TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint(t *testing.T) { // 1607/AC6, AC7
b := startBed(t)
b.fake.failBuild.Store(true)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildFailed))
if !strings.Contains(v.Error, " on n") || !strings.Contains(v.Error, "boom") {
t.Errorf("build = %+v, want failed naming the node and the step's reason", v)
}
key, _ := delivery.BuilderBuildKey("ca-trust", fp)
failed, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
b.fake.failBuild.Store(false)
// The indexer writes the module again (here at a later commit), with the same fingerprint.
b.archive(t, "ca-trust", "modules/ca-trust", "89abcdef0123456789abcdef0123456789abcdef")
b.await(t, "ca-trust", isState(delivery.BuildDone))
done, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
if done.Revision() <= failed.Revision() {
t.Errorf("revision %d, was %d: the key was not written as a new revision", done.Revision(), failed.Revision())
}
}
// TestAToolchainChangeDoesNotBuildTheOldFingerprint: the fingerprint names go1.27.1, the nodes then agree on go1.27.2 (a
// report of both nodes), and the builder writes no key for the old fingerprint although it was woken by the report; a
// module value naming go1.27.2 (the new fingerprint, computed by the indexer) is built.
func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/AC4
b := startBed(t)
b.report(t, "mesh-build-go", "n1", "go", "go1.27.2", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.2", "1")
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // names go1.27.1
b.start(t, "n1", nil)
// Another module, whose fingerprint names the new toolchain, is built; by then the old one has been looked at.
b.moduleCommit(t, "netcheck2", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck",
map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.2", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}, commit)
b.await(t, "netcheck2", isState(delivery.BuildDone))
if _, err := b.builds.Get(b.ctx, mustKey(t, "netcheck")); err == nil {
t.Error("the old fingerprint, naming go1.27.1, was built with go1.27.2")
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("build_go_build was called %d times, want 1 (the new fingerprint on one node only)", n)
}
}
// TestARootModuleIsBuilt: a module whose path is "." is given to the build tools as it is.
func TestARootModuleIsBuilt(t *testing.T) { // 1615/AC4
b := startBed(t)
b.archive(t, "ca-trust", delivery.RootPath, commit)
b.start(t, "n1", nil)
b.await(t, "ca-trust", isState(delivery.BuildDone))
}
func mustKey(t *testing.T, module string) string {
k, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
return k
}
// moduleAt is module with the go toolchain version the fingerprint names and the commit it was computed at.
func (b *bed) moduleAt(t *testing.T, name, fixture, path, goVersion, at string) {
t.Helper()
b.moduleCommit(t, name, fixture, path, map[string]delivery.ToolchainUsed{"go": {Version: goVersion, Recipe: "1"}}, at)
}
// TestAKeyLeftBuildingForAModuleThatIsGoneIsClosedAtStart: the builds bucket holds a key left building by another node for
// a module the indexer no longer has; the program, at start, fails it naming why (issue 1660, ADR 0478 clause 6).
func TestAKeyLeftBuildingForAModuleThatIsGoneIsClosedAtStart(t *testing.T) {
b := startBed(t)
data, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "n2.mesh-delivery-builder", Manifest: strings.Repeat("a", 64)})
if err != nil {
t.Fatal(err)
}
if _, err := b.builds.Put(b.ctx, mustKey(t, "ghost"), data); err != nil {
t.Fatal(err)
}
b.start(t, "n1", nil)
v := b.await(t, "ghost", isState(delivery.BuildFailed))
if !strings.Contains(v.Error, "no longer has this fingerprint") || !strings.Contains(v.Error, "n2.mesh-delivery-builder") {
t.Errorf("build = %+v, want it closed naming who left it", v)
}
}
// TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk: the trunk's record of netcheck carries a fingerprint of its
// own; a candidate the indexer wrote gives netcheck the fingerprint fp at a pull request's head; the program writes
// builds.netcheck.<fp> done, its fetch given the head, the ref that carries it and the trunk commit (issue 1684).
func TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1
b := startBed(t)
other := "src2:" + strings.Repeat("e", 64)
digest := b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: commit, FP: "src1:" + strings.Repeat("f", 64), FP2: other,
Manifest: digest, Path: "modules/netcheck", Toolchains: map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, "netcheck", data); err != nil {
t.Fatal(err)
}
const head = "1111111111111111111111111111111111111111"
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"netcheck"},
Fps: map[string]delivery.CandidateModule{"netcheck": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, err := delivery.CandidatesKey("feat/x")
if err != nil {
t.Fatal(err)
}
if _, err := b.candidates.Put(b.ctx, ckey, cand); err != nil {
t.Fatal(err)
}
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
if len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
b.fake.mu.Lock()
defer b.fake.mu.Unlock()
found := false
for _, f := range b.fake.fetches {
if f["commit"] == head && f["ref"] == "refs/pull/7/head" && f["base"] == commit {
found = true
}
}
if !found {
t.Errorf("fetches = %v, want one of the candidate's head with its ref and the trunk commit", b.fake.fetches)
}
}
// TestAnImageIsBuiltOnOneNodeAndOnlyATrunkKeyPublishes: modules.imaged (one image artifact) is built on n1, where
// mesh-build-image runs: a trunk record's key is done with the digest and the reference, its build call carrying publish; a
// candidate's key for the same module is built first with no publish and no reference; when the pull request is merged and the
// trunk's record names the candidate's fingerprint, the key is built again with publish and ends with a reference
// (ADR 0487 clauses 5 and 12).
func TestAnImageIsBuiltOnOneNodeAndOnlyATrunkKeyPublishes(t *testing.T) { // 1705
b := startBed(t)
other := "src2:" + strings.Repeat("e", 64)
digest := b.moduleCommit(t, "imaged", "catalogue-a67f0196-imaged.module.json", "modules/imaged", nil, commit)
// The trunk carries `other`; its key is built and published.
trunk := func(fp2 string) {
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: commit, FP: "src1:" + strings.Repeat("f", 64), FP2: fp2,
Manifest: digest, Path: "modules/imaged"})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, "imaged", data); err != nil {
t.Fatal(err)
}
}
trunk(other)
b.start(t, "n1", nil)
okey, _ := delivery.BuilderBuildKey("imaged", other)
awaitKey := func(key string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
var last delivery.BuilderBuild
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.builds.Get(b.ctx, key); err == nil {
if last, err = delivery.DecodeBuilderBuild(e.Value()); err == nil && pred(last) {
return last
}
}
}
t.Fatalf("builds.%s = %+v within 20 s", key, last)
return last
}
v := awaitKey(okey, isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Digest != digestOf("server") || !strings.HasSuffix(v.Artifacts[0].Reference, "@"+digestOf("server")) {
t.Fatalf("trunk build = %+v, want the digest and the reference", v)
}
// The candidate carries fp: built, not published.
const head = "1111111111111111111111111111111111111111"
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"imaged"},
Fps: map[string]delivery.CandidateModule{"imaged": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
if _, err := b.candidates.Put(b.ctx, ckey, cand); err != nil {
t.Fatal(err)
}
v = b.await(t, "imaged", isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Reference != "" {
t.Fatalf("candidate build = %+v, want done with no reference", v)
}
b.fake.mu.Lock()
if len(b.fake.builds) != 2 || b.fake.builds[0]["publish"] != true || b.fake.builds[1]["publish"] != nil {
t.Errorf("builds = %v, want the trunk's with publish, then the candidate's with none", b.fake.builds)
}
b.fake.mu.Unlock()
// The pull request is merged: the trunk's record names fp, and its key ends published, built once more with publish.
trunk(fp)
v = b.await(t, "imaged", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && len(v.Artifacts) == 1 && v.Artifacts[0].Reference != ""
})
if v.Artifacts[0].Digest != digestOf("server") {
t.Errorf("merged build = %+v", v)
}
b.fake.mu.Lock()
defer b.fake.mu.Unlock()
published := 0
for _, a := range b.fake.builds {
if a["publish"] == true {
published++
}
}
if len(b.fake.builds) != 3 || published != 2 {
t.Errorf("builds = %v, want three, two of them with publish (the trunk's two fingerprints)", b.fake.builds)
}
}
// TestEveryDoneKeyNamesItsReferencesAndARefusedPushIsBuiltAgain: with the artifact store refusing pushes, the key of an
// archive module ends failed opening "not stored:"; once it takes them, a new modules value builds the key again, done, its
// every artifact naming a reference (issue 1587, B8, ADR 0498 clauses 1, 2 and 8).
func TestEveryDoneKeyNamesItsReferencesAndARefusedPushIsBuiltAgain(t *testing.T) {
b := startBed(t)
b.fake.refusePush.Store(true)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildFailed))
if !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "507") {
t.Fatalf("build = %+v, want failed opening %q with the store's words", v, delivery.NotStored)
}
b.fake.refusePush.Store(false)
b.archive(t, "ca-trust", "modules/ca-trust", "89abcdef0123456789abcdef0123456789abcdef")
v = b.await(t, "ca-trust", isState(delivery.BuildDone))
if ok, missing := v.Referenced(); !ok {
t.Errorf("build = %+v, artifacts naming no reference: %v", v, missing)
}
}
// TestAPinnedUpstreamIsCopiedByTheImageBuildModule: hello-web's manifest (captured from mesh-catalog 49caf17f) names an
// upstream image pinned by its digest and an archive; the upstream is copied on n1, where mesh-build-image runs, with no
// fetch of the image module, and the key ends done with both references (issue 1587, B8, ADR 0498 clause 4).
func TestAPinnedUpstreamIsCopiedByTheImageBuildModule(t *testing.T) {
b := startBed(t)
b.moduleCommit(t, "hello-web", "catalogue-49caf17f-hello-web.module.json", "modules/hello-web", nil, commit)
b.start(t, "n1", nil)
v := b.await(t, "hello-web", isState(delivery.BuildDone))
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, artifacts naming no reference: %v", v, missing)
}
if n := b.fake.count("mesh-build-image.build_image_copy"); n != 1 {
t.Errorf("build_image_copy called %d times, want once", n)
}
if n := b.fake.count("mesh-build-image.build_image_fetch"); n != 0 {
t.Errorf("build_image_fetch called %d times, want none for a copy", n)
}
}
// TestAManifestFallingBackToAPredefinedKeyIsBuilt: claude-code's manifest (mesh-catalog 835bf22b) has a config value falling
// back to a predefined key; the builder, which holds no published list of them, reads it and builds the key done (issue
// 1751, mesh-sdk catalogue.ParseManifestForInputs).
func TestAManifestFallingBackToAPredefinedKeyIsBuilt(t *testing.T) {
b := startBed(t)
b.module(t, "claude-code", "catalogue-835bf22b-claude-code.module.json", "modules/claude-code")
b.start(t, "n1", nil)
v := b.await(t, "claude-code", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone || v.State == delivery.BuildFailed
})
if v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done: the manifest is read", v)
}
}
// 1886/AC5: a builder on n1 whose Go, Node and image build modules each answered a reference says under n1, in its bucket
// of instances, that it writes references, and takes the say away when it stops: the controller delivers from its keys
// only while it holds.
func TestABuilderWhoseBuildModulesAnsweredReferencesSaysItAndStoppedSaysNothing(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // a Go and a TypeScript bundle
b.moduleCommit(t, "imaged", "catalogue-a67f0196-imaged.module.json", "modules/imaged", nil, commit) // an image, for the trunk
b.start(t, "n1", nil)
var said string
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && said == ""; time.Sleep(50 * time.Millisecond) {
if e, err := b.instances.Get(b.ctx, "n1"); err == nil {
said = string(e.Value())
}
}
if !strings.Contains(said, `"writes":["references"]`) {
t.Fatalf("the builder said %q", said)
}
b.stop()
if _, err := b.instances.Get(b.ctx, "n1"); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("the stopped builder still says: %v", err)
}
}
// 1887/AC1, AC2 on the real bus: the builder puts the trunk fingerprint's build as asks on the build seats' delivery
// streams and calls no node's tool; the build modules that pulled them wrote the steps, and the key is done naming the
// references.
func TestTheBuilderAsksTheBuildSeatsAndCallsNoNode(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
var calls atomic.Int64
s, err := sub.Subscribe("mesh.mod.*.call.>", func(*nats.Msg) { calls.Add(1) })
if err != nil {
t.Fatal(err)
}
defer s.Unsubscribe()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
for _, a := range v.Artifacts {
if a.Reference == "" {
t.Errorf("artifact %s names no reference", a.Name)
}
}
if n := calls.Load(); n != 0 {
t.Errorf("%d tool calls reached a build module", n)
}
// 1887/AC1: one ask per build module the build needs (netcheck: a Go and a TypeScript bundle), each on its seat's
// delivery stream.
got := asked()
if len(got) != 2 {
t.Fatalf("asks put = %+v, want one Go and one Node delivery ask", got)
}
for _, a := range got {
if a.Kind != delivery.AskDelivery || a.Build == nil || a.Build.FP != fp {
t.Errorf("ask %+v, want a delivery ask of the fingerprint", a)
}
}
}
// asksPut hears every ask put on a build seat's accept subjects (a core subscription beside the stream) and answers the
// asks heard so far.
func (b *bed) asksPut(t *testing.T, nc *nats.Conn) func() []delivery.StepAsk {
t.Helper()
var mu sync.Mutex
var asks []delivery.StepAsk
s, err := nc.Subscribe("mesh.seat.*.accept.>", func(m *nats.Msg) {
if a, err := delivery.DecodeStepAsk(m.Data); err == nil {
mu.Lock()
asks = append(asks, a)
mu.Unlock()
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Unsubscribe() })
return func() []delivery.StepAsk {
mu.Lock()
defer mu.Unlock()
return append([]delivery.StepAsk(nil), asks...)
}
}
// 1887/AC3: a builder that started again while its asks waited in the streams (no node had a free slot) puts no second
// ask: it waits for the asks its building key names, and the key is done once a node took them.
func TestARestartedBuilderPutsNoSecondAsk(t *testing.T) {
b := startBedWith(t, true) // 1928/AC1: no node can take an ask from the start, so no pull in hand takes the first one
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", func(v delivery.BuilderBuild) bool { return v.State == delivery.BuildBuilding && len(v.Asks) == 2 })
for deadline := time.Now().Add(20 * time.Second); len(asked()) < 2; time.Sleep(20 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("asks put = %+v within 20 s, want the two the key names", asked())
}
}
b.stop()
b.start(t, "n1", nil) // the same builder, started again, while both asks wait
b.fake.open()
done := b.await(t, "netcheck", isState(delivery.BuildDone))
if got := asked(); len(got) != 2 {
t.Fatalf("asks put = %d (%+v), want the two the key named and none again", len(got), got)
}
for lang, id := range v.Asks {
found := false
for _, a := range asked() {
found = found || a.ID == id
}
if !found {
t.Errorf("the %s ask %s the key named was never put", lang, id)
}
}
if len(done.Artifacts) != 2 {
t.Errorf("done = %+v", done)
}
}
@@ -0,0 +1,284 @@
// Command mesh-delivery-builder is delivery's builder (novox/hq ADR 0453 clause 4, ADR 0478, issue 1607): it builds every
// new-form fingerprint the indexer writes, dark, into its own build keys (builds.<module>.<fingerprint>), through the build
// modules' fetch and build tools. Nothing reads its keys to deliver until release R4e. It serves no request subject and
// holds no timer: every reaction is a watch or a bus event (ADR 0437 §8).
package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"io"
"log"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build"
)
const (
// defaultStepBound is how long one ask's step may run once a build node took it (a fetch downloads the Go modules, a
// build compiles): from its start, never from its put, so an ask waiting in its stream for a free slot is not failed
// (buildrun.Asker.Bound; ADR 0525 §3, §6).
defaultStepBound = 30 * time.Minute
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-builder:", err)
os.Exit(1)
}
}
// credential is the broker file the controller issues to the module: JSON of the bus address, the certificate
// fingerprint to pin, the principal and its password (as the indexer reads it; issue 1242 moves this into the SDK).
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
func readCredential(path string) (credential, error) {
raw, err := os.ReadFile(path)
if err != nil {
return credential{}, err
}
var c credential
if err := json.Unmarshal(raw, &c); err != nil {
return credential{}, fmt.Errorf("the broker credential is not one: %w", err)
}
if c.URL == "" || c.User == "" {
return credential{}, errors.New("the broker credential names no bus address or user")
}
return c, nil
}
// pinned accepts exactly the certificate with this SHA-256 and no other.
func pinned(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
if got := "sha256:" + hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%.23s…)", got)
}
return nil
},
}
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-builder: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to hear the bus with")
}
cred, err := readCredential(path)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
nats.CustomInboxPrefix("_INBOX." + cred.User),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
bucket := func(name string) (jetstream.KeyValue, error) {
kv, err := js.KeyValue(ctx, name)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", name, err)
}
return kv, nil
}
builds, err := bucket(delivery.BuilderBuildsBucket)
if err != nil {
return err
}
modules, err := bucket(delivery.ModulesBucket)
if err != nil {
return err
}
manifests, err := bucket(delivery.ManifestsBucket)
if err != nil {
return err
}
candidates, err := bucket(delivery.CandidatesBucket)
if err != nil {
return err
}
bound := defaultStepBound
if v := strings.TrimSpace(getenv("MESH_BUILDER_STEP_BOUND")); v != "" {
if bound, err = time.ParseDuration(v); err != nil {
return fmt.Errorf("MESH_BUILDER_STEP_BOUND: %w", err)
}
}
tools := &buildrun.ToolchainSet{}
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
asks := buildrun.JetStreamAsks{JS: js}
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
calls, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer calls.Close()
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: bound},
Held: heldOnTheBus{js},
Toolchains: tools,
Manifests: manifestReader{manifests},
Logf: logger.Printf,
Stored: storedVia(calls.Call),
}
// **Said once every build module answered a reference, taken away when stopped** (issue 1886, ADR 0522): the
// controller delivers from this builder's keys only while an instance says it writes references. A bucket not there
// yet (the manifest's state not made) is said, and the controller's plan waits for the keys, named (no build-agent
// is asked since R4g, ADR 0524 §7).
switch kv, err := js.KeyValue(ctx, instancesBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: the controller is not told that this builder writes references until it starts again", instancesBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", instancesBucket, err)
default:
says := newSayer(ctx, kv, cred.Node, []string{buildrun.GoBuildModule, buildrun.NodeBuildModule, build.ImageBuildModule}, logger.Printf)
b.Answered = says.Answered
defer func() {
stopping, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if err := says.unsay(stopping); err != nil {
logger.Printf("%v", err)
}
}()
}
// The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, b.NodesChanged, logger)
if err != nil {
return err
}
if err := watch.start(); err != nil {
return err
}
// A build module says each time it wrote its toolchains key: its stream, and the builder's durable consumer on it, are
// made by the controller. Where the stream is not there yet the builder says so and is started again with the build
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := js.Consumer(ctx, stream, cred.Node+"_"+cred.Module)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the builder starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := cons.Consume(func(m jetstream.Msg) { watch.Reported(m) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
if err != nil {
return err
}
defer cc.Stop()
}
// The controller says artifacts-gone when its take-in found a key's blob gone (issue 1937): the builder's durable
// consumer on EVENTS, made by the controller from the manifest's consumes. Where it is not there yet the builder says
// so; it still asks held at its start and at every module value.
switch cons, err := js.Consumer(ctx, eventsStream, cred.Node+"_"+cred.Module); {
case err != nil:
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s is not heard until the builder starts again: %v",
cred.Node, cred.Module, eventsStream, delivery.ArtifactsGoneEvent, err)
default:
cc, err := cons.Consume(func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
if err != nil {
return err
}
defer cc.Stop()
}
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
b.Module(ctx, e.Key(), e.Value())
} else {
b.Forget(e.Key())
}
}
w, err := settle(ctx, modules, each)
if err != nil {
return err
}
// A key left building for a fingerprint its module no longer has is closed (issue 1660); the modules are all known now.
if err := b.Sweep(ctx); err != nil {
return err
}
// The fingerprints the indexer gives the candidates' modules are built too (issue 1684), once the trunk's modules are known.
eachCandidate := func(e jetstream.KeyValueEntry) {
if opOf(e) {
b.Candidate(ctx, e.Key(), e.Value())
} else {
b.ForgetCandidate(e.Key())
}
}
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
return err
}
logger.Printf("started on %s", cred.Node)
errs := make(chan error, 2)
go func() { errs <- follow(ctx, w, each) }()
go func() { errs <- follow(ctx, cw, eachCandidate) }()
select {
case <-ctx.Done():
b.Wait()
return nil
case err := <-errs:
return err
case err := <-assignErrs:
return err
case err := <-watch.errs:
return err
}
}
@@ -0,0 +1,79 @@
package main
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
)
// What the builder says it writes (novox/hq issue 1886, ADR 0522): each running instance keeps a key, named by its node,
// in its own bucket of instances (`mesh-delivery-builder_instances`, the manifest's state `instances`), saying that every
// done key it writes names where its artifacts are (ADR 0498 §1). The references come from the build modules' own pushes,
// so an instance says it only once a step of every build module it calls answered one since it started: a builder beside
// a build module from before R4e says nothing. The controller delivers from the builder's keys only while an instance
// says so; since R4g it asks no build-agent, and a module whose key is not yet done waits, named (ADR 0524 §7). The bucket is read by the controller alone, so its name and shape
// are not in mesh-sdk.
const (
instancesBucket = "mesh-delivery-builder_instances"
writesReferences = `{"v":1,"writes":["references"]}`
)
// sayStore is the bucket of instances as the builder writes it.
type sayStore interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
Delete(ctx context.Context, key string, opts ...jetstream.KVDeleteOpt) error
}
// sayer puts this instance's say once every build module in need answered a reference, and takes it away at the stop.
type sayer struct {
ctx context.Context
store sayStore
node string
need []string
logf func(string, ...any)
mu sync.Mutex
answered map[string]bool
said bool
}
func newSayer(ctx context.Context, store sayStore, node string, need []string, logf func(string, ...any)) *sayer {
return &sayer{ctx: ctx, store: store, node: node, need: need, logf: logf, answered: map[string]bool{}}
}
// Answered is told the build module whose step answered a reference (build.Builder.Answered).
func (s *sayer) Answered(module string) {
s.mu.Lock()
defer s.mu.Unlock()
s.answered[module] = true
if s.said {
return
}
for _, m := range s.need {
if !s.answered[m] {
return
}
}
if _, err := s.store.Put(s.ctx, s.node, []byte(writesReferences)); err != nil {
s.logf("saying in %s that this builder writes references: %v; said again at the next reference", instancesBucket, err)
return
}
s.said = true
s.logf("every build module answered a reference: said in %s that this builder writes references", instancesBucket)
}
// unsay takes the say away, if it was made: a builder replaced by one that does not write references leaves none.
func (s *sayer) unsay(ctx context.Context) error {
s.mu.Lock()
defer s.mu.Unlock()
if !s.said {
return nil
}
if err := s.store.Delete(ctx, s.node); err != nil {
return fmt.Errorf("taking away the say in %s: %w", instancesBucket, err)
}
s.said = false
return nil
}
@@ -0,0 +1,46 @@
package main
import (
"context"
"encoding/json"
"testing"
"github.com/nats-io/nats.go/jetstream"
)
type saysOf map[string]string
func (s saysOf) Put(_ context.Context, key string, value []byte) (uint64, error) {
s[key] = string(value)
return 1, nil
}
func (s saysOf) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(s, key)
return nil
}
// 1886/AC5: a builder says, under its node, that it writes references only once every build module it calls answered
// one; stopped, it says nothing any more.
func TestTheBuilderSaysItWritesReferencesOnceEveryBuildModuleAnsweredOne(t *testing.T) {
s := saysOf{}
say := newSayer(t.Context(), s, "shanks", []string{"mesh-build-go", "mesh-build-node", "mesh-build-image"}, t.Logf)
say.Answered("mesh-build-go")
say.Answered("mesh-build-node")
if v, ok := s["shanks"]; ok {
t.Fatalf("said before the image build module answered a reference: %q", v)
}
say.Answered("mesh-build-image")
var v struct {
Writes []string `json:"writes"`
}
if err := json.Unmarshal([]byte(s["shanks"]), &v); err != nil || len(v.Writes) != 1 || v.Writes[0] != "references" {
t.Fatalf("the builder's say: %q %v", s["shanks"], err)
}
if err := say.unsay(t.Context()); err != nil {
t.Fatal(err)
}
if _, ok := s["shanks"]; ok {
t.Fatalf("the stopped builder still says: %v", s)
}
}
@@ -0,0 +1,174 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node; it is passed on as deleted otherwise,
// and passed on again when the module is assigned there again. Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]bool // <node>.<module> with an open assignment
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]bool // <module>/<key> said as left out, so it is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]bool{},
raw: map[string]map[string][]byte{}, left: map[string]bool{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in and answers whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
open := !deleted && openAssignment(value)
g.mu.Lock()
defer g.mu.Unlock()
if g.standing[key] == open {
return false
}
if open {
g.standing[key] = true
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned (or the assignments are not known), deleted
// otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
if err != nil || !g.known || g.standing[node+"."+module] {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if !g.left[module+"/"+key] {
g.left[module+"/"+key] = true
g.logf("%s.%s is left out of the agreed toolchain: %s has no assignment on %s, so it is no builder (issue 1940)",
delivery.ToolchainsBucket(module), key, module, node)
}
return g.forward(module, key, nil, true)
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands: one that has not
// ended and has no state (a pending one is not on the machine yet; mesh-controller internal/chain Assignment).
func openAssignment(value []byte) bool {
var a struct {
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return false
}
return a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e)) }
w, err := settle(ctx, kv, each)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", provider.AssignmentsBucket, err)
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e)) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -0,0 +1,112 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false)
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false)
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true)
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
}
// A newly assigned build node that has not reported yet changes nothing; its report counts as it comes.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
set := &buildrun.ToolchainSet{}
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(string, ...any) {})
gate.Settled()
if gate.Assignment("g14.mesh-build-node", []byte(`{}`), false) {
t.Fatal("an assignment with no report passed something on")
}
key, value := gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an assigned node's report was not passed on")
}
key, value = gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("a report of a node with no assignment key was passed on after the assignments settled")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
}
@@ -0,0 +1,61 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// eventsStream is the stream every module's and role's events are on, where the controller makes the builder's durable
// consumer of artifacts-gone (mesh-controller internal/broker EventsStream).
const eventsStream = "EVENTS"
// toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call).
type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the controller's verb held (mesh-sdk delivery.HeldVerb; novox/hq issue
// 1937) and answers the references the store no longer holds. One it could not ask about, with none gone, is an error:
// the key is then left done, and the controller's take-in still refuses a gone blob.
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) {
raw, err := call(delivery.HeldVerb, map[string]any{"references": references})
if err != nil {
return nil, fmt.Errorf("asking %s: %w", delivery.HeldVerb, err)
}
a, err := delivery.DecodeHeldAnswer(raw, references)
if err != nil {
return nil, err
}
if len(a.Gone) == 0 && len(a.Unknown) > 0 {
var why []string
for r, w := range a.Unknown {
why = append(why, r+": "+w)
}
sort.Strings(why)
return nil, fmt.Errorf("the artifact store could not be asked about %s", strings.Join(why, "; "))
}
return a.Gone, nil
}
}
// heardGone takes one artifacts-gone event in: acknowledged, then handed to the builder, which asks the store and builds
// the key again. A body that is not one is said and acknowledged, so it is not handed over again.
func heardGone(ctx context.Context, b *build.Builder, m jetstream.Msg, logf func(string, ...any)) {
e, err := delivery.DecodeArtifactsGone(m.Data())
if ackErr := m.Ack(); ackErr != nil {
logf("acknowledging %s: %v", m.Subject(), ackErr)
}
if err != nil {
logf("%s holds what is not %s, left: %v", m.Subject(), delivery.ArtifactsGoneEvent, err)
return
}
b.ArtifactsGone(ctx, e)
}
@@ -0,0 +1,82 @@
{
"module": "hello-web",
"slug": "hello",
"version": "1",
"capabilities": [
"container-runtime"
],
"requires": [
"route"
],
"contributes": {
"route": {
"label": "hello",
"endpoint": "web"
}
},
"binds": {
"route": "${dir:state}/route.json"
},
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
"why": "the demo web page; only the route-proxy reaches it, and the public name is a route grant"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "page",
"type": "file",
"path": "${dir:state}/index.html",
"mode": "0644",
"template": "files/index.html"
},
{
"id": "net",
"type": "network",
"name": "hello-web"
},
{
"id": "server",
"type": "container",
"name": "hello-web",
"network": "hello-web",
"ports": [
"8080"
],
"volumes": [
"${dir:state}/index.html:/www/index.html:ro"
],
"args": [
"sh",
"-c",
"while true; do { printf 'HTTP/1.1 200 OK\\r\\nContent-Type: text/plain\\r\\nConnection: close\\r\\n\\r\\n'; cat /www/index.html; } | nc -l -p 8080; done"
],
"artifact": "server"
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "upstream",
"from": "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,249 @@
{
"module": "claude-code",
"version": "1",
"moved": true,
"slug": "agent",
"capabilities": [
"package-manager"
],
"requires": [
"mcp-endpoint",
{
"name": "gitea-read",
"optional": true
}
],
"binds": {
"mcp-endpoint": "${dir:state}/mcp-endpoint.json"
},
"secrets": {
"gitea-read": "${dir:state}/gitea-read.secret"
},
"uses": [
"operator-channel"
],
"state": [
"servers",
"holdings",
"config",
"proposals",
"questions",
"memory",
"waits"
],
"reads": [
"claude-licence-manager.bindings"
],
"tools": [
"claude_code_status",
"claude_code_render",
"claude_code_guard",
"claude_code_pull",
"claude_code_grant",
"claude_code_add_api_key",
"claude_code_registrations",
"claude_code_memory",
"claude_code_mcp_list",
"claude_code_mcp_register",
"claude_code_mcp_unregister",
"claude_code_skill_list",
"claude_code_skill_register",
"claude_code_skill_unregister",
"claude_code_agent_list",
"claude_code_agent_register",
"claude_code_agent_unregister",
"claude_code_command_list",
"claude_code_command_register",
"claude_code_command_unregister",
"claude_code_hook_list",
"claude_code_hook_register",
"claude_code_hook_unregister",
"claude_code_output_style_list",
"claude_code_output_style_register",
"claude_code_output_style_unregister",
"claude_code_instructions_list",
"claude_code_instructions_register",
"claude_code_instructions_unregister",
"claude_code_instructions_propose",
"claude_code_proposals",
"claude_code_ask",
"claude_code_answer",
"claude_code_wait",
"claude_code_settings_get",
"claude_code_settings_set",
"claude_code_settings_clear",
"claude_code_permission_add",
"claude_code_permission_remove",
"claude_code_config_list",
"claude_code_config_show",
"claude_code_config_status",
"claude_code_config_import",
"claude_code_home_show",
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge",
"claude_code_judged"
],
"data": {
"own": [
{
"id": "agent-home",
"path": "${dir:agent-home}",
"class": "valuable",
"why": "the operator's agent's own files: its memory, history and projects"
}
]
},
"config": {
"role": {
"type": "string",
"description": "What this machine is for, in a sentence, written into the instructions every Claude Code session on it reads; empty says nothing.",
"trusted": true,
"default": ""
},
"mcp_servers": {
"type": "object",
"description": "The tool servers every session on this machine is given beside the mesh's own, by name, each as Claude Code's managed MCP configuration takes it; empty adds none.",
"trusted": true,
"default": {}
},
"managed_settings": {
"type": "object",
"description": "Keys of Claude Code's managed settings every session on this machine obeys, such as spinnerTipsEnabled, laid over the mesh's own; empty adds none. Falls back to agent-managed-settings, the mesh's or this node's.",
"trusted": true,
"predefined": "agent-managed-settings",
"default": {}
},
"instructions": {
"type": "object",
"description": "The instructions every session on this machine reads, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"output_styles": {
"type": "object",
"description": "The output styles offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"skills": {
"type": "object",
"description": "The skills offered to every session on this machine, by name, each its files by path, a text as one line or a list of its lines; a skill without its SKILL.md is left out; empty adds none.",
"trusted": true,
"default": {}
},
"commands": {
"type": "object",
"description": "The slash commands offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"agents": {
"type": "object",
"description": "The agents offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
}
},
"resources": [
{
"id": "package",
"type": "package",
"package": "claude-code"
},
{
"id": "managed",
"type": "directory",
"path": "/etc/claude-code",
"mode": "0755",
"unknown": "program"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/claude-agent",
"mode": "0755",
"template": "files/claude-agent"
},
{
"id": "agent-account-name",
"type": "file",
"path": "/etc/claude-code/agent-account",
"mode": "0644",
"template": "files/agent-account"
},
{
"id": "agent-account",
"type": "user",
"name": "${machine:agent-account}",
"home": "${machine:agent-home}",
"root": "${machine:agent-root}"
},
{
"id": "agent-home",
"type": "directory",
"path": "${machine:agent-home}/.claude",
"mode": "0700",
"owner": "${machine:agent-account}",
"unknown": "module"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "${machine:account}",
"place": ".",
"unknown": "program"
},
{
"id": "facts",
"type": "file",
"path": "${dir:state}/facts.json",
"mode": "0600",
"owner": "${machine:account}",
"template": "files/facts.json"
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.json",
"mode": "0600",
"owner": "${machine:account}",
"takes-config": ["role", "mcp_servers", "managed_settings", "instructions", "output_styles", "skills", "commands", "agents"],
"merge": "json",
"template": "files/settings.json"
}
],
"items": [
{
"collection": "shell:zsh:normal",
"template": "files/zsh-normal"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/claude-code",
"binary": "claude-code",
"loads": [
"claude-code"
],
"env": {
"MESH_CLAUDE_CODE_STATE": "${dir:state}",
"MESH_CLAUDE_CODE_FACTS": "${dir:state}/facts.json",
"MESH_CLAUDE_CODE_SETTINGS": "${dir:state}/settings.json"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,66 @@
{
"module": "ca-trust",
"version": "1",
"slug": "catrust",
"capabilities": [
"service-manager"
],
"requires": [
"internal-acme-ca"
],
"seats": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"claims": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "anchor",
"type": "file",
"path": "${dir:state}/anchor",
"mode": "0755",
"template": "files/anchor"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-ca-trust.service",
"mode": "0644",
"template": "files/mesh-ca-trust.service"
},
{
"id": "trust",
"type": "service",
"unit": "mesh-ca-trust.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"anchor",
"unit"
]
}
],
"build": {
"artifacts": [
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,14 @@
{
"module": "imaged",
"version": "1",
"slug": "imaged",
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile"
}
]
}
}
@@ -0,0 +1,57 @@
{
"module": "netcheck",
"version": "1",
"emits": [
"tunnel.endpoint-changed",
"tunnel.handshake-stale",
"tunnel.handshake-fresh"
],
"tools": [
"netcheck_tunnel",
"netcheck_tcp",
"netcheck_dns",
"netcheck_http",
"netcheck_listening"
],
"replaces": {
"netcheck_listening": [
"ss -lunt",
"ss -ltn",
"netstat -lnt"
],
"netcheck_dns": [
"dig",
"nslookup"
],
"netcheck_tunnel": [
"wg show",
"wg show mesh0 dump"
]
},
"build": {
"artifacts": [
{
"name": "tools-go",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/netcheck",
"binary": "netcheck",
"loads": [
"netcheck"
]
},
{
"name": "tools-typescript",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"tools/index.js"
],
"loads": [
"tools/index.js"
]
}
]
}
}
@@ -0,0 +1,105 @@
package main
import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"log"
"strings"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// toolchainStreams are the stream each build module carries its toolchain.reported event on, as the bus names it
// (declared by the build modules, heard by this module through consumes).
var toolchainStreams = map[string]string{buildrun.GoBuildModule: "TOOLCHAINS_GO", buildrun.NodeBuildModule: "TOOLCHAINS_NODE"}
// toolchainWatch keeps the build modules' toolchains in a buildrun.ToolchainSet and tells the builder when one changed. A
// build module's bucket is made when it is assigned, after the builder started, and the builder holds no timer (ADR 0437
// §8): it watches the buckets that exist at start, and a build module says each time it wrote its toolchains key
// (toolchain.reported, handled by Reported); a bucket not watched yet is then watched.
type toolchainWatch struct {
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
// put takes a change of a toolchains bucket in: set.Put, or the gate of the assigned build nodes before it (issue 1940).
put func(module, key string, value []byte, deleted bool) bool
changed func(ctx context.Context) // a language's entries changed: the failed builds are tried again
logger *log.Logger
errs chan error
mu sync.Mutex
watched map[string]bool
}
func newToolchainWatch(ctx context.Context, js bucketOpener, set *buildrun.ToolchainSet, changed func(context.Context), logger *log.Logger) *toolchainWatch {
return &toolchainWatch{ctx: ctx, js: js, set: set, put: set.Put, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
}
// start watches the buckets that exist now; a missing one is said and waits for its build module's event.
func (w *toolchainWatch) start() error {
for module := range buildrun.LanguageOf {
if err := w.ensure(module); err != nil {
return err
}
}
return nil
}
func (w *toolchainWatch) ensure(module string) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.watched[module] {
return nil
}
name := delivery.ToolchainsBucket(module)
kv, err := w.js.KeyValue(w.ctx, name)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
w.logger.Printf("the bucket %s does not exist yet: it is watched when %s reports a toolchain", name, module)
return nil
case err != nil:
return fmt.Errorf("the bucket %s: %w", name, err)
}
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.put(module, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
w.watched[module] = true
go func() {
w.errs <- follow(w.ctx, watcher, func(e jetstream.KeyValueEntry) {
if w.put(module, e.Key(), e.Value(), !opOf(e)) {
w.changed(w.ctx)
}
})
}()
w.changed(w.ctx) // what the settle read is new to the builder
return nil
}
// delivered is the part of a JetStream message the watch uses.
type delivered interface {
Subject() string
Ack() error
Term() error
}
// Reported takes one toolchain.reported event of a build module in: the module is read off the subject,
// mesh.stream.<module>.event.toolchain.reported, and its bucket is watched if it is not. A subject of no build module is
// terminated; a failure stays unacknowledged, and the server delivers it again.
func (w *toolchainWatch) Reported(m delivered) {
module, _, _ := strings.Cut(strings.TrimPrefix(m.Subject(), "mesh.stream."), ".event.")
if _, known := buildrun.LanguageOf[module]; !known || !strings.HasPrefix(m.Subject(), "mesh.stream.") {
w.logger.Printf("%s: not a build module's event, dropped", m.Subject())
_ = m.Term()
return
}
if err := w.ensure(module); err != nil {
w.logger.Printf("%s: left for redelivery: %v", m.Subject(), err)
return
}
_ = m.Ack()
}
+22
View File
@@ -0,0 +1,22 @@
module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27
View File
@@ -0,0 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1 h1:fzOkrz9CSp3mEjeV7NfZx8yoivI1sD1AOtIPtyyF8RI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,108 @@
package build
import (
"context"
"fmt"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// How a candidate's fingerprint is built (novox/hq ADR 0484 clause 3, issue 1684): the indexer writes, for each module a
// pull request touches, the fingerprint of the new form it has at the head merged onto the trunk (candidates.<group>), and
// the builder builds each, once, as a key of its own (builds.<module>.<fingerprint>) that the checker's line reads. The
// build is the trunk module's: the same build modules, the same nodes and capacity; what differs is where the source comes
// from: the fetch step is given the pull request's head, the ref that carries it and the trunk commit to merge it onto, as
// the checker's fetch is, and builds the merge's module directory.
// via is how a build's fetch step reaches its source when it is a candidate's: the ref that carries the head and the trunk
// commit the head is merged onto.
type via struct{ Ref, Base string }
// Candidate takes one value of the indexer's candidates bucket in (the key is the group) and builds the fingerprints it
// carries. A value that is not a candidate is said and left.
func (b *Builder) Candidate(ctx context.Context, key string, value []byte) {
b.init()
c, err := delivery.DecodeCandidate(value)
if err != nil {
b.logf("candidates.%s holds what is not a candidate, left: %v", key, err)
return
}
b.mu.Lock()
b.cands[key] = c
b.mu.Unlock()
b.considerCandidate(ctx, c)
}
// ForgetCandidate takes a candidate out (its key was deleted).
func (b *Builder) ForgetCandidate(key string) {
b.init()
b.mu.Lock()
delete(b.cands, key)
b.mu.Unlock()
}
func (b *Builder) considerCandidate(ctx context.Context, c delivery.Candidate) {
for _, p := range c.Pulls {
names := make([]string, 0, len(p.Fps))
for name := range p.Fps {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
cm := p.Fps[name]
if cm.FP2 == "" || cm.Manifest == "" {
continue // no fingerprint of the new form for it (a reason in none2) or no manifest to build by
}
b.mu.Lock()
trunk, known := b.known[name]
b.mu.Unlock()
if !known || trunk.Path == "" {
b.logf("candidates.%s: the module %s has no record of the trunk yet, so its fingerprint %s is not built", c.Group, name, cm.FP2)
continue
}
key, err := delivery.BuilderBuildKey(name, cm.FP2)
if err != nil {
b.logf("candidates.%s: %v", c.Group, err)
continue
}
m := delivery.Module{Repository: p.Repository, Commit: p.Head.Head, FP2: cm.FP2, Manifest: cm.Manifest, Path: trunk.Path,
Toolchains: b.agreedFor(ctx, cm.Manifest)}
if len(cm.Paths) > 0 {
// The named paths' objects the preview gave it: the fetch names the paths (runLane, ADR 0499 clause 4).
m.Inputs2 = &delivery.KeptInputs2{Paths: cm.Paths}
}
b.mu.Lock()
if _, onTrunk := b.trunkFP[key]; !onTrunk {
b.via[key] = via{Ref: fmt.Sprintf("refs/pull/%d/head", p.Number), Base: trunk.Commit}
}
b.mu.Unlock()
b.consider(ctx, name, m)
}
}
}
// agreedFor are the toolchains the build nodes agree on now, for the languages of the manifest's plan: the toolchains the
// candidate's fingerprint was computed with, as the indexer recomputes it whenever they change. A language with none agreed
// is left out, and the build then waits for it as the trunk's does.
func (b *Builder) agreedFor(ctx context.Context, manifest string) map[string]delivery.ToolchainUsed {
plan, err, _ := b.planOf(ctx, manifest)
if err != nil {
return nil
}
out := map[string]delivery.ToolchainUsed{}
for _, bm := range plan.Modules() {
if !plan.NeedsToolchain(bm) {
continue
}
language := buildrun.LanguageOf[bm]
if tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language)); none == "" {
out[language] = delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
}
}
if len(out) == 0 {
return nil
}
return out
}
@@ -0,0 +1,114 @@
package build
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// trunkModule writes the trunk's record of the module m at a fingerprint of its own, so that the candidate's fingerprint (fp1)
// is one the trunk does not carry.
func trunkModule(t *testing.T, r *rig) {
t.Helper()
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'),
FP2: "src2:" + hex64('e'), Manifest: mdigest('a'), Path: "modules/m", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
r.b.Wait()
r.bm.mu.Lock()
r.bm.calls = nil
r.bm.mu.Unlock()
}
func candidateValue(t *testing.T, group string, number int, head string) []byte {
t.Helper()
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: number, Head: head}, Modules: []string{"m"},
Fps: map[string]delivery.CandidateModule{"m": {FP2: fp1, Manifest: mdigest('a')}}}}
hash, err := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: group, HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
return data
}
const (
headX = "1111111111111111111111111111111111111111"
headY = "2222222222222222222222222222222222222222"
)
func TestACandidatesFingerprintIsBuiltOnceAsAKeyOfItsOwnFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1, N7/AC2 (issue 1684)
r := newRig(t, goBundle)
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Fatalf("build = %+v, want the candidate's fingerprint built", v)
}
fetches := r.bm.callsOf("build_go_fetch")
if len(fetches) != 1 { // one node per fingerprint (ADR 0478 clause 4)
t.Fatalf("%d fetches, want one", len(fetches))
}
f := fetches[0].body.(map[string]any)
if f["commit"] != headX || f["ref"] != "refs/pull/7/head" || f["base"] != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" || f["repository"] != "novox/mesh-catalog" {
t.Errorf("fetch args = %+v, want the head, its ref and the trunk commit to merge onto", f)
}
if b := r.bm.callsOf("build_go_build")[0].body.(map[string]any); b["module"] != "modules/m" {
t.Errorf("build args = %+v", b)
}
// Two heads carrying one fingerprint are one build.
r.b.Candidate(context.Background(), "feat%2Fy", candidateValue(t, "feat/y", 8, headY))
r.b.Wait()
if n := len(r.bm.callsOf("build_go_fetch")); n != 1 {
t.Errorf("%d fetches after a second head with the same fingerprint, want no more", n)
}
}
func TestACandidateWhoseModuleHasNoTrunkRecordIsNotBuilt(t *testing.T) {
r := newRig(t, goBundle)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if _, ok := r.st.vals[r.key]; ok {
t.Error("a build key was written for a module the builder knows no directory of")
}
}
func TestAFingerprintTheTrunkCarriesIsBuiltFromTheTrunksCommit(t *testing.T) { // the trunk's source wins
r := newRig(t, goBundle)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the trunk now carries fp1
r.b.Wait()
for _, c := range r.bm.callsOf("build_go_fetch") {
if f := c.body.(map[string]any); f["ref"] != nil || f["base"] != nil {
t.Errorf("fetch args = %+v: a trunk fingerprint is built from the trunk's commit alone", f)
}
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v", v)
}
}
func TestACandidateWhoseHeadDoesNotMergeIsFailedAndNoTrunkBuildIsStoredUnderIt(t *testing.T) { // review D4
for name, set := range map[string]func(*rig){
"conflict": func(r *rig) { r.bm.conflicts = "go.mod" },
"no merge": func(r *rig) { r.bm.noMerge = true },
} {
r := newRig(t, goBundle)
trunkModule(t, r)
set(r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildFailed || (name == "conflict" && !strings.Contains(v.Error, "go.mod")) || len(r.bm.callsOf("build_go_build")) != 0 {
t.Errorf("%s: build = %+v, builds %d, want failed and nothing built", name, v, len(r.bm.callsOf("build_go_build")))
}
}
}
@@ -0,0 +1,260 @@
package build
import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// ErrConflict is what a Store answers when a Create finds the key or an Update finds another revision than the one given.
var ErrConflict = errors.New("the key exists or changed")
// Store is the builder's bucket (builds): create and compare-and-set, each answering the key's new revision. Only the
// builder's credential writes it (ADR 0429 clause 5).
type Store interface {
Get(ctx context.Context, key string) (value []byte, revision uint64, found bool, err error)
Create(ctx context.Context, key string, value []byte) (uint64, error)
Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error)
// Keys are every key of the bucket.
Keys(ctx context.Context) ([]string, error)
}
// Claimer claims fingerprints for the builder instance By (ADR 0429 clause 5: a builder claims a fingerprint by creating
// its key and moves it by compare-and-set).
type Claimer struct {
Store Store
By string
}
// Claim is a build the builder holds: its key, the revision the next write must be made at and the value last written.
type Claim struct {
Key string
Rev uint64
Build delivery.BuilderBuild
// Gone are the references of a done key the artifact store no longer holds, for which it was claimed again
// (ClaimOptions.Stored; issue 1937): said in the log and the key's step.
Gone []string
// Unasked is why the store could not be asked about a done key taken as it stands to publish: said in the log.
Unasked error
}
// ClaimOptions say what the builder knows about the moment of a claim.
type ClaimOptions struct {
// TakeOver takes a key found building whoever holds it: the builder has just started and looks at the key for the first
// time, and a build left building by a builder that is gone, on this node or another, is nobody's (ADR 0478 clause 6).
TakeOver bool
// ImageBuilder says whether a node runs a healthy image build module now: a key failed for NoImageBuilderPrefix is taken
// again only then. Nil means none does.
ImageBuilder func() bool
// Trunk is whether the trunk's record names this fingerprint. A done key whose image or package has no reference was
// built as a candidate's, which never pushes or publishes (ADR 0487 clause 12, ADR 0498 clauses 5 and 6); on the trunk it
// is taken again to publish them, an image only when an image builder runs (ImageBuilder).
Trunk bool
// Published says whether a package version `<name>@<version>` is published: one of the builder's done keys names it as
// a package's reference. A key queued waiting for a version (BuilderBuild.Waits) is taken only once it is (ADR 0500
// clause 6). Nil means none is.
Published func(ref string) bool
// Stored answers which of the references given the artifact store no longer holds (the controller's verb held,
// mesh-sdk delivery.HeldVerb; novox/hq issue 1937). A done key of the trunk's fingerprint is asked before it is left
// done: the store's nightly collector deletes a blob nothing holds, and a key the controller never took in holds
// nothing, so a key whose blob is gone is claimed building as a new revision and built again. Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
}
// StoredReferences are the references of a done build's bundles and archives that name a blob in the artifact store
// (`artifact-store://<module>/<artifact>/blobs/<digest>`): what the store's collector can delete. An image or an upstream
// copy is a manifest, which the collector keeps, and a package lives in the package registry.
func StoredReferences(b delivery.BuilderBuild) []string {
var out []string
for _, a := range b.Artifacts {
if strings.HasPrefix(a.Reference, "artifact-store://") && strings.Contains(a.Reference, "/blobs/") &&
!slices.Contains(out, a.Reference) {
out = append(out, a.Reference)
}
}
return out
}
// GoneStep is the step a key claimed again for gone blobs says (issue 1937).
func GoneStep(gone []string) string {
return "claimed again: the artifact store no longer holds " + strings.Join(gone, ", ")
}
// Unpublished is whether a done build holds an image built without a push or a package built without a publish: it has
// no reference (a candidate's, ADR 0487 clause 12, ADR 0498 clauses 5 and 6).
func Unpublished(b delivery.BuilderBuild) bool { return len(UnpublishedModules(b)) > 0 }
// UnpublishedModules are the build modules that publish what a done build holds unpublished: the image build module for
// an image, the Node build module for a package; sorted, none when it holds nothing unpublished.
func UnpublishedModules(b delivery.BuilderBuild) []string {
if b.State != delivery.BuildDone {
return nil
}
var image, pkg bool
for _, a := range b.Artifacts {
image = image || (a.Kind == catalogue.ArtifactImage && a.Reference == "")
pkg = pkg || (a.Kind == catalogue.ArtifactPackage && a.Reference == "")
}
var out []string
if image {
out = append(out, ImageBuildModule)
}
if pkg {
out = append(out, buildrun.NodeBuildModule)
}
sort.Strings(out)
return out
}
// publishable are the build modules of UnpublishedModules that can publish now: the image build module only while a node
// runs it (imageBuilder).
func publishable(b delivery.BuilderBuild, imageBuilder func() bool) []string {
var out []string
for _, bm := range UnpublishedModules(b) {
if bm != ImageBuildModule || (imageBuilder != nil && imageBuilder()) {
out = append(out, bm)
}
}
return out
}
// Unreferenced is whether a done build holds a bundle, an archive or an upstream copy that names no reference: a key done
// before R4e, whose artifacts stayed on their build nodes (ADR 0498 clause 8). An artifact of no kind (a key written before
// kinds) is one too. Claim takes such a key again, once: its new revision names them.
func Unreferenced(b delivery.BuilderBuild) bool {
if b.State != delivery.BuildDone {
return false
}
for _, a := range b.Artifacts {
if a.Reference == "" && (a.Kind == "" || stored(a.Kind)) {
return true
}
}
return false
}
// Claim takes the build of module at fingerprint, or answers nil when it is not this builder's to build now:
// - no key: it is created building, by this builder; a create that finds the key made meanwhile gets nothing;
// - a done key is not taken again (the same source is never built twice), except one whose bundle, archive or upstream
// copy names no reference (Unreferenced, a key done before R4e: ADR 0498 clause 8), which is claimed building as a new
// revision and built whole, and one whose image or package was built unpublished as a candidate's once the trunk
// names it (ClaimOptions.Trunk): that is taken as it stands, with the done value, and its builder writes the new
// revision when the publishing build ends (ADR 0498 clause 6). From R4e one node builds each fingerprint (ADR 0478
// clause 4), so a key built on one node is never built again for a second;
// - a done key of the trunk's fingerprint whose bundle or archive blob the artifact store no longer holds
// (ClaimOptions.Stored, issue 1937) is claimed building as a new revision and built whole, its step saying which;
// - a failed key is taken again as a new revision of the same key (ADR 0429 clause 18), except one failed for a kind no
// builder builds, for an upstream pinned by no digest or a package version taken (delivery.FailedUntilChanged: ADR 0478
// clause 2, ADR 0498 clauses 4 and 5) or for digests that differ (buildrun.DigestsDifferPrefix, clause 4),
// which wait for another fingerprint, and one failed for want of an image builder (NoImageBuilderPrefix), which waits for
// another fingerprint or a node that runs the image build module (ClaimOptions.ImageBuilder);
// - a key queued waiting for a package version (Waits) is claimed building as a new revision once that version is
// published (ClaimOptions.Published, ADR 0500 clause 6), and not before;
// - a key left queued or building by this builder is taken as it stands (ADR 0478 clause 5: a stopped builder takes its
// own again when it starts), and so is one left by another builder when ClaimOptions.TakeOver (clause 6).
func (c Claimer) Claim(ctx context.Context, module, fingerprint, manifest string, opt ClaimOptions) (*Claim, error) {
key, err := delivery.BuilderBuildKey(module, fingerprint)
if err != nil {
return nil, err
}
building := delivery.BuilderBuild{State: delivery.BuildBuilding, By: c.By, Step: "claimed", Manifest: manifest}
value, rev, found, err := c.Store.Get(ctx, key)
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
if !found {
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Create(ctx, key, data) })
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
return nil, fmt.Errorf("%s holds what is not a build: %w", key, err)
}
switch {
case cur.State == delivery.BuildDone:
if Unreferenced(cur) {
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
}
if opt.Trunk && opt.Stored != nil {
if refs := StoredReferences(cur); len(refs) > 0 {
gone, err := opt.Stored(ctx, refs)
if err != nil {
err = fmt.Errorf("%s: whether the artifact store still holds %s cannot be asked, so the key is "+
"left done (the controller's take-in refuses it if one is gone): %w", key, strings.Join(refs, ", "), err)
if len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur, Unasked: err}, nil
}
return nil, err
}
if len(gone) > 0 {
again := building
again.Step = GoneStep(gone)
cl, err := c.write(ctx, key, again, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
if cl != nil {
cl.Gone = gone
}
return cl, err
}
}
}
if opt.Trunk && len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur}, nil
}
return nil, nil
case cur.State == delivery.BuildFailed && (delivery.FailedUntilChanged(cur.Error) || strings.HasPrefix(cur.Error, buildrun.DigestsDifferPrefix)):
return nil, nil
case cur.State == delivery.BuildFailed && strings.HasPrefix(cur.Error, NoImageBuilderPrefix) && (opt.ImageBuilder == nil || !opt.ImageBuilder()):
return nil, nil
case cur.State == delivery.BuildFailed:
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
case cur.State == delivery.BuildQueued && cur.Waits != "":
if opt.Published == nil || !opt.Published(cur.Waits) {
return nil, nil // the package it waits for is not published yet
}
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
case cur.By == c.By || opt.TakeOver:
return &Claim{Key: key, Rev: rev, Build: cur}, nil
}
return nil, nil
}
func (c Claimer) write(_ context.Context, key string, b delivery.BuilderBuild, put func([]byte) (uint64, error)) (*Claim, error) {
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
return nil, err
}
rev, err := put(data)
if errors.Is(err, ErrConflict) {
return nil, nil // another builder made it first
}
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
b.V = delivery.Version
return &Claim{Key: key, Rev: rev, Build: b}, nil
}
// Finish moves the claimed key to b by compare-and-set at the claim's revision, and writes nothing when b is the value
// the key holds. A key that changed meanwhile is ErrConflict: the claim is no longer this builder's.
func (c Claimer) Finish(ctx context.Context, cl *Claim, b delivery.BuilderBuild) error {
b.V = delivery.Version
if delivery.SameBuilderBuild(cl.Build, b) {
return nil
}
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
return err
}
rev, err := c.Store.Update(ctx, cl.Key, data, cl.Rev)
if err != nil {
return fmt.Errorf("%s: %w", cl.Key, err)
}
cl.Rev, cl.Build = rev, b
return nil
}
@@ -0,0 +1,219 @@
package build
import (
"context"
"errors"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// memStore is a bucket's create and compare-and-set in memory, with a revision per write.
type memStore struct {
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
next uint64
writes int
// before runs once ahead of the next write, so a test changes the key under the writer.
before func()
}
func newMem() *memStore { return &memStore{vals: map[string][]byte{}, revs: map[string]uint64{}} }
func (m *memStore) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
m.mu.Lock()
defer m.mu.Unlock()
v, ok := m.vals[key]
return v, m.revs[key], ok, nil
}
func (m *memStore) Keys(_ context.Context) ([]string, error) {
m.mu.Lock()
defer m.mu.Unlock()
var keys []string
for k := range m.vals {
keys = append(keys, k)
}
return keys, nil
}
func (m *memStore) hook() {
if h := m.before; h != nil {
m.before = nil
h()
}
}
func (m *memStore) put(key string, value []byte) uint64 {
m.next++
m.writes++
m.vals[key], m.revs[key] = value, m.next
return m.next
}
func (m *memStore) Create(_ context.Context, key string, value []byte) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.hook()
if _, ok := m.vals[key]; ok {
return 0, ErrConflict
}
return m.put(key, value), nil
}
func (m *memStore) Update(_ context.Context, key string, value []byte, rev uint64) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.hook()
if _, ok := m.vals[key]; !ok || m.revs[key] != rev {
return 0, ErrConflict
}
return m.put(key, value), nil
}
const (
fp1 = "src2:1111111111111111111111111111111111111111111111111111111111111111"
digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
)
func seed(t *testing.T, st *memStore, b delivery.BuilderBuild) string {
t.Helper()
key, err := delivery.BuilderBuildKey("m", fp1)
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
t.Fatal(err)
}
st.put(key, data)
return key
}
func valueOf(t *testing.T, st *memStore, key string) delivery.BuilderBuild {
t.Helper()
b, err := delivery.DecodeBuilderBuild(st.vals[key])
if err != nil {
t.Fatal(err)
}
return b
}
func TestClaimCreatesTheKeyBuildingAndASecondBuilderGetsNothing(t *testing.T) { // 1607/AC3
st := newMem()
a, b := Claimer{Store: st, By: "novox.a"}, Claimer{Store: st, By: "novox.b"}
got, err := a.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got == nil {
t.Fatalf("first claim = %v, %v, want a claim", got, err)
}
if v := valueOf(t, st, got.Key); v.State != delivery.BuildBuilding || v.By != "novox.a" || v.Manifest != hex64('a') {
t.Errorf("the key holds %+v, want building by novox.a with the manifest digest", v)
}
if second, err := b.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{}); err != nil || second != nil {
t.Errorf("second builder's claim = %v, %v, want none: the same source is never built twice", second, err)
}
}
func TestClaimLosingTheCreateRaceGetsNothing(t *testing.T) { // 1607/AC3
st := newMem()
st.before = func() { seed(t, st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.b"}) }
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
if err != nil || got != nil {
t.Errorf("claim = %v, %v, want none: the create failed because the key exists", got, err)
}
}
func TestADoneKeyIsNeverTakenAgain(t *testing.T) { // 1607/AC3; a key whose artifacts name references (ADR 0498 clause 8)
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.a", Manifest: hex64('a'), Artifacts: []delivery.BuiltArtifact{{Name: "x", Kind: "bundle", Digest: digest, Reference: "artifact-store://m/x/blobs/" + digest}}})
writes := st.writes
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
}
func TestAFailedKeyIsTakenAgainAsANewRevisionExceptARefusedKind(t *testing.T) { // 1607/AC3, ADR 0478 clause 2
st := newMem()
key := seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: "the step failed"})
rev := st.revs[key]
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got == nil || st.revs[key] <= rev {
t.Fatalf("claim = %v, %v, revision %d (was %d), want a claim as a new revision of the same key", got, err, st.revs[key], rev)
}
if v := valueOf(t, st, key); v.State != delivery.BuildBuilding || v.Error != "" {
t.Errorf("the key holds %+v, want building with no error", v)
}
st = newMem()
key = seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: delivery.RefusedKind + "package of language \"python\" (artifact x)"})
writes := st.writes
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got != nil || st.writes != writes {
t.Errorf("a refused kind: claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
for _, e := range []string{delivery.NotPinned + "up: nats:2 names no digest", delivery.VersionTaken + "@novox/mesh-sdk@0.1.15"} { // B3, B5: ADR 0498 clauses 4, 5
st = newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: e})
writes = st.writes
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got != nil || st.writes != writes {
t.Errorf("%s: claim = %v, %v, writes %d (was %d), want none until the fingerprint changes", e, got, err, st.writes, writes)
}
}
}
func TestAKeyFailedForNoImageBuilderIsTakenAgainOnlyWhenANodeHasTheModule(t *testing.T) { // 1705b
st := newMem()
key := seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: NoImageBuilderPrefix + " no node has mesh-build-image"})
writes := st.writes
c := Claimer{Store: st, By: "novox.a"}
for _, opt := range []ClaimOptions{{}, {ImageBuilder: func() bool { return false }}} {
if got, err := c.Claim(context.Background(), "m", fp1, "", opt); err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
}
got, err := c.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{ImageBuilder: func() bool { return true }})
if err != nil || got == nil || valueOf(t, st, key).State != delivery.BuildBuilding {
t.Errorf("claim = %v, %v, want a claim with the key building", got, err)
}
}
func TestFinishMovesTheKeyByCompareAndSetAndWritesOnlyADifferentValue(t *testing.T) { // 1607/AC3
st := newMem()
c := Claimer{Store: st, By: "novox.a"}
got, _ := c.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
done := delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.a", Manifest: hex64('a'), Artifacts: []delivery.BuiltArtifact{{Name: "x", Kind: "archive", Digest: digest}}}
if err := c.Finish(context.Background(), got, done); err != nil {
t.Fatal(err)
}
if v := valueOf(t, st, got.Key); v.State != delivery.BuildDone || v.Artifacts[0].Digest != digest {
t.Errorf("the key holds %+v", v)
}
writes := st.writes
if err := c.Finish(context.Background(), got, done); err != nil && !errors.Is(err, ErrConflict) {
t.Fatal(err)
}
if st.writes != writes {
t.Errorf("the same value was written again (%d writes, was %d)", st.writes, writes)
}
}
func TestFinishAfterTheKeyChangedIsAConflictNotAnOverwrite(t *testing.T) { // 1607/AC3
st := newMem()
c := Claimer{Store: st, By: "novox.a"}
got, _ := c.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
st.put(got.Key, []byte(`{"v":1,"state":"failed","error":"someone else"}`))
err := c.Finish(context.Background(), got, delivery.BuilderBuild{State: delivery.BuildFailed, Error: "mine"})
if !errors.Is(err, ErrConflict) {
t.Errorf("Finish = %v, want ErrConflict", err)
}
}
func hex64(c byte) string {
b := make([]byte, 64)
for i := range b {
b[i] = c
}
return string(b)
}
@@ -0,0 +1,94 @@
package build
// The fakes of the build modules the builder tests run against: a caller answering a long tool at once with the step and its key,
// and the steps watch. The caller, the steps watch and the runner themselves are the SDK's (mesh-sdk go/delivery/buildrun, issue 1671).
import (
"context"
"encoding/json"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
// presenceValue is a watchman presence value, as its writer encodes it (mesh-sdk go/delivery EncodePresence).
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
// tcValue is a toolchains value, as the build module's writer encodes it (mesh-sdk go/delivery EncodeToolchain).
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
// tcs are the toolchain entries of a language on nodes, as the toolchains set gives them.
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
@@ -0,0 +1,59 @@
package build
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// result reads a captured step value of mesh-build-image (testdata/captured/README) and returns its result.
func captured(t *testing.T, name string) json.RawMessage {
t.Helper()
raw, err := os.ReadFile("testdata/captured/" + name)
if err != nil {
t.Fatal(err)
}
var step delivery.Step
if err := json.Unmarshal(raw, &step); err != nil {
t.Fatalf("%s: %v", name, err)
}
if step.State != delivery.StepPassed || len(step.Result) == 0 {
t.Fatalf("%s: state %q, result %s, want a passed step with a result", name, step.State, step.Result)
}
return step.Result
}
// The captured image steps, run as a pulled ask runs them (buildstep.RunAsk), answer what the builder reads: the trunk's
// build its digest and reference, a candidate's its digest and no reference (issue 1887 A4a).
func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // review r2, mesh-catalog 2; ADR 0487 clause 1
for _, tc := range []struct {
build string
publish bool
ref bool
}{{"build-trunk.json", true, true}, {"build-candidate.json", false, false}} {
steps := buildstep.AskSteps{
Fetch: func(context.Context, map[string]any) (any, error) {
return json.RawMessage(captured(t, "fetch.json")), nil
},
Build: func(context.Context, map[string]any) (any, error) { return json.RawMessage(captured(t, tc.build)), nil },
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskDelivery, Language: delivery.AskImage, Asker: delivery.BuilderModule,
At: time.Now(), Build: &delivery.BuildJob{Module: "m", Path: "modules/m", Publish: tc.publish,
Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "server"}}}}
out, err := buildstep.RunAsk(context.Background(), steps, ask)
if err != nil {
t.Fatalf("%s: %v", tc.build, err)
}
a := out.(delivery.BuildAnswer)
if len(a.Artifacts) != 1 || a.Artifacts[0].Digest == "" || (a.Artifacts[0].Reference != "") != tc.ref {
t.Errorf("%s answered %+v, want a digest and a reference only for the trunk", tc.build, a)
}
}
}
@@ -0,0 +1,46 @@
package build
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1660: at start the builder closes a key left building for a fingerprint the module no longer has.
func TestAKeyLeftBuildingForAFingerprintTheModuleNoLongerHasIsClosedAtStart(t *testing.T) { // issue 1660
r := newRig(t, goBundle)
stale := seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD}) // m at fp1
other, _ := delivery.BuilderBuildKey("gone", fp1)
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
r.st.put(other, data)
// The module m has moved on to another fingerprint.
moved := delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'),
FP2: "src2:" + hex64('7'), Manifest: manifestD, Path: "modules/m", Toolchains: agreedToolchains}
enc, _ := delivery.EncodeModule(moved)
r.b.Module(context.Background(), "m", enc)
r.b.Wait()
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
for _, k := range []string{stale, other} {
v := valueOf(t, r.st, k)
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "no longer") || !strings.Contains(v.Error, "ace.builder") {
t.Errorf("%s = %+v, want it closed as failed, naming why and who left it", k, v)
}
}
}
func TestSweepLeavesAKeyOfTheModulesCurrentFingerprintForTheBuildToTakeOver(t *testing.T) { // issue 1660
r := newRig(t, goBundle)
seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // takes it over and builds it
r.b.Wait()
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.By != "novox.builder" {
t.Errorf("build = %+v, want done by novox.builder", v)
}
}
@@ -0,0 +1,204 @@
package build
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again (novox/hq issue 1937): on 2026-10-24 the
// store's 03:30 collection deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which the
// controller had never taken in, and the builder kept them done.
// fakeHeld answers the controller's verb held: the references in gone are gone.
type fakeHeld struct {
mu sync.Mutex
gone map[string]bool
err error
asked [][]string
}
func (f *fakeHeld) stored(_ context.Context, refs []string) ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.asked = append(f.asked, slices.Clone(refs))
if f.err != nil {
return nil, f.err
}
var out []string
for _, r := range refs {
if f.gone[r] {
out = append(out, r)
}
}
return out, nil
}
func (f *fakeHeld) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.asked)
}
// logged keeps what the builder says.
type logged struct {
mu sync.Mutex
lines []string
}
func (l *logged) logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *logged) has(words ...string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
all := true
for _, w := range words {
all = all && strings.Contains(line, w)
}
if all {
return true
}
}
return false
}
const goneRef = "artifact-store://m/tool/blobs/" + digest
func doneKeyNaming(ref string) delivery.BuilderBuild {
return delivery.BuilderBuild{State: delivery.BuildDone, By: "shanks.builder", Manifest: manifestD, Step: "built mesh-build-go on shanks",
Artifacts: []delivery.BuiltArtifact{{Name: "tool", Kind: "bundle", Digest: digest, Reference: ref}}}
}
// 1937/AC3 (unit, the claim): a done key of the trunk's fingerprint whose blob is gone is claimed building as a new
// revision, its step naming the gone reference; a candidate's done key is not asked about.
func TestADoneTrunkKeyWhoseBlobIsGoneIsClaimedAgain(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
rev := putBuild(t, st, key, doneKeyNaming(goneRef))
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Stored: held.stored})
if err != nil || cl != nil || held.asks() != 0 {
t.Fatalf("a candidate's done key: claim %+v, %v, %d asks; want nothing and no ask", cl, err, held.asks())
}
cl, err = c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if err != nil || cl == nil {
t.Fatalf("claim = %+v, %v; want the key claimed again", cl, err)
}
if cl.Rev <= rev || cl.Build.State != delivery.BuildBuilding || !slices.Equal(cl.Gone, []string{goneRef}) {
t.Errorf("claim = %+v (revision was %d); want building as a new revision naming the gone reference", cl, rev)
}
v := valueOf(t, st, key)
if v.State != delivery.BuildBuilding || !strings.Contains(v.Step, "no longer holds") || !strings.Contains(v.Step, goneRef) {
t.Errorf("the key holds %+v; want building, its step naming the gone reference", v)
}
}
// 1937/AC5 (unit, the claim): a done trunk key whose blobs are all held stays done; a store that cannot be asked leaves
// it done and says why; images and packages are not asked about.
func TestADoneTrunkKeyStaysDoneWhileTheStoreHoldsItsBlobs(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
b := doneKeyNaming(goneRef)
b.Artifacts = append(b.Artifacts, delivery.BuiltArtifact{Name: "img", Kind: "image", Digest: digest, Reference: "registry.example/m/img@" + digest})
rev := putBuild(t, st, key, b)
held := &fakeHeld{gone: map[string]bool{}}
if cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored}); err != nil || cl != nil {
t.Fatalf("claim = %+v, %v; want nothing: the store holds the blob", cl, err)
}
if len(held.asked) != 1 || !slices.Equal(held.asked[0], []string{goneRef}) {
t.Errorf("asked %v; want the bundle's reference alone", held.asked)
}
held.err = errors.New("the controller did not answer")
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if cl != nil || err == nil || !strings.Contains(err.Error(), "cannot be asked") || !strings.Contains(err.Error(), "left done") {
t.Errorf("claim = %+v, %v; want the key left done and said", cl, err)
}
if st.revs[key] != rev {
t.Errorf("the key moved to revision %d (was %d)", st.revs[key], rev)
}
}
// 1937/AC3 (unit, the builder): the builder's look at a module whose done key's blob is gone builds it again, says so
// in a log line and in the done key's step, and asks the store once per process while it answers held.
func TestTheBuilderBuildsAgainADoneKeyWhoseBlobIsGone(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
r.b.Stored = held.stored
rev := putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again", v, r.st.revs[r.key], rev)
}
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+goneRef) {
t.Errorf("the done key's step %q does not say it was built again for the gone blob", v.Step)
}
if !log.has(r.key, "built again", goneRef) {
t.Errorf("no log line says the key was built again: %v", log.lines)
}
// The new revision's blob is held: asked once, then not again in this process.
held.mu.Lock()
held.gone = map[string]bool{}
held.mu.Unlock()
rev = r.st.revs[r.key]
asks := held.asks()
r.consider(t, "modules/m")
r.consider(t, "modules/m")
if r.st.revs[r.key] != rev {
t.Errorf("a key whose blob is held was built again")
}
if held.asks() != asks+1 {
t.Errorf("asked the store %d times for two looks at a held key; want once", held.asks()-asks)
}
}
// 1937/AC4 (unit, the builder): artifacts-gone for the module's fingerprint makes the builder ask again, even for a key
// it found held before, and build it again; one for another fingerprint builds nothing.
func TestArtifactsGoneMakesTheBuilderAskAndBuildAgain(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{}}
r.b.Stored = held.stored
putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
if v := r.consider(t, "modules/m"); v.State != delivery.BuildDone || len(r.bm.callsOf("build_go_build")) != 0 {
t.Fatalf("a key whose blob is held was built: %+v", v)
}
rev := r.st.revs[r.key]
held.mu.Lock()
held.gone[goneRef] = true // collected after the builder's look; the controller's take-in found it
held.mu.Unlock()
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Fingerprint: "src2:" + strings.Repeat("9", 64), Gone: []string{goneRef}})
r.b.Wait()
if r.st.revs[r.key] != rev || !log.has("a fingerprint the module does not have now") {
t.Fatalf("an event for another fingerprint moved the key (revision %d, was %d): %v", r.st.revs[r.key], rev, log.lines)
}
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Key: r.key, Fingerprint: fp1, Gone: []string{goneRef}})
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again on the event", v, r.st.revs[r.key], rev)
}
if !log.has("heard artifacts-gone") || !log.has(r.key, "built again", goneRef) {
t.Errorf("the builder did not say what it heard and did: %v", log.lines)
}
}
@@ -0,0 +1,12 @@
package build
import (
"context"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Manifests reads a module's manifest by its digest, the key of the indexer's manifests bucket.
type Manifests interface {
Manifest(ctx context.Context, digest string) (catalogue.Manifest, error)
}
@@ -0,0 +1,118 @@
// Package build is the builder's logic (novox/hq ADR 0453 clause 4, ADR 0478), apart from the bus: what a manifest asks
// to be built and where, the claim of a build key, the choice of build nodes, the build modules' long steps and the run of
// one build. The bus is reached through the small interfaces here, so each part is tested with fakes.
package build
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// ImageBuildModule is the build module that builds an image artifact (novox/hq ADR 0487): buildah, rootless, on the nodes that
// run it. It has no toolchain language; the builder finds its nodes from the node-engine's have.
const ImageBuildModule = "mesh-build-image"
// NoImageBuilderPrefix opens the error of a build key failed because no node runs ImageBuildModule (ADR 0487 clause 5): such a
// key is not built again as a new revision, only when a node reports the module (ClaimOptions.ImageBuilder) or the fingerprint
// changes.
const NoImageBuilderPrefix = "no image builder:"
// Task is one artifact to build: its name and kind in the manifest, the build module that builds it and the language
// of the toolchain it is built with.
type Task struct {
Artifact string
Kind string
Module string
Language string
// From is, for an upstream artifact, the reference its manifest pins, which the image build module copies (ADR 0498
// clause 4).
From string
}
// Plan is what a manifest asks the builder to build, by artifact name.
type Plan struct{ Tasks []Task }
// Modules are the build modules the plan calls, sorted.
func (p Plan) Modules() []string {
seen := map[string]bool{}
var out []string
for _, t := range p.Tasks {
if !seen[t.Module] {
seen[t.Module] = true
out = append(out, t.Module)
}
}
sort.Strings(out)
return out
}
// NeedsToolchain is whether any task the build module builds is built with a toolchain: a bundle is, an archive is not
// (the indexer gives an archive no language, so a fingerprint names no toolchain for it; ADR 0478 clause 1).
func (p Plan) NeedsToolchain(module string) bool {
for _, t := range p.Tasks {
if t.Module == module && t.Kind != catalogue.ArtifactArchive && t.Kind != catalogue.ArtifactImage && t.Kind != catalogue.ArtifactUpstream {
return true
}
}
return false
}
// Refused is the error of a plan the builder does not build: an artifact of a kind (or of a language) no build module
// builds. It opens delivery.RefusedKind, so its key is not built again until the fingerprint changes.
type Refused struct{ Artifact, What string }
func (r *Refused) Error() string {
return fmt.Sprintf("%s%s (artifact %s): no build module builds it; the builder builds Go and TypeScript bundles, archives, images, upstream copies and TypeScript packages", delivery.RefusedKind, r.What, r.Artifact)
}
// NotPinned is the error of a plan with an upstream artifact whose reference names no digest (ADR 0498 clause 4): it opens
// delivery.NotPinned, so its key is not built again until the fingerprint changes.
type NotPinned struct{ Artifact, From string }
func (n *NotPinned) Error() string {
return fmt.Sprintf("%s%s: %q names no digest (@sha256:), so what would be copied is whatever its tag means today", delivery.NotPinned, n.Artifact, n.From)
}
// PlanOf is the plan of a manifest (ADR 0478 clauses 1 and 2, ADR 0498 clauses 4 and 5): a Go bundle and an archive are
// built by the Go build module, an image by the image build module (ADR 0487), which also copies an upstream image pinned
// by its digest, and a TypeScript bundle or package by the Node one, each in the order of the artifacts' names. A manifest
// with an upstream artifact pinned by no digest, or a bundle or a package of another language, or a kind no build module
// builds, is refused as a whole, naming the first such artifact by name. A manifest with no artifact has an empty plan.
func PlanOf(m catalogue.Manifest) (Plan, error) {
if m.Build == nil {
return Plan{}, nil
}
artifacts := append([]catalogue.Artifact(nil), m.Build.Artifacts...)
sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Name < artifacts[j].Name })
var p Plan
for _, a := range artifacts {
switch {
case a.Kind == catalogue.ArtifactArchive:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.GoBuildModule, Language: "go"})
case a.Kind == catalogue.ArtifactBundle && a.Language == "go":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.GoBuildModule, Language: "go"})
case a.Kind == catalogue.ArtifactBundle && a.Language == "typescript":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.NodeBuildModule, Language: "typescript"})
case a.Kind == catalogue.ArtifactImage:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: ImageBuildModule})
case a.Kind == catalogue.ArtifactUpstream && !strings.Contains(a.From, "@sha256:"):
return Plan{}, &NotPinned{Artifact: a.Name, From: a.From}
case a.Kind == catalogue.ArtifactUpstream:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: ImageBuildModule, From: a.From})
case a.Kind == catalogue.ArtifactPackage && a.Language == "typescript":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.NodeBuildModule, Language: "typescript"})
case a.Kind == catalogue.ArtifactBundle:
return Plan{}, &Refused{Artifact: a.Name, What: fmt.Sprintf("bundle of language %q", a.Language)}
case a.Kind == catalogue.ArtifactPackage:
return Plan{}, &Refused{Artifact: a.Name, What: fmt.Sprintf("package of language %q", a.Language)}
default:
return Plan{}, &Refused{Artifact: a.Name, What: a.Kind}
}
}
return p, nil
}
@@ -0,0 +1,96 @@
package build
import (
"errors"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// manifestOf is a manifest holding these artifacts: the plan reads only their name, kind and language.
func manifestOf(artifacts ...catalogue.Artifact) catalogue.Manifest {
return catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: artifacts}}
}
func TestPlanRoutesEachKindToItsBuildModule(t *testing.T) { // 1607/AC2
m := manifestOf(
catalogue.Artifact{Name: "tool", Kind: "bundle", Language: "go"},
catalogue.Artifact{Name: "web", Kind: "bundle", Language: "typescript"},
catalogue.Artifact{Name: "files", Kind: "archive"})
p, err := PlanOf(m)
if err != nil {
t.Fatal(err)
}
want := []Task{
{Artifact: "files", Kind: "archive", Module: buildrun.GoBuildModule, Language: "go"},
{Artifact: "tool", Kind: "bundle", Module: buildrun.GoBuildModule, Language: "go"},
{Artifact: "web", Kind: "bundle", Module: buildrun.NodeBuildModule, Language: "typescript"},
}
if len(p.Tasks) != len(want) {
t.Fatalf("tasks = %+v, want %+v", p.Tasks, want)
}
for i := range want {
if p.Tasks[i] != want[i] {
t.Errorf("task %d = %+v, want %+v", i, p.Tasks[i], want[i])
}
}
if modules := p.Modules(); len(modules) != 2 || modules[0] != buildrun.GoBuildModule || modules[1] != buildrun.NodeBuildModule {
t.Errorf("modules = %v", modules)
}
}
const pinned = "nats:2.11.17-alpine@sha256:3333333333333333333333333333333333333333333333333333333333333333"
func TestPlanSendsAPinnedUpstreamToTheImageBuildModulesCopyAndRefusesAnUnpinnedOne(t *testing.T) { // B3, ADR 0498 clause 4
p, err := PlanOf(manifestOf(catalogue.Artifact{Name: "ok", Kind: "archive"}, catalogue.Artifact{Name: "up", Kind: "upstream", From: pinned}))
if err != nil || len(p.Tasks) != 2 {
t.Fatalf("plan %+v, error %v", p, err)
}
if up := p.Tasks[1]; up.Artifact != "up" || up.Kind != "upstream" || up.Module != ImageBuildModule || up.From != pinned || up.Language != "" {
t.Errorf("task = %+v, want the upstream sent to %s with its pinned reference and no toolchain", up, ImageBuildModule)
}
if p.NeedsToolchain(ImageBuildModule) {
t.Error("a copy needs no toolchain")
}
_, err = PlanOf(manifestOf(catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine"}))
if err == nil || !strings.HasPrefix(err.Error(), delivery.NotPinned+"up") || !delivery.FailedUntilChanged(err.Error()) {
t.Errorf("an upstream with no digest: error = %v, want it to open %q naming the artifact", err, delivery.NotPinned)
}
}
func TestPlanRefusesAPackageOfAnotherLanguageByName(t *testing.T) { // ADR 0498 clause 5
_, err := PlanOf(manifestOf(catalogue.Artifact{Name: "ok", Kind: "archive"}, catalogue.Artifact{Name: "pkg", Kind: "package", Language: "python"}))
var r *Refused
if err == nil || !asRefused(err, &r) || !strings.HasPrefix(err.Error(), delivery.RefusedKind) || !strings.Contains(err.Error(), "python") || !strings.Contains(err.Error(), "pkg") {
t.Errorf("error = %v, want it to open %q naming the artifact and its language", err, delivery.RefusedKind)
}
}
func TestPlanRefusesABundleOfAnotherLanguageAndAModuleWithNoArtifact(t *testing.T) {
if _, err := PlanOf(manifestOf(catalogue.Artifact{Name: "x", Kind: "bundle", Language: "python"})); err == nil || !strings.HasPrefix(err.Error(), delivery.RefusedKind) {
t.Errorf("a python bundle: error = %v, want a refusal", err)
}
p, err := PlanOf(manifestOf())
if err != nil || len(p.Tasks) != 0 {
t.Errorf("no artifact: plan %+v error %v, want an empty plan", p, err)
}
}
func asRefused(err error, target **Refused) bool { return errors.As(err, target) }
func TestPlanSendsAnImageToMeshBuildImageWithNoToolchain(t *testing.T) { // 1705a, ADR 0487 clause 5
p, err := PlanOf(manifestOf(catalogue.Artifact{Name: "img", Kind: "image"}, catalogue.Artifact{Name: "files", Kind: "archive"}))
if err != nil || len(p.Tasks) != 2 {
t.Fatalf("plan %+v, error %v", p, err)
}
if p.Tasks[1].Artifact != "img" || p.Tasks[1].Module != ImageBuildModule {
t.Errorf("task = %+v, want the image on %s", p.Tasks[1], ImageBuildModule)
}
if p.NeedsToolchain(ImageBuildModule) {
t.Error("an image needs no toolchain")
}
}
@@ -0,0 +1,403 @@
package build
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// R4e: references in the key (novox/hq ADR 0498), upstream copies, packages published from the trunk, the package wait
// (ADR 0500 clause 6) and one node per fingerprint (ADR 0478 clause 4).
func TestACandidatesBundleKeyNamesItsReferenceAndTheTrunkBuildsItNoMore(t *testing.T) { // B1, ADR 0498 clause 3 and its check 3
r := newRig(t, goBundle, archiveA)
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone {
t.Fatalf("build = %+v, want done", v)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("a candidate's bundle and archive name no reference: %v (%+v)", missing, v.Artifacts)
}
rev := r.st.revs[r.key]
r.bm.mu.Lock()
r.bm.calls = nil
r.bm.mu.Unlock()
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the merge: the trunk now names the same fingerprint
r.b.Wait()
if n := len(r.bm.calls); n != 0 || r.st.revs[r.key] != rev {
t.Errorf("the trunk built its candidate's fingerprint again: %d calls, revision %d (was %d)", n, r.st.revs[r.key], rev)
}
}
func TestABundleWhosePushAnsweredNoReferenceFailsNotStoredAndIsBuiltAgain(t *testing.T) { // B1, ADR 0498 clauses 1 and 2
r := newRig(t, goBundle)
r.bm.noRef["ace/tool"] = true
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "tool") {
t.Fatalf("build = %+v, want failed opening %q and naming the artifact", v, delivery.NotStored)
}
delete(r.bm.noRef, "ace/tool")
if v = r.consider(t, "modules/m"); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want a not-stored key built again as a new revision", v)
}
}
func TestAPushRefusedByTheStoreFailsTheKeyOpeningNotStored(t *testing.T) { // B1, ADR 0498 clause 2
r := newRig(t, goBundle)
r.bm.fails["ace/build_go_build"] = delivery.NotStored + "tool: the artifact store answered 507 insufficient storage"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "507") || !strings.Contains(v.Error, "ace") {
t.Fatalf("build = %+v, want failed opening %q with the store's words and the node", v, delivery.NotStored)
}
}
func TestOnlyATrunkKeysBuildsAreAskedToPublish(t *testing.T) { // B1, the lead's interface (a): publish only for a trunk key
r := newRig(t, goBundle)
r.consider(t, "modules/m")
for _, c := range r.bm.callsOf("build_go_build") {
if c.body.(map[string]any)["publish"] != true {
t.Errorf("trunk build args = %+v, want publish", c.body)
}
}
c := newRig(t, goBundle)
trunkModule(t, c)
c.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
c.b.Wait()
for _, call := range c.bm.callsOf("build_go_build") {
if _, ok := call.body.(map[string]any)["publish"]; ok {
t.Errorf("candidate build args = %+v, want no publish", call.body)
}
}
}
// putBuild writes a builder key's value as the store would hold it.
func putBuild(t *testing.T, st *memStore, key string, b delivery.BuilderBuild) uint64 {
t.Helper()
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
t.Fatal(err)
}
st.mu.Lock()
defer st.mu.Unlock()
return st.put(key, data)
}
func TestADoneKeyWrittenBeforeR4eIsBuiltAgainOnceAsANewRevision(t *testing.T) { // B2, ADR 0498 clause 8
r := newRig(t, goBundle, archiveA)
rev := putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: manifestD, Step: "built mesh-build-go on ace and novox",
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Digest: digest}, {Name: "tool", Kind: "bundle", Digest: digest}}})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev {
t.Fatalf("build = %+v at revision %d (was %d), want built again as a new revision", v, r.st.revs[r.key], rev)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("the new revision names no reference for %v", missing)
}
rev = r.st.revs[r.key]
r.b.tried = map[string]bool{} // a builder started anew looks again
r.consider(t, "modules/m")
if r.st.revs[r.key] != rev {
t.Errorf("a key whose artifacts all name references was built again (revision %d, was %d): once only", r.st.revs[r.key], rev)
}
}
func TestAPinnedUpstreamIsCopiedOnceOnOneImageNodeWithNoFetchAndItsKeyNamesTheReference(t *testing.T) { // B4, ADR 0498 clause 4
up := catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest}
r := newRig(t, up)
r.puller = "ace"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Fatalf("build = %+v, want done with the copy", v)
}
if a := v.Artifacts[0]; a.Kind != "upstream" || a.Digest != digest || a.Reference != "registry.example/m/up@"+digest {
t.Errorf("artifact = %+v, want the upstream's digest and the registry's reference", a)
}
copies := r.bm.callsOf("build_image_copy")
if len(copies) != 1 {
t.Fatalf("%d copies, want one on one node", len(copies))
}
if args := copies[0].body.(map[string]any); args["module"] != "m" || args["artifact"] != "up" || args["from"] != up.From || len(args) != 3 {
t.Errorf("copy args = %+v, want the module's name, the artifact and the pinned reference", args)
}
if n := len(r.bm.callsOf("build_image_fetch")) + len(r.bm.callsOf("build_image_build")); n != 0 {
t.Errorf("%d fetch or build calls for a copy, want none", n)
}
}
func TestAnUpstreamBesideAnImageIsCopiedAndTheImageFetchedAndBuiltOnTheSameNode(t *testing.T) { // B4
r := newRig(t, imageArt, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.puller = "ace"
v := r.consider(t, "modules/m")
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, missing references %v", v, missing)
}
if f, b, c := len(r.bm.callsOf("build_image_fetch")), len(r.bm.callsOf("build_image_build")), len(r.bm.callsOf("build_image_copy")); f != 1 || b != 1 || c != 1 {
t.Errorf("fetch %d, build %d, copy %d; want one each", f, b, c)
}
}
func TestACopyThatFailsNotStoredFailsTheKeyAndIsBuiltAgain(t *testing.T) { // B4, ADR 0498 clause 2
r := newRig(t, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.puller = "ace"
r.bm.fails["ace/build_image_copy"] = delivery.NotStored + "up: the registry answered 500"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || delivery.FailedUntilChanged(v.Error) {
t.Fatalf("build = %+v, want failed 'not stored:' and built again", v)
}
delete(r.bm.fails, "ace/build_image_copy")
if v = r.consider(t, "modules/m"); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done on the new revision", v)
}
}
var tsPackage = catalogue.Artifact{Name: "lib", Kind: "package", Language: "typescript"}
func TestATypeScriptPackageIsBuiltByTheNodeModuleAndPublishedOnlyForTheTrunk(t *testing.T) { // B5, ADR 0498 clause 5, ADR 0500 clause 4
c := newRig(t, tsPackage)
c.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, c)
c.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
c.b.Wait()
v := valueOf(t, c.st, c.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "package" || v.Artifacts[0].Reference != "" {
t.Fatalf("candidate build = %+v, want done with the packed digest and no reference", v)
}
for _, call := range c.bm.callsOf("build_node_build") {
if _, ok := call.body.(map[string]any)["publish"]; ok {
t.Errorf("a candidate's package build was asked to publish: %+v", call.body)
}
}
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
v = r.consider(t, "modules/m")
if v.State != delivery.BuildDone || v.Artifacts[0].Reference != "@novox/mesh-sdk@0.1.15" {
t.Fatalf("trunk build = %+v, want done naming the published version", v)
}
if n := len(r.bm.callsOf("build_node_build")); n == 0 {
t.Error("the package was not built by the Node build module")
}
}
func TestAVersionTakenFailsTheKeyUntilTheFingerprintChanges(t *testing.T) { // B5, ADR 0498 clause 5
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
r.bm.fails["ace/build_node_build"] = delivery.VersionTaken + "@novox/mesh-sdk@0.1.15 is published with another packed file"
r.bm.fails["novox/build_node_build"] = r.bm.fails["ace/build_node_build"]
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.VersionTaken) || !strings.Contains(v.Error, "0.1.15") {
t.Fatalf("build = %+v, want failed opening %q naming the version", v, delivery.VersionTaken)
}
writes := r.st.writes
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Errorf("a version-taken key was built again (%d writes, was %d)", r.st.writes, writes)
}
}
func TestACandidatesPackageIsPublishedOnceWhenTheTrunkNamesItsFingerprint(t *testing.T) { // B6, ADR 0498 clause 6
r := newRig(t, tsPackage, tsBundle)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Fatalf("candidate build = %+v", v)
}
rev := r.st.revs[r.key]
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the merge names the candidate's fingerprint
r.b.Wait()
v := valueOf(t, r.st, r.key)
if r.st.revs[r.key] <= rev {
t.Fatalf("the trunk did not build the unpublished package again (revision %d, was %d)", r.st.revs[r.key], rev)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, missing references %v", v, missing)
}
rev = r.st.revs[r.key]
r.b.tried = map[string]bool{}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if r.st.revs[r.key] != rev {
t.Error("a published package was built again: once only")
}
}
func TestAPublishThatFailsVersionTakenOnTheTrunkKeepsThePrefixFirst(t *testing.T) { // B6, ADR 0498 clauses 5 and 6
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
for _, n := range []string{"ace", "novox"} {
r.bm.fails[n+"/build_node_build"] = delivery.VersionTaken + "@novox/mesh-sdk@0.1.15 is published with another packed file"
}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.VersionTaken) {
t.Errorf("build = %+v, want failed opening %q", v, delivery.VersionTaken)
}
}
// sdkModule adds to the rig a module "sdk" whose manifest (digest b) holds the TypeScript package lib, published as ref.
func sdkModule(t *testing.T, r *rig, ref string) []byte {
t.Helper()
r.mf.byDigest[mdigest('b')] = catalogue.Manifest{Module: "sdk", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{tsPackage}}}
r.bm.packages["lib"] = ref
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: "8a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP2: "src2:" + hex64('5'),
Manifest: mdigest('b'), Path: ".", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
return data
}
func TestAModuleWhoseLockNamesAnUnpublishedPackageWaitsQueuedAndIsBuiltOncePublished(t *testing.T) { // B7, ADR 0500 clause 6
const ref = "@novox/mesh-sdk@0.1.15"
r := newRig(t, tsBundle)
for _, n := range []string{"ace", "novox"} {
r.bm.fails[n+"/build_node_fetch"] = waitsFor + ref + ": the lock names a version the registry does not hold"
}
v := r.consider(t, "modules/m")
if v.State != delivery.BuildQueued || v.Waits != ref || v.Error != "" {
t.Fatalf("build = %+v, want queued, waiting for %s, not failed", v, ref)
}
delete(r.bm.fails, "ace/build_node_fetch")
delete(r.bm.fails, "novox/build_node_fetch")
writes := r.st.writes
r.b.tried = map[string]bool{}
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Fatalf("a key waiting for an unpublished package was claimed (%d writes, was %d)", r.st.writes, writes)
}
r.b.Module(context.Background(), "sdk", sdkModule(t, r, ref)) // the package's trunk key is built and published
r.b.Wait()
if v = valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.Waits != "" {
t.Errorf("build = %+v, want built once the package it waits for is published", v)
}
}
func TestAKeyWaitingForAPackageAlreadyPublishedIsBuiltWhenTheBuilderStarts(t *testing.T) { // B7: the scan at start
const ref = "@novox/mesh-sdk@0.1.15"
r := newRig(t, tsBundle)
putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildQueued, By: "novox.builder", Manifest: manifestD, Waits: ref, Step: "waits"})
sdkKey, _ := delivery.BuilderBuildKey("sdk", "src2:"+hex64('5'))
putBuild(t, r.st, sdkKey, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: mdigest('b'), Step: "built",
Artifacts: []delivery.BuiltArtifact{{Name: "lib", Kind: "package", Digest: digest, Reference: ref}}})
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want built: the package it waits for was published before the builder started", v)
}
}
func TestTheFetchOfAModuleNamingItsPathsCarriesItsDirectoryAndThosePaths(t *testing.T) { // the lead's interface (c), ADR 0499 clause 4
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
paths := map[string]string{"src": hex40('1'), "package.json": hex40('2')}
tree, err := delivery.NamedTree(manifestD, paths)
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP2: fp1, Manifest: manifestD, Path: ".",
Toolchains: agreedToolchains, Inputs2: &delivery.KeptInputs2{Tree: tree, Paths: paths}})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
r.b.Wait()
f := r.bm.callsOf("build_node_fetch")
if len(f) == 0 {
t.Fatal("no fetch")
}
req, err := buildstep.ParseFetchRequest(f[0].body.(map[string]any))
if err != nil || req.Dir != "." || strings.Join(req.Paths, ",") != "package.json,src" || req.Commit != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" {
t.Errorf("fetch request = %+v, %v; want dir . and the kept paths, sorted", req, err)
}
r2 := newRig(t, goBundle)
r2.consider(t, "modules/m")
if a := r2.bm.callsOf("build_go_fetch")[0].body.(map[string]any); a["dir"] != nil || a["paths"] != nil {
t.Errorf("fetch args = %+v, want no dir or paths for a module naming none", a)
}
}
func hex40(c byte) string { return strings.Repeat(string(c), 40) }
func TestOneNodeBuildsEachFingerprint(t *testing.T) { // B9, ADR 0478 clause 4: "From R4e one node builds each fingerprint"
r := newRig(t, goBundle, archiveA, tsBundle) // ace and novox both qualify for both build modules
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || strings.Contains(v.Step, " and ") || strings.HasPrefix(v.Step, "one node (") {
t.Fatalf("build = %+v, want done on one node of each build module, unmarked", v)
}
if f, b := len(r.bm.callsOf("build_go_fetch")), len(r.bm.callsOf("build_go_build")); f != 1 || b != 2 {
t.Errorf("go fetches %d, builds %d; want one fetch and one build of each artifact, on one node", f, b)
}
if f := len(r.bm.callsOf("build_node_fetch")); f != 1 {
t.Errorf("node fetches %d, want one", f)
}
}
func TestAKeyDoneOnOneNodeIsNeverBuiltAgainForASecondNode(t *testing.T) { // B9: the second-node retake is gone
r := newRig(t, goBundle)
putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: manifestD,
Step: "one node (mesh-build-go): built mesh-build-go on one node, novox",
Artifacts: []delivery.BuiltArtifact{{Name: "tool", Kind: "bundle", Digest: digest, Reference: "artifact-store://m/tool/blobs/" + digest}}})
writes := r.st.writes
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.NodesChanged(context.Background())
r.b.Wait()
if r.st.writes != writes || len(r.bm.calls) != 0 {
t.Errorf("a key done on one node was built again (%d writes, was %d; %d calls)", r.st.writes, writes, len(r.bm.calls))
}
}
func TestACandidatesFetchNamesTheNamedPathsTheIndexerGaveIt(t *testing.T) { // ADR 0499 clause 4, mesh-sdk CandidateModule.Paths
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: headX}, Modules: []string{"m"},
Fps: map[string]delivery.CandidateModule{"m": {FP2: fp1, Manifest: mdigest('a'), Paths: map[string]string{"src": hex40('1'), "package.json": hex40('2')}}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
data, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
r.b.Candidate(context.Background(), "feat%2Fx", data)
r.b.Wait()
f := r.bm.callsOf("build_node_fetch")
if len(f) == 0 {
t.Fatal("no fetch")
}
req, err := buildstep.ParseFetchRequest(f[0].body.(map[string]any))
if err != nil || req.Dir != "modules/m" || strings.Join(req.Paths, ",") != "package.json,src" || req.Base == "" {
t.Errorf("fetch request = %+v, %v; want the candidate's base, the module's directory and its named paths", req, err)
}
}
// 1886/AC5: the builder tells who answered a reference: the build module whose step named one, and none for a step that
// answered a digest alone (a build module from before R4e).
func TestTheBuilderTellsWhichBuildModuleAnsweredAReference(t *testing.T) {
r := newRig(t, goBundle)
var answered []string
r.b.Answered = func(module string) { answered = append(answered, module) }
r.bm.noRef["ace/tool"] = true
r.consider(t, "modules/m")
if len(answered) != 0 {
t.Fatalf("a step that answered no reference was told as one: %v", answered)
}
delete(r.bm.noRef, "ace/tool")
r.consider(t, "modules/m")
if len(answered) != 1 || answered[0] != "mesh-build-go" {
t.Fatalf("the build module that answered a reference: %v, want mesh-build-go", answered)
}
}
@@ -0,0 +1,84 @@
package build
import (
"context"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func haveValue(t *testing.T, phase delivery.Phase, health delivery.Health) []byte {
t.Helper()
b, err := delivery.EncodeHave(delivery.Have{Phase: phase, Health: health})
if err != nil {
t.Fatal(err)
}
return b
}
// 1. A module of an archive only: the real indexer gives an archive no language (inputs.go languagesOf), so its
// fingerprint names no toolchain, and the builder builds it on the build module's nodes with no toolchain agreed.
func TestAnArchiveOnlyModuleIsBuiltAlthoughItsFingerprintNamesNoToolchain(t *testing.T) { // review round 4, 1
r := newRig(t, archiveA)
v := func() delivery.BuilderBuild {
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", nil))
r.b.Wait()
if _, ok := r.st.vals[r.key]; !ok {
t.Fatal("no build key was written for the archive-only module")
}
return valueOf(t, r.st, r.key)
}()
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || len(v.Toolchains) != 0 {
t.Fatalf("build = %+v, want done with the archive and no toolchain named", v)
}
if n := len(r.bm.callsOf("build_go_build")); n != 1 {
t.Errorf("%d go builds, want one, on one node (ADR 0478 clause 4)", n)
}
}
func TestABundleAndAnArchiveOfOneBuildModuleStillNeedTheAgreedToolchain(t *testing.T) { // review round 4, 1
p, err := PlanOf(catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle, archiveA}}})
if err != nil || !p.NeedsToolchain(buildrun.GoBuildModule) {
t.Errorf("a plan with a Go bundle: %v, %v", p.NeedsToolchain(buildrun.GoBuildModule), err)
}
p, _ = PlanOf(catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{archiveA}}})
if p.NeedsToolchain(buildrun.GoBuildModule) {
t.Error("a plan of an archive needs a toolchain")
}
}
func TestClaimLeavesAKeyFailedForDifferingDigestsAlone(t *testing.T) { // review round 4, 2
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, Error: buildrun.DigestsDifferPrefix + " the nodes built tool to different digests: ace x; novox y"})
writes := st.writes
got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{TakeOver: true})
if err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, %d writes (was %d), want none and no write", got, err, st.writes, writes)
}
}
// 6. A key found building is taken over, whoever holds it.
func TestABuildLeftBuildingByAnyBuilderIsTakenOverAtStart(t *testing.T) { // review round 4, 6; 1607/AC3, ADR 0478 clauses 5, 6
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.a"})
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got == nil {
t.Errorf("own building key: claim = %v, %v, want it taken again", got, err)
}
if got, err := (Claimer{Store: st, By: "ace.b"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{TakeOver: true}); err != nil || got == nil || got.Build.By != "novox.a" {
t.Errorf("another node's building key at start: claim = %v, %v, want it taken over", got, err)
}
if got, _ := (Claimer{Store: st, By: "ace.b"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); got != nil {
t.Errorf("another builder's building key after the start was taken: a running builder's key is its own")
}
}
func TestABuilderFindingAKeyBuildingByAnotherNodeBuildsItOnStart(t *testing.T) { // review round 4, 6
r := newRig(t, goBundle)
seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || v.By != "novox.builder" {
t.Errorf("build = %+v, want it taken over and done by novox.builder", v)
}
}
@@ -0,0 +1,812 @@
package build
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// ToolchainEntries are the nodes' toolchain entries of a language (buildrun.ToolchainSet).
type ToolchainEntries interface {
Entries(language string) []delivery.NodeToolchain
}
// Builder builds, dark, every new-form fingerprint the indexer writes (novox/hq ADR 0453 clause 4, ADR 0478): it is
// handed each modules value as the watch of the indexer's bucket sees it, and for each module with a fingerprint and an
// artifact to build claims the fingerprint's build key, builds it by asking the build seats (one ask per build module,
// pulled by whichever build node has a free slot: ADR 0525), and writes the key done or failed. It holds no timer; a build starts from a modules value or a
// change of the toolchains.
type Builder struct {
Claimer Claimer
// Asks puts each build module's part of a build as one ask on its build seat's stream and waits for its step (novox/hq
// ADR 0525 §1, §3, §6; issue 1887 A4): the builder chooses no node, whichever build node pulls the ask builds it.
Asks interface {
Put(ctx context.Context, a delivery.StepAsk) error
Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, error)
}
// Held says whether an ask a key names is on its stream or a build node wrote a step for it (issue 1910): a key taken
// over at start whose ask is neither was left by a builder that stopped between naming the ask and putting it, and
// the ask is put then, under the same id. Nil takes every named ask as held.
Held interface {
Held(ctx context.Context, a delivery.StepAsk) (bool, error)
}
Toolchains ToolchainEntries
Manifests Manifests
// Limit is how many builds the builder has asked at once; their asks wait in the build seats' streams, which are the
// backlog (ADR 0525 §9), not here. 256 when 0.
Limit int
Logf func(format string, args ...any)
// Answered is told the build module whose step answered a reference (issue 1886): what the builder says it writes
// rests on it. Nil tells nobody.
Answered func(buildModule string)
// Stored answers which references the artifact store no longer holds (the controller's verb held; issue 1937): a done
// key of the trunk's fingerprint is asked before it is left done (ClaimOptions.Stored). Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
once sync.Once
sem chan struct{}
wg sync.WaitGroup
mu sync.Mutex
known map[string]delivery.Module // by modules key
inflight map[string]bool // by build key
again map[string]delivery.Module // by build key: the newest value that arrived while the key was being built
plans map[string]planned // by manifest digest
cands map[string]delivery.Candidate // by candidates key (candidates.go)
via map[string]via // by build key: a candidate's fetch (candidates.go)
trunkFP map[string]bool // by build key: a trunk module value carries this fingerprint
tried map[string]bool // by build key: looked at since this process started
// published are the package versions `<name>@<version>` a done key of this bucket names as a package's reference: a key
// waiting for one is built once it is here (ADR 0500 clause 6).
published map[string]bool
// held are the done keys' references the store answered held, by `<key> <references>`, asked once per process: a
// blob collected later is found by the take-in, whose artifacts-gone forgets the key here (issue 1937).
held map[string]bool
}
type planned struct {
plan Plan
err error
}
func (b *Builder) init() {
b.once.Do(func() {
limit := b.Limit
if limit <= 0 {
limit = 256
}
b.sem = make(chan struct{}, limit)
b.cands, b.via, b.trunkFP = map[string]delivery.Candidate{}, map[string]via{}, map[string]bool{}
b.known, b.inflight, b.again, b.plans, b.tried = map[string]delivery.Module{}, map[string]bool{}, map[string]delivery.Module{}, map[string]planned{}, map[string]bool{}
b.published, b.held = map[string]bool{}, map[string]bool{}
})
}
func (b *Builder) logf(format string, args ...any) {
if b.Logf != nil {
b.Logf(format, args...)
}
}
// Wait returns when every build started so far has ended.
func (b *Builder) Wait() { b.wg.Wait() }
// Module takes one value of the indexer's modules bucket in (the key is the module's name) and builds it if it is
// to be built. A value that is not a module is said and left.
func (b *Builder) Module(ctx context.Context, key string, value []byte) {
b.init()
m, err := delivery.DecodeModule(value)
if err != nil {
b.logf("modules.%s holds what is not a module, left: %v", key, err)
return
}
b.mu.Lock()
b.known[key] = m
if bk, err := delivery.BuilderBuildKey(key, m.FP2); err == nil && m.FP2 != "" {
// A fingerprint the trunk carries is built from the trunk's commit, whoever asked for it first.
b.trunkFP[bk] = true
delete(b.via, bk)
}
b.mu.Unlock()
b.consider(ctx, key, m)
}
// NodesChanged is Reconsider after a toolchain moved, so a build that waited for the nodes to agree on one may go on.
func (b *Builder) NodesChanged(ctx context.Context) { b.Reconsider(ctx) }
// Sweep reads, at start, every key of the bucket: it keeps the package versions done keys name as published (a key
// waiting for one is then looked at again, ADR 0500 clause 6), and closes every key left building for a fingerprint its
// module no longer has (issue 1660, ADR 0478 clause 6): the key can never be built, since nothing asks for it, and would
// stay building for ever. It fails the key naming who left it. A key of the fingerprint the module has now is not
// touched: the build takes it over. Call it once the modules bucket has been read, so every module is known.
func (b *Builder) Sweep(ctx context.Context) error {
b.init()
keys, err := b.Claimer.Store.Keys(ctx)
if err != nil {
return err
}
b.mu.Lock()
current := map[string]bool{}
for name, m := range b.known {
if m.FP2 != "" {
if k, err := delivery.BuilderBuildKey(name, m.FP2); err == nil {
current[k] = true
}
}
}
b.mu.Unlock()
newly := false
for _, key := range keys {
value, rev, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil || !found {
continue
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
continue
}
newly = b.notePublished(cur) || newly
if current[key] || cur.State != delivery.BuildBuilding {
continue
}
closed := delivery.BuilderBuild{State: delivery.BuildFailed, By: b.Claimer.By, Manifest: cur.Manifest, Step: cur.Step,
Error: fmt.Sprintf("closed at start: left building by %s, and the module no longer has this fingerprint, so nothing will build it", cur.By)}
data, err := delivery.EncodeBuilderBuild(closed)
if err != nil {
return err
}
if _, err := b.Claimer.Store.Update(ctx, key, data, rev); err != nil && !errors.Is(err, ErrConflict) {
return fmt.Errorf("%s: %w", key, err)
}
b.logf("%s: closed, left building by %s for a fingerprint the module no longer has", key, cur.By)
}
if newly {
// A key read before the scan may wait for a package the scan found published (ADR 0500 clause 6).
b.Reconsider(ctx)
}
return nil
}
// notePublished keeps the package versions a done build names as references, and says whether one was new.
func (b *Builder) notePublished(v delivery.BuilderBuild) bool {
if v.State != delivery.BuildDone {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
newly := false
for _, a := range v.Artifacts {
if a.Kind == catalogue.ArtifactPackage && a.Reference != "" && !b.published[a.Reference] {
b.published[a.Reference] = true
newly = true
}
}
return newly
}
func (b *Builder) isPublished(ref string) bool {
b.mu.Lock()
defer b.mu.Unlock()
return b.published[ref]
}
// Forget takes a module out (its key was deleted).
func (b *Builder) Forget(key string) {
b.init()
b.mu.Lock()
delete(b.known, key)
b.mu.Unlock()
}
// storedOnce is Stored for one key, asked once per process while the store answers every reference held.
func (b *Builder) storedOnce(key string) func(ctx context.Context, references []string) ([]string, error) {
if b.Stored == nil {
return nil
}
return func(ctx context.Context, references []string) ([]string, error) {
id := key + " " + strings.Join(references, " ")
b.mu.Lock()
known := b.held[id]
b.mu.Unlock()
if known {
return nil, nil
}
gone, err := b.Stored(ctx, references)
if err == nil && len(gone) == 0 {
b.mu.Lock()
b.held[id] = true
b.mu.Unlock()
}
return gone, err
}
}
// ArtifactsGone takes the controller's event artifacts-gone in (issue 1937): a take-in found that the store does not
// hold what the module's key names. The module is looked at again, which asks the store (ClaimOptions.Stored) and
// builds the key again; an event for a fingerprint the module no longer has is said and left. The event only wakes the
// builder: what is built rests on the store's answer, never on the event alone.
func (b *Builder) ArtifactsGone(ctx context.Context, e delivery.ArtifactsGone) {
b.init()
b.mu.Lock()
m, ok := b.known[e.Module]
if key, err := delivery.BuilderBuildKey(e.Module, e.Fingerprint); err == nil {
for id := range b.held {
if strings.HasPrefix(id, key+" ") {
delete(b.held, id)
}
}
}
b.mu.Unlock()
if !ok || m.FP2 != e.Fingerprint {
b.logf("%s: heard %s for %s, a fingerprint the module does not have now: nothing to build", e.Module,
delivery.ArtifactsGoneEvent, e.Fingerprint)
return
}
b.logf("%s: heard %s (%s): asking the artifact store and building the key again", e.Module,
delivery.ArtifactsGoneEvent, strings.Join(e.Gone, ", "))
b.consider(ctx, e.Module, m)
}
// Reconsider looks at every module again: the toolchains changed, so a build that failed for them may build now.
func (b *Builder) Reconsider(ctx context.Context) {
b.init()
b.mu.Lock()
known := make(map[string]delivery.Module, len(b.known))
for k, m := range b.known {
known[k] = m
}
b.mu.Unlock()
for _, k := range sortedKeys(known) {
b.consider(ctx, k, known[k])
}
b.mu.Lock()
cands := make([]delivery.Candidate, 0, len(b.cands))
for _, c := range b.cands {
cands = append(cands, c)
}
b.mu.Unlock()
sort.Slice(cands, func(i, j int) bool { return cands[i].Group < cands[j].Group })
for _, c := range cands {
b.considerCandidate(ctx, c)
}
}
func sortedKeys(m map[string]delivery.Module) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// planOf is the plan of the manifest with this digest, read once: a manifest by digest never changes. A manifest that
// cannot be read is not kept, so it is read again when the module is.
func (b *Builder) planOf(ctx context.Context, digest string) (Plan, error, bool) {
b.mu.Lock()
p, ok := b.plans[digest]
b.mu.Unlock()
if ok {
return p.plan, p.err, true
}
m, err := b.Manifests.Manifest(ctx, digest)
if err != nil {
return Plan{}, fmt.Errorf("its manifest %s cannot be read: %w", digest, err), false
}
plan, err := PlanOf(m)
b.mu.Lock()
b.plans[digest] = planned{plan, err}
b.mu.Unlock()
return plan, err, true
}
func (b *Builder) consider(ctx context.Context, module string, m delivery.Module) {
if m.FP2 == "" || m.Manifest == "" {
return
}
key, err := delivery.BuilderBuildKey(module, m.FP2)
if err != nil {
b.logf("modules.%s: %v", module, err)
return
}
b.mu.Lock()
if b.inflight[key] {
b.again[key] = m // looked at once more when the build ends, so a change is never lost
b.mu.Unlock()
return
}
b.inflight[key] = true
b.mu.Unlock()
b.wg.Add(1)
go func() {
defer b.wg.Done()
b.sem <- struct{}{}
defer func() { <-b.sem }()
for {
b.try(ctx, module, m)
b.mu.Lock()
next, more := b.again[key]
delete(b.again, key)
if !more || ctx.Err() != nil {
delete(b.inflight, key)
b.mu.Unlock()
return
}
b.mu.Unlock()
m = next
}
}()
}
// try claims the fingerprint and builds it, writing the key done or failed. A build that ends because ctx did is left
// building by this builder: it takes it again when it starts (ADR 0478 clause 5).
func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
plan, planErr, read := b.planOf(ctx, m.Manifest)
nothing := read && planErr == nil && len(plan.Tasks) == 0
if planErr == nil && !nothing {
if why := b.stale(m, plan); why != "" {
// The fingerprint names a toolchain the build nodes no longer agree on: a new fingerprint is coming, and a build
// now would be of this source with another toolchain (ADR 0477 clause 2). Nothing is written.
b.logf("%s at %s is not built: %s", module, m.FP2, why)
return
}
}
key, _ := delivery.BuilderBuildKey(module, m.FP2)
b.mu.Lock()
trunk := b.trunkFP[key]
first := !b.tried[key]
b.tried[key] = true
b.mu.Unlock()
cl, err := b.Claimer.Claim(ctx, module, m.FP2, m.Manifest, ClaimOptions{TakeOver: first, ImageBuilder: imageSeat, Trunk: trunk, Published: b.isPublished,
Stored: b.storedOnce(key)})
if err != nil {
b.logf("%s: %v", module, err)
return
}
if cl == nil {
return
}
if cl.Unasked != nil {
b.logf("%v", cl.Unasked)
}
if len(cl.Gone) > 0 {
// Said aloud (issue 1937): the key was done, and the store's collector deleted what it names.
b.logf("%s: built again: the artifact store no longer holds %s", cl.Key, strings.Join(cl.Gone, ", "))
}
// A done key taken again builds only what is missing: the image or the package built unpublished as a candidate's; the
// other artifacts stay as the done key has them (ADR 0487 clauses 5 and 8, ADR 0498 clause 6).
var prior *delivery.BuilderBuild
var only map[string]bool
publishing := false
if cl.Build.State == delivery.BuildDone {
prior, only = &cl.Build, map[string]bool{}
if trunk {
for _, bm := range publishable(cl.Build, imageSeat) {
only[bm] = true
publishing = true
}
}
}
var result delivery.BuilderBuild
if nothing {
// A manifest with no artifact is built by building nothing: its key is done with no artifact, claimed and written as
// any other, so the controller has a done key to register it from (issue 1613, ADR 0478 clause 1, ADR 0501 clause 1).
result = delivery.BuilderBuild{State: delivery.BuildDone, Step: "nothing to build: the manifest names no artifact"}
} else {
result = b.build(ctx, module, m, plan, planErr, prior, only, b.notes(cl))
}
if ctx.Err() != nil {
return
}
if result.State == "" {
b.logf("%s: left building, the toolchains moved under it", cl.Key)
return
}
result.By, result.Manifest = b.Claimer.By, m.Manifest
if len(cl.Gone) > 0 && result.State == delivery.BuildDone {
result.Step += " (built again: the artifact store no longer held " + strings.Join(cl.Gone, ", ") + ")"
}
if publishing && result.State == delivery.BuildFailed {
// The trunk's image or package was built only as a candidate's and its push or publish failed: the key says so, so the builds view, the
// checker and the failed-key retake see it, and a look does not build the image again for nothing.
result.Error = keyError([]string{"publishing what the merged commit built failed: " + result.Error})
}
if err := b.Claimer.Finish(ctx, cl, result); err != nil {
b.logf("%s: the build could not be written: %v", cl.Key, err)
return
}
if result.State == delivery.BuildFailed {
b.logf("%s: failed: %s", cl.Key, result.Error)
}
if result.State == delivery.BuildQueued {
b.logf("%s: waits for the package %s to be published", cl.Key, result.Waits)
}
if b.notePublished(result) {
// A key may wait for the package this build published (ADR 0500 clause 6).
b.Reconsider(ctx)
}
}
// stale is why the module's fingerprint is not built now, or empty when it is: the toolchain each language of the plan
// has agreed (novox/hq ADR 0465) must be the one the fingerprint was computed with (Module.Toolchains).
func (b *Builder) stale(m delivery.Module, plan Plan) string {
for _, bm := range plan.Modules() {
if !plan.NeedsToolchain(bm) {
continue // an archive is packed with no toolchain
}
language := buildrun.LanguageOf[bm]
tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language))
if none != "" {
return fmt.Sprintf("the %s toolchain is not agreed: %s", language, none)
}
want, ok := m.Toolchains[language]
if !ok || want.Version != tc.Version || want.Recipe != tc.Recipe {
return fmt.Sprintf("the fingerprint names %s %s (recipe %s) and the nodes agree on %s (recipe %s)", language, want.Version, want.Recipe, tc.Version, tc.Recipe)
}
}
return ""
}
func failed(format string, args ...any) delivery.BuilderBuild {
return delivery.BuilderBuild{State: delivery.BuildFailed, Error: fmt.Sprintf(format, args...)}
}
// lane is the build of a plan's tasks of one build module on one node.
type lane struct {
module, node string // node is the one that pulled the lane's ask
tasks []Task
toolchain *delivery.ToolchainUsed
digests map[string]string // artifact -> digest
references map[string]string // artifact -> the reference an image was pushed to (a trunk build only)
err error
}
func (b *Builder) build(ctx context.Context, module string, m delivery.Module, plan Plan, planErr error, prior *delivery.BuilderBuild, only map[string]bool, asked *askNotes) delivery.BuilderBuild {
switch {
case planErr != nil:
return failed("%s", planErr)
case m.Path == "":
// Empty is not known (a root module's path is "."): the indexer writes it again (issue 1615).
return failed("the module's key names no directory (path), so the build tools cannot be told where %s lives", module)
}
repository := delivery.RepositoryName(m.Repository)
var lanes []*lane
used := map[string]delivery.ToolchainUsed{}
var kept []string // build modules whose artifacts stay as the prior done key has them
for _, bm := range plan.Modules() {
if prior != nil && len(only) > 0 && !only[bm] {
kept = append(kept, bm)
continue
}
var toolchain *delivery.ToolchainUsed
if bm != ImageBuildModule && plan.NeedsToolchain(bm) {
// The ask names the toolchain the fingerprint was computed with; a node building with another refuses it.
language := buildrun.LanguageOf[bm]
tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language))
if none != "" {
return failed("the %s toolchain is not agreed: %s", language, none)
}
if want := m.Toolchains[language]; want.Version != tc.Version || want.Recipe != tc.Recipe {
return delivery.BuilderBuild{} // the agreed toolchain moved since try looked: nothing is written
}
u := delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
used[language], toolchain = u, &u
}
var tasks []Task
for _, t := range plan.Tasks {
if t.Module == bm {
tasks = append(tasks, t)
}
}
// One ask builds each fingerprint's part of a build module, on whichever node pulls it (ADR 0525 §3).
lanes = append(lanes, &lane{module: bm, tasks: tasks, toolchain: toolchain, digests: map[string]string{}, references: map[string]string{}})
}
var wg sync.WaitGroup
for _, l := range lanes {
wg.Add(1)
go func() { defer wg.Done(); b.runLane(ctx, l, repository, m, module, asked) }()
}
wg.Wait()
var errs []string
for _, l := range lanes {
if l.err != nil {
errs = append(errs, l.err.Error())
}
}
if len(errs) > 0 {
if ref := waitsOf(errs); ref != "" {
// The module's lock names a version of a package the registry does not hold yet: the key waits for it, queued, and
// is not failed (ADR 0500 clause 6).
return delivery.BuilderBuild{State: delivery.BuildQueued, Waits: ref, Step: "waits for the package " + ref + " to be published"}
}
return failed("%s", keyError(errs))
}
var artifacts []delivery.BuiltArtifact
keptModule := map[string]bool{}
for _, bm := range kept {
keptModule[bm] = true
}
for _, t := range plan.Tasks {
if keptModule[t.Module] {
found := false
for _, a := range prior.Artifacts {
if a.Name == t.Artifact {
artifacts, found = append(artifacts, a), true
}
}
if !found {
return failed("the done build holds no artifact %s to keep", t.Artifact)
}
continue
}
byNode := map[string]string{}
for _, l := range lanes {
if d, ok := l.digests[t.Artifact]; ok && l.module == t.Module {
byNode[l.node] = d
}
}
d, err := buildrun.Agree(t.Artifact, byNode)
if err != nil {
return failed("%s", err)
}
built := delivery.BuiltArtifact{Name: t.Artifact, Kind: t.Kind, Digest: d}
for _, l := range lanes {
if l.module != t.Module {
continue
}
r := l.references[t.Artifact]
if r == "" && stored(t.Kind) {
// A bundle, an archive or an upstream copy is done only when its push was answered with where it is (ADR 0498
// clause 1); a candidate's included (clause 3).
return failed("%s%s on %s: the build answered no reference, so its push to the artifact store was not answered", delivery.NotStored, t.Artifact, l.node)
}
if r != "" {
built.Reference = r
}
}
artifacts = append(artifacts, built)
}
var steps []string
for _, l := range lanes {
steps = append(steps, fmt.Sprintf("%s on %s", l.module, l.node))
}
for _, bm := range kept {
steps = append(steps, bm+" kept")
for lang, u := range prior.Toolchains {
if _, ok := used[lang]; !ok && buildrun.LanguageOf[bm] == lang {
used[lang] = u
}
}
}
step := "built " + strings.Join(steps, "; ")
if len(used) == 0 {
used = nil
}
done := delivery.BuilderBuild{State: delivery.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used}
if _, err := delivery.EncodeBuilderBuild(done); err != nil {
// A reference the build module answered that does not name the artifact and its digest is no reference.
return failed("%s%v", delivery.NotStored, err)
}
return done
}
// stored is whether an artifact of this kind is in the artifact store once built, a candidate's too, so its done key
// names its reference (ADR 0498 clauses 1, 3 and 4); an image or a package names one only when built for the trunk.
func stored(kind string) bool {
return kind == catalogue.ArtifactBundle || kind == catalogue.ArtifactArchive || kind == catalogue.ArtifactUpstream
}
// waitsFor opens the reason of a fetch step whose module's lock names a package version the registry does not hold, then
// `<name>@<version>` (the Node build module's, ADR 0500 clause 6).
const waitsFor = delivery.WaitsFor
// waitsOf is the package version a lane's error says the module waits for, or "" when none does or what it names is not
// `<name>@<version>`.
func waitsOf(errs []string) string {
for _, e := range errs {
at := strings.Index(e, waitsFor)
if at < 0 {
continue
}
ref, _, _ := strings.Cut(e[at+len(waitsFor):], " ")
ref = strings.TrimRight(ref, ":;,")
if _, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildQueued, Waits: ref}); err == nil {
return ref
}
}
return ""
}
// keyPrefixes are the errors a build module's step may open its reason with that the key's error must open with too, so
// the claim reads them (delivery.FailedUntilChanged), first the ones not built again.
var keyPrefixes = []string{delivery.NotPinned, delivery.VersionTaken, delivery.RefusedKind, delivery.NotStored}
// keyError is the error of a key from its lanes' errors: when one names a prefix of keyPrefixes inside the step's words
// (`<tool> on <node>: the step <key> failed: not stored: ...`), the key's error opens with that prefix, so a reader of the
// key need not search for it.
func keyError(errs []string) string {
for _, prefix := range keyPrefixes {
for i, e := range errs {
if strings.HasPrefix(e, prefix) {
return strings.Join(errs, "; ")
}
if at := strings.Index(e, prefix); at >= 0 {
rest := append([]string{prefix + e[:at] + e[at+len(prefix):]}, append(append([]string(nil), errs[:i]...), errs[i+1:]...)...)
return strings.Join(rest, "; ")
}
}
}
return strings.Join(errs, "; ")
}
// imageSeat says an image build is asked of the image build seat, which takes it when a node pulls it: a key failed for
// want of an image builder before R4g is taken again (ADR 0525 §1).
func imageSeat() bool { return true }
// askLanguage is the build seat language of each build module (delivery.AskSubject).
var askLanguage = map[string]string{buildrun.GoBuildModule: delivery.AskGo, buildrun.NodeBuildModule: delivery.AskNode, ImageBuildModule: delivery.AskImage}
// askNotes writes a claimed key's asks into it, building, before each is put (ADR 0525 §1; issue 1887 A4b), and holds
// the asks a key taken over at start already names, so they are waited for and never put again.
type askNotes struct {
mu sync.Mutex
b *Builder
cl *Claim
resumed map[string]string
}
func (b *Builder) notes(cl *Claim) *askNotes {
n := &askNotes{b: b, cl: cl}
if cl.Build.State == delivery.BuildBuilding && len(cl.Build.Asks) > 0 {
n.resumed = maps.Clone(cl.Build.Asks)
}
return n
}
// resume is the id of the ask a taken-over key names for language, or "".
func (n *askNotes) resume(language string) string {
n.mu.Lock()
defer n.mu.Unlock()
return n.resumed[language]
}
// note writes the ask's id into the key, building, by compare-and-set at the claim's revision.
func (n *askNotes) note(ctx context.Context, language, id, step string) error {
n.mu.Lock()
defer n.mu.Unlock()
v := n.cl.Build
v.State, v.By, v.Step, v.Error, v.Artifacts = delivery.BuildBuilding, n.b.Claimer.By, step, "", nil
v.Asks = maps.Clone(v.Asks)
if v.Asks == nil {
v.Asks = map[string]string{}
}
v.Asks[language] = id
return n.b.Claimer.Finish(ctx, n.cl, v)
}
// runLane puts the lane's part of the build as one ask on its build seat's stream (the delivery stream for a trunk key, the
// check stream for a candidate's) and reads what the node that pulled it built (ADR 0525 §1, §3, §6). An upstream artifact
// is copied by its pinned reference with no fetch (ADR 0498 clause 4); a lane of copies alone fetches nothing. A lane of a
// key taken over at start waits for the ask the key names and puts none.
func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m delivery.Module, module string, asked *askNotes) {
req := buildstep.FetchRequest{Repository: repository, Commit: m.Commit}
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.mu.Lock()
v, candidate := b.via[key]
b.mu.Unlock()
if candidate {
req.Ref, req.Base = v.Ref, v.Base
}
}
if m.Inputs2 != nil && len(m.Inputs2.Paths) > 0 {
// A module whose every artifact names its paths is built from those alone (ADR 0499 clause 4): the paths come from
// the indexer's kept inputs, never from a manifest (the lead's decision L8 of R4e).
req.Dir = m.Path
for p := range m.Inputs2.Paths {
req.Paths = append(req.Paths, p)
}
sort.Strings(req.Paths)
}
candidate := req.Base != ""
language := askLanguage[l.module]
kind := delivery.AskDelivery
if candidate {
kind = delivery.AskCheck
}
job := &delivery.BuildJob{Module: module, Path: m.Path, FP: m.FP2, Toolchain: l.toolchain,
// Only a trunk key's build pushes an image or publishes a package; a candidate's never does (ADR 0487 clause 12,
// ADR 0498 clauses 5 and 6). A bundle or an archive is pushed either way (clause 3).
Publish: !candidate}
for _, t := range l.tasks {
a := delivery.AskArtifact{Artifact: t.Artifact}
if t.Kind == catalogue.ArtifactUpstream {
a.From = t.From // the copy runs no code of a change: the module's name, the artifact and its pin
}
job.Artifacts = append(job.Artifacts, a)
}
if needsFetch(l.tasks) {
job.Fetch = req.Args()
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: language, Asker: delivery.BuilderModule, At: time.Now(), Build: job}
if id := asked.resume(language); id != "" {
ask.ID = id // put before this builder started again: the stream holds it, or a node already ran it
if b.Held != nil {
held, err := b.Held.Held(ctx, ask)
if err != nil {
// Said, and the key is not failed: the ask may well be on its stream and its build finish (issue 1910).
b.logf("whether the ask %s of %s the key names is on its stream cannot be read, so it is waited for as put: %v", ask.ID, l.module, err)
held = true
}
if !held {
// The builder that named it stopped before it put it (issue 1910): put it now, under the same id.
b.logf("the ask %s of %s that the key names was never put: it is put now", ask.ID, l.module)
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
}
}
} else {
if err := asked.note(ctx, language, ask.ID, "asked "+l.module); err != nil {
l.err = fmt.Errorf("the ask of %s could not be written to the key: %w", l.module, err)
return
}
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
}
step, err := b.Asks.Wait(ctx, ask)
if err != nil {
l.err = fmt.Errorf("%s: %w", l.module, err)
return
}
l.node = step.Node
if step.State != delivery.StepPassed {
l.err = fmt.Errorf("%s on %s: the ask %s %s: %s", l.module, step.Node, ask.ID, step.State, step.Reason)
return
}
var answer delivery.BuildAnswer
if err := json.Unmarshal(step.Result, &answer); err != nil {
l.err = fmt.Errorf("the ask of %s on %s answered what is not a build: %s", l.module, step.Node, step.Result)
return
}
built := map[string]delivery.AskBuilt{}
for _, a := range answer.Artifacts {
built[a.Artifact] = a
}
for _, t := range l.tasks {
a, ok := built[t.Artifact]
if !ok || a.Digest == "" {
l.err = fmt.Errorf("the build of %s of %s on %s answered no digest: %s", t.Artifact, module, step.Node, step.Result)
return
}
l.digests[t.Artifact] = a.Digest
if a.Reference != "" && (stored(t.Kind) || !candidate) {
l.references[t.Artifact] = a.Reference
}
if a.Reference != "" && b.Answered != nil {
b.Answered(l.module)
}
}
}
// needsFetch is whether any of the tasks builds from the module's source: every one but an upstream copy.
func needsFetch(tasks []Task) bool {
for _, t := range tasks {
if t.Kind != catalogue.ArtifactUpstream {
return true
}
}
return false
}
@@ -0,0 +1,699 @@
package build
import (
"context"
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// buildModules simulates the two build modules on the nodes: a long tool answers a step and key at once and writes the
// step's end into the steps (as mesh-build-go and mesh-build-node do), with a digest by node and artifact, or a failure.
type buildModules struct {
mu sync.Mutex
n int
calls []call
steps *fakeSteps
digests map[string]string // `<node>/<artifact>` -> digest; the default is `digest`
fails map[string]string // `<node>/<tool>` -> the reason the step fails with
// noRef names `<node>/<artifact>` whose bundle or archive build answers no reference (its push not answered).
noRef map[string]bool
// packages are the package artifacts by name, each with the `<name>@<version>` its trunk build publishes (ADR 0498 clause 5).
packages map[string]string
// hold, when set before the builds start, blocks every call until it is closed; entered is sent to as a call arrives.
// conflicts, when set, is a file a fetch with a base reports in conflict; noMerge makes it answer neither a merge nor a conflict.
conflicts string
noMerge bool
hold chan struct{}
entered chan struct{}
}
func newBuildModules() *buildModules {
return &buildModules{steps: &fakeSteps{steps: map[string]delivery.Step{}}, digests: map[string]string{}, fails: map[string]string{}, noRef: map[string]bool{}, packages: map[string]string{}}
}
func (b *buildModules) CallOn(node, key string, body any) (json.RawMessage, error) {
if b.hold != nil {
select {
case b.entered <- struct{}{}:
default:
}
<-b.hold
}
b.mu.Lock()
defer b.mu.Unlock()
b.calls = append(b.calls, call{node, key, body})
b.n++
id := fmt.Sprintf("s%d", b.n)
module, tool, _ := strings.Cut(key, ".")
args, _ := body.(map[string]any)
step := delivery.Step{ID: id, Tool: tool, Node: node, State: delivery.StepPassed}
if reason, ok := b.fails[node+"/"+tool]; ok {
step.State, step.Reason = delivery.StepFailed, reason
} else if strings.HasSuffix(tool, "_fetch") {
step.Result = json.RawMessage(`{"workspace":"w-` + node + `"}`)
if args["base"] != nil { // a fetch with a base answers the merge, or the files in conflict (mesh-build-go FetchResult)
if b.conflicts != "" {
step.Result = json.RawMessage(`{"workspace":"w-` + node + `","base":"` + fmt.Sprint(args["base"]) + `","conflicts":["` + b.conflicts + `"]}`)
} else if !b.noMerge {
step.Result = json.RawMessage(`{"workspace":"w-` + node + `","base":"` + fmt.Sprint(args["base"]) + `","merge":"` + strings.Repeat("9", 40) + `"}`)
}
}
} else {
d := digest
if v, ok := b.digests[node+"/"+fmt.Sprint(args["artifact"])]; ok {
d = v
}
if tool == "build_image_copy" { // mesh-build-image's copy answers the digest and where it copied to (ADR 0498 clause 4)
step.Result = json.RawMessage(`{"artifact":"` + fmt.Sprint(args["artifact"]) + `","digest":"` + d + `","reference":"registry.example/` + fmt.Sprint(args["module"]) + `/` + fmt.Sprint(args["artifact"]) + `@` + d + `","pushed":true}`)
} else if pkg, ok := b.packages[fmt.Sprint(args["artifact"])]; ok { // a package names its version only when published
step.Result = json.RawMessage(`{"digest":"` + d + `"}`)
if args["publish"] == true {
step.Result = json.RawMessage(`{"digest":"` + d + `","reference":"` + pkg + `"}`)
}
} else if tool == "build_image_build" && args["publish"] == true {
step.Result = json.RawMessage(`{"digest":"` + d + `","reference":"registry.example/m/` + fmt.Sprint(args["artifact"]) + `@` + d + `"}`)
} else if tool != "build_image_build" && !b.noRef[node+"/"+fmt.Sprint(args["artifact"])] {
// A bundle or an archive is pushed to the artifact store and its reference answered (ADR 0498 clause 1).
step.Result = json.RawMessage(`{"digest":"` + d + `","reference":"artifact-store://m/` + fmt.Sprint(args["artifact"]) + `/blobs/` + d + `"}`)
} else {
step.Result = json.RawMessage(`{"digest":"` + d + `"}`)
}
}
sk, _ := delivery.StepKey(node, id)
b.steps.mu.Lock()
b.steps.steps[module+"/"+sk] = step
b.steps.mu.Unlock()
return json.RawMessage(`{"step":"` + id + `","key":"` + sk + `","state":"running"}`), nil
}
func (b *buildModules) callsOf(tool string) (out []call) {
b.mu.Lock()
defer b.mu.Unlock()
for _, c := range b.calls {
if strings.HasSuffix(c.key, "."+tool) {
out = append(out, c)
}
}
return out
}
type fakeManifests struct {
byDigest map[string]catalogue.Manifest
}
func (f *fakeManifests) Manifest(_ context.Context, d string) (catalogue.Manifest, error) {
m, ok := f.byDigest[d]
if !ok {
return catalogue.Manifest{}, fmt.Errorf("manifests.%s is not there", d)
}
return m, nil
}
func mdigest(c byte) string { return hex64(c) }
type rig struct {
b *Builder
st *memStore
bm *buildModules
tools *buildrun.ToolchainSet
mf *fakeManifests
key string
// asks are the build seats' streams; puller is the node that pulls every ask ("ace" when empty).
asks *buildrun.FakeAsks
puller string
}
// toolPrefix is the start of each build module's tool names, as the fake build modules answer them.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_", ImageBuildModule: "build_image_"}
// answer runs an ask as the build module that pulls it does (buildstep.RunAsk), with the fake build modules' tools on the
// rig's pulling node, and answers the step that node writes.
func (r *rig) answer(a delivery.StepAsk) delivery.Step {
node := r.puller
if node == "" {
node = "ace"
}
module, _ := delivery.BuildModuleOf(a.Language)
runner := buildrun.Runner{Caller: r.bm, Steps: r.bm.steps, Bound: time.Second}
tool := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
raw, err := runner.Run(ctx, module, node, toolPrefix[module]+name, args)
if err != nil {
return nil, err
}
return raw, nil
}
}
steps := buildstep.AskSteps{Fetch: tool("fetch"), Build: tool("build"), Copy: tool("copy")}
out, err := buildstep.RunAsk(context.Background(), steps, a)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
return step
}
step.Result, _ = json.Marshal(out)
return step
}
func newRig(t *testing.T, artifacts ...catalogue.Artifact) *rig {
t.Helper()
r := &rig{st: newMem(), bm: newBuildModules(), tools: &buildrun.ToolchainSet{}, mf: &fakeManifests{byDigest: map[string]catalogue.Manifest{mdigest('a'): {Module: "m", Build: &catalogue.Build{Artifacts: artifacts}}}}}
for _, n := range []string{"ace", "novox"} {
r.tools.Put(buildrun.GoBuildModule, n+".go", tcValue(t, "go", "go1.27.1"), false)
r.tools.Put(buildrun.NodeBuildModule, n+".typescript", tcValue(t, "typescript", "tsc5"), false)
}
r.asks = &buildrun.FakeAsks{Answer: r.answer}
r.b = &Builder{
Claimer: Claimer{Store: r.st, By: "novox.builder"}, Asks: buildrun.Asker{Publisher: r.asks, Waiter: r.asks, Bound: 2 * time.Second},
Toolchains: r.tools, Manifests: r.mf, Logf: t.Logf,
}
var err error
if r.key, err = delivery.BuilderBuildKey("m", fp1); err != nil {
t.Fatal(err)
}
return r
}
// agreedToolchains are what the rig's nodes report, and so what a fingerprint computed now names.
var agreedToolchains = map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}
func moduleValue(t *testing.T, path string) []byte { return moduleValueWith(t, path, agreedToolchains) }
func moduleValueWith(t *testing.T, path string, toolchains map[string]delivery.ToolchainUsed) []byte {
t.Helper()
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'), FP2: fp1, Manifest: mdigest('a'), Path: path, Toolchains: toolchains})
if err != nil {
t.Fatal(err)
}
return data
}
func (r *rig) consider(t *testing.T, path string) delivery.BuilderBuild {
t.Helper()
r.b.Module(context.Background(), "m", moduleValue(t, path))
r.b.Wait()
if _, ok := r.st.vals[r.key]; !ok {
t.Fatal("no build key was written")
}
return valueOf(t, r.st, r.key)
}
var (
goBundle = catalogue.Artifact{Name: "tool", Kind: "bundle", Language: "go"}
tsBundle = catalogue.Artifact{Name: "web", Kind: "bundle", Language: "typescript"}
archiveA = catalogue.Artifact{Name: "files", Kind: "archive"}
imageArt = catalogue.Artifact{Name: "img", Kind: "image"}
manifestD = mdigest('a')
)
func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7, ADR 0478 clauses 1, 4
r := newRig(t, goBundle, archiveA, tsBundle)
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || v.By != "novox.builder" || v.Manifest != manifestD || len(v.Artifacts) != 3 {
t.Fatalf("build = %+v, want done by novox.builder with three artifacts", v)
}
for i, want := range []string{"files", "tool", "web"} {
if a := v.Artifacts[i]; a.Name != want || a.Digest != digest || a.Reference != "artifact-store://m/"+want+"/blobs/"+digest {
t.Errorf("artifact %d = %+v, want %s with the digest and its artifact-store reference (ADR 0498 clause 1)", i, a, want)
}
}
if v.Artifacts[0].Kind != "archive" || v.Artifacts[1].Kind != "bundle" {
t.Errorf("kinds = %+v", v.Artifacts)
}
if got := v.Toolchains["go"]; got.Version != "go1.27.1" || got.Recipe != "1" {
t.Errorf("toolchains = %+v, want go and typescript named", v.Toolchains)
}
if got := v.Toolchains["typescript"]; got.Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
// One fetch per build module, on one node (ADR 0478 clause 4); every build names the module's directory.
if n := len(r.bm.callsOf("build_go_fetch")); n != 1 {
t.Errorf("%d go fetches, want one", n)
}
if n := len(r.bm.callsOf("build_go_build")); n != 2 { // tool and files
t.Errorf("%d go builds, want 2", n)
}
if n := len(r.bm.callsOf("build_node_build")); n != 1 {
t.Errorf("%d node builds, want 1", n)
}
c := r.bm.callsOf("build_go_build")[0]
if args := c.body.(map[string]any); args["module"] != "modules/m" || !strings.HasPrefix(fmt.Sprint(args["workspace"]), "w-") {
t.Errorf("build args = %+v, want the module's path and the fetch's workspace", args)
}
f := r.bm.callsOf("build_go_fetch")[0].body.(map[string]any)
if f["repository"] != "novox/mesh-catalog" || f["commit"] != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" {
t.Errorf("fetch args = %+v", f)
}
}
// 1887/AC1, AC2: a trunk fingerprint with no key is one ask on the Go seat's delivery stream, named in the building key
// before it is put, and the builder calls no node; the step a node wrote for it makes the key done naming its reference.
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
r := newRig(t, goBundle)
r.asks.Answer = nil // no node has pulled it yet
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
var asks []delivery.StepAsk
for deadline := time.Now().Add(2 * time.Second); len(asks) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
asks = r.asks.Asks()
}
a := asks[0]
if len(asks) != 1 || a.Kind != delivery.AskDelivery || a.Language != delivery.AskGo || a.Build == nil || a.Build.FP != fp1 {
t.Fatalf("asks = %+v, want one Go delivery ask of the fingerprint", asks)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildBuilding || v.Asks[delivery.AskGo] != a.ID {
t.Fatalf("the key = %+v, want building naming the ask %s", v, a.ID)
}
if n := len(r.bm.callsOf("build_go_build")) + len(r.bm.callsOf("build_go_fetch")); n != 0 {
t.Fatalf("the builder called a node %d times", n)
}
r.asks.End(r.answer(a)) // a node pulled it and ran it
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || !strings.Contains(v.Step, "on ace") || v.Artifacts[0].Reference == "" || len(v.Asks) != 0 {
t.Fatalf("the key = %+v, want done on ace naming the reference", v)
}
}
// 1887/AC3: a builder that starts again while its ask waits puts no second ask: it waits for the step of the ask the
// building key names.
func TestARestartedBuilderWaitsForTheAskTheKeyNames(t *testing.T) {
r := newRig(t, goBundle)
const id = "mvl0frv18d1957"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
v := r.consider(t, "modules/m")
if n := len(r.asks.Asks()); n != 0 {
t.Fatalf("%d asks put again", n)
}
if v.State != delivery.BuildDone || !strings.Contains(v.Step, "on shanks") {
t.Fatalf("the key = %+v, want done from the ask's step on shanks", v)
}
}
func TestAFailingStepFailsTheBuildWithItsReasonAndNodeAndIsTriedAgainAsANewRevision(t *testing.T) { // 1607/AC7, ADR 0429 clause 18
r := newRig(t, goBundle)
r.bm.fails["ace/build_go_build"] = "go build of tool: undefined: x"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "ace") || !strings.Contains(v.Error, "undefined: x") {
t.Fatalf("build = %+v, want failed with the node and the reason", v)
}
rev := r.st.revs[r.key]
delete(r.bm.fails, "ace/build_go_build")
v = r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev {
t.Errorf("build = %+v at revision %d (was %d), want done as a new revision of the same key", v, r.st.revs[r.key], rev)
}
}
func TestAnImageGoesToOneNodeAndGetsOneDoneKeyWithItsReference(t *testing.T) { // 1705a, ADR 0487 clause 5
r := newRig(t, imageArt)
r.puller = "ace" // two nodes qualify; an image still builds on one
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "image" || v.Artifacts[0].Digest != digest {
t.Fatalf("build = %+v, want done with one image artifact and its digest", v)
}
if v.Artifacts[0].Reference == "" {
t.Errorf("a trunk image has no reference: %+v", v.Artifacts[0])
}
if n := len(r.bm.callsOf("build_image_fetch")); n != 1 {
t.Errorf("%d image fetches, want one", n)
}
writes := r.st.writes
r.consider(t, "modules/m")
if r.st.writes != writes || len(r.bm.callsOf("build_image_build")) != 1 {
t.Errorf("a done image key was built again")
}
}
func TestABundleAndAnImageOfOneModuleBuildOnOneNodeEach(t *testing.T) { // 1705a
r := newRig(t, goBundle, imageArt)
r.puller = "ace"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
if n := len(r.bm.callsOf("build_go_build")); n != 1 {
t.Errorf("%d go builds, want 1 (one node per fingerprint, ADR 0478 clause 4)", n)
}
if n := len(r.bm.callsOf("build_image_build")); n != 1 {
t.Errorf("%d image builds, want 1", n)
}
}
func TestACandidatesImageCallCarriesNoPublishAndItsKeyNoReference(t *testing.T) { // 1705b, lead's decision
r := newRig(t, imageArt)
r.puller = "ace"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Reference != "" {
t.Fatalf("build = %+v, want done with no reference", v)
}
calls := r.bm.callsOf("build_image_build")
if len(calls) != 1 {
t.Fatalf("%d image builds, want 1", len(calls))
}
if _, has := calls[0].body.(map[string]any)["publish"]; has {
t.Errorf("a candidate's image call carries publish: %+v", calls[0].body)
}
}
func TestATrunkImageCallCarriesPublish(t *testing.T) { // 1705b
r := newRig(t, imageArt)
r.puller = "ace"
r.consider(t, "modules/m")
calls := r.bm.callsOf("build_image_build")
if len(calls) != 1 || calls[0].body.(map[string]any)["publish"] != true {
t.Errorf("calls = %+v, want publish true", calls)
}
}
func TestAnUnpinnedUpstreamFailsNotPinnedAndIsNeverBuiltAgain(t *testing.T) { // B3, ADR 0498 clause 4
r := newRig(t, archiveA, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine"})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotPinned) || !strings.Contains(v.Error, "up") {
t.Fatalf("build = %+v, want failed 'not pinned:' naming the artifact", v)
}
if len(r.bm.calls) != 0 {
t.Errorf("%d calls, want none for an unpinned upstream", len(r.bm.calls))
}
writes := r.st.writes
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Errorf("an unpinned upstream's key was written again (%d writes, was %d)", r.st.writes, writes)
}
}
func TestAModuleKeyWithNoPathFailsNamingThatReason(t *testing.T) { // 1607/AC8, ADR 0478 clause 8
r := newRig(t, goBundle)
v := r.consider(t, "")
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "path") || len(r.bm.calls) != 0 {
t.Errorf("build = %+v with %d calls, want failed naming the missing path and nothing called", v, len(r.bm.calls))
}
}
func TestAnUnreadableManifestFailsTheBuild(t *testing.T) {
r := newRig(t, goBundle)
delete(r.mf.byDigest, mdigest('a'))
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "manifest") {
t.Errorf("build = %+v, want failed naming the manifest", v)
}
}
func TestAModuleWithNoFingerprintIsNotBuilt(t *testing.T) { // 1607/AC2
r := newRig(t)
data, _ := delivery.EncodeModule(delivery.Module{Repository: "novox/r", Commit: "c", FP: "src1:" + hex64('f'), None2: "toolchain not yet reported"})
r.b.Module(context.Background(), "n", data)
r.b.Wait()
if len(r.st.vals) != 0 {
t.Errorf("keys %v were written: a module with no new-form fingerprint has nothing to build", r.st.vals)
}
}
func TestAModuleWithNoArtifactGetsADoneKeyWithNoArtifactAndNoBuildCall(t *testing.T) { // issue 1613, ADR 0478 clause 1
for name, manifest := range map[string]catalogue.Manifest{"no build": {Module: "m"}, "no artifact": {Module: "m", Build: &catalogue.Build{}}} {
r := newRig(t)
r.mf.byDigest[mdigest('a')] = manifest
v := r.consider(t, "") // a module with nothing to build needs no directory
if v.State != delivery.BuildDone || len(v.Artifacts) != 0 || v.Manifest != manifestD || v.By != "novox.builder" {
t.Errorf("%s: build = %+v, want done with no artifact", name, v)
}
if len(r.bm.calls) != 0 {
t.Errorf("%s: %d build module calls, want none", name, len(r.bm.calls))
}
writes := r.st.writes
r.b.tried = map[string]bool{}
r.consider(t, "")
if r.st.writes != writes {
t.Errorf("%s: the done key was written again", name)
}
}
}
func TestAKeyTheBuilderLeftBuildingIsBuiltAtStartAndDoneKeysAreNotBuiltTwice(t *testing.T) { // 1607/AC3, AC7
r := newRig(t, goBundle)
seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD})
if v := r.consider(t, "modules/m"); v.State != delivery.BuildDone {
t.Fatalf("build = %+v, want the builder's own building key built", v)
}
calls := len(r.bm.calls)
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Reconsider(context.Background())
r.b.Wait()
if len(r.bm.calls) != calls {
t.Errorf("a done key was built again (%d calls, was %d)", len(r.bm.calls), calls)
}
}
// Two instances that start together on one key: the second finds the key building and takes it over (ADR 0478 clause 6:
// a key found building at start is taken over, whoever holds it), so both may build; the key is written once, by the one
// whose compare-and-set comes first, and the other's write is refused, never overwriting it.
func TestTwoInstancesStartingTogetherEndWithOneDoneKey(t *testing.T) { // 1607/AC7, review round 4
r := newRig(t, goBundle)
other := &Builder{Claimer: Claimer{Store: r.st, By: "ace.builder"}, Asks: r.b.Asks, Toolchains: r.tools, Manifests: r.mf, Logf: t.Logf}
value := moduleValue(t, "modules/m")
var wg sync.WaitGroup
for _, b := range []*Builder{r.b, other} {
wg.Add(1)
go func() { defer wg.Done(); b.Module(context.Background(), "m", value); b.Wait() }()
}
wg.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Errorf("build = %+v, want one done key", v)
}
if n := len(r.bm.callsOf("build_go_build")); n < 1 {
t.Errorf("%d builds, want one for each instance that built", n)
}
}
func TestAModulesValueThatArrivesWhileItsKeyIsBeingBuiltIsLookedAtAgainWhenItEnds(t *testing.T) { // 1607/AC6 (no event is dropped, no timer)
r := newRig(t, goBundle)
r.bm.fails["ace/build_go_build"] = "go build of tool: boom"
r.bm.hold, r.bm.entered = make(chan struct{}), make(chan struct{}, 8)
value := moduleValue(t, "modules/m")
r.b.Module(context.Background(), "m", value)
<-r.bm.entered // the first build is under way
r.b.Module(context.Background(), "m", value)
r.b.Reconsider(context.Background())
delete(r.bm.fails, "ace/build_go_build") // what the event is about: the build can succeed now
close(r.bm.hold)
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want the failed key built again as a new revision after the first build ended", v)
}
}
func TestAKeyIsBuiltOnlyWithTheToolchainItsFingerprintNamesAndNothingIsWrittenOtherwise(t *testing.T) { // 1615/AC4
r := newRig(t, goBundle)
old := map[string]delivery.ToolchainUsed{"go": {Version: "go1.26.0", Recipe: "1"}}
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", old))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Errorf("keys %v and %d calls for a fingerprint of go1.26.0 while the nodes agree on go1.27.1, want neither", r.st.vals, len(r.bm.calls))
}
// A recipe that differs is a different toolchain too, and a value that names none (written before) is not built.
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "0"}}))
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", nil))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Errorf("keys %v and %d calls for a recipe that differs or a value that names no toolchain", r.st.vals, len(r.bm.calls))
}
// The toolchain moves under a key that is not built yet: the old fingerprint stays unbuilt, the new one builds.
r.tools.Put(buildrun.GoBuildModule, "ace.go", tcValue(t, "go", "go1.27.2"), false)
r.tools.Put(buildrun.GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.2"), false)
r.b.Reconsider(context.Background())
r.b.Wait()
if len(r.st.vals) != 0 {
t.Errorf("keys %v: the old fingerprint was built with the new toolchain", r.st.vals)
}
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.2", Recipe: "1"}}))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.Toolchains["go"].Version != "go1.27.2" {
t.Errorf("build = %+v, want done with go1.27.2", v)
}
}
func TestNodesTheyDisagreeOnLeaveTheKeyUnwritten(t *testing.T) { // 1615/AC4
r := newRig(t, goBundle)
r.tools.Put(buildrun.GoBuildModule, "ace.go", tcValue(t, "go", "go1.26.0"), false)
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if len(r.st.vals) != 0 || len(r.bm.calls) != 0 {
t.Errorf("keys %v and %d calls while the nodes disagree", r.st.vals, len(r.bm.calls))
}
}
func TestARootModulesPathIsDotAndIsGivenToTheBuildTools(t *testing.T) { // 1615/AC4
r := newRig(t, goBundle)
if v := r.consider(t, delivery.RootPath); v.State != delivery.BuildDone {
t.Fatalf("build = %+v, want done for a root module", v)
}
if args := r.bm.callsOf("build_go_build")[0].body.(map[string]any); args["module"] != "." {
t.Errorf("build args = %+v, want module \".\"", args)
}
}
func TestACandidatesImageIsPublishedOnceWhenTheTrunkNamesItsFingerprint(t *testing.T) { // review r1, mesh-delivery 1; ADR 0487 clause 5
r := newRig(t, imageArt)
r.puller = "ace"
trunkModule(t, r) // the trunk carries another fingerprint; fp1 is the candidate's
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || v.Artifacts[0].Reference != "" {
t.Fatalf("candidate build = %+v, want done with no reference", v)
}
if n := len(r.bm.callsOf("build_image_build")); n != 1 {
t.Fatalf("%d image builds after the candidate, want 1", n)
}
// The pull request is merged: the trunk's record names the candidate's fingerprint.
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
v = valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Reference == "" {
t.Fatalf("trunk build = %+v, want done with a reference", v)
}
calls := r.bm.callsOf("build_image_build")
published := 0
for _, c := range calls {
if c.body.(map[string]any)["publish"] == true {
published++
}
}
if len(calls) != 2 || published != 1 {
t.Errorf("%d image builds, %d with publish, want 2 and exactly 1", len(calls), published)
}
// Written once more only: looking again builds nothing.
rev := r.st.revs[r.key]
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if r.st.revs[r.key] != rev || len(r.bm.callsOf("build_image_build")) != 2 {
t.Error("a published key was built again")
}
}
func TestAFailedPublishOfACandidatesImageIsWrittenToTheKey(t *testing.T) { // review r2, mesh-delivery 1; ADR 0487 clause 5
r := newRig(t, imageArt)
r.puller = "ace"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Fatalf("candidate build = %+v", v)
}
r.bm.fails["ace/build_image_build"] = "push to the registry: connection refused"
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the merge: the trunk names the fingerprint
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "connection refused") || !strings.Contains(v.Error, "publish") {
t.Fatalf("build = %+v, want failed naming the publish and its reason", v)
}
// The failed key is retaken as any failed key is, and ends published once the registry answers.
delete(r.bm.fails, "ace/build_image_build")
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.Artifacts[0].Reference == "" {
t.Errorf("build = %+v, want done with a reference", v)
}
}
// Review round 2 item 1: a build ask that waits in its stream longer than the step bound is not failed and not asked
// again; the key is done when a node takes it.
func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r := newRig(t, goBundle)
r.asks.Answer = nil
r.b.Asks = buildrun.Asker{Publisher: r.asks, Waiter: r.asks, Bound: 20 * time.Millisecond}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
for deadline := time.Now().Add(2 * time.Second); len(r.asks.Asks()) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
}
time.Sleep(100 * time.Millisecond)
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildBuilding {
t.Fatalf("the key = %+v while its ask waits, want building", v)
}
r.asks.End(r.answer(r.asks.Asks()[0]))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || len(r.asks.Asks()) != 1 {
t.Fatalf("the key = %+v after %d asks, want done from the one ask", v, len(r.asks.Asks()))
}
}
// heldAsks says an ask is held when its id is among held.
type heldAsks map[string]bool
func (h heldAsks) Held(_ context.Context, a delivery.StepAsk) (bool, error) { return h[a.ID], nil }
// 1910: a builder that stopped between naming the ask in its key and putting it left a key naming an ask neither the
// stream holds nor any node ran: started again, the builder puts that ask, under the same id, once, and the key is
// built from its step; an ask the stream holds is not put again.
func TestAnAskTheKeyNamesAndNobodyPutIsPutAtStart(t *testing.T) {
for _, held := range []bool{false, true} {
r := newRig(t, goBundle)
const id = "mvl0frv18d1958"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.b.Held = heldAsks{id: held}
if held {
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
}
v := r.consider(t, "modules/m")
asks := r.asks.Asks()
switch {
case held && len(asks) != 0:
t.Errorf("an ask the stream holds was put again: %+v", asks)
case !held && (len(asks) != 1 || asks[0].ID != id):
t.Errorf("the ask nobody put: asks %+v, want one under %s", asks, id)
}
if v.State != delivery.BuildDone {
t.Errorf("held %v: the key = %+v, want done", held, v)
}
}
}
// unreadableHeld cannot tell whether an ask is held.
type unreadableHeld struct{}
func (unreadableHeld) Held(context.Context, delivery.StepAsk) (bool, error) {
return false, errors.New("permissions violation for publish to $JS.API.STREAM.INFO.SEAT_NODE_BUILD_GO")
}
// 1910: a take-over that cannot read whether the ask its key names is on its stream says so and waits for the ask's step:
// the key is not failed, and the build finishes when a node ran the ask.
func TestATakeOverThatCannotReadTheStreamWaitsForTheAsk(t *testing.T) {
r := newRig(t, goBundle)
const id = "mvl0frv18d1959"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.b.Held = unreadableHeld{}
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(r.asks.Asks()) != 0 {
t.Fatalf("the key = %+v after %d asks, want done from the ask's step, none put", v, len(r.asks.Asks()))
}
}
@@ -0,0 +1,4 @@
Captured step values of mesh-build-image (fetch, a candidate's build, a trunk's build), copied from
novox/mesh-catalog modules/mesh-build-image/cmd/mesh-build-image/testdata/captured/. Both copies must change together: the
module's capture test writes them, and the builder's TestCapturedImageStepsDecodeWithTheBuildersOwnTypes (internal/build,
fixtures_test.go) decodes these with the types the builder reads results with and fails on a missing field.
@@ -0,0 +1,14 @@
{
"ended": "2026-10-22T08:00:01Z",
"id": "STEPID",
"node": "ace",
"result": {
"artifact": "server",
"digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
"pushed": false
},
"started": "2026-10-22T08:00:00Z",
"state": "passed",
"tool": "build_image_build",
"v": 1
}
@@ -0,0 +1,15 @@
{
"ended": "2026-10-22T08:00:01Z",
"id": "STEPID",
"node": "ace",
"result": {
"artifact": "server",
"digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
"pushed": true,
"reference": "REGISTRY/fixture-image/server@sha256:1111111111111111111111111111111111111111111111111111111111111111"
},
"started": "2026-10-22T08:00:00Z",
"state": "passed",
"tool": "build_image_build",
"v": 1
}
@@ -0,0 +1,13 @@
{
"ended": "2026-10-22T08:00:01Z",
"id": "STEPID",
"node": "ace",
"result": {
"commit": "COMMIT",
"workspace": "WORKSPACE"
},
"started": "2026-10-22T08:00:00Z",
"state": "passed",
"tool": "build_image_fetch",
"v": 1
}
@@ -0,0 +1,188 @@
package manifest
import (
"bytes"
"encoding/json"
"io/fs"
"os"
"path/filepath"
"reflect"
"slices"
"strings"
"testing"
)
// The controller's manifest reader is mesh-controller internal/catalogue (Manifest, manifest.go: Consumes :521,
// Invokes :590, Streams :613, Reads :659; StreamDeclaration, streams.go), an internal package of another repository.
// These tests decode the manifest strictly into the fields those types name (DisallowUnknownFields) and hold the
// rules of issue 1607 (ADR 0453 §4, ADR 0478). The merge gate's module check proves the controller takes the manifest.
type state struct {
Name string `json:"name"`
History int `json:"history,omitempty"`
}
type artifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
System string `json:"system"`
From string `json:"from"`
Binary string `json:"binary"`
}
type manifest struct {
Module string `json:"module"`
Version string `json:"version"`
Slug string `json:"slug"`
Consumes []string `json:"consumes"`
State []state `json:"state"`
Reads []string `json:"reads"`
Uses []string `json:"uses"`
Invokes []string `json:"invokes"`
Config map[string]struct {
Type string `json:"type"`
Description string `json:"description"`
Pattern string `json:"pattern"`
Default string `json:"default"`
} `json:"config"`
OwnSecrets map[string]string `json:"own-secrets"`
SecretsOwner string `json:"secrets-owner"`
Resources []map[string]any `json:"resources"`
Build struct {
Artifacts []artifact `json:"artifacts"`
} `json:"build"`
}
func load(t *testing.T) (manifest, map[string]json.RawMessage) {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
var m manifest
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&m); err != nil {
t.Fatalf("the controller's reader refuses a key this manifest has: %v", err)
}
var top map[string]json.RawMessage
if err := json.Unmarshal(raw, &top); err != nil {
t.Fatal(err)
}
return m, top
}
func sorted(s []string) []string {
s = slices.Clone(s)
slices.Sort(s)
return s
}
func TestManifestDeclarations(t *testing.T) { // 1607/AC1
m, top := load(t)
if m.Module != "mesh-delivery-builder" {
t.Errorf("module = %q", m.Module)
}
if want := []state{{Name: "builds", History: 5}, {Name: "instances", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
t.Errorf("state = %+v, want %+v", m.State, want)
}
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
// holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5).
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
t.Errorf("uses = %v, want the three build seats", m.Uses)
}
// 1937/AC3: the builder asks the controller whether the artifact store still holds a done key's blobs (held), and no
// other tool; it holds no capacity.
if want := []string{"seat:mesh-controller.held"}; !reflect.DeepEqual(m.Invokes, want) || len(m.Config) != 0 {
t.Errorf("invokes = %v, config = %v, want %v and no config", m.Invokes, m.Config, want)
}
// The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it
// starts; and by the controller's artifacts-gone, a take-in refused for a gone blob (1937/AC4).
if want := []string{"mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want)
}
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
t.Errorf("own-secrets = %v, want exactly broker", m.OwnSecrets)
}
if m.SecretsOwner != "mesh-delivery-builder" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
for _, k := range []string{"streams", "logs", "emits", "claims", "provides", "requires", "defines-seats", "tools", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the builder has none", k)
}
}
}
func TestManifestProcess(t *testing.T) { // 1607/AC1
m, _ := load(t)
var procs []map[string]any
for _, r := range m.Resources {
if r["type"] == "process" {
procs = append(procs, r)
}
if r["type"] == "tool" || r["type"] == "seat" {
t.Errorf("resource %v: the builder serves no tool and holds no seat", r)
}
}
if len(procs) != 1 {
t.Fatalf("%d process resources, want one", len(procs))
}
p := procs[0]
if p["artifact"] != "program" || !reflect.DeepEqual(p["run"], []any{"./mesh-delivery-builder"}) {
t.Errorf("process = %v", p)
}
env, _ := p["env"].(map[string]any)
if len(env) != 1 || env["MESH_BROKER_FILE"] != "${dir:state}/broker" {
t.Errorf("env = %v, want the broker file alone", p["env"])
}
for _, r := range m.Resources {
if r["id"] == "settings" {
t.Errorf("a settings resource %v: the builder holds no setting since the capacity is the build modules' (ADR 0525 §5)", r)
}
}
if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential")
}
if len(m.Build.Artifacts) != 1 {
t.Fatalf("artifacts = %+v, want the program alone", m.Build.Artifacts)
}
a := m.Build.Artifacts[0]
if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-builder" || !strings.HasSuffix(a.From, a.Binary) || a.Kind != "bundle" || a.Language != "go" {
t.Errorf("artifact = %+v", a)
}
}
func TestServesNoRequestSubjectAndHoldsNoTicker(t *testing.T) { // 1607/AC6 (ADR 0437 §8: no timer, a wait is a context's bound)
forbidden := []string{"Subscribe(", "QueueSubscribe", "ChanSubscribe", "time.NewTicker", "time.Tick", "time.NewTimer", "time.AfterFunc", "time.After(", "time.Sleep"}
var files int
for _, root := range []string{"cmd", "internal"} {
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return err
}
files++
src, err := os.ReadFile(path)
if err != nil {
return err
}
for _, f := range forbidden {
if strings.Contains(string(src), f) {
t.Errorf("%s holds %s: the builder serves no request subject and holds no timer", path, f)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
}
if files < 1 {
t.Fatalf("read %d source files, want the program's", files)
}
}
+83
View File
@@ -0,0 +1,83 @@
{
"module": "mesh-delivery-builder",
"version": "1",
"slug": "builder",
"consumes": [
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported",
"mesh-controller.artifacts-gone"
],
"invokes": [
"seat:mesh-controller.held"
],
"state": [
{ "name": "builds", "history": 5 },
{ "name": "instances", "history": 1 }
],
"reads": [
"mesh-build-go.steps",
"mesh-build-go.toolchains",
"mesh-build-image.steps",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules"
],
"uses": [
"node-build-go",
"node-build-image",
"node-build-node"
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-delivery-builder",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-delivery-builder",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-delivery-builder"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-delivery-builder",
"place": ".",
"unknown": "program"
},
{
"id": "builder",
"type": "process",
"name": "mesh-delivery-builder",
"artifact": "program",
"run": [
"./mesh-delivery-builder"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker"
},
"user": "mesh-delivery-builder",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "program",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-builder",
"binary": "mesh-delivery-builder"
}
]
}
}
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1,191 @@
// Command mesh-delivery-checker-tools serves the checker's read tools (novox/hq ADR 0484 clause 13, issue 1686): the runtime
// launches it for a call, it reads the checker's verdicts and kept states and the indexer's candidates through the checker's
// state and reads, and writes nothing. The checker is the other program: it serves no tool (ADR 0437 §4).
package main
import (
"encoding/json"
"fmt"
"os"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
func read(s stdio.KeptState) check.StateRead {
return check.StateRead{Keys: s.Keys, Get: func(key string) (json.RawMessage, error) {
e, err := s.Get(key)
if err != nil || e == nil {
return nil, err
}
return e.Value, nil
}}
}
// stateStore is a state the tools write: the tries.
type stateStore struct {
check.StateRead
create func(key string, value []byte) error
}
func (s stateStore) Keys() ([]string, error) { return s.StateRead.Keys() }
func (s stateStore) Get(key string) (json.RawMessage, error) { return s.StateRead.Get(key) }
func (s stateStore) Create(key string, value []byte) error { return s.create(key, value) }
// Trying is what the try's tools work on: the tries, and the gate's trunk heads.
type Trying struct {
Tries check.TryStore
Trunks func() (map[string]string, error)
Now func() time.Time
}
// TryTools are the try's two tools (ADR 0501 §5): ask one, read one. mesh-controller's try calls them.
func TryTools(tr Trying) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return s }
return []stdio.Tool{
{Name: check.TryTool,
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused.",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"head": map[string]any{"type": "string", "description": "the full commit to try, 40 or 64 hex digits"},
"trunk": map[string]any{"type": "string", "description": "the trunk commit to merge it onto, or the trunk's branch for its head now; the trunk's head when absent"},
"tests": map[string]any{"type": "string", "description": "a go test -run pattern: only those tests run, and nothing is kept"},
"level": map[string]any{"type": "string", "enum": []string{"unit", "integration"}, "description": "only this level; both when absent"}}},
Run: func(args map[string]any) (any, error) {
if tr.Tries == nil || tr.Trunks == nil {
return nil, fmt.Errorf("the tries state cannot be written here")
}
trunks, err := tr.Trunks()
if err != nil {
return nil, fmt.Errorf("the trunk heads cannot be read: %w", err)
}
now := time.Now
if tr.Now != nil {
now = tr.Now
}
id, err := check.AskTry(tr.Tries, trunks, check.TryAsk{Repository: str(args, "repository"), Head: str(args, "head"),
Trunk: str(args, "trunk"), Tests: str(args, "tests"), Level: str(args, "level")}, now())
if err != nil {
return nil, err
}
return map[string]any{"id": id}, nil
}},
{Name: check.TryReadTool,
Description: "Where a try stands, as stored: queued, running, or its end (pass, fail, or error when it could not run), with what was said of it, " +
"the head and the trunk commit it was merged onto, the time it took and what failed first. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"id"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"}}},
Run: func(args map[string]any) (any, error) {
if tr.Tries == nil {
return nil, fmt.Errorf("the tries state cannot be read here")
}
return check.ReadTry(tr.Tries, str(args, "id"))
}},
}
}
// Tools are the read tools over the checker's states.
func Tools(r check.Reader) []stdio.Tool {
return []stdio.Tool{
{Name: check.VerdictsReadTool,
Description: "The verdicts the checker holds, each as stored under verdicts.<group>.<heads>: its state, what failed first, the tests run, kept and skipped per level, the time of each part, the trunk heads it judged against and the composition, with the full " +
"report of the status each pull request carries (the line the forge shows is cut at 140 characters, this is not). With a group, that group's only. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"group": map[string]any{"type": "string", "description": "the delivery group's branch name"}}},
Run: func(args map[string]any) (any, error) {
group, _ := args["group"].(string)
vs, err := r.Verdicts(group)
return map[string]any{"verdicts": vs}, err
}},
{Name: check.PullReadTool,
Description: "The verdict that judges one pull request and the full report of its status mesh/check. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"repository", "number"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"number": map[string]any{"type": "integer", "description": "the pull request's number"}}},
Run: func(args map[string]any) (any, error) {
repository, _ := args["repository"].(string)
n, _ := args["number"].(float64)
return r.Pull(repository, int(n))
}},
{Name: check.KeptReadTool,
Description: "The kept verdicts of passing packages, each as stored under kept.<level>.<digest>: the package, its tests, the time they took, the toolchain " +
"and when it was kept. With a level, that level's; with a level and a digest, that one. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"level": map[string]any{"type": "string", "enum": []string{"unit", "integration"}, "description": "the level"},
"digest": map[string]any{"type": "string", "description": "the package's digest, with a level"}}},
Run: func(args map[string]any) (any, error) {
level, _ := args["level"].(string)
digest, _ := args["digest"].(string)
kept, err := r.Kept(level, digest)
return map[string]any{"kept": kept}, err
}},
{Name: check.PreviewsReadTool,
Description: "The merge previews the checker holds, each as stored under previews.<repository>.<head>.<trunk>: the merge commit, per module directory " +
"its tree, manifest digest and lock hashes, or the files in conflict. With a repository, that repository's; with a head (7 or more characters) " +
"only that head's. Keys that are no preview key are listed as unreadable. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"head": map[string]any{"type": "string", "description": "a pull request head commit, or a prefix of at least 7 characters"}}},
Run: func(args map[string]any) (any, error) {
repository, _ := args["repository"].(string)
head, _ := args["head"].(string)
if head != "" && len(head) < 7 {
return nil, fmt.Errorf("the head %q is too short: give at least 7 characters", head)
}
ps, bad, err := r.Previews(repository, head)
return map[string]any{"previews": ps, "unreadable": bad}, err
}},
{Name: check.StatusReadTool,
Description: "What runs: the steps in flight on the build modules by node, the candidates without a verdict for their heads, " +
"and the candidates whose check is owed by kind (waiting_by_kind: new, nodes, and trunk, a re-judge pulled after the other two). A key a " +
"build module left running (started past the step bound, or its build module not running on its node) is listed apart under " +
"left_running with why, and not counted. Nothing holds the checker and it has no capacity of its own: a check is an ask on the " +
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return r.Status() }},
}
}
// stepBound is the checker's step bound as the checker reads it (MESH_CHECKER_STEP_BOUND), its default when unset or not a
// duration.
func stepBound() time.Duration {
if d, err := time.ParseDuration(os.Getenv("MESH_CHECKER_STEP_BOUND")); err == nil && d > 0 {
return d
}
return check.DefaultStepBound
}
func main() {
r := check.Reader{VerdictState: read(stdio.State("verdicts")), KeptState: read(stdio.State("kept")), CandidateState: read(stdio.State("mesh-delivery-indexer.candidates")),
PreviewState: read(stdio.State("previews")),
StepStates: map[string]check.StateRead{"mesh-build-go": read(stdio.State("mesh-build-go.steps")), "mesh-build-node": read(stdio.State("mesh-build-node.steps"))},
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound()}
tries := stdio.State(check.TriesState)
source := stdio.State("mesh-delivery-gate.source")
trunks := func() (map[string]string, error) {
e, err := source.Get(delivery.SourceMainKey)
if err != nil || e == nil {
return nil, err
}
s, err := delivery.DecodeSource(e.Value)
return s.Heads, err
}
r.Trunks = trunks
trying := Trying{
Tries: stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}},
Trunks: trunks,
}
if err := stdio.Serve("", append(Tools(r), TryTools(trying)...)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1)
}
}
@@ -0,0 +1,81 @@
package main
import (
"encoding/json"
"errors"
"slices"
"strings"
"testing"
"time"
"mesh-delivery-checker/internal/check"
)
func TestTheToolsServedAreTheFiveReadTools(t *testing.T) {
var names []string
for _, tl := range Tools(check.Reader{}) {
names = append(names, tl.Name)
if tl.Description == "" || tl.Run == nil {
t.Errorf("%s has no description or no run", tl.Name)
}
}
want := []string{check.VerdictsReadTool, check.PullReadTool, check.KeptReadTool, check.PreviewsReadTool, check.StatusReadTool}
if !slices.Equal(names, want) {
t.Errorf("tools = %v, want %v", names, want)
}
// With nothing to read from a tool says so and changes nothing.
for _, tl := range Tools(check.Reader{}) {
if tl.Name == check.StatusReadTool {
continue // it answers from what it can read, and says what it could not
}
if _, err := tl.Run(map[string]any{"repository": "novox/x", "number": float64(1)}); err == nil {
t.Errorf("%s answered with no state to read", tl.Name)
}
}
}
// memStore is a tries state in memory.
type memStore struct{ vals map[string]json.RawMessage }
func (m *memStore) Keys() ([]string, error) {
var out []string
for k := range m.vals {
out = append(out, k)
}
return out, nil
}
func (m *memStore) Get(key string) (json.RawMessage, error) { return m.vals[key], nil }
func (m *memStore) Create(key string, value []byte) error {
if _, ok := m.vals[key]; ok {
return errors.New("exists")
}
m.vals[key] = value
return nil
}
func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed interface with mesh-controller's try
store := &memStore{vals: map[string]json.RawMessage{}}
tools := TryTools(Trying{Tries: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(tools) != 2 || tools[0].Name != "checker_try" || tools[1].Name != "checker_try_read" {
t.Fatalf("tools = %+v", tools)
}
out, err := tools[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
if err != nil {
t.Fatal(err)
}
id := out.(map[string]any)["id"].(string)
if id != "try-1000000005" {
t.Errorf("id = %q", id)
}
raw, err := tools[1].Run(map[string]any{"id": id})
if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) {
t.Errorf("read = %s, %v", raw, err)
}
if _, err := tools[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil ||
err.Error() != "no level of novox/hq is checked by the checker" {
t.Errorf("hq: %v", err)
}
}
@@ -0,0 +1,132 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// bucketOpener is the part of the bus the watches use.
type bucketOpener interface {
KeyValue(ctx context.Context, bucket string) (jetstream.KeyValue, error)
}
// kvStore is the builds bucket as the builder's store: create and compare-and-set.
type kvStore struct{ kv jetstream.KeyValue }
func conflict(err error) error {
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return check.ErrConflict
}
return err
}
func (s kvStore) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
e, err := s.kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, false, nil
}
if err != nil {
return nil, 0, false, err
}
return e.Value(), e.Revision(), true, nil
}
func (s kvStore) Create(ctx context.Context, key string, value []byte) (uint64, error) {
rev, err := s.kv.Create(ctx, key, value)
return rev, conflict(err)
}
func (s kvStore) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
rev, err := s.kv.Update(ctx, key, value, revision)
return rev, conflict(err)
}
func (s kvStore) Keys(ctx context.Context) ([]string, error) {
keys, err := s.kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
// Delete removes a key (an ended try past its keeping).
func (s kvStore) Delete(ctx context.Context, key string) error { return s.kv.Delete(ctx, key) }
func opOf(e jetstream.KeyValueEntry) bool { return e.Operation() == jetstream.KeyValuePut }
// settle reads the entries a watcher holds now, up to its mark of the last values, and returns the watcher for the
// changes that follow.
func settle(ctx context.Context, kv jetstream.KeyValue, each func(jetstream.KeyValueEntry)) (jetstream.KeyWatcher, error) {
w, err := kv.WatchAll(ctx)
if err != nil {
return nil, err
}
for {
select {
case <-ctx.Done():
w.Stop()
return nil, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return nil, errors.New("the watch ended")
}
if e == nil {
return w, nil
}
each(e)
}
}
}
// follow hands each change of a watcher to each until ctx ends; a watch that ends is an error, and the process manager
// starts the program again.
func follow(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.KeyValueEntry)) error {
defer w.Stop()
for {
select {
case <-ctx.Done():
return nil
case e, ok := <-w.Updates():
if !ok {
return errors.New("a watch ended")
}
if e != nil {
each(e)
}
}
}
}
// manifestReader reads manifests.<digest> of the indexer and parses the manifest it holds.
type manifestReader struct{ kv jetstream.KeyValue }
func (m manifestReader) Manifest(ctx context.Context, digest string) (catalogue.Manifest, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return catalogue.Manifest{}, err
}
e, err := m.kv.Get(ctx, key)
if err != nil {
return catalogue.Manifest{}, fmt.Errorf("manifests.%s: %w", key, err)
}
kept, err := delivery.DecodeManifest(e.Value())
if err != nil {
return catalogue.Manifest{}, err
}
return parseManifest(kept.JSON)
}
// parseManifest reads a manifest the indexer kept as the indexer and the assigner read it (issues 1751, 1906): a config value that
// falls back to a predefined key is judged by the merge gate in the catalogue, against the predefined.json at its root that this
// process never holds, so it is not unreadable here (novox/hq ADR 0348 §7).
func parseManifest(raw string) (catalogue.Manifest, error) {
return catalogue.ParseManifestForInputs([]byte(raw))
}
@@ -0,0 +1,63 @@
//go:build integration
package main
import (
"strings"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// assignerWrote raises the assigner's compositions bucket, the assigner online in the watchman's presence, and its verdict
// under the check's key, as the assigner leaves them on the bus.
func (b *bed) assignerWrote(t *testing.T, key string, v delivery.CompositionVerdict) {
t.Helper()
kv, err := b.js.CreateKeyValue(b.ctx, jetstream.KeyValueConfig{Bucket: delivery.AssignerCompositionsBucket, History: delivery.AssignerCompositionsHistory})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PresencePartKey(delivery.AssignerModule, "novox")
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
if _, err := b.buckets[delivery.PresenceBucket].Put(b.ctx, pkey, on); err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
}
// 1297/AC3: started whole with the assigner's verdict on the bus, the checker's line says the composition's verdict, not
// "not yet run": a pass names its machine, a fail fails the check with the assigner's words.
func TestTheCheckersLineSaysTheAssignersComposition(t *testing.T) {
for _, c := range []struct {
state, owed, words string
}{
{delivery.CompositionPass, "success", "composition passed"},
{delivery.CompositionFail, "failure", "g14 runs web, which the change removes from its source"},
} {
b := startBed(t)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
group, heads, _ := delivery.ParseCheckerVerdictKey(key)
v := delivery.CompositionVerdict{Group: group, Heads: heads, State: c.state, SDK: "v0.1.14-x", Machines: []string{"g14"}}
if c.state == delivery.CompositionFail {
v.Machine, v.Words = "g14", c.words
}
b.assignerWrote(t, key, v)
b.start(t, nil)
got := b.verdict(t, key, ended)
if got.Composition == nil || got.Composition.State != c.state {
t.Fatalf("%s: the verdict's composition is %+v", c.state, got.Composition)
}
o := b.owed(t, 7, head1, c.owed)
if !strings.Contains(o.Text, c.words) || strings.Contains(o.Text, "not yet run") {
t.Errorf("%s: the line says %q", c.state, o.Text)
}
}
}
@@ -0,0 +1,131 @@
package main
import (
"context"
"errors"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// compositionWatch reads the assigner's composition of a check (ADR 0528 clause 5) by watching its key and the
// assigner's presence: no poll and no timer. With no assigner online it answers check.ErrAssignerNotRunning at once.
type compositionWatch struct {
kv jetstream.KeyValue
// presence is the watchman's presence bucket; nil (not there) counts as no assigner online.
presence jetstream.KeyValue
}
// Await answers the composition under key once the assigner wrote one that does not wait (check.AwaitComposition).
func (c compositionWatch) Await(ctx context.Context, key string) (delivery.CompositionVerdict, error) {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
comps := make(chan check.CompositionEvent)
presence := make(chan check.PresenceEvent)
cw, err := c.kv.Watch(ctx, key)
if err != nil {
return delivery.CompositionVerdict{}, err
}
defer cw.Stop()
go func() {
for e := range cw.Updates() {
ev := check.CompositionEvent{Settled: e == nil}
if e != nil && e.Operation() == jetstream.KeyValuePut {
ev.Value = e.Value()
}
select {
case comps <- ev:
case <-ctx.Done():
return
}
}
}()
if c.presence == nil {
go func() {
select {
case presence <- check.PresenceEvent{Settled: true}:
case <-ctx.Done():
}
}()
} else {
pw, err := c.presence.Watch(ctx, "part."+delivery.AssignerModule+".*")
if err != nil {
return delivery.CompositionVerdict{}, err
}
defer pw.Stop()
go func() {
for e := range pw.Updates() {
ev := check.PresenceEvent{Settled: e == nil}
if e != nil {
ev.Key = e.Key()
if e.Operation() == jetstream.KeyValuePut {
if p, err := delivery.DecodePresence(e.Value()); err == nil {
ev.Online = p.Online
}
}
}
select {
case presence <- ev:
case <-ctx.Done():
return
}
}
}()
}
return check.AwaitComposition(ctx, presence, comps)
}
// followCompositions hands the checker each composition the assigner writes from now on, so a verdict that said its
// composition was not judged takes it in when the assigner starts (event-driven).
func followCompositions(ctx context.Context, kv jetstream.KeyValue, ch *check.Checker) error {
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
return err
}
go func() {
defer w.Stop()
for e := range w.Updates() {
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
if comp, err := delivery.DecodeCompositionVerdict(e.Value()); err == nil {
ch.Composed(ctx, comp)
}
}
}()
return nil
}
// lazyCompositions opens the assigner's compositions when a check first asks for one and the bucket is there: until it
// is, no assigner ever ran, so the check says at once that its composition is not judged (check.ErrAssignerNotRunning),
// and the next check looks again. No timer: each look is a check asking.
type lazyCompositions struct {
open func(ctx context.Context) (check.CompositionReader, error)
mu sync.Mutex
reader check.CompositionReader
}
// Await is the opened reader's Await, opening it first when it is not yet.
func (l *lazyCompositions) Await(ctx context.Context, key string) (delivery.CompositionVerdict, error) {
l.mu.Lock()
r := l.reader
if r == nil {
opened, err := l.open(ctx)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
l.mu.Unlock()
return delivery.CompositionVerdict{}, check.ErrAssignerNotRunning
case err != nil:
l.mu.Unlock()
return delivery.CompositionVerdict{}, err
}
l.reader, r = opened, opened
}
l.mu.Unlock()
return r.Await(ctx, key)
}
@@ -0,0 +1,106 @@
//go:build integration
package main
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// 1297/AC3 (its bus half): on the real bus, the checker's read of the assigner's composition waits while the key is
// absent and while the assigner says it waits, and answers the verdict the assigner then writes under the check's key.
func TestTheCheckerAwaitsTheAssignersComposition(t *testing.T) {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "comp")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
srv, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: filepath.Join(dir, "js"), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
nc, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.AssignerCompositionsBucket, History: delivery.AssignerCompositionsHistory})
if err != nil {
t.Fatal(err)
}
pkv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.PresenceBucket, History: 5})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PresencePartKey(delivery.AssignerModule, "novox")
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
if _, err := pkv.Put(ctx, pkey, on); err != nil {
t.Fatal(err)
}
// With no assigner online the wait answers at once (a second key, before the assigner of this test is online).
off, _ := delivery.EncodePresence(delivery.Presence{Online: false})
if _, err := pkv.Put(ctx, pkey, off); err != nil {
t.Fatal(err)
}
other, _ := delivery.CompositionKey("feat/y", "0123456789ab")
if _, err := (compositionWatch{kv: kv, presence: pkv}).Await(ctx, other); !errors.Is(err, check.ErrAssignerNotRunning) {
t.Fatalf("with no assigner online the wait answered %v", err)
}
if _, err := pkv.Put(ctx, pkey, on); err != nil {
t.Fatal(err)
}
key, _ := delivery.CompositionKey("feat/x", "0123456789ab")
got := make(chan delivery.CompositionVerdict, 1)
go func() {
v, err := compositionWatch{kv: kv, presence: pkv}.Await(ctx, key)
if err != nil {
t.Error(err)
}
got <- v
}()
put := func(v delivery.CompositionVerdict) {
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(ctx, key, data); err != nil {
t.Fatal(err)
}
}
time.Sleep(200 * time.Millisecond)
put(delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionWaits, Words: "the indexer holds no candidates yet"})
select {
case v := <-got:
t.Fatalf("a waiting composition was answered: %+v", v)
case <-time.After(300 * time.Millisecond):
}
put(delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "x", Machines: []string{"g14"}})
select {
case v := <-got:
if v.State != delivery.CompositionPass || v.Machines[0] != "g14" {
t.Errorf("answered %+v", v)
}
case <-time.After(5 * time.Second):
t.Fatal("the composition written was not answered")
}
}
@@ -0,0 +1,46 @@
package main
import (
"context"
"errors"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
type answering struct{ v delivery.CompositionVerdict }
func (a answering) Await(context.Context, string) (delivery.CompositionVerdict, error) {
return a.v, nil
}
// 1297 review item 3: the compositions bucket missing when the checker starts is looked up again at the next check: while
// it is missing a check says at once the assigner is not running; once it is there the check reads its composition, and
// it is opened once.
func TestTheCompositionsBucketIsLookedUpAtEachCheck(t *testing.T) {
opens, there := 0, false
l := &lazyCompositions{open: func(context.Context) (check.CompositionReader, error) {
opens++
if !there {
return nil, jetstream.ErrBucketNotFound
}
return answering{delivery.CompositionVerdict{State: delivery.CompositionPass, SDK: "x"}}, nil
}}
if _, err := l.Await(context.Background(), "k"); !errors.Is(err, check.ErrAssignerNotRunning) {
t.Fatalf("without the bucket: %v", err)
}
there = true
for range 2 {
v, err := l.Await(context.Background(), "k")
if err != nil || v.State != delivery.CompositionPass {
t.Fatalf("with the bucket: %+v, %v", v, err)
}
}
if opens != 2 {
t.Errorf("opened %d times, want twice (missing, then once)", opens)
}
}
@@ -0,0 +1,129 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"github.com/nats-io/nats.go"
"mesh-delivery-checker/internal/check"
)
// The forge's tools the checker reads with (mesh-catalog modules/gitea): gitea_get_file answers {file: {content}}, the text
// of a file at a ref, and gitea_compare answers {total_commits, commits}, what head holds that base does not. The checker
// writes nothing to the forge: the forge bridge sets the statuses (ADR 0484 clause 2).
const (
getFileTool = "gitea.gitea_get_file"
compareTool = "gitea.gitea_compare"
commitsTool = "gitea.gitea_commits"
// gitea_list_branches answers {branches: [{name, commit}]}, every branch with the commit it points at.
branchesTool = "gitea.gitea_list_branches"
)
// notFoundStatus is the 404 of the forge's error, `Gitea API <path>: <status> <body>`.
var notFoundStatus = regexp.MustCompile(`: 404( |$)`)
// forgeReads reads the forge through call (mesh-sdk process.Conn.Call as the checker's own account).
type forgeReads struct {
call func(key string, body any) (json.RawMessage, error)
}
// ask calls a forge tool; a call that reached nobody (no responders: the forge's tool runner re-serving, or the forge
// down) is marked Unheard, so the check fails at the forge and is judged again when the nodes change (issue 1624).
func (f forgeReads) ask(key string, body any) (json.RawMessage, error) {
raw, err := f.call(key, body)
if errors.Is(err, nats.ErrNoResponders) || errors.Is(err, nats.ErrTimeout) || errors.Is(err, context.DeadlineExceeded) {
return nil, &check.Unheard{Err: err}
}
return raw, err
}
func ownerRepo(repository string) (string, string) {
owner, repo, _ := strings.Cut(repository, "/")
return owner, repo
}
// File is the text of a file of a repository at a ref.
func (f forgeReads) File(_ context.Context, repository, path, ref string) (string, error) {
owner, repo := ownerRepo(repository)
raw, err := f.ask(getFileTool, map[string]string{"owner": owner, "repo": repo, "path": path, "ref": ref})
if err != nil {
if notFoundStatus.MatchString(err.Error()) {
// The forge answers 404 for a file the ref does not hold (issue 1845). The status is matched in its own place,
// after the path's colon: the path carries a sha that may hold the digits 404.
return "", fmt.Errorf("%w: %v", check.ErrNotFound, err)
}
return "", err
}
var a struct {
File *struct {
Content *string `json:"content"`
} `json:"file"`
}
if err := json.Unmarshal(raw, &a); err != nil || a.File == nil || a.File.Content == nil {
return "", fmt.Errorf("the answer of %s is not {file: {content}}: %v", getFileTool, err)
}
return *a.File.Content, nil
}
// OnTrunk says whether the commit is already on the repository's main: main holds it when the commit holds nothing main does
// not.
func (f forgeReads) OnTrunk(_ context.Context, repository, commit string) (bool, error) {
owner, repo := ownerRepo(repository)
raw, err := f.ask(compareTool, map[string]string{"owner": owner, "repo": repo, "base": "main", "head": commit})
if err != nil {
return false, err
}
var a struct {
Total *int `json:"total_commits"`
}
if err := json.Unmarshal(raw, &a); err != nil || a.Total == nil {
return false, fmt.Errorf("the answer of %s is not {total_commits}: %v", compareTool, err)
}
return *a.Total == 0, nil
}
// FullSHA is the whole sha of a commit named by its first digits: the newest commit of the history at that ref.
func (f forgeReads) FullSHA(_ context.Context, repository, commit string) (string, error) {
owner, repo := ownerRepo(repository)
raw, err := f.ask(commitsTool, map[string]any{"owner": owner, "repo": repo, "ref": commit, "limit": 1})
if err != nil {
return "", err
}
var a struct {
Commits []struct {
SHA string `json:"sha"`
} `json:"commits"`
}
if err := json.Unmarshal(raw, &a); err != nil || len(a.Commits) == 0 || len(a.Commits[0].SHA) != 40 || !strings.HasPrefix(a.Commits[0].SHA, commit) {
return "", fmt.Errorf("the answer of %s for %s is not a commit that starts with it: %v", commitsTool, commit, err)
}
return a.Commits[0].SHA, nil
}
// BranchHeads is the commit each branch of the repository points at.
func (f forgeReads) BranchHeads(_ context.Context, repository string) (map[string]string, error) {
owner, repo := ownerRepo(repository)
raw, err := f.ask(branchesTool, map[string]string{"owner": owner, "repo": repo})
if err != nil {
return nil, err
}
var a struct {
Branches *[]struct {
Name string `json:"name"`
Commit string `json:"commit"`
} `json:"branches"`
}
if err := json.Unmarshal(raw, &a); err != nil || a.Branches == nil {
return nil, fmt.Errorf("the answer of %s is not {branches: [{name, commit}]}: %v", branchesTool, err)
}
heads := map[string]string{}
for _, b := range *a.Branches {
heads[b.Name] = b.Commit
}
return heads, nil
}
@@ -0,0 +1,105 @@
package main
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/nats-io/nats.go"
"mesh-delivery-checker/internal/check"
)
// capturedAnswer is what the forge's tool answered (internal/check/testdata/captured/README).
func capturedAnswer(t *testing.T, name string) json.RawMessage {
t.Helper()
raw, err := os.ReadFile(filepath.Join("..", "..", "internal", "check", "testdata", "captured", name+".json"))
if err != nil {
t.Fatal(err)
}
return raw
}
func TestTheForgeReadsParseWhatTheForgeAnswered(t *testing.T) { // 1624/AC4
var asked []string
f := forgeReads{call: func(key string, body any) (json.RawMessage, error) {
b, _ := json.Marshal(body)
asked = append(asked, key+" "+string(b))
switch {
case key == getFileTool:
return capturedAnswer(t, "forge-get-file-go-mod"), nil
case strings.Contains(string(b), "cb54756c3865"):
return capturedAnswer(t, "forge-compare-trunk"), nil
}
return capturedAnswer(t, "forge-compare-branch"), nil
}}
text, err := f.File(context.Background(), "novox/mesh-delivery", "modules/mesh-delivery-builder/go.mod", "main")
if err != nil || !strings.Contains(text, "git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021100209-cb54756c3865") {
t.Errorf("file = %q, %v", text, err)
}
if on, err := f.OnTrunk(context.Background(), "novox/mesh-sdk", "cb54756c3865"); err != nil || !on {
t.Errorf("a commit on the trunk: %v, %v", on, err)
}
if on, err := f.OnTrunk(context.Background(), "novox/mesh-sdk", "9aaa5c80ac5c"); err != nil || on {
t.Errorf("a commit of the branch: %v, %v", on, err)
}
if len(asked) != 3 || !strings.Contains(asked[0], `"owner":"novox"`) || !strings.Contains(asked[2], `"base":"main"`) {
t.Errorf("asked %v", asked)
}
bad := forgeReads{call: func(string, any) (json.RawMessage, error) { return json.RawMessage(`{}`), nil }}
if _, err := bad.File(context.Background(), "novox/x", "go.mod", "main"); err == nil {
t.Error("an answer with no file was taken")
}
if _, err := bad.OnTrunk(context.Background(), "novox/x", "abc"); err == nil {
t.Error("an answer with no total was taken")
}
}
func TestTheForgeNamesTheWholeShaOfAShortCommit(t *testing.T) { // review D2
f := forgeReads{call: func(key string, body any) (json.RawMessage, error) {
if key != commitsTool {
t.Errorf("called %s", key)
}
return capturedAnswer(t, "forge-commits"), nil
}}
if sha, err := f.FullSHA(context.Background(), "novox/mesh-sdk", "cb54756c3865"); err != nil || sha != "cb54756c3865df5ac9c0dc0481ad9ebe2bf013a1" {
t.Errorf("sha = %q, %v", sha, err)
}
if _, err := f.FullSHA(context.Background(), "novox/mesh-sdk", "deadbeefdead"); err == nil {
t.Error("a commit that does not start with the pin was taken")
}
}
func TestAFileTheRefDoesNotHoldIsNotFoundAndOtherFailuresAreNot(t *testing.T) { // 1845/AC1, AC4
f := forgeReads{call: func(string, any) (json.RawMessage, error) {
return nil, errors.New("Gitea API contents/checks/particulars/go.mod?ref=98ca97fc: 404 object does not exist")
}}
if _, err := f.File(context.Background(), "novox/mesh-catalog", "checks/particulars/go.mod", "98ca97fc"); !errors.Is(err, check.ErrNotFound) {
t.Errorf("a 404 is %v, want ErrNotFound", err)
}
f = forgeReads{call: func(string, any) (json.RawMessage, error) {
return nil, errors.New("Gitea API contents/go.mod?ref=1111111111111111111111111111111404abcde: 500 internal error")
}}
if _, err := f.File(context.Background(), "novox/mesh-catalog", "go.mod", "main"); err == nil || errors.Is(err, check.ErrNotFound) {
t.Errorf("a 500 is %v, want a failure that is not ErrNotFound", err)
}
}
func TestAForgeReadThatReachesNobodyIsUnheard(t *testing.T) { // 1624 C7: "nats: no responders" on mesh-catalog #293
f := forgeReads{call: func(string, any) (json.RawMessage, error) { return nil, nats.ErrNoResponders }}
_, err := f.File(context.Background(), "novox/mesh-catalog", "checks/agenthome/go.mod", "950e37a46e8d")
var u *check.Unheard
if !errors.As(err, &u) {
t.Errorf("err = %v, want it marked as reaching nobody", err)
}
if _, err := f.OnTrunk(context.Background(), "novox/mesh-sdk", "abc"); !errors.As(err, &u) {
t.Errorf("compare: %v", err)
}
if _, err := f.FullSHA(context.Background(), "novox/mesh-sdk", "abc"); !errors.As(err, &u) {
t.Errorf("commits: %v", err)
}
}
@@ -0,0 +1,708 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
// controller's user and the checker's own principal <node>.<module>. It carries no per-principal permissions: the grants
// the controller composes from the manifest's state and reads (only the checker writes its buckets) are the controller's,
// and are proved by its own module check, not here.
const busConf = `
listen: 127.0.0.1:-1
jetstream { store_dir: %q }
accounts {
MESH {
jetstream: enabled
users = [
{ user: "ctl", password: "pw" }
{ user: "n1.mesh-delivery-checker", password: "pw" }
]
}
}
`
const (
repo = "novox/mesh-delivery"
head1 = "1111111111111111111111111111111111111111"
head2 = "2222222222222222222222222222222222222222"
trunk = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
merged = "fedcba9876543210fedcba9876543210fedcba98" // the merge commit of the captured fetch answer
digU = "aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11aa11"
digI = "bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22bb22"
)
var captured = filepath.Join("..", "..", "internal", "check", "testdata", "captured")
func readCaptured(t *testing.T, name string) map[string]any {
t.Helper()
raw, err := os.ReadFile(filepath.Join(captured, name+".json"))
if err != nil {
t.Fatal(err)
}
var v map[string]any
if err := json.Unmarshal(raw, &v); err != nil {
t.Fatal(err)
}
return v
}
// fakeBuildModules serve the tools of mesh-build-go and mesh-build-node as the real ones do (ADR 0469 clause 2): a call on
// mesh.mod.<module>.call.<tool>.<node>.<caller> answers at once with the step and its key, and the step's end is written to
// the key <node>.<id> of the module's steps bucket. What the steps end with are the results the build modules' own code
// marshalled (internal/check/testdata/captured, README).
type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
conflict atomic.Bool
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
testTool *testing.T
// beside is, by workspace, the siblings the fetch placed in it as `<owner/repo> <commit>`, sorted: a digest echoes them
// (1929/AC2; mesh-sdk buildstep.besideOf), or the level runs whole and keeps nothing.
beside map[string][]string
}
func (f *fakeBuildModules) count(tool string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[tool]
}
func (f *fakeBuildModules) result(t *testing.T, tool string, args map[string]any) json.RawMessage {
level, _ := args["level"].(string)
var v any
switch {
case strings.HasSuffix(tool, "_fetch"):
name := "go-fetch-clean"
if f.conflict.Load() {
name = "go-fetch-conflict"
}
fetched := readCaptured(t, name)
fetched["commit"], fetched["base"] = args["commit"], args["base"]
var refs []struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
}
raw, _ := json.Marshal(args["beside"])
_ = json.Unmarshal(raw, &refs)
var lines []string
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
sort.Strings(lines)
f.mu.Lock()
if f.beside == nil {
f.beside = map[string][]string{}
}
f.beside[fmt.Sprint(fetched["workspace"])] = lines
f.mu.Unlock()
v = fetched
case strings.HasSuffix(tool, "_digest"):
d := map[string]string{"unit": digU, "integration": digI}[level]
f.mu.Lock()
beside := f.beside[fmt.Sprint(args["workspace"])]
f.mu.Unlock()
v = map[string]any{"level": level, "packages": map[string]string{"fixture/slow": d}, "beside": beside}
case strings.HasSuffix(tool, "_test"):
r := readCaptured(t, "go-test-slow")
r["level"] = level
v = r
}
data, _ := json.Marshal(v)
return data
}
// serve makes module's steps bucket and its build seat's stream and workers, and pulls its asks on n1 as the real module
// does since R4g (ADR 0525; issue 1887 A5): each ask's job runs with step bodies answering the captured results, and a
// check takes the kept verdicts of the checker's kept bucket by their digests.
func (f *fakeBuildModules) serve(t *testing.T, module string, kept jetstream.KeyValue) {
kv, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
language := strings.TrimPrefix(module, "mesh-build-")
if err := buildstep.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
prefix := map[string]string{"mesh-build-go": "build_go_", "mesh-build-node": "build_node_"}[module]
call := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
tool := prefix + name
f.mu.Lock()
f.calls[module+"."+tool]++
f.mu.Unlock()
return f.result(t, tool, args), nil
}
}
steps := buildstep.AskSteps{Fetch: call("fetch"), Test: call("test"),
Kept: func(ctx context.Context, level, digest string) (int, bool, error) {
key, err := delivery.KeptKey(level, digest)
if err != nil {
return 0, false, err
}
e, err := kept.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return 0, false, nil
}
if err != nil {
return 0, false, err
}
v, err := delivery.DecodeKept(e.Value())
return v.Tests, err == nil, err
}}
if module == "mesh-build-go" {
steps.Digest = call("digest")
}
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: "n1", Put: put,
Capacity: func() int { return 2 }, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
go func() { _ = p.Pull(f.ctx) }()
}
// serveForge answers the forge's two reads as its tools do, with the answers the forge gave (internal/check/testdata/captured):
// the go.mod of the captured file, whose mesh-sdk pin is cb54756c3865, and gitea_compare for the pin: on the trunk, unless the
// pin is the one the test set as stale.
func (b *bed) serveForge(t *testing.T, nc *nats.Conn) {
reply := func(m *nats.Msg, name string) {
var result map[string]any
_ = json.Unmarshal(mustRead(t, name), &result)
answer, _ := json.Marshal(map[string]any{"result": result, "error": "", "node": "n0"})
_ = m.Respond(answer)
}
for _, tool := range []string{"gitea_get_file", "gitea_compare"} {
tool := tool
sub, err := nc.Subscribe("mesh.mod.gitea.tool."+tool, func(m *nats.Msg) {
var a map[string]string
_ = json.Unmarshal(m.Data, &a)
switch {
case tool == "gitea_get_file" && b.stalePin.Load():
content := strings.Replace(string(mustRead(t, "forge-get-file-go-mod")), "cb54756c3865", "9aaa5c80ac5c", 1)
var result map[string]any
_ = json.Unmarshal([]byte(content), &result)
answer, _ := json.Marshal(map[string]any{"result": result, "error": "", "node": "n0"})
_ = m.Respond(answer)
case tool == "gitea_get_file":
reply(m, "forge-get-file-go-mod")
case strings.HasPrefix(a["head"], "9aaa5c80ac5c"):
reply(m, "forge-compare-branch")
default:
reply(m, "forge-compare-trunk")
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
}
func mustRead(t *testing.T, name string) []byte {
t.Helper()
raw, err := os.ReadFile(filepath.Join(captured, name+".json"))
if err != nil {
t.Fatal(err)
}
return raw
}
type bed struct {
stalePin atomic.Bool
ctx context.Context
js jetstream.JetStream
buckets map[string]jetstream.KeyValue
tools map[string]jetstream.KeyValue
fake *fakeBuildModules
srvURL string
announce []string
mu sync.Mutex
}
// startBed starts the bus, the buckets the controller makes from the checker's and the other parts' manifests, the build
// modules' streams and the checker's durable consumers on them, one node's toolchains, the gate's source naming the trunk, and
// the fake build modules. It runs no checker.
func startBed(t *testing.T) *bed {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "c")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
conf := filepath.Join(dir, "bus.conf")
if err := os.WriteFile(conf, []byte(fmt.Sprintf(busConf, filepath.Join(dir, "js"))), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatal(err)
}
opts.NoLog, opts.NoSigs = true, true
srv, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
ctl, err := nats.Connect(srv.ClientURL(), nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(ctl.Close)
js, err := jetstream.New(ctl)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
t.Cleanup(cancel)
b := &bed{ctx: ctx, js: js, buckets: map[string]jetstream.KeyValue{}, tools: map[string]jetstream.KeyValue{}, srvURL: srv.ClientURL()}
for module, stream := range toolchainStreams {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{"mesh.stream." + module + ".event.toolchain.reported"}}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateOrUpdateConsumer(ctx, stream, jetstream.ConsumerConfig{Durable: "n1_mesh-delivery-checker", AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
b.tools[module] = kv
}
// The checker's stream of events, which the indexer and the forge bridge hear.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "CHECKS", Subjects: []string{"mesh.stream.mesh-delivery-checker.event.>"}}); err != nil {
t.Fatal(err)
}
for name, history := range map[string]int{delivery.CheckerVerdictsBucket: delivery.CheckerVerdictsHistory, delivery.CheckerKeptBucket: delivery.CheckerKeptHistory,
delivery.CheckerOwedBucket: delivery.CheckerOwedHistory, delivery.PreviewsBucket: delivery.PreviewsHistory, delivery.CandidatesBucket: delivery.CandidatesHistory,
delivery.ModulesBucket: delivery.ModulesHistory, delivery.ManifestsBucket: delivery.ManifestsHistory, delivery.SourceBucket: delivery.SourceHistory,
delivery.PresenceBucket: delivery.PresenceHistory, delivery.HaveBucket: delivery.HaveHistory, check.TriesBucket: 1} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: uint8(history)})
if err != nil {
t.Fatal(err)
}
b.buckets[name] = kv
}
// 1929/AC1: the indexer's value of the repository's module, with no TypeScript manifest, so the Go build module alone
// judges the repository (an unknown repository is judged by both, issue 1887 A12).
known, err := delivery.EncodeModule(delivery.Module{Repository: repo, Commit: trunk, Path: "modules/m", None2: "the fingerprint is computed with its chain"})
if err != nil {
t.Fatal(err)
}
if _, err := b.buckets[delivery.ModulesBucket].Put(ctx, "m", known); err != nil {
t.Fatal(err)
}
b.setOnline(t, "n1", true)
for _, module := range []string{"mesh-build-go", "mesh-build-node"} {
b.setHealth(t, "n1", module)
}
b.serveForge(t, ctl)
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.serve(t, "mesh-build-go", b.buckets[delivery.CheckerKeptBucket])
b.fake.serve(t, "mesh-build-node", b.buckets[delivery.CheckerKeptBucket])
b.report(t, "mesh-build-go", "n1", "go", "go1.27.1", "1")
b.report(t, "mesh-build-node", "n1", "typescript", "tsc5", "1")
b.setTrunk(t, map[string]string{repo: trunk})
return b
}
func (b *bed) setTrunk(t *testing.T, heads map[string]string) {
t.Helper()
data, err := delivery.EncodeSource(delivery.Source{Group: "g", Heads: heads})
if err != nil {
t.Fatal(err)
}
if _, err := b.buckets[delivery.SourceBucket].Put(b.ctx, delivery.SourceMainKey, data); err != nil {
t.Fatal(err)
}
}
func (b *bed) setOnline(t *testing.T, node string, online bool) {
t.Helper()
data, _ := delivery.EncodePresence(delivery.Presence{Online: online})
if _, err := b.buckets[delivery.PresenceBucket].Put(b.ctx, node, data); err != nil {
t.Fatal(err)
}
}
func (b *bed) setHealth(t *testing.T, node, module string) {
t.Helper()
data, err := delivery.EncodeHave(delivery.Have{Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.HaveKey(node, module)
if _, err := b.buckets[delivery.HaveBucket].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
}
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper()
data, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: recipe, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.ToolchainKey(node, language)
if _, err := b.tools[module].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, "mesh.stream."+module+".event.toolchain.reported", []byte(`{}`)); err != nil {
t.Fatal(err)
}
}
// candidate writes candidates.<group> as the indexer does (the SDK's EncodeCandidate is the indexer's own encoder).
func (b *bed) candidate(t *testing.T, group string, pulls ...delivery.CandidatePull) string {
t.Helper()
heads := make([]delivery.Head, len(pulls))
for i, p := range pulls {
heads[i] = p.Head
}
hash, err := delivery.HeadsHash(heads)
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: group, HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.CandidatesKey(group)
if _, err := b.buckets[delivery.CandidatesBucket].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
vk, _ := delivery.CheckerVerdictKey(group, hash)
return vk
}
func pull(repository string, number int, head string) delivery.CandidatePull {
return delivery.CandidatePull{Head: delivery.Head{Repository: repository, Number: number, Head: head}, Modules: []string{}}
}
// start runs the program as the checker of a node.
func (b *bed) start(t *testing.T, env map[string]string) {
t.Helper()
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(credential{URL: b.srvURL, Node: "n1", Module: "mesh-delivery-checker", User: "n1.mesh-delivery-checker", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
e := map[string]string{"MESH_BROKER_FILE": broker}
for k, v := range env {
e[k] = v
}
runCtx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return e[k] }, io.Discard) }()
t.Cleanup(func() { stop(); <-done })
}
// verdict reads the verdict key until pred holds of it, within 20 s.
func (b *bed) verdict(t *testing.T, key string, pred func(delivery.Verdict) bool) delivery.Verdict {
t.Helper()
var last delivery.Verdict
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.buckets[delivery.CheckerVerdictsBucket].Get(b.ctx, key)
if err != nil {
continue
}
if last, err = delivery.DecodeVerdict(e.Value()); err == nil && pred(last) {
return last
}
}
t.Fatalf("verdicts.%s = %+v within 20 s", key, last)
return last
}
func ended(v delivery.Verdict) bool { return v.State != delivery.VerdictRunning }
// owed reads the status the checker owes on a head.
func (b *bed) owed(t *testing.T, number int, head, state string) delivery.Owed {
t.Helper()
key, err := delivery.CheckerOwedKey(repo, number, head)
if err != nil {
t.Fatal(err)
}
var last delivery.Owed
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.buckets[delivery.CheckerOwedBucket].Get(b.ctx, key)
if err != nil {
continue
}
if last, err = delivery.DecodeCheckerOwed(e.Value()); err == nil && last.State == state {
return last
}
}
t.Fatalf("owed.%s = %+v within 20 s, want the state %s", key, last, state)
return last
}
// TestCheckerJudgesACandidateAndOwesItsStatus: a candidate the indexer wrote, one pull request of mesh-delivery, is fetched
// merged onto the trunk the gate names, digested and tested at both levels on the node, kept, previewed; the verdict passes
// with the counts of both levels and the composition not run, and the status mesh/check is owed on the head as success.
func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300/AC5, 1445/AC1, 1929/AC1
b := startBed(t)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
v := b.verdict(t, key, ended)
if v.State != delivery.VerdictPass || v.Levels[delivery.LevelUnit].Run != 2 || v.Levels[delivery.LevelIntegration].Run != 2 || v.Trunk[repo] != trunk {
t.Fatalf("verdict = %+v", v)
}
if v.Composition == nil || v.Composition.State != delivery.CompositionNotRun || len(v.Machines) != 1 || v.Machines[0] != "n1" {
t.Errorf("composition %+v machines %v", v.Composition, v.Machines)
}
o := b.owed(t, 7, head1, "success")
if o.Context != delivery.StatusContextCheck || !strings.HasPrefix(o.Text, "pass 111111111111 on aaaaaaaaaaaa") || !strings.Contains(o.Text, "unit 2 run") {
t.Errorf("owed = %+v", o)
}
for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_digest": 2, "mesh-build-go.build_go_test": 2} {
if got := b.fake.count(tool); got != want {
t.Errorf("%s called %d times, want %d", tool, got, want)
}
}
// What passed is kept once under each level's digest.
for _, k := range []string{"unit." + digU, "integration." + digI} {
if _, err := b.buckets[delivery.CheckerKeptBucket].Get(b.ctx, k); err != nil {
t.Errorf("kept.%s: %v", k, err)
}
}
// The preview of the head merged onto the trunk, with the tree and the manifest digest the fetch named.
pk, _ := delivery.PreviewKey(repo, head1, trunk)
e, err := b.buckets[delivery.PreviewsBucket].Get(b.ctx, pk)
if err != nil {
t.Fatalf("previews.%s: %v", pk, err)
}
pv, err := delivery.DecodePreview(e.Value())
if err != nil || pv.Merge != merged || pv.Modules["modules/m"].Tree != "1111111111111111111111111111111111111111" {
t.Errorf("preview = %+v, %v", pv, err)
}
}
// TestACheckOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every package
// taken from the first's kept verdicts: no test step runs and no second kept key appears.
func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2
b := startBed(t)
first := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
b.verdict(t, first, ended)
tests, kept := b.fake.count("mesh-build-go.build_go_test"), keptCount(t, b)
second := b.candidate(t, "feat/y", pull(repo, 8, head2))
v := b.verdict(t, second, ended)
if v.State != delivery.VerdictPass || v.Levels[delivery.LevelUnit].Run != 0 || v.Levels[delivery.LevelUnit].Kept != 2 || v.Levels[delivery.LevelIntegration].Run != 0 {
t.Fatalf("verdict = %+v, want everything taken from the kept verdicts", v)
}
if got := b.fake.count("mesh-build-go.build_go_test"); got != tests {
t.Errorf("build_go_test called %d times, was %d: a test ran for equal inputs", got, tests)
}
if got := keptCount(t, b); got != kept {
t.Errorf("%d kept keys, was %d", got, kept)
}
}
func keptCount(t *testing.T, b *bed) int {
t.Helper()
keys, err := b.buckets[delivery.CheckerKeptBucket].Keys(b.ctx)
if err != nil {
return 0
}
return len(keys)
}
// TestAHeadThatDoesNotMergeFailsNamingTheFilesAndRunsNoTest: the fetch answers the files in conflict; the verdict fails naming
// them, no test or digest step is run, a preview of the conflicts alone is written and the status is owed as failure.
func TestAHeadThatDoesNotMergeFailsNamingTheFilesAndRunsNoTest(t *testing.T) { // 1289/AC4
b := startBed(t)
b.fake.conflict.Store(true)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
v := b.verdict(t, key, ended)
if v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "files in conflict: a.go, sub/b.go") {
t.Fatalf("verdict = %+v", v)
}
if o := b.owed(t, 7, head1, "failure"); !strings.HasPrefix(o.Text, "failed: n1: ") || !strings.Contains(o.Text, "files in conflict: a.go, sub/b.go") {
t.Errorf("owed = %+v", o)
}
if n := b.fake.count("mesh-build-go.build_go_digest") + b.fake.count("mesh-build-go.build_go_test"); n != 0 {
t.Errorf("%d digest and test steps ran after a conflict", n)
}
pk, _ := delivery.PreviewKey(repo, head1, trunk)
if e, err := b.buckets[delivery.PreviewsBucket].Get(b.ctx, pk); err != nil {
t.Errorf("previews.%s: %v", pk, err)
} else if pv, err := delivery.DecodePreview(e.Value()); err != nil || strings.Join(pv.Conflicts, ",") != "a.go,sub/b.go" || pv.Merge != "" {
t.Errorf("preview = %+v, %v, want the files in conflict alone", pv, err)
}
}
// TestAnHqPullRequestGetsNoVerdictAndAnotherDoes: hq's pull request is judged by no one (ADR 0484 clause 8), the other one in
// the same run is.
// 1887/A10, ADR 0527 §2: a pull request into hq gets a verdict and a mesh/check from the checker, as any repository's.
func TestAnHqPullRequestGetsAVerdictAndAStatus(t *testing.T) {
b := startBed(t)
b.setTrunk(t, map[string]string{repo: trunk, "novox/hq": trunk})
hq := b.candidate(t, "feat/hq", pull("novox/hq", 3, head1))
b.start(t, nil)
if v := b.verdict(t, hq, ended); v.State != delivery.VerdictPass {
t.Fatalf("hq's verdict = %+v", v)
}
key, _ := delivery.CheckerOwedKey("novox/hq", 3, head1)
for deadline := time.Now().Add(20 * time.Second); ; time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.CheckerOwedBucket].Get(b.ctx, key); err == nil {
if o, err := delivery.DecodeCheckerOwed(e.Value()); err == nil && o.State == "success" {
return
}
}
if time.Now().After(deadline) {
t.Fatalf("hq#3 owes no success within 20 s")
}
}
}
// TestACheckLeftRunningIsTakenOverWhenTheCheckerStarts: the verdict key holds running, left by a checker that stopped; the
// started checker judges it again and ends it (ADR 0478 clause 6, as the builder's).
func TestACheckLeftRunningIsTakenOverWhenTheCheckerStarts(t *testing.T) { // N6/AC2
b := startBed(t)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
group, heads, _ := delivery.ParseCheckerVerdictKey(key)
left, err := delivery.EncodeVerdict(delivery.Verdict{Group: group, Heads: heads, Trunk: map[string]string{repo: trunk}, State: delivery.VerdictRunning})
if err != nil {
t.Fatal(err)
}
if _, err := b.buckets[delivery.CheckerVerdictsBucket].Create(b.ctx, key, left); err != nil {
t.Fatal(err)
}
b.start(t, nil)
if v := b.verdict(t, key, ended); v.State != delivery.VerdictPass {
t.Errorf("verdict = %+v", v)
}
}
// TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit: the go.mod the forge answers names a commit that main does not
// hold; the verdict fails naming the member and the commit and no test runs (ADR 0484 clause 9).
func TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit(t *testing.T) { // 1624/AC2, 1929/AC3
b := startBed(t)
b.stalePin.Store(true)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
v := b.verdict(t, key, ended)
if v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "9aaa5c80ac5c") || !strings.Contains(v.Failed.Words, repo+"#7") {
t.Fatalf("verdict = %+v", v)
}
// 1929/AC3: no assertion that no test ran: one ask runs fetch, digest and tests (ADR 0525 §3).
}
// triesStore is the tries bucket as the tools write it.
type triesStore struct {
kv jetstream.KeyValue
ctx context.Context
}
func (s triesStore) Keys() ([]string, error) {
keys, err := s.kv.Keys(s.ctx)
if err == jetstream.ErrNoKeysFound {
return nil, nil
}
return keys, err
}
func (s triesStore) Get(key string) (json.RawMessage, error) {
e, err := s.kv.Get(s.ctx, key)
if err == jetstream.ErrKeyNotFound {
return nil, nil
}
if err != nil {
return nil, err
}
return e.Value(), nil
}
func (s triesStore) Create(key string, value []byte) error {
_, err := s.kv.Create(s.ctx, key, value)
return err
}
// TestATryIsJudgedAndItsEndWrittenWithNoStatusOwed: a try written as checker_try writes it is taken by the running checker,
// judged through the build modules as a pull request's head merged onto its trunk, and ended pass in its key, with no
// verdict and no status owed (ADR 0501 §5).
func TestATryIsJudgedAndItsEndWrittenWithNoStatusOwed(t *testing.T) {
b := startBed(t)
b.start(t, nil)
store := triesStore{b.buckets[check.TriesBucket], b.ctx}
id, err := check.AskTry(store, map[string]string{repo: trunk}, check.TryAsk{Repository: repo, Head: head1}, time.Now())
if err != nil {
t.Fatal(err)
}
var last check.Try
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
raw, _ := store.Get(id)
if last, err = check.DecodeTry(raw); err == nil && last.Over() {
break
}
}
if last.State != check.TryPass || last.Trunk != trunk || last.Took == "" {
t.Fatalf("try = %+v, want passed on the trunk", last)
}
if b.fake.count("mesh-build-go.build_go_fetch") == 0 {
t.Errorf("the try fetched nothing")
}
for _, name := range []string{delivery.CheckerOwedBucket, delivery.CheckerVerdictsBucket, delivery.PreviewsBucket} {
if keys, _ := b.buckets[name].Keys(b.ctx); len(keys) != 0 {
t.Errorf("a try wrote %s: %v", name, keys)
}
}
}
// TestTheCheckerSaysOwedReadableOnTheBus: started, the checker says on its own states subject what it keeps, owed among
// the readable ones, which is what grants the forge module its read of owed (ADR 0495 §2-§3, ADR 0521; R4e finding R2-1).
func TestTheCheckerSaysOwedReadableOnTheBus(t *testing.T) { // R2-1
b := startBed(t)
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
b.start(t, nil)
events, err := b.js.Stream(b.ctx, "EVENTS")
if err != nil {
t.Fatal(err)
}
var said announce.States
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
m, err := events.GetLastMsgForSubject(b.ctx, announce.Subject("mesh-delivery-checker"))
if err != nil {
continue
}
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
break
}
if said.Module != "mesh-delivery-checker" || said.Node != "n1" || !said.Keeps("owed") || !said.Readable("owed") ||
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || said.Readable("tries") || len(said.States) != 5 {
t.Fatalf("the checker said %+v, want its five states with kept, owed, previews and verdicts readable (1929/AC4; 1887 A5b)", said)
}
}
@@ -0,0 +1,190 @@
package main
import (
"context"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-checker/internal/check"
)
// judgeDeps are the parts the judge of a check is made of.
type judgeDeps struct {
asks check.Asks
tools *buildrun.ToolchainSet
trunks *check.Trunks
kept check.KV
previews check.KV
repos *repoModules
forge check.Forge
announce check.Announcer
builds check.BuildStates
// compositions is the assigner's bucket of compositions; nil leaves every composition not run.
compositions check.CompositionReader
logf func(format string, args ...any)
}
func newJudge(d judgeDeps) check.Judge {
return &check.Examiner{Compositions: d.compositions, Asks: d.asks, Toolchains: d.tools,
Kept: check.Kept{Store: d.kept}, Previews: d.previews, Modules: d.repos, Packages: d.repos, Forge: d.forge, Runs: d.repos, Builds: d.builds, Announce: d.announce, Trunks: d.trunks, Logf: d.logf}
}
// manifestKV reads a manifest by its digest.
type manifestKV interface {
Get(ctx context.Context, key string) (value []byte, revision uint64, found bool, err error)
}
// repoModules knows which build modules judge a repository, from the indexer's modules and their manifests: the Go build
// module always (a repository's go.mod files are its own), the Node one when a module of the repository has a TypeScript
// artifact (ADR 0484 clause 10). Safe for use from the watch and from a check at once.
type repoModules struct {
manifests manifestKV
logf func(format string, args ...any)
mu sync.Mutex
langs map[string]map[string]bool // by manifest digest: the languages of its artifacts
tsPkgs map[string][]string // by manifest digest: the names of its TypeScript package artifacts
path map[string]string // by module: its directory in its repository
named map[string]map[string][]string // by module: each artifact's named paths, as the indexer keeps them
repoOf map[string]string // by module: its repository
commit map[string]string // by module: the commit its record was computed at (the commit the mesh runs)
typed map[string]string // by module: its manifest digest
}
// Put takes a change of the indexer's modules bucket in: a module's repository and the languages its manifest names.
func (r *repoModules) Put(ctx context.Context, key string, value []byte, deleted bool) {
r.mu.Lock()
if r.repoOf == nil {
r.repoOf, r.typed, r.langs, r.commit = map[string]string{}, map[string]string{}, map[string]map[string]bool{}, map[string]string{}
r.tsPkgs, r.path, r.named = map[string][]string{}, map[string]string{}, map[string]map[string][]string{}
}
if deleted {
delete(r.repoOf, key)
delete(r.typed, key)
delete(r.commit, key)
delete(r.path, key)
delete(r.named, key)
r.mu.Unlock()
return
}
r.mu.Unlock()
m, err := delivery.DecodeModule(value)
if err != nil {
return
}
r.mu.Lock()
r.repoOf[key] = m.Repository
r.typed[key] = m.Manifest
r.commit[key] = m.Commit
r.path[key] = m.Path
r.named[key] = m.ArtifactPaths
_, known := r.langs[m.Manifest]
r.mu.Unlock()
if m.Manifest == "" || known {
return
}
set := map[string]bool{}
var pkgs []string
if mk, err := delivery.ManifestsKey(m.Manifest); err == nil {
if raw, _, found, err := r.manifests.Get(ctx, mk); err == nil && found {
if kept, err := delivery.DecodeManifest(raw); err == nil {
if man, err := parseManifest(kept.JSON); err == nil && man.Build != nil {
for _, a := range man.Build.Artifacts {
if a.Language != "" {
set[a.Language] = true
}
if a.Kind == "package" && a.Language == "typescript" {
pkgs = append(pkgs, a.Name)
}
}
} else if err != nil && r.logf != nil {
r.logf("the manifest %s cannot be read, so its module counts as no TypeScript: %v", m.Manifest, err)
}
}
}
}
r.mu.Lock()
r.langs[m.Manifest] = set
r.tsPkgs[m.Manifest] = pkgs
r.mu.Unlock()
}
// PackagePaths are the paths, relative to the repository, that the TypeScript package artifacts of a repository's modules
// are built from, sorted: the named paths the indexer keeps in each module's value (delivery.Module.ArtifactPaths; only the
// indexer reads them from a manifest, the lead's decision L8), under the module's directory. None for a module whose value
// holds no paths.
func (r *repoModules) PackagePaths(repository string) []string {
r.mu.Lock()
defer r.mu.Unlock()
var out []string
for module, repo := range r.repoOf {
if repo != repository {
continue
}
dir := strings.Trim(r.path[module], "/")
for _, name := range r.tsPkgs[r.typed[module]] {
for _, p := range r.named[module][name] {
p = strings.Trim(strings.TrimPrefix(p, "./"), "/")
if dir != "" && dir != delivery.RootPath {
p = dir + "/" + p
}
out = append(out, p)
}
}
}
sort.Strings(out)
return out
}
// BuildModules are the build modules that judge a repository, sorted: the Go build module always, and the Node build
// module when a module of the repository builds TypeScript. A repository of which the mesh knows no module (mesh-lab, hq)
// is judged by both, each finding its own directories in the tree (go.mod files; package.json files), so a repository
// with tests in either language and no manifest is judged whole (issue 1887 A12).
func (r *repoModules) BuildModules(repository string) []string {
out := []string{buildrun.GoBuildModule}
r.mu.Lock()
defer r.mu.Unlock()
known := false
for module, repo := range r.repoOf {
if repo != repository {
continue
}
known = true
if r.langs[r.typed[module]]["typescript"] {
out = append(out, buildrun.NodeBuildModule)
break
}
}
if !known {
out = append(out, buildrun.NodeBuildModule)
}
sort.Strings(out)
return out
}
// buildReader reads the builder's build of a module at a fingerprint (mesh-delivery-builder_builds).
type buildReader struct{ kv kvStore }
func (r buildReader) Build(ctx context.Context, module, fp2 string) (delivery.BuilderBuild, bool, error) {
key, err := delivery.BuilderBuildKey(module, fp2)
if err != nil {
return delivery.BuilderBuild{}, false, err
}
value, _, found, err := r.kv.Get(ctx, key)
if err != nil || !found {
return delivery.BuilderBuild{}, false, err
}
b, err := delivery.DecodeBuilderBuild(value)
return b, err == nil, err
}
// CommitOf is the commit the indexer's record of the module was computed at: the commit of its repository the mesh runs.
func (r *repoModules) CommitOf(module string) string {
r.mu.Lock()
defer r.mu.Unlock()
return r.commit[module]
}
@@ -0,0 +1,143 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
type memManifests map[string][]byte
func (m memManifests) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
v, ok := m[key]
return v, 1, ok, nil
}
func putModule(t *testing.T, r *repoModules, name, repository, manifest string) {
t.Helper()
data, err := delivery.EncodeModule(delivery.Module{Repository: repository, Commit: strings.Repeat("a", 40), FP: "src1:" + strings.Repeat("a", 64), Manifest: delivery.ManifestDigest([]byte(manifest))})
if err != nil {
t.Fatal(err)
}
r.Put(context.Background(), name, data, false)
}
func TestARepositoryWithATypeScriptModuleIsJudgedByTheNodeBuildModuleToo(t *testing.T) { // 1292/AC2
own, err := os.ReadFile("../../module.json") // a manifest with a Go bundle: this module's own
if err != nil {
t.Fatal(err)
}
goManifest := string(own)
// A TypeScript bundle names neither a system nor a directory it is built from (the catalogue's parser says so).
var doc map[string]any
if err := json.Unmarshal(own, &doc); err != nil {
t.Fatal(err)
}
artifacts := doc["build"].(map[string]any)["artifacts"].([]any)
program := artifacts[0].(map[string]any)
program["language"] = "typescript"
delete(program, "system")
delete(program, "from")
delete(program, "binary")
tsRaw, err := json.Marshal(doc)
if err != nil {
t.Fatal(err)
}
tsManifest := string(tsRaw)
store := memManifests{}
for _, m := range []string{goManifest, tsManifest} {
data, err := delivery.EncodeManifest(delivery.Manifest{Module: "demo", JSON: m})
if err != nil {
t.Fatal(err)
}
store[delivery.ManifestDigest([]byte(m))] = data
}
r := &repoModules{manifests: store}
putModule(t, r, "mesh-delivery-checker", "novox/mesh-delivery", goManifest)
putModule(t, r, "mesh-ui", "novox/mesh-catalog", tsManifest)
if got := r.BuildModules("novox/mesh-delivery"); strings.Join(got, ",") != buildrun.GoBuildModule {
t.Errorf("mesh-delivery: %v, want the Go build module alone", got)
}
if got := r.BuildModules("novox/mesh-catalog"); strings.Join(got, ",") != buildrun.GoBuildModule+","+buildrun.NodeBuildModule {
t.Errorf("mesh-catalog: %v, want both", got)
}
if got := r.CommitOf("mesh-delivery-checker"); got != strings.Repeat("a", 40) || r.CommitOf("gone") != "" {
t.Errorf("the commit the mesh runs = %q", got)
}
// 1887/A12: a repository with no module the mesh knows (mesh-lab: nested go.mod files and a root package.json) is
// judged by both build modules.
if got := r.BuildModules("novox/mesh-lab"); strings.Join(got, ",") != buildrun.GoBuildModule+","+buildrun.NodeBuildModule {
t.Errorf("a repository with no module: %v, want both build modules", got)
}
}
// sdkManifest is mesh-sdk's own manifest (its module.json at feat/r4e-sdk): one TypeScript package naming its paths.
const sdkManifest = `{"module":"mesh-sdk","version":"1","slug":"sdk","build":{"artifacts":[{"name":"lib","kind":"package","language":"typescript",` +
`"paths":["src","test","package.json","package-lock.json","tsconfig.json"]}]},"resources":[]}`
func TestTheTypeScriptPackagesPathsComeFromTheIndexersModulesNotTheManifest(t *testing.T) { // ADR 0500 §5 with lead decision L8
store := memManifests{}
data, err := delivery.EncodeManifest(delivery.Manifest{Module: "mesh-sdk", JSON: sdkManifest})
if err != nil {
t.Fatal(err)
}
store[delivery.ManifestDigest([]byte(sdkManifest))] = data
r := &repoModules{manifests: store}
put := func(path string, paths map[string][]string) {
v, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: strings.Repeat("a", 40), FP: "src1:" + strings.Repeat("a", 64),
Manifest: delivery.ManifestDigest([]byte(sdkManifest)), Path: path, ArtifactPaths: paths})
if err != nil {
t.Fatal(err)
}
r.Put(context.Background(), "mesh-sdk", v, false)
}
put(".", map[string][]string{"lib": {"src", "package.json"}}) // the indexer's word, not the manifest's five paths
if got := strings.Join(r.PackagePaths("novox/mesh-sdk"), ","); got != "package.json,src" {
t.Errorf("paths = %q, want the indexer's, at the repository's root", got)
}
put("ts", map[string][]string{"lib": {"src"}})
if got := strings.Join(r.PackagePaths("novox/mesh-sdk"), ","); got != "ts/src" {
t.Errorf("paths = %q, want them under the module's directory", got)
}
put(".", nil) // a value written before the indexer kept paths: nothing to serve by
if got := r.PackagePaths("novox/mesh-sdk"); len(got) != 0 {
t.Errorf("paths = %v, want none", got)
}
if got := r.PackagePaths("novox/mesh-delivery"); len(got) != 0 {
t.Errorf("another repository: %v", got)
}
}
// A manifest whose config value falls back to a predefined key (ADR 0348 §7) is read, not logged as unreadable for the
// catalogue's predefined.json this process never holds: the indexer and the assigner read it with
// catalogue.ParseManifestForInputs (issues 1751, 1906). xterm's module.json from mesh-catalog at 835bf22b.
func TestAManifestFallingBackToAPredefinedKeyIsRead(t *testing.T) {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
manifest := string(raw)
data, err := delivery.EncodeManifest(delivery.Manifest{Module: "xterm", JSON: manifest})
if err != nil {
t.Fatal(err)
}
store := memManifests{delivery.ManifestDigest([]byte(manifest)): data}
var said []string
r := &repoModules{manifests: store, logf: func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }}
putModule(t, r, "xterm", "novox/mesh-catalog", manifest)
r.BuildModules("novox/mesh-catalog")
for _, s := range said {
if strings.Contains(s, "cannot be read") {
t.Errorf("the manifest was not read: %s", s)
}
}
if _, err := parseManifest(manifest); err != nil {
t.Errorf("manifestReader: %v", err)
}
}
@@ -0,0 +1,56 @@
package main
import (
"encoding/json"
"os"
"sort"
"testing"
)
// The checker says on the bus which states it keeps and which others may read (novox/hq ADR 0495 clauses 2 and 3): a
// reader is granted from that announcement. owed is read by the forge module, which writes mesh/check from it (ADR 0521,
// R4e round 2 finding R2-1), and by the forge bridge; previews by the indexer; verdicts by the gate. The list names every
// state of the manifest, so no state is refused at the call for want of being announced (clause 5).
func TestTheCheckerAnnouncesEveryStateItKeepsAndOwedReadable(t *testing.T) { // R2-1, ADR 0495 §2-§3, ADR 0521
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
State []struct{ Name string } `json:"state"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
var manifest, announced, readable []string
for _, s := range m.State {
manifest = append(manifest, s.Name)
}
for _, s := range keptStates {
announced = append(announced, s.Name)
if s.Readable {
readable = append(readable, s.Name)
}
}
sort.Strings(manifest)
sort.Strings(announced)
if !equal(manifest, announced) {
t.Errorf("announced %v, the manifest keeps %v", announced, manifest)
}
// kept is read by the Go build module for a check ask (issue 1887 design limit 1).
if want := []string{"kept", "owed", "previews", "verdicts"}; !equal(readable, want) {
t.Errorf("readable %v, want %v", readable, want)
}
}
func equal(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
@@ -0,0 +1,385 @@
// Command mesh-delivery-checker is delivery's checker (novox/hq ADR 0453 clause 4, ADR 0447, ADR 0484, issues 1678 to 1686): for
// each candidate the indexer writes it judges every pull request's head merged onto its trunk, through the build modules'
// fetch, digest and test steps, keeps the passing packages' verdicts, writes the verdict, and owes the forge the status
// `mesh/check`, which the forge bridge sets. It runs dark: nothing reads its answers to merge or deliver (ADR 0453 clause 4).
// It serves no request subject and holds no timer: every reaction is a watch or a bus event (ADR 0437 §8).
package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-checker/internal/check"
)
// eventSubject is the subject of an event of the checker's stream checks (the bus names the stream CHECKS): the indexer's
// watch of the previews bucket and the forge bridge's watch of the owed bucket are woken by them.
func eventSubject(event string) string {
return "mesh.stream." + delivery.CheckerModule + ".event." + event
}
const (
// defaultLink is the base of the address a status points at; the verdict key follows.
defaultLink = "mesh-delivery-view/checks"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-checker:", err)
os.Exit(1)
}
}
// credential is the broker file the controller issues to the module: JSON of the bus address, the certificate
// fingerprint to pin, the principal and its password (as the indexer reads it; issue 1242 moves this into the SDK).
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
func readCredential(path string) (credential, error) {
raw, err := os.ReadFile(path)
if err != nil {
return credential{}, err
}
var c credential
if err := json.Unmarshal(raw, &c); err != nil {
return credential{}, fmt.Errorf("the broker credential is not one: %w", err)
}
if c.URL == "" || c.User == "" {
return credential{}, errors.New("the broker credential names no bus address or user")
}
return c, nil
}
// pinned accepts exactly the certificate with this SHA-256 and no other.
func pinned(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
if got := "sha256:" + hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%.23s…)", got)
}
return nil
},
}
}
// durationEnv reads a duration setting from the environment, def when it is not set.
func durationEnv(getenv func(string) string, name string, def time.Duration) (time.Duration, error) {
v := strings.TrimSpace(getenv(name))
if v == "" {
return def, nil
}
d, err := time.ParseDuration(v)
if err != nil || d <= 0 {
return 0, fmt.Errorf("%s: %q is not a duration of more than 0", name, v)
}
return d, nil
}
// run is the program: it returns when ctx ends or the bus cannot be used.
// keptStates is every state the checker keeps and which others may read (ADR 0495 §2-§3): owed by the forge module,
// which writes mesh/check from it (ADR 0521), and by the forge bridge; previews by the indexer; verdicts by the gate.
var keptStates = []announce.State{
// The kept verdicts are read by the Go build module, which takes them for a check ask's unchanged packages (issue 1887's
// design limit 1); the checker alone writes them (ADR 0444 clause 5).
{Name: "kept", Readable: true},
{Name: "owed", Readable: true},
{Name: "previews", Readable: true},
{Name: "tries"},
{Name: "verdicts", Readable: true},
}
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-checker: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to hear the bus with")
}
cred, err := readCredential(path)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
nats.CustomInboxPrefix("_INBOX." + cred.User),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
bucket := func(name string) (jetstream.KeyValue, error) {
kv, err := js.KeyValue(ctx, name)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", name, err)
}
return kv, nil
}
buckets := map[string]jetstream.KeyValue{}
for _, name := range []string{delivery.CheckerVerdictsBucket, delivery.CheckerKeptBucket, delivery.CheckerOwedBucket, delivery.PreviewsBucket,
delivery.CandidatesBucket, delivery.ModulesBucket, delivery.ManifestsBucket, check.TriesBucket} {
if buckets[name], err = bucket(name); err != nil {
return err
}
}
// Calls to the build modules go out as this module's own account, naming the machine (mesh-sdk process.CallOn); the
// buckets are read and written on the connection above.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
caller, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer caller.Close()
// What the checker keeps, said on the bus (ADR 0495 §2-§3): a reader is granted from it. Not fatal when it cannot be said
// now: process.Conn says it again at every reconnect.
if err := caller.Keeps(keptStates...); err != nil {
logger.Printf("what the checker keeps could not be said on the bus, so readers of owed, previews and verdicts are not granted until it is: %v", err)
}
stepBound, err := durationEnv(getenv, "MESH_CHECKER_STEP_BOUND", check.DefaultStepBound)
if err != nil {
return err
}
checkBound, err := durationEnv(getenv, "MESH_CHECKER_BOUND", check.DefaultBound)
if err != nil {
return err
}
link := strings.TrimSpace(getenv("MESH_CHECKER_LINK"))
if link == "" {
link = defaultLink
}
tools := &buildrun.ToolchainSet{}
presence := &buildrun.PresenceSet{}
trunks := &check.Trunks{}
// A check is one ask per pull request and build module on the build seats' check stream, pulled by a build node with a
// free slot (novox/hq ADR 0525 §1, §3): the checker chooses no node and starts no step.
asks := buildrun.JetStreamAsks{JS: js}
ch := &check.Checker{
Verdicts: kvStore{buckets[delivery.CheckerVerdictsBucket]},
Owed: kvStore{buckets[delivery.CheckerOwedBucket]},
Trunks: trunks,
Bound: checkBound,
Link: link,
Logf: logger.Printf,
}
// The builder's builds say how the candidates' fingerprints stand (issue 1684); where its bucket is not there yet the line
// says nothing of the builds, until the checker starts again.
var builds check.BuildStates
switch kv, err := js.KeyValue(ctx, delivery.BuilderBuildsBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: the line says nothing of the builds until the checker starts again", delivery.BuilderBuildsBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.BuilderBuildsBucket, err)
default:
builds = buildReader{kvStore{kv}}
}
// The assigner's compositions (ADR 0528 clause 5, issue 1297): their bucket is looked up at each check until it is
// there, so an assigner delivered after the checker started is read at the next check, and followed from then on.
compositions := &lazyCompositions{open: func(at context.Context) (check.CompositionReader, error) {
kv, err := js.KeyValue(at, delivery.AssignerCompositionsBucket)
if err != nil {
return nil, err
}
cw := compositionWatch{kv: kv}
if pkv, err := js.KeyValue(at, delivery.PresenceBucket); err == nil {
cw.presence = pkv
}
if err := followCompositions(ctx, kv, ch); err != nil {
return nil, fmt.Errorf("the bucket %s cannot be watched: %w", delivery.AssignerCompositionsBucket, err)
}
logger.Printf("the assigner's compositions are read from %s", delivery.AssignerCompositionsBucket)
return cw, nil
}}
// The tries (ADR 0501 §5) are judged by the same judge, held by the same deliveries.
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, Logf: logger.Printf}
announce := func(ctx context.Context, event, key string) {
body, _ := json.Marshal(map[string]string{"key": key})
if _, err := js.Publish(ctx, eventSubject(event), body); err != nil {
logger.Printf("%s of %s was written and the part that reads it could not be told: %v", event, key, err)
}
}
repos := &repoModules{manifests: kvStore{buckets[delivery.ManifestsBucket]}, logf: logger.Printf}
ch.Judge = newJudge(judgeDeps{
asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: stepBound}, tools: tools, trunks: trunks,
kept: kvStore{buckets[delivery.CheckerKeptBucket]}, previews: kvStore{buckets[delivery.PreviewsBucket]},
repos: repos, builds: builds, forge: forgeReads{call: caller.Call}, logf: logger.Printf,
announce: announce, compositions: compositions,
})
ch.Announce = announce
tryer.Judge = ch.Judge
// The toolchains first, so a check that starts from the first candidate finds the nodes.
watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, ch.NodesChanged, logger)
if err != nil {
return err
}
if err := watch.start(); err != nil {
return err
}
// A build module says each time it wrote its toolchains key: its stream, and the checker's durable consumer on it, are
// made by the controller. Where the stream is not there yet the checker says so and is started again with the build
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := js.Consumer(ctx, stream, cred.Node+"_"+cred.Module)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the checker starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := cons.Consume(func(m jetstream.Msg) { watch.Reported(m) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
if err != nil {
return err
}
defer cc.Stop()
}
errs := make(chan error, 8)
// The watchman's presence: a node coming back online asks again a check whose forge was not heard (UnheardWords), and
// the assigner's own presence says whether a composition is being made (compositionWatch, ADR 0537 §4). A check is no
// longer held for a node: its asks wait in the build seats' streams, which redeliver a lost node's ask (ADR 0525 §6,
// issue 1912), so the node-engine's have is not watched here.
switch kv, err := js.KeyValue(ctx, delivery.PresenceBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no node counts as online until the checker starts again", delivery.PresenceBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.PresenceBucket, err)
default:
pw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { presence.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, pw, func(e jetstream.KeyValueEntry) {
if presence.Put(e.Key(), e.Value(), !opOf(e)) {
ch.NodesChanged(ctx) // a node came online or went: a check whose forge was not heard may now be heard
}
})
}()
}
// The gate's source says the trunk head of each repository, which a pull request is merged onto. Where it is not there
// yet no trunk is known, so nothing is checked.
switch kv, err := js.KeyValue(ctx, delivery.SourceBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no trunk is known until the checker starts again", delivery.SourceBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.SourceBucket, err)
default:
sw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) {
if e.Key() == delivery.SourceMainKey {
trunks.Put(e.Value(), !opOf(e))
}
})
if err != nil {
return err
}
go func() {
errs <- follow(ctx, sw, func(e jetstream.KeyValueEntry) {
if e.Key() == delivery.SourceMainKey && trunks.Put(e.Value(), !opOf(e)) {
ch.TrunkMoved(ctx) // the trunk moved: its statuses go pending, and a verdict for the old trunk is judged again
}
})
}()
}
// The modules say which repository holds a TypeScript module, so the Node build module judges it too.
mw, err := settle(ctx, buckets[delivery.ModulesBucket], func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, mw, func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
}()
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
ch.Candidate(ctx, e.Key(), e.Value())
} else {
ch.Forget(e.Key())
}
}
w, err := settle(ctx, buckets[delivery.CandidatesBucket], each)
if err != nil {
return err
}
// A key left running for heads no candidate holds is closed; the candidates are all known now.
if err := ch.Sweep(ctx); err != nil {
return err
}
// The tries: a queued one starts, one left running is taken over, an ended one past its keeping is removed.
tw, err := settle(ctx, buckets[check.TriesBucket], func(jetstream.KeyValueEntry) {})
if err != nil {
return err
}
tryer.Reconsider(ctx)
go func() {
errs <- follow(ctx, tw, func(e jetstream.KeyValueEntry) {
if opOf(e) {
tryer.Put(ctx, e.Key(), e.Value())
}
})
}()
logger.Printf("started on %s", cred.Node)
go func() { errs <- follow(ctx, w, each) }()
select {
case <-ctx.Done():
ch.Wait()
tryer.Wait()
return nil
case err := <-errs:
return err
case err := <-assignErrs:
return err
case err := <-watch.errs:
return err
}
}
@@ -0,0 +1,21 @@
package main
import (
"os"
"strings"
"testing"
)
// 1912: with checks as asks on the build seats' streams, the stream redelivers a lost node's ask (ADR 0525 §6), so the
// checker program opens no node-engine have to start again a check that failed for want of a node.
func TestTheCheckerWatchesNoHave(t *testing.T) {
raw, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
for _, s := range []string{"delivery.HaveBucket", "buildrun.HaveSet"} {
if strings.Contains(string(raw), s) {
t.Errorf("the checker program still reads %s", s)
}
}
}
@@ -0,0 +1,174 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node; it is passed on as deleted otherwise,
// and passed on again when the module is assigned there again. Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]bool // <node>.<module> with an open assignment
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]bool // <module>/<key> said as left out, so it is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]bool{},
raw: map[string]map[string][]byte{}, left: map[string]bool{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in and answers whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
open := !deleted && openAssignment(value)
g.mu.Lock()
defer g.mu.Unlock()
if g.standing[key] == open {
return false
}
if open {
g.standing[key] = true
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned (or the assignments are not known), deleted
// otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
if err != nil || !g.known || g.standing[node+"."+module] {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if !g.left[module+"/"+key] {
g.left[module+"/"+key] = true
g.logf("%s.%s is left out of the agreed toolchain: %s has no assignment on %s, so it is no builder (issue 1940)",
delivery.ToolchainsBucket(module), key, module, node)
}
return g.forward(module, key, nil, true)
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands: one that has not
// ended and has no state (a pending one is not on the machine yet; mesh-controller internal/chain Assignment).
func openAssignment(value []byte) bool {
var a struct {
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return false
}
return a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e)) }
w, err := settle(ctx, kv, each)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", provider.AssignmentsBucket, err)
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e)) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -0,0 +1,112 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false)
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false)
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true)
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
}
// A newly assigned build node that has not reported yet changes nothing; its report counts as it comes.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
set := &buildrun.ToolchainSet{}
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(string, ...any) {})
gate.Settled()
if gate.Assignment("g14.mesh-build-node", []byte(`{}`), false) {
t.Fatal("an assignment with no report passed something on")
}
key, value := gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an assigned node's report was not passed on")
}
key, value = gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("a report of a node with no assignment key was passed on after the assignments settled")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,106 @@
package main
import (
"context"
"errors"
"fmt"
"log"
"strings"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// toolchainStreams are the stream each build module carries its toolchain.reported event on, as the bus names it
// (declared by the build modules, heard by this module through consumes).
var toolchainStreams = map[string]string{buildrun.GoBuildModule: "TOOLCHAINS_GO", buildrun.NodeBuildModule: "TOOLCHAINS_NODE"}
// toolchainWatch keeps the build modules' toolchains in a buildrun.ToolchainSet and tells the builder when one changed. A
// build module's bucket is made when it is assigned, after the builder started, and the builder holds no timer (ADR 0437
// §8): it watches the buckets that exist at start, and a build module says each time it wrote its toolchains key
// (toolchain.reported, handled by Reported); a bucket not watched yet is then watched.
type toolchainWatch struct {
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
// put takes a change of a toolchains bucket in: set.Put, or the gate of the assigned build nodes before it (issue 1940).
put func(module, key string, value []byte, deleted bool) bool
changed func(ctx context.Context) // a language's entries changed: the failed builds are tried again
logger *log.Logger
errs chan error
mu sync.Mutex
watched map[string]bool
}
func newToolchainWatch(ctx context.Context, js bucketOpener, set *buildrun.ToolchainSet, changed func(context.Context), logger *log.Logger) *toolchainWatch {
return &toolchainWatch{ctx: ctx, js: js, set: set, put: set.Put, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
}
// start watches the buckets that exist now; a missing one is said and waits for its build module's event.
func (w *toolchainWatch) start() error {
for module := range buildrun.LanguageOf {
if err := w.ensure(module); err != nil {
return err
}
}
return nil
}
func (w *toolchainWatch) ensure(module string) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.watched[module] {
return nil
}
name := delivery.ToolchainsBucket(module)
kv, err := w.js.KeyValue(w.ctx, name)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
w.logger.Printf("the bucket %s does not exist yet: it is watched when %s reports a toolchain", name, module)
return nil
case err != nil:
return fmt.Errorf("the bucket %s: %w", name, err)
}
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.put(module, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
w.watched[module] = true
go func() {
w.errs <- follow(w.ctx, watcher, func(e jetstream.KeyValueEntry) {
if w.put(module, e.Key(), e.Value(), !opOf(e)) {
w.changed(w.ctx)
}
})
}()
w.changed(w.ctx) // what the settle read is new to the builder
return nil
}
// delivered is the part of a JetStream message the watch uses.
type delivered interface {
Subject() string
Ack() error
Term() error
}
// Reported takes one toolchain.reported event of a build module in: the module is read off the subject,
// mesh.stream.<module>.event.toolchain.reported, and its bucket is watched if it is not. A subject of no build module is
// terminated; a failure stays unacknowledged, and the server delivers it again.
func (w *toolchainWatch) Reported(m delivered) {
module, _, _ := strings.Cut(strings.TrimPrefix(m.Subject(), "mesh.stream."), ".event.")
if _, known := buildrun.LanguageOf[module]; !known || !strings.HasPrefix(m.Subject(), "mesh.stream.") {
w.logger.Printf("%s: not a build module's event, dropped", m.Subject())
_ = m.Term()
return
}
if err := w.ensure(module); err != nil {
w.logger.Printf("%s: left for redelivery: %v", m.Subject(), err)
return
}
_ = m.Ack()
}
+22
View File
@@ -0,0 +1,22 @@
module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27
View File
@@ -0,0 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,141 @@
package check
import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ErrAssignerNotRunning is what a wait for a composition answers when no instance of the assigner is online: nobody is
// making it, so the check does not wait for it (issue 1297). The verdict says the composition is not judged, and when the
// assigner starts and writes it, the checker takes it in then (Checker.Composed).
var ErrAssignerNotRunning = errors.New(delivery.AssignerNotRunning)
// PresenceEvent is a change of the watchman's presence bucket, or Settled once what stands was handed.
type PresenceEvent struct {
Key string
Online bool
Settled bool
}
// CompositionEvent is a change of the composition's key (its value, nil for a delete), or Settled once what stands was
// handed.
type CompositionEvent struct {
Value []byte
Settled bool
}
// AwaitComposition answers the assigner's composition once one is written that does not wait, while an instance of the
// assigner is online by the watchman's presence (`part.mesh-delivery-assigner.<instance>`). Once both watches handed
// what stands, no instance online is ErrAssignerNotRunning at once, and so is the last instance going offline during the
// wait. No timer: every answer follows an event; ctx ending is ctx's error.
func AwaitComposition(ctx context.Context, presence <-chan PresenceEvent, comps <-chan CompositionEvent) (delivery.CompositionVerdict, error) {
online := map[string]bool{}
presenceSettled, compSettled := false, false
for {
select {
case <-ctx.Done():
return delivery.CompositionVerdict{}, ctx.Err()
case e, ok := <-presence:
if !ok {
return delivery.CompositionVerdict{}, errors.New("the watch of the assigner's presence ended")
}
if e.Settled {
presenceSettled = true
break
}
if _, part, instance, err := delivery.ParsePresenceKey(e.Key); err == nil && part == delivery.AssignerModule {
if e.Online {
online[instance] = true
} else {
delete(online, instance)
}
}
case e, ok := <-comps:
if !ok {
return delivery.CompositionVerdict{}, errors.New("the watch of the composition ended")
}
if e.Settled {
compSettled = true
break
}
if e.Value == nil {
break
}
v, err := delivery.DecodeCompositionVerdict(e.Value)
if err != nil {
return delivery.CompositionVerdict{}, fmt.Errorf("the composition cannot be read: %w", err)
}
if v.State != delivery.CompositionWaits {
return v, nil
}
}
if presenceSettled && compSettled && len(online) == 0 {
return delivery.CompositionVerdict{}, ErrAssignerNotRunning
}
}
}
// withComposition is v with the assigner's composition taken in: a failed one fails the check, naming its machine.
func withComposition(v *delivery.Verdict, comp delivery.CompositionVerdict) {
v.Composition = comp.Composition()
v.Took.Composition = comp.Took
if comp.State == delivery.CompositionFail {
machine := comp.Machine
if machine == "" {
machine = "the composition"
}
v.State, v.Failed = delivery.VerdictFail, delivery.FailedFirst{Machine: machine, Words: comp.Words}
}
}
// notJudged says whether a verdict's composition was not judged because the assigner was not running.
func notJudged(v delivery.Verdict) bool {
return v.Composition != nil && v.Composition.State == delivery.CompositionNotRun && v.Composition.Words == delivery.AssignerNotRunning
}
// Composed takes in a composition the assigner wrote after the check's verdict said it was not judged because the
// assigner was not running (the assigner started since): the verdict and its status are written again with it. A
// composition of heads no candidate holds now, or of a verdict that judged its composition, is left.
func (ch *Checker) Composed(ctx context.Context, comp delivery.CompositionVerdict) {
ch.init()
if comp.State == delivery.CompositionWaits {
return
}
ckey, err := delivery.CandidatesKey(comp.Group)
if err != nil {
return
}
ch.mu.Lock()
c, ok := ch.known[ckey]
ch.mu.Unlock()
if !ok {
return
}
cs := CaseOf(c, ch.heads())
if cs == nil || cs.HeadsHash != comp.Heads {
return
}
cur, rev, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return
}
v, err := delivery.DecodeVerdict(cur)
if err != nil || v.State != delivery.VerdictPass || !notJudged(v) {
return
}
withComposition(&v, comp)
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: the verdict with its composition cannot be written: %v", cs.Key, err)
return
}
if _, err := ch.Verdicts.Update(ctx, cs.Key, data, rev); err != nil {
ch.logf("%s: the verdict with its composition could not be written: %v", cs.Key, err)
return
}
ch.writeOwed(ctx, cs, v, Outcome{Verdict: v})
ch.logf("%s: the composition came after the verdict: %s", cs.Key, comp.State)
}
@@ -0,0 +1,149 @@
package check
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func assignerOn(instance string, online bool) PresenceEvent {
key, _ := delivery.PresencePartKey(delivery.AssignerModule, instance)
return PresenceEvent{Key: key, Online: online}
}
func compEvent(t *testing.T, v delivery.CompositionVerdict) CompositionEvent {
t.Helper()
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
return CompositionEvent{Value: data}
}
var passed = delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "x", Machines: []string{"g14"}}
// 1297/B3d: with no assigner online, the wait answers ErrAssignerNotRunning as soon as both watches handed what stands.
func TestNoAssignerAnswersAtOnce(t *testing.T) {
presence, comps := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence <- assignerOn("novox", false)
presence <- PresenceEvent{Settled: true}
comps <- CompositionEvent{Settled: true}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if _, err := AwaitComposition(ctx, presence, comps); !errors.Is(err, ErrAssignerNotRunning) {
t.Fatalf("answered %v, want the assigner not running", err)
}
}
// 1297/B3d: with the assigner online it waits for the key, past a composition that waits, and answers the verdict.
func TestAnOnlineAssignerIsWaitedFor(t *testing.T) {
presence, comps := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence <- assignerOn("novox", true)
presence <- PresenceEvent{Settled: true}
comps <- CompositionEvent{Settled: true}
done := make(chan error, 1)
var got delivery.CompositionVerdict
go func() {
var err error
got, err = AwaitComposition(context.Background(), presence, comps)
done <- err
}()
comps <- compEvent(t, delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionWaits, Words: "w"})
select {
case err := <-done:
t.Fatalf("answered before the composition was written: %v", err)
case <-time.After(50 * time.Millisecond):
}
comps <- compEvent(t, passed)
if err := <-done; err != nil || got.State != delivery.CompositionPass {
t.Fatalf("answered %+v, %v", got, err)
}
// And the last instance going offline during a wait ends it as not running.
presence2, comps2 := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence2 <- assignerOn("novox", true)
presence2 <- PresenceEvent{Settled: true}
comps2 <- CompositionEvent{Settled: true}
presence2 <- assignerOn("novox", false)
if _, err := AwaitComposition(context.Background(), presence2, comps2); !errors.Is(err, ErrAssignerNotRunning) {
t.Errorf("the assigner going offline answered %v", err)
}
}
// 1297/B3d: a passing check with no assigner says at once, loud, that its composition is not judged; never a pass of it.
func TestAPassingCheckWithNoAssignerSaysTheCompositionIsNotJudged(t *testing.T) {
b := newBench(t)
b.ex.Compositions = &fakeCompositions{err: ErrAssignerNotRunning}
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.Composition == nil || v.Composition.State != delivery.CompositionNotRun || v.Composition.Words != delivery.AssignerNotRunning {
t.Fatalf("composition = %+v", v.Composition)
}
if words := compositionWords(v.Composition, 0); words != "composition not judged: the assigner is not running" {
t.Errorf("the line says %q", words)
}
}
// 1297/B3d: the assigner starting writes the composition later, and the checker takes it into the verdict that said it
// was not judged, writing the status again; a failed one fails the check.
func TestTheAssignerStartingBringsTheComposition(t *testing.T) {
for _, comp := range []delivery.CompositionVerdict{passed,
{Group: "feat/x", Heads: "", State: delivery.CompositionFail, SDK: "x", Machine: "g14", Words: "g14 runs web, which the change removes"}} {
ch, cs := checkerWithNotJudged(t)
comp.Heads = cs.HeadsHash
ch.Composed(context.Background(), comp)
raw, _, _, _ := ch.Verdicts.Get(context.Background(), cs.Key)
v, err := delivery.DecodeVerdict(raw)
if err != nil || v.Composition == nil || v.Composition.State != comp.State {
t.Fatalf("after the composition came, the verdict is %+v, %v", v, err)
}
if comp.State == delivery.CompositionFail && (v.State != delivery.VerdictFail || v.Failed.Machine != "g14") {
t.Errorf("a failed composition left the verdict %+v", v)
}
owed := ch.Owed.(*memKV)
keys, _ := owed.Keys(context.Background())
if len(keys) == 0 {
t.Fatal("no status was written again")
}
data, _, _, _ := owed.Get(context.Background(), keys[0])
o, _ := delivery.DecodeCheckerOwed(data)
if !strings.Contains(o.Text, "composition passed") && !strings.Contains(o.Text, "g14 runs web") {
t.Errorf("the status's line says %q", o.Text)
}
}
}
// checkerWithNotJudged is a checker holding a candidate whose passing verdict said its composition was not judged.
func checkerWithNotJudged(t *testing.T) (*Checker, *Case) {
t.Helper()
cs := caseOf(t, pullOf(repoA, 7, head1))
raw := candidateOf(t, cs.Group, pullOf(repoA, 7, head1))
ch := &Checker{Verdicts: newMemKV(), Owed: newMemKV(), Trunks: fixedTrunks{cs.Trunk}}
ckey, _ := delivery.CandidatesKey(cs.Group)
ch.init()
c, err := delivery.DecodeCandidate(raw)
if err != nil {
t.Fatal(err)
}
ch.known[ckey] = c
if got := CaseOf(c, ch.heads()); got == nil || got.Key != cs.Key {
t.Fatalf("the case is %+v, want %s", got, cs.Key)
}
v := delivery.Verdict{Group: cs.Group, Heads: cs.Heads, State: delivery.VerdictPass,
Levels: map[string]delivery.LevelResult{delivery.LevelUnit: {Run: 1}, delivery.LevelIntegration: {Run: 1}},
Composition: &delivery.Composition{State: delivery.CompositionNotRun, Words: delivery.AssignerNotRunning}}
data, err := delivery.EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := ch.Verdicts.Create(context.Background(), cs.Key, data); err != nil {
t.Fatal(err)
}
return ch, cs
}
type fixedTrunks struct{ heads map[string]string }
func (f fixedTrunks) Heads() map[string]string { return f.heads }
@@ -0,0 +1,110 @@
package check
import (
"context"
"fmt"
"strconv"
)
// BesideRef is a sibling repository a repository's tests read, placed beside the fetched tree as a plain directory (ADR
// 0484 clause 4). Ref names the pull request's head when the commit is only there.
type BesideRef struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Ref string `json:"ref,omitempty"`
}
const (
catalogueRepository = "novox/mesh-catalog"
nodeEngineRepo = "novox/mesh-host"
nodeEngineModule = "mesh-host"
controllerRepo = "novox/mesh-controller"
)
// besideRules are the repositories each core repository's tests read, as the tests state it (mesh-sdk go/internal/beside
// beside.go): the catalogue, the node-engine, the SDK, and the delivery's pinned forge dependencies (mesh-catalog
// checks/deps). Which commit of each is besideCommit's rule.
var besideRules = map[string][]string{
controllerRepo: {catalogueRepository, nodeEngineRepo, SDKRepository},
nodeEngineRepo: {catalogueRepository, SDKRepository, controllerRepo},
SDKRepository: {catalogueRepository, nodeEngineRepo},
catalogueRepository: {SDKRepository, controllerRepo},
"novox/mesh-delivery": {SDKRepository, catalogueRepository},
}
// besideFor are the siblings a pull request's tests read (ADR 0484 clause 4): the catalogue at its trunk, always, even in a
// group holding a catalogue pull request; the node-engine at the commit the mesh runs (the commit of the indexer's module
// mesh-host, its trunk head when none is known); mesh-sdk, for the controller's tests, at the commit the controller's go.mod
// pins; any other sibling at the head of the group's pull request for it when the group holds one, else at its trunk. And
// first, the change's own repository at the trunk it is merged onto, so a test comparing a shrink-only list with the
// trunk's copy reads it at $MESH_CHECK_BESIDE/<name> (issue 1887 A12, ADR 0529 §2).
func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef, error) {
var heads map[string]string
if j.Trunks != nil {
heads = j.Trunks.Heads()
}
var out []BesideRef
if p.Trunk != "" {
out = append(out, BesideRef{Repository: p.Repository, Commit: p.Trunk})
}
for _, repo := range besideRules[p.Repository] {
switch {
case repo == catalogueRepository:
if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
}
case repo == nodeEngineRepo:
commit := ""
if j.Runs != nil {
commit = j.Runs.CommitOf(nodeEngineModule)
}
if commit == "" {
commit = heads[repo]
}
if commit != "" {
out = append(out, BesideRef{Repository: repo, Commit: commit})
}
case repo == SDKRepository && p.Repository == controllerRepo && j.Forge != nil:
text, err := j.Forge.File(ctx, p.Repository, "go.mod", p.Head)
if err != nil {
return nil, fmt.Errorf("go.mod of %s#%d cannot be read to place the mesh-sdk its tests read: %w", p.Repository, p.Number, err)
}
pin, ok := SDKPin(text)
if !ok {
return nil, fmt.Errorf("go.mod of %s#%d names no mesh-sdk commit, so the mesh-sdk its tests read cannot be placed", p.Repository, p.Number)
}
full, err := j.Forge.FullSHA(ctx, SDKRepository, pin)
if err != nil {
return nil, fmt.Errorf("mesh-sdk %s, which %s#%d pins, cannot be found: %w", pin, p.Repository, p.Number, err)
}
out = append(out, BesideRef{Repository: repo, Commit: full})
default:
if member := c.member(repo); member != nil {
out = append(out, BesideRef{Repository: repo, Commit: member.Head, Ref: refOf(member.Number)})
} else if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
} else if repo == controllerRepo && p.Repository == nodeEngineRepo {
// The node-engine's step-nine test judges the controller's own manifest and fails without it (issue 1295): a
// checker that cannot place it says so, where placing none left that test to skip.
return nil, fmt.Errorf("the trunk head of %s is not known, so the controller the tests of %s#%d read cannot be placed beside them", repo, p.Repository, p.Number)
}
}
}
return out, nil
}
func (c *Case) member(repository string) *Pull {
for i := range c.Pulls {
if c.Pulls[i].Repository == repository {
return &c.Pulls[i]
}
}
return nil
}
func refOf(number int) string {
if number <= 0 {
return ""
}
return "refs/pull/" + strconv.Itoa(number) + "/head"
}
@@ -0,0 +1,66 @@
package check
import (
"context"
"sort"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The states of a build the status line says (ADR 0484 clause 7).
const (
BuildBuilt = "built"
BuildFailed = "failed"
BuildWaiting = "waiting"
)
// BuildStates reads the builder's build of a module at a fingerprint of the new form (builds.<module>.<fingerprint>): the
// value and whether the key exists.
type BuildStates interface {
Build(ctx context.Context, module, fp2 string) (delivery.BuilderBuild, bool, error)
}
// NoteOf is the state of a build as the line says it: done is built, failed is failed, and one queued, building or not
// begun is waiting.
func NoteOf(b delivery.BuilderBuild, found bool) string {
switch {
case !found:
return BuildWaiting
case b.State == delivery.BuildDone:
return BuildBuilt
case b.State == delivery.BuildFailed:
return BuildFailed
}
return BuildWaiting
}
// buildNotes are the state of each build the case's modules carry a fingerprint of, by module, sorted. A module two pull
// requests give is said once.
func (j *Examiner) buildNotes(ctx context.Context, c *Case) []BuildNote {
if j.Builds == nil {
return nil
}
fps := map[string]string{}
for _, p := range c.Pulls {
for module, cm := range p.Fps {
if cm.FP2 != "" {
fps[module] = cm.FP2
}
}
}
modules := make([]string, 0, len(fps))
for m := range fps {
modules = append(modules, m)
}
sort.Strings(modules)
var notes []BuildNote
for _, m := range modules {
b, found, err := j.Builds.Build(ctx, m, fps[m])
if err != nil {
j.logf("the build of %s at %s cannot be read: %v", m, fps[m], err)
found = false
}
notes = append(notes, BuildNote{Module: m, State: NoteOf(b, found)})
}
return notes
}
@@ -0,0 +1,170 @@
package check
import (
"fmt"
"maps"
"slices"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Pull is one pull request the checker judges: its head, the trunk head of its repository it is merged onto and the
// modules the indexer says it touches, with the fingerprints the indexer wrote for them.
type Pull struct {
Repository string
Number int
Head string
Trunk string
Modules []string
Fps map[string]delivery.CandidateModule
}
// Case is one check: a candidate's pull requests that are judged (those of a repository whose trunk is not known are left
// out; hq is judged as any repository since R4g, its checks Go tests: ADR 0527), the trunk head of each repository and the key of the verdict.
type Case struct {
// Key is the verdict key, `<group>.<heads hash>`.
Key string
Group string
HeadsHash string
// Heads is the heads hash the gate matches an Approve's members by (GateHeads): a verdict carries it.
Heads string
Pulls []Pull
// Trunk is the trunk head of each repository of Pulls.
Trunk map[string]string
// Left says why pull requests of the candidate are not judged (a repository with no known trunk); empty when all are.
Left []string
// Queued is when the checker took the case up, for the time of its queue.
Queued time.Time
// Try narrows a try's check (its level, its tests); nil for a pull request's.
Try *TryOptions
notesMu sync.Mutex
notes []string
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a re-judge after a trunk move is asked on the build
// seats' recheck stream, every other check on check (ADR 0536 §4).
Kind string
}
// Progress is the step a check is at, written by the check and read when its bound passes.
type Progress struct {
mu sync.Mutex
stage, node string
}
// Set records the step now running and the node it runs on, empty for none.
func (p *Progress) Set(stage, node string) {
p.mu.Lock()
p.stage, p.node = stage, node
p.mu.Unlock()
}
// Now is the step recorded last and its node.
func (p *Progress) Now() (stage, node string) {
p.mu.Lock()
defer p.mu.Unlock()
return p.stage, p.node
}
// CaseOf is the check of a candidate against the trunk heads (the gate's source.main): the pull requests that are judged.
// The checker judges pull requests into main only (ADR 0484 clause 11): the pull message carries no base, a merge into
// another branch is a closed message, so every candidate is into the trunk; a repository whose trunk head is not known has
// none to merge onto. It answers nil when no pull request is left to judge.
func CaseOf(c delivery.Candidate, trunk map[string]string) *Case {
key, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash)
if err != nil {
return nil
}
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}}
for _, p := range c.Pulls {
switch head, known := trunk[p.Repository]; {
case !known || head == "":
cs.Left = append(cs.Left, fmt.Sprintf("%s#%d: the trunk head of %s is not known", p.Repository, p.Number, p.Repository))
default:
cs.Pulls = append(cs.Pulls, Pull{Repository: p.Repository, Number: p.Number, Head: p.Head.Head, Trunk: head, Modules: p.Modules, Fps: p.Fps})
cs.Trunk[p.Repository] = head
}
}
if len(cs.Pulls) == 0 {
return nil
}
sort.SliceStable(cs.Pulls, func(i, j int) bool {
if cs.Pulls[i].Repository != cs.Pulls[j].Repository {
return cs.Pulls[i].Repository < cs.Pulls[j].Repository
}
return cs.Pulls[i].Number < cs.Pulls[j].Number
})
return cs
}
// GateHeads is the hash the gate matches the members an Approve covers by (mesh-catalog mesh-delivery-gate onestatus.go
// headsOf): delivery.HeadsHash over every pull request of the candidate, hq's included, as
// `<owner/name lowercase>#<number>=<full head sha lowercase>`. A lone pull request is a group of one. Empty for a candidate
// with no pull request.
func GateHeads(c delivery.Candidate) string {
heads := make([]delivery.Head, 0, len(c.Pulls))
for _, p := range c.Pulls {
heads = append(heads, delivery.Head{Repository: strings.ToLower(p.Repository), Number: p.Number, Head: strings.ToLower(p.Head.Head)})
}
hash, err := delivery.HeadsHash(heads)
if err != nil {
return ""
}
return hash
}
// sameTrunk says whether two verdicts' trunk heads are the same.
func sameTrunk(a, b map[string]string) bool { return maps.Equal(a, b) }
// Trunks holds the trunk head of each repository, as the gate's source.main says (a watch puts it in).
type Trunks struct {
mu sync.Mutex
heads map[string]string
}
// Put takes a source.main value in and answers whether the heads changed.
func (t *Trunks) Put(value []byte, deleted bool) bool {
var heads map[string]string
if !deleted {
s, err := delivery.DecodeSource(value)
if err != nil {
return false
}
heads = s.Heads
}
t.mu.Lock()
defer t.mu.Unlock()
if maps.Equal(t.heads, heads) {
return false
}
t.heads = maps.Clone(heads)
return true
}
// Heads are the trunk heads now (a copy).
func (t *Trunks) Heads() map[string]string {
t.mu.Lock()
defer t.mu.Unlock()
return maps.Clone(t.heads)
}
// note puts a word of the check's way on the case, for a try's line (issue 1921).
func (c *Case) note(s string) {
c.notesMu.Lock()
defer c.notesMu.Unlock()
if !slices.Contains(c.notes, s) {
c.notes = append(c.notes, s)
}
}
// Notes are the words noted on the case, in the order they were noted.
func (c *Case) Notes() []string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return slices.Clone(c.notes)
}
@@ -0,0 +1,522 @@
package check
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// DefaultBound is how long a check may run before it is ended failed naming the step it was stuck at (ADR 0484, issue 1678):
// none since R4g. A check's asks wait in the build seats' streams as long as the backlog is (ADR 0525 §9), and each ask's
// step is bounded from its start on the node that took it (buildrun.Asker.Bound, DefaultStepBound); a bound set by
// MESH_CHECKER_BOUND still ends a check that ran past it.
const DefaultBound time.Duration = 0
// UnheardWords end the Words of a failure because a part of the mesh the check reads was not heard (the forge, issue 1624):
// like a failure of the nodes, it is not the change's, and is judged again when the nodes change.
const UnheardWords = "not heard; judged again when the nodes change"
// The events the checker says on its stream checks (module.json): a preview or a status was written, so a part that reads
// the bucket, which is made when the checker is assigned and may not exist when that part starts, finds it (ADR 0437 §8: no
// timer). The body is {"key": ...}.
const (
EventPreviewWritten = "preview.written"
EventOwedWritten = "owed.written"
)
// Announcer says an event of the checker's stream, with the key written.
type Announcer func(ctx context.Context, event, key string)
// Outcome is what judging a case gives: the verdict (state, failure, levels, times, composition; the checker adds the
// group, the heads and the trunk) and, for the line, what the check knows of the builds of the modules it touches.
type Outcome struct {
Verdict delivery.Verdict
Builds []BuildNote
// Slowest names the slowest tests of a level that went over its bound ("unit pkg.Test 1.2s").
Slowest []string
// Judged says, by pull request (pullKey), which program judged a change to a build module (ADR 0535 §5); a pull request
// that changes none is not named.
Judged map[string]string
}
// BuildNote is the state of a candidate's module build, by module: built, failed or waiting (ADR 0484 clause 7).
type BuildNote struct {
Module string
State string
}
// Judge runs the check of a case: the merge, the levels. It writes what it is doing to c.Progress. It answers an error
// only when it could not judge at all; a failure of the change is a Verdict.
type Judge interface {
Judge(ctx context.Context, c *Case) (Outcome, error)
}
// Checker judges the candidates the indexer writes, dark (ADR 0453 clause 4, ADR 0484): for each candidate it claims a
// verdict key, runs the check through its Judge, writes the verdict and, for each pull request judged, the owed status
// `mesh/check` the forge bridge performs. It holds no timer; a check starts from a candidate or from a change of the
// nodes, the trunk or the controller's plan, and ends when its judge does or its bound passes.
type Checker struct {
Verdicts KV
Owed KV
Judge Judge
Trunks interface{ Heads() map[string]string }
// Bound is how long one check may run; none when 0 (DefaultBound).
Bound time.Duration
// Link is the base of the address a status points at; the verdict key follows.
Link string
// Announce says a status was written (EventOwedWritten); nil says nothing.
Announce Announcer
Now func() time.Time
Logf func(format string, args ...any)
once sync.Once
wg sync.WaitGroup
mu sync.Mutex
known map[string]delivery.Candidate // by candidates key
inflight map[string]bool // by verdict key
again map[string]bool // by verdict key: asked again while it ran
tried map[string]bool // by verdict key: looked at since this process started
epoch uint64 // changes of the nodes seen since this process started
nodesFail map[string]uint64 // by verdict key: the epoch a failure of the nodes was written at
}
func (ch *Checker) init() {
ch.once.Do(func() {
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail = map[string]uint64{}
})
}
func (ch *Checker) logf(format string, args ...any) {
if ch.Logf != nil {
ch.Logf(format, args...)
}
}
func (ch *Checker) now() time.Time {
if ch.Now != nil {
return ch.Now()
}
return time.Now()
}
func (ch *Checker) bound() time.Duration {
if ch.Bound > 0 {
return ch.Bound
}
return DefaultBound
}
// Wait returns when every check started so far has ended.
func (ch *Checker) Wait() { ch.wg.Wait() }
// Candidate takes one value of the indexer's candidates bucket in (the key is the group) and checks it if it is to be
// checked. A value that is not a candidate is said and left.
func (ch *Checker) Candidate(ctx context.Context, key string, value []byte) {
ch.init()
c, err := delivery.DecodeCandidate(value)
if err != nil {
ch.logf("candidates.%s holds what is not a candidate, left: %v", key, err)
return
}
ch.mu.Lock()
ch.known[key] = c
ch.mu.Unlock()
ch.consider(ctx, key, c)
}
// Forget takes a candidate out (its key was deleted).
func (ch *Checker) Forget(key string) {
ch.init()
ch.mu.Lock()
delete(ch.known, key)
ch.mu.Unlock()
}
// NodesChanged is Reconsider after a change of the nodes: one came online or went, a build module became healthy or
// stopped being, or a toolchain moved. A check that failed because no node qualified is run again only after such a change.
func (ch *Checker) NodesChanged(ctx context.Context) {
ch.init()
ch.mu.Lock()
ch.epoch++
ch.mu.Unlock()
ch.Reconsider(ctx)
}
// Reconsider looks at every candidate again: the trunk moved, the nodes changed or a delivery ended. A candidate whose
// verdict stands for the trunk it was judged against is left.
func (ch *Checker) Reconsider(ctx context.Context) {
ch.init()
ch.mu.Lock()
known := make(map[string]delivery.Candidate, len(ch.known))
for k, c := range ch.known {
known[k] = c
}
ch.mu.Unlock()
for k, c := range known {
ch.consider(ctx, k, c)
}
}
func (ch *Checker) heads() map[string]string {
if ch.Trunks == nil {
return nil
}
return ch.Trunks.Heads()
}
func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidate) {
cs := CaseOf(c, ch.heads())
if cs == nil {
return // nothing to judge: no trunk known
}
cs.Queued = ch.now()
ch.mu.Lock()
if ch.inflight[cs.Key] {
ch.again[cs.Key] = true // looked at once more when the check ends, so a change of the trunk is never lost
ch.mu.Unlock()
return
}
ch.inflight[cs.Key] = true
ch.mu.Unlock()
ch.wg.Add(1)
go func() {
defer ch.wg.Done()
for {
// Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a
// failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9;
// ADR 0536 §4). The checker holds no place and no capacity of its own.
cs.Kind = ch.kindOf(ctx, cs)
ch.try(ctx, key, cs)
ch.mu.Lock()
more := ch.again[cs.Key]
delete(ch.again, cs.Key)
if !more || ctx.Err() != nil {
delete(ch.inflight, cs.Key)
ch.mu.Unlock()
return
}
ch.mu.Unlock()
if next := CaseOf(ch.candidate(key, c), ch.heads()); next != nil {
cs = next
}
cs.Queued = ch.now()
}
}()
}
// The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on
// the build seats' check stream, a head asked again only because its trunk moved on the recheck stream, pulled after.
const (
KindNew = "new"
KindNodes = "nodes"
KindTrunk = "trunk"
)
// kindOf is the kind of a case's check, read from its verdict key: none (or one left running) is a head never judged; a
// failure of the nodes is asked again after the nodes changed; a verdict on other trunk heads is a re-judge after a trunk
// move. A verdict that stands is ranked with the re-judges: its claim finds nothing to do.
func (ch *Checker) kindOf(ctx context.Context, cs *Case) string {
value, _, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return KindNew
}
v, err := delivery.DecodeVerdict(value)
switch {
case err != nil || v.State == delivery.VerdictRunning:
return KindNew
case IsNodesFailure(v):
return KindNodes
}
return KindTrunk
}
func (ch *Checker) candidate(key string, fallback delivery.Candidate) delivery.Candidate {
ch.mu.Lock()
defer ch.mu.Unlock()
if c, ok := ch.known[key]; ok {
return c
}
return fallback
}
// claimed is a verdict key the checker holds: the revision the next write is made at.
type claimed struct{ rev uint64 }
// claim takes the check of the case, or answers nil when it is not this checker's to run now:
// - no key: it is created running; a create that finds the key made meanwhile gets nothing;
// - a key left running is taken over as it stands when this process looks at it for the first time (it was left by the
// checker that stopped; there is one instance, as there is one builder per node, ADR 0478 clause 6), and left otherwise;
// - a pass or a failure for the trunk heads the case has stands; one for other trunk heads (the trunk moved) is taken
// again by compare-and-set on the same key, and so is a failure of the nodes after the nodes changed.
func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64) (*claimed, error) {
running := delivery.Verdict{Group: cs.Group, Heads: cs.Heads, Trunk: cs.Trunk, State: delivery.VerdictRunning}
data, err := delivery.EncodeVerdict(running)
if err != nil {
return nil, err
}
value, rev, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil {
return nil, fmt.Errorf("%s: %w", cs.Key, err)
}
if !found {
rev, err := ch.Verdicts.Create(ctx, cs.Key, data)
if errors.Is(err, ErrConflict) {
return nil, nil // another checker made it first
}
if err != nil {
return nil, fmt.Errorf("%s: %w", cs.Key, err)
}
return &claimed{rev}, nil
}
cur, err := delivery.DecodeVerdict(value)
if err != nil {
return nil, fmt.Errorf("%s holds what is not a verdict: %w", cs.Key, err)
}
retake := false
switch {
case cur.State == delivery.VerdictRunning:
if first {
return &claimed{rev}, nil
}
return nil, nil
case !sameTrunk(cur.Trunk, cs.Trunk):
retake = true
case cur.State == delivery.VerdictFail && IsNodesFailure(cur):
ch.mu.Lock()
at, ok := ch.nodesFail[cs.Key]
ch.mu.Unlock()
retake = !ok || epoch > at
}
if !retake {
return nil, nil
}
rev, err = ch.Verdicts.Update(ctx, cs.Key, data, rev)
if errors.Is(err, ErrConflict) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("%s: %w", cs.Key, err)
}
return &claimed{rev}, nil
}
// IsNodesFailure says whether a verdict failed because a part of the mesh the check reads was not heard (UnheardWords):
// it is judged again when the nodes change. No check fails for want of a build node any more: its asks wait in the build
// seats' streams (ADR 0525 §6, issue 1912).
func IsNodesFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, UnheardWords)
}
// try claims the check, judges it within its bound and writes the verdict and the owed statuses. A check that ends
// because ctx did is left running: the checker takes it again when it starts.
func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
ch.mu.Lock()
first := !ch.tried[cs.Key]
ch.tried[cs.Key] = true
epoch := ch.epoch
ch.mu.Unlock()
cl, err := ch.claim(ctx, cs, first, epoch)
if err != nil {
ch.logf("%s: %v", cs.Key, err)
return
}
if cl == nil {
return
}
began := ch.now()
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
out := ch.judge(ctx, cs)
if ctx.Err() != nil {
return
}
v := out.Verdict
v.Group, v.Heads, v.Trunk = cs.Group, cs.Heads, cs.Trunk
if v.Took.Queue == 0 && !cs.Queued.IsZero() && began.After(cs.Queued) {
v.Took.Queue = began.Sub(cs.Queued).Milliseconds()
}
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: the verdict cannot be written: %v", cs.Key, err)
v = delivery.Verdict{Group: cs.Group, Heads: cs.Heads, Trunk: cs.Trunk, State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Machine: "checker", Words: "the verdict could not be written: " + err.Error()}}
if data, err = delivery.EncodeVerdict(v); err != nil {
return
}
}
if _, err := ch.Verdicts.Update(ctx, cs.Key, data, cl.rev); err != nil {
ch.logf("%s: the verdict could not be written: %v", cs.Key, err)
return
}
if IsNodesFailure(v) {
ch.mu.Lock()
ch.nodesFail[cs.Key] = epoch
ch.mu.Unlock()
}
if ch.movedSince(cs) {
// Judged on a trunk that moved while it ran: the verdict is kept as judged, and the status stays pending until the
// check of the new trunk, which Reconsider starts, ends.
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending", cs.Key, v.State)
return
}
ch.writeOwed(ctx, cs, v, out)
ch.logf("%s: %s", cs.Key, v.State)
}
// judge runs the case within the bound. A judge that fails to judge, and a check that ran past its bound, are verdicts of
// failure naming the machine and the step.
func (ch *Checker) judge(ctx context.Context, cs *Case) Outcome {
out, _ := judgeWithin(ctx, ch.Judge, cs, ch.bound())
return out
}
var errBound = errors.New("the check ran past its bound")
func failure(machine, words string) delivery.Verdict {
return delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: machine, Words: words}}
}
// writeOwed writes, for each pull request judged, the status the forge bridge performs: pending while the check runs, success
// or failure at its end (ADR 0484 clause 2). A key already holding the same value is not written again, so the bridge
// performs one act for each change; an older head's key is never touched.
func (ch *Checker) writeOwed(ctx context.Context, cs *Case, v delivery.Verdict, out Outcome) {
for _, p := range cs.Pulls {
key, err := delivery.CheckerOwedKey(p.Repository, p.Number, p.Head)
if err != nil {
ch.logf("%s: %v", cs.Key, err)
continue
}
state := map[string]string{delivery.VerdictRunning: "pending", delivery.VerdictPass: "success", delivery.VerdictFail: "failure"}[v.State]
o := delivery.Owed{Act: delivery.OwedStatus, Context: delivery.StatusContextCheck, State: state,
Target: strings.TrimSuffix(ch.Link, "/") + "/" + cs.Key, Text: Line(v, cs, p, out.Builds, out.Slowest, out.Judged)}
data, err := delivery.EncodeCheckerOwed(o)
if err != nil {
ch.logf("%s: the status of %s cannot be written: %v", cs.Key, key, err)
continue
}
wrote, err := putOnce(ctx, ch.Owed, key, data, func(cur []byte) bool {
held, err := delivery.DecodeCheckerOwed(cur)
o.V = delivery.Version
return err == nil && delivery.SameOwed(held, o)
})
if err != nil {
ch.logf("%s: the status of %s cannot be written: %v", cs.Key, key, err)
} else if wrote && ch.Announce != nil {
ch.Announce(ctx, EventOwedWritten, key)
}
}
}
// putOnce writes data under key by create, or by compare-and-set when the key exists, and writes nothing when same says the
// key holds the value already.
func putOnce(ctx context.Context, kv KV, key string, data []byte, same func(cur []byte) bool) (bool, error) {
for try := 0; try < 4; try++ {
cur, rev, found, err := kv.Get(ctx, key)
if err != nil {
return false, err
}
if found && same(cur) {
return false, nil
}
if !found {
_, err = kv.Create(ctx, key, data)
} else {
_, err = kv.Update(ctx, key, data, rev)
}
if errors.Is(err, ErrConflict) {
continue
}
return err == nil, err
}
return false, fmt.Errorf("%s changed under every one of 4 tries", key)
}
// Sweep closes, at start, every verdict left running for a candidate that no longer holds those heads: nothing will check
// them again, and the key would stay running for ever (the builder's Sweep, issue 1660). Call it once the candidates have been
// read. A key of heads a candidate holds now is not touched: the check takes it over.
func (ch *Checker) Sweep(ctx context.Context) error {
ch.init()
keys, err := ch.Verdicts.Keys(ctx)
if err != nil {
return err
}
ch.mu.Lock()
current := map[string]bool{}
for _, c := range ch.known {
if k, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash); err == nil {
current[k] = true
}
}
ch.mu.Unlock()
for _, key := range keys {
if current[key] {
continue
}
value, rev, found, err := ch.Verdicts.Get(ctx, key)
if err != nil || !found {
continue
}
cur, err := delivery.DecodeVerdict(value)
if err != nil || cur.State != delivery.VerdictRunning {
continue
}
cur.State = delivery.VerdictFail
cur.Failed = delivery.FailedFirst{Machine: "checker", Words: "closed at start: left running, and no candidate holds these heads any more, so nothing will check them"}
data, err := delivery.EncodeVerdict(cur)
if err != nil {
return err
}
if _, err := ch.Verdicts.Update(ctx, key, data, rev); err != nil && !errors.Is(err, ErrConflict) {
return fmt.Errorf("%s: %w", key, err)
}
ch.logf("%s: closed, left running for heads no candidate holds", key)
}
return nil
}
// TrunkMoved is Reconsider after a move of a trunk, which first owes mesh/check pending on every open pull request of a
// candidate whose verdict was judged on other trunk heads, every member's for a group (the lead's decision L4, the checker's
// part of issue 1302): the forge's success of the old trunk never stands while the check waits for a place or for a delivery
// that holds it, so the gate never reads it as the head's.
func (ch *Checker) TrunkMoved(ctx context.Context) {
ch.init()
ch.mu.Lock()
known := make(map[string]delivery.Candidate, len(ch.known))
for k, c := range ch.known {
known[k] = c
}
ch.mu.Unlock()
heads := ch.heads()
for _, c := range known {
cs := CaseOf(c, heads)
if cs == nil {
continue
}
value, _, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
continue // never judged: no status to put out of date
}
if v, err := delivery.DecodeVerdict(value); err == nil && sameTrunk(v.Trunk, cs.Trunk) {
continue
}
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
ch.Reconsider(ctx)
}
// movedSince says whether a trunk the case was judged on is no longer the trunk's head.
func (ch *Checker) movedSince(cs *Case) bool {
now := ch.heads()
for repo, head := range cs.Trunk {
if now[repo] != head {
return true
}
}
return false
}
@@ -0,0 +1,380 @@
package check
import (
"context"
"encoding/json"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const hqRepo = "novox/hq"
const (
repoA = "novox/mesh-delivery"
repoSDK = "novox/mesh-sdk"
head1 = "1111111111111111111111111111111111111111"
head2 = "2222222222222222222222222222222222222222"
trunkA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
trunkA2 = "abababababababababababababababababababab"
trunkS = "5555555555555555555555555555555555555555"
trunkS2 = "5656565656565656565656565656565656565656"
)
// candidateOf is a candidate of a group with the given pull requests, as the indexer writes one.
func candidateOf(t testing.TB, group string, pulls ...delivery.CandidatePull) []byte {
t.Helper()
heads := make([]delivery.Head, len(pulls))
for i, p := range pulls {
heads[i] = p.Head
}
hash, err := delivery.HeadsHash(heads)
if err != nil {
t.Fatal(err)
}
c := delivery.Candidate{V: delivery.Version, Group: group, HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)}
data, err := delivery.EncodeCandidate(c)
if err != nil {
t.Fatal(err)
}
return data
}
func pullOf(repo string, number int, head string) delivery.CandidatePull {
return delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: head}, Modules: []string{}}
}
// judgeFunc is a Judge written as a function; it counts its calls.
type judgeFunc struct {
mu sync.Mutex
calls int
f func(ctx context.Context, c *Case, call int) (Outcome, error)
}
func (j *judgeFunc) Judge(ctx context.Context, c *Case) (Outcome, error) {
j.mu.Lock()
j.calls++
n := j.calls
j.mu.Unlock()
return j.f(ctx, c, n)
}
func (j *judgeFunc) count() int { j.mu.Lock(); defer j.mu.Unlock(); return j.calls }
func passing() delivery.Verdict {
return delivery.Verdict{State: delivery.VerdictPass, Levels: map[string]delivery.LevelResult{
delivery.LevelUnit: {Run: 2, Kept: 3}, delivery.LevelIntegration: {Run: 1}}}
}
func passJudge() *judgeFunc {
return &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) { return Outcome{Verdict: passing()}, nil }}
}
type rig struct {
ch *Checker
verdicts *memKV
owed *memKV
trunks *Trunks
}
func newRig(t testing.TB, j Judge) *rig {
t.Helper()
r := &rig{verdicts: newMemKV(), owed: newMemKV(), trunks: &Trunks{}}
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS})
r.ch = &Checker{Verdicts: r.verdicts, Owed: r.owed, Judge: j, Trunks: r.trunks, Link: "https://view.invalid/checks", Now: time.Now}
return r
}
func (r *rig) setTrunks(t testing.TB, heads map[string]string) bool {
t.Helper()
data, err := delivery.EncodeSource(delivery.Source{Group: "g", Heads: heads})
if err != nil {
t.Fatal(err)
}
return r.trunks.Put(data, false)
}
func (r *rig) verdict(t testing.TB, key string) delivery.Verdict {
t.Helper()
raw, _, found, _ := r.verdicts.Get(context.Background(), key)
if !found {
t.Fatalf("no verdict %s; the bucket holds %v", key, r.verdicts.vals)
}
v, err := delivery.DecodeVerdict(raw)
if err != nil {
t.Fatal(err)
}
return v
}
func keyOf(t testing.TB, group string, value []byte) string {
t.Helper()
c, err := delivery.DecodeCandidate(value)
if err != nil {
t.Fatal(err)
}
k, err := delivery.CheckerVerdictKey(group, c.HeadsHash)
if err != nil {
t.Fatal(err)
}
return k
}
func TestACandidateNotYetCheckedGetsItsVerdictKeyCreatedOnceAndIsJudged(t *testing.T) { // N6/AC1, 1678
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
key := keyOf(t, "feat/x", cand)
if v := r.verdict(t, key); v.State != delivery.VerdictPass || v.Group != "feat/x" || v.Trunk[repoA] != trunkA {
t.Fatalf("verdict = %+v", v)
}
if r.verdicts.creates[key] != 1 || j.count() != 1 {
t.Errorf("creates = %d, judged = %d, want one each", r.verdicts.creates[key], j.count())
}
// The same candidate heard again, with nothing changed, is not judged again.
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 1 {
t.Errorf("judged %d times after the same candidate again, want once", j.count())
}
}
// racyKV lets another instance create the key just before this one's create.
type racyKV struct {
*memKV
other []byte
}
func (r racyKV) Create(ctx context.Context, key string, value []byte) (uint64, error) {
if r.other != nil {
_, _ = r.memKV.Create(ctx, key, r.other)
}
return r.memKV.Create(ctx, key, value)
}
func TestASecondInstancesCreateIsRefusedAndItDoesNothing(t *testing.T) { // N6/AC1
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
key := keyOf(t, "feat/x", cand)
other, _ := delivery.EncodeVerdict(delivery.Verdict{Group: "feat/x", Heads: strings.SplitN(key, ".", 2)[1], State: delivery.VerdictRunning})
r.ch.Verdicts = racyKV{r.verdicts, other}
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 0 {
t.Errorf("the instance that lost the create judged %d times, want none", j.count())
}
if v := r.verdict(t, key); v.State != delivery.VerdictRunning {
t.Errorf("the other instance's key was changed: %+v", v)
}
}
func TestARestartWithARunningKeyJudgesItAgain(t *testing.T) { // N6/AC2
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
key := keyOf(t, "feat/x", cand)
left, _ := delivery.EncodeVerdict(delivery.Verdict{Group: "feat/x", Heads: strings.SplitN(key, ".", 2)[1], Trunk: map[string]string{repoA: trunkA}, State: delivery.VerdictRunning})
if _, err := r.verdicts.Create(ctx, key, left); err != nil {
t.Fatal(err)
}
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 1 || r.verdict(t, key).State != delivery.VerdictPass {
t.Fatalf("judged %d times, verdict %+v, want the left check judged once and passed", j.count(), r.verdict(t, key))
}
}
func TestACheckPastItsBoundFailsNamingTheStuckStep(t *testing.T) { // N6/AC3
ctx := context.Background()
j := &judgeFunc{f: func(ctx context.Context, c *Case, _ int) (Outcome, error) {
c.Progress.Set("test the unit level of mesh-delivery", "shanks")
<-ctx.Done()
return Outcome{}, ctx.Err()
}}
r := newRig(t, j)
r.ch.Bound = 30 * time.Millisecond
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if v.State != delivery.VerdictFail || v.Failed.Machine != "shanks" || !strings.Contains(v.Failed.Words, "stuck at: test the unit level of mesh-delivery") {
t.Fatalf("verdict = %+v, want a failure naming shanks and the stuck step", v)
}
}
// A check whose forge was not heard is judged again when the nodes change (issue 1912: no check fails for want of a build
// node any more; its asks wait in the streams).
func TestAnUnheardForgeFailsWithTheReasonAndIsJudgedAgainWhenTheNodesChange(t *testing.T) { // N6/AC4
ctx := context.Background()
j := &judgeFunc{f: func(_ context.Context, _ *Case, n int) (Outcome, error) {
if n == 1 {
return Outcome{Verdict: failure("the forge", "novox/mesh-delivery#7: go.mod cannot be read: no responders (the forge was "+UnheardWords+")")}, nil
}
return Outcome{Verdict: passing()}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
key := keyOf(t, "feat/x", cand)
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if v := r.verdict(t, key); !IsNodesFailure(v) {
t.Fatalf("verdict = %+v, want a failure of the nodes", v)
}
r.ch.Reconsider(ctx) // nothing changed: not judged again
r.ch.Wait()
if j.count() != 1 {
t.Fatalf("judged %d times with the nodes unchanged, want once", j.count())
}
r.ch.NodesChanged(ctx)
r.ch.Wait()
if j.count() != 2 || r.verdict(t, key).State != delivery.VerdictPass {
t.Fatalf("after the nodes changed: judged %d times, verdict %+v", j.count(), r.verdict(t, key))
}
}
func TestARepositoryWithNoTrunkGetsNoVerdict(t *testing.T) { // N6/AC6, ADR 0484 clause 11
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
r.ch.Candidate(ctx, "feat/y", candidateOf(t, "feat/y", pullOf("novox/unknown", 4, head1)))
r.ch.Wait()
if j.count() != 0 || len(r.verdicts.vals) != 0 || len(r.owed.vals) != 0 {
t.Errorf("judged %d times, %d verdicts, %d statuses, want none", j.count(), len(r.verdicts.vals), len(r.owed.vals))
}
}
// 1887/A10, ADR 0527 §2: hq is judged as any repository, its checks Go tests, and its pull request gets a verdict and a
// mesh/check; in a group, hq's member is judged with the others.
func TestAnHqPullRequestGetsAVerdictAndAStatus(t *testing.T) {
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA2})
r.ch.Candidate(ctx, "feat/hq", candidateOf(t, "feat/hq", pullOf(hqRepo, 3, head1)))
r.ch.Wait()
if j.count() != 1 || len(r.verdicts.vals) != 1 {
t.Fatalf("judged %d times, %d verdicts, want one", j.count(), len(r.verdicts.vals))
}
if s := owedState(t, r, hqRepo, 3, head1); s != "success" {
t.Errorf("hq#3 owes %q, want success", s)
}
r.ch.Candidate(ctx, "feat/x", candidateOf(t, "feat/x", pullOf(hqRepo, 4, head2), pullOf(repoA, 7, head2)))
r.ch.Wait()
if s := owedState(t, r, hqRepo, 4, head2); s != "success" {
t.Errorf("hq#4 in a group owes %q, want success", s)
}
}
func TestATrunkMoveRewritesTheSameVerdictKeyAndTheHeadsStatus(t *testing.T) { // 1300/AC7
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
key := keyOf(t, "feat/x", cand)
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
_, rev1, _, _ := r.verdicts.Get(ctx, key)
owedKey, _ := delivery.CheckerOwedKey(repoA, 7, head1)
_, orev1, _, _ := r.owed.Get(ctx, owedKey)
if !r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) {
t.Fatal("the trunks did not change")
}
r.ch.Reconsider(ctx)
r.ch.Wait()
_, rev2, _, _ := r.verdicts.Get(ctx, key)
if j.count() != 2 || rev2 <= rev1 || r.verdict(t, key).Trunk[repoA] != trunkA2 {
t.Fatalf("judged %d times, revisions %d then %d, trunk %v", j.count(), rev1, rev2, r.verdict(t, key).Trunk)
}
if r.verdicts.creates[key] != 1 {
t.Errorf("the key was created %d times, want the same key rewritten", r.verdicts.creates[key])
}
if _, orev2, _, _ := r.owed.Get(ctx, owedKey); orev2 <= orev1 {
// the line is the same when nothing differs; the status is rewritten when the words change, here the trunk is in them
t.Logf("the status keeps revision %d (its words did not change)", orev2)
}
// A new head is another key, and the older head's status is not touched.
older, _, _, _ := r.owed.Get(ctx, owedKey)
cand2 := candidateOf(t, "feat/x", pullOf(repoA, 7, head2))
r.ch.Candidate(ctx, "feat%2Fx", cand2)
r.ch.Wait()
if again, _, _, _ := r.owed.Get(ctx, owedKey); string(again) != string(older) {
t.Errorf("the older head's status was rewritten")
}
if n := len(r.owed.vals); n != 2 {
t.Errorf("%d statuses, want one for each head", n)
}
}
func TestTheStatusIsPendingWhileTheCheckRunsAndEndsSuccessOrFailure(t *testing.T) { // 1300/AC5 (unit part)
ctx := context.Background()
var seen []string
var r *rig
j := &judgeFunc{f: func(ctx context.Context, c *Case, _ int) (Outcome, error) {
key, _ := delivery.CheckerOwedKey(repoA, 7, head1)
raw, _, _, _ := r.owed.Get(ctx, key)
o, _ := delivery.DecodeCheckerOwed(raw)
seen = append(seen, o.State)
return Outcome{Verdict: delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Test: "m/a.TestX"}}}, nil
}}
r = newRig(t, j)
r.ch.Candidate(ctx, "feat/x", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
r.ch.Wait()
key, _ := delivery.CheckerOwedKey(repoA, 7, head1)
raw, _, _, _ := r.owed.Get(ctx, key)
o, err := delivery.DecodeCheckerOwed(raw)
if err != nil {
t.Fatal(err)
}
if len(seen) != 1 || seen[0] != "pending" || o.State != "failure" || o.Context != delivery.StatusContextCheck || !strings.HasPrefix(o.Target, "https://view.invalid/checks/") {
t.Errorf("while running %v, at the end %+v", seen, o)
}
}
func TestSweepClosesARunningKeyNoCandidateHoldsAndLeavesOneItDoes(t *testing.T) {
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
live := keyOf(t, "feat/x", cand)
dead, _ := delivery.CheckerVerdictKey("feat/gone", "0123456789ab")
for _, k := range []string{live, dead} {
group, heads, _ := delivery.ParseCheckerVerdictKey(k)
raw, _ := delivery.EncodeVerdict(delivery.Verdict{Group: group, Heads: heads, State: delivery.VerdictRunning})
_, _ = r.verdicts.Create(ctx, k, raw)
}
r.ch.init()
var c delivery.Candidate
_ = json.Unmarshal(cand, &c)
r.ch.known["feat%2Fx"] = c
if err := r.ch.Sweep(ctx); err != nil {
t.Fatal(err)
}
if v := r.verdict(t, dead); v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "closed at start") {
t.Errorf("the dead key = %+v", v)
}
if v := r.verdict(t, live); v.State != delivery.VerdictRunning {
t.Errorf("the live key = %+v, want it left for the check to take over", v)
}
}
func TestTheForgeBridgeIsToldOnceForEachChangeOfAStatus(t *testing.T) { // 1683 (the checker's half of AC5)
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
var told []string
r.ch.Announce = func(_ context.Context, event, key string) { told = append(told, event+" "+key) }
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
key, _ := delivery.CheckerOwedKey(repoA, 7, head1)
want := []string{EventOwedWritten + " " + key, EventOwedWritten + " " + key} // pending, then success
if len(told) != 2 || told[0] != want[0] || told[1] != want[1] {
t.Fatalf("told %v, want the status said when pending and when success", told)
}
r.ch.Candidate(ctx, "feat%2Fx", cand) // nothing changed: nothing written, nothing said
r.ch.Wait()
if len(told) != 2 {
t.Errorf("told %v after a check that changed nothing", told)
}
}
@@ -0,0 +1,80 @@
package check
import (
"context"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// fakeCompositions answers the assigner's composition of a check from memory, recording the keys asked.
type fakeCompositions struct {
v delivery.CompositionVerdict
err error
asked []string
}
func (f *fakeCompositions) Await(_ context.Context, key string) (delivery.CompositionVerdict, error) {
f.asked = append(f.asked, key)
return f.v, f.err
}
// 1297/AC3 (its unit half): a check whose tests pass takes the assigner's composition of it, by the check's own key,
// into its verdict: a pass says it passed with the mesh-sdk and machines it composed, and is no longer "not run".
func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
b := newBench(t)
f := &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "v0.1.14-x", Machines: []string{"g14"}, Took: 40}}
b.ex.Compositions = f
cs := caseOf(t, pullOf(repoA, 7, head1))
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionPass || v.Composition.SDK != "v0.1.14-x" || v.Took.Composition != 40 {
t.Fatalf("verdict = %+v, composition %+v", v, v.Composition)
}
if len(f.asked) != 1 || f.asked[0] != cs.Key {
t.Errorf("asked %v, want the check's key %s", f.asked, cs.Key)
}
if !compositionMentions(v, "passed") {
t.Errorf("the line says %q", compositionWords(v.Composition, v.Took.Composition))
}
}
func compositionMentions(v delivery.Verdict, word string) bool {
return strings.Contains(compositionWords(v.Composition, v.Took.Composition), word)
}
// 1297/AC2 at the checker: a failed composition fails the check, naming the machine it failed at and its words.
func TestAFailedCompositionFailsTheCheck(t *testing.T) {
b := newBench(t)
b.ex.Compositions = &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionFail, SDK: "x",
Machine: "g14", Words: "g14 runs web, which the change removes from its source"}}
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.State != delivery.VerdictFail || v.Failed.Machine != "g14" || v.Composition == nil || v.Composition.State != delivery.CompositionFail {
t.Fatalf("verdict = %+v", v)
}
v.Group, v.Heads = "feat/x", "0123456789ab" // the checker names them when it writes the verdict
if _, err := delivery.EncodeVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
}
// 1297/B3d: a composition that cannot be read leaves it not run; a try is not composed.
func TestACompositionNotHadIsNotRun(t *testing.T) {
b := newBench(t)
f := &fakeCompositions{err: errors.New("the check's bound passed")}
b.ex.Compositions = f
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionNotRun {
t.Fatalf("verdict = %+v", v)
}
b2 := newBench(t)
f2 := &fakeCompositions{v: delivery.CompositionVerdict{State: delivery.CompositionFail, Words: "w", SDK: "x"}}
b2.ex.Compositions = f2
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.Try = &TryOptions{}
b2.judge(t, cs)
if len(f2.asked) != 0 {
t.Errorf("a try asked for a composition: %v", f2.asked)
}
}
@@ -0,0 +1,83 @@
package check
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1930/AC3: given a check ask whose stored end keeps the digest's packages as a list of {package, digest} (issue 1930),
// or one stored before it with the packages a map keyed by package path, when the same heads are checked a second time,
// then the kept verdicts the first check wrote are read from it: every package it passed is kept, none run again.
func TestKeptVerdictsAreReadFromTheDigestsListForm(t *testing.T) {
for _, tc := range []struct {
name string
old bool
}{{"the list form", false}, {"a map stored before issue 1930", true}} {
t.Run(tc.name, func(t *testing.T) {
b := newBench(t)
var stored []string
b.asks.Answer = func(a delivery.StepAsk) delivery.Step {
s := b.answer(a)
if tc.old && s.State == delivery.StepPassed {
s.Result = oldDigestForm(t, s.Result)
}
stored = append(stored, string(s.Result))
return s
}
first := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
u := first.Levels[delivery.LevelUnit]
if first.State != delivery.VerdictPass || u.Run == 0 || u.Kept != 0 {
t.Fatalf("the first check = %+v", first)
}
if len(stored) == 0 {
t.Fatal("no ask's end was stored")
}
list := strings.Contains(stored[0], `"packages":[{"package":"fixture/slow"`)
if list == tc.old || strings.Contains(stored[0], `"fixture/slow":"`) != tc.old {
t.Fatalf("the stored end is not in the form under test: %s", stored[0])
}
b.steps.calls = nil
second := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if s := second.Levels[delivery.LevelUnit]; second.State != delivery.VerdictPass || s.Kept != u.Run || s.Run != 0 {
t.Fatalf("the second check's unit level = %+v, want %d kept and none run", s, u.Run)
}
if n := len(b.steps.tool("test")); n != 0 {
t.Errorf("%d test steps ran on the second check", n)
}
})
}
}
// oldDigestForm rewrites a check answer's level digests to the form stored before issue 1930: packages as a map keyed by
// package path.
func oldDigestForm(t *testing.T, result json.RawMessage) json.RawMessage {
t.Helper()
var a struct {
Fetch json.RawMessage `json:"fetch"`
Levels []map[string]any `json:"levels"`
}
if err := json.Unmarshal(result, &a); err != nil {
t.Fatal(err)
}
for _, l := range a.Levels {
d, ok := l["digest"].(map[string]any)
if !ok {
continue
}
m := map[string]any{}
list, _ := d["packages"].([]any)
for _, p := range list {
pd := p.(map[string]any)
m[pd["package"].(string)] = pd["digest"]
}
d["packages"] = m
}
raw, err := json.Marshal(a)
if err != nil {
t.Fatal(err)
}
return raw
}
@@ -0,0 +1,6 @@
// Package check is the checker's logic (novox/hq ADR 0484, ADR 0453 clause 4, ADR 0447, ADR 0444), apart from the bus: the
// claim of a verdict key, the judging of a pull request's head merged onto its trunk through the build modules' fetch,
// digest and test steps, the kept verdicts, the line of the status and the act the forge bridge performs. The bus is
// reached through the small interfaces here, so each part is tested with fakes. The checker runs dark: nothing reads its
// answers to merge or deliver (ADR 0453 clause 4).
package check
@@ -0,0 +1,845 @@
package check
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// captured reads a result the build module's own code marshalled (testdata/captured/README).
func captured[T any](t testing.TB, name string) T {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", "captured", name+".json"))
if err != nil {
t.Fatal(err)
}
var v T
if err := json.Unmarshal(raw, &v); err != nil {
t.Fatal(err)
}
return v
}
type stepCall struct {
module, node, tool string
args map[string]any
}
// fakeSteps stands for the build modules: a handler for each tool answers what the module's code answers.
type fakeSteps struct {
mu sync.Mutex
calls []stepCall
handlers map[string]func(c stepCall) (any, error)
}
func (f *fakeSteps) Run(_ context.Context, module, node, tool string, args map[string]any) (json.RawMessage, error) {
c := stepCall{module, node, tool, args}
f.mu.Lock()
f.calls = append(f.calls, c)
h := f.handlers[tool]
f.mu.Unlock()
if h == nil {
return nil, fmt.Errorf("the fake build module has no tool %s", tool)
}
v, err := h(c)
if err != nil {
return nil, err
}
return json.Marshal(v)
}
func (f *fakeSteps) tool(name string) []stepCall {
f.mu.Lock()
defer f.mu.Unlock()
var out []stepCall
for _, c := range f.calls {
if strings.HasSuffix(c.tool, "_"+name) {
out = append(out, c)
}
}
return out
}
type entries []delivery.NodeToolchain
func (e entries) Entries(language string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, x := range e {
if x.Language == language {
out = append(out, x)
}
}
return out
}
type langs map[string][]string
func (l langs) BuildModules(repository string) []string {
if m, ok := l[repository]; ok {
return m
}
return []string{buildrun.GoBuildModule}
}
func goEntries(nodes ...string) entries {
var e entries
for _, n := range nodes {
e = append(e, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: "go", Version: "go1.27.1", Recipe: "7"}})
e = append(e, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: "typescript", Version: "node24", Recipe: "3"}})
}
return e
}
const digC = "cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33cc33"
const digD = "dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44"
type bench struct {
ex *Examiner
asks *buildrun.FakeAsks
steps *fakeSteps
kept *memKV
prev *memKV
fetch Fetched
digest map[string]Digested // by level
test func(c stepCall) (any, error)
}
// refsOf reads the siblings of a fetch's arguments, as they arrive in an ask (JSON) or as the checker gave them.
func refsOf(v any) []BesideRef {
raw, _ := json.Marshal(v)
var refs []BesideRef
_ = json.Unmarshal(raw, &refs)
return refs
}
// toolPrefix is the start of each build module's tool names, as the fake build modules answer them.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_"}
// answer runs a check ask as the build module that pulls it does (buildstep.RunAsk), with the fake tools on shanks and the
// bench's kept verdicts, and answers the step it writes.
func (b *bench) answer(a delivery.StepAsk) delivery.Step {
module, _ := delivery.BuildModuleOf(a.Language)
const node = "shanks"
tool := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
raw, err := b.steps.Run(ctx, module, node, toolPrefix[module]+name, args)
if err != nil {
return nil, err
}
return raw, nil
}
}
steps := buildstep.AskSteps{Fetch: tool("fetch"), Test: tool("test"),
Kept: func(ctx context.Context, level, digest string) (int, bool, error) {
k, found, err := (Kept{Store: b.kept}).Lookup(ctx, level, digest)
return k.Tests, found, err
}}
if module == buildrun.GoBuildModule {
steps.Digest = tool("digest")
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
began := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed, Started: began, Ended: began.Add(2 * time.Second)}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
return step
}
step.Result, _ = json.Marshal(out)
return step
}
// newBench is an examiner over fake build modules that merge cleanly, digest the captured fixture's packages and pass
// the captured slow fixture's result at both levels, on shanks.
func newBench(t testing.TB) *bench {
b := &bench{steps: &fakeSteps{}, kept: newMemKV(), prev: newMemKV()}
b.fetch = captured[Fetched](t, "go-fetch-clean")
b.fetch.Modules, b.fetch.Changed, b.fetch.Trees = []string{"."}, []string{"README.md"}, nil
one := captured[Digested](t, "go-digest")
b.digest = map[string]Digested{
delivery.LevelUnit: {Level: "unit", Packages: map[string]string{"fixture/slow": digA}},
delivery.LevelIntegration: {Level: "integration", Packages: map[string]string{"fixture/slow": digB}},
}
_ = one
slow := captured[Tested](t, "go-test-slow")
b.test = func(c stepCall) (any, error) {
r := slow
r.Level, _ = c.args["level"].(string)
return r, nil
}
b.steps.handlers = map[string]func(stepCall) (any, error){
"build_go_fetch": func(stepCall) (any, error) { return b.fetch, nil },
"build_node_fetch": func(stepCall) (any, error) { return b.fetch, nil },
"build_go_digest": func(c stepCall) (any, error) { return echoBeside(b, b.digest[c.args["level"].(string)]), nil },
"build_go_test": func(c stepCall) (any, error) { return b.test(c) },
"build_node_test": func(c stepCall) (any, error) { return b.test(c) },
}
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
b.asks = &buildrun.FakeAsks{Answer: b.answer}
b.ex = &Examiner{Asks: buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 5 * time.Second}, Toolchains: goEntries("shanks"), Kept: Kept{Store: b.kept}, Previews: b.prev,
Modules: langs{}, Trunks: &Trunks{}, Now: func() time.Time { now = now.Add(50 * time.Millisecond); return now }}
return b
}
func caseOf(t testing.TB, pulls ...delivery.CandidatePull) *Case {
t.Helper()
raw := candidateOf(t, "feat/x", pulls...)
c, err := delivery.DecodeCandidate(raw)
if err != nil {
t.Fatal(err)
}
cs := CaseOf(c, map[string]string{repoA: trunkA, repoSDK: trunkS, "novox/mesh-controller": trunkA, "novox/mesh-catalog": trunkA})
if cs == nil {
t.Fatal("no case")
}
return cs
}
func (b *bench) judge(t testing.TB, cs *Case) Outcome {
t.Helper()
out, err := b.ex.Judge(context.Background(), cs)
if err != nil {
t.Fatal(err)
}
return out
}
func testedWith(t testing.TB, name string, edit func(*Tested)) Tested {
r := captured[Tested](t, name)
if edit != nil {
edit(&r)
}
return r
}
func packagesOf(c stepCall) []string {
var out []string
if l, ok := c.args["packages"].([]string); ok {
out = l
}
return out
}
func TestACleanMergeIsFetchedAtItsTrunkAndBothLevelsPass(t *testing.T) { // 1679/AC1
b := newBench(t)
cs := caseOf(t, pullOf(repoA, 7, head1))
out := b.judge(t, cs)
v := out.Verdict
if v.State != delivery.VerdictPass || v.Levels[delivery.LevelUnit].Run != 2 || v.Levels[delivery.LevelIntegration].Run != 2 {
t.Fatalf("verdict = %+v", v)
}
if v.Composition == nil || v.Composition.State != delivery.CompositionNotRun {
t.Errorf("composition = %+v, want not run", v.Composition)
}
if len(v.Machines) != 1 || v.Machines[0] != "shanks" {
t.Errorf("machines = %v", v.Machines)
}
fetch := b.steps.tool("fetch")
if len(fetch) != 1 || fetch[0].args["repository"] != repoA || fetch[0].args["commit"] != head1 || fetch[0].args["base"] != trunkA || fetch[0].args["ref"] != "refs/pull/7/head" {
t.Fatalf("fetch calls = %+v", fetch)
}
if v.Took.Clone <= 0 || v.Took.Unit <= 0 {
t.Errorf("took = %+v, want the clone and the unit level timed", v.Took)
}
// What passed is kept under each level's digest.
for _, k := range []string{"unit." + digA, "integration." + digB} {
if _, _, found, _ := b.kept.Get(context.Background(), k); !found {
t.Errorf("%s was not kept", k)
}
}
}
func TestOnlyThePackageWithoutAKeptVerdictRuns(t *testing.T) { // 1288/AC1
b := newBench(t)
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"fixture/slow": digA, "fixture/other": digC}}
if _, err := (Kept{Store: b.kept}).Keep(context.Background(), "unit", digC, keptValue("fixture/other", 40)); err != nil {
t.Fatal(err)
}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
tests := b.steps.tool("test")
if len(tests) != 2 || strings.Join(packagesOf(tests[0]), ",") != "fixture/slow" || tests[0].args["level"] != "unit" {
t.Fatalf("test calls = %+v, want the unit level of fixture/slow alone, then the integration level", tests)
}
if u := out.Verdict.Levels[delivery.LevelUnit]; u.Run != 2 || u.Kept != 40 {
t.Errorf("unit = %+v, want 2 run and 40 kept", u)
}
}
func TestAReAskRunsOnlyWhatFailedAndWhatChanged(t *testing.T) { // 1288/AC2, AC3
b := newBench(t)
failing := captured[Tested](t, "go-test-failing")
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"fixture/failing/bad": digA, "fixture/failing/good": digB}}
b.test = func(c stepCall) (any, error) {
if c.args["level"] == "unit" {
return failing, nil
}
return captured[Tested](t, "go-test-slow"), nil
}
cs := caseOf(t, pullOf(repoA, 7, head1))
first := b.judge(t, cs).Verdict
if first.State != delivery.VerdictFail || first.Failed.Test != "unit: fixture/failing/bad.TestThreeIsFour" {
t.Fatalf("first = %+v", first)
}
if _, _, found, _ := b.kept.Get(context.Background(), "unit."+digB); !found {
t.Fatal("the package that passed in the failing run was not kept")
}
if _, _, found, _ := b.kept.Get(context.Background(), "unit."+digA); found {
t.Fatal("the package that failed was kept")
}
// Asked again with nothing changed: only the failed package runs; the good one is taken.
b.steps.calls = nil
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
tests := b.steps.tool("test")
if len(tests) == 0 || strings.Join(packagesOf(tests[0]), ",") != "fixture/failing/bad" {
t.Fatalf("the re-ask ran %+v, want only fixture/failing/bad", tests)
}
// The trunk moved and one digest changed: that package runs, the unchanged one is taken.
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"fixture/failing/bad": digA, "fixture/failing/good": digD}}
b.steps.calls = nil
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
tests = b.steps.tool("test")
if got := strings.Join(packagesOf(tests[0]), ","); got != "fixture/failing/bad,fixture/failing/good" && got != "fixture/failing/good,fixture/failing/bad" {
t.Errorf("after a digest changed the run was %q, want the failed package and the changed one", got)
}
}
func TestANodePackageIsNeverKeptAndRunsEveryCheck(t *testing.T) { // 1288/AC4, 1292/AC1
b := newBench(t)
b.ex.Modules = langs{repoA: {buildrun.NodeBuildModule}}
b.fetch = captured[Fetched](t, "node-fetch-clean")
b.test = func(c stepCall) (any, error) {
return Tested{Level: c.args["level"].(string), Module: "modules/m", Passed: true, Elapsed: 0.4, Bound: 5}, nil
}
cs := caseOf(t, pullOf(repoA, 7, head1))
for i := 0; i < 2; i++ {
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass || v.Levels[delivery.LevelUnit].Kept != 0 || v.Levels[delivery.LevelUnit].Run != 1 {
t.Fatalf("round %d: verdict = %+v", i, v)
}
}
if n := len(b.steps.tool("test")); n != 4 {
t.Errorf("%d node test calls in two checks, want 4 (both levels each time)", n)
}
if n := len(b.steps.tool("digest")); n != 0 {
t.Errorf("the node build module was asked for a digest %d times", n)
}
if keys, _ := b.kept.Keys(context.Background()); len(keys) != 0 {
t.Errorf("kept %v for a node package", keys)
}
}
func TestTheChangesOwnFailingTestRunsFirstAndNothingElseRuns(t *testing.T) { // 1289/AC1, AC2
b := newBench(t)
b.fetch.Changed = []string{"modules/m/internal/b/b_test.go"}
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}}
failing := captured[Tested](t, "go-test-failing")
b.test = func(c stepCall) (any, error) {
if got := packagesOf(c); len(got) == 1 && got[0] == "m/internal/b" {
return failing, nil
}
return captured[Tested](t, "go-test-slow"), nil
}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if out.Verdict.State != delivery.VerdictFail || !strings.Contains(out.Verdict.Failed.Test, "TestThreeIsFour") {
t.Fatalf("verdict = %+v", out.Verdict)
}
tests := b.steps.tool("test")
if len(tests) != 1 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" {
t.Fatalf("test calls = %+v, want only m/internal/b, once", tests)
}
if n := len(b.steps.tool("digest")); n != 1 {
t.Errorf("%d digest calls, want the unit level's alone (the check stopped)", n)
}
if out.Verdict.Composition == nil || out.Verdict.Composition.State != delivery.CompositionNotRun {
t.Errorf("composition = %+v", out.Verdict.Composition)
}
}
func TestTheChangesOwnPassingTestRunsFirstAndTheRestFollow(t *testing.T) { // 1289/AC3
b := newBench(t)
b.fetch.Changed = []string{"modules/m/internal/b/b_test.go", "modules/m/internal/a/a.go"}
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if out.Verdict.State != delivery.VerdictPass {
t.Fatalf("verdict = %+v", out.Verdict)
}
var unit []string
for _, c := range b.steps.tool("test") {
if c.args["level"] == "unit" {
unit = append(unit, strings.Join(packagesOf(c), ","))
}
}
if strings.Join(unit, " | ") != "m/internal/b | m/internal/a" {
t.Errorf("the unit level ran %v, want the changed test's package first, then the rest", unit)
}
}
func TestAHeadThatDoesNotMergeEndsFailedNamingTheFilesAndRunsNoTest(t *testing.T) { // 1289/AC4
b := newBench(t)
b.fetch = captured[Fetched](t, "go-fetch-conflict")
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
f := out.Verdict.Failed
if out.Verdict.State != delivery.VerdictFail || f.Machine != "shanks" || !strings.Contains(f.Words, "files in conflict: a.go, sub/b.go") {
t.Fatalf("verdict = %+v", out.Verdict)
}
if n := len(b.steps.tool("digest")) + len(b.steps.tool("test")); n != 0 {
t.Errorf("%d digest and test steps ran after a conflict", n)
}
}
func TestALevelWhoseDigestIsNotNamedRunsWholeAndKeepsNothing(t *testing.T) { // 1287/AC4
b := newBench(t)
b.steps.handlers["build_go_digest"] = func(stepCall) (any, error) { return nil, fmt.Errorf("go list failed") }
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if out.Verdict.State != delivery.VerdictPass {
t.Fatalf("verdict = %+v", out.Verdict)
}
tests := b.steps.tool("test")
if len(tests) != 2 || tests[0].args["packages"] != nil {
t.Errorf("test calls = %+v, want each level once, whole", tests)
}
if keys, _ := b.kept.Keys(context.Background()); len(keys) != 0 {
t.Errorf("kept %v with no digest", keys)
}
}
func TestALevelOverItsBoundIsNamedWithItsSlowestTestsAndKeepsItsVerdict(t *testing.T) { // 1300/AC3
b := newBench(t)
b.test = func(c stepCall) (any, error) {
return testedWith(t, "go-test-slow", func(r *Tested) {
r.Level = c.args["level"].(string)
r.OverBound = true
r.OverWhich = "run"
r.Elapsed = 7
}), nil
}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
v := out.Verdict
if v.State != delivery.VerdictPass || strings.Join(v.OverBound, ",") != "unit,integration" {
t.Fatalf("verdict = %+v", v)
}
if len(out.Slowest) == 0 || !strings.Contains(out.Slowest[0], "TestTakesAWhile") {
t.Errorf("slowest = %v", out.Slowest)
}
}
func TestSkippedTestsAreCountedPerLevel(t *testing.T) { // 1300/AC6, 1251
b := newBench(t)
b.test = func(c stepCall) (any, error) {
return testedWith(t, "go-test-slow", func(r *Tested) { r.Level = c.args["level"].(string); r.Skipped = 3 }), nil
}
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.Levels[delivery.LevelUnit].Skipped != 3 || v.Levels[delivery.LevelIntegration].Skipped != 3 {
t.Errorf("levels = %+v", v.Levels)
}
}
func TestAGroupIsOneVerdictAndEachMembersTestsReadTheOthersAtTheirHeads(t *testing.T) { // 1624/AC1
b := newBench(t)
cs := caseOf(t, pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2))
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass {
t.Fatalf("verdict = %+v", v)
}
fetch := b.steps.tool("fetch")
if len(fetch) != 2 {
t.Fatalf("%d fetches, want one for each member", len(fetch))
}
var forA []BesideRef
for _, f := range fetch {
if f.args["repository"] == repoA {
forA, _ = refsOf(f.args["beside"]), true
}
}
found := false
for _, r := range forA {
if r.Repository == repoSDK && r.Commit == head2 && r.Ref == "refs/pull/12/head" {
found = true
}
}
if !found {
t.Errorf("mesh-delivery's tests were not given the SDK member at its head: %+v", forA)
}
}
func TestAnExaminerIsAJudgeOfTheChecker(t *testing.T) {
var _ Judge = (*Examiner)(nil)
}
const (
manA = "0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a"
)
func TestACleanMergeWritesItsPreviewWithTheTreesManifestsAndLocks(t *testing.T) { // 1445/AC1
b := newBench(t)
b.fetch.Trees = map[string]MergeTree{
"modules/avahi": {Tree: "7777777777777777777777777777777777777777", Manifest: manA, Locks: map[string]string{"go.sum": "ff00"}},
"modules/bare": {Tree: "8888888888888888888888888888888888888888"}, // no manifest digest: left out
}
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
key, _ := delivery.PreviewKey(repoA, head1, trunkA)
raw, _, found, _ := b.prev.Get(context.Background(), key)
if !found {
t.Fatalf("no preview %s; the bucket holds %v", key, b.prev.vals)
}
pv, err := delivery.DecodePreview(raw)
if err != nil {
t.Fatal(err)
}
m := pv.Modules["modules/avahi"]
if pv.Merge != b.fetch.Merge || pv.Trunk != trunkA || pv.Head != head1 || len(pv.Modules) != 1 || m.Tree != "7777777777777777777777777777777777777777" || m.Manifest != manA || m.Locks["go.sum"] != "ff00" {
t.Errorf("preview = %+v", pv)
}
}
func TestAHeadThatDoesNotMergeWritesAPreviewOfItsConflictsAlone(t *testing.T) { // 1445/AC5
b := newBench(t)
b.fetch = captured[Fetched](t, "go-fetch-conflict")
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
key, _ := delivery.PreviewKey(repoA, head1, trunkA)
raw, _, found, _ := b.prev.Get(context.Background(), key)
if !found {
t.Fatalf("no preview for a head that does not merge; the bucket holds %v", b.prev.vals)
}
pv, err := delivery.DecodePreview(raw)
if err != nil || strings.Join(pv.Conflicts, ",") != "a.go,sub/b.go" || pv.Merge != "" || len(pv.Modules) != 0 {
t.Errorf("preview = %+v, %v, want the files in conflict and no merge", pv, err)
}
}
func TestAPreviewIsWrittenOnceAndAnExistingKeyStands(t *testing.T) {
b := newBench(t)
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
b.fetch.Merge = "5656565656565656565656565656565656565656"
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
key, _ := delivery.PreviewKey(repoA, head1, trunkA)
if b.prev.creates[key] != 1 {
t.Errorf("the preview was created %d times, want once", b.prev.creates[key])
}
raw, _, _, _ := b.prev.Get(context.Background(), key)
if pv, _ := delivery.DecodePreview(raw); pv.Merge == b.fetch.Merge {
t.Error("the second merge commit replaced the first")
}
}
func TestTheIndexerIsToldOfAPreviewOnceWhenItIsCreated(t *testing.T) { // 1445/AC4 (the checker's half)
b := newBench(t)
var told []string
b.ex.Announce = func(_ context.Context, event, key string) { told = append(told, event+" "+key) }
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
key, _ := delivery.PreviewKey(repoA, head1, trunkA)
if len(told) != 1 || told[0] != EventPreviewWritten+" "+key {
t.Errorf("told %v, want once of %s", told, key)
}
}
type fakeBuilds map[string]delivery.BuilderBuild
func (f fakeBuilds) Build(_ context.Context, module, fp2 string) (delivery.BuilderBuild, bool, error) {
b, ok := f[module+" "+fp2]
return b, ok, nil
}
func TestTheOutcomeSaysIfEachCandidatesModuleIsBuiltFailedOrWaiting(t *testing.T) { // N7/AC1 (the checker reads the builder's keys)
b := newBench(t)
fpA := "src2:" + strings.Repeat("a", 64)
fpB := "src2:" + strings.Repeat("b", 64)
fpC := "src2:" + strings.Repeat("c", 64)
b.ex.Builds = fakeBuilds{
"avahi " + fpA: {State: delivery.BuildDone},
"mesh-tools " + fpB: {State: delivery.BuildFailed, Error: "boom"},
// netcheck has no key yet
}
p := pullOf(repoA, 7, head1)
p.Modules = []string{"avahi", "mesh-tools", "netcheck", "nofp"}
p.Fps = map[string]delivery.CandidateModule{"avahi": {FP2: fpA}, "mesh-tools": {FP2: fpB}, "netcheck": {FP2: fpC}, "nofp": {None2: "toolchain not yet reported"}}
out := b.judge(t, caseOf(t, p))
var got []string
for _, n := range out.Builds {
got = append(got, n.Module+" "+n.State)
}
if strings.Join(got, ", ") != "avahi built, mesh-tools failed, netcheck waiting" {
t.Errorf("builds = %v, want one note for each module that has a fingerprint, sorted", got)
}
}
// trunksOf are trunk heads of every repository, as the gate's source names them.
func trunksOf(t *testing.T, heads map[string]string) *Trunks {
t.Helper()
data, err := delivery.EncodeSource(delivery.Source{Group: "g", Heads: heads})
if err != nil {
t.Fatal(err)
}
tr := &Trunks{}
tr.Put(data, false)
return tr
}
func TestNothingIsKeptForAWorkspaceGivenSiblingsWhenTheDigestDoesNotSayItCoversThem(t *testing.T) { // review D1, ADR 0444, issue 1287
b := newBench(t)
// A build module whose digest does not echo the siblings (before DigestVersion 3).
b.steps.handlers["build_go_digest"] = func(c stepCall) (any, error) { return b.digest[c.args["level"].(string)], nil }
b.ex.Trunks = trunksOf(t, map[string]string{repoSDK: trunkS, "novox/mesh-catalog": trunkA})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if got := len(b.steps.tool("fetch")); got != 1 || b.steps.tool("fetch")[0].args["beside"] == nil {
t.Fatalf("the fetch was not given siblings: %+v", b.steps.tool("fetch"))
}
if keys, _ := b.kept.Keys(context.Background()); len(keys) != 0 {
t.Errorf("kept %v for a package whose digest does not say it covers the siblings", keys)
}
// And a kept verdict of the same digest from elsewhere is not taken for it: everything runs.
if _, err := (Kept{Store: b.kept}).Keep(context.Background(), "unit", digA, keptValue("fixture/slow", 2)); err != nil {
t.Fatal(err)
}
b.steps.calls = nil
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if n := len(b.steps.tool("test")); n != 2 {
t.Errorf("%d test steps, want both levels run whole", n)
}
}
// besideDigest stands for the Go build module's digest over a workspace with siblings (DigestVersion 3, lead decision L5):
// each package's digest folds the siblings' commits in, and the answer echoes the lines it folded.
// echoBeside is d with the siblings the fetches placed, as the Go build module's digest echoes them (its DigestResult's
// Beside): every check now places the change's own repository at its trunk (issue 1887 A12).
func echoBeside(b *bench, d Digested) Digested {
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
}
sort.Strings(lines)
d.Beside = lines
return d
}
func besideDigest(b *bench) func(c stepCall) (any, error) {
return func(c stepCall) (any, error) {
level := c.args["level"].(string)
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
}
sort.Strings(lines)
sum := sha256.Sum256([]byte(level + "\n" + strings.Join(lines, "\n")))
return Digested{Level: level, Packages: map[string]string{"fixture/slow": hex.EncodeToString(sum[:])}, Beside: lines}, nil
}
}
func TestACoreRepositorysPassingPackageIsKeptUnderADigestCoveringItsSiblings(t *testing.T) { // 1287 AC1-AC3, AC7 with L5
b := newBench(t)
b.steps.handlers["build_go_digest"] = besideDigest(b)
b.ex.Trunks = trunksOf(t, map[string]string{repoSDK: trunkS, "novox/mesh-catalog": trunkA})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
keys, _ := b.kept.Keys(context.Background())
if len(keys) != 2 {
t.Fatalf("kept %v, want the passing package at both levels", keys)
}
// The same siblings again: everything is taken, nothing runs.
b.steps.calls = nil
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if n := len(b.steps.tool("test")); n != 0 || v.Levels[delivery.LevelUnit].Run != 0 || v.Levels[delivery.LevelUnit].Kept == 0 {
t.Fatalf("%d test steps, unit %+v, want none run and the kept verdict taken", n, v.Levels[delivery.LevelUnit])
}
// A sibling's commit moves: a new digest, so the package runs again and is kept under it.
b.ex.Trunks = trunksOf(t, map[string]string{repoSDK: trunkS2, "novox/mesh-catalog": trunkA})
b.steps.calls = nil
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if n := len(b.steps.tool("test")); n != 2 {
t.Errorf("%d test steps after a sibling moved, want both levels run", n)
}
if keys, _ := b.kept.Keys(context.Background()); len(keys) != 4 {
t.Errorf("kept %v, want a second key per level under the new digest", keys)
}
}
func TestADigestThatEchoesOtherSiblingsKeepsNothing(t *testing.T) { // 1287 with L5: an echo that differs is not trusted
b := newBench(t)
inner := besideDigest(b)
b.steps.handlers["build_go_digest"] = func(c stepCall) (any, error) {
v, err := inner(c)
d := v.(Digested)
d.Beside = d.Beside[:len(d.Beside)-1]
return d, err
}
b.ex.Trunks = trunksOf(t, map[string]string{repoSDK: trunkS, "novox/mesh-catalog": trunkA})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if keys, _ := b.kept.Keys(context.Background()); len(keys) != 0 {
t.Errorf("kept %v under a digest that does not cover every sibling", keys)
}
}
type pkgPaths map[string][]string
func (p pkgPaths) PackagePaths(repository string) []string { return p[repository] }
// groupPackageBench is a group of a TypeScript member (mesh-delivery) and the mesh-sdk member, whose merge changes sdkChanged.
func groupPackageBench(t *testing.T, sdkChanged []string) *bench {
b := newBench(t)
b.ex.Modules = langs{repoA: {buildrun.GoBuildModule, buildrun.NodeBuildModule}, repoSDK: {buildrun.GoBuildModule, buildrun.NodeBuildModule}}
b.ex.Packages = pkgPaths{repoSDK: {"src", "test", "package.json", "package-lock.json", "tsconfig.json"}}
fetch := func(c stepCall) (any, error) {
f := b.fetch
if c.args["repository"] == repoSDK {
f.Changed = sdkChanged
}
return f, nil
}
b.steps.handlers["build_go_fetch"], b.steps.handlers["build_node_fetch"] = fetch, fetch
return b
}
func nodeFetchOf(b *bench, repo string) stepCall {
for _, c := range b.steps.tool("fetch") {
if c.module == buildrun.NodeBuildModule && c.args["repository"] == repo {
return c
}
}
return stepCall{}
}
func TestATypeScriptMemberTakesTheGroupsPackageWhenTheSDKMemberChangesItsPaths(t *testing.T) { // ADR 0500 §5, C11b
b := groupPackageBench(t, []string{"go/delivery/x.go", "src/state.ts"})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2)))
want := map[string]any{"repository": repoSDK, "commit": head2, "ref": "refs/pull/12/head"}
if got := nodeFetchOf(b, repoA).args["package"]; !reflect.DeepEqual(got, want) {
t.Errorf("mesh-delivery's Node fetch was given the package %v, want %v", got, want)
}
if got := nodeFetchOf(b, repoSDK).args["package"]; got != nil {
t.Errorf("the mesh-sdk member's own fetch was given %v: it builds its package itself", got)
}
}
func TestATypeScriptMemberResolvesTheRegistryWhenTheSDKMemberLeavesItsPaths(t *testing.T) { // ADR 0500 §5: the manifest alone, or Go alone
for _, changed := range [][]string{{"go/delivery/x.go"}, {"module.json"}, {"srcx/a.ts"}} {
b := groupPackageBench(t, changed)
b.judge(t, caseOf(t, pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2)))
if got := nodeFetchOf(b, repoA).args["package"]; got != nil {
t.Errorf("%v: the Node fetch was given %v, want the registry", changed, got)
}
}
// A lone TypeScript pull request: no group package either.
b := groupPackageBench(t, []string{"src/state.ts"})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if got := nodeFetchOf(b, repoA).args["package"]; got != nil {
t.Errorf("a lone pull request's Node fetch was given %v", got)
}
}
func TestNoStepOfACheckCarriesPublish(t *testing.T) { // ADR 0500 §4, 1309 (the checker's half): only the builder's trunk build publishes
b := groupPackageBench(t, []string{"src/state.ts"})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2)))
if len(b.steps.calls) == 0 {
t.Fatal("no step ran")
}
for _, c := range b.steps.calls {
if _, ok := c.args["publish"]; ok {
t.Errorf("%s was given publish: %v", c.tool, c.args)
}
if strings.HasSuffix(c.tool, "_build") || strings.HasSuffix(c.tool, "_pack") {
t.Errorf("the checker asked %s: a check builds no artifact of its own", c.tool)
}
}
}
// 1890/AC1, 1887/A5a: an ask whose step never ends (the build node stopped with it running, and nothing redelivered it in
// time) ends the check loudly at the asker's bound; the checker holds no place to free.
func TestAnAskWhoseStepNeverEndsEndsTheCheckLoudly(t *testing.T) {
b := newBench(t)
// A node took the ask and its step stays running: the bound runs from its start (review round 2 item 1).
b.asks.Answer = func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Tool: "check ask", Node: "shanks", State: delivery.StepRunning, Started: time.Now()}
}
b.ex.Asks = buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 20 * time.Millisecond}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if f := out.Verdict.Failed; out.Verdict.State != delivery.VerdictFail || !strings.Contains(f.Words, "on shanks did not end within") {
t.Fatalf("verdict = %+v", out.Verdict)
}
if n := len(b.asks.Asks()); n != 1 || b.asks.Asks()[0].Kind != delivery.AskCheck || b.asks.Asks()[0].Check == nil {
t.Fatalf("asks = %+v, want one check ask", b.asks.Asks())
}
}
// 1887/A5a, ADR 0536 §4: a check is one ask per pull request and build module on the check stream, a re-judge after a
// trunk move one on the recheck stream, each carrying both levels with the kept verdicts taken by the build module.
func TestACheckIsOneAskAndAReJudgeGoesOnTheRecheckStream(t *testing.T) {
b := newBench(t)
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
asks := b.asks.Asks()
if len(asks) != 1 || asks[0].Kind != delivery.AskCheck || asks[0].Language != delivery.AskGo || asks[0].Asker != delivery.CheckerModule {
t.Fatalf("asks = %+v, want one Go check ask", asks)
}
if j := asks[0].Check; j == nil || !j.Kept || strings.Join(j.Levels, ",") != "unit,integration" || j.Fetch["repository"] != repoA {
t.Fatalf("the job = %+v", asks[0].Check)
}
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.Kind = KindTrunk
b.judge(t, cs)
if asks := b.asks.Asks(); len(asks) != 2 || asks[1].Kind != delivery.AskRecheck {
t.Fatalf("asks = %+v, want the re-judge on recheck", asks)
}
}
// Review round 2 item 1: a check ask that waits in its stream longer than the step bound (no node has a free slot during a
// large rebuild) is not failed: the check goes on when a node takes it, and no second ask is put.
func TestACheckAskWaitingInItsStreamIsNotBounded(t *testing.T) {
b := newBench(t)
answer := b.asks.Answer
b.asks.Answer = nil
b.ex.Asks = buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 20 * time.Millisecond}
done := make(chan Outcome, 1)
go func() { done <- b.judge(t, caseOf(t, pullOf(repoA, 7, head1))) }()
for deadline := time.Now().Add(2 * time.Second); len(b.asks.Asks()) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
}
time.Sleep(100 * time.Millisecond) // five bounds in the stream
select {
case out := <-done:
t.Fatalf("the check ended while its ask waited: %+v", out.Verdict)
default:
}
b.asks.End(answer(b.asks.Asks()[0])) // a node took it and ran it
if out := <-done; out.Verdict.State != delivery.VerdictPass || len(b.asks.Asks()) != 1 {
t.Fatalf("verdict %+v after %d asks, want a pass from the one ask", out.Verdict, len(b.asks.Asks()))
}
}
// 1917: the digest is taken inside the check ask, as the build module's own digest step answers it (no step key is asked
// of it), so a second check of the same heads keeps every package the first one passed: "N kept", nothing run again.
func TestASecondCheckOfTheSameHeadsKeepsWhatTheFirstPassed(t *testing.T) {
b := newBench(t)
first := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
u := first.Levels[delivery.LevelUnit]
if first.State != delivery.VerdictPass || u.Run == 0 || u.Kept != 0 {
t.Fatalf("the first check = %+v", first)
}
b.steps.calls = nil
second := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if s := second.Levels[delivery.LevelUnit]; second.State != delivery.VerdictPass || s.Kept != u.Run || s.Run != 0 {
t.Fatalf("the second check's unit level = %+v, want %d kept and none run", s, u.Run)
}
if n := len(b.steps.tool("test")); n != 0 {
t.Errorf("%d test steps ran on the second check", n)
}
}
@@ -0,0 +1,73 @@
package check
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The failure the build module reported for group r4g on 2026-10-23: the test logged a line, then failed with a multi-line
// t.Fatalf. Its output is what the build module now sends (its last lines, with the error line as Cause).
func fatalAfterLog() Failure {
return Failure{
Package: "github.com/novox/mesh-controller/internal/broker", Test: "TestTheCataloguesDerivedSubjectsMeet",
Cause: "3 subscription(s) derive to a subject no emitter owns:",
Output: " agreement_catalogue_test.go:130: judged against /var/lib/mesh-build-go-work/id/beside/mesh-catalog as cloned\n" +
" agreement_catalogue_test.go:186: 3 subscription(s) derive to a subject no emitter owns:\n" +
" a.b.c\n d.e.f\n g.h.i\n--- FAIL: TestTheCataloguesDerivedSubjectsMeet (0.00s)",
}
}
func TestAFailedTestsVerdictKeepsItsWholeFailureMessage(t *testing.T) { // 1901/AC1
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{fatalAfterLog()}})
for _, want := range []string{"3 subscription(s) derive to a subject no emitter owns:", "a.b.c", "d.e.f", "g.h.i"} {
if !strings.Contains(v.Failed.Words, want) {
t.Errorf("words lack %q: %q", want, v.Failed.Words)
}
}
if len(v.Failed.Words) > MaxWords+200 {
t.Errorf("words are %d bytes, bound %d", len(v.Failed.Words), MaxWords)
}
}
func TestTheStatusLineNamesTheFailureNotTheFirstLoggedLine(t *testing.T) { // 1901/AC2
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{fatalAfterLog()}})
got := firstFailed(*v)
if !strings.Contains(got, "3 subscription(s) derive to a subject no emitter owns:") || strings.Contains(got, "judged against") || strings.Contains(got, "\n") {
t.Errorf("firstFailed = %q", got)
}
// With no cause told apart (an older build module), the words still start at the first line, as before.
f := fatalAfterLog()
f.Cause = ""
v = levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if !strings.HasPrefix(v.Failed.Words, "agreement_catalogue_test.go:130") {
t.Errorf("words = %q", v.Failed.Words)
}
}
func TestWordsLongerThanTheBoundKeepTheirLastLines(t *testing.T) { // 1901/AC1
f := Failure{Output: strings.Repeat("log line of some length\n", 500) + "the cause is here"}
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if !strings.HasSuffix(v.Failed.Words, "the cause is here") || len(v.Failed.Words) > MaxWords+200 {
t.Errorf("%d bytes, ends %q", len(v.Failed.Words), v.Failed.Words[len(v.Failed.Words)-30:])
}
_ = delivery.VerdictFail
}
// An older build module sends no "cause": its answer is still accepted and the verdict's words are its output, as before.
func TestAnOlderBuildModulesFailureIsAcceptedWithItsWholeOutput(t *testing.T) { // 1901/AC3
old := `{"package":"p","test":"TestX","output":"x_test.go:1: logged\nx_test.go:2: the old kept lines"}`
var f Failure
if err := json.Unmarshal([]byte(old), &f); err != nil || f.Cause != "" {
t.Fatalf("1901/AC3: err = %v, cause = %q", err, f.Cause)
}
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if v.Failed.Words != "x_test.go:1: logged\nx_test.go:2: the old kept lines" {
t.Errorf("1901/AC3: words = %q", v.Failed.Words)
}
if got := firstFailed(*v); !strings.Contains(got, "logged") || strings.Contains(got, "\n") {
t.Errorf("1901/AC3: status = %q", got)
}
}
@@ -0,0 +1,62 @@
package check
import (
"context"
"sort"
"sync"
)
// memKV is a bucket in memory with the revisions a real one gives.
type memKV struct {
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
next uint64
// creates counts the creates that succeeded, by key.
creates map[string]int
}
func newMemKV() *memKV {
return &memKV{vals: map[string][]byte{}, revs: map[string]uint64{}, creates: map[string]int{}}
}
func (m *memKV) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
m.mu.Lock()
defer m.mu.Unlock()
v, ok := m.vals[key]
return v, m.revs[key], ok, nil
}
func (m *memKV) Create(_ context.Context, key string, value []byte) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if _, ok := m.vals[key]; ok {
return 0, ErrConflict
}
m.next++
m.vals[key], m.revs[key] = append([]byte(nil), value...), m.next
m.creates[key]++
return m.next, nil
}
func (m *memKV) Update(_ context.Context, key string, value []byte, revision uint64) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if _, ok := m.vals[key]; !ok || m.revs[key] != revision {
return 0, ErrConflict
}
m.next++
m.vals[key], m.revs[key] = append([]byte(nil), value...), m.next
return m.next, nil
}
func (m *memKV) Keys(context.Context) ([]string, error) {
m.mu.Lock()
defer m.mu.Unlock()
keys := make([]string, 0, len(m.vals))
for k := range m.vals {
keys = append(keys, k)
}
sort.Strings(keys)
return keys, nil
}
@@ -0,0 +1,116 @@
package check
import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// What the gate reads of the checker (R4e, lead's interface with dev-r4e-gate): the verdict's heads are the gate's hash over
// `<owner/name lowercase>#<number>=<full head sha>`, a lone pull request is a group of one, and a move of a trunk marks
// mesh/check pending on every open pull request into it, every member's for a group, before any check starts.
func owedState(t testing.TB, r *rig, repo string, number int, head string) string {
t.Helper()
key, err := delivery.CheckerOwedKey(repo, number, head)
if err != nil {
t.Fatal(err)
}
raw, _, found, _ := r.owed.Get(context.Background(), key)
if !found {
return ""
}
o, err := delivery.DecodeCheckerOwed(raw)
if err != nil {
t.Fatal(err)
}
return o.State
}
func TestTheVerdictsHeadsAreTheGatesHashOverLowercaseHeads(t *testing.T) { // gate (b), (c)
ctx := context.Background()
r := newRig(t, passJudge())
upper := strings.ToUpper(head1[:20]) + head1[20:]
r.setTrunks(t, map[string]string{"Novox/Mesh-Delivery": trunkA, repoA: trunkA, repoSDK: trunkS})
cand := candidateOf(t, "feat/x", pullOf("Novox/Mesh-Delivery", 7, upper))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
want, _ := delivery.HeadsHash([]delivery.Head{{Repository: repoA, Number: 7, Head: head1}})
if v := r.verdict(t, keyOf(t, "feat/x", cand)); v.Heads != want {
t.Errorf("heads = %q, want the gate's hash of the lone pull request, a group of one, %q", v.Heads, want)
}
}
func TestAGroupsVerdictHeadsCoverEveryMember(t *testing.T) { // gate (b): hq's member too, as the gate's ask covers it
ctx := context.Background()
r := newRig(t, passJudge())
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2), pullOf(hqRepo, 3, head2))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
want, _ := delivery.HeadsHash([]delivery.Head{{Repository: repoA, Number: 7, Head: head1}, {Repository: repoSDK, Number: 12, Head: head2},
{Repository: hqRepo, Number: 3, Head: head2}})
if v := r.verdict(t, keyOf(t, "feat/x", cand)); v.Heads != want {
t.Errorf("heads = %q, want %q", v.Heads, want)
}
}
func TestATrunkMoveMarksEveryOpenPullRequestIntoItPendingAtOnce(t *testing.T) { // gate (a), lead decision L4 (1302's checker part)
ctx := context.Background()
r := newRig(t, passJudge())
r.ch.Candidate(ctx, "feat%2Fg", candidateOf(t, "feat/g", pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2)))
r.ch.Candidate(ctx, "feat%2Fs", candidateOf(t, "feat/s", pullOf(repoSDK, 13, head1)))
r.ch.Wait()
for _, p := range []struct {
repo string
n int
head string
}{{repoA, 7, head1}, {repoSDK, 12, head2}, {repoSDK, 13, head1}} {
if s := owedState(t, r, p.repo, p.n, p.head); s != "success" {
t.Fatalf("%s#%d is %q before the move", p.repo, p.n, s)
}
}
// mesh-delivery's trunk moves; the re-judges it asks are held in their judge (in the stream, since R4g) meanwhile.
release := make(chan struct{})
r.ch.Judge = &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) { <-release; return Outcome{Verdict: passing()}, nil }}
defer func() { close(release); r.ch.Wait() }()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("mesh-delivery#7, into the moved trunk, is %q, want pending", s)
}
if s := owedState(t, r, repoSDK, 12, head2); s != "pending" {
t.Errorf("mesh-sdk#12, a member of the same group, is %q, want pending", s)
}
if s := owedState(t, r, repoSDK, 13, head1); s != "success" {
t.Errorf("mesh-sdk#13, alone into a trunk that did not move, is %q, want its success to stand", s)
}
}
func TestACheckThatEndsAfterItsTrunkMovedOwesPendingNotItsVerdict(t *testing.T) { // gate (a): a pass on an old trunk is never shown
ctx := context.Background()
var r *rig
release, judged := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
defer close(judged)
} else {
<-release // the check asked again for the new trunk waits in the stream meanwhile
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
<-judged
for deadline := time.Now().Add(2 * time.Second); owedState(t, r, repoA, 7, head1) == "" && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
time.Sleep(20 * time.Millisecond)
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status is %q, want pending: the pass was judged on a trunk that moved", s)
}
close(release)
r.ch.Wait()
}
@@ -0,0 +1,21 @@
package check
import (
"errors"
"github.com/nats-io/nats.go"
)
// Unheard is the error of a call that reached nobody: the bus had no responder for it, or none answered within the call's
// bound. The program wraps the errors of its forge calls in it where the bus gives a timeout; a no-responders error is
// known as it is. A build step is no call since R4g: an ask whose node stops is delivered again by its stream (ADR 0525 §6).
type Unheard struct{ Err error }
func (u *Unheard) Error() string { return u.Err.Error() }
func (u *Unheard) Unwrap() error { return u.Err }
// isUnheard says whether err is a call that reached nobody. A step that began and failed, or ended failed, is not.
func isUnheard(err error) bool {
var u *Unheard
return errors.As(err, &u) || errors.Is(err, nats.ErrNoResponders)
}
@@ -0,0 +1,21 @@
package check
import (
"fmt"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Issue 1820 AC5: a read of the forge that nobody answers is the forge's failure, not a build node's. (AC1-AC4 moved a
// step to another node; since R4g an ask whose node stops is delivered again by its stream: ADR 0525 §6.)
func TestAPinReadThatNobodyAnswersIsTheForgesFailureNotABuildNodes(t *testing.T) { // 1820/AC5
b := groupBench(t, "cb54756c3865")
b.ex.Forge.(*fakeForge).err = fmt.Errorf("gitea.gitea_get_file: %w", nats.ErrNoResponders)
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2))).Verdict
if v.State != delivery.VerdictFail || v.Failed.Machine != "forge" {
t.Errorf("verdict = %+v, want a failure at the forge: no build node was to blame", v)
}
}

Some files were not shown because too many files have changed in this diff Show More