Compare commits

..
Author SHA1 Message Date
jschoubben 64d364a768 Assigner pins mesh-sdk 76f7c4d, a pull request's head holding 8eafc0c and 77b48a4: the checker refuses a pin to a merge commit on main (issue 774)
mesh/delivery ready: it is merged on the Approve
mesh/check failed: the composition: the manifest of mesh-delivery-checker cannot be read: this manifest cannot be used: | 64d364a7686c on fa075dc880af…
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (d9a0e4aec26c): the composition
2026-10-29 00:20:03 +01:00
jschoubben 2caaa6fc92 Merge main into L3: the assigner pins mesh-sdk main d747c11, which holds both L3's sdk and the seed's held-as-today fix (issue 774)
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/774-vault-mints-l2 checking: 3 of 4 member(s) ready
2026-10-28 23:50:43 +01:00
mesh-tools fa075dc880 Merge pull request 'Assigner: pin mesh-sdk 06b13ec, so a seed's brief shows what stays held as today (issue 1126)' (#107) from fix/1126-seed-held into main 2026-10-28 22:20:54 +00:00
jschoubben f6df07605d Merge remote-tracking branch 'origin/main' into l3r/774-vault-mints
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (1c492872d414): shanks
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#105 does not merge into the trunk: files in conflict: modules/mesh-delivery-assigner/go.mod, modules/me…
# Conflicts:
#	modules/mesh-delivery-assigner/cmd/mesh-delivery-assigner/seat_tools_test.go
2026-10-28 23:00:04 +01:00
jschoubben fc3883b0b6 Assigner: pin mesh-sdk 77b48a4, so the seed's Approve names the differences it carries (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@fa075dc880af
mesh/delivery-group group fix/1126-seed-held merged: every member is merged
mesh/check pass fc3883b0b650 on 5b2e5bc53776 | unit 1199 run, 1015 kept, 33 skipped | integration 1218 run, 1081 kept, 26 skipped | over bound: unit (…
2026-10-28 22:41:03 +01:00
mesh-tools 5b2e5bc537 Merge pull request 'The checker writes a loud verdict where it cannot judge, so no Approve waits silently (issue 1733)' (#104) from fix/1733-no-hour into main 2026-10-28 21:29:56 +00:00
jschoubben 25953c6ae1 Assigner: pin mesh-sdk 06b13ec, so a seed's brief shows what stays held as today (issue 1126)
mesh/delivery-group group fix/1126-seed-held checking: 2 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 25953c6ae110 on 5b2e5bc53776 | novox/mesh-delivery#107 in group fix/1126-seed-held
2026-10-28 22:18:31 +01:00
mesh-tools 38f89c6976 Merge pull request 'The assigner serves the seat mesh-placement's verbs on the seat's own name (issue 1126)' (#106) from fix/1126-seat-names into main 2026-10-28 21:14:13 +00:00
jschoubben 49ea5c96e3 Assigner: serve the seat mesh-placement's verbs as <seat>.<verb>, the name the tool runner serves a seat's verb from (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@38f89c6976c9
mesh/check being carried to 49ea5c96e390 on 38f89c6976c9 | from 1f6f4eda3208
2026-10-28 22:07:05 +01:00
jschoubben 7692f201df Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-sdk@8eafc0c5ce3c was judged on trunk 53f26493697e,…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 7692f201df51 on 5b2e5bc53776 | novox/mesh-delivery#105 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 09da8d741e The assigner's laying again and its seed again know the lease write too, so a proposal laid again after it is judged as its first laying was (issue 774, ADR 0567 §8; re-review) 2026-10-28 21:54:47 +01:00
jschoubben b2351a213d Follow L2 under L3: the builder, checker, indexer and view read the seat mesh-placement's placement, the seat's Leased is the lease handed away, and the view keeps no read of the indexer's placement L2 removed (issue 774, ADR 0567 §8, ADR 0549 §3) 2026-10-28 21:54:47 +01:00
jschoubben 2abceb1913 Pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2, in the six modules the group changes (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 685810720e mesh-grants: place by the catalogue's placement alone once the controller's lease is handed away, and refuse any other reader of the assignments in this repository (issue 774, ADR 0549 §3, §6)
Why: after the lease write the assignments are frozen, so the part's union kept a module the catalogue takes off a node
with its account, grant, membership and user-list entry. The lease is read as the assigner reads it
(buscompose.LeaseHandedAway); before it, or with no placement on the bus, the union stands. The read stays declared until
the first change after the lease write, named in modules/assignments-readers with the assigner; a TestGrants row refuses
any other module.json that reads mesh-controller.assignments, as mesh-catalog's checks/placement does for the catalogue.
2026-10-28 21:54:47 +01:00
jschoubben 8e25926e91 mesh-grants: the node-engine's manifest is placed on every node, so its declared state is asserted and created as a placed module's, and it adds no account (issue 774, ADR 0563 §2, §3) 2026-10-28 21:54:47 +01:00
jschoubben 68628bfd78 Pin mesh-sdk 277a5bd, the L3 group's head, in the assigner, builder, checker, indexer, view and grants; the assigner's manifest test names mesh-grants.grants, its integration test no unused import (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 1dbb7c5740 mesh-delivery-view: node_trusts answers in one call whether every node's record names the vault's signing key, the gate before the controller's lease is handed away (issue 774, ADR 0550 §2, design 47 L3 s6b) 2026-10-28 21:54:47 +01:00
jschoubben 9e3d1d5e68 The assigner reads the catalogue's placement alone once the controller's lease is handed away, and reads the lease to know it (issue 774, ADR 0549 §5, §6) 2026-10-28 21:54:47 +01:00
jschoubben 9a70d697af The view shows a row per module the catalogue's placement puts on a node, and reads the controller's assignments no more (issue 774, ADR 0549 §3) 2026-10-28 21:54:47 +01:00
jschoubben a4116a1de7 Builder, checker and indexer count a build node's toolchain only where the catalogue's placement puts its build module, through mesh-sdk's one gate, and read the controller's assignments no more (issue 774, ADR 0549 §3, issue 1940) 2026-10-28 21:54:47 +01:00
jschoubben 6f721dd536 mesh-grants: say every placed module left without a grant, in the pass and as the condition mesh-grants.grant-missing once the lease is handed away (issue 1126, ADR 0547 §5)
Why: the controller's grant writer skipped link2pay, snake, f1-circuits and txt-game in silence (it read the fingerprint by
their unchanged commit, a build of the old form) and the take-in said 'its grant written'. The part already reads the
fingerprint from the indexer's value, so those four are granted at their src2 done key's fingerprint; a row proves it.
What a pass leaves without a grant (no fingerprint of the new form with the indexer's none2, a refused manifest, no
hashed account) is now said rather than passed over. Quick: log it. Thorough (chosen): the pass's Missing, shown by the
grants tool, and a warning condition raised and cleared on change, as consumer-differs is.
2026-10-28 21:54:47 +01:00
jschoubben 354b9ea396 mesh-delivery-assigner: read grants from the source the lease names (issue 774, ADR 0547 §4)
Why: from the lease write the grants part's state grants is the source and the controller's bucket otherwise; the assigner follows the lease key (a pass only when the source changes), and neither bucket stops its start. Needs the group's mesh-sdk pin (GrantSource, LeaseHandedAway, the assigner a named reader of the lease).
2026-10-28 21:54:47 +01:00
jschoubben 3618831992 mesh-grants: write grants.<module>.<fp> in shadow into the state grants, say grant-written, and show every difference from the controller's in grants_grants (issue 774, ADR 0547 §5)
Why: the assigner switches its grants source when the controller's lease is handed away; until then the written keys are compared with the controller's, so the switch rests on a measured equality. Needs the group's mesh-sdk pin (PartGrantKey) before it builds.
2026-10-28 21:54:47 +01:00
jschoubben 843a1589d7 Fold the seat verbs' test into the try tools test; pin the sdk head (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben fc6d5dcbdb mesh-delivery-checker claims its seat and serves try, try-read and try-end by the tools' own code, so the try outlives the controller (ADR 0547 §10, issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 6d709e1fc1 mesh-grants: grants_accept_hashes releases the clause 7 hold once, for exactly the held users of the held list, with why, said as hashes-accepted (issue 774, ADR 0561 §7, §9) 2026-10-28 21:54:35 +01:00
jschoubben eb11b2c844 mesh-grants: hold a list that changes several loaded users' hashes at once with no rotation by the vault naming them, and raise user-list-held naming them (issue 774, ADR 0561 §7) 2026-10-28 21:54:35 +01:00
jschoubben 67e3ebf22f mesh-grants: grants_accounts reads the account consumers' differences in shadow, where no condition is raised (issue 774, ADR 0561 §6, §9) 2026-10-28 21:54:35 +01:00
jschoubben 9b31746cdb mesh-grants: raise a condition when an account consumer differs from its composed configuration, and bring one to it by hand with grants_consumer_match, never under the controller's lease (issue 774, ADR 0561 §6, §9) 2026-10-28 21:54:35 +01:00
jschoubben ebad58f7e2 mesh-grants: create the durable consumers an account needs from the lease write, compared in shadow until then, so a module assigned after the controller leaves still hears what it consumes (issue 774, ADR 0561 §6) 2026-10-28 21:54:35 +01:00
jschoubben 3c6fb7fc9e mesh-grants: compose the user list again when the vault says account-issued, so a minted account reaches the bus within one pass (issue 774, ADR 0561 §1) 2026-10-28 21:54:35 +01:00
jschoubben 71aa3e86c6 mesh-grants: write the bus accounts the mesh wants to a state only the vault reads, so the vault can mint them; grants_accounts reads it (issue 774, ADR 0561 §2, §9) 2026-10-28 21:54:35 +01:00
jschoubben 2d8ad2e66a Checker: no candidate is left without a verdict (issue 1733)
mesh/delivery merged: source.main moved to novox/mesh-delivery@5b2e5bc53776
mesh/delivery-group group fix/1733-no-hour merged: every member is merged
mesh/check pass 2d8ad2e66a8f on 38f89c6976c9 | carried from 1f6f4eda3208 | unit 470 run, 4760 kept, 13 skipped | integration 503 run, 4844 kept, 22 sk…
Once the gate's one-hour ask bound goes (mesh-catalog fix/1733-no-hour), the
gate asks the operator's Approve only on a verdict, so every path that left
a candidate without one now writes one, from an event and with no timer:
- a group with a member whose repository's trunk head is not known (a pull
  request the checker cannot claim) is not judged in part: its verdict fails
  naming that repository, and every status of the group says so (a pass of
  the others would have let it merge with that member never checked);
- the verdict of heads that cannot be judged replaces whatever the key held,
  such as a running verdict left by a check that can no longer run;
- a verdict key deleted is judged again on its delete (the checker watches its
  verdicts), and a check running when its key is deleted writes it anew.

Tests: three rows at the end of the no-trunk test; counts unchanged.
2026-10-28 21:50:03 +01:00
mesh-tools 1f6f4eda32 Merge pull request 'The checker and the builder use it; checker_ask_again asks a lost check again; the checker's integration bed writes a node record (issue 734's kind)' (#103) from fix/lost-check into main 2026-10-28 20:41:01 +00:00
mesh-tools 769815ac0f Merge pull request 'The assigner's seed and mesh-grants take mesh-wireguard as provided by the mesh (issue 1126)' (#102) from fix/1126-provided into main 2026-10-28 20:39:23 +00:00
jschoubben e7888024a8 Checker: delete a check's asks key at its end, so the state asks holds only the checks that run
mesh/delivery merged: source.main moved to novox/mesh-delivery@1f6f4eda3208
mesh/delivery-group group fix/lost-check merged: every member is merged
mesh/check checking e7888024a826 on 769815ac0fea | novox/mesh-delivery#103 in group fix/lost-check
Re-review of fix/lost-check: writing an empty list at the end kept one key
for every verdict key ever checked. Tests: every row of the lost-ask table
proves the key gone at the end; one row more (deaf after its build module
replaced it, asked again by hand); counts unchanged.
2026-10-28 20:39:50 +01:00
jschoubben 2007a79c45 Checker and builder pin mesh-sdk f91340f, which opens a silent steps watch again
mesh/delivery-group group fix/lost-check checking: 1 of 2 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 2007a79c45c3 on a3a0d8f2ceb3 | novox/mesh-delivery#103 in group fix/lost-check
2026-10-28 20:38:31 +01:00
jschoubben d8286d1790 Assigner and grants: take the private network as the mesh's own through mesh-sdk's one predicate, so the seed is not refused for it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@769815ac0fea
mesh/delivery-group group fix/1126-provided merged: every member is merged
mesh/check checking d8286d1790fb on a3a0d8f2ceb3 | novox/mesh-delivery#102 in group fix/1126-provided
2026-10-28 20:38:14 +01:00
jschoubben ca74639a74 Checker integration test: the checker says its seventh state, asks, as its own
The new state asks (checker_ask_again's record of each running check's
asks) is kept and not readable, as hearing is.
2026-10-28 20:37:07 +01:00
jschoubben a58a73bb55 Checker integration tests: give the bed's node a record in the have, so the names check finds a name to ask with (ADR 0567 §12, issue 1126)
Since 9db9499 the checker asks no check while no node record is on the bus; the bed wrote none, so eight integration tests failed on main (PR #101's check). The fake build module now records the names an ask carries, and the first test asserts n1 is named.
2026-10-28 20:36:25 +01:00
jschoubben 16594feff1 checker_ask_again: refuse while an ask of the check waits in its build seat's stream
Review of fix/lost-check: an ask no node took yet has no step key, so
asking again then would run the check twice (issue 1791's kind). The
checker keeps the asks each running check put in a new state asks, and the
tool refuses unless every one of them has a step key. Asking again under the
same ask id is not done: the stream drops a second put of an id within its
de-duplication window and keeps both past it, so the waiting copy is never
replaced.

Tests: the hand row of the lost-ask table refuses while its ask waits;
counts unchanged.
2026-10-28 20:36:25 +01:00
jschoubben cb46ff8735 Checker and builder wait through mesh-sdk's HearEnds, so a deaf watch loses no step end in either
The checker's HeardEnds moved to mesh-sdk buildrun (fix/lost-check,
6aeb401): both modules pin it, and the builder, which waits on the same
Asker, now reads an ask's step at each step end of its build module too.
checker_ask_again and checker_status's lost stay in the checker.

Tests: one row in the builder's ask table (its watch deaf, read at a
nudge, done with no new ask); counts unchanged.
2026-10-28 20:31:04 +01:00
jschoubben 6f079f13a8 Checker: never lose a step's end, and ask a lost check again (issue 734's kind)
mesh-catalog #553's verdict stayed running after its step mvsgthyjed5786
passed on shanks at 19:09:31Z: the watch of that one step key, opened at
18:46:05Z, heard neither its start nor its end and never closed, so the
wait had no bound running and nothing would ever judge it.

- HeardEnds stands between the asker and each ask's watch: at every step
  end of the same build module (one watch per steps state), at a
  reconnect, the asks waited on read their keys now and are judged; a
  watch that closes is opened again, at most 3 times.
- A step the build module's next process ended (RestartReason, a delivery
  replacing it) counts as a lost ask: asked again under the claim, at most
  LostAskTries, then failed as not heard, judged again when the nodes change.
- checker_ask_again {group, why} ends a running verdict as asked again by
  hand; the checker hears its verdicts, ends the wait and claims it anew.
  Refused while a step of its pull requests runs, or for a verdict that
  is not running.
- checker_status names such checks under lost, and each step's what.

Tests: one table replaces the given-up test (rows for given up, replaced,
deaf and nudged, deaf and asked by hand); counts unchanged (147 unit, 10
integration in the checker).
2026-10-28 20:27:41 +01:00
mesh-tools a3a0d8f2ce Merge pull request 'Format two assigner files as gofmt writes them, so mesh-delivery's check passes again (issue 1126)' (#101) from fix/1126-assigner-gofmt into main 2026-10-28 19:08:03 +00:00
jschoubben 5fe8ab11f1 Assigner: format pass.go and placement.go as gofmt writes them, which the check requires (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@a3a0d8f2ceb3
mesh/check checking 5fe8ab11f1b4 on a3a0d8f2ceb3
2026-10-28 20:03:16 +01:00
mesh-tools b6eb43dd62 Merge pull request 'mesh-delivery-builder pins mesh-sdk 38dbe1d, so it accepts the assigner's claim on mesh-placement (issue 1126)' (#100) from fix/1126-builder-pin into main 2026-10-28 19:02:33 +00:00
jschoubben dc4419b5c5 Builder: pin mesh-sdk 38dbe1d, so it knows the seat mesh-placement and builds the assigner that claims it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b6eb43dd6271
mesh/check checking dc4419b5c50f on b6eb43dd6271
2026-10-28 19:57:29 +01:00
mesh-tools 24181c0242 Merge pull request 'The assigner holds the seat mesh-placement; the indexer, bridge and view drop the repository path (issue 1126, delivery 2 of 2)' (#99) from feat/1126-placement-seat into main 2026-10-28 18:33:09 +00:00
mesh-tools db737df597 Merge pull request 'mesh-grants: pin mesh-sdk 38dbe1d, so the bus loads the node agents' reads of the seat's states first (issue 1126, delivery 1 of 2)' (#98) from feat/1126-grants-first into main 2026-10-28 18:21:36 +00:00
jschoubben dd9acaaeb3 Assigner: bind the seat's states at each pass and say while it waits for them, since their creation can come before this credential may hear it (ADR 0567 §1, review round 3)
mesh/delivery merged: source.main moved to novox/mesh-delivery@24181c024212
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
2026-10-28 19:14:43 +01:00
jschoubben 451a4dfe74 Grants: pin mesh-sdk 38dbe1d, so the bus loads the node agents' reads of the seat mesh-placement's states before anything reads them (ADR 0567 §9, issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@db737df597c3
mesh/delivery-group group feat/1126-grants-first merged: every member is merged
mesh/check pass 451a4dfe74d9 on 2ac146078f47 | unit 1909 run, 715 kept, 3 skipped | integration 1931 run, 813 kept, 3 skipped | over bound: unit (slow…
2026-10-28 19:11:58 +01:00
jschoubben fb26ca5112 Pin mesh-sdk 38dbe1d, the group's head, in every module the seat touches (issue 1126) 2026-10-28 18:48:56 +01:00
jschoubben 389bb96ae4 Grants: read placement from the seat mesh-placement's state, as every reader does (ADR 0567 §9, issue 1126) 2026-10-28 18:47:49 +01:00
jschoubben 06e668cb90 Assigner: bound answers and asks by pulse keys instead of a timer, ask the operator for a hand-back, follow a partial write through, and call no controller verb for the seed (ADR 0567 §4, §5, §7, §9, review round 2) 2026-10-28 18:46:39 +01:00
jschoubben 297a3cb549 View: build_usage reads each node's capacity as the integer mesh-config resolves for it, default 0, since no repository names a node (ADR 0567 §12, issue 1126) 2026-10-28 18:35:14 +01:00
jschoubben 1d0467d4c3 Forge bridge: hold that a catalogue merge carries no placement against the retired field, as mesh-sdk now names it (ADR 0567 §11, issue 1126) 2026-10-28 18:34:19 +01:00
jschoubben 9db9499267 Checker: put every node's name on each check ask for the names check, so no repository lists them (ADR 0567 §12, issue 1126) 2026-10-28 18:34:02 +01:00
jschoubben deda4c985e Assigner: fold three small tests into their neighbours, so the seat's tests leave the counts flat (test budget) 2026-10-28 18:32:11 +01:00
jschoubben 06df6a6167 Assigner: prove a proposal answered, approved and written whole on the bus, and a moved key laid again (ADR 0567 §5, issue 1126) 2026-10-28 18:31:44 +01:00
jschoubben 1718d4b1d7 Assigner: follow every proposal from its answers through the Approve to the write at its revisions, keeping the controller in step (ADR 0567 §3-§8, issue 1126) 2026-10-28 18:31:43 +01:00
jschoubben 16b71ca685 Assigner: serve the seat's seven verbs, so a person proposes, withdraws, adopts, hands back and reads placement through the seat (ADR 0567 §2, §7, §10, issue 1126) 2026-10-28 18:31:43 +01:00
jschoubben 5d57b78fe9 Assigner: claim the seat mesh-placement with its stream, invokes and the resolved values it composes with (ADR 0567 §1, §6, §8, §9, issue 1126) 2026-10-28 18:22:40 +01:00
jschoubben 36332abf35 Assigner: drop the pull request's placement check and hold the seat's states as its own, since a placement change is no repository change (ADR 0567 §1, §4, §11, issue 1126) 2026-10-28 18:19:12 +01:00
jschoubben dbbdc95c0b View: drop a comment about the removed placement read 2026-10-28 17:59:34 +01:00
jschoubben fb0be84e24 View: drop indexer_placement and its reads, the seat mesh-placement answers placement (issue 1126) 2026-10-28 17:59:27 +01:00
jschoubben 730e44d762 Forge bridge: carry no placement files, placement is the seat mesh-placement's (issue 1126)
A catalogue merge or pull request no longer lists or reads placement/, so the Listed read and ConvertWith have no use and go.
2026-10-28 17:58:58 +01:00
jschoubben f4dd334b49 Indexer: stop holding the catalogue's placement, it is the seat mesh-placement's now (issue 1126)
Deletes the placement and candidate writes, the reread tool and its forge reads, the two buckets and the placement-changed event.
D1a and D1b are one commit: the index package, the named reads, the tools and the manifest do not compile apart.
2026-10-28 17:58:09 +01:00
mesh-tools 2ac146078f Merge pull request 'The delivery view forgets a write or event the bus refused, so 'on the machines' is not left stale (issue 1126)' (#97) from fix/1126-reach-now into main 2026-10-28 15:18:54 +00:00
jschoubben 726bbf1f1f The view writes a reach or says an event again when the bus refused it, so a refused write never leaves the reach state at an older have (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2ac146078f47
mesh/check being carried to 726bbf1f1ffa on 2ac146078f47 | from e753009fbc71
The view remembered a reach value, a reached head and a motion event as written or said before the bus took them; a write refused at its start (no responders) was never made again while the derivation stayed the same, and mesh-delivery.show said mesh-delivery-view not running on shanks while node_haves said it running.

Quick fix: forget only the refused reach write. Thorough fix, chosen: every write and event the view makes after its lock (reach, reached head, delivered revision, motion) answers its error, and the model forgets what was refused, so the next change it takes in makes it again; one place, the writer of the state both mesh-delivery.show and knowledge-base.progress read.
2026-10-28 16:11:52 +01:00
mesh-tools e753009fbc Merge pull request 'The indexer never moves a module backwards: a late merged message writes nothing; indexer_heard and indexer_module_history read what it heard and kept (issue 1126)' (#96) from fix/1126-index-order into main 2026-10-28 14:44:34 +00:00
mesh-tools f39bc10e3b Merge pull request 'builder_build_again: a person can build a failed key again, as the builder does by itself (issue 1126)' (#95) from fix/1126-build-again into main 2026-10-28 14:27:08 +00:00
jschoubben 3629bcf057 Read only gitea's 404 status as not-found, so a 502 naming a sha with 404 in it is redelivered, not written backwards (issue 1126, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@e753009fbc71
mesh/check being carried to 3629bcf057c3 on e753009fbc71 | from f39bc10e3be7
2026-10-28 15:01:51 +01:00
jschoubben 743a3618ea Keep back only a merge older than the one taken in, asked once per message before any write (issue 1126, review)
Asking whether the merge contains the held commit refused forward merges
forever on a diverged or rewritten main or a squash-merged registered head,
failed every later merge when the forge no longer had the held commit, and
cost one forge call per held commit. The question is now the reverse: a merge
is kept back only when it is in the taken merge and differs from it. A merged
message asks it once, before it writes anything, and a late message writes no
module, deletion, placement or taken key. A take-in asks it once for its
module. A 404 is not known to be older and is written, and said; any other
failure to say leaves the message for redelivery.
2026-10-28 14:56:40 +01:00
jschoubben 77d738d973 Never let a merge move a module backwards, and make the forge messages and a module's history readable (issue 1126)
The indexer wrote mesh-catalog's merge 9cdac09a back over the later 69056948,
rolling kernel and mesh-build-image back on the nodes: a merged message's
commit was written whatever the held one was. Before writing or deleting a
module's value, the indexer now asks the forge whether the merge contains the
held commit, keeps the held value when it does not and says so, and leaves the
message for redelivery when the forge cannot say. Each applied merged message
is said in one line with the modules written.

No tool showed what the indexer heard of FORGE nor a modules key's history
with times, so the cause could not be proven: the indexer now records each
FORGE message it handles (sequence, deliveries, times, commit, outcome) in its
state heard, read by indexer_heard, and indexer_module_history reads
modules.<module>'s kept values with their times.

Quick fix (Src order in writeOverlay) rejected: Src is 0 once the source moved
on, as here. Thorough fix chosen: the order is the forge's, on every path that
writes a module (merged message, take-in).
2026-10-28 14:48:13 +01:00
jschoubben a3b4ac79af builder_build_again: a look the builder queued during the by-hand build still runs; say a look that ended early is not asked
mesh/delivery merged: source.main moved to novox/mesh-delivery@f39bc10e3be7
mesh/check being carried to a3b4ac79afaa on f39bc10e3be7 | from 31bbf207ef3b
2026-10-28 14:32:33 +01:00
jschoubben c6dd04a7cc builder_build_again: register the look in one critical section, and answer what the build did
Review round 1 (issue 1126 note 72): the inflight check, the watch and the look were three locks, so a second request or the builder's own look could build the key twice; the answer said asked when the look wrote nothing or the build failed before its asks.
2026-10-28 14:30:31 +01:00
jschoubben 48cd554f46 Add builder_build_again: build a failed key again by hand as a new revision (issue 1126 note 72)
The builder retakes a failed key only when it looks at its module again, which nothing does for an unchanged module. The tool makes it look now, with the claim's own rules, and refuses a key that is done or waiting.
2026-10-28 14:25:03 +01:00
mesh-tools 31bbf207ef Merge pull request 'Say loudly what cannot be judged, read only a merge's changed directories, and move the taken key only by an applied merge (issues 1422, 1126; group fix/1422-trunk-unknown)' (#94) from fix/1422-trunk-unknown into main 2026-10-28 10:34:50 +00:00
jschoubben 2b801f5bd4 indexer_take_in reads a held module again only when its record is behind the source's head, and the two new tests are rows (issue 1126, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@31bbf207ef3b
mesh/delivery-group group fix/1422-trunk-unknown merged: every member is merged
mesh/check being carried to 2b801f5bd49a on 31bbf207ef3b | from ae7ec702a5e9, mesh-catalog 9fd95693db54
A head older than the record, or one whose order the forge cannot say, would have written
an older manifest back. The new tests become rows of existing ones: the repository is over
its 500 unit tests.
2026-10-28 11:18:01 +01:00
jschoubben 790653dfde indexer_take_in reads a held module again at the source's head when its record stands behind it (issue 1126)
A merge whose merged message never reached the indexer had no recovery: take-in answered
held for mesh-vault, whose record stayed at 78d7ef7 after mesh-catalog#535.
2026-10-28 11:10:09 +01:00
jschoubben 89be76bb6a The forge bridge keeps the directories a failed try read whole, so a large merge converges over its tries (issue 1126)
Without it the first merge after the bridge starts, with nothing kept, still reads all 140
catalogue directories in one 30 s try and times out as before.
2026-10-28 11:10:09 +01:00
jschoubben 704fc8d467 The forge bridge reads a merge's module directories only where their tree changed (issue 1126)
mesh-catalog's 140 module directories are 420 forge reads per merge, past the 30 s bound
of a handled event while the forge is busy: the merged message of #535 timed out on every
try, and #3fb7951 needed seven (issue 1458 met the same kind at 131). A tree hash names its
bytes, so what was read at a tree is kept in the new state inputs and in memory, and a
merge reads only the directories it changed.
2026-10-28 11:10:09 +01:00
jschoubben 458e403a04 The indexer's taken key moves only by a merge it applied, never by the gate's source head (issue 1126)
A drained forge stream does not say a merge was applied: the forge bridge was still
retrying mesh-catalog#535's merged message when the indexer started, and the source's
head moved taken.mesh-catalog to 9fd9569, so the view and show said the merge was on the
machines while mesh-vault's record stood at 78d7ef7 (ADR 0556 clause 1).
2026-10-28 11:10:09 +01:00
jschoubben 93c0ae19dd Forge bridge: write a head pushed to a pull request into another base as closed, and say so (issue 1422)
The checker judges pull requests into main only and source.main names main's head only (ADR 0484 §11), but a live update
into another base (de-spiegel#6 into nox-mesh) became a candidate, unlike at the start (OpenPulls) and at its merge. Once
the gate seeds a repository's trunk head, such a candidate would be judged against main's head: a verdict that lies.
2026-10-28 11:03:39 +01:00
jschoubben 4e9b076408 Checker: say loudly that a pull request cannot be judged when its repository has no trunk head (issue 1422)
A repository never merged through the gate has no entry in source.main, so its pull requests were dropped without a log
line, a verdict or a status, and checker_status counted them as new. Now the verdict says it cannot be judged and why,
the forge status is error with those words, checker_status names it under cannot_be_judged, the log says it once per
head, and the heads are judged whole once the trunk head is known. Quick fix: this. Thorough fix (the gate seeding
source.main from the pull message's base head) is still to do: it needs mesh-sdk, the forge bridge and the gate.
2026-10-28 11:03:39 +01:00
mesh-tools ae7ec702a5 Merge pull request 'Delete a closed pull request's placement candidates, and give indexer_placement a summary (issue 1126, group fix/1126-seed-same)' (#93) from fix/1126-seed-same into main 2026-10-28 09:46:34 +00:00
jschoubben 2cccbfd096 mesh-delivery-indexer and mesh-delivery-view pin mesh-sdk 3028c7e, the group's head after review round 2 (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ae7ec702a5e9
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check checking 2cccbfd096d8 on ae7ec702a5e9 | novox/mesh-delivery#93 in group fix/1126-seed-same
2026-10-28 10:02:46 +01:00
jschoubben 8f5e7f0c8f Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 75d3c40f41 mesh-delivery-indexer and mesh-delivery-view pin the group's mesh-sdk head 1062b33 (issue 1126; go.sum is filled once that head is pushed) 2026-10-28 09:51:48 +01:00
jschoubben 63582e44f3 Delete a closed pull request's placement candidates even when its closed message was missed, and give indexer_placement a compact summary form (issue 1126) 2026-10-28 09:25:09 +01:00
mesh-tools 4d4743d900 Merge pull request 'mesh-delivery-view passes a node's placement answer through as written, so its bodies reach readers (issue 1126, group fix/1126-shadow-bodies)' (#92) from fix/1126-shadow-bodies into main 2026-10-28 07:46:28 +00:00
jschoubben 289d1e8fa6 Format mesh-delivery-view's read.go as gofmt writes it (issue 1126, the group check's failure)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4d4743d90065
mesh/delivery-group group fix/1126-shadow-bodies merged: every member is merged
mesh/check checking 289d1e8fa69a on 4d4743d90065 | novox/mesh-delivery#92 in group fix/1126-shadow-bodies
2026-10-28 01:49:54 +01:00
jschoubben 968b9bf2e6 mesh-delivery-view: a node's candidate answer is shown as the node wrote it; pin mesh-sdk 362a3f1 (issue 1126, probe #532)
mesh/delivery-group group fix/1126-shadow-bodies rejected: composed together: the checker's verdict at the members' heads (46e1e846f8db): unit: modules/mesh-de…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (2 tests): unit: modules/mesh-delivery-view.gofmt: not as gofmt writes them (run gofmt -w): modules/mesh-delivery-view/internal/view…
The bodies of each difference never reached the answer shown: the view decoded
the node's answer with its mesh-sdk, which did not know the field, and encoded
it again. The answer now rides through as stored; a decoded copy only says its
state.
2026-10-28 01:39:34 +01:00
mesh-tools bab2e1391a Merge pull request 'mesh-grants: stop rewriting the shadow user list every 3 s when nothing changed (issue 1126, group fix/1126-switch-safe)' (#91) from fix/1126-switch-safe into main 2026-10-27 23:52:23 +00:00
jschoubben 663c9cbc34 mesh-grants: pin mesh-sdk 7359d18, the group's head (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bab2e1391ae5
mesh/delivery-group group fix/1126-switch-safe merged: every member is merged
mesh/check pass 663c9cbc348d on a49fd9cc65f7 | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
2026-10-28 00:34:13 +01:00
jschoubben 3d916a685e mesh-grants: the same list said again moves neither the sealed key nor the shadow list (issue 1126)
Live since the take-over the shadow list was written every 3 s (revision 99270
to 99357 in 90 s, nothing differing): the sealed key read back says [] for the
nil left-out a pass composes, so every pass wrote the key again; each write
moved the bus node's report of what it loaded, whose revision and time were
inputs of the shadow list, and woke the next pass. The bus itself reloads only
when the list's bytes change (its log: two configurations alternating with the
placement, f8567865 and 473f9ad3), so this loop was load, not the reload.
2026-10-28 00:20:44 +01:00
mesh-tools a49fd9cc65 Merge pull request 'mesh-grants: take-over allowed when the grants part's list only adds rights, every loss still refused (issue 1945, ADR 0562)' (#90) from fix/1945-takeover-adds into main 2026-10-27 21:36:25 +00:00
jschoubben 3192d89edf mesh-grants: refuse a take-over over an uncompared deny; ask words only for added rights
mesh/delivery merged: source.main moved to novox/mesh-delivery@a49fd9cc65f7
mesh/check checking 3192d89edfcb on a49fd9cc65f7
Review round 1 of issue 1945's fix found two ways the guard could let a loss
through or hide one from the operator.

A publish deny was never compared: grants.User carried no deny, so a widened
allow that brings a new deny (a lost publish right) passed as an added right.
The user now carries its deny; a deny the shadow adds is a loss, and while
nats_users answers no deny for the loaded list, a user whose grants differ
and whose shadow grant has a deny refuses, named as a deny the loaded list
cannot be compared on. None of today's 41 differing users carries a deny
(only the four node-tools users do, and they are the same in both lists).

The ask said "takes none away" whenever rights were added, whatever else
differed. Its words now name the added rights only when nothing else refuses
(Refusals: unread list, Removing, a missing hash), and otherwise say what the
yes would be refused for, naming the users; Decide uses the same Refusals.

issue 1945
2026-10-27 22:33:53 +01:00
jschoubben 9edfcc40e9 mesh-grants: allow take-over when the lists differ only by added rights
The operator's yes to take-over was refused twice (2026-10-26 13:48 and
2026-10-27 21:19 UTC) because 41 users differ, every difference a right the
grants part's list adds. The controller's list is not changed (ADR 0547 §1),
so ADR 0547 §8's equality could never hold again. ADR 0562 changes it:
take-over is allowed when no user is added or dropped, no loaded user loses a
publish, subscribe or responses right and every hash is held; the ask names
the added rights by user and kind, and a yes after they changed is refused.

Tests: rows from the captured live comparison of 21:19 (testdata, no
hashes) in the compare table, and rows in the take-over table for a right
removed, added rights changed since the ask, and only added rights recorded.

issue 1945
2026-10-27 22:28:22 +01:00
mesh-tools 68bdda1a4d Merge pull request 'mesh-grants and the assigner: grant every node the served keys it composes with (issue 1126, group fix/1126-node-creds)' (#89) from fix/1126-node-creds into main 2026-10-27 20:32:34 +00:00
jschoubben bb7f2ff365 Merge remote-tracking branch 'origin/main' into fix/1126-node-creds
mesh/delivery merged: source.main moved to novox/mesh-delivery@68bdda1a4dbf
mesh/delivery-group group fix/1126-node-creds merged: every member is merged
mesh/check checking bb7f2ff365c7 on 68bdda1a4dbf | novox/mesh-delivery#89 in group fix/1126-node-creds
2026-10-27 21:18:57 +01:00
mesh-tools da51481620 Merge pull request 'mesh-grants: a seat keeps its holder under any placement, and no list or membership leaves a claimed seat without one (issue 1945)' (#88) from fix/1945-grants-seats into main 2026-10-27 20:14:04 +00:00
jschoubben c93f0d15e4 Grant the served keys of the trunk's manifest too, and pin mesh-sdk 7a8e9f9 in the assigner and mesh-grants on main's bus helpers (issue 1126, review round 1)
A node composing a consumer reads the served keys of the trunk's manifest of a
provider elsewhere, which may serve a setting the build it runs does not.
2026-10-27 21:13:41 +01:00
jschoubben e0cb285993 Merge remote-tracking branch 'origin/main' into fix/1126-node-creds
# Conflicts:
#	modules/mesh-delivery-assigner/go.mod
#	modules/mesh-delivery-assigner/go.sum
2026-10-27 21:12:25 +01:00
jschoubben a7d03f7c82 Pin mesh-sdk 538ffba, the group's head, in the assigner and mesh-grants (issue 1126) 2026-10-27 21:04:08 +01:00
jschoubben ea3bfa5e6e Grant every node the keys a provider elsewhere serves from, and give the composition check frombus's new reads, pinning mesh-sdk cebc651 (issue 1126)
mesh-grants declares each module's served settings (catalogue.ServedSettings), so
the user list grants every other node's agent those keys of mesh-config_resolved
alone: shanks then settles what novox's mailu serves. The assigner's check of
every node reads no build, archive or served value: it composes as before.
2026-10-27 21:02:49 +01:00
jschoubben ec0b7e2dcc Keep holding a membership that would drop a capability its holder still asks for (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@da514816204b
mesh/check being carried to ec0b7e2dcc06 on da514816204b | from 24cb379434f5
A dropped verified-sender comes from a root-free word that failed for one
pass, not from a release, and losing it cut the operator's approvals on
2026-10-27. Holdings.CapabilitiesLost holds such a membership with its own
reason while the holder is assigned and its manifest still asks for it; a
dropped tool or field and a moved seat stay published.
2026-10-27 20:56:09 +01:00
jschoubben feb32600bb Hold a list or membership only when a seat would lose its last holder (issue 1945)
Holding the user list on any loss would freeze it after take-over at the
first release that drops a tool or moves a seat (ADR 0547 §7), with nothing
to accept a deliberate loss; memberships had the same gap. The lead's decision
in review round 1: one pure guard, Holdings.LastHolders, holds the list and
each membership only when a seat held in force would have no holder in what
is composed now while an assigned module still claims it, the 09:21 kind.
A dropped tool, verb or field, or a seat handed to another node, is written.
2026-10-27 20:54:30 +01:00
jschoubben bc1978ec67 Keep every seat's holder under a placement, and hold a user list that would lose a grant (issue 1945)
The user list of 2026-10-27 09:21 granted no holder for nine seats: the seeded
placements hold only the seats the controller had on record, and the grants part
read those holds as every holder (records.go heldWherePlaced). A placement's
holds are now the holders on record, as catalogue.World's Holdings are: a mesh
seat no placement holds is held by derivation, wherever its claimant is placed.

The membership loss guard (Losses) did not cover the user list: after take-over
the list that lost those seats went out. UserLosses applies the same guard per
user against the loaded list, and holdsBack holds a list that would lose a grant,
or whose loaded grants cannot be read, raising the held warning.

Quick fix chosen over moving the on-record rule into mesh-sdk: mesh-grants is the
only part reading placement holds itself; the others pass them to the catalogue.
2026-10-27 20:54:30 +01:00
mesh-tools 24cb379434 Merge pull request 'Every delivery part opens its bus connection through mesh-sdk's module helpers (issue 1242)' (#87) from fix/1242-bus-helper into main 2026-10-27 19:49:22 +00:00
jschoubben 96be0c3377 Move every delivery part's bus connection onto mesh-sdk's module helpers: credential, pinned dial, own-stream subjects, durable consumer, and refuse a hand-written one (issue 1242)
mesh/delivery merged: source.main moved to novox/mesh-delivery@24cb379434f5
mesh/delivery-group group fix/1242-bus-helper merged: every member is merged
mesh/check checking 96be0c3377c4 on c38e81c7f5e9 | novox/mesh-delivery#87 in group fix/1242-bus-helper
2026-10-27 20:40:49 +01:00
mesh-tools c38e81c7f5 Merge pull request 'A want's self comes from the current placement alone, so a node no longer placed gets the controller's composition instead of a stale self want (issue 1126)' (#86) from fix/1126-self-wants into main 2026-10-27 15:25:20 +00:00
jschoubben bb63d83f84 Derive a want's Self from the placement now, in every rule, so a want written while a node was placed does not outlive it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c38e81c7f5e9
mesh/check being carried to bb63d83f849e on c38e81c7f5e9 | from 2915d74822cd
After the placement seed's revert the rules that hold a want (a hold, rules 3 and 5) kept g14's and shanks's Self wants: the held branch copied them whole, and the rule branch wrote a want naming no composition, which EncodeWant refuses. Module.Self, set once from the placement, is now the one source, and every want the rules write or keep is derived from it: an unplaced node's kept want names the controller's composition of the same fingerprint, asked as any record move; with none, it stays and rule 7 says why.
2026-10-27 16:19:02 +01:00
mesh-tools 2915d74822 Merge pull request 'A placement pull request is carried, kept as candidates, and checked against every placed node's answer before it merges (issue 1126, ADR 0559)' (#85) from feat/1126-l2b into main 2026-10-27 07:21:16 +00:00
jschoubben 06d4f45e18 Pin mesh-sdk cadd7b1, the group's final head, in every member module (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2915d74822cd
mesh/delivery-group group feat/1126-l2b merged: every member is merged
mesh/check checking 06d4f45e1885 on 2915d74822cd | novox/mesh-delivery#85 in group feat/1126-l2b
2026-10-27 02:44:11 +01:00
jschoubben 77dacfbb17 Grant the node agents the placement candidates through the grants part at the group's sdk head, and show in indexer_placement the answer of every node placed at the head or on the trunk, so the seed's nodes can answer and be seen (issue 1126, ADR 0559 §3, §7; review round 1) 2026-10-27 02:43:00 +01:00
jschoubben d8592d9a4f Pin mesh-sdk 062c29c, the group's final head, in every member module that uses the candidate contract (issue 1126) 2026-10-27 02:30:16 +01:00
jschoubben ea4a163187 Pin mesh-sdk a37d262, the group's head, in the bridge, the indexer and the view (issue 1126) 2026-10-27 02:29:28 +01:00
jschoubben fc92202e04 Take a later composition into a verdict that failed only because placed nodes had not answered, so a sleeping node's answer passes the check on its event with no new check (issue 1126, ADR 0559 §5) 2026-10-27 02:29:20 +01:00
jschoubben d8da56c6da Compose a catalogue pull request's own placement in the assigner's check and fail it on any placed node's unintended difference, refusal or silence past ten minutes, so a placement change is proven equal to what runs before it merges (issue 1126, ADR 0559 §5) 2026-10-27 02:29:20 +01:00
jschoubben c2b1ba4c5d Pin mesh-sdk ba398e2, the group's head with the node's whole candidate answer (issue 1126) 2026-10-27 02:20:17 +01:00
jschoubben 04bb2d1398 Show placement candidates and each node's answer in indexer_placement, add indexer_placement_reread, and drop a group's candidates when its head changes no placement file, so a missed message or a revert is recoverable by tool (issue 1126, ADR 0559 §3, §7, §8) 2026-10-27 02:18:21 +01:00
jschoubben ad60ba3a46 Write a catalogue pull request's changed placement as placement-candidates <group>.<node> and delete them when it merges or closes, so each placed node composes it in shadow and answers (issue 1126) 2026-10-27 02:13:55 +01:00
jschoubben ea3eac51bd Carry a catalogue pull request's placement files at its head when it touches placement/, so the indexer can write them as candidates for the nodes to answer before the merge (issue 1126) 2026-10-27 02:13:55 +01:00
mesh-tools cacf537004 Merge pull request 'The checker's failure is the build step's failure, and a verdict that hides its failing test is refused (issue 1901)' (#84) from fix/1901-failure-contract into main 2026-10-26 23:44:21 +00:00
jschoubben f4a86f5884 Merge main into fix/1901-failure-contract: the checker pins mesh-sdk a2759ec, the group's sdk head (issue 1901)
mesh/delivery merged: source.main moved to novox/mesh-delivery@cacf537004c9
mesh/delivery-group group fix/1901-failure-contract merged: every member is merged
mesh/check checking f4a86f58843d on cacf537004c9 | novox/mesh-delivery#84 in group fix/1901-failure-contract
2026-10-27 00:39:35 +01:00
mesh-tools 9f1b227fb8 Merge pull request 'The indexer fills each repository's taken key from merges it applied or a drained stream's heads, so a merge heard while the state was missing reads on the machines (issue 1280)' (#83) from fix/1280-indexer-fills-taken into main 2026-10-26 23:22:01 +00:00
jschoubben 6c79e456a1 The indexer confirms a forge message's ack with the server before it fills taken keys (issue 1280)
mesh/delivery merged: source.main moved to novox/mesh-delivery@9f1b227fb8c0
mesh/check checking 6c79e456a100 on 9f1b227fb8c0
2026-10-27 00:20:16 +01:00
jschoubben 0b980203fe Taken keys are filled only from merges the indexer applied or a drained forge stream's heads (issue 1280) 2026-10-27 00:18:23 +01:00
jschoubben 73bf290f58 The indexer fills every repository's taken key from the source's head at start and when the state appears (issue 1280) 2026-10-27 00:14:41 +01:00
mesh-tools 475bb4bb0b Merge pull request 'The view's tools read their own reach by its bare name, and a test holds every state the commands read to what is issued (issue 1280)' (#82) from fix/1280-view-reads-its-reach into main 2026-10-26 22:48:25 +00:00
jschoubben efadc691e7 The view's tools read their own reach by its bare name, as the mesh issues it, and a test holds every state the commands read to what is issued (issue 1280)
mesh/delivery merged: source.main moved to novox/mesh-delivery@475bb4bb0b8f
mesh/check being carried to efadc691e7ea on 475bb4bb0b8f | from 4af36c69a761
view_reach asked for mesh-delivery-view.reach, a name the mesh never issued, so it failed on the machines;
the same line was not gofmt'd, which failed the try of main. Quick fix: the name. Thorough for this module: the
manifest test now refuses any stdio.State literal in cmd/ that is neither an own state nor listed in reads.
2026-10-26 23:27:05 +01:00
mesh-tools 4af36c69a7 Merge pull request 'Delivery records are true at the source: the indexer records what it took in, the view derives what reached the machines, a settle verb for held deliveries (issue 1280)' (#80) from fix/delivery-records-true into main 2026-10-26 21:48:50 +00:00
mesh-tools bd107d2015 Merge pull request 'A version that fails on one machine while another runs it healthy is that machine's failure, and goes no further meanwhile (issue 1496, ADR 0557 §1-§3)' (#81) from fix/1496-start-health into main 2026-10-26 21:37:58 +00:00
jschoubben f007ca4227 A node that runs a version alone-failed elsewhere still gets its want at it (ADR 0557 §2, issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bd107d20154c
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking f007ca42279a on 4b36cc29a6ae | novox/mesh-delivery#81 in group fix/1496-start-health
Review of 50c4674: the bar on new wants naming F while a node fails it alone
also caught a node that runs F with no live want, the issue-1945 recovery path:
it went on neither list and its node-engine gave the module up. The bar now
holds only a node that neither wants nor runs F.
2026-10-26 22:34:25 +01:00
jschoubben 50c4674dfa While a node fails a version alone, no further node gets a want for it (ADR 0557 §2, issue 1496)
ADR 0557 version 9: a failure that is one node's holds nothing and sets nothing
back, but the version reaches no node that does not want it yet until it is
over. A node whose want does not name F gets no new want naming it (rule 2, in
words naming the failing nodes); wants already written stay.
2026-10-26 22:30:00 +01:00
jschoubben c12b870104 The indexer binds its taken state when it appears, as issue 1790 asks of every state a delivery adds: an indexer started before the bus holds it still takes merges in and says the key it did not write (issue 1280, ADR 0556 clause 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4af36c69a761
mesh/delivery-group group fix/delivery-records-true merged: every member is merged
mesh/check checking c12b870104e8 on bd107d20154c | novox/mesh-delivery#80 in group fix/delivery-records-true
2026-10-26 22:25:39 +01:00
jschoubben 92aedb9365 The first pass after a start clears failed-alone for every module and known node (ADR 0557, issue 1496)
Review round 2 of 16e4e61: gating the first-pass clear on a failed have left a
condition an earlier process raised standing for good once its node recovered
while no assigner watched. The first pass now clears it for every module and
every node the presence bucket knows, a quiet clear at the router for one that
is not open.
2026-10-26 22:21:00 +01:00
jschoubben 368204115c Pin the teller at mesh-sdk 68d65e6, the group's sdk head merged with sdk main (issue 1280) 2026-10-26 22:20:45 +01:00
jschoubben 971cdb44e2 Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-indexer/cmd/mesh-delivery-indexer/main.go
#	modules/mesh-delivery-indexer/go.mod
#	modules/mesh-delivery-indexer/go.sum
#	modules/mesh-delivery-indexer/manifest_test.go
#	modules/mesh-delivery-indexer/module.json
#	modules/mesh-delivery-view/cmd/mesh-delivery-view-tools/main.go
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
#	modules/mesh-delivery-view/internal/view/read.go
#	modules/mesh-delivery-view/manifest_test.go
#	modules/mesh-delivery-view/module.json
2026-10-26 22:20:45 +01:00
jschoubben 16e4e61a0f Only an online healthy witness, and only a first failure, keep a version from being held; failed-alone clears when a node leaves (ADR 0557, issue 1496)
Review of 2ee7589: a node that sleeps after running F healthy kept its stale
have as a witness, so F stayed unheld while it failed on the online nodes; and a
failed-alone condition was never cleared once its node left the module.

A witness must now be online by the watchman's presence. F failing on a second
node is no longer one node's failure: the hold and the set-back stand (ADR 0557
version 6, §2). A remembered failed-alone key whose node is no longer one of the
module's nodes is cleared once; the first pass after a start clears it for every
node the presence bucket knows, for a module with a failed have.
2026-10-26 22:16:38 +01:00
mesh-tools 4b36cc29a6 Merge pull request 'Placement from the catalogue: the bridge carries it, the indexer writes it, the assigner compares it (issue 1126)' (#76) from feat/1126-nodes-compose into main 2026-10-26 21:11:26 +00:00
mesh-tools 31ddd09087 Merge pull request 'The checker ends a check whose heads no candidate holds, and starts a group's check once its heads settle (issue 1890)' (#79) from fix/1890-end-superseded-checks into main 2026-10-26 21:10:56 +00:00
jschoubben 68173de8bd Pin the indexer, the teller and the view at mesh-sdk b4310df: the delivery-records work merged with sdk main (issue 1280) 2026-10-26 22:08:40 +01:00
jschoubben 960e3309bc Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 22:07:28 +01:00
jschoubben 2ee7589474 A version healthy on another node is not held for one node's failed health check (ADR 0557, issue 1496)
On ace, 2026-10-26 21:19, a newly assigned mesh-build-node failed its first
health check at 8a98386f (later 31be2f24) while shanks, g14 and novox had run it
healthy for an hour. Being first by name, ace was the canary, and the assigner
held the version and moved the three healthy nodes back.

A failed health check of the newest fingerprint now holds it only when no other
node's have names it healthy (ADR 0557 §2 keeps ADR 0429 §6's hold and set-back
for that case). Otherwise the failure is that node's (ADR 0557 §1): nothing is
held or set back, the node keeps its want, and the assigner raises the node's
condition once (failed-alone.<node>.<module>), naming the node, the module, the
fingerprint and the have's words, and clears it once when it ends; the first
pass after a start clears what a previous process may have raised.
2026-10-26 22:06:32 +01:00
jschoubben 297992a4ae Pin mesh-sdk 9796444 (l2-next: feat/1126-nodes-compose merged with sdk main #101, the standing-test race fixes) (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4b36cc29a6ae
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 297992a4ae55 on 4b36cc29a6ae | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 22:05:41 +01:00
jschoubben 3f85230417 Pin mesh-sdk 5f45175 (l2-next: feat/1126-nodes-compose merged with sdk main) after merging main (issue 1126) 2026-10-26 21:59:21 +01:00
jschoubben 43c3158bc6 Merge remote-tracking branch 'origin/main' into feat/1126-nodes-compose
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 21:57:03 +01:00
jschoubben d95e9f3229 End a check whose heads no candidate holds, and start a group's check once its heads settle (issue 1890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@31ddd090876f
mesh/check pass d95e9f322913 on d581f9659a78 | unit 191 run, 525 kept | integration 201 run, 591 kept | composition passed on novox
2026-10-26 21:44:12 +01:00
mesh-tools d581f9659a Merge pull request 'Tries record their asks; the try limit counts tries on a build node; build_usage shows capacity, hours and steps (issue 1890)' (#78) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:34 +00:00
jschoubben ea94f9234c Merge remote-tracking branch 'origin/main' into fix/capacity-follows-cpu-share
mesh/delivery merged: source.main moved to novox/mesh-delivery@d581f9659a78
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking ea94f9234c97 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
2026-10-26 21:29:44 +01:00
jschoubben c19468b04e Pin the checker and the view at mesh-sdk 805b4d9: the capacity work merged with sdk main 2026-10-26 21:29:44 +01:00
jschoubben c47afd8f4b build_usage says a seat's build hours, and when it is outside them
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group fix/capacity-follows-cpu-share checking: 1 of 3 member(s) ready
mesh/check checking c47afd8f4b88 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
A build module's capacity.<node> now carries its node's build window (the operator, 2026-10-26: g14
builds only from 18:00 to 06:00); the seat shows it, outside it says it takes nothing new, and a
value the module cannot read shows as any hour with the value and why. The checker's and the view's
SDK pin moves to mesh-sdk d00dac1.
2026-10-26 21:22:36 +01:00
jschoubben afd5a3ba07 A module a node wants and runs nothing countable of holds no head back; the reached head's event names its merges as not yet on the machines; pin mesh-sdk e44cbaf (review of issue 1280, ADR 0556 clause 2) 2026-10-26 21:15:31 +01:00
jschoubben 7dbad4c6d2 Pin mesh-sdk 33144781 (taken, reach, settle layouts) in the indexer, the view and the teller (issue 1280) 2026-10-26 21:05:29 +01:00
jschoubben 083a5042f6 The view reads the wants by module once per derivation of the reach 2026-10-26 21:03:56 +01:00
jschoubben 351ca7c290 The teller checks a settle ask's warrant against mesh-delivery.settle, the act its options bind (issue 1280, ADR 0556 clause 5) 2026-10-26 20:58:20 +01:00
jschoubben f907be2c14 The view derives per repository up to which merge every node runs it, keeps it as its state reach and says each reached head, so a fix on the machines reads delivered (issue 1280, ADR 0556 clauses 2 and 3) 2026-10-26 20:56:12 +01:00
jschoubben 6201f58c20 The indexer says per repository the last merge it took in, so the view knows its modules speak for that merge (issue 1280, ADR 0556 clause 1) 2026-10-26 20:49:00 +01:00
jschoubben 8fee764c73 Pin mesh-sdk feat/1126-nodes-compose 03dce12, the group's head: the checker refuses a pin that is neither trunk nor the group's head (issue 1126)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 8fee764c734e on d581f9659a78 | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:35:45 +01:00
mesh-tools 3dd94f714d Merge pull request 'The checker fails any repository's change that adds a binary, in any commit (issue 1577)' (#77) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:27 +00:00
jschoubben c716670be0 The checker pins the SDK of review round 1, and its hid-the-failure cases join the older build module's test (issue 1901) 2026-10-26 20:19:07 +01:00
jschoubben 352066dde6 Review of L2: the assigner binds the indexer's placement state when it appears, with no restart (D1); the three new unit tests folded into existing ones, so no count rises (D2); pin mesh-sdk 3968849 (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 352066dde611 on 3dd94f714d6e | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:12:00 +01:00
jschoubben d6bd31a166 Say the try limits as what they are: a check when a try is asked
Tries let in while waiting may later all run at once; each node's build capacity bounds what runs
on it (review of issue 1890's try cap).
2026-10-26 20:11:08 +01:00
jschoubben c0a4b139a2 A try queued behind the build seat's checks no longer blocks the next try
Only tries a build node holds count toward the 4 at once; tries waiting for a node are bounded
apart at 8, and the refusal names both counts (issue 1890: at 19:53 four tries queued behind 44
checks for up to 47 minutes refused every further try).
2026-10-26 20:05:51 +01:00
jschoubben 6d60953f98 A running try names its asks and where each stands; build_usage says a seat's CPU-bound capacity
The checker wrote a try's asks only at its end, so a 31-minute try could not be told queued from
running (issue 1890). The try's record now names each ask as it is put, then the node that took
it, its state and the phase its step says, written on the step key's own changes. build_usage
names a running try's asks and takes a build module's capacity.<node> (configured bounded by its
node's build CPU, issue 1806's follow-up) when it is current with mesh-config.
2026-10-26 19:58:39 +01:00
jschoubben 17ed06ee4d Pin mesh-sdk's fix/1901-failure-contract head 81ec7bd7 (issue 1901) 2026-10-26 19:50:47 +01:00
jschoubben 9427dac823 The checker says loudly when a build module hid a failure's cause (issue 1901)
A failed test with no cause, or a passing line as its cause, was worded as if it were the failure: a try of mesh-catalog
failed with a passing '✔' line of the first test file. Under mesh-sdk's failure contract the verdict's words and its
status line now start with 'the build module hid the failure (<module> at <version>)', and the check still fails.
2026-10-26 19:50:05 +01:00
jschoubben d74e082155 Checker refuses a program a commit of the branch added and a later one removed: it stays in history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-delivery@3dd94f714d6e
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking d74e0821552c on df8b9ac3b1b5 | novox/mesh-delivery#77 in group fix/1577-checker-refuses-binaries
2026-10-26 19:47:53 +01:00
jschoubben 2e0013ec25 The assigner's wants test goes on to placement: a placed node's want names the build alone and its own want says it composes its own (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2e0013ec2500 on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:44:55 +01:00
jschoubben 2493aa6529 Pin mesh-sdk feat/1126-nodes-compose 9713c30: placement carries a settings-form module's values (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2493aa65299e on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:40:38 +01:00
jschoubben 85c89b16ef The grants part composes what the catalogue's placement places beside the assignments, and holds a seat held once for the mesh where placement says (issue 1126, ADR 0549 §3, G5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:33:27 +01:00
jschoubben 4f75925410 Nodes compose: placement from the catalogue, node records, self wants, own and pair credentials, core boxes and the authority from the vault (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
One commit for the branch feat/1126-nodes-compose of mesh-delivery: its 16 commits squashed so its history does not carry the 17.8 MB compiled mesh-delivery-assigner a commit added by mistake and a later one removed (issue 1577). The tree equals the former head ff6c038d. The operator allowed this rewrite of the feature branch on 2026-10-26.
2026-10-26 19:30:18 +01:00
jschoubben 820c458f4c The captured clean fetches carry the programs a current build module answers, so lines asserted whole do not warn (issue 1577) 2026-10-26 19:16:25 +01:00
jschoubben dc9d65524d A check judged by a build module that reports no programs says so in its status line, still passing: the refusal did not run (issue 1577) 2026-10-26 19:08:00 +01:00
jschoubben 1a4c788ed7 Checker fails a change whose fetch reports a compiled program, and warns when a build module reports none: build output kept getting committed (issue 1577) 2026-10-26 19:01:40 +01:00
mesh-tools df8b9ac3b1 Merge pull request 'Checker, builder and indexer bind states created after they start (issue 1790)' (#75) from fix/1790-state-appears into main 2026-10-26 14:47:40 +00:00
jschoubben fcb1e40582 Pin mesh-sdk fix/1790-state-appears at 65ee43f: a refused hearing of a bucket's creation is said and heard again at a reconnect (issue 1790, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@df8b9ac3b1b5
mesh/delivery-group group fix/1790-state-appears merged: every member is merged
mesh/check being carried to fcb1e4058262 on df8b9ac3b1b5
2026-10-26 15:25:53 +01:00
jschoubben 4973372612 Merge remote-tracking branch 'origin/main' into fix/1790-state-appears 2026-10-26 15:25:06 +01:00
jschoubben 3f6c085e8a Pin mesh-sdk fix/1790-state-appears at 3ac644b (issue 1790) 2026-10-26 15:14:00 +01:00
jschoubben 040e3097b5 checker, builder, indexer: a state of their own not on the bus at start is bound when it appears; mesh-grants composes each principal's hearing of its buckets' creation (issue 1790)
A delivery restarted the checker and the builder before the grants part created their new state hearing;
OptionalBucket answered nil and the hearing was never kept. They, and the builder's reasks and queues and the
indexer's takeins, now open their own states through mesh-sdk process.AwaitBucket (mesh-sdk
fix/1790-state-appears d515c71), which binds on the bus's STREAM.CREATED advisory and writes the latest kept
value. mesh-grants takes the same SDK, whose composer grants that advisory.
2026-10-26 15:13:00 +01:00
mesh-tools feef96ed16 Merge pull request 'build_usage: how much each build node is used and whether it may build, in one call' (#74) from feat/build-usage-tool into main 2026-10-26 14:09:32 +00:00
jschoubben 4f722899df View: build_usage does not answer eligibility it cannot read; a resolved capacity key with no readable value is the puller's 0
mesh/delivery merged: source.main moved to novox/mesh-delivery@feef96ed160e
mesh/check checking 4f722899df25 on feef96ed160e
A capacity this tool cannot read is unknown to it, not 0: the seat says eligible_known false and the node not known.
A key mesh-config holds without a readable value is what the puller reads as 0, and was dropped silently.
2026-10-26 14:58:47 +01:00
jschoubben 166a918b43 View: build_usage counts keys a build module ended for being left running apart, lists every node a capacity names, and says capacity unknown when mesh-config cannot be read
Since mesh-sdk #90 a build module ends a left-running key as failed at the moment it notices; counted as finished,
such a step put hours into the medians and the time at capacity. The view takes mesh-sdk's step bound and restart
reason from the SDK it now pins.
2026-10-26 14:58:47 +01:00
jschoubben 1b4c860f63 Checker: its announcement test expects tries readable, as the delivery view now reads them 2026-10-26 14:58:47 +01:00
jschoubben 3b6945b2db View: build_usage answers how the build nodes are used in one call; the checker's tries name their asks
The operator's question (how much are the build nodes used, is ace eligible) took six calls over four tools.
build_usage reads it from the bus's states: seats held (have), capacity (mesh-config resolved), pauses, the
build modules' steps by kind with left-running keys, the last hour and day with total and median time and time
at capacity, the running tries, and the backlog the view program reads from the seats' streams. A try now names
the asks it put, and the checker announces its tries readable, so the view tells a try's asks from a check's.
2026-10-26 14:58:41 +01:00
mesh-tools 4174e23c04 Merge pull request 'The checker and the builder end or re-ask what the build seat gave up, and a try can be ended by hand (issue 734)' (#73) from fix/734-lost-ask into main 2026-10-26 13:54:17 +00:00
jschoubben 3993175e45 Checker, builder and grants part: pin mesh-sdk main, which holds both the dead-letter hearing (#92) and the placement (#91) (issue 734)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4174e23c0492
mesh/delivery-group group fix/734-lost-ask merged: every member is merged
mesh/check checking 3993175e457f on 4174e23c0492
2026-10-26 14:45:33 +01:00
jschoubben 60c7ffc981 Checker, builder and grants part: hear DEAD_LETTERS again until heard, say when they do not, and compose the same grant (issue 734, review round 1)
mesh-grants moves to the same mesh-sdk (3203358c), so its shadow user list grants the checker and the builder the
DEAD_LETTERS reads the controller's list will, and the take-over is not blocked by a difference. The checker and the
builder hear DEAD_LETTERS through buildrun.DeadLetterHearing: in the background, again at every reconnect and every ask
waited for while deaf, and keep whether they hear it in their new state hearing, which checker_status and builder_stuck
say as dead_letters. checker_try says that a try ended by the not-taken bound leaves its ask in the stream.

Not done as asked: a module health tool. The catalogue refuses a tool check on a module with no runtime of its own and no
other check beside it (ADR 0227 rule 8, ADR 0366 §1), so the deaf state is said in the status tools and the log.
2026-10-26 14:44:53 +01:00
jschoubben 33de9dde4f Checker and builder: wait on the build seats through mesh-sdk's LostAskWaiter, so an ask the seat gave up on ends every asker's wait (issue 734)
The waiter moved to mesh-sdk buildrun (fix/734-lost-ask, d96b9fd6), where every asker of the build seats shares it.
The builder hears DEAD_LETTERS for the go, node and image seats and asks a given-up part of a build again at once, with
the seat's words as why, instead of waiting for the check bound. The checker keeps what it does with a lost ask (a try
ends as error, a check is asked again) and its verdict words.
2026-10-26 14:44:52 +01:00
jschoubben 6b85220fa4 Checker: an ask the build seat gives up on ends its try as error and asks a check again; a try no node takes ends at the step bound; checker_try_end ends one by hand (issue 734)
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
2026-10-26 14:44:40 +01:00
mesh-tools 1cbae7d41c Merge pull request 'The checker asks the packages a node could not test again, on a node that has what they need (issue 890, ADR 0552)' (#72) from feat/890-placement into main 2026-10-26 13:40:45 +00:00
mesh-tools b3a4b09f5b Merge pull request 'The indexer takes in a registered module whose repository became readable, and serves indexer_take_in (issue 1945)' (#71) from fix/indexer-takes-in into main 2026-10-26 13:39:13 +00:00
mesh-tools c8b26735c8 Merge pull request 'The assigner says when it holds the bus back instead of skipping it silently (issue 1945)' (#70) from fix/bus-bar-loud into main 2026-10-26 13:13:22 +00:00
jschoubben 907e092abc mesh-delivery-indexer reads unread registered modules again in one background run, so a forge timeout holds no merged message or source revision past its ack wait (issue 1945, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b3a4b09f5b6f
mesh/check being carried to 907e092abc45 on b3a4b09f5b6f
The take-in tool's bound now says the take-in is still running and where its answer will be.
2026-10-26 14:11:12 +01:00
jschoubben 3073495a06 checker: pin mesh-sdk 33395f68, feat/890-placement merged with main (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@1cbae7d41cd0
mesh/delivery-group group feat/890-placement merged: every member is merged
mesh/check checking 3073495a0670 on 1cbae7d41cd0
2026-10-26 14:06:10 +01:00
jschoubben 16c411e436 Merge remote-tracking branch 'origin/main' into feat/890-placement 2026-10-26 14:05:40 +01:00
jschoubben 2d841870a8 Assigner: only a put still on its steps is named when a node leaves neither list (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c8b26735c8cf
mesh/check pass 2d841870a873 on c8b26735c8cf | carried from 1d416827157d | unit 185 run, 518 kept | integration 195 run, 584 kept | composition passed…
Review round 2: an ended put stays in the bucket and owns no want.
2026-10-26 14:03:48 +01:00
jschoubben 7f8ce7f5c0 checker: the placed ask names the unmet packages, so a package without a kept verdict is not run or counted twice (issue 890, ADR 0552 §4; review round 1). Pins mesh-sdk 45564e69. 2026-10-26 14:02:16 +01:00
jschoubben 07b5f13474 mesh-delivery-indexer reads a registered module again once its repository becomes readable, and takes one in on ask (issue 1945)
A registered module whose repository the forge answered 404 (car-hunter, snake, before the forge's account could read
them) was read again only when the indexer started again or the controller wrote a new build record of it, so the grants
part's composed user list stayed short of their subjects and the take-over was refused. A failed read now keeps the
module as unread, read again on the next source revision and the next merged message (no timer of its own), and the new
tool indexer_take_in (the tools bundle mesh-delivery-indexer-tools, through the indexer's takeins state, as the
builder's builder_ask_again) takes one in now and answers what it stored or why not.
2026-10-26 14:01:34 +01:00
jschoubben f4aac599ca Assigner: name a running put when a node leaves neither list, and say the bus bar in plain words (issue 1945)
Review round 1: while an operator's put runs through its steps it owns the want, so the clear said
'the node does not run it' of a node that runs the bus.
2026-10-26 14:00:32 +01:00
jschoubben 2d63f0ba62 Assigner: say the bus bar instead of skipping it silently, and clear a neither condition only in true words (issue 1945)
A bus whose move only the operator's put may make (ADR 0546 §8) got no want and nothing in the outcome:
nats on novox, running an old-form build, was on no list while the pass cleared its condition and logged
that it had a want again. The bar now names its holding, puts a node without a live want on neither list
(its condition needs the operator: assigner_put), and a neither condition is cleared as 'has a want again'
only when a live want exists.
2026-10-26 13:55:42 +01:00
mesh-tools 1d41682715 Merge pull request 'mesh-grants says loudly when it holds the user list back, and clears it after a restart (issue 1945)' (#69) from fix/grants-hold-loud into main 2026-10-26 12:44:36 +00:00
jschoubben 705765d113 checker: a placed ask that every node declines for longer than its bound (30 min) fails the check naming the need (issue 890, ADR 0552 review) 2026-10-26 13:42:46 +01:00
jschoubben 16a97ed65d checker: ask the packages a node left unmet again of a node that has what their tests need, and put a declined ask again (issue 890)
A check that landed on ace passed with mesh-build-node's integration tests skipped. The build module now leaves such a
package unmet; the checker keeps the passes and asks the levels from the first unmet one with the need named, putting the
ask again under a new id when a node lacking it declines it. Re-pins mesh-sdk feat/890-placement.
2026-10-26 13:40:00 +01:00
mesh-tools ecf532af0e Merge pull request 'checker_status reads the node-engine at its running commit, as the checker decides (issue 568)' (#68) from fix/status-and-holds into main 2026-10-26 12:39:56 +00:00
jschoubben ce600df98a Say the held-list state on a part's first pass, so a warning raised before a restart is cleared
mesh/delivery merged: source.main moved to novox/mesh-delivery@1d416827157d
mesh/check being carried to ce600df98afc on 1d416827157d
2026-10-26 13:37:31 +01:00
jschoubben b4a6a34705 mesh-grants: a user list held back after take-over raises the warning mesh-grants.user-list-held and clears it when the list advances, so the hold is not only in a tool's output (issue 1945) 2026-10-26 13:36:39 +01:00
jschoubben a7e51b416b checker_status reads the node-engine at its running commit, as the checker's decision does, so it stops listing every candidate as waiting for a trunk move (issue 568)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ecf532af0e85
mesh/check being carried to a7e51b416bc1 on ecf532af0e85 | from 71f786f14662
2026-10-26 13:25:18 +01:00
mesh-tools 71f786f146 Merge pull request 'mesh-grants: take-over and hand-back by the operator's Telegram Approve, the sealed list, and every grant the loaded list holds (issue 1945)' (#67) from feat/1945-user-list-switch into main 2026-10-26 12:21:50 +00:00
mesh-tools 848c6fdc56 Merge pull request 'Checker: one check per set of heads, a trunk or sibling move carries the verdict, a merge turn holds a claim (ADR 0548, issue 568)' (#65) from feat/548-one-check into main 2026-10-26 12:01:32 +00:00
mesh-tools ead9ee2d13 Merge pull request 'Indexer: index a module registered from any branch, by its registered ref, never a newer pull request's (issue 1945)' (#66) from fix/indexer-registered-modules-2 into main 2026-10-26 12:01:24 +00:00
mesh-tools 4861d1f78d Merge pull request 'Checker: keep no package whose tests skipped, so a skipped pass is never reused as proof (issue 890)' (#64) from fix/checker-skips-not-kept into main 2026-10-26 12:01:19 +00:00
jschoubben 5c76c9d3ac mesh-grants: a list in force with no users kept is checked against the users the bus loaded, so the first list after take-over keeps every user (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@71f786f14662
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check being carried to 5c76c9d3ac84 on 71f786f14662 | novox/mesh-delivery#67 in group feat/1945-user-list-switch
2026-10-26 12:54:55 +01:00
jschoubben 699cbfb7c1 Indexer: a registered module takes its registered ref, not a newer branch's record; no read at an older commit after a failure
mesh/delivery merged: source.main moved to novox/mesh-delivery@ead9ee2d138d
mesh/check checking 699cbfb7c131 on ead9ee2d138d
2026-10-26 12:52:29 +01:00
jschoubben 319d56be7e Indexer: a registered module is read from every repository it was built from, whatever the branch, and at the registered commit when the head is gone 2026-10-26 12:45:35 +01:00
mesh-tools 11d3678984 Merge pull request 'Indexer: index a module registered from its own repository, so the grants part composes its grants (issue 1945)' (#63) from fix/indexer-registered-modules into main 2026-10-26 11:33:41 +00:00
jschoubben f55f150896 mesh-grants: after take-over the list in force is not replaced by one composed past a failed read or one dropping its users for want of a hash; the switch's effect says when the bus node does not watch the key (review of the switch, issue 1945). Pins mesh-sdk c2a410bb. 2026-10-26 12:31:31 +01:00
jschoubben e1f1b5d18b Checker: a package with skipped tests is withheld from the kept verdicts, not the whole result (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4861d1f78d28
mesh/delivery-group group fix/checker-skips-not-kept merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:29:14 +01:00
jschoubben f0fe454145 Checker: a result with skipped tests keeps no verdict, so a skipped pass is run again (issue 890) 2026-10-26 12:23:57 +01:00
jschoubben 4026f6c188 checker: a check is never held by a turn its own heads wait in, the node-engine read at its running commit is named and carried when it moves, and a gate bucket made after the start is opened before the next claim (ADR 0548 §3, §6; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@848c6fdc5648
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-delivery#65 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:14:50 +01:00
jschoubben 29cfb8c177 Pin mesh-sdk at the group's head 83ee7769 (issue 1945) 2026-10-26 12:06:58 +01:00
jschoubben a092ee9ff9 mesh-grants writes the user list sealed to the bus node and names it in user-list.sealed; the switch turns that key; take-over state says switched only once the bus node loaded it (ADR 0547 clauses 7, 8; issue 1945)
The bus node can only take over a list that exists: each pass composes the accounts block with the vault's hashes, seals
it to the key the bus node says in its have, puts it in the part's own object bucket by digest and writes the key with
digest, revision, user count, users left out and which list to follow. take-over is refused, naming why, while no list is
sealed to the bus node's current key or the loaded list does not grant the bus node its read.

Review of fix/1945-takeover-blockers: no responders on the router's record is an error, not 'no record'; an answer whose
record cannot be reached is not closed; an ask that failed to publish is taken back at the router; an error acting on an
answer is logged. Pins mesh-sdk 175a5567.
2026-10-26 11:59:41 +01:00
jschoubben 6c3cbee35c checker: pin mesh-sdk at the group's head f5c233a9288c, which holds the merge turns and the carried verdict (ADR 0548 §10; issue 568) 2026-10-26 11:45:13 +01:00
jschoubben 646f83642a Indexer: read a registered module at the source's head when it names one, else at the registered commit
mesh/delivery merged: source.main moved to novox/mesh-delivery@11d3678984af
mesh/check pass 646f83642a01 on 11d3678984af | unit 171 run, 502 kept | integration 181 run, 568 kept | composition passed in 39ms on novox
2026-10-26 11:41:46 +01:00
jschoubben d9bfba1e62 Indexer: read one directory of a registered module's repository, leave a repository the source names to its merges, accept master as a trunk 2026-10-26 11:36:22 +01:00
jschoubben f1bb7747ae checker: one integration test through the real bus proves a merge turn holds a check off until it goes and a carry of unchanged digests runs nothing; it extends the equal-inputs test, so the count stays (ADR 0548 §3, §5; issue 568 D7) 2026-10-26 11:36:15 +01:00
jschoubben d2eb3e5ecf checker: the line of a carried verdict names the head on the new trunk, then where it was carried from; a verdict being carried is pending and says so (ADR 0548 §6, §8; issue 568 D6) 2026-10-26 11:35:16 +01:00
jschoubben d91ea41f64 checker: only a head git refused gets a whole check after a trunk move; a conflicting carry fails naming the files and asks nothing (ADR 0548 §7; issue 568 D5) 2026-10-26 11:34:08 +01:00
jschoubben 5fed429827 checker: a verdict names the siblings it read at their trunk, and a sibling's move is carried as a trunk move is (ADR 0548 §6; issue 568 D4) 2026-10-26 11:33:35 +01:00
jschoubben e2c41ffa5a checker: a carry keeps the composition it passed while no composed fingerprint changed, and composes again otherwise (ADR 0548 §5; issue 568 D3) 2026-10-26 11:32:31 +01:00
jschoubben 13e4ed8515 checker: a trunk move never ends a check; once it ended, its pass is carried and names the commits it was carried from, and a failed carry names the move (ADR 0548 §1, §5; issue 568 D2) 2026-10-26 11:30:46 +01:00
jschoubben c66fe11e16 checker: claim no check of a repository an approved merge holds a turn on, and claim it when the turn goes (ADR 0548 §3; issue 568 D1) 2026-10-26 11:29:25 +01:00
jschoubben c44b9f91cd Indexer: gofmt 2026-10-26 11:23:27 +01:00
jschoubben 82da252cf0 Indexer: index the manifests of modules registered from their own repositories, so the grants part has no gap for them 2026-10-26 11:23:21 +01:00
mesh-tools 32bc4f3c48 Merge pull request 'Checker: read every test result of a level and name every failure in the verdict, the try and the status line (issue 1921)' (#62) from fix/1921-every-failure-r into main 2026-10-26 10:18:11 +00:00
mesh-tools ed1125b66d Merge pull request 'Assigner: take a stale ask away only once its held condition's clear was said, so a failed clear is retried (issue 1906)' (#61) from fix/1906-clear-retried into main 2026-10-26 10:09:42 +00:00
jschoubben 366bc750da Assigner: clear a stale hold's condition before taking its ask away, so a failed clear is retried at the next pass (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ed1125b66d91
mesh/check pass 366bc750da87 on ed1125b66d91 | unit 168 run, 489 kept | integration 21 run, 712 kept | composition passed on ace
2026-10-26 11:03:09 +01:00
jschoubben 04a56b7aff Name every failed test in a check's verdict and a try's failed list (issue 1921)
mesh/delivery merged: source.main moved to novox/mesh-delivery@32bc4f3c48b2
mesh/delivery-group group fix/1921-every-failure-r merged: every member is merged
mesh/check checking 04a56b7aff66 on 32bc4f3c48b2
The checker stopped reading a level's test results at the first that did
not pass and kept only its first failure, so a check named one fault per
round. It now reads and counts every result and writes every failed test
with its first line into the verdict's Failures and the try's failed
list (at most 50, then how many more); Failed keeps the first one's
words whole (issue 1901). Pins mesh-sdk fix/1921-every-failure.
2026-10-26 10:58:13 +01:00
191 changed files with 18460 additions and 2903 deletions
+3
View File
@@ -1 +1,4 @@
vendor/
# A compiled program never enters a commit (issue 1577): `go build .` in a command directory leaves one there.
modules/*/cmd/*/mesh-*
!modules/*/cmd/*/mesh-*.go
+7
View File
@@ -0,0 +1,7 @@
# The modules of this repository that still read the controller's assignments (mesh-controller.assignments), each with
# why. mesh-grants' test (TestGrants, "assignments readers") refuses any other module.json that reads them, and an entry
# here whose module no longer does: the list only shrinks, and it is empty in the first change after the controller's
# lease is written to a holder that is no controller (novox/hq ADR 0549 §3, §6; issue 774). Each module named here
# uses the assignments only while the controller holds the lease (buscompose.LeaseHandedAway), placement alone after it.
mesh-delivery-assigner unioned with the placement until the lease write (assign.Placed with handedAway); placement alone after
mesh-grants unioned with the placement until the lease write (ReadInputs); placement alone after
@@ -9,6 +9,7 @@ import (
"io"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
@@ -17,9 +18,11 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
@@ -69,7 +72,11 @@ func startBed(t *testing.T, leaveOut string) *bed {
if name == leaveOut {
continue
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 5})
cfg := jetstream.KeyValueConfig{Bucket: name, History: 5}
if name == delivery.PlacementPulsesBucket {
cfg.History, cfg.LimitMarkerTTL = 1, time.Minute // a time to live per key, as the controller makes it (key-ttl)
}
kv, err := js.CreateKeyValue(ctx, cfg)
if err != nil {
t.Fatal(err)
}
@@ -128,9 +135,7 @@ func (b *bed) startStoppable(t *testing.T) (<-chan error, func()) {
// 1297/B2b: started whole on its credential, the assigner says on the bus every state its manifest declares, each readable,
// having opened every bucket it names. The states are read from module.json, so a state added there is expected here too.
func TestTheAssignerStartsAndSaysWhatItKeeps(t *testing.T) {
b := startBed(t, "")
done := b.start(t)
func saysWhatItKeeps(t *testing.T, b *bed, done <-chan error) {
events, err := b.js.Stream(b.ctx, "EVENTS")
if err != nil {
t.Fatal(err)
@@ -172,6 +177,9 @@ func TestTheAssignerStartsAndSaysWhatItKeeps(t *testing.T) {
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets {
if missing == delivery.PlacementBucket || sourceOfGrants(missing) {
continue // it starts without it, below (the grants' source: main_test.go)
}
b := startBed(t, missing)
select {
case err := <-b.start(t):
@@ -218,6 +226,34 @@ func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
t.Fatal("the put was not taken further once its bucket was made")
}
}
// Without the seat's placement state it starts all the same, and once the bucket is made (the controller makes
// it from the manifest) it is read with no restart: a module only the placement names is
// said as placed by one source only.
pb := startBed(t, delivery.PlacementBucket)
pdone := pb.start(t)
select {
case err := <-pdone:
t.Fatalf("without %s the assigner stopped: %v", delivery.PlacementBucket, err)
case <-time.After(2 * time.Second):
}
placement, err := pb.js.CreateKeyValue(pb.ctx, jetstream.KeyValueConfig{Bucket: delivery.PlacementBucket, History: 5})
if err != nil {
t.Fatal(err)
}
pl, _ := delivery.EncodePlacement(delivery.Placement{Node: "n1", Modules: []delivery.PlacedModule{{Module: "web"}}})
plkey, _ := delivery.PlacementKey("n1")
if _, err := placement.Put(pb.ctx, plkey, pl); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(15 * time.Second); ; time.Sleep(200 * time.Millisecond) {
if c, ok := pb.conditionsSaid(t)[delivery.AssignerModule+".placement.n1.web"]; ok && c.State == asks.StateRaised {
break
}
if time.Now().After(deadline) {
t.Fatal("the placement state made after the start was not read")
}
}
}
// putJSON writes a value into one of the bed's buckets.
@@ -406,6 +442,32 @@ func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
if !cleared {
t.Errorf("the condition of web on neither list on g14 was not cleared once g14 got its want")
}
// Issue 1126 (ADR 0547 §2): the catalogue's placement reaches the indexer's state naming web on every node the
// assignments name it on. Each node then composes its own: its want names the build alone (Self, no composition), and
// its own want says it composes its own, which the node agent reads (mesh-host's TestNats...ComposesAndAppliesItsOwn).
for _, node := range []string{"ace", "g14", "shanks"} {
pl, err := delivery.EncodePlacement(delivery.Placement{Node: node, Modules: []delivery.PlacedModule{{Module: "web"}}})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PlacementKey(node)
b.putJSON(t, delivery.PlacementBucket, pkey, pl)
}
if w := b.wantNow(t, "ace", "web", "ace's want once placed: the build alone, composed by the node", func(w delivery.Want) bool { return w.Self }); w.Composed != "" || w.FP != fNew {
t.Errorf("ace's want once placed = %+v", w)
}
nkey, _ := delivery.WantNodeKey("ace")
var self bool
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && !self; time.Sleep(100 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, nkey); err == nil {
nw, err := delivery.DecodeNodeWant(e.Value())
self = err == nil && nw.Self
}
}
if !self {
t.Error("ace's own want does not say it composes its own once placed")
}
}
// 1297/B9 (the assigner's half, ADR 0528 clause 6): started whole, the assigner holds a fingerprint whose health check
@@ -943,3 +1005,196 @@ func TestAHoldWrittenAgainAfterALiftAndARestartIsAskedAgain(t *testing.T) {
}
}
}
// bedStore is the runtime's states over the bed's buckets, as the seat's verbs read and write them: the assigner's own by
// their local names, another module's `<module>.<state>`.
type bedStore struct{ b *bed }
func (s bedStore) kv(state string) jetstream.KeyValue {
if strings.Contains(state, ".") {
return s.b.buckets[strings.Replace(state, ".", "_", 1)]
}
return s.b.buckets[delivery.AssignerModule+"_"+state]
}
func (s bedStore) get(state, key string) ([]byte, uint64, bool, error) {
e, err := s.kv(state).Get(s.b.ctx, key)
if err != nil {
return nil, 0, false, nil
}
return e.Value(), e.Revision(), true, nil
}
func (s bedStore) put(state, key string, value []byte) error {
_, err := s.kv(state).Put(s.b.ctx, key, value)
return err
}
func (s bedStore) keys(state string) ([]string, error) {
keys, err := s.kv(state).Keys(s.b.ctx)
if err != nil && strings.Contains(err.Error(), "no keys") {
return nil, nil
}
return keys, err
}
func (s bedStore) del(state, key string) error { return s.kv(state).Delete(s.b.ctx, key) }
func (s bedStore) history(string, string) ([]stdio.StateRevision, error) { return nil, nil }
// 1126 (ADR 0567 §2-§6), the seat mesh-placement whole on the bus: a proposal moving web from the anchor to the laptop is
// proposed through the verb, the assigner finds both nodes must answer, they answer the same in their haves, the
// operator is asked and approves, and each key is written at a new revision while placement-changed is heard on the
// assigner's stream naming both nodes. A second proposal laid over the old revision of a key moved since is laid again,
// and declined, writes nothing and is withdrawn.
func TestAProposalIsAnsweredApprovedAndWritten(t *testing.T) {
b := startBed(t, "")
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: "MOD_" + delivery.AssignerModule + "_placement",
Subjects: []string{process.ModuleStreamSubject(delivery.AssignerModule, ">")}}); err != nil {
t.Fatal(err)
}
heard := make(chan delivery.PlacementEvent, 16)
nc, err := nats.Connect(b.srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
if _, err := nc.Subscribe(process.ModuleStreamSubject(delivery.AssignerModule, delivery.PlacementChangedEvent), func(m *nats.Msg) {
var e delivery.PlacementEvent
if json.Unmarshal(m.Data, &e) == nil {
heard <- e
}
}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: node}})
b.putJSON(t, delivery.HaveBucket, node+".node", rec)
}
b.putJSON(t, delivery.ModulesBucket, "web", []byte(`{"v":1}`))
b.putJSON(t, delivery.ModulesBucket, "shell", []byte(`{"v":1}`))
place := func(node string, modules ...string) uint64 {
p := delivery.Placement{Node: node, Modules: []delivery.PlacedModule{}}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
raw, _ := delivery.EncodePlacement(p)
key, _ := delivery.PlacementKey(node)
rev, err := b.buckets[delivery.PlacementBucket].Put(b.ctx, key, raw)
if err != nil {
t.Fatal(err)
}
return rev
}
anchorRev, laptopRev := place("anchor", "web"), place("laptop", "shell")
bound := judge.AnswerBound
t.Cleanup(func() { judge.AnswerBound = bound })
judge.AnswerBound = 2 * time.Second
saysWhatItKeeps(t, b, b.start(t))
verbs := newSeatTools(bedStore{b}, func(string, any) (json.RawMessage, error) { return json.RawMessage(`{"ok":true}`), nil })
propose := func() delivery.PlacementProposal {
t.Helper()
got, err := verbs.doPropose(map[string]any{"why": "move web", "changes": `[{"op":"take-off","node":"anchor","module":"web"},` +
`{"op":"place","node":"laptop","module":"web"}]`}, "operator@laptop")
if err != nil {
t.Fatal(err)
}
return got.(proposed).Proposal
}
proposalNow := func(id, what string, ok func(delivery.PlacementProposal) bool) delivery.PlacementProposal {
t.Helper()
var last delivery.PlacementProposal
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.PlacementProposalsBucket].Get(b.ctx, id); err == nil {
if p, err := delivery.DecodePlacementProposal(e.Value()); err == nil {
if last = p; ok(p) {
return p
}
}
}
}
t.Fatalf("the proposal %s never was %s: %+v", id, what, last)
return last
}
answerAll := func(p delivery.PlacementProposal) {
t.Helper()
for _, node := range p.Must {
key, _ := delivery.PlacementCandidateKey(p.ID, node)
e, err := b.buckets[delivery.PlacementCandidatesBucket].Get(b.ctx, key)
if err != nil {
t.Fatalf("no candidate %s: %v", key, err)
}
have, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: node}, Candidates: map[string]delivery.CandidateAnswer{
p.ID: {Proposal: p.ID, Revision: e.Revision(), Verdict: delivery.CandidateSame, Same: []string{"node"}}}})
b.putJSON(t, delivery.HaveBucket, node+".node", have)
}
}
p := propose()
p = proposalNow(p.ID, "told who must answer", func(p delivery.PlacementProposal) bool { return len(p.Must) > 0 })
if !slices.Equal(p.Must, []string{"anchor", "laptop"}) {
t.Fatalf("must answer = %v, want both nodes", p.Must)
}
answerAll(p)
p = proposalNow(p.ID, "asked", func(p delivery.PlacementProposal) bool { return p.State == delivery.ProposalAsked })
akey, _ := delivery.PlacementAskKey(p.ID)
b.answer(t, akey, assign.OptionYes)
select {
case e := <-heard:
if e.Proposal.ID != p.ID || len(e.Nodes) != 2 || e.Nodes[0].Before != anchorRev || e.Nodes[1].Before != laptopRev ||
e.Nodes[0].After <= anchorRev || e.Warrant == nil || !slices.Equal(e.Nodes[0].Removed, []string{"web"}) {
t.Errorf("placement-changed = %+v", e)
}
case <-time.After(15 * time.Second):
t.Fatal("no placement-changed heard")
}
for node, want := range map[string][]string{"anchor": {}, "laptop": {"shell", "web"}} {
key, _ := delivery.PlacementKey(node)
e, err := b.buckets[delivery.PlacementBucket].Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
if got, err := delivery.DecodePlacement(e.Value()); err != nil || !slices.Equal(got.Assigned(), want) {
t.Errorf("%s = %v, %v, want %v", node, got.Assigned(), err, want)
}
}
if b.hasKey(delivery.PlacementProposalsBucket, p.ID) {
t.Error("an applied proposal stays in the state")
}
// A proposal whose key moves before its warrant is laid again; then declined, it writes nothing and ends.
q, err := verbs.doPropose(map[string]any{"why": "back", "changes": `[{"op":"take-off","node":"laptop","module":"web"}]`}, "op")
if err != nil {
t.Fatal(err)
}
moved := q.(proposed).Proposal
place("laptop", "shell", "web") // another writer moved the key
relaid := proposalNow(moved.ID, "laid again", func(p delivery.PlacementProposal) bool {
return strings.Contains(p.Words, "laid again") && len(p.Must) > 0
})
answerAll(relaid)
proposalNow(moved.ID, "asked", func(p delivery.PlacementProposal) bool { return p.State == delivery.ProposalAsked })
akey, _ = delivery.PlacementAskKey(moved.ID)
b.answer(t, akey, assign.OptionNo)
for deadline := time.Now().Add(15 * time.Second); b.hasKey(delivery.PlacementProposalsBucket, moved.ID); time.Sleep(50 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("a declined proposal was not ended")
}
}
key, _ := delivery.PlacementKey("laptop")
if e, _ := b.buckets[delivery.PlacementBucket].Get(b.ctx, key); e == nil || !strings.Contains(string(e.Value()), "web") {
t.Error("a declined proposal wrote its key")
}
// ADR 0567 §4: a node that does not answer fails the verdict once its pulse key's time to live ends, heard on the bus
// with no timer of the assigner's, naming it.
q, err = verbs.doPropose(map[string]any{"why": "again", "changes": `[{"op":"take-off","node":"laptop","module":"web"}]`}, "op")
if err != nil {
t.Fatal(err)
}
silent := q.(proposed).Proposal
failed := proposalNow(silent.ID, "failed for a silent node", func(p delivery.PlacementProposal) bool { return p.State == delivery.ProposalFailed })
if failed.Verdict == nil || !strings.Contains(strings.Join(failed.Verdict.Fails, "; "), "laptop (offline) did not answer") {
t.Errorf("the silent node's verdict = %+v", failed.Verdict)
}
}
@@ -14,7 +14,7 @@ import (
"log"
"os"
"os/signal"
"strings"
"slices"
"syscall"
"time"
@@ -23,6 +23,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
@@ -30,7 +31,7 @@ import (
)
func main() {
if os.Getenv("MESH_BROKER_FILE") == "" {
if os.Getenv(process.BrokerFileEnv) == "" {
// Started by the tool runner, with no credential of its own: the put's tools (ADR 0546), served through the runtime,
// since a process never serves (ADR 0436). The process the manifest runs names its credential.
if err := serveTools(); err != nil {
@@ -50,9 +51,12 @@ func main() {
// readBuckets are the buckets of other parts the assigner reads, each named in its manifest's reads (the manifest test
// holds the two lists equal), so the controller's broker grants exactly these.
var readBuckets = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.BuildsBucket,
delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
@@ -68,14 +72,27 @@ var readBuckets = []string{
// their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go"
// ownBuckets are the assigner's own states, as its manifest declares them.
// ownBuckets are the assigner's own states, as its manifest declares them: the seat mesh-placement's three among them
// (ADR 0567 §1), named after the assigner as every module state is.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket}
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket}
// seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants
// them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind).
var seatBuckets = []string{delivery.PlacementBucket, delivery.PlacementProposalsBucket, delivery.PlacementCandidatesBucket,
delivery.PlacementPulsesBucket}
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546).
// and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9), and its placement by
// the builder, the checker and the indexer, through mesh-sdk's toolchain gate (issue 774, ADR 0549 §3).
var keptStates = []announce.State{
{Name: delivery.PlacementState, Readable: true},
{Name: delivery.PlacementProposalsState, Readable: true},
{Name: delivery.PlacementCandidatesState, Readable: true},
{Name: delivery.PlacementPulsesState, Readable: true},
{Name: "asks", Readable: true},
{Name: "compositions", Readable: true},
{Name: "differences", Readable: true},
@@ -92,12 +109,49 @@ type buckets map[string]jetstream.KeyValue
// without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{}
// The controller's lease first (ADR 0549 §5, ADR 0561 §3): once it is handed away the seat's placement alone names
// where modules run, and the controller's assignments are no bucket the assigner needs. Not granted or not there, the
// controller holds it, as before.
away, why := false, ""
if kv, err := js.KeyValue(ctx, buscompose.LeaseBucket); err != nil {
logf("the bucket %s cannot be opened (%v): the controller is taken to hold its lease, so the assignments are read beside the placement", buscompose.LeaseBucket, err)
} else {
out[buscompose.LeaseBucket] = kv
e, gerr := kv.Get(ctx, buscompose.ControllerLeaseKey)
var raw []byte
if gerr == nil {
raw = e.Value()
}
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
gerr = nil
}
away, why = buscompose.LeaseHandedAway(raw, raw != nil, gerr)
}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == buscompose.LeaseBucket {
continue
}
kv, err := js.KeyValue(ctx, name)
if name == provider.AssignmentsBucket && err != nil && away {
logf("the bucket %s cannot be opened (%v), and none is needed: %s, so the seat's placement alone is read", name, err, why)
continue
}
if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue
}
if slices.Contains(seatBuckets, name) && err != nil {
// The seat's states come in the same delivery as this assigner (ADR 0567 §1): without placement the controller's
// assignments alone name where modules run, as before any placement was written (ADR 0549 §5).
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if sourceOfGrants(name) && err != nil {
// The grants part's grants and the controller's lease choose where grants are read from (issue 774): without
// them the controller's grants are read, as before the lease is handed away.
logf("the bucket %s cannot be opened (%v): the controller's grants are read until the assigner starts again", name, err)
continue
}
if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
}
@@ -106,36 +160,24 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
return out, nil
}
// sourceOfGrants says the bucket is one of the two that choose the grants' source, the grants part's grants and the
// controller's lease: neither stops the assigner when it is not there (delivery.GrantSource).
func sourceOfGrants(name string) bool {
return name == delivery.PartGrantsBucket || name == buscompose.LeaseBucket
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to read the bus with")
}
account, err := process.ReadAccount(path)
account, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(account.Node + "." + account.Module),
nats.UserInfo(account.User, account.Password),
nats.CustomInboxPrefix(account.InboxPrefix()),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(account.Fingerprint) != "" {
opts = append(opts, nats.Secure(process.PinnedToFingerprint(account.Fingerprint)))
}
conn, err := nats.Connect(account.URL, opts...)
conn, js, err := account.Dial()
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
return err
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
kvs, err := open(ctx, js, logger.Printf)
if err != nil {
return err
@@ -150,10 +192,14 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := said.Keeps(keptStates...); err != nil {
logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err)
}
thisNode = account.Node
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, kvs, sayVia(ctx, js), emitting{said}, logger)
return follow(ctx, js, conn, kvs, sayVia(ctx, js), emitting{said}, logger)
}
// thisNode is the node this process runs on, as its credential names it: said with each of the seat's events.
var thisNode string
// emitting is the process connection as the puts' caller: its calls, and its events with no options.
type emitting struct{ *process.Conn }
@@ -162,13 +208,20 @@ func (e emitting) Emit(eventType string, body any) error { return e.Conn.Emit(ev
// watched are the buckets whose every value a pass reads; the manifests are read by digest when a pass needs one, and
// watched only for keys, so a manifest that arrives wakes a composition that waits for it.
var watched = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ModulesBucket,
delivery.PlacementBucket,
delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket,
delivery.PlacementPulsesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.AssignerAsksBucket,
@@ -179,7 +232,7 @@ var watched = []string{
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error {
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
@@ -223,7 +276,13 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
w.put(name, e.Key(), value)
before := w.grantsBucket()
w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
if name == buscompose.LeaseBucket && w.grantsBucket() == before {
// The controller renews its lease every few seconds: only a change of where grants are read from
// makes a pass (delivery.GrantSource).
continue
}
wake()
}
}
@@ -263,7 +322,12 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
}
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
}
started := 0
for _, name := range watched {
if kvs[name] == nil && (slices.Contains(seatBuckets, name) || sourceOfGrants(name) || name == provider.AssignmentsBucket) {
continue // not on the bus at this start (open said so)
}
started++
var opts []jetstream.WatchOpt
if name == delivery.HaveBucket {
opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back
@@ -275,7 +339,37 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err
}
for range len(watched) + 1 {
// The seat's states arrive with the same delivery as this assigner (ADR 0567 §1): one not on the bus at this start is
// bound once the bus says it was created, and watched from then on, with no restart (issue 1790's kind). Until then the
// controller's assignments alone name where modules run (ADR 0549 §5).
type appeared struct {
name string
kv jetstream.KeyValue
}
placementAppeared := make(chan appeared, len(seatBuckets)+2)
for _, name := range seatBuckets {
if kvs[name] == nil && js != nil {
process.AwaitBucket(ctx, js, nc, name, "the seat mesh-placement waits for it",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
// And the two that choose where grants are read from, with the same delivery as the grants part that writes one.
for _, name := range []string{delivery.PartGrantsBucket, buscompose.LeaseBucket} {
if kvs[name] == nil && js != nil {
name := name
process.AwaitBucket(ctx, js, nc, name, "the controller's grants are read",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
takePlacement := func(a appeared) error {
if kvs[a.name] != nil {
return nil // bound already, by the other of its two events
}
kvs[a.name] = a.kv
logger.Printf("the bucket %s is open and watched", a.name)
return watch(a.name)
}
for range started + 1 {
select {
case <-ctx.Done():
return nil
@@ -289,6 +383,12 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
}
if bus != nil {
p.seat = newSeatActs(bus, wake, logger.Printf)
}
if js != nil {
p.publish = streamPublisher(ctx, js, thisNode)
}
wake()
for {
select {
@@ -297,8 +397,29 @@ func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(a
case err := <-errs:
return err
case <-settled:
case kv := <-placementAppeared:
if err := takePlacement(kv); err != nil {
return err
}
case <-dirty:
reopenPuts(p)
// The grants part makes the seat's buckets in the pass that seals the list granting them, so their creation can
// be said before this credential may hear it: every pass, itself started by a watched change, binds each one the
// bus now holds, and says the seat waits while one is missing.
var unbound []string
for _, name := range seatBuckets {
if kvs[name] != nil || js == nil {
continue
}
if kv, err := js.KeyValue(ctx, name); err == nil {
if err := takePlacement(appeared{name, kv}); err != nil {
return err
}
continue
}
unbound = append(unbound, name)
}
p.sayWaits(unbound)
p.pass(ctx)
}
}
@@ -17,13 +17,19 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
// (`<module>.<state>` read as the bucket `<module>_<state>`), so it is granted what it reads and reads nothing else.
func TestOpensWhatItsManifestNames(t *testing.T) {
// 1297/B2a: without a credential the program does not start, and says why.
if err := run(context.Background(), func(string) string { return "" }, io.Discard); err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
}
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
t.Fatal(err)
@@ -58,20 +64,30 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
}
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
// Issue 774 (ADR 0549 §5): the controller's lease not granted, the assigner starts taking it as the controller's; the
// assignments refused while the controller holds it stop the start, and once it is handed away they are not needed.
handed := []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)
for _, c := range []struct {
missing string
err error
starts bool
lease []byte
said string
}{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false},
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true, nil, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, nil, ""},
// Issue 774: the grants part's grants and the controller's lease choose the grants' source; neither stops the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, nil, "the controller's grants are read"},
{buscompose.LeaseBucket, errors.New("nats: permissions violation"), true, nil, "taken to hold its lease"},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), false, []byte(`{"instance":"novox/1"}`), ""},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), true, handed, "the seat's placement alone is read"},
} {
var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged = fmt.Sprintf(f, a...) })
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err, lease: c.lease}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, "no put is taken further")):
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, c.said)):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant
t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err)
@@ -94,7 +110,7 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0))
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0))
select {
case key := <-wrote:
if key != "want.ace.node" {
@@ -105,26 +121,23 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
}
}
// 1297/B2a: without a credential the program does not start, and says why.
func TestNoCredential(t *testing.T) {
err := run(context.Background(), func(string) string { return "" }, io.Discard)
if err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
}
}
// refusingJS opens every bucket but one, which it refuses with err.
type refusingJS struct {
jetstream.JetStream
missing string
err error
lease []byte // the controller's lease, when it holds one
}
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing {
return nil, r.err
}
return &memKV{values: map[string][]byte{}}, nil
kv := &memKV{values: map[string][]byte{}}
if name == buscompose.LeaseBucket && r.lease != nil {
kv.values[buscompose.ControllerLeaseKey] = r.lease
}
return kv, nil
}
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
@@ -170,3 +183,5 @@ type kvEntry struct {
func (e kvEntry) Key() string { return e.key }
func (e kvEntry) Value() []byte { return e.value }
func (e kvEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
func (e kvEntry) Revision() uint64 { return 1 }
func (e kvEntry) Created() time.Time { return time.Time{} }
@@ -3,7 +3,7 @@ package main
import (
"context"
"errors"
"fmt"
"maps"
"slices"
"strings"
"time"
@@ -40,6 +40,17 @@ type passer struct {
// noWant is, by node and module, the reason last said why the node gets no want for the module (issue 1945): a reason is
// said in the journal when it appears or changes, and its clearing when the node has a want again; one line per change.
noWant map[string]string
// nodeFailed is, by node and module, the words of the condition last raised for a failure that is the node's own (ADR 0557
// §1, issue 1496): raised once per words, cleared once when it ends.
nodeFailed map[string]string
// The seat mesh-placement's holder (seat.go, ADR 0567): its calls of the controller (nil: none is made), how it says
// the seat's events (nil: none is said), the proposals it said, and what Intended computed by proposal and base.
seat *seatActs
publish publisher
saidProposed map[string]bool
// saidWaits is the last said of the seat's wait for its states (sayWaits).
saidWaits string
intended map[string]map[string]map[string]bool
}
func (p *passer) clock() time.Time {
@@ -50,6 +61,7 @@ func (p *passer) clock() time.Time {
}
func (p *passer) pass(ctx context.Context) {
p.placementSeat(ctx)
p.compositions(ctx)
p.wants(ctx)
p.puts(ctx)
@@ -61,7 +73,23 @@ func (p *passer) pass(ctx context.Context) {
// controller has composed it and again whenever that digest changes (ADR 0539 §6).
func (p *passer) nodeWants(ctx context.Context) {
written := p.w.read(delivery.AssignerWantsBucket)
placed := p.w.placedNodes()
// A node the catalogue places composes its own composition (ADR 0547 §2): its own want names none.
for node := range placed {
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
}
if before, err := delivery.DecodeNodeWant(written[key]); err == nil && before.Self {
continue
}
want := delivery.NodeWant{Self: true, Words: "the node composes its own composition (ADR 0547 §2)"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
for node, d := range p.w.nodeComposed() {
if placed[node] {
continue
}
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
@@ -98,6 +126,8 @@ func (p *passer) wants(ctx context.Context) {
}
}
wanted := p.w.wantsOf()
frozen := p.sayPlacement(p.withCalls(p.w.placementDiffers()))
placed := p.w.placedNodes()
puts := p.putsByModule()
p.staleAsks(ctx, trunk)
names := map[string]bool{}
@@ -114,15 +144,26 @@ func (p *passer) wants(ctx context.Context) {
// quiet at the router.
first := p.neither == nil
stale := map[string]neitherOne{}
defer func() { p.sayNeither(neither, stale) }()
left := map[string]string{}
defer func() { p.sayNeither(neither, stale, left) }()
// Where grants are read from, once for the pass: the grants part's while the lease is handed away (issue 774).
grants := p.w.grantsBucket()
for name := range names {
fp := trunk[name].FP2
name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
Composed: func(node, fp string) (delivery.ComposedDigest, bool) { return p.w.composed(node, name, fp) },
Self: func(node string) bool { return placed[node] },
Composed: func(node, fp string) (delivery.ComposedDigest, bool) {
if placed[node] {
// The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want
// names the build alone, and the node judges the move itself.
return delivery.ComposedDigest{Judgement: delivery.JudgedNeither, Words: "the node composes its own"}, true
}
return p.w.composed(node, name, fp)
},
GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key)
return err == nil && p.w.has(grants, key)
},
Restored: func(node string) bool {
key, err := delivery.RestoreAskKey(node, name, fp)
@@ -138,8 +179,8 @@ func (p *passer) wants(ctx context.Context) {
hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(delivery.ControllerGrantsBucket, grant)
if raw := p.w.value(delivery.ControllerGrantsBucket, grant); raw != nil {
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(grants, grant)
if raw := p.w.value(grants, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
}
@@ -153,8 +194,11 @@ func (p *passer) wants(ctx context.Context) {
last[node] = p.w.lastHealthy(node, name, fp)
}
out := assign.Wants(m, nodes, wanted[name], last)
p.yeses(ctx, name, m, wanted[name], &out)
gatherNeither(neither, name, out)
leftNeither(left, m, wanted[name], out)
p.sayNoWant(name, m.Nodes, wanted[name], out)
p.sayNodeFailed(name, fp, m.Nodes, nodes, out, first)
if first {
startClears(stale, m, out)
if fp != "" && !m.Held && out.Held == nil && !out.Lifted {
@@ -182,6 +226,9 @@ func (p *passer) wants(ctx context.Context) {
p.recordAsk(ctx, name, fp, need)
}
for node, want := range out.Wants {
if frozen[node+"."+name] {
continue // one source alone names it (ADR 0549 §5): no want, and the one written before stays
}
key, err := delivery.WantKey(node, name)
if err != nil {
p.logf("no want of %s on %s is written: %v", name, node, err)
@@ -210,7 +257,7 @@ func (p *passer) sayNoWant(module string, nodes []string, before map[string]deli
}
reason := ""
if h, held := out.Rules[node]; held && (!written || w.Removed) {
reason = fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
reason = h.Reason()
}
last, said := p.noWant[key]
switch {
@@ -229,6 +276,60 @@ func (p *passer) sayNoWant(module string, nodes []string, before map[string]deli
}
}
// sayNodeFailed raises, once per node and words, the condition of a failed health check of a module's newest fingerprint fp
// that is the node's own, since fp runs healthy elsewhere and is not held (ADR 0557 §1, issue 1496), and says it in the journal;
// it clears the condition once when the node no longer fails alone, or is no longer one of the module's nodes. The first pass
// after a start remembers nothing a previous process raised: it clears, for every module, the condition of every node the
// presence bucket knows that does not fail alone, since a node that recovered while no process watched leaves no failed have to
// find it by (a clear of a condition not open is quiet at the router). A condition the router did not take is said again at the next pass.
func (p *passer) sayNodeFailed(module, fp string, nodes []string, known assign.Nodes, out assign.Outcome, first bool) {
if p.nodeFailed == nil {
p.nodeFailed = map[string]string{}
}
look := map[string]bool{}
for _, node := range nodes {
look[node] = true
}
for key := range p.nodeFailed {
if node, m, _ := strings.Cut(key, "."); m == module {
look[node] = true
}
}
if first {
for node := range known.Online {
look[node] = true
}
}
for _, node := range slices.Sorted(maps.Keys(look)) {
key := node + "." + module
f, failing := out.NodeFailed[node]
last, said := p.nodeFailed[key]
switch {
case failing && f.Words+"|"+strings.Join(f.Others, ",") != last:
p.logf("%s on %s: the health check at %s failed here (%s), while it runs healthy on %s: the failure is %s's, so nothing "+
"is held or set back", module, node, fp, f.Words, strings.Join(f.Others, ", "), node)
if p.say != nil {
if err := p.say(assign.NodeFailedCondition(node, module, fp, f, true)); err != nil {
p.logf("the condition of %s failing on %s alone could not be said, and is said at the next pass: %v", module, node, err)
continue
}
}
p.nodeFailed[key] = f.Words + "|" + strings.Join(f.Others, ",")
case !failing && (said || first):
if said {
p.logf("%s on %s no longer fails its health check alone", module, node)
}
if p.say != nil {
if err := p.say(assign.NodeFailedCondition(node, module, fp, assign.NodeFailure{}, false)); err != nil && said {
p.logf("the condition of %s failing on %s alone could not be cleared, and is cleared at the next pass: %v", module, node, err)
continue
}
}
delete(p.nodeFailed, key)
}
}
}
// neitherOne is a module assigned to a node that holds no want for it after a pass, and the rule holding its first want back.
type neitherOne struct {
node, module string
@@ -242,6 +343,48 @@ func gatherNeither(into map[string]neitherOne, module string, out assign.Outcome
}
}
// leftNeither says, by node and module, for every node of m not on neither list after the pass, why: "" when the node has a live
// want for it (written before and kept, or written in this pass), else why it has none. sayNeither clears a neither condition in
// these words, so a clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while the bus bar
// left it with none).
func leftNeither(into map[string]string, m assign.Module, before map[string]delivery.Want, out assign.Outcome) {
for _, node := range m.Nodes {
if _, on := out.Neither[node]; on {
continue
}
w, written := out.Wants[node]
if !written {
w, written = before[node]
}
why := ""
h, ruled := out.Rules[node]
switch {
case written && !w.Removed:
case (m.Held && !out.Lifted) || out.Held != nil:
why = "it is held, and its held condition says so"
case slices.ContainsFunc(out.Record, func(r assign.RecordNeed) bool { return r.Node == node }):
why = "its move waits for the operator's yes on its record ask"
case putOn(m, node):
why = "its put waits on its own steps and the operator's answer"
case m.FP == "":
why = "the trunk has no build of it"
case ruled:
why = h.Reason()
default:
why = "the node does not run it"
}
into[node+"."+m.Name] = why
}
}
// putOn says whether an operator's put on node owns m's want there: one still on its steps (ADR 0546), so the rules write none
// meanwhile. An ended put (done, refused, released) stays in the bucket and owns nothing.
func putOn(m assign.Module, node string) bool {
p, put := m.Puts[node]
_, owns := assign.PutOwns(p)
return put && owns
}
// startClears adds, for the first pass after a start, every node of m that is not on neither list after the pass: a neither
// condition a previous process raised for it is cleared. That covers a node with a want and one with none that is not on neither
// list, such as a held module (issue 1945: the hold is said on its own, so a neither raised for it before the restart would stay up
@@ -258,7 +401,9 @@ func startClears(into map[string]neitherOne, m assign.Module, out assign.Outcome
// once, the condition of each one said before that has a want again (issue 1945). A say that fails is made again at the next pass.
// stale is, on the first pass after a start, the modules whose condition a previous process may have raised and that have a want:
// each is cleared once.
func (p *passer) sayNeither(now, stale map[string]neitherOne) {
// left is, by node and module, why each assigned node not on neither list after this pass has no want ("" when it has one); a key
// it does not hold is no longer assigned.
func (p *passer) sayNeither(now, stale map[string]neitherOne, left map[string]string) {
if p.neither == nil {
p.neither = map[string]neitherOne{}
for key, one := range stale {
@@ -278,21 +423,29 @@ func (p *passer) sayNeither(now, stale map[string]neitherOne) {
continue
}
}
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; rule %d holds its first want back: %s",
one.module, one.node, one.holding.Rule, one.holding.Words)
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; what holds its first want back: %s",
one.module, one.node, one.holding.Reason())
p.neither[key] = one
}
for key, one := range p.neither {
if _, still := now[key]; still {
continue
}
why, assigned := left[key]
if !assigned {
why = "it is no longer assigned there"
}
if p.say != nil {
if err := p.say(assign.NeitherCondition(one.node, one.module, assign.Holding{}, false)); err != nil {
if err := p.say(assign.NeitherLeftCondition(one.node, one.module, why)); err != nil {
p.logf("the condition of %s on neither list on %s could not be cleared, and is cleared at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s on %s has a want again", one.module, one.node)
if why == "" {
p.logf("%s on %s has a want again", one.module, one.node)
} else {
p.logf("%s on %s is no longer on neither list and has no want: %s", one.module, one.node, why)
}
delete(p.neither, key)
}
}
@@ -481,13 +634,19 @@ func (p *passer) staleAsks(ctx context.Context, trunk map[string]delivery.Module
}
for key := range p.w.read(delivery.AssignerAsksBucket) {
if p.staleAsk(key, trunk) {
// The condition is cleared first and the ask goes only once that was said: the ask is what brings this key back at the
// next pass, so a clear that failed once is made again, with no state of its own (issue 1906, note 29).
k, kerr := delivery.ParseAnyAskKey(key)
held := kerr == nil && (k.Kind == delivery.AskKindHold || k.Kind == delivery.AskKindRestore)
if held && !p.clearHeld(k.Module, k.FP) {
continue
}
p.logf("the ask %s is about a fingerprint or composition the trunk has left behind and is taken away", key)
p.remove(ctx, delivery.AssignerAsksBucket, key)
// The held record stays: it is the memory that this fingerprint failed (ADR 0408 §4), and a revert returns the trunk to
// exactly it. Only the condition, which would else ask for an answer that can no longer be given, is cleared, and it is
// raised again if the trunk returns to the fingerprint.
if k, err := delivery.ParseAnyAskKey(key); err == nil && (k.Kind == delivery.AskKindHold || k.Kind == delivery.AskKindRestore) {
p.clearHeld(k.Module, k.FP)
if held {
delete(p.said, assign.HeldCondition(k.Module, k.FP, "", "", false, false).Key+".raised")
}
}
@@ -566,16 +725,18 @@ func (p *passer) answeredWith(key, option string) bool {
}
// clearHeld clears, once, the condition of a hold the operator released.
func (p *passer) clearHeld(module, fp string) {
// It says whether nothing is left to say: the clear was said now or before, or there is no router to say it to.
func (p *passer) clearHeld(module, fp string) bool {
c := assign.HeldCondition(module, fp, "", "", false, false)
if p.say == nil || p.said[c.Key+".cleared"] {
return
return true
}
if err := p.say(c); err != nil {
p.logf("the condition of the released %s could not be cleared, and is cleared at the next pass: %v", module, err)
return
return false
}
p.said[c.Key+".cleared"] = true
return true
}
// recordAsk writes, once, the ask about a move of a module of policy record that waits for the operator's yes.
@@ -667,3 +828,83 @@ func (p *passer) backupAsk(ctx context.Context, key, node, module string, want d
}
p.said[c.Key] = true
}
// sayPlacement raises, once, the condition of every pair the controller's assignments and the catalogue's placement do not
// both name, clears it once they agree, and answers those pairs by node and module (ADR 0549 §5): the pass writes no want
// for them and keeps the one written before. A say that fails is made again at the next pass.
func (p *passer) sayPlacement(differ []assign.Pair) map[string]bool {
frozen := map[string]bool{}
now := map[string]bool{}
for _, d := range differ {
frozen[d.Node+"."+d.Module] = true
c := assign.PlacementCondition(d, true)
now[c.Key] = true
if p.said[c.Key+".raised"] {
continue
}
p.logf("%s on %s is named by %s alone: no want is written for it there until both sources name it (ADR 0549 §5)", d.Module, d.Node, d.Only)
if err := p.say(c); err != nil {
p.logf("the placement condition of %s on %s could not be said, and is said at the next pass: %v", d.Module, d.Node, err)
continue
}
p.said[c.Key+".raised"] = true
}
prefix := delivery.AssignerModule + ".placement."
for key := range p.said {
c, raised := strings.CutSuffix(key, ".raised")
if !raised || !strings.HasPrefix(c, prefix) || now[c] {
continue
}
node, module, _ := strings.Cut(strings.TrimPrefix(c, prefix), ".")
if err := p.say(assign.PlacementCondition(assign.Pair{Node: node, Module: module}, false)); err != nil {
p.logf("the placement condition of %s on %s could not be cleared, and is cleared at the next pass: %v", module, node, err)
continue
}
delete(p.said, key)
}
return frozen
}
// yeses writes a placed node's wants naming the build alone, and sees to the moves a node that composes its own judged a
// person must allow (ADR 0547 §3): for a have of phase
// waiting-yes it raises the record ask once, naming the composition waiting and the node's judgement, and on a yes writes
// the want again naming it; a yes once written is kept while the want stays at its build, so the node is not asked again.
func (p *passer) yeses(ctx context.Context, module string, m assign.Module, before map[string]delivery.Want, out *assign.Outcome) {
for _, node := range m.Nodes {
if m.Self == nil || !m.Self(node) {
continue
}
was, had := before[node]
w, written := out.Wants[node]
if !written {
if !had || was.Removed {
continue
}
w = was
}
if w.Removed {
continue
}
yes := ""
if had && was.FP == w.FP {
yes = was.Yes
}
key, err := delivery.HaveKey(node, module)
if err != nil {
continue
}
if h, err := delivery.DecodeHave(p.w.value(delivery.HaveBucket, key)); err == nil && h.Phase == delivery.PhaseWaitingYes && h.Waits != yes {
if m.Allowed != nil && m.Allowed(node, h.Waits) {
yes = h.Waits
} else {
p.recordAsk(ctx, module, w.FP, assign.RecordNeed{Node: node, Digest: h.Waits, Judgement: h.Judgement, Words: h.Detail})
}
}
// A want carried over from before the node composed its own may still name the controller's composition; a placed
// node's want names the build alone (ADR 0547 §2).
if yes != w.Yes || w.Composed != "" || !w.Self {
w.Yes, w.Composed, w.Self = yes, "", true
out.Wants[node] = w
}
}
}
@@ -13,6 +13,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
@@ -135,6 +136,9 @@ func (m *memKV) Update(_ context.Context, key string, value []byte, revision uin
type memStore map[string]*memKV
func (s memStore) get(state, key string) ([]byte, uint64, bool, error) {
if s[state] == nil {
return nil, 0, false, nil
}
v, ok := s[state].values[key]
return v, uint64(len(v)), ok, nil
}
@@ -342,6 +346,9 @@ func putOfTheBus(t *testing.T) {
// again after a say that failed, and clears it once when the module has a want again; a module on neither list is also written to
// the journal, so it is loud with no operator channel.
func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
// A pair only one source names is a condition of the same kind (issue 1126): folded here to hold the count.
aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t)
t.Run("gathers", testThePassGathersTheModulesOnNeitherList)
p := newPasser(buckets{}, time.Date(2026, 10, 25, 15, 0, 0, 0, time.UTC))
var said []asks.Condition
failing := true
@@ -357,30 +364,66 @@ func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
p.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
one := map[string]neitherOne{"g14.screen-lock": {node: "g14", module: "screen-lock",
holding: assign.Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}}}
p.sayNeither(one, nil) // the say fails
p.sayNeither(one, nil)
p.sayNeither(one, nil)
p.sayNeither(one, nil, nil) // the say fails
p.sayNeither(one, nil, nil)
p.sayNeither(one, nil, nil)
if len(said) != 1 || said[0].State != asks.StateRaised || said[0].Subject.Machine != "g14" || said[0].Subject.ID != "screen-lock" {
t.Fatalf("said = %+v, want one raised condition of screen-lock on g14", said)
}
if !strings.Contains(strings.Join(logged, "\n"), "screen-lock") {
t.Errorf("the journal does not name the module on neither list: %q", logged)
}
p.sayNeither(map[string]neitherOne{}, nil)
p.sayNeither(map[string]neitherOne{}, nil)
wants := map[string]string{"g14.screen-lock": ""}
p.sayNeither(map[string]neitherOne{}, nil, wants)
p.sayNeither(map[string]neitherOne{}, nil, wants)
if len(said) != 2 || said[1].State != asks.StateCleared || said[1].Key != said[0].Key {
t.Errorf("said = %+v, want the condition cleared once under its key", said)
}
if last := logged[len(logged)-1]; last != "screen-lock on g14 has a want again" {
t.Errorf("the clear said %q", last)
}
// 1945 (the bus bar): a module that leaves neither list with no want is cleared in words saying so, never "has a want again".
bus := map[string]neitherOne{"novox.nats": {node: "novox", module: "nats", holding: assign.Holding{Bus: true, Words: "the bus moves only by the operator's put"}}}
p.sayNeither(bus, nil, nil)
if c := said[len(said)-1]; c.State != asks.StateRaised || !strings.HasPrefix(c.Explanation, asks.NeedsYou) || !strings.Contains(c.Explanation, "the bus moves only by the operator's put") {
t.Errorf("the bus's neither condition = %+v", c)
}
p.sayNeither(map[string]neitherOne{}, nil, map[string]string{"novox.nats": "it is held, and its held condition says so"})
if c := said[len(said)-1]; c.State != asks.StateCleared || strings.Contains(c.Resolved, "has a want") {
t.Errorf("a module that left neither list with no want was cleared as %+v", c)
}
if last := logged[len(logged)-1]; last != "nats on novox is no longer on neither list and has no want: it is held, and its held condition says so" {
t.Errorf("the clear without a want said %q", last)
}
}
// 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module.
func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
func testThePassGathersTheModulesOnNeitherList(t *testing.T) {
got := map[string]neitherOne{}
gatherNeither(got, "screen-lock", assign.Outcome{Neither: map[string]assign.Holding{"g14": {Rule: delivery.RuleGrant, Words: "w"}}})
gatherNeither(got, "rofi", assign.Outcome{})
if len(got) != 1 || got["g14.screen-lock"].node != "g14" || got["g14.screen-lock"].module != "screen-lock" || got["g14.screen-lock"].holding.Rule != delivery.RuleGrant {
t.Errorf("gathered = %+v", got)
}
// 1945 (the bus bar): a node off neither list is cleared in words that say why it has no want; an operator's put running
// through its steps owns the want, so it is named, never "the node does not run it".
for _, c := range []struct {
name string
puts map[string]delivery.Put
before map[string]delivery.Want
why string
}{
{"a live want", nil, map[string]delivery.Want{"novox": {FP: fpTest}}, ""},
{"a put on its steps", map[string]delivery.Put{"novox": {FP: fpTest, Step: delivery.PutPerson}}, nil, "its put waits on its own steps and the operator's answer"},
{"nothing", nil, nil, "the node does not run it"},
{"an ended put", map[string]delivery.Put{"novox": {FP: fpTest, Step: delivery.PutRefused}}, nil, "the node does not run it"},
} {
left := map[string]string{}
leftNeither(left, assign.Module{Name: "nats", FP: fpTest, Nodes: []string{"novox"}, Puts: c.puts}, c.before, assign.Outcome{})
if why, there := left["novox.nats"]; !there || why != c.why {
t.Errorf("%s: left %q (%v), want %q", c.name, why, there, c.why)
}
}
// 1945 (a first want held back silently): the journal says why a node gets no want when the reason appears or changes, and
// when it has a want again: one line per change, not per pass.
p := newPasser(buckets{}, time.Date(2026, 10, 25, 21, 30, 0, 0, time.UTC))
@@ -399,6 +442,7 @@ func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
{nil, assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Rule: delivery.RuleGrant, Words: "no credential"}}}, "rofi on shanks gets no want: rule 6: no credential"},
{nil, wanted, "rofi on shanks has a want"},
{map[string]delivery.Want{"shanks": wanted.Wants["shanks"]}, assign.Outcome{}, ""},
{nil, assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Bus: true, Words: "the bus moves only by the operator's put"}}}, "rofi on shanks gets no want: the bus moves only by the operator's put"},
} {
logged = nil
p.sayNoWant("rofi", []string{"shanks"}, c.before, c.out)
@@ -406,12 +450,71 @@ func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
t.Errorf("pass %d said %q, want %q", i, got, c.line)
}
}
// 1496, ADR 0557 §1: a failure that is the node's own (its fingerprint healthy elsewhere, not held) raises the node's condition
// once, naming the node, the module, F and the have's words, and clears it once when it ends.
var conds []asks.Condition
p.say = func(c asks.Condition) error { conds = append(conds, c); return nil }
failing := map[string]assign.NodeFailure{"shanks": {Others: []string{"ace"}, Words: "health failed, phase running: w"}}
for i, c := range []struct {
failed map[string]assign.NodeFailure
state string // the condition said in this pass, "" for none
}{
{failing, asks.StateRaised},
{failing, ""},
{nil, asks.StateCleared},
{nil, ""},
} {
conds = nil
p.sayNodeFailed("rofi", fpTest, []string{"shanks"}, assign.Nodes{}, assign.Outcome{NodeFailed: c.failed}, false)
switch {
case c.state == "" && len(conds) != 0, c.state != "" && (len(conds) != 1 || conds[0].State != c.state):
t.Errorf("node failed, pass %d said %+v, want %q", i, conds, c.state)
case c.state == asks.StateRaised && (conds[0].Key != "mesh-delivery-assigner.failed-alone.shanks.rofi" ||
!strings.Contains(conds[0].Explanation, "failed on shanks (health failed, phase running: w), while it runs healthy on ace")):
t.Errorf("node failed, raised %+v", conds[0])
}
}
// D2: a node that leaves the module's nodes while it failed alone is cleared once; the first pass after a start clears every
// node the presence bucket knows, for every module, failed have or not (a node that recovered unwatched leaves none).
conds = nil
p.sayNodeFailed("rofi", fpTest, []string{"shanks"}, assign.Nodes{}, assign.Outcome{NodeFailed: failing}, false)
p.sayNodeFailed("rofi", fpTest, nil, assign.Nodes{}, assign.Outcome{}, false)
p.sayNodeFailed("rofi", fpTest, nil, assign.Nodes{}, assign.Outcome{}, false)
if len(conds) != 2 || conds[1].State != asks.StateCleared || conds[1].Key != "mesh-delivery-assigner.failed-alone.shanks.rofi" {
t.Errorf("a node leaving the module said %+v, want one raise and one clear", conds)
}
known := assign.Nodes{Online: map[string]bool{"ace": true, "g14": false},
Haves: map[string]map[string]delivery.Have{"g14": {"rofi": {FP: fpTest, Health: delivery.HealthFailed}}}}
for _, c := range []struct {
name string
known assign.Nodes
keys []string
}{
{"a failed have", known, []string{"mesh-delivery-assigner.failed-alone.ace.rofi", "mesh-delivery-assigner.failed-alone.g14.rofi"}},
{"no failed have", assign.Nodes{Online: known.Online}, []string{"mesh-delivery-assigner.failed-alone.ace.rofi", "mesh-delivery-assigner.failed-alone.g14.rofi"}},
} {
conds = nil
fresh := newPasser(buckets{}, time.Date(2026, 10, 26, 22, 0, 0, 0, time.UTC))
fresh.logf = func(string, ...any) {}
fresh.say = func(c asks.Condition) error { conds = append(conds, c); return nil }
fresh.sayNodeFailed("rofi", fpTest, nil, c.known, assign.Outcome{}, true)
var keys []string
for _, cond := range conds {
if cond.State == asks.StateCleared {
keys = append(keys, cond.Key)
}
}
if !slices.Equal(keys, c.keys) || len(conds) != len(c.keys) {
t.Errorf("first pass, %s: said %+v, want clears of %q", c.name, conds, c.keys)
}
}
}
// A neither or held condition a previous process raised is cleared by the first pass after a start: a fresh pass remembers
// nothing it said, so it clears, once, the neither condition of every assigned module not on neither list and the held condition of
// every module that is not held. Clearing a condition that is not open is quiet at the router.
func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
t.Run("a held module", testAHeldModuleRaisesNoNeitherAndClearsAStandingOne)
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerWantsBucket: wantsKV, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
var said []asks.Condition
@@ -449,7 +552,7 @@ func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
// 1945/AC24: given a module held at F under rule 2 on a node that runs it with no want, when the assigner passes, then it raises no
// neither condition for it, a neither condition this process raised for it before is cleared, and so is one a previous process
// raised (the first pass after a start), each once.
func TestAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) {
func testAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) {
setup := func() (*passer, *[]asks.Condition) {
p := newPasser(buckets{delivery.AssignerWantsBucket: &memKV{values: map[string][]byte{}}, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
said := &[]asks.Condition{}
@@ -606,6 +709,36 @@ func TestAsksOfAFingerprintTheTrunkLeftBehindAreTakenAway(t *testing.T) {
})
}
t.Run("a clear that failed to be said is said at the next pass, and the ask goes after it", func(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
var said []asks.Condition
fail := true
p.say = func(c asks.Condition) error {
if fail {
fail = false
return errors.New("router unreachable")
}
said = append(said, c)
return nil
}
a := assign.HoldAsk(holdOld, "web", fpOld, "novox", "w", 1, time.Now())
v, err := delivery.EncodeAssignerAsk(holdOld, a)
if err != nil {
t.Fatal(err)
}
asksKV.values[holdOld] = v
p.w.put(delivery.AssignerAsksBucket, holdOld, v)
p.staleAsks(context.Background(), trunkOf(fpNew))
if _, there := asksKV.values[holdOld]; !there || len(said) != 0 {
t.Fatalf("after a failed clear: ask left = %v, conditions said = %d; want the ask kept and nothing said", there, len(said))
}
p.staleAsks(context.Background(), trunkOf(fpNew))
if _, there := asksKV.values[holdOld]; there || len(said) != 1 || said[0].State != asks.StateCleared {
t.Errorf("at the next pass: ask left = %v, conditions said = %v; want the ask gone and the clear said", there, said)
}
})
t.Run("a lift takes the restore asks of its fingerprint on every node", func(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
@@ -635,6 +768,8 @@ func mustKey(key string, err error) string {
// (ADR 0545). That record ask is not about the trunk's fingerprint: it survives every pass, so the operator is asked, and on a
// yes the want names the revised digest.
func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing.T) {
// A placed node's yes comes from its have (issue 1126): folded here to hold the count.
aPlacedNodesWantsNameTheBuildAloneAndItsYesComesFromItsHave(t)
fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64)
revised, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64)
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
@@ -683,3 +818,151 @@ func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing
t.Error("the answered record ask was taken away")
}
}
// 1126 (ADR 0549 §5): until the third milestone a pair the controller's assignments and the catalogue's placement do not
// both name gets no want and one condition, raised once and cleared once both name it; before any placement was read the
// assignments are the one source and nothing is raised.
func aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t *testing.T) {
p := newPasser(buckets{}, time.Date(2026, 10, 26, 18, 0, 0, 0, time.UTC))
var said []asks.Condition
p.say = func(c asks.Condition) error { said = append(said, c); return nil }
assigned := func(node, module string) {
raw, _ := json.Marshal(map[string]string{"node": node, "module": module})
p.w.put(provider.AssignmentsBucket, node+"."+module, raw)
}
place := func(node string, modules ...string) {
pl := delivery.Placement{Node: node}
for _, m := range modules {
pl.Modules = append(pl.Modules, delivery.PlacedModule{Module: m})
}
raw, err := delivery.EncodePlacement(pl)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.PlacementKey(node)
p.w.put(delivery.PlacementBucket, key, raw)
}
assigned("g14", "a")
assigned("g14", "b")
if got, _ := p.w.assigned(); !slices.Equal(got["g14"], []string{"a", "b"}) || len(p.sayPlacement(p.w.placementDiffers())) != 0 || len(said) != 0 {
t.Fatalf("before any placement: assigned %v, said %v", got, said)
}
none := func(context.Context, string) ([]byte, error) { return nil, errNotThere }
if m := p.w.mesh(context.Background(), none); m.Input != nil || m.InputUnread != "" {
t.Errorf("before any placement the check composes over %+v (%q)", m.Input, m.InputUnread)
}
place("g14", "a", "c")
got, _ := p.w.assigned()
frozen := p.sayPlacement(p.w.placementDiffers())
p.sayPlacement(p.w.placementDiffers())
if !slices.Equal(got["g14"], []string{"a"}) || !frozen["g14.b"] || !frozen["g14.c"] || frozen["g14.a"] || len(said) != 2 ||
said[0].State != asks.StateRaised || !strings.Contains(said[0].Explanation, "the controller's assignments names b on g14") ||
!strings.Contains(said[1].Explanation, "the seat mesh-placement's placement names c on g14") {
t.Fatalf("one source alone: assigned %v, frozen %v, said %+v", got, frozen, said)
}
place("g14", "a", "b", "c")
assigned("g14", "c")
if frozen := p.sayPlacement(p.w.placementDiffers()); len(frozen) != 0 || len(said) != 4 || said[2].State != asks.StateCleared || said[3].State != asks.StateCleared {
t.Errorf("both agree again: frozen %v, said %+v", frozen, said)
}
// 1126 (ADR 0547 §2): placed, the check composes over the input the nodes compose from (frombus), once each placed node
// wrote its record; until then it says why and composes the assignments alone.
if m := p.w.mesh(context.Background(), none); m.Input != nil || !strings.Contains(m.InputUnread, "g14") {
t.Errorf("a placed node with no record: input %+v, unread %q", m.Input, m.InputUnread)
}
rec, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: "g14", TunnelKey: "k"}})
p.w.put(delivery.HaveBucket, "g14.node", rec)
m := p.w.mesh(context.Background(), none)
if m.Input == nil || len(m.Input.Nodes) != 1 || !slices.Equal(m.Input.Nodes[0].Assigned, []string{"a", "b", "c"}) || m.Input.Nodes[0].Place.Key != "k" {
t.Errorf("the check's input from the bus: %+v (%q)", m.Input, m.InputUnread)
}
// Issue 774 (ADR 0549 §5, §6): once the controller's lease is handed away the placement alone names where modules
// run: a pair in the placement only gets its want and no condition, and one in the assignments only is no module of
// the node. A lease that goes back to a controller brings the comparison back.
place("n2", "d")
assigned("g14", "e")
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`))
got, all := p.w.assigned()
if !slices.Equal(got["n2"], []string{"d"}) || !slices.Equal(got["g14"], []string{"a", "b", "c"}) || len(all) != 0 {
t.Errorf("handed away: assigned %v, assignments %v; want the placement alone", got, all)
}
if frozen := p.sayPlacement(p.w.placementDiffers()); len(frozen) != 0 || len(said) != 4 {
t.Errorf("handed away: frozen %v, said %+v; want no condition", frozen, said[4:])
}
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"instance":"novox/1","epoch":58}`))
if frozen := p.sayPlacement(p.w.placementDiffers()); !frozen["n2.d"] || !frozen["g14.e"] {
t.Errorf("a controller's lease again: frozen %v; want both pairs held", frozen)
}
}
// 1126 (ADR 0547 §2 to §4): a node the catalogue places composes its own: its wants name the build alone and its own want
// names no composition; a have waiting for a yes raises the record ask once, and the yes is written into the want and
// kept there while the want stays at its build.
func aPlacedNodesWantsNameTheBuildAloneAndItsYesComesFromItsHave(t *testing.T) {
fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64)
waiting, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64)
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
p.logf = t.Logf
put := func(bucket, key string, v []byte) { p.w.put(bucket, key, v) }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
put(delivery.ModulesBucket, "web", mod) // the trunk's build is not done: rule 1 holds g14 at fpOld
put(provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"b-1"}`))
pl, _ := delivery.EncodePlacement(delivery.Placement{Node: "g14", Modules: []delivery.PlacedModule{{Module: "web"}}})
put(delivery.PlacementBucket, "placement.g14", pl)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
put(delivery.PresenceBucket, "g14", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: was, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
put(delivery.HaveBucket, "g14.web", have)
wkey, _ := delivery.WantKey("g14", "web")
want, _ := delivery.EncodeWant(delivery.Want{FP: fpOld, Composed: was})
wantsKV.values[wkey] = want
put(delivery.AssignerWantsBucket, wkey, want)
sync := func() {
for k, v := range wantsKV.values {
put(delivery.AssignerWantsBucket, k, v)
}
}
p.wants(context.Background())
p.nodeWants(context.Background())
sync()
got, err := delivery.DecodeWant(wantsKV.values[wkey])
if err != nil || got.FP != fpOld || got.Composed != "" || !got.Self {
t.Fatalf("the placed node's want is %+v (%v), want it at %s, the node composing it", got, err, fpOld)
}
nkey, _ := delivery.WantNodeKey("g14")
if nw, err := delivery.DecodeNodeWant(wantsKV.values[nkey]); err != nil || nw.Composed != "" || !nw.Self {
t.Fatalf("the placed node's own want is %+v (%v), want one naming no composition", nw, err)
}
waits, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: was, Phase: delivery.PhaseWaitingYes, Health: delivery.HealthOK,
Judgement: delivery.JudgedRecreates, Waits: waiting, Detail: "it recreates web.server"})
put(delivery.HaveBucket, "g14.web", waits)
akey, _ := delivery.RecordAskKey("g14", "web", waiting)
p.wants(context.Background())
if _, there := asksKV.values[akey]; !there {
t.Fatal("a have waiting for a yes raised no record ask")
}
a, err := delivery.DecodeAssignerAsk(akey, asksKV.values[akey])
if err != nil {
t.Fatal(err)
}
put(delivery.AssignerAsksBucket, akey, asksKV.values[akey])
yes, _ := delivery.EncodeAnswer(delivery.Answer{Option: assign.OptionYes, Warrant: a.Ask.ID + "@2026-10-26T19:00:00Z", Words: "the operator"})
put(delivery.AnswersBucket, akey, yes)
p.wants(context.Background())
sync()
if got, err = delivery.DecodeWant(wantsKV.values[wkey]); err != nil || got.Yes != waiting || got.FP != fpOld {
t.Fatalf("after the yes the want is %+v (%v), want it to name the yes %s", got, err, waiting)
}
// The node applied it: its have runs the composition; the yes stays while the want stays at its build.
ran, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Composed: waiting, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
put(delivery.HaveBucket, "g14.web", ran)
p.wants(context.Background())
if got, _ = delivery.DecodeWant(wantsKV.values[wkey]); got.Yes != waiting {
t.Errorf("a pass after the move dropped the yes: %+v", got)
}
}
@@ -213,8 +213,7 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
// composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one
// place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and
// mesh-controller_grants); when composing and grants move out of the controller into delivery's own parts, this reads them
// from their new owner and nothing else changes.
// mesh-controller_grants); the grants are the grants part's while the controller's lease is handed away (grantsBucket).
func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) {
var out *delivery.ComposedDigest
if d, ok := p.w.composed(node, module, fp); ok {
@@ -222,7 +221,7 @@ func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest,
}
granted := false
if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil {
if raw := p.w.value(delivery.ControllerGrantsBucket, gkey); raw != nil {
if raw := p.w.value(p.w.grantsBucket(), gkey); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
granted = g.NamesNode(node, module)
}
@@ -0,0 +1,810 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"reflect"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-placement's holder in the assigner's process (novox/hq ADR 0567 §2-§8): each pass follows every open
// proposal one step, from what the world holds. A proposal the verbs wrote is said (placement-proposed); the nodes it
// must hear from are found by composing every node with and without it, and those its key does not change get a reached
// candidate; their answers are judged (placement-judged) and, on a passing verdict, the operator is asked to approve it;
// on the warrant each changed key is written by compare-and-set at the revision it was laid over (placement-changed),
// and the controller's assignments are kept in step through its verbs until the lease write. A key moved meanwhile
// lays the proposal again; a decline, an expiry or a withdrawal ends it (placement-withdrawn). Every event is said on the
// assigner's own stream, so only this process says them. Nothing polls: the wake at an answer's bound is the one wake.
// publisher says one of the seat's events on the assigner's stream, under a message id the bus keeps one of.
type publisher func(event string, body any, id string) error
// streamPublisher publishes on mesh.stream.mesh-delivery-assigner.event.<event>, with the headers an emit carries.
func streamPublisher(ctx context.Context, js jetstream.JetStream, node string) publisher {
return func(event string, body any, id string) error {
data, err := json.Marshal(body)
if err != nil {
return err
}
msg := nats.NewMsg(process.ModuleStreamSubject(delivery.AssignerModule, event))
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", delivery.AssignerModule)
msg.Header.Set("x-node", node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
_, err = js.PublishMsg(ctx, msg, jetstream.WithMsgID(id))
return err
}
}
// seatActs keeps what the seat's calls of the controller came to while this process runs: the export read again at a
// seed's warrant, and each call that keeps the controller's assignments in step (§8).
type seatActs struct {
mu sync.Mutex
bus caller
wake func()
logf func(string, ...any)
export map[string]exportCheck
// calls is, by pair (node and module, or node and provision for a pin), what the last call for it came to.
calls map[string]string
}
// exportCheck is the export read again for one proposal: running, its answer, or why it was not read.
type exportCheck struct {
running bool
export map[string]delivery.Placement
err string
}
func newSeatActs(bus caller, wake func(), logf func(string, ...any)) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{}}
}
// exportOf is the export read again for a proposal, asked once in the background.
func (a *seatActs) exportOf(id string) exportCheck {
a.mu.Lock()
defer a.mu.Unlock()
if c, ok := a.export[id]; ok {
return c
}
a.export[id] = exportCheck{running: true}
go func() {
defer a.wake()
raw, err := a.bus.Call(exportCall, map[string]any{})
c := exportCheck{}
if err != nil {
c.err = "the controller's placement-export did not answer: " + err.Error()
} else if c.export, err = assign.ExportOf(raw); err != nil {
c.err = err.Error()
}
a.mu.Lock()
a.export[id] = c
a.mu.Unlock()
}()
return exportCheck{running: true}
}
// forget drops what is kept of a proposal's export, so a later warrant reads it again.
func (a *seatActs) forget(id string) {
if a == nil {
return
}
a.mu.Lock()
delete(a.export, id)
a.mu.Unlock()
}
// record records once, in the controller's hand-act log, that placement was written on the operator's warrant (§5): the
// router's record of the teller's ask says who chose and how (ADR 0274).
func (a *seatActs) record(warrant string) {
if a == nil || warrant == "" {
return
}
ask, _, _ := strings.Cut(warrant, "@")
go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil {
a.logf("the placement's warrant %s is not recorded in the hand-act log: %v", ask, err)
}
}()
}
// handAct records a hand-back in the controller's hand-act log (§7).
func (a *seatActs) handAct(prop delivery.PlacementProposal) {
go func() {
if _, err := a.bus.Call(handActCall, map[string]any{"what": "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "),
"why": prop.Why, "cause": "asked by " + prop.By + ", approved by the operator"}); err != nil {
a.logf("the hand-back %s is not recorded in the hand-act log: %v", prop.ID, err)
}
}()
}
// callAll makes the controller's calls in order in the background, each said in the journal and kept by its pair.
func (a *seatActs) callAll(calls []assign.ControllerCall) {
if len(calls) == 0 {
return
}
a.mu.Lock()
for _, c := range calls {
a.calls[c.Node+"."+c.Module] = c.Key() + " is asked"
}
a.mu.Unlock()
go func() {
defer a.wake()
for _, c := range calls {
words := c.Key() + " answered"
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+"."+c.Verb, c.Args); err != nil {
words = c.Key() + " refused: " + err.Error()
}
a.logf("keeping the controller in step with placement: %s", words)
a.mu.Lock()
a.calls[c.Node+"."+c.Module] = words
a.mu.Unlock()
}
}()
}
// callOf is what the last call for a pair came to, "" when none was made.
func (a *seatActs) callOf(node, module string) string {
if a == nil {
return ""
}
a.mu.Lock()
defer a.mu.Unlock()
return a.calls[node+"."+module]
}
// withCalls names, on each pair the two sources do not both name, the call the assigner made for it (§8).
func (p *passer) withCalls(pairs []assign.Pair) []assign.Pair {
for i := range pairs {
pairs[i].Call = p.seat.callOf(pairs[i].Node, pairs[i].Module)
}
return pairs
}
// sayWaits raises, once, that the seat waits for the states not yet bound, and clears it once, when all are.
func (p *passer) sayWaits(unbound []string) {
c := assign.SeatWaitsCondition(unbound)
state := string(c.State) + c.Summary
if p.say == nil || p.said[c.Key] && p.saidWaits == state {
return
}
if c.State == asks.StateCleared && p.saidWaits == "" {
return // nothing was raised by this process
}
if err := p.say(c); err != nil {
p.logf("that the seat waits for its states could not be said, and is said at the next pass: %v", err)
return
}
p.said[c.Key], p.saidWaits = true, state
}
// seatPlacements is every node's placement key and its revision, as the watch holds them.
func (p *passer) seatPlacements() (map[string]delivery.Placement, map[string]uint64) {
out, revs := map[string]delivery.Placement{}, map[string]uint64{}
for key, value := range p.w.read(delivery.PlacementBucket) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
if pl, err := delivery.DecodePlacement(value); err == nil && pl.Node == node {
out[node], revs[node] = pl, p.w.metaOf(delivery.PlacementBucket, key).revision
}
}
return out, revs
}
// candidatesOf is a proposal's candidates by node, with their keys' revisions and the newest write.
func (p *passer) candidatesOf(id string) (map[string]delivery.PlacementCandidate, map[string]uint64, time.Time) {
out, revs := map[string]delivery.PlacementCandidate{}, map[string]uint64{}
var newest time.Time
for key, value := range p.w.read(delivery.PlacementCandidatesBucket) {
pid, node, err := delivery.ParsePlacementCandidateKey(key)
if err != nil || pid != id {
continue
}
c, err := delivery.DecodePlacementCandidate(value)
if err != nil {
continue
}
meta := p.w.metaOf(delivery.PlacementCandidatesBucket, key)
out[node], revs[node] = c, meta.revision
if meta.at.After(newest) {
newest = meta.at
}
}
return out, revs, newest
}
// answersTo is, by node, each node's answer to the proposal in its `<node>.node` have.
func (p *passer) answersTo(id string) map[string]*delivery.CandidateAnswer {
out := map[string]*delivery.CandidateAnswer{}
for key, value := range p.w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
}
h, err := delivery.DecodeHaveNode(value)
if err != nil {
continue
}
if a, ok := h.Candidates[id]; ok {
out[node] = &a
}
}
return out
}
// placementSeat takes every open proposal one step further.
func (p *passer) placementSeat(ctx context.Context) {
for _, b := range seatBuckets {
if p.kvs[b] == nil {
return
}
}
if p.saidProposed == nil {
p.saidProposed, p.intended = map[string]bool{}, map[string]map[string]map[string]bool{}
}
current, revs := p.seatPlacements()
ids := make([]string, 0)
raws := p.w.read(delivery.PlacementProposalsBucket)
for id := range raws {
ids = append(ids, id)
}
sort.Strings(ids)
for _, id := range ids {
prop, err := delivery.DecodePlacementProposal(raws[id])
if err != nil || prop.ID != id {
p.logf("the proposal %s cannot be read, so it is left: %v", id, err)
continue
}
p.stepProposal(ctx, prop, p.w.metaOf(delivery.PlacementProposalsBucket, id).revision, current, revs)
}
}
// pulse writes a bound's key with its time to live (§4): its expiry, heard on the watch as the server's delete marker,
// is the event that ends the wait. Written again over one that stands, so its time starts anew.
func (p *passer) pulse(ctx context.Context, key string, ttl time.Duration) {
kv := p.kvs[delivery.PlacementPulsesBucket]
_ = kv.Delete(ctx, key)
rev, err := kv.Create(ctx, key, []byte(`{}`), jetstream.KeyTTL(ttl))
if err != nil {
p.logf("the bound %s is not written, so its wait ends at once: %v", key, err)
return
}
p.w.putEntry(delivery.PlacementPulsesBucket, key, []byte(`{}`), rev, p.clock())
}
// pulsing says whether a bound's key stands.
func (p *passer) pulsing(key string) bool { return p.w.has(delivery.PlacementPulsesBucket, key) }
// deletePulses deletes every bound of a proposal.
func (p *passer) deletePulses(ctx context.Context, id string) {
for key := range p.w.read(delivery.PlacementPulsesBucket) {
if strings.HasSuffix(key, "."+id) || strings.Contains(key, "."+id+".") {
if err := p.kvs[delivery.PlacementPulsesBucket].Delete(ctx, key); err == nil {
p.w.put(delivery.PlacementPulsesBucket, key, nil)
}
}
}
}
// ask raises (or raises again) a proposal's authorising ask and its bound, and answers the proposal asked.
func (p *passer) ask(ctx context.Context, prop delivery.PlacementProposal, akey, brief string) (delivery.PlacementProposal, bool) {
a, err := assign.PlacementAsk(akey, prop, brief, assign.PlacementAsking(prop, p.clock()), p.clock())
if err != nil {
p.logf("the ask about the proposal %s is not written: %v", prop.ID, err)
return prop, false
}
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
if pk, err := delivery.PulseAskKey(prop.ID); err == nil {
p.pulse(ctx, pk, a.Ask.Expires.Sub(p.clock()))
}
prop.State, prop.Ask = delivery.ProposalAsked, a.Ask.ID
return prop, true
}
// stepProposal takes one proposal one step further.
func (p *passer) stepProposal(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, revs map[string]uint64) {
if !p.saidProposed[prop.ID] && prop.State == delivery.ProposalAnswering && prop.Verdict == nil {
p.sayEvent(delivery.PlacementProposedEvent, delivery.PlacementEvent{Proposal: prop}, prop.ID)
p.saidProposed[prop.ID] = true
}
if prop.Withdrawn != "" {
p.endProposal(ctx, prop, prop.Withdrawn)
return
}
akey, _ := delivery.PlacementAskKey(prop.ID)
again := !prop.Again.IsZero() && (prop.Verdict == nil || prop.Again.After(prop.Verdict.At))
if prop.State == delivery.ProposalAsked {
askPulse, _ := delivery.PulseAskKey(prop.ID)
switch {
case p.answeredWith(akey, assign.OptionYes):
if prop.HandBack {
p.handBack(ctx, prop, akey, revs)
} else {
p.warranted(ctx, prop, rev, akey, current, revs)
}
return
case p.answeredWith(akey, assign.OptionNo):
p.endProposal(ctx, prop, "declined by the operator")
return
case !p.pulsing(askPulse) && !again:
p.endProposal(ctx, prop, "its ask expired unanswered")
return
}
}
if prop.HandBack {
// The operator's alone (§7): no node answers it; its ask is raised at once, and again on again.
if prop.State != delivery.ProposalAsked || again {
brief := fmt.Sprintf("%s (asked by %s): every node's placement key is deleted, %s, and each node returns to the "+
"controller's compositions; the history is kept", prop.Why, prop.By, strings.Join(prop.Nodes(), ", "))
now := p.clock().UTC()
prop.Verdict = &delivery.PlacementVerdict{State: delivery.VerdictPass, At: now}
if prop, ok := p.ask(ctx, prop, akey, brief); ok {
p.writeProposal(ctx, prop, rev)
}
}
return
}
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key it was laid over moved since")
return
}
cands, candRevs, written := p.candidatesOf(prop.ID)
if len(prop.Must) == 0 {
p.mustAnswer(ctx, prop, rev, current, cands)
return
}
for _, node := range prop.Must {
if _, ok := cands[node]; !ok {
return // a candidate just written is not heard yet: its watch wakes the next pass
}
}
if written.After(prop.Written) {
prop.Written = written
}
pulsing := map[string]bool{}
for _, node := range prop.Must {
if k, err := delivery.PulseAnswerKey(prop.ID, node); err == nil {
pulsing[node] = p.pulsing(k)
}
}
v := judge.JudgeAnswers(prop, p.intendedOf(prop, current, cands), p.answersTo(prop.ID), candRevs, p.w.online(), pulsing, p.clock())
if prop.Verdict != nil && sameVerdict(*prop.Verdict, v) && !again {
return
}
prop.Verdict = &v
switch v.State {
case delivery.VerdictFail:
prop.State, prop.Ask = delivery.ProposalFailed, ""
case delivery.VerdictWaits:
prop.State, prop.Ask = delivery.ProposalAnswering, ""
case delivery.VerdictPass:
if prop.State != delivery.ProposalAsked || again {
var ok bool
if prop, ok = p.ask(ctx, prop, akey, judge.Brief(prop, v)); !ok {
return
}
if again {
prop.Words = "judged and asked again (again)"
}
}
}
if p.writeProposal(ctx, prop, rev) {
p.sayEvent(delivery.PlacementJudgedEvent, delivery.PlacementEvent{Proposal: prop},
fmt.Sprintf("%s.judged.%d", prop.ID, v.At.UnixNano()))
}
}
// sameVerdict says two verdicts say the same, their time aside.
func sameVerdict(a, b delivery.PlacementVerdict) bool {
a.At, b.At = time.Time{}, time.Time{}
return reflect.DeepEqual(a, b)
}
// moved says whether a key the proposal changes is no longer at the revision it was laid over.
func moved(prop delivery.PlacementProposal, revs map[string]uint64) bool {
for node, base := range prop.Base {
if revs[node] != base {
return true
}
}
return false
}
// intended is, by proposal and the revisions it was laid over, what Intended computed, kept while this process runs.
func (p *passer) intendedOf(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) map[string]map[string]bool {
key := fmt.Sprint(prop.ID, prop.Base)
if got, ok := p.intended[key]; ok {
return got
}
got, _, _ := p.compute(prop, current, cands)
p.intended[key] = got
return got
}
// compute composes every node with the current placement and with the proposal laid over it (judge.Intended).
func (p *passer) compute(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) (map[string]map[string]bool, []string, []string) {
laid := map[string]*delivery.Placement{}
for _, node := range prop.Nodes() {
if c, ok := cands[node]; ok && !c.Reached {
laid[node] = c.Placement
}
}
get := kvGetter(p.kvs[delivery.ManifestsBucket])
r := worldReader{w: p.w, manifests: get}
var now, with *compose.Input
if in, err := frombus.Input(context.Background(), r, frombus.Mesh{}, frombus.Scope{Every: true}); err == nil {
now = &in
}
if in, err := frombus.Input(context.Background(), r, frombus.Mesh{}, frombus.Scope{Every: true, Placements: laid}); err == nil {
with = &in
}
nodes := map[string]bool{}
for n := range current {
nodes[n] = true
}
if records, err := r.Records(context.Background()); err == nil {
for n := range records {
nodes[n] = true
}
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
fps := map[string]string{}
for name, m := range p.w.trunk() {
if m.FP2 != "" {
fps[name] = m.FP2
}
}
return judge.Intended(now, with, current, laid, names, fps)
}
// mustAnswer finds the nodes the proposal must hear from (§3), writes a reached candidate for each whose key it does not
// change, and writes them into the proposal.
func (p *passer) mustAnswer(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) {
for _, node := range prop.Nodes() {
if _, ok := cands[node]; !ok {
return // the verbs write the candidates before the proposal: wait to hear them
}
}
intended, must, notes := p.compute(prop, current, cands)
p.intended[fmt.Sprint(prop.ID, prop.Base)] = intended
for _, node := range must {
if _, ok := cands[node]; ok {
continue
}
key, err := delivery.PlacementCandidateKey(prop.ID, node)
if err != nil {
continue
}
c := delivery.PlacementCandidate{Proposal: prop.ID, Reached: true}
if pl, ok := current[node]; ok {
c.Placement = &pl
}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
for _, node := range must {
if k, err := delivery.PulseAnswerKey(prop.ID, node); err == nil {
p.pulse(ctx, k, judge.AnswerBound)
}
}
prop.Must = must
if len(notes) > 0 {
prop.Words = strings.TrimPrefix(prop.Words+"; "+strings.Join(notes, "; "), "; ")
}
p.writeProposal(ctx, prop, rev)
}
// layAgain lays a proposal over the keys as they stand now (§5: a key moved): its candidates are written again, the
// nodes that must answer found again, and it is judged and asked again; a proposal that no longer lays ends.
func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, revs map[string]uint64, why string) {
held, recorded := p.lays(ctx)
again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes,
Current: current, Revisions: revs, Held: held, Recorded: recorded,
Seed: prop.Seed, ExportDigest: prop.ExportDigest, Leased: p.w.handedAway()})
if len(findings) > 0 {
p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; "))
return
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
for _, node := range again.Nodes() {
key, _ := delivery.PlacementCandidateKey(prop.ID, node)
c := delivery.PlacementCandidate{Proposal: prop.ID, Placement: laid[node]}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
again.Written, again.Words = p.clock().UTC(), why+": laid again over the keys as they stand, to be judged and asked again"
p.seat.forget(prop.ID)
p.writeProposal(ctx, again, rev)
}
// lays is what a proposal is checked against, as the verbs check it: the modules the indexer holds, by key, and the
// nodes with a record.
func (p *passer) lays(ctx context.Context) (func(string) bool, func(string) bool) {
held := map[string]bool{}
for key := range p.w.read(delivery.ModulesBucket) {
if name, err := delivery.ParseModulesKey(key); err == nil {
held[name] = true
}
}
records, _ := worldReader{w: p.w}.Records(ctx)
return func(m string) bool { return held[m] }, func(n string) bool { _, ok := records[n]; return ok }
}
// warranted writes an approved proposal (§5): for the seed, after the export is read again and found unchanged (§7);
// each changed key by compare-and-set at the revision it was laid over; then the warrant recorded, the candidates and the
// proposal deleted, placement-changed said, and the controller kept in step (§8).
func (p *passer) warranted(ctx context.Context, prop delivery.PlacementProposal, rev uint64, akey string, current map[string]delivery.Placement, revs map[string]uint64) {
if err := p.performs(akey, assign.PlacementAct(prop)); err != nil {
p.endProposal(ctx, prop, "its warrant does not bind these changes, so nothing is written: "+err.Error())
return
}
answer, _ := p.w.answer(akey)
if prop.Seed {
if p.seat == nil {
return
}
c := p.seat.exportOf(prop.ID)
switch {
case c.running:
return // its answer wakes the next pass
case c.err != "":
if words := "the export cannot be read again at the warrant: " + c.err; prop.Words != words {
prop.Words = words
p.writeProposal(ctx, prop, rev)
}
return
case delivery.ExportDigest(c.export) != prop.ExportDigest:
p.endProposal(ctx, prop, "the controller's export changed since the seed was proposed: the new export is proposed as the seed")
p.proposeSeed(ctx, prop, c.export)
return
}
}
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key moved since it was approved, so its warrant is spent")
return
}
cands, _, _ := p.candidatesOf(prop.ID)
laid := map[string]*delivery.Placement{}
for _, node := range prop.Nodes() {
laid[node] = cands[node].Placement
}
writes, err := assign.Writes(prop, laid)
if err != nil {
p.logf("the proposal %s cannot be written: %v", prop.ID, err)
return
}
kv := p.kvs[delivery.PlacementBucket]
var changes []delivery.PlacementChange
writtenLaid := map[string]*delivery.Placement{}
after := maps.Clone(current)
afterRevs := maps.Clone(revs)
for _, w := range writes {
var rev2 uint64
var err error
switch {
case w.Value == nil:
err = kv.Delete(ctx, w.Key, jetstream.LastRevision(w.Base))
default:
raw, encErr := delivery.EncodePlacement(*w.Value)
if encErr != nil {
err = encErr
} else if w.Base == 0 {
rev2, err = kv.Create(ctx, w.Key, raw)
} else {
rev2, err = kv.Update(ctx, w.Key, raw, w.Base)
}
}
if err != nil {
// Another writer came first (a hand-back): what is written stands and is followed through; the rest is laid
// again over the keys as this write left them.
p.logf("placement %s is not written at revision %d (%v): the rest of the proposal %s is laid again", w.Key, w.Base, err, prop.ID)
p.followWrite(prop, answer, changes, current, writtenLaid)
if rest, ok := assign.Rest(prop, writtenLaid); ok {
p.layAgain(ctx, rest, rev, after, afterRevs, "a key moved while it was written, after "+strings.Join(sortedNodes(writtenLaid), ", ")+" were")
} else {
p.endProposal(ctx, prop, "a key moved while it was written: "+err.Error())
}
return
}
before := current[w.Node]
now := delivery.Placement{Node: w.Node}
if w.Value != nil {
now = *w.Value
after[w.Node] = *w.Value
} else {
delete(after, w.Node)
}
afterRevs[w.Node] = rev2
c := delivery.ChangeOf(before, now)
c.Before, c.After = w.Base, rev2
changes = append(changes, c)
writtenLaid[w.Node] = w.Value
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.followWrite(prop, answer, changes, current, writtenLaid)
}
// followWrite follows the keys a warrant wrote: the warrant recorded, placement-changed said, and, until the lease
// write, the controller's assignments kept in step for them (§8). The seed calls no controller verb: it is what the
// controller holds already (§9).
func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange,
before map[string]delivery.Placement, written map[string]*delivery.Placement) {
if len(changes) == 0 {
return
}
p.seat.record(answer.Warrant)
p.sayChanged(prop, answer, changes)
if p.seat != nil && !p.w.handedAway() && !prop.Seed {
p.seat.callAll(assign.ControllerCalls(before, written, prop.ID, answer.Warrant))
}
}
func sortedNodes(m map[string]*delivery.Placement) []string {
out := make([]string, 0, len(m))
for n := range m {
out = append(out, n)
}
sort.Strings(out)
return out
}
// handBack deletes every key on the operator's warrant (§7) at the revision the hand-back read, records it in the
// controller's hand-act log and says placement-changed; every node returns to the controller's compositions. A key moved
// since ends it, and the operator hands back again.
func (p *passer) handBack(ctx context.Context, prop delivery.PlacementProposal, akey string, revs map[string]uint64) {
if err := p.performs(akey, assign.PlacementAct(prop)); err != nil {
p.endProposal(ctx, prop, "its warrant does not bind this hand-back, so nothing is deleted: "+err.Error())
return
}
answer, _ := p.w.answer(akey)
current, _ := p.seatPlacements()
var changes []delivery.PlacementChange
for _, node := range prop.Nodes() {
key, _ := delivery.PlacementKey(node)
if err := p.kvs[delivery.PlacementBucket].Delete(ctx, key, jetstream.LastRevision(prop.Base[node])); err != nil {
p.sayChanged(prop, answer, changes)
p.endProposal(ctx, prop, fmt.Sprintf("%s moved since the hand-back was asked (%v): hand back again", key, err))
return
}
c := delivery.ChangeOf(current[node], delivery.Placement{Node: node})
c.Before = prop.Base[node]
changes = append(changes, c)
}
if p.seat != nil {
p.seat.handAct(prop)
}
p.seat.record(answer.Warrant)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.deletePulses(ctx, prop.ID)
p.sayChanged(prop, answer, changes)
}
// sayChanged says placement-changed for the keys written.
func (p *passer) sayChanged(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange) {
if len(changes) == 0 {
return
}
at := p.clock().UTC()
if _, when, ok := strings.Cut(answer.Warrant, "@"); ok {
if t, err := time.Parse(time.RFC3339Nano, when); err == nil {
at = t
}
}
delivery.SortChanges(changes)
p.sayEvent(delivery.PlacementChangedEvent, delivery.PlacementEvent{Proposal: prop, Nodes: changes,
Warrant: &delivery.PlacementWarrant{Warrant: answer.Warrant, Words: answer.Words, At: at}}, prop.ID+".changed")
}
// proposeSeed proposes an export as the seed again, by the same proposer and why.
func (p *passer) proposeSeed(ctx context.Context, was delivery.PlacementProposal, export map[string]delivery.Placement) {
held, recorded := p.lays(ctx)
now := p.clock()
seed, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: was.By, Why: was.Why, At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: p.w.handedAway()})
if len(findings) > 0 {
p.logf("the new export is not proposed as the seed: %s", strings.Join(findings, "; "))
return
}
for _, node := range seed.Nodes() {
key, _ := delivery.PlacementCandidateKey(seed.ID, node)
c := delivery.PlacementCandidate{Proposal: seed.ID, Placement: laid[node]}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
seed.Written = now.UTC()
p.writeProposal(ctx, seed, 0)
}
// endProposal ends a proposal: its ask, its candidates and its key deleted, and placement-withdrawn said carrying it
// whole with why it ended.
func (p *passer) endProposal(ctx context.Context, prop delivery.PlacementProposal, why string) {
if akey, err := delivery.PlacementAskKey(prop.ID); err == nil && p.w.has(delivery.AssignerAsksBucket, akey) {
if err := p.kvs[delivery.AssignerAsksBucket].Delete(ctx, akey); err == nil {
p.w.put(delivery.AssignerAsksBucket, akey, nil)
}
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
if err := p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID); err != nil && !errors.Is(err, context.Canceled) {
p.logf("the proposal %s is not deleted, and is ended at the next pass: %v", prop.ID, err)
return
}
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.seat.forget(prop.ID)
p.logf("the proposal %s ends: %s", prop.ID, why)
p.sayEvent(delivery.PlacementWithdrawnEvent, delivery.PlacementEvent{Proposal: prop, Ended: why}, prop.ID+".withdrawn")
}
// deleteCandidates deletes every candidate of a proposal.
func (p *passer) deleteCandidates(ctx context.Context, id string) {
for key := range p.w.read(delivery.PlacementCandidatesBucket) {
if pid, _, err := delivery.ParsePlacementCandidateKey(key); err == nil && pid == id {
if err := p.kvs[delivery.PlacementCandidatesBucket].Delete(ctx, key); err == nil {
p.w.put(delivery.PlacementCandidatesBucket, key, nil)
}
}
}
}
// writeProposal writes a proposal over the revision the pass read, by compare-and-set: a verb that marked it meanwhile
// (withdraw, again) wins, and the next pass reads that. It says whether it wrote.
func (p *passer) writeProposal(ctx context.Context, prop delivery.PlacementProposal, rev uint64) bool {
raw, err := delivery.EncodePlacementProposal(prop)
if err != nil {
p.logf("the proposal %s is not written: %v", prop.ID, err)
return false
}
kv := p.kvs[delivery.PlacementProposalsBucket]
var at uint64
if rev == 0 {
at, err = kv.Create(ctx, prop.ID, raw)
} else {
at, err = kv.Update(ctx, prop.ID, raw, rev)
}
if err != nil {
if !errors.Is(err, context.Canceled) {
p.logf("the proposal %s is not written, and is taken further at the next pass: %v", prop.ID, err)
}
return false
}
p.w.putEntry(delivery.PlacementProposalsBucket, prop.ID, raw, at, p.clock())
return true
}
// say says one of the seat's events, said in the journal when the bus refuses it.
func (p *passer) sayEvent(event string, body delivery.PlacementEvent, id string) {
if p.publish == nil {
return
}
if err := p.publish(event, body, id); err != nil {
p.logf("%s of %s is not said: %v", event, body.Proposal.ID, err)
}
}
@@ -0,0 +1,543 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-placement's seven verbs (novox/hq ADR 0567 §10), served through the tool runner beside the put tools: a
// process never serves (ADR 0436). They read every state the seat keeps and write what a person asks of it: a proposal
// and its candidates (propose, adopt), a proposal marked to end (withdraw) or to be judged again (again), and the keys
// deleted (hand-back). Everything that follows a proposal — composing it, judging the answers, the ask, the write at the
// warrant and the seat's events on its stream — is the assigner's process's (seat.go), started by these writes.
// seatStore is what the seat's verbs need of the states: the runtime's, or a test's.
type seatStore interface {
store
del(state, key string) error
history(state, key string) ([]stdio.StateRevision, error)
}
func (runtimeStore) del(state, key string) error { return stdio.State(state).Delete(key) }
func (runtimeStore) history(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
// The states the verbs read, as the runtime names them: the seat's own by their local names, the others'
// `<module>.<state>`.
const (
modulesState = delivery.IndexerModule + ".modules"
haveState = "node-engine.have"
)
// seatTools serves the seat's verbs over a store, asking the controller through ask (stdio.Ask, or a test's).
type seatTools struct {
st seatStore
ask func(key string, body any) (json.RawMessage, error)
now func() time.Time
}
func newSeatTools(st seatStore, ask func(string, any) (json.RawMessage, error)) *seatTools {
return &seatTools{st: st, ask: ask, now: time.Now}
}
// keyed is one placement key as a verb answers it.
type keyed struct {
Placement delivery.Placement `json:"placement"`
Revision uint64 `json:"revision"`
}
// placements reads every node's placement and its key's revision.
func (s *seatTools) placements() (map[string]delivery.Placement, map[string]uint64, error) {
keys, err := s.st.keys(delivery.PlacementState)
if err != nil {
return nil, nil, fmt.Errorf("placement cannot be listed: %w", err)
}
out, revs := map[string]delivery.Placement{}, map[string]uint64{}
for _, key := range keys {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
raw, rev, found, err := s.st.get(delivery.PlacementState, key)
if err != nil {
return nil, nil, fmt.Errorf("placement of %s cannot be read: %w", node, err)
}
if !found {
continue
}
p, err := delivery.DecodePlacement(raw)
if err != nil {
return nil, nil, fmt.Errorf("placement of %s cannot be read: %w", node, err)
}
out[node], revs[node] = p, rev
}
return out, revs, nil
}
// lays is what a proposal is checked against: the modules the indexer holds and the nodes with a record.
func (s *seatTools) lays() (func(string) bool, func(string) bool, error) {
mods, err := s.st.keys(modulesState)
if err != nil {
return nil, nil, fmt.Errorf("the indexer's modules cannot be listed: %w", err)
}
held := map[string]bool{}
for _, k := range mods {
if name, err := delivery.ParseModulesKey(k); err == nil {
held[name] = true
}
}
haves, err := s.st.keys(haveState)
if err != nil {
return nil, nil, fmt.Errorf("the nodes' records cannot be listed: %w", err)
}
recorded := map[string]bool{}
for _, k := range haves {
node, _, isNode, err := delivery.ParseHaveKey(k)
if err != nil || !isNode {
continue
}
if raw, _, found, err := s.st.get(haveState, k); err == nil && found {
if h, err := delivery.DecodeHaveNode(raw); err == nil && h.Record != nil {
recorded[node] = true
}
}
}
return func(m string) bool { return held[m] }, func(n string) bool { return recorded[n] }, nil
}
// proposal reads one open proposal.
func (s *seatTools) proposal(id string) (delivery.PlacementProposal, error) {
if err := delivery.CheckProposalID(id); err != nil {
return delivery.PlacementProposal{}, err
}
raw, _, found, err := s.st.get(delivery.PlacementProposalsState, id)
if err != nil {
return delivery.PlacementProposal{}, fmt.Errorf("the proposal %s cannot be read: %w", id, err)
}
if !found {
return delivery.PlacementProposal{}, fmt.Errorf("no proposal %s is open: mesh-placement.proposals lists those that are", id)
}
return delivery.DecodePlacementProposal(raw)
}
// write writes a proposal laid by assign.Propose: each candidate, then the proposal, whose write the assigner's process
// takes as the event to compose and judge it.
func (s *seatTools) write(p delivery.PlacementProposal, laid map[string]*delivery.Placement) error {
p.Written = s.now().UTC()
for _, node := range p.Nodes() {
key, err := delivery.PlacementCandidateKey(p.ID, node)
if err != nil {
return err
}
raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: p.ID, Placement: laid[node]})
if err != nil {
return err
}
if err := s.st.put(delivery.PlacementCandidatesState, key, raw); err != nil {
return fmt.Errorf("the candidate %s is not written: %w", key, err)
}
}
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return fmt.Errorf("the proposal %s is not written: %w", p.ID, err)
}
return nil
}
// proposed is what propose and adopt answer.
type proposed struct {
Proposal delivery.PlacementProposal `json:"proposal"`
Follow string `json:"follow"`
}
func followProposal(id string) string {
return fmt.Sprintf("the nodes it touches answer it; mesh-placement.proposals with id %q says where it stands, and the "+
"operator is asked to approve it once its verdict passes", id)
}
// changesOf reads propose's changes: a JSON array, given as one or as its text.
func changesOf(v any) ([]delivery.PlacementProposalChange, error) {
var raw []byte
switch c := v.(type) {
case string:
raw = []byte(c)
case nil:
return nil, errors.New("propose names its changes")
default:
var err error
if raw, err = json.Marshal(c); err != nil {
return nil, err
}
}
var out []delivery.PlacementProposalChange
dec := json.NewDecoder(strings.NewReader(string(raw)))
dec.DisallowUnknownFields()
if err := dec.Decode(&out); err != nil {
return nil, fmt.Errorf("the changes are no list of changes: %w", err)
}
return out, nil
}
// doPropose is the verb propose (§2).
func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
changes, err := changesOf(args["changes"])
if err != nil {
return nil, err
}
for _, c := range changes {
if c.Op == delivery.OpSeed {
return nil, errors.New("the seed is adopt's alone: propose takes place, take-off, pin, unpin, hold, unhold, decide and config")
}
}
current, revs, err := s.placements()
if err != nil {
return nil, err
}
held, recorded, err := s.lays()
if err != nil {
return nil, err
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Leased: s.leased()})
if len(findings) > 0 {
return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; "))
}
if err := s.write(p, laid); err != nil {
return nil, err
}
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// doWithdraw is the verb withdraw (§2): the proposal is marked, and the assigner deletes it and its candidates and says
// placement-withdrawn carrying it whole.
func (s *seatTools) doWithdraw(args map[string]any, by string) (any, error) {
why := arg(args, "why")
if why == "" {
return nil, errors.New("withdraw says why: it is kept in the event that ends the proposal")
}
p, err := s.proposal(arg(args, "id"))
if err != nil {
return nil, err
}
p.Withdrawn = fmt.Sprintf("withdrawn by %s: %s", by, why)
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the proposal %s is not withdrawn: %w", p.ID, err)
}
return map[string]any{"proposal": p.ID, "withdrawn": p.Withdrawn}, nil
}
// doAgain is the verb again (§10): the proposal is marked, and the assigner judges it now and raises its ask again
// when its verdict passes.
func (s *seatTools) doAgain(args map[string]any) (any, error) {
p, err := s.proposal(arg(args, "id"))
if err != nil {
return nil, err
}
p.Again = s.now().UTC()
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the proposal %s is not asked again: %w", p.ID, err)
}
return map[string]any{"proposal": p.ID, "again": p.Again, "follow": followProposal(p.ID)}, nil
}
// exportCall is the controller's verb the seed is read from, by request and reply on its seat (§7).
const exportCall = "seat:" + catalogue.ControllerSeatName + ".placement-export"
// doAdopt is the verb adopt (§7): the controller's export proposed whole as the seed.
func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
current, _, err := s.placements()
if err != nil {
return nil, err
}
if len(current) > 0 {
return nil, fmt.Errorf("placement is taken over already (%d keys): propose a change instead, or hand it back first", len(current))
}
if ids, err := s.st.keys(delivery.PlacementProposalsState); err == nil {
for _, id := range ids {
if p, err := s.proposal(id); err == nil && p.Seed {
return nil, fmt.Errorf("the seed %s is open: withdraw it first, or answer its ask", id)
}
}
}
raw, err := s.ask(exportCall, map[string]any{})
if err != nil {
return nil, fmt.Errorf("the controller's placement-export did not answer: %w", err)
}
export, err := assign.ExportOf(raw)
if err != nil {
return nil, err
}
held, recorded, err := s.lays()
if err != nil {
return nil, err
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: s.leased()})
if len(findings) > 0 {
return nil, fmt.Errorf("the seed is refused: %s", strings.Join(findings, "; "))
}
if err := s.write(p, laid); err != nil {
return nil, err
}
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// leased says the controller's lease is handed away (mesh-sdk buscompose.LeaseHandedAway, ADR 0561 §3; issue 774): the lease
// write of ADR 0547 that ends keeping the controller in step (ADR 0567 §8). A lease not read, a controller's, expired or
// unreadable, is not handed away.
func (s *seatTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// handActCall records a hand act in the controller's hand-act log.
const handActCall = "seat:" + catalogue.ControllerSeatName + ".hand-act"
// doHandBack is the verb hand-back (§7): the operator's alone, so it is asked as a proposal is. It writes a hand-back
// proposal naming every key at its revision; the assigner raises its authorising ask and, on its warrant, deletes the keys,
// records the hand act and says placement-changed; every node returns to the controller's compositions.
func (s *seatTools) doHandBack(args map[string]any, by string) (any, error) {
if s.leased() {
return nil, errors.New("placement is the mesh's alone since the lease write: there is no controller to hand it back to")
}
why := arg(args, "why")
if why == "" {
return nil, errors.New("hand-back says why: it is recorded with the hand act")
}
if by == "" {
return nil, errors.New("the runtime names no caller: a hand-back is made by someone")
}
_, revs, err := s.placements()
if err != nil {
return nil, err
}
if len(revs) == 0 {
return nil, errors.New("there is no placement key to hand back")
}
now := s.now()
p := delivery.PlacementProposal{ID: delivery.NewProposalID(now), By: by, Why: why, At: now.UTC(), Written: now.UTC(),
HandBack: true, Base: revs, State: delivery.ProposalAnswering}
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the hand-back is not written: %w", err)
}
return proposed{Proposal: p, Follow: "the operator is asked to approve it; on the warrant every key is deleted and each " +
"node returns to the controller's compositions"}, nil
}
// placementAnswer is what the verb placement answers.
type placementAnswer struct {
Keys map[string]keyed `json:"keys,omitempty"`
Nodes []string `json:"nodes,omitempty"`
Count int `json:"count"`
History int `json:"history-per-key"`
Changes []historyEntry `json:"changes,omitempty"`
Older string `json:"older,omitempty"`
}
// historyEntry is one change of a key in the state's revisions.
type historyEntry struct {
Node string `json:"node"`
Revision uint64 `json:"revision"`
At time.Time `json:"at"`
Change delivery.PlacementChange `json:"change"`
Deleted bool `json:"deleted,omitempty"`
}
// doPlacement is the verb placement (§10, §6): one node's key, one module's nodes, or every key; with history, the
// changes in order from the revisions the state keeps.
func (s *seatTools) doPlacement(args map[string]any) (any, error) {
all, revs, err := s.placements()
if err != nil {
return nil, err
}
node, module := arg(args, "node"), arg(args, "module")
out := placementAnswer{Keys: map[string]keyed{}, Count: len(all), History: delivery.PlacementHistory}
for n, p := range all {
if node != "" && n != node {
continue
}
if module != "" && !strings.Contains(" "+strings.Join(p.Assigned(), " ")+" ", " "+module+" ") {
continue
}
out.Keys[n] = keyed{Placement: p, Revision: revs[n]}
out.Nodes = append(out.Nodes, n)
}
sort.Strings(out.Nodes)
if node != "" && len(out.Keys) == 0 && arg(args, "history") != "true" {
return nil, fmt.Errorf("%s has no placement key", node)
}
if arg(args, "history") != "true" {
return out, nil
}
nodes := out.Nodes
if node != "" {
nodes = []string{node}
} else if module == "" {
nodes = nil
for n := range all {
nodes = append(nodes, n)
}
sort.Strings(nodes)
}
for _, n := range nodes {
key, err := delivery.PlacementKey(n)
if err != nil {
return nil, err
}
revisions, err := s.st.history(delivery.PlacementState, key)
if err != nil {
return nil, fmt.Errorf("the history of %s cannot be read: %w", key, err)
}
var before delivery.Placement
for _, r := range revisions {
e := historyEntry{Node: n, Revision: r.Revision, At: r.At}
after := delivery.Placement{}
if r.Op == "" || r.Op == "put" || r.Op == "PUT" {
if p, err := delivery.DecodePlacement(r.Value); err == nil {
after = p
}
} else {
e.Deleted = true
}
e.Change = delivery.ChangeOf(before, after)
e.Change.Before, e.Change.After = 0, r.Revision
before = after
if module != "" && !strings.Contains(" "+strings.Join(append(append(e.Change.Added, e.Change.Removed...), e.Change.Changed...), " ")+" ", " "+module+" ") {
continue
}
out.Changes = append(out.Changes, e)
}
}
out.Older = fmt.Sprintf("the state keeps %d revisions of each key; an older change is in its placement-changed event on "+
"the assigner's stream %s, kept %d days", delivery.PlacementHistory, delivery.PlacementStream, delivery.PlacementStreamDays)
return out, nil
}
// openProposal is one open proposal as the verb proposals answers it: the proposal (its changes, base revisions,
// verdict with the differences and the answers it judged, its ask), its candidates by node with their revisions, and
// every node's answer now.
type openProposal struct {
Proposal delivery.PlacementProposal `json:"proposal"`
Candidates map[string]uint64 `json:"candidates"`
Answers map[string]delivery.CandidateAnswer `json:"answers,omitempty"`
}
// doProposals is the verb proposals (§10).
func (s *seatTools) doProposals(args map[string]any) (any, error) {
ids, err := s.st.keys(delivery.PlacementProposalsState)
if err != nil {
return nil, fmt.Errorf("the proposals cannot be listed: %w", err)
}
cands, err := s.st.keys(delivery.PlacementCandidatesState)
if err != nil {
return nil, fmt.Errorf("the candidates cannot be listed: %w", err)
}
haves, err := s.st.keys(haveState)
if err != nil {
return nil, fmt.Errorf("the nodes' haves cannot be listed: %w", err)
}
answers := map[string]map[string]delivery.CandidateAnswer{} // by proposal, then node
for _, k := range haves {
node, _, isNode, err := delivery.ParseHaveKey(k)
if err != nil || !isNode {
continue
}
raw, _, found, err := s.st.get(haveState, k)
if err != nil || !found {
continue
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil {
continue
}
for id, a := range h.Candidates {
if answers[id] == nil {
answers[id] = map[string]delivery.CandidateAnswer{}
}
answers[id][node] = a
}
}
sort.Strings(ids)
out := []openProposal{}
for _, id := range ids {
if want := arg(args, "id"); want != "" && want != id {
continue
}
p, err := s.proposal(id)
if err != nil {
return nil, err
}
o := openProposal{Proposal: p, Candidates: map[string]uint64{}, Answers: answers[id]}
for _, k := range cands {
pid, node, err := delivery.ParsePlacementCandidateKey(k)
if err != nil || pid != id {
continue
}
if _, rev, found, err := s.st.get(delivery.PlacementCandidatesState, k); err == nil && found {
o.Candidates[node] = rev
}
}
out = append(out, o)
}
if arg(args, "id") != "" && len(out) == 0 {
return nil, fmt.Errorf("no proposal %s is open", arg(args, "id"))
}
return map[string]any{"proposals": out}, nil
}
// tools are the seat's verbs as the runtime serves them, each named <seat>.<verb> (the tool runner serves a seat's verb
// only from a tool of that name), with the seat's own schemas.
func (s *seatTools) tools() []stdio.Tool {
seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat)
input := map[string]map[string]any{}
description := map[string]string{}
for _, v := range seat.Serves {
input[v.Name], description[v.Name] = v.Input, v.Description
}
run := map[string]func(map[string]any) (any, error){
catalogue.PlacementVerbPlacement: s.doPlacement,
catalogue.PlacementVerbProposals: s.doProposals,
catalogue.PlacementVerbPropose: func(a map[string]any) (any, error) { return s.doPropose(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbWithdraw: func(a map[string]any) (any, error) { return s.doWithdraw(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbAgain: s.doAgain,
catalogue.PlacementVerbAdopt: func(a map[string]any) (any, error) { return s.doAdopt(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbHandBack: func(a map[string]any) (any, error) { return s.doHandBack(a, stdio.CallerOf(a)) },
}
var out []stdio.Tool
for _, v := range seat.Serves {
out = append(out, stdio.Tool{Name: catalogue.PlacementSeat + "." + v.Name, Description: description[v.Name], Input: input[v.Name], Run: run[v.Name]})
}
return out
}
@@ -0,0 +1,167 @@
package main
import (
"encoding/json"
"os"
"slices"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func (s memStore) del(state, key string) error {
delete(s[state].values, key)
return nil
}
func (s memStore) history(state, key string) ([]stdio.StateRevision, error) {
v, ok := s[state].values[key]
if !ok {
return nil, nil
}
return []stdio.StateRevision{{Key: key, Value: v, Revision: uint64(len(v)), Op: "put"}}, nil
}
// ADR 0567 §2, §7, §10: the seat's seven verbs over the states. propose lays a proposal and writes it with its candidates,
// or refuses naming every finding and writes nothing; withdraw and again mark it for the holder; adopt proposes the
// controller's export whole as the seed, once; hand-back records the hand act and deletes every key; placement and
// proposals read every state the seat keeps, and history names the stream for what the state no longer keeps.
func TestTheSeatsVerbs(t *testing.T) {
anchor := delivery.Placement{V: delivery.Version, Node: "anchor", Modules: []delivery.PlacedModule{{Module: "web"}}}
st := memStore{}
for _, state := range []string{delivery.PlacementState, delivery.PlacementProposalsState, delivery.PlacementCandidatesState, modulesState, haveState} {
st[state] = &memKV{values: map[string][]byte{}}
}
for _, m := range []string{"web", "shell"} {
st[modulesState].values[m] = []byte("{}")
}
for _, n := range []string{"anchor", "laptop"} {
raw, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: n}})
st[haveState].values[n+".node"] = raw
}
export, _ := json.Marshal(map[string]any{"anchor": anchor})
var asked []string
s := newSeatTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
if key == exportCall {
out, _ := json.Marshal(map[string]any{"ok": true, "output": string(export)})
return out, nil
}
return json.RawMessage(`{"ok":true}`), nil
})
s.now = func() time.Time { return time.Date(2026, 10, 28, 20, 0, 0, 0, time.UTC) }
// adopt: the export proposed whole as the seed, once.
got, err := s.doAdopt(map[string]any{"why": "take over"}, "operator@laptop")
if err != nil {
t.Fatal(err)
}
seed := got.(proposed).Proposal
if !seed.Seed || seed.ExportDigest != delivery.ExportDigest(map[string]delivery.Placement{"anchor": anchor}) || !slices.Equal(asked, []string{exportCall}) ||
len(st[delivery.PlacementCandidatesState].values) != 1 || st[delivery.PlacementProposalsState].values[seed.ID] == nil {
t.Fatalf("adopt = %+v, asked %v", seed, asked)
}
if _, err := s.doAdopt(map[string]any{"why": "again"}, "op"); err == nil || !strings.Contains(err.Error(), "is open") {
t.Errorf("a second adopt with a seed open: %v", err)
}
// withdraw and again mark the proposal; the holder ends or judges it.
if _, err := s.doWithdraw(map[string]any{"id": seed.ID, "why": "not yet"}, "op"); err != nil {
t.Fatal(err)
}
if _, err := s.doAgain(map[string]any{"id": seed.ID}); err != nil {
t.Fatal(err)
}
if p, err := s.proposal(seed.ID); err != nil || !strings.Contains(p.Withdrawn, "not yet") || p.Again.IsZero() {
t.Errorf("marked = %+v, %v", p, err)
}
if _, err := s.doWithdraw(map[string]any{"id": "p-none", "why": "x"}, "op"); err == nil {
t.Error("an unknown proposal was withdrawn")
}
delete(st[delivery.PlacementProposalsState].values, seed.ID) // the holder ended it
st[delivery.PlacementCandidatesState].values = map[string][]byte{}
// propose: refused whole, naming every finding; a valid move laid over the keys at their revisions.
raw, _ := delivery.EncodePlacement(anchor)
st[delivery.PlacementState].values["placement.anchor"] = raw
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"place","node":"anchor","module":"ghost"},{"op":"hold","node":"anchor","module":"web","claim":"x"}]`}, "op"); err == nil ||
!strings.Contains(err.Error(), "ghost") || !strings.Contains(err.Error(), "lease write") || len(st[delivery.PlacementProposalsState].values) != 0 {
t.Errorf("a refused proposal: %v", err)
}
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"swap","node":"anchor"}]`}, "op"); err == nil {
t.Error("an unknown change was taken")
}
got, err = s.doPropose(map[string]any{"why": "move web", "changes": []any{
map[string]any{"op": "take-off", "node": "anchor", "module": "web"}, map[string]any{"op": "place", "node": "laptop", "module": "web"}}}, "operator@laptop")
if err != nil {
t.Fatal(err)
}
move := got.(proposed).Proposal
if move.By != "operator@laptop" || move.Base["anchor"] != uint64(len(raw)) || move.Base["laptop"] != 0 || len(st[delivery.PlacementCandidatesState].values) != 2 {
t.Fatalf("propose = %+v", move)
}
// placement and proposals read every state the seat keeps (§10); history names the stream for what is older.
reads := map[string]bool{}
pa, err := s.doPlacement(map[string]any{"module": "web", "history": "true"})
if a := pa.(placementAnswer); err != nil || !slices.Equal(a.Nodes, []string{"anchor"}) || len(a.Changes) != 1 || !strings.Contains(a.Older, delivery.PlacementStream) {
t.Errorf("placement of web = %+v, %v", pa, err)
} else {
reads[delivery.PlacementState], reads["stream "+delivery.PlacementStream] = true, true
}
if _, err := s.doPlacement(map[string]any{"node": "laptop"}); err == nil {
t.Error("a node with no key was answered")
}
ps, err := s.doProposals(map[string]any{"id": move.ID})
if list := ps.(map[string]any)["proposals"].([]openProposal); err != nil || len(list) != 1 || len(list[0].Candidates) != 2 {
t.Errorf("proposals = %+v, %v", ps, err)
} else {
reads[delivery.PlacementProposalsState], reads[delivery.PlacementCandidatesState] = true, true
}
raw0, err := readManifest()
if err != nil {
t.Fatal(err)
}
for _, state := range raw0.State {
if slices.Contains([]string{delivery.PlacementState, delivery.PlacementProposalsState, delivery.PlacementCandidatesState}, state.Name) && !reads[state.Name] {
t.Errorf("no verb reads the seat's state %s", state.Name)
}
}
for _, stream := range raw0.Streams {
if !reads["stream "+stream.Name] {
t.Errorf("no verb reads the seat's stream %s", stream.Name)
}
}
// hand-back: the operator's alone, written as a hand-back proposal naming every key at its revision, for the assigner
// to ask; nothing is deleted by the verb.
got, err = s.doHandBack(map[string]any{"why": "back"}, "op")
if hb := got.(proposed).Proposal; err != nil || !hb.HandBack || hb.Base["anchor"] != uint64(len(raw)) || len(st[delivery.PlacementState].values) != 1 {
t.Errorf("hand-back = %+v, %v", got, err)
}
// Issue 774 (ADR 0567 §8): once the controller's lease is handed away a change its verbs cannot take is proposed, and
// there is no controller to hand placement back to.
st[buscompose.ControllerLease] = &memKV{values: map[string][]byte{buscompose.ControllerLeaseKey: []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)}}
if _, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op"); err != nil {
t.Errorf("a hold after the lease write: %v", err)
}
if _, err := s.doHandBack(map[string]any{"why": "back"}, "op"); err == nil || !strings.Contains(err.Error(), "since the lease write") {
t.Errorf("a hand-back after the lease write: %v", err)
}
if tools := s.tools(); len(tools) != 7 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
}
// readManifest is the module's manifest, read as the controller reads it.
func readManifest() (catalogue.Manifest, error) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
return catalogue.Manifest{}, err
}
return catalogue.ParseManifest(raw)
}
@@ -18,8 +18,8 @@ import (
// The assigner's tools (novox/hq ADR 0546), served through the tool runner from its bundle: a process never serves (ADR
// 0436, mesh-sdk process). They write and read the puts; the assigner's process takes each put further (puts.go) and says
// every step as the event put. They are the module's own tools, not a seat's verbs: a new seat needs a change of the
// controller, which takes none; they become the verbs of a seat once composing leaves the controller.
// every step as the event put. They are the module's own tools; the seat mesh-placement's verbs are served beside them
// (seat_tools.go, ADR 0567 §10).
// putsState is the assigner's state of puts, as the runtime names a module's own state.
const putsState = "puts"
@@ -351,7 +351,7 @@ func serveTools() error {
time.Sleep(wait)
}
}()
return stdio.Serve("", s.tools())
return stdio.Serve("", append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...))
}
// shortFP is a fingerprint's first twelve hex digits.
@@ -13,7 +13,10 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
@@ -46,6 +49,14 @@ type world struct {
healthy map[string][]string
// changes counts, by bucket and key, the values this process has seen the key take: a value written again is another.
changes map[string]uint64
// meta is, by bucket and key, the revision and the time of the value held now, where the watch said them (putEntry).
meta map[string]entryMeta
}
// entryMeta is a value's revision in its bucket and when it was written.
type entryMeta struct {
revision uint64
at time.Time
}
// parsed is a manifest read by digest: its module and its manifest, or why it cannot be read.
@@ -56,7 +67,27 @@ type parsed struct {
}
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{}}
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{},
meta: map[string]entryMeta{}}
}
// putEntry is put, keeping the value's revision and the time it was written.
func (w *world) putEntry(bucket, key string, value []byte, revision uint64, at time.Time) {
w.put(bucket, key, value)
w.mu.Lock()
defer w.mu.Unlock()
if value == nil {
delete(w.meta, bucket+"|"+key)
return
}
w.meta[bucket+"|"+key] = entryMeta{revision: revision, at: at}
}
// metaOf is the revision and time of the value a bucket holds at key, as the watch said them.
func (w *world) metaOf(bucket, key string) entryMeta {
w.mu.Lock()
defer w.mu.Unlock()
return w.meta[bucket+"|"+key]
}
// put takes a change of a watched bucket in: a value, or nil for a key deleted or purged.
@@ -162,8 +193,73 @@ func (w *world) manifest(ctx context.Context, get getter, digest string) parsed
return p
}
// assigned is each node's modules with an open assignment, read from the controller's assignments, in name order.
// assigned is each node's modules both the controller's assignments and the seat mesh-placement's placement name, in name order
// (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module.
// Once the controller's lease is handed away it is the seat's placement alone, with no assignment (ADR 0549 §5, §6).
func (w *world) assigned() (map[string][]string, map[string]assignment) {
// A lease handed away before any placement was read keeps the assignments as they last stood, never an empty mesh:
// the lease write's own check refuses that order (ADR 0547 §11).
placed, seeded := w.placed()
byAssignment, all := w.assignedByController()
if away := w.handedAway(); away && seeded {
both, _ := assign.Placed(byAssignment, placed, seeded, away)
return both, map[string]assignment{}
}
both, _ := assign.Placed(byAssignment, placed, seeded, false)
return both, all
}
// placementDiffers is every pair the controller's assignments and the seat's placement do not both name: none once
// the controller's lease is handed away, since the placement is then the one source.
func (w *world) placementDiffers() []assign.Pair {
placed, seeded := w.placed()
byAssignment, _ := w.assignedByController()
_, differ := assign.Placed(byAssignment, placed, seeded, w.handedAway())
return differ
}
// handedAway says the controller's lease is handed to a holder that is no controller (mesh-sdk buscompose.LeaseHandedAway,
// ADR 0561 §3): from then the seat mesh-placement's placement alone names where modules run (ADR 0549 §5). A lease not read (its
// bucket not granted or not there), a controller's, expired or unreadable, is not handed away.
func (w *world) handedAway() bool {
raw, found := w.read(buscompose.LeaseBucket)[buscompose.ControllerLeaseKey]
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, nil)
return away
}
// placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key
// that does not read places nothing on its node and is said by the indexer's own read tool.
func (w *world) placed() (map[string][]string, bool) {
out := map[string][]string{}
values := w.read(delivery.PlacementBucket)
for key, value := range values {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
p, err := delivery.DecodePlacement(value)
if err != nil || p.Node != node {
continue
}
out[node] = p.Assigned()
}
return out, len(values) > 0
}
// placedNodes is every node the seat's placement names (ADR 0567 §1): those compose their own modules (ADR 0547 §2),
// so their wants name the build alone and the seventh rule does not hold them.
func (w *world) placedNodes() map[string]bool {
placed, _ := w.placed()
out := make(map[string]bool, len(placed))
for node := range placed {
out[node] = true
}
return out
}
// assignedByController is each node's modules with an open assignment, read from the controller's assignments, in
// name order.
func (w *world) assignedByController() (map[string][]string, map[string]assignment) {
out, all := map[string][]string{}, map[string]assignment{}
for _, value := range w.read(provider.AssignmentsBucket) {
var a assignment
@@ -210,9 +306,128 @@ func (w *world) mesh(ctx context.Context, manifests getter) assign.Mesh {
}
m.Manifests[name] = p.manifest
}
// Once the catalogue places the nodes, the check composes over the input they compose from (ADR 0547 §2, ADR 0549
// §3): one composer's input for the check and the nodes. An input that cannot be read is said, and the nodes and their
// assignments alone are composed, as before any placement.
if _, seeded := w.placed(); seeded {
in, err := frombus.Input(ctx, worldReader{w: w, manifests: manifests}, frombus.Mesh{}, frombus.Scope{Every: true})
if err != nil {
m.InputUnread = err.Error()
} else {
m.Input = &in
}
}
return m
}
// worldReader is what the assigner watched as frombus reads the bus: the seat's placement, the indexer's modules, each node's
// record in its own have key, the manifests by digest; the check reads no config value, own secret or credential (the
// composition it asks resolves, and stands in for what it does not carry).
type worldReader struct {
w *world
manifests getter
}
func (r worldReader) Placements(context.Context) (map[string]delivery.Placement, error) {
out := map[string]delivery.Placement{}
for key, value := range r.w.read(delivery.PlacementBucket) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
if p, err := delivery.DecodePlacement(value); err == nil && p.Node == node {
out[node] = p
}
}
return out, nil
}
func (r worldReader) Records(context.Context) (map[string]delivery.NodeRecord, error) {
out := map[string]delivery.NodeRecord{}
for key, value := range r.w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
}
if h, err := delivery.DecodeHaveNode(value); err == nil && h.Record != nil {
out[node] = *h.Record
}
}
return out, nil
}
func (r worldReader) Modules(context.Context) (map[string]delivery.Module, error) {
return r.w.trunk(), nil
}
func (r worldReader) Manifest(ctx context.Context, digest string) ([]byte, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return nil, err
}
raw, err := r.manifests(ctx, key)
if err != nil {
return nil, err
}
m, err := delivery.DecodeManifest(raw)
if err != nil {
return nil, err
}
return []byte(m.JSON), nil
}
// Values is one module's config values on one node as mesh-config resolved them, each a value its source lays
// (catalogue.LaidValue): the holder composes every node a proposal touches with them (ADR 0567 §9).
func (r worldReader) Values(_ context.Context, node, module string) (map[string]any, error) {
var out map[string]any
prefix := node + "." + module + "."
for key, raw := range r.w.read(buscompose.ResolvedBucket) {
name, ok := strings.CutPrefix(key, prefix)
if !ok || strings.Contains(name, ".") {
continue
}
if v, laid := catalogue.LaidValue(raw); laid {
if out == nil {
out = map[string]any{}
}
out[name] = v
}
}
return out, nil
}
func (worldReader) OwnSecrets(context.Context, string) ([]compose.OwnSecretRow, error) {
return nil, nil
}
func (worldReader) Credentials(context.Context, string, []compose.CredentialKey, []compose.CredentialKey) ([]compose.Credential, []compose.Issued, error) {
return nil, nil, nil
}
// Build and Archive are read only for a node composing itself (frombus.Scope without Every): the check of every node
// composes from the manifests the indexer read, and reads no build and no archive.
func (worldReader) Build(context.Context, string, string) (*delivery.BuilderBuild, error) {
return nil, nil
}
func (worldReader) Archive(_ context.Context, url, _ string) ([]byte, error) {
return nil, fmt.Errorf("the composition check reads no archive (%s)", url)
}
func (r worldReader) Served(ctx context.Context, node, module string, keys []string) (map[string]any, error) {
all, err := r.Values(ctx, node, module)
var out map[string]any
for _, k := range keys {
if v, ok := all[k]; ok {
if out == nil {
out = map[string]any{}
}
out[k] = v
}
}
return out, err
}
// change is a candidate's change for the composition check, or why the composition waits for it.
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
@@ -265,6 +480,21 @@ func (w *world) change(ctx context.Context, manifests getter, c delivery.Candida
return ch, ""
}
// online is, by node, whether the watchman holds the node online.
func (w *world) online() map[string]bool {
out := map[string]bool{}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
continue
}
if p, err := delivery.DecodePresence(value); err == nil {
out[node] = p.Online
}
}
return out
}
// wantsOf is the wants the assigner wrote, by module then node.
func (w *world) wantsOf() map[string]map[string]delivery.Want {
out := map[string]map[string]delivery.Want{}
@@ -333,6 +563,16 @@ func (w *world) value(bucket, key string) []byte {
return w.values[bucket][key]
}
// grantsBucket is the bucket the grants are read from now (delivery.GrantSource): the grants part's while the controller's
// lease is handed away, as the lease bucket's key says it (buscompose.LeaseHandedAway), the controller's in every other
// case, a lease not held here included.
func (w *world) grantsBucket() string {
raw := w.value(buscompose.LeaseBucket, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, raw != nil, nil)
bucket, _ := delivery.GrantSource(away)
return bucket
}
// has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool {
w.mu.Lock()
@@ -6,7 +6,9 @@ import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
@@ -48,6 +50,7 @@ func pull(repo string, number int, modules map[string]delivery.CandidateModule)
// module with no manifest in the preview as removed, a mesh-sdk member's head, and a module the trunk holds from
// another repository as defined twice.
func TestTheChangeOfACandidate(t *testing.T) {
theWorldCountsTheValuesAKeyTakes(t)
f := fakeManifests{}
web := f.add(t, "web", `{"module":"web","version":"2"}`)
mail := f.add(t, "mail", `{"module":"mail","version":"1"}`)
@@ -113,7 +116,7 @@ func TestAManifestFallingBackToAPredefinedKeyIsReadable(t *testing.T) {
}
// Review round 1 (items 3 and 4): the world counts the values a key takes, so a have written again after an answer is seen.
func TestTheWorldCountsTheValuesAKeyTakes(t *testing.T) {
func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
w := newWorld()
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
@@ -124,4 +127,40 @@ func TestTheWorldCountsTheValuesAKeyTakes(t *testing.T) {
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
// Issue 774: grants are read from the grants part's bucket only while the controller's lease is handed away: a grant
// only the part holds lets a node's put through then, and with a controller's lease (or none, or an unreadable one) it
// holds it, a grant only the controller holds being the one that counts instead.
fp := delivery.SourcePrefix2 + strings.Repeat("d", 64)
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{Accounts: []string{"ace.web"}})
for _, row := range []struct {
name, lease string
inPart, inController bool
bucket string
granted bool
}{
{"handed away, the part's grant", `{"handed-to":"the operator's hand-over"}`, true, false, delivery.PartGrantsBucket, true},
{"handed away, only the controller's grant", `{"handed-to":"x"}`, false, true, delivery.PartGrantsBucket, false},
{"a controller holds it, only the part's grant", `{"instance":"novox/1","epoch":3}`, true, false, delivery.ControllerGrantsBucket, false},
{"a controller holds it, its own grant", `{"instance":"novox/1"}`, false, true, delivery.ControllerGrantsBucket, true},
{"no lease read", ``, true, false, delivery.ControllerGrantsBucket, false},
{"an unreadable lease", `[1]`, true, false, delivery.ControllerGrantsBucket, false},
} {
p := newPasser(buckets{}, time.Now())
if row.lease != "" {
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(row.lease))
}
if row.inPart {
p.w.put(delivery.PartGrantsBucket, gkey, grant)
}
if row.inController {
p.w.put(delivery.ControllerGrantsBucket, gkey, grant)
}
if got := p.w.grantsBucket(); got != row.bucket {
t.Errorf("%s: grants are read from %s, want %s", row.name, got, row.bucket)
}
if _, granted := p.composedFor("ace", "web", fp); granted != row.granted {
t.Errorf("%s: granted = %v, want %v", row.name, granted, row.granted)
}
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028225633-76f7c4d2238f
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -10
View File
@@ -1,13 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c h1:pY2YpHSSsfDPBrOEiWTNSIPn0zdnAayRmtmmyQ3hCHY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8 h1:jA7V7hlfJAigVUvJDpKwfgd9ylQ9bqppg3rK9XgLWoA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8 h1:vhYk/iv43u2WGSTrFyQGDvGBxLaRITgMAiiI9DjuA10=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5 h1:gE80bR0tPME7MTMBLcdP62v+F/9soVXmdcBYNMOye44=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028225633-76f7c4d2238f h1:jCd8WN+Olo4nlaB1cz1HYpEV4xc6Mzw7viXdwU+0fic=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028225633-76f7c4d2238f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -81,6 +81,29 @@ func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.C
return c
}
// NodeFailedCondition is the condition the assigner raises for a module whose newest fingerprint fp failed its health check on
// node while other nodes run it healthy (ADR 0557 §1, issue 1496): the failure is the node's, so nothing is held or set back and
// the node keeps its want. It names the node, the module, fp and the have's words, keyed by node and module, and is cleared
// (raised false) once the node no longer fails alone.
func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) asks.Condition {
key := delivery.AssignerModule + ".failed-alone." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s no longer fails its health check alone.", module, node)}
}
headline := fmt.Sprintf("%s fails on %s alone", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s The health check of %s at %s failed on %s (%s), while it runs healthy on %s. The failure is %s's, "+
"so the assigner holds nothing and sets no node back, and %s keeps its want. This clears once %s runs it healthy or "+
"another version is the newest.", asks.NothingToDo, module, short(fp), node, f.Words, strings.Join(f.Others, ", "), node,
node, node)}
}
// NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
@@ -96,6 +119,14 @@ func NeitherCondition(node, module string, h Holding, raised bool) asks.Conditio
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
if h.Bus {
needs := fmt.Sprintf("put %s on %s with assigner_put", module, node)
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner never moves the bus: %s. This clears once the node has a want for it.",
asks.NeedsYou, needs, module, node, h.Words)}
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
@@ -103,6 +134,17 @@ func NeitherCondition(node, module string, h Holding, raised bool) asks.Conditio
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
}
// NeitherLeftCondition clears the neither condition of module on node when it is no longer on neither list: why is "" when the
// node has a live want for it again, else why it has none (it is held, its move waits for an ask, the node no longer runs it), so
// the clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while it had none).
func NeitherLeftCondition(node, module, why string) asks.Condition {
c := NeitherCondition(node, module, Holding{}, false)
if why != "" {
c.Resolved = fmt.Sprintf("%s on %s is no longer on neither list and has no want: %s.", module, node, why)
}
return c
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
c := HeldCondition(module, fp, "", "", false, false)
@@ -35,6 +35,13 @@ type Mesh struct {
// Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a
// composer at another mesh-sdk reads (issue 1622).
Digests map[string]string
// Input is the rest of the mesh as the nodes compose from it, filled by mesh-sdk's frombus from the catalogue's
// placement and the nodes' records (novox/hq ADR 0547 §2, ADR 0549 §3, issue 1126), so the check and the nodes share
// one composer's input; nil before any placement is read, when the nodes and their assignments alone are composed.
Input *compose.Input
// InputUnread says why that input could not be read from the bus, when it could not: the check then composes the
// nodes and their assignments alone, and says so in its warnings.
InputUnread string
}
// Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those
@@ -57,7 +64,8 @@ type Change struct {
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
// removes, those running a consumer of a provision such a module provides (before or after), and those running a
// provider of a provision such a module requires after the change. A module the change adds that is assigned nowhere
// touches no machine.
// touches no machine. A change of placement is no change of a repository (ADR 0567 §4): the seat
// mesh-placement judges it.
func Touched(m Mesh, c Change) []string {
moved := map[string]bool{}
for name := range c.Manifests {
@@ -272,6 +280,18 @@ func GrantsAlone(m Mesh, c Change, composed []string) []string {
return out
}
// withOverlay is base with every manifest of over laid over it, a copy.
func withOverlay(base, over map[string]catalogue.Manifest) map[string]catalogue.Manifest {
out := make(map[string]catalogue.Manifest, len(base)+len(over))
for name, man := range base {
out[name] = man
}
for name, man := range over {
out[name] = man
}
return out
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
@@ -42,9 +42,17 @@ func HeadJob(m Mesh, c Change) (delivery.ComposeJob, error) {
for _, name := range c.Removed {
delete(change, name)
}
req := composer.Request{V: composer.Version, Input: plainInput(m.Assigned), Depth: composer.DepthResolve,
HoldingsFromClaims: true, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, {Name: CaseChange, Manifests: change}}}
in, fromClaims := plainInput(m.Assigned), true
if m.Input != nil {
// The input the nodes compose from, without its catalogue: the cases carry the manifests (issue 1126).
in = *m.Input
in.Catalogue = nil
fromClaims = len(in.Holdings) == 0
}
changeCase := composer.Case{Name: CaseChange, Manifests: change}
req := composer.Request{V: composer.Version, Input: in, Depth: composer.DepthResolve,
HoldingsFromClaims: fromClaims, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, changeCase}}
return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group)
}
@@ -195,6 +203,10 @@ func FromComposer(m Mesh, c Change, head Ran, readers []Reader, took time.Durati
readerFails, readerWarnings := judgeReaders(c, change, readers)
fails = append(fails, readerFails...)
v.Warnings = append(v.Warnings, readerWarnings...)
if m.InputUnread != "" {
v.Warnings = append(v.Warnings, "composed over the nodes and their assignments alone: the input the nodes compose from "+
"cannot be read from the bus ("+m.InputUnread+")")
}
return finish(v, m, c, machines, fails)
}
@@ -8,6 +8,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -100,6 +101,25 @@ func TestAGroupWithAnSDKMemberAsksTheComposerAtItsHead(t *testing.T) {
if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims {
t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims)
}
// 1126 (ADR 0547 §2, ADR 0549 §3): once the catalogue places the nodes, the check composes over the input the nodes
// compose from (frombus), its catalogue left to the cases, and the seats held as placement says.
m := digestedMesh()
m.Input = &compose.Input{Catalogue: map[string]catalogue.Manifest{"web": {Module: "web"}},
Nodes: []compose.Node{{Name: "g14", Assigned: []string{"web"}, Place: compose.Place{Endpoint: "g14.example:51820"}}},
Holdings: []catalogue.Held{{Claim: "web-seat", Scope: "mesh", Node: "g14", Module: "web"}}}
placedJob, err := HeadJob(m, c)
if err != nil {
t.Fatal(err)
}
placed, err := composer.Decode(placedJob.Request)
if err != nil {
t.Fatal(err)
}
if placed.Input.Catalogue != nil || len(placed.Input.Nodes) != 1 || placed.Input.Nodes[0].Place.Endpoint != "g14.example:51820" ||
len(placed.Input.Holdings) != 1 || placed.HoldingsFromClaims {
t.Errorf("the placed check's request = %+v, holdings from claims %v", placed.Input, placed.HoldingsFromClaims)
}
}
// 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the
@@ -0,0 +1,132 @@
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Until ADR 0547's third milestone two sources name which module runs on which node: the controller's assignments and
// the seat mesh-placement's placement (ADR 0549 §5, ADR 0567 §8). The assigner writes wants only for a pair both name:
// for a pair one source names alone it writes no want and touches none it wrote before, so nothing moves and nothing is
// removed on one source's word, and it raises a condition naming the pair until both agree. After an approved proposal
// the assigner keeps the controller in step itself, calling its assign, unassign, pin and unpin; a call refused or not
// yet answered leaves the pair differing, and the condition names the call (Pair.Call). Before the first placement key
// is written the controller's assignments are the one source, as before. From the lease write (the controller's lease
// handed away) the placement is the one source.
// Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or
// OnlyPlacement.
type Pair struct {
Node, Module, Only string
// Call is what became of the assigner's call of the controller's verb for the pair, in words, when it made one.
Call string
}
// The source that names a pair alone.
const (
OnlyAssignments = "the controller's assignments"
OnlyPlacement = "the seat mesh-placement's placement"
)
// Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs. Once the controller's
// lease is handed away (handedAway, ADR 0549 §5, §6) it is the placement alone, in name order, and no pair differs: the
// assignments are then a source of nothing.
func Placed(assigned, placed map[string][]string, seeded, handedAway bool) (map[string][]string, []Pair) {
if handedAway && seeded {
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
return out, nil
}
if !seeded {
return assigned, nil
}
set := func(m map[string][]string) map[string]bool {
out := map[string]bool{}
for node, modules := range m {
for _, module := range modules {
out[node+"\x00"+module] = true
}
}
return out
}
inAssigned, inPlaced := set(assigned), set(placed)
both := map[string][]string{}
var differ []Pair
for node, modules := range assigned {
for _, module := range modules {
if inPlaced[node+"\x00"+module] {
both[node] = append(both[node], module)
} else {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyAssignments})
}
}
}
for node, modules := range placed {
for _, module := range modules {
if !inAssigned[node+"\x00"+module] {
differ = append(differ, Pair{Node: node, Module: module, Only: OnlyPlacement})
}
}
}
for _, modules := range both {
sort.Strings(modules)
}
sort.Slice(differ, func(i, j int) bool {
if differ[i].Node != differ[j].Node {
return differ[i].Node < differ[j].Node
}
return differ[i].Module < differ[j].Module
})
return both, differ
}
// PlacementCondition is the condition of a pair the two sources do not both name: raised while they differ, cleared once
// both name it or neither does.
func PlacementCondition(p Pair, raised bool) asks.Condition {
key := delivery.AssignerModule + ".placement." + p.Node + "." + p.Module
subject := asks.Scope{Scope: "module", ID: p.Module, Machine: p.Node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("the controller's assignments and the seat's placement agree on %s on %s.", p.Module, p.Node)}
}
headline := fmt.Sprintf("%s on %s is placed by one source only", p.Module, p.Node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
var needs string
if p.Only == OnlyAssignments {
needs = fmt.Sprintf("propose placing %s on %s (mesh-placement.propose), or unassign it at the controller", p.Module, p.Node)
} else {
needs = fmt.Sprintf("assign %s to %s at the controller, as the approved placement says, or propose taking it off", p.Module, p.Node)
}
if p.Call != "" {
needs += "; the assigner's call: " + p.Call
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: fmt.Sprintf("%s%s. %s names %s on %s and the other source does not. Until ADR 0547's third milestone both must "+
"name a module on a node (ADR 0549 §5): the assigner writes no want for it there and leaves the one it wrote before, so "+
"nothing moves or is removed. This clears once both name it or neither does.", asks.NeedsYou, needs, p.Only, p.Module, p.Node)}
}
// SeatWaitsCondition is the assigner's word on the seat mesh-placement while a state of the seat is not bound (ADR 0567
// §1): raised naming each, cleared when none is left.
func SeatWaitsCondition(unbound []string) asks.Condition {
key := delivery.AssignerModule + ".seat-states"
subject := asks.Scope{Scope: "module", ID: delivery.AssignerModule}
if len(unbound) == 0 {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared, Resolved: "the seat mesh-placement holds all its states."}
}
why := "the seat waits for its states: " + strings.Join(unbound, ", ") + " are not on the bus or not granted yet"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: "the seat waits for its states", Headline: "the seat waits for its states",
Explanation: asks.NothingToDo + " " + why + "; each is bound at the next pass once it is, with no restart."}
}
@@ -0,0 +1,296 @@
package assign
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The seat mesh-placement (novox/hq ADR 0567), held by the assigner beside its wants: placement is its state
// `placement`, changed only by a proposal (§2) the nodes it touches answer (§3), judged (§4) and written on the
// operator's Approve at the revision it was laid over (§5). What here decides; the tools and the passes do.
// ProposeIn is what a proposal is laid over: its id, who and why, its changes; every node's current placement and the
// revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's
// digest.
type ProposeIn struct {
ID, By, Why string
At time.Time
Changes []delivery.PlacementProposalChange
Current map[string]delivery.Placement
Revisions map[string]uint64
Held func(module string) bool
Recorded func(node string) bool
Seed bool
ExportDigest string
// Leased says the lease write of ADR 0547 has happened (the controller's lease handed away, mesh-sdk
// buscompose.LeaseHandedAway; issue 774), after which the controller's assignments are read no more (§8): until then
// the holder keeps the controller in step and refuses what its verbs cannot take.
Leased bool
}
// Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision),
// the laid placement of every node whose key it changes, and every finding, which refuses it whole: a change the
// controller's verbs cannot take before the lease write (§8), a change that does not fit its node, and what
// CheckPlacements refuses of the result laid over every placement (two modules on one host port, a provider not placed,
// a seat held once held twice, two nodes with one address, two hubs, a module the indexer does not hold, a node with no
// record). A key whose value the proposal leaves as it is changes nothing and is not laid.
func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Placement, []string) {
p := delivery.PlacementProposal{ID: in.ID, By: strings.TrimSpace(in.By), Why: strings.TrimSpace(in.Why), At: in.At.UTC(),
Changes: in.Changes, Seed: in.Seed, ExportDigest: in.ExportDigest, Base: map[string]uint64{}, State: delivery.ProposalAnswering}
var findings []string
if p.By == "" {
findings = append(findings, "the runtime names no caller: a proposal is made by someone")
}
if p.Why == "" {
findings = append(findings, "a proposal says why: it is shown to the operator")
}
if len(in.Changes) == 0 {
findings = append(findings, "a proposal changes something: it names no change")
}
for i, c := range in.Changes {
if (c.Op == delivery.OpSeed) != in.Seed {
findings = append(findings, fmt.Sprintf("change %d: the seed is adopt's alone, and never mixed with other changes", i+1))
continue
}
if !in.Leased {
if why := c.BeforeLease(); why != "" {
findings = append(findings, fmt.Sprintf("change %d: %s", i+1, why))
}
}
}
laid, lay := delivery.LayChanges(in.Current, in.Changes)
findings = append(findings, lay...)
all := map[string]delivery.Placement{}
for node, pl := range in.Current {
all[node] = pl
}
out := map[string]*delivery.Placement{}
for node, pl := range laid {
if before, ok := in.Current[node]; ok && delivery.SamePlacement(before, pl) {
continue
}
pl := pl
all[node] = pl
out[node] = &pl
p.Base[node] = in.Revisions[node]
}
if len(findings) == 0 && len(out) == 0 {
findings = append(findings, "the changes leave every placement as it is")
}
held, recorded := in.Held, in.Recorded
if held == nil {
held = func(string) bool { return false }
}
if recorded == nil {
recorded = func(string) bool { return false }
}
for _, r := range delivery.CheckPlacements(all, held, recorded) {
findings = append(findings, r)
}
return p, out, findings
}
// Write is one key a warrant writes (§5): the node, its key, the revision it is written at (0: it has none, so it is
// created) and its value (nil: it is deleted).
type Write struct {
Node, Key string
Base uint64
Value *delivery.Placement
}
// Writes is every key a proposal writes, by node order.
func Writes(p delivery.PlacementProposal, laid map[string]*delivery.Placement) ([]Write, error) {
var out []Write
for _, node := range p.Nodes() {
key, err := delivery.PlacementKey(node)
if err != nil {
return nil, err
}
out = append(out, Write{Node: node, Key: key, Base: p.Base[node], Value: laid[node]})
}
return out, nil
}
// ControllerCall is one call of the controller's seat verbs the holder makes after writing an approved proposal, until
// the lease write (§8): assign, unassign, pin or unpin, with its arguments.
type ControllerCall struct {
Verb string
Args map[string]any
// Node and Module name the pair the call keeps in step (a pin names no module).
Node, Module string
}
// Key names the call, as the condition of its pair says it.
func (c ControllerCall) Key() string {
return fmt.Sprintf("mesh-controller.%s %s %s", c.Verb, c.Node, c.Module)
}
// ControllerCalls is every call that brings the controller's assignments and pins to the placement after: assign for a
// module placed on a node before not, unassign for one taken off, pin for a pin set or changed, unpin for one removed,
// each naming the proposal and its warrant in its why. In node order, unassigns before assigns, so a module moved
// between nodes is first taken off.
func ControllerCalls(before map[string]delivery.Placement, after map[string]*delivery.Placement, proposal, warrant string) []ControllerCall {
why := fmt.Sprintf("placement proposal %s, approved (%s)", proposal, warrant)
nodes := make([]string, 0, len(after))
for node := range after {
nodes = append(nodes, node)
}
sort.Strings(nodes)
var off, on, pins []ControllerCall
for _, node := range nodes {
was, now := before[node], delivery.Placement{}
if a := after[node]; a != nil {
now = *a
}
had, has := map[string]bool{}, map[string]bool{}
for _, m := range was.Modules {
had[m.Module] = true
}
for _, m := range now.Modules {
has[m.Module] = true
if !had[m.Module] {
on = append(on, ControllerCall{Verb: "assign", Node: node, Module: m.Module, Args: map[string]any{"node": node, "module": m.Module}})
}
}
for _, m := range was.Modules {
if !has[m.Module] {
off = append(off, ControllerCall{Verb: "unassign", Node: node, Module: m.Module, Args: map[string]any{"node": node, "module": m.Module}})
}
}
for _, provision := range sortedKeys(now.Pins) {
to := now.Pins[provision]
if old, ok := was.Pins[provision]; !ok || old != to {
pins = append(pins, ControllerCall{Verb: "pin", Node: node, Module: provision,
Args: map[string]any{"node": node, "provision": provision, "from": to.Node, "module": to.Module}})
}
}
for _, provision := range sortedKeys(was.Pins) {
if _, ok := now.Pins[provision]; !ok {
pins = append(pins, ControllerCall{Verb: "unpin", Node: node, Module: provision, Args: map[string]any{"node": node, "provision": provision}})
}
}
}
out := append(append(off, on...), pins...)
for i := range out {
out[i].Args["why"] = why
}
return out
}
// PlacementAct is the act an approved proposal's yes binds: writing exactly its changes at its base revisions.
func PlacementAct(p delivery.PlacementProposal) asks.Act {
return asks.Act{"placement": "write", "proposal": p.ID, "changes": delivery.ChangesDigest(p)}
}
// PlacementAsking is a proposal's asking number: each judging that raises its ask again (again, a key moved) is
// another asking, under an id of its own.
func PlacementAsking(p delivery.PlacementProposal, at time.Time) int {
return int(at.Unix())
}
// PlacementAsk is the authorising ask about a proposal (§5): an Approve that binds its act, so only a channel that
// proves who answered takes it, with the brief of the verdict by node (judge.Brief) as its explanation.
func PlacementAsk(key string, p delivery.PlacementProposal, brief string, asking int, now time.Time) (delivery.AssignerAsk, error) {
binds, err := asks.ActDigest(PlacementAct(p))
if err != nil {
return delivery.AssignerAsk{}, err
}
nodes := p.Nodes()
headline := fmt.Sprintf("Change placement on %s?", strings.Join(nodes, ", "))
if p.Seed {
headline = "Take placement over from the controller?"
}
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = fmt.Sprintf("Change placement on %d machines?", len(nodes))
}
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: headline,
Explanation: brief,
Options: []asks.Option{
{ID: OptionYes, Label: "Approve", Does: "the placement of " + strings.Join(nodes, ", ") + " is written as proposed",
Level: asks.Approve, Binds: binds},
{ID: OptionNo, Label: "Decline", Does: "nothing is written, and the proposal ends", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(asks.ApproveLasts),
OnExpiry: "nothing is written, and the proposal ends",
About: key,
}, Asking: asking}, nil
}
// ExportOf reads the controller's answer to placement-export (§7): its verb answers what the command printed, the
// JSON of every node's placement, in `output` (the controller's verbAnswer; `answer` when it carries it as data). A
// refused or unreadable answer is an error naming it.
func ExportOf(raw []byte) (map[string]delivery.Placement, error) {
var a struct {
Output string `json:"output"`
OK *bool `json:"ok"`
Answer json.RawMessage `json:"answer"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return nil, fmt.Errorf("the controller's placement-export answered no verb answer: %w", err)
}
if a.OK != nil && !*a.OK {
return nil, fmt.Errorf("the controller's placement-export refused: %s", strings.TrimSpace(a.Output))
}
var byNode map[string]json.RawMessage
if len(a.Answer) > 0 && string(a.Answer) != "null" {
if err := json.Unmarshal(a.Answer, &byNode); err != nil {
return nil, fmt.Errorf("the controller's placement-export is no placement by node: %w", err)
}
} else if err := json.NewDecoder(strings.NewReader(a.Output)).Decode(&byNode); err != nil {
// The command prints its JSON first; what its standard error said after it is left.
return nil, fmt.Errorf("the controller's placement-export printed no placement by node: %w", err)
}
out := map[string]delivery.Placement{}
for node, rawP := range byNode {
p, err := delivery.DecodePlacement(rawP)
if err != nil {
return nil, fmt.Errorf("the controller's placement-export of %s cannot be read: %w", node, err)
}
if p.Node != node {
return nil, fmt.Errorf("the controller's placement-export names %s under %s", p.Node, node)
}
out[node] = p
}
if len(out) == 0 {
return nil, fmt.Errorf("the controller's placement-export names no node")
}
return out, nil
}
// SeedChanges is the seed's changes: every node's placement of the export, whole, in node order.
func SeedChanges(export map[string]delivery.Placement) []delivery.PlacementProposalChange {
var out []delivery.PlacementProposalChange
for _, node := range sortedKeys(export) {
p := export[node]
out = append(out, delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: node, Whole: &p})
}
return out
}
// Rest is what a proposal leaves to write once the keys of written are: its changes of the other nodes, to be laid again
// over the keys as the write left them (§5: a key moved while it was written). False when nothing is left.
func Rest(p delivery.PlacementProposal, written map[string]*delivery.Placement) (delivery.PlacementProposal, bool) {
rest := p
rest.Changes, rest.Base = nil, map[string]uint64{}
for _, c := range p.Changes {
if _, done := written[c.Node]; !done {
rest.Changes = append(rest.Changes, c)
}
}
for node, base := range p.Base {
if _, done := written[node]; !done {
rest.Base[node] = base
}
}
rest.Must, rest.Verdict, rest.Ask, rest.State = nil, nil, "", delivery.ProposalAnswering
return rest, len(rest.Changes) > 0
}
@@ -0,0 +1,157 @@
package assign
import (
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0567 §2, §5, §7, §8: what the seat's holder decides of a proposal: every refusal named at once, a valid one laid
// with the base revision of each key it changes, the writes at those revisions, the controller's calls that keep its
// assignments in step, the authorising ask bound to the changes, and the export read as the controller answers it.
func TestThePlacementSeatDecides(t *testing.T) {
anchor := delivery.Placement{V: delivery.Version, Node: "anchor", Decided: delivery.NodeDecided{Hub: true, Address: "10.0.0.1"},
Modules: []delivery.PlacedModule{{Module: "store", Ports: map[string]int{"5432": 5432}}, {Module: "web"}}}
laptop := delivery.Placement{V: delivery.Version, Node: "laptop", Decided: delivery.NodeDecided{Address: "10.0.0.2"}, Modules: []delivery.PlacedModule{{Module: "shell"}}}
in := func(changes ...delivery.PlacementProposalChange) ProposeIn {
return ProposeIn{ID: "p-one", By: "operator@laptop", Why: "move web", At: time.Unix(1, 0), Changes: changes,
Current: map[string]delivery.Placement{"anchor": anchor, "laptop": laptop}, Revisions: map[string]uint64{"anchor": 4, "laptop": 9},
Held: func(m string) bool { return m != "ghost" },
Recorded: func(n string) bool { return n == "anchor" || n == "laptop" }}
}
move := []delivery.PlacementProposalChange{{Op: delivery.OpTakeOff, Node: "anchor", Module: "web"}, {Op: delivery.OpPlace, Node: "laptop", Module: "web"}}
p, laid, findings := Propose(in(move...))
if len(findings) != 0 || p.State != delivery.ProposalAnswering || p.Base["anchor"] != 4 || p.Base["laptop"] != 9 ||
!slices.Equal(laid["laptop"].Assigned(), []string{"shell", "web"}) || !slices.Equal(laid["anchor"].Assigned(), []string{"store"}) {
t.Fatalf("a move: %+v, %+v, %v", p, laid, findings)
}
if _, err := delivery.EncodePlacementProposal(p); err != nil {
t.Fatalf("the proposal cannot be written: %v", err)
}
for name, c := range map[string]struct {
in ProposeIn
want string
}{
"a port before the lease": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}}), "lease write"},
"a holding": {in(delivery.PlacementProposalChange{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "mesh-store", Scope: "mesh"}), "lease write"},
"a binding": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "web", Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}}), "lease write"},
"a config place": {in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}}), "lease write"},
"a config access": {in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", MeshWide: map[string]any{"accesses": []any{}}}), "lease write"},
"a module not held": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "ghost"}), "the indexer holds no module ghost"},
"a node with no record": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "stranger", Module: "web"}), "stranger is a node with no record"},
"nothing changes": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "anchor", Module: "web"}), "leave every placement as it is"},
"no why": {func() ProposeIn { i := in(move...); i.Why = " "; return i }(), "says why"},
"no caller": {func() ProposeIn { i := in(move...); i.By = ""; return i }(), "names no caller"},
"a seed by hand": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "laptop", Whole: &laptop}), "adopt's alone"},
"a pin of nobody placed": {in(delivery.PlacementProposalChange{Op: delivery.OpPin, Node: "laptop", Provision: "db", Provider: &delivery.Provider{Node: "laptop", Module: "store"}}), "not placed there"},
} {
if _, _, f := Propose(c.in); !slices.ContainsFunc(f, func(s string) bool { return strings.Contains(s, c.want) }) {
t.Errorf("%s: findings %v, want one naming %q", name, f, c.want)
}
}
// A module's config values may be proposed before the lease write (§13); once the lease is written every change is
// the state's alone, and none waits.
if _, _, f := Propose(in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}})); len(f) != 0 {
t.Errorf("config values before the lease write: %v", f)
}
for _, c := range []delivery.PlacementProposalChange{
{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}},
{Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)},
{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"},
} {
leased := in(c)
leased.Leased = true
if _, _, f := Propose(leased); len(f) != 0 {
t.Errorf("after the lease write %s is refused: %v", c.Op, f)
}
}
// A key moved while the warrant wrote: what is left to write is the other nodes' changes, to be laid and asked again.
if rest, ok := Rest(p, map[string]*delivery.Placement{"anchor": laid["anchor"]}); !ok || len(rest.Changes) != 1 || rest.Changes[0].Node != "laptop" ||
len(rest.Base) != 1 || rest.State != delivery.ProposalAnswering || rest.Must != nil {
t.Errorf("the rest = %+v, %v", rest, ok)
}
if _, ok := Rest(p, laid); ok {
t.Error("a proposal written whole leaves something to write")
}
// The seat says it waits while one of its states is not bound, naming it, and clears once all are.
for _, unbound := range [][]string{{delivery.PlacementPulsesBucket}, nil} {
c := SeatWaitsCondition(unbound)
if _, err := asks.EncodeCondition(c); err != nil || (len(unbound) > 0) != (c.State == asks.StateRaised) ||
(len(unbound) > 0 && !strings.Contains(c.Explanation, "the seat waits for its states: "+delivery.PlacementPulsesBucket)) {
t.Errorf("waiting for %v: %+v, %v", unbound, c, err)
}
}
// Issue 1126: the seed places the private network, which no repository holds; it is the mesh's own, and passes.
wired := laptop
wired.Modules = append(slices.Clone(laptop.Modules), delivery.PlacedModule{Module: "mesh-wireguard"})
seed := in(SeedChanges(map[string]delivery.Placement{"laptop": wired})...)
seed.Seed, seed.ExportDigest, seed.Current, seed.Revisions = true, "sha256:x", map[string]delivery.Placement{}, map[string]uint64{}
seed.Held = func(m string) bool { return m != "mesh-wireguard" } // the indexer holds every module but it
if _, _, f := Propose(seed); len(f) != 0 {
t.Errorf("a seed placing the private network is refused: %v", f)
}
writes, err := Writes(p, laid)
if err != nil || len(writes) != 2 || writes[0].Key != "placement.anchor" || writes[0].Base != 4 || writes[1].Base != 9 {
t.Fatalf("writes = %+v, %v", writes, err)
}
pinned := *laid["laptop"]
pinned.Pins = map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}
calls := ControllerCalls(map[string]delivery.Placement{"anchor": anchor, "laptop": laptop},
map[string]*delivery.Placement{"anchor": laid["anchor"], "laptop": &pinned}, "p-one", "w1@t")
var said []string
for _, c := range calls {
said = append(said, c.Key())
if !strings.Contains(c.Args["why"].(string), "p-one") {
t.Errorf("%s does not name the proposal: %v", c.Key(), c.Args)
}
}
if !slices.Equal(said, []string{"mesh-controller.unassign anchor web", "mesh-controller.assign laptop web", "mesh-controller.pin laptop db"}) {
t.Errorf("the controller's calls = %v", said)
}
a, err := PlacementAsk("placement.p-one", p, "move web (proposed by operator@laptop)", 7, time.Unix(100, 0))
if err != nil || a.Ask.Check(time.Unix(100, 0)) != nil || a.Ask.Options[0].Level != asks.Approve || a.Ask.Options[0].Performs(PlacementAct(p)) != nil {
t.Fatalf("the ask = %+v, %v", a, err)
}
q := p
q.Changes = append(slices.Clone(p.Changes), delivery.PlacementProposalChange{Op: delivery.OpUnpin, Node: "laptop", Provision: "x"})
if a.Ask.Options[0].Performs(PlacementAct(q)) == nil {
t.Error("the ask's yes binds changes it was not asked about")
}
export, _ := json.Marshal(map[string]any{"anchor": anchor, "laptop": laptop})
for name, c := range map[string]struct {
raw string
want int
}{
"in output": {`{"ok":true,"output":` + quote(string(export)+"\n") + `}`, 2},
"as answer": {`{"ok":true,"output":"","answer":` + string(export) + `}`, 2},
"refused": {`{"ok":false,"output":"no store"}`, 0},
"no node": {`{"ok":true,"output":"{}"}`, 0},
"no version": {`{"ok":true,"output":"{\"anchor\":{\"node\":\"anchor\",\"modules\":[]}}"}`, 0},
} {
got, err := ExportOf([]byte(c.raw))
if len(got) != c.want || (c.want == 0) != (err != nil) {
t.Errorf("%s: %d nodes, %v", name, len(got), err)
}
}
if seed := SeedChanges(map[string]delivery.Placement{"laptop": laptop, "anchor": anchor}); len(seed) != 2 || seed[0].Node != "anchor" || seed[0].Op != delivery.OpSeed {
t.Errorf("the seed's changes = %+v", seed)
}
}
func quote(s string) string {
raw, _ := json.Marshal(s)
return string(raw)
}
@@ -2,6 +2,7 @@ package assign
import (
"fmt"
"maps"
"slices"
"sort"
"strings"
@@ -46,8 +47,14 @@ type Module struct {
Now time.Time
// Puts are the puts of the module, by node (ADR 0546): while one owns a node's want (PutOwns), the rules pass over it.
Puts map[string]delivery.Put
// Self says whether node composes the module itself: the catalogue's placement names the node (ADR 0547 §2). It is the one
// source of a want's Self: every want the rules write or keep is derived from it (selfDerived), so a want written Self while
// the node was placed does not outlive the placement (issue 1126). Nil: no node does.
Self func(node string) bool
}
func (m Module) selfOn(node string) bool { return m.Self != nil && m.Self(node) }
func (m Module) composition(node, fp string) (delivery.ComposedDigest, bool) {
if m.Composed == nil {
return delivery.ComposedDigest{}, false
@@ -169,6 +176,9 @@ type Outcome struct {
// Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held
// record and its ask are taken away, and the wants move by the rules again in this pass.
Lifted bool
// NodeFailed is, by node, a failed health check of m.FP on a node while another node runs m.FP healthy: the failure is
// that node's, so m.FP is not held and nothing is set back (ADR 0557 §1, issue 1496). The pass raises its condition once.
NodeFailed map[string]NodeFailure
// Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14).
Record []RecordNeed
// Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set
@@ -192,10 +202,27 @@ type RecordNeed struct {
Words string
}
// Holding is a rule holding a node back, and why.
// Holding is a rule holding a node back, and why. Bus marks the bar on the bus instead of one of the seven rules (Rule is then
// 0): the bus moves only by the operator's put (ADR 0539 §15, ADR 0546 §8), never by the assigner (issue 1945: nats on novox,
// running an old-form build, got no want and the pass said nothing).
type Holding struct {
Rule int
Words string
Bus bool
}
// Reason is the holding in words: the rule and its words, or for the bus bar its words alone.
func (h Holding) Reason() string {
if h.Bus {
return h.Words
}
return fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
}
// BusBarWords say why the assigner writes no want moving the bus on node: it runs prev, the trunk builds fp, and only the
// operator's put moves it (ADR 0539 §15, ADR 0546 §8). The words reach the operator, so they name no record.
func BusBarWords(node, prev, fp string) string {
return fmt.Sprintf("only your put moves the bus: %s runs %s, the trunk builds %s; put it with assigner_put", node, short(prev), short(fp))
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
@@ -211,11 +238,25 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
}
if !held && !out.Lifted && !m.Released && m.FP != "" {
for _, node := range slices.Sorted(slices.Values(m.Nodes)) {
if h, ok := n.Haves[node][m.Name]; ok && h.FP == m.FP && failed(h) {
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
h, ok := n.Haves[node][m.Name]
if !ok || h.FP != m.FP || !failed(h) {
continue
}
if others := healthyElsewhere(m, n, node); len(others) > 0 && !failedElsewhere(m, n, node) {
// F runs healthy on another node, so its failure here is this node's, not F's: F is not held, no node is set back,
// and the failing node keeps its want (ADR 0557 §1; issue 1496: on ace a newly assigned mesh-build-node failed its
// first check and the hold moved shanks, g14 and novox, healthy on F for an hour, back). The pass raises the node's
// condition. With no online healthy witness, or with F failed on a second node, the hold and the set-back stand (ADR 0557
// §2, ADR 0429 §6).
if out.NodeFailed == nil {
out.NodeFailed = map[string]NodeFailure{}
}
out.NodeFailed[node] = NodeFailure{Others: others, Words: HeldWords(h, time.Time{})} // said once: no clock in it
continue
}
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
}
}
mm := m
@@ -281,6 +322,13 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
ok, target = true, have.FP
}
}
if ok && target == m.FP && len(out.NodeFailed) > 0 && !(had && !before.Removed && before.FP == m.FP) &&
n.Haves[node][m.Name].FP != m.FP {
// While a node fails F alone, F reaches no further node: a node that neither wants nor runs F gets no new want
// naming it, and wants already written stay (ADR 0557 §2). A node that runs F with no live want gets its want at F, or
// it would be on neither list and its node-engine would give the module up (issue 1945).
ok, rule, words = false, delivery.RuleNotHeld, FailedAloneWords(m.Name, m.FP, slices.Sorted(maps.Keys(out.NodeFailed)))
}
switch {
case ok:
d, _ := m.composition(node, target)
@@ -303,6 +351,14 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
} else if prev, barred := at.busBar(asked, hadLive, running.FP); barred {
// The bus is barred: no want moves it but the operator's put. Said, never silent (issue 1945): its holding is named,
// and a node that runs it with no live want has it on neither list, so its condition stays raised.
h := Holding{Bus: true, Words: BusBarWords(node, prev, at.FP)}
out.Rules[node] = h
if !hadLive && running.Phase != delivery.PhaseRemoved && !held {
out.Neither[node] = h
}
}
continue
}
@@ -368,6 +424,10 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
}
}
}
next, write := m.selfDerived(node, before, had, next, n, &out)
if !write {
continue
}
if !had || !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
@@ -380,6 +440,43 @@ func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[st
return out
}
// selfDerived is next with its Self derived from the placement now (Module.Self), whichever rule made it, and whether it is
// written. A placed node's want names the build alone (ADR 0547 §2). An unplaced node's want that a rule kept from the want
// written while the node was placed (held at F, held back by rule 3 or 5, a data module asked about) names the controller's
// composition of the same fingerprint, asked as any move of a record module is (ADR 0539 §14): on 2026-10-27, after the
// placement seed's revert, g14's systemd-resolved (held) and forticlient (rule 5), and shanks's forticlient (rule 3), kept
// their Self wants, which their node agents, no longer placed, could not compose (issue 1126). With no such composition, or
// while its ask waits, the want written stays and the node's holding says why.
func (m Module) selfDerived(node string, before delivery.Want, had bool, next delivery.Want, n Nodes, out *Outcome) (delivery.Want, bool) {
switch {
case next.Removed:
return next, true
case m.selfOn(node):
next.Self, next.Composed = true, ""
return next, true
case !next.Self && next.Composed != "":
return next, true
}
next.Self, next.Yes = false, ""
d, there := m.composition(node, next.FP)
if !there || d.Digest == "" {
out.Rules[node] = Holding{Rule: delivery.RuleComposed, Words: fmt.Sprintf("%s on %s was composed by the node while it was placed, "+
"and the controller has not composed it at %s, so its want stays", m.Name, node, short(next.FP))}
return next, false
}
at := m
at.FP = next.FP
running, ran := n.Haves[node][m.Name]
if need, wait := at.mustAsk(node, before, had, ran, running.FP, running.Composed == "", d); wait {
if need != nil && !slices.Contains(out.Record, *need) { // the branch held back at an older fingerprint may have asked it
out.Record = append(out.Record, *need)
}
return next, false
}
next.Composed = d.Digest
return next, true
}
// NoWantWords say why a node gets no want for a module: the rule that holds its first want back and why (issue 1945).
func NoWantWords(rule int, words string) string {
return fmt.Sprintf("no want: rule %d: %s", rule, words)
@@ -394,6 +491,43 @@ func failed(h delivery.Have) bool {
return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed)
}
// healthyElsewhere is every online node of m but node that runs m.FP healthy, by name.
func healthyElsewhere(m Module, n Nodes, node string) []string {
var out []string
for _, other := range slices.Sorted(slices.Values(m.Nodes)) {
// A witness is online: a node that sleeps keeps the have it last wrote, and a stale have is no evidence that F runs
// healthy now.
if other != node && n.Online[other] && healthy(n.Haves[other][m.Name], m.FP) {
out = append(out, other)
}
}
return out
}
// failedElsewhere says whether another node of m has a have naming m.FP failed: then the failure is not one node's alone, and
// the hold and the set-back stand at the second failure (ADR 0557 §2).
func failedElsewhere(m Module, n Nodes, node string) bool {
for _, other := range m.Nodes {
if h, ok := n.Haves[other][m.Name]; ok && other != node && h.FP == m.FP && failed(h) {
return true
}
}
return false
}
// FailedAloneWords say why a node gets no new want naming fp: fp fails its health check on the nodes named, alone.
func FailedAloneWords(module, fp string, failing []string) string {
return fmt.Sprintf("%s at %s fails its health check on %s, so no further node moves to it until that is over", module, short(fp),
strings.Join(failing, ", "))
}
// NodeFailure is a failed health check of a module's newest fingerprint on one node while it runs healthy on Others: the have's
// health, phase and words (HeldWords without a time), and the healthy witnesses by name.
type NodeFailure struct {
Others []string
Words string
}
// runs says whether a have is of a module the node runs: running, whatever its health.
func runs(h delivery.Have) bool {
return h.Phase == delivery.PhaseRunning
@@ -466,12 +600,7 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
// The fingerprint the bus ran before: its want's, or with no readable want the one its have names (ADR 0539 §15 bars a new one).
prev := before.FP
if !had {
prev = running
}
if catalogue.ProvidesBus(m.Manifest) && prev != m.FP {
if _, barred := m.busBar(before, had, running); barred {
return nil, true // the bus is never moved by the assigner; the operator writes its want (§15)
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
@@ -483,6 +612,16 @@ func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, runnin
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// busBar says whether m is the bus and the move would take it off the fingerprint it ran before, prev: its want's, or with no
// readable want the one its have names (ADR 0539 §15 bars a new one).
func (m Module) busBar(before delivery.Want, had bool, running string) (prev string, barred bool) {
prev = before.FP
if !had {
prev = running
}
return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP
}
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
@@ -87,6 +87,96 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
}
}
// 1126/AC-SW1, AC-SW2, AC-SW3: a want written Self while g14 was placed, kept by a rule after the placement is gone, names
// the controller's composition of the fingerprint the rule keeps (on 2026-10-27 g14's systemd-resolved, held, and forticlient,
// rule 5, kept their Self wants, which g14 could no longer compose); with no such composition, or while a record ask waits, it
// stays and is said; a placed node's want is Self with no composition.
selfAt := func(fp string) map[string]delivery.Want {
return map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}, "g14": {FP: fp, Self: true, Yes: dig(fp)}}
}
atF := func(_ *Module, n *Nodes) {
n.Haves["g14"]["mail"] = delivery.Have{FP: fNew, Composed: dig(fNew), Phase: delivery.PhaseFailed, Health: delivery.HealthOK}
}
for _, c := range []struct {
name string
set func(*Module, *Nodes)
before string
want delivery.Want // the want of g14 written; zero: none written
rule int // the rule g14's holding names
asked bool
}{
{name: "rule 5 at F", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
n.Haves["ace"]["store"] = delivery.Have{FP: fpOf('s'), Phase: delivery.PhaseRunning, Health: delivery.HealthWaits}
}, want: delivery.Want{FP: fNew, Composed: dig(fNew)}, rule: delivery.RuleProviders},
{name: "rule 3 at F", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
}, want: delivery.Want{FP: fNew, Composed: dig(fNew)}, rule: delivery.RuleCanary},
{name: "rule 3 held back at an older fingerprint", before: fOld, set: func(_ *Module, n *Nodes) {
n.Haves["ace"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
}, want: delivery.Want{FP: fOld, Composed: dig(fOld), Newest: fNew, Rule: delivery.RuleCanary,
Words: "the canary ace does not run bbbbbbbbbbbb healthy yet"}, rule: delivery.RuleCanary},
{name: "held at F with nothing to set back to", before: fNew, set: func(m *Module, n *Nodes) { atF(m, n); m.Held = true },
want: delivery.Want{FP: fNew, Composed: dig(fNew), Words: "mail is held at bbbbbbbbbbbb and g14 ran no other fingerprint healthy to set it back to"},
rule: delivery.RuleNotHeld},
{name: "held data module", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
m.Held = true
m.Manifest.Data = &catalogue.Data{Own: []catalogue.DataItem{{ID: "mail", Class: catalogue.ClassIrreplaceable}}}
}, want: delivery.Want{FP: fNew, Composed: dig(fNew),
Words: "mail holds data, so it is not set back from bbbbbbbbbbbb: the operator is asked whether to restore it"}, rule: delivery.RuleNotHeld},
{name: "a record ask waits", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
recording(m)
m.Held = true
m.Composed = func(_, fp string) (delivery.ComposedDigest, bool) {
return delivery.ComposedDigest{Digest: dig(fp), Judgement: delivery.JudgedRecreates, Against: dig(fOld)}, true
}
}, rule: delivery.RuleNotHeld, asked: true},
{name: "no composition of F", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
if node == "g14" {
return delivery.ComposedDigest{}, false
}
return composedEverywhere(node, fp)
}
}, rule: delivery.RuleComposed},
{name: "placed", before: fNew, set: func(m *Module, n *Nodes) {
atF(m, n)
m.Self = func(node string) bool { return node == "g14" }
m.Composed = func(node, fp string) (delivery.ComposedDigest, bool) {
if node == "g14" {
return delivery.ComposedDigest{Judgement: delivery.JudgedNeither}, true
}
return composedEverywhere(node, fp)
}
}, want: delivery.Want{FP: fNew, Self: true}},
} {
m, n := ready()
c.set(&m, &n)
before := selfAt(c.before)
if c.name == "placed" {
before["g14"] = delivery.Want{FP: fNew, Composed: dig(fNew)}
}
out := Wants(m, n, before, nil)
if got := out.Wants["g14"]; got != c.want {
t.Errorf("%s: g14's want = %+v, want %+v", c.name, got, c.want)
}
if got := out.Rules["g14"].Rule; got != c.rule {
t.Errorf("%s: g14 is held by rule %d (%s), want %d", c.name, got, out.Rules["g14"].Words, c.rule)
}
if asked := len(out.Record) == 1 && out.Record[0].Node == "g14"; asked != c.asked {
t.Errorf("%s: record asks %+v", c.name, out.Record)
}
if w, ok := out.Wants["g14"]; ok {
if _, err := delivery.EncodeWant(w); err != nil {
t.Errorf("%s: g14's want does not encode: %v", c.name, err)
}
}
}
// Rule 6 per node: a grant written for the module's other nodes does not grant a node whose account it does not name
// (issue 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus).
m, n := ready()
@@ -150,6 +240,72 @@ func TestAFailedHealthHoldsAndSetsBack(t *testing.T) {
if w, ok := out.Wants["g14"]; ok && w.FP == fNew {
t.Errorf("g14 moved to a held fingerprint: %+v", w)
}
// 1496, ADR 0557 §1, §2: the same failure while F runs healthy on another node is that node's: F is not held, nothing is set
// back, the failing node keeps its want, and the outcome names it (on ace a newly assigned module's first check held a version healthy for an hour on three nodes).
n.Haves["g14"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
before := map[string]delivery.Want{"ace": {FP: fNew, Composed: dig(fNew)}, "g14": {FP: fNew, Composed: dig(fNew)}}
out = Wants(m, n, before, map[string]string{"ace": fOld, "g14": fOld})
if out.Held != nil || len(out.Wants) != 0 {
t.Fatalf("held %+v, wants %+v: want no hold and no want changed", out.Held, out.Wants)
}
if f := out.NodeFailed["ace"]; !slices.Equal(f.Others, []string{"g14"}) || f.Words != "health failed, phase running: connection refused" ||
len(out.NodeFailed) != 1 {
t.Errorf("node failed = %+v", out.NodeFailed)
}
// ADR 0557 §2 (version 9): while one node fails F alone, novox, whose want names fOld, gets no new want naming F; the wants
// at F stay. The canary ace runs F healthy here and g14 fails it, so the canary rule (3) would let novox move: only §2 holds it.
three := m
three.Nodes = []string{"ace", "g14", "novox"}
failedHave := n.Haves["ace"]["mail"]
n.Haves["ace"]["mail"], n.Haves["g14"]["mail"] = n.Haves["g14"]["mail"], failedHave
n.Online["novox"] = true
n.Haves["novox"] = map[string]delivery.Have{"mail": {FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}}
before["novox"] = delivery.Want{FP: fOld, Composed: dig(fOld)}
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if _, alone := out.NodeFailed["g14"]; !alone {
t.Errorf("g14 does not fail alone: %+v", out.NodeFailed)
}
if w, written := out.Wants["novox"]; out.Held != nil || len(out.NodeFailed) != 1 || (written && w.FP == fNew) || out.Rules["novox"].Rule != delivery.RuleNotHeld {
t.Errorf("held %+v, node failed %+v, novox's want %+v (%+v): want novox kept off F", out.Held, out.NodeFailed, out.Wants["novox"], out.Rules["novox"])
}
if _, written := out.Wants["g14"]; written {
t.Errorf("g14's want at F was rewritten: %+v", out.Wants["g14"])
}
// A node that runs F healthy with no want, while g14 fails it alone, gets its want at F and is on no neither list (issue 1945).
n.Haves["novox"]["mail"] = delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
delete(before, "novox")
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if w := out.Wants["novox"]; w.FP != fNew || len(out.Neither) != 0 {
t.Errorf("novox running F with no want: want %+v, neither %+v; want its want at F and no neither", w, out.Neither)
}
n.Haves["ace"]["mail"], n.Haves["g14"]["mail"] = failedHave, n.Haves["ace"]["mail"]
// ADR 0557 §2: a witness that is offline keeps the have it last wrote and witnesses nothing (D1); F failing on a second node is
// not one node's failure (D3). Either way F is held and ace is set back.
for _, c := range []struct {
name string
online map[string]bool
novox delivery.Health
}{
{"the witness sleeps", map[string]bool{"ace": true, "g14": false, "novox": true}, delivery.HealthOK},
{"F fails on a second node", map[string]bool{"ace": true, "g14": true, "novox": true}, delivery.HealthFailed},
} {
n.Online = c.online
n.Haves["novox"] = map[string]delivery.Have{"mail": {FP: fNew, Phase: delivery.PhaseRunning, Health: c.novox}}
if c.novox == delivery.HealthOK {
n.Haves["novox"]["mail"] = delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
}
before["novox"] = delivery.Want{FP: fNew, Composed: dig(fNew)}
out = Wants(three, n, before, map[string]string{"ace": fOld, "g14": fOld, "novox": fOld})
if out.Held == nil || out.Held.Node != "ace" || len(out.NodeFailed) != 0 {
t.Errorf("%s: held %+v, node failed %+v: want F held at ace", c.name, out.Held, out.NodeFailed)
}
if w := out.Wants["ace"]; w.FP != fOld {
t.Errorf("%s: ace's want = %+v, want set back", c.name, w)
}
}
}
// 1297/B4: a module holding data is not set back; its want stays and says why.
@@ -427,6 +583,23 @@ func TestTheBusIsNotMovedButItsUserListRevisionIsWrittenAtOnce(t *testing.T) {
t.Errorf("a held bus did not take its user-list revision as a put: %+v", w)
}
m.Puts = nil
// 1945 (the bus bar made loud): a node that runs the bus at an old-form build and holds no want for it gets none, since only
// the operator's put moves the bus, and the pass says so: a holding naming the bar, and the node on neither list so its
// condition stays raised. A node with a live want keeps it and is not on neither list.
m.Composed = composedEverywhere
n.Haves["ace"]["mail"] = delivery.Have{FP: "b-1700000000-oldform", Phase: delivery.PhaseRunning, Health: delivery.HealthOK}
out = Wants(m, n, map[string]delivery.Want{"g14": {FP: fOld, Composed: dig(fOld)}}, nil)
if w, wrote := out.Wants["ace"]; wrote {
t.Errorf("the bus bar wrote ace's want %+v", w)
}
h, on := out.Neither["ace"]
if !on || !h.Bus || out.Rules["ace"] != h || !strings.Contains(h.Words, "only your put moves the bus") || !strings.Contains(h.Words, "ace runs b-1700000000-oldform") ||
!strings.Contains(h.Words, "assigner_put") {
t.Errorf("the barred bus on ace: neither %+v (%v), rules %+v", h, on, out.Rules["ace"])
}
if _, on := out.Neither["g14"]; on || out.Wants["g14"].FP == fNew {
t.Errorf("g14 has a live want: neither %+v, its want %+v", out.Neither, out.Wants["g14"])
}
stepsOfAPut(t)
}
@@ -7,6 +7,7 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func load(t *testing.T) catalogue.Manifest {
@@ -32,9 +33,9 @@ func TestManifest(t *testing.T) {
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants",
want := []string{"mesh-build-go.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants", "mesh-controller.lease",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"}
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
@@ -42,7 +43,8 @@ func TestManifest(t *testing.T) {
for _, s := range m.State {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20}; !reflect.DeepEqual(states, want) {
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -57,12 +59,27 @@ func TestManifest(t *testing.T) {
t.Errorf("uses = %v, want exactly node-build-go and operator-channel", m.Uses)
}
// ADR 0546: it serves its own tools (put, release, puts) from its bundle through the tool runner (the artifact loads the
// module), claims no seat (a new seat needs a change of the controller), says each put's step as the event put, and
// calls the controller's warranted and the bus node's backup holder.
if len(m.Claims) != 0 || !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts"}) {
t.Errorf("the assigner claims %v and serves tools %v, want no claim and its three put tools", m.Claims, m.Tools)
// module), says each put's step as the event put, and calls the controller's warranted and the bus node's backup holder.
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its seven verbs; keeps the seat's events on its stream
// placement for 30 days; calls the controller's export, its four verbs that keep its assignments in step and its
// hand-act log; and reads mesh-config's resolved values to compose every node a proposal touches.
if len(m.Claims) != 1 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs", m.Claims)
}
if want := []string{"seat:mesh-controller.warranted", "seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat))
}
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts"}) {
t.Errorf("the assigner serves tools %v, want its three put tools", m.Tools)
}
if len(m.Streams) != 1 || m.Streams[0].Name != delivery.PlacementStream || m.Streams[0].MaxAgeDays != delivery.PlacementStreamDays ||
!slices.Equal(slices.Sorted(slices.Values(m.Streams[0].Events)), slices.Sorted(slices.Values(delivery.PlacementEvents))) {
t.Errorf("streams = %+v, want placement carrying the seat's four events for %d days", m.Streams, delivery.PlacementStreamDays)
}
if want := []string{"seat:mesh-controller.assign", "seat:mesh-controller.hand-act", "seat:mesh-controller.pin",
"seat:mesh-controller.placement-export", "seat:mesh-controller.unassign", "seat:mesh-controller.unpin",
"seat:mesh-controller.warranted", "seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
if !slices.Equal(m.Emits, []string{"put"}) {
+42 -2
View File
@@ -2,18 +2,36 @@
"module": "mesh-delivery-assigner",
"version": "1",
"slug": "assigner",
"claims": [
{
"name": "mesh-placement",
"scope": "mesh",
"serves": [
"placement",
"proposals",
"propose",
"withdraw",
"again",
"adopt",
"hand-back"
]
}
],
"reads": [
"mesh-build-go.steps",
"mesh-config.resolved",
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-controller.composed",
"mesh-controller.grants",
"mesh-controller.lease",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"mesh-grants.grants",
"node-engine.have"
],
"uses": [
@@ -21,9 +39,15 @@
"operator-channel"
],
"invokes": [
"seat:mesh-controller.assign",
"seat:mesh-controller.hand-act",
"seat:mesh-controller.pin",
"seat:mesh-controller.placement-export",
"seat:mesh-controller.unassign",
"seat:mesh-controller.unpin",
"seat:mesh-controller.warranted",
"seat:node-backup.now",
"seat:node-backup.backed-up"
"seat:node-backup.backed-up",
"seat:node-backup.now"
],
"emits": [
"put"
@@ -34,12 +58,28 @@
"assigner_puts"
],
"state": [
{ "name": "placement", "history": 64 },
{ "name": "proposals", "history": 5 },
{ "name": "candidates", "history": 5 },
{ "name": "pulses", "key-ttl": true },
{ "name": "asks", "history": 5 },
{ "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 }
],
"streams": [
{
"name": "placement",
"events": [
"placement-proposed",
"placement-judged",
"placement-changed",
"placement-withdrawn"
],
"max-age-days": 30
}
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
@@ -7,6 +7,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
@@ -15,6 +16,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-builder/internal/build"
)
@@ -23,15 +25,16 @@ import (
type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue queueAsker, now func() time.Time) []stdio.Tool {
// the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{
{Name: build.StuckTool,
Description: "Every build key that waits (building or queued), oldest first: its module, fingerprint, step words and since when, and each ask it " +
"waits on by language with its id, when it was put, its asking (1 for the first; the builder asks an ask again when it can no longer be " +
"answered, at most 3 times) and what its build module's steps hold of it (none, or its state on its node since its start). The " +
"seat workers' delivery counts are not readable by the builder. With module, that module's keys only. Read-only.",
"seat workers' delivery counts are not readable by the builder. dead_letters says whether the builder hears the asks the build seats give up " +
"on (issue 734), and why not: heard, a given-up ask is asked again at once; deaf, at its check bound. With module, that module's keys only. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's keys"}}},
Run: func(args map[string]any) (any, error) { return r.Stuck(str(args, "module")) }},
@@ -55,6 +58,42 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
return map[string]any{"key": key, "asked": true,
"next": "the builder answers in " + build.ReAsksState + "." + key + "; " + build.StuckTool + " shows the answer and the new ask"}, nil
}},
{Name: build.BuildAgainTool,
Description: "Build a failed key again (rebuild a module, build a failed key again): the builder writes the key of module and " +
"fingerprint building as a new revision of the same key and asks the build seats, as it does by itself when it looks at a module " +
"again (ADR 0429 clause 18). Answers the new revision and the ask id of each build module (go, node, image). Refuses a key that " +
"is not failed, and says why: done (nothing to do), or waiting (building or queued: " + build.AskAgainTool + " asks its waiting asks " +
"again); a key failed for a reason the builder never retries on the same fingerprint (a pin missing, a version taken, digests that " +
"differ) and a fingerprint its module no longer has are refused too. Use it for a key builder_stuck does not show, which failed.",
Input: map[string]any{"type": "object", "required": []string{"module", "fp", "why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name"},
"fp": map[string]any{"type": "string", "description": "its new-form fingerprint, src2:<hex>"},
"why": map[string]any{"type": "string", "description": "why it is built again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := build.NewBuildAgain(str(args, "module"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
raw, err := buildAgain(key, value)
if err != nil {
return nil, err
}
r, err := build.DecodeReAsk(raw)
if err != nil {
return nil, err
}
if r.Refused {
return nil, errors.New(r.Answer)
}
if r.Failed != "" {
return nil, errors.New(r.Answer)
}
if r.Revision == 0 && !r.Pending {
return nil, fmt.Errorf("the builder answered neither a new revision nor a refusal (an older builder?): %s", r.Answer)
}
return map[string]any{"key": key, "revision": r.Revision, "asks": r.Asks, "answer": r.Answer,
"next": build.StuckTool + " shows the key and its asks while they wait"}, nil
}},
{Name: build.SeatQueueTool,
Description: "What waits on a build seat's stream (SEAT_NODE_BUILD_<LANGUAGE>), as the builder reads it: the stream's message count and " +
"sequences, and per kind (delivery, check, recheck) its messages oldest first, at most limit each, with sequence, ask id, when asked " +
@@ -85,8 +124,17 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
// askQueue writes the queues value at key and waits, on a watch of the key, at most bound for the builder's answer.
func askQueue(s stdio.KeptState, bound time.Duration) queueAsker {
return askKept(s, build.QueuesState, build.QueuesBucket, "the read", bound, func(v []byte) bool {
a, err := build.DecodeQueueAsk(v)
return err == nil && !a.Answered.IsZero()
})
}
// askKept writes a value at key of a state the builder watches and answers there, and waits, on a watch of the key, at most
// bound for a value written after it that answered says is the builder's answer.
func askKept(s stdio.KeptState, state, bucket, what string, bound time.Duration, answered func([]byte) bool) queueAsker {
return func(key string, value []byte) (json.RawMessage, error) {
answered := make(chan json.RawMessage, 1)
heard := make(chan json.RawMessage, 1)
var rev uint64
var mu sync.Mutex
if err := s.Watch(key, func(c stdio.StateChange) error {
@@ -96,36 +144,34 @@ func askQueue(s stdio.KeptState, bound time.Duration) queueAsker {
if c.Op != "put" || mine == 0 || c.Revision <= mine {
return nil
}
if a, err := build.DecodeQueueAsk(c.Value); err == nil && !a.Answered.IsZero() {
if answered(c.Value) {
select {
case answered <- c.Value:
case heard <- c.Value:
default:
}
}
return nil
}); err != nil {
return nil, fmt.Errorf("watching %s.%s: %w", build.QueuesState, key, err)
return nil, fmt.Errorf("watching %s.%s: %w", state, key, err)
}
r, err := s.Put(key, json.RawMessage(value))
if err != nil {
return nil, fmt.Errorf("the read could not be written to %s: %w", build.QueuesBucket, err)
return nil, fmt.Errorf("%s could not be written to %s: %w", what, bucket, err)
}
mu.Lock()
rev = r
mu.Unlock()
// The watch may have heard the answer before the revision was known: read it once too.
if e, err := s.Get(key); err == nil && e != nil && e.Revision > r {
if a, err := build.DecodeQueueAsk(e.Value); err == nil && !a.Answered.IsZero() {
return e.Value, nil
}
if e, err := s.Get(key); err == nil && e != nil && e.Revision > r && answered(e.Value) {
return e.Value, nil
}
wait, cancel := context.WithTimeout(context.Background(), bound) // a context's bound, no timer (ADR 0437 §8)
defer cancel()
select {
case v := <-answered:
case v := <-heard:
return v, nil
case <-wait.Done():
return nil, fmt.Errorf("the builder did not answer %s.%s within %s: it is not running, or cannot read %s", build.QueuesState, key, bound, build.QueuesBucket)
return nil, fmt.Errorf("the builder did not answer %s.%s within %s: it is not running, or cannot read %s", state, key, bound, bucket)
}
}
}
@@ -166,12 +212,17 @@ func main() {
steps[bm] = read(stdio.State(bm + ".steps"))
}
reasks := stdio.State(build.ReAsksState)
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps}
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps,
Hearing: read(stdio.State(buildrun.HearingState))}
put := func(key string, value []byte) error {
_, err := reasks.Put(key, json.RawMessage(value))
return err
}
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), time.Now)); err != nil {
buildAgain := askKept(reasks, build.ReAsksState, build.ReAsksBucket, "the request", 25*time.Second, func(v []byte) bool {
a, err := build.DecodeReAsk(v)
return err == nil && !a.Answered.IsZero()
})
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1)
}
@@ -24,6 +24,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build"
)
@@ -345,7 +346,7 @@ func (b *bed) archive(t *testing.T, name, path, at string) string {
func (b *bed) start(t *testing.T, node string, env map[string]string) {
t.Helper()
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(credential{URL: b.srvURL, Node: node, Module: "mesh-delivery-builder", User: node + ".mesh-delivery-builder", Password: "pw"})
cred, _ := json.Marshal(process.Account{URL: b.srvURL, Node: node, Module: "mesh-delivery-builder", User: node + ".mesh-delivery-builder", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
@@ -6,12 +6,6 @@ package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
@@ -51,80 +45,18 @@ func main() {
}
}
// credential is the broker file the controller issues to the module: JSON of the bus address, the certificate
// fingerprint to pin, the principal and its password (as the indexer reads it; issue 1242 moves this into the SDK).
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
func readCredential(path string) (credential, error) {
raw, err := os.ReadFile(path)
if err != nil {
return credential{}, err
}
var c credential
if err := json.Unmarshal(raw, &c); err != nil {
return credential{}, fmt.Errorf("the broker credential is not one: %w", err)
}
if c.URL == "" || c.User == "" {
return credential{}, errors.New("the broker credential names no bus address or user")
}
return c, nil
}
// pinned accepts exactly the certificate with this SHA-256 and no other.
func pinned(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
if got := "sha256:" + hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%.23s…)", got)
}
return nil
},
}
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-builder: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to hear the bus with")
}
cred, err := readCredential(path)
cred, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
nats.CustomInboxPrefix("_INBOX." + cred.User),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
conn, js, err := cred.Dial()
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
return err
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
builds, err := process.EssentialBucket(ctx, js, "builder", delivery.BuilderBuildsBucket, "claim or write a build key")
if err != nil {
return err
@@ -149,20 +81,37 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
asks := buildrun.JetStreamAsks{JS: js}
// The asks the build seats gave up on (issue 734): heard from DEAD_LETTERS, each ends the wait for its step at once,
// and the lane asks its part again (Builder.waitOnce) instead of waiting for the check bound.
// Heard in the background, and again at a reconnect or an ask while it is not; whether it is heard is kept in the
// state hearing, which builder_stuck says (dead_letters).
dead := &buildrun.DeadLetters{}
// The state may not be on the bus yet when a delivery started the builder (the grants part creates it, issue 1790): it
// is bound when the bus says it was created, and the latest hearing is written then.
hearingKV := process.AwaitBucket(ctx, js, conn, delivery.BuilderModule+"_"+buildrun.HearingState, "builder_stuck cannot say whether DEAD_LETTERS is heard", logger.Printf)
hearing := buildrun.NewDeadLetterHearing(ctx, buildrun.JetStreamDeadLetters{JS: js}, delivery.AskLanguages, dead, logger.Printf,
buildrun.SayHearing(ctx, hearingKV, logger.Printf))
defer hearing.Stop()
// Each ask's step is heard by its own watch and, at every step end of its build module and every reconnect, read
// (buildrun.HearEnds): a watch that goes deaf loses no end (the lost check of 2026-10-28).
var buildModules []string
for _, l := range delivery.AskLanguages {
if m, err := delivery.BuildModuleOf(l); err == nil {
buildModules = append(buildModules, m)
}
}
heard := buildrun.HearEnds(ctx, asks, asks, buildModules, logger.Printf)
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
calls, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer calls.Close()
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: bound},
Asks: buildrun.Asker{Publisher: asks, Waiter: buildrun.LostAskWaiter{Inner: heard, Dead: dead, Hearing: hearing}, Bound: bound},
Held: heldOnTheBus{js},
// An ask that can no longer be answered is asked again (issue 1945), read at the step bound: a context's bound.
Gone: heldOnTheBus{js},
@@ -204,11 +153,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
// The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, b.NodesChanged, logger)
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, b.NodesChanged, logger.Printf)
if err != nil {
return err
}
@@ -220,13 +169,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := js.Consumer(ctx, stream, cred.Node+"_"+cred.Module)
cons, err := cred.EventConsumer(ctx, js, stream)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the builder starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := cons.Consume(func(m jetstream.Msg) { watch.Reported(m) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { watch.Reported(m) }, logger.Printf)
if err != nil {
return err
}
@@ -235,13 +183,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The controller says artifacts-gone when its take-in found a key's blob gone (issue 1937): the builder's durable
// consumer on EVENTS, made by the controller from the manifest's consumes. Where it is not there yet the builder says
// so; it still asks held at its start and at every module value.
switch cons, err := js.Consumer(ctx, eventsStream, cred.Node+"_"+cred.Module); {
switch cons, err := cred.EventConsumer(ctx, js, eventsStream); {
case err != nil:
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s is not heard until the builder starts again: %v",
cred.Node, cred.Module, eventsStream, delivery.ArtifactsGoneEvent, err)
default:
cc, err := cons.Consume(func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) }, logger.Printf)
if err != nil {
return err
}
@@ -280,24 +227,25 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
// A person asks a key's build again with builder_ask_again, which writes the reasks state (issue 1945): watched, as
// every other input. The builder asks again by itself without it.
if kv := process.OptionalBucket(ctx, js, build.ReAsksBucket, "builder_ask_again is not heard until the builder starts again; an ask that can no longer be answered is still asked again by itself", logger.Printf); kv != nil {
// A state not on the bus yet at this start is watched once it appears (issue 1790).
process.AwaitBucket(ctx, js, conn, build.ReAsksBucket, "builder_ask_again is not heard; an ask that can no longer be answered is still asked again by itself", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachReAsk := func(e jetstream.KeyValueEntry) { reAsked(ctx, b, kv, e, logger.Printf) }
if rw, err := settle(ctx, kv, eachReAsk); err != nil {
process.NotWatched(logger.Printf, build.ReAsksBucket, err, "builder_ask_again is not heard until the builder starts again")
} else {
go func() { errs <- follow(ctx, rw, eachReAsk) }()
}
}
})
// A person reads a build seat's stream with builder_seat_queue, which writes the queues state (issue 1945 note 62): the
// builder alone may read the stream, so it answers there.
if kv := process.OptionalBucket(ctx, js, build.QueuesBucket, "builder_seat_queue is not answered until the builder starts again", logger.Printf); kv != nil {
process.AwaitBucket(ctx, js, conn, build.QueuesBucket, "builder_seat_queue is not answered", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachRead := func(e jetstream.KeyValueEntry) { queueRead(ctx, heldOnTheBus{js}, kv, e, logger.Printf) }
if qw, err := settle(ctx, kv, eachRead); err != nil {
process.NotWatched(logger.Printf, build.QueuesBucket, err, "builder_seat_queue is not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, qw, eachRead) }()
}
}
})
logger.Printf("started on %s", cred.Node)
go func() { errs <- follow(ctx, w, each) }()
select {
@@ -306,7 +254,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
case err := <-errs:
return err
case err := <-assignErrs:
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
@@ -8,6 +8,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
@@ -17,8 +18,13 @@ import (
// askAgainer is the part of the builder a reasks value asks through.
type askAgainer interface {
AskAgain(key, why string) []string
BuildAgain(ctx context.Context, module, fp, why, by string, bound time.Duration) build.BuildAgainResult
}
// buildAgainBound is how long the builder waits for the claim and the asks of a key built again by hand; the tool waits
// longer for the builder's answer.
const buildAgainBound = 15 * time.Second
// reAsked answers one value of the reasks state (the tool builder_ask_again, issue 1945): a value not yet answered asks the
// key's waiting lanes again and is written back, by compare-and-set at its revision, with what the builder did. A value
// answered already (this builder's own write) or deleted is left.
@@ -34,6 +40,11 @@ func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstre
if r.Answer != "" {
return
}
if r.Mode == build.ModeBuild {
// The answer comes when the claim and the asks are written, so it is made beside the watch, not in it.
go builtAgain(ctx, b, kv, e, r, logf)
return
}
asked := b.AskAgain(e.Key(), r.Why)
if len(asked) == 0 {
r.Answer = "no ask of this key waits in this builder now (it is not building, or its build has not reached its ask yet): nothing was asked; builder_stuck shows where it stands"
@@ -53,12 +64,12 @@ func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstre
// emitAskedAgain says the builder's event build.asked-again (issue 1945): mesh.mod.<module>.event.build.asked-again,
// with the SDK's envelope headers and the new ask's id as the bus's dedup id.
func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred credential, e build.AskedAgain) error {
func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred process.Account, e build.AskedAgain) error {
data, err := json.Marshal(e)
if err != nil {
return err
}
msg := nats.NewMsg("mesh.mod." + cred.Module + ".event." + build.AskedAgainEvent)
msg := nats.NewMsg(cred.EventSubject(build.AskedAgainEvent))
id := "asked-again-" + e.New
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", cred.Module)
@@ -73,3 +84,33 @@ func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred credential
}
return nil
}
// builtAgain answers a reasks value of mode build (the tool builder_build_again, issue 1126 note 72): the failed key is
// built again as a new revision, and the value is written back with the revision and the asks put, or why nothing was done.
func builtAgain(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, r build.ReAsk, logf func(string, ...any)) {
res := b.BuildAgain(ctx, r.Module, r.FP, r.Why, r.By, buildAgainBound)
switch {
case res.Refused != "":
r.Answer, r.Refused = res.Refused, true
case res.Failed != "":
r.Answer, r.Revision, r.Failed = fmt.Sprintf("the key is written building as revision %d, and the build failed before any ask was put: %s", res.Revision, res.Failed), res.Revision, res.Failed
default:
r.Pending = res.Revision == 0
r.Answer, r.Revision, r.Asks = "built again: the key is written building as revision "+fmt.Sprint(res.Revision)+" and its asks are put", res.Revision, res.Asks
if res.Pending != "" {
r.Answer = res.Pending
if res.Revision != 0 {
r.Answer = "built again: the key is written building as revision " + fmt.Sprint(res.Revision) + "; " + res.Pending
}
}
}
r.Answered = time.Now().UTC()
logf("%s: build again asked by %s (%s): %s", e.Key(), r.By, r.Why, r.Answer)
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
@@ -0,0 +1,73 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,228 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1 h1:fzOkrz9CSp3mEjeV7NfZx8yoivI1sD1AOtIPtyyF8RI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c h1:hcqOCd5q3M0ZiEzB6MhwxT/AOKkm9gaskKqPcpkAAg8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -12,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **An ask that can no longer be answered is asked again** (novox/hq issue 1945). A build seat's stream gives an ask up
@@ -20,7 +21,8 @@ import (
// restart take over the same id and wait again. The builder notices it at the check bound (CheckAfter, the step bound),
// a context's bound and no timer of its own: it asks Gone, which reads the bus, and asks such an ask again under a new
// id, written into the key, said in its log and as the event build.asked-again, at most Askings times; then the key is
// failed with why. A person asks the same by hand with the tool builder_ask_again (AskAgain).
// failed with why. An ask DEAD_LETTERS says the seat gave up (buildrun.LostAskWaiter, issue 734) is asked again at once.
// A person asks the same by hand with the tool builder_ask_again (AskAgain).
// DefaultAskings is how many times a build module's part of a build is asked, the first included, when Builder.Askings is 0.
const DefaultAskings = 3
@@ -315,6 +317,11 @@ func (b *Builder) waitOnce(ctx context.Context, ask delivery.StepAsk, again <-ch
defer stop()
}
step, err := b.Asks.Wait(wait, ask)
if err == nil && buildrun.IsLost(step) {
// The build seat gave the ask up (issue 734): it can no longer be answered, so it is asked again now, with the
// words of why, not at the check bound.
return step, step.Reason, errAgain
}
if err == nil || ctx.Err() != nil {
return step, "", err
}
@@ -0,0 +1,183 @@
package build
import (
"context"
"fmt"
"maps"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **A failed key is built again by hand** (novox/hq issue 1126 note 72; ADR 0429 clause 18: "a fingerprint is built again
// only when its build key failed, as a new revision of the same key"). The builder does it by itself whenever it looks at a
// module again (a modules value, a change of the toolchains, artifacts-gone, a moved trunk): Claimer.Claim writes a failed
// key building as a new revision and the build asks the build seats. Nothing looks at an unchanged module on its own, so a key
// that failed (a node refused the build, a toolchain was missing) stayed failed until the module changed. The tool
// builder_build_again (BuildAgain) makes the builder look at one module now, which does exactly that and no more: it keeps
// the claim's rules, so a key failed for a reason the builder never retries on the same fingerprint stays refused.
// BuildAgainResult is the builder's answer to a request to build a key again.
type BuildAgainResult struct {
// Refused is why nothing was done, "" when the key was written building as a new revision.
Refused string
// Revision is the revision the claim wrote; Asks are the ask ids put by language, as far as they were put within the bound.
Revision uint64
Asks map[string]string
// Pending says the builder had not claimed the key, or not put every ask, when it answered: not a refusal.
Pending string
// Failed is the error of the key when the build ended failed before its asks were put (a plan that cannot be read, a
// toolchain not agreed): the new revision was written, and nothing was asked.
Failed string
}
// handWatch follows one request to build a key again until its asks are put.
type handWatch struct {
rev uint64
lanes int
asks map[string]string
done chan struct{}
shut bool
// ended is set when the look ended before the asks were all put.
ended bool
}
// settle closes done once the claim is written and every lane has put its ask. Call with b.mu held.
func (w *handWatch) settle() {
if !w.shut && w.rev > 0 && w.lanes >= 0 && len(w.asks) >= w.lanes {
w.shut = true
close(w.done)
}
}
func (b *Builder) hand(key string, f func(w *handWatch)) {
b.mu.Lock()
defer b.mu.Unlock()
if w := b.hands[key]; w != nil {
f(w)
}
}
func (b *Builder) handClaimed(key string, rev uint64) {
b.hand(key, func(w *handWatch) { w.rev = rev; w.settle() })
}
func (b *Builder) handLanes(key string, n int) {
b.hand(key, func(w *handWatch) { w.lanes = n; w.settle() })
}
func (b *Builder) handAsk(key, language, id string) {
b.hand(key, func(w *handWatch) { w.asks[language] = id; w.settle() })
}
// handEnded ends the watch of a request whose look ended: it wrote no claim, or the build ended.
func (b *Builder) handEnded(key string) {
b.hand(key, func(w *handWatch) {
if !w.shut {
w.shut, w.ended = true, true
close(w.done)
}
})
}
// BuildAgain builds the failed key of module at fp again, as the builder does by itself: the key is written building as a
// new revision and the build module asks are put on the build seats. It answers once the asks are put, or at bound. A key
// that is not failed is refused with why: done (nothing to do) or waiting (builder_ask_again asks its waiting asks again).
// ctx is the builder's own context: the build it starts lives on it.
func (b *Builder) BuildAgain(ctx context.Context, module, fp, why, by string, bound time.Duration) BuildAgainResult {
b.init()
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return BuildAgainResult{Refused: err.Error()}
}
value, _, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s cannot be read: %v", key, err)}
}
if !found {
return BuildAgainResult{Refused: fmt.Sprintf("%s has no build key: nothing failed. The builder builds a fingerprint by itself once the indexer writes it to its module", key)}
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s holds what is not a build: %v", key, err)}
}
switch {
case cur.State == delivery.BuildDone:
return BuildAgainResult{Refused: fmt.Sprintf("%s is done: nothing to do", key)}
case cur.State == delivery.BuildBuilding || cur.State == delivery.BuildQueued:
return BuildAgainResult{Refused: fmt.Sprintf("%s is waiting (%s: %s): %s asks its waiting asks again", key, cur.State, cur.Step, AskAgainTool)}
case cur.State != delivery.BuildFailed:
return BuildAgainResult{Refused: fmt.Sprintf("%s is %q, not failed", key, cur.State)}
case delivery.FailedUntilChanged(cur.Error) || strings.HasPrefix(cur.Error, buildrun.DigestsDifferPrefix):
return BuildAgainResult{Refused: fmt.Sprintf("%s failed for a reason the builder does not retry on the same fingerprint, it waits for another fingerprint: %s", key, cur.Error)}
}
// One critical section: the key is refused when it is looked at or asked already, the module still has the fingerprint,
// and the watch and the look are registered before anything else can take the key.
w := &handWatch{lanes: -1, asks: map[string]string{}, done: make(chan struct{})}
b.mu.Lock()
m, known := b.known[module]
switch {
case !known || m.FP2 != fp:
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("module %s does not have the fingerprint %s now (a newer one superseded it, or its modules value is not read yet): the builder builds the module's current fingerprint", module, fp)}
case m.Manifest == "":
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("module %s names no manifest at %s yet, so the builder looks at nothing", module, fp)}
case b.inflight[key] || b.hands[key] != nil:
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("%s is being looked at by this builder now (or asked again by hand): wait for it to end", key)}
}
b.hands[key] = w
b.inflight[key] = true
b.mu.Unlock()
defer func() {
b.mu.Lock()
if b.hands[key] == w {
delete(b.hands, key)
}
b.mu.Unlock()
}()
b.logf("%s: failed (%s); built again by hand, asked by %s: %s", key, cur.Error, by, why)
b.startOnce(ctx, key, module, m)
wait, stop := context.WithTimeout(ctx, bound) // a context's bound for a person's answer (ADR 0437 §8)
defer stop()
select {
case <-w.done:
case <-wait.Done():
}
b.mu.Lock()
res := BuildAgainResult{Revision: w.rev, Asks: maps.Clone(w.asks)}
ended, shut, lanes := w.ended, w.shut, w.lanes
b.mu.Unlock()
switch {
case res.Revision == 0 && shut:
res.Refused = fmt.Sprintf("the builder looked at %s and wrote nothing (a toolchain moved, the claim was lost or could not be written): its log says why", key)
case res.Revision == 0:
res.Pending = fmt.Sprintf("the builder has not claimed %s within %s: it waits behind other builds; builder_stuck shows where it stands", key, bound)
case ended && len(res.Asks) < max(lanes, 0)+boolInt(lanes < 0):
// The look ended before every ask was put: say how the key stands, not that it was asked.
if v, _, found, err := b.Claimer.Store.Get(ctx, key); err == nil && found {
if cur, err := delivery.DecodeBuilderBuild(v); err == nil {
switch cur.State {
case delivery.BuildFailed:
res.Failed = cur.Error
case delivery.BuildDone:
default:
res.Pending = fmt.Sprintf("the look ended with the key %s (%s) before every ask was put: its log says why", cur.State, cur.Step)
}
}
}
case !shut:
res.Pending = fmt.Sprintf("the key is building at revision %d, and not every ask was put within %s; builder_stuck shows the asks", res.Revision, bound)
}
return res
}
func boolInt(b bool) int {
if b {
return 1
}
return 0
}
@@ -89,6 +89,8 @@ type Builder struct {
held map[string]bool
// waiting are the lanes that wait for an ask's step, by build key and language: AskAgain asks through them.
waiting map[string]map[string]chan string
// hands are the requests to build a key again by hand that wait for the claim and its asks (BuildAgain), by build key.
hands map[string]*handWatch
}
type planned struct {
@@ -106,6 +108,7 @@ func (b *Builder) init() {
b.cands, b.via, b.trunkFP = map[string]delivery.Candidate{}, map[string]via{}, map[string]bool{}
b.known, b.inflight, b.again, b.plans, b.tried = map[string]delivery.Module{}, map[string]bool{}, map[string]delivery.Module{}, map[string]planned{}, map[string]bool{}
b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{}
b.hands = map[string]*handWatch{}
})
}
@@ -372,9 +375,35 @@ func (b *Builder) consider(ctx context.Context, module string, m delivery.Module
}()
}
// startOnce looks at the module for a key the caller has set inflight already (BuildAgain): the by-hand look runs once, and
// a look the builder queued for the key meanwhile (again) still runs after it, as consider's loop does.
func (b *Builder) startOnce(ctx context.Context, key, module string, m delivery.Module) {
b.wg.Add(1)
go func() {
defer b.wg.Done()
b.sem <- struct{}{}
defer func() { <-b.sem }()
for {
b.try(ctx, module, m)
b.mu.Lock()
next, more := b.again[key]
delete(b.again, key)
if !more || ctx.Err() != nil {
delete(b.inflight, key)
b.mu.Unlock()
return
}
b.mu.Unlock()
m = next // a look the builder queued meanwhile is not lost (consider's rule); it is not a second by-hand build
}
}()
}
// try claims the fingerprint and builds it, writing the key done or failed. A build that ends because ctx did is left
// building by this builder: it takes it again when it starts (ADR 0478 clause 5).
func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
key, _ := delivery.BuilderBuildKey(module, m.FP2)
defer b.handEnded(key)
plan, planErr, read := b.planOf(ctx, m.Manifest)
nothing := read && planErr == nil && len(plan.Tasks) == 0
if planErr == nil && !nothing {
@@ -385,7 +414,6 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
return
}
}
key, _ := delivery.BuilderBuildKey(module, m.FP2)
b.mu.Lock()
trunk := b.trunkFP[key]
first := !b.tried[key]
@@ -400,6 +428,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
if cl == nil {
return
}
b.handClaimed(key, cl.Rev)
if cl.Unasked != nil {
b.logf("%v", cl.Unasked)
}
@@ -535,6 +564,9 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
// One ask builds each fingerprint's part of a build module, on whichever node pulls it (ADR 0525 §3).
lanes = append(lanes, &lane{module: bm, tasks: tasks, toolchain: toolchain, digests: map[string]string{}, references: map[string]string{}})
}
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.handLanes(key, len(lanes))
}
var wg sync.WaitGroup
for _, l := range lanes {
wg.Add(1)
@@ -717,7 +749,11 @@ func (n *askNotes) note(ctx context.Context, language, id, step string) error {
v.Asks = map[string]string{}
}
v.Asks[language] = id
return n.b.Claimer.Finish(ctx, n.cl, v)
if err := n.b.Claimer.Finish(ctx, n.cl, v); err != nil {
return err
}
n.b.handAsk(n.cl.Key, language, id)
return nil
}
// runLane puts the lane's part of the build as one ask on its build seat's stream (the delivery stream for a trunk key, the
@@ -690,17 +690,130 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
step string
byHand bool
always bool // Gone says every ask cannot be answered, not only the first
dead bool // DEAD_LETTERS keeps the first ask: the seat gave it up (issue 734)
deaf bool // the ask's own watch hears nothing though its step ends (the lost check of 2026-10-28)
puts int
done bool
asking int // the asking the event says, 0 when none is said
// again, when set, is a request to build the key again by hand (BuildAgain, issue 1126 note 72) of the key as
// it stands (key, absent when nil), the module known at known's fingerprint ("" for the rig's): refuse is a part of why it
// is refused, "" when it is built again as a new revision and its ask id answered.
again bool
key *delivery.BuilderBuild
known string
refuse string
// edit changes the module the builder has read, busy marks the key looked at (1) or asked by hand (2), failed is a part of
// the error the key then holds when its build ends before its asks are put.
edit func(m *delivery.Module)
busy int
failed string
// during is a look of the builder's own at the module while the by-hand build runs: it still runs afterwards.
during bool
}{
{name: "build again: a look the builder queued during the by-hand build still runs afterwards", again: true, during: true,
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a key looked at by the builder now is refused, not built twice", again: true, refuse: "is being looked at", busy: 1,
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a key already asked by hand is refused, its watch kept", again: true, refuse: "is being looked at", busy: 2,
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a module with no manifest is refused", again: true, refuse: "names no manifest", edit: func(m *delivery.Module) { m.Manifest = "" },
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a toolchain that moved writes nothing and is said at once", again: true, refuse: "wrote nothing",
edit: func(m *delivery.Module) {
m.Toolchains = map[string]delivery.ToolchainUsed{"go": {Version: "go1.0", Recipe: "1"}}
},
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a build that fails before its asks says the key failed, not that it was asked", again: true, failed: "cannot be read",
edit: func(m *delivery.Module) { m.Manifest = mdigest('b') },
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "build again: a failed key is written building as a new revision and asks the seat", again: true,
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: the step failed: no tun"}},
{name: "build again: a done key is refused, nothing to do", again: true, refuse: "is done: nothing to do",
key: &delivery.BuilderBuild{State: delivery.BuildDone, Step: "built"}},
{name: "build again: a building key is refused, builder_ask_again is for it", again: true, refuse: "is waiting",
key: &delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: "mvl0frv18d1960"}}},
{name: "build again: a key failed for a pin missing is refused, it waits for another fingerprint", again: true, refuse: "does not retry",
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: delivery.NotPinned + "web"}},
{name: "build again: a key with no build is refused", again: true, refuse: "has no build key"},
{name: "build again: a fingerprint its module no longer has is refused", again: true, refuse: "does not have the fingerprint", known: "src2:" + strings.Repeat("9", 64),
key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "ace: no tun"}},
{name: "given up by its stream: asked again under a new id and done from it", puts: 2, done: true, asking: 2},
{name: "asked again by hand: the same, with no check bound", byHand: true, puts: 2, done: true, asking: 2},
{name: "given up into DEAD_LETTERS: its wait ends failed with the words and it is asked again at once", dead: true, puts: 2, done: true, asking: 2},
{name: "its step ended while its watch was deaf: read at its build module's next step end, done with no new ask", deaf: true, puts: 1, done: true},
{name: "never answerable: failed after its askings", askings: 2, always: true, puts: 2},
{name: "taken over at its last asking: failed, not asked again", askings: 3, always: true, step: "asked mesh-build-go again (asking 3): x", puts: 0},
} {
t.Run(c.name, func(t *testing.T) { // 1945
r := newRig(t, goBundle)
if c.again {
var rev uint64
if c.key != nil {
c.key.Manifest = manifestD
rev = putBuild(t, r.st, r.key, *c.key)
}
m, err := delivery.DecodeModule(moduleValue(t, "modules/m"))
if err != nil {
t.Fatal(err)
}
if c.known != "" {
m.FP2 = c.known
}
if c.edit != nil {
c.edit(&m)
}
r.b.init()
var stored atomic.Int64
if c.during {
r.asks.Answer = nil
r.b.Stored = func(context.Context, []string) ([]string, error) { stored.Add(1); return nil, nil }
}
switch c.busy {
case 1:
r.b.inflight[r.key] = true
case 2:
r.b.hands[r.key] = &handWatch{}
}
r.b.known["m"] = m // the builder has read the modules value, and looks at nothing until asked
log := &logged{}
r.b.Logf = log.logf
res := r.b.BuildAgain(context.Background(), "m", fp1, "no tun on ace", "shanks.lead", 2*time.Second)
if c.during {
// The trunk's modules value arrives while the by-hand build waits for its ask: it is queued, and run after it.
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.asks.End(r.answer(waitForAsks(t, r, 1)[0]))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || res.Revision <= rev || stored.Load() != 1 {
t.Fatalf("key %+v, answer %+v, %d asks of the store; want built from the new revision and the queued look run once after it", v, res, stored.Load())
}
return
}
r.b.Wait()
after := r.st.revs[r.key]
if c.refuse != "" {
if !strings.Contains(res.Refused, c.refuse) || res.Revision != 0 || len(r.asks.Asks()) != 0 || after != rev {
t.Fatalf("refused %q (revision %d, %d asks, key at revision %d was %d); want refused with %q and nothing written or asked", res.Refused, res.Revision, len(r.asks.Asks()), after, rev, c.refuse)
}
return
}
if c.failed != "" {
if !strings.Contains(res.Failed, c.failed) || res.Revision <= rev || len(res.Asks) != 0 || len(r.asks.Asks()) != 0 {
t.Fatalf("answer %+v after %d asks; want a new revision (was %d), no ask and the key's failure %q", res, len(r.asks.Asks()), rev, c.failed)
}
return
}
asks := r.asks.Asks()
if res.Refused != "" || res.Pending != "" || res.Revision <= rev || len(asks) != 1 || res.Asks[delivery.AskGo] != asks[0].ID {
t.Fatalf("answer %+v after %d asks; want the new revision (was %d) and the ask id of the one ask", res, len(asks), rev)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || after <= res.Revision {
t.Errorf("the key = %+v at revision %d; want built from the new revision %d", v, after, res.Revision)
}
if !log.has(r.key, "built again by hand", "no tun on ace", "shanks.lead") {
t.Errorf("the why is not logged: %v", log.lines)
}
return
}
r.asks.Answer = nil
r.b.Askings = c.askings
var said []AskedAgain
@@ -714,7 +827,16 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r.st.vals[r.key] = data
}
gone := &goneAsks{}
if !c.byHand {
dead := &buildrun.DeadLetters{}
if c.dead {
r.b.Asks = buildrun.Asker{Publisher: r.asks, Waiter: buildrun.LostAskWaiter{Inner: r.asks, Dead: dead}, Bound: 2 * time.Second}
}
var heard *buildrun.HeardEnds
if c.deaf {
heard = &buildrun.HeardEnds{Inner: silentWatch{}, Read: r.asks}
r.b.Asks = buildrun.Asker{Publisher: r.asks, Waiter: heard, Bound: 2 * time.Second}
}
if !c.byHand && !c.dead && !c.deaf {
gone.why = func(a delivery.StepAsk) string {
if c.always || a.ID == first || first == "" {
return "it waits first in line while a node took a later ask"
@@ -724,6 +846,10 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
r.b.Gone, r.b.CheckAfter = gone, 5*time.Millisecond
}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
if c.dead {
dead.Heard(buildrun.GivenUp{AskID: waitForAsks(t, r, 1)[0].ID, Stream: "SEAT_NODE_BUILD_GO", Deliveries: 5,
At: time.Date(2026, 10, 26, 12, 26, 40, 0, time.UTC)})
}
if c.byHand {
waitForAsks(t, r, 1)
for deadline := time.Now().Add(2 * time.Second); len(r.b.AskAgain(r.key, "stuck since 19:06")) == 0; time.Sleep(time.Millisecond) {
@@ -732,7 +858,16 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
}
}
}
if c.done {
if c.deaf {
r.asks.End(r.answer(waitForAsks(t, r, 1)[0]))
for deadline := time.Now().Add(2 * time.Second); heard.Waiting() == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask is waited on")
}
}
heard.Nudge(buildrun.GoBuildModule) // a step of mesh-build-go ended
}
if c.done && !c.deaf {
asks := waitForAsks(t, r, c.puts)
if first == "" && !c.byHand {
first = asks[0].ID
@@ -757,13 +892,16 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
mu.Lock()
defer mu.Unlock()
// Note 62: the old message of an ask asked again because its stream gave it up is dropped from the stream.
if gone.mu.Lock(); c.done && !c.byHand && (len(gone.dropped) != 1 || gone.dropped[0] != first) {
if gone.mu.Lock(); c.done && !c.byHand && !c.dead && !c.deaf && (len(gone.dropped) != 1 || gone.dropped[0] != first) {
t.Errorf("dropped %v from the stream, want the old ask %s", gone.dropped, first)
}
gone.mu.Unlock()
if c.asking > 0 && (len(said) != 1 || said[0].Asking != c.asking || said[0].New == said[0].Old || said[0].Key != r.key || said[0].Why == "") {
t.Errorf("said %+v, want once at asking %d", said, c.asking)
}
if c.dead && (len(said) != 1 || !strings.Contains(said[0].Why, buildrun.LostWords+" after 5 deliveries at 2026-10-26T12:26:40Z")) {
t.Errorf("said %+v, want the ask asked again saying the seat gave it up", said)
}
})
}
t.Run("what the bus says of an ask", func(t *testing.T) { // 1945
@@ -836,12 +974,17 @@ func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
running, _ := json.Marshal(delivery.Step{V: delivery.Version, ID: "mvo53jcm70b5f8", Tool: "delivery ask", Node: "g14", State: delivery.StepRunning, Started: at})
_, reask, _ := NewReAsk("m", fp1, "stuck", "shanks.lead", at)
r := StuckReader{Builds: state(map[string]string{key: string(building), done: string(finished)}), ReAsks: state(map[string]string{key: string(reask)}),
Steps: map[string]StateRead{buildrun.GoBuildModule: state(map[string]string{}), ImageBuildModule: state(map[string]string{"g14.mvo53jcm70b5f8": string(running)})},
Now: func() time.Time { return at.Add(time.Hour) }}
Steps: map[string]StateRead{buildrun.GoBuildModule: state(map[string]string{}), ImageBuildModule: state(map[string]string{"g14.mvo53jcm70b5f8": string(running)})},
Hearing: state(map[string]string{buildrun.HearingKey: `{"heard":false,"streams":["SEAT_NODE_BUILD_GO"],"why":"DEAD_LETTERS is not heard: SEAT_NODE_BUILD_GO: permissions violation","at":"2026-10-26T13:00:00Z"}`}),
Now: func() time.Time { return at.Add(time.Hour) }}
out, err := r.Stuck("")
if err != nil {
t.Fatal(err)
}
// Issue 734, review round 1: a builder that does not hear DEAD_LETTERS says so, with why.
if dl := out["dead_letters"].(map[string]any); dl["heard"] != false || !strings.Contains(dl["why"].(string), "permissions violation") {
t.Errorf("dead_letters = %v, want not heard, saying why", dl)
}
stuck := out["stuck"].([]Stuck)
if len(stuck) != 1 || stuck[0].Key != key || stuck[0].Age != "1h0m0s" || stuck[0].NoneStep != 1 || stuck[0].ReAsk == nil || stuck[0].ReAsk.Why != "stuck" {
t.Fatalf("stuck = %+v, want the building key alone, an hour old, one ask with no step, with its ask again", stuck)
@@ -959,3 +1102,10 @@ func TestATakeOverThatCannotReadTheStreamWaitsForTheAsk(t *testing.T) {
t.Fatalf("the key = %+v after %d asks, want done from the ask's step, none put", v, len(r.asks.Asks()))
}
}
// silentWatch is a watch of an ask's step that stays open and hears nothing.
type silentWatch struct{}
func (silentWatch) WatchAsk(context.Context, string, string) (<-chan delivery.Step, error) {
return make(chan delivery.Step), nil
}
@@ -22,6 +22,8 @@ import (
const (
StuckTool = "builder_stuck"
AskAgainTool = "builder_ask_again"
// BuildAgainTool builds a failed key again by hand (issue 1126 note 72): it writes a reasks value of Mode ModeBuild.
BuildAgainTool = "builder_build_again"
// ReAsksState is the builder's state of asks to ask again: the bucket mesh-delivery-builder_reasks, keyed as the build key.
ReAsksState = "reasks"
ReAsksBucket = delivery.BuilderModule + "_" + ReAsksState
@@ -36,8 +38,24 @@ type ReAsk struct {
At time.Time `json:"at"`
Answer string `json:"answer,omitempty"`
Answered time.Time `json:"answered,omitzero"`
// Mode is ModeBuild for a request to build a failed key again (BuildAgainTool); empty for an ask asked again.
Mode string `json:"mode,omitempty"`
Module string `json:"module,omitempty"`
FP string `json:"fp,omitempty"`
// Refused, Revision and Asks are the builder's answer to a ModeBuild request: why it did nothing, or the key's new
// revision (building) and the ask ids it put by language.
Refused bool `json:"refused,omitempty"`
// Pending says the builder answered before the key was claimed or all its asks were put: neither a refusal nor done.
// Failed is the error of a key written building that failed before its asks were put.
Pending bool `json:"pending,omitempty"`
Failed string `json:"failed,omitempty"`
Revision uint64 `json:"revision,omitempty"`
Asks map[string]string `json:"asks,omitempty"`
}
// ModeBuild is the Mode of a reasks value that asks the key to be built again, not its waiting asks to be asked again.
const ModeBuild = "build"
// DecodeReAsk reads a reasks value.
func DecodeReAsk(data []byte) (ReAsk, error) {
var r ReAsk
@@ -63,6 +81,19 @@ func NewReAsk(module, fp, why, by string, at time.Time) (string, []byte, error)
return key, data, err
}
// NewBuildAgain is the value builder_build_again writes for a module's fingerprint: its key and the value.
func NewBuildAgain(module, fp, why, by string, at time.Time) (string, []byte, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the key is built again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModeBuild, Module: module, FP: fp})
return key, data, err
}
// StateRead is a state the tools read: its keys, and a key's value with when it was written (zero when not known).
type StateRead struct {
Keys func() ([]string, error)
@@ -106,7 +137,9 @@ type StuckReader struct {
ReAsks StateRead
// Steps are the build modules' steps states, by build module.
Steps map[string]StateRead
Now func() time.Time
// Hearing is the builder's state hearing: whether it hears DEAD_LETTERS (issue 734); without it none is said.
Hearing StateRead
Now func() time.Time
// Logf, when set, gets the one line a call logs: the values it read and the time it took.
Logf func(format string, args ...any)
}
@@ -149,6 +182,20 @@ func parallel(n int, read func(i int) error) error {
}
// Stuck is every key building or queued with its asks, oldest first; with module, that module's only.
// deadLetters says whether the builder hears the asks the build seats give up on (issue 734), and why not: deaf, a lane
// whose ask is given up waits for its check bound (buildrun.HearingHealth over the kept hearing).
func (r StuckReader) deadLetters() map[string]any {
if r.Hearing.Get == nil {
return map[string]any{"heard": false, "why": "the builder's state hearing is not readable here"}
}
raw, _, err := r.Hearing.Get(buildrun.HearingKey)
if err != nil {
return map[string]any{"heard": false, "why": "the builder's state hearing cannot be read: " + err.Error()}
}
a := buildrun.HearingHealth(raw)
return map[string]any{"heard": a.Healthy, "why": a.Why}
}
func (r StuckReader) Stuck(module string) (map[string]any, error) {
started := time.Now()
reads := 0
@@ -268,6 +315,7 @@ func (r StuckReader) Stuck(module string) (map[string]any, error) {
})
answer := map[string]any{"stuck": out, "count": len(out),
"delivery_counts": "not readable by the builder: the build seats' workers' information is granted to the delivery view alone (its backlog); an ask given up on is told by the builder at its check bound and asked again"}
answer["dead_letters"] = r.deadLetters()
if cut {
answer["cut"], answer["note"] = true, fmt.Sprintf("read %d of %d build keys in key order and stopped; give a module to read its keys alone", MaxReads, total)
}
@@ -87,13 +87,13 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if m.Module != "mesh-delivery-builder" {
t.Errorf("module = %q", m.Module)
}
if want := []state{{Name: "builds", History: 5}, {Name: "instances", History: 1}, {Name: "queues", History: 1}, {Name: "reasks", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
if want := []state{{Name: "builds", History: 5}, {Name: "hearing", History: 1}, {Name: "instances", History: 1}, {Name: "queues", History: 1}, {Name: "reasks", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
t.Errorf("state = %+v, want %+v", m.State, want)
}
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
// holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5).
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-assigner.placement", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
@@ -116,7 +116,7 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1945: the builder's tools (builder_seat_queue: note 62), served by its tools bundle, and the event it says when it asks an ask again.
if want := []string{"builder_ask_again", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
if want := []string{"builder_ask_again", "builder_build_again", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
if want := []string{"build.asked-again"}; !reflect.DeepEqual(m.Emits, want) {
+3 -1
View File
@@ -4,6 +4,7 @@
"slug": "builder",
"tools": [
"builder_ask_again",
"builder_build_again",
"builder_seat_queue",
"builder_stuck"
],
@@ -20,6 +21,7 @@
],
"state": [
{ "name": "builds", "history": 5 },
{ "name": "hearing", "history": 1 },
{ "name": "instances", "history": 1 },
{ "name": "queues", "history": 1 },
{ "name": "reasks", "history": 1 }
@@ -30,7 +32,7 @@
"mesh-build-image.steps",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-assigner.placement",
"mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
@@ -1,6 +1,6 @@
// Command mesh-delivery-checker-tools serves the checker's read tools (novox/hq ADR 0484 clause 13, issue 1686): the runtime
// launches it for a call, it reads the checker's verdicts and kept states and the indexer's candidates through the checker's
// state and reads, and writes nothing. The checker is the other program: it serves no tool (ADR 0437 §4).
// state and reads. It writes only the tries (the try's tools) and, asking a check again by hand, one verdict. The checker is the other program: it serves no tool (ADR 0437 §4).
package main
import (
@@ -12,6 +12,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-checker/internal/check"
)
@@ -53,10 +54,36 @@ func read(s stdio.KeptState) check.StateRead {
}}
}
// runsOf says which commit the mesh runs of a module from the indexer's modules state, as the checker's own record of it does
// (the commit the module's record was computed at); "" when the record is not readable.
type runsOf struct{ modules check.StateRead }
func (r runsOf) CommitOf(module string) string {
if r.modules.Get == nil {
return ""
}
raw, err := r.modules.Get(module)
if err != nil || len(raw) == 0 {
return ""
}
m, err := delivery.DecodeModule(raw)
if err != nil {
return ""
}
return m.Commit
}
// stateStore is a state the tools write: the tries.
type stateStore struct {
check.StateRead
create func(key string, value []byte) error
entry func(key string) (json.RawMessage, uint64, error)
update func(key string, value []byte, revision uint64) error
}
func (s stateStore) Entry(key string) (json.RawMessage, uint64, error) { return s.entry(key) }
func (s stateStore) Update(key string, value []byte, revision uint64) error {
return s.update(key, value, revision)
}
func (s stateStore) Keys() ([]string, error) { return s.StateRead.Keys() }
@@ -66,19 +93,21 @@ func (s stateStore) Create(key string, value []byte) error { return s.create(k
// Trying is what the try's tools work on: the tries, and the gate's trunk heads.
type Trying struct {
Tries check.TryStore
Ender check.TryEnder
Trunks func() (map[string]string, error)
Now func() time.Time
}
// TryTools are the try's two tools (ADR 0501 §5): ask one, read one. mesh-controller's try calls them.
// TryTools are the try's tools (ADR 0501 §5): ask one, read one, and end one by hand (issue 734). mesh-controller's try calls them.
func TryTools(tr Trying) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return s }
return []stdio.Tool{
{Name: check.TryTool,
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused.",
"where it stands. A try of the same head that waits or runs is the answer to a second ask; a new try is refused while 4 tries run on a build node or 8 wait for one, mesh-wide; tries let in while waiting may later all run at once, and each node's build capacity bounds what runs on it. A repository " +
"the checker checks no level of (one whose trunk the gate does not name) is refused. A try whose ask no build node took within the step " +
"bound ends as error; its ask stays in the build seat's stream and may still run later, judged by nobody (issue 734).",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"head": map[string]any{"type": "string", "description": "the full commit to try, 40 or 64 hex digits"},
@@ -106,7 +135,8 @@ func TryTools(tr Trying) []stdio.Tool {
}},
{Name: check.TryReadTool,
Description: "Where a try stands, as stored: queued, running, or its end (pass, fail, or error when it could not run), with what was said of it, " +
"the head and the trunk commit it was merged onto, the time it took and what failed first. Read-only.",
"the head and the trunk commit it was merged onto, the time it took and what failed first; while it runs, each ask it put as soon as it is put, " +
"with the node that took it, its state and the phase its step says (fetch and merge, a level's digest, lint, tests) and since when. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"id"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"}}},
Run: func(args map[string]any) (any, error) {
@@ -115,9 +145,46 @@ func TryTools(tr Trying) []stdio.Tool {
}
return check.ReadTry(tr.Tries, str(args, "id"))
}},
{Name: check.TryEndTool,
Description: "End a try that waits or runs, as error, by hand: say why (kept in what it says). Its place among the 4 is free at once, " +
"the checker stops waiting for its asks, and the next " + check.TryTool + " of its head asks anew with a new id. For a try whose ask " +
"was lost (no step runs for it in checker_status); the checker ends one by itself when the build seat gives its ask up, or no node " +
"takes it within the step bound. A try that ended already is refused.",
Input: map[string]any{"type": "object", "required": []string{"id", "why"}, "properties": map[string]any{
"id": map[string]any{"type": "string", "description": "the try's id, try-<digits>"},
"why": map[string]any{"type": "string", "description": "why it is ended, in words"}}},
Run: func(args map[string]any) (any, error) {
if tr.Ender == nil {
return nil, fmt.Errorf("the tries state cannot be written here")
}
now := time.Now
if tr.Now != nil {
now = tr.Now
}
t, err := check.EndTry(tr.Ender, str(args, "id"), str(args, "why"), now())
if err != nil {
return nil, err
}
return map[string]any{"id": t.ID, "state": t.State, "said": t.Said}, nil
}},
}
}
// Seat is the seat the checker holds (novox/hq ADR 0547 §10, issue 774); its verbs are served as tools named <seat>.<verb>.
const Seat = "mesh-delivery-checker"
// SeatTryTools are the try's tools as the seat's verbs: try, try-read and try-end, run by the same code as checker_try,
// checker_try_read and checker_try_end, so the answer is the same. The module tools stay beside them.
func SeatTryTools(tr Trying) []stdio.Tool {
verbs := map[string]string{check.TryTool: "try", check.TryReadTool: "try-read", check.TryEndTool: "try-end"}
var out []stdio.Tool
for _, t := range TryTools(tr) {
t.Name = Seat + "." + verbs[t.Name]
out = append(out, t)
}
return out
}
// Tools are the read tools over the checker's states.
func Tools(r check.Reader) []stdio.Tool {
return []stdio.Tool{
@@ -174,12 +241,33 @@ func Tools(r check.Reader) []stdio.Tool {
"and the candidates whose check is owed by kind (waiting_by_kind: new, nodes, and trunk, a re-judge pulled after the other two). A key a " +
"build module left running (started past the step bound, or its build module not running on its node) is listed apart under " +
"left_running with why, and not counted. Nothing holds the checker and it has no capacity of its own: a check is an ask on the " +
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). Read-only.",
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). held_by_turn names the candidates whose check an approved " +
"merge's turn holds off (novox/hq ADR 0548 §3), with the repository and the merge that waits; the gate's turns tool reads every turn and its " +
"hand-now tool steers one. dead_letters says whether the checker hears the asks the build seats give up on (issue 734), and why not: " +
"deaf, a try or check whose ask is given up waits until its bound. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return r.Status() }},
}
}
// RecoveryTools are the tools that steer a check by hand (tools-are-how-a-module-is-spoken-to): ask a check again whose
// wait judged nothing (the lost check of 2026-10-28). They write the checker's verdicts through w.
func RecoveryTools(r check.Reader, w check.VerdictEntry) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return s }
return []stdio.Tool{{Name: check.AskAgainTool,
Description: "Ask a candidate's check again by hand, whose verdict stays running while nothing runs for it (" + check.StatusReadTool + " names it " +
"under lost): its verdict is ended as asked again, saying why, and the checker, which hears its verdicts, stops the wait that judged nothing " +
"and asks the check anew, whole. Refused for a group with no candidate, a verdict that is not running (a pass or failure stands until a push or " +
"a trunk move), and while a step of its pull requests can still run on a build module (its end judges it). The checker reads a step that " +
"ended while its watch was deaf by itself at the next step end of its build module; this is for what that does not reach.",
Input: map[string]any{"type": "object", "required": []string{"group", "why"}, "properties": map[string]any{
"group": map[string]any{"type": "string", "description": "the delivery group's branch name"},
"why": map[string]any{"type": "string", "description": "why it is asked again, in words; kept in the verdict it ends"}}},
Run: func(args map[string]any) (any, error) {
return r.AskAgain(w, str(args, "group"), str(args, "why"))
}}}
}
// stepBound is the checker's step bound as the checker reads it (MESH_CHECKER_STEP_BOUND), its default when unset or not a
// duration.
func stepBound() time.Duration {
@@ -193,7 +281,10 @@ func main() {
r := check.Reader{VerdictState: read(stdio.State("verdicts")), KeptState: read(stdio.State("kept")), CandidateState: read(stdio.State("mesh-delivery-indexer.candidates")),
PreviewState: read(stdio.State("previews")),
StepStates: map[string]check.StateRead{"mesh-build-go": read(stdio.State("mesh-build-go.steps")), "mesh-build-node": read(stdio.State("mesh-build-node.steps"))},
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound()}
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound(),
TurnState: read(stdio.State("mesh-delivery-gate.turns")), HearingState: read(stdio.State(buildrun.HearingState)),
AskState: read(stdio.State(check.AsksState))}
r.Runs = runsOf{read(stdio.State("mesh-delivery-indexer.modules"))} // the node-engine at the commit the mesh runs, as the checker reads it
tries := stdio.State(check.TriesState)
source := stdio.State("mesh-delivery-gate.source")
trunks := func() (map[string]string, error) {
@@ -205,14 +296,35 @@ func main() {
return s.Heads, err
}
r.Trunks = trunks
trying := Trying{
Tries: stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}},
Trunks: trunks,
}
if err := stdio.Serve("", logged(append(Tools(r), TryTools(trying)...))); err != nil {
store := stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}, entry: func(key string) (json.RawMessage, uint64, error) {
reads.Add(1)
e, err := tries.Get(key)
if err != nil || e == nil {
return nil, 0, err
}
return e.Value, e.Revision, nil
}, update: func(key string, value []byte, revision uint64) error {
_, err := tries.Update(key, json.RawMessage(value), revision)
return err
}}
trying := Trying{Tries: store, Ender: store, Trunks: trunks}
verdicts := stdio.State("verdicts")
written := stateStore{entry: func(key string) (json.RawMessage, uint64, error) {
reads.Add(1)
e, err := verdicts.Get(key)
if err != nil || e == nil {
return nil, 0, err
}
return e.Value, e.Revision, nil
}, update: func(key string, value []byte, revision uint64) error {
_, err := verdicts.Update(key, json.RawMessage(value), revision)
return err
}}
tools := append(append(append(Tools(r), TryTools(trying)...), SeatTryTools(trying)...), RecoveryTools(r, written)...)
if err := stdio.Serve("", logged(tools)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1)
}
@@ -23,6 +23,14 @@ func TestTheToolsServedAreTheFiveReadTools(t *testing.T) {
if !slices.Equal(names, want) {
t.Errorf("tools = %v, want %v", names, want)
}
// The recovery tool beside them asks a lost check again; with nothing to write it refuses.
recovery := RecoveryTools(check.Reader{}, nil)
if len(recovery) != 1 || recovery[0].Name != check.AskAgainTool || recovery[0].Description == "" {
t.Errorf("recovery tools = %+v", recovery)
}
if _, err := recovery[0].Run(map[string]any{"group": "feat/x", "why": "lost"}); err == nil {
t.Errorf("%s answered with no state to write", check.AskAgainTool)
}
// With nothing to read from a tool says so and changes nothing.
for _, tl := range Tools(check.Reader{}) {
if tl.Name == check.StatusReadTool {
@@ -53,13 +61,19 @@ func (m *memStore) Create(key string, value []byte) error {
return nil
}
func (m *memStore) Entry(key string) (json.RawMessage, uint64, error) { return m.vals[key], 1, nil }
func (m *memStore) Update(key string, value []byte, revision uint64) error {
m.vals[key] = value
return nil
}
func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed interface with mesh-controller's try
store := &memStore{vals: map[string]json.RawMessage{}}
tools := TryTools(Trying{Tries: store, Now: func() time.Time { return time.Unix(1, 5) },
tools := TryTools(Trying{Tries: store, Ender: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(tools) != 2 || tools[0].Name != "checker_try" || tools[1].Name != "checker_try_read" {
if len(tools) != 3 || tools[0].Name != "checker_try" || tools[1].Name != "checker_try_read" || tools[2].Name != "checker_try_end" {
t.Fatalf("tools = %+v", tools)
}
out, err := tools[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
@@ -74,6 +88,32 @@ func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed i
if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) {
t.Errorf("read = %s, %v", raw, err)
}
// The seat's verbs answer as the tools do (ADR 0547 §10, issue 774): the same ask is the same try, and the read is alike.
verbs := SeatTryTools(Trying{Tries: store, Ender: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(verbs) != 3 || verbs[0].Name != Seat+".try" || verbs[1].Name != Seat+".try-read" || verbs[2].Name != Seat+".try-end" {
t.Fatalf("verbs = %+v", verbs)
}
again, err := verbs[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
if err != nil || again.(map[string]any)["id"] != id {
t.Errorf("verb try = %v, %v, want the waiting try %s", again, err, id)
}
if viaVerb, err := verbs[1].Run(map[string]any{"id": id}); err != nil || string(viaVerb.(json.RawMessage)) != string(raw.(json.RawMessage)) {
t.Errorf("verb try-read = %s, %v; tool = %s", viaVerb, err, raw)
}
if _, err := verbs[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil {
t.Error("the verb tried a repository nothing checks")
}
// Issue 734: a try whose ask was lost is ended by hand with why, and its head is asked anew.
ended, err := tools[2].Run(map[string]any{"id": id, "why": "its ask was lost"})
if err != nil || ended.(map[string]any)["state"] != "error" {
t.Errorf("end = %v, %v", ended, err)
}
if _, err := tools[2].Run(map[string]any{"id": id, "why": "again"}); err == nil {
t.Error("an ended try was ended again")
}
if _, err := tools[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil ||
err.Error() != "no level of novox/hq is checked by the checker" {
t.Errorf("hq: %v", err)
@@ -9,6 +9,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-checker/internal/check"
)
@@ -130,3 +131,55 @@ func (m manifestReader) Manifest(ctx context.Context, digest string) (catalogue.
func parseManifest(raw string) (catalogue.Manifest, error) {
return catalogue.ParseManifestForInputs([]byte(raw))
}
// awaitedKV is a bucket that may appear after the start (process.AwaitBucket) as a check.KV: a call while it is not on the
// bus fails saying so.
type awaitedKV struct{ b *process.AwaitedBucket }
func (a awaitedKV) kv() (kvStore, error) {
kv := a.b.KV()
if kv == nil {
return kvStore{}, errors.New("the bucket is not on the bus yet")
}
return kvStore{kv}, nil
}
func (a awaitedKV) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
s, err := a.kv()
if err != nil {
return nil, 0, false, err
}
return s.Get(ctx, key)
}
func (a awaitedKV) Create(ctx context.Context, key string, value []byte) (uint64, error) {
s, err := a.kv()
if err != nil {
return 0, err
}
return s.Create(ctx, key, value)
}
func (a awaitedKV) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
s, err := a.kv()
if err != nil {
return 0, err
}
return s.Update(ctx, key, value, revision)
}
func (a awaitedKV) Keys(ctx context.Context) ([]string, error) {
s, err := a.kv()
if err != nil {
return nil, err
}
return s.Keys(ctx)
}
func (a awaitedKV) Delete(ctx context.Context, key string) error {
s, err := a.kv()
if err != nil {
return err
}
return s.Delete(ctx, key)
}
@@ -24,6 +24,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-checker/internal/check"
)
@@ -79,12 +81,16 @@ type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
conflict atomic.Bool
// hold is a commit whose fetch never answers: its ask stays taken, as one lost in a bus reload (issue 734).
hold atomic.Value
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
testTool *testing.T
// beside is, by workspace, the siblings the fetch placed in it as `<owner/repo> <commit>`, sorted: a digest echoes them
// (1929/AC2; mesh-sdk buildstep.besideOf), or the level runs whole and keeps nothing.
beside map[string][]string
// names are the nodes' names the last check ask carried, for the names check (ADR 0567 §12).
names []string
}
func (f *fakeBuildModules) count(tool string) int {
@@ -98,6 +104,9 @@ func (f *fakeBuildModules) result(t *testing.T, tool string, args map[string]any
var v any
switch {
case strings.HasSuffix(tool, "_fetch"):
if h, _ := f.hold.Load().(string); h != "" && args["commit"] == h {
<-f.ctx.Done()
}
name := "go-fetch-clean"
if f.conflict.Load() {
name = "go-fetch-conflict"
@@ -184,7 +193,14 @@ func (f *fakeBuildModules) serve(t *testing.T, module string, kept jetstream.Key
}
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: "n1", Put: put,
Capacity: func() int { return 2 }, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) {
if a.Check != nil {
f.mu.Lock()
f.names = a.Check.NodeNames
f.mu.Unlock()
}
return buildstep.RunAsk(ctx, steps, a)
}}
go func() { _ = p.Pull(f.ctx) }()
}
@@ -298,6 +314,10 @@ func startBed(t *testing.T) *bed {
}
b.tools[module] = kv
}
// What the controller keeps of the messages a consumer gave up on (issue 734).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buildrun.DeadLettersStream, Subjects: []string{"mesh.events.dead.>"}}); err != nil {
t.Fatal(err)
}
// The checker's stream of events, which the indexer and the forge bridge hear.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "CHECKS", Subjects: []string{"mesh.stream.mesh-delivery-checker.event.>"}}); err != nil {
t.Fatal(err)
@@ -305,7 +325,8 @@ func startBed(t *testing.T) *bed {
for name, history := range map[string]int{delivery.CheckerVerdictsBucket: delivery.CheckerVerdictsHistory, delivery.CheckerKeptBucket: delivery.CheckerKeptHistory,
delivery.CheckerOwedBucket: delivery.CheckerOwedHistory, delivery.PreviewsBucket: delivery.PreviewsHistory, delivery.CandidatesBucket: delivery.CandidatesHistory,
delivery.ModulesBucket: delivery.ModulesHistory, delivery.ManifestsBucket: delivery.ManifestsHistory, delivery.SourceBucket: delivery.SourceHistory,
delivery.PresenceBucket: delivery.PresenceHistory, delivery.HaveBucket: delivery.HaveHistory, check.TriesBucket: 1} {
delivery.PresenceBucket: delivery.PresenceHistory, delivery.HaveBucket: delivery.HaveHistory, check.TriesBucket: 1,
delivery.TurnsBucket: delivery.TurnsHistory} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: uint8(history)})
if err != nil {
t.Fatal(err)
@@ -322,6 +343,7 @@ func startBed(t *testing.T) *bed {
t.Fatal(err)
}
b.setOnline(t, "n1", true)
b.setNode(t, "n1")
for _, module := range []string{"mesh-build-go", "mesh-build-node"} {
b.setHealth(t, "n1", module)
}
@@ -354,6 +376,20 @@ func (b *bed) setOnline(t *testing.T, node string, online bool) {
}
}
// setNode writes the node's record in the node-engine's have, as the node-engine does, so the checker names it in each
// check ask (ADR 0567 §12); with no record the checker asks no check.
func (b *bed) setNode(t *testing.T, node string) {
t.Helper()
data, err := delivery.EncodeNodeRecord(delivery.NodeRecord{V: 1, Node: node})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.HaveNodeKey(node)
if _, err := b.buckets[delivery.HaveBucket].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
}
func (b *bed) setHealth(t *testing.T, node, module string) {
t.Helper()
data, err := delivery.EncodeHave(delivery.Have{Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
@@ -412,11 +448,12 @@ func pull(repository string, number int, head string) delivery.CandidatePull {
func (b *bed) start(t *testing.T, env map[string]string) {
t.Helper()
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(credential{URL: b.srvURL, Node: "n1", Module: "mesh-delivery-checker", User: "n1.mesh-delivery-checker", Password: "pw"})
cred, _ := json.Marshal(process.Account{URL: b.srvURL, Node: "n1", Module: "mesh-delivery-checker", User: "n1.mesh-delivery-checker", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
e := map[string]string{"MESH_BROKER_FILE": broker}
// The settle window of a group's heads (check.DefaultSettle) is cut short: each bed's candidate is written whole.
e := map[string]string{"MESH_BROKER_FILE": broker, "MESH_CHECKER_SETTLE": "10ms"}
for k, v := range env {
e[k] = v
}
@@ -480,6 +517,12 @@ func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300
if v.Composition == nil || v.Composition.State != delivery.CompositionNotRun || len(v.Machines) != 1 || v.Machines[0] != "n1" {
t.Errorf("composition %+v machines %v", v.Composition, v.Machines)
}
b.fake.mu.Lock()
names := b.fake.names
b.fake.mu.Unlock()
if len(names) != 1 || names[0] != "n1" {
t.Errorf("the check ask named the nodes %v, want [n1] from the have's node record", names)
}
o := b.owed(t, 7, head1, "success")
if o.Context != delivery.StatusContextCheck || !strings.HasPrefix(o.Text, "pass 111111111111 on aaaaaaaaaaaa") || !strings.Contains(o.Text, "unit 2 run") {
t.Errorf("owed = %+v", o)
@@ -507,9 +550,12 @@ func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300
}
}
// TestACheckOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every package
// taken from the first's kept verdicts: no test step runs and no second kept key appears.
func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2
// TestACheckOrACarryOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every
// package taken from the first's kept verdicts: no test step runs and no second kept key appears. Then, through the real bus
// (novox/hq ADR 0548 §3, §5): a third candidate is not claimed while the gate's merge turn on its repository stands, and is
// claimed when the turn's key is deleted; and a move of the trunk carries the first verdict on the recheck stream by its
// unchanged digests, running no test, the verdict naming the trunk it was carried from.
func TestACheckOrACarryOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2, ADR 0548 §3, §5
b := startBed(t)
first := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
@@ -526,6 +572,42 @@ func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1
if got := keptCount(t, b); got != kept {
t.Errorf("%d kept keys, was %d", got, kept)
}
// §3: the gate's turn on the repository holds a new check off; its delete claims it.
at := time.Now().UTC()
turn, err := delivery.EncodeTurn(delivery.Turn{Repository: repo, Waiting: []delivery.TurnWait{{Group: "fix/other", Heads: "0123456789ab", Approved: at, Since: at}}})
if err != nil {
t.Fatal(err)
}
tk, _ := delivery.TurnsKey(repo)
if _, err := b.buckets[delivery.TurnsBucket].Put(b.ctx, tk, turn); err != nil {
t.Fatal(err)
}
time.Sleep(300 * time.Millisecond) // the checker hears the turn
third := b.candidate(t, "feat/z", pull(repo, 9, "3333333333333333333333333333333333333333"))
time.Sleep(time.Second)
if _, err := b.buckets[delivery.CheckerVerdictsBucket].Get(b.ctx, third); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("verdicts.%s exists (%v) while a merge turn holds its repository", third, err)
}
if err := b.buckets[delivery.TurnsBucket].Delete(b.ctx, tk); err != nil {
t.Fatal(err)
}
b.verdict(t, third, ended)
// §5: the trunk moves; the first verdict is carried by its digests, nothing runs.
tests = b.fake.count("mesh-build-go.build_go_test")
const moved = "abababababababababababababababababababab"
b.setTrunk(t, map[string]string{repo: moved})
c := b.verdict(t, first, func(v delivery.Verdict) bool { return ended(v) && v.Trunk[repo] == moved })
if c.State != delivery.VerdictPass || c.CarriedFrom[repo] != trunk || c.Levels[delivery.LevelUnit].Run != 0 || c.Levels[delivery.LevelIntegration].Run != 0 {
t.Fatalf("the carried verdict = %+v, want a pass carried from %s with no test run", c, trunk[:12])
}
if got := b.fake.count("mesh-build-go.build_go_test"); got != tests {
t.Errorf("build_go_test called %d times on the carry, was %d: a carry of unchanged digests runs nothing", got, tests)
}
if o := b.owed(t, 7, head1, "success"); !strings.HasPrefix(o.Text, "pass 111111111111 on abababababab | carried from aaaaaaaaaaaa") {
t.Errorf("the carried status says %q", o.Text)
}
}
func keptCount(t *testing.T, b *bed) int {
@@ -676,6 +758,49 @@ func TestATryIsJudgedAndItsEndWrittenWithNoStatusOwed(t *testing.T) {
t.Errorf("a try wrote %s: %v", name, keys)
}
}
// Issue 734: a try whose ask the build seat gives up on (the controller keeps its copy in DEAD_LETTERS with the ask's
// message id) ends as error saying so, and its head asked again is a new try.
b.fake.hold.Store(head2)
lost, err := check.AskTry(store, map[string]string{repo: trunk}, check.TryAsk{Repository: repo, Head: head2}, time.Now())
if err != nil {
t.Fatal(err)
}
steps, err := b.js.KeyValue(b.ctx, delivery.StepsBucket("mesh-build-go"))
if err != nil {
t.Fatal(err)
}
askID := ""
for deadline := time.Now().Add(20 * time.Second); askID == "" && time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
keys, _ := steps.Keys(b.ctx)
for _, k := range keys {
if e, err := steps.Get(b.ctx, k); err == nil && strings.Contains(string(e.Value()), `"running"`) {
askID = strings.TrimPrefix(k, "n1.")
}
}
}
if askID == "" {
t.Fatal("the held try's ask was never taken")
}
dead := &nats.Msg{Subject: "mesh.events.dead.SEAT_NODE_BUILD_GO.SEAT_NODE_BUILD_GO_CHECK_worker", Header: nats.Header{}}
dead.Header.Set(buildrun.DeadHeaderMsgID, askID)
dead.Header.Set(buildrun.DeadHeaderDeliveries, "5")
dead.Header.Set(buildrun.DeadHeaderStream, "SEAT_NODE_BUILD_GO")
dead.Header.Set(buildrun.DeadHeaderGaveUp, time.Now().UTC().Format(time.RFC3339Nano))
if _, err := b.js.PublishMsg(b.ctx, dead); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
raw, _ := store.Get(lost)
if last, err = check.DecodeTry(raw); err == nil && last.Over() {
break
}
}
if last.State != check.TryError || !strings.Contains(last.Said, buildrun.LostWords+" after 5 deliveries") {
t.Fatalf("the try whose ask was given up = %+v, want error saying so", last)
}
if again, err := check.AskTry(store, map[string]string{repo: trunk}, check.TryAsk{Repository: repo, Head: head2}, time.Now()); err != nil || again == lost {
t.Errorf("its head asked again answered %q, %v, want a new try", again, err)
}
}
// TestTheCheckerSaysOwedReadableOnTheBus: started, the checker says on its own states subject what it keeps, owed among
@@ -702,7 +827,8 @@ func TestTheCheckerSaysOwedReadableOnTheBus(t *testing.T) { // R2-1
break
}
if said.Module != "mesh-delivery-checker" || said.Node != "n1" || !said.Keeps("owed") || !said.Readable("owed") ||
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || said.Readable("tries") || len(said.States) != 5 {
t.Fatalf("the checker said %+v, want its five states with kept, owed, previews and verdicts readable (1929/AC4; 1887 A5b)", said)
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || !said.Readable("tries") || said.Readable("hearing") ||
!said.Keeps("asks") || said.Readable("asks") || len(said.States) != 7 {
t.Fatalf("the checker said %+v, want its seven states, asks and hearing its own, with kept, owed, previews, tries and verdicts readable, tries by the delivery view (1929/AC4; 1887 A5b; build_usage)", said)
}
}
@@ -2,12 +2,16 @@ package main
import (
"context"
"fmt"
"sort"
"strings"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-checker/internal/check"
)
@@ -25,11 +29,46 @@ type judgeDeps struct {
builds check.BuildStates
// compositions is the assigner's bucket of compositions; nil leaves every composition not run.
compositions check.CompositionReader
logf func(format string, args ...any)
// nodes names every node with a record; nil hands no names.
nodes *nodeNames
logf func(format string, args ...any)
}
// nodeNames reads every node's name from the node-engine's have, one key `<node>.node` per node with a record (novox/hq
// ADR 0567 §12), at each ask: a node enrolled since is named in the next check.
type nodeNames struct{ kv jetstream.KeyValue }
// openNodeNames opens the node-engine's have for the nodes' names alone: it is not watched, so no check waits on a node.
func openNodeNames(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) *nodeNames {
return &nodeNames{kv: process.OptionalBucket(ctx, js, delivery.HaveBucket, "no check is asked until it is: the names check needs the nodes' names", logf)}
}
func (n nodeNames) Names(ctx context.Context) ([]string, error) {
if n.kv == nil {
return nil, fmt.Errorf("the bucket %s could not be opened at the checker's start", delivery.HaveBucket)
}
lister, err := n.kv.ListKeysFiltered(ctx, "*."+delivery.HaveNodeWord)
if err != nil {
return nil, err
}
var out []string
for key := range lister.Keys() {
if node, _, isNode, err := delivery.ParseHaveKey(key); err == nil && isNode {
out = append(out, node)
}
}
sort.Strings(out)
return out, nil
}
func newJudge(d judgeDeps) check.Judge {
return &check.Examiner{Compositions: d.compositions, Asks: d.asks, Toolchains: d.tools,
var nodes interface {
Names(ctx context.Context) ([]string, error)
}
if d.nodes != nil {
nodes = d.nodes
}
return &check.Examiner{Nodes: nodes, Compositions: d.compositions, Asks: d.asks, Toolchains: d.tools,
Kept: check.Kept{Store: d.kept}, Previews: d.previews, Modules: d.repos, Packages: d.repos, Forge: d.forge, Runs: d.repos, Builds: d.builds, Announce: d.announce, Trunks: d.trunks, Logf: d.logf}
}
@@ -37,8 +37,8 @@ func TestTheCheckerAnnouncesEveryStateItKeepsAndOwedReadable(t *testing.T) { //
if !equal(manifest, announced) {
t.Errorf("announced %v, the manifest keeps %v", announced, manifest)
}
// kept is read by the Go build module for a check ask (issue 1887 design limit 1).
if want := []string{"kept", "owed", "previews", "verdicts"}; !equal(readable, want) {
// kept is read by the Go build module for a check ask (issue 1887 design limit 1); tries by the delivery view (build_usage).
if want := []string{"kept", "owed", "previews", "tries", "verdicts"}; !equal(readable, want) {
t.Errorf("readable %v, want %v", readable, want)
}
}
@@ -0,0 +1,50 @@
package main
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// lazyTurns is the gate's merge turns as the checker reads them (novox/hq ADR 0548 §3), opened when the gate's bucket is
// there: before each claim it is opened again until it is, so a checker that started before the gate made the bucket holds
// checks from the gate's first turn on, with no timer and no restart. Until then it holds nothing, and says so once, loudly,
// and once again when it opens.
type lazyTurns struct {
turns *check.Turns
open func() error
logf func(format string, args ...any)
mu sync.Mutex
opened bool
said bool
}
// ensure opens the bucket once it is there; a failure is said once.
func (l *lazyTurns) ensure() {
l.mu.Lock()
defer l.mu.Unlock()
if l.opened || l.open == nil {
return
}
if err := l.open(); err != nil {
if !l.said {
l.said = true
l.logf("NO MERGE TURN HOLDS A CHECK: the gate's bucket %s cannot be read (%v); every check is claimed as if no approved merge waited, "+
"and the bucket is opened again before each claim until it is there", delivery.TurnsBucket, err)
}
return
}
l.opened = true
if l.said {
l.logf("the gate's merge turns are read from %s from now on: an approved merge's turn holds checks off its repositories again", delivery.TurnsBucket)
}
}
// Holding opens the bucket when it is not yet, then answers what the turns hold.
func (l *lazyTurns) Holding(repositories []string, heads string) (string, delivery.TurnWait, bool) {
l.ensure()
return l.turns.Holding(repositories, heads)
}
@@ -7,12 +7,7 @@ package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
@@ -36,7 +31,7 @@ import (
// eventSubject is the subject of an event of the checker's stream checks (the bus names the stream CHECKS): the indexer's
// watch of the previews bucket and the forge bridge's watch of the owed bucket are woken by them.
func eventSubject(event string) string {
return "mesh.stream." + delivery.CheckerModule + ".event." + event
return process.ModuleStreamSubject(delivery.CheckerModule, event)
}
const (
@@ -53,50 +48,6 @@ func main() {
}
}
// credential is the broker file the controller issues to the module: JSON of the bus address, the certificate
// fingerprint to pin, the principal and its password (as the indexer reads it; issue 1242 moves this into the SDK).
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
func readCredential(path string) (credential, error) {
raw, err := os.ReadFile(path)
if err != nil {
return credential{}, err
}
var c credential
if err := json.Unmarshal(raw, &c); err != nil {
return credential{}, fmt.Errorf("the broker credential is not one: %w", err)
}
if c.URL == "" || c.User == "" {
return credential{}, errors.New("the broker credential names no bus address or user")
}
return c, nil
}
// pinned accepts exactly the certificate with this SHA-256 and no other.
func pinned(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
if got := "sha256:" + hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%.23s…)", got)
}
return nil
},
}
}
// durationEnv reads a duration setting from the environment, def when it is not set.
func durationEnv(getenv func(string) string, name string, def time.Duration) (time.Duration, error) {
v := strings.TrimSpace(getenv(name))
@@ -112,46 +63,33 @@ func durationEnv(getenv func(string) string, name string, def time.Duration) (ti
// run is the program: it returns when ctx ends or the bus cannot be used.
// keptStates is every state the checker keeps and which others may read (ADR 0495 §2-§3): owed by the forge module,
// which writes mesh/check from it (ADR 0521), and by the forge bridge; previews by the indexer; verdicts by the gate.
// which writes mesh/check from it (ADR 0521), and by the forge bridge; previews by the indexer; verdicts by the gate; tries by the delivery view.
var keptStates = []announce.State{
// The kept verdicts are read by the Go build module, which takes them for a check ask's unchanged packages (issue 1887's
// design limit 1); the checker alone writes them (ADR 0444 clause 5).
// Whether DEAD_LETTERS is heard (issue 734): read by the checker's own status tool alone.
{Name: "hearing"},
// The asks each running check put (check.AsksState): read by the checker's own checker_ask_again alone.
{Name: "asks"},
{Name: "kept", Readable: true},
{Name: "owed", Readable: true},
{Name: "previews", Readable: true},
{Name: "tries"},
// The tries are read by the delivery view, which tells a try's asks from a candidate's by the asks each names (build_usage).
{Name: "tries", Readable: true},
{Name: "verdicts", Readable: true},
}
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-checker: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to hear the bus with")
}
cred, err := readCredential(path)
cred, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
nats.CustomInboxPrefix("_INBOX." + cred.User),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
conn, js, err := cred.Dial()
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
return err
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
// Its own states, the candidates it checks, and the modules and manifests that say which build modules judge a repository:
// a check without one of them would be wrong or unwritten, so each stops the start, naming its grant (issue 1945).
buckets := map[string]jetstream.KeyValue{}
@@ -163,11 +101,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
// Calls to the build modules go out as this module's own account, naming the machine (mesh-sdk process.CallOn); the
// buckets are read and written on the connection above.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
caller, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
caller, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
@@ -186,6 +120,10 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
settleWindow, err := durationEnv(getenv, "MESH_CHECKER_SETTLE", check.DefaultSettle)
if err != nil {
return err
}
link := strings.TrimSpace(getenv("MESH_CHECKER_LINK"))
if link == "" {
link = defaultLink
@@ -193,16 +131,37 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
tools := &buildrun.ToolchainSet{}
presence := &buildrun.PresenceSet{}
trunks := &check.Trunks{}
turns := &check.Turns{}
lazy := &lazyTurns{turns: turns, logf: logger.Printf}
// A check is one ask per pull request and build module on the build seats' check stream, pulled by a build node with a
// free slot (novox/hq ADR 0525 §1, §3): the checker chooses no node and starts no step.
asks := buildrun.JetStreamAsks{JS: js}
// The asks the build seats gave up on (issue 734): heard from DEAD_LETTERS, each ends the step its asker waits for.
// Heard in the background, and again at a reconnect or an ask while it is not; whether it is heard is kept in the
// state hearing, which checker_status says (dead_letters).
dead := &buildrun.DeadLetters{}
// The state may not be on the bus yet when a delivery started the checker (the grants part creates it, issue 1790): it
// is bound when the bus says it was created, and the latest hearing is written then.
hearingKV := process.AwaitBucket(ctx, js, conn, check.HearingBucket, "checker_status cannot say whether DEAD_LETTERS is heard", logger.Printf)
hearing := buildrun.NewDeadLetterHearing(ctx, buildrun.JetStreamDeadLetters{JS: js}, []string{delivery.AskGo, delivery.AskNode}, dead,
logger.Printf, buildrun.SayHearing(ctx, hearingKV, logger.Printf))
defer hearing.Stop()
// Each ask's step is heard by its own watch and, at every step end of its build module and every reconnect, read
// (buildrun.HearEnds): a watch that goes deaf loses no end (the lost check of 2026-10-28).
heard := buildrun.HearEnds(ctx, asks, asks, []string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, logger.Printf)
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
waiter := buildrun.LostAskWaiter{Inner: heard, Dead: dead, Hearing: hearing}
asksKV := process.AwaitBucket(ctx, js, conn, check.AsksBucket, "checker_ask_again refuses every check, as it cannot tell whether its ask waits in a stream", logger.Printf)
ch := &check.Checker{
Asks: awaitedKV{asksKV},
Verdicts: kvStore{buckets[delivery.CheckerVerdictsBucket]},
Owed: kvStore{buckets[delivery.CheckerOwedBucket]},
Trunks: trunks,
Turns: lazy,
Bound: checkBound,
Link: link,
Logf: logger.Printf,
Settle: settleWindow,
}
// The builder's builds say how the candidates' fingerprints stand (issue 1684); where its bucket is not there yet the line
// says nothing of the builds, until the checker starts again.
@@ -228,7 +187,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return cw, nil
}}
// The tries (ADR 0501 §5) are judged by the same judge, held by the same deliveries.
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, Logf: logger.Printf}
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, NotTaken: stepBound, Logf: logger.Printf}
announce := func(ctx context.Context, event, key string) {
body, _ := json.Marshal(map[string]string{"key": key})
if _, err := js.Publish(ctx, eventSubject(event), body); err != nil {
@@ -236,22 +195,24 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
}
repos := &repoModules{manifests: kvStore{buckets[delivery.ManifestsBucket]}, logf: logger.Printf}
ch.Runs = repos // the node-engine a check reads beside at its running commit (ADR 0548 §6)
ch.Judge = newJudge(judgeDeps{
asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: stepBound}, tools: tools, trunks: trunks,
asks: buildrun.Asker{Publisher: asks, Waiter: waiter, Bound: stepBound}, tools: tools, trunks: trunks,
kept: kvStore{buckets[delivery.CheckerKeptBucket]}, previews: kvStore{buckets[delivery.PreviewsBucket]},
repos: repos, builds: builds, forge: forgeReads{call: caller.Call}, logf: logger.Printf,
announce: announce, compositions: compositions,
nodes: openNodeNames(ctx, js, logger.Printf),
})
ch.Announce = announce
tryer.Judge = ch.Judge
// The toolchains first, so a check that starts from the first candidate finds the nodes.
watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, ch.NodesChanged, logger)
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, ch.NodesChanged, logger.Printf)
if err != nil {
return err
}
@@ -263,13 +224,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := js.Consumer(ctx, stream, cred.Node+"_"+cred.Module)
cons, err := cred.EventConsumer(ctx, js, stream)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the checker starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := cons.Consume(func(m jetstream.Msg) { watch.Reported(m) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { watch.Reported(m) }, logger.Printf)
if err != nil {
return err
}
@@ -313,13 +273,42 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}()
}
}
// The gate's merge turns (novox/hq ADR 0548 §3): no check is claimed of a repository an approved merge waits on, and the
// delete of its turn claims it. Read before the candidates, so no check starts past a turn written before this start.
// Where the gate's bucket is not there yet it is read as empty, and opened again before each claim until it is there
// (lazyTurns), so a gate delivered after the checker started holds checks from its first turn on.
lazy.open = func() error {
kv, err := js.KeyValue(ctx, delivery.TurnsBucket)
if err != nil {
return err
}
tw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { turns.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, tw, func(e jetstream.KeyValueEntry) {
if turns.Put(e.Key(), e.Value(), !opOf(e)) {
ch.Reconsider(ctx) // a turn went or changed: a check it held is claimed now
}
})
}()
return nil
}
lazy.ensure()
// The modules say which repository holds a TypeScript module, so the Node build module judges it too.
mw, err := settle(ctx, buckets[delivery.ModulesBucket], func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, mw, func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
errs <- follow(ctx, mw, func(e jetstream.KeyValueEntry) {
before := repos.CommitOf(check.NodeEngineModule)
repos.Put(ctx, e.Key(), e.Value(), !opOf(e))
if after := repos.CommitOf(check.NodeEngineModule); after != before {
ch.TrunkMoved(ctx) // the node-engine the mesh runs moved: a verdict that read it beside is carried (ADR 0548 §6)
}
})
}()
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
@@ -349,6 +338,21 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
})
}()
// The checker's own verdicts, updates only, in one watch: one checker_ask_again wrote asks its check again
// (Checker.VerdictWritten); a deleted key of heads a candidate holds is judged again (Checker.VerdictDeleted, issue 1733).
vw, err := buckets[delivery.CheckerVerdictsBucket].WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
return fmt.Errorf("the checker's verdicts cannot be watched, so no check can be asked again by hand and a deleted verdict would never be written again: %w", err)
}
go func() {
errs <- follow(ctx, vw, func(e jetstream.KeyValueEntry) {
if opOf(e) {
ch.VerdictWritten(ctx, e.Key(), e.Value())
} else {
ch.VerdictDeleted(ctx, e.Key())
}
})
}()
logger.Printf("started on %s", cred.Node)
go func() { errs <- follow(ctx, w, each) }()
select {
@@ -358,7 +362,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
case err := <-errs:
return err
case err := <-assignErrs:
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
@@ -7,7 +7,8 @@ import (
)
// 1912: with checks as asks on the build seats' streams, the stream redelivers a lost node's ask (ADR 0525 §6), so the
// checker program opens no node-engine have to start again a check that failed for want of a node.
// checker program watches no node-engine have to start again a check that failed for want of a node. It reads the have's
// keys only for the nodes' names at each ask (judge.go, novox/hq ADR 0567 §12).
func TestTheCheckerWatchesNoHave(t *testing.T) {
raw, err := os.ReadFile("main.go")
if err != nil {
@@ -0,0 +1,62 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,218 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c h1:hcqOCd5q3M0ZiEzB6MhwxT/AOKkm9gaskKqPcpkAAg8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028205404-8eafc0c5ce3c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,355 @@
package check
import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// One check per delivery (novox/hq ADR 0548): the checker's part. A merge turn holds new checks off its repository (§3);
// a trunk or sibling move never ends a running check and carries its verdict once it ended (§1, §5, §6); only a head git
// refused gets a whole check after a move (§7); the line of a carried verdict says so (§8).
// turnOf is a turn's value on a repository for a delivery of the heads hash.
func turnOf(t testing.TB, repo, group, heads string) []byte {
t.Helper()
at := time.Date(2026, 10, 26, 9, 0, 0, 0, time.UTC)
data, err := delivery.EncodeTurn(delivery.Turn{Repository: repo, Waiting: []delivery.TurnWait{{Group: group, Heads: heads, Approved: at, Since: at}}})
if err != nil {
t.Fatal(err)
}
return data
}
func turnKey(t testing.TB, repo string) string {
t.Helper()
k, err := delivery.TurnsKey(repo)
if err != nil {
t.Fatal(err)
}
return k
}
// waitFor polls a condition the checker's goroutines make true, failing after two seconds.
func waitFor(t testing.TB, what string, ok func() bool) {
t.Helper()
for deadline := time.Now().Add(2 * time.Second); !ok(); time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("%s did not happen", what)
}
}
}
// pullWithFP is a pull request touching the module m at the fingerprint `src2:` and 64 times the hex digit d.
func pullWithFP(repo string, number int, head string, d string) delivery.CandidatePull {
p := pullOf(repo, number, head)
p.Modules = []string{"m"}
p.Fps = map[string]delivery.CandidateModule{"m": {FP2: "src2:" + strings.Repeat(d, 64)}}
return p
}
func TestOneCheckPerDelivery(t *testing.T) {
ctx := context.Background()
for name, run := range map[string]func(t *testing.T){
// §3: no claim while a turn names one of its repositories; a claim on the turn's delete event.
"a turn on its repository holds a new check off until the turn goes": func(t *testing.T) {
j := passJudge()
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "fix/other", "0123456789ab"), false)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 0 || len(r.verdicts.vals) != 0 {
t.Fatalf("judged %d times with a turn on %s, verdicts %v", j.count(), repoA, r.verdicts.vals)
}
if held := r.ch.Held()[keyOf(t, "feat/x", cand)]; !strings.Contains(held, repoA) || !strings.Contains(held, "fix/other") {
t.Errorf("the hold is said as %q, want the turn's repository and group", held)
}
if !turns.Put(turnKey(t, repoA), nil, true) {
t.Fatal("the turn's delete changed nothing")
}
r.ch.Reconsider(ctx) // what the watch of turns does on the delete
r.ch.Wait()
if j.count() != 1 || r.verdict(t, keyOf(t, "feat/x", cand)).State != delivery.VerdictPass {
t.Errorf("judged %d times after the turn went, want once", j.count())
}
},
"a turn on another repository, or of its own heads, holds nothing": func(t *testing.T) {
j := passJudge()
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
c, _ := delivery.DecodeCandidate(cand)
turns.Put(turnKey(t, repoSDK), turnOf(t, repoSDK, "fix/other", "0123456789ab"), false)
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "feat/x", GateHeads(c)), false)
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if j.count() != 1 {
t.Errorf("judged %d times, want once: the turns hold another repository and its own merge", j.count())
}
},
// §3, the race: a check claimed before the gate's turn is judged on the commit it read, never restarted.
"a check claimed before a turn was written runs to its verdict": func(t *testing.T) {
release, started := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) {
close(started)
<-release
return Outcome{Verdict: passing()}, nil
}}
r := newRig(t, j)
turns := &Turns{}
r.ch.Turns = turns
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
<-started
turns.Put(turnKey(t, repoA), turnOf(t, repoA, "fix/other", "0123456789ab"), false)
r.ch.Reconsider(ctx)
close(release)
r.ch.Wait()
if j.count() != 1 || r.verdict(t, keyOf(t, "feat/x", cand)).State != delivery.VerdictPass {
t.Errorf("judged %d times; the running check is not ended by the turn", j.count())
}
},
// §1, §5: a trunk move during a running check calls the judge once, writes its verdict, keeps the status pending (a pass
// on an old trunk is never shown), then asks one carry, on the recheck stream, and no check.
"a trunk move during a check ends nothing and asks one carry after it": func(t *testing.T) {
var r *rig
var kinds []string
var from []map[string]string
judged := make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
kinds, from = append(kinds, c.Kind), append(from, c.CarriedFrom)
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
r.ch.TrunkMoved(ctx) // as the watch of the source says it
close(judged)
} else if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status during the carry is %q, want pending: the pass on the old trunk is never shown", s)
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
<-judged
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || kinds[0] != KindNew || kinds[1] != KindTrunk || from[0] != nil || from[1][repoA] != trunkA {
t.Fatalf("judged %d times as %v carried from %v; want a whole check, then one carry from %s", j.count(), kinds, from, trunkA[:12])
}
if v.State != delivery.VerdictPass || v.Trunk[repoA] != trunkA2 || v.CarriedFrom[repoA] != trunkA {
t.Errorf("the verdict is %s on %v carried from %v; want a pass on the new trunk carried from the old", v.State, v.Trunk, v.CarriedFrom)
}
if s := owedState(t, r, repoA, 7, head1); s != "success" {
t.Errorf("the status after the carry is %q, want success", s)
}
if r.verdicts.creates[keyOf(t, "feat/x", cand)] != 1 {
t.Error("the verdict key was made again: a carry rewrites it")
}
},
// §5: the composition is kept when no fingerprint of a module it composed changed, and runs again otherwise.
"a carry keeps a passed composition while the fingerprints stay the same": func(t *testing.T) {
var kept []bool
j := &judgeFunc{f: func(_ context.Context, c *Case, _ int) (Outcome, error) {
kept = append(kept, c.Composition != nil)
v := passing()
v.Composition = &delivery.Composition{State: delivery.CompositionPass, SDK: "v0.1.14-x", FPSet: c.FPSet}
return Outcome{Verdict: v}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullWithFP(repoA, 7, head1, "a"))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx) // the indexer's fingerprints of the candidate stay the same
r.ch.Wait()
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullWithFP(repoA, 7, head1, "b"))) // a fingerprint changed with the move
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if len(kept) != 3 || kept[0] || !kept[1] || kept[2] {
t.Errorf("the composition was kept in the checks %v; want composed, kept on the carry, composed again once a fingerprint changed", kept)
}
},
// §6: a sibling's move puts the verdict out of date, its status pending saying it is carried, and the carry's verdict
// names the sibling's old commit; the status follows the carry's verdict. Which packages run is the digest's (only
// those that name MESH_CHECK_BESIDE change with a sibling), the build module's part.
"a sibling's move is carried and the status follows the carry": func(t *testing.T) {
var r *rig
var from map[string]string
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
if call == 2 {
from = c.CarriedFrom
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status during the sibling's carry is %q, want pending", s)
}
}
return Outcome{Verdict: passing(), Beside: map[string]string{catalogueRepository: r.trunks.Heads()[catalogueRepository]}}, nil
}}
r = newRig(t, j)
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, catalogueRepository: trunkS})
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
if b := r.verdict(t, keyOf(t, "feat/x", cand)).Beside; b[catalogueRepository] != trunkS {
t.Fatalf("the verdict names the siblings %v, want the catalogue at %s", b, trunkS[:12])
}
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, catalogueRepository: trunkS2})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || from[catalogueRepository] != trunkS || v.CarriedFrom[catalogueRepository] != trunkS || v.Beside[catalogueRepository] != trunkS2 {
t.Errorf("judged %d times, carried from %v, verdict beside %v carried from %v; want one carry over the catalogue's move", j.count(), from, v.Beside, v.CarriedFrom)
}
if s := owedState(t, r, repoA, 7, head1); s != "success" {
t.Errorf("the status after the carry is %q, want success", s)
}
},
// §7: a carry whose merge conflicts fails naming the files and asks no check; a later move onto which the head merges
// cleanly asks a whole check, not a carry.
"a carry git refuses fails naming the files, and the next move asks a whole check": func(t *testing.T) {
var kinds []string
var from []map[string]string
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
kinds, from = append(kinds, c.Kind), append(from, c.CarriedFrom)
if call == 2 {
return Outcome{Verdict: delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: "ace",
Words: repoA + "#7 " + ConflictWords + "go.mod"}}}, nil
}
return Outcome{Verdict: passing()}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
r.ch.Reconsider(ctx) // anything else that looks again asks nothing of a conflict
r.ch.Wait()
if v := r.verdict(t, keyOf(t, "feat/x", cand)); j.count() != 2 || !IsConflictFailure(v) || !strings.Contains(v.Failed.Words, "go.mod") {
t.Fatalf("judged %d times, verdict %+v; want the carry failed naming go.mod and nothing asked after it", j.count(), v.Failed)
}
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if j.count() != 3 || kinds[1] != KindTrunk || from[1] == nil || kinds[2] != KindNew || from[2] != nil {
t.Errorf("checks %v carried from %v; want a whole check, a carry, then a whole check after the conflict", kinds, from)
}
},
// §8, §6: the line of a carried verdict names the head on the new trunk commit first, as the gate and the merge tool
// read a line, then where it was carried from and what the carry ran and kept; while it is carried it is pending and says so.
"the line of a carry says where it was carried from": func(t *testing.T) {
cs, p := lineCase(t, pullOf(repoA, 7, head1))
v := passVerdict()
v.Trunk = map[string]string{repoA: trunkA}
v.Beside = map[string]string{catalogueRepository: trunkS2}
v.CarriedFrom = map[string]string{repoA: trunkA2, catalogueRepository: trunkS}
line := Line(v, cs, p, nil, nil, nil)
if !strings.HasPrefix(line, "pass 111111111111 on aaaaaaaaaaaa | carried from abababababab, mesh-catalog 555555555555 | unit 12 run") {
t.Errorf("the carried line = %q", line)
}
if head, trunk := checkWordsOf(line); head != head1[:12] || trunk != trunkA[:12] {
t.Errorf("the line names %s on %s first, want the head on the new trunk", head, trunk)
}
cs.CarriedFrom = map[string]string{repoA: trunkA2}
pending := Line(delivery.Verdict{State: delivery.VerdictRunning}, cs, p, nil, nil, nil)
if !strings.HasPrefix(pending, "being carried to 111111111111 on aaaaaaaaaaaa | from abababababab") {
t.Errorf("the pending line of a carry = %q", pending)
}
},
// §3 (review round 1): two approved merges in one turn whose trunk moved each get their carry; neither holds the other's.
"two merges waiting in one turn each get their carry": func(t *testing.T) {
var r *rig
j := &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) { return Outcome{Verdict: passing()}, nil }}
r = newRig(t, j)
candA, candC := candidateOf(t, "feat/a", pullOf(repoA, 7, head1)), candidateOf(t, "feat/c", pullOf(repoA, 8, head2))
r.ch.Candidate(ctx, "feat%2Fa", candA)
r.ch.Candidate(ctx, "feat%2Fc", candC)
r.ch.Wait()
a, _ := delivery.DecodeCandidate(candA)
c, _ := delivery.DecodeCandidate(candC)
at := time.Date(2026, 10, 26, 9, 0, 0, 0, time.UTC)
turn, err := delivery.EncodeTurn(delivery.Turn{Repository: repoA, Waiting: []delivery.TurnWait{
{Group: "feat/a", Heads: GateHeads(a), Approved: at, Since: at}, {Group: "feat/c", Heads: GateHeads(c), Approved: at.Add(time.Minute), Since: at}}})
if err != nil {
t.Fatal(err)
}
turns := &Turns{}
turns.Put(turnKey(t, repoA), turn, false)
r.ch.Turns = turns
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // a third merge moves the trunk under both
r.ch.TrunkMoved(ctx)
r.ch.Wait()
for _, cand := range [][]byte{candA, candC} {
group := map[bool]string{true: "feat/a", false: "feat/c"}[string(cand) == string(candA)]
if v := r.verdict(t, keyOf(t, group, cand)); v.Trunk[repoA] != trunkA2 || v.CarriedFrom[repoA] != trunkA {
t.Errorf("%s: verdict on %v carried from %v; want its carry claimed and done", group, v.Trunk, v.CarriedFrom)
}
}
},
// §6 (review round 1): the node-engine read at the commit the mesh runs is named, and a new running commit is carried.
"a new running node-engine is carried": func(t *testing.T) {
runs := runsAt{nodeEngineModule: trunkS}
j := &judgeFunc{f: func(_ context.Context, c *Case, _ int) (Outcome, error) {
return Outcome{Verdict: passing(), Beside: map[string]string{nodeEngineRepo: runs[nodeEngineModule]}}, nil
}}
r := newRig(t, j)
r.ch.Runs = runs
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.ch.TrunkMoved(ctx) // nothing moved: nothing is carried
r.ch.Wait()
runs[nodeEngineModule] = trunkS2
r.ch.TrunkMoved(ctx) // what the indexer's watch does when mesh-host's running commit changes
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if j.count() != 2 || v.CarriedFrom[nodeEngineRepo] != trunkS || v.Beside[nodeEngineRepo] != trunkS2 {
t.Errorf("judged %d times, verdict beside %v carried from %v; want one carry over the node-engine's new running commit", j.count(), v.Beside, v.CarriedFrom)
}
},
// §5: a carry that fails names the move it failed on.
"a carry that fails names the move": func(t *testing.T) {
j := &judgeFunc{f: func(_ context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
return Outcome{Verdict: passing()}, nil
}
return Outcome{Verdict: delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Test: "unit: m/a.TestX", Words: "want 4"}}}, nil
}}
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
v := r.verdict(t, keyOf(t, "feat/x", cand))
if v.State != delivery.VerdictFail || v.CarriedFrom != nil || !strings.HasPrefix(v.Failed.Words, "want 4") ||
!strings.Contains(v.Failed.Words, repoA+" "+trunkA[:12]+" to "+trunkA2[:12]) {
t.Errorf("the failed carry is %s carried from %v with words %q; want the test's words, then the move", v.State, v.CarriedFrom, v.Failed.Words)
}
},
} {
t.Run(name, run)
}
}
// checkWordsOf is the head and trunk a line names first, as the gate reads it (mesh-delivery-gate onestatus.go checkLine).
func checkWordsOf(line string) (string, string) {
for _, part := range strings.Split(line, " | ") {
f := strings.Fields(part)
for i := 0; i+2 < len(f); i++ {
if f[i+1] == "on" && len(f[i]) == 12 && len(f[i+2]) == 12 {
return f[i], f[i+2]
}
}
}
return "", ""
}
@@ -0,0 +1,198 @@
package check
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A check is asked again by hand** (the lost check of 2026-10-28; tools-are-how-a-module-is-spoken-to: every recovery
// has a tool). checker_ask_again ends a candidate's running verdict, whose wait judged nothing, with AskedAgainWords and
// the why; the checker hears it on its watch of its verdicts (Checker.VerdictWritten), ends the check of those heads
// running in it, and claims and asks it anew, a whole check (IsEnded). It is refused while a step of the candidate's pull
// requests can still run (checker_status's running_steps): that step's end judges it; and while an ask the check put is
// still in its build seat's stream, taken by no node (it has no step key yet): asking again then would run the check twice
// (issue 1791's kind). The checker keeps the asks each running check put in its state asks (AsksState, by verdict key).
// An ask is not put again under its own id instead: the stream de-duplicates an id within its window, so the copy waiting
// stays and the new one is dropped, and past the window both would run; nothing takes a put ask back (buildrun.Asker).
// AsksState is the checker's state of the asks each running check put, by verdict key (CheckAsks); AsksBucket its bucket.
const (
AsksState = "asks"
AsksBucket = delivery.CheckerModule + "_" + AsksState
)
// CheckAsks are the asks a check put, in order, as the state asks keeps them while it runs; its end deletes the key.
type CheckAsks struct {
Asks []string `json:"asks"`
}
// AsksKV is the state asks: a KV whose key a check's end deletes, so the state holds only the checks that run.
type AsksKV interface {
KV
Delete(ctx context.Context, key string) error
}
// forgetAsks deletes the asks a check put at its end; one that cannot be deleted is said, and the next check of the same
// heads writes the key anew.
func (ch *Checker) forgetAsks(ctx context.Context, key string) {
if err := ch.Asks.Delete(ctx, key); err != nil {
ch.logf("%s: the asks it put could not be forgotten at its end: %v", key, err)
}
}
// recordAsks keeps the asks a check put under its verdict key; nil Asks keeps nothing. A record that cannot be written
// is said: checker_ask_again then refuses the check, as it cannot tell whether its ask waits in a stream.
func (ch *Checker) recordAsks(ctx context.Context, key string, asks []string) {
if ch.Asks == nil {
return
}
data, err := json.Marshal(CheckAsks{Asks: asks})
if err == nil {
_, err = putOnce(ctx, ch.Asks, key, data, func(cur []byte) bool { return string(cur) == string(data) })
}
if err != nil {
ch.logf("%s: the asks it put cannot be kept (%v), so %s refuses it", key, err, AskAgainTool)
}
}
// AskAgainTool asks a candidate's check again by hand.
const AskAgainTool = "checker_ask_again"
// AskedAgainWords open the words of a verdict AskAgainTool wrote; the why follows.
const AskedAgainWords = EndedWords + "asked again by hand (" + AskAgainTool + "): "
// errAskedAgain is the cause a running check is ended with when its verdict was asked again by hand.
var errAskedAgain = errors.New("its verdict was asked again by hand (" + AskAgainTool + ")")
// IsAskedAgain says whether a verdict was written by AskAgainTool.
func IsAskedAgain(v delivery.Verdict) bool {
return IsEnded(v) && strings.HasPrefix(v.Failed.Words, AskedAgainWords)
}
// VerdictEntry is the verdicts state as the ask-again tool writes it: a key's value and revision (nil and 0 for none),
// and a compare-and-set.
type VerdictEntry interface {
Entry(key string) (json.RawMessage, uint64, error)
Update(key string, value []byte, revision uint64) error
}
// AskedAgain is the answer of AskAgainTool.
type AskedAgain struct {
Key string `json:"key"`
Was string `json:"was"`
Said string `json:"said"`
Note string `json:"note"`
}
// AskAgain ends the running verdict of the group's candidate as asked again by hand, saying why. It refuses a group with
// no candidate, a verdict that is not running (a pass or a failure stands until a push or a trunk move), one a step of
// whose pull requests can still run, and one whose asks are not known or one of which no node took yet.
func (r Reader) AskAgain(w VerdictEntry, group, why string) (AskedAgain, error) {
why = strings.TrimSpace(why)
if group == "" || why == "" {
return AskedAgain{}, fmt.Errorf("give the group and why it is asked again")
}
if w == nil {
return AskedAgain{}, fmt.Errorf("the verdicts state cannot be written here")
}
c, ok := r.candidate(group)
if !ok {
return AskedAgain{}, fmt.Errorf("no candidate is held for the group %q", group)
}
key, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash)
if err != nil {
return AskedAgain{}, err
}
raw, rev, err := w.Entry(key)
if err != nil {
return AskedAgain{}, fmt.Errorf("%s cannot be read: %w", key, err)
}
if len(raw) == 0 {
return AskedAgain{}, fmt.Errorf("%s holds no verdict: the checker claims its check when it looks at the candidate (%s)", key, StatusReadTool)
}
v, err := delivery.DecodeVerdict(raw)
if err != nil {
return AskedAgain{}, fmt.Errorf("%s holds what is not a verdict: %w", key, err)
}
if v.State != delivery.VerdictRunning {
return AskedAgain{}, fmt.Errorf("%s is %s, not running: a verdict stands until a push or a trunk move asks it again", key, v.State)
}
steps := r.readSteps()
for _, rs := range steps.running {
if asksOf(c, rs.What) {
return AskedAgain{}, fmt.Errorf("a step of it runs: %s %s on %s since %s (%s); its end judges it", rs.Module, rs.ID, rs.Node, rs.Since, rs.What)
}
}
if r.AskState.Get == nil {
return AskedAgain{}, fmt.Errorf("the checker's asks are not readable here, so whether its ask waits in a build seat's stream cannot be told")
}
kept, err := r.AskState.Get(key)
var put CheckAsks
if err == nil && len(kept) > 0 {
err = json.Unmarshal(kept, &put)
}
if err != nil || len(put.Asks) == 0 {
return AskedAgain{}, fmt.Errorf("the checker names no ask of %s (%v), so whether one waits in a build seat's stream cannot be told; asking again could run it twice", key, err)
}
taken := map[string]bool{}
for _, s := range steps.all {
taken[s.step.ID] = true
}
for _, id := range put.Asks {
if !taken[id] {
return AskedAgain{}, fmt.Errorf("its ask %s waits in its build seat's stream, taken by no node yet: asking again would run it twice; "+
"the seats' backlog is in mesh-delivery-view.build_usage", id)
}
}
words := AskedAgainWords + why
ended := delivery.Verdict{Group: v.Group, Heads: v.Heads, Trunk: v.Trunk, State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Machine: "checker", Words: words}}
data, err := delivery.EncodeVerdict(ended)
if err != nil {
return AskedAgain{}, err
}
if err := w.Update(key, data, rev); err != nil {
return AskedAgain{}, fmt.Errorf("%s changed while it was asked again, so it was not; read it again: %w", key, err)
}
return AskedAgain{Key: key, Was: v.State, Said: words,
Note: "the checker hears it on its watch of its verdicts, ends the wait that judged nothing and asks the check anew, whole; " +
VerdictsReadTool + " shows it running again"}, nil
}
// VerdictWritten takes a change of the checker's verdicts in (its watch, updates only): a verdict AskAgainTool wrote ends
// the check of those heads running in this process, which then considers them again; with none running, the candidate
// that holds them is considered at once. Any other verdict is the checker's own, and left.
func (ch *Checker) VerdictWritten(ctx context.Context, key string, value []byte) {
v, err := delivery.DecodeVerdict(value)
if err != nil || !IsAskedAgain(v) {
return
}
ch.init()
ch.mu.Lock()
if l, ok := ch.live[key]; ok {
l.end(errAskedAgain)
ch.mu.Unlock()
ch.logf("%s: %s; the check running for it is ended and asked again", key, v.Failed.Words)
return
}
var group string
var cand delivery.Candidate
for k, c := range ch.known {
if vk, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash); err == nil && vk == key {
group, cand = k, c
break
}
}
ch.mu.Unlock()
if group == "" {
ch.logf("%s: %s, but no candidate holds these heads, so nothing is asked", key, v.Failed.Words)
return
}
ch.logf("%s: %s; asked again", key, v.Failed.Words)
ch.consider(ctx, group, cand)
}
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -96,9 +97,19 @@ func notJudged(v delivery.Verdict) bool {
return v.Composition != nil && v.Composition.State == delivery.CompositionNotRun && v.Composition.Words == delivery.AssignerNotRunning
}
// unanswered says whether a verdict failed only on its composition, and that only because nodes the change places did
// not answer its placement in time (delivery.CompositionUnanswered, ADR 0559 §5): its tests passed (a composition is read
// only into a passing verdict), so a later composition of the same heads is taken in on its event, with no new check
// (ADR 0548 §1: a part of the mesh not heard is judged again when it is heard).
func unanswered(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && v.Composition != nil && v.Composition.State == delivery.CompositionFail &&
strings.Contains(v.Composition.Words, delivery.CompositionUnanswered) && v.Failed.Words == v.Composition.Words
}
// Composed takes in a composition the assigner wrote after the check's verdict said it was not judged because the
// assigner was not running (the assigner started since): the verdict and its status are written again with it. A
// composition of heads no candidate holds now, or of a verdict that judged its composition, is left.
// assigner was not running (the assigner started since), or after it failed only because placed nodes had not answered
// (unanswered): the verdict and its status are written again with it. A composition of heads no candidate holds now, or
// of a verdict that judged its composition otherwise, is left.
func (ch *Checker) Composed(ctx context.Context, comp delivery.CompositionVerdict) {
ch.init()
if comp.State == delivery.CompositionWaits {
@@ -123,10 +134,21 @@ func (ch *Checker) Composed(ctx context.Context, comp delivery.CompositionVerdic
return
}
v, err := delivery.DecodeVerdict(cur)
if err != nil || v.State != delivery.VerdictPass || !notJudged(v) {
if err != nil {
return
}
switch {
case v.State == delivery.VerdictPass && notJudged(v):
case unanswered(v) && comp.Words != v.Composition.Words:
// The nodes answered since (or another failure stands now): the verdict is its tests' pass with this composition.
v.State, v.Failed = delivery.VerdictPass, delivery.FailedFirst{}
default:
return
}
withComposition(&v, comp)
if v.Composition != nil && v.Composition.State != delivery.CompositionNotRun {
v.Composition.FPSet = cs.FPSet
}
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: the verdict with its composition cannot be written: %v", cs.Key, err)
@@ -113,6 +113,27 @@ func TestTheAssignerStartingBringsTheComposition(t *testing.T) {
t.Errorf("the status's line says %q", o.Text)
}
}
// ADR 0559 §5: a verdict that failed only because a placed node had not answered takes the same heads' later
// composition in on its event, a pass making it pass with no new check; a composition failing the same way is left.
ch, cs := checkerWithNotJudged(t)
silent := delivery.CompositionVerdict{Group: cs.Group, Heads: cs.HeadsHash, State: delivery.CompositionFail, SDK: "x", Machine: "g14",
Words: "g14 (offline) " + delivery.CompositionUnanswered}
ch.Composed(context.Background(), silent)
for _, row := range []struct {
name string
comp delivery.CompositionVerdict
state string
}{
{"the same failure again", silent, delivery.VerdictFail},
{"the node's answer", delivery.CompositionVerdict{Group: cs.Group, Heads: cs.HeadsHash, State: delivery.CompositionPass, SDK: "x"}, delivery.VerdictPass},
} {
ch.Composed(context.Background(), row.comp)
raw, _, _, _ := ch.Verdicts.Get(context.Background(), cs.Key)
if v, err := delivery.DecodeVerdict(raw); err != nil || v.State != row.state || v.Composition == nil || v.Composition.State != row.comp.State {
t.Errorf("%s: the verdict is %+v, %v", row.name, v, err)
}
}
}
// checkerWithNotJudged is a checker holding a candidate whose passing verdict said its composition was not judged.
@@ -12,8 +12,18 @@ type BesideRef struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
Ref string `json:"ref,omitempty"`
// atTrunk says the sibling was read at its trunk's head, so a move of that trunk puts the verdict out of date and is
// carried (novox/hq ADR 0548 §6). Never sent: the node-engine at the commit the mesh runs, mesh-sdk at a pin and a
// group's member at its head do not move with a trunk.
atTrunk bool
// runs says the sibling was read at the commit the mesh runs (the node-engine, the indexer's record of mesh-host): a new
// running commit puts the verdict out of date and is carried (ADR 0548 §6). Never sent.
runs bool
}
// NodeEngineModule is the module whose running commit the node-engine sibling is read at.
const NodeEngineModule = nodeEngineModule
const (
catalogueRepository = "novox/mesh-catalog"
nodeEngineRepo = "novox/mesh-host"
@@ -51,18 +61,21 @@ func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef,
switch {
case repo == catalogueRepository:
if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
out = append(out, BesideRef{Repository: repo, Commit: head, atTrunk: c.member(repo) == nil})
}
case repo == nodeEngineRepo:
commit := ""
if j.Runs != nil {
commit = j.Runs.CommitOf(nodeEngineModule)
}
atTrunk := false
if commit == "" {
commit = heads[repo]
commit, atTrunk = heads[repo], true
}
if commit != "" {
out = append(out, BesideRef{Repository: repo, Commit: commit})
// Named in the verdict either way (ADR 0548 §6, issue 568's own example): at the commit the mesh runs, a new
// running commit puts the verdict out of date; at the trunk, a trunk move does (Checker.besideNow).
out = append(out, BesideRef{Repository: repo, Commit: commit, atTrunk: atTrunk, runs: !atTrunk})
}
case repo == SDKRepository && p.Repository == controllerRepo && j.Forge != nil:
text, err := j.Forge.File(ctx, p.Repository, "go.mod", p.Head)
@@ -82,7 +95,7 @@ func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef,
if member := c.member(repo); member != nil {
out = append(out, BesideRef{Repository: repo, Commit: member.Head, Ref: refOf(member.Number)})
} else if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
out = append(out, BesideRef{Repository: repo, Commit: head, atTrunk: true})
} else if repo == controllerRepo && p.Repository == nodeEngineRepo {
// The node-engine's step-nine test judges the controller's own manifest and fails without it (issue 1295): a
// checker that cannot place it says so, where placing none left that test to skip.
@@ -108,3 +121,13 @@ func refOf(number int) string {
}
return "refs/pull/" + strconv.Itoa(number) + "/head"
}
// besideAtTrunk are the siblings of refs read at their trunk's head that the case does not merge, by repository: what a
// verdict names as Beside (ADR 0548 §6).
func besideAtTrunk(c *Case, refs []BesideRef, into map[string]string) {
for _, r := range refs {
if _, merged := c.Trunk[r.Repository]; (r.atTrunk || r.runs) && !merged {
into[r.Repository] = r.Commit
}
}
}
@@ -0,0 +1,42 @@
package check
import (
"fmt"
"strings"
)
// A change that adds or modifies a compiled program fails (issue 1577): it is build output, `go build` in a module's folder
// writes one beside go.mod, and `git add -A` takes it. The build module's fetch step, which holds the merged clone, reads the
// first bytes of each changed file and reports the programs (mesh-sdk buildstep.MergeInto); the checker judges the answer.
// NoProgramsReported is the warning of a build module whose fetch answered no programs field: an older module, which cannot
// tell, so the check does not refuse a program it did not look for.
const NoProgramsReported = "the build module did not report programs: a compiled program in the change is not refused"
// checkBinariesOf fails a pull request whose change adds or modifies a compiled program, naming each file. It judges the
// first fetch answer of the pull request, whichever build module gave it; a build module that reports no programs leaves a
// warning on the check and passes it.
func (st *state) checkBinariesOf(t *target) verdictPtr {
key := fmt.Sprintf("%s#%d", t.pull.Repository, t.pull.Number)
if t.fetched.Merge == "" {
return nil // no merge was made: nothing was changed to judge
}
if t.fetched.Programs == nil {
st.c.note(fmt.Sprintf("%s: %s (%s)", key, NoProgramsReported, t.module))
st.c.warn(key, fmt.Sprintf("a compiled program is not refused: %s reported no programs", t.module))
st.j.logf("%s: %s (%s)", key, NoProgramsReported, t.module)
return nil
}
var found []string
for _, p := range *t.fetched.Programs {
if p.Commit != "" {
found = append(found, fmt.Sprintf("%s: a compiled program added in %s and removed later: it stays in history; squash the branch (issue 1577)", p.Path, short12(p.Commit)))
continue
}
found = append(found, p.Path+": a compiled program is build output: remove it and add it to .gitignore (issue 1577)")
}
if len(found) > 0 {
return failed(t.node, fmt.Sprintf("%s: %s", key, strings.Join(found, "; ")))
}
return nil
}
@@ -0,0 +1,63 @@
package check
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func TestAChangeWhoseFetchReportsProgramsFailsNamingEachFile(t *testing.T) { // 1577
prog := func(p ...string) *[]ProgramFile {
out := []ProgramFile{}
for _, f := range p {
path, commit, _ := strings.Cut(f, "@") // `path@commit`: added by a commit of the branch and removed later
out = append(out, ProgramFile{Path: path, Kind: "ELF", Commit: commit})
}
return &out
}
for _, c := range []struct {
name string
programs *[]ProgramFile
fail []string // paths named; none passes
warned bool // the build module reported no programs field
}{
{"none found", prog(), nil, false},
{"one program", prog("modules/m/m"), []string{"modules/m/m"}, false},
{"two programs", prog("a/x", "b/y"), []string{"a/x", "b/y"}, false},
{"added and removed in the branch", prog("gone/prog@0123456789abcdef0123"), []string{"gone/prog: a compiled program added in 0123456789ab and removed later: it stays in history; squash the branch (issue 1577)"}, false},
{"an older build module reports nothing", nil, nil, true},
} {
b := newBench(t)
b.fetch.Merge, b.fetch.Programs = "mmmm", c.programs
cs := caseOf(t, pullOf(repoA, 7, head1))
v := b.judge(t, cs).Verdict
if got := slices.ContainsFunc(cs.Notes(), func(n string) bool { return strings.Contains(n, NoProgramsReported) }); got != c.warned {
t.Errorf("%s: warned = %v, want %v (%v)", c.name, got, c.warned, cs.Notes())
}
line := Line(v, cs, cs.Pulls[0], nil, nil, nil)
if got := strings.Contains(line, "a compiled program is not refused: mesh-build-go reported no programs"); got != c.warned {
t.Errorf("%s: the status line holds the warning = %v, want %v: %s", c.name, got, c.warned, line)
}
if len(c.fail) == 0 {
if v.State != delivery.VerdictPass {
t.Errorf("%s: verdict = %+v", c.name, v)
}
continue
}
if v.State != delivery.VerdictFail {
t.Errorf("%s: verdict = %+v, want a failure", c.name, v)
continue
}
for _, p := range c.fail {
want := p
if !strings.Contains(p, " added in ") {
want = p + ": a compiled program is build output: remove it and add it to .gitignore (issue 1577)"
}
if !strings.Contains(v.Failed.Words, want) {
t.Errorf("%s: words %q do not name %s", c.name, v.Failed.Words, p)
}
}
}
}
@@ -43,12 +43,43 @@ type Case struct {
Try *TryOptions
notesMu sync.Mutex
notes []string
// warnings are, by pull request, what part of its check did not run (Case.warn).
warnings map[string]string
notes []string
// asks are the ids of the asks the check put on the build seats' streams, in order: a try names them (Try.Asks).
asks []string
// Watch, when set, hears each ask the check puts, as it is put (step nil), and each value its step key takes while
// the check waits for it: the node that took it, the phase it says, its end. A try writes them into its record, so a
// running try names its asks and where each stands (issue 1890). Called from the check's goroutines; it must not block
// for long.
Watch func(askID string, step *delivery.Step)
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a re-judge after a trunk move is asked on the build
// seats' recheck stream, every other check on check (ADR 0536 §4).
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a carry after a trunk or sibling move is asked on the
// build seats' recheck stream, every other check on check (ADR 0536 §4, ADR 0548 §5).
Kind string
// CarriedFrom are, for the carry of a verdict that passed, the trunk and sibling commits it was judged on (ADR 0548 §5);
// nil for a whole check.
CarriedFrom map[string]string
// FPSet is the candidate's fingerprint set (Candidate.FPSet): a composition names the set it judged.
FPSet string
// Composition is, for a carry, the composition that passed on the same fingerprint set: it is kept and not asked again,
// since no fingerprint of a module it composed changed (ADR 0548 §5). Nil composes.
Composition *delivery.Composition
}
// Asked records the id of an ask the check put.
func (c *Case) Asked(id string) {
c.notesMu.Lock()
c.asks = append(c.asks, id)
c.notesMu.Unlock()
}
// Asks are the ids of the asks the check put, in order.
func (c *Case) Asks() []string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return append([]string(nil), c.asks...)
}
// Progress is the step a check is at, written by the check and read when its bound passes.
@@ -80,11 +111,11 @@ func CaseOf(c delivery.Candidate, trunk map[string]string) *Case {
if err != nil {
return nil
}
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}}
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}, FPSet: c.FPSet}
for _, p := range c.Pulls {
switch head, known := trunk[p.Repository]; {
case !known || head == "":
cs.Left = append(cs.Left, fmt.Sprintf("%s#%d: the trunk head of %s is not known", p.Repository, p.Number, p.Repository))
cs.Left = append(cs.Left, fmt.Sprintf("%s#%d: %s", p.Repository, p.Number, UntrunkedWordsOf([]string{p.Repository})))
default:
cs.Pulls = append(cs.Pulls, Pull{Repository: p.Repository, Number: p.Number, Head: p.Head.Head, Trunk: head, Modules: p.Modules, Fps: p.Fps})
cs.Trunk[p.Repository] = head
@@ -102,6 +133,42 @@ func CaseOf(c delivery.Candidate, trunk map[string]string) *Case {
return cs
}
// UntrunkedWords open the words of a verdict, and of a status, of a pull request that cannot be judged because the gate's
// source.main names no trunk head of its repository (issue 1422): never a failure of the change, and judged whole once the
// trunk head is known. It is said loudly, never left silently "new".
const UntrunkedWords = "cannot be judged: the trunk head of "
// UntrunkedWordsOf are those words for the repositories whose trunk head is not known, in order, each named once.
func UntrunkedWordsOf(repos []string) string {
rs := slices.Clone(repos)
slices.Sort(rs)
rs = slices.Compact(rs)
return UntrunkedWords + strings.Join(rs, ", ") + " is not known: the gate's source.main names none, so there is no commit to merge it onto"
}
// IsUntrunked says whether a verdict says its heads cannot be judged for want of a trunk head (UntrunkedWords).
func IsUntrunked(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && v.Failed.Machine == "checker" && strings.HasPrefix(v.Failed.Words, UntrunkedWords)
}
// leftOf are the pull requests of a candidate whose repository has no trunk head known, with Trunk empty, in repository and
// number order; nil when every repository's is known.
func leftOf(c delivery.Candidate, trunk map[string]string) []Pull {
var out []Pull
for _, p := range c.Pulls {
if trunk[p.Repository] == "" {
out = append(out, Pull{Repository: p.Repository, Number: p.Number, Head: p.Head.Head, Modules: p.Modules, Fps: p.Fps})
}
}
sort.SliceStable(out, func(i, j int) bool {
if out[i].Repository != out[j].Repository {
return out[i].Repository < out[j].Repository
}
return out[i].Number < out[j].Number
})
return out
}
// GateHeads is the hash the gate matches the members an Approve covers by (mesh-catalog mesh-delivery-gate onestatus.go
// headsOf): delivery.HeadsHash over every pull request of the candidate, hq's included, as
// `<owner/name lowercase>#<number>=<full head sha lowercase>`. A lone pull request is a group of one. Empty for a candidate
@@ -162,6 +229,23 @@ func (c *Case) note(s string) {
}
}
// warn says of a pull request (pullKey) that a part of its check did not run, in words for its status line (issue 1577).
func (c *Case) warn(key, words string) {
c.notesMu.Lock()
defer c.notesMu.Unlock()
if c.warnings == nil {
c.warnings = map[string]string{}
}
c.warnings[key] = words
}
// Warning is what warn said of a pull request, or "".
func (c *Case) Warning(key string) string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return c.warnings[key]
}
// Notes are the words noted on the case, in the order they were noted.
func (c *Case) Notes() []string {
c.notesMu.Lock()
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
@@ -42,6 +43,9 @@ type Outcome struct {
// Judged says, by pull request (pullKey), which program judged a change to a build module (ADR 0535 §5); a pull request
// that changes none is not named.
Judged map[string]string
// Beside are the siblings the check read at their trunk's head, by repository (ADR 0548 §6): the verdict names them, and
// a move of one is carried.
Beside map[string]string
}
// BuildNote is the state of a candidate's module build, by module: built, failed or waiting (ADR 0484 clause 7).
@@ -65,14 +69,28 @@ type Checker struct {
Owed KV
Judge Judge
Trunks interface{ Heads() map[string]string }
// Runs says which commit the mesh runs of a module (the indexer's modules): the node-engine a verdict read beside at its
// running commit is out of date when that commit moves (ADR 0548 §6); nil reads every sibling at its trunk.
Runs interface{ CommitOf(module string) string }
// Turns are the gate's merge turns (ADR 0548 §3): no check is claimed of a repository a turn names, but the check of the
// waiting delivery's own heads; nil holds none.
Turns interface {
Holding(repositories []string, heads string) (string, delivery.TurnWait, bool)
}
// Bound is how long one check may run; none when 0 (DefaultBound).
Bound time.Duration
// Asks keeps the asks each running check put, by verdict key (AsksState), for checker_ask_again; nil keeps none.
Asks AsksKV
// Link is the base of the address a status points at; the verdict key follows.
Link string
// Announce says a status was written (EventOwedWritten); nil says nothing.
Announce Announcer
Now func() time.Time
Logf func(format string, args ...any)
// Settle is how long a group's heads must stay the same after the candidate last changed them before its check starts
// (issue 1890): the members of a group are announced one after another, and a check of each partial set is thrown
// away at the next. None when 0; DefaultSettle in the module.
Settle time.Duration
once sync.Once
wg sync.WaitGroup
@@ -83,15 +101,78 @@ type Checker struct {
tried map[string]bool // by verdict key: looked at since this process started
epoch uint64 // changes of the nodes seen since this process started
nodesFail map[string]uint64 // by verdict key: the epoch a failure of the nodes was written at
held map[string]string // by verdict key: the turn that holds its check off, as last said
live map[string]liveCheck // by verdict key: the check running in this process, ended when its heads go
settling map[string]*window // by candidates key: the window its heads settle in
}
// liveCheck is a check running in this process: the candidates key of its group and the end of its run.
type liveCheck struct {
group string
end context.CancelCauseFunc
}
// window is a group's settle window: aborted with errUnsettled when its heads change again or the group goes, and with
// errSettled to end it at once.
type window struct{ abort context.CancelCauseFunc }
var (
errUnsettled = errors.New("the group's heads changed again")
errSettled = errors.New("the group's heads settled")
)
// settleNow ends the settle window of every group at once, as if Settle had passed.
func (ch *Checker) settleNow() {
ch.mu.Lock()
defer ch.mu.Unlock()
for _, w := range ch.settling {
w.abort(errSettled)
}
}
// DefaultSettle is the module's Settle (issue 1890). On 2026-10-26 the three members of a group were announced within 2.8
// seconds of each other (20:31:52.3, :53.7, :55.1 UTC), a session opening or pushing them one API call after another;
// fifteen seconds holds five times that, and costs a check of 10 to 18 minutes under 2.5 % of its time. A member that comes
// later still is no waste beyond its asks already taken: the check of the heads it replaced is ended at once.
const DefaultSettle = 15 * time.Second
// EndedWords open the words of a verdict the checker ended without judging it, its heads no candidate's any more (issue
// 1890): never a failure of the change, and judged whole if a candidate holds those heads again.
const EndedWords = "ended: "
// IsEnded says whether a verdict was ended without being judged (EndedWords).
func IsEnded(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && v.Failed.Machine == "checker" && strings.HasPrefix(v.Failed.Words, EndedWords)
}
// superseded is the cause a running check is ended with: the heads that replaced its own, or none when its group went.
type superseded struct{ by string }
func (s superseded) Error() string {
if s.by == "" {
return EndedWords + "no candidate holds these heads any more: the group was closed or merged"
}
return EndedWords + "superseded by " + s.by
}
func (ch *Checker) init() {
ch.once.Do(func() {
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail = map[string]uint64{}
ch.nodesFail, ch.held = map[string]uint64{}, map[string]string{}
ch.live, ch.settling = map[string]liveCheck{}, map[string]*window{}
})
}
// endOthers ends every check of the group running in this process whose verdict key is not keep: no candidate holds its
// heads any more (issue 1890). Called with ch.mu held.
func (ch *Checker) endOthers(group, keep, by string) {
for k, l := range ch.live {
if l.group == group && k != keep {
l.end(superseded{by: by})
}
}
}
func (ch *Checker) logf(format string, args ...any) {
if ch.Logf != nil {
ch.Logf(format, args...)
@@ -124,17 +205,91 @@ func (ch *Checker) Candidate(ctx context.Context, key string, value []byte) {
ch.logf("candidates.%s holds what is not a candidate, left: %v", key, err)
return
}
vkey, _ := delivery.CheckerVerdictKey(c.Group, c.HeadsHash)
ch.mu.Lock()
prev, had := ch.known[key]
ch.known[key] = c
// A check of heads this candidate no longer holds is ended at once (issue 1890): a member's new head, a member joining
// or leaving.
ch.endOthers(key, vkey, GateHeads(c))
moved := !had || prev.HeadsHash != c.HeadsHash
if ch.Settle > 0 && (moved || ch.settling[key] != nil) {
// The group's heads changed: its check waits until they have been the same for Settle, so a group whose members
// are announced one after another is checked once, whole. A later change starts the window again. The wait is a
// context's bound, no timer (ADR 0437 §8).
if w := ch.settling[key]; w != nil {
w.abort(errUnsettled)
}
base, abort := context.WithCancelCause(ctx)
w := &window{abort: abort}
ch.settling[key] = w
ch.wg.Add(1)
go func() {
defer ch.wg.Done()
wait, done := context.WithTimeout(base, ch.Settle)
defer done()
<-wait.Done()
settled := errors.Is(wait.Err(), context.DeadlineExceeded) || errors.Is(context.Cause(wait), errSettled)
ch.mu.Lock()
if !settled || ch.settling[key] != w {
ch.mu.Unlock()
return // changed again, forgotten, or the checker stops
}
delete(ch.settling, key)
now, ok := ch.known[key]
ch.mu.Unlock()
abort(nil)
if ok {
ch.consider(ctx, key, now)
}
}()
ch.mu.Unlock()
return
}
ch.mu.Unlock()
ch.consider(ctx, key, c)
}
// Forget takes a candidate out (its key was deleted).
// VerdictDeleted takes the delete of a verdict key in (the checker's watch of its verdicts, issue 1733): the gate asks the
// operator's Approve only on a verdict, so heads a candidate holds are never left without one. A check of them running in
// this process writes its verdict anew at its end; otherwise the candidate is considered at once and judged again (or said
// again to be unjudgeable). A key no candidate holds is said and left.
func (ch *Checker) VerdictDeleted(ctx context.Context, key string) {
ch.init()
ch.mu.Lock()
if ch.inflight[key] {
ch.mu.Unlock()
ch.logf("%s: deleted while its check runs; its end writes it anew", key)
return
}
var group string
var cand delivery.Candidate
for k, c := range ch.known {
if vk, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash); err == nil && vk == key {
group, cand = k, c
break
}
}
ch.mu.Unlock()
if group == "" {
ch.logf("%s: deleted, and no candidate holds these heads, so nothing is judged", key)
return
}
ch.logf("%s: deleted; its heads are judged again", key)
ch.consider(ctx, group, cand)
}
// Forget takes a candidate out (its key was deleted): a check of its heads still running is ended, as no candidate holds
// them any more.
func (ch *Checker) Forget(key string) {
ch.init()
ch.mu.Lock()
delete(ch.known, key)
if w := ch.settling[key]; w != nil {
w.abort(errUnsettled)
delete(ch.settling, key)
}
ch.endOthers(key, "", "")
ch.mu.Unlock()
}
@@ -171,38 +326,67 @@ func (ch *Checker) heads() map[string]string {
}
func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidate) {
cs := CaseOf(c, ch.heads())
heads := ch.heads()
if left := leftOf(c, heads); len(left) > 0 {
// A pull request the checker cannot claim (its repository's trunk head is not known) leaves its whole group
// unjudged, said in a failure verdict (issue 1422, issue 1733): the gate asks the Approve only on a verdict, and a
// pass of the other members would let the group merge with that one never checked.
ch.untrunked(ctx, c, left)
return
}
cs := CaseOf(c, heads)
if cs == nil {
return // nothing to judge: no trunk known
return // no pull request at all: nothing to judge
}
cs.Queued = ch.now()
ch.mu.Lock()
if ch.settling[key] != nil {
ch.mu.Unlock()
return // its heads are settling: the window's end considers it (Settle)
}
if ch.inflight[cs.Key] {
ch.again[cs.Key] = true // looked at once more when the check ends, so a change of the trunk is never lost
ch.mu.Unlock()
return
}
ch.inflight[cs.Key] = true
run, end := context.WithCancelCause(ctx)
ch.live[cs.Key] = liveCheck{group: key, end: end}
ch.mu.Unlock()
ch.wg.Add(1)
go func() {
defer ch.wg.Done()
vkey := cs.Key
defer func() {
ch.mu.Lock()
delete(ch.inflight, vkey)
delete(ch.again, vkey)
delete(ch.live, vkey)
ch.mu.Unlock()
again := errors.Is(context.Cause(run), errAskedAgain)
end(nil)
if again && ctx.Err() == nil {
ch.consider(ctx, key, ch.candidate(key, c)) // asked again by hand (AskAgainTool): claimed anew
}
}()
for {
// Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a
// failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9;
// ADR 0536 §4). The checker holds no place and no capacity of its own.
cs.Kind = ch.kindOf(ctx, cs)
ch.try(ctx, key, cs)
ch.prepare(ctx, cs)
ch.try(ctx, run, key, cs)
ch.mu.Lock()
more := ch.again[cs.Key]
delete(ch.again, cs.Key)
if !more || ctx.Err() != nil {
delete(ch.inflight, cs.Key)
ch.mu.Unlock()
more := ch.again[vkey]
delete(ch.again, vkey)
ch.mu.Unlock()
if !more || run.Err() != nil {
return
}
ch.mu.Unlock()
// Looked at again for the same heads only: a candidate that moved to other heads has its own check (issue 1890).
if next := CaseOf(ch.candidate(key, c), ch.heads()); next != nil {
if next.Key != vkey {
return
}
cs = next
}
cs.Queued = ch.now()
@@ -210,30 +394,203 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
}()
}
// untrunked says, loudly, that pull requests of a candidate cannot be judged because the gate's source.main names no trunk
// head of their repository (issue 1422): its verdict key holds a failure that says so, whatever it held before (a running
// verdict left by a check that can no longer run, or a verdict judged on a trunk that is no longer named: issue 1733), which
// checker_verdicts and checker_status show; and every pull request of the candidate gets the status error with
// UntrunkedWords, the members whose trunk is known too, as their group is not judged. It is logged once per head: when a
// status or the verdict is written, never at a look that finds them already said. Once the trunk head is known the verdict
// is stale (it names no trunk) and the heads are judged whole (prepare, claim).
func (ch *Checker) untrunked(ctx context.Context, c delivery.Candidate, left []Pull) {
key, err := delivery.CheckerVerdictKey(c.Group, c.HeadsHash)
if err != nil {
ch.logf("candidates of %s: no verdict key for heads %q, so it cannot be said that they cannot be judged: %v", c.Group, c.HeadsHash, err)
return
}
repos := make([]string, 0, len(left))
for _, p := range left {
repos = append(repos, p.Repository)
}
words := UntrunkedWordsOf(repos)
v := delivery.Verdict{Group: c.Group, Heads: GateHeads(c), Trunk: map[string]string{}, State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Machine: "checker", Words: words}}
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: %s, and the verdict cannot say so: %v", key, words, err)
return
}
said, err := putOnce(ctx, ch.Verdicts, key, data, func(cur []byte) bool {
held, err := delivery.DecodeVerdict(cur)
return err == nil && IsUntrunked(held) && held.Failed.Words == words // the same words stand; anything else is replaced
})
if err != nil {
ch.logf("%s: %s, and the verdict cannot say so: %v", key, words, err)
}
for _, cp := range c.Pulls {
okey, err := delivery.CheckerOwedKey(cp.Repository, cp.Number, cp.Head.Head)
if err != nil {
ch.logf("%s: %v", key, err)
continue
}
o := delivery.Owed{Act: delivery.OwedStatus, Context: delivery.StatusContextCheck, State: "error",
Target: strings.TrimSuffix(ch.Link, "/") + "/" + key, Text: clipLine(words, MaxLine)}
data, err := delivery.EncodeCheckerOwed(o)
if err != nil {
ch.logf("%s: the status of %s cannot be written: %v", key, okey, err)
continue
}
wrote, err := putOnce(ctx, ch.Owed, okey, data, func(cur []byte) bool {
held, err := delivery.DecodeCheckerOwed(cur)
o.V = delivery.Version
return err == nil && delivery.SameOwed(held, o)
})
if err != nil {
ch.logf("%s: the status of %s cannot be written: %v", key, okey, err)
continue
}
if wrote {
said = true
if ch.Announce != nil {
ch.Announce(ctx, EventOwedWritten, okey)
}
}
}
if said {
names := make([]string, 0, len(left))
for _, p := range left {
names = append(names, fmt.Sprintf("%s#%d@%s", p.Repository, p.Number, short12(p.Head)))
}
ch.logf("%s: %s %s; the verdict and every status of its group say so (error), and it is judged once the trunk head is known", key, strings.Join(names, ", "), words)
}
}
// The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on
// the build seats' check stream, a head asked again only because its trunk moved on the recheck stream, pulled after.
// the build seats' check stream; a carry, a verdict moved to a newer trunk or sibling commit by its package digests (ADR 0548
// §5), on the recheck stream, pulled after.
const (
KindNew = "new"
KindNodes = "nodes"
KindTrunk = "trunk"
)
// kindOf is the kind of a case's check, read from its verdict key: none (or one left running) is a head never judged; a
// failure of the nodes is asked again after the nodes changed; a verdict on other trunk heads is a re-judge after a trunk
// move. A verdict that stands is ranked with the re-judges: its claim finds nothing to do.
func (ch *Checker) kindOf(ctx context.Context, cs *Case) string {
// prepare reads the case's verdict key and says what its check is (ADR 0548 §1, §5, §7): none (or one left running) is a
// head never judged, a whole check; a failure of the nodes is asked again after the nodes changed; a verdict judged on trunk
// or sibling commits that moved since is carried, its packages run only where the move changed their digest, and a pass's
// commits are named as the carry's CarriedFrom. A verdict that stands is ranked with the carries: its claim finds nothing
// to do.
func (ch *Checker) prepare(ctx context.Context, cs *Case) {
cs.Kind, cs.CarriedFrom, cs.Composition = KindNew, nil, nil
value, _, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return KindNew
return
}
v, err := delivery.DecodeVerdict(value)
switch {
case err != nil || v.State == delivery.VerdictRunning:
return KindNew
return
case IsNodesFailure(v):
return KindNodes
cs.Kind = KindNodes
return
case IsConflictFailure(v):
return // a head git refused: its next check is whole (ADR 0548 §7)
case IsEnded(v), IsUntrunked(v):
return // ended unjudged when its heads went (issue 1890), or never judged for want of a trunk head (issue 1422): checked whole
}
return KindTrunk
cs.Kind = KindTrunk
if v.State == delivery.VerdictPass && ch.stale(v, cs) {
cs.CarriedFrom = judgedOn(v)
if c := v.Composition; c != nil && c.State == delivery.CompositionPass && c.FPSet != "" && c.FPSet == cs.FPSet {
kept := *c
cs.Composition = &kept
}
}
}
// stale says whether a verdict was judged on commits that are no longer the case's: a trunk it merged moved, or a sibling it
// read at its trunk's head (ADR 0548 §6).
func (ch *Checker) stale(v delivery.Verdict, cs *Case) bool {
return !sameTrunk(v.Trunk, cs.Trunk) || besideMoved(v.Beside, ch.besideNow())
}
// besideNow are the commits a sibling is read at now, by repository: its trunk head, and for the node-engine the commit the
// mesh runs when the indexer names one, as besideFor places them.
func (ch *Checker) besideNow() map[string]string {
return besideAt(ch.heads(), ch.Runs)
}
// besideAt places the commits a sibling is read at: the trunk heads, with the node-engine at the commit the mesh runs when
// runs names one (ADR 0548 §6). The checker's decision and the status read the same commits through it.
func besideAt(now map[string]string, runs interface{ CommitOf(module string) string }) map[string]string {
if runs != nil {
if c := runs.CommitOf(nodeEngineModule); c != "" {
out := make(map[string]string, len(now)+1)
for r, h := range now {
out[r] = h
}
out[nodeEngineRepo] = c
return out
}
}
return now
}
// besideMoved says whether a sibling read at these commits moved: the commit it is read at now is known and another.
func besideMoved(beside, now map[string]string) bool {
for r, c := range beside {
if h := now[r]; h != "" && h != c {
return true
}
}
return false
}
// judgedOn are the trunk and sibling commits a verdict was judged on, by repository.
func judgedOn(v delivery.Verdict) map[string]string {
out := make(map[string]string, len(v.Trunk)+len(v.Beside))
for r, c := range v.Beside {
out[r] = c
}
for r, c := range v.Trunk {
out[r] = c
}
return out
}
// carried is what a carry's verdict names as carried from (ADR 0548 §5): the commits it was judged on before, of the
// repositories the new verdict still judges or reads; nil when none of them moved.
func carried(from map[string]string, v delivery.Verdict) map[string]string {
out, moved := map[string]string{}, false
for r, c := range from {
now, ok := v.Trunk[r]
if !ok {
now, ok = v.Beside[r]
}
if ok && now != "" && c != "" {
out[r] = c
moved = moved || now != c
}
}
if !moved {
return nil
}
return out
}
// moveWords say a carry's moves, for the words of a failure: "carried from <old12> to <new12> of <repository>", each moved
// repository, sorted.
func moveWords(from map[string]string, v delivery.Verdict) string {
var moves []string
for r, c := range from {
now, ok := v.Trunk[r]
if !ok {
now = v.Beside[r]
}
if now != "" && now != c {
moves = append(moves, fmt.Sprintf("%s %s to %s", r, short12(c), short12(now)))
}
}
sort.Strings(moves)
return strings.Join(moves, ", ")
}
func (ch *Checker) candidate(key string, fallback delivery.Candidate) delivery.Candidate {
@@ -265,6 +622,9 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return nil, fmt.Errorf("%s: %w", cs.Key, err)
}
if !found {
if ch.heldByTurn(cs) {
return nil, nil
}
rev, err := ch.Verdicts.Create(ctx, cs.Key, data)
if errors.Is(err, ErrConflict) {
return nil, nil // another checker made it first
@@ -285,15 +645,15 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil
}
return nil, nil
case !sameTrunk(cur.Trunk, cs.Trunk):
retake = true
case IsEnded(cur), ch.stale(cur, cs):
retake = true // heads held again after their check was ended unjudged (issue 1890), or a trunk that moved
case cur.State == delivery.VerdictFail && IsNodesFailure(cur):
ch.mu.Lock()
at, ok := ch.nodesFail[cs.Key]
ch.mu.Unlock()
retake = !ok || epoch > at
}
if !retake {
if !retake || ch.heldByTurn(cs) {
return nil, nil
}
rev, err = ch.Verdicts.Update(ctx, cs.Key, data, rev)
@@ -306,6 +666,59 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil
}
// heldByTurn says whether a merge turn holds the case's check off (ADR 0548 §3): an approved merge waits on one of the
// repositories it merges heads into, and it is not the merge of the case's own heads. The check is claimed when the turn's
// key goes (Reconsider on the turns watch). Said once per turn in the log. A check already claimed is never held: one
// claimed in the moment before the gate wrote its turn is judged on the commit it read and carried once after.
func (ch *Checker) heldByTurn(cs *Case) bool {
if ch.Turns == nil {
return false
}
repos := make([]string, 0, len(cs.Trunk))
for r := range cs.Trunk {
repos = append(repos, r)
}
repo, w, held := ch.Turns.Holding(repos, cs.Heads)
why := ""
if held {
why = repo + " " + w.Group + " " + w.Heads
}
ch.mu.Lock()
said := ch.held[cs.Key]
if why == "" {
delete(ch.held, cs.Key)
} else {
ch.held[cs.Key] = why
}
ch.mu.Unlock()
if held && said != why {
ch.logf("%s: not claimed: the approved merge of %s (heads %s) holds a turn on %s, so its check waits for that merge (ADR 0548 §3)",
cs.Key, w.Group, w.Heads, repo)
}
return held
}
// Held is every check a merge turn holds off now, by verdict key, with the turn's repository, group and heads.
func (ch *Checker) Held() map[string]string {
ch.init()
ch.mu.Lock()
defer ch.mu.Unlock()
out := make(map[string]string, len(ch.held))
for k, v := range ch.held {
out[k] = v
}
return out
}
// ConflictWords are what the words of a head that git would not merge onto its trunk say before the files in conflict.
const ConflictWords = "does not merge into the trunk: files in conflict: "
// IsConflictFailure says whether a verdict failed because its head did not merge onto its trunk (ADR 0548 §7): it has no
// pass to carry, so its next check, at a push or at a later trunk move, is a whole check of its head.
func IsConflictFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, ConflictWords)
}
// IsNodesFailure says whether a verdict failed because a part of the mesh the check reads was not heard (UnheardWords):
// it is judged again when the nodes change. No check fails for want of a build node any more: its asks wait in the build
// seats' streams (ADR 0525 §6, issue 1912).
@@ -313,9 +726,13 @@ func IsNodesFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, UnheardWords)
}
// try claims the check, judges it within its bound and writes the verdict and the owed statuses. A check that ends
// because ctx did is left running: the checker takes it again when it starts.
func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
// try claims the check, judges it within its bound under run and writes the verdict and the owed statuses. A check that ends
// because ctx did is left running: the checker takes it again when it starts. A check whose run was ended because no
// candidate holds its heads any more (issue 1890) is ended: its verdict says so, and no status is written.
func (ch *Checker) try(ctx, run context.Context, key string, cs *Case) {
if run.Err() != nil {
return // ended before it was claimed: nothing to write
}
ch.mu.Lock()
first := !ch.tried[cs.Key]
ch.tried[cs.Key] = true
@@ -330,13 +747,58 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
return
}
began := ch.now()
if cs.Watch == nil && ch.Asks != nil {
cs.Watch = func(_ string, step *delivery.Step) {
if step == nil { // put: kept before its wait starts
ch.recordAsks(ctx, cs.Key, cs.Asks())
}
}
defer ch.forgetAsks(ctx, cs.Key)
}
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
out := ch.judge(ctx, cs)
if ctx.Err() != nil {
out := ch.judge(run, cs)
// An ask the build seat gave up on, or whose step the build module's next process ended because it was replaced under
// it, judged nothing (issue 734, IsLostAsk): the check is asked again under the same claim, its status pending, and after
// LostAskTries it fails as not heard, judged again when the nodes change. A step that ended while its watch was deaf is
// read at the next nudge (HeardEnds) and judged.
for try := 1; IsLostAsk(out.Verdict) && run.Err() == nil; try++ {
if try >= LostAskTries {
out.Verdict.Failed.Words = strings.TrimSpace(out.Verdict.Failed.Words + "\nasked " + fmt.Sprint(LostAskTries) +
" times and given up each time: " + UnheardWords)
break
}
ch.logf("%s: %s; asked again (%d of %d)", cs.Key, out.Verdict.Failed.Words, try+1, LostAskTries)
out = ch.judge(run, cs)
}
if ctx.Err() != nil || errors.Is(context.Cause(run), errAskedAgain) {
return // the checker stops, or the verdict was asked again by hand, which wrote it: its next check claims it anew
}
var why superseded
if errors.As(context.Cause(run), &why) {
ch.end(ctx, cs, cl, why)
return
}
v := out.Verdict
v.Group, v.Heads, v.Trunk = cs.Group, cs.Heads, cs.Trunk
v.Group, v.Heads, v.Trunk, v.Beside = cs.Group, cs.Heads, cs.Trunk, nil
for r, c := range out.Beside {
if _, merged := cs.Trunk[r]; !merged && c != "" {
if v.Beside == nil {
v.Beside = map[string]string{}
}
v.Beside[r] = c
}
}
if cs.CarriedFrom != nil {
// A carry (ADR 0548 §5): a pass names the commits it was carried from; a failure names the move it failed on.
switch v.State {
case delivery.VerdictPass:
v.CarriedFrom = carried(cs.CarriedFrom, v)
case delivery.VerdictFail:
if moves := moveWords(cs.CarriedFrom, v); moves != "" {
v.Failed.Words = strings.TrimSpace(v.Failed.Words + "\nfailed on the carry over the move of " + moves + ": the move changed what these tests read")
}
}
}
if v.Took.Queue == 0 && !cs.Queued.IsZero() && began.After(cs.Queued) {
v.Took.Queue = began.Sub(cs.Queued).Milliseconds()
}
@@ -350,25 +812,50 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
}
}
if _, err := ch.Verdicts.Update(ctx, cs.Key, data, cl.rev); err != nil {
ch.logf("%s: the verdict could not be written: %v", cs.Key, err)
return
if _, _, found, gerr := ch.Verdicts.Get(ctx, cs.Key); gerr == nil && !found {
_, err = ch.Verdicts.Create(ctx, cs.Key, data) // its key was deleted while it ran (issue 1733): written anew
}
if err != nil {
ch.logf("%s: the verdict could not be written: %v", cs.Key, err)
return
}
}
if IsNodesFailure(v) {
ch.mu.Lock()
ch.nodesFail[cs.Key] = epoch
ch.mu.Unlock()
}
if ch.movedSince(cs) {
if ch.movedSince(cs) || besideMoved(v.Beside, ch.besideNow()) {
// Judged on a trunk that moved while it ran: the verdict is kept as judged, and the status stays pending until the
// check of the new trunk, which Reconsider starts, ends.
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending", cs.Key, v.State)
ch.owePending(ctx, cs, v)
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending until it is carried", cs.Key, v.State)
return
}
ch.writeOwed(ctx, cs, v, out)
ch.logf("%s: %s", cs.Key, v.State)
}
// end writes the verdict of a check whose heads no candidate holds any more (issue 1890), by compare-and-set on its claim,
// and says it once in the log with the asks it had put: they stay in the build seats' streams, since an ask once put cannot
// be taken back (buildrun.Asker has no withdraw), and are no longer waited for. No status is written: the heads that
// replaced these have their own check, whose pending status stands.
func (ch *Checker) end(ctx context.Context, cs *Case, cl *claimed, why superseded) {
v := delivery.Verdict{Group: cs.Group, Heads: cs.Heads, Trunk: cs.Trunk, State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Machine: "checker", Words: why.Error()}}
data, err := delivery.EncodeVerdict(v)
if err == nil {
_, err = ch.Verdicts.Update(ctx, cs.Key, data, cl.rev)
}
if err != nil {
ch.logf("%s: %s, but the verdict could not say so: %v", cs.Key, why.Error(), err)
return
}
stage, _ := cs.Progress.Now()
asks := cs.Asks()
ch.logf("%s: %s; it was at %q with %d asks put %v, which are no longer waited for", cs.Key, why.Error(), stage, len(asks), asks)
}
// judge runs the case within the bound. A judge that fails to judge, and a check that ran past its bound, are verdicts of
// failure naming the machine and the step.
func (ch *Checker) judge(ctx context.Context, cs *Case) Outcome {
@@ -467,7 +954,7 @@ func (ch *Checker) Sweep(ctx context.Context) error {
continue
}
cur.State = delivery.VerdictFail
cur.Failed = delivery.FailedFirst{Machine: "checker", Words: "closed at start: left running, and no candidate holds these heads any more, so nothing will check them"}
cur.Failed = delivery.FailedFirst{Machine: "checker", Words: EndedWords + "closed at start: left running, and no candidate holds these heads any more, so nothing will check them"}
data, err := delivery.EncodeVerdict(cur)
if err != nil {
return err
@@ -502,14 +989,37 @@ func (ch *Checker) TrunkMoved(ctx context.Context) {
if err != nil || !found {
continue // never judged: no status to put out of date
}
if v, err := delivery.DecodeVerdict(value); err == nil && sameTrunk(v.Trunk, cs.Trunk) {
v, err := delivery.DecodeVerdict(value)
if err == nil && !ch.stale(v, cs) {
continue
}
if err == nil && v.State == delivery.VerdictPass {
cs.CarriedFrom = judgedOn(v) // the status says it is being carried to the new commit (ADR 0548 §6)
}
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
ch.Reconsider(ctx)
}
// owePending owes mesh/check pending on the heads of a verdict judged on commits that moved while it ran: a pass says it is
// being carried to the trunk's head now (ADR 0548 §6), anything else that it is checked again.
func (ch *Checker) owePending(ctx context.Context, judged *Case, v delivery.Verdict) {
next := judged
ckey, err := delivery.CandidatesKey(v.Group)
ch.mu.Lock()
c, ok := ch.known[ckey]
ch.mu.Unlock()
if err == nil && ok {
if now := CaseOf(c, ch.heads()); now != nil && now.Key == judged.Key {
next = now
}
}
if v.State == delivery.VerdictPass {
next.CarriedFrom = judgedOn(v)
}
ch.writeOwed(ctx, next, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
// movedSince says whether a trunk the case was judged on is no longer the trunk's head.
func (ch *Checker) movedSince(cs *Case) bool {
now := ch.heads()
@@ -3,6 +3,7 @@ package check
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
@@ -238,14 +239,119 @@ func TestAnUnheardForgeFailsWithTheReasonAndIsJudgedAgainWhenTheNodesChange(t *t
}
}
func TestARepositoryWithNoTrunkGetsNoVerdict(t *testing.T) { // N6/AC6, ADR 0484 clause 11
// Issue 1422: a pull request of a repository the gate's source.main names no trunk head of (one never merged through the
// gate) is never dropped silently. It is not judged, its verdict says it cannot be judged and why, its status is error with
// those words, checker_status names it, and the log says it once per head. Once its trunk head is known it is judged whole;
// in a group, the members whose trunk is known are judged and the others' status says why they are not.
func TestARepositoryWithNoTrunkCannotBeJudgedLoudlyUntilItsTrunkIsKnown(t *testing.T) { // N6/AC6, ADR 0484 clause 11, 1422
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
r.ch.Candidate(ctx, "feat/y", candidateOf(t, "feat/y", pullOf("novox/unknown", 4, head1)))
var logged []string
r.ch.Logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
const unknown = "novox/unknown"
cand := candidateOf(t, "feat/y", pullOf(unknown, 4, head1))
key := keyOf(t, "feat/y", cand)
r.ch.Candidate(ctx, "feat/y", cand)
r.ch.Wait()
if j.count() != 0 || len(r.verdicts.vals) != 0 || len(r.owed.vals) != 0 {
t.Errorf("judged %d times, %d verdicts, %d statuses, want none", j.count(), len(r.verdicts.vals), len(r.owed.vals))
r.ch.Reconsider(ctx) // looked at again: nothing is written or said twice
r.ch.NodesChanged(ctx)
r.ch.Wait()
v := r.verdict(t, key)
if j.count() != 0 || !IsUntrunked(v) || !strings.Contains(v.Failed.Words, "the trunk head of "+unknown+" is not known") {
t.Fatalf("judged %d times, verdict %+v, want none judged and a verdict that says it cannot be judged", j.count(), v)
}
if s := owedState(t, r, unknown, 4, head1); s != "error" {
t.Errorf("%s#4 owes %q, want error", unknown, s)
}
if n := countWith(logged, "cannot be judged"); n != 1 {
t.Errorf("said %d times in the log, want once for the head: %q", n, logged)
}
reader := func() Reader {
ckey, _ := delivery.CandidatesKey("feat/y")
return Reader{CandidateState: fakeState{ckey: cand}.read(), VerdictState: fakeState(r.verdicts.vals).read(),
Trunks: func() (map[string]string, error) { return r.trunks.Heads(), nil }}
}
st, err := reader().Status()
if err != nil {
t.Fatal(err)
}
if why := st.CannotBeJudged["feat/y"]; !strings.Contains(why, unknown) || len(st.Waiting[KindNew]) != 0 || len(st.Unjudged) != 0 {
t.Errorf("cannot_be_judged %v, waiting %v, unjudged %v: want feat/y named as cannot be judged, not as new", st.CannotBeJudged, st.Waiting, st.Unjudged)
}
if rows, _, err := reader().Verdicts("feat/y"); err != nil || len(rows) != 1 || !strings.Contains(rows[0].Reports[unknown+"#4"], "cannot be judged") {
t.Errorf("checker_verdicts %+v (%v), want the report of %s#4 to say it cannot be judged", rows, err, unknown)
}
// Its trunk head becomes known: the heads are judged whole, the same key rewritten, the status success.
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, unknown: trunkA2})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if v := r.verdict(t, key); j.count() != 1 || v.State != delivery.VerdictPass || v.Trunk[unknown] != trunkA2 {
t.Errorf("judged %d times, verdict %+v, want one pass on %s", j.count(), v, trunkA2)
}
if s := owedState(t, r, unknown, 4, head1); s != "success" {
t.Errorf("%s#4 owes %q once its trunk is known, want success", unknown, s)
}
// Issue 1733: the gate asks the Approve only on a verdict, so no candidate is left without one. A group with a member
// whose trunk is not known is not judged in part: its verdict fails naming that repository, and every status says so; a
// running verdict left for heads that cannot be judged is replaced; a verdict key deleted is judged again on its delete.
for _, row := range []struct {
name string
pulls []delivery.CandidatePull
running bool // a running verdict is left under the key first
deleted bool // the key is deleted after its verdict
judged int
want string // the verdict's state, and for a failure the repository it names
}{
{name: "a member the checker cannot claim", pulls: []delivery.CandidatePull{pullOf(repoA, 9, head2), pullOf("novox/other", 2, head2)}, want: "novox/other"},
{name: "a running verdict left for heads that cannot be judged", pulls: []delivery.CandidatePull{pullOf(unknown, 5, head2)}, running: true, want: unknown},
{name: "a verdict key deleted", pulls: []delivery.CandidatePull{pullOf(repoA, 5, head2)}, deleted: true, judged: 2, want: delivery.VerdictPass},
} {
t.Run(row.name, func(t *testing.T) {
j := passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/z", row.pulls...)
key := keyOf(t, "feat/z", cand)
if row.running {
group, heads, _ := delivery.ParseCheckerVerdictKey(key)
raw, _ := delivery.EncodeVerdict(delivery.Verdict{Group: group, Heads: heads, State: delivery.VerdictRunning})
_, _ = r.verdicts.Create(ctx, key, raw)
}
r.ch.Candidate(ctx, "feat%2Fz", cand)
r.ch.Wait()
if row.deleted {
_ = r.verdicts.Delete(ctx, key)
r.ch.VerdictDeleted(ctx, key) // the watch of the verdicts hands the delete over
r.ch.Wait()
}
v := r.verdict(t, key)
if row.want == delivery.VerdictPass {
if v.State != delivery.VerdictPass || j.count() != row.judged {
t.Errorf("verdict %s after %d judgings, want a pass after %d", v.State, j.count(), row.judged)
}
return
}
if !IsUntrunked(v) || !strings.Contains(v.Failed.Words, row.want) || j.count() != 0 {
t.Errorf("verdict %+v after %d judgings, want a failure naming %s and none judged", v, j.count(), row.want)
}
for _, p := range row.pulls {
if s := owedState(t, r, p.Repository, p.Number, p.Head.Head); s != "error" {
t.Errorf("%s#%d owes %q, want error", p.Repository, p.Number, s)
}
}
})
}
}
func countWith(lines []string, words string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, words) {
n++
}
}
return n
}
// 1887/A10, ADR 0527 §2: hq is judged as any repository, its checks Go tests, and its pull request gets a verdict and a
@@ -333,7 +439,10 @@ func TestTheStatusIsPendingWhileTheCheckRunsAndEndsSuccessOrFailure(t *testing.T
}
}
func TestSweepClosesARunningKeyNoCandidateHoldsAndLeavesOneItDoes(t *testing.T) {
// No check of heads no candidate holds is left running (issue 1890): at start the sweep ends a key left running; while the
// checker runs, a check whose group moved to other heads is ended at once, unjudged, and writes no status; heads held again
// are checked whole; and a group's check starts only once its heads settled, so a partial set of members is never checked.
func TestHeadsNoCandidateHoldsAreNeverLeftRunning(t *testing.T) {
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
@@ -351,12 +460,91 @@ func TestSweepClosesARunningKeyNoCandidateHoldsAndLeavesOneItDoes(t *testing.T)
if err := r.ch.Sweep(ctx); err != nil {
t.Fatal(err)
}
if v := r.verdict(t, dead); v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "closed at start") {
if v := r.verdict(t, dead); !IsEnded(v) || !strings.Contains(v.Failed.Words, "closed at start") {
t.Errorf("the dead key = %+v", v)
}
if v := r.verdict(t, live); v.State != delivery.VerdictRunning {
t.Errorf("the live key = %+v, want it left for the check to take over", v)
}
// A member joins while the check of the lone member's heads runs: that check is ended, its judge's run cancelled.
started, ended := make(chan struct{}), make(chan error, 1)
j = &judgeFunc{f: func(run context.Context, _ *Case, call int) (Outcome, error) {
if call == 1 {
close(started)
<-run.Done()
ended <- context.Cause(run)
return Outcome{}, run.Err()
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
var logged []string
var logMu sync.Mutex
r.ch.Logf = func(f string, a ...any) { logMu.Lock(); logged = append(logged, fmt.Sprintf(f, a...)); logMu.Unlock() }
alone, both := cand, candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(repoSDK, 3, head2))
r.ch.Candidate(ctx, "feat%2Fx", alone)
<-started
r.ch.Candidate(ctx, "feat%2Fx", both)
var bothC delivery.Candidate
_ = json.Unmarshal(both, &bothC)
select {
case err := <-ended:
if err == nil || !strings.Contains(err.Error(), "superseded by "+GateHeads(bothC)) {
t.Errorf("the old check's run ended with %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("the check of the heads the candidate left still runs")
}
r.ch.Wait()
if v := r.verdict(t, live); !IsEnded(v) || v.Failed.Words != EndedWords+"superseded by "+GateHeads(bothC) {
t.Errorf("the superseded verdict = %+v", v.Failed)
}
if v := r.verdict(t, keyOf(t, "feat/x", both)); v.State != delivery.VerdictPass {
t.Errorf("the group's verdict = %s", v.State)
}
okey, _ := delivery.CheckerOwedKey(repoA, 7, head1)
if raw, _, _, _ := r.owed.Get(ctx, okey); !strings.Contains(string(raw), `"success"`) {
t.Errorf("the head's status = %s, want the group's success, never the ended check's", raw)
}
logMu.Lock()
said := 0
for _, l := range logged {
if strings.Contains(l, "ended: superseded") {
said++
}
}
logMu.Unlock()
if said != 1 {
t.Errorf("the end was said %d times: %v", said, logged)
}
// The member leaves again: the ended heads are held again and checked whole.
r.ch.Candidate(ctx, "feat%2Fx", alone)
r.ch.Wait()
if v := r.verdict(t, live); v.State != delivery.VerdictPass || j.count() != 3 {
t.Errorf("the heads held again = %s after %d judgings, want a pass of a third", v.State, j.count())
}
// Members announced one after another: only the settled, whole set is checked.
j = passJudge()
r = newRig(t, j)
r.ch.Settle = time.Hour
sdk := candidateOf(t, "feat/y", pullOf(repoSDK, 4, head2))
whole := candidateOf(t, "feat/y", pullOf(repoSDK, 4, head2), pullOf(repoA, 8, head1))
r.ch.Candidate(ctx, "feat%2Fy", sdk)
r.ch.Candidate(ctx, "feat%2Fy", whole)
r.ch.Reconsider(ctx) // a trunk move or a turn while the heads settle starts nothing
if j.count() != 0 {
t.Fatalf("judged %d times before the heads settled", j.count())
}
r.ch.settleNow()
r.ch.Wait()
if _, _, found, _ := r.verdicts.Get(ctx, keyOf(t, "feat/y", sdk)); found || j.count() != 1 {
t.Errorf("the partial set has a verdict (%v) or the judge ran %d times, want the whole set alone", found, j.count())
}
if v := r.verdict(t, keyOf(t, "feat/y", whole)); v.State != delivery.VerdictPass {
t.Errorf("the whole set = %s", v.State)
}
}
func TestTheForgeBridgeIsToldOnceForEachChangeOfAStatus(t *testing.T) { // 1683 (the checker's half of AC5)
@@ -28,6 +28,7 @@ func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
f := &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "v0.1.14-x", Machines: []string{"g14"}, Took: 40}}
b.ex.Compositions = f
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.FPSet = "fps-of-the-candidate"
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionPass || v.Composition.SDK != "v0.1.14-x" || v.Took.Composition != 40 {
t.Fatalf("verdict = %+v, composition %+v", v, v.Composition)
@@ -38,6 +39,15 @@ func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
if !compositionMentions(v, "passed") {
t.Errorf("the line says %q", compositionWords(v.Composition, v.Took.Composition))
}
if v.Composition.FPSet == "" || v.Composition.FPSet != cs.FPSet {
t.Errorf("the composition names the fingerprint set %q, want the candidate's %q (ADR 0548 §5)", v.Composition.FPSet, cs.FPSet)
}
// ADR 0548 §5: a carry whose fingerprints did not change keeps the composition it passed and asks the assigner nothing.
carry := caseOf(t, pullOf(repoA, 7, head1))
carry.CarriedFrom, carry.Composition = map[string]string{repoA: trunkA2}, v.Composition
if kept := b.judge(t, carry).Verdict; kept.Composition == nil || kept.Composition.SDK != "v0.1.14-x" || len(f.asked) != 1 {
t.Errorf("the carry's composition is %+v and the assigner was asked %d times; want the kept one and no second ask", kept.Composition, len(f.asked))
}
}
func compositionMentions(v delivery.Verdict, word string) bool {
@@ -0,0 +1,90 @@
package check
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// failingResult is a level's test result in which the named tests failed, each in its own package.
func failingResult(t *testing.T, tests int, failed ...Failure) json.RawMessage {
t.Helper()
raw, err := json.Marshal(Tested{Level: delivery.LevelIntegration, Tests: tests, Elapsed: 2, Failed: failed, Passed: len(failed) == 0})
if err != nil {
t.Fatal(err)
}
return raw
}
func judgeLevel(t *testing.T, tests ...json.RawMessage) (*delivery.Verdict, *state) {
t.Helper()
st := &state{j: &Examiner{}, c: &Case{}, levels: map[string]*acc{delivery.LevelUnit: {}, delivery.LevelIntegration: {}}, nodes: map[string]bool{}}
v, _ := st.level(t.Context(), &target{pull: Pull{Repository: "novox/x"}}, delivery.LevelAnswer{Level: delivery.LevelIntegration, Tests: tests})
return v, st
}
// Issue 1921: two failing tests in two packages are two entries of the verdict's Failures and two lines of a try's
// failed list, each with its own first line (its cause, issue 1901), whether the build module answered them in one test
// result (one go test run over both packages) or in two (the changed packages first, then the rest); every result is
// counted, the ones after a failure too; and Failed keeps the first one's words whole.
func TestTwoFailingTestsInTwoPackagesAreTwoEntries(t *testing.T) {
a := Failure{Package: "x/a", Test: "TestA", Cause: "a_test.go:9: want 1, got 2", Output: " a_test.go:3: logged\n a_test.go:9: want 1, got 2\n--- FAIL: TestA (0.00s)"}
b := Failure{Package: "x/b", Test: "TestB", Cause: "b_test.go:4: boom", Output: " b_test.go:4: boom\n--- FAIL: TestB (0.00s)"}
for name, tests := range map[string][]json.RawMessage{
"one result": {failingResult(t, 10, a, b)},
"two results": {failingResult(t, 4, a), failingResult(t, 3), failingResult(t, 3, b)},
} {
v, st := judgeLevel(t, tests...)
if v == nil {
t.Fatalf("%s: the level passed", name)
}
want := []string{"integration: x/a.TestA: a_test.go:9: want 1, got 2", "integration: x/b.TestB: b_test.go:4: boom"}
if got := failedLines(*v); strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("%s: failed lines\n%s\nwant\n%s", name, strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if len(v.Failures) != 2 || v.Failed.Test != "integration: x/a.TestA" || !strings.Contains(v.Failed.Words, "a_test.go:3: logged") {
t.Errorf("%s: verdict %+v", name, v)
}
if run := st.levels[delivery.LevelIntegration].run; run != 10 {
t.Errorf("%s: %d tests counted, want all 10", name, run)
}
if _, err := delivery.EncodeVerdict(delivery.Verdict{Group: "g", Heads: "0123456789ab", State: v.State, Failed: v.Failed, Failures: v.Failures}); err != nil {
t.Errorf("%s: the verdict is not written: %v", name, err)
}
}
}
// Issue 1921: past delivery.MaxFailures the list names the first fifty and how many more, the status line says how many
// failed, and each entry's line is cut to MaxFailureLine.
func TestFailuresAreCappedWithHowManyMore(t *testing.T) {
var fs []Failure
for i := range 60 {
fs = append(fs, Failure{Package: fmt.Sprintf("x/p%d", i), Test: "TestT", Cause: strings.Repeat("y", 500)})
}
v, _ := judgeLevel(t, failingResult(t, 60, fs...))
if len(v.Failures) != delivery.MaxFailures+1 || v.Failures[delivery.MaxFailures].Test != "and 10 more" {
t.Fatalf("%d entries, last %+v", len(v.Failures), v.Failures[len(v.Failures)-1])
}
if len(v.Failures[0].Words) > MaxFailureLine {
t.Errorf("an entry's line is %d bytes", len(v.Failures[0].Words))
}
if n := failedCount(*v); n != 60 {
t.Errorf("failedCount = %d, want 60", n)
}
cs := &Case{Pulls: []Pull{{Repository: "novox/x", Head: "0123456789abcdef", Trunk: "fedcba9876543210"}}}
if line := Line(*v, cs, cs.Pulls[0], nil, nil, nil); !strings.HasPrefix(line, "failed (60 tests): integration: x/p0.TestT") {
t.Errorf("line %q", line)
}
}
// A verdict that names no list (a conflict, a refused step, a verdict written before issue 1921) is one failed line, as
// before.
func TestAVerdictWithoutFailuresIsOneLine(t *testing.T) {
v := delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: "g14", Words: "files in conflict: go.mod\nmore"}}
if got := failedLines(v); len(got) != 1 || got[0] != "g14: files in conflict: go.mod" {
t.Fatalf("%q", got)
}
}
@@ -5,6 +5,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
@@ -116,6 +117,9 @@ type bench struct {
fetch Fetched
digest map[string]Digested // by level
test func(c stepCall) (any, error)
// declines is how many asks naming needs a node lacking them declines before shanks, which has every need, takes
// one (issue 890); an ask naming none runs on shanks, which then has none.
declines int
}
// refsOf reads the siblings of a fetch's arguments, as they arrive in an ask (JSON) or as the checker gave them.
@@ -151,8 +155,17 @@ func (b *bench) answer(a delivery.StepAsk) delivery.Step {
if module == buildrun.GoBuildModule {
steps.Digest = tool("digest")
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
began := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
if len(a.Needs) > 0 {
if b.declines > 0 {
b.declines--
result, _ := json.Marshal(delivery.StepDeclined{Declined: true, Needs: a.Needs, Lacks: a.Needs})
return delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: "ace", State: delivery.StepRefused, Started: began, Ended: began,
Reason: "declined", Result: result}
}
steps.Has = func(string) bool { return true }
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed, Started: began, Ended: began.Add(2 * time.Second)}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
@@ -260,8 +273,36 @@ func TestACleanMergeIsFetchedAtItsTrunkAndBothLevelsPass(t *testing.T) { // 1679
t.Errorf("%s was not kept", k)
}
}
// ADR 0567 §12: every check ask carries every node's name for the names check; a check whose names cannot be read, or
// that reads none, is not asked and fails saying so.
for _, c := range []struct {
names []string
err error
fails string
}{{[]string{"anchor", "laptop"}, nil, ""}, {nil, nil, "no node record"}, {nil, errors.New("no bucket"), "no bucket"}} {
b := newBench(t)
b.ex.Nodes = namesOf{c.names, c.err}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
asks := b.asks.Asks()
if c.fails != "" {
if out.Verdict.State != delivery.VerdictFail || len(asks) != 0 || !strings.Contains(out.Verdict.Failed.Words, c.fails) {
t.Errorf("names %v, %v: verdict %+v, %d asks", c.names, c.err, out.Verdict, len(asks))
}
} else if len(asks) == 0 || strings.Join(asks[0].Check.NodeNames, ",") != "anchor,laptop" {
t.Errorf("the check ask carries the names %v", asks)
}
}
}
// namesOf is the nodes' names as a test gives them.
type namesOf struct {
names []string
err error
}
func (n namesOf) Names(context.Context) ([]string, error) { return n.names, n.err }
func TestOnlyThePackageWithoutAKeptVerdictRuns(t *testing.T) { // 1288/AC1
b := newBench(t)
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"fixture/slow": digA, "fixture/other": digC}}
@@ -276,6 +317,49 @@ func TestOnlyThePackageWithoutAKeptVerdictRuns(t *testing.T) { // 1288/AC1
if u := out.Verdict.Levels[delivery.LevelUnit]; u.Run != 2 || u.Kept != 40 {
t.Errorf("unit = %+v, want 2 run and 40 kept", u)
}
// Issue 890/AC5: a package needing node-build-node is left unmet where the node lacks it; the check keeps the passes
// and asks the integration level again for a node with it, puts it again when a node declines it, and that node tests
// the package alone, its kept passes not counted twice.
// fixture/outside has no kept verdict after its pass (it reads outside itself: its digest moves every call), so a
// placed ask that ran more than its unmet packages would run and count it again.
b = newBench(t)
b.declines = 1
outside := 0
b.steps.handlers["build_go_digest"] = func(c stepCall) (any, error) {
d := echoBeside(b, b.digest[c.args["level"].(string)])
if d.Level == "integration" {
outside++
d.Packages = map[string]string{"fixture/slow": digB, "fixture/node": digC, "fixture/outside": fmt.Sprintf("%064x", outside)}
}
return d, nil
}
b.digest[delivery.LevelIntegration] = Digested{Level: "integration",
Needs: []delivery.PackageNeeds{{Package: "fixture/node", Needs: []string{delivery.NeedNodeBuildNode}}}}
out = b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
asks := b.asks.Asks()
if out.Verdict.State != delivery.VerdictPass || len(asks) != 3 || len(asks[0].Needs) != 0 {
t.Fatalf("verdict %+v, asks %+v: want a pass of three asks, the first naming no need", out.Verdict, asks)
}
for _, a := range asks[1:] {
if strings.Join(a.Needs, ",") != delivery.NeedNodeBuildNode || strings.Join(a.Check.Levels, ",") != delivery.LevelIntegration || strings.Join(a.Check.Packages, ",") != "fixture/node" {
t.Errorf("a placed ask %+v, want the integration level needing node-build-node", a)
}
}
tests = b.steps.tool("test")
if last := tests[len(tests)-1]; len(tests) != 3 || strings.Join(packagesOf(last), ",") != "fixture/node" || last.args["level"] != "integration" {
t.Errorf("test calls = %+v, want the unit level, the integration level of fixture/slow and fixture/outside, then fixture/node alone", tests)
}
slow := captured[Tested](t, "go-test-slow")
if i := out.Verdict.Levels[delivery.LevelIntegration]; i.Run != 2*slow.Tests || i.Kept != 0 {
t.Errorf("integration = %+v, want %d run and nothing counted kept", i, 2*slow.Tests)
}
// And where no node has the need, the check fails once the declines outlast the bound, naming the need.
b.kept, b.declines = newMemKV(), 1000
b.ex.Kept, b.ex.DeclineBound = Kept{Store: b.kept}, 200*time.Millisecond
if v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict; v.State != delivery.VerdictFail || !strings.Contains(fmt.Sprint(v), "needs a node with node-build-node") {
t.Errorf("with no node having the need: %+v", v)
}
}
func TestAReAskRunsOnlyWhatFailedAndWhatChanged(t *testing.T) { // 1288/AC2, AC3
@@ -341,7 +425,9 @@ func TestANodePackageIsNeverKeptAndRunsEveryCheck(t *testing.T) { // 1288/AC4, 1
}
}
func TestTheChangesOwnFailingTestRunsFirstAndNothingElseRuns(t *testing.T) { // 1289/AC1, AC2
// 1289/AC1, AC2 as issue 1921 changed it: the change's own failing test runs first, the level's other packages still run
// after it (so one check names every failure), and no later level runs.
func TestTheChangesOwnFailingTestRunsFirstThenTheRestAndNoLaterLevel(t *testing.T) {
b := newBench(t)
b.fetch.Changed = []string{"modules/m/internal/b/b_test.go"}
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}}
@@ -357,8 +443,8 @@ func TestTheChangesOwnFailingTestRunsFirstAndNothingElseRuns(t *testing.T) { //
t.Fatalf("verdict = %+v", out.Verdict)
}
tests := b.steps.tool("test")
if len(tests) != 1 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" {
t.Fatalf("test calls = %+v, want only m/internal/b, once", tests)
if len(tests) != 2 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" || strings.Join(packagesOf(tests[1]), ",") != "m/internal/a" {
t.Fatalf("test calls = %+v, want m/internal/b, then m/internal/a, each once", tests)
}
if n := len(b.steps.tool("digest")); n != 1 {
t.Errorf("%d digest calls, want the unit level's alone (the check stopped)", n)
@@ -618,9 +704,8 @@ func TestNothingIsKeptForAWorkspaceGivenSiblingsWhenTheDigestDoesNotSayItCoversT
// Beside): every check now places the change's own repository at its trunk (issue 1887 A12).
func echoBeside(b *bench, d Digested) Digested {
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
if fetches := b.steps.tool("fetch"); len(fetches) > 0 { // the workspace digested is the last fetch's (issue 890: a placed ask fetches again)
for _, r := range refsOf(fetches[len(fetches)-1].args["beside"]) {
lines = append(lines, r.Repository+" "+r.Commit)
}
}
@@ -42,7 +42,7 @@ func TestTheStatusLineNamesTheFailureNotTheFirstLoggedLine(t *testing.T) { // 19
f := fatalAfterLog()
f.Cause = ""
v = levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if !strings.HasPrefix(v.Failed.Words, "agreement_catalogue_test.go:130") {
if !strings.HasPrefix(afterHid(v.Failed.Words), "agreement_catalogue_test.go:130") {
t.Errorf("words = %q", v.Failed.Words)
}
}
@@ -56,18 +56,45 @@ func TestWordsLongerThanTheBoundKeepTheirLastLines(t *testing.T) { // 1901/AC1
_ = delivery.VerdictFail
}
// An older build module sends no "cause": its answer is still accepted and the verdict's words are its output, as before.
func TestAnOlderBuildModulesFailureIsAcceptedWithItsWholeOutput(t *testing.T) { // 1901/AC3
// An older build module sends no "cause": its answer is still accepted and the verdict's words are its output, as before
// (1901/AC3). A failure that hides its cause (none, or a passing line) is said loudly first in the words and in the status
// line, naming the build module and the version it says it is, and the level still fails; a failure that keeps the
// contract is worded as before (1901/AC8).
func TestAnOlderBuildModulesFailureIsAcceptedWithItsWholeOutput(t *testing.T) { // 1901/AC3, AC8
old := `{"package":"p","test":"TestX","output":"x_test.go:1: logged\nx_test.go:2: the old kept lines"}`
var f Failure
if err := json.Unmarshal([]byte(old), &f); err != nil || f.Cause != "" {
t.Fatalf("1901/AC3: err = %v, cause = %q", err, f.Cause)
}
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if v.Failed.Words != "x_test.go:1: logged\nx_test.go:2: the old kept lines" {
if afterHid(v.Failed.Words) != "x_test.go:1: logged\nx_test.go:2: the old kept lines" {
t.Errorf("1901/AC3: words = %q", v.Failed.Words)
}
if got := firstFailed(*v); !strings.Contains(got, "logged") || strings.Contains(got, "\n") {
t.Errorf("1901/AC3: status = %q", got)
if got := firstFailed(*v); !strings.Contains(got, "the build module hid the failure") || strings.Contains(got, "\n") {
t.Errorf("1901/AC3, AC8: status = %q", got)
}
by := "mesh-build-node with mesh-sdk v0.1.14-0.20261026184535-e40dd0f56a78"
for _, c := range []struct {
name, module, by, cause, output, loud string
}{
{"no cause", "mesh-build-node", by, "", "test/b.test.ts: something", "the build module hid the failure (mesh-build-node at mesh-sdk v0.1.14-0.20261026184535-e40dd0f56a78): it sent no cause"},
{"a passing line as its cause", "mesh-build-node", by, "\u2714 the same merge has the same id (8.006913ms)", "", "the build module hid the failure (mesh-build-node at mesh-sdk v0.1.14-0.20261026184535-e40dd0f56a78): its cause is a passing line"},
{"from before the contract", "mesh-build-node", "", "", "\u2714 the same merge has the same id (8.006913ms)", "the build module hid the failure (mesh-build-node at a version before the failure contract): it sent no cause"},
{"a real cause", "mesh-build-node", by, "one plus one is not three", "not ok 3 - fails", ""},
} {
t.Run(c.name, func(t *testing.T) {
v := levelFailure(&target{module: c.module}, "unit", Tested{By: c.by, Failed: []Failure{{Package: "modules/gitea", Test: "fails", Cause: c.cause, Output: c.output}}})
if v.State != delivery.VerdictFail {
t.Errorf("1901/AC8: the verdict is %q, want it failed", v.State)
}
line := firstFailed(*v)
switch {
case c.loud == "" && strings.Contains(v.Failed.Words, "hid the failure"):
t.Errorf("1901/AC8: a failure that keeps the contract is said hidden: %q", v.Failed.Words)
case c.loud != "" && (!strings.HasPrefix(v.Failed.Words, c.loud) || !strings.Contains(line, c.loud) || !strings.Contains(v.Failures[0].Words, "hid the failure")):
t.Errorf("1901/AC8: words %q, status %q, want both to start with %q", v.Failed.Words, line, c.loud)
}
})
}
}
@@ -60,3 +60,11 @@ func (m *memKV) Keys(context.Context) ([]string, error) {
sort.Strings(keys)
return keys, nil
}
func (m *memKV) Delete(_ context.Context, key string) error {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.vals, key)
delete(m.revs, key)
return nil
}
@@ -4,7 +4,6 @@ import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -88,29 +87,3 @@ func TestATrunkMoveMarksEveryOpenPullRequestIntoItPendingAtOnce(t *testing.T) {
t.Errorf("mesh-sdk#13, alone into a trunk that did not move, is %q, want its success to stand", s)
}
}
func TestACheckThatEndsAfterItsTrunkMovedOwesPendingNotItsVerdict(t *testing.T) { // gate (a): a pass on an old trunk is never shown
ctx := context.Background()
var r *rig
release, judged := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
defer close(judged)
} else {
<-release // the check asked again for the new trunk waits in the stream meanwhile
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
<-judged
for deadline := time.Now().Add(2 * time.Second); owedState(t, r, repoA, 7, head1) == "" && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
time.Sleep(20 * time.Millisecond)
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status is %q, want pending: the pass was judged on a trunk that moved", s)
}
close(release)
r.ch.Wait()
}
@@ -80,6 +80,9 @@ var askLanguage = map[string]string{buildrun.GoBuildModule: delivery.AskGo, buil
// stopping at the first failure; it judges the step's answer as it judged the tools' answers.
type Examiner struct {
Asks Asks
// DeclineBound is how long a placed ask may go on being declined, from its first decline, before the check fails
// saying no node with its need took it (issue 890, ADR 0552); DefaultDeclineBound when zero.
DeclineBound time.Duration
// Toolchains are the nodes' toolchains: a kept verdict names the toolchain the nodes agree on.
Toolchains ToolchainEntries
Kept Kept
@@ -107,8 +110,14 @@ type Examiner struct {
// waiting until the assigner wrote a verdict that does not wait, or ctx ends (ADR 0528 clause 5, issue 1297); nil
// leaves the composition not run.
Compositions CompositionReader
Now func() time.Time
Logf func(format string, args ...any)
// Nodes names every node with a record, read from each `<node>.node` of the node-engine's have (novox/hq ADR 0567
// §12): every check ask carries them (CheckJob.NodeNames) for the names check, so no repository lists them. A check
// whose names cannot be read, or that reads none, is not asked and fails saying so; nil hands no names.
Nodes interface {
Names(ctx context.Context) ([]string, error)
}
Now func() time.Time
Logf func(format string, args ...any)
}
// CompositionReader answers the assigner's composition of a check by the check's key, waiting until the assigner wrote
@@ -203,12 +212,15 @@ type state struct {
nodes map[string]bool
clone time.Duration
slow []string
// beside are the siblings read at their trunk's head, by repository (ADR 0548 §6).
beside map[string]string
}
// Judge runs the case (it is the Judge of a Checker). A failure of the change, of a merge or of a step is a verdict; an error is only a case that cannot
// be judged at all.
func (j *Examiner) Judge(ctx context.Context, c *Case) (Outcome, error) {
st := &state{j: j, c: c, levels: map[string]*acc{delivery.LevelUnit: {}, delivery.LevelIntegration: {}}, nodes: map[string]bool{}}
st := &state{j: j, c: c, levels: map[string]*acc{delivery.LevelUnit: {}, delivery.LevelIntegration: {}}, nodes: map[string]bool{},
beside: map[string]string{}}
if v := st.judgeAll(ctx); v != nil {
return st.finish(ctx, *v), nil
}
@@ -230,6 +242,13 @@ func (st *state) finish(ctx context.Context, v delivery.Verdict) Outcome {
// fails the check, naming the machine it failed at. A try is not composed (it has no candidate); a composition that
// cannot be had leaves it not run, said; with no assigner running it is not judged, said at once.
func (st *state) composed(ctx context.Context, v *delivery.Verdict) {
if st.c.Composition != nil && st.c.Try == nil {
// A carry whose candidate's fingerprints did not change keeps the composition it passed (ADR 0548 §5).
kept := *st.c.Composition
v.Composition = &kept
st.c.note("the composition is kept: no fingerprint of a module it composed changed")
return
}
if st.j.Compositions == nil || st.c.Try != nil {
return
}
@@ -243,6 +262,9 @@ func (st *state) composed(ctx context.Context, v *delivery.Verdict) {
return
}
withComposition(v, comp)
if v.Composition != nil && v.Composition.State != delivery.CompositionNotRun {
v.Composition.FPSet = st.c.FPSet // a carry keeps it while the fingerprints stay the same (ADR 0548 §5)
}
}
// outcome fills the verdict's levels, times, machines and composition from what ran.
@@ -282,6 +304,9 @@ func (st *state) outcome(v delivery.Verdict) Outcome {
}
}
out := Outcome{Verdict: v, Slowest: slow}
if len(st.beside) > 0 {
out.Beside = st.beside
}
for _, t := range st.targets {
if t.judged != "" {
if out.Judged == nil {
@@ -343,6 +368,7 @@ func (st *state) judgeAll(ctx context.Context) *delivery.Verdict {
if len(beside) > 0 {
args["beside"] = beside
t.beside = besideLines(beside)
besideAtTrunk(st.c, beside, st.beside)
}
if p.Repository == catalogueRepository {
t.runner = RunnerWorkspace // used by the build module when the merge changes it (ADR 0535 §1)
@@ -364,11 +390,35 @@ func (st *state) judgeAll(ctx context.Context) *delivery.Verdict {
return nil
}
// DefaultDeclineBound is an Examiner's DeclineBound when it names none: longer than a check runs, so a node with the
// need that is busy is waited for, and short enough that a mesh where no node has it says so.
const DefaultDeclineBound = 30 * time.Minute
// ask puts the target's check ask, waits for the step of the node that pulled it and judges its answer: the fetch, the
// pins of its go.mod files, then each level's digest, kept verdicts and tests.
// pins of its go.mod files, then each level's digest, kept verdicts and tests. When the answer leaves packages unmet,
// their tests needing what that node lacks (issue 890), the passes are kept and the levels from the first such one are
// asked again of a node that has it (placed).
func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *delivery.Verdict {
return st.askNeeding(ctx, t, merged, st.levelsOf(), nil, nil)
}
// askNeeding is ask with the levels to run, what the node must have and the packages it tests: none for the first ask
// of a target; for the placed one the unmet packages' needs and those packages alone, so no package the first ask judged
// is run or counted again (ADR 0552 §4).
func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, levels, needs, packages []string) *delivery.Verdict {
p := t.pull
job := &delivery.CheckJob{Fetch: t.fetchArgs, Levels: st.levelsOf(), Kept: true}
placed := len(needs) > 0
job := &delivery.CheckJob{Fetch: t.fetchArgs, Levels: levels, Kept: true, Packages: packages}
if st.j.Nodes != nil {
names, err := st.j.Nodes.Names(ctx)
if err == nil && len(names) == 0 {
err = fmt.Errorf("no node record is on the bus")
}
if err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d is not asked: the nodes' names the names check needs cannot be read: %v", p.Repository, p.Number, err))
}
job.NodeNames = names
}
if t.runner != "" {
job.Runner, job.RunnerWhen = t.runner, buildGoDir
}
@@ -380,17 +430,53 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
kind = delivery.AskRecheck // a re-judge after a trunk move goes after every new head's check (ADR 0536 §4)
}
a := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: askLanguage[t.module], Asker: delivery.CheckerModule,
At: st.j.now().UTC(), Check: job}
st.c.Progress.Set(fmt.Sprintf("the %s ask %s of %s#%d", kind, a.ID, p.Repository, p.Number), "")
if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
}
step, err := st.j.Asks.Wait(ctx, a)
if err != nil {
if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error())
At: st.j.now().UTC(), Check: job, Needs: needs}
var step delivery.Step
var firstDecline time.Time
for {
what := fmt.Sprintf("the %s ask %s of %s#%d", kind, a.ID, p.Repository, p.Number)
if placed {
what += " for a node with " + strings.Join(needs, ", ")
}
return fail(t.module, fmt.Sprintf("the check of %s#%d: %v", p.Repository, p.Number, err))
st.c.Progress.Set(what, "")
st.c.Asked(a.ID)
if err := st.j.Asks.Put(ctx, a); err != nil {
return fail(t.module, fmt.Sprintf("the check of %s#%d could not be asked: %v", p.Repository, p.Number, err))
}
waiting := ctx
if watch := st.c.Watch; watch != nil {
id := a.ID
watch(id, nil) // put: a try names it at once, not only when it ends (issue 1890)
waiting = buildrun.WithStepSeen(ctx, func(s delivery.Step) { watch(id, &s) })
}
var err error
step, err = st.j.Asks.Wait(waiting, a)
if err != nil {
if ctx.Err() != nil {
return fail(t.module, ctx.Err().Error())
}
return fail(t.module, fmt.Sprintf("the check of %s#%d: %v", p.Repository, p.Number, err))
}
d, declined := delivery.DeclinedOf(step)
if !declined {
break
}
// A node that lacks the need dropped the ask (issue 890): it is put again, for a node that has it to take; the
// node that declined looks at the stream again only after its next pull's wait.
now := st.j.now()
if firstDecline.IsZero() {
firstDecline = now
}
bound := st.j.DeclineBound
if bound <= 0 {
bound = DefaultDeclineBound
}
if now.Sub(firstDecline) > bound {
return fail(step.Node, fmt.Sprintf("the check of %s#%d needs a node with %s, and none took it in %s: every node that pulled it lacks it (last %s)",
p.Repository, p.Number, strings.Join(needs, ", "), bound, step.Node))
}
st.j.logf("%s#%d: the ask %s was declined on %s, which lacks %s; asked again", p.Repository, p.Number, a.ID, step.Node, strings.Join(d.Lacks, ", "))
a.ID, a.At = buildstep.NewID(), now.UTC()
}
// The fetch and the digests are what of the ask's time its levels' running did not take (Took.Clone).
took := time.Duration(0)
@@ -414,18 +500,31 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
if st.c.Try == nil {
st.j.writePreview(ctx, p, append(*merged, t.fetched))
}
return fail(step.Node, fmt.Sprintf("%s#%d does not merge into the trunk: files in conflict: %s", p.Repository, p.Number, strings.Join(t.fetched.Conflicts, ", ")))
return fail(step.Node, fmt.Sprintf("%s#%d %s%s", p.Repository, p.Number, ConflictWords, strings.Join(t.fetched.Conflicts, ", ")))
}
t.changed = t.fetched.Changed
if !changesBuildGo(p.Repository, t.changed) {
t.runner = "" // the build module used no runner: the merge does not change it
}
*merged = append(*merged, t.fetched)
if v := st.checkPinsOf(ctx, t); v != nil {
return v
if !placed { // the placed ask fetched the same merge: its preview and pins were read from the first
*merged = append(*merged, t.fetched)
if v := st.checkPinsOf(ctx, t); v != nil {
return v
}
if v := st.checkBinariesOf(t); v != nil {
return v
}
}
ran := 0.0
for _, la := range answer.Levels {
if placed {
if len(la.Unmet) > 0 {
st.clone += max(0, took-time.Duration(ran*float64(time.Second)))
return fail(step.Node, fmt.Sprintf("the %s level of %s#%d left %s untested on %s, which was asked for a node with %s",
la.Level, p.Repository, p.Number, unmetNames(la.Unmet), step.Node, strings.Join(needs, ", ")))
}
la.Kept = nil // it names its packages and takes no other's kept verdict; this is only a guard against an older module
}
v, elapsed := st.level(ctx, t, la)
ran += elapsed
if v != nil {
@@ -434,12 +533,52 @@ func (st *state) ask(ctx context.Context, t *target, merged *[]Fetched) *deliver
}
}
st.clone += max(0, took-time.Duration(ran*float64(time.Second)))
if unmet := delivery.UnmetNeeds(answer.Levels); len(unmet) > 0 && !placed {
// Every level passed, its passes kept, and some packages were left to a node with what their tests need: the levels
// from the first that left one are asked of such a node, which takes the kept passes and tests those (issue 890).
var again, packages []string
seen := map[string]bool{}
for _, la := range answer.Levels {
if len(again) > 0 || len(la.Unmet) > 0 {
again = append(again, la.Level)
}
for _, u := range la.Unmet {
if !seen[u.Package] {
seen[u.Package] = true
packages = append(packages, u.Package)
}
}
}
st.j.logf("%s#%d: %s on %s left untested; asked of a node with %s", p.Repository, p.Number, unmetOf(answer.Levels), step.Node, strings.Join(unmet, ", "))
return st.askNeeding(ctx, t, merged, again, unmet, packages)
}
return nil
}
// unmetNames names the unmet packages and their needs.
func unmetNames(unmet []delivery.PackageNeeds) string {
var out []string
for _, u := range unmet {
out = append(out, u.Package+" (needs "+strings.Join(u.Needs, ", ")+")")
}
return strings.Join(out, ", ")
}
// unmetOf names the unmet packages of every level.
func unmetOf(levels []delivery.LevelAnswer) string {
var out []string
for _, la := range levels {
if len(la.Unmet) > 0 {
out = append(out, "at "+la.Level+" "+unmetNames(la.Unmet))
}
}
return strings.Join(out, "; ")
}
// level judges one level of a check ask's answer as the tools' answers were judged: the digest keys the kept verdicts
// of the packages that passed, the kept ones taken are counted, and the first test result that did not pass is the
// verdict. It answers the level's running time (its tests' and setup's).
// of the packages that passed, the kept ones taken are counted, and every test result is read and counted, also after
// one that did not pass, so the verdict names every failed test of the level (issue 1921). It answers the level's
// running time (its tests' and setup's).
func (st *state) level(ctx context.Context, t *target, la delivery.LevelAnswer) (*delivery.Verdict, float64) {
level := la.Level
a := st.levels[level]
@@ -461,6 +600,7 @@ func (st *state) level(ctx context.Context, t *target, la delivery.LevelAnswer)
a.kept += k.Tests
}
elapsed := 0.0
var failedResults []Tested
for _, raw := range la.Tests {
var res Tested
if err := json.Unmarshal(raw, &res); err != nil {
@@ -489,9 +629,12 @@ func (st *state) level(ctx context.Context, t *target, la delivery.LevelAnswer)
}
}
if !res.Passed {
return levelFailure(t, level, res), elapsed
failedResults = append(failedResults, res)
}
}
if len(failedResults) > 0 {
return levelFailure(t, level, failedResults...), elapsed
}
return nil, elapsed
}
@@ -536,6 +679,12 @@ const MaxWords = 4096
// no cause told apart (an older build module) the cause is the output's first "panic:", "fatal error:" or "signal:"
// line, read before the output is cut, or the words say the tests passed and the program failed after them.
func failureWords(f Failure) string {
return failureWordsAfter(f, "")
}
// failureWordsAfter is failureWords with the line loud put first, within the same MaxWords: what the checker itself says of
// the failure (that the build module hid it), which the status line then names.
func failureWordsAfter(f Failure, loud string) string {
out, passed := dropSummary(strings.TrimSpace(f.Output))
cause := strings.TrimSpace(f.Cause)
if cause == "" {
@@ -551,11 +700,14 @@ func failureWords(f Failure) string {
if len(cause) > MaxCause {
cause = strings.ToValidUTF8(cause[:MaxCause], "") + "..."
}
// The words keep MaxWords exactly: the cause and its newline, the marker and the kept lines (review round 1).
// The words keep MaxWords exactly: the loud line, the cause and their newlines, the marker and the kept lines.
budget := MaxWords
if cause != "" {
budget -= len(cause) + 1
}
if loud != "" {
budget -= len(loud) + 1
}
if len(out) > budget {
const marker = "... "
tail := out[len(out)-(budget-len(marker)):]
@@ -568,9 +720,31 @@ func failureWords(f Failure) string {
if cause != "" && !strings.HasPrefix(firstLine(out), cause) {
out = strings.TrimRight(cause+"\n"+out, "\n")
}
if loud != "" {
out = strings.TrimRight(loud+"\n"+out, "\n")
}
return out
}
// hidWords is the checker's loud line for a failure the build module sent hiding its cause (issue 1901, its thorough
// fix): with no cause, or a passing line as its cause, under the failure contract (buildstep.Hidden). It names the build
// module and the version it says it is (Tested.By, "<module> with mesh-sdk <version>"); one from before the contract
// says nothing of itself. "" when the failure keeps the contract. The check fails either way: only its words change.
func hidWords(module, by string, f Failure) string {
why := buildstep.Hidden(f)
if why == "" {
return ""
}
version := "a version before the failure contract"
if module == "" {
module = "an unnamed build module"
}
if m, v, ok := strings.Cut(by, " with "); ok && m != "" {
module, version = m, v
}
return fmt.Sprintf("the build module hid the failure (%s at %s): %s", module, version, why)
}
// MaxCause is the most a cause in a failure's words holds: one long line (the 1901 review: a cause of a kilobyte could
// push the words past MaxWords).
const MaxCause = 1024
@@ -633,23 +807,60 @@ func firstLine(s string) string {
return ""
}
// failure turns a level that did not pass into a verdict naming what failed first.
func levelFailure(t *target, level string, res Tested) *delivery.Verdict {
// MaxFailureLine is the most bytes of a failure's line in a verdict's Failures: the first line of its words, which is
// its cause where the build module told one apart (issue 1901). Failed keeps the first failure's words whole.
const MaxFailureLine = 300
// levelFailure turns a level that did not pass into a verdict: Failed names what failed first with its words whole,
// and Failures every failed test of the level's results with its first line, at most delivery.MaxFailures and an entry
// saying how many more (issue 1921).
func levelFailure(t *target, level string, results ...Tested) *delivery.Verdict {
v := delivery.Verdict{State: delivery.VerdictFail}
switch {
case len(res.Failed) > 0:
f := res.Failed[0]
name := failedName(f)
if t.module == buildrun.NodeBuildModule && f.Package == "" {
name = res.Module
var all []delivery.FailedFirst
for _, res := range results {
if len(res.Failed) == 0 {
why := res.Why
if why == "" {
why = "the " + level + " level did not pass"
}
all = append(all, delivery.FailedFirst{Test: level + ": " + t.pull.Repository, Words: why})
continue
}
v.Failed = delivery.FailedFirst{Test: level + ": " + name, Words: failureWords(f)}
default:
why := res.Why
if why == "" {
why = "the " + level + " level did not pass"
for _, f := range res.Failed {
name := failedName(f)
if t.module == buildrun.NodeBuildModule && f.Package == "" {
name = res.Module
}
all = append(all, delivery.FailedFirst{Test: level + ": " + name, Words: failureWordsAfter(f, hidWords(t.module, res.By, f))})
}
v.Failed = delivery.FailedFirst{Test: level + ": " + t.pull.Repository, Words: why}
}
if len(all) == 0 { // no result: a level said failed with nothing to name
all = append(all, delivery.FailedFirst{Test: level + ": " + t.pull.Repository, Words: "the " + level + " level did not pass"})
}
v.Failed = all[0]
v.Failures = failureLines(all)
return &v
}
// failureLines is every failure with the first line of its words, cut to MaxFailureLine, at most delivery.MaxFailures
// and then one entry saying how many more failed.
func failureLines(all []delivery.FailedFirst) []delivery.FailedFirst {
n := min(len(all), delivery.MaxFailures)
out := make([]delivery.FailedFirst, 0, n+1)
for _, f := range all[:n] {
out = append(out, delivery.FailedFirst{Test: f.Test, Machine: f.Machine, Words: clipLine(firstLine(f.Words), MaxFailureLine)})
}
if more := len(all) - n; more > 0 {
out = append(out, delivery.FailedFirst{Test: fmt.Sprintf("and %d more", more)})
}
return out
}
// clipLine cuts a line to at most n bytes on a whole character, marking the cut.
func clipLine(s string, n int) string {
if len(s) <= n {
return s
}
s = strings.ToValidUTF8(s[:n-3], "")
return s + "..."
}
@@ -56,12 +56,22 @@ func (k Kept) Keep(ctx context.Context, level, digest string, v delivery.Kept) (
// KeepPasses keeps, for the packages of a test result that passed with tests, a verdict under the digest the build module
// named for the package at the level. A package that failed, has no tests or has no digest is never kept (ADR 0444 clause
// 1): it runs every check. It answers the packages it kept now.
// 1): it runs every check. A package with skipped tests is not kept either: a pass that skipped (a test needing a
// toolchain the node lacks) would be reused for its digest and never run anywhere (issue 890). A build module that
// counts skips only per result, not per package, keeps nothing of a result in which a test skipped. It answers the
// packages it kept now.
func (k Kept) KeepPasses(ctx context.Context, repository, level, toolchain string, res Tested, digests map[string]string, at time.Time) ([]string, error) {
var kept []string
perPackage := false
for _, p := range res.Packages {
perPackage = perPackage || p.Skipped > 0
}
if res.Skipped > 0 && !perPackage {
return kept, nil // an older build module: the skip cannot be placed in a package
}
for _, p := range res.Packages {
digest, ok := digests[p.Package]
if !ok || !p.Passed || p.NoTests || p.Tests == 0 {
if !ok || !p.Passed || p.NoTests || p.Tests == 0 || p.Skipped > 0 {
continue
}
if failedIn(res, p.Package) {
@@ -74,6 +74,32 @@ func TestOnlyPassingPackagesWithADigestAreKept(t *testing.T) { // 1287/AC3, AC4
}
}
func TestAResultWithSkippedTestsKeepsNothing(t *testing.T) { // issue 890: a pass that skipped proved nothing
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
res := Tested{Level: "integration", Passed: true, Skipped: 2, Packages: []GoPackage{{Package: "m/a", Passed: true, Tests: 4}}}
kept, err := k.KeepPasses(ctx, "novox/mesh-catalog", "integration", "go1.26.0", res, map[string]string{"m/a": digA}, at0)
if err != nil || len(kept) != 0 {
t.Fatalf("kept %v, %v, want nothing", kept, err)
}
if keys, _ := st.Keys(ctx); len(keys) != 0 {
t.Errorf("the bucket holds %v, want none", keys)
}
}
func TestOnlyTheSkippingPackageOfAResultIsWithheld(t *testing.T) { // issue 890
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
res := Tested{Level: "integration", Passed: true, Skipped: 2, Packages: []GoPackage{
{Package: "m/a", Passed: true, Tests: 4, Skipped: 2},
{Package: "m/b", Passed: true, Tests: 3},
}}
kept, err := k.KeepPasses(ctx, "novox/mesh-catalog", "integration", "go1.26.0", res, map[string]string{"m/a": digA, "m/b": digB}, at0)
if err != nil || strings.Join(kept, ",") != "m/b" {
t.Fatalf("kept %v, %v, want only m/b", kept, err)
}
}
func TestAKeptValueOfAnotherDigestIsNotRead(t *testing.T) { // 1287/AC7
ctx, st := context.Background(), newMemKV()
k := Kept{Store: st}
@@ -49,6 +49,34 @@ func firstFailed(v delivery.Verdict) string {
return firstWordsLine(f.Words)
}
// failedLines is everything a verdict names as failed, one line each: every failed test with its first line (issue
// 1921), or what failed first when the verdict names no list (a conflict, a refused step, an older verdict).
func failedLines(v delivery.Verdict) []string {
if len(v.Failures) == 0 {
if f := firstFailed(v); f != "" {
return []string{f}
}
return nil
}
lines := make([]string, 0, len(v.Failures))
for _, f := range v.Failures {
lines = append(lines, firstFailed(delivery.Verdict{Failed: f}))
}
return lines
}
// failedCount is how many tests a verdict names as failed, the ones after delivery.MaxFailures counted too.
func failedCount(v delivery.Verdict) int {
n := len(v.Failures)
if n > delivery.MaxFailures {
var more int
if _, err := fmt.Sscanf(v.Failures[n-1].Test, "and %d more", &more); err == nil {
return delivery.MaxFailures + more
}
}
return n
}
// firstWordsLine is the first line of a failure's words: the words are several lines (issue 1901) and the status line one.
func firstWordsLine(w string) string { return firstLine(w) }
@@ -57,18 +85,22 @@ type pieces struct {
head, commits, counts, shortCounts, times, shortTimes, over, builds, composition, shortComposition, group string
// judged names the program that judged a change to a build module (ADR 0535 §5); never dropped.
judged string
// carried says, of a carried verdict, the commits it was carried from (ADR 0548 §8); never dropped.
carried string
// warning says that a part of the check did not run, for instance that an older build module could not refuse a compiled program; never dropped.
warning string
}
// pullKey names a pull request of a case: `<owner/repo>#<number>`.
func pullKey(p Pull) string { return fmt.Sprintf("%s#%d", p.Repository, p.Number) }
func (p pieces) join(drop int) string {
parts := []string{p.head, p.commits, p.counts, p.judged}
parts := []string{p.head, p.commits, p.carried, p.warning, p.counts, p.judged}
switch {
case drop < 3 && p.times != "":
parts = append(parts, p.times)
case drop == 3 && p.shortTimes != "":
parts = []string{p.head, p.commits, p.shortCounts, p.judged, p.shortTimes}
parts = []string{p.head, p.commits, p.carried, p.warning, p.shortCounts, p.judged, p.shortTimes}
}
if p.over != "" {
parts = append(parts, p.over)
@@ -143,18 +175,67 @@ func buildWords(builds []BuildNote) string {
return "builds: " + strings.Join(parts, ", ")
}
// carriedWords say the commits a verdict was carried from, for the line of pull request p (ADR 0548 §8): its own trunk's
// commit as twelve characters, a sibling's with its name before it; only those that moved, sorted. Empty when none did.
func carriedWords(from map[string]string, v delivery.Verdict, cs *Case, p Pull) string {
var own string
var others []string
for r, old := range from {
now, ok := v.Trunk[r]
if cs != nil && !ok {
now, ok = cs.Trunk[r]
}
if !ok {
now = v.Beside[r]
}
if now == "" || now == old {
continue
}
if strings.EqualFold(r, p.Repository) {
own = short12(old)
continue
}
name := r
if i := strings.LastIndex(r, "/"); i >= 0 {
name = r[i+1:]
}
others = append(others, name+" "+short12(old))
}
slices.Sort(others)
if own != "" {
others = append([]string{own}, others...)
}
return strings.Join(others, ", ")
}
// parts makes the pieces of the line of a pull request of a case.
func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) pieces {
var pc pieces
pc.judged = judged[pullKey(p)]
if cs != nil {
pc.warning = cs.Warning(pullKey(p))
}
commits := short12(p.Head) + " on " + short12(p.Trunk)
switch v.State {
case delivery.VerdictRunning:
pc.head = "checking " + commits
if cs != nil && cs.CarriedFrom != nil {
// Out of date and being carried to the new commit (ADR 0548 §6): pending, and saying so.
pc.head = "being carried to " + commits
if w := carriedWords(cs.CarriedFrom, v, cs, p); w != "" {
pc.carried = "from " + w
}
}
case delivery.VerdictPass:
pc.head = "pass " + commits
if w := carriedWords(v.CarriedFrom, v, cs, p); w != "" {
pc.carried = "carried from " + w // the counts that follow say what the carry ran and kept
}
default:
pc.head = "failed: " + firstFailed(v)
if n := failedCount(v); n > 1 {
pc.head = fmt.Sprintf("failed (%d tests): %s", n, firstFailed(v))
}
pc.commits = commits
}
var counts []string
@@ -0,0 +1,39 @@
package check
import (
"strings"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// **An ask the build seat gave up on ends its step** (novox/hq issue 734): mesh-sdk buildrun.LostAskWaiter hands the
// waiting asker a failed step on the node buildrun.LostNode when DEAD_LETTERS keeps the ask's copy (buildrun.HearDeadLetters,
// granted by buscompose DeadLetterReads), and, under buildrun.NotTakenBound, when no build node took the ask within the
// bound. What follows is the checker's: a try ends as error (Tryer.run), with the not-taken bound at the step bound
// (Tryer.NotTaken); a check is asked again under its claim, at most LostAskTries times, then fails as not heard and is
// judged again when the nodes change (Checker.try). A check's ask has no not-taken bound: it waits in the stream as long as
// the backlog is (ADR 0525 §9), and a check ended for a backlog would owe its pull request a failure.
// LostAskTries is how often a check asks again whose ask was given up, before it fails as not heard (UnheardWords) and is
// judged again when the nodes change.
const LostAskTries = 3
// IsLostAsk says whether a verdict failed because its ask judged nothing: the build seat gave it up (buildrun.LostWords), or
// the build module that ran its step was replaced under it, and its next process ended the step it left running
// (buildstep.RestartReason; a delivery of the build module, as on 2026-10-28). Neither is the change's failure: the check
// is asked again, as for a given-up ask.
func IsLostAsk(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && (strings.Contains(v.Failed.Words, buildrun.LostWords) ||
strings.Contains(v.Failed.Words, buildstep.RestartReason))
}
// IsNotTaken says whether a verdict failed because no build node took its ask within the bound (buildrun.NotTakenWords).
func IsNotTaken(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, buildrun.NotTakenWords)
}
// HearingBucket is the checker's state hearing: whether it hears DEAD_LETTERS (buildrun.Hearing under
// buildrun.HearingKey), which checker_status says (dead_letters).
const HearingBucket = delivery.CheckerModule + "_" + buildrun.HearingState
@@ -0,0 +1,347 @@
package check
import (
"context"
"encoding/json"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// losingAsks is the build seat over a bus that gives up the first `lose` asks put (DEAD_LETTERS keeps them, as on
// 2026-10-26 during a bus reload: try-1793017340003993788) and answers the rest; with never set it neither answers nor
// gives up (an ask no node takes).
type losingAsks struct {
*buildrun.FakeAsks
answer func(delivery.StepAsk) delivery.Step
dead *buildrun.DeadLetters
never bool
mu sync.Mutex
lose int
}
func (l *losingAsks) PublishAsk(ctx context.Context, subject, id string, data []byte) error {
if err := l.FakeAsks.PublishAsk(ctx, subject, id, data); err != nil {
return err
}
l.mu.Lock()
lost := l.lose > 0
if lost {
l.lose--
}
l.mu.Unlock()
switch {
case l.never:
case lost:
h := map[string][]string{buildrun.DeadHeaderMsgID: {id}, buildrun.DeadHeaderDeliveries: {"5"}, buildrun.DeadHeaderStream: {"SEAT_NODE_BUILD_GO"},
buildrun.DeadHeaderGaveUp: {"2026-10-26T12:26:40Z"}}
g, ok := buildrun.GivenUpOf(h)
if !ok {
return nil
}
go l.dead.Heard(g)
default:
a, err := delivery.DecodeStepAsk(data)
if err != nil {
return err
}
l.Write(l.answer(a))
}
return nil
}
func losing(b *bench, lose int) *losingAsks {
l := &losingAsks{FakeAsks: &buildrun.FakeAsks{}, answer: b.answer, dead: &buildrun.DeadLetters{}, lose: lose}
b.ex.Asks = buildrun.Asker{Publisher: l, Waiter: buildrun.LostAskWaiter{Inner: l.FakeAsks, Dead: l.dead}, Bound: 5 * time.Second}
return l
}
// endable is the tries state as the end tool sees it.
type endable struct{ kv *memKV }
func (e endable) Entry(key string) (json.RawMessage, uint64, error) {
v, rev, _, err := e.kv.Get(context.Background(), key)
return v, rev, err
}
func (e endable) Update(key string, value []byte, revision uint64) error {
_, err := e.kv.Update(context.Background(), key, value, revision)
return err
}
// Issue 734: a try whose ask the build seat gave up on ends as error saying so, and the next ask of its head is a new try;
// one no node takes ends at the step bound; one ended by hand frees its place and stops being waited for.
func TestAnAskGivenUpOrNeverTakenEndsItsTryAsErrorAndItsHeadIsAskedAnew(t *testing.T) {
ctx := context.Background()
for _, row := range []struct {
name string
lose int
never bool
want string
}{
{"given up by the build seat", 1, false, buildrun.LostWords + " after 5 deliveries at 2026-10-26T12:26:40Z"},
{"taken by no build node within the bound", 0, true, buildrun.NotTakenWords + " within 200ms"},
} {
t.Run(row.name, func(t *testing.T) {
kv := newMemKV()
at := time.Date(2026, 10, 26, 12, 22, 20, 0, time.UTC)
id, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, at)
if err != nil {
t.Fatal(err)
}
b := newBench(t)
l := losing(b, row.lose)
l.never = row.never
tr := &Tryer{Tries: kv, Judge: b.ex, NotTaken: 200 * time.Millisecond}
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
tr.Wait()
got := readTry(t, kv, id)
if got.State != TryError || len(got.Failed) != 1 || !strings.Contains(got.Failed[0], row.want) || !strings.Contains(got.Said, row.want) {
t.Fatalf("try = %+v, want error saying %q", got, row.want)
}
again, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, at.Add(time.Minute))
if err != nil || again == id {
t.Errorf("the head asked again answered %q, %v, want a new try", again, err)
}
})
}
t.Run("ended by hand", func(t *testing.T) {
kv := newMemKV()
id, _ := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoA, Head: head1}, time.Now())
b := newBench(t)
losing(b, 0).never = true
tr := &Tryer{Tries: kv, Judge: b.ex} // no bound: only the hand ends it
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
deadline := time.Now().Add(2 * time.Second)
for readTry(t, kv, id).State != TryRunning && time.Now().Before(deadline) {
time.Sleep(5 * time.Millisecond)
}
if _, err := EndTry(endable{kv}, id, "", time.Now()); err == nil {
t.Error("an end without a why was taken")
}
ended, err := EndTry(endable{kv}, id, "its ask was lost in the bus reload", time.Now())
if err != nil || ended.State != TryError || !strings.HasPrefix(ended.Said, "ended by hand ("+TryEndTool+"): its ask was lost") {
t.Fatalf("ended = %+v, %v", ended, err)
}
raw, _, _, _ = kv.Get(ctx, id)
tr.Put(ctx, id, raw) // the watch of the tries hands the end over
tr.Wait() // returns: the try's judging stopped
if got := readTry(t, kv, id); got.State != TryError || got.Said != ended.Said {
t.Errorf("try = %+v, want the hand's end kept", got)
}
if _, err := EndTry(endable{kv}, id, "again", time.Now()); err == nil || !strings.Contains(err.Error(), "ended already") {
t.Errorf("a second end answered %v", err)
}
})
}
// troubled is the build seat as a check meets it when its step's end does not reach its wait: the first `restarts` asks'
// steps are ended by the build module's next process (buildstep.RestartReason), and the first `deaf` asks' watches hear
// nothing though their steps end (the lost check of 2026-10-28). Its steps state is read through AskSteps and Steps.
type troubled struct {
*buildrun.FakeAsks
answer func(delivery.StepAsk) delivery.Step
mu sync.Mutex
restarts, deaf int
deafIDs map[string]bool
steps map[string]delivery.Step
}
func (tr *troubled) PublishAsk(ctx context.Context, subject, id string, data []byte) error {
if err := tr.FakeAsks.PublishAsk(ctx, subject, id, data); err != nil {
return err
}
a, err := delivery.DecodeStepAsk(data)
if err != nil {
return err
}
step := tr.answer(a)
step.What = buildstep.AskWhat(a)
tr.mu.Lock()
if tr.restarts > 0 {
tr.restarts--
step.State, step.Reason, step.Result = delivery.StepFailed, buildstep.RestartReason, nil
}
deaf := tr.deaf > 0
if deaf {
tr.deaf--
tr.deafIDs[id] = true
}
tr.steps[id] = step
tr.mu.Unlock()
if !deaf {
tr.Write(step)
}
return nil
}
func (tr *troubled) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
tr.mu.Lock()
deaf := tr.deafIDs[askID]
tr.mu.Unlock()
if deaf {
return make(chan delivery.Step), nil // open, and silent for good
}
return tr.FakeAsks.WatchAsk(ctx, module, askID)
}
func (tr *troubled) AskSteps(_ context.Context, _, askID string) ([]delivery.Step, error) {
tr.mu.Lock()
defer tr.mu.Unlock()
if s, ok := tr.steps[askID]; ok {
return []delivery.Step{s}, nil
}
return nil, nil
}
// Steps is the steps state as the read tools read it, with running in place of every step the hand has not let end.
func (tr *troubled) Steps(running bool) StateRead {
tr.mu.Lock()
defer tr.mu.Unlock()
vals := map[string]json.RawMessage{}
for id, s := range tr.steps {
if running {
s.State, s.Ended, s.Started = delivery.StepRunning, time.Time{}, time.Now()
}
raw, _ := delivery.EncodeStep(s)
vals[s.Node+"."+id] = raw
}
return StateRead{Keys: func() ([]string, error) {
var out []string
for k := range vals {
out = append(out, k)
}
return out, nil
}, Get: func(k string) (json.RawMessage, error) { return vals[k], nil }}
}
// kvRead is a bucket in memory as the read tools read it, live.
func kvRead(kv *memKV) StateRead {
return StateRead{Keys: func() ([]string, error) { return kv.Keys(context.Background()) },
Get: func(k string) (json.RawMessage, error) {
v, _, _, err := kv.Get(context.Background(), k)
return v, err
}}
}
// asVerdicts is the rig's verdicts as the ask-again tool reads and writes them.
type asVerdicts struct{ kv *memKV }
func (a asVerdicts) Entry(key string) (json.RawMessage, uint64, error) {
v, rev, _, err := a.kv.Get(context.Background(), key)
return v, rev, err
}
func (a asVerdicts) Update(key string, value []byte, revision uint64) error {
_, err := a.kv.Update(context.Background(), key, value, revision)
return err
}
// Issue 734 and the lost check of 2026-10-28: a check whose ask judged nothing is asked again, and one whose step ended
// unheard is judged. Given up by the build seat once, or its step ended by the build module's next process after a
// delivery replaced it, the check is asked again under its claim and passes; given up every time it fails loudly as not
// heard, judged again when the nodes change. A step whose watch went deaf is read at the next step end of its build module
// (a nudge) and judged with no new ask; one no event reaches is named lost by checker_status, refused by checker_ask_again
// while its step still runs, and asked again by it once its step ended.
func TestACheckWhoseAskJudgedNothingIsAskedAgainOrRead(t *testing.T) {
ctx := context.Background()
for _, row := range []struct {
name string
lose, restarts, deaf int
how string // "nudge": a step of the build module ends; "hand": checker_ask_again
state string
asks int
}{
{name: "given up once", lose: 1, state: delivery.VerdictPass, asks: 2},
{name: "given up every time", lose: LostAskTries, state: delivery.VerdictFail, asks: LostAskTries},
{name: "its build module replaced under it", restarts: 1, state: delivery.VerdictPass, asks: 2},
{name: "replaced under every ask", restarts: LostAskTries, state: delivery.VerdictFail, asks: LostAskTries},
{name: "its watch deaf, a step of its build module ends", deaf: 1, how: "nudge", state: delivery.VerdictPass, asks: 1},
{name: "its watch deaf, asked again by hand", deaf: 1, how: "hand", state: delivery.VerdictPass, asks: 2},
{name: "its watch deaf, asked again by hand after its build module replaced it", deaf: 1, restarts: 1, how: "hand", state: delivery.VerdictPass, asks: 2},
} {
t.Run(row.name, func(t *testing.T) {
b := newBench(t)
var asked func() int
var heard *buildrun.HeardEnds
var tr *troubled
if row.lose > 0 {
l := losing(b, row.lose)
asked = func() int { return len(l.Asks()) }
} else {
tr = &troubled{FakeAsks: &buildrun.FakeAsks{}, answer: b.answer, restarts: row.restarts, deaf: row.deaf,
deafIDs: map[string]bool{}, steps: map[string]delivery.Step{}}
heard = &buildrun.HeardEnds{Inner: tr, Read: tr}
b.ex.Asks = buildrun.Asker{Publisher: tr, Waiter: heard, Bound: 5 * time.Second}
asked = func() int { return len(tr.Asks()) }
}
r := newRig(t, b.ex)
asksKept := newMemKV()
r.ch.Asks = asksKept
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
key := keyOf(t, "feat/x", cand)
r.ch.Candidate(ctx, "feat%2Fx", cand)
if row.deaf > 0 {
for deadline := time.Now().Add(2 * time.Second); heard.Waiting() == 0 && time.Now().Before(deadline); {
time.Sleep(2 * time.Millisecond)
}
if v := r.verdict(t, key); v.State != delivery.VerdictRunning {
t.Fatalf("a check whose step ended unheard = %s, want running until it is read", v.State)
}
}
switch row.how {
case "nudge":
heard.Nudge(buildrun.NodeBuildModule) // another build module's step: not this one's to read
heard.Nudge(buildrun.GoBuildModule)
case "hand":
cands := newMemKV()
ckey, _ := delivery.CandidatesKey("feat/x")
_, _ = cands.Create(ctx, ckey, cand)
none := StateRead{Keys: func() ([]string, error) { return nil, nil }, Get: func(string) (json.RawMessage, error) { return nil, nil }}
rd := Reader{CandidateState: kvRead(cands), VerdictState: kvRead(r.verdicts), AskState: kvRead(asksKept),
StepStates: map[string]StateRead{buildrun.GoBuildModule: none}} // no node took its ask yet
if _, err := rd.AskAgain(asVerdicts{r.verdicts}, "feat/x", "its watch went deaf"); err == nil || !strings.Contains(err.Error(), "waits in its build seat's stream") {
t.Errorf("asked again while its ask waits in the stream (issue 1791's kind): %v", err)
}
rd.StepStates[buildrun.GoBuildModule] = tr.Steps(true)
if _, err := rd.AskAgain(asVerdicts{r.verdicts}, "feat/x", "its watch went deaf"); err == nil || !strings.Contains(err.Error(), "a step of it runs") {
t.Errorf("asked again while its step runs: %v", err)
}
rd.StepStates[buildrun.GoBuildModule] = tr.Steps(false)
if st, _ := rd.Status(); !strings.Contains(st.Lost["feat/x"], AskAgainTool) {
t.Errorf("status lost = %v, want feat/x named with %s", st.Lost, AskAgainTool)
}
got, err := rd.AskAgain(asVerdicts{r.verdicts}, "feat/x", "its watch went deaf")
if err != nil || got.Key != key || !strings.HasPrefix(got.Said, AskedAgainWords) {
t.Fatalf("asked again = %+v, %v", got, err)
}
raw, _, _, _ := r.verdicts.Get(ctx, key)
r.ch.VerdictWritten(ctx, key, raw) // the checker's watch of its verdicts hands it over
}
r.ch.Wait()
v := r.verdict(t, key)
if v.State != row.state || asked() != row.asks {
t.Errorf("verdict %s (%s) after %d asks, want %s after %d", v.State, v.Failed.Words, asked(), row.state, row.asks)
}
if _, _, kept, _ := asksKept.Get(ctx, key); kept {
t.Errorf("the asks of the ended check are still kept under %s", key)
}
if row.state == delivery.VerdictFail && (!IsLostAsk(v) || !IsNodesFailure(v)) {
t.Errorf("a check that judged nothing every time = %q, want lost and judged again when the nodes change", v.Failed.Words)
}
if row.how == "hand" {
rd := Reader{VerdictState: kvRead(r.verdicts), CandidateState: kvRead(newMemKV())}
if _, err := rd.AskAgain(asVerdicts{r.verdicts}, "feat/x", "again"); err == nil {
t.Error("a judged check was asked again")
}
}
})
}
}
@@ -38,11 +38,11 @@ func TestAPackageFailureIsNeverWordedPass(t *testing.T) { // 1901/AC5
t.Errorf("1901/AC5: the words keep %q: %q", l, v.Failed.Words)
}
}
if !strings.HasPrefix(v.Failed.Words, c.want) {
t.Errorf("1901/AC5: words start %q, want %q", firstLine(v.Failed.Words), c.want)
if !strings.HasPrefix(afterHid(v.Failed.Words), c.want) {
t.Errorf("1901/AC5: words after the loud line start %q, want %q", firstLine(afterHid(v.Failed.Words)), c.want)
}
if got := firstFailed(*v); !strings.Contains(got, c.want) || strings.HasSuffix(got, ": PASS") {
t.Errorf("1901/AC5: status = %q", got)
if got := firstFailed(*v); !strings.Contains(got, "the build module hid the failure") || strings.HasSuffix(got, ": PASS") {
t.Errorf("1901/AC5, AC8: status = %q", got)
}
})
}
@@ -63,7 +63,7 @@ func TestALongCauseIsCutToALinesBound(t *testing.T) { // 1901/AC5
func TestAPackageStoppedWithOnlyItsFailLineIsStillWorded(t *testing.T) { // 1901/AC5
for _, out := range []string{"FAIL\texample.com/timeout\t2.017s", "FAIL\texample.com/exiting\t0.006s", "FAIL\texample.com/killed\t0.006s"} {
v := levelFailure(&target{}, "integration", Tested{Failed: []Failure{{Package: "example.com/x", Output: out}}})
if !strings.HasPrefix(v.Failed.Words, "the package failed with no test to name and no panic, signal or build error in its output") {
if !strings.HasPrefix(afterHid(v.Failed.Words), "the package failed with no test to name and no panic, signal or build error in its output") {
t.Errorf("1901/AC5: %q is worded %q", out, v.Failed.Words)
}
}
@@ -74,7 +74,7 @@ func TestAPackageStoppedWithOnlyItsFailLineIsStillWorded(t *testing.T) { // 1901
func TestOnlyGosExactFormsAreSummaryOrCause(t *testing.T) { // 1901/AC5
out := " x_test.go:3: ok so far\nok then: the seat is free\n panic: child retried and recovered\npanic: the real one\nFAIL\tp\t1.0s"
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{{Package: "p", Output: out}}})
if !strings.HasPrefix(v.Failed.Words, "panic: the real one") || !strings.Contains(v.Failed.Words, "ok then: the seat is free") {
if !strings.HasPrefix(afterHid(v.Failed.Words), "panic: the real one") || !strings.Contains(v.Failed.Words, "ok then: the seat is free") {
t.Errorf("1901/AC5: words = %q", v.Failed.Words)
}
}
@@ -90,3 +90,13 @@ func TestTheWordsKeepTheirBoundExactly(t *testing.T) { // 1901/AC5
}
}
}
// afterHid is a verdict's words after the checker's loud line, which a failure with no cause from an older build module
// now starts with (1901/AC8).
func afterHid(w string) string {
if strings.HasPrefix(w, "the build module hid the failure") {
_, rest, _ := strings.Cut(w, "\n")
return rest
}
return w
}
@@ -240,7 +240,12 @@ func TestTheSiblingsAreWhereTheTestsStateThem(t *testing.T) { // review D2, ADR
catalogue + " go.mod " + head2: "module git.novox.be/novox/mesh-catalog\n"}, onTrunk: map[string]bool{"cb54756c3865": true}}
b.ex.Forge = forge
// A group of the controller and the catalogue: the controller's tests read the catalogue's trunk, not the group's head.
b.judge(t, caseOf(t, pullOf(controller, 7, head1), pullOf(catalogue, 12, head2)))
out := b.judge(t, caseOf(t, pullOf(controller, 7, head1), pullOf(catalogue, 12, head2)))
// ADR 0548 §6, issue 568's own example: the verdict names the node-engine at the commit the mesh runs, and mesh-sdk at the
// trunk the catalogue's tests read it at; the catalogue, a member here, is not named.
if out.Beside[host] != hostRuns || out.Beside[catalogue] != "" || out.Beside[SDKRepository] != "9999999999999999999999999999999999999999" {
t.Errorf("the siblings named for the verdict = %v, want the node-engine at %s and mesh-sdk at its trunk", out.Beside, hostRuns[:12])
}
got := besideOf(t, b, controller)
if c := got[catalogue]; c.Commit != trunkS || c.Ref != "" {
t.Errorf("the catalogue beside the controller = %+v, want its trunk %s", c, trunkS)
@@ -324,7 +329,11 @@ func TestTheChangesOwnRepositoryIsBesideItAtItsTrunk(t *testing.T) {
func TestACatalogueChangeHasTheCatalogueBesideItAtItsTrunk(t *testing.T) {
b := newBench(t)
b.ex.Trunks = trunksOf(t, map[string]string{catalogueRepository: trunkA, SDKRepository: trunkS, controllerRepo: trunkA2})
b.judge(t, caseOf(t, pullOf(catalogueRepository, 12, head1)))
out := b.judge(t, caseOf(t, pullOf(catalogueRepository, 12, head1)))
// ADR 0548 §6: the outcome names the siblings read at their trunk's head, never the repository it merges.
if out.Beside[SDKRepository] != trunkS || out.Beside[controllerRepo] != trunkA2 || out.Beside[catalogueRepository] != "" {
t.Errorf("the siblings named for the verdict = %v, want mesh-sdk and mesh-controller at their trunks and not the catalogue", out.Beside)
}
got := besideOf(t, b, catalogueRepository)
if c := got[catalogueRepository]; c.Commit != trunkA || c.Ref != "" {
t.Fatalf("the catalogue beside its own change = %+v, want its trunk %s", c, trunkA)
@@ -75,3 +75,40 @@ func TestCheckerStatusNamesTheWaitingChecksByKind(t *testing.T) { // 1897/AC4
t.Errorf("waiting = %v, want %v", st.Waiting, want)
}
}
// 568 note 21: a verdict that read the node-engine at its running commit is not waiting for a trunk move while that commit
// is still the one the mesh runs, though the node-engine's trunk has moved on (ADR 0548 §6).
func TestCheckerStatusReadsTheNodeEngineAtItsRunningCommit(t *testing.T) { // 568/21
cand := candidateOf(t, group(1), pullOf(repoA, 1, head1))
ckey, _ := delivery.CandidatesKey(group(1))
vkey := keyOf(t, group(1), cand)
dc, _ := delivery.DecodeCandidate(cand)
judged, err := delivery.EncodeVerdict(delivery.Verdict{Group: group(1), Heads: dc.HeadsHash, Trunk: map[string]string{repoA: trunkA}, Beside: map[string]string{nodeEngineRepo: "run111"}, State: delivery.VerdictPass, Levels: passing().Levels})
if err != nil {
t.Fatal(err)
}
heads := func() (map[string]string, error) {
return map[string]string{repoA: trunkA, nodeEngineRepo: "trunk222"}, nil
}
for _, row := range []struct {
name string
runs runsAt
want int
}{
{"the running commit the verdict read", runsAt{nodeEngineModule: "run111"}, 0},
{"the running commit moved", runsAt{nodeEngineModule: "run999"}, 1},
{"no running commit known: the trunk's", nil, 1},
} {
rd := Reader{CandidateState: fakeState{ckey: cand}.read(), VerdictState: fakeState{vkey: judged}.read(), Trunks: heads}
if row.runs != nil {
rd.Runs = row.runs
}
st, err := rd.Status()
if err != nil {
t.Fatal(err)
}
if got := len(st.Waiting[KindTrunk]); got != row.want {
t.Errorf("%s: waiting for a trunk move = %v, want %d", row.name, st.Waiting[KindTrunk], row.want)
}
}
}
@@ -8,6 +8,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// The read tools (novox/hq ADR 0484 clauses 8 and 13, issue 1686): what the checker stored, as stored, with the line of the
@@ -39,7 +40,16 @@ type Reader struct {
StepBound time.Duration // how long the checker waits for one step; DefaultStepBound when 0
// Trunks are the trunk heads of every repository (the gate's source); nil names no re-judge after a trunk move.
Trunks func() (map[string]string, error)
Now func() time.Time
// TurnState is the gate's merge turns (ADR 0548 §3); without it no check is named as held by a turn.
TurnState StateRead
// Runs says which commit the mesh runs of a module (the indexer's modules), as the checker's decision reads it: the
// node-engine a verdict names is compared at its running commit, not at the trunk's (ADR 0548 §6); nil knows none.
Runs interface{ CommitOf(module string) string }
// HearingState is the checker's state hearing: whether it hears DEAD_LETTERS (issue 734).
HearingState StateRead
// AskState is the checker's state asks: the asks each running check put (AsksState), which checker_ask_again reads.
AskState StateRead
Now func() time.Time
}
// VerdictRead is a verdict as stored with the full report of the status of each pull request it judged (Report), when the
@@ -78,6 +88,11 @@ func (r Reader) read(key string, raw json.RawMessage) (VerdictRead, error) {
cs := CaseOf(c, v.Trunk)
out.Reports = map[string]string{}
if cs == nil {
if IsUntrunked(v) {
for _, p := range leftOf(c, v.Trunk) {
out.Reports[pullKey(p)] = v.Failed.Words
}
}
return out, nil
}
for _, p := range cs.Pulls {
@@ -293,6 +308,8 @@ type RunningStep struct {
ID string `json:"id"`
Tool string `json:"tool"`
Since string `json:"since,omitempty"`
// What says what the step is (buildstep.AskWhat): an ask's kind, repository@head and levels.
What string `json:"what,omitempty"`
}
// LeftStep is a key a build module left running that no step behind it runs any more, and why.
@@ -312,68 +329,83 @@ type Status struct {
// Waiting are the candidates whose check waits, by kind (ADR 0536 §4): new (no verdict for its heads) and nodes (failed
// for the nodes, asked again when they change) start before trunk (judged on trunk heads that moved since).
Waiting map[string][]string `json:"waiting_by_kind"`
Notes []string `json:"notes,omitempty"`
// HeldByTurn are the candidates whose check an approved merge's turn holds off (ADR 0548 §3), by group: the repository and
// the group of the merge that waits. Their check is claimed when that turn goes.
HeldByTurn map[string]string `json:"held_by_turn"`
// CannotBeJudged are the candidates no pull request of which can be judged, by group, with why: the gate's source.main
// names no trunk head of their repositories (issue 1422). They are judged once it does; until then their status is error.
CannotBeJudged map[string]string `json:"cannot_be_judged"`
// Lost are the candidates whose verdict is running while no step of theirs can still run and their newest step ended, by
// group, with that step: nothing will judge them (the lost check of 2026-10-28); checker_ask_again asks one again.
Lost map[string]string `json:"lost"`
// DeadLetters says whether the checker hears the asks the build seats give up on (issue 734): deaf, a try or check
// whose ask is given up waits until its bound.
DeadLetters DeadLettersHeard `json:"dead_letters"`
Notes []string `json:"notes,omitempty"`
}
// DeadLettersHeard is the checker's hearing of DEAD_LETTERS as the status says it: healthy or not, and why.
type DeadLettersHeard struct {
Heard bool `json:"heard"`
Why string `json:"why"`
}
// deadLetters reads the checker's kept hearing (buildrun.HearingHealth).
func (r Reader) deadLetters() DeadLettersHeard {
if r.HearingState.Get == nil {
return DeadLettersHeard{Why: "the checker's state hearing is not readable here"}
}
raw, err := r.HearingState.Get(buildrun.HearingKey)
if err != nil {
return DeadLettersHeard{Why: "the checker's state hearing cannot be read: " + err.Error()}
}
a := buildrun.HearingHealth(raw)
return DeadLettersHeard{Heard: a.Healthy, Why: a.Why}
}
// turns reads the gate's merge turns, by repository; nil when they cannot be read, said in a note.
func (r Reader) turns(st *Status) *Turns {
if r.TurnState.Keys == nil || r.TurnState.Get == nil {
st.Notes = append(st.Notes, "the gate's merge turns are not readable here, so no check is named as held by one")
return nil
}
keys, err := r.TurnState.Keys()
if err != nil {
st.Notes = append(st.Notes, "the gate's merge turns cannot be listed, so no check is named as held by one: "+err.Error())
return nil
}
t := &Turns{}
for _, k := range keys {
if raw, err := r.TurnState.Get(k); err == nil && len(raw) > 0 {
t.Put(k, raw, false)
}
}
return t
}
// heldBy says the turn that holds a candidate's check off, or "".
func heldBy(t *Turns, c delivery.Candidate, heads map[string]string) string {
cs := CaseOf(c, heads)
if t == nil || cs == nil {
return ""
}
repos := make([]string, 0, len(cs.Trunk))
for r := range cs.Trunk {
repos = append(repos, r)
}
if repo, w, ok := t.Holding(repos, cs.Heads); ok {
return repo + ": the approved merge of " + w.Group + " waits on it"
}
return ""
}
// Status reads the build modules' steps and the candidates against their verdicts.
func (r Reader) Status() (Status, error) {
st := Status{RunningByNode: map[string]int{}}
mods := make([]string, 0, len(r.StepStates))
for m := range r.StepStates {
mods = append(mods, m)
}
sort.Strings(mods)
st.Running, st.LeftRunning = []RunningStep{}, []LeftStep{}
now, bound := time.Now(), r.StepBound
if r.Now != nil {
now = r.Now()
}
if bound <= 0 {
bound = DefaultStepBound
}
for _, m := range mods {
sr := r.StepStates[m]
if sr.Keys == nil || sr.Get == nil {
continue
}
keys, err := sr.Keys()
if err != nil {
st.Notes = append(st.Notes, "the steps of "+m+" cannot be listed: "+err.Error())
continue
}
// A step that runs started lately: the newest ids are read first, at most MaxReads of them.
newestFirst(keys)
keys, cut := capKeys(keys, "")
if cut != "" {
st.Notes = append(st.Notes, "the steps of "+m+": read the newest "+fmt.Sprint(MaxReads)+" of its keys; a step older than those is not shown")
}
steps := make([]delivery.Step, len(keys))
isRunning := make([]bool, len(keys))
_ = parallel(len(keys), func(i int) error {
raw, err := sr.Get(keys[i])
if err != nil || len(raw) == 0 {
return nil
}
step, err := delivery.DecodeStep(raw)
steps[i], isRunning[i] = step, err == nil && step.State == delivery.StepRunning
return nil
})
for i := range keys {
if !isRunning[i] {
continue
}
step := steps[i]
rs := RunningStep{Module: m, Node: step.Node, ID: step.ID, Tool: step.Tool}
if !step.Started.IsZero() {
rs.Since = step.Started.UTC().Format("2006-01-02T15:04:05Z")
}
if why := r.leftWhy(m, step, now, bound); why != "" {
st.LeftRunning = append(st.LeftRunning, LeftStep{RunningStep: rs, Why: why})
continue
}
st.Running = append(st.Running, rs)
st.RunningByNode[step.Node]++
}
st := Status{RunningByNode: map[string]int{}, Lost: map[string]string{}}
read := r.readSteps()
st.Running, st.LeftRunning, st.Notes = read.running, read.left, read.notes
for _, rs := range st.Running {
st.RunningByNode[rs.Node]++
}
if r.HaveState.Get == nil {
st.Notes = append(st.Notes, "the node-engine's have is not readable here, so a step is not checked against its build module being on its node")
@@ -383,6 +415,8 @@ func (r Reader) Status() (Status, error) {
}
st.Unjudged = []string{}
st.Waiting = map[string][]string{KindNew: {}, KindNodes: {}, KindTrunk: {}}
st.HeldByTurn, st.CannotBeJudged = map[string]string{}, map[string]string{}
turns := r.turns(&st)
var heads map[string]string
if r.Trunks == nil {
st.Notes = append(st.Notes, "the trunk heads are not readable here, so no check is named as waiting for a trunk move")
@@ -432,27 +466,162 @@ func (r Reader) Status() (Status, error) {
continue
}
c, raw := row.c, row.raw
if heads != nil && len(leftOf(c, heads)) > 0 {
// Nothing of it can be judged now, whatever its verdict says (issue 1422): never counted as waiting.
var repos []string
for _, p := range leftOf(c, heads) {
repos = append(repos, p.Repository)
}
st.CannotBeJudged[c.Group] = UntrunkedWordsOf(repos)
continue
}
if len(raw) == 0 {
st.Unjudged = append(st.Unjudged, c.Group)
st.Waiting[KindNew] = append(st.Waiting[KindNew], c.Group)
if why := heldBy(turns, c, heads); why != "" {
st.HeldByTurn[c.Group] = why
}
continue
}
v, err := delivery.DecodeVerdict(raw)
switch {
case err != nil || v.State == delivery.VerdictRunning:
case err != nil:
case v.State == delivery.VerdictRunning:
if why := read.lostWhy(c); why != "" {
st.Lost[c.Group] = why
}
case IsNodesFailure(v):
st.Waiting[KindNodes] = append(st.Waiting[KindNodes], c.Group)
case IsUntrunked(v):
// Its trunk head is known now: it waits for its whole check.
st.Waiting[KindNew] = append(st.Waiting[KindNew], c.Group)
case heads != nil:
if cs := CaseOf(c, heads); cs != nil && !sameTrunk(v.Trunk, cs.Trunk) {
if cs := CaseOf(c, heads); cs != nil && (!sameTrunk(v.Trunk, cs.Trunk) || besideMoved(v.Beside, besideAt(heads, r.Runs))) {
st.Waiting[KindTrunk] = append(st.Waiting[KindTrunk], c.Group)
if why := heldBy(turns, c, heads); why != "" {
st.HeldByTurn[c.Group] = why
}
}
}
}
}
st.DeadLetters = r.deadLetters()
st.Notes = append(st.Notes, "nothing holds the checker and it holds no place: running_steps are the build modules' keys that can still be running (started within the step bound, their build module running on the node); a candidate without a verdict has its check ask in a build seat's stream, waiting for a node with a free slot, whose backlog the delivery view shows (ADR 0525 §4, §9)")
return st, nil
}
// stepsRead is what the build modules' steps states hold: the steps that can still run, those left running, and every
// ask step read, by module, for the candidate it belongs to (stepsOf).
type stepsRead struct {
running []RunningStep
left []LeftStep
all []moduleStep
notes []string
}
type moduleStep struct {
module string
step delivery.Step
}
// readSteps reads the newest MaxReads keys of each build module's steps state.
func (r Reader) readSteps() stepsRead {
out := stepsRead{running: []RunningStep{}, left: []LeftStep{}}
mods := make([]string, 0, len(r.StepStates))
for m := range r.StepStates {
mods = append(mods, m)
}
sort.Strings(mods)
now, bound := time.Now(), r.StepBound
if r.Now != nil {
now = r.Now()
}
if bound <= 0 {
bound = DefaultStepBound
}
for _, m := range mods {
sr := r.StepStates[m]
if sr.Keys == nil || sr.Get == nil {
continue
}
keys, err := sr.Keys()
if err != nil {
out.notes = append(out.notes, "the steps of "+m+" cannot be listed: "+err.Error())
continue
}
// A step that runs started lately: the newest ids are read first, at most MaxReads of them.
newestFirst(keys)
keys, cut := capKeys(keys, "")
if cut != "" {
out.notes = append(out.notes, "the steps of "+m+": read the newest "+fmt.Sprint(MaxReads)+" of its keys; a step older than those is not shown")
}
steps := make([]delivery.Step, len(keys))
decoded := make([]bool, len(keys))
_ = parallel(len(keys), func(i int) error {
raw, err := sr.Get(keys[i])
if err != nil || len(raw) == 0 {
return nil
}
step, err := delivery.DecodeStep(raw)
steps[i], decoded[i] = step, err == nil
return nil
})
for i := range keys {
if !decoded[i] {
continue
}
step := steps[i]
out.all = append(out.all, moduleStep{module: m, step: step})
if step.State != delivery.StepRunning {
continue
}
rs := RunningStep{Module: m, Node: step.Node, ID: step.ID, Tool: step.Tool, What: step.What}
if !step.Started.IsZero() {
rs.Since = step.Started.UTC().Format("2006-01-02T15:04:05Z")
}
if why := r.leftWhy(m, step, now, bound); why != "" {
out.left = append(out.left, LeftStep{RunningStep: rs, Why: why})
continue
}
out.running = append(out.running, rs)
}
}
return out
}
// asksOf says whether a step is an ask of one of the candidate's pull requests: its What names `<repository>@<head12>`
// (buildstep.AskWhat).
func asksOf(c delivery.Candidate, what string) bool {
for _, p := range c.Pulls {
if p.Repository != "" && strings.Contains(what, p.Repository+"@"+short12(p.Head.Head)) {
return true
}
}
return false
}
// lostWhy says why a candidate whose verdict is running has a check that nothing will judge, or "": no step of its pull
// requests can still run, and its newest one ended.
func (s stepsRead) lostWhy(c delivery.Candidate) string {
for _, rs := range s.running {
if asksOf(c, rs.What) {
return ""
}
}
var last *delivery.Step
for i := range s.all {
st := s.all[i].step
if st.State != delivery.StepRunning && asksOf(c, st.What) && (last == nil || st.Ended.After(last.Ended)) {
last = &s.all[i].step
}
}
if last == nil {
return "" // its ask waits in its build seat's stream: no node took it yet
}
return fmt.Sprintf("its verdict is running, but no step of it runs: its last step %s on %s ended %s at %s and no verdict followed; %s asks it again",
last.ID, last.Node, last.State, last.Ended.UTC().Format(time.RFC3339), AskAgainTool)
}
// DefaultStepBound is how long one check ask's step may run once a build node took it (its key written running; the fetch,
// the digests and the levels), never counting its wait in the stream; past it the check fails naming the node.
const DefaultStepBound = 30 * time.Minute
@@ -36,6 +36,7 @@ func (f fakeState) read() StateRead {
func readerOf(t *testing.T) (Reader, string) {
t.Helper()
r := newRig(t, passJudge())
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA2})
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(hqRepo, 3, head2))
r.ch.Candidate(context.Background(), "feat%2Fx", cand)
r.ch.Wait()
@@ -144,7 +145,10 @@ func TestTheStatusToolReadsRunningSteps(t *testing.T) {
CandidateState: fakeState{ckey: cand}.read(),
VerdictState: fakeState{}.read(),
HaveState: fakeState{"shanks.mesh-build-go": haveOf(t, delivery.PhaseRunning)}.read(),
Now: func() time.Time { return time.Date(2026, 10, 22, 15, 10, 0, 0, time.UTC) },
HearingState: fakeState{"dead-letters": []byte(`{"heard":false,"streams":["SEAT_NODE_BUILD_GO"],` +
`"why":"DEAD_LETTERS is not heard, so an ask a build seat gives up on is waited for until its bound: SEAT_NODE_BUILD_GO: permissions violation",` +
`"at":"2026-10-26T13:00:00Z"}`)}.read(),
Now: func() time.Time { return time.Date(2026, 10, 22, 15, 10, 0, 0, time.UTC) },
}
st, err := rd.Status()
if err != nil {
@@ -156,9 +160,13 @@ func TestTheStatusToolReadsRunningSteps(t *testing.T) {
if len(st.Unjudged) != 1 || st.Unjudged[0] != "feat/x" {
t.Errorf("unjudged = %v", st.Unjudged)
}
// Issue 734, review round 1: a checker that does not hear DEAD_LETTERS says so in its status, with why.
if st.DeadLetters.Heard || !strings.Contains(st.DeadLetters.Why, "permissions violation") {
t.Errorf("dead letters = %+v, want not heard, saying why", st.DeadLetters)
}
// What cannot be read is said, not guessed.
st, err = Reader{}.Status()
if err != nil || len(st.Notes) == 0 {
if err != nil || len(st.Notes) == 0 || st.DeadLetters.Heard {
t.Errorf("empty status = %+v, %v", st, err)
}
}
@@ -234,6 +242,18 @@ func TestTheStatusCountsAnHQOnlyGroupAsUnjudged(t *testing.T) {
if len(st.Unjudged) != 1 || st.Unjudged[0] != "pb-x" {
t.Errorf("unjudged = %v, want pb-x", st.Unjudged)
}
// ADR 0548 §3: a candidate whose check an approved merge's turn holds off is named with the turn.
held := candidateOf(t, "feat/x", pullOf(repoA, 7, head1))
hkey, _ := delivery.CandidatesKey("feat/x")
st, err = Reader{CandidateState: fakeState{hkey: held}.read(), VerdictState: fakeState{}.read(),
Trunks: func() (map[string]string, error) { return map[string]string{repoA: trunkA}, nil },
TurnState: fakeState{turnKey(t, repoA): turnOf(t, repoA, "fix/other", "0123456789ab")}.read()}.Status()
if err != nil {
t.Fatal(err)
}
if why := st.HeldByTurn["feat/x"]; !strings.Contains(why, repoA) || !strings.Contains(why, "fix/other") {
t.Errorf("held_by_turn = %v, want feat/x held by the merge of fix/other on %s", st.HeldByTurn, repoA)
}
}
// slowBucket wraps a state with a Get that takes delay and counts the values read, the lists and the most Gets in flight.
@@ -3,6 +3,9 @@ package check
import (
"encoding/json"
"strings"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The shapes of what the build modules' steps answer, as their code writes them (mesh-catalog modules/mesh-build-go
@@ -42,6 +45,17 @@ type Fetched struct {
Conflicts []string `json:"conflicts,omitempty"`
Changed []string `json:"changed,omitempty"`
Trees map[string]MergeTree `json:"trees,omitempty"`
// Programs are the changed files whose first bytes are an executable binary, read by the build module in its workspace
// (issue 1577); nil for a build module that does not report it, a pointer to an empty list for one that found none.
Programs *[]ProgramFile `json:"programs,omitempty"`
}
// ProgramFile is a changed file that is a compiled program, and its format (ELF, Mach-O or PE).
type ProgramFile struct {
Path string `json:"path"`
Kind string `json:"kind"`
// Commit is set for a program a commit of the branch added and a later one removed: it stays in history once merged.
Commit string `json:"commit,omitempty"`
}
// GoPackage is one package of a Go test result.
@@ -52,16 +66,13 @@ type GoPackage struct {
Elapsed float64 `json:"elapsed"`
Tests int `json:"tests"`
NoTests bool `json:"no_tests,omitempty"`
// Skipped is how many of the package's tests skipped; an older build module names none.
Skipped int `json:"skipped,omitempty"`
}
// Failure is a test that failed, or a package that failed with no test to name.
type Failure struct {
Package string `json:"package"`
Test string `json:"test,omitempty"`
Output string `json:"output,omitempty"`
// Cause is the first line of what the test reported as its error, when the build module told it apart (issue 1901).
Cause string `json:"cause,omitempty"`
}
// Failure is a test that failed, or a package that failed with no test to name: the build modules' failure contract
// (buildstep.Failure, issue 1901), its cause never empty and never a passing line from a build module that keeps it.
type Failure = buildstep.Failure
// TestTime is one test and the seconds it took.
type TestTime struct {
@@ -94,6 +105,9 @@ type Tested struct {
// JudgedBy is the program that ran the step (ADR 0535 §5): "installed", or "candidate <merge commit>" under runner
// workspace; absent from a build module that does not take the runner.
JudgedBy string `json:"judged_by,omitempty"`
// By is the build module that answered and the SDK it was built with (buildstep.By); absent from a build module from
// before the failure contract.
By string `json:"by,omitempty"`
}
// Digested is the answer of the Go build module's digest step: one digest per package that has tests at the level, by import
@@ -107,4 +121,6 @@ type Digested struct {
WithoutTests []string `json:"without_tests,omitempty"`
// JudgedBy is the program that ran the digest, as Tested.JudgedBy says.
JudgedBy string `json:"judged_by,omitempty"`
// Needs are the packages whose tests need something of the node at this level (issue 890).
Needs []delivery.PackageNeeds `json:"needs,omitempty"`
}
@@ -10,7 +10,7 @@ TestFetchWithABaseMergesAndAnswersThePreview and TestFetchWithAConflictEndsPasse
into a throwaway test that wrote them out, at mesh-catalog feat/r4c5-checker d5804927 (the git commands are the test's fake,
so the shas are its constants; the shape and the field names are the module's own marshalling). node-fetch-clean.json is the
FetchResult json.Marshal of mesh-build-node's fetch (fetch.go, preview.go) in its test TestFetchWithABaseMergesAndAnswersThePreview
(cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679.
(cmd/mesh-build-node/preview_test.go) at the same commit. Captured on 2026-10-21 for issue 1679. Both clean fetches carry `"programs":[]`, which a merging fetch of the build modules answers since issue 1577 (mesh-sdk buildstep.MergeInto's Programs), added by hand to the captured bytes: a fixture without it is an older build module's, and its line warns that no compiled program is refused.
forge-get-file-go-mod.json is the answer of the forge's gitea_get_file (mesh-catalog modules/gitea, tools/index.ts) for modules/mesh-delivery-builder/go.mod of
novox/mesh-delivery at main, and forge-compare-trunk.json the answer of gitea_compare for novox/mesh-sdk base main, head cb54756c3865 (a commit on the
@@ -1 +1 @@
{"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}}
{"workspace":"mvii1iwu5414e4","commit":"0123456789abcdef0123456789abcdef01234567","modules":["."],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"fedcba9876543210fedcba9876543210fedcba98","changed":["a.go","modules/m/main.go"],"programs":[],"trees":{"modules/m":{"tree":"1111111111111111111111111111111111111111","manifest":"ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356","locks":{"go.sum":"c5fc83c01e92404452b986527d239140ccf9a48b88e0c268fbf38c2e1429e9c9"}}}}
@@ -1 +1 @@
{"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}}
{"workspace":"mvii1sxt979f67","commit":"0123456789abcdef0123456789abcdef01234567","modules":["modules/m"],"installed":[],"refused":[],"base":"abcdef0123456789abcdef0123456789abcdef01","merge":"0123456789abcdef0123456789abcdef01234567","changed":["modules/m/package.json"],"programs":[],"trees":{"modules/m":{"tree":"0123456789abcdef0123456789abcdef01234567","manifest":"0581b699d9d05517e69b21f5cedbeb2eee8aba2b205280b561be731594d7feef"}}}
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"regexp"
"slices"
"sort"
"strconv"
"strings"
@@ -13,6 +14,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// A try is a pushed head judged by the checker as a pull request's head is, before its pull request is opened (ADR 0501
@@ -24,11 +26,17 @@ import (
const (
TryTool = "checker_try"
TryReadTool = "checker_try_read"
TryEndTool = "checker_try_end"
// TriesState is the checker's state of tries: the bucket mesh-delivery-checker_tries, keys try-<unix nanoseconds>.
TriesState = "tries"
TriesBucket = delivery.CheckerModule + "_" + TriesState
// MaxTries is how many tries may wait or run at once, mesh-wide: each holds a build node for minutes.
MaxTries = 4
// MaxTries and MaxTriesWaiting are checked when a try is asked, mesh-wide: a new try is refused while MaxTries tries run
// on a build node or MaxTriesWaiting wait for one (queued, or their asks not yet taken by a node). It is no bound on
// what runs later: tries let in while waiting may later all run at once, and each node's build capacity bounds what
// runs on it. A try waiting behind the seat's other asks holds no node, so it no longer blocks the next try (issue
// 1890: at 19:53 on 2026-10-26 four tries sat queued behind 44 checks for up to 47 minutes and no fifth could be asked).
MaxTries = 4
MaxTriesWaiting = 8
// TryScan is how many of the newest tries a new ask reads to find the ones that wait or run (issue 1827).
TryScan = 64
// TryKeep is how long an ended try's key is kept before the checker removes it.
@@ -69,13 +77,110 @@ type Try struct {
// Said is the try's report in words: what failed first, the tests run, kept and skipped per level, the time of each part.
Said string `json:"said,omitempty"`
// Took is the time from its start to its end, as Go writes a duration.
Took string `json:"took,omitempty"`
Took string `json:"took,omitempty"`
// Failed is every test that failed, each with its first line, at most delivery.MaxFailures and then how many more
// (issue 1921); or what else failed first (a conflict, a refused step).
Failed []string `json:"failed,omitempty"`
Levels map[string]delivery.LevelResult `json:"levels,omitempty"`
Machines []string `json:"machines,omitempty"`
Asked time.Time `json:"asked"`
Started time.Time `json:"started,omitzero"`
Ended time.Time `json:"ended,omitzero"`
// Asks are the ids of the asks the try put on the build seats' streams, each ending on the step key <node>.<id> of its
// build module: the delivery view counts those steps as a try's, not a candidate's check (build_usage).
Asks []string `json:"asks,omitempty"`
// Steps say where each ask stands while the try runs and after: asked until a node takes it, then the node, its
// state, the phase its step says and when (issue 1890: a 31-minute try could not be told queued from running).
Steps []TryStep `json:"steps,omitempty"`
Asked time.Time `json:"asked"`
Started time.Time `json:"started,omitzero"`
Ended time.Time `json:"ended,omitzero"`
}
// TryAsked is a TryStep's state before a build node took its ask: it waits in the seat's stream.
const TryAsked = "asked"
// TryStep is one ask of a try as its step key last said it.
type TryStep struct {
Ask string `json:"ask"`
Node string `json:"node,omitempty"`
State string `json:"state"`
Phase string `json:"phase,omitempty"`
Started time.Time `json:"started,omitzero"`
PhaseAt time.Time `json:"phase-at,omitzero"`
Ended time.Time `json:"ended,omitzero"`
}
// tryWriter writes a running try's record again as its asks are put and their steps move: one writer per try, holding
// the revision it wrote last, so the try's end is written at that revision. A write refused (the try ended by hand)
// stops the writes; the end's own write then says it.
type tryWriter struct {
mu sync.Mutex
tr *Tryer
ctx context.Context
key string
t Try
rev uint64
stopped bool
}
// heard takes an ask put (step nil) or a value of its step key in, and writes the record when it changed.
func (w *tryWriter) heard(askID string, step *delivery.Step) {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped {
return
}
changed := false
if !slices.Contains(w.t.Asks, askID) {
w.t.Asks, changed = append(w.t.Asks, askID), true
}
i := slices.IndexFunc(w.t.Steps, func(s TryStep) bool { return s.Ask == askID })
if i < 0 {
w.t.Steps, changed = append(w.t.Steps, TryStep{Ask: askID, State: TryAsked}), true
i = len(w.t.Steps) - 1
}
if step != nil {
now := TryStep{Ask: askID, Node: step.Node, State: step.State, Phase: step.Phase, Started: step.Started,
PhaseAt: step.PhaseAt, Ended: step.Ended}
changed = changed || now != w.t.Steps[i]
w.t.Steps[i] = now
}
if changed {
w.write()
}
}
// write writes the record at the revision written last; a refusal stops the writes.
func (w *tryWriter) write() {
data, err := encodeTry(w.t)
if err != nil {
return
}
rev, err := w.tr.Tries.Update(w.ctx, w.key, data, w.rev)
if err != nil {
w.stopped = true
w.tr.logf("%s: its asks' progress is no longer written: %v", w.key, err)
return
}
w.rev = rev
}
// end stops the writes and answers the record and its revision, for the try's end.
func (w *tryWriter) end() (Try, uint64) {
w.mu.Lock()
defer w.mu.Unlock()
w.stopped = true
return w.t, w.rev
}
// HoldsANode says whether the try holds a build node now: it runs, and a step of its asks runs on a node, or it names
// no step (a checker that writes its asks only at the end, or one about to put its first).
func (t Try) HoldsANode() bool {
if t.State != TryRunning {
return false
}
if len(t.Steps) == 0 {
return true
}
return slices.ContainsFunc(t.Steps, func(s TryStep) bool { return s.State == delivery.StepRunning })
}
// Over says whether the try came to its end.
@@ -117,8 +222,8 @@ func NotChecked(repository string) error {
}
// AskTry writes a new try, queued, and answers its id: a try of the same head that waits or runs is the answer to a second
// ask, and at most MaxTries wait or run at once. trunks are the gate's trunk heads of every repository (source.main): a
// repository with none is no repository the checker checks. A trunk asked by name is its trunk head now.
// ask, and a new try is refused while MaxTries tries run on a build node or MaxTriesWaiting wait for one. trunks are the
// gate's trunk heads of every repository (source.main): a repository with none is no repository the checker checks. A trunk asked by name is its trunk head now.
func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (string, error) {
repo := strings.ToLower(strings.TrimSuffix(strings.TrimSpace(a.Repository), ".git"))
head := strings.ToLower(strings.TrimSpace(a.Head))
@@ -181,7 +286,7 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
}); err != nil {
return "", err
}
var open []string
var holding, waiting []string
for i, cur := range tries {
if !live[i] {
continue
@@ -189,10 +294,18 @@ func AskTry(store TryStore, trunks map[string]string, a TryAsk, now time.Time) (
if cur.Repository == t.Repository && cur.Head == t.Head && cur.Trunk == t.Trunk && cur.Tests == t.Tests && cur.Level == t.Level {
return cur.ID, nil
}
open = append(open, fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State))
said := fmt.Sprintf("%s (%s@%.12s, %s)", cur.ID, cur.Repository, cur.Head, cur.State)
if cur.HoldsANode() {
holding = append(holding, said)
} else {
waiting = append(waiting, said)
}
}
if len(open) >= MaxTries {
return "", fmt.Errorf("%d tries wait or run, the most at once: %s; ask again when one ends", len(open), strings.Join(open, ", "))
// Checked at this ask only: a try queued behind the seat's checks holds no node; tries let in while waiting may later
// all run at once, and each node's build capacity bounds what runs on it.
if len(holding) >= MaxTries || len(waiting) >= MaxTriesWaiting {
return "", fmt.Errorf("a new try is refused while %d tries run on a build node or %d wait for one; now %d run and %d wait: running %s; waiting %s; ask again when one ends",
MaxTries, MaxTriesWaiting, len(holding), len(waiting), orNone(holding), orNone(waiting))
}
t.ID = "try-" + strconv.FormatInt(now.UnixNano(), 10)
data, err := encodeTry(t)
@@ -220,6 +333,52 @@ func ReadTry(store TryStore, id string) (json.RawMessage, error) {
return raw, nil
}
// TryEnder is the tries state as the end tool sees it: one key with its revision, and a write at that revision.
type TryEnder interface {
Entry(key string) (json.RawMessage, uint64, error)
Update(key string, value []byte, revision uint64) error
}
// EndTry ends a try that waits or runs as error, saying why and that it was ended by hand (issue 734): its place is
// free at once, and the next checker_try of its head asks anew. A try that ended already is refused, naming its end.
func EndTry(store TryEnder, id, why string, now time.Time) (Try, error) {
if !tryKeyOK.MatchString(id) {
return Try{}, fmt.Errorf("%q is not a try's id, try-<digits>", id)
}
why = strings.TrimSpace(why)
if why == "" {
return Try{}, errors.New("say why the try is ended: the why is kept in what it says")
}
raw, rev, err := store.Entry(id)
if err != nil {
return Try{}, err
}
if len(raw) == 0 {
return Try{}, fmt.Errorf("no try %s: it was never asked, or it ended more than %s ago and was removed", id, TryKeep)
}
t, err := DecodeTry(raw)
if err != nil {
return Try{}, err
}
if t.Over() {
return Try{}, fmt.Errorf("%s ended already, %s at %s", id, t.State, t.Ended.Format(time.RFC3339))
}
t.State, t.Ended = TryError, now.UTC()
if !t.Started.IsZero() {
t.Took = t.Ended.Sub(t.Started).Round(time.Second).String()
}
t.Said = strings.TrimSpace("ended by hand (" + TryEndTool + "): " + why + " | " + t.Said)
t.Said = strings.TrimSuffix(t.Said, " |")
data, err := encodeTry(t)
if err != nil {
return Try{}, err
}
if err := store.Update(id, data, rev); err != nil {
return Try{}, fmt.Errorf("%s changed while it was ended, read it and end it again: %w", id, err)
}
return t, nil
}
// TryOptions narrow what a try runs; a pull request's check has none.
type TryOptions struct {
Level string
@@ -232,12 +391,14 @@ type Tryer struct {
Tries KV
Judge Judge
Bound time.Duration
Now func() time.Time
Logf func(format string, args ...any)
// NotTaken ends a try whose ask no build node took within it as error (issue 734); none when 0.
NotTaken time.Duration
Now func() time.Time
Logf func(format string, args ...any)
mu sync.Mutex
wg sync.WaitGroup
inflight map[string]bool
inflight map[string]context.CancelFunc
seen map[string]bool // looked at since this process started: a try left running is taken over the first time
}
@@ -265,25 +426,31 @@ func (tr *Tryer) Put(ctx context.Context, key string, value []byte) {
tr.logf("tries.%s holds what is not a try, left: %v", key, err)
return
}
if t.Over() {
return
}
tr.mu.Lock()
if tr.inflight == nil {
tr.inflight, tr.seen = map[string]bool{}, map[string]bool{}
tr.inflight, tr.seen = map[string]context.CancelFunc{}, map[string]bool{}
}
if tr.inflight[key] || (t.State == TryRunning && tr.seen[key]) {
if t.Over() {
// Ended while it ran (checker_try_end): its judging stops, and its asks are no longer waited for.
if stop := tr.inflight[key]; stop != nil {
stop()
}
tr.mu.Unlock()
return
}
if tr.inflight[key] != nil || (t.State == TryRunning && tr.seen[key]) {
tr.mu.Unlock()
return
}
run, stop := context.WithCancel(ctx)
tr.seen[key] = true
tr.inflight[key] = true
tr.inflight[key] = stop
tr.mu.Unlock()
tr.wg.Add(1)
go func() {
defer tr.wg.Done()
defer func() { tr.mu.Lock(); delete(tr.inflight, key); tr.mu.Unlock() }()
tr.run(ctx, key)
defer func() { tr.mu.Lock(); delete(tr.inflight, key); tr.mu.Unlock(); stop() }()
tr.run(run, key)
}()
}
@@ -334,34 +501,42 @@ func (tr *Tryer) run(ctx context.Context, key string) {
}
return
}
cs := &Case{Key: key, Group: key,
progress := &tryWriter{tr: tr, ctx: ctx, key: key, t: t, rev: rev}
cs := &Case{Key: key, Group: key, Watch: progress.heard,
Pulls: []Pull{{Repository: t.Repository, Head: t.Head, Trunk: t.Trunk}},
Trunk: map[string]string{t.Repository: t.Trunk},
Queued: t.Asked,
Progress: &Progress{},
Try: &TryOptions{Level: t.Level, Run: t.Tests}}
out, judged := judgeWithin(ctx, tr.Judge, cs, tr.Bound)
if ctx.Err() != nil {
return // left running: the checker takes it over when it starts
judging := ctx
if tr.NotTaken > 0 {
judging = buildrun.NotTakenBound(ctx, tr.NotTaken)
}
out, judged := judgeWithin(judging, tr.Judge, cs, tr.Bound)
if ctx.Err() != nil {
return // left running: the checker takes it over when it starts; or ended by hand, which wrote its end
}
t, rev = progress.end()
v := out.Verdict
t.Ended = tr.now().UTC()
t.Took = t.Ended.Sub(t.Started).Round(time.Second).String()
t.Levels, t.Machines = v.Levels, v.Machines
t.Levels, t.Machines, t.Asks = v.Levels, v.Machines, cs.Asks()
t.Said = Report(v, cs, cs.Pulls[0], out.Builds, out.Slowest, out.Judged)
for _, n := range cs.Notes() {
t.Said += " | " + n
}
switch {
case !judged || IsNodesFailure(v):
case !judged || IsNodesFailure(v) || IsLostAsk(v) || IsNotTaken(v):
// Not the change's: its ask was given up or never taken (issue 734), or a part of the mesh was not heard.
t.State = TryError
if IsLostAsk(v) || IsNotTaken(v) {
t.Failed = []string{v.Failed.Words}
}
case v.State == delivery.VerdictPass:
t.State = TryPass
default:
t.State = TryFail
if f := firstFailed(v); f != "" {
t.Failed = []string{f}
}
t.Failed = failedLines(v)
}
if t.Tests != "" {
t.Said += " | only the tests " + t.Tests + " (Go); nothing kept"
@@ -403,3 +578,11 @@ func judgeWithin(ctx context.Context, j Judge, cs *Case, bound time.Duration) (O
}
return out, true
}
// orNone joins the tries said, or says none.
func orNone(said []string) string {
if len(said) == 0 {
return "none"
}
return strings.Join(said, ", ")
}
@@ -3,6 +3,7 @@ package check
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"time"
@@ -23,6 +24,19 @@ func (m memTries) Create(key string, value []byte) error {
return err
}
// writtenTries is a memKV that keeps every running try it was given to write, in order.
type writtenTries struct {
*memKV
running []Try
}
func (w *writtenTries) Update(ctx context.Context, key string, value []byte, rev uint64) (uint64, error) {
if t, err := DecodeTry(value); err == nil && !t.Over() {
w.running = append(w.running, t)
}
return w.memKV.Update(ctx, key, value, rev)
}
var tryTrunks = map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA}
func readTry(t testing.TB, kv *memKV, id string) Try {
@@ -54,14 +68,38 @@ func TestATryIsAskedQueuedOncePerHeadAndAtMostFourAtOnce(t *testing.T) { // ADR
if err != nil || again != id {
t.Errorf("a second ask of the same head answered %q, %v, want %q", again, err, id)
}
heads := []string{head2, trunkA2, trunkS2}
for i, h := range heads {
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoA, Head: h, Trunk: trunkA}, at.Add(time.Duration(i+2)*time.Second)); err != nil {
t.Fatalf("try %d: %v", i+2, err)
}
// Issue 1890: tries on a build node (a step of theirs running, or none named yet) and tries waiting for one (queued, or
// their asks not taken) are bounded apart, MaxTries and MaxTriesWaiting; the refusal names both counts.
onNode := func(i int) Try {
return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
Steps: []TryStep{{Ask: "a", State: delivery.StepRunning}}}
}
if _, err := AskTry(store, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute)); err == nil || !strings.Contains(err.Error(), "4 tries wait or run") {
t.Errorf("a fifth try answered %v, want refused naming the four", err)
waitingFor := func(i int) Try {
return Try{ID: fmt.Sprintf("try-%d", i), Repository: repoA, Head: fmt.Sprintf("%040x", i), Trunk: trunkA, State: TryRunning,
Steps: []TryStep{{Ask: "a", State: TryAsked}}}
}
for _, c := range []struct {
what string
holding, wait int
refused string
}{
{"four queued behind the checks, none on a node: a fifth is taken", 0, 4, ""},
{"four on a node", MaxTries, 0, "now 4 run and 0 wait"},
{"three on a node, eight waiting", 3, MaxTriesWaiting, "a new try is refused while 4 tries run on a build node or 8 wait for one; now 3 run and 8 wait"},
} {
kv := newMemKV()
for i := range c.holding + c.wait {
tr := waitingFor(i + 1)
if i < c.holding {
tr = onNode(i + 1)
}
data, _ := encodeTry(tr)
_, _ = kv.Create(context.Background(), tr.ID, data)
}
_, err := AskTry(memTries{kv}, tryTrunks, TryAsk{Repository: repoSDK, Head: head1}, at.Add(time.Minute))
if (c.refused == "") != (err == nil) || (err != nil && !strings.Contains(err.Error(), c.refused)) {
t.Errorf("%s: answered %v, want refused %q", c.what, err, c.refused)
}
}
}
@@ -89,7 +127,8 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
t.Fatal(err)
}
b := newBench(t)
tr := &Tryer{Tries: kv, Judge: b.ex}
written := &writtenTries{memKV: kv}
tr := &Tryer{Tries: written, Judge: b.ex}
raw, _, _, _ := kv.Get(ctx, id)
tr.Put(ctx, id, raw)
tr.Wait()
@@ -97,6 +136,21 @@ func TestATryIsJudgedAsAPullRequestsHeadAndOwesTheForgeNothing(t *testing.T) { /
if got.State != TryPass || got.Took == "" || got.Head != head1 || got.Trunk != trunkA || got.Said == "" || got.Levels[delivery.LevelUnit].Run != 2 {
t.Fatalf("try = %+v", got)
}
// Issue 1890: while it runs, the try names its ask as soon as it is put, and then where its step stands; the end
// keeps each ask's node and state.
if len(written.running) < 3 || len(written.running[1].Asks) != 1 || written.running[1].Steps[0].State != TryAsked ||
written.running[2].Steps[0].Node != "shanks" {
t.Errorf("the running try's writes = %+v, want its ask named while it ran, then its node", written.running)
}
if len(got.Steps) != 1 || got.Steps[0].Node != "shanks" || got.Steps[0].State != delivery.StepPassed {
t.Errorf("the ended try's steps = %+v", got.Steps)
}
// The try names the asks it put, so the delivery view counts their steps as a try's (build_usage).
if len(got.Asks) != 1 {
t.Errorf("asks = %v, want the one check ask the try put", got.Asks)
} else if _, err := delivery.AskStepKey("shanks", got.Asks[0]); err != nil {
t.Errorf("the try names %q, which is no ask id: %v", got.Asks[0], err)
}
fetch := b.steps.tool("fetch")
if len(fetch) != 1 || fetch[0].args["commit"] != head1 || fetch[0].args["base"] != trunkA || fetch[0].args["ref"] != "" {
t.Errorf("fetch = %+v, want the head merged onto the trunk, with no pull request's ref", fetch)
@@ -0,0 +1,78 @@
package check
import (
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Turns holds the gate's merge turns (novox/hq ADR 0548 §3), by repository, as a watch of the gate's turns bucket puts them
// in: the checker claims no check of a repository a turn names, except the check of a waiting delivery's own heads, so an
// approved merge goes before a check not yet started. A bucket not there is read as empty.
type Turns struct {
mu sync.Mutex
byRepo map[string]delivery.Turn
}
// Put takes a value of the turns bucket in, or its delete, and answers whether what the checker holds changed. A value that
// is not a turn is dropped as a delete would be: a turn the checker cannot read holds nothing.
func (t *Turns) Put(key string, value []byte, deleted bool) bool {
repository, err := delivery.ParseTurnsKey(key)
if err != nil {
return false
}
t.mu.Lock()
defer t.mu.Unlock()
if t.byRepo == nil {
t.byRepo = map[string]delivery.Turn{}
}
cur, had := t.byRepo[repository]
if !deleted {
if turn, err := delivery.DecodeTurn(value); err == nil {
t.byRepo[repository] = turn
return !had || !delivery.SameTurn(cur, turn)
}
}
delete(t.byRepo, repository)
return had
}
// Holding names the first of the repositories (by name) a turn holds off the check of the heads, and the delivery that
// waits first in it; false when none does.
func (t *Turns) Holding(repositories []string, heads string) (string, delivery.TurnWait, bool) {
if t == nil {
return "", delivery.TurnWait{}, false
}
sorted := make([]string, 0, len(repositories))
for _, r := range repositories {
sorted = append(sorted, strings.ToLower(r))
}
sort.Strings(sorted)
t.mu.Lock()
defer t.mu.Unlock()
for _, r := range sorted {
turn, ok := t.byRepo[r]
if !ok || !delivery.TurnHolds(turn, heads) {
continue
}
for _, w := range turn.Waiting {
if w.Heads != heads {
return r, w, true
}
}
}
return "", delivery.TurnWait{}, false
}
// All is every turn held now, by repository (a copy).
func (t *Turns) All() map[string]delivery.Turn {
t.mu.Lock()
defer t.mu.Unlock()
out := make(map[string]delivery.Turn, len(t.byRepo))
for r, turn := range t.byRepo {
out[r] = turn
}
return out
}
+17 -6
View File
@@ -38,6 +38,11 @@ type stream struct {
}
type manifest struct {
Claims []struct {
Name string `json:"name"`
Scope string `json:"scope"`
Serves []string `json:"serves"`
} `json:"claims"`
Streams []stream `json:"streams"`
Tools []string `json:"tools"`
Module string `json:"module"`
@@ -93,11 +98,11 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
t.Errorf("module = %q", m.Module)
}
// delivery.CheckerKeptHistory, CheckerOwedHistory, PreviewsHistory, CheckerVerdictsHistory; the names follow the SDK's buckets.
if want := []state{{Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
if want := []state{{Name: "asks", History: 1}, {Name: "hearing", History: 1}, {Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-assigner.compositions", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates",
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-assigner.compositions", "mesh-delivery-assigner.placement", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
@@ -124,11 +129,17 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
t.Errorf("streams = %+v, want %+v", m.Streams, want)
}
// The read tools (issue 1686) are served by the tools bundle, not by the checker's process.
// The try's two tools (ADR 0501 §5) beside them: mesh-controller's try calls them.
if want := []string{"checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
// The try's tools (ADR 0501 §5; the end by hand, issue 734) beside them: mesh-controller's try calls them. A lost check
// is asked again by hand (checker_ask_again).
if want := []string{"checker_ask_again", "checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
for _, k := range []string{"logs", "emits", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
// The try is the checker's own seat's verbs (ADR 0547 §10, issue 774), served beside the tools by the same code.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-delivery-checker" || m.Claims[0].Scope != "mesh" ||
!reflect.DeepEqual(m.Claims[0].Serves, []string{"try", "try-read", "try-end"}) {
t.Errorf("claims = %+v, want the mesh seat mesh-delivery-checker serving try, try-read and try-end", m.Claims)
}
for _, k := range []string{"logs", "emits", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the checker has none", k)
}
+17 -1
View File
@@ -2,6 +2,17 @@
"module": "mesh-delivery-checker",
"version": "1",
"slug": "checker",
"claims": [
{
"name": "mesh-delivery-checker",
"scope": "mesh",
"serves": [
"try",
"try-read",
"try-end"
]
}
],
"consumes": [
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported"
@@ -10,6 +21,8 @@
{ "name": "checks", "events": ["preview.**", "owed.**"] }
],
"state": [
{ "name": "asks", "history": 1 },
{ "name": "hearing", "history": 1 },
{ "name": "kept", "history": 1 },
{ "name": "owed", "history": 5 },
{ "name": "previews", "history": 5 },
@@ -21,10 +34,11 @@
"mesh-build-go.toolchains",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-assigner.compositions",
"mesh-delivery-assigner.placement",
"mesh-delivery-builder.builds",
"mesh-delivery-gate.source",
"mesh-delivery-gate.turns",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
@@ -36,11 +50,13 @@
"node-build-node"
],
"tools": [
"checker_ask_again",
"checker_kept",
"checker_previews",
"checker_pull",
"checker_status",
"checker_try",
"checker_try_end",
"checker_try_read",
"checker_verdicts"
],
@@ -21,6 +21,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream,
@@ -71,11 +72,17 @@ accounts {
"mesh.mod.gitea.tool.gitea_open_pulls"
"mesh.mod.gitea.tool.gitea_open_pulls.*"
"$JS.API.CONSUMER.CREATE.KV_mesh-delivery-forge-bridge_performed.>"
"$JS.API.CONSUMER.CREATE.KV_mesh-delivery-forge-bridge_inputs.>"
"$JS.API.CONSUMER.DELETE.KV_mesh-delivery-forge-bridge_performed.>"
"$JS.API.CONSUMER.DELETE.KV_mesh-delivery-forge-bridge_inputs.>"
"$JS.API.DIRECT.GET.KV_mesh-delivery-forge-bridge_performed.>"
"$JS.API.DIRECT.GET.KV_mesh-delivery-forge-bridge_inputs.>"
"$JS.API.STREAM.INFO.KV_mesh-delivery-forge-bridge_performed"
"$JS.API.STREAM.INFO.KV_mesh-delivery-forge-bridge_inputs"
"$JS.FC.KV_mesh-delivery-forge-bridge_performed.>"
"$JS.FC.KV_mesh-delivery-forge-bridge_inputs.>"
"$KV.mesh-delivery-forge-bridge_performed.>"
"$KV.mesh-delivery-forge-bridge_inputs.>"
"$JS.API.CONSUMER.CREATE.KV_mesh-delivery-gate_merges.>"
"$JS.API.CONSUMER.DELETE.KV_mesh-delivery-gate_merges.>"
"$JS.API.DIRECT.GET.KV_mesh-delivery-gate_merges.>"
@@ -218,7 +225,7 @@ func startBed(t *testing.T, openReply string, prepare func(ctx context.Context,
}
// Its other state bucket, and the gate's buckets the bridge reads, made by the controller from the gate's manifest
// (mesh-sdk go/delivery MergesBucket, OwedBucket with their histories).
for name, history := range map[string]uint8{performedBucket: 1, delivery.MergesBucket: delivery.MergesHistory, delivery.OwedBucket: delivery.OwedHistory} {
for name, history := range map[string]uint8{performedBucket: 1, inputsBucket: 1, delivery.MergesBucket: delivery.MergesHistory, delivery.OwedBucket: delivery.OwedHistory} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: history}); err != nil {
t.Fatal(err)
}
@@ -271,7 +278,7 @@ func startBed(t *testing.T, openReply string, prepare func(ctx context.Context,
}
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(credential{URL: srv.ClientURL(), Node: "n1", Module: "mesh-delivery-forge-bridge", User: "n1.mesh-delivery-forge-bridge", Password: "pw"})
cred, _ := json.Marshal(process.Account{URL: srv.ClientURL(), Node: "n1", Module: "mesh-delivery-forge-bridge", User: "n1.mesh-delivery-forge-bridge", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
@@ -0,0 +1,81 @@
package main
import (
"context"
"encoding/json"
"log"
"regexp"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-forge-bridge/internal/bridge"
)
// inputsBucket is the module's state bucket inputs (manifest state entry inputs): what the bridge read of a module
// directory, its manifest and lock-file hashes, by the directory's git tree hash, so a merge reads the forge only for the
// directories it changed (issue 1126; internal/bridge kept.go). A tree hash names its bytes, so a value is never stale;
// a merge adds a value only for each directory it changed (a few kilobytes each).
const inputsBucket = delivery.ForgeBridgeModule + "_inputs"
// keptTimeout bounds one read or write of the bucket: the bucket is a saving, never a reason to fail a merge.
const keptTimeout = 2 * time.Second
var treeHash = regexp.MustCompile(`^[0-9a-f]{40}$|^[0-9a-f]{64}$`)
// kept is the bridge's Kept: in memory always, and in the bucket inputs when it is on the bus, so the first merge after a
// start reads only what changed too. A value the bucket cannot give or take is read from the forge, and said.
type kept struct {
kv jetstream.KeyValue // nil when the bucket is not on the bus
logger *log.Logger
mu sync.Mutex
mem map[string]bridge.Inputs
}
func newKept(kv jetstream.KeyValue, logger *log.Logger) *kept {
return &kept{kv: kv, logger: logger, mem: map[string]bridge.Inputs{}}
}
func (k *kept) Get(tree string) (bridge.Inputs, bool) {
k.mu.Lock()
in, ok := k.mem[tree]
k.mu.Unlock()
if ok || k.kv == nil || !treeHash.MatchString(tree) {
return in, ok
}
ctx, cancel := context.WithTimeout(context.Background(), keptTimeout)
defer cancel()
e, err := k.kv.Get(ctx, tree)
if err != nil {
return bridge.Inputs{}, false // not kept (or not readable): read from the forge
}
if err := json.Unmarshal(e.Value(), &in); err != nil || in.Manifest == "" {
k.logger.Printf("%s.%s cannot be read, so its directory is read from the forge: %v", inputsBucket, tree, err)
return bridge.Inputs{}, false
}
k.mu.Lock()
k.mem[tree] = in
k.mu.Unlock()
return in, true
}
func (k *kept) Put(tree string, in bridge.Inputs) {
k.mu.Lock()
k.mem[tree] = in
k.mu.Unlock()
if k.kv == nil || !treeHash.MatchString(tree) {
return
}
data, err := json.Marshal(in)
if err != nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), keptTimeout)
defer cancel()
if _, err := k.kv.Put(ctx, tree, data); err != nil {
k.logger.Printf("%s.%s is not kept, so the next merge reads its directory from the forge again: %v", inputsBucket, tree, err)
}
}
@@ -6,18 +6,12 @@ package main
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"strings"
"sync"
"syscall"
"time"
@@ -39,7 +33,7 @@ const eventsStream = "EVENTS"
// The bridge's own stream FORGE takes `mesh.stream.<module>.event.<event>` (mesh-control internal/broker
// modstreams.go ModuleStreamSubject).
func forgeSubject(event string) string {
return "mesh.stream." + delivery.ForgeBridgeModule + ".event." + event
return process.ModuleStreamSubject(delivery.ForgeBridgeModule, event)
}
func main() {
@@ -51,50 +45,6 @@ func main() {
}
}
// credential is the broker file the controller issues to the module: JSON of the bus address, the certificate
// fingerprint to pin, the principal and its password (as the watchman reads it; issue 1242 moves this into the SDK).
type credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint"`
Node string `json:"node"`
Module string `json:"module"`
User string `json:"user"`
Password string `json:"password"`
}
func readCredential(path string) (credential, error) {
raw, err := os.ReadFile(path)
if err != nil {
return credential{}, err
}
var c credential
if err := json.Unmarshal(raw, &c); err != nil {
return credential{}, fmt.Errorf("the broker credential is not one: %w", err)
}
if c.URL == "" || c.User == "" {
return credential{}, errors.New("the broker credential names no bus address or user")
}
return c, nil
}
// pinned accepts exactly the certificate with this SHA-256 and no other.
func pinned(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
if got := "sha256:" + hex.EncodeToString(sum[:]); got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%.23s…)", got)
}
return nil
},
}
}
// delivered is the part of a JetStream message the bridge uses.
type delivered interface {
Subject() string
@@ -117,8 +67,8 @@ type publisher interface {
// handle turns one event into its message on FORGE and acknowledges it. An event it cannot read is terminated, never
// redelivered, and the log names its subject and why. A message it could not write stays unacknowledged: the server
// delivers it again (the server's ack wait is the only clock).
func handle(ctx context.Context, m delivered, pub publisher, trees bridge.Trees, files bridge.Files, logger *log.Logger) {
msg, err := bridge.Convert(m.Subject(), m.Data(), trees, files)
func handle(ctx context.Context, m delivered, pub publisher, trees bridge.Trees, files bridge.Files, kept bridge.Kept, logger *log.Logger) {
msg, err := bridge.ConvertKept(m.Subject(), m.Data(), trees, files, kept)
switch {
case errors.Is(err, bridge.ErrRetry):
logger.Printf("%v; heard again in %s", err, retryDelay)
@@ -146,46 +96,32 @@ func handle(ctx context.Context, m delivered, pub publisher, trees bridge.Trees,
logger.Printf("%s: %s could not be written on FORGE, left for redelivery: %v", m.Subject(), msg.Event, err)
return
}
if msg.Said != "" {
logger.Printf("%s: %s", m.Subject(), msg.Said)
}
_ = m.Ack()
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-forge-bridge: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to hear the bus with")
}
cred, err := readCredential(path)
cred, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
again := make(chan struct{}, 1)
opts := []nats.Option{
nats.Name(cred.Node + "." + cred.Module),
nats.UserInfo(cred.User, cred.Password),
nats.CustomInboxPrefix("_INBOX." + cred.User),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
conn, js, err := cred.Dial(
nats.ReconnectHandler(func(*nats.Conn) {
select {
case again <- struct{}{}:
default:
}
}),
}
if strings.TrimSpace(cred.Fingerprint) != "" {
opts = append(opts, nats.Secure(pinned(cred.Fingerprint)))
}
conn, err := nats.Connect(cred.URL, opts...)
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
}
defer conn.Close()
js, err := jetstream.New(conn)
)
if err != nil {
return err
}
defer conn.Close()
// What the forge holds open, said before any event is heard, and again at each reconnect (ADR 0429 §13).
call := callerOn(conn, cred.User)
// No forge event is heard while the open pull requests are read and said (ADR 0443 §5): events and syncs take turns.
@@ -227,20 +163,21 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
go onReconnect(ctx, again, said, logger)
cons, err := js.Consumer(ctx, eventsStream, cred.Node+"_"+cred.Module)
// What a merge read of each module directory, by its tree (issue 1126): a merge reads only the directories it changed.
inputs := newKept(process.OptionalBucket(ctx, js, inputsBucket, "a merge's module directories are kept in memory only, so the first merge after a start reads every one", logger.Printf), logger)
cons, err := cred.EventConsumer(ctx, js, eventsStream)
if err != nil {
// Essential: the forge's events are what the bridge carries.
return fmt.Errorf("the consumer %s_%s on %s cannot be read (%v): the bridge hears no forge event without it, so it stops; where the bus refused it or did not answer, this module's consumes (made by the controller with its grant) have not reached the bus yet", cred.Node, cred.Module, eventsStream, err)
}
cc, err := cons.Consume(func(m jetstream.Msg) {
cc, err := process.HearEvents(cons, func(m jetstream.Msg) {
hear(&turn, func() {
// Bounded: the turn is held, and no sync or other event goes on without it.
hctx, cancel := context.WithTimeout(ctx, syncDeadline)
defer cancel()
handle(hctx, m, rec, treesVia(hctx, call), filesVia(hctx, call), logger)
handle(hctx, m, rec, treesVia(hctx, call), filesVia(hctx, call), inputs, logger)
})
},
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
}, logger.Printf)
if err != nil {
return err
}
@@ -81,7 +81,7 @@ func TestReadableEventIsWrittenOnForgeAndAcked(t *testing.T) { // 1235/AC2 (unit
})
var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0))
handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if len(pub.subjects) != 1 || pub.subjects[0] != "mesh.stream.mesh-delivery-forge-bridge.event.forge.novox.mesh-delivery.pull" {
t.Fatalf("published on %v", pub.subjects)
}
@@ -97,7 +97,7 @@ func TestReadableEventIsWrittenOnForgeAndAcked(t *testing.T) { // 1235/AC2 (unit
func TestUnreadableEventIsTerminatedAndLogged(t *testing.T) { // 1235/AC5
var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(map[string]any{"head_sha": "short"})}, &fakePub{}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0))
handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if len(pub.subjects) != 0 {
t.Errorf("a message was written: %v", pub.subjects)
}
@@ -112,7 +112,7 @@ func TestUnreadableEventIsTerminatedAndLogged(t *testing.T) { // 1235/AC5
func TestUnwrittenMessageIsNotAcked(t *testing.T) { // 1235 (a failed write is redelivered by the server, not retried here)
var logs bytes.Buffer
m, pub := &fakeMsg{subject: "mesh.mod.gitea.event.pull.updated", data: body(nil)}, &fakePub{err: errors.New("no stream")}
handle(context.Background(), m, pub, nil, someFiles, log.New(&logs, "", 0))
handle(context.Background(), m, pub, nil, someFiles, nil, log.New(&logs, "", 0))
if m.acked != 0 || m.termd != 0 {
t.Errorf("acked %d terminated %d, want neither", m.acked, m.termd)
}
@@ -134,7 +134,7 @@ func TestTreesRefusedLeavesTheEventForTenSecondsAndLogs(t *testing.T) { // 1236/
call := func(context.Context, string, any) (json.RawMessage, error) {
return nil, errors.New("the tree listing of novox/mesh-delivery was cut")
}
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), log.New(&logs, "", 0))
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), nil, log.New(&logs, "", 0))
if len(pub.subjects) != 0 {
t.Errorf("a message was written: %v", pub.subjects)
}
@@ -163,7 +163,7 @@ func TestMergeCallsTheToolOnceAndWritesTheMergedMessage(t *testing.T) { // 1236/
}
return json.RawMessage(`{"root":"r0","trees":{"":"t0","modules/a":"t1"}}`), nil
}
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), log.New(&logs, "", 0))
handle(context.Background(), m, pub, treesVia(context.Background(), call), filesVia(context.Background(), call), nil, log.New(&logs, "", 0))
if len(keys) == 0 || keys[0] != "gitea.gitea_commit_trees" || slices.Index(keys[1:], "gitea.gitea_commit_trees") >= 0 {
t.Fatalf("calls %v, want the trees once and first", keys)
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-forge-bridge
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -4
View File
@@ -1,7 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9 h1:Ufmd2cYEXICQ817IGflccP9bexABE4WhIO49HPbaoSo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022190740-104533e8c0c9/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"path"
"regexp"
"slices"
@@ -53,6 +54,8 @@ var LockFiles = []string{"go.sum", "package-lock.json"}
type Message struct {
Event string
Data []byte
// Said is what the bridge did that the log says, beside writing the message; empty for none.
Said string
}
// The subjects of the forge module's events the bridge writes pull messages for (mesh-catalog modules/gitea,
@@ -77,6 +80,10 @@ type pull struct {
ModuleDirsSaid bool `json:"module_dirs_said"`
Base string `json:"base"`
MergeSHA string `json:"merge_commit_sha"`
// Paths are the files the pull request changes (a removed or renamed one included), PathsTruncated says the forge
// cut the list; nil when the forge did not say them.
Paths *[]string `json:"paths"`
PathsTruncated bool `json:"paths_truncated"`
}
func unreadable(subject, why string, args ...any) error {
@@ -88,6 +95,12 @@ func unreadable(subject, why string, args ...any) error {
// the trunk asks trees once and becomes a merged message; a merge into any other branch is a closed pull message, as the
// pull request left its group. A forge that cannot say is ErrRetry; a merge with no module.json is ErrNoModules.
func Convert(subject string, data []byte, trees Trees, files Files) (Message, error) {
return ConvertKept(subject, data, trees, files, nil)
}
// ConvertKept is Convert; with kept, a merge reads the forge only for the module directories whose tree hash kept
// does not hold, and keeps what it read (kept.go).
func ConvertKept(subject string, data []byte, trees Trees, files Files, kept Kept) (Message, error) {
var action string
switch subject {
case UpdatedSubject:
@@ -114,7 +127,16 @@ func Convert(subject string, data []byte, trees Trees, files Files) (Message, er
return Message{}, unreadable(subject, "no branch")
}
if subject == MergedSubject && p.Base == Trunk {
return merged(subject, p, trees, files)
return merged(subject, p, trees, files, kept)
}
said := ""
if action == delivery.PullUpdated && p.Base != "" && p.Base != Trunk {
// The checker judges pull requests into main only, and source.main names main's head only (novox/hq ADR 0484 §11): a
// pull request into another base is no candidate, as it is none at the start (OpenPulls) and at its merge. It is
// written closed, so a candidate an older head made is taken out, and said (issue 1422).
action = delivery.PullClosed
said = fmt.Sprintf("%s/%s#%d at %s is into %s, not %s: written closed, so it is no candidate; the checker judges pull requests into %s only (novox/hq ADR 0484 §11)",
p.Owner, p.Repo, p.Number, p.HeadSHA[:12], p.Base, Trunk, Trunk)
}
f := delivery.Forge{Kind: delivery.ForgePull, Repository: p.Owner + "/" + p.Repo, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Action: action}
@@ -134,7 +156,7 @@ func Convert(subject string, data []byte, trees Trees, files Files) (Message, er
if err != nil {
return Message{}, unreadable(subject, "%v", err)
}
return Message{Event: event, Data: body}, nil
return Message{Event: event, Data: body, Said: said}, nil
}
// MaxReadsInFlight is how many files of the forge inputsOf asks for at once. A merge into a repository of many
@@ -147,6 +169,7 @@ const MaxReadsInFlight = 8
type read struct {
dir, path string
manifest bool
asked bool
raw []byte
err error
}
@@ -160,6 +183,14 @@ type read struct {
// order the calls finish in, and the error is that of the first failing file in the order of dirs and LockFiles. After
// a failure no further file is asked for, and inputsOf returns only when every call it started has returned.
func inputsOf(owner, repo, ref string, dirs []string, files Files) (manifests map[string]string, locks map[string]map[string]string, err error) {
manifests, locks, _, err = inputsRead(owner, repo, ref, dirs, files)
return manifests, locks, err
}
// inputsRead is inputsOf, and also answers, on a failure too, the directories whose manifest and lock files were all
// read, with their manifests and lock hashes: a merge keeps them by their tree, so its next try reads only the rest
// (issue 1126).
func inputsRead(owner, repo, ref string, dirs []string, files Files) (manifests map[string]string, locks map[string]map[string]string, whole []string, err error) {
at := func(dir, name string) string {
if dir == "" {
return name
@@ -188,6 +219,7 @@ func inputsOf(owner, repo, ref string, dirs []string, files Files) (manifests ma
<-slots
break
}
r.asked = true
wg.Add(1)
go func(r *read) {
defer func() { <-slots; wg.Done() }()
@@ -202,10 +234,16 @@ func inputsOf(owner, repo, ref string, dirs []string, files Files) (manifests ma
wg.Wait()
manifests = map[string]string{}
locks = map[string]map[string]string{}
broken := map[string]bool{} // a directory with a file not read (failed, or never asked after a failure)
for _, r := range reads {
switch {
case r.err != nil && (r.manifest || !errors.Is(r.err, ErrNotFound)):
return nil, nil, fmt.Errorf("%s: %w", r.path, r.err)
broken[r.dir] = true
if err == nil {
err = fmt.Errorf("%s: %w", r.path, r.err)
}
case r.err == nil && r.raw == nil && !r.asked:
broken[r.dir] = true
case r.err != nil: // a lock file the forge does not have
case r.manifest:
manifests[r.dir] = string(r.raw)
@@ -217,17 +255,29 @@ func inputsOf(owner, repo, ref string, dirs []string, files Files) (manifests ma
locks[r.dir][path.Base(r.path)] = hex.EncodeToString(sum[:])
}
}
for _, dir := range dirs {
if !broken[dir] {
whole = append(whole, dir)
}
}
if err != nil {
for dir := range broken {
delete(manifests, dir)
delete(locks, dir)
}
return manifests, locks, whole, err
}
if len(manifests) == 0 {
manifests = nil
}
if len(locks) == 0 {
locks = nil
}
return manifests, locks, nil
return manifests, locks, whole, nil
}
// merged is the merged message of a merge into the trunk: the trees of the merge commit, asked once.
func merged(subject string, p pull, trees Trees, files Files) (Message, error) {
func merged(subject string, p pull, trees Trees, files Files, kept Kept) (Message, error) {
if !sha40.MatchString(p.MergeSHA) {
return Message{}, unreadable(subject, "the merge commit %q is not 40 hex digits", p.MergeSHA)
}
@@ -244,16 +294,28 @@ func merged(subject string, p pull, trees Trees, files Files) (Message, error) {
names = append(names, dir)
}
slices.Sort(names)
manifests, locks, err := inputsOf(p.Owner, p.Repo, p.MergeSHA, names, files)
manifests, locks, toRead := keptInputs(names, dirs, kept)
read, readLocks, whole, err := inputsRead(p.Owner, p.Repo, p.MergeSHA, toRead, files)
if err != nil {
return Message{}, fmt.Errorf("%w: %s: the manifests of %s at the merge commit %s: %v", ErrRetry, subject, repository, p.MergeSHA, err)
keep(whole, dirs, read, readLocks, kept) // what was read whole is not asked again at the next try
return Message{}, fmt.Errorf("%w: %s: the manifests of %s at the merge commit %s (%d of %d module directories read from the forge, the others kept by their tree): %v", ErrRetry, subject, repository, p.MergeSHA, len(toRead), len(names), err)
}
keep(toRead, dirs, read, readLocks, kept)
maps.Copy(manifests, read)
maps.Copy(locks, readLocks)
if len(manifests) == 0 {
manifests = nil
}
if len(locks) == 0 {
locks = nil
}
event, err := delivery.ForgeEvent(repository, delivery.ForgeMerged)
if err != nil {
return Message{}, unreadable(subject, "%v", err)
}
body, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks})
f := delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks}
body, err := delivery.EncodeForge(f)
if err != nil {
return Message{}, unreadable(subject, "%v", err)
}

Some files were not shown because too many files have changed in this diff Show More