Compare commits

...
Author SHA1 Message Date
jschoubben 4b23db28b3 mesh-grants: an open take-over ask is always closed (router record, expiry, failed publish, any outcome), the switch is said recorded not switched, the refusal condition is cleared, the excuse is limited to the claiming machines' subscriptions (issue 1945) 2026-10-26 11:43:39 +01:00
jschoubben b0b1cf4f01 mesh-grants: a retired verb and a multiply claimed mesh seat's added subjects are named, not counted, in the comparison (issue 1945) 2026-10-26 11:34:52 +01:00
jschoubben 9a569bfb8d mesh-grants: derive that a bundle loads from build.artifacts, as the indexer's manifest has no derived bundles, so route-proxy's helper is carried (issue 1945) 2026-10-26 11:33:28 +01:00
jschoubben 75d0eb32be mesh-grants: take-over and hand-back are asked of the operator on their channel and run on their yes after the refusals; take-over no longer always refused; grants_take_over_state (issue 1945) 2026-10-26 11:26:37 +01:00
jschoubben ce72b67cef mesh-grants: hear the condition events through the controller's durable consumer and acknowledge them; the provided mesh-wireguard manifest always wins (issue 1945) 2026-10-26 11:20:42 +01:00
jschoubben 2a94d16834 mesh-grants: ask the controller's root-free verb and wake on its condition events, so a verified sender is composed (issue 1945)
The part composed every membership with no root-free input, so telegram's verified-sender was dropped and the membership held as a shadow
difference. It now asks mesh-controller.root-free for the machines a composition needs, per pass, failing closed and said once, and wakes
on root-not-free condition-raised and condition-cleared.
2026-10-26 11:11:43 +01:00
jschoubben da7baa5e39 mesh-grants: take mesh-wireguard's manifest from mesh-sdk, where the mesh provides it, so its node-tools grants are composed (issue 1945) 2026-10-26 11:11:43 +01:00
mesh-tools 2f697216ee Merge pull request 'Assigner: take away hold, restore and record asks whose fingerprint the trunk left behind (issue 1906)' (#60) from fix/1906-stale-asks into main 2026-10-26 09:49:22 +00:00
jschoubben 769dab43a2 Assigner: keep the held record of a fingerprint whose asks went stale (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2f697216eee4
mesh/check pass 769dab43a24d on 2f697216eee4 | unit 168 run, 488 kept | integration 178 run, 554 kept | composition passed on ace
The held record is the memory that a fingerprint failed (ADR 0408 section 4); a revert returns the
trunk to exactly it, and without the key rule 3 would move the canary onto the failed build with
nothing asked. A stale hold ask now clears only the condition (raised again on a revisit), and the
hold is asked again under a new asking. Also read the single want key for a record ask.
2026-10-26 10:33:30 +01:00
jschoubben 98a05997ae Assigner: keep a held-back node's record ask, and clear the held record and condition of a stale hold (issue 1906)
Review round 1: a record ask is about the composition of any fingerprint a node may still want
(trunk, want, have), not only the trunk's, so deleting it looped with the wants rule that asks it
again and the operator was never asked. It is now stale only when the module left the trunk or no
read composition of those fingerprints has its digest. A hold or restore ask taken away as stale
also takes its held record and clears its held condition.
2026-10-26 10:31:01 +01:00
jschoubben 07dee528c5 Assigner: take away hold, restore and record asks whose fingerprint the trunk left behind (issue 1906)
An ask outlived its fingerprint: a lift left the restore asks of its fingerprint on every node,
and no pass removed an ask of a fingerprint that is no longer the module's trunk fingerprint,
so the operator kept being asked about a module that runs the trunk healthy. Each pass now
takes those asks away (the teller then withdraws the router ask), and a lift takes the restore
asks of its fingerprint. Nothing is taken away while the trunk or a composition is unread.
2026-10-26 10:25:50 +01:00
mesh-tools 7ddbf40715 Merge pull request 'mesh-grants: read manifests as the indexer does and say every refused one, so claude-code counts as placed (issue 1945)' (#59) from fix/1945-grants-inputs into main 2026-10-26 09:00:58 +00:00
jschoubben aea22bf793 mesh-grants: read manifests for their inputs, as every delivery part does, not against a list from the controller
mesh/delivery merged: source.main moved to novox/mesh-delivery@7ddbf40715ff
mesh/check pass aea22bf793b4 on 7ddbf40715ff | unit 168 run, 479 kept | integration 178 run, 545 kept | composition passed on novox
The part reads seats, buckets, logs and streams, never a config value, and
the merge gate already judged the fallbacks; reading the controller's
predefined mirror would tie the part to the leaving controller (ADR 0547
clause 1). ParseManifestForInputs takes a fallback as it declares itself.

Issue: 1945
2026-10-26 09:36:18 +01:00
jschoubben 72889be87f mesh-grants: judge manifests against the predefined keys on the bus, and say every refused one
A manifest falling back to a predefined key (claude-code, dunst) was parsed
with no list, refused silently, and its module counted as placed nowhere, so
three memberships would lose claude-code's reach addresses and were held back.
The part now reads the list the controller mirrors (mesh-controller.predefined,
key list), follows it, and lists every manifest it cannot read in its gaps,
its pass (refused) and its log.

Issue: 1945
2026-10-26 09:33:29 +01:00
mesh-tools 86db64904b Merge pull request 'mesh-grants: create a declared module's missing buckets, logs and streams on the event; publish memberships past retired seat verbs (issues 1790, 1945)' (#58) from fix/1790-grants-create-buckets into main 2026-10-26 08:17:22 +00:00
jschoubben 12f16bbebd mesh-grants: create a declared bucket, log or stream the bus lacks, so a new module state no longer waits for a controller start
mesh/delivery merged: source.main moved to novox/mesh-delivery@86db64904b2a
mesh/delivery-group group fix/1790-grants-create-buckets merged: every member is merged
mesh/check checking 12f16bbebd40 on 13380e3d0e52 | novox/mesh-delivery#58 in group fix/1790-grants-create-buckets
A module's new state reached the bus only when the controller started (issue 1790, met three
times: puts, queues, mesh-vault's bus-accounts). The operator allowed one more grant in the
controller's composer on 2026-10-26 08:54, creation only: the part's process now creates each
bucket, log or stream a placed module's manifest declares and the bus lacks, with the same
configuration the controller asserts (mesh-sdk BucketStreamConfig, Log.Config,
ModuleStreamConfig), on the event that brought the manifest, never on a timer. It says asserted
and logs one line per module; a stream the bus holds is never updated or deleted, its difference
only said; a create refused is said and tried at the next event; one created at this manifest
and gone since is made again only by hand. grants_assert asks the process to do the same for one
module now. The user list, the grants and any change to a held stream stay in shadow. mesh-sdk
moves to fix/1790-grants-create-buckets, whose composer grants this, so the shadow list is
composed as the controller's.

Issue: 1790
2026-10-26 09:11:23 +01:00
jschoubben 4ec40eec9a mesh-grants: publish a membership that drops only a retired seat verb, and say every one held back
The guard that holds back a membership losing anything held back five since 01:24: the frozen
controller's memberships still serve the delivery seat's mark-ready, which mesh-sdk d506f21
retired on purpose (ADR 0421), so every membership composed with today's seat catalogue looked
like a loss. A seat verb the catalogue defines no longer is now dropped from the comparison and
named in the pass (retired); any other loss, a module's own seat's verbs included, is still
refused. The log named only the first held membership: it now says each one, with what it would
lose, whenever the set changes, and grants_membership with neither node nor module lists them.

Issue: 1945
2026-10-26 09:11:03 +01:00
mesh-tools 13380e3d0e Merge pull request 'mesh-grants: the shadow user list takes each user's hash from the vault and says per user whether one is held (issue 1945, ADR 0547 clause 9)' (#57) from fix/1945-vault-hashing into main 2026-10-26 07:48:07 +00:00
jschoubben 9132b6a296 mesh-grants: take each user's hash from the vault's bus-accounts and say per user whether one is held, so take-over waits until every user has a credential
mesh/delivery merged: source.main moved to novox/mesh-delivery@13380e3d0e52
mesh/check pass 9132b6a296b9 on 13380e3d0e52 | unit 168 run, 473 kept | integration 178 run, 539 kept | composition passed in 36ms on novox
ADR 0547 clause 9: the vault writes the hash of each bus account it holds into bus-accounts.<user>. The grants part
watches that state (no timer), carries the hash with each shadow user in memory only, and its user-list tool says per
user whether a hash is held and how its line would differ from the loaded one (same, adds, left out for want of a
credential, loses its login); never the hash, since bcrypt salts make two hashes incomparable. Take-over names the
users with no hash.

Issue: 1945
2026-10-26 02:29:54 +01:00
mesh-tools bcb5aafdbe Merge pull request 'Builder: given-up asks no longer hold a re-ask back for ever, and a tool reads a build seat's queue (issue 1945)' (#56) from fix/1945-reask-dead-ahead into main 2026-10-26 00:24:30 +00:00
jschoubben 3e8bd9b0de Builder: count only deliverable asks ahead, drop the old message of an ask asked again, and read a seat's queue (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bcb5aafdbe80
mesh/check pass 3e8bd9b0def2 on bcb5aafdbe80 | unit 168 run, 472 kept | integration 178 run, 538 kept | composition passed on shanks
mesh-delivery #51's re-ask never fired for mesh-build-node.4e7b118e: messages the
SEAT_NODE_BUILD_GO workers had given up on stay on the work queue, and each counted
as waiting ahead, so the ask was never asked again (note 62). A message ahead now
waits only when no node wrote its step and no ask of its kind asked after it was
taken; the old message of an ask asked again is deleted from its stream; and
builder_seat_queue (answered by the builder through its queues state, as only its
account may read the stream) shows each message and whether it still waits.
2026-10-26 01:19:30 +01:00
mesh-tools b842b9422d Merge pull request 'Builder and checker read tools read only what they answer, in parallel and bounded (issue 1827)' (#55) from fix/1827-bounded-reads into main 2026-10-25 23:44:25 +00:00
jschoubben 9db15c7cf4 Builder and checker read tools read only the keys they need, in parallel, at most 1000 a call (issue 1827)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b842b9422d2a
mesh/check checking 9db15c7cf4fc on b842b9422d2a
builder_stuck, checker_kept, checker_verdicts, checker_status, checker_previews, checker_pull and checker_try listed a
bucket of thousands of keys and read each value (builder_stuck also its history) one round trip at a time before
filtering, so they timed out at 30 s. Now keys are narrowed by what the key carries first, history is read only for rows
kept, values are read sixteen at a time, at most 1000 a call with the answer saying it was cut, steps newest first, a new
try reads only the newest 64 tries, and each tool logs the values read and the time taken. Same kind as the view's fix
(PR 52); the helper is copied, not put in mesh-sdk, as a few lines do not earn a group delivery.
2026-10-26 00:39:08 +01:00
jschoubben ba47b47e24 Merge pull request 'mesh-delivery-view: pin mesh-sdk at its main so a have's user-list field decodes (issue 1945)' (#54) from fix/1945-view-sdk-pin into main
Reviewed-on: #54
2026-10-25 23:06:21 +00:00
jschoubben b566563dc5 mesh-delivery-view: pin mesh-sdk at its main so a have's user-list field decodes (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ba47b47e2421
mesh/check pass b566563dc592 on ba47b47e2421 | unit 576 run, 63 kept | integration 177 run, 538 kept | composition passed on shanks
2026-10-26 00:01:12 +01:00
jschoubben 2a856e2a76 Merge pull request 'mesh-grants: publish memberships for new placements now; the user list, assertions and grants in shadow (ADR 0547 milestone 1)' (#53) from fix/1945-memberships into main
Reviewed-on: #53
2026-10-25 22:44:11 +00:00
jschoubben ce7d60cb6e Merge pull request 'The delivery view reads only the keys a call names, in parallel, so node_haves and assigner_wants answer within their time (issue 1945)' (#52) from fix/view-reads-fast into main
Reviewed-on: #52
2026-10-25 22:03:21 +00:00
jschoubben 26ba7404bf Pin mesh-grants to mesh-sdk b7b5c6e, whose composer grants it reading every membership (review of mesh-grants, B1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2a856e2a76ea
mesh/delivery-group group fix/1945-memberships merged: every member is merged
mesh/check checking 26ba7404bfc4 on ce7d60cb6ed0 | novox/mesh-delivery#53 in group fix/1945-memberships
2026-10-25 22:54:01 +01:00
jschoubben b4af9cdb7a mesh-grants publishes only a first or an additive membership, and holds no lock across the bus (review of 1ff5049)
Composed without a root-free judgement, a membership would drop a kind's verified-sender and the
operator's channel approvals would stop being honoured; any loss now stays a shadow difference,
named. A tool no longer waits on a pass's round trip. compose-now says the controller serves no
verb to write a module's grant again.
2026-10-25 22:52:00 +01:00
mesh-tools db53ec975f Merge pull request 'Delivery parts keep running when the bus refuses a bucket they can work without; the builder asks a lost build again; the assigner says why a node gets no want (issue 1945)' (#51) from fix/1945-ask-again-and-why into main 2026-10-25 21:43:11 +00:00
jschoubben 1ff5049d03 mesh-grants publishes memberships on change; the user list, assertions and grants stay in shadow (issue 1945)
Since release 6 nobody publishes memberships, so a module placed since cannot serve its seat
or reach a new state (the operator, 2026-10-25 22:31). The part publishes each membership on
the assignments stream only when it differs from the one last published, composes from the
manifest the wanted build names, and says a refused publish and tries again at the next event.
Pins mesh-sdk 881da7f, whose composer grants mesh-grants that publish.
2026-10-25 22:37:06 +01:00
jschoubben 41f25d3129 Read only the keys a view tool needs, in parallel, so node_haves and assigner_wants answer within the call's 30 s
mesh/delivery merged: source.main moved to novox/mesh-delivery@ce7d60cb6ed0
mesh/check pass 41f25d3129fd on 73be3df28cd4 | unit 21 run, 579 kept | integration 21 run, 655 kept | composition passed on shanks
On 2026-10-25 node_haves and assigner_wants timed out at 30 s four times, even narrowed to one node and one module,
while assigner_want answered at once. Each listed every key of its bucket and then read every value one after the
other, filtering only afterwards; every value is a round trip from the tool through the runtime to the bus, which
opens the bucket and gets the key, so a few hundred keys outlasted the call. Now a node and a module read that one
key directly, a node or a module alone filters the keys before any value is read, and the values are read sixteen
at a time, at most 1000 a call, the answer saying when it was cut. indexer_modules, indexer_candidates and
builder_builds read their values the same parallel way. The kind: a read tool that reads a whole bucket key by key;
every list tool of the view is covered.
2026-10-25 22:33:02 +01:00
jschoubben 4628ba26b4 Delivery parts: take the essential and optional bucket opens from mesh-sdk (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@db53ec975f01
mesh/delivery-group group fix/1945-ask-again-and-why merged: every member is merged
mesh/check pass 4628ba26b41a on 73be3df28cd4 | unit 167 run, 443 kept | integration 177 run, 509 kept | composition passed on 3 machines
The open helpers were copied into seven parts; code shared between
modules lives in mesh-sdk, so they are mesh-sdk process's now and the
parts are pinned at fix/1945-optional-buckets (e0d1de04), moved to its
main once it merges, one delivery group with mesh-sdk.
2026-10-25 22:30:22 +01:00
jschoubben 20027e53ff Delivery parts: a bucket the bus refuses at start stops a part only when it cannot work without it (issue 1945)
Met twice on 2026-10-25: the builder of #48 exited on a refused read of
the gate's source and crash-looped, and its new reasks bucket would have
done the same. A grant reaches the bus with a delivery, so a refusal or a
timeout at start is often only a grant not there yet; exiting for it
stopped the whole part until a hand step on the bus.

Every startup open in the parts is now essential or optional. Essential
ones (a part's own state and the input it exists for) still stop the
start, with an error naming the bucket and its grant. Optional ones are
said and left: watches the bus refuses too. The builder builds trunk keys
without the gate's source, which it opens again at the next candidate
that asks a trunk head; toolchains and previews open again at their next
event, as a missing bucket did. builder_ask_again says that asking a
running step again builds it twice.
2026-10-25 22:29:31 +01:00
jschoubben d4a054bfed Assigner: say why a node gets no want, in the journal and in its removed want (issue 1945)
A new assignment whose trunk build was not done fell through the rules
with nothing said: no want, no neither-list entry, no log line, and the
rule holding it back was computed and dropped. The pass now logs the rule
when a node's reason appears or changes and when it has a want again, one
line per change. A node whose want is removed and that does not run the
module gets the reason in that removed want's words, so assigner_want
answers why; a node with no key gets no removed want, since its node agent
would then remove the module and stop holding its resources back.
2026-10-25 22:29:03 +01:00
jschoubben c623e8d235 Builder: ask again an ask its stream can no longer answer, and give it tools to read and ask again (issue 1945)
A build ask its seat's stream gave up on, dropped or ended with no step was
waited for with no end, and on restart taken over and waited for again: no
build node would ever answer it, and nothing could ask it again (trunk key
of mesh-build-node at 4e7b118e, building since 19:06; seven keys at 18:44).

At the step bound, a context's bound and no timer of its own, the builder
reads what the bus says of an ask with no step: gone from its stream, or
first in line while a node took a later ask. Such an ask is asked again
under a new id the key names, logged and said as build.asked-again, at most
three askings, then the key fails with why. The bus's max-deliveries
advisory would tell it sooner, but the builder is granted neither it nor the
workers' information, and the controller is not changed here.

builder_stuck reads every waiting key with its asks, askings and steps;
builder_ask_again writes the new reasks state, which the builder watches
and answers.
2026-10-25 22:29:03 +01:00
jschoubben 998fe3a515 Add mesh-grants in shadow: compose the bus's user list, memberships and assertions from the bus and compare, writing only its own state (ADR 0547 clauses 5-8, issue 1945)
The grants part must compose with the controller's own composer before it can take over the
list; in shadow it proves the two agree, names every input the bus does not carry yet, and
takes no effect. It claims no seat (mesh-sdk's seat list does not name mesh-grants for the
running composer), serves module tools, and keeps running when a read or write is refused.
2026-10-25 22:27:47 +01:00
mesh-tools 73be3df28c Merge pull request 'The assigner writes wants whatever keeps its puts from opening, and never moves a held put's want (ADR 0546, issue 1945)' (#50) from fix/assigner-starts-without-puts into main 2026-10-25 21:22:55 +00:00
jschoubben eb65535f45 A want marked put and held stands while its put is not heard: the rules give it back only on a put heard released, so a start before the puts are open moves no node (ADR 0546 clause 5)
mesh/delivery merged: source.main moved to novox/mesh-delivery@73be3df28cd4
mesh/check pass eb65535f45db on 73be3df28cd4 | unit 167 run, 433 kept | integration 177 run, 499 kept | composition passed on ace
2026-10-25 22:11:44 +01:00
jschoubben fa6cdbbe61 The puts' watch never gates the assigner's first pass: it is not waited for at the start, and one that ends before it handed what stands only stops the puts (ADR 0546) 2026-10-25 22:06:36 +01:00
jschoubben 462ec9cfb1 The assigner opens the puts' bucket again on a later pass once it is there and granted, so a put works without a restart (ADR 0546, ADR 0201) 2026-10-25 21:59:24 +01:00
jschoubben 9950689a11 The assigner starts and writes wants whatever keeps the puts' bucket from opening or being watched (not there, not granted, no answer): it takes no put further until it starts again, said in its journal (ADR 0546) 2026-10-25 21:58:36 +01:00
mesh-tools 71a6a5bf77 Merge pull request 'Put a version of a module on a node with one tool call: the assigner's put, release and puts (ADR 0546)' (#49) from feat/put-version into main 2026-10-25 20:52:05 +00:00
jschoubben 22d4cbd144 The assigner's start test expects every state its manifest declares, read from module.json, not a count: the state puts broke the count of four (ADR 0546)
mesh/delivery merged: source.main moved to novox/mesh-delivery@71a6a5bf7717
mesh/check pass 22d4cbd14423 on 82180bffd2c6 | unit 22 run, 578 kept | integration 251 run, 425 kept | composition passed on 2 machines
2026-10-25 21:48:32 +01:00
jschoubben 7b618b2317 Pin the assigner and the teller to mesh-sdk main db26f7e, which carries the put (ADR 0546)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestTheAssignerStartsAndSaysWhatItKeeps: the assigner said {V:1 Modu…
2026-10-25 21:37:17 +01:00
jschoubben d8477c71b1 A put asks under its own asking and writes its want only while it stands: an earlier put's answer never counts, a put released meanwhile writes nothing, the bus's want is written only for the act the yes bound, an expired ask refuses the put, and a warrant not recorded is said in the put's words (ADR 0546, review round 1) 2026-10-25 21:36:38 +01:00
jschoubben 45fae9da8a Serve the put as the assigner's own tools from the compositions and grants the controller already writes, with no seat and no compose verb: the controller takes no change (ADR 0546) 2026-10-25 21:36:38 +01:00
jschoubben 2f742fd3fe Put a version of a module on a node with one verb: the assigner's seat put, release and puts, held until released, the bus on the operator's warrant after a snapshot (ADR 0546) 2026-10-25 21:36:38 +01:00
mesh-tools 82180bffd2 Merge pull request 'Indexer: write a module's value only when its fingerprint changes, so a catalogue merge no longer floods the take-in (issue 1945)' (#48) from fix/indexer-writes-on-change into main 2026-10-25 20:35:24 +00:00
jschoubben ac82187b85 Indexer integration test: a disagreement keeps a computed fingerprint, as issue 1940's fix decided
mesh/delivery merged: source.main moved to novox/mesh-delivery@82180bffd2c6
mesh/check pass ac82187b852c on 82180bffd2c6 | unit 167 run, 432 kept | integration 177 run, 498 kept | composition passed on 2 machines
The test predates the fix that keeps a held fingerprint while builders disagree (#47, 3e7f852) and still asserted the
blanking that fix removed, so it fails on main 4efe6cf as well. It now asserts the fingerprint is kept after n2's
assignment, and that a merge moving avahi during the disagreement gets none until the builders agree.
2026-10-25 21:28:03 +01:00
jschoubben 8d38c6bac9 Write the source revision a candidate was previewed onto, so the builder merges its preview onto that trunk head
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
Candidate.Base was never written non-zero (review round 2): the builder's base lookup would have found no trunk for any
live candidate and built no preview at all. The indexer now writes, whenever it writes a candidate's fingerprints, the gate's
source revision whose trunk heads it read the previews onto; that head of the pull request's repository is the preview's
trunk. No mesh-sdk change: the field and its meaning were there. The builder's test now takes the candidate the indexer's
own test writes (testdata/candidate-written.json, kept equal to it), not a hand-made one.
2026-10-25 21:16:20 +01:00
jschoubben 9afef800c0 Merge a candidate's preview onto the trunk head it was previewed onto, not the module value's commit
The builder took a candidate build's merge base from the module's value commit. Now that a value keeps the commit its
fingerprint first appeared at, that commit can be many merges behind the trunk, so a preview would be built onto a stale
trunk (review round 1). The base is now the head of the pull request's repository in the gate's source revision the
candidate names (Candidate.Base): the builder reads mesh-delivery-gate.source, watched with its history (a direct get of
one revision is not what a read grants), and a candidate heard before its revision is built once the revision is heard.
2026-10-25 21:13:59 +01:00
jschoubben eaf27f36af Write a module's value only when the module changed, so a merge no longer rewrites every module of its repository
The operator, 2026-10-25: only if a module has a new version in commit x is a new value written naming commit x. A merge
wrote every module of the repository at the new commit, fingerprint unchanged (about 186 keys per mesh-catalog merge), and
the controller's take-in took each in again: twice tonight it flooded. Now a merge that gives a module the value it holds
but for the commit writes nothing, and the value keeps the commit its fingerprint first appeared at. The kind: a write
that differs only in where it was computed. The other instance, a source revision re-stamping Src on every module at its
repository's head after any merge anywhere, now stamps Src once. The trunk head stays the gate's source.main.
2026-10-25 21:13:59 +01:00
jschoubben 4efe6cfb26 Merge pull request 'Builders: a stale toolchain report no longer counts, and a disagreement no longer blanks a module already fingerprinted (issue 1940)' (#47) from fix/1940-stale-toolchain-record into main
Reviewed-on: #47
2026-10-25 20:09:27 +00:00
jschoubben 3e7f852472 Count a build node's toolchain report only once written since its assignment, and keep a computed fingerprint while builders disagree (issue 1940)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
On 2026-10-25 20:56 assigning mesh-build-node to ace again brought ace's report of 2026-10-23 (v26.12.0) back into the
agreement before the module ran there; the indexer blanked every module's fingerprint to "builders disagree" and the
controller's take-in refused modules already computed and built.

- The toolchain gate (indexer, builder, checker; still one file in three) leaves a report out, said once, until its node
  writes it since its current assignment: the assignment's own time, or the revision that opened it.
- The indexer keeps a module's held fingerprint while a language has no agreed toolchain, when its kept inputs with the
  toolchains it was computed with give it again; only a new fingerprint (a merge, a base, an agreed toolchain moved)
  needs the agreement.
2026-10-25 21:03:16 +01:00
jschoubben 5ba6881f9f Merge pull request 'Assigner rule 6 per node: a node is granted only when the grant names its own account (issue 1945 note 35)' (#46) from fix/1945-credential-when-row-without-seal into main
Reviewed-on: #46
2026-10-25 19:31:51 +00:00
jschoubben 61196c99e6 Hold a node's want until the grant names its account (issue 1945, rule 6 per node)
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 61196c99e6a4 on 5ba6881f9fd3 | unit 163 run, 421 kept | integration 173 run, 487 kept | composition passed on ace
Rule 6 read only whether a grant of the module's fingerprint existed. A grant
written for the module's other nodes then let a node whose account was never
issued or sealed be wanted: on 2026-10-25 at 20:04 ace's mesh-build-go was
wanted with no ace.mesh-build-go on the bus, and crash-looped 20 times for
its missing credential. A grant that names accounts now grants a node only
when it names that node's own or a process's account; one that names none
grants every node, as before.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 20:13:39 +01:00
jschoubben 250364a57b Merge pull request 'The assigner follows a kept composition's new digest, so a want held at an earlier build gets today's config (issue 1945, ADR 0545)' (#45) from fix/1945-kept-compositions-follow-config into main
Reviewed-on: #45
2026-10-25 18:38:03 +00:00
jschoubben 73779a5dd1 Let a want held back at an earlier fingerprint follow that fingerprint's new revision (issue 1945, ADR 0545)
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
The controller now composes a module's kept compositions again when its config
changes, so going back keeps today's config. A want held back at an earlier
fingerprint kept the digest it was first written with, and the node applied
the old revision from the bucket's history: on g14 a rollback of mesh-build-go
put its capacity back to 0. The want now names the new digest, asked through
the record hold as any move of a record module is. A want held at F by the
rules still waits for them, as a new fingerprint does (ADR 0539 §5).
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:24:44 +01:00
jschoubben ec76ab77c0 Merge pull request 'Assigner: a held module is on the held list, never on neither, and a neither from before a restart is cleared (issue 1945)' (#44) from fix/held-is-not-neither into main
Reviewed-on: #44
2026-10-25 16:11:33 +00:00
jschoubben 91d041b62a mesh-delivery-assigner: a held module is never on neither list (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 91d041b62a1a on ec76ab77c058 | unit 163 run, 421 kept | integration 173 run, 487 kept | over bound: integration (slowest integration m…
systemd-resolved, held on shanks by the operator under rule 2, was raised
as an urgent "on no list" condition and reminded every hour, although its
hold is already said and answered on the held list. A node now goes on
neither list only when it gets no want and the module is not held, by a
hold read at its fingerprint or one this pass raises. The first pass after
a start clears the neither condition of every assigned node not on neither
list, not only those with a want, so a neither raised before the restart
for a held module (or a move waiting for its record ask) is cleared too.
2026-10-25 17:07:50 +01:00
jschoubben 51444a8341 Merge pull request 'mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the runtime's line or say so (issue 1901)' (#43) from fix/1901-package-cause into main
Reviewed-on: #43
2026-10-25 15:07:47 +00:00
jschoubben dd9138310c mesh-delivery-checker: a package sent as its FAIL line alone is still worded, and the words keep their bound exactly (issue 1901, review round 1)
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass dd9138310c48 on 51444a834173 | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
The build module on main sends a timeout, exit or kill as only 'FAIL <pkg>':
dropping summary lines left empty words. They now say the package failed with
no cause sent. Causes match at column 0 only, summary lines only in go's exact
forms, and the words count the cause, the marker and the newlines within
MaxWords.
2026-10-25 15:58:09 +01:00
mesh-tools 653da54c0d Merge pull request 'Assigner: a running node gets its first want past the ordering rules, a module on neither list is raised, and a failed health holds only once a node is done moving (issue 1945)' (#42) from fix/r6-no-module-in-neither into main 2026-10-25 14:56:26 +00:00
jschoubben f72f57f6e2 mesh-delivery-checker: with no cause told apart, a race report is the cause (issue 1901)
An older build module's output for a race after the tests' PASS is now worded
by the race's heading, access and frame, not by go test's PASS or the report's
separator line (1901/AC5).
2026-10-25 15:52:21 +01:00
jschoubben 975f0ed654 Assigner: a hold lifts over haves not running, the first pass after a start clears stale conditions, and a first want after a removed want is asked (issue 1945, review round 2 of 3d7d65ed)
mesh/delivery merged: source.main moved to novox/mesh-delivery@653da54c0da3
mesh/check pass 975f0ed65460 on 653da54c0da3 | unit 154 run, 419 kept | integration 164 run, 485 kept | composition passed on ace
Phase guard, same kind as the failed-health hold. allRunHealthy counted
every have whatever its phase, so a node with a removed, held or still
moving have kept a hold up for ever: it counts only phase running (healthy
or not) and failed (not healthy). The failed-health hold already ignores
removed and held haves (phase running or failed only).

Stale conditions (B1). The neither and held conditions this process said
lived only in its memory, so one raised by a previous process stayed urgent
for ever once the module had its want or was no longer held. The first pass
after a start clears, once, the neither condition of every assigned module
with a want and not on neither list, and the held condition of every
module's newest fingerprint that is not held; a clear of a condition that is
not open is quiet at the router. A hold raised again forgets that it was
cleared, so a later release clears it again.

Record ask (review note 3). mustAsk wrote no ask when the want before was
removed, so the first-want path could write a recreates or moves-data
composition of a policy-record module with no ask. A removed want is now
no want, and the first want after it goes through the record ask.

Tests first, each seen failing: a hold lifted over removed, held and moving
haves; the record ask after a removed want; a fresh pass clearing what a
previous process raised, once.
2026-10-25 15:47:57 +01:00
jschoubben 706169aa8a Assigner: a failed health holds a fingerprint only once the node is done moving (phase running or failed)
On novox at 15:27:36 a have reading "health failed, phase applying" at the
controller's new fingerprint held its canary two seconds before the new
build was healthy: Wants held on a failed health whatever the phase. A
failed health while the node backs up, applies or checks is a step of the
move, not its outcome, so it holds nothing now.

Every other Health read in the assigner (rule 3's healthy, rule 5's
providedHealthy, the world's lastHealthy) already asks for phase running.
Test from the live row first, seen failing before the fix.
2026-10-25 15:39:45 +01:00
jschoubben 6d2d50b1e1 mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the panic line (issue 1901)
A try of mesh-controller's status read 'cmd/mesh-controller: PASS': the build
module sent a package failure whose output began with go test's PASS. The
checker now drops go test's summary lines, takes the first panic, fatal error
or signal line as the cause when the build module told none apart (read before
the words are cut), says the tests passed and the program failed after them
when nothing else is left, and cuts a cause to 1 KB so the words keep their
bound (1901/AC5).
2026-10-25 15:38:40 +01:00
jschoubben 3d7d65edce Assigner: a node running an old-form build gets its first want at F once the canary runs F healthy (issue 1945)
An old-form build (not src2:) has no composition, so no want can ever name
what such a node runs. Design limit, decided by the lead (lead-47): its first
want names the newest F, by every rule as usual - the canary (rule 3: this
node is the canary, or the canary runs F healthy), everything F needs (built,
not held, granted, composed) and the providers (rule 5). Otherwise nothing is
written and the neither condition stays. The canary stays in force, and
old-form modules (clipmenu and screen-lock on g14) leave neither list by
themselves, without a hand-written want.

The code already behaved so (an old-form have falls through to Holds); the
case is now named in Wants, and proven: a unit test for both sides of the
canary, the providers and the grant, and the integration test gives g14 F
and clears its neither condition once the canary ace runs F healthy.
2026-10-25 15:32:35 +01:00
mesh-tools 06fd53e5cb Merge pull request 'List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)' (#41) from fix/r6-hold-verbs into main 2026-10-25 14:30:12 +00:00
jschoubben 81182e9dcd Assigner: a running node's first want names the fingerprint it runs, never a newer one, and only a node that runs the module is on neither list (issue 1945, review of f7b23033)
A first want at the newest fingerprint passed over the canary for every module right after the switch. It now names what the node
runs: F when it runs F (by the rules without which there is nothing to want), or an older fingerprint of the new form whose
composition and grant exist, held back from F by the rule; otherwise nothing is written and the node stays on neither list. A new
assignment, with no have, waits for the rules and raises no condition (1945/AC19-AC21).
2026-10-25 15:26:03 +01:00
jschoubben f7b23033a2 Assigner: the integration test of wants on the bus proves a running node's first want and the neither condition; held-condition reads filter by key (issue 1945, try-1792937443523744996) 2026-10-25 15:18:07 +01:00
jschoubben cf26e95691 A view answer rewritten with the same option changes nothing; pin a unit test's clock against the integration tests' (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@06fd53e5cbf7
mesh/check pass cf26e956915c on 06fd53e5cbf7 | unit 154 run, 408 kept | integration 164 run, 474 kept | composition passed in 1ms on 2 machines
2026-10-25 15:15:40 +01:00
jschoubben 8daa5977f9 Assigner: a node running a module with no want gets its first want past the ordering rules, and a module on neither list is raised (issue 1945)
The node's own composition gives up every module with a fingerprint of the new form, so such a module is applied only through its
want. The assigner wrote no want while a rule held it back and the node had none, so clipmenu, screen-lock, i3status-rust and every
module requiring x11-display lay on neither list on g14 and shanks: xorg's health is unknown (rule 5) and the canary never ran them
healthy (rule 3), and their files were removed and never made again.

A node that runs a module and has no want for it now gets its first want by the rules without which there is nothing to want (built,
not held, granted, composed); the canary and providers rules order every move after it. Any module still on neither list is raised
as an urgent condition naming module, node and rule, written to the journal, and cleared once it has a want (1945/AC19-AC21).
2026-10-25 15:10:34 +01:00
jschoubben 30ca8a22bc List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)
A hold answered keep on the phone could not be changed: its ask ended with the
answer, nothing listed it, and a kept hold is never lifted by itself. The view
lists every hold with its node, cause and answer (mesh-delivery-cli holds) and
answers the standing hold ask with hold-release or hold-keep through its one
write; the teller takes a later view answer that changes a hold's option into
the answer of that asking, never an older one.
2026-10-25 15:09:40 +01:00
mesh-tools 97e605edb0 Merge pull request 'Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, and keep its cause (issue 1906)' (#40) from fix/r6-hold-asks into main 2026-10-25 13:27:29 +00:00
jschoubben 7ef796b04b Format the assigner's integration test (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@97e605edb0ad
mesh/check pass 7ef796b04b92 on 97e605edb0ad | unit 154 run, 404 kept | integration 164 run, 469 kept | composition passed on 2 machines
2026-10-25 14:16:58 +01:00
jschoubben ec93585736 Ask a hold again after a restart, leave a merge ask room, take an answer only for its asking (issue 1906, review)
A hold written again after a lift and a restart reused its first ask id, which
the router drops; the hold's time is now its asking. Seven holds could take the
router's three places for the teller and keep a merge Approve from being asked;
the teller keeps at most two assigner asks open. A release or keep answered for
one hold carried over to a later hold of the same fingerprint; an answer now
counts only for the asking its warrant names.
2026-10-25 14:15:44 +01:00
jschoubben 6b63f13da6 Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, keep the hold's cause (issue 1906)
Seven modules stayed held on every machine and the operator was never asked:
the teller only filled assigner asks from the view, a single failed check held
a fingerprint for good, and the held record kept no cause. The teller now asks
every assigner ask as it asks merge asks and writes the phone's answer under
the ask's key; the assigner lifts a hold whose fingerprint runs healthy on
every node that runs the module, says so once and takes its ask back; the held
record names health, phase, when and the node's words.
2026-10-25 14:06:58 +01:00
jschoubben 2268f94ef2 Merge pull request 'The view lists a node's own want among the assigner's wants (issue 1945)' (#39) from fix/r6-node-composition into main
Reviewed-on: #39
2026-10-25 00:18:11 +00:00
jschoubben 48aaecf556 List a node's own want among the assigner's wants, never as unreadable (issue 1945)
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check checking 48aaecf556c5 on 650a06ceab1d | novox/mesh-delivery#39 in group fix/r6-node-composition
2026-10-25 01:50:00 +02:00
jschoubben 650a06ceab Merge pull request 'R6: the assigner writes wants by rule 7 with the composition digest; the watchman and the view follow wants and haves (issue 1906)' (#38) from feat/r6 into main
Reviewed-on: #38
2026-10-24 23:17:46 +00:00
jschoubben 3e863884fd gofmt the passer's fields after main's merge
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/delivery delivering: 0 machine step(s) passed
mesh/check checking 3e863884fda8 on 650a06ceab1d | novox/mesh-delivery#38 in group feat/r6
2026-10-24 23:33:59 +02:00
jschoubben c21bed20a3 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943) 2026-10-24 23:06:02 +02:00
jschoubben aa31e85665 Merge main into feat/r6: keep R4g's composer at a group's mesh-sdk head beside R6's wants and asks
The assigner reads mesh-controller.builds again (the composer's runningCommits reads the build record of every open
assignment), puts its compose asks on node-build-go and says its conditions on operator-channel.
2026-10-24 23:03:42 +02:00
jschoubben 171bddacab Merge pull request 'R4g finish: the composer as a job on node-build-go's check stream, merge-check.sh gone, reads-outside markers (issues 1295, 1621-1623, 1935)' (#35) from feat/r4g-finish into main
Reviewed-on: #35
2026-10-24 20:57:15 +00:00
jschoubben 65496fc4f7 Merge main into feat/r4g-finish, keeping #37's same-branch mesh-sdk pin rule beside the node-engine check's controller placing (issue 1295)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking 65496fc4f7a6 on 171bddacabe2 | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 21:54:46 +02:00
jschoubben 36a1fd197f The switch exception skips the record hold only for a composition the controller judges unchanged; pin mesh-sdk d7a9b96 (issue 1906, ADR 0540 §9) 2026-10-24 21:18:48 +02:00
jschoubben 9a5b3d4881 Pin the assigner, watchman and view to mesh-sdk feat/r6 7f9cc0e (issue 1943) 2026-10-24 20:43:46 +02:00
jschoubben e263049551 Right after the switch a node running F with a have that names no composition is given its want at F with no record hold, the bus included (issue 1906, review round 2 N1) 2026-10-24 20:35:17 +02:00
jschoubben 0f7bc9f531 Pin the assigner, watchman and view to mesh-sdk feat/r6 e45bdc0 (issue 1943) 2026-10-24 20:21:57 +02:00
jschoubben da31740f00 The view's motion events carry the time they are about and call a held or past-bound module stopped, as the messenger reads them (issue 1906 D10) 2026-10-24 18:31:29 +02:00
jschoubben a3815051cc The view says modules in motion: moving, settled and stuck, once for a node, a module and a composition (issue 1906 D10, for the messenger) 2026-10-24 18:27:01 +02:00
jschoubben 8280506cee Start the world's change counter, which the pass of a held backup reads (issue 1906, review round 1) 2026-10-24 18:22:22 +02:00
jschoubben 3b00f3890d Review round 1: a user-list-only revision of the bus is written at once at the fingerprint it runs, before the record hold; pin mesh-sdk a523a8c (issue 1906) 2026-10-24 18:17:25 +02:00
jschoubben fcac03a4f2 Review round 1: the bus with no want is compared with its have; a backup ask is bound to its composition and number and each answer is a distinct want value; the view reads wants and haves, with dotted module names (issue 1906) 2026-10-24 18:15:13 +02:00
jschoubben 14c20b21ee Pin the assigner, watchman and view to the head of mesh-sdk feat/r6 (issue 1943) 2026-10-24 17:52:44 +02:00
jschoubben 333e0cd07d The checker reads a kept manifest as the indexer and the assigner do, so a value falling back to a predefined key is not logged unreadable (issue 1906, issues 1751 and 1921 class) 2026-10-24 17:42:24 +02:00
jschoubben 03e5951dc5 Merge remote-tracking branch 'origin/main' into feat/r6 2026-10-24 17:41:53 +02:00
mesh-tools 4c1d36c9cd Merge pull request 'A try accepts a mesh-sdk pin that is the head of mesh-sdk's branch of the same name, so a group's members can be tried before its pull requests (issue 1921)' (#37) from fix/1921-try-same-branch-pin into main 2026-10-24 15:37:07 +00:00
jschoubben 37d0ebba7a Let the view's manifest test read the emits key the delivered event adds 2026-10-24 16:47:33 +02:00
jschoubben 86fd0bbe96 The view says its event delivered once for each source revision whose wants and haves are all there (issue 1906 D9, for issue 1946) 2026-10-24 16:47:09 +02:00
jschoubben 37a5ab8f1c A node that runs a module of policy record with no want yet is asked like any other, not treated as one that never ran it (issue 1906 D4) 2026-10-24 16:43:01 +02:00
jschoubben 55f19283f4 Do not wait twice at the cleanup for an assigner that ended on its own, which made the refused-start test take ten seconds a bucket 2026-10-24 16:42:28 +02:00
jschoubben e7a3ff1e7e The record policy is the one the controller holds beside the composition, the operator's own choice included; pin mesh-sdk 64a502d (issue 1906 D4, ADR 0539 §14) 2026-10-24 16:41:58 +02:00
jschoubben 991589fdd2 A try of a branch accepts a mesh-sdk pin at the head of mesh-sdk's branch of the same name, so a milestone's repositories can be tried before their pull requests (issue 1921)
mesh/delivery delivered
mesh/check checking 991589fdd20a on 4c1d36c9cd7f
2026-10-24 16:41:39 +02:00
jschoubben f947070b7b The view shows the controller's judgement of the move beside each want (issue 1906 D7, ADR 0539 §14) 2026-10-24 16:38:24 +02:00
jschoubben 04c2c60f02 The assigner acts on answers: a release writes the held want again, a backup answer goes into the node's want, a record move waits for its yes, a restore answer sets a data module back (issue 1906 D4) 2026-10-24 16:37:37 +02:00
jschoubben 0df20b5d77 The assigner's seventh rule: a want names the digest of its composition and waits for it; a revision at the same fingerprint is a new want; the node's own want (issue 1906 D2, ADR 0539 §6) 2026-10-24 16:33:48 +02:00
jschoubben 806c8bf0e4 Pin the assigner, watchman and view to mesh-sdk's release 6 head (issue 1943 S3) and give their test wants a composition digest 2026-10-24 16:32:09 +02:00
jschoubben 12b01badb0 Prove the assigner replaced mid-pass resumes with nothing lost or written again (issue 1906 D8) 2026-10-24 16:27:34 +02:00
jschoubben 7b18696cd5 Give the data module's test manifest the directory and backup the catalogue asks of an irreplaceable item 2026-10-24 16:26:43 +02:00
jschoubben 204a59dab4 The assigner asks whether to restore a data module whose held fingerprint failed its health check, and raises the condition for a held fingerprint (issue 1906 D3) 2026-10-24 16:26:20 +02:00
jschoubben f048e2b443 The view shows each want beside the have of its node, and a source revision's derived state, in place of the comparison with sends (issue 1906 D7) 2026-10-24 16:23:51 +02:00
jschoubben 9d2e9dc6c9 The watchman mirrors each want of the assigner as a pulse owed by the node's agent until the have names its fingerprint (issue 1906 D6) 2026-10-24 16:22:02 +02:00
jschoubben 024a3787da End the assigner's comparison with the send: the controller sends nothing to compare with, so the differences bucket is kept read-only (issue 1906 D5, ADR 0539 §8) 2026-10-24 16:20:43 +02:00
jschoubben 4fd6b934c8 Read the assigner's wants, differences and the haves with tools, so the gap between wants and sends can be read live (issue 1906 D1) 2026-10-24 16:19:42 +02:00
mesh-tools fa46e7df7d Merge pull request 'The assigner reads a change's manifests as the indexer does, so a value falling back to a predefined key no longer fails every composition touching it' (#36) from fix/assigner-predefined-fallback into main 2026-10-24 14:09:58 +00:00
jschoubben 18b522184f Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds: a group touching pacman or xterm failed its composition on it (ADR 0348 §7, issue 1751)
mesh/delivery delivered
mesh/check pass 18b522184f73 on fa46e7df7dd1 | unit 128 run, 375 kept | integration 138 run, 426 kept | composition passed on ace
2026-10-24 15:03:14 +02:00
jschoubben 1d4dcdf41a Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds (ADR 0348 §7, issue 1751); pin mesh-sdk bfc3883a
mesh/delivery-group group feat/r4g-finish rejected: composed together: the checker's verdict at the members' heads (171a15384fa1): shanks
mesh/delivery superseded: a newer head of the same pull request
mesh/check checking 1d4dcdf41a22 on 4c1d36c9cd7f | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 14:57:36 +02:00
jschoubben f5fd74380c Pin to the group's new mesh-sdk head 19c1d359, whose composer names the reader's pinned SDK instead of a bare (devel) (ADR 0482 clause 5)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk 19c1d35939bc, which is neither on mesh-sdk's t…
2026-10-24 14:12:10 +02:00
jschoubben e78bbfbdee Pin the assigner to the group's mesh-sdk head after main's merge, so it composes with the SDK the group delivers
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's t…
2026-10-24 13:28:42 +02:00
jschoubben ce7dcccb1d Merge main into feat/r4g-finish (#33, #34) 2026-10-24 13:26:24 +02:00
jschoubben 42d880f075 Merge pull request 'Count a node's toolchain report only while its build module is assigned there, so a removed node no longer makes builders disagree (issue 1940)' (#34) from fix/1940-stale-toolchain into main
Reviewed-on: #34
2026-10-24 11:10:45 +00:00
jschoubben 3553cbaf08 Count only the toolchain reports of nodes a build module is assigned to, so a node it left does not make the builders disagree (issue 1940)
mesh/delivery delivered
mesh-build-node was unassigned from ace on 2026-10-23 and its ace.typescript key stayed in
mesh-build-node_toolchains at v26.12.0: nothing deletes it, so the indexer, the builder and the
checker took it for a builder and every Node fingerprint said 'builders disagree'. The three now
read mesh-controller_assignments and pass a toolchain report on only while its build module has an
open assignment on its node; builders that run different versions still disagree.
2026-10-24 12:48:41 +02:00
jschoubben d8b19cb332 Merge pull request 'Build a done key again when the store no longer holds its blob (issue 1937)' (#33) from fix/1937-done-key-blobs into main
Reviewed-on: #33
2026-10-24 10:35:21 +00:00
jschoubben 2259d98c5b Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:39:03 +02:00
jschoubben 4702c6bd77 Pin to the finish group's mesh-sdk head dd0d156b, so every member names one SDK (issue 1622) 2026-10-24 04:28:15 +02:00
jschoubben 23f4304d70 Merge remote-tracking branch 'origin/feat/r4g-finish-composer' into HEAD 2026-10-24 04:26:19 +02:00
jschoubben 28ee22ef27 Give the integration test's EVENTS stream the artifacts-gone subject alone, so it does not overlap the build seats' streams (issue 1937)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792840300970624489 (novox/mesh-delivery at 42d880f0 into main, d…
mesh/delivery-group group fix/1937-done-key-blobs closed: a member was closed before the group merged
mesh/check pass 28ee22ef27e2 on d8b19cb332b6 | unit 126 run, 370 kept | integration 136 run, 420 kept | composition passed on shanks
2026-10-24 04:25:19 +02:00
jschoubben f0c5be8c30 Say which artifacts the store check asks about: bundles and archives, whose blobs the collector deletes (issue 1937) 2026-10-24 04:19:17 +02:00
jschoubben 80a0183567 Build a done key again when the artifact store no longer holds its blob, and on the controller's artifacts-gone (issue 1937)
The store's nightly collector deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which
the controller had never taken in, and the builder kept them done: their next delivery would refuse. The builder now asks
the controller's held before it leaves a done trunk key done (once per key per process, at its start and on each module
value), claims a key whose blob is gone building as a new revision and says so in its log and the key's step, and hears
artifacts-gone, said when a take-in found a gone blob. No timer: a start, a value or an event.
2026-10-24 04:17:01 +02:00
jschoubben c256d732f4 Pin the assigner to mesh-sdk's head that reads a composer request from a file (issue 1622) 2026-10-24 04:14:33 +02:00
jschoubben 93427c08e7 Ask the Go build seat for the composer at a group's mesh-sdk head and at its readers' mesh-sdk, and follow each step (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben c58342786f Judge a group with a mesh-sdk member from the composer at its head, and fail a field its readers cannot read (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben 840e56eddb Fail the node-engine's check when the controller's trunk is unknown, so its step-nine test never skips (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 90fd07ac26 Merge pull request 'Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)' (#32) from fix/1933-rebuild-indexer into main
Reviewed-on: #32
2026-10-23 23:38:21 +00:00
jschoubben 90a79b1585 Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)
mesh/delivery delivered
mesh/check pass 90a79b15854b on 90fd07ac260a | unit 126 run, 366 kept | integration 136 run, 414 kept | composition passed on novox
2026-10-24 01:35:33 +02:00
jschoubben 45ab994777 Merge pull request 'The checker keeps a failed test's whole message with its cause first, and its status names the cause (issue 1901, with mesh-catalog)' (#30) from fix/1901-rebased into main
Reviewed-on: #30
2026-10-23 22:36:03 +00:00
jschoubben 55f486e040 Merge pull request 'The indexer keeps an unreadable manifest's inputs, so an SDK upgrade judges it again at start (issue 1923)' (#29) from fix/1923-stale-none2 into main
Reviewed-on: #29
2026-10-23 22:35:49 +00:00
jschoubben 7a88eb0a56 Merge pull request 'The checker reads mesh-sdk's list-form level digest, so kept verdicts hold for a check naming a package like token (issue 1930, with mesh-sdk and mesh-catalog)' (#31) from fix/1930-digest-shape into main
Reviewed-on: #31
2026-10-23 22:08:43 +00:00
jschoubben 9261347a8f Read a level's digest from mesh-sdk's list form, so kept verdicts hold for a check whose end names a package like token (issue 1930)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792794985391468127 (novox/mesh-delivery at 45ab9947 into main…
mesh/check checking 9261347a8f91 on 7a88eb0a5609
mesh/delivery-group group fix/1930-digest-shape merged: every member is merged
2026-10-23 23:37:49 +02:00
mesh-tools 263ff54a4e Merge pull request 'Integration tests that R4g never ran now pass: the assigner's fixture, the builder's restart test, four checker tests (issues 1927, 1928, 1929)' (#28) from fix/1927-assigner-fpset into main 2026-10-23 21:24:32 +00:00
jschoubben cd3e55e056 Bring the checker's integration tests to what R4g decided: Go-only repository, echoed beside, readable kept (issue 1929)
mesh/delivery delivered
mesh/check pass cd3e55e05605 on fae4d48e09a5 | unit 0 run 483 kept | int 0 run 540 kept | clone 17s | composition passed on 3 machines
2026-10-23 23:18:11 +02:00
jschoubben b303e32a41 Close the builder test's nodes from the bed's start, so no pull in hand takes the first ask (issue 1928) 2026-10-23 23:03:38 +02:00
jschoubben a31bd7d4a5 Give the assigner test's candidate the SDK's fingerprint set, which the SDK refuses without (issue 1927) 2026-10-23 22:57:05 +02:00
jschoubben 32542ebc14 Keep an unreadable manifest's inputs and judge its verdict again at start, so an upgraded SDK clears it (issue 1923)
mesh/delivery delivered
mesh/check pass 32542ebc14c3 on 7a88eb0a5609 | unit 122 run, 366 kept | integration 132 run, 414 kept | composition passed on novox
2026-10-23 22:17:09 +02:00
jschoubben a697ee86d4 Test 1901/AC3: an older pair of build module and checker still agree (issue 1901)
mesh/delivery delivered
mesh/delivery-group group fix/1901-rebased merged: every member is merged
mesh/check checking a697ee86d4af on 45ab99477701
2026-10-23 22:14:29 +02:00
jschoubben 0373a3b24e mesh-delivery-checker: a failed test's verdict keeps its last lines and the status line names its error (issue 1901) 2026-10-23 22:13:41 +02:00
jschoubben fae4d48e09 Merge pull request 'R4g+R5 (mesh-delivery): builder and checker put asks, the assigner and its composition check, the view's wants and backlog (design 47)' (#27) from feat/r4g into main
Reviewed-on: #27
2026-10-23 17:49:01 +00:00
jschoubben b2c545c40a Place mesh-controller beside the node-engine's tests, which read its manifest (issue 1295, R4g)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792777718506798022 (novox/mesh-delivery at fae4d48e into main…
mesh/check checking b2c545c40aec on 85aca46348a0 | novox/mesh-delivery#27 in group feat/r4g
mesh/delivery-group group feat/r4g merged: every member is merged
2026-10-23 18:36:13 +02:00
jschoubben ad26670996 Prove that a second check of the same heads keeps every package the first passed, the digest being taken inside the check ask as the build module answers it (issue 1917)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/apply.TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing: judge_directories_…
2026-10-23 17:18:37 +02:00
jschoubben fad96ae67b Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/config.TestTheUnmovedListHoldsNoModuleTheTrunksDoesNot: realtree_test.go:14: mesh-cata…
2026-10-23 17:06:09 +02:00
jschoubben b83af04f0c Read an ask's stream at take-over by its name, as the broker grants it, and wait for the ask when that read fails rather than fail a key whose build may still finish (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: novox: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group…
2026-10-23 16:59:37 +02:00
jschoubben c7bddbbeab Bound a running build step's pulse by the step's own bound from its start, since nothing rewrites the key while it runs (issue 1908) 2026-10-23 16:58:26 +02:00
jschoubben d3561923aa Say the unit level's gofmt and go vet time in the check's line, and pin the group's mesh-sdk head that carries it (issue 1887 D7-go, review round 2 item 5) 2026-10-23 16:51:55 +02:00
jschoubben d32ec6b7f8 Say what the builder, the indexer and the watchman do since R4g in their comments: no build-agent is asked, no build record moves a fingerprint, no checks-running bucket is mirrored (issue 1887, review round 2 item 6) 2026-10-23 16:47:36 +02:00
jschoubben 52d8b9b753 Say in the checker's status that a check without a verdict waits in a build seat's stream, not behind a plan key or a capacity, as A5b left it (issue 1887, review round 2 item 4) 2026-10-23 16:47:20 +02:00
jschoubben ce51457974 Prove on the real bus that a builder started again while its asks waited puts no second ask (issue 1887 AC3, review round 2 item 2) 2026-10-23 16:46:58 +02:00
jschoubben bac82bb91e Count the asks the builder puts and read their kind in its integration test, so 1887/AC1 is proven on the real bus (issue 1887, review round 2 item 3) 2026-10-23 16:46:57 +02:00
jschoubben 357e28f31d Merge remote-tracking branch 'origin/feat/r4g' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-builder/internal/build/run_test.go
2026-10-23 16:46:05 +02:00
jschoubben 5ddc221833 Bound a build or check ask's step from its start, and drop the checker's whole-check bound, so asks waiting in the streams during a large rebuild are never failed and asked twice (issue 1887, review round 2 item 1; ADR 0525 §3, §6) 2026-10-23 16:45:35 +02:00
jschoubben 37860d46a0 Stop watching the node-engine's have in the checker: a check's asks wait in the build seats' streams, which redeliver a lost node's ask, so no check fails for want of a node; presence stays for an unheard forge and the assigner's presence (issue 1912) 2026-10-23 16:41:34 +02:00
jschoubben 84b2c92ccf Put again, under its own id, an ask a build key names that neither its stream holds nor any node ran, so a builder that stopped between naming and putting it leaves no key waiting for an ask nobody put (issue 1910) 2026-10-23 16:39:56 +02:00
jschoubben c0f24f56b5 Drop the view's use of the build seats: it reads their workers and publishes no ask (issue 1887 A8a) 2026-10-23 16:35:05 +02:00
jschoubben 76bc810bda Test that an ask running when the watchman starts gets its pulse from the steps that stand (issue 1909) 2026-10-23 16:34:50 +02:00
jschoubben e927469460 Pulse a build ask only from when a node takes it, and end the pulse with its step, so a waiting ask is a backlog and not a stuck row (issue 1908) 2026-10-23 16:34:39 +02:00
jschoubben 97b42e3f0d Name in every composition, by machine, those that could run a moved or added module and are not assigned it and those whose grants alone the change alters, as not judged, as ADR 0537 clause 1 asks (issue 1297) 2026-10-23 16:29:57 +02:00
jschoubben 1b019f0353 Record the assigner's full requirements and mesh-sdk feat/r4g's checksums, so the new module builds without a workspace (R4g) 2026-10-23 16:27:05 +02:00
jschoubben 95f52afab2 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g) 2026-10-23 16:26:11 +02:00
jschoubben e4a40a1cb0 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, so every module r4g touches builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:02 +02:00
jschoubben dd8e05142d Merge branch 'feat/r4g-scripts' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 66be3dd0e4 Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 527f87f94c Merge branch 'feat/r4g-asks' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-checker/cmd/mesh-delivery-checker/judge.go
#	modules/mesh-delivery-checker/manifest_test.go
#	modules/mesh-delivery-checker/module.json
2026-10-23 16:20:17 +02:00
jschoubben 0463f21026 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 16:19:45 +02:00
jschoubben a4629bc649 Compute every module's fingerprint with no builder input and read no build-agent record for it, so a change of the builder rebuilds nothing (issue 1887 I1, ADR 0530 §1) 2026-10-23 16:14:19 +02:00
jschoubben 50836eb4d6 Format the watchman's integration test (issue 1887 A8b) 2026-10-23 16:14:03 +02:00
jschoubben 79ef768ea4 Show the backlog of build asks per language and kind: waiting, held by a node, and the oldest waiting ask's age (issue 1887 A8a) 2026-10-23 16:14:00 +02:00
jschoubben a8ba5c2fd5 Mirror a waiting build ask as a pulse until a node writes its step, and drop the read of the controller's checks-running (issue 1887 A8b) 2026-10-23 16:12:43 +02:00
jschoubben d0354574c6 Look the assigner's compositions bucket up at each check until it is there, so an assigner delivered after the checker started is read without a restart (issue 1297, review round 1 item 3) 2026-10-23 16:12:18 +02:00
jschoubben 69577f739d Judge hq as any repository, so its pull requests get a verdict and mesh/check from its Go tests and no script judges it (issue 1887 A10, ADR 0527 §2) 2026-10-23 16:11:43 +02:00
jschoubben d4977d9c9d Start the checker whole with the assigner's verdict on the bus and read its line, the level 1297/AC3 asks (issue 1297, review round 1 item 2) 2026-10-23 16:11:31 +02:00
jschoubben 179664898a Merge feat/r4g-asks into the view lane, keeping the checker's assigner compositions 2026-10-23 16:11:15 +02:00
jschoubben 77af82cfa2 Keep the asks state and write the ask about a held fingerprint, so the view's and the teller's read of mesh-delivery-assigner.asks is granted and has something to answer (issue 1297, review round 1 item 1; ADR 0528 clause 6) 2026-10-23 16:11:07 +02:00
jschoubben f37e5d5fa1 Delete merge-check.sh; the checker's levels and the build module's fetch beside run what it ran (issue 1295, S8; ADR 0529) 2026-10-23 16:10:46 +02:00
jschoubben d9d1e57e3a Delete the checker's hold, its own capacity and places, its node moves and its read of the controller's plans-building, so a check waits only in the build seats' streams (issue 1887 A5b, closes 1685; ADR 0525 §4, §7) 2026-10-23 16:09:34 +02:00
jschoubben e4b639c6de Judge each pull request's check as one ask on the build seats' check stream, a re-judge on recheck, so the checker chooses no node and a lost node's check is redelivered, not moved (issue 1887 A5a, ADR 0525 §1, §3, ADR 0536 §4) 2026-10-23 16:05:55 +02:00
jschoubben 271d1ca759 Pin mesh-sdk at feat/r4g's head c3915ffb, the merge of lane B's layouts, so every r4g module builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:01:15 +02:00
jschoubben 2227e7a81f Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 173178df86 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 9f27e2b1f8 Build each build module's part of a fingerprint as one ask on its seat's stream, named in the key before it is put, so the builder chooses no node, holds no capacity and never asks twice (issue 1887 A4a, A4b, A4c, ADR 0525 §1-§6) 2026-10-23 15:57:03 +02:00
jschoubben 85aca46348 Merge pull request 'Checks under 5 minutes (checker): a new head's check before trunk-move re-judges, and a build module's change judged by its own code (issues 1897, 1898; ADR 0535, 0536)' (#25) from feat/fastcheck into main
Reviewed-on: #25
2026-10-23 13:50:08 +00:00
jschoubben d5868c178f Take the words of a module the change removes from mesh-sdk in the indexer and the assigner, so the assigner's reading of a removal cannot drift from what the indexer writes (issue 1297, B3b) 2026-10-23 15:44:00 +02:00
jschoubben 7cbb1c4207 Wait for the assigner's composition only while the assigner is online, say at once that it is not judged when it is not, and take it in when the assigner writes it later, so no check waits its bound for a composition nobody makes (issue 1297, B3d) 2026-10-23 15:44:00 +02:00
jschoubben 4c98841d2a Judge a repository the mesh knows no module of with both build modules, so mesh-lab's nested Go modules and its npm test both run (issue 1887 A12) 2026-10-23 15:41:57 +02:00
jschoubben c9ee92a4ab Pin the assigner and the checker to the SDK commit that holds the assigner's ask value, so both build on the layouts the view reads (issue 1297, subtask B1c) 2026-10-23 15:39:49 +02:00
jschoubben 0f7091e4ac Place the change's own repository beside it at its trunk, so a shrink-only list test compares with the trunk's copy at $MESH_CHECK_BESIDE/<name> (issue 1887 A12, ADR 0529 §2) 2026-10-23 15:35:37 +02:00
jschoubben ed98a7519e Write every difference between a want and what the controller sends, and delete one that ended, so the view and acceptance test 17 read them on the bus (issue 1297, subtask B5b; ADR 0528 clause 4) 2026-10-23 15:31:11 +02:00
jschoubben d7ba5aa91e Compare each node's want with the fingerprint of the build the controller sends it, explaining a difference by the rule that holds the want back or saying it unexplained (issue 1297, subtask B5a; ADR 0528 clause 4) 2026-10-23 15:30:31 +02:00
jschoubben 016385751d Write each node's want by the six rules, the canary the first online node by name, and hold a fingerprint whose health failed with its nodes set back, so R5's wants can be compared with what the controller sends (issue 1297, subtasks B4a-B4b; ADR 0429 §6, ADR 0528 clauses 1 and 3, ADR 0435 §3) 2026-10-23 15:29:57 +02:00
jschoubben 5091cab652 Read the assigner's composition into the checker's verdict in place of 'not yet run', failing a check whose composition failed, naming its machine (issue 1297, subtask B3d; ADR 0528 clause 5) 2026-10-23 15:27:59 +02:00
jschoubben d8599855e1 Compose the machines a change touches, from the trunk's and the change's manifests and the controller's assignments on the bus, and write the verdict under the checker's key; what needs the facts snapshot is said as not judged (issue 1297, subtasks B3a-B3c; ADR 0446 clauses 1-3, ADR 0528 clause 5, ADR 0482 clause 3 as R4g delivers it) 2026-10-23 15:25:22 +02:00
jschoubben 57258b6fa4 Take the assigner's ask in the SDK's layout and name the asking in the view's answer, so a stale answer never fills a later asking (issue 1297 B9 and B10) 2026-10-23 15:22:32 +02:00
jschoubben c2b7b8867e Start the assigner whole on an embedded bus in an integration test: it says its three states and refuses to start without a bucket it reads (issue 1297, subtask B2b) 2026-10-23 15:19:49 +02:00
jschoubben a48279b31a Fill an assigner ask's answer from the view's answer under its view key, once and never for an ask that needs proof (issue 1297 B10) 2026-10-23 15:19:37 +02:00
jschoubben 034a6de9a7 Make the module mesh-delivery-assigner: its manifest's reads and states, its own credential and the states it announces, so the controller grants it what R5 reads (issue 1297, subtask B2a; ADR 0528, ADR 0429 §1) 2026-10-23 15:19:06 +02:00
jschoubben e7ed48b81a Give the command an answer verb, so the operator answers a hold or restore ask from a terminal as in the page (issue 1297 B9b) 2026-10-23 15:18:04 +02:00
jschoubben f7dd4ab586 Let the view answer the assigner's asks that need no proof, under the ask's view key, and refuse the merge's Approve on the phone's behalf (issue 1297 B9a) 2026-10-23 15:17:40 +02:00
jschoubben 57b0030258 Show every difference between the assigner's wants and the controller's sends with its rule, so the operator sees what holds a node back (issue 1297 B8) 2026-10-23 15:15:32 +02:00
mesh-tools db5d88f893 Merge pull request 'Drop the indexer's modules keys the catalogue's trunk no longer holds, at each merge and at start (issue 1900)' (#26) from fix/1900-indexer-drops-removed-modules into main 2026-10-23 12:56:31 +00:00
jschoubben 24753bfc2c Grant gitea_contains in the indexer's invokes and test every called forge tool is granted (issue 1900)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792794967663205383 (novox/mesh-delivery at 55f486e0 into main, d…
mesh/check pass 24753bfc2c02 on 75a4821eff9e | unit 431 run, 0 kept | integration 476 run, 0 kept | over bound: integration (slowest integration mesh-…
2026-10-23 14:48:41 +02:00
jschoubben 1dc9d80cfb Keep a key written ahead of the source head and a key with no path when reconciling (issue 1900) 2026-10-23 14:44:14 +02:00
jschoubben 26bfdbb637 Drop the indexer's modules keys the trunk no longer holds, at each merge and at start (issue 1900) 2026-10-23 14:31:55 +02:00
jschoubben 4ffe5f4d52 Start a never-judged head's check before the re-judges a trunk move asks, oldest first (issue 1897 S1)
mesh/delivery delivered
mesh/delivery-group group feat/fastcheck merged: every member is merged
mesh/check checking 4ffe5f4d5243 on 85aca46348a0 | novox/mesh-delivery#25 in group feat/fastcheck
A new pull request waited 25 to 45 minutes behind every open pull request's re-judge after a
merge. The semaphore becomes a ranked admission (ADR 0536 §4); checker_status names the
waiting checks by kind.
2026-10-23 14:09:42 +02:00
jschoubben 2bdb8d0e00 Ask the Go build module to run the candidate when a merge changes it, and say in the line which program judged (issue 1898 K3)
A change to mesh-build-go was judged by the installed module it replaces, so such pull
requests were merged by hand. ADR 0535 §1, §2, §4, §5.
2026-10-23 14:06:57 +02:00
jschoubben 75a4821eff Merge pull request 'Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)' (#24) from fix/1890-checker-stale-steps into main
Reviewed-on: #24
2026-10-23 11:01:10 +00:00
jschoubben e762028760 Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)
mesh/delivery delivered
mesh/check pass e76202876032 on b714eab995a5 | unit 422 run, 0 kept | integration 467 run, 0 kept | over bound: integration (slowest integration mesh-…
The checker's own places are released when a step ends or its wait passes the step bound; checker_status read every
'running' key in the build modules' steps buckets as a step in flight, so keys left by stopped, restarted or unassigned
build modules looked like full nodes. It now lists them under left_running with why (past the bound, module not on the
node, module not running), and leaves hq-only groups out of the unjudged candidates.
2026-10-23 09:03:45 +02:00
jschoubben 36bf3b7143 Merge pull request 'The builder says it writes references only once its build modules proved it (issue 1886, ADR 0522)' (#23) from fix/1886-builder-says-references into main
Reviewed-on: #23
2026-10-23 00:26:18 +00:00
jschoubben 9fad90ee89 Say that the builder writes references only once each build module answered one (issue 1886, ADR 0522 review)
mesh/merge-gate head 9fad90ee890a merged into trunk b714eab995a5 — pass: builds mesh-delivery-builder → novox; no bus step; the change touches no module of…
mesh/repo-check head 9fad90ee890a merged into trunk b714eab995a5 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/1886-builder-says-references merged: every member is merged
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The references come from the build modules' own pushes, so a new builder beside
an old build module writes none; saying it at start would send the controller
to keys that fail not stored.
2026-10-23 00:52:50 +02:00
jschoubben 4b8b1c047c Say in the builder's own bucket of instances that it writes references (issue 1886)
The controller delivers from the builder's keys only while a running builder
says its done keys name references; a builder from before R4e says nothing,
so R4e's own delivery is asked of build-agent instead of waiting for ever.
The say is put once started and taken away when stopped.
2026-10-23 00:48:28 +02:00
mesh-tools b714eab995 Merge pull request 'R4e (mesh-delivery): the builder pushes and keys artifacts by reference, the checker's verdict and try, the view, and the checker's owed state read by the forge' (#22) from feat/r4e into main 2026-10-22 22:13:05 +00:00
jschoubben efca168c55 Stop the forge bridge performing mesh/check: the forge module writes it (R2-1)
mesh/repo-check head efca168c5522 merged into trunk 0e372c87f1df beside mesh-catalog#457@896bd883 +1 more — pass: its merge-check.sh passed
mesh/merge-gate head efca168c5522 merged into trunk 0e372c87f1df — pass: group feat/r4e composed: every machine composes with the change as it did without …
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792707174627079192 (novox/mesh-delivery at b714eab9 into main, d…
mesh/delivery-group group feat/r4e closed: a member was closed before the group merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:989 | efca168c5522 on b714eab995a5 | unit 2092 run, 0 kept, 3 skipped | integration…
R4e round 2 finding R2-1, ADR 0521: nothing in the delivery plan orders the
forge module before the bridge, and the forge module's durable consumer keeps
its deliver policy when its filters change, so a check.status emitted before
it consumed the event was lost. The forge module now watches the checker's
owed state and writes mesh/check itself, once per revision.

The bridge emits no check.status, declares no emits, and neither reads the
checker's owed nor hears its stream; OwedAct still refuses mesh/check. The
integration test shows the bridge calls no tool and publishes nothing of its
own when the checker owes mesh/check.
2026-10-22 21:40:09 +02:00
jschoubben 19d06163de Announce the checker's states, owed readable, so the forge module may read it (R2-1)
R4e round 2 finding R2-1, ADR 0521: the forge module writes mesh/check from
the checker's owed state, read by a watch, because no event order between the
forge bridge and the forge module is kept by the delivery plan. A reader is
granted from the owner's announcement (ADR 0495 §2-§3), and the checker
announced nothing. It now says every state it keeps, with owed, previews and
verdicts readable: the forge module and the bridge read owed, the indexer
previews, the gate verdicts.
2026-10-22 21:38:53 +02:00
jschoubben 0997b071a3 Pin every R4e member module to mesh-sdk's final R4e head 104533e8, which holds review fixes S1 and S2, so the group pins one SDK 2026-10-22 21:29:20 +02:00
226 changed files with 27982 additions and 3545 deletions
-74
View File
@@ -1,74 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# This repository's own check, mesh/repo-check (novox/hq ADR 0238 §4), written with its first commit (ADR 0428 §4):
# every Go module in the tree is vetted, its unit tests run, and then its integration tests (-tags integration).
# A tree with no Go module has nothing to test, and the check says so and passes.
#
# No module commits its dependencies (no vendor directory: .gitignore). A module's dependencies are fetched, or a
# requirement of the mesh's own forge (the pinned mesh-sdk commit) is taken from its repository's clone beside the check
# (MESH_CHECK_BESIDE), through a go.mod of its own (-modfile), by mesh-catalog's helper checks/deps, built from the
# clone of mesh-catalog beside this check and not copied here: the build seat forgets the forge credential before the
# check runs, so a commit the seat's module cache does not hold cannot be fetched from the forge (novox/hq issue 385). A
# module whose dependencies cannot be had **fails the check**, by name and why: its tests could not run, and a test that
# could not run is a failure, never a note. Run by hand, without MESH_CHECK_BESIDE, the dependencies are fetched.
set -eu
cd "$(dirname "$0")"
# The toolchain the check runs in sets -mod=vendor; there is no vendor directory, so the modules are used as they are.
GOFLAGS="$(printf '%s' "${GOFLAGS:-}" | sed 's/-mod=[a-z]*//g') -mod=mod"
export GOFLAGS
export GOPRIVATE=git.novox.be
modules=$(find . -name go.mod -not -path '*/vendor/*' -not -path './.git/*' | sort)
if [ -z "$modules" ]; then
echo "This tree holds no Go module, so there are no tests to run; the check passes."
exit 0
fi
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
deps=""
helper="${MESH_CHECK_BESIDE:-}/mesh-catalog/checks/deps"
if [ -n "${MESH_CHECK_BESIDE:-}" ] && [ -f "$helper/main.go" ]; then
if ! (cd "$helper" && go build -o "$work/deps" .); then
echo "FAILED: the dependency helper of mesh-catalog beside this check could not be built"
exit 1
fi
deps="$work/deps"
elif [ -n "${MESH_CHECK_BESIDE:-}" ]; then
echo "NOT TAKEN FROM BESIDE: no mesh-catalog/checks/deps beside this check ($MESH_CHECK_BESIDE); the dependencies are fetched"
fi
untestable=""
for mod in $modules; do
dir=$(dirname "$mod")
modfile=""
if [ -n "$deps" ]; then
if ! modfile=$("$deps" -beside "$MESH_CHECK_BESIDE" -work "$work" "$dir"); then
echo "FAILED: $dir could not be tested: $modfile"
untestable="$untestable $dir"
continue
fi
[ -n "$modfile" ] && echo "$dir takes its forge dependencies from the clones beside the check"
elif ! fetched=$(cd "$dir" && go mod download 2>&1); then
echo "FAILED: $dir could not be tested: its dependencies cannot be fetched: $(printf '%s\n' "$fetched" | tail -n 1)"
untestable="$untestable $dir"
continue
fi
mf=""
[ -n "$modfile" ] && mf="-modfile=$modfile"
unformatted=$(cd "$dir" && gofmt -l .)
if [ -n "$unformatted" ]; then
echo "$dir is not gofmt'd: $unformatted"
exit 1
fi
echo "== $dir: go vet and the unit tests"
(cd "$dir" && go vet $mf ./... && go test $mf ./...)
echo "== $dir: the integration tests (-tags integration)"
(cd "$dir" && go vet $mf -tags integration ./... && go test $mf -tags integration ./...)
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
@@ -0,0 +1,161 @@
package main
import (
"context"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"mesh-delivery-assigner/internal/assign"
)
// AssignerModule is the asker the assigner's compose asks name.
const AssignerModule = "mesh-delivery-assigner"
// composeAsks are the compose asks the assigner put on the Go build seat's check stream (novox/hq ADR 0482 clause 4,
// issues 1622 and 1623), by what each composes: one ask per key, put once and followed by a watch of its step key until
// it ends, which wakes a pass. No timer: an ask that waits in the stream waits (its backlog is the view's and the
// watchman's to show). A restart forgets them and asks again; the build module keeps a composer built for a commit, so a
// second ask builds nothing.
type composeAsks struct {
ctx context.Context
asker buildrun.Asker
wake func()
logf func(string, ...any)
now func() time.Time
mu sync.Mutex
asks map[string]*composeAsk
}
type composeAsk struct {
id string
cancel context.CancelFunc
ran *assign.Ran
}
func newComposeAsks(ctx context.Context, asker buildrun.Asker, wake func(), logf func(string, ...any)) *composeAsks {
return &composeAsks{ctx: ctx, asker: asker, wake: wake, logf: logf, now: time.Now, asks: map[string]*composeAsk{}}
}
// get is the end of the ask that composes key, or nil and its id while it runs. The first get of a key puts the ask.
// A job its layout refuses ends at once, failed with why: never silent.
func (a *composeAsks) get(key string, job delivery.ComposeJob) (*assign.Ran, string) {
a.mu.Lock()
defer a.mu.Unlock()
if k, ok := a.asks[key]; ok {
return k.ran, k.id
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: AssignerModule,
At: a.now(), Compose: &job}
ctx, cancel := context.WithCancel(a.ctx)
k := &composeAsk{id: ask.ID, cancel: cancel}
a.asks[key] = k
if err := a.asker.Put(ctx, ask); err != nil {
a.logf("the compose ask %s for %s could not be put: %v", ask.ID, key, err)
k.ran = &assign.Ran{State: delivery.StepFailed, Reason: "the compose ask could not be put on the Go build seat: " + err.Error()}
cancel()
return k.ran, k.id
}
go func() {
step, err := a.asker.Wait(ctx, ask)
if ctx.Err() != nil {
return
}
ran := assign.RanOf(step)
if err != nil {
ran = assign.Ran{State: delivery.StepFailed, Reason: "the compose ask's step could not be followed: " + err.Error()}
}
a.mu.Lock()
k.ran = &ran
a.mu.Unlock()
a.wake()
}()
return nil, k.id
}
// sweep forgets, and stops following, every ask whose key no pass asked for this time: a candidate gone or moved on.
func (a *composeAsks) sweep(seen map[string]bool) {
a.mu.Lock()
defer a.mu.Unlock()
for key, k := range a.asks {
if !seen[key] {
k.cancel()
delete(a.asks, key)
}
}
}
// composeAtHead is the composition of a group with a mesh-sdk member: the composer at its head and at each reader's
// mesh-sdk, asked of the Go build seat, and the verdict once every ask ended; until then the verdict waits, naming the
// asks. The keys it used are marked in seen.
func (p *passer) composeAtHead(ctx context.Context, m assign.Mesh, ch assign.Change, seen map[string]bool, began time.Time) delivery.CompositionVerdict {
waiting := func(words string) delivery.CompositionVerdict {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionWaits, Words: words}
}
head, err := assign.HeadJob(m, ch)
if err != nil {
return delivery.CompositionVerdict{Group: ch.Group, Heads: ch.Heads, State: delivery.CompositionFail, SDK: "mesh-sdk " + ch.SDKHead,
Words: "the compose ask at the group's mesh-sdk head cannot be made: " + err.Error()}
}
var pending []string
key := "head." + ch.Group + "." + ch.Heads + "." + ch.SDKHead
seen[key] = true
headRan, id := p.asks.get(key, head)
if headRan == nil {
pending = append(pending, "mesh-sdk "+ch.SDKHead+" (ask "+id+")")
}
var readers []assign.Reader
var notJudged []string
for _, module := range sortedNames(assign.ReaderModules) {
repository := assign.ReaderModules[module]
var commits []string
member := false
if h := ch.Members[repository]; h != "" {
commits, member = []string{h}, true
} else {
commits = p.w.runningCommits(ctx, kvGetter(p.kvs[delivery.BuildsBucket]), module)
}
if len(commits) == 0 {
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: no build of it is assigned with a "+
"commit on record, and the group holds no member of "+repository)
continue
}
for _, commit := range commits {
job, err := assign.ReaderJob(ch, module, repository, commit)
if err != nil {
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member,
Ran: assign.Ran{State: delivery.StepFailed, Reason: err.Error()}})
continue
}
rkey := "reader." + ch.Group + "." + ch.Heads + "." + module + "." + commit
seen[rkey] = true
ran, rid := p.asks.get(rkey, job)
if ran == nil {
pending = append(pending, module+" at "+commit+" (ask "+rid+")")
continue
}
readers = append(readers, assign.Reader{Module: module, Commit: commit, Member: member, Ran: *ran})
}
}
if len(pending) > 0 {
return waiting("composing on the Go build seat with the mesh-sdk of " + strings.Join(pending, ", "))
}
v := assign.FromComposer(m, ch, *headRan, readers, time.Since(began))
v.Warnings = append(v.Warnings, notJudged...)
return v
}
func sortedNames(m map[string]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,180 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// holder is a test holder of the Go build seat: it takes every ask from the seat's check worker, as mesh-build-go does,
// and answers a compose ask by running mesh-sdk's composer over the request filled from the indexer's manifests (the
// linked composer standing for the one the build module builds at the asked commit), writing its step on its node's key.
type holder struct {
mu sync.Mutex
asks []delivery.StepAsk
}
func (h *holder) taken() []delivery.StepAsk {
h.mu.Lock()
defer h.mu.Unlock()
return append([]delivery.StepAsk(nil), h.asks...)
}
func (b *bed) hold(t *testing.T) *holder {
t.Helper()
seat, _ := delivery.AskStream(delivery.AskGo)
worker, _ := delivery.AskWorker(delivery.AskGo, delivery.AskCheck)
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskCheck)
cons, err := b.js.CreateOrUpdateConsumer(b.ctx, seat, jetstream.ConsumerConfig{Durable: worker, FilterSubject: subject, AckPolicy: jetstream.AckExplicitPolicy})
if err != nil {
t.Fatal(err)
}
steps := b.buckets[delivery.StepsBucket(goBuildModule)]
h := &holder{}
cc, err := cons.Consume(func(m jetstream.Msg) {
a, err := delivery.DecodeStepAsk(m.Data())
if err != nil || a.Compose == nil {
t.Errorf("the seat was put an ask that is not a compose: %v %s", err, m.Data())
m.Ack()
return
}
h.mu.Lock()
h.asks = append(h.asks, a)
h.mu.Unlock()
req, err := composer.Decode(a.Compose.Request)
if err == nil {
err = composer.Fill(&req, func(digest string) ([]byte, error) {
key, _ := delivery.ManifestsKey(digest)
e, err := b.buckets[delivery.ManifestsBucket].Get(b.ctx, key)
if err != nil {
return nil, err
}
man, err := delivery.DecodeManifest(e.Value())
return []byte(man.JSON), err
})
}
step := delivery.Step{ID: a.ID, Tool: "compose", Node: "ace", State: delivery.StepPassed, Started: time.Now().UTC(), Ended: time.Now().UTC()}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else if v, err := composer.Run(req); err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
} else {
vr, _ := json.Marshal(v)
step.Result, _ = json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0123456789abcdef", Verdict: vr})
}
raw, err := delivery.EncodeStep(step)
if err != nil {
t.Errorf("the step cannot be written: %v", err)
}
key, _ := delivery.AskStepKey("ace", a.ID)
if _, err := steps.Put(b.ctx, key, raw); err != nil {
t.Errorf("the step %s: %v", key, err)
}
m.Ack()
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(cc.Stop)
return h
}
// 1622/AC1-AC3 and 1623 on the bus: started whole, the assigner composes a group holding a mesh-sdk pull request by
// asking the Go build seat for the composer at the member's head, and for the composer at the mesh-sdk the running
// controller pins; it waits naming the asks while they run, writes the verdict from their answers naming the mesh-sdk the
// head's composer was built with, and asks nothing more on a later pass of the same heads.
func TestAGroupWithAnSDKMemberIsComposedByTheComposerAtItsHead(t *testing.T) {
b := startBed(t, "")
trunk := b.manifest(t, "web", `{"module":"web","version":"1"}`)
changed := b.manifest(t, "web", `{"module":"web","version":"2"}`)
mod, err := delivery.EncodeModule(delivery.Module{FP2: delivery.SourcePrefix2 + strings.Repeat("a", 64), Manifest: trunk, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ModulesBucket, "web", mod)
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
running := strings.Repeat("3", 40)
b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`))
b.putJSON(t, delivery.BuildsBucket, delivery.BuildKey("mesh-controller", "c-1"), []byte(`{"id":"c-1","repository":"novox/mesh-controller","commit":"`+running+`"}`))
sdkHead := strings.Repeat("4", 40)
heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
pulls := []delivery.CandidatePull{{Head: heads[0]}, {Head: heads[1],
Modules: []string{"web"}, Fps: map[string]delivery.CandidateModule{"web": {FP2: delivery.SourcePrefix2 + strings.Repeat("b", 64), Manifest: changed}}}}
cand, err := delivery.EncodeCandidate(delivery.Candidate{Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
b.putJSON(t, delivery.CandidatesBucket, ckey, cand)
b.start(t)
key, _ := delivery.CompositionKey("feat/x", hash)
// No holder takes the asks yet: the composition waits, naming them.
waits := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State == delivery.CompositionWaits })
if !strings.Contains(waits.Words, "mesh-sdk "+sdkHead) || !strings.Contains(waits.Words, "mesh-controller at "+running) {
t.Fatalf("the waiting composition says %q, want the head's and the running controller's asks", waits.Words)
}
h := b.hold(t)
v := b.verdict(t, key, func(v delivery.CompositionVerdict) bool { return v.State != delivery.CompositionWaits })
if v.State != delivery.CompositionPass || len(v.Machines) != 1 || v.Machines[0] != "g14" || v.SDK == "" {
t.Fatalf("composition = %+v, want a pass on g14", v)
}
asks := h.taken()
if len(asks) != 2 {
t.Fatalf("the seat was asked %d composes, want two (the head, the running controller)", len(asks))
}
byRepo := map[string]delivery.StepAsk{}
for _, a := range asks {
byRepo[a.Compose.Fetch["repository"].(string)] = a
if a.Asker != AssignerModule || a.Kind != delivery.AskCheck || a.Language != delivery.AskGo {
t.Errorf("ask = %+v", a)
}
}
if a := byRepo["novox/mesh-sdk"]; a.Compose == nil || a.Compose.Fetch["commit"] != sdkHead || a.Compose.Dir != "go" {
t.Errorf("the head's ask = %+v", a.Compose)
}
if a := byRepo["novox/mesh-controller"]; a.Compose == nil || a.Compose.Fetch["commit"] != running || a.Compose.Dir != delivery.RootPath {
t.Errorf("the running controller's ask = %+v", a.Compose)
}
// A later pass of the same heads (woken by any change) asks nothing more.
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
time.Sleep(500 * time.Millisecond)
if n := len(h.taken()); n != 2 {
t.Errorf("a later pass of the same heads asked again: %d asks", n)
}
}
// verdict waits for the composition under key to satisfy ok, and answers it.
func (b *bed) verdict(t *testing.T, key string, ok func(delivery.CompositionVerdict) bool) delivery.CompositionVerdict {
t.Helper()
var last delivery.CompositionVerdict
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.buckets[delivery.AssignerCompositionsBucket].Get(b.ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
continue
}
if err != nil {
t.Fatal(err)
}
if last, err = delivery.DecodeCompositionVerdict(e.Value()); err != nil {
t.Fatal(err)
}
if ok(last) {
return last
}
}
t.Fatalf("no composition under %s as wanted after 10 s; last %+v", key, last)
return last
}
@@ -0,0 +1,945 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// bed is an embedded bus with JetStream, every bucket the assigner reads and keeps, and the stream the mesh's events
// are kept in, as the controller's broker raises them.
type bed struct {
srv *server.Server
js jetstream.JetStream
ctx context.Context
dir string
buckets map[string]jetstream.KeyValue
}
func startBed(t *testing.T, leaveOut string) *bed {
t.Helper()
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "assigner")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
srv, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: filepath.Join(dir, "js"), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
nc, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
t.Cleanup(cancel)
b := &bed{srv: srv, js: js, ctx: ctx, dir: dir, buckets: map[string]jetstream.KeyValue{}}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == leaveOut {
continue
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: 5})
if err != nil {
t.Fatal(err)
}
b.buckets[name] = kv
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
// The operator-channel seat's stream of conditions, which the assigner says its conditions on.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "CONDITIONS", Subjects: []string{"mesh.seat.operator-channel.accept.condition.>"}}); err != nil {
t.Fatal(err)
}
// The Go build seat's work queue, as the broker derives it from the seat's accepted kinds (delivery.AskStream).
seat, _ := delivery.AskStream(delivery.AskGo)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: seat, Subjects: []string{"mesh.seat.node-build-go.accept.>"},
Retention: jetstream.WorkQueuePolicy}); err != nil {
t.Fatal(err)
}
return b
}
// start runs the program as the node does, through run with its environment and a broker file; it answers the
// channel run's error arrives on.
func (b *bed) start(t *testing.T) <-chan error {
t.Helper()
done, _ := b.startStoppable(t)
return done
}
// startStoppable is start, and answers the function that stops this one program (a replaced assigner is stopped, then
// another started).
func (b *bed) startStoppable(t *testing.T) (<-chan error, func()) {
t.Helper()
broker := filepath.Join(b.dir, "broker")
cred, _ := json.Marshal(process.Account{URL: b.srv.ClientURL(), Node: "n1", Module: "mesh-delivery-assigner", User: "n1.mesh-delivery-assigner", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
env := map[string]string{"MESH_BROKER_FILE": broker}
ctx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() {
done <- run(ctx, func(k string) string { return env[k] }, io.Discard)
close(done) // a program that ended on its own (a refused start) is not waited for again at the cleanup
}()
stopped := func() {
stop()
select {
case <-done:
case <-time.After(10 * time.Second):
}
}
t.Cleanup(stopped)
return done, stopped
}
// 1297/B2b: started whole on its credential, the assigner says on the bus every state its manifest declares, each readable,
// having opened every bucket it names. The states are read from module.json, so a state added there is expected here too.
func TestTheAssignerStartsAndSaysWhatItKeeps(t *testing.T) {
b := startBed(t, "")
done := b.start(t)
events, err := b.js.Stream(b.ctx, "EVENTS")
if err != nil {
t.Fatal(err)
}
var said announce.States
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
select {
case err := <-done:
t.Fatalf("the assigner stopped: %v", err)
default:
}
m, err := events.GetLastMsgForSubject(b.ctx, announce.Subject("mesh-delivery-assigner"))
if err != nil {
continue
}
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
break
}
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
if said.Module != "mesh-delivery-assigner" || said.Node != "n1" || len(said.States) != len(m.State) {
t.Fatalf("the assigner said %+v, want the %d states its manifest declares", said, len(m.State))
}
for _, s := range m.State {
if !said.Readable(s.Name) {
t.Errorf("the assigner said %+v: its state %s is not readable", said, s.Name)
}
}
}
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets {
b := startBed(t, missing)
select {
case err := <-b.start(t):
if err == nil || !strings.Contains(err.Error(), missing) {
t.Errorf("without %s the assigner stopped with %v, want the bucket named", missing, err)
}
case <-time.After(15 * time.Second):
t.Errorf("without %s the assigner started", missing)
}
}
// ADR 0546: without the puts' bucket it starts all the same, and once the bucket is made (by the controller at its start,
// ADR 0201) a pass opens it, so a put is taken further without a restart.
was := PutsReopenEvery
PutsReopenEvery = 0
defer func() { PutsReopenEvery = was }()
b := startBed(t, delivery.AssignerPutsBucket)
done := b.start(t)
select {
case err := <-done:
t.Fatalf("without %s the assigner stopped: %v", delivery.AssignerPutsBucket, err)
case <-time.After(2 * time.Second):
}
puts, err := b.js.CreateKeyValue(b.ctx, jetstream.KeyValueConfig{Bucket: delivery.AssignerPutsBucket, History: 5})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PutKey("n1", "web")
asked := delivery.Put{FP: delivery.SourcePrefix2 + strings.Repeat("b", 64), Why: "a try", At: time.Now()}.Said(time.Now(), delivery.PutAsked, "asked")
raw, _ := delivery.EncodePut(asked)
if _, err := puts.Put(b.ctx, pkey, raw); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(15 * time.Second); ; time.Sleep(200 * time.Millisecond) {
// A change of a bucket it watches starts a pass, which opens the puts' bucket.
if _, err := b.buckets[delivery.PresenceBucket].Put(b.ctx, "n1", []byte(time.Now().String())); err != nil {
t.Fatal(err)
}
if e, err := puts.Get(b.ctx, pkey); err == nil {
if got, err := delivery.DecodePut(e.Value()); err == nil && got.Step == delivery.PutRefused && strings.Contains(got.Words, "not assigned") {
break
}
}
if time.Now().After(deadline) {
t.Fatal("the put was not taken further once its bucket was made")
}
}
}
// putJSON writes a value into one of the bed's buckets.
func (b *bed) putJSON(t *testing.T, bucket, key string, value []byte) {
t.Helper()
if _, err := b.buckets[bucket].Put(b.ctx, key, value); err != nil {
t.Fatalf("%s %s: %v", bucket, key, err)
}
}
// compose says the controller composed module for node at fp, with the digest d (a letter repeated) and the judgement of the
// move to it, in mesh-controller_composed.
func (b *bed) compose(t *testing.T, node, module, fp string, d byte, judgement delivery.Judgement) string {
t.Helper()
key, err := delivery.ComposedKey(node, module, fp)
if err != nil {
t.Fatal(err)
}
digest := "sha256:" + strings.Repeat(string(d), 64)
cd := delivery.ComposedDigest{Digest: digest, Judgement: judgement}
if judgement != delivery.JudgedNeither {
cd.Against, cd.Words = "sha256:"+strings.Repeat("0", 64), "cannot be told"
}
v, err := delivery.EncodeComposedDigest(cd)
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ControllerComposedBucket, key, v)
return digest
}
// manifest keeps a manifest in the indexer's bucket by its digest and answers the digest.
func (b *bed) manifest(t *testing.T, module, body string) string {
t.Helper()
digest := delivery.ManifestDigest([]byte(body))
key, _ := delivery.ManifestsKey(digest)
value, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: body})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ManifestsBucket, key, value)
return digest
}
// 1297/AC1: against the real bus, a fixture change touching one module on one machine is composed and answered within
// seconds under the checker's key of the check, naming the machine composed and the mesh-sdk it composed with.
func TestAChangeIsComposedOnTheBus(t *testing.T) {
b := startBed(t, "")
trunk := b.manifest(t, "web", `{"module":"web","version":"1"}`)
changed := b.manifest(t, "web", `{"module":"web","version":"2"}`)
mod, err := delivery.EncodeModule(delivery.Module{FP2: delivery.SourcePrefix2 + strings.Repeat("a", 64), Manifest: trunk, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ModulesBucket, "web", mod)
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
b.putJSON(t, provider.AssignmentsBucket, "ace.other", []byte(`{"node":"ace","module":"other","build":"build-2"}`))
heads := []delivery.Head{{Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
// 1927/AC1: the candidate carries the fingerprint set the SDK computes, as the indexer writes it.
pulls := []delivery.CandidatePull{{Head: heads[0],
Modules: []string{"web"}, Fps: map[string]delivery.CandidateModule{"web": {FP2: delivery.SourcePrefix2 + strings.Repeat("b", 64), Manifest: changed}}}}
cand, err := delivery.EncodeCandidate(delivery.Candidate{Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
b.putJSON(t, delivery.CandidatesBucket, ckey, cand)
began := time.Now()
b.start(t)
key, _ := delivery.CompositionKey("feat/x", hash)
var v delivery.CompositionVerdict
for deadline := time.Now().Add(10 * time.Second); ; time.Sleep(50 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("no composition under %s after 10 s", key)
}
e, err := b.buckets[delivery.AssignerCompositionsBucket].Get(b.ctx, key)
if err != nil {
continue
}
if v, err = delivery.DecodeCompositionVerdict(e.Value()); err != nil {
t.Fatal(err)
}
break
}
if v.State != delivery.CompositionPass || len(v.Machines) != 1 || v.Machines[0] != "g14" || v.SDK == "" {
t.Fatalf("composition = %+v, want a pass on g14 alone naming its mesh-sdk", v)
}
if took := time.Since(began); took > 5*time.Second {
t.Errorf("the composition took %s", took)
}
// 1622/AC3: a group without a mesh-sdk member builds no composer: no ask is put on the Go build seat.
seat, _ := delivery.AskStream(delivery.AskGo)
if st, err := b.js.Stream(b.ctx, seat); err != nil {
t.Fatal(err)
} else if info, err := st.Info(b.ctx); err != nil || info.State.Msgs != 0 {
t.Errorf("a group without a mesh-sdk member put %d asks on the Go build seat (%v)", info.State.Msgs, err)
}
}
// get waits for a key of one of the bed's buckets and answers its value.
func (b *bed) get(t *testing.T, bucket, key string) []byte {
t.Helper()
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[bucket].Get(b.ctx, key); err == nil {
return e.Value()
}
}
t.Fatalf("no %s %s after 10 s", bucket, key)
return nil
}
// 1297/B4 on the bus, 1906/D5: the controller sends ace, g14 and shanks builds while the canary ace does not run F healthy yet.
// Started whole, the assigner wants F on the canary and writes no difference at all: there is no send to compare with (ADR 0539
// §8). 1945/AC19, AC21: shanks, which runs the older fingerprint (composed and granted) with no want, gets its first want at that
// fingerprint, never at F; g14, which runs a build of the old form, gets none, so web is on neither list there and the assigner
// raises the condition naming g14, web and the canary rule; once ace runs F healthy, g14's first want names F and the condition is
// cleared.
func TestWantsAreWrittenAndNoDifferenceIs(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
built, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
if err != nil {
t.Fatal(err)
}
bkey, _ := delivery.BuilderBuildKey("web", fNew)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
for _, fp := range []string{fNew, fOld} {
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
}
runs := map[string]string{"ace": fOld, "g14": "build-9", "shanks": fOld}
for _, node := range []string{"ace", "g14", "shanks"} {
b.compose(t, node, "web", fNew, 'c', delivery.JudgedNeither)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, node, on)
have, _ := delivery.EncodeHave(delivery.Have{FP: runs[node], Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, node+".web", have)
b.putJSON(t, provider.AssignmentsBucket, node+".web", []byte(`{"node":"`+node+`","module":"web","build":"b-`+node+`"}`))
}
b.compose(t, "shanks", "web", fOld, 'a', delivery.JudgedNeither)
b.start(t)
wkey, _ := delivery.WantKey("ace", "web")
want, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey))
if err != nil || want.FP != fNew {
t.Fatalf("the canary ace's want = %+v, %v", want, err)
}
if w := b.wantNow(t, "shanks", "web", "shanks' first want at what it runs", func(w delivery.Want) bool { return w.FP != "" }); w.FP != fOld || w.Composed != "sha256:"+strings.Repeat("a", 64) || w.Newest != fNew {
t.Errorf("shanks' want = %+v, want the fingerprint it runs naming its composition, held back from F", w)
}
var neither asks.Condition
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && neither.Key == ""; time.Sleep(100 * time.Millisecond) {
for k, c := range b.conditionsSaid(t) {
if strings.Contains(k, ".neither.") && c.State == asks.StateRaised {
neither = c
}
}
}
if neither.Subject.Machine != "g14" || neither.Subject.ID != "web" || neither.Severity != asks.SeverityUrgent || !strings.Contains(neither.Explanation, "canary") {
t.Errorf("the condition of web on neither list on g14 = %+v", neither)
}
if keys, err := b.buckets[delivery.AssignerDifferencesBucket].Keys(b.ctx); err == nil && len(keys) != 0 {
t.Errorf("differences were written: %v", keys)
}
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, "want.g14.web"); err == nil {
t.Errorf("g14, which runs an old-form build and is held back by the canary, was given a want: %s", e.Value())
}
// 1945/AC21: once the canary ace runs F healthy, g14's first want names F by the rules as usual, and web leaves neither list
// there by itself: the condition is cleared.
healthyF, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthyF)
if w := b.wantNow(t, "g14", "web", "g14's first want at F once the canary runs it healthy", func(w delivery.Want) bool { return w.FP != "" }); w.FP != fNew || w.Composed != "sha256:"+strings.Repeat("c", 64) || w.Newest != "" {
t.Errorf("g14's want = %+v, want F with its composition", w)
}
cleared := false
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && !cleared; time.Sleep(100 * time.Millisecond) {
c, ok := b.conditionsSaid(t)[neither.Key]
cleared = ok && c.State == asks.StateCleared
}
if !cleared {
t.Errorf("the condition of web on neither list on g14 was not cleared once g14 got its want")
}
}
// 1297/B9 (the assigner's half, ADR 0528 clause 6): started whole, the assigner holds a fingerprint whose health check
// failed on a node and writes the ask about it in its own asks, which the view and the teller read; its asks bucket is
// the one its manifest declares (ownBuckets, held equal to the manifest by TestOpensWhatItsManifestNames).
func TestAHeldFingerprintIsAskedAbout(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("c", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putJSON(t, provider.AssignmentsBucket, "ace.web", []byte(`{"node":"ace","module":"web","build":"b-1"}`))
b.start(t)
hkey, _ := delivery.HeldKey("web", fNew)
// 1906/H3: the held record names the machine and keeps the health, when and the node's words.
if h, err := delivery.DecodeHeld(b.get(t, delivery.AssignerWantsBucket, hkey)); err != nil || h.Node != "ace" ||
!strings.HasPrefix(h.Words, "health failed, phase running, seen at ") || !strings.HasSuffix(h.Words, ": connection refused") {
t.Fatalf("held = %+v, %v", h, err)
}
akey, _ := delivery.HoldAskKey("web", fNew)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || a.Asking <= 0 || a.Ask.ID != delivery.AskingID(akey, a.Asking) || len(a.Ask.Options) != 2 {
t.Fatalf("the ask = %+v, %v", a, err)
}
}
// conditionsSaid reads what the assigner said on the operator channel's subject for conditions, by condition key.
func (b *bed) conditionsSaid(t *testing.T) map[string]asks.Condition {
t.Helper()
stream, err := b.js.Stream(b.ctx, "CONDITIONS")
if err != nil {
t.Fatal(err)
}
out := map[string]asks.Condition{}
cons, err := stream.CreateOrUpdateConsumer(b.ctx, jetstream.ConsumerConfig{AckPolicy: jetstream.AckNonePolicy})
if err != nil {
t.Fatal(err)
}
for {
batch, err := cons.FetchNoWait(100)
if err != nil {
t.Fatal(err)
}
n := 0
for m := range batch.Messages() {
n++
c, err := asks.DecodeCondition(m.Data())
if err != nil {
t.Fatal(err)
}
out[c.Key] = c
}
if n == 0 {
return out
}
}
}
// 1906/D3 (ADR 0539 §19), whole on the bus: a module holding data whose new fingerprint failed its health check on the
// canary is held, not set back, and asked about (restore it, or leave it stopped); the condition names the module, the node
// and the check's words.
func TestAFailedDataModuleIsAskedToRestore(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("d", 64)
digest := b.manifest(t, "mail", `{"module":"mail","version":"1","resources":[{"id":"data","type":"directory","path":"/var/lib/mail"}],"data":{"own":[{"id":"mail","path":"${dir:data}","class":"irreplaceable","backup":"copy"}]}}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/mail", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "mail", mod)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.mail", failed)
wantKey, _ := delivery.WantKey("ace", "mail")
want, _ := delivery.EncodeWant(delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("c", 64)})
b.putJSON(t, delivery.AssignerWantsBucket, wantKey, want)
b.putJSON(t, provider.AssignmentsBucket, "ace.mail", []byte(`{"node":"ace","module":"mail","build":"b-1"}`))
b.start(t)
akey, _ := delivery.RestoreAskKey("ace", "mail", fNew)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || len(a.Ask.Options) != 2 || a.Ask.Options[0].ID != "restore" || a.Ask.Options[1].ID != "leave-stopped" {
t.Fatalf("the restore ask = %+v, %v", a, err)
}
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wantKey)); err != nil || w.FP != fNew {
t.Errorf("the data module was set back: want = %+v, %v", w, err)
}
key := ""
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline) && key == ""; time.Sleep(100 * time.Millisecond) {
for k, c := range b.conditionsSaid(t) {
if c.State == asks.StateRaised && c.Subject.ID == "mail" && strings.Contains(k, ".held.") {
if c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") {
t.Fatalf("the condition = %+v, want it to name ace and the check's words", c)
}
key = k
}
}
}
if key == "" {
t.Fatal("no condition for the held fingerprint was said")
}
}
// 1906/D8 (scenario D of acceptance test 18, ADR 0539 §19), whole on the bus: the assigner is stopped while it writes the wants
// of forty modules and another is started; every module's want is written, and each exactly once, by revision: nothing lost
// and nothing written again.
func TestTheAssignerReplacedResumesWithNothingLostOrRepeated(t *testing.T) {
b := startBed(t, "")
const modules = 40
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
names := make([]string, modules)
for i := range names {
name := fmt.Sprintf("mod%02d", i)
names[i] = name
fp := delivery.SourcePrefix2 + strings.Repeat(string(rune('a'+i%6)), 63) + fmt.Sprintf("%x", i%16)
digest := b.manifest(t, name, fmt.Sprintf(`{"module":%q,"version":"1"}`, name))
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/" + name, Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, name, mod)
built, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
if err != nil {
t.Fatal(err)
}
bkey, _ := delivery.BuilderBuildKey(name, fp)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey(name, fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
b.compose(t, "ace", name, fp, 'c', delivery.JudgedNeither)
b.putJSON(t, provider.AssignmentsBucket, "ace."+name, []byte(fmt.Sprintf(`{"node":"ace","module":%q,"build":"b-%d"}`, name, i)))
}
count := func() int {
keys, err := b.buckets[delivery.AssignerWantsBucket].Keys(b.ctx)
if err != nil {
return 0
}
return len(keys)
}
_, stop := b.startStoppable(t)
for deadline := time.Now().Add(20 * time.Second); count() == 0 && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
stop() // replaced, wherever its pass had got to
b.start(t)
for deadline := time.Now().Add(30 * time.Second); count() < modules && time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
}
time.Sleep(time.Second) // and no pass of the second writes any again
for _, name := range names {
key, _ := delivery.WantKey("ace", name)
history, err := b.buckets[delivery.AssignerWantsBucket].History(b.ctx, key)
if err != nil {
t.Errorf("the want of %s was never written: %v", name, err)
continue
}
if len(history) != 1 {
t.Errorf("the want of %s was written %d times, want once", name, len(history))
}
}
}
// 1906/D2 (ADR 0539 §6), whole on the bus: a want waits for the composition of its module for the node, names its digest once it
// appears, and a new revision of the composition at the same fingerprint is written as a new want; the node's own want
// `want.<node>.node` is written once the node's composition exists, and again when its digest changes.
func TestAWantWaitsForItsComposition(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("e", 64)
digest := b.manifest(t, "web", `{"module":"web","version":"1"}`)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fNew, Manifest: digest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, "web", mod)
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
bkey, _ := delivery.BuilderBuildKey("web", fNew)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey("web", fNew)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
b.putJSON(t, provider.AssignmentsBucket, "ace.web", []byte(`{"node":"ace","module":"web","build":"b-1"}`))
b.start(t)
wkey, _ := delivery.WantKey("ace", "web")
time.Sleep(time.Second)
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, wkey); err == nil {
t.Fatalf("a want was written before the composition existed: %s", e.Value())
}
first := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedNeither)
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey)); err != nil || w.FP != fNew || w.Composed != first {
t.Fatalf("the want = %+v, %v; want F naming %s", w, err, first)
}
second := b.compose(t, "ace", "web", fNew, 'd', delivery.JudgedNeither)
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, wkey); err == nil {
if w, err := delivery.DecodeWant(e.Value()); err == nil && w.Composed == second {
break
}
}
}
if w, err := delivery.DecodeWant(b.get(t, delivery.AssignerWantsBucket, wkey)); err != nil || w.Composed != second {
t.Fatalf("after a new revision at the same fingerprint the want = %+v, %v; want it naming %s", w, err, second)
}
// the node's own want
nkey, _ := delivery.WantNodeKey("ace")
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, nkey); err == nil {
t.Fatalf("the node's own want was written before its composition: %s", e.Value())
}
ownKey, _ := delivery.ComposedNodeKey("ace")
own, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("a", 64), Judgement: delivery.JudgedNeither})
b.putJSON(t, delivery.ControllerComposedBucket, ownKey, own)
if w, err := delivery.DecodeNodeWant(b.get(t, delivery.AssignerWantsBucket, nkey)); err != nil || w.Composed != "sha256:"+strings.Repeat("a", 64) {
t.Fatalf("the node's own want = %+v, %v", w, err)
}
}
// fixture is a module web (or the manifest given) at fp on the node ace, which is online: built, granted and assigned.
func (b *bed) fixture(t *testing.T, name, fp, manifest string) {
t.Helper()
digest := b.manifest(t, name, manifest)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: digest, Repository: "novox/" + name, Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
b.putJSON(t, delivery.ModulesBucket, name, mod)
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "builder.novox.1", Step: "pushed", Manifest: digest})
bkey, _ := delivery.BuilderBuildKey(name, fp)
b.putJSON(t, delivery.BuilderBuildsBucket, bkey, built)
gkey, _ := delivery.ControllerGrantKey(name, fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
b.putJSON(t, delivery.ControllerGrantsBucket, gkey, grant)
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
b.putJSON(t, delivery.PresenceBucket, "ace", on)
b.putJSON(t, provider.AssignmentsBucket, "ace."+name, []byte(`{"node":"ace","module":"`+name+`","build":"b-1"}`))
}
func (b *bed) putWant(t *testing.T, node, module string, w delivery.Want) {
t.Helper()
key, _ := delivery.WantKey(node, module)
v, err := delivery.EncodeWant(w)
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.AssignerWantsBucket, key, v)
}
// wantNow waits until the want of module on node satisfies ok and answers it.
func (b *bed) wantNow(t *testing.T, node, module, what string, ok func(delivery.Want) bool) delivery.Want {
t.Helper()
key, _ := delivery.WantKey(node, module)
var last delivery.Want
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerWantsBucket].Get(b.ctx, key); err == nil {
if w, err := delivery.DecodeWant(e.Value()); err == nil {
last = w
if ok(w) {
return w
}
}
}
}
t.Fatalf("never saw: %s; the want is %+v", what, last)
return last
}
// answer says the teller wrote the operator's answer to the ask that stands at key (its warrant names that asking's id).
func (b *bed) answer(t *testing.T, key, option string) {
t.Helper()
a, err := delivery.DecodeAssignerAsk(key, b.get(t, delivery.AssignerAsksBucket, key))
if err != nil {
t.Fatal(err)
}
b.answerTo(t, key, option, a.Ask.ID)
}
// answerTo is answer to the ask with this id (the router's warrant names it).
func (b *bed) answerTo(t *testing.T, key, option, askID string) {
t.Helper()
v, err := delivery.EncodeAnswer(delivery.Answer{Option: option, Warrant: askID + "@2026-10-24T12:00:00Z", Words: "the operator"})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.AnswersBucket, key, v)
}
func (b *bed) hasKey(bucket, key string) bool {
_, err := b.buckets[bucket].Get(b.ctx, key)
return err == nil
}
// 1906/D4 (ADR 0539 §19), whole on the bus: the operator's release of a held fingerprint writes its want again, by the rules,
// and clears the condition.
func TestAReleaseAnswerWritesTheWantAgain(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
b.compose(t, "ace", "web", fOld, 'a', delivery.JudgedNeither)
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy) // it ran the older fingerprint healthy before
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
b.start(t)
w := b.wantNow(t, "ace", "web", "set back to the older fingerprint", func(w delivery.Want) bool { return w.FP == fOld })
if w.Newest != fNew {
t.Fatalf("the set-back want = %+v", w)
}
hold, _ := delivery.HoldAskKey("web", fNew)
b.get(t, delivery.AssignerAsksBucket, hold)
b.answer(t, hold, "release")
w = b.wantNow(t, "ace", "web", "the held fingerprint's want written again", func(w delivery.Want) bool { return w.FP == fNew })
if w.Composed != "sha256:"+strings.Repeat("b", 64) || w.Rule != 0 {
t.Errorf("the want = %+v", w)
}
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
cleared := false
for k, c := range b.conditionsSaid(t) {
cleared = cleared || (c.Subject.ID == "web" && c.State == asks.StateCleared && strings.Contains(k, ".held."))
}
if cleared {
return
}
}
t.Error("the condition of the held fingerprint was not cleared")
}
// askNow waits for the backup ask at key to be the one numbered asking and answers it.
func (b *bed) askNow(t *testing.T, key string, asking int) delivery.AssignerAsk {
t.Helper()
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.AssignerAsksBucket].Get(b.ctx, key); err == nil {
if a, err := delivery.DecodeAssignerAsk(key, e.Value()); err == nil && a.Asking == asking {
return a
}
}
}
t.Fatalf("the ask at %s never became asking %d", key, asking)
return delivery.AssignerAsk{}
}
// 1906/D4 (ADR 0539 §16), whole on the bus: a module held on a node because its backup failed is asked about, and the answer
// is written into the node's want; an answer "again" followed by another failure is asked again, and its answer is another want
// value, so the node retries; an old answer is never taken for a new revision at the same fingerprint.
func TestABackupAnswerIsWrittenIntoTheNodesWant(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
first := b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: first})
held, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseHeld, Health: delivery.HealthWaits, Detail: "no space left"})
b.putJSON(t, delivery.HaveBucket, "ace.web", held)
b.start(t)
akey, _ := delivery.BackupAskKey("ace", "web", fNew)
a := b.askNow(t, akey, 0)
if !strings.Contains(a.Ask.Explanation, "no space left") || !strings.Contains(a.Ask.Explanation, first) {
t.Fatalf("the backup ask = %+v", a)
}
// an answer to another ask is no answer to this one
b.answerTo(t, akey, delivery.BackupSkip, "someone-else")
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Backup != "" {
t.Fatalf("an answer to another ask was taken: %+v", w)
}
b.answerTo(t, akey, delivery.BackupAgain, a.Ask.ID)
w1 := b.wantNow(t, "ace", "web", "the answer in the want", func(w delivery.Want) bool { return w.Backup == delivery.BackupAgain })
if w1.FP != fNew || w1.Composed != first {
t.Errorf("the want = %+v", w1)
}
// the node retries and the backup fails again: a new have value, asked again under the next number
again, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseHeld, Health: delivery.HealthWaits, Detail: "no space left, again"})
b.putJSON(t, delivery.HaveBucket, "ace.web", again)
a2 := b.askNow(t, akey, 2)
if a2.Ask.ID == a.Ask.ID {
t.Fatalf("asked again under the same id")
}
b.answerTo(t, akey, delivery.BackupAgain, a2.Ask.ID)
w2 := b.wantNow(t, "ace", "web", "the second answer another want value", func(w delivery.Want) bool { return w.Backup == delivery.BackupAgain && w.Words != w1.Words })
if w2.Words == w1.Words {
t.Errorf("the second answer left the want as it was: %+v", w2)
}
// a new revision at the same fingerprint, whose backup fails as well, is asked, and the old answer is not taken for it
second := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedNeither)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: second})
a3 := b.askNow(t, akey, 3)
if !strings.Contains(a3.Ask.Explanation, second) {
t.Errorf("the ask for the new revision = %+v", a3)
}
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Backup != "" {
t.Errorf("the old answer was taken for the new revision: %+v", w)
}
}
const recorded = `{"module":"web","version":"1","upgrade":{"policy":"record","why":"a person takes each build"}}`
// 1906/D4 (ADR 0539 §14), whole on the bus: a move of a module of policy record waits for its ask's yes when judged to recreate
// a container, to move data or not to be judgeable, at a new fingerprint and at a new revision of the same alike; one judged
// neither is written at once.
func TestARecordMoveWaitsForTheYesAndANeitherMoveDoesNot(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "web", fNew, recorded)
old := "sha256:" + strings.Repeat("a", 64)
b.putWant(t, "ace", "web", delivery.Want{FP: fOld, Composed: old})
nw := b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedRecreates)
b.start(t)
akey, _ := delivery.RecordAskKey("ace", "web", nw)
a, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil || !strings.Contains(a.Ask.Explanation, "recreates a container") {
t.Fatalf("the record ask = %+v, %v", a, err)
}
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the old want", func(delivery.Want) bool { return true }); w.FP != fOld {
t.Fatalf("the move was written before the yes: %+v", w)
}
b.answer(t, akey, assign.OptionYes)
b.wantNow(t, "ace", "web", "the move after the yes", func(w delivery.Want) bool { return w.FP == fNew && w.Composed == nw })
// a new revision at the same fingerprint, judged to move data, waits as well
rev := b.compose(t, "ace", "web", fNew, 'c', delivery.JudgedMovesData)
rkey, _ := delivery.RecordAskKey("ace", "web", rev)
b.get(t, delivery.AssignerAsksBucket, rkey)
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "web", "the want", func(delivery.Want) bool { return true }); w.Composed != nw {
t.Fatalf("the revision was written before the yes: %+v", w)
}
// and a revision judged neither is written at once
free := b.compose(t, "ace", "web", fNew, 'd', delivery.JudgedNeither)
b.wantNow(t, "ace", "web", "a move judged neither written at once", func(w delivery.Want) bool { return w.Composed == free })
}
// 1906/D4 (ADR 0539 §7, §15), whole on the bus: the bus's new fingerprint is not written, and a revision of its composition at the
// fingerprint it runs that differs only in its user list is written at once, with the shape the controller gives it: judged
// recreates (the file is mounted), policy record from the bus, user-list-only; no record ask is made.
func TestAUserListRevisionOfTheBusIsWrittenAtOnce(t *testing.T) {
b := startBed(t, "")
fNew, fOld := delivery.SourcePrefix2+strings.Repeat("b", 64), delivery.SourcePrefix2+strings.Repeat("a", 64)
b.fixture(t, "bus", fNew, `{"module":"bus","version":"1","provides":[{"name":"mesh-bus"}]}`)
running, _ := delivery.EncodeHave(delivery.Have{FP: fOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.bus", running)
b.putWant(t, "ace", "bus", delivery.Want{FP: fOld, Composed: "sha256:" + strings.Repeat("a", 64)})
b.compose(t, "ace", "bus", fNew, 'b', delivery.JudgedNeither)
b.start(t)
time.Sleep(time.Second)
if w := b.wantNow(t, "ace", "bus", "the want", func(delivery.Want) bool { return true }); w.FP != fOld {
t.Fatalf("the assigner moved the bus to a new fingerprint: %+v", w)
}
// the controller composes the bus at the fingerprint it runs with a new user list
key, _ := delivery.ComposedKey("ace", "bus", fOld)
free := "sha256:" + strings.Repeat("e", 64)
v, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: free, Judgement: delivery.JudgedRecreates, Against: "sha256:" + strings.Repeat("a", 64),
Words: "the user list file is mounted", UserListOnly: true, Policy: delivery.PolicyRecord, PolicyFrom: "the bus"})
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.ControllerComposedBucket, key, v)
w := b.wantNow(t, "ace", "bus", "the user-list revision written at once", func(w delivery.Want) bool { return w.Composed == free })
if w.FP != fOld {
t.Errorf("the want = %+v, want it at the fingerprint the bus runs", w)
}
rkey, _ := delivery.RecordAskKey("ace", "bus", free)
if b.hasKey(delivery.AssignerAsksBucket, rkey) {
t.Errorf("the user-list revision was asked")
}
}
// 1906/H2, whole on the bus: given a fingerprint held after one failed check, when every node that runs the module then runs
// it healthy, the assigner lifts the hold by itself: the held record and its ask are taken away, the held condition is
// cleared saying why, and the want stays at the fingerprint.
func TestAHoldIsLiftedOnceEveryNodeRunsItHealthy(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
b.start(t)
hkey, _ := delivery.HeldKey("web", fNew)
akey, _ := delivery.HoldAskKey("web", fNew)
b.get(t, delivery.AssignerWantsBucket, hkey)
b.get(t, delivery.AssignerAsksBucket, akey)
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
if b.hasKey(delivery.AssignerWantsBucket, hkey) || b.hasKey(delivery.AssignerAsksBucket, akey) {
continue
}
for k, c := range b.conditionsSaid(t) {
if c.Subject.ID == "web" && c.State == asks.StateCleared && strings.Contains(k, ".held.") && strings.Contains(c.Resolved, "lifted its hold by itself") {
if w := b.wantNow(t, "ace", "web", "the want at the lifted fingerprint", func(w delivery.Want) bool { return w.FP == fNew }); w.Rule != 0 {
t.Errorf("the want = %+v", w)
}
return
}
}
}
t.Fatalf("the hold was not lifted: held %v, ask %v", b.hasKey(delivery.AssignerWantsBucket, hkey), b.hasKey(delivery.AssignerAsksBucket, akey))
}
// 1906/H2, whole on the bus: a hold lifted by itself and written again after the assigner was started again is asked under a
// new id, so the teller raises it; the answer to the first hold does not carry over (1906/H4).
func TestAHoldWrittenAgainAfterALiftAndARestartIsAskedAgain(t *testing.T) {
b := startBed(t, "")
fNew := delivery.SourcePrefix2 + strings.Repeat("b", 64)
b.fixture(t, "web", fNew, `{"module":"web","version":"1"}`)
b.compose(t, "ace", "web", fNew, 'b', delivery.JudgedNeither)
failed, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed, Detail: "connection refused"})
healthy, _ := delivery.EncodeHave(delivery.Have{FP: fNew, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.putWant(t, "ace", "web", delivery.Want{FP: fNew, Composed: "sha256:" + strings.Repeat("b", 64)})
_, stop := b.startStoppable(t)
hkey, _ := delivery.HeldKey("web", fNew)
akey, _ := delivery.HoldAskKey("web", fNew)
first, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil {
t.Fatal(err)
}
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); b.hasKey(delivery.AssignerWantsBucket, hkey); time.Sleep(100 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("the hold was not lifted")
}
}
stop()
time.Sleep(1100 * time.Millisecond) // the next hold is written in another second
b.answerTo(t, akey, "keep", first.Ask.ID) // an answer to the first hold, which the later hold must not take
b.putJSON(t, delivery.HaveBucket, "ace.web", failed)
b.start(t)
again, err := delivery.DecodeAssignerAsk(akey, b.get(t, delivery.AssignerAsksBucket, akey))
if err != nil {
t.Fatal(err)
}
if again.Ask.ID == first.Ask.ID {
t.Fatalf("the hold written again is asked under the first id %s", first.Ask.ID)
}
b.putJSON(t, delivery.HaveBucket, "ace.web", healthy)
for deadline := time.Now().Add(15 * time.Second); b.hasKey(delivery.AssignerWantsBucket, hkey); time.Sleep(100 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("the stale \"keep\" of the first hold kept the second one held")
}
}
}
@@ -0,0 +1,325 @@
// Command mesh-delivery-assigner is delivery's assigner (novox/hq ADR 0429 §1 and §6, ADR 0446, ADR 0528, issue 1297): a
// process on its own bus credential that watches what the indexer, the builder, the controller, the node-engines and
// the watchman write, and writes what each node should run (`want.<node>.<module>`, by the six rules), the
// fingerprints it holds, every difference between a want and what the controller sends, and the composition check's
// verdict of each group the checker judges. In release 5 the wants are only compared: the controller still sends what
// each node runs (ADR 0429 §7). It holds no timer: every pass is started by a change it watched.
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"strings"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
)
func main() {
if os.Getenv("MESH_BROKER_FILE") == "" {
// Started by the tool runner, with no credential of its own: the put's tools (ADR 0546), served through the runtime,
// since a process never serves (ADR 0436). The process the manifest runs names its credential.
if err := serveTools(); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-assigner:", err)
os.Exit(1)
}
return
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-assigner:", err)
os.Exit(1)
}
}
// readBuckets are the buckets of other parts the assigner reads, each named in its manifest's reads (the manifest test
// holds the two lists equal), so the controller's broker grants exactly these.
var readBuckets = []string{
provider.AssignmentsBucket,
delivery.BuildsBucket,
delivery.ControllerGrantsBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ManifestsBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.StepsBucket(goBuildModule),
}
// goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes
// their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go"
// ownBuckets are the assigner's own states, as its manifest declares them.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket}
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546).
var keptStates = []announce.State{
{Name: "asks", Readable: true},
{Name: "compositions", Readable: true},
{Name: "differences", Readable: true},
{Name: "puts", Readable: true},
{Name: "wants", Readable: true},
}
// buckets is every bucket the assigner opened, by name.
type buckets map[string]jetstream.KeyValue
// open opens every bucket the assigner reads and writes; a bucket it cannot open stops the start, named with its grant (issue 1945),
// since a pass without it would write wants from half the mesh. The puts' bucket is the exception: whatever keeps it from opening (it
// does not exist yet, the credential is not granted it yet, the bus does not answer in time), the assigner writes wants
// without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kv, err := js.KeyValue(ctx, name)
if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue
}
if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
}
out[name] = kv
}
return out, nil
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
if path == "" {
return errors.New("MESH_BROKER_FILE is not set, so there is no credential to read the bus with")
}
account, err := process.ReadAccount(path)
if err != nil {
return err
}
opts := []nats.Option{
nats.Name(account.Node + "." + account.Module),
nats.UserInfo(account.User, account.Password),
nats.CustomInboxPrefix(account.InboxPrefix()),
nats.MaxReconnects(-1),
nats.RetryOnFailedConnect(true),
}
if strings.TrimSpace(account.Fingerprint) != "" {
opts = append(opts, nats.Secure(process.PinnedToFingerprint(account.Fingerprint)))
}
conn, err := nats.Connect(account.URL, opts...)
if err != nil {
return fmt.Errorf("connecting to the bus: %w", err)
}
defer conn.Close()
js, err := jetstream.New(conn)
if err != nil {
return err
}
kvs, err := open(ctx, js, logger.Printf)
if err != nil {
return err
}
// What the assigner keeps, said on the bus (ADR 0495 §2-§3): a reader is granted from it. Not fatal when it cannot
// be said now: process.Conn says it again at every reconnect.
said, err := process.Connect(account, nats.Name(account.Node+"."+account.Module+".keeps"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer said.Close()
if err := said.Keeps(keptStates...); err != nil {
logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err)
}
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, kvs, sayVia(ctx, js), emitting{said}, logger)
}
// emitting is the process connection as the puts' caller: its calls, and its events with no options.
type emitting struct{ *process.Conn }
func (e emitting) Emit(eventType string, body any) error { return e.Conn.Emit(eventType, body) }
// watched are the buckets whose every value a pass reads; the manifests are read by digest when a pass needs one, and
// watched only for keys, so a manifest that arrives wakes a composition that waits for it.
var watched = []string{
provider.AssignmentsBucket,
delivery.ControllerGrantsBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.CandidatesBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.AssignerAsksBucket,
delivery.AssignerCompositionsBucket,
delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket,
}
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
select {
case dirty <- struct{}{}:
default:
}
}
errs := make(chan error, len(watched)+2)
settled := make(chan struct{}, len(watched)+2)
watch := func(name string, opts ...jetstream.WatchOpt) error {
kw, err := kvs[name].WatchAll(ctx, opts...)
if err != nil {
return fmt.Errorf("the bucket %s cannot be watched: %w", name, err)
}
// The puts' watch never stops the wants (ADR 0546): when it ends, the puts are no longer heard, said in the journal.
optional := name == delivery.AssignerPutsBucket
go func() {
defer kw.Stop()
for {
select {
case <-ctx.Done():
errs <- nil
return
case e, ok := <-kw.Updates():
if !ok {
if optional {
logger.Printf("the watch of %s ended: no put is taken further until the assigner starts again", name)
return
}
errs <- fmt.Errorf("the watch of %s ended", name)
return
}
if e == nil {
if !optional { // the puts never gate the first pass, so never the wants (ADR 0546)
settled <- struct{}{}
}
continue
}
var value []byte
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
w.put(name, e.Key(), value)
wake()
}
}
}()
return nil
}
// The puts' watch is not waited for before the first pass: a watch of them that ends before it handed what stands only
// stops the puts, never the wants.
if kvs[delivery.AssignerPutsBucket] != nil {
if err := watch(delivery.AssignerPutsBucket); err != nil {
logger.Printf("%v: no put is taken further until it is opened again", err)
delete(kvs, delivery.AssignerPutsBucket)
}
}
// The puts' bucket is made by the controller from the manifest's states at its start (ADR 0201), and the assigner's
// credential reads it once the bus's user list carries the grant: neither waits for the assigner. So while it is not open,
// a pass started by a change opens it again, at most once a minute, and from then on the puts are watched and taken
// further, without a restart.
var triedPuts time.Time
reopenPuts := func(p *passer) {
if kvs[delivery.AssignerPutsBucket] != nil || time.Since(triedPuts) < PutsReopenEvery {
return
}
triedPuts = time.Now()
kv, err := js.KeyValue(ctx, delivery.AssignerPutsBucket)
if err != nil {
return
}
kvs[delivery.AssignerPutsBucket] = kv
if err := watch(delivery.AssignerPutsBucket); err != nil {
logger.Printf("%v: no put is taken further until it is opened again", err)
delete(kvs, delivery.AssignerPutsBucket)
return
}
if bus != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
}
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
}
for _, name := range watched {
var opts []jetstream.WatchOpt
if name == delivery.HaveBucket {
opts = append(opts, jetstream.IncludeHistory()) // what each node ran healthy before, for a set-back
}
if err := watch(name, opts...); err != nil {
return err
}
}
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err
}
for range len(watched) + 1 {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf)
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
}
wake()
for {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
case <-dirty:
reopenPuts(p)
p.pass(ctx)
}
}
}
// PutsReopenEvery is how often, at most, a pass opens the puts' bucket again while it is not open.
var PutsReopenEvery = time.Minute
// sayVia publishes a condition on the operator-channel seat's one subject for conditions, under the assigner's own name
// (ADR 0437 clause 9).
func sayVia(ctx context.Context, js jetstream.JetStream) func(asks.Condition) error {
return func(c asks.Condition) error {
subject, err := asks.ConditionSubject(delivery.AssignerModule)
if err != nil {
return err
}
data, err := asks.EncodeCondition(c)
if err != nil {
return err
}
_, err = js.Publish(ctx, subject, data)
return err
}
}
@@ -0,0 +1,172 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"os"
"path/filepath"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
// (`<module>.<state>` read as the bucket `<module>_<state>`), so it is granted what it reads and reads nothing else.
func TestOpensWhatItsManifestNames(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
var reads []string
for _, r := range m.Reads {
reads = append(reads, strings.Replace(r, ".", "_", 1))
}
var states []string
for _, s := range m.State {
states = append(states, m.Module+"_"+s.Name)
}
for _, pair := range []struct {
what string
code, want []string
}{{"reads", readBuckets, reads}, {"states", ownBuckets, states}} {
got, want := slices.Sorted(slices.Values(pair.code)), slices.Sorted(slices.Values(pair.want))
if !reflect.DeepEqual(got, want) {
t.Errorf("the program's %s are %v, the manifest's %v", pair.what, got, want)
}
}
var kept []string
for _, s := range keptStates {
kept = append(kept, m.Module+"_"+s.Name)
}
if got, want := slices.Sorted(slices.Values(kept)), slices.Sorted(slices.Values(ownBuckets)); !reflect.DeepEqual(got, want) {
t.Errorf("the program announces %v, keeps %v", got, want)
}
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
for _, c := range []struct {
missing string
err error
starts bool
}{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false},
} {
var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged = fmt.Sprintf(f, a...) })
switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, "no put is taken further")):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant
t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err)
}
}
// And a watch of the puts that ends before it handed what stands (its subscription closed, its grant withdrawn) does not
// keep the first pass from being made: the wants are written all the same.
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
wrote := make(chan string, 8)
kvs := buckets{}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kvs[name] = &watchKV{memKV: &memKV{values: map[string][]byte{}}}
}
ckey, _ := delivery.ComposedNodeKey("ace")
cd, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("d", 64), Judgement: delivery.JudgedUnchanged})
if err != nil {
t.Fatal(err)
}
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0))
select {
case key := <-wrote:
if key != "want.ace.node" {
t.Errorf("the first pass wrote %s", key)
}
case <-ctx.Done():
t.Error("a puts watch that ended before its first marker kept the first pass from being made: no want was written")
}
}
// 1297/B2a: without a credential the program does not start, and says why.
func TestNoCredential(t *testing.T) {
err := run(context.Background(), func(string) string { return "" }, io.Discard)
if err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
}
}
// refusingJS opens every bucket but one, which it refuses with err.
type refusingJS struct {
jetstream.JetStream
missing string
err error
}
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing {
return nil, r.err
}
return &memKV{values: map[string][]byte{}}, nil
}
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
// closed; a put on it is said on wrote.
type watchKV struct {
*memKV
entries []kvEntry
closed bool
wrote chan string
}
func (w *watchKV) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
ch := make(chan jetstream.KeyValueEntry, len(w.entries)+1)
if w.closed {
close(ch)
return keyWatcher{ch}, nil
}
for _, e := range w.entries {
ch <- e
}
ch <- nil
return keyWatcher{ch}, nil
}
func (w *watchKV) Put(ctx context.Context, key string, value []byte) (uint64, error) {
if w.wrote != nil {
w.wrote <- key
}
return 1, nil
}
type keyWatcher struct{ ch chan jetstream.KeyValueEntry }
func (k keyWatcher) Updates() <-chan jetstream.KeyValueEntry { return k.ch }
func (k keyWatcher) Stop() error { return nil }
type kvEntry struct {
jetstream.KeyValueEntry
key string
value []byte
}
func (e kvEntry) Key() string { return e.key }
func (e kvEntry) Value() []byte { return e.value }
func (e kvEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
@@ -0,0 +1,669 @@
package main
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/sdkversion"
"mesh-delivery-assigner/internal/assign"
)
// passer makes the assigner's passes: from what the world holds now, it writes each key whose value differs from the
// one written, and only those (the Same functions).
type passer struct {
kvs buckets
w *world
logf func(string, ...any)
sdk string
// say publishes a condition to the operator; said is, by condition key and state, what this process has said, so a
// pass says a condition once and a restart says it again (the router keeps one per key).
say func(asks.Condition) error
said map[string]bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64
asks *composeAsks
// now is the clock; nil is time.Now.
now func() time.Time
// neither is, by node and module, each module on neither list whose condition this process raised (issue 1945). Nil until the
// first pass after a start, which clears what a previous process may have raised (startClears).
neither map[string]neitherOne
// acts are the puts' acts (ADR 0546); nil: the puts are not taken further (no bus to call, or no puts bucket).
acts *putActs
// noWant is, by node and module, the reason last said why the node gets no want for the module (issue 1945): a reason is
// said in the journal when it appears or changes, and its clearing when the node has a want again; one line per change.
noWant map[string]string
}
func (p *passer) clock() time.Time {
if p.now != nil {
return p.now()
}
return time.Now()
}
func (p *passer) pass(ctx context.Context) {
p.compositions(ctx)
p.wants(ctx)
p.puts(ctx)
p.nodeWants(ctx)
p.backups(ctx)
}
// nodeWants writes each node's own want, `want.<node>.node`, naming the digest of the node's own composition, once the
// controller has composed it and again whenever that digest changes (ADR 0539 §6).
func (p *passer) nodeWants(ctx context.Context) {
written := p.w.read(delivery.AssignerWantsBucket)
for node, d := range p.w.nodeComposed() {
key, err := delivery.WantNodeKey(node)
if err != nil {
continue
}
if before, err := delivery.DecodeNodeWant(written[key]); err == nil && before.Composed == d.Digest {
continue
}
want := delivery.NodeWant{Composed: d.Digest, Words: "the node's own composition"}
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeNodeWant(want) })
}
}
// wants writes, for every module the trunk holds a fingerprint of, each node's want by the six rules (ADR 0429 §6), and
// the hold of a fingerprint whose health check failed. The assigner compares its wants with nothing the controller
// sends: the differences bucket is kept read-only with its history (ADR 0539 §8).
func (p *passer) wants(ctx context.Context) {
get := kvGetter(p.kvs[delivery.ManifestsBucket])
assigned, _ := p.w.assigned()
trunk := p.w.trunk()
manifests := map[string]catalogue.Manifest{}
for name, mod := range trunk {
if mod.Manifest == "" {
continue
}
if got := p.w.manifest(ctx, get, mod.Manifest); got.err == "" {
manifests[name] = got.manifest
}
}
nodes := p.w.nodes(manifests, assigned)
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
nodesOf[m] = append(nodesOf[m], node)
}
}
wanted := p.w.wantsOf()
puts := p.putsByModule()
p.staleAsks(ctx, trunk)
names := map[string]bool{}
for name := range trunk {
names[name] = true
}
for name := range wanted {
names[name] = true
}
neither := map[string]neitherOne{}
// The first pass after a start remembers nothing a previous process said: a neither or held condition it raised would stay
// raised for ever once the module has its want or is no longer held. That pass clears, once, the neither condition of every
// assigned module not on neither list and the held condition of every module not held; a clear of a condition that is not open is
// quiet at the router.
first := p.neither == nil
stale := map[string]neitherOne{}
defer func() { p.sayNeither(neither, stale) }()
for name := range names {
fp := trunk[name].FP2
name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
Composed: func(node, fp string) (delivery.ComposedDigest, bool) { return p.w.composed(node, name, fp) },
GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key)
},
Restored: func(node string) bool {
key, err := delivery.RestoreAskKey(node, name, fp)
return err == nil && p.answeredWith(key, assign.OptionRestore)
},
Allowed: func(node, digest string) bool {
key, err := delivery.RecordAskKey(node, name, digest)
return err == nil && p.answeredWith(key, assign.OptionYes)
}}
if fp != "" {
held, _ := delivery.HeldKey(name, fp)
grant, _ := delivery.ControllerGrantKey(name, fp)
hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(delivery.ControllerGrantsBucket, grant)
if raw := p.w.value(delivery.ControllerGrantsBucket, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
}
}
if m.Released {
p.clearHeld(name, fp)
}
}
last := map[string]string{}
for _, node := range m.Nodes {
last[node] = p.w.lastHealthy(node, name, fp)
}
out := assign.Wants(m, nodes, wanted[name], last)
gatherNeither(neither, name, out)
p.sayNoWant(name, m.Nodes, wanted[name], out)
if first {
startClears(stale, m, out)
if fp != "" && !m.Held && out.Held == nil && !out.Lifted {
p.clearHeld(name, fp)
}
}
if out.Held != nil {
key, err := delivery.HeldKey(name, fp)
if err == nil {
held := *out.Held
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeHeld(held) })
}
}
if out.Lifted {
p.lift(ctx, name, fp)
}
if out.Held != nil || (m.Held && !out.Lifted) {
p.holdAsk(ctx, name, fp, out.Held)
p.heldCondition(ctx, name, fp, out.Held, len(out.Restore) > 0)
}
for _, node := range out.Restore {
p.restoreAsk(ctx, name, node, fp)
}
for _, need := range out.Record {
p.recordAsk(ctx, name, fp, need)
}
for node, want := range out.Wants {
key, err := delivery.WantKey(node, name)
if err != nil {
p.logf("no want of %s on %s is written: %v", name, node, err)
continue
}
want := want
p.put(ctx, delivery.AssignerWantsBucket, key, func() ([]byte, error) { return delivery.EncodeWant(want) })
}
}
}
// sayNoWant says in the journal why a node of module gets no want after this pass, from the rule the outcome names for it
// (issue 1945: a new assignment whose trunk build was not done was held back with nothing said), once per reason, and says
// when the node has a want again. A node with a live want, written before and kept or written in this pass, has one.
func (p *passer) sayNoWant(module string, nodes []string, before map[string]delivery.Want, out assign.Outcome) {
if p.noWant == nil {
p.noWant = map[string]string{}
}
assigned := map[string]bool{}
for _, node := range nodes {
assigned[node] = true
key := node + "." + module
w, written := out.Wants[node]
if !written {
w, written = before[node]
}
reason := ""
if h, held := out.Rules[node]; held && (!written || w.Removed) {
reason = fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
}
last, said := p.noWant[key]
switch {
case reason != "" && reason != last:
p.logf("%s on %s gets no want: %s", module, node, reason)
p.noWant[key] = reason
case reason == "" && said && written && !w.Removed:
p.logf("%s on %s has a want", module, node)
delete(p.noWant, key)
}
}
for key := range p.noWant {
if node, m, _ := strings.Cut(key, "."); m == module && !assigned[node] {
delete(p.noWant, key) // no longer assigned there: its removed want says so
}
}
}
// neitherOne is a module assigned to a node that holds no want for it after a pass, and the rule holding its first want back.
type neitherOne struct {
node, module string
holding assign.Holding
}
// gatherNeither adds the nodes on which module is on neither list in its outcome, keyed by node and module.
func gatherNeither(into map[string]neitherOne, module string, out assign.Outcome) {
for node, h := range out.Neither {
into[node+"."+module] = neitherOne{node: node, module: module, holding: h}
}
}
// startClears adds, for the first pass after a start, every node of m that is not on neither list after the pass: a neither
// condition a previous process raised for it is cleared. That covers a node with a want and one with none that is not on neither
// list, such as a held module (issue 1945: the hold is said on its own, so a neither raised for it before the restart would stay up
// for ever) or a move waiting for its record ask. Clearing a condition that is not open is quiet at the router.
func startClears(into map[string]neitherOne, m assign.Module, out assign.Outcome) {
for _, node := range m.Nodes {
if _, on := out.Neither[node]; !on {
into[node+"."+m.Name] = neitherOne{node: node, module: m.Name}
}
}
}
// sayNeither raises, once, the condition of every module on neither list in this pass and writes it to the journal, and clears,
// once, the condition of each one said before that has a want again (issue 1945). A say that fails is made again at the next pass.
// stale is, on the first pass after a start, the modules whose condition a previous process may have raised and that have a want:
// each is cleared once.
func (p *passer) sayNeither(now, stale map[string]neitherOne) {
if p.neither == nil {
p.neither = map[string]neitherOne{}
for key, one := range stale {
if _, on := now[key]; !on {
p.neither[key] = one
}
}
}
for key, one := range now {
if _, said := p.neither[key]; said {
continue
}
c := assign.NeitherCondition(one.node, one.module, one.holding, true)
if p.say != nil {
if err := p.say(c); err != nil {
p.logf("the condition of %s on neither list on %s could not be said, and is said at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s is assigned to %s and on neither list: no want, and not in the node's own composition; rule %d holds its first want back: %s",
one.module, one.node, one.holding.Rule, one.holding.Words)
p.neither[key] = one
}
for key, one := range p.neither {
if _, still := now[key]; still {
continue
}
if p.say != nil {
if err := p.say(assign.NeitherCondition(one.node, one.module, assign.Holding{}, false)); err != nil {
p.logf("the condition of %s on neither list on %s could not be cleared, and is cleared at the next pass: %v", one.module, one.node, err)
continue
}
}
p.logf("%s on %s has a want again", one.module, one.node)
delete(p.neither, key)
}
}
// compositions writes the composition check's verdict of every group the indexer holds a candidate for, under the
// checker's key of that check: waits, with why, while the indexer holds what it composes from only in part.
func (p *passer) compositions(ctx context.Context) {
if p.sdk == "" {
p.sdk = sdkversion.Version()
}
written := p.w.read(delivery.AssignerCompositionsBucket)
trunk := p.w.trunk()
var mesh *assign.Mesh
seen := map[string]bool{}
if p.asks != nil {
defer p.asks.sweep(seen)
}
for key, value := range p.w.read(delivery.CandidatesBucket) {
c, err := delivery.DecodeCandidate(value)
if err != nil {
p.logf("the candidate %s cannot be read, so it is not composed: %v", key, err)
continue
}
out, err := delivery.CompositionKey(c.Group, c.HeadsHash)
if err != nil {
p.logf("the candidate %s names no composition key: %v", key, err)
continue
}
began := time.Now()
ch, waits := p.w.change(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]), c, trunk)
var v delivery.CompositionVerdict
if waits != "" {
v = delivery.CompositionVerdict{Group: c.Group, Heads: c.HeadsHash, State: delivery.CompositionWaits, Words: waits}
} else {
if mesh == nil {
m := p.w.mesh(ctx, kvGetter(p.kvs[delivery.ManifestsBucket]))
mesh = &m
}
if ch.SDKHead != "" && p.asks != nil {
// A group with a mesh-sdk member is composed by the composer built at its head (ADR 0482 clauses 3 to 5).
v = p.composeAtHead(ctx, *mesh, ch, seen, began)
} else {
v = assign.Compose(*mesh, ch, p.sdk, time.Since(began))
}
}
if before, err := delivery.DecodeCompositionVerdict(written[out]); err == nil {
before.Took, before.V = v.Took, 0
if delivery.SameCompositionVerdict(before, v) {
continue
}
}
p.put(ctx, delivery.AssignerCompositionsBucket, out, func() ([]byte, error) { return delivery.EncodeCompositionVerdict(v) })
}
}
// put writes one key of the assigner's own bucket; a write that fails is said and made again at the next pass.
func (p *passer) put(ctx context.Context, bucket, key string, encode func() ([]byte, error)) {
value, err := encode()
if err != nil {
p.logf("%s %s is not written: %v", bucket, key, err)
return
}
if _, err := p.kvs[bucket].Put(ctx, key, value); err != nil && !errors.Is(err, context.Canceled) {
p.logf("%s %s could not be written: %v", bucket, key, err)
return
}
p.w.put(bucket, key, value)
}
// holdAsk writes, once, the ask about a held fingerprint (ADR 0528 clause 6, ADR 0437 clause 10): the first ask that needs
// no proof of who answered, which the view may answer and the teller fills. In release 5 an answer changes nothing on a
// node (wants are only compared, ADR 0429 §7), and the options say so.
func (p *passer) holdAsk(ctx context.Context, module, fp string, fresh *delivery.Held) {
key, err := delivery.HoldAskKey(module, fp)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
// The asking is the hold's time in seconds: a hold of the same fingerprint written again after a lift is another asking, under
// an id of its own, also after a restart, since the router drops a second ask under an id it keeps (issue 1906).
now := p.clock()
asking := int(now.Unix())
h := p.heldHold(module, fp, fresh)
a := assign.HoldAsk(key, module, fp, h.Node, h.Words, asking, now)
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// lift takes away a hold the assigner lifted by itself, since every node that runs the module runs the held fingerprint healthy
// (issue 1906): the held record and its ask (the teller then takes the ask back from the operator), and the held condition is
// cleared saying why, once. A hold of the same fingerprint written later is said and asked again.
func (p *passer) lift(ctx context.Context, module, fp string) {
held, err := delivery.HeldKey(module, fp)
if err != nil {
return
}
ask, err := delivery.HoldAskKey(module, fp)
if err != nil {
return
}
c := assign.LiftedCondition(module, fp)
if !p.said[c.Key+".lifted"] {
if p.say != nil {
if err := p.say(c); err != nil {
// The hold stays written, so the next pass lifts it again and says so then.
p.logf("the condition of the lifted hold of %s could not be cleared, and is cleared at the next pass: %v", module, err)
return
}
}
p.logf("%s", assign.LiftedWords(module, fp))
p.said[c.Key+".lifted"] = true
delete(p.said, c.Key+".raised") // a hold written again is said again
}
p.remove(ctx, delivery.AssignerWantsBucket, held)
p.remove(ctx, delivery.AssignerAsksBucket, ask)
// The restore asks of that fingerprint, on every node, go with the hold: every node runs it healthy, so no node waits to be
// set back from it (issue 1906, note 28).
for key := range p.w.read(delivery.AssignerAsksBucket) {
if k, err := delivery.ParseAnyAskKey(key); err == nil && k.Kind == delivery.AskKindRestore && k.Module == module && k.FP == fp {
p.remove(ctx, delivery.AssignerAsksBucket, key)
}
}
}
// staleAsk says whether the hold, restore or record ask at key (read as the SDK reads it) is about something the trunk has left
// behind: a module no longer on the trunk, a fingerprint that is no longer the module's trunk fingerprint, or a composition the
// controller no longer composes for the node at that fingerprint. A backup ask and an ask of another kind are never stale here.
// A module of the trunk whose fingerprint is not read yet, and a composition not read yet, are not stale: nothing known is
// nothing taken away.
func (p *passer) staleAsk(key string, trunk map[string]delivery.Module) bool {
k, err := delivery.ParseAnyAskKey(key)
if err != nil {
return false
}
switch k.Kind {
case delivery.AskKindHold, delivery.AskKindRestore:
m, on := trunk[k.Module]
return !on || (m.FP2 != "" && m.FP2 != k.FP)
case delivery.AskKindRecord:
m, on := trunk[k.Module]
if !on {
return true
}
// The assigner asks about a composition at more than the trunk fingerprint: a node held back by the rules is asked about the
// revised composition of the fingerprint it runs (ADR 0545), and a node's first want at the older fingerprint it runs. So a
// record ask is stale only when its digest is the composition of none of the fingerprints the node can still want (the
// trunk's, its want's, its have's), and only when every one of them is read.
for _, fp := range p.fingerprintsOf(k.Node, k.Module, m.FP2) {
d, ok := p.w.composed(k.Node, k.Module, fp)
if !ok || d.Digest == k.Composed {
return false
}
}
return len(p.fingerprintsOf(k.Node, k.Module, m.FP2)) > 0
}
return false
}
// fingerprintsOf are the fingerprints a node may still want of module: the trunk's, the one its want names and the one its have
// names, each once.
func (p *passer) fingerprintsOf(node, module, trunkFP string) []string {
var out []string
add := func(fp string) {
if fp != "" && !slices.Contains(out, fp) {
out = append(out, fp)
}
}
add(trunkFP)
if key, err := delivery.WantKey(node, module); err == nil {
if w, err := delivery.DecodeWant(p.w.value(delivery.AssignerWantsBucket, key)); err == nil {
add(w.FP)
}
}
if key, err := delivery.HaveKey(node, module); err == nil {
if h, err := delivery.DecodeHave(p.w.value(delivery.HaveBucket, key)); err == nil {
add(h.FP)
}
}
return out
}
// staleAsks takes away, at every pass, each hold, restore and record ask that staleAsk names (issue 1906, note 28): an ask
// outlived its fingerprint, and the teller takes its router ask back from the operator when the key goes. With the trunk not
// read at all (no module), nothing is taken away.
func (p *passer) staleAsks(ctx context.Context, trunk map[string]delivery.Module) {
if len(trunk) == 0 {
return
}
for key := range p.w.read(delivery.AssignerAsksBucket) {
if p.staleAsk(key, trunk) {
p.logf("the ask %s is about a fingerprint or composition the trunk has left behind and is taken away", key)
p.remove(ctx, delivery.AssignerAsksBucket, key)
// The held record stays: it is the memory that this fingerprint failed (ADR 0408 §4), and a revert returns the trunk to
// exactly it. Only the condition, which would else ask for an answer that can no longer be given, is cleared, and it is
// raised again if the trunk returns to the fingerprint.
if k, err := delivery.ParseAnyAskKey(key); err == nil && (k.Kind == delivery.AskKindHold || k.Kind == delivery.AskKindRestore) {
p.clearHeld(k.Module, k.FP)
delete(p.said, assign.HeldCondition(k.Module, k.FP, "", "", false, false).Key+".raised")
}
}
}
}
// remove deletes one key of the assigner's own bucket when it holds it; a delete that fails is said and made again at the next
// pass.
func (p *passer) remove(ctx context.Context, bucket, key string) {
if !p.w.has(bucket, key) {
return
}
if err := p.kvs[bucket].Delete(ctx, key); err != nil && !errors.Is(err, context.Canceled) {
p.logf("%s %s could not be taken away: %v", bucket, key, err)
return
}
p.w.put(bucket, key, nil)
}
// heldHold is the hold of fp as written: the node whose health check failed and its words.
func (p *passer) heldHold(module, fp string, fresh *delivery.Held) delivery.Held {
if fresh != nil {
return *fresh
}
key, err := delivery.HeldKey(module, fp)
if err != nil {
return delivery.Held{}
}
h, _ := delivery.DecodeHeld(p.w.read(delivery.AssignerWantsBucket)[key])
return h
}
// heldCondition raises, once, the condition for a fingerprint the assigner holds (ADR 0429 §6, ADR 0539 §19): the module, the
// node whose health check failed and the check's words. A module holding data needs the operator's answer to its restore ask.
func (p *passer) heldCondition(ctx context.Context, module, fp string, fresh *delivery.Held, data bool) {
h := p.heldHold(module, fp, fresh)
c := assign.HeldCondition(module, fp, h.Node, h.Words, true, data)
if p.say == nil || p.said[c.Key+".raised"] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the held %s could not be said, and is said at the next pass: %v", module, err)
return
}
p.said[c.Key+".raised"] = true
delete(p.said, c.Key+".lifted") // a hold raised again and lifted again is said lifted again
delete(p.said, c.Key+".cleared") // and released again is cleared again
}
// restoreAsk writes, once, the ask about a module holding data whose held fingerprint failed its health check on node.
func (p *passer) restoreAsk(ctx context.Context, module, node, fp string) {
key, err := delivery.RestoreAskKey(node, module, fp)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
words := p.heldHold(module, fp, nil).Words
a := assign.RestoreAsk(key, node, module, fp, words, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// answeredWith says whether the teller wrote an answer to the ask at key as it stands now and the option chosen is option. The
// answer is the operator's, given in the view or on the phone, and stays under its key (ADR 0539 §14, §16, §19); it counts only
// for the asking it answered (its warrant names that asking's id), so an answer to an ask taken away, or to an earlier hold of
// the same fingerprint, does not carry over to a later one (issue 1906).
func (p *passer) answeredWith(key, option string) bool {
a, ok := p.w.answer(key)
if !ok || a.Option != option {
return false
}
ask, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return false
}
id, _, _ := strings.Cut(a.Warrant, "@")
return id == ask.Ask.ID
}
// clearHeld clears, once, the condition of a hold the operator released.
func (p *passer) clearHeld(module, fp string) {
c := assign.HeldCondition(module, fp, "", "", false, false)
if p.say == nil || p.said[c.Key+".cleared"] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the released %s could not be cleared, and is cleared at the next pass: %v", module, err)
return
}
p.said[c.Key+".cleared"] = true
}
// recordAsk writes, once, the ask about a move of a module of policy record that waits for the operator's yes.
func (p *passer) recordAsk(ctx context.Context, module, fp string, need assign.RecordNeed) {
key, err := delivery.RecordAskKey(need.Node, module, need.Digest)
if err != nil || p.w.has(delivery.AssignerAsksBucket, key) {
return
}
a := assign.RecordAsk(key, need.Node, module, fp, need, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
}
// backups sees to the modules held on a node because the backup before an apply failed (the have's phase held, ADR 0539 §16):
// it asks whether to back up again or go on without a backup, raises the condition naming node, module and the backup's words,
// and writes the answer into the node's want for the module, which the node agent reads from it and from nothing else.
//
// The ask is bound to the composition (it names the digest, and a new revision at the same fingerprint is asked again under a
// new number) and an answer counts only if the router's warrant names the id of the ask now at the key (the warrant's first part,
// before the `@`). Each answer is another want value (the asking's number is in the want's words), so a backup that fails again
// after "back up again" and is answered "again" again changes the key and the node retries. That the backup failed again is seen
// in a new have value after the answer was written.
func (p *passer) backups(ctx context.Context) {
wanted := p.w.wantsOf()
for key, value := range p.w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
continue
}
have, err := delivery.DecodeHave(value)
if err != nil || have.Phase != delivery.PhaseHeld {
continue
}
want, ok := wanted[module][node]
if !ok || want.Removed {
continue
}
akey, err := delivery.BackupAskKey(node, module, want.FP)
if err != nil {
continue
}
current, there := delivery.DecodeAssignerAsk(akey, p.w.read(delivery.AssignerAsksBucket)[akey])
if there != nil || assign.BackupAskDigest(current) != want.Composed {
after := -1
if there == nil {
after = current.Asking
}
p.backupAsk(ctx, akey, node, module, want, have.Detail, assign.NextBackupAsking(after))
continue
}
answer, answered := p.w.answer(akey)
if router, _, _ := strings.Cut(answer.Warrant, "@"); !answered || router != current.Ask.ID ||
(answer.Option != delivery.BackupAgain && answer.Option != delivery.BackupSkip) {
continue // not answered yet: the ask stands
}
words := assign.BackupAnswerWords(answer.Option, current.Asking)
haveChanges := p.w.changed(delivery.HaveBucket, key)
if want.Backup != answer.Option || want.Words != words {
want.Backup, want.Words = answer.Option, words
if wkey, err := delivery.WantKey(node, module); err == nil {
w := want
p.put(ctx, delivery.AssignerWantsBucket, wkey, func() ([]byte, error) { return delivery.EncodeWant(w) })
p.answeredAt[current.Ask.ID] = haveChanges
}
continue
}
if answer.Option != delivery.BackupAgain {
continue
}
// The answer is in the want. The node retried and the backup failed again when the have took another value since.
if seen, known := p.answeredAt[current.Ask.ID]; !known {
p.answeredAt[current.Ask.ID] = haveChanges // after a restart: wait for the next failure
} else if haveChanges > seen {
p.backupAsk(ctx, akey, node, module, want, have.Detail, assign.NextBackupAsking(current.Asking))
}
}
}
// backupAsk writes the backup ask at its asking and raises the condition of the failed backup, once.
func (p *passer) backupAsk(ctx context.Context, key, node, module string, want delivery.Want, words string, asking int) {
a := assign.BackupAsk(key, node, module, want.FP, want.Composed, words, asking, time.Now())
p.put(ctx, delivery.AssignerAsksBucket, key, func() ([]byte, error) { return delivery.EncodeAssignerAsk(key, a) })
c := assign.BackupCondition(node, module, want.FP, words, true)
if p.say == nil || p.said[c.Key] {
return
}
if err := p.say(c); err != nil {
p.logf("the condition of the failed backup of %s on %s could not be said: %v", module, node, err)
return
}
p.said[c.Key] = true
}
@@ -0,0 +1,685 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// memKV is the part of a bucket a pass writes: puts and deletes.
type memKV struct {
jetstream.KeyValue
values map[string][]byte
}
func (m *memKV) Put(_ context.Context, key string, value []byte) (uint64, error) {
m.values[key] = value
return uint64(len(m.values)), nil
}
func (m *memKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.values, key)
return nil
}
// newPasser is a pass on a fresh world, as after a start, writing to the buckets given, at the clock given.
func newPasser(kvs buckets, at time.Time) *passer {
return &passer{kvs: kvs, w: newWorld(), logf: func(string, ...any) {}, said: map[string]bool{}, answeredAt: map[string]uint64{},
now: func() time.Time { return at }}
}
var fpTest = delivery.SourcePrefix2 + strings.Repeat("b", 64)
// 1906/H2: given a hold lifted by itself, when the assigner is started again and the same fingerprint is held again, then the
// hold ask is written under another asking, with an id of its own, so the teller raises it (the router drops a second ask under
// an id it keeps).
func TestAHoldAskedAgainAfterALiftAndARestartHasANewID(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
kvs := buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}
key, _ := delivery.HoldAskKey("web", fpTest)
held := &delivery.Held{Node: "ace", Words: "health failed"}
first := newPasser(kvs, time.Date(2026, 10, 25, 1, 0, 0, 0, time.UTC))
first.holdAsk(context.Background(), "web", fpTest, held)
a1, err := delivery.DecodeAssignerAsk(key, asksKV.values[key])
if err != nil {
t.Fatal(err)
}
first.lift(context.Background(), "web", fpTest)
if _, there := asksKV.values[key]; there {
t.Fatal("the lift left the hold ask")
}
again := newPasser(kvs, time.Date(2026, 10, 25, 2, 0, 0, 0, time.UTC)) // started again: it remembers nothing
again.holdAsk(context.Background(), "web", fpTest, held)
a2, err := delivery.DecodeAssignerAsk(key, asksKV.values[key])
if err != nil {
t.Fatal(err)
}
if a2.Ask.ID == a1.Ask.ID {
t.Errorf("the ask written again after a lift and a restart has the id %s of the first", a1.Ask.ID)
}
}
// 1906/H4: an answer counts only for the asking it answered: an answer to an earlier hold of the same fingerprint, or to an ask
// taken away, is not taken for the hold that stands now.
func TestAnAnswerCountsOnlyForTheAskingItAnswered(t *testing.T) {
key, _ := delivery.HoldAskKey("web", fpTest)
p := newPasser(buckets{}, time.Now())
put := func(asking int) string {
a := assign.HoldAsk(key, "web", fpTest, "ace", "health failed", asking, time.Now())
v, err := delivery.EncodeAssignerAsk(key, a)
if err != nil {
t.Fatal(err)
}
p.w.put(delivery.AssignerAsksBucket, key, v)
return a.Ask.ID
}
first := put(100)
v, _ := delivery.EncodeAnswer(delivery.Answer{Option: "release", Warrant: first + "@2026-10-25T01:00:00Z", Words: "the operator"})
p.w.put(delivery.AnswersBucket, key, v)
if !p.answeredWith(key, "release") {
t.Fatal("the answer to the ask that stands is not taken")
}
if p.answeredWith(key, "keep") {
t.Error("an answer was taken for an option not chosen")
}
put(200) // the same fingerprint held again, after a lift
if p.answeredWith(key, "release") {
t.Error("the answer to an earlier hold carried over to a later one")
}
p.w.put(delivery.AssignerAsksBucket, key, nil) // the ask taken away
if p.answeredWith(key, "release") {
t.Error("an answer to an ask taken away was taken")
}
t.Run("a put of the bus acts on the operator's warrant alone", putOfTheBus)
}
// memEntry is a value a memKV holds, as a bucket's Get answers it.
type memEntry struct {
jetstream.KeyValueEntry
value []byte
revision uint64
}
func (e memEntry) Value() []byte { return e.value }
func (e memEntry) Revision() uint64 { return e.revision }
func (m *memKV) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
v, ok := m.values[key]
if !ok {
return nil, jetstream.ErrKeyNotFound
}
return memEntry{value: v, revision: uint64(len(v))}, nil
}
func (m *memKV) Update(_ context.Context, key string, value []byte, revision uint64) (uint64, error) {
if uint64(len(m.values[key])) != revision {
return 0, errors.New("wrong last sequence")
}
m.values[key] = value
return uint64(len(value)), nil
}
// memStore is the runtime's states for the put's tools, over the same memKVs a pass writes.
type memStore map[string]*memKV
func (s memStore) get(state, key string) ([]byte, uint64, bool, error) {
v, ok := s[state].values[key]
return v, uint64(len(v)), ok, nil
}
func (s memStore) put(state, key string, value []byte) error {
s[state].values[key] = value
return nil
}
func (s memStore) keys(state string) ([]string, error) {
var out []string
for k := range s[state].values {
out = append(out, k)
}
return out, nil
}
// fakeBus is the assigner's process account in a test: every call recorded, each answered at once.
type fakeBus struct {
calls chan string
events []string
}
func (b *fakeBus) Call(key string, body any) (json.RawMessage, error) {
b.calls <- fmt.Sprintf("%s %v", key, body)
return json.RawMessage(`{"ok":true,"output":""}`), nil
}
func (b *fakeBus) CallOn(node, key string, body any) (json.RawMessage, error) {
return b.Call(node+"/"+key, body)
}
func (b *fakeBus) Emit(eventType string, body any) error {
b.events = append(b.events, fmt.Sprintf("%s %+v", eventType, body))
return nil
}
// ADR 0546, the first put (2026-10-25, want.novox.nats): the tool writes the put; a pass waits for the composition and grant
// the controller's take-in writes; once they are there it asks the operator with a yes that approves; only on that warrant does it have
// the streams snapshotted, then writes the want held and records the warrant; the have running it healthy holds it; a release
// gives the want back to the rules. Each step is said as the event put.
func putOfTheBus(t *testing.T) {
ctx := context.Background()
now := time.Date(2026, 10, 25, 22, 0, 0, 0, time.UTC)
fp := delivery.SourcePrefix2 + "fd497ab8" + strings.Repeat("0", 56)
digest := "sha256:" + strings.Repeat("d", 64)
mem := func() *memKV { return &memKV{values: map[string][]byte{}} }
puts, asksKV, wantsKV := mem(), mem(), mem()
kvs := buckets{delivery.AssignerPutsBucket: puts, delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}
p := newPasser(kvs, now)
bus := &fakeBus{calls: make(chan string, 8)}
woke := make(chan struct{}, 8) // each act's goroutine wakes a pass once it is done
p.acts = newPutActs(bus, func() { woke <- struct{}{} }, func(string, ...any) {})
p.acts.snapshot = func(context.Context, string, string) (string, error) {
return "novox's restore point of nats (streams) taken 22:01", nil
}
// The world: nats assigned to novox, built at fp, the bus by its manifest, novox running it from before the switch.
world := func(bucket, key string, v any) {
var data []byte
switch x := v.(type) {
case []byte:
data = x
default:
data, _ = json.Marshal(x)
}
p.w.put(bucket, key, data)
}
world(provider.AssignmentsBucket, "novox.nats", map[string]string{"node": "novox", "module": "nats"})
mdigest := strings.Repeat("e", 64)
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fp, Manifest: mdigest, Repository: "novox/mesh-catalog", Commit: strings.Repeat("c", 40)})
world(delivery.ModulesBucket, "nats", mod)
p.w.manifests[mdigest] = parsed{module: "nats", manifest: catalogue.Manifest{Module: "nats", Provides: []catalogue.Offer{{Name: "mesh-bus"}}}}
bkey, _ := delivery.BuilderBuildKey("nats", fp)
built, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: mdigest})
world(delivery.BuilderBuildsBucket, bkey, built)
hkey, _ := delivery.HaveKey("novox", "nats")
before, _ := delivery.EncodeHave(delivery.Have{FP: delivery.SourcePrefix2 + strings.Repeat("a", 64), Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
world(delivery.HaveBucket, hkey, before)
builds := mem()
builds.values[bkey] = built
modules := mem()
modules.values["nats"] = mod
s := newPutTools(memStore{putsState: puts, delivery.BuilderModule + ".builds": builds, delivery.IndexerModule + ".modules": modules})
s.now = func() time.Time { return now }
was := answerWithin
answerWithin = 0
defer func() { answerWithin = was }()
if _, err := s.doPut(map[string]any{"node": "novox", "module": "nats", "version": "fd497ab8ffff", "why": "x"}, "shanks.node-tools"); err == nil {
t.Error("a prefix of a fingerprint the builder never built was taken")
}
if got, err := s.version("nats", "fd497ab80000"); err != nil || got != fp {
t.Errorf("the prefix of the fingerprint built reads %q, %v", got, err)
}
if got, err := s.version("nats", "newest"); err != nil || got != fp {
t.Errorf("newest reads %q, %v", got, err)
}
ans, err := s.doPut(map[string]any{"node": "novox", "module": "nats", "version": fp, "hold": "true", "why": "the bus's user list"}, "shanks.node-tools")
if err != nil {
t.Fatal(err)
}
if a := ans.(putAnswer); a.Step != delivery.PutAsked || !a.Hold || !strings.Contains(a.Follow, "mesh_wait") {
t.Fatalf("put answered %+v", a)
}
pkey, _ := delivery.PutKey("novox", "nats")
step := func() delivery.Put {
t.Helper()
world(delivery.AssignerPutsBucket, pkey, puts.values[pkey])
p.pass(ctx)
got, err := delivery.DecodePut(puts.values[pkey])
if err != nil {
t.Fatal(err)
}
return got
}
if got := step(); got.Step != delivery.PutCompose || !strings.Contains(got.Words, "no composition") {
t.Fatalf("the first pass left the put at %s: %s", got.Step, got.Words)
}
select {
case call := <-bus.calls:
t.Errorf("a put asked %q: it reads the compositions and asks nobody for one", call)
default:
}
ckey, _ := delivery.ComposedKey("novox", "nats", fp)
cd, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: digest, Judgement: delivery.JudgedCannot, Words: "the have names no composition"})
world(delivery.ControllerComposedBucket, ckey, cd)
gkey, _ := delivery.ControllerGrantKey("nats", fp)
g, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{})
world(delivery.ControllerGrantsBucket, gkey, g)
got := step()
akey, _ := delivery.RecordAskKey("novox", "nats", digest)
if got.Step != delivery.PutPerson || got.Ask != akey {
t.Fatalf("composed, the put stands at %s (%s), ask %q", got.Step, got.Words, got.Ask)
}
a, err := delivery.DecodeAssignerAsk(akey, asksKV.values[akey])
if err != nil || a.Ask.Options[0].Level != asks.Approve {
t.Fatalf("the ask is %+v, %v; want a yes that approves", a, err)
}
if _, there := wantsKV.values["want.novox.nats"]; there {
t.Fatal("the bus's want was written before the operator's warrant")
}
world(delivery.AssignerAsksBucket, akey, asksKV.values[akey])
yes, _ := delivery.EncodeAnswer(delivery.Answer{Option: assign.OptionYes, Warrant: a.Ask.ID + "@2026-10-25T22:01:00Z", Words: "the operator, verified"})
world(delivery.AnswersBucket, akey, yes)
if got := step(); got.Step != delivery.PutSnapshot {
t.Fatalf("on the warrant the put stands at %s: %s", got.Step, got.Words)
}
<-woke
got = step()
w, err := delivery.DecodeWant(wantsKV.values["want.novox.nats"])
if got.Step != delivery.PutWanted || err != nil || w.FP != fp || w.Composed != digest || !w.Put || !w.Hold {
t.Fatalf("snapshotted, the put stands at %s (%s); the want %+v, %v", got.Step, got.Words, w, err)
}
if call := <-bus.calls; !strings.Contains(call, "warranted") || !strings.Contains(call, a.Ask.ID) {
t.Errorf("the warrant was recorded as %q", call)
}
world(delivery.AssignerWantsBucket, "want.novox.nats", wantsKV.values["want.novox.nats"])
running, _ := delivery.EncodeHave(delivery.Have{FP: fp, Composed: digest, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
world(delivery.HaveBucket, hkey, running)
if got := step(); got.Step != delivery.PutHolding {
t.Fatalf("running healthy, the put stands at %s: %s", got.Step, got.Words)
}
if _, err := s.doRelease(map[string]any{"node": "novox", "module": "nats", "why": "done"}, "shanks.node-tools"); err != nil {
t.Fatal(err)
}
if got := step(); got.Step != delivery.PutReleased {
t.Fatalf("released, the put stands at %s", got.Step)
}
listed, _ := s.doPuts(map[string]any{"module": "nats"})
if l := listed.(map[string]any)["puts"].([]putAnswer); len(l) != 1 || l[0].Step != delivery.PutReleased || len(l[0].Steps) < 6 {
t.Errorf("puts lists %+v", l)
}
if len(bus.events) < 5 || !strings.Contains(strings.Join(bus.events, "\n"), "holding") {
t.Errorf("the steps said as events: %v", bus.events)
}
// The same bus put again at the same composition after the release is its own asking: its ask is written again under an
// id of its own, and the earlier yes, still at the answers' key, does not count for it (ADR 0546 clause 8).
world(delivery.HaveBucket, hkey, before)
s.now = func() time.Time { return now.Add(time.Minute) }
if _, err := s.doPut(map[string]any{"node": "novox", "module": "nats", "version": fp, "hold": "true", "why": "again"}, "shanks.node-tools"); err != nil {
t.Fatal(err)
}
again := step()
a2, err := delivery.DecodeAssignerAsk(akey, asksKV.values[akey])
if again.Step != delivery.PutPerson || err != nil || a2.Ask.ID == a.Ask.ID {
t.Fatalf("put again, it stands at %s (%s); its ask %q, %v; the first ask was %q", again.Step, again.Words, a2.Ask.ID, err, a.Ask.ID)
}
world(delivery.AssignerAsksBucket, akey, asksKV.values[akey])
yes2, _ := delivery.EncodeAnswer(delivery.Answer{Option: assign.OptionYes, Warrant: a2.Ask.ID + "@2026-10-25T22:03:00Z", Words: "the operator, verified"})
world(delivery.AnswersBucket, akey, yes2)
if got := step(); got.Step != delivery.PutSnapshot {
t.Fatalf("on the second warrant the put stands at %s: %s", got.Step, got.Words)
}
<-woke
// Released between the pass's read and its write: the pass writes no want for the put that no longer stands.
oldWant, _ := delivery.EncodeWant(delivery.Want{FP: delivery.SourcePrefix2 + strings.Repeat("a", 64)})
wantsKV.values["want.novox.nats"] = oldWant
world(delivery.AssignerWantsBucket, "want.novox.nats", oldWant)
if _, err := s.doRelease(map[string]any{"node": "novox", "module": "nats", "why": "changed my mind"}, "shanks.node-tools"); err != nil {
t.Fatal(err)
}
p.pass(ctx)
if string(wantsKV.values["want.novox.nats"]) != string(oldWant) {
t.Errorf("a want was written for a put released meanwhile: %s", wantsKV.values["want.novox.nats"])
}
}
// 1945/AC21: given a module on neither list on a node, when passes follow, then the assigner raises its condition once, says it
// again after a say that failed, and clears it once when the module has a want again; a module on neither list is also written to
// the journal, so it is loud with no operator channel.
func TestAModuleOnNeitherListIsRaisedOnceAndClearedOnce(t *testing.T) {
p := newPasser(buckets{}, time.Date(2026, 10, 25, 15, 0, 0, 0, time.UTC))
var said []asks.Condition
failing := true
p.say = func(c asks.Condition) error {
if failing {
failing = false
return errors.New("the channel is away")
}
said = append(said, c)
return nil
}
var logged []string
p.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
one := map[string]neitherOne{"g14.screen-lock": {node: "g14", module: "screen-lock",
holding: assign.Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}}}
p.sayNeither(one, nil) // the say fails
p.sayNeither(one, nil)
p.sayNeither(one, nil)
if len(said) != 1 || said[0].State != asks.StateRaised || said[0].Subject.Machine != "g14" || said[0].Subject.ID != "screen-lock" {
t.Fatalf("said = %+v, want one raised condition of screen-lock on g14", said)
}
if !strings.Contains(strings.Join(logged, "\n"), "screen-lock") {
t.Errorf("the journal does not name the module on neither list: %q", logged)
}
p.sayNeither(map[string]neitherOne{}, nil)
p.sayNeither(map[string]neitherOne{}, nil)
if len(said) != 2 || said[1].State != asks.StateCleared || said[1].Key != said[0].Key {
t.Errorf("said = %+v, want the condition cleared once under its key", said)
}
}
// 1945/AC21: the pass gathers the modules on neither list of every module's outcome, keyed by node and module.
func TestThePassGathersTheModulesOnNeitherList(t *testing.T) {
got := map[string]neitherOne{}
gatherNeither(got, "screen-lock", assign.Outcome{Neither: map[string]assign.Holding{"g14": {Rule: delivery.RuleGrant, Words: "w"}}})
gatherNeither(got, "rofi", assign.Outcome{})
if len(got) != 1 || got["g14.screen-lock"].node != "g14" || got["g14.screen-lock"].module != "screen-lock" || got["g14.screen-lock"].holding.Rule != delivery.RuleGrant {
t.Errorf("gathered = %+v", got)
}
// 1945 (a first want held back silently): the journal says why a node gets no want when the reason appears or changes, and
// when it has a want again: one line per change, not per pass.
p := newPasser(buckets{}, time.Date(2026, 10, 25, 21, 30, 0, 0, time.UTC))
var logged []string
p.logf = func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) }
notDone := assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Rule: delivery.RuleBuildDone, Words: "the build of rofi at 4e7b118e is not done"}}}
wanted := assign.Outcome{Wants: map[string]delivery.Want{"shanks": {FP: fpTest, Composed: "sha256:" + strings.Repeat("c", 64)}}}
for i, c := range []struct {
before map[string]delivery.Want
out assign.Outcome
line string // the line said in this pass, "" for none
}{
{nil, notDone, "rofi on shanks gets no want: rule 1: the build of rofi at 4e7b118e is not done"},
{nil, notDone, ""},
{map[string]delivery.Want{"shanks": {Removed: true}}, notDone, ""},
{nil, assign.Outcome{Rules: map[string]assign.Holding{"shanks": {Rule: delivery.RuleGrant, Words: "no credential"}}}, "rofi on shanks gets no want: rule 6: no credential"},
{nil, wanted, "rofi on shanks has a want"},
{map[string]delivery.Want{"shanks": wanted.Wants["shanks"]}, assign.Outcome{}, ""},
} {
logged = nil
p.sayNoWant("rofi", []string{"shanks"}, c.before, c.out)
if got := strings.Join(logged, "\n"); got != c.line {
t.Errorf("pass %d said %q, want %q", i, got, c.line)
}
}
}
// A neither or held condition a previous process raised is cleared by the first pass after a start: a fresh pass remembers
// nothing it said, so it clears, once, the neither condition of every assigned module not on neither list and the held condition of
// every module that is not held. Clearing a condition that is not open is quiet at the router.
func TestTheFirstPassAfterAStartClearsWhatAPreviousProcessRaised(t *testing.T) {
wantsKV := &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerWantsBucket: wantsKV, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
var said []asks.Condition
p.say = func(c asks.Condition) error { said = append(said, c); return nil }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
p.w.put(delivery.ModulesBucket, "web", mod)
p.w.put(provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"b-1"}`))
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
p.w.put(delivery.PresenceBucket, "g14", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpTest, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
p.w.put(delivery.HaveBucket, "g14.web", have)
key, _ := delivery.WantKey("g14", "web")
want, _ := delivery.EncodeWant(delivery.Want{FP: fpTest, Composed: "sha256:" + strings.Repeat("c", 64)})
p.w.put(delivery.AssignerWantsBucket, key, want)
p.wants(context.Background())
cleared := map[string]bool{}
for _, c := range said {
if c.State != asks.StateCleared {
t.Errorf("the first pass raised %+v", c)
}
cleared[c.Key] = true
}
neither := assign.NeitherCondition("g14", "web", assign.Holding{}, false).Key
held := assign.HeldCondition("web", fpTest, "", "", false, false).Key
if !cleared[neither] || !cleared[held] {
t.Errorf("cleared = %v, want %s and %s", cleared, neither, held)
}
n := len(said)
p.wants(context.Background())
if len(said) != n {
t.Errorf("a second pass said again: %+v", said[n:])
}
}
// 1945/AC24: given a module held at F under rule 2 on a node that runs it with no want, when the assigner passes, then it raises no
// neither condition for it, a neither condition this process raised for it before is cleared, and so is one a previous process
// raised (the first pass after a start), each once.
func TestAHeldModuleRaisesNoNeitherAndClearsAStandingOne(t *testing.T) {
setup := func() (*passer, *[]asks.Condition) {
p := newPasser(buckets{delivery.AssignerWantsBucket: &memKV{values: map[string][]byte{}}, delivery.AssignerAsksBucket: &memKV{values: map[string][]byte{}}}, time.Now())
said := &[]asks.Condition{}
p.say = func(c asks.Condition) error { *said = append(*said, c); return nil }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/resolved", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
p.w.put(delivery.ModulesBucket, "systemd-resolved", mod)
p.w.put(provider.AssignmentsBucket, "shanks.systemd-resolved", []byte(`{"node":"shanks","module":"systemd-resolved","build":"b-1"}`))
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
p.w.put(delivery.PresenceBucket, "shanks", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpTest, Phase: delivery.PhaseRunning, Health: delivery.HealthFailed})
p.w.put(delivery.HaveBucket, "shanks.systemd-resolved", have)
hkey, _ := delivery.HeldKey("systemd-resolved", fpTest)
held, _ := delivery.EncodeHeld(delivery.Held{Node: "shanks", Words: "health failed"})
p.w.put(delivery.AssignerWantsBucket, hkey, held)
return p, said
}
key := assign.NeitherCondition("shanks", "systemd-resolved", assign.Holding{}, false).Key
neithers := func(said []asks.Condition) (raised, cleared int) {
for _, c := range said {
if c.Key != key {
continue
}
if c.State == asks.StateRaised {
raised++
} else {
cleared++
}
}
return
}
// a standing neither this process raised
p, said := setup()
p.neither = map[string]neitherOne{"shanks.systemd-resolved": {node: "shanks", module: "systemd-resolved", holding: assign.Holding{Rule: delivery.RuleNotHeld}}}
p.wants(context.Background())
p.wants(context.Background())
if r, c := neithers(*said); r != 0 || c != 1 {
t.Errorf("standing in this process: %d raised, %d cleared, want 0 and 1: %+v", r, c, *said)
}
// one a previous process raised: the first pass after a start clears it
p, said = setup()
p.wants(context.Background())
p.wants(context.Background())
if r, c := neithers(*said); r != 0 || c != 1 {
t.Errorf("after a start: %d raised, %d cleared, want 0 and 1: %+v", r, c, *said)
}
}
// 1906 note 28: an ask outlives its fingerprint. Each pass takes away the hold, restore and record asks of a module whose
// fingerprint (or composition) is not the trunk's now, or whose module left the trunk; a trunk not read takes nothing away; a
// backup ask is left; and a lift takes the restore asks of its fingerprint on every node.
func TestAsksOfAFingerprintTheTrunkLeftBehindAreTakenAway(t *testing.T) {
fpOld, fpNew := delivery.SourcePrefix2+strings.Repeat("a", 64), delivery.SourcePrefix2+strings.Repeat("c", 64)
digOld, digNew := "sha256:"+strings.Repeat("d", 64), "sha256:"+strings.Repeat("e", 64)
trunkOf := func(fp string) map[string]delivery.Module {
return map[string]delivery.Module{"web": {FP2: fp, Manifest: "sha256:" + strings.Repeat("1", 64), Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath}}
}
type ask struct{ name, key string }
holdOld, _ := delivery.HoldAskKey("web", fpOld)
holdNew, _ := delivery.HoldAskKey("web", fpNew)
holdGone, _ := delivery.HoldAskKey("gone", fpOld)
restOld, _ := delivery.RestoreAskKey("novox", "web", fpOld)
restNew, _ := delivery.RestoreAskKey("novox", "web", fpNew)
recOld, _ := delivery.RecordAskKey("novox", "web", digOld)
recNew, _ := delivery.RecordAskKey("novox", "web", digNew)
backOld, _ := delivery.BackupAskKey("novox", "web", fpOld)
all := []string{holdOld, holdNew, holdGone, restOld, restNew, recOld, recNew, backOld}
for _, c := range []struct {
name string
trunk map[string]delivery.Module
gone []string // the keys taken away
noDigit bool // the controller's composition is not read
heldAt string // the node's want is at this fingerprint, whose composition is digOld
}{
{"the trunk moved on", trunkOf(fpNew), []string{holdOld, holdGone, restOld, recOld}, false, ""},
{"the trunk is at the old fingerprint", trunkOf(fpOld), []string{holdNew, holdGone, restNew, recOld}, false, ""},
{"the trunk's fingerprint is not read yet", trunkOf(""), []string{holdGone}, false, ""},
{"the composition is not read yet", trunkOf(fpNew), []string{holdOld, holdGone, restOld}, true, ""},
{"a node held back at the old fingerprint is still asked about its composition", trunkOf(fpNew), []string{holdOld, holdGone, restOld}, false, fpOld},
{"the trunk is not read at all", nil, nil, false, ""},
} {
t.Run(c.name, func(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
var said []asks.Condition
p.say = func(c asks.Condition) error { said = append(said, c); return nil }
for _, key := range all {
k, _ := delivery.ParseAnyAskKey(key)
var a delivery.AssignerAsk
switch k.Kind {
case delivery.AskKindHold:
a = assign.HoldAsk(key, k.Module, k.FP, "novox", "w", 1, time.Now())
case delivery.AskKindRestore:
a = assign.RestoreAsk(key, k.Node, k.Module, k.FP, "w", time.Now())
case delivery.AskKindRecord:
a = assign.RecordAsk(key, k.Node, k.Module, fpNew, assign.RecordNeed{Node: k.Node, Digest: k.Composed}, time.Now())
default:
a = assign.BackupAsk(key, k.Node, k.Module, k.FP, digOld, "w", 2, time.Now())
}
v, err := delivery.EncodeAssignerAsk(key, a)
if err != nil {
t.Fatal(err)
}
asksKV.values[key] = v
p.w.put(delivery.AssignerAsksBucket, key, v)
}
if !c.noDigit {
ckey, _ := delivery.ComposedKey("novox", "web", fpNew)
v, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: digNew, Judgement: delivery.JudgedUnchanged})
p.w.put(delivery.ControllerComposedBucket, ckey, v)
okey, _ := delivery.ComposedKey("novox", "web", fpOld)
old, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: digNew, Judgement: delivery.JudgedUnchanged})
if c.heldAt != "" {
old, _ = delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: digOld, Judgement: delivery.JudgedUnchanged})
}
p.w.put(delivery.ControllerComposedBucket, okey, old)
}
if c.heldAt != "" {
wkey, _ := delivery.WantKey("novox", "web")
w, _ := delivery.EncodeWant(delivery.Want{FP: c.heldAt, Composed: digOld})
p.w.put(delivery.AssignerWantsBucket, wkey, w)
}
heldOld, _ := delivery.HeldKey("web", fpOld)
wantsKV.values[heldOld] = []byte("x")
p.w.put(delivery.AssignerWantsBucket, heldOld, []byte("x"))
firstAsk, _ := delivery.DecodeAssignerAsk(holdOld, asksKV.values[holdOld])
p.staleAsks(context.Background(), c.trunk)
// the held record and condition of a fingerprint whose hold ask went are taken away with it
heldOld, _ = delivery.HeldKey("web", fpOld)
_, heldThere := wantsKV.values[heldOld]
cleared := slices.ContainsFunc(said, func(c asks.Condition) bool {
return c.Key == assign.HeldCondition("web", fpOld, "", "", false, false).Key && c.State == asks.StateCleared
})
if gone := slices.Contains(c.gone, holdOld); !heldThere || cleared != gone {
t.Errorf("hold ask of the old fingerprint gone = %v: its held record stays = %v (must), its condition cleared = %v", gone, heldThere, cleared)
}
for _, key := range all {
_, there := asksKV.values[key]
want := !slices.Contains(c.gone, key)
if there != want {
t.Errorf("%s: left in the bucket = %v, want %v", key, there, want)
}
}
if slices.Contains(c.gone, holdOld) {
// the trunk returns to the fingerprint (a revert): the held record still holds it, and it is asked and said again
p.now = func() time.Time { return time.Now().Add(time.Hour) }
p.holdAsk(context.Background(), "web", fpOld, nil)
p.heldCondition(context.Background(), "web", fpOld, nil, false)
again, err := delivery.DecodeAssignerAsk(holdOld, asksKV.values[holdOld])
raised := slices.ContainsFunc(said, func(c asks.Condition) bool { return c.State == asks.StateRaised })
if err != nil || again.Ask.ID == firstAsk.Ask.ID || !raised {
t.Errorf("the trunk returned to the held fingerprint: ask %q (first %q, %v), condition raised again = %v", again.Ask.ID, firstAsk.Ask.ID, err, raised)
}
}
})
}
t.Run("a lift takes the restore asks of its fingerprint on every node", func(t *testing.T) {
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
keep, _ := delivery.RestoreAskKey("ace", "web", fpNew)
keepOther, _ := delivery.RestoreAskKey("ace", "other", fpOld)
for _, key := range []string{restOld, mustKey(delivery.RestoreAskKey("ace", "web", fpOld)), keep, keepOther} {
asksKV.values[key] = []byte("x")
p.w.put(delivery.AssignerAsksBucket, key, []byte("x"))
}
p.lift(context.Background(), "web", fpOld)
for key, gone := range map[string]bool{restOld: true, mustKey(delivery.RestoreAskKey("ace", "web", fpOld)): true, keep: false, keepOther: false} {
if _, there := asksKV.values[key]; there == gone {
t.Errorf("%s: left in the bucket = %v after the lift", key, there)
}
}
})
}
func mustKey(key string, err error) string {
if err != nil {
panic(err)
}
return key
}
// 1906 note 28, review: a node held back at an earlier fingerprint is asked about the revised composition of that fingerprint
// (ADR 0545). That record ask is not about the trunk's fingerprint: it survives every pass, so the operator is asked, and on a
// yes the want names the revised digest.
func TestARecordAskOfAHeldBackNodeSurvivesPassesAndItsYesMovesTheWant(t *testing.T) {
fpOld := delivery.SourcePrefix2 + strings.Repeat("a", 64)
revised, was := "sha256:"+strings.Repeat("e", 64), "sha256:"+strings.Repeat("d", 64)
asksKV, wantsKV := &memKV{values: map[string][]byte{}}, &memKV{values: map[string][]byte{}}
p := newPasser(buckets{delivery.AssignerAsksBucket: asksKV, delivery.AssignerWantsBucket: wantsKV}, time.Now())
p.logf = t.Logf
put := func(bucket, key string, v []byte) { p.w.put(bucket, key, v) }
mod, _ := delivery.EncodeModule(delivery.Module{FP2: fpTest, Repository: "novox/web", Commit: strings.Repeat("1", 40), Path: delivery.RootPath})
put(delivery.ModulesBucket, "web", mod) // the trunk's build is not done: rule 1 holds g14 back at fpOld
put(provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"b-1"}`))
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
put(delivery.PresenceBucket, "g14", on)
have, _ := delivery.EncodeHave(delivery.Have{FP: fpOld, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
put(delivery.HaveBucket, "g14.web", have)
wkey, _ := delivery.WantKey("g14", "web")
want, _ := delivery.EncodeWant(delivery.Want{FP: fpOld, Composed: was})
wantsKV.values[wkey] = want
put(delivery.AssignerWantsBucket, wkey, want)
ckey, _ := delivery.ComposedKey("g14", "web", fpOld)
cd, cerr := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: revised, Judgement: delivery.JudgedRecreates, Policy: delivery.PolicyRecord, PolicyFrom: "default", Against: was, Words: "recreates"})
if cerr != nil {
t.Fatal(cerr)
}
put(delivery.ControllerComposedBucket, ckey, cd)
tkey, _ := delivery.ComposedKey("g14", "web", fpTest) // the trunk's own composition, another digest than the revised one
tcd, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("f", 64), Judgement: delivery.JudgedUnchanged})
put(delivery.ControllerComposedBucket, tkey, tcd)
akey, _ := delivery.RecordAskKey("g14", "web", revised)
for i := 0; i < 2; i++ {
p.wants(context.Background())
if _, there := asksKV.values[akey]; !there {
t.Fatalf("pass %d: the record ask of the held-back node is not there", i+1)
}
}
a, err := delivery.DecodeAssignerAsk(akey, asksKV.values[akey])
if err != nil {
t.Fatal(err)
}
yes, _ := delivery.EncodeAnswer(delivery.Answer{Option: assign.OptionYes, Warrant: a.Ask.ID + "@2026-10-26T01:00:00Z", Words: "the operator"})
put(delivery.AnswersBucket, akey, yes)
p.wants(context.Background())
got, err := delivery.DecodeWant(wantsKV.values[wkey])
if err != nil || got.FP != fpOld || got.Composed != revised {
t.Errorf("after the yes the want is %+v (%v), want it at %s naming %s", got, err, fpOld, revised)
}
if _, there := asksKV.values[akey]; !there {
t.Error("the answered record ask was taken away")
}
}
@@ -0,0 +1,449 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// The puts (novox/hq ADR 0546): the tool `assigner_put` writes the key `<node>.<module>` of the assigner's puts, and each pass
// takes every put that has not ended one step further (assign.StepPut), doing the acts the step asks for: the put's ask, the
// bus's snapshot, the warranted record. Each act's outcome is heard back as a change of the world (the composition, the grant,
// the answer) or, for the calls, as a wake once they answer; nothing polls but the snapshot's read-back of the backup holder,
// as the controller's bus step reads it (ADR 0235), bounded. Every step is said as the event mesh-delivery-assigner.put, so a
// caller waits on it.
// PutEvent is the event a put's every step is said as: mesh.mod.mesh-delivery-assigner.event.put.
const PutEvent = "put"
// caller is what the puts need of the bus as the assigner's own process account: a call of a seat's verb, on one node or on
// the seat held once, and an event said. process.Conn, or a test's.
type caller interface {
Call(key string, body any) (json.RawMessage, error)
CallOn(node, key string, body any) (json.RawMessage, error)
Emit(eventType string, body any) error
}
// putActs keeps, by put (its key and when it was asked), what the acts of a put came to while this process runs: a restart
// takes the snapshot again, without harm.
type putActs struct {
mu sync.Mutex
bus caller
wake func()
logf func(string, ...any)
snapping map[string]bool
snapped map[string]string
snapNo map[string]string
warranted map[string]bool
warrantNo map[string]string
said map[string]string
// snapshot takes the bus's snapshot on node; a variable so a test takes none.
snapshot func(ctx context.Context, node, module string) (string, error)
}
func newPutActs(bus caller, wake func(), logf func(string, ...any)) *putActs {
a := &putActs{bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
said: map[string]string{}}
a.snapshot = a.snapshotTheBus
return a
}
// identity names one put: its key and when it was asked, so a put asked again under the key is another.
func identity(key string, p delivery.Put) string {
return key + "@" + p.At.UTC().Format(time.RFC3339Nano)
}
// putsByModule is every put the assigner holds, by module then node.
func (p *passer) putsByModule() map[string]map[string]delivery.Put {
out := map[string]map[string]delivery.Put{}
for key, value := range p.w.read(delivery.AssignerPutsBucket) {
node, module, err := delivery.ParsePutKey(key)
if err != nil {
continue
}
put, err := delivery.DecodePut(value)
if err != nil {
p.logf("the put %s cannot be read, and is left: %v", key, err)
continue
}
if out[module] == nil {
out[module] = map[string]delivery.Put{}
}
out[module][node] = put
}
return out
}
// puts takes every put that has not ended one step further.
func (p *passer) puts(ctx context.Context) {
if p.acts == nil {
return
}
all := p.putsByModule()
if len(all) == 0 {
return
}
assigned, _ := p.w.assigned()
trunk := p.w.trunk()
get := kvGetter(p.kvs[delivery.ManifestsBucket])
wanted := p.w.wantsOf()
modules := make([]string, 0, len(all))
for m := range all {
modules = append(modules, m)
}
sort.Strings(modules)
for _, module := range modules {
var manifest catalogue.Manifest
if mod, ok := trunk[module]; ok && mod.Manifest != "" {
if got := p.w.manifest(ctx, get, mod.Manifest); got.err == "" {
manifest = got.manifest
}
}
for node, put := range all[module] {
if delivery.PutEnded(put.Step) {
continue
}
key, _ := delivery.PutKey(node, module)
id := identity(key, put)
world := p.putWorld(node, module, put, manifest, assigned[node])
world.Snapshot, world.Snapshotting, world.SnapshotFailed = p.acts.state(id)
next := assign.StepPut(world)
// A held bus whose user list the rules moved on (ADR 0539 §7): the put follows the composition its want names.
if w, ok := wanted[module][node]; ok && w.Put && w.FP == next.Put.FP && w.Composed != "" && w.Composed != next.Put.Composed &&
(next.Put.Step == delivery.PutWanted || next.Put.Step == delivery.PutHolding) {
if d, _ := p.composedFor(node, module, w.FP); d != nil && d.UserListOnly && d.Digest == w.Composed {
next.Put = next.Put.Said(world.Now, next.Put.Step, fmt.Sprintf("the bus's user list changed: the want names composition %s", shortHex(w.Composed)))
next.Put.Composed = w.Composed
}
}
// The bus's want is written only for the act the operator's yes bound (ADR 0546 clause 8).
if next.Put.Step == delivery.PutWanted && put.Step != delivery.PutWanted && world.Bus && next.Put.Ask != "" {
if err := p.performs(next.Put.Ask, assign.PutAct(node, module, next.Put.FP, next.Put.Composed)); err != nil {
next = assign.PutNext{Put: next.Put.Said(world.Now, delivery.PutRefused, "the bus's want is not written: "+err.Error())}
}
}
if failed := p.acts.warrantFailed(id); failed != "" && (next.Put.Step == delivery.PutWanted || next.Put.Step == delivery.PutHolding) &&
!strings.Contains(next.Put.Words, failed) {
next.Put = next.Put.Said(world.Now, next.Put.Step, next.Put.Words+"; "+failed)
}
p.actOn(ctx, node, module, id, world, next)
written := samePut(put, next.Put) || p.writePut(ctx, key, put, next.Put)
if !written {
continue // released or put again meanwhile: nothing is done for the put that no longer stands
}
if next.Put.Step == delivery.PutWanted && put.Step != delivery.PutWanted {
// The want is written in this pass too, before the next one would: Module.Puts writes the same.
if wkey, err := delivery.WantKey(node, module); err == nil {
want := assign.PutWant(next.Put)
if before, ok := wanted[module][node]; !ok || !delivery.SameWant(normalWant(before), normalWant(want)) {
p.put(ctx, delivery.AssignerWantsBucket, wkey, func() ([]byte, error) { return delivery.EncodeWant(want) })
}
}
if next.Warranted {
p.acts.warrant(id, delivery.AskingID(next.Put.Ask, assign.PutAsking(next.Put)))
}
}
}
}
}
func normalWant(w delivery.Want) delivery.Want {
w.V = 0
return w
}
// putWorld is what one step of a put reads from the world.
func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalogue.Manifest, assignedHere []string) assign.PutWorld {
w := assign.PutWorld{Node: node, Module: module, Put: put, Now: p.clock(), Bus: catalogue.ProvidesBus(manifest), Newest: p.w.trunk()[module].FP2}
for _, m := range assignedHere {
if m == module {
w.Assigned = true
}
}
if bkey, err := delivery.BuilderBuildKey(module, put.FP); err == nil {
if raw := p.w.value(delivery.BuilderBuildsBucket, bkey); raw != nil {
if b, err := delivery.DecodeBuilderBuild(raw); err == nil {
w.Build, w.BuildWords = b.State, b.Error
} else {
w.Build = "unreadable: " + err.Error()
}
}
}
w.Composed, w.Granted = p.composedFor(node, module, put.FP)
if d := w.Composed; d != nil {
policy := d.Policy
if policy == "" {
policy, _, _ = catalogue.DerivedUpgrade(manifest)
}
w.Recorded = policy == catalogue.PolicyRecord
if akey, err := delivery.RecordAskKey(node, module, d.Digest); err == nil && put.Ask == akey {
// Only this put's own ask counts: an ask (and its answer) of an earlier put under the same key is another asking.
if a, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey)); err == nil && a.Asking == assign.PutAsking(put) {
switch {
case p.answeredWith(akey, assign.OptionYes):
w.Answer = assign.OptionYes
case p.answeredWith(akey, assign.OptionNo):
w.Answer = assign.OptionNo
default:
w.AskExpired = !a.Ask.Expires.IsZero() && w.Now.After(a.Ask.Expires)
}
}
}
}
if hkey, err := delivery.HaveKey(node, module); err == nil {
if raw := p.w.value(delivery.HaveBucket, hkey); raw != nil {
if h, err := delivery.DecodeHave(raw); err == nil {
w.Have = &h
}
}
}
return w
}
// composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one
// place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and
// mesh-controller_grants); when composing and grants move out of the controller into delivery's own parts, this reads them
// from their new owner and nothing else changes.
func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) {
var out *delivery.ComposedDigest
if d, ok := p.w.composed(node, module, fp); ok {
out = &d
}
granted := false
if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil {
if raw := p.w.value(delivery.ControllerGrantsBucket, gkey); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
granted = g.NamesNode(node, module)
}
}
}
return out, granted
}
// actOn does the acts a step asks for.
func (p *passer) actOn(ctx context.Context, node, module, id string, w assign.PutWorld, next assign.PutNext) {
if next.Put.Step == delivery.PutPerson && next.Put.Ask != "" && w.Composed != nil {
// The put's own ask, written again over one an earlier put left under the key (its asking differs), so the teller
// raises it anew under its own id.
akey := next.Put.Ask
current, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey))
if err != nil || current.Asking != assign.PutAsking(next.Put) {
a, err := assign.PutAsk(akey, node, module, next.Put, *w.Composed, w.Bus, w.Now)
if err != nil {
p.logf("the ask about putting %s on %s is not written: %v", module, node, err)
} else {
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
}
}
}
if next.Snapshot {
p.acts.snap(ctx, id, node, module)
}
}
// performs checks that the act about to be performed is the one the yes of the ask at key bound (asks.Option.Performs).
func (p *passer) performs(key string, act asks.Act) error {
a, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return fmt.Errorf("the ask %s cannot be read: %w", key, err)
}
for _, o := range a.Ask.Options {
if o.ID == assign.OptionYes {
return o.Performs(act)
}
}
return fmt.Errorf("the ask %s offers no yes", key)
}
// samePut says two puts say the same: their step, words and what they found.
func samePut(a, b delivery.Put) bool {
return a.Step == b.Step && a.Words == b.Words && a.Composed == b.Composed && a.Ask == b.Ask && a.Snapshot == b.Snapshot
}
// writePut writes a put's next value over the one the pass read, by compare-and-set, unless the tool has put again or
// released it since; and says it as the event put. It says whether it wrote.
func (p *passer) writePut(ctx context.Context, key string, read, next delivery.Put) bool {
kv := p.kvs[delivery.AssignerPutsBucket]
if kv == nil {
return false
}
e, err := kv.Get(ctx, key)
if err != nil {
p.logf("the put %s cannot be read again, and is taken further at the next pass: %v", key, err)
return false
}
now, err := delivery.DecodePut(e.Value())
if err != nil || !now.At.Equal(read.At) || now.Step != read.Step {
return false // asked again or released meanwhile: the next pass reads that
}
data, err := delivery.EncodePut(next)
if err != nil {
p.logf("the put %s is not written: %v", key, err)
return false
}
if _, err := kv.Update(ctx, key, data, e.Revision()); err != nil {
if !errors.Is(err, context.Canceled) {
p.logf("the put %s could not be written, and is taken further at the next pass: %v", key, err)
}
return false
}
p.w.put(delivery.AssignerPutsBucket, key, data)
p.logf("put %s: %s: %s", key, next.Step, next.Words)
p.acts.say(key, next)
return true
}
// state is what this process knows of a put's acts.
func (a *putActs) state(id string) (snapshot string, snapping bool, snapNo string) {
a.mu.Lock()
defer a.mu.Unlock()
return a.snapped[id], a.snapping[id], a.snapNo[id]
}
// snap has the bus's snapshot taken once for a put, in the background.
func (a *putActs) snap(ctx context.Context, id, node, module string) {
a.mu.Lock()
if a.snapping[id] || a.snapped[id] != "" || a.snapNo[id] != "" {
a.mu.Unlock()
return
}
a.snapping[id] = true
a.mu.Unlock()
go func() {
defer a.wake()
where, err := a.snapshot(ctx, node, module)
a.mu.Lock()
defer a.mu.Unlock()
a.snapping[id] = false
if err != nil {
a.snapNo[id] = err.Error()
return
}
a.snapped[id] = where
}()
}
// SnapshotWithin is how long the bus node's backup holder is given to take the bus's snapshot (as the controller's bus step).
var SnapshotWithin = 15 * time.Minute
// snapshotLook is how often the backup holder's record is read back while the snapshot is taken.
var snapshotLook = 10 * time.Second
// backedUp is one module as the backup holder's `backed-up` answers it.
type backedUp struct {
Module string `json:"module"`
Data []struct {
Item string `json:"item"`
LastBackup *time.Time `json:"last_backup"`
Error string `json:"error"`
} `json:"data"`
}
// snapshotTheBus asks the bus node's backup holder to back the bus module up now, whose dump is the streams' snapshot
// (ADR 0235), and reads its record back until it names a backup newer than the ask: where the snapshot is.
func (a *putActs) snapshotTheBus(ctx context.Context, node, module string) (string, error) {
asked := time.Now()
if _, err := a.bus.CallOn(node, "seat:"+catalogue.BackupSeat+".now", map[string]any{"module": module}); err != nil {
return "", fmt.Errorf("%s's backup holder was not asked to take the bus's snapshot: %w", node, err)
}
deadline := asked.Add(SnapshotWithin)
for {
if raw, err := a.bus.CallOn(node, "seat:"+catalogue.BackupSeat+".backed-up", map[string]any{}); err == nil {
var modules []backedUp
if json.Unmarshal(raw, &modules) == nil {
for _, m := range modules {
for _, it := range m.Data {
if m.Module == module && it.LastBackup != nil && it.LastBackup.After(asked) && it.Error == "" {
return fmt.Sprintf("%s's restore point of %s (%s) taken %s", node, module, it.Item, it.LastBackup.UTC().Format(time.RFC3339)), nil
}
}
}
}
}
if time.Now().After(deadline) {
return "", fmt.Errorf("%s's backup holder did not say the bus's snapshot was taken within %s", node, SnapshotWithin)
}
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(snapshotLook):
}
}
}
// warrant records once, in the controller's hand-act log, that the bus's want was written on the operator's warrant: the
// router's record of the teller's ask says who chose and how (ADR 0274).
func (a *putActs) warrant(id, ask string) {
a.mu.Lock()
if a.warranted[id] {
a.mu.Unlock()
return
}
a.warranted[id] = true
a.mu.Unlock()
go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil {
a.logf("the put's warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Lock()
a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the controller's hand-act log: %v", ask, err)
a.mu.Unlock()
a.wake()
}
}()
}
// warrantFailed says why the put's warrant was not recorded, "" when it was or is not yet known.
func (a *putActs) warrantFailed(id string) string {
a.mu.Lock()
defer a.mu.Unlock()
return a.warrantNo[id]
}
// putSaid is the body of the event put.
type putSaid struct {
Node string `json:"node"`
Module string `json:"module"`
FP string `json:"fp"`
Hold bool `json:"hold"`
Step string `json:"step"`
Words string `json:"words"`
Composed string `json:"composed,omitempty"`
}
// say says a put's step as the event put, once per step and words.
func (a *putActs) say(key string, p delivery.Put) {
node, module, err := delivery.ParsePutKey(key)
if err != nil || a.bus == nil {
return
}
a.mu.Lock()
if a.said[key] == p.Step+p.Words {
a.mu.Unlock()
return
}
a.said[key] = p.Step + p.Words
a.mu.Unlock()
if err := a.bus.Emit(PutEvent, putSaid{Node: node, Module: module, FP: p.FP, Hold: p.Hold, Step: p.Step, Words: p.Words, Composed: p.Composed}); err != nil {
a.logf("the put %s's step %s is not said as an event: %v", key, p.Step, err)
}
}
// shortHex is a digest's first twelve hex digits.
func shortHex(digest string) string {
h := strings.TrimPrefix(digest, "sha256:")
if len(h) > 12 {
return h[:12]
}
return h
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,364 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"os"
"regexp"
"slices"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The assigner's tools (novox/hq ADR 0546), served through the tool runner from its bundle: a process never serves (ADR
// 0436, mesh-sdk process). They write and read the puts; the assigner's process takes each put further (puts.go) and says
// every step as the event put. They are the module's own tools, not a seat's verbs: a new seat needs a change of the
// controller, which takes none; they become the verbs of a seat once composing leaves the controller.
// putsState is the assigner's state of puts, as the runtime names a module's own state.
const putsState = "puts"
// answerWithin is how long assigner_put waits for the assigner's process to take the put up before it answers what stands.
var answerWithin = 20 * time.Second
// store is what the tools need of the states: the runtime's (stdio.State), or a test's.
type store interface {
get(state, key string) ([]byte, uint64, bool, error)
put(state, key string, value []byte) error
keys(state string) ([]string, error)
}
// putTools serves the tools over a store, and keeps the newest value of every put it heard, to wait on.
type putTools struct {
st store
now func() time.Time
mu sync.Mutex
// heard is every put the watch handed, by key; changed is closed and replaced at each.
heard map[string][]byte
changed chan struct{}
}
func newPutTools(st store) *putTools {
return &putTools{st: st, now: time.Now, heard: map[string][]byte{}, changed: make(chan struct{})}
}
// hear takes a put's change in from the watch.
func (s *putTools) hear(key string, value []byte) {
s.mu.Lock()
s.heard[key] = value
close(s.changed)
s.changed = make(chan struct{})
s.mu.Unlock()
}
// fullFP is a fingerprint of the new form, whole; prefix at least its first twelve hex digits.
var (
fullFP = regexp.MustCompile(`^src2:[0-9a-f]{64}$`)
prefix = regexp.MustCompile(`^(src2:)?[0-9a-f]{12,63}$`)
)
// version reads the version a put names: newest (the indexer's fingerprint at the trunk), a whole fingerprint, or the start
// of one the builder built, which must name one build alone.
func (s *putTools) version(module, version string) (string, error) {
v := strings.ToLower(strings.TrimSpace(version))
switch {
case v == "newest":
raw, _, found, err := s.st.get(delivery.IndexerModule+".modules", module)
if err != nil {
return "", fmt.Errorf("the indexer's modules cannot be read: %w", err)
}
if !found {
return "", fmt.Errorf("the indexer holds no module %s at the trunk", module)
}
m, err := delivery.DecodeModule(raw)
if err != nil {
return "", fmt.Errorf("the indexer's %s cannot be read: %w", module, err)
}
if m.FP2 == "" {
return "", fmt.Errorf("the indexer holds no fingerprint of the new form for %s: %s", module, m.None2)
}
return m.FP2, nil
case fullFP.MatchString(v):
return v, nil
case prefix.MatchString(v):
start := strings.TrimPrefix(v, delivery.SourcePrefix2)
keys, err := s.st.keys(delivery.BuilderModule + ".builds")
if err != nil {
return "", fmt.Errorf("the builder's builds cannot be listed: %w", err)
}
var found []string
for _, k := range keys {
m, fp, err := delivery.ParseBuilderBuildKey(k)
if err == nil && m == module && strings.HasPrefix(strings.TrimPrefix(fp, delivery.SourcePrefix2), start) {
found = append(found, fp)
}
}
switch len(found) {
case 0:
return "", fmt.Errorf("the builder built no %s whose fingerprint starts %s: give it whole", module, start)
case 1:
return found[0], nil
}
sort.Strings(found)
return "", fmt.Errorf("%s starts %d fingerprints of %s the builder built (%s): give more of it", start, len(found), module, strings.Join(found, ", "))
}
return "", fmt.Errorf("%q is no version: newest, a fingerprint (src2: and 64 hex digits) or at least its first 12 hex digits", version)
}
// putAnswer is what assigner_put and assigner_puts answer of one put.
type putAnswer struct {
Node string `json:"node"`
Module string `json:"module"`
delivery.Put
Follow string `json:"follow,omitempty"`
}
// doPut is the tool assigner_put.
func (s *putTools) doPut(args map[string]any, by string) (any, error) {
node, module, why := arg(args, "node"), arg(args, "module"), arg(args, "why")
key, err := delivery.PutKey(node, module)
if err != nil {
return nil, err
}
if why == "" {
return nil, errors.New("put says why: it is kept with the put and said in any ask")
}
hold := false
switch arg(args, "hold") {
case "true":
hold = true
case "", "false":
default:
return nil, fmt.Errorf("hold is true or false, not %q", arg(args, "hold"))
}
fp, err := s.version(module, arg(args, "version"))
if err != nil {
return nil, err
}
now := s.now().UTC()
words := fmt.Sprintf("asked by %s: %s", by, why)
if raw, _, found, err := s.st.get(putsState, key); err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
} else if found {
if before, err := delivery.DecodePut(raw); err == nil && !delivery.PutEnded(before.Step) {
if before.FP == fp && before.Hold == hold {
return putAnswer{Node: node, Module: module, Put: before, Follow: followPut(node, module)}, nil
}
words += fmt.Sprintf("; it replaces the put of %s at %s, which stood at %s", module, shortFP(before.FP), before.Step)
}
}
p := delivery.Put{FP: fp, Hold: hold, Why: why, By: by, At: now}.Said(now, delivery.PutAsked, words)
data, err := delivery.EncodePut(p)
if err != nil {
return nil, err
}
if err := s.st.put(putsState, key, data); err != nil {
return nil, fmt.Errorf("the put of %s on %s is not written: %w", module, node, err)
}
s.hear(key, data)
return putAnswer{Node: node, Module: module, Put: s.await(key, p), Follow: followPut(node, module)}, nil
}
// await is the put at key once the assigner's process took it up past its asking, or as it stands after answerWithin.
func (s *putTools) await(key string, p delivery.Put) delivery.Put {
deadline := time.After(answerWithin)
for {
s.mu.Lock()
raw, changed := s.heard[key], s.changed
s.mu.Unlock()
if now, err := delivery.DecodePut(raw); err == nil && now.At.Equal(p.At) {
p = now
if now.Step != delivery.PutAsked && now.Step != delivery.PutBuild && now.Step != delivery.PutCompose {
return p // it waits on a person or the node, or ended
}
}
select {
case <-changed:
case <-deadline:
return p
}
}
}
// followPut says how a caller follows a put after the answer.
func followPut(node, module string) string {
return fmt.Sprintf("the put goes on: mesh_wait on mesh-delivery-assigner.put with filter {\"node\": %q, \"module\": %q}, then "+
"mesh-delivery-assigner.assigner_puts for where it stands", node, module)
}
// doRelease is the tool assigner_release.
func (s *putTools) doRelease(args map[string]any, by string) (any, error) {
node, module, why := arg(args, "node"), arg(args, "module"), arg(args, "why")
key, err := delivery.PutKey(node, module)
if err != nil {
return nil, err
}
if why == "" {
return nil, errors.New("release says why: it is kept with the put")
}
raw, _, found, err := s.st.get(putsState, key)
if err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
}
if !found {
return nil, fmt.Errorf("%s holds no put of %s: nothing to release", node, module)
}
p, err := delivery.DecodePut(raw)
if err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
}
if delivery.PutEnded(p.Step) {
return nil, fmt.Errorf("the put of %s on %s ended already (%s: %s): nothing to release", module, node, p.Step, p.Words)
}
p = p.Said(s.now(), delivery.PutReleased, fmt.Sprintf("released by %s: %s; the assigner's rules write the want again", by, why))
data, err := delivery.EncodePut(p)
if err != nil {
return nil, err
}
if err := s.st.put(putsState, key, data); err != nil {
return nil, fmt.Errorf("the release of %s on %s is not written: %w", module, node, err)
}
s.hear(key, data)
return putAnswer{Node: node, Module: module, Put: p}, nil
}
// doPuts is the tool assigner_puts.
func (s *putTools) doPuts(args map[string]any) (any, error) {
keys, err := s.st.keys(putsState)
if err != nil {
return nil, fmt.Errorf("the puts cannot be listed: %w", err)
}
sort.Strings(keys)
out := []putAnswer{}
for _, key := range keys {
node, module, err := delivery.ParsePutKey(key)
if err != nil || (arg(args, "node") != "" && node != arg(args, "node")) || (arg(args, "module") != "" && module != arg(args, "module")) {
continue
}
raw, _, found, err := s.st.get(putsState, key)
if err != nil || !found {
continue
}
p, err := delivery.DecodePut(raw)
if err != nil {
continue
}
out = append(out, putAnswer{Node: node, Module: module, Put: p})
}
return map[string]any{"puts": out}, nil
}
func arg(args map[string]any, name string) string {
switch v := args[name].(type) {
case string:
return strings.TrimSpace(v)
case bool:
return fmt.Sprint(v)
}
return ""
}
// The tools' names, as the manifest lists them.
const (
toolPut = "assigner_put"
toolRelease = "assigner_release"
toolPuts = "assigner_puts"
)
// schema is a tool's arguments, each a string: required ones first named, and switches true or false.
func schema(properties map[string]string, required []string, switches ...string) map[string]any {
props := map[string]any{}
for name, description := range properties {
p := map[string]any{"type": "string", "description": description}
if slices.Contains(switches, name) {
p["enum"] = []string{"true", "false"}
}
props[name] = p
}
out := map[string]any{"type": "object", "properties": props}
if len(required) > 0 {
out["required"] = required
}
return out
}
// tools are the assigner's tools as the runtime serves them (ADR 0546 clause 1).
func (s *putTools) tools() []stdio.Tool {
return []stdio.Tool{
{Name: toolPut, Description: "Put one version of a module on one machine (novox/hq ADR 0546): the build at that fingerprint, " +
"the composition and grant the controller wrote for that machine, then the machine's want naming it, held there until " +
"released when hold is true. A module whose moves a person takes (the bus, a module of policy record) is asked of the " +
"operator first, and the bus has its streams snapshotted after the yes. Answers what was done or what it waits on; the " +
"put goes on after the answer, each step said as the event mesh-delivery-assigner.put, and assigner_puts says where it stands.",
Input: schema(map[string]string{
"node": "the machine, where the module is assigned",
"module": "the module",
"version": "newest, a fingerprint (src2: and 64 hex digits), or at least its first 12 hex digits",
"hold": "\"true\": the want stays at this version until released; \"false\" (the default): the rules move it again once it runs",
"why": "why, kept with the put and said in any ask",
}, []string{"node", "module", "version", "why"}, "hold"),
Run: func(a map[string]any) (any, error) { return s.doPut(a, stdio.CallerOf(a)) }},
{Name: toolRelease, Description: "End a put on one machine (novox/hq ADR 0546): its hold is taken away and the assigner's " +
"rules write the machine's want again from their next pass. Nothing changes on the machine until they move it.",
Input: schema(map[string]string{"node": "the machine", "module": "the module", "why": "why, kept with the put"},
[]string{"node", "module", "why"}),
Run: func(a map[string]any) (any, error) { return s.doRelease(a, stdio.CallerOf(a)) }},
{Name: toolPuts, Description: "Every put and where it stands (novox/hq ADR 0546): its version, hold, why, each step with " +
"its time and words. Narrowed by machine or module. Read-only.",
Input: schema(map[string]string{"node": "one machine", "module": "one module"}, nil),
Run: s.doPuts},
}
}
// runtimeStore is the runtime's states.
type runtimeStore struct{}
func (runtimeStore) get(state, key string) ([]byte, uint64, bool, error) {
e, err := stdio.State(state).Get(key)
if err != nil || e == nil {
return nil, 0, false, err
}
return e.Value, e.Revision, true, nil
}
func (runtimeStore) put(state, key string, value []byte) error {
_, err := stdio.State(state).Put(key, json.RawMessage(value))
return err
}
func (runtimeStore) keys(state string) ([]string, error) { return stdio.State(state).Keys() }
// serveTools serves the tools to the runtime until it stops, hearing every put to answer assigner_put with its first steps.
func serveTools() error {
s := newPutTools(runtimeStore{})
go func() {
<-stdio.Served()
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
err := stdio.State(putsState).Watch("", func(c stdio.StateChange) error {
s.hear(c.Key, c.Value)
return nil
})
if err == nil {
return
}
fmt.Fprintf(os.Stderr, "mesh-delivery-assigner: the puts are not heard yet (%v); asked again in %s\n", err, wait)
time.Sleep(wait)
}
}()
return stdio.Serve("", s.tools())
}
// shortFP is a fingerprint's first twelve hex digits.
func shortFP(fp string) string {
part, err := delivery.FingerprintKeyPart(fp)
if err != nil || len(part) < 12 {
return fp
}
return part[:12]
}
@@ -0,0 +1,437 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
"mesh-delivery-assigner/internal/assign"
)
// noModuleWords is the indexer's reason for a module of a candidate whose directory the merge preview holds no manifest
// for (mesh-sdk delivery.CandidateNoneNoModule, which the indexer writes): the change removes the module.
const noModuleWords = delivery.CandidateNoneNoModule
// assignment holds only the fields of the controller's Assignment the assigner reads (the view's own, mesh-delivery-view
// internal/view/view.go): an assignment stands while it has not ended and has no state.
type assignment struct {
Node string `json:"node"`
Module string `json:"module"`
Build string `json:"build"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
func (a assignment) open() bool { return a.Ended.IsZero() && a.State == "" }
// world is what the assigner watched, as it stands now: every value by bucket and key, and the manifests read by
// digest. The watches write it; a pass reads it.
type world struct {
mu sync.Mutex
values map[string]map[string][]byte
manifests map[string]parsed
// healthy is, by have key, every fingerprint the node ran healthy, oldest first, each once: what a set-back goes to.
healthy map[string][]string
// changes counts, by bucket and key, the values this process has seen the key take: a value written again is another.
changes map[string]uint64
}
// parsed is a manifest read by digest: its module and its manifest, or why it cannot be read.
type parsed struct {
module string
manifest catalogue.Manifest
err string
}
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{}}
}
// put takes a change of a watched bucket in: a value, or nil for a key deleted or purged.
func (w *world) put(bucket, key string, value []byte) {
w.mu.Lock()
defer w.mu.Unlock()
if w.values[bucket] == nil {
w.values[bucket] = map[string][]byte{}
}
if value == nil {
delete(w.values[bucket], key)
return
}
w.values[bucket][key] = value
w.changes[bucket+"|"+key]++
if bucket == delivery.HaveBucket {
if h, err := delivery.DecodeHave(value); err == nil && h.FP != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
list := slices.DeleteFunc(w.healthy[key], func(fp string) bool { return fp == h.FP })
w.healthy[key] = append(list, h.FP)
}
}
}
// lastHealthy is the newest fingerprint other than fp the node ran the module healthy at, or empty.
func (w *world) lastHealthy(node, module, fp string) string {
key, err := delivery.HaveKey(node, module)
if err != nil {
return ""
}
w.mu.Lock()
defer w.mu.Unlock()
list := w.healthy[key]
for i := len(list) - 1; i >= 0; i-- {
if list[i] != fp {
return list[i]
}
}
return ""
}
// read is a copy of what a bucket holds now.
func (w *world) read(bucket string) map[string][]byte {
w.mu.Lock()
defer w.mu.Unlock()
out := make(map[string][]byte, len(w.values[bucket]))
for k, v := range w.values[bucket] {
out[k] = v
}
return out
}
// getter reads one key's value; errNotThere when the key does not exist.
type getter func(ctx context.Context, key string) ([]byte, error)
var errNotThere = errors.New("no such key")
// kvGetter reads keys of a bucket.
func kvGetter(kv jetstream.KeyValue) getter {
return func(ctx context.Context, key string) ([]byte, error) {
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, errNotThere
}
if err != nil {
return nil, err
}
return e.Value(), nil
}
}
// manifest reads the manifest of a digest from the indexer's bucket once and keeps it: a digest's manifest never changes.
func (w *world) manifest(ctx context.Context, get getter, digest string) parsed {
w.mu.Lock()
p, ok := w.manifests[digest]
w.mu.Unlock()
if ok {
return p
}
key, err := delivery.ManifestsKey(digest)
if err != nil {
return parsed{err: err.Error()}
}
value, err := get(ctx, key)
if errors.Is(err, errNotThere) {
return parsed{err: "the indexer holds no manifest of digest " + digest + " yet"}
}
if err != nil {
return parsed{err: err.Error()}
}
m, err := delivery.DecodeManifest(value)
if err != nil {
p = parsed{err: err.Error()}
} else if man, err := catalogue.ParseManifestForInputs([]byte(m.JSON)); err != nil {
// Read as the indexer reads it (issue 1751): a value falling back to a predefined key is judged by the merge gate
// in the catalogue, against the predefined.json at its root that this process never holds (ADR 0348 §7).
p = parsed{module: m.Module, err: err.Error()}
} else {
p = parsed{module: m.Module, manifest: man}
}
w.mu.Lock()
w.manifests[digest] = p
w.mu.Unlock()
return p
}
// assigned is each node's modules with an open assignment, read from the controller's assignments, in name order.
func (w *world) assigned() (map[string][]string, map[string]assignment) {
out, all := map[string][]string{}, map[string]assignment{}
for _, value := range w.read(provider.AssignmentsBucket) {
var a assignment
if json.Unmarshal(value, &a) != nil || a.Node == "" || a.Module == "" || !a.open() {
continue
}
out[a.Node] = append(out[a.Node], a.Module)
all[a.Node+"."+a.Module] = a
}
for _, mods := range out {
sort.Strings(mods)
}
return out, all
}
// trunk is every module the indexer holds at the trunk, with its value.
func (w *world) trunk() map[string]delivery.Module {
out := map[string]delivery.Module{}
for key, value := range w.read(delivery.ModulesBucket) {
name, err := delivery.ParseModulesKey(key)
if err != nil {
continue
}
if m, err := delivery.DecodeModule(value); err == nil {
out[name] = m
}
}
return out
}
// mesh is the mesh as the trunk holds it, for the composition check.
func (w *world) mesh(ctx context.Context, manifests getter) assign.Mesh {
m := assign.Mesh{Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{}, Digests: map[string]string{}}
m.Assigned, _ = w.assigned()
for name, mod := range w.trunk() {
if mod.Manifest == "" {
continue
}
m.Digests[name] = mod.Manifest
p := w.manifest(ctx, manifests, mod.Manifest)
if p.err != "" {
m.Unreadable[name] = p.err
continue
}
m.Manifests[name] = p.manifest
}
return m
}
// change is a candidate's change for the composition check, or why the composition waits for it.
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}}
definedIn := map[string]map[string]bool{}
for _, p := range c.Pulls {
if p.Repository == assign.SDKRepository {
ch.SDKHead = p.Head.Head
}
ch.Members[p.Repository] = p.Head.Head
for _, name := range p.Modules {
fp, ok := p.Fps[name]
if !ok || fp.Manifest == "" {
if ok && fp.None2 == noModuleWords {
ch.Removed = append(ch.Removed, name)
continue
}
return ch, fmt.Sprintf("the indexer holds no manifest of %s for %s#%d yet", name, p.Repository, p.Number)
}
if fp.FP2 != "" && fp.FP2 != trunk[name].FP2 {
ch.Rebuilt++
}
got := w.manifest(ctx, manifests, fp.Manifest)
if strings.HasPrefix(got.err, "the indexer holds no manifest") {
return ch, got.err
}
ch.Digests[name] = fp.Manifest
if got.err != "" {
ch.Unreadable[name] = got.err
continue
}
ch.Manifests[got.module] = got.manifest
if definedIn[got.module] == nil {
definedIn[got.module] = map[string]bool{}
}
definedIn[got.module][p.Repository] = true
if t, ok := trunk[got.module]; ok && t.Repository != "" && t.Repository != p.Repository {
definedIn[got.module][t.Repository] = true
}
}
}
for name, repos := range definedIn {
if len(repos) > 1 {
for r := range repos {
ch.Twice[name] = append(ch.Twice[name], r)
}
sort.Strings(ch.Twice[name])
}
}
return ch, ""
}
// wantsOf is the wants the assigner wrote, by module then node.
func (w *world) wantsOf() map[string]map[string]delivery.Want {
out := map[string]map[string]delivery.Want{}
for key, value := range w.read(delivery.AssignerWantsBucket) {
node, module, err := delivery.ParseWantKey(key)
if err != nil {
continue
}
want, err := delivery.DecodeWant(value)
if err != nil {
continue
}
if out[module] == nil {
out[module] = map[string]delivery.Want{}
}
out[module][node] = want
}
return out
}
// nodes is what the six rules read of the nodes: presence, haves, and every provision's providers with their nodes.
func (w *world) nodes(manifests map[string]catalogue.Manifest, assigned map[string][]string) assign.Nodes {
n := assign.Nodes{Online: map[string]bool{}, Haves: map[string]map[string]delivery.Have{}, Providers: map[string]map[string][]string{}}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
continue
}
if p, err := delivery.DecodePresence(value); err == nil {
n.Online[node] = p.Online
}
}
for key, value := range w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
continue
}
if h, err := delivery.DecodeHave(value); err == nil {
if n.Haves[node] == nil {
n.Haves[node] = map[string]delivery.Have{}
}
n.Haves[node][module] = h
}
}
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
nodesOf[m] = append(nodesOf[m], node)
}
}
for name, man := range manifests {
for _, o := range man.Provides {
if n.Providers[o.Name] == nil {
n.Providers[o.Name] = map[string][]string{}
}
n.Providers[o.Name][name] = nodesOf[name]
}
}
return n
}
// value is what a bucket holds at a key now, nil when nothing.
func (w *world) value(bucket, key string) []byte {
w.mu.Lock()
defer w.mu.Unlock()
return w.values[bucket][key]
}
// has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool {
w.mu.Lock()
defer w.mu.Unlock()
_, ok := w.values[bucket][key]
return ok
}
// built says whether the builder's build of a module at a fingerprint is done.
func (w *world) built(module, fp string) bool {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return false
}
w.mu.Lock()
value, ok := w.values[delivery.BuilderBuildsBucket][key]
w.mu.Unlock()
if !ok {
return false
}
b, err := delivery.DecodeBuilderBuild(value)
return err == nil && b.State == delivery.BuildDone
}
// composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it
// (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read.
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
key, err := delivery.ComposedKey(node, module, fp)
if err != nil {
return delivery.ComposedDigest{}, false
}
w.mu.Lock()
value, ok := w.values[delivery.ControllerComposedBucket][key]
w.mu.Unlock()
if !ok {
return delivery.ComposedDigest{}, false
}
d, err := delivery.DecodeComposedDigest(value)
return d, err == nil
}
// nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`).
func (w *world) nodeComposed() map[string]delivery.ComposedDigest {
out := map[string]delivery.ComposedDigest{}
for key, value := range w.read(delivery.ControllerComposedBucket) {
node, _, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || !isNode {
continue
}
if d, err := delivery.DecodeComposedDigest(value); err == nil {
out[node] = d
}
}
return out
}
// answer is the teller's answer to the ask at key (mesh-delivery-teller_answers, the same key), and whether there is one.
func (w *world) answer(key string) (delivery.Answer, bool) {
w.mu.Lock()
value, ok := w.values[delivery.AnswersBucket][key]
w.mu.Unlock()
if !ok {
return delivery.Answer{}, false
}
a, err := delivery.DecodeAnswer(value)
return a, err == nil
}
// changed is how many values the key has taken in this process (see world.changes).
func (w *world) changed(bucket, key string) uint64 {
w.mu.Lock()
defer w.mu.Unlock()
return w.changes[bucket+"|"+key]
}
// runningCommits is, of every open assignment of module, the commit of the build it names, read from the controller's
// build record (`mesh-controller_builds` `<module>.build.<id>`), distinct and sorted: the commits a reader of the
// manifests runs at (novox/hq ADR 0482 clause 5, issue 1623). A build whose record names no full commit is left out.
func (w *world) runningCommits(ctx context.Context, get getter, module string) []string {
_, all := w.assigned()
seen := map[string]bool{}
for _, a := range all {
if a.Module != module || a.Build == "" {
continue
}
value, err := get(ctx, delivery.BuildKey(module, a.Build))
if err != nil {
continue
}
var rec delivery.BuildRecord
if json.Unmarshal(value, &rec) != nil || len(rec.Commit) != 40 {
continue
}
seen[rec.Commit] = true
}
out := make([]string, 0, len(seen))
for c := range seen {
out = append(out, c)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,127 @@
package main
import (
"context"
"os"
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// fakeManifests is the indexer's manifests bucket in memory.
type fakeManifests map[string][]byte
func (f fakeManifests) add(t *testing.T, module, body string) string {
t.Helper()
digest := delivery.ManifestDigest([]byte(body))
value, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: body})
if err != nil {
t.Fatal(err)
}
f[digest] = value
return digest
}
func (f fakeManifests) get(_ context.Context, key string) ([]byte, error) {
if v, ok := f[key]; ok {
return v, nil
}
return nil, errNotThere
}
func fp(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
func pull(repo string, number int, modules map[string]delivery.CandidateModule) delivery.CandidatePull {
p := delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: strings.Repeat("3", 40)}, Fps: modules}
for m := range modules {
p.Modules = append(p.Modules, m)
}
slices.Sort(p.Modules)
return p
}
// 1297/B3b: a candidate's change reads each module's manifest by its digest, counts the modules rebuilt, takes a
// module with no manifest in the preview as removed, a mesh-sdk member's head, and a module the trunk holds from
// another repository as defined twice.
func TestTheChangeOfACandidate(t *testing.T) {
f := fakeManifests{}
web := f.add(t, "web", `{"module":"web","version":"2"}`)
mail := f.add(t, "mail", `{"module":"mail","version":"1"}`)
trunk := map[string]delivery.Module{"web": {FP2: fp('a'), Repository: "novox/web"}, "mail": {FP2: fp('c'), Repository: "novox/mail"}}
c := delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: web}, "old": {None2: noModuleWords}}),
pull("novox/other", 2, map[string]delivery.CandidateModule{"mail": {FP2: fp('c'), Manifest: mail}}),
pull(assign.SDKRepository, 3, nil),
}}
ch, waits := newWorld().change(context.Background(), f.get, c, trunk)
if waits != "" {
t.Fatalf("the change waits: %s", waits)
}
if ch.Manifests["web"].Version != "2" || ch.Rebuilt != 1 || !slices.Equal(ch.Removed, []string{"old"}) || ch.SDKHead != strings.Repeat("3", 40) {
t.Errorf("change = %+v", ch)
}
if !slices.Equal(ch.Twice["mail"], []string{"novox/mail", "novox/other"}) || len(ch.Twice) != 1 {
t.Errorf("twice = %v, want mail in novox/mail and novox/other", ch.Twice)
}
}
// 1297/B3b: a candidate whose module has no manifest digest yet, or whose manifest the indexer does not hold yet, waits,
// saying for what; an unreadable manifest is the change's, not a wait.
func TestAChangeWaitsForItsManifests(t *testing.T) {
f := fakeManifests{}
bad := f.add(t, "web", `{"module":"web","version":"2","scope":"x"}`)
for _, c := range []delivery.CandidatePull{
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {None2: "no preview yet"}}),
pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: strings.Repeat("9", 64)}}),
} {
_, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab", Pulls: []delivery.CandidatePull{c}}, nil)
if waits == "" {
t.Errorf("a change without its manifest %+v does not wait", c.Fps)
}
}
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/web", 1, map[string]delivery.CandidateModule{"web": {FP2: fp('b'), Manifest: bad}})}}, nil)
if waits != "" || ch.Unreadable["web"] == "" {
t.Errorf("an unreadable manifest: waits %q, change %+v", waits, ch)
}
}
// A manifest of the change whose config value falls back to a predefined key (novox/hq ADR 0348 §7) is read, not
// unreadable for the catalogue's predefined.json the assigner never holds: the merge gate judges the fallback in the
// catalogue, and the indexer reads it the same way (catalogue.ParseManifestForInputs, issue 1751). A group whose
// mesh-catalog member touched pacman and xterm failed its composition on this (group check 18df81380db3). The manifest is
// xterm's module.json from mesh-catalog at 835bf22b.
func TestAManifestFallingBackToAPredefinedKeyIsReadable(t *testing.T) {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(raw), `"predefined"`) {
t.Fatal("the fixture names no predefined key")
}
f := fakeManifests{}
xterm := f.add(t, "xterm", string(raw))
ch, waits := newWorld().change(context.Background(), f.get, delivery.Candidate{Group: "g", HeadsHash: "0123456789ab",
Pulls: []delivery.CandidatePull{pull("novox/mesh-catalog", 1, map[string]delivery.CandidateModule{"xterm": {FP2: fp('b'), Manifest: xterm}})}}, nil)
if waits != "" || ch.Unreadable["xterm"] != "" || ch.Manifests["xterm"].Module != "xterm" {
t.Errorf("xterm: waits %q, unreadable %q, want read", waits, ch.Unreadable["xterm"])
}
}
// Review round 1 (items 3 and 4): the world counts the values a key takes, so a have written again after an answer is seen.
func TestTheWorldCountsTheValuesAKeyTakes(t *testing.T) {
w := newWorld()
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.mail", []byte(`{"v":1}`))
if got := w.changed(delivery.HaveBucket, "ace.web"); got != 2 {
t.Errorf("changed = %d, want 2", got)
}
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
}
+22
View File
@@ -0,0 +1,22 @@
module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+35
View File
@@ -0,0 +1,35 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c h1:pY2YpHSSsfDPBrOEiWTNSIPn0zdnAayRmtmmyQ3hCHY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024210118-748c564cdf7c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8 h1:jA7V7hlfJAigVUvJDpKwfgd9ylQ9bqppg3rK9XgLWoA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025195051-20a6dc4920f8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8 h1:vhYk/iv43u2WGSTrFyQGDvGBxLaRITgMAiiI9DjuA10=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025200810-d4ae4b6790b8/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5 h1:gE80bR0tPME7MTMBLcdP62v+F/9soVXmdcBYNMOye44=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025203528-db26f7e8b6f5/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,206 @@
package assign
import (
"fmt"
"regexp"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HoldAsk is the ask about a module's fingerprint the assigner holds, at its key (delivery.HoldAskKey): keep it held or
// release it. Releasing writes the held fingerprint's want again by the rules (ADR 0539 §19). node and words are the held
// record's: the machine whose health check failed and the hold's words (issue 1906). asking numbers the writing: 0 the first, a
// larger number an ask written again after a hold lifted by itself, so the router takes it as new.
func HoldAsk(key, module, fp, node, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp)),
Explanation: fmt.Sprintf("The health check of %s failed on %s at %s (%s), so the assigner holds that version and wants "+
"no node to move to it. It lifts the hold by itself once every node that runs %s runs it healthy; releasing it "+
"lets the wants move to it now.", module, node, short(fp), words, module),
Options: []asks.Option{
{ID: "keep", Label: "Keep it held", Does: "the version stays held", Level: asks.Acknowledge},
{ID: "release", Label: "Release it", Does: "the version is released and the wants move to it by the rules again", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the version stays held",
About: key,
}}
}
// The options of the restore ask (ADR 0537 §3, ADR 0539 §19), and what each does on the node.
const (
OptionRestore = "restore"
OptionLeaveStopped = "leave-stopped"
)
// RestoreAsk is the ask about a module holding data whose fingerprint failed its health check on a node (ADR 0429 §6): it is
// not set back by itself, since a set-back would lose the writes made since. words is the check's words. Restore sets the
// node back to the last healthy fingerprint and loses the newer writes; leaving it stopped keeps the data as it is.
func RestoreAsk(key, node, module, fp, words string, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s failed its health check on %s", module, node),
Explanation: fmt.Sprintf("%s holds data, and its health check failed on %s at %s: %s. Setting it back to the version that ran healthy "+
"would lose the writes made since; leaving it stopped keeps the data as it is.", module, node, short(fp), words),
Options: []asks.Option{
{ID: OptionRestore, Label: "Restore it", Does: "the node is set back to the last healthy version and the newer writes are lost", Level: asks.Acknowledge},
{ID: OptionLeaveStopped, Label: "Leave it stopped", Does: "the module stays as it is on the node and the data is kept", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module stays as it is on the node",
About: key,
}}
}
// HeldCondition is the condition the assigner raises when it holds a fingerprint of a module (ADR 0429 §6, ADR 0539 §19), or
// clears (raised false) when the hold is released. It names the module, the node whose health check failed and the check's
// words, under the assigner's own name (ADR 0437 clause 9), keyed by the held fingerprint. needsYou is true for a module holding
// data, whose restore ask waits for the operator.
func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.Condition {
key := delivery.AssignerModule + ".held." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
c := asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: fmt.Sprintf("%s is held at %s", module, short(fp)),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp))}
what := fmt.Sprintf("The health check of %s failed on %s at %s: %s. The assigner holds that version and wants no node to move to it.", module, node, short(fp), words)
if needsYou {
c.Needs = "answer whether to restore " + module + " on " + node
c.Explanation = asks.NeedsYou + c.Needs + ". " + what
} else {
c.Explanation = asks.NothingToDo + " " + what + " Nodes that wanted it are set back to the version that ran healthy."
}
return c
}
// NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
// node and module, and cleared (raised false) when the node has a want for it again.
func NeitherCondition(node, module string, h Holding, raised bool) asks.Condition {
key := delivery.AssignerModule + ".neither." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s has a want again, so its own composition is applied there.", module, node)}
}
headline := fmt.Sprintf("%s on %s is on no list", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner's rule %d holds its first want back: %s. The assigner writes the want once the "+
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
c := HeldCondition(module, fp, "", "", false, false)
c.Resolved = LiftedWords(module, fp) + "."
return c
}
// The options of the record ask (ADR 0539 §14) and of the backup ask (§16).
const (
OptionYes = "yes"
OptionNo = "no"
OptionBackupAgain = delivery.BackupAgain
OptionBackupSkip = delivery.BackupSkip
)
// RecordAsk is the ask about a move of a module of policy record on node to the composition with the digest, judged by the
// controller to recreate a container, to move data, or not to be judgeable (ADR 0539 §14). The want names the composition only
// after the answer yes. The way the move was judged is in the ask.
func RecordAsk(key, node, module, fp string, need RecordNeed, now time.Time) delivery.AssignerAsk {
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + need.Words,
}[need.Judgement]
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s waits for your yes on %s", module, node),
Explanation: fmt.Sprintf("%s is recorded: a person takes each of its moves. The next one on %s (version %s) %s. "+
"The node is told to apply it only after you say yes.", module, node, short(fp), what),
Options: []asks.Option{
{ID: OptionYes, Label: "Yes, move it", Does: "the node is told to apply the new composition of " + module, Level: asks.Acknowledge},
{ID: OptionNo, Label: "No, not now", Does: "the node keeps what it runs", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the node keeps what it runs",
About: key,
}}
}
// NextBackupAsking is the number of the asking that follows the one numbered after (a negative number: none yet). Numbers 0 and 1
// give one id (delivery.AskingID), so the first asking is 0 and the next 2.
func NextBackupAsking(after int) int {
if after < 0 {
return 0
}
return max(after+1, 2)
}
var backupDigest = regexp.MustCompile(`\(composition (sha256:[0-9a-f]{64})\)`)
// BackupAskDigest is the digest of the composition a backup ask was made about, from its explanation; empty for an ask that names
// none. An answer is taken for the composition of the ask it answers, and a new revision at the same fingerprint is asked again
// under a new number (ADR 0539 §16).
func BackupAskDigest(a delivery.AssignerAsk) string {
if m := backupDigest.FindStringSubmatch(a.Ask.Explanation); m != nil {
return m[1]
}
return ""
}
// BackupAnswerWords are the words of a want carrying the answer to the asking-th ask: each asking's answer is another want
// value, so a backup that fails again after "back up again" and is answered "again" again changes the key and the node retries.
func BackupAnswerWords(option string, asking int) string {
return fmt.Sprintf("the operator answered %q to the backup ask (asking %d)", option, asking+1)
}
// BackupAsk is the ask about a backup of a module on node that failed before an apply (ADR 0539 §16): back up again, or go on
// without a backup. digest is the composition about to be applied and asking the number of the asking, 0 the first; words are the
// backup's own words, from the node's have.
func BackupAsk(key, node, module, fp, digest, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("The backup of %s failed on %s", module, node),
Explanation: fmt.Sprintf("The backup before applying %s at %s on %s failed: %s. %s keeps running what it has until you answer. (composition %s)",
module, short(fp), node, words, module, digest),
Options: []asks.Option{
{ID: OptionBackupAgain, Label: "Back up again", Does: "the node tries the backup again before it applies", Level: asks.Acknowledge},
{ID: OptionBackupSkip, Label: "Go on without a backup", Does: "the node applies the new version without a backup", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module keeps running what it has",
About: key,
}}
}
// BackupCondition is the condition raised when a backup before an apply fails and the module is held on the node (ADR 0539 §16),
// naming the node, the module and the backup's words; cleared under the same key.
func BackupCondition(node, module, fp, words string, raised bool) asks.Condition {
key := delivery.AssignerModule + ".backup." + node + "." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
headline := fmt.Sprintf("The backup of %s failed on %s", module, node)
needs := "answer whether to back up again or go on without a backup"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: asks.NeedsYou + needs + ". The backup before applying " + module + " at " + short(fp) + " on " + node + " failed: " + words + "."}
}
@@ -0,0 +1,203 @@
package assign
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at
// its key, of a kind the view may answer.
func TestTheHoldAskIsValid(t *testing.T) {
key, err := delivery.HoldAskKey("mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := HoldAsk(key, "mail", fNew, "ace", "health failed, phase running: connection refused", 0, now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
// 1906/H3: the ask names the machine whose check failed and the hold's words.
if !strings.Contains(a.Ask.Explanation, "on ace") || !strings.Contains(a.Ask.Explanation, "connection refused") {
t.Errorf("the ask does not name the node and the hold's words: %q", a.Ask.Explanation)
}
// 1906/H2: an ask written again after a lift is another asking, under an id of its own.
again := HoldAsk(key, "mail", fNew, "ace", "x", 7, now)
if again.Asking != 7 || again.Ask.ID != delivery.AskingID(key, 7) || again.Ask.ID == a.Ask.ID {
t.Errorf("the asking after a lift = %d %s", again.Asking, again.Ask.ID)
}
if _, err := delivery.EncodeAssignerAsk(key, again); err != nil {
t.Errorf("the ask asked again cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
}
// 1906/D3 (ADR 0539 §19): the ask about a data module held on a node is a valid ask of the router and of mesh-sdk's layout at
// its restore key, of a kind the view may answer; it offers restoring (losing the newer writes) and leaving it stopped.
func TestTheRestoreAskIsValid(t *testing.T) {
key, err := delivery.RestoreAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := RestoreAsk(key, "ace", "mail", fNew, "connection refused", now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
var ids []string
for _, o := range a.Ask.Options {
ids = append(ids, o.ID)
}
if len(ids) != 2 || ids[0] != OptionRestore || ids[1] != OptionLeaveStopped {
t.Errorf("options = %v", ids)
}
if !strings.Contains(a.Ask.Explanation, "connection refused") || !strings.Contains(a.Ask.Explanation, "ace") {
t.Errorf("the ask does not name the node and the check's words: %q", a.Ask.Explanation)
}
}
// 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own
// name (ADR 0437 clause 9), and is cleared under the same key.
func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Subject.ID != "mail" || c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") || !strings.Contains(c.Key, "mail") {
t.Errorf("condition = %+v", c)
}
if got := HeldCondition("mail", fNew, "", "", false, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateCleared || c.Key != HeldCondition("mail", fNew, "ace", "x", true, false).Key || !strings.Contains(c.Resolved, "lifted its hold by itself") {
t.Errorf("condition = %+v", c)
}
}
// 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in
// the view, and name the move and the node.
func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
now := time.Now()
rkey, err := delivery.RecordAskKey("ace", "mail", "sha256:"+strings.Repeat("b", 64))
if err != nil {
t.Fatal(err)
}
r := RecordAsk(rkey, "ace", "mail", fNew, RecordNeed{Node: "ace", Digest: "sha256:" + strings.Repeat("b", 64), Judgement: delivery.JudgedCannot, Words: "the composition it runs is gone"}, now)
bkey, err := delivery.BackupAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
dgst := "sha256:" + strings.Repeat("b", 64)
b := BackupAsk(bkey, "ace", "mail", fNew, dgst, "no space left", 0, now)
for key, a := range map[string]delivery.AssignerAsk{rkey: r, bkey: b} {
if err := a.Ask.Check(now); err != nil {
t.Errorf("%s: the router refuses the ask: %v", key, err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("%s: cannot be written: %v", key, err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("%s: the view may not answer it: %v", key, err)
}
}
if !strings.Contains(r.Ask.Explanation, "gone") || r.Ask.Options[0].ID != OptionYes {
t.Errorf("the record ask = %+v", r.Ask)
}
if !strings.Contains(b.Ask.Explanation, "no space left") || b.Ask.Options[0].ID != delivery.BackupAgain || b.Ask.Options[1].ID != delivery.BackupSkip {
t.Errorf("the backup ask = %+v", b.Ask)
}
// the id binds the ask to the composition and to its number (review round 1, items 3 and 4)
if again := BackupAsk(bkey, "ace", "mail", fNew, dgst, "x", NextBackupAsking(0), now); again.Ask.ID == b.Ask.ID {
t.Errorf("asked again under the same id")
}
if NextBackupAsking(-1) != 0 || NextBackupAsking(0) != 2 || NextBackupAsking(2) != 3 {
t.Errorf("the numbers of the askings: %d %d %d", NextBackupAsking(-1), NextBackupAsking(0), NextBackupAsking(2))
}
if BackupAskDigest(b) != dgst {
t.Errorf("the ask names composition %q, want %q", BackupAskDigest(b), dgst)
}
if BackupAnswerWords("again", 0) == BackupAnswerWords("again", 1) {
t.Errorf("two answers of the same option are one want value")
}
c := BackupCondition("ace", "mail", fNew, "no space left", true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Errorf("the backup condition is refused: %v", err)
}
// ADR 0546 clauses 7 and 8: a put's ask is valid at the record ask's key; for the bus its yes approves and binds the act,
// so the view cannot answer it, and it lasts no longer than an approving ask may.
put := delivery.Put{FP: fNew, Why: "the bus's user list", Hold: true}
d := delivery.ComposedDigest{Digest: dgst, Judgement: delivery.JudgedCannot, Words: "the have names no composition"}
for _, bus := range []bool{false, true} {
a, err := PutAsk(rkey, "ace", "mail", put, d, bus, now)
if err != nil {
t.Fatalf("bus %v: %v", bus, err)
}
if err := a.Ask.Check(now); err != nil {
t.Errorf("bus %v: the router refuses the put's ask: %v", bus, err)
}
if _, err := delivery.EncodeAssignerAsk(rkey, a); err != nil {
t.Errorf("bus %v: the put's ask cannot be written: %v", bus, err)
}
viewed := delivery.ViewAnswerFits(delivery.ViewAnswer{Asking: a.Ask.ID, Option: OptionYes}, a)
yes := a.Ask.Options[0]
switch {
case !strings.Contains(a.Ask.Explanation, "the bus's user list"):
t.Errorf("bus %v: the put's ask does not say why: %s", bus, a.Ask.Explanation)
case bus && (yes.Level != asks.Approve || yes.Binds == "" || viewed == nil || a.Ask.Expires.After(now.Add(asks.ApproveLasts))):
t.Errorf("bus: the yes is %+v, the view's answer %v, expiring %s", yes, viewed, a.Ask.Expires)
case !bus && (yes.Level != asks.Acknowledge || viewed != nil):
t.Errorf("a recorded module: the yes is %+v, the view's answer %v", yes, viewed)
}
later := put
later.At = now.Add(time.Minute)
b, _ := PutAsk(rkey, "ace", "mail", later, d, bus, now)
if b.Ask.ID == a.Ask.ID || b.Asking != PutAsking(later) {
t.Errorf("bus %v: a later put's ask is asking %d under id %q, the earlier's %q", bus, b.Asking, b.Ask.ID, a.Ask.ID)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dgst)); err != nil {
t.Errorf("bus %v: the yes does not perform the act it bound: %v", bus, err)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dig(fOld))); bus && err == nil {
t.Errorf("bus: the yes performs another composition's act")
}
}
}
// 1945/AC21: the condition of a module on neither list is urgent, names the module, the node and the rule holding its want back,
// under the assigner's own name keyed by node and module, and is cleared under the same key.
func TestTheNeitherConditionNamesModuleNodeAndRule(t *testing.T) {
c := NeitherCondition("g14", "i3status-rust", Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent || c.Subject.ID != "i3status-rust" || c.Subject.Machine != "g14" ||
!strings.Contains(c.Explanation, "x11-display") || !strings.Contains(c.Explanation, "rule 5") || c.Key != "mesh-delivery-assigner.neither.g14.i3status-rust" {
t.Errorf("condition = %+v", c)
}
if got := NeitherCondition("g14", "i3status-rust", Holding{}, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
@@ -0,0 +1,282 @@
// Package assign is the assigner's judgement, as functions over what it read from the bus: the composition check of a
// change (novox/hq ADR 0446 clauses 1 to 5, ADR 0528 clause 5, issue 1297), the six rules of what each node should
// run (ADR 0429 §6, ADR 0528 clause 3) and the comparison with what the controller sends (ADR 0528 clause 4). It reads
// and writes nothing itself: the program hands it what it watched and writes what it answers.
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// SDKRepository is the repository of mesh-sdk: a group with a member there changes the composition itself.
const SDKRepository = "novox/mesh-sdk"
// NotJudged names, in a composition's warnings, the failures of ADR 0446 clause 1 the assigner cannot judge from the
// bus, because what they need is only in the controller's facts snapshot, which is not on the bus yet (issue 1461;
// the design limit of R4g, issue 1297): said on every verdict, never passed silently.
const NotJudged = "not judged until the facts snapshot is on the bus (issue 1461): a consumer left out of a grant, a " +
"credential left bound elsewhere, a stored setting the changed definition cannot keep, and a new module the " +
"node-engine's validator would refuse"
// Mesh is the mesh as the trunk holds it: every module's manifest by name, the manifests that cannot be read with
// why, and each node's open assignments.
type Mesh struct {
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Assigned map[string][]string
// Digests is every trunk module's manifest digest in the indexer's manifests bucket, readable or not: what a
// composer at another mesh-sdk reads (issue 1622).
Digests map[string]string
}
// Change is a group's change as the indexer's candidate holds it: the manifest of every module it moves or adds, those
// it cannot read and why, the modules it removes, every module defined twice with where, the heads of a mesh-sdk
// member (empty when none), and how many modules it rebuilds.
type Change struct {
Group, Heads string
Manifests map[string]catalogue.Manifest
Unreadable map[string]string
Removed []string
Twice map[string][]string
SDKHead string
Rebuilt int
// Digests is the manifest digest of every module the change moves or adds, readable by the assigner or not, and
// Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623).
Digests map[string]string
Members map[string]string
}
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
// removes, those running a consumer of a provision such a module provides (before or after), and those running a
// provider of a provision such a module requires after the change. A module the change adds that is assigned nowhere
// touches no machine.
func Touched(m Mesh, c Change) []string {
moved := map[string]bool{}
for name := range c.Manifests {
moved[name] = true
}
for name := range c.Unreadable {
moved[name] = true
}
for _, name := range c.Removed {
moved[name] = true
}
provided, required := map[string]bool{}, map[string]bool{}
for name := range moved {
for _, man := range []catalogue.Manifest{m.Manifests[name], c.Manifests[name]} {
for _, o := range man.Provides {
provided[o.Name] = true
}
}
for _, r := range c.Manifests[name].Requires {
required[r] = true
}
}
var out []string
for node, modules := range m.Assigned {
for _, mod := range modules {
man := m.Manifests[mod]
if moved[mod] || slices.ContainsFunc(man.Requires, func(r string) bool { return provided[r] }) ||
slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return required[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// input is the composition's input for the mesh's catalogue and assignments: the nodes by name with what is assigned
// to them, and every seat the assigned modules claim as held there. What the bus does not carry (places on the
// private network, settings, credentials, licences) is left empty, so a trunk composition and a change's are made
// with the same gaps and compared (ADR 0446 clause 1: what was already broken is said and fails nothing).
func input(catalog map[string]catalogue.Manifest, assigned map[string][]string) compose.Input {
in := compose.Input{Catalogue: catalog}
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
for _, mod := range assigned[n] {
for _, cl := range catalog[mod].Claims {
scope := cl.Scope
if seat, ok := catalogue.SeatNamed(cl.Name); ok && scope == "" {
scope = seat.Scope
}
in.Holdings = append(in.Holdings, catalogue.Held{Claim: cl.Name, Scope: scope, Node: n, Module: mod})
}
}
}
return in
}
// Compose is the composition check of change c over the mesh m, composed with the mesh-sdk version sdk: every touched
// machine resolved as the trunk holds it and as the change leaves it. It fails on a manifest of the change that cannot
// be read, a module a machine runs that the change removes, a module defined twice, and a machine that resolves on the
// trunk and not with the change; a machine that does not resolve on either is said and fails nothing. It warns past
// CompositionRebuildWarning modules rebuilt, says what it does not judge (NotJudged), and for a group with a mesh-sdk
// member says it composed with its own mesh-sdk, not the member's head (issue 1621). took is how long it took.
func Compose(m Mesh, c Change, sdk string, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: sdk, Took: took.Milliseconds()}
var fails []string
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Unreadable) {
failAt("", fmt.Sprintf("the manifest of %s cannot be read: %s", name, c.Unreadable[name]))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
machines := Touched(m, c)
v.Machines = machines
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
}
trunk := input(m.Manifests, m.Assigned)
changed := map[string]catalogue.Manifest{}
for name, man := range m.Manifests {
changed[name] = man
}
for name, man := range c.Manifests {
changed[name] = man
}
for _, name := range c.Removed {
delete(changed, name)
}
after := input(changed, m.Assigned)
for _, node := range machines {
_, before := compose.Resolve(trunk, node)
_, now := compose.Resolve(after, node)
switch {
case now == nil:
case before != nil:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %v", node, before))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %v", node, now))
}
}
if c.SDKHead != "" {
v.Warnings = append(v.Warnings, fmt.Sprintf("composed with the assigner's own mesh-sdk %s, not the group's mesh-sdk head %s, "+
"and a field the running controller cannot read is not judged (issues 1621, 1622, 1623)", sdk, c.SDKHead))
}
return finish(v, m, c, machines, fails)
}
// finish adds what every composition says beside its own judgement (more modules rebuilt than CompositionRebuildWarning,
// what is not judged, the machines not composed, a change that touches none) and its state from fails.
func finish(v delivery.CompositionVerdict, m Mesh, c Change, machines []string, fails []string) delivery.CompositionVerdict {
if c.Rebuilt > delivery.CompositionRebuildWarning {
v.Warnings = append(v.Warnings, fmt.Sprintf("the merge rebuilds %d modules, more than %d", c.Rebuilt, delivery.CompositionRebuildWarning))
}
v.Warnings = append(v.Warnings, NotJudged)
if could := CouldRun(m, c, machines); len(could) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines that could run a moved or added module and are not assigned it: "+strings.Join(could, ", "))
}
if grants := GrantsAlone(m, c, machines); len(grants) > 0 {
v.Warnings = append(v.Warnings, "not judged until the facts snapshot is on the bus (issue 1461, ADR 0537 clause 1): "+
"the machines whose grants alone the change alters: "+strings.Join(grants, ", "))
}
if len(machines) == 0 {
v.Warnings = append(v.Warnings, "the change touches no machine: no module it moves is assigned anywhere")
}
if len(fails) > 0 {
v.State, v.Words = delivery.CompositionFail, strings.Join(fails, "; ")
return v
}
v.State = delivery.CompositionPass
return v
}
// CouldRun is, by name, every machine of the mesh not composed that is not assigned a module the change moves or adds
// (ADR 0537 clause 1): which of them could run it, the bus does not say (capabilities, settings and places are only in
// the facts snapshot), so each is named as not judged.
func CouldRun(m Mesh, c Change, composed []string) []string {
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for name := range c.Manifests {
if !slices.Contains(modules, name) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
// grantsOf is what of a manifest makes grants: its bus declarations and the provisions it requires and offers.
func grantsOf(man catalogue.Manifest) string {
var offers []string
for _, o := range man.Provides {
offers = append(offers, o.Name)
}
return fmt.Sprint(man.Requires, offers, man.Reads, man.State, man.Emits, man.Consumes, man.Invokes, man.Claims, man.Grants)
}
// GrantsAlone is, by name, every machine not composed whose grants alone the change alters (ADR 0537 clause 1): for a
// moved module whose grants differ from the trunk's, the machines assigned the bus's holder (whose user list carries
// every module's bus grants) and those assigned a provider of a provision it requires, before or after.
func GrantsAlone(m Mesh, c Change, composed []string) []string {
provisions := map[string]bool{}
altered := false
for name, man := range c.Manifests {
before, had := m.Manifests[name]
if had && grantsOf(before) == grantsOf(man) {
continue
}
altered = true
for _, r := range append(slices.Clone(man.Requires), before.Requires...) {
provisions[r] = true
}
}
if !altered {
return nil
}
var out []string
for node, modules := range m.Assigned {
if slices.Contains(composed, node) {
continue
}
for _, mod := range modules {
man := m.Manifests[mod]
if man.ClaimsSeat("mesh-broker") || slices.ContainsFunc(man.Provides, func(o catalogue.Offer) bool { return provisions[o.Name] }) {
out = append(out, node)
break
}
}
}
sort.Strings(out)
return out
}
func sortedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -0,0 +1,141 @@
package assign
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func meshOf() Mesh {
return Mesh{
Manifests: map[string]catalogue.Manifest{
"mail": {Module: "mail", Version: "1", Requires: []string{"database"}},
"store": {Module: "store", Version: "1", Provides: []catalogue.Offer{{Name: "database"}}},
"web": {Module: "web", Version: "1"},
},
Assigned: map[string][]string{"ace": {"mail", "store"}, "g14": {"web"}},
}
}
func hasWarning(v delivery.CompositionVerdict, part string) bool {
return slices.ContainsFunc(v.Warnings, func(w string) bool { return strings.Contains(w, part) })
}
// 1297/AC1 (its unit half): a change touching one module on one machine composes that machine alone, passes, names
// the machine and the mesh-sdk it composed with, and says what it did not judge.
func TestAChangeOfOneModuleComposesItsMachine(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}}
v := Compose(meshOf(), c, "v0.1.14-x", 0)
if v.State != delivery.CompositionPass || !slices.Equal(v.Machines, []string{"g14"}) || v.SDK != "v0.1.14-x" {
t.Fatalf("composition = %+v, want a pass on g14 alone", v)
}
if !hasWarning(v, "issue 1461") {
t.Errorf("the verdict does not say what it did not judge: %v", v.Warnings)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
}
// 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module.
func TestRemovingAModuleAMachineRunsFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Removed: []string{"web"}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "g14" || !strings.Contains(v.Words, "g14 runs web") {
t.Fatalf("composition = %+v, want a fail naming g14 and web", v)
}
}
// 1297/B3c: a machine that composes on the trunk and not with the change fails it, naming the machine and why; the
// consumer's machine is touched when its provider changes.
func TestAMachineTheChangeBreaksFails(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"store": {Module: "store", Version: "2"}}}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || v.Machine != "ace" || !strings.Contains(v.Words, "database") {
t.Fatalf("composition = %+v, want a fail on ace naming the provision nothing provides", v)
}
}
// 1297/B3c: what was already broken on the trunk is said and fails nothing.
func TestWhatWasBrokenBeforeFailsNothing(t *testing.T) {
m := meshOf()
m.Manifests["store"] = catalogue.Manifest{Module: "store", Version: "1"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"mail": {Module: "mail", Version: "2", Requires: []string{"database"}}}}
v := Compose(m, c, "sdk", 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "on the trunk either") {
t.Fatalf("composition = %+v, want a pass saying ace was already broken", v)
}
}
// 1297/B3c: an unreadable manifest and a module defined twice fail; more than twelve rebuilt modules warn.
func TestUnreadableTwiceAndWide(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Unreadable: map[string]string{"web": "unknown field \"scope\""}, Rebuilt: 13}
v := Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web cannot be read") || !hasWarning(v, "rebuilds 13 modules") {
t.Fatalf("composition = %+v", v)
}
c = Change{Group: "g", Heads: "0123456789ab", Twice: map[string][]string{"web": {"novox/a", "novox/b"}}, Rebuilt: 12}
v = Compose(meshOf(), c, "sdk", 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "web is defined twice") || hasWarning(v, "rebuilds") {
t.Fatalf("composition = %+v", v)
}
}
// 1622's criterion as R4g delivers it: a group with a mesh-sdk member is composed with the assigner's own mesh-sdk, and
// the verdict says so, naming the member's head it did not use; a group without one says nothing of it.
func TestAGroupWithAnSDKMemberSaysWhichSDKComposedIt(t *testing.T) {
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": {Module: "web", Version: "2"}}, SDKHead: "f625d98713e4"}
v := Compose(meshOf(), c, "v0.1.14-own", 0)
if !hasWarning(v, "not the group's mesh-sdk head f625d98713e4") || !hasWarning(v, "v0.1.14-own") || !hasWarning(v, "1623") {
t.Errorf("the verdict does not say it composed with its own mesh-sdk: %v", v.Warnings)
}
c.SDKHead = ""
if v := Compose(meshOf(), c, "v0.1.14-own", 0); hasWarning(v, "mesh-sdk head") {
t.Errorf("a group without a mesh-sdk member says it has one: %v", v.Warnings)
}
}
// 1297/B3a: the machines touched are those running a moved module or a consumer of what it provides; an added module
// assigned nowhere touches none, and the verdict says so.
func TestTouched(t *testing.T) {
m := meshOf()
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace"}) {
t.Errorf("touched by store = %v", got)
}
m.Assigned["g14"] = append(m.Assigned["g14"], "mail")
if got := Touched(m, Change{Manifests: map[string]catalogue.Manifest{"store": m.Manifests["store"]}}); !slices.Equal(got, []string{"ace", "g14"}) {
t.Errorf("touched by store with a consumer on g14 = %v", got)
}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"new": {Module: "new", Version: "1"}}}
if v := Compose(m, c, "sdk", 0); len(v.Machines) != 0 || !hasWarning(v, "touches no machine") || v.State != delivery.CompositionPass {
t.Errorf("an added module assigned nowhere: %+v", v)
}
}
// ADR 0537 clause 1: every verdict names, by machine, those that could run a moved or added module and are not assigned
// it, and those whose grants alone the change alters (the bus's machine, a provider of what the module requires), as not
// judged; shanks, providing what web now requires, is composed and so named in neither. A change that alters no grant
// names none of the second.
func TestTheMachinesNotJudgedAreNamed(t *testing.T) {
m := meshOf()
m.Manifests["bus"] = catalogue.Manifest{Module: "bus", Version: "1", Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: "mesh"}}}
m.Manifests["db"] = catalogue.Manifest{Module: "db", Version: "1", Provides: []catalogue.Offer{{Name: "cache"}}}
m.Assigned["novox"] = []string{"bus"}
m.Assigned["shanks"] = []string{"db"}
c := Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{
"web": {Module: "web", Version: "2", Reads: []string{"mesh-controller.assignments"}, Requires: []string{"cache"}}}}
v := Compose(m, c, "sdk", 0)
if !hasWarning(v, "could run a moved or added module and are not assigned it: ace, novox") || !slices.Contains(v.Machines, "shanks") {
t.Errorf("the machines that could run web are not named: %v", v.Warnings)
}
if !hasWarning(v, "whose grants alone the change alters: novox") {
t.Errorf("the machines whose grants alone change are not named: %v", v.Warnings)
}
c.Manifests["web"] = catalogue.Manifest{Module: "web", Version: "3"}
if v := Compose(meshOf(), c, "sdk", 0); hasWarning(v, "grants alone") {
t.Errorf("a change altering no grant names machines whose grants change: %v", v.Warnings)
}
}
@@ -0,0 +1,241 @@
package assign
import (
"encoding/json"
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The cases of a group's composition at its mesh-sdk head: the trunk as it is, and the trunk with the change.
const (
CaseTrunk = "trunk"
CaseChange = "change"
)
// SDKDir is the directory of mesh-sdk's Go module in its repository, where the composer is built at a head; a reader's
// composer is built at its repository's root, at the mesh-sdk its go.mod pins.
const SDKDir = "go"
// The modules whose mesh-sdk reads the manifests a plan sends (ADR 0482 clause 5): the controller composes and sends
// them, the node-engine applies them. Each is judged at its repository.
var ReaderModules = map[string]string{"mesh-controller": "novox/mesh-controller", "mesh-host": "novox/mesh-host"}
// HeadJob is the compose job of a group with a mesh-sdk member (ADR 0482 clauses 3 and 4, issue 1622): the composer
// built at the member's head composes the machines the change touches, as the trunk holds them and as the change leaves
// them, every manifest named by its digest in the indexer's bucket, the seats held from the claims (the assigner's input).
func HeadJob(m Mesh, c Change) (delivery.ComposeJob, error) {
trunk, change := map[string]composer.Manifest{}, map[string]composer.Manifest{}
for name, digest := range m.Digests {
trunk[name] = composer.Manifest{Digest: digest}
change[name] = composer.Manifest{Digest: digest}
}
for name, digest := range c.Digests {
change[name] = composer.Manifest{Digest: digest}
}
for _, name := range c.Removed {
delete(change, name)
}
req := composer.Request{V: composer.Version, Input: plainInput(m.Assigned), Depth: composer.DepthResolve,
HoldingsFromClaims: true, Machines: Touched(m, c),
Cases: []composer.Case{{Name: CaseTrunk, Manifests: trunk}, {Name: CaseChange, Manifests: change}}}
return job(SDKRepository, c.SDKHead, SDKDir, req, "the mesh-sdk head of "+c.Group)
}
// ReaderJob is the compose job that reads the manifests a group changes with the mesh-sdk a reader module pins at
// commit (ADR 0482 clause 5, issue 1623): no machine is composed, every changed manifest is only read.
func ReaderJob(c Change, module, repository, commit string) (delivery.ComposeJob, error) {
changed := map[string]composer.Manifest{}
for name, digest := range c.Digests {
changed[name] = composer.Manifest{Digest: digest}
}
req := composer.Request{V: composer.Version, Cases: []composer.Case{{Name: CaseChange, Manifests: changed}}}
return job(repository, commit, delivery.RootPath, req, module+" at "+commit)
}
func job(repository, commit, dir string, req composer.Request, reader string) (delivery.ComposeJob, error) {
raw, err := json.Marshal(req)
if err != nil {
return delivery.ComposeJob{}, err
}
return delivery.ComposeJob{Fetch: map[string]any{"repository": repository, "commit": commit}, Dir: dir,
Package: composer.Package, Request: raw, Reader: reader}, nil
}
// plainInput is the composition's input without its catalogue: the nodes by name with what is assigned to them (the
// composer derives the seats held from the claims, as input does).
func plainInput(assigned map[string][]string) compose.Input {
var in compose.Input
nodes := make([]string, 0, len(assigned))
for n := range assigned {
nodes = append(nodes, n)
}
sort.Strings(nodes)
for _, n := range nodes {
in.Nodes = append(in.Nodes, compose.Node{Name: n, Assigned: slices.Clone(assigned[n])})
}
return in
}
// Ran is one compose ask's end: its step's state and reason, and the composer's verdict when the step passed.
type Ran struct {
State string
Reason string
Verdict *composer.Verdict
}
// RanOf reads a compose ask's end from its step value; a step that passed with an answer that cannot be read is a
// failure, named.
func RanOf(step delivery.Step) Ran {
r := Ran{State: step.State, Reason: step.Reason}
if step.State != delivery.StepPassed {
return r
}
var a delivery.ComposeAnswer
var v composer.Verdict
if err := json.Unmarshal(step.Result, &a); err != nil {
return Ran{State: delivery.StepFailed, Reason: "the compose step's answer cannot be read: " + err.Error()}
}
if err := json.Unmarshal(a.Verdict, &v); err != nil || v.V != composer.Version {
return Ran{State: delivery.StepFailed, Reason: fmt.Sprintf("the composer's verdict cannot be read (version %d): %v", v.V, err)}
}
r.Verdict = &v
return r
}
func caseOf(v *composer.Verdict, name string) (composer.CaseVerdict, bool) {
if v == nil {
return composer.CaseVerdict{}, false
}
for _, c := range v.Cases {
if c.Name == name {
return c, true
}
}
return composer.CaseVerdict{}, false
}
// Reader is one module whose mesh-sdk reads the group's manifests, at the commit judged: the group's member head when
// the group holds one in its repository, else a build its node runs.
type Reader struct {
Module, Commit string
Member bool
Ran Ran
}
// FromComposer is the composition check of a group with a mesh-sdk member, judged from the composer built at its head
// (ADR 0482 clauses 3 to 5, issues 1622 and 1623): the same failures as Compose, read from the composer's two cases, the
// verdict naming the mesh-sdk the composer was built with; a trunk manifest the head's mesh-sdk refuses fails too (the
// group would deliver a reader that refuses it); and each reader that refuses a changed manifest the head reads fails
// it, naming the module and its words, unless the reader's mesh-sdk has no composer to ask (before issue 1621), which is
// said and judged nothing.
func FromComposer(m Mesh, c Change, head Ran, readers []Reader, took time.Duration) delivery.CompositionVerdict {
v := delivery.CompositionVerdict{Group: c.Group, Heads: c.Heads, SDK: "mesh-sdk " + c.SDKHead, Took: took.Milliseconds()}
machines := Touched(m, c)
v.Machines = machines
var fails []string
if head.State != delivery.StepPassed {
v.Words = fmt.Sprintf("the composer at mesh-sdk %s did not run: %s %s", c.SDKHead, head.State, head.Reason)
v.State = delivery.CompositionFail
return v
}
if head.Verdict.SDK != "" {
v.SDK = head.Verdict.SDK
}
trunk, okT := caseOf(head.Verdict, CaseTrunk)
change, okC := caseOf(head.Verdict, CaseChange)
if !okT || !okC {
v.State, v.Words = delivery.CompositionFail, "the composer's verdict holds no trunk or no change case"
return v
}
failAt := func(machine, words string) {
if v.Machine == "" && machine != "" {
v.Machine = machine
}
fails = append(fails, words)
}
for _, name := range sortedKeys(c.Digests) {
if r := change.Manifests[name]; !r.Accepted {
failAt("", fmt.Sprintf("the manifest of %s cannot be read by mesh-sdk %s: %s", name, c.SDKHead, r.Refused))
}
}
for _, name := range sortedKeys(trunk.Manifests) {
if _, changed := c.Digests[name]; changed || slices.Contains(c.Removed, name) || trunk.Manifests[name].Accepted {
continue
}
if _, before := m.Unreadable[name]; before {
continue // already unreadable on the trunk: not the group's doing
}
failAt("", fmt.Sprintf("mesh-sdk %s refuses the trunk's manifest of %s, which the group does not change: %s", c.SDKHead, name, trunk.Manifests[name].Refused))
}
for _, name := range sortedKeys(c.Twice) {
failAt("", fmt.Sprintf("%s is defined twice, in %s", name, strings.Join(c.Twice[name], " and ")))
}
for _, node := range machines {
for _, name := range c.Removed {
if slices.Contains(m.Assigned[node], name) {
failAt(node, fmt.Sprintf("%s runs %s, which the change removes from its source", node, name))
}
}
before, now := trunk.Machines[node], change.Machines[node]
switch {
case now.Composes:
case !before.Composes:
v.Warnings = append(v.Warnings, fmt.Sprintf("%s does not compose on the trunk either, so the change is not blamed: %s", node, before.Reason))
default:
failAt(node, fmt.Sprintf("%s composes on the trunk and not with the change: %s", node, now.Reason))
}
}
readerFails, readerWarnings := judgeReaders(c, change, readers)
fails = append(fails, readerFails...)
v.Warnings = append(v.Warnings, readerWarnings...)
return finish(v, m, c, machines, fails)
}
// NoComposer says whether a compose step failed because the mesh-sdk it was built at has no composer command (a mesh-sdk
// from before issue 1621): Go's words for a package its module does not hold.
func NoComposer(reason string) bool {
for _, words := range []string{"cannot find package", "no required module provides package", "is not in std", "cannot find module providing package", "does not contain package"} {
if strings.Contains(reason, words) {
return true
}
}
return false
}
func judgeReaders(c Change, head composer.CaseVerdict, readers []Reader) (fails, warnings []string) {
for _, r := range readers {
at := r.Module + " at " + r.Commit
if r.Member {
at = r.Module + "'s member head " + r.Commit
}
if r.Ran.State != delivery.StepPassed {
if NoComposer(r.Ran.Reason) {
warnings = append(warnings, fmt.Sprintf("not judged whether %s reads the group's manifests: the mesh-sdk it pins has no "+
"composer command (before issue 1621)", at))
continue
}
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins did not run: %s %s", at, r.Ran.State, r.Ran.Reason))
continue
}
read, ok := caseOf(r.Ran.Verdict, CaseChange)
if !ok {
fails = append(fails, fmt.Sprintf("the composer of the mesh-sdk %s pins answered no change case", at))
continue
}
for _, name := range sortedKeys(c.Digests) {
if !head.Manifests[name].Accepted || read.Manifests[name].Accepted {
continue
}
fails = append(fails, fmt.Sprintf("the manifest of %s uses what %s (mesh-sdk %s) cannot read: %s; the group must move %s onto "+
"mesh-sdk %s", name, at, r.Ran.Verdict.SDK, read.Manifests[name].Refused, r.Module, c.SDKHead))
}
}
return fails, warnings
}
@@ -0,0 +1,184 @@
package assign
import (
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
var testTime = time.Date(2026, 10, 24, 3, 0, 0, 0, time.UTC)
const sdkHead = "1111111111111111111111111111111111111111"
// raws are the manifests of meshOf as their repositories hold them, by digest, as the indexer keeps them; `web2` is web
// with a field `shares` that the assigner's own mesh-sdk does not read (the shape of issue 1593).
var raws = map[string]string{
"d-mail": `{"module": "mail", "version": "1", "requires": ["postgres-database"]}`,
"d-store": `{"module": "store", "version": "1", "provides": [{"name": "postgres-database"}]}`,
"d-web": `{"module": "web", "version": "1"}`,
"d-web2": `{"module": "web", "version": "2", "shares": ["photos"]}`,
}
// asTheHeadReads is how mesh-sdk at the group's head reads each manifest: as the assigner's own reads it, but for the
// field it adds, which it accepts. The head's composer is built and run by the Go build module (ADR 0482 clause 4); here
// its verdict is made by mesh-sdk's composer.Run over the same request, with `web2` read as the head reads it.
var asTheHeadReads = map[string]string{"d-web2": `{"module": "web", "version": "2"}`}
func digestedMesh() Mesh {
m := meshOf()
m.Digests = map[string]string{"mail": "d-mail", "store": "d-store", "web": "d-web"}
return m
}
// webShares is a group of mesh-sdk and mesh-catalog whose catalogue member gives web the field only the SDK head reads:
// the assigner's own reader refuses it (Unreadable), as the indexer's manifest is read in world.change.
func webShares(members map[string]string) Change {
_, err := catalogue.ParseManifest([]byte(raws["d-web2"]))
return Change{Group: "g", Heads: "0123456789ab", SDKHead: sdkHead, Manifests: map[string]catalogue.Manifest{},
Unreadable: map[string]string{"web": err.Error()}, Digests: map[string]string{"web": "d-web2"}, Members: members}
}
// run is the composer's step for a job, as the build module answers it: the request filled from the indexer's digests
// (with read standing in for how that mesh-sdk reads a digest), run, and its verdict put in a passed step.
func run(t *testing.T, job delivery.ComposeJob, read map[string]string) Ran {
t.Helper()
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if err := composer.Fill(&req, func(d string) ([]byte, error) {
if r, ok := read[d]; ok {
return []byte(r), nil
}
return []byte(raws[d]), nil
}); err != nil {
t.Fatal(err)
}
v, err := composer.Run(req)
if err != nil {
t.Fatal(err)
}
v.SDK = "(devel) " + job.Fetch["commit"].(string)
vr, _ := json.Marshal(v)
ans, _ := json.Marshal(delivery.ComposeAnswer{Built: true, Key: "composer-0", Verdict: vr})
return RanOf(delivery.Step{State: delivery.StepPassed, Result: ans})
}
// 1622/AC1: a group holding a mesh-sdk pull request is composed by the composer built at the member's head: the job
// fetches mesh-sdk at that head, builds mesh-sdk's composer command in its Go module, and hands it the trunk and the
// change by digest, never the manifests' bytes, composing the machines the change touches.
func TestAGroupWithAnSDKMemberAsksTheComposerAtItsHead(t *testing.T) {
c := webShares(nil)
job, err := HeadJob(digestedMesh(), c)
if err != nil {
t.Fatal(err)
}
if job.Fetch["repository"] != SDKRepository || job.Fetch["commit"] != sdkHead || job.Dir != "go" || job.Package != composer.Package {
t.Fatalf("job = %+v", job)
}
ask := delivery.StepAsk{ID: "mh3k2x9qa1b2c5", Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: "mesh-delivery-assigner",
At: testTime, Compose: &job}
if _, err := delivery.EncodeStepAsk(ask); err != nil {
t.Fatalf("the ask is refused by its layout: %v", err)
}
req, err := composer.Decode(job.Request)
if err != nil {
t.Fatal(err)
}
if len(req.Cases) != 2 || req.Cases[0].Name != CaseTrunk || req.Cases[1].Name != CaseChange {
t.Fatalf("cases = %+v", req.Cases)
}
if req.Cases[0].Manifests["web"].Digest != "d-web" || req.Cases[1].Manifests["web"].Digest != "d-web2" || len(req.Cases[1].Manifests["web"].Raw) != 0 {
t.Errorf("the cases do not name web by its trunk and its change digest: %+v", req.Cases)
}
if !slices.Equal(req.Machines, []string{"g14"}) || req.Depth != composer.DepthResolve || !req.HoldingsFromClaims {
t.Errorf("request = machines %v, depth %q, holdings from claims %v", req.Machines, req.Depth, req.HoldingsFromClaims)
}
}
// 1622/AC2, AC3: a catalogue member using a field only the SDK head reads passes with the head's composer, and the
// verdict names the mesh-sdk the composer was built with; with the assigner's own mesh-sdk it fails.
func TestAFieldTheGroupAddsPassesWithTheHeadsComposer(t *testing.T) {
c := webShares(map[string]string{"novox/mesh-controller": "2222222222222222222222222222222222222222"})
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
v := FromComposer(digestedMesh(), c, head, nil, 0)
if v.State != delivery.CompositionPass || v.SDK != "(devel) "+sdkHead || !slices.Equal(v.Machines, []string{"g14"}) {
t.Fatalf("with the head's composer: %+v", v)
}
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
own := Compose(digestedMesh(), c, "v0.1.14-own", 0)
if own.State != delivery.CompositionFail || !strings.Contains(own.Words, "shares") {
t.Fatalf("with the assigner's own mesh-sdk: %+v, want a fail naming the field", own)
}
}
// 1623/AC1, AC2: the group of mesh-sdk and mesh-catalog adding and using a field, without a controller member, fails
// naming the field and mesh-controller, whose running build pins an older mesh-sdk; with a controller member pinned to
// the SDK head (its composer reads what the head reads) it passes.
func TestAFieldTheRunningControllerCannotReadFailsTheGroup(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
running := "3333333333333333333333333333333333333333"
rjob, err := ReaderJob(c, "mesh-controller", "novox/mesh-controller", running)
if err != nil {
t.Fatal(err)
}
if rjob.Fetch["commit"] != running || rjob.Dir != delivery.RootPath {
t.Fatalf("reader job = %+v", rjob)
}
old := run(t, rjob, nil) // the running controller's mesh-sdk reads web2 as the assigner's own does: refused
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: running, Ran: old}}, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "shares") || !strings.Contains(v.Words, "move mesh-controller onto mesh-sdk "+sdkHead) {
t.Fatalf("without a controller member: %+v, want a fail naming the field and mesh-controller", v)
}
member := "2222222222222222222222222222222222222222"
pinned := run(t, rjob, asTheHeadReads)
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-controller", Commit: member, Member: true, Ran: pinned}}, 0)
if v.State != delivery.CompositionPass {
t.Fatalf("with a controller member pinned to the head: %+v, want a pass", v)
}
}
// 1623: a reader whose mesh-sdk has no composer command (before issue 1621) is said, not failed; a reader's composer that
// failed for another cause fails the group, loud.
func TestAReaderWithNoComposerIsSaidAndAnotherFailureFails(t *testing.T) {
c := webShares(nil)
job, _ := HeadJob(digestedMesh(), c)
head := run(t, job, asTheHeadReads)
none := Ran{State: delivery.StepFailed, Reason: "go build: no required module provides package git.novox.be/novox/mesh-sdk/go/cmd/mesh-compose"}
v := FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: none}}, 0)
if v.State != delivery.CompositionPass || !hasWarning(v, "mesh-host at abc") || !hasWarning(v, "before issue 1621") {
t.Fatalf("a reader with no composer: %+v", v)
}
broke := Ran{State: delivery.StepFailed, Reason: "the fetch could not reach the forge"}
v = FromComposer(digestedMesh(), c, head, []Reader{{Module: "mesh-host", Commit: "abc", Ran: broke}}, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "could not reach the forge") {
t.Fatalf("a reader whose composer failed: %+v", v)
}
}
// 1622: the head's composer that did not run fails the composition naming why, never passes it; a trunk manifest the
// head's mesh-sdk refuses fails it too, since the group would deliver a reader that refuses it.
func TestTheHeadsComposerFailingOrRefusingTheTrunkFails(t *testing.T) {
c := webShares(nil)
v := FromComposer(digestedMesh(), c, Ran{State: delivery.StepRefused, Reason: "the guard refuses it"}, nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "the guard refuses it") || !strings.Contains(v.Words, sdkHead) {
t.Fatalf("a head composer that did not run: %+v", v)
}
job, _ := HeadJob(digestedMesh(), c)
strict := map[string]string{"d-web2": asTheHeadReads["d-web2"], "d-mail": `{"module": "mail", "version": "1", "gone": true}`}
v = FromComposer(digestedMesh(), c, run(t, job, strict), nil, 0)
if v.State != delivery.CompositionFail || !strings.Contains(v.Words, "refuses the trunk's manifest of mail") {
t.Fatalf("a head that refuses a trunk manifest: %+v", v)
}
}
@@ -0,0 +1,252 @@
package assign
import (
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A put (novox/hq ADR 0546): one version of a module put on one node by the assigner's tool assigner_put. The tool writes the put
// key; each pass takes it one step further from what the world holds (StepPut), and the rules pass over the node's want for
// the module from its asking until it ends (Module.Puts).
// ComposeWithin is how long a put waits for the composition and grant it needs (a take-in of a build just done writes them)
// before it is refused.
const ComposeWithin = 10 * time.Minute
// PutWorld is what one step of a put reads.
type PutWorld struct {
Node, Module string
Put delivery.Put
// Assigned says the module stands assigned to the node.
Assigned bool
// Bus says the module provides the mesh's bus (ADR 0539 §15); Recorded that it is of policy record (§14).
Bus, Recorded bool
// Build is the state of the builder's build at the put's fingerprint, "" when no key is written; BuildWords its error.
Build, BuildWords string
// Newest is the module's newest fingerprint at the trunk, which the builder builds by itself.
Newest string
// Composed is the composition of the module for the node at the fingerprint, as its writer holds it; nil when there is
// none. Granted says the grant at the fingerprint names the node's accounts. Both are read, never asked for (clause 4).
Composed *delivery.ComposedDigest
Granted bool
// Answer is the option the operator chose on this put's own ask (its asking, PutAsking), "" when none; AskExpired says
// that ask expired unanswered.
Answer string
AskExpired bool
// Snapshot says where the bus's streams were snapshotted for this put, once they were; Snapshotting that it was
// asked; SnapshotFailed why it was not taken.
Snapshot string
Snapshotting bool
SnapshotFailed string
// Have is what the node says it runs of the module; nil when it says nothing.
Have *delivery.Have
Now time.Time
}
// PutNext is what a step decides: the put as it stands after it, and the acts the pass does for it.
type PutNext struct {
Put delivery.Put
// Ask writes the put's ask about the move (PutAsk).
Ask bool
// Snapshot has the bus's node back the bus up now, its dump the streams' snapshot (ADR 0235).
Snapshot bool
// Warranted records, once, in the controller's hand-act log that the bus's want was written on the operator's warrant.
Warranted bool
}
// StepPut takes a put one step further from what the world holds (ADR 0546 clauses 3 to 8).
func StepPut(w PutWorld) PutNext {
p := w.Put
if delivery.PutEnded(p.Step) {
return PutNext{Put: p}
}
say := func(step, format string, args ...any) PutNext {
return PutNext{Put: p.Said(w.Now, step, fmt.Sprintf(format, args...))}
}
fp := short(p.FP)
if !w.Assigned {
return say(delivery.PutRefused, "%s is not assigned to %s: a put moves a module where it stands assigned (mesh-controller.assign first)", w.Module, w.Node)
}
switch w.Build {
case delivery.BuildDone:
case delivery.BuildFailed:
return say(delivery.PutRefused, "the builder's build of %s at %s failed: %s", w.Module, fp, w.BuildWords)
case "":
if p.FP != w.Newest {
// The builder builds the trunk's newest fingerprint and the candidates', and takes no ask for another yet (ADR 0546
// clause 3): an older fingerprint it never built, or whose key it removed, cannot be put.
return say(delivery.PutRefused, "the builder holds no build of %s at %s, and builds by itself only the trunk's newest (%s): put that, or a version it built", w.Module, fp, short(w.Newest))
}
return say(delivery.PutBuild, "waits for the builder's build of %s at %s, the trunk's newest, which it builds by itself", w.Module, fp)
default:
return say(delivery.PutBuild, "waits for the builder's build of %s at %s, which is %s", w.Module, fp, w.Build)
}
if w.Composed == nil || !w.Granted {
missing := fmt.Sprintf("no composition of %s for %s at %s is written", w.Module, w.Node, fp)
if w.Composed != nil {
missing = fmt.Sprintf("the grant of %s at %s does not name %s's accounts", w.Module, fp, w.Node)
}
since := w.Now
if p.Step == delivery.PutCompose && len(p.Steps) > 0 {
since = p.Steps[len(p.Steps)-1].At
}
if w.Now.Sub(since) > ComposeWithin {
return say(delivery.PutRefused, "%s after %s: only a composition written by the take-in of its build (or one of the five kept) is put; "+
"for another the hand path is mesh-hand-compose (ADR 0539 §10)", missing, ComposeWithin)
}
return say(delivery.PutCompose, "waits up to %s: %s yet", ComposeWithin, missing)
}
d := *w.Composed
if p.Step == delivery.PutWanted || p.Step == delivery.PutHolding {
return PutNext{Put: applied(w, p)}
}
p.Composed = d.Digest
if needsPerson(w, d) {
switch w.Answer {
case OptionNo:
return say(delivery.PutRefused, "the operator answered no to putting %s at %s on %s", w.Module, fp, w.Node)
case OptionYes:
default:
if w.AskExpired {
return say(delivery.PutRefused, "the operator's ask about putting %s at %s on %s expired unanswered: the node keeps what it runs", w.Module, fp, w.Node)
}
next := say(delivery.PutPerson, "waits for the operator's yes to putting %s at %s on %s (composition %s, judged %s)", w.Module, fp, w.Node, short(d.Digest), d.Judgement)
if p.Ask == "" {
next.Ask = true
next.Put.Ask, _ = delivery.RecordAskKey(w.Node, w.Module, d.Digest)
}
return next
}
if w.Bus {
switch {
case w.Snapshot != "":
p.Snapshot = w.Snapshot
case w.SnapshotFailed != "":
return say(delivery.PutRefused, "the bus's streams were not snapshotted, so the bus is not put: %s", w.SnapshotFailed)
case w.Snapshotting:
return say(delivery.PutSnapshot, "the operator said yes; waits for %s's backup holder to snapshot the bus's streams", w.Node)
default:
next := say(delivery.PutSnapshot, "the operator said yes; asked %s's backup holder to snapshot the bus's streams", w.Node)
next.Snapshot = true
return next
}
}
}
words := fmt.Sprintf("the want of %s on %s names %s (composition %s)", w.Module, w.Node, fp, short(d.Digest))
if p.Hold {
words += ", held until released"
}
if p.Snapshot != "" {
words += "; the streams were snapshotted: " + p.Snapshot
}
next := PutNext{Put: applied(w, p.Said(w.Now, delivery.PutWanted, words))}
next.Warranted = w.Bus && p.Ask != ""
return next
}
// applied is a wanted put once the node's have says how it went: holding or done when it runs the composition healthy.
func applied(w PutWorld, p delivery.Put) delivery.Put {
h := w.Have
if h == nil || h.FP != p.FP || h.Composed != p.Composed {
return p
}
switch {
case h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK && p.Hold:
return p.Said(w.Now, delivery.PutHolding, fmt.Sprintf("%s runs %s at %s healthy; held there until released", w.Node, w.Module, short(p.FP)))
case h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK:
return p.Said(w.Now, delivery.PutDone, fmt.Sprintf("%s runs %s at %s healthy; the rules write its want again", w.Node, w.Module, short(p.FP)))
case failed(*h):
return p.Said(w.Now, p.Step, fmt.Sprintf("%s runs %s at %s and its health check failed: %s", w.Node, w.Module, short(p.FP), h.Detail))
}
return p
}
// needsPerson says whether the put's move waits for the operator: any move of the bus but a revision at the fingerprint the
// node runs that changes only its user list (ADR 0539 §7, §15), and a move of a module of policy record judged to recreate a
// container, move data, or that cannot be judged (§14).
func needsPerson(w PutWorld, d delivery.ComposedDigest) bool {
runs := w.Have != nil && w.Have.FP == w.Put.FP
if runs && w.Have.Composed == d.Digest {
return false // the node runs this composition already: nothing moves
}
if w.Bus {
return !(runs && d.UserListOnly)
}
if !w.Recorded {
return false
}
return d.Judgement == delivery.JudgedRecreates || d.Judgement == delivery.JudgedMovesData || d.Judgement == delivery.JudgedCannot
}
// PutWant is the want a put writes once it is wanted: the fingerprint and composition, marked put, and held when it holds.
func PutWant(p delivery.Put) delivery.Want {
return delivery.Want{FP: p.FP, Composed: p.Composed, Put: true, Hold: p.Hold, Words: "put: " + p.Why}
}
// PutOwns says what a put does to the node's want: owns says the rules pass over it, want the want it writes (nil: what
// stands stays). A put owns the want from its asking until it ends; a held put ends only when released.
func PutOwns(p delivery.Put) (want *delivery.Want, owns bool) {
switch p.Step {
case delivery.PutWanted, delivery.PutHolding:
w := PutWant(p)
return &w, true
case delivery.PutDone, delivery.PutReleased, delivery.PutRefused:
return nil, false
}
return nil, true
}
// PutAsking is a put's own asking number: when it was asked, in milliseconds. Each put asked under the same key is another
// asking, so the ask is written again under an id of its own and an answer to an earlier put, whose warrant names that put's
// asking, never counts for a later one (ADR 0546 clause 8: one approval binds one act).
func PutAsking(p delivery.Put) int {
if n := p.At.UnixMilli(); n > 0 {
return int(n)
}
return 0
}
// PutAct is the act a bus put's yes binds and that is performed when its want is written.
func PutAct(node, module, fp, composed string) asks.Act {
return asks.Act{"put": "want", "node": node, "module": module, "fp": fp, "composed": composed}
}
// PutAsk is the ask about a put's move (ADR 0546 clauses 7 and 8), under the record ask's key of the composition so the
// teller relays it and its answer as any record ask's. For the bus its yes is of level approve and binds the act, so only a
// channel that proves who answered takes it (ADR 0259), never the view.
func PutAsk(key, node, module string, p delivery.Put, d delivery.ComposedDigest, bus bool, now time.Time) (delivery.AssignerAsk, error) {
yes := asks.Option{ID: OptionYes, Label: "Yes, put it", Does: "the node is told to apply " + module + " at " + short(p.FP), Level: asks.Acknowledge}
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + d.Words,
delivery.JudgedNeither: "recreates no container and moves no data",
delivery.JudgedUnchanged: "changes nothing the node runs",
}[d.Judgement]
explanation := fmt.Sprintf("Asked to put %s at version %s on %s: %s. The move %s.", module, short(p.FP), node, p.Why, what)
expires := now.Add(7 * 24 * time.Hour)
if bus {
binds, err := asks.ActDigest(PutAct(node, module, p.FP, d.Digest))
if err != nil {
return delivery.AssignerAsk{}, err
}
yes.Level, yes.Binds = asks.Approve, binds
yes.Does = "the bus's streams are snapshotted, then " + node + " is told to apply the bus at " + short(p.FP)
explanation += " It is the mesh's bus: every module reconnects while it restarts, and its streams are snapshotted first."
expires = now.Add(asks.ApproveLasts)
}
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, PutAsking(p)),
Headline: fmt.Sprintf("Put %s on %s?", module, node),
Explanation: explanation,
Options: []asks.Option{yes,
{ID: OptionNo, Label: "No", Does: "the node keeps what it runs, and the put is refused", Level: asks.Acknowledge}},
Who: asks.Operator,
Expires: expires,
OnExpiry: "the node keeps what it runs",
About: key,
}, Asking: PutAsking(p)}, nil
}
@@ -0,0 +1,522 @@
package assign
import (
"fmt"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
// trunk (the indexer's modules), whether the builder's build of F is done, whether F is held, whether the controller
// issued its credential for F, its manifest at F, and the nodes it is assigned to.
type Module struct {
Name string
FP string
Built bool
Held bool
Granted bool
// GrantedAt says whether the controller issued the module's credential for another fingerprint than FP: the first want of a node
// that runs an older one names it (issue 1945). Nil: none.
GrantedAt func(fp string) bool
// GrantedOn says whether the controller's grant of FP names the module's accounts on a node, when the grant names any:
// a grant written for the module's other nodes says nothing of a node whose account is not issued and sealed (issue
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone.
GrantedOn func(node string) bool
Manifest catalogue.Manifest
Nodes []string
// Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its
// digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool)
// Allowed says whether the operator answered yes to the record ask about the move of this module on node to the composition
// with this digest (ADR 0539 §14). Nil: none did.
Allowed func(node, digest string) bool
// Released is a hold on FP the operator released (ADR 0539 §19): FP is not held again by a health check that failed.
Released bool
// Restored says whether the operator answered "restore" to the restore ask of this data module on node: it is then set back
// as any module is, losing the writes made since (ADR 0539 §19). Nil: none did.
Restored func(node string) bool
// Kept says the operator answered "keep" to the hold ask of FP: a hold kept so is not lifted by itself.
Kept bool
// Now is when this pass reads the haves: the held record names it as when the health check was seen to fail. Zero: not said.
Now time.Time
// Puts are the puts of the module, by node (ADR 0546): while one owns a node's want (PutOwns), the rules pass over it.
Puts map[string]delivery.Put
}
func (m Module) composition(node, fp string) (delivery.ComposedDigest, bool) {
if m.Composed == nil {
return delivery.ComposedDigest{}, false
}
return m.Composed(node, fp)
}
// Node is what the rules read of the nodes: whether each is online (the watchman's presence) and each module's have.
type Nodes struct {
Online map[string]bool
Haves map[string]map[string]delivery.Have
// Providers is, by provision, the modules that provide it, with the nodes each is assigned to.
Providers map[string]map[string][]string
}
// healthy says whether a have runs fp healthy.
func healthy(h delivery.Have, fp string) bool {
return h.FP == fp && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
// Canary is the module's canary: the first of its nodes, by name, that is online (ADR 0435 §3; one ring, ordered by
// name, ADR 0528 clause 3). Empty when none is online.
func Canary(nodes []string, online map[string]bool) string {
sorted := slices.Sorted(slices.Values(nodes))
for _, n := range sorted {
if online[n] {
return n
}
}
return ""
}
// Holds is whether every rule holds for moving node's want for m to m.FP, and if not, the first rule that does not,
// with why (ADR 0429 §6). Rule 4, at most `step` others ahead, always holds: the mesh keeps no step, and step 0 is no
// limit (ADR 0528 clause 3).
func Holds(m Module, node string, n Nodes) (bool, int, string) {
return holds(m, node, n, true)
}
// holds is Holds; with ordering false it leaves out the two rules that order a move among nodes and modules, the canary (3)
// and the providers (5), and keeps those without which there is nothing to want: the build done, not held, the grant and the
// composition.
func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
switch {
case !m.Built:
return false, delivery.RuleBuildDone, fmt.Sprintf("the build of %s at %s is not done", m.Name, short(m.FP))
case m.Held:
return false, delivery.RuleNotHeld, fmt.Sprintf("%s is held at %s", m.Name, short(m.FP))
}
if ordering {
canary := Canary(m.Nodes, n.Online)
switch {
case canary == "":
return false, delivery.RuleCanary, fmt.Sprintf("none of the nodes of %s is online, so it has no canary", m.Name)
case node != canary && !healthy(n.Haves[canary][m.Name], m.FP):
return false, delivery.RuleCanary, fmt.Sprintf("the canary %s does not run %s healthy yet", canary, short(m.FP))
}
for _, provision := range m.Manifest.Requires {
if ok, why := providedHealthy(provision, node, n); !ok {
return false, delivery.RuleProviders, why
}
}
}
if !m.Granted {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s for %s", m.Name, short(m.FP))
}
if m.GrantedOn != nil && !m.GrantedOn(node) {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s on %s for %s", m.Name, node,
short(m.FP))
}
if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP))
}
return true, 0, ""
}
// providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned
// elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node
// offers it) and holds.
func providedHealthy(provision, node string, n Nodes) (bool, string) {
providers := n.Providers[provision]
if len(providers) == 0 {
return true, ""
}
names := make([]string, 0, len(providers))
for p := range providers {
names = append(names, p)
}
sort.Strings(names)
for _, p := range names {
nodes := providers[p]
at := node
if !slices.Contains(nodes, node) {
at = Canary(nodes, n.Online)
}
if h, ok := n.Haves[at][p]; ok && at != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return true, ""
}
}
return false, fmt.Sprintf("no provider of %s runs healthy on %s or at its canary", provision, node)
}
// keepsData says whether a module holds data a set-back would lose: an own item of a class that keeps (irreplaceable,
// valuable or rebuildable; cache and none do not).
func keepsData(m catalogue.Manifest) bool {
for _, it := range m.DataItems() {
if it.Class == catalogue.ClassIrreplaceable || it.Class == catalogue.ClassValuable || it.Class == catalogue.ClassRebuildable {
return true
}
}
return false
}
// Outcome is what one module's pass decides: each node's want to write (only those that change), and a hold to write.
type Outcome struct {
Wants map[string]delivery.Want
// Held is the hold of m.FP to write, when a node's health check failed on it and it is not held yet.
Held *delivery.Held
// Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held
// record and its ask are taken away, and the wants move by the rules again in this pass.
Lifted bool
// Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14).
Record []RecordNeed
// Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set
// back: the operator is asked whether to restore each (ADR 0539 §19).
Restore []string
// Rules is, by node, the rule holding the node back from m.FP and why.
Rules map[string]Holding
// Neither is, by node, the nodes the module is assigned to that run it (or whose have names it), hold no want for it after this
// pass and are not held (a held module is on the held list), with the rule holding the first want back: the node's own composition does not hold a module with a fingerprint of the new form, so such a module is on
// neither list and nothing applies it on that node (issue 1945). A module with no fingerprint of the new form goes with the node's
// own composition and is never here.
Neither map[string]Holding
}
// RecordNeed is a move of a module of policy record on a node that waits for the operator's yes: the composition's digest, the
// controller's judgement of the move and its words.
type RecordNeed struct {
Node string
Digest string
Judgement delivery.Judgement
Words string
}
// Holding is a rule holding a node back, and why.
type Holding struct {
Rule int
Words string
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
// before (wants, by node); lastHealthy is, by node, the newest fingerprint other than m.FP the node ran healthy. A failed
// health check of m.FP on a node holds it: every node wanting m.FP is set back to its last healthy fingerprint, but a
// module holding data is not set back but asked about (Restore). A
// node that wants m but is no longer among its nodes gets its want written removed, never deleted (ADR 0408 §4).
func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[string]string) Outcome {
out := Outcome{Wants: map[string]delivery.Want{}, Rules: map[string]Holding{}, Neither: map[string]Holding{}}
held := m.Held
if held && !m.Kept && m.FP != "" && allRunHealthy(m, n) {
out.Lifted, held = true, false
}
if !held && !out.Lifted && !m.Released && m.FP != "" {
for _, node := range slices.Sorted(slices.Values(m.Nodes)) {
if h, ok := n.Haves[node][m.Name]; ok && h.FP == m.FP && failed(h) {
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
}
}
}
mm := m
mm.Held = held
for _, node := range m.Nodes {
if m.FP == "" {
break
}
before, had := wants[node]
var next delivery.Want
p, put := m.Puts[node]
if !put && had && before.Put && before.Hold {
// The want is the truth (ADR 0546 clause 5): a want marked put and held stands while its put is not heard (the
// puts not open, or not yet handed after a start); only a put heard released gives it back to the rules. A put
// without hold is not kept this way: its put takes it further once heard.
p, put = delivery.Put{FP: before.FP, Composed: before.Composed, Hold: true, Why: strings.TrimPrefix(before.Words, "put: "),
Step: delivery.PutHolding}, true
}
if put {
if want, owns := PutOwns(p); owns {
if want != nil {
w := *want
// A held bus still takes a revision of its composition that changes only its user list (ADR 0539 §7).
if ul, ok := m.userListRevision(node, before, had, n); ok && ul.FP == w.FP {
w.Composed = ul.Composed
}
if !had || !delivery.SameWant(normal(before), normal(w)) {
out.Wants[node] = w
}
}
continue
}
}
// A revision of the bus module's composition that differs only in its user list is written at once, at the fingerprint the
// node runs, outside the rules, the record hold and the bar on a new fingerprint: a new module's credential must reach the
// bus before the module can connect (ADR 0539 §7). The controller marks it; whatever it judges of the file's mount stands.
if w, ok := m.userListRevision(node, before, had, n); ok {
if !had || !delivery.SameWant(normal(before), normal(w)) {
out.Wants[node] = w
}
continue
}
ok, rule, words := Holds(mm, node, n)
target := m.FP
if have, ran := n.Haves[node][m.Name]; !ok && (!had || before.Removed) && (rule == delivery.RuleCanary || rule == delivery.RuleProviders) && runs(have) {
// A node that runs the module and has no want for it has no want the ordering rules could hold it back to: holding its first
// want back leaves the module on neither list, since the node's own composition gives up every module with a fingerprint of
// the new form (issue 1945: on g14 the bundles of clipmenu, screen-lock and every module requiring x11-display were removed
// and never made again, because xorg's health is unknown and screen-lock's canary never ran it healthy). Its first want names
// what it runs, never a newer fingerprint, so the canary is never passed over: F when it runs F (by the rules without which
// there is nothing to want), or an older fingerprint of the new form whose composition and grant exist. Rules 3 and 5 then
// order the move from it as from any want.
switch _, composed := m.composition(node, have.FP); {
case !ran:
case have.FP == m.FP:
ok, rule, words = holds(mm, node, n, false)
case !strings.HasPrefix(have.FP, delivery.SourcePrefix2):
// An old-form build (not src2:) has no composition, so no want can ever name it: its first want names F, and only by every
// rule as usual, the canary (3) and the providers (5) included, which here hold it back. Nothing is written and the module
// stays on neither list until the canary runs F healthy; then Holds passes and the want names F (issue 1945, a design
// limit set by the lead: the canary stays in force and old-form modules leave neither list by themselves).
case composed && m.GrantedAt != nil && m.GrantedAt(have.FP):
ok, target = true, have.FP
}
}
switch {
case ok:
d, _ := m.composition(node, target)
next = delivery.Want{FP: target, Composed: d.Digest}
at := m
if target != m.FP {
// the first want at the older fingerprint the node runs: held back from F by the rule, and judged as a want at it
next.Newest, next.Rule, next.Words = m.FP, rule, words
at.FP = target
}
if had && !before.Removed && !before.Put && before.FP == target && before.Composed == d.Digest {
next.Backup, next.Words = before.Backup, before.Words // the answer to a failed backup stays while its target stays (ADR 0539 §16)
}
running, ran := n.Haves[node][m.Name]
// A removed want is no want: a first want after it goes through the record ask as a first want does.
asked, hadLive := before, had
if before.Removed {
asked, hadLive = delivery.Want{}, false
}
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
}
continue
}
case held && had && before.FP == m.FP:
out.Rules[node] = Holding{Rule: rule, Words: words}
last := lastHealthy[node]
switch {
case keepsData(m.Manifest) && (m.Restored == nil || !m.Restored(node)):
next = before
next.Words = fmt.Sprintf("%s holds data, so it is not set back from %s: the operator is asked whether to restore it", m.Name, short(m.FP))
out.Restore = append(out.Restore, node)
case last == "":
next = before
next.Words = fmt.Sprintf("%s is held at %s and %s ran no other fingerprint healthy to set it back to", m.Name, short(m.FP), node)
default:
d, there := m.composition(node, last)
if !there {
next = before
next.Words = fmt.Sprintf("%s is held at %s, and the composition of %s at %s is gone, so %s is not set back", m.Name, short(m.FP), m.Name, short(last), node)
break
}
next = delivery.Want{FP: last, Composed: d.Digest, Newest: m.FP, Rule: rule, Words: words}
}
default:
out.Rules[node] = Holding{Rule: rule, Words: words}
if !had || before.Removed {
// The node runs the module, or its have names it, and gets no want for it: the module is on neither list (issue 1945), and
// the pass says so. A new assignment, with no have, waits for the rules and is on no list yet by design. A held module (rule 2,
// held before or by this pass) is on the held list instead, whose hold is said and answered on its own: it is never on neither
// list (issue 1945: systemd-resolved, held on shanks by the operator, was said as urgent every hour).
h, there := n.Haves[node][m.Name]
switch {
case there && h.Phase != delivery.PhaseRemoved && !held:
out.Neither[node] = Holding{Rule: rule, Words: words}
case had && (!there || h.Phase == delivery.PhaseRemoved):
// A removed want of a node that does not run the module says why no want follows it (issue 1945), so the view's
// assigner_want answers it; it stays removed. A node with no key gets none: a removed want written where none was
// would have its node agent remove the module and stop holding its composition's resources back.
next = delivery.Want{Removed: true, Words: NoWantWords(rule, words)}
if !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
}
continue
}
next = delivery.Want{FP: before.FP, Composed: before.Composed, Newest: m.FP, Rule: rule, Words: words}
if before.FP == m.FP {
// Held at F by the rules: a new revision of F's composition waits for them, as a new fingerprint does (ADR 0539 §5).
next = delivery.Want{FP: m.FP, Composed: before.Composed}
} else if d, there := m.composition(node, before.FP); there && d.Digest != "" && d.Digest != before.Composed {
// Held back at an earlier fingerprint whose kept composition the controller composed again with today's config
// (novox/hq ADR 0545): the want follows its new revision, so the node does not run that build with the config of
// the moment it was made (on g14 a capacity of 0 after a rollback). Asked as any move of a record module is.
at := m
at.FP = before.FP
running, ran := n.Haves[node][m.Name]
need, wait := at.mustAsk(node, before, true, ran, running.FP, running.Composed == "", d)
if need != nil {
out.Record = append(out.Record, *need)
}
if !wait {
next.Composed = d.Digest
}
}
}
if !had || !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
}
for node, before := range wants {
if !slices.Contains(m.Nodes, node) && !before.Removed {
out.Wants[node] = delivery.Want{Removed: true, Words: fmt.Sprintf("%s is no longer assigned to %s", m.Name, node)}
}
}
return out
}
// NoWantWords say why a node gets no want for a module: the rule that holds its first want back and why (issue 1945).
func NoWantWords(rule int, words string) string {
return fmt.Sprintf("no want: rule %d: %s", rule, words)
}
// failed says whether a have's health check failed once the node is done moving: phase running or failed (never removed, held, or
// moving). A failed health while it
// backs up, applies or checks is a step of the move, not its outcome (on novox at 15:27:36 on 2026-10-25, "health failed, phase
// applying" held the controller's canary two seconds before the new build was healthy). Every healthy read (healthy, providedHealthy,
// the world's lastHealthy) already asks for phase running.
func failed(h delivery.Have) bool {
return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed)
}
// runs says whether a have is of a module the node runs: running, whatever its health.
func runs(h delivery.Have) bool {
return h.Phase == delivery.PhaseRunning
}
// allRunHealthy says whether every node of m that runs the module (its have's phase is running, or failed) runs m.FP healthy, and
// at least one does: a hold whose fingerprint runs healthy wherever the module runs holds nothing back any more (issue 1906: seven
// modules stayed held on every machine after one transient failed check). A node set back to another fingerprint does not run
// m.FP, and a failed one does not run it healthy, so the hold stays; a have that is removed, held for its backup or still moving
// (backing up, applying, checking) is not counted, or it would keep the hold up for ever.
func allRunHealthy(m Module, n Nodes) bool {
ran := 0
for _, node := range m.Nodes {
h, ok := n.Haves[node][m.Name]
if !ok || (h.Phase != delivery.PhaseRunning && h.Phase != delivery.PhaseFailed) {
continue
}
if !healthy(h, m.FP) {
return false
}
ran++
}
return ran > 0
}
// HeldWords are the words a hold keeps of its cause (issue 1906): the health and phase the node reported, when the assigner saw
// it, and the node's own words. The node is the held record's own field.
func HeldWords(h delivery.Have, at time.Time) string {
words := fmt.Sprintf("health %s, phase %s", h.Health, h.Phase)
if !at.IsZero() {
words += ", seen at " + at.UTC().Format(time.RFC3339)
}
if d := strings.TrimSpace(h.Detail); d != "" {
return words + ": " + d
}
return words + ": the node gave no words"
}
// LiftedWords say why a hold was lifted by itself.
func LiftedWords(module, fp string) string {
return fmt.Sprintf("every node that runs %s runs %s healthy, so the assigner lifted its hold by itself", module, short(fp))
}
// normal is a want without its version, as a decoded and a composed one are compared.
func normal(w delivery.Want) delivery.Want {
w.V = 0
return w
}
func short(fp string) string {
part, err := delivery.FingerprintKeyPart(fp)
if err != nil || len(part) < 12 {
return fp
}
return part[:12]
}
// mustAsk says whether the move to the composition d on node waits: for the operator's yes (need names the ask) or, for the bus,
// for the operator's hand (ADR 0539 §14, §15). A module the node never ran (it has no want and no have), a move judged neither, a module that is not of
// policy record and a want that stays are written as any want; so is a revision of the bus module's composition at the
// same fingerprint judged neither (a user list only, §7).
func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, running string, haveNamesNoComposition bool, d delivery.ComposedDigest) (*RecordNeed, bool) {
if (!had && !ran) || before.Removed || (before.FP == m.FP && before.Composed == d.Digest) {
return nil, false
}
// Right after the switch no node has a readable want, and a have from the old node-engine names a fingerprint and no composition
// (ADR 0539 §9, §11): a node running this fingerprint is given its want at it with no hold when the controller judges the composition
// unchanged, equal to the declaration the node last applied (ADR 0540 §9), since the node agent then applies nothing. A composition that
// differs from it (a changed setting or secret), or that the controller could not judge, waits for the ask. A new fingerprint still waits.
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
// The fingerprint the bus ran before: its want's, or with no readable want the one its have names (ADR 0539 §15 bars a new one).
prev := before.FP
if !had {
prev = running
}
if catalogue.ProvidesBus(m.Manifest) && prev != m.FP {
return nil, true // the bus is never moved by the assigner; the operator writes its want (§15)
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
if m.Allowed != nil && m.Allowed(node, d.Digest) {
return nil, false
}
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
if d.Policy != "" {
return d.Policy == delivery.PolicyRecord
}
policy, _, _ := catalogue.DerivedUpgrade(m.Manifest)
return policy == catalogue.PolicyRecord
}
// userListRevision is the want that names the bus module's composition at the fingerprint the node runs when that composition
// differs from the one the node runs only in the bus's user list (ComposedDigest.UserListOnly), and whether there is one. A node
// whose want names another fingerprint than the one it runs, or a module that does not provide the bus, has none.
func (m Module) userListRevision(node string, before delivery.Want, had bool, n Nodes) (delivery.Want, bool) {
if !catalogue.ProvidesBus(m.Manifest) {
return delivery.Want{}, false
}
running, ran := n.Haves[node][m.Name]
if !ran || running.FP == "" || (had && (before.Removed || before.FP != running.FP)) {
return delivery.Want{}, false
}
d, ok := m.composition(node, running.FP)
if !ok || !d.UserListOnly {
return delivery.Want{}, false
}
w := delivery.Want{FP: running.FP, Composed: d.Digest}
if had && before.Composed == d.Digest {
w.Backup, w.Words = before.Backup, before.Words
}
return w, true
}
// GrantNamesNode says whether a grant of module names its accounts on node (mesh-sdk's ControllerGrant.NamesNode, which a
// put reads by too).
func GrantNamesNode(g delivery.ControllerGrant, node, module string) bool {
return g.NamesNode(node, module)
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,71 @@
package manifest
import (
"os"
"reflect"
"slices"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
func load(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatalf("mesh-sdk's manifest reader refuses the manifest: %v", err)
}
return m
}
// 1297/B2a: the controller's reader (mesh-sdk catalogue, which the controller reads manifests with) takes the
// manifest; it reads exactly the buckets the assigner opens (the program's list, held equal in its own test), keeps
// the states it declares (the differences kept read-only, 1906/D5), and holds a credential of its own.
func TestManifest(t *testing.T) {
m := load(t)
if m.Module != "mesh-delivery-assigner" {
t.Errorf("module = %q", m.Module)
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
states := map[string]int{}
for _, s := range m.State {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
t.Errorf("own-secrets = %v, want exactly broker (or no credential is issued)", m.OwnSecrets)
}
if m.SecretsOwner != "mesh-delivery-assigner" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1906/D3: it says its conditions on the operator channel; 1622: it puts its compose asks on the Go build seat, as the
// checker puts its checks.
if !slices.Equal(m.Uses, []string{"node-build-go", "operator-channel"}) {
t.Errorf("uses = %v, want exactly node-build-go and operator-channel", m.Uses)
}
// ADR 0546: it serves its own tools (put, release, puts) from its bundle through the tool runner (the artifact loads the
// module), claims no seat (a new seat needs a change of the controller), says each put's step as the event put, and
// calls the controller's warranted and the bus node's backup holder.
if len(m.Claims) != 0 || !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts"}) {
t.Errorf("the assigner claims %v and serves tools %v, want no claim and its three put tools", m.Claims, m.Tools)
}
if want := []string{"seat:mesh-controller.warranted", "seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
if !slices.Equal(m.Emits, []string{"put"}) {
t.Errorf("emits = %v, want put", m.Emits)
}
}
@@ -0,0 +1,95 @@
{
"module": "mesh-delivery-assigner",
"version": "1",
"slug": "assigner",
"reads": [
"mesh-build-go.steps",
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-controller.composed",
"mesh-controller.grants",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"node-engine.have"
],
"uses": [
"node-build-go",
"operator-channel"
],
"invokes": [
"seat:mesh-controller.warranted",
"seat:node-backup.now",
"seat:node-backup.backed-up"
],
"emits": [
"put"
],
"tools": [
"assigner_put",
"assigner_release",
"assigner_puts"
],
"state": [
{ "name": "asks", "history": 5 },
{ "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 }
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-delivery-assigner",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-delivery-assigner",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-delivery-assigner"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-delivery-assigner",
"place": ".",
"unknown": "program"
},
{
"id": "assigner",
"type": "process",
"name": "mesh-delivery-assigner",
"artifact": "program",
"run": [
"./mesh-delivery-assigner"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker"
},
"user": "mesh-delivery-assigner",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "program",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-assigner",
"binary": "mesh-delivery-assigner",
"loads": [
"mesh-delivery-assigner"
]
}
]
}
}
@@ -0,0 +1,178 @@
// Command mesh-delivery-builder-tools serves the builder's tools (novox/hq issue 1945): the runtime launches it for a call.
// builder_stuck reads the builder's builds and reasks states and the build modules' steps through the builder's state and
// reads; builder_ask_again writes one value of the reasks state, which the builder watches and answers. The builder is the
// other program: it serves no tool (ADR 0437 §4).
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// queueAsker writes a queues value and waits for the builder's answer to it.
type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue queueAsker, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{
{Name: build.StuckTool,
Description: "Every build key that waits (building or queued), oldest first: its module, fingerprint, step words and since when, and each ask it " +
"waits on by language with its id, when it was put, its asking (1 for the first; the builder asks an ask again when it can no longer be " +
"answered, at most 3 times) and what its build module's steps hold of it (none, or its state on its node since its start). The " +
"seat workers' delivery counts are not readable by the builder. With module, that module's keys only. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's keys"}}},
Run: func(args map[string]any) (any, error) { return r.Stuck(str(args, "module")) }},
{Name: build.AskAgainTool,
Description: "Ask a waiting build key's asks again now, under new ids the key then names, as the builder does by itself for an ask its build " +
"seat's stream gave up on. The builder answers in the reasks state, which " + build.StuckTool + " shows beside the key; a key no ask of " +
"which waits is answered so and nothing is asked. An ask whose step already runs on a node is asked again too: it is then " +
"built twice, the first run's work wasted and its end read by nobody, never a wrong key; builder_stuck shows whether a step runs.",
Input: map[string]any{"type": "object", "required": []string{"module", "fp", "why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name"},
"fp": map[string]any{"type": "string", "description": "its new-form fingerprint, src2:<hex>, as builder_stuck shows it"},
"why": map[string]any{"type": "string", "description": "why it is asked again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := build.NewReAsk(str(args, "module"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
if err := askAgain(key, value); err != nil {
return nil, fmt.Errorf("the ask could not be written to %s: %w", build.ReAsksBucket, err)
}
return map[string]any{"key": key, "asked": true,
"next": "the builder answers in " + build.ReAsksState + "." + key + "; " + build.StuckTool + " shows the answer and the new ask"}, nil
}},
{Name: build.SeatQueueTool,
Description: "What waits on a build seat's stream (SEAT_NODE_BUILD_<LANGUAGE>), as the builder reads it: the stream's message count and " +
"sequences, and per kind (delivery, check, recheck) its messages oldest first, at most limit each, with sequence, ask id, when asked " +
"and put, age, the state of the step a build node wrote for it, and whether the builder counts it as still waiting or as given up " +
"on (its worker delivered it its most times) or taken; and the ask asked last that a node took. The workers' delivery counts are " +
"not readable by the builder (nats.nats_streams shows them). Read by the builder, which alone may read the stream; read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"language": map[string]any{"type": "string", "description": "go, node or image (default go)"},
"seat": map[string]any{"type": "string", "description": "or the seat: node-build-go, node-build-node, node-build-image"},
"limit": map[string]any{"type": "integer", "description": fmt.Sprintf("messages read of each kind (default %d, at most %d)", build.DefaultQueueLimit, build.MaxQueueLimit)}}},
Run: func(args map[string]any) (any, error) {
language := build.LanguageOf(str(args, "language"))
if language == "" {
language = build.LanguageOf(str(args, "seat"))
}
if language == "" {
language = delivery.AskGo
}
limit, _ := args["limit"].(float64)
key, value, err := build.NewQueueAsk(language, int(limit), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
return queue(key, value)
}},
}
}
// askQueue writes the queues value at key and waits, on a watch of the key, at most bound for the builder's answer.
func askQueue(s stdio.KeptState, bound time.Duration) queueAsker {
return func(key string, value []byte) (json.RawMessage, error) {
answered := make(chan json.RawMessage, 1)
var rev uint64
var mu sync.Mutex
if err := s.Watch(key, func(c stdio.StateChange) error {
mu.Lock()
mine := rev
mu.Unlock()
if c.Op != "put" || mine == 0 || c.Revision <= mine {
return nil
}
if a, err := build.DecodeQueueAsk(c.Value); err == nil && !a.Answered.IsZero() {
select {
case answered <- c.Value:
default:
}
}
return nil
}); err != nil {
return nil, fmt.Errorf("watching %s.%s: %w", build.QueuesState, key, err)
}
r, err := s.Put(key, json.RawMessage(value))
if err != nil {
return nil, fmt.Errorf("the read could not be written to %s: %w", build.QueuesBucket, err)
}
mu.Lock()
rev = r
mu.Unlock()
// The watch may have heard the answer before the revision was known: read it once too.
if e, err := s.Get(key); err == nil && e != nil && e.Revision > r {
if a, err := build.DecodeQueueAsk(e.Value); err == nil && !a.Answered.IsZero() {
return e.Value, nil
}
}
wait, cancel := context.WithTimeout(context.Background(), bound) // a context's bound, no timer (ADR 0437 §8)
defer cancel()
select {
case v := <-answered:
return v, nil
case <-wait.Done():
return nil, fmt.Errorf("the builder did not answer %s.%s within %s: it is not running, or cannot read %s", build.QueuesState, key, bound, build.QueuesBucket)
}
}
}
func read(s stdio.KeptState) build.StateRead {
return build.StateRead{Keys: s.Keys, Get: func(key string) (json.RawMessage, time.Time, error) {
e, err := s.Get(key)
if err != nil || e == nil {
return nil, time.Time{}, err
}
return e.Value, time.Time{}, nil
}, Written: func(key string) time.Time {
e, err := s.Get(key)
if err != nil || e == nil {
return time.Time{}
}
return written(s, key, e.Revision)
}}
}
// written is when the key's revision was written, from its history, or zero.
func written(s stdio.KeptState, key string, revision uint64) time.Time {
h, err := s.History(key)
if err != nil {
return time.Time{}
}
for _, r := range h {
if r.Revision == revision {
return r.At
}
}
return time.Time{}
}
func main() {
steps := map[string]build.StateRead{}
for _, bm := range build.StuckBuildModules {
steps[bm] = read(stdio.State(bm + ".steps"))
}
reasks := stdio.State(build.ReAsksState)
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps}
put := func(key string, value []byte) error {
_, err := reasks.Put(key, json.RawMessage(value))
return err
}
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1)
}
}
@@ -66,6 +66,11 @@ func settle(ctx context.Context, kv jetstream.KeyValue, each func(jetstream.KeyV
if err != nil {
return nil, err
}
return settled(ctx, w, each)
}
// settled hands each entry the watch delivers before its end-of-initial marker to each, and answers the watch.
func settled(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.KeyValueEntry)) (jetstream.KeyWatcher, error) {
for {
select {
case <-ctx.Done():
@@ -102,43 +107,6 @@ func follow(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.Key
}
}
// stepsWaiter waits for a build module's step by watching its key in the module's steps bucket (ADR 0469 clause 2): the
// watch hands over the value the key holds now, so a step that ended before the watch began is read at once, and every
// change after. No timer: the caller bounds the wait with its context.
type stepsWaiter struct{ js bucketOpener }
func (w stepsWaiter) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
kv, err := w.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return delivery.Step{}, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
watch, err := kv.Watch(ctx, key)
if err != nil {
return delivery.Step{}, err
}
defer watch.Stop()
for {
select {
case <-ctx.Done():
return delivery.Step{}, ctx.Err()
case e, ok := <-watch.Updates():
if !ok {
return delivery.Step{}, errors.New("the watch of the step ended")
}
if e == nil || !opOf(e) {
continue
}
step, err := delivery.DecodeStep(e.Value())
if err != nil {
return delivery.Step{}, fmt.Errorf("the step %s is not a step: %w", key, err)
}
if step.State != delivery.StepRunning {
return step, nil
}
}
}
}
// manifestReader reads manifests.<digest> of the indexer and parses the manifest it holds.
type manifestReader struct{ kv jetstream.KeyValue }
@@ -0,0 +1,185 @@
//go:build integration
package main
import (
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again, by the builder whole against the bus
// (novox/hq issue 1937): on 2026-10-24 the store's 03:30 collection deleted the bare blobs of three done keys the
// controller had never taken in, and the builder kept them done.
// heldVerb answers the controller seat's verb held on the bus, as the serving controller does: {result: {output, ok,
// answer}}; a reference is gone while gone says so.
type heldVerb struct {
mu sync.Mutex
gone func(reference string) bool
asked []string
}
func (h *heldVerb) serve(t *testing.T, b *bed) {
t.Helper()
sub, err := b.js.Conn().Subscribe("mesh.seat.mesh-controller.call.held.>", func(m *nats.Msg) {
var args struct {
References []string `json:"references"`
}
_ = json.Unmarshal(m.Data, &args)
a := delivery.HeldAnswer{Store: "store.test:5100", Held: []string{}, Gone: []string{}}
h.mu.Lock()
for _, r := range args.References {
h.asked = append(h.asked, r)
if h.gone(r) {
a.Gone = append(a.Gone, r)
} else {
a.Held = append(a.Held, r)
}
}
h.mu.Unlock()
reply, _ := json.Marshal(map[string]any{"result": map[string]any{"output": "held", "ok": true, "answer": a}, "node": "ctl"})
_ = m.Respond(reply)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
func (h *heldVerb) asks() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.asked)
}
// eventsConsumer makes EVENTS and the builder's durable consumer of artifacts-gone on it, as the controller makes them
// from the manifest's consumes.
func eventsConsumer(t *testing.T, b *bed, node string) {
t.Helper()
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: eventsStream, Subjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
if _, err := b.js.CreateOrUpdateConsumer(b.ctx, eventsStream, jetstream.ConsumerConfig{Durable: node + "_mesh-delivery-builder",
AckPolicy: jetstream.AckExplicitPolicy, FilterSubjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
}
// doneKey writes builds.<module>.<fp> done, naming one archive at reference, and answers its revision.
func doneKey(t *testing.T, b *bed, module, manifest, reference, digest string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
data, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "n2.mesh-delivery-builder",
Manifest: manifest, Step: "built mesh-build-go on n2",
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Digest: digest, Reference: reference}}})
if err != nil {
t.Fatal(err)
}
rev, err := b.builds.Put(b.ctx, key, data)
if err != nil {
t.Fatal(err)
}
return rev
}
func revisionOf(t *testing.T, b *bed, module string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
e, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
return e.Revision()
}
// 1937/AC3 (integration): the builder, started whole, finds the done key of the trunk's fingerprint naming a blob held
// answers gone, builds it again as a new revision whose step says so, and leaves the rebuilt key done.
func TestADoneKeyWhoseBlobIsGoneIsBuiltAgainAtStart(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
old := digestOf("collected last night")
oldRef := "artifact-store://ca-trust/files/blobs/" + old
rev := doneKey(t, b, "ca-trust", manifest, oldRef, old)
h := &heldVerb{gone: func(r string) bool { return r == oldRef }}
h.serve(t, b)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && len(v.Artifacts) == 1 && v.Artifacts[0].Digest == digestOf("files")
})
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+oldRef) {
t.Errorf("the rebuilt key's step %q does not say why it was built again", v.Step)
}
if got := revisionOf(t, b, "ca-trust"); got <= rev {
t.Errorf("revision %d, was %d: want a new revision", got, rev)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("the archive was built %d times, want once", n)
}
if h.asks() == 0 {
t.Error("the builder never asked held")
}
}
// 1937/AC4 (integration): a done key whose blob held answers held at start stays done; when the controller says
// artifacts-gone for it, the builder asks again and builds it again.
func TestArtifactsGoneLeadsTheBuilderToBuildTheKeyAgain(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
d := digestOf("files")
ref := "artifact-store://ca-trust/files/blobs/" + d
rev := doneKey(t, b, "ca-trust", manifest, ref, d)
var mu sync.Mutex
collected := false
h := &heldVerb{gone: func(r string) bool {
mu.Lock()
defer mu.Unlock()
if r == ref && collected {
collected = false // gone once: the rebuild pushes it again
return true
}
return false
}}
h.serve(t, b)
b.start(t, "n1", nil)
for deadline := time.Now().Add(20 * time.Second); h.asks() == 0 && time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
}
if h.asks() == 0 {
t.Fatal("the builder never asked held at its start")
}
time.Sleep(300 * time.Millisecond)
if got := revisionOf(t, b, "ca-trust"); got != rev || b.fake.count("mesh-build-go.build_go_build") != 0 {
t.Fatalf("a key whose blob is held was built again (revision %d, was %d)", got, rev)
}
mu.Lock()
collected = true
mu.Unlock()
key, _ := delivery.BuilderBuildKey("ca-trust", fp)
body, err := delivery.EncodeArtifactsGone(delivery.ArtifactsGone{Module: "ca-trust", Key: key, Fingerprint: fp,
Gone: []string{ref}, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, delivery.ArtifactsGoneSubject, body); err != nil {
t.Fatal(err)
}
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && strings.Contains(v.Step, "built again")
})
if got := revisionOf(t, b, "ca-trust"); got <= rev || b.fake.count("mesh-build-go.build_go_build") != 1 {
t.Errorf("build = %+v at revision %d (was %d), %d builds: want built again once on the event", v, got, rev,
b.fake.count("mesh-build-go.build_go_build"))
}
}
@@ -0,0 +1,306 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// heldOnTheBus says whether an ask a build key names is on its build seat's stream, or a build node wrote a step for it
// (issue 1910), read once at the take-over of a key at start; and, at the builder's check bound, whether an ask can still
// be answered (issue 1945). Never polled.
type heldOnTheBus struct{ js jetstream.JetStream }
// Held is build.Builder's Held: a step key `<node>.<ask id>` in the build module's steps, or a message of the stream under
// the ask's subject whose Nats-Msg-Id is the ask's id.
func (h heldOnTheBus) Held(ctx context.Context, a delivery.StepAsk) (bool, error) {
seen, err := h.seen(ctx, a, false)
if err != nil {
return false, err
}
return seen.Step || seen.OnStream, nil
}
// Gone is build.Builder's Gone: why the ask can no longer be answered (build.AskSeen.Why), or "".
func (h heldOnTheBus) Gone(ctx context.Context, a delivery.StepAsk) (string, error) {
seen, err := h.seen(ctx, a, true)
if err != nil {
return "", err
}
return seen.Why(), nil
}
// maxOnSubject bounds the messages of one subject a read walks (each a read by sequence, next by subject), and maxSteps
// the steps it reads whole to find the last ask taken: past them the read says it cannot tell, and the ask is waited for.
const (
maxOnSubject = 5000
maxSteps = 200
)
// seen reads what the bus holds of the ask; with later, the ask asked last that a build node took too (AskSeen.Last), when
// the ask is on its stream.
func (h heldOnTheBus) seen(ctx context.Context, a delivery.StepAsk, later bool) (build.AskSeen, error) {
seen := build.AskSeen{Kind: a.Kind, At: build.AskedAt(a.ID)}
module, err := delivery.BuildModuleOf(a.Language)
if err != nil {
return seen, err
}
steps, err := h.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return seen, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
pattern, err := delivery.AskStepWatch(a.ID)
if err != nil {
return seen, err
}
keys, err := steps.ListKeysFiltered(ctx, pattern)
if err != nil {
return seen, err
}
for range keys.Keys() {
keys.Stop()
seen.Step = true // a node took it
return seen, nil
}
stream, name, subject, err := h.stream(ctx, a.Language, a.Kind)
seen.Stream = name
if errors.Is(err, jetstream.ErrStreamNotFound) {
return seen, nil
}
if err != nil {
return seen, err
}
var ahead []build.Queued
err = onSubject(ctx, stream, subject, func(q build.Queued) bool {
if q.Ask == a.ID {
seen.OnStream, seen.PutAt = true, q.PutAt
return false
}
ahead = append(ahead, q)
return true
})
if err != nil || !seen.OnStream || !later {
return seen, err
}
taken, err := stepKeys(ctx, steps)
if err != nil {
return seen, err
}
for i := range ahead {
ahead[i].Step = stepState(ctx, steps, taken[ahead[i].Ask])
}
seen.Ahead = ahead
since := seen.At
if since.IsZero() {
since = seen.PutAt
}
for _, q := range ahead {
if q.At.Before(since) {
since = q.At
}
}
seen.Last = lastTaken(ctx, steps, taken, a.Kind, since)
return seen, nil
}
// stream is the build seat's stream of language, its name and the subject of kind on it.
func (h heldOnTheBus) stream(ctx context.Context, language, kind string) (jetstream.Stream, string, string, error) {
subject, err := delivery.AskSubject(language, kind)
if err != nil {
return nil, "", "", err
}
// The seat's stream by its name (delivery.AskStream), as the broker grants the builder its information and its
// messages by sequence and nothing wider (issue 1910).
name, err := delivery.AskStream(language)
if err != nil {
return nil, "", "", err
}
stream, err := h.js.Stream(ctx, name)
return stream, name, subject, err
}
// onSubject hands each message of subject on stream to each, in order, until each answers false or the subject's
// messages end. A seat's stream is a work queue, which takes no second consumer over its subjects: its messages are read
// by sequence instead, each the next of the subject (one read per message, none for the sequences taken between), and
// none consumed.
func onSubject(ctx context.Context, stream jetstream.Stream, subject string, each func(build.Queued) bool) error {
for seq, n := uint64(1), 0; ; n++ {
if n >= maxOnSubject {
return fmt.Errorf("%s holds more than %d messages before the one asked after: not read further", subject, maxOnSubject)
}
m, err := stream.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(subject))
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil
}
if err != nil {
return err
}
q := build.Queued{Seq: m.Sequence, Ask: m.Header.Get(jetstream.MsgIDHeader), PutAt: m.Time}
if _, kind, err := delivery.ParseAskSubject(m.Subject); err == nil {
q.Kind = kind
}
if q.At = build.AskedAt(q.Ask); q.At.IsZero() {
q.At = q.PutAt
}
if !each(q) {
return nil
}
seq = m.Sequence + 1
}
}
// stepKeys are the build module's step keys by ask id: `<node>.<ask id>` (a node's own key, such as its pause, has no
// ask id and is left out).
func stepKeys(ctx context.Context, steps jetstream.KeyValue) (map[string][]string, error) {
all, err := steps.ListKeys(ctx)
if err != nil {
if errors.Is(err, jetstream.ErrNoKeysFound) {
return map[string][]string{}, nil
}
return nil, err
}
defer all.Stop()
out := map[string][]string{}
for k := range all.Keys() {
if _, id, ok := strings.Cut(k, "."); ok && !build.AskedAt(id).IsZero() {
out[id] = append(out[id], k)
}
}
return out, nil
}
// stepState is the state of the first of keys a node wrote, "" for none.
func stepState(ctx context.Context, steps jetstream.KeyValue, keys []string) string {
for _, k := range keys {
e, err := steps.Get(ctx, k)
if err != nil {
continue
}
if s, err := delivery.DecodeStep(e.Value()); err == nil {
return s.State
}
}
return ""
}
// lastTaken is the ask asked last after since that a build node took, of a kind whose taking proves kind passed
// (build.Proves), read newest first and at most maxSteps of them; nil for none. A step's tool is "<kind> ask"
// (mesh-sdk buildstep).
func lastTaken(ctx context.Context, steps jetstream.KeyValue, taken map[string][]string, kind string, since time.Time) *build.Taken {
var ids []string
for id := range taken {
if build.AskedAt(id).After(since) {
ids = append(ids, id)
}
}
sort.Slice(ids, func(i, j int) bool { return build.AskedAt(ids[i]).After(build.AskedAt(ids[j])) })
read := 0
for _, id := range ids {
for _, k := range taken[id] {
if read >= maxSteps {
return nil
}
read++
e, err := steps.Get(ctx, k)
if err != nil {
continue
}
s, err := delivery.DecodeStep(e.Value())
if err != nil {
continue
}
stepKind, ok := strings.CutSuffix(s.Tool, " ask")
if !ok || !build.Proves(stepKind, kind) {
continue
}
return &build.Taken{Ask: id, Kind: stepKind, At: build.AskedAt(id), Node: s.Node, Started: s.Started}
}
}
return nil
}
// Drop is build.Dropper's Drop: it removes the ask's message from its build seat's stream, found by its Nats-Msg-Id on its
// subject, when the stream still holds it. The builder's account must be granted $JS.API.STREAM.MSG.DELETE.<stream>.
func (h heldOnTheBus) Drop(ctx context.Context, a delivery.StepAsk) error {
stream, name, subject, err := h.stream(ctx, a.Language, a.Kind)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return nil
}
if err != nil {
return err
}
var seq uint64
if err := onSubject(ctx, stream, subject, func(q build.Queued) bool {
if q.Ask == a.ID {
seq = q.Seq
return false
}
return true
}); err != nil || seq == 0 {
return err
}
if err := stream.DeleteMsg(ctx, seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return fmt.Errorf("deleting message %d of %s (is $JS.API.STREAM.MSG.DELETE.%s granted to the builder?): %w", seq, name, name, err)
}
return nil
}
// Queue is what the tool builder_seat_queue answers (build.SeatQueue): the messages of language's build seat's stream, of
// every kind, oldest first and at most limit of each kind, each with its step's state and whether it still waits.
func (h heldOnTheBus) Queue(ctx context.Context, language string, limit int) (build.SeatQueue, error) {
q := build.SeatQueue{Language: language, Note: build.QueueNote}
module, err := delivery.BuildModuleOf(language)
if err != nil {
return q, err
}
stream, name, _, err := h.stream(ctx, language, delivery.AskDelivery)
q.Stream = name
if errors.Is(err, jetstream.ErrStreamNotFound) {
q.Note = name + " does not exist"
return q, nil
}
if err != nil {
return q, err
}
info, err := stream.Info(ctx)
if err != nil {
return q, err
}
q.Messages, q.FirstSeq, q.LastSeq = info.State.Msgs, info.State.FirstSeq, info.State.LastSeq
steps, err := h.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return q, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
taken, err := stepKeys(ctx, steps)
if err != nil {
return q, err
}
for _, kind := range delivery.AskKinds {
subject, _ := delivery.AskSubject(language, kind)
var ahead []build.Queued
if err := onSubject(ctx, stream, subject, func(m build.Queued) bool {
ahead = append(ahead, m)
return len(ahead) < limit
}); err != nil {
return q, err
}
if len(ahead) == 0 {
continue
}
for i := range ahead {
ahead[i].Step = stepState(ctx, steps, taken[ahead[i].Ask])
}
seen := build.AskSeen{Kind: kind, Ahead: ahead, Last: lastTaken(ctx, steps, taken, kind, ahead[0].At)}
q.Kinds = append(q.Kinds, build.NewKindQueue(kind, seen, time.Now()))
}
return q, nil
}
@@ -7,6 +7,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
@@ -21,7 +22,10 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
@@ -46,13 +50,12 @@ const (
fp = "src2:3333333333333333333333333333333333333333333333333333333333333333"
)
// fakeBuildModules serve the tools of mesh-build-go and mesh-build-node as the real ones do (ADR 0469 clause 2): a call on
// mesh.mod.<module>.call.<tool>.<node>.<caller> answers at once with the step and its key, and the step's end is written to
// the key <node>.<id> of the module's steps bucket, some milliseconds later.
// fakeBuildModules are the build modules on the nodes as the real ones run since R4g (ADR 0525; issue 1887 A4): each pulls
// its build seat's asks on its nodes (buildstep.Puller over the seat's workers) and runs an ask's job with step bodies that
// answer as the real tools do, ending the step on <node>.<ask id> of its steps bucket.
type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
n atomic.Int64
failBuild atomic.Bool
// refusePush makes every bundle and archive build fail as a push the artifact store refused (ADR 0498 clause 2).
refusePush atomic.Bool
@@ -60,6 +63,21 @@ type fakeBuildModules struct {
calls map[string]int // by `<module>.<tool>`
fetches []map[string]any
builds []map[string]any // the arguments of every build call
// closed, while set, keeps every node's capacity at 0: no ask is pulled until open is called (a test of the asks
// waiting in their streams).
closed atomic.Bool
pullers []*buildstep.Puller
}
// open lets the nodes pull again and wakes their pullers.
func (f *fakeBuildModules) open() {
f.closed.Store(false)
f.mu.Lock()
ps := append([]*buildstep.Puller(nil), f.pullers...)
f.mu.Unlock()
for _, p := range ps {
p.Wake()
}
}
func (f *fakeBuildModules) count(tool string) int {
@@ -74,76 +92,93 @@ func digestOf(artifact string) string {
return "sha256:" + hex.EncodeToString(sum[:])
}
func (f *fakeBuildModules) serve(t *testing.T, nc *nats.Conn, module string) {
steps, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
var prefixOf = map[string]string{"mesh-build-go": "build_go_", "mesh-build-node": "build_node_", "mesh-build-image": "build_image_"}
// step is one tool's answer, as the real module's.
func (f *fakeBuildModules) step(module, name string, args map[string]any) (any, error) {
tool := prefixOf[module] + name
f.mu.Lock()
f.calls[module+"."+tool]++
if name == "fetch" {
f.fetches = append(f.fetches, args)
} else {
f.builds = append(f.builds, args)
}
f.mu.Unlock()
switch {
case name == "fetch":
merge := ""
if args["base"] != nil { // a fetch with a base answers the merge commit (mesh-build-go FetchResult)
merge = `,"merge":"` + strings.Repeat("9", 40) + `"`
}
return json.RawMessage(`{"workspace":"w` + buildstep.NewID() + `","commit":"` + commit + `"` + merge + `}`), nil
case f.failBuild.Load() && tool != "build_image_copy": // the build fails wherever it runs
return nil, fmt.Errorf("go build of %v: boom", args["artifact"])
case tool == "build_image_copy": // the upstream copy answers the digest and the registry's reference (ADR 0498 clause 4)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"artifact":"` + fmt.Sprint(args["artifact"]) + `","digest":"` + d + `","reference":"registry.example/` + fmt.Sprint(args["module"]) + `/` + fmt.Sprint(args["artifact"]) + `@` + d + `","pushed":true}`), nil
case tool != "build_image_build" && f.refusePush.Load():
return nil, fmt.Errorf("%s%v: the artifact store answered 507 Insufficient Storage", delivery.NotStored, args["artifact"])
case tool != "build_image_build": // a bundle or an archive is pushed and its reference answered (ADR 0498 clause 1)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"artifact-store://` + filepath.Base(fmt.Sprint(args["module"])) + `/` + fmt.Sprint(args["artifact"]) + `/blobs/` + d + `"}`), nil
case tool == "build_image_build" && args["publish"] == true: // a trunk image is pushed: the digest and the reference
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"registry.example/imaged/server@` + d + `"}`), nil
default:
return json.RawMessage(`{"digest":"` + digestOf(fmt.Sprint(args["artifact"])) + `"}`), nil
}
}
// serve makes module's steps bucket and its build seat's stream and workers, and pulls its asks on nodes.
func (f *fakeBuildModules) serve(t *testing.T, module string, nodes ...string) {
kv, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
sub, err := nc.Subscribe("mesh.mod."+module+".call.>", func(m *nats.Msg) {
tokens := strings.Split(m.Subject, ".") // mesh mod <module> call <tool> <node> <caller>
tool, node := tokens[4], tokens[5]
f.mu.Lock()
f.calls[module+"."+tool]++
f.mu.Unlock()
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
if strings.HasSuffix(tool, "_fetch") {
f.mu.Lock()
f.fetches = append(f.fetches, args)
f.mu.Unlock()
} else {
f.mu.Lock()
f.builds = append(f.builds, args)
f.mu.Unlock()
}
id := fmt.Sprintf("s%d", f.n.Add(1))
key, _ := delivery.StepKey(node, id)
put := func(s delivery.Step) {
data, err := delivery.EncodeStep(s)
if err == nil {
_, err = steps.Put(f.ctx, key, data)
}
if err != nil {
t.Errorf("writing step %s: %v", key, err)
}
}
step := delivery.Step{ID: id, Tool: tool, Node: node, State: delivery.StepRunning, Started: time.Now().UTC()}
put(step)
answer, _ := json.Marshal(map[string]any{"result": map[string]any{"step": id, "key": key, "state": "running"}, "error": "", "node": node})
_ = m.Respond(answer)
go func() {
time.Sleep(30 * time.Millisecond)
step.Ended = time.Now().UTC()
switch {
case strings.HasSuffix(tool, "_fetch"):
merge := ""
if args["base"] != nil { // a fetch with a base answers the merge commit (mesh-build-go FetchResult)
merge = `,"merge":"` + strings.Repeat("9", 40) + `"`
language := strings.TrimPrefix(module, "mesh-build-")
if err := buildstep.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
call := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) { return f.step(module, name, args) }
}
steps := buildstep.AskSteps{Fetch: call("fetch"), Build: call("build"), Copy: call("copy")}
for _, node := range nodes {
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: node, Put: put,
Capacity: func() int {
if f.closed.Load() {
return 0
}
step.State, step.Result = delivery.StepPassed, json.RawMessage(`{"workspace":"w`+id+`","commit":"`+commit+`"`+merge+`}`)
case f.failBuild.Load() && tool != "build_image_copy": // every node's build fails: one node builds a fingerprint
step.State, step.Reason = delivery.StepFailed, fmt.Sprintf("go build of %v: boom", args["artifact"])
case tool == "build_image_copy": // the upstream copy answers the digest and the registry's reference (ADR 0498 clause 4)
d := digestOf(fmt.Sprint(args["artifact"]))
step.State, step.Result = delivery.StepPassed, json.RawMessage(`{"artifact":"`+fmt.Sprint(args["artifact"])+`","digest":"`+d+`","reference":"registry.example/`+fmt.Sprint(args["module"])+`/`+fmt.Sprint(args["artifact"])+`@`+d+`","pushed":true}`)
case tool != "build_image_build" && f.refusePush.Load():
step.State, step.Reason = delivery.StepFailed, fmt.Sprintf("%s%v: the artifact store answered 507 Insufficient Storage", delivery.NotStored, args["artifact"])
case tool != "build_image_build": // a bundle or an archive is pushed and its reference answered (ADR 0498 clause 1)
d := digestOf(fmt.Sprint(args["artifact"]))
step.State, step.Result = delivery.StepPassed, json.RawMessage(`{"digest":"`+d+`","reference":"artifact-store://`+filepath.Base(fmt.Sprint(args["module"]))+`/`+fmt.Sprint(args["artifact"])+`/blobs/`+d+`"}`)
case tool == "build_image_build" && args["publish"] == true: // a trunk image is pushed: the digest and the reference
d := digestOf(fmt.Sprint(args["artifact"]))
step.State, step.Result = delivery.StepPassed, json.RawMessage(`{"digest":"`+d+`","reference":"registry.example/imaged/server@`+d+`"}`)
default:
step.State, step.Result = delivery.StepPassed, json.RawMessage(`{"digest":"`+digestOf(fmt.Sprint(args["artifact"]))+`"}`)
}
put(step)
}()
})
return 2
}, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
f.mu.Lock()
f.pullers = append(f.pullers, p)
f.mu.Unlock()
go func() { _ = p.Pull(f.ctx) }()
}
}
// trunkHead is the trunk head of novox/mesh-catalog the bed's source revision names: newer than the module values' commit.
const trunkHead = "3333333333333333333333333333333333333333"
// trunk writes the gate's source.main naming head as novox/mesh-catalog's trunk head, and answers its revision.
func (b *bed) trunk(t *testing.T, head string) uint64 {
t.Helper()
data, err := delivery.EncodeSource(delivery.Source{V: delivery.Version, Group: "g", Heads: map[string]string{"novox/mesh-catalog": head}})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
rev, err := b.source.Put(b.ctx, delivery.SourceMainKey, data)
if err != nil {
t.Fatal(err)
}
return rev
}
type bed struct {
@@ -151,10 +186,11 @@ type bed struct {
js jetstream.JetStream
modules jetstream.KeyValue
candidates jetstream.KeyValue
source jetstream.KeyValue
builds jetstream.KeyValue
tools map[string]jetstream.KeyValue
presence jetstream.KeyValue
have jetstream.KeyValue
instances jetstream.KeyValue
stop func() // stops the builder started last
fake *fakeBuildModules
srvURL string
dir string
@@ -163,7 +199,11 @@ type bed struct {
// startBed starts the bus, the buckets the controller makes from the builder's and the build modules' manifests, the
// build modules' streams and the builder's durable consumers on them, two nodes' toolchains, the indexer's manifests, and
// the fake build modules. It runs no builder.
func startBed(t *testing.T) *bed {
func startBed(t *testing.T) *bed { return startBedWith(t, false) }
// startBedWith is startBed with the nodes' capacity 0 from the start when closed: a puller that reads its capacity before a
// test could close the nodes may be in a pull already and takes the first ask put (issue 1928).
func startBedWith(t *testing.T, closed bool) *bed {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "b")
if err != nil {
t.Fatal(err)
@@ -223,21 +263,15 @@ func startBed(t *testing.T) *bed {
}
b.modules = mk(delivery.ModulesBucket, delivery.ModulesHistory)
b.candidates = mk(delivery.CandidatesBucket, delivery.CandidatesHistory)
b.source = mk(delivery.SourceBucket, delivery.SourceHistory)
mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
presence := mk(delivery.PresenceBucket, delivery.PresenceHistory)
b.presence, b.have = presence, mk(delivery.HaveBucket, delivery.HaveHistory)
for _, node := range []string{"n1", "n2"} {
b.setOnline(t, node, true)
for _, module := range []string{"mesh-build-go", "mesh-build-node"} {
b.setHealth(t, node, module, delivery.HealthOK)
}
}
b.instances = mk(instancesBucket, 1)
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.serve(t, ctl, "mesh-build-go")
b.fake.serve(t, ctl, "mesh-build-node")
b.fake.serve(t, ctl, "mesh-build-image")
b.setHealth(t, "n1", "mesh-build-image", delivery.HealthOK) // the image build module runs on n1 only
b.fake.closed.Store(closed)
b.fake.serve(t, "mesh-build-go", "n1", "n2")
b.fake.serve(t, "mesh-build-node", "n1", "n2")
b.fake.serve(t, "mesh-build-image", "n1") // the image build module runs on n1 only
b.report(t, "mesh-build-go", "n1", "go", "go1.27.1", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.1", "1")
b.report(t, "mesh-build-node", "n1", "typescript", "tsc5", "1")
@@ -245,28 +279,6 @@ func startBed(t *testing.T) *bed {
return b
}
// setOnline writes the watchman's presence of a node.
func (b *bed) setOnline(t *testing.T, node string, online bool) {
t.Helper()
data, _ := delivery.EncodePresence(delivery.Presence{Online: online})
if _, err := b.presence.Put(b.ctx, node, data); err != nil {
t.Fatal(err)
}
}
// setHealth writes the node-engine's have of a build module on a node, running with this health.
func (b *bed) setHealth(t *testing.T, node, module string, health delivery.Health) {
t.Helper()
data, err := delivery.EncodeHave(delivery.Have{Phase: delivery.PhaseRunning, Health: health})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.HaveKey(node, module)
if _, err := b.have.Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
}
// report writes a node's toolchains key of a build module and says so on the module's stream, as the module does.
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper()
@@ -344,7 +356,9 @@ func (b *bed) start(t *testing.T, node string, env map[string]string) {
runCtx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return e[k] }, io.Discard) }()
t.Cleanup(func() { stop(); <-done })
var once sync.Once
b.stop = func() { once.Do(func() { stop(); <-done }) }
t.Cleanup(b.stop)
}
// await reads the build key until pred holds of it, within 20 s.
@@ -427,20 +441,8 @@ func TestAnArchiveOnlyModuleNamingNoToolchainIsBuilt(t *testing.T) {
}
}
// TestANodeWhoseBuildModuleIsUnhealthyIsNotUsed: the node-engine's have of mesh-build-go on n2 says failed, so only n1 builds.
func TestANodeWhoseBuildModuleIsUnhealthyIsNotUsed(t *testing.T) {
b := startBed(t)
b.setHealth(t, "n2", "mesh-build-go", delivery.HealthFailed)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if !strings.Contains(v.Step, "on n1") {
t.Errorf("build = %+v, want it built on n1 alone", v)
}
}
// TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint: a build step fails on one node, so the
// key is failed naming it; the indexer writing modules.<m> again with the same fingerprint wakes the builder, which builds
// TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint: a build step fails, so the key is
// failed naming it; the indexer writing modules.<m> again with the same fingerprint wakes the builder, which builds
// the key again as a new revision of the same key, now done.
func TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint(t *testing.T) { // 1607/AC6, AC7
b := startBed(t)
@@ -532,7 +534,9 @@ func TestAKeyLeftBuildingForAModuleThatIsGoneIsClosedAtStart(t *testing.T) {
// TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk: the trunk's record of netcheck carries a fingerprint of its
// own; a candidate the indexer wrote gives netcheck the fingerprint fp at a pull request's head; the program writes
// builds.netcheck.<fp> done, its fetch given the head, the ref that carries it and the trunk commit (issue 1684).
// builds.netcheck.<fp> done, its fetch given the head, the ref that carries it and the trunk head of the candidate's source
// revision, newer than the module value's commit (issue 1684; the operator 2026-10-25: a value names the commit its
// fingerprint first appeared at).
func TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1
b := startBed(t)
other := "src2:" + strings.Repeat("e", 64)
@@ -549,7 +553,7 @@ func TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk(t *tes
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"netcheck"},
Fps: map[string]delivery.CandidateModule{"netcheck": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", Base: b.trunk(t, trunkHead), HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
@@ -569,12 +573,12 @@ func TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk(t *tes
defer b.fake.mu.Unlock()
found := false
for _, f := range b.fake.fetches {
if f["commit"] == head && f["ref"] == "refs/pull/7/head" && f["base"] == commit {
if f["commit"] == head && f["ref"] == "refs/pull/7/head" && f["base"] == trunkHead {
found = true
}
}
if !found {
t.Errorf("fetches = %v, want one of the candidate's head with its ref and the trunk commit", b.fake.fetches)
t.Errorf("fetches = %v, want one of the candidate's head with its ref and the trunk head it was previewed onto, not the module value's commit", b.fake.fetches)
}
}
@@ -622,7 +626,7 @@ func TestAnImageIsBuiltOnOneNodeAndOnlyATrunkKeyPublishes(t *testing.T) { // 170
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"imaged"},
Fps: map[string]delivery.CandidateModule{"imaged": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", Base: b.trunk(t, trunkHead), HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
@@ -713,3 +717,158 @@ func TestAManifestFallingBackToAPredefinedKeyIsBuilt(t *testing.T) {
t.Errorf("build = %+v, want done: the manifest is read", v)
}
}
// 1886/AC5: a builder on n1 whose Go, Node and image build modules each answered a reference says under n1, in its bucket
// of instances, that it writes references, and takes the say away when it stops: the controller delivers from its keys
// only while it holds.
func TestABuilderWhoseBuildModulesAnsweredReferencesSaysItAndStoppedSaysNothing(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // a Go and a TypeScript bundle
b.moduleCommit(t, "imaged", "catalogue-a67f0196-imaged.module.json", "modules/imaged", nil, commit) // an image, for the trunk
b.start(t, "n1", nil)
var said string
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && said == ""; time.Sleep(50 * time.Millisecond) {
if e, err := b.instances.Get(b.ctx, "n1"); err == nil {
said = string(e.Value())
}
}
if !strings.Contains(said, `"writes":["references"]`) {
t.Fatalf("the builder said %q", said)
}
b.stop()
if _, err := b.instances.Get(b.ctx, "n1"); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("the stopped builder still says: %v", err)
}
}
// 1887/AC1, AC2 on the real bus: the builder puts the trunk fingerprint's build as asks on the build seats' delivery
// streams and calls no node's tool; the build modules that pulled them wrote the steps, and the key is done naming the
// references.
func TestTheBuilderAsksTheBuildSeatsAndCallsNoNode(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
var calls atomic.Int64
s, err := sub.Subscribe("mesh.mod.*.call.>", func(*nats.Msg) { calls.Add(1) })
if err != nil {
t.Fatal(err)
}
defer s.Unsubscribe()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
for _, a := range v.Artifacts {
if a.Reference == "" {
t.Errorf("artifact %s names no reference", a.Name)
}
}
if n := calls.Load(); n != 0 {
t.Errorf("%d tool calls reached a build module", n)
}
// 1887/AC1: one ask per build module the build needs (netcheck: a Go and a TypeScript bundle), each on its seat's
// delivery stream.
got := asked()
if len(got) != 2 {
t.Fatalf("asks put = %+v, want one Go and one Node delivery ask", got)
}
for _, a := range got {
if a.Kind != delivery.AskDelivery || a.Build == nil || a.Build.FP != fp {
t.Errorf("ask %+v, want a delivery ask of the fingerprint", a)
}
}
}
// asksPut hears every ask put on a build seat's accept subjects (a core subscription beside the stream) and answers the
// asks heard so far.
func (b *bed) asksPut(t *testing.T, nc *nats.Conn) func() []delivery.StepAsk {
t.Helper()
var mu sync.Mutex
var asks []delivery.StepAsk
s, err := nc.Subscribe("mesh.seat.*.accept.>", func(m *nats.Msg) {
if a, err := delivery.DecodeStepAsk(m.Data); err == nil {
mu.Lock()
asks = append(asks, a)
mu.Unlock()
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Unsubscribe() })
return func() []delivery.StepAsk {
mu.Lock()
defer mu.Unlock()
return append([]delivery.StepAsk(nil), asks...)
}
}
// 1887/AC3: a builder that started again while its asks waited in the streams (no node had a free slot) puts no second
// ask: it waits for the asks its building key names, and the key is done once a node took them.
func TestARestartedBuilderPutsNoSecondAsk(t *testing.T) {
b := startBedWith(t, true) // 1928/AC1: no node can take an ask from the start, so no pull in hand takes the first one
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", func(v delivery.BuilderBuild) bool { return v.State == delivery.BuildBuilding && len(v.Asks) == 2 })
for deadline := time.Now().Add(20 * time.Second); len(asked()) < 2; time.Sleep(20 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("asks put = %+v within 20 s, want the two the key names", asked())
}
}
// 1945 note 62, on the real bus: the stream read as builder_seat_queue reads it shows the Go ask waiting; the ask can
// still be answered (nothing was taken since); and a message dropped as an ask asked again is gone from the stream.
held := heldOnTheBus{b.js}
goAsk := delivery.StepAsk{ID: v.Asks[delivery.AskGo], Language: delivery.AskGo, Kind: delivery.AskDelivery}
stale := delivery.StepAsk{ID: "mvo53jcm70b5f7", Language: delivery.AskGo, Kind: delivery.AskDelivery}
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskDelivery)
if _, err := b.js.Publish(b.ctx, subject, []byte("{}"), jetstream.WithMsgID(stale.ID)); err != nil {
t.Fatal(err)
}
queued := func() []build.QueuedState {
q, err := held.Queue(b.ctx, delivery.AskGo, 10)
if err != nil || len(q.Kinds) != 1 || q.Kinds[0].Kind != delivery.AskDelivery {
t.Fatalf("the Go seat's queue = %+v, %v; want its delivery asks", q, err)
}
return q.Kinds[0].Messages
}
if m := queued(); len(m) != 2 || m[0].Ask != goAsk.ID || !m[0].Waits || m[1].Ask != stale.ID {
t.Fatalf("the Go delivery queue = %+v, want the key's ask waiting, then the stale one", m)
}
if why, err := held.Gone(b.ctx, goAsk); why != "" || err != nil {
t.Errorf("Gone(the waiting ask) = %q, %v; want it answerable", why, err)
}
if err := held.Drop(b.ctx, stale); err != nil {
t.Fatal(err)
}
if m := queued(); len(m) != 1 || m[0].Ask != goAsk.ID {
t.Fatalf("after the drop the Go delivery queue = %+v, want the key's ask alone", m)
}
b.stop()
b.start(t, "n1", nil) // the same builder, started again, while both asks wait
b.fake.open()
done := b.await(t, "netcheck", isState(delivery.BuildDone))
if got := asked(); len(got) != 2 {
t.Fatalf("asks put = %d (%+v), want the two the key named and none again", len(got), got)
}
for lang, id := range v.Asks {
found := false
for _, a := range asked() {
found = found || a.ID == id
}
if !found {
t.Errorf("the %s ask %s the key named was never put", lang, id)
}
}
if len(done.Artifacts) != 2 {
t.Errorf("done = %+v", done)
}
}
@@ -13,15 +13,19 @@ import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"io"
"log"
"maps"
"os"
"os/signal"
"slices"
"strings"
"sync"
"syscall"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
@@ -32,8 +36,9 @@ import (
)
const (
// defaultStepBound is how long the builder waits for one step of a build module (a fetch downloads the Go modules, a
// build compiles).
// defaultStepBound is how long one ask's step may run once a build node took it (a fetch downloads the Go modules, a
// build compiles): from its start, never from its put, so an ask waiting in its stream for a free slot is not failed
// (buildrun.Asker.Bound; ADR 0525 §3, §6).
defaultStepBound = 30 * time.Minute
)
@@ -120,66 +125,93 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
bucket := func(name string) (jetstream.KeyValue, error) {
kv, err := js.KeyValue(ctx, name)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", name, err)
}
return kv, nil
}
builds, err := bucket(delivery.BuilderBuildsBucket)
builds, err := process.EssentialBucket(ctx, js, "builder", delivery.BuilderBuildsBucket, "claim or write a build key")
if err != nil {
return err
}
modules, err := bucket(delivery.ModulesBucket)
modules, err := process.EssentialBucket(ctx, js, "builder", delivery.ModulesBucket, "know any fingerprint to build")
if err != nil {
return err
}
manifests, err := bucket(delivery.ManifestsBucket)
manifests, err := process.EssentialBucket(ctx, js, "builder", delivery.ManifestsBucket, "read what a module builds")
if err != nil {
return err
}
candidates, err := bucket(delivery.CandidatesBucket)
if err != nil {
return err
}
// Calls to the build modules go out as this module's own account, naming the machine (mesh-sdk process.CallOn); the
// buckets are read and written on the connection above.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
caller, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer caller.Close()
// The candidates' fingerprints are built too (issue 1684): a builder that cannot read them builds the trunk's.
candidates := process.OptionalBucket(ctx, js, delivery.CandidatesBucket, "no candidate's fingerprint is built until the builder starts again; the trunk's are", logger.Printf)
bound := defaultStepBound
if v := strings.TrimSpace(getenv("MESH_BUILDER_STEP_BOUND")); v != "" {
if bound, err = time.ParseDuration(v); err != nil {
return fmt.Errorf("MESH_BUILDER_STEP_BOUND: %w", err)
}
}
capacity, err := capacityFrom(strings.TrimSpace(getenv("MESH_BUILDER_SETTINGS")), getenv("MESH_BUILDER_CAPACITY"))
if err != nil {
return fmt.Errorf("the build capacity setting: %w", err)
}
tools := &buildrun.ToolchainSet{}
presence := &buildrun.PresenceSet{}
have := &buildrun.HaveSet{}
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
asks := buildrun.JetStreamAsks{JS: js}
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
account, err := process.ReadAccount(path)
if err != nil {
return err
}
calls, err := process.Connect(account, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer calls.Close()
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Runner: buildrun.Runner{Caller: caller, Steps: stepsWaiter{js}, Bound: bound},
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: bound},
Held: heldOnTheBus{js},
// An ask that can no longer be answered is asked again (issue 1945), read at the step bound: a context's bound.
Gone: heldOnTheBus{js},
CheckAfter: bound,
Toolchains: tools,
Presence: presence,
Health: have,
Places: &buildrun.Places{Capacity: capacity},
Manifests: manifestReader{manifests},
Logf: logger.Printf,
Stored: storedVia(calls.Call),
}
// A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base
// revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision
// is not what a read grants): a revision heard after the candidate that names it builds that candidate then. Where it
// cannot be read the trunk's keys are built and the candidates wait (issue 1945).
errs := make(chan error, 6)
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf}
trunks.start()
b.Trunk = trunks.head
b.Said = func(e build.AskedAgain) {
if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
}
// **Said once every build module answered a reference, taken away when stopped** (issue 1886, ADR 0522): the
// controller delivers from this builder's keys only while an instance says it writes references. A bucket not there
// yet (the manifest's state not made) is said, and the controller's plan waits for the keys, named (no build-agent
// is asked since R4g, ADR 0524 §7).
switch kv := process.OptionalBucket(ctx, js, instancesBucket, "the controller is not told that this builder writes references until it starts again", logger.Printf); {
case kv == nil:
default:
says := newSayer(ctx, kv, cred.Node, []string{buildrun.GoBuildModule, buildrun.NodeBuildModule, build.ImageBuildModule}, logger.Printf)
b.Answered = says.Answered
defer func() {
stopping, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if err := says.unsay(stopping); err != nil {
logger.Printf("%v", err)
}
}()
}
// The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, b.NodesChanged, logger)
if err != nil {
return err
}
if err := watch.start(); err != nil {
return err
}
@@ -200,47 +232,20 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
defer cc.Stop()
}
// The watchman's presence says which nodes are asleep. Where its bucket is not there yet no node is online, so nothing
// builds; the builder says so and is started again with the watchman.
presenceErrs := make(chan error, 1)
switch kv, err := js.KeyValue(ctx, delivery.PresenceBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no node counts as online until the builder starts again", delivery.PresenceBucket)
// The controller says artifacts-gone when its take-in found a key's blob gone (issue 1937): the builder's durable
// consumer on EVENTS, made by the controller from the manifest's consumes. Where it is not there yet the builder says
// so; it still asks held at its start and at every module value.
switch cons, err := js.Consumer(ctx, eventsStream, cred.Node+"_"+cred.Module); {
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.PresenceBucket, err)
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s is not heard until the builder starts again: %v",
cred.Node, cred.Module, eventsStream, delivery.ArtifactsGoneEvent, err)
default:
pw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { presence.Put(e.Key(), e.Value(), !opOf(e)) })
cc, err := cons.Consume(func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) },
jetstream.ConsumeErrHandler(func(_ jetstream.ConsumeContext, err error) { logger.Printf("hearing the bus: %v", err) }))
if err != nil {
return err
}
go func() {
presenceErrs <- follow(ctx, pw, func(e jetstream.KeyValueEntry) {
if presence.Put(e.Key(), e.Value(), !opOf(e)) {
b.NodesChanged(ctx) // a node came online or went: a build that waited for one may go on
}
})
}()
}
// The node-engine's have says whether a node's build module runs and is healthy: a node whose build module is not is not
// chosen (ADR 0478 clause 3). Where its bucket is not there yet no build module counts as healthy, so nothing builds.
haveErrs := make(chan error, 1)
switch kv, err := js.KeyValue(ctx, delivery.HaveBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no build module counts as healthy until the builder starts again", delivery.HaveBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.HaveBucket, err)
default:
hw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { have.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
haveErrs <- follow(ctx, hw, func(e jetstream.KeyValueEntry) {
if have.Put(e.Key(), e.Value(), !opOf(e)) {
b.NodesChanged(ctx) // a build module became healthy or stopped being: a build that waited may go on
}
})
}()
defer cc.Stop()
}
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
@@ -265,25 +270,92 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
b.ForgetCandidate(e.Key())
}
}
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
return err
if candidates != nil {
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
process.NotWatched(logger.Printf, delivery.CandidatesBucket, err, "no candidate's fingerprint is built until the builder starts again")
} else {
go func() { errs <- follow(ctx, cw, eachCandidate) }()
}
}
// A person asks a key's build again with builder_ask_again, which writes the reasks state (issue 1945): watched, as
// every other input. The builder asks again by itself without it.
if kv := process.OptionalBucket(ctx, js, build.ReAsksBucket, "builder_ask_again is not heard until the builder starts again; an ask that can no longer be answered is still asked again by itself", logger.Printf); kv != nil {
eachReAsk := func(e jetstream.KeyValueEntry) { reAsked(ctx, b, kv, e, logger.Printf) }
if rw, err := settle(ctx, kv, eachReAsk); err != nil {
process.NotWatched(logger.Printf, build.ReAsksBucket, err, "builder_ask_again is not heard until the builder starts again")
} else {
go func() { errs <- follow(ctx, rw, eachReAsk) }()
}
}
// A person reads a build seat's stream with builder_seat_queue, which writes the queues state (issue 1945 note 62): the
// builder alone may read the stream, so it answers there.
if kv := process.OptionalBucket(ctx, js, build.QueuesBucket, "builder_seat_queue is not answered until the builder starts again", logger.Printf); kv != nil {
eachRead := func(e jetstream.KeyValueEntry) { queueRead(ctx, heldOnTheBus{js}, kv, e, logger.Printf) }
if qw, err := settle(ctx, kv, eachRead); err != nil {
process.NotWatched(logger.Printf, build.QueuesBucket, err, "builder_seat_queue is not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, qw, eachRead) }()
}
}
logger.Printf("started on %s", cred.Node)
errs := make(chan error, 2)
go func() { errs <- follow(ctx, w, each) }()
go func() { errs <- follow(ctx, cw, eachCandidate) }()
select {
case <-ctx.Done():
b.Wait()
return nil
case err := <-errs:
return err
case err := <-assignErrs:
return err
case err := <-watch.errs:
return err
case err := <-presenceErrs:
return err
case err := <-haveErrs:
return err
}
}
// sourceHeads are the trunk heads of each revision of the gate's source.main the watch delivered, the newest
// delivery.SourceHistory of them: what a candidate's base revision names.
type sourceHeads struct {
mu sync.Mutex
byRev map[uint64]map[string]string
}
// put takes one revision in and answers whether it is new.
func (s *sourceHeads) put(e jetstream.KeyValueEntry) bool {
if !opOf(e) {
return false
}
src, err := delivery.DecodeSource(e.Value())
if err != nil {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if s.byRev == nil {
s.byRev = map[uint64]map[string]string{}
}
if _, had := s.byRev[e.Revision()]; had {
return false
}
s.byRev[e.Revision()] = src.Heads
for len(s.byRev) > delivery.SourceHistory {
delete(s.byRev, slices.Min(slices.Collect(maps.Keys(s.byRev))))
}
return true
}
// head is the head of repository in revision rev; an error when the revision is not heard (yet, or any more).
func (s *sourceHeads) head(_ context.Context, rev uint64, repository string) (string, error) {
s.mu.Lock()
defer s.mu.Unlock()
heads, ok := s.byRev[rev]
if !ok {
return "", fmt.Errorf("source revision %d is not heard (it is built once it is)", rev)
}
for r, head := range heads {
if delivery.SameRepository(r, repository) {
return head, nil
}
}
return "", nil
}
@@ -0,0 +1,78 @@
package main
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A part does not stop for a bucket it can work without** (mesh-sdk process.EssentialBucket, OptionalBucket) (novox/hq issue 1945, met twice on 2026-10-25: the builder of
// mesh-delivery #48 exited on a refused read of the gate's source, and its reasks bucket would have done the same). A grant
// reaches the bus with a delivery, so a bucket the bus refuses, or times out on, at start is often only not granted yet:
// exiting for it stops every build until a person's hand. A bucket the builder cannot build at all without (its builds,
// the indexer's modules and manifests) still stops the start, naming the grant to look at; every other is said and left,
// and opened again at the next event that needs it where there is one.
// trunkSource is the gate's source history the candidates' previews are merged onto (sourceHeads), opened at start and,
// while it cannot be, again at the next candidate that asks a trunk head: a builder without it builds the trunk's keys,
// and a candidate's preview waits, saying why.
type trunkSource struct {
ctx context.Context
js bucketOpener
heads *sourceHeads
moved func(context.Context)
errs chan<- error
logf func(string, ...any)
mu sync.Mutex
opened bool
}
// ensure opens and settles the source's watch once, and follows it; an error leaves it closed for the next ask.
func (t *trunkSource) ensure() error {
t.mu.Lock()
defer t.mu.Unlock()
if t.opened {
return nil
}
kv, err := t.js.KeyValue(t.ctx, delivery.SourceBucket)
if err != nil {
return fmt.Errorf("the bucket %s cannot be read (%v)", delivery.SourceBucket, err)
}
w, err := kv.Watch(t.ctx, delivery.SourceMainKey, jetstream.IncludeHistory())
if err != nil {
return fmt.Errorf("the bucket %s cannot be watched (%v)", delivery.SourceBucket, err)
}
sw, err := settled(t.ctx, w, func(e jetstream.KeyValueEntry) { t.heads.put(e) })
if err != nil {
return fmt.Errorf("the bucket %s: %w", delivery.SourceBucket, err)
}
t.opened = true
go func() {
t.errs <- follow(t.ctx, sw, func(e jetstream.KeyValueEntry) {
if t.heads.put(e) && t.moved != nil {
t.moved(t.ctx)
}
})
}()
return nil
}
// start opens the source at start; one that cannot be is said, and the builder goes on.
func (t *trunkSource) start() {
if err := t.ensure(); err != nil {
t.logf("%v: the trunk's keys are built, and a candidate's preview waits until the source can be read, tried again at its next candidate", err)
}
}
// head is Builder.Trunk: the head of repository at source revision rev, opening the source first if it is not yet.
func (t *trunkSource) head(ctx context.Context, rev uint64, repository string) (string, error) {
if err := t.ensure(); err != nil {
return "", err
}
return t.heads.head(ctx, rev, repository)
}
@@ -0,0 +1,49 @@
package main
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-builder/internal/build"
)
// queueReader is the part of the builder a queues value reads through: heldOnTheBus.
type queueReader interface {
Queue(ctx context.Context, language string, limit int) (build.SeatQueue, error)
}
// queueRead answers one value of the queues state (the tool builder_seat_queue, issue 1945 note 62): a value not yet
// answered is read now, from the stream the builder's account may read, and written back by compare-and-set at its
// revision with what was read or why it could not be. A value answered already (this builder's own write, or another
// instance's) or deleted is left.
func queueRead(ctx context.Context, r queueReader, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
a, err := build.DecodeQueueAsk(e.Value())
if err != nil {
logf("%s.%s holds what is not a queue read, left: %v", build.QueuesState, e.Key(), err)
return
}
if !a.Answered.IsZero() {
return
}
read, cancel := context.WithTimeout(ctx, 20*time.Second)
q, err := r.Queue(read, e.Key(), max(1, min(a.Limit, build.MaxQueueLimit)))
cancel()
if err != nil {
a.Error = err.Error()
}
a.Queue, a.Answered = &q, time.Now().UTC()
data, err := json.Marshal(a)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the read could not be written: %v", build.QueuesState, e.Key(), err)
}
}
@@ -0,0 +1,75 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-builder/internal/build"
)
// askAgainer is the part of the builder a reasks value asks through.
type askAgainer interface {
AskAgain(key, why string) []string
}
// reAsked answers one value of the reasks state (the tool builder_ask_again, issue 1945): a value not yet answered asks the
// key's waiting lanes again and is written back, by compare-and-set at its revision, with what the builder did. A value
// answered already (this builder's own write) or deleted is left.
func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
r, err := build.DecodeReAsk(e.Value())
if err != nil {
logf("%s.%s holds what is not an ask to ask again, left: %v", build.ReAsksState, e.Key(), err)
return
}
if r.Answer != "" {
return
}
asked := b.AskAgain(e.Key(), r.Why)
if len(asked) == 0 {
r.Answer = "no ask of this key waits in this builder now (it is not building, or its build has not reached its ask yet): nothing was asked; builder_stuck shows where it stands"
} else {
r.Answer = "asked again: the " + strings.Join(asked, ", ") + " ask, under a new id the key names"
}
r.Answered = time.Now().UTC()
logf("%s: asked again by %s (%s): %s", e.Key(), r.By, r.Why, r.Answer)
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
// emitAskedAgain says the builder's event build.asked-again (issue 1945): mesh.mod.<module>.event.build.asked-again,
// with the SDK's envelope headers and the new ask's id as the bus's dedup id.
func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred credential, e build.AskedAgain) error {
data, err := json.Marshal(e)
if err != nil {
return err
}
msg := nats.NewMsg("mesh.mod." + cred.Module + ".event." + build.AskedAgainEvent)
id := "asked-again-" + e.New
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", cred.Module)
msg.Header.Set("x-node", cred.Node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
return fmt.Errorf("saying %s for %s: %w", build.AskedAgainEvent, e.Key, err)
}
return nil
}
@@ -0,0 +1,79 @@
package main
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
)
// What the builder says it writes (novox/hq issue 1886, ADR 0522): each running instance keeps a key, named by its node,
// in its own bucket of instances (`mesh-delivery-builder_instances`, the manifest's state `instances`), saying that every
// done key it writes names where its artifacts are (ADR 0498 §1). The references come from the build modules' own pushes,
// so an instance says it only once a step of every build module it calls answered one since it started: a builder beside
// a build module from before R4e says nothing. The controller delivers from the builder's keys only while an instance
// says so; since R4g it asks no build-agent, and a module whose key is not yet done waits, named (ADR 0524 §7). The bucket is read by the controller alone, so its name and shape
// are not in mesh-sdk.
const (
instancesBucket = "mesh-delivery-builder_instances"
writesReferences = `{"v":1,"writes":["references"]}`
)
// sayStore is the bucket of instances as the builder writes it.
type sayStore interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
Delete(ctx context.Context, key string, opts ...jetstream.KVDeleteOpt) error
}
// sayer puts this instance's say once every build module in need answered a reference, and takes it away at the stop.
type sayer struct {
ctx context.Context
store sayStore
node string
need []string
logf func(string, ...any)
mu sync.Mutex
answered map[string]bool
said bool
}
func newSayer(ctx context.Context, store sayStore, node string, need []string, logf func(string, ...any)) *sayer {
return &sayer{ctx: ctx, store: store, node: node, need: need, logf: logf, answered: map[string]bool{}}
}
// Answered is told the build module whose step answered a reference (build.Builder.Answered).
func (s *sayer) Answered(module string) {
s.mu.Lock()
defer s.mu.Unlock()
s.answered[module] = true
if s.said {
return
}
for _, m := range s.need {
if !s.answered[m] {
return
}
}
if _, err := s.store.Put(s.ctx, s.node, []byte(writesReferences)); err != nil {
s.logf("saying in %s that this builder writes references: %v; said again at the next reference", instancesBucket, err)
return
}
s.said = true
s.logf("every build module answered a reference: said in %s that this builder writes references", instancesBucket)
}
// unsay takes the say away, if it was made: a builder replaced by one that does not write references leaves none.
func (s *sayer) unsay(ctx context.Context) error {
s.mu.Lock()
defer s.mu.Unlock()
if !s.said {
return nil
}
if err := s.store.Delete(ctx, s.node); err != nil {
return fmt.Errorf("taking away the say in %s: %w", instancesBucket, err)
}
s.said = false
return nil
}
@@ -0,0 +1,46 @@
package main
import (
"context"
"encoding/json"
"testing"
"github.com/nats-io/nats.go/jetstream"
)
type saysOf map[string]string
func (s saysOf) Put(_ context.Context, key string, value []byte) (uint64, error) {
s[key] = string(value)
return 1, nil
}
func (s saysOf) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(s, key)
return nil
}
// 1886/AC5: a builder says, under its node, that it writes references only once every build module it calls answered
// one; stopped, it says nothing any more.
func TestTheBuilderSaysItWritesReferencesOnceEveryBuildModuleAnsweredOne(t *testing.T) {
s := saysOf{}
say := newSayer(t.Context(), s, "shanks", []string{"mesh-build-go", "mesh-build-node", "mesh-build-image"}, t.Logf)
say.Answered("mesh-build-go")
say.Answered("mesh-build-node")
if v, ok := s["shanks"]; ok {
t.Fatalf("said before the image build module answered a reference: %q", v)
}
say.Answered("mesh-build-image")
var v struct {
Writes []string `json:"writes"`
}
if err := json.Unmarshal([]byte(s["shanks"]), &v); err != nil || len(v.Writes) != 1 || v.Writes[0] != "references" {
t.Fatalf("the builder's say: %q %v", s["shanks"], err)
}
if err := say.unsay(t.Context()); err != nil {
t.Fatal(err)
}
if _, ok := s["shanks"]; ok {
t.Fatalf("the stopped builder still says: %v", s)
}
}
@@ -1,28 +0,0 @@
package main
import (
"encoding/json"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"os"
)
// capacityFrom is the build capacity of each node (ADR 0478 clause 3): the module's setting `capacity`, `<node>=<steps at
// once>` pairs separated by commas, laid into the settings file the controller composes (issue 1661); empty is one step at a
// time on every node. With no settings file named, fallback is read instead, for a run outside the module's launch.
func capacityFrom(path, fallback string) (map[string]int, error) {
if path == "" {
return buildrun.ParseCapacity(fallback)
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("the settings file: %w", err)
}
var s struct {
Capacity string `json:"capacity"`
}
if err := json.Unmarshal(raw, &s); err != nil {
return nil, fmt.Errorf("the settings file %s is not JSON: %w", path, err)
}
return buildrun.ParseCapacity(s.Capacity)
}
@@ -1,67 +0,0 @@
package main
import (
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"os"
"path/filepath"
"testing"
)
func settingsFile(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "settings.json")
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
return p
}
// 1661: the build capacity is the module's setting `capacity`, laid into the settings file: empty is one step at a time everywhere.
func TestCapacityIsReadFromTheSettingsFile(t *testing.T) {
for body, want := range map[string]map[string]int{
`{"capacity": "shanks=3,ace=2"}`: {"shanks": 3, "ace": 2},
`{"capacity": ""}`: {},
`{}`: {},
} {
got, err := capacityFrom(settingsFile(t, body), "")
if err != nil || len(got) != len(want) || got["shanks"] != want["shanks"] || got["ace"] != want["ace"] {
t.Errorf("%s: capacity %v, %v, want %v", body, got, err, want)
}
}
p := &buildrun.Places{Capacity: mustCapacity(t, settingsFile(t, `{"capacity": ""}`))}
if p.CapacityOf("shanks") != 1 || p.CapacityOf("ace") != 1 {
t.Errorf("an empty setting gives %d and %d, want 1 everywhere", p.CapacityOf("shanks"), p.CapacityOf("ace"))
}
}
func mustCapacity(t *testing.T, path string) map[string]int {
t.Helper()
c, err := capacityFrom(path, "")
if err != nil {
t.Fatal(err)
}
return c
}
func TestACapacitySettingThatIsNotOneIsRefusedNamingIt(t *testing.T) {
if _, err := capacityFrom(settingsFile(t, `{"capacity": "shanks=-1"}`), ""); err == nil {
t.Error("shanks=-1 accepted")
}
if c, err := capacityFrom(settingsFile(t, `{"capacity": "shanks=3,ace=2,g14=0,novox=0"}`), ""); err != nil || c["g14"] != 0 || len(c) != 4 {
t.Errorf("the starting value: %v, %v", c, err)
}
if _, err := capacityFrom(settingsFile(t, `not json`), ""); err == nil {
t.Error("a file that is not JSON accepted")
}
if _, err := capacityFrom(filepath.Join(t.TempDir(), "none"), ""); err == nil {
t.Error("a settings file that is not there accepted")
}
}
// With no settings file named (a run outside the module's launch, as the tests do), the capacity is the environment's.
func TestWithNoSettingsFileTheCapacityIsTheEnvironments(t *testing.T) {
got, err := capacityFrom("", "ace=2")
if err != nil || got["ace"] != 2 {
t.Errorf("capacity %v, %v", got, err)
}
}
@@ -0,0 +1,211 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -0,0 +1,228 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
@@ -0,0 +1,61 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// eventsStream is the stream every module's and role's events are on, where the controller makes the builder's durable
// consumer of artifacts-gone (mesh-controller internal/broker EventsStream).
const eventsStream = "EVENTS"
// toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call).
type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the controller's verb held (mesh-sdk delivery.HeldVerb; novox/hq issue
// 1937) and answers the references the store no longer holds. One it could not ask about, with none gone, is an error:
// the key is then left done, and the controller's take-in still refuses a gone blob.
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) {
raw, err := call(delivery.HeldVerb, map[string]any{"references": references})
if err != nil {
return nil, fmt.Errorf("asking %s: %w", delivery.HeldVerb, err)
}
a, err := delivery.DecodeHeldAnswer(raw, references)
if err != nil {
return nil, err
}
if len(a.Gone) == 0 && len(a.Unknown) > 0 {
var why []string
for r, w := range a.Unknown {
why = append(why, r+": "+w)
}
sort.Strings(why)
return nil, fmt.Errorf("the artifact store could not be asked about %s", strings.Join(why, "; "))
}
return a.Gone, nil
}
}
// heardGone takes one artifacts-gone event in: acknowledged, then handed to the builder, which asks the store and builds
// the key again. A body that is not one is said and acknowledged, so it is not handed over again.
func heardGone(ctx context.Context, b *build.Builder, m jetstream.Msg, logf func(string, ...any)) {
e, err := delivery.DecodeArtifactsGone(m.Data())
if ackErr := m.Ack(); ackErr != nil {
logf("acknowledging %s: %v", m.Subject(), ackErr)
}
if err != nil {
logf("%s holds what is not %s, left: %v", m.Subject(), delivery.ArtifactsGoneEvent, err)
return
}
b.ArtifactsGone(ctx, e)
}
@@ -4,11 +4,13 @@ import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"log"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -23,9 +25,11 @@ var toolchainStreams = map[string]string{buildrun.GoBuildModule: "TOOLCHAINS_GO"
// §8): it watches the buckets that exist at start, and a build module says each time it wrote its toolchains key
// (toolchain.reported, handled by Reported); a bucket not watched yet is then watched.
type toolchainWatch struct {
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
// put takes a change of a toolchains bucket in: set.Put, or the gate of the assigned build nodes before it (issue 1940).
put func(module, key string, value []byte, deleted bool) bool
changed func(ctx context.Context) // a language's entries changed: the failed builds are tried again
logger *log.Logger
errs chan error
@@ -34,7 +38,7 @@ type toolchainWatch struct {
}
func newToolchainWatch(ctx context.Context, js bucketOpener, set *buildrun.ToolchainSet, changed func(context.Context), logger *log.Logger) *toolchainWatch {
return &toolchainWatch{ctx: ctx, js: js, set: set, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
return &toolchainWatch{ctx: ctx, js: js, set: set, put: set.Put, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
}
// start watches the buckets that exist now; a missing one is said and waits for its build module's event.
@@ -60,16 +64,19 @@ func (w *toolchainWatch) ensure(module string) error {
w.logger.Printf("the bucket %s does not exist yet: it is watched when %s reports a toolchain", name, module)
return nil
case err != nil:
return fmt.Errorf("the bucket %s: %w", name, err)
// Refused or not answered (its grant not on the bus yet, issue 1945): said, and opened again at the module's next report.
w.logger.Printf("the bucket %s cannot be read (%v): %s's toolchains are not heard until its next report", name, err, module)
return nil
}
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.set.Put(module, e.Key(), e.Value(), !opOf(e)) })
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.put(module, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
process.NotWatched(w.logger.Printf, name, err, fmt.Sprintf("%s's toolchains are not heard until its next report", module))
return nil
}
w.watched[module] = true
go func() {
w.errs <- follow(w.ctx, watcher, func(e jetstream.KeyValueEntry) {
if w.set.Put(module, e.Key(), e.Value(), !opOf(e)) {
if w.put(module, e.Key(), e.Value(), !opOf(e)) {
w.changed(w.ctx)
}
})
@@ -1,3 +0,0 @@
{
"capacity": "shanks=3,ace=2,g14=0,novox=0"
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+4 -2
View File
@@ -1,5 +1,7 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876 h1:dtiIJzrX7ey1wmsdzapiFgCKDllCph7/vxfUp1ib/t8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1 h1:fzOkrz9CSp3mEjeV7NfZx8yoivI1sD1AOtIPtyyF8RI=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024020722-34f210e5eba1/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,337 @@
package build
import (
"context"
"errors"
"fmt"
"regexp"
"slices"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **An ask that can no longer be answered is asked again** (novox/hq issue 1945). A build seat's stream gives an ask up
// once it was delivered as often as its worker may (delivery.AskMaxDeliver), and a message given up on, dropped or
// terminated with no step written is answered by no build node: the lane waiting for its step would wait for ever, and on
// restart take over the same id and wait again. The builder notices it at the check bound (CheckAfter, the step bound),
// a context's bound and no timer of its own: it asks Gone, which reads the bus, and asks such an ask again under a new
// id, written into the key, said in its log and as the event build.asked-again, at most Askings times; then the key is
// failed with why. A person asks the same by hand with the tool builder_ask_again (AskAgain).
// DefaultAskings is how many times a build module's part of a build is asked, the first included, when Builder.Askings is 0.
const DefaultAskings = 3
// AskedAgainEvent is the event the builder says when it asked an ask again: mesh.mod.mesh-delivery-builder.event.<this>.
const AskedAgainEvent = "build.asked-again"
// AskedAgain is the body of AskedAgainEvent.
type AskedAgain struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp"`
BuildModule string `json:"build_module"`
Language string `json:"language"`
Old string `json:"old"`
New string `json:"new"`
Asking int `json:"asking"`
Askings int `json:"askings"`
Why string `json:"why"`
At time.Time `json:"at"`
}
// AskSeen is what the bus says of an ask whose step has not ended, as the builder may read it: its build seat's stream
// (information and messages by sequence) and its build module's steps. The builder may not read the seat workers'
// delivery counts or ack floors (granted to the delivery view alone: $JS.API.CONSUMER.INFO.<stream>.<worker>), so a
// message given up on is told by what the workers did after it: a build node took an ask asked later, of the same kind or
// of one pulled after it (issue 1945 note 62: seven messages SEAT_NODE_BUILD_GO had given up on sat ahead of a Go ask for
// five hours, each counted as waiting, so the ask was never asked again).
type AskSeen struct {
Stream string
// Kind is the ask's kind, At when it was asked (its id's time).
Kind string
At time.Time
// Step is whether a build node wrote a step for the ask (running: the step bound of buildrun.Asker judges it).
Step bool
// OnStream is whether the stream still holds the ask's message; PutAt is when it was put.
OnStream bool
PutAt time.Time
// Ahead are the messages of its subject before it on the stream.
Ahead []Queued
// Last is the ask asked last that a build node took (wrote a step for) of the ask's kind or of a kind pulled after
// it, when one was taken: every message of the kind asked before it was then delivered already, or given up on.
Last *Taken
}
// Queued is one message of a build seat's stream: its sequence, its ask's id and kind, when the ask was asked (its id's
// time, the put time when the id says none) and put, and the state of the step a build node wrote for it ("" for none).
type Queued struct {
Seq uint64 `json:"seq"`
Ask string `json:"ask"`
Kind string `json:"kind"`
At time.Time `json:"asked"`
PutAt time.Time `json:"put"`
Step string `json:"step,omitempty"`
}
// Taken is an ask a build node took: its id, kind and when it was asked, and the step's node and start.
type Taken struct {
Ask string `json:"ask"`
Kind string `json:"kind"`
At time.Time `json:"asked"`
Node string `json:"node"`
Started time.Time `json:"started"`
}
// Proves is whether a taken ask of kind taken shows that a message of kind waiting asked before it is no longer
// delivered: a holder pulls the kinds in delivery.AskKinds' order, so taking one of the same kind or of a later one means
// its worker had none of waiting's kind left to give.
func Proves(taken, waiting string) bool {
t, w := slices.Index(delivery.AskKinds, taken), slices.Index(delivery.AskKinds, waiting)
return t >= 0 && w >= 0 && t >= w
}
// Waits is whether a message ahead still waits to be delivered: no build node wrote a step for it, and no ask of its
// kind asked after it was taken. A message its worker gave up on (delivered its most times) stays on a work queue until
// its age, and is not counted.
func (s AskSeen) Waits(q Queued) bool {
return q.Step == "" && !s.passed(q.At)
}
func (s AskSeen) passed(at time.Time) bool {
return s.Last != nil && Proves(s.Last.Kind, s.Kind) && s.Last.At.After(at)
}
// Waiting is how many messages ahead still wait to be delivered (Waits).
func (s AskSeen) Waiting() int {
n := 0
for _, q := range s.Ahead {
if s.Waits(q) {
n++
}
}
return n
}
// Why is why the ask can no longer be answered, or "" when it can: it has a step, waits behind others of its kind still
// deliverable, or no build node took a later ask of its kind (the nodes are busy or paused: a backlog, never a fault,
// ADR 0525 §9).
func (s AskSeen) Why() string {
switch {
case s.Step:
return ""
case !s.OnStream:
return fmt.Sprintf("no build node wrote a step for it and %s no longer holds it: it was given up on or dropped", s.Stream)
case s.Waiting() > 0 || !s.passed(s.at()):
return ""
default:
behind := ""
if n := len(s.Ahead); n > 0 {
behind = fmt.Sprintf(" behind %d given up on or taken", n)
}
return fmt.Sprintf("it waits on %s since %s%s while %s took the %s ask %s asked later at %s: its stream no longer delivers it (given up after its most deliveries)",
s.Stream, s.PutAt.UTC().Format(time.RFC3339), behind, s.Last.Node, s.Last.Kind, s.Last.Ask, s.Last.At.UTC().Format(time.RFC3339))
}
}
func (s AskSeen) at() time.Time {
if s.At.IsZero() {
return s.PutAt
}
return s.At
}
// AskedAt is when an ask id was made (buildstep.NewID: its milliseconds in base 36 before six hex digits), or zero.
func AskedAt(id string) time.Time {
if len(id) <= 6 {
return time.Time{}
}
ms, err := strconv.ParseInt(id[:len(id)-6], 36, 64)
if err != nil || ms <= 0 {
return time.Time{}
}
return time.UnixMilli(ms)
}
// Dropper removes an ask's message from its build seat's stream (Builder.Gone may be one): the old message of an ask asked
// again, so a message given up on does not sit ahead of every later ask of its kind until its age (issue 1945 note 62).
type Dropper interface {
Drop(ctx context.Context, a delivery.StepAsk) error
}
// drop removes the old ask's message when Gone can; a refusal is said and changes nothing else: the new ask is put.
func (b *Builder) drop(ctx context.Context, ask delivery.StepAsk, old string) {
d, ok := b.Gone.(Dropper)
if !ok {
return
}
prior := ask
prior.ID = old
if err := d.Drop(ctx, prior); err != nil {
b.logf("the message of the ask %s asked again as %s stays on its stream: %v", old, ask.ID, err)
}
}
func (b *Builder) askings() int {
if b.Askings > 0 {
return b.Askings
}
return DefaultAskings
}
var askingWords = regexp.MustCompile(`asked (\S+) again \(asking (\d+)\)`)
// askingOf is the asking a key's step words name for a build module's ask, 1 when none.
func askingOf(step, buildModule string) int {
for _, m := range askingWords.FindAllStringSubmatch(step, -1) {
if m[1] == buildModule {
if n, err := strconv.Atoi(m[2]); err == nil && n > 0 {
return n
}
}
}
return 1
}
// AskAgain asks the waiting asks of a build key again now, under new ids (the tool builder_ask_again): it answers the
// languages whose lane waits, none when no lane of this builder waits on the key.
func (b *Builder) AskAgain(key, why string) []string {
b.init()
b.mu.Lock()
defer b.mu.Unlock()
var asked []string
for language, c := range b.waiting[key] {
select {
case c <- why:
default: // one is already asked
}
asked = append(asked, language)
}
return asked
}
// Waiting are the build keys whose lanes wait for a step now, each with its languages.
func (b *Builder) Waiting() map[string][]string {
b.init()
b.mu.Lock()
defer b.mu.Unlock()
out := map[string][]string{}
for key, ls := range b.waiting {
for l := range ls {
out[key] = append(out[key], l)
}
}
return out
}
func (b *Builder) listen(key, language string) chan string {
b.init()
c := make(chan string, 1)
b.mu.Lock()
if b.waiting[key] == nil {
b.waiting[key] = map[string]chan string{}
}
b.waiting[key][language] = c
b.mu.Unlock()
return c
}
func (b *Builder) unlisten(key, language string, c chan string) {
b.mu.Lock()
if b.waiting[key][language] == c {
delete(b.waiting[key], language)
if len(b.waiting[key]) == 0 {
delete(b.waiting, key)
}
}
b.mu.Unlock()
}
var (
errCheck = errors.New("the check bound passed")
errAgain = errors.New("asked again")
)
type byHand struct{ why string }
func (h byHand) Error() string { return "asked again by hand: " + h.why }
// answer waits for the step of the lane's ask, and asks it again while it can no longer be answered (Gone) or a person
// asked it again (AskAgain); it answers the step and the ask that has it.
func (b *Builder) answer(ctx context.Context, l *lane, ask delivery.StepAsk, asked *askNotes, module, fp string) (delivery.Step, delivery.StepAsk, error) {
key, _ := delivery.BuilderBuildKey(module, fp)
again := b.listen(key, ask.Language)
defer b.unlisten(key, ask.Language, again)
asking := askingOf(asked.step, l.module)
for {
step, why, err := b.waitOnce(ctx, ask, again)
switch {
case errors.Is(err, errCheck):
continue
case errors.Is(err, errAgain):
if asking >= b.askings() {
return step, ask, fmt.Errorf("the ask %s could not be answered and was asked %d times, the most it is: %s", ask.ID, asking, why)
}
asking++
old := ask.ID
ask.ID, ask.At = buildstep.NewID(), time.Now()
if err := asked.note(ctx, ask.Language, ask.ID, fmt.Sprintf("asked %s again (asking %d): %s", l.module, asking, why)); err != nil {
return step, ask, fmt.Errorf("the ask of %s asked again could not be written to the key: %w", l.module, err)
}
if err := b.Asks.Put(ctx, ask); err != nil {
return step, ask, err
}
b.drop(ctx, ask, old)
b.logf("%s: the ask %s of %s is asked again as %s (asking %d of %d): %s", key, old, l.module, ask.ID, asking, b.askings(), why)
if b.Said != nil {
b.Said(AskedAgain{Key: key, Module: module, FP: fp, BuildModule: l.module, Language: ask.Language, Old: old, New: ask.ID,
Asking: asking, Askings: b.askings(), Why: why, At: ask.At})
}
default:
return step, ask, err
}
}
}
// waitOnce waits for the ask's step until it ends, a person asks it again, or the check bound passes: then it asks Gone
// and answers errAgain with why when the ask can no longer be answered, errCheck when it waits on.
func (b *Builder) waitOnce(ctx context.Context, ask delivery.StepAsk, again <-chan string) (delivery.Step, string, error) {
hand, cancel := context.WithCancelCause(ctx)
defer cancel(nil)
go func() {
select {
case why := <-again:
cancel(byHand{why})
case <-hand.Done():
}
}()
wait := hand
if b.Gone != nil && b.CheckAfter > 0 {
var stop context.CancelFunc
wait, stop = context.WithTimeoutCause(hand, b.CheckAfter, errCheck)
defer stop()
}
step, err := b.Asks.Wait(wait, ask)
if err == nil || ctx.Err() != nil {
return step, "", err
}
var h byHand
if errors.As(context.Cause(hand), &h) {
return step, strings.TrimSpace("asked again by hand: " + h.why), errAgain
}
if errors.Is(context.Cause(wait), errCheck) {
why, gerr := b.Gone.Gone(ctx, ask)
if gerr != nil {
b.logf("whether the ask %s can still be answered cannot be read, so it is waited for: %v", ask.ID, gerr)
return step, "", errCheck
}
if why == "" {
return step, "", errCheck
}
return step, why, errAgain
}
return step, "", err
}
@@ -73,9 +73,17 @@ func (b *Builder) considerCandidate(ctx context.Context, c delivery.Candidate) {
// The named paths' objects the preview gave it: the fetch names the paths (runLane, ADR 0499 clause 4).
m.Inputs2 = &delivery.KeptInputs2{Paths: cm.Paths}
}
// The preview is merged onto the trunk head the candidate was previewed onto: the head of the pull request's
// repository in the gate's source revision c.Base (ADR 0484 §3). Never the module's value's commit: that is the
// commit its fingerprint first appeared at, and may be many merges behind the trunk.
base, err := b.trunkHead(ctx, c.Base, p.Repository)
if err != nil {
b.logf("candidates.%s: the module %s's fingerprint %s is not built: %v", c.Group, name, cm.FP2, err)
continue
}
b.mu.Lock()
if _, onTrunk := b.trunkFP[key]; !onTrunk {
b.via[key] = via{Ref: fmt.Sprintf("refs/pull/%d/head", p.Number), Base: trunk.Commit}
b.via[key] = via{Ref: fmt.Sprintf("refs/pull/%d/head", p.Number), Base: base}
}
b.mu.Unlock()
b.consider(ctx, name, m)
@@ -106,3 +114,18 @@ func (b *Builder) agreedFor(ctx context.Context, manifest string) map[string]del
}
return out
}
// trunkHead is the head of repository in the gate's source revision rev (Builder.Trunk), or why it cannot be said.
func (b *Builder) trunkHead(ctx context.Context, rev uint64, repository string) (string, error) {
if b.Trunk == nil {
return "", fmt.Errorf("the gate's source cannot be read here, so the trunk head %s was previewed onto is not known", repository)
}
head, err := b.Trunk(ctx, rev, repository)
if err != nil {
return "", fmt.Errorf("the trunk head of %s at source revision %d cannot be read: %w", repository, rev, err)
}
if head == "" {
return "", fmt.Errorf("source revision %d names no trunk head of %s", rev, repository)
}
return head, nil
}
@@ -2,9 +2,13 @@ package build
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -40,11 +44,74 @@ func candidateValue(t *testing.T, group string, number int, head string) []byte
}
const (
headX = "1111111111111111111111111111111111111111"
headY = "2222222222222222222222222222222222222222"
// trunkHead is the trunk head the rig's source revision names: newer than the module value's commit, which stays at the
// commit its fingerprint first appeared at.
trunkHead = "3333333333333333333333333333333333333333"
headX = "1111111111111111111111111111111111111111"
headY = "2222222222222222222222222222222222222222"
)
// indexerCandidate is a candidate as the indexer writes it, with the gate's source revision it names (mesh-delivery-indexer
// internal/index testdata/candidate-written.json, kept equal to what the indexer writes by its own test).
func indexerCandidate(t *testing.T) (rev uint64, heads map[string]string, c delivery.Candidate, raw []byte) {
t.Helper()
data, err := os.ReadFile("../../../mesh-delivery-indexer/internal/index/testdata/candidate-written.json")
if err != nil {
t.Fatal(err)
}
var w struct {
Source struct {
Rev uint64 `json:"rev"`
Heads map[string]string `json:"heads"`
} `json:"source"`
Candidate json.RawMessage `json:"candidate"`
}
if err := json.Unmarshal(data, &w); err != nil {
t.Fatal(err)
}
if c, err = delivery.DecodeCandidate(w.Candidate); err != nil {
t.Fatal(err)
}
return w.Source.Rev, w.Source.Heads, c, w.Candidate
}
func TestACandidatesFingerprintIsBuiltOnceAsAKeyOfItsOwnFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1, N7/AC2 (issue 1684)
// The candidate the indexer writes is merged onto the trunk head of the source revision it names, never onto the module
// value's commit, which stays at the commit its fingerprint first appeared at (review of the indexer writing on change).
{
rev, heads, c, raw := indexerCandidate(t)
p := c.Pulls[0]
cm := p.Fps["avahi"]
ir := newRig(t, goBundle)
ir.mf.byDigest[cm.Manifest] = catalogue.Manifest{Module: "avahi", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
ir.b.Trunk = func(_ context.Context, at uint64, repository string) (string, error) {
if at != rev {
return "", fmt.Errorf("source revision %d is not heard", at)
}
return heads[repository], nil
}
const valueCommit = "7a74cee83f88e6585a5511c3cc53bdf0f5303d20"
value, err := delivery.EncodeModule(delivery.Module{Repository: p.Repository, Commit: valueCommit, FP2: "src2:" + hex64('e'),
Manifest: cm.Manifest, Path: "modules/avahi", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
ir.b.Module(context.Background(), "avahi", value)
ir.b.Wait()
ir.bm.mu.Lock()
ir.bm.calls = nil
ir.bm.mu.Unlock()
ir.b.Candidate(context.Background(), "feat%2Fx", raw)
ir.b.Wait()
fetches := ir.bm.callsOf("build_go_fetch")
if len(fetches) != 1 {
t.Fatalf("%d fetches of the indexer's candidate, want one", len(fetches))
}
f := fetches[0].body.(map[string]any)
if want := heads[p.Repository]; want == "" || f["base"] != want || f["base"] == valueCommit || f["commit"] != p.Head.Head {
t.Errorf("fetch args = %+v, want the head merged onto the trunk head %.8s of source revision %d", f, want, rev)
}
}
r := newRig(t, goBundle)
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
@@ -58,8 +125,8 @@ func TestACandidatesFingerprintIsBuiltOnceAsAKeyOfItsOwnFromTheHeadMergedOntoThe
t.Fatalf("%d fetches, want one", len(fetches))
}
f := fetches[0].body.(map[string]any)
if f["commit"] != headX || f["ref"] != "refs/pull/7/head" || f["base"] != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" || f["repository"] != "novox/mesh-catalog" {
t.Errorf("fetch args = %+v, want the head, its ref and the trunk commit to merge onto", f)
if f["commit"] != headX || f["ref"] != "refs/pull/7/head" || f["base"] != trunkHead || f["repository"] != "novox/mesh-catalog" {
t.Errorf("fetch args = %+v, want the head, its ref and the trunk head the candidate was previewed onto", f)
}
if b := r.bm.callsOf("build_go_build")[0].body.(map[string]any); b["module"] != "modules/m" {
t.Errorf("build args = %+v", b)
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
@@ -37,6 +38,11 @@ type Claim struct {
Key string
Rev uint64
Build delivery.BuilderBuild
// Gone are the references of a done key the artifact store no longer holds, for which it was claimed again
// (ClaimOptions.Stored; issue 1937): said in the log and the key's step.
Gone []string
// Unasked is why the store could not be asked about a done key taken as it stands to publish: said in the log.
Unasked error
}
// ClaimOptions say what the builder knows about the moment of a claim.
@@ -55,6 +61,30 @@ type ClaimOptions struct {
// a package's reference. A key queued waiting for a version (BuilderBuild.Waits) is taken only once it is (ADR 0500
// clause 6). Nil means none is.
Published func(ref string) bool
// Stored answers which of the references given the artifact store no longer holds (the controller's verb held,
// mesh-sdk delivery.HeldVerb; novox/hq issue 1937). A done key of the trunk's fingerprint is asked before it is left
// done: the store's nightly collector deletes a blob nothing holds, and a key the controller never took in holds
// nothing, so a key whose blob is gone is claimed building as a new revision and built again. Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
}
// StoredReferences are the references of a done build's bundles and archives that name a blob in the artifact store
// (`artifact-store://<module>/<artifact>/blobs/<digest>`): what the store's collector can delete. An image or an upstream
// copy is a manifest, which the collector keeps, and a package lives in the package registry.
func StoredReferences(b delivery.BuilderBuild) []string {
var out []string
for _, a := range b.Artifacts {
if strings.HasPrefix(a.Reference, "artifact-store://") && strings.Contains(a.Reference, "/blobs/") &&
!slices.Contains(out, a.Reference) {
out = append(out, a.Reference)
}
}
return out
}
// GoneStep is the step a key claimed again for gone blobs says (issue 1937).
func GoneStep(gone []string) string {
return "claimed again: the artifact store no longer holds " + strings.Join(gone, ", ")
}
// Unpublished is whether a done build holds an image built without a push or a package built without a publish: it has
@@ -118,6 +148,8 @@ func Unreferenced(b delivery.BuilderBuild) bool {
// names it (ClaimOptions.Trunk): that is taken as it stands, with the done value, and its builder writes the new
// revision when the publishing build ends (ADR 0498 clause 6). From R4e one node builds each fingerprint (ADR 0478
// clause 4), so a key built on one node is never built again for a second;
// - a done key of the trunk's fingerprint whose bundle or archive blob the artifact store no longer holds
// (ClaimOptions.Stored, issue 1937) is claimed building as a new revision and built whole, its step saying which;
// - a failed key is taken again as a new revision of the same key (ADR 0429 clause 18), except one failed for a kind no
// builder builds, for an upstream pinned by no digest or a package version taken (delivery.FailedUntilChanged: ADR 0478
// clause 2, ADR 0498 clauses 4 and 5) or for digests that differ (buildrun.DigestsDifferPrefix, clause 4),
@@ -149,6 +181,28 @@ func (c Claimer) Claim(ctx context.Context, module, fingerprint, manifest string
if Unreferenced(cur) {
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
}
if opt.Trunk && opt.Stored != nil {
if refs := StoredReferences(cur); len(refs) > 0 {
gone, err := opt.Stored(ctx, refs)
if err != nil {
err = fmt.Errorf("%s: whether the artifact store still holds %s cannot be asked, so the key is "+
"left done (the controller's take-in refuses it if one is gone): %w", key, strings.Join(refs, ", "), err)
if len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur, Unasked: err}, nil
}
return nil, err
}
if len(gone) > 0 {
again := building
again.Step = GoneStep(gone)
cl, err := c.write(ctx, key, again, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
if cl != nil {
cl.Gone = gone
}
return cl, err
}
}
}
if opt.Trunk && len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur}, nil
}
@@ -1,10 +1,14 @@
package build
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -25,17 +29,31 @@ func captured(t *testing.T, name string) json.RawMessage {
return step.Result
}
// The captured image steps, run as a pulled ask runs them (buildstep.RunAsk), answer what the builder reads: the trunk's
// build its digest and reference, a candidate's its digest and no reference (issue 1887 A4a).
func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // review r2, mesh-catalog 2; ADR 0487 clause 1
var f fetchResult
if err := json.Unmarshal(captured(t, "fetch.json"), &f); err != nil || f.Workspace == "" {
t.Errorf("fetch = %+v, %v, want a workspace (the field the builder reads)", f, err)
}
var trunk buildResult
if err := json.Unmarshal(captured(t, "build-trunk.json"), &trunk); err != nil || trunk.Digest == "" || trunk.Reference == "" {
t.Errorf("trunk build = %+v, %v, want a digest and a reference", trunk, err)
}
var cand buildResult
if err := json.Unmarshal(captured(t, "build-candidate.json"), &cand); err != nil || cand.Digest == "" || cand.Reference != "" {
t.Errorf("candidate build = %+v, %v, want a digest and no reference", cand, err)
for _, tc := range []struct {
build string
publish bool
ref bool
}{{"build-trunk.json", true, true}, {"build-candidate.json", false, false}} {
steps := buildstep.AskSteps{
Fetch: func(context.Context, map[string]any) (any, error) {
return json.RawMessage(captured(t, "fetch.json")), nil
},
Build: func(context.Context, map[string]any) (any, error) { return json.RawMessage(captured(t, tc.build)), nil },
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskDelivery, Language: delivery.AskImage, Asker: delivery.BuilderModule,
At: time.Now(), Build: &delivery.BuildJob{Module: "m", Path: "modules/m", Publish: tc.publish,
Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "server"}}}}
out, err := buildstep.RunAsk(context.Background(), steps, ask)
if err != nil {
t.Fatalf("%s: %v", tc.build, err)
}
a := out.(delivery.BuildAnswer)
if len(a.Artifacts) != 1 || a.Artifacts[0].Digest == "" || (a.Artifacts[0].Reference != "") != tc.ref {
t.Errorf("%s answered %+v, want a digest and a reference only for the trunk", tc.build, a)
}
}
}
@@ -0,0 +1,204 @@
package build
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again (novox/hq issue 1937): on 2026-10-24 the
// store's 03:30 collection deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which the
// controller had never taken in, and the builder kept them done.
// fakeHeld answers the controller's verb held: the references in gone are gone.
type fakeHeld struct {
mu sync.Mutex
gone map[string]bool
err error
asked [][]string
}
func (f *fakeHeld) stored(_ context.Context, refs []string) ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.asked = append(f.asked, slices.Clone(refs))
if f.err != nil {
return nil, f.err
}
var out []string
for _, r := range refs {
if f.gone[r] {
out = append(out, r)
}
}
return out, nil
}
func (f *fakeHeld) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.asked)
}
// logged keeps what the builder says.
type logged struct {
mu sync.Mutex
lines []string
}
func (l *logged) logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *logged) has(words ...string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
all := true
for _, w := range words {
all = all && strings.Contains(line, w)
}
if all {
return true
}
}
return false
}
const goneRef = "artifact-store://m/tool/blobs/" + digest
func doneKeyNaming(ref string) delivery.BuilderBuild {
return delivery.BuilderBuild{State: delivery.BuildDone, By: "shanks.builder", Manifest: manifestD, Step: "built mesh-build-go on shanks",
Artifacts: []delivery.BuiltArtifact{{Name: "tool", Kind: "bundle", Digest: digest, Reference: ref}}}
}
// 1937/AC3 (unit, the claim): a done key of the trunk's fingerprint whose blob is gone is claimed building as a new
// revision, its step naming the gone reference; a candidate's done key is not asked about.
func TestADoneTrunkKeyWhoseBlobIsGoneIsClaimedAgain(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
rev := putBuild(t, st, key, doneKeyNaming(goneRef))
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Stored: held.stored})
if err != nil || cl != nil || held.asks() != 0 {
t.Fatalf("a candidate's done key: claim %+v, %v, %d asks; want nothing and no ask", cl, err, held.asks())
}
cl, err = c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if err != nil || cl == nil {
t.Fatalf("claim = %+v, %v; want the key claimed again", cl, err)
}
if cl.Rev <= rev || cl.Build.State != delivery.BuildBuilding || !slices.Equal(cl.Gone, []string{goneRef}) {
t.Errorf("claim = %+v (revision was %d); want building as a new revision naming the gone reference", cl, rev)
}
v := valueOf(t, st, key)
if v.State != delivery.BuildBuilding || !strings.Contains(v.Step, "no longer holds") || !strings.Contains(v.Step, goneRef) {
t.Errorf("the key holds %+v; want building, its step naming the gone reference", v)
}
}
// 1937/AC5 (unit, the claim): a done trunk key whose blobs are all held stays done; a store that cannot be asked leaves
// it done and says why; images and packages are not asked about.
func TestADoneTrunkKeyStaysDoneWhileTheStoreHoldsItsBlobs(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
b := doneKeyNaming(goneRef)
b.Artifacts = append(b.Artifacts, delivery.BuiltArtifact{Name: "img", Kind: "image", Digest: digest, Reference: "registry.example/m/img@" + digest})
rev := putBuild(t, st, key, b)
held := &fakeHeld{gone: map[string]bool{}}
if cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored}); err != nil || cl != nil {
t.Fatalf("claim = %+v, %v; want nothing: the store holds the blob", cl, err)
}
if len(held.asked) != 1 || !slices.Equal(held.asked[0], []string{goneRef}) {
t.Errorf("asked %v; want the bundle's reference alone", held.asked)
}
held.err = errors.New("the controller did not answer")
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if cl != nil || err == nil || !strings.Contains(err.Error(), "cannot be asked") || !strings.Contains(err.Error(), "left done") {
t.Errorf("claim = %+v, %v; want the key left done and said", cl, err)
}
if st.revs[key] != rev {
t.Errorf("the key moved to revision %d (was %d)", st.revs[key], rev)
}
}
// 1937/AC3 (unit, the builder): the builder's look at a module whose done key's blob is gone builds it again, says so
// in a log line and in the done key's step, and asks the store once per process while it answers held.
func TestTheBuilderBuildsAgainADoneKeyWhoseBlobIsGone(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
r.b.Stored = held.stored
rev := putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again", v, r.st.revs[r.key], rev)
}
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+goneRef) {
t.Errorf("the done key's step %q does not say it was built again for the gone blob", v.Step)
}
if !log.has(r.key, "built again", goneRef) {
t.Errorf("no log line says the key was built again: %v", log.lines)
}
// The new revision's blob is held: asked once, then not again in this process.
held.mu.Lock()
held.gone = map[string]bool{}
held.mu.Unlock()
rev = r.st.revs[r.key]
asks := held.asks()
r.consider(t, "modules/m")
r.consider(t, "modules/m")
if r.st.revs[r.key] != rev {
t.Errorf("a key whose blob is held was built again")
}
if held.asks() != asks+1 {
t.Errorf("asked the store %d times for two looks at a held key; want once", held.asks()-asks)
}
}
// 1937/AC4 (unit, the builder): artifacts-gone for the module's fingerprint makes the builder ask again, even for a key
// it found held before, and build it again; one for another fingerprint builds nothing.
func TestArtifactsGoneMakesTheBuilderAskAndBuildAgain(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{}}
r.b.Stored = held.stored
putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
if v := r.consider(t, "modules/m"); v.State != delivery.BuildDone || len(r.bm.callsOf("build_go_build")) != 0 {
t.Fatalf("a key whose blob is held was built: %+v", v)
}
rev := r.st.revs[r.key]
held.mu.Lock()
held.gone[goneRef] = true // collected after the builder's look; the controller's take-in found it
held.mu.Unlock()
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Fingerprint: "src2:" + strings.Repeat("9", 64), Gone: []string{goneRef}})
r.b.Wait()
if r.st.revs[r.key] != rev || !log.has("a fingerprint the module does not have now") {
t.Fatalf("an event for another fingerprint moved the key (revision %d, was %d): %v", r.st.revs[r.key], rev, log.lines)
}
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Key: r.key, Fingerprint: fp1, Gone: []string{goneRef}})
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again on the event", v, r.st.revs[r.key], rev)
}
if !log.has("heard artifacts-gone") || !log.has(r.key, "built again", goneRef) {
t.Errorf("the builder did not say what it heard and did: %v", log.lines)
}
}
@@ -0,0 +1,115 @@
package build
import (
"encoding/json"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **The tool builder_seat_queue reads a build seat's stream as the builder reads it** (novox/hq issue 1945 note 62; the
// instruction "tools are how a module is spoken to"): the messages waiting on SEAT_NODE_BUILD_<LANGUAGE>, bounded, each
// with its ask, its age, the state of the step a build node wrote for it, and whether the builder counts it as still
// waiting (AskSeen.Waits) or as given up on. Only the builder's own account may read the stream (issue 1910): the tools
// bundle writes the read into the builder's state queues, keyed by the language, which the builder watches and answers
// there, as builder_ask_again does through reasks.
const (
SeatQueueTool = "builder_seat_queue"
// QueuesState is the builder's state of queue reads: the bucket mesh-delivery-builder_queues, keyed by the language.
QueuesState = "queues"
QueuesBucket = delivery.BuilderModule + "_" + QueuesState
// DefaultQueueLimit and MaxQueueLimit bound the messages read of each kind.
DefaultQueueLimit = 50
MaxQueueLimit = 500
)
// QueueNote says what the read cannot show.
const QueueNote = "the builder may not read the seat workers' delivery counts or ack floors ($JS.API.CONSUMER.INFO.<stream>.<worker> is " +
"granted to the delivery view alone): a message counts as given up on when no node wrote its step and a node took an ask of its kind " +
"asked after it; nats.nats_streams shows the workers' pending and waiting pulls"
// QueueAsk is a value of the queues state: a read asked (limit, who, when) and, once the builder read the stream, its answer.
type QueueAsk struct {
V int `json:"v"`
Limit int `json:"limit"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Answered time.Time `json:"answered,omitzero"`
Error string `json:"error,omitempty"`
Queue *SeatQueue `json:"queue,omitempty"`
}
// SeatQueue is what the builder read of one build seat's stream.
type SeatQueue struct {
Language string `json:"language"`
Stream string `json:"stream"`
Messages uint64 `json:"messages"`
FirstSeq uint64 `json:"first_seq"`
LastSeq uint64 `json:"last_seq"`
Kinds []KindQueue `json:"kinds"`
Note string `json:"note"`
}
// KindQueue is one kind's messages on the stream, oldest first: how many the builder counts as waiting, the ask asked
// last a build node took of the kind (or of one pulled after it), and each message.
type KindQueue struct {
Kind string `json:"kind"`
Read int `json:"read"`
Waiting int `json:"waiting"`
GivenUp int `json:"given_up_or_taken"`
Last *Taken `json:"last_taken,omitempty"`
Messages []QueuedState `json:"messages"`
}
// QueuedState is one message as the tool shows it.
type QueuedState struct {
Queued
AgeSeconds int64 `json:"age_s"`
Waits bool `json:"waits"`
}
// NewKindQueue is seen's messages ahead as one kind's queue at now.
func NewKindQueue(kind string, seen AskSeen, now time.Time) KindQueue {
k := KindQueue{Kind: kind, Read: len(seen.Ahead), Last: seen.Last}
for _, q := range seen.Ahead {
w := seen.Waits(q)
if w {
k.Waiting++
} else {
k.GivenUp++
}
k.Messages = append(k.Messages, QueuedState{Queued: q, AgeSeconds: int64(now.Sub(q.PutAt) / time.Second), Waits: w})
}
return k
}
// NewQueueAsk is the value builder_seat_queue writes for a language: its key and the value.
func NewQueueAsk(language string, limit int, by string, at time.Time) (string, []byte, error) {
if _, err := delivery.AskStream(language); err != nil {
return "", nil, fmt.Errorf("language: %w", err)
}
if limit <= 0 {
limit = DefaultQueueLimit
}
data, err := json.Marshal(QueueAsk{V: 1, Limit: min(limit, MaxQueueLimit), By: by, At: at.UTC()})
return language, data, err
}
// DecodeQueueAsk reads a queues value.
func DecodeQueueAsk(data []byte) (QueueAsk, error) {
var a QueueAsk
if err := json.Unmarshal(data, &a); err != nil {
return a, err
}
if a.V != 1 {
return a, fmt.Errorf("a queues value of version %d, not 1", a.V)
}
return a, nil
}
// LanguageOf is the language a tool's seat or language argument names: go, node or image, or node-build-<language>.
func LanguageOf(arg string) string {
return strings.TrimPrefix(strings.TrimSpace(arg), "node-build-")
}
@@ -111,7 +111,7 @@ func TestADoneKeyWrittenBeforeR4eIsBuiltAgainOnceAsANewRevision(t *testing.T) {
func TestAPinnedUpstreamIsCopiedOnceOnOneImageNodeWithNoFetchAndItsKeyNamesTheReference(t *testing.T) { // B4, ADR 0498 clause 4
up := catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest}
r := newRig(t, up)
r.b.Health = imageHave(t, "ace", "novox")
r.puller = "ace"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Fatalf("build = %+v, want done with the copy", v)
@@ -133,7 +133,7 @@ func TestAPinnedUpstreamIsCopiedOnceOnOneImageNodeWithNoFetchAndItsKeyNamesTheRe
func TestAnUpstreamBesideAnImageIsCopiedAndTheImageFetchedAndBuiltOnTheSameNode(t *testing.T) { // B4
r := newRig(t, imageArt, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
v := r.consider(t, "modules/m")
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, missing references %v", v, missing)
@@ -145,7 +145,7 @@ func TestAnUpstreamBesideAnImageIsCopiedAndTheImageFetchedAndBuiltOnTheSameNode(
func TestACopyThatFailsNotStoredFailsTheKeyAndIsBuiltAgain(t *testing.T) { // B4, ADR 0498 clause 2
r := newRig(t, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
r.bm.fails["ace/build_image_copy"] = delivery.NotStored + "up: the registry answered 500"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || delivery.FailedUntilChanged(v.Error) {
@@ -379,7 +379,26 @@ func TestACandidatesFetchNamesTheNamedPathsTheIndexerGaveIt(t *testing.T) { // A
t.Fatal("no fetch")
}
req, err := buildstep.ParseFetchRequest(f[0].body.(map[string]any))
if err != nil || req.Dir != "modules/m" || strings.Join(req.Paths, ",") != "package.json,src" || req.Base == "" {
t.Errorf("fetch request = %+v, %v; want the candidate's base, the module's directory and its named paths", req, err)
// The base is the trunk head the candidate was previewed onto, never the module value's commit (an older one).
if err != nil || req.Dir != "modules/m" || strings.Join(req.Paths, ",") != "package.json,src" || req.Base != trunkHead {
t.Errorf("fetch request = %+v, %v; want the trunk head %.8s as the base, the module's directory and its named paths", req, err, trunkHead)
}
}
// 1886/AC5: the builder tells who answered a reference: the build module whose step named one, and none for a step that
// answered a digest alone (a build module from before R4e).
func TestTheBuilderTellsWhichBuildModuleAnsweredAReference(t *testing.T) {
r := newRig(t, goBundle)
var answered []string
r.b.Answered = func(module string) { answered = append(answered, module) }
r.bm.noRef["ace/tool"] = true
r.consider(t, "modules/m")
if len(answered) != 0 {
t.Fatalf("a step that answered no reference was told as one: %v", answered)
}
delete(r.bm.noRef, "ace/tool")
r.consider(t, "modules/m")
if len(answered) != 1 || answered[0] != "mesh-build-go" {
t.Fatalf("the build module that answered a reference: %v, want mesh-build-go", answered)
}
}
@@ -3,7 +3,6 @@ package build
import (
"context"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -39,26 +38,6 @@ func TestAnArchiveOnlyModuleIsBuiltAlthoughItsFingerprintNamesNoToolchain(t *tes
}
}
func TestAnArchiveOnlyModuleNeedsNoAgreedToolchainButNeedsNodesThatQualify(t *testing.T) { // review round 4, 1
r := newRig(t, archiveA)
r.tools.Put(buildrun.GoBuildModule, "ace.go", tcValue(t, "go", "go1.26.0"), false) // the nodes disagree: a bundle would wait
r.b.Health = &buildrun.HaveSet{} // no build module is healthy anywhere
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", nil))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildFailed || !strings.Contains(v.Error, "healthy") {
t.Errorf("build = %+v, want failed: no node's build module is healthy", v)
}
have := &buildrun.HaveSet{}
have.Put("novox.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
r.b.Health = have
r.b.Reconsider(context.Background())
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || !strings.Contains(v.Step, "on novox") {
t.Errorf("build = %+v, want done on novox, the one node that qualifies", v)
}
}
func TestABundleAndAnArchiveOfOneBuildModuleStillNeedTheAgreedToolchain(t *testing.T) { // review round 4, 1
p, err := PlanOf(catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle, archiveA}}})
if err != nil || !p.NeedsToolchain(buildrun.GoBuildModule) {
@@ -80,115 +59,6 @@ func TestClaimLeavesAKeyFailedForDifferingDigestsAlone(t *testing.T) { // review
}
}
// 4. A step never waits on a full node while another qualifying node has room.
func TestChoosesMadeAtOnceSeeEachOthersReservationsSoNoLaneWaitsWhileANodeHasRoom(t *testing.T) { // review round 4, 4
p := &buildrun.Places{Capacity: map[string]int{"ace": 2, "g14": 1, "novox": 1, "shanks": 3}}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks")
total := 2 + 1 + 1 + 3
var waiting int
for i := 1; i <= 4; i++ { // four modules choose before any lane has started
room := 0
for _, n := range []string{"ace", "g14", "novox", "shanks"} {
room += p.CapacityOf(n) - p.Running()[n] - p.Reserved()[n]
}
nodes, _, err := buildrun.ChooseNodes("go", entries, p, nil, 2)
if err != nil || len(nodes) != 2 {
t.Fatalf("choice %d: %v, %v", i, nodes, err)
}
if room >= 2 {
for _, n := range nodes {
if p.Reserved()[n] > p.CapacityOf(n) {
t.Errorf("choice %d: %v put a lane on full %s while %d places were free", i, nodes, n, room)
}
}
} else {
waiting += 2 - room
}
}
if want := 4*2 - total; waiting != want {
t.Errorf("%d lanes wait, want %d: the 8 lanes less the %d places", waiting, want, total)
}
loads := p.Reserved()
for n, c := range loads {
if c > p.CapacityOf(n) && n != "shanks" && n != "ace" && n != "g14" && n != "novox" {
t.Errorf("unknown node %s", n)
}
}
if loads["shanks"] != 3 || loads["ace"] != 2 || loads["g14"]+loads["novox"] != 3 {
t.Errorf("reservations = %v, want shanks 3, ace 2 and the single-place nodes taking the rest", loads)
}
}
func TestACancelledReservationFreesTheChoiceAndAnAcquireTakesItsReservation(t *testing.T) { // review round 4, 4
p := &buildrun.Places{}
nodes, _, _ := buildrun.ChooseNodes("go", tcs("go", "go1.27.1", "ace"), p, nil, 1)
if len(nodes) != 1 || p.Reserved()["ace"] != 1 {
t.Fatalf("nodes %v, reserved %v", nodes, p.Reserved())
}
release, err := p.Acquire(context.Background(), "ace")
if err != nil || p.Reserved()["ace"] != 0 || p.Running()["ace"] != 1 {
t.Fatalf("acquire: %v, reserved %v, running %v", err, p.Reserved(), p.Running())
}
release()
buildrun.ChooseNodes("go", tcs("go", "go1.27.1", "ace"), p, nil, 1)
p.Cancel("ace")
if len(p.Reserved()) != 0 {
t.Errorf("reserved after Cancel = %v", p.Reserved())
}
}
// 5. A node whose build module is not healthy is not chosen.
func TestHaveSetKnowsOnlyBuildModulesRunningAndHealthy(t *testing.T) { // review round 4, 5
var s buildrun.HaveSet
if s.Healthy("ace", buildrun.GoBuildModule) {
t.Error("a module with no have is healthy")
}
if !s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false) || !s.Healthy("ace", buildrun.GoBuildModule) {
t.Error("a running, healthy module is not")
}
if s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false) {
t.Error("the same state counted as a change")
}
if !s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthFailed), false) || s.Healthy("ace", buildrun.GoBuildModule) {
t.Error("a module whose health failed is healthy")
}
s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
if !s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseFailed, delivery.HealthOK), false) || s.Healthy("ace", buildrun.GoBuildModule) {
t.Error("a module whose phase failed is healthy")
}
s.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
if !s.Put("ace.mesh-build-go", nil, true) || s.Healthy("ace", buildrun.GoBuildModule) {
t.Error("a deleted have is healthy")
}
if s.Put("ace.node", []byte(`{}`), false) || s.Put("ace.mesh-build-go", []byte(`not json`), false) {
t.Error("a node's key or a non-have counted")
}
}
func TestANodeWhoseBuildModuleIsNotHealthyIsNotChosen(t *testing.T) { // review round 4, 5
r := newRig(t, goBundle)
have := &buildrun.HaveSet{}
have.Put("ace.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthFailed), false) // go is not the recipe's
have.Put("novox.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
r.b.Health = have
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || !strings.Contains(v.Step, "on novox") {
t.Fatalf("build = %+v, want done on novox alone", v)
}
for _, c := range r.bm.calls {
if c.node == "ace" {
t.Errorf("a step was called on ace, whose build module is unhealthy: %+v", c)
}
}
have.Put("novox.mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthFailed), false)
r.st.vals = map[string][]byte{} // a fresh key
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildFailed || !strings.Contains(v.Error, "healthy") {
t.Errorf("build = %+v, want failed: no node has a healthy build module", v)
}
}
// 6. A key found building is taken over, whoever holds it.
func TestABuildLeftBuildingByAnyBuilderIsTakenOverAtStart(t *testing.T) { // review round 4, 6; 1607/AC3, ADR 0478 clauses 5, 6
st := newMem()
@@ -5,9 +5,11 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
@@ -20,31 +22,52 @@ type ToolchainEntries interface {
Entries(language string) []delivery.NodeToolchain
}
// toolPrefix is the start of each build module's tool names: <prefix>fetch and <prefix>build.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_", ImageBuildModule: "build_image_"}
// Builder builds, dark, every new-form fingerprint the indexer writes (novox/hq ADR 0453 clause 4, ADR 0478): it is
// handed each modules value as the watch of the indexer's bucket sees it, and for each module with a fingerprint and an
// artifact to build claims the fingerprint's build key, builds it through the build modules on the nodes
// the toolchains choose, and writes the key done or failed. It holds no timer; a build starts from a modules value or a
// artifact to build claims the fingerprint's build key, builds it by asking the build seats (one ask per build module,
// pulled by whichever build node has a free slot: ADR 0525), and writes the key done or failed. It holds no timer; a build starts from a modules value or a
// change of the toolchains.
type Builder struct {
Claimer Claimer
Runner buildrun.Runner
Toolchains ToolchainEntries
// Presence says which nodes are online (the watchman's); a node that is not is not chosen.
Presence interface{ Online(node string) bool }
Manifests Manifests
// Health says whether a node's build module is running and healthy (the node-engine's have); a node whose build module
// is not is not chosen. Nil means every node's is.
Health interface {
Healthy(node, module string) bool
Claimer Claimer
// Asks puts each build module's part of a build as one ask on its build seat's stream and waits for its step (novox/hq
// ADR 0525 §1, §3, §6; issue 1887 A4): the builder chooses no node, whichever build node pulls the ask builds it.
Asks interface {
Put(ctx context.Context, a delivery.StepAsk) error
Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, error)
}
// buildrun.Places holds a step back while its node is at its build capacity; a Builder with none runs one step at a time on each node.
Places *buildrun.Places
// Limit is how many builds run at once; four when 0.
// Held says whether an ask a key names is on its stream or a build node wrote a step for it (issue 1910): a key taken
// over at start whose ask is neither was left by a builder that stopped between naming the ask and putting it, and
// the ask is put then, under the same id. Nil takes every named ask as held.
Held interface {
Held(ctx context.Context, a delivery.StepAsk) (bool, error)
}
Toolchains ToolchainEntries
Manifests Manifests
// Limit is how many builds the builder has asked at once; their asks wait in the build seats' streams, which are the
// backlog (ADR 0525 §9), not here. 256 when 0.
Limit int
Logf func(format string, args ...any)
// Answered is told the build module whose step answered a reference (issue 1886): what the builder says it writes
// rests on it. Nil tells nobody.
Answered func(buildModule string)
// Stored answers which references the artifact store no longer holds (the controller's verb held; issue 1937): a done
// key of the trunk's fingerprint is asked before it is left done (ClaimOptions.Stored). Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
// Trunk answers the head of a repository in the gate's source revision rev (source.main at that revision): the trunk a
// candidate's preview is merged onto (Candidate.Base). Nil builds no candidate's fingerprint, and says why.
Trunk func(ctx context.Context, rev uint64, repository string) (string, error)
// Gone says why an ask with no ended step can no longer be answered, or "" when it can (issue 1945): asked once the
// ask has waited CheckAfter, and again each CheckAfter it goes on waiting. An ask that cannot be answered is asked
// again under a new id, at most Askings times. Nil, or CheckAfter 0, never asks it.
Gone interface {
Gone(ctx context.Context, a delivery.StepAsk) (string, error)
}
CheckAfter time.Duration
// Askings is how many times one build module's part of a build is asked, the first included, before its key is failed
// with why; DefaultAskings when 0.
Askings int
// Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody.
Said func(AskedAgain)
once sync.Once
sem chan struct{}
@@ -61,6 +84,11 @@ type Builder struct {
// published are the package versions `<name>@<version>` a done key of this bucket names as a package's reference: a key
// waiting for one is built once it is here (ADR 0500 clause 6).
published map[string]bool
// held are the done keys' references the store answered held, by `<key> <references>`, asked once per process: a
// blob collected later is found by the take-in, whose artifacts-gone forgets the key here (issue 1937).
held map[string]bool
// waiting are the lanes that wait for an ask's step, by build key and language: AskAgain asks through them.
waiting map[string]map[string]chan string
}
type planned struct {
@@ -72,15 +100,12 @@ func (b *Builder) init() {
b.once.Do(func() {
limit := b.Limit
if limit <= 0 {
limit = 4
limit = 256
}
b.sem = make(chan struct{}, limit)
b.cands, b.via, b.trunkFP = map[string]delivery.Candidate{}, map[string]via{}, map[string]bool{}
b.known, b.inflight, b.again, b.plans, b.tried = map[string]delivery.Module{}, map[string]bool{}, map[string]delivery.Module{}, map[string]planned{}, map[string]bool{}
b.published = map[string]bool{}
if b.Places == nil {
b.Places = &buildrun.Places{}
}
b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{}
})
}
@@ -113,8 +138,7 @@ func (b *Builder) Module(ctx context.Context, key string, value []byte) {
b.consider(ctx, key, m)
}
// NodesChanged is Reconsider after a change of the nodes: one came online or went, a build module became healthy or
// stopped being, or a toolchain moved, so a build that waited for a node may go on.
// NodesChanged is Reconsider after a toolchain moved, so a build that waited for the nodes to agree on one may go on.
func (b *Builder) NodesChanged(ctx context.Context) { b.Reconsider(ctx) }
// Sweep reads, at start, every key of the bucket: it keeps the package versions done keys name as published (a key
@@ -201,6 +225,55 @@ func (b *Builder) Forget(key string) {
b.mu.Unlock()
}
// storedOnce is Stored for one key, asked once per process while the store answers every reference held.
func (b *Builder) storedOnce(key string) func(ctx context.Context, references []string) ([]string, error) {
if b.Stored == nil {
return nil
}
return func(ctx context.Context, references []string) ([]string, error) {
id := key + " " + strings.Join(references, " ")
b.mu.Lock()
known := b.held[id]
b.mu.Unlock()
if known {
return nil, nil
}
gone, err := b.Stored(ctx, references)
if err == nil && len(gone) == 0 {
b.mu.Lock()
b.held[id] = true
b.mu.Unlock()
}
return gone, err
}
}
// ArtifactsGone takes the controller's event artifacts-gone in (issue 1937): a take-in found that the store does not
// hold what the module's key names. The module is looked at again, which asks the store (ClaimOptions.Stored) and
// builds the key again; an event for a fingerprint the module no longer has is said and left. The event only wakes the
// builder: what is built rests on the store's answer, never on the event alone.
func (b *Builder) ArtifactsGone(ctx context.Context, e delivery.ArtifactsGone) {
b.init()
b.mu.Lock()
m, ok := b.known[e.Module]
if key, err := delivery.BuilderBuildKey(e.Module, e.Fingerprint); err == nil {
for id := range b.held {
if strings.HasPrefix(id, key+" ") {
delete(b.held, id)
}
}
}
b.mu.Unlock()
if !ok || m.FP2 != e.Fingerprint {
b.logf("%s: heard %s for %s, a fingerprint the module does not have now: nothing to build", e.Module,
delivery.ArtifactsGoneEvent, e.Fingerprint)
return
}
b.logf("%s: heard %s (%s): asking the artifact store and building the key again", e.Module,
delivery.ArtifactsGoneEvent, strings.Join(e.Gone, ", "))
b.consider(ctx, e.Module, m)
}
// Reconsider looks at every module again: the toolchains changed, so a build that failed for them may build now.
func (b *Builder) Reconsider(ctx context.Context) {
b.init()
@@ -213,6 +286,13 @@ func (b *Builder) Reconsider(ctx context.Context) {
for _, k := range sortedKeys(known) {
b.consider(ctx, k, known[k])
}
b.TrunkMoved(ctx)
}
// TrunkMoved considers every candidate again once a revision of the gate's source is heard: a candidate heard before the
// revision it was previewed onto had no trunk head to be merged onto.
func (b *Builder) TrunkMoved(ctx context.Context) {
b.init()
b.mu.Lock()
cands := make([]delivery.Candidate, 0, len(b.cands))
for _, c := range b.cands {
@@ -311,7 +391,8 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
first := !b.tried[key]
b.tried[key] = true
b.mu.Unlock()
cl, err := b.Claimer.Claim(ctx, module, m.FP2, m.Manifest, ClaimOptions{TakeOver: first, ImageBuilder: b.imageBuilderRuns, Trunk: trunk, Published: b.isPublished})
cl, err := b.Claimer.Claim(ctx, module, m.FP2, m.Manifest, ClaimOptions{TakeOver: first, ImageBuilder: imageSeat, Trunk: trunk, Published: b.isPublished,
Stored: b.storedOnce(key)})
if err != nil {
b.logf("%s: %v", module, err)
return
@@ -319,6 +400,13 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
if cl == nil {
return
}
if cl.Unasked != nil {
b.logf("%v", cl.Unasked)
}
if len(cl.Gone) > 0 {
// Said aloud (issue 1937): the key was done, and the store's collector deleted what it names.
b.logf("%s: built again: the artifact store no longer holds %s", cl.Key, strings.Join(cl.Gone, ", "))
}
// A done key taken again builds only what is missing: the image or the package built unpublished as a candidate's; the
// other artifacts stay as the done key has them (ADR 0487 clauses 5 and 8, ADR 0498 clause 6).
var prior *delivery.BuilderBuild
@@ -327,7 +415,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
if cl.Build.State == delivery.BuildDone {
prior, only = &cl.Build, map[string]bool{}
if trunk {
for _, bm := range publishable(cl.Build, b.imageBuilderRuns) {
for _, bm := range publishable(cl.Build, imageSeat) {
only[bm] = true
publishing = true
}
@@ -339,7 +427,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
// any other, so the controller has a done key to register it from (issue 1613, ADR 0478 clause 1, ADR 0501 clause 1).
result = delivery.BuilderBuild{State: delivery.BuildDone, Step: "nothing to build: the manifest names no artifact"}
} else {
result = b.build(ctx, module, m, plan, planErr, prior, only)
result = b.build(ctx, module, m, plan, planErr, prior, only, b.notes(cl))
}
if ctx.Err() != nil {
return
@@ -349,6 +437,9 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
return
}
result.By, result.Manifest = b.Claimer.By, m.Manifest
if len(cl.Gone) > 0 && result.State == delivery.BuildDone {
result.Step += " (built again: the artifact store no longer held " + strings.Join(cl.Gone, ", ") + ")"
}
if publishing && result.State == delivery.BuildFailed {
// The trunk's image or package was built only as a candidate's and its push or publish failed: the key says so, so the builds view, the
// checker and the failed-key retake see it, and a look does not build the image again for nothing.
@@ -396,14 +487,15 @@ func failed(format string, args ...any) delivery.BuilderBuild {
// lane is the build of a plan's tasks of one build module on one node.
type lane struct {
module, node string
module, node string // node is the one that pulled the lane's ask
tasks []Task
toolchain *delivery.ToolchainUsed
digests map[string]string // artifact -> digest
references map[string]string // artifact -> the reference an image was pushed to (a trunk build only)
err error
}
func (b *Builder) build(ctx context.Context, module string, m delivery.Module, plan Plan, planErr error, prior *delivery.BuilderBuild, only map[string]bool) delivery.BuilderBuild {
func (b *Builder) build(ctx context.Context, module string, m delivery.Module, plan Plan, planErr error, prior *delivery.BuilderBuild, only map[string]bool, asked *askNotes) delivery.BuilderBuild {
switch {
case planErr != nil:
return failed("%s", planErr)
@@ -414,76 +506,39 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
repository := delivery.RepositoryName(m.Repository)
var lanes []*lane
used := map[string]delivery.ToolchainUsed{}
var steps []string
// Every node chosen holds a reservation until its lane takes the place; a build that ends before its lanes start gives them up.
giveUp := func(nodes ...string) {
for _, n := range nodes {
b.Places.Cancel(n)
}
}
giveUpLanes := func() {
for _, l := range lanes {
b.Places.Cancel(l.node)
}
}
var kept []string // build modules whose artifacts stay as the prior done key has them
for _, bm := range plan.Modules() {
if prior != nil && len(only) > 0 && !only[bm] {
kept = append(kept, bm)
continue
}
language := buildrun.LanguageOf[bm]
entries := b.Toolchains.Entries(language)
qualifies := func(node string) bool { return b.qualifies(node, bm) }
var nodes []string
if bm == ImageBuildModule {
// An image is built on one node, chosen from the nodes whose image build module runs healthy (ADR 0487 clause 8):
// the module has no toolchain language, so no toolchain entry names its nodes, and a second build would only
// push the image again.
ns, err := buildrun.ChooseAnyNodes("image", b.imageEntries(), b.Places, qualifies, 1)
if err != nil {
giveUpLanes()
return failed("%s no node has %s, so the image cannot be built: the provision rootless-image-build is held by the module buildah; assign buildah and %s to a node (%v)", NoImageBuilderPrefix, ImageBuildModule, ImageBuildModule, err)
}
nodes = ns
} else if plan.NeedsToolchain(bm) {
ns, tc, err := buildrun.ChooseNodes(language, entries, b.Places, qualifies, 1)
if err != nil {
giveUpLanes()
return failed("%s", err)
var toolchain *delivery.ToolchainUsed
if bm != ImageBuildModule && plan.NeedsToolchain(bm) {
// The ask names the toolchain the fingerprint was computed with; a node building with another refuses it.
language := buildrun.LanguageOf[bm]
tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language))
if none != "" {
return failed("the %s toolchain is not agreed: %s", language, none)
}
if want := m.Toolchains[language]; want.Version != tc.Version || want.Recipe != tc.Recipe {
giveUp(ns...)
giveUpLanes()
return delivery.BuilderBuild{} // the agreed toolchain moved since try looked: nothing is written
}
used[language] = delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
nodes = ns
} else {
// An archive is packed with no toolchain (the fingerprint names none): any node of the build module that qualifies.
ns, err := buildrun.ChooseAnyNodes(language, entries, b.Places, qualifies, 1)
if err != nil {
giveUpLanes()
return failed("%s", err)
}
nodes = ns
u := delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
used[language], toolchain = u, &u
}
// One node builds each fingerprint (ADR 0478 clause 4, from R4e).
steps = append(steps, fmt.Sprintf("%s on %s", bm, nodes[0]))
var tasks []Task
for _, t := range plan.Tasks {
if t.Module == bm {
tasks = append(tasks, t)
}
}
for _, n := range nodes {
lanes = append(lanes, &lane{module: bm, node: n, tasks: tasks, digests: map[string]string{}, references: map[string]string{}})
}
// One ask builds each fingerprint's part of a build module, on whichever node pulls it (ADR 0525 §3).
lanes = append(lanes, &lane{module: bm, tasks: tasks, toolchain: toolchain, digests: map[string]string{}, references: map[string]string{}})
}
var wg sync.WaitGroup
for _, l := range lanes {
wg.Add(1)
go func() { defer wg.Done(); b.runLane(ctx, l, repository, m, module) }()
go func() { defer wg.Done(); b.runLane(ctx, l, repository, m, module, asked) }()
}
wg.Wait()
var errs []string
@@ -545,6 +600,10 @@ func (b *Builder) build(ctx context.Context, module string, m delivery.Module, p
}
artifacts = append(artifacts, built)
}
var steps []string
for _, l := range lanes {
steps = append(steps, fmt.Sprintf("%s on %s", l.module, l.node))
}
for _, bm := range kept {
steps = append(steps, bm+" kept")
for lang, u := range prior.Toolchains {
@@ -614,73 +673,58 @@ func keyError(errs []string) string {
return strings.Join(errs, "; ")
}
func (b *Builder) online(node string) bool { return b.Presence == nil || b.Presence.Online(node) }
// imageSeat says an image build is asked of the image build seat, which takes it when a node pulls it: a key failed for
// want of an image builder before R4g is taken again (ADR 0525 §1).
func imageSeat() bool { return true }
// qualifies is whether the node can build with the build module now: it is online and the build module on it is healthy.
func (b *Builder) qualifies(node, bm string) bool {
return b.Places.CapacityOf(node) > 0 && b.online(node) && (b.Health == nil || b.Health.Healthy(node, bm))
// askLanguage is the build seat language of each build module (delivery.AskSubject).
var askLanguage = map[string]string{buildrun.GoBuildModule: delivery.AskGo, buildrun.NodeBuildModule: delivery.AskNode, ImageBuildModule: delivery.AskImage}
// askNotes writes a claimed key's asks into it, building, before each is put (ADR 0525 §1; issue 1887 A4b), and holds
// the asks a key taken over at start already names, so they are waited for and never put again.
type askNotes struct {
mu sync.Mutex
b *Builder
cl *Claim
resumed map[string]string
// step is the step words of a key taken over at start: an ask asked again says its asking there (askingOf).
step string
}
// imageEntries are the nodes an image may be built on, as toolchain entries for the chooser: every node the builder has a
// capacity for or a build module toolchain from. Whether the image build module runs there is qualifies', from the have.
func (b *Builder) imageEntries() []delivery.NodeToolchain {
seen := map[string]bool{}
var entries []delivery.NodeToolchain
add := func(n string) {
if !seen[n] {
seen[n] = true
entries = append(entries, delivery.NodeToolchain{Node: n})
}
func (b *Builder) notes(cl *Claim) *askNotes {
n := &askNotes{b: b, cl: cl}
if cl.Build.State == delivery.BuildBuilding && len(cl.Build.Asks) > 0 {
n.resumed, n.step = maps.Clone(cl.Build.Asks), cl.Build.Step
}
for n := range b.Places.Capacity {
add(n)
return n
}
// resume is the id of the ask a taken-over key names for language, or "".
func (n *askNotes) resume(language string) string {
n.mu.Lock()
defer n.mu.Unlock()
return n.resumed[language]
}
// note writes the ask's id into the key, building, by compare-and-set at the claim's revision.
func (n *askNotes) note(ctx context.Context, language, id, step string) error {
n.mu.Lock()
defer n.mu.Unlock()
v := n.cl.Build
v.State, v.By, v.Step, v.Error, v.Artifacts = delivery.BuildBuilding, n.b.Claimer.By, step, "", nil
v.Asks = maps.Clone(v.Asks)
if v.Asks == nil {
v.Asks = map[string]string{}
}
for _, language := range []string{"go", "typescript"} {
for _, e := range b.Toolchains.Entries(language) {
add(e.Node)
}
}
sort.Slice(entries, func(i, j int) bool { return entries[i].Node < entries[j].Node })
return entries
v.Asks[language] = id
return n.b.Claimer.Finish(ctx, n.cl, v)
}
// imageBuilderRuns is whether a node can build an image now.
func (b *Builder) imageBuilderRuns() bool {
for _, e := range b.imageEntries() {
if b.qualifies(e.Node, ImageBuildModule) {
return true
}
}
return false
}
// step runs one step of a build module; the lane that runs it holds its node's place.
func (b *Builder) step(ctx context.Context, bm, node, tool string, args map[string]any) (json.RawMessage, error) {
return b.Runner.Run(ctx, bm, node, toolPrefix[bm]+tool, args)
}
// fetchResult and buildResult are the results of a build module's fetch and build steps as the builder reads them.
type fetchResult struct {
Workspace string `json:"workspace"`
Merge string `json:"merge"`
Conflicts []string `json:"conflicts"`
}
type buildResult struct {
Digest string `json:"digest"`
Reference string `json:"reference"`
}
// runLane takes a place on the node, fetches the commit there and builds each of the lane's artifacts, one step at a time.
// An upstream artifact is copied by its pinned reference with no fetch (ADR 0498 clause 4); a lane of copies alone fetches
// nothing.
func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m delivery.Module, module string) {
release, err := b.Places.Acquire(ctx, l.node)
if err != nil {
l.err = err
return
}
defer release()
// runLane puts the lane's part of the build as one ask on its build seat's stream (the delivery stream for a trunk key, the
// check stream for a candidate's) and reads what the node that pulled it built (ADR 0525 §1, §3, §6). An upstream artifact
// is copied by its pinned reference with no fetch (ADR 0498 clause 4); a lane of copies alone fetches nothing. A lane of a
// key taken over at start waits for the ask the key names and puts none.
func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m delivery.Module, module string, asked *askNotes) {
req := buildstep.FetchRequest{Repository: repository, Commit: m.Commit}
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.mu.Lock()
@@ -699,55 +743,86 @@ func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m del
}
sort.Strings(req.Paths)
}
fetchArgs := req.Args()
candidate := req.Base != ""
var fetched fetchResult
language := askLanguage[l.module]
kind := delivery.AskDelivery
if candidate {
kind = delivery.AskCheck
}
job := &delivery.BuildJob{Module: module, Path: m.Path, FP: m.FP2, Toolchain: l.toolchain,
// Only a trunk key's build pushes an image or publishes a package; a candidate's never does (ADR 0487 clause 12,
// ADR 0498 clauses 5 and 6). A bundle or an archive is pushed either way (clause 3).
Publish: !candidate}
for _, t := range l.tasks {
a := delivery.AskArtifact{Artifact: t.Artifact}
if t.Kind == catalogue.ArtifactUpstream {
a.From = t.From // the copy runs no code of a change: the module's name, the artifact and its pin
}
job.Artifacts = append(job.Artifacts, a)
}
if needsFetch(l.tasks) {
raw, err := b.step(ctx, l.module, l.node, "fetch", fetchArgs)
if err != nil {
job.Fetch = req.Args()
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: language, Asker: delivery.BuilderModule, At: time.Now(), Build: job}
if id := asked.resume(language); id != "" {
ask.ID = id // put before this builder started again: the stream holds it, or a node already ran it
if b.Held != nil {
held, err := b.Held.Held(ctx, ask)
if err != nil {
// Said, and the key is not failed: the ask may well be on its stream and its build finish (issue 1910).
b.logf("whether the ask %s of %s the key names is on its stream cannot be read, so it is waited for as put: %v", ask.ID, l.module, err)
held = true
}
if !held {
// The builder that named it stopped before it put it (issue 1910): put it now, under the same id.
b.logf("the ask %s of %s that the key names was never put: it is put now", ask.ID, l.module)
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
}
}
} else {
if err := asked.note(ctx, language, ask.ID, "asked "+l.module); err != nil {
l.err = fmt.Errorf("the ask of %s could not be written to the key: %w", l.module, err)
return
}
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
if err := json.Unmarshal(raw, &fetched); err != nil || fetched.Workspace == "" {
l.err = fmt.Errorf("the fetch on %s answered no workspace: %s", l.node, raw)
return
}
// A candidate is built from its head merged onto the trunk. A head that does not merge ends the fetch passed with the tree at
// the trunk, and building that would store the trunk's build under the candidate's fingerprint.
if candidate {
if len(fetched.Conflicts) > 0 {
l.err = fmt.Errorf("the head does not merge into the trunk on %s: files in conflict: %s", l.node, strings.Join(fetched.Conflicts, ", "))
return
}
if fetched.Merge == "" {
l.err = fmt.Errorf("the fetch with a base on %s answered no merge: %s", l.node, raw)
return
}
}
}
step, ask, err := b.answer(ctx, l, ask, asked, module, m.FP2)
if err != nil {
l.err = fmt.Errorf("%s: %w", l.module, err)
return
}
l.node = step.Node
if step.State != delivery.StepPassed {
l.err = fmt.Errorf("%s on %s: the ask %s %s: %s", l.module, step.Node, ask.ID, step.State, step.Reason)
return
}
var answer delivery.BuildAnswer
if err := json.Unmarshal(step.Result, &answer); err != nil {
l.err = fmt.Errorf("the ask of %s on %s answered what is not a build: %s", l.module, step.Node, step.Result)
return
}
built := map[string]delivery.AskBuilt{}
for _, a := range answer.Artifacts {
built[a.Artifact] = a
}
for _, t := range l.tasks {
tool, args := "build", map[string]any{"workspace": fetched.Workspace, "module": m.Path, "artifact": t.Artifact}
if t.Kind == catalogue.ArtifactUpstream {
// The copy runs no code of a change: the module's name (the reference's path), the artifact and its pin.
tool, args = "copy", map[string]any{"module": module, "artifact": t.Artifact, "from": t.From}
} else if !candidate {
// Only a trunk key's build pushes an image or publishes a package; a candidate's never does (ADR 0487 clause 12,
// ADR 0498 clauses 5 and 6). A bundle or an archive is pushed either way (clause 3).
args["publish"] = true
}
raw, err := b.step(ctx, l.module, l.node, tool, args)
if err != nil {
l.err = err
a, ok := built[t.Artifact]
if !ok || a.Digest == "" {
l.err = fmt.Errorf("the build of %s of %s on %s answered no digest: %s", t.Artifact, module, step.Node, step.Result)
return
}
var built buildResult
if err := json.Unmarshal(raw, &built); err != nil || built.Digest == "" {
l.err = fmt.Errorf("the %s of %s of %s on %s answered no digest: %s", tool, t.Artifact, module, l.node, raw)
return
l.digests[t.Artifact] = a.Digest
if a.Reference != "" && (stored(t.Kind) || !candidate) {
l.references[t.Artifact] = a.Reference
}
l.digests[t.Artifact] = built.Digest
if built.Reference != "" && (stored(t.Kind) || !candidate) {
l.references[t.Artifact] = built.Reference
if a.Reference != "" && b.Answered != nil {
b.Answered(l.module)
}
}
}
@@ -3,13 +3,17 @@ package build
import (
"context"
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -124,22 +128,57 @@ type rig struct {
st *memStore
bm *buildModules
tools *buildrun.ToolchainSet
pres *buildrun.PresenceSet
mf *fakeManifests
key string
// asks are the build seats' streams; puller is the node that pulls every ask ("ace" when empty).
asks *buildrun.FakeAsks
puller string
}
// toolPrefix is the start of each build module's tool names, as the fake build modules answer them.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_", ImageBuildModule: "build_image_"}
// answer runs an ask as the build module that pulls it does (buildstep.RunAsk), with the fake build modules' tools on the
// rig's pulling node, and answers the step that node writes.
func (r *rig) answer(a delivery.StepAsk) delivery.Step {
node := r.puller
if node == "" {
node = "ace"
}
module, _ := delivery.BuildModuleOf(a.Language)
runner := buildrun.Runner{Caller: r.bm, Steps: r.bm.steps, Bound: time.Second}
tool := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
raw, err := runner.Run(ctx, module, node, toolPrefix[module]+name, args)
if err != nil {
return nil, err
}
return raw, nil
}
}
steps := buildstep.AskSteps{Fetch: tool("fetch"), Build: tool("build"), Copy: tool("copy")}
out, err := buildstep.RunAsk(context.Background(), steps, a)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
return step
}
step.Result, _ = json.Marshal(out)
return step
}
func newRig(t *testing.T, artifacts ...catalogue.Artifact) *rig {
t.Helper()
r := &rig{st: newMem(), bm: newBuildModules(), tools: &buildrun.ToolchainSet{}, pres: &buildrun.PresenceSet{}, mf: &fakeManifests{byDigest: map[string]catalogue.Manifest{mdigest('a'): {Module: "m", Build: &catalogue.Build{Artifacts: artifacts}}}}}
r := &rig{st: newMem(), bm: newBuildModules(), tools: &buildrun.ToolchainSet{}, mf: &fakeManifests{byDigest: map[string]catalogue.Manifest{mdigest('a'): {Module: "m", Build: &catalogue.Build{Artifacts: artifacts}}}}}
for _, n := range []string{"ace", "novox"} {
r.tools.Put(buildrun.GoBuildModule, n+".go", tcValue(t, "go", "go1.27.1"), false)
r.tools.Put(buildrun.NodeBuildModule, n+".typescript", tcValue(t, "typescript", "tsc5"), false)
r.pres.Put(n, presenceValue(t, true), false)
}
r.asks = &buildrun.FakeAsks{Answer: r.answer}
r.b = &Builder{
Claimer: Claimer{Store: r.st, By: "novox.builder"}, Runner: buildrun.Runner{Caller: r.bm, Steps: r.bm.steps, Bound: time.Second},
Toolchains: r.tools, Presence: r.pres, Manifests: r.mf, Logf: t.Logf,
Claimer: Claimer{Store: r.st, By: "novox.builder"}, Asks: buildrun.Asker{Publisher: r.asks, Waiter: r.asks, Bound: 2 * time.Second},
Toolchains: r.tools, Manifests: r.mf, Logf: t.Logf,
Trunk: func(context.Context, uint64, string) (string, error) { return trunkHead, nil },
}
var err error
if r.key, err = delivery.BuilderBuildKey("m", fp1); err != nil {
@@ -220,15 +259,53 @@ func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7
}
}
func TestOneQualifyingNodeBuildsOnce(t *testing.T) { // 1607/AC4, ADR 0478 clause 4
// 1887/AC1, AC2: a trunk fingerprint with no key is one ask on the Go seat's delivery stream, named in the building key
// before it is put, and the builder calls no node; the step a node wrote for it makes the key done naming its reference.
func TestATrunkFingerprintIsOneAskOnTheDeliveryStream(t *testing.T) {
r := newRig(t, goBundle)
r.tools.Put(buildrun.GoBuildModule, "ace.go", nil, true)
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || !strings.Contains(v.Step, "on novox") {
t.Errorf("build = %+v, want done with its step saying novox", v)
r.asks.Answer = nil // no node has pulled it yet
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
var asks []delivery.StepAsk
for deadline := time.Now().Add(2 * time.Second); len(asks) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
asks = r.asks.Asks()
}
if n := len(r.bm.callsOf("build_go_build")); n != 1 {
t.Errorf("%d builds, want one", n)
a := asks[0]
if len(asks) != 1 || a.Kind != delivery.AskDelivery || a.Language != delivery.AskGo || a.Build == nil || a.Build.FP != fp1 {
t.Fatalf("asks = %+v, want one Go delivery ask of the fingerprint", asks)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildBuilding || v.Asks[delivery.AskGo] != a.ID {
t.Fatalf("the key = %+v, want building naming the ask %s", v, a.ID)
}
if n := len(r.bm.callsOf("build_go_build")) + len(r.bm.callsOf("build_go_fetch")); n != 0 {
t.Fatalf("the builder called a node %d times", n)
}
r.asks.End(r.answer(a)) // a node pulled it and ran it
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || !strings.Contains(v.Step, "on ace") || v.Artifacts[0].Reference == "" || len(v.Asks) != 0 {
t.Fatalf("the key = %+v, want done on ace naming the reference", v)
}
}
// 1887/AC3: a builder that starts again while its ask waits puts no second ask: it waits for the step of the ask the
// building key names.
func TestARestartedBuilderWaitsForTheAskTheKeyNames(t *testing.T) {
r := newRig(t, goBundle)
const id = "mvl0frv18d1957"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
v := r.consider(t, "modules/m")
if n := len(r.asks.Asks()); n != 0 {
t.Fatalf("%d asks put again", n)
}
if v.State != delivery.BuildDone || !strings.Contains(v.Step, "on shanks") {
t.Fatalf("the key = %+v, want done from the ask's step on shanks", v)
}
}
@@ -247,20 +324,9 @@ func TestAFailingStepFailsTheBuildWithItsReasonAndNodeAndIsTriedAgainAsANewRevis
}
}
// imageHave marks mesh-build-image running and healthy on these nodes.
func imageHave(t *testing.T, nodes ...string) *buildrun.HaveSet {
t.Helper()
have := &buildrun.HaveSet{}
for _, n := range nodes {
have.Put(n+".mesh-build-go", haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
have.Put(n+"."+ImageBuildModule, haveValue(t, delivery.PhaseRunning, delivery.HealthOK), false)
}
return have
}
func TestAnImageGoesToOneNodeAndGetsOneDoneKeyWithItsReference(t *testing.T) { // 1705a, ADR 0487 clause 5
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "ace", "novox") // two nodes qualify; an image still builds on one
r.puller = "ace" // two nodes qualify; an image still builds on one
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "image" || v.Artifacts[0].Digest != digest {
t.Fatalf("build = %+v, want done with one image artifact and its digest", v)
@@ -278,19 +344,9 @@ func TestAnImageGoesToOneNodeAndGetsOneDoneKeyWithItsReference(t *testing.T) { /
}
}
func TestAnImageIsBuiltOnlyWhereMeshBuildImageIsHealthy(t *testing.T) { // 1705a
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "novox")
r.consider(t, "modules/m")
c := r.bm.callsOf("build_image_build")
if len(c) != 1 || c[0].node != "novox" {
t.Errorf("calls = %+v, want one on novox", c)
}
}
func TestABundleAndAnImageOfOneModuleBuildOnOneNodeEach(t *testing.T) { // 1705a
r := newRig(t, goBundle, imageArt)
r.b.Health = imageHave(t, "ace", "novox")
r.puller = "ace"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
@@ -303,33 +359,9 @@ func TestABundleAndAnImageOfOneModuleBuildOnOneNodeEach(t *testing.T) { // 1705a
}
}
func TestWithNoImageNodeTheKeyFailsNamingTheProvisionAndIsNotRetakenUntilANodeReportsTheModule(t *testing.T) { // 1705b, lead's decision
r := newRig(t, imageArt)
r.b.Health = imageHave(t) // mesh-build-image runs nowhere
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, NoImageBuilderPrefix) || !strings.Contains(v.Error, "rootless-image-build") || !strings.Contains(v.Error, "buildah") || !strings.Contains(v.Error, ImageBuildModule) {
t.Fatalf("build = %+v, want failed naming rootless-image-build, buildah and mesh-build-image", v)
}
if len(r.bm.calls) != 0 {
t.Errorf("%d calls, want none", len(r.bm.calls))
}
writes := r.st.writes
r.consider(t, "modules/m")
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Errorf("the failed key was written again (%d writes, was %d)", r.st.writes, writes)
}
// A node reports the module: the key is taken again.
r.b.Health = imageHave(t, "ace")
v = r.consider(t, "modules/m")
if v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done once a node reports mesh-build-image", v)
}
}
func TestACandidatesImageCallCarriesNoPublishAndItsKeyNoReference(t *testing.T) { // 1705b, lead's decision
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
@@ -348,7 +380,7 @@ func TestACandidatesImageCallCarriesNoPublishAndItsKeyNoReference(t *testing.T)
func TestATrunkImageCallCarriesPublish(t *testing.T) { // 1705b
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
r.consider(t, "modules/m")
calls := r.bm.callsOf("build_image_build")
if len(calls) != 1 || calls[0].body.(map[string]any)["publish"] != true {
@@ -439,7 +471,7 @@ func TestAKeyTheBuilderLeftBuildingIsBuiltAtStartAndDoneKeysAreNotBuiltTwice(t *
// whose compare-and-set comes first, and the other's write is refused, never overwriting it.
func TestTwoInstancesStartingTogetherEndWithOneDoneKey(t *testing.T) { // 1607/AC7, review round 4
r := newRig(t, goBundle)
other := &Builder{Claimer: Claimer{Store: r.st, By: "ace.builder"}, Runner: r.b.Runner, Toolchains: r.tools, Presence: r.pres, Manifests: r.mf, Logf: t.Logf}
other := &Builder{Claimer: Claimer{Store: r.st, By: "ace.builder"}, Asks: r.b.Asks, Toolchains: r.tools, Manifests: r.mf, Logf: t.Logf}
value := moduleValue(t, "modules/m")
var wg sync.WaitGroup
for _, b := range []*Builder{r.b, other} {
@@ -522,36 +554,9 @@ func TestARootModulesPathIsDotAndIsGivenToTheBuildTools(t *testing.T) { // 1615/
}
}
func TestANodeThatIsNotOnlineIsNotChosenAndComingBackWakesTheBuild(t *testing.T) { // 1607 review 3
r := newRig(t, goBundle)
r.tools.Put(buildrun.GoBuildModule, "g14.go", tcValue(t, "go", "go1.27.1"), false) // a laptop that reports its toolchain and sleeps
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone {
t.Fatalf("build = %+v", v)
}
for _, c := range r.bm.calls {
if c.node == "g14" {
t.Errorf("a call went to g14, which is not online: %+v", c)
}
}
// With no node online the key fails naming why; a node coming online builds it as a new revision.
r2 := newRig(t, goBundle)
r2.pres.Put("ace", nil, true)
r2.pres.Put("novox", nil, true)
if v := r2.consider(t, "modules/m"); v.State != delivery.BuildFailed || !strings.Contains(v.Error, "online") {
t.Fatalf("build = %+v, want failed because no node is online", v)
}
r2.pres.Put("ace", presenceValue(t, true), false)
r2.b.Reconsider(context.Background())
r2.b.Wait()
if v := valueOf(t, r2.st, r2.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done once a node is online", v)
}
}
func TestACandidatesImageIsPublishedOnceWhenTheTrunkNamesItsFingerprint(t *testing.T) { // review r1, mesh-delivery 1; ADR 0487 clause 5
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
trunkModule(t, r) // the trunk carries another fingerprint; fp1 is the candidate's
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
@@ -590,7 +595,7 @@ func TestACandidatesImageIsPublishedOnceWhenTheTrunkNamesItsFingerprint(t *testi
func TestAFailedPublishOfACandidatesImageIsWrittenToTheKey(t *testing.T) { // review r2, mesh-delivery 1; ADR 0487 clause 5
r := newRig(t, imageArt)
r.b.Health = imageHave(t, "ace")
r.puller = "ace"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
@@ -612,3 +617,345 @@ func TestAFailedPublishOfACandidatesImageIsWrittenToTheKey(t *testing.T) { // re
t.Errorf("build = %+v, want done with a reference", v)
}
}
// goneAsks answers Gone from a function of the ask, and counts how often it was asked; it keeps the ids it was asked to
// drop from their stream (Dropper).
type goneAsks struct {
mu sync.Mutex
n int
why func(a delivery.StepAsk) string
dropped []string
}
func (g *goneAsks) Drop(_ context.Context, a delivery.StepAsk) error {
g.mu.Lock()
defer g.mu.Unlock()
g.dropped = append(g.dropped, a.ID)
return nil
}
func (g *goneAsks) Gone(_ context.Context, a delivery.StepAsk) (string, error) {
g.mu.Lock()
g.n++
g.mu.Unlock()
return g.why(a), nil
}
func (g *goneAsks) asked() int {
g.mu.Lock()
defer g.mu.Unlock()
return g.n
}
// waitForAsks waits until n asks were put.
func waitForAsks(t *testing.T, r *rig, n int) []delivery.StepAsk {
t.Helper()
for deadline := time.Now().Add(2 * time.Second); len(r.asks.Asks()) < n; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("%d asks put, want %d", len(r.asks.Asks()), n)
}
}
return r.asks.Asks()
}
// Review round 2 item 1: a build ask that waits in its stream longer than the step bound is not failed and not asked
// again; the key is done when a node takes it. Issue 1945: an ask that can no longer be answered (its stream gave it up)
// is asked again under a new id, said, at most Askings times; a person asks it again by hand the same way.
func TestABuildAskWaitingInItsStreamIsNotBounded(t *testing.T) {
t.Run("waits in its stream: never asked again", func(t *testing.T) {
r := newRig(t, goBundle)
r.asks.Answer = nil
r.b.Asks = buildrun.Asker{Publisher: r.asks, Waiter: r.asks, Bound: 20 * time.Millisecond}
gone := &goneAsks{why: func(delivery.StepAsk) string { return "" }}
r.b.Gone, r.b.CheckAfter = gone, 5*time.Millisecond
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
waitForAsks(t, r, 1)
time.Sleep(100 * time.Millisecond)
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildBuilding {
t.Fatalf("the key = %+v while its ask waits, want building", v)
}
if gone.asked() == 0 {
t.Fatal("whether the ask can still be answered was never read at the check bound") // 1945
}
r.asks.End(r.answer(r.asks.Asks()[0]))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || len(r.asks.Asks()) != 1 {
t.Fatalf("the key = %+v after %d asks, want done from the one ask", v, len(r.asks.Asks()))
}
})
for _, c := range []struct {
name string
askings int
// step is the key's step words at start when it names an ask (a key taken over), "" for a fresh build.
step string
byHand bool
always bool // Gone says every ask cannot be answered, not only the first
puts int
done bool
asking int // the asking the event says, 0 when none is said
}{
{name: "given up by its stream: asked again under a new id and done from it", puts: 2, done: true, asking: 2},
{name: "asked again by hand: the same, with no check bound", byHand: true, puts: 2, done: true, asking: 2},
{name: "never answerable: failed after its askings", askings: 2, always: true, puts: 2},
{name: "taken over at its last asking: failed, not asked again", askings: 3, always: true, step: "asked mesh-build-go again (asking 3): x", puts: 0},
} {
t.Run(c.name, func(t *testing.T) { // 1945
r := newRig(t, goBundle)
r.asks.Answer = nil
r.b.Askings = c.askings
var said []AskedAgain
var mu sync.Mutex
r.b.Said = func(e AskedAgain) { mu.Lock(); said = append(said, e); mu.Unlock() }
first := ""
if c.step != "" {
first = "mvl0frv18d1960"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: c.step, Asks: map[string]string{delivery.AskGo: first}})
r.st.vals[r.key] = data
}
gone := &goneAsks{}
if !c.byHand {
gone.why = func(a delivery.StepAsk) string {
if c.always || a.ID == first || first == "" {
return "it waits first in line while a node took a later ask"
}
return ""
}
r.b.Gone, r.b.CheckAfter = gone, 5*time.Millisecond
}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
if c.byHand {
waitForAsks(t, r, 1)
for deadline := time.Now().Add(2 * time.Second); len(r.b.AskAgain(r.key, "stuck since 19:06")) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no lane of the key waits")
}
}
}
if c.done {
asks := waitForAsks(t, r, c.puts)
if first == "" && !c.byHand {
first = asks[0].ID
}
last := asks[len(asks)-1]
if v := valueOf(t, r.st, r.key); v.Asks[delivery.AskGo] != last.ID || !strings.Contains(v.Step, "again (asking 2)") {
t.Fatalf("the key = %+v, want building naming the ask asked again %s", v, last.ID)
}
r.asks.End(r.answer(last))
}
r.b.Wait()
v := valueOf(t, r.st, r.key)
if n := len(r.asks.Asks()); n != c.puts {
t.Errorf("%d asks put, want %d", n, c.puts)
}
if c.done && v.State != delivery.BuildDone {
t.Errorf("the key = %+v, want done from the ask asked again", v)
}
if !c.done && (v.State != delivery.BuildFailed || !strings.Contains(v.Error, "could not be answered and was asked")) {
t.Errorf("the key = %+v, want failed saying the ask could not be answered", v)
}
mu.Lock()
defer mu.Unlock()
// Note 62: the old message of an ask asked again because its stream gave it up is dropped from the stream.
if gone.mu.Lock(); c.done && !c.byHand && (len(gone.dropped) != 1 || gone.dropped[0] != first) {
t.Errorf("dropped %v from the stream, want the old ask %s", gone.dropped, first)
}
gone.mu.Unlock()
if c.asking > 0 && (len(said) != 1 || said[0].Asking != c.asking || said[0].New == said[0].Old || said[0].Key != r.key || said[0].Why == "") {
t.Errorf("said %+v, want once at asking %d", said, c.asking)
}
})
}
t.Run("what the bus says of an ask", func(t *testing.T) { // 1945
put := time.Date(2026, 10, 25, 17, 6, 0, 0, time.UTC)
taken := func(kind string, at time.Time) *Taken {
return &Taken{Ask: "later", Kind: kind, At: at, Node: "g14", Started: at}
}
later := taken(delivery.AskDelivery, put.Add(time.Minute))
// ahead is n messages put before the ask, a minute apart, each with step (its state, "" for none).
ahead := func(n int, step string) []Queued {
var qs []Queued
for i := range n {
at := put.Add(-time.Duration(n-i) * time.Minute)
qs = append(qs, Queued{Seq: uint64(i + 1), Ask: fmt.Sprint("a", i), Kind: delivery.AskDelivery, At: at, PutAt: at, Step: step})
}
return qs
}
seen := func(qs []Queued, last *Taken) AskSeen {
return AskSeen{Stream: "S", Kind: delivery.AskDelivery, At: put, OnStream: true, PutAt: put, Ahead: qs, Last: last}
}
for _, c := range []struct {
name string
seen AskSeen
want string // a part of the why, "" when it can be answered
waiting int
}{
{"a node took it", AskSeen{Stream: "S", Step: true}, "", 0},
{"no longer on its stream", AskSeen{Stream: "S"}, "no longer holds it", 0},
{"first in line, a later ask taken", seen(nil, later), "no longer delivers it", 0},
{"first in line, nothing taken since", seen(nil, nil), "", 0},
{"first in line, the ask taken last was asked before it", seen(nil, taken(delivery.AskDelivery, put.Add(-time.Second))), "", 0},
{"two wait ahead, nothing taken since", seen(ahead(2, ""), nil), "", 2},
{"two ahead passed over, the ask not yet", seen(ahead(2, ""), taken(delivery.AskDelivery, put.Add(-time.Second))), "", 0},
// Note 62: seven messages SEAT_NODE_BUILD_GO gave up on sat ahead of a Go ask for five hours.
{"seven given up on ahead, a later ask taken", seen(ahead(7, ""), later), "behind 7 given up on or taken", 0},
{"seven taken ahead (failed as poison), a later ask taken", seen(ahead(7, delivery.StepFailed), later), "no longer delivers it", 0},
{"a check ask taken later proves a delivery ask passed", seen(ahead(7, ""), taken(delivery.AskCheck, put.Add(time.Minute))), "no longer delivers it", 0},
{"a delivery ask taken later does not prove a check ask passed", func() AskSeen { s := seen(ahead(3, ""), later); s.Kind = delivery.AskCheck; return s }(), "", 3},
} {
got := c.seen.Why()
if (c.want == "") != (got == "") || !strings.Contains(got, c.want) || c.seen.Waiting() != c.waiting {
t.Errorf("%s: why %q and %d waiting, want %q and %d", c.name, got, c.seen.Waiting(), c.want, c.waiting)
}
}
if at := time.UnixMilli(1793120000123); !AskedAt(strconv.FormatInt(at.UnixMilli(), 36)+"0a1b2c").Equal(at) || !AskedAt("x").IsZero() {
t.Errorf("AskedAt does not read an ask id's time")
}
})
t.Run("the tool builder_stuck", func(t *testing.T) { // 1945
at := time.Date(2026, 10, 25, 17, 6, 0, 0, time.UTC)
state := func(vals map[string]string) StateRead {
return StateRead{Keys: func() ([]string, error) {
var ks []string
for k := range vals {
ks = append(ks, k)
}
return ks, nil
}, Get: func(k string) (json.RawMessage, time.Time, error) {
if v, ok := vals[k]; ok {
return json.RawMessage(v), at, nil
}
return nil, time.Time{}, nil
}}
}
key, _ := delivery.BuilderBuildKey("m", fp1)
done, _ := delivery.BuilderBuildKey("d", fp1)
building, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go again (asking 2): x", Asks: map[string]string{delivery.AskGo: "mvo53jcm70b5f7", delivery.AskImage: "mvo53jcm70b5f8"}})
finished, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, Manifest: manifestD, Step: "built"})
running, _ := json.Marshal(delivery.Step{V: delivery.Version, ID: "mvo53jcm70b5f8", Tool: "delivery ask", Node: "g14", State: delivery.StepRunning, Started: at})
_, reask, _ := NewReAsk("m", fp1, "stuck", "shanks.lead", at)
r := StuckReader{Builds: state(map[string]string{key: string(building), done: string(finished)}), ReAsks: state(map[string]string{key: string(reask)}),
Steps: map[string]StateRead{buildrun.GoBuildModule: state(map[string]string{}), ImageBuildModule: state(map[string]string{"g14.mvo53jcm70b5f8": string(running)})},
Now: func() time.Time { return at.Add(time.Hour) }}
out, err := r.Stuck("")
if err != nil {
t.Fatal(err)
}
stuck := out["stuck"].([]Stuck)
if len(stuck) != 1 || stuck[0].Key != key || stuck[0].Age != "1h0m0s" || stuck[0].NoneStep != 1 || stuck[0].ReAsk == nil || stuck[0].ReAsk.Why != "stuck" {
t.Fatalf("stuck = %+v, want the building key alone, an hour old, one ask with no step, with its ask again", stuck)
}
a := stuck[0].Asks
if len(a) != 2 || a[0].Language != delivery.AskGo || a[0].Asking != 2 || a[0].Step != "none" || a[0].Asked.IsZero() ||
a[1].Asking != 1 || !strings.HasPrefix(a[1].Step, "running on g14") {
t.Fatalf("asks = %+v, want go at asking 2 with no step and image at asking 1 running on g14", a)
}
// 1827: on the live shape (3000 build keys over 100 modules, 1 ms a Get, three keys of each module building) a module reads only its
// own keys, the whole answer reads MaxReads values in parallel and says it was cut, and the history (Written) is asked
// only for the rows kept.
var gets, written, inFlight, maxFlight atomic.Int64
vals := map[string]string{}
for i := 0; i < 3000; i++ {
k, _ := delivery.BuilderBuildKey(fmt.Sprintf("mod%02d", i%100), fmt.Sprintf("src2:%064x", i))
vals[k] = string(finished)
if (i/100)%10 == 0 {
vals[k] = string(building)
}
}
slow := state(vals)
get := slow.Get
slow.Get = func(k string) (json.RawMessage, time.Time, error) {
gets.Add(1)
n := inFlight.Add(1)
for m := maxFlight.Load(); n > m && !maxFlight.CompareAndSwap(m, n); m = maxFlight.Load() {
}
time.Sleep(time.Millisecond)
inFlight.Add(-1)
raw, _, err := get(k)
return raw, time.Time{}, err
}
slow.Written = func(string) time.Time { written.Add(1); return at }
for _, row := range []struct {
module string
stuck int
gets int64
cut bool
}{{"mod10", 3, 30, false}, {"", 100, MaxReads, true}} {
gets.Store(0)
written.Store(0)
out, err := StuckReader{Builds: slow, ReAsks: state(map[string]string{}), Steps: map[string]StateRead{}, Now: func() time.Time { return at }}.Stuck(row.module)
if err != nil {
t.Fatal(err)
}
_, cut := out["cut"]
if n := len(out["stuck"].([]Stuck)); n != row.stuck || cut != row.cut || written.Load() != int64(n) || gets.Load() != row.gets {
t.Errorf("%q: %d stuck, %d gets, %d written, cut %v; want %d stuck and gets, written for each, cut %v", row.module, n, gets.Load(), written.Load(), cut, row.stuck, row.cut)
}
}
if maxFlight.Load() < 2 {
t.Errorf("the values were read one at a time")
}
})
}
// heldAsks says an ask is held when its id is among held.
type heldAsks map[string]bool
func (h heldAsks) Held(_ context.Context, a delivery.StepAsk) (bool, error) { return h[a.ID], nil }
// 1910: a builder that stopped between naming the ask in its key and putting it left a key naming an ask neither the
// stream holds nor any node ran: started again, the builder puts that ask, under the same id, once, and the key is
// built from its step; an ask the stream holds is not put again.
func TestAnAskTheKeyNamesAndNobodyPutIsPutAtStart(t *testing.T) {
for _, held := range []bool{false, true} {
r := newRig(t, goBundle)
const id = "mvl0frv18d1958"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.b.Held = heldAsks{id: held}
if held {
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
}
v := r.consider(t, "modules/m")
asks := r.asks.Asks()
switch {
case held && len(asks) != 0:
t.Errorf("an ask the stream holds was put again: %+v", asks)
case !held && (len(asks) != 1 || asks[0].ID != id):
t.Errorf("the ask nobody put: asks %+v, want one under %s", asks, id)
}
if v.State != delivery.BuildDone {
t.Errorf("held %v: the key = %+v, want done", held, v)
}
}
}
// unreadableHeld cannot tell whether an ask is held.
type unreadableHeld struct{}
func (unreadableHeld) Held(context.Context, delivery.StepAsk) (bool, error) {
return false, errors.New("permissions violation for publish to $JS.API.STREAM.INFO.SEAT_NODE_BUILD_GO")
}
// 1910: a take-over that cannot read whether the ask its key names is on its stream says so and waits for the ask's step:
// the key is not failed, and the build finishes when a node ran the ask.
func TestATakeOverThatCannotReadTheStreamWaitsForTheAsk(t *testing.T) {
r := newRig(t, goBundle)
const id = "mvl0frv18d1959"
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Manifest: manifestD,
Step: "asked mesh-build-go", Asks: map[string]string{delivery.AskGo: id}})
r.st.vals[r.key] = data
r.b.Held = unreadableHeld{}
r.asks.Answer = nil
r.asks.End(delivery.Step{ID: id, Node: "shanks", State: delivery.StepPassed,
Result: []byte(`{"artifacts":[{"artifact":"tool","digest":"` + digest + `","reference":"artifact-store://m/tool/blobs/` + digest + `"}]}`)})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(r.asks.Asks()) != 0 {
t.Fatalf("the key = %+v after %d asks, want done from the ask's step, none put", v, len(r.asks.Asks()))
}
}
@@ -0,0 +1,290 @@
package build
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// The builder's tools (novox/hq issue 1945; the instruction "tools are how a module is spoken to"): builder_stuck reads
// every key that waits on an ask, builder_ask_again asks a key's build again now. The tools bundle serves them through
// the tool runner; the builder's process serves no tool (ADR 0437 §4), so builder_ask_again writes the ask into the
// builder's state reasks, which the builder watches and answers there.
const (
StuckTool = "builder_stuck"
AskAgainTool = "builder_ask_again"
// ReAsksState is the builder's state of asks to ask again: the bucket mesh-delivery-builder_reasks, keyed as the build key.
ReAsksState = "reasks"
ReAsksBucket = delivery.BuilderModule + "_" + ReAsksState
)
// ReAsk is a value of the reasks state: why a person asked the key's build again, who and when, and, once the builder
// read it, its answer.
type ReAsk struct {
V int `json:"v"`
Why string `json:"why"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Answer string `json:"answer,omitempty"`
Answered time.Time `json:"answered,omitzero"`
}
// DecodeReAsk reads a reasks value.
func DecodeReAsk(data []byte) (ReAsk, error) {
var r ReAsk
if err := json.Unmarshal(data, &r); err != nil {
return r, err
}
if r.V != 1 {
return r, fmt.Errorf("a reasks value of version %d, not 1", r.V)
}
return r, nil
}
// NewReAsk is the value builder_ask_again writes for a module's fingerprint: its key and the value.
func NewReAsk(module, fp, why, by string, at time.Time) (string, []byte, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the build is asked again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC()})
return key, data, err
}
// StateRead is a state the tools read: its keys, and a key's value with when it was written (zero when not known).
type StateRead struct {
Keys func() ([]string, error)
Get func(key string) (json.RawMessage, time.Time, error)
// Written is when the key's current value was written, from the state's history: one more round trip, so the tools ask it
// only for a row they keep, and only when Get did not say.
Written func(key string) time.Time
}
// StuckAsk is one ask a key waits on.
type StuckAsk struct {
Language string `json:"language"`
BuildModule string `json:"build_module"`
ID string `json:"id"`
Asking int `json:"asking"`
// Asked is when the ask was put, read from its id; Step is what its build module's steps hold of it: none, or its
// state on its node since its start.
Asked time.Time `json:"asked,omitzero"`
Step string `json:"step"`
}
// Stuck is one build key that waits on an ask.
type Stuck struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp"`
State string `json:"state"`
Step string `json:"step"`
By string `json:"by"`
Since time.Time `json:"since,omitzero"`
Age string `json:"age,omitempty"`
Asks []StuckAsk `json:"asks"`
ReAsk *ReAsk `json:"reask,omitempty"`
Unasked bool `json:"unasked,omitempty"`
NoneStep int `json:"asks_with_no_step"`
}
// StuckReader reads what builder_stuck answers.
type StuckReader struct {
Builds StateRead
ReAsks StateRead
// Steps are the build modules' steps states, by build module.
Steps map[string]StateRead
Now func() time.Time
// Logf, when set, gets the one line a call logs: the values it read and the time it took.
Logf func(format string, args ...any)
}
// ReadWorkers is how many values builder_stuck asks the bus for at once, and MaxReads how many it reads in a call (issue 1827:
// every value is a round trip through the runtime, so one after the other a few thousand keys outlast the call's 30 s).
// A copy of the delivery view's and the checker's: a few lines are not worth a group delivery through mesh-sdk.
const (
ReadWorkers = 16
MaxReads = 1000
)
// parallel runs read(i) for i in [0, n), at most ReadWorkers at a time, and answers the first error.
func parallel(n int, read func(i int) error) error {
var (
next atomic.Int64
failed atomic.Bool
once sync.Once
first error
wg sync.WaitGroup
)
for w := 0; w < min(ReadWorkers, n); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for !failed.Load() {
i := int(next.Add(1) - 1)
if i >= n {
return
}
if err := read(i); err != nil {
once.Do(func() { first = err })
failed.Store(true)
}
}
}()
}
wg.Wait()
return first
}
// Stuck is every key building or queued with its asks, oldest first; with module, that module's only.
func (r StuckReader) Stuck(module string) (map[string]any, error) {
started := time.Now()
reads := 0
defer func() {
if r.Logf != nil {
r.Logf("builder_stuck read %d values in %s", reads, time.Since(started).Round(time.Millisecond))
}
}()
keys, err := r.Builds.Keys()
if err != nil {
return nil, err
}
now := time.Now
if r.Now != nil {
now = r.Now
}
// Steps are found by the ask's id, which their key ends in: one index per build module, no value read until an ask names it.
stepKeys := map[string]map[string]string{}
for bm, s := range r.Steps {
ks, err := s.Keys()
if err != nil {
return nil, fmt.Errorf("the steps of %s: %w", bm, err)
}
byID := make(map[string]string, len(ks))
for _, k := range ks {
if _, id, err := delivery.ParseStepKey(k); err == nil {
byID[id] = k
}
}
stepKeys[bm] = byID
}
// Narrowed by the key (the module) before any value is read; the state is in the value, so every matching key is read, in
// parallel, at most MaxReads of them.
type parsed struct{ key, mod, fp string }
var matching []parsed
for _, key := range keys {
mod, fp, err := delivery.ParseBuilderBuildKey(key)
if err != nil || (module != "" && mod != module) {
continue
}
matching = append(matching, parsed{key, mod, fp})
}
sort.Slice(matching, func(i, j int) bool { return matching[i].key < matching[j].key })
cut := len(matching) > MaxReads
total := len(matching)
if cut {
matching = matching[:MaxReads]
}
values := make([]json.RawMessage, len(matching))
ats := make([]time.Time, len(matching))
if err := parallel(len(matching), func(i int) error {
raw, at, err := r.Builds.Get(matching[i].key)
if err == nil {
values[i], ats[i] = raw, at
}
return nil
}); err != nil {
return nil, err
}
reads += len(matching)
out := []Stuck{}
for i, m := range matching {
key, mod, fp := m.key, m.mod, m.fp
raw, at := values[i], ats[i]
if raw == nil {
continue
}
v, err := delivery.DecodeBuilderBuild(raw)
if err != nil || (v.State != delivery.BuildBuilding && v.State != delivery.BuildQueued) {
continue
}
if at.IsZero() && r.Builds.Written != nil {
at = r.Builds.Written(key)
}
s := Stuck{Key: key, Module: mod, FP: fp, State: v.State, Step: v.Step, By: v.By, Since: at, Asks: []StuckAsk{}}
if !at.IsZero() {
s.Age = now().Sub(at).Round(time.Second).String()
}
for language, id := range v.Asks {
bm, _ := delivery.BuildModuleOf(language)
a := StuckAsk{Language: language, BuildModule: bm, ID: id, Asking: askingOf(v.Step, bm), Step: "none"}
if t, ok := idTime(id); ok {
a.Asked = t
}
if st, ok := r.Steps[bm]; ok {
if k, ok := stepKeys[bm][id]; ok {
reads++
if raw, _, err := st.Get(k); err == nil && raw != nil {
if step, err := delivery.DecodeStep(raw); err == nil {
a.Step = fmt.Sprintf("%s on %s since %s", step.State, step.Node, step.Started.UTC().Format(time.RFC3339))
}
}
}
}
if a.Step == "none" {
s.NoneStep++
}
s.Asks = append(s.Asks, a)
}
sort.Slice(s.Asks, func(i, j int) bool { return s.Asks[i].Language < s.Asks[j].Language })
s.Unasked = v.State == delivery.BuildBuilding && len(v.Asks) == 0
if r.ReAsks.Get != nil {
reads++
if raw, _, err := r.ReAsks.Get(key); err == nil && raw != nil {
if ra, err := DecodeReAsk(raw); err == nil {
s.ReAsk = &ra
}
}
}
out = append(out, s)
}
sort.Slice(out, func(i, j int) bool {
if !out[i].Since.Equal(out[j].Since) {
return out[i].Since.Before(out[j].Since)
}
return out[i].Key < out[j].Key
})
answer := map[string]any{"stuck": out, "count": len(out),
"delivery_counts": "not readable by the builder: the build seats' workers' information is granted to the delivery view alone (its backlog); an ask given up on is told by the builder at its check bound and asked again"}
if cut {
answer["cut"], answer["note"] = true, fmt.Sprintf("read %d of %d build keys in key order and stopped; give a module to read its keys alone", MaxReads, total)
}
return answer, nil
}
// StuckBuildModules are the build modules whose steps builder_stuck reads.
var StuckBuildModules = []string{buildrun.GoBuildModule, buildrun.NodeBuildModule, ImageBuildModule}
// idTime is when a step id was made (buildstep.NewID: the time in milliseconds in base 36, then six hex digits).
func idTime(id string) (time.Time, bool) {
if len(id) <= 6 {
return time.Time{}, false
}
ms, err := strconv.ParseInt(id[:len(id)-6], 36, 64)
if err != nil || ms <= 0 {
return time.Time{}, false
}
return time.UnixMilli(ms).UTC(), true
}
+44 -36
View File
@@ -23,21 +23,25 @@ type state struct {
}
type artifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
System string `json:"system"`
From string `json:"from"`
Binary string `json:"binary"`
Name string `json:"name"`
Kind string `json:"kind"`
Language string `json:"language"`
System string `json:"system"`
From string `json:"from"`
Binary string `json:"binary"`
Loads []string `json:"loads"`
}
type manifest struct {
Module string `json:"module"`
Version string `json:"version"`
Slug string `json:"slug"`
Tools []string `json:"tools"`
Emits []string `json:"emits"`
Consumes []string `json:"consumes"`
State []state `json:"state"`
Reads []string `json:"reads"`
Uses []string `json:"uses"`
Invokes []string `json:"invokes"`
Config map[string]struct {
Type string `json:"type"`
@@ -83,18 +87,26 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if m.Module != "mesh-delivery-builder" {
t.Errorf("module = %q", m.Module)
}
if want := []state{{Name: "builds", History: 5}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
if want := []state{{Name: "builds", History: 5}, {Name: "instances", History: 1}, {Name: "queues", History: 1}, {Name: "reasks", History: 1}}; !reflect.DeepEqual(m.State, want) { // delivery.BuilderBuildsHistory
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) {
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
// holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5).
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
if want := []string{"mesh-build-go.build_go_build", "mesh-build-go.build_go_fetch", "mesh-build-image.build_image_build", "mesh-build-image.build_image_copy", "mesh-build-image.build_image_fetch", "mesh-build-node.build_node_build", "mesh-build-node.build_node_fetch"}; !reflect.DeepEqual(sorted(m.Invokes), want) {
t.Errorf("invokes = %v, want only the build modules' fetch and build (go, image, node) and the image module's copy (ADR 0498 clause 4)", m.Invokes)
if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
t.Errorf("uses = %v, want the three build seats", m.Uses)
}
// The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it starts.
if want := []string{"mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
// 1937/AC3: the builder asks the controller whether the artifact store still holds a done key's blobs (held), and no
// other tool; it holds no capacity.
if want := []string{"seat:mesh-controller.held"}; !reflect.DeepEqual(m.Invokes, want) || len(m.Config) != 0 {
t.Errorf("invokes = %v, config = %v, want %v and no config", m.Invokes, m.Config, want)
}
// The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it
// starts; and by the controller's artifacts-gone, a take-in refused for a gone blob (1937/AC4).
if want := []string{"mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want)
}
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
@@ -103,7 +115,14 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
if m.SecretsOwner != "mesh-delivery-builder" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
for _, k := range []string{"streams", "logs", "emits", "claims", "provides", "requires", "uses", "defines-seats", "tools", "seat", "seats", "bus"} {
// 1945: the builder's tools (builder_seat_queue: note 62), served by its tools bundle, and the event it says when it asks an ask again.
if want := []string{"builder_ask_again", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
if want := []string{"build.asked-again"}; !reflect.DeepEqual(m.Emits, want) {
t.Errorf("emits = %v, want %v", m.Emits, want)
}
for _, k := range []string{"streams", "logs", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the builder has none", k)
}
@@ -118,7 +137,7 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
procs = append(procs, r)
}
if r["type"] == "tool" || r["type"] == "seat" {
t.Errorf("resource %v: the builder serves no tool and holds no seat", r)
t.Errorf("resource %v: the builder's tools are its bundle's, and it holds no seat", r)
}
}
if len(procs) != 1 {
@@ -129,35 +148,24 @@ func TestManifestProcess(t *testing.T) { // 1607/AC1
t.Errorf("process = %v", p)
}
env, _ := p["env"].(map[string]any)
// The build capacity of each node is the module's setting `capacity` (issue 1661, ADR 0478 clause 3), laid into the
// settings file the program reads; empty is one step at a time everywhere. The environment holds no capacity.
if len(env) != 2 || env["MESH_BROKER_FILE"] != "${dir:state}/broker" || env["MESH_BUILDER_SETTINGS"] != "${dir:state}/settings.json" {
t.Errorf("env = %v", p["env"])
if len(env) != 1 || env["MESH_BROKER_FILE"] != "${dir:state}/broker" {
t.Errorf("env = %v, want the broker file alone", p["env"])
}
if !reflect.DeepEqual(p["restart-on"], []any{"settings"}) {
t.Errorf("restart-on = %v, want the settings file, so a changed capacity reaches the running program", p["restart-on"])
}
c, ok := m.Config["capacity"]
if !ok || len(m.Config) != 1 || c.Type != "string" || c.Default != "shanks=3,ace=2,g14=0,novox=0" || c.Description == "" {
t.Errorf("config = %+v, want one string setting, capacity, shanks=3,ace=2,g14=0,novox=0 to start", m.Config)
}
var settings map[string]any
for _, r := range m.Resources {
if r["id"] == "settings" {
settings = r
t.Errorf("a settings resource %v: the builder holds no setting since the capacity is the build modules' (ADR 0525 §5)", r)
}
}
if settings == nil || settings["type"] != "file" || settings["merge"] != "json" || !reflect.DeepEqual(settings["takes-config"], []any{"capacity"}) || settings["path"] != "${dir:state}/settings.json" {
t.Errorf("settings resource = %v", settings)
}
if raw, err := os.ReadFile("files/settings.json"); err != nil || !strings.Contains(string(raw), `"capacity": "shanks=3,ace=2,g14=0,novox=0"`) {
t.Errorf("files/settings.json = %q, %v, want the setting's starting value", raw, err)
}
if _, bus := p["bus"]; bus {
t.Errorf("the process declares a bus: it runs on its own broker credential")
}
if len(m.Build.Artifacts) != 2 || m.Build.Artifacts[1].Name != "files" || m.Build.Artifacts[1].Kind != "archive" || m.Build.Artifacts[1].From != "files" {
t.Fatalf("artifacts = %+v, want the program and the files", m.Build.Artifacts)
if len(m.Build.Artifacts) != 2 {
t.Fatalf("artifacts = %+v, want the program and the tools", m.Build.Artifacts)
}
if a := m.Build.Artifacts[1]; a.Name != "tools" || a.Kind != "bundle" || a.Binary != "mesh-delivery-builder-tools" || !reflect.DeepEqual(a.Loads, []string{"mesh-delivery-builder-tools"}) {
t.Errorf("tools artifact = %+v", a) // 1945
} else if st, err := os.Stat(a.From); err != nil || !st.IsDir() {
t.Errorf("the tools' source %s: %v", a.From, err)
}
a := m.Build.Artifacts[0]
if st, err := os.Stat(a.From); err != nil || !st.IsDir() || a.Binary != "mesh-delivery-builder" || !strings.HasSuffix(a.From, a.Binary) || a.Kind != "bundle" || a.Language != "go" {
+35 -40
View File
@@ -2,12 +2,27 @@
"module": "mesh-delivery-builder",
"version": "1",
"slug": "builder",
"tools": [
"builder_ask_again",
"builder_seat_queue",
"builder_stuck"
],
"emits": [
"build.asked-again"
],
"consumes": [
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported"
"mesh-build-node.toolchain.reported",
"mesh-controller.artifacts-gone"
],
"invokes": [
"seat:mesh-controller.held"
],
"state": [
{ "name": "builds", "history": 5 }
{ "name": "builds", "history": 5 },
{ "name": "instances", "history": 1 },
{ "name": "queues", "history": 1 },
{ "name": "reasks", "history": 1 }
],
"reads": [
"mesh-build-go.steps",
@@ -15,29 +30,18 @@
"mesh-build-image.steps",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-watchman.presence",
"node-engine.have"
"mesh-delivery-indexer.modules"
],
"invokes": [
"mesh-build-go.build_go_build",
"mesh-build-go.build_go_fetch",
"mesh-build-image.build_image_build",
"mesh-build-image.build_image_copy",
"mesh-build-image.build_image_fetch",
"mesh-build-node.build_node_build",
"mesh-build-node.build_node_fetch"
"uses": [
"node-build-go",
"node-build-image",
"node-build-node"
],
"config": {
"capacity": {
"type": "string",
"description": "How many build steps the builder runs at once on each node, as node=steps pairs separated by commas, such as shanks=3,ace=2 (ADR 0478 clause 3); a node set to 0 never builds, a node not named runs one at a time, and empty is one everywhere. The builder builds on shanks and ace only (the operator, 2026-10-21).",
"pattern": "|[a-z0-9_-]+=[0-9]+(,[a-z0-9_-]+=[0-9]+)*",
"default": "shanks=3,ace=2,g14=0,novox=0"
}
},
"own-secrets": {
"broker": "${dir:state}/broker"
},
@@ -58,17 +62,6 @@
"place": ".",
"unknown": "program"
},
{
"id": "settings",
"trusted": false,
"type": "file",
"path": "${dir:state}/settings.json",
"mode": "0600",
"owner": "mesh-delivery-builder",
"takes-config": ["capacity"],
"merge": "json",
"template": "files/settings.json"
},
{
"id": "builder",
"type": "process",
@@ -78,12 +71,8 @@
"./mesh-delivery-builder"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker",
"MESH_BUILDER_SETTINGS": "${dir:state}/settings.json"
"MESH_BROKER_FILE": "${dir:state}/broker"
},
"restart-on": [
"settings"
],
"user": "mesh-delivery-builder",
"health": {
"kind": "unit"
@@ -101,9 +90,15 @@
"binary": "mesh-delivery-builder"
},
{
"name": "files",
"kind": "archive",
"from": "files"
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-builder-tools",
"binary": "mesh-delivery-builder-tools",
"loads": [
"mesh-delivery-builder-tools"
]
}
]
}
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -7,6 +7,7 @@ import (
"encoding/json"
"fmt"
"os"
"sync/atomic"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
@@ -15,8 +16,35 @@ import (
"mesh-delivery-checker/internal/check"
)
// reads counts the values this call read from the bus: the runtime launches the tools for one call, so the count is the call's.
var reads atomic.Int64
// cutAnswer says in the answer that it was cut, and how to narrow it.
func cutAnswer(answer map[string]any, cut string) map[string]any {
if cut != "" {
answer["cut"], answer["note"] = true, cut
}
return answer
}
// logged runs every tool with one line to stderr per call: the tool, the values it read and the time it took.
func logged(tools []stdio.Tool) []stdio.Tool {
for i := range tools {
name, run := tools[i].Name, tools[i].Run
tools[i].Run = func(args map[string]any) (any, error) {
reads.Store(0)
start := time.Now()
out, err := run(args)
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %s read %d values in %s (error: %v)\n", name, reads.Load(), time.Since(start).Round(time.Millisecond), err)
return out, err
}
}
return tools
}
func read(s stdio.KeptState) check.StateRead {
return check.StateRead{Keys: s.Keys, Get: func(key string) (json.RawMessage, error) {
reads.Add(1)
e, err := s.Get(key)
if err != nil || e == nil {
return nil, err
@@ -50,7 +78,7 @@ func TryTools(tr Trying) []stdio.Tool {
Description: "Try a pushed head before its pull request is opened: the checker judges it as it judges a pull request's head, merged onto its trunk's head, " +
"its unit and integration levels through the build modules, and owes the forge nothing. Answers the try's id at once; " + check.TryReadTool + " reads " +
"where it stands. A try of the same head that waits or runs is the answer to a second ask; at most 4 wait or run at once, mesh-wide. A repository " +
"the checker checks no level of (hq, one whose trunk the gate does not name) is refused.",
"the checker checks no level of (one whose trunk the gate does not name) is refused.",
Input: map[string]any{"type": "object", "required": []string{"repository", "head"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"head": map[string]any{"type": "string", "description": "the full commit to try, 40 or 64 hex digits"},
@@ -100,12 +128,11 @@ func Tools(r check.Reader) []stdio.Tool {
"group": map[string]any{"type": "string", "description": "the delivery group's branch name"}}},
Run: func(args map[string]any) (any, error) {
group, _ := args["group"].(string)
vs, err := r.Verdicts(group)
return map[string]any{"verdicts": vs}, err
vs, cut, err := r.Verdicts(group)
return cutAnswer(map[string]any{"verdicts": vs}, cut), err
}},
{Name: check.PullReadTool,
Description: "The verdict that judges one pull request and the full report of its status mesh/check; for an hq pull request, that hq is judged by " +
"merge-check.sh until issue 1296. Read-only.",
Description: "The verdict that judges one pull request and the full report of its status mesh/check. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"repository", "number"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"number": map[string]any{"type": "integer", "description": "the pull request's number"}}},
@@ -123,8 +150,8 @@ func Tools(r check.Reader) []stdio.Tool {
Run: func(args map[string]any) (any, error) {
level, _ := args["level"].(string)
digest, _ := args["digest"].(string)
kept, err := r.Kept(level, digest)
return map[string]any{"kept": kept}, err
kept, cut, err := r.Kept(level, digest)
return cutAnswer(map[string]any{"kept": kept}, cut), err
}},
{Name: check.PreviewsReadTool,
Description: "The merge previews the checker holds, each as stored under previews.<repository>.<head>.<trunk>: the merge commit, per module directory " +
@@ -139,56 +166,53 @@ func Tools(r check.Reader) []stdio.Tool {
if head != "" && len(head) < 7 {
return nil, fmt.Errorf("the head %q is too short: give at least 7 characters", head)
}
ps, bad, err := r.Previews(repository, head)
return map[string]any{"previews": ps, "unreadable": bad}, err
ps, bad, cut, err := r.Previews(repository, head)
return cutAnswer(map[string]any{"previews": ps, "unreadable": bad}, cut), err
}},
{Name: check.StatusReadTool,
Description: "What holds the checker back and what runs: the capacity per node (the setting, the default one step per node), whether the controller's " +
"delivery plans in their build phase hold a key (no new check starts while one does), the steps in flight on the build modules by node, and the " +
"candidates without a verdict for their heads. The steps the checker holds waiting for a place are in its process, not in a bucket, and the " +
"answer says so. Read-only.",
Description: "What runs: the steps in flight on the build modules by node, the candidates without a verdict for their heads, " +
"and the candidates whose check is owed by kind (waiting_by_kind: new, nodes, and trunk, a re-judge pulled after the other two). A key a " +
"build module left running (started past the step bound, or its build module not running on its node) is listed apart under " +
"left_running with why, and not counted. Nothing holds the checker and it has no capacity of its own: a check is an ask on the " +
"build seats' streams, whose backlog the delivery view shows (novox/hq ADR 0525). Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) { return r.Status() }},
}
}
func capacity() (string, error) {
path := os.Getenv("MESH_CHECKER_SETTINGS")
if path == "" {
return os.Getenv("MESH_CHECKER_CAPACITY"), nil
// stepBound is the checker's step bound as the checker reads it (MESH_CHECKER_STEP_BOUND), its default when unset or not a
// duration.
func stepBound() time.Duration {
if d, err := time.ParseDuration(os.Getenv("MESH_CHECKER_STEP_BOUND")); err == nil && d > 0 {
return d
}
raw, err := os.ReadFile(path)
if err != nil {
return "", err
}
var s struct {
Capacity string `json:"capacity"`
}
return s.Capacity, json.Unmarshal(raw, &s)
return check.DefaultStepBound
}
func main() {
r := check.Reader{VerdictState: read(stdio.State("verdicts")), KeptState: read(stdio.State("kept")), CandidateState: read(stdio.State("mesh-delivery-indexer.candidates")),
PreviewState: read(stdio.State("previews")), PlansState: read(stdio.State("mesh-controller.plans-building")),
StepStates: map[string]check.StateRead{"mesh-build-go": read(stdio.State("mesh-build-go.steps")), "mesh-build-node": read(stdio.State("mesh-build-node.steps"))},
Capacity: capacity}
PreviewState: read(stdio.State("previews")),
StepStates: map[string]check.StateRead{"mesh-build-go": read(stdio.State("mesh-build-go.steps")), "mesh-build-node": read(stdio.State("mesh-build-node.steps"))},
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound()}
tries := stdio.State(check.TriesState)
source := stdio.State("mesh-delivery-gate.source")
trunks := func() (map[string]string, error) {
e, err := source.Get(delivery.SourceMainKey)
if err != nil || e == nil {
return nil, err
}
s, err := delivery.DecodeSource(e.Value)
return s.Heads, err
}
r.Trunks = trunks
trying := Trying{
Tries: stateStore{StateRead: read(tries), create: func(key string, value []byte) error {
_, err := tries.Create(key, json.RawMessage(value))
return err
}},
Trunks: func() (map[string]string, error) {
e, err := source.Get(delivery.SourceMainKey)
if err != nil || e == nil {
return nil, err
}
s, err := delivery.DecodeSource(e.Value)
return s.Heads, err
},
Trunks: trunks,
}
if err := stdio.Serve("", append(Tools(r), TryTools(trying)...)); err != nil {
if err := stdio.Serve("", logged(append(Tools(r), TryTools(trying)...))); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1)
}
@@ -105,43 +105,6 @@ func follow(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.Key
}
}
// stepsWaiter waits for a build module's step by watching its key in the module's steps bucket (ADR 0469 clause 2): the
// watch hands over the value the key holds now, so a step that ended before the watch began is read at once, and every
// change after. No timer: the caller bounds the wait with its context.
type stepsWaiter struct{ js bucketOpener }
func (w stepsWaiter) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
kv, err := w.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return delivery.Step{}, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
watch, err := kv.Watch(ctx, key)
if err != nil {
return delivery.Step{}, err
}
defer watch.Stop()
for {
select {
case <-ctx.Done():
return delivery.Step{}, ctx.Err()
case e, ok := <-watch.Updates():
if !ok {
return delivery.Step{}, errors.New("the watch of the step ended")
}
if e == nil || !opOf(e) {
continue
}
step, err := delivery.DecodeStep(e.Value())
if err != nil {
return delivery.Step{}, fmt.Errorf("the step %s is not a step: %w", key, err)
}
if step.State != delivery.StepRunning {
return step, nil
}
}
}
}
// manifestReader reads manifests.<digest> of the indexer and parses the manifest it holds.
type manifestReader struct{ kv jetstream.KeyValue }
@@ -158,5 +121,12 @@ func (m manifestReader) Manifest(ctx context.Context, digest string) (catalogue.
if err != nil {
return catalogue.Manifest{}, err
}
return catalogue.ParseManifest([]byte(kept.JSON))
return parseManifest(kept.JSON)
}
// parseManifest reads a manifest the indexer kept as the indexer and the assigner read it (issues 1751, 1906): a config value that
// falls back to a predefined key is judged by the merge gate in the catalogue, against the predefined.json at its root that this
// process never holds, so it is not unreadable here (novox/hq ADR 0348 §7).
func parseManifest(raw string) (catalogue.Manifest, error) {
return catalogue.ParseManifestForInputs([]byte(raw))
}
@@ -0,0 +1,63 @@
//go:build integration
package main
import (
"strings"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// assignerWrote raises the assigner's compositions bucket, the assigner online in the watchman's presence, and its verdict
// under the check's key, as the assigner leaves them on the bus.
func (b *bed) assignerWrote(t *testing.T, key string, v delivery.CompositionVerdict) {
t.Helper()
kv, err := b.js.CreateKeyValue(b.ctx, jetstream.KeyValueConfig{Bucket: delivery.AssignerCompositionsBucket, History: delivery.AssignerCompositionsHistory})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PresencePartKey(delivery.AssignerModule, "novox")
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
if _, err := b.buckets[delivery.PresenceBucket].Put(b.ctx, pkey, on); err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
}
// 1297/AC3: started whole with the assigner's verdict on the bus, the checker's line says the composition's verdict, not
// "not yet run": a pass names its machine, a fail fails the check with the assigner's words.
func TestTheCheckersLineSaysTheAssignersComposition(t *testing.T) {
for _, c := range []struct {
state, owed, words string
}{
{delivery.CompositionPass, "success", "composition passed"},
{delivery.CompositionFail, "failure", "g14 runs web, which the change removes from its source"},
} {
b := startBed(t)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
group, heads, _ := delivery.ParseCheckerVerdictKey(key)
v := delivery.CompositionVerdict{Group: group, Heads: heads, State: c.state, SDK: "v0.1.14-x", Machines: []string{"g14"}}
if c.state == delivery.CompositionFail {
v.Machine, v.Words = "g14", c.words
}
b.assignerWrote(t, key, v)
b.start(t, nil)
got := b.verdict(t, key, ended)
if got.Composition == nil || got.Composition.State != c.state {
t.Fatalf("%s: the verdict's composition is %+v", c.state, got.Composition)
}
o := b.owed(t, 7, head1, c.owed)
if !strings.Contains(o.Text, c.words) || strings.Contains(o.Text, "not yet run") {
t.Errorf("%s: the line says %q", c.state, o.Text)
}
}
}
@@ -0,0 +1,131 @@
package main
import (
"context"
"errors"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// compositionWatch reads the assigner's composition of a check (ADR 0528 clause 5) by watching its key and the
// assigner's presence: no poll and no timer. With no assigner online it answers check.ErrAssignerNotRunning at once.
type compositionWatch struct {
kv jetstream.KeyValue
// presence is the watchman's presence bucket; nil (not there) counts as no assigner online.
presence jetstream.KeyValue
}
// Await answers the composition under key once the assigner wrote one that does not wait (check.AwaitComposition).
func (c compositionWatch) Await(ctx context.Context, key string) (delivery.CompositionVerdict, error) {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
comps := make(chan check.CompositionEvent)
presence := make(chan check.PresenceEvent)
cw, err := c.kv.Watch(ctx, key)
if err != nil {
return delivery.CompositionVerdict{}, err
}
defer cw.Stop()
go func() {
for e := range cw.Updates() {
ev := check.CompositionEvent{Settled: e == nil}
if e != nil && e.Operation() == jetstream.KeyValuePut {
ev.Value = e.Value()
}
select {
case comps <- ev:
case <-ctx.Done():
return
}
}
}()
if c.presence == nil {
go func() {
select {
case presence <- check.PresenceEvent{Settled: true}:
case <-ctx.Done():
}
}()
} else {
pw, err := c.presence.Watch(ctx, "part."+delivery.AssignerModule+".*")
if err != nil {
return delivery.CompositionVerdict{}, err
}
defer pw.Stop()
go func() {
for e := range pw.Updates() {
ev := check.PresenceEvent{Settled: e == nil}
if e != nil {
ev.Key = e.Key()
if e.Operation() == jetstream.KeyValuePut {
if p, err := delivery.DecodePresence(e.Value()); err == nil {
ev.Online = p.Online
}
}
}
select {
case presence <- ev:
case <-ctx.Done():
return
}
}
}()
}
return check.AwaitComposition(ctx, presence, comps)
}
// followCompositions hands the checker each composition the assigner writes from now on, so a verdict that said its
// composition was not judged takes it in when the assigner starts (event-driven).
func followCompositions(ctx context.Context, kv jetstream.KeyValue, ch *check.Checker) error {
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
return err
}
go func() {
defer w.Stop()
for e := range w.Updates() {
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
if comp, err := delivery.DecodeCompositionVerdict(e.Value()); err == nil {
ch.Composed(ctx, comp)
}
}
}()
return nil
}
// lazyCompositions opens the assigner's compositions when a check first asks for one and the bucket is there: until it
// is, no assigner ever ran, so the check says at once that its composition is not judged (check.ErrAssignerNotRunning),
// and the next check looks again. No timer: each look is a check asking.
type lazyCompositions struct {
open func(ctx context.Context) (check.CompositionReader, error)
mu sync.Mutex
reader check.CompositionReader
}
// Await is the opened reader's Await, opening it first when it is not yet.
func (l *lazyCompositions) Await(ctx context.Context, key string) (delivery.CompositionVerdict, error) {
l.mu.Lock()
r := l.reader
if r == nil {
opened, err := l.open(ctx)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
l.mu.Unlock()
return delivery.CompositionVerdict{}, check.ErrAssignerNotRunning
case err != nil:
l.mu.Unlock()
return delivery.CompositionVerdict{}, err
}
l.reader, r = opened, opened
}
l.mu.Unlock()
return r.Await(ctx, key)
}
@@ -0,0 +1,106 @@
//go:build integration
package main
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
// 1297/AC3 (its bus half): on the real bus, the checker's read of the assigner's composition waits while the key is
// absent and while the assigner says it waits, and answers the verdict the assigner then writes under the check's key.
func TestTheCheckerAwaitsTheAssignersComposition(t *testing.T) {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "comp")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
srv, err := server.NewServer(&server.Options{Host: "127.0.0.1", Port: -1, JetStream: true, StoreDir: filepath.Join(dir, "js"), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
nc, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(nc.Close)
js, _ := jetstream.New(nc)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.AssignerCompositionsBucket, History: delivery.AssignerCompositionsHistory})
if err != nil {
t.Fatal(err)
}
pkv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.PresenceBucket, History: 5})
if err != nil {
t.Fatal(err)
}
pkey, _ := delivery.PresencePartKey(delivery.AssignerModule, "novox")
on, _ := delivery.EncodePresence(delivery.Presence{Online: true})
if _, err := pkv.Put(ctx, pkey, on); err != nil {
t.Fatal(err)
}
// With no assigner online the wait answers at once (a second key, before the assigner of this test is online).
off, _ := delivery.EncodePresence(delivery.Presence{Online: false})
if _, err := pkv.Put(ctx, pkey, off); err != nil {
t.Fatal(err)
}
other, _ := delivery.CompositionKey("feat/y", "0123456789ab")
if _, err := (compositionWatch{kv: kv, presence: pkv}).Await(ctx, other); !errors.Is(err, check.ErrAssignerNotRunning) {
t.Fatalf("with no assigner online the wait answered %v", err)
}
if _, err := pkv.Put(ctx, pkey, on); err != nil {
t.Fatal(err)
}
key, _ := delivery.CompositionKey("feat/x", "0123456789ab")
got := make(chan delivery.CompositionVerdict, 1)
go func() {
v, err := compositionWatch{kv: kv, presence: pkv}.Await(ctx, key)
if err != nil {
t.Error(err)
}
got <- v
}()
put := func(v delivery.CompositionVerdict) {
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := kv.Put(ctx, key, data); err != nil {
t.Fatal(err)
}
}
time.Sleep(200 * time.Millisecond)
put(delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionWaits, Words: "the indexer holds no candidates yet"})
select {
case v := <-got:
t.Fatalf("a waiting composition was answered: %+v", v)
case <-time.After(300 * time.Millisecond):
}
put(delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "x", Machines: []string{"g14"}})
select {
case v := <-got:
if v.State != delivery.CompositionPass || v.Machines[0] != "g14" {
t.Errorf("answered %+v", v)
}
case <-time.After(5 * time.Second):
t.Fatal("the composition written was not answered")
}
}
@@ -0,0 +1,46 @@
package main
import (
"context"
"errors"
"testing"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
)
type answering struct{ v delivery.CompositionVerdict }
func (a answering) Await(context.Context, string) (delivery.CompositionVerdict, error) {
return a.v, nil
}
// 1297 review item 3: the compositions bucket missing when the checker starts is looked up again at the next check: while
// it is missing a check says at once the assigner is not running; once it is there the check reads its composition, and
// it is opened once.
func TestTheCompositionsBucketIsLookedUpAtEachCheck(t *testing.T) {
opens, there := 0, false
l := &lazyCompositions{open: func(context.Context) (check.CompositionReader, error) {
opens++
if !there {
return nil, jetstream.ErrBucketNotFound
}
return answering{delivery.CompositionVerdict{State: delivery.CompositionPass, SDK: "x"}}, nil
}}
if _, err := l.Await(context.Background(), "k"); !errors.Is(err, check.ErrAssignerNotRunning) {
t.Fatalf("without the bucket: %v", err)
}
there = true
for range 2 {
v, err := l.Await(context.Background(), "k")
if err != nil || v.State != delivery.CompositionPass {
t.Fatalf("with the bucket: %+v, %v", v, err)
}
}
if opens != 2 {
t.Errorf("opened %d times, want twice (missing, then once)", opens)
}
}
@@ -20,6 +20,8 @@ const (
getFileTool = "gitea.gitea_get_file"
compareTool = "gitea.gitea_compare"
commitsTool = "gitea.gitea_commits"
// gitea_list_branches answers {branches: [{name, commit}]}, every branch with the commit it points at.
branchesTool = "gitea.gitea_list_branches"
)
// notFoundStatus is the 404 of the forge's error, `Gitea API <path>: <status> <body>`.
@@ -102,3 +104,26 @@ func (f forgeReads) FullSHA(_ context.Context, repository, commit string) (strin
}
return a.Commits[0].SHA, nil
}
// BranchHeads is the commit each branch of the repository points at.
func (f forgeReads) BranchHeads(_ context.Context, repository string) (map[string]string, error) {
owner, repo := ownerRepo(repository)
raw, err := f.ask(branchesTool, map[string]string{"owner": owner, "repo": repo})
if err != nil {
return nil, err
}
var a struct {
Branches *[]struct {
Name string `json:"name"`
Commit string `json:"commit"`
} `json:"branches"`
}
if err := json.Unmarshal(raw, &a); err != nil || a.Branches == nil {
return nil, fmt.Errorf("the answer of %s is not {branches: [{name, commit}]}: %v", branchesTool, err)
}
heads := map[string]string{}
for _, b := range *a.Branches {
heads[b.Name] = b.Commit
}
return heads, nil
}
@@ -1,26 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"github.com/nats-io/nats.go"
"mesh-delivery-checker/internal/check"
)
// heardCaller is the build runner's caller with the calls that reached nobody marked: no responders, or none answering within
// the call's bound. The step never began there, so the check may give it to another node (issue 1820). A call that was
// answered, with a failure or not, is left as it is.
type heardCaller struct {
call func(node, key string, body any) (json.RawMessage, error)
}
func (h heardCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
raw, err := h.call(node, key, body)
if errors.Is(err, nats.ErrNoResponders) || errors.Is(err, nats.ErrTimeout) || errors.Is(err, context.DeadlineExceeded) {
return nil, &check.Unheard{Err: err}
}
return raw, err
}
@@ -1,30 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"mesh-delivery-checker/internal/check"
)
func TestACallThatReachedNobodyIsMarkedAndAnAnsweredFailureIsNot(t *testing.T) { // 1820/AC6
for _, err := range []error{nats.ErrNoResponders, nats.ErrTimeout, context.DeadlineExceeded} {
h := heardCaller{call: func(string, string, any) (json.RawMessage, error) { return nil, fmt.Errorf("call: %w", err) }}
_, got := h.CallOn("g14", "m.t", nil)
var u *check.Unheard
if !errors.As(got, &u) {
t.Errorf("%v: %v is not marked unheard", err, got)
}
}
h := heardCaller{call: func(string, string, any) (json.RawMessage, error) { return nil, errors.New("the module refused") }}
_, got := h.CallOn("g14", "m.t", nil)
var u *check.Unheard
if errors.As(got, &u) {
t.Errorf("an answered failure %v is marked unheard", got)
}
}
@@ -1,101 +0,0 @@
package main
import (
"context"
"errors"
"log"
"sync"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-checker/internal/check"
)
// holdWatch holds the checker back while a delivery builds (ADR 0484 clause 12): whether one does is the list of the keys of
// the controller's bucket of the delivery plans in their build phase, read each time a check is about to start. Its keys age
// out (a time to live, and no delete marker), which a watch never hears, so a watch cannot decide it (mesh-controller
// internal/link/holds.go lists the keys too); the watch is only the wake-up, for a plan that ended by a delete. The bucket
// is read as empty where it is absent (issue 1652 makes it) or cannot be read, saying so once. It may come into being after
// the checker started and the checker holds no timer (ADR 0437 §8), so each time a check is about to start and the bucket is
// not known yet, the checker looks for it.
type holdWatch struct {
ctx context.Context
js bucketOpener
keys func(ctx context.Context) ([]string, error)
changed func(ctx context.Context) // a plan key changed: the checks that waited may start
logger *log.Logger
errs chan error
mu sync.Mutex
watched bool
said bool
}
func newHoldWatch(ctx context.Context, js bucketOpener, changed func(context.Context), logger *log.Logger) *holdWatch {
return &holdWatch{ctx: ctx, js: js, changed: changed, logger: logger, errs: make(chan error, 1)}
}
// Held says whether a delivery builds now.
func (w *holdWatch) Held() bool {
w.ensure()
w.mu.Lock()
keys := w.keys
w.mu.Unlock()
if keys == nil {
return false
}
list, err := keys(w.ctx)
if err != nil {
w.say("the keys of %s cannot be listed, so it is read as empty: %v", check.PlansBuildingBucket, err)
return false
}
return len(list) > 0
}
func (w *holdWatch) say(format string, args ...any) {
w.mu.Lock()
defer w.mu.Unlock()
if !w.said {
w.said = true
w.logger.Printf(format, args...)
}
}
// ensure finds the bucket if it is not known and exists, and starts the wake-up watch.
func (w *holdWatch) ensure() {
w.mu.Lock()
if w.watched || w.js == nil {
w.mu.Unlock()
return
}
w.mu.Unlock()
kv, err := w.js.KeyValue(w.ctx, check.PlansBuildingBucket)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
w.say("the bucket %s does not exist yet: it is read as empty, so no check is held back", check.PlansBuildingBucket)
return
case err != nil:
w.say("the bucket %s cannot be read, so it is read as empty: %v", check.PlansBuildingBucket, err)
return
}
w.mu.Lock()
defer w.mu.Unlock()
if w.watched {
return
}
w.watched, w.said = true, false
w.keys = func(ctx context.Context) ([]string, error) {
list, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return list, err
}
go func() {
wake, err := kv.WatchAll(w.ctx)
if err != nil {
w.errs <- err
return
}
w.errs <- follow(w.ctx, wake, func(jetstream.KeyValueEntry) { w.changed(w.ctx) })
}()
}
@@ -1,35 +0,0 @@
package main
import (
"context"
"errors"
"io"
"log"
"testing"
)
// The controller's plans-building keys age out of the bucket (a time to live, no delete marker), which a watch never hears:
// whether a delivery builds is the list of the keys now, and the watch is only a wake-up (review D3).
func TestADeliveryHoldsWhileItsKeyIsListedAndNotWhenItAgedOutWithNoEvent(t *testing.T) {
listed := []string{"plan-1"}
w := &holdWatch{watched: true, logger: log.New(io.Discard, "", 0), keys: func(context.Context) ([]string, error) { return listed, nil }}
if !w.Held() {
t.Fatal("a plan key is listed and nothing holds")
}
listed = nil // the key aged out: no delete reached the watch
if w.Held() {
t.Error("the key is gone from the bucket and the checker is still held")
}
}
func TestABucketThatCannotBeListedIsReadAsEmpty(t *testing.T) {
w := &holdWatch{watched: true, logger: log.New(io.Discard, "", 0), keys: func(context.Context) ([]string, error) { return []string{"plan-1"}, errors.New("the bus is down") }}
if w.Held() {
t.Error("an unreadable bucket held the checker")
}
var absent holdWatch
absent.logger = log.New(io.Discard, "", 0)
if absent.Held() {
t.Error("a watch with no bucket held the checker")
}
}
@@ -5,10 +5,12 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
@@ -19,6 +21,8 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-checker/internal/check"
@@ -74,11 +78,13 @@ func readCaptured(t *testing.T, name string) map[string]any {
type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
n atomic.Int64
conflict atomic.Bool
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
testTool *testing.T
// beside is, by workspace, the siblings the fetch placed in it as `<owner/repo> <commit>`, sorted: a digest echoes them
// (1929/AC2; mesh-sdk buildstep.besideOf), or the level runs whole and keeps nothing.
beside map[string][]string
}
func (f *fakeBuildModules) count(tool string) int {
@@ -98,10 +104,30 @@ func (f *fakeBuildModules) result(t *testing.T, tool string, args map[string]any
}
fetched := readCaptured(t, name)
fetched["commit"], fetched["base"] = args["commit"], args["base"]
var refs []struct {
Repository string `json:"repository"`
Commit string `json:"commit"`
}
raw, _ := json.Marshal(args["beside"])
_ = json.Unmarshal(raw, &refs)
var lines []string
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
sort.Strings(lines)
f.mu.Lock()
if f.beside == nil {
f.beside = map[string][]string{}
}
f.beside[fmt.Sprint(fetched["workspace"])] = lines
f.mu.Unlock()
v = fetched
case strings.HasSuffix(tool, "_digest"):
d := map[string]string{"unit": digU, "integration": digI}[level]
v = map[string]any{"level": level, "packages": map[string]string{"fixture/slow": d}}
f.mu.Lock()
beside := f.beside[fmt.Sprint(args["workspace"])]
f.mu.Unlock()
v = map[string]any{"level": level, "packages": map[string]string{"fixture/slow": d}, "beside": beside}
case strings.HasSuffix(tool, "_test"):
r := readCaptured(t, "go-test-slow")
r["level"] = level
@@ -111,45 +137,55 @@ func (f *fakeBuildModules) result(t *testing.T, tool string, args map[string]any
return data
}
func (f *fakeBuildModules) serve(t *testing.T, nc *nats.Conn, module string) {
steps, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
// serve makes module's steps bucket and its build seat's stream and workers, and pulls its asks on n1 as the real module
// does since R4g (ADR 0525; issue 1887 A5): each ask's job runs with step bodies answering the captured results, and a
// check takes the kept verdicts of the checker's kept bucket by their digests.
func (f *fakeBuildModules) serve(t *testing.T, module string, kept jetstream.KeyValue) {
kv, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
sub, err := nc.Subscribe("mesh.mod."+module+".call.>", func(m *nats.Msg) {
tokens := strings.Split(m.Subject, ".") // mesh mod <module> call <tool> <node> <caller>
tool, node := tokens[4], tokens[5]
f.mu.Lock()
f.calls[module+"."+tool]++
f.mu.Unlock()
var args map[string]any
_ = json.Unmarshal(m.Data, &args)
id := fmt.Sprintf("s%d", f.n.Add(1))
key, _ := delivery.StepKey(node, id)
put := func(s delivery.Step) {
data, err := delivery.EncodeStep(s)
if err == nil {
_, err = steps.Put(f.ctx, key, data)
language := strings.TrimPrefix(module, "mesh-build-")
if err := buildstep.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
prefix := map[string]string{"mesh-build-go": "build_go_", "mesh-build-node": "build_node_"}[module]
call := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
tool := prefix + name
f.mu.Lock()
f.calls[module+"."+tool]++
f.mu.Unlock()
return f.result(t, tool, args), nil
}
}
steps := buildstep.AskSteps{Fetch: call("fetch"), Test: call("test"),
Kept: func(ctx context.Context, level, digest string) (int, bool, error) {
key, err := delivery.KeptKey(level, digest)
if err != nil {
return 0, false, err
}
e, err := kept.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return 0, false, nil
}
if err != nil {
t.Errorf("writing step %s: %v", key, err)
return 0, false, err
}
}
step := delivery.Step{ID: id, Tool: tool, Node: node, State: delivery.StepRunning, Started: time.Now().UTC()}
put(step)
answer, _ := json.Marshal(map[string]any{"result": map[string]any{"step": id, "key": key, "state": "running"}, "error": "", "node": node})
_ = m.Respond(answer)
go func() {
time.Sleep(20 * time.Millisecond)
step.Ended = time.Now().UTC()
step.State, step.Result = delivery.StepPassed, f.result(t, tool, args)
put(step)
}()
})
if err != nil {
t.Fatal(err)
v, err := delivery.DecodeKept(e.Value())
return v.Tests, err == nil, err
}}
if module == "mesh-build-go" {
steps.Digest = call("digest")
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
p := &buildstep.Puller{Source: &buildstep.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: "n1", Put: put,
Capacity: func() int { return 2 }, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return buildstep.RunAsk(ctx, steps, a) }}
go func() { _ = p.Pull(f.ctx) }()
}
// serveForge answers the forge's two reads as its tools do, with the answers the forge gave (internal/check/testdata/captured):
@@ -276,14 +312,23 @@ func startBed(t *testing.T) *bed {
}
b.buckets[name] = kv
}
// 1929/AC1: the indexer's value of the repository's module, with no TypeScript manifest, so the Go build module alone
// judges the repository (an unknown repository is judged by both, issue 1887 A12).
known, err := delivery.EncodeModule(delivery.Module{Repository: repo, Commit: trunk, Path: "modules/m", None2: "the fingerprint is computed with its chain"})
if err != nil {
t.Fatal(err)
}
if _, err := b.buckets[delivery.ModulesBucket].Put(ctx, "m", known); err != nil {
t.Fatal(err)
}
b.setOnline(t, "n1", true)
for _, module := range []string{"mesh-build-go", "mesh-build-node"} {
b.setHealth(t, "n1", module)
}
b.serveForge(t, ctl)
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.serve(t, ctl, "mesh-build-go")
b.fake.serve(t, ctl, "mesh-build-node")
b.fake.serve(t, "mesh-build-go", b.buckets[delivery.CheckerKeptBucket])
b.fake.serve(t, "mesh-build-node", b.buckets[delivery.CheckerKeptBucket])
b.report(t, "mesh-build-go", "n1", "go", "go1.27.1", "1")
b.report(t, "mesh-build-node", "n1", "typescript", "tsc5", "1")
b.setTrunk(t, map[string]string{repo: trunk})
@@ -424,7 +469,7 @@ func (b *bed) owed(t *testing.T, number int, head, state string) delivery.Owed {
// TestCheckerJudgesACandidateAndOwesItsStatus: a candidate the indexer wrote, one pull request of mesh-delivery, is fetched
// merged onto the trunk the gate names, digested and tested at both levels on the node, kept, previewed; the verdict passes
// with the counts of both levels and the composition not run, and the status mesh/check is owed on the head as success.
func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300/AC5, 1445/AC1
func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300/AC5, 1445/AC1, 1929/AC1
b := startBed(t)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
@@ -464,7 +509,7 @@ func TestCheckerJudgesACandidateAndOwesItsStatus(t *testing.T) { // N6/AC5, 1300
// TestACheckOfEqualInputsRunsNothing: a second candidate (another head, the same digests) is checked with every package
// taken from the first's kept verdicts: no test step runs and no second kept key appears.
func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6
func TestACheckOfEqualInputsRunsNothing(t *testing.T) { // 1287/AC5, 1288/AC6, 1929/AC2
b := startBed(t)
first := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
@@ -519,14 +564,25 @@ func TestAHeadThatDoesNotMergeFailsNamingTheFilesAndRunsNoTest(t *testing.T) { /
// TestAnHqPullRequestGetsNoVerdictAndAnotherDoes: hq's pull request is judged by no one (ADR 0484 clause 8), the other one in
// the same run is.
func TestAnHqPullRequestGetsNoVerdictAndAnotherDoes(t *testing.T) { // ADR 0484 clause 8
// 1887/A10, ADR 0527 §2: a pull request into hq gets a verdict and a mesh/check from the checker, as any repository's.
func TestAnHqPullRequestGetsAVerdictAndAStatus(t *testing.T) {
b := startBed(t)
b.setTrunk(t, map[string]string{repo: trunk, "novox/hq": trunk})
hq := b.candidate(t, "feat/hq", pull("novox/hq", 3, head1))
other := b.candidate(t, "feat/x", pull(repo, 7, head2))
b.start(t, nil)
b.verdict(t, other, ended)
if _, err := b.buckets[delivery.CheckerVerdictsBucket].Get(b.ctx, hq); err == nil {
t.Error("hq's pull request has a verdict")
if v := b.verdict(t, hq, ended); v.State != delivery.VerdictPass {
t.Fatalf("hq's verdict = %+v", v)
}
key, _ := delivery.CheckerOwedKey("novox/hq", 3, head1)
for deadline := time.Now().Add(20 * time.Second); ; time.Sleep(50 * time.Millisecond) {
if e, err := b.buckets[delivery.CheckerOwedBucket].Get(b.ctx, key); err == nil {
if o, err := delivery.DecodeCheckerOwed(e.Value()); err == nil && o.State == "success" {
return
}
}
if time.Now().After(deadline) {
t.Fatalf("hq#3 owes no success within 20 s")
}
}
}
@@ -549,37 +605,9 @@ func TestACheckLeftRunningIsTakenOverWhenTheCheckerStarts(t *testing.T) { // N6/
}
}
// TestNoCheckStartsWhileADeliveryBuildsAndOneStartsWhenItEnds: the controller's bucket of the plans in their build phase
// holds a plan's key; the candidate waits, no step is called; the key deleted, the check starts and passes (ADR 0484 clause 12).
func TestNoCheckStartsWhileADeliveryBuildsAndOneStartsWhenItEnds(t *testing.T) { // N8/AC1
b := startBed(t)
plans, err := b.js.CreateKeyValue(b.ctx, jetstream.KeyValueConfig{Bucket: "mesh-controller_plans-building", History: 1})
if err != nil {
t.Fatal(err)
}
if _, err := plans.Put(b.ctx, "plan-1", []byte(`{"plan":"plan-1","since":"2026-10-21T19:00:00Z","seen":1}`)); err != nil {
t.Fatal(err)
}
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
b.start(t, nil)
time.Sleep(1500 * time.Millisecond) // a wait for nothing to happen: there is no event to wait on
if _, err := b.buckets[delivery.CheckerVerdictsBucket].Get(b.ctx, key); err == nil {
t.Fatal("a check started while a delivery builds")
}
if n := b.fake.count("mesh-build-go.build_go_fetch"); n != 0 {
t.Fatalf("%d fetches while a delivery builds", n)
}
if err := plans.Delete(b.ctx, "plan-1"); err != nil {
t.Fatal(err)
}
if v := b.verdict(t, key, ended); v.State != delivery.VerdictPass {
t.Errorf("verdict = %+v", v)
}
}
// TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit: the go.mod the forge answers names a commit that main does not
// hold; the verdict fails naming the member and the commit and no test runs (ADR 0484 clause 9).
func TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit(t *testing.T) { // 1624/AC2
func TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit(t *testing.T) { // 1624/AC2, 1929/AC3
b := startBed(t)
b.stalePin.Store(true)
key := b.candidate(t, "feat/x", pull(repo, 7, head1))
@@ -588,9 +616,7 @@ func TestAMemberNamingAMeshSDKCommitOffTheTrunkFailsNamingTheCommit(t *testing.T
if v.State != delivery.VerdictFail || !strings.Contains(v.Failed.Words, "9aaa5c80ac5c") || !strings.Contains(v.Failed.Words, repo+"#7") {
t.Fatalf("verdict = %+v", v)
}
if n := b.fake.count("mesh-build-go.build_go_test"); n != 0 {
t.Errorf("%d test steps ran after a stale pin", n)
}
// 1929/AC3: no assertion that no test ran: one ask runs fetch, digest and tests (ADR 0525 §3).
}
// triesStore is the tries bucket as the tools write it.
@@ -651,3 +677,32 @@ func TestATryIsJudgedAndItsEndWrittenWithNoStatusOwed(t *testing.T) {
}
}
}
// TestTheCheckerSaysOwedReadableOnTheBus: started, the checker says on its own states subject what it keeps, owed among
// the readable ones, which is what grants the forge module its read of owed (ADR 0495 §2-§3, ADR 0521; R4e finding R2-1).
func TestTheCheckerSaysOwedReadableOnTheBus(t *testing.T) { // R2-1
b := startBed(t)
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
b.start(t, nil)
events, err := b.js.Stream(b.ctx, "EVENTS")
if err != nil {
t.Fatal(err)
}
var said announce.States
for deadline := time.Now().Add(10 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
m, err := events.GetLastMsgForSubject(b.ctx, announce.Subject("mesh-delivery-checker"))
if err != nil {
continue
}
if err := json.Unmarshal(m.Data, &said); err != nil {
t.Fatal(err)
}
break
}
if said.Module != "mesh-delivery-checker" || said.Node != "n1" || !said.Keeps("owed") || !said.Readable("owed") ||
!said.Readable("previews") || !said.Readable("verdicts") || !said.Readable("kept") || said.Readable("tries") || len(said.States) != 5 {
t.Fatalf("the checker said %+v, want its five states with kept, owed, previews and verdicts readable (1929/AC4; 1887 A5b)", said)
}
}
@@ -6,7 +6,6 @@ import (
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
@@ -15,11 +14,8 @@ import (
// judgeDeps are the parts the judge of a check is made of.
type judgeDeps struct {
runner buildrun.Runner
asks check.Asks
tools *buildrun.ToolchainSet
presence *buildrun.PresenceSet
have *buildrun.HaveSet
places *buildrun.Places
trunks *check.Trunks
kept check.KV
previews check.KV
@@ -27,11 +23,13 @@ type judgeDeps struct {
forge check.Forge
announce check.Announcer
builds check.BuildStates
logf func(format string, args ...any)
// compositions is the assigner's bucket of compositions; nil leaves every composition not run.
compositions check.CompositionReader
logf func(format string, args ...any)
}
func newJudge(d judgeDeps) check.Judge {
return &check.Examiner{Steps: d.runner, Toolchains: d.tools, Presence: d.presence, Health: d.have, Places: d.places,
return &check.Examiner{Compositions: d.compositions, Asks: d.asks, Toolchains: d.tools,
Kept: check.Kept{Store: d.kept}, Previews: d.previews, Modules: d.repos, Packages: d.repos, Forge: d.forge, Runs: d.repos, Builds: d.builds, Announce: d.announce, Trunks: d.trunks, Logf: d.logf}
}
@@ -94,7 +92,7 @@ func (r *repoModules) Put(ctx context.Context, key string, value []byte, deleted
if mk, err := delivery.ManifestsKey(m.Manifest); err == nil {
if raw, _, found, err := r.manifests.Get(ctx, mk); err == nil && found {
if kept, err := delivery.DecodeManifest(raw); err == nil {
if man, err := catalogue.ParseManifest([]byte(kept.JSON)); err == nil && man.Build != nil {
if man, err := parseManifest(kept.JSON); err == nil && man.Build != nil {
for _, a := range man.Build.Artifacts {
if a.Language != "" {
set[a.Language] = true
@@ -142,17 +140,28 @@ func (r *repoModules) PackagePaths(repository string) []string {
return out
}
// BuildModules are the build modules that judge a repository, sorted.
// BuildModules are the build modules that judge a repository, sorted: the Go build module always, and the Node build
// module when a module of the repository builds TypeScript. A repository of which the mesh knows no module (mesh-lab, hq)
// is judged by both, each finding its own directories in the tree (go.mod files; package.json files), so a repository
// with tests in either language and no manifest is judged whole (issue 1887 A12).
func (r *repoModules) BuildModules(repository string) []string {
out := []string{buildrun.GoBuildModule}
r.mu.Lock()
defer r.mu.Unlock()
known := false
for module, repo := range r.repoOf {
if repo == repository && r.langs[r.typed[module]]["typescript"] {
if repo != repository {
continue
}
known = true
if r.langs[r.typed[module]]["typescript"] {
out = append(out, buildrun.NodeBuildModule)
break
}
}
if !known {
out = append(out, buildrun.NodeBuildModule)
}
sort.Strings(out)
return out
}
@@ -173,7 +182,8 @@ func (r buildReader) Build(ctx context.Context, module, fp2 string) (delivery.Bu
return b, err == nil, err
}
// CommitOf is the commit the indexer's record of the module was computed at: the commit of its repository the mesh runs.
// CommitOf is the commit the indexer's record of the module names: the commit its fingerprint first appeared at, so the source
// of the build the mesh runs (a later merge that leaves the module alone writes no record, the operator 2026-10-25).
func (r *repoModules) CommitOf(module string) string {
r.mu.Lock()
defer r.mu.Unlock()
@@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
@@ -69,8 +70,10 @@ func TestARepositoryWithATypeScriptModuleIsJudgedByTheNodeBuildModuleToo(t *test
if got := r.CommitOf("mesh-delivery-checker"); got != strings.Repeat("a", 40) || r.CommitOf("gone") != "" {
t.Errorf("the commit the mesh runs = %q", got)
}
if got := r.BuildModules("novox/mesh-sdk"); strings.Join(got, ",") != buildrun.GoBuildModule {
t.Errorf("a repository with no module: %v, want the Go build module", got)
// 1887/A12: a repository with no module the mesh knows (mesh-lab: nested go.mod files and a root package.json) is
// judged by both build modules.
if got := r.BuildModules("novox/mesh-lab"); strings.Join(got, ",") != buildrun.GoBuildModule+","+buildrun.NodeBuildModule {
t.Errorf("a repository with no module: %v, want both build modules", got)
}
}
@@ -110,3 +113,31 @@ func TestTheTypeScriptPackagesPathsComeFromTheIndexersModulesNotTheManifest(t *t
t.Errorf("another repository: %v", got)
}
}
// A manifest whose config value falls back to a predefined key (ADR 0348 §7) is read, not logged as unreadable for the
// catalogue's predefined.json this process never holds: the indexer and the assigner read it with
// catalogue.ParseManifestForInputs (issues 1751, 1906). xterm's module.json from mesh-catalog at 835bf22b.
func TestAManifestFallingBackToAPredefinedKeyIsRead(t *testing.T) {
raw, err := os.ReadFile("testdata/catalogue-835bf22b-xterm.module.json")
if err != nil {
t.Fatal(err)
}
manifest := string(raw)
data, err := delivery.EncodeManifest(delivery.Manifest{Module: "xterm", JSON: manifest})
if err != nil {
t.Fatal(err)
}
store := memManifests{delivery.ManifestDigest([]byte(manifest)): data}
var said []string
r := &repoModules{manifests: store, logf: func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) }}
putModule(t, r, "xterm", "novox/mesh-catalog", manifest)
r.BuildModules("novox/mesh-catalog")
for _, s := range said {
if strings.Contains(s, "cannot be read") {
t.Errorf("the manifest was not read: %s", s)
}
}
if _, err := parseManifest(manifest); err != nil {
t.Errorf("manifestReader: %v", err)
}
}
@@ -0,0 +1,56 @@
package main
import (
"encoding/json"
"os"
"sort"
"testing"
)
// The checker says on the bus which states it keeps and which others may read (novox/hq ADR 0495 clauses 2 and 3): a
// reader is granted from that announcement. owed is read by the forge module, which writes mesh/check from it (ADR 0521,
// R4e round 2 finding R2-1), and by the forge bridge; previews by the indexer; verdicts by the gate. The list names every
// state of the manifest, so no state is refused at the call for want of being announced (clause 5).
func TestTheCheckerAnnouncesEveryStateItKeepsAndOwedReadable(t *testing.T) { // R2-1, ADR 0495 §2-§3, ADR 0521
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
var m struct {
State []struct{ Name string } `json:"state"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
var manifest, announced, readable []string
for _, s := range m.State {
manifest = append(manifest, s.Name)
}
for _, s := range keptStates {
announced = append(announced, s.Name)
if s.Readable {
readable = append(readable, s.Name)
}
}
sort.Strings(manifest)
sort.Strings(announced)
if !equal(manifest, announced) {
t.Errorf("announced %v, the manifest keeps %v", announced, manifest)
}
// kept is read by the Go build module for a check ask (issue 1887 design limit 1).
if want := []string{"kept", "owed", "previews", "verdicts"}; !equal(readable, want) {
t.Errorf("readable %v, want %v", readable, want)
}
}
func equal(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
@@ -25,6 +25,7 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
@@ -39,9 +40,6 @@ func eventSubject(event string) string {
}
const (
// defaultStepBound is how long the checker waits for one step of a build module (a fetch downloads the Go modules, a
// test level runs a repository's tests).
defaultStepBound = 30 * time.Minute
// defaultLink is the base of the address a status points at; the verdict key follows.
defaultLink = "mesh-delivery-view/checks"
)
@@ -113,6 +111,18 @@ func durationEnv(getenv func(string) string, name string, def time.Duration) (ti
}
// run is the program: it returns when ctx ends or the bus cannot be used.
// keptStates is every state the checker keeps and which others may read (ADR 0495 §2-§3): owed by the forge module,
// which writes mesh/check from it (ADR 0521), and by the forge bridge; previews by the indexer; verdicts by the gate.
var keptStates = []announce.State{
// The kept verdicts are read by the Go build module, which takes them for a check ask's unchanged packages (issue 1887's
// design limit 1); the checker alone writes them (ADR 0444 clause 5).
{Name: "kept", Readable: true},
{Name: "owed", Readable: true},
{Name: "previews", Readable: true},
{Name: "tries"},
{Name: "verdicts", Readable: true},
}
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-checker: ", log.LstdFlags)
path := strings.TrimSpace(getenv("MESH_BROKER_FILE"))
@@ -142,17 +152,12 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
bucket := func(name string) (jetstream.KeyValue, error) {
kv, err := js.KeyValue(ctx, name)
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", name, err)
}
return kv, nil
}
// Its own states, the candidates it checks, and the modules and manifests that say which build modules judge a repository:
// a check without one of them would be wrong or unwritten, so each stops the start, naming its grant (issue 1945).
buckets := map[string]jetstream.KeyValue{}
for _, name := range []string{delivery.CheckerVerdictsBucket, delivery.CheckerKeptBucket, delivery.CheckerOwedBucket, delivery.PreviewsBucket,
delivery.CandidatesBucket, delivery.ModulesBucket, delivery.ManifestsBucket, check.TriesBucket} {
if buckets[name], err = bucket(name); err != nil {
if buckets[name], err = process.EssentialBucket(ctx, js, "checker", name, "judge or keep a check"); err != nil {
return err
}
}
@@ -167,8 +172,13 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
defer caller.Close()
// What the checker keeps, said on the bus (ADR 0495 §2-§3): a reader is granted from it. Not fatal when it cannot be said
// now: process.Conn says it again at every reconnect.
if err := caller.Keeps(keptStates...); err != nil {
logger.Printf("what the checker keeps could not be said on the bus, so readers of owed, previews and verdicts are not granted until it is: %v", err)
}
stepBound, err := durationEnv(getenv, "MESH_CHECKER_STEP_BOUND", defaultStepBound)
stepBound, err := durationEnv(getenv, "MESH_CHECKER_STEP_BOUND", check.DefaultStepBound)
if err != nil {
return err
}
@@ -176,20 +186,16 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err != nil {
return err
}
capacity, err := capacityFrom(strings.TrimSpace(getenv("MESH_CHECKER_SETTINGS")), getenv("MESH_CHECKER_CAPACITY"))
if err != nil {
return fmt.Errorf("the capacity setting: %w", err)
}
link := strings.TrimSpace(getenv("MESH_CHECKER_LINK"))
if link == "" {
link = defaultLink
}
tools := &buildrun.ToolchainSet{}
presence := &buildrun.PresenceSet{}
have := &buildrun.HaveSet{}
trunks := &check.Trunks{}
places := &buildrun.Places{Capacity: capacity}
runner := buildrun.Runner{Caller: heardCaller{caller.CallOn}, Steps: stepsWaiter{js}, Bound: stepBound}
// A check is one ask per pull request and build module on the build seats' check stream, pulled by a build node with a
// free slot (novox/hq ADR 0525 §1, §3): the checker chooses no node and starts no step.
asks := buildrun.JetStreamAsks{JS: js}
ch := &check.Checker{
Verdicts: kvStore{buckets[delivery.CheckerVerdictsBucket]},
Owed: kvStore{buckets[delivery.CheckerOwedBucket]},
@@ -201,19 +207,28 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The builder's builds say how the candidates' fingerprints stand (issue 1684); where its bucket is not there yet the line
// says nothing of the builds, until the checker starts again.
var builds check.BuildStates
switch kv, err := js.KeyValue(ctx, delivery.BuilderBuildsBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: the line says nothing of the builds until the checker starts again", delivery.BuilderBuildsBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.BuilderBuildsBucket, err)
default:
if kv := process.OptionalBucket(ctx, js, delivery.BuilderBuildsBucket, "the line says nothing of the builds until the checker starts again", logger.Printf); kv != nil {
builds = buildReader{kvStore{kv}}
}
// The assigner's compositions (ADR 0528 clause 5, issue 1297): their bucket is looked up at each check until it is
// there, so an assigner delivered after the checker started is read at the next check, and followed from then on.
compositions := &lazyCompositions{open: func(at context.Context) (check.CompositionReader, error) {
kv, err := js.KeyValue(at, delivery.AssignerCompositionsBucket)
if err != nil {
return nil, err
}
cw := compositionWatch{kv: kv}
if pkv, err := js.KeyValue(at, delivery.PresenceBucket); err == nil {
cw.presence = pkv
}
if err := followCompositions(ctx, kv, ch); err != nil {
return nil, fmt.Errorf("the bucket %s cannot be watched: %w", delivery.AssignerCompositionsBucket, err)
}
logger.Printf("the assigner's compositions are read from %s", delivery.AssignerCompositionsBucket)
return cw, nil
}}
// The tries (ADR 0501 §5) are judged by the same judge, held by the same deliveries.
tryer := &check.Tryer{Tries: kvStore{buckets[check.TriesBucket]}, Bound: checkBound, Logf: logger.Printf}
holds := newHoldWatch(ctx, js, func(ctx context.Context) { ch.Reconsider(ctx); tryer.Reconsider(ctx) }, logger)
ch.Hold = holds.Held
tryer.Hold = holds.Held
announce := func(ctx context.Context, event, key string) {
body, _ := json.Marshal(map[string]string{"key": key})
if _, err := js.Publish(ctx, eventSubject(event), body); err != nil {
@@ -222,16 +237,24 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
repos := &repoModules{manifests: kvStore{buckets[delivery.ManifestsBucket]}, logf: logger.Printf}
ch.Judge = newJudge(judgeDeps{
runner: runner, tools: tools, presence: presence, have: have, places: places, trunks: trunks,
asks: buildrun.Asker{Publisher: asks, Waiter: asks, Bound: stepBound}, tools: tools, trunks: trunks,
kept: kvStore{buckets[delivery.CheckerKeptBucket]}, previews: kvStore{buckets[delivery.PreviewsBucket]},
repos: repos, builds: builds, forge: forgeReads{call: caller.Call}, logf: logger.Printf,
announce: announce,
announce: announce, compositions: compositions,
})
ch.Announce = announce
tryer.Judge = ch.Judge
// The toolchains first, so a check that starts from the first candidate finds the nodes.
watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, ch.NodesChanged, logger)
if err != nil {
return err
}
if err := watch.start(); err != nil {
return err
}
@@ -253,69 +276,42 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
defer cc.Stop()
}
errs := make(chan error, 8)
// The watchman's presence says which nodes are asleep. Where its bucket is not there yet no node is online, so nothing
// is checked; the checker says so and is started again with the watchman.
switch kv, err := js.KeyValue(ctx, delivery.PresenceBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no node counts as online until the checker starts again", delivery.PresenceBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.PresenceBucket, err)
default:
pw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { presence.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
// The watchman's presence: a node coming back online asks again a check whose forge was not heard (UnheardWords), and
// the assigner's own presence says whether a composition is being made (compositionWatch, ADR 0537 §4). A check is no
// longer held for a node: its asks wait in the build seats' streams, which redeliver a lost node's ask (ADR 0525 §6,
// issue 1912), so the node-engine's have is not watched here.
if kv := process.OptionalBucket(ctx, js, delivery.PresenceBucket, "no node counts as online until the checker starts again", logger.Printf); kv != nil {
if pw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { presence.Put(e.Key(), e.Value(), !opOf(e)) }); err != nil {
process.NotWatched(logger.Printf, delivery.PresenceBucket, err, "no node counts as online until the checker starts again")
} else {
go func() {
errs <- follow(ctx, pw, func(e jetstream.KeyValueEntry) {
if presence.Put(e.Key(), e.Value(), !opOf(e)) {
ch.NodesChanged(ctx) // a node came online or went: a check whose forge was not heard may now be heard
}
})
}()
}
go func() {
errs <- follow(ctx, pw, func(e jetstream.KeyValueEntry) {
if presence.Put(e.Key(), e.Value(), !opOf(e)) {
ch.NodesChanged(ctx) // a node came online or went: a check that waited for one may go on
}
})
}()
}
// The node-engine's have says whether a node's build module runs and is healthy: a node whose build module is not is not
// chosen. Where its bucket is not there yet no build module counts as healthy, so nothing is checked.
switch kv, err := js.KeyValue(ctx, delivery.HaveBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no build module counts as healthy until the checker starts again", delivery.HaveBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.HaveBucket, err)
default:
hw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) { have.Put(e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
}
go func() {
errs <- follow(ctx, hw, func(e jetstream.KeyValueEntry) {
if have.Put(e.Key(), e.Value(), !opOf(e)) {
ch.NodesChanged(ctx)
}
})
}()
}
// The gate's source says the trunk head of each repository, which a pull request is merged onto. Where it is not there
// yet no trunk is known, so nothing is checked.
switch kv, err := js.KeyValue(ctx, delivery.SourceBucket); {
case errors.Is(err, jetstream.ErrBucketNotFound):
logger.Printf("the bucket %s does not exist yet: no trunk is known until the checker starts again", delivery.SourceBucket)
case err != nil:
return fmt.Errorf("the bucket %s: %w", delivery.SourceBucket, err)
default:
if kv := process.OptionalBucket(ctx, js, delivery.SourceBucket, "no trunk is known, so nothing is checked until the checker starts again", logger.Printf); kv != nil {
sw, err := settle(ctx, kv, func(e jetstream.KeyValueEntry) {
if e.Key() == delivery.SourceMainKey {
trunks.Put(e.Value(), !opOf(e))
}
})
if err != nil {
return err
process.NotWatched(logger.Printf, delivery.SourceBucket, err, "no trunk is known, so nothing is checked until the checker starts again")
} else {
go func() {
errs <- follow(ctx, sw, func(e jetstream.KeyValueEntry) {
if e.Key() == delivery.SourceMainKey && trunks.Put(e.Value(), !opOf(e)) {
ch.TrunkMoved(ctx) // the trunk moved: its statuses go pending, and a verdict for the old trunk is judged again
}
})
}()
}
go func() {
errs <- follow(ctx, sw, func(e jetstream.KeyValueEntry) {
if e.Key() == delivery.SourceMainKey && trunks.Put(e.Value(), !opOf(e)) {
ch.TrunkMoved(ctx) // the trunk moved: its statuses go pending, and a verdict for the old trunk is judged again
}
})
}()
}
// The modules say which repository holds a TypeScript module, so the Node build module judges it too.
mw, err := settle(ctx, buckets[delivery.ModulesBucket], func(e jetstream.KeyValueEntry) { repos.Put(ctx, e.Key(), e.Value(), !opOf(e)) })
@@ -362,9 +358,9 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
case err := <-errs:
return err
case err := <-watch.errs:
case err := <-assignErrs:
return err
case err := <-holds.errs:
case err := <-watch.errs:
return err
}
}
@@ -0,0 +1,21 @@
package main
import (
"os"
"strings"
"testing"
)
// 1912: with checks as asks on the build seats' streams, the stream redelivers a lost node's ask (ADR 0525 §6), so the
// checker program opens no node-engine have to start again a check that failed for want of a node.
func TestTheCheckerWatchesNoHave(t *testing.T) {
raw, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
for _, s := range []string{"delivery.HaveBucket", "buildrun.HaveSet"} {
if strings.Contains(string(raw), s) {
t.Errorf("the checker program still reads %s", s)
}
}
}
@@ -1,29 +0,0 @@
package main
import (
"encoding/json"
"fmt"
"os"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// capacityFrom is the build capacity of each node (ADR 0478 clause 3): the module's setting `capacity`, `<node>=<steps at
// once>` pairs separated by commas, laid into the settings file the controller composes (issue 1661); empty is one step at a
// time on every node. With no settings file named, fallback is read instead, for a run outside the module's launch.
func capacityFrom(path, fallback string) (map[string]int, error) {
if path == "" {
return buildrun.ParseCapacity(fallback)
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("the settings file: %w", err)
}
var s struct {
Capacity string `json:"capacity"`
}
if err := json.Unmarshal(raw, &s); err != nil {
return nil, fmt.Errorf("the settings file %s is not JSON: %w", path, err)
}
return buildrun.ParseCapacity(s.Capacity)
}
@@ -1,68 +0,0 @@
package main
import (
"os"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func settingsFile(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "settings.json")
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
return p
}
// ADR 0484 clause 12: the build capacity is the module's setting `capacity`, laid into the settings file: empty is one step at a time everywhere.
func TestCapacityIsReadFromTheSettingsFile(t *testing.T) {
for body, want := range map[string]map[string]int{
`{"capacity": "shanks=3,ace=2"}`: {"shanks": 3, "ace": 2},
`{"capacity": ""}`: {},
`{}`: {},
} {
got, err := capacityFrom(settingsFile(t, body), "")
if err != nil || len(got) != len(want) || got["shanks"] != want["shanks"] || got["ace"] != want["ace"] {
t.Errorf("%s: capacity %v, %v, want %v", body, got, err, want)
}
}
p := &buildrun.Places{Capacity: mustCapacity(t, settingsFile(t, `{"capacity": ""}`))}
if p.CapacityOf("shanks") != 1 || p.CapacityOf("ace") != 1 {
t.Errorf("an empty setting gives %d and %d, want 1 everywhere", p.CapacityOf("shanks"), p.CapacityOf("ace"))
}
}
func mustCapacity(t *testing.T, path string) map[string]int {
t.Helper()
c, err := capacityFrom(path, "")
if err != nil {
t.Fatal(err)
}
return c
}
func TestACapacitySettingThatIsNotOneIsRefusedNamingIt(t *testing.T) {
if _, err := capacityFrom(settingsFile(t, `{"capacity": "shanks=-1"}`), ""); err == nil {
t.Error("shanks=-1 accepted")
}
if c, err := capacityFrom(settingsFile(t, `{"capacity": "shanks=3,ace=2,g14=0,novox=0"}`), ""); err != nil || c["g14"] != 0 || len(c) != 4 {
t.Errorf("the starting value: %v, %v", c, err)
}
if _, err := capacityFrom(settingsFile(t, `not json`), ""); err == nil {
t.Error("a file that is not JSON accepted")
}
if _, err := capacityFrom(filepath.Join(t.TempDir(), "none"), ""); err == nil {
t.Error("a settings file that is not there accepted")
}
}
// With no settings file named (a run outside the module's launch, as the tests do), the capacity is the environment's.
func TestWithNoSettingsFileTheCapacityIsTheEnvironments(t *testing.T) {
got, err := capacityFrom("", "ace=2")
if err != nil || got["ace"] != 2 {
t.Errorf("capacity %v, %v", got, err)
}
}
@@ -0,0 +1,211 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -0,0 +1,218 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -8,6 +8,8 @@ import (
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -24,9 +26,11 @@ var toolchainStreams = map[string]string{buildrun.GoBuildModule: "TOOLCHAINS_GO"
// §8): it watches the buckets that exist at start, and a build module says each time it wrote its toolchains key
// (toolchain.reported, handled by Reported); a bucket not watched yet is then watched.
type toolchainWatch struct {
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
// put takes a change of a toolchains bucket in: set.Put, or the gate of the assigned build nodes before it (issue 1940).
put func(module, key string, value []byte, deleted bool) bool
changed func(ctx context.Context) // a language's entries changed: the failed builds are tried again
logger *log.Logger
errs chan error
@@ -35,7 +39,7 @@ type toolchainWatch struct {
}
func newToolchainWatch(ctx context.Context, js bucketOpener, set *buildrun.ToolchainSet, changed func(context.Context), logger *log.Logger) *toolchainWatch {
return &toolchainWatch{ctx: ctx, js: js, set: set, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
return &toolchainWatch{ctx: ctx, js: js, set: set, put: set.Put, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
}
// start watches the buckets that exist now; a missing one is said and waits for its build module's event.
@@ -61,16 +65,19 @@ func (w *toolchainWatch) ensure(module string) error {
w.logger.Printf("the bucket %s does not exist yet: it is watched when %s reports a toolchain", name, module)
return nil
case err != nil:
return fmt.Errorf("the bucket %s: %w", name, err)
// Refused or not answered (its grant not on the bus yet, issue 1945): said, and opened again at the module's next report.
w.logger.Printf("the bucket %s cannot be read (%v): %s's toolchains are not heard until its next report", name, err, module)
return nil
}
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.set.Put(module, e.Key(), e.Value(), !opOf(e)) })
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.put(module, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
return err
process.NotWatched(w.logger.Printf, name, err, fmt.Sprintf("%s's toolchains are not heard until its next report", module))
return nil
}
w.watched[module] = true
go func() {
w.errs <- follow(w.ctx, watcher, func(e jetstream.KeyValueEntry) {
if w.set.Put(module, e.Key(), e.Value(), !opOf(e)) {
if w.put(module, e.Key(), e.Value(), !opOf(e)) {
w.changed(w.ctx)
}
})
@@ -1,3 +0,0 @@
{
"capacity": ""
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+4 -2
View File
@@ -1,5 +1,7 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876 h1:dtiIJzrX7ey1wmsdzapiFgCKDllCph7/vxfUp1ib/t8=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022183749-875c680d9876/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585 h1:7QoFH5qjXRpx5M4uPdYsVJ5qKl2082gq8n0lFxhoNIg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025211220-e0d1de047585/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1,141 @@
package check
import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ErrAssignerNotRunning is what a wait for a composition answers when no instance of the assigner is online: nobody is
// making it, so the check does not wait for it (issue 1297). The verdict says the composition is not judged, and when the
// assigner starts and writes it, the checker takes it in then (Checker.Composed).
var ErrAssignerNotRunning = errors.New(delivery.AssignerNotRunning)
// PresenceEvent is a change of the watchman's presence bucket, or Settled once what stands was handed.
type PresenceEvent struct {
Key string
Online bool
Settled bool
}
// CompositionEvent is a change of the composition's key (its value, nil for a delete), or Settled once what stands was
// handed.
type CompositionEvent struct {
Value []byte
Settled bool
}
// AwaitComposition answers the assigner's composition once one is written that does not wait, while an instance of the
// assigner is online by the watchman's presence (`part.mesh-delivery-assigner.<instance>`). Once both watches handed
// what stands, no instance online is ErrAssignerNotRunning at once, and so is the last instance going offline during the
// wait. No timer: every answer follows an event; ctx ending is ctx's error.
func AwaitComposition(ctx context.Context, presence <-chan PresenceEvent, comps <-chan CompositionEvent) (delivery.CompositionVerdict, error) {
online := map[string]bool{}
presenceSettled, compSettled := false, false
for {
select {
case <-ctx.Done():
return delivery.CompositionVerdict{}, ctx.Err()
case e, ok := <-presence:
if !ok {
return delivery.CompositionVerdict{}, errors.New("the watch of the assigner's presence ended")
}
if e.Settled {
presenceSettled = true
break
}
if _, part, instance, err := delivery.ParsePresenceKey(e.Key); err == nil && part == delivery.AssignerModule {
if e.Online {
online[instance] = true
} else {
delete(online, instance)
}
}
case e, ok := <-comps:
if !ok {
return delivery.CompositionVerdict{}, errors.New("the watch of the composition ended")
}
if e.Settled {
compSettled = true
break
}
if e.Value == nil {
break
}
v, err := delivery.DecodeCompositionVerdict(e.Value)
if err != nil {
return delivery.CompositionVerdict{}, fmt.Errorf("the composition cannot be read: %w", err)
}
if v.State != delivery.CompositionWaits {
return v, nil
}
}
if presenceSettled && compSettled && len(online) == 0 {
return delivery.CompositionVerdict{}, ErrAssignerNotRunning
}
}
}
// withComposition is v with the assigner's composition taken in: a failed one fails the check, naming its machine.
func withComposition(v *delivery.Verdict, comp delivery.CompositionVerdict) {
v.Composition = comp.Composition()
v.Took.Composition = comp.Took
if comp.State == delivery.CompositionFail {
machine := comp.Machine
if machine == "" {
machine = "the composition"
}
v.State, v.Failed = delivery.VerdictFail, delivery.FailedFirst{Machine: machine, Words: comp.Words}
}
}
// notJudged says whether a verdict's composition was not judged because the assigner was not running.
func notJudged(v delivery.Verdict) bool {
return v.Composition != nil && v.Composition.State == delivery.CompositionNotRun && v.Composition.Words == delivery.AssignerNotRunning
}
// Composed takes in a composition the assigner wrote after the check's verdict said it was not judged because the
// assigner was not running (the assigner started since): the verdict and its status are written again with it. A
// composition of heads no candidate holds now, or of a verdict that judged its composition, is left.
func (ch *Checker) Composed(ctx context.Context, comp delivery.CompositionVerdict) {
ch.init()
if comp.State == delivery.CompositionWaits {
return
}
ckey, err := delivery.CandidatesKey(comp.Group)
if err != nil {
return
}
ch.mu.Lock()
c, ok := ch.known[ckey]
ch.mu.Unlock()
if !ok {
return
}
cs := CaseOf(c, ch.heads())
if cs == nil || cs.HeadsHash != comp.Heads {
return
}
cur, rev, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return
}
v, err := delivery.DecodeVerdict(cur)
if err != nil || v.State != delivery.VerdictPass || !notJudged(v) {
return
}
withComposition(&v, comp)
data, err := delivery.EncodeVerdict(v)
if err != nil {
ch.logf("%s: the verdict with its composition cannot be written: %v", cs.Key, err)
return
}
if _, err := ch.Verdicts.Update(ctx, cs.Key, data, rev); err != nil {
ch.logf("%s: the verdict with its composition could not be written: %v", cs.Key, err)
return
}
ch.writeOwed(ctx, cs, v, Outcome{Verdict: v})
ch.logf("%s: the composition came after the verdict: %s", cs.Key, comp.State)
}
@@ -0,0 +1,149 @@
package check
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func assignerOn(instance string, online bool) PresenceEvent {
key, _ := delivery.PresencePartKey(delivery.AssignerModule, instance)
return PresenceEvent{Key: key, Online: online}
}
func compEvent(t *testing.T, v delivery.CompositionVerdict) CompositionEvent {
t.Helper()
data, err := delivery.EncodeCompositionVerdict(v)
if err != nil {
t.Fatal(err)
}
return CompositionEvent{Value: data}
}
var passed = delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "x", Machines: []string{"g14"}}
// 1297/B3d: with no assigner online, the wait answers ErrAssignerNotRunning as soon as both watches handed what stands.
func TestNoAssignerAnswersAtOnce(t *testing.T) {
presence, comps := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence <- assignerOn("novox", false)
presence <- PresenceEvent{Settled: true}
comps <- CompositionEvent{Settled: true}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if _, err := AwaitComposition(ctx, presence, comps); !errors.Is(err, ErrAssignerNotRunning) {
t.Fatalf("answered %v, want the assigner not running", err)
}
}
// 1297/B3d: with the assigner online it waits for the key, past a composition that waits, and answers the verdict.
func TestAnOnlineAssignerIsWaitedFor(t *testing.T) {
presence, comps := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence <- assignerOn("novox", true)
presence <- PresenceEvent{Settled: true}
comps <- CompositionEvent{Settled: true}
done := make(chan error, 1)
var got delivery.CompositionVerdict
go func() {
var err error
got, err = AwaitComposition(context.Background(), presence, comps)
done <- err
}()
comps <- compEvent(t, delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionWaits, Words: "w"})
select {
case err := <-done:
t.Fatalf("answered before the composition was written: %v", err)
case <-time.After(50 * time.Millisecond):
}
comps <- compEvent(t, passed)
if err := <-done; err != nil || got.State != delivery.CompositionPass {
t.Fatalf("answered %+v, %v", got, err)
}
// And the last instance going offline during a wait ends it as not running.
presence2, comps2 := make(chan PresenceEvent, 4), make(chan CompositionEvent, 4)
presence2 <- assignerOn("novox", true)
presence2 <- PresenceEvent{Settled: true}
comps2 <- CompositionEvent{Settled: true}
presence2 <- assignerOn("novox", false)
if _, err := AwaitComposition(context.Background(), presence2, comps2); !errors.Is(err, ErrAssignerNotRunning) {
t.Errorf("the assigner going offline answered %v", err)
}
}
// 1297/B3d: a passing check with no assigner says at once, loud, that its composition is not judged; never a pass of it.
func TestAPassingCheckWithNoAssignerSaysTheCompositionIsNotJudged(t *testing.T) {
b := newBench(t)
b.ex.Compositions = &fakeCompositions{err: ErrAssignerNotRunning}
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.Composition == nil || v.Composition.State != delivery.CompositionNotRun || v.Composition.Words != delivery.AssignerNotRunning {
t.Fatalf("composition = %+v", v.Composition)
}
if words := compositionWords(v.Composition, 0); words != "composition not judged: the assigner is not running" {
t.Errorf("the line says %q", words)
}
}
// 1297/B3d: the assigner starting writes the composition later, and the checker takes it into the verdict that said it
// was not judged, writing the status again; a failed one fails the check.
func TestTheAssignerStartingBringsTheComposition(t *testing.T) {
for _, comp := range []delivery.CompositionVerdict{passed,
{Group: "feat/x", Heads: "", State: delivery.CompositionFail, SDK: "x", Machine: "g14", Words: "g14 runs web, which the change removes"}} {
ch, cs := checkerWithNotJudged(t)
comp.Heads = cs.HeadsHash
ch.Composed(context.Background(), comp)
raw, _, _, _ := ch.Verdicts.Get(context.Background(), cs.Key)
v, err := delivery.DecodeVerdict(raw)
if err != nil || v.Composition == nil || v.Composition.State != comp.State {
t.Fatalf("after the composition came, the verdict is %+v, %v", v, err)
}
if comp.State == delivery.CompositionFail && (v.State != delivery.VerdictFail || v.Failed.Machine != "g14") {
t.Errorf("a failed composition left the verdict %+v", v)
}
owed := ch.Owed.(*memKV)
keys, _ := owed.Keys(context.Background())
if len(keys) == 0 {
t.Fatal("no status was written again")
}
data, _, _, _ := owed.Get(context.Background(), keys[0])
o, _ := delivery.DecodeCheckerOwed(data)
if !strings.Contains(o.Text, "composition passed") && !strings.Contains(o.Text, "g14 runs web") {
t.Errorf("the status's line says %q", o.Text)
}
}
}
// checkerWithNotJudged is a checker holding a candidate whose passing verdict said its composition was not judged.
func checkerWithNotJudged(t *testing.T) (*Checker, *Case) {
t.Helper()
cs := caseOf(t, pullOf(repoA, 7, head1))
raw := candidateOf(t, cs.Group, pullOf(repoA, 7, head1))
ch := &Checker{Verdicts: newMemKV(), Owed: newMemKV(), Trunks: fixedTrunks{cs.Trunk}}
ckey, _ := delivery.CandidatesKey(cs.Group)
ch.init()
c, err := delivery.DecodeCandidate(raw)
if err != nil {
t.Fatal(err)
}
ch.known[ckey] = c
if got := CaseOf(c, ch.heads()); got == nil || got.Key != cs.Key {
t.Fatalf("the case is %+v, want %s", got, cs.Key)
}
v := delivery.Verdict{Group: cs.Group, Heads: cs.Heads, State: delivery.VerdictPass,
Levels: map[string]delivery.LevelResult{delivery.LevelUnit: {Run: 1}, delivery.LevelIntegration: {Run: 1}},
Composition: &delivery.Composition{State: delivery.CompositionNotRun, Words: delivery.AssignerNotRunning}}
data, err := delivery.EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
if _, err := ch.Verdicts.Create(context.Background(), cs.Key, data); err != nil {
t.Fatal(err)
}
return ch, cs
}
type fixedTrunks struct{ heads map[string]string }
func (f fixedTrunks) Heads() map[string]string { return f.heads }
@@ -26,7 +26,7 @@ const (
// checks/deps). Which commit of each is besideCommit's rule.
var besideRules = map[string][]string{
controllerRepo: {catalogueRepository, nodeEngineRepo, SDKRepository},
nodeEngineRepo: {catalogueRepository, SDKRepository},
nodeEngineRepo: {catalogueRepository, SDKRepository, controllerRepo},
SDKRepository: {catalogueRepository, nodeEngineRepo},
catalogueRepository: {SDKRepository, controllerRepo},
"novox/mesh-delivery": {SDKRepository, catalogueRepository},
@@ -35,13 +35,18 @@ var besideRules = map[string][]string{
// besideFor are the siblings a pull request's tests read (ADR 0484 clause 4): the catalogue at its trunk, always, even in a
// group holding a catalogue pull request; the node-engine at the commit the mesh runs (the commit of the indexer's module
// mesh-host, its trunk head when none is known); mesh-sdk, for the controller's tests, at the commit the controller's go.mod
// pins; any other sibling at the head of the group's pull request for it when the group holds one, else at its trunk.
// pins; any other sibling at the head of the group's pull request for it when the group holds one, else at its trunk. And
// first, the change's own repository at the trunk it is merged onto, so a test comparing a shrink-only list with the
// trunk's copy reads it at $MESH_CHECK_BESIDE/<name> (issue 1887 A12, ADR 0529 §2).
func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef, error) {
var heads map[string]string
if j.Trunks != nil {
heads = j.Trunks.Heads()
}
var out []BesideRef
if p.Trunk != "" {
out = append(out, BesideRef{Repository: p.Repository, Commit: p.Trunk})
}
for _, repo := range besideRules[p.Repository] {
switch {
case repo == catalogueRepository:
@@ -78,6 +83,10 @@ func (j *Examiner) besideFor(ctx context.Context, c *Case, p Pull) ([]BesideRef,
out = append(out, BesideRef{Repository: repo, Commit: member.Head, Ref: refOf(member.Number)})
} else if head := heads[repo]; head != "" {
out = append(out, BesideRef{Repository: repo, Commit: head})
} else if repo == controllerRepo && p.Repository == nodeEngineRepo {
// The node-engine's step-nine test judges the controller's own manifest and fails without it (issue 1295): a
// checker that cannot place it says so, where placing none left that test to skip.
return nil, fmt.Errorf("the trunk head of %s is not known, so the controller the tests of %s#%d read cannot be placed beside them", repo, p.Repository, p.Number)
}
}
}
@@ -0,0 +1,79 @@
package check
import (
"fmt"
"sort"
"sync"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ReadWorkers is how many values a read tool asks the bus for at once. Each value is a round trip from the tool through the
// runtime to the bus, so reading a bucket of thousands of keys one after the other outlasts the 30 s a tool call is given
// (issue 1827, as the delivery view's issue 1945). It is a copy of the view's: a few lines are not worth a group delivery
// through mesh-sdk (rules-are-weighed).
const ReadWorkers = 16
// MaxReads bounds the values one call of a read tool reads; past it the answer says it was cut and how to narrow it.
const MaxReads = 1000
// parallel runs read(i) for i in [0, n), at most ReadWorkers at a time, and answers the first error; once one failed no
// further read starts.
func parallel(n int, read func(i int) error) error {
var (
next atomic.Int64
failed atomic.Bool
once sync.Once
first error
wg sync.WaitGroup
)
for w := 0; w < min(ReadWorkers, n); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for !failed.Load() {
i := int(next.Add(1) - 1)
if i >= n {
return
}
if err := read(i); err != nil {
once.Do(func() { first = err })
failed.Store(true)
}
}
}()
}
wg.Wait()
return first
}
// capKeys keeps the first MaxReads keys and answers a note when it cut the rest, "" otherwise.
func capKeys(keys []string, narrow string) ([]string, string) {
if len(keys) <= MaxReads {
return keys, ""
}
return keys[:MaxReads], fmt.Sprintf("read %d of %d matching keys and stopped; %s", MaxReads, len(keys), narrow)
}
// newestFirst sorts step keys (<node>.<id>) by id, newest first: an id is the time in base 36, then six hex digits, so the
// longer id is the later and equal lengths compare as text. A key that is no step key goes last.
func newestFirst(keys []string) {
id := func(k string) (string, bool) {
_, id, err := delivery.ParseStepKey(k)
return id, err == nil
}
sort.Slice(keys, func(i, j int) bool {
a, aok := id(keys[i])
b, bok := id(keys[j])
switch {
case aok != bok:
return aok
case len(a) != len(b):
return len(a) > len(b)
case a != b:
return a > b
}
return keys[i] > keys[j]
})
}
@@ -3,6 +3,7 @@ package check
import (
"fmt"
"maps"
"slices"
"sort"
"strings"
"sync"
@@ -11,13 +12,6 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HQRepository is the repository the dark checker writes no status for (ADR 0484 clause 8): hq builds no module and its
// check is a script no build module can run, so it is judged by its merge-check.sh until issue 1296 settles how.
const HQRepository = "novox/hq"
// HQNote is what the read tools say of an hq pull request (ADR 0484 clause 8).
const HQNote = "hq: judged by merge-check.sh until issue 1296"
// Pull is one pull request the checker judges: its head, the trunk head of its repository it is merged onto and the
// modules the indexer says it touches, with the fingerprints the indexer wrote for them.
type Pull struct {
@@ -29,8 +23,8 @@ type Pull struct {
Fps map[string]delivery.CandidateModule
}
// Case is one check: a candidate's pull requests that are judged (those of hq and of a repository whose trunk is not known
// are left out), the trunk head of each repository and the key of the verdict.
// Case is one check: a candidate's pull requests that are judged (those of a repository whose trunk is not known are left
// out; hq is judged as any repository since R4g, its checks Go tests: ADR 0527), the trunk head of each repository and the key of the verdict.
type Case struct {
// Key is the verdict key, `<group>.<heads hash>`.
Key string
@@ -41,14 +35,20 @@ type Case struct {
Pulls []Pull
// Trunk is the trunk head of each repository of Pulls.
Trunk map[string]string
// Left says why pull requests of the candidate are not judged (hq's, a repository with no known trunk); empty when all are.
// Left says why pull requests of the candidate are not judged (a repository with no known trunk); empty when all are.
Left []string
// Queued is when the checker took the case up, for the time of its queue.
Queued time.Time
// Try narrows a try's check (its level, its tests); nil for a pull request's.
Try *TryOptions
notesMu sync.Mutex
notes []string
// Progress says what the check is doing, so a check that ran past its bound names the step it was stuck at.
Progress *Progress
// Kind is what asked the check (KindNew, KindNodes, KindTrunk): a re-judge after a trunk move is asked on the build
// seats' recheck stream, every other check on check (ADR 0536 §4).
Kind string
}
// Progress is the step a check is at, written by the check and read when its bound passes.
@@ -83,8 +83,6 @@ func CaseOf(c delivery.Candidate, trunk map[string]string) *Case {
cs := &Case{Key: key, Group: c.Group, HeadsHash: c.HeadsHash, Heads: GateHeads(c), Trunk: map[string]string{}, Progress: &Progress{}}
for _, p := range c.Pulls {
switch head, known := trunk[p.Repository]; {
case p.Repository == HQRepository:
cs.Left = append(cs.Left, fmt.Sprintf("%s#%d: %s", p.Repository, p.Number, HQNote))
case !known || head == "":
cs.Left = append(cs.Left, fmt.Sprintf("%s#%d: the trunk head of %s is not known", p.Repository, p.Number, p.Repository))
default:
@@ -154,3 +152,19 @@ func (t *Trunks) Heads() map[string]string {
defer t.mu.Unlock()
return maps.Clone(t.heads)
}
// note puts a word of the check's way on the case, for a try's line (issue 1921).
func (c *Case) note(s string) {
c.notesMu.Lock()
defer c.notesMu.Unlock()
if !slices.Contains(c.notes, s) {
c.notes = append(c.notes, s)
}
}
// Notes are the words noted on the case, in the order they were noted.
func (c *Case) Notes() []string {
c.notesMu.Lock()
defer c.notesMu.Unlock()
return slices.Clone(c.notes)
}
@@ -11,12 +11,11 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// DefaultBound is how long a check may run before it is ended failed naming the step it was stuck at (ADR 0484, issue 1678).
const DefaultBound = 30 * time.Minute
// NodesWords is in the Words of a failure that is the build nodes' (none qualifies, the toolchains disagree): such a verdict
// is judged again when the nodes change, by a watch and never a timer (ADR 0437 §8).
const NodesWords = "build node qualifies"
// DefaultBound is how long a check may run before it is ended failed naming the step it was stuck at (ADR 0484, issue 1678):
// none since R4g. A check's asks wait in the build seats' streams as long as the backlog is (ADR 0525 §9), and each ask's
// step is bounded from its start on the node that took it (buildrun.Asker.Bound, DefaultStepBound); a bound set by
// MESH_CHECKER_BOUND still ends a check that ran past it.
const DefaultBound time.Duration = 0
// UnheardWords end the Words of a failure because a part of the mesh the check reads was not heard (the forge, issue 1624):
// like a failure of the nodes, it is not the change's, and is judged again when the nodes change.
@@ -40,6 +39,9 @@ type Outcome struct {
Builds []BuildNote
// Slowest names the slowest tests of a level that went over its bound ("unit pkg.Test 1.2s").
Slowest []string
// Judged says, by pull request (pullKey), which program judged a change to a build module (ADR 0535 §5); a pull request
// that changes none is not named.
Judged map[string]string
}
// BuildNote is the state of a candidate's module build, by module: built, failed or waiting (ADR 0484 clause 7).
@@ -63,12 +65,8 @@ type Checker struct {
Owed KV
Judge Judge
Trunks interface{ Heads() map[string]string }
// Bound is how long one check may run; DefaultBound when 0.
// Bound is how long one check may run; none when 0 (DefaultBound).
Bound time.Duration
// Limit is how many checks run at once; two when 0.
Limit int
// Hold says whether a delivery builds, so no new check starts (ADR 0484 clause 12); nil never holds.
Hold func() bool
// Link is the base of the address a status points at; the verdict key follows.
Link string
// Announce says a status was written (EventOwedWritten); nil says nothing.
@@ -77,7 +75,6 @@ type Checker struct {
Logf func(format string, args ...any)
once sync.Once
sem chan struct{}
wg sync.WaitGroup
mu sync.Mutex
known map[string]delivery.Candidate // by candidates key
@@ -90,11 +87,6 @@ type Checker struct {
func (ch *Checker) init() {
ch.once.Do(func() {
limit := ch.Limit
if limit <= 0 {
limit = 2
}
ch.sem = make(chan struct{}, limit)
ch.known, ch.inflight, ch.again, ch.tried = map[string]delivery.Candidate{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
ch.nodesFail = map[string]uint64{}
})
@@ -181,11 +173,7 @@ func (ch *Checker) heads() map[string]string {
func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidate) {
cs := CaseOf(c, ch.heads())
if cs == nil {
return // nothing to judge: hq's pull requests, or no trunk known
}
if ch.Hold != nil && ch.Hold() {
ch.logf("%s: not started, a delivery builds", cs.Key)
return // started when the plan bucket empties (Reconsider)
return // nothing to judge: no trunk known
}
cs.Queued = ch.now()
ch.mu.Lock()
@@ -199,9 +187,11 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
ch.wg.Add(1)
go func() {
defer ch.wg.Done()
ch.sem <- struct{}{}
defer func() { <-ch.sem }()
for {
// Every check is asked at once: the build seats' streams hold the backlog, a new head's check and a check after a
// failure of the nodes on check, a re-judge after a trunk move on recheck, pulled after them (ADR 0525 §4, §9;
// ADR 0536 §4). The checker holds no place and no capacity of its own.
cs.Kind = ch.kindOf(ctx, cs)
ch.try(ctx, key, cs)
ch.mu.Lock()
more := ch.again[cs.Key]
@@ -215,10 +205,37 @@ func (ch *Checker) consider(ctx context.Context, key string, c delivery.Candidat
if next := CaseOf(ch.candidate(key, c), ch.heads()); next != nil {
cs = next
}
cs.Queued = ch.now()
}
}()
}
// The kinds of a check (ADR 0536 §4): a head never judged and a head asked again after a failure of the nodes are asked on
// the build seats' check stream, a head asked again only because its trunk moved on the recheck stream, pulled after.
const (
KindNew = "new"
KindNodes = "nodes"
KindTrunk = "trunk"
)
// kindOf is the kind of a case's check, read from its verdict key: none (or one left running) is a head never judged; a
// failure of the nodes is asked again after the nodes changed; a verdict on other trunk heads is a re-judge after a trunk
// move. A verdict that stands is ranked with the re-judges: its claim finds nothing to do.
func (ch *Checker) kindOf(ctx context.Context, cs *Case) string {
value, _, found, err := ch.Verdicts.Get(ctx, cs.Key)
if err != nil || !found {
return KindNew
}
v, err := delivery.DecodeVerdict(value)
switch {
case err != nil || v.State == delivery.VerdictRunning:
return KindNew
case IsNodesFailure(v):
return KindNodes
}
return KindTrunk
}
func (ch *Checker) candidate(key string, fallback delivery.Candidate) delivery.Candidate {
ch.mu.Lock()
defer ch.mu.Unlock()
@@ -289,9 +306,11 @@ func (ch *Checker) claim(ctx context.Context, cs *Case, first bool, epoch uint64
return &claimed{rev}, nil
}
// IsNodesFailure says whether a verdict failed because no build node could take the check.
// IsNodesFailure says whether a verdict failed because a part of the mesh the check reads was not heard (UnheardWords):
// it is judged again when the nodes change. No check fails for want of a build node any more: its asks wait in the build
// seats' streams (ADR 0525 §6, issue 1912).
func IsNodesFailure(v delivery.Verdict) bool {
return v.State == delivery.VerdictFail && (strings.Contains(v.Failed.Words, NodesWords) || strings.Contains(v.Failed.Words, UnheardWords))
return v.State == delivery.VerdictFail && strings.Contains(v.Failed.Words, UnheardWords)
}
// try claims the check, judges it within its bound and writes the verdict and the owed statuses. A check that ends
@@ -311,7 +330,7 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
return
}
began := ch.now()
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, nil, nil)
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
out := ch.judge(ctx, cs)
if ctx.Err() != nil {
return
@@ -342,11 +361,11 @@ func (ch *Checker) try(ctx context.Context, key string, cs *Case) {
if ch.movedSince(cs) {
// Judged on a trunk that moved while it ran: the verdict is kept as judged, and the status stays pending until the
// check of the new trunk, which Reconsider starts, ends.
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, nil, nil)
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
ch.logf("%s: %s on a trunk that moved while it ran; the status stays pending", cs.Key, v.State)
return
}
ch.writeOwed(ctx, cs, v, out.Builds, out.Slowest)
ch.writeOwed(ctx, cs, v, out)
ch.logf("%s: %s", cs.Key, v.State)
}
@@ -366,7 +385,7 @@ func failure(machine, words string) delivery.Verdict {
// writeOwed writes, for each pull request judged, the status the forge bridge performs: pending while the check runs, success
// or failure at its end (ADR 0484 clause 2). A key already holding the same value is not written again, so the bridge
// performs one act for each change; an older head's key is never touched.
func (ch *Checker) writeOwed(ctx context.Context, cs *Case, v delivery.Verdict, builds []BuildNote, slowest []string) {
func (ch *Checker) writeOwed(ctx context.Context, cs *Case, v delivery.Verdict, out Outcome) {
for _, p := range cs.Pulls {
key, err := delivery.CheckerOwedKey(p.Repository, p.Number, p.Head)
if err != nil {
@@ -375,7 +394,7 @@ func (ch *Checker) writeOwed(ctx context.Context, cs *Case, v delivery.Verdict,
}
state := map[string]string{delivery.VerdictRunning: "pending", delivery.VerdictPass: "success", delivery.VerdictFail: "failure"}[v.State]
o := delivery.Owed{Act: delivery.OwedStatus, Context: delivery.StatusContextCheck, State: state,
Target: strings.TrimSuffix(ch.Link, "/") + "/" + cs.Key, Text: Line(v, cs, p, builds, slowest)}
Target: strings.TrimSuffix(ch.Link, "/") + "/" + cs.Key, Text: Line(v, cs, p, out.Builds, out.Slowest, out.Judged)}
data, err := delivery.EncodeCheckerOwed(o)
if err != nil {
ch.logf("%s: the status of %s cannot be written: %v", cs.Key, key, err)
@@ -486,7 +505,7 @@ func (ch *Checker) TrunkMoved(ctx context.Context) {
if v, err := delivery.DecodeVerdict(value); err == nil && sameTrunk(v.Trunk, cs.Trunk) {
continue
}
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, nil, nil)
ch.writeOwed(ctx, cs, delivery.Verdict{State: delivery.VerdictRunning}, Outcome{})
}
ch.Reconsider(ctx)
}
@@ -11,6 +11,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const hqRepo = "novox/hq"
const (
repoA = "novox/mesh-delivery"
repoSDK = "novox/mesh-sdk"
@@ -206,11 +208,13 @@ func TestACheckPastItsBoundFailsNamingTheStuckStep(t *testing.T) { // N6/AC3
}
}
func TestNoQualifyingNodeFailsWithTheReasonAndIsJudgedAgainWhenTheNodesChange(t *testing.T) { // N6/AC4
// A check whose forge was not heard is judged again when the nodes change (issue 1912: no check fails for want of a build
// node any more; its asks wait in the streams).
func TestAnUnheardForgeFailsWithTheReasonAndIsJudgedAgainWhenTheNodesChange(t *testing.T) { // N6/AC4
ctx := context.Background()
j := &judgeFunc{f: func(_ context.Context, _ *Case, n int) (Outcome, error) {
if n == 1 {
return Outcome{Verdict: failure("shanks", "no go "+NodesWords+": none is online")}, nil
return Outcome{Verdict: failure("the forge", "novox/mesh-delivery#7: go.mod cannot be read: no responders (the forge was "+UnheardWords+")")}, nil
}
return Outcome{Verdict: passing()}, nil
}}
@@ -234,10 +238,9 @@ func TestNoQualifyingNodeFailsWithTheReasonAndIsJudgedAgainWhenTheNodesChange(t
}
}
func TestHqAndARepositoryWithNoTrunkGetNoVerdict(t *testing.T) { // N6/AC6, ADR 0484 clauses 8 and 11
func TestARepositoryWithNoTrunkGetsNoVerdict(t *testing.T) { // N6/AC6, ADR 0484 clause 11
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
r.ch.Candidate(ctx, "feat/hq", candidateOf(t, "feat/hq", pullOf(HQRepository, 3, head1)))
r.ch.Candidate(ctx, "feat/y", candidateOf(t, "feat/y", pullOf("novox/unknown", 4, head1)))
r.ch.Wait()
if j.count() != 0 || len(r.verdicts.vals) != 0 || len(r.owed.vals) != 0 {
@@ -245,17 +248,24 @@ func TestHqAndARepositoryWithNoTrunkGetNoVerdict(t *testing.T) { // N6/AC6, ADR
}
}
func TestAGroupWithHqJudgesItsOtherMembersAndWritesNoStatusForHq(t *testing.T) { // ADR 0484 clause 8
// 1887/A10, ADR 0527 §2: hq is judged as any repository, its checks Go tests, and its pull request gets a verdict and a
// mesh/check; in a group, hq's member is judged with the others.
func TestAnHqPullRequestGetsAVerdictAndAStatus(t *testing.T) {
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
r.ch.Candidate(ctx, "feat/x", candidateOf(t, "feat/x", pullOf(HQRepository, 3, head1), pullOf(repoA, 7, head2)))
r.setTrunks(t, map[string]string{repoA: trunkA, repoSDK: trunkS, hqRepo: trunkA2})
r.ch.Candidate(ctx, "feat/hq", candidateOf(t, "feat/hq", pullOf(hqRepo, 3, head1)))
r.ch.Wait()
if j.count() != 1 {
t.Fatalf("judged %d times, want once", j.count())
if j.count() != 1 || len(r.verdicts.vals) != 1 {
t.Fatalf("judged %d times, %d verdicts, want one", j.count(), len(r.verdicts.vals))
}
keys, _ := r.owed.Keys(ctx)
if len(keys) != 1 || !strings.HasPrefix(keys[0], "novox/mesh-delivery.7.") && !strings.Contains(keys[0], "mesh-delivery") {
t.Errorf("statuses %v, want one for mesh-delivery#7 only", keys)
if s := owedState(t, r, hqRepo, 3, head1); s != "success" {
t.Errorf("hq#3 owes %q, want success", s)
}
r.ch.Candidate(ctx, "feat/x", candidateOf(t, "feat/x", pullOf(hqRepo, 4, head2), pullOf(repoA, 7, head2)))
r.ch.Wait()
if s := owedState(t, r, hqRepo, 4, head2); s != "success" {
t.Errorf("hq#4 in a group owes %q, want success", s)
}
}
@@ -298,24 +308,6 @@ func TestATrunkMoveRewritesTheSameVerdictKeyAndTheHeadsStatus(t *testing.T) { //
}
}
func TestNoCheckStartsWhileADeliveryBuildsAndOneStartsWhenItEnds(t *testing.T) { // N8/AC1
ctx, j := context.Background(), passJudge()
r := newRig(t, j)
held := true
r.ch.Hold = func() bool { return held }
r.ch.Candidate(ctx, "feat/x", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
r.ch.Wait()
if j.count() != 0 {
t.Fatalf("judged %d times while held", j.count())
}
held = false
r.ch.Reconsider(ctx)
r.ch.Wait()
if j.count() != 1 {
t.Fatalf("judged %d times after the hold ended, want once", j.count())
}
}
func TestTheStatusIsPendingWhileTheCheckRunsAndEndsSuccessOrFailure(t *testing.T) { // 1300/AC5 (unit part)
ctx := context.Background()
var seen []string
@@ -0,0 +1,80 @@
package check
import (
"context"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// fakeCompositions answers the assigner's composition of a check from memory, recording the keys asked.
type fakeCompositions struct {
v delivery.CompositionVerdict
err error
asked []string
}
func (f *fakeCompositions) Await(_ context.Context, key string) (delivery.CompositionVerdict, error) {
f.asked = append(f.asked, key)
return f.v, f.err
}
// 1297/AC3 (its unit half): a check whose tests pass takes the assigner's composition of it, by the check's own key,
// into its verdict: a pass says it passed with the mesh-sdk and machines it composed, and is no longer "not run".
func TestAPassingCheckTakesTheAssignersComposition(t *testing.T) {
b := newBench(t)
f := &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionPass, SDK: "v0.1.14-x", Machines: []string{"g14"}, Took: 40}}
b.ex.Compositions = f
cs := caseOf(t, pullOf(repoA, 7, head1))
v := b.judge(t, cs).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionPass || v.Composition.SDK != "v0.1.14-x" || v.Took.Composition != 40 {
t.Fatalf("verdict = %+v, composition %+v", v, v.Composition)
}
if len(f.asked) != 1 || f.asked[0] != cs.Key {
t.Errorf("asked %v, want the check's key %s", f.asked, cs.Key)
}
if !compositionMentions(v, "passed") {
t.Errorf("the line says %q", compositionWords(v.Composition, v.Took.Composition))
}
}
func compositionMentions(v delivery.Verdict, word string) bool {
return strings.Contains(compositionWords(v.Composition, v.Took.Composition), word)
}
// 1297/AC2 at the checker: a failed composition fails the check, naming the machine it failed at and its words.
func TestAFailedCompositionFailsTheCheck(t *testing.T) {
b := newBench(t)
b.ex.Compositions = &fakeCompositions{v: delivery.CompositionVerdict{Group: "feat/x", Heads: "0123456789ab", State: delivery.CompositionFail, SDK: "x",
Machine: "g14", Words: "g14 runs web, which the change removes from its source"}}
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.State != delivery.VerdictFail || v.Failed.Machine != "g14" || v.Composition == nil || v.Composition.State != delivery.CompositionFail {
t.Fatalf("verdict = %+v", v)
}
v.Group, v.Heads = "feat/x", "0123456789ab" // the checker names them when it writes the verdict
if _, err := delivery.EncodeVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
}
// 1297/B3d: a composition that cannot be read leaves it not run; a try is not composed.
func TestACompositionNotHadIsNotRun(t *testing.T) {
b := newBench(t)
f := &fakeCompositions{err: errors.New("the check's bound passed")}
b.ex.Compositions = f
v := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if v.State != delivery.VerdictPass || v.Composition == nil || v.Composition.State != delivery.CompositionNotRun {
t.Fatalf("verdict = %+v", v)
}
b2 := newBench(t)
f2 := &fakeCompositions{v: delivery.CompositionVerdict{State: delivery.CompositionFail, Words: "w", SDK: "x"}}
b2.ex.Compositions = f2
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.Try = &TryOptions{}
b2.judge(t, cs)
if len(f2.asked) != 0 {
t.Errorf("a try asked for a composition: %v", f2.asked)
}
}
@@ -0,0 +1,83 @@
package check
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1930/AC3: given a check ask whose stored end keeps the digest's packages as a list of {package, digest} (issue 1930),
// or one stored before it with the packages a map keyed by package path, when the same heads are checked a second time,
// then the kept verdicts the first check wrote are read from it: every package it passed is kept, none run again.
func TestKeptVerdictsAreReadFromTheDigestsListForm(t *testing.T) {
for _, tc := range []struct {
name string
old bool
}{{"the list form", false}, {"a map stored before issue 1930", true}} {
t.Run(tc.name, func(t *testing.T) {
b := newBench(t)
var stored []string
b.asks.Answer = func(a delivery.StepAsk) delivery.Step {
s := b.answer(a)
if tc.old && s.State == delivery.StepPassed {
s.Result = oldDigestForm(t, s.Result)
}
stored = append(stored, string(s.Result))
return s
}
first := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
u := first.Levels[delivery.LevelUnit]
if first.State != delivery.VerdictPass || u.Run == 0 || u.Kept != 0 {
t.Fatalf("the first check = %+v", first)
}
if len(stored) == 0 {
t.Fatal("no ask's end was stored")
}
list := strings.Contains(stored[0], `"packages":[{"package":"fixture/slow"`)
if list == tc.old || strings.Contains(stored[0], `"fixture/slow":"`) != tc.old {
t.Fatalf("the stored end is not in the form under test: %s", stored[0])
}
b.steps.calls = nil
second := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if s := second.Levels[delivery.LevelUnit]; second.State != delivery.VerdictPass || s.Kept != u.Run || s.Run != 0 {
t.Fatalf("the second check's unit level = %+v, want %d kept and none run", s, u.Run)
}
if n := len(b.steps.tool("test")); n != 0 {
t.Errorf("%d test steps ran on the second check", n)
}
})
}
}
// oldDigestForm rewrites a check answer's level digests to the form stored before issue 1930: packages as a map keyed by
// package path.
func oldDigestForm(t *testing.T, result json.RawMessage) json.RawMessage {
t.Helper()
var a struct {
Fetch json.RawMessage `json:"fetch"`
Levels []map[string]any `json:"levels"`
}
if err := json.Unmarshal(result, &a); err != nil {
t.Fatal(err)
}
for _, l := range a.Levels {
d, ok := l["digest"].(map[string]any)
if !ok {
continue
}
m := map[string]any{}
list, _ := d["packages"].([]any)
for _, p := range list {
pd := p.(map[string]any)
m[pd["package"].(string)] = pd["digest"]
}
d["packages"] = m
}
raw, err := json.Marshal(a)
if err != nil {
t.Fatal(err)
}
return raw
}
@@ -15,6 +15,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
@@ -108,6 +109,7 @@ const digD = "dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44dd44"
type bench struct {
ex *Examiner
asks *buildrun.FakeAsks
steps *fakeSteps
kept *memKV
prev *memKV
@@ -116,6 +118,50 @@ type bench struct {
test func(c stepCall) (any, error)
}
// refsOf reads the siblings of a fetch's arguments, as they arrive in an ask (JSON) or as the checker gave them.
func refsOf(v any) []BesideRef {
raw, _ := json.Marshal(v)
var refs []BesideRef
_ = json.Unmarshal(raw, &refs)
return refs
}
// toolPrefix is the start of each build module's tool names, as the fake build modules answer them.
var toolPrefix = map[string]string{buildrun.GoBuildModule: "build_go_", buildrun.NodeBuildModule: "build_node_"}
// answer runs a check ask as the build module that pulls it does (buildstep.RunAsk), with the fake tools on shanks and the
// bench's kept verdicts, and answers the step it writes.
func (b *bench) answer(a delivery.StepAsk) delivery.Step {
module, _ := delivery.BuildModuleOf(a.Language)
const node = "shanks"
tool := func(name string) buildstep.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) {
raw, err := b.steps.Run(ctx, module, node, toolPrefix[module]+name, args)
if err != nil {
return nil, err
}
return raw, nil
}
}
steps := buildstep.AskSteps{Fetch: tool("fetch"), Test: tool("test"),
Kept: func(ctx context.Context, level, digest string) (int, bool, error) {
k, found, err := (Kept{Store: b.kept}).Lookup(ctx, level, digest)
return k.Tests, found, err
}}
if module == buildrun.GoBuildModule {
steps.Digest = tool("digest")
}
out, err := buildstep.RunAsk(context.Background(), steps, a)
began := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
step := delivery.Step{ID: a.ID, Tool: a.Kind + " ask", Node: node, State: delivery.StepPassed, Started: began, Ended: began.Add(2 * time.Second)}
if err != nil {
step.State, step.Reason = delivery.StepFailed, err.Error()
return step
}
step.Result, _ = json.Marshal(out)
return step
}
// newBench is an examiner over fake build modules that merge cleanly, digest the captured fixture's packages and pass
// the captured slow fixture's result at both levels, on shanks.
func newBench(t testing.TB) *bench {
@@ -137,12 +183,13 @@ func newBench(t testing.TB) *bench {
b.steps.handlers = map[string]func(stepCall) (any, error){
"build_go_fetch": func(stepCall) (any, error) { return b.fetch, nil },
"build_node_fetch": func(stepCall) (any, error) { return b.fetch, nil },
"build_go_digest": func(c stepCall) (any, error) { return b.digest[c.args["level"].(string)], nil },
"build_go_digest": func(c stepCall) (any, error) { return echoBeside(b, b.digest[c.args["level"].(string)]), nil },
"build_go_test": func(c stepCall) (any, error) { return b.test(c) },
"build_node_test": func(c stepCall) (any, error) { return b.test(c) },
}
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
b.ex = &Examiner{Steps: b.steps, Toolchains: goEntries("shanks"), Places: &buildrun.Places{}, Kept: Kept{Store: b.kept}, Previews: b.prev,
b.asks = &buildrun.FakeAsks{Answer: b.answer}
b.ex = &Examiner{Asks: buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 5 * time.Second}, Toolchains: goEntries("shanks"), Kept: Kept{Store: b.kept}, Previews: b.prev,
Modules: langs{}, Trunks: &Trunks{}, Now: func() time.Time { now = now.Add(50 * time.Millisecond); return now }}
return b
}
@@ -353,33 +400,6 @@ func TestAHeadThatDoesNotMergeEndsFailedNamingTheFilesAndRunsNoTest(t *testing.T
}
}
func TestNoQualifyingNodeIsAFailureOfTheNodes(t *testing.T) { // N6/AC4
b := newBench(t)
b.ex.Toolchains = entries{}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if !IsNodesFailure(out.Verdict) {
t.Fatalf("verdict = %+v, want a failure of the nodes", out.Verdict)
}
b.ex.Toolchains = goEntries("shanks")
b.ex.Presence = offline{}
if out = b.judge(t, caseOf(t, pullOf(repoA, 7, head1))); !IsNodesFailure(out.Verdict) {
t.Fatalf("an offline node: verdict = %+v", out.Verdict)
}
}
type offline struct{}
func (offline) Online(string) bool { return false }
func TestAStepThatFailsIsAVerdictNamingTheNode(t *testing.T) {
b := newBench(t)
b.steps.handlers["build_go_fetch"] = func(stepCall) (any, error) { return nil, fmt.Errorf("the step failed: ssh refused") }
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if f := out.Verdict.Failed; out.Verdict.State != delivery.VerdictFail || f.Machine != "shanks" || !strings.Contains(f.Words, "ssh refused") {
t.Fatalf("verdict = %+v", out.Verdict)
}
}
func TestALevelWhoseDigestIsNotNamedRunsWholeAndKeepsNothing(t *testing.T) { // 1287/AC4
b := newBench(t)
b.steps.handlers["build_go_digest"] = func(stepCall) (any, error) { return nil, fmt.Errorf("go list failed") }
@@ -441,7 +461,7 @@ func TestAGroupIsOneVerdictAndEachMembersTestsReadTheOthersAtTheirHeads(t *testi
var forA []BesideRef
for _, f := range fetch {
if f.args["repository"] == repoA {
forA, _ = f.args["beside"].([]BesideRef)
forA, _ = refsOf(f.args["beside"]), true
}
}
found := false
@@ -571,6 +591,8 @@ func trunksOf(t *testing.T, heads map[string]string) *Trunks {
func TestNothingIsKeptForAWorkspaceGivenSiblingsWhenTheDigestDoesNotSayItCoversThem(t *testing.T) { // review D1, ADR 0444, issue 1287
b := newBench(t)
// A build module whose digest does not echo the siblings (before DigestVersion 3).
b.steps.handlers["build_go_digest"] = func(c stepCall) (any, error) { return b.digest[c.args["level"].(string)], nil }
b.ex.Trunks = trunksOf(t, map[string]string{repoSDK: trunkS, "novox/mesh-catalog": trunkA})
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if got := len(b.steps.tool("fetch")); got != 1 || b.steps.tool("fetch")[0].args["beside"] == nil {
@@ -592,12 +614,27 @@ func TestNothingIsKeptForAWorkspaceGivenSiblingsWhenTheDigestDoesNotSayItCoversT
// besideDigest stands for the Go build module's digest over a workspace with siblings (DigestVersion 3, lead decision L5):
// each package's digest folds the siblings' commits in, and the answer echoes the lines it folded.
// echoBeside is d with the siblings the fetches placed, as the Go build module's digest echoes them (its DigestResult's
// Beside): every check now places the change's own repository at its trunk (issue 1887 A12).
func echoBeside(b *bench, d Digested) Digested {
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
}
sort.Strings(lines)
d.Beside = lines
return d
}
func besideDigest(b *bench) func(c stepCall) (any, error) {
return func(c stepCall) (any, error) {
level := c.args["level"].(string)
var lines []string
for _, f := range b.steps.tool("fetch") {
refs, _ := f.args["beside"].([]BesideRef)
refs, _ := refsOf(f.args["beside"]), true
for _, r := range refs {
lines = append(lines, r.Repository+" "+r.Commit)
}
@@ -723,3 +760,86 @@ func TestNoStepOfACheckCarriesPublish(t *testing.T) { // ADR 0500 §4, 1309 (the
}
}
}
// 1890/AC1, 1887/A5a: an ask whose step never ends (the build node stopped with it running, and nothing redelivered it in
// time) ends the check loudly at the asker's bound; the checker holds no place to free.
func TestAnAskWhoseStepNeverEndsEndsTheCheckLoudly(t *testing.T) {
b := newBench(t)
// A node took the ask and its step stays running: the bound runs from its start (review round 2 item 1).
b.asks.Answer = func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Tool: "check ask", Node: "shanks", State: delivery.StepRunning, Started: time.Now()}
}
b.ex.Asks = buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 20 * time.Millisecond}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if f := out.Verdict.Failed; out.Verdict.State != delivery.VerdictFail || !strings.Contains(f.Words, "on shanks did not end within") {
t.Fatalf("verdict = %+v", out.Verdict)
}
if n := len(b.asks.Asks()); n != 1 || b.asks.Asks()[0].Kind != delivery.AskCheck || b.asks.Asks()[0].Check == nil {
t.Fatalf("asks = %+v, want one check ask", b.asks.Asks())
}
}
// 1887/A5a, ADR 0536 §4: a check is one ask per pull request and build module on the check stream, a re-judge after a
// trunk move one on the recheck stream, each carrying both levels with the kept verdicts taken by the build module.
func TestACheckIsOneAskAndAReJudgeGoesOnTheRecheckStream(t *testing.T) {
b := newBench(t)
b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
asks := b.asks.Asks()
if len(asks) != 1 || asks[0].Kind != delivery.AskCheck || asks[0].Language != delivery.AskGo || asks[0].Asker != delivery.CheckerModule {
t.Fatalf("asks = %+v, want one Go check ask", asks)
}
if j := asks[0].Check; j == nil || !j.Kept || strings.Join(j.Levels, ",") != "unit,integration" || j.Fetch["repository"] != repoA {
t.Fatalf("the job = %+v", asks[0].Check)
}
cs := caseOf(t, pullOf(repoA, 7, head1))
cs.Kind = KindTrunk
b.judge(t, cs)
if asks := b.asks.Asks(); len(asks) != 2 || asks[1].Kind != delivery.AskRecheck {
t.Fatalf("asks = %+v, want the re-judge on recheck", asks)
}
}
// Review round 2 item 1: a check ask that waits in its stream longer than the step bound (no node has a free slot during a
// large rebuild) is not failed: the check goes on when a node takes it, and no second ask is put.
func TestACheckAskWaitingInItsStreamIsNotBounded(t *testing.T) {
b := newBench(t)
answer := b.asks.Answer
b.asks.Answer = nil
b.ex.Asks = buildrun.Asker{Publisher: b.asks, Waiter: b.asks, Bound: 20 * time.Millisecond}
done := make(chan Outcome, 1)
go func() { done <- b.judge(t, caseOf(t, pullOf(repoA, 7, head1))) }()
for deadline := time.Now().Add(2 * time.Second); len(b.asks.Asks()) == 0; time.Sleep(time.Millisecond) {
if time.Now().After(deadline) {
t.Fatal("no ask was put")
}
}
time.Sleep(100 * time.Millisecond) // five bounds in the stream
select {
case out := <-done:
t.Fatalf("the check ended while its ask waited: %+v", out.Verdict)
default:
}
b.asks.End(answer(b.asks.Asks()[0])) // a node took it and ran it
if out := <-done; out.Verdict.State != delivery.VerdictPass || len(b.asks.Asks()) != 1 {
t.Fatalf("verdict %+v after %d asks, want a pass from the one ask", out.Verdict, len(b.asks.Asks()))
}
}
// 1917: the digest is taken inside the check ask, as the build module's own digest step answers it (no step key is asked
// of it), so a second check of the same heads keeps every package the first one passed: "N kept", nothing run again.
func TestASecondCheckOfTheSameHeadsKeepsWhatTheFirstPassed(t *testing.T) {
b := newBench(t)
first := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
u := first.Levels[delivery.LevelUnit]
if first.State != delivery.VerdictPass || u.Run == 0 || u.Kept != 0 {
t.Fatalf("the first check = %+v", first)
}
b.steps.calls = nil
second := b.judge(t, caseOf(t, pullOf(repoA, 7, head1))).Verdict
if s := second.Levels[delivery.LevelUnit]; second.State != delivery.VerdictPass || s.Kept != u.Run || s.Run != 0 {
t.Fatalf("the second check's unit level = %+v, want %d kept and none run", s, u.Run)
}
if n := len(b.steps.tool("test")); n != 0 {
t.Errorf("%d test steps ran on the second check", n)
}
}
@@ -0,0 +1,73 @@
package check
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The failure the build module reported for group r4g on 2026-10-23: the test logged a line, then failed with a multi-line
// t.Fatalf. Its output is what the build module now sends (its last lines, with the error line as Cause).
func fatalAfterLog() Failure {
return Failure{
Package: "github.com/novox/mesh-controller/internal/broker", Test: "TestTheCataloguesDerivedSubjectsMeet",
Cause: "3 subscription(s) derive to a subject no emitter owns:",
Output: " agreement_catalogue_test.go:130: judged against /var/lib/mesh-build-go-work/id/beside/mesh-catalog as cloned\n" +
" agreement_catalogue_test.go:186: 3 subscription(s) derive to a subject no emitter owns:\n" +
" a.b.c\n d.e.f\n g.h.i\n--- FAIL: TestTheCataloguesDerivedSubjectsMeet (0.00s)",
}
}
func TestAFailedTestsVerdictKeepsItsWholeFailureMessage(t *testing.T) { // 1901/AC1
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{fatalAfterLog()}})
for _, want := range []string{"3 subscription(s) derive to a subject no emitter owns:", "a.b.c", "d.e.f", "g.h.i"} {
if !strings.Contains(v.Failed.Words, want) {
t.Errorf("words lack %q: %q", want, v.Failed.Words)
}
}
if len(v.Failed.Words) > MaxWords+200 {
t.Errorf("words are %d bytes, bound %d", len(v.Failed.Words), MaxWords)
}
}
func TestTheStatusLineNamesTheFailureNotTheFirstLoggedLine(t *testing.T) { // 1901/AC2
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{fatalAfterLog()}})
got := firstFailed(*v)
if !strings.Contains(got, "3 subscription(s) derive to a subject no emitter owns:") || strings.Contains(got, "judged against") || strings.Contains(got, "\n") {
t.Errorf("firstFailed = %q", got)
}
// With no cause told apart (an older build module), the words still start at the first line, as before.
f := fatalAfterLog()
f.Cause = ""
v = levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if !strings.HasPrefix(v.Failed.Words, "agreement_catalogue_test.go:130") {
t.Errorf("words = %q", v.Failed.Words)
}
}
func TestWordsLongerThanTheBoundKeepTheirLastLines(t *testing.T) { // 1901/AC1
f := Failure{Output: strings.Repeat("log line of some length\n", 500) + "the cause is here"}
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if !strings.HasSuffix(v.Failed.Words, "the cause is here") || len(v.Failed.Words) > MaxWords+200 {
t.Errorf("%d bytes, ends %q", len(v.Failed.Words), v.Failed.Words[len(v.Failed.Words)-30:])
}
_ = delivery.VerdictFail
}
// An older build module sends no "cause": its answer is still accepted and the verdict's words are its output, as before.
func TestAnOlderBuildModulesFailureIsAcceptedWithItsWholeOutput(t *testing.T) { // 1901/AC3
old := `{"package":"p","test":"TestX","output":"x_test.go:1: logged\nx_test.go:2: the old kept lines"}`
var f Failure
if err := json.Unmarshal([]byte(old), &f); err != nil || f.Cause != "" {
t.Fatalf("1901/AC3: err = %v, cause = %q", err, f.Cause)
}
v := levelFailure(&target{}, "unit", Tested{Failed: []Failure{f}})
if v.Failed.Words != "x_test.go:1: logged\nx_test.go:2: the old kept lines" {
t.Errorf("1901/AC3: words = %q", v.Failed.Words)
}
if got := firstFailed(*v); !strings.Contains(got, "logged") || strings.Contains(got, "\n") {
t.Errorf("1901/AC3: status = %q", got)
}
}
@@ -4,6 +4,7 @@ import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -46,11 +47,11 @@ func TestTheVerdictsHeadsAreTheGatesHashOverLowercaseHeads(t *testing.T) { // ga
func TestAGroupsVerdictHeadsCoverEveryMember(t *testing.T) { // gate (b): hq's member too, as the gate's ask covers it
ctx := context.Background()
r := newRig(t, passJudge())
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2), pullOf(HQRepository, 3, head2))
cand := candidateOf(t, "feat/x", pullOf(repoA, 7, head1), pullOf(repoSDK, 12, head2), pullOf(hqRepo, 3, head2))
r.ch.Candidate(ctx, "feat%2Fx", cand)
r.ch.Wait()
want, _ := delivery.HeadsHash([]delivery.Head{{Repository: repoA, Number: 7, Head: head1}, {Repository: repoSDK, Number: 12, Head: head2},
{Repository: HQRepository, Number: 3, Head: head2}})
{Repository: hqRepo, Number: 3, Head: head2}})
if v := r.verdict(t, keyOf(t, "feat/x", cand)); v.Heads != want {
t.Errorf("heads = %q, want %q", v.Heads, want)
}
@@ -71,11 +72,12 @@ func TestATrunkMoveMarksEveryOpenPullRequestIntoItPendingAtOnce(t *testing.T) {
t.Fatalf("%s#%d is %q before the move", p.repo, p.n, s)
}
}
// A delivery builds, so no check starts; mesh-delivery's trunk moves.
r.ch.Hold = func() bool { return true }
// mesh-delivery's trunk moves; the re-judges it asks are held in their judge (in the stream, since R4g) meanwhile.
release := make(chan struct{})
r.ch.Judge = &judgeFunc{f: func(context.Context, *Case, int) (Outcome, error) { <-release; return Outcome{Verdict: passing()}, nil }}
defer func() { close(release); r.ch.Wait() }()
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS})
r.ch.TrunkMoved(ctx)
r.ch.Wait()
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("mesh-delivery#7, into the moved trunk, is %q, want pending", s)
}
@@ -90,17 +92,25 @@ func TestATrunkMoveMarksEveryOpenPullRequestIntoItPendingAtOnce(t *testing.T) {
func TestACheckThatEndsAfterItsTrunkMovedOwesPendingNotItsVerdict(t *testing.T) { // gate (a): a pass on an old trunk is never shown
ctx := context.Background()
var r *rig
release, judged := make(chan struct{}), make(chan struct{})
j := &judgeFunc{f: func(ctx context.Context, c *Case, call int) (Outcome, error) {
if call == 1 {
r.setTrunks(t, map[string]string{repoA: trunkA2, repoSDK: trunkS}) // the trunk moves while it runs
r.ch.Hold = func() bool { return true } // and a delivery holds the next check
defer close(judged)
} else {
<-release // the check asked again for the new trunk waits in the stream meanwhile
}
return Outcome{Verdict: passing()}, nil
}}
r = newRig(t, j)
r.ch.Candidate(ctx, "feat%2Fx", candidateOf(t, "feat/x", pullOf(repoA, 7, head1)))
r.ch.Wait()
<-judged
for deadline := time.Now().Add(2 * time.Second); owedState(t, r, repoA, 7, head1) == "" && time.Now().Before(deadline); time.Sleep(time.Millisecond) {
}
time.Sleep(20 * time.Millisecond)
if s := owedState(t, r, repoA, 7, head1); s != "pending" {
t.Errorf("the status is %q, want pending: the pass was judged on a trunk that moved", s)
}
close(release)
r.ch.Wait()
}
@@ -1,18 +1,14 @@
package check
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"github.com/nats-io/nats.go"
)
// Unheard is the error of a call that reached nobody: the bus had no responder for it, or none answered within the call's
// bound. The step it asked for never began, so the same step may be asked of another node (issue 1820). The program wraps
// the errors of its calls in it where the bus gives a timeout; a no-responders error is known as it is.
// bound. The program wraps the errors of its forge calls in it where the bus gives a timeout; a no-responders error is
// known as it is. A build step is no call since R4g: an ask whose node stops is delivered again by its stream (ADR 0525 §6).
type Unheard struct{ Err error }
func (u *Unheard) Error() string { return u.Err.Error() }
@@ -23,79 +19,3 @@ func isUnheard(err error) bool {
var u *Unheard
return errors.As(err, &u) || errors.Is(err, nats.ErrNoResponders)
}
// deafNode is a node that did not answer a build module's call during this judging.
type deafNode struct{ node, why string }
// markDeaf takes a node out of the choice for a build module for the rest of the judging.
func (st *state) markDeaf(bm, node string, err error) {
if st.deaf == nil {
st.deaf = map[string][]deafNode{}
}
st.deaf[bm] = append(st.deaf[bm], deafNode{node, err.Error()})
}
func (st *state) isDeaf(bm, node string) bool {
for _, d := range st.deaf[bm] {
if d.node == node {
return true
}
}
return false
}
// triedWords names every node that did not answer a build module, with what the bus said.
func (st *state) triedWords(bm string) string {
var parts []string
for _, d := range st.deaf[bm] {
parts = append(parts, fmt.Sprintf("%s (%s)", d.node, d.why))
}
return strings.Join(parts, "; ")
}
// moveStep is called when the target's node did not answer: the node is left out of the choice, and another build node of the
// same build module takes the target's place. It answers an error naming every node tried when none is left (the words hold
// NodesWords, so the verdict is judged again when the nodes change).
func (st *state) moveStep(t *target, err error) error {
old := t.node
st.markDeaf(t.module, old, err)
node, _, cerr := st.choose(t.module)
if cerr != nil {
return fmt.Errorf("%s; tried: %s", cerr, st.triedWords(t.module))
}
st.j.logf("%s#%d: %s did not answer (%v), the step goes to %s", t.pull.Repository, t.pull.Number, old, err, node)
t.node = node
other := false
for _, o := range st.targets {
if o != t && o.node == old {
other = true
}
}
if !other {
delete(st.nodes, old)
}
st.nodes[node] = true
return nil
}
// refetch makes the target's workspace again on its new node: the workspace of a step that was lost is on the node that did
// not answer.
func (st *state) refetch(ctx context.Context, t *target) error {
raw, err := st.step(ctx, t, "fetch", fmt.Sprintf("fetch %s#%d on %s", t.pull.Repository, t.pull.Number, t.node), t.fetchArgs)
if err != nil {
return err
}
var f Fetched
if err := json.Unmarshal(raw, &f); err != nil || f.Workspace == "" {
return fmt.Errorf("the fetch of %s#%d on %s answered no workspace: %s", t.pull.Repository, t.pull.Number, t.node, raw)
}
t.fetched = f
return nil
}
// stepOnWorkspace sets the workspace a step reads to the target's current one.
func stepOnWorkspace(t *target, args map[string]any) {
if _, ok := args["workspace"]; ok {
args["workspace"] = t.fetched.Workspace
}
}

Some files were not shown because too many files have changed in this diff Show More