Compare commits

..
907 Commits
Author SHA1 Message Date
jschoubben 945906ac97 The assigner waits for a requirement nothing offers, and for a provision the provisioner keys whoever is placed yet; a requirement something offers (a module's own name, a module the mesh provides itself) is left to Awaited as before (ADR 0603 §5)
mesh/delivery ready: it is merged on the Approve
mesh/check pass 945906ac976c on c4470f42dba9 | unit 1186 run, 5555 kept, 27 skipped | integration 1205 run, 5717 kept, 35 skipped | over bound: unit (…
mesh/delivery-group group feat/l3b-bus ready: every member ready, and composed together they pass
2026-11-03 19:45:15 +01:00
jschoubben 1c66d58bce The assigner waits for a provision whose provider is not known yet: an unknown provider has provided nothing, so a consumer never moves ahead of a provider the indexer has not taken in (ADR 0603 §5); the integration test's write order is unordered again 2026-11-03 19:36:18 +01:00
jschoubben 34ef185ba5 The assigner's integration test puts the provider's module before the consumer's, so the consumer's requirement always has a provider to be judged against (the group check's flaky failure, 5 of 40 runs) 2026-11-03 19:33:43 +01:00
jschoubben f12823681c Merge the indexer's integration test: a field its judge does not know waits while the running build is unknown (ADR 0601 §3; the group check's failure)
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestWantsAreWrittenAndNoDifferenceIs:… | f12823681c06 on c4470f42dba9
mesh/delivery-group group feat/l3b-bus rejected: composed together: the checker's verdict at the members' heads (62ca67fc6b58): integration: mesh-delivery-assi…
mesh/delivery closed: a newer head of the same pull request
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:51:14 +01:00
jschoubben b96a4f1d99 Pin mesh-sdk b7abe203a929, the group's head of feat/l3b-bus: it keeps, deprecated, the exported names the running parts still use
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:50:12 +01:00
jschoubben 70ac4ff79a The indexer's integration test expects a field from later to wait, not be refused, while no engine says which build runs (ADR 0601 §3)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 18:41:10 +01:00
jschoubben 51deed51e3 Merge ADR 0603 §10: the check waits on the builder's builds of its candidate before it composes, asks itself again when a failed build is done again, and names what it composes from manifests (the operator, 2026-11-03: build §10 in L3)
mesh/check checking 51deed51e3b4 on c4470f42dba9 | novox/mesh-delivery#147 in group feat/l3b-bus
mesh/delivery-group group feat/l3b-bus checking: 4 of 4 member(s) ready
mesh/delivery closed: a newer head of the same pull request
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:56:03 +01:00
jschoubben 3842a464ed A check failed on a build asks itself again once the build is done again, and a failed check never waits on builds (ADR 0603 §10)
A failed candidate build no longer needs mesh-delivery.recheck, an exception verb (ADR 0421 §12): the checker hears the
builder's keys the failed verdict names and, once each is written done (builder_build_again), retakes the check of the
same heads as it does after a failure of the nodes, at its start too. When the tests have already failed the check
composes at once and names the builds it did not wait on, so its red status is never held back by a build.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:55:41 +01:00
jschoubben 877b09e92b checker_status names the builds a running check waits on (ADR 0603 §10)
Worked out from the builder's builds state at each read, stored nowhere: of each running check, the candidate's builds
that are neither done nor failed.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:51:09 +01:00
jschoubben 453cdfa607 The check waits on the builder's builds of its candidate before it composes, so it composes with them (ADR 0603 §10)
The builder already builds every candidate fingerprint into its own key. The check now watches exactly those keys on the
bus and composes once each is done or failed, within the new setting build-wait-bound (1800 s). A failed build fails the
check, naming it; a build not ended within the bound is named and the check composes without it. A module built by a
build module the change alters (mesh-build-go or mesh-build-node), one standing on a base the change alters, and one the
trunk holds no record of are not waited on, and the composition's warnings name each with why.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:51:09 +01:00
jschoubben 8f7945a109 Merge round 2 finding C and the check's true words: the new rows fold into existing tests, the check names what it composes from manifests (ADR 0603 §10; test budget)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:36:49 +01:00
jschoubben 2773885600 Merge round 2 findings A and B: only the newest manifest's provisions hold a want, and a wait keeps its first start while it stands (ADR 0603 §8, §9)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:36:49 +01:00
jschoubben 45bd63427f A wait the watchman mirrors keeps its earliest start while the same mark stands, so a key written again with new words does not start the bound again (ADR 0603 §8) 2026-11-03 17:26:45 +01:00
jschoubben effa12c4b4 Fold the three new tests into existing ones (test budget); the base note says what the check does
The check builds no artifacts and no changed base before its dependents; build_change.go now says only that they are composed from their manifests.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:26:31 +01:00
jschoubben 9da14440e8 The assigner waits only on the keys of provisions the newest manifest still requires: an older edge's asked or retiring key holds nothing (ADR 0603 §9) 2026-11-03 17:26:30 +01:00
jschoubben fd638b99d5 Merge D9, D13b, D15 and review round 1 findings 3 and 4: the indexer runs without its graph state, a package inside a module's directory has its publisher (ADR 0603 §3, §4, §10; ADR 0604 §1)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:28 +01:00
jschoubben 0d1edef846 Merge D11, D12, D14: delivered over the nodes present, the watchman's waits with a bound, the newest provider edge (ADR 0603 §7, §8, §9; L3 review round 1, finding 2)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:28 +01:00
jschoubben 5c4d770aa1 The check names the modules built with a changed build module or base; rows for D9 and the assigner's integration test (ADR 0603 §3, §4, §10)
D13b: the composition's warnings, and so the check's line, name the changed modules built with the Go build module the change alters and those standing on a changed base. D9: builder and verdict rows. D15: the assigner's integration test gets a source revision changing a provider, its new consumer and a build module.

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:07 +01:00
jschoubben b4855f21c3 Test rows: a missing graph state fails no recompute (review finding 3); a recipe change builds a Go module again (D9)
Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:07 +01:00
jschoubben 1387b10e78 Take a missing graph state as no graph yet and a Publisher's directory as the nearest module above it
The indexer's recompute and merge failed on listing a graph bucket not yet on the bus, and a package published from inside a module's directory had no publisher, so the dependents of a build module missed what imports its packages (review findings 3 and 4 of L3).

Claude-Session: https://claude.ai/code/session_01JG73KG7AfSdh6FP8VrBqCn
2026-11-03 17:21:01 +01:00
jschoubben a1814c6571 Assigner row: the newest source revision's provider edge replaces the older one, and only it is waited on (ADR 0603 §9) 2026-11-03 17:19:05 +01:00
jschoubben 92b4307922 The watchman mirrors a want held by a rule, a provision asked and a build queued behind a package as waits with a bound, and raises a finding past it (ADR 0603 §8) 2026-11-03 17:18:36 +01:00
jschoubben 2c37f48f9d A source revision is delivered over the nodes present: the view leaves out a node the watchman says is away and names it, and the node counts once it runs its wants (ADR 0603 §7) 2026-11-03 17:16:30 +01:00
jschoubben 4a736858b6 Pin mesh-sdk cf5812101b0b, the group's head of feat/l3b-bus with the module graph taken out 2026-11-03 16:56:37 +01:00
jschoubben 3411b91571 The module graph is mesh-delivery's own: a Go module sdk/ with package graph, used by path by the indexer, the view and the checker, and the seat mesh-catalog's answers built from mesh-sdk's contract types (the operator, 2026-11-03: no graph-related code in mesh-sdk, only contracts; research 081 D11, D13) 2026-11-03 16:56:37 +01:00
jschoubben 0ce16ad773 Pin mesh-sdk 18b3eac47c8e, the group's head of feat/l3b-bus rebased onto main 2341e8e9 2026-11-03 16:46:40 +01:00
jschoubben 875d42fe3e On B1: the ledger test names its tool runner by its manifest's tool-runner, and ShelfWaiting takes B1's old-section counts (rebase onto main c4470f42) 2026-11-03 16:46:40 +01:00
jschoubben aacdba44e3 The judge's copy of the mesh-catalog seat is the SDK's entry: four optional verbs, so the old module's claim is not outgrown (ADR 0604 §1; F3) 2026-11-03 16:45:06 +01:00
jschoubben 39b3ce3c73 builder_builds_since names the operator's command nox builds as its reader, not mesh-catalog (ADR 0604) 2026-11-03 16:45:06 +01:00
jschoubben e1d0c9fe03 A merge that leaves a repository no module never deletes a module another trunk holds, and indexer_take_in takes a repository in (issue 1367)
dropRemoved hands the module over from the other trunk, read from the forge at the head the source names, so no wait in memory is needed. indexer_take_in with a repository takes in a head with no module.json as a merge that leaves none, and answers by name for a head with modules.
2026-11-03 16:45:06 +01:00
jschoubben 6bc16d74db Write and take in a merge that leaves no module.json, so a module-less repository's reach is not held for ever and a removed last module is dropped (issue 1126)
The forge bridge wrote no merged message for such a merge, so taken.<repo> never moved
and the view held 'the indexer has not yet taken in <head>' (hq #490-#492). The bridge
now writes it with no trees; the indexer drops every module of the repository on it,
after the same order check as any merged message. The view needed no change: with the
taken key moving it reaches the merge, which its new row proves.

Needs mesh-sdk fix/1126-no-module-reach (a merged message may carry no trees).
2026-11-03 16:45:06 +01:00
jschoubben 9f49bc92ad The seat mesh-catalog's four verbs answered from the indexer's modules, manifests and graph, with the old catalog's field names (captured answers as fixtures); served once the indexer claims the seat, in the release after L3's (ADR 0604 §1, §2) 2026-11-03 16:45:06 +01:00
jschoubben 4f6e1021c6 view_tiers: a source revision's tiers worked out from the module graph at each call, and per module what it waits on now, its build, each node's want and have and each provision it awaits, the nodes offline named away (ADR 0603 §6) 2026-11-03 16:45:06 +01:00
jschoubben 027c445bae Pin mesh-sdk 182d8757dba9, the group's head of feat/l3b-bus with the graph's ordering edges (ADR 0603 §1, §6) 2026-11-03 16:45:06 +01:00
jschoubben 7ba2668aa8 The check refuses a change whose module graph has a cycle and names it, worked out from the catalogue as the change leaves it; a cycle the trunk already has among the same modules is said, not the change's failure (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben a901b840a0 Pin mesh-sdk acad60547c84, and the indexer's graph nodes take their bases and provision edges from the contract's GraphEdgesOf (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben f70faed5b6 The indexer keeps the module graph as its state graph: each module's build and provision edges written at every recompute when they change, at most 2000 modules, a module's key deleted once the modules state no longer holds it (ADR 0603 §2) 2026-11-03 16:45:06 +01:00
jschoubben 5ee3b4c15f Pin mesh-sdk 25b371bc8397, the group's head of feat/l3b-bus with the module graph and the seat mesh-catalog's contract (ADR 0603 §2, ADR 0604 §1) 2026-11-03 16:44:47 +01:00
jschoubben 2f387a3f4c Pin mesh-sdk e7ce86ff62b5, the group's head of feat/l3b-bus (review round 2) 2026-11-03 16:44:47 +01:00
jschoubben 22aedbe97f A recompute with no merge writes the fingerprint's source revision as not known, so no release's Approve answers a build it never carried, though the source revision is still that release's (ADR 0601 §1; review round 2) 2026-11-03 16:44:47 +01:00
jschoubben 478f0110e9 Pin mesh-sdk 825483043549, the group's head of feat/l3b-bus (review round 1) 2026-11-03 16:44:47 +01:00
jschoubben 6d24922172 The node-engine's witness ping is granted by the tool runner's name its manifest gives, any service's instance on its own node while none says so, never a name in mesh-grants (ADR 0601 §4; review round 1, finding 4) 2026-11-03 16:44:47 +01:00
jschoubben 0fa00d5e6c The watchman raises what a node keeps of a module as it runs, for a manifest that does not yet say it, from the module's have, and clears it once the have says none (ADR 0601 §4; review round 1, finding 3) 2026-11-03 16:44:47 +01:00
jschoubben a41d2851c7 A release answers only the build whose fingerprint it moved: the indexer stamps the source revision at which a fingerprint last changed, a recompute with no merge the revision of its moment, and the assigner reads that stamp, never the Src a recompute keeps (ADR 0601 §1; review round 1, finding 1) 2026-11-03 16:44:47 +01:00
jschoubben ea5be10532 Pin mesh-sdk 09ec82cb3551, the group's head of feat/l3b-bus (ADR 0601) 2026-11-03 16:44:47 +01:00
jschoubben 6b9d211f72 The checker decides a node the sandbox did not judge by the contract's field (composer.Machine.NotJudged, verdict not-judged) and names it with the composer's words, never by a phrase in its reason (ADR 0593 §9; the prepare's seams finding on "not judged") 2026-11-03 16:44:47 +01:00
jschoubben b342c67b19 The grants part knows the tool runner by what its manifest says, never by its name, and keeps a node's tool runner user as the list in force holds it while no manifest there says it yet, said in the held condition (ADR 0601 §4; the prepare's seams finding on mesh-grants's own name) 2026-11-03 16:44:47 +01:00
jschoubben daf178c4ca A release's Approve answers the record ask of a move of the build alone to the build it carries: the assigner reads the release by the build's source revision from the gate's source and merges, and names it in the want; a move that changes a setting or a secret too is asked as before (ADR 0601 §1) 2026-11-03 16:44:37 +01:00
jschoubben a9aa6d29ea The assigner moves the bus module by its rules as any module, with no bar and no snapshot of its own, and a person may choose its policy as any module's: its node backs its streams up before it applies (ADR 0601 §1, §2) 2026-11-03 16:44:37 +01:00
jschoubben ecf96be8ce The judge's row for a field it does not know names a field mesh-sdk does not know now: core-credentials is one of its manifest words (ADR 0601 §4) 2026-11-03 16:44:37 +01:00
jschoubben ab3cb1ca9e The judge knows the tool runner by what its manifest says and keeps the kept streams itself, and this repository's check that no manifest names the ended controller reads them as text: mesh-sdk names no module (ADR 0601 §4) 2026-11-03 16:44:37 +01:00
jschoubben 054f089614 The bus has the upgrade policy its manifest says, as any module: the judge and the assigner rows read it from the manifest (ADR 0601 §1) 2026-11-03 16:44:37 +01:00
jschoubben 6301e49c73 Review round 2: a held module's object reads come from its running manifest; one with none keeps the list in force, said; indexer_judge asks the engine for the running build again (issue 774) 2026-11-03 16:44:37 +01:00
jschoubben 78fe1ce960 Review round 1: the indexer learns its build from the node-engine's seat; a held module's seats and hearings come from its running manifest; indexer_judge; one judge pass at a time (issue 774) 2026-11-03 16:44:37 +01:00
jschoubben 6645ac58c2 A manifest with a field the judge does not know waits instead of being refused; a held module keeps its bus users; tools list and redo the verdicts (ADR 0578 amended, issue 774) 2026-11-03 16:44:19 +01:00
mesh-tools c4470f42db Merge pull request 'Bus state by template: the judge's rules, manifest-accepted, the old-section count and the ledger account (issue 1966, B1)' (#146) from feat/bus-state into main 2026-11-03 15:24:36 +00:00
jschoubben 4bb790bedf Pin the group's mesh-sdk head f8bd12e in place of the local replace (issue 1966)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c4470f42dba9
mesh/delivery-group group feat/bus-state merged: every member is merged
mesh/check pass 4bb790bedffe on 8f83608503da | unit 6460 run, 262 kept, 27 skipped | integration 6639 run, 264 kept, 35 skipped | over bound: unit, in…
2026-11-03 15:35:28 +01:00
jschoubben aa98729457 The zero rule takes the count of the accepted set the last pass held, and is not applied when it cannot be read (issue 1966 D4) 2026-11-03 15:28:58 +01:00
jschoubben 2fdb0516cd The checker's judge counts the old sections from the trunk case as the indexer does, so a change adding a section at 0 is refused at the merge check (issue 1966 D4) 2026-11-03 15:14:28 +01:00
jschoubben 28a6858863 The ledger account's subjects come from mesh-sdk's one naming of them (issue 1966 D5) 2026-11-03 15:13:38 +01:00
jschoubben 1ab0274547 The wanted accounts include the ledger kind, held in the tool runner's ledgers, in step with mesh-sdk's buscompose (issue 1966 D5b) 2026-11-03 13:39:21 +01:00
jschoubben 07af86b153 The grants part composes a ledger account per assignment of a module asking for a ledger, publishing only on its ledger histories and snapshots (issue 1966 D5) 2026-11-03 13:38:20 +01:00
jschoubben dd2e7c8e64 The indexer counts the modules on each old section, a tool reads it, and the judge refuses a section at 0 outside the foundation (issue 1966 D4) 2026-11-03 13:34:54 +01:00
jschoubben 756a3813e2 The indexer says manifest-accepted, with the bus-state entries whole, for a newly accepted verdict (issue 1966 D3) 2026-11-03 13:32:41 +01:00
jschoubben 9a508701ff One local name is one form, and the old state says what it accepts: 0 is taken as 1 (issue 1966 D2) 2026-11-03 13:31:42 +01:00
jschoubben 7ee7248fad The judge refuses bus state in requires that breaks a template's bounds, naming the entry and the field (issue 1966 D1) 2026-11-03 13:30:42 +01:00
mesh-tools 8f83608503 Merge pull request 'The checker waits its compose answer bound, not the SDK's 30 s, for the machines' compositions' (#144) from fix/1126-compose-answer-bound into main 2026-11-03 00:40:26 +00:00
jschoubben bedda86a22 Fold the call-bound check into the answer-bound settings test (test budget)
mesh/delivery merged: source.main moved to novox/mesh-delivery@8f83608503da
mesh/check being carried to bedda86a22c5 on 8f83608503da | from a62276bcafad
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-03 01:26:26 +01:00
jschoubben 0e26fd8ed5 The checker's calls to node-engines wait the compose-answer-bound, not the SDK's 30 s (issue 1126)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-03 01:13:36 +01:00
mesh-tools a62276bcaf Merge pull request 'Tests no longer race the wall clock; the checker invokes compose-candidate again' (#143) from fix/timing-tests into main 2026-11-02 22:17:53 +00:00
jschoubben 34a607a1f5 Give the checker back its compose-candidate invoke, now that the new indexer runs
mesh/delivery merged: source.main moved to novox/mesh-delivery@a62276bcafad
mesh/check being carried to 34a607a1f5cf on a62276bcafad | from 15d28ddf5bd3
The invoke was left out (mesh-delivery #142) only while the indexer built before the verb ran; the new indexer and the
node-engine's compose-candidate run on every machine since 22:57, so a check composes on the machines again instead of
passing with a warning (ADR 0593 §7, §9).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 23:04:44 +01:00
jschoubben f73b0bf6f6 Tests wait on events or the clock they are given, never on milliseconds a loaded machine outlasts
The check of 2026-11-02 22:35 on a loaded shanks failed two buildrun tests on wall-clock bounds of 60 ms, the
second failure of this kind tonight (the first: mesh-sdk's busretry tests, f02f142).

Quick fix: give the two buildrun asker tests bounds of seconds.
Thorough fix (chosen, because the kind recurs across every module of the repository): every test that raced a
wall clock now waits on an event, steps a clock it is given, or has a bound only a failing test reaches.
- buildrun asker tests: the steps' own times stand for the clock (bound an hour, starts and phases written in the
  past), no goroutine paces phases with sleeps.
- builder: the decline bound reads Builder.Now; the decline test steps it; the stream-wait test waits for five
  check reads instead of 100 ms.
- checker: the lost-ask try bounds an untaken ask at 200 ms only where that is what ends it; the program says when a
  merge turn changes, and the integration test waits on that line instead of 300 ms.
- teller: the bed makes the teller's own ends bucket (a 10 s stall at start the tests raced by 0.5 s), waits for its
  warrant subscription instead of 500 ms pauses, and the package-level now is restored with t.Cleanup.
- watchman: the recount, run in its own goroutine, is waited for instead of 10 ms.
- polls of 2-5 s become 10 s (unit) and 20 s (integration).

Under a throttled cgroup (half a core shared with two busy loops) beside the node-engine's suite, the buildrun
tests failed 5 of 20 runs before and passed 30 of 30 after; no test added.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 23:04:44 +01:00
mesh-tools 15d28ddf5b Merge pull request 'mesh-delivery-checker: leave compose-candidate out of its invokes until the new indexer runs' (#142) from fix/checker-invoke-later into main 2026-11-02 21:55:29 +00:00
jschoubben 22f3658a36 Leave compose-candidate out of the checker's invokes until the new indexer runs
mesh/delivery merged: source.main moved to novox/mesh-delivery@15d28ddf5bd3
mesh/check checking 22f3658a36a6 on 15d28ddf5bd3
The running indexer, built before the verb existed, refuses a manifest invoking it, and one refused manifest holds the
grants part's whole list: no new grant is written, so the new indexer never moves (issue 1126, 2026-11-02). Quick fix,
chosen because the thorough one (the grants part keeps only the refused module's grants) cannot be delivered while the
list is held; a follow-up adds the invoke back. Until then a check's compose-candidate call is refused on the bus and the
node is named silent: the check passes with that warning (engines.go, ADR 0593 §9).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 22:22:23 +01:00
mesh-tools 073e3fab65 Merge pull request 'sdk1: the indexer fingerprints by imported content; the checker asks the node-engines to compose' (#141) from feat/sdk1 into main 2026-11-02 20:42:16 +00:00
jschoubben 2b5a7fe831 Pack the checker's settings template as its files archive, or the indexer's judge refuses the manifest
mesh/delivery merged: source.main moved to novox/mesh-delivery@073e3fab65db
mesh/delivery-group group feat/sdk1 merged: every member is merged
mesh/check checking 2b5a7fe831c1 on 511e4b741497 | novox/mesh-delivery#141 in group feat/sdk1
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 21:19:38 +01:00
jschoubben 3b3b096be8 Place mesh-networking beside a node-engine check, so route-proxy's proofs of a composition are judged once this release runs
mesh/delivery-group group feat/sdk1 rejected: composed together: the checker's verdict at the members' heads (5fc952f974ca): mesh-build-go
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: mesh-build-go: the indexer's judge at the group's mesh-delivery head 3b3b096be8cdf8f49246ed3961ece3eb5c349d62 refuses the manifest …
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:40:01 +01:00
jschoubben 6313f3ea10 Place the node-engine beside a catalogue check, so mesh-build-go's composer test is judged once this release runs
The composer command moved to the node-engine (ADR 0593 §7); a catalogue check had only mesh-sdk beside it. Pin the build runner to the catalogue head whose composer test skips aloud until then.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:36:19 +01:00
jschoubben 712d136fe6 Pin the build runner to the release's new mesh-catalog head
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/sdk1 checking: 6 of 7 member(s) ready
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:23:41 +01:00
jschoubben 09ac178888 Pin mesh-sdk at f02f142, the sdk1 head whose tests no longer leak package globals, so the group's check builds against it
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 20:14:55 +01:00
jschoubben d4600efd7d Pin the build runner to the release's mesh-catalog head, computed by go
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/sdk1 checking: 6 of 7 member(s) ready
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: NOT CHECKED: the node-engine's composer c…
The placeholder was written in local time and go refused it as a pseudo-version.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:41:56 +01:00
jschoubben 0edd0bca74 mesh-delivery-builder: fold the step-choice and step-run tests into two tables, within the test budget
Review finding 5: the repository's unit tests rose from 810 to 814 (over the 500 budget); eleven test
functions of buildrun become two, so the count is 805.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:38:09 +01:00
jschoubben 2fb1dc3214 mesh-delivery-checker: a placement that cannot be read leaves the composition not made, said
Review finding 4: an unreadable placement gave no nodes, so nobody was asked and the check passed. The
placement read now answers its error, and the verdict says the composition was not made and why.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:37:22 +01:00
jschoubben e5ce35cd00 mesh-delivery-checker: a composer at the node-engine's head that cannot run fails the composition
Review finding 3 (ADR 0593 §9): only a node that does not answer may let a check pass with a warning; a
sandbox composer that cannot be built or run was filed with the silent nodes. It now fails that node's
composition, naming the head.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:36:42 +01:00
jschoubben e31da597bb mesh-delivery-indexer: the version line takes a directory-replaced sibling by its content, so it never goes stale
Review finding 2 (ADR 0593 §2, §3): after the release no build has reported imports, and a module that
replaces a mesh Go module of its own repository by a directory had no go.sum line and no version for it,
so a change of buildstep or buildrun alone moved no importer. The indexer now reads go.mod's directory
replaces, publishes those modules, and puts their content digest on the version line; builds are pinned
to the other repositories' versions alone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:35:58 +01:00
jschoubben 33675f548a Pin mesh-sdk at the sdk1 group's head 1c7356c, so the group's check builds every module against the narrowed SDK it was written for
The build runner's pin stays at its placeholder until mesh-catalog's final head is pushed.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:33:10 +01:00
jschoubben 008a85b73c mesh-delivery-indexer: the judge's tests ask nothing of the composition's offers, overflow or left-out reasons
ADR 0593 §7: OffersIssuedOutside, Resolution and UnknownFieldReason left mesh-sdk with the composition,
so the judge's tests keep its own reading and refusals and those proofs go to the node-engine's tests.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:23:17 +01:00
jschoubben 6fb743f521 mesh-delivery-indexer: the judge's tests judge manifests and compose nothing
ADR 0593 §7: the catalogue's composition (Resolution.Compose and Declaration, LeftOut, Resolve,
JudgeSettings, ApplySettings, HeldTemplateProblem) leaves mesh-sdk for the node-engine, so the judge's
tests keep only its verdicts and refusals; each proof that a manifest composes goes to the node-engine's
tests, and the helpers only those proofs used go with them.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:08:41 +01:00
jschoubben c0209b3d8e Builder's fetch test takes the named tree as data, not computed
The named tree is the indexer's computation (ADR 0593 §2): the builder reads it
from the module's value, so its test gives one as the indexer would write it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:00:57 +01:00
jschoubben c12804f4d9 mesh-delivery-indexer: its integration test no longer reads verdicts through mesh-sdk's composition reader
ADR 0593 §7: compose and frombus left mesh-sdk for the node-engine, so how a composer holds a refused
module is the engine's to test; the indexer's test keeps proving the verdicts and refusals it writes.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 19:00:16 +01:00
jschoubben fbb50f8fcd mesh-delivery-checker: run check asks in its tests with the Go build module's real runner
ADR 0593 §1: the step runner left mesh-sdk for mesh-build-go's buildstep package, so the examiner's unit
tests and the module's integration test require it by the group's version (as the builder does) rather
than an imitation of its plan and kept-verdict rules.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:58:18 +01:00
jschoubben 0ce3409141 mesh-delivery-checker: make a check's composition from every placed node's engine answer
ADR 0593 §7, §9: the checker asks compose-candidate of each placed node in parallel within its own setting
compose-answer-bound (60 s); a refusal fails the check naming node and module, a silent, unready or
unreachable node passes with "not composed on that node: no answer". A group with a node-engine member is
composed by the composer at that head on the Go build seat over each node's input. The wait on the
assigner's compositions goes, and the checker invokes the one engine verb it needs.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 46e53a81c2 mesh-delivery-checker: build mesh packages at the versions the indexer publishes instead of computing them
ADR 0593 §2: only the indexer computes a package's version; the checker follows its packages state and
hands those commits to a Go check, so LatestPackages leaves repoModules and mesh-sdk's
PackageVersionsOf has no reader here.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 7cf47c04eb mesh-delivery-assigner: compose nothing; judge proposals with its own judge and compare the published compositions
ADR 0593 §7, §8: the check's composition, its compose asks to the Go build seat and the state
compositions go (the checker asks the engines); a proposal is judged without frombus; mesh-wants.upgrade
compare reads every node's composed-<node> through mesh-host.composed. The assigner no longer imports
compose, composer, frombus or buildrun.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben 8855735341 mesh-delivery-assigner: the placement judge is the assigner's own, judging a proposal from placement alone
ADR 0593 §1, §8: compose/judge moves back from mesh-sdk; every node answers a proposal, what a placement
changes is read from its entries, and a node silent past the answer bound (60 s) is named and the
proposal decided on the answers it has.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:20 +01:00
jschoubben c62a501bf0 mesh-grants: a node reads the indexer's candidates and verdicts whole, as its engine reads a key through a short consumer
ADR 0593 §7: compose-candidate's read of one key times out without the consumer grants, so both buckets
are granted as the builder's builds are; the node grant now equals mesh-host's golden node.json.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 0b03b9c7c5 mesh-grants: create a module's object buckets and history streams, grant their readers read-only, and warn at 75 percent
ADR 0592 §5, §6, §9, §25 (the grants part's share by ADR 0593 §10): an object bucket and a stream that
keeps its history are created create-only with their caps and no age limit, refusing new data when full;
the owner alone writes its bucket and a module its grant names reads it; the part raises
mesh-grants.history-full once any passes 75 percent. Each engine also reads the indexer's manifest
verdicts by one key for compose-candidate.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben d283efdc9f mesh-grants: the machine's runtime hears a node's compositions bucket made later for the bundles it carries
ADR 0593 §6: a bundle such as the provisioner reads mesh-host.composed on the runtime's connection, so the
runtime account gets the stream listing and the creation advisory beside the read of each bucket.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 74304052b4 mesh-grants: every node's engine writes its own compositions and the named readers read them all
ADR 0593 §6, §7: the grants part creates composed-<node> per node, grants each engine the writes of its
own bucket and a direct get of the indexer's candidates for compose-candidate, and grants
mesh-host.composed read-only, with the stream listing and creation advisory a per-node read needs, to
the provisioner, the assigner and the view alone; memberships issue it per node.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 2a91caddaf mesh-delivery-checker: place no sibling for the controller's repository, which this group empties
ADR 0593 §11: the controller's repository holds nothing that builds once route-proxy moves, so its
sibling rule goes; its tests fold into the sibling test, which now uses mesh-sdk's rule.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 78fedabeb1 mesh-grants: own the bus's composer, so a change to who may say what rebuilds only the grants part
ADR 0593 §1: buscompose's engine moves from mesh-sdk into mesh-grants/internal/buscompose, its golden
user lists unchanged; the controller's raise and assert code is left behind, and the merged switch
test runs inside the part's whole integration test.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 5c20b9ef15 mesh-delivery-checker: the line test holds ADR 0591's case of a first failure longer than the forge's limit
The first failure's words are shortened; "failed (N)" and "and N-1 more" stay whole within the limit.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben fa85de8ef6 mesh-delivery-checker: write a member's preview whenever its fetch merged, and do not wait on a composition that cannot come
Review of 650a1d0: a member that only failed its tests got no preview, so the indexer held no manifest
of it, the assigner's composition only said it waits, and the check waited for ever. The preview is now
written whenever every build module's fetch merged; a member with no preview of a merge (a conflict, a
fetch that did not answer) makes the composition not run, said with the members it names, unawaited.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben 290e1696b2 mesh-delivery-checker: a check runs every member, level and the composition to its end and names every failure
The operator, 2026-11-02: "A failed check shows only its first fault". Group feat/1126-removal's
check failed four times, each verdict naming one fault, so three faults took three runs of 20-40 minutes.

The checker no longer returns at a member's first failure: every member is asked, every level's
failures and every member's refused step or conflict are joined in judging order, the manifests are
judged and the composition is read also when the tests failed, and a failed composition is added after
them. The status line says how many failed and names the first, cut so "; and N more" stays whole.
checker_verdicts and checker_pull answer the whole list (failed). Needs mesh-sdk's buildstep change
(every level runs after a failure) pinned at the group's head.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:56:19 +01:00
jschoubben b7551d2efb Indexer's tests compute with its own fingerprint package
So mesh-sdk can drop SourceFingerprint2, NamedTree and NoPackageVersion (SDK1)
without breaking them.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:41:22 +01:00
jschoubben 2e91f0b061 Fold the release's new tests into existing ones to stay within the test budget
mesh-delivery is over its budget (810 unit, 73 integration on main), so this
branch adds none net: its new cases are subtests, the three work-queue process
tests and the consumes test become one, the history stream bounds join the
stream values table, and the go/-only mesh-sdk merge check joins the one new
integration test. buildrun's 20 unit and 1 integration tests moved in from
mesh-sdk with it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:40:57 +01:00
jschoubben 61b2b718e6 View reads the indexer's published packages with indexer_packages
A module's fingerprint takes the digests of the packages its last build
imported, so the published packages say why a merge moved a module or not;
the state needs a read tool like every other (ADR 0593 §2; D2).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:38:58 +01:00
jschoubben 2e23b0b736 Prove against the bus that a mesh-sdk merge moves only importers of a changed package
The indexer whole: two catalogue modules report their imports at their builds'
end, mesh-sdk merges a change of buildstep alone, the packages key names the
merge with a digest per package, the buildstep importer moves and the other
keeps its fingerprint though the version line would have moved it (ADR 0593
§2-§4; D11).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:37:37 +01:00
jschoubben b81e05387b Judge says its rules are its own
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:35:17 +01:00
jschoubben 525d7374b2 Judge object buckets and history streams as ADR 0592 decides
The indexer's judge takes a module's own object bucket (a plain local name,
once, no name its states take, none of the mesh's derived prefixes, 1 to 8192
MiB with no default, a grant only by a trusted value) and a stream that keeps
its history (no max age, deliveries or duplicates), counts such a stream among
the bus's predefined forms, gives a bucket's name one owner on the bus, and
refuses a read of a bucket its owner grants nobody. The seat table expects
the node-engine's compose-candidate verb (ADR 0593 §7; D5).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:34:54 +01:00
jschoubben 4c4380b94b Builder's integration test pulls asks with mesh-build-go's runner
The pulling code leaves mesh-sdk for mesh-build-go's own package (ADR 0593 §1),
so the bed imports it by its published version and tests the builder against
the real runner, not an imitation. The required version is a placeholder at
mesh-catalog 9c57f7af: the group pins the catalogue's head before its push.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:33:31 +01:00
jschoubben f6fa75e189 Builder's unit tests run an ask with a local runner, not mesh-sdk's
The build runner leaves mesh-sdk for mesh-build-go (ADR 0593 §1), another
repository the builder does not import; the tests answer asks as the builder
needs them answered.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:31:07 +01:00
jschoubben 604f1528d6 Builder owns the step runner; a build's imports reach its done key and log
buildrun leaves mesh-sdk for the builder's own Go module (ADR 0593 §1); the
checker, assigner, indexer and watchman require it by a replace to the sibling
directory. Issue 1930's bound was already on mesh-sdk's main and moves with it.
The builder copies the imports a build module answers to the done key and the
log's end entry, where the indexer keeps them (§2; D7).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:29:56 +01:00
jschoubben 2aa43ec249 Bridge reads no mesh package from a go.sum
Only the indexer computes what a fingerprint takes (ADR 0593 §2): the merged and
pull messages carry no Packages, and the kept inputs no longer hold them (D6).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:26:53 +01:00
jschoubben fba98ee80f Indexer computes fingerprints alone, over the packages a module imports
The fingerprint, named tree and package versions move from mesh-sdk into the
indexer's internal/fingerprint (ADR 0593 §2). The indexer publishes each reached
mesh Go module's package digests from the forge's listing in its new packages
state, keeps the imports a done build reports, and writes the import form only
when an input differs from the build that reported them (§4), so the form change
costs no rebuild and a package merge moves only its importers (§3). Candidates
take the same form against the group's member heads (D1-D4).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 18:25:57 +01:00
mesh-tools 511e4b7414 Merge pull request 'The grants part and the indexer without the controller; consumers far behind raise a condition (issue 1126 group B)' (#138) from feat/1126-removal into main 2026-11-02 13:41:16 +00:00
mesh-tools 4dd7dcfc14 Merge pull request 'A group check places its own catalogue head beside the other members (issue 1126)' (#139) from fix/1126-group-catalogue-beside into main 2026-11-02 12:33:04 +00:00
jschoubben d26781b403 Place a group's catalogue head beside its other members' tests (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4dd7dcfc1411
mesh/check being carried to d26781b403df on 4dd7dcfc1411 | from be88162cde31
A group's check judges each member against the group's heads (ADR 0482 §2,
ADR 0573 §4), but the checker placed mesh-catalog at its trunk beside every
other member even when the group held a catalogue pull request, so
mesh-delivery's judge tests in group feat/1126-removal judged the new judge
against catalogue main and failed. The catalogue now follows the rule of
every other sibling: the member's head and ref in a group, the trunk alone.
A catalogue change's own trunk copy (ADR 0529 §2) is placed separately and
is unchanged.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 13:12:54 +01:00
jschoubben e2cb398186 Pin mesh-sdk to the group head of issue 1126
mesh/delivery merged: source.main moved to novox/mesh-delivery@511e4b741497
mesh/delivery-group group feat/1126-removal merged: every member is merged
mesh/check checking e2cb3981866e on 4dd7dcfc1411 | novox/mesh-delivery#138 in group feat/1126-removal
Every member builds against mesh-sdk abf5eac, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:27:33 +01:00
jschoubben 6a4d88265d Say mesh-grants.consumer-behind again only when the consumers behind change, not their counts
The counts change every pass, so a consumer that stays behind was said to the operator again at each pass. The
condition names the consumers alone; how far each is stays in the last pass.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:23:23 +01:00
jschoubben 655692c7ba Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk f569924, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:20:14 +01:00
jschoubben 8f9766e46c Let the grants part say an account consumer far behind as mesh-grants.consumer-behind
The self-check's bus-consumer probe judged consumers the controller defined, and with the controller ended it judges none
(issue 1126). The grants part owns the account consumers, so it says when one stays more than 1000 messages behind its
stream's head at two passes running, the probe's bound and confirmation, and clears it once none is.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 12:03:29 +01:00
jschoubben 83c1b3ce38 Pin mesh-sdk to the group head of issue 1126
Every member builds against mesh-sdk 729a86b, the head of feat/1126-removal, so the group is checked as one contract.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:56:55 +01:00
jschoubben 000a24d261 Match the judge's copy of mesh-wants' upgrade to mesh-sdk (issue 1126)
mesh-sdk now says what upgrade's compare answers without the controller:
each node's own composition and the assigner's findings. The judge keeps
a copy of the seat, which must read the same.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:49:17 +01:00
jschoubben 3228d911cb Let a module move between repositories by the merges that move it (issue 1126, route-proxy)
route-proxy leaves the controller's repository, which is archived, and nothing registered a move since the controller
went: the indexer left a module named by another repository's merge as it was, forever. Now that merge waits, and the
holding repository's merge that drops the module hands it over in place, so its key never goes missing between the two;
merged the other way round, the key is deleted and written new. A trunk that still holds the module keeps it (ADR 0420
clause 7, ADR 0501 clause 3).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:48:34 +01:00
jschoubben 7e358b0f1d Recognise the control node by the bus module, not the controller (issue 1126)
The watchman held a node a control node, urgent after half an hour
silent, while its have named mesh-controller; the controller ended by
its unassignment (ADR 0590), so no node was one any more and the bus
node's silence stayed a warning. The control node is now the node whose
have holds the bus module, nats, which the watchman already hears by
that name.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:47:53 +01:00
jschoubben 8d4c13427a Build against mesh-sdk without the controller (issue 1126)
mesh-sdk's group B member renames the grant to Grant, drops the
controller principal, the signer and the lease fields from the user
list key, and the keeper's and config's take-over verbs. The grants
part, the assigner and the view use the new names; the judge's copies of
the health and config seats follow; the tests' fixtures lose the
controller's user, and the refused principal is one removing what the
closed set does not name, the controller's mirror being gone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:46:12 +01:00
jschoubben 2049b1a370 Say the assigner's conditions as findings alone, the keeper being the one source (issue 1126, B8)
ADR 0590 clause 3 ends the choice of a condition source by the keeper's
take-over key, which mesh-sdk no longer defines. The assigner watched
that key and said every condition straight to the router while it named
the controller. It now writes each as a finding the keeper carries, and
says one straight only where it has no findings bucket at all.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:46:12 +01:00
jschoubben 43d4f511c5 Drop the lease write's findings, the controller's artifacts-gone and the issuing gate (issue 1126, B9)
ADR 0590 supersedes the lease write: mesh-grants says no lease event any
more, so the watchman's lease-altered and removal-failed findings and
their row had nothing to hear. The ended controller's consumers are
removed as hand acts, so their deletion is no consumer lost; only the
one on DEAD_LETTER_NOTICES is still read, since its removal hands the
notices to the watchman. The builder hears artifacts-gone from the
artifact sweep seat alone, and mesh-enrolment's tokens no longer report
an issuing gate that nothing can turn off.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:35:55 +01:00
jschoubben f7f3518a04 Take the controller out of the judge, the checker's siblings and the assigner (issue 1126, B8)
The controller ended by its unassignment (ADR 0590) and mesh-sdk drops
its seat, the grants seat and placement's adopt and hand-back. The
indexer's judge still carried the controller's seat entry, the old
removal words and the two placement verbs, and counted 58 seats; the
checker still placed mesh-controller beside the node-engine's and the
catalogue's tests and failed a node-engine check without its trunk; the
assigner still asked a compose of the running controller, served adopt
and hand-back as stubs, and kept a policy-split finding that could no
longer arise.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:32:56 +01:00
jschoubben d4aa1fe11b Hear the keeper alone and stop reading the controller's lease in mesh-grants (issue 1126, B7b)
ADR 0590 §3 ends the choice of a condition source by the keeper's
take-over key: the controller is unassigned and the keeper is the one
source. The part still read the take-over key, fell back to the
controller's condition events when it was absent, and read and watched
the controller's lease, which nothing writes any more. It now hears the
keeper's events alone and reads neither.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:27:19 +01:00
jschoubben 14046f9b14 Remove the lease write, take-over and their tools from mesh-grants (issue 1126, B7a)
The controller ended by its unassignment (ADR 0590), which supersedes the
lease write and its hand-back whole and makes the grants part's list the
only one the bus node follows. The lease write, its check, the removal
marker, the take-over and hand-back asks, their five tools, the seat
claim and the lease and takeover states had nothing left to act on; the
membership notice on its subject had no reader. A whole-module test on
the bus replaces the lease write's: with no controller at all, the part
publishes a membership in its state alone, makes the module's consumer
and writes the key the bus node follows.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 11:25:59 +01:00
mesh-tools be88162cde Merge pull request 'Without the controller: mesh-grants makes consumers, enrolment issues, the watchman keeps dead letters, and the controller cannot be placed again (issue 1126)' (#137) from fix/1126-gates-delivery into main 2026-11-02 09:51:14 +00:00
jschoubben a192052d82 Refuse any placement of the controller (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@be88162cde31
mesh/check being carried to a192052d82fe on be88162cde31 | from 0490043990e2
The grants part, the enrolment seat and the watchman now act without
reading the lease, so a controller placed again would write beside them
with nothing saying so. The assigner refuses every proposal, seed or
joining node's write that would place mesh-controller, naming why.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:38:22 +01:00
jschoubben 1dae2de835 Issue enrolments and keep dead letters without the lease (issue 1126, A2)
The controller ended by its unassignment, so its lease key is absent for
good and mesh-sdk reads that as not handed away: mesh-enrolment's issue
stayed off and the watchman never took DEAD_LETTER_NOTICES, whose notices
age out after a week. Both now act without reading the lease; the
watchman still waits, and says so, while the controller's consumer on the
notices stands. Neither manifest reads the lease any more.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:35:40 +01:00
mesh-tools 0490043990 Merge pull request 'view_reach answers one module's delivery reach' (#136) from fix/view-reach-filter into main 2026-11-02 09:33:51 +00:00
jschoubben e094e362f4 Run mesh-grants' handed behaviour without the lease (issue 1126, A1)
The controller ended by its unassignment and its lease key expired, so
every gate on "the lease is handed away" stayed shut for good: no account
consumer was created or matched, grant-missing was never raised, the
enrolment users kept the controller's subject, memberships were still
said on their subject and the sealed key never said LeaseHanded. The
handed branch now always runs, and a hand-back is gone.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:33:14 +01:00
jschoubben 8850a5673e view_reach takes a module, so one module's reach is asked without reading the whole repository
mesh/delivery merged: source.main moved to novox/mesh-delivery@0490043990e2
mesh/check being carried to 8850a5673e61 on 0490043990e2 | from cc9c681fa348
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 10:24:54 +01:00
mesh-tools cc9c681fa3 Merge pull request 'The lease write stops the controller on its placed node and reads the operator's answer from the right bucket (issue 774)' (#135) from fix/774-lease-write-node into main 2026-11-02 08:52:47 +00:00
jschoubben 8f87dbe94d mesh-delivery-teller: read asks from the router's bucket the grant names
mesh/delivery merged: source.main moved to novox/mesh-delivery@cc9c681fa348
mesh/check being carried to 8f87dbe94d76 on cc9c681fa348 | from f1c33439292d
The teller read the router's record from KV_asks, a bucket no composed grant
names, so every read was denied and its operator answers could not be read
back. It now reads messenger_asks, as mesh-grants does. The integration bed
grants the real subject and reads a record on the teller's credential, so it
no longer agrees with the mistake. (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 09:44:26 +01:00
jschoubben 7cee3842b8 mesh-grants: stop the controller on its placed node and read asks from the router's bucket
The lease write addressed node-service-manager at the lease holder's host, a
hostname no node carries ("novox-be" on novox), so the operator's yes timed out
at its first step and failed without a log line. The node now comes from
placement (exactly one node placing mesh-controller, else refused by name), a
failed act is logged beside its condition, and the asks record is read from
messenger_asks, the bucket the composed grant names, instead of asks, which no
grant allows. The fixtures give the lease a host unlike the node, so the tests
no longer agree with the mistake. (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 09:42:19 +01:00
mesh-tools f1c3343929 Merge pull request 'mesh-enrolment holds the enrolment seat; placement takes add-node and remove-node; the grants part composes enrolling and rotating nodes (issue 774)' (#134) from feat/l3-enrol into main 2026-11-02 07:56:39 +00:00
jschoubben 63c8687166 Let a bus module's tools bundle declare what it invokes (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@f1c33439292d
mesh/delivery-group group feat/l3-enrol merged: every member is merged
mesh/check failed (2 tests): integration: mesh-mail-reader/cmd/mesh-mail-reader.TestAcceptanceOneMatteringMailSendsExactlyOneTellNamingIt: timed out w…
The judge refused nats's manifest for its invokes (seat:mesh-hand-acts.hand-act,
which nats_bucket_expiry asks to record its hand act), and with one manifest
refused the grants part holds the whole bus user list, freezing every build
that adds bus permissions. The bundle asks through the node's tool runner,
which buscompose grants every tool and seat verb, so the invokes are granted:
the same condition that already lets the bundle declare emits (ADR 0357).
ADR 0235's rule stands for a bus module with no bundle.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 02:21:05 +01:00
jschoubben b13f65674b Pin mesh-sdk at dff0ce2, the enrolment group's rebased head, so every member builds the same contract (issue 774)
mesh/delivery-group group feat/l3-enrol checking: 4 of 4 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking b13f65674b35 on 9125fcc71206 | novox/mesh-delivery#134 in group feat/l3-enrol
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben e6ee939bd4 The indexer's closed seat set counts fifty-eight with mesh-enrolment (ADR 0587 §1)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben bde60130a8 The assigner's add-node test writes no lease: since the removal group the assigner reads none and its bed has no lease bucket
The enrolment's issue waits for the lease write itself (ADR 0587 §4); the assigner takes an add-node on its warrant at any time (ADR 0567 §8). (issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:48:53 +01:00
jschoubben 795724f5ca mesh-enrolment: catch up only on an ask answered yes, and drop the address key first at confirm, decline and refused add
ADR 0589 version 29, clause 6. At start and after a catch-up an address key is an expiry never heard only when its
enrolment key is gone and the router records its ask answered yes; one declined or not answered yet is no expiry, and one
not answered stays until its ask ends. A confirm, a decline and a refused add delete the address key before their other
steps, so one stopped midway never leaves a key the catch-up would take for an expiry and unplace a node that joined. A
refused add (mesh-placement's refusal of the proposal, not a call that did not reach it) now ends the enrolment as refused;
a call that failed still waits for again. Tests: rows in the end table (after a confirm stopped before its enrolment key
went, after a decline, after a refused add, a declined ask, an ask not answered: nothing unplaced) and the integration test's
holder-was-down case reading the yes; no new function.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 44fee9eb0e mesh-enrolment: end an enrolment that expired unseen from its address key, taking the node out of placement
ADR 0589 clause 6: a placed but unconfirmed node is always taken out again, on the warrant of its ask. The address key now
names the ask and the issue's time (delivery.EnrolmentAddress, mesh-sdk c7130e3), written just before the ask is raised and
with no expiry of its own, so it outlives the enrolment key; the holder deletes it at every end. An expiry, seen or not, ends
the enrolment in this order, each step safe to repeat: remove-node on the ask's warrant (only when placement holds the
node), enrolment-expired naming the rebuilt issued event (delivery.EnrolmentIssuedID), the vault's drop of the token, and
the address key last. Reconcile finds every address key that names an ask and whose enrolment key is gone (expired while
the holder was down, before its watch caught up, or an end that stopped) and ends it; it runs once the watch has caught up,
on every expiry, and again with a growing wait while a step fails. An address reserved by an issue under way (no ask yet) is
left alone. Tests: rows in the end table (expired unseen; the holder down with a drop that fails once; a reservation under
way) and the integration test's expiry case (the address key goes; a holder that was down unplaces on its pass); no new
function.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben cbcbb9644b mesh-enrolment: keep the tunnel key in delivery.Enrolment's own field
mesh-sdk 22c03af carries the machine's tunnel key on the enrolment (OverlayKey, overlay-key, required at every stage), so the
holder's wrapper that kept it beside the contract is gone; the wire is unchanged. Test fixtures give the seeded enrolments a
real tunnel key, which the contract now requires.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben d1a256b429 mesh-grants: give the enrolment fixture its tunnel key, which an enrolment now requires
mesh-sdk 22c03af (feat/l3-enrol) requires overlay-key on every enrolment; the fixture of the enrolment-user subtest carries one.

(issue 774, review round 1b)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 40bbc7e0d3 mesh-enrolment: take the tunnel key at issue, place the node on the yes, take it off again at an unconfirmed end
The operator's decision of 2026-11-02 01:05, on mesh-sdk 4224665:
- issue takes overlay_key, the machine's tunnel public key as `nox-mesh-host key` prints it, and refuses a missing one. The
  token's first tunnel is whole: key, address, range (the hub's tunnel range in placement, the one the controller's token
  carries today; a hub with none is a finding, never a guess), hub key and hub endpoint.
- On the operator's yes the node is proposed into placement at once (add-node with its tunnel key alone, on the warrant), and
  only then is the token made, so the hub peers it before redeem. Redeem writes no placement: it refuses a tunnel key other
  than the one issued for, or a node placement does not hold yet, and the certificate's call carries all four keys presented.
- A revoke, a refusal or an expiry before confirm takes the node off placement (remove-node, same warrant) before the end is
  said, and the end brings the token's drop.
- The assigner applies a remove-node: the node's key deleted at once, with no ask, placement-changed carrying the warrant.
- The tunnel key is kept as one more member of the node key's JSON, beside delivery.Enrolment, until the contract carries it.
Tests: rows in the existing tables and both integration tests (the token's Missing() is empty; the node is placed before
redeem and leaves placement when its enrolment expires); no function added, mesh-delivery stays at 816.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:53 +01:00
jschoubben 70f13a3775 mesh-enrolment: place before the vault is asked, read the grants pass whole, name the expired enrolment's issued event, carry the first tunnel
- redeem writes the node into placement before it asks the vault for the account and the certificate of the serving key
  presented (the vault reads the node's address from placement); a retry by the same keys still finishes (blocker 3).
- loadedVia reads mesh-grants.user-list's pass as the part's whole Result, the list state under user_list; the fixture
  testdata/grants-pass.json was written by mesh-grants' own Result, not by hand (blocker 4).
- enrolment-expired is delivery.EnrolmentExpiry with IssuedID, through its codec; a key that expired before its content was
  seen names no issued event, so nothing is said and its token is dropped at once (blocker 6).
- issue builds the joining node's first tunnel from placement (the hub, its endpoint, the node's address) and the hub's
  record (its tunnel key) and passes it to enrolment-token; no hub, or a hub with no key or endpoint, is a finding of issue
  (ADR 0587 §3, insight of 2026-11-02).
- The assigner takes an add-node only from mesh-enrolment's own bus user (stdio.Callers), never person.mesh-enrolment.
- The watchman raises machine.<node>.node-account-fallback from a node's health report's fall back and clears it when the
  report no longer carries one (ADR 0583 §6, blocker 7).
- mesh-enrolment's ten unit test functions are two table tests now (blocker 8): the repository is back at 816.

(issue 774)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 189e64c486 mesh-enrolment: its integration test, whole on a real bus
The holder's code over a real nats-server bucket with a time to live per key: issue, the operator's yes, redeem as enrol.ace,
the node confirming as the login its key of node-logins names (the stand-in vault writes it at node-account), node-enrolled
and placement-changed said on their subjects, the user enrol.ace composed while the key is live and gone with it, the token
dropped only after the end; and a token never redeemed expires on the bus, is heard, says enrolment-expired and is dropped.
Found by it: a bucket with a time to live per key must keep one revision per key, or the bus expires nothing (nats-server
2.11.17), so the state enrolments declares no history. Paid for: the integration test is paid by merging the builder's
archive-only and root-module integration tests into one table test; the module's ten unit tests by merging the assigner's
three small composition tests into one table (net 2 of 10 paid; the rest is for the lead, the repository being over budget).

(issue 774, 774/S60)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 1a1d3a1ecc Compose both of a rotating node's logins from the vault's node-logins, so its engine can test-dial the new one
ADR 0583 §1 (insight 2026-11-01) and §2: the grants part reads the vault's node-logins as its named reader and
composes a node user per login, both while the node rotates and the new one alone after rotation-done; each node's
engine reads its own node's key alone (composed by mesh-sdk while the vault serves rotate-node). The part follows the
state by watch, so a rotation begun or ended composes again at once. A second login is judged its node's when the
list is compared (Placed), and a hash changing by a rotation node-logins names is not counted as a writer changing
several hashes at once (sealed.go). The state not on the bus yet is a standing gap: every node logs in as node.<n>,
as today. Tested as a subtest of TestGrants, no new test function. Built against the group's mesh-sdk head 5000774
through a go.work outside the worktree; the pin is the lead's.

(issue 774, 774/S53)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:44 +01:00
jschoubben 288364f28d Compose each enrol.<node> user from the seat mesh-enrolment's keys, so a token issued by the seat can be redeemed
ADR 0587 §4: the grants part composes enrol.<node> from a key at live and the vault's hash, granted only the seat's
redeem and its own inbox; §5: when the key ends the user goes. The part reads mesh-enrolment.enrolments as its named
reader and follows it by watch: a key reaching live composes the user, and its end (by the holder, or by the bus at
its expiry) is a change of the key that takes it out, so no consume of the seat's ending events is added beside the
watch (the subtask named one; the watch already hears every end, the bus's expiry included). From the lease write the
user loses the controller's enrolment subject (§6) and the standing gap of the controller's tokens is dropped; before
it the gap stands. Tested as a subtest of TestGrants, no new test function. Built against the group's mesh-sdk head
5000774 through a go.work outside the worktree; the pin is the lead's.

(issue 774, 774/S52)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:32 +01:00
jschoubben 447bcf3a6b The assigner takes an add-node from mesh-enrolment and writes the joining node with no ask
propose takes the enrolment's warrant {node, address, ask, warrant} and refuses an add-node from any caller but
mesh-enrolment, with no warrant or with a warrant naming another node or address (delivery.AddNodeRefusal); the joining node
counts as having a record for this one change, as ADR 0587 §7 changes ADR 0567 §2. The assigner's process then writes
placement.<node> at once, with no node to answer and no ask raised, and says placement-changed carrying the enrolment's
warrant; the hand-act record of that warrant is the enrolment's, made when it was given. The holder of mesh-enrolment calls
propose with the change, the keys and the warrant read from the router's record of the ask. Tests: rows appended to the
seat's verbs test and a case to the whole-seat integration test; none added, so none to pay for (the budget is paid in
774/S60).

(issue 774, 774/S59)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:45:12 +01:00
jschoubben 07d26adcd3 mesh-enrolment: read the group's mesh-sdk contract instead of its own copies of the shapes
The seat, the enrolment state, redeem and its proof, the answer the vault signs, and the vault's two verbs' answers are now
mesh-sdk's (catalogue and delivery), which the vault, the grants part and the engine read too; the proof is the contract's
RedeemProof, the answer carries the login, account, certificate and the vault's signing key (read from its state signers),
and the bucket's values are written through the contract's codecs. The expiry event names the issued event by message id as
well as by the sequence, which a bundle cannot learn. The module builds against the group's mesh-sdk head through a go.work
outside the tree; the lead re-pins go.mod before the push. Tests merged into the same functions; none added.

(issue 774, 774/S54-S58)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 02214cf633 mesh-enrolment: revoke, the expiry marker and the drop of the token
revoke ends an enrolment at any stage without an Approve and records it as an act done by hand (the vault retires an
account minted for a redeem under way); the bus's expiry marker of a key at live says enrolment-expired with the issued
event; the durable consumer of the seat's own ending events asks the vault to drop the token and its hash, acknowledging
only once the vault answers, never before an end. No event or key holds a token (ADR 0587 §5, §8). Run starts the watch,
the consumer and the operator's answers. Tests added: two table-style tests.

(issue 774, 774/S58)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben f904ff425e mesh-enrolment: confirm closes an enrolment on the node's own call
Only the node dialled as node.<node> confirms (the runtime's proof of the caller is the confirmation, ADR 0587 §6); the
holder then says node-enrolled with the key's content and deletes the address key (while it still names the node) and the
node's key. A stop between the event and the deletes is finished by calling confirm again; the event keeps one message.
release now answers what it could not delete. Tests added: one table-style test.

(issue 774, 774/S57)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 89370b6f5c mesh-enrolment: redeem, the joining machine's one call
redeem answers only enrol.<node> as the runtime proves it, checks the proof against the identity key presented, spends the
token once by compare-and-set (the keys lose their expiry), asks the vault for the node's account and certificate, writes
the node into placement on the issue's warrant, answers once the bus node loaded node.<node>, signed by the vault; the
same keys called again finish what is not done and answer the same (ADR 0587 §5-§7). The proof is mesh-host's EnrolProof
with the node's name in the token's place, since the seat never holds the secret. Tests added: two table-style tests.

(issue 774, 774/S56)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 4edd0e526c mesh-enrolment: issue, the operator's ask and the warrant, and again
issue refuses at once naming every finding (a bad name, a name a node record, a placement key or an open enrolment
holds, an expiry past a day, any call before the lease write), reserves the lowest free address by creating its key,
writes the enrolment at asked and asks the operator (ADR 0587 §4). On the yes the warrant is recorded, the vault
makes the token and both keys go live with the bus's expiry; a no or an expired ask takes both keys and says
enrolment-refused. Tests added: three table-style tests; none deleted here (paid with 774/S59's merges).

(issue 774, 774/S55)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
jschoubben 5d5ee69775 Add the module mesh-enrolment: the seat's skeleton, its bucket of open enrolments and the tokens verb
The holder of the seat mesh-enrolment (ADR 0587 §1, §2, §8) runs as a bundle on its own account, keeps the bucket
enrolments with a time to live per key, and answers tokens: every open enrolment or one node's with stage, expiry,
ask and revision, the bucket's fill, whether issuing is on and why, and for one node at live the token sealed by
the vault and relayed, kept nowhere. Tests added: two (one table-style test of tokens, the manifest test); the
repository is over its budget, paid by merging tests in the assigner's seat tests in 774/S59's commit.

(issue 774, 774/S54)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 01:44:46 +01:00
mesh-tools 9125fcc712 Merge pull request 'mesh-grants writes the controller's lease away; the assigner and the grants part stop reading the controller (issue 774)' (#133) from feat/l3-removal into main 2026-11-02 00:35:38 +00:00
jschoubben 17ef1d577d Pin mesh-sdk at c21d8fd, the removal group's head, so every member builds the same contract (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@9125fcc71206
mesh/delivery-group group feat/l3-removal merged: every member is merged
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#590: the ask mvyhumdm860c7f's step on shanks did not end within 30m0s of its start |…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:53:42 +01:00
jschoubben da27cd1593 Name the lease write's earlier steps by their issues, not milestone labels
(issue 774, 774/S14)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:06 +01:00
jschoubben aa6a4e7a59 The indexer's closed seat set counts fifty-seven with mesh-grants
ADR 0582 clause 1 adds the seat mesh-grants to mesh-sdk's set; the indexer's guard of the closed set counts it.

(issue 774, 774/S1)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:51:06 +01:00
jschoubben 8bf45fbafe Integration test: mesh-grants writes the lease and hands it back on a real bus
ADR 0582 "How it is checked": against an embedded bus, a stand-in controller holds the lease in a bucket with an
expiry (2 s standing in for 15 s, through grants.LeaseExpiry, so the wait past it is seconds), with stand-in
node-service-manager and bus-tool seats (the expiry tool acting on the real bucket's stream, as nats-tools'
SetBucketExpiry does). On the warrant the key holder names mesh-grants, the expiry is gone, the controller's
create of the key is refused, and the lease still reads handed away past the old expiry; on the hand-back's
warrant the expiry is back and the controller takes the lease again. mesh-grants requires nats-server for it,
at mesh-sdk's version.

Budget: TestTheLeaseWriteOnTheBus (new, integration) is paid by merging the checker's
TestTheCheckerAwaitsTheAssignersComposition into TestTheCheckersLineSaysTheAssignersComposition as its first case.

(issue 774, 774/S23)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:40:00 +01:00
jschoubben e18be5cabe The watchman raises lease-altered and removal-failed as findings, and clears them when the lease and the removal read right
ADR 0582 clauses 4 and 6: mesh-grants says lease-altered and removal-failed "as a finding the watchman raises".
The watchman hears the seat mesh-grants' three events on core subjects (mesh-sdk WatchmanHears, its grant and
its subscriptions from one list, rather than a consume and a durable consumer), raises lease-altered naming
what changed and clears it at the next lease-handed, and raises removal-failed naming what failed and clears it
once what it names reads gone: the controller's consumer of DEAD_LETTER_NOTICES by its deletion advisory
(heard already), the stream ASSIGNMENTS by its own. The finding DEAD_LETTER_NOTICES.controller stays as it is.

Test: a row in TestTheWatchmansFindings; no new function.

(issue 774, 774/S22)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:37:54 +01:00
jschoubben a78fa3db9d mesh-grants reads placement alone and hears no controller event; this repository's manifests depend on the controller only by the lease
ADR 0549 §5, ADR 0564 §9, ADR 0571 §10: the part no longer reads mesh-controller.assignments (the union with
placement goes; placement not read is a failed read that places nothing) nor consumes the controller's
condition-raised and condition-cleared, the keeper's being the one source. ADR 0571 §13's test for this
repository replaces the guard of ADR 0549 §6 and its list ../assignments-readers, now empty and deleted: at
phase write no module.json here depends on mesh-controller beyond the lease and the three kept to the write
(catalogue.ManifestRefusals).

Tests: in TestGrants the fixture places through the seat's placement (assign, unassign), the subtest of the
stale assignments becomes 774/S29/AC1 (placement alone gives the fixture's three memberships with no read,
consume or call of the controller; the fixture is the suite's own, whose assignments named the same modules,
not a copy of live memberships), and the assignments-readers subtest becomes 774/S24/AC1. No new test function.

(issue 774, 774/S29, 774/S24)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:35:59 +01:00
jschoubben b81d7475e8 mesh-grants judges the controller's dependents with mesh-sdk's judge, and its keep list holds the three kept to the write
ADR 0571 §13: the judge moved to mesh-sdk (catalogue.Dependencies, Judge, KeepList; issue 774, 774/S4) so the
manifest tests of three repositories and grants_lease_check give one judgement; mesh-grants re-exports it by
its names. The keep list there holds the three kept to the write alone (774/S30), superseding the keep of
pacman's read at 4a84370. grants_lease_check no longer counts the controller's standing assignments, which
nothing reads any more.

Tests: TestTheLeaseCheckJudgesEveryDependencyOnTheController keeps the tool's own rows (what it read, the
lease, the streams); its dependency rows moved to mesh-sdk's TestTheLeaseWritesCheckJudgesEveryDependencyOnTheController
(774/S24's replacement of the moved rows), and TestTheKeepListNamesItsRecordsAndThreeToTheWrite went with them.

(issue 774, 774/S30, 774/S24)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:35:59 +01:00
jschoubben 7ec6767418 mesh-grants holds the seat mesh-grants and writes the controller's lease on the operator's warrant
ADR 0582: lease-write is refused, naming each check, until the lease check at phase write and every condition
of clause 2 read green (design 47's earlier issues resolved, every node trusting the vault's key, mesh-config's
compare empty from the indexer, the upgrade compare empty, the keeper's take-over done, the vault minting with
nothing held back, the bus module listing its expiry and removal tools). It asks the operator once; the yes
binds the act, the holder, the ask and the checks' digest, and the checks are read again on the warrant (one
that reads otherwise refuses it, named). Then, each step read back: node-service-manager.stop on the
controller's node, the list composed with Records.LeaseCut and waited on until the bus node's have says it
loaded it, the expiry removed through nats_bucket_expiry, the key holder written (with the ask and the time),
lease-handed said, the act recorded through mesh-hand-acts.warranted, and removal-due written with a day to
live. lease-altered is said once when the key or the expiry differs; lease-hand-back reverses it for a day
and is refused once the removal ran; removal-due's marker alone starts the removal of the controller's
consumer and the stream ASSIGNMENTS, recorded as hand acts, and a failure says removal-failed and writes the
key again with a day (grants_removal_due is the recovery for a missed marker). From the write memberships are
written to the state alone (ADR 0571 clause 10), the user-list key says lease-handed, and the user list's
hand-back answers gone (ADR 0547 §11). Every step waits on an event, never a timer.

Bent rule: S14-S21 are one commit, not eight: the steps are one state machine in leasewrite.go, and splitting
it by hunks would review the same file eight times for nothing it protects.

Tests: TestTheLeaseWrite (new, one function with a subtest per subtask), paid by the deletion of
TestTheKeepListNamesItsRecordsAndThreeToTheWrite (moved to mesh-sdk, S4/S30) and the assigner's two deletions
(S25). Fixtures shaped from each writer's code: systemd's Manager.Status, nats-tools' BucketExpirySet,
ConsumerRemoved and StreamRemoved, mesh-issues' Brief, mesh-delivery-view's TrustsAnswer, mesh-config's
Comparison, the assigner's upgrade compare, mesh-vault's vault_status.

(issue 774, 774/S14, 774/S15, 774/S16, 774/S17, 774/S18, 774/S19, 774/S20, 774/S21)

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:31:35 +01:00
jschoubben 30ec4329e0 The assigner's integration tests place their modules and name the node's own composition; a placed node's backup answer stays in its want
The integration tests put placement (not assignments) and the node's own composition (not the controller's) on the bus; the
three that tested the controller's judgements of a move (waiting for a composition, a record move, a user-list revision) go,
their unit counterparts standing. Run locally against the embedded bus, all of this module's integration tests pass.
Found by them: the answer to a failed backup was dropped from a placed node's want at every pass, because the carry-over
compared the want's digest, which a placed node's want does not carry; fixed in wants.go, with a case in the existing unit test.
Tests: three integration tests deleted, none added; one case added to an existing unit test.

(issue 774, 774/S26, 774/S27)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:30:51 +01:00
jschoubben 3bae0d8fa2 Placement's adopt and hand-back are answered as gone, naming their records, and change nothing (ADR 0547 §11, ADR 0567 §11)
The controller's placement-export is no longer called, its answer reader, the seed and the hand-back flow of the holder are
removed, and a seed or hand-back proposal a previous process left open ends at its next pass.
Tests: the adopt, hand-back and seed cases of the seat's verb test and of Propose are replaced by one gone-case; the export
reader's table and the seed test go.

(issue 774, 774/S28)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:21:46 +01:00
jschoubben 7bd18306d4 The assigner reads no composition of the controller: upgrade compare and rule 7 read the digest a node's own have names (ADR 0571 §8)
mesh-controller.composed leaves the reads and the watches; upgrade compare lists the node composition digests of the haves
instead of comparing with the controller's policies, which the controller no longer writes.

(issue 774, 774/S27)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:20:13 +01:00
jschoubben 1302eb9e1f The assigner reads no assignment and no lease: the placement alone names where modules run (ADR 0567 §8, ADR 0549 §5)
Every handed-away branch is the only one: world.assigned is the placement, the pair comparison and its conditions go, the
lease and assignments buckets are no longer opened, watched or declared, and a proposal of any kind is taken.
Tests: the pair-comparison test and the controller-composition record-ask test are replaced by one placement test; the
rest are edited to place their modules instead of assigning them.

(issue 774, 774/S26)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:18:50 +01:00
jschoubben f2ed0938ce The assigner calls no verb of the controller after a placement is written: placement is the one source (ADR 0567 §7, §8)
Drops callAll, the refused-call conditions and the pin reads; the conditions a previous process raised are cleared once.
Tests: deletes four (the refused-call condition, its clearing, the contract of the calls, the condition's shape) and adds one.

(issue 774, 774/S25)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-02 00:13:50 +01:00
jschoubben 4a84370b2f mesh-grants: the lease check keeps pacman's read of declared-packages until the write's delivery
pacman still falls back to the controller's declared-packages for a node whose engine has
not written its own packages (ADR 0565 §6), so the check named that read as ended and would
refuse the lease write. It is now a kept read, dropped with the others at the write's own
delivery (issue 774, design 47 L3).

Quick fix chosen over removing pacman's fallback: it matches the record as it stands, and
the read goes with the other kept reads at the write's delivery.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 23:27:29 +01:00
mesh-tools 7d1b8864cc Merge pull request 'mesh-grants pins the mesh-sdk that grants each engine a read of its own declared-packages key (ADR 0565 §6, issue 1953)' (#132) from feat/node-packages into main 2026-11-01 17:02:47 +00:00
jschoubben b18c62f671 Pin the grants part to the group's mesh-sdk head b2a27ed5, so each engine may read its node's key of the packages mirror (ADR 0565 clause 6, issue 1953)
mesh/delivery merged: source.main moved to novox/mesh-delivery@7d1b8864cc08
mesh/delivery-group group feat/node-packages merged: every member is merged
mesh/check checking b18c62f671e1 on cc38ca3f8dc8 | novox/mesh-delivery#132 in group feat/node-packages
The node seat's verb packages compares the engine's key with the controller's mirror until the lease write; the grants part composes the engine's read of its own node's mirror key only from this mesh-sdk.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 16:24:04 +01:00
mesh-tools cc38ca3f8d Merge pull request 'Agree a toolchain by strict majority; a node on another version sits out (ADR 0584, issue 1578)' (#131) from fix/toolchain-majority into main 2026-11-01 15:04:20 +00:00
jschoubben d739516056 Name the agreed toolchain in the checker's check asks; pin mesh-sdk 67354f8 (ADR 0584, issue 1578)
mesh/delivery merged: source.main moved to novox/mesh-delivery@cc38ca3f8dc8
mesh/delivery-group group fix/toolchain-majority merged: every member is merged
mesh/check pass d739516056dd on 55a506975b01 | unit 7056 run, 243 kept, 16 skipped | integration 7194 run, 275 kept, 25 skipped | over bound: unit, in…
A check ask named no toolchain, so a node on the minority version could run it and KeepPasses record the pass as run on the agreed one. The check ask now carries the agreed toolchain and a node off it declines (mesh-sdk), for the checker's decline loop to put again. Pins the builder, indexer and checker.

Known limit: the checker puts a declined check ask again at once, with no pause; the declining puller skips that kind until its next pull wait, which paces a lone minority node.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:38:16 +01:00
jschoubben a6c225eb45 Pause before a declined ask is put again with a context's bound, not a timer (ADR 0437 §8, issue 1578)
The builder's manifest test refuses time.After. The previous commit's pause used it.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:36:52 +01:00
jschoubben 5085b681e9 Count no decline as an asking; bound declines by time and name the nodes (ADR 0584, issue 1578)
A build node off the agreed toolchain declines the ask and, idle, takes the ask put again at once, so three declines used the three askings and failed the build. A decline now uses no asking: the ask is put again after a short pause, and past a bound from the first decline the key fails naming the declining nodes and their reasons. A person reads which node sits out in the key's step text, which the view's builder_builds answers as stored.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:36:23 +01:00
jschoubben 9e407f8f76 Build with the toolchain a majority of build nodes report; say which nodes sit out (ADR 0584, issue 1578)
One build node on another version refused every build of the language. The builder and the checker now follow mesh-sdk's majority agreement; the key's step names a node that sits out ("ace sits out: typescript 26.12.1, agreed 26.13.0"), and a build ask a minority node declined is put again for a node on the agreed version. Check steps compare no toolchain, so clause 2 already holds for them. Pins builder, indexer and checker to mesh-sdk 71be15f.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:26:55 +01:00
jschoubben 8545e7fc9f Cite ADR 0584 where the builder waits on a toolchain the nodes do not agree on (issue 1578)
Comment only; the majority rule itself is in mesh-sdk's AgreedToolchain.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 15:22:04 +01:00
mesh-tools 55a506975b Merge pull request 'The assigner, builder and grants read off the controller; upgrade compare by watch (design 47 L5, L7)' (#130) from feat/l5-l7-main into main 2026-11-01 13:53:33 +00:00
jschoubben b7dfe4c637 builder: the held fake in its integration tests answers on the artifact sweep's seat, which the builder now asks instead of the controller
mesh/delivery merged: source.main moved to novox/mesh-delivery@55a506975b01
mesh/delivery-group group feat/l5-l7-main merged: every member is merged
mesh/check checking b7dfe4c6375a on 58748c2be161 | novox/mesh-delivery#130 in group feat/l5-l7-main
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 10:58:15 +01:00
jschoubben c4b05a3e93 Pin group A's mesh-sdk head on main
mesh/delivery-group group feat/l5-l7-main rejected: composed together: the checker's verdict at the members' heads (e32e2c9c4fea): integration: mesh-delivery-b…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (2 tests): integration: mesh-delivery-builder/cmd/mesh-delivery-builder.TestADoneKeyWhoseBlobIsGoneIsBuiltAgainAtStart: builds.ca-tr…
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:57:47 +01:00
jschoubben 38f2401f17 mesh-grants and the builder pin mesh-sdk at 8311ecb
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:28 +01:00
jschoubben c8b041ab37 mesh-wants upgrade reads through mesh-sdk's watched reader, keeping no mirror of its own
One helper serves every part that reads whole states (ADR 0580 clause 3): the assigner's own watched mirror goes, its
compare reads the large states from stdio.WatchedStates and each manifest once per digest. mesh-sdk pinned at 8311ecb.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben f52f47e1ff mesh-grants and the assigner read the grants part's key alone, root-free is each engine's word, and the hand-act log is the seat's alone (ADR 0571 clauses 9 and 11, ADR 0565 clause 11, issues 1953 and 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben 213ebe7cfd The builder asks the artifact sweep's seat for held and no longer the controller (ADR 0571 clause 2, issue 1955)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
jschoubben 90e278cb97 mesh-wants upgrade: read the large states by one watch each, started once, not a get per key, so compare answers in time (issue 1953 L5f)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 09:54:22 +01:00
mesh-tools 58748c2be1 Merge pull request 'The indexer judges every manifest once and says its verdict; the checker judges a change with the same judge (ADR 0578, 0579)' (#129) from feat/judge-once into main 2026-11-01 07:54:35 +00:00
jschoubben 8e9cc58301 Fall back to a captured copy only for mesh-host, which the checker places beside no mesh-delivery check (issue 1585)
mesh/delivery merged: source.main moved to novox/mesh-delivery@58748c2be161
mesh/delivery-group group feat/judge-once merged: every member is merged
mesh/check checking 8e9cc5830119 on b1f262d7d06e | novox/mesh-delivery#129 in group feat/judge-once
Review of fb96216: a missing mesh-sdk or catalogue in a merge check is the checker's fault and fails, never judged
against a stale copy. The captured placement guard is refreshed to mesh-host's main 233bb23.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:49:23 +01:00
jschoubben fb96216e46 Read a sibling the checker does not place beside mesh-delivery from its captured copy, said (issue 1585)
The judge's tests moved from mesh-sdk, where mesh-host was beside them; the checker places only mesh-sdk and the
catalogue beside mesh-delivery (besideRules), so the node-engine's path rules failed as not beside. A sibling the run
holds beside is read there; one it does not is read from testdata/beside and the log says which; only a sibling
with neither fails.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:47:22 +01:00
jschoubben 5059fef3e6 Expect the mesh packages a merged message carries in the forge bridge's whole test (ADR 0573 §2)
mesh/delivery-group group feat/judge-once rejected: composed together: the checker's verdict at the members' heads (253f35815198): unit: mesh-delivery-indexer/…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: mesh-delivery-indexer/judge.TestTheResolvedPathRulesAreTheNodeEnginesOwn: mesh-host is not beside this check in MESH_CHECK_BE…
Issue 1585: the bridge names the mesh packages each module's go.sum names (builder-1585b's fold); its integration test
still wanted none. All nine modules' integration suites pass locally (the lead allowed a whole local run this round).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 04:20:05 +01:00
jschoubben 564a04b45c Pin the group's mesh-sdk head after the gofmt fix
mesh/delivery-group group feat/judge-once checking: 6 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 564a04b45c42 on b1f262d7d06e | novox/mesh-delivery#129 in group feat/judge-once
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:26:58 +01:00
jschoubben edf7b3c0bf Answer review round 1: fresh is only what a take-in newly wrote, a merge keeps the verdict, the check compares with the trunk (ADR 0578, 0579)
mesh/delivery-group group feat/judge-once checking: 6 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking edf7b3c0bf3a on b1f262d7d06e | novox/mesh-delivery#129 in group feat/judge-once
Issue 1585, R1: fresh holds the digests a take-in newly wrote (a new module, a changed manifest); a registered or
taken-in module is judged too; a start pass that fails is judged again at the next message heard, no timer.
R2: a module's verdict, problems and warnings are carried while its digest stands and are no part of the compare.
R3: the checker judges the trunk beside the change and fails on every problem the change case adds, an untouched
manifest's included. R4: the judge reads the indexer's kept predefined keys, or the catalogue member's when it
changes them. A log line names a package no held module publishes. Every module pins mesh-sdk 62cddf4.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 03:12:09 +01:00
jschoubben 0e819eea4d Never refuse what the judge accepted before: warn, and refuse only a new or changed manifest (ADR 0579)
Issue 1585: the indexer keeps an accepted manifest accepted when a stricter judge re-judges it, its problems as
warnings carried in its value; a merge's new manifests are judged in full. The watchman raises manifest-outgrown as
a warning until mended. The checker fails only on a manifest the change adds or changes and names the rest as
warnings. Every module pins mesh-sdk 19c2042.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:45:18 +01:00
jschoubben f4ecc6fa64 Pin every mesh-delivery module to mesh-sdk feat/judge-once a20bdf7, the judge out and package versions in
Issue 1585: builder-1585b's package versions folded onto the judge once; one mesh-sdk head for the group.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:42:05 +01:00
jschoubben fa918318b2 Test the checker's latest package versions
The checker builds every Go check against them (issue 1585, S14); a row
in the repoModules test proves they come from the modules that hold each
package and leave out a package no module holds.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:41:28 +01:00
jschoubben 3f2303a086 Build checks against the group's package heads, not go.mod pins
ADR 0573 §4 makes refusing a member for its mesh-sdk pin a fault of the
check (issue 1585, S14). Every Go check ask now carries the latest
published version of each package, or the head of the group member that
publishes it, so the pin check, its forge reads and its tests go, and
the controller's tests read mesh-sdk at the version it is built with.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:41:22 +01:00
jschoubben ae6a54bca2 Take each mesh package's latest version into the fingerprint
ADR 0573 §2: a module builds against the latest published version of
each mesh package it uses, and a package's trunk merge moves every
dependent (issue 1585, S13). The bridge reads the packages each go.sum
names, the indexer keeps them and takes each one's trunk merge into the
fingerprint, a candidate takes a group member's head instead, and the
builder's ask carries the versions so the build module writes them.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:41:04 +01:00
jschoubben d6807233f2 Keep the reserved stream prefixes in the judge, the one part that reads them (ADR 0578 §4)
Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:38:00 +01:00
jschoubben 3ae308426a Merge four of the judge's moved tests into one, so the move adds no test to the budget
Issue 1585: three tests are split between mesh-sdk (their composition half) and the judge (their judging half).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:36:23 +01:00
jschoubben 8a98271680 Move the judge's rules and their tests from mesh-sdk into the indexer's judge (ADR 0578 §4)
Issue 1585, S6-S8: 192 judge-only declarations (the strict parser, every manifest problem function, the tables)
moved whole by an AST mover, methods becoming functions; 163 helpers composition keeps too are copied. 245 tests
move with them, reading the captured catalogue through a copy of internal/beside. judge.WithTemplatesFrom and
ReadTreeManifest judge what mesh-sdk's readers now only fill. mesh-delivery's own tests decode. Pinned to mesh-sdk 25a2cec.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:34:26 +01:00
jschoubben 0469641b9d Pin mesh-delivery's modules to mesh-sdk feat/judge-once 7cab079
Issue 1585: the group's mesh-sdk head (lenient Decode, verdict contract, frombus holding a refused module).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:21:32 +01:00
jschoubben 4f6f38e16a Keep every strict read of a manifest in the indexer's judge, and add no test to the budget
Issue 1585, S5-S8: the indexer's fingerprint reads go through judge.ForInputs, so judge/ is the one place in
mesh-delivery that calls mesh-sdk's rules; the command's request judging moves into judge.Request, tested in the
judge's table; two predefined tests become one.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:21:32 +01:00
jschoubben 5b6602544e Prove the judge once whole: verdicts, manifest-refused, and a bus reader that holds and goes on (ADR 0578)
Issue 1585: the indexer's merge test takes in three manifests more against the real bus; it writes their
verdicts, says each refusal, and mesh-sdk's bus reader composes the valid one and holds the refused, named; before
any verdict it takes each as accepted and names the field set aside. One test extended, no test added.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:17:57 +01:00
jschoubben efef7183ef Let grants, builder and checker decode manifests, and the checker judge a change with the indexer's judge (ADR 0578 §2, §3)
Issue 1585, S12: grants, builder and checker read with catalogue.Decode and grants holds a module the indexer
refused; the checker asks the Go build seat to run mesh-judge, built at mesh-delivery's trunk or at the group's
mesh-delivery head when it changes the judge, over the catalogue as the change leaves it, and fails on a refusal.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:15:54 +01:00
jschoubben a4772a5c1d Let the assigner decode manifests and take the indexer's verdicts, never judging (ADR 0578 §3, §5, §6)
Issue 1585, S11: world and wants read with catalogue.Decode; a trunk refusal is held, named and handed to the
composer in its input; a proposal's composition read that fails is said, not dropped; a reader that cannot read a
changed field is a warning, no longer a failure.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:11:54 +01:00
jschoubben 26852f4a9d Raise a refused manifest as the watchman's finding, so the operator hears it once (ADR 0578 §1)
Issue 1585, S10: the watchman watches the indexer's modules and finds manifest-refused for a module whose value
carries a refusal, ending it once accepted. Its mesh-sdk pin moves to the group head, so it takes health.Finding's
Ref (issue 1952 renamed Token).

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:09:50 +01:00
jschoubben 82776466b2 Let the indexer write a verdict on every manifest it holds and say each refusal (ADR 0578 §1)
Issue 1585, S9: verdicts keyed by digest, written only when they differ; a module's value carries a refusal; a
new refusal is said as manifest-refused. Judged at start (a new judge judges again) and after every merge.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:08:21 +01:00
jschoubben 4837548349 Give the mesh one judge of manifests, in the indexer, and a command the checker runs (ADR 0578 §1, §2)
Issue 1585, S5: judge.Shelf judges each manifest alone and the rules between them, giving a problem between
manifests to each module whose removal ends it (naming alone blamed the module mounts for f1-circuits' problem).
cmd/mesh-judge answers a composer request so a Go build module runs it as it runs the composer.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 02:08:21 +01:00
mesh-tools b1f262d7d0 Merge pull request 'mesh-grants takes the grants of the node-engine's data items (ADR 0577)' (#128) from feat/data-items into main 2026-11-01 00:03:16 +00:00
jschoubben ddadb75bf4 mesh-grants pins mesh-sdk feat/data-items, so it grants the node-engine's data items state (ADR 0577)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b1f262d7d06e
mesh/delivery-group group feat/data-items merged: every member is merged
mesh/check pass ddadb75bf474 on 91128d4f0af6 | unit 4510 run, 4259 kept, 49 skipped | integration 4589 run, 4361 kept, 51 skipped | over bound: unit, …
The only change taken in is ADR 0577's: each node's engine writes its own keys of mesh-host_data,
and each node's disk-load reads its own node's keys alone. Without it the engine's writes are
refused and disk-load measures nothing.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-11-01 00:19:09 +01:00
mesh-tools 91128d4f0a Merge pull request 'The watchman confirms a module unhealthy by time, and keeps a standing finding across its restart (issue 1952)' (#127) from fix/watchman-confirm into main 2026-10-31 21:34:06 +00:00
mesh-tools 5dbde07b7b Merge pull request 'mesh-grants builds memberships from the group mesh-sdk (issue 1952)' (#126) from fix/l4-quick into main 2026-10-31 21:34:03 +00:00
jschoubben 8a658a4290 Watchman: an unhealthy finding read back at the start is kept as it was (issue 1952)
mesh/delivery merged: source.main moved to novox/mesh-delivery@91128d4f0af6
mesh/check checking 8a658a4290c9 on 91128d4f0af6
At a restart a standing finding is observed again by the replayed failed have: kept, not confirmed again, and its urgent
signal runs on from the finding's Since; an urgent one stays urgent with no signal restarted.
Quick: skip the confirm when a finding stands. Thorough, chosen: the same, plus urgency read from the finding itself.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 22:01:59 +01:00
jschoubben fb8d2386b9 Watchman: a module is unhealthy when its have still says failed a minute after it first did (issue 1952)
A have that turned failed once and stayed unchanged was written once, so the old count of two writes never fired and the
finding was never written (fail2ban on novox). Confirm by time, with a signal, also at a restart.
Quick: raise the count to one write (loses the confirmation). Thorough, chosen: confirm by a time bound, as ADR 0564 s4's
signals are keys that expire, so no write count is needed.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 21:59:46 +01:00
jschoubben a2558171e1 Compose memberships with a carried holder's own seat events (issue 1952)
mesh/delivery merged: source.main moved to novox/mesh-delivery@5dbde07b7bb8
mesh/delivery-group group fix/l4-quick merged: every member is merged
mesh/check pass a2558171e11e on f83835b4b6ad | unit 2729 run, 4570 kept, 16 skipped | integration 2975 run, 4493 kept, 25 skipped | over bound: unit, …
The grants part composes every membership and runtime grant from
mesh-sdk, which now lists a held seat's events for its holder, so the
self-check the tool runner carries can say `checked`. Pinned to that
mesh-sdk head.

Quick vs thorough: quick, these fixes for L4's take-over now; the
thorough fix (research 032 / ADR 0240's health view extension) is
decided later.

Claude-Session: https://claude.ai/code/session_01NcbgfJ9Gum47zkFKWRaovL
2026-10-31 21:35:59 +01:00
mesh-tools f83835b4b6 Merge pull request 'mesh-placement keyed per node and module, and memberships from mesh-grants' own state (issue 774)' (#125) from feat/072-moves into main 2026-10-31 10:31:10 +00:00
jschoubben a313c3e843 Pin mesh-sdk at the group's head acbf80a in every delivery module the group touches (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@f83835b4b6ad
mesh/delivery-group group feat/072-moves merged: every member is merged
mesh/check failed: shanks: the check of novox/mesh-controller#487 failed: go mod download in modules/route-proxy: go: git.novox.be/novox/mesh-controll…
2026-10-31 10:25:20 +01:00
jschoubben e7f51caedb Read each membership's last value from mesh-grants' own state and say it on its subject without the assignments stream, so memberships keep flowing once nats_stream_remove takes the stream at the lease write (issue 774, research 072 row 33) 2026-10-31 10:25:20 +01:00
jschoubben 4b194215dc Read and write the seat mesh-placement's state as one key per module and node, converting each node's placement.<node> once at the holder's start (issue 774, research 072 row 37)
The holder writes a node's changed module keys, then its node key as the commit by compare-and-set at
the revision the proposal was laid over, then the placement.<node> mirror for readers not yet delivered.
Every reader assembles through mesh-sdk's PlacementKeys, which reads both forms.
2026-10-31 09:32:32 +01:00
mesh-tools af0fd83aec Merge pull request 'Teller: teller_end_ask ends one stale ask by hand (issue 774)' (#124) from fix/merge-ask-ends-on-merge into main 2026-10-31 01:33:02 +00:00
jschoubben 17abe6af32 Teller: teller_end_ask ends one stale ask by hand, so an ask whose key is never taken away cannot hold the router's room for a day (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@af0fd83aec09
mesh/delivery-group group fix/merge-ask-ends-on-merge merged: every member is merged
mesh/check checking 17abe6af32ce on af0fd83aec09 | novox/mesh-delivery#124 in group fix/merge-ask-ends-on-merge
On 2026-10-31 two merge asks of pull requests merged by another path held two of
the router's three places for the teller and a placement ask was refused for want
of room. The gate's fix ends such asks by itself; this tool is the recovery for any
ask that still outlives its question: it cancels the open ask at the router, or
drops one waiting for room, by its router id or its key, and logs who and why. The
tools bundle asks through the teller's new state ends, as indexer_take_in does.
2026-10-31 02:13:39 +01:00
mesh-tools 804d5cc065 Merge pull request 'A node-engine answers any verb of its own node, so a new engine verb needs no grants release (issue 774)' (#123) from fix/engine-seat-any-verb into main 2026-10-31 00:30:55 +00:00
mesh-tools fd62a36041 Merge pull request 'The assigner puts only changed values and keeps a placed node's record ask while it waits (issue 774)' (#121) from fix/assigner-puts-only-changes into main 2026-10-31 00:21:36 +00:00
jschoubben 14f68219cf Grants: pin mesh-sdk 748543c, so every node's engine is granted any verb and event of its node seat on its own node, composition and compose included (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@804d5cc06559
mesh/delivery-group group fix/engine-seat-any-verb merged: every member is merged
mesh/check checking 14f68219cf29 on 804d5cc06559 | novox/mesh-delivery#123 in group fix/engine-seat-any-verb
The grants part pinned eff81f1, before the verb composition (6bce1fb): the bus refused every engine's subscription to it and each engine reopened its link every few minutes. With 748543c a verb added to the node-engine seat needs no grants release first. Group with mesh-sdk fix/engine-seat-any-verb.
2026-10-31 01:14:33 +01:00
jschoubben 7235e4fb92 Keep a record ask the node's have waits on, and put no value a bucket already holds, so a node that composes its own is asked once and hears no want that did not change (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@fd62a36041be
mesh/check checking 7235e4fb921d on fd62a36041be
On novox the asks of postgres, keycloak and mongodb were judged stale against the controller's compositions, taken away and made again under a new id every pass (8 s), so no answer could count; and every put is a change the node composes again on.
Quick fix: the stale check keeps a record ask whose digest the node's have waits on; the pass skips a put whose value the bucket holds.
Thorough fix beside it (not here, another branch owns the file): the node-engine's composer composes again only when a want's value changed.
2026-10-31 01:09:13 +01:00
mesh-tools e559798972 Merge pull request 'Answer the controller's lease, what still depends on it and the lease write's check in one call (issue 774)' (#120) from feat/lease-tool into main 2026-10-30 22:49:20 +00:00
jschoubben 2dd5239726 Refuse the lease write's check when the indexer's manifests were not all read (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@e559798972b4
mesh/check being carried to 2dd5239726ca on e559798972b4 | from d409bdf20f48
Review of 09791f5: with the indexer's modules bucket readable but empty, nothing was read, so
nothing was found and both phases passed; the operator could have handed the lease away while an
unkept read still stood. Judge now refuses when no manifest was read, or fewer of the indexer's
modules than it holds (a provided manifest does not count for a missing one), saying why; a
missing or refused manifest is named as before, now under test.

The controller's grants leave the keep list: ADR 0571 §9 has no manifest read them from L7's
delivery, changing ADR 0567 §10 and ADR 0547 §5's shadow compare, so the assigner's and the grants
part's reads are refused, and a refusal names the record that ended a declaration where one does.

Quick fix: the counts in the grants part's Judge, as here. Thorough fix: the same coverage rule
in the shared mesh-sdk derivation once the keep list moves there; the quick one stands until then.
2026-10-30 23:41:11 +01:00
mesh-tools d409bdf20f Merge pull request 'The assigner waits only for provisions the composition keys (issue 774)' (#119) from fix/one-rule-for-provisions into main 2026-10-30 22:33:50 +00:00
jschoubben 09791f5a82 Answer the controller's lease, what still depends on it and the lease write's check in one call (issue 774)
The operator asked to remove the controller by finishing its hand-over; no tool said who holds
the lease, which manifests still read, invoke or consume mesh-controller, or whether the lease
write's check (ADR 0571 §13) would pass. grants_lease_check answers all three from bus state on
the grants part, which already reads the lease, every manifest at the indexer's trunk and the
bus's streams: the holder and what follows from it, every dependency with the record on the
named keep list that keeps it, the controller's streams and buckets someone consumes, and the
check run dry at the write (the three kept to it alone) and before its delivery, with reasons.

Quick fix: the judgement and the keep list in the grants part, as here.
Thorough fix: the keep list and the dependency derivation in mesh-sdk, shared by this tool and
the §13 manifest tests in mesh-catalog, mesh-delivery and mesh-tools. Chosen quick: the tests do
not exist yet and an sdk move needs a group; the move is named for the lead.
2026-10-30 23:30:09 +01:00
jschoubben 6298d9d67c Pin mesh-sdk at 6c5a8ef893b4, the one rule of what is keyed with its compose row (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@d409bdf20f48
mesh/delivery-group group fix/one-rule-for-provisions merged: every member is merged
mesh/check being carried to 6298d9d67c83 on 52a1570b3e3c | from mesh-sdk b74e23f973ab | novox/mesh-delivery#119 in group fix/one-rule-for-provisions
2026-10-30 22:29:35 +01:00
jschoubben 83391dba1b Wait in rule 8 only for the provisions the composition keys (issue 774)
Rule 8 asked a key for every `requires`, and the plans key only a handed module's needs and what a helper answers:
grafana waited on ace for oidc-client, which the controller still makes, and mesh-build-go for go-toolchain on its own
machine, for ever. MissingProvisions now takes catalogue.Awaited, the one rule the plan composition and mesh-provisioner
key by, judged against the providers placed somewhere.

Quick fix: drop the missing-key check. Thorough fix (chosen): the shared rule in mesh-sdk, so the three parts cannot
drift again. Needs mesh-sdk at 30098229b831.
2026-10-30 22:29:25 +01:00
mesh-tools 52a1570b3e Merge pull request 'The assigner calls the controller only with the arguments its contract declares, and a refused call is a condition (issue 774)' (#118) from fix/held-module-and-assigner-contract into main 2026-10-30 21:17:41 +00:00
jschoubben 8fe6442c47 Clear a refused pin's condition once the controller's pins agree, and take over a previous process's conditions only once the findings are read (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@52a1570b3e3c
mesh/delivery-group group fix/held-module-and-assigner-contract merged: every member is merged
mesh/check checking 8fe6442c4754 on c104e7500d21 | novox/mesh-delivery#118 in group fix/held-module-and-assigner-contract
A pin made by hand, as the condition's Needs asks, changed nothing the assigner looked at, so its urgent condition
never cleared, and a restart left every one raised. The assigner now reads the controller's pins (placement-export,
asked again only after the controller's compositions change, never on a timer) and clears a pin's condition once
they agree with the placement's after being seen differing, as a module's place already did. The first pass takes
over the call conditions a previous process raised, pins and places alike, and retries at the next pass when the
findings cannot be read.
2026-10-30 21:27:35 +01:00
jschoubben 1fef458c5f Send the controller only the arguments its contract declares, and make a refused call a condition (issue 774)
The assigner added "why" to every assign, unassign, pin and unpin since 2026-10-28; the controller's seat contract
takes no such argument, so it refused each one and did nothing. No approved placement reached the controller's
assignments, no bus account was minted for a newly placed module, and the refusal was only a journal line.

Quick fix: drop "why" from ControllerCalls. Thorough fix: a contract test that every argument ControllerCalls gives,
for every verb, is declared (and every required one given) in catalogue.ControllerVerbs, plus an urgent condition per
pair whose call was refused (verb, node, module, the controller's words), cleared when a later call is answered or the
two sources agree. Chosen: both here; a caller-side check of arguments against the seat's contract for every seat
call belongs in mesh-sdk and is noted for the lead. The warrant stays in the assigner's own record (placement-changed,
the hand-act log).

Every module pins mesh-sdk eff81f1220e2 (in review).
2026-10-30 21:23:39 +01:00
jschoubben c104e7500d Merge pull request 'The grants part writes its list whole however large, one refused principal no longer freezes it, and stale watchman keys close (issue 774)' (#117) from fix/sweep-2026-10-30 into main
Reviewed-on: #117
2026-10-30 18:13:00 +00:00
jschoubben e0b56c83f9 Pin mesh-sdk a3718126, whose bus retry never asks again a call its callee answered (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c104e7500d21
mesh/delivery-group group fix/sweep-2026-10-30 merged: every member is merged
mesh/check checking e0b56c83f94c on 33973d533825 | novox/mesh-delivery#117 in group fix/sweep-2026-10-30
2026-10-30 18:43:13 +01:00
jschoubben d4ac50c4b1 Pin mesh-sdk 206f794e, the fix group's final library head (issue 774) 2026-10-30 18:37:07 +01:00
jschoubben f756731700 Pin mesh-sdk ef1ec1c9, the fix group's reviewed library head (issue 774) 2026-10-30 18:27:08 +01:00
jschoubben 255f8b2aca Close a stuck key whose watched key aged out or was deleted unheard, so its condition clears (issue 774)
A key that ages out of its bucket (the build seats' asks pass the stream's 168 h) leaves no marker a watch hears,
and one deleted while the watchman was down is not heard either: its stuck key stood unmoved for ever, and 26
stuck-key conditions of asks from 2026-10-21 were open on 10-30.

Quick fix: mark stuck keys older than the stream's maximum age moved.
Thorough fix, chosen because it covers every bucket the watchman mirrors and any way a key goes: at the start's
end, a stuck key not moved whose bucket was read whole at this start (each build seat's steps, the assigner's
wants) and whose key was not read standing is written moved, which clears its finding, and the closing is said
in the log. A bucket that could not be read closes nothing. A key that ages out while the watchman runs is
closed at its next start: a KV age-out emits no event, and no timer is added.
2026-10-30 18:25:56 +01:00
jschoubben ff58b16cf9 Pin mesh-sdk 61b67d61, so mesh-grants writes through the size guard and every delivery module rides out a user-list reload (issue 774) 2026-10-30 18:23:08 +01:00
jschoubben 788969bcf8 Write a kept user through mesh-sdk's AccountsEntry, so its lines are ComposeAccounts' own (issue 774)
The grants part repeated the composer's entry lines to keep a refused principal's grants of the list in force;
mesh-sdk 61b67d6 exports them as buscompose.AccountsEntry, so the copy goes. The byte-for-byte test of the kept
entry still passes. Needs the mesh-sdk pin at 61b67d6 or later.
2026-10-30 18:20:28 +01:00
jschoubben ce054dc8dd Judge nothing and owe no status for a candidate outside the mesh (issue 774)
The forge carries the operator's own repositories (jschoubben/hal, jschoubben/novadock) beside the mesh's; the
indexer writes their pull requests as candidates, the gate never names their trunk, so the checker said each
"cannot be judged" and owed the status error on mesh/check, which the forge module wrote into repositories the
mesh's forge account may not write: 403, 119 times each (sweep 2026-10-30 item 7).

Quick fix: name the two repositories as not the mesh's in the checker.
Thorough fix, chosen because any repository on the forge can meet it: a candidate every pull request of which is
of a repository the gate's source.main names no trunk of and touches no module is outside the mesh. The checker
writes it no verdict and owes it no status, takes away a status owed before, says it once in the log, and lists it
in checker_status as outside_the_mesh. A new mesh repository's first pull request adds a module, so it is still
said untrunked and owed error (issue 1422). Nothing is outside before the gate's source is read.
2026-10-30 18:12:33 +01:00
jschoubben 94f555277c Never name the private network the mesh provides as a module missing its grant (issue 774)
mesh-wireguard is provided by the mesh itself (mesh-sdk delivery.ProvidedByMesh): no repository builds it, so the
indexer holds no fingerprint of it and the assigner never wants it by one; the controller composes it with the
node's own. The grants part named it "no fingerprint of the new form: the indexer holds no value of it" on every
pass, a warning nothing could clear (sweep 2026-10-30 item 6).

Quick fix: drop mesh-wireguard by name from the missing list.
Thorough fix, chosen as cheap and covering every module the mesh may provide later: MissingGrants passes over any
module ProvidedByMesh names. The controller says the same of it in mesh-control (its grants and switch lines).
2026-10-30 18:10:55 +01:00
jschoubben 031cfb8277 Compose the user list principal by principal, so one refused principal never freezes it (issue 774)
Met twice on 2026-10-30 (the engine's acts, then mesh-host_health): the composer refused one principal as a
second writer, and that refusal ended the whole composition, so the list in force stood for every principal
while changes piled up; the next list that composed moved 52 users at once (16:37:28).

Quick fix: leave the refused principal out and write the rest (locks that principal out of the bus).
Thorough fix, chosen because the kind was met twice and it hides every other change: ComposeEach composes each
principal on its own; a refused one keeps its grants of the list in force (the part's sealed grants), written as
the composer writes an entry; one the list in force does not carry stays out; a list that would drop one the
list in force carries, whose grants were not kept, is held, said. The refusal is raised as the warning
mesh-grants.principal-refused, by name and why, cleared once it composes, and listed in the pass
(principals_refused). The entry renderer mirrors buscompose.ComposeAccounts's lines and is pinned to them by the
test (the entry the bus loaded is written back byte for byte); it belongs in mesh-sdk as an exported entry writer.
2026-10-30 18:10:26 +01:00
jschoubben c1b28cc4fe Keep each sealed list's permissions one key per user, so no value outgrows the bucket (issue 774)
The sealed-grants value held every user of the last two lists in one value; at 61 users it was refused with
"message size exceeds maximum" (JetStream 10054, 16:37 and 16:55 on 2026-10-30), so the permissions of the
list in force were never kept and a grant judged against them waited.

Quick fix: keep only the newest list in the one value (halves it; it still grows with the mesh).
Thorough fix, chosen because the value grows with every placed module and the kind was met twice: one key per
user and list, sealed-grants.<digest hex>.<user>, and an index sealed-lists naming the kept lists and their users,
written only once every user's key is there; the third list's keys are taken away, and the old one-value key read
as a fallback and removed. mesh-sdk's kvlimit guard (7b02f63) refuses an oversized value before the put through
process.State.Put, which the part's own store (cmd/mesh-grants ownStore.Put) writes through once pinned.
2026-10-30 18:07:26 +01:00
jschoubben 33973d5338 Merge pull request 'mesh-grants grants each provisioner client every machine's plans (issue 774)' (#116) from fix/provisioner-reads-every-plan into main
Reviewed-on: #116
2026-10-30 15:53:41 +00:00
jschoubben f800d672f3 mesh-grants grants each provisioner client every machine's plans through mesh-sdk af66ccb, so g14 and shanks read their plans again (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@33973d533825
mesh/delivery-group group fix/provisioner-reads-every-plan merged: every member is merged
mesh/check being carried to f800d672f344 on 98594c990287 | from mesh-sdk 42988a734b3d
2026-10-30 16:41:02 +01:00
jschoubben 98594c9902 Merge pull request 'mesh-grants composes the user list without granting the engine's state to a tool runner (issue 774)' (#115) from fix/runner-carries-no-engine-state into main
Reviewed-on: #115
2026-10-30 15:35:58 +00:00
jschoubben 57193ab2e4 Pin mesh-grants to the mesh-sdk that composes no user from the node-engine's manifest, so the whole user list composes again (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@98594c990287
mesh/delivery-group group fix/runner-carries-no-engine-state merged: every member is merged
mesh/check checking 57193ab2e474 on f7cd4698db7e | novox/mesh-delivery#115 in group fix/runner-carries-no-engine-state
Second meeting of the engine-state-in-runner kind; thorough: the runner
never carries the engine module. The live placement names mesh-host,
and with its health state the runner was a second writer, so every
grant since 13:54 was held back. The seeded-placement test now carries
the engine's manifest and refuses any runner write of mesh-host_*.
2026-10-30 16:32:43 +01:00
jschoubben f7cd4698db Merge pull request 'mesh-grants writes its memberships bare through the mesh-sdk codec (issue 774)' (#114) from fix/memberships-shape into main
Reviewed-on: #114
2026-10-30 14:42:13 +00:00
jschoubben 180a3fe5db Pin mesh-sdk at its fix/memberships-shape head after review round 1 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@f7cd4698db7e
mesh/delivery-group group fix/memberships-shape merged: every member is merged
mesh/check checking 180a3fe5db30 on ab210fc5ab40 | novox/mesh-delivery#114 in group fix/memberships-shape
The decoder now refuses an old wrapper of a held-back membership; grants_membership says so for such a key.
2026-10-30 15:36:48 +01:00
jschoubben 4bb1970fef Write the grants part's memberships bare, through mesh-sdk's codec, so the tool runner serves again
The state memberships held the part's wrapper ({node, module, digest, membership, published, retired}) while every
reader takes the bare membership the assignments stream carries: novox's tool runner decoded an empty one and served
no tool. The part now writes membership.EncodeGrantsMembership of a membership published or unchanged, never one held
back. The pass record moves to the pass the part already keeps (user-list at pass): published or why not was there, the
retired seat verbs are added as Result.Retired; grants_membership reads both, so no new state is declared.

Issue 774 (the bus login list, L3). Quick fix: marshal m.Membership here. Thorough, chosen: one codec in mesh-sdk the
writer and the tool runner share, because a hand-built fixture agreed with the mistake. Pins mesh-sdk at the head of
its fix/memberships-shape.
2026-10-30 15:27:06 +01:00
mesh-tools ab210fc5ab Merge pull request 'Design 47's last milestones: watchman, grants gate, assigner wants and placement, builder log, view (issues 774, 1952, 1953, 1954, 1955)' (#113) from feat/774-l3-reland into main 2026-10-30 11:02:07 +00:00
jschoubben 8edc83950c Pin mesh-sdk 2da0de7, the release's sdk with main merged in (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ab210fc5ab40
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
mesh/check checking 8edc83950cea on 5588875c3f21 | novox/mesh-delivery#113 in group feat/774-l3-reland
2026-10-30 01:12:57 +01:00
jschoubben 8e470f3af4 Merge origin/main (delivery 1, #112) into the release, keeping the release's mesh-sdk pins (issue 774) 2026-10-30 01:12:57 +01:00
mesh-tools 5588875c3f Merge pull request 'Pin the parse-only mesh-sdk in the assigner, indexer, checker, builder and grants part, so they read design 47's manifests (issue 774)' (#112) from fix/774-new-words-first into main 2026-10-30 00:10:15 +00:00
jschoubben a2dd7a5c37 Pin the parse-only mesh-sdk 78c9ec8 in place of 6e4a301, so delivery 1 changes nothing but what a manifest may say, and keep main's checker test as it is (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@5588875c3f21
mesh/delivery-group group fix/774-new-words-first merged: every member is merged
mesh/check pass a2dd7a5c37ef on 883d976b5f00 | unit 9174 run, 546 kept, 51 skipped | integration 9952 run, 659 kept, 60 skipped | over bound: unit, in…
2026-10-30 00:38:11 +01:00
jschoubben 7536d9d14e Pin mesh-sdk 6e4a301, the release's final head, which keeps the old provisioner client's needs grant for one release (issue 774) 2026-10-30 00:18:04 +01:00
jschoubben 9b60eda451 Pin mesh-sdk 6e4a301 in the checker, and hold that the change's own failing test ends its level, as that sdk's build asks do (issue 774, issue 1289) 2026-10-30 00:17:24 +01:00
jschoubben 541ad1d2e2 Pin mesh-sdk 6e4a301 in the modules that parse manifests, so the running assigner, indexer, builder and grants part read the release's new manifest words before the release itself is checked (issue 774) 2026-10-30 00:17:24 +01:00
jschoubben ce7032c49b Pin mesh-sdk 007b6fd, the release's final head (issue 774) 2026-10-30 00:05:25 +01:00
jschoubben 08ecf10ef3 Grants: write withheld hearings in mesh-sdk's layout, so the self-check reads them by the sdk's names (issue 1955) 2026-10-29 23:56:21 +01:00
jschoubben ba502479e8 Builder: ask and hear the seat mesh-artifact-sweep, the name mesh-sdk 7bd0f6c gives the store's sweep, so its held and artifacts-gone reach their holder (issue 1955, ADR 0571 §2) 2026-10-29 23:56:21 +01:00
jschoubben e564eaf75f Grants: write the hearings it withholds to its state withheld on change, so the self-check's probe D-heard reads them from the bus and not the controller's store (issue 1955) 2026-10-29 23:51:48 +01:00
jschoubben a54d1acf45 View: answer builder_history from the builder's log, each build's end with its duration from its start (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben e82ea89c23 Assigner: read the running commit of a reader of the manifests from the builder's log, and hold the buckets it starts without in one list so the start test skips them (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 06b58bc83b Indexer: take directories and outside-built registrations from the builder's log, watched from its start, so no build record of the controller's is read (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 17c0734fb2 Builder: keep each build's start and end in its own module log build-log, and answer builder_builds_since from it, so readers and mesh-catalog read the builder's history without a bare stream (issue 1955, ADR 0571 §1, §9) 2026-10-29 23:48:14 +01:00
jschoubben c045057ca0 Assigner: write who holds which seat from placement into seat-holders, only the keys a change moves, and answer it from placement's seats, so readers no longer ask the controller's seats (issue 1955, ADR 0571 §4)
Pins mesh-sdk f9a496d.
2026-10-29 23:14:28 +01:00
jschoubben b44f01f89c Builder: write each build's start and end to its own log on the bus, and ask the seat artifact-store's held and hear its artifacts-gone, the controller's only until the lease write, so build history and the store's sweep leave the controller (issue 1955, ADR 0571 §1, §2)
Pins mesh-sdk f9a496d.
2026-10-29 23:13:32 +01:00
jschoubben 229a3e9855 Assigner: from the lease write rule 7 reads a placed node's own composition digest from its have, an unplaced node gets no want and is said, and the controller's compositions are read only until then (issue 1955, ADR 0571 §6) 2026-10-29 23:10:31 +01:00
jschoubben e5192f55ee View: read no build record and no composition of the controller's: fingerprints against the builder's done keys alone, a move's judgement from the node agent's have (issue 1955, ADR 0571 §1, §6) 2026-10-29 23:08:46 +01:00
jschoubben ee2f1efe78 Watchman: hold a failing network part at the machines it points at, as the controller does, so a machine others cannot reach is said once there and not as each reporter's own (issue 774)
Pins mesh-sdk 6eae076.
2026-10-29 23:06:19 +01:00
jschoubben 8a91871de6 Watchman: raise machine-network, network-rewritten, network-unreachable, machine-units, relogin-needed and directory-used-as-found from each node's own health report, under the controller's keys, so these alerts outlive the controller's lease (issue 774)
Pins mesh-sdk 94d0de5.
2026-10-29 23:01:03 +01:00
jschoubben 64a5074d58 Assigner: lay each held value of the group's own pull requests at their heads over its module's values in the composition check, judged against the head's declaration and named, so a check reads the held values itself (issue 1953 D7, ADR 0565 §5) 2026-10-29 22:43:27 +01:00
jschoubben 7b7796b6fa Assigner: hold a want while a provision its module requires has no key yet, and say it once it stays without one past the delivery bound, so a module never starts without its database unseen (issue 1954 D1)
Pins mesh-sdk 66ff33d.
2026-10-29 22:41:44 +01:00
jschoubben f56eb62d8b Assigner: hold a want until every provision its module consumes on that node is provided, and say a failed one as a finding, so a module never moves ahead of what it needs (issue 1954 D1, ADR 0570 §4)
Pins mesh-sdk 6e39ed1.
2026-10-29 22:39:38 +01:00
jschoubben 78f03b7422 Watchman: say needs-operator from mesh-provisioner's provisions in waits, so the row the haves could not carry has its facts on the bus (issue 1954 D2)
Pins mesh-sdk 4a96793.
2026-10-29 22:37:39 +01:00
jschoubben 3aec7f9b02 Assigner: hold the seat mesh-wants, keep a person's upgrade policy and judge moves by it, say a policy split with the controller as a finding, and answer the bus step, so the policies and the bus step leave the controller (issue 1953 D4-D6) 2026-10-29 22:34:08 +01:00
jschoubben 2488d839ab Bridge, indexer and view: carry the catalogue's predefined list on its merged message, and keep it and each open pull request's config declarations in the indexer's states, so mesh-config reads them in place of the controller's mirror and pull-declares (issue 1953 D1-D3)
The view's tools read both states. Pins mesh-sdk 2d2aa7b.
2026-10-29 22:29:04 +01:00
jschoubben 2531cd42b6 Checker: hold that the change's own failing test ends its level there, as mesh-sdk's build asks now do, so a failing check costs only its own tests (issue 1289)
Pins mesh-sdk 2d2aa7b.
2026-10-29 22:24:50 +01:00
jschoubben 5235bb2e79 Assigner and grants part: record hand acts and warrants through the seat mesh-hand-acts, the controller's log only while that seat has no holder and the controller holds its lease (issue 1953 D8)
Pins mesh-sdk 2d2aa7b in the L5 modules.
2026-10-29 22:24:00 +01:00
jschoubben 3b6f3af38e Grants part: create a module's log with the max age it declares, so a log's entries age out as its manifest says (issue 1952)
Its nats fixture gives the tools bundle's account a secret the tool runtime can reach, as mesh-sdk 2e5cc4f asks. Pins mesh-sdk 2e5cc4f.
2026-10-29 22:19:35 +01:00
jschoubben 15a15cbda5 Watchman: observe S1, S9 and S21 as the controller does and stop the needs-operator row, so the keeper's compare can empty with no condition lost (issue 1952 WM2, WM3, WM5)
S1 is urgent after half an hour for a control node (the node running mesh-controller). S9 also hears the bus module's
slow consumers and closed connections, each closure counted once. S21 is one finding per heard machine from its failed
haves and its provisioner clients' reports (PROV_APPLIED). DEAD_LETTERS is counted again at a letter's age and every
five minutes while letters are held, on the watchman's own pulse. needs-operator waits for L6: its inputs are not on the
bus. Pins mesh-sdk b6ef01d.
2026-10-29 21:59:46 +01:00
jschoubben 7dbf7a19a1 Grants part: compose a principal for every own bus account, a tools bundle's with what it may remove, so the bus tools remove the controller's dead-letter consumer and the engine's acts on their own account (issue 1952 S5)
Pins mesh-sdk 7000bd5.
2026-10-29 21:56:13 +01:00
jschoubben e1fd3eb0f6 View: answer status from the buckets it reads and derive motion with mesh-sdk's MotionOf, so status leaves the controller and the keeper judges a delivery as the view does (issue 1952 V1) 2026-10-29 21:52:42 +01:00
jschoubben 73280f85c3 Assigner: write each condition it says as a finding too, and say it straight only until the take-over, so its holds reach the operator through the condition keeper and never twice (issue 1952 WM6) 2026-10-29 21:49:25 +01:00
jschoubben b8e8981730 Grants part: hear root-not-free from the seat the take-over key names, never both, so the switch to the condition keeper moves it with the router (issue 1952 G1)
Watchman: subscribe exactly what its grant lets it hear (buscompose.WatchmanHears). Both pin mesh-sdk 69fcb6c.
2026-10-29 21:46:29 +01:00
jschoubben da787ae360 Watchman: hold mesh-watchdogs and write its watchdogs as findings, so health can leave the controller (issue 1952 WM1-WM5)
The signals rows S1, S8, S9, S10, S11, S21 and the haves' unhealthy and needs-operator are pulses whose expiry is a finding;
stuck keys and presence are its own findings; it keeps the dead letters from the lease write on its own consumer, lists and
shows them, and refuses deliver and drop until the hand-act log's home (issue 1953). Tools read every state it keeps and
start resync. Pins mesh-sdk 2a04bf0.
2026-10-29 21:44:18 +01:00
jschoubben 013b7420f6 mesh-grants: hold a grant only while the loaded list lacks its permissions, read from the part's own record of the lists it sealed, so a release whose bus tools are down still moves (issue 774) 2026-10-29 21:18:21 +01:00
jschoubben c7fbc530da mesh-grants: compose the user list from the running and the trunk manifest, and write a grant only once the loaded list carries it, so L3's parts never use a subject the bus has not loaded (issue 774) 2026-10-29 21:15:55 +01:00
jschoubben 98ad1aa252 Pin mesh-sdk 22c9ddd in the modules L3 changes, the L3 re-land group's sdk head (issue 774) 2026-10-29 21:07:00 +01:00
jschoubben b01cf63237 Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#109) from revert/774-l3 into main"
This reverts commit c1b4533201, reversing
changes made to d8274204fb.
2026-10-29 21:05:14 +01:00
mesh-tools 883d976b5f Merge pull request 'Builder: announce a done build again, so the controller composes a module whose composition and grant were lost (recovery of novox and ace, 2026-10-29)' (#111) from fix/announce-done-again into main 2026-10-29 19:47:58 +00:00
jschoubben ba4bfea0a4 Remove the PR description draft from the branch; it is kept outside the repository
mesh/delivery merged: source.main moved to novox/mesh-delivery@883d976b5f00
mesh/check checking ba4bfea0a407 on 883d976b5f00
2026-10-29 17:59:28 +01:00
jschoubben fb7ad5cd73 Add builder_announce_again so a done build can be taken in and composed again
Recovery of photos and link2pay on novox, 2026-10-29: their composition and grant were lost and no tool wrote a done key again.
Quick fix: rewrite the done key unchanged as a new revision. Thorough fix: ADR 0547, the node agent composes its own and the
assigner stops waiting on controller compositions (design 47).
2026-10-29 17:58:39 +01:00
mesh-tools 5e931eac19 Merge pull request 'The assigner never keeps a want back because a provider is unhealthy, and the canary gates only modules that run (issue 1960)' (#110) from fix/1960-no-hold-spread into main 2026-10-29 15:35:28 +00:00
jschoubben d54eab4052 Never keep a want back for an unhealthy provider, nor a first want for the canary where nothing runs: the module is wanted and its provider down is said (issue 1960)
mesh/delivery merged: source.main moved to novox/mesh-delivery@5e931eac191a
mesh/delivery-group group fix/1960-no-hold-spread merged: every member is merged
mesh/check pass d54eab4052d1 on c1b4533201c8 | unit 3318 run, 1951 kept, 38 skipped | integration 3994 run, 2125 kept, 38 skipped | over bound: unit, …
On novox on 2026-10-29 the assigner wrote no want for de-spiegel and link2pay (rule 5: no provider of route runs
healthy) nor photos (rule 5: no provider of s3-bucket), and none where no node of a module was online (rule 3). A
module with no want is on no list and nothing applies it; a site runs while the proxy in front of it is down.

- Quick fix: skip rules 3 and 5 for the three modules' first wants. Leaves rule 5 holding every move and the
  canary holding new assignments.
- Thorough fix, chosen: rule 5 holds no want; it is said per node (Outcome.Unreached, journaled once per words by the
  pass). The canary (rule 3) no longer holds the first want of a node that does not run the module. A node that runs
  it keeps the canary order: at an older composed and granted fingerprint of the new form its first want stays at what
  it runs; at an old-form build it waits for the canary as before (issue 1945); at an older fingerprint with no
  composition or grant it gets F. This departs from ADR 0429 §6, which makes every provider healthy a rule of every want.

Tests: rows changed in the existing rule tests, and rows for today's modules (route-proxy down: de-spiegel and
link2pay wanted; minio not healthy: photos moves; novox offline: a first want written, a move kept back).
2026-10-29 16:04:39 +01:00
mesh-tools c1b4533201 Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#109) from revert/774-l3 into main 2026-10-29 08:59:55 +00:00
jschoubben 6771db22ae Revert L3 (mesh-delivery, merge d827420): its node-engine removed every held module on novox, the bus server among them, at 08:54 CET on 2026-10-29 (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c1b4533201c8
mesh/delivery-group group revert/774-l3 merged: every member is merged
mesh/check being carried to 6771db22ae69 on d8274204fbed | from mesh-catalog 1555241393fe | novox/mesh-delivery#109 in group revert/774-l3
L3 lands again with two fixes: a held module is never removed, and the grants cover L3's new subjects.
2026-10-29 09:42:29 +01:00
mesh-tools d8274204fb Merge pull request 'L3: mesh-grants writes the wanted accounts and grants, the checker's try seat, placement alone after the lease (issue 774, ADR 0561), rebased onto L2' (#105) from feat/774-vault-mints-l2 into main 2026-10-29 07:44:27 +00:00
jschoubben 17c79f25f0 Merge remote-tracking branch 'origin/main' into l3-md-main
mesh/delivery merged: source.main moved to novox/mesh-delivery@d8274204fbed
mesh/delivery-group group feat/774-vault-mints-l2 merged: every member is merged
mesh/check pass 17c79f25f047 on d4cbfc135918 | unit 1601 run, 5122 kept, 47 skipped | integration 1637 run, 5215 kept, 49 skipped | over bound: unit, …
2026-10-29 02:15:27 +01:00
mesh-tools d4cbfc1359 Merge pull request 'Assigner pins mesh-sdk 76f7c4d, which defines the seat mesh-delivery-checker, so L3's check can compose (issue 774)' (#108) from fix/assigner-pin-checker-seat into main 2026-10-28 23:59:03 +00:00
jschoubben e0af965d8b Assigner pins mesh-sdk 76f7c4d, which defines the seat mesh-delivery-checker: the check composes with the running assigner's mesh-sdk, so L3's checker claim is refused until this runs (issue 774, kind of issue 1585)
mesh/delivery merged: source.main moved to novox/mesh-delivery@d4cbfc135918
mesh/check being carried to e0af965d8b40 on d4cbfc135918 | from fa075dc880af
2026-10-29 00:35:07 +01:00
jschoubben 64d364a768 Assigner pins mesh-sdk 76f7c4d, a pull request's head holding 8eafc0c and 77b48a4: the checker refuses a pin to a merge commit on main (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 ready: every member ready, and composed together they pass
mesh/delivery closed: a newer head of the same pull request
mesh/check pass 64d364a7686c on d4cbfc135918 | unit 1319 run, 5401 kept, 47 skipped | integration 1359 run, 5490 kept, 49 skipped | over bound: unit, …
2026-10-29 00:20:03 +01:00
jschoubben 2caaa6fc92 Merge main into L3: the assigner pins mesh-sdk main d747c11, which holds both L3's sdk and the seed's held-as-today fix (issue 774)
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group feat/774-vault-mints-l2 checking: 3 of 4 member(s) ready
mesh/check failed: ace: the check of novox/mesh-host#168 failed: the pin of git.novox.be/novox/mesh-sdk/go at commit d4c61ee8ffd0 is on no pull reques…
2026-10-28 23:50:43 +01:00
mesh-tools fa075dc880 Merge pull request 'Assigner: pin mesh-sdk 06b13ec, so a seed's brief shows what stays held as today (issue 1126)' (#107) from fix/1126-seed-held into main 2026-10-28 22:20:54 +00:00
jschoubben f6df07605d Merge remote-tracking branch 'origin/main' into l3r/774-vault-mints
mesh/delivery-group group feat/774-vault-mints-l2 rejected: composed together: the checker's verdict at the members' heads (1c492872d414): shanks
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#105 does not merge into the trunk: files in conflict: modules/mesh-delivery-assigner/go.mod, modules/me…
# Conflicts:
#	modules/mesh-delivery-assigner/cmd/mesh-delivery-assigner/seat_tools_test.go
2026-10-28 23:00:04 +01:00
jschoubben fc3883b0b6 Assigner: pin mesh-sdk 77b48a4, so the seed's Approve names the differences it carries (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@fa075dc880af
mesh/delivery-group group fix/1126-seed-held merged: every member is merged
mesh/check pass fc3883b0b650 on 5b2e5bc53776 | unit 1199 run, 1015 kept, 33 skipped | integration 1218 run, 1081 kept, 26 skipped | over bound: unit (…
2026-10-28 22:41:03 +01:00
mesh-tools 5b2e5bc537 Merge pull request 'The checker writes a loud verdict where it cannot judge, so no Approve waits silently (issue 1733)' (#104) from fix/1733-no-hour into main 2026-10-28 21:29:56 +00:00
jschoubben 25953c6ae1 Assigner: pin mesh-sdk 06b13ec, so a seed's brief shows what stays held as today (issue 1126)
mesh/delivery-group group fix/1126-seed-held checking: 2 of 3 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 25953c6ae110 on 5b2e5bc53776 | novox/mesh-delivery#107 in group fix/1126-seed-held
2026-10-28 22:18:31 +01:00
mesh-tools 38f89c6976 Merge pull request 'The assigner serves the seat mesh-placement's verbs on the seat's own name (issue 1126)' (#106) from fix/1126-seat-names into main 2026-10-28 21:14:13 +00:00
jschoubben 49ea5c96e3 Assigner: serve the seat mesh-placement's verbs as <seat>.<verb>, the name the tool runner serves a seat's verb from (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@38f89c6976c9
mesh/check being carried to 49ea5c96e390 on 38f89c6976c9 | from 1f6f4eda3208
2026-10-28 22:07:05 +01:00
jschoubben 7692f201df Pin mesh-sdk 8eafc0c, the L3 group's head on mesh-sdk main with #120 and #121 (issue 774)
mesh/delivery-group group feat/774-vault-mints-l2 checking: the checker's verdict is out of date: novox/mesh-sdk@8eafc0c5ce3c was judged on trunk 53f26493697e,…
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 7692f201df51 on 5b2e5bc53776 | novox/mesh-delivery#105 in group feat/774-vault-mints-l2
2026-10-28 21:56:06 +01:00
jschoubben 09da8d741e The assigner's laying again and its seed again know the lease write too, so a proposal laid again after it is judged as its first laying was (issue 774, ADR 0567 §8; re-review) 2026-10-28 21:54:47 +01:00
jschoubben b2351a213d Follow L2 under L3: the builder, checker, indexer and view read the seat mesh-placement's placement, the seat's Leased is the lease handed away, and the view keeps no read of the indexer's placement L2 removed (issue 774, ADR 0567 §8, ADR 0549 §3) 2026-10-28 21:54:47 +01:00
jschoubben 2abceb1913 Pin mesh-sdk 4e6e5cc, the L3 group's head rebased on L2, in the six modules the group changes (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 685810720e mesh-grants: place by the catalogue's placement alone once the controller's lease is handed away, and refuse any other reader of the assignments in this repository (issue 774, ADR 0549 §3, §6)
Why: after the lease write the assignments are frozen, so the part's union kept a module the catalogue takes off a node
with its account, grant, membership and user-list entry. The lease is read as the assigner reads it
(buscompose.LeaseHandedAway); before it, or with no placement on the bus, the union stands. The read stays declared until
the first change after the lease write, named in modules/assignments-readers with the assigner; a TestGrants row refuses
any other module.json that reads mesh-controller.assignments, as mesh-catalog's checks/placement does for the catalogue.
2026-10-28 21:54:47 +01:00
jschoubben 8e25926e91 mesh-grants: the node-engine's manifest is placed on every node, so its declared state is asserted and created as a placed module's, and it adds no account (issue 774, ADR 0563 §2, §3) 2026-10-28 21:54:47 +01:00
jschoubben 68628bfd78 Pin mesh-sdk 277a5bd, the L3 group's head, in the assigner, builder, checker, indexer, view and grants; the assigner's manifest test names mesh-grants.grants, its integration test no unused import (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 1dbb7c5740 mesh-delivery-view: node_trusts answers in one call whether every node's record names the vault's signing key, the gate before the controller's lease is handed away (issue 774, ADR 0550 §2, design 47 L3 s6b) 2026-10-28 21:54:47 +01:00
jschoubben 9e3d1d5e68 The assigner reads the catalogue's placement alone once the controller's lease is handed away, and reads the lease to know it (issue 774, ADR 0549 §5, §6) 2026-10-28 21:54:47 +01:00
jschoubben 9a70d697af The view shows a row per module the catalogue's placement puts on a node, and reads the controller's assignments no more (issue 774, ADR 0549 §3) 2026-10-28 21:54:47 +01:00
jschoubben a4116a1de7 Builder, checker and indexer count a build node's toolchain only where the catalogue's placement puts its build module, through mesh-sdk's one gate, and read the controller's assignments no more (issue 774, ADR 0549 §3, issue 1940) 2026-10-28 21:54:47 +01:00
jschoubben 6f721dd536 mesh-grants: say every placed module left without a grant, in the pass and as the condition mesh-grants.grant-missing once the lease is handed away (issue 1126, ADR 0547 §5)
Why: the controller's grant writer skipped link2pay, snake, f1-circuits and txt-game in silence (it read the fingerprint by
their unchanged commit, a build of the old form) and the take-in said 'its grant written'. The part already reads the
fingerprint from the indexer's value, so those four are granted at their src2 done key's fingerprint; a row proves it.
What a pass leaves without a grant (no fingerprint of the new form with the indexer's none2, a refused manifest, no
hashed account) is now said rather than passed over. Quick: log it. Thorough (chosen): the pass's Missing, shown by the
grants tool, and a warning condition raised and cleared on change, as consumer-differs is.
2026-10-28 21:54:47 +01:00
jschoubben 354b9ea396 mesh-delivery-assigner: read grants from the source the lease names (issue 774, ADR 0547 §4)
Why: from the lease write the grants part's state grants is the source and the controller's bucket otherwise; the assigner follows the lease key (a pass only when the source changes), and neither bucket stops its start. Needs the group's mesh-sdk pin (GrantSource, LeaseHandedAway, the assigner a named reader of the lease).
2026-10-28 21:54:47 +01:00
jschoubben 3618831992 mesh-grants: write grants.<module>.<fp> in shadow into the state grants, say grant-written, and show every difference from the controller's in grants_grants (issue 774, ADR 0547 §5)
Why: the assigner switches its grants source when the controller's lease is handed away; until then the written keys are compared with the controller's, so the switch rests on a measured equality. Needs the group's mesh-sdk pin (PartGrantKey) before it builds.
2026-10-28 21:54:47 +01:00
jschoubben 843a1589d7 Fold the seat verbs' test into the try tools test; pin the sdk head (issue 774) 2026-10-28 21:54:47 +01:00
jschoubben fc6d5dcbdb mesh-delivery-checker claims its seat and serves try, try-read and try-end by the tools' own code, so the try outlives the controller (ADR 0547 §10, issue 774) 2026-10-28 21:54:47 +01:00
jschoubben 6d709e1fc1 mesh-grants: grants_accept_hashes releases the clause 7 hold once, for exactly the held users of the held list, with why, said as hashes-accepted (issue 774, ADR 0561 §7, §9) 2026-10-28 21:54:35 +01:00
jschoubben eb11b2c844 mesh-grants: hold a list that changes several loaded users' hashes at once with no rotation by the vault naming them, and raise user-list-held naming them (issue 774, ADR 0561 §7) 2026-10-28 21:54:35 +01:00
jschoubben 67e3ebf22f mesh-grants: grants_accounts reads the account consumers' differences in shadow, where no condition is raised (issue 774, ADR 0561 §6, §9) 2026-10-28 21:54:35 +01:00
jschoubben 9b31746cdb mesh-grants: raise a condition when an account consumer differs from its composed configuration, and bring one to it by hand with grants_consumer_match, never under the controller's lease (issue 774, ADR 0561 §6, §9) 2026-10-28 21:54:35 +01:00
jschoubben ebad58f7e2 mesh-grants: create the durable consumers an account needs from the lease write, compared in shadow until then, so a module assigned after the controller leaves still hears what it consumes (issue 774, ADR 0561 §6) 2026-10-28 21:54:35 +01:00
jschoubben 3c6fb7fc9e mesh-grants: compose the user list again when the vault says account-issued, so a minted account reaches the bus within one pass (issue 774, ADR 0561 §1) 2026-10-28 21:54:35 +01:00
jschoubben 71aa3e86c6 mesh-grants: write the bus accounts the mesh wants to a state only the vault reads, so the vault can mint them; grants_accounts reads it (issue 774, ADR 0561 §2, §9) 2026-10-28 21:54:35 +01:00
jschoubben 2d8ad2e66a Checker: no candidate is left without a verdict (issue 1733)
mesh/delivery merged: source.main moved to novox/mesh-delivery@5b2e5bc53776
mesh/delivery-group group fix/1733-no-hour merged: every member is merged
mesh/check pass 2d8ad2e66a8f on 38f89c6976c9 | carried from 1f6f4eda3208 | unit 470 run, 4760 kept, 13 skipped | integration 503 run, 4844 kept, 22 sk…
Once the gate's one-hour ask bound goes (mesh-catalog fix/1733-no-hour), the
gate asks the operator's Approve only on a verdict, so every path that left
a candidate without one now writes one, from an event and with no timer:
- a group with a member whose repository's trunk head is not known (a pull
  request the checker cannot claim) is not judged in part: its verdict fails
  naming that repository, and every status of the group says so (a pass of
  the others would have let it merge with that member never checked);
- the verdict of heads that cannot be judged replaces whatever the key held,
  such as a running verdict left by a check that can no longer run;
- a verdict key deleted is judged again on its delete (the checker watches its
  verdicts), and a check running when its key is deleted writes it anew.

Tests: three rows at the end of the no-trunk test; counts unchanged.
2026-10-28 21:50:03 +01:00
mesh-tools 1f6f4eda32 Merge pull request 'The checker and the builder use it; checker_ask_again asks a lost check again; the checker's integration bed writes a node record (issue 734's kind)' (#103) from fix/lost-check into main 2026-10-28 20:41:01 +00:00
mesh-tools 769815ac0f Merge pull request 'The assigner's seed and mesh-grants take mesh-wireguard as provided by the mesh (issue 1126)' (#102) from fix/1126-provided into main 2026-10-28 20:39:23 +00:00
jschoubben e7888024a8 Checker: delete a check's asks key at its end, so the state asks holds only the checks that run
mesh/delivery merged: source.main moved to novox/mesh-delivery@1f6f4eda3208
mesh/delivery-group group fix/lost-check merged: every member is merged
mesh/check checking e7888024a826 on 769815ac0fea | novox/mesh-delivery#103 in group fix/lost-check
Re-review of fix/lost-check: writing an empty list at the end kept one key
for every verdict key ever checked. Tests: every row of the lost-ask table
proves the key gone at the end; one row more (deaf after its build module
replaced it, asked again by hand); counts unchanged.
2026-10-28 20:39:50 +01:00
jschoubben 2007a79c45 Checker and builder pin mesh-sdk f91340f, which opens a silent steps watch again
mesh/delivery-group group fix/lost-check checking: 1 of 2 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 2007a79c45c3 on a3a0d8f2ceb3 | novox/mesh-delivery#103 in group fix/lost-check
2026-10-28 20:38:31 +01:00
jschoubben d8286d1790 Assigner and grants: take the private network as the mesh's own through mesh-sdk's one predicate, so the seed is not refused for it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@769815ac0fea
mesh/delivery-group group fix/1126-provided merged: every member is merged
mesh/check checking d8286d1790fb on a3a0d8f2ceb3 | novox/mesh-delivery#102 in group fix/1126-provided
2026-10-28 20:38:14 +01:00
jschoubben ca74639a74 Checker integration test: the checker says its seventh state, asks, as its own
The new state asks (checker_ask_again's record of each running check's
asks) is kept and not readable, as hearing is.
2026-10-28 20:37:07 +01:00
jschoubben a58a73bb55 Checker integration tests: give the bed's node a record in the have, so the names check finds a name to ask with (ADR 0567 §12, issue 1126)
Since 9db9499 the checker asks no check while no node record is on the bus; the bed wrote none, so eight integration tests failed on main (PR #101's check). The fake build module now records the names an ask carries, and the first test asserts n1 is named.
2026-10-28 20:36:25 +01:00
jschoubben 16594feff1 checker_ask_again: refuse while an ask of the check waits in its build seat's stream
Review of fix/lost-check: an ask no node took yet has no step key, so
asking again then would run the check twice (issue 1791's kind). The
checker keeps the asks each running check put in a new state asks, and the
tool refuses unless every one of them has a step key. Asking again under the
same ask id is not done: the stream drops a second put of an id within its
de-duplication window and keeps both past it, so the waiting copy is never
replaced.

Tests: the hand row of the lost-ask table refuses while its ask waits;
counts unchanged.
2026-10-28 20:36:25 +01:00
jschoubben cb46ff8735 Checker and builder wait through mesh-sdk's HearEnds, so a deaf watch loses no step end in either
The checker's HeardEnds moved to mesh-sdk buildrun (fix/lost-check,
6aeb401): both modules pin it, and the builder, which waits on the same
Asker, now reads an ask's step at each step end of its build module too.
checker_ask_again and checker_status's lost stay in the checker.

Tests: one row in the builder's ask table (its watch deaf, read at a
nudge, done with no new ask); counts unchanged.
2026-10-28 20:31:04 +01:00
jschoubben 6f079f13a8 Checker: never lose a step's end, and ask a lost check again (issue 734's kind)
mesh-catalog #553's verdict stayed running after its step mvsgthyjed5786
passed on shanks at 19:09:31Z: the watch of that one step key, opened at
18:46:05Z, heard neither its start nor its end and never closed, so the
wait had no bound running and nothing would ever judge it.

- HeardEnds stands between the asker and each ask's watch: at every step
  end of the same build module (one watch per steps state), at a
  reconnect, the asks waited on read their keys now and are judged; a
  watch that closes is opened again, at most 3 times.
- A step the build module's next process ended (RestartReason, a delivery
  replacing it) counts as a lost ask: asked again under the claim, at most
  LostAskTries, then failed as not heard, judged again when the nodes change.
- checker_ask_again {group, why} ends a running verdict as asked again by
  hand; the checker hears its verdicts, ends the wait and claims it anew.
  Refused while a step of its pull requests runs, or for a verdict that
  is not running.
- checker_status names such checks under lost, and each step's what.

Tests: one table replaces the given-up test (rows for given up, replaced,
deaf and nudged, deaf and asked by hand); counts unchanged (147 unit, 10
integration in the checker).
2026-10-28 20:27:41 +01:00
mesh-tools a3a0d8f2ce Merge pull request 'Format two assigner files as gofmt writes them, so mesh-delivery's check passes again (issue 1126)' (#101) from fix/1126-assigner-gofmt into main 2026-10-28 19:08:03 +00:00
jschoubben 5fe8ab11f1 Assigner: format pass.go and placement.go as gofmt writes them, which the check requires (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@a3a0d8f2ceb3
mesh/check checking 5fe8ab11f1b4 on a3a0d8f2ceb3
2026-10-28 20:03:16 +01:00
mesh-tools b6eb43dd62 Merge pull request 'mesh-delivery-builder pins mesh-sdk 38dbe1d, so it accepts the assigner's claim on mesh-placement (issue 1126)' (#100) from fix/1126-builder-pin into main 2026-10-28 19:02:33 +00:00
jschoubben dc4419b5c5 Builder: pin mesh-sdk 38dbe1d, so it knows the seat mesh-placement and builds the assigner that claims it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b6eb43dd6271
mesh/check checking dc4419b5c50f on b6eb43dd6271
2026-10-28 19:57:29 +01:00
mesh-tools 24181c0242 Merge pull request 'The assigner holds the seat mesh-placement; the indexer, bridge and view drop the repository path (issue 1126, delivery 2 of 2)' (#99) from feat/1126-placement-seat into main 2026-10-28 18:33:09 +00:00
mesh-tools db737df597 Merge pull request 'mesh-grants: pin mesh-sdk 38dbe1d, so the bus loads the node agents' reads of the seat's states first (issue 1126, delivery 1 of 2)' (#98) from feat/1126-grants-first into main 2026-10-28 18:21:36 +00:00
jschoubben dd9acaaeb3 Assigner: bind the seat's states at each pass and say while it waits for them, since their creation can come before this credential may hear it (ADR 0567 §1, review round 3)
mesh/delivery merged: source.main moved to novox/mesh-delivery@24181c024212
mesh/delivery-group group feat/1126-placement-seat merged: every member is merged
mesh/check failed: integration: meshbuildgo/cmd/mesh-build-go.TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus: the compose ask mvsg85b34d25b4 never ende…
2026-10-28 19:14:43 +01:00
jschoubben 451a4dfe74 Grants: pin mesh-sdk 38dbe1d, so the bus loads the node agents' reads of the seat mesh-placement's states before anything reads them (ADR 0567 §9, issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@db737df597c3
mesh/delivery-group group feat/1126-grants-first merged: every member is merged
mesh/check pass 451a4dfe74d9 on 2ac146078f47 | unit 1909 run, 715 kept, 3 skipped | integration 1931 run, 813 kept, 3 skipped | over bound: unit (slow…
2026-10-28 19:11:58 +01:00
jschoubben fb26ca5112 Pin mesh-sdk 38dbe1d, the group's head, in every module the seat touches (issue 1126) 2026-10-28 18:48:56 +01:00
jschoubben 389bb96ae4 Grants: read placement from the seat mesh-placement's state, as every reader does (ADR 0567 §9, issue 1126) 2026-10-28 18:47:49 +01:00
jschoubben 06e668cb90 Assigner: bound answers and asks by pulse keys instead of a timer, ask the operator for a hand-back, follow a partial write through, and call no controller verb for the seed (ADR 0567 §4, §5, §7, §9, review round 2) 2026-10-28 18:46:39 +01:00
jschoubben 297a3cb549 View: build_usage reads each node's capacity as the integer mesh-config resolves for it, default 0, since no repository names a node (ADR 0567 §12, issue 1126) 2026-10-28 18:35:14 +01:00
jschoubben 1d0467d4c3 Forge bridge: hold that a catalogue merge carries no placement against the retired field, as mesh-sdk now names it (ADR 0567 §11, issue 1126) 2026-10-28 18:34:19 +01:00
jschoubben 9db9499267 Checker: put every node's name on each check ask for the names check, so no repository lists them (ADR 0567 §12, issue 1126) 2026-10-28 18:34:02 +01:00
jschoubben deda4c985e Assigner: fold three small tests into their neighbours, so the seat's tests leave the counts flat (test budget) 2026-10-28 18:32:11 +01:00
jschoubben 06df6a6167 Assigner: prove a proposal answered, approved and written whole on the bus, and a moved key laid again (ADR 0567 §5, issue 1126) 2026-10-28 18:31:44 +01:00
jschoubben 1718d4b1d7 Assigner: follow every proposal from its answers through the Approve to the write at its revisions, keeping the controller in step (ADR 0567 §3-§8, issue 1126) 2026-10-28 18:31:43 +01:00
jschoubben 16b71ca685 Assigner: serve the seat's seven verbs, so a person proposes, withdraws, adopts, hands back and reads placement through the seat (ADR 0567 §2, §7, §10, issue 1126) 2026-10-28 18:31:43 +01:00
jschoubben 5d57b78fe9 Assigner: claim the seat mesh-placement with its stream, invokes and the resolved values it composes with (ADR 0567 §1, §6, §8, §9, issue 1126) 2026-10-28 18:22:40 +01:00
jschoubben 36332abf35 Assigner: drop the pull request's placement check and hold the seat's states as its own, since a placement change is no repository change (ADR 0567 §1, §4, §11, issue 1126) 2026-10-28 18:19:12 +01:00
jschoubben dbbdc95c0b View: drop a comment about the removed placement read 2026-10-28 17:59:34 +01:00
jschoubben fb0be84e24 View: drop indexer_placement and its reads, the seat mesh-placement answers placement (issue 1126) 2026-10-28 17:59:27 +01:00
jschoubben 730e44d762 Forge bridge: carry no placement files, placement is the seat mesh-placement's (issue 1126)
A catalogue merge or pull request no longer lists or reads placement/, so the Listed read and ConvertWith have no use and go.
2026-10-28 17:58:58 +01:00
jschoubben f4dd334b49 Indexer: stop holding the catalogue's placement, it is the seat mesh-placement's now (issue 1126)
Deletes the placement and candidate writes, the reread tool and its forge reads, the two buckets and the placement-changed event.
D1a and D1b are one commit: the index package, the named reads, the tools and the manifest do not compile apart.
2026-10-28 17:58:09 +01:00
mesh-tools 2ac146078f Merge pull request 'The delivery view forgets a write or event the bus refused, so 'on the machines' is not left stale (issue 1126)' (#97) from fix/1126-reach-now into main 2026-10-28 15:18:54 +00:00
jschoubben 726bbf1f1f The view writes a reach or says an event again when the bus refused it, so a refused write never leaves the reach state at an older have (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2ac146078f47
mesh/check being carried to 726bbf1f1ffa on 2ac146078f47 | from e753009fbc71
The view remembered a reach value, a reached head and a motion event as written or said before the bus took them; a write refused at its start (no responders) was never made again while the derivation stayed the same, and mesh-delivery.show said mesh-delivery-view not running on shanks while node_haves said it running.

Quick fix: forget only the refused reach write. Thorough fix, chosen: every write and event the view makes after its lock (reach, reached head, delivered revision, motion) answers its error, and the model forgets what was refused, so the next change it takes in makes it again; one place, the writer of the state both mesh-delivery.show and knowledge-base.progress read.
2026-10-28 16:11:52 +01:00
mesh-tools e753009fbc Merge pull request 'The indexer never moves a module backwards: a late merged message writes nothing; indexer_heard and indexer_module_history read what it heard and kept (issue 1126)' (#96) from fix/1126-index-order into main 2026-10-28 14:44:34 +00:00
mesh-tools f39bc10e3b Merge pull request 'builder_build_again: a person can build a failed key again, as the builder does by itself (issue 1126)' (#95) from fix/1126-build-again into main 2026-10-28 14:27:08 +00:00
jschoubben 3629bcf057 Read only gitea's 404 status as not-found, so a 502 naming a sha with 404 in it is redelivered, not written backwards (issue 1126, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@e753009fbc71
mesh/check being carried to 3629bcf057c3 on e753009fbc71 | from f39bc10e3be7
2026-10-28 15:01:51 +01:00
jschoubben 743a3618ea Keep back only a merge older than the one taken in, asked once per message before any write (issue 1126, review)
Asking whether the merge contains the held commit refused forward merges
forever on a diverged or rewritten main or a squash-merged registered head,
failed every later merge when the forge no longer had the held commit, and
cost one forge call per held commit. The question is now the reverse: a merge
is kept back only when it is in the taken merge and differs from it. A merged
message asks it once, before it writes anything, and a late message writes no
module, deletion, placement or taken key. A take-in asks it once for its
module. A 404 is not known to be older and is written, and said; any other
failure to say leaves the message for redelivery.
2026-10-28 14:56:40 +01:00
jschoubben 77d738d973 Never let a merge move a module backwards, and make the forge messages and a module's history readable (issue 1126)
The indexer wrote mesh-catalog's merge 9cdac09a back over the later 69056948,
rolling kernel and mesh-build-image back on the nodes: a merged message's
commit was written whatever the held one was. Before writing or deleting a
module's value, the indexer now asks the forge whether the merge contains the
held commit, keeps the held value when it does not and says so, and leaves the
message for redelivery when the forge cannot say. Each applied merged message
is said in one line with the modules written.

No tool showed what the indexer heard of FORGE nor a modules key's history
with times, so the cause could not be proven: the indexer now records each
FORGE message it handles (sequence, deliveries, times, commit, outcome) in its
state heard, read by indexer_heard, and indexer_module_history reads
modules.<module>'s kept values with their times.

Quick fix (Src order in writeOverlay) rejected: Src is 0 once the source moved
on, as here. Thorough fix chosen: the order is the forge's, on every path that
writes a module (merged message, take-in).
2026-10-28 14:48:13 +01:00
jschoubben a3b4ac79af builder_build_again: a look the builder queued during the by-hand build still runs; say a look that ended early is not asked
mesh/delivery merged: source.main moved to novox/mesh-delivery@f39bc10e3be7
mesh/check being carried to a3b4ac79afaa on f39bc10e3be7 | from 31bbf207ef3b
2026-10-28 14:32:33 +01:00
jschoubben c6dd04a7cc builder_build_again: register the look in one critical section, and answer what the build did
Review round 1 (issue 1126 note 72): the inflight check, the watch and the look were three locks, so a second request or the builder's own look could build the key twice; the answer said asked when the look wrote nothing or the build failed before its asks.
2026-10-28 14:30:31 +01:00
jschoubben 48cd554f46 Add builder_build_again: build a failed key again by hand as a new revision (issue 1126 note 72)
The builder retakes a failed key only when it looks at its module again, which nothing does for an unchanged module. The tool makes it look now, with the claim's own rules, and refuses a key that is done or waiting.
2026-10-28 14:25:03 +01:00
mesh-tools 31bbf207ef Merge pull request 'Say loudly what cannot be judged, read only a merge's changed directories, and move the taken key only by an applied merge (issues 1422, 1126; group fix/1422-trunk-unknown)' (#94) from fix/1422-trunk-unknown into main 2026-10-28 10:34:50 +00:00
jschoubben 2b801f5bd4 indexer_take_in reads a held module again only when its record is behind the source's head, and the two new tests are rows (issue 1126, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@31bbf207ef3b
mesh/delivery-group group fix/1422-trunk-unknown merged: every member is merged
mesh/check being carried to 2b801f5bd49a on 31bbf207ef3b | from ae7ec702a5e9, mesh-catalog 9fd95693db54
A head older than the record, or one whose order the forge cannot say, would have written
an older manifest back. The new tests become rows of existing ones: the repository is over
its 500 unit tests.
2026-10-28 11:18:01 +01:00
jschoubben 790653dfde indexer_take_in reads a held module again at the source's head when its record stands behind it (issue 1126)
A merge whose merged message never reached the indexer had no recovery: take-in answered
held for mesh-vault, whose record stayed at 78d7ef7 after mesh-catalog#535.
2026-10-28 11:10:09 +01:00
jschoubben 89be76bb6a The forge bridge keeps the directories a failed try read whole, so a large merge converges over its tries (issue 1126)
Without it the first merge after the bridge starts, with nothing kept, still reads all 140
catalogue directories in one 30 s try and times out as before.
2026-10-28 11:10:09 +01:00
jschoubben 704fc8d467 The forge bridge reads a merge's module directories only where their tree changed (issue 1126)
mesh-catalog's 140 module directories are 420 forge reads per merge, past the 30 s bound
of a handled event while the forge is busy: the merged message of #535 timed out on every
try, and #3fb7951 needed seven (issue 1458 met the same kind at 131). A tree hash names its
bytes, so what was read at a tree is kept in the new state inputs and in memory, and a
merge reads only the directories it changed.
2026-10-28 11:10:09 +01:00
jschoubben 458e403a04 The indexer's taken key moves only by a merge it applied, never by the gate's source head (issue 1126)
A drained forge stream does not say a merge was applied: the forge bridge was still
retrying mesh-catalog#535's merged message when the indexer started, and the source's
head moved taken.mesh-catalog to 9fd9569, so the view and show said the merge was on the
machines while mesh-vault's record stood at 78d7ef7 (ADR 0556 clause 1).
2026-10-28 11:10:09 +01:00
jschoubben 93c0ae19dd Forge bridge: write a head pushed to a pull request into another base as closed, and say so (issue 1422)
The checker judges pull requests into main only and source.main names main's head only (ADR 0484 §11), but a live update
into another base (de-spiegel#6 into nox-mesh) became a candidate, unlike at the start (OpenPulls) and at its merge. Once
the gate seeds a repository's trunk head, such a candidate would be judged against main's head: a verdict that lies.
2026-10-28 11:03:39 +01:00
jschoubben 4e9b076408 Checker: say loudly that a pull request cannot be judged when its repository has no trunk head (issue 1422)
A repository never merged through the gate has no entry in source.main, so its pull requests were dropped without a log
line, a verdict or a status, and checker_status counted them as new. Now the verdict says it cannot be judged and why,
the forge status is error with those words, checker_status names it under cannot_be_judged, the log says it once per
head, and the heads are judged whole once the trunk head is known. Quick fix: this. Thorough fix (the gate seeding
source.main from the pull message's base head) is still to do: it needs mesh-sdk, the forge bridge and the gate.
2026-10-28 11:03:39 +01:00
mesh-tools ae7ec702a5 Merge pull request 'Delete a closed pull request's placement candidates, and give indexer_placement a summary (issue 1126, group fix/1126-seed-same)' (#93) from fix/1126-seed-same into main 2026-10-28 09:46:34 +00:00
jschoubben 2cccbfd096 mesh-delivery-indexer and mesh-delivery-view pin mesh-sdk 3028c7e, the group's head after review round 2 (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ae7ec702a5e9
mesh/delivery-group group fix/1126-seed-same merged: every member is merged
mesh/check checking 2cccbfd096d8 on ae7ec702a5e9 | novox/mesh-delivery#93 in group fix/1126-seed-same
2026-10-28 10:02:46 +01:00
jschoubben 8f5e7f0c8f Fill go.sum for mesh-sdk 1062b33, the group's head (issue 1126) 2026-10-28 09:53:34 +01:00
jschoubben 75d3c40f41 mesh-delivery-indexer and mesh-delivery-view pin the group's mesh-sdk head 1062b33 (issue 1126; go.sum is filled once that head is pushed) 2026-10-28 09:51:48 +01:00
jschoubben 63582e44f3 Delete a closed pull request's placement candidates even when its closed message was missed, and give indexer_placement a compact summary form (issue 1126) 2026-10-28 09:25:09 +01:00
mesh-tools 4d4743d900 Merge pull request 'mesh-delivery-view passes a node's placement answer through as written, so its bodies reach readers (issue 1126, group fix/1126-shadow-bodies)' (#92) from fix/1126-shadow-bodies into main 2026-10-28 07:46:28 +00:00
jschoubben 289d1e8fa6 Format mesh-delivery-view's read.go as gofmt writes it (issue 1126, the group check's failure)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4d4743d90065
mesh/delivery-group group fix/1126-shadow-bodies merged: every member is merged
mesh/check checking 289d1e8fa69a on 4d4743d90065 | novox/mesh-delivery#92 in group fix/1126-shadow-bodies
2026-10-28 01:49:54 +01:00
jschoubben 968b9bf2e6 mesh-delivery-view: a node's candidate answer is shown as the node wrote it; pin mesh-sdk 362a3f1 (issue 1126, probe #532)
mesh/delivery-group group fix/1126-shadow-bodies rejected: composed together: the checker's verdict at the members' heads (46e1e846f8db): unit: modules/mesh-de…
mesh/delivery closed: a newer head of the same pull request
mesh/check failed (2 tests): unit: modules/mesh-delivery-view.gofmt: not as gofmt writes them (run gofmt -w): modules/mesh-delivery-view/internal/view…
The bodies of each difference never reached the answer shown: the view decoded
the node's answer with its mesh-sdk, which did not know the field, and encoded
it again. The answer now rides through as stored; a decoded copy only says its
state.
2026-10-28 01:39:34 +01:00
mesh-tools bab2e1391a Merge pull request 'mesh-grants: stop rewriting the shadow user list every 3 s when nothing changed (issue 1126, group fix/1126-switch-safe)' (#91) from fix/1126-switch-safe into main 2026-10-27 23:52:23 +00:00
jschoubben 663c9cbc34 mesh-grants: pin mesh-sdk 7359d18, the group's head (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bab2e1391ae5
mesh/delivery-group group fix/1126-switch-safe merged: every member is merged
mesh/check pass 663c9cbc348d on a49fd9cc65f7 | unit 2992 run, 1008 kept, 36 skipped | integration 3025 run, 1102 kept, 29 skipped | over bound: unit (…
2026-10-28 00:34:13 +01:00
jschoubben 3d916a685e mesh-grants: the same list said again moves neither the sealed key nor the shadow list (issue 1126)
Live since the take-over the shadow list was written every 3 s (revision 99270
to 99357 in 90 s, nothing differing): the sealed key read back says [] for the
nil left-out a pass composes, so every pass wrote the key again; each write
moved the bus node's report of what it loaded, whose revision and time were
inputs of the shadow list, and woke the next pass. The bus itself reloads only
when the list's bytes change (its log: two configurations alternating with the
placement, f8567865 and 473f9ad3), so this loop was load, not the reload.
2026-10-28 00:20:44 +01:00
mesh-tools a49fd9cc65 Merge pull request 'mesh-grants: take-over allowed when the grants part's list only adds rights, every loss still refused (issue 1945, ADR 0562)' (#90) from fix/1945-takeover-adds into main 2026-10-27 21:36:25 +00:00
jschoubben 3192d89edf mesh-grants: refuse a take-over over an uncompared deny; ask words only for added rights
mesh/delivery merged: source.main moved to novox/mesh-delivery@a49fd9cc65f7
mesh/check checking 3192d89edfcb on a49fd9cc65f7
Review round 1 of issue 1945's fix found two ways the guard could let a loss
through or hide one from the operator.

A publish deny was never compared: grants.User carried no deny, so a widened
allow that brings a new deny (a lost publish right) passed as an added right.
The user now carries its deny; a deny the shadow adds is a loss, and while
nats_users answers no deny for the loaded list, a user whose grants differ
and whose shadow grant has a deny refuses, named as a deny the loaded list
cannot be compared on. None of today's 41 differing users carries a deny
(only the four node-tools users do, and they are the same in both lists).

The ask said "takes none away" whenever rights were added, whatever else
differed. Its words now name the added rights only when nothing else refuses
(Refusals: unread list, Removing, a missing hash), and otherwise say what the
yes would be refused for, naming the users; Decide uses the same Refusals.

issue 1945
2026-10-27 22:33:53 +01:00
jschoubben 9edfcc40e9 mesh-grants: allow take-over when the lists differ only by added rights
The operator's yes to take-over was refused twice (2026-10-26 13:48 and
2026-10-27 21:19 UTC) because 41 users differ, every difference a right the
grants part's list adds. The controller's list is not changed (ADR 0547 §1),
so ADR 0547 §8's equality could never hold again. ADR 0562 changes it:
take-over is allowed when no user is added or dropped, no loaded user loses a
publish, subscribe or responses right and every hash is held; the ask names
the added rights by user and kind, and a yes after they changed is refused.

Tests: rows from the captured live comparison of 21:19 (testdata, no
hashes) in the compare table, and rows in the take-over table for a right
removed, added rights changed since the ask, and only added rights recorded.

issue 1945
2026-10-27 22:28:22 +01:00
mesh-tools 68bdda1a4d Merge pull request 'mesh-grants and the assigner: grant every node the served keys it composes with (issue 1126, group fix/1126-node-creds)' (#89) from fix/1126-node-creds into main 2026-10-27 20:32:34 +00:00
jschoubben bb7f2ff365 Merge remote-tracking branch 'origin/main' into fix/1126-node-creds
mesh/delivery merged: source.main moved to novox/mesh-delivery@68bdda1a4dbf
mesh/delivery-group group fix/1126-node-creds merged: every member is merged
mesh/check checking bb7f2ff365c7 on 68bdda1a4dbf | novox/mesh-delivery#89 in group fix/1126-node-creds
2026-10-27 21:18:57 +01:00
mesh-tools da51481620 Merge pull request 'mesh-grants: a seat keeps its holder under any placement, and no list or membership leaves a claimed seat without one (issue 1945)' (#88) from fix/1945-grants-seats into main 2026-10-27 20:14:04 +00:00
jschoubben c93f0d15e4 Grant the served keys of the trunk's manifest too, and pin mesh-sdk 7a8e9f9 in the assigner and mesh-grants on main's bus helpers (issue 1126, review round 1)
A node composing a consumer reads the served keys of the trunk's manifest of a
provider elsewhere, which may serve a setting the build it runs does not.
2026-10-27 21:13:41 +01:00
jschoubben e0cb285993 Merge remote-tracking branch 'origin/main' into fix/1126-node-creds
# Conflicts:
#	modules/mesh-delivery-assigner/go.mod
#	modules/mesh-delivery-assigner/go.sum
2026-10-27 21:12:25 +01:00
jschoubben a7d03f7c82 Pin mesh-sdk 538ffba, the group's head, in the assigner and mesh-grants (issue 1126) 2026-10-27 21:04:08 +01:00
jschoubben ea3bfa5e6e Grant every node the keys a provider elsewhere serves from, and give the composition check frombus's new reads, pinning mesh-sdk cebc651 (issue 1126)
mesh-grants declares each module's served settings (catalogue.ServedSettings), so
the user list grants every other node's agent those keys of mesh-config_resolved
alone: shanks then settles what novox's mailu serves. The assigner's check of
every node reads no build, archive or served value: it composes as before.
2026-10-27 21:02:49 +01:00
jschoubben ec0b7e2dcc Keep holding a membership that would drop a capability its holder still asks for (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@da514816204b
mesh/check being carried to ec0b7e2dcc06 on da514816204b | from 24cb379434f5
A dropped verified-sender comes from a root-free word that failed for one
pass, not from a release, and losing it cut the operator's approvals on
2026-10-27. Holdings.CapabilitiesLost holds such a membership with its own
reason while the holder is assigned and its manifest still asks for it; a
dropped tool or field and a moved seat stay published.
2026-10-27 20:56:09 +01:00
jschoubben feb32600bb Hold a list or membership only when a seat would lose its last holder (issue 1945)
Holding the user list on any loss would freeze it after take-over at the
first release that drops a tool or moves a seat (ADR 0547 §7), with nothing
to accept a deliberate loss; memberships had the same gap. The lead's decision
in review round 1: one pure guard, Holdings.LastHolders, holds the list and
each membership only when a seat held in force would have no holder in what
is composed now while an assigned module still claims it, the 09:21 kind.
A dropped tool, verb or field, or a seat handed to another node, is written.
2026-10-27 20:54:30 +01:00
jschoubben bc1978ec67 Keep every seat's holder under a placement, and hold a user list that would lose a grant (issue 1945)
The user list of 2026-10-27 09:21 granted no holder for nine seats: the seeded
placements hold only the seats the controller had on record, and the grants part
read those holds as every holder (records.go heldWherePlaced). A placement's
holds are now the holders on record, as catalogue.World's Holdings are: a mesh
seat no placement holds is held by derivation, wherever its claimant is placed.

The membership loss guard (Losses) did not cover the user list: after take-over
the list that lost those seats went out. UserLosses applies the same guard per
user against the loaded list, and holdsBack holds a list that would lose a grant,
or whose loaded grants cannot be read, raising the held warning.

Quick fix chosen over moving the on-record rule into mesh-sdk: mesh-grants is the
only part reading placement holds itself; the others pass them to the catalogue.
2026-10-27 20:54:30 +01:00
mesh-tools 24cb379434 Merge pull request 'Every delivery part opens its bus connection through mesh-sdk's module helpers (issue 1242)' (#87) from fix/1242-bus-helper into main 2026-10-27 19:49:22 +00:00
jschoubben 96be0c3377 Move every delivery part's bus connection onto mesh-sdk's module helpers: credential, pinned dial, own-stream subjects, durable consumer, and refuse a hand-written one (issue 1242)
mesh/delivery merged: source.main moved to novox/mesh-delivery@24cb379434f5
mesh/delivery-group group fix/1242-bus-helper merged: every member is merged
mesh/check checking 96be0c3377c4 on c38e81c7f5e9 | novox/mesh-delivery#87 in group fix/1242-bus-helper
2026-10-27 20:40:49 +01:00
mesh-tools c38e81c7f5 Merge pull request 'A want's self comes from the current placement alone, so a node no longer placed gets the controller's composition instead of a stale self want (issue 1126)' (#86) from fix/1126-self-wants into main 2026-10-27 15:25:20 +00:00
jschoubben bb63d83f84 Derive a want's Self from the placement now, in every rule, so a want written while a node was placed does not outlive it (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c38e81c7f5e9
mesh/check being carried to bb63d83f849e on c38e81c7f5e9 | from 2915d74822cd
After the placement seed's revert the rules that hold a want (a hold, rules 3 and 5) kept g14's and shanks's Self wants: the held branch copied them whole, and the rule branch wrote a want naming no composition, which EncodeWant refuses. Module.Self, set once from the placement, is now the one source, and every want the rules write or keep is derived from it: an unplaced node's kept want names the controller's composition of the same fingerprint, asked as any record move; with none, it stays and rule 7 says why.
2026-10-27 16:19:02 +01:00
mesh-tools 2915d74822 Merge pull request 'A placement pull request is carried, kept as candidates, and checked against every placed node's answer before it merges (issue 1126, ADR 0559)' (#85) from feat/1126-l2b into main 2026-10-27 07:21:16 +00:00
jschoubben 06d4f45e18 Pin mesh-sdk cadd7b1, the group's final head, in every member module (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2915d74822cd
mesh/delivery-group group feat/1126-l2b merged: every member is merged
mesh/check checking 06d4f45e1885 on 2915d74822cd | novox/mesh-delivery#85 in group feat/1126-l2b
2026-10-27 02:44:11 +01:00
jschoubben 77dacfbb17 Grant the node agents the placement candidates through the grants part at the group's sdk head, and show in indexer_placement the answer of every node placed at the head or on the trunk, so the seed's nodes can answer and be seen (issue 1126, ADR 0559 §3, §7; review round 1) 2026-10-27 02:43:00 +01:00
jschoubben d8592d9a4f Pin mesh-sdk 062c29c, the group's final head, in every member module that uses the candidate contract (issue 1126) 2026-10-27 02:30:16 +01:00
jschoubben ea4a163187 Pin mesh-sdk a37d262, the group's head, in the bridge, the indexer and the view (issue 1126) 2026-10-27 02:29:28 +01:00
jschoubben fc92202e04 Take a later composition into a verdict that failed only because placed nodes had not answered, so a sleeping node's answer passes the check on its event with no new check (issue 1126, ADR 0559 §5) 2026-10-27 02:29:20 +01:00
jschoubben d8da56c6da Compose a catalogue pull request's own placement in the assigner's check and fail it on any placed node's unintended difference, refusal or silence past ten minutes, so a placement change is proven equal to what runs before it merges (issue 1126, ADR 0559 §5) 2026-10-27 02:29:20 +01:00
jschoubben c2b1ba4c5d Pin mesh-sdk ba398e2, the group's head with the node's whole candidate answer (issue 1126) 2026-10-27 02:20:17 +01:00
jschoubben 04bb2d1398 Show placement candidates and each node's answer in indexer_placement, add indexer_placement_reread, and drop a group's candidates when its head changes no placement file, so a missed message or a revert is recoverable by tool (issue 1126, ADR 0559 §3, §7, §8) 2026-10-27 02:18:21 +01:00
jschoubben ad60ba3a46 Write a catalogue pull request's changed placement as placement-candidates <group>.<node> and delete them when it merges or closes, so each placed node composes it in shadow and answers (issue 1126) 2026-10-27 02:13:55 +01:00
jschoubben ea3eac51bd Carry a catalogue pull request's placement files at its head when it touches placement/, so the indexer can write them as candidates for the nodes to answer before the merge (issue 1126) 2026-10-27 02:13:55 +01:00
mesh-tools cacf537004 Merge pull request 'The checker's failure is the build step's failure, and a verdict that hides its failing test is refused (issue 1901)' (#84) from fix/1901-failure-contract into main 2026-10-26 23:44:21 +00:00
jschoubben f4a86f5884 Merge main into fix/1901-failure-contract: the checker pins mesh-sdk a2759ec, the group's sdk head (issue 1901)
mesh/delivery merged: source.main moved to novox/mesh-delivery@cacf537004c9
mesh/delivery-group group fix/1901-failure-contract merged: every member is merged
mesh/check checking f4a86f58843d on cacf537004c9 | novox/mesh-delivery#84 in group fix/1901-failure-contract
2026-10-27 00:39:35 +01:00
mesh-tools 9f1b227fb8 Merge pull request 'The indexer fills each repository's taken key from merges it applied or a drained stream's heads, so a merge heard while the state was missing reads on the machines (issue 1280)' (#83) from fix/1280-indexer-fills-taken into main 2026-10-26 23:22:01 +00:00
jschoubben 6c79e456a1 The indexer confirms a forge message's ack with the server before it fills taken keys (issue 1280)
mesh/delivery merged: source.main moved to novox/mesh-delivery@9f1b227fb8c0
mesh/check checking 6c79e456a100 on 9f1b227fb8c0
2026-10-27 00:20:16 +01:00
jschoubben 0b980203fe Taken keys are filled only from merges the indexer applied or a drained forge stream's heads (issue 1280) 2026-10-27 00:18:23 +01:00
jschoubben 73bf290f58 The indexer fills every repository's taken key from the source's head at start and when the state appears (issue 1280) 2026-10-27 00:14:41 +01:00
mesh-tools 475bb4bb0b Merge pull request 'The view's tools read their own reach by its bare name, and a test holds every state the commands read to what is issued (issue 1280)' (#82) from fix/1280-view-reads-its-reach into main 2026-10-26 22:48:25 +00:00
jschoubben efadc691e7 The view's tools read their own reach by its bare name, as the mesh issues it, and a test holds every state the commands read to what is issued (issue 1280)
mesh/delivery merged: source.main moved to novox/mesh-delivery@475bb4bb0b8f
mesh/check being carried to efadc691e7ea on 475bb4bb0b8f | from 4af36c69a761
view_reach asked for mesh-delivery-view.reach, a name the mesh never issued, so it failed on the machines;
the same line was not gofmt'd, which failed the try of main. Quick fix: the name. Thorough for this module: the
manifest test now refuses any stdio.State literal in cmd/ that is neither an own state nor listed in reads.
2026-10-26 23:27:05 +01:00
mesh-tools 4af36c69a7 Merge pull request 'Delivery records are true at the source: the indexer records what it took in, the view derives what reached the machines, a settle verb for held deliveries (issue 1280)' (#80) from fix/delivery-records-true into main 2026-10-26 21:48:50 +00:00
mesh-tools bd107d2015 Merge pull request 'A version that fails on one machine while another runs it healthy is that machine's failure, and goes no further meanwhile (issue 1496, ADR 0557 §1-§3)' (#81) from fix/1496-start-health into main 2026-10-26 21:37:58 +00:00
jschoubben f007ca4227 A node that runs a version alone-failed elsewhere still gets its want at it (ADR 0557 §2, issue 1496)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bd107d20154c
mesh/delivery-group group fix/1496-start-health merged: every member is merged
mesh/check checking f007ca42279a on 4b36cc29a6ae | novox/mesh-delivery#81 in group fix/1496-start-health
Review of 50c4674: the bar on new wants naming F while a node fails it alone
also caught a node that runs F with no live want, the issue-1945 recovery path:
it went on neither list and its node-engine gave the module up. The bar now
holds only a node that neither wants nor runs F.
2026-10-26 22:34:25 +01:00
jschoubben 50c4674dfa While a node fails a version alone, no further node gets a want for it (ADR 0557 §2, issue 1496)
ADR 0557 version 9: a failure that is one node's holds nothing and sets nothing
back, but the version reaches no node that does not want it yet until it is
over. A node whose want does not name F gets no new want naming it (rule 2, in
words naming the failing nodes); wants already written stay.
2026-10-26 22:30:00 +01:00
jschoubben c12b870104 The indexer binds its taken state when it appears, as issue 1790 asks of every state a delivery adds: an indexer started before the bus holds it still takes merges in and says the key it did not write (issue 1280, ADR 0556 clause 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4af36c69a761
mesh/delivery-group group fix/delivery-records-true merged: every member is merged
mesh/check checking c12b870104e8 on bd107d20154c | novox/mesh-delivery#80 in group fix/delivery-records-true
2026-10-26 22:25:39 +01:00
jschoubben 92aedb9365 The first pass after a start clears failed-alone for every module and known node (ADR 0557, issue 1496)
Review round 2 of 16e4e61: gating the first-pass clear on a failed have left a
condition an earlier process raised standing for good once its node recovered
while no assigner watched. The first pass now clears it for every module and
every node the presence bucket knows, a quiet clear at the router for one that
is not open.
2026-10-26 22:21:00 +01:00
jschoubben 368204115c Pin the teller at mesh-sdk 68d65e6, the group's sdk head merged with sdk main (issue 1280) 2026-10-26 22:20:45 +01:00
jschoubben 971cdb44e2 Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-indexer/cmd/mesh-delivery-indexer/main.go
#	modules/mesh-delivery-indexer/go.mod
#	modules/mesh-delivery-indexer/go.sum
#	modules/mesh-delivery-indexer/manifest_test.go
#	modules/mesh-delivery-indexer/module.json
#	modules/mesh-delivery-view/cmd/mesh-delivery-view-tools/main.go
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
#	modules/mesh-delivery-view/internal/view/read.go
#	modules/mesh-delivery-view/manifest_test.go
#	modules/mesh-delivery-view/module.json
2026-10-26 22:20:45 +01:00
jschoubben 16e4e61a0f Only an online healthy witness, and only a first failure, keep a version from being held; failed-alone clears when a node leaves (ADR 0557, issue 1496)
Review of 2ee7589: a node that sleeps after running F healthy kept its stale
have as a witness, so F stayed unheld while it failed on the online nodes; and a
failed-alone condition was never cleared once its node left the module.

A witness must now be online by the watchman's presence. F failing on a second
node is no longer one node's failure: the hold and the set-back stand (ADR 0557
version 6, §2). A remembered failed-alone key whose node is no longer one of the
module's nodes is cleared once; the first pass after a start clears it for every
node the presence bucket knows, for a module with a failed have.
2026-10-26 22:16:38 +01:00
mesh-tools 4b36cc29a6 Merge pull request 'Placement from the catalogue: the bridge carries it, the indexer writes it, the assigner compares it (issue 1126)' (#76) from feat/1126-nodes-compose into main 2026-10-26 21:11:26 +00:00
mesh-tools 31ddd09087 Merge pull request 'The checker ends a check whose heads no candidate holds, and starts a group's check once its heads settle (issue 1890)' (#79) from fix/1890-end-superseded-checks into main 2026-10-26 21:10:56 +00:00
jschoubben 68173de8bd Pin the indexer, the teller and the view at mesh-sdk b4310df: the delivery-records work merged with sdk main (issue 1280) 2026-10-26 22:08:40 +01:00
jschoubben 960e3309bc Merge remote-tracking branch 'origin/main' into fix/delivery-records-true
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 22:07:28 +01:00
jschoubben 2ee7589474 A version healthy on another node is not held for one node's failed health check (ADR 0557, issue 1496)
On ace, 2026-10-26 21:19, a newly assigned mesh-build-node failed its first
health check at 8a98386f (later 31be2f24) while shanks, g14 and novox had run it
healthy for an hour. Being first by name, ace was the canary, and the assigner
held the version and moved the three healthy nodes back.

A failed health check of the newest fingerprint now holds it only when no other
node's have names it healthy (ADR 0557 §2 keeps ADR 0429 §6's hold and set-back
for that case). Otherwise the failure is that node's (ADR 0557 §1): nothing is
held or set back, the node keeps its want, and the assigner raises the node's
condition once (failed-alone.<node>.<module>), naming the node, the module, the
fingerprint and the have's words, and clears it once when it ends; the first
pass after a start clears what a previous process may have raised.
2026-10-26 22:06:32 +01:00
jschoubben 297992a4ae Pin mesh-sdk 9796444 (l2-next: feat/1126-nodes-compose merged with sdk main #101, the standing-test race fixes) (issue 1126)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4b36cc29a6ae
mesh/delivery-group group feat/1126-nodes-compose merged: every member is merged
mesh/check checking 297992a4ae55 on 4b36cc29a6ae | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 22:05:41 +01:00
jschoubben 3f85230417 Pin mesh-sdk 5f45175 (l2-next: feat/1126-nodes-compose merged with sdk main) after merging main (issue 1126) 2026-10-26 21:59:21 +01:00
jschoubben 43c3158bc6 Merge remote-tracking branch 'origin/main' into feat/1126-nodes-compose
# Conflicts:
#	modules/mesh-delivery-view/go.mod
#	modules/mesh-delivery-view/go.sum
2026-10-26 21:57:03 +01:00
jschoubben d95e9f3229 End a check whose heads no candidate holds, and start a group's check once its heads settle (issue 1890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@31ddd090876f
mesh/check pass d95e9f322913 on d581f9659a78 | unit 191 run, 525 kept | integration 201 run, 591 kept | composition passed on novox
2026-10-26 21:44:12 +01:00
mesh-tools d581f9659a Merge pull request 'Tries record their asks; the try limit counts tries on a build node; build_usage shows capacity, hours and steps (issue 1890)' (#78) from fix/capacity-follows-cpu-share into main 2026-10-26 20:33:34 +00:00
jschoubben ea94f9234c Merge remote-tracking branch 'origin/main' into fix/capacity-follows-cpu-share
mesh/delivery merged: source.main moved to novox/mesh-delivery@d581f9659a78
mesh/delivery-group group fix/capacity-follows-cpu-share merged: every member is merged
mesh/check checking ea94f9234c97 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
2026-10-26 21:29:44 +01:00
jschoubben c19468b04e Pin the checker and the view at mesh-sdk 805b4d9: the capacity work merged with sdk main 2026-10-26 21:29:44 +01:00
jschoubben c47afd8f4b build_usage says a seat's build hours, and when it is outside them
mesh/delivery closed: a newer head of the same pull request
mesh/delivery-group group fix/capacity-follows-cpu-share checking: 1 of 3 member(s) ready
mesh/check checking c47afd8f4b88 on 3dd94f714d6e | novox/mesh-delivery#78 in group fix/capacity-follows-cpu-share
A build module's capacity.<node> now carries its node's build window (the operator, 2026-10-26: g14
builds only from 18:00 to 06:00); the seat shows it, outside it says it takes nothing new, and a
value the module cannot read shows as any hour with the value and why. The checker's and the view's
SDK pin moves to mesh-sdk d00dac1.
2026-10-26 21:22:36 +01:00
jschoubben afd5a3ba07 A module a node wants and runs nothing countable of holds no head back; the reached head's event names its merges as not yet on the machines; pin mesh-sdk e44cbaf (review of issue 1280, ADR 0556 clause 2) 2026-10-26 21:15:31 +01:00
jschoubben 7dbad4c6d2 Pin mesh-sdk 33144781 (taken, reach, settle layouts) in the indexer, the view and the teller (issue 1280) 2026-10-26 21:05:29 +01:00
jschoubben 083a5042f6 The view reads the wants by module once per derivation of the reach 2026-10-26 21:03:56 +01:00
jschoubben 351ca7c290 The teller checks a settle ask's warrant against mesh-delivery.settle, the act its options bind (issue 1280, ADR 0556 clause 5) 2026-10-26 20:58:20 +01:00
jschoubben f907be2c14 The view derives per repository up to which merge every node runs it, keeps it as its state reach and says each reached head, so a fix on the machines reads delivered (issue 1280, ADR 0556 clauses 2 and 3) 2026-10-26 20:56:12 +01:00
jschoubben 6201f58c20 The indexer says per repository the last merge it took in, so the view knows its modules speak for that merge (issue 1280, ADR 0556 clause 1) 2026-10-26 20:49:00 +01:00
jschoubben 8fee764c73 Pin mesh-sdk feat/1126-nodes-compose 03dce12, the group's head: the checker refuses a pin that is neither trunk nor the group's head (issue 1126)
mesh/delivery-group group feat/1126-nodes-compose checking: 4 of 6 member(s) ready
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 8fee764c734e on d581f9659a78 | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:35:45 +01:00
mesh-tools 3dd94f714d Merge pull request 'The checker fails any repository's change that adds a binary, in any commit (issue 1577)' (#77) from fix/1577-checker-refuses-binaries into main 2026-10-26 19:32:27 +00:00
jschoubben c716670be0 The checker pins the SDK of review round 1, and its hid-the-failure cases join the older build module's test (issue 1901) 2026-10-26 20:19:07 +01:00
jschoubben 352066dde6 Review of L2: the assigner binds the indexer's placement state when it appears, with no restart (D1); the three new unit tests folded into existing ones, so no count rises (D2); pin mesh-sdk 3968849 (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check checking 352066dde611 on 3dd94f714d6e | novox/mesh-delivery#76 in group feat/1126-nodes-compose
2026-10-26 20:12:00 +01:00
jschoubben d6bd31a166 Say the try limits as what they are: a check when a try is asked
Tries let in while waiting may later all run at once; each node's build capacity bounds what runs
on it (review of issue 1890's try cap).
2026-10-26 20:11:08 +01:00
jschoubben c0a4b139a2 A try queued behind the build seat's checks no longer blocks the next try
Only tries a build node holds count toward the 4 at once; tries waiting for a node are bounded
apart at 8, and the refusal names both counts (issue 1890: at 19:53 four tries queued behind 44
checks for up to 47 minutes refused every further try).
2026-10-26 20:05:51 +01:00
jschoubben 6d60953f98 A running try names its asks and where each stands; build_usage says a seat's CPU-bound capacity
The checker wrote a try's asks only at its end, so a 31-minute try could not be told queued from
running (issue 1890). The try's record now names each ask as it is put, then the node that took
it, its state and the phase its step says, written on the step key's own changes. build_usage
names a running try's asks and takes a build module's capacity.<node> (configured bounded by its
node's build CPU, issue 1806's follow-up) when it is current with mesh-config.
2026-10-26 19:58:39 +01:00
jschoubben 17ed06ee4d Pin mesh-sdk's fix/1901-failure-contract head 81ec7bd7 (issue 1901) 2026-10-26 19:50:47 +01:00
jschoubben 9427dac823 The checker says loudly when a build module hid a failure's cause (issue 1901)
A failed test with no cause, or a passing line as its cause, was worded as if it were the failure: a try of mesh-catalog
failed with a passing '✔' line of the first test file. Under mesh-sdk's failure contract the verdict's words and its
status line now start with 'the build module hid the failure (<module> at <version>)', and the check still fails.
2026-10-26 19:50:05 +01:00
jschoubben d74e082155 Checker refuses a program a commit of the branch added and a later one removed: it stays in history (issue 1577)
mesh/delivery merged: source.main moved to novox/mesh-delivery@3dd94f714d6e
mesh/delivery-group group fix/1577-checker-refuses-binaries merged: every member is merged
mesh/check checking d74e0821552c on df8b9ac3b1b5 | novox/mesh-delivery#77 in group fix/1577-checker-refuses-binaries
2026-10-26 19:47:53 +01:00
jschoubben 2e0013ec25 The assigner's wants test goes on to placement: a placed node's want names the build alone and its own want says it composes its own (issue 1126, ADR 0547 §2)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2e0013ec2500 on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:44:55 +01:00
jschoubben 2493aa6529 Pin mesh-sdk feat/1126-nodes-compose 9713c30: placement carries a settings-form module's values (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: ace: novox/mesh-catalog#505 does not merge into the trunk: files in conflict: .gitignore | 2493aa65299e on df8b9ac3b1b5 | unit 1245…
2026-10-26 19:40:38 +01:00
jschoubben 85c89b16ef The grants part composes what the catalogue's placement places beside the assignments, and holds a seat held once for the mesh where placement says (issue 1126, ADR 0549 §3, G5)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
2026-10-26 19:33:27 +01:00
jschoubben 4f75925410 Nodes compose: placement from the catalogue, node records, self wants, own and pair credentials, core boxes and the authority from the vault (issue 1126)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/nobinaries.TestTheCatalogueHoldsNoCompiledProgram: modules/mesh-config/cmd/mesh-config…
One commit for the branch feat/1126-nodes-compose of mesh-delivery: its 16 commits squashed so its history does not carry the 17.8 MB compiled mesh-delivery-assigner a commit added by mistake and a later one removed (issue 1577). The tree equals the former head ff6c038d. The operator allowed this rewrite of the feature branch on 2026-10-26.
2026-10-26 19:30:18 +01:00
jschoubben 820c458f4c The captured clean fetches carry the programs a current build module answers, so lines asserted whole do not warn (issue 1577) 2026-10-26 19:16:25 +01:00
jschoubben dc9d65524d A check judged by a build module that reports no programs says so in its status line, still passing: the refusal did not run (issue 1577) 2026-10-26 19:08:00 +01:00
jschoubben 1a4c788ed7 Checker fails a change whose fetch reports a compiled program, and warns when a build module reports none: build output kept getting committed (issue 1577) 2026-10-26 19:01:40 +01:00
mesh-tools df8b9ac3b1 Merge pull request 'Checker, builder and indexer bind states created after they start (issue 1790)' (#75) from fix/1790-state-appears into main 2026-10-26 14:47:40 +00:00
jschoubben fcb1e40582 Pin mesh-sdk fix/1790-state-appears at 65ee43f: a refused hearing of a bucket's creation is said and heard again at a reconnect (issue 1790, review)
mesh/delivery merged: source.main moved to novox/mesh-delivery@df8b9ac3b1b5
mesh/delivery-group group fix/1790-state-appears merged: every member is merged
mesh/check being carried to fcb1e4058262 on df8b9ac3b1b5
2026-10-26 15:25:53 +01:00
jschoubben 4973372612 Merge remote-tracking branch 'origin/main' into fix/1790-state-appears 2026-10-26 15:25:06 +01:00
jschoubben 3f6c085e8a Pin mesh-sdk fix/1790-state-appears at 3ac644b (issue 1790) 2026-10-26 15:14:00 +01:00
jschoubben 040e3097b5 checker, builder, indexer: a state of their own not on the bus at start is bound when it appears; mesh-grants composes each principal's hearing of its buckets' creation (issue 1790)
A delivery restarted the checker and the builder before the grants part created their new state hearing;
OptionalBucket answered nil and the hearing was never kept. They, and the builder's reasks and queues and the
indexer's takeins, now open their own states through mesh-sdk process.AwaitBucket (mesh-sdk
fix/1790-state-appears d515c71), which binds on the bus's STREAM.CREATED advisory and writes the latest kept
value. mesh-grants takes the same SDK, whose composer grants that advisory.
2026-10-26 15:13:00 +01:00
mesh-tools feef96ed16 Merge pull request 'build_usage: how much each build node is used and whether it may build, in one call' (#74) from feat/build-usage-tool into main 2026-10-26 14:09:32 +00:00
jschoubben 4f722899df View: build_usage does not answer eligibility it cannot read; a resolved capacity key with no readable value is the puller's 0
mesh/delivery merged: source.main moved to novox/mesh-delivery@feef96ed160e
mesh/check checking 4f722899df25 on feef96ed160e
A capacity this tool cannot read is unknown to it, not 0: the seat says eligible_known false and the node not known.
A key mesh-config holds without a readable value is what the puller reads as 0, and was dropped silently.
2026-10-26 14:58:47 +01:00
jschoubben 166a918b43 View: build_usage counts keys a build module ended for being left running apart, lists every node a capacity names, and says capacity unknown when mesh-config cannot be read
Since mesh-sdk #90 a build module ends a left-running key as failed at the moment it notices; counted as finished,
such a step put hours into the medians and the time at capacity. The view takes mesh-sdk's step bound and restart
reason from the SDK it now pins.
2026-10-26 14:58:47 +01:00
jschoubben 1b4c860f63 Checker: its announcement test expects tries readable, as the delivery view now reads them 2026-10-26 14:58:47 +01:00
jschoubben 3b6945b2db View: build_usage answers how the build nodes are used in one call; the checker's tries name their asks
The operator's question (how much are the build nodes used, is ace eligible) took six calls over four tools.
build_usage reads it from the bus's states: seats held (have), capacity (mesh-config resolved), pauses, the
build modules' steps by kind with left-running keys, the last hour and day with total and median time and time
at capacity, the running tries, and the backlog the view program reads from the seats' streams. A try now names
the asks it put, and the checker announces its tries readable, so the view tells a try's asks from a check's.
2026-10-26 14:58:41 +01:00
mesh-tools 4174e23c04 Merge pull request 'The checker and the builder end or re-ask what the build seat gave up, and a try can be ended by hand (issue 734)' (#73) from fix/734-lost-ask into main 2026-10-26 13:54:17 +00:00
jschoubben 3993175e45 Checker, builder and grants part: pin mesh-sdk main, which holds both the dead-letter hearing (#92) and the placement (#91) (issue 734)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4174e23c0492
mesh/delivery-group group fix/734-lost-ask merged: every member is merged
mesh/check checking 3993175e457f on 4174e23c0492
2026-10-26 14:45:33 +01:00
jschoubben 60c7ffc981 Checker, builder and grants part: hear DEAD_LETTERS again until heard, say when they do not, and compose the same grant (issue 734, review round 1)
mesh-grants moves to the same mesh-sdk (3203358c), so its shadow user list grants the checker and the builder the
DEAD_LETTERS reads the controller's list will, and the take-over is not blocked by a difference. The checker and the
builder hear DEAD_LETTERS through buildrun.DeadLetterHearing: in the background, again at every reconnect and every ask
waited for while deaf, and keep whether they hear it in their new state hearing, which checker_status and builder_stuck
say as dead_letters. checker_try says that a try ended by the not-taken bound leaves its ask in the stream.

Not done as asked: a module health tool. The catalogue refuses a tool check on a module with no runtime of its own and no
other check beside it (ADR 0227 rule 8, ADR 0366 §1), so the deaf state is said in the status tools and the log.
2026-10-26 14:44:53 +01:00
jschoubben 33de9dde4f Checker and builder: wait on the build seats through mesh-sdk's LostAskWaiter, so an ask the seat gave up on ends every asker's wait (issue 734)
The waiter moved to mesh-sdk buildrun (fix/734-lost-ask, d96b9fd6), where every asker of the build seats shares it.
The builder hears DEAD_LETTERS for the go, node and image seats and asks a given-up part of a build again at once, with
the seat's words as why, instead of waiting for the check bound. The checker keeps what it does with a lost ask (a try
ends as error, a check is asked again) and its verdict words.
2026-10-26 14:44:52 +01:00
jschoubben 6b85220fa4 Checker: an ask the build seat gives up on ends its try as error and asks a check again; a try no node takes ends at the step bound; checker_try_end ends one by hand (issue 734)
A check ask handed out five times during a bus reload was given up into DEAD_LETTERS and no build node wrote a step for it,
so try-1793017340003993788 stayed running for ever, held one of the four try places, and checker_try of its head kept
answering its id. The checker now hears the copies DEAD_LETTERS keeps of the build seats' given-up asks (an ordered
consumer from its start; the grant is mesh-sdk/mesh-controller DeadLetterReads, branch fix/734-lost-ask) and ends the
step it waits for: a try ends as error saying why, a check is asked again under its claim, up to three times, then fails
as not heard. A try whose ask no node takes within the step bound ends the same way (a check's backlog stays no fault,
ADR 0525 §9). checker_try_end ends a try by hand, freeing its place and stopping its judging.
2026-10-26 14:44:40 +01:00
mesh-tools 1cbae7d41c Merge pull request 'The checker asks the packages a node could not test again, on a node that has what they need (issue 890, ADR 0552)' (#72) from feat/890-placement into main 2026-10-26 13:40:45 +00:00
mesh-tools b3a4b09f5b Merge pull request 'The indexer takes in a registered module whose repository became readable, and serves indexer_take_in (issue 1945)' (#71) from fix/indexer-takes-in into main 2026-10-26 13:39:13 +00:00
mesh-tools c8b26735c8 Merge pull request 'The assigner says when it holds the bus back instead of skipping it silently (issue 1945)' (#70) from fix/bus-bar-loud into main 2026-10-26 13:13:22 +00:00
jschoubben 907e092abc mesh-delivery-indexer reads unread registered modules again in one background run, so a forge timeout holds no merged message or source revision past its ack wait (issue 1945, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b3a4b09f5b6f
mesh/check being carried to 907e092abc45 on b3a4b09f5b6f
The take-in tool's bound now says the take-in is still running and where its answer will be.
2026-10-26 14:11:12 +01:00
jschoubben 3073495a06 checker: pin mesh-sdk 33395f68, feat/890-placement merged with main (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@1cbae7d41cd0
mesh/delivery-group group feat/890-placement merged: every member is merged
mesh/check checking 3073495a0670 on 1cbae7d41cd0
2026-10-26 14:06:10 +01:00
jschoubben 16c411e436 Merge remote-tracking branch 'origin/main' into feat/890-placement 2026-10-26 14:05:40 +01:00
jschoubben 2d841870a8 Assigner: only a put still on its steps is named when a node leaves neither list (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@c8b26735c8cf
mesh/check pass 2d841870a873 on c8b26735c8cf | carried from 1d416827157d | unit 185 run, 518 kept | integration 195 run, 584 kept | composition passed…
Review round 2: an ended put stays in the bucket and owns no want.
2026-10-26 14:03:48 +01:00
jschoubben 7f8ce7f5c0 checker: the placed ask names the unmet packages, so a package without a kept verdict is not run or counted twice (issue 890, ADR 0552 §4; review round 1). Pins mesh-sdk 45564e69. 2026-10-26 14:02:16 +01:00
jschoubben 07b5f13474 mesh-delivery-indexer reads a registered module again once its repository becomes readable, and takes one in on ask (issue 1945)
A registered module whose repository the forge answered 404 (car-hunter, snake, before the forge's account could read
them) was read again only when the indexer started again or the controller wrote a new build record of it, so the grants
part's composed user list stayed short of their subjects and the take-over was refused. A failed read now keeps the
module as unread, read again on the next source revision and the next merged message (no timer of its own), and the new
tool indexer_take_in (the tools bundle mesh-delivery-indexer-tools, through the indexer's takeins state, as the
builder's builder_ask_again) takes one in now and answers what it stored or why not.
2026-10-26 14:01:34 +01:00
jschoubben f4aac599ca Assigner: name a running put when a node leaves neither list, and say the bus bar in plain words (issue 1945)
Review round 1: while an operator's put runs through its steps it owns the want, so the clear said
'the node does not run it' of a node that runs the bus.
2026-10-26 14:00:32 +01:00
jschoubben 2d63f0ba62 Assigner: say the bus bar instead of skipping it silently, and clear a neither condition only in true words (issue 1945)
A bus whose move only the operator's put may make (ADR 0546 §8) got no want and nothing in the outcome:
nats on novox, running an old-form build, was on no list while the pass cleared its condition and logged
that it had a want again. The bar now names its holding, puts a node without a live want on neither list
(its condition needs the operator: assigner_put), and a neither condition is cleared as 'has a want again'
only when a live want exists.
2026-10-26 13:55:42 +01:00
mesh-tools 1d41682715 Merge pull request 'mesh-grants says loudly when it holds the user list back, and clears it after a restart (issue 1945)' (#69) from fix/grants-hold-loud into main 2026-10-26 12:44:36 +00:00
jschoubben 705765d113 checker: a placed ask that every node declines for longer than its bound (30 min) fails the check naming the need (issue 890, ADR 0552 review) 2026-10-26 13:42:46 +01:00
jschoubben 16a97ed65d checker: ask the packages a node left unmet again of a node that has what their tests need, and put a declined ask again (issue 890)
A check that landed on ace passed with mesh-build-node's integration tests skipped. The build module now leaves such a
package unmet; the checker keeps the passes and asks the levels from the first unmet one with the need named, putting the
ask again under a new id when a node lacking it declines it. Re-pins mesh-sdk feat/890-placement.
2026-10-26 13:40:00 +01:00
mesh-tools ecf532af0e Merge pull request 'checker_status reads the node-engine at its running commit, as the checker decides (issue 568)' (#68) from fix/status-and-holds into main 2026-10-26 12:39:56 +00:00
jschoubben ce600df98a Say the held-list state on a part's first pass, so a warning raised before a restart is cleared
mesh/delivery merged: source.main moved to novox/mesh-delivery@1d416827157d
mesh/check being carried to ce600df98afc on 1d416827157d
2026-10-26 13:37:31 +01:00
jschoubben b4a6a34705 mesh-grants: a user list held back after take-over raises the warning mesh-grants.user-list-held and clears it when the list advances, so the hold is not only in a tool's output (issue 1945) 2026-10-26 13:36:39 +01:00
jschoubben a7e51b416b checker_status reads the node-engine at its running commit, as the checker's decision does, so it stops listing every candidate as waiting for a trunk move (issue 568)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ecf532af0e85
mesh/check being carried to a7e51b416bc1 on ecf532af0e85 | from 71f786f14662
2026-10-26 13:25:18 +01:00
mesh-tools 71f786f146 Merge pull request 'mesh-grants: take-over and hand-back by the operator's Telegram Approve, the sealed list, and every grant the loaded list holds (issue 1945)' (#67) from feat/1945-user-list-switch into main 2026-10-26 12:21:50 +00:00
mesh-tools 848c6fdc56 Merge pull request 'Checker: one check per set of heads, a trunk or sibling move carries the verdict, a merge turn holds a claim (ADR 0548, issue 568)' (#65) from feat/548-one-check into main 2026-10-26 12:01:32 +00:00
mesh-tools ead9ee2d13 Merge pull request 'Indexer: index a module registered from any branch, by its registered ref, never a newer pull request's (issue 1945)' (#66) from fix/indexer-registered-modules-2 into main 2026-10-26 12:01:24 +00:00
mesh-tools 4861d1f78d Merge pull request 'Checker: keep no package whose tests skipped, so a skipped pass is never reused as proof (issue 890)' (#64) from fix/checker-skips-not-kept into main 2026-10-26 12:01:19 +00:00
jschoubben 5c76c9d3ac mesh-grants: a list in force with no users kept is checked against the users the bus loaded, so the first list after take-over keeps every user (review of the switch, issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@71f786f14662
mesh/delivery-group group feat/1945-user-list-switch merged: every member is merged
mesh/check being carried to 5c76c9d3ac84 on 71f786f14662 | novox/mesh-delivery#67 in group feat/1945-user-list-switch
2026-10-26 12:54:55 +01:00
jschoubben 699cbfb7c1 Indexer: a registered module takes its registered ref, not a newer branch's record; no read at an older commit after a failure
mesh/delivery merged: source.main moved to novox/mesh-delivery@ead9ee2d138d
mesh/check checking 699cbfb7c131 on ead9ee2d138d
2026-10-26 12:52:29 +01:00
jschoubben 319d56be7e Indexer: a registered module is read from every repository it was built from, whatever the branch, and at the registered commit when the head is gone 2026-10-26 12:45:35 +01:00
mesh-tools 11d3678984 Merge pull request 'Indexer: index a module registered from its own repository, so the grants part composes its grants (issue 1945)' (#63) from fix/indexer-registered-modules into main 2026-10-26 11:33:41 +00:00
jschoubben f55f150896 mesh-grants: after take-over the list in force is not replaced by one composed past a failed read or one dropping its users for want of a hash; the switch's effect says when the bus node does not watch the key (review of the switch, issue 1945). Pins mesh-sdk c2a410bb. 2026-10-26 12:31:31 +01:00
jschoubben e1f1b5d18b Checker: a package with skipped tests is withheld from the kept verdicts, not the whole result (issue 890)
mesh/delivery merged: source.main moved to novox/mesh-delivery@4861d1f78d28
mesh/delivery-group group fix/checker-skips-not-kept merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-catalog#492 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:29:14 +01:00
jschoubben f0fe454145 Checker: a result with skipped tests keeps no verdict, so a skipped pass is run again (issue 890) 2026-10-26 12:23:57 +01:00
jschoubben 4026f6c188 checker: a check is never held by a turn its own heads wait in, the node-engine read at its running commit is named and carried when it moves, and a gate bucket made after the start is opened before the next claim (ADR 0548 §3, §6; issue 568, review round 1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@848c6fdc5648
mesh/delivery-group group feat/548-one-check merged: every member is merged
mesh/check failed: ace: the check of novox/mesh-delivery#65 refused: the installed go version could not be read: `go version` with GOTOOLCHAIN=local f…
2026-10-26 12:14:50 +01:00
jschoubben 29cfb8c177 Pin mesh-sdk at the group's head 83ee7769 (issue 1945) 2026-10-26 12:06:58 +01:00
jschoubben a092ee9ff9 mesh-grants writes the user list sealed to the bus node and names it in user-list.sealed; the switch turns that key; take-over state says switched only once the bus node loaded it (ADR 0547 clauses 7, 8; issue 1945)
The bus node can only take over a list that exists: each pass composes the accounts block with the vault's hashes, seals
it to the key the bus node says in its have, puts it in the part's own object bucket by digest and writes the key with
digest, revision, user count, users left out and which list to follow. take-over is refused, naming why, while no list is
sealed to the bus node's current key or the loaded list does not grant the bus node its read.

Review of fix/1945-takeover-blockers: no responders on the router's record is an error, not 'no record'; an answer whose
record cannot be reached is not closed; an ask that failed to publish is taken back at the router; an error acting on an
answer is logged. Pins mesh-sdk 175a5567.
2026-10-26 11:59:41 +01:00
jschoubben 6c3cbee35c checker: pin mesh-sdk at the group's head f5c233a9288c, which holds the merge turns and the carried verdict (ADR 0548 §10; issue 568) 2026-10-26 11:45:13 +01:00
jschoubben 4b23db28b3 mesh-grants: an open take-over ask is always closed (router record, expiry, failed publish, any outcome), the switch is said recorded not switched, the refusal condition is cleared, the excuse is limited to the claiming machines' subscriptions (issue 1945) 2026-10-26 11:43:39 +01:00
jschoubben 646f83642a Indexer: read a registered module at the source's head when it names one, else at the registered commit
mesh/delivery merged: source.main moved to novox/mesh-delivery@11d3678984af
mesh/check pass 646f83642a01 on 11d3678984af | unit 171 run, 502 kept | integration 181 run, 568 kept | composition passed in 39ms on novox
2026-10-26 11:41:46 +01:00
jschoubben d9bfba1e62 Indexer: read one directory of a registered module's repository, leave a repository the source names to its merges, accept master as a trunk 2026-10-26 11:36:22 +01:00
jschoubben f1bb7747ae checker: one integration test through the real bus proves a merge turn holds a check off until it goes and a carry of unchanged digests runs nothing; it extends the equal-inputs test, so the count stays (ADR 0548 §3, §5; issue 568 D7) 2026-10-26 11:36:15 +01:00
jschoubben d2eb3e5ecf checker: the line of a carried verdict names the head on the new trunk, then where it was carried from; a verdict being carried is pending and says so (ADR 0548 §6, §8; issue 568 D6) 2026-10-26 11:35:16 +01:00
jschoubben b0b1cf4f01 mesh-grants: a retired verb and a multiply claimed mesh seat's added subjects are named, not counted, in the comparison (issue 1945) 2026-10-26 11:34:52 +01:00
jschoubben d91ea41f64 checker: only a head git refused gets a whole check after a trunk move; a conflicting carry fails naming the files and asks nothing (ADR 0548 §7; issue 568 D5) 2026-10-26 11:34:08 +01:00
jschoubben 5fed429827 checker: a verdict names the siblings it read at their trunk, and a sibling's move is carried as a trunk move is (ADR 0548 §6; issue 568 D4) 2026-10-26 11:33:35 +01:00
jschoubben 9a569bfb8d mesh-grants: derive that a bundle loads from build.artifacts, as the indexer's manifest has no derived bundles, so route-proxy's helper is carried (issue 1945) 2026-10-26 11:33:28 +01:00
jschoubben e2c41ffa5a checker: a carry keeps the composition it passed while no composed fingerprint changed, and composes again otherwise (ADR 0548 §5; issue 568 D3) 2026-10-26 11:32:31 +01:00
jschoubben 13e4ed8515 checker: a trunk move never ends a check; once it ended, its pass is carried and names the commits it was carried from, and a failed carry names the move (ADR 0548 §1, §5; issue 568 D2) 2026-10-26 11:30:46 +01:00
jschoubben c66fe11e16 checker: claim no check of a repository an approved merge holds a turn on, and claim it when the turn goes (ADR 0548 §3; issue 568 D1) 2026-10-26 11:29:25 +01:00
jschoubben 75d0eb32be mesh-grants: take-over and hand-back are asked of the operator on their channel and run on their yes after the refusals; take-over no longer always refused; grants_take_over_state (issue 1945) 2026-10-26 11:26:37 +01:00
jschoubben c44b9f91cd Indexer: gofmt 2026-10-26 11:23:27 +01:00
jschoubben 82da252cf0 Indexer: index the manifests of modules registered from their own repositories, so the grants part has no gap for them 2026-10-26 11:23:21 +01:00
jschoubben ce72b67cef mesh-grants: hear the condition events through the controller's durable consumer and acknowledge them; the provided mesh-wireguard manifest always wins (issue 1945) 2026-10-26 11:20:42 +01:00
mesh-tools 32bc4f3c48 Merge pull request 'Checker: read every test result of a level and name every failure in the verdict, the try and the status line (issue 1921)' (#62) from fix/1921-every-failure-r into main 2026-10-26 10:18:11 +00:00
jschoubben 2a94d16834 mesh-grants: ask the controller's root-free verb and wake on its condition events, so a verified sender is composed (issue 1945)
The part composed every membership with no root-free input, so telegram's verified-sender was dropped and the membership held as a shadow
difference. It now asks mesh-controller.root-free for the machines a composition needs, per pass, failing closed and said once, and wakes
on root-not-free condition-raised and condition-cleared.
2026-10-26 11:11:43 +01:00
jschoubben da7baa5e39 mesh-grants: take mesh-wireguard's manifest from mesh-sdk, where the mesh provides it, so its node-tools grants are composed (issue 1945) 2026-10-26 11:11:43 +01:00
mesh-tools ed1125b66d Merge pull request 'Assigner: take a stale ask away only once its held condition's clear was said, so a failed clear is retried (issue 1906)' (#61) from fix/1906-clear-retried into main 2026-10-26 10:09:42 +00:00
jschoubben 366bc750da Assigner: clear a stale hold's condition before taking its ask away, so a failed clear is retried at the next pass (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ed1125b66d91
mesh/check pass 366bc750da87 on ed1125b66d91 | unit 168 run, 489 kept | integration 21 run, 712 kept | composition passed on ace
2026-10-26 11:03:09 +01:00
jschoubben 04a56b7aff Name every failed test in a check's verdict and a try's failed list (issue 1921)
mesh/delivery merged: source.main moved to novox/mesh-delivery@32bc4f3c48b2
mesh/delivery-group group fix/1921-every-failure-r merged: every member is merged
mesh/check checking 04a56b7aff66 on 32bc4f3c48b2
The checker stopped reading a level's test results at the first that did
not pass and kept only its first failure, so a check named one fault per
round. It now reads and counts every result and writes every failed test
with its first line into the verdict's Failures and the try's failed
list (at most 50, then how many more); Failed keeps the first one's
words whole (issue 1901). Pins mesh-sdk fix/1921-every-failure.
2026-10-26 10:58:13 +01:00
mesh-tools 2f697216ee Merge pull request 'Assigner: take away hold, restore and record asks whose fingerprint the trunk left behind (issue 1906)' (#60) from fix/1906-stale-asks into main 2026-10-26 09:49:22 +00:00
jschoubben 769dab43a2 Assigner: keep the held record of a fingerprint whose asks went stale (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2f697216eee4
mesh/check pass 769dab43a24d on 2f697216eee4 | unit 168 run, 488 kept | integration 178 run, 554 kept | composition passed on ace
The held record is the memory that a fingerprint failed (ADR 0408 section 4); a revert returns the
trunk to exactly it, and without the key rule 3 would move the canary onto the failed build with
nothing asked. A stale hold ask now clears only the condition (raised again on a revisit), and the
hold is asked again under a new asking. Also read the single want key for a record ask.
2026-10-26 10:33:30 +01:00
jschoubben 98a05997ae Assigner: keep a held-back node's record ask, and clear the held record and condition of a stale hold (issue 1906)
Review round 1: a record ask is about the composition of any fingerprint a node may still want
(trunk, want, have), not only the trunk's, so deleting it looped with the wants rule that asks it
again and the operator was never asked. It is now stale only when the module left the trunk or no
read composition of those fingerprints has its digest. A hold or restore ask taken away as stale
also takes its held record and clears its held condition.
2026-10-26 10:31:01 +01:00
jschoubben 07dee528c5 Assigner: take away hold, restore and record asks whose fingerprint the trunk left behind (issue 1906)
An ask outlived its fingerprint: a lift left the restore asks of its fingerprint on every node,
and no pass removed an ask of a fingerprint that is no longer the module's trunk fingerprint,
so the operator kept being asked about a module that runs the trunk healthy. Each pass now
takes those asks away (the teller then withdraws the router ask), and a lift takes the restore
asks of its fingerprint. Nothing is taken away while the trunk or a composition is unread.
2026-10-26 10:25:50 +01:00
mesh-tools 7ddbf40715 Merge pull request 'mesh-grants: read manifests as the indexer does and say every refused one, so claude-code counts as placed (issue 1945)' (#59) from fix/1945-grants-inputs into main 2026-10-26 09:00:58 +00:00
jschoubben aea22bf793 mesh-grants: read manifests for their inputs, as every delivery part does, not against a list from the controller
mesh/delivery merged: source.main moved to novox/mesh-delivery@7ddbf40715ff
mesh/check pass aea22bf793b4 on 7ddbf40715ff | unit 168 run, 479 kept | integration 178 run, 545 kept | composition passed on novox
The part reads seats, buckets, logs and streams, never a config value, and
the merge gate already judged the fallbacks; reading the controller's
predefined mirror would tie the part to the leaving controller (ADR 0547
clause 1). ParseManifestForInputs takes a fallback as it declares itself.

Issue: 1945
2026-10-26 09:36:18 +01:00
jschoubben 72889be87f mesh-grants: judge manifests against the predefined keys on the bus, and say every refused one
A manifest falling back to a predefined key (claude-code, dunst) was parsed
with no list, refused silently, and its module counted as placed nowhere, so
three memberships would lose claude-code's reach addresses and were held back.
The part now reads the list the controller mirrors (mesh-controller.predefined,
key list), follows it, and lists every manifest it cannot read in its gaps,
its pass (refused) and its log.

Issue: 1945
2026-10-26 09:33:29 +01:00
mesh-tools 86db64904b Merge pull request 'mesh-grants: create a declared module's missing buckets, logs and streams on the event; publish memberships past retired seat verbs (issues 1790, 1945)' (#58) from fix/1790-grants-create-buckets into main 2026-10-26 08:17:22 +00:00
jschoubben 12f16bbebd mesh-grants: create a declared bucket, log or stream the bus lacks, so a new module state no longer waits for a controller start
mesh/delivery merged: source.main moved to novox/mesh-delivery@86db64904b2a
mesh/delivery-group group fix/1790-grants-create-buckets merged: every member is merged
mesh/check checking 12f16bbebd40 on 13380e3d0e52 | novox/mesh-delivery#58 in group fix/1790-grants-create-buckets
A module's new state reached the bus only when the controller started (issue 1790, met three
times: puts, queues, mesh-vault's bus-accounts). The operator allowed one more grant in the
controller's composer on 2026-10-26 08:54, creation only: the part's process now creates each
bucket, log or stream a placed module's manifest declares and the bus lacks, with the same
configuration the controller asserts (mesh-sdk BucketStreamConfig, Log.Config,
ModuleStreamConfig), on the event that brought the manifest, never on a timer. It says asserted
and logs one line per module; a stream the bus holds is never updated or deleted, its difference
only said; a create refused is said and tried at the next event; one created at this manifest
and gone since is made again only by hand. grants_assert asks the process to do the same for one
module now. The user list, the grants and any change to a held stream stay in shadow. mesh-sdk
moves to fix/1790-grants-create-buckets, whose composer grants this, so the shadow list is
composed as the controller's.

Issue: 1790
2026-10-26 09:11:23 +01:00
jschoubben 4ec40eec9a mesh-grants: publish a membership that drops only a retired seat verb, and say every one held back
The guard that holds back a membership losing anything held back five since 01:24: the frozen
controller's memberships still serve the delivery seat's mark-ready, which mesh-sdk d506f21
retired on purpose (ADR 0421), so every membership composed with today's seat catalogue looked
like a loss. A seat verb the catalogue defines no longer is now dropped from the comparison and
named in the pass (retired); any other loss, a module's own seat's verbs included, is still
refused. The log named only the first held membership: it now says each one, with what it would
lose, whenever the set changes, and grants_membership with neither node nor module lists them.

Issue: 1945
2026-10-26 09:11:03 +01:00
mesh-tools 13380e3d0e Merge pull request 'mesh-grants: the shadow user list takes each user's hash from the vault and says per user whether one is held (issue 1945, ADR 0547 clause 9)' (#57) from fix/1945-vault-hashing into main 2026-10-26 07:48:07 +00:00
jschoubben 9132b6a296 mesh-grants: take each user's hash from the vault's bus-accounts and say per user whether one is held, so take-over waits until every user has a credential
mesh/delivery merged: source.main moved to novox/mesh-delivery@13380e3d0e52
mesh/check pass 9132b6a296b9 on 13380e3d0e52 | unit 168 run, 473 kept | integration 178 run, 539 kept | composition passed in 36ms on novox
ADR 0547 clause 9: the vault writes the hash of each bus account it holds into bus-accounts.<user>. The grants part
watches that state (no timer), carries the hash with each shadow user in memory only, and its user-list tool says per
user whether a hash is held and how its line would differ from the loaded one (same, adds, left out for want of a
credential, loses its login); never the hash, since bcrypt salts make two hashes incomparable. Take-over names the
users with no hash.

Issue: 1945
2026-10-26 02:29:54 +01:00
mesh-tools bcb5aafdbe Merge pull request 'Builder: given-up asks no longer hold a re-ask back for ever, and a tool reads a build seat's queue (issue 1945)' (#56) from fix/1945-reask-dead-ahead into main 2026-10-26 00:24:30 +00:00
jschoubben 3e8bd9b0de Builder: count only deliverable asks ahead, drop the old message of an ask asked again, and read a seat's queue (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@bcb5aafdbe80
mesh/check pass 3e8bd9b0def2 on bcb5aafdbe80 | unit 168 run, 472 kept | integration 178 run, 538 kept | composition passed on shanks
mesh-delivery #51's re-ask never fired for mesh-build-node.4e7b118e: messages the
SEAT_NODE_BUILD_GO workers had given up on stay on the work queue, and each counted
as waiting ahead, so the ask was never asked again (note 62). A message ahead now
waits only when no node wrote its step and no ask of its kind asked after it was
taken; the old message of an ask asked again is deleted from its stream; and
builder_seat_queue (answered by the builder through its queues state, as only its
account may read the stream) shows each message and whether it still waits.
2026-10-26 01:19:30 +01:00
mesh-tools b842b9422d Merge pull request 'Builder and checker read tools read only what they answer, in parallel and bounded (issue 1827)' (#55) from fix/1827-bounded-reads into main 2026-10-25 23:44:25 +00:00
jschoubben 9db15c7cf4 Builder and checker read tools read only the keys they need, in parallel, at most 1000 a call (issue 1827)
mesh/delivery merged: source.main moved to novox/mesh-delivery@b842b9422d2a
mesh/check checking 9db15c7cf4fc on b842b9422d2a
builder_stuck, checker_kept, checker_verdicts, checker_status, checker_previews, checker_pull and checker_try listed a
bucket of thousands of keys and read each value (builder_stuck also its history) one round trip at a time before
filtering, so they timed out at 30 s. Now keys are narrowed by what the key carries first, history is read only for rows
kept, values are read sixteen at a time, at most 1000 a call with the answer saying it was cut, steps newest first, a new
try reads only the newest 64 tries, and each tool logs the values read and the time taken. Same kind as the view's fix
(PR 52); the helper is copied, not put in mesh-sdk, as a few lines do not earn a group delivery.
2026-10-26 00:39:08 +01:00
jschoubben ba47b47e24 Merge pull request 'mesh-delivery-view: pin mesh-sdk at its main so a have's user-list field decodes (issue 1945)' (#54) from fix/1945-view-sdk-pin into main
Reviewed-on: #54
2026-10-25 23:06:21 +00:00
jschoubben b566563dc5 mesh-delivery-view: pin mesh-sdk at its main so a have's user-list field decodes (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ba47b47e2421
mesh/check pass b566563dc592 on ba47b47e2421 | unit 576 run, 63 kept | integration 177 run, 538 kept | composition passed on shanks
2026-10-26 00:01:12 +01:00
jschoubben 2a856e2a76 Merge pull request 'mesh-grants: publish memberships for new placements now; the user list, assertions and grants in shadow (ADR 0547 milestone 1)' (#53) from fix/1945-memberships into main
Reviewed-on: #53
2026-10-25 22:44:11 +00:00
jschoubben ce7d60cb6e Merge pull request 'The delivery view reads only the keys a call names, in parallel, so node_haves and assigner_wants answer within their time (issue 1945)' (#52) from fix/view-reads-fast into main
Reviewed-on: #52
2026-10-25 22:03:21 +00:00
jschoubben 26ba7404bf Pin mesh-grants to mesh-sdk b7b5c6e, whose composer grants it reading every membership (review of mesh-grants, B1)
mesh/delivery merged: source.main moved to novox/mesh-delivery@2a856e2a76ea
mesh/delivery-group group fix/1945-memberships merged: every member is merged
mesh/check checking 26ba7404bfc4 on ce7d60cb6ed0 | novox/mesh-delivery#53 in group fix/1945-memberships
2026-10-25 22:54:01 +01:00
jschoubben b4af9cdb7a mesh-grants publishes only a first or an additive membership, and holds no lock across the bus (review of 1ff5049)
Composed without a root-free judgement, a membership would drop a kind's verified-sender and the
operator's channel approvals would stop being honoured; any loss now stays a shadow difference,
named. A tool no longer waits on a pass's round trip. compose-now says the controller serves no
verb to write a module's grant again.
2026-10-25 22:52:00 +01:00
mesh-tools db53ec975f Merge pull request 'Delivery parts keep running when the bus refuses a bucket they can work without; the builder asks a lost build again; the assigner says why a node gets no want (issue 1945)' (#51) from fix/1945-ask-again-and-why into main 2026-10-25 21:43:11 +00:00
jschoubben 1ff5049d03 mesh-grants publishes memberships on change; the user list, assertions and grants stay in shadow (issue 1945)
Since release 6 nobody publishes memberships, so a module placed since cannot serve its seat
or reach a new state (the operator, 2026-10-25 22:31). The part publishes each membership on
the assignments stream only when it differs from the one last published, composes from the
manifest the wanted build names, and says a refused publish and tries again at the next event.
Pins mesh-sdk 881da7f, whose composer grants mesh-grants that publish.
2026-10-25 22:37:06 +01:00
jschoubben 41f25d3129 Read only the keys a view tool needs, in parallel, so node_haves and assigner_wants answer within the call's 30 s
mesh/delivery merged: source.main moved to novox/mesh-delivery@ce7d60cb6ed0
mesh/check pass 41f25d3129fd on 73be3df28cd4 | unit 21 run, 579 kept | integration 21 run, 655 kept | composition passed on shanks
On 2026-10-25 node_haves and assigner_wants timed out at 30 s four times, even narrowed to one node and one module,
while assigner_want answered at once. Each listed every key of its bucket and then read every value one after the
other, filtering only afterwards; every value is a round trip from the tool through the runtime to the bus, which
opens the bucket and gets the key, so a few hundred keys outlasted the call. Now a node and a module read that one
key directly, a node or a module alone filters the keys before any value is read, and the values are read sixteen
at a time, at most 1000 a call, the answer saying when it was cut. indexer_modules, indexer_candidates and
builder_builds read their values the same parallel way. The kind: a read tool that reads a whole bucket key by key;
every list tool of the view is covered.
2026-10-25 22:33:02 +01:00
jschoubben 4628ba26b4 Delivery parts: take the essential and optional bucket opens from mesh-sdk (issue 1945)
mesh/delivery merged: source.main moved to novox/mesh-delivery@db53ec975f01
mesh/delivery-group group fix/1945-ask-again-and-why merged: every member is merged
mesh/check pass 4628ba26b41a on 73be3df28cd4 | unit 167 run, 443 kept | integration 177 run, 509 kept | composition passed on 3 machines
The open helpers were copied into seven parts; code shared between
modules lives in mesh-sdk, so they are mesh-sdk process's now and the
parts are pinned at fix/1945-optional-buckets (e0d1de04), moved to its
main once it merges, one delivery group with mesh-sdk.
2026-10-25 22:30:22 +01:00
jschoubben 20027e53ff Delivery parts: a bucket the bus refuses at start stops a part only when it cannot work without it (issue 1945)
Met twice on 2026-10-25: the builder of #48 exited on a refused read of
the gate's source and crash-looped, and its new reasks bucket would have
done the same. A grant reaches the bus with a delivery, so a refusal or a
timeout at start is often only a grant not there yet; exiting for it
stopped the whole part until a hand step on the bus.

Every startup open in the parts is now essential or optional. Essential
ones (a part's own state and the input it exists for) still stop the
start, with an error naming the bucket and its grant. Optional ones are
said and left: watches the bus refuses too. The builder builds trunk keys
without the gate's source, which it opens again at the next candidate
that asks a trunk head; toolchains and previews open again at their next
event, as a missing bucket did. builder_ask_again says that asking a
running step again builds it twice.
2026-10-25 22:29:31 +01:00
jschoubben d4a054bfed Assigner: say why a node gets no want, in the journal and in its removed want (issue 1945)
A new assignment whose trunk build was not done fell through the rules
with nothing said: no want, no neither-list entry, no log line, and the
rule holding it back was computed and dropped. The pass now logs the rule
when a node's reason appears or changes and when it has a want again, one
line per change. A node whose want is removed and that does not run the
module gets the reason in that removed want's words, so assigner_want
answers why; a node with no key gets no removed want, since its node agent
would then remove the module and stop holding its resources back.
2026-10-25 22:29:03 +01:00
jschoubben c623e8d235 Builder: ask again an ask its stream can no longer answer, and give it tools to read and ask again (issue 1945)
A build ask its seat's stream gave up on, dropped or ended with no step was
waited for with no end, and on restart taken over and waited for again: no
build node would ever answer it, and nothing could ask it again (trunk key
of mesh-build-node at 4e7b118e, building since 19:06; seven keys at 18:44).

At the step bound, a context's bound and no timer of its own, the builder
reads what the bus says of an ask with no step: gone from its stream, or
first in line while a node took a later ask. Such an ask is asked again
under a new id the key names, logged and said as build.asked-again, at most
three askings, then the key fails with why. The bus's max-deliveries
advisory would tell it sooner, but the builder is granted neither it nor the
workers' information, and the controller is not changed here.

builder_stuck reads every waiting key with its asks, askings and steps;
builder_ask_again writes the new reasks state, which the builder watches
and answers.
2026-10-25 22:29:03 +01:00
jschoubben 998fe3a515 Add mesh-grants in shadow: compose the bus's user list, memberships and assertions from the bus and compare, writing only its own state (ADR 0547 clauses 5-8, issue 1945)
The grants part must compose with the controller's own composer before it can take over the
list; in shadow it proves the two agree, names every input the bus does not carry yet, and
takes no effect. It claims no seat (mesh-sdk's seat list does not name mesh-grants for the
running composer), serves module tools, and keeps running when a read or write is refused.
2026-10-25 22:27:47 +01:00
mesh-tools 73be3df28c Merge pull request 'The assigner writes wants whatever keeps its puts from opening, and never moves a held put's want (ADR 0546, issue 1945)' (#50) from fix/assigner-starts-without-puts into main 2026-10-25 21:22:55 +00:00
jschoubben eb65535f45 A want marked put and held stands while its put is not heard: the rules give it back only on a put heard released, so a start before the puts are open moves no node (ADR 0546 clause 5)
mesh/delivery merged: source.main moved to novox/mesh-delivery@73be3df28cd4
mesh/check pass eb65535f45db on 73be3df28cd4 | unit 167 run, 433 kept | integration 177 run, 499 kept | composition passed on ace
2026-10-25 22:11:44 +01:00
jschoubben fa6cdbbe61 The puts' watch never gates the assigner's first pass: it is not waited for at the start, and one that ends before it handed what stands only stops the puts (ADR 0546) 2026-10-25 22:06:36 +01:00
jschoubben 462ec9cfb1 The assigner opens the puts' bucket again on a later pass once it is there and granted, so a put works without a restart (ADR 0546, ADR 0201) 2026-10-25 21:59:24 +01:00
jschoubben 9950689a11 The assigner starts and writes wants whatever keeps the puts' bucket from opening or being watched (not there, not granted, no answer): it takes no put further until it starts again, said in its journal (ADR 0546) 2026-10-25 21:58:36 +01:00
mesh-tools 71a6a5bf77 Merge pull request 'Put a version of a module on a node with one tool call: the assigner's put, release and puts (ADR 0546)' (#49) from feat/put-version into main 2026-10-25 20:52:05 +00:00
jschoubben 22d4cbd144 The assigner's start test expects every state its manifest declares, read from module.json, not a count: the state puts broke the count of four (ADR 0546)
mesh/delivery merged: source.main moved to novox/mesh-delivery@71a6a5bf7717
mesh/check pass 22d4cbd14423 on 82180bffd2c6 | unit 22 run, 578 kept | integration 251 run, 425 kept | composition passed on 2 machines
2026-10-25 21:48:32 +01:00
jschoubben 7b618b2317 Pin the assigner and the teller to mesh-sdk main db26f7e, which carries the put (ADR 0546)
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-assigner/cmd/mesh-delivery-assigner.TestTheAssignerStartsAndSaysWhatItKeeps: the assigner said {V:1 Modu…
2026-10-25 21:37:17 +01:00
jschoubben d8477c71b1 A put asks under its own asking and writes its want only while it stands: an earlier put's answer never counts, a put released meanwhile writes nothing, the bus's want is written only for the act the yes bound, an expired ask refuses the put, and a warrant not recorded is said in the put's words (ADR 0546, review round 1) 2026-10-25 21:36:38 +01:00
jschoubben 45fae9da8a Serve the put as the assigner's own tools from the compositions and grants the controller already writes, with no seat and no compose verb: the controller takes no change (ADR 0546) 2026-10-25 21:36:38 +01:00
jschoubben 2f742fd3fe Put a version of a module on a node with one verb: the assigner's seat put, release and puts, held until released, the bus on the operator's warrant after a snapshot (ADR 0546) 2026-10-25 21:36:38 +01:00
mesh-tools 82180bffd2 Merge pull request 'Indexer: write a module's value only when its fingerprint changes, so a catalogue merge no longer floods the take-in (issue 1945)' (#48) from fix/indexer-writes-on-change into main 2026-10-25 20:35:24 +00:00
jschoubben ac82187b85 Indexer integration test: a disagreement keeps a computed fingerprint, as issue 1940's fix decided
mesh/delivery merged: source.main moved to novox/mesh-delivery@82180bffd2c6
mesh/check pass ac82187b852c on 82180bffd2c6 | unit 167 run, 432 kept | integration 177 run, 498 kept | composition passed on 2 machines
The test predates the fix that keeps a held fingerprint while builders disagree (#47, 3e7f852) and still asserted the
blanking that fix removed, so it fails on main 4efe6cf as well. It now asserts the fingerprint is kept after n2's
assignment, and that a merge moving avahi during the disagreement gets none until the builders agree.
2026-10-25 21:28:03 +01:00
jschoubben 8d38c6bac9 Write the source revision a candidate was previewed onto, so the builder merges its preview onto that trunk head
mesh/delivery closed: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
Candidate.Base was never written non-zero (review round 2): the builder's base lookup would have found no trunk for any
live candidate and built no preview at all. The indexer now writes, whenever it writes a candidate's fingerprints, the gate's
source revision whose trunk heads it read the previews onto; that head of the pull request's repository is the preview's
trunk. No mesh-sdk change: the field and its meaning were there. The builder's test now takes the candidate the indexer's
own test writes (testdata/candidate-written.json, kept equal to it), not a hand-made one.
2026-10-25 21:16:20 +01:00
jschoubben 9afef800c0 Merge a candidate's preview onto the trunk head it was previewed onto, not the module value's commit
The builder took a candidate build's merge base from the module's value commit. Now that a value keeps the commit its
fingerprint first appeared at, that commit can be many merges behind the trunk, so a preview would be built onto a stale
trunk (review round 1). The base is now the head of the pull request's repository in the gate's source revision the
candidate names (Candidate.Base): the builder reads mesh-delivery-gate.source, watched with its history (a direct get of
one revision is not what a read grants), and a candidate heard before its revision is built once the revision is heard.
2026-10-25 21:13:59 +01:00
jschoubben eaf27f36af Write a module's value only when the module changed, so a merge no longer rewrites every module of its repository
The operator, 2026-10-25: only if a module has a new version in commit x is a new value written naming commit x. A merge
wrote every module of the repository at the new commit, fingerprint unchanged (about 186 keys per mesh-catalog merge), and
the controller's take-in took each in again: twice tonight it flooded. Now a merge that gives a module the value it holds
but for the commit writes nothing, and the value keeps the commit its fingerprint first appeared at. The kind: a write
that differs only in where it was computed. The other instance, a source revision re-stamping Src on every module at its
repository's head after any merge anywhere, now stamps Src once. The trunk head stays the gate's source.main.
2026-10-25 21:13:59 +01:00
jschoubben 4efe6cfb26 Merge pull request 'Builders: a stale toolchain report no longer counts, and a disagreement no longer blanks a module already fingerprinted (issue 1940)' (#47) from fix/1940-stale-toolchain-record into main
Reviewed-on: #47
2026-10-25 20:09:27 +00:00
jschoubben 3e7f852472 Count a build node's toolchain report only once written since its assignment, and keep a computed fingerprint while builders disagree (issue 1940)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check failed: integration: mesh-delivery-indexer/cmd/mesh-delivery-indexer.TestIndexerCountsOnlyTheAssignedBuildNodesToolchains: within 5 s of n2…
On 2026-10-25 20:56 assigning mesh-build-node to ace again brought ace's report of 2026-10-23 (v26.12.0) back into the
agreement before the module ran there; the indexer blanked every module's fingerprint to "builders disagree" and the
controller's take-in refused modules already computed and built.

- The toolchain gate (indexer, builder, checker; still one file in three) leaves a report out, said once, until its node
  writes it since its current assignment: the assignment's own time, or the revision that opened it.
- The indexer keeps a module's held fingerprint while a language has no agreed toolchain, when its kept inputs with the
  toolchains it was computed with give it again; only a new fingerprint (a merge, a base, an agreed toolchain moved)
  needs the agreement.
2026-10-25 21:03:16 +01:00
jschoubben 5ba6881f9f Merge pull request 'Assigner rule 6 per node: a node is granted only when the grant names its own account (issue 1945 note 35)' (#46) from fix/1945-credential-when-row-without-seal into main
Reviewed-on: #46
2026-10-25 19:31:51 +00:00
jschoubben 61196c99e6 Hold a node's want until the grant names its account (issue 1945, rule 6 per node)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1945-credential-when-row-without-seal held: a member is held: novox/mesh-controller@fbf14b68dbab (merged without a passing check:…
mesh/check pass 61196c99e6a4 on 5ba6881f9fd3 | unit 163 run, 421 kept | integration 173 run, 487 kept | composition passed on ace
Rule 6 read only whether a grant of the module's fingerprint existed. A grant
written for the module's other nodes then let a node whose account was never
issued or sealed be wanted: on 2026-10-25 at 20:04 ace's mesh-build-go was
wanted with no ace.mesh-build-go on the bus, and crash-looped 20 times for
its missing credential. A grant that names accounts now grants a node only
when it names that node's own or a process's account; one that names none
grants every node, as before.
Research 072: the grants piece (memberships and credentials heard from placement).
2026-10-25 20:13:39 +01:00
jschoubben 250364a57b Merge pull request 'The assigner follows a kept composition's new digest, so a want held at an earlier build gets today's config (issue 1945, ADR 0545)' (#45) from fix/1945-kept-compositions-follow-config into main
Reviewed-on: #45
2026-10-25 18:38:03 +00:00
jschoubben 73779a5dd1 Let a want held back at an earlier fingerprint follow that fingerprint's new revision (issue 1945, ADR 0545)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1945-kept-compositions-follow-config held: a member is held: novox/mesh-controller@ff3cc47027e0 (merged without a passing check: …
mesh/check failed: integration: github.com/novox/mesh-controller/cmd/mesh-controller.TestTheDeclarationsStreamGoesOnceEveryNodeRunsTheNewEngine: with …
The controller now composes a module's kept compositions again when its config
changes, so going back keeps today's config. A want held back at an earlier
fingerprint kept the digest it was first written with, and the node applied
the old revision from the bucket's history: on g14 a rollback of mesh-build-go
put its capacity back to 0. The want now names the new digest, asked through
the record hold as any move of a record module is. A want held at F by the
rules still waits for them, as a new fingerprint does (ADR 0539 §5).
Research 072: milestone 2, the composer reacting to the change of an input.
2026-10-25 18:24:44 +01:00
jschoubben ec76ab77c0 Merge pull request 'Assigner: a held module is on the held list, never on neither, and a neither from before a restart is cleared (issue 1945)' (#44) from fix/held-is-not-neither into main
Reviewed-on: #44
2026-10-25 16:11:33 +00:00
jschoubben 91d041b62a mesh-delivery-assigner: a held module is never on neither list (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/check pass 91d041b62a1a on ec76ab77c058 | unit 163 run, 421 kept | integration 173 run, 487 kept | over bound: integration (slowest integration m…
systemd-resolved, held on shanks by the operator under rule 2, was raised
as an urgent "on no list" condition and reminded every hour, although its
hold is already said and answered on the held list. A node now goes on
neither list only when it gets no want and the module is not held, by a
hold read at its fingerprint or one this pass raises. The first pass after
a start clears the neither condition of every assigned node not on neither
list, not only those with a want, so a neither raised before the restart
for a held module (or a move waiting for its record ask) is cleared too.
2026-10-25 17:07:50 +01:00
jschoubben 51444a8341 Merge pull request 'mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the runtime's line or say so (issue 1901)' (#43) from fix/1901-package-cause into main
Reviewed-on: #43
2026-10-25 15:07:47 +00:00
jschoubben dd9138310c mesh-delivery-checker: a package sent as its FAIL line alone is still worded, and the words keep their bound exactly (issue 1901, review round 1)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/1901-package-cause held: a member is held: novox/mesh-catalog@155124b59af4 (merged without a passing check: only a person decides…
mesh/check pass dd9138310c48 on 51444a834173 | unit 2015 run, 2883 kept, 5 skipped | integration 2037 run, 2978 kept, 9 skipped | over bound: unit, in…
The build module on main sends a timeout, exit or kill as only 'FAIL <pkg>':
dropping summary lines left empty words. They now say the package failed with
no cause sent. Causes match at column 0 only, summary lines only in go's exact
forms, and the words count the cause, the marker and the newlines within
MaxWords.
2026-10-25 15:58:09 +01:00
mesh-tools 653da54c0d Merge pull request 'Assigner: a running node gets its first want past the ordering rules, a module on neither list is raised, and a failed health holds only once a node is done moving (issue 1945)' (#42) from fix/r6-no-module-in-neither into main 2026-10-25 14:56:26 +00:00
jschoubben f72f57f6e2 mesh-delivery-checker: with no cause told apart, a race report is the cause (issue 1901)
An older build module's output for a race after the tests' PASS is now worded
by the race's heading, access and frame, not by go test's PASS or the report's
separator line (1901/AC5).
2026-10-25 15:52:21 +01:00
jschoubben 975f0ed654 Assigner: a hold lifts over haves not running, the first pass after a start clears stale conditions, and a first want after a removed want is asked (issue 1945, review round 2 of 3d7d65ed)
mesh/delivery merged: source.main moved to novox/mesh-delivery@653da54c0da3
mesh/check pass 975f0ed65460 on 653da54c0da3 | unit 154 run, 419 kept | integration 164 run, 485 kept | composition passed on ace
Phase guard, same kind as the failed-health hold. allRunHealthy counted
every have whatever its phase, so a node with a removed, held or still
moving have kept a hold up for ever: it counts only phase running (healthy
or not) and failed (not healthy). The failed-health hold already ignores
removed and held haves (phase running or failed only).

Stale conditions (B1). The neither and held conditions this process said
lived only in its memory, so one raised by a previous process stayed urgent
for ever once the module had its want or was no longer held. The first pass
after a start clears, once, the neither condition of every assigned module
with a want and not on neither list, and the held condition of every
module's newest fingerprint that is not held; a clear of a condition that is
not open is quiet at the router. A hold raised again forgets that it was
cleared, so a later release clears it again.

Record ask (review note 3). mustAsk wrote no ask when the want before was
removed, so the first-want path could write a recreates or moves-data
composition of a policy-record module with no ask. A removed want is now
no want, and the first want after it goes through the record ask.

Tests first, each seen failing: a hold lifted over removed, held and moving
haves; the record ask after a removed want; a fresh pass clearing what a
previous process raised, once.
2026-10-25 15:47:57 +01:00
jschoubben 706169aa8a Assigner: a failed health holds a fingerprint only once the node is done moving (phase running or failed)
On novox at 15:27:36 a have reading "health failed, phase applying" at the
controller's new fingerprint held its canary two seconds before the new
build was healthy: Wants held on a failed health whatever the phase. A
failed health while the node backs up, applies or checks is a step of the
move, not its outcome, so it holds nothing now.

Every other Health read in the assigner (rule 3's healthy, rule 5's
providedHealthy, the world's lastHealthy) already asks for phase running.
Test from the live row first, seen failing before the fix.
2026-10-25 15:39:45 +01:00
jschoubben 6d2d50b1e1 mesh-delivery-checker: a failure's words are never "PASS"; with no cause, take the panic line (issue 1901)
A try of mesh-controller's status read 'cmd/mesh-controller: PASS': the build
module sent a package failure whose output began with go test's PASS. The
checker now drops go test's summary lines, takes the first panic, fatal error
or signal line as the cause when the build module told none apart (read before
the words are cut), says the tests passed and the program failed after them
when nothing else is left, and cuts a cause to 1 KB so the words keep their
bound (1901/AC5).
2026-10-25 15:38:40 +01:00
jschoubben 3d7d65edce Assigner: a node running an old-form build gets its first want at F once the canary runs F healthy (issue 1945)
An old-form build (not src2:) has no composition, so no want can ever name
what such a node runs. Design limit, decided by the lead (lead-47): its first
want names the newest F, by every rule as usual - the canary (rule 3: this
node is the canary, or the canary runs F healthy), everything F needs (built,
not held, granted, composed) and the providers (rule 5). Otherwise nothing is
written and the neither condition stays. The canary stays in force, and
old-form modules (clipmenu and screen-lock on g14) leave neither list by
themselves, without a hand-written want.

The code already behaved so (an old-form have falls through to Holds); the
case is now named in Wants, and proven: a unit test for both sides of the
canary, the providers and the grant, and the integration test gives g14 F
and clears its neither condition once the canary ace runs F healthy.
2026-10-25 15:32:35 +01:00
mesh-tools 06fd53e5cb Merge pull request 'List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)' (#41) from fix/r6-hold-verbs into main 2026-10-25 14:30:12 +00:00
jschoubben 81182e9dcd Assigner: a running node's first want names the fingerprint it runs, never a newer one, and only a node that runs the module is on neither list (issue 1945, review of f7b23033)
A first want at the newest fingerprint passed over the canary for every module right after the switch. It now names what the node
runs: F when it runs F (by the rules without which there is nothing to want), or an older fingerprint of the new form whose
composition and grant exist, held back from F by the rule; otherwise nothing is written and the node stays on neither list. A new
assignment, with no have, waits for the rules and raises no condition (1945/AC19-AC21).
2026-10-25 15:26:03 +01:00
jschoubben f7b23033a2 Assigner: the integration test of wants on the bus proves a running node's first want and the neither condition; held-condition reads filter by key (issue 1945, try-1792937443523744996) 2026-10-25 15:18:07 +01:00
jschoubben cf26e95691 A view answer rewritten with the same option changes nothing; pin a unit test's clock against the integration tests' (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@06fd53e5cbf7
mesh/check pass cf26e956915c on 06fd53e5cbf7 | unit 154 run, 408 kept | integration 164 run, 474 kept | composition passed in 1ms on 2 machines
2026-10-25 15:15:40 +01:00
jschoubben 8daa5977f9 Assigner: a node running a module with no want gets its first want past the ordering rules, and a module on neither list is raised (issue 1945)
The node's own composition gives up every module with a fingerprint of the new form, so such a module is applied only through its
want. The assigner wrote no want while a rule held it back and the node had none, so clipmenu, screen-lock, i3status-rust and every
module requiring x11-display lay on neither list on g14 and shanks: xorg's health is unknown (rule 5) and the canary never ran them
healthy (rule 3), and their files were removed and never made again.

A node that runs a module and has no want for it now gets its first want by the rules without which there is nothing to want (built,
not held, granted, composed); the canary and providers rules order every move after it. Any module still on neither list is raised
as an urgent condition naming module, node and rule, written to the journal, and cleared once it has a want (1945/AC19-AC21).
2026-10-25 15:10:34 +01:00
jschoubben 30ca8a22bc List holds and let the operator release or keep one again from mesh-delivery-cli (issue 1906)
A hold answered keep on the phone could not be changed: its ask ended with the
answer, nothing listed it, and a kept hold is never lifted by itself. The view
lists every hold with its node, cause and answer (mesh-delivery-cli holds) and
answers the standing hold ask with hold-release or hold-keep through its one
write; the teller takes a later view answer that changes a hold's option into
the answer of that asking, never an older one.
2026-10-25 15:09:40 +01:00
mesh-tools 97e605edb0 Merge pull request 'Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, and keep its cause (issue 1906)' (#40) from fix/r6-hold-asks into main 2026-10-25 13:27:29 +00:00
jschoubben 7ef796b04b Format the assigner's integration test (issue 1906)
mesh/delivery merged: source.main moved to novox/mesh-delivery@97e605edb0ad
mesh/check pass 7ef796b04b92 on 97e605edb0ad | unit 154 run, 404 kept | integration 164 run, 469 kept | composition passed on 2 machines
2026-10-25 14:16:58 +01:00
jschoubben ec93585736 Ask a hold again after a restart, leave a merge ask room, take an answer only for its asking (issue 1906, review)
A hold written again after a lift and a restart reused its first ask id, which
the router drops; the hold's time is now its asking. Seven holds could take the
router's three places for the teller and keep a merge Approve from being asked;
the teller keeps at most two assigner asks open. A release or keep answered for
one hold carried over to a later hold of the same fingerprint; an answer now
counts only for the asking its warrant names.
2026-10-25 14:15:44 +01:00
jschoubben 6b63f13da6 Raise the assigner's asks on the operator channel, lift a hold once every node runs it healthy, keep the hold's cause (issue 1906)
Seven modules stayed held on every machine and the operator was never asked:
the teller only filled assigner asks from the view, a single failed check held
a fingerprint for good, and the held record kept no cause. The teller now asks
every assigner ask as it asks merge asks and writes the phone's answer under
the ask's key; the assigner lifts a hold whose fingerprint runs healthy on
every node that runs the module, says so once and takes its ask back; the held
record names health, phase, when and the node's words.
2026-10-25 14:06:58 +01:00
jschoubben 2268f94ef2 Merge pull request 'The view lists a node's own want among the assigner's wants (issue 1945)' (#39) from fix/r6-node-composition into main
Reviewed-on: #39
2026-10-25 00:18:11 +00:00
jschoubben 48aaecf556 List a node's own want among the assigner's wants, never as unreadable (issue 1945)
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/check checking 48aaecf556c5 on 650a06ceab1d | novox/mesh-delivery#39 in group fix/r6-node-composition
2026-10-25 01:50:00 +02:00
jschoubben 650a06ceab Merge pull request 'R6: the assigner writes wants by rule 7 with the composition digest; the watchman and the view follow wants and haves (issue 1906)' (#38) from feat/r6 into main
Reviewed-on: #38
2026-10-24 23:17:46 +00:00
jschoubben 3e863884fd gofmt the passer's fields after main's merge
mesh/delivery delivering: 0 machine step(s) passed
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/check checking 3e863884fda8 on 650a06ceab1d | novox/mesh-delivery#38 in group feat/r6
2026-10-24 23:33:59 +02:00
jschoubben c21bed20a3 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943) 2026-10-24 23:06:02 +02:00
jschoubben aa31e85665 Merge main into feat/r6: keep R4g's composer at a group's mesh-sdk head beside R6's wants and asks
The assigner reads mesh-controller.builds again (the composer's runningCommits reads the build record of every open
assignment), puts its compose asks on node-build-go and says its conditions on operator-channel.
2026-10-24 23:03:42 +02:00
jschoubben 171bddacab Merge pull request 'R4g finish: the composer as a job on node-build-go's check stream, merge-check.sh gone, reads-outside markers (issues 1295, 1621-1623, 1935)' (#35) from feat/r4g-finish into main
Reviewed-on: #35
2026-10-24 20:57:15 +00:00
jschoubben 65496fc4f7 Merge main into feat/r4g-finish, keeping #37's same-branch mesh-sdk pin rule beside the node-engine check's controller placing (issue 1295)
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
mesh/check checking 65496fc4f7a6 on 171bddacabe2 | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 21:54:46 +02:00
jschoubben 36a1fd197f The switch exception skips the record hold only for a composition the controller judges unchanged; pin mesh-sdk d7a9b96 (issue 1906, ADR 0540 §9) 2026-10-24 21:18:48 +02:00
jschoubben 9a5b3d4881 Pin the assigner, watchman and view to mesh-sdk feat/r6 7f9cc0e (issue 1943) 2026-10-24 20:43:46 +02:00
jschoubben e263049551 Right after the switch a node running F with a have that names no composition is given its want at F with no record hold, the bus included (issue 1906, review round 2 N1) 2026-10-24 20:35:17 +02:00
jschoubben 0f7bc9f531 Pin the assigner, watchman and view to mesh-sdk feat/r6 e45bdc0 (issue 1943) 2026-10-24 20:21:57 +02:00
jschoubben da31740f00 The view's motion events carry the time they are about and call a held or past-bound module stopped, as the messenger reads them (issue 1906 D10) 2026-10-24 18:31:29 +02:00
jschoubben a3815051cc The view says modules in motion: moving, settled and stuck, once for a node, a module and a composition (issue 1906 D10, for the messenger) 2026-10-24 18:27:01 +02:00
jschoubben 8280506cee Start the world's change counter, which the pass of a held backup reads (issue 1906, review round 1) 2026-10-24 18:22:22 +02:00
jschoubben 3b00f3890d Review round 1: a user-list-only revision of the bus is written at once at the fingerprint it runs, before the record hold; pin mesh-sdk a523a8c (issue 1906) 2026-10-24 18:17:25 +02:00
jschoubben fcac03a4f2 Review round 1: the bus with no want is compared with its have; a backup ask is bound to its composition and number and each answer is a distinct want value; the view reads wants and haves, with dotted module names (issue 1906) 2026-10-24 18:15:13 +02:00
jschoubben 14c20b21ee Pin the assigner, watchman and view to the head of mesh-sdk feat/r6 (issue 1943) 2026-10-24 17:52:44 +02:00
jschoubben 333e0cd07d The checker reads a kept manifest as the indexer and the assigner do, so a value falling back to a predefined key is not logged unreadable (issue 1906, issues 1751 and 1921 class) 2026-10-24 17:42:24 +02:00
jschoubben 03e5951dc5 Merge remote-tracking branch 'origin/main' into feat/r6 2026-10-24 17:41:53 +02:00
mesh-tools 4c1d36c9cd Merge pull request 'A try accepts a mesh-sdk pin that is the head of mesh-sdk's branch of the same name, so a group's members can be tried before its pull requests (issue 1921)' (#37) from fix/1921-try-same-branch-pin into main 2026-10-24 15:37:07 +00:00
jschoubben 37d0ebba7a Let the view's manifest test read the emits key the delivered event adds 2026-10-24 16:47:33 +02:00
jschoubben 86fd0bbe96 The view says its event delivered once for each source revision whose wants and haves are all there (issue 1906 D9, for issue 1946) 2026-10-24 16:47:09 +02:00
jschoubben 37a5ab8f1c A node that runs a module of policy record with no want yet is asked like any other, not treated as one that never ran it (issue 1906 D4) 2026-10-24 16:43:01 +02:00
jschoubben 55f19283f4 Do not wait twice at the cleanup for an assigner that ended on its own, which made the refused-start test take ten seconds a bucket 2026-10-24 16:42:28 +02:00
jschoubben e7a3ff1e7e The record policy is the one the controller holds beside the composition, the operator's own choice included; pin mesh-sdk 64a502d (issue 1906 D4, ADR 0539 §14) 2026-10-24 16:41:58 +02:00
jschoubben 991589fdd2 A try of a branch accepts a mesh-sdk pin at the head of mesh-sdk's branch of the same name, so a milestone's repositories can be tried before their pull requests (issue 1921)
mesh/delivery delivered
mesh/check checking 991589fdd20a on 4c1d36c9cd7f
2026-10-24 16:41:39 +02:00
jschoubben f947070b7b The view shows the controller's judgement of the move beside each want (issue 1906 D7, ADR 0539 §14) 2026-10-24 16:38:24 +02:00
jschoubben 04c2c60f02 The assigner acts on answers: a release writes the held want again, a backup answer goes into the node's want, a record move waits for its yes, a restore answer sets a data module back (issue 1906 D4) 2026-10-24 16:37:37 +02:00
jschoubben 0df20b5d77 The assigner's seventh rule: a want names the digest of its composition and waits for it; a revision at the same fingerprint is a new want; the node's own want (issue 1906 D2, ADR 0539 §6) 2026-10-24 16:33:48 +02:00
jschoubben 806c8bf0e4 Pin the assigner, watchman and view to mesh-sdk's release 6 head (issue 1943 S3) and give their test wants a composition digest 2026-10-24 16:32:09 +02:00
jschoubben 12b01badb0 Prove the assigner replaced mid-pass resumes with nothing lost or written again (issue 1906 D8) 2026-10-24 16:27:34 +02:00
jschoubben 7b18696cd5 Give the data module's test manifest the directory and backup the catalogue asks of an irreplaceable item 2026-10-24 16:26:43 +02:00
jschoubben 204a59dab4 The assigner asks whether to restore a data module whose held fingerprint failed its health check, and raises the condition for a held fingerprint (issue 1906 D3) 2026-10-24 16:26:20 +02:00
jschoubben f048e2b443 The view shows each want beside the have of its node, and a source revision's derived state, in place of the comparison with sends (issue 1906 D7) 2026-10-24 16:23:51 +02:00
jschoubben 9d2e9dc6c9 The watchman mirrors each want of the assigner as a pulse owed by the node's agent until the have names its fingerprint (issue 1906 D6) 2026-10-24 16:22:02 +02:00
jschoubben 024a3787da End the assigner's comparison with the send: the controller sends nothing to compare with, so the differences bucket is kept read-only (issue 1906 D5, ADR 0539 §8) 2026-10-24 16:20:43 +02:00
jschoubben 4fd6b934c8 Read the assigner's wants, differences and the haves with tools, so the gap between wants and sends can be read live (issue 1906 D1) 2026-10-24 16:19:42 +02:00
mesh-tools fa46e7df7d Merge pull request 'The assigner reads a change's manifests as the indexer does, so a value falling back to a predefined key no longer fails every composition touching it' (#36) from fix/assigner-predefined-fallback into main 2026-10-24 14:09:58 +00:00
jschoubben 18b522184f Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds: a group touching pacman or xterm failed its composition on it (ADR 0348 §7, issue 1751)
mesh/delivery delivered
mesh/check pass 18b522184f73 on fa46e7df7dd1 | unit 128 run, 375 kept | integration 138 run, 426 kept | composition passed on ace
2026-10-24 15:03:14 +02:00
jschoubben 1d4dcdf41a Read a change's manifest as the indexer reads it, so a fallback to a predefined key is not unreadable for the catalogue's predefined.json the assigner never holds (ADR 0348 §7, issue 1751); pin mesh-sdk bfc3883a
mesh/delivery-group group feat/r4g-finish rejected: composed together: the checker's verdict at the members' heads (171a15384fa1): shanks
mesh/delivery superseded: a newer head of the same pull request
mesh/check checking 1d4dcdf41a22 on 4c1d36c9cd7f | novox/mesh-delivery#35 in group feat/r4g-finish
2026-10-24 14:57:36 +02:00
jschoubben f5fd74380c Pin to the group's new mesh-sdk head 19c1d359, whose composer names the reader's pinned SDK instead of a bare (devel) (ADR 0482 clause 5)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk 19c1d35939bc, which is neither on mesh-sdk's t…
2026-10-24 14:12:10 +02:00
jschoubben e78bbfbdee Pin the assigner to the group's mesh-sdk head after main's merge, so it composes with the SDK the group delivers
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-delivery#35: modules/mesh-delivery-assigner/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's t…
2026-10-24 13:28:42 +02:00
jschoubben ce7dcccb1d Merge main into feat/r4g-finish (#33, #34) 2026-10-24 13:26:24 +02:00
jschoubben 42d880f075 Merge pull request 'Count a node's toolchain report only while its build module is assigned there, so a removed node no longer makes builders disagree (issue 1940)' (#34) from fix/1940-stale-toolchain into main
Reviewed-on: #34
2026-10-24 11:10:45 +00:00
jschoubben 3553cbaf08 Count only the toolchain reports of nodes a build module is assigned to, so a node it left does not make the builders disagree (issue 1940)
mesh/delivery delivered
mesh-build-node was unassigned from ace on 2026-10-23 and its ace.typescript key stayed in
mesh-build-node_toolchains at v26.12.0: nothing deletes it, so the indexer, the builder and the
checker took it for a builder and every Node fingerprint said 'builders disagree'. The three now
read mesh-controller_assignments and pass a toolchain report on only while its build module has an
open assignment on its node; builders that run different versions still disagree.
2026-10-24 12:48:41 +02:00
jschoubben d8b19cb332 Merge pull request 'Build a done key again when the store no longer holds its blob (issue 1937)' (#33) from fix/1937-done-key-blobs into main
Reviewed-on: #33
2026-10-24 10:35:21 +00:00
jschoubben 2259d98c5b Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:39:03 +02:00
jschoubben 4702c6bd77 Pin to the finish group's mesh-sdk head dd0d156b, so every member names one SDK (issue 1622) 2026-10-24 04:28:15 +02:00
jschoubben 23f4304d70 Merge remote-tracking branch 'origin/feat/r4g-finish-composer' into HEAD 2026-10-24 04:26:19 +02:00
jschoubben 28ee22ef27 Give the integration test's EVENTS stream the artifacts-gone subject alone, so it does not overlap the build seats' streams (issue 1937)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792840300970624489 (novox/mesh-delivery at 42d880f0 into main, d…
mesh/delivery-group group fix/1937-done-key-blobs closed: a member was closed before the group merged
mesh/check pass 28ee22ef27e2 on d8b19cb332b6 | unit 126 run, 370 kept | integration 136 run, 420 kept | composition passed on shanks
2026-10-24 04:25:19 +02:00
jschoubben f0c5be8c30 Say which artifacts the store check asks about: bundles and archives, whose blobs the collector deletes (issue 1937) 2026-10-24 04:19:17 +02:00
jschoubben 80a0183567 Build a done key again when the artifact store no longer holds its blob, and on the controller's artifacts-gone (issue 1937)
The store's nightly collector deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which
the controller had never taken in, and the builder kept them done: their next delivery would refuse. The builder now asks
the controller's held before it leaves a done trunk key done (once per key per process, at its start and on each module
value), claims a key whose blob is gone building as a new revision and says so in its log and the key's step, and hears
artifacts-gone, said when a take-in found a gone blob. No timer: a start, a value or an event.
2026-10-24 04:17:01 +02:00
jschoubben c256d732f4 Pin the assigner to mesh-sdk's head that reads a composer request from a file (issue 1622) 2026-10-24 04:14:33 +02:00
jschoubben 93427c08e7 Ask the Go build seat for the composer at a group's mesh-sdk head and at its readers' mesh-sdk, and follow each step (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben c58342786f Judge a group with a mesh-sdk member from the composer at its head, and fail a field its readers cannot read (issues 1622, 1623) 2026-10-24 03:57:51 +02:00
jschoubben 840e56eddb Fail the node-engine's check when the controller's trunk is unknown, so its step-nine test never skips (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 90fd07ac26 Merge pull request 'Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)' (#32) from fix/1933-rebuild-indexer into main
Reviewed-on: #32
2026-10-23 23:38:21 +00:00
jschoubben 90a79b1585 Give the indexer a new build: its last build's program never reached the artifact store, so novox cannot apply (issue 1933)
mesh/delivery delivered
mesh/check pass 90a79b15854b on 90fd07ac260a | unit 126 run, 366 kept | integration 136 run, 414 kept | composition passed on novox
2026-10-24 01:35:33 +02:00
jschoubben 45ab994777 Merge pull request 'The checker keeps a failed test's whole message with its cause first, and its status names the cause (issue 1901, with mesh-catalog)' (#30) from fix/1901-rebased into main
Reviewed-on: #30
2026-10-23 22:36:03 +00:00
jschoubben 55f486e040 Merge pull request 'The indexer keeps an unreadable manifest's inputs, so an SDK upgrade judges it again at start (issue 1923)' (#29) from fix/1923-stale-none2 into main
Reviewed-on: #29
2026-10-23 22:35:49 +00:00
jschoubben 7a88eb0a56 Merge pull request 'The checker reads mesh-sdk's list-form level digest, so kept verdicts hold for a check naming a package like token (issue 1930, with mesh-sdk and mesh-catalog)' (#31) from fix/1930-digest-shape into main
Reviewed-on: #31
2026-10-23 22:08:43 +00:00
jschoubben 9261347a8f Read a level's digest from mesh-sdk's list form, so kept verdicts hold for a check whose end names a package like token (issue 1930)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792794985391468127 (novox/mesh-delivery at 45ab9947 into main…
mesh/delivery-group group fix/1930-digest-shape merged: every member is merged
mesh/check checking 9261347a8f91 on 7a88eb0a5609
2026-10-23 23:37:49 +02:00
mesh-tools 263ff54a4e Merge pull request 'Integration tests that R4g never ran now pass: the assigner's fixture, the builder's restart test, four checker tests (issues 1927, 1928, 1929)' (#28) from fix/1927-assigner-fpset into main 2026-10-23 21:24:32 +00:00
jschoubben cd3e55e056 Bring the checker's integration tests to what R4g decided: Go-only repository, echoed beside, readable kept (issue 1929)
mesh/delivery delivered
mesh/check pass cd3e55e05605 on fae4d48e09a5 | unit 0 run 483 kept | int 0 run 540 kept | clone 17s | composition passed on 3 machines
2026-10-23 23:18:11 +02:00
jschoubben b303e32a41 Close the builder test's nodes from the bed's start, so no pull in hand takes the first ask (issue 1928) 2026-10-23 23:03:38 +02:00
jschoubben a31bd7d4a5 Give the assigner test's candidate the SDK's fingerprint set, which the SDK refuses without (issue 1927) 2026-10-23 22:57:05 +02:00
jschoubben 32542ebc14 Keep an unreadable manifest's inputs and judge its verdict again at start, so an upgraded SDK clears it (issue 1923)
mesh/delivery delivered
mesh/check pass 32542ebc14c3 on 7a88eb0a5609 | unit 122 run, 366 kept | integration 132 run, 414 kept | composition passed on novox
2026-10-23 22:17:09 +02:00
jschoubben a697ee86d4 Test 1901/AC3: an older pair of build module and checker still agree (issue 1901)
mesh/delivery delivered
mesh/delivery-group group fix/1901-rebased merged: every member is merged
mesh/check checking a697ee86d4af on 45ab99477701
2026-10-23 22:14:29 +02:00
jschoubben 0373a3b24e mesh-delivery-checker: a failed test's verdict keeps its last lines and the status line names its error (issue 1901) 2026-10-23 22:13:41 +02:00
jschoubben fae4d48e09 Merge pull request 'R4g+R5 (mesh-delivery): builder and checker put asks, the assigner and its composition check, the view's wants and backlog (design 47)' (#27) from feat/r4g into main
Reviewed-on: #27
2026-10-23 17:49:01 +00:00
jschoubben b2c545c40a Place mesh-controller beside the node-engine's tests, which read its manifest (issue 1295, R4g)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792777718506798022 (novox/mesh-delivery at fae4d48e into main…
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking b2c545c40aec on 85aca46348a0 | novox/mesh-delivery#27 in group feat/r4g
2026-10-23 18:36:13 +02:00
jschoubben ad26670996 Prove that a second check of the same heads keeps every package the first passed, the digest being taken inside the check ask as the build module answers it (issue 1917)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-host/internal/apply.TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing: judge_directories_…
2026-10-23 17:18:37 +02:00
jschoubben fad96ae67b Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/config.TestTheUnmovedListHoldsNoModuleTheTrunksDoesNot: realtree_test.go:14: mesh-cata…
2026-10-23 17:06:09 +02:00
jschoubben b83af04f0c Read an ask's stream at take-over by its name, as the broker grants it, and wait for the ask when that read fails rather than fail a key whose build may still finish (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: novox: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group…
2026-10-23 16:59:37 +02:00
jschoubben c7bddbbeab Bound a running build step's pulse by the step's own bound from its start, since nothing rewrites the key while it runs (issue 1908) 2026-10-23 16:58:26 +02:00
jschoubben d3561923aa Say the unit level's gofmt and go vet time in the check's line, and pin the group's mesh-sdk head that carries it (issue 1887 D7-go, review round 2 item 5) 2026-10-23 16:51:55 +02:00
jschoubben d32ec6b7f8 Say what the builder, the indexer and the watchman do since R4g in their comments: no build-agent is asked, no build record moves a fingerprint, no checks-running bucket is mirrored (issue 1887, review round 2 item 6) 2026-10-23 16:47:36 +02:00
jschoubben 52d8b9b753 Say in the checker's status that a check without a verdict waits in a build seat's stream, not behind a plan key or a capacity, as A5b left it (issue 1887, review round 2 item 4) 2026-10-23 16:47:20 +02:00
jschoubben ce51457974 Prove on the real bus that a builder started again while its asks waited puts no second ask (issue 1887 AC3, review round 2 item 2) 2026-10-23 16:46:58 +02:00
jschoubben bac82bb91e Count the asks the builder puts and read their kind in its integration test, so 1887/AC1 is proven on the real bus (issue 1887, review round 2 item 3) 2026-10-23 16:46:57 +02:00
jschoubben 357e28f31d Merge remote-tracking branch 'origin/feat/r4g' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-builder/internal/build/run_test.go
2026-10-23 16:46:05 +02:00
jschoubben 5ddc221833 Bound a build or check ask's step from its start, and drop the checker's whole-check bound, so asks waiting in the streams during a large rebuild are never failed and asked twice (issue 1887, review round 2 item 1; ADR 0525 §3, §6) 2026-10-23 16:45:35 +02:00
jschoubben 37860d46a0 Stop watching the node-engine's have in the checker: a check's asks wait in the build seats' streams, which redeliver a lost node's ask, so no check fails for want of a node; presence stays for an unheard forge and the assigner's presence (issue 1912) 2026-10-23 16:41:34 +02:00
jschoubben 84b2c92ccf Put again, under its own id, an ask a build key names that neither its stream holds nor any node ran, so a builder that stopped between naming and putting it leaves no key waiting for an ask nobody put (issue 1910) 2026-10-23 16:39:56 +02:00
jschoubben c0f24f56b5 Drop the view's use of the build seats: it reads their workers and publishes no ask (issue 1887 A8a) 2026-10-23 16:35:05 +02:00
jschoubben 76bc810bda Test that an ask running when the watchman starts gets its pulse from the steps that stand (issue 1909) 2026-10-23 16:34:50 +02:00
jschoubben e927469460 Pulse a build ask only from when a node takes it, and end the pulse with its step, so a waiting ask is a backlog and not a stuck row (issue 1908) 2026-10-23 16:34:39 +02:00
jschoubben 97b42e3f0d Name in every composition, by machine, those that could run a moved or added module and are not assigned it and those whose grants alone the change alters, as not judged, as ADR 0537 clause 1 asks (issue 1297) 2026-10-23 16:29:57 +02:00
jschoubben 1b019f0353 Record the assigner's full requirements and mesh-sdk feat/r4g's checksums, so the new module builds without a workspace (R4g) 2026-10-23 16:27:05 +02:00
jschoubben 95f52afab2 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g) 2026-10-23 16:26:11 +02:00
jschoubben e4a40a1cb0 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, so every module r4g touches builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:02 +02:00
jschoubben dd8e05142d Merge branch 'feat/r4g-scripts' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 66be3dd0e4 Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 16:20:17 +02:00
jschoubben 527f87f94c Merge branch 'feat/r4g-asks' into feat/r4g
# Conflicts:
#	modules/mesh-delivery-checker/cmd/mesh-delivery-checker/judge.go
#	modules/mesh-delivery-checker/manifest_test.go
#	modules/mesh-delivery-checker/module.json
2026-10-23 16:20:17 +02:00
jschoubben 0463f21026 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 16:19:45 +02:00
jschoubben a4629bc649 Compute every module's fingerprint with no builder input and read no build-agent record for it, so a change of the builder rebuilds nothing (issue 1887 I1, ADR 0530 §1) 2026-10-23 16:14:19 +02:00
jschoubben 50836eb4d6 Format the watchman's integration test (issue 1887 A8b) 2026-10-23 16:14:03 +02:00
jschoubben 79ef768ea4 Show the backlog of build asks per language and kind: waiting, held by a node, and the oldest waiting ask's age (issue 1887 A8a) 2026-10-23 16:14:00 +02:00
jschoubben a8ba5c2fd5 Mirror a waiting build ask as a pulse until a node writes its step, and drop the read of the controller's checks-running (issue 1887 A8b) 2026-10-23 16:12:43 +02:00
jschoubben d0354574c6 Look the assigner's compositions bucket up at each check until it is there, so an assigner delivered after the checker started is read without a restart (issue 1297, review round 1 item 3) 2026-10-23 16:12:18 +02:00
jschoubben 69577f739d Judge hq as any repository, so its pull requests get a verdict and mesh/check from its Go tests and no script judges it (issue 1887 A10, ADR 0527 §2) 2026-10-23 16:11:43 +02:00
jschoubben d4977d9c9d Start the checker whole with the assigner's verdict on the bus and read its line, the level 1297/AC3 asks (issue 1297, review round 1 item 2) 2026-10-23 16:11:31 +02:00
jschoubben 179664898a Merge feat/r4g-asks into the view lane, keeping the checker's assigner compositions 2026-10-23 16:11:15 +02:00
jschoubben 77af82cfa2 Keep the asks state and write the ask about a held fingerprint, so the view's and the teller's read of mesh-delivery-assigner.asks is granted and has something to answer (issue 1297, review round 1 item 1; ADR 0528 clause 6) 2026-10-23 16:11:07 +02:00
jschoubben f37e5d5fa1 Delete merge-check.sh; the checker's levels and the build module's fetch beside run what it ran (issue 1295, S8; ADR 0529) 2026-10-23 16:10:46 +02:00
jschoubben d9d1e57e3a Delete the checker's hold, its own capacity and places, its node moves and its read of the controller's plans-building, so a check waits only in the build seats' streams (issue 1887 A5b, closes 1685; ADR 0525 §4, §7) 2026-10-23 16:09:34 +02:00
jschoubben e4b639c6de Judge each pull request's check as one ask on the build seats' check stream, a re-judge on recheck, so the checker chooses no node and a lost node's check is redelivered, not moved (issue 1887 A5a, ADR 0525 §1, §3, ADR 0536 §4) 2026-10-23 16:05:55 +02:00
jschoubben 271d1ca759 Pin mesh-sdk at feat/r4g's head c3915ffb, the merge of lane B's layouts, so every r4g module builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:01:15 +02:00
jschoubben 2227e7a81f Merge branch 'feat/r4g-view' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 173178df86 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 15:59:55 +02:00
jschoubben 9f27e2b1f8 Build each build module's part of a fingerprint as one ask on its seat's stream, named in the key before it is put, so the builder chooses no node, holds no capacity and never asks twice (issue 1887 A4a, A4b, A4c, ADR 0525 §1-§6) 2026-10-23 15:57:03 +02:00
jschoubben 85aca46348 Merge pull request 'Checks under 5 minutes (checker): a new head's check before trunk-move re-judges, and a build module's change judged by its own code (issues 1897, 1898; ADR 0535, 0536)' (#25) from feat/fastcheck into main
Reviewed-on: #25
2026-10-23 13:50:08 +00:00
jschoubben d5868c178f Take the words of a module the change removes from mesh-sdk in the indexer and the assigner, so the assigner's reading of a removal cannot drift from what the indexer writes (issue 1297, B3b) 2026-10-23 15:44:00 +02:00
jschoubben 7cbb1c4207 Wait for the assigner's composition only while the assigner is online, say at once that it is not judged when it is not, and take it in when the assigner writes it later, so no check waits its bound for a composition nobody makes (issue 1297, B3d) 2026-10-23 15:44:00 +02:00
jschoubben 4c98841d2a Judge a repository the mesh knows no module of with both build modules, so mesh-lab's nested Go modules and its npm test both run (issue 1887 A12) 2026-10-23 15:41:57 +02:00
jschoubben c9ee92a4ab Pin the assigner and the checker to the SDK commit that holds the assigner's ask value, so both build on the layouts the view reads (issue 1297, subtask B1c) 2026-10-23 15:39:49 +02:00
jschoubben 0f7091e4ac Place the change's own repository beside it at its trunk, so a shrink-only list test compares with the trunk's copy at $MESH_CHECK_BESIDE/<name> (issue 1887 A12, ADR 0529 §2) 2026-10-23 15:35:37 +02:00
jschoubben ed98a7519e Write every difference between a want and what the controller sends, and delete one that ended, so the view and acceptance test 17 read them on the bus (issue 1297, subtask B5b; ADR 0528 clause 4) 2026-10-23 15:31:11 +02:00
jschoubben d7ba5aa91e Compare each node's want with the fingerprint of the build the controller sends it, explaining a difference by the rule that holds the want back or saying it unexplained (issue 1297, subtask B5a; ADR 0528 clause 4) 2026-10-23 15:30:31 +02:00
jschoubben 016385751d Write each node's want by the six rules, the canary the first online node by name, and hold a fingerprint whose health failed with its nodes set back, so R5's wants can be compared with what the controller sends (issue 1297, subtasks B4a-B4b; ADR 0429 §6, ADR 0528 clauses 1 and 3, ADR 0435 §3) 2026-10-23 15:29:57 +02:00
jschoubben 5091cab652 Read the assigner's composition into the checker's verdict in place of 'not yet run', failing a check whose composition failed, naming its machine (issue 1297, subtask B3d; ADR 0528 clause 5) 2026-10-23 15:27:59 +02:00
jschoubben d8599855e1 Compose the machines a change touches, from the trunk's and the change's manifests and the controller's assignments on the bus, and write the verdict under the checker's key; what needs the facts snapshot is said as not judged (issue 1297, subtasks B3a-B3c; ADR 0446 clauses 1-3, ADR 0528 clause 5, ADR 0482 clause 3 as R4g delivers it) 2026-10-23 15:25:22 +02:00
jschoubben 57258b6fa4 Take the assigner's ask in the SDK's layout and name the asking in the view's answer, so a stale answer never fills a later asking (issue 1297 B9 and B10) 2026-10-23 15:22:32 +02:00
jschoubben c2b7b8867e Start the assigner whole on an embedded bus in an integration test: it says its three states and refuses to start without a bucket it reads (issue 1297, subtask B2b) 2026-10-23 15:19:49 +02:00
jschoubben a48279b31a Fill an assigner ask's answer from the view's answer under its view key, once and never for an ask that needs proof (issue 1297 B10) 2026-10-23 15:19:37 +02:00
jschoubben 034a6de9a7 Make the module mesh-delivery-assigner: its manifest's reads and states, its own credential and the states it announces, so the controller grants it what R5 reads (issue 1297, subtask B2a; ADR 0528, ADR 0429 §1) 2026-10-23 15:19:06 +02:00
jschoubben e7ed48b81a Give the command an answer verb, so the operator answers a hold or restore ask from a terminal as in the page (issue 1297 B9b) 2026-10-23 15:18:04 +02:00
jschoubben f7dd4ab586 Let the view answer the assigner's asks that need no proof, under the ask's view key, and refuse the merge's Approve on the phone's behalf (issue 1297 B9a) 2026-10-23 15:17:40 +02:00
jschoubben 57b0030258 Show every difference between the assigner's wants and the controller's sends with its rule, so the operator sees what holds a node back (issue 1297 B8) 2026-10-23 15:15:32 +02:00
mesh-tools db5d88f893 Merge pull request 'Drop the indexer's modules keys the catalogue's trunk no longer holds, at each merge and at start (issue 1900)' (#26) from fix/1900-indexer-drops-removed-modules into main 2026-10-23 12:56:31 +00:00
jschoubben 24753bfc2c Grant gitea_contains in the indexer's invokes and test every called forge tool is granted (issue 1900)
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792794967663205383 (novox/mesh-delivery at 55f486e0 into main, d…
mesh/check pass 24753bfc2c02 on 75a4821eff9e | unit 431 run, 0 kept | integration 476 run, 0 kept | over bound: integration (slowest integration mesh-…
2026-10-23 14:48:41 +02:00
jschoubben 1dc9d80cfb Keep a key written ahead of the source head and a key with no path when reconciling (issue 1900) 2026-10-23 14:44:14 +02:00
jschoubben 26bfdbb637 Drop the indexer's modules keys the trunk no longer holds, at each merge and at start (issue 1900) 2026-10-23 14:31:55 +02:00
jschoubben 4ffe5f4d52 Start a never-judged head's check before the re-judges a trunk move asks, oldest first (issue 1897 S1)
mesh/delivery delivered
mesh/delivery-group group feat/fastcheck merged: every member is merged
mesh/check checking 4ffe5f4d5243 on 85aca46348a0 | novox/mesh-delivery#25 in group feat/fastcheck
A new pull request waited 25 to 45 minutes behind every open pull request's re-judge after a
merge. The semaphore becomes a ranked admission (ADR 0536 §4); checker_status names the
waiting checks by kind.
2026-10-23 14:09:42 +02:00
jschoubben 2bdb8d0e00 Ask the Go build module to run the candidate when a merge changes it, and say in the line which program judged (issue 1898 K3)
A change to mesh-build-go was judged by the installed module it replaces, so such pull
requests were merged by hand. ADR 0535 §1, §2, §4, §5.
2026-10-23 14:06:57 +02:00
jschoubben 75a4821eff Merge pull request 'Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)' (#24) from fix/1890-checker-stale-steps into main
Reviewed-on: #24
2026-10-23 11:01:10 +00:00
jschoubben e762028760 Make checker_status count only steps that can still run, so a key a build module left running is not read as a full node (issue 1890)
mesh/delivery delivered
mesh/check pass e76202876032 on b714eab995a5 | unit 422 run, 0 kept | integration 467 run, 0 kept | over bound: integration (slowest integration mesh-…
The checker's own places are released when a step ends or its wait passes the step bound; checker_status read every
'running' key in the build modules' steps buckets as a step in flight, so keys left by stopped, restarted or unassigned
build modules looked like full nodes. It now lists them under left_running with why (past the bound, module not on the
node, module not running), and leaves hq-only groups out of the unjudged candidates.
2026-10-23 09:03:45 +02:00
jschoubben 36bf3b7143 Merge pull request 'The builder says it writes references only once its build modules proved it (issue 1886, ADR 0522)' (#23) from fix/1886-builder-says-references into main
Reviewed-on: #23
2026-10-23 00:26:18 +00:00
jschoubben 9fad90ee89 Say that the builder writes references only once each build module answered one (issue 1886, ADR 0522 review)
mesh/merge-gate head 9fad90ee890a merged into trunk b714eab995a5 — pass: builds mesh-delivery-builder → novox; no bus step; the change touches no module of…
mesh/repo-check head 9fad90ee890a merged into trunk b714eab995a5 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/1886-builder-says-references merged: every member is merged
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The references come from the build modules' own pushes, so a new builder beside
an old build module writes none; saying it at start would send the controller
to keys that fail not stored.
2026-10-23 00:52:50 +02:00
jschoubben 4b8b1c047c Say in the builder's own bucket of instances that it writes references (issue 1886)
The controller delivers from the builder's keys only while a running builder
says its done keys name references; a builder from before R4e says nothing,
so R4e's own delivery is asked of build-agent instead of waiting for ever.
The say is put once started and taken away when stopped.
2026-10-23 00:48:28 +02:00
mesh-tools b714eab995 Merge pull request 'R4e (mesh-delivery): the builder pushes and keys artifacts by reference, the checker's verdict and try, the view, and the checker's owed state read by the forge' (#22) from feat/r4e into main 2026-10-22 22:13:05 +00:00
jschoubben efca168c55 Stop the forge bridge performing mesh/check: the forge module writes it (R2-1)
mesh/repo-check head efca168c5522 merged into trunk 0e372c87f1df beside mesh-catalog#457@896bd883 +1 more — pass: its merge-check.sh passed
mesh/merge-gate head efca168c5522 merged into trunk 0e372c87f1df — pass: group feat/r4e composed: every machine composes with the change as it did without …
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792707174627079192 (novox/mesh-delivery at b714eab9 into main, d…
mesh/delivery-group group feat/r4e closed: a member was closed before the group merged
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:989 | efca168c5522 on b714eab995a5 | unit 2092 run, 0 kept, 3 skipped | integration…
R4e round 2 finding R2-1, ADR 0521: nothing in the delivery plan orders the
forge module before the bridge, and the forge module's durable consumer keeps
its deliver policy when its filters change, so a check.status emitted before
it consumed the event was lost. The forge module now watches the checker's
owed state and writes mesh/check itself, once per revision.

The bridge emits no check.status, declares no emits, and neither reads the
checker's owed nor hears its stream; OwedAct still refuses mesh/check. The
integration test shows the bridge calls no tool and publishes nothing of its
own when the checker owes mesh/check.
2026-10-22 21:40:09 +02:00
jschoubben 19d06163de Announce the checker's states, owed readable, so the forge module may read it (R2-1)
R4e round 2 finding R2-1, ADR 0521: the forge module writes mesh/check from
the checker's owed state, read by a watch, because no event order between the
forge bridge and the forge module is kept by the delivery plan. A reader is
granted from the owner's announcement (ADR 0495 §2-§3), and the checker
announced nothing. It now says every state it keeps, with owed, previews and
verdicts readable: the forge module and the bridge read owed, the indexer
previews, the gate verdicts.
2026-10-22 21:38:53 +02:00
jschoubben 0997b071a3 Pin every R4e member module to mesh-sdk's final R4e head 104533e8, which holds review fixes S1 and S2, so the group pins one SDK 2026-10-22 21:29:20 +02:00
jschoubben bb4a4a82dc Emit mesh/check as the bridge's own check.status, never through a tool (K1)
Review finding K1 of R4e, issue 1301 S4 (ADR 0503 §3, ADR 0513 §4: the forge
requires mesh/check alone). gitea_check_status rested on "only the bridge
invokes it", but node-tools invokes every tool and a tool is told no caller,
so any session could set mesh/check and merge past the checker.

The bridge now emits check.status on mesh.mod.mesh-delivery-forge-bridge.event.
check.status, which the bus grants to its own account alone (emits), with the
owed status, its key and its revision; the event id check.<key>.<rev> is the
bus's dedup id. A status the bus did not take is not kept as done. OwedAct
refuses mesh/check, and gitea.gitea_check_status leaves the invokes. The
integration test shows the tool runner's account refused on that subject and
gitea's durable consumer handed statuses emitted before it fetched.
2026-10-22 21:23:27 +02:00
jschoubben 08bc672488 Pin every R4e member module to mesh-sdk's final R4e head 875c680d, which holds main's #68 and #69, so the group pins one SDK 2026-10-22 20:45:16 +02:00
jschoubben 90ac287e59 Drop the checker tools binary committed by mistake with the try verbs (7abe230): no module commits a build output 2026-10-22 20:29:53 +02:00
jschoubben 91e68ac13b Merge feat/r4e-builder into feat/r4e (R4e integration)
The checker's go.mod and go.sum conflicted on the mesh-sdk pin (checker lane 9cb4d59, builder lane e76a5a3): every
module touched is pinned to the sdk lane's newest head d1af87a until the final SDK head.
2026-10-22 20:29:53 +02:00
jschoubben 173becf35e Merge feat/r4e-checker into feat/r4e (R4e integration) 2026-10-22 20:29:10 +02:00
jschoubben 612658e788 Ask the forge for the objects of a merge's named paths in the indexer, the one reader of a manifest's paths, and narrow the module's tree from the answer (issue 1758, I6 reworked; ADR 0499 §2 §3, ADR 0495 §1, L8)
The manifest invokes gitea.gitea_commit_trees. A forge that cannot say gives the module none with its words.
2026-10-22 18:24:38 +02:00
jschoubben d801a6460e Read no manifest's paths in the forge bridge: the indexer, the one reader of paths (ADR 0495 §1, the lead's decision L8), asks the forge for their objects itself (issue 1758, I6 reworked)
Undoes 8ab8c5d's bridge change; the bridge keeps the mesh-sdk pin.
2026-10-22 18:23:29 +02:00
jschoubben 12cf0cc1f5 Format the builder's integration test 2026-10-22 18:21:04 +02:00
jschoubben 7cc97cd6b6 Name a candidate's named paths in its fetch, and prove on a real bus that claude-code's manifest is built (issues 1758 and 1751) 2026-10-22 18:20:59 +02:00
jschoubben ece328d0cc Give a candidate's module its named paths' objects from the preview, so the builder's fetch for it names them (issue 1758, ADR 0499 §4; mesh-sdk CandidateModule.Paths) 2026-10-22 18:20:37 +02:00
jschoubben b39090aab9 Read manifests with mesh-sdk's ParseManifestForInputs in the indexer and the builder, so claude-code, dunst and xterm are built too (issue 1751)
The indexer's own copy goes. Pins mesh-sdk feat/r4e-sdk e76a5a3 in every module of the group.
2026-10-22 18:20:37 +02:00
jschoubben 4bb04a47f9 Build each fingerprint on one node and drop the second-node retake, as ADR 0478 §4 has it from R4e (issue 1587, B9)
A key done on one node before R4e is not built again for a second; a key failed for differing digests stays as it is.
2026-10-22 18:18:59 +02:00
jschoubben 9a3f38629f Prove on a real bus that a merge of mesh-sdk changing only what its lib does not name leaves its tree input and new form unchanged (issue 1758, I9, ADR 0499 check 6 in miniature) 2026-10-22 18:16:54 +02:00
jschoubben 9bd84941ee Show a fingerprint of the new form against the builder's key of it, since the indexer no longer writes the old form and no build record names the new (issue 1851, I5)
The view watches the builder's builds; a value written before R4e keeps today's comparison. Pins mesh-sdk feat/r4e-sdk
9cb4d59.
2026-10-22 18:16:18 +02:00
jschoubben cc63123b78 Carry each named path's object from a fetch's module trees into the merge preview, so the indexer narrows a candidate's tree as the merge's (issue 1758, I8, ADR 0499 §2)
Pins mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:15:09 +02:00
jschoubben 23daa2c03c Take a module's tree input from the paths its artifacts name, by mesh-sdk's NamedTree, and keep their objects and each artifact's paths, so a module outside them moves for no change there (issue 1758, I7 and the indexer's I8, ADR 0499 §2 §3)
A merged message or a preview without the objects gives the module none, naming the paths. Every other module keeps its
directory's tree. Module.ArtifactPaths is filled for the checker (the lead's interface).
2026-10-22 18:14:30 +02:00
jschoubben 8ab8c5d04a Carry each named path's git object in the merged and pull messages of a module whose every artifact names its paths, read at its manifest's commit (issue 1758, I6, ADR 0499 §3)
Asks gitea_commit_trees with paths (the interface agreed with the lead); a forge that cannot say is a retry. Pins
mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:13:25 +02:00
jschoubben e81ef8bb63 Read a manifest whose config value falls back to a predefined key, so claude-code, dunst and xterm get a new-form fingerprint (issue 1751, ADR 0348 §7)
The indexer holds no published list of predefined keys; each key a value names is taken as the value declares it.
No fingerprint input comes from the list: the gate and the controller still judge the value.
2026-10-22 18:11:21 +02:00
jschoubben af4eaefb14 Read the controller's build records only for the builder's own input and to fill a key's missing directory once, so a key written before paths gets one with no merge (issue 1851 I3, issue 1614)
Both reads go in R4g with build-agent. A path a key names is never overwritten, and no fingerprint changes.
2026-10-22 18:09:56 +02:00
jschoubben 9f7e082688 List a repository's modules from the merged message's manifests and the indexer's own bucket, not the controller's build records, so a new module gets a value and a key with no build asked (issue 1851, I1 I2 I4, ADR 0501 §6, ADR 0453 §6)
I1: a merge writes each module its manifests name, at its directory. I2: a pull request's changed directory, a
preview's touched module and a source revision's modules come from the modules bucket (a module a pull request adds is
found in the preview by its manifest). I4: the value of today's form (fp, none) is no longer written, since it was
computed from the build records; mesh-sdk keeps it readable until R4g. One commit: the three share applyMerged and its
tests. Pins mesh-sdk feat/r4e-sdk 9cb4d59.
2026-10-22 18:08:48 +02:00
jschoubben 7d740be255 Fail a check whose forge read reached nobody at the forge, and judge it again when the nodes change (issue 1624, C7)
mesh-catalog #293 failed with 'go.mod cannot be read to check its mesh-sdk pin: nats: no responders', put at the
build node, and stayed failed: the forge's tool runner was not answering for a moment, which is not the change's
failure. The forge's reads now mark such a call unheard, as the build steps' calls are (issue 1820).
2026-10-22 18:07:22 +02:00
jschoubben 66ae4dc791 Format the judge (gofmt) 2026-10-22 18:06:11 +02:00
jschoubben 03eaf332ad Say the composition's time and the machines it composed in a check's line, once the assigner reports them (issue 1300, C10)
The line said only that the composition passed with an SDK commit, and its time sat among the parts that are dropped
first, so a forge status never said how long the composition took or which machines it covered.
2026-10-22 18:06:07 +02:00
jschoubben cbdcf6da0e Give a group's TypeScript members the mesh-sdk member's package when its merge changes the package's paths, and pin the SDK's R4e head (ADR 0500 §4-§5, issue 1309's checker half)
A TypeScript member of a group that changes mesh-sdk's package resolved the old version from the registry. Its Node
fetch now names the mesh-sdk member's head, which the Node build module packs and serves, publishing nothing. The paths
come from the indexer's modules value (lead decision L8), so a change of Go files or of the manifest alone resolves
the registry as before. A test holds that no step of a check carries publish.
2026-10-22 18:05:12 +02:00
jschoubben 8b2e8895c2 Prove on a real bus that every done key names its references, a refused push is built again, and a pinned upstream is copied (issue 1587, B8, ADR 0498 checks) 2026-10-22 18:03:39 +02:00
jschoubben 7abe230701 Judge a pushed head as a try on the checker, through checker_try and checker_try_read (ADR 0501 §5)
mesh-controller's try asked build-agent, which leaves the checking at R4e. A try is now a key in the checker's state
tries: the checker judges the head merged onto its trunk as it judges a pull request's head, through the same
judge, places and holds, writes its end in the key and owes the forge nothing (no status, no preview, no verdict).
One try per head, at most four at once; hq and a repository whose trunk the gate does not name are refused with
"no level of <repo> is checked by the checker".
2026-10-22 18:03:28 +02:00
jschoubben 7651e47cc6 Write a module with no artifact a done key with no artifact, so the controller has a done key to register it from (issue 1613, ADR 0478 §1, ADR 0501 §1)
Moves the mesh-sdk pin to feat/r4e-sdk 9cb4d59, which accepts such a key and names delivery.WaitsFor; the builder's
package wait now uses that shared prefix (B7).
2026-10-22 18:03:02 +02:00
jschoubben 957afa6926 Ask the fetch with mesh-sdk's FetchRequest, naming a module's directory and the paths its kept inputs hold, so a module naming its paths is built from those alone (issue 1758, ADR 0499 §4; the lead's fetch interface) 2026-10-22 18:02:08 +02:00
jschoubben 2e567b9b30 Let a key whose lock names an unpublished package version wait queued, naming it, and build it once the builder's own done key publishes it (issue 1587, B7, ADR 0500 §6)
The prefix 'waits for: ' is kept in the builder until mesh-sdk's delivery.WaitsFor lands on feat/r4e-sdk.
2026-10-22 18:01:33 +02:00
jschoubben 851811a318 Build a candidate's done package key again once the trunk names it, so the package is published from the trunk (issue 1587, B6, ADR 0498 §6) 2026-10-22 18:00:19 +02:00
jschoubben 2399a4f4cc Build a TypeScript package with the Node build module, published only for a trunk fingerprint; a version taken fails the key until the fingerprint changes (issue 1587, B5, ADR 0498 §5, ADR 0500 §4) 2026-10-22 17:59:43 +02:00
jschoubben 2855ae7ebf Set the owed mesh/check through gitea_check_status, the one tool that writes it (issue 1301's delivery part, option A)
The forge's gitea_commit_status refuses the context mesh/check from R4e, so that only the checker's account, through
this bridge, can make a pull request pass the one required status.
2026-10-22 17:59:30 +02:00
jschoubben 081f32d6c9 Copy a pinned upstream image with mesh-build-image's copy step, with no fetch, so its key names the registry's reference (issue 1587, B4, ADR 0498 §4) 2026-10-22 17:59:19 +02:00
jschoubben d9ace64681 Mark mesh/check pending on every open pull request into a moved trunk, and carry the gate's heads hash in each verdict (issue 1302's checker part, lead decision L4; issue 1300)
The gate decides on mesh/check alone and matches an Approve's members by the heads hash over lowercase repositories
and heads. A success judged on an old trunk stood on the forge while the next check waited for a place or a delivery,
and a candidate whose repository the forge named in capitals carried a hash the gate never finds. A check that ends
after its trunk moved now owes pending, not its verdict.
2026-10-22 17:58:49 +02:00
jschoubben c26f909152 Send a pinned upstream image to mesh-build-image's copy and fail an unpinned one 'not pinned:' until its fingerprint changes; refusals use mesh-sdk's RefusedKind (issue 1587, B3, ADR 0498 §4) 2026-10-22 17:58:33 +02:00
jschoubben 9871ed2d09 Build a key done before R4e again once, so every bundle and archive the controller delivers names its stored reference (issue 1587, B2, ADR 0498 §8) 2026-10-22 17:57:35 +02:00
jschoubben f26ff8f8bb Write a key done only when every bundle and archive names where it is stored, a candidate's too, so a merge builds nothing its check built (issue 1587, B1, ADR 0498 §1-§3)
Pins mesh-sdk feat/r4e-sdk a90f523 (R4e's group member). A trunk key's builds are asked to publish; a build answering
no reference fails the key 'not stored:', and a step's own 'not stored:', 'not pinned:', 'version taken:' or
'refused kind:' opens the key's error.
2026-10-22 17:57:01 +02:00
jschoubben f7d8622deb Keep a core repository's passing package under a digest that covers its siblings' commits, so core checks keep verdicts (issue 1287, C3a/C3b, lead decision L5)
The checker took and kept nothing for a workspace given siblings, and every core repository has siblings, so the
kept store stayed empty. The digest step now folds the siblings' commits in and echoes them; a verdict is kept or
taken only when the echo equals the siblings the fetch was given, so a digest that misses one never poisons another
check (ADR 0444, issue 482).
2026-10-22 17:56:59 +02:00
jschoubben d83156fb21 Add checker_previews and checker_status read tools so the live acceptance of issues 1680 and 1685 can be read (C1, C2) 2026-10-22 17:25:11 +02:00
jschoubben 16ca51dc1f Bound builder_builds by key prefix, limit and read budget, list unreadable keys, refuse fp2 with commit, add candidate reads (issue 1827 V1-V3, 1818, 1681) 2026-10-22 17:23:46 +02:00
mesh-tools 0e372c87f1 Merge pull request 'Read a go.mod the head does not hold at the trunk, so a branch older than a new Go module passes the pin check (issue 1845)' (#21) from fix/1845-pin-read-at-trunk into main 2026-10-22 13:44:08 +00:00
jschoubben 4a0c771059 Match the forge's 404 by its status, not by digits a path's sha may hold (issue 1845)
mesh/repo-check head 4a0c77105943 merged into trunk 2b635e2138ae — pass: its merge-check.sh passed
mesh/merge-gate head 4a0c77105943 merged into trunk 2b635e2138ae — pass: builds mesh-delivery-checker → novox; no bus step; every machine composes with the…
mesh/delivery delivered
mesh/check pass 4a0c77105943 on 2b635e2138ae | unit 371 run 0 kept | int 410 run 0 kept | clone 13s unit 206ms int 14s+45s | composition not yet run
2026-10-22 15:09:25 +02:00
jschoubben 14c78e485f Read a Go module's go.mod at the trunk when the head does not hold it, so a branch cut before the module was added passes the pin check (issue 1845) 2026-10-22 15:00:58 +02:00
mesh-tools 2b635e2138 Merge pull request 'Send image artifacts to mesh-build-image, one node, publishing only from the trunk (issue 1705)' (#20) from feat/r4c9-image-build into main 2026-10-22 08:57:21 +00:00
jschoubben db62e82e26 Write a failed publish of a merged image to the key; read mesh-build-image's captured steps (review r2)
mesh/repo-check head db62e82e26d7 merged into trunk 47ab8cd3cf45 beside mesh-catalog#435@7c5d11e5 — pass: its merge-check.sh passed
mesh/merge-gate head db62e82e26d7 merged into trunk 47ab8cd3cf45 — pass: group feat/r4c9-image-build composed: every machine composes with the change as it…
mesh/delivery delivered
mesh/delivery-group group feat/r4c9-image-build merged: every member is merged
mesh/check pass db62e82e26d7 on 2b635e2138ae | unit 366 run 0 kept | int 405 run 0 kept | clone 9.2s unit 191ms int 6.7s+44s | composition not yet run
2026-10-22 10:37:21 +02:00
mesh-tools 47ab8cd3cf Merge pull request 'The checker moves a step that no build node answers to another node (issue 1820)' (#19) from fix/1820-checker-heard-nodes into main 2026-10-22 08:36:34 +00:00
jschoubben 12f61998c5 Publish a merged candidate's image once; a retaken key builds only what is missing (review r1) 2026-10-22 10:33:52 +02:00
mesh-tools 5b09d1ccb5 Merge pull request 'Read tools for the builder's builds in the view (issue 1723)' (#18) from fix/1723-builder-builds-tool into main 2026-10-22 08:30:42 +00:00
jschoubben bbc690539b Build an image on one node through mesh-build-image; only a trunk key publishes (issue 1705) 2026-10-22 10:19:26 +02:00
jschoubben bbb42d704c The checker moves a step that no build node answers to another node and fails naming every node tried (issue 1820)
mesh/merge-gate head bbb42d704ce0 merged into trunk 5b09d1ccb52d — pass: builds mesh-delivery-checker → novox; no bus step; every machine composes with the…
mesh/repo-check head bbb42d704ce0 merged into trunk 5b09d1ccb52d — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check pass bbb42d704ce0 on 5b09d1ccb52d | unit 354 run 0 kept | int 392 run 0 kept | clone 8.7s unit 199ms int 7.1s+44s | composition not yet run
2026-10-22 10:04:29 +02:00
jschoubben f2379dfeea Add builder_build and builder_builds read tools to the view, so the builder's keys can be checked live (issue 1723)
mesh/merge-gate head f2379dfeea2d merged into trunk 113f7aa93385 — pass: builds mesh-delivery-view → shanks; no bus step; every machine composes with the c…
mesh/repo-check head f2379dfeea2d merged into trunk 113f7aa93385 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: g14: fetch of novox/mesh-delivery#18: build_go_fetch on g14: nats: no responders available for request | f2379dfeea2d on 113f7aa933…
2026-10-22 09:54:46 +02:00
mesh-tools 113f7aa933 Merge pull request 'The checker runs beside today's check: the checker, the indexer's candidate fingerprints, the bridge's status' (#17) from feat/the-checker into main 2026-10-21 21:46:33 +00:00
jschoubben b519c8dde2 Keep nothing for a workspace given siblings, place the siblings where the tests state them, decide the hold from the listed plan keys (review D1-D3)
mesh/repo-check head b519c8dde2da merged into trunk a5aa69a4e815 beside mesh-catalog#423@12ff1cfb +1 more — pass: its merge-check.sh passed
mesh/merge-gate head b519c8dde2da merged into trunk a5aa69a4e815 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker merged: every member is merged
2026-10-21 21:52:41 +02:00
jschoubben 7214bf4368 Fail a candidate's build when its fetch answers files in conflict or no merge, so the trunk's build is not stored under the candidate's fingerprint (review D4) 2026-10-21 21:50:30 +02:00
jschoubben 284f21c4e3 Key the builder's candidate test as the indexer keys a group 2026-10-21 21:35:53 +02:00
jschoubben a20af30d54 A group's Go member naming a mesh-sdk commit that is neither on the trunk nor the group's head fails the check naming the member and the commit; a pin only a head carries names issue 1148 (issue 1624) 2026-10-21 21:34:35 +02:00
jschoubben 4cdba63b9b Read tools for the checker's verdicts, a pull request's report and the kept verdicts (issue 1686) 2026-10-21 21:32:08 +02:00
jschoubben ee994f57b9 Do not pin the integration test to one form of the line 2026-10-21 21:30:29 +02:00
jschoubben fecabfca8b Fit the status line in the forge's 140 characters by giving the times and counts in their compact form before dropping them (issue 1682) 2026-10-21 21:30:24 +02:00
jschoubben 4b063252e7 The checker's line says whether each candidate module's build is built, failed or waiting, read from the builder's keys (issue 1684) 2026-10-21 21:29:13 +02:00
jschoubben 66908d9bbc The builder builds the fingerprints a candidate carries, once each, from the head merged onto the trunk (issue 1684) 2026-10-21 21:28:38 +02:00
jschoubben 7433b0d047 Start no check while a delivery builds: the checker reads the controller's bucket of the plans in their build phase, absent as empty (issue 1685) 2026-10-21 21:26:46 +02:00
jschoubben a2a5167d72 Write a preview of the conflicts for a head that does not merge, and give the candidate's modules the reason; pin mesh-sdk to the R4c-5a head (issue 1445) 2026-10-21 21:25:45 +02:00
jschoubben e554094b74 The builder uses mesh-sdk's buildrun package for the caller, the steps, the places, the presence, toolchain and have sets and the choice of nodes, so the checker shares them (issue 1671) 2026-10-21 21:25:00 +02:00
jschoubben 64599db3fa Capture the fetch-with-base fixtures from mesh-catalog's code, and prove the checker whole against a real bus with fake build modules (issues 1679, 1287, 1288, 1289, 1678) 2026-10-21 21:23:22 +02:00
jschoubben 96ba6bd3dc Say the verdict in a line of at most 140 characters and owe the status mesh/check in the checker's bucket, which the forge bridge performs once per revision; the checker's stream checks wakes the readers of its buckets (issues 1682, 1683) 2026-10-21 21:20:42 +02:00
jschoubben 4242fb5b0b The indexer gives a candidate's touched module the fingerprint its merge preview gives it, and wakes on the checker's preview.written without a restart; pin mesh-sdk to the R4c-5a head (issues 1445, 1681) 2026-10-21 21:17:09 +02:00
jschoubben e175343837 Write the merge preview of each head the checker fetched, for the indexer's fingerprints (issue 1680) 2026-10-21 21:12:48 +02:00
jschoubben cfbd5b6d2d Judge a head merged onto its trunk through the build modules: run only packages without a kept verdict, the change's own tests first, stop at the first failure (issues 1679, 1288, 1289) 2026-10-21 21:11:43 +02:00
jschoubben 27072a01b5 Add the checker module: it watches the candidates, claims a verdict once, owes the status mesh/check and ends a check past its bound (issue 1678) 2026-10-21 21:07:14 +02:00
jschoubben a00df90c47 Keep a passing package's verdict once under its digest, so a second check can take it (issue 1287) 2026-10-21 21:02:54 +02:00
mesh-tools a5aa69a4e8 Merge pull request 'The builder takes build-agent's newest trunk build as its builder input (issue 1727)' (#16) from fix/1727-builder-input into main 2026-10-21 15:44:42 +00:00
mesh-tools 318064ce20 Merge pull request 'Ask again a merge ask the router refused for want of room, so the operator is asked (issue 1610)' (#15) from fix/1610-ask-reaches-channel into main 2026-10-21 15:39:20 +00:00
jschoubben 61fea8314f Compute from a build of build-agent only once a toolchain is read, so a restart drops no fingerprint; fixtures as live (issue 1727)
mesh/merge-gate head 61fea8314fe8 merged into trunk 318064ce206d — pass: builds mesh-delivery-indexer → novox; no bus step; every machine composes with the…
mesh/repo-check head 61fea8314fe8 merged into trunk 318064ce206d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 16:45:51 +02:00
jschoubben 037bf1ef36 Ask again at start a merge ask whose refusal the teller did not hear, since the router keeps no record of a refused ask (issue 1610, review round 1)
mesh/merge-gate head 037bf1ef361b merged into trunk b04ee764739a — pass: builds mesh-delivery-teller → novox; no bus step; every machine composes with the …
mesh/repo-check head 037bf1ef361b merged into trunk b04ee764739a — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 16:45:31 +02:00
jschoubben 38116fe00a Ask again a merge ask the router refused for want of room, once one of the teller's asks ends, so the operator is asked (issue 1610)
The router holds at most three open asks of one asker and refuses a fourth without keeping it; the teller only logged the refusal, so a merge ask in a burst never reached the operator while the gate read it as asked (issues 1610, 1620). Bounded at six refusals, then said as a condition.
2026-10-21 15:54:08 +02:00
jschoubben 5f9ed2b886 Take the builder's Builder input from build-agent's newest trunk build record, not its absent fingerprint, so modules get a new-form fingerprint (issue 1727) 2026-10-21 15:49:33 +02:00
mesh-tools b04ee76473 Merge pull request 'Make the builder's build capacity a setting per node, hold a failed second-node build until the nodes change, and close keys left building for a vanished fingerprint (issues 1661, 1664, 1660)' (#14) from fix/1661-builder-follow-ups into main 2026-10-21 12:26:03 +00:00
mesh-tools 2000c841bf Merge pull request 'Read the indexer's modules keys as stored from the view's tools, so the new-form fingerprint, path and toolchains can be checked live (issue 1562)' (#13) from fix/1562-indexer-reads into main 2026-10-21 12:04:07 +00:00
jschoubben 46702d1af3 Keep the builder off a node whose capacity setting is 0, and start the setting at shanks=3,ace=2,g14=0,novox=0 (issue 1661)
mesh/merge-gate head 46702d1af311 merged into trunk 2000c841bf70 — pass: builds mesh-delivery-builder, sent nowhere; no bus step; every machine composes wi…
mesh/repo-check head 46702d1af311 merged into trunk 2000c841bf70 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:46:14 +02:00
jschoubben 1239199b35 Name the view's tools binary as what its bundle loads, as the controller's module check requires of a bundle compiled to a binary (issue 1562)
mesh/merge-gate head 1239199b35dd merged into trunk d68d7ef3d9af — pass: builds mesh-delivery-view → shanks; no bus step; every machine composes with the c…
mesh/repo-check head 1239199b35dd merged into trunk d68d7ef3d9af — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:43:26 +02:00
jschoubben dcf62b0413 Make the builder's capacity a module setting, hold back a failed second-node build until the nodes change, and close at start a key left building for a fingerprint the module no longer has (issues 1661, 1664, 1660) 2026-10-21 13:34:39 +02:00
mesh-tools d68d7ef3d9 Merge pull request 'Add the builder that builds each fingerprint's bundles and archives on two build nodes beside build-agent, and have the indexer write a module's path and toolchains (R4c-4: issues 1606, 1607, 1615)' (#12) from feat/r4c4-builder into main 2026-10-21 11:31:05 +00:00
jschoubben 6c2b79acf1 Serve the indexer's read tools from the view, not the indexer (ADR 0437 4): indexer_module and indexer_modules read mesh-delivery-indexer.modules through the view's reads. Side effect: carrying the view's tools gives the tool runner read access to every bucket the view reads. (issue 1562)
mesh/merge-gate head 6c2b79acf106 merged into trunk d68d7ef3d9af — fail: builds mesh-delivery-view → shanks; no bus step; a manifest the change touches fai…
mesh/repo-check head 6c2b79acf106 merged into trunk d68d7ef3d9af — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 13:24:10 +02:00
jschoubben bfc64ac65e Build an archive with no toolchain, mark digests that differ and a one-node key so each is handled, reserve a place when a node is chosen, choose only nodes whose build module is healthy, and take over any key found building (issue 1607, review round 4)
mesh/merge-gate head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: builds mesh-delivery-indexer, new: modules/mesh-delivery-builder → novox; no bus s…
mesh/repo-check head bfc64ac65ea1 merged into trunk 26de6bc62558 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 13:15:28 +02:00
jschoubben e6ce4be4b4 Read the indexer's modules keys with two read-only tools, so the new-form fingerprint and the reason for none can be checked live (issue 1562) 2026-10-21 13:06:05 +02:00
jschoubben ebac167cd6 Move the indexer and builder to mesh-sdk main with the toolchains and root path (issue 1615) 2026-10-21 12:45:10 +02:00
jschoubben 27e4c3891f Run at most a node's build capacity of build steps on it at once, and hold a step back for a place with no timer (issue 1607, round 3) 2026-10-21 12:41:42 +02:00
jschoubben f555a05cfb Build a key only with the toolchain its fingerprint names, accept '.' as a root module's path, and choose only online nodes (issues 1615, 1607 review) 2026-10-21 11:36:32 +02:00
jschoubben 3bd2a065f9 The indexer writes '.' for a root module's path and the toolchains each fingerprint was computed with (issue 1615) 2026-10-21 11:33:49 +02:00
jschoubben 0b0cfb6e37 Test the builder whole over the real bus with fake build modules, and look again at a value that arrives while its key is built (issue 1607, AC6, AC7) 2026-10-21 11:25:13 +02:00
jschoubben 46c9f33d64 Build each new-form fingerprint through the build modules on two nodes and write its key done or failed (issue 1607, AC7, AC8) 2026-10-21 11:23:39 +02:00
jschoubben 71bb5a5fb3 Watch the build modules' toolchains and steps and keep the toolchains by node, with no timer (issue 1607, AC6) 2026-10-21 11:21:46 +02:00
jschoubben 3298ad58d3 Run a build module's long step: call it on its node, read the result at its steps key, bound the wait (issue 1607, AC5) 2026-10-21 11:21:14 +02:00
jschoubben 74d11f95a2 Choose build nodes by the agreed toolchain and fewest running steps, and agree the two nodes' digests (issue 1607, AC4) 2026-10-21 11:20:31 +02:00
jschoubben 1e6d00b4c7 Claim a build key by create and compare-and-set, never twice, retry a failed key except a refused kind (issue 1607, AC3) 2026-10-21 11:20:09 +02:00
jschoubben 87e550681f Plan a manifest's artifacts for the build modules and refuse image, upstream and package by kind (issue 1607, AC2) 2026-10-21 11:19:38 +02:00
jschoubben c3804b012f Add the builder part: its manifest, its credential and its bus connection (issue 1607, AC1) 2026-10-21 11:18:30 +02:00
jschoubben 07a28b03c4 The indexer writes each module's directory into modules.<m> and keeps it (issue 1606, ADR 0478 clause 8) 2026-10-21 11:16:16 +02:00
mesh-tools 26de6bc625 Merge pull request 'The indexer reads the build modules' toolchains, and an artifact with no lock carried has no lock entry (design 47 R4c-3)' (#11) from feat/r4c3-build-modules into main 2026-10-20 21:43:20 +00:00
mesh-tools a5da6079de Write each module once per merge and watch a build module's toolchains bucket on its toolchain.reported event, with no timer (issue 1444, review round 1)
mesh/repo-check head a5da6079de01 merged into trunk 340bf643883d beside mesh-catalog#389@bcee61ed — pass: its merge-check.sh passed
mesh/merge-gate head a5da6079de01 merged into trunk 340bf643883d — pass: group feat/r4c3-build-modules composed: every machine composes with the change as …
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792582266656254343 (novox/mesh-delivery at d68d7ef3 into main…
mesh/delivery-group group feat/r4c3-build-modules merged: every member is merged
2026-10-20 21:02:44 +00:00
mesh-tools 127ac64cce Leave an artifact without its lock file out of Locks (issue 1459) and read the build modules' toolchains (issue 1444), part 2 of 2 2026-10-20 20:48:30 +00:00
mesh-tools 74fef7dd5a Recompute the new-form fingerprints when a build module reports its toolchain (issue 1444), part 1 of 2 2026-10-20 20:47:35 +00:00
mesh-tools 340bf64388 Merge pull request 'The bridge carries each module's manifest and lock hashes, and the indexer writes manifests by digest and each module's new-form inputs (design 47 R4c-2, issues 1448, 1449, 1450, 1444, 1451, 1458)' (#10) from feat/r4c2-new-keys into main 2026-10-20 19:31:05 +00:00
mesh-tools c9f742d5f2 Read a merge's manifests and lock files at most eight at once, so a catalogue merge stays inside the bridge's bound (issues 1448, 1458, design 47 R4c-2)
mesh/repo-check head c9f742d5f20e merged into trunk 2520cc4a88f7 — pass: its merge-check.sh passed
mesh/merge-gate head c9f742d5f20e merged into trunk 2520cc4a88f7 — pass: group feat/r4c2-new-keys composed: every machine composes with the change as it di…
mesh/delivery delivered
mesh/delivery-group group feat/r4c2-new-keys merged: every member is merged
2026-10-20 19:06:03 +00:00
mesh-tools f69a561a16 Test how a manifest and lock hashes become a module's new-form inputs (issue 1450, design 47 R4c-2) 2026-10-20 18:55:54 +00:00
mesh-tools 27bfd83fa0 Test the kept manifests, the new-form reasons and the candidate's manifests against real files (issues 1449, 1444, 1451, design 47 R4c-2) 2026-10-20 18:55:29 +00:00
mesh-tools a75d9cc851 Keep manifests and say why no new-form fingerprint exists, so the group check can start from facts (issues 1449, 1450, 1444, 1451, design 47 R4c-2) 2026-10-20 18:54:28 +00:00
mesh-tools ba7ce55887 Give the indexer a manifests bucket and the SDK that knows the new keys (issue 1449, design 47 R4c-2) 2026-10-20 18:53:40 +00:00
mesh-tools 3c1e7ad464 Test the bridge's manifest and lock file conversion (issue 1448, design 47 R4c-2) 2026-10-20 18:53:07 +00:00
mesh-tools 41225bf459 Test the forge bridge's file reads against the stand-in forge and the bus (issue 1448, design 47 R4c-2) 2026-10-20 18:52:27 +00:00
mesh-tools 543d91d626 Read each module's manifest and lock files in the forge bridge, so the indexer gets them (issue 1448, design 47 R4c-2) 2026-10-20 18:51:27 +00:00
mesh-tools d650a41ba6 Copy the real manifests the indexer's new-form tests read (issues 1450, 1444)
Fixtures are copied from the writer, never written by hand: four catalogue manifests at mesh-catalog a67f0196 and build-agent's at mesh-controller f2d7bc94.
2026-10-20 18:48:08 +00:00
mesh-tools 2520cc4a88 Merge pull request 'The parts serve the gate: the bridge performs owed acts and merges, the teller carries asks, the indexer reads source.main, the view shows asks (design 47 R3e, issues 1341-1345)' (#9) from feat/r3e-the-gate into main 2026-10-20 06:49:19 +00:00
mesh-tools c93954e53c Pin mesh-sdk to main's merge of #47 (88581d0a), the same tree as ba684e54 the review read
mesh/repo-check head c93954e53c8e merged into trunk 9c15112d935a beside mesh-catalog#380@cf5c0273 +1 more — pass: its merge-check.sh passed
mesh/merge-gate head c93954e53c8e merged into trunk 9c15112d935a — pass: group feat/r3e-the-gate composed: every machine composes with the change as it did…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792478961597980735 (novox/mesh-delivery at 2520cc4a into main…
mesh/delivery-group group feat/r3e-the-gate merged: every member is merged
2026-10-20 06:33:01 +00:00
mesh-tools bf59b5a27b An answer to an earlier asking no longer blocks the next asking: the teller and the view tell the asking by the answer's warrant id (review round 2, R2-1) 2026-10-20 01:35:26 +00:00
mesh-tools 14e9d56a22 Ask under each asking's own router id, spell the heads in lower case as the gate does, show a removed have plainly; pin mesh-sdk ba684e5 (review round 1: G2, T1, issue 1370) 2026-10-20 01:30:54 +00:00
mesh-tools 34f585106f Show the open asks in the delivery view: the gate's asks with no answer, in the table, the page and the command's JSON (issue 1345) 2026-10-20 00:34:28 +00:00
mesh-tools 05c180cf87 Add the indexer's source revision logic and its tests that the last commit named (issue 1344) 2026-10-20 00:31:19 +00:00
mesh-tools 409c41d4ae Stamp the modules with the gate's source revision, read from source.main (issue 1344) 2026-10-20 00:30:52 +00:00
mesh-tools 5ba1da5b82 Carry the gate's merge asks to the router and write the operator's answer back, a warrant that does not hold said as a condition (issue 1343) 2026-10-20 00:28:36 +00:00
mesh-tools 8ee56dbb5c Grant the bridge's test principal only a read of the gate's buckets, as the controller composes it (issues 1341, 1342) 2026-10-20 00:24:40 +00:00
mesh-tools be5c1b371e Merge the pull request a gate's merges key hands the bridge, with the Approve's heads, one pass stopping at a refusal (issue 1342) 2026-10-20 00:17:44 +00:00
mesh-tools eeb5561a23 Perform the gate's owed acts once per revision from the bridge, a refusal written on FORGE (issue 1341) 2026-10-20 00:16:43 +00:00
mesh-tools a0883a174b Test the bridge's two state buckets, heads and performed (issue 1341) 2026-10-20 00:15:54 +00:00
mesh-tools 7686cfc55d Turn an owed key into the forge call that performs it, and a refusal into a refused message (issue 1341) 2026-10-20 00:15:29 +00:00
mesh-tools 507fac41fe Declare the bridge's performed bucket (issue 1341) 2026-10-20 00:14:42 +00:00
mesh-tools 9c15112d93 Merge pull request 'Take the SDK that counts a build outcome as the controller writes it, so modules get their fingerprints (issue 1281)' (#8) from fix/1281-outcome-is-a-build into main 2026-10-19 23:02:00 +00:00
mesh-tools 3c655c5ab4 Write a failed build as the controller does, an outcome with failed set (issue 1281)
mesh/repo-check head 3c655c5ab449 merged into trunk 6a23627426c6 — pass: its merge-check.sh passed
mesh/merge-gate head 3c655c5ab449 merged into trunk 6a23627426c6 — pass: group fix/1281-outcome-is-a-build composed: every machine composes with the change…
mesh/delivery delivered
mesh/delivery-group group fix/1281-outcome-is-a-build merged: every member is merged
2026-10-19 22:47:41 +00:00
mesh-tools 8ffcf02a9c Take the SDK that counts a stateless build record as an outcome; fixtures in the controller's layout (issue 1281) 2026-10-19 22:44:21 +00:00
mesh-tools 6a23627426 Merge pull request 'Recognise a trunk build by repository identity and an empty ref in the indexer and the view (issue 1281)' (#6) from fix/1281-trunk-builds-by-repository into main 2026-10-19 20:58:07 +00:00
jschoubben 4b8e71370c Recognise a trunk build by repository identity and an empty ref in the indexer and the view (issue 1281)
mesh/repo-check head 4b8e71370cc2 merged into trunk fa4be27c1e40 — pass: its merge-check.sh passed
mesh/merge-gate head 4b8e71370cc2 merged into trunk fa4be27c1e40 — pass: group fix/1281-trunk-builds-by-repository composed: every machine composes with th…
mesh/delivery delivered
mesh/delivery-group group fix/1281-trunk-builds-by-repository merged: every member is merged
Build records name their repository by clone URL and a plan-asked build has no ref, so no module fingerprint was ever written or compared; both now use the SDK's tests.
2026-10-19 22:39:51 +02:00
924 changed files with 135360 additions and 1446 deletions
+3
View File
@@ -1 +1,4 @@
vendor/
# A compiled program never enters a commit (issue 1577): `go build .` in a command directory leaves one there.
modules/*/cmd/*/mesh-*
!modules/*/cmd/*/mesh-*.go
-74
View File
@@ -1,74 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# This repository's own check, mesh/repo-check (novox/hq ADR 0238 §4), written with its first commit (ADR 0428 §4):
# every Go module in the tree is vetted, its unit tests run, and then its integration tests (-tags integration).
# A tree with no Go module has nothing to test, and the check says so and passes.
#
# No module commits its dependencies (no vendor directory: .gitignore). A module's dependencies are fetched, or a
# requirement of the mesh's own forge (the pinned mesh-sdk commit) is taken from its repository's clone beside the check
# (MESH_CHECK_BESIDE), through a go.mod of its own (-modfile), by mesh-catalog's helper checks/deps, built from the
# clone of mesh-catalog beside this check and not copied here: the build seat forgets the forge credential before the
# check runs, so a commit the seat's module cache does not hold cannot be fetched from the forge (novox/hq issue 385). A
# module whose dependencies cannot be had **fails the check**, by name and why: its tests could not run, and a test that
# could not run is a failure, never a note. Run by hand, without MESH_CHECK_BESIDE, the dependencies are fetched.
set -eu
cd "$(dirname "$0")"
# The toolchain the check runs in sets -mod=vendor; there is no vendor directory, so the modules are used as they are.
GOFLAGS="$(printf '%s' "${GOFLAGS:-}" | sed 's/-mod=[a-z]*//g') -mod=mod"
export GOFLAGS
export GOPRIVATE=git.novox.be
modules=$(find . -name go.mod -not -path '*/vendor/*' -not -path './.git/*' | sort)
if [ -z "$modules" ]; then
echo "This tree holds no Go module, so there are no tests to run; the check passes."
exit 0
fi
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
deps=""
helper="${MESH_CHECK_BESIDE:-}/mesh-catalog/checks/deps"
if [ -n "${MESH_CHECK_BESIDE:-}" ] && [ -f "$helper/main.go" ]; then
if ! (cd "$helper" && go build -o "$work/deps" .); then
echo "FAILED: the dependency helper of mesh-catalog beside this check could not be built"
exit 1
fi
deps="$work/deps"
elif [ -n "${MESH_CHECK_BESIDE:-}" ]; then
echo "NOT TAKEN FROM BESIDE: no mesh-catalog/checks/deps beside this check ($MESH_CHECK_BESIDE); the dependencies are fetched"
fi
untestable=""
for mod in $modules; do
dir=$(dirname "$mod")
modfile=""
if [ -n "$deps" ]; then
if ! modfile=$("$deps" -beside "$MESH_CHECK_BESIDE" -work "$work" "$dir"); then
echo "FAILED: $dir could not be tested: $modfile"
untestable="$untestable $dir"
continue
fi
[ -n "$modfile" ] && echo "$dir takes its forge dependencies from the clones beside the check"
elif ! fetched=$(cd "$dir" && go mod download 2>&1); then
echo "FAILED: $dir could not be tested: its dependencies cannot be fetched: $(printf '%s\n' "$fetched" | tail -n 1)"
untestable="$untestable $dir"
continue
fi
mf=""
[ -n "$modfile" ] && mf="-modfile=$modfile"
unformatted=$(cd "$dir" && gofmt -l .)
if [ -n "$unformatted" ]; then
echo "$dir is not gofmt'd: $unformatted"
exit 1
fi
echo "== $dir: go vet and the unit tests"
(cd "$dir" && go vet $mf ./... && go test $mf ./...)
echo "== $dir: the integration tests (-tags integration)"
(cd "$dir" && go vet $mf -tags integration ./... && go test $mf -tags integration ./...)
done
if [ -n "$untestable" ]; then
echo "the check fails: the tests of$untestable could not run (novox/hq issue 385)"
exit 1
fi
@@ -0,0 +1,73 @@
package main
import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Every node's published compositions, read only to compare (novox/hq ADR 0593 §8): each node's engine writes its own
// `composed-<node>` (delivery.ComposedRead, one bucket per node), and `mesh-wants.upgrade` with compare names, for every
// node, the composition its own have runs beside the newest it published of each module and of the node, so a person sees
// where they part. The assigner composes nothing and never writes these.
// publishedRow is one composition a node published: the module ("node" for its own), the fingerprint, the digest, or why
// it was refused or cannot be read.
type publishedRow struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp,omitempty"`
Digest string `json:"digest,omitempty"`
Refused string `json:"refused,omitempty"`
Unread string `json:"unread,omitempty"`
}
// published is, by node, every composition it published, sorted by key, from every key of the per-node state (a key
// names its node first): each value decoded and digested here, as the node-engine digests it (delivery.ComposedDigestOf).
// A value or key that does not read is named with why, never left out.
func published(values map[string][]byte) map[string][]publishedRow {
byNode := map[string]map[string][]byte{}
for key, raw := range values {
node, _, _ := strings.Cut(key, ".")
if byNode[node] == nil {
byNode[node] = map[string][]byte{}
}
byNode[node][key] = raw
}
out := map[string][]publishedRow{}
for node, keys := range byNode {
var rows []publishedRow
for key, raw := range keys {
row := publishedRow{Key: key}
_, module, fp, isNode, err := delivery.ParseComposedKey(key)
switch {
case err != nil:
row.Unread = err.Error()
case isNode:
row.Module = "node"
default:
row.Module, row.FP = module, fp
}
if err == nil {
c, derr := delivery.DecodeComposed(raw)
switch {
case derr != nil:
row.Unread = derr.Error()
case c.Refused != "":
row.Refused = c.Refused
default:
if d, err := delivery.ComposedDigestOf(c); err != nil {
row.Unread = err.Error()
} else {
row.Digest = d
}
}
}
rows = append(rows, row)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Key < rows[j].Key })
out[node] = rows
}
return out
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,463 @@
// Command mesh-delivery-assigner is delivery's assigner (novox/hq ADR 0429 §1 and §6, ADR 0446, ADR 0528, issue 1297): a
// process on its own bus credential that watches what the indexer, the builder, the controller, the node-engines and
// the watchman write, and writes what each node should run (`want.<node>.<module>`, by the six rules), the
// fingerprints it holds, every difference between a want and what the controller sends, and the composition check's
// verdict of each group the checker judges. In release 5 the wants are only compared: the controller still sends what
// each node runs (ADR 0429 §7). It holds no timer: every pass is started by a change it watched.
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"slices"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provisions"
"mesh-delivery-assigner/internal/assign"
)
func main() {
if os.Getenv(process.BrokerFileEnv) == "" {
// Started by the tool runner, with no credential of its own: the put's tools (ADR 0546), served through the runtime,
// since a process never serves (ADR 0436). The process the manifest runs names its credential.
if err := serveTools(); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-assigner:", err)
os.Exit(1)
}
return
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-assigner:", err)
os.Exit(1)
}
}
// readBuckets are the buckets of other parts the assigner reads, each named in its manifest's reads (the manifest test
// holds the two lists equal), so the controller's broker grants exactly these.
var readBuckets = []string{
delivery.PartGrantsBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.ManifestsBucket,
delivery.ModulesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
provisions.ProvisionsBucket,
delivery.SourceBucket,
delivery.MergesBucket,
}
// readFamilies are the states kept one bucket per node the assigner reads, each named in its manifest's reads: every
// node's compositions, which mesh-wants.upgrade compares (novox/hq ADR 0593 §8).
var readFamilies = []string{delivery.ComposedRead}
// ownBuckets are the assigner's own states, as its manifest declares them: the seat mesh-placement's three among them
// (ADR 0567 §1), named after the assigner as every module state is.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket, assign.FindingsBucket, assign.PoliciesBucket, delivery.SeatHoldersBucket}
// seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants
// them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind).
var seatBuckets = []string{delivery.PlacementBucket, delivery.PlacementProposalsBucket, delivery.PlacementCandidatesBucket,
delivery.PlacementPulsesBucket}
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9), and its placement by
// the builder, the checker and the indexer, through mesh-sdk's toolchain gate (issue 774, ADR 0549 §3).
var keptStates = []announce.State{
{Name: delivery.PlacementState, Readable: true},
{Name: delivery.PlacementProposalsState, Readable: true},
{Name: delivery.PlacementCandidatesState, Readable: true},
{Name: delivery.PlacementPulsesState, Readable: true},
{Name: "asks", Readable: true},
{Name: "differences", Readable: true},
{Name: "puts", Readable: true},
{Name: "wants", Readable: true},
{Name: health.FindingsState, Readable: true},
{Name: delivery.AssignerPoliciesState, Readable: true},
{Name: delivery.SeatHoldersState, Readable: true},
}
// buckets is every bucket the assigner opened, by name.
type buckets map[string]jetstream.KeyValue
// open opens every bucket the assigner reads and writes; a bucket it cannot open stops the start, named with its grant (issue 1945),
// since a pass without it would write wants from half the mesh. The puts' bucket is the exception: whatever keeps it from opening (it
// does not exist yet, the credential is not granted it yet, the bus does not answer in time), the assigner writes wants
// without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kv, err := js.KeyValue(ctx, name)
if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue
}
if slices.Contains(seatBuckets, name) && err != nil {
// The seat's states come in the same delivery as this assigner (ADR 0567 §1): without placement no module is placed.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == delivery.SeatHoldersBucket && err != nil {
// The seat's state, new with ADR 0571 §4: bound once the bus says it was made; until then nobody is told who
// holds which seat here, and the controller's seats still answer.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == provisions.ProvisionsBucket && err != nil {
// The seat mesh-provisioner's state, with its own delivery (ADR 0570 §4): bound once the bus says it was made;
// until then no want waits for a provision, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == assign.PoliciesBucket && err != nil {
// The seat mesh-wants' state, new with it (ADR 0565 §7): bound once the bus says it was made; until then no
// person's policy is read, so the controller's beside each composition holds, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == assign.FindingsBucket && err != nil {
// New with health leaving the controller (issue 1952 WM6): without it the conditions are said straight to the
// router and kept as no finding.
logf("the bucket %s cannot be opened (%v): the assigner's conditions are said straight to the router until it starts again", name, err)
continue
}
if (name == delivery.SourceBucket || name == delivery.MergesBucket) && err != nil {
// The gate's source and merges (ADR 0601 §1): without them no release is known to carry a build, so every move of
// a record module is asked of a person, as before.
logf("the bucket %s cannot be opened (%v): no release's Approve answers a record ask until it is bound", name, err)
continue
}
if name == delivery.PartGrantsBucket && err != nil {
// The grants part's grants (ADR 0571 clause 9): without them no module counts as granted until it is bound.
logf("the bucket %s cannot be opened (%v): no grant is read until it is bound once the bus says it was made", name, err)
continue
}
if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
}
out[name] = kv
}
return out, nil
}
// startsWithout says the assigner starts without the bucket when it cannot be opened (open says why for each): the seat's
// states and those new with a delivery of their own, bound once made, and the two that choose the grants' source.
func startsWithout(name string) bool {
switch name {
case delivery.SeatHoldersBucket, provisions.ProvisionsBucket, assign.PoliciesBucket, assign.FindingsBucket,
delivery.AssignerPutsBucket, delivery.SourceBucket, delivery.MergesBucket:
return true
}
return slices.Contains(seatBuckets, name) || name == delivery.PartGrantsBucket
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
account, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
conn, js, err := account.Dial()
if err != nil {
return err
}
defer conn.Close()
kvs, err := open(ctx, js, logger.Printf)
if err != nil {
return err
}
// What the assigner keeps, said on the bus (ADR 0495 §2-§3): a reader is granted from it. Not fatal when it cannot
// be said now: process.Conn says it again at every reconnect.
said, err := process.Connect(account, nats.Name(account.Node+"."+account.Module+".keeps"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer said.Close()
if err := said.Keeps(keptStates...); err != nil {
logger.Printf("what the assigner keeps could not be said on the bus, so its readers are not granted until it is: %v", err)
}
thisNode = account.Node
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
findings := findingsOf(ctx, kvs, logger.Printf)
return follow(ctx, js, conn, kvs, findings.Say(sayVia(ctx, js)), findings, emitting{said}, logger)
}
// findingsOf is where the assigner's said conditions go as findings (issue 1952 WM6): its findings bucket, which the
// condition keeper carries (novox/hq ADR 0590 §3). Without it the conditions are said straight to the router.
func findingsOf(ctx context.Context, kvs buckets, logf func(string, ...any)) assign.Findings {
var f assign.Findings
if kv := kvs[assign.FindingsBucket]; kv != nil {
f.Put = func(key string, value []byte) error { _, err := kv.Put(ctx, key, value); return err }
f.Delete = func(key string) error { return kv.Delete(ctx, key) }
f.Keys = func() ([]string, error) {
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
}
return f
}
// thisNode is the node this process runs on, as its credential names it: said with each of the seat's events.
var thisNode string
// emitting is the process connection as the puts' caller: its calls, and its events with no options.
type emitting struct{ *process.Conn }
func (e emitting) Emit(eventType string, body any) error { return e.Conn.Emit(eventType, body) }
// watched are the buckets whose every value a pass reads; the manifests are read by digest when a pass needs one, and
// watched only for keys, so a manifest that arrives wakes a composition that waits for it.
var watched = []string{
delivery.PartGrantsBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
delivery.ModulesBucket,
delivery.PlacementBucket,
delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket,
delivery.PlacementPulsesBucket,
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.AssignerAsksBucket,
delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket,
assign.PoliciesBucket,
provisions.ProvisionsBucket,
delivery.SeatHoldersBucket,
delivery.SourceBucket,
delivery.MergesBucket,
}
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, findings assign.Findings, bus caller, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
select {
case dirty <- struct{}{}:
default:
}
}
errs := make(chan error, len(watched)+2)
settled := make(chan struct{}, len(watched)+2)
watch := func(name string, opts ...jetstream.WatchOpt) error {
kw, err := kvs[name].WatchAll(ctx, opts...)
if err != nil {
return fmt.Errorf("the bucket %s cannot be watched: %w", name, err)
}
// The puts' watch never stops the wants (ADR 0546): when it ends, the puts are no longer heard, said in the journal.
optional := name == delivery.AssignerPutsBucket
go func() {
defer kw.Stop()
for {
select {
case <-ctx.Done():
errs <- nil
return
case e, ok := <-kw.Updates():
if !ok {
if optional {
logger.Printf("the watch of %s ended: no put is taken further until the assigner starts again", name)
return
}
errs <- fmt.Errorf("the watch of %s ended", name)
return
}
if e == nil {
if !optional { // the puts never gate the first pass, so never the wants (ADR 0546)
settled <- struct{}{}
}
continue
}
var value []byte
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
wake()
}
}
}()
return nil
}
// The puts' watch is not waited for before the first pass: a watch of them that ends before it handed what stands only
// stops the puts, never the wants.
if kvs[delivery.AssignerPutsBucket] != nil {
if err := watch(delivery.AssignerPutsBucket); err != nil {
logger.Printf("%v: no put is taken further until it is opened again", err)
delete(kvs, delivery.AssignerPutsBucket)
}
}
// The puts' bucket is made by the controller from the manifest's states at its start (ADR 0201), and the assigner's
// credential reads it once the bus's user list carries the grant: neither waits for the assigner. So while it is not open,
// a pass started by a change opens it again, at most once a minute, and from then on the puts are watched and taken
// further, without a restart.
var triedPuts time.Time
reopenPuts := func(p *passer) {
if kvs[delivery.AssignerPutsBucket] != nil || time.Since(triedPuts) < PutsReopenEvery {
return
}
triedPuts = time.Now()
kv, err := js.KeyValue(ctx, delivery.AssignerPutsBucket)
if err != nil {
return
}
kvs[delivery.AssignerPutsBucket] = kv
if err := watch(delivery.AssignerPutsBucket); err != nil {
logger.Printf("%v: no put is taken further until it is opened again", err)
delete(kvs, delivery.AssignerPutsBucket)
return
}
if bus != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
}
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
}
started := 0
for _, name := range watched {
if kvs[name] == nil && startsWithout(name) {
continue // not on the bus at this start (open said so)
}
started++
var opts []jetstream.WatchOpt
if name == delivery.HaveBucket || name == delivery.SourceBucket {
// what each node ran healthy before, for a set-back; and the release behind each source revision (ADR 0601 §1)
opts = append(opts, jetstream.IncludeHistory())
}
if err := watch(name, opts...); err != nil {
return err
}
}
if err := watch(delivery.ManifestsBucket, jetstream.MetaOnly()); err != nil {
return err
}
// The seat's states arrive with the same delivery as this assigner (ADR 0567 §1): one not on the bus at this start is
// bound once the bus says it was created, and watched from then on, with no restart (issue 1790's kind). Until then no
// module is placed.
type appeared struct {
name string
kv jetstream.KeyValue
}
placementAppeared := make(chan appeared, len(seatBuckets)+2)
for _, name := range seatBuckets {
if kvs[name] == nil && js != nil {
process.AwaitBucket(ctx, js, nc, name, "the seat mesh-placement waits for it",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
// And the two that choose where grants are read from, with the same delivery as the grants part that writes one.
for _, name := range []string{delivery.PartGrantsBucket, assign.PoliciesBucket, provisions.ProvisionsBucket, delivery.SeatHoldersBucket} {
if kvs[name] == nil && js != nil {
name := name
process.AwaitBucket(ctx, js, nc, name, "until then the bucket is not read",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
takePlacement := func(a appeared) error {
if kvs[a.name] != nil {
return nil // bound already, by the other of its two events
}
kvs[a.name] = a.kv
logger.Printf("the bucket %s is open and watched", a.name)
return watch(a.name)
}
for range started + 1 {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{}}
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
}
if bus != nil {
p.seat = newSeatActs(bus, wake, logger.Printf)
}
if js != nil {
p.publish = streamPublisher(ctx, js, thisNode)
}
wake()
for {
select {
case <-ctx.Done():
return nil
case err := <-errs:
return err
case <-settled:
case kv := <-placementAppeared:
if err := takePlacement(kv); err != nil {
return err
}
case <-dirty:
reopenPuts(p)
// The grants part makes the seat's buckets in the pass that seals the list granting them, so their creation can
// be said before this credential may hear it: every pass, itself started by a watched change, binds each one the
// bus now holds, and says the seat waits while one is missing.
var unbound []string
for _, name := range seatBuckets {
if kvs[name] != nil || js == nil {
continue
}
if kv, err := js.KeyValue(ctx, name); err == nil {
if err := takePlacement(appeared{name, kv}); err != nil {
return err
}
continue
}
unbound = append(unbound, name)
}
p.sayWaits(unbound)
p.pass(ctx)
}
}
}
// PutsReopenEvery is how often, at most, a pass opens the puts' bucket again while it is not open.
var PutsReopenEvery = time.Minute
// sayVia publishes a condition on the operator-channel seat's one subject for conditions, under the assigner's own name
// (ADR 0437 clause 9).
func sayVia(ctx context.Context, js jetstream.JetStream) func(asks.Condition) error {
return func(c asks.Condition) error {
subject, err := asks.ConditionSubject(delivery.AssignerModule)
if err != nil {
return err
}
data, err := asks.EncodeCondition(c)
if err != nil {
return err
}
_, err = js.Publish(ctx, subject, data)
return err
}
}
@@ -0,0 +1,185 @@
package main
import (
"mesh-delivery-assigner/internal/assign"
"context"
"errors"
"fmt"
"io"
"log"
"os"
"path/filepath"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
// (`<module>.<state>` read as the bucket `<module>_<state>`), so it is granted what it reads and reads nothing else.
func TestOpensWhatItsManifestNames(t *testing.T) {
// 1297/B2a: without a credential the program does not start, and says why.
if err := run(context.Background(), func(string) string { return "" }, io.Discard); err == nil || !strings.Contains(err.Error(), "MESH_BROKER_FILE") {
t.Errorf("run without a credential = %v", err)
}
raw, err := os.ReadFile(filepath.Join("..", "..", "module.json"))
if err != nil {
t.Fatal(err)
}
m, err := catalogue.Decode(raw)
if err != nil {
t.Fatal(err)
}
var reads []string
for _, r := range m.Reads {
if slices.Contains(readFamilies, r) {
continue // a state kept one bucket per node, read by its prefix (ADR 0593 §6), not one bucket
}
reads = append(reads, strings.Replace(r, ".", "_", 1))
}
for _, l := range readFamilies {
if !slices.Contains(m.Reads, l) {
t.Errorf("the program reads the per-node state %s, the manifest does not name it", l)
}
}
var states []string
for _, s := range m.State {
states = append(states, m.Module+"_"+s.Name)
}
for _, pair := range []struct {
what string
code, want []string
}{{"reads", readBuckets, reads}, {"states", ownBuckets, states}} {
got, want := slices.Sorted(slices.Values(pair.code)), slices.Sorted(slices.Values(pair.want))
if !reflect.DeepEqual(got, want) {
t.Errorf("the program's %s are %v, the manifest's %v", pair.what, got, want)
}
}
var kept []string
for _, s := range keptStates {
kept = append(kept, m.Module+"_"+s.Name)
}
if got, want := slices.Sorted(slices.Values(kept)), slices.Sorted(slices.Values(ownBuckets)); !reflect.DeepEqual(got, want) {
t.Errorf("the program announces %v, keeps %v", got, want)
}
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
for _, c := range []struct {
missing string
err error
starts bool
said string
}{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, ""},
// Issue 774: the grants part's grants not there do not stop the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, "no grant is read"},
} {
var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, c.said)):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant
t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err)
}
}
// And a watch of the puts that ends before it handed what stands (its subscription closed, its grant withdrawn) does not
// keep the first pass from being made: the wants are written all the same.
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
wrote := make(chan string, 8)
kvs := buckets{}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
kvs[name] = &watchKV{memKV: &memKV{values: map[string][]byte{}}}
}
placed, err := delivery.PlacementValues(delivery.Placement{Node: "ace"})
if err != nil {
t.Fatal(err)
}
for key, raw := range placed {
kv := kvs[delivery.PlacementBucket].(*watchKV)
kv.entries = append(kv.entries, kvEntry{key: key, value: raw})
}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, assign.Findings{}, nil, log.New(io.Discard, "", 0))
select {
case key := <-wrote:
if key != "want.ace.node" {
t.Errorf("the first pass wrote %s", key)
}
case <-ctx.Done():
t.Error("a puts watch that ended before its first marker kept the first pass from being made: no want was written")
}
}
// refusingJS opens every bucket but one, which it refuses with err.
type refusingJS struct {
jetstream.JetStream
missing string
err error
}
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing {
return nil, r.err
}
return &memKV{values: map[string][]byte{}}, nil
}
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
// closed; a put on it is said on wrote.
type watchKV struct {
*memKV
entries []kvEntry
closed bool
wrote chan string
}
func (w *watchKV) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
ch := make(chan jetstream.KeyValueEntry, len(w.entries)+1)
if w.closed {
close(ch)
return keyWatcher{ch}, nil
}
for _, e := range w.entries {
ch <- e
}
ch <- nil
return keyWatcher{ch}, nil
}
func (w *watchKV) Put(ctx context.Context, key string, value []byte) (uint64, error) {
if w.wrote != nil {
w.wrote <- key
}
return 1, nil
}
type keyWatcher struct{ ch chan jetstream.KeyValueEntry }
func (k keyWatcher) Updates() <-chan jetstream.KeyValueEntry { return k.ch }
func (k keyWatcher) Stop() error { return nil }
type kvEntry struct {
jetstream.KeyValueEntry
key string
value []byte
}
func (e kvEntry) Key() string { return e.key }
func (e kvEntry) Value() []byte { return e.value }
func (e kvEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
func (e kvEntry) Revision() uint64 { return 1 }
func (e kvEntry) Created() time.Time { return time.Time{} }
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,455 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"mesh-delivery-assigner/internal/assign"
)
// The puts (novox/hq ADR 0546): the tool `assigner_put` writes the key `<node>.<module>` of the assigner's puts, and each pass
// takes every put that has not ended one step further (assign.StepPut), doing the acts the step asks for: the put's ask, the
// bus's snapshot, the warranted record. Each act's outcome is heard back as a change of the world (the composition, the grant,
// the answer) or, for the calls, as a wake once they answer; nothing polls but the snapshot's read-back of the backup holder,
// as the controller's bus step reads it (ADR 0235), bounded. Every step is said as the event mesh-delivery-assigner.put, so a
// caller waits on it.
// PutEvent is the event a put's every step is said as: mesh.mod.mesh-delivery-assigner.event.put.
const PutEvent = "put"
// caller is what the puts need of the bus as the assigner's own process account: a call of a seat's verb, on one node or on
// the seat held once, and an event said. process.Conn, or a test's.
type caller interface {
Call(key string, body any) (json.RawMessage, error)
CallOn(node, key string, body any) (json.RawMessage, error)
Emit(eventType string, body any) error
}
// putActs keeps, by put (its key and when it was asked), what the acts of a put came to while this process runs: a restart
// takes the snapshot again, without harm.
type putActs struct {
mu sync.Mutex
bus caller
wake func()
logf func(string, ...any)
snapping map[string]bool
snapped map[string]string
snapNo map[string]string
warranted map[string]bool
// rec records the put's warrant in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
warrantNo map[string]string
said map[string]string
// snapshot takes the bus's snapshot on node; a variable so a test takes none.
snapshot func(ctx context.Context, node, module string) (string, error)
}
func newPutActs(bus caller, wake func(), logf func(string, ...any)) *putActs {
a := &putActs{rec: handacts.Recorder{Ask: bus.Call}, bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
said: map[string]string{}}
a.snapshot = a.snapshotTheBus
return a
}
// identity names one put: its key and when it was asked, so a put asked again under the key is another.
func identity(key string, p delivery.Put) string {
return key + "@" + p.At.UTC().Format(time.RFC3339Nano)
}
// putsByModule is every put the assigner holds, by module then node.
func (p *passer) putsByModule() map[string]map[string]delivery.Put {
out := map[string]map[string]delivery.Put{}
for key, value := range p.w.read(delivery.AssignerPutsBucket) {
node, module, err := delivery.ParsePutKey(key)
if err != nil {
continue
}
put, err := delivery.DecodePut(value)
if err != nil {
p.logf("the put %s cannot be read, and is left: %v", key, err)
continue
}
if out[module] == nil {
out[module] = map[string]delivery.Put{}
}
out[module][node] = put
}
return out
}
// puts takes every put that has not ended one step further.
func (p *passer) puts(ctx context.Context) {
if p.acts == nil {
return
}
all := p.putsByModule()
if len(all) == 0 {
return
}
assigned := p.w.assigned()
trunk := p.w.trunk()
get := kvGetter(p.kvs[delivery.ManifestsBucket])
wanted := p.w.wantsOf()
modules := make([]string, 0, len(all))
for m := range all {
modules = append(modules, m)
}
sort.Strings(modules)
for _, module := range modules {
var manifest catalogue.Manifest
if mod, ok := trunk[module]; ok && mod.Manifest != "" {
if got := p.w.manifest(ctx, get, mod.Manifest); got.err == "" {
manifest = got.manifest
}
}
for node, put := range all[module] {
if delivery.PutEnded(put.Step) {
continue
}
key, _ := delivery.PutKey(node, module)
id := identity(key, put)
world := p.putWorld(node, module, put, manifest, assigned[node])
world.Snapshot, world.Snapshotting, world.SnapshotFailed = p.acts.state(id)
next := assign.StepPut(world)
// A held bus whose user list the rules moved on (ADR 0539 §7): the put follows the composition its want names.
if w, ok := wanted[module][node]; ok && w.Put && w.FP == next.Put.FP && w.Composed != "" && w.Composed != next.Put.Composed &&
(next.Put.Step == delivery.PutWanted || next.Put.Step == delivery.PutHolding) {
if d, _ := p.composedFor(node, module, w.FP); d != nil && d.UserListOnly && d.Digest == w.Composed {
next.Put = next.Put.Said(world.Now, next.Put.Step, fmt.Sprintf("the bus's user list changed: the want names composition %s", shortHex(w.Composed)))
next.Put.Composed = w.Composed
}
}
// The bus's want is written only for the act the operator's yes bound (ADR 0546 clause 8).
if next.Put.Step == delivery.PutWanted && put.Step != delivery.PutWanted && world.Bus && next.Put.Ask != "" {
if err := p.performs(next.Put.Ask, assign.PutAct(node, module, next.Put.FP, next.Put.Composed)); err != nil {
next = assign.PutNext{Put: next.Put.Said(world.Now, delivery.PutRefused, "the bus's want is not written: "+err.Error())}
}
}
if failed := p.acts.warrantFailed(id); failed != "" && (next.Put.Step == delivery.PutWanted || next.Put.Step == delivery.PutHolding) &&
!strings.Contains(next.Put.Words, failed) {
next.Put = next.Put.Said(world.Now, next.Put.Step, next.Put.Words+"; "+failed)
}
p.actOn(ctx, node, module, id, world, next)
written := samePut(put, next.Put) || p.writePut(ctx, key, put, next.Put)
if !written {
continue // released or put again meanwhile: nothing is done for the put that no longer stands
}
if next.Put.Step == delivery.PutWanted && put.Step != delivery.PutWanted {
// The want is written in this pass too, before the next one would: Module.Puts writes the same.
if wkey, err := delivery.WantKey(node, module); err == nil {
want := assign.PutWant(next.Put)
if before, ok := wanted[module][node]; !ok || !delivery.SameWant(normalWant(before), normalWant(want)) {
p.put(ctx, delivery.AssignerWantsBucket, wkey, func() ([]byte, error) { return delivery.EncodeWant(want) })
}
}
if next.Warranted {
p.acts.warrant(id, delivery.AskingID(next.Put.Ask, assign.PutAsking(next.Put)))
}
}
}
}
}
func normalWant(w delivery.Want) delivery.Want {
w.V = 0
return w
}
// putWorld is what one step of a put reads from the world.
func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalogue.Manifest, assignedHere []string) assign.PutWorld {
w := assign.PutWorld{Node: node, Module: module, Put: put, Now: p.clock(), Bus: catalogue.ProvidesBus(manifest), Newest: p.w.trunk()[module].FP2}
for _, m := range assignedHere {
if m == module {
w.Assigned = true
}
}
if bkey, err := delivery.BuilderBuildKey(module, put.FP); err == nil {
if raw := p.w.value(delivery.BuilderBuildsBucket, bkey); raw != nil {
if b, err := delivery.DecodeBuilderBuild(raw); err == nil {
w.Build, w.BuildWords = b.State, b.Error
} else {
w.Build = "unreadable: " + err.Error()
}
}
}
w.Composed, w.Granted = p.composedFor(node, module, put.FP)
if d := w.Composed; d != nil {
policy := d.Policy
chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket))
if c, ok := chosen[module]; ok {
policy, _, _, _ = catalogue.Upgrade(manifest, &c) // a person's choice at the assigner (ADR 0565 §7)
} else if policy == "" {
policy, _, _ = catalogue.DerivedUpgrade(manifest)
}
w.Recorded = policy == catalogue.PolicyRecord
if akey, err := delivery.RecordAskKey(node, module, d.Digest); err == nil && put.Ask == akey {
// Only this put's own ask counts: an ask (and its answer) of an earlier put under the same key is another asking.
if a, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey)); err == nil && a.Asking == assign.PutAsking(put) {
switch {
case p.answeredWith(akey, assign.OptionYes):
w.Answer = assign.OptionYes
case p.answeredWith(akey, assign.OptionNo):
w.Answer = assign.OptionNo
default:
w.AskExpired = !a.Ask.Expires.IsZero() && w.Now.After(a.Ask.Expires)
}
}
}
}
if hkey, err := delivery.HaveKey(node, module); err == nil {
if raw := p.w.value(delivery.HaveBucket, hkey); raw != nil {
if h, err := delivery.DecodeHave(raw); err == nil {
w.Have = &h
}
}
}
return w
}
// composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one
// place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and
// mesh-controller_grants); the grants are the grants part's while the controller's lease is handed away (grantsBucket).
func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) {
var out *delivery.ComposedDigest
if d, ok := p.w.composed(node, module, fp); ok {
out = &d
}
granted := false
if gkey, err := delivery.PartGrantKey(module, fp); err == nil {
if raw := p.w.value(p.w.grantsBucket(), gkey); raw != nil {
if g, err := delivery.DecodeGrant(raw); err == nil {
granted = g.NamesNode(node, module)
}
}
}
return out, granted
}
// actOn does the acts a step asks for.
func (p *passer) actOn(ctx context.Context, node, module, id string, w assign.PutWorld, next assign.PutNext) {
if next.Put.Step == delivery.PutPerson && next.Put.Ask != "" && w.Composed != nil {
// The put's own ask, written again over one an earlier put left under the key (its asking differs), so the teller
// raises it anew under its own id.
akey := next.Put.Ask
current, err := delivery.DecodeAssignerAsk(akey, p.w.value(delivery.AssignerAsksBucket, akey))
if err != nil || current.Asking != assign.PutAsking(next.Put) {
a, err := assign.PutAsk(akey, node, module, next.Put, *w.Composed, w.Bus, w.Now)
if err != nil {
p.logf("the ask about putting %s on %s is not written: %v", module, node, err)
} else {
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
}
}
}
if next.Snapshot {
p.acts.snap(ctx, id, node, module)
}
}
// performs checks that the act about to be performed is the one the yes of the ask at key bound (asks.Option.Performs).
func (p *passer) performs(key string, act asks.Act) error {
a, err := delivery.DecodeAssignerAsk(key, p.w.value(delivery.AssignerAsksBucket, key))
if err != nil {
return fmt.Errorf("the ask %s cannot be read: %w", key, err)
}
for _, o := range a.Ask.Options {
if o.ID == assign.OptionYes {
return o.Performs(act)
}
}
return fmt.Errorf("the ask %s offers no yes", key)
}
// samePut says two puts say the same: their step, words and what they found.
func samePut(a, b delivery.Put) bool {
return a.Step == b.Step && a.Words == b.Words && a.Composed == b.Composed && a.Ask == b.Ask && a.Snapshot == b.Snapshot
}
// writePut writes a put's next value over the one the pass read, by compare-and-set, unless the tool has put again or
// released it since; and says it as the event put. It says whether it wrote.
func (p *passer) writePut(ctx context.Context, key string, read, next delivery.Put) bool {
kv := p.kvs[delivery.AssignerPutsBucket]
if kv == nil {
return false
}
e, err := kv.Get(ctx, key)
if err != nil {
p.logf("the put %s cannot be read again, and is taken further at the next pass: %v", key, err)
return false
}
now, err := delivery.DecodePut(e.Value())
if err != nil || !now.At.Equal(read.At) || now.Step != read.Step {
return false // asked again or released meanwhile: the next pass reads that
}
data, err := delivery.EncodePut(next)
if err != nil {
p.logf("the put %s is not written: %v", key, err)
return false
}
if _, err := kv.Update(ctx, key, data, e.Revision()); err != nil {
if !errors.Is(err, context.Canceled) {
p.logf("the put %s could not be written, and is taken further at the next pass: %v", key, err)
}
return false
}
p.w.put(delivery.AssignerPutsBucket, key, data)
p.logf("put %s: %s: %s", key, next.Step, next.Words)
p.acts.say(key, next)
return true
}
// state is what this process knows of a put's acts.
func (a *putActs) state(id string) (snapshot string, snapping bool, snapNo string) {
a.mu.Lock()
defer a.mu.Unlock()
return a.snapped[id], a.snapping[id], a.snapNo[id]
}
// snap has the bus's snapshot taken once for a put, in the background.
func (a *putActs) snap(ctx context.Context, id, node, module string) {
a.mu.Lock()
if a.snapping[id] || a.snapped[id] != "" || a.snapNo[id] != "" {
a.mu.Unlock()
return
}
a.snapping[id] = true
a.mu.Unlock()
go func() {
defer a.wake()
where, err := a.snapshot(ctx, node, module)
a.mu.Lock()
defer a.mu.Unlock()
a.snapping[id] = false
if err != nil {
a.snapNo[id] = err.Error()
return
}
a.snapped[id] = where
}()
}
// SnapshotWithin is how long the bus node's backup holder is given to take the bus's snapshot (as the controller's bus step).
var SnapshotWithin = 15 * time.Minute
// snapshotLook is how often the backup holder's record is read back while the snapshot is taken.
var snapshotLook = 10 * time.Second
// backedUp is one module as the backup holder's `backed-up` answers it.
type backedUp struct {
Module string `json:"module"`
Data []struct {
Item string `json:"item"`
LastBackup *time.Time `json:"last_backup"`
Error string `json:"error"`
} `json:"data"`
}
// snapshotTheBus asks the bus node's backup holder to back the bus module up now, whose dump is the streams' snapshot
// (ADR 0235), and reads its record back until it names a backup newer than the ask: where the snapshot is.
func (a *putActs) snapshotTheBus(ctx context.Context, node, module string) (string, error) {
asked := time.Now()
if _, err := a.bus.CallOn(node, "seat:"+catalogue.BackupSeat+".now", map[string]any{"module": module}); err != nil {
return "", fmt.Errorf("%s's backup holder was not asked to take the bus's snapshot: %w", node, err)
}
deadline := asked.Add(SnapshotWithin)
for {
if raw, err := a.bus.CallOn(node, "seat:"+catalogue.BackupSeat+".backed-up", map[string]any{}); err == nil {
var modules []backedUp
if json.Unmarshal(raw, &modules) == nil {
for _, m := range modules {
for _, it := range m.Data {
if m.Module == module && it.LastBackup != nil && it.LastBackup.After(asked) && it.Error == "" {
return fmt.Sprintf("%s's restore point of %s (%s) taken %s", node, module, it.Item, it.LastBackup.UTC().Format(time.RFC3339)), nil
}
}
}
}
}
if time.Now().After(deadline) {
return "", fmt.Errorf("%s's backup holder did not say the bus's snapshot was taken within %s", node, SnapshotWithin)
}
select {
case <-ctx.Done():
return "", ctx.Err()
case <-time.After(snapshotLook):
}
}
}
// warrant records once, in the hand-act log (the seat mesh-hand-acts', or the controller's while that seat has no holder and
// the controller holds its lease, ADR 0565 §11), that the bus's want was written on the operator's warrant: the router's
// record of the teller's ask says who chose and how (ADR 0274).
func (a *putActs) warrant(id, ask string) {
a.mu.Lock()
if a.warranted[id] {
a.mu.Unlock()
return
}
a.warranted[id] = true
a.mu.Unlock()
go func() {
if _, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the put's warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Lock()
a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Unlock()
a.wake()
}
}()
}
// warrantFailed says why the put's warrant was not recorded, "" when it was or is not yet known.
func (a *putActs) warrantFailed(id string) string {
a.mu.Lock()
defer a.mu.Unlock()
return a.warrantNo[id]
}
// putSaid is the body of the event put.
type putSaid struct {
Node string `json:"node"`
Module string `json:"module"`
FP string `json:"fp"`
Hold bool `json:"hold"`
Step string `json:"step"`
Words string `json:"words"`
Composed string `json:"composed,omitempty"`
}
// say says a put's step as the event put, once per step and words.
func (a *putActs) say(key string, p delivery.Put) {
node, module, err := delivery.ParsePutKey(key)
if err != nil || a.bus == nil {
return
}
a.mu.Lock()
if a.said[key] == p.Step+p.Words {
a.mu.Unlock()
return
}
a.said[key] = p.Step + p.Words
a.mu.Unlock()
if err := a.bus.Emit(PutEvent, putSaid{Node: node, Module: module, FP: p.FP, Hold: p.Hold, Step: p.Step, Words: p.Words, Composed: p.Composed}); err != nil {
a.logf("the put %s's step %s is not said as an event: %v", key, p.Step, err)
}
}
// shortHex is a digest's first twelve hex digits.
func shortHex(digest string) string {
h := strings.TrimPrefix(digest, "sha256:")
if len(h) > 12 {
return h[:12]
}
return h
}
@@ -0,0 +1,748 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"reflect"
"slices"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-assigner/internal/assign"
"mesh-delivery-assigner/internal/judge"
)
// The seat mesh-placement's holder in the assigner's process (novox/hq ADR 0567 §2-§8): each pass follows every open
// proposal one step, from what the world holds. A proposal the verbs wrote is said (placement-proposed); the nodes it
// must hear from are found by composing every node with and without it, and those its key does not change get a reached
// candidate; their answers are judged (placement-judged) and, on a passing verdict, the operator is asked to approve it;
// on the warrant each changed key is written by compare-and-set at the revision it was laid over (placement-changed),
// and the controller's assignments are kept in step through its verbs until the lease write. A key moved meanwhile
// lays the proposal again; a decline, an expiry or a withdrawal ends it (placement-withdrawn). Every event is said on the
// assigner's own stream, so only this process says them. Nothing polls: the wake at an answer's bound is the one wake.
// publisher says one of the seat's events on the assigner's stream, under a message id the bus keeps one of.
type publisher func(event string, body any, id string) error
// streamPublisher publishes on mesh.stream.mesh-delivery-assigner.event.<event>, with the headers an emit carries.
func streamPublisher(ctx context.Context, js jetstream.JetStream, node string) publisher {
return func(event string, body any, id string) error {
data, err := json.Marshal(body)
if err != nil {
return err
}
msg := nats.NewMsg(process.ModuleStreamSubject(delivery.AssignerModule, event))
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", delivery.AssignerModule)
msg.Header.Set("x-node", node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
_, err = js.PublishMsg(ctx, msg, jetstream.WithMsgID(id))
return err
}
}
// seatActs keeps the seat's acts while this process runs: the warrant recorded in the hand-act log. The seat calls no verb of
// the controller: placement is the one source (ADR 0567 §8, issue 774).
type seatActs struct {
bus caller
wake func()
logf func(string, ...any)
// rec records the seat's acts in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
}
func newSeatActs(bus caller, wake func(), logf func(string, ...any)) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, rec: handacts.Recorder{Ask: bus.Call}}
}
// record records once, in the hand-act log, that placement was written on the operator's warrant (§5): the router's
// record of the teller's ask says who chose and how (ADR 0274). The log is the seat mesh-hand-acts' from its delivery, and
// the controller's while that seat has no holder (ADR 0565 §11, handacts.Recorder).
func (a *seatActs) record(warrant string) {
if a == nil || warrant == "" {
return
}
ask, _, _ := strings.Cut(warrant, "@")
go func() {
if taken, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the placement's warrant %s is not recorded in the hand-act log: %v", ask, err)
} else {
a.logf("the placement's warrant %s is recorded in the log of %s", ask, taken.Log)
}
}()
}
// clearStaleCalls clears, once, the conditions of the refused controller calls a previous process raised (findings of the
// kinds controller-call and controller-pin): the assigner calls no verb of the controller any more (ADR 0567 §8, issue 774),
// so nothing would clear them. It is tried again at the next pass while the findings cannot be read.
func (p *passer) clearStaleCalls() {
if p.callsCleared || p.say == nil || p.findings.Keys == nil {
return
}
keys, err := p.findings.Keys()
if err != nil {
p.logf("the findings were not read, so the refused controller calls a previous process raised are cleared at the next pass: %v", err)
return
}
p.callsCleared = true
for _, k := range keys {
rest, ok := strings.CutPrefix(k, "module.")
if !ok {
continue
}
for _, kind := range []string{assign.KindControllerCall, assign.KindControllerPin} {
if pair, ok := strings.CutSuffix(rest, "."+kind); ok {
if node, module, two := strings.Cut(pair, "."); two {
if err := p.say(assign.CallClearedCondition(kind, node, module)); err != nil {
p.callsCleared = false
p.logf("the condition of the old %s call for %s on %s could not be cleared, and is cleared at the next pass: %v", kind, module, node, err)
}
}
}
}
}
}
// sayWaits raises, once, that the seat waits for the states not yet bound, and clears it once, when all are.
func (p *passer) sayWaits(unbound []string) {
c := assign.SeatWaitsCondition(unbound)
state := string(c.State) + c.Summary
if p.say == nil || p.said[c.Key] && p.saidWaits == state {
return
}
if c.State == asks.StateCleared && p.saidWaits == "" {
return // nothing was raised by this process
}
if err := p.say(c); err != nil {
p.logf("that the seat waits for its states could not be said, and is said at the next pass: %v", err)
return
}
p.said[c.Key], p.saidWaits = true, state
}
// seatPlacements is every node's placement and the revision of its node key (its legacy key's for a node not yet
// converted), as the watch holds them: the revision a proposal is laid over (§2, research 072 row 37).
func (p *passer) seatPlacements() (map[string]delivery.Placement, map[string]uint64) {
return p.w.placement.AllRevisions()
}
// candidatesOf is a proposal's candidates by node, with their keys' revisions and the newest write.
func (p *passer) candidatesOf(id string) (map[string]delivery.PlacementCandidate, map[string]uint64, time.Time) {
out, revs := map[string]delivery.PlacementCandidate{}, map[string]uint64{}
var newest time.Time
for key, value := range p.w.read(delivery.PlacementCandidatesBucket) {
pid, node, err := delivery.ParsePlacementCandidateKey(key)
if err != nil || pid != id {
continue
}
c, err := delivery.DecodePlacementCandidate(value)
if err != nil {
continue
}
meta := p.w.metaOf(delivery.PlacementCandidatesBucket, key)
out[node], revs[node] = c, meta.revision
if meta.at.After(newest) {
newest = meta.at
}
}
return out, revs, newest
}
// answersTo is, by node, each node's answer to the proposal in its `<node>.node` have.
func (p *passer) answersTo(id string) map[string]*delivery.CandidateAnswer {
out := map[string]*delivery.CandidateAnswer{}
for key, value := range p.w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
}
h, err := delivery.DecodeHaveNode(value)
if err != nil {
continue
}
if a, ok := h.Candidates[id]; ok {
out[node] = &a
}
}
return out
}
// placementSeat takes every open proposal one step further.
func (p *passer) placementSeat(ctx context.Context) {
for _, b := range seatBuckets {
if p.kvs[b] == nil {
return
}
}
if p.saidProposed == nil {
p.saidProposed, p.intended = map[string]bool{}, map[string]map[string]map[string]bool{}
}
p.convertPlacement(ctx)
current, revs := p.seatPlacements()
ids := make([]string, 0)
raws := p.w.read(delivery.PlacementProposalsBucket)
for id := range raws {
ids = append(ids, id)
}
sort.Strings(ids)
for _, id := range ids {
prop, err := delivery.DecodePlacementProposal(raws[id])
if err != nil || prop.ID != id {
p.logf("the proposal %s cannot be read, so it is left: %v", id, err)
continue
}
p.stepProposal(ctx, prop, p.w.metaOf(delivery.PlacementProposalsBucket, id).revision, current, revs)
}
}
// pulse writes a bound's key with its time to live (§4): its expiry, heard on the watch as the server's delete marker,
// is the event that ends the wait. Written again over one that stands, so its time starts anew.
func (p *passer) pulse(ctx context.Context, key string, ttl time.Duration) {
kv := p.kvs[delivery.PlacementPulsesBucket]
_ = kv.Delete(ctx, key)
rev, err := kv.Create(ctx, key, []byte(`{}`), jetstream.KeyTTL(ttl))
if err != nil {
p.logf("the bound %s is not written, so its wait ends at once: %v", key, err)
return
}
p.w.putEntry(delivery.PlacementPulsesBucket, key, []byte(`{}`), rev, p.clock())
}
// pulsing says whether a bound's key stands.
func (p *passer) pulsing(key string) bool { return p.w.has(delivery.PlacementPulsesBucket, key) }
// deletePulses deletes every bound of a proposal.
func (p *passer) deletePulses(ctx context.Context, id string) {
for key := range p.w.read(delivery.PlacementPulsesBucket) {
if strings.HasSuffix(key, "."+id) || strings.Contains(key, "."+id+".") {
if err := p.kvs[delivery.PlacementPulsesBucket].Delete(ctx, key); err == nil {
p.w.put(delivery.PlacementPulsesBucket, key, nil)
}
}
}
}
// ask raises (or raises again) a proposal's authorising ask and its bound, and answers the proposal asked.
func (p *passer) ask(ctx context.Context, prop delivery.PlacementProposal, akey, brief string) (delivery.PlacementProposal, bool) {
a, err := assign.PlacementAsk(akey, prop, brief, assign.PlacementAsking(prop, p.clock()), p.clock())
if err != nil {
p.logf("the ask about the proposal %s is not written: %v", prop.ID, err)
return prop, false
}
p.put(ctx, delivery.AssignerAsksBucket, akey, func() ([]byte, error) { return delivery.EncodeAssignerAsk(akey, a) })
if pk, err := delivery.PulseAskKey(prop.ID); err == nil {
p.pulse(ctx, pk, a.Ask.Expires.Sub(p.clock()))
}
prop.State, prop.Ask = delivery.ProposalAsked, a.Ask.ID
return prop, true
}
// stepProposal takes one proposal one step further.
func (p *passer) stepProposal(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, revs map[string]uint64) {
if !p.saidProposed[prop.ID] && prop.State == delivery.ProposalAnswering && prop.Verdict == nil {
p.sayEvent(delivery.PlacementProposedEvent, delivery.PlacementEvent{Proposal: prop}, prop.ID)
p.saidProposed[prop.ID] = true
}
if prop.Withdrawn != "" {
p.endProposal(ctx, prop, prop.Withdrawn)
return
}
if prop.Seed || prop.HandBack {
// Adopt and hand-back are gone since the lease write (ADR 0547 §11, ADR 0567 §11): a proposal one left open ends.
p.endProposal(ctx, prop, "adopt and hand-back are gone since the lease write: placement is the mesh's alone")
return
}
if prop.Warrant != nil {
p.addNode(ctx, prop, revs)
return
}
akey, _ := delivery.PlacementAskKey(prop.ID)
again := !prop.Again.IsZero() && (prop.Verdict == nil || prop.Again.After(prop.Verdict.At))
if prop.State == delivery.ProposalAsked {
askPulse, _ := delivery.PulseAskKey(prop.ID)
switch {
case p.answeredWith(akey, assign.OptionYes):
p.warranted(ctx, prop, rev, akey, current, revs)
return
case p.answeredWith(akey, assign.OptionNo):
p.endProposal(ctx, prop, "declined by the operator")
return
case !p.pulsing(askPulse) && !again:
p.endProposal(ctx, prop, "its ask expired unanswered")
return
}
}
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key it was laid over moved since")
return
}
cands, candRevs, written := p.candidatesOf(prop.ID)
if len(prop.Must) == 0 {
p.mustAnswer(ctx, prop, rev, current, cands)
return
}
for _, node := range prop.Must {
if _, ok := cands[node]; !ok {
return // a candidate just written is not heard yet: its watch wakes the next pass
}
}
if written.After(prop.Written) {
prop.Written = written
}
pulsing := map[string]bool{}
for _, node := range prop.Must {
if k, err := delivery.PulseAnswerKey(prop.ID, node); err == nil {
pulsing[node] = p.pulsing(k)
}
}
v := judge.JudgeAnswers(prop, p.intendedOf(prop, current, cands), p.answersTo(prop.ID), candRevs, p.w.online(), pulsing, p.clock())
if prop.Verdict != nil && sameVerdict(*prop.Verdict, v) && !again {
return
}
prop.Verdict = &v
switch v.State {
case delivery.VerdictFail:
prop.State, prop.Ask = delivery.ProposalFailed, ""
case delivery.VerdictWaits:
prop.State, prop.Ask = delivery.ProposalAnswering, ""
case delivery.VerdictPass:
if prop.State != delivery.ProposalAsked || again {
var ok bool
if prop, ok = p.ask(ctx, prop, akey, judge.Brief(prop, v)); !ok {
return
}
if again {
prop.Words = "judged and asked again (again)"
}
}
}
if p.writeProposal(ctx, prop, rev) {
p.sayEvent(delivery.PlacementJudgedEvent, delivery.PlacementEvent{Proposal: prop},
fmt.Sprintf("%s.judged.%d", prop.ID, v.At.UnixNano()))
}
}
// sameVerdict says two verdicts say the same, their time aside.
func sameVerdict(a, b delivery.PlacementVerdict) bool {
a.At, b.At = time.Time{}, time.Time{}
return reflect.DeepEqual(a, b)
}
// moved says whether a key the proposal changes is no longer at the revision it was laid over.
func moved(prop delivery.PlacementProposal, revs map[string]uint64) bool {
for node, base := range prop.Base {
if revs[node] != base {
return true
}
}
return false
}
// intendedOf is, by proposal and the revisions it was laid over, what Intended computed, kept while this process runs.
func (p *passer) intendedOf(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) map[string]map[string]bool {
key := fmt.Sprint(prop.ID, prop.Base)
if got, ok := p.intended[key]; ok {
return got
}
got, _ := p.compute(prop, current, cands)
p.intended[key] = got
return got
}
// compute is what the proposal changes on each node by its placement, and the nodes that answer it: every node with a
// placement or a record (judge.Intended, ADR 0593 §8). The holder composes nothing: each node says what its compositions
// change in its answer.
func (p *passer) compute(prop delivery.PlacementProposal, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) (map[string]map[string]bool, []string) {
laid := map[string]*delivery.Placement{}
for _, node := range prop.Nodes() {
if c, ok := cands[node]; ok && !c.Reached {
laid[node] = c.Placement
}
}
nodes := map[string]bool{}
for n := range current {
nodes[n] = true
}
for n := range p.w.records() {
nodes[n] = true
}
return judge.Intended(current, laid, slices.Sorted(maps.Keys(nodes)))
}
// mustAnswer finds the nodes the proposal must hear from (§3), writes a reached candidate for each whose key it does not
// change, and writes them into the proposal.
func (p *passer) mustAnswer(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, cands map[string]delivery.PlacementCandidate) {
for _, node := range prop.Nodes() {
if _, ok := cands[node]; !ok {
return // the verbs write the candidates before the proposal: wait to hear them
}
}
intended, must := p.compute(prop, current, cands)
p.intended[fmt.Sprint(prop.ID, prop.Base)] = intended
for _, node := range must {
if _, ok := cands[node]; ok {
continue
}
key, err := delivery.PlacementCandidateKey(prop.ID, node)
if err != nil {
continue
}
c := delivery.PlacementCandidate{Proposal: prop.ID, Reached: true}
if pl, ok := current[node]; ok {
c.Placement = &pl
}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
for _, node := range must {
if k, err := delivery.PulseAnswerKey(prop.ID, node); err == nil {
p.pulse(ctx, k, judge.AnswerBound)
}
}
prop.Must = must
p.writeProposal(ctx, prop, rev)
}
// layAgain lays a proposal over the keys as they stand now (§5: a key moved): its candidates are written again, the
// nodes that must answer found again, and it is judged and asked again; a proposal that no longer lays ends.
func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal, rev uint64, current map[string]delivery.Placement, revs map[string]uint64, why string) {
held, recorded := p.lays(ctx)
again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes,
Current: current, Revisions: revs, Held: held, Recorded: recorded})
if len(findings) > 0 {
p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; "))
return
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
for _, node := range again.Nodes() {
key, _ := delivery.PlacementCandidateKey(prop.ID, node)
c := delivery.PlacementCandidate{Proposal: prop.ID, Placement: laid[node]}
p.put(ctx, delivery.PlacementCandidatesBucket, key, func() ([]byte, error) { return delivery.EncodePlacementCandidate(c) })
}
again.Written, again.Words = p.clock().UTC(), why+": laid again over the keys as they stand, to be judged and asked again"
p.writeProposal(ctx, again, rev)
}
// lays is what a proposal is checked against, as the verbs check it: the modules the indexer holds, by key, and the
// nodes with a record.
func (p *passer) lays(ctx context.Context) (func(string) bool, func(string) bool) {
held := map[string]bool{}
for key := range p.w.read(delivery.ModulesBucket) {
if name, err := delivery.ParseModulesKey(key); err == nil {
held[name] = true
}
}
records := p.w.records()
return func(m string) bool { return held[m] }, func(n string) bool { _, ok := records[n]; return ok }
}
// warranted writes an approved proposal (§5): each changed key by compare-and-set at the revision it was laid over; then the warrant recorded, the candidates and the
// proposal deleted, and placement-changed said.
func (p *passer) warranted(ctx context.Context, prop delivery.PlacementProposal, rev uint64, akey string, current map[string]delivery.Placement, revs map[string]uint64) {
if err := p.performs(akey, assign.PlacementAct(prop)); err != nil {
p.endProposal(ctx, prop, "its warrant does not bind these changes, so nothing is written: "+err.Error())
return
}
answer, _ := p.w.answer(akey)
if moved(prop, revs) {
p.layAgain(ctx, prop, rev, current, revs, "a key moved since it was approved, so its warrant is spent")
return
}
cands, _, _ := p.candidatesOf(prop.ID)
laid := map[string]*delivery.Placement{}
for _, node := range prop.Nodes() {
laid[node] = cands[node].Placement
}
writes, err := assign.Writes(prop, laid)
if err != nil {
p.logf("the proposal %s cannot be written: %v", prop.ID, err)
return
}
var changes []delivery.PlacementChange
writtenLaid := map[string]*delivery.Placement{}
after := maps.Clone(current)
afterRevs := maps.Clone(revs)
for _, w := range writes {
rev2, err := p.writeNode(ctx, w.Node, w.Value)
if err != nil {
// Another writer came first (a hand-back): what is written stands and is followed through; the rest is laid
// again over the keys as this write left them.
p.logf("placement %s is not written at revision %d (%v): the rest of the proposal %s is laid again", w.Key, w.Base, err, prop.ID)
p.followWrite(prop, answer, changes)
if rest, ok := assign.Rest(prop, writtenLaid); ok {
p.layAgain(ctx, rest, rev, after, afterRevs, "a key moved while it was written, after "+strings.Join(sortedNodes(writtenLaid), ", ")+" were")
} else {
p.endProposal(ctx, prop, "a key moved while it was written: "+err.Error())
}
return
}
before := current[w.Node]
now := delivery.Placement{Node: w.Node}
if w.Value != nil {
now = *w.Value
after[w.Node] = *w.Value
} else {
delete(after, w.Node)
}
afterRevs[w.Node] = rev2
c := delivery.ChangeOf(before, now)
c.Before, c.After = w.Base, rev2
changes = append(changes, c)
writtenLaid[w.Node] = w.Value
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.followWrite(prop, answer, changes)
}
// addNode writes a joining node on the warrant of the enrolment that issued its token (ADR 0587 §7): the operator's Approve
// was the issue's, so no node is waited on and no ask is raised. The key is created at the revision it was laid over (none: a
// node with a key is no joining node); the enrolment recorded the warrant in the hand-act log when it was given, so this does
// not record it again; placement-changed carries the warrant. A key written meanwhile ends the proposal, saying so.
func (p *passer) addNode(ctx context.Context, prop delivery.PlacementProposal, revs map[string]uint64) {
current, _ := p.seatPlacements()
if moved(prop, revs) {
p.endProposal(ctx, prop, "a key it was laid over was written since: the enrolment proposes the node again")
return
}
laid, findings := delivery.LayChanges(current, prop.Changes)
findings = append(findings, assign.ControllerRefusals(prop.Changes, laid)...)
node := prop.Changes[0].Node
if prop.Changes[0].Op == delivery.OpRemoveNode {
p.removeNode(ctx, prop, current, findings)
return
}
value, ok := laid[node]
if len(findings) > 0 || !ok {
p.endProposal(ctx, prop, "the node cannot be added: "+strings.Join(findings, "; "))
return
}
rev, err := p.writeNode(ctx, node, &value)
if err != nil {
p.endProposal(ctx, prop, "the node's key was not written: "+err.Error())
return
}
c := delivery.ChangeOf(delivery.Placement{Node: node}, value)
c.Before, c.After = prop.Base[node], rev
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
w := prop.Warrant
p.sayChanged(prop, delivery.Answer{Option: assign.OptionYes, Warrant: w.Warrant,
Words: fmt.Sprintf("the operator approved the machine %s at %s through the enrolment's ask %s", w.Node, w.Address, w.Ask)},
[]delivery.PlacementChange{c})
}
// removeNode deletes the key of a joining node whose enrolment ended unconfirmed, on that enrolment's warrant (the operator's
// decision of 2026-11-02 01:05): no node answers and no ask is raised, as for its add-node; placement-changed carries the warrant.
// A node that runs modules or holds seats is no joining node and stays (LayChanges' finding ends the proposal).
func (p *passer) removeNode(ctx context.Context, prop delivery.PlacementProposal, current map[string]delivery.Placement, findings []string) {
node := prop.Changes[0].Node
before, ok := current[node]
if len(findings) > 0 || !ok {
p.endProposal(ctx, prop, "the node cannot be taken off: "+strings.Join(findings, "; "))
return
}
rev, err := p.writeNode(ctx, node, nil)
if err != nil {
p.endProposal(ctx, prop, "the node's key was not deleted: "+err.Error())
return
}
c := delivery.ChangeOf(before, delivery.Placement{Node: node})
c.Before, c.After = prop.Base[node], rev
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
_ = p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID)
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
w := prop.Warrant
p.sayChanged(prop, delivery.Answer{Option: assign.OptionYes, Warrant: w.Warrant,
Words: fmt.Sprintf("the machine %s left placement: its enrolment (ask %s) ended before it confirmed", w.Node, w.Ask)},
[]delivery.PlacementChange{c})
}
// followWrite follows the keys a warrant wrote: the warrant recorded and placement-changed said. No verb of the
// controller is called: the placement is the one source of where modules run (ADR 0567 §8, issue 774).
func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange) {
if len(changes) == 0 {
return
}
p.seat.record(answer.Warrant)
p.sayChanged(prop, answer, changes)
}
func sortedNodes(m map[string]*delivery.Placement) []string {
out := make([]string, 0, len(m))
for n := range m {
out = append(out, n)
}
sort.Strings(out)
return out
}
// sayChanged says placement-changed for the keys written.
func (p *passer) sayChanged(prop delivery.PlacementProposal, answer delivery.Answer, changes []delivery.PlacementChange) {
if len(changes) == 0 {
return
}
at := p.clock().UTC()
if _, when, ok := strings.Cut(answer.Warrant, "@"); ok {
if t, err := time.Parse(time.RFC3339Nano, when); err == nil {
at = t
}
}
delivery.SortChanges(changes)
p.sayEvent(delivery.PlacementChangedEvent, delivery.PlacementEvent{Proposal: prop, Nodes: changes,
Warrant: &delivery.PlacementWarrant{Warrant: answer.Warrant, Words: answer.Words, At: at}}, prop.ID+".changed")
}
// endProposal ends a proposal: its ask, its candidates and its key deleted, and placement-withdrawn said carrying it
// whole with why it ended.
func (p *passer) endProposal(ctx context.Context, prop delivery.PlacementProposal, why string) {
if akey, err := delivery.PlacementAskKey(prop.ID); err == nil && p.w.has(delivery.AssignerAsksBucket, akey) {
if err := p.kvs[delivery.AssignerAsksBucket].Delete(ctx, akey); err == nil {
p.w.put(delivery.AssignerAsksBucket, akey, nil)
}
}
p.deleteCandidates(ctx, prop.ID)
p.deletePulses(ctx, prop.ID)
if err := p.kvs[delivery.PlacementProposalsBucket].Delete(ctx, prop.ID); err != nil && !errors.Is(err, context.Canceled) {
p.logf("the proposal %s is not deleted, and is ended at the next pass: %v", prop.ID, err)
return
}
p.w.put(delivery.PlacementProposalsBucket, prop.ID, nil)
p.logf("the proposal %s ends: %s", prop.ID, why)
p.sayEvent(delivery.PlacementWithdrawnEvent, delivery.PlacementEvent{Proposal: prop, Ended: why}, prop.ID+".withdrawn")
}
// deleteCandidates deletes every candidate of a proposal.
func (p *passer) deleteCandidates(ctx context.Context, id string) {
for key := range p.w.read(delivery.PlacementCandidatesBucket) {
if pid, _, err := delivery.ParsePlacementCandidateKey(key); err == nil && pid == id {
if err := p.kvs[delivery.PlacementCandidatesBucket].Delete(ctx, key); err == nil {
p.w.put(delivery.PlacementCandidatesBucket, key, nil)
}
}
}
}
// writeProposal writes a proposal over the revision the pass read, by compare-and-set: a verb that marked it meanwhile
// (withdraw, again) wins, and the next pass reads that. It says whether it wrote.
func (p *passer) writeProposal(ctx context.Context, prop delivery.PlacementProposal, rev uint64) bool {
raw, err := delivery.EncodePlacementProposal(prop)
if err != nil {
p.logf("the proposal %s is not written: %v", prop.ID, err)
return false
}
kv := p.kvs[delivery.PlacementProposalsBucket]
var at uint64
if rev == 0 {
at, err = kv.Create(ctx, prop.ID, raw)
} else {
at, err = kv.Update(ctx, prop.ID, raw, rev)
}
if err != nil {
if !errors.Is(err, context.Canceled) {
p.logf("the proposal %s is not written, and is taken further at the next pass: %v", prop.ID, err)
}
return false
}
p.w.putEntry(delivery.PlacementProposalsBucket, prop.ID, raw, at, p.clock())
return true
}
// say says one of the seat's events, said in the journal when the bus refuses it.
func (p *passer) sayEvent(event string, body delivery.PlacementEvent, id string) {
if p.publish == nil {
return
}
if err := p.publish(event, body, id); err != nil {
p.logf("%s of %s is not said: %v", event, body.Proposal.ID, err)
}
}
// writeNode writes one node's placement after (nil: none) as its keys (mesh-sdk delivery.PlacementKeys.Writes; research
// 072 row 37): the module keys that change, each carrying the value it replaces, then the node key, the commit, then the
// module keys no longer placed. It answers the node key's revision after (0: deleted). A write that fails up to the
// commit ends it with nothing changed for any reader, the module keys written before it read at their previous values;
// a delete after the commit that fails leaves a module key the node key does not name, which places nothing and is
// deleted at the node's next write. Each key written is taken into the world at once, so the next write of the pass
// reads it.
func (p *passer) writeNode(ctx context.Context, node string, after *delivery.Placement) (uint64, error) {
writes, err := p.w.placement.Writes(node, after)
if err != nil {
return 0, err
}
kv := p.kvs[delivery.PlacementBucket]
committed, rev := false, uint64(0)
for _, w := range writes {
got, err := writeKey(ctx, kv, w)
if err != nil {
if committed {
p.logf("placement %s is not deleted after %s's write (%v): it places nothing, and goes at the node's next write", w.Key, node, err)
continue
}
return 0, fmt.Errorf("placement %s at revision %d: %w", w.Key, w.Base, err)
}
p.w.putEntry(delivery.PlacementBucket, w.Key, w.Value, got, p.clock())
if w.Commit {
committed, rev = true, got
}
}
return rev, nil
}
// writeKey makes one write of a placement key by compare-and-set and answers the key's revision after (0: deleted).
func writeKey(ctx context.Context, kv jetstream.KeyValue, w delivery.PlacementKeyWrite) (uint64, error) {
switch {
case w.Value == nil:
return 0, kv.Delete(ctx, w.Key, jetstream.LastRevision(w.Base))
case w.Base == 0:
return kv.Create(ctx, w.Key, w.Value)
default:
return kv.Update(ctx, w.Key, w.Value, w.Base)
}
}
// convertPlacement converts every node the state holds only in the form before, `placement.<node>`, into its node key
// and module keys (research 072 row 37), once: each node converted is said, and a node whose conversion fails is said
// and tried again at the next pass. The legacy key is left for the readers not yet delivered; a node key wins over it
// for every reader that reads both, and it is deleted with the node key when the node's placement is.
func (p *passer) convertPlacement(ctx context.Context) {
if p.convertedAll {
return
}
migrations, err := p.w.placement.Migrations()
if err != nil {
p.logf("placement: some nodes in the form before cannot be converted, and stay read from it: %v", err)
}
failed := false
for _, node := range slices.Sorted(maps.Keys(migrations)) {
before, rev, _ := p.w.placement.Of(node)
got, err := p.writeNode(ctx, node, &before)
if err != nil {
failed = true
p.logf("placement: %s is not converted from placement.%s (revision %d) to its node and module keys, and is tried again at the next pass: %v", node, node, rev, err)
continue
}
p.logf("placement: %s converted from placement.%s (revision %d) to %s.node (revision %d) and %d module keys (research 072 row 37); placement.%s stays for readers not yet delivered",
node, node, rev, node, got, len(before.Modules), node)
}
p.convertedAll = !failed && err == nil
}
@@ -0,0 +1,620 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-placement's seven verbs (novox/hq ADR 0567 §10), served through the tool runner beside the put tools: a
// process never serves (ADR 0436). They read every state the seat keeps and write what a person asks of it: a proposal
// and its candidates (propose, adopt), a proposal marked to end (withdraw) or to be judged again (again), and the keys
// deleted (hand-back). Everything that follows a proposal — composing it, judging the answers, the ask, the write at the
// warrant and the seat's events on its stream — is the assigner's process's (seat.go), started by these writes.
// seatStore is what the seat's verbs need of the states: the runtime's, or a test's.
type seatStore interface {
store
del(state, key string) error
history(state, key string) ([]stdio.StateRevision, error)
}
func (runtimeStore) del(state, key string) error { return stdio.State(state).Delete(key) }
func (runtimeStore) history(state, key string) ([]stdio.StateRevision, error) {
return stdio.State(state).History(key)
}
// The states the verbs read, as the runtime names them: the seat's own by their local names, the others'
// `<module>.<state>`.
const (
modulesState = delivery.IndexerModule + ".modules"
haveState = "node-engine.have"
)
// seatTools serves the seat's verbs over a store, asking the controller through ask (stdio.Ask, or a test's).
type seatTools struct {
st seatStore
ask func(key string, body any) (json.RawMessage, error)
now func() time.Time
}
func newSeatTools(st seatStore, ask func(string, any) (json.RawMessage, error)) *seatTools {
return &seatTools{st: st, ask: ask, now: time.Now}
}
// keyed is one placement key as a verb answers it.
type keyed struct {
Placement delivery.Placement `json:"placement"`
Revision uint64 `json:"revision"`
}
// placements reads every node's placement and the revision of its node key (its legacy key's before it is converted),
// assembled from its keys in either form (delivery.PlacementKeys; research 072 row 37).
func (s *seatTools) placements() (map[string]delivery.Placement, map[string]uint64, error) {
keys, err := s.st.keys(delivery.PlacementState)
if err != nil {
return nil, nil, fmt.Errorf("placement cannot be listed: %w", err)
}
sort.Strings(keys)
held := delivery.NewPlacementKeys()
var bad []string
for _, key := range keys {
if _, _, _, err := delivery.ParsePlacementKey(key); err != nil {
continue
}
raw, rev, found, err := s.st.get(delivery.PlacementState, key)
if err != nil {
return nil, nil, fmt.Errorf("placement key %s cannot be read: %w", key, err)
}
if !found {
continue
}
if _, err := held.Put(key, raw, rev, false); err != nil {
bad = append(bad, err.Error())
}
}
out, revs := held.AllRevisions()
if len(bad) > 0 && len(out) == 0 {
return nil, nil, fmt.Errorf("placement cannot be read: %s", strings.Join(bad, "; "))
}
return out, revs, nil
}
// lays is what a proposal is checked against: the modules the indexer holds and the nodes with a record.
func (s *seatTools) lays() (func(string) bool, func(string) bool, error) {
mods, err := s.st.keys(modulesState)
if err != nil {
return nil, nil, fmt.Errorf("the indexer's modules cannot be listed: %w", err)
}
held := map[string]bool{}
for _, k := range mods {
if name, err := delivery.ParseModulesKey(k); err == nil {
held[name] = true
}
}
haves, err := s.st.keys(haveState)
if err != nil {
return nil, nil, fmt.Errorf("the nodes' records cannot be listed: %w", err)
}
recorded := map[string]bool{}
for _, k := range haves {
node, _, isNode, err := delivery.ParseHaveKey(k)
if err != nil || !isNode {
continue
}
if raw, _, found, err := s.st.get(haveState, k); err == nil && found {
if h, err := delivery.DecodeHaveNode(raw); err == nil && h.Record != nil {
recorded[node] = true
}
}
}
return func(m string) bool { return held[m] }, func(n string) bool { return recorded[n] }, nil
}
// proposal reads one open proposal.
func (s *seatTools) proposal(id string) (delivery.PlacementProposal, error) {
if err := delivery.CheckProposalID(id); err != nil {
return delivery.PlacementProposal{}, err
}
raw, _, found, err := s.st.get(delivery.PlacementProposalsState, id)
if err != nil {
return delivery.PlacementProposal{}, fmt.Errorf("the proposal %s cannot be read: %w", id, err)
}
if !found {
return delivery.PlacementProposal{}, fmt.Errorf("no proposal %s is open: mesh-placement.proposals lists those that are", id)
}
return delivery.DecodePlacementProposal(raw)
}
// write writes a proposal laid by assign.Propose: each candidate, then the proposal, whose write the assigner's process
// takes as the event to compose and judge it.
func (s *seatTools) write(p delivery.PlacementProposal, laid map[string]*delivery.Placement) error {
p.Written = s.now().UTC()
for _, node := range p.Nodes() {
key, err := delivery.PlacementCandidateKey(p.ID, node)
if err != nil {
return err
}
raw, err := delivery.EncodePlacementCandidate(delivery.PlacementCandidate{Proposal: p.ID, Placement: laid[node]})
if err != nil {
return err
}
if err := s.st.put(delivery.PlacementCandidatesState, key, raw); err != nil {
return fmt.Errorf("the candidate %s is not written: %w", key, err)
}
}
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return fmt.Errorf("the proposal %s is not written: %w", p.ID, err)
}
return nil
}
// proposed is what propose and adopt answer.
type proposed struct {
Proposal delivery.PlacementProposal `json:"proposal"`
Follow string `json:"follow"`
}
func followProposal(id string) string {
return fmt.Sprintf("the nodes it touches answer it; mesh-placement.proposals with id %q says where it stands, and the "+
"operator is asked to approve it once its verdict passes", id)
}
// changesOf reads propose's changes: a JSON array, given as one or as its text.
func changesOf(v any) ([]delivery.PlacementProposalChange, error) {
var raw []byte
switch c := v.(type) {
case string:
raw = []byte(c)
case nil:
return nil, errors.New("propose names its changes")
default:
var err error
if raw, err = json.Marshal(c); err != nil {
return nil, err
}
}
var out []delivery.PlacementProposalChange
dec := json.NewDecoder(strings.NewReader(string(raw)))
dec.DisallowUnknownFields()
if err := dec.Decode(&out); err != nil {
return nil, fmt.Errorf("the changes are no list of changes: %w", err)
}
return out, nil
}
// doPropose is the verb propose (§2).
func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
changes, err := changesOf(args["changes"])
if err != nil {
return nil, err
}
for _, c := range changes {
if c.Op == delivery.OpSeed {
return nil, errors.New("the seed is gone with adopt: propose takes place, take-off, pin, unpin, hold, unhold, decide and config")
}
}
current, revs, err := s.placements()
if err != nil {
return nil, err
}
held, recorded, err := s.lays()
if err != nil {
return nil, err
}
warrant, err := warrantOf(args["warrant"])
if err != nil {
return nil, err
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Warrant: warrant})
if refusal := delivery.AddNodeRefusal(p, moduleOfCaller(by)); refusal != "" {
findings = append(findings, refusal)
}
if len(findings) > 0 {
return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; "))
}
if err := s.write(p, laid); err != nil {
return nil, err
}
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// warrantOf reads propose's warrant, an enrolment's for an add-node: an object, given as one or as its text; none is none.
func warrantOf(v any) (*delivery.AddNodeWarrant, error) {
var raw []byte
switch w := v.(type) {
case nil:
return nil, nil
case string:
if strings.TrimSpace(w) == "" {
return nil, nil
}
raw = []byte(w)
default:
var err error
if raw, err = json.Marshal(w); err != nil {
return nil, err
}
}
var out delivery.AddNodeWarrant
dec := json.NewDecoder(strings.NewReader(string(raw)))
dec.DisallowUnknownFields()
if err := dec.Decode(&out); err != nil {
return nil, fmt.Errorf("the warrant is no enrolment's warrant {node, address, ask, warrant}: %w", err)
}
return &out, nil
}
// moduleOfCaller is the module a caller is, as the runtime proves it, for AddNodeRefusal: mesh-enrolment when the proven
// caller is that module's (its bus user `<node>.mesh-enrolment`, never a person's `person.mesh-enrolment`), else the caller
// as proven, which the refusal names.
func moduleOfCaller(by string) string {
if (stdio.Callers{delivery.EnrolmentModule}).Allows(by) {
return delivery.EnrolmentModule
}
return by
}
// doWithdraw is the verb withdraw (§2): the proposal is marked, and the assigner deletes it and its candidates and says
// placement-withdrawn carrying it whole.
func (s *seatTools) doWithdraw(args map[string]any, by string) (any, error) {
why := arg(args, "why")
if why == "" {
return nil, errors.New("withdraw says why: it is kept in the event that ends the proposal")
}
p, err := s.proposal(arg(args, "id"))
if err != nil {
return nil, err
}
p.Withdrawn = fmt.Sprintf("withdrawn by %s: %s", by, why)
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the proposal %s is not withdrawn: %w", p.ID, err)
}
return map[string]any{"proposal": p.ID, "withdrawn": p.Withdrawn}, nil
}
// doAgain is the verb again (§10): the proposal is marked, and the assigner judges it now and raises its ask again
// when its verdict passes.
func (s *seatTools) doAgain(args map[string]any) (any, error) {
p, err := s.proposal(arg(args, "id"))
if err != nil {
return nil, err
}
p.Again = s.now().UTC()
raw, err := delivery.EncodePlacementProposal(p)
if err != nil {
return nil, err
}
if err := s.st.put(delivery.PlacementProposalsState, p.ID, raw); err != nil {
return nil, fmt.Errorf("the proposal %s is not asked again: %w", p.ID, err)
}
return map[string]any{"proposal": p.ID, "again": p.Again, "follow": followProposal(p.ID)}, nil
}
// placementAnswer is what the verb placement answers.
type placementAnswer struct {
Keys map[string]keyed `json:"keys,omitempty"`
Nodes []string `json:"nodes,omitempty"`
Count int `json:"count"`
History int `json:"history-per-key"`
Changes []historyEntry `json:"changes,omitempty"`
Older string `json:"older,omitempty"`
}
// historyEntry is one change of a key in the state's revisions.
type historyEntry struct {
Node string `json:"node"`
Revision uint64 `json:"revision"`
At time.Time `json:"at"`
Change delivery.PlacementChange `json:"change"`
Deleted bool `json:"deleted,omitempty"`
}
// doPlacement is the verb placement (§10, §6): one node's key, one module's nodes, or every key; with history, the
// changes in order from the revisions the state keeps.
func (s *seatTools) doPlacement(args map[string]any) (any, error) {
if arg(args, "seats") == "true" {
return s.seats(arg(args, "node"))
}
all, revs, err := s.placements()
if err != nil {
return nil, err
}
node, module := arg(args, "node"), arg(args, "module")
out := placementAnswer{Keys: map[string]keyed{}, Count: len(all), History: delivery.PlacementHistory}
for n, p := range all {
if node != "" && n != node {
continue
}
if module != "" && !strings.Contains(" "+strings.Join(p.Assigned(), " ")+" ", " "+module+" ") {
continue
}
out.Keys[n] = keyed{Placement: p, Revision: revs[n]}
out.Nodes = append(out.Nodes, n)
}
sort.Strings(out.Nodes)
if node != "" && len(out.Keys) == 0 && arg(args, "history") != "true" {
return nil, fmt.Errorf("%s has no placement key", node)
}
if arg(args, "history") != "true" {
return out, nil
}
nodes := out.Nodes
if node != "" {
nodes = []string{node}
} else if module == "" {
nodes = nil
for n := range all {
nodes = append(nodes, n)
}
sort.Strings(nodes)
}
for _, n := range nodes {
revisions, err := s.nodeHistory(n)
if err != nil {
return nil, err
}
var before sketch
for _, r := range revisions {
e := historyEntry{Node: n, Revision: r.Revision, At: r.At}
after := sketch{}
if r.Op == "" || r.Op == "put" || r.Op == "PUT" {
after = sketchOf(r.Key, r.Value)
} else {
e.Deleted = true
}
e.Change = after.changeFrom(n, before)
e.Change.Before, e.Change.After = 0, r.Revision
before = after
if module != "" && !strings.Contains(" "+strings.Join(append(append(e.Change.Added, e.Change.Removed...), e.Change.Changed...), " ")+" ", " "+module+" ") {
continue
}
out.Changes = append(out.Changes, e)
}
}
out.Older = fmt.Sprintf("the state keeps %d revisions of each key; an older change is in its placement-changed event on "+
"the assigner's stream %s, kept %d days", delivery.PlacementHistory, delivery.PlacementStream, delivery.PlacementStreamDays)
return out, nil
}
// openProposal is one open proposal as the verb proposals answers it: the proposal (its changes, base revisions,
// verdict with the differences and the answers it judged, its ask), its candidates by node with their revisions, and
// every node's answer now.
type openProposal struct {
Proposal delivery.PlacementProposal `json:"proposal"`
Candidates map[string]uint64 `json:"candidates"`
Answers map[string]delivery.CandidateAnswer `json:"answers,omitempty"`
}
// doProposals is the verb proposals (§10).
func (s *seatTools) doProposals(args map[string]any) (any, error) {
ids, err := s.st.keys(delivery.PlacementProposalsState)
if err != nil {
return nil, fmt.Errorf("the proposals cannot be listed: %w", err)
}
cands, err := s.st.keys(delivery.PlacementCandidatesState)
if err != nil {
return nil, fmt.Errorf("the candidates cannot be listed: %w", err)
}
haves, err := s.st.keys(haveState)
if err != nil {
return nil, fmt.Errorf("the nodes' haves cannot be listed: %w", err)
}
answers := map[string]map[string]delivery.CandidateAnswer{} // by proposal, then node
for _, k := range haves {
node, _, isNode, err := delivery.ParseHaveKey(k)
if err != nil || !isNode {
continue
}
raw, _, found, err := s.st.get(haveState, k)
if err != nil || !found {
continue
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil {
continue
}
for id, a := range h.Candidates {
if answers[id] == nil {
answers[id] = map[string]delivery.CandidateAnswer{}
}
answers[id][node] = a
}
}
sort.Strings(ids)
out := []openProposal{}
for _, id := range ids {
if want := arg(args, "id"); want != "" && want != id {
continue
}
p, err := s.proposal(id)
if err != nil {
return nil, err
}
o := openProposal{Proposal: p, Candidates: map[string]uint64{}, Answers: answers[id]}
for _, k := range cands {
pid, node, err := delivery.ParsePlacementCandidateKey(k)
if err != nil || pid != id {
continue
}
if _, rev, found, err := s.st.get(delivery.PlacementCandidatesState, k); err == nil && found {
o.Candidates[node] = rev
}
}
out = append(out, o)
}
if arg(args, "id") != "" && len(out) == 0 {
return nil, fmt.Errorf("no proposal %s is open", arg(args, "id"))
}
return map[string]any{"proposals": out}, nil
}
// tools are the seat's verbs as the runtime serves them, each named <seat>.<verb> (the tool runner serves a seat's verb
// only from a tool of that name), with the seat's own schemas.
func (s *seatTools) tools() []stdio.Tool {
seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat)
input := map[string]map[string]any{}
description := map[string]string{}
for _, v := range seat.Serves {
input[v.Name], description[v.Name] = v.Input, v.Description
}
run := map[string]func(map[string]any) (any, error){
catalogue.PlacementVerbPlacement: s.doPlacement,
catalogue.PlacementVerbProposals: s.doProposals,
catalogue.PlacementVerbPropose: func(a map[string]any) (any, error) { return s.doPropose(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbWithdraw: func(a map[string]any) (any, error) { return s.doWithdraw(a, stdio.CallerOf(a)) },
catalogue.PlacementVerbAgain: s.doAgain,
}
var out []stdio.Tool
for _, v := range seat.Serves {
out = append(out, stdio.Tool{Name: catalogue.PlacementSeat + "." + v.Name, Description: description[v.Name], Input: input[v.Name], Run: run[v.Name]})
}
return out
}
// seats is the verb placement with seats (ADR 0571 §4): who holds each seat and on which node, as the state seat-holders
// holds it; with node, the keys of seats held on that node.
func (s *seatTools) seats(node string) (any, error) {
keys, err := s.st.keys(delivery.SeatHoldersState)
if err != nil {
return nil, fmt.Errorf("seat-holders cannot be listed: %w", err)
}
sort.Strings(keys)
out := []delivery.SeatHolders{}
for _, key := range keys {
raw, _, found, err := s.st.get(delivery.SeatHoldersState, key)
if err != nil {
return nil, fmt.Errorf("seat-holders %s cannot be read: %w", key, err)
}
if !found {
continue
}
var sh delivery.SeatHolders
if json.Unmarshal(raw, &sh) != nil {
continue
}
if node != "" && sh.Node != node && !slices.ContainsFunc(sh.Holders, func(h delivery.SeatHolder) bool { return h.Node == node }) {
continue
}
out = append(out, sh)
}
return map[string]any{"seats": out, "holds": len(keys)}, nil
}
// keyedRevision is one revision of a node's placement key, the key it is of beside it.
type keyedRevision struct {
stdio.StateRevision
Key string
}
// nodeHistory is the revisions of a node's placement, in the order the bus wrote them: its legacy key's
// `placement.<node>` before it was converted, then its node key's `<node>.node`, whose every revision is one write of
// the node (research 072 row 37: the node key is a write's commit, so its history is the node's placement history).
func (s *seatTools) nodeHistory(node string) ([]keyedRevision, error) {
legacy, err := delivery.LegacyPlacementKey(node)
if err != nil {
return nil, err
}
nodeKey, err := delivery.PlacementNodeKey(node)
if err != nil {
return nil, err
}
var out []keyedRevision
for _, key := range []string{legacy, nodeKey} {
revisions, err := s.st.history(delivery.PlacementState, key)
if err != nil {
return nil, fmt.Errorf("the history of %s cannot be read: %w", key, err)
}
for _, r := range revisions {
out = append(out, keyedRevision{StateRevision: r, Key: key})
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].Revision < out[j].Revision })
return out, nil
}
// sketch is a revision of a node's placement as far as its history says a change: each module's digest, and its pins
// and holdings, which change every module (delivery.ChangeOf).
type sketch struct {
modules []delivery.PlacementRef
nodeWide string
}
// sketchOf reads one revision of a node key or a legacy key; a value that does not read places nothing.
func sketchOf(key string, value []byte) sketch {
_, _, kind, err := delivery.ParsePlacementKey(key)
if err != nil {
return sketch{}
}
wide := func(pins map[string]delivery.Provider, holds []delivery.Holding) string {
raw, _ := json.Marshal([]any{pins, holds})
return string(raw)
}
if kind == delivery.PlacementNodeKind {
n, err := delivery.DecodePlacementNode(value)
if err != nil {
return sketch{}
}
return sketch{modules: n.Modules, nodeWide: wide(n.Pins, n.Holds)}
}
p, err := delivery.DecodePlacement(value)
if err != nil {
return sketch{}
}
out := sketch{nodeWide: wide(p.Pins, p.Holds)}
for _, m := range p.Modules {
out.modules = append(out.modules, delivery.PlacementRef{Module: m.Module, Digest: m.Digest()})
}
return out
}
// changeFrom is what changed from before to s on node, as delivery.ChangeOf says it.
func (s sketch) changeFrom(node string, before sketch) delivery.PlacementChange {
c := delivery.PlacementChange{Node: node}
was := map[string]string{}
for _, r := range before.modules {
was[r.Module] = r.Digest
}
now := map[string]bool{}
nodeWide := before.nodeWide != s.nodeWide && len(before.modules) > 0
for _, r := range s.modules {
now[r.Module] = true
d, ok := was[r.Module]
switch {
case !ok:
c.Added = append(c.Added, r.Module)
case nodeWide || d != r.Digest:
c.Changed = append(c.Changed, r.Module)
}
}
for _, r := range before.modules {
if !now[r.Module] {
c.Removed = append(c.Removed, r.Module)
}
}
sort.Strings(c.Added)
sort.Strings(c.Removed)
sort.Strings(c.Changed)
return c
}
@@ -0,0 +1,572 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"reflect"
"slices"
"strings"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/provisions"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-assigner/internal/assign"
)
func (s memStore) del(state, key string) error {
delete(s[state].values, key)
return nil
}
func (s memStore) history(state, key string) ([]stdio.StateRevision, error) {
v, ok := s[state].values[key]
if !ok {
return nil, nil
}
return []stdio.StateRevision{{Key: key, Value: v, Revision: uint64(len(v)), Op: "put"}}, nil
}
// ADR 0567 §2, §7, §10: the seat's five verbs over the states. propose lays a proposal and writes it with its candidates,
// or refuses naming every finding and writes nothing; withdraw and again mark it for the holder; placement and
// proposals read every state the seat keeps, and history names the stream for what the state no longer keeps.
func TestTheSeatsVerbs(t *testing.T) {
anchor := delivery.Placement{V: delivery.Version, Node: "anchor", Modules: []delivery.PlacedModule{{Module: "web"}}}
st := memStore{}
for _, state := range []string{delivery.PlacementState, delivery.PlacementProposalsState, delivery.PlacementCandidatesState, modulesState, haveState} {
st[state] = &memKV{values: map[string][]byte{}}
}
for _, m := range []string{"web", "shell"} {
st[modulesState].values[m] = []byte("{}")
}
for _, n := range []string{"anchor", "laptop"} {
raw, _ := delivery.EncodeHaveNode(delivery.HaveNode{Record: &delivery.NodeRecord{Node: n}})
st[haveState].values[n+".node"] = raw
}
var asked []string
s := newSeatTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
return json.RawMessage(`{"ok":true}`), nil
})
s.now = func() time.Time { return time.Date(2026, 10, 28, 20, 0, 0, 0, time.UTC) }
raw, _ := delivery.EncodePlacement(anchor)
st[delivery.PlacementState].values["placement.anchor"] = raw
// withdraw and again mark the proposal; the holder ends or judges it.
got, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op")
if err != nil {
t.Fatal(err)
}
seed := got.(proposed).Proposal
if _, err := s.doWithdraw(map[string]any{"id": seed.ID, "why": "not yet"}, "op"); err != nil {
t.Fatal(err)
}
if _, err := s.doAgain(map[string]any{"id": seed.ID}); err != nil {
t.Fatal(err)
}
if p, err := s.proposal(seed.ID); err != nil || !strings.Contains(p.Withdrawn, "not yet") || p.Again.IsZero() {
t.Errorf("marked = %+v, %v", p, err)
}
if _, err := s.doWithdraw(map[string]any{"id": "p-none", "why": "x"}, "op"); err == nil {
t.Error("an unknown proposal was withdrawn")
}
delete(st[delivery.PlacementProposalsState].values, seed.ID) // the holder ended it
st[delivery.PlacementCandidatesState].values = map[string][]byte{}
// propose: refused whole, naming every finding; a valid move laid over the keys at their revisions.
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"place","node":"anchor","module":"ghost"},{"op":"hold","node":"anchor","module":"web","claim":"x"}]`}, "op"); err == nil ||
!strings.Contains(err.Error(), "ghost") || len(st[delivery.PlacementProposalsState].values) != 0 {
t.Errorf("a refused proposal: %v", err)
}
if _, err := s.doPropose(map[string]any{"why": "x", "changes": `[{"op":"swap","node":"anchor"}]`}, "op"); err == nil {
t.Error("an unknown change was taken")
}
got, err = s.doPropose(map[string]any{"why": "move web", "changes": []any{
map[string]any{"op": "take-off", "node": "anchor", "module": "web"}, map[string]any{"op": "place", "node": "laptop", "module": "web"}}}, "operator@laptop")
if err != nil {
t.Fatal(err)
}
move := got.(proposed).Proposal
if move.By != "operator@laptop" || move.Base["anchor"] != uint64(len(raw)) || move.Base["laptop"] != 0 || len(st[delivery.PlacementCandidatesState].values) != 2 {
t.Fatalf("propose = %+v", move)
}
// placement and proposals read every state the seat keeps (§10); history names the stream for what is older.
reads := map[string]bool{}
pa, err := s.doPlacement(map[string]any{"module": "web", "history": "true"})
if a := pa.(placementAnswer); err != nil || !slices.Equal(a.Nodes, []string{"anchor"}) || len(a.Changes) != 1 || !strings.Contains(a.Older, delivery.PlacementStream) {
t.Errorf("placement of web = %+v, %v", pa, err)
} else {
reads[delivery.PlacementState], reads["stream "+delivery.PlacementStream] = true, true
}
if _, err := s.doPlacement(map[string]any{"node": "laptop"}); err == nil {
t.Error("a node with no key was answered")
}
ps, err := s.doProposals(map[string]any{"id": move.ID})
if list := ps.(map[string]any)["proposals"].([]openProposal); err != nil || len(list) != 1 || len(list[0].Candidates) != 2 {
t.Errorf("proposals = %+v, %v", ps, err)
} else {
reads[delivery.PlacementProposalsState], reads[delivery.PlacementCandidatesState] = true, true
}
raw0, err := readManifest()
if err != nil {
t.Fatal(err)
}
for _, state := range raw0.State {
if slices.Contains([]string{delivery.PlacementState, delivery.PlacementProposalsState, delivery.PlacementCandidatesState}, state.Name) && !reads[state.Name] {
t.Errorf("no verb reads the seat's state %s", state.Name)
}
}
for _, stream := range raw0.Streams {
if !reads["stream "+stream.Name] {
t.Errorf("no verb reads the seat's stream %s", stream.Name)
}
}
// Issue 774 (ADR 0567 §8): every kind of change is proposed, and there is no controller to hand placement back to.
if _, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op"); err != nil {
t.Errorf("a hold: %v", err)
}
// 774/S59/AC1 (ADR 0587 §7): an add-node from mesh-enrolment whose warrant names this node and this address is proposed, with
// the node a joining one with no record; from another caller, naming another node or address, or with no warrant it is refused.
addNode := func(by, warrant string) error {
_, err := s.doPropose(map[string]any{"why": "enrolment of ace", "warrant": warrant, "changes": []any{map[string]any{"op": "add-node", "node": "ace",
"decided": map[string]any{"address": "10.9.0.2"}, "keys": map[string]any{"overlay": "dHVubmVs"}}}}, by)
return err
}
good := `{"node":"ace","address":"10.9.0.2","ask":"enrol-ace-1","warrant":"enrol-ace-1@2026-10-28T20:00:00Z"}`
for _, c := range []struct{ by, warrant, want string }{
{"novox.mesh-grants", good, "mesh-enrolment's alone"},
{"person.mesh-enrolment", good, "mesh-enrolment's alone"},
{"novox.mesh-enrolment", `{"node":"g14","address":"10.9.0.2","ask":"a","warrant":"a@t"}`, "carries the warrant of g14"},
{"novox.mesh-enrolment", `{"node":"ace","address":"10.9.0.9","ask":"a","warrant":"a@t"}`, "carries the warrant of ace at 10.9.0.9"},
{"novox.mesh-enrolment", "", "carries no warrant"},
} {
if err := addNode(c.by, c.warrant); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("an add-node by %s with warrant %q: %v, want a refusal saying %q", c.by, c.warrant, err, c.want)
}
}
if err := addNode("novox.mesh-enrolment", good); err != nil {
t.Errorf("an add-node from mesh-enrolment: %v", err)
}
// A remove-node (the operator's decision of 2026-11-02 01:05) takes a joining node that never confirmed off placement: from
// mesh-enrolment, on the warrant naming that node, of a node running nothing, it is proposed with the node's key to delete;
// of a node running modules, or with another node's warrant, it is refused.
joiner, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "joiner", Decided: delivery.NodeDecided{Address: "10.9.0.7"}, Modules: []delivery.PlacedModule{}})
st[delivery.PlacementState].values["placement.joiner"] = joiner
removeNode := func(node, warrant string) (any, error) {
return s.doPropose(map[string]any{"why": "enrolment ended", "warrant": warrant,
"changes": []any{map[string]any{"op": "remove-node", "node": node}}}, "novox.mesh-enrolment")
}
for _, c := range []struct{ node, warrant, want string }{
{"anchor", `{"node":"anchor","address":"10.9.0.1","ask":"a","warrant":"a@t"}`, "runs modules or holds seats"},
{"joiner", `{"node":"g14","address":"10.9.0.7","ask":"a","warrant":"a@t"}`, "carries the warrant of g14"},
} {
if _, err := removeNode(c.node, c.warrant); err == nil || !strings.Contains(err.Error(), c.want) {
t.Errorf("a remove-node of %s: %v, want a refusal saying %q", c.node, err, c.want)
}
}
if got, err := removeNode("joiner", `{"node":"joiner","address":"10.9.0.7","ask":"a","warrant":"a@t"}`); err != nil {
t.Errorf("a remove-node from mesh-enrolment: %v", err)
} else if rm := got.(proposed).Proposal; !slices.Equal(rm.Nodes(), []string{"joiner"}) || !slices.Equal(rm.RemovedNodes(), []string{"joiner"}) {
t.Errorf("the remove-node proposed %+v", rm)
}
// 1126/B8: the seat serves its five verbs; adopt and hand-back went with the controller (ADR 0590).
if tools := s.tools(); len(tools) != 5 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
t.Run("1953/D4-D6: the seat mesh-wants' upgrade and bus", wantsVerbs)
t.Run("1955/L7 (ADR 0571 §6): rule 7 reads the node's own composition; an unplaced node is said", func(t *testing.T) {
w := newWorld()
fp := delivery.SourcePrefix2 + strings.Repeat("2", 64)
pl, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "ace", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey, _ := delivery.LegacyPlacementKey("ace")
w.putEntry(delivery.PlacementBucket, pkey, pl, 1, time.Now())
for node, digest := range map[string]string{"ace": "sha256:" + strings.Repeat("a", 64), "g14": ""} {
hn, _ := delivery.EncodeHaveNode(delivery.HaveNode{Composed: digest})
hk, _ := delivery.HaveNodeKey(node)
w.putEntry(delivery.HaveBucket, hk, hn, 1, time.Now())
}
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("a", 64) {
t.Errorf("ace: %+v %v", d, ok)
}
if _, ok := w.composed("g14", "web", fp); ok || !slices.Equal(w.unplacedNodes(), []string{"g14"}) {
t.Errorf("g14 unplaced: composed %v, unplaced %v", ok, w.unplacedNodes())
}
findings := map[string][]byte{}
p := &passer{w: w, logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
p.unplacedFindings()
if _, ok := findings[assign.UnplacedKey("g14")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
pl2, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "g14", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey2, _ := delivery.LegacyPlacementKey("g14")
w.putEntry(delivery.PlacementBucket, pkey2, pl2, 1, time.Now())
p.unplacedFindings()
if len(findings) != 0 {
t.Errorf("placed again: findings %v", findings)
}
})
t.Run("1955/L7 (ADR 0571 §4): seat-holders is written from placement, only the keys a change moves, and answered by placement seats", func(t *testing.T) {
writes := 0
kv := &countingKV{memKV: memKV{values: map[string][]byte{}}, writes: &writes}
p := &passer{w: newWorld(), logf: t.Logf, kvs: buckets{delivery.SeatHoldersBucket: kv}}
place := func(node string, holds ...delivery.Holding) {
pl, err := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{{Module: "web"}, {Module: "nats"}}, Holds: holds})
if err != nil {
t.Fatal(err)
}
k, _ := delivery.LegacyPlacementKey(node)
p.w.putEntry(delivery.PlacementBucket, k, pl, 1, time.Now())
}
place("ace", delivery.Holding{Claim: "mesh-broker", Scope: "mesh", Module: "nats"}, delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
place("g14", delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
ctx := context.Background()
p.seatHolders(ctx)
if writes != 3 || kv.values["mesh-broker"] == nil || kv.values["node-shell.ace"] == nil || kv.values["node-shell.g14"] == nil {
t.Fatalf("%d writes: %v", writes, kv.values)
}
p.seatHolders(ctx)
if writes != 3 {
t.Errorf("an unchanged placement wrote %d keys", writes-3)
}
place("g14") // g14 holds nothing now
p.seatHolders(ctx)
if writes != 3 || kv.values["node-shell.g14"] != nil || kv.values["node-shell.ace"] == nil {
t.Errorf("after g14's change: %d writes, %v", writes, kv.values)
}
st := memStore{delivery.SeatHoldersState: &memKV{values: kv.values}}
out, err := newSeatTools(st, nil).seats("ace")
if err != nil || len(out.(map[string]any)["seats"].([]delivery.SeatHolders)) != 2 {
t.Errorf("seats on ace = %+v, %v", out, err)
}
})
t.Run("research 072 row 37: the form before is converted once, a node is written as its keys, and history reads both", func(t *testing.T) {
ctx := context.Background()
kv := &revKV{values: map[string][]byte{}, revs: map[string]uint64{}}
var logs []string
p := &passer{w: newWorld(), kvs: buckets{delivery.PlacementBucket: kv}, now: time.Now,
logf: func(f string, a ...any) { logs = append(logs, fmt.Sprintf(f, a...)) }}
put := func(key string, value []byte, rev uint64) {
p.w.putEntry(delivery.PlacementBucket, key, value, rev, time.Now())
}
was := delivery.Placement{Node: "anchor", Modules: []delivery.PlacedModule{{Module: "web"}}}
raw, _ := delivery.EncodePlacement(was)
put("placement.anchor", raw, kv.write("placement.anchor", raw))
p.convertPlacement(ctx)
if strings.Join(kv.log, " ") != "create anchor.web create anchor.node" || !p.convertedAll ||
!strings.Contains(strings.Join(logs, "\n"), "anchor converted from placement.anchor") {
t.Fatalf("the conversion wrote %v, said %v", kv.log, logs)
}
p.convertPlacement(ctx)
if len(kv.log) != 2 {
t.Errorf("a converted node is converted again: %v", kv.log)
}
// A write: the module keys, the node key (its commit), the mirror, then the module key no longer placed.
after := delivery.Placement{Node: "anchor", Modules: []delivery.PlacedModule{{Module: "shell", Ports: map[string]int{"22": 2222}}}}
kv.log = nil
rev, err := p.writeNode(ctx, "anchor", &after)
if err != nil || strings.Join(kv.log, " ") != "create anchor.shell update anchor.node update placement.anchor delete anchor.web" || rev != kv.revs["anchor.node"] {
t.Fatalf("the write = %v, %d, %v", kv.log, rev, err)
}
if got, _ := p.seatPlacements(); !delivery.SamePlacement(got["anchor"], after) {
t.Errorf("the write is not read back at once: %+v", got)
}
// A commit refused leaves every reader on what stood: the module key written before it carries the value it replaced.
kv.refuse, kv.log = "anchor.node", nil
moved := delivery.Placement{Node: "anchor", Modules: []delivery.PlacedModule{{Module: "shell", Ports: map[string]int{"22": 2223}}}}
if _, err := p.writeNode(ctx, "anchor", &moved); err == nil || strings.Join(kv.log, " ") != "update anchor.shell" {
t.Fatalf("a refused commit = %v, %v", kv.log, err)
}
fresh, err := delivery.PlacementsOf(kv.values)
if err != nil || !delivery.SamePlacement(fresh["anchor"], after) {
t.Errorf("a reader of the keys after a refused commit reads %+v, %v", fresh["anchor"], err)
}
kv.refuse = ""
// The verb's history reads the form before, then the node key.
st := memStore{delivery.PlacementState: &memKV{values: kv.values}}
hs := newSeatTools(st, nil)
hs.st = historyStore{memStore: st, revisions: map[string][]stdio.StateRevision{
"placement.anchor": {{Key: "placement.anchor", Value: raw, Revision: 1, Op: "put"}},
"anchor.node": {{Key: "anchor.node", Value: kv.values["anchor.node"], Revision: 5, Op: "put"}, {Key: "anchor.node", Revision: 9, Op: "del"}},
}}
pa, err := hs.doPlacement(map[string]any{"node": "anchor", "history": "true"})
a, _ := pa.(placementAnswer)
if err != nil || len(a.Changes) != 3 || !slices.Equal(a.Changes[0].Change.Added, []string{"web"}) ||
!slices.Equal(a.Changes[1].Change.Added, []string{"shell"}) || !slices.Equal(a.Changes[1].Change.Removed, []string{"web"}) ||
!a.Changes[2].Deleted || !slices.Equal(a.Changes[2].Change.Removed, []string{"shell"}) {
t.Errorf("history = %+v, %v", pa, err)
}
// No placement: the mirror first, then the node key, then the module keys.
kv.log = nil
if _, err := p.writeNode(ctx, "anchor", nil); err != nil || strings.Join(kv.log, " ") != "delete placement.anchor delete anchor.node delete anchor.shell" {
t.Errorf("the delete = %v, %v", kv.log, err)
}
})
t.Run("1954/D1: a failed provision is the assigner's finding until it is provided", func(t *testing.T) {
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, kvs: buckets{provisions.ProvisionsBucket: nil}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
failed, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Failed, Why: "the role exists already"})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", failed, 1, time.Now())
p.kvs[provisions.ProvisionsBucket] = &memKV{}
of := p.provisionsOf()
if got := of("photos")("ace"); got["ace.photos.db"].State != provisions.Failed {
t.Fatalf("provisions = %+v", got)
}
if _, ok := findings[assign.ProvisionFailedKey("ace.photos.db")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
provided, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Provided})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", provided, 2, time.Now())
p.provisionsOf()
if len(findings) != 0 {
t.Errorf("findings after provided = %v", findings)
}
// A required provision with no key: pulsed with the bound first, said once the pulse expired, ended once keyed.
ctx := context.Background()
p.kvs[delivery.PlacementPulsesBucket] = &memKV{values: map[string][]byte{}}
gone := map[string]bool{"ace.photos.s3": true}
p.provisionsMissing(ctx, gone)
if !p.pulsing("provision-missing.ace.photos.s3") || len(findings) != 0 {
t.Fatalf("first seen: pulsing %v, findings %v", p.pulsing("provision-missing.ace.photos.s3"), findings)
}
p.provisionsMissing(ctx, gone)
if len(findings) != 0 {
t.Fatal("said before its bound")
}
p.w.put(delivery.PlacementPulsesBucket, "provision-missing.ace.photos.s3", nil) // the server expired it
p.provisionsMissing(ctx, gone)
if _, ok := findings[assign.ProvisionMissingKey("ace.photos.s3")]; !ok {
t.Fatalf("past its bound: findings %v", findings)
}
p.provisionsMissing(ctx, map[string]bool{})
if len(findings) != 0 {
t.Errorf("keyed again: findings %v", findings)
}
})
}
// wantsVerbs (novox/hq ADR 0565 §7, §8, §9): upgrade answers each module's policy and its source; a person's choice is
// written with who and why, recorded as a hand act in the seat's log and said as policy-changed, and default takes it back;
// the bus is refused; compare lists each node's composition and the findings; bus names the node behind the indexer's newest.
func wantsVerbs(t *testing.T) {
fpOld, fpNew := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64)
st := memStore{}
for _, state := range []string{modulesState, manifestsState, policiesState, haveState, putsState, "findings", delivery.ComposedRead} {
st[state] = &memKV{values: map[string][]byte{}}
}
// ADR 0593 §6, §8: what each node published, every node's bucket as one state: anchor's own composition and its web at
// a fingerprint, and laptop's web refused there.
webKey, _ := delivery.ComposedKey("anchor", "web", fpNew)
refusedKey, _ := delivery.ComposedKey("laptop", "web", fpNew)
st[delivery.ComposedRead].values["anchor.node"], _ = delivery.EncodeComposed(delivery.Composed{Node: "anchor"})
st[delivery.ComposedRead].values[webKey], _ = delivery.EncodeComposed(delivery.Composed{Node: "anchor", Module: "web", FP: fpNew})
st[delivery.ComposedRead].values[refusedKey], _ = delivery.EncodeComposed(delivery.Composed{Node: "laptop", Module: "web", FP: fpNew, Refused: "a required value is unset"})
manifest := func(name, extra string) {
raw := `{"module":"` + name + `","version":"1"` + extra + `}`
digest := delivery.ManifestDigest([]byte(raw))
st[manifestsState].values[digest], _ = delivery.EncodeManifest(delivery.Manifest{V: delivery.Version, Module: name, JSON: raw})
st[modulesState].values[name], _ = delivery.EncodeModule(delivery.Module{V: delivery.Version, FP2: fpNew, Manifest: digest, Repository: "novox/x", Commit: "c"})
}
manifest("web", "")
manifest("photos", `,"upgrade":{"policy":"record","why":"a person checks the albums"}`)
manifest("nats", `,"provides":[{"name":"mesh-bus","scope":"mesh"}],"upgrade":{"policy":"record","why":"its streams"}`)
for node, fp := range map[string]string{"anchor": fpOld, "laptop": fpNew} {
st[haveState].values[node+".nats"], _ = delivery.EncodeHave(delivery.Have{FP: fp, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
}
st[haveState].values["anchor.node"], _ = delivery.EncodeHaveNode(delivery.HaveNode{Composed: "sha256:" + strings.Repeat("c", 64)})
var asked []string
var said []string
w := newWantsTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
return json.RawMessage(`{"ok":true}`), nil
}, func(event, id string, body any) error { said = append(said, event+" "+id); return nil })
policyOf := func(module string) policyRow {
out, err := w.doUpgrade(map[string]any{"module": module}, "op")
if err != nil {
t.Fatal(err)
}
return out.(map[string]any)["modules"].([]policyRow)[0]
}
if p := policyOf("photos"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule {
t.Errorf("photos = %+v", p)
}
if p := policyOf("nats"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule { // the policy its manifest says (issue 774)
t.Errorf("nats = %+v", p)
}
// compare: each node's own composition is the digest its have names, and the assigner's findings; nothing splits, the
// controller being gone (1126/B8).
out, _ := w.doUpgrade(map[string]any{"compare": "true"}, "op")
if got := out.(map[string]any); got["splits"] != nil || got["findings"] == nil ||
got["composed"].(map[string]string)["anchor"] != "sha256:"+strings.Repeat("c", 64) || len(got["composed"].(map[string]string)) != 1 {
t.Errorf("compare = %+v", got)
}
pub, _ := out.(map[string]any)["published"].(map[string][]publishedRow)
if a, l := pub["anchor"], pub["laptop"]; len(a) != 2 || a[0].Module != "node" || a[1].Module != "web" || a[1].FP != fpNew ||
!strings.HasPrefix(a[1].Digest, "sha256:") || len(l) != 1 || l[0].Refused != "a required value is unset" || l[0].Digest != "" {
t.Errorf("published = %+v (unread %v)", pub, out.(map[string]any)["published-unread"])
}
// Read by watch (issue 1953 L5f): the same answer, each watched state started once, and no get per key of the
// large states (the modules, their manifests, the composed record).
watches := map[string]int{}
byWatch := newWantsTools(st, nil, nil)
byWatch.watched = stdio.WatchedStatesOver(func(state string, handle func(stdio.StateChange) error) error {
watches[state]++
for k, v := range st[state].values {
_ = handle(stdio.StateChange{Key: k, Op: stdio.OpPut, Value: v, Current: true})
}
return nil
}, func(state, key string) ([]byte, bool, error) {
watches[state+" by key"]++
v, ok := st[state].values[key]
return v, ok, nil
})
for range 2 {
wout, werr := byWatch.doUpgrade(map[string]any{"compare": "true"}, "op")
if werr != nil {
t.Fatal(werr)
}
if got := wout.(map[string]any); got["composed"].(map[string]string)["anchor"] != "sha256:"+strings.Repeat("c", 64) {
t.Errorf("compare by watch = %+v", got)
}
}
if n := watches[manifestsState+" by key"]; n == 0 || n > len(st[manifestsState].values) {
t.Errorf("the manifests were read by key %d times, want each once", n)
}
for _, state := range []string{modulesState, haveState, health.FindingsState} {
if watches[state] != 1 {
t.Errorf("%s was watched %d times, want once", state, watches[state])
}
}
// ADR 0580 clause 3: the compare reads through mesh-sdk's watched reader and the seat keeps no mirror of its own.
for f := range reflect.TypeFor[wantsTools]().Fields() {
if f.Type.Kind() == reflect.Map || (f.Type.Kind() == reflect.Pointer && f.Type.Elem().Kind() == reflect.Map) {
t.Errorf("wantsTools keeps %s, a copy of its own beside the watched reader", f.Name)
}
}
if watches[policiesState] != 0 {
t.Errorf("the seat's own small policies state was watched: %v", watches)
}
// a person's record for web: written, recorded, said.
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "record", "why": "the operator checks it"}, "person.jochen"); err != nil {
t.Fatal(err)
}
chosen, _ := assign.DecodePolicies(st[policiesState].values)
if c := chosen["web"]; c.Policy != catalogue.PolicyRecord || c.By != "person.jochen" || !slices.Contains(asked, "seat:mesh-hand-acts.hand-act") ||
len(said) != 1 || !strings.HasPrefix(said[0], "policy-changed policy-changed.web.") {
t.Errorf("chosen %+v, asked %v, said %v", c, asked, said)
}
// A person may choose the bus's policy as any module's (ADR 0601 §1: no part treats the module that provides the bus apart).
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "roll", "why": "x"}, "op"); err != nil {
t.Errorf("a person's policy for the bus was refused: %v", err)
}
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "default", "why": "back to the module"}, "op"); err != nil {
t.Fatal(err)
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "roll"}, "op"); err == nil {
t.Error("a policy without why was set")
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "default", "why": "back to the module"}, "op"); err != nil || len(st[policiesState].values) != 0 {
t.Errorf("default left %v (%v)", st[policiesState].values, err)
}
// bus: the node behind the indexer's newest.
b, err := w.doBus(nil)
if err != nil {
t.Fatal(err)
}
nodes := b.(map[string]any)["nodes"].([]busNode)
if b.(map[string]any)["module"] != "nats" || len(nodes) != 2 || !nodes[0].Behind || nodes[1].Behind || !slices.Contains(asked, "seat:mesh-hand-acts.hand-acts") {
t.Errorf("bus = %+v, asked %v", b, asked)
}
if tools := w.tools(); len(tools) != 2 || tools[0].Name != "mesh-wants.upgrade" || tools[1].Run == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
}
// readManifest is the module's manifest, read as the controller reads it.
func readManifest() (catalogue.Manifest, error) {
raw, err := os.ReadFile("../../module.json")
if err != nil {
return catalogue.Manifest{}, err
}
return catalogue.Decode(raw)
}
// countingKV is a memKV that counts its puts.
type countingKV struct {
memKV
writes *int
}
func (c *countingKV) Put(ctx context.Context, key string, value []byte) (uint64, error) {
*c.writes++
return c.memKV.Put(ctx, key, value)
}
// revKV is a bucket with real revisions, refusing a write at a revision that is not the key's, and a write of refuse;
// log is every write in order.
type revKV struct {
jetstream.KeyValue
rev uint64
values map[string][]byte
revs map[string]uint64
log []string
refuse string
}
func (k *revKV) write(key string, value []byte) uint64 {
k.rev++
k.values[key], k.revs[key] = value, k.rev
return k.rev
}
func (k *revKV) Create(_ context.Context, key string, value []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
if _, ok := k.values[key]; ok || key == k.refuse {
return 0, errors.New("key exists")
}
k.log = append(k.log, "create "+key)
return k.write(key, value), nil
}
func (k *revKV) Update(_ context.Context, key string, value []byte, revision uint64) (uint64, error) {
if k.revs[key] != revision || key == k.refuse {
return 0, errors.New("wrong last sequence")
}
k.log = append(k.log, "update "+key)
return k.write(key, value), nil
}
func (k *revKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
if key == k.refuse {
return errors.New("wrong last sequence")
}
k.log = append(k.log, "delete "+key)
delete(k.values, key)
delete(k.revs, key)
return nil
}
// historyStore is a memStore whose history is given.
type historyStore struct {
memStore
revisions map[string][]stdio.StateRevision
}
func (s historyStore) history(_, key string) ([]stdio.StateRevision, error) {
return s.revisions[key], nil
}
@@ -0,0 +1,236 @@
{
"module": "xterm",
"version": "1",
"moved": true,
"capabilities": [
"package-manager"
],
"requires": [
"x11-display"
],
"claims": [
{
"name": "node-terminal-emulator",
"scope": "node",
"serves": [
"open"
]
}
],
"tools": [
"xterm_font",
"xterm_colours"
],
"config": {
"face": {
"type": "string",
"description": "The font family xterm draws with, as fc-list names it: JetBrainsMono Nerd Font, because research 026/04 chose it for its glyphs the prompt uses. A family the machine does not have falls back to the nearest installed face (xterm_font says which).",
"pattern": "[A-Za-z0-9 ._-]{1,64}",
"default": "JetBrainsMono Nerd Font"
},
"size": {
"type": "number",
"description": "The size of xterm's text, in points: 12 reads well on a screen of about 100 DPI, and a denser screen needs a larger size.",
"minimum": 6,
"maximum": 72,
"predefined": "font-size",
"default": 12
},
"save-lines": {
"type": "number",
"description": "How many lines of scrollback xterm keeps per terminal: 10000 holds a long build log, and each line costs memory in every open terminal.",
"minimum": 0,
"maximum": 1000000,
"default": 10000
},
"term-name": {
"type": "string",
"description": "The TERM value xterm sets for the programs it runs: xterm-256color, because the palette has 256 colours and every program knows the name.",
"pattern": "[A-Za-z0-9._-]{1,64}",
"default": "xterm-256color"
},
"render-font": {
"type": "boolean",
"description": "Whether xterm draws with fontconfig faces (true) or the old X core fonts (false): true, because the face is a fontconfig family and core fonts cannot show it.",
"default": true
},
"bold-mode": {
"type": "boolean",
"description": "Whether xterm draws bold text by overstriking the normal face (true): false, because the face has a real bold style and overstriking blurs it.",
"default": false
},
"always-bold-mode": {
"type": "boolean",
"description": "Whether xterm overstrikes bold even where the face has a bold style (true): false, for the same reason as bold-mode.",
"default": false
},
"allow-bold-fonts": {
"type": "boolean",
"description": "Whether xterm may use the face's bold style for bold text (true): true, so bold text is drawn with the real bold weight.",
"default": true
},
"background": {
"type": "string",
"description": "The terminal's background colour, as #rrggbb: black, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"foreground": {
"type": "string",
"description": "The terminal's text colour, as #rrggbb: a light grey, easier on the eyes than white on black.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a8a8a8"
},
"color0": {
"type": "string",
"description": "Palette entry 0, the normal black, as #rrggbb: #000000, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#000000"
},
"color1": {
"type": "string",
"description": "Palette entry 1, the normal red, as #rrggbb: #9e1828, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#9e1828"
},
"color2": {
"type": "string",
"description": "Palette entry 2, the normal green, as #rrggbb: #008800, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#008800"
},
"color3": {
"type": "string",
"description": "Palette entry 3, the normal yellow, as #rrggbb: #d2bb4b, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#d2bb4b"
},
"color4": {
"type": "string",
"description": "Palette entry 4, the normal blue, as #rrggbb: #414171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#414171"
},
"color5": {
"type": "string",
"description": "Palette entry 5, the normal magenta, as #rrggbb: #963c59, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#963c59"
},
"color6": {
"type": "string",
"description": "Palette entry 6, the normal cyan, as #rrggbb: #418179, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#418179"
},
"color7": {
"type": "string",
"description": "Palette entry 7, the normal white, as #rrggbb: #bebebe, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bebebe"
},
"color8": {
"type": "string",
"description": "Palette entry 8, the bright black, as #rrggbb: #666666, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#666666"
},
"color9": {
"type": "string",
"description": "Palette entry 9, the bright red, as #rrggbb: #bc5766, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#bc5766"
},
"color10": {
"type": "string",
"description": "Palette entry 10, the bright green, as #rrggbb: #61a171, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#61a171"
},
"color11": {
"type": "string",
"description": "Palette entry 11, the bright yellow, as #rrggbb: #e7db52, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#e7db52"
},
"color12": {
"type": "string",
"description": "Palette entry 12, the bright blue, as #rrggbb: #5085af, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#5085af"
},
"color13": {
"type": "string",
"description": "Palette entry 13, the bright magenta, as #rrggbb: #a97a99, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#a97a99"
},
"color14": {
"type": "string",
"description": "Palette entry 14, the bright cyan, as #rrggbb: #6ba4a4, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#6ba4a4"
},
"color15": {
"type": "string",
"description": "Palette entry 15, the bright white, as #rrggbb: #ffffff, as both workstations ran it.",
"pattern": "#[0-9a-fA-F]{6}",
"default": "#ffffff"
},
"copy-binding": {
"type": "string",
"description": "The key combination that copies the selection to the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> C, so a plain Ctrl+C still interrupts the program. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> C"
},
"paste-binding": {
"type": "string",
"description": "The key combination that pastes the clipboard, in X translation syntax: Ctrl Shift ~Alt <Key> V, the pair of copy-binding. It holds no colon, backslash or line break, which would end the resource.",
"pattern": "[^:\\\\\\n]+",
"default": "Ctrl Shift ~Alt <Key> V"
}
},
"items": [
{
"collection": "environment",
"name": "terminal",
"attributes": {
"kind": "variable",
"name": "TERMINAL"
},
"template": "files/env-terminal"
},
{
"collection": "shell:xresources:normal",
"name": "xresources-normal",
"template": "files/xresources-normal"
}
],
"resources": [
{
"id": "package",
"type": "package",
"package": "xterm"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/xterm-tools",
"binary": "xterm-tools",
"loads": [
"xterm-tools"
]
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,371 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"os"
"regexp"
"slices"
"sort"
"strings"
"sync"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The assigner's tools (novox/hq ADR 0546), served through the tool runner from its bundle: a process never serves (ADR
// 0436, mesh-sdk process). They write and read the puts; the assigner's process takes each put further (puts.go) and says
// every step as the event put. They are the module's own tools; the seat mesh-placement's verbs are served beside them
// (seat_tools.go, ADR 0567 §10).
// putsState is the assigner's state of puts, as the runtime names a module's own state.
const putsState = "puts"
// answerWithin is how long assigner_put waits for the assigner's process to take the put up before it answers what stands.
var answerWithin = 20 * time.Second
// store is what the tools need of the states: the runtime's (stdio.State), or a test's.
type store interface {
get(state, key string) ([]byte, uint64, bool, error)
put(state, key string, value []byte) error
keys(state string) ([]string, error)
}
// putTools serves the tools over a store, and keeps the newest value of every put it heard, to wait on.
type putTools struct {
st store
now func() time.Time
mu sync.Mutex
// heard is every put the watch handed, by key; changed is closed and replaced at each.
heard map[string][]byte
changed chan struct{}
}
func newPutTools(st store) *putTools {
return &putTools{st: st, now: time.Now, heard: map[string][]byte{}, changed: make(chan struct{})}
}
// hear takes a put's change in from the watch.
func (s *putTools) hear(key string, value []byte) {
s.mu.Lock()
s.heard[key] = value
close(s.changed)
s.changed = make(chan struct{})
s.mu.Unlock()
}
// fullFP is a fingerprint of the new form, whole; prefix at least its first twelve hex digits.
var (
fullFP = regexp.MustCompile(`^src2:[0-9a-f]{64}$`)
prefix = regexp.MustCompile(`^(src2:)?[0-9a-f]{12,63}$`)
)
// version reads the version a put names: newest (the indexer's fingerprint at the trunk), a whole fingerprint, or the start
// of one the builder built, which must name one build alone.
func (s *putTools) version(module, version string) (string, error) {
v := strings.ToLower(strings.TrimSpace(version))
switch {
case v == "newest":
raw, _, found, err := s.st.get(delivery.IndexerModule+".modules", module)
if err != nil {
return "", fmt.Errorf("the indexer's modules cannot be read: %w", err)
}
if !found {
return "", fmt.Errorf("the indexer holds no module %s at the trunk", module)
}
m, err := delivery.DecodeModule(raw)
if err != nil {
return "", fmt.Errorf("the indexer's %s cannot be read: %w", module, err)
}
if m.FP2 == "" {
return "", fmt.Errorf("the indexer holds no fingerprint of the new form for %s: %s", module, m.None2)
}
return m.FP2, nil
case fullFP.MatchString(v):
return v, nil
case prefix.MatchString(v):
start := strings.TrimPrefix(v, delivery.SourcePrefix2)
keys, err := s.st.keys(delivery.BuilderModule + ".builds")
if err != nil {
return "", fmt.Errorf("the builder's builds cannot be listed: %w", err)
}
var found []string
for _, k := range keys {
m, fp, err := delivery.ParseBuilderBuildKey(k)
if err == nil && m == module && strings.HasPrefix(strings.TrimPrefix(fp, delivery.SourcePrefix2), start) {
found = append(found, fp)
}
}
switch len(found) {
case 0:
return "", fmt.Errorf("the builder built no %s whose fingerprint starts %s: give it whole", module, start)
case 1:
return found[0], nil
}
sort.Strings(found)
return "", fmt.Errorf("%s starts %d fingerprints of %s the builder built (%s): give more of it", start, len(found), module, strings.Join(found, ", "))
}
return "", fmt.Errorf("%q is no version: newest, a fingerprint (src2: and 64 hex digits) or at least its first 12 hex digits", version)
}
// putAnswer is what assigner_put and assigner_puts answer of one put.
type putAnswer struct {
Node string `json:"node"`
Module string `json:"module"`
delivery.Put
Follow string `json:"follow,omitempty"`
}
// doPut is the tool assigner_put.
func (s *putTools) doPut(args map[string]any, by string) (any, error) {
node, module, why := arg(args, "node"), arg(args, "module"), arg(args, "why")
key, err := delivery.PutKey(node, module)
if err != nil {
return nil, err
}
if why == "" {
return nil, errors.New("put says why: it is kept with the put and said in any ask")
}
hold := false
switch arg(args, "hold") {
case "true":
hold = true
case "", "false":
default:
return nil, fmt.Errorf("hold is true or false, not %q", arg(args, "hold"))
}
fp, err := s.version(module, arg(args, "version"))
if err != nil {
return nil, err
}
now := s.now().UTC()
words := fmt.Sprintf("asked by %s: %s", by, why)
if raw, _, found, err := s.st.get(putsState, key); err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
} else if found {
if before, err := delivery.DecodePut(raw); err == nil && !delivery.PutEnded(before.Step) {
if before.FP == fp && before.Hold == hold {
return putAnswer{Node: node, Module: module, Put: before, Follow: followPut(node, module)}, nil
}
words += fmt.Sprintf("; it replaces the put of %s at %s, which stood at %s", module, shortFP(before.FP), before.Step)
}
}
p := delivery.Put{FP: fp, Hold: hold, Why: why, By: by, At: now}.Said(now, delivery.PutAsked, words)
data, err := delivery.EncodePut(p)
if err != nil {
return nil, err
}
if err := s.st.put(putsState, key, data); err != nil {
return nil, fmt.Errorf("the put of %s on %s is not written: %w", module, node, err)
}
s.hear(key, data)
return putAnswer{Node: node, Module: module, Put: s.await(key, p), Follow: followPut(node, module)}, nil
}
// await is the put at key once the assigner's process took it up past its asking, or as it stands after answerWithin.
func (s *putTools) await(key string, p delivery.Put) delivery.Put {
deadline := time.After(answerWithin)
for {
s.mu.Lock()
raw, changed := s.heard[key], s.changed
s.mu.Unlock()
if now, err := delivery.DecodePut(raw); err == nil && now.At.Equal(p.At) {
p = now
if now.Step != delivery.PutAsked && now.Step != delivery.PutBuild && now.Step != delivery.PutCompose {
return p // it waits on a person or the node, or ended
}
}
select {
case <-changed:
case <-deadline:
return p
}
}
}
// followPut says how a caller follows a put after the answer.
func followPut(node, module string) string {
return fmt.Sprintf("the put goes on: mesh_wait on mesh-delivery-assigner.put with filter {\"node\": %q, \"module\": %q}, then "+
"mesh-delivery-assigner.assigner_puts for where it stands", node, module)
}
// doRelease is the tool assigner_release.
func (s *putTools) doRelease(args map[string]any, by string) (any, error) {
node, module, why := arg(args, "node"), arg(args, "module"), arg(args, "why")
key, err := delivery.PutKey(node, module)
if err != nil {
return nil, err
}
if why == "" {
return nil, errors.New("release says why: it is kept with the put")
}
raw, _, found, err := s.st.get(putsState, key)
if err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
}
if !found {
return nil, fmt.Errorf("%s holds no put of %s: nothing to release", node, module)
}
p, err := delivery.DecodePut(raw)
if err != nil {
return nil, fmt.Errorf("the put of %s on %s cannot be read: %w", module, node, err)
}
if delivery.PutEnded(p.Step) {
return nil, fmt.Errorf("the put of %s on %s ended already (%s: %s): nothing to release", module, node, p.Step, p.Words)
}
p = p.Said(s.now(), delivery.PutReleased, fmt.Sprintf("released by %s: %s; the assigner's rules write the want again", by, why))
data, err := delivery.EncodePut(p)
if err != nil {
return nil, err
}
if err := s.st.put(putsState, key, data); err != nil {
return nil, fmt.Errorf("the release of %s on %s is not written: %w", module, node, err)
}
s.hear(key, data)
return putAnswer{Node: node, Module: module, Put: p}, nil
}
// doPuts is the tool assigner_puts.
func (s *putTools) doPuts(args map[string]any) (any, error) {
keys, err := s.st.keys(putsState)
if err != nil {
return nil, fmt.Errorf("the puts cannot be listed: %w", err)
}
sort.Strings(keys)
out := []putAnswer{}
for _, key := range keys {
node, module, err := delivery.ParsePutKey(key)
if err != nil || (arg(args, "node") != "" && node != arg(args, "node")) || (arg(args, "module") != "" && module != arg(args, "module")) {
continue
}
raw, _, found, err := s.st.get(putsState, key)
if err != nil || !found {
continue
}
p, err := delivery.DecodePut(raw)
if err != nil {
continue
}
out = append(out, putAnswer{Node: node, Module: module, Put: p})
}
return map[string]any{"puts": out}, nil
}
func arg(args map[string]any, name string) string {
switch v := args[name].(type) {
case string:
return strings.TrimSpace(v)
case bool:
return fmt.Sprint(v)
}
return ""
}
// The tools' names, as the manifest lists them.
const (
toolPut = "assigner_put"
toolRelease = "assigner_release"
toolPuts = "assigner_puts"
)
// schema is a tool's arguments, each a string: required ones first named, and switches true or false.
func schema(properties map[string]string, required []string, switches ...string) map[string]any {
props := map[string]any{}
for name, description := range properties {
p := map[string]any{"type": "string", "description": description}
if slices.Contains(switches, name) {
p["enum"] = []string{"true", "false"}
}
props[name] = p
}
out := map[string]any{"type": "object", "properties": props}
if len(required) > 0 {
out["required"] = required
}
return out
}
// tools are the assigner's tools as the runtime serves them (ADR 0546 clause 1).
func (s *putTools) tools() []stdio.Tool {
return []stdio.Tool{
{Name: toolPut, Description: "Put one version of a module on one machine (novox/hq ADR 0546): the build at that fingerprint, " +
"the composition and grant the controller wrote for that machine, then the machine's want naming it, held there until " +
"released when hold is true. A module whose moves a person takes (the bus, a module of policy record) is asked of the " +
"operator first, and the bus has its streams snapshotted after the yes. Answers what was done or what it waits on; the " +
"put goes on after the answer, each step said as the event mesh-delivery-assigner.put, and assigner_puts says where it stands.",
Input: schema(map[string]string{
"node": "the machine, where the module is assigned",
"module": "the module",
"version": "newest, a fingerprint (src2: and 64 hex digits), or at least its first 12 hex digits",
"hold": "\"true\": the want stays at this version until released; \"false\" (the default): the rules move it again once it runs",
"why": "why, kept with the put and said in any ask",
}, []string{"node", "module", "version", "why"}, "hold"),
Run: func(a map[string]any) (any, error) { return s.doPut(a, stdio.CallerOf(a)) }},
{Name: toolRelease, Description: "End a put on one machine (novox/hq ADR 0546): its hold is taken away and the assigner's " +
"rules write the machine's want again from their next pass. Nothing changes on the machine until they move it.",
Input: schema(map[string]string{"node": "the machine", "module": "the module", "why": "why, kept with the put"},
[]string{"node", "module", "why"}),
Run: func(a map[string]any) (any, error) { return s.doRelease(a, stdio.CallerOf(a)) }},
{Name: toolPuts, Description: "Every put and where it stands (novox/hq ADR 0546): its version, hold, why, each step with " +
"its time and words. Narrowed by machine or module. Read-only.",
Input: schema(map[string]string{"node": "one machine", "module": "one module"}, nil),
Run: s.doPuts},
}
}
// runtimeStore is the runtime's states.
type runtimeStore struct{}
func (runtimeStore) get(state, key string) ([]byte, uint64, bool, error) {
e, err := stdio.State(state).Get(key)
if err != nil || e == nil {
return nil, 0, false, err
}
return e.Value, e.Revision, true, nil
}
func (runtimeStore) put(state, key string, value []byte) error {
_, err := stdio.State(state).Put(key, json.RawMessage(value))
return err
}
func (runtimeStore) keys(state string) ([]string, error) { return stdio.State(state).Keys() }
// serveTools serves the tools to the runtime until it stops, hearing every put to answer assigner_put with its first steps.
func serveTools() error {
s := newPutTools(runtimeStore{})
go func() {
<-stdio.Served()
for wait := 2 * time.Second; ; wait = min(wait*2, time.Minute) {
err := stdio.State(putsState).Watch("", func(c stdio.StateChange) error {
s.hear(c.Key, c.Value)
return nil
})
if err == nil {
return
}
fmt.Fprintf(os.Stderr, "mesh-delivery-assigner: the puts are not heard yet (%v); asked again in %s\n", err, wait)
time.Sleep(wait)
}
}()
tools := append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...)
wants := newWantsTools(runtimeStore{}, stdio.Ask, func(e, id string, b any) error {
// The seat's event, said on the seat: a claimed seat's event is never the module's own.
return stdio.SeatPublish(wantsEventSubject(e), b, id)
})
wants.watched = stdio.NewWatchedStates() // the large states read whole from one watched copy (ADR 0580 clause 3)
tools = append(tools, wants.tools()...)
return stdio.Serve("", tools)
}
// shortFP is a fingerprint's first twelve hex digits.
func shortFP(fp string) string {
part, err := delivery.FingerprintKeyPart(fp)
if err != nil || len(part) < 12 {
return fp
}
return part[:12]
}
@@ -0,0 +1,320 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-wants' verbs (novox/hq ADR 0565 §7, §8, §9, §15; issue 1953 D4-D6), served through the tool runner beside the
// put tools and mesh-placement's verbs. `upgrade` reads and sets a person's upgrade policy, kept in the state `policies`
// (each set recorded as a hand act and said as policy-changed), and with compare lists the digest of the composition each node's
// own have names beside every composition each node published (ADR 0593 §8, composed.go); `bus` says what a bus upgrade would do. The assigner's process reads
// `policies` when it judges a move and writes the finding policy-split while a split stands.
// The states the verbs read, as the runtime names them.
const (
policiesState = delivery.AssignerPoliciesState
manifestsState = delivery.IndexerModule + ".manifests"
// PolicyChangedEvent is the seat's event of a person's policy set or taken back.
PolicyChangedEvent = "policy-changed"
// busStepCause is the cause the bus's step is recorded under in the hand-act log.
busStepCause = "bus-upgrade"
)
// wantsTools serves the seat's verbs over a store, asking seats through ask and saying events through emit.
type wantsTools struct {
st seatStore
ask func(key string, body any) (json.RawMessage, error)
emit func(event, id string, body any) error
now func() time.Time
// watched is the copy of the large states it reads whole, each by one watch started on its first read and kept by
// its changes after (mesh-sdk's WatchedStates, novox/hq ADR 0580 clause 3), never a get per key, which timed out on
// a large state (issue 1953 L5f). Nil reads key by key.
watched *stdio.WatchedStates
}
// wantsEventSubject is the subject of one of the seat's events.
func wantsEventSubject(event string) string {
return "mesh.seat." + catalogue.WantsSeat + ".event." + event
}
func newWantsTools(st seatStore, ask func(string, any) (json.RawMessage, error), emit func(event, id string, body any) error) *wantsTools {
return &wantsTools{st: st, ask: ask, emit: emit, now: time.Now}
}
// values reads every key of a state: from the watched copy when there is one, else key by key.
func (s *wantsTools) values(state string) (map[string][]byte, error) {
if s.watched != nil {
return s.watched.Values(state)
}
return s.direct(state)
}
// direct reads a state key by key, for a small state of the seat's own.
func (s *wantsTools) direct(state string) (map[string][]byte, error) {
keys, err := s.st.keys(state)
if err != nil {
return nil, fmt.Errorf("%s cannot be listed: %w", state, err)
}
out := map[string][]byte{}
for _, k := range keys {
raw, _, found, err := s.st.get(state, k)
if err != nil {
return nil, fmt.Errorf("%s %s cannot be read: %w", state, k, err)
}
if found {
out[k] = raw
}
}
return out, nil
}
// manifest reads a manifest by its digest: kept once read when there is a watched copy, a manifest never changing
// under its digest.
func (s *wantsTools) manifest(digest string) ([]byte, bool, error) {
if s.watched != nil {
return s.watched.Once(manifestsState, digest)
}
raw, _, found, err := s.st.get(manifestsState, digest)
return raw, found, err
}
// manifests is every module the indexer holds, with its trunk manifest and fingerprint.
func (s *wantsTools) manifests() (map[string]catalogue.Manifest, map[string]delivery.Module, error) {
mods, err := s.values(modulesState)
if err != nil {
return nil, nil, err
}
out, held := map[string]catalogue.Manifest{}, map[string]delivery.Module{}
for name, raw := range mods {
m, err := delivery.DecodeModule(raw)
if err != nil {
continue
}
held[name] = m
if m.Manifest == "" {
continue
}
mraw, found, err := s.manifest(m.Manifest)
if err != nil || !found {
continue
}
kept, err := delivery.DecodeManifest(mraw)
if err != nil {
continue
}
// Decoded, never judged (ADR 0578 §3); one that does not decode is left out of the answer, as before.
if man, err := catalogue.Decode([]byte(kept.JSON)); err == nil {
out[name] = man
}
}
return out, held, nil
}
// policyRow is one module's policy as upgrade answers it.
type policyRow struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Why string `json:"why,omitempty"`
By string `json:"by,omitempty"`
}
// doUpgrade is the verb upgrade: read every module's policy or one's; with policy and why set a person's choice, or take it
// back with default; with compare each node's composition and the assigner's findings.
func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
manifests, _, err := s.manifests()
if err != nil {
return nil, err
}
raw, err := s.direct(policiesState) // the seat's own small state, read whole after its own writes
if err != nil {
return nil, err
}
chosen, unreadable := assign.DecodePolicies(raw)
if arg(args, "compare") == "true" {
haves, err := s.values(haveState)
if err != nil {
return nil, err
}
composed := map[string]string{}
for key, raw := range haves {
if node, _, isNode, err := delivery.ParseHaveKey(key); err == nil && isNode {
if h, err := delivery.DecodeHaveNode(raw); err == nil && h.Composed != "" {
composed[node] = h.Composed
}
}
}
out := map[string]any{"composed": composed}
// And what every node published (ADR 0593 §8), read only to compare.
if values, err := s.values(delivery.ComposedRead); err != nil {
out["published-unread"] = err.Error()
} else {
out["published"] = published(values)
}
// The assigner's findings, its said conditions as the condition keeper reads them (§15).
if found, err := s.values(health.FindingsState); err != nil {
out["findings-unread"] = err.Error()
} else {
keys := []string{}
for k := range found {
keys = append(keys, k)
}
sort.Strings(keys)
out["findings"] = keys
}
return out, nil
}
module, policy := arg(args, "module"), arg(args, "policy")
if policy != "" {
return s.setPolicy(module, policy, arg(args, "why"), by, manifests)
}
var rows []policyRow
for name, m := range manifests {
if module != "" && name != module {
continue
}
p, from, why, who := assign.PolicyOf(m, chosen)
rows = append(rows, policyRow{Module: name, Policy: p, From: from, Why: why, By: who})
}
if module != "" && len(rows) == 0 {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Module < rows[j].Module })
return map[string]any{"modules": rows, "unreadable": unreadable}, nil
}
// setPolicy writes a person's choice of a module's policy, or deletes it for default, records the act in the hand-act log
// and says policy-changed. The bus's choice is a person's as any module's (ADR 0601 §1).
func (s *wantsTools) setPolicy(module, policy, why, by string, manifests map[string]catalogue.Manifest) (any, error) {
if module == "" {
return nil, errors.New("a policy is set for one module: name it")
}
if _, ok := manifests[module]; !ok {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
if why == "" {
return nil, errors.New("a person's policy says why")
}
if by == "" || by == stdio.Unproven {
return nil, errors.New("a person's policy names who chose it, and the runtime proved no caller")
}
what := fmt.Sprintf("mesh-wants.upgrade %s %s", module, policy)
if policy == "default" {
if err := s.st.del(policiesState, module); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be taken back: %w", module, err)
}
} else {
p := catalogue.PersonPolicy{V: catalogue.PersonPolicyVersion, Module: module, Policy: policy, Why: why, By: by, At: s.now().UTC()}
if err := p.Check(); err != nil {
return nil, err
}
raw, _ := json.Marshal(p)
if err := s.st.put(policiesState, module, raw); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be written: %w", module, err)
}
}
rec := handacts.Recorder{Ask: s.ask}
taken, rerr := rec.Record(handacts.Request{What: what, Why: why, Cause: "upgrade-policy", Verb: "mesh-wants.upgrade"})
recorded := taken.Log
if rerr != nil {
recorded = "not recorded: " + rerr.Error()
}
body := map[string]any{"module": module, "policy": policy, "why": why, "by": by}
said := ""
if s.emit != nil {
id := PolicyChangedEvent + "." + module + "." + strconv.FormatInt(s.now().UnixNano(), 10)
if err := s.emit(PolicyChangedEvent, id, body); err != nil {
said = "not said: " + err.Error()
}
}
return map[string]any{"set": body, "hand-act": recorded, "event": said}, nil
}
// busNode is one node's bus as bus answers it.
type busNode struct {
Node string `json:"node"`
FP string `json:"fp"`
Behind bool `json:"behind"`
}
// doBus is the verb bus: the fingerprint of the bus module each node runs against the indexer's newest, the nodes behind,
// the last bus steps of the hand-act log and the assigner's puts of the bus. Read-only.
func (s *wantsTools) doBus(map[string]any) (any, error) {
manifests, held, err := s.manifests()
if err != nil {
return nil, err
}
bus := ""
for name, m := range manifests {
if catalogue.ProvidesBus(m) {
bus = name
}
}
if bus == "" {
return nil, errors.New("the indexer holds no module that provides the bus")
}
newest := held[bus].FP2
haves, err := s.values(haveState)
if err != nil {
return nil, err
}
var nodes []busNode
for key, raw := range haves {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode || module != bus {
continue
}
h, err := delivery.DecodeHave(raw)
if err != nil || h.Phase == delivery.PhaseRemoved {
continue
}
nodes = append(nodes, busNode{Node: node, FP: h.FP, Behind: newest != "" && h.FP != newest})
}
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Node < nodes[j].Node })
out := map[string]any{"module": bus, "newest": newest, "nodes": nodes}
if steps, err := s.ask("seat:"+handacts.Seat+".hand-acts", map[string]any{"cause": busStepCause, "days": 90}); err != nil {
out["steps-unread"] = err.Error()
} else {
out["steps"] = steps
}
puts, err := s.values(putsState)
if err != nil {
out["puts-unread"] = err.Error()
} else {
mine := map[string]json.RawMessage{}
for k, v := range puts {
if _, module, err := delivery.ParsePutKey(k); err == nil && module == bus {
mine[k] = v
}
}
out["puts"] = mine
}
return out, nil
}
// tools are the seat's verbs as the runtime serves them, each named <seat>.<verb>, with the seat's own schemas.
func (s *wantsTools) tools() []stdio.Tool {
seat, _ := catalogue.SeatNamed(catalogue.WantsSeat)
run := map[string]func(map[string]any) (any, error){
"upgrade": func(a map[string]any) (any, error) { return s.doUpgrade(a, stdio.CallerOf(a)) },
"bus": s.doBus,
}
var out []stdio.Tool
for _, v := range seat.Serves {
out = append(out, stdio.Tool{Name: catalogue.WantsSeat + "." + v.Name, Description: v.Description, Input: v.Input, Run: run[v.Name]})
}
return out
}
@@ -0,0 +1,484 @@
package main
import (
"context"
"errors"
"slices"
"sort"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
)
// noModuleWords is the indexer's reason for a module of a candidate whose directory the merge preview holds no manifest
// for (mesh-sdk delivery.CandidateNoneNoModule, which the indexer writes): the change removes the module.
const noModuleWords = delivery.CandidateNoneNoModule
// world is what the assigner watched, as it stands now: every value by bucket and key, and the manifests read by
// digest. The watches write it; a pass reads it.
type world struct {
mu sync.Mutex
values map[string]map[string][]byte
manifests map[string]parsed
// healthy is, by have key, every fingerprint the node ran healthy, oldest first, each once: what a set-back goes to.
healthy map[string][]string
// changes counts, by bucket and key, the values this process has seen the key take: a value written again is another.
changes map[string]uint64
// meta is, by bucket and key, the revision and the time of the value held now, where the watch said them (putEntry).
meta map[string]entryMeta
// trunkBuilt is, by module, its newest done trunk build in the builder's log (putBuilt).
trunkBuilt map[string]delivery.BuildLogEntry
// placement is the seat's state placement assembled node by node from its keys in either form (mesh-sdk
// delivery.PlacementKeys; research 072 row 37): the one reader of its keys in the assigner.
placement *delivery.PlacementKeys
// sourceGroups is, by revision of the gate's source.main, the release (group) that moved it, from every revision its
// history keeps (watched with history): a build's source revision names the release that carries it (ADR 0601 §1).
sourceGroups map[uint64]string
}
// entryMeta is a value's revision in its bucket and when it was written.
type entryMeta struct {
revision uint64
at time.Time
}
// parsed is a manifest read by digest: its module and its manifest, or why it cannot be read.
type parsed struct {
module string
manifest catalogue.Manifest
err string
}
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{},
meta: map[string]entryMeta{}, trunkBuilt: map[string]delivery.BuildLogEntry{}, placement: delivery.NewPlacementKeys(),
sourceGroups: map[uint64]string{}}
}
// putEntry is put, keeping the value's revision and the time it was written.
func (w *world) putEntry(bucket, key string, value []byte, revision uint64, at time.Time) {
w.put(bucket, key, value)
w.mu.Lock()
defer w.mu.Unlock()
if value == nil {
delete(w.meta, bucket+"|"+key)
return
}
w.meta[bucket+"|"+key] = entryMeta{revision: revision, at: at}
if bucket == delivery.PlacementBucket {
_, _ = w.placement.Put(key, value, revision, false) // put took it already; this is its revision
}
if bucket == delivery.SourceBucket && key == delivery.SourceMainKey {
if s, err := delivery.DecodeSource(value); err == nil && s.Group != "" {
w.sourceGroups[revision] = s.Group
}
}
}
// release is the release whose Approve carries module's build fp (ADR 0601 §1), and whether one does: the trunk's value of
// the module names fp, the source revision at which that fingerprint last changed (FPSrc, never the Src of its last merge,
// which a recompute with no merge keeps) names the release that moved it, and the gate's merge of that release, which it
// writes on the operator's Approve, holds the module's own trunk merge among its merged pull requests. Any of them
// unknown: none, and the move is asked of a person.
func (w *world) release(module, fp string) (string, bool) {
t, ok := w.trunk()[module]
if !ok || fp == "" || t.FP2 != fp || t.FPSrc == 0 || t.Commit == "" {
return "", false
}
w.mu.Lock()
group := w.sourceGroups[t.FPSrc]
w.mu.Unlock()
key, err := delivery.MergesKey(group)
if group == "" || err != nil {
return "", false
}
raw := w.value(delivery.MergesBucket, key)
if raw == nil {
return "", false
}
merge, err := delivery.DecodeMerge(raw)
if err != nil {
return "", false
}
for _, pull := range merge.Pulls {
if pull.State == delivery.MergeMerged && pull.Commit == t.Commit && delivery.SameRepository(pull.Repository, t.Repository) {
return group, true
}
}
return "", false
}
// metaOf is the revision and time of the value a bucket holds at key, as the watch said them.
func (w *world) metaOf(bucket, key string) entryMeta {
w.mu.Lock()
defer w.mu.Unlock()
return w.meta[bucket+"|"+key]
}
// put takes a change of a watched bucket in: a value, or nil for a key deleted or purged.
func (w *world) put(bucket, key string, value []byte) {
w.mu.Lock()
defer w.mu.Unlock()
if w.values[bucket] == nil {
w.values[bucket] = map[string][]byte{}
}
w.changes[bucket]++ // the bucket's generation (bucketGen), a delete included
if value == nil {
delete(w.values[bucket], key)
if bucket == delivery.PlacementBucket {
_, _ = w.placement.Put(key, nil, 0, true)
}
return
}
w.values[bucket][key] = value
w.changes[bucket+"|"+key]++
if bucket == delivery.PlacementBucket {
// A key or value that does not read places nothing new on its node; the indexer's read tool says it.
_, _ = w.placement.Put(key, value, 0, false)
}
if bucket == delivery.HaveBucket {
if h, err := delivery.DecodeHave(value); err == nil && h.FP != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
list := slices.DeleteFunc(w.healthy[key], func(fp string) bool { return fp == h.FP })
w.healthy[key] = append(list, h.FP)
}
}
}
// lastHealthy is the newest fingerprint other than fp the node ran the module healthy at, or empty.
func (w *world) lastHealthy(node, module, fp string) string {
key, err := delivery.HaveKey(node, module)
if err != nil {
return ""
}
w.mu.Lock()
defer w.mu.Unlock()
list := w.healthy[key]
for i := len(list) - 1; i >= 0; i-- {
if list[i] != fp {
return list[i]
}
}
return ""
}
// read is a copy of what a bucket holds now.
func (w *world) read(bucket string) map[string][]byte {
w.mu.Lock()
defer w.mu.Unlock()
out := make(map[string][]byte, len(w.values[bucket]))
for k, v := range w.values[bucket] {
out[k] = v
}
return out
}
// getter reads one key's value; errNotThere when the key does not exist.
type getter func(ctx context.Context, key string) ([]byte, error)
var errNotThere = errors.New("no such key")
// kvGetter reads keys of a bucket.
func kvGetter(kv jetstream.KeyValue) getter {
return func(ctx context.Context, key string) ([]byte, error) {
e, err := kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, errNotThere
}
if err != nil {
return nil, err
}
return e.Value(), nil
}
}
// manifest reads the manifest of a digest from the indexer's bucket once and keeps it: a digest's manifest never changes.
func (w *world) manifest(ctx context.Context, get getter, digest string) parsed {
w.mu.Lock()
p, ok := w.manifests[digest]
w.mu.Unlock()
if ok {
return p
}
key, err := delivery.ManifestsKey(digest)
if err != nil {
return parsed{err: err.Error()}
}
value, err := get(ctx, key)
if errors.Is(err, errNotThere) {
return parsed{err: "the indexer holds no manifest of digest " + digest + " yet"}
}
if err != nil {
return parsed{err: err.Error()}
}
m, err := delivery.DecodeManifest(value)
if err != nil {
p = parsed{err: err.Error()}
} else if man, err := catalogue.Decode([]byte(m.JSON)); err != nil {
// Decoded, never judged (novox/hq ADR 0578 §3): the indexer's judge judged it, and its verdict is taken from the
// module's value where it is read (world.mesh).
p = parsed{module: m.Module, err: "it does not decode: " + err.Error()}
} else {
p = parsed{module: m.Module, manifest: man}
}
w.mu.Lock()
w.manifests[digest] = p
w.mu.Unlock()
return p
}
// assigned is each node's modules the seat mesh-placement's placement names, in name order (ADR 0549 §5, ADR 0567 §8): the
// placement is the one source of where modules run. The controller's assignments are read no more (issue 774).
func (w *world) assigned() map[string][]string {
placed, _ := w.placed()
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
return out
}
// placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key
// that does not read places nothing on its node and is said by the indexer's own read tool.
func (w *world) placed() (map[string][]string, bool) {
out := map[string][]string{}
for node, p := range w.placements() {
out[node] = p.Assigned()
}
return out, len(w.read(delivery.PlacementBucket)) > 0
}
// placements is every node's placement, assembled from its keys in either form (delivery.PlacementKeys).
func (w *world) placements() map[string]delivery.Placement {
out, _ := w.placement.AllRevisions()
return out
}
// placedNodes is every node the seat's placement names (ADR 0567 §1): those compose their own modules (ADR 0547 §2),
// so their wants name the build alone and the seventh rule does not hold them.
func (w *world) placedNodes() map[string]bool {
placed, _ := w.placed()
out := make(map[string]bool, len(placed))
for node := range placed {
out[node] = true
}
return out
}
// trunk is every module the indexer holds at the trunk, with its value.
func (w *world) trunk() map[string]delivery.Module {
out := map[string]delivery.Module{}
for key, value := range w.read(delivery.ModulesBucket) {
name, err := delivery.ParseModulesKey(key)
if err != nil {
continue
}
if m, err := delivery.DecodeModule(value); err == nil {
out[name] = m
}
}
return out
}
// records is each node's record in its own have key (`<node>.node`): the nodes a placement proposal is answered by,
// with those placement names (ADR 0593 §8).
func (w *world) records() map[string]delivery.NodeRecord {
out := map[string]delivery.NodeRecord{}
for key, value := range w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err != nil || !isNode {
continue
}
if h, err := delivery.DecodeHaveNode(value); err == nil && h.Record != nil {
out[node] = *h.Record
}
}
return out
}
// online is, by node, whether the watchman holds the node online.
func (w *world) online() map[string]bool {
out := map[string]bool{}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
continue
}
if p, err := delivery.DecodePresence(value); err == nil {
out[node] = p.Online
}
}
return out
}
// wantsOf is the wants the assigner wrote, by module then node.
func (w *world) wantsOf() map[string]map[string]delivery.Want {
out := map[string]map[string]delivery.Want{}
for key, value := range w.read(delivery.AssignerWantsBucket) {
node, module, err := delivery.ParseWantKey(key)
if err != nil {
continue
}
want, err := delivery.DecodeWant(value)
if err != nil {
continue
}
if out[module] == nil {
out[module] = map[string]delivery.Want{}
}
out[module][node] = want
}
return out
}
// nodes is what the six rules read of the nodes: presence, haves, and every provision's providers with their nodes.
func (w *world) nodes(manifests map[string]catalogue.Manifest, assigned map[string][]string) assign.Nodes {
n := assign.Nodes{Online: map[string]bool{}, Haves: map[string]map[string]delivery.Have{}, Providers: map[string]map[string][]string{}}
for key, value := range w.read(delivery.PresenceBucket) {
node, _, _, err := delivery.ParsePresenceKey(key)
if err != nil || node == "" {
continue
}
if p, err := delivery.DecodePresence(value); err == nil {
n.Online[node] = p.Online
}
}
for key, value := range w.read(delivery.HaveBucket) {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
continue
}
if h, err := delivery.DecodeHave(value); err == nil {
if n.Haves[node] == nil {
n.Haves[node] = map[string]delivery.Have{}
}
n.Haves[node][module] = h
}
}
nodesOf := map[string][]string{}
for node, modules := range assigned {
for _, m := range modules {
nodesOf[m] = append(nodesOf[m], node)
}
}
for name, man := range manifests {
for _, o := range man.Provides {
if n.Providers[o.Name] == nil {
n.Providers[o.Name] = map[string][]string{}
}
n.Providers[o.Name][name] = nodesOf[name]
}
}
return n
}
// value is what a bucket holds at a key now, nil when nothing.
func (w *world) value(bucket, key string) []byte {
w.mu.Lock()
defer w.mu.Unlock()
return w.values[bucket][key]
}
// grantsBucket is the bucket the grants are read from: the grants part's state `grants`, always (ADR 0571 clause 9); no
// manifest but the controller's reads the controller's grants.
func (w *world) grantsBucket() string { return delivery.PartGrantsBucket }
// has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool {
w.mu.Lock()
defer w.mu.Unlock()
_, ok := w.values[bucket][key]
return ok
}
// built says whether the builder's build of a module at a fingerprint is done.
func (w *world) built(module, fp string) bool {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return false
}
w.mu.Lock()
value, ok := w.values[delivery.BuilderBuildsBucket][key]
w.mu.Unlock()
if !ok {
return false
}
b, err := delivery.DecodeBuilderBuild(value)
return err == nil && b.State == delivery.BuildDone
}
// composed is the composition of module for node: a placed node composes its own, and its composition is the digest its own
// have names (ADR 0571 §6); an unplaced node has none, so it gets no want (and the pass says so, unplacedFindings). The
// controller's copy (mesh-controller_composed) is read no more (issue 774).
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
if !w.placedNodes()[node] {
return delivery.ComposedDigest{}, false
}
return w.ownComposed(node)
}
// answer is the teller's answer to the ask at key (mesh-delivery-teller_answers, the same key), and whether there is one.
func (w *world) answer(key string) (delivery.Answer, bool) {
w.mu.Lock()
value, ok := w.values[delivery.AnswersBucket][key]
w.mu.Unlock()
if !ok {
return delivery.Answer{}, false
}
a, err := delivery.DecodeAnswer(value)
return a, err == nil
}
// bucketGen is how many times any key of the bucket was written or deleted in this process: a read of what it shows
// elsewhere is asked again only after it moves.
func (w *world) bucketGen(bucket string) uint64 {
w.mu.Lock()
defer w.mu.Unlock()
return w.changes[bucket]
}
// changed is how many values the key has taken in this process (see world.changes).
func (w *world) changed(bucket, key string) uint64 {
w.mu.Lock()
defer w.mu.Unlock()
return w.changes[bucket+"|"+key]
}
// ownComposed is a placed node's own composition, as the digest its own have names (`<node>.node` of the haves, Composed):
// what rule 7 reads. A node whose have names none is not composed yet.
func (w *world) ownComposed(node string) (delivery.ComposedDigest, bool) {
key, err := delivery.HaveNodeKey(node)
if err != nil {
return delivery.ComposedDigest{}, false
}
raw, ok := w.read(delivery.HaveBucket)[key]
if !ok {
return delivery.ComposedDigest{}, false
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil || h.Composed == "" {
return delivery.ComposedDigest{}, false
}
return delivery.ComposedDigest{V: delivery.Version, Digest: h.Composed, Judgement: delivery.JudgedNeither,
Words: "the node composes its own"}, true
}
// unplacedNodes are the nodes whose engine has a have and that no placement names: they get no
// want, since nothing composes them.
func (w *world) unplacedNodes() []string {
placed := w.placedNodes()
var out []string
for key := range w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err == nil && isNode && !placed[node] {
out = append(out, node)
}
}
sort.Strings(out)
return out
}
@@ -0,0 +1,80 @@
package main
import (
"context"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// fakeManifests is the indexer's manifests bucket in memory.
type fakeManifests map[string][]byte
func (f fakeManifests) add(t *testing.T, module, body string) string {
t.Helper()
digest := delivery.ManifestDigest([]byte(body))
value, err := delivery.EncodeManifest(delivery.Manifest{Module: module, JSON: body})
if err != nil {
t.Fatal(err)
}
f[digest] = value
return digest
}
func (f fakeManifests) get(_ context.Context, key string) ([]byte, error) {
if v, ok := f[key]; ok {
return v, nil
}
return nil, errNotThere
}
func fp(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
func pull(repo string, number int, modules map[string]delivery.CandidateModule) delivery.CandidatePull {
p := delivery.CandidatePull{Head: delivery.Head{Repository: repo, Number: number, Head: strings.Repeat("3", 40)}, Fps: modules}
for m := range modules {
p.Modules = append(p.Modules, m)
}
slices.Sort(p.Modules)
return p
}
// Review round 1 (items 3 and 4): the world counts the values a key takes, so a have written again after an answer is seen.
func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
w := newWorld()
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.web", []byte(`{"v":1}`))
w.put(delivery.HaveBucket, "ace.mail", []byte(`{"v":1}`))
if got := w.changed(delivery.HaveBucket, "ace.web"); got != 2 {
t.Errorf("changed = %d, want 2", got)
}
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
// ADR 0571 clause 9: grants are read from the grants part's bucket, the one bucket of grants (ADR 0590).
fp := delivery.SourcePrefix2 + strings.Repeat("d", 64)
gkey, _ := delivery.PartGrantKey("web", fp)
grant, _ := delivery.EncodeGrant(delivery.Grant{Accounts: []string{"ace.web"}})
for _, row := range []struct {
name string
inPart bool
granted bool
}{
{"the part's grant", true, true},
{"no grant", false, false},
} {
p := newPasser(buckets{}, time.Now())
if row.inPart {
p.w.put(delivery.PartGrantsBucket, gkey, grant)
}
if got := p.w.grantsBucket(); got != delivery.PartGrantsBucket {
t.Errorf("%s: grants are read from %s, want the part's", row.name, got)
}
if _, granted := p.composedFor("ace", "web", fp); granted != row.granted {
t.Errorf("%s: granted = %v, want %v", row.name, granted, row.granted)
}
}
}
+22
View File
@@ -0,0 +1,22 @@
module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
+27
View File
@@ -0,0 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929 h1:RtAFbUJRP1qO9AIMM2ue+cXNzqj7WZ7IOlSqNvttwuU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,240 @@
package assign
import (
"fmt"
"regexp"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HoldAsk is the ask about a module's fingerprint the assigner holds, at its key (delivery.HoldAskKey): keep it held or
// release it. Releasing writes the held fingerprint's want again by the rules (ADR 0539 §19). node and words are the held
// record's: the machine whose health check failed and the hold's words (issue 1906). asking numbers the writing: 0 the first, a
// larger number an ask written again after a hold lifted by itself, so the router takes it as new.
func HoldAsk(key, module, fp, node, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp)),
Explanation: fmt.Sprintf("The health check of %s failed on %s at %s (%s), so the assigner holds that version and wants "+
"no node to move to it. It lifts the hold by itself once every node that runs %s runs it healthy; releasing it "+
"lets the wants move to it now.", module, node, short(fp), words, module),
Options: []asks.Option{
{ID: "keep", Label: "Keep it held", Does: "the version stays held", Level: asks.Acknowledge},
{ID: "release", Label: "Release it", Does: "the version is released and the wants move to it by the rules again", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the version stays held",
About: key,
}}
}
// The options of the restore ask (ADR 0537 §3, ADR 0539 §19), and what each does on the node.
const (
OptionRestore = "restore"
OptionLeaveStopped = "leave-stopped"
)
// RestoreAsk is the ask about a module holding data whose fingerprint failed its health check on a node (ADR 0429 §6): it is
// not set back by itself, since a set-back would lose the writes made since. words is the check's words. Restore sets the
// node back to the last healthy fingerprint and loses the newer writes; leaving it stopped keeps the data as it is.
func RestoreAsk(key, node, module, fp, words string, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s failed its health check on %s", module, node),
Explanation: fmt.Sprintf("%s holds data, and its health check failed on %s at %s: %s. Setting it back to the version that ran healthy "+
"would lose the writes made since; leaving it stopped keeps the data as it is.", module, node, short(fp), words),
Options: []asks.Option{
{ID: OptionRestore, Label: "Restore it", Does: "the node is set back to the last healthy version and the newer writes are lost", Level: asks.Acknowledge},
{ID: OptionLeaveStopped, Label: "Leave it stopped", Does: "the module stays as it is on the node and the data is kept", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module stays as it is on the node",
About: key,
}}
}
// HeldCondition is the condition the assigner raises when it holds a fingerprint of a module (ADR 0429 §6, ADR 0539 §19), or
// clears (raised false) when the hold is released. It names the module, the node whose health check failed and the check's
// words, under the assigner's own name (ADR 0437 clause 9), keyed by the held fingerprint. needsYou is true for a module holding
// data, whose restore ask waits for the operator.
func HeldCondition(module, fp, node, words string, raised, needsYou bool) asks.Condition {
key := delivery.AssignerModule + ".held." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
c := asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: fmt.Sprintf("%s is held at %s", module, short(fp)),
Headline: fmt.Sprintf("%s is held at %s", module, short(fp))}
what := fmt.Sprintf("The health check of %s failed on %s at %s: %s. The assigner holds that version and wants no node to move to it.", module, node, short(fp), words)
if needsYou {
c.Needs = "answer whether to restore " + module + " on " + node
c.Explanation = asks.NeedsYou + c.Needs + ". " + what
} else {
c.Explanation = asks.NothingToDo + " " + what + " Nodes that wanted it are set back to the version that ran healthy."
}
return c
}
// NodeFailedCondition is the condition the assigner raises for a module whose newest fingerprint fp failed its health check on
// node while other nodes run it healthy (ADR 0557 §1, issue 1496): the failure is the node's, so nothing is held or set back and
// the node keeps its want. It names the node, the module, fp and the have's words, keyed by node and module, and is cleared
// (raised false) once the node no longer fails alone.
func NodeFailedCondition(node, module, fp string, f NodeFailure, raised bool) asks.Condition {
key := delivery.AssignerModule + ".failed-alone." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s no longer fails its health check alone.", module, node)}
}
headline := fmt.Sprintf("%s fails on %s alone", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s The health check of %s at %s failed on %s (%s), while it runs healthy on %s. The failure is %s's, "+
"so the assigner holds nothing and sets no node back, and %s keeps its want. This clears once %s runs it healthy or "+
"another version is the newest.", asks.NothingToDo, module, short(fp), node, f.Words, strings.Join(f.Others, ", "), node,
node, node)}
}
// NeitherCondition is the condition of a module assigned to node that holds no want for it (issue 1945): the node's own composition
// gives up every module with a fingerprint of the new form, so nothing applies it there and its files are gone. It is raised (urgent)
// naming the module, the node and the rule holding its first want back with the rule's words, under the assigner's own name keyed by
// node and module, and cleared (raised false) when the node has a want for it again.
func NeitherCondition(node, module string, h Holding, raised bool) asks.Condition {
key := delivery.AssignerModule + ".neither." + node + "." + module
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared,
Resolved: fmt.Sprintf("%s on %s has a want again, so its own composition is applied there.", module, node)}
}
headline := fmt.Sprintf("%s on %s is on no list", module, node)
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = string([]rune(headline)[:asks.HeadlineLength])
}
return asks.Condition{Key: key, Severity: asks.SeverityUrgent, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline,
Explanation: fmt.Sprintf("%s %s is assigned to %s, but the node holds no want for it and the node's own composition does not "+
"hold it either, so nothing applies it there. The assigner's rule %d holds its first want back: %s. The assigner writes the want once the "+
"rule holds, and clears this.", asks.NothingToDo, module, node, h.Rule, h.Words)}
}
// NeitherLeftCondition clears the neither condition of module on node when it is no longer on neither list: why is "" when the
// node has a live want for it again, else why it has none (it is held, its move waits for an ask, the node no longer runs it), so
// the clear never says a want the node does not hold (issue 1945: "nats on novox has a want again" while it had none).
func NeitherLeftCondition(node, module, why string) asks.Condition {
c := NeitherCondition(node, module, Holding{}, false)
if why != "" {
c.Resolved = fmt.Sprintf("%s on %s is no longer on neither list and has no want: %s.", module, node, why)
}
return c
}
// LiftedCondition clears the condition of a held fingerprint the assigner lifted by itself, saying why (issue 1906).
func LiftedCondition(module, fp string) asks.Condition {
c := HeldCondition(module, fp, "", "", false, false)
c.Resolved = LiftedWords(module, fp) + "."
return c
}
// The options of the record ask (ADR 0539 §14) and of the backup ask (§16).
const (
OptionYes = "yes"
OptionNo = "no"
OptionBackupAgain = delivery.BackupAgain
OptionBackupSkip = delivery.BackupSkip
)
// RecordAsk is the ask about a move of a module of policy record on node to the composition with the digest, judged by the
// controller to recreate a container, to move data, or not to be judgeable (ADR 0539 §14). The want names the composition only
// after the answer yes. The way the move was judged is in the ask.
func RecordAsk(key, node, module, fp string, need RecordNeed, now time.Time) delivery.AssignerAsk {
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + need.Words,
}[need.Judgement]
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, 0),
Headline: fmt.Sprintf("%s waits for your yes on %s", module, node),
Explanation: fmt.Sprintf("%s is recorded: a person takes each of its moves. The next one on %s (version %s) %s. "+
"The node is told to apply it only after you say yes.", module, node, short(fp), what),
Options: []asks.Option{
{ID: OptionYes, Label: "Yes, move it", Does: "the node is told to apply the new composition of " + module, Level: asks.Acknowledge},
{ID: OptionNo, Label: "No, not now", Does: "the node keeps what it runs", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the node keeps what it runs",
About: key,
}}
}
// NextBackupAsking is the number of the asking that follows the one numbered after (a negative number: none yet). Numbers 0 and 1
// give one id (delivery.AskingID), so the first asking is 0 and the next 2.
func NextBackupAsking(after int) int {
if after < 0 {
return 0
}
return max(after+1, 2)
}
var backupDigest = regexp.MustCompile(`\(composition (sha256:[0-9a-f]{64})\)`)
// BackupAskDigest is the digest of the composition a backup ask was made about, from its explanation; empty for an ask that names
// none. An answer is taken for the composition of the ask it answers, and a new revision at the same fingerprint is asked again
// under a new number (ADR 0539 §16).
func BackupAskDigest(a delivery.AssignerAsk) string {
if m := backupDigest.FindStringSubmatch(a.Ask.Explanation); m != nil {
return m[1]
}
return ""
}
// BackupAnswerWords are the words of a want carrying the answer to the asking-th ask: each asking's answer is another want
// value, so a backup that fails again after "back up again" and is answered "again" again changes the key and the node retries.
func BackupAnswerWords(option string, asking int) string {
return fmt.Sprintf("the operator answered %q to the backup ask (asking %d)", option, asking+1)
}
// BackupAsk is the ask about a backup of a module on node that failed before an apply (ADR 0539 §16): back up again, or go on
// without a backup. digest is the composition about to be applied and asking the number of the asking, 0 the first; words are the
// backup's own words, from the node's have.
func BackupAsk(key, node, module, fp, digest, words string, asking int, now time.Time) delivery.AssignerAsk {
return delivery.AssignerAsk{Asking: asking, Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: fmt.Sprintf("The backup of %s failed on %s", module, node),
Explanation: fmt.Sprintf("The backup before applying %s at %s on %s failed: %s. %s keeps running what it has until you answer. (composition %s)",
module, short(fp), node, words, module, digest),
Options: []asks.Option{
{ID: OptionBackupAgain, Label: "Back up again", Does: "the node tries the backup again before it applies", Level: asks.Acknowledge},
{ID: OptionBackupSkip, Label: "Go on without a backup", Does: "the node applies the new version without a backup", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(7 * 24 * time.Hour),
OnExpiry: "the module keeps running what it has",
About: key,
}}
}
// BackupCondition is the condition raised when a backup before an apply fails and the module is held on the node (ADR 0539 §16),
// naming the node, the module and the backup's words; cleared under the same key.
func BackupCondition(node, module, fp, words string, raised bool) asks.Condition {
key := delivery.AssignerModule + ".backup." + node + "." + module + "." + strings.TrimPrefix(fp, delivery.SourcePrefix2)
subject := asks.Scope{Scope: "module", ID: module, Machine: node}
if !raised {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared}
}
headline := fmt.Sprintf("The backup of %s failed on %s", module, node)
needs := "answer whether to back up again or go on without a backup"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: headline, Headline: headline, Needs: needs,
Explanation: asks.NeedsYou + needs + ". The backup before applying " + module + " at " + short(fp) + " on " + node + " failed: " + words + "."}
}
@@ -0,0 +1,254 @@
package assign
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at
// its key, of a kind the view may answer.
func TestTheHoldAskIsValid(t *testing.T) {
key, err := delivery.HoldAskKey("mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := HoldAsk(key, "mail", fNew, "ace", "health failed, phase running: connection refused", 0, now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
// 1906/H3: the ask names the machine whose check failed and the hold's words.
if !strings.Contains(a.Ask.Explanation, "on ace") || !strings.Contains(a.Ask.Explanation, "connection refused") {
t.Errorf("the ask does not name the node and the hold's words: %q", a.Ask.Explanation)
}
// 1906/H2: an ask written again after a lift is another asking, under an id of its own.
again := HoldAsk(key, "mail", fNew, "ace", "x", 7, now)
if again.Asking != 7 || again.Ask.ID != delivery.AskingID(key, 7) || again.Ask.ID == a.Ask.ID {
t.Errorf("the asking after a lift = %d %s", again.Asking, again.Ask.ID)
}
if _, err := delivery.EncodeAssignerAsk(key, again); err != nil {
t.Errorf("the ask asked again cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
}
// 1906/D3 (ADR 0539 §19): the ask about a data module held on a node is a valid ask of the router and of mesh-sdk's layout at
// its restore key, of a kind the view may answer; it offers restoring (losing the newer writes) and leaving it stopped.
func TestTheRestoreAskIsValid(t *testing.T) {
key, err := delivery.RestoreAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
now := time.Now()
a := RestoreAsk(key, "ace", "mail", fNew, "connection refused", now)
if err := a.Ask.Check(now); err != nil {
t.Errorf("the router refuses the ask: %v", err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("the ask cannot be written: %v", err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("the view may not answer it: %v", err)
}
var ids []string
for _, o := range a.Ask.Options {
ids = append(ids, o.ID)
}
if len(ids) != 2 || ids[0] != OptionRestore || ids[1] != OptionLeaveStopped {
t.Errorf("options = %v", ids)
}
if !strings.Contains(a.Ask.Explanation, "connection refused") || !strings.Contains(a.Ask.Explanation, "ace") {
t.Errorf("the ask does not name the node and the check's words: %q", a.Ask.Explanation)
}
}
// 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own
// name (ADR 0437 clause 9), and is cleared under the same key.
func TestTheAssignersConditions(t *testing.T) {
t.Run("the held condition names module, node and words", heldConditionNamesModuleNodeAndWords)
t.Run("a lifted hold clears its condition saying why", liftedHoldClearsItsConditionSayingWhy)
t.Run("1952/WM6: each said condition is a finding, its clear deletes it, said straight only with no findings bucket", saidConditionsAreFindings)
}
func heldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Subject.ID != "mail" || c.Subject.Machine != "ace" || !strings.Contains(c.Explanation, "connection refused") || !strings.Contains(c.Key, "mail") {
t.Errorf("condition = %+v", c)
}
if got := HeldCondition("mail", fNew, "", "", false, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func liftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateCleared || c.Key != HeldCondition("mail", fNew, "ace", "x", true, false).Key || !strings.Contains(c.Resolved, "lifted its hold by itself") {
t.Errorf("condition = %+v", c)
}
}
// 1952/WM6 (ADR 0564 §2, issue 1952 note 16): every condition the assigner says is a finding in its own bucket, under
// `<scope>.<id>.<kind>`, written at the raise and deleted at the clear, and only as a finding: the keeper is the one
// source since the controller's end (novox/hq ADR 0590 §3, issue 1126 B8). Said straight to the router only where the
// findings bucket cannot be written to at all.
func saidConditionsAreFindings(t *testing.T) {
at := time.Date(2026, 10, 29, 21, 0, 0, 0, time.UTC)
for _, c := range []struct {
raised, cleared asks.Condition
key string
}{
{HeldCondition("mail", fNew, "ace", "refused", true, true), LiftedCondition("mail", fNew), "module.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".held"},
{NodeFailedCondition("ace", "mail", fNew, NodeFailure{Words: "x", Others: []string{"g14"}}, true), NodeFailedCondition("ace", "mail", fNew, NodeFailure{}, false), "module.ace.mail.failed-alone"},
{NeitherCondition("ace", "mail", Holding{Rule: 3, Words: "w"}, true), NeitherCondition("ace", "mail", Holding{}, false), "module.ace.mail.neither"},
{SeatWaitsCondition([]string{"placement"}), SeatWaitsCondition(nil), "module.mesh-delivery-assigner.seat-states"},
{BackupCondition("ace", "mail", fNew, "disk full", true), BackupCondition("ace", "mail", fNew, "", false), "module.ace.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".backup"},
} {
puts, deletes, said := map[string]health.Finding{}, []string{}, 0
f := Findings{Put: func(k string, v []byte) error {
got, err := health.DecodeFinding(v)
puts[k] = got
return err
}, Delete: func(k string) error { deletes = append(deletes, k); return nil }, Now: func() time.Time { return at }}
say := f.Say(func(asks.Condition) error { said++; return nil })
if err := say(c.raised); err != nil {
t.Fatalf("%s: %v", c.key, err)
}
got, ok := puts[c.key]
if !ok || got.Severity != health.Severity(c.raised.Severity) || got.Headline != c.raised.Headline || got.Observer != delivery.AssignerModule ||
got.Subject.Machine != c.raised.Subject.Machine || said != 0 {
t.Fatalf("%s: finding %+v (written %v), said %d", c.key, got, puts, said)
}
if err := say(c.cleared); err != nil || !slices.Equal(deletes, []string{c.key}) || said != 0 {
t.Fatalf("%s: deleted %v, said %d (%v)", c.key, deletes, said, err)
}
if err := (Findings{}).Say(func(asks.Condition) error { said++; return nil })(c.raised); err != nil || said != 1 {
t.Fatalf("%s: with no findings bucket said %d (%v), want straight once", c.key, said, err)
}
}
if _, _, _, err := FindingOf(asks.Condition{Key: "another.x", State: asks.StateRaised}, at); err == nil {
t.Error("a condition not the assigner's is a finding")
}
}
// 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in
// the view, and name the move and the node.
func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
now := time.Now()
rkey, err := delivery.RecordAskKey("ace", "mail", "sha256:"+strings.Repeat("b", 64))
if err != nil {
t.Fatal(err)
}
r := RecordAsk(rkey, "ace", "mail", fNew, RecordNeed{Node: "ace", Digest: "sha256:" + strings.Repeat("b", 64), Judgement: delivery.JudgedCannot, Words: "the composition it runs is gone"}, now)
bkey, err := delivery.BackupAskKey("ace", "mail", fNew)
if err != nil {
t.Fatal(err)
}
dgst := "sha256:" + strings.Repeat("b", 64)
b := BackupAsk(bkey, "ace", "mail", fNew, dgst, "no space left", 0, now)
for key, a := range map[string]delivery.AssignerAsk{rkey: r, bkey: b} {
if err := a.Ask.Check(now); err != nil {
t.Errorf("%s: the router refuses the ask: %v", key, err)
}
if _, err := delivery.EncodeAssignerAsk(key, a); err != nil {
t.Errorf("%s: cannot be written: %v", key, err)
}
if _, err := delivery.ViewAnswerKey(key); err != nil {
t.Errorf("%s: the view may not answer it: %v", key, err)
}
}
if !strings.Contains(r.Ask.Explanation, "gone") || r.Ask.Options[0].ID != OptionYes {
t.Errorf("the record ask = %+v", r.Ask)
}
if !strings.Contains(b.Ask.Explanation, "no space left") || b.Ask.Options[0].ID != delivery.BackupAgain || b.Ask.Options[1].ID != delivery.BackupSkip {
t.Errorf("the backup ask = %+v", b.Ask)
}
// the id binds the ask to the composition and to its number (review round 1, items 3 and 4)
if again := BackupAsk(bkey, "ace", "mail", fNew, dgst, "x", NextBackupAsking(0), now); again.Ask.ID == b.Ask.ID {
t.Errorf("asked again under the same id")
}
if NextBackupAsking(-1) != 0 || NextBackupAsking(0) != 2 || NextBackupAsking(2) != 3 {
t.Errorf("the numbers of the askings: %d %d %d", NextBackupAsking(-1), NextBackupAsking(0), NextBackupAsking(2))
}
if BackupAskDigest(b) != dgst {
t.Errorf("the ask names composition %q, want %q", BackupAskDigest(b), dgst)
}
if BackupAnswerWords("again", 0) == BackupAnswerWords("again", 1) {
t.Errorf("two answers of the same option are one want value")
}
c := BackupCondition("ace", "mail", fNew, "no space left", true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Errorf("the backup condition is refused: %v", err)
}
// ADR 0546 clauses 7 and 8: a put's ask is valid at the record ask's key; for the bus its yes approves and binds the act,
// so the view cannot answer it, and it lasts no longer than an approving ask may.
put := delivery.Put{FP: fNew, Why: "the bus's user list", Hold: true}
d := delivery.ComposedDigest{Digest: dgst, Judgement: delivery.JudgedCannot, Words: "the have names no composition"}
for _, bus := range []bool{false, true} {
a, err := PutAsk(rkey, "ace", "mail", put, d, bus, now)
if err != nil {
t.Fatalf("bus %v: %v", bus, err)
}
if err := a.Ask.Check(now); err != nil {
t.Errorf("bus %v: the router refuses the put's ask: %v", bus, err)
}
if _, err := delivery.EncodeAssignerAsk(rkey, a); err != nil {
t.Errorf("bus %v: the put's ask cannot be written: %v", bus, err)
}
viewed := delivery.ViewAnswerFits(delivery.ViewAnswer{Asking: a.Ask.ID, Option: OptionYes}, a)
yes := a.Ask.Options[0]
switch {
case !strings.Contains(a.Ask.Explanation, "the bus's user list"):
t.Errorf("bus %v: the put's ask does not say why: %s", bus, a.Ask.Explanation)
case bus && (yes.Level != asks.Approve || yes.Binds == "" || viewed == nil || a.Ask.Expires.After(now.Add(asks.ApproveLasts))):
t.Errorf("bus: the yes is %+v, the view's answer %v, expiring %s", yes, viewed, a.Ask.Expires)
case !bus && (yes.Level != asks.Acknowledge || viewed != nil):
t.Errorf("a recorded module: the yes is %+v, the view's answer %v", yes, viewed)
}
later := put
later.At = now.Add(time.Minute)
b, _ := PutAsk(rkey, "ace", "mail", later, d, bus, now)
if b.Ask.ID == a.Ask.ID || b.Asking != PutAsking(later) {
t.Errorf("bus %v: a later put's ask is asking %d under id %q, the earlier's %q", bus, b.Asking, b.Ask.ID, a.Ask.ID)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dgst)); err != nil {
t.Errorf("bus %v: the yes does not perform the act it bound: %v", bus, err)
}
if err := yes.Performs(PutAct("ace", "mail", fNew, dig(fOld))); bus && err == nil {
t.Errorf("bus: the yes performs another composition's act")
}
}
}
// 1945/AC21: the condition of a module on neither list is urgent, names the module, the node and the rule holding its want back,
// under the assigner's own name keyed by node and module, and is cleared under the same key.
func TestTheNeitherConditionNamesModuleNodeAndRule(t *testing.T) {
c := NeitherCondition("g14", "i3status-rust", Holding{Rule: delivery.RuleProviders, Words: "no provider of x11-display runs healthy on g14 or at its canary"}, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
}
if c.State != asks.StateRaised || c.Severity != asks.SeverityUrgent || c.Subject.ID != "i3status-rust" || c.Subject.Machine != "g14" ||
!strings.Contains(c.Explanation, "x11-display") || !strings.Contains(c.Explanation, "rule 5") || c.Key != "mesh-delivery-assigner.neither.g14.i3status-rust" {
t.Errorf("condition = %+v", c)
}
if got := NeitherCondition("g14", "i3status-rust", Holding{}, false); got.State != asks.StateCleared || got.Key != c.Key {
t.Errorf("cleared = %+v", got)
}
}
@@ -0,0 +1,104 @@
package assign
import (
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// The assigner's said conditions become findings (novox/hq ADR 0564 §2, issue 1952 WM6, note 16): every condition it says
// under its own name (ADR 0437 §9) — a hold and its lift, a node failing alone, a module on neither list, a backup, a pair
// placed by one source, the seat waiting for its states — is also a finding in its own bucket `findings`, written at its
// raise and deleted at its clear, which the condition keeper raises and clears. Until the operator's take-over the router
// hears it straight from the assigner, as before; from the take-over the keeper's condition is the one said, so the
// assigner no longer says it straight (never both).
// FindingsBucket is the assigner's findings.
var FindingsBucket = health.FindingsBucket(delivery.AssignerModule)
// FindingOf is a said condition as a finding: its key `<scope>.<id>.<kind>`, from the condition's own key
// `mesh-delivery-assigner.<kind>.<id…>` (the subject's id when the key names none), its words unchanged. raised is false for
// a clear, whose key is to be deleted; value is nil then.
func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised bool, err error) {
rest, ok := strings.CutPrefix(c.Key, delivery.AssignerModule+".")
if !ok || rest == "" {
return "", nil, false, fmt.Errorf("the condition %q is not the assigner's", c.Key)
}
kind, id, _ := strings.Cut(rest, ".")
if id == "" {
id = c.Subject.ID
}
scope := c.Subject.Scope
if scope == "" {
scope = health.ScopeModule
}
key = health.FindingKey(scope, id, kind)
if c.State == asks.StateCleared {
return key, nil, false, nil
}
if c.State != asks.StateRaised {
return "", nil, false, fmt.Errorf("the condition %s is neither raised nor cleared", c.Key)
}
summary := c.Summary
if summary == "" {
summary = c.Headline
}
var actions []health.Action
for _, a := range c.Actions {
actions = append(actions, health.Action{Label: a.Label, Verb: a.Verb, Machine: a.Machine, Arguments: a.Arguments, Level: string(a.Level)})
}
value, err = health.EncodeFinding(health.Finding{Key: key, Kind: kind, Subject: health.Subject{Scope: scope, ID: id, Machine: c.Subject.Machine},
Severity: health.Severity(c.Severity), Summary: summary, Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved,
Needs: c.Needs, Actions: actions, Evidence: []health.Evidence{{At: at.UTC(), Said: summary}}, Source: "assigner." + kind,
Observer: delivery.AssignerModule, At: at.UTC()})
return key, value, true, err
}
// Findings is where the assigner's findings go: the bucket's put and delete.
type Findings struct {
Put func(key string, value []byte) error
Delete func(key string) error
Keys func() ([]string, error)
Now func() time.Time
}
// Say is say with each condition written as a finding (or its finding deleted), which the condition keeper carries, the
// one source since the controller's end (novox/hq ADR 0590 §3); said straight to the router only where there is no
// findings bucket at all. A finding that cannot be written is the error.
func (f Findings) Say(say func(asks.Condition) error) func(asks.Condition) error {
return func(c asks.Condition) error {
var errs []error
if f.Put != nil {
now := time.Now
if f.Now != nil {
now = f.Now
}
key, value, raised, err := FindingOf(c, now())
switch {
case err != nil:
errs = append(errs, err)
case raised:
if err := f.Put(key, value); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be written: %w", key, err))
}
default:
if err := f.Delete(key); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be deleted: %w", key, err))
}
}
}
if f.Put == nil {
if say != nil {
if err := say(c); err != nil {
errs = append(errs, err)
}
}
}
return errors.Join(errs...)
}
}
@@ -0,0 +1,22 @@
package assign
import (
"strings"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// SeatWaitsCondition is the assigner's word on the seat mesh-placement while a state of the seat is not bound (ADR 0567
// §1): raised naming each, cleared when none is left.
func SeatWaitsCondition(unbound []string) asks.Condition {
key := delivery.AssignerModule + ".seat-states"
subject := asks.Scope{Scope: "module", ID: delivery.AssignerModule}
if len(unbound) == 0 {
return asks.Condition{Key: key, Subject: subject, State: asks.StateCleared, Resolved: "the seat mesh-placement holds all its states."}
}
why := "the seat waits for its states: " + strings.Join(unbound, ", ") + " are not on the bus or not granted yet"
return asks.Condition{Key: key, Severity: asks.SeverityWarning, Subject: subject, State: asks.StateRaised,
Summary: "the seat waits for its states", Headline: "the seat waits for its states",
Explanation: asks.NothingToDo + " " + why + "; each is bound at the next pass once it is, with no restart."}
}
@@ -0,0 +1,135 @@
package assign
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// A person's upgrade policy is the assigner's (novox/hq ADR 0565 §7, §8, issue 1953 D4, D5): kept in its state `policies`, one
// key per module, set and taken back through `mesh-wants.upgrade`. Every reader takes one derivation, mesh-sdk's
// catalogue.Upgrade: a person's choice over the module's word, its irreplaceable data and the default, the bus as any module (ADR 0601 §1).
// PoliciesBucket is the assigner's state `policies`.
var PoliciesBucket = delivery.AssignerModule + "_" + delivery.AssignerPoliciesState
// DecodePolicies reads the policies state's values by key (the module); a value that cannot be read is left out, named.
func DecodePolicies(values map[string][]byte) (map[string]catalogue.PersonPolicy, []string) {
out := map[string]catalogue.PersonPolicy{}
var bad []string
for key, raw := range values {
var p catalogue.PersonPolicy
if err := json.Unmarshal(raw, &p); err != nil || p.V != catalogue.PersonPolicyVersion || p.Check() != nil || p.Module != key {
bad = append(bad, key)
continue
}
out[key] = p
}
sort.Strings(bad)
return out, bad
}
// PolicyOf is a module's policy and where it comes from, with the person's choice for it when there is one.
func PolicyOf(m catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) (policy, from, why, by string) {
var c *catalogue.PersonPolicy
if p, ok := chosen[m.Module]; ok {
c = &p
}
return catalogue.Upgrade(m, c)
}
func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
// KindProvisionFailed is the assigner's finding of a provision its module's want waits for that failed (ADR 0570 §4).
const KindProvisionFailed = "provision-failed"
// ProvisionFailedKey is a failed provision's finding's key, `provider.<node>.<consumer>.<provision>[.<local>].provision-failed`.
func ProvisionFailedKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionFailed)
}
// ProvisionFailedFinding is the finding of a failed provision, naming its consumer, its provider and why.
func ProvisionFailedFinding(key string, p provisions.Provision, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s for %s on %s failed at %s on %s: %s, so %s is not moved there until it is provided", p.Provision,
p.Consumer.Module, p.Consumer.Node, p.Provider.Module, p.Provider.Node, p.Why, p.Consumer.Module)
return health.EncodeFinding(health.Finding{Key: ProvisionFailedKey(key), Kind: KindProvisionFailed,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: p.Consumer.Node, Also: also(p.Provider.Node, p.Consumer.Node)},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(p.Consumer.Module + " waits for a provision that failed"),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provider provides it.",
Resolved: p.Provision + " for " + p.Consumer.Module + " is provided", Evidence: []health.Evidence{{At: at.UTC(), Said: p.Why}},
Source: "assigner." + KindProvisionFailed, Observer: delivery.AssignerModule, At: at.UTC()})
}
func also(provider, consumer string) []string {
if provider == "" || provider == consumer {
return nil
}
return []string{provider}
}
// KindProvisionMissing is the assigner's finding of a required provision that has had no key for longer than its bound.
const KindProvisionMissing = "provision-missing"
// ProvisionMissingKey is its key, `provider.<node>.<module>.<provision>.provision-missing`.
func ProvisionMissingKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionMissing)
}
// ProvisionMissingFinding is the finding of a required provision with no key past bound (key `<node>.<module>.<provision>`).
func ProvisionMissingFinding(key string, bound time.Duration, at time.Time) ([]byte, error) {
parts := strings.SplitN(key, ".", 3)
if len(parts) != 3 {
return nil, fmt.Errorf("%q is not <node>.<module>.<provision>", key)
}
node, module, provision := parts[0], parts[1], parts[2]
summary := fmt.Sprintf("%s on %s requires %s and mesh-provisioner has composed no key for it in %s, so %s is not moved there: "+
"no provider of it may be placed", module, node, provision, bound, module)
return health.EncodeFinding(health.Finding{Key: ProvisionMissingKey(key), Kind: KindProvisionMissing,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: node},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(module + " on " + node + " lacks " + provision),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provision has its key.",
Resolved: provision + " for " + module + " on " + node + " has its key", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindProvisionMissing, Observer: delivery.AssignerModule, At: at.UTC()})
}
// KindUnplaced is, from the lease write, a node that runs but no placement names (ADR 0571 §6).
const KindUnplaced = "unplaced"
// UnplacedKey is its finding's key, `machine.<node>.unplaced`.
func UnplacedKey(node string) string {
return health.FindingKey(health.ScopeMachine, node, KindUnplaced)
}
// UnplacedFinding is the finding of a node no placement names since the lease write.
func UnplacedFinding(node string, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s runs, but no placement names it: since the controller handed its lease away nothing composes it, "+
"so the assigner writes it no want", node)
return health.EncodeFinding(health.Finding{Key: UnplacedKey(node), Kind: KindUnplaced,
Subject: health.Subject{Scope: health.ScopeMachine, ID: node, Machine: node}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(node + " has no placement"), Needs: "place it with mesh-placement.propose.",
Explanation: asks.NeedsYou + "place it with mesh-placement.propose. " + summary + ".",
Resolved: node + " is placed", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindUnplaced, Observer: delivery.AssignerModule, At: at.UTC()})
}
func clipHeadline(s string) string {
if len([]rune(s)) <= 60 {
return s
}
return string([]rune(s)[:60])
}
@@ -0,0 +1,252 @@
package assign
import (
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A put (novox/hq ADR 0546): one version of a module put on one node by the assigner's tool assigner_put. The tool writes the put
// key; each pass takes it one step further from what the world holds (StepPut), and the rules pass over the node's want for
// the module from its asking until it ends (Module.Puts).
// ComposeWithin is how long a put waits for the composition and grant it needs (a take-in of a build just done writes them)
// before it is refused.
const ComposeWithin = 10 * time.Minute
// PutWorld is what one step of a put reads.
type PutWorld struct {
Node, Module string
Put delivery.Put
// Assigned says the module stands assigned to the node.
Assigned bool
// Bus says the module provides the mesh's bus (ADR 0539 §15); Recorded that it is of policy record (§14).
Bus, Recorded bool
// Build is the state of the builder's build at the put's fingerprint, "" when no key is written; BuildWords its error.
Build, BuildWords string
// Newest is the module's newest fingerprint at the trunk, which the builder builds by itself.
Newest string
// Composed is the composition of the module for the node at the fingerprint, as its writer holds it; nil when there is
// none. Granted says the grant at the fingerprint names the node's accounts. Both are read, never asked for (clause 4).
Composed *delivery.ComposedDigest
Granted bool
// Answer is the option the operator chose on this put's own ask (its asking, PutAsking), "" when none; AskExpired says
// that ask expired unanswered.
Answer string
AskExpired bool
// Snapshot says where the bus's streams were snapshotted for this put, once they were; Snapshotting that it was
// asked; SnapshotFailed why it was not taken.
Snapshot string
Snapshotting bool
SnapshotFailed string
// Have is what the node says it runs of the module; nil when it says nothing.
Have *delivery.Have
Now time.Time
}
// PutNext is what a step decides: the put as it stands after it, and the acts the pass does for it.
type PutNext struct {
Put delivery.Put
// Ask writes the put's ask about the move (PutAsk).
Ask bool
// Snapshot has the bus's node back the bus up now, its dump the streams' snapshot (ADR 0235).
Snapshot bool
// Warranted records, once, in the controller's hand-act log that the bus's want was written on the operator's warrant.
Warranted bool
}
// StepPut takes a put one step further from what the world holds (ADR 0546 clauses 3 to 8).
func StepPut(w PutWorld) PutNext {
p := w.Put
if delivery.PutEnded(p.Step) {
return PutNext{Put: p}
}
say := func(step, format string, args ...any) PutNext {
return PutNext{Put: p.Said(w.Now, step, fmt.Sprintf(format, args...))}
}
fp := short(p.FP)
if !w.Assigned {
return say(delivery.PutRefused, "%s is not assigned to %s: a put moves a module where it stands assigned (propose its placement first)", w.Module, w.Node)
}
switch w.Build {
case delivery.BuildDone:
case delivery.BuildFailed:
return say(delivery.PutRefused, "the builder's build of %s at %s failed: %s", w.Module, fp, w.BuildWords)
case "":
if p.FP != w.Newest {
// The builder builds the trunk's newest fingerprint and the candidates', and takes no ask for another yet (ADR 0546
// clause 3): an older fingerprint it never built, or whose key it removed, cannot be put.
return say(delivery.PutRefused, "the builder holds no build of %s at %s, and builds by itself only the trunk's newest (%s): put that, or a version it built", w.Module, fp, short(w.Newest))
}
return say(delivery.PutBuild, "waits for the builder's build of %s at %s, the trunk's newest, which it builds by itself", w.Module, fp)
default:
return say(delivery.PutBuild, "waits for the builder's build of %s at %s, which is %s", w.Module, fp, w.Build)
}
if w.Composed == nil || !w.Granted {
missing := fmt.Sprintf("no composition of %s for %s at %s is written", w.Module, w.Node, fp)
if w.Composed != nil {
missing = fmt.Sprintf("the grant of %s at %s does not name %s's accounts", w.Module, fp, w.Node)
}
since := w.Now
if p.Step == delivery.PutCompose && len(p.Steps) > 0 {
since = p.Steps[len(p.Steps)-1].At
}
if w.Now.Sub(since) > ComposeWithin {
return say(delivery.PutRefused, "%s after %s: only a composition written by the take-in of its build (or one of the five kept) is put; "+
"for another the hand path is mesh-hand-compose (ADR 0539 §10)", missing, ComposeWithin)
}
return say(delivery.PutCompose, "waits up to %s: %s yet", ComposeWithin, missing)
}
d := *w.Composed
if p.Step == delivery.PutWanted || p.Step == delivery.PutHolding {
return PutNext{Put: applied(w, p)}
}
p.Composed = d.Digest
if needsPerson(w, d) {
switch w.Answer {
case OptionNo:
return say(delivery.PutRefused, "the operator answered no to putting %s at %s on %s", w.Module, fp, w.Node)
case OptionYes:
default:
if w.AskExpired {
return say(delivery.PutRefused, "the operator's ask about putting %s at %s on %s expired unanswered: the node keeps what it runs", w.Module, fp, w.Node)
}
next := say(delivery.PutPerson, "waits for the operator's yes to putting %s at %s on %s (composition %s, judged %s)", w.Module, fp, w.Node, short(d.Digest), d.Judgement)
if p.Ask == "" {
next.Ask = true
next.Put.Ask, _ = delivery.RecordAskKey(w.Node, w.Module, d.Digest)
}
return next
}
if w.Bus {
switch {
case w.Snapshot != "":
p.Snapshot = w.Snapshot
case w.SnapshotFailed != "":
return say(delivery.PutRefused, "the bus's streams were not snapshotted, so the bus is not put: %s", w.SnapshotFailed)
case w.Snapshotting:
return say(delivery.PutSnapshot, "the operator said yes; waits for %s's backup holder to snapshot the bus's streams", w.Node)
default:
next := say(delivery.PutSnapshot, "the operator said yes; asked %s's backup holder to snapshot the bus's streams", w.Node)
next.Snapshot = true
return next
}
}
}
words := fmt.Sprintf("the want of %s on %s names %s (composition %s)", w.Module, w.Node, fp, short(d.Digest))
if p.Hold {
words += ", held until released"
}
if p.Snapshot != "" {
words += "; the streams were snapshotted: " + p.Snapshot
}
next := PutNext{Put: applied(w, p.Said(w.Now, delivery.PutWanted, words))}
next.Warranted = w.Bus && p.Ask != ""
return next
}
// applied is a wanted put once the node's have says how it went: holding or done when it runs the composition healthy.
func applied(w PutWorld, p delivery.Put) delivery.Put {
h := w.Have
if h == nil || h.FP != p.FP || h.Composed != p.Composed {
return p
}
switch {
case h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK && p.Hold:
return p.Said(w.Now, delivery.PutHolding, fmt.Sprintf("%s runs %s at %s healthy; held there until released", w.Node, w.Module, short(p.FP)))
case h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK:
return p.Said(w.Now, delivery.PutDone, fmt.Sprintf("%s runs %s at %s healthy; the rules write its want again", w.Node, w.Module, short(p.FP)))
case failed(*h):
return p.Said(w.Now, p.Step, fmt.Sprintf("%s runs %s at %s and its health check failed: %s", w.Node, w.Module, short(p.FP), h.Detail))
}
return p
}
// needsPerson says whether the put's move waits for the operator: any move of the bus but a revision at the fingerprint the
// node runs that changes only its user list (ADR 0539 §7, §15), and a move of a module of policy record judged to recreate a
// container, move data, or that cannot be judged (§14).
func needsPerson(w PutWorld, d delivery.ComposedDigest) bool {
runs := w.Have != nil && w.Have.FP == w.Put.FP
if runs && w.Have.Composed == d.Digest {
return false // the node runs this composition already: nothing moves
}
if w.Bus {
return !(runs && d.UserListOnly)
}
if !w.Recorded {
return false
}
return d.Judgement == delivery.JudgedRecreates || d.Judgement == delivery.JudgedMovesData || d.Judgement == delivery.JudgedCannot
}
// PutWant is the want a put writes once it is wanted: the fingerprint and composition, marked put, and held when it holds.
func PutWant(p delivery.Put) delivery.Want {
return delivery.Want{FP: p.FP, Composed: p.Composed, Put: true, Hold: p.Hold, Words: "put: " + p.Why}
}
// PutOwns says what a put does to the node's want: owns says the rules pass over it, want the want it writes (nil: what
// stands stays). A put owns the want from its asking until it ends; a held put ends only when released.
func PutOwns(p delivery.Put) (want *delivery.Want, owns bool) {
switch p.Step {
case delivery.PutWanted, delivery.PutHolding:
w := PutWant(p)
return &w, true
case delivery.PutDone, delivery.PutReleased, delivery.PutRefused:
return nil, false
}
return nil, true
}
// PutAsking is a put's own asking number: when it was asked, in milliseconds. Each put asked under the same key is another
// asking, so the ask is written again under an id of its own and an answer to an earlier put, whose warrant names that put's
// asking, never counts for a later one (ADR 0546 clause 8: one approval binds one act).
func PutAsking(p delivery.Put) int {
if n := p.At.UnixMilli(); n > 0 {
return int(n)
}
return 0
}
// PutAct is the act a bus put's yes binds and that is performed when its want is written.
func PutAct(node, module, fp, composed string) asks.Act {
return asks.Act{"put": "want", "node": node, "module": module, "fp": fp, "composed": composed}
}
// PutAsk is the ask about a put's move (ADR 0546 clauses 7 and 8), under the record ask's key of the composition so the
// teller relays it and its answer as any record ask's. For the bus its yes is of level approve and binds the act, so only a
// channel that proves who answered takes it (ADR 0259), never the view.
func PutAsk(key, node, module string, p delivery.Put, d delivery.ComposedDigest, bus bool, now time.Time) (delivery.AssignerAsk, error) {
yes := asks.Option{ID: OptionYes, Label: "Yes, put it", Does: "the node is told to apply " + module + " at " + short(p.FP), Level: asks.Acknowledge}
what := map[delivery.Judgement]string{
delivery.JudgedRecreates: "recreates a container",
delivery.JudgedMovesData: "moves the module's data",
delivery.JudgedCannot: "cannot be judged: " + d.Words,
delivery.JudgedNeither: "recreates no container and moves no data",
delivery.JudgedUnchanged: "changes nothing the node runs",
}[d.Judgement]
explanation := fmt.Sprintf("Asked to put %s at version %s on %s: %s. The move %s.", module, short(p.FP), node, p.Why, what)
expires := now.Add(7 * 24 * time.Hour)
if bus {
binds, err := asks.ActDigest(PutAct(node, module, p.FP, d.Digest))
if err != nil {
return delivery.AssignerAsk{}, err
}
yes.Level, yes.Binds = asks.Approve, binds
yes.Does = "the bus's streams are snapshotted, then " + node + " is told to apply the bus at " + short(p.FP)
explanation += " It is the mesh's bus: every module reconnects while it restarts, and its streams are snapshotted first."
expires = now.Add(asks.ApproveLasts)
}
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, PutAsking(p)),
Headline: fmt.Sprintf("Put %s on %s?", module, node),
Explanation: explanation,
Options: []asks.Option{yes,
{ID: OptionNo, Label: "No", Does: "the node keeps what it runs, and the put is refused", Level: asks.Acknowledge}},
Who: asks.Operator,
Expires: expires,
OnExpiry: "the node keeps what it runs",
About: key,
}, Asking: PutAsking(p)}, nil
}
@@ -0,0 +1,235 @@
package assign
import (
"fmt"
"maps"
"slices"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The seat mesh-placement (novox/hq ADR 0567), held by the assigner beside its wants: placement is its state
// `placement`, changed only by a proposal (§2) the nodes it touches answer (§3), judged (§4) and written on the
// operator's Approve at the revision it was laid over (§5). What here decides; the tools and the passes do.
// ProposeIn is what a proposal is laid over: its id, who and why, its changes; every node's current placement and the
// revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's
// digest.
type ProposeIn struct {
ID, By, Why string
At time.Time
Changes []delivery.PlacementProposalChange
Current map[string]delivery.Placement
Revisions map[string]uint64
Held func(module string) bool
Recorded func(node string) bool
// Warrant is the enrolment's warrant an add-node carries (ADR 0587 §7); only that change has one.
Warrant *delivery.AddNodeWarrant
}
// Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision),
// the laid placement of every node whose key it changes, and every finding, which refuses it whole: a change that does not fit its node, and what
// CheckPlacements refuses of the result laid over every placement (two modules on one host port, a provider not placed,
// a seat held once held twice, two nodes with one address, two hubs, a module the indexer does not hold, a node with no
// record). A key whose value the proposal leaves as it is changes nothing and is not laid.
func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Placement, []string) {
p := delivery.PlacementProposal{ID: in.ID, By: strings.TrimSpace(in.By), Why: strings.TrimSpace(in.Why), At: in.At.UTC(),
Changes: in.Changes, Warrant: in.Warrant, Base: map[string]uint64{}, State: delivery.ProposalAnswering}
var findings []string
if p.By == "" {
findings = append(findings, "the runtime names no caller: a proposal is made by someone")
}
if p.Why == "" {
findings = append(findings, "a proposal says why: it is shown to the operator")
}
if len(in.Changes) == 0 {
findings = append(findings, "a proposal changes something: it names no change")
}
for i, c := range in.Changes {
if c.Op == delivery.OpSeed {
findings = append(findings, fmt.Sprintf("change %d: the seed is gone with adopt, the controller's export being gone", i+1))
}
}
laid, lay := delivery.LayChanges(in.Current, in.Changes)
findings = append(findings, lay...)
findings = append(findings, ControllerRefusals(in.Changes, laid)...)
all := map[string]delivery.Placement{}
for node, pl := range in.Current {
all[node] = pl
}
out := map[string]*delivery.Placement{}
for node, pl := range laid {
if before, ok := in.Current[node]; ok && delivery.SamePlacement(before, pl) {
continue
}
pl := pl
all[node] = pl
out[node] = &pl
p.Base[node] = in.Revisions[node]
}
// A remove-node deletes the key of a joining node that never confirmed (the operator's decision of 2026-11-02 01:05): its
// candidate is no placement, and the checks across nodes read the mesh without it. LayChanges said why one cannot go.
for _, node := range p.RemovedNodes() {
if _, ok := in.Current[node]; !ok {
continue
}
delete(all, node)
out[node] = nil
p.Base[node] = in.Revisions[node]
}
if len(findings) == 0 && len(out) == 0 {
findings = append(findings, "the changes leave every placement as it is")
}
held, recorded := in.Held, in.Recorded
if held == nil {
held = func(string) bool { return false }
}
if recorded == nil {
recorded = func(string) bool { return false }
}
// A joining node writes its record only once it holds its account, which the add-node's write gives it: this one change by
// this one caller is the exception to a node with no record (ADR 0587 §7, changing ADR 0567 §2).
joining := map[string]bool{}
for _, c := range in.Changes {
if c.Op == delivery.OpAddNode {
joining[c.Node] = true
}
}
if len(joining) > 0 {
was := recorded
recorded = func(node string) bool { return joining[node] || was(node) }
}
for _, r := range delivery.CheckPlacements(all, held, recorded) {
findings = append(findings, r)
}
return p, out, findings
}
// ControllerModule is the controller's module, which nothing places again (issue 1126).
const ControllerModule = "mesh-controller"
// ControllerEnded is why a placement of the controller is refused: the parts that acted only once its lease was handed
// (consumers, the user list, enrolment, dead letters) now act without reading the lease, so a controller placed again
// would write beside them, silently.
const ControllerEnded = "the controller ended by its unassignment on 2026-11-02; placing it again would make two writers (issue 1126)"
// ControllerRefusals names every change, and every node as laid, that places the controller: by a place, by a seed's
// whole placement, or in what the changes lay. None when nothing does.
func ControllerRefusals(changes []delivery.PlacementProposalChange, laid map[string]delivery.Placement) []string {
var out []string
nodes := map[string]bool{}
for i, c := range changes {
places := c.Module == ControllerModule && c.Op != delivery.OpTakeOff
if c.Whole != nil && slices.Contains(c.Whole.Assigned(), ControllerModule) {
places = true
}
if places {
out = append(out, fmt.Sprintf("change %d places %s on %s: %s", i+1, ControllerModule, c.Node, ControllerEnded))
nodes[c.Node] = true
}
}
for _, node := range slices.Sorted(maps.Keys(laid)) {
if !nodes[node] && slices.Contains(laid[node].Assigned(), ControllerModule) {
out = append(out, fmt.Sprintf("%s would run %s: %s", node, ControllerModule, ControllerEnded))
}
}
return out
}
// Write is one node a warrant writes (§5): the node, its node key `<node>.node` (the commit of the node's keys,
// mesh-sdk delivery.PlacementKeys.Writes), the revision of the node's placement it is written at (0: it has none, so
// it is created) and its value (nil: it is deleted).
type Write struct {
Node, Key string
Base uint64
Value *delivery.Placement
}
// Writes is every key a proposal writes, by node order.
func Writes(p delivery.PlacementProposal, laid map[string]*delivery.Placement) ([]Write, error) {
var out []Write
for _, node := range p.Nodes() {
key, err := delivery.PlacementNodeKey(node)
if err != nil {
return nil, err
}
out = append(out, Write{Node: node, Key: key, Base: p.Base[node], Value: laid[node]})
}
return out, nil
}
// The kinds of the condition (and finding) of a refused controller call, which the assigner raised until it stopped calling
// the controller's verbs (ADR 0567 §8, issue 774).
const (
KindControllerCall = "controller-call"
KindControllerPin = "controller-pin"
)
// CallClearedCondition clears the condition of a refused controller call a previous process raised, for its pair (node and
// module, or node and provision for a pin).
func CallClearedCondition(kind, node, module string) asks.Condition {
return asks.Condition{Key: delivery.AssignerModule + "." + kind + "." + node + "." + module,
Subject: asks.Scope{Scope: "module", ID: module, Machine: node}, State: asks.StateCleared,
Resolved: "the assigner calls no verb of the controller any more: placement is the one source."}
}
// PlacementAct is the act an approved proposal's yes binds: writing exactly its changes at its base revisions.
func PlacementAct(p delivery.PlacementProposal) asks.Act {
return asks.Act{"placement": "write", "proposal": p.ID, "changes": delivery.ChangesDigest(p)}
}
// PlacementAsking is a proposal's asking number: each judging that raises its ask again (again, a key moved) is
// another asking, under an id of its own.
func PlacementAsking(p delivery.PlacementProposal, at time.Time) int {
return int(at.Unix())
}
// PlacementAsk is the authorising ask about a proposal (§5): an Approve that binds its act, so only a channel that
// proves who answered takes it, with the brief of the verdict by node (judge.Brief) as its explanation.
func PlacementAsk(key string, p delivery.PlacementProposal, brief string, asking int, now time.Time) (delivery.AssignerAsk, error) {
binds, err := asks.ActDigest(PlacementAct(p))
if err != nil {
return delivery.AssignerAsk{}, err
}
nodes := p.Nodes()
headline := fmt.Sprintf("Change placement on %s?", strings.Join(nodes, ", "))
if n := len([]rune(headline)); n > asks.HeadlineLength {
headline = fmt.Sprintf("Change placement on %d machines?", len(nodes))
}
return delivery.AssignerAsk{Ask: asks.Ask{
ID: delivery.AskingID(key, asking),
Headline: headline,
Explanation: brief,
Options: []asks.Option{
{ID: OptionYes, Label: "Approve", Does: "the placement of " + strings.Join(nodes, ", ") + " is written as proposed",
Level: asks.Approve, Binds: binds},
{ID: OptionNo, Label: "Decline", Does: "nothing is written, and the proposal ends", Level: asks.Acknowledge},
},
Who: asks.Operator,
Expires: now.Add(asks.ApproveLasts),
OnExpiry: "nothing is written, and the proposal ends",
About: key,
}, Asking: asking}, nil
}
// Rest is what a proposal leaves to write once the keys of written are: its changes of the other nodes, to be laid again
// over the keys as the write left them (§5: a key moved while it was written). False when nothing is left.
func Rest(p delivery.PlacementProposal, written map[string]*delivery.Placement) (delivery.PlacementProposal, bool) {
rest := p
rest.Changes, rest.Base = nil, map[string]uint64{}
for _, c := range p.Changes {
if _, done := written[c.Node]; !done {
rest.Changes = append(rest.Changes, c)
}
}
for node, base := range p.Base {
if _, done := written[node]; !done {
rest.Base[node] = base
}
}
rest.Must, rest.Verdict, rest.Ask, rest.State = nil, nil, "", delivery.ProposalAnswering
return rest, len(rest.Changes) > 0
}
@@ -0,0 +1,103 @@
package assign
import (
"encoding/json"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0567 §2, §5, §7, §8: what the seat's holder decides of a proposal: every refusal named at once, a valid one laid
// with the base revision of each key it changes, the writes at those revisions, the controller's calls that keep its
// assignments in step, the authorising ask bound to the changes, and the export read as the controller answers it.
func TestThePlacementSeatDecides(t *testing.T) {
anchor := delivery.Placement{V: delivery.Version, Node: "anchor", Decided: delivery.NodeDecided{Hub: true, Address: "10.0.0.1"},
Modules: []delivery.PlacedModule{{Module: "store", Ports: map[string]int{"5432": 5432}}, {Module: "web"}}}
laptop := delivery.Placement{V: delivery.Version, Node: "laptop", Decided: delivery.NodeDecided{Address: "10.0.0.2"}, Modules: []delivery.PlacedModule{{Module: "shell"}}}
in := func(changes ...delivery.PlacementProposalChange) ProposeIn {
return ProposeIn{ID: "p-one", By: "operator@laptop", Why: "move web", At: time.Unix(1, 0), Changes: changes,
Current: map[string]delivery.Placement{"anchor": anchor, "laptop": laptop}, Revisions: map[string]uint64{"anchor": 4, "laptop": 9},
Held: func(m string) bool { return m != "ghost" },
Recorded: func(n string) bool { return n == "anchor" || n == "laptop" }}
}
move := []delivery.PlacementProposalChange{{Op: delivery.OpTakeOff, Node: "anchor", Module: "web"}, {Op: delivery.OpPlace, Node: "laptop", Module: "web"}}
p, laid, findings := Propose(in(move...))
if len(findings) != 0 || p.State != delivery.ProposalAnswering || p.Base["anchor"] != 4 || p.Base["laptop"] != 9 ||
!slices.Equal(laid["laptop"].Assigned(), []string{"shell", "web"}) || !slices.Equal(laid["anchor"].Assigned(), []string{"store"}) {
t.Fatalf("a move: %+v, %+v, %v", p, laid, findings)
}
if _, err := delivery.EncodePlacementProposal(p); err != nil {
t.Fatalf("the proposal cannot be written: %v", err)
}
for name, c := range map[string]struct {
in ProposeIn
want string
}{
"a module not held": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "ghost"}), "the indexer holds no module ghost"},
"a node with no record": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "stranger", Module: "web"}), "stranger is a node with no record"},
"nothing changes": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "anchor", Module: "web"}), "leave every placement as it is"},
"no why": {func() ProposeIn { i := in(move...); i.Why = " "; return i }(), "says why"},
"no caller": {func() ProposeIn { i := in(move...); i.By = ""; return i }(), "names no caller"},
"a seed by hand": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "laptop", Whole: &laptop}), "gone with adopt"},
// 1126: the controller ended by its unassignment; placed again it would be a second writer beside the parts that
// act without its lease.
"the controller placed again": {in(delivery.PlacementProposalChange{Op: delivery.OpPlace, Node: "laptop", Module: "mesh-controller"}), "ended by its unassignment"},
"the controller seeded whole": {in(delivery.PlacementProposalChange{Op: delivery.OpSeed, Node: "anchor", Whole: &delivery.Placement{V: delivery.Version, Node: "anchor",
Decided: anchor.Decided, Modules: []delivery.PlacedModule{{Module: "mesh-controller"}}}}), "two writers (issue 1126)"},
"a pin of nobody placed": {in(delivery.PlacementProposalChange{Op: delivery.OpPin, Node: "laptop", Provision: "db", Provider: &delivery.Provider{Node: "laptop", Module: "store"}}), "not placed there"},
} {
if _, _, f := Propose(c.in); !slices.ContainsFunc(f, func(s string) bool { return strings.Contains(s, c.want) }) {
t.Errorf("%s: findings %v, want one naming %q", name, f, c.want)
}
}
// Every kind of change is taken: the placement is the one source, and the controller's verbs limit nothing (ADR 0567 §8).
for _, c := range []delivery.PlacementProposalChange{
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}},
{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", MeshWide: map[string]any{"accesses": []any{}}},
{Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)},
{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"},
} {
if _, _, f := Propose(in(c)); len(f) != 0 {
t.Errorf("%s is refused: %v", c.Op, f)
}
}
// A key moved while the warrant wrote: what is left to write is the other nodes' changes, to be laid and asked again.
if rest, ok := Rest(p, map[string]*delivery.Placement{"anchor": laid["anchor"]}); !ok || len(rest.Changes) != 1 || rest.Changes[0].Node != "laptop" ||
len(rest.Base) != 1 || rest.State != delivery.ProposalAnswering || rest.Must != nil {
t.Errorf("the rest = %+v, %v", rest, ok)
}
if _, ok := Rest(p, laid); ok {
t.Error("a proposal written whole leaves something to write")
}
// The seat says it waits while one of its states is not bound, naming it, and clears once all are.
for _, unbound := range [][]string{{delivery.PlacementPulsesBucket}, nil} {
c := SeatWaitsCondition(unbound)
if _, err := asks.EncodeCondition(c); err != nil || (len(unbound) > 0) != (c.State == asks.StateRaised) ||
(len(unbound) > 0 && !strings.Contains(c.Explanation, "the seat waits for its states: "+delivery.PlacementPulsesBucket)) {
t.Errorf("waiting for %v: %+v, %v", unbound, c, err)
}
}
writes, err := Writes(p, laid)
if err != nil || len(writes) != 2 || writes[0].Key != "anchor.node" || writes[0].Base != 4 || writes[1].Base != 9 {
t.Fatalf("writes = %+v, %v", writes, err)
}
a, err := PlacementAsk("placement.p-one", p, "move web (proposed by operator@laptop)", 7, time.Unix(100, 0))
if err != nil || a.Ask.Check(time.Unix(100, 0)) != nil || a.Ask.Options[0].Level != asks.Approve || a.Ask.Options[0].Performs(PlacementAct(p)) != nil {
t.Fatalf("the ask = %+v, %v", a, err)
}
q := p
q.Changes = append(slices.Clone(p.Changes), delivery.PlacementProposalChange{Op: delivery.OpUnpin, Node: "laptop", Provision: "x"})
if a.Ask.Options[0].Performs(PlacementAct(q)) == nil {
t.Error("the ask's yes binds changes it was not asked about")
}
}
@@ -0,0 +1,744 @@
package assign
import (
"fmt"
"maps"
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
// trunk (the indexer's modules), whether the builder's build of F is done, whether F is held, whether the controller
// issued its credential for F, its manifest at F, and the nodes it is assigned to.
type Module struct {
Name string
FP string
Built bool
Held bool
Granted bool
// GrantedAt says whether the controller issued the module's credential for another fingerprint than FP: the first want of a node
// that runs an older one names it (issue 1945). Nil: none.
GrantedAt func(fp string) bool
// GrantedOn says whether the controller's grant of FP names the module's accounts on a node, when the grant names any:
// a grant written for the module's other nodes says nothing of a node whose account is not issued and sealed (issue
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone.
GrantedOn func(node string) bool
Manifest catalogue.Manifest
// Provisions is the seat mesh-provisioner's provisions this module consumes on a node, by key (ADR 0570 §4): the want
// waits until each is provided. Nil: the seat's state is not read (not delivered yet), and nothing waits on it.
Provisions func(node string) map[string]provisions.Provision
// Answering is the manifests of the modules placed somewhere that provide a provision: what catalogue.Awaited
// judges a want's provisions by (issue 774). Nil: none known, and no key is waited for that is not there.
Answering func(provision string) []catalogue.Manifest
// Offered says whether any module offers a provision by name or by its own name (catalogue.Manifest.Offers) among every
// manifest the trunk holds and the modules the mesh provides itself, placed or not. Nil: not known, and no requirement
// is waited on for want of an offer.
Offered func(provision string) bool
// Chosen is a person's choice of the module's upgrade policy (the assigner's `policies`, ADR 0565 §7); nil: none.
Chosen *catalogue.PersonPolicy
Nodes []string
// Composed answers the composition of the module for a node at a fingerprint: the digest the node's own have names (a
// placed node composes its own, ADR 0571 §6), and whether it names one. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool)
// Allowed says whether the operator answered yes to the record ask about the move of this module on node to the composition
// with this digest (ADR 0539 §14). Nil: none did.
Allowed func(node, digest string) bool
// Release names the release whose Approve carries the build fp, as mesh-delivery's gate holds it for the release's heads,
// read by the build's source revision (ADR 0601 §1): that Approve is the yes to a move of the build alone. False when no
// approved release is known to carry it. Nil: none is read.
Release func(fp string) (string, bool)
// Released is a hold on FP the operator released (ADR 0539 §19): FP is not held again by a health check that failed.
Released bool
// Restored says whether the operator answered "restore" to the restore ask of this data module on node: it is then set back
// as any module is, losing the writes made since (ADR 0539 §19). Nil: none did.
Restored func(node string) bool
// Kept says the operator answered "keep" to the hold ask of FP: a hold kept so is not lifted by itself.
Kept bool
// Now is when this pass reads the haves: the held record names it as when the health check was seen to fail. Zero: not said.
Now time.Time
// Puts are the puts of the module, by node (ADR 0546): while one owns a node's want (PutOwns), the rules pass over it.
Puts map[string]delivery.Put
// Self says whether node composes the module itself: the catalogue's placement names the node (ADR 0547 §2). It is the one
// source of a want's Self: every want the rules write or keep is derived from it (selfDerived), so a want written Self while
// the node was placed does not outlive the placement (issue 1126). Nil: no node does.
Self func(node string) bool
}
func (m Module) selfOn(node string) bool { return m.Self != nil && m.Self(node) }
func (m Module) composition(node, fp string) (delivery.ComposedDigest, bool) {
if m.Composed == nil {
return delivery.ComposedDigest{}, false
}
return m.Composed(node, fp)
}
// Node is what the rules read of the nodes: whether each is online (the watchman's presence) and each module's have.
type Nodes struct {
Online map[string]bool
Haves map[string]map[string]delivery.Have
// Providers is, by provision, the modules that provide it, with the nodes each is assigned to.
Providers map[string]map[string][]string
}
// healthy says whether a have runs fp healthy.
func healthy(h delivery.Have, fp string) bool {
return h.FP == fp && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
// Canary is the module's canary: the first of its nodes, by name, that is online (ADR 0435 §3; one ring, ordered by
// name, ADR 0528 clause 3). Empty when none is online.
func Canary(nodes []string, online map[string]bool) string {
sorted := slices.Sorted(slices.Values(nodes))
for _, n := range sorted {
if online[n] {
return n
}
}
return ""
}
// Holds is whether every rule holds for moving node's want for m to m.FP, and if not, the first rule that does not,
// with why (ADR 0429 §6). Rule 4, at most `step` others ahead, always holds: the mesh keeps no step, and step 0 is no
// limit (ADR 0528 clause 3). Rule 5, every provider healthy, holds no want (issue 1960): a module whose provider is down
// is wanted all the same, and said as not reached (Unreached), since a site runs while the proxy in front of it is
// down and one module never blocks another.
func Holds(m Module, node string, n Nodes) (bool, int, string) {
return holds(m, node, n, true)
}
// holds is Holds; with ordering false it leaves out the canary (3), which orders a move among nodes, and keeps those
// without which there is nothing to want: the build done, not held, the grant and the composition.
func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
switch {
case !m.Built:
return false, delivery.RuleBuildDone, fmt.Sprintf("the build of %s at %s is not done", m.Name, short(m.FP))
case m.Held:
return false, delivery.RuleNotHeld, fmt.Sprintf("%s is held at %s", m.Name, short(m.FP))
}
if ordering {
canary := Canary(m.Nodes, n.Online)
switch {
case canary == "":
return false, delivery.RuleCanary, fmt.Sprintf("none of the nodes of %s is online, so it has no canary", m.Name)
case node != canary && !healthy(n.Haves[canary][m.Name], m.FP):
return false, delivery.RuleCanary, fmt.Sprintf("the canary %s does not run %s healthy yet", canary, short(m.FP))
}
}
if !m.Granted {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s for %s", m.Name, short(m.FP))
}
if m.GrantedOn != nil && !m.GrantedOn(node) {
return false, delivery.RuleGrant, fmt.Sprintf("the controller has not issued the credential of %s on %s for %s", m.Name, node,
short(m.FP))
}
if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("%s has named no composition of its own for %s at %s yet", node, m.Name, short(m.FP))
}
if words, waiting := m.provisionWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
return true, 0, ""
}
// provisionWaiting says why the module's want on node waits for a provision (ADR 0570 §4): a provision it requires that has
// no key yet, or the first of its keys that is not provided. Nil Provisions (the seat's state not on the bus) waits for none.
func (m Module) provisionWaiting(node string) (string, bool) {
if m.Provisions == nil {
return "", false
}
if missing := m.MissingProvisions(node); len(missing) > 0 {
return fmt.Sprintf("provision %s has no key yet for %s on %s", missing[0], m.Name, node), true
}
keyed := m.Provisions(node)
// Only a provision the newest manifest still requires is waited on (ADR 0603 §9): a key of an older revision's edge,
// asked or retiring for the operator's answer (ADR 0570 §5), holds nothing. A requirement catalogue.Awaited leaves out is
// not waited on when something offers it (a module's own name, a module the mesh provides itself, a provision the
// composition does not key: optional, a toolchain on the module's own machine, a login the controller makes). One that
// nothing offers is waited on (ADR 0603 §5: an unknown provider has provided nothing), so a consumer never moves ahead of
// a provider whose module the indexer has not yet taken in; it holds loudly, since the watchman raises the hold past its
// bound (ADR 0603 §8).
needed := m.Manifest.Requires
if m.Answering != nil {
needed = catalogue.Awaited(m.Manifest, m.Answering)
have := map[string]bool{}
for _, p := range keyed {
have[p.Provision] = true
}
for _, r := range slices.Sorted(slices.Values(m.Manifest.Requires)) {
if have[r] && !m.Manifest.OptionalRequires[r] && !slices.Contains(needed, r) {
needed = append(needed, r) // the provisioner keys it from the manifests, whoever is placed yet
continue
}
if m.Offered == nil || m.Manifest.OptionalRequires[r] || len(m.Answering(r)) > 0 || m.Offered(r) || slices.Contains(needed, r) {
continue
}
needed = append(needed, r)
if !have[r] {
return fmt.Sprintf("no module offers %s yet for %s on %s: an unknown provider has provided nothing, so the want waits for one, and the watchman raises it past its bound", r, m.Name, node), true
}
}
}
for _, key := range slices.Sorted(maps.Keys(keyed)) {
if p := keyed[key]; p.State != provisions.Provided && slices.Contains(needed, p.Provision) {
words := fmt.Sprintf("the provision %s of %s on %s is %s, not provided", key, m.Name, node, p.State)
if p.Why != "" {
words += ": " + p.Why
}
return words, true
}
}
return "", false
}
// MissingProvisions are the provisions a want of the module waits for (catalogue.Awaited, the one rule the plan
// composition and the seat mesh-provisioner key by, issue 774) that have no key on node yet, sorted; none while the
// seat's state is not read. A requirement the composition keys nothing for (a toolchain on the module's own machine, a
// login the controller still makes for a module not handed) is not waited for: on 2026-10-30 every new build of
// grafana, dnsmasq, mesh-knowledge, mesh-mail and the builders waited here for a key nobody would make.
func (m Module) MissingProvisions(node string) []string {
if m.Provisions == nil || m.Answering == nil {
return nil
}
have := map[string]bool{}
for _, p := range m.Provisions(node) {
have[p.Provision] = true
}
var out []string
for _, r := range catalogue.Awaited(m.Manifest, m.Answering) {
if !have[r] {
out = append(out, r)
}
}
return out
}
// providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned
// elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node
// offers it) and holds.
func providedHealthy(provision, node string, n Nodes) (bool, string) {
providers := n.Providers[provision]
if len(providers) == 0 {
return true, ""
}
names := make([]string, 0, len(providers))
for p := range providers {
names = append(names, p)
}
sort.Strings(names)
for _, p := range names {
nodes := providers[p]
at := node
if !slices.Contains(nodes, node) {
at = Canary(nodes, n.Online)
}
if h, ok := n.Haves[at][p]; ok && at != "" && h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return true, ""
}
}
return false, fmt.Sprintf("no provider of %s runs healthy on %s or at its canary", provision, node)
}
// unreached is why m on node may be wanted and not reached: the first provision it requires that no provider runs
// healthy on node or at its canary (rule 5, said and never holding a want; issue 1960); empty when all do.
func unreached(m Module, node string, n Nodes) string {
for _, provision := range m.Manifest.Requires {
if ok, why := providedHealthy(provision, node, n); !ok {
return why
}
}
return ""
}
// keepsData says whether a module holds data a set-back would lose: an own item of a class that keeps (irreplaceable,
// valuable or rebuildable; cache and none do not).
func keepsData(m catalogue.Manifest) bool {
for _, it := range m.DataItems() {
if it.Class == catalogue.ClassIrreplaceable || it.Class == catalogue.ClassValuable || it.Class == catalogue.ClassRebuildable {
return true
}
}
return false
}
// Outcome is what one module's pass decides: each node's want to write (only those that change), and a hold to write.
type Outcome struct {
Wants map[string]delivery.Want
// Held is the hold of m.FP to write, when a node's health check failed on it and it is not held yet.
Held *delivery.Held
// Lifted is the hold of m.FP lifted by itself: every node that runs the module runs m.FP healthy (issue 1906). The held
// record and its ask are taken away, and the wants move by the rules again in this pass.
Lifted bool
// NodeFailed is, by node, a failed health check of m.FP on a node while another node runs m.FP healthy: the failure is
// that node's, so m.FP is not held and nothing is set back (ADR 0557 §1, issue 1496). The pass raises its condition once.
NodeFailed map[string]NodeFailure
// Record is the moves a person must allow before the want names them: each is asked once (ADR 0539 §14).
Record []RecordNeed
// Restore is the nodes of a module holding data whose held fingerprint failed its health check there and that are not set
// back: the operator is asked whether to restore each (ADR 0539 §19).
Restore []string
// Rules is, by node, the rule holding the node back from m.FP and why.
Rules map[string]Holding
// Unreached is, by node, why the module may be wanted there and not reached: no provider of a provision it requires runs
// healthy (rule 5, issue 1960). It holds no want back; the pass says it.
Unreached map[string]string
// Neither is, by node, the nodes the module is assigned to that run it (or whose have names it), hold no want for it after this
// pass and are not held (a held module is on the held list), with the rule holding the first want back: the node's own composition does not hold a module with a fingerprint of the new form, so such a module is on
// neither list and nothing applies it on that node (issue 1945). A module with no fingerprint of the new form goes with the node's
// own composition and is never here.
Neither map[string]Holding
}
// RecordNeed is a move of a module of policy record on a node that waits for the operator's yes: the composition's digest, the
// controller's judgement of the move and its words.
type RecordNeed struct {
Node string
Digest string
Judgement delivery.Judgement
Words string
}
// Holding is a rule holding a node back, and why. The bus module is held back by the rules alone, as every module (ADR 0601 §1,
// §2): no bar of its own.
type Holding struct {
Rule int
Words string
}
// Reason is the holding in words: the rule and its words.
func (h Holding) Reason() string {
return fmt.Sprintf("rule %d: %s", h.Rule, h.Words)
}
// Wants is the pass of one module m: for each of its nodes, the want by the six rules, starting from the want written
// before (wants, by node); lastHealthy is, by node, the newest fingerprint other than m.FP the node ran healthy. A failed
// health check of m.FP on a node holds it: every node wanting m.FP is set back to its last healthy fingerprint, but a
// module holding data is not set back but asked about (Restore). A
// node that wants m but is no longer among its nodes gets its want written removed, never deleted (ADR 0408 §4).
func Wants(m Module, n Nodes, wants map[string]delivery.Want, lastHealthy map[string]string) Outcome {
out := Outcome{Wants: map[string]delivery.Want{}, Rules: map[string]Holding{}, Neither: map[string]Holding{}}
held := m.Held
if held && !m.Kept && m.FP != "" && allRunHealthy(m, n) {
out.Lifted, held = true, false
}
if !held && !out.Lifted && !m.Released && m.FP != "" {
for _, node := range slices.Sorted(slices.Values(m.Nodes)) {
h, ok := n.Haves[node][m.Name]
if !ok || h.FP != m.FP || !failed(h) {
continue
}
if others := healthyElsewhere(m, n, node); len(others) > 0 && !failedElsewhere(m, n, node) {
// F runs healthy on another node, so its failure here is this node's, not F's: F is not held, no node is set back,
// and the failing node keeps its want (ADR 0557 §1; issue 1496: on ace a newly assigned mesh-build-node failed its
// first check and the hold moved shanks, g14 and novox, healthy on F for an hour, back). The pass raises the node's
// condition. With no online healthy witness, or with F failed on a second node, the hold and the set-back stand (ADR 0557
// §2, ADR 0429 §6).
if out.NodeFailed == nil {
out.NodeFailed = map[string]NodeFailure{}
}
out.NodeFailed[node] = NodeFailure{Others: others, Words: HeldWords(h, time.Time{})} // said once: no clock in it
continue
}
out.Held = &delivery.Held{Node: node, Words: HeldWords(h, m.Now)}
held = true
break
}
}
mm := m
mm.Held = held
for _, node := range m.Nodes {
if m.FP == "" {
break
}
before, had := wants[node]
var next delivery.Want
p, put := m.Puts[node]
if !put && had && before.Put && before.Hold {
// The want is the truth (ADR 0546 clause 5): a want marked put and held stands while its put is not heard (the
// puts not open, or not yet handed after a start); only a put heard released gives it back to the rules. A put
// without hold is not kept this way: its put takes it further once heard.
p, put = delivery.Put{FP: before.FP, Composed: before.Composed, Hold: true, Why: strings.TrimPrefix(before.Words, "put: "),
Step: delivery.PutHolding}, true
}
if put {
if want, owns := PutOwns(p); owns {
if want != nil {
w := *want
// A held bus still takes a revision of its composition that changes only its user list (ADR 0539 §7).
if ul, ok := m.userListRevision(node, before, had, n); ok && ul.FP == w.FP {
w.Composed = ul.Composed
}
if !had || !delivery.SameWant(normal(before), normal(w)) {
out.Wants[node] = w
}
}
continue
}
}
// A revision of the bus module's composition that differs only in its user list is written at once, at the fingerprint the
// node runs, outside the rules, the record hold and the bar on a new fingerprint: a new module's credential must reach the
// bus before the module can connect (ADR 0539 §7). The controller marks it; whatever it judges of the file's mount stands.
if w, ok := m.userListRevision(node, before, had, n); ok {
if !had || !delivery.SameWant(normal(before), normal(w)) {
out.Wants[node] = w
}
continue
}
ok, rule, words := Holds(mm, node, n)
target := m.FP
if have, ran := n.Haves[node][m.Name]; !ok && (!had || before.Removed) && rule == delivery.RuleCanary {
// The canary orders a move, never the first want of a node that does not run the module (issue 1960: de-spiegel,
// link2pay and photos on novox were left with no want). A node that runs an older fingerprint of the new form,
// composed and granted, gets its first want at what it runs, and rule 3 then orders its move to F as from any want.
// A node that runs an old-form build, which no want can name, stays under the canary (issue 1945, a design limit):
// its first want names F once the canary runs F healthy. Any other node that runs it (F itself, or an older
// fingerprint with no composition or grant) gets F by the rules without which there is nothing to want.
older := ran && runs(have) && have.FP != m.FP && strings.HasPrefix(have.FP, delivery.SourcePrefix2)
if older {
_, composed := m.composition(node, have.FP)
older = composed && m.GrantedAt != nil && m.GrantedAt(have.FP)
}
oldForm := ran && runs(have) && have.FP != m.FP && !strings.HasPrefix(have.FP, delivery.SourcePrefix2)
switch {
case older:
ok, target = true, have.FP
case oldForm:
// held back by rule 3 as written: on neither list until the canary runs F healthy
default:
ok, rule, words = holds(mm, node, n, false)
}
}
if why := unreached(m, node, n); why != "" {
if out.Unreached == nil {
out.Unreached = map[string]string{}
}
out.Unreached[node] = why
}
if ok && target == m.FP && len(out.NodeFailed) > 0 && !(had && !before.Removed && before.FP == m.FP) &&
n.Haves[node][m.Name].FP != m.FP {
// While a node fails F alone, F reaches no further node: a node that neither wants nor runs F gets no new want
// naming it, and wants already written stay (ADR 0557 §2). A node that runs F with no live want gets its want at F, or
// it would be on neither list and its node-engine would give the module up (issue 1945).
ok, rule, words = false, delivery.RuleNotHeld, FailedAloneWords(m.Name, m.FP, slices.Sorted(maps.Keys(out.NodeFailed)))
}
switch {
case ok:
d, _ := m.composition(node, target)
next = delivery.Want{FP: target, Composed: d.Digest}
at := m
if target != m.FP {
// the first want at the older fingerprint the node runs: held back from F by the rule, and judged as a want at it
next.Newest, next.Rule, next.Words = m.FP, rule, words
at.FP = target
}
// A placed node's want names the build alone: it is the same target while the build is.
if had && !before.Removed && !before.Put && before.FP == target && (before.Composed == d.Digest || (before.Self && m.selfOn(node))) {
next.Backup, next.Words = before.Backup, before.Words // the answer to a failed backup stays while its target stays (ADR 0539 §16)
}
running, ran := n.Haves[node][m.Name]
// A removed want is no want: a first want after it goes through the record ask as a first want does.
asked, hadLive := before, had
if before.Removed {
asked, hadLive = delivery.Want{}, false
}
if need, wait := at.mustAsk(node, asked, hadLive, ran, running.FP, running.Composed == "", d); wait {
if need != nil {
out.Record = append(out.Record, *need)
}
continue
}
case held && had && before.FP == m.FP:
out.Rules[node] = Holding{Rule: rule, Words: words}
last := lastHealthy[node]
switch {
case keepsData(m.Manifest) && (m.Restored == nil || !m.Restored(node)):
next = before
next.Words = fmt.Sprintf("%s holds data, so it is not set back from %s: the operator is asked whether to restore it", m.Name, short(m.FP))
out.Restore = append(out.Restore, node)
case last == "":
next = before
next.Words = fmt.Sprintf("%s is held at %s and %s ran no other fingerprint healthy to set it back to", m.Name, short(m.FP), node)
default:
d, there := m.composition(node, last)
if !there {
next = before
next.Words = fmt.Sprintf("%s is held at %s, and the composition of %s at %s is gone, so %s is not set back", m.Name, short(m.FP), m.Name, short(last), node)
break
}
next = delivery.Want{FP: last, Composed: d.Digest, Newest: m.FP, Rule: rule, Words: words}
}
default:
out.Rules[node] = Holding{Rule: rule, Words: words}
if !had || before.Removed {
// The node runs the module, or its have names it, and gets no want for it: the module is on neither list (issue 1945), and
// the pass says so. A new assignment, with no have, waits for the rules and is on no list yet by design. A held module (rule 2,
// held before or by this pass) is on the held list instead, whose hold is said and answered on its own: it is never on neither
// list (issue 1945: systemd-resolved, held on shanks by the operator, was said as urgent every hour).
h, there := n.Haves[node][m.Name]
switch {
case there && h.Phase != delivery.PhaseRemoved && !held:
out.Neither[node] = Holding{Rule: rule, Words: words}
case had && (!there || h.Phase == delivery.PhaseRemoved):
// A removed want of a node that does not run the module says why no want follows it (issue 1945), so the view's
// assigner_want answers it; it stays removed. A node with no key gets none: a removed want written where none was
// would have its node agent remove the module and stop holding its composition's resources back.
next = delivery.Want{Removed: true, Words: NoWantWords(rule, words)}
if !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
}
continue
}
next = delivery.Want{FP: before.FP, Composed: before.Composed, Newest: m.FP, Rule: rule, Words: words}
if before.FP == m.FP {
// Held at F by the rules: a new revision of F's composition waits for them, as a new fingerprint does (ADR 0539 §5).
next = delivery.Want{FP: m.FP, Composed: before.Composed}
} else if d, there := m.composition(node, before.FP); there && d.Digest != "" && d.Digest != before.Composed {
// Held back at an earlier fingerprint whose kept composition the controller composed again with today's config
// (novox/hq ADR 0545): the want follows its new revision, so the node does not run that build with the config of
// the moment it was made (on g14 a capacity of 0 after a rollback). Asked as any move of a record module is.
at := m
at.FP = before.FP
running, ran := n.Haves[node][m.Name]
need, wait := at.mustAsk(node, before, true, ran, running.FP, running.Composed == "", d)
if need != nil {
out.Record = append(out.Record, *need)
}
if !wait {
next.Composed = d.Digest
}
}
}
next, write := m.selfDerived(node, before, had, next, n, &out)
if !write {
continue
}
if !had || !delivery.SameWant(normal(before), normal(next)) {
out.Wants[node] = next
}
}
for node, before := range wants {
if !slices.Contains(m.Nodes, node) && !before.Removed {
out.Wants[node] = delivery.Want{Removed: true, Words: fmt.Sprintf("%s is no longer assigned to %s", m.Name, node)}
}
}
return out
}
// selfDerived is next with its Self derived from the placement now (Module.Self), whichever rule made it, and whether it is
// written. A placed node's want names the build alone (ADR 0547 §2). An unplaced node's want that a rule kept from the want
// written while the node was placed (held at F, held back by rule 3 or 5, a data module asked about) names the controller's
// composition of the same fingerprint, asked as any move of a record module is (ADR 0539 §14): on 2026-10-27, after the
// placement seed's revert, g14's systemd-resolved (held) and forticlient (rule 5), and shanks's forticlient (rule 3), kept
// their Self wants, which their node agents, no longer placed, could not compose (issue 1126). With no such composition, or
// while its ask waits, the want written stays and the node's holding says why.
func (m Module) selfDerived(node string, before delivery.Want, had bool, next delivery.Want, n Nodes, out *Outcome) (delivery.Want, bool) {
switch {
case next.Removed:
return next, true
case m.selfOn(node):
next.Self, next.Composed = true, ""
return next, true
case !next.Self && next.Composed != "":
return next, true
}
next.Self, next.Yes = false, ""
d, there := m.composition(node, next.FP)
if !there || d.Digest == "" {
out.Rules[node] = Holding{Rule: delivery.RuleComposed, Words: fmt.Sprintf("%s on %s was composed by the node while it was placed, "+
"and no composition of its own is named at %s now, so its want stays", m.Name, node, short(next.FP))}
return next, false
}
at := m
at.FP = next.FP
running, ran := n.Haves[node][m.Name]
if need, wait := at.mustAsk(node, before, had, ran, running.FP, running.Composed == "", d); wait {
if need != nil && !slices.Contains(out.Record, *need) { // the branch held back at an older fingerprint may have asked it
out.Record = append(out.Record, *need)
}
return next, false
}
next.Composed = d.Digest
return next, true
}
// NoWantWords say why a node gets no want for a module: the rule that holds its first want back and why (issue 1945).
func NoWantWords(rule int, words string) string {
return fmt.Sprintf("no want: rule %d: %s", rule, words)
}
// failed says whether a have's health check failed once the node is done moving: phase running or failed (never removed, held, or
// moving). A failed health while it
// backs up, applies or checks is a step of the move, not its outcome (on novox at 15:27:36 on 2026-10-25, "health failed, phase
// applying" held the controller's canary two seconds before the new build was healthy). Every healthy read (healthy, providedHealthy,
// the world's lastHealthy) already asks for phase running.
func failed(h delivery.Have) bool {
return h.Health == delivery.HealthFailed && (h.Phase == delivery.PhaseRunning || h.Phase == delivery.PhaseFailed)
}
// healthyElsewhere is every online node of m but node that runs m.FP healthy, by name.
func healthyElsewhere(m Module, n Nodes, node string) []string {
var out []string
for _, other := range slices.Sorted(slices.Values(m.Nodes)) {
// A witness is online: a node that sleeps keeps the have it last wrote, and a stale have is no evidence that F runs
// healthy now.
if other != node && n.Online[other] && healthy(n.Haves[other][m.Name], m.FP) {
out = append(out, other)
}
}
return out
}
// failedElsewhere says whether another node of m has a have naming m.FP failed: then the failure is not one node's alone, and
// the hold and the set-back stand at the second failure (ADR 0557 §2).
func failedElsewhere(m Module, n Nodes, node string) bool {
for _, other := range m.Nodes {
if h, ok := n.Haves[other][m.Name]; ok && other != node && h.FP == m.FP && failed(h) {
return true
}
}
return false
}
// FailedAloneWords say why a node gets no new want naming fp: fp fails its health check on the nodes named, alone.
func FailedAloneWords(module, fp string, failing []string) string {
return fmt.Sprintf("%s at %s fails its health check on %s, so no further node moves to it until that is over", module, short(fp),
strings.Join(failing, ", "))
}
// NodeFailure is a failed health check of a module's newest fingerprint on one node while it runs healthy on Others: the have's
// health, phase and words (HeldWords without a time), and the healthy witnesses by name.
type NodeFailure struct {
Others []string
Words string
}
// runs says whether a have is of a module the node runs: running, whatever its health.
func runs(h delivery.Have) bool {
return h.Phase == delivery.PhaseRunning
}
// allRunHealthy says whether every node of m that runs the module (its have's phase is running, or failed) runs m.FP healthy, and
// at least one does: a hold whose fingerprint runs healthy wherever the module runs holds nothing back any more (issue 1906: seven
// modules stayed held on every machine after one transient failed check). A node set back to another fingerprint does not run
// m.FP, and a failed one does not run it healthy, so the hold stays; a have that is removed, held for its backup or still moving
// (backing up, applying, checking) is not counted, or it would keep the hold up for ever.
func allRunHealthy(m Module, n Nodes) bool {
ran := 0
for _, node := range m.Nodes {
h, ok := n.Haves[node][m.Name]
if !ok || (h.Phase != delivery.PhaseRunning && h.Phase != delivery.PhaseFailed) {
continue
}
if !healthy(h, m.FP) {
return false
}
ran++
}
return ran > 0
}
// HeldWords are the words a hold keeps of its cause (issue 1906): the health and phase the node reported, when the assigner saw
// it, and the node's own words. The node is the held record's own field.
func HeldWords(h delivery.Have, at time.Time) string {
words := fmt.Sprintf("health %s, phase %s", h.Health, h.Phase)
if !at.IsZero() {
words += ", seen at " + at.UTC().Format(time.RFC3339)
}
if d := strings.TrimSpace(h.Detail); d != "" {
return words + ": " + d
}
return words + ": the node gave no words"
}
// LiftedWords say why a hold was lifted by itself.
func LiftedWords(module, fp string) string {
return fmt.Sprintf("every node that runs %s runs %s healthy, so the assigner lifted its hold by itself", module, short(fp))
}
// normal is a want without its version, as a decoded and a composed one are compared.
func normal(w delivery.Want) delivery.Want {
w.V = 0
return w
}
func short(fp string) string {
part, err := delivery.FingerprintKeyPart(fp)
if err != nil || len(part) < 12 {
return fp
}
return part[:12]
}
// mustAsk says whether the move to the composition d on node waits for the operator's yes (need names the ask, ADR 0539 §14). A
// module the node never ran (it has no want and no have), a move judged neither, a module that is not of policy record and a want
// that stays are written as any want. The bus module is no exception: it moves by the rules (ADR 0601 §1), its node backing up its
// streams before it applies (§2); the assigner asks no snapshot for it.
func (m Module) mustAsk(node string, before delivery.Want, had, ran bool, running string, haveNamesNoComposition bool, d delivery.ComposedDigest) (*RecordNeed, bool) {
if (!had && !ran) || before.Removed || (before.FP == m.FP && before.Composed == d.Digest) {
return nil, false
}
// Right after the switch no node has a readable want, and a have from the old node-engine names a fingerprint and no composition
// (ADR 0539 §9, §11): a node running this fingerprint is given its want at it with no hold when the controller judges the composition
// unchanged, equal to the declaration the node last applied (ADR 0540 §9), since the node agent then applies nothing. A composition that
// differs from it (a changed setting or secret), or that the controller could not judge, waits for the ask. A new fingerprint still waits.
if !had && ran && running == m.FP && haveNamesNoComposition && d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
if !m.recorded(d) || d.Judgement == delivery.JudgedNeither || d.Judgement == delivery.JudgedUnchanged {
return nil, false
}
if m.Allowed != nil && m.Allowed(node, d.Digest) {
return nil, false
}
return &RecordNeed{Node: node, Digest: d.Digest, Judgement: d.Judgement, Words: d.Words}, true
}
// recorded says whether the module is of policy record: a person's choice at the assigner over the manifest's word (ADR
// 0565 §7, catalogue.Upgrade); for a module with none, as the controller holds it beside the composition's digest until the
// lease write (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
if m.Chosen != nil {
policy, _, _, _ := catalogue.Upgrade(m.Manifest, m.Chosen)
return policy == catalogue.PolicyRecord
}
if d.Policy != "" {
return d.Policy == delivery.PolicyRecord
}
policy, _, _ := catalogue.DerivedUpgrade(m.Manifest)
return policy == catalogue.PolicyRecord
}
// userListRevision is the want that names the bus module's composition at the fingerprint the node runs when that composition
// differs from the one the node runs only in the bus's user list (ComposedDigest.UserListOnly), and whether there is one. A node
// whose want names another fingerprint than the one it runs, or a module that does not provide the bus, has none.
func (m Module) userListRevision(node string, before delivery.Want, had bool, n Nodes) (delivery.Want, bool) {
if !catalogue.ProvidesBus(m.Manifest) {
return delivery.Want{}, false
}
running, ran := n.Haves[node][m.Name]
if !ran || running.FP == "" || (had && (before.Removed || before.FP != running.FP)) {
return delivery.Want{}, false
}
d, ok := m.composition(node, running.FP)
if !ok || !d.UserListOnly {
return delivery.Want{}, false
}
w := delivery.Want{FP: running.FP, Composed: d.Digest}
if had && before.Composed == d.Digest {
w.Backup, w.Words = before.Backup, before.Words
}
return w, true
}
// GrantNamesNode says whether a grant of module names its accounts on node (mesh-sdk's Grant.NamesNode, which a
// put reads by too).
func GrantNamesNode(g delivery.Grant, node, module string) bool {
return g.NamesNode(node, module)
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,331 @@
// Package judge is how the seat mesh-placement's holder judges a proposal (novox/hq ADR 0567 §3, §4, as ADR 0593 §8
// changes them): what it changes on each node by its placement (Intended), every node answering it, and the verdict of
// their answers (JudgeAnswers). The assigner's own (ADR 0593 §1), moved back from mesh-sdk's compose/judge: the holder
// composes nothing, since every composition is its node's engine's, and a node answers through the candidates path
// with what it composed. It holds no bus client and no timer: the holder gives it what it read.
package judge
import (
"encoding/json"
"fmt"
"reflect"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// AnswerBound is how long the holder waits for a node's answer after its candidate was written: the time to live of the
// node's pulse key (delivery.PulseAnswerKey), whose expiry names it as giving no answer, never passed. The same answer
// bound a check gives a node (ADR 0593 §8, §7: 60 seconds, delivery.ComposeAnswerBound). A variable so a test on the bus
// can shorten it.
var AnswerBound = delivery.ComposeAnswerBound
// Intended is, by node, the modules whose placed entry the proposal changes there, and the nodes that answer it: every
// node (ADR 0593 §8, changing ADR 0567 §4's list of the nodes that must answer), sorted. current is the placement as it
// stands, laid the proposal's placements by node (nil removes the node's key) and nodes every node with a record. A
// node's set holds the modules the proposal adds, removes or whose placed entry differs on that node; a node with no
// key now gets an empty set (§7: the seed is answered as the same). What a node's compositions change beyond its own
// placement it says itself in its answer, marked in the brief as not computed.
func Intended(current map[string]delivery.Placement, laid map[string]*delivery.Placement, nodes []string) (intended map[string]map[string]bool, must []string) {
intended = map[string]map[string]bool{}
all := map[string]bool{}
for _, n := range nodes {
all[n] = true
}
for n := range laid {
all[n] = true
}
for node := range all {
must = append(must, node)
set := map[string]bool{}
intended[node] = set
before, placed := current[node]
after, changesKey := laid[node]
if !changesKey || !placed {
continue // untouched, or new to placement: every answer must be the same
}
var afterModules []delivery.PlacedModule
if after != nil {
afterModules = after.Modules
}
entries := map[string]delivery.PlacedModule{}
for _, m := range before.Modules {
entries[m.Module] = m
}
seen := map[string]bool{}
for _, m := range afterModules {
seen[m.Module] = true
if e, ok := entries[m.Module]; !ok || !reflect.DeepEqual(e, m) {
set[m.Module] = true
}
}
for name := range entries {
if !seen[name] {
set[name] = true
}
}
}
sort.Strings(must)
return intended, must
}
// JudgeAnswers is the verdict of a proposal's answers (§4). answers is by node its HaveNode.Candidates entry for the
// proposal (nil when it has none) and revisions by node the revision of its candidate key: an answer counts when it
// names the proposal and that revision. It fails on a refusal by a node that must answer; a node of Must with no answer
// that counts is waited for while its pulse key stands, and once it has ended (pulsing false: the bound's expiry, heard
// on the bus) is named as giving no answer and the proposal decided on the answers it has (ADR 0593 §8).
// A difference fails nothing: one Intended computed is listed under Intended, one it did not under Unintended, both
// shown in the ask's brief. The seed alone fails on any difference.
func JudgeAnswers(p delivery.PlacementProposal, intended map[string]map[string]bool, answers map[string]*delivery.CandidateAnswer,
revisions map[string]uint64, online map[string]bool, pulsing map[string]bool, now time.Time) delivery.PlacementVerdict {
v := delivery.PlacementVerdict{At: now.UTC(), Intended: map[string][]string{}, Unintended: map[string][]string{}, Carried: map[string][]string{},
Answers: map[string]delivery.CandidateAnswer{}}
for node, set := range intended {
for m := range set {
v.Intended[node] = append(v.Intended[node], m)
}
sort.Strings(v.Intended[node])
}
must := p.Must
if len(must) == 0 {
must = p.Nodes()
}
for _, node := range must {
a := answers[node]
if a == nil || a.Proposal != p.ID || a.Revision == 0 || a.Revision != revisions[node] {
v.Silent = append(v.Silent, node)
continue
}
v.Answers[node] = *a
if a.Refused != "" {
v.Fails = append(v.Fails, fmt.Sprintf("%s cannot compose the proposal: %s", node, a.Refused))
continue
}
for _, r := range a.Refusals {
v.Fails = append(v.Fails, fmt.Sprintf("%s refuses the proposal for %s: %s", node, r.Module, r.Why))
}
want := intended[node]
differs := func(module, words string) {
if p.Seed {
// The seed carries what its nodes found, each named in the brief for the operator who approves it.
v.Carried[node] = append(v.Carried[node], words)
return
}
if !want[module] {
v.Unintended[node] = append(v.Unintended[node], module)
}
}
for _, d := range a.Differs {
differs(d.Module, differenceWords(d))
}
for _, d := range a.Added {
differs(d.Module, "would run "+d.Module)
}
for _, m := range a.Removed {
differs(m, "would stop running "+m)
}
sort.Strings(v.Unintended[node])
}
if len(v.Intended) == 0 {
v.Intended = nil
}
if len(v.Unintended) == 0 {
v.Unintended = nil
}
if len(v.Carried) == 0 {
v.Carried = nil
}
if len(v.Answers) == 0 {
v.Answers = nil
}
var waiting, gone []string
for _, node := range v.Silent {
named := node
if online != nil && !online[node] {
named += " (offline)"
}
if pulsing[node] {
waiting = append(waiting, named)
} else {
gone = append(gone, named)
}
}
if len(waiting) > 0 {
v.Notes = append(v.Notes, fmt.Sprintf("waits for the answers of %s, each for %s from its candidate's write",
strings.Join(waiting, ", "), AnswerBound))
}
if len(gone) > 0 {
// ADR 0593 §8: decided on the answers it has; that node's own composition judges again when it applies (§9).
v.Notes = append(v.Notes, fmt.Sprintf("no answer from %s within %s of its candidate's write: decided on the answers "+
"it has, and that node's own composition judges again when it applies", strings.Join(gone, ", "), AnswerBound))
}
switch {
case len(v.Fails) > 0:
v.State = delivery.VerdictFail
case len(waiting) > 0:
v.State = delivery.VerdictWaits
default:
v.State = delivery.VerdictPass
}
return v
}
// Brief is the verdict as the ask's brief shows it, by node (§5): what the proposal changes and what each node answered
// beside it, marked, and what stays held as the node holds it today, with the proposal's why.
func Brief(p delivery.PlacementProposal, v delivery.PlacementVerdict) string {
var b strings.Builder
fmt.Fprintf(&b, "%s (proposed by %s)", p.Why, p.By)
nodes := map[string]bool{}
for n := range v.Intended {
nodes[n] = true
}
for n := range v.Unintended {
nodes[n] = true
}
for _, n := range p.Nodes() {
nodes[n] = true
}
for n, a := range v.Answers {
if len(a.Held) > 0 {
nodes[n] = true
}
}
for n := range v.Carried {
nodes[n] = true
}
names := make([]string, 0, len(nodes))
for n := range nodes {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
fmt.Fprintf(&b, "; %s:", n)
if m := v.Intended[n]; len(m) > 0 {
fmt.Fprintf(&b, " changes %s", strings.Join(m, ", "))
} else {
b.WriteString(" composes the same")
}
if m := v.Unintended[n]; len(m) > 0 {
fmt.Fprintf(&b, "; also differs, not computed: %s", strings.Join(m, ", "))
}
if m := v.Carried[n]; len(m) > 0 {
fmt.Fprintf(&b, "; the seed carries: %s", strings.Join(m, "; "))
}
if held := v.Answers[n].Held; len(held) > 0 {
var names []string
for _, h := range held {
today := h.Today
if today == "" {
today = "held"
}
names = append(names, h.Module+" ("+today+")")
}
fmt.Fprintf(&b, "; stays as today: %s", strings.Join(names, ", "))
}
}
return b.String()
}
// differenceWords names one module's difference: the module, the first field that differs in it with its value as it
// runs and as the seed composes it, each cut to about 60 characters ("(value cut)" where a body shown is cut before the
// field), and how many fields more; or the resource and why when no body is shown.
func differenceWords(d delivery.ModuleDiffers) string {
for _, b := range d.Bodies {
if len(b.Paths) == 0 {
continue
}
path := b.Paths[0]
words := fmt.Sprintf("%s: %s %s → %s", d.Module, fieldName(b.Running, b.Composed, path),
valueAt(b.Running, path), valueAt(b.Composed, path))
if n := len(b.Paths) - 1; n > 0 {
words += fmt.Sprintf(" (and %d more fields)", n)
}
return words
}
if d.Resource != "" {
return fmt.Sprintf("%s: %s (%s)", d.Module, d.Resource, d.Why)
}
return fmt.Sprintf("%s: %s", d.Module, d.Why)
}
// shownValue bounds a value named in a brief.
const shownValue = 60
// walk follows a differing path (delivery.DifferingPaths: keys joined by dots, a key may hold dots, and a JSON value
// written in a string is read as JSON) through a body, answering the keys it passed and the value at its end.
func walk(body, path string) (keys []string, value any, ok bool) {
var v any
if json.Unmarshal([]byte(body), &v) != nil {
return nil, nil, false
}
rest := path
for rest != "" {
if s, isString := v.(string); isString {
var inner any
if json.Unmarshal([]byte(s), &inner) != nil {
return nil, nil, false
}
v = inner
}
switch t := v.(type) {
case map[string]any:
best := ""
for k := range t {
if (rest == k || strings.HasPrefix(rest, k+".")) && len(k) > len(best) {
best = k
}
}
if best == "" {
return nil, nil, false
}
keys, v, rest = append(keys, best), t[best], strings.TrimPrefix(strings.TrimPrefix(rest, best), ".")
case []any:
seg, after, _ := strings.Cut(rest, ".")
n, err := strconv.Atoi(seg)
if err != nil || n < 0 || n >= len(t) {
return nil, nil, false
}
keys, v, rest = append(keys, seg), t[n], after
default:
return nil, nil, false
}
}
return keys, v, true
}
// valueAt is the value at path in a shown body, cut to shownValue, or why it is not shown.
func valueAt(body, path string) string {
if body == "" {
return "(none)"
}
_, v, ok := walk(body, path)
if !ok {
return "(value cut)"
}
s, isString := v.(string)
if !isString {
raw, _ := json.Marshal(v)
s = string(raw)
}
if r := []rune(s); len(r) > shownValue {
s = string(r[:shownValue-1]) + "…"
}
return s
}
// fieldName is the differing field as a person reads it: its parent's name and its own, `PARENT[key]`, from whichever
// body reaches it; the path's end when neither does.
func fieldName(running, composed, path string) string {
for _, body := range []string{running, composed} {
if keys, _, ok := walk(body, path); ok && len(keys) > 0 {
if len(keys) == 1 {
return keys[0]
}
return keys[len(keys)-2] + "[" + keys[len(keys)-1] + "]"
}
}
return path
}
@@ -0,0 +1,91 @@
package judge
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0567 §3, §4 (rows moved from the assigner's check of a pull request's placement, ADR 0559 §5): what a proposal
// changes by node, and the verdict of the answers: differences shown and failing nothing, a refusal and a silent node
// failing, a later answer judging again, the seed failing on any difference.
func TestJudge(t *testing.T) {
current := map[string]delivery.Placement{"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail"}}}, "server": {Node: "server"}}
laid := map[string]*delivery.Placement{
"anchor": {Node: "anchor", Modules: []delivery.PlacedModule{{Module: "mail", Ports: map[string]int{"25": 2525}}, {Module: "web"}}},
"laptop": {Node: "laptop", Modules: []delivery.PlacedModule{{Module: "web"}}}}
// ADR 0593 §8: every node answers; what the placement changes is read from the placement alone.
got, must := Intended(current, laid, []string{"anchor", "laptop", "server"})
if !got["anchor"]["mail"] || !got["anchor"]["web"] || len(got["laptop"]) != 0 || len(got["server"]) != 0 ||
!slices.Equal(must, []string{"anchor", "laptop", "server"}) {
t.Errorf("intended = %v, must %v: want anchor's mail and web, nothing elsewhere, every node answering", got, must)
}
at := time.Date(2026, 10, 27, 2, 0, 0, 0, time.UTC)
p := delivery.PlacementProposal{ID: "p-one", By: "op", Why: "move web", Written: at, Base: map[string]uint64{"anchor": 3, "laptop": 0},
Must: []string{"anchor", "laptop"}}
revisions := map[string]uint64{"anchor": 7, "laptop": 8}
intended := map[string]map[string]bool{"anchor": {"web": true}}
same := func(rev uint64) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateSame, Same: []string{"mail"}}
}
differs := func(rev uint64, module string) *delivery.CandidateAnswer {
return &delivery.CandidateAnswer{Proposal: "p-one", Revision: rev, Verdict: delivery.CandidateDiffers,
Differs: []delivery.ModuleDiffers{{Module: module, Why: "the port differs"}}}
}
for _, row := range []struct {
name string
seed bool
answers map[string]*delivery.CandidateAnswer
pulsing map[string]bool
state string
words string
marked string
}{
{"all the same", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a module added where intended", false, map[string]*delivery.CandidateAnswer{"anchor": {Proposal: "p-one", Revision: 7, Verdict: delivery.CandidateDiffers,
Added: []delivery.ModuleAdded{{Module: "web", Unissued: []string{"its own secret"}}}}, "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
{"a difference not computed passes, marked", false, map[string]*delivery.CandidateAnswer{"anchor": differs(7, "mail"), "laptop": same(8)}, nil, delivery.VerdictPass, "", "mail"},
{"the seed carries a difference, named", true, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateDiffers, Differs: []delivery.ModuleDiffers{{Module: "node-tools", Resource: "node-tools.runtime", Why: "the resource differs",
Bodies: []delivery.ResourceBodies{{ID: "node-tools.runtime", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf", "env.MESH_TOOL_MODULES"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/hosts\":\"hostname\",\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}","MESH_TOOL_MODULES":"… (cut at 8192 bytes)`}}}}}}, nil, delivery.VerdictPass, "", ""},
{"a refusal", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8, Verdict: delivery.CandidateRefused,
Refusals: []delivery.ModuleRefused{{Module: "web", Why: "no own secret"}}}}, nil, delivery.VerdictFail, "laptop refuses", ""},
{"a node not answering, within its bound", false, map[string]*delivery.CandidateAnswer{"anchor": same(7)}, map[string]bool{"laptop": true}, delivery.VerdictWaits, "laptop (offline)", ""},
{"an answer of an earlier revision, past its bound, is no answer and decided without it", false, map[string]*delivery.CandidateAnswer{"anchor": same(6), "laptop": same(8)}, nil, delivery.VerdictPass, "no answer from anchor", ""},
{"a module held as today passes, shown in the brief", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": {Proposal: "p-one", Revision: 8,
Verdict: delivery.CandidateSame, Held: []delivery.ModuleRefused{{Module: "spiegel", Why: "a required value is unset"}}}}, nil, delivery.VerdictPass, "", ""},
{"its later answer", false, map[string]*delivery.CandidateAnswer{"anchor": same(7), "laptop": same(8)}, nil, delivery.VerdictPass, "", ""},
} {
q := p
q.Seed = row.seed
v := JudgeAnswers(q, intended, row.answers, revisions, map[string]bool{"anchor": true}, row.pulsing, at)
all := strings.Join(append(append([]string{}, v.Fails...), v.Notes...), "; ")
if v.State != row.state || !strings.Contains(all, row.words) {
t.Errorf("%s: %s, %q", row.name, v.State, all)
}
if row.seed && row.state == delivery.VerdictPass && !strings.Contains(Brief(q, v), "the seed carries: node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → (value cut) (and 1 more fields)") {
t.Errorf("%s: the brief does not name it: %s", row.name, Brief(q, v))
}
if row.seed && row.state == delivery.VerdictPass {
whole := row.answers["laptop"].Differs[0]
whole.Bodies = []delivery.ResourceBodies{{ID: "r", Paths: []string{"env.MESH_OWNED_PATHS./etc/resolv.conf"},
Running: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"networkmanager (node-uplink)\"}"}}`,
Composed: `{"env":{"MESH_OWNED_PATHS":"{\"/etc/resolv.conf\":\"systemd-resolved (node-resolver)\"}"}}`}}
if got := differenceWords(whole); got != "node-tools: MESH_OWNED_PATHS[/etc/resolv.conf] networkmanager (node-uplink) → systemd-resolved (node-resolver)" {
t.Errorf("the difference is named %q", got)
}
}
if strings.HasPrefix(row.name, "a module held as today") && !strings.Contains(Brief(q, v), "stays as today: spiegel (held)") {
t.Errorf("%s: the brief does not show it: %s", row.name, Brief(q, v))
}
if row.marked != "" && (!slices.Contains(v.Unintended["anchor"], row.marked) || !strings.Contains(Brief(q, v), "not computed: "+row.marked)) {
t.Errorf("%s: the difference is not marked: %+v, %s", row.name, v.Unintended, Brief(q, v))
}
}
}
@@ -0,0 +1,95 @@
package manifest
import (
"os"
"reflect"
"slices"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func load(t *testing.T) catalogue.Manifest {
t.Helper()
raw, err := os.ReadFile("module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.Decode(raw)
if err != nil {
t.Fatalf("mesh-sdk cannot decode the manifest: %v", err)
}
return m
}
// 1297/B2a: the controller's reader (mesh-sdk catalogue, which the controller reads manifests with) takes the
// manifest; it reads exactly the buckets the assigner opens (the program's list, held equal in its own test), keeps
// the states it declares (the differences kept read-only, 1906/D5), and holds a credential of its own.
func TestManifest(t *testing.T) {
m := load(t)
if m.Module != "mesh-delivery-assigner" {
t.Errorf("module = %q", m.Module)
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
// ADR 0593 §8: it composes nothing, so it reads no candidate, held or resolved value, compose step or build log; it reads
// every node's published compositions to compare them, and the gate's source and merges, for the release that carries a build (ADR 0601 §1).
want := []string{"mesh-delivery-builder.builds", "mesh-delivery-gate.merges", "mesh-delivery-gate.source", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "mesh-host.composed",
"mesh-provisioner.provisions", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
states := map[string]int{}
for _, s := range m.State {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
t.Errorf("own-secrets = %v, want exactly broker (or no credential is issued)", m.OwnSecrets)
}
if m.SecretsOwner != "mesh-delivery-assigner" {
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1906/D3: it says its conditions on the operator channel; it asks the Go build seat nothing since the composition is
// the node-engine's (ADR 0593 §7).
if !slices.Equal(m.Uses, []string{"operator-channel"}) {
t.Errorf("uses = %v, want exactly operator-channel", m.Uses)
}
// ADR 0546: it serves its own tools (put, release, puts) from its bundle through the tool runner (the artifact loads the
// module), says each put's step as the event put, and calls the hand-act seat and the bus node's backup holder.
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its five verbs (adopt and hand-back went with the
// controller, ADR 0590); keeps the seat's events on its stream placement for 30 days; calls the hand-act log; and reads
// mesh-config's resolved values to compose every node a proposal touches.
// ADR 0565 §7, §9 (issue 1953 D4): and the seat mesh-wants, serving upgrade and bus.
if len(m.Claims) != 2 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again"}) ||
m.Claims[1].Name != catalogue.WantsSeat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its five verbs and mesh-wants with its two", m.Claims)
}
if seat, _ := catalogue.SeatNamed(catalogue.WantsSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold mesh-wants: %v", catalogue.CanHold(m, seat))
}
if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat))
}
if !slices.Equal(m.Tools, []string{"assigner_put", "assigner_release", "assigner_puts"}) {
t.Errorf("the assigner serves tools %v, want its three put tools", m.Tools)
}
if len(m.Streams) != 1 || m.Streams[0].Name != delivery.PlacementStream || m.Streams[0].MaxAgeDays != delivery.PlacementStreamDays ||
!slices.Equal(slices.Sorted(slices.Values(m.Streams[0].Events)), slices.Sorted(slices.Values(delivery.PlacementEvents))) {
t.Errorf("streams = %+v, want placement carrying the seat's four events for %d days", m.Streams, delivery.PlacementStreamDays)
}
if want := []string{
"seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-acts", "seat:mesh-hand-acts.warranted", // ADR 0565 §9, §11: the seat alone
"seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
if !slices.Equal(m.Emits, []string{"put"}) {
t.Errorf("emits = %v, want put", m.Emits)
}
}
+134
View File
@@ -0,0 +1,134 @@
{
"module": "mesh-delivery-assigner",
"version": "1",
"slug": "assigner",
"claims": [
{
"name": "mesh-placement",
"scope": "mesh",
"serves": [
"placement",
"proposals",
"propose",
"withdraw",
"again"
]
},
{
"name": "mesh-wants",
"scope": "mesh",
"serves": [
"upgrade",
"bus"
]
}
],
"reads": [
"mesh-delivery-builder.builds",
"mesh-delivery-gate.merges",
"mesh-delivery-gate.source",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"mesh-grants.grants",
"mesh-host.composed",
"mesh-provisioner.provisions",
"node-engine.have"
],
"uses": [
"operator-channel"
],
"invokes": [
"seat:mesh-hand-acts.hand-act",
"seat:mesh-hand-acts.hand-acts",
"seat:mesh-hand-acts.warranted",
"seat:node-backup.backed-up",
"seat:node-backup.now"
],
"emits": [
"put"
],
"tools": [
"assigner_put",
"assigner_release",
"assigner_puts"
],
"state": [
{ "name": "placement", "history": 64 },
{ "name": "proposals", "history": 5 },
{ "name": "candidates", "history": 5 },
{ "name": "pulses", "key-ttl": true },
{ "name": "asks", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 },
{ "name": "findings", "history": 5 },
{ "name": "policies", "history": 20 },
{ "name": "seat-holders", "history": 5 }
],
"streams": [
{
"name": "placement",
"events": [
"placement-proposed",
"placement-judged",
"placement-changed",
"placement-withdrawn"
],
"max-age-days": 30
}
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
"secrets-owner": "mesh-delivery-assigner",
"resources": [
{
"id": "account",
"type": "user",
"name": "mesh-delivery-assigner",
"shell": "/usr/bin/nologin",
"home": "/var/lib/mesh-delivery-assigner"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "mesh-delivery-assigner",
"place": ".",
"unknown": "program"
},
{
"id": "assigner",
"type": "process",
"name": "mesh-delivery-assigner",
"artifact": "program",
"run": [
"./mesh-delivery-assigner"
],
"env": {
"MESH_BROKER_FILE": "${dir:state}/broker"
},
"user": "mesh-delivery-assigner",
"health": {
"kind": "unit"
}
}
],
"build": {
"artifacts": [
{
"name": "program",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/mesh-delivery-assigner",
"binary": "mesh-delivery-assigner",
"loads": [
"mesh-delivery-assigner"
]
}
]
}
}
@@ -0,0 +1,316 @@
package buildrun
import (
"context"
"errors"
"fmt"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **An asker puts an ask on its build seat's stream and waits for its step** (novox/hq ADR 0525 §1, §6; issue 1887 A4,
// A5). It chooses no node: whichever build node pulls the ask writes `<node>.<ask id>` in its build module's steps state,
// and the asker reads the first such key that is not running. The builder and the checker share it.
// AskPublisher puts an encoded ask on subject, de-duplicated by its id.
type AskPublisher interface {
PublishAsk(ctx context.Context, subject, id string, data []byte) error
}
// AskWaiter watches an ask's steps: every value a key `*.<ask id>` of the build module's steps state takes, running or
// ended, on the channel it answers, until ctx ends. A watch, not a poll.
type AskWaiter interface {
WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error)
}
// Asker puts asks and waits for their steps.
type Asker struct {
Publisher AskPublisher
Waiter AskWaiter
// Bound is how long a running step (a build node took the ask, its key written running) may stand still, with no new phase,
// before the asker stops waiting; a step that keeps moving is waited for up to 3 Bounds from its start (issue 1930); zero waits as long as ctx. An ask waiting in its stream is not bounded: the stream's backlog is no fault
// (ADR 0525 §6, §9), and the bound starts again when another node takes the ask.
Bound time.Duration
}
// Put puts a on its stream: the seat's accept subject of its language and kind.
func (k Asker) Put(ctx context.Context, a delivery.StepAsk) error {
raw, err := delivery.EncodeStepAsk(a)
if err != nil {
return err
}
subject, err := delivery.AskSubject(a.Language, a.Kind)
if err != nil {
return err
}
if err := k.Publisher.PublishAsk(ctx, subject, a.ID, raw); err != nil {
return fmt.Errorf("putting the %s ask %s: %w", a.Kind, a.ID, err)
}
return nil
}
// Wait waits for the step of an ask already put (a builder that started again takes over its own asks with it, never
// putting them twice) and answers it when it ended, whatever its state. The bound runs from the step's start on the node
// that took the ask, read from its running key, so an ask that waits in the stream for a free slot waits as long as ctx.
func (k Asker) Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) {
module, err := delivery.BuildModuleOf(a.Language)
if err != nil {
return delivery.Step{}, err
}
watch, cancel := context.WithCancel(ctx)
defer cancel()
steps, err := k.Waiter.WatchAsk(watch, module, a.ID)
if err != nil {
return delivery.Step{}, err
}
var bound, outerCap <-chan time.Time
var timer, capTimer *time.Timer
defer func() {
if timer != nil {
timer.Stop()
}
if capTimer != nil {
capTimer.Stop()
}
}()
running, since, phase, phaseAt := "", time.Time{}, "", time.Time{}
standStill := func() error {
if phase == "" {
return fmt.Errorf("the ask %s's step on %s did not end within %s of its start: it stood still, saying no phase", a.ID, running, k.Bound)
}
return fmt.Errorf("the ask %s's step on %s did not end within %s: it stood still in %q since %s",
a.ID, running, k.Bound, phase, phaseAt.UTC().Format(time.RFC3339))
}
for {
select {
case <-ctx.Done():
return delivery.Step{}, ctx.Err()
case <-bound:
return delivery.Step{}, standStill()
case <-outerCap:
return delivery.Step{}, fmt.Errorf("the ask %s's step on %s did not end within %s of its start: it kept moving (last phase %q at %s) but never ended",
a.ID, running, 3*k.Bound, phase, phaseAt.UTC().Format(time.RFC3339))
case step, ok := <-steps:
if !ok {
return delivery.Step{}, errors.New("the watch of the ask's step ended")
}
if seen := stepSeenOf(ctx); seen != nil {
seen(step)
}
if step.State != delivery.StepRunning {
return step, nil
}
if k.Bound <= 0 {
continue
}
newRun := step.Node != running || !step.Started.Equal(since)
if !newRun && step.Phase == phase && step.PhaseAt.Equal(phaseAt) {
continue
}
// A node took the ask, or took it again after its run stopped (another node, or the same node's build module
// replaced mid-run: a new start), or its step entered a phase. The bound is on standing still: it runs from the
// step's last movement (its phase's start, else its start), so a step that keeps moving is waited for, up to
// three bounds from its start (issue 1930).
running, since, phase, phaseAt = step.Node, step.Started, step.Phase, step.PhaseAt
last := step.Started
if step.PhaseAt.After(last) {
last = step.PhaseAt
}
left := k.Bound
if !last.IsZero() {
left = time.Until(last.Add(k.Bound))
}
if timer != nil {
timer.Stop()
}
timer = time.NewTimer(max(left, 0))
bound = timer.C
if newRun {
if capTimer != nil {
capTimer.Stop()
}
outer := 3 * k.Bound
if !step.Started.IsZero() {
outer = time.Until(step.Started.Add(3 * k.Bound))
}
capTimer = time.NewTimer(max(outer, 0))
outerCap = capTimer.C
}
}
}
}
type stepSeenKey struct{}
// WithStepSeen is ctx carrying seen, which Asker.Wait hands every value the ask's step key takes while it waits: the node
// that took it, each phase it says while running (delivery.Step.Phase), and its end. A try names its asks' progress so
// (issue 1890). seen runs in the waiting goroutine and must not block.
func WithStepSeen(ctx context.Context, seen func(delivery.Step)) context.Context {
return context.WithValue(ctx, stepSeenKey{}, seen)
}
func stepSeenOf(ctx context.Context) func(delivery.Step) {
seen, _ := ctx.Value(stepSeenKey{}).(func(delivery.Step))
return seen
}
// Ask puts a and waits for its step; a step that did not pass is an error naming its node, its state and its reason.
func (k Asker) Ask(ctx context.Context, a delivery.StepAsk) (delivery.Step, error) {
if err := k.Put(ctx, a); err != nil {
return delivery.Step{}, err
}
return k.Passed(k.Wait(ctx, a))
}
// Passed is step when it passed, else an error naming where it ended and why.
func (k Asker) Passed(step delivery.Step, err error) (delivery.Step, error) {
if err != nil {
return step, err
}
if step.State != delivery.StepPassed {
return step, fmt.Errorf("the ask %s on %s %s: %s", step.ID, step.Node, step.State, step.Reason)
}
return step, nil
}
// JetStreamAsks is the bus side of an Asker: a JetStream publish de-duplicated by the ask's id (Nats-Msg-Id), and a watch
// of `*.<ask id>` in the build module's steps bucket.
type JetStreamAsks struct {
JS jetstream.JetStream
}
// PublishAsk is AskPublisher's.
func (s JetStreamAsks) PublishAsk(ctx context.Context, subject, id string, data []byte) error {
msg := &nats.Msg{Subject: subject, Data: data, Header: nats.Header{}}
msg.Header.Set(jetstream.MsgIDHeader, id)
_, err := s.JS.PublishMsg(ctx, msg)
return err
}
// WatchAsk is AskWaiter's: the watch hands over what the keys hold now, so a step that ended before it began is read at
// once, and every change after.
func (s JetStreamAsks) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
pattern, err := delivery.AskStepWatch(askID)
if err != nil {
return nil, err
}
kv, err := s.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
w, err := kv.Watch(ctx, pattern)
if err != nil {
return nil, err
}
out := make(chan delivery.Step)
go func() {
defer close(out)
defer w.Stop()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
return
}
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
step, err := delivery.DecodeStep(e.Value())
if err != nil {
continue // not a step (a node's pause key never matches an ask id; anything else is not this ask's)
}
select {
case out <- step:
case <-ctx.Done():
return
}
}
}
}()
return out, nil
}
// FakeAsks is a test double of the bus for an asker's tests: each ask put is kept and, when Answer is set, its step is
// written at once with what Answer says a build node wrote (running or ended). Write writes a step later. Safe for
// concurrent use.
type FakeAsks struct {
Answer func(a delivery.StepAsk) delivery.Step
mu sync.Mutex
asks []delivery.StepAsk
steps map[string][]delivery.Step
watchers map[string][]chan delivery.Step
}
// PublishAsk is AskPublisher's.
func (f *FakeAsks) PublishAsk(_ context.Context, _, id string, data []byte) error {
a, err := delivery.DecodeStepAsk(data)
if err != nil {
return err
}
f.mu.Lock()
f.asks = append(f.asks, a)
f.mu.Unlock()
if f.Answer != nil {
f.Write(f.Answer(a))
}
return nil
}
// Write writes an ask's step, as a build node does.
func (f *FakeAsks) Write(step delivery.Step) {
f.mu.Lock()
if f.steps == nil {
f.steps, f.watchers = map[string][]delivery.Step{}, map[string][]chan delivery.Step{}
}
f.steps[step.ID] = append(f.steps[step.ID], step)
ws := append([]chan delivery.Step(nil), f.watchers[step.ID]...)
f.mu.Unlock()
for _, w := range ws {
w <- step
}
}
// End is Write: kept for the tests that end a step.
func (f *FakeAsks) End(step delivery.Step) { f.Write(step) }
// WatchAsk is AskWaiter's: what was written so far, then every write.
func (f *FakeAsks) WatchAsk(ctx context.Context, _ string, askID string) (<-chan delivery.Step, error) {
out := make(chan delivery.Step, 64)
f.mu.Lock()
if f.steps == nil {
f.steps, f.watchers = map[string][]delivery.Step{}, map[string][]chan delivery.Step{}
}
for _, s := range f.steps[askID] {
out <- s
}
f.watchers[askID] = append(f.watchers[askID], out)
f.mu.Unlock()
go func() {
<-ctx.Done()
f.mu.Lock()
ws := f.watchers[askID]
for i, w := range ws {
if w == out {
f.watchers[askID] = append(ws[:i], ws[i+1:]...)
break
}
}
f.mu.Unlock()
}()
return out, nil
}
// Asks are the asks put so far.
func (f *FakeAsks) Asks() []delivery.StepAsk {
f.mu.Lock()
defer f.mu.Unlock()
return append([]delivery.StepAsk(nil), f.asks...)
}
@@ -0,0 +1,61 @@
//go:build integration
package buildrun
import (
"context"
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// 1887/A4a, 1887/AC3 on the real bus: an ask put twice under one id is one message, and the asker reads the step a node
// wrote under `<node>.<ask id>`.
func TestAnAskIsPutOnceAndItsStepIsRead(t *testing.T) {
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, _ := jetstream.New(nc)
ctx := context.Background()
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "SEAT_NODE_BUILD_GO", Subjects: []string{"mesh.seat.node-build-go.accept.>"},
Retention: jetstream.WorkQueuePolicy}); err != nil {
t.Fatal(err)
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket("mesh-build-go")})
if err != nil {
t.Fatal(err)
}
bus := JetStreamAsks{JS: js}
k := Asker{Publisher: bus, Waiter: bus, Bound: 5 * time.Second}
a := goAsk("mvl0frv18d1960")
for i := 0; i < 2; i++ {
if err := k.Put(ctx, a); err != nil {
t.Fatal(err)
}
}
s, _ := js.Stream(ctx, "SEAT_NODE_BUILD_GO")
if info, _ := s.Info(ctx); info.State.Msgs != 1 {
t.Fatalf("%d messages for one ask put twice", info.State.Msgs)
}
go func() {
time.Sleep(100 * time.Millisecond)
key, _ := delivery.AskStepKey("ace", a.ID)
running, _ := delivery.EncodeStep(delivery.Step{ID: a.ID, Tool: "delivery ask", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
_, _ = kv.Put(ctx, key, running)
passed, _ := delivery.EncodeStep(delivery.Step{ID: a.ID, Tool: "delivery ask", Node: "ace", State: delivery.StepPassed, Started: time.Now(),
Result: json.RawMessage(`{"artifacts":[]}`)})
_, _ = kv.Put(ctx, key, passed)
}()
step, err := k.Passed(k.Wait(ctx, a))
if err != nil || step.Node != "ace" {
t.Fatalf("step %+v, %v", step, err)
}
}
@@ -0,0 +1,234 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
func goAsk(id string) delivery.StepAsk {
return delivery.StepAsk{ID: id, Kind: delivery.AskDelivery, Language: delivery.AskGo, Asker: "mesh-delivery-builder", At: time.Now(),
Build: &delivery.BuildJob{Module: "m", Path: ".", Fetch: map[string]any{"repository": "novox/x", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "program"}}}}
}
// 1887/A4a: an asker puts the ask once and answers the step a build node wrote for it, naming the node; one that did not
// pass is an error naming the node and why; a wait of an ask already put puts nothing.
func TestAnAskerPutsOnceAndReadsTheStep(t *testing.T) {
f := &FakeAsks{Answer: func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Node: "ace", State: delivery.StepPassed, Result: []byte(`{"artifacts":[]}`)}
}}
k := Asker{Publisher: f, Waiter: f}
var seen []delivery.Step // issue 1890: a waiter that asks to see each value of the step sees it
step, err := k.Ask(WithStepSeen(context.Background(), func(s delivery.Step) { seen = append(seen, s) }), goAsk("mvl0frv18d1957"))
if err != nil || step.Node != "ace" || len(f.Asks()) != 1 || len(seen) != 1 || seen[0].Node != "ace" {
t.Fatalf("step %+v, %v, asks %d, seen %+v", step, err, len(f.Asks()), seen)
}
f.Answer = func(a delivery.StepAsk) delivery.Step {
return delivery.Step{ID: a.ID, Node: "shanks", State: delivery.StepRefused, Reason: "go 1.27 installed"}
}
if _, err := k.Ask(context.Background(), goAsk("mvl0frv18d1958")); err == nil || !strings.Contains(err.Error(), "shanks refused: go 1.27 installed") {
t.Fatalf("a refused step: %v", err)
}
// review round 2 item 1: an ask that waits in its stream for a free slot is not bounded; once a node writes its step
// running, the step is bounded from its start, and the error names the node.
g := &FakeAsks{}
k = Asker{Publisher: g, Waiter: g, Bound: 20 * time.Millisecond}
waiting, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
if _, err := k.Wait(waiting, goAsk("mvl0frv18d1959")); err != context.DeadlineExceeded {
t.Fatalf("an ask no node took: %v, want it still waiting when ctx ended", err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1961", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1961")); err == nil || !strings.Contains(err.Error(), "on ace did not end within") {
t.Fatalf("a step that never ends: %v", err)
}
if len(g.Asks()) != 0 {
t.Fatal("a wait put an ask")
}
// Issue 1930: the bound is on a step that stands still, not on its age. A step past one bound from its start that entered a
// new phase inside the bound is waited for until it ends; one that stood a whole bound in a phase fails at once, naming
// the phase; one three bounds from its start ends, said as kept moving. The steps' own times stand for the clock, so the
// bound is an hour no loaded machine reaches and no case waits on a pause (the check of 2026-11-02 22:35 tripped one
// of 60 ms).
k = Asker{Publisher: g, Waiter: g, Bound: time.Hour}
now := time.Now()
g.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "ace", State: delivery.StepRunning, Started: now.Add(-90 * time.Minute), Phase: "pb", PhaseAt: now})
g.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "ace", State: delivery.StepPassed})
if step, err := k.Wait(context.Background(), goAsk("mvl0frv18d1980")); err != nil || step.State != delivery.StepPassed {
t.Fatalf("a step that kept moving past its bound: %+v, %v", step, err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1981", Node: "ace", State: delivery.StepRunning, Started: now.Add(-time.Hour), Phase: "integration: tests", PhaseAt: now.Add(-time.Hour)})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1981")); err == nil || !strings.Contains(err.Error(), `stood still in "integration: tests"`) {
t.Fatalf("a step that stood still in a phase: %v", err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1982", Node: "ace", State: delivery.StepRunning, Started: now.Add(-3 * time.Hour), Phase: "qz", PhaseAt: now})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1982")); err == nil || !strings.Contains(err.Error(), "kept moving") {
t.Fatalf("a step that never stops moving: %v", err)
}
// Issue 734: an ask the build seat gave up on (DEAD_LETTERS keeps its copy with the ask's id) ends its wait as failed
// with the words, though a node had taken it; under NotTakenBound one no node takes ends at the bound; a step written
// first is answered as the node wrote it.
dead := &DeadLetters{}
lw := LostAskWaiter{Inner: g, Dead: dead}
k = Asker{Publisher: g, Waiter: lw, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1970", Node: "ace", State: delivery.StepRunning, Started: time.Now()})
given, ok := GivenUpOf(map[string][]string{"Mesh-Dead-Msg-Id": {"mvl0frv18d1970"}, "Mesh-Dead-Deliveries": {"5"},
"Mesh-Dead-Stream": {"SEAT_NODE_BUILD_GO"}, "Mesh-Dead-Gave-Up": {"2026-10-26T12:26:40Z"}})
if !ok {
t.Fatal("a dead letter's headers were not read")
}
go dead.Heard(given)
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1970"))
if err != nil || step.State != delivery.StepFailed || !IsLost(step) ||
!strings.Contains(step.Reason, LostWords+" after 5 deliveries at 2026-10-26T12:26:40Z") {
t.Fatalf("a given-up ask: %+v, %v", step, err)
}
if _, err := k.Passed(step, nil); err == nil || !strings.Contains(err.Error(), "on "+LostNode+" failed: "+LostWords) {
t.Fatalf("a given-up ask's error: %v", err)
}
step, err = k.Wait(NotTakenBound(context.Background(), 20*time.Millisecond), goAsk("mvl0frv18d1971"))
if err != nil || !IsNotTaken(step) || IsLost(step) {
t.Fatalf("an ask no node took: %+v, %v", step, err)
}
g.Write(delivery.Step{ID: "mvl0frv18d1972", Node: "g14", State: delivery.StepPassed})
if step, err = k.Wait(NotTakenBound(context.Background(), time.Minute), goAsk("mvl0frv18d1972")); err != nil || step.Node != "g14" {
t.Fatalf("a step written: %+v, %v", step, err)
}
// Review round 1 of issue 734: a hearing the bus refuses at start is deaf and says so, never holds the start up, and
// tries again on the next ask waited for; heard, it says so and hands the given-up asks on.
fol := &refusingFollower{refuse: 1}
var mu sync.Mutex
var told []Hearing
hearing := NewDeadLetterHearing(context.Background(), fol, []string{delivery.AskGo}, dead, nil,
func(h Hearing) { mu.Lock(); told = append(told, h); mu.Unlock() })
hearing.Wait()
if why := hearing.Deaf(); !strings.Contains(why, "SEAT_NODE_BUILD_GO: permissions violation") {
t.Fatalf("a refused hearing says %q", why)
}
k = Asker{Publisher: g, Waiter: LostAskWaiter{Inner: g, Dead: dead, Hearing: hearing}, Bound: time.Minute}
g.Write(delivery.Step{ID: "mvl0frv18d1973", Node: "g14", State: delivery.StepPassed})
if _, err := k.Wait(context.Background(), goAsk("mvl0frv18d1973")); err != nil {
t.Fatal(err)
}
hearing.Wait()
mu.Lock()
if hearing.Deaf() != "" || len(told) != 2 || told[0].Heard || !told[1].Heard || fol.calls() != 2 {
t.Fatalf("after an ask: deaf %q, told %+v, followed %d times", hearing.Deaf(), told, fol.calls())
}
mu.Unlock()
fol.heard(GivenUp{AskID: "mvl0frv18d1974", Deliveries: 5})
step, err = k.Wait(context.Background(), goAsk("mvl0frv18d1974"))
if err != nil || !IsLost(step) {
t.Fatalf("an ask heard given up after the hearing came back: %+v, %v", step, err)
}
// The lost check of 2026-10-28: a step whose own watch went deaf is read at the next nudge (a step of its build module
// ended) and answered; a nudge of another build module reads nothing; a watch that closes is opened again.
d := &FakeAsks{}
heard := &HeardEnds{Inner: deafWatch{}, Read: d}
k = Asker{Publisher: d, Waiter: heard, Bound: time.Minute}
d.Write(delivery.Step{ID: "mvl0frv18d1980", Node: "shanks", State: delivery.StepPassed})
got := make(chan delivery.Step, 1)
go func() { s, _ := k.Wait(context.Background(), goAsk("mvl0frv18d1980")); got <- s }()
for heard.Waiting() == 0 {
time.Sleep(time.Millisecond)
}
heard.Nudge(NodeBuildModule)
select {
case s := <-got:
t.Fatalf("a nudge of another build module answered %+v", s)
case <-time.After(20 * time.Millisecond):
}
heard.Nudge(GoBuildModule)
if s := <-got; s.State != delivery.StepPassed || s.Node != "shanks" {
t.Fatalf("a step its watch never handed over: %+v", s)
}
closing := &HeardEnds{Inner: &closingWatch{inner: d}, Read: d}
if s, err := (Asker{Publisher: d, Waiter: closing}).Wait(context.Background(), goAsk("mvl0frv18d1980")); err != nil || s.State != delivery.StepPassed {
t.Fatalf("a watch that closed once: %+v, %v", s, err)
}
for _, row := range []struct {
h *Hearing
want string
}{{nil, health.StateUnhealthy}, {&told[0], health.StateUnhealthy}, {&told[1], health.StateHealthy}} {
var raw []byte
if row.h != nil {
raw, _ = json.Marshal(row.h)
}
if a := HearingHealth(raw); a.State != row.want || a.Why == "" {
t.Errorf("health of %+v = %+v, want %s", row.h, a, row.want)
}
}
}
// Review (issue 734 path): an ask taken again by the same node (its build module replaced mid-run) is bounded from the
// new run's start, not the first run's. The steps' own times stand for the clock: the first run started now, the second a
// whole bound ago, so the wait ends at once on the second run's bound where the first run's would wait an hour; no case
// waits on a pause a loaded machine can outlast (the check of 2026-11-02 22:35 tripped one of 60 ms). The context's
// bound is reached only by a wait that kept the first run's start.
func TestARedeliveryToTheSameNodeRestartsTheBound(t *testing.T) {
f := &FakeAsks{}
k := Asker{Publisher: f, Waiter: f, Bound: time.Hour}
const id = "mvl0frv18d1962"
now := time.Now()
f.Write(delivery.Step{ID: id, Node: "ace", State: delivery.StepRunning, Started: now})
f.Write(delivery.Step{ID: id, Node: "ace", State: delivery.StepRunning, Started: now.Add(-time.Hour)})
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if _, err := k.Wait(ctx, goAsk(id)); err == nil || !strings.Contains(err.Error(), "on ace did not end within 1h0m0s of its start") {
t.Fatalf("a redelivered run a bound old: %v, want its own bound to end the wait", err)
}
}
// refusingFollower refuses the first `refuse` follows, as the bus does before the asker's grant is composed.
type refusingFollower struct {
mu sync.Mutex
refuse int
n int
hand func(GivenUp)
}
func (f *refusingFollower) Follow(_ context.Context, _ string, _ time.Time, heard func(GivenUp)) (func(), error) {
f.mu.Lock()
defer f.mu.Unlock()
f.n++
if f.n <= f.refuse {
return nil, errors.New("permissions violation")
}
f.hand = heard
return func() {}, nil
}
func (f *refusingFollower) calls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.n }
func (f *refusingFollower) heard(g GivenUp) { f.mu.Lock(); h := f.hand; f.mu.Unlock(); h(g) }
// deafWatch is a watch of an ask's step that stays open and hears nothing.
type deafWatch struct{}
func (deafWatch) WatchAsk(context.Context, string, string) (<-chan delivery.Step, error) {
return make(chan delivery.Step), nil
}
// closingWatch closes its first watch at once, and hands the later ones to inner.
type closingWatch struct {
inner AskWaiter
closed bool
}
func (c *closingWatch) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if !c.closed {
c.closed = true
ch := make(chan delivery.Step)
close(ch)
return ch, nil
}
return c.inner.WatchAsk(ctx, module, askID)
}
@@ -0,0 +1,88 @@
package buildrun
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ChooseNodes picks up to want build nodes for a language (ADR 0478 clause 3): the nodes whose toolchains key reports the
// version the nodes agree on, the ones with room under their build capacity first (places), then the fewest lanes the
// part has running or chosen there, a tie broken by the node's name. A place is reserved on each node chosen, in the
// same step as the choice, so builds choosing at once do not choose the same full node while another has room; the
// caller takes the place (Places.Acquire) or gives it up (Places.Cancel). A node at its capacity is still chosen when
// the others are too (fewer than want have room): its lane waits for a place. A node qualifies only when qualifies is true
// of it (online and its build module healthy; nil means every node does). It answers the toolchain all the reporting nodes
// agree on, qualifying or not, as the indexer computes it. When nothing is reported, or the nodes report different
// versions (so the indexer agreed none), it fails naming the language and why; when no reporting node qualifies it fails
// naming the nodes. A node whose build module refuses on its toolchain refuses the step, and the build fails naming it.
func ChooseNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, delivery.Toolchain2, error) {
tc, none := delivery.AgreedToolchain(entries)
if none != "" {
return nil, delivery.Toolchain2{}, fmt.Errorf("no %s build node qualifies: %s", language, none)
}
nodes, err := choose(language, entries, places, qualifies, want)
return nodes, tc, err
}
// ChooseAnyNodes is ChooseNodes for artifacts that need no toolchain, such as an archive (ADR 0478 clause 1): the
// build module's nodes agree on nothing, so any node of the build module that qualifies is chosen, by capacity and load only.
func ChooseAnyNodes(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
return choose(language, entries, places, qualifies, want)
}
func choose(language string, entries []delivery.NodeToolchain, places *Places, qualifies func(node string) bool, want int) ([]string, error) {
if places == nil {
places = &Places{}
}
nodes := make([]string, 0, len(entries))
var reporting []string
seen := map[string]bool{}
for _, e := range entries {
if seen[e.Node] {
continue
}
seen[e.Node] = true
reporting = append(reporting, e.Node)
if places.CapacityOf(e.Node) > 0 && (qualifies == nil || qualifies(e.Node)) { // a capacity of 0 keeps the part off a node
nodes = append(nodes, e.Node)
}
}
if len(nodes) == 0 {
sort.Strings(reporting)
return nil, fmt.Errorf("no %s build node qualifies: %s report the toolchain, and none is online with a healthy build module and a build capacity", language, strings.Join(reporting, ", "))
}
return places.choose(nodes, want), nil
}
// DigestsDifferPrefix opens the error of a build whose nodes built an artifact to different digests (ADR 0478 clause 4):
// such a key is not built again as a new revision, since the same source on the same toolchains gave two answers and
// building again settles nothing; it waits for another fingerprint.
const DigestsDifferPrefix = "digests differ:"
// Agree is the one digest every node built an artifact to (ADR 0478 clause 4), or an error naming each node and its
// digest when they differ. One node's digest agrees with itself.
func Agree(artifact string, byNode map[string]string) (string, error) {
nodes := make([]string, 0, len(byNode))
for n := range byNode {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var first string
for i, n := range nodes {
if i == 0 {
first = byNode[n]
continue
}
if byNode[n] != first {
msg := fmt.Sprintf("%s the nodes built %s to different digests:", DigestsDifferPrefix, artifact)
for _, n := range nodes {
msg += fmt.Sprintf(" %s %s;", n, byNode[n])
}
return "", fmt.Errorf("%s", msg[:len(msg)-1])
}
}
return first, nil
}
@@ -0,0 +1,109 @@
package buildrun
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
func hex64(c byte) string { return strings.Repeat(string(c), 64) }
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
func chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName(t *testing.T) { // 1607/AC4, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "novox", "ace", "g14", "shanks")
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 3, "ace": 1, "g14": 1, "shanks": 0}), nil, 2)
if err != nil {
t.Fatal(err)
}
if len(nodes) != 2 || nodes[0] != "shanks" || nodes[1] != "ace" {
t.Errorf("nodes = %v, want [shanks ace]: fewest steps first, a tie by name", nodes)
}
if tc.Version != "go1.27.1" || tc.Recipe != "1" || tc.Language != "go" {
t.Errorf("toolchain = %+v", tc)
}
}
func chooseWithOneNodeAnswersOneNode(t *testing.T) { // 1607/AC4
nodes, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "novox"), nil, nil, 2)
if err != nil || len(nodes) != 1 || nodes[0] != "novox" {
t.Errorf("nodes = %v, err %v, want [novox]", nodes, err)
}
}
func chooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies(t *testing.T) { // 1607/AC4
_, _, err := ChooseNodes("typescript", nil, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "typescript") || !strings.Contains(err.Error(), delivery.ToolchainNoReport) {
t.Errorf("no report: error = %v, want it to name typescript and %q", err, delivery.ToolchainNoReport)
}
entries := append(tcs("go", "go1.27.1", "novox"), tcs("go", "go1.26.0", "ace")...)
_, _, err = ChooseNodes("go", entries, nil, nil, 2)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), "go1.26.0") || !strings.Contains(err.Error(), "go") {
t.Errorf("disagreeing nodes: error = %v, want both versions named", err)
}
}
func agreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer(t *testing.T) { // 1607/AC4, ADR 0478 clause 4
d, err := Agree("tool", map[string]string{"ace": digest, "novox": digest})
if err != nil || d != digest {
t.Errorf("Agree = %q, %v, want %q", d, err, digest)
}
_, err = Agree("tool", map[string]string{"ace": digest, "novox": "sha256:" + hex64('9')})
if err == nil || !strings.Contains(err.Error(), "ace") || !strings.Contains(err.Error(), "novox") || !strings.Contains(err.Error(), digest) || !strings.Contains(err.Error(), hex64('9')) || !strings.Contains(err.Error(), "tool") {
t.Errorf("differing digests: error = %v, want the artifact, both nodes and both digests", err)
}
if d, err := Agree("tool", map[string]string{"ace": digest}); err != nil || d != digest {
t.Errorf("one node: %q, %v", d, err)
}
}
func chooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs(t *testing.T) { // 1607 review 3, ADR 0478 clause 3
entries := tcs("go", "go1.27.1", "ace", "g14", "novox")
online := func(n string) bool { return n != "g14" } // a sleeping laptop
nodes, tc, err := ChooseNodes("go", entries, placesRunning(map[string]int{"novox": 5}), online, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "ace" || nodes[1] != "novox" || tc.Version != "go1.27.1" {
t.Errorf("nodes = %v, toolchain %+v, err %v, want [ace novox] with g14 left out although it runs nothing", nodes, tc, err)
}
_, tc, err = ChooseNodes("go", entries, nil, func(string) bool { return false }, 2)
if err == nil || !strings.Contains(err.Error(), "online") || !strings.Contains(err.Error(), "g14") || !strings.Contains(err.Error(), "go") {
t.Errorf("none online: error = %v, want it to name the nodes that report and that none is online", err)
}
if tc.Version != "go1.27.1" {
t.Errorf("none online: the agreed toolchain %+v is still answered", tc)
}
}
// placesRunning is Places with these steps counted as running, for a test of the choice.
func placesRunning(running map[string]int) *Places {
return &Places{running: running, changed: make(chan struct{})}
}
func choosePrefersNodesBelowTheirCapacity(t *testing.T) { // round 3
entries := tcs("go", "go1.27.1", "ace", "novox", "shanks")
// shanks has capacity 3 with 2 running: room; ace has the default 1 with 1 running: none; novox has none running.
p := placesRunning(map[string]int{"shanks": 2, "ace": 1})
p.Capacity = map[string]int{"shanks": 3}
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil || len(nodes) != 2 || nodes[0] != "novox" || nodes[1] != "shanks" {
t.Errorf("nodes = %v, err %v, want [novox shanks]: the nodes with room, ace at capacity left out", nodes, err)
}
}
// The choice of the nodes a step runs on and the agreement of their answers, one table of cases (the test budget).
func TestChoosingTheNodesOfAStep(t *testing.T) {
t.Run("choose takes the agreed nodes with fewest running steps ties by name", chooseTakesTheAgreedNodesWithFewestRunningStepsTiesByName)
t.Run("choose with one node answers one node", chooseWithOneNodeAnswersOneNode)
t.Run("choose fails naming the language and the versions when no node qualifies", chooseFailsNamingTheLanguageAndTheVersionsWhenNoNodeQualifies)
t.Run("agree needs equal digests and names both nodes when they differ", agreeNeedsEqualDigestsAndNamesBothNodesWhenTheyDiffer)
t.Run("choose skips nodes that are not online and fails naming them when none is", chooseSkipsNodesThatAreNotOnlineAndFailsNamingThemWhenNoneIs)
t.Run("choose prefers nodes below their capacity", choosePrefersNodesBelowTheirCapacity)
}
@@ -0,0 +1,5 @@
// Package buildrun holds the parts every module that runs steps on the build modules shares: the caller, the steps
// watch and the runner of a long step (ADR 0469), the sets of presence, toolchains and healthy build modules, the
// places a node offers by capacity, and the choice of build nodes (ADR 0478). The builder and the checker of the
// delivery use them; they moved here from the builder's own package so neither keeps a copy (issue 1671, ADR 0484).
package buildrun
@@ -0,0 +1,22 @@
module git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929
github.com/nats-io/nats.go v1.54.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nats-server/v2 v2.11.17 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
@@ -0,0 +1,27 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929 h1:RtAFbUJRP1qO9AIMM2ue+cXNzqj7WZ7IOlSqNvttwuU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,56 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// HaveSet holds what each node's node-engine says of each module on it (node-engine_have, key `<node>.<module>`): the
// builder reads it to know whether a node's build module is running and healthy, since a build module whose installed
// toolchain is not the one its recipe builds with is unhealthy (mesh-build-go build_go_health) and its steps would
// refuse (ADR 0478 clause 3). A module the node-engine has written no key of is not healthy. Safe for use from the watch
// and from a build at once.
type HaveSet struct {
mu sync.Mutex
healthy map[string]bool // by `<node>.<module>`
}
// Put takes a change of the have bucket in and answers whether a module's healthy state changed. A node's own key
// (`<node>.node`) and a value that is not a have are left out.
func (s *HaveSet) Put(key string, value []byte, deleted bool) bool {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode {
return false
}
ok := false
if !deleted {
h, err := delivery.DecodeHave(value)
if err != nil {
return false
}
ok = h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK
}
k := node + "." + module
s.mu.Lock()
defer s.mu.Unlock()
if s.healthy[k] == ok {
return false
}
if s.healthy == nil {
s.healthy = map[string]bool{}
}
if ok {
s.healthy[k] = true
} else {
delete(s.healthy, k)
}
return true
}
// Healthy is whether the module on the node was last seen running and healthy.
func (s *HaveSet) Healthy(node, module string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.healthy[node+"."+module]
}
@@ -0,0 +1,236 @@
package buildrun
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A step's end is never only its own watch's to hear** (the lost check of 2026-10-28: mesh-catalog #553's verdict
// stayed running after its step mvsgthyjed5786 had passed on shanks at 19:09:31Z; the watch of that one key, opened at
// 18:46:05Z, heard neither the step's start nor its end and never closed, while a watch opened at 18:58:48Z heard its own
// step end). HeardEnds stands between the asker (Asker) and the watch of each ask's step key: when it is
// nudged, every ask it waits on reads its step keys now (a read, not the watch) and hands what it reads to its wait, so a
// step that ended is judged even when its own watch went deaf. A watch that closes under a wait is opened again, at most
// reopenTries times. It is nudged on an event (Nudge): any step of the same build module ending (one watch of each steps
// state, HearEnds), a reconnect to the bus, and whatever else the asker hears (the checker: checker_ask_again).
// It holds no timer. Every asker of the build seats waits through it (the checker and the builder, HearEnds).
// StepReader reads, now, every step a build module's steps state holds for an ask: its keys `*.<ask id>`.
type StepReader interface {
AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error)
}
// reopenTries is how often a wait opens its ask's watch again after it closed, before the wait ends failed.
const reopenTries = 3
// HeardEnds is an AskWaiter over Inner that also reads an ask's steps when nudged. Safe for concurrent use.
type HeardEnds struct {
Inner AskWaiter
Read StepReader
Logf func(format string, args ...any)
mu sync.Mutex
waiting map[string]*waitedAsk // by ask id
}
type waitedAsk struct {
module string
nudge chan struct{}
}
func (h *HeardEnds) logf(format string, args ...any) {
if h.Logf != nil {
h.Logf(format, args...)
}
}
// WatchAsk is AskWaiter's: what Inner's watch hands, and, at each nudge, what the ask's keys hold now.
func (h *HeardEnds) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
in, err := h.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
w := &waitedAsk{module: module, nudge: make(chan struct{}, 1)}
h.mu.Lock()
if h.waiting == nil {
h.waiting = map[string]*waitedAsk{}
}
h.waiting[askID] = w
h.mu.Unlock()
out := make(chan delivery.Step)
go func() {
defer close(out)
defer func() {
h.mu.Lock()
if h.waiting[askID] == w {
delete(h.waiting, askID)
}
h.mu.Unlock()
}()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
reopened := 0
for {
select {
case <-ctx.Done():
return
case s, ok := <-in:
if !ok {
if ctx.Err() != nil || reopened >= reopenTries {
h.logf("the watch of the ask %s's step on %s closed %d times; its wait ends", askID, module, reopened+1)
return
}
reopened++
if in, err = h.Inner.WatchAsk(ctx, module, askID); err != nil {
h.logf("the watch of the ask %s's step on %s closed and cannot be opened again, so its wait ends: %v", askID, module, err)
return
}
h.logf("the watch of the ask %s's step on %s closed; opened again (%d of %d)", askID, module, reopened, reopenTries)
continue
}
if !send(s) {
return
}
case <-w.nudge:
if h.Read == nil {
continue
}
steps, err := h.Read.AskSteps(ctx, module, askID)
if err != nil {
h.logf("the steps of the ask %s on %s cannot be read: %v", askID, module, err)
continue
}
for _, s := range steps {
if s.State != delivery.StepRunning {
h.logf("the ask %s's step on %s ended %s, read at a nudge: its watch had not handed it over", askID, s.Node, s.State)
}
if !send(s) {
return
}
}
}
}
}()
return out, nil
}
// Nudge has every ask waited on of module read its steps now; module "" nudges every ask.
func (h *HeardEnds) Nudge(module string) {
h.mu.Lock()
defer h.mu.Unlock()
for _, w := range h.waiting {
if module == "" || w.module == module {
select {
case w.nudge <- struct{}{}:
default: // a nudge is pending already
}
}
}
}
// Waiting is how many asks are waited on now.
func (h *HeardEnds) Waiting() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.waiting)
}
// AskSteps is StepReader's: the keys `*.<ask id>` of the build module's steps state, read now.
func (s JetStreamAsks) AskSteps(ctx context.Context, module, askID string) ([]delivery.Step, error) {
pattern, err := delivery.AskStepWatch(askID)
if err != nil {
return nil, err
}
kv, err := s.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return nil, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
lister, err := kv.ListKeysFiltered(ctx, pattern)
if err != nil {
return nil, err
}
var out []delivery.Step
for key := range lister.Keys() {
e, err := kv.Get(ctx, key)
if err != nil {
continue // deleted between the list and the read
}
if step, err := delivery.DecodeStep(e.Value()); err == nil {
out = append(out, step)
}
}
return out, nil
}
// HearEnds is the HeardEnds every asker of the build seats waits through: over inner (the asks' own watches, or a
// LostAskWaiter's inner), reading with asks, and nudged at each step end of each of modules (one watch of each steps state,
// updates only; one that closes is opened again, and said and left after reopenTries closes in a row that heard nothing). The asker nudges it at a reconnect too (Nudge("")).
func HearEnds(ctx context.Context, asks JetStreamAsks, inner AskWaiter, modules []string, logf func(string, ...any)) *HeardEnds {
if logf == nil {
logf = func(string, ...any) {}
}
h := &HeardEnds{Inner: inner, Read: asks, Logf: logf}
for _, module := range modules {
kv, err := asks.JS.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
logf("the steps of %s cannot be read (%v): an ask whose own watch goes deaf is read only at a reconnect", module, err)
continue
}
go h.follow(ctx, kv, module)
}
return h
}
func (h *HeardEnds) follow(ctx context.Context, kv jetstream.KeyValue, module string) {
silent := 0 // closes in a row having heard nothing
for ctx.Err() == nil {
w, err := kv.WatchAll(ctx, jetstream.UpdatesOnly())
if err != nil {
h.logf("the steps of %s cannot be watched, so an ask whose own watch goes deaf is read only at a reconnect: %v", module, err)
return
}
heard := false
for e := range w.Updates() {
if e == nil || e.Operation() != jetstream.KeyValuePut {
continue
}
heard = true
if step, err := delivery.DecodeStep(e.Value()); err == nil && step.State != delivery.StepRunning {
h.Nudge(module)
}
}
_ = w.Stop()
if ctx.Err() != nil {
return
}
h.Nudge(module) // a step may have ended while the watch was closing
if heard {
silent = 0
} else if silent++; silent >= reopenTries {
h.logf("the watch of the steps of %s closed %d times in a row having heard nothing, so it is left: an ask whose own watch goes deaf is read only at a reconnect", module, silent)
return
}
h.logf("the watch of the steps of %s closed; opened again", module)
}
}
// AskSteps is StepReader's: the last step written for the ask, if any.
func (f *FakeAsks) AskSteps(_ context.Context, _ string, askID string) ([]delivery.Step, error) {
f.mu.Lock()
defer f.mu.Unlock()
if s := f.steps[askID]; len(s) > 0 {
return []delivery.Step{s[len(s)-1]}, nil
}
return nil, nil
}
@@ -0,0 +1,503 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
)
// **An ask the build seat gave up on ends its step** (novox/hq issue 734). A build seat's worker offers an ask
// delivery.AskMaxDeliver times; one handed out that often without an answer (a bus reload that drops every hand-out, a node
// that dies taking it each time) is given up by the server, and the controller keeps a copy of it in DEAD_LETTERS under
// mesh.events.dead.<seat stream>.<worker>, with the ask's message id (its ask id), how often it was handed out and when it
// was given up. No build node writes a step for it, so an asker that waits for the step waits for ever: a try stays
// running and holds a try place, a check's verdict stays running and its pull request waits.
//
// An asker of the build seats (the checker, the builder) hears those copies (DeadLetterHearing: an ordered consumer on
// DEAD_LETTERS filtered to the build seats' streams, granted to a user of a seat pulled by kind, buscompose
// DeadLetterReads) into DeadLetters, and LostAskWaiter ends the waiting step at once as failed on the node LostNode, its
// reason saying LostWords; the asker decides what follows (the checker ends a try as error and asks a check again, the
// builder asks its part of a build again).
//
// Belt and braces, where the asker says so in the context (NotTakenBound; the checker's tries): an ask no build node took
// within a bound ends the same way, saying NotTakenWords. An ask waits in the stream as long as the backlog is otherwise
// (ADR 0525 §9): a backlog is no fault.
// LostNode is the node a step ended for a lost ask names: no build node wrote it.
const LostNode = "bus"
// The words of a step ended for its ask.
const (
LostWords = "its ask was given up by the build seat"
NotTakenWords = "no build node took its ask"
)
// The headers the controller keeps a given-up message's copy with in DEAD_LETTERS (mesh-controller internal/link
// deadletters.go): the original message id (an ask's is its ask id), how often it was handed out, when it was given up.
const (
DeadHeaderMsgID = "Mesh-Dead-Msg-Id"
DeadHeaderDeliveries = "Mesh-Dead-Deliveries"
DeadHeaderGaveUp = "Mesh-Dead-Gave-Up"
DeadHeaderStream = "Mesh-Dead-Stream"
)
// DeadLettersStream is the stream the controller keeps given-up messages in, and DeadLetterFilter the subjects of one seat
// stream's copies there (buscompose DeadLetterSubject).
const DeadLettersStream = "DEAD_LETTERS"
// DeadLetterFilter is the subjects DEAD_LETTERS keeps a seat stream's given-up messages under.
func DeadLetterFilter(stream string) string { return "mesh.events.dead." + stream + ".>" }
// SeatStream is the stream of a build seat's language (buscompose seatStreamName of node-build-<language>).
func SeatStream(language string) string { return "SEAT_NODE_BUILD_" + strings.ToUpper(language) }
// GivenUp is one ask a build seat gave up on, as DEAD_LETTERS keeps it.
type GivenUp struct {
AskID string
Stream string
Deliveries int
At time.Time
}
// GivenUpOf reads a DEAD_LETTERS copy's headers; false when it names no message id.
func GivenUpOf(header map[string][]string) (GivenUp, bool) {
get := func(k string) string {
if v := header[k]; len(v) > 0 {
return strings.TrimSpace(v[0])
}
return ""
}
g := GivenUp{AskID: get(DeadHeaderMsgID), Stream: get(DeadHeaderStream)}
if g.AskID == "" {
return GivenUp{}, false
}
g.Deliveries, _ = strconv.Atoi(get(DeadHeaderDeliveries))
g.At, _ = time.Parse(time.RFC3339Nano, get(DeadHeaderGaveUp))
return g, true
}
// Words are what a step ended for the given-up ask says.
func (g GivenUp) Words() string {
at := "at an unknown time"
if !g.At.IsZero() {
at = "at " + g.At.UTC().Format(time.RFC3339)
}
return fmt.Sprintf("%s after %d deliveries %s (kept in %s); ask again", LostWords, g.Deliveries, at, DeadLettersStream)
}
// keptGivenUp is how many given-up asks DeadLetters remembers for a waiter that starts after the copy was heard.
const keptGivenUp = 1024
// DeadLetters holds the given-up asks heard, and hands each to the waiter of its ask. Safe for concurrent use.
type DeadLetters struct {
mu sync.Mutex
heard map[string]GivenUp
order []string
watchers map[string][]chan GivenUp
}
// Heard takes one given-up ask in.
func (d *DeadLetters) Heard(g GivenUp) {
d.mu.Lock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if _, ok := d.heard[g.AskID]; !ok {
d.order = append(d.order, g.AskID)
if len(d.order) > keptGivenUp {
delete(d.heard, d.order[0])
d.order = d.order[1:]
}
}
d.heard[g.AskID] = g
ws := d.watchers[g.AskID]
delete(d.watchers, g.AskID)
d.mu.Unlock()
for _, w := range ws {
w <- g // buffered, one each
}
}
// watch answers a channel that gets the ask's given-up copy once (at once when it was heard already), and its cancel.
func (d *DeadLetters) watch(askID string) (<-chan GivenUp, func()) {
ch := make(chan GivenUp, 1)
d.mu.Lock()
defer d.mu.Unlock()
if d.heard == nil {
d.heard, d.watchers = map[string]GivenUp{}, map[string][]chan GivenUp{}
}
if g, ok := d.heard[askID]; ok {
ch <- g
return ch, func() {}
}
d.watchers[askID] = append(d.watchers[askID], ch)
return ch, func() {
d.mu.Lock()
defer d.mu.Unlock()
ws := d.watchers[askID]
for i, w := range ws {
if w == ch {
d.watchers[askID] = append(ws[:i], ws[i+1:]...)
break
}
}
if len(d.watchers[askID]) == 0 {
delete(d.watchers, askID)
}
}
}
type notTakenKey struct{}
// NotTakenBound says, in ctx, that an ask no build node took within bound ends its step (a try's, never a check's).
func NotTakenBound(ctx context.Context, bound time.Duration) context.Context {
return context.WithValue(ctx, notTakenKey{}, bound)
}
func notTakenBoundOf(ctx context.Context) time.Duration {
d, _ := ctx.Value(notTakenKey{}).(time.Duration)
return d
}
// LostAskWaiter is an AskWaiter that ends an ask's step when the build seat gave the ask up (DeadLetters), or, under
// NotTakenBound, when no build node took it within the bound: it hands the asker a failed step whose reason says why, on
// the node "bus". Every step the build module writes is handed on as it is.
type LostAskWaiter struct {
Inner AskWaiter
Dead *DeadLetters
// Hearing, when set, is asked to hear DEAD_LETTERS again each time an ask is waited for while it cannot: an ask put is
// the event that makes hearing matter.
Hearing interface {
Deaf() string
Again()
}
Now func() time.Time
}
func (w LostAskWaiter) now() time.Time {
if w.Now != nil {
return w.Now()
}
return time.Now()
}
// WatchAsk is AskWaiter's.
func (w LostAskWaiter) WatchAsk(ctx context.Context, module, askID string) (<-chan delivery.Step, error) {
if w.Hearing != nil && w.Hearing.Deaf() != "" {
w.Hearing.Again()
}
inner, err := w.Inner.WatchAsk(ctx, module, askID)
if err != nil {
return nil, err
}
var dead <-chan GivenUp
cancel := func() {}
if w.Dead != nil {
dead, cancel = w.Dead.watch(askID)
}
// The not-taken bound is a context's (ADR 0437 §8: no timer), ended when a node takes the ask.
var notTaken <-chan struct{}
bound := notTakenBoundOf(ctx)
stopBound := func() {}
if bound > 0 {
var nt context.Context
nt, stopBound = context.WithTimeout(context.Background(), bound)
notTaken = nt.Done()
}
out := make(chan delivery.Step)
go func() {
defer close(out)
defer cancel()
defer func() { stopBound() }()
send := func(s delivery.Step) bool {
select {
case out <- s:
return true
case <-ctx.Done():
return false
}
}
ended := func(reason string) delivery.Step {
at := w.now().UTC()
return delivery.Step{ID: askID, Node: LostNode, State: delivery.StepFailed, Reason: reason, Ended: at}
}
for {
select {
case <-ctx.Done():
return
case s, ok := <-inner:
if !ok {
return
}
if notTaken != nil {
stopBound() // a node took it: its step is bounded by the asker from its start
notTaken = nil
}
if !send(s) {
return
}
case g := <-dead:
dead = nil
send(ended(g.Words()))
return
case <-notTaken:
send(ended(fmt.Sprintf("%s within %s of its put; it was lost or waits behind a backlog; ask again", NotTakenWords, bound)))
return
}
}
}()
return out, nil
}
// IsLost says whether a step was ended by LostAskWaiter because the build seat gave its ask up (LostWords).
func IsLost(s delivery.Step) bool { return s.Node == LostNode && strings.Contains(s.Reason, LostWords) }
// IsNotTaken says whether a step was ended by LostAskWaiter because no build node took its ask within the bound.
func IsNotTaken(s delivery.Step) bool {
return s.Node == LostNode && strings.Contains(s.Reason, NotTakenWords)
}
// **The asker hears DEAD_LETTERS, tries again when it cannot, and says whether it hears** (issue 734). The read is
// granted with the asker's seat traffic (buscompose DeadLetterReads); an asker started before its grant was composed is
// refused, and an asker that only tried once would stay deaf until it starts again, silently. DeadLetterHearing tries in
// the background, never holding the start up, and again on an event (Again): the asker's reconnect to the bus, and every
// ask waited for while it is deaf (LostAskWaiter.Hearing); no timer. Each change of whether it hears is said (Said), so
// the module's health tool says it: a deaf asker is not healthy.
// Hearing is what a DeadLetterHearing says: whether it hears every build seat stream it follows, and why not.
type Hearing struct {
Heard bool `json:"heard"`
Streams []string `json:"streams"`
Why string `json:"why,omitempty"`
At time.Time `json:"at"`
}
// HearingKey is the key a module keeps its Hearing under, in its own state (the checker's and the builder's hearing).
const HearingKey = "dead-letters"
// DeadLetterFollower starts following one seat stream's copies in DEAD_LETTERS from since, handing each given-up ask to
// heard; it answers the stop, or why it cannot.
type DeadLetterFollower interface {
Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error)
}
// JetStreamDeadLetters is the bus side of a DeadLetterFollower: an ordered consumer on DEAD_LETTERS filtered to the
// stream's copies, which resumes where it was after a reconnect and, reset before it heard any, starts at since again.
type JetStreamDeadLetters struct{ JS jetstream.JetStream }
// Follow is DeadLetterFollower's.
func (j JetStreamDeadLetters) Follow(ctx context.Context, stream string, since time.Time, heard func(GivenUp)) (func(), error) {
cons, err := j.JS.OrderedConsumer(ctx, DeadLettersStream, jetstream.OrderedConsumerConfig{
FilterSubjects: []string{DeadLetterFilter(stream)}, DeliverPolicy: jetstream.DeliverByStartTimePolicy, OptStartTime: &since})
if err != nil {
return nil, err
}
cc, err := cons.Consume(func(m jetstream.Msg) {
if g, ok := GivenUpOf(m.Headers()); ok {
heard(g)
}
})
if err != nil {
return nil, err
}
return cc.Stop, nil
}
// DeadLetterHearing follows the build seats' given-up asks into Dead. Make it with NewDeadLetterHearing.
type DeadLetterHearing struct {
follower DeadLetterFollower
streams []string
dead *DeadLetters
logf func(string, ...any)
said func(Hearing)
ctx context.Context
since time.Time
mu sync.Mutex
stops map[string]func()
why map[string]string
trying bool
again bool
told *Hearing
wg sync.WaitGroup
}
// NewDeadLetterHearing starts hearing the given-up asks of each language's build seat stream, in the background. said
// is told each change of whether it hears (nil tells nothing); ctx ends it.
func NewDeadLetterHearing(ctx context.Context, follower DeadLetterFollower, languages []string, dead *DeadLetters,
logf func(string, ...any), said func(Hearing)) *DeadLetterHearing {
h := &DeadLetterHearing{follower: follower, dead: dead, logf: logf, said: said, ctx: ctx, since: time.Now(),
stops: map[string]func(){}, why: map[string]string{}}
for _, l := range languages {
s := SeatStream(l)
h.streams = append(h.streams, s)
h.why[s] = "not tried yet"
}
if h.logf == nil {
h.logf = func(string, ...any) {}
}
h.Again()
return h
}
// Deaf says why a stream is not heard, or "" when every one is.
func (h *DeadLetterHearing) Deaf() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.deafLocked()
}
func (h *DeadLetterHearing) deafLocked() string {
var out []string
for _, s := range h.streams {
if why, deaf := h.why[s]; deaf {
out = append(out, s+": "+why)
}
}
if len(out) == 0 {
return ""
}
return DeadLettersStream + " is not heard, so an ask a build seat gives up on is waited for until its bound: " +
strings.Join(out, "; ")
}
// Again tries, in the background, every stream not heard yet; a call while it tries makes it try once more after.
func (h *DeadLetterHearing) Again() {
h.mu.Lock()
if h.trying {
h.again = true
h.mu.Unlock()
return
}
h.trying = true
h.mu.Unlock()
h.wg.Add(1)
go func() {
defer h.wg.Done()
for {
h.try()
h.mu.Lock()
if !h.again || h.ctx.Err() != nil {
h.trying = false
h.mu.Unlock()
return
}
h.again = false
h.mu.Unlock()
}
}()
}
// Wait returns when no try runs.
func (h *DeadLetterHearing) Wait() { h.wg.Wait() }
// Stop ends every stream's following.
func (h *DeadLetterHearing) Stop() {
h.mu.Lock()
defer h.mu.Unlock()
for s, stop := range h.stops {
stop()
delete(h.stops, s)
}
}
func (h *DeadLetterHearing) try() {
for _, s := range h.streams {
h.mu.Lock()
_, deaf := h.why[s]
h.mu.Unlock()
if !deaf || h.ctx.Err() != nil {
continue
}
stream := s
stop, err := h.follower.Follow(h.ctx, stream, h.since, func(g GivenUp) {
h.logf("the ask %s on %s was given up after %d deliveries at %s: the wait for its step ends", g.AskID, g.Stream,
g.Deliveries, g.At.Format(time.RFC3339))
h.dead.Heard(g)
})
h.mu.Lock()
if err != nil {
h.why[stream] = err.Error()
h.logf("the asks %s gives up on are not heard (%s of %s); tried again at the next reconnect or ask: %v",
stream, DeadLetterFilter(stream), DeadLettersStream, err)
} else {
delete(h.why, stream)
h.stops[stream] = stop
h.logf("the asks %s gives up on are heard from %s", stream, DeadLettersStream)
}
h.mu.Unlock()
}
h.tell()
}
// tell says the hearing when it changed.
func (h *DeadLetterHearing) tell() {
h.mu.Lock()
why := h.deafLocked()
now := Hearing{Heard: why == "", Streams: append([]string(nil), h.streams...), Why: why, At: time.Now().UTC()}
same := h.told != nil && h.told.Heard == now.Heard && h.told.Why == now.Why
if !same {
h.told = &now
}
h.mu.Unlock()
if !same && h.said != nil {
h.said(now)
}
}
// HearingState is the name of the state an asker keeps its Hearing in (`<module>_hearing`, key HearingKey), which its
// health tool reads.
const HearingState = "hearing"
// HearingKeeper is where a Hearing is kept: a jetstream.KeyValue, or a process.AwaitedBucket for a state that may not be
// on the bus at the part's start (issue 1790), which keeps the latest until it appears and answers process.ErrNotYet.
type HearingKeeper interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
}
// SayHearing is a DeadLetterHearing's said that keeps each Hearing in kv under HearingKey; nil kv keeps nothing and
// says so once. A write kv keeps for later (process.ErrNotYet) is not an error: the awaited bucket said it waits.
func SayHearing(ctx context.Context, kv HearingKeeper, logf func(string, ...any)) func(Hearing) {
if kv == nil {
if logf != nil {
logf("whether %s is heard is not kept: the state %s is not there, so the health tool cannot say it", DeadLettersStream, HearingState)
}
return nil
}
return func(h Hearing) {
data, err := json.Marshal(h)
if err == nil {
_, err = kv.Put(ctx, HearingKey, data)
}
if err != nil && !errors.Is(err, process.ErrNotYet) && logf != nil {
logf("whether %s is heard could not be kept: %v", DeadLettersStream, err)
}
}
}
// HearingHealth is the health answer of an asker's kept Hearing (raw, empty when none is kept): healthy while it hears
// DEAD_LETTERS, unhealthy saying why while it does not or has not said.
func HearingHealth(raw []byte) health.Answer {
if len(raw) == 0 {
return health.Unhealthy("it has not said whether it hears " + DeadLettersStream + " (issue 734): it is starting, or its state " + HearingState + " is not there")
}
var h Hearing
if err := json.Unmarshal(raw, &h); err != nil {
return health.Unhealthy("what it keeps of hearing " + DeadLettersStream + " is not a hearing: " + err.Error())
}
if !h.Heard {
return health.Unhealthy(h.Why + " (since " + h.At.Format(time.RFC3339) + "; tried again at its next reconnect or ask)")
}
return health.Healthy("it hears " + DeadLettersStream + " for " + strings.Join(h.Streams, ", "))
}
@@ -0,0 +1,7 @@
package buildrun
// The build modules the mesh runs, by the language each builds (ADR 0469).
const (
GoBuildModule = "mesh-build-go"
NodeBuildModule = "mesh-build-node"
)
@@ -0,0 +1,145 @@
package buildrun
import (
"context"
"fmt"
"sort"
"sync"
)
// DefaultCapacity is the build capacity of a node that has no setting: one build step at a time.
const DefaultCapacity = 1
// Places counts the build steps a part runs on each node and holds a lane back while its node is at its
// capacity (the operator's decision of 2026-10-21; ADR 0478 clause 3). A lane is the fetch and the builds of one build
// module on one node, run one step at a time, so a lane holds one place while it runs. A node is chosen for a lane by
// reserving a place on it at the moment of the choice (ChooseNodes), so that builds choosing at once see each other's
// choices and none waits on a full node while another has room. A lane waits for its place by watching the ends of the
// other lanes, not by a timer: ending a lane wakes the waiting ones.
type Places struct {
// Capacity is the number of steps at once by node; a node not in it has DefaultCapacity.
Capacity map[string]int
mu sync.Mutex
running map[string]int // lanes holding a place
reserved map[string]int // lanes chosen for the node that do not hold a place yet
changed chan struct{} // closed, and replaced, each time a lane ends
}
// CapacityOf is how many steps the part runs on the node at once: DefaultCapacity for a node the setting does not
// name, and 0 for one it names so, where the part never builds.
func (p *Places) CapacityOf(node string) int {
if n, ok := p.Capacity[node]; ok && n >= 0 {
return n
}
return DefaultCapacity
}
// Running is the steps in flight now, by node.
func (p *Places) Running() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.running))
for n, c := range p.running {
if c > 0 {
out[n] = c
}
}
return out
}
// Reserved is the lanes chosen for a node that hold no place yet, by node.
func (p *Places) Reserved() map[string]int {
p.mu.Lock()
defer p.mu.Unlock()
out := make(map[string]int, len(p.reserved))
for n, c := range p.reserved {
if c > 0 {
out[n] = c
}
}
return out
}
func (p *Places) init() {
if p.running == nil {
p.running = map[string]int{}
}
if p.reserved == nil {
p.reserved = map[string]int{}
}
if p.changed == nil {
p.changed = make(chan struct{})
}
}
// Acquire takes a place on the node, waiting while the node is at its capacity, and answers the function that gives it
// back. A place reserved for the node by a choice is the one taken. It returns an error when ctx ends first, and then
// gives up the reservation too.
func (p *Places) Acquire(ctx context.Context, node string) (release func(), err error) {
for {
p.mu.Lock()
p.init()
if p.running[node] < p.CapacityOf(node) {
p.running[node]++
if p.reserved[node] > 0 {
p.reserved[node]--
}
p.mu.Unlock()
var once sync.Once
return func() {
once.Do(func() {
p.mu.Lock()
p.running[node]--
close(p.changed)
p.changed = make(chan struct{})
p.mu.Unlock()
})
}, nil
}
wait := p.changed
p.mu.Unlock()
select {
case <-wait:
case <-ctx.Done():
p.Cancel(node)
return nil, fmt.Errorf("waiting for a place on %s: %w", node, ctx.Err())
}
}
}
// Cancel gives up a place reserved for the node that will not be taken.
func (p *Places) Cancel(node string) {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
if p.reserved[node] > 0 {
p.reserved[node]--
}
}
// choose orders the nodes (those with room first, counting the lanes running and the lanes reserved, then the fewest of
// them, then by name), keeps the first want and reserves a place on each, all at once.
func (p *Places) choose(nodes []string, want int) []string {
p.mu.Lock()
defer p.mu.Unlock()
p.init()
load := func(n string) int { return p.running[n] + p.reserved[n] }
sort.SliceStable(nodes, func(i, j int) bool {
ri, rj := load(nodes[i]) < p.CapacityOf(nodes[i]), load(nodes[j]) < p.CapacityOf(nodes[j])
if ri != rj {
return ri
}
if load(nodes[i]) != load(nodes[j]) {
return load(nodes[i]) < load(nodes[j])
}
return nodes[i] < nodes[j]
})
if len(nodes) > want {
nodes = nodes[:want]
}
for _, n := range nodes {
p.reserved[n]++
}
return nodes
}
@@ -0,0 +1,104 @@
package buildrun
import (
"context"
"strings"
"testing"
"time"
)
func acquireAsync(p *Places, node string) chan func() {
got := make(chan func(), 1)
go func() {
release, err := p.Acquire(context.Background(), node)
if err != nil {
close(got)
return
}
got <- release
}()
return got
}
func TestThreeStepsOnANodeOfCapacityTwoRunTwoAtOnceAndTheThirdStartsWhenOneEnds(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"ace": 2}}
a, b := acquireAsync(p, "ace"), acquireAsync(p, "ace")
ra, rb := <-a, <-b
third := acquireAsync(p, "ace")
select {
case <-third:
t.Fatal("a third step started on a node of capacity 2 with two running")
case <-time.After(10 * time.Millisecond):
}
if p.Running()["ace"] != 2 {
t.Errorf("running = %v, want 2 on ace", p.Running())
}
ra()
select {
case r := <-third:
r()
case <-time.After(2 * time.Second):
t.Fatal("the third step did not start when one ended")
}
rb()
}
func TestANodeWithNoSettingRunsOneStepAtATime(t *testing.T) { // round 3
p := &Places{Capacity: map[string]int{"shanks": 3}}
first := <-acquireAsync(p, "g14")
second := acquireAsync(p, "g14")
select {
case <-second:
t.Fatal("two steps on a node without a setting")
case <-time.After(10 * time.Millisecond):
}
first()
r := <-second
r()
if p.CapacityOf("g14") != 1 || p.CapacityOf("shanks") != 3 {
t.Errorf("capacities = %d, %d", p.CapacityOf("g14"), p.CapacityOf("shanks"))
}
}
func TestAWaitingStepStopsWhenItsContextEnds(t *testing.T) { // round 3
p := &Places{}
r := <-acquireAsync(p, "ace")
defer r()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { _, err := p.Acquire(ctx, "ace"); done <- err }()
cancel()
select {
case err := <-done:
if err == nil {
t.Error("Acquire returned no error for a cancelled wait")
}
case <-time.After(2 * time.Second):
t.Fatal("a cancelled wait did not return")
}
}
// The operator's decision of 2026-10-21 13:40: the builder builds on shanks and ace only; a capacity of 0 keeps it off a node.
func TestACapacityOfZeroKeepsThePartOffTheNodeAndANodeNotNamedKeepsOne(t *testing.T) { // issue 1661
p := &Places{Capacity: map[string]int{"shanks": 3, "ace": 2, "g14": 0, "novox": 0}}
if p.CapacityOf("g14") != 0 || p.CapacityOf("novox") != 0 || p.CapacityOf("ace") != 2 || p.CapacityOf("other") != DefaultCapacity {
t.Errorf("capacities = %d %d %d %d, want 0 0 2 and the default for a node not named", p.CapacityOf("g14"), p.CapacityOf("novox"), p.CapacityOf("ace"), p.CapacityOf("other"))
}
entries := tcs("go", "go1.27.1", "ace", "g14", "novox", "shanks", "other")
for i := 0; i < 6; i++ {
nodes, _, err := ChooseNodes("go", entries, p, nil, 2)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if n == "g14" || n == "novox" {
t.Fatalf("choice %d: %v includes a node of capacity 0", i, nodes)
}
}
}
// Only nodes of capacity 0 report: nothing qualifies, and the build says why.
_, _, err := ChooseNodes("go", tcs("go", "go1.27.1", "g14", "novox"), p, nil, 2)
if err == nil || !strings.Contains(err.Error(), "capacity") {
t.Errorf("only capacity-0 nodes: error = %v, want it to name the capacity", err)
}
}
@@ -0,0 +1,53 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// PresenceSet holds which nodes are online, as the watchman writes it (mesh-delivery-watchman_presence, key `<node>`, the
// node-engine's connection): a node asleep or off the bus is offline and its build module cannot answer. A node the
// watchman has no key of is not online. Safe for use from the watch and from a build at once.
type PresenceSet struct {
mu sync.Mutex
online map[string]bool
}
// Put takes a change of the presence bucket in and answers whether a node's online state changed. A part's key and a
// value that is not a presence are left out.
func (s *PresenceSet) Put(key string, value []byte, deleted bool) bool {
node, part, _, err := delivery.ParsePresenceKey(key)
if err != nil || part != "" {
return false
}
on := false
if !deleted {
p, err := delivery.DecodePresence(value)
if err != nil {
return false
}
on = p.Online
}
s.mu.Lock()
defer s.mu.Unlock()
if s.online[node] == on {
return false
}
if s.online == nil {
s.online = map[string]bool{}
}
if on {
s.online[node] = true
} else {
delete(s.online, node)
}
return true
}
// Online is whether the watchman last saw the node online.
func (s *PresenceSet) Online(node string) bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.online[node]
}
@@ -0,0 +1,43 @@
package buildrun
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
func TestPresenceSetKnowsOnlyNodesTheWatchmanSawOnline(t *testing.T) { // 1607 review 3
var s PresenceSet
if s.Online("ace") {
t.Error("a node the watchman never wrote is online")
}
if !s.Put("ace", presenceValue(t, true), false) || !s.Online("ace") {
t.Error("ace online was not taken in")
}
if s.Put("ace", presenceValue(t, true), false) {
t.Error("the same presence again counted as a change")
}
if !s.Put("ace", presenceValue(t, false), false) || s.Online("ace") {
t.Error("ace going offline was not taken in")
}
s.Put("g14", presenceValue(t, true), false)
if !s.Put("g14", nil, true) || s.Online("g14") {
t.Error("a deleted key is still online")
}
// A part's instance (part.<part>.<instance>) is not a node.
if s.Put("part.mesh-delivery-builder.novox", presenceValue(t, true), false) || s.Online("part") {
t.Error("a part's key was taken for a node")
}
if s.Put("novox", []byte("junk"), false) {
t.Error("junk counted")
}
}
@@ -0,0 +1,225 @@
package buildrun
import (
"context"
"errors"
"fmt"
"maps"
"slices"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module leaves a node: no part writes another module's state, and the module is gone. A report of a node the
// build module no longer runs on is therefore no builder's (issue 1940: mesh-build-node taken off ace on 2026-10-23 left
// ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// ToolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// ToolchainSet): an entry is passed on only while the placement (the seat mesh-placement's state `placement`, held by
// the assigner, delivery.PlacementBucket, read by delivery.Placed; ADR 0567, ADR 0549 §3) places its build module on
// its node, and only once it was written since that placement; it is passed on as deleted otherwise, said once, and
// passed on when its node writes it again. A report written before the placement is one an earlier run of the module
// left (issue 1940, 2026-10-25 20:56: mesh-build-node put on ace again brought ace.typescript v26.12.0 of 2026-10-23
// back into the agreement before the module ran there). Until the placement is settled, and while no node's placement
// is held (before the assigner's adopt writes the first one, or where the state cannot be read), every entry is passed
// on. Builders that run and report different versions still disagree: that refusal is the guard of novox/hq ADR 0465,
// and it stays.
//
// This replaces the copies of the gate the indexer, the builder and the checker each kept over the controller's
// assignments (issue 1940's subtask, issue 774: those go with the controller's lease, ADR 0549 §6).
type ToolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
placed *delivery.Placed
known bool // the placement is settled
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
// NewToolchainGate is a gate over the toolchains of modules, passing what stands on to forward and saying with logf what
// it leaves out.
func NewToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *ToolchainGate {
g := &ToolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, placed: delivery.NewPlaced(),
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *ToolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Placement takes one revision of the seat mesh-placement's placement in (at is when the bus wrote it) and answers whether what it
// passed on changed. A value that does not read is said, and places nothing on its node.
func (g *ToolchainGate) Placement(key string, value []byte, deleted bool, at time.Time) bool {
g.mu.Lock()
defer g.mu.Unlock()
was := g.placed.Seeded()
node, changed, err := g.placed.Put(key, value, deleted, at)
if err != nil {
g.logf("%s: %v (issue 1940)", delivery.PlacementBucket, err)
}
if !g.known {
return false
}
if was != g.placed.Seeded() {
// The first placement came, or the last went: every report's standing changed.
return g.all()
}
return g.again(node, changed)
}
// Settled says the placement's history is read: from now on only the reports of placed nodes pass, once a node's
// placement is held. It answers whether what it passed on changed.
func (g *ToolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
return g.all()
}
// all passes on every report again. The caller holds mu.
func (g *ToolchainGate) all() bool {
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// again passes on every report of node by a build module among modules. The caller holds mu.
func (g *ToolchainGate) again(node string, modules []string) bool {
changed := false
for _, module := range modules {
if !g.modules[module] {
continue
}
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is placed and wrote it since (or the placement is not
// known), deleted otherwise. The caller holds mu.
func (g *ToolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known && g.placed.Seeded() {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: the catalogue does not place the module on its
// node, or the report was written before it did. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *ToolchainGate) leftOut(module, node string, value []byte) string {
since, placed := g.placed.Since(node, module)
if !placed {
return fmt.Sprintf("the catalogue does not place %s on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s was placed on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// FollowPlacement reads the seat mesh-placement's placement with its history into gate, says it settled, and follows it; changed
// is told whenever what the gate passes on changed. Where the state cannot be read or watched (not there, or this part
// may not read it) it says so and the gate passes every report on: not fatal, since an indexer, builder or checker that
// stops for it stops every delivery (issue 1945). An error of the watch after that ends on the channel.
func FollowPlacement(ctx context.Context, js process.BucketOpener, gate *ToolchainGate, changed func(context.Context), logf func(string, ...any)) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, delivery.PlacementBucket)
if err != nil {
logf("the state %s cannot be read, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
w, err := kv.WatchAll(ctx, jetstream.IncludeHistory())
if err != nil {
logf("the state %s cannot be watched, so every node's toolchain report counts, a build node's the catalogue does not place too (issue 1940): %v", delivery.PlacementBucket, err)
return errs, nil
}
put := func(e jetstream.KeyValueEntry) bool {
return gate.Placement(e.Key(), e.Value(), e.Operation() != jetstream.KeyValuePut, e.Created())
}
for settled := false; !settled; {
select {
case <-ctx.Done():
_ = w.Stop()
return errs, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return errs, errors.New("the watch of " + delivery.PlacementBucket + " ended before it settled")
}
if e == nil {
settled = true
continue
}
put(e)
}
}
if gate.Settled() {
changed(ctx)
}
go func() {
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
errs <- ctx.Err()
return
case e, ok := <-w.Updates():
if !ok {
errs <- errors.New("the watch of " + delivery.PlacementBucket + " ended")
return
}
if e != nil && put(e) {
changed(ctx)
}
}
}
}()
return errs, nil
}
@@ -0,0 +1,263 @@
package buildrun
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func gateReport(t *testing.T, node, version string, at time.Time) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
// placing is a value of the state `placement` placing modules on node.
func placing(t *testing.T, node string, modules ...string) (string, []byte) {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
values, err := delivery.PlacementValues(p)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.PlacementNodeKey(node)
return key, values[key]
}
// placingLegacy is placing in the form before, `placement.<node>` (research 072 row 37).
func placingLegacy(t *testing.T, node string, modules ...string) (string, []byte) {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, _ := delivery.LegacyPlacementKey(node)
value, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// Issue 1940, the gate over the catalogue's placement (issue 774, ADR 0549 §3): a build node's report counts only while
// the placement places its build module on its node, and only once written since; before the placement is settled, and
// while no node's placement is held, every report counts. The live cases: mesh-build-node taken off ace left
// ace.typescript at v26.12.0 against v26.12.1 on the three others; put on ace again, its report of 2026-10-23 06:29 came
// back before the module ran there.
func TestToolchainGate(t *testing.T) {
old := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
placed := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node placed on ace again
three := []string{"g14", "novox", "shanks"}
start := func(t *testing.T, seeded bool) (*ToolchainGate, *ToolchainSet, *[]string) {
set := &ToolchainSet{}
said := &[]string{}
gate := NewToolchainGate([]string{GoBuildModule, NodeBuildModule}, set.Put, func(f string, a ...any) { *said = append(*said, fmt.Sprintf(f, a...)) })
for _, node := range three {
if seeded {
key, value := placing(t, node, NodeBuildModule)
gate.Placement(key, value, false, old.Add(-time.Hour))
}
key, value := gateReport(t, node, "v26.12.1", placed.Add(time.Minute))
gate.Toolchain(NodeBuildModule, key, value, false)
}
return gate, set, said
}
t.Run("not placed on ace: its report is no builder's", func(t *testing.T) {
gate, set, said := start(t, true)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
if a := delivery.ToolchainAgreement(set.Entries("typescript")); len(a.SittingOut) != 1 || a.SittingOut[0].Node != "ace" { // ADR 0584: the majority is agreed, ace's report counts as the one that sits out
t.Fatalf("before the placement is settled every report counts; got %+v", a)
}
k, v := placing(t, "ace", GoBuildModule) // another build module on ace: no matter
gate.Placement(k, v, false, old)
if !gate.Settled() {
t.Fatal("settling left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the placed nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "does not place mesh-build-node on ace") {
t.Fatalf("ace's report is said left out once; said %q", *said)
}
if gate.Toolchain(NodeBuildModule, key, value, false) {
t.Fatal("a report of a node not placed changed what was passed on")
}
// Placed on ace with a report written since: it counts, and the builders really disagree.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
gate.Placement(k, v, false, old.Add(-time.Minute))
if a := delivery.ToolchainAgreement(set.Entries("typescript")); a.Toolchain.Version != "v26.12.1" || len(a.SittingOut) != 1 || a.SittingOut[0].Version != "v26.12.0" {
t.Fatalf("builders that run different versions: the majority is agreed and ace sits out; got %+v", a)
}
// Taken off ace again (its key deleted): its report goes again.
if !gate.Placement(k, nil, true, placed) {
t.Fatal("taking ace's placement away passed nothing on")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
// A report deleted from the bucket goes whether its node is placed or not.
key, _ = gateReport(t, "g14", "v26.12.1", placed)
if !gate.Toolchain(NodeBuildModule, key, nil, true) || len(set.Entries("typescript")) != 2 {
t.Fatalf("deleting g14's report: %d entries left, want novox and shanks", len(set.Entries("typescript")))
}
})
for _, c := range []struct {
name string
at time.Time // when ace wrote its report
counts bool
}{
{"a report left before the placement", old, false},
{"a report written since the placement", placed.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
gate, set, said := start(t, true)
k, v := placing(t, "ace") // ace's key, first heard placing nothing
gate.Placement(k, v, false, old.Add(-time.Hour))
gate.Settled()
k, v = placing(t, "ace", NodeBuildModule)
if gate.Placement(k, v, false, placed) {
t.Fatal("a placement with no report passed something on")
}
key, value := gateReport(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// ace's placement written again with another module beside: its standing does not move, nothing is said again.
k, v = placing(t, "ace", GoBuildModule, NodeBuildModule)
if gate.Placement(k, v, false, placed.Add(time.Minute)) {
t.Fatal("the same placement of mesh-build-node changed what was passed on")
}
if c.counts {
return
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's; got %q, %q", v, none)
}
if len(*said) != 1 || !strings.Contains((*said)[0], "before mesh-build-node was placed on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", *said)
}
key, value = gateReport(t, "ace", "v26.12.1", placed.Add(time.Minute))
if !gate.Toolchain(NodeBuildModule, key, value, false) || len(set.Entries("typescript")) != 4 {
t.Fatal("ace's new report was not passed on")
}
})
}
t.Run("no placement held: every report counts", func(t *testing.T) {
gate, set, _ := start(t, false)
key, value := gateReport(t, "ace", "v26.12.0", old)
gate.Toolchain(NodeBuildModule, key, value, false)
gate.Settled()
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("before the first placement file every report counts; got %d", n)
}
k, v := placing(t, "g14", NodeBuildModule)
if !gate.Placement(k, v, false, old.Add(-time.Hour)) || len(set.Entries("typescript")) != 1 {
t.Fatalf("the first placement leaves every node it does not place out; got %d", len(set.Entries("typescript")))
}
if !gate.Placement(k, nil, true, placed) || len(set.Entries("typescript")) != 4 {
t.Fatal("the last placement gone: every report counts again")
}
})
t.Run("the history: a module in the first revision heard of a key stands since before it", func(t *testing.T) {
p := delivery.NewPlaced()
// ace in the form before, then converted: its module stands since before, the legacy key read no more.
k, v := placingLegacy(t, "ace", NodeBuildModule)
p.Put(k, v, false, old)
k, v = placing(t, "ace", NodeBuildModule)
if _, changed, err := p.Put(k, v, false, placed); len(changed) != 0 || err != nil {
t.Errorf("the conversion of ace's key changed %v, %v", changed, err)
}
if _, changed, _ := p.Put("ace."+NodeBuildModule, []byte("{}"), false, placed); len(changed) != 0 {
t.Errorf("a module key says which modules stand: %v", changed)
}
if k, v := placingLegacy(t, "ace"); true {
if _, changed, _ := p.Put(k, v, false, placed); len(changed) != 0 {
t.Errorf("the legacy key is read after the node key: %v", changed)
}
}
k2, v2 := placing(t, "g14")
p.Put(k2, v2, false, old)
k2, v2 = placing(t, "g14", NodeBuildModule)
if _, changed, _ := p.Put(k2, v2, false, placed); strings.Join(changed, ",") != NodeBuildModule {
t.Errorf("a module placed says changed %v", changed)
}
if _, _, err := p.Put("placement.novox", []byte("not json"), false, old); err == nil {
t.Error("a value that does not read was not said")
}
if s, ok := p.Since("ace", NodeBuildModule); !ok || !s.IsZero() {
t.Errorf("ace's module stands since %v, %v; want the zero time", s, ok)
}
if s, ok := p.Since("g14", NodeBuildModule); !ok || !s.Equal(placed) {
t.Errorf("g14's module stands since %v, %v; want the revision that placed it", s, ok)
}
if got := p.PlacedNodes(NodeBuildModule); strings.Join(got, ",") != "ace,g14" || len(p.ByNode()) != 2 || !p.Seeded() {
t.Errorf("PlacedNodes = %v, ByNode = %v", got, p.ByNode())
}
})
t.Run("a placement the bus refuses is said, and the part goes on", func(t *testing.T) {
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-delivery-assigner_placement")
gate, _, _ := start(t, false)
for _, bus := range []refusingBus{{err: refused}, {refuseOpen: true, err: refused}} {
if _, err := FollowPlacement(context.Background(), bus, gate, func(context.Context) {}, logger.Printf); err != nil {
t.Errorf("a refused placement stopped the start: %v", err)
}
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
})
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -0,0 +1,67 @@
package buildrun
import (
"context"
"encoding/json"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Caller calls a tool of a module on one named machine (mesh-sdk process.Conn.CallOn; ADR 0469 clause 3).
type Caller interface {
CallOn(node, key string, body any) (json.RawMessage, error)
}
// Steps waits for the end of a build module's long step: the value of its key `<node>.<id>` in the module's steps state
// (ADR 0469 clause 2) once its state is not running. It is a watch of that key, not a poll; it returns when the step
// ends or ctx does.
type Steps interface {
Wait(ctx context.Context, module, key string) (delivery.Step, error)
}
// Runner runs a build module's long steps: the call answers at once with the step's key, and the result is read from that
// key when the step ends (ADR 0469 clause 2).
type Runner struct {
Caller Caller
Steps Steps
// Bound is how long a step may run before the builder stops waiting for it.
Bound time.Duration
}
// Run calls tool of the build module on node with args, waits for the step's end and answers its result. A step that
// failed or was refused is an error naming the node, the state and the reason.
func (r Runner) Run(ctx context.Context, module, node, tool string, args map[string]any) (json.RawMessage, error) {
raw, err := r.Caller.CallOn(node, module+"."+tool, args)
if err != nil {
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
var answer struct {
Step string `json:"step"`
Key string `json:"key"`
State string `json:"state"`
}
if err := json.Unmarshal(raw, &answer); err != nil {
return nil, fmt.Errorf("%s on %s answered what is not a step: %w", tool, node, err)
}
if answer.Key == "" {
return nil, fmt.Errorf("%s on %s answered no step key: %s", tool, node, raw)
}
if n, _, err := delivery.ParseStepKey(answer.Key); err != nil || n != node {
return nil, fmt.Errorf("%s on %s answered the step key %q, which is not a key of that node", tool, node, answer.Key)
}
wait, cancel := context.WithTimeoutCause(ctx, r.Bound, fmt.Errorf("the step %s did not end within %s", answer.Key, r.Bound))
defer cancel()
step, err := r.Steps.Wait(wait, module, answer.Key)
if err != nil {
if cause := context.Cause(wait); cause != nil && ctx.Err() == nil {
err = cause
}
return nil, fmt.Errorf("%s on %s: %w", tool, node, err)
}
if step.State != delivery.StepPassed {
return nil, fmt.Errorf("%s on %s: the step %s %s: %s", tool, node, answer.Key, step.State, step.Reason)
}
return step.Result, nil
}
@@ -0,0 +1,125 @@
package buildrun
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
func runCallsTheToolOnTheNodeAndReadsTheStepsKey(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", delivery.StepPassed, "", `{"workspace":"w1","commit":"c"}`)}}
res, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", map[string]any{"repository": "novox/r", "commit": "c"})
if err != nil {
t.Fatal(err)
}
if len(caller.calls) != 1 || caller.calls[0].node != "ace" || caller.calls[0].key != "mesh-build-go.build_go_fetch" {
t.Errorf("calls = %+v, want one on ace for mesh-build-go.build_go_fetch", caller.calls)
}
var fr struct{ Workspace string }
if err := json.Unmarshal(res, &fr); err != nil || fr.Workspace != "w1" {
t.Errorf("result = %s (%v), want the step's result", res, err)
}
}
func runFailsOnAFailedOrRefusedStepNamingItsReason(t *testing.T) { // 1607/AC5
for _, state := range []string{delivery.StepFailed, delivery.StepRefused} {
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
steps := &fakeSteps{steps: map[string]delivery.Step{"mesh-build-go/ace.abc1": stepAt("ace", "abc1", state, "go is go1.26.0, the recipe builds with go1.27.1", "")}}
_, err := Runner{Caller: caller, Steps: steps, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "go1.27.1") || !strings.Contains(err.Error(), state) || !strings.Contains(err.Error(), "ace") {
t.Errorf("%s: error = %v, want the node, the state and the reason", state, err)
}
}
}
func runStopsWaitingAtTheBound(t *testing.T) { // 1607/AC5
caller := &fakeCaller{answer: `{"step":"abc1","key":"ace.abc1","state":"running"}`}
_, err := Runner{Caller: caller, Steps: &fakeSteps{}, Bound: 20 * time.Millisecond}.Run(context.Background(), "mesh-build-go", "ace", "build_go_build", nil)
if err == nil || !strings.Contains(err.Error(), "ace.abc1") || !strings.Contains(err.Error(), "did not end") {
t.Errorf("error = %v, want the step key and that it did not end within its bound", err)
}
}
func runRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode(t *testing.T) { // 1607/AC5
for _, answer := range []string{`{"state":"running"}`, `{"step":"abc1","key":"novox.abc1","state":"running"}`, `not json`} {
_, err := Runner{Caller: &fakeCaller{answer: answer}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if err == nil {
t.Errorf("answer %s: no error", answer)
}
}
}
func runReturnsTheCallsOwnError(t *testing.T) { // 1607/AC5
boom := errors.New("no responders")
_, err := Runner{Caller: &fakeCaller{err: boom}, Steps: &fakeSteps{}, Bound: time.Second}.Run(context.Background(), "mesh-build-go", "ace", "build_go_fetch", nil)
if !errors.Is(err, boom) || !strings.Contains(err.Error(), "ace") {
t.Errorf("error = %v, want the call's error naming the node", err)
}
}
// A step run on its node and read from its steps key, one table of cases (the test budget).
func TestRunningAStepOnANode(t *testing.T) {
t.Run("run calls the tool on the node and reads the steps key", runCallsTheToolOnTheNodeAndReadsTheStepsKey)
t.Run("run fails on a failed or refused step naming its reason", runFailsOnAFailedOrRefusedStepNamingItsReason)
t.Run("run stops waiting at the bound", runStopsWaitingAtTheBound)
t.Run("run refuses an answer with no key and a key of another node", runRefusesAnAnswerWithNoKeyAndAKeyOfAnotherNode)
t.Run("run returns the calls own error", runReturnsTheCallsOwnError)
}
@@ -0,0 +1,64 @@
package buildrun
import (
"sync"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// LanguageOf is the language each build module's toolchains state holds (the indexer's ToolchainModules, reversed).
var LanguageOf = map[string]string{GoBuildModule: "go", NodeBuildModule: "typescript"}
// ToolchainSet holds what the build modules' toolchains buckets hold: each node's entry of a language, as the watches
// put them in. Safe for use from the watches and from a build at once.
type ToolchainSet struct {
mu sync.Mutex
by map[string]map[string]delivery.Toolchain // language -> node -> entry
}
// Put takes a change of a build module's toolchains bucket in: a put of a node's entry (key <node>.<language>) or its
// delete. It answers whether the language's entries changed (a node's version or recipe, or a node appearing or going).
// A key or a value that is not an entry of the module's language is left out.
func (s *ToolchainSet) Put(module, key string, value []byte, deleted bool) bool {
language, ok := LanguageOf[module]
if !ok {
return false
}
node, keyLanguage, err := delivery.ParseToolchainKey(key)
if err != nil || keyLanguage != language {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if deleted {
if _, had := s.by[language][node]; !had {
return false
}
delete(s.by[language], node)
return true
}
tc, err := delivery.DecodeToolchain(value)
if err != nil || tc.Language != language {
return false
}
if s.by == nil {
s.by = map[string]map[string]delivery.Toolchain{}
}
if s.by[language] == nil {
s.by[language] = map[string]delivery.Toolchain{}
}
cur, had := s.by[language][node]
s.by[language][node] = tc
return !had || cur.Version != tc.Version || cur.Recipe != tc.Recipe
}
// Entries are the nodes' entries of a language, in no particular order.
func (s *ToolchainSet) Entries(language string) []delivery.NodeToolchain {
s.mu.Lock()
defer s.mu.Unlock()
var out []delivery.NodeToolchain
for node, tc := range s.by[language] {
out = append(out, delivery.NodeToolchain{Node: node, Toolchain: tc})
}
return out
}
@@ -0,0 +1,49 @@
package buildrun
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
func TestToolchainSetKeepsEachNodesEntryByLanguageAndSaysWhenOneChanged(t *testing.T) { // 1607/AC6
var s ToolchainSet
if !s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("a first entry did not count as a change")
}
if s.Put(GoBuildModule, "novox.go", tcValue(t, "go", "go1.27.1"), false) {
t.Error("the same entry again counted as a change")
}
s.Put(GoBuildModule, "ace.go", tcValue(t, "go", "go1.27.1"), false)
s.Put(NodeBuildModule, "ace.typescript", tcValue(t, "typescript", "tsc5"), false)
if got := s.Entries("go"); len(got) != 2 {
t.Errorf("go entries = %+v, want two", got)
}
if got := s.Entries("typescript"); len(got) != 1 || got[0].Node != "ace" || got[0].Version != "tsc5" {
t.Errorf("typescript entries = %+v", got)
}
if !s.Put(GoBuildModule, "ace.go", nil, true) || len(s.Entries("go")) != 1 {
t.Error("a delete did not remove the node's entry")
}
// What is not a toolchain of the module's language is left out.
s = ToolchainSet{}
if s.Put(GoBuildModule, "novox.typescript", tcValue(t, "typescript", "x"), false) {
t.Error("a typescript toolchain in the go build module's bucket counted")
}
if s.Put(GoBuildModule, "novox.go", []byte("not json"), false) || s.Put("other-module", "novox.go", tcValue(t, "go", "x"), false) {
t.Error("garbage or an unknown module counted")
}
if len(s.Entries("go")) != 0 {
t.Error("an entry was kept")
}
}
@@ -0,0 +1,271 @@
// Command mesh-delivery-builder-tools serves the builder's tools (novox/hq issue 1945): the runtime launches it for a call.
// builder_stuck reads the builder's builds and reasks states and the build modules' steps through the builder's state and
// reads; builder_ask_again writes one value of the reasks state, which the builder watches and answers. The builder is the
// other program: it serves no tool (ADR 0437 §4).
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// queueAsker writes a queues value and waits for the builder's answer to it.
type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, since build.LogSince, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{
{Name: build.StuckTool,
Description: "Every build key that waits (building or queued), oldest first: its module, fingerprint, step words and since when, and each ask it " +
"waits on by language with its id, when it was put, its asking (1 for the first; the builder asks an ask again when it can no longer be " +
"answered, at most 3 times) and what its build module's steps hold of it (none, or its state on its node since its start). The " +
"seat workers' delivery counts are not readable by the builder. dead_letters says whether the builder hears the asks the build seats give up " +
"on (issue 734), and why not: heard, a given-up ask is asked again at once; deaf, at its check bound. With module, that module's keys only. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's keys"}}},
Run: func(args map[string]any) (any, error) { return r.Stuck(str(args, "module")) }},
{Name: build.AskAgainTool,
Description: "Ask a waiting build key's asks again now, under new ids the key then names, as the builder does by itself for an ask its build " +
"seat's stream gave up on. The builder answers in the reasks state, which " + build.StuckTool + " shows beside the key; a key no ask of " +
"which waits is answered so and nothing is asked. An ask whose step already runs on a node is asked again too: it is then " +
"built twice, the first run's work wasted and its end read by nobody, never a wrong key; builder_stuck shows whether a step runs.",
Input: map[string]any{"type": "object", "required": []string{"module", "fp", "why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name"},
"fp": map[string]any{"type": "string", "description": "its new-form fingerprint, src2:<hex>, as builder_stuck shows it"},
"why": map[string]any{"type": "string", "description": "why it is asked again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := build.NewReAsk(str(args, "module"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
if err := askAgain(key, value); err != nil {
return nil, fmt.Errorf("the ask could not be written to %s: %w", build.ReAsksBucket, err)
}
return map[string]any{"key": key, "asked": true,
"next": "the builder answers in " + build.ReAsksState + "." + key + "; " + build.StuckTool + " shows the answer and the new ask"}, nil
}},
{Name: build.BuildAgainTool,
Description: "Build a failed key again (rebuild a module, build a failed key again): the builder writes the key of module and " +
"fingerprint building as a new revision of the same key and asks the build seats, as it does by itself when it looks at a module " +
"again (ADR 0429 clause 18). Answers the new revision and the ask id of each build module (go, node, image). Refuses a key that " +
"is not failed, and says why: done (nothing to do), or waiting (building or queued: " + build.AskAgainTool + " asks its waiting asks " +
"again); a key failed for a reason the builder never retries on the same fingerprint (a pin missing, a version taken, digests that " +
"differ) and a fingerprint its module no longer has are refused too. Use it for a key builder_stuck does not show, which failed.",
Input: map[string]any{"type": "object", "required": []string{"module", "fp", "why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name"},
"fp": map[string]any{"type": "string", "description": "its new-form fingerprint, src2:<hex>"},
"why": map[string]any{"type": "string", "description": "why it is built again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := build.NewBuildAgain(str(args, "module"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
raw, err := buildAgain(key, value)
if err != nil {
return nil, err
}
r, err := build.DecodeReAsk(raw)
if err != nil {
return nil, err
}
if r.Refused {
return nil, errors.New(r.Answer)
}
if r.Failed != "" {
return nil, errors.New(r.Answer)
}
if r.Revision == 0 && !r.Pending {
return nil, fmt.Errorf("the builder answered neither a new revision nor a refusal (an older builder?): %s", r.Answer)
}
return map[string]any{"key": key, "revision": r.Revision, "asks": r.Asks, "answer": r.Answer,
"next": build.StuckTool + " shows the key and its asks while they wait"}, nil
}},
{Name: delivery.BuildsSinceTool,
Description: "Every build that ended since a time, oldest first, from the builder's log build-log (kept 30 days): per build its module, " +
"fingerprint, source revision, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest built, the bases it was built against and each artifact made with its digest and reference. more says the " +
"log held more than one answer carries: ask again from the last build's time. What the operator's command nox builds reads. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"since"}, "properties": map[string]any{
"since": map[string]any{"type": "string", "description": "RFC 3339 time; the builds that ended at or after it"}}},
Run: func(args map[string]any) (any, error) {
at, err := time.Parse(time.RFC3339Nano, str(args, "since"))
if err != nil {
return nil, fmt.Errorf("since %q is not an RFC 3339 time: %w", str(args, "since"), err)
}
return build.BuildsSince(since, at)
}},
{Name: build.AnnounceAgainTool,
Description: "Announce a done build again (take in again, compose again): the builder writes the done key of module and fingerprint " +
"again, unchanged, as a new revision of the same key (its content, artifacts and manifest stay identical), so the controller's " +
"take-in hears it and composes the module and writes its grant again. Use it when a done build was taken in but its composition " +
"or grant is gone and every want is refused for it. Answers the new revision. Refuses a key that is not done: " + build.BuildAgainTool +
" builds a failed key again, " + build.AskAgainTool + " asks a waiting key's asks again.",
Input: map[string]any{"type": "object", "required": []string{"module", "fp", "why"}, "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "the module's name"},
"fp": map[string]any{"type": "string", "description": "its new-form fingerprint, src2:<hex>"},
"why": map[string]any{"type": "string", "description": "why it is announced again, in plain words"}}},
Run: func(args map[string]any) (any, error) {
key, value, err := build.NewAnnounceAgain(str(args, "module"), str(args, "fp"), str(args, "why"), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
raw, err := buildAgain(key, value)
if err != nil {
return nil, err
}
r, err := build.DecodeReAsk(raw)
if err != nil {
return nil, err
}
if r.Refused || r.Revision == 0 {
return nil, errors.New(r.Answer)
}
return map[string]any{"key": key, "revision": r.Revision, "answer": r.Answer}, nil
}},
{Name: build.SeatQueueTool,
Description: "What waits on a build seat's stream (SEAT_NODE_BUILD_<LANGUAGE>), as the builder reads it: the stream's message count and " +
"sequences, and per kind (delivery, check, recheck) its messages oldest first, at most limit each, with sequence, ask id, when asked " +
"and put, age, the state of the step a build node wrote for it, and whether the builder counts it as still waiting or as given up " +
"on (its worker delivered it its most times) or taken; and the ask asked last that a node took. The workers' delivery counts are " +
"not readable by the builder (nats.nats_streams shows them). Read by the builder, which alone may read the stream; read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"language": map[string]any{"type": "string", "description": "go, node or image (default go)"},
"seat": map[string]any{"type": "string", "description": "or the seat: node-build-go, node-build-node, node-build-image"},
"limit": map[string]any{"type": "integer", "description": fmt.Sprintf("messages read of each kind (default %d, at most %d)", build.DefaultQueueLimit, build.MaxQueueLimit)}}},
Run: func(args map[string]any) (any, error) {
language := build.LanguageOf(str(args, "language"))
if language == "" {
language = build.LanguageOf(str(args, "seat"))
}
if language == "" {
language = delivery.AskGo
}
limit, _ := args["limit"].(float64)
key, value, err := build.NewQueueAsk(language, int(limit), stdio.CallerOf(args), now())
if err != nil {
return nil, err
}
return queue(key, value)
}},
}
}
// askQueue writes the queues value at key and waits, on a watch of the key, at most bound for the builder's answer.
func askQueue(s stdio.KeptState, bound time.Duration) queueAsker {
return askKept(s, build.QueuesState, build.QueuesBucket, "the read", bound, func(v []byte) bool {
a, err := build.DecodeQueueAsk(v)
return err == nil && !a.Answered.IsZero()
})
}
// askKept writes a value at key of a state the builder watches and answers there, and waits, on a watch of the key, at most
// bound for a value written after it that answered says is the builder's answer.
func askKept(s stdio.KeptState, state, bucket, what string, bound time.Duration, answered func([]byte) bool) queueAsker {
return func(key string, value []byte) (json.RawMessage, error) {
heard := make(chan json.RawMessage, 1)
var rev uint64
var mu sync.Mutex
if err := s.Watch(key, func(c stdio.StateChange) error {
mu.Lock()
mine := rev
mu.Unlock()
if c.Op != "put" || mine == 0 || c.Revision <= mine {
return nil
}
if answered(c.Value) {
select {
case heard <- c.Value:
default:
}
}
return nil
}); err != nil {
return nil, fmt.Errorf("watching %s.%s: %w", state, key, err)
}
r, err := s.Put(key, json.RawMessage(value))
if err != nil {
return nil, fmt.Errorf("%s could not be written to %s: %w", what, bucket, err)
}
mu.Lock()
rev = r
mu.Unlock()
// The watch may have heard the answer before the revision was known: read it once too.
if e, err := s.Get(key); err == nil && e != nil && e.Revision > r && answered(e.Value) {
return e.Value, nil
}
wait, cancel := context.WithTimeout(context.Background(), bound) // a context's bound, no timer (ADR 0437 §8)
defer cancel()
select {
case v := <-heard:
return v, nil
case <-wait.Done():
return nil, fmt.Errorf("the builder did not answer %s.%s within %s: it is not running, or cannot read %s", state, key, bound, bucket)
}
}
}
func read(s stdio.KeptState) build.StateRead {
return build.StateRead{Keys: s.Keys, Get: func(key string) (json.RawMessage, time.Time, error) {
e, err := s.Get(key)
if err != nil || e == nil {
return nil, time.Time{}, err
}
return e.Value, time.Time{}, nil
}, Written: func(key string) time.Time {
e, err := s.Get(key)
if err != nil || e == nil {
return time.Time{}
}
return written(s, key, e.Revision)
}}
}
// written is when the key's revision was written, from its history, or zero.
func written(s stdio.KeptState, key string, revision uint64) time.Time {
h, err := s.History(key)
if err != nil {
return time.Time{}
}
for _, r := range h {
if r.Revision == revision {
return r.At
}
}
return time.Time{}
}
func main() {
steps := map[string]build.StateRead{}
for _, bm := range build.StuckBuildModules {
steps[bm] = read(stdio.State(bm + ".steps"))
}
reasks := stdio.State(build.ReAsksState)
r := build.StuckReader{Logf: func(f string, a ...any) { fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] "+f+"\n", a...) }, Builds: read(stdio.State("builds")), ReAsks: read(reasks), Steps: steps,
Hearing: read(stdio.State(buildrun.HearingState))}
put := func(key string, value []byte) error {
_, err := reasks.Put(key, json.RawMessage(value))
return err
}
buildAgain := askKept(reasks, build.ReAsksState, build.ReAsksBucket, "the request", 25*time.Second, func(v []byte) bool {
a, err := build.DecodeReAsk(v)
return err == nil && !a.Answered.IsZero()
})
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, stdio.Log(delivery.BuilderLog).Since, time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1)
}
}
@@ -0,0 +1,128 @@
package main
import (
"context"
"errors"
"fmt"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// bucketOpener is the part of the bus the watches use.
type bucketOpener interface {
KeyValue(ctx context.Context, bucket string) (jetstream.KeyValue, error)
}
// kvStore is the builds bucket as the builder's store: create and compare-and-set.
type kvStore struct{ kv jetstream.KeyValue }
func conflict(err error) error {
if errors.Is(err, jetstream.ErrKeyExists) || errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
return build.ErrConflict
}
return err
}
func (s kvStore) Get(ctx context.Context, key string) ([]byte, uint64, bool, error) {
e, err := s.kv.Get(ctx, key)
if errors.Is(err, jetstream.ErrKeyNotFound) {
return nil, 0, false, nil
}
if err != nil {
return nil, 0, false, err
}
return e.Value(), e.Revision(), true, nil
}
func (s kvStore) Create(ctx context.Context, key string, value []byte) (uint64, error) {
rev, err := s.kv.Create(ctx, key, value)
return rev, conflict(err)
}
func (s kvStore) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
rev, err := s.kv.Update(ctx, key, value, revision)
return rev, conflict(err)
}
func (s kvStore) Keys(ctx context.Context) ([]string, error) {
keys, err := s.kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
func opOf(e jetstream.KeyValueEntry) bool { return e.Operation() == jetstream.KeyValuePut }
// settle reads the entries a watcher holds now, up to its mark of the last values, and returns the watcher for the
// changes that follow.
func settle(ctx context.Context, kv jetstream.KeyValue, each func(jetstream.KeyValueEntry)) (jetstream.KeyWatcher, error) {
w, err := kv.WatchAll(ctx)
if err != nil {
return nil, err
}
return settled(ctx, w, each)
}
// settled hands each entry the watch delivers before its end-of-initial marker to each, and answers the watch.
func settled(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.KeyValueEntry)) (jetstream.KeyWatcher, error) {
for {
select {
case <-ctx.Done():
w.Stop()
return nil, ctx.Err()
case e, ok := <-w.Updates():
if !ok {
return nil, errors.New("the watch ended")
}
if e == nil {
return w, nil
}
each(e)
}
}
}
// follow hands each change of a watcher to each until ctx ends; a watch that ends is an error, and the process manager
// starts the program again.
func follow(ctx context.Context, w jetstream.KeyWatcher, each func(jetstream.KeyValueEntry)) error {
defer w.Stop()
for {
select {
case <-ctx.Done():
return nil
case e, ok := <-w.Updates():
if !ok {
return errors.New("a watch ended")
}
if e != nil {
each(e)
}
}
}
}
// manifestReader reads manifests.<digest> of the indexer and parses the manifest it holds.
type manifestReader struct{ kv jetstream.KeyValue }
func (m manifestReader) Manifest(ctx context.Context, digest string) (catalogue.Manifest, error) {
key, err := delivery.ManifestsKey(digest)
if err != nil {
return catalogue.Manifest{}, err
}
e, err := m.kv.Get(ctx, key)
if err != nil {
return catalogue.Manifest{}, fmt.Errorf("manifests.%s: %w", key, err)
}
kept, err := delivery.DecodeManifest(e.Value())
if err != nil {
return catalogue.Manifest{}, err
}
// Decoded, never judged (novox/hq ADR 0578 §3): what a build needs of it is read, a field it does not know set aside.
return catalogue.Decode([]byte(kept.JSON))
}
@@ -0,0 +1,186 @@
//go:build integration
package main
import (
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again, by the builder whole against the bus
// (novox/hq issue 1937): on 2026-10-24 the store's 03:30 collection deleted the bare blobs of three done keys the
// controller had never taken in, and the builder kept them done.
// heldVerb answers the seat mesh-artifact-sweep's verb held on the bus, as its holder does: {result: {output, ok,
// answer}}; a reference is gone while gone says so.
type heldVerb struct {
mu sync.Mutex
gone func(reference string) bool
asked []string
}
func (h *heldVerb) serve(t *testing.T, b *bed) {
t.Helper()
sub, err := b.js.Conn().Subscribe("mesh.seat."+catalogue.ArtifactSweepSeat+".call.held.>", func(m *nats.Msg) {
var args struct {
References []string `json:"references"`
}
_ = json.Unmarshal(m.Data, &args)
a := delivery.HeldAnswer{Store: "store.test:5100", Held: []string{}, Gone: []string{}}
h.mu.Lock()
for _, r := range args.References {
h.asked = append(h.asked, r)
if h.gone(r) {
a.Gone = append(a.Gone, r)
} else {
a.Held = append(a.Held, r)
}
}
h.mu.Unlock()
reply, _ := json.Marshal(map[string]any{"result": map[string]any{"output": "held", "ok": true, "answer": a}, "node": "ctl"})
_ = m.Respond(reply)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
func (h *heldVerb) asks() int {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.asked)
}
// eventsConsumer makes EVENTS and the builder's durable consumer of artifacts-gone on it, as the controller makes them
// from the manifest's consumes.
func eventsConsumer(t *testing.T, b *bed, node string) {
t.Helper()
if _, err := b.js.CreateStream(b.ctx, jetstream.StreamConfig{Name: eventsStream, Subjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
if _, err := b.js.CreateOrUpdateConsumer(b.ctx, eventsStream, jetstream.ConsumerConfig{Durable: node + "_mesh-delivery-builder",
AckPolicy: jetstream.AckExplicitPolicy, FilterSubjects: []string{delivery.ArtifactsGoneSubject}}); err != nil {
t.Fatal(err)
}
}
// doneKey writes builds.<module>.<fp> done, naming one archive at reference, and answers its revision.
func doneKey(t *testing.T, b *bed, module, manifest, reference, digest string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
data, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildDone, By: "n2.mesh-delivery-builder",
Manifest: manifest, Step: "built mesh-build-go on n2",
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Digest: digest, Reference: reference}}})
if err != nil {
t.Fatal(err)
}
rev, err := b.builds.Put(b.ctx, key, data)
if err != nil {
t.Fatal(err)
}
return rev
}
func revisionOf(t *testing.T, b *bed, module string) uint64 {
t.Helper()
key, _ := delivery.BuilderBuildKey(module, fp)
e, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
return e.Revision()
}
// 1937/AC3 (integration): the builder, started whole, finds the done key of the trunk's fingerprint naming a blob held
// answers gone, builds it again as a new revision whose step says so, and leaves the rebuilt key done.
func TestADoneKeyWhoseBlobIsGoneIsBuiltAgainAtStart(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
old := digestOf("collected last night")
oldRef := "artifact-store://ca-trust/files/blobs/" + old
rev := doneKey(t, b, "ca-trust", manifest, oldRef, old)
h := &heldVerb{gone: func(r string) bool { return r == oldRef }}
h.serve(t, b)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && len(v.Artifacts) == 1 && v.Artifacts[0].Digest == digestOf("files")
})
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+oldRef) {
t.Errorf("the rebuilt key's step %q does not say why it was built again", v.Step)
}
if got := revisionOf(t, b, "ca-trust"); got <= rev {
t.Errorf("revision %d, was %d: want a new revision", got, rev)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("the archive was built %d times, want once", n)
}
if h.asks() == 0 {
t.Error("the builder never asked held")
}
}
// 1937/AC4 (integration): a done key whose blob held answers held at start stays done; when the controller says
// artifacts-gone for it, the builder asks again and builds it again.
func TestArtifactsGoneLeadsTheBuilderToBuildTheKeyAgain(t *testing.T) {
b := startBed(t)
eventsConsumer(t, b, "n1")
manifest := b.archive(t, "ca-trust", "modules/ca-trust", commit)
d := digestOf("files")
ref := "artifact-store://ca-trust/files/blobs/" + d
rev := doneKey(t, b, "ca-trust", manifest, ref, d)
var mu sync.Mutex
collected := false
h := &heldVerb{gone: func(r string) bool {
mu.Lock()
defer mu.Unlock()
if r == ref && collected {
collected = false // gone once: the rebuild pushes it again
return true
}
return false
}}
h.serve(t, b)
b.start(t, "n1", nil)
for deadline := time.Now().Add(20 * time.Second); h.asks() == 0 && time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
}
if h.asks() == 0 {
t.Fatal("the builder never asked held at its start")
}
time.Sleep(300 * time.Millisecond)
if got := revisionOf(t, b, "ca-trust"); got != rev || b.fake.count("mesh-build-go.build_go_build") != 0 {
t.Fatalf("a key whose blob is held was built again (revision %d, was %d)", got, rev)
}
mu.Lock()
collected = true
mu.Unlock()
key, _ := delivery.BuilderBuildKey("ca-trust", fp)
body, err := delivery.EncodeArtifactsGone(delivery.ArtifactsGone{Module: "ca-trust", Key: key, Fingerprint: fp,
Gone: []string{ref}, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, delivery.ArtifactsGoneSubject, body); err != nil {
t.Fatal(err)
}
v := b.await(t, "ca-trust", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && strings.Contains(v.Step, "built again")
})
if got := revisionOf(t, b, "ca-trust"); got <= rev || b.fake.count("mesh-build-go.build_go_build") != 1 {
t.Errorf("build = %+v at revision %d (was %d), %d builds: want built again once on the event", v, got, rev,
b.fake.count("mesh-build-go.build_go_build"))
}
}
@@ -0,0 +1,306 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// heldOnTheBus says whether an ask a build key names is on its build seat's stream, or a build node wrote a step for it
// (issue 1910), read once at the take-over of a key at start; and, at the builder's check bound, whether an ask can still
// be answered (issue 1945). Never polled.
type heldOnTheBus struct{ js jetstream.JetStream }
// Held is build.Builder's Held: a step key `<node>.<ask id>` in the build module's steps, or a message of the stream under
// the ask's subject whose Nats-Msg-Id is the ask's id.
func (h heldOnTheBus) Held(ctx context.Context, a delivery.StepAsk) (bool, error) {
seen, err := h.seen(ctx, a, false)
if err != nil {
return false, err
}
return seen.Step || seen.OnStream, nil
}
// Gone is build.Builder's Gone: why the ask can no longer be answered (build.AskSeen.Why), or "".
func (h heldOnTheBus) Gone(ctx context.Context, a delivery.StepAsk) (string, error) {
seen, err := h.seen(ctx, a, true)
if err != nil {
return "", err
}
return seen.Why(), nil
}
// maxOnSubject bounds the messages of one subject a read walks (each a read by sequence, next by subject), and maxSteps
// the steps it reads whole to find the last ask taken: past them the read says it cannot tell, and the ask is waited for.
const (
maxOnSubject = 5000
maxSteps = 200
)
// seen reads what the bus holds of the ask; with later, the ask asked last that a build node took too (AskSeen.Last), when
// the ask is on its stream.
func (h heldOnTheBus) seen(ctx context.Context, a delivery.StepAsk, later bool) (build.AskSeen, error) {
seen := build.AskSeen{Kind: a.Kind, At: build.AskedAt(a.ID)}
module, err := delivery.BuildModuleOf(a.Language)
if err != nil {
return seen, err
}
steps, err := h.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return seen, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
pattern, err := delivery.AskStepWatch(a.ID)
if err != nil {
return seen, err
}
keys, err := steps.ListKeysFiltered(ctx, pattern)
if err != nil {
return seen, err
}
for range keys.Keys() {
keys.Stop()
seen.Step = true // a node took it
return seen, nil
}
stream, name, subject, err := h.stream(ctx, a.Language, a.Kind)
seen.Stream = name
if errors.Is(err, jetstream.ErrStreamNotFound) {
return seen, nil
}
if err != nil {
return seen, err
}
var ahead []build.Queued
err = onSubject(ctx, stream, subject, func(q build.Queued) bool {
if q.Ask == a.ID {
seen.OnStream, seen.PutAt = true, q.PutAt
return false
}
ahead = append(ahead, q)
return true
})
if err != nil || !seen.OnStream || !later {
return seen, err
}
taken, err := stepKeys(ctx, steps)
if err != nil {
return seen, err
}
for i := range ahead {
ahead[i].Step = stepState(ctx, steps, taken[ahead[i].Ask])
}
seen.Ahead = ahead
since := seen.At
if since.IsZero() {
since = seen.PutAt
}
for _, q := range ahead {
if q.At.Before(since) {
since = q.At
}
}
seen.Last = lastTaken(ctx, steps, taken, a.Kind, since)
return seen, nil
}
// stream is the build seat's stream of language, its name and the subject of kind on it.
func (h heldOnTheBus) stream(ctx context.Context, language, kind string) (jetstream.Stream, string, string, error) {
subject, err := delivery.AskSubject(language, kind)
if err != nil {
return nil, "", "", err
}
// The seat's stream by its name (delivery.AskStream), as the broker grants the builder its information and its
// messages by sequence and nothing wider (issue 1910).
name, err := delivery.AskStream(language)
if err != nil {
return nil, "", "", err
}
stream, err := h.js.Stream(ctx, name)
return stream, name, subject, err
}
// onSubject hands each message of subject on stream to each, in order, until each answers false or the subject's
// messages end. A seat's stream is a work queue, which takes no second consumer over its subjects: its messages are read
// by sequence instead, each the next of the subject (one read per message, none for the sequences taken between), and
// none consumed.
func onSubject(ctx context.Context, stream jetstream.Stream, subject string, each func(build.Queued) bool) error {
for seq, n := uint64(1), 0; ; n++ {
if n >= maxOnSubject {
return fmt.Errorf("%s holds more than %d messages before the one asked after: not read further", subject, maxOnSubject)
}
m, err := stream.GetMsg(ctx, seq, jetstream.WithGetMsgSubject(subject))
if errors.Is(err, jetstream.ErrMsgNotFound) {
return nil
}
if err != nil {
return err
}
q := build.Queued{Seq: m.Sequence, Ask: m.Header.Get(jetstream.MsgIDHeader), PutAt: m.Time}
if _, kind, err := delivery.ParseAskSubject(m.Subject); err == nil {
q.Kind = kind
}
if q.At = build.AskedAt(q.Ask); q.At.IsZero() {
q.At = q.PutAt
}
if !each(q) {
return nil
}
seq = m.Sequence + 1
}
}
// stepKeys are the build module's step keys by ask id: `<node>.<ask id>` (a node's own key, such as its pause, has no
// ask id and is left out).
func stepKeys(ctx context.Context, steps jetstream.KeyValue) (map[string][]string, error) {
all, err := steps.ListKeys(ctx)
if err != nil {
if errors.Is(err, jetstream.ErrNoKeysFound) {
return map[string][]string{}, nil
}
return nil, err
}
defer all.Stop()
out := map[string][]string{}
for k := range all.Keys() {
if _, id, ok := strings.Cut(k, "."); ok && !build.AskedAt(id).IsZero() {
out[id] = append(out[id], k)
}
}
return out, nil
}
// stepState is the state of the first of keys a node wrote, "" for none.
func stepState(ctx context.Context, steps jetstream.KeyValue, keys []string) string {
for _, k := range keys {
e, err := steps.Get(ctx, k)
if err != nil {
continue
}
if s, err := delivery.DecodeStep(e.Value()); err == nil {
return s.State
}
}
return ""
}
// lastTaken is the ask asked last after since that a build node took, of a kind whose taking proves kind passed
// (build.Proves), read newest first and at most maxSteps of them; nil for none. A step's tool is "<kind> ask"
// (mesh-sdk buildstep).
func lastTaken(ctx context.Context, steps jetstream.KeyValue, taken map[string][]string, kind string, since time.Time) *build.Taken {
var ids []string
for id := range taken {
if build.AskedAt(id).After(since) {
ids = append(ids, id)
}
}
sort.Slice(ids, func(i, j int) bool { return build.AskedAt(ids[i]).After(build.AskedAt(ids[j])) })
read := 0
for _, id := range ids {
for _, k := range taken[id] {
if read >= maxSteps {
return nil
}
read++
e, err := steps.Get(ctx, k)
if err != nil {
continue
}
s, err := delivery.DecodeStep(e.Value())
if err != nil {
continue
}
stepKind, ok := strings.CutSuffix(s.Tool, " ask")
if !ok || !build.Proves(stepKind, kind) {
continue
}
return &build.Taken{Ask: id, Kind: stepKind, At: build.AskedAt(id), Node: s.Node, Started: s.Started}
}
}
return nil
}
// Drop is build.Dropper's Drop: it removes the ask's message from its build seat's stream, found by its Nats-Msg-Id on its
// subject, when the stream still holds it. The builder's account must be granted $JS.API.STREAM.MSG.DELETE.<stream>.
func (h heldOnTheBus) Drop(ctx context.Context, a delivery.StepAsk) error {
stream, name, subject, err := h.stream(ctx, a.Language, a.Kind)
if errors.Is(err, jetstream.ErrStreamNotFound) {
return nil
}
if err != nil {
return err
}
var seq uint64
if err := onSubject(ctx, stream, subject, func(q build.Queued) bool {
if q.Ask == a.ID {
seq = q.Seq
return false
}
return true
}); err != nil || seq == 0 {
return err
}
if err := stream.DeleteMsg(ctx, seq); err != nil && !errors.Is(err, jetstream.ErrMsgNotFound) {
return fmt.Errorf("deleting message %d of %s (is $JS.API.STREAM.MSG.DELETE.%s granted to the builder?): %w", seq, name, name, err)
}
return nil
}
// Queue is what the tool builder_seat_queue answers (build.SeatQueue): the messages of language's build seat's stream, of
// every kind, oldest first and at most limit of each kind, each with its step's state and whether it still waits.
func (h heldOnTheBus) Queue(ctx context.Context, language string, limit int) (build.SeatQueue, error) {
q := build.SeatQueue{Language: language, Note: build.QueueNote}
module, err := delivery.BuildModuleOf(language)
if err != nil {
return q, err
}
stream, name, _, err := h.stream(ctx, language, delivery.AskDelivery)
q.Stream = name
if errors.Is(err, jetstream.ErrStreamNotFound) {
q.Note = name + " does not exist"
return q, nil
}
if err != nil {
return q, err
}
info, err := stream.Info(ctx)
if err != nil {
return q, err
}
q.Messages, q.FirstSeq, q.LastSeq = info.State.Msgs, info.State.FirstSeq, info.State.LastSeq
steps, err := h.js.KeyValue(ctx, delivery.StepsBucket(module))
if err != nil {
return q, fmt.Errorf("the bucket %s: %w", delivery.StepsBucket(module), err)
}
taken, err := stepKeys(ctx, steps)
if err != nil {
return q, err
}
for _, kind := range delivery.AskKinds {
subject, _ := delivery.AskSubject(language, kind)
var ahead []build.Queued
if err := onSubject(ctx, stream, subject, func(m build.Queued) bool {
ahead = append(ahead, m)
return len(ahead) < limit
}); err != nil {
return q, err
}
if len(ahead) == 0 {
continue
}
for i := range ahead {
ahead[i].Step = stepState(ctx, steps, taken[ahead[i].Ask])
}
seen := build.AskSeen{Kind: kind, Ahead: ahead, Last: lastTaken(ctx, steps, taken, kind, ahead[0].At)}
q.Kinds = append(q.Kinds, build.NewKindQueue(kind, seen, time.Now()))
}
return q, nil
}
@@ -0,0 +1,912 @@
//go:build integration
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
// The build runner the build modules pull and run asks with: mesh-build-go's own package (novox/hq ADR 0593 §1),
// another repository's, by its published version, so the builder is tested against the real pulling code.
runner "git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build"
)
// busConf is the shape of the composed accounts file (mesh-controller internal/broker): one account with JetStream, the
// controller's user and the builder's own principals <node>.<module> (two instances, for the test of one build).
const busConf = `
listen: 127.0.0.1:-1
jetstream { store_dir: %q }
accounts {
MESH {
jetstream: enabled
users = [
{ user: "ctl", password: "pw" }
{ user: "n1.mesh-delivery-builder", password: "pw" }
{ user: "n2.mesh-delivery-builder", password: "pw" }
]
}
}
`
const (
commit = "7a74cee83f88e6585a5511c3cc53bdf0f5303d20"
fp = "src2:3333333333333333333333333333333333333333333333333333333333333333"
)
// fakeBuildModules are the build modules on the nodes as the real ones run since R4g (ADR 0525; issue 1887 A4): each pulls
// its build seat's asks on its nodes (runner.Puller over the seat's workers) and runs an ask's job with step bodies that
// answer as the real tools do, ending the step on <node>.<ask id> of its steps bucket.
type fakeBuildModules struct {
js jetstream.JetStream
ctx context.Context
failBuild atomic.Bool
// refusePush makes every bundle and archive build fail as a push the artifact store refused (ADR 0498 clause 2).
refusePush atomic.Bool
mu sync.Mutex
calls map[string]int // by `<module>.<tool>`
fetches []map[string]any
builds []map[string]any // the arguments of every build call
// closed, while set, keeps every node's capacity at 0: no ask is pulled until open is called (a test of the asks
// waiting in their streams).
closed atomic.Bool
pullers []*runner.Puller
}
// open lets the nodes pull again and wakes their pullers.
func (f *fakeBuildModules) open() {
f.closed.Store(false)
f.mu.Lock()
ps := append([]*runner.Puller(nil), f.pullers...)
f.mu.Unlock()
for _, p := range ps {
p.Wake()
}
}
func (f *fakeBuildModules) count(tool string) int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls[tool]
}
// digestOf is what every node builds an artifact to.
func digestOf(artifact string) string {
sum := sha256.Sum256([]byte(artifact))
return "sha256:" + hex.EncodeToString(sum[:])
}
var prefixOf = map[string]string{"mesh-build-go": "build_go_", "mesh-build-node": "build_node_", "mesh-build-image": "build_image_"}
// step is one tool's answer, as the real module's.
func (f *fakeBuildModules) step(module, name string, args map[string]any) (any, error) {
tool := prefixOf[module] + name
f.mu.Lock()
f.calls[module+"."+tool]++
if name == "fetch" {
f.fetches = append(f.fetches, args)
} else {
f.builds = append(f.builds, args)
}
f.mu.Unlock()
switch {
case name == "fetch":
merge := ""
if args["base"] != nil { // a fetch with a base answers the merge commit (mesh-build-go FetchResult)
merge = `,"merge":"` + strings.Repeat("9", 40) + `"`
}
return json.RawMessage(`{"workspace":"w` + buildstep.NewID() + `","commit":"` + commit + `"` + merge + `}`), nil
case f.failBuild.Load() && tool != "build_image_copy": // the build fails wherever it runs
return nil, fmt.Errorf("go build of %v: boom", args["artifact"])
case tool == "build_image_copy": // the upstream copy answers the digest and the registry's reference (ADR 0498 clause 4)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"artifact":"` + fmt.Sprint(args["artifact"]) + `","digest":"` + d + `","reference":"registry.example/` + fmt.Sprint(args["module"]) + `/` + fmt.Sprint(args["artifact"]) + `@` + d + `","pushed":true}`), nil
case tool != "build_image_build" && f.refusePush.Load():
return nil, fmt.Errorf("%s%v: the artifact store answered 507 Insufficient Storage", delivery.NotStored, args["artifact"])
case tool != "build_image_build": // a bundle or an archive is pushed and its reference answered (ADR 0498 clause 1)
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"artifact-store://` + filepath.Base(fmt.Sprint(args["module"])) + `/` + fmt.Sprint(args["artifact"]) + `/blobs/` + d + `"}`), nil
case tool == "build_image_build" && args["publish"] == true: // a trunk image is pushed: the digest and the reference
d := digestOf(fmt.Sprint(args["artifact"]))
return json.RawMessage(`{"digest":"` + d + `","reference":"registry.example/imaged/server@` + d + `"}`), nil
default:
return json.RawMessage(`{"digest":"` + digestOf(fmt.Sprint(args["artifact"])) + `"}`), nil
}
}
// serve makes module's steps bucket and its build seat's stream and workers, and pulls its asks on nodes.
func (f *fakeBuildModules) serve(t *testing.T, module string, nodes ...string) {
kv, err := f.js.CreateKeyValue(f.ctx, jetstream.KeyValueConfig{Bucket: delivery.StepsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
language := strings.TrimPrefix(module, "mesh-build-")
if err := runner.DeclareAskWorkers(f.ctx, f.js, language, 30*time.Second); err != nil {
t.Fatal(err)
}
put := func(key string, value any) (uint64, error) {
raw, _ := value.(json.RawMessage)
return kv.Put(f.ctx, key, raw)
}
call := func(name string) runner.StepFunc {
return func(ctx context.Context, args map[string]any) (any, error) { return f.step(module, name, args) }
}
steps := runner.AskSteps{Fetch: call("fetch"), Build: call("build"), Copy: call("copy")}
for _, node := range nodes {
p := &runner.Puller{Source: &runner.JetStreamAsks{JS: f.js, Language: language}, Language: language, Node: node, Put: put,
Capacity: func() int {
if f.closed.Load() {
return 0
}
return 2
}, PullWait: 200 * time.Millisecond,
Run: func(ctx context.Context, a delivery.StepAsk) (any, error) { return runner.RunAsk(ctx, steps, a) }}
f.mu.Lock()
f.pullers = append(f.pullers, p)
f.mu.Unlock()
go func() { _ = p.Pull(f.ctx) }()
}
}
// trunkHead is the trunk head of novox/mesh-catalog the bed's source revision names: newer than the module values' commit.
const trunkHead = "3333333333333333333333333333333333333333"
// trunk writes the gate's source.main naming head as novox/mesh-catalog's trunk head, and answers its revision.
func (b *bed) trunk(t *testing.T, head string) uint64 {
t.Helper()
data, err := delivery.EncodeSource(delivery.Source{V: delivery.Version, Group: "g", Heads: map[string]string{"novox/mesh-catalog": head}})
if err != nil {
t.Fatal(err)
}
rev, err := b.source.Put(b.ctx, delivery.SourceMainKey, data)
if err != nil {
t.Fatal(err)
}
return rev
}
type bed struct {
ctx context.Context
js jetstream.JetStream
modules jetstream.KeyValue
candidates jetstream.KeyValue
source jetstream.KeyValue
builds jetstream.KeyValue
tools map[string]jetstream.KeyValue
instances jetstream.KeyValue
stop func() // stops the builder started last
fake *fakeBuildModules
srvURL string
dir string
}
// startBed starts the bus, the buckets the controller makes from the builder's and the build modules' manifests, the
// build modules' streams and the builder's durable consumers on them, two nodes' toolchains, the indexer's manifests, and
// the fake build modules. It runs no builder.
func startBed(t *testing.T) *bed { return startBedWith(t, false) }
// startBedWith is startBed with the nodes' capacity 0 from the start when closed: a puller that reads its capacity before a
// test could close the nodes may be in a pull already and takes the first ask put (issue 1928).
func startBedWith(t *testing.T, closed bool) *bed {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "b")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
conf := filepath.Join(dir, "bus.conf")
if err := os.WriteFile(conf, []byte(fmt.Sprintf(busConf, filepath.Join(dir, "js"))), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatal(err)
}
opts.NoLog, opts.NoSigs = true, true
srv, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(srv.Shutdown)
ctl, err := nats.Connect(srv.ClientURL(), nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(ctl.Close)
js, err := jetstream.New(ctl)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
t.Cleanup(cancel)
b := &bed{ctx: ctx, js: js, tools: map[string]jetstream.KeyValue{}, srvURL: srv.ClientURL(), dir: dir}
for module, stream := range toolchainStreams {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: stream, Subjects: []string{"mesh.stream." + module + ".event.toolchain.reported"}}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"n1", "n2"} {
if _, err := js.CreateOrUpdateConsumer(ctx, stream, jetstream.ConsumerConfig{Durable: node + "_mesh-delivery-builder", AckPolicy: jetstream.AckExplicitPolicy}); err != nil {
t.Fatal(err)
}
}
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket(module), History: 1})
if err != nil {
t.Fatal(err)
}
b.tools[module] = kv
}
mk := func(name string, history int) jetstream.KeyValue {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: name, History: uint8(history)})
if err != nil {
t.Fatal(err)
}
return kv
}
b.modules = mk(delivery.ModulesBucket, delivery.ModulesHistory)
b.candidates = mk(delivery.CandidatesBucket, delivery.CandidatesHistory)
b.source = mk(delivery.SourceBucket, delivery.SourceHistory)
mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
b.instances = mk(instancesBucket, 1)
// The builder's module log, as the runtime raises it (ADR 0297): its stream on the log's subjects.
logStream := buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: logStream, Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"},
AllowDirect: true}); err != nil {
t.Fatal(err)
}
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.closed.Store(closed)
b.fake.serve(t, "mesh-build-go", "n1", "n2")
b.fake.serve(t, "mesh-build-node", "n1", "n2")
b.fake.serve(t, "mesh-build-image", "n1") // the image build module runs on n1 only
b.report(t, "mesh-build-go", "n1", "go", "go1.27.1", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.1", "1")
b.report(t, "mesh-build-node", "n1", "typescript", "tsc5", "1")
b.report(t, "mesh-build-node", "n2", "typescript", "tsc5", "1")
return b
}
// logged is every entry of module's key in the builder's log, read through process.Conn.ReadLog on the controller's user.
func (b *bed) logged(t *testing.T, module string) []delivery.BuildLogEntry {
t.Helper()
conn, err := process.Connect(process.Account{URL: b.srvURL, Module: "mesh-delivery-view", User: "ctl", Password: "pw"})
if err != nil {
t.Fatal(err)
}
defer conn.Close()
entries, err := conn.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Since(time.Unix(0, 0), module)
if err != nil {
t.Fatal(err)
}
var out []delivery.BuildLogEntry
for _, le := range entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
t.Fatal(err)
}
out = append(out, e)
}
return out
}
// report writes a node's toolchains key of a build module and says so on the module's stream, as the module does.
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper()
data, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: recipe, At: time.Now().UTC()})
if err != nil {
t.Fatal(err)
}
key, _ := delivery.ToolchainKey(node, language)
if _, err := b.tools[module].Put(b.ctx, key, data); err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, "mesh.stream."+module+".event.toolchain.reported", []byte(`{}`)); err != nil {
t.Fatal(err)
}
}
// module writes modules.<name> as the indexer does for a module with a new-form fingerprint, and its manifest by digest.
func (b *bed) module(t *testing.T, name, fixture, path string) string {
return b.moduleWith(t, name, fixture, path, map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}})
}
// moduleWith is module with the toolchains its fingerprint was computed with.
func (b *bed) moduleWith(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed) string {
return b.moduleCommit(t, name, fixture, path, toolchains, commit)
}
func (b *bed) moduleCommit(t *testing.T, name, fixture, path string, toolchains map[string]delivery.ToolchainUsed, at string) string {
t.Helper()
raw, err := os.ReadFile(filepath.Join("testdata", fixture))
if err != nil {
t.Fatal(err)
}
digest := delivery.ManifestDigest(raw)
manifests, err := b.js.KeyValue(b.ctx, delivery.ManifestsBucket)
if err != nil {
t.Fatal(err)
}
mv, err := delivery.EncodeManifest(delivery.Manifest{Module: name, JSON: string(raw)})
if err != nil {
t.Fatal(err)
}
if _, err := manifests.Put(b.ctx, digest, mv); err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: at, FP: "src1:" + strings.Repeat("f", 64), FP2: fp,
Manifest: digest, Path: path, Toolchains: toolchains})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, name, data); err != nil {
t.Fatal(err)
}
return digest
}
// archive writes modules.<name> as the real indexer does for a module of an archive only: its fingerprint names no
// toolchain (mesh-delivery-indexer internal/index inputs.go languagesOf gives an archive no language), so Toolchains is empty.
func (b *bed) archive(t *testing.T, name, path, at string) string {
t.Helper()
return b.moduleCommit(t, name, "catalogue-a67f0196-ca-trust.module.json", path, nil, at)
}
// start runs the program as the builder of a node.
func (b *bed) start(t *testing.T, node string, env map[string]string) {
t.Helper()
broker := filepath.Join(t.TempDir(), "broker")
cred, _ := json.Marshal(process.Account{URL: b.srvURL, Node: node, Module: "mesh-delivery-builder", User: node + ".mesh-delivery-builder", Password: "pw"})
if err := os.WriteFile(broker, cred, 0o600); err != nil {
t.Fatal(err)
}
e := map[string]string{"MESH_BROKER_FILE": broker}
for k, v := range env {
e[k] = v
}
runCtx, stop := context.WithCancel(b.ctx)
done := make(chan error, 1)
go func() { done <- run(runCtx, func(k string) string { return e[k] }, io.Discard) }()
var once sync.Once
b.stop = func() { once.Do(func() { stop(); <-done }) }
t.Cleanup(b.stop)
}
// await reads the build key until pred holds of it, within 20 s.
func (b *bed) await(t *testing.T, module string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
t.Helper()
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
var last delivery.BuilderBuild
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
e, err := b.builds.Get(b.ctx, key)
if err != nil {
continue
}
if last, err = delivery.DecodeBuilderBuild(e.Value()); err == nil && pred(last) {
return last
}
}
t.Fatalf("builds.%s = %+v within 20 s", key, last)
return last
}
func isState(state string) func(delivery.BuilderBuild) bool {
return func(v delivery.BuilderBuild) bool { return v.State == state }
}
// TestBuilderMakesADoneKeyWithTheDigests: modules.netcheck (a Go and a TypeScript bundle) and its manifest are in the
// indexer's buckets, and two nodes report each build module's toolchain; the program writes builds.netcheck.<fp> done,
// the digests of both bundles, both build modules' toolchains, after a fetch and a build on one node (ADR 0478 clause 4).
func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
b := startBed(t)
digest := b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
if v.By != "n1.mesh-delivery-builder" || v.Manifest != digest || len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
for _, a := range v.Artifacts {
if a.Digest != digestOf(a.Name) || a.Kind != "bundle" || a.Reference != "artifact-store://netcheck/"+a.Name+"/blobs/"+a.Digest {
t.Errorf("artifact = %+v, want the digest every node built and the reference its push answered (ADR 0498 clause 1)", a)
}
}
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made.
ended := b.logged(t, "netcheck")
if len(ended) != 2 || ended[0].Event != delivery.BuildStartedEvent || ended[1].Event != delivery.BuildEndedEvent ||
ended[1].Manifest != digest || len(ended[1].Made) != 2 || ended[1].Node != "n1" {
t.Errorf("logged = %+v, want the start and the end of netcheck's build", ended)
}
for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_build": 1, "mesh-build-node.build_node_fetch": 1, "mesh-build-node.build_node_build": 1} {
if got := b.fake.count(tool); got != want {
t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want)
}
}
}
// TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey: two builders on two nodes start together on one module key of an
// archive only (no toolchain named, as the indexer writes it); the second takes over the key the first holds building
// (ADR 0478 clause 6), and the key ends done, written once.
func TestTwoBuilderInstancesStartingTogetherEndWithOneDoneKey(t *testing.T) { // 1607/AC7
b := startBed(t)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
b.start(t, "n2", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || v.Artifacts[0].Digest != digestOf("files") {
t.Fatalf("build = %+v, want the archive's digest", v)
}
if n := b.fake.count("mesh-build-go.build_go_build"); n < 1 { // each instance that built, on one node
t.Errorf("build_go_build was called %d times, want one for each instance that built", n)
}
}
// TestAnArchiveOnlyModuleAndARootModuleAreBuilt: the indexer gives an archive no language, so modules.ca-trust has no
// toolchains; the builder builds it on the build module's nodes all the same (review round 4). A module whose path is "."
// is given to the build tools as it is (1615/AC4). One table: the two tests were merged to pay for the enrolment's.
func TestAnArchiveOnlyModuleAndARootModuleAreBuilt(t *testing.T) {
for _, c := range []struct{ name, path string }{{"no toolchain", "modules/ca-trust"}, {"root", delivery.RootPath}} {
t.Run(c.name, func(t *testing.T) {
b := startBed(t)
b.archive(t, "ca-trust", c.path, commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildDone))
if c.name == "no toolchain" && (len(v.Toolchains) != 0 || len(v.Artifacts) != 1 || strings.HasPrefix(v.Step, "one node (")) {
t.Errorf("build = %+v, want done naming no toolchain", v)
}
})
}
}
// TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint: a build step fails, so the key is
// failed naming it; the indexer writing modules.<m> again with the same fingerprint wakes the builder, which builds
// the key again as a new revision of the same key, now done.
func TestAFailedKeyIsBuiltAgainAsANewRevisionFromANewModulesValueWithTheSameFingerprint(t *testing.T) { // 1607/AC6, AC7
b := startBed(t)
b.fake.failBuild.Store(true)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildFailed))
if !strings.Contains(v.Error, " on n") || !strings.Contains(v.Error, "boom") {
t.Errorf("build = %+v, want failed naming the node and the step's reason", v)
}
key, _ := delivery.BuilderBuildKey("ca-trust", fp)
failed, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
b.fake.failBuild.Store(false)
// The indexer writes the module again (here at a later commit), with the same fingerprint.
b.archive(t, "ca-trust", "modules/ca-trust", "89abcdef0123456789abcdef0123456789abcdef")
b.await(t, "ca-trust", isState(delivery.BuildDone))
done, err := b.builds.Get(b.ctx, key)
if err != nil {
t.Fatal(err)
}
if done.Revision() <= failed.Revision() {
t.Errorf("revision %d, was %d: the key was not written as a new revision", done.Revision(), failed.Revision())
}
}
// TestAToolchainChangeDoesNotBuildTheOldFingerprint: the fingerprint names go1.27.1, the nodes then agree on go1.27.2 (a
// report of both nodes), and the builder writes no key for the old fingerprint although it was woken by the report; a
// module value naming go1.27.2 (the new fingerprint, computed by the indexer) is built.
func TestAToolchainChangeDoesNotBuildTheOldFingerprint(t *testing.T) { // 1615/AC4
b := startBed(t)
b.report(t, "mesh-build-go", "n1", "go", "go1.27.2", "1")
b.report(t, "mesh-build-go", "n2", "go", "go1.27.2", "1")
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // names go1.27.1
b.start(t, "n1", nil)
// Another module, whose fingerprint names the new toolchain, is built; by then the old one has been looked at.
b.moduleCommit(t, "netcheck2", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck",
map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.2", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}, commit)
b.await(t, "netcheck2", isState(delivery.BuildDone))
if _, err := b.builds.Get(b.ctx, mustKey(t, "netcheck")); err == nil {
t.Error("the old fingerprint, naming go1.27.1, was built with go1.27.2")
}
if n := b.fake.count("mesh-build-go.build_go_build"); n != 1 {
t.Errorf("build_go_build was called %d times, want 1 (the new fingerprint on one node only)", n)
}
}
func mustKey(t *testing.T, module string) string {
k, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
return k
}
// moduleAt is module with the go toolchain version the fingerprint names and the commit it was computed at.
func (b *bed) moduleAt(t *testing.T, name, fixture, path, goVersion, at string) {
t.Helper()
b.moduleCommit(t, name, fixture, path, map[string]delivery.ToolchainUsed{"go": {Version: goVersion, Recipe: "1"}}, at)
}
// TestAKeyLeftBuildingForAModuleThatIsGoneIsClosedAtStart: the builds bucket holds a key left building by another node for
// a module the indexer no longer has; the program, at start, fails it naming why (issue 1660, ADR 0478 clause 6).
func TestAKeyLeftBuildingForAModuleThatIsGoneIsClosedAtStart(t *testing.T) {
b := startBed(t)
data, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "n2.mesh-delivery-builder", Manifest: strings.Repeat("a", 64)})
if err != nil {
t.Fatal(err)
}
if _, err := b.builds.Put(b.ctx, mustKey(t, "ghost"), data); err != nil {
t.Fatal(err)
}
b.start(t, "n1", nil)
v := b.await(t, "ghost", isState(delivery.BuildFailed))
if !strings.Contains(v.Error, "no longer has this fingerprint") || !strings.Contains(v.Error, "n2.mesh-delivery-builder") {
t.Errorf("build = %+v, want it closed naming who left it", v)
}
}
// TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk: the trunk's record of netcheck carries a fingerprint of its
// own; a candidate the indexer wrote gives netcheck the fingerprint fp at a pull request's head; the program writes
// builds.netcheck.<fp> done, its fetch given the head, the ref that carries it and the trunk head of the candidate's source
// revision, newer than the module value's commit (issue 1684; the operator 2026-10-25: a value names the commit its
// fingerprint first appeared at).
func TestBuilderBuildsACandidatesFingerprintFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1
b := startBed(t)
other := "src2:" + strings.Repeat("e", 64)
digest := b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: commit, FP: "src1:" + strings.Repeat("f", 64), FP2: other,
Manifest: digest, Path: "modules/netcheck", Toolchains: map[string]delivery.ToolchainUsed{"go": {Version: "go1.27.1", Recipe: "1"}, "typescript": {Version: "tsc5", Recipe: "1"}}})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, "netcheck", data); err != nil {
t.Fatal(err)
}
const head = "1111111111111111111111111111111111111111"
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"netcheck"},
Fps: map[string]delivery.CandidateModule{"netcheck": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", Base: b.trunk(t, trunkHead), HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, err := delivery.CandidatesKey("feat/x")
if err != nil {
t.Fatal(err)
}
if _, err := b.candidates.Put(b.ctx, ckey, cand); err != nil {
t.Fatal(err)
}
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
if len(v.Artifacts) != 2 {
t.Fatalf("build = %+v", v)
}
b.fake.mu.Lock()
defer b.fake.mu.Unlock()
found := false
for _, f := range b.fake.fetches {
if f["commit"] == head && f["ref"] == "refs/pull/7/head" && f["base"] == trunkHead {
found = true
}
}
if !found {
t.Errorf("fetches = %v, want one of the candidate's head with its ref and the trunk head it was previewed onto, not the module value's commit", b.fake.fetches)
}
}
// TestAnImageIsBuiltOnOneNodeAndOnlyATrunkKeyPublishes: modules.imaged (one image artifact) is built on n1, where
// mesh-build-image runs: a trunk record's key is done with the digest and the reference, its build call carrying publish; a
// candidate's key for the same module is built first with no publish and no reference; when the pull request is merged and the
// trunk's record names the candidate's fingerprint, the key is built again with publish and ends with a reference
// (ADR 0487 clauses 5 and 12).
func TestAnImageIsBuiltOnOneNodeAndOnlyATrunkKeyPublishes(t *testing.T) { // 1705
b := startBed(t)
other := "src2:" + strings.Repeat("e", 64)
digest := b.moduleCommit(t, "imaged", "catalogue-a67f0196-imaged.module.json", "modules/imaged", nil, commit)
// The trunk carries `other`; its key is built and published.
trunk := func(fp2 string) {
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: commit, FP: "src1:" + strings.Repeat("f", 64), FP2: fp2,
Manifest: digest, Path: "modules/imaged"})
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(b.ctx, "imaged", data); err != nil {
t.Fatal(err)
}
}
trunk(other)
b.start(t, "n1", nil)
okey, _ := delivery.BuilderBuildKey("imaged", other)
awaitKey := func(key string, pred func(delivery.BuilderBuild) bool) delivery.BuilderBuild {
var last delivery.BuilderBuild
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if e, err := b.builds.Get(b.ctx, key); err == nil {
if last, err = delivery.DecodeBuilderBuild(e.Value()); err == nil && pred(last) {
return last
}
}
}
t.Fatalf("builds.%s = %+v within 20 s", key, last)
return last
}
v := awaitKey(okey, isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Digest != digestOf("server") || !strings.HasSuffix(v.Artifacts[0].Reference, "@"+digestOf("server")) {
t.Fatalf("trunk build = %+v, want the digest and the reference", v)
}
// The candidate carries fp: built, not published.
const head = "1111111111111111111111111111111111111111"
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: head}, Modules: []string{"imaged"},
Fps: map[string]delivery.CandidateModule{"imaged": {FP2: fp, Manifest: digest}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
cand, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", Base: b.trunk(t, trunkHead), HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
ckey, _ := delivery.CandidatesKey("feat/x")
if _, err := b.candidates.Put(b.ctx, ckey, cand); err != nil {
t.Fatal(err)
}
v = b.await(t, "imaged", isState(delivery.BuildDone))
if len(v.Artifacts) != 1 || v.Artifacts[0].Reference != "" {
t.Fatalf("candidate build = %+v, want done with no reference", v)
}
b.fake.mu.Lock()
if len(b.fake.builds) != 2 || b.fake.builds[0]["publish"] != true || b.fake.builds[1]["publish"] != nil {
t.Errorf("builds = %v, want the trunk's with publish, then the candidate's with none", b.fake.builds)
}
b.fake.mu.Unlock()
// The pull request is merged: the trunk's record names fp, and its key ends published, built once more with publish.
trunk(fp)
v = b.await(t, "imaged", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone && len(v.Artifacts) == 1 && v.Artifacts[0].Reference != ""
})
if v.Artifacts[0].Digest != digestOf("server") {
t.Errorf("merged build = %+v", v)
}
b.fake.mu.Lock()
defer b.fake.mu.Unlock()
published := 0
for _, a := range b.fake.builds {
if a["publish"] == true {
published++
}
}
if len(b.fake.builds) != 3 || published != 2 {
t.Errorf("builds = %v, want three, two of them with publish (the trunk's two fingerprints)", b.fake.builds)
}
}
// TestEveryDoneKeyNamesItsReferencesAndARefusedPushIsBuiltAgain: with the artifact store refusing pushes, the key of an
// archive module ends failed opening "not stored:"; once it takes them, a new modules value builds the key again, done, its
// every artifact naming a reference (issue 1587, B8, ADR 0498 clauses 1, 2 and 8).
func TestEveryDoneKeyNamesItsReferencesAndARefusedPushIsBuiltAgain(t *testing.T) {
b := startBed(t)
b.fake.refusePush.Store(true)
b.archive(t, "ca-trust", "modules/ca-trust", commit)
b.start(t, "n1", nil)
v := b.await(t, "ca-trust", isState(delivery.BuildFailed))
if !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "507") {
t.Fatalf("build = %+v, want failed opening %q with the store's words", v, delivery.NotStored)
}
b.fake.refusePush.Store(false)
b.archive(t, "ca-trust", "modules/ca-trust", "89abcdef0123456789abcdef0123456789abcdef")
v = b.await(t, "ca-trust", isState(delivery.BuildDone))
if ok, missing := v.Referenced(); !ok {
t.Errorf("build = %+v, artifacts naming no reference: %v", v, missing)
}
}
// TestAPinnedUpstreamIsCopiedByTheImageBuildModule: hello-web's manifest (captured from mesh-catalog 49caf17f) names an
// upstream image pinned by its digest and an archive; the upstream is copied on n1, where mesh-build-image runs, with no
// fetch of the image module, and the key ends done with both references (issue 1587, B8, ADR 0498 clause 4).
func TestAPinnedUpstreamIsCopiedByTheImageBuildModule(t *testing.T) {
b := startBed(t)
b.moduleCommit(t, "hello-web", "catalogue-49caf17f-hello-web.module.json", "modules/hello-web", nil, commit)
b.start(t, "n1", nil)
v := b.await(t, "hello-web", isState(delivery.BuildDone))
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, artifacts naming no reference: %v", v, missing)
}
if n := b.fake.count("mesh-build-image.build_image_copy"); n != 1 {
t.Errorf("build_image_copy called %d times, want once", n)
}
if n := b.fake.count("mesh-build-image.build_image_fetch"); n != 0 {
t.Errorf("build_image_fetch called %d times, want none for a copy", n)
}
}
// TestAManifestFallingBackToAPredefinedKeyIsBuilt: claude-code's manifest (mesh-catalog 835bf22b) has a config value falling
// back to a predefined key; the builder, which holds no published list of them, reads it and builds the key done (issue
// 1751, mesh-sdk catalogue.ParseManifestForInputs).
func TestAManifestFallingBackToAPredefinedKeyIsBuilt(t *testing.T) {
b := startBed(t)
b.module(t, "claude-code", "catalogue-835bf22b-claude-code.module.json", "modules/claude-code")
b.start(t, "n1", nil)
v := b.await(t, "claude-code", func(v delivery.BuilderBuild) bool {
return v.State == delivery.BuildDone || v.State == delivery.BuildFailed
})
if v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done: the manifest is read", v)
}
}
// 1886/AC5: a builder on n1 whose Go, Node and image build modules each answered a reference says under n1, in its bucket
// of instances, that it writes references, and takes the say away when it stops: the controller delivers from its keys
// only while it holds.
func TestABuilderWhoseBuildModulesAnsweredReferencesSaysItAndStoppedSaysNothing(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck") // a Go and a TypeScript bundle
b.moduleCommit(t, "imaged", "catalogue-a67f0196-imaged.module.json", "modules/imaged", nil, commit) // an image, for the trunk
b.start(t, "n1", nil)
var said string
for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && said == ""; time.Sleep(50 * time.Millisecond) {
if e, err := b.instances.Get(b.ctx, "n1"); err == nil {
said = string(e.Value())
}
}
if !strings.Contains(said, `"writes":["references"]`) {
t.Fatalf("the builder said %q", said)
}
b.stop()
if _, err := b.instances.Get(b.ctx, "n1"); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Fatalf("the stopped builder still says: %v", err)
}
}
// 1887/AC1, AC2 on the real bus: the builder puts the trunk fingerprint's build as asks on the build seats' delivery
// streams and calls no node's tool; the build modules that pulled them wrote the steps, and the key is done naming the
// references.
func TestTheBuilderAsksTheBuildSeatsAndCallsNoNode(t *testing.T) {
b := startBed(t)
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
var calls atomic.Int64
s, err := sub.Subscribe("mesh.mod.*.call.>", func(*nats.Msg) { calls.Add(1) })
if err != nil {
t.Fatal(err)
}
defer s.Unsubscribe()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", isState(delivery.BuildDone))
for _, a := range v.Artifacts {
if a.Reference == "" {
t.Errorf("artifact %s names no reference", a.Name)
}
}
if n := calls.Load(); n != 0 {
t.Errorf("%d tool calls reached a build module", n)
}
// 1887/AC1: one ask per build module the build needs (netcheck: a Go and a TypeScript bundle), each on its seat's
// delivery stream.
got := asked()
if len(got) != 2 {
t.Fatalf("asks put = %+v, want one Go and one Node delivery ask", got)
}
for _, a := range got {
if a.Kind != delivery.AskDelivery || a.Build == nil || a.Build.FP != fp {
t.Errorf("ask %+v, want a delivery ask of the fingerprint", a)
}
}
}
// asksPut hears every ask put on a build seat's accept subjects (a core subscription beside the stream) and answers the
// asks heard so far.
func (b *bed) asksPut(t *testing.T, nc *nats.Conn) func() []delivery.StepAsk {
t.Helper()
var mu sync.Mutex
var asks []delivery.StepAsk
s, err := nc.Subscribe("mesh.seat.*.accept.>", func(m *nats.Msg) {
if a, err := delivery.DecodeStepAsk(m.Data); err == nil {
mu.Lock()
asks = append(asks, a)
mu.Unlock()
}
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Unsubscribe() })
return func() []delivery.StepAsk {
mu.Lock()
defer mu.Unlock()
return append([]delivery.StepAsk(nil), asks...)
}
}
// 1887/AC3: a builder that started again while its asks waited in the streams (no node had a free slot) puts no second
// ask: it waits for the asks its building key names, and the key is done once a node took them.
func TestARestartedBuilderPutsNoSecondAsk(t *testing.T) {
b := startBedWith(t, true) // 1928/AC1: no node can take an ask from the start, so no pull in hand takes the first one
b.module(t, "netcheck", "catalogue-a67f0196-netcheck.module.json", "modules/netcheck")
sub, err := nats.Connect(b.srvURL, nats.UserInfo("ctl", "pw"))
if err != nil {
t.Fatal(err)
}
defer sub.Close()
asked := b.asksPut(t, sub)
b.start(t, "n1", nil)
v := b.await(t, "netcheck", func(v delivery.BuilderBuild) bool { return v.State == delivery.BuildBuilding && len(v.Asks) == 2 })
for deadline := time.Now().Add(20 * time.Second); len(asked()) < 2; time.Sleep(20 * time.Millisecond) {
if time.Now().After(deadline) {
t.Fatalf("asks put = %+v within 20 s, want the two the key names", asked())
}
}
// 1945 note 62, on the real bus: the stream read as builder_seat_queue reads it shows the Go ask waiting; the ask can
// still be answered (nothing was taken since); and a message dropped as an ask asked again is gone from the stream.
held := heldOnTheBus{b.js}
goAsk := delivery.StepAsk{ID: v.Asks[delivery.AskGo], Language: delivery.AskGo, Kind: delivery.AskDelivery}
stale := delivery.StepAsk{ID: "mvo53jcm70b5f7", Language: delivery.AskGo, Kind: delivery.AskDelivery}
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskDelivery)
if _, err := b.js.Publish(b.ctx, subject, []byte("{}"), jetstream.WithMsgID(stale.ID)); err != nil {
t.Fatal(err)
}
queued := func() []build.QueuedState {
q, err := held.Queue(b.ctx, delivery.AskGo, 10)
if err != nil || len(q.Kinds) != 1 || q.Kinds[0].Kind != delivery.AskDelivery {
t.Fatalf("the Go seat's queue = %+v, %v; want its delivery asks", q, err)
}
return q.Kinds[0].Messages
}
if m := queued(); len(m) != 2 || m[0].Ask != goAsk.ID || !m[0].Waits || m[1].Ask != stale.ID {
t.Fatalf("the Go delivery queue = %+v, want the key's ask waiting, then the stale one", m)
}
if why, err := held.Gone(b.ctx, goAsk); why != "" || err != nil {
t.Errorf("Gone(the waiting ask) = %q, %v; want it answerable", why, err)
}
if err := held.Drop(b.ctx, stale); err != nil {
t.Fatal(err)
}
if m := queued(); len(m) != 1 || m[0].Ask != goAsk.ID {
t.Fatalf("after the drop the Go delivery queue = %+v, want the key's ask alone", m)
}
b.stop()
b.start(t, "n1", nil) // the same builder, started again, while both asks wait
b.fake.open()
done := b.await(t, "netcheck", isState(delivery.BuildDone))
if got := asked(); len(got) != 2 {
t.Fatalf("asks put = %d (%+v), want the two the key named and none again", len(got), got)
}
for lang, id := range v.Asks {
found := false
for _, a := range asked() {
found = found || a.ID == id
}
if !found {
t.Errorf("the %s ask %s the key named was never put", lang, id)
}
}
if len(done.Artifacts) != 2 {
t.Errorf("done = %+v", done)
}
}
@@ -0,0 +1,317 @@
// Command mesh-delivery-builder is delivery's builder (novox/hq ADR 0453 clause 4, ADR 0478, issue 1607): it builds every
// new-form fingerprint the indexer writes, dark, into its own build keys (builds.<module>.<fingerprint>), through the build
// modules' fetch and build tools. Nothing reads its keys to deliver until release R4e. It serves no request subject and
// holds no timer: every reaction is a watch or a bus event (ADR 0437 §8).
package main
import (
"context"
"fmt"
"io"
"log"
"maps"
"os"
"os/signal"
"slices"
"strings"
"sync"
"syscall"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-builder/internal/build"
)
const (
// defaultStepBound is how long one ask's step may run once a build node took it (a fetch downloads the Go modules, a
// build compiles): from its start, never from its put, so an ask waiting in its stream for a free slot is not failed
// (buildrun.Asker.Bound; ADR 0525 §3, §6).
defaultStepBound = 30 * time.Minute
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, os.Getenv, os.Stderr); err != nil {
fmt.Fprintln(os.Stderr, "mesh-delivery-builder:", err)
os.Exit(1)
}
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-builder: ", log.LstdFlags)
cred, err := process.ReadBrokerAccount(getenv)
if err != nil {
return err
}
conn, js, err := cred.Dial()
if err != nil {
return err
}
defer conn.Close()
builds, err := process.EssentialBucket(ctx, js, "builder", delivery.BuilderBuildsBucket, "claim or write a build key")
if err != nil {
return err
}
modules, err := process.EssentialBucket(ctx, js, "builder", delivery.ModulesBucket, "know any fingerprint to build")
if err != nil {
return err
}
manifests, err := process.EssentialBucket(ctx, js, "builder", delivery.ManifestsBucket, "read what a module builds")
if err != nil {
return err
}
// The candidates' fingerprints are built too (issue 1684): a builder that cannot read them builds the trunk's.
candidates := process.OptionalBucket(ctx, js, delivery.CandidatesBucket, "no candidate's fingerprint is built until the builder starts again; the trunk's are", logger.Printf)
bound := defaultStepBound
if v := strings.TrimSpace(getenv("MESH_BUILDER_STEP_BOUND")); v != "" {
if bound, err = time.ParseDuration(v); err != nil {
return fmt.Errorf("MESH_BUILDER_STEP_BOUND: %w", err)
}
}
tools := &buildrun.ToolchainSet{}
// Each build module's part of a build is an ask on its build seat's stream, pulled by a build node with a free slot
// (novox/hq ADR 0525): the builder chooses no node, so it reads neither presence nor have and holds no capacity.
asks := buildrun.JetStreamAsks{JS: js}
// The asks the build seats gave up on (issue 734): heard from DEAD_LETTERS, each ends the wait for its step at once,
// and the lane asks its part again (Builder.waitOnce) instead of waiting for the check bound.
// Heard in the background, and again at a reconnect or an ask while it is not; whether it is heard is kept in the
// state hearing, which builder_stuck says (dead_letters).
dead := &buildrun.DeadLetters{}
// The state may not be on the bus yet when a delivery started the builder (the grants part creates it, issue 1790): it
// is bound when the bus says it was created, and the latest hearing is written then.
hearingKV := process.AwaitBucket(ctx, js, conn, delivery.BuilderModule+"_"+buildrun.HearingState, "builder_stuck cannot say whether DEAD_LETTERS is heard", logger.Printf)
hearing := buildrun.NewDeadLetterHearing(ctx, buildrun.JetStreamDeadLetters{JS: js}, delivery.AskLanguages, dead, logger.Printf,
buildrun.SayHearing(ctx, hearingKV, logger.Printf))
defer hearing.Stop()
// Each ask's step is heard by its own watch and, at every step end of its build module and every reconnect, read
// (buildrun.HearEnds): a watch that goes deaf loses no end (the lost check of 2026-10-28).
var buildModules []string
for _, l := range delivery.AskLanguages {
if m, err := delivery.BuildModuleOf(l); err == nil {
buildModules = append(buildModules, m)
}
}
heard := buildrun.HearEnds(ctx, asks, asks, buildModules, logger.Printf)
conn.SetReconnectHandler(func(*nats.Conn) { hearing.Again(); heard.Nudge("") })
// The builder asks the controller, as its own account, whether the artifact store still holds a done key's blobs
// (the seat verb held, novox/hq issue 1937) before it leaves the key done.
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
if err != nil {
return err
}
defer calls.Close()
b := &build.Builder{
Claimer: build.Claimer{Store: kvStore{builds}, By: cred.Node + "." + cred.Module},
Asks: buildrun.Asker{Publisher: asks, Waiter: buildrun.LostAskWaiter{Inner: heard, Dead: dead, Hearing: hearing}, Bound: bound},
Held: heldOnTheBus{js},
// An ask that can no longer be answered is asked again (issue 1945), read at the step bound: a context's bound.
Gone: heldOnTheBus{js},
CheckAfter: bound,
Toolchains: tools,
Manifests: manifestReader{manifests},
Logf: logger.Printf,
Stored: storedVia(calls.Call),
Node: cred.Node,
}
// A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base
// revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision
// is not what a read grants): a revision heard after the candidate that names it builds that candidate then. Where it
// cannot be read the trunk's keys are built and the candidates wait (issue 1945).
errs := make(chan error, 6)
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf}
trunks.start()
b.Trunk = trunks.head
// The builder's log (ADR 0571 §1): each build's start and end in its module log `build-log`, read by the view, the
// indexer and the assigner through their reads, and by the operator's command nox builds through the tool builder_builds_since.
b.Log = func(e delivery.BuildLogEntry) {
if err := appendLog(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
}
b.Said = func(e build.AskedAgain) {
if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
}
// **Said once every build module answered a reference, taken away when stopped** (issue 1886, ADR 0522): the
// controller delivers from this builder's keys only while an instance says it writes references. A bucket not there
// yet (the manifest's state not made) is said, and the controller's plan waits for the keys, named (no build-agent
// is asked since R4g, ADR 0524 §7).
switch kv := process.OptionalBucket(ctx, js, instancesBucket, "the controller is not told that this builder writes references until it starts again", logger.Printf); {
case kv == nil:
default:
says := newSayer(ctx, kv, cred.Node, []string{buildrun.GoBuildModule, buildrun.NodeBuildModule, build.ImageBuildModule}, logger.Printf)
b.Answered = says.Answered
defer func() {
stopping, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if err := says.unsay(stopping); err != nil {
logger.Printf("%v", err)
}
}()
}
// The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, b.NodesChanged, logger.Printf)
if err != nil {
return err
}
if err := watch.start(); err != nil {
return err
}
// A build module says each time it wrote its toolchains key: its stream, and the builder's durable consumer on it, are
// made by the controller. Where the stream is not there yet the builder says so and is started again with the build
// module.
for _, module := range []string{buildrun.GoBuildModule, buildrun.NodeBuildModule} {
stream := toolchainStreams[module]
cons, err := cred.EventConsumer(ctx, js, stream)
if err != nil {
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s's reports are not heard until the builder starts again: %v", cred.Node, cred.Module, stream, module, err)
continue
}
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { watch.Reported(m) }, logger.Printf)
if err != nil {
return err
}
defer cc.Stop()
}
// The seat mesh-artifact-sweep says artifacts-gone when it found a key's blob gone (issue 1937, ADR 0571 §2): the
// builder's durable consumer on EVENTS, made by the grants part from the manifest's consumes. Where it is not there yet the builder says
// so; it still asks held at its start and at every module value.
switch cons, err := cred.EventConsumer(ctx, js, eventsStream); {
case err != nil:
logger.Printf("the consumer %s_%s on %s does not exist yet, so %s is not heard until the builder starts again: %v",
cred.Node, cred.Module, eventsStream, delivery.ArtifactsGoneEvent, err)
default:
cc, err := process.HearEvents(cons, func(m jetstream.Msg) { heardGone(ctx, b, m, logger.Printf) }, logger.Printf)
if err != nil {
return err
}
defer cc.Stop()
}
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
b.Module(ctx, e.Key(), e.Value())
} else {
b.Forget(e.Key())
}
}
w, err := settle(ctx, modules, each)
if err != nil {
return err
}
// A key left building for a fingerprint its module no longer has is closed (issue 1660); the modules are all known now.
if err := b.Sweep(ctx); err != nil {
return err
}
// The fingerprints the indexer gives the candidates' modules are built too (issue 1684), once the trunk's modules are known.
eachCandidate := func(e jetstream.KeyValueEntry) {
if opOf(e) {
b.Candidate(ctx, e.Key(), e.Value())
} else {
b.ForgetCandidate(e.Key())
}
}
if candidates != nil {
cw, err := settle(ctx, candidates, eachCandidate)
if err != nil {
process.NotWatched(logger.Printf, delivery.CandidatesBucket, err, "no candidate's fingerprint is built until the builder starts again")
} else {
go func() { errs <- follow(ctx, cw, eachCandidate) }()
}
}
// A person asks a key's build again with builder_ask_again, which writes the reasks state (issue 1945): watched, as
// every other input. The builder asks again by itself without it.
// A state not on the bus yet at this start is watched once it appears (issue 1790).
process.AwaitBucket(ctx, js, conn, build.ReAsksBucket, "builder_ask_again is not heard; an ask that can no longer be answered is still asked again by itself", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachReAsk := func(e jetstream.KeyValueEntry) { reAsked(ctx, b, kv, e, logger.Printf) }
if rw, err := settle(ctx, kv, eachReAsk); err != nil {
process.NotWatched(logger.Printf, build.ReAsksBucket, err, "builder_ask_again is not heard until the builder starts again")
} else {
go func() { errs <- follow(ctx, rw, eachReAsk) }()
}
})
// A person reads a build seat's stream with builder_seat_queue, which writes the queues state (issue 1945 note 62): the
// builder alone may read the stream, so it answers there.
process.AwaitBucket(ctx, js, conn, build.QueuesBucket, "builder_seat_queue is not answered", logger.Printf).Ready(func(kv jetstream.KeyValue) {
eachRead := func(e jetstream.KeyValueEntry) { queueRead(ctx, heldOnTheBus{js}, kv, e, logger.Printf) }
if qw, err := settle(ctx, kv, eachRead); err != nil {
process.NotWatched(logger.Printf, build.QueuesBucket, err, "builder_seat_queue is not answered until the builder starts again")
} else {
go func() { errs <- follow(ctx, qw, eachRead) }()
}
})
logger.Printf("started on %s", cred.Node)
go func() { errs <- follow(ctx, w, each) }()
select {
case <-ctx.Done():
b.Wait()
return nil
case err := <-errs:
return err
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
}
}
// sourceHeads are the trunk heads of each revision of the gate's source.main the watch delivered, the newest
// delivery.SourceHistory of them: what a candidate's base revision names.
type sourceHeads struct {
mu sync.Mutex
byRev map[uint64]map[string]string
}
// put takes one revision in and answers whether it is new.
func (s *sourceHeads) put(e jetstream.KeyValueEntry) bool {
if !opOf(e) {
return false
}
src, err := delivery.DecodeSource(e.Value())
if err != nil {
return false
}
s.mu.Lock()
defer s.mu.Unlock()
if s.byRev == nil {
s.byRev = map[uint64]map[string]string{}
}
if _, had := s.byRev[e.Revision()]; had {
return false
}
s.byRev[e.Revision()] = src.Heads
for len(s.byRev) > delivery.SourceHistory {
delete(s.byRev, slices.Min(slices.Collect(maps.Keys(s.byRev))))
}
return true
}
// head is the head of repository in revision rev; an error when the revision is not heard (yet, or any more).
func (s *sourceHeads) head(_ context.Context, rev uint64, repository string) (string, error) {
s.mu.Lock()
defer s.mu.Unlock()
heads, ok := s.byRev[rev]
if !ok {
return "", fmt.Errorf("source revision %d is not heard (it is built once it is)", rev)
}
for r, head := range heads {
if delivery.SameRepository(r, repository) {
return head, nil
}
}
return "", nil
}
@@ -0,0 +1,78 @@
package main
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A part does not stop for a bucket it can work without** (mesh-sdk process.EssentialBucket, OptionalBucket) (novox/hq issue 1945, met twice on 2026-10-25: the builder of
// mesh-delivery #48 exited on a refused read of the gate's source, and its reasks bucket would have done the same). A grant
// reaches the bus with a delivery, so a bucket the bus refuses, or times out on, at start is often only not granted yet:
// exiting for it stops every build until a person's hand. A bucket the builder cannot build at all without (its builds,
// the indexer's modules and manifests) still stops the start, naming the grant to look at; every other is said and left,
// and opened again at the next event that needs it where there is one.
// trunkSource is the gate's source history the candidates' previews are merged onto (sourceHeads), opened at start and,
// while it cannot be, again at the next candidate that asks a trunk head: a builder without it builds the trunk's keys,
// and a candidate's preview waits, saying why.
type trunkSource struct {
ctx context.Context
js bucketOpener
heads *sourceHeads
moved func(context.Context)
errs chan<- error
logf func(string, ...any)
mu sync.Mutex
opened bool
}
// ensure opens and settles the source's watch once, and follows it; an error leaves it closed for the next ask.
func (t *trunkSource) ensure() error {
t.mu.Lock()
defer t.mu.Unlock()
if t.opened {
return nil
}
kv, err := t.js.KeyValue(t.ctx, delivery.SourceBucket)
if err != nil {
return fmt.Errorf("the bucket %s cannot be read (%v)", delivery.SourceBucket, err)
}
w, err := kv.Watch(t.ctx, delivery.SourceMainKey, jetstream.IncludeHistory())
if err != nil {
return fmt.Errorf("the bucket %s cannot be watched (%v)", delivery.SourceBucket, err)
}
sw, err := settled(t.ctx, w, func(e jetstream.KeyValueEntry) { t.heads.put(e) })
if err != nil {
return fmt.Errorf("the bucket %s: %w", delivery.SourceBucket, err)
}
t.opened = true
go func() {
t.errs <- follow(t.ctx, sw, func(e jetstream.KeyValueEntry) {
if t.heads.put(e) && t.moved != nil {
t.moved(t.ctx)
}
})
}()
return nil
}
// start opens the source at start; one that cannot be is said, and the builder goes on.
func (t *trunkSource) start() {
if err := t.ensure(); err != nil {
t.logf("%v: the trunk's keys are built, and a candidate's preview waits until the source can be read, tried again at its next candidate", err)
}
}
// head is Builder.Trunk: the head of repository at source revision rev, opening the source first if it is not yet.
func (t *trunkSource) head(ctx context.Context, rev uint64, repository string) (string, error) {
if err := t.ensure(); err != nil {
return "", err
}
return t.heads.head(ctx, rev, repository)
}
@@ -0,0 +1,49 @@
package main
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-builder/internal/build"
)
// queueReader is the part of the builder a queues value reads through: heldOnTheBus.
type queueReader interface {
Queue(ctx context.Context, language string, limit int) (build.SeatQueue, error)
}
// queueRead answers one value of the queues state (the tool builder_seat_queue, issue 1945 note 62): a value not yet
// answered is read now, from the stream the builder's account may read, and written back by compare-and-set at its
// revision with what was read or why it could not be. A value answered already (this builder's own write, or another
// instance's) or deleted is left.
func queueRead(ctx context.Context, r queueReader, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
a, err := build.DecodeQueueAsk(e.Value())
if err != nil {
logf("%s.%s holds what is not a queue read, left: %v", build.QueuesState, e.Key(), err)
return
}
if !a.Answered.IsZero() {
return
}
read, cancel := context.WithTimeout(ctx, 20*time.Second)
q, err := r.Queue(read, e.Key(), max(1, min(a.Limit, build.MaxQueueLimit)))
cancel()
if err != nil {
a.Error = err.Error()
}
a.Queue, a.Answered = &q, time.Now().UTC()
data, err := json.Marshal(a)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the read could not be written: %v", build.QueuesState, e.Key(), err)
}
}
@@ -0,0 +1,165 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"mesh-delivery-builder/internal/build"
)
// askAgainer is the part of the builder a reasks value asks through.
type askAgainer interface {
AskAgain(key, why string) []string
BuildAgain(ctx context.Context, module, fp, why, by string, bound time.Duration) build.BuildAgainResult
AnnounceAgain(ctx context.Context, module, fp, why, by string) build.BuildAgainResult
}
// buildAgainBound is how long the builder waits for the claim and the asks of a key built again by hand; the tool waits
// longer for the builder's answer.
const buildAgainBound = 15 * time.Second
// reAsked answers one value of the reasks state (the tool builder_ask_again, issue 1945): a value not yet answered asks the
// key's waiting lanes again and is written back, by compare-and-set at its revision, with what the builder did. A value
// answered already (this builder's own write) or deleted is left.
func reAsked(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, logf func(string, ...any)) {
if e.Operation() != jetstream.KeyValuePut {
return
}
r, err := build.DecodeReAsk(e.Value())
if err != nil {
logf("%s.%s holds what is not an ask to ask again, left: %v", build.ReAsksState, e.Key(), err)
return
}
if r.Answer != "" {
return
}
if r.Mode == build.ModeBuild {
// The answer comes when the claim and the asks are written, so it is made beside the watch, not in it.
go builtAgain(ctx, b, kv, e, r, logf)
return
}
if r.Mode == build.ModeAnnounce {
announcedAgain(ctx, b, kv, e, r, logf)
return
}
asked := b.AskAgain(e.Key(), r.Why)
if len(asked) == 0 {
r.Answer = "no ask of this key waits in this builder now (it is not building, or its build has not reached its ask yet): nothing was asked; builder_stuck shows where it stands"
} else {
r.Answer = "asked again: the " + strings.Join(asked, ", ") + " ask, under a new id the key names"
}
r.Answered = time.Now().UTC()
logf("%s: asked again by %s (%s): %s", e.Key(), r.By, r.Why, r.Answer)
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
// emitAskedAgain says the builder's event build.asked-again (issue 1945): mesh.mod.<module>.event.build.asked-again,
// with the SDK's envelope headers and the new ask's id as the bus's dedup id.
func emitAskedAgain(ctx context.Context, js jetstream.JetStream, cred process.Account, e build.AskedAgain) error {
data, err := json.Marshal(e)
if err != nil {
return err
}
msg := nats.NewMsg(cred.EventSubject(build.AskedAgainEvent))
id := "asked-again-" + e.New
msg.Header.Set("x-event-id", id)
msg.Header.Set("x-source", cred.Module)
msg.Header.Set("x-node", cred.Node)
msg.Header.Set("x-time", time.Now().UTC().Format(time.RFC3339Nano))
msg.Header.Set("content-type", "application/json")
msg.Data = data
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
return fmt.Errorf("saying %s for %s: %w", build.AskedAgainEvent, e.Key, err)
}
return nil
}
// builtAgain answers a reasks value of mode build (the tool builder_build_again, issue 1126 note 72): the failed key is
// built again as a new revision, and the value is written back with the revision and the asks put, or why nothing was done.
func builtAgain(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, r build.ReAsk, logf func(string, ...any)) {
res := b.BuildAgain(ctx, r.Module, r.FP, r.Why, r.By, buildAgainBound)
switch {
case res.Refused != "":
r.Answer, r.Refused = res.Refused, true
case res.Failed != "":
r.Answer, r.Revision, r.Failed = fmt.Sprintf("the key is written building as revision %d, and the build failed before any ask was put: %s", res.Revision, res.Failed), res.Revision, res.Failed
default:
r.Pending = res.Revision == 0
r.Answer, r.Revision, r.Asks = "built again: the key is written building as revision "+fmt.Sprint(res.Revision)+" and its asks are put", res.Revision, res.Asks
if res.Pending != "" {
r.Answer = res.Pending
if res.Revision != 0 {
r.Answer = "built again: the key is written building as revision " + fmt.Sprint(res.Revision) + "; " + res.Pending
}
}
}
r.Answered = time.Now().UTC()
logf("%s: build again asked by %s (%s): %s", e.Key(), r.By, r.Why, r.Answer)
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
// announcedAgain answers a reasks value of mode announce (the tool builder_announce_again): the done key is written again
// unchanged, and the value is written back with the new revision, or why nothing was done.
func announcedAgain(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e jetstream.KeyValueEntry, r build.ReAsk, logf func(string, ...any)) {
res := b.AnnounceAgain(ctx, r.Module, r.FP, r.Why, r.By)
if res.Refused != "" {
r.Answer, r.Refused = res.Refused, true
} else {
r.Revision, r.Answer = res.Revision, fmt.Sprintf("announced again: the done key is written again, unchanged, as revision %d", res.Revision)
}
r.Answered = time.Now().UTC()
data, err := json.Marshal(r)
if err != nil {
return
}
if _, err := kv.Update(ctx, e.Key(), data, e.Revision()); err != nil && !errors.Is(err, jetstream.ErrKeyExists) && !errors.Is(err, jetstream.ErrKeyRevisionMismatch) {
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
// appendLog appends one entry to the builder's module log `build-log` (ADR 0297, ADR 0571 §1) under the module's key
// (delivery.BuildLogKey), on the log's own subject: a module's program appends to the logs it keeps on its own credential.
// The id makes a repeated append of one entry one entry.
func appendLog(ctx context.Context, js jetstream.JetStream, cred process.Account, e delivery.BuildLogEntry) error {
data, err := delivery.EncodeBuildLogEntry(e)
if err != nil {
return err
}
key, err := delivery.BuildLogKey(e.Module)
if err != nil {
return err
}
msg := nats.NewMsg(buscompose.LogSubject(cred.Module, delivery.BuilderLog) + "." + key)
msg.Data = data
id := e.Event + "." + e.Module + "." + e.FP + "." + e.At.Format(time.RFC3339Nano)
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
return fmt.Errorf("logging %s of %s at %s: %w", e.Event, e.Module, e.FP, err)
}
return nil
}
@@ -0,0 +1,73 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -0,0 +1,79 @@
package main
import (
"context"
"fmt"
"sync"
"github.com/nats-io/nats.go/jetstream"
)
// What the builder says it writes (novox/hq issue 1886, ADR 0522): each running instance keeps a key, named by its node,
// in its own bucket of instances (`mesh-delivery-builder_instances`, the manifest's state `instances`), saying that every
// done key it writes names where its artifacts are (ADR 0498 §1). The references come from the build modules' own pushes,
// so an instance says it only once a step of every build module it calls answered one since it started: a builder beside
// a build module from before R4e says nothing. The controller delivers from the builder's keys only while an instance
// says so; since R4g it asks no build-agent, and a module whose key is not yet done waits, named (ADR 0524 §7). The bucket is read by the controller alone, so its name and shape
// are not in mesh-sdk.
const (
instancesBucket = "mesh-delivery-builder_instances"
writesReferences = `{"v":1,"writes":["references"]}`
)
// sayStore is the bucket of instances as the builder writes it.
type sayStore interface {
Put(ctx context.Context, key string, value []byte) (uint64, error)
Delete(ctx context.Context, key string, opts ...jetstream.KVDeleteOpt) error
}
// sayer puts this instance's say once every build module in need answered a reference, and takes it away at the stop.
type sayer struct {
ctx context.Context
store sayStore
node string
need []string
logf func(string, ...any)
mu sync.Mutex
answered map[string]bool
said bool
}
func newSayer(ctx context.Context, store sayStore, node string, need []string, logf func(string, ...any)) *sayer {
return &sayer{ctx: ctx, store: store, node: node, need: need, logf: logf, answered: map[string]bool{}}
}
// Answered is told the build module whose step answered a reference (build.Builder.Answered).
func (s *sayer) Answered(module string) {
s.mu.Lock()
defer s.mu.Unlock()
s.answered[module] = true
if s.said {
return
}
for _, m := range s.need {
if !s.answered[m] {
return
}
}
if _, err := s.store.Put(s.ctx, s.node, []byte(writesReferences)); err != nil {
s.logf("saying in %s that this builder writes references: %v; said again at the next reference", instancesBucket, err)
return
}
s.said = true
s.logf("every build module answered a reference: said in %s that this builder writes references", instancesBucket)
}
// unsay takes the say away, if it was made: a builder replaced by one that does not write references leaves none.
func (s *sayer) unsay(ctx context.Context) error {
s.mu.Lock()
defer s.mu.Unlock()
if !s.said {
return nil
}
if err := s.store.Delete(ctx, s.node); err != nil {
return fmt.Errorf("taking away the say in %s: %w", instancesBucket, err)
}
s.said = false
return nil
}
@@ -0,0 +1,46 @@
package main
import (
"context"
"encoding/json"
"testing"
"github.com/nats-io/nats.go/jetstream"
)
type saysOf map[string]string
func (s saysOf) Put(_ context.Context, key string, value []byte) (uint64, error) {
s[key] = string(value)
return 1, nil
}
func (s saysOf) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(s, key)
return nil
}
// 1886/AC5: a builder says, under its node, that it writes references only once every build module it calls answered
// one; stopped, it says nothing any more.
func TestTheBuilderSaysItWritesReferencesOnceEveryBuildModuleAnsweredOne(t *testing.T) {
s := saysOf{}
say := newSayer(t.Context(), s, "shanks", []string{"mesh-build-go", "mesh-build-node", "mesh-build-image"}, t.Logf)
say.Answered("mesh-build-go")
say.Answered("mesh-build-node")
if v, ok := s["shanks"]; ok {
t.Fatalf("said before the image build module answered a reference: %q", v)
}
say.Answered("mesh-build-image")
var v struct {
Writes []string `json:"writes"`
}
if err := json.Unmarshal([]byte(s["shanks"]), &v); err != nil || len(v.Writes) != 1 || v.Writes[0] != "references" {
t.Fatalf("the builder's say: %q %v", s["shanks"], err)
}
if err := say.unsay(t.Context()); err != nil {
t.Fatal(err)
}
if _, ok := s["shanks"]; ok {
t.Fatalf("the stopped builder still says: %v", s)
}
}
@@ -0,0 +1,66 @@
package main
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-builder/internal/build"
)
// eventsStream is the stream every module's and role's events are on, where the grants part makes the builder's durable
// consumer of artifacts-gone (mesh-sdk buscompose EventsStream).
const eventsStream = "EVENTS"
// SeatHeld is the seat mesh-artifact-sweep's verb held (ADR 0571 §2).
const SeatHeld = "seat:" + catalogue.ArtifactSweepSeat + ".held"
// toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call).
type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the seat mesh-artifact-sweep's `held` (novox/hq ADR 0571 §2, issue 1937)
// and answers the references the store no longer holds. One it could not ask about, with none gone, is an error: the key
// is then left done, and asked again at the next module value or artifacts-gone.
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) {
body := map[string]any{"references": references}
raw, err := call(SeatHeld, body)
if err != nil {
return nil, fmt.Errorf("asking held: %w", err)
}
a, err := delivery.DecodeHeldAnswer(raw, references)
if err != nil {
return nil, err
}
if len(a.Gone) == 0 && len(a.Unknown) > 0 {
var why []string
for r, w := range a.Unknown {
why = append(why, r+": "+w)
}
sort.Strings(why)
return nil, fmt.Errorf("the artifact store could not be asked about %s", strings.Join(why, "; "))
}
return a.Gone, nil
}
}
// heardGone takes one artifacts-gone event in: acknowledged, then handed to the builder, which asks the store and builds
// the key again. A body that is not one is said and acknowledged, so it is not handed over again.
func heardGone(ctx context.Context, b *build.Builder, m jetstream.Msg, logf func(string, ...any)) {
e, err := delivery.DecodeArtifactsGone(m.Data())
if ackErr := m.Ack(); ackErr != nil {
logf("acknowledging %s: %v", m.Subject(), ackErr)
}
if err != nil {
logf("%s holds what is not %s, left: %v", m.Subject(), delivery.ArtifactsGoneEvent, err)
return
}
b.ArtifactsGone(ctx, e)
}
@@ -0,0 +1,82 @@
{
"module": "hello-web",
"slug": "hello",
"version": "1",
"capabilities": [
"container-runtime"
],
"requires": [
"route"
],
"contributes": {
"route": {
"label": "hello",
"endpoint": "web"
}
},
"binds": {
"route": "${dir:state}/route.json"
},
"listens": [
{
"name": "web",
"port": 8080,
"protocol": "tcp",
"from": "mesh",
"why": "the demo web page; only the route-proxy reaches it, and the public name is a route grant"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "page",
"type": "file",
"path": "${dir:state}/index.html",
"mode": "0644",
"template": "files/index.html"
},
{
"id": "net",
"type": "network",
"name": "hello-web"
},
{
"id": "server",
"type": "container",
"name": "hello-web",
"network": "hello-web",
"ports": [
"8080"
],
"volumes": [
"${dir:state}/index.html:/www/index.html:ro"
],
"args": [
"sh",
"-c",
"while true; do { printf 'HTTP/1.1 200 OK\\r\\nContent-Type: text/plain\\r\\nConnection: close\\r\\n\\r\\n'; cat /www/index.html; } | nc -l -p 8080; done"
],
"artifact": "server"
}
],
"build": {
"artifacts": [
{
"name": "server",
"kind": "upstream",
"from": "alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b"
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,249 @@
{
"module": "claude-code",
"version": "1",
"moved": true,
"slug": "agent",
"capabilities": [
"package-manager"
],
"requires": [
"mcp-endpoint",
{
"name": "gitea-read",
"optional": true
}
],
"binds": {
"mcp-endpoint": "${dir:state}/mcp-endpoint.json"
},
"secrets": {
"gitea-read": "${dir:state}/gitea-read.secret"
},
"uses": [
"operator-channel"
],
"state": [
"servers",
"holdings",
"config",
"proposals",
"questions",
"memory",
"waits"
],
"reads": [
"claude-licence-manager.bindings"
],
"tools": [
"claude_code_status",
"claude_code_render",
"claude_code_guard",
"claude_code_pull",
"claude_code_grant",
"claude_code_add_api_key",
"claude_code_registrations",
"claude_code_memory",
"claude_code_mcp_list",
"claude_code_mcp_register",
"claude_code_mcp_unregister",
"claude_code_skill_list",
"claude_code_skill_register",
"claude_code_skill_unregister",
"claude_code_agent_list",
"claude_code_agent_register",
"claude_code_agent_unregister",
"claude_code_command_list",
"claude_code_command_register",
"claude_code_command_unregister",
"claude_code_hook_list",
"claude_code_hook_register",
"claude_code_hook_unregister",
"claude_code_output_style_list",
"claude_code_output_style_register",
"claude_code_output_style_unregister",
"claude_code_instructions_list",
"claude_code_instructions_register",
"claude_code_instructions_unregister",
"claude_code_instructions_propose",
"claude_code_proposals",
"claude_code_ask",
"claude_code_answer",
"claude_code_wait",
"claude_code_settings_get",
"claude_code_settings_set",
"claude_code_settings_clear",
"claude_code_permission_add",
"claude_code_permission_remove",
"claude_code_config_list",
"claude_code_config_show",
"claude_code_config_status",
"claude_code_config_import",
"claude_code_home_show",
"claude_code_home_remove",
"claude_code_home_removed",
"claude_code_home_restore",
"claude_code_judge",
"claude_code_judged"
],
"data": {
"own": [
{
"id": "agent-home",
"path": "${dir:agent-home}",
"class": "valuable",
"why": "the operator's agent's own files: its memory, history and projects"
}
]
},
"config": {
"role": {
"type": "string",
"description": "What this machine is for, in a sentence, written into the instructions every Claude Code session on it reads; empty says nothing.",
"trusted": true,
"default": ""
},
"mcp_servers": {
"type": "object",
"description": "The tool servers every session on this machine is given beside the mesh's own, by name, each as Claude Code's managed MCP configuration takes it; empty adds none.",
"trusted": true,
"default": {}
},
"managed_settings": {
"type": "object",
"description": "Keys of Claude Code's managed settings every session on this machine obeys, such as spinnerTipsEnabled, laid over the mesh's own; empty adds none. Falls back to agent-managed-settings, the mesh's or this node's.",
"trusted": true,
"predefined": "agent-managed-settings",
"default": {}
},
"instructions": {
"type": "object",
"description": "The instructions every session on this machine reads, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"output_styles": {
"type": "object",
"description": "The output styles offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"skills": {
"type": "object",
"description": "The skills offered to every session on this machine, by name, each its files by path, a text as one line or a list of its lines; a skill without its SKILL.md is left out; empty adds none.",
"trusted": true,
"default": {}
},
"commands": {
"type": "object",
"description": "The slash commands offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
},
"agents": {
"type": "object",
"description": "The agents offered to every session on this machine, by name, each a text as one line or a list of its lines; empty adds none.",
"trusted": true,
"default": {}
}
},
"resources": [
{
"id": "package",
"type": "package",
"package": "claude-code"
},
{
"id": "managed",
"type": "directory",
"path": "/etc/claude-code",
"mode": "0755",
"unknown": "program"
},
{
"id": "start",
"type": "file",
"path": "/usr/local/bin/claude-agent",
"mode": "0755",
"template": "files/claude-agent"
},
{
"id": "agent-account-name",
"type": "file",
"path": "/etc/claude-code/agent-account",
"mode": "0644",
"template": "files/agent-account"
},
{
"id": "agent-account",
"type": "user",
"name": "${machine:agent-account}",
"home": "${machine:agent-home}",
"root": "${machine:agent-root}"
},
{
"id": "agent-home",
"type": "directory",
"path": "${machine:agent-home}/.claude",
"mode": "0700",
"owner": "${machine:agent-account}",
"unknown": "module"
},
{
"id": "state",
"type": "directory",
"mode": "0700",
"owner": "${machine:account}",
"place": ".",
"unknown": "program"
},
{
"id": "facts",
"type": "file",
"path": "${dir:state}/facts.json",
"mode": "0600",
"owner": "${machine:account}",
"template": "files/facts.json"
},
{
"id": "settings",
"type": "file",
"path": "${dir:state}/settings.json",
"mode": "0600",
"owner": "${machine:account}",
"takes-config": ["role", "mcp_servers", "managed_settings", "instructions", "output_styles", "skills", "commands", "agents"],
"merge": "json",
"template": "files/settings.json"
}
],
"items": [
{
"collection": "shell:zsh:normal",
"template": "files/zsh-normal"
}
],
"build": {
"artifacts": [
{
"name": "tools",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/claude-code",
"binary": "claude-code",
"loads": [
"claude-code"
],
"env": {
"MESH_CLAUDE_CODE_STATE": "${dir:state}",
"MESH_CLAUDE_CODE_FACTS": "${dir:state}/facts.json",
"MESH_CLAUDE_CODE_SETTINGS": "${dir:state}/settings.json"
}
},
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,66 @@
{
"module": "ca-trust",
"version": "1",
"slug": "catrust",
"capabilities": [
"service-manager"
],
"requires": [
"internal-acme-ca"
],
"seats": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"claims": [
{
"name": "the-mesh-trust-anchor",
"scope": "node"
}
],
"resources": [
{
"id": "state",
"type": "directory",
"mode": "0700",
"place": ".",
"unknown": "program"
},
{
"id": "anchor",
"type": "file",
"path": "${dir:state}/anchor",
"mode": "0755",
"template": "files/anchor"
},
{
"id": "unit",
"type": "file",
"path": "/etc/systemd/system/mesh-ca-trust.service",
"mode": "0644",
"template": "files/mesh-ca-trust.service"
},
{
"id": "trust",
"type": "service",
"unit": "mesh-ca-trust.service",
"state": "running",
"boot": "enabled",
"restart-on": [
"anchor",
"unit"
]
}
],
"build": {
"artifacts": [
{
"name": "files",
"kind": "archive",
"from": "files"
}
]
}
}
@@ -0,0 +1,14 @@
{
"module": "imaged",
"version": "1",
"slug": "imaged",
"build": {
"artifacts": [
{
"name": "server",
"kind": "image",
"from": "Dockerfile"
}
]
}
}
@@ -0,0 +1,57 @@
{
"module": "netcheck",
"version": "1",
"emits": [
"tunnel.endpoint-changed",
"tunnel.handshake-stale",
"tunnel.handshake-fresh"
],
"tools": [
"netcheck_tunnel",
"netcheck_tcp",
"netcheck_dns",
"netcheck_http",
"netcheck_listening"
],
"replaces": {
"netcheck_listening": [
"ss -lunt",
"ss -ltn",
"netstat -lnt"
],
"netcheck_dns": [
"dig",
"nslookup"
],
"netcheck_tunnel": [
"wg show",
"wg show mesh0 dump"
]
},
"build": {
"artifacts": [
{
"name": "tools-go",
"kind": "bundle",
"language": "go",
"system": "arch",
"from": "cmd/netcheck",
"binary": "netcheck",
"loads": [
"netcheck"
]
},
{
"name": "tools-typescript",
"kind": "bundle",
"language": "typescript",
"entrypoints": [
"tools/index.js"
],
"loads": [
"tools/index.js"
]
}
]
}
}
@@ -0,0 +1,110 @@
package main
import (
"context"
"errors"
"fmt"
"log"
"strings"
"sync"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// toolchainStreams are the stream each build module carries its toolchain.reported event on, as the bus names it
// (declared by the build modules, heard by this module through consumes).
var toolchainStreams = map[string]string{buildrun.GoBuildModule: "TOOLCHAINS_GO", buildrun.NodeBuildModule: "TOOLCHAINS_NODE"}
// toolchainWatch keeps the build modules' toolchains in a buildrun.ToolchainSet and tells the builder when one changed. A
// build module's bucket is made when it is assigned, after the builder started, and the builder holds no timer (ADR 0437
// §8): it watches the buckets that exist at start, and a build module says each time it wrote its toolchains key
// (toolchain.reported, handled by Reported); a bucket not watched yet is then watched.
type toolchainWatch struct {
ctx context.Context
js bucketOpener
set *buildrun.ToolchainSet
// put takes a change of a toolchains bucket in: set.Put, or the gate of the assigned build nodes before it (issue 1940).
put func(module, key string, value []byte, deleted bool) bool
changed func(ctx context.Context) // a language's entries changed: the failed builds are tried again
logger *log.Logger
errs chan error
mu sync.Mutex
watched map[string]bool
}
func newToolchainWatch(ctx context.Context, js bucketOpener, set *buildrun.ToolchainSet, changed func(context.Context), logger *log.Logger) *toolchainWatch {
return &toolchainWatch{ctx: ctx, js: js, set: set, put: set.Put, changed: changed, logger: logger, errs: make(chan error, len(buildrun.LanguageOf)), watched: map[string]bool{}}
}
// start watches the buckets that exist now; a missing one is said and waits for its build module's event.
func (w *toolchainWatch) start() error {
for module := range buildrun.LanguageOf {
if err := w.ensure(module); err != nil {
return err
}
}
return nil
}
func (w *toolchainWatch) ensure(module string) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.watched[module] {
return nil
}
name := delivery.ToolchainsBucket(module)
kv, err := w.js.KeyValue(w.ctx, name)
switch {
case errors.Is(err, jetstream.ErrBucketNotFound):
w.logger.Printf("the bucket %s does not exist yet: it is watched when %s reports a toolchain", name, module)
return nil
case err != nil:
// Refused or not answered (its grant not on the bus yet, issue 1945): said, and opened again at the module's next report.
w.logger.Printf("the bucket %s cannot be read (%v): %s's toolchains are not heard until its next report", name, err, module)
return nil
}
watcher, err := settle(w.ctx, kv, func(e jetstream.KeyValueEntry) { w.put(module, e.Key(), e.Value(), !opOf(e)) })
if err != nil {
process.NotWatched(w.logger.Printf, name, err, fmt.Sprintf("%s's toolchains are not heard until its next report", module))
return nil
}
w.watched[module] = true
go func() {
w.errs <- follow(w.ctx, watcher, func(e jetstream.KeyValueEntry) {
if w.put(module, e.Key(), e.Value(), !opOf(e)) {
w.changed(w.ctx)
}
})
}()
w.changed(w.ctx) // what the settle read is new to the builder
return nil
}
// delivered is the part of a JetStream message the watch uses.
type delivered interface {
Subject() string
Ack() error
Term() error
}
// Reported takes one toolchain.reported event of a build module in: the module is read off the subject,
// mesh.stream.<module>.event.toolchain.reported, and its bucket is watched if it is not. A subject of no build module is
// terminated; a failure stays unacknowledged, and the server delivers it again.
func (w *toolchainWatch) Reported(m delivered) {
module, _, _ := strings.Cut(strings.TrimPrefix(m.Subject(), "mesh.stream."), ".event.")
if _, known := buildrun.LanguageOf[module]; !known || !strings.HasPrefix(m.Subject(), "mesh.stream.") {
w.logger.Printf("%s: not a build module's event, dropped", m.Subject())
_ = m.Term()
return
}
if err := w.ensure(module); err != nil {
w.logger.Printf("%s: left for redelivery: %v", m.Subject(), err)
return
}
_ = m.Ack()
}
+27
View File
@@ -0,0 +1,27 @@
module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
require git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261102193506-d221c633799a
require (
git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun v0.0.0-00010101000000-000000000000
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/time v0.15.0 // indirect
)
replace git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun => ./buildrun
+29
View File
@@ -0,0 +1,29 @@
git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261102193506-d221c633799a h1:oxgksCiq272/afXbUION/dxtAS5MPGnmyq6+AZQxmXg=
git.novox.be/novox/mesh-catalog/modules/mesh-build-go/buildstep v0.0.0-20261102193506-d221c633799a/go.mod h1:kpsX4Pgss5EeuBvel1zWa2ogj7MkR3jZso6oa/J0YUQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929 h1:RtAFbUJRP1qO9AIMM2ue+cXNzqj7WZ7IOlSqNvttwuU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261103174208-b7abe203a929/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
github.com/nats-io/nkeys v0.4.16/go.mod h1:llLgWoI0o4z/Q57q2R1kHfmocyhGV6VG/U18Glg1Afs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
@@ -0,0 +1,53 @@
package build
import (
"context"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A done build is announced again by hand** (recovery of photos and link2pay on novox, 2026-10-29). The controller takes a
// build in when a write of the build key reaches its take-in consumer, and composes then; a done key is never written again
// by the builder, so a composition or a grant lost afterwards could not be made again by any tool. AnnounceAgain writes the
// done key again with its own bytes, as a new revision of the same key: its content, state, artifacts and manifest stay
// identical, and only the write is new. Who asked and why is in the builder's log and in the reasks answer.
// AnnounceAgain rewrites the done key of module at fp unchanged, as a new revision, and answers the revision. A key that is
// not done is refused with why: failed (builder_build_again builds it) or waiting (builder_ask_again).
func (b *Builder) AnnounceAgain(ctx context.Context, module, fp, why, by string) BuildAgainResult {
b.init()
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return BuildAgainResult{Refused: err.Error()}
}
value, rev, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s cannot be read: %v", key, err)}
}
if !found {
return BuildAgainResult{Refused: fmt.Sprintf("%s has no build key: nothing to announce", key)}
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s holds what is not a build: %v", key, err)}
}
switch cur.State {
case delivery.BuildDone:
case delivery.BuildFailed:
return BuildAgainResult{Refused: fmt.Sprintf("%s is failed, not done: %s builds it again", key, BuildAgainTool)}
default:
return BuildAgainResult{Refused: fmt.Sprintf("%s is %q, not done: %s asks its waiting asks again", key, cur.State, AskAgainTool)}
}
// Compare-and-set at the revision read: a build written meanwhile is not overwritten.
nrev, err := b.Claimer.Store.Update(ctx, key, value, rev)
if err != nil {
if errors.Is(err, ErrConflict) {
return BuildAgainResult{Refused: fmt.Sprintf("%s changed while it was announced: read it again", key)}
}
return BuildAgainResult{Refused: fmt.Sprintf("%s could not be written: %v", key, err)}
}
b.logf("%s: done, announced again by %s (revision %d): %s", key, by, nrev, why)
return BuildAgainResult{Revision: nrev}
}
@@ -0,0 +1,49 @@
package build
import (
"bytes"
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func TestAnnounceAgainRewritesOnlyADoneKeyUnchanged(t *testing.T) { // recovery of photos and link2pay, 2026-10-29
for _, c := range []struct {
name string
key *delivery.BuilderBuild
fp string
refuse string
}{
{name: "a done key is written again, unchanged, as a new revision", key: &delivery.BuilderBuild{State: delivery.BuildDone, Step: "built"}},
{name: "a failed key is refused, builder_build_again is for it", refuse: "builder_build_again", key: &delivery.BuilderBuild{State: delivery.BuildFailed, Error: "x"}},
{name: "a building key is refused, builder_ask_again is for it", refuse: "builder_ask_again", key: &delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.builder", Step: "asked"}},
{name: "a key with no build is refused", refuse: "has no build key"},
{name: "a fingerprint that is not one is refused", refuse: "fingerprint", fp: "nope"},
} {
t.Run(c.name, func(t *testing.T) {
r := newRig(t, goBundle)
var rev uint64
if c.key != nil {
c.key.Manifest = manifestD
rev = putBuild(t, r.st, r.key, *c.key)
}
before := bytes.Clone(r.st.vals[r.key])
fp := fp1
if c.fp != "" {
fp = c.fp
}
res := r.b.AnnounceAgain(context.Background(), "m", fp, "grant lost", "shanks.lead")
if c.refuse != "" {
if !strings.Contains(res.Refused, c.refuse) || res.Revision != 0 || r.st.revs[r.key] != rev || !bytes.Equal(r.st.vals[r.key], before) {
t.Fatalf("answer %+v, key at revision %d (was %d); want refused with %q and nothing written", res, r.st.revs[r.key], rev, c.refuse)
}
return
}
if res.Refused != "" || res.Revision <= rev || r.st.revs[r.key] != res.Revision || !bytes.Equal(r.st.vals[r.key], before) {
t.Fatalf("answer %+v, key at revision %d (was %d); want a new revision and the same bytes", res, r.st.revs[r.key], rev)
}
})
}
}
@@ -0,0 +1,419 @@
package build
import (
"context"
"errors"
"fmt"
"regexp"
"slices"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **An ask that can no longer be answered is asked again** (novox/hq issue 1945). A build seat's stream gives an ask up
// once it was delivered as often as its worker may (delivery.AskMaxDeliver), and a message given up on, dropped or
// terminated with no step written is answered by no build node: the lane waiting for its step would wait for ever, and on
// restart take over the same id and wait again. The builder notices it at the check bound (CheckAfter, the step bound),
// a context's bound and no timer of its own: it asks Gone, which reads the bus, and asks such an ask again under a new
// id, written into the key, said in its log and as the event build.asked-again, at most Askings times; then the key is
// failed with why. An ask DEAD_LETTERS says the seat gave up (buildrun.LostAskWaiter, issue 734) is asked again at once.
// A person asks the same by hand with the tool builder_ask_again (AskAgain).
// DefaultAskings is how many times a build module's part of a build is asked, the first included, when Builder.Askings is 0.
const DefaultAskings = 3
// AskedAgainEvent is the event the builder says when it asked an ask again: mesh.mod.mesh-delivery-builder.event.<this>.
const AskedAgainEvent = "build.asked-again"
// AskedAgain is the body of AskedAgainEvent.
type AskedAgain struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp"`
BuildModule string `json:"build_module"`
Language string `json:"language"`
Old string `json:"old"`
New string `json:"new"`
Asking int `json:"asking"`
Askings int `json:"askings"`
Why string `json:"why"`
At time.Time `json:"at"`
}
// AskSeen is what the bus says of an ask whose step has not ended, as the builder may read it: its build seat's stream
// (information and messages by sequence) and its build module's steps. The builder may not read the seat workers'
// delivery counts or ack floors (granted to the delivery view alone: $JS.API.CONSUMER.INFO.<stream>.<worker>), so a
// message given up on is told by what the workers did after it: a build node took an ask asked later, of the same kind or
// of one pulled after it (issue 1945 note 62: seven messages SEAT_NODE_BUILD_GO had given up on sat ahead of a Go ask for
// five hours, each counted as waiting, so the ask was never asked again).
type AskSeen struct {
Stream string
// Kind is the ask's kind, At when it was asked (its id's time).
Kind string
At time.Time
// Step is whether a build node wrote a step for the ask (running: the step bound of buildrun.Asker judges it).
Step bool
// OnStream is whether the stream still holds the ask's message; PutAt is when it was put.
OnStream bool
PutAt time.Time
// Ahead are the messages of its subject before it on the stream.
Ahead []Queued
// Last is the ask asked last that a build node took (wrote a step for) of the ask's kind or of a kind pulled after
// it, when one was taken: every message of the kind asked before it was then delivered already, or given up on.
Last *Taken
}
// Queued is one message of a build seat's stream: its sequence, its ask's id and kind, when the ask was asked (its id's
// time, the put time when the id says none) and put, and the state of the step a build node wrote for it ("" for none).
type Queued struct {
Seq uint64 `json:"seq"`
Ask string `json:"ask"`
Kind string `json:"kind"`
At time.Time `json:"asked"`
PutAt time.Time `json:"put"`
Step string `json:"step,omitempty"`
}
// Taken is an ask a build node took: its id, kind and when it was asked, and the step's node and start.
type Taken struct {
Ask string `json:"ask"`
Kind string `json:"kind"`
At time.Time `json:"asked"`
Node string `json:"node"`
Started time.Time `json:"started"`
}
// Proves is whether a taken ask of kind taken shows that a message of kind waiting asked before it is no longer
// delivered: a holder pulls the kinds in delivery.AskKinds' order, so taking one of the same kind or of a later one means
// its worker had none of waiting's kind left to give.
func Proves(taken, waiting string) bool {
t, w := slices.Index(delivery.AskKinds, taken), slices.Index(delivery.AskKinds, waiting)
return t >= 0 && w >= 0 && t >= w
}
// Waits is whether a message ahead still waits to be delivered: no build node wrote a step for it, and no ask of its
// kind asked after it was taken. A message its worker gave up on (delivered its most times) stays on a work queue until
// its age, and is not counted.
func (s AskSeen) Waits(q Queued) bool {
return q.Step == "" && !s.passed(q.At)
}
func (s AskSeen) passed(at time.Time) bool {
return s.Last != nil && Proves(s.Last.Kind, s.Kind) && s.Last.At.After(at)
}
// Waiting is how many messages ahead still wait to be delivered (Waits).
func (s AskSeen) Waiting() int {
n := 0
for _, q := range s.Ahead {
if s.Waits(q) {
n++
}
}
return n
}
// Why is why the ask can no longer be answered, or "" when it can: it has a step, waits behind others of its kind still
// deliverable, or no build node took a later ask of its kind (the nodes are busy or paused: a backlog, never a fault,
// ADR 0525 §9).
func (s AskSeen) Why() string {
switch {
case s.Step:
return ""
case !s.OnStream:
return fmt.Sprintf("no build node wrote a step for it and %s no longer holds it: it was given up on or dropped", s.Stream)
case s.Waiting() > 0 || !s.passed(s.at()):
return ""
default:
behind := ""
if n := len(s.Ahead); n > 0 {
behind = fmt.Sprintf(" behind %d given up on or taken", n)
}
return fmt.Sprintf("it waits on %s since %s%s while %s took the %s ask %s asked later at %s: its stream no longer delivers it (given up after its most deliveries)",
s.Stream, s.PutAt.UTC().Format(time.RFC3339), behind, s.Last.Node, s.Last.Kind, s.Last.Ask, s.Last.At.UTC().Format(time.RFC3339))
}
}
func (s AskSeen) at() time.Time {
if s.At.IsZero() {
return s.PutAt
}
return s.At
}
// AskedAt is when an ask id was made (buildstep.NewID: its milliseconds in base 36 before six hex digits), or zero.
func AskedAt(id string) time.Time {
if len(id) <= 6 {
return time.Time{}
}
ms, err := strconv.ParseInt(id[:len(id)-6], 36, 64)
if err != nil || ms <= 0 {
return time.Time{}
}
return time.UnixMilli(ms)
}
// Dropper removes an ask's message from its build seat's stream (Builder.Gone may be one): the old message of an ask asked
// again, so a message given up on does not sit ahead of every later ask of its kind until its age (issue 1945 note 62).
type Dropper interface {
Drop(ctx context.Context, a delivery.StepAsk) error
}
// drop removes the old ask's message when Gone can; a refusal is said and changes nothing else: the new ask is put.
func (b *Builder) drop(ctx context.Context, ask delivery.StepAsk, old string) {
d, ok := b.Gone.(Dropper)
if !ok {
return
}
prior := ask
prior.ID = old
if err := d.Drop(ctx, prior); err != nil {
b.logf("the message of the ask %s asked again as %s stays on its stream: %v", old, ask.ID, err)
}
}
// DefaultDeclineBound is how long one ask may go on being declined, from its first decline, before its key fails (the
// checker's DefaultDeclineBound, issue 890); DefaultDeclinePause is how long the builder waits before it puts a declined
// ask again, so a lone node off the agreed toolchain cannot spin on it.
const (
DefaultDeclineBound = 30 * time.Minute
DefaultDeclinePause = 5 * time.Second
)
func (b *Builder) declineBound() time.Duration {
if b.DeclineBound > 0 {
return b.DeclineBound
}
return DefaultDeclineBound
}
func (b *Builder) declinePause() time.Duration {
if b.DeclinePause != 0 {
return max(b.DeclinePause, 0)
}
return DefaultDeclinePause
}
// declinedWords names the nodes that declined an ask and why, in the order of their names.
func declinedWords(by map[string]string) string {
nodes := make([]string, 0, len(by))
for n := range by {
nodes = append(nodes, n)
}
sort.Strings(nodes)
var parts []string
for _, n := range nodes {
parts = append(parts, n+": "+by[n])
}
return strings.Join(parts, "; ")
}
func (b *Builder) askings() int {
if b.Askings > 0 {
return b.Askings
}
return DefaultAskings
}
var askingWords = regexp.MustCompile(`asked (\S+) again \(asking (\d+)\)`)
// askingOf is the asking a key's step words name for a build module's ask, 1 when none.
func askingOf(step, buildModule string) int {
for _, m := range askingWords.FindAllStringSubmatch(step, -1) {
if m[1] == buildModule {
if n, err := strconv.Atoi(m[2]); err == nil && n > 0 {
return n
}
}
}
return 1
}
// AskAgain asks the waiting asks of a build key again now, under new ids (the tool builder_ask_again): it answers the
// languages whose lane waits, none when no lane of this builder waits on the key.
func (b *Builder) AskAgain(key, why string) []string {
b.init()
b.mu.Lock()
defer b.mu.Unlock()
var asked []string
for language, c := range b.waiting[key] {
select {
case c <- why:
default: // one is already asked
}
asked = append(asked, language)
}
return asked
}
// Waiting are the build keys whose lanes wait for a step now, each with its languages.
func (b *Builder) Waiting() map[string][]string {
b.init()
b.mu.Lock()
defer b.mu.Unlock()
out := map[string][]string{}
for key, ls := range b.waiting {
for l := range ls {
out[key] = append(out[key], l)
}
}
return out
}
func (b *Builder) listen(key, language string) chan string {
b.init()
c := make(chan string, 1)
b.mu.Lock()
if b.waiting[key] == nil {
b.waiting[key] = map[string]chan string{}
}
b.waiting[key][language] = c
b.mu.Unlock()
return c
}
func (b *Builder) unlisten(key, language string, c chan string) {
b.mu.Lock()
if b.waiting[key][language] == c {
delete(b.waiting[key], language)
if len(b.waiting[key]) == 0 {
delete(b.waiting, key)
}
}
b.mu.Unlock()
}
var (
errCheck = errors.New("the check bound passed")
errAgain = errors.New("asked again")
// errDeclined: a build node on another toolchain than the agreed one sat out of the ask (ADR 0584 clause 2).
errDeclined = errors.New("declined")
)
type byHand struct{ why string }
func (h byHand) Error() string { return "asked again by hand: " + h.why }
// answer waits for the step of the lane's ask, and asks it again while it can no longer be answered (Gone) or a person
// asked it again (AskAgain); it answers the step and the ask that has it.
func (b *Builder) answer(ctx context.Context, l *lane, ask delivery.StepAsk, asked *askNotes, module, fp string) (delivery.Step, delivery.StepAsk, error) {
key, _ := delivery.BuilderBuildKey(module, fp)
again := b.listen(key, ask.Language)
defer b.unlisten(key, ask.Language, again)
asking := askingOf(asked.step, l.module)
var firstDecline time.Time
declines := map[string]string{} // node -> why it declined, the last
for {
step, why, err := b.waitOnce(ctx, ask, again)
switch {
case errors.Is(err, errCheck):
continue
case errors.Is(err, errDeclined):
// Not an asking: a node that sits out would use them all while it idles and pulls the ask again at once.
// Bounded by time from the first decline, then failed loudly naming the nodes and their reasons (as the
// checker's decline loop is, issue 890).
now := b.now()
if firstDecline.IsZero() {
firstDecline = now
}
declines[step.Node] = why
if bound := b.declineBound(); now.Sub(firstDecline) > bound {
return step, ask, fmt.Errorf("the ask %s of %s was declined for %s and no build node took it: %s", ask.ID, l.module, bound.Round(time.Second), declinedWords(declines))
}
// The pause is a context's bound, not a timer (ADR 0437 §8).
pause, stop := context.WithTimeout(ctx, b.declinePause())
<-pause.Done()
stop()
if ctx.Err() != nil {
return step, ask, ctx.Err()
}
old := ask.ID
ask.ID, ask.At = buildstep.NewID(), time.Now()
if err := asked.note(ctx, ask.Language, ask.ID, fmt.Sprintf("put %s again after %s declined it", l.module, step.Node)); err != nil {
return step, ask, fmt.Errorf("the ask of %s put again after a decline could not be written to the key: %w", l.module, err)
}
if err := b.Asks.Put(ctx, ask); err != nil {
return step, ask, err
}
b.drop(ctx, ask, old)
b.logf("%s: the ask %s of %s was declined on %s and is put again as %s: %s", key, old, l.module, step.Node, ask.ID, why)
case errors.Is(err, errAgain):
if asking >= b.askings() {
return step, ask, fmt.Errorf("the ask %s could not be answered and was asked %d times, the most it is: %s", ask.ID, asking, why)
}
asking++
old := ask.ID
ask.ID, ask.At = buildstep.NewID(), time.Now()
if err := asked.note(ctx, ask.Language, ask.ID, fmt.Sprintf("asked %s again (asking %d): %s", l.module, asking, why)); err != nil {
return step, ask, fmt.Errorf("the ask of %s asked again could not be written to the key: %w", l.module, err)
}
if err := b.Asks.Put(ctx, ask); err != nil {
return step, ask, err
}
b.drop(ctx, ask, old)
b.logf("%s: the ask %s of %s is asked again as %s (asking %d of %d): %s", key, old, l.module, ask.ID, asking, b.askings(), why)
if b.Said != nil {
b.Said(AskedAgain{Key: key, Module: module, FP: fp, BuildModule: l.module, Language: ask.Language, Old: old, New: ask.ID,
Asking: asking, Askings: b.askings(), Why: why, At: ask.At})
}
default:
return step, ask, err
}
}
}
// waitOnce waits for the ask's step until it ends, a person asks it again, or the check bound passes: then it asks Gone
// and answers errAgain with why when the ask can no longer be answered, errCheck when it waits on.
func (b *Builder) waitOnce(ctx context.Context, ask delivery.StepAsk, again <-chan string) (delivery.Step, string, error) {
hand, cancel := context.WithCancelCause(ctx)
defer cancel(nil)
go func() {
select {
case why := <-again:
cancel(byHand{why})
case <-hand.Done():
}
}()
wait := hand
if b.Gone != nil && b.CheckAfter > 0 {
var stop context.CancelFunc
wait, stop = context.WithTimeoutCause(hand, b.CheckAfter, errCheck)
defer stop()
}
step, err := b.Asks.Wait(wait, ask)
if _, declined := delivery.DeclinedOf(step); err == nil && declined {
// A build node off the agreed toolchain sat out (ADR 0584 clause 2). It is no asking: answer() puts the ask again
// after a pause, within DeclineBound.
return step, step.Reason, errDeclined
}
if err == nil && buildrun.IsLost(step) {
// The build seat gave the ask up (issue 734): it can no longer be answered, so it is asked again now, with the
// words of why, not at the check bound.
return step, step.Reason, errAgain
}
if err == nil || ctx.Err() != nil {
return step, "", err
}
var h byHand
if errors.As(context.Cause(hand), &h) {
return step, strings.TrimSpace("asked again by hand: " + h.why), errAgain
}
if errors.Is(context.Cause(wait), errCheck) {
why, gerr := b.Gone.Gone(ctx, ask)
if gerr != nil {
b.logf("whether the ask %s can still be answered cannot be read, so it is waited for: %v", ask.ID, gerr)
return step, "", errCheck
}
if why == "" {
return step, "", errCheck
}
return step, why, errAgain
}
return step, "", err
}
@@ -0,0 +1,94 @@
package build
import (
"context"
"encoding/json"
"fmt"
"maps"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The build runner a build node runs an ask with is mesh-build-go's (novox/hq ADR 0593 §1), another repository's,
// which the builder does not import. These tests run an ask as the builder needs it run: fetch once at the job's
// fetch arguments with its package versions, build or copy each artifact, answer each one's digest and reference. They
// test the builder, not the runner.
// stepFunc is one step of a build module, as its tool answers it.
type stepFunc func(ctx context.Context, args map[string]any) (any, error)
// askSteps are a build module's steps an ask is run with.
type askSteps struct{ Fetch, Build, Copy stepFunc }
// runAsk runs a build ask with steps and answers its delivery.BuildAnswer, or the error that fails the step.
func runAsk(ctx context.Context, steps askSteps, ask delivery.StepAsk) (any, error) {
if ask.Build == nil {
return nil, fmt.Errorf("the ask %s holds no build job", ask.ID)
}
job := *ask.Build
fetch := job.Fetch
if len(ask.Packages) > 0 && fetch != nil {
fetch = maps.Clone(fetch)
fetch["packages"] = maps.Clone(ask.Packages)
}
answer := delivery.BuildAnswer{}
workspace := ""
for _, a := range job.Artifacts {
var out any
var err error
if a.From != "" {
if steps.Copy == nil {
return nil, fmt.Errorf("this build module copies no artifact (%s)", a.Artifact)
}
out, err = steps.Copy(ctx, map[string]any{"module": job.Module, "artifact": a.Artifact, "from": a.From})
} else {
if workspace == "" {
raw, err := asRaw(steps.Fetch(ctx, fetch))
if err != nil {
return nil, err
}
var f struct {
Workspace string `json:"workspace"`
Merge string `json:"merge"`
Conflicts []string `json:"conflicts"`
}
if err := json.Unmarshal(raw, &f); err != nil || f.Workspace == "" {
return nil, fmt.Errorf("the fetch answered no workspace: %s", raw)
}
if _, candidate := fetch["base"]; candidate && (len(f.Conflicts) > 0 || f.Merge == "") {
return nil, fmt.Errorf("the head does not merge into the trunk: %s", strings.Join(f.Conflicts, ", "))
}
workspace = f.Workspace
}
args := map[string]any{"workspace": workspace, "module": job.Path, "artifact": a.Artifact}
if job.Publish {
args["publish"] = true
}
out, err = steps.Build(ctx, args)
}
raw, err := asRaw(out, err)
if err != nil {
return nil, fmt.Errorf("%s: %w", a.Artifact, err)
}
var built struct {
Digest string `json:"digest"`
Reference string `json:"reference"`
}
if err := json.Unmarshal(raw, &built); err != nil || built.Digest == "" {
return nil, fmt.Errorf("the build of %s answered no digest: %s", a.Artifact, raw)
}
answer.Artifacts = append(answer.Artifacts, delivery.AskBuilt{Artifact: a.Artifact, Digest: built.Digest, Reference: built.Reference})
}
return answer, nil
}
func asRaw(v any, err error) (json.RawMessage, error) {
if err != nil {
return nil, err
}
if r, ok := v.(json.RawMessage); ok {
return r, nil
}
return json.Marshal(v)
}
@@ -0,0 +1,183 @@
package build
import (
"context"
"fmt"
"maps"
"strings"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **A failed key is built again by hand** (novox/hq issue 1126 note 72; ADR 0429 clause 18: "a fingerprint is built again
// only when its build key failed, as a new revision of the same key"). The builder does it by itself whenever it looks at a
// module again (a modules value, a change of the toolchains, artifacts-gone, a moved trunk): Claimer.Claim writes a failed
// key building as a new revision and the build asks the build seats. Nothing looks at an unchanged module on its own, so a key
// that failed (a node refused the build, a toolchain was missing) stayed failed until the module changed. The tool
// builder_build_again (BuildAgain) makes the builder look at one module now, which does exactly that and no more: it keeps
// the claim's rules, so a key failed for a reason the builder never retries on the same fingerprint stays refused.
// BuildAgainResult is the builder's answer to a request to build a key again.
type BuildAgainResult struct {
// Refused is why nothing was done, "" when the key was written building as a new revision.
Refused string
// Revision is the revision the claim wrote; Asks are the ask ids put by language, as far as they were put within the bound.
Revision uint64
Asks map[string]string
// Pending says the builder had not claimed the key, or not put every ask, when it answered: not a refusal.
Pending string
// Failed is the error of the key when the build ended failed before its asks were put (a plan that cannot be read, a
// toolchain not agreed): the new revision was written, and nothing was asked.
Failed string
}
// handWatch follows one request to build a key again until its asks are put.
type handWatch struct {
rev uint64
lanes int
asks map[string]string
done chan struct{}
shut bool
// ended is set when the look ended before the asks were all put.
ended bool
}
// settle closes done once the claim is written and every lane has put its ask. Call with b.mu held.
func (w *handWatch) settle() {
if !w.shut && w.rev > 0 && w.lanes >= 0 && len(w.asks) >= w.lanes {
w.shut = true
close(w.done)
}
}
func (b *Builder) hand(key string, f func(w *handWatch)) {
b.mu.Lock()
defer b.mu.Unlock()
if w := b.hands[key]; w != nil {
f(w)
}
}
func (b *Builder) handClaimed(key string, rev uint64) {
b.hand(key, func(w *handWatch) { w.rev = rev; w.settle() })
}
func (b *Builder) handLanes(key string, n int) {
b.hand(key, func(w *handWatch) { w.lanes = n; w.settle() })
}
func (b *Builder) handAsk(key, language, id string) {
b.hand(key, func(w *handWatch) { w.asks[language] = id; w.settle() })
}
// handEnded ends the watch of a request whose look ended: it wrote no claim, or the build ended.
func (b *Builder) handEnded(key string) {
b.hand(key, func(w *handWatch) {
if !w.shut {
w.shut, w.ended = true, true
close(w.done)
}
})
}
// BuildAgain builds the failed key of module at fp again, as the builder does by itself: the key is written building as a
// new revision and the build module asks are put on the build seats. It answers once the asks are put, or at bound. A key
// that is not failed is refused with why: done (nothing to do) or waiting (builder_ask_again asks its waiting asks again).
// ctx is the builder's own context: the build it starts lives on it.
func (b *Builder) BuildAgain(ctx context.Context, module, fp, why, by string, bound time.Duration) BuildAgainResult {
b.init()
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return BuildAgainResult{Refused: err.Error()}
}
value, _, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s cannot be read: %v", key, err)}
}
if !found {
return BuildAgainResult{Refused: fmt.Sprintf("%s has no build key: nothing failed. The builder builds a fingerprint by itself once the indexer writes it to its module", key)}
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
return BuildAgainResult{Refused: fmt.Sprintf("%s holds what is not a build: %v", key, err)}
}
switch {
case cur.State == delivery.BuildDone:
return BuildAgainResult{Refused: fmt.Sprintf("%s is done: nothing to do", key)}
case cur.State == delivery.BuildBuilding || cur.State == delivery.BuildQueued:
return BuildAgainResult{Refused: fmt.Sprintf("%s is waiting (%s: %s): %s asks its waiting asks again", key, cur.State, cur.Step, AskAgainTool)}
case cur.State != delivery.BuildFailed:
return BuildAgainResult{Refused: fmt.Sprintf("%s is %q, not failed", key, cur.State)}
case delivery.FailedUntilChanged(cur.Error) || strings.HasPrefix(cur.Error, buildrun.DigestsDifferPrefix):
return BuildAgainResult{Refused: fmt.Sprintf("%s failed for a reason the builder does not retry on the same fingerprint, it waits for another fingerprint: %s", key, cur.Error)}
}
// One critical section: the key is refused when it is looked at or asked already, the module still has the fingerprint,
// and the watch and the look are registered before anything else can take the key.
w := &handWatch{lanes: -1, asks: map[string]string{}, done: make(chan struct{})}
b.mu.Lock()
m, known := b.known[module]
switch {
case !known || m.FP2 != fp:
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("module %s does not have the fingerprint %s now (a newer one superseded it, or its modules value is not read yet): the builder builds the module's current fingerprint", module, fp)}
case m.Manifest == "":
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("module %s names no manifest at %s yet, so the builder looks at nothing", module, fp)}
case b.inflight[key] || b.hands[key] != nil:
b.mu.Unlock()
return BuildAgainResult{Refused: fmt.Sprintf("%s is being looked at by this builder now (or asked again by hand): wait for it to end", key)}
}
b.hands[key] = w
b.inflight[key] = true
b.mu.Unlock()
defer func() {
b.mu.Lock()
if b.hands[key] == w {
delete(b.hands, key)
}
b.mu.Unlock()
}()
b.logf("%s: failed (%s); built again by hand, asked by %s: %s", key, cur.Error, by, why)
b.startOnce(ctx, key, module, m)
wait, stop := context.WithTimeout(ctx, bound) // a context's bound for a person's answer (ADR 0437 §8)
defer stop()
select {
case <-w.done:
case <-wait.Done():
}
b.mu.Lock()
res := BuildAgainResult{Revision: w.rev, Asks: maps.Clone(w.asks)}
ended, shut, lanes := w.ended, w.shut, w.lanes
b.mu.Unlock()
switch {
case res.Revision == 0 && shut:
res.Refused = fmt.Sprintf("the builder looked at %s and wrote nothing (a toolchain moved, the claim was lost or could not be written): its log says why", key)
case res.Revision == 0:
res.Pending = fmt.Sprintf("the builder has not claimed %s within %s: it waits behind other builds; builder_stuck shows where it stands", key, bound)
case ended && len(res.Asks) < max(lanes, 0)+boolInt(lanes < 0):
// The look ended before every ask was put: say how the key stands, not that it was asked.
if v, _, found, err := b.Claimer.Store.Get(ctx, key); err == nil && found {
if cur, err := delivery.DecodeBuilderBuild(v); err == nil {
switch cur.State {
case delivery.BuildFailed:
res.Failed = cur.Error
case delivery.BuildDone:
default:
res.Pending = fmt.Sprintf("the look ended with the key %s (%s) before every ask was put: its log says why", cur.State, cur.Step)
}
}
}
case !shut:
res.Pending = fmt.Sprintf("the key is building at revision %d, and not every ask was put within %s; builder_stuck shows the asks", res.Revision, bound)
}
return res
}
func boolInt(b bool) int {
if b {
return 1
}
return 0
}
@@ -0,0 +1,131 @@
package build
import (
"context"
"fmt"
"sort"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// How a candidate's fingerprint is built (novox/hq ADR 0484 clause 3, issue 1684): the indexer writes, for each module a
// pull request touches, the fingerprint of the new form it has at the head merged onto the trunk (candidates.<group>), and
// the builder builds each, once, as a key of its own (builds.<module>.<fingerprint>) that the checker's line reads. The
// build is the trunk module's: the same build modules, the same nodes and capacity; what differs is where the source comes
// from: the fetch step is given the pull request's head, the ref that carries it and the trunk commit to merge it onto, as
// the checker's fetch is, and builds the merge's module directory.
// via is how a build's fetch step reaches its source when it is a candidate's: the ref that carries the head and the trunk
// commit the head is merged onto.
type via struct{ Ref, Base string }
// Candidate takes one value of the indexer's candidates bucket in (the key is the group) and builds the fingerprints it
// carries. A value that is not a candidate is said and left.
func (b *Builder) Candidate(ctx context.Context, key string, value []byte) {
b.init()
c, err := delivery.DecodeCandidate(value)
if err != nil {
b.logf("candidates.%s holds what is not a candidate, left: %v", key, err)
return
}
b.mu.Lock()
b.cands[key] = c
b.mu.Unlock()
b.considerCandidate(ctx, c)
}
// ForgetCandidate takes a candidate out (its key was deleted).
func (b *Builder) ForgetCandidate(key string) {
b.init()
b.mu.Lock()
delete(b.cands, key)
b.mu.Unlock()
}
func (b *Builder) considerCandidate(ctx context.Context, c delivery.Candidate) {
for _, p := range c.Pulls {
names := make([]string, 0, len(p.Fps))
for name := range p.Fps {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
cm := p.Fps[name]
if cm.FP2 == "" || cm.Manifest == "" {
continue // no fingerprint of the new form for it (a reason in none2) or no manifest to build by
}
b.mu.Lock()
trunk, known := b.known[name]
b.mu.Unlock()
if !known || trunk.Path == "" {
b.logf("candidates.%s: the module %s has no record of the trunk yet, so its fingerprint %s is not built", c.Group, name, cm.FP2)
continue
}
key, err := delivery.BuilderBuildKey(name, cm.FP2)
if err != nil {
b.logf("candidates.%s: %v", c.Group, err)
continue
}
m := delivery.Module{Repository: p.Repository, Commit: p.Head.Head, FP2: cm.FP2, Manifest: cm.Manifest, Path: trunk.Path,
Toolchains: b.agreedFor(ctx, cm.Manifest), Packages: cm.Packages} // the group's package heads (ADR 0573 §4)
if len(cm.Paths) > 0 {
// The named paths' objects the preview gave it: the fetch names the paths (runLane, ADR 0499 clause 4).
m.Inputs2 = &delivery.KeptInputs2{Paths: cm.Paths}
}
// The preview is merged onto the trunk head the candidate was previewed onto: the head of the pull request's
// repository in the gate's source revision c.Base (ADR 0484 §3). Never the module's value's commit: that is the
// commit its fingerprint first appeared at, and may be many merges behind the trunk.
base, err := b.trunkHead(ctx, c.Base, p.Repository)
if err != nil {
b.logf("candidates.%s: the module %s's fingerprint %s is not built: %v", c.Group, name, cm.FP2, err)
continue
}
b.mu.Lock()
if _, onTrunk := b.trunkFP[key]; !onTrunk {
b.via[key] = via{Ref: fmt.Sprintf("refs/pull/%d/head", p.Number), Base: base}
}
b.mu.Unlock()
b.consider(ctx, name, m)
}
}
}
// agreedFor are the toolchains the build nodes agree on now, for the languages of the manifest's plan: the toolchains the
// candidate's fingerprint was computed with, as the indexer recomputes it whenever they change. A language with none agreed
// is left out, and the build then waits for it as the trunk's does.
func (b *Builder) agreedFor(ctx context.Context, manifest string) map[string]delivery.ToolchainUsed {
plan, err, _ := b.planOf(ctx, manifest)
if err != nil {
return nil
}
out := map[string]delivery.ToolchainUsed{}
for _, bm := range plan.Modules() {
if !plan.NeedsToolchain(bm) {
continue
}
language := buildrun.LanguageOf[bm]
if tc, none := delivery.AgreedToolchain(b.Toolchains.Entries(language)); none == "" {
out[language] = delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
}
}
if len(out) == 0 {
return nil
}
return out
}
// trunkHead is the head of repository in the gate's source revision rev (Builder.Trunk), or why it cannot be said.
func (b *Builder) trunkHead(ctx context.Context, rev uint64, repository string) (string, error) {
if b.Trunk == nil {
return "", fmt.Errorf("the gate's source cannot be read here, so the trunk head %s was previewed onto is not known", repository)
}
head, err := b.Trunk(ctx, rev, repository)
if err != nil {
return "", fmt.Errorf("the trunk head of %s at source revision %d cannot be read: %w", repository, rev, err)
}
if head == "" {
return "", fmt.Errorf("source revision %d names no trunk head of %s", rev, repository)
}
return head, nil
}
@@ -0,0 +1,181 @@
package build
import (
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// trunkModule writes the trunk's record of the module m at a fingerprint of its own, so that the candidate's fingerprint (fp1)
// is one the trunk does not carry.
func trunkModule(t *testing.T, r *rig) {
t.Helper()
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'),
FP2: "src2:" + hex64('e'), Manifest: mdigest('a'), Path: "modules/m", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
r.b.Wait()
r.bm.mu.Lock()
r.bm.calls = nil
r.bm.mu.Unlock()
}
func candidateValue(t *testing.T, group string, number int, head string) []byte {
t.Helper()
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: number, Head: head}, Modules: []string{"m"},
Fps: map[string]delivery.CandidateModule{"m": {FP2: fp1, Manifest: mdigest('a')}}}}
hash, err := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: group, HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
return data
}
const (
// trunkHead is the trunk head the rig's source revision names: newer than the module value's commit, which stays at the
// commit its fingerprint first appeared at.
trunkHead = "3333333333333333333333333333333333333333"
headX = "1111111111111111111111111111111111111111"
headY = "2222222222222222222222222222222222222222"
)
// indexerCandidate is a candidate as the indexer writes it, with the gate's source revision it names (mesh-delivery-indexer
// internal/index testdata/candidate-written.json, kept equal to what the indexer writes by its own test).
func indexerCandidate(t *testing.T) (rev uint64, heads map[string]string, c delivery.Candidate, raw []byte) {
t.Helper()
data, err := os.ReadFile("../../../mesh-delivery-indexer/internal/index/testdata/candidate-written.json")
if err != nil {
t.Fatal(err)
}
var w struct {
Source struct {
Rev uint64 `json:"rev"`
Heads map[string]string `json:"heads"`
} `json:"source"`
Candidate json.RawMessage `json:"candidate"`
}
if err := json.Unmarshal(data, &w); err != nil {
t.Fatal(err)
}
if c, err = delivery.DecodeCandidate(w.Candidate); err != nil {
t.Fatal(err)
}
return w.Source.Rev, w.Source.Heads, c, w.Candidate
}
func TestACandidatesFingerprintIsBuiltOnceAsAKeyOfItsOwnFromTheHeadMergedOntoTheTrunk(t *testing.T) { // N7/AC1, N7/AC2 (issue 1684)
// The candidate the indexer writes is merged onto the trunk head of the source revision it names, never onto the module
// value's commit, which stays at the commit its fingerprint first appeared at (review of the indexer writing on change).
{
rev, heads, c, raw := indexerCandidate(t)
p := c.Pulls[0]
cm := p.Fps["avahi"]
ir := newRig(t, goBundle)
ir.mf.byDigest[cm.Manifest] = catalogue.Manifest{Module: "avahi", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle}}}
ir.b.Trunk = func(_ context.Context, at uint64, repository string) (string, error) {
if at != rev {
return "", fmt.Errorf("source revision %d is not heard", at)
}
return heads[repository], nil
}
const valueCommit = "7a74cee83f88e6585a5511c3cc53bdf0f5303d20"
value, err := delivery.EncodeModule(delivery.Module{Repository: p.Repository, Commit: valueCommit, FP2: "src2:" + hex64('e'),
Manifest: cm.Manifest, Path: "modules/avahi", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
ir.b.Module(context.Background(), "avahi", value)
ir.b.Wait()
ir.bm.mu.Lock()
ir.bm.calls = nil
ir.bm.mu.Unlock()
ir.b.Candidate(context.Background(), "feat%2Fx", raw)
ir.b.Wait()
fetches := ir.bm.callsOf("build_go_fetch")
if len(fetches) != 1 {
t.Fatalf("%d fetches of the indexer's candidate, want one", len(fetches))
}
f := fetches[0].body.(map[string]any)
if want := heads[p.Repository]; want == "" || f["base"] != want || f["base"] == valueCommit || f["commit"] != p.Head.Head {
t.Errorf("fetch args = %+v, want the head merged onto the trunk head %.8s of source revision %d", f, want, rev)
}
}
r := newRig(t, goBundle)
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Fatalf("build = %+v, want the candidate's fingerprint built", v)
}
fetches := r.bm.callsOf("build_go_fetch")
if len(fetches) != 1 { // one node per fingerprint (ADR 0478 clause 4)
t.Fatalf("%d fetches, want one", len(fetches))
}
f := fetches[0].body.(map[string]any)
if f["commit"] != headX || f["ref"] != "refs/pull/7/head" || f["base"] != trunkHead || f["repository"] != "novox/mesh-catalog" {
t.Errorf("fetch args = %+v, want the head, its ref and the trunk head the candidate was previewed onto", f)
}
if b := r.bm.callsOf("build_go_build")[0].body.(map[string]any); b["module"] != "modules/m" {
t.Errorf("build args = %+v", b)
}
// Two heads carrying one fingerprint are one build.
r.b.Candidate(context.Background(), "feat%2Fy", candidateValue(t, "feat/y", 8, headY))
r.b.Wait()
if n := len(r.bm.callsOf("build_go_fetch")); n != 1 {
t.Errorf("%d fetches after a second head with the same fingerprint, want no more", n)
}
}
func TestACandidateWhoseModuleHasNoTrunkRecordIsNotBuilt(t *testing.T) {
r := newRig(t, goBundle)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if _, ok := r.st.vals[r.key]; ok {
t.Error("a build key was written for a module the builder knows no directory of")
}
}
func TestAFingerprintTheTrunkCarriesIsBuiltFromTheTrunksCommit(t *testing.T) { // the trunk's source wins
r := newRig(t, goBundle)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the trunk now carries fp1
r.b.Wait()
for _, c := range r.bm.callsOf("build_go_fetch") {
if f := c.body.(map[string]any); f["ref"] != nil || f["base"] != nil {
t.Errorf("fetch args = %+v: a trunk fingerprint is built from the trunk's commit alone", f)
}
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v", v)
}
}
func TestACandidateWhoseHeadDoesNotMergeIsFailedAndNoTrunkBuildIsStoredUnderIt(t *testing.T) { // review D4
for name, set := range map[string]func(*rig){
"conflict": func(r *rig) { r.bm.conflicts = "go.mod" },
"no merge": func(r *rig) { r.bm.noMerge = true },
} {
r := newRig(t, goBundle)
trunkModule(t, r)
set(r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildFailed || (name == "conflict" && !strings.Contains(v.Error, "go.mod")) || len(r.bm.callsOf("build_go_build")) != 0 {
t.Errorf("%s: build = %+v, builds %d, want failed and nothing built", name, v, len(r.bm.callsOf("build_go_build")))
}
}
}
@@ -0,0 +1,260 @@
package build
import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ErrConflict is what a Store answers when a Create finds the key or an Update finds another revision than the one given.
var ErrConflict = errors.New("the key exists or changed")
// Store is the builder's bucket (builds): create and compare-and-set, each answering the key's new revision. Only the
// builder's credential writes it (ADR 0429 clause 5).
type Store interface {
Get(ctx context.Context, key string) (value []byte, revision uint64, found bool, err error)
Create(ctx context.Context, key string, value []byte) (uint64, error)
Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error)
// Keys are every key of the bucket.
Keys(ctx context.Context) ([]string, error)
}
// Claimer claims fingerprints for the builder instance By (ADR 0429 clause 5: a builder claims a fingerprint by creating
// its key and moves it by compare-and-set).
type Claimer struct {
Store Store
By string
}
// Claim is a build the builder holds: its key, the revision the next write must be made at and the value last written.
type Claim struct {
Key string
Rev uint64
Build delivery.BuilderBuild
// Gone are the references of a done key the artifact store no longer holds, for which it was claimed again
// (ClaimOptions.Stored; issue 1937): said in the log and the key's step.
Gone []string
// Unasked is why the store could not be asked about a done key taken as it stands to publish: said in the log.
Unasked error
}
// ClaimOptions say what the builder knows about the moment of a claim.
type ClaimOptions struct {
// TakeOver takes a key found building whoever holds it: the builder has just started and looks at the key for the first
// time, and a build left building by a builder that is gone, on this node or another, is nobody's (ADR 0478 clause 6).
TakeOver bool
// ImageBuilder says whether a node runs a healthy image build module now: a key failed for NoImageBuilderPrefix is taken
// again only then. Nil means none does.
ImageBuilder func() bool
// Trunk is whether the trunk's record names this fingerprint. A done key whose image or package has no reference was
// built as a candidate's, which never pushes or publishes (ADR 0487 clause 12, ADR 0498 clauses 5 and 6); on the trunk it
// is taken again to publish them, an image only when an image builder runs (ImageBuilder).
Trunk bool
// Published says whether a package version `<name>@<version>` is published: one of the builder's done keys names it as
// a package's reference. A key queued waiting for a version (BuilderBuild.Waits) is taken only once it is (ADR 0500
// clause 6). Nil means none is.
Published func(ref string) bool
// Stored answers which of the references given the artifact store no longer holds (the seat mesh-artifact-sweep's verb held,
// novox/hq ADR 0571 §2, issue 1937). A done key of the trunk's fingerprint is asked before it is left
// done: the store's nightly collector deletes a blob nothing holds, and a key the controller never took in holds
// nothing, so a key whose blob is gone is claimed building as a new revision and built again. Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
}
// StoredReferences are the references of a done build's bundles and archives that name a blob in the artifact store
// (`artifact-store://<module>/<artifact>/blobs/<digest>`): what the store's collector can delete. An image or an upstream
// copy is a manifest, which the collector keeps, and a package lives in the package registry.
func StoredReferences(b delivery.BuilderBuild) []string {
var out []string
for _, a := range b.Artifacts {
if strings.HasPrefix(a.Reference, "artifact-store://") && strings.Contains(a.Reference, "/blobs/") &&
!slices.Contains(out, a.Reference) {
out = append(out, a.Reference)
}
}
return out
}
// GoneStep is the step a key claimed again for gone blobs says (issue 1937).
func GoneStep(gone []string) string {
return "claimed again: the artifact store no longer holds " + strings.Join(gone, ", ")
}
// Unpublished is whether a done build holds an image built without a push or a package built without a publish: it has
// no reference (a candidate's, ADR 0487 clause 12, ADR 0498 clauses 5 and 6).
func Unpublished(b delivery.BuilderBuild) bool { return len(UnpublishedModules(b)) > 0 }
// UnpublishedModules are the build modules that publish what a done build holds unpublished: the image build module for
// an image, the Node build module for a package; sorted, none when it holds nothing unpublished.
func UnpublishedModules(b delivery.BuilderBuild) []string {
if b.State != delivery.BuildDone {
return nil
}
var image, pkg bool
for _, a := range b.Artifacts {
image = image || (a.Kind == catalogue.ArtifactImage && a.Reference == "")
pkg = pkg || (a.Kind == catalogue.ArtifactPackage && a.Reference == "")
}
var out []string
if image {
out = append(out, ImageBuildModule)
}
if pkg {
out = append(out, buildrun.NodeBuildModule)
}
sort.Strings(out)
return out
}
// publishable are the build modules of UnpublishedModules that can publish now: the image build module only while a node
// runs it (imageBuilder).
func publishable(b delivery.BuilderBuild, imageBuilder func() bool) []string {
var out []string
for _, bm := range UnpublishedModules(b) {
if bm != ImageBuildModule || (imageBuilder != nil && imageBuilder()) {
out = append(out, bm)
}
}
return out
}
// Unreferenced is whether a done build holds a bundle, an archive or an upstream copy that names no reference: a key done
// before R4e, whose artifacts stayed on their build nodes (ADR 0498 clause 8). An artifact of no kind (a key written before
// kinds) is one too. Claim takes such a key again, once: its new revision names them.
func Unreferenced(b delivery.BuilderBuild) bool {
if b.State != delivery.BuildDone {
return false
}
for _, a := range b.Artifacts {
if a.Reference == "" && (a.Kind == "" || stored(a.Kind)) {
return true
}
}
return false
}
// Claim takes the build of module at fingerprint, or answers nil when it is not this builder's to build now:
// - no key: it is created building, by this builder; a create that finds the key made meanwhile gets nothing;
// - a done key is not taken again (the same source is never built twice), except one whose bundle, archive or upstream
// copy names no reference (Unreferenced, a key done before R4e: ADR 0498 clause 8), which is claimed building as a new
// revision and built whole, and one whose image or package was built unpublished as a candidate's once the trunk
// names it (ClaimOptions.Trunk): that is taken as it stands, with the done value, and its builder writes the new
// revision when the publishing build ends (ADR 0498 clause 6). From R4e one node builds each fingerprint (ADR 0478
// clause 4), so a key built on one node is never built again for a second;
// - a done key of the trunk's fingerprint whose bundle or archive blob the artifact store no longer holds
// (ClaimOptions.Stored, issue 1937) is claimed building as a new revision and built whole, its step saying which;
// - a failed key is taken again as a new revision of the same key (ADR 0429 clause 18), except one failed for a kind no
// builder builds, for an upstream pinned by no digest or a package version taken (delivery.FailedUntilChanged: ADR 0478
// clause 2, ADR 0498 clauses 4 and 5) or for digests that differ (buildrun.DigestsDifferPrefix, clause 4),
// which wait for another fingerprint, and one failed for want of an image builder (NoImageBuilderPrefix), which waits for
// another fingerprint or a node that runs the image build module (ClaimOptions.ImageBuilder);
// - a key queued waiting for a package version (Waits) is claimed building as a new revision once that version is
// published (ClaimOptions.Published, ADR 0500 clause 6), and not before;
// - a key left queued or building by this builder is taken as it stands (ADR 0478 clause 5: a stopped builder takes its
// own again when it starts), and so is one left by another builder when ClaimOptions.TakeOver (clause 6).
func (c Claimer) Claim(ctx context.Context, module, fingerprint, manifest string, opt ClaimOptions) (*Claim, error) {
key, err := delivery.BuilderBuildKey(module, fingerprint)
if err != nil {
return nil, err
}
building := delivery.BuilderBuild{State: delivery.BuildBuilding, By: c.By, Step: "claimed", Manifest: manifest}
value, rev, found, err := c.Store.Get(ctx, key)
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
if !found {
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Create(ctx, key, data) })
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
return nil, fmt.Errorf("%s holds what is not a build: %w", key, err)
}
switch {
case cur.State == delivery.BuildDone:
if Unreferenced(cur) {
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
}
if opt.Trunk && opt.Stored != nil {
if refs := StoredReferences(cur); len(refs) > 0 {
gone, err := opt.Stored(ctx, refs)
if err != nil {
err = fmt.Errorf("%s: whether the artifact store still holds %s cannot be asked, so the key is "+
"left done (the controller's take-in refuses it if one is gone): %w", key, strings.Join(refs, ", "), err)
if len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur, Unasked: err}, nil
}
return nil, err
}
if len(gone) > 0 {
again := building
again.Step = GoneStep(gone)
cl, err := c.write(ctx, key, again, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
if cl != nil {
cl.Gone = gone
}
return cl, err
}
}
}
if opt.Trunk && len(publishable(cur, opt.ImageBuilder)) > 0 {
return &Claim{Key: key, Rev: rev, Build: cur}, nil
}
return nil, nil
case cur.State == delivery.BuildFailed && (delivery.FailedUntilChanged(cur.Error) || strings.HasPrefix(cur.Error, buildrun.DigestsDifferPrefix)):
return nil, nil
case cur.State == delivery.BuildFailed && strings.HasPrefix(cur.Error, NoImageBuilderPrefix) && (opt.ImageBuilder == nil || !opt.ImageBuilder()):
return nil, nil
case cur.State == delivery.BuildFailed:
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
case cur.State == delivery.BuildQueued && cur.Waits != "":
if opt.Published == nil || !opt.Published(cur.Waits) {
return nil, nil // the package it waits for is not published yet
}
return c.write(ctx, key, building, func(data []byte) (uint64, error) { return c.Store.Update(ctx, key, data, rev) })
case cur.By == c.By || opt.TakeOver:
return &Claim{Key: key, Rev: rev, Build: cur}, nil
}
return nil, nil
}
func (c Claimer) write(_ context.Context, key string, b delivery.BuilderBuild, put func([]byte) (uint64, error)) (*Claim, error) {
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
return nil, err
}
rev, err := put(data)
if errors.Is(err, ErrConflict) {
return nil, nil // another builder made it first
}
if err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
b.V = delivery.Version
return &Claim{Key: key, Rev: rev, Build: b}, nil
}
// Finish moves the claimed key to b by compare-and-set at the claim's revision, and writes nothing when b is the value
// the key holds. A key that changed meanwhile is ErrConflict: the claim is no longer this builder's.
func (c Claimer) Finish(ctx context.Context, cl *Claim, b delivery.BuilderBuild) error {
b.V = delivery.Version
if delivery.SameBuilderBuild(cl.Build, b) {
return nil
}
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
return err
}
rev, err := c.Store.Update(ctx, cl.Key, data, cl.Rev)
if err != nil {
return fmt.Errorf("%s: %w", cl.Key, err)
}
cl.Rev, cl.Build = rev, b
return nil
}
@@ -0,0 +1,219 @@
package build
import (
"context"
"errors"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// memStore is a bucket's create and compare-and-set in memory, with a revision per write.
type memStore struct {
mu sync.Mutex
vals map[string][]byte
revs map[string]uint64
next uint64
writes int
// before runs once ahead of the next write, so a test changes the key under the writer.
before func()
}
func newMem() *memStore { return &memStore{vals: map[string][]byte{}, revs: map[string]uint64{}} }
func (m *memStore) Get(_ context.Context, key string) ([]byte, uint64, bool, error) {
m.mu.Lock()
defer m.mu.Unlock()
v, ok := m.vals[key]
return v, m.revs[key], ok, nil
}
func (m *memStore) Keys(_ context.Context) ([]string, error) {
m.mu.Lock()
defer m.mu.Unlock()
var keys []string
for k := range m.vals {
keys = append(keys, k)
}
return keys, nil
}
func (m *memStore) hook() {
if h := m.before; h != nil {
m.before = nil
h()
}
}
func (m *memStore) put(key string, value []byte) uint64 {
m.next++
m.writes++
m.vals[key], m.revs[key] = value, m.next
return m.next
}
func (m *memStore) Create(_ context.Context, key string, value []byte) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.hook()
if _, ok := m.vals[key]; ok {
return 0, ErrConflict
}
return m.put(key, value), nil
}
func (m *memStore) Update(_ context.Context, key string, value []byte, rev uint64) (uint64, error) {
m.mu.Lock()
defer m.mu.Unlock()
m.hook()
if _, ok := m.vals[key]; !ok || m.revs[key] != rev {
return 0, ErrConflict
}
return m.put(key, value), nil
}
const (
fp1 = "src2:1111111111111111111111111111111111111111111111111111111111111111"
digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
)
func seed(t *testing.T, st *memStore, b delivery.BuilderBuild) string {
t.Helper()
key, err := delivery.BuilderBuildKey("m", fp1)
if err != nil {
t.Fatal(err)
}
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
t.Fatal(err)
}
st.put(key, data)
return key
}
func valueOf(t *testing.T, st *memStore, key string) delivery.BuilderBuild {
t.Helper()
b, err := delivery.DecodeBuilderBuild(st.vals[key])
if err != nil {
t.Fatal(err)
}
return b
}
func TestClaimCreatesTheKeyBuildingAndASecondBuilderGetsNothing(t *testing.T) { // 1607/AC3
st := newMem()
a, b := Claimer{Store: st, By: "novox.a"}, Claimer{Store: st, By: "novox.b"}
got, err := a.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got == nil {
t.Fatalf("first claim = %v, %v, want a claim", got, err)
}
if v := valueOf(t, st, got.Key); v.State != delivery.BuildBuilding || v.By != "novox.a" || v.Manifest != hex64('a') {
t.Errorf("the key holds %+v, want building by novox.a with the manifest digest", v)
}
if second, err := b.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{}); err != nil || second != nil {
t.Errorf("second builder's claim = %v, %v, want none: the same source is never built twice", second, err)
}
}
func TestClaimLosingTheCreateRaceGetsNothing(t *testing.T) { // 1607/AC3
st := newMem()
st.before = func() { seed(t, st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.b"}) }
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
if err != nil || got != nil {
t.Errorf("claim = %v, %v, want none: the create failed because the key exists", got, err)
}
}
func TestADoneKeyIsNeverTakenAgain(t *testing.T) { // 1607/AC3; a key whose artifacts name references (ADR 0498 clause 8)
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.a", Manifest: hex64('a'), Artifacts: []delivery.BuiltArtifact{{Name: "x", Kind: "bundle", Digest: digest, Reference: "artifact-store://m/x/blobs/" + digest}}})
writes := st.writes
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
}
func TestAFailedKeyIsTakenAgainAsANewRevisionExceptARefusedKind(t *testing.T) { // 1607/AC3, ADR 0478 clause 2
st := newMem()
key := seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: "the step failed"})
rev := st.revs[key]
got, err := Claimer{Store: st, By: "novox.a"}.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
if err != nil || got == nil || st.revs[key] <= rev {
t.Fatalf("claim = %v, %v, revision %d (was %d), want a claim as a new revision of the same key", got, err, st.revs[key], rev)
}
if v := valueOf(t, st, key); v.State != delivery.BuildBuilding || v.Error != "" {
t.Errorf("the key holds %+v, want building with no error", v)
}
st = newMem()
key = seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: delivery.RefusedKind + "package of language \"python\" (artifact x)"})
writes := st.writes
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got != nil || st.writes != writes {
t.Errorf("a refused kind: claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
for _, e := range []string{delivery.NotPinned + "up: nats:2 names no digest", delivery.VersionTaken + "@novox/mesh-sdk@0.1.15"} { // B3, B5: ADR 0498 clauses 4, 5
st = newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: e})
writes = st.writes
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got != nil || st.writes != writes {
t.Errorf("%s: claim = %v, %v, writes %d (was %d), want none until the fingerprint changes", e, got, err, st.writes, writes)
}
}
}
func TestAKeyFailedForNoImageBuilderIsTakenAgainOnlyWhenANodeHasTheModule(t *testing.T) { // 1705b
st := newMem()
key := seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, By: "novox.a", Error: NoImageBuilderPrefix + " no node has mesh-build-image"})
writes := st.writes
c := Claimer{Store: st, By: "novox.a"}
for _, opt := range []ClaimOptions{{}, {ImageBuilder: func() bool { return false }}} {
if got, err := c.Claim(context.Background(), "m", fp1, "", opt); err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, writes %d (was %d), want none and no write", got, err, st.writes, writes)
}
}
got, err := c.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{ImageBuilder: func() bool { return true }})
if err != nil || got == nil || valueOf(t, st, key).State != delivery.BuildBuilding {
t.Errorf("claim = %v, %v, want a claim with the key building", got, err)
}
}
func TestFinishMovesTheKeyByCompareAndSetAndWritesOnlyADifferentValue(t *testing.T) { // 1607/AC3
st := newMem()
c := Claimer{Store: st, By: "novox.a"}
got, _ := c.Claim(context.Background(), "m", fp1, hex64('a'), ClaimOptions{})
done := delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.a", Manifest: hex64('a'), Artifacts: []delivery.BuiltArtifact{{Name: "x", Kind: "archive", Digest: digest}}}
if err := c.Finish(context.Background(), got, done); err != nil {
t.Fatal(err)
}
if v := valueOf(t, st, got.Key); v.State != delivery.BuildDone || v.Artifacts[0].Digest != digest {
t.Errorf("the key holds %+v", v)
}
writes := st.writes
if err := c.Finish(context.Background(), got, done); err != nil && !errors.Is(err, ErrConflict) {
t.Fatal(err)
}
if st.writes != writes {
t.Errorf("the same value was written again (%d writes, was %d)", st.writes, writes)
}
}
func TestFinishAfterTheKeyChangedIsAConflictNotAnOverwrite(t *testing.T) { // 1607/AC3
st := newMem()
c := Claimer{Store: st, By: "novox.a"}
got, _ := c.Claim(context.Background(), "m", fp1, "", ClaimOptions{})
st.put(got.Key, []byte(`{"v":1,"state":"failed","error":"someone else"}`))
err := c.Finish(context.Background(), got, delivery.BuilderBuild{State: delivery.BuildFailed, Error: "mine"})
if !errors.Is(err, ErrConflict) {
t.Errorf("Finish = %v, want ErrConflict", err)
}
}
func hex64(c byte) string {
b := make([]byte, 64)
for i := range b {
b[i] = c
}
return string(b)
}
@@ -0,0 +1,94 @@
package build
// The fakes of the build modules the builder tests run against: a caller answering a long tool at once with the step and its key,
// and the steps watch. The caller, the steps watch and the runner themselves are the SDK's (mesh-sdk go/delivery/buildrun, issue 1671).
import (
"context"
"encoding/json"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
type call struct {
node, key string
body any
}
// fakeCaller answers a long tool as the build modules do: at once, with the step and its key.
type fakeCaller struct {
calls []call
answer string
err error
}
func (f *fakeCaller) CallOn(node, key string, body any) (json.RawMessage, error) {
f.calls = append(f.calls, call{node, key, body})
if f.err != nil {
return nil, f.err
}
return json.RawMessage(f.answer), nil
}
// fakeSteps holds the steps values by `<module>/<key>`; Wait on a key it does not hold blocks until the context ends.
type fakeSteps struct {
mu sync.Mutex
steps map[string]delivery.Step
// during, when set, is called as a wait starts and the function it answers as the wait ends.
during func(key string) func()
}
func (f *fakeSteps) Wait(ctx context.Context, module, key string) (delivery.Step, error) {
f.mu.Lock()
s, ok := f.steps[module+"/"+key]
during := f.during
f.mu.Unlock()
if ok {
if during != nil {
defer during(key)()
}
return s, nil
}
<-ctx.Done()
return delivery.Step{}, context.Cause(ctx)
}
func stepAt(node, id, state, reason string, result string) delivery.Step {
s := delivery.Step{ID: id, Tool: "build_go_fetch", Node: node, State: state, Reason: reason}
if result != "" {
s.Result = json.RawMessage(result)
}
return s
}
// presenceValue is a watchman presence value, as its writer encodes it (mesh-sdk go/delivery EncodePresence).
func presenceValue(t *testing.T, online bool) []byte {
t.Helper()
b, err := delivery.EncodePresence(delivery.Presence{Online: online})
if err != nil {
t.Fatal(err)
}
return b
}
// tcValue is a toolchains value, as the build module's writer encodes it (mesh-sdk go/delivery EncodeToolchain).
func tcValue(t *testing.T, language, version string) []byte {
t.Helper()
b, err := delivery.EncodeToolchain(delivery.Toolchain{Language: language, Version: version, Recipe: "1", At: time.Now()})
if err != nil {
t.Fatal(err)
}
return b
}
// tcs are the toolchain entries of a language on nodes, as the toolchains set gives them.
func tcs(language, version string, nodes ...string) []delivery.NodeToolchain {
var out []delivery.NodeToolchain
for _, n := range nodes {
out = append(out, delivery.NodeToolchain{Node: n, Toolchain: delivery.Toolchain{Language: language, Version: version, Recipe: "1"}})
}
return out
}
@@ -0,0 +1,59 @@
package build
import (
"context"
"encoding/json"
"os"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// result reads a captured step value of mesh-build-image (testdata/captured/README) and returns its result.
func captured(t *testing.T, name string) json.RawMessage {
t.Helper()
raw, err := os.ReadFile("testdata/captured/" + name)
if err != nil {
t.Fatal(err)
}
var step delivery.Step
if err := json.Unmarshal(raw, &step); err != nil {
t.Fatalf("%s: %v", name, err)
}
if step.State != delivery.StepPassed || len(step.Result) == 0 {
t.Fatalf("%s: state %q, result %s, want a passed step with a result", name, step.State, step.Result)
}
return step.Result
}
// The captured image steps, run as a pulled ask runs them (runAsk), answer what the builder reads: the trunk's
// build its digest and reference, a candidate's its digest and no reference (issue 1887 A4a).
func TestCapturedImageStepsDecodeWithTheBuildersOwnTypes(t *testing.T) { // review r2, mesh-catalog 2; ADR 0487 clause 1
for _, tc := range []struct {
build string
publish bool
ref bool
}{{"build-trunk.json", true, true}, {"build-candidate.json", false, false}} {
steps := askSteps{
Fetch: func(context.Context, map[string]any) (any, error) {
return json.RawMessage(captured(t, "fetch.json")), nil
},
Build: func(context.Context, map[string]any) (any, error) { return json.RawMessage(captured(t, tc.build)), nil },
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskDelivery, Language: delivery.AskImage, Asker: delivery.BuilderModule,
At: time.Now(), Build: &delivery.BuildJob{Module: "m", Path: "modules/m", Publish: tc.publish,
Fetch: map[string]any{"repository": "novox/mesh-catalog", "commit": strings.Repeat("a", 40)},
Artifacts: []delivery.AskArtifact{{Artifact: "server"}}}}
out, err := runAsk(context.Background(), steps, ask)
if err != nil {
t.Fatalf("%s: %v", tc.build, err)
}
a := out.(delivery.BuildAnswer)
if len(a.Artifacts) != 1 || a.Artifacts[0].Digest == "" || (a.Artifacts[0].Reference != "") != tc.ref {
t.Errorf("%s answered %+v, want a digest and a reference only for the trunk", tc.build, a)
}
}
}
@@ -0,0 +1,46 @@
package build
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1660: at start the builder closes a key left building for a fingerprint the module no longer has.
func TestAKeyLeftBuildingForAFingerprintTheModuleNoLongerHasIsClosedAtStart(t *testing.T) { // issue 1660
r := newRig(t, goBundle)
stale := seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD}) // m at fp1
other, _ := delivery.BuilderBuildKey("gone", fp1)
data, _ := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
r.st.put(other, data)
// The module m has moved on to another fingerprint.
moved := delivery.Module{Repository: "novox/mesh-catalog", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP: "src1:" + hex64('f'),
FP2: "src2:" + hex64('7'), Manifest: manifestD, Path: "modules/m", Toolchains: agreedToolchains}
enc, _ := delivery.EncodeModule(moved)
r.b.Module(context.Background(), "m", enc)
r.b.Wait()
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
for _, k := range []string{stale, other} {
v := valueOf(t, r.st, k)
if v.State != delivery.BuildFailed || !strings.Contains(v.Error, "no longer") || !strings.Contains(v.Error, "ace.builder") {
t.Errorf("%s = %+v, want it closed as failed, naming why and who left it", k, v)
}
}
}
func TestSweepLeavesAKeyOfTheModulesCurrentFingerprintForTheBuildToTakeOver(t *testing.T) { // issue 1660
r := newRig(t, goBundle)
seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // takes it over and builds it
r.b.Wait()
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.By != "novox.builder" {
t.Errorf("build = %+v, want done by novox.builder", v)
}
}
@@ -0,0 +1,204 @@
package build
import (
"context"
"errors"
"fmt"
"slices"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A done key whose blob the artifact store no longer holds is built again (novox/hq issue 1937): on 2026-10-24 the
// store's 03:30 collection deleted the bare blobs of node-tools', mesh-host's and the forge bridge's done keys, which the
// controller had never taken in, and the builder kept them done.
// fakeHeld answers the controller's verb held: the references in gone are gone.
type fakeHeld struct {
mu sync.Mutex
gone map[string]bool
err error
asked [][]string
}
func (f *fakeHeld) stored(_ context.Context, refs []string) ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.asked = append(f.asked, slices.Clone(refs))
if f.err != nil {
return nil, f.err
}
var out []string
for _, r := range refs {
if f.gone[r] {
out = append(out, r)
}
}
return out, nil
}
func (f *fakeHeld) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.asked)
}
// logged keeps what the builder says.
type logged struct {
mu sync.Mutex
lines []string
}
func (l *logged) logf(format string, args ...any) {
l.mu.Lock()
defer l.mu.Unlock()
l.lines = append(l.lines, fmt.Sprintf(format, args...))
}
func (l *logged) has(words ...string) bool {
l.mu.Lock()
defer l.mu.Unlock()
for _, line := range l.lines {
all := true
for _, w := range words {
all = all && strings.Contains(line, w)
}
if all {
return true
}
}
return false
}
const goneRef = "artifact-store://m/tool/blobs/" + digest
func doneKeyNaming(ref string) delivery.BuilderBuild {
return delivery.BuilderBuild{State: delivery.BuildDone, By: "shanks.builder", Manifest: manifestD, Step: "built mesh-build-go on shanks",
Artifacts: []delivery.BuiltArtifact{{Name: "tool", Kind: "bundle", Digest: digest, Reference: ref}}}
}
// 1937/AC3 (unit, the claim): a done key of the trunk's fingerprint whose blob is gone is claimed building as a new
// revision, its step naming the gone reference; a candidate's done key is not asked about.
func TestADoneTrunkKeyWhoseBlobIsGoneIsClaimedAgain(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
rev := putBuild(t, st, key, doneKeyNaming(goneRef))
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Stored: held.stored})
if err != nil || cl != nil || held.asks() != 0 {
t.Fatalf("a candidate's done key: claim %+v, %v, %d asks; want nothing and no ask", cl, err, held.asks())
}
cl, err = c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if err != nil || cl == nil {
t.Fatalf("claim = %+v, %v; want the key claimed again", cl, err)
}
if cl.Rev <= rev || cl.Build.State != delivery.BuildBuilding || !slices.Equal(cl.Gone, []string{goneRef}) {
t.Errorf("claim = %+v (revision was %d); want building as a new revision naming the gone reference", cl, rev)
}
v := valueOf(t, st, key)
if v.State != delivery.BuildBuilding || !strings.Contains(v.Step, "no longer holds") || !strings.Contains(v.Step, goneRef) {
t.Errorf("the key holds %+v; want building, its step naming the gone reference", v)
}
}
// 1937/AC5 (unit, the claim): a done trunk key whose blobs are all held stays done; a store that cannot be asked leaves
// it done and says why; images and packages are not asked about.
func TestADoneTrunkKeyStaysDoneWhileTheStoreHoldsItsBlobs(t *testing.T) {
st := newMem()
c := Claimer{Store: st, By: "novox.builder"}
key, _ := delivery.BuilderBuildKey("m", fp1)
b := doneKeyNaming(goneRef)
b.Artifacts = append(b.Artifacts, delivery.BuiltArtifact{Name: "img", Kind: "image", Digest: digest, Reference: "registry.example/m/img@" + digest})
rev := putBuild(t, st, key, b)
held := &fakeHeld{gone: map[string]bool{}}
if cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored}); err != nil || cl != nil {
t.Fatalf("claim = %+v, %v; want nothing: the store holds the blob", cl, err)
}
if len(held.asked) != 1 || !slices.Equal(held.asked[0], []string{goneRef}) {
t.Errorf("asked %v; want the bundle's reference alone", held.asked)
}
held.err = errors.New("the controller did not answer")
cl, err := c.Claim(t.Context(), "m", fp1, manifestD, ClaimOptions{Trunk: true, Stored: held.stored})
if cl != nil || err == nil || !strings.Contains(err.Error(), "cannot be asked") || !strings.Contains(err.Error(), "left done") {
t.Errorf("claim = %+v, %v; want the key left done and said", cl, err)
}
if st.revs[key] != rev {
t.Errorf("the key moved to revision %d (was %d)", st.revs[key], rev)
}
}
// 1937/AC3 (unit, the builder): the builder's look at a module whose done key's blob is gone builds it again, says so
// in a log line and in the done key's step, and asks the store once per process while it answers held.
func TestTheBuilderBuildsAgainADoneKeyWhoseBlobIsGone(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{goneRef: true}}
r.b.Stored = held.stored
rev := putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again", v, r.st.revs[r.key], rev)
}
if !strings.Contains(v.Step, "built again: the artifact store no longer held "+goneRef) {
t.Errorf("the done key's step %q does not say it was built again for the gone blob", v.Step)
}
if !log.has(r.key, "built again", goneRef) {
t.Errorf("no log line says the key was built again: %v", log.lines)
}
// The new revision's blob is held: asked once, then not again in this process.
held.mu.Lock()
held.gone = map[string]bool{}
held.mu.Unlock()
rev = r.st.revs[r.key]
asks := held.asks()
r.consider(t, "modules/m")
r.consider(t, "modules/m")
if r.st.revs[r.key] != rev {
t.Errorf("a key whose blob is held was built again")
}
if held.asks() != asks+1 {
t.Errorf("asked the store %d times for two looks at a held key; want once", held.asks()-asks)
}
}
// 1937/AC4 (unit, the builder): artifacts-gone for the module's fingerprint makes the builder ask again, even for a key
// it found held before, and build it again; one for another fingerprint builds nothing.
func TestArtifactsGoneMakesTheBuilderAskAndBuildAgain(t *testing.T) {
r := newRig(t, goBundle)
log := &logged{}
r.b.Logf = log.logf
held := &fakeHeld{gone: map[string]bool{}}
r.b.Stored = held.stored
putBuild(t, r.st, r.key, doneKeyNaming(goneRef))
if v := r.consider(t, "modules/m"); v.State != delivery.BuildDone || len(r.bm.callsOf("build_go_build")) != 0 {
t.Fatalf("a key whose blob is held was built: %+v", v)
}
rev := r.st.revs[r.key]
held.mu.Lock()
held.gone[goneRef] = true // collected after the builder's look; the controller's take-in found it
held.mu.Unlock()
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Fingerprint: "src2:" + strings.Repeat("9", 64), Gone: []string{goneRef}})
r.b.Wait()
if r.st.revs[r.key] != rev || !log.has("a fingerprint the module does not have now") {
t.Fatalf("an event for another fingerprint moved the key (revision %d, was %d): %v", r.st.revs[r.key], rev, log.lines)
}
r.b.ArtifactsGone(context.Background(), delivery.ArtifactsGone{Event: delivery.ArtifactsGoneEvent, Module: "m",
Key: r.key, Fingerprint: fp1, Gone: []string{goneRef}})
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev || len(r.bm.callsOf("build_go_build")) == 0 {
t.Fatalf("build = %+v at revision %d (was %d); want built again on the event", v, r.st.revs[r.key], rev)
}
if !log.has("heard artifacts-gone") || !log.has(r.key, "built again", goneRef) {
t.Errorf("the builder did not say what it heard and did: %v", log.lines)
}
}
@@ -0,0 +1,36 @@
package build
import (
"encoding/json"
"strings"
"sync"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ADR 0593 §2 (F3): the imports a Go build module answers at a build's end reach the done key and the log's end entry,
// where the indexer keeps them on the module.
func importsABuildAnswersReachTheDoneKeyAndTheLog(t *testing.T) {
r := newRig(t, goBundle)
imports := []string{"git.novox.be/novox/mesh-sdk/go", "git.novox.be/novox/mesh-sdk/go/announce"}
r.asks.Answer = func(a delivery.StepAsk) delivery.Step {
step := r.answer(a)
var answer map[string]any
if err := json.Unmarshal(step.Result, &answer); err == nil && a.Language == "go" {
answer["imports"] = imports
step.Result, _ = json.Marshal(answer)
}
return step
}
var logged []delivery.BuildLogEntry
var mu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { mu.Lock(); logged = append(logged, e); mu.Unlock() }
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || strings.Join(v.Imports, ",") != strings.Join(imports, ",") {
t.Fatalf("done key %+v, want the imports", v)
}
if len(logged) != 2 || strings.Join(logged[1].Imports, ",") != strings.Join(imports, ",") || len(logged[0].Imports) != 0 {
t.Fatalf("log %+v, want the imports on the end alone", logged)
}
}
@@ -0,0 +1,12 @@
package build
import (
"context"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Manifests reads a module's manifest by its digest, the key of the indexer's manifests bucket.
type Manifests interface {
Manifest(ctx context.Context, digest string) (catalogue.Manifest, error)
}
@@ -0,0 +1,118 @@
// Package build is the builder's logic (novox/hq ADR 0453 clause 4, ADR 0478), apart from the bus: what a manifest asks
// to be built and where, the claim of a build key, the choice of build nodes, the build modules' long steps and the run of
// one build. The bus is reached through the small interfaces here, so each part is tested with fakes.
package build
import (
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ImageBuildModule is the build module that builds an image artifact (novox/hq ADR 0487): buildah, rootless, on the nodes that
// run it. It has no toolchain language; the builder finds its nodes from the node-engine's have.
const ImageBuildModule = "mesh-build-image"
// NoImageBuilderPrefix opens the error of a build key failed because no node runs ImageBuildModule (ADR 0487 clause 5): such a
// key is not built again as a new revision, only when a node reports the module (ClaimOptions.ImageBuilder) or the fingerprint
// changes.
const NoImageBuilderPrefix = "no image builder:"
// Task is one artifact to build: its name and kind in the manifest, the build module that builds it and the language
// of the toolchain it is built with.
type Task struct {
Artifact string
Kind string
Module string
Language string
// From is, for an upstream artifact, the reference its manifest pins, which the image build module copies (ADR 0498
// clause 4).
From string
}
// Plan is what a manifest asks the builder to build, by artifact name.
type Plan struct{ Tasks []Task }
// Modules are the build modules the plan calls, sorted.
func (p Plan) Modules() []string {
seen := map[string]bool{}
var out []string
for _, t := range p.Tasks {
if !seen[t.Module] {
seen[t.Module] = true
out = append(out, t.Module)
}
}
sort.Strings(out)
return out
}
// NeedsToolchain is whether any task the build module builds is built with a toolchain: a bundle is, an archive is not
// (the indexer gives an archive no language, so a fingerprint names no toolchain for it; ADR 0478 clause 1).
func (p Plan) NeedsToolchain(module string) bool {
for _, t := range p.Tasks {
if t.Module == module && t.Kind != catalogue.ArtifactArchive && t.Kind != catalogue.ArtifactImage && t.Kind != catalogue.ArtifactUpstream {
return true
}
}
return false
}
// Refused is the error of a plan the builder does not build: an artifact of a kind (or of a language) no build module
// builds. It opens delivery.RefusedKind, so its key is not built again until the fingerprint changes.
type Refused struct{ Artifact, What string }
func (r *Refused) Error() string {
return fmt.Sprintf("%s%s (artifact %s): no build module builds it; the builder builds Go and TypeScript bundles, archives, images, upstream copies and TypeScript packages", delivery.RefusedKind, r.What, r.Artifact)
}
// NotPinned is the error of a plan with an upstream artifact whose reference names no digest (ADR 0498 clause 4): it opens
// delivery.NotPinned, so its key is not built again until the fingerprint changes.
type NotPinned struct{ Artifact, From string }
func (n *NotPinned) Error() string {
return fmt.Sprintf("%s%s: %q names no digest (@sha256:), so what would be copied is whatever its tag means today", delivery.NotPinned, n.Artifact, n.From)
}
// PlanOf is the plan of a manifest (ADR 0478 clauses 1 and 2, ADR 0498 clauses 4 and 5): a Go bundle and an archive are
// built by the Go build module, an image by the image build module (ADR 0487), which also copies an upstream image pinned
// by its digest, and a TypeScript bundle or package by the Node one, each in the order of the artifacts' names. A manifest
// with an upstream artifact pinned by no digest, or a bundle or a package of another language, or a kind no build module
// builds, is refused as a whole, naming the first such artifact by name. A manifest with no artifact has an empty plan.
func PlanOf(m catalogue.Manifest) (Plan, error) {
if m.Build == nil {
return Plan{}, nil
}
artifacts := append([]catalogue.Artifact(nil), m.Build.Artifacts...)
sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Name < artifacts[j].Name })
var p Plan
for _, a := range artifacts {
switch {
case a.Kind == catalogue.ArtifactArchive:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.GoBuildModule, Language: "go"})
case a.Kind == catalogue.ArtifactBundle && a.Language == "go":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.GoBuildModule, Language: "go"})
case a.Kind == catalogue.ArtifactBundle && a.Language == "typescript":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.NodeBuildModule, Language: "typescript"})
case a.Kind == catalogue.ArtifactImage:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: ImageBuildModule})
case a.Kind == catalogue.ArtifactUpstream && !strings.Contains(a.From, "@sha256:"):
return Plan{}, &NotPinned{Artifact: a.Name, From: a.From}
case a.Kind == catalogue.ArtifactUpstream:
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: ImageBuildModule, From: a.From})
case a.Kind == catalogue.ArtifactPackage && a.Language == "typescript":
p.Tasks = append(p.Tasks, Task{Artifact: a.Name, Kind: a.Kind, Module: buildrun.NodeBuildModule, Language: "typescript"})
case a.Kind == catalogue.ArtifactBundle:
return Plan{}, &Refused{Artifact: a.Name, What: fmt.Sprintf("bundle of language %q", a.Language)}
case a.Kind == catalogue.ArtifactPackage:
return Plan{}, &Refused{Artifact: a.Name, What: fmt.Sprintf("package of language %q", a.Language)}
default:
return Plan{}, &Refused{Artifact: a.Name, What: a.Kind}
}
}
return p, nil
}
@@ -0,0 +1,96 @@
package build
import (
"errors"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// manifestOf is a manifest holding these artifacts: the plan reads only their name, kind and language.
func manifestOf(artifacts ...catalogue.Artifact) catalogue.Manifest {
return catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: artifacts}}
}
func TestPlanRoutesEachKindToItsBuildModule(t *testing.T) { // 1607/AC2
m := manifestOf(
catalogue.Artifact{Name: "tool", Kind: "bundle", Language: "go"},
catalogue.Artifact{Name: "web", Kind: "bundle", Language: "typescript"},
catalogue.Artifact{Name: "files", Kind: "archive"})
p, err := PlanOf(m)
if err != nil {
t.Fatal(err)
}
want := []Task{
{Artifact: "files", Kind: "archive", Module: buildrun.GoBuildModule, Language: "go"},
{Artifact: "tool", Kind: "bundle", Module: buildrun.GoBuildModule, Language: "go"},
{Artifact: "web", Kind: "bundle", Module: buildrun.NodeBuildModule, Language: "typescript"},
}
if len(p.Tasks) != len(want) {
t.Fatalf("tasks = %+v, want %+v", p.Tasks, want)
}
for i := range want {
if p.Tasks[i] != want[i] {
t.Errorf("task %d = %+v, want %+v", i, p.Tasks[i], want[i])
}
}
if modules := p.Modules(); len(modules) != 2 || modules[0] != buildrun.GoBuildModule || modules[1] != buildrun.NodeBuildModule {
t.Errorf("modules = %v", modules)
}
}
const pinned = "nats:2.11.17-alpine@sha256:3333333333333333333333333333333333333333333333333333333333333333"
func TestPlanSendsAPinnedUpstreamToTheImageBuildModulesCopyAndRefusesAnUnpinnedOne(t *testing.T) { // B3, ADR 0498 clause 4
p, err := PlanOf(manifestOf(catalogue.Artifact{Name: "ok", Kind: "archive"}, catalogue.Artifact{Name: "up", Kind: "upstream", From: pinned}))
if err != nil || len(p.Tasks) != 2 {
t.Fatalf("plan %+v, error %v", p, err)
}
if up := p.Tasks[1]; up.Artifact != "up" || up.Kind != "upstream" || up.Module != ImageBuildModule || up.From != pinned || up.Language != "" {
t.Errorf("task = %+v, want the upstream sent to %s with its pinned reference and no toolchain", up, ImageBuildModule)
}
if p.NeedsToolchain(ImageBuildModule) {
t.Error("a copy needs no toolchain")
}
_, err = PlanOf(manifestOf(catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine"}))
if err == nil || !strings.HasPrefix(err.Error(), delivery.NotPinned+"up") || !delivery.FailedUntilChanged(err.Error()) {
t.Errorf("an upstream with no digest: error = %v, want it to open %q naming the artifact", err, delivery.NotPinned)
}
}
func TestPlanRefusesAPackageOfAnotherLanguageByName(t *testing.T) { // ADR 0498 clause 5
_, err := PlanOf(manifestOf(catalogue.Artifact{Name: "ok", Kind: "archive"}, catalogue.Artifact{Name: "pkg", Kind: "package", Language: "python"}))
var r *Refused
if err == nil || !asRefused(err, &r) || !strings.HasPrefix(err.Error(), delivery.RefusedKind) || !strings.Contains(err.Error(), "python") || !strings.Contains(err.Error(), "pkg") {
t.Errorf("error = %v, want it to open %q naming the artifact and its language", err, delivery.RefusedKind)
}
}
func TestPlanRefusesABundleOfAnotherLanguageAndAModuleWithNoArtifact(t *testing.T) {
if _, err := PlanOf(manifestOf(catalogue.Artifact{Name: "x", Kind: "bundle", Language: "python"})); err == nil || !strings.HasPrefix(err.Error(), delivery.RefusedKind) {
t.Errorf("a python bundle: error = %v, want a refusal", err)
}
p, err := PlanOf(manifestOf())
if err != nil || len(p.Tasks) != 0 {
t.Errorf("no artifact: plan %+v error %v, want an empty plan", p, err)
}
}
func asRefused(err error, target **Refused) bool { return errors.As(err, target) }
func TestPlanSendsAnImageToMeshBuildImageWithNoToolchain(t *testing.T) { // 1705a, ADR 0487 clause 5
p, err := PlanOf(manifestOf(catalogue.Artifact{Name: "img", Kind: "image"}, catalogue.Artifact{Name: "files", Kind: "archive"}))
if err != nil || len(p.Tasks) != 2 {
t.Fatalf("plan %+v, error %v", p, err)
}
if p.Tasks[1].Artifact != "img" || p.Tasks[1].Module != ImageBuildModule {
t.Errorf("task = %+v, want the image on %s", p.Tasks[1], ImageBuildModule)
}
if p.NeedsToolchain(ImageBuildModule) {
t.Error("an image needs no toolchain")
}
}
@@ -0,0 +1,115 @@
package build
import (
"encoding/json"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// **The tool builder_seat_queue reads a build seat's stream as the builder reads it** (novox/hq issue 1945 note 62; the
// instruction "tools are how a module is spoken to"): the messages waiting on SEAT_NODE_BUILD_<LANGUAGE>, bounded, each
// with its ask, its age, the state of the step a build node wrote for it, and whether the builder counts it as still
// waiting (AskSeen.Waits) or as given up on. Only the builder's own account may read the stream (issue 1910): the tools
// bundle writes the read into the builder's state queues, keyed by the language, which the builder watches and answers
// there, as builder_ask_again does through reasks.
const (
SeatQueueTool = "builder_seat_queue"
// QueuesState is the builder's state of queue reads: the bucket mesh-delivery-builder_queues, keyed by the language.
QueuesState = "queues"
QueuesBucket = delivery.BuilderModule + "_" + QueuesState
// DefaultQueueLimit and MaxQueueLimit bound the messages read of each kind.
DefaultQueueLimit = 50
MaxQueueLimit = 500
)
// QueueNote says what the read cannot show.
const QueueNote = "the builder may not read the seat workers' delivery counts or ack floors ($JS.API.CONSUMER.INFO.<stream>.<worker> is " +
"granted to the delivery view alone): a message counts as given up on when no node wrote its step and a node took an ask of its kind " +
"asked after it; nats.nats_streams shows the workers' pending and waiting pulls"
// QueueAsk is a value of the queues state: a read asked (limit, who, when) and, once the builder read the stream, its answer.
type QueueAsk struct {
V int `json:"v"`
Limit int `json:"limit"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Answered time.Time `json:"answered,omitzero"`
Error string `json:"error,omitempty"`
Queue *SeatQueue `json:"queue,omitempty"`
}
// SeatQueue is what the builder read of one build seat's stream.
type SeatQueue struct {
Language string `json:"language"`
Stream string `json:"stream"`
Messages uint64 `json:"messages"`
FirstSeq uint64 `json:"first_seq"`
LastSeq uint64 `json:"last_seq"`
Kinds []KindQueue `json:"kinds"`
Note string `json:"note"`
}
// KindQueue is one kind's messages on the stream, oldest first: how many the builder counts as waiting, the ask asked
// last a build node took of the kind (or of one pulled after it), and each message.
type KindQueue struct {
Kind string `json:"kind"`
Read int `json:"read"`
Waiting int `json:"waiting"`
GivenUp int `json:"given_up_or_taken"`
Last *Taken `json:"last_taken,omitempty"`
Messages []QueuedState `json:"messages"`
}
// QueuedState is one message as the tool shows it.
type QueuedState struct {
Queued
AgeSeconds int64 `json:"age_s"`
Waits bool `json:"waits"`
}
// NewKindQueue is seen's messages ahead as one kind's queue at now.
func NewKindQueue(kind string, seen AskSeen, now time.Time) KindQueue {
k := KindQueue{Kind: kind, Read: len(seen.Ahead), Last: seen.Last}
for _, q := range seen.Ahead {
w := seen.Waits(q)
if w {
k.Waiting++
} else {
k.GivenUp++
}
k.Messages = append(k.Messages, QueuedState{Queued: q, AgeSeconds: int64(now.Sub(q.PutAt) / time.Second), Waits: w})
}
return k
}
// NewQueueAsk is the value builder_seat_queue writes for a language: its key and the value.
func NewQueueAsk(language string, limit int, by string, at time.Time) (string, []byte, error) {
if _, err := delivery.AskStream(language); err != nil {
return "", nil, fmt.Errorf("language: %w", err)
}
if limit <= 0 {
limit = DefaultQueueLimit
}
data, err := json.Marshal(QueueAsk{V: 1, Limit: min(limit, MaxQueueLimit), By: by, At: at.UTC()})
return language, data, err
}
// DecodeQueueAsk reads a queues value.
func DecodeQueueAsk(data []byte) (QueueAsk, error) {
var a QueueAsk
if err := json.Unmarshal(data, &a); err != nil {
return a, err
}
if a.V != 1 {
return a, fmt.Errorf("a queues value of version %d, not 1", a.V)
}
return a, nil
}
// LanguageOf is the language a tool's seat or language argument names: go, node or image, or node-build-<language>.
func LanguageOf(arg string) string {
return strings.TrimPrefix(strings.TrimSpace(arg), "node-build-")
}
@@ -0,0 +1,403 @@
package build
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// R4e: references in the key (novox/hq ADR 0498), upstream copies, packages published from the trunk, the package wait
// (ADR 0500 clause 6) and one node per fingerprint (ADR 0478 clause 4).
func TestACandidatesBundleKeyNamesItsReferenceAndTheTrunkBuildsItNoMore(t *testing.T) { // B1, ADR 0498 clause 3 and its check 3
r := newRig(t, goBundle, archiveA)
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
v := valueOf(t, r.st, r.key)
if v.State != delivery.BuildDone {
t.Fatalf("build = %+v, want done", v)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("a candidate's bundle and archive name no reference: %v (%+v)", missing, v.Artifacts)
}
rev := r.st.revs[r.key]
r.bm.mu.Lock()
r.bm.calls = nil
r.bm.mu.Unlock()
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the merge: the trunk now names the same fingerprint
r.b.Wait()
if n := len(r.bm.calls); n != 0 || r.st.revs[r.key] != rev {
t.Errorf("the trunk built its candidate's fingerprint again: %d calls, revision %d (was %d)", n, r.st.revs[r.key], rev)
}
}
func TestABundleWhosePushAnsweredNoReferenceFailsNotStoredAndIsBuiltAgain(t *testing.T) { // B1, ADR 0498 clauses 1 and 2
r := newRig(t, goBundle)
r.bm.noRef["ace/tool"] = true
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "tool") {
t.Fatalf("build = %+v, want failed opening %q and naming the artifact", v, delivery.NotStored)
}
delete(r.bm.noRef, "ace/tool")
if v = r.consider(t, "modules/m"); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want a not-stored key built again as a new revision", v)
}
}
func TestAPushRefusedByTheStoreFailsTheKeyOpeningNotStored(t *testing.T) { // B1, ADR 0498 clause 2
r := newRig(t, goBundle)
r.bm.fails["ace/build_go_build"] = delivery.NotStored + "tool: the artifact store answered 507 insufficient storage"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || !strings.Contains(v.Error, "507") || !strings.Contains(v.Error, "ace") {
t.Fatalf("build = %+v, want failed opening %q with the store's words and the node", v, delivery.NotStored)
}
}
func TestOnlyATrunkKeysBuildsAreAskedToPublish(t *testing.T) { // B1, the lead's interface (a): publish only for a trunk key
r := newRig(t, goBundle)
r.consider(t, "modules/m")
for _, c := range r.bm.callsOf("build_go_build") {
if c.body.(map[string]any)["publish"] != true {
t.Errorf("trunk build args = %+v, want publish", c.body)
}
}
c := newRig(t, goBundle)
trunkModule(t, c)
c.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
c.b.Wait()
for _, call := range c.bm.callsOf("build_go_build") {
if _, ok := call.body.(map[string]any)["publish"]; ok {
t.Errorf("candidate build args = %+v, want no publish", call.body)
}
}
}
// putBuild writes a builder key's value as the store would hold it.
func putBuild(t *testing.T, st *memStore, key string, b delivery.BuilderBuild) uint64 {
t.Helper()
data, err := delivery.EncodeBuilderBuild(b)
if err != nil {
t.Fatal(err)
}
st.mu.Lock()
defer st.mu.Unlock()
return st.put(key, data)
}
func TestADoneKeyWrittenBeforeR4eIsBuiltAgainOnceAsANewRevision(t *testing.T) { // B2, ADR 0498 clause 8
r := newRig(t, goBundle, archiveA)
rev := putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: manifestD, Step: "built mesh-build-go on ace and novox",
Artifacts: []delivery.BuiltArtifact{{Name: "files", Kind: "archive", Digest: digest}, {Name: "tool", Kind: "bundle", Digest: digest}}})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || r.st.revs[r.key] <= rev {
t.Fatalf("build = %+v at revision %d (was %d), want built again as a new revision", v, r.st.revs[r.key], rev)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("the new revision names no reference for %v", missing)
}
rev = r.st.revs[r.key]
r.b.tried = map[string]bool{} // a builder started anew looks again
r.consider(t, "modules/m")
if r.st.revs[r.key] != rev {
t.Errorf("a key whose artifacts all name references was built again (revision %d, was %d): once only", r.st.revs[r.key], rev)
}
}
func TestAPinnedUpstreamIsCopiedOnceOnOneImageNodeWithNoFetchAndItsKeyNamesTheReference(t *testing.T) { // B4, ADR 0498 clause 4
up := catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest}
r := newRig(t, up)
r.puller = "ace"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 {
t.Fatalf("build = %+v, want done with the copy", v)
}
if a := v.Artifacts[0]; a.Kind != "upstream" || a.Digest != digest || a.Reference != "registry.example/m/up@"+digest {
t.Errorf("artifact = %+v, want the upstream's digest and the registry's reference", a)
}
copies := r.bm.callsOf("build_image_copy")
if len(copies) != 1 {
t.Fatalf("%d copies, want one on one node", len(copies))
}
if args := copies[0].body.(map[string]any); args["module"] != "m" || args["artifact"] != "up" || args["from"] != up.From || len(args) != 3 {
t.Errorf("copy args = %+v, want the module's name, the artifact and the pinned reference", args)
}
if n := len(r.bm.callsOf("build_image_fetch")) + len(r.bm.callsOf("build_image_build")); n != 0 {
t.Errorf("%d fetch or build calls for a copy, want none", n)
}
}
func TestAnUpstreamBesideAnImageIsCopiedAndTheImageFetchedAndBuiltOnTheSameNode(t *testing.T) { // B4
r := newRig(t, imageArt, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.puller = "ace"
v := r.consider(t, "modules/m")
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, missing references %v", v, missing)
}
if f, b, c := len(r.bm.callsOf("build_image_fetch")), len(r.bm.callsOf("build_image_build")), len(r.bm.callsOf("build_image_copy")); f != 1 || b != 1 || c != 1 {
t.Errorf("fetch %d, build %d, copy %d; want one each", f, b, c)
}
}
func TestACopyThatFailsNotStoredFailsTheKeyAndIsBuiltAgain(t *testing.T) { // B4, ADR 0498 clause 2
r := newRig(t, catalogue.Artifact{Name: "up", Kind: "upstream", From: "nats:2.11.17-alpine@" + digest})
r.puller = "ace"
r.bm.fails["ace/build_image_copy"] = delivery.NotStored + "up: the registry answered 500"
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.NotStored) || delivery.FailedUntilChanged(v.Error) {
t.Fatalf("build = %+v, want failed 'not stored:' and built again", v)
}
delete(r.bm.fails, "ace/build_image_copy")
if v = r.consider(t, "modules/m"); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want done on the new revision", v)
}
}
var tsPackage = catalogue.Artifact{Name: "lib", Kind: "package", Language: "typescript"}
func TestATypeScriptPackageIsBuiltByTheNodeModuleAndPublishedOnlyForTheTrunk(t *testing.T) { // B5, ADR 0498 clause 5, ADR 0500 clause 4
c := newRig(t, tsPackage)
c.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, c)
c.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
c.b.Wait()
v := valueOf(t, c.st, c.key)
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "package" || v.Artifacts[0].Reference != "" {
t.Fatalf("candidate build = %+v, want done with the packed digest and no reference", v)
}
for _, call := range c.bm.callsOf("build_node_build") {
if _, ok := call.body.(map[string]any)["publish"]; ok {
t.Errorf("a candidate's package build was asked to publish: %+v", call.body)
}
}
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
v = r.consider(t, "modules/m")
if v.State != delivery.BuildDone || v.Artifacts[0].Reference != "@novox/mesh-sdk@0.1.15" {
t.Fatalf("trunk build = %+v, want done naming the published version", v)
}
if n := len(r.bm.callsOf("build_node_build")); n == 0 {
t.Error("the package was not built by the Node build module")
}
}
func TestAVersionTakenFailsTheKeyUntilTheFingerprintChanges(t *testing.T) { // B5, ADR 0498 clause 5
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
r.bm.fails["ace/build_node_build"] = delivery.VersionTaken + "@novox/mesh-sdk@0.1.15 is published with another packed file"
r.bm.fails["novox/build_node_build"] = r.bm.fails["ace/build_node_build"]
v := r.consider(t, "modules/m")
if v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.VersionTaken) || !strings.Contains(v.Error, "0.1.15") {
t.Fatalf("build = %+v, want failed opening %q naming the version", v, delivery.VersionTaken)
}
writes := r.st.writes
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Errorf("a version-taken key was built again (%d writes, was %d)", r.st.writes, writes)
}
}
func TestACandidatesPackageIsPublishedOnceWhenTheTrunkNamesItsFingerprint(t *testing.T) { // B6, ADR 0498 clause 6
r := newRig(t, tsPackage, tsBundle)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Fatalf("candidate build = %+v", v)
}
rev := r.st.revs[r.key]
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m")) // the merge names the candidate's fingerprint
r.b.Wait()
v := valueOf(t, r.st, r.key)
if r.st.revs[r.key] <= rev {
t.Fatalf("the trunk did not build the unpublished package again (revision %d, was %d)", r.st.revs[r.key], rev)
}
if ok, missing := v.Referenced(); !ok {
t.Fatalf("build = %+v, missing references %v", v, missing)
}
rev = r.st.revs[r.key]
r.b.tried = map[string]bool{}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if r.st.revs[r.key] != rev {
t.Error("a published package was built again: once only")
}
}
func TestAPublishThatFailsVersionTakenOnTheTrunkKeepsThePrefixFirst(t *testing.T) { // B6, ADR 0498 clauses 5 and 6
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
r.b.Candidate(context.Background(), "feat%2Fx", candidateValue(t, "feat/x", 7, headX))
r.b.Wait()
for _, n := range []string{"ace", "novox"} {
r.bm.fails[n+"/build_node_build"] = delivery.VersionTaken + "@novox/mesh-sdk@0.1.15 is published with another packed file"
}
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildFailed || !strings.HasPrefix(v.Error, delivery.VersionTaken) {
t.Errorf("build = %+v, want failed opening %q", v, delivery.VersionTaken)
}
}
// sdkModule adds to the rig a module "sdk" whose manifest (digest b) holds the TypeScript package lib, published as ref.
func sdkModule(t *testing.T, r *rig, ref string) []byte {
t.Helper()
r.mf.byDigest[mdigest('b')] = catalogue.Manifest{Module: "sdk", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{tsPackage}}}
r.bm.packages["lib"] = ref
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: "8a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP2: "src2:" + hex64('5'),
Manifest: mdigest('b'), Path: ".", Toolchains: agreedToolchains})
if err != nil {
t.Fatal(err)
}
return data
}
func TestAModuleWhoseLockNamesAnUnpublishedPackageWaitsQueuedAndIsBuiltOncePublished(t *testing.T) { // B7, ADR 0500 clause 6
const ref = "@novox/mesh-sdk@0.1.15"
r := newRig(t, tsBundle)
for _, n := range []string{"ace", "novox"} {
r.bm.fails[n+"/build_node_fetch"] = waitsFor + ref + ": the lock names a version the registry does not hold"
}
v := r.consider(t, "modules/m")
if v.State != delivery.BuildQueued || v.Waits != ref || v.Error != "" {
t.Fatalf("build = %+v, want queued, waiting for %s, not failed", v, ref)
}
delete(r.bm.fails, "ace/build_node_fetch")
delete(r.bm.fails, "novox/build_node_fetch")
writes := r.st.writes
r.b.tried = map[string]bool{}
r.consider(t, "modules/m")
if r.st.writes != writes {
t.Fatalf("a key waiting for an unpublished package was claimed (%d writes, was %d)", r.st.writes, writes)
}
r.b.Module(context.Background(), "sdk", sdkModule(t, r, ref)) // the package's trunk key is built and published
r.b.Wait()
if v = valueOf(t, r.st, r.key); v.State != delivery.BuildDone || v.Waits != "" {
t.Errorf("build = %+v, want built once the package it waits for is published", v)
}
}
func TestAKeyWaitingForAPackageAlreadyPublishedIsBuiltWhenTheBuilderStarts(t *testing.T) { // B7: the scan at start
const ref = "@novox/mesh-sdk@0.1.15"
r := newRig(t, tsBundle)
putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildQueued, By: "novox.builder", Manifest: manifestD, Waits: ref, Step: "waits"})
sdkKey, _ := delivery.BuilderBuildKey("sdk", "src2:"+hex64('5'))
putBuild(t, r.st, sdkKey, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: mdigest('b'), Step: "built",
Artifacts: []delivery.BuiltArtifact{{Name: "lib", Kind: "package", Digest: digest, Reference: ref}}})
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
if err := r.b.Sweep(context.Background()); err != nil {
t.Fatal(err)
}
r.b.Wait()
if v := valueOf(t, r.st, r.key); v.State != delivery.BuildDone {
t.Errorf("build = %+v, want built: the package it waits for was published before the builder started", v)
}
}
func TestTheFetchOfAModuleNamingItsPathsCarriesItsDirectoryAndThosePaths(t *testing.T) { // the lead's interface (c), ADR 0499 clause 4
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
paths := map[string]string{"src": hex40('1'), "package.json": hex40('2')}
// The named tree the indexer computes from those paths (its own, ADR 0593 §2), given as data: the builder reads it,
// never computes it.
const tree = delivery.NamedTreePrefix + "5b5e0f3b6f3c1f2a9d8e7c6b5a4938271605f4e3d2c1b0a9f8e7d6c5b4a39281"
data, err := delivery.EncodeModule(delivery.Module{Repository: "novox/mesh-sdk", Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", FP2: fp1, Manifest: manifestD, Path: ".",
Toolchains: agreedToolchains, Inputs2: &delivery.KeptInputs2{Tree: tree, Paths: paths}})
if err != nil {
t.Fatal(err)
}
r.b.Module(context.Background(), "m", data)
r.b.Wait()
f := r.bm.callsOf("build_node_fetch")
if len(f) == 0 {
t.Fatal("no fetch")
}
req, err := buildstep.ParseFetchRequest(f[0].body.(map[string]any))
if err != nil || req.Dir != "." || strings.Join(req.Paths, ",") != "package.json,src" || req.Commit != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" {
t.Errorf("fetch request = %+v, %v; want dir . and the kept paths, sorted", req, err)
}
r2 := newRig(t, goBundle)
r2.consider(t, "modules/m")
if a := r2.bm.callsOf("build_go_fetch")[0].body.(map[string]any); a["dir"] != nil || a["paths"] != nil {
t.Errorf("fetch args = %+v, want no dir or paths for a module naming none", a)
}
}
func hex40(c byte) string { return strings.Repeat(string(c), 40) }
func TestOneNodeBuildsEachFingerprint(t *testing.T) { // B9, ADR 0478 clause 4: "From R4e one node builds each fingerprint"
r := newRig(t, goBundle, archiveA, tsBundle) // ace and novox both qualify for both build modules
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || strings.Contains(v.Step, " and ") || strings.HasPrefix(v.Step, "one node (") {
t.Fatalf("build = %+v, want done on one node of each build module, unmarked", v)
}
if f, b := len(r.bm.callsOf("build_go_fetch")), len(r.bm.callsOf("build_go_build")); f != 1 || b != 2 {
t.Errorf("go fetches %d, builds %d; want one fetch and one build of each artifact, on one node", f, b)
}
if f := len(r.bm.callsOf("build_node_fetch")); f != 1 {
t.Errorf("node fetches %d, want one", f)
}
}
func TestAKeyDoneOnOneNodeIsNeverBuiltAgainForASecondNode(t *testing.T) { // B9: the second-node retake is gone
r := newRig(t, goBundle)
putBuild(t, r.st, r.key, delivery.BuilderBuild{State: delivery.BuildDone, By: "novox.builder", Manifest: manifestD,
Step: "one node (mesh-build-go): built mesh-build-go on one node, novox",
Artifacts: []delivery.BuiltArtifact{{Name: "tool", Kind: "bundle", Digest: digest, Reference: "artifact-store://m/tool/blobs/" + digest}}})
writes := r.st.writes
r.b.Module(context.Background(), "m", moduleValue(t, "modules/m"))
r.b.NodesChanged(context.Background())
r.b.Wait()
if r.st.writes != writes || len(r.bm.calls) != 0 {
t.Errorf("a key done on one node was built again (%d writes, was %d; %d calls)", r.st.writes, writes, len(r.bm.calls))
}
}
func TestACandidatesFetchNamesTheNamedPathsTheIndexerGaveIt(t *testing.T) { // ADR 0499 clause 4, mesh-sdk CandidateModule.Paths
r := newRig(t, tsPackage)
r.bm.packages["lib"] = "@novox/mesh-sdk@0.1.15"
trunkModule(t, r)
pulls := []delivery.CandidatePull{{Head: delivery.Head{Repository: "novox/mesh-catalog", Number: 7, Head: headX}, Modules: []string{"m"},
Fps: map[string]delivery.CandidateModule{"m": {FP2: fp1, Manifest: mdigest('a'), Paths: map[string]string{"src": hex40('1'), "package.json": hex40('2')}}}}}
hash, _ := delivery.HeadsHash([]delivery.Head{pulls[0].Head})
data, err := delivery.EncodeCandidate(delivery.Candidate{V: delivery.Version, Group: "feat/x", HeadsHash: hash, Pulls: pulls, FPSet: delivery.FingerprintSet(pulls)})
if err != nil {
t.Fatal(err)
}
r.b.Candidate(context.Background(), "feat%2Fx", data)
r.b.Wait()
f := r.bm.callsOf("build_node_fetch")
if len(f) == 0 {
t.Fatal("no fetch")
}
req, err := buildstep.ParseFetchRequest(f[0].body.(map[string]any))
// The base is the trunk head the candidate was previewed onto, never the module value's commit (an older one).
if err != nil || req.Dir != "modules/m" || strings.Join(req.Paths, ",") != "package.json,src" || req.Base != trunkHead {
t.Errorf("fetch request = %+v, %v; want the trunk head %.8s as the base, the module's directory and its named paths", req, err, trunkHead)
}
}
// 1886/AC5: the builder tells who answered a reference: the build module whose step named one, and none for a step that
// answered a digest alone (a build module from before R4e).
func TestTheBuilderTellsWhichBuildModuleAnsweredAReference(t *testing.T) {
r := newRig(t, goBundle)
var answered []string
r.b.Answered = func(module string) { answered = append(answered, module) }
r.bm.noRef["ace/tool"] = true
r.consider(t, "modules/m")
if len(answered) != 0 {
t.Fatalf("a step that answered no reference was told as one: %v", answered)
}
delete(r.bm.noRef, "ace/tool")
r.consider(t, "modules/m")
if len(answered) != 1 || answered[0] != "mesh-build-go" {
t.Fatalf("the build module that answered a reference: %v, want mesh-build-go", answered)
}
}
@@ -0,0 +1,84 @@
package build
import (
"context"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
func haveValue(t *testing.T, phase delivery.Phase, health delivery.Health) []byte {
t.Helper()
b, err := delivery.EncodeHave(delivery.Have{Phase: phase, Health: health})
if err != nil {
t.Fatal(err)
}
return b
}
// 1. A module of an archive only: the real indexer gives an archive no language (inputs.go languagesOf), so its
// fingerprint names no toolchain, and the builder builds it on the build module's nodes with no toolchain agreed.
func TestAnArchiveOnlyModuleIsBuiltAlthoughItsFingerprintNamesNoToolchain(t *testing.T) { // review round 4, 1
r := newRig(t, archiveA)
v := func() delivery.BuilderBuild {
r.b.Module(context.Background(), "m", moduleValueWith(t, "modules/m", nil))
r.b.Wait()
if _, ok := r.st.vals[r.key]; !ok {
t.Fatal("no build key was written for the archive-only module")
}
return valueOf(t, r.st, r.key)
}()
if v.State != delivery.BuildDone || len(v.Artifacts) != 1 || v.Artifacts[0].Kind != "archive" || len(v.Toolchains) != 0 {
t.Fatalf("build = %+v, want done with the archive and no toolchain named", v)
}
if n := len(r.bm.callsOf("build_go_build")); n != 1 {
t.Errorf("%d go builds, want one, on one node (ADR 0478 clause 4)", n)
}
}
func TestABundleAndAnArchiveOfOneBuildModuleStillNeedTheAgreedToolchain(t *testing.T) { // review round 4, 1
p, err := PlanOf(catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{goBundle, archiveA}}})
if err != nil || !p.NeedsToolchain(buildrun.GoBuildModule) {
t.Errorf("a plan with a Go bundle: %v, %v", p.NeedsToolchain(buildrun.GoBuildModule), err)
}
p, _ = PlanOf(catalogue.Manifest{Module: "m", Build: &catalogue.Build{Artifacts: []catalogue.Artifact{archiveA}}})
if p.NeedsToolchain(buildrun.GoBuildModule) {
t.Error("a plan of an archive needs a toolchain")
}
}
func TestClaimLeavesAKeyFailedForDifferingDigestsAlone(t *testing.T) { // review round 4, 2
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildFailed, Error: buildrun.DigestsDifferPrefix + " the nodes built tool to different digests: ace x; novox y"})
writes := st.writes
got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{TakeOver: true})
if err != nil || got != nil || st.writes != writes {
t.Errorf("claim = %v, %v, %d writes (was %d), want none and no write", got, err, st.writes, writes)
}
}
// 6. A key found building is taken over, whoever holds it.
func TestABuildLeftBuildingByAnyBuilderIsTakenOverAtStart(t *testing.T) { // review round 4, 6; 1607/AC3, ADR 0478 clauses 5, 6
st := newMem()
seed(t, st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "novox.a"})
if got, err := (Claimer{Store: st, By: "novox.a"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); err != nil || got == nil {
t.Errorf("own building key: claim = %v, %v, want it taken again", got, err)
}
if got, err := (Claimer{Store: st, By: "ace.b"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{TakeOver: true}); err != nil || got == nil || got.Build.By != "novox.a" {
t.Errorf("another node's building key at start: claim = %v, %v, want it taken over", got, err)
}
if got, _ := (Claimer{Store: st, By: "ace.b"}).Claim(context.Background(), "m", fp1, "", ClaimOptions{}); got != nil {
t.Errorf("another builder's building key after the start was taken: a running builder's key is its own")
}
}
func TestABuilderFindingAKeyBuildingByAnotherNodeBuildsItOnStart(t *testing.T) { // review round 4, 6
r := newRig(t, goBundle)
seed(t, r.st, delivery.BuilderBuild{State: delivery.BuildBuilding, By: "ace.builder", Manifest: manifestD})
v := r.consider(t, "modules/m")
if v.State != delivery.BuildDone || v.By != "novox.builder" {
t.Errorf("build = %+v, want it taken over and done by novox.builder", v)
}
}
@@ -0,0 +1,986 @@
package build
import (
"context"
"encoding/json"
"errors"
"fmt"
"maps"
"slices"
"sort"
"strconv"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ToolchainEntries are the nodes' toolchain entries of a language (buildrun.ToolchainSet).
type ToolchainEntries interface {
Entries(language string) []delivery.NodeToolchain
}
// Builder builds, dark, every new-form fingerprint the indexer writes (novox/hq ADR 0453 clause 4, ADR 0478): it is
// handed each modules value as the watch of the indexer's bucket sees it, and for each module with a fingerprint and an
// artifact to build claims the fingerprint's build key, builds it by asking the build seats (one ask per build module,
// pulled by whichever build node has a free slot: ADR 0525), and writes the key done or failed. It holds no timer; a build starts from a modules value or a
// change of the toolchains.
type Builder struct {
Claimer Claimer
// Asks puts each build module's part of a build as one ask on its build seat's stream and waits for its step (novox/hq
// ADR 0525 §1, §3, §6; issue 1887 A4): the builder chooses no node, whichever build node pulls the ask builds it.
Asks interface {
Put(ctx context.Context, a delivery.StepAsk) error
Wait(ctx context.Context, a delivery.StepAsk) (delivery.Step, error)
}
// Held says whether an ask a key names is on its stream or a build node wrote a step for it (issue 1910): a key taken
// over at start whose ask is neither was left by a builder that stopped between naming the ask and putting it, and
// the ask is put then, under the same id. Nil takes every named ask as held.
Held interface {
Held(ctx context.Context, a delivery.StepAsk) (bool, error)
}
Toolchains ToolchainEntries
Manifests Manifests
// Limit is how many builds the builder has asked at once; their asks wait in the build seats' streams, which are the
// backlog (ADR 0525 §9), not here. 256 when 0.
Limit int
Logf func(format string, args ...any)
// Answered is told the build module whose step answered a reference (issue 1886): what the builder says it writes
// rests on it. Nil tells nobody.
Answered func(buildModule string)
// Stored answers which references the artifact store no longer holds (the seat mesh-artifact-sweep's verb held; issue 1937): a done
// key of the trunk's fingerprint is asked before it is left done (ClaimOptions.Stored). Nil asks nothing.
Stored func(ctx context.Context, references []string) (gone []string, err error)
// Trunk answers the head of a repository in the gate's source revision rev (source.main at that revision): the trunk a
// candidate's preview is merged onto (Candidate.Base). Nil builds no candidate's fingerprint, and says why.
Trunk func(ctx context.Context, rev uint64, repository string) (string, error)
// Gone says why an ask with no ended step can no longer be answered, or "" when it can (issue 1945): asked once the
// ask has waited CheckAfter, and again each CheckAfter it goes on waiting. An ask that cannot be answered is asked
// again under a new id, at most Askings times. Nil, or CheckAfter 0, never asks it.
Gone interface {
Gone(ctx context.Context, a delivery.StepAsk) (string, error)
}
CheckAfter time.Duration
// Askings is how many times one build module's part of a build is asked, the first included, before its key is failed
// with why; DefaultAskings when 0.
Askings int
// DeclineBound and DeclinePause bound and pace the asks build nodes off the agreed toolchain declined (ADR 0584 clause 2):
// they are no askings. DefaultDeclineBound and DefaultDeclinePause when 0; a negative DeclinePause puts it again at once.
DeclineBound, DeclinePause time.Duration
// Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody.
Said func(AskedAgain)
// Log is told each build's start and end, the entries of the builder's log (novox/hq ADR 0571 §1). Nil tells nobody.
Log func(delivery.BuildLogEntry)
// Now is the builder's clock: of the log, and of the decline bound (DeclineBound runs from the first decline); nil is
// time.Now. Node is the node the builder runs on, which asked the build.
Now func() time.Time
Node string
once sync.Once
sem chan struct{}
wg sync.WaitGroup
mu sync.Mutex
known map[string]delivery.Module // by modules key
inflight map[string]bool // by build key
again map[string]delivery.Module // by build key: the newest value that arrived while the key was being built
plans map[string]planned // by manifest digest
cands map[string]delivery.Candidate // by candidates key (candidates.go)
via map[string]via // by build key: a candidate's fetch (candidates.go)
trunkFP map[string]bool // by build key: a trunk module value carries this fingerprint
tried map[string]bool // by build key: looked at since this process started
// published are the package versions `<name>@<version>` a done key of this bucket names as a package's reference: a key
// waiting for one is built once it is here (ADR 0500 clause 6).
published map[string]bool
// held are the done keys' references the store answered held, by `<key> <references>`, asked once per process: a
// blob collected later is found by the take-in, whose artifacts-gone forgets the key here (issue 1937).
held map[string]bool
// waiting are the lanes that wait for an ask's step, by build key and language: AskAgain asks through them.
waiting map[string]map[string]chan string
// hands are the requests to build a key again by hand that wait for the claim and its asks (BuildAgain), by build key.
hands map[string]*handWatch
}
type planned struct {
plan Plan
err error
}
func (b *Builder) init() {
b.once.Do(func() {
limit := b.Limit
if limit <= 0 {
limit = 256
}
b.sem = make(chan struct{}, limit)
b.cands, b.via, b.trunkFP = map[string]delivery.Candidate{}, map[string]via{}, map[string]bool{}
b.known, b.inflight, b.again, b.plans, b.tried = map[string]delivery.Module{}, map[string]bool{}, map[string]delivery.Module{}, map[string]planned{}, map[string]bool{}
b.published, b.held, b.waiting = map[string]bool{}, map[string]bool{}, map[string]map[string]chan string{}
b.hands = map[string]*handWatch{}
})
}
func (b *Builder) logf(format string, args ...any) {
if b.Logf != nil {
b.Logf(format, args...)
}
}
// Wait returns when every build started so far has ended.
func (b *Builder) Wait() { b.wg.Wait() }
// Module takes one value of the indexer's modules bucket in (the key is the module's name) and builds it if it is
// to be built. A value that is not a module is said and left.
func (b *Builder) Module(ctx context.Context, key string, value []byte) {
b.init()
m, err := delivery.DecodeModule(value)
if err != nil {
b.logf("modules.%s holds what is not a module, left: %v", key, err)
return
}
b.mu.Lock()
b.known[key] = m
if bk, err := delivery.BuilderBuildKey(key, m.FP2); err == nil && m.FP2 != "" {
// A fingerprint the trunk carries is built from the trunk's commit, whoever asked for it first.
b.trunkFP[bk] = true
delete(b.via, bk)
}
b.mu.Unlock()
b.consider(ctx, key, m)
}
// NodesChanged is Reconsider after a toolchain moved, so a build that waited for the nodes to agree on one may go on.
func (b *Builder) NodesChanged(ctx context.Context) { b.Reconsider(ctx) }
// Sweep reads, at start, every key of the bucket: it keeps the package versions done keys name as published (a key
// waiting for one is then looked at again, ADR 0500 clause 6), and closes every key left building for a fingerprint its
// module no longer has (issue 1660, ADR 0478 clause 6): the key can never be built, since nothing asks for it, and would
// stay building for ever. It fails the key naming who left it. A key of the fingerprint the module has now is not
// touched: the build takes it over. Call it once the modules bucket has been read, so every module is known.
func (b *Builder) Sweep(ctx context.Context) error {
b.init()
keys, err := b.Claimer.Store.Keys(ctx)
if err != nil {
return err
}
b.mu.Lock()
current := map[string]bool{}
for name, m := range b.known {
if m.FP2 != "" {
if k, err := delivery.BuilderBuildKey(name, m.FP2); err == nil {
current[k] = true
}
}
}
b.mu.Unlock()
newly := false
for _, key := range keys {
value, rev, found, err := b.Claimer.Store.Get(ctx, key)
if err != nil || !found {
continue
}
cur, err := delivery.DecodeBuilderBuild(value)
if err != nil {
continue
}
newly = b.notePublished(cur) || newly
if current[key] || cur.State != delivery.BuildBuilding {
continue
}
closed := delivery.BuilderBuild{State: delivery.BuildFailed, By: b.Claimer.By, Manifest: cur.Manifest, Step: cur.Step,
Error: fmt.Sprintf("closed at start: left building by %s, and the module no longer has this fingerprint, so nothing will build it", cur.By)}
data, err := delivery.EncodeBuilderBuild(closed)
if err != nil {
return err
}
if _, err := b.Claimer.Store.Update(ctx, key, data, rev); err != nil && !errors.Is(err, ErrConflict) {
return fmt.Errorf("%s: %w", key, err)
}
b.logf("%s: closed, left building by %s for a fingerprint the module no longer has", key, cur.By)
}
if newly {
// A key read before the scan may wait for a package the scan found published (ADR 0500 clause 6).
b.Reconsider(ctx)
}
return nil
}
// notePublished keeps the package versions a done build names as references, and says whether one was new.
func (b *Builder) notePublished(v delivery.BuilderBuild) bool {
if v.State != delivery.BuildDone {
return false
}
b.mu.Lock()
defer b.mu.Unlock()
newly := false
for _, a := range v.Artifacts {
if a.Kind == catalogue.ArtifactPackage && a.Reference != "" && !b.published[a.Reference] {
b.published[a.Reference] = true
newly = true
}
}
return newly
}
func (b *Builder) isPublished(ref string) bool {
b.mu.Lock()
defer b.mu.Unlock()
return b.published[ref]
}
// Forget takes a module out (its key was deleted).
func (b *Builder) Forget(key string) {
b.init()
b.mu.Lock()
delete(b.known, key)
b.mu.Unlock()
}
// storedOnce is Stored for one key, asked once per process while the store answers every reference held.
func (b *Builder) storedOnce(key string) func(ctx context.Context, references []string) ([]string, error) {
if b.Stored == nil {
return nil
}
return func(ctx context.Context, references []string) ([]string, error) {
id := key + " " + strings.Join(references, " ")
b.mu.Lock()
known := b.held[id]
b.mu.Unlock()
if known {
return nil, nil
}
gone, err := b.Stored(ctx, references)
if err == nil && len(gone) == 0 {
b.mu.Lock()
b.held[id] = true
b.mu.Unlock()
}
return gone, err
}
}
// ArtifactsGone takes the seat mesh-artifact-sweep's event artifacts-gone in (issue 1937, ADR 0571 §2): the sweep found
// that the store does not hold what the module's key names. The module is looked at again, which asks the store (ClaimOptions.Stored) and
// builds the key again; an event for a fingerprint the module no longer has is said and left. The event only wakes the
// builder: what is built rests on the store's answer, never on the event alone.
func (b *Builder) ArtifactsGone(ctx context.Context, e delivery.ArtifactsGone) {
b.init()
b.mu.Lock()
m, ok := b.known[e.Module]
if key, err := delivery.BuilderBuildKey(e.Module, e.Fingerprint); err == nil {
for id := range b.held {
if strings.HasPrefix(id, key+" ") {
delete(b.held, id)
}
}
}
b.mu.Unlock()
if !ok || m.FP2 != e.Fingerprint {
b.logf("%s: heard %s for %s, a fingerprint the module does not have now: nothing to build", e.Module,
delivery.ArtifactsGoneEvent, e.Fingerprint)
return
}
b.logf("%s: heard %s (%s): asking the artifact store and building the key again", e.Module,
delivery.ArtifactsGoneEvent, strings.Join(e.Gone, ", "))
b.consider(ctx, e.Module, m)
}
// Reconsider looks at every module again: the toolchains changed, so a build that failed for them may build now.
func (b *Builder) Reconsider(ctx context.Context) {
b.init()
b.mu.Lock()
known := make(map[string]delivery.Module, len(b.known))
for k, m := range b.known {
known[k] = m
}
b.mu.Unlock()
for _, k := range sortedKeys(known) {
b.consider(ctx, k, known[k])
}
b.TrunkMoved(ctx)
}
// TrunkMoved considers every candidate again once a revision of the gate's source is heard: a candidate heard before the
// revision it was previewed onto had no trunk head to be merged onto.
func (b *Builder) TrunkMoved(ctx context.Context) {
b.init()
b.mu.Lock()
cands := make([]delivery.Candidate, 0, len(b.cands))
for _, c := range b.cands {
cands = append(cands, c)
}
b.mu.Unlock()
sort.Slice(cands, func(i, j int) bool { return cands[i].Group < cands[j].Group })
for _, c := range cands {
b.considerCandidate(ctx, c)
}
}
func sortedKeys(m map[string]delivery.Module) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// planOf is the plan of the manifest with this digest, read once: a manifest by digest never changes. A manifest that
// cannot be read is not kept, so it is read again when the module is.
func (b *Builder) planOf(ctx context.Context, digest string) (Plan, error, bool) {
b.mu.Lock()
p, ok := b.plans[digest]
b.mu.Unlock()
if ok {
return p.plan, p.err, true
}
m, err := b.Manifests.Manifest(ctx, digest)
if err != nil {
return Plan{}, fmt.Errorf("its manifest %s cannot be read: %w", digest, err), false
}
plan, err := PlanOf(m)
b.mu.Lock()
b.plans[digest] = planned{plan, err}
b.mu.Unlock()
return plan, err, true
}
func (b *Builder) consider(ctx context.Context, module string, m delivery.Module) {
if m.FP2 == "" || m.Manifest == "" {
return
}
key, err := delivery.BuilderBuildKey(module, m.FP2)
if err != nil {
b.logf("modules.%s: %v", module, err)
return
}
b.mu.Lock()
if b.inflight[key] {
b.again[key] = m // looked at once more when the build ends, so a change is never lost
b.mu.Unlock()
return
}
b.inflight[key] = true
b.mu.Unlock()
b.wg.Add(1)
go func() {
defer b.wg.Done()
b.sem <- struct{}{}
defer func() { <-b.sem }()
for {
b.try(ctx, module, m)
b.mu.Lock()
next, more := b.again[key]
delete(b.again, key)
if !more || ctx.Err() != nil {
delete(b.inflight, key)
b.mu.Unlock()
return
}
b.mu.Unlock()
m = next
}
}()
}
// startOnce looks at the module for a key the caller has set inflight already (BuildAgain): the by-hand look runs once, and
// a look the builder queued for the key meanwhile (again) still runs after it, as consider's loop does.
func (b *Builder) startOnce(ctx context.Context, key, module string, m delivery.Module) {
b.wg.Add(1)
go func() {
defer b.wg.Done()
b.sem <- struct{}{}
defer func() { <-b.sem }()
for {
b.try(ctx, module, m)
b.mu.Lock()
next, more := b.again[key]
delete(b.again, key)
if !more || ctx.Err() != nil {
delete(b.inflight, key)
b.mu.Unlock()
return
}
b.mu.Unlock()
m = next // a look the builder queued meanwhile is not lost (consider's rule); it is not a second by-hand build
}
}()
}
// try claims the fingerprint and builds it, writing the key done or failed. A build that ends because ctx did is left
// building by this builder: it takes it again when it starts (ADR 0478 clause 5).
func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
key, _ := delivery.BuilderBuildKey(module, m.FP2)
defer b.handEnded(key)
plan, planErr, read := b.planOf(ctx, m.Manifest)
nothing := read && planErr == nil && len(plan.Tasks) == 0
if planErr == nil && !nothing {
if why := b.stale(m, plan); why != "" {
// The fingerprint names a toolchain the build nodes no longer agree on: a new fingerprint is coming, and a build
// now would be of this source with another toolchain (ADR 0584 clause 1, which changes ADR 0477 clause 2). Nothing is written.
b.logf("%s at %s is not built: %s", module, m.FP2, why)
return
}
}
b.mu.Lock()
trunk := b.trunkFP[key]
first := !b.tried[key]
b.tried[key] = true
b.mu.Unlock()
cl, err := b.Claimer.Claim(ctx, module, m.FP2, m.Manifest, ClaimOptions{TakeOver: first, ImageBuilder: imageSeat, Trunk: trunk, Published: b.isPublished,
Stored: b.storedOnce(key)})
if err != nil {
b.logf("%s: %v", module, err)
return
}
if cl == nil {
return
}
b.handClaimed(key, cl.Rev)
b.logBuild(LogStarted(module, m, b.refOf(module, m.FP2), b.Node, b.now()))
if cl.Unasked != nil {
b.logf("%v", cl.Unasked)
}
if len(cl.Gone) > 0 {
// Said aloud (issue 1937): the key was done, and the store's collector deleted what it names.
b.logf("%s: built again: the artifact store no longer holds %s", cl.Key, strings.Join(cl.Gone, ", "))
}
// A done key taken again builds only what is missing: the image or the package built unpublished as a candidate's; the
// other artifacts stay as the done key has them (ADR 0487 clauses 5 and 8, ADR 0498 clause 6).
var prior *delivery.BuilderBuild
var only map[string]bool
publishing := false
if cl.Build.State == delivery.BuildDone {
prior, only = &cl.Build, map[string]bool{}
if trunk {
for _, bm := range publishable(cl.Build, imageSeat) {
only[bm] = true
publishing = true
}
}
}
var result delivery.BuilderBuild
if nothing {
// A manifest with no artifact is built by building nothing: its key is done with no artifact, claimed and written as
// any other, so the controller has a done key to register it from (issue 1613, ADR 0478 clause 1, ADR 0501 clause 1).
result = delivery.BuilderBuild{State: delivery.BuildDone, Step: "nothing to build: the manifest names no artifact"}
} else {
result = b.build(ctx, module, m, plan, planErr, prior, only, b.notes(cl))
}
if ctx.Err() != nil {
return
}
if result.State == "" {
b.logf("%s: left building, the toolchains moved under it", cl.Key)
return
}
result.By, result.Manifest = b.Claimer.By, m.Manifest
if len(cl.Gone) > 0 && result.State == delivery.BuildDone {
result.Step += " (built again: the artifact store no longer held " + strings.Join(cl.Gone, ", ") + ")"
}
if publishing && result.State == delivery.BuildFailed {
// The trunk's image or package was built only as a candidate's and its push or publish failed: the key says so, so the builds view, the
// checker and the failed-key retake see it, and a look does not build the image again for nothing.
result.Error = keyError([]string{"publishing what the merged commit built failed: " + result.Error})
}
if err := b.Claimer.Finish(ctx, cl, result); err != nil {
b.logf("%s: the build could not be written: %v", cl.Key, err)
return
}
b.logBuild(LogEnded(module, m, b.refOf(module, m.FP2), b.Node, b.now(), result))
if result.State == delivery.BuildFailed {
b.logf("%s: failed: %s", cl.Key, result.Error)
}
if result.State == delivery.BuildQueued {
b.logf("%s: waits for the package %s to be published", cl.Key, result.Waits)
}
if b.notePublished(result) {
// A key may wait for the package this build published (ADR 0500 clause 6).
b.Reconsider(ctx)
}
}
// stale is why the module's fingerprint is not built now, or empty when it is: the toolchain each language of the plan
// has agreed (novox/hq ADR 0465) must be the one the fingerprint was computed with (Module.Toolchains).
func (b *Builder) stale(m delivery.Module, plan Plan) string {
for _, bm := range plan.Modules() {
if !plan.NeedsToolchain(bm) {
continue // an archive is packed with no toolchain
}
language := buildrun.LanguageOf[bm]
ag := delivery.ToolchainAgreement(b.Toolchains.Entries(language))
if ag.Wait != "" {
return fmt.Sprintf("the %s toolchain is not agreed: %s", language, ag.Wait)
}
tc := ag.Toolchain
want, ok := m.Toolchains[language]
if !ok || want.Version != tc.Version || want.Recipe != tc.Recipe {
return fmt.Sprintf("the fingerprint names %s %s (recipe %s) and the nodes agree on %s (recipe %s)", language, want.Version, want.Recipe, tc.Version, tc.Recipe)
}
}
return ""
}
func failed(format string, args ...any) delivery.BuilderBuild {
return delivery.BuilderBuild{State: delivery.BuildFailed, Error: fmt.Sprintf(format, args...)}
}
// lane is the build of a plan's tasks of one build module on one node.
type lane struct {
module, node string // node is the one that pulled the lane's ask
tasks []Task
toolchain *delivery.ToolchainUsed
digests map[string]string // artifact -> digest
references map[string]string // artifact -> the reference an image was pushed to (a trunk build only)
imports []string // the mesh packages the lane's programs import, as its build module answered them
err error
}
func (b *Builder) build(ctx context.Context, module string, m delivery.Module, plan Plan, planErr error, prior *delivery.BuilderBuild, only map[string]bool, asked *askNotes) delivery.BuilderBuild {
switch {
case planErr != nil:
return failed("%s", planErr)
case m.Path == "":
// Empty is not known (a root module's path is "."): the indexer writes it again (issue 1615).
return failed("the module's key names no directory (path), so the build tools cannot be told where %s lives", module)
}
repository := delivery.RepositoryName(m.Repository)
var lanes []*lane
used := map[string]delivery.ToolchainUsed{}
var sitting []string // the nodes that sit out of a language's build, said in the key's step (ADR 0584 clause 2)
var kept []string // build modules whose artifacts stay as the prior done key has them
for _, bm := range plan.Modules() {
if prior != nil && len(only) > 0 && !only[bm] {
kept = append(kept, bm)
continue
}
var toolchain *delivery.ToolchainUsed
if bm != ImageBuildModule && plan.NeedsToolchain(bm) {
// The ask names the toolchain the fingerprint was computed with; a node building with another refuses it.
language := buildrun.LanguageOf[bm]
ag := delivery.ToolchainAgreement(b.Toolchains.Entries(language))
if ag.Wait != "" {
return failed("the %s toolchain is not agreed: %s", language, ag.Wait)
}
tc := ag.Toolchain
sitting = append(sitting, SittingOutWords(language, ag)...)
if want := m.Toolchains[language]; want.Version != tc.Version || want.Recipe != tc.Recipe {
return delivery.BuilderBuild{} // the agreed toolchain moved since try looked: nothing is written
}
u := delivery.ToolchainUsed{Version: tc.Version, Recipe: tc.Recipe}
used[language], toolchain = u, &u
}
var tasks []Task
for _, t := range plan.Tasks {
if t.Module == bm {
tasks = append(tasks, t)
}
}
// One ask builds each fingerprint's part of a build module, on whichever node pulls it (ADR 0525 §3).
lanes = append(lanes, &lane{module: bm, tasks: tasks, toolchain: toolchain, digests: map[string]string{}, references: map[string]string{}})
}
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.handLanes(key, len(lanes))
}
var wg sync.WaitGroup
for _, l := range lanes {
wg.Add(1)
go func() { defer wg.Done(); b.runLane(ctx, l, repository, m, module, asked) }()
}
wg.Wait()
var errs []string
for _, l := range lanes {
if l.err != nil {
errs = append(errs, l.err.Error())
}
}
if len(errs) > 0 {
if ref := waitsOf(errs); ref != "" {
// The module's lock names a version of a package the registry does not hold yet: the key waits for it, queued, and
// is not failed (ADR 0500 clause 6).
return delivery.BuilderBuild{State: delivery.BuildQueued, Waits: ref, Step: "waits for the package " + ref + " to be published"}
}
return failed("%s", keyError(errs))
}
var artifacts []delivery.BuiltArtifact
keptModule := map[string]bool{}
for _, bm := range kept {
keptModule[bm] = true
}
for _, t := range plan.Tasks {
if keptModule[t.Module] {
found := false
for _, a := range prior.Artifacts {
if a.Name == t.Artifact {
artifacts, found = append(artifacts, a), true
}
}
if !found {
return failed("the done build holds no artifact %s to keep", t.Artifact)
}
continue
}
byNode := map[string]string{}
for _, l := range lanes {
if d, ok := l.digests[t.Artifact]; ok && l.module == t.Module {
byNode[l.node] = d
}
}
d, err := buildrun.Agree(t.Artifact, byNode)
if err != nil {
return failed("%s", err)
}
built := delivery.BuiltArtifact{Name: t.Artifact, Kind: t.Kind, Digest: d}
for _, l := range lanes {
if l.module != t.Module {
continue
}
r := l.references[t.Artifact]
if r == "" && stored(t.Kind) {
// A bundle, an archive or an upstream copy is done only when its push was answered with where it is (ADR 0498
// clause 1); a candidate's included (clause 3).
return failed("%s%s on %s: the build answered no reference, so its push to the artifact store was not answered", delivery.NotStored, t.Artifact, l.node)
}
if r != "" {
built.Reference = r
}
}
artifacts = append(artifacts, built)
}
var steps []string
for _, l := range lanes {
steps = append(steps, fmt.Sprintf("%s on %s", l.module, l.node))
}
for _, bm := range kept {
steps = append(steps, bm+" kept")
for lang, u := range prior.Toolchains {
if _, ok := used[lang]; !ok && buildrun.LanguageOf[bm] == lang {
used[lang] = u
}
}
}
step := "built " + strings.Join(steps, "; ")
if len(sitting) > 0 {
step += " (" + strings.Join(sitting, "; ") + ")"
}
if len(used) == 0 {
used = nil
}
done := delivery.BuilderBuild{State: delivery.BuildDone, By: b.Claimer.By, Manifest: m.Manifest, Step: step, Artifacts: artifacts, Toolchains: used,
Imports: importsOf(lanes)}
if _, err := delivery.EncodeBuilderBuild(done); err != nil {
// A reference the build module answered that does not name the artifact and its digest is no reference.
return failed("%s%v", delivery.NotStored, err)
}
return done
}
// stored is whether an artifact of this kind is in the artifact store once built, a candidate's too, so its done key
// names its reference (ADR 0498 clauses 1, 3 and 4); an image or a package names one only when built for the trunk.
func stored(kind string) bool {
return kind == catalogue.ArtifactBundle || kind == catalogue.ArtifactArchive || kind == catalogue.ArtifactUpstream
}
// waitsFor opens the reason of a fetch step whose module's lock names a package version the registry does not hold, then
// `<name>@<version>` (the Node build module's, ADR 0500 clause 6).
const waitsFor = delivery.WaitsFor
// waitsOf is the package version a lane's error says the module waits for, or "" when none does or what it names is not
// `<name>@<version>`.
func waitsOf(errs []string) string {
for _, e := range errs {
at := strings.Index(e, waitsFor)
if at < 0 {
continue
}
ref, _, _ := strings.Cut(e[at+len(waitsFor):], " ")
ref = strings.TrimRight(ref, ":;,")
if _, err := delivery.EncodeBuilderBuild(delivery.BuilderBuild{State: delivery.BuildQueued, Waits: ref}); err == nil {
return ref
}
}
return ""
}
// keyPrefixes are the errors a build module's step may open its reason with that the key's error must open with too, so
// the claim reads them (delivery.FailedUntilChanged), first the ones not built again.
var keyPrefixes = []string{delivery.NotPinned, delivery.VersionTaken, delivery.RefusedKind, delivery.NotStored}
// keyError is the error of a key from its lanes' errors: when one names a prefix of keyPrefixes inside the step's words
// (`<tool> on <node>: the step <key> failed: not stored: ...`), the key's error opens with that prefix, so a reader of the
// key need not search for it.
func keyError(errs []string) string {
for _, prefix := range keyPrefixes {
for i, e := range errs {
if strings.HasPrefix(e, prefix) {
return strings.Join(errs, "; ")
}
if at := strings.Index(e, prefix); at >= 0 {
rest := append([]string{prefix + e[:at] + e[at+len(prefix):]}, append(append([]string(nil), errs[:i]...), errs[i+1:]...)...)
return strings.Join(rest, "; ")
}
}
}
return strings.Join(errs, "; ")
}
// imageSeat says an image build is asked of the image build seat, which takes it when a node pulls it: a key failed for
// want of an image builder before R4g is taken again (ADR 0525 §1).
func imageSeat() bool { return true }
// askLanguage is the build seat language of each build module (delivery.AskSubject).
var askLanguage = map[string]string{buildrun.GoBuildModule: delivery.AskGo, buildrun.NodeBuildModule: delivery.AskNode, ImageBuildModule: delivery.AskImage}
// askNotes writes a claimed key's asks into it, building, before each is put (ADR 0525 §1; issue 1887 A4b), and holds
// the asks a key taken over at start already names, so they are waited for and never put again.
type askNotes struct {
mu sync.Mutex
b *Builder
cl *Claim
resumed map[string]string
// step is the step words of a key taken over at start: an ask asked again says its asking there (askingOf).
step string
}
func (b *Builder) notes(cl *Claim) *askNotes {
n := &askNotes{b: b, cl: cl}
if cl.Build.State == delivery.BuildBuilding && len(cl.Build.Asks) > 0 {
n.resumed, n.step = maps.Clone(cl.Build.Asks), cl.Build.Step
}
return n
}
// resume is the id of the ask a taken-over key names for language, or "".
func (n *askNotes) resume(language string) string {
n.mu.Lock()
defer n.mu.Unlock()
return n.resumed[language]
}
// note writes the ask's id into the key, building, by compare-and-set at the claim's revision.
func (n *askNotes) note(ctx context.Context, language, id, step string) error {
n.mu.Lock()
defer n.mu.Unlock()
v := n.cl.Build
v.State, v.By, v.Step, v.Error, v.Artifacts = delivery.BuildBuilding, n.b.Claimer.By, step, "", nil
v.Asks = maps.Clone(v.Asks)
if v.Asks == nil {
v.Asks = map[string]string{}
}
v.Asks[language] = id
if err := n.b.Claimer.Finish(ctx, n.cl, v); err != nil {
return err
}
n.b.handAsk(n.cl.Key, language, id)
return nil
}
// runLane puts the lane's part of the build as one ask on its build seat's stream (the delivery stream for a trunk key, the
// check stream for a candidate's) and reads what the node that pulled it built (ADR 0525 §1, §3, §6). An upstream artifact
// is copied by its pinned reference with no fetch (ADR 0498 clause 4); a lane of copies alone fetches nothing. A lane of a
// key taken over at start waits for the ask the key names and puts none.
func (b *Builder) runLane(ctx context.Context, l *lane, repository string, m delivery.Module, module string, asked *askNotes) {
req := buildstep.FetchRequest{Repository: repository, Commit: m.Commit}
if key, err := delivery.BuilderBuildKey(module, m.FP2); err == nil {
b.mu.Lock()
v, candidate := b.via[key]
b.mu.Unlock()
if candidate {
req.Ref, req.Base = v.Ref, v.Base
}
}
if m.Inputs2 != nil && len(m.Inputs2.Paths) > 0 {
// A module whose every artifact names its paths is built from those alone (ADR 0499 clause 4): the paths come from
// the indexer's kept inputs, never from a manifest (the lead's decision L8 of R4e).
req.Dir = m.Path
for p := range m.Inputs2.Paths {
req.Paths = append(req.Paths, p)
}
sort.Strings(req.Paths)
}
candidate := req.Base != ""
language := askLanguage[l.module]
kind := delivery.AskDelivery
if candidate {
kind = delivery.AskCheck
}
job := &delivery.BuildJob{Module: module, Path: m.Path, FP: m.FP2, Toolchain: l.toolchain,
// Only a trunk key's build pushes an image or publishes a package; a candidate's never does (ADR 0487 clause 12,
// ADR 0498 clauses 5 and 6). A bundle or an archive is pushed either way (clause 3).
Publish: !candidate}
for _, t := range l.tasks {
a := delivery.AskArtifact{Artifact: t.Artifact}
if t.Kind == catalogue.ArtifactUpstream {
a.From = t.From // the copy runs no code of a change: the module's name, the artifact and its pin
}
job.Artifacts = append(job.Artifacts, a)
}
if needsFetch(l.tasks) {
job.Fetch = req.Args()
}
ask := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: language, Asker: delivery.BuilderModule, At: time.Now(), Build: job}
if language == delivery.AskGo && job.Fetch != nil && len(m.Packages) > 0 {
// The versions of the mesh packages the fingerprint was computed with (novox/hq ADR 0573 §2): the build module writes
// them into the workspace, whatever the repository's go.mod names.
ask.Packages = maps.Clone(m.Packages)
}
if id := asked.resume(language); id != "" {
ask.ID = id // put before this builder started again: the stream holds it, or a node already ran it
if b.Held != nil {
held, err := b.Held.Held(ctx, ask)
if err != nil {
// Said, and the key is not failed: the ask may well be on its stream and its build finish (issue 1910).
b.logf("whether the ask %s of %s the key names is on its stream cannot be read, so it is waited for as put: %v", ask.ID, l.module, err)
held = true
}
if !held {
// The builder that named it stopped before it put it (issue 1910): put it now, under the same id.
b.logf("the ask %s of %s that the key names was never put: it is put now", ask.ID, l.module)
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
}
}
} else {
if err := asked.note(ctx, language, ask.ID, "asked "+l.module); err != nil {
l.err = fmt.Errorf("the ask of %s could not be written to the key: %w", l.module, err)
return
}
if err := b.Asks.Put(ctx, ask); err != nil {
l.err = err
return
}
}
step, ask, err := b.answer(ctx, l, ask, asked, module, m.FP2)
if err != nil {
l.err = fmt.Errorf("%s: %w", l.module, err)
return
}
l.node = step.Node
if step.State != delivery.StepPassed {
l.err = fmt.Errorf("%s on %s: the ask %s %s: %s", l.module, step.Node, ask.ID, step.State, step.Reason)
return
}
var answer delivery.BuildAnswer
if err := json.Unmarshal(step.Result, &answer); err != nil {
l.err = fmt.Errorf("the ask of %s on %s answered what is not a build: %s", l.module, step.Node, step.Result)
return
}
l.imports = answer.Imports
built := map[string]delivery.AskBuilt{}
for _, a := range answer.Artifacts {
built[a.Artifact] = a
}
for _, t := range l.tasks {
a, ok := built[t.Artifact]
if !ok || a.Digest == "" {
l.err = fmt.Errorf("the build of %s of %s on %s answered no digest: %s", t.Artifact, module, step.Node, step.Result)
return
}
l.digests[t.Artifact] = a.Digest
if a.Reference != "" && (stored(t.Kind) || !candidate) {
l.references[t.Artifact] = a.Reference
}
if a.Reference != "" && b.Answered != nil {
b.Answered(l.module)
}
}
}
// needsFetch is whether any of the tasks builds from the module's source: every one but an upstream copy.
func needsFetch(tasks []Task) bool {
for _, t := range tasks {
if t.Kind != catalogue.ArtifactUpstream {
return true
}
}
return false
}
func (b *Builder) now() time.Time {
if b.Now != nil {
return b.Now()
}
return time.Now()
}
// logBuild tells the log an entry, said in the journal when it cannot be one.
func (b *Builder) logBuild(e delivery.BuildLogEntry) {
if b.Log == nil {
return
}
if _, err := delivery.EncodeBuildLogEntry(e); err != nil {
b.logf("%s at %s: no log entry: %v", e.Module, e.FP, err)
return
}
b.Log(e)
}
// refOf is the ref a candidate's build of module's fingerprint is built for, "" for the trunk's.
func (b *Builder) refOf(module, fp string) string {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return ""
}
b.mu.Lock()
defer b.mu.Unlock()
return b.via[key].Ref
}
// LogStarted is the log's entry of a build of module's fingerprint starting (ADR 0571 §1): the module, the fingerprint, the
// source revision, where it is built from (repository, directory, commit, the candidate's ref), the node of the builder
// that asked it and the time.
func LogStarted(module string, m delivery.Module, ref, by string, at time.Time) delivery.BuildLogEntry {
dir := m.Path
if dir == "" {
dir = delivery.RootPath
}
return delivery.BuildLogEntry{Event: delivery.BuildStartedEvent, Module: module, FP: m.FP2, Source: strconv.FormatUint(m.Src, 10),
Repository: m.Repository, Directory: dir, Commit: m.Commit, Ref: ref, Node: by, At: at.UTC()}
}
// LogEnded is the log's entry of that build ending: the outcome (done, failed or queued), the manifest built, the bases it
// was built against and each artifact made, as the done key has them: what the operator's command nox builds shows of a build.
func LogEnded(module string, m delivery.Module, ref, by string, at time.Time, result delivery.BuilderBuild) delivery.BuildLogEntry {
e := LogStarted(module, m, ref, by, at)
e.Event, e.Outcome, e.Manifest = delivery.BuildEndedEvent, result.State, m.Manifest
if result.Manifest != "" {
e.Manifest = result.Manifest
}
if m.Inputs2 != nil {
e.Against = append([]string(nil), m.Inputs2.Bases...)
}
e.Made = append([]delivery.BuiltArtifact(nil), result.Artifacts...)
e.Imports = append([]string(nil), result.Imports...) // the indexer keeps them on the module (novox/hq ADR 0593 §2)
return e
}
// importsOf are the mesh packages the lanes' programs import, as their build modules answered them (novox/hq ADR 0593
// §2): sorted and each once; nil when none answered any. A lane kept from a prior build answered none, so a build that
// kept one reports only what it built.
func importsOf(lanes []*lane) []string {
seen := map[string]bool{}
for _, l := range lanes {
for _, p := range l.imports {
seen[p] = true
}
}
if len(seen) == 0 {
return nil
}
return slices.Sorted(maps.Keys(seen))
}
// SittingOutWords says, per node that is not on the agreed toolchain of a language, that it sits out and which version
// it has against the agreed one: "ace sits out: typescript 26.12.1, agreed 26.13.0" (novox/hq ADR 0584 clause 2).
func SittingOutWords(language string, a delivery.Agreement) []string {
var out []string
for _, s := range a.SittingOut {
out = append(out, fmt.Sprintf("%s sits out: %s %s, agreed %s", s.Node, language, s.Version, a.Toolchain.Version))
}
return out
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,43 @@
package build
import (
"fmt"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// BuildsSinceLimit is how many entries one answer of BuildsSinceTool reads of the log; More says the log held more.
const BuildsSinceLimit = 500
// LogSince is one page of the builder's log from a time on: stdio.Log(BuilderLog).Since.
type LogSince func(at time.Time, keys string, limit int) (*stdio.LogPage, error)
// BuildsSince answers delivery.BuildsSinceTool (ADR 0571 clause 9): every ended build in the builder's log from since on,
// oldest first, read through the runtime's log verbs; More when the page was not the log's last, so the caller asks again
// from the last build's time. An entry the log holds that does not read is refused, naming its sequence: a reader that
// skipped it would register too little.
func BuildsSince(read LogSince, since time.Time) (delivery.BuildsSince, error) {
out := delivery.BuildsSince{Since: since.UTC(), Builds: []delivery.BuildLogEntry{}}
page, err := read(since, "", BuildsSinceLimit)
if err != nil {
return out, fmt.Errorf("the builder's log %s from %s: %w", delivery.BuilderLog, since.Format(time.RFC3339), err)
}
if page == nil {
return out, nil
}
var entries []delivery.BuildLogEntry
for _, le := range page.Entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
return out, fmt.Errorf("the builder's log %s, entry %d: %w", delivery.BuilderLog, le.Sequence, err)
}
entries = append(entries, e)
}
if ended := delivery.EndedSince(entries, since); ended != nil {
out.Builds = ended
}
out.More = page.Next != 0
return out, nil
}
@@ -0,0 +1,357 @@
package build
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
"git.novox.be/novox/mesh-delivery/modules/mesh-delivery-builder/buildrun"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The builder's tools (novox/hq issue 1945; the instruction "tools are how a module is spoken to"): builder_stuck reads
// every key that waits on an ask, builder_ask_again asks a key's build again now. The tools bundle serves them through
// the tool runner; the builder's process serves no tool (ADR 0437 §4), so builder_ask_again writes the ask into the
// builder's state reasks, which the builder watches and answers there.
const (
StuckTool = "builder_stuck"
AskAgainTool = "builder_ask_again"
// BuildAgainTool builds a failed key again by hand (issue 1126 note 72): it writes a reasks value of Mode ModeBuild.
BuildAgainTool = "builder_build_again"
// AnnounceAgainTool writes a done key again unchanged, so the controller takes it in and composes again: a reasks value
// of Mode ModeAnnounce.
AnnounceAgainTool = "builder_announce_again"
// ReAsksState is the builder's state of asks to ask again: the bucket mesh-delivery-builder_reasks, keyed as the build key.
ReAsksState = "reasks"
ReAsksBucket = delivery.BuilderModule + "_" + ReAsksState
)
// ReAsk is a value of the reasks state: why a person asked the key's build again, who and when, and, once the builder
// read it, its answer.
type ReAsk struct {
V int `json:"v"`
Why string `json:"why"`
By string `json:"by,omitempty"`
At time.Time `json:"at"`
Answer string `json:"answer,omitempty"`
Answered time.Time `json:"answered,omitzero"`
// Mode is ModeBuild for a request to build a failed key again (BuildAgainTool); empty for an ask asked again.
Mode string `json:"mode,omitempty"`
Module string `json:"module,omitempty"`
FP string `json:"fp,omitempty"`
// Refused, Revision and Asks are the builder's answer to a ModeBuild request: why it did nothing, or the key's new
// revision (building) and the ask ids it put by language.
Refused bool `json:"refused,omitempty"`
// Pending says the builder answered before the key was claimed or all its asks were put: neither a refusal nor done.
// Failed is the error of a key written building that failed before its asks were put.
Pending bool `json:"pending,omitempty"`
Failed string `json:"failed,omitempty"`
Revision uint64 `json:"revision,omitempty"`
Asks map[string]string `json:"asks,omitempty"`
}
// ModeBuild is the Mode of a reasks value that asks the key to be built again, not its waiting asks to be asked again.
const ModeBuild = "build"
// ModeAnnounce is the Mode of a reasks value that asks a done key to be announced again.
const ModeAnnounce = "announce"
// DecodeReAsk reads a reasks value.
func DecodeReAsk(data []byte) (ReAsk, error) {
var r ReAsk
if err := json.Unmarshal(data, &r); err != nil {
return r, err
}
if r.V != 1 {
return r, fmt.Errorf("a reasks value of version %d, not 1", r.V)
}
return r, nil
}
// NewReAsk is the value builder_ask_again writes for a module's fingerprint: its key and the value.
func NewReAsk(module, fp, why, by string, at time.Time) (string, []byte, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the build is asked again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC()})
return key, data, err
}
// NewBuildAgain is the value builder_build_again writes for a module's fingerprint: its key and the value.
func NewBuildAgain(module, fp, why, by string, at time.Time) (string, []byte, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the key is built again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModeBuild, Module: module, FP: fp})
return key, data, err
}
// NewAnnounceAgain is the value builder_announce_again writes for a module's fingerprint: its key and the value.
func NewAnnounceAgain(module, fp, why, by string, at time.Time) (string, []byte, error) {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return "", nil, err
}
if strings.TrimSpace(why) == "" {
return "", nil, errors.New("say why the done build is announced again (why)")
}
data, err := json.Marshal(ReAsk{V: 1, Why: strings.TrimSpace(why), By: by, At: at.UTC(), Mode: ModeAnnounce, Module: module, FP: fp})
return key, data, err
}
// StateRead is a state the tools read: its keys, and a key's value with when it was written (zero when not known).
type StateRead struct {
Keys func() ([]string, error)
Get func(key string) (json.RawMessage, time.Time, error)
// Written is when the key's current value was written, from the state's history: one more round trip, so the tools ask it
// only for a row they keep, and only when Get did not say.
Written func(key string) time.Time
}
// StuckAsk is one ask a key waits on.
type StuckAsk struct {
Language string `json:"language"`
BuildModule string `json:"build_module"`
ID string `json:"id"`
Asking int `json:"asking"`
// Asked is when the ask was put, read from its id; Step is what its build module's steps hold of it: none, or its
// state on its node since its start.
Asked time.Time `json:"asked,omitzero"`
Step string `json:"step"`
}
// Stuck is one build key that waits on an ask.
type Stuck struct {
Key string `json:"key"`
Module string `json:"module"`
FP string `json:"fp"`
State string `json:"state"`
Step string `json:"step"`
By string `json:"by"`
Since time.Time `json:"since,omitzero"`
Age string `json:"age,omitempty"`
Asks []StuckAsk `json:"asks"`
ReAsk *ReAsk `json:"reask,omitempty"`
Unasked bool `json:"unasked,omitempty"`
NoneStep int `json:"asks_with_no_step"`
}
// StuckReader reads what builder_stuck answers.
type StuckReader struct {
Builds StateRead
ReAsks StateRead
// Steps are the build modules' steps states, by build module.
Steps map[string]StateRead
// Hearing is the builder's state hearing: whether it hears DEAD_LETTERS (issue 734); without it none is said.
Hearing StateRead
Now func() time.Time
// Logf, when set, gets the one line a call logs: the values it read and the time it took.
Logf func(format string, args ...any)
}
// ReadWorkers is how many values builder_stuck asks the bus for at once, and MaxReads how many it reads in a call (issue 1827:
// every value is a round trip through the runtime, so one after the other a few thousand keys outlast the call's 30 s).
// A copy of the delivery view's and the checker's: a few lines are not worth a group delivery through mesh-sdk.
const (
ReadWorkers = 16
MaxReads = 1000
)
// parallel runs read(i) for i in [0, n), at most ReadWorkers at a time, and answers the first error.
func parallel(n int, read func(i int) error) error {
var (
next atomic.Int64
failed atomic.Bool
once sync.Once
first error
wg sync.WaitGroup
)
for w := 0; w < min(ReadWorkers, n); w++ {
wg.Add(1)
go func() {
defer wg.Done()
for !failed.Load() {
i := int(next.Add(1) - 1)
if i >= n {
return
}
if err := read(i); err != nil {
once.Do(func() { first = err })
failed.Store(true)
}
}
}()
}
wg.Wait()
return first
}
// Stuck is every key building or queued with its asks, oldest first; with module, that module's only.
// deadLetters says whether the builder hears the asks the build seats give up on (issue 734), and why not: deaf, a lane
// whose ask is given up waits for its check bound (buildrun.HearingHealth over the kept hearing).
func (r StuckReader) deadLetters() map[string]any {
if r.Hearing.Get == nil {
return map[string]any{"heard": false, "why": "the builder's state hearing is not readable here"}
}
raw, _, err := r.Hearing.Get(buildrun.HearingKey)
if err != nil {
return map[string]any{"heard": false, "why": "the builder's state hearing cannot be read: " + err.Error()}
}
a := buildrun.HearingHealth(raw)
return map[string]any{"heard": a.Healthy, "why": a.Why}
}
func (r StuckReader) Stuck(module string) (map[string]any, error) {
started := time.Now()
reads := 0
defer func() {
if r.Logf != nil {
r.Logf("builder_stuck read %d values in %s", reads, time.Since(started).Round(time.Millisecond))
}
}()
keys, err := r.Builds.Keys()
if err != nil {
return nil, err
}
now := time.Now
if r.Now != nil {
now = r.Now
}
// Steps are found by the ask's id, which their key ends in: one index per build module, no value read until an ask names it.
stepKeys := map[string]map[string]string{}
for bm, s := range r.Steps {
ks, err := s.Keys()
if err != nil {
return nil, fmt.Errorf("the steps of %s: %w", bm, err)
}
byID := make(map[string]string, len(ks))
for _, k := range ks {
if _, id, err := delivery.ParseStepKey(k); err == nil {
byID[id] = k
}
}
stepKeys[bm] = byID
}
// Narrowed by the key (the module) before any value is read; the state is in the value, so every matching key is read, in
// parallel, at most MaxReads of them.
type parsed struct{ key, mod, fp string }
var matching []parsed
for _, key := range keys {
mod, fp, err := delivery.ParseBuilderBuildKey(key)
if err != nil || (module != "" && mod != module) {
continue
}
matching = append(matching, parsed{key, mod, fp})
}
sort.Slice(matching, func(i, j int) bool { return matching[i].key < matching[j].key })
cut := len(matching) > MaxReads
total := len(matching)
if cut {
matching = matching[:MaxReads]
}
values := make([]json.RawMessage, len(matching))
ats := make([]time.Time, len(matching))
if err := parallel(len(matching), func(i int) error {
raw, at, err := r.Builds.Get(matching[i].key)
if err == nil {
values[i], ats[i] = raw, at
}
return nil
}); err != nil {
return nil, err
}
reads += len(matching)
out := []Stuck{}
for i, m := range matching {
key, mod, fp := m.key, m.mod, m.fp
raw, at := values[i], ats[i]
if raw == nil {
continue
}
v, err := delivery.DecodeBuilderBuild(raw)
if err != nil || (v.State != delivery.BuildBuilding && v.State != delivery.BuildQueued) {
continue
}
if at.IsZero() && r.Builds.Written != nil {
at = r.Builds.Written(key)
}
s := Stuck{Key: key, Module: mod, FP: fp, State: v.State, Step: v.Step, By: v.By, Since: at, Asks: []StuckAsk{}}
if !at.IsZero() {
s.Age = now().Sub(at).Round(time.Second).String()
}
for language, id := range v.Asks {
bm, _ := delivery.BuildModuleOf(language)
a := StuckAsk{Language: language, BuildModule: bm, ID: id, Asking: askingOf(v.Step, bm), Step: "none"}
if t, ok := idTime(id); ok {
a.Asked = t
}
if st, ok := r.Steps[bm]; ok {
if k, ok := stepKeys[bm][id]; ok {
reads++
if raw, _, err := st.Get(k); err == nil && raw != nil {
if step, err := delivery.DecodeStep(raw); err == nil {
a.Step = fmt.Sprintf("%s on %s since %s", step.State, step.Node, step.Started.UTC().Format(time.RFC3339))
}
}
}
}
if a.Step == "none" {
s.NoneStep++
}
s.Asks = append(s.Asks, a)
}
sort.Slice(s.Asks, func(i, j int) bool { return s.Asks[i].Language < s.Asks[j].Language })
s.Unasked = v.State == delivery.BuildBuilding && len(v.Asks) == 0
if r.ReAsks.Get != nil {
reads++
if raw, _, err := r.ReAsks.Get(key); err == nil && raw != nil {
if ra, err := DecodeReAsk(raw); err == nil {
s.ReAsk = &ra
}
}
}
out = append(out, s)
}
sort.Slice(out, func(i, j int) bool {
if !out[i].Since.Equal(out[j].Since) {
return out[i].Since.Before(out[j].Since)
}
return out[i].Key < out[j].Key
})
answer := map[string]any{"stuck": out, "count": len(out),
"delivery_counts": "not readable by the builder: the build seats' workers' information is granted to the delivery view alone (its backlog); an ask given up on is told by the builder at its check bound and asked again"}
answer["dead_letters"] = r.deadLetters()
if cut {
answer["cut"], answer["note"] = true, fmt.Sprintf("read %d of %d build keys in key order and stopped; give a module to read its keys alone", MaxReads, total)
}
return answer, nil
}
// StuckBuildModules are the build modules whose steps builder_stuck reads.
var StuckBuildModules = []string{buildrun.GoBuildModule, buildrun.NodeBuildModule, ImageBuildModule}
// idTime is when a step id was made (buildstep.NewID: the time in milliseconds in base 36, then six hex digits).
func idTime(id string) (time.Time, bool) {
if len(id) <= 6 {
return time.Time{}, false
}
ms, err := strconv.ParseInt(id[:len(id)-6], 36, 64)
if err != nil || ms <= 0 {
return time.Time{}, false
}
return time.UnixMilli(ms).UTC(), true
}

Some files were not shown because too many files have changed in this diff Show More