Compare commits

..
Author SHA1 Message Date
jschoubben d51324df45 Format the help's test (issue 1851) 2026-10-22 21:20:06 +02:00
jschoubben 58512a9c6d Pin mesh-sdk at the R4e member's final head 104533e8 and vendor it (R4e E)
The SDK's S1 and S2 review fixes landed after 875c680d.
2026-10-22 21:19:46 +02:00
jschoubben 1fd833bfe8 List the store tests R4e adds, and drop those of the hand builds and the stopped checks it deleted (issue 1851, issue 1299)
The merge check refused the eight deleted tests as listed and not run.
2026-10-22 21:18:58 +02:00
jschoubben f9014452d3 List register and switch in the help, and none of the removed commands (issue 1851 A11a, A11b)
The help still offered build, rebuild and replay --register, each now
refused as removed, and did not name the terminal's register.
2026-10-22 21:10:20 +02:00
jschoubben 61449249a8 Name build-agent's own as the builder's module, build-agent and mesh-tools (ADR 0520 §1, issue 1851)
ADR 0520 narrowed after its review: mesh-tools-go builds from its own
directory on a pinned base and has its key from mesh-build-image, and
mesh-tools-python is no module, so the set is named, not derived from
the toolchain bases.
2026-10-22 21:08:47 +02:00
jschoubben 61f7e89ad5 Ask a repository check whenever the trunk requires it, beside mesh/check (issue 1299 C1, ADR 0513 §4)
A trunk requiring both contexts (hq) matched mesh/check first and had
its own merge-check.sh never asked.
2026-10-22 21:07:55 +02:00
jschoubben 9713ceb84d Ask build-agent for itself and its toolchain images, never a builder key (ADR 0520, issue 1851)
The builder writes no key of build-agent or of the toolchain bases its
builds stand on, so a walk moving one would hold its tier for ever.
Until R4g they are build-agent's own, as the builder's module is; the
bases are read from the toolchains, not copied.
2026-10-22 21:06:33 +02:00
jschoubben 93fc2f48a1 Name a retry's modules still waiting for the bus, read as the plan it resumes (issue 1851 A11a, issue 1403)
With the hand rebuild gone, a retry of a plan failed past the bus's bound
read the waiting modules while the plan still said failed, named none,
and answered "asked" with nothing after it.
2026-10-22 21:05:01 +02:00
jschoubben d5d53893a4 Send the bus its user list before a starting controller writes (issue 1800)
A new controller's grants reached the bus only after its gate, so the
buckets it newly writes were refused until then. At start it now sends
the bus's machine the user list alone, every build there kept.
2026-10-22 20:51:08 +02:00
jschoubben a0b2d98002 Merge main (upgrade safety's declared packages, issue 1840) and pin the R4e mesh-sdk member's final head 875c680d, re-vendored here and in modules/route-proxy; the controller announces declared-packages readable (issue 1790, D1; E) 2026-10-22 20:44:59 +02:00
jschoubben bc80683b9e Count the work the take-ins start in the background, so a test's mesh closes only once that work has ended (issue 1851, A5-A7; issue 1790, D3) 2026-10-22 20:41:30 +02:00
mesh-tools 96441ea28a Merge pull request 'Keep each node's declared packages in a state the pacman module reads (upgrade safety, issue 1840, ADR 0507 §5)' (#439) from feat/upgrade-safety into main 2026-10-22 18:34:19 +00:00
jschoubben 420ae3d0e7 Test the group check's answer without swapping standard output, and wait for a take-in to forget a new module's wait (issue 1299, C2; issue 1851, A5) 2026-10-22 20:19:57 +02:00
jschoubben 7c0e1d223f Serve a removed verb as its refusal, not as a verb this build is behind on, and fit the tests of the verbs and the stopped checks (issue 1851, A11a; issue 1299, C1; issue 1318, C3) 2026-10-22 19:38:02 +02:00
jschoubben a10ef8995b Vendor what go.mod pins after the rebase on main (issue 1790; the SDK's 1833 fix returns with E's re-pin) 2026-10-22 19:05:21 +02:00
jschoubben dfc2f2aa98 Take a walk's key in again when it was recorded at an earlier merge, so the module is registered at the commit the walk carries before it is built (issue 1851, A6) 2026-10-22 19:05:05 +02:00
jschoubben ab33f015ec Say again what the controller keeps at every push, so a word refused before its grant is kept once a push carries it (issue 1790, D1) 2026-10-22 19:05:05 +02:00
jschoubben 791c2a3d48 Refuse a removed verb by name before reading its arguments, as the SDK member's table now offers none (issue 1851, A11a; issue 1318, C3) 2026-10-22 19:05:05 +02:00
jschoubben d36e0ea278 Announce the controller's states, keep every module's word, and grant a read only where its owner announced the state readable; the copied list of the controller's mirrors goes (issue 1790, D1-D4, ADR 0495, ADR 0513 §3) 2026-10-22 19:05:04 +02:00
jschoubben b6ed18197b Vendor the R4e mesh-sdk member's head with the states announcement (issue 1790; re-pinned to the final head in E) 2026-10-22 19:05:04 +02:00
jschoubben 45d091778c Refuse delivery-plan and forget-checks by name and take a merge's warrant from the teller alone, no longer mesh-delivery's (issue 1318, C3, ADR 0450 §8) 2026-10-22 19:05:04 +02:00
jschoubben 3a1937914e Ask build-agent no composed check of a delivery group, and answer that the checker judges it (issue 1299, C2, ADR 0503 §3) 2026-10-22 19:05:04 +02:00
jschoubben 25fdf9920e Read a commit as the same as itself however short, and fit the pending assignments' tests to a mesh that asks no build (issue 1851, A9, A10) 2026-10-22 19:05:04 +02:00
jschoubben 323e1270cc Ask no pull request check of build-agent but the own merge-check.sh of a repository whose trunk requires mesh/repo-check alone, and say every verdict with no gate layer (issue 1299, C1, ADR 0503 §3, §5) 2026-10-22 19:05:04 +02:00
jschoubben e7e23c57cc Read a module whose every artifact names its paths from those paths and its manifest alone, so a change elsewhere in its directory moves nothing (issue 1758, B1, ADR 0499 §5) 2026-10-22 19:05:04 +02:00
jschoubben a8387ebad8 Name, before the switch, which modules it moves: the held build's digests against the builder's done key, images and packages said as what cannot be compared (issue 1851, A13) 2026-10-22 19:05:04 +02:00
jschoubben 1963d84a7a Refuse, in the one function every ask goes through, an ask of build-agent for any module but the builder's own (issue 1851, A12, ADR 0453 §5) 2026-10-22 19:05:04 +02:00
jschoubben b71425867f Register a module from a repository the catalogue does not build at the controller's terminal alone, from its done trunk key, naming exactly its repository and path (issue 1851, A11b, ADR 0501 §4) 2026-10-22 19:05:04 +02:00
jschoubben c7039d81e1 Refuse build, rebuild and replay --register by name, naming what replaced them, and delete the hand builds (issue 1851, A11a, ADR 0501 §5) 2026-10-22 19:05:04 +02:00
jschoubben f76da2ce9c Ask the checker for a try of a branch head, not build-agent, and read its word in today's answer (issue 1851, A11, ADR 0501 §5) 2026-10-22 19:05:04 +02:00
jschoubben 7f6d418205 Mark the walks the integration tests drive around the one ask left as the builder's own, and expect no ask of a moved module (issue 1851, A5, A6) 2026-10-22 19:05:04 +02:00
jschoubben 0aec975d19 Keep an assignment of a module whose key is coming pending on that key and ask no build; nothing asks a request again at start (issue 1851, A10) 2026-10-22 19:05:04 +02:00
jschoubben 9da79ef8c3 Move no module whose fingerprint is unchanged, and let a key's build stand on the held build-agent build of the same digests at the switch (issue 1851, A9) 2026-10-22 19:05:04 +02:00
jschoubben 7743bd0d4b Delete the walk's reuse of a check's build: a walk takes keys and asks build-agent only for the builder's own module (issue 1851, A8) 2026-10-22 19:05:04 +02:00
jschoubben 62268cf43d Take a walk's modules from the builder's done keys and ask no build but the builder's own: a key building or unreferenced holds the tier and says so, a failed key fails the walk in its words (issue 1851, A6, A7) 2026-10-22 19:05:04 +02:00
jschoubben ea3c8ad7f6 Register a new module from the builder's first done key and ask no first build; its condition reads the key and names a manifest word this controller does not read (issue 1851, A5a, A5b) 2026-10-22 19:05:04 +02:00
jschoubben d984f0a55d Take a module in from the builder's done key of its trunk fingerprint: one record of the key's artifacts, its manifest judged against the catalogue trunk's predefined keys (issue 1851, A3, A4b, A4c) 2026-10-22 19:05:04 +02:00
jschoubben f00b6aff8b Judge a recorded build, not build-agent's outcome, and read a registration's warrant from the merge a key names, so a done key registers under the same rules (issue 1851, A4a) 2026-10-22 19:05:04 +02:00
jschoubben eff0d5cbf4 Vendor the R4e mesh-sdk member's head for its shared launcher rule and the manifest word paths (issue 1851; re-pinned to the final head in E) 2026-10-22 19:05:04 +02:00
jschoubben 1ed7e34155 Read the builder's keys, the indexer's values by commit and its manifests, so the controller can take a module from a done key (issue 1851, A1, A2) 2026-10-22 19:05:04 +02:00
jschoubben abe356a262 Merge main and take the SDK at the milestone's mesh-sdk head 8892a117, which holds main's SDK and the milestone's (issue 1840)
mesh/merge-gate head abe356a262e8 merged into trunk 41c96287578e — pass: group feat/upgrade-safety composed: every machine composes with the change as it d…
mesh/repo-check head abe356a262e8 merged into trunk 41c96287578e beside mesh-catalog#450@86ba54b0 — pass: its merge-check.sh passed
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | abe356a262e8 on 41c96287578e | unit 3372 run, 0 kept, 7 skipped | integration…
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792694058124013462 (novox/mesh-controller at 96441ea2 into ma…
mesh/delivery-group group feat/upgrade-safety delivered: every member is delivered
2026-10-22 18:14:14 +02:00
mesh-tools 41c9628757 Merge pull request 'Pin mesh-sdk at the fix that refuses an unresolvable machine alone (issue 1833)' (#437) from fix/1833-one-machine-alone into main 2026-10-22 16:10:20 +00:00
jschoubben 3e3400f510 Take the SDK at the milestone's mesh-sdk head f8b8f3a6, which the delivery group's members must pin exactly (issue 1840) 2026-10-22 18:07:38 +02:00
jschoubben e022a3fd6f issue 1840/S3: look the declared packages up by name in the mirror's integration test, since every node also runs the mesh's own modules and their packages 2026-10-22 17:52:40 +02:00
jschoubben 684a8783e5 Merge remote-tracking branch 'origin/main' into feat/upgrade-safety 2026-10-22 17:40:00 +02:00
mesh-tools 11e364bd43 Merge pull request 'Hold a first merge whose new module the running build-agent refuses for an unknown manifest word, name it, and ask it again when build-agent is delivered (issue 1173)' (#438) from fix/1173-held-first-build into main 2026-10-22 15:28:11 +00:00
jschoubben 1e9b87e03a Hold every refused module of a first merge and ask each again by its own path (issue 1173)
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
mesh/merge-gate head 1e9b87e03acd merged into trunk b8f77577fed2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 1e9b87e03acd merged into trunk b8f77577fed2 — pass: its merge-check.sh passed
mesh/delivery delivered
A held merge was asked again by the rule of a first merge, which finds nothing once a sibling module of the
same merge registered, so it stayed held forever; and a second refusal in a merge already held fell through to
a failed build. The hold now keeps a list of module, path and word, a refusal in a held waiting merge joins it,
and the delivery of build-agent asks each held path again on the merge's Approve warrant.
2026-10-22 16:50:06 +02:00
jschoubben 45a09f90e3 Carry a held merge's new announcement into its record (issue 1173)
HoldMerge replaced the event on the flag only; the record kept the old one.
2026-10-22 16:33:28 +02:00
jschoubben 4538d05732 Ask a held first build again when a delivery of build-agent ends (issue 1173)
ADR 0482 section 5: the build waiting for a build-agent that reads its manifest is asked once the plan delivering it ends.
2026-10-22 16:19:45 +02:00
jschoubben 980d396d40 Keep a first merge whose manifest word the running build-agent does not read, and name it (issue 1173)
ADR 0482 section 5: a merged manifest the running reader refuses waits for the newer reader, never failed.
2026-10-22 16:18:39 +02:00
mesh-tools b8f77577fe Merge pull request 'Say a delivery's start and end with the machines its merges declare, and word a needed new head as its author's (ADR 0505)' (#436) from feat/quiet-deliveries into main 2026-10-22 14:01:22 +00:00
jschoubben 28a278c146 Take mesh-sdk at bd7731f: a machine that does not resolve keeps its name and its address in the mesh as well as its tunnel (issue 1833)
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 28a278c1462a on b8f77577fed2 | unit 3353 run, 0 kept, 7 skipped | integration…
mesh/repo-check head 28a278c1462a merged into trunk 11e364bd439f — pass: its merge-check.sh passed
mesh/merge-gate head 28a278c1462a merged into trunk 11e364bd439f — pass: group fix/1833-one-machine-alone composed: every machine composes with the change …
mesh/delivery delivered
mesh/delivery-group group fix/1833-one-machine-alone delivered: every member is delivered
2026-10-22 15:27:27 +02:00
jschoubben c34ab0e43a Take mesh-sdk at c154bde: a machine that does not resolve keeps its place on the private network, so it alone is refused (issue 1833)
One machine whose modules could not be resolved left the private network uncomposable when it was the hub, and every
send to every machine was held. The SDK now keeps a refused machine at the place the controller already holds for it.
2026-10-22 15:04:47 +02:00
jschoubben 771cc24fcb Take the SDK whose process can delete a key (issue 1840/S1) 2026-10-22 14:37:18 +02:00
jschoubben b3786cc67e Merge remote-tracking branch 'origin/main' into feat/upgrade-safety
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-22 14:25:33 +02:00
jschoubben b7d03b6897 Word a delivery that needs its author's new head as its author's, with no Stop (ADR 0505 §5)
mesh/repo-check head b7d03b68974d merged into trunk 8f2d22db0caa beside mesh-catalog#444@09e749bc — pass: its merge-check.sh passed
mesh/merge-gate head b7d03b68974d merged into trunk 8f2d22db0caa — pass: group feat/quiet-deliveries composed: every machine composes with the change as it…
mesh/check checking b7d03b68974d on 8f2d22db0caa
mesh/delivery delivered
mesh/delivery-group group feat/quiet-deliveries delivered: every member is delivered
2026-10-22 14:25:26 +02:00
jschoubben 21954ad1bc issue 1840/S3b: prove a process may watch its own state's bucket and no other (AC9) 2026-10-22 14:23:58 +02:00
jschoubben 485eb44de4 issue 1840/S3: keep each node's declared packages on the bus, so the pacman verbs can say what a node declares 2026-10-22 14:23:58 +02:00
jschoubben 8d562f9b22 Say a walk's start and end again with the machines whose declaration its merges change (ADR 0505 §2) 2026-10-22 14:23:54 +02:00
mesh-tools 8f2d22db0c Merge pull request 'Move route-proxy's program into its module and run it on the pinned alpine image (issues 1633, 1634, 1635, ADR 0480 §1-§4, §11)' (#433) from feat/r4d into main 2026-10-22 12:19:07 +00:00
jschoubben cc8a74fce4 issue 1840/S3: take the SDK that names declared-packages 2026-10-22 14:18:39 +02:00
mesh-tools aa9da94260 Merge pull request 'A build node leaves its builds' images behind no more, and takes no work under a floor of free space (issue 1834)' (#435) from fix/1834-build-node-disk into main 2026-10-22 12:05:14 +00:00
jschoubben 01f615aa0a Read the bus release from the nats module's upstream artifact, so the tests' bus is still held to the mesh's once the Dockerfile is gone (issue 1640)
mesh/merge-gate head 01f615aa0ad6 merged into trunk aa9da942606d — pass: group feat/r4d composed: every machine composes with the change as it did without …
mesh/delivery delivered
mesh/repo-check head 01f615aa0ad6 merged into trunk aa9da942606d — pass: its merge-check.sh passed
mesh/delivery-group group feat/r4d delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 01f615aa0ad6 on aa9da942606d | unit 3345 run, 0 kept, 7 skipped | integration…
mesh-catalog #434 runs the nats server as the upstream image named in module.json (ADR 0497) and
deletes modules/nats/Dockerfile, so the check failed opening it. The manifest's upstream nats
artifact is read first; the Dockerfile only for a catalogue from before the move (the captured copy).
2026-10-22 13:38:00 +02:00
jschoubben f501eaca43 Remove an image's pushed names with its build tag, and take no work under the free-space floor (issue 1834)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@01f615aa0ad6 (merged as 8f2d22db into main, walk plan-17926707…
mesh/merge-gate head f501eaca4305 merged into trunk 4463991e0609 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f501eaca4305 merged into trunk 4463991e0609 — pass: its merge-check.sh passed
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The registry push tags the image a second time, so removing only the build tag freed nothing.
The held image a comparison pulls goes after it is read, and a node under the floor leaves the ask
to another build node instead of taking it and failing.
2026-10-22 13:19:58 +02:00
jschoubben 995f8237ce Free a build node's disk after every build and refuse work under a floor of free space (issue 1834)
A full / made every check on shanks end 'store never became ready' with the reason erased by --rm.
Builds now remove their local image tag, narrowed source and context clone; a build or check
refuses under 20 GiB free on / and the workspace, saying so; the throwaway store is no longer
started with --rm and a store that does not come up reports its exit code and last 20 log lines.
2026-10-22 13:14:00 +02:00
jschoubben 707bfba4ea Check an image's Dockerfile place instead of refusing the image kind (ADR 0487 clause 13)
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 707bfba4ea36 merged into trunk 4463991e0609 beside mesh-catalog#434@68183ee4 — fail: its merge-check.sh failed: --- FAIL: TestTheBusTh…
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
2026-10-22 11:45:06 +02:00
mesh-tools 4463991e06 Merge pull request 'Compose a pull request's check with the config values held for it (issue 1826, ADR 0502)' (#434) from fix/1826-held-config into main 2026-10-22 09:38:35 +00:00
jschoubben 4357b3688f Tell mesh-config what a head declares at every head, so a head that reverts a key drops its held value (issue 1826)
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
mesh/repo-check head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: its merge-check.sh passed
mesh/merge-gate head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: group fix/1826-held-config composed: every machine composes with the change as it …
mesh/delivery delivered
mesh/delivery-group group fix/1826-held-config delivered: every member is delivered
2026-10-22 11:07:34 +02:00
jschoubben 03927a6f6c Carry a config value held for a pull request's new key into its composed check (issue 1826)
The controller tells mesh-config what a pull request declares, asks the approved held values with each check, and the build agent and merge gate lay them over the change's composition only: a key the trunk does not declare yet can then be approved before the merge (ADR 0502).
2026-10-22 11:01:12 +02:00
jschoubben 499ff5c218 Move the route proxy into its module and run it from the trust image, on mesh-sdk's public membership (issues 1633, 1634, 1635)
mesh/repo-check out of date: judged beside mesh-catalog#434 at c948f3a61c74, it is now at 7451e7edf4e5; judged again on a recheck or when it is merged
mesh/merge-gate group feat/r4d: composed again for the group's new heads
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
A bundle is built only from its module's directory (ADR 0480 clause 4), so the proxy leaves
examples/ and the controller's internal/broker; the broker keeps no second copy of what the
proxy shares with mesh-sdk. The image and its Dockerfile go, and a test holds that only
build-agent declares one (clause 11).
2026-10-22 09:53:56 +02:00
mesh-tools 3d4ad6fefb Merge pull request 'Limit what a build or check starts on a build node: a per-node build slice (issue 1798)' (#432) from fix/1798-limit-build-resources into main 2026-10-21 23:20:57 +00:00
jschoubben 6bf38072ca Say in the build-agent settings that changing one restarts the build slice (issue 1798)
mesh/merge-gate head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 01:01:56 +02:00
jschoubben b174d743c0 Name the build slice without a dash so it sits beside user.slice, and restart it when its limits change (issue 1798)
mesh/merge-gate building on ace since 00:57
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:56:48 +02:00
jschoubben 37119ea31f Fill the build slice's limits as the mesh fills a module's template today (issue 1798)
mesh/merge-gate building on ace since 00:53
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:38:02 +02:00
jschoubben af2c6b5e94 Run builds and checks in a per-node build slice with CPU, IO and memory limits (issue 1798) 2026-10-22 00:24:45 +02:00
mesh-tools a929df9b0e Merge pull request 'The checker runs beside today's check: checks wait while a delivery builds; a group's SDK member merges; a flaky test fixed' (#430) from feat/the-checker into main 2026-10-21 21:46:26 +00:00
mesh-tools a3c965b9a0 Merge pull request 'Remove a check's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)' (#431) from fix/1772-check-volumes into main 2026-10-21 21:15:07 +00:00
jschoubben b9ec1919b8 Name plans-building among the controller's readable states and pin the SDK that does (issue 1687, ADR 0483 clause 2)
mesh/repo-check head b9ec1919b865 merged into trunk a3c965b9a0d5 beside mesh-catalog#423@12ff1cfb — pass: its merge-check.sh passed
mesh/merge-gate head b9ec1919b865 merged into trunk a3c965b9a0d5 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker delivered: every member is delivered
2026-10-21 22:47:04 +02:00
jschoubben d491356f59 Remove a build's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)
mesh/merge-gate head d491356f59dd merged into trunk 1f3fdf311167 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head d491356f59dd merged into trunk 1f3fdf311167 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 22:40:16 +02:00
jschoubben 47e1819101 Merge remote-tracking branch 'origin/fix/1106-sdk-member-merges' into feat/the-checker
mesh/repo-check out of date: judged beside mesh-sdk#66 at 78364e318bdc, it is now at 580da61a9107; judged again on a recheck or when it is merged
mesh/delivery-group group feat/the-checker checking: 4 of 4 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 47e1819101a1 merged into trunk 1f3fdf311167 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
2026-10-21 22:01:42 +02:00
jschoubben 0e25ed7c5a Merge remote-tracking branch 'origin/fix/1399-person-word-test' into feat/the-checker 2026-10-21 22:01:42 +02:00
jschoubben 79e2923dc8 Merge remote-tracking branch 'origin/fix/1652-checks-wait' into feat/the-checker
# Conflicts:
#	internal/broker/testdata/composed.conf
#	internal/link/builds_nats.go
2026-10-21 22:01:42 +02:00
mesh-tools 1f3fdf3111 Merge pull request 'Judge a duplicate hand-out before marking a build in hand, and never hand every build back over a stale mark (issue 1768)' (#429) from fix/1768-in-hand-marker into main 2026-10-21 19:59:30 +00:00
jschoubben 9902a3f16c Name every group member's trunk in the composed verdict, the announced one when the seat names none (issue 1106) 2026-10-21 21:57:30 +02:00
jschoubben 11815c0fb3 Have the delivery test's view hear a plan a verb let go, so a loaded machine cannot read it still waiting (issue 1399) 2026-10-21 21:45:58 +02:00
mesh-tools f973cefd7f Merge pull request 'Ask the operator for a held move and a failed release, and send no image the store lacks (issues 1732, 1320, 1742)' (#428) from fix/1732-held-moves-asked into main 2026-10-21 19:07:51 +00:00
jschoubben 696300b9e2 Judge a duplicate hand-out before the in-hand claim, and treat a stale mark as stale (issue 1768)
mesh/merge-gate head 696300b9e280 merged into trunk f973cefd7fd3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 696300b9e280 merged into trunk f973cefd7fd3 — pass: its merge-check.sh passed
mesh/delivery delivered
A dropped duplicate left the in-hand marker, so every later claim failed and every ask spent its deliveries.
2026-10-21 20:57:12 +02:00
jschoubben e7cc315fd0 Guard the taking loop's subscriptions, and update the two tests that held a busy machine's pulls
The loop now lets go of and binds queues around every ask, so Close takes the lock the loop does;
the busy machine's test says it holds no pull, and the stray test keeps its long check alive
within its two-second ack wait.
2026-10-21 20:53:58 +02:00
mesh-tools 98155e327b Merge pull request 'Take mesh-sdk at 07a6592: an item's text places a config value (issue 1658), and fit the unit level in its bound (issue 1763)' (#420) from fix/1656-xterm-config into main 2026-10-21 18:36:39 +00:00
jschoubben 60253c82b7 Fix the review of the checks-wait milestone: redeliveries, deleted modules, the kept queue
A check pending redelivery is read before the let-through fetches (none is handed back);
a deleted module is done for the plan's key; the queue an ask came from is let go of when
it ends. Tries say they run under their own id.
2026-10-21 20:17:37 +02:00
jschoubben d6db334840 Offer no Release on a machine that also holds a person's own hold (issue 1732, review round 1)
mesh/repo-check head d6db334840a5 merged into trunk 98155e327b6d beside mesh-host#129@09c77d57 — pass: its merge-check.sh passed
mesh/merge-gate head d6db334840a5 merged into trunk 98155e327b6d — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/delivery delivered
mesh/delivery-group group fix/1732-held-moves-asked delivered: every member is delivered
2026-10-21 20:15:58 +02:00
jschoubben 69d016bfdf Merge remote-tracking branch 'origin/main' into fix/1732-held-moves-asked 2026-10-21 20:15:13 +02:00
jschoubben bb8f076cc0 Merge remote-tracking branch 'origin/main' into fix/1652-checks-wait 2026-10-21 20:14:12 +02:00
jschoubben a58b38cee3 Keep the missing image out of the condition's summary (issue 1742) 2026-10-21 20:06:10 +02:00
jschoubben 9d260412b3 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/repo-check head 9d260412b330 merged into trunk 44033f4b6d72 beside mesh-catalog#416@b9dcf19c — pass: its merge-check.sh passed
mesh/merge-gate head 9d260412b330 merged into trunk 44033f4b6d72 — pass: group fix/1656-xterm-config composed: every machine composes with the change as it…
mesh/delivery-group group fix/1656-xterm-config failed: a member failed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792609673527907956 (novox/mesh-controller at f973cefd into ma…
2026-10-21 19:59:59 +02:00
jschoubben 99610e9569 Cut the controller's slowest unit tests (a source scan that regexed every file, resolver stand-in timeouts, a wait bound) so mesh-controller's unit level fits its bound (issue 1763) 2026-10-21 19:59:58 +02:00
jschoubben bb4d2e1da4 Let go of the other queues while an ask runs, and read the let-through race right
A message that reaches a builder's subscription outside a fetch was held unheartbeated for
the whole run of another ask, behind which a delivery waited (issue 1747). The queues the
running ask did not come from are let go of and bound again after it.
2026-10-21 19:56:23 +02:00
mesh-tools 44033f4b6d Merge pull request 'Issue a bus process's account when its module is assigned or gains it, and warn of one still missing instead of composing it silently (issue 1226)' (#427) from fix/1226-issue-process-accounts into main 2026-10-21 17:53:47 +00:00
jschoubben dc06af1f5a Leave a module out of a send when the store lacks its image, and say it (issue 1742) 2026-10-21 19:43:36 +02:00
jschoubben b62fbeabbf Ask a held move and the builds held after a failed release, and say them on the plan's line (issue 1732, 1320) 2026-10-21 19:43:36 +02:00
jschoubben 373aefd14e Take mesh-sdk at 07a6592, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config checking: composed check out of date: novox/mesh-sdk@c8e35097e861 was judged against trunk 85b5a2687e6a, the tr…
mesh/merge-gate building on ace since 19:52
mesh/repo-check out of date: judged against trunk 22c9815f6522, the trunk is now 44033f4b6d72; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:43:32 +02:00
jschoubben f66458fe25 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config 2026-10-21 19:43:02 +02:00
mesh-tools 22c9815f65 Merge pull request 'Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)' (#423) from fix/1310-failed-check-summary into main 2026-10-21 17:39:15 +00:00
jschoubben a14dc04b7b Hold the build seat's checks while a delivery builds, and let one check that waited 30 minutes through
ADR 0483: the controller keeps a key per plan in its build phase; holders read it and
take no new check while one is there, but for the oldest check past 30 minutes, one
at a time by a let-through key. The queue says why checks wait.
2026-10-21 19:32:47 +02:00
jschoubben 181b579ac6 Assert the next compose carries the issued account, not a stand-in (issue 1226, review round 2)
mesh/merge-gate head 181b579ac6ea merged into trunk 22c9815f6522 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 181b579ac6ea merged into trunk 22c9815f6522 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:27:22 +02:00
jschoubben b08c6372e1 Assert the next compose carries the issued account (issue 1226, review round 2) 2026-10-21 19:26:57 +02:00
jschoubben 4e11965a2a Merge remote-tracking branch 'origin/main' into fix/1226-issue-process-accounts 2026-10-21 19:26:00 +02:00
jschoubben faf8beee84 Warn instead of holding the machine for an unissued process account, and drive the delivery path in the test (issue 1226, review round 1) 2026-10-21 19:25:44 +02:00
mesh-tools 3827c3eb60 Merge pull request 'Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920)' (#426) from fix/920-plan-line-says-the-refusal into main 2026-10-21 17:23:01 +00:00
jschoubben 6bcfbeb5b2 Issue a bus process's account when its module is assigned or gains it (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben deac3e232c Refuse a process account's stand-in, naming issue (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben 99da867a98 Merge origin/main into fix/1656-xterm-config 2026-10-21 19:16:11 +02:00
jschoubben 612f493c49 Say only a refusal as a plan's hold, not a tier's summary left in its note (issue 920, review round 1)
mesh/merge-gate head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:01:42 +02:00
jschoubben 2377d75ed8 Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920) 2026-10-21 18:57:59 +02:00
mesh-tools 5ba2ffe8e6 Merge pull request 'Send a declaration's large files by reference through a bucket per node, and hold no check twice on the build seat' (#425) from feat/1696-declaration-files into main 2026-10-21 16:51:05 +00:00
jschoubben 8fc0d135bb Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files
mesh/merge-gate head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: its merge-check.sh passed
mesh/delivery delivered
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-21 18:32:56 +02:00
mesh-tools d568d76b97 Merge pull request 'Take mesh-sdk 90b20f5, so node-files' contract names upload and download (issue 1718, design 60 F3 group)' (#424) from feat/files-through-the-store into main 2026-10-21 16:31:27 +00:00
jschoubben 610d88e6bf Merge commit '3217f097d38d9f1c4302ec827ad6b371fd8ba95b' into feat/1696-declaration-files 2026-10-21 18:21:37 +02:00
jschoubben 0b11b6c98d Take mesh-sdk at its merge 725590e0 of the per-node bucket (issue 1694, ADR 0488) 2026-10-21 18:03:47 +02:00
jschoubben 92a5193e31 Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files 2026-10-21 18:03:06 +02:00
jschoubben 550cf37d71 Merge remote-tracking branch 'origin/main' into feat/files-through-the-store-files-verbs
mesh/repo-check head 550cf37d7109 merged into trunk d96c9f56a3f1 beside mesh-catalog#417@aef2a1a9 — pass: its merge-check.sh passed
mesh/merge-gate head 550cf37d7109 merged into trunk d96c9f56a3f1 — pass: group feat/files-through-the-store composed: every machine composes with the chang…
mesh/delivery superseded: its walk was superseded and names no walk that took it over: a walk that took it over answers its merge, and it follows that wa…
mesh/delivery-group group feat/files-through-the-store stopped: superseded, not carried by a later delivery: novox/mesh-catalog@6cf51173d586, novox/mesh-contro…
2026-10-21 17:49:29 +02:00
mesh-tools d96c9f56a3 Merge pull request 'Pin route-proxy to a mesh-sdk commit on main, so a fresh go mod download resolves it once the feature branch goes (issue 1669)' (#419) from fix/1669-route-proxy-pin into main 2026-10-21 15:12:49 +00:00
jschoubben 69a67876c2 Take mesh-sdk at 6f63275, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 69a67876c269 merged into trunk d96c9f56a3f1 beside mesh-catalog#416@f178aab6 — fail: its merge-check.sh failed: --- FAIL: TestModuleCh…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 17:03:35 +02:00
jschoubben bb31f6bc49 Take mesh-sdk d4deecdb, main's merge of node-files' upload and download, over main's container-groups pin (issue 1718)
mesh/delivery-group group feat/files-through-the-store checking: 0 of 2 member(s) ready
mesh/merge-gate building on g14 since 17:08
mesh/repo-check out of date: judged against trunk 12752911b644, the trunk is now d96c9f56a3f1; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 16:57:02 +02:00
jschoubben 516af253c6 Merge main into feat/1696-declaration-files, keeping mesh-sdk at 86f9a1a5, which holds main's pin 85b5a268 2026-10-21 16:46:21 +02:00
mesh-tools 12752911b6 Merge pull request 'Send a container's supplementary groups only to a node that reports container-groups, holding the module elsewhere (issue 1638, R4d)' (#422) from feat/r4d-groups into main 2026-10-21 14:44:55 +00:00
jschoubben 087e9266d1 Test the capability gate, the sweep's mark re-read and the lock use of a send and of the sweep, each failing under its mutation (issues 1696, 1697; review round 2) 2026-10-21 16:28:24 +02:00
jschoubben 3ac1944168 Announce the pull request's head again, so its delivery is made with its base and asks the Approve (issue 1726; tree unchanged)
mesh/merge-gate head 3ac1944168e4 merged into trunk 12752911b644 — pass: builds route-proxy → ace, novox; no bus step; every machine composes with the chan…
mesh/repo-check head 3ac1944168e4 merged into trunk 12752911b644 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 16:17:00 +02:00
jschoubben a3b4e0601d Take mesh-sdk at 86f9a1a5 and name each node's bucket with its FileBucketFor, refusing a node name that could carry a wildcard (issues 1696, 1697; ADR 0488) 2026-10-21 16:05:24 +02:00
jschoubben 227c86ebb3 Take mesh-sdk 90b20f5, so node-files' contract names upload and download (issue 1718)
mesh/delivery-group group feat/files-through-the-store rejected: a member's own check failed
mesh/merge-gate head 227c86ebb388 merged into trunk 12752911b644 — fail: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 227c86ebb388 merged into trunk 12752911b644 — fail: head 227c86ebb388 does not merge cleanly into trunk 12752911b644: in conflict go.m…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 15:57:18 +02:00
jschoubben da364c2730 Give each node a bucket of its own for the files that travel by digest, and keep the sweep from removing an object a send relies on (issues 1696, 1697; review of ADR 0486 step 3)
A literal * in a published subject matches the grant's *, so in one shared bucket a node could read every node's files. Each node now
has declaration-files-<node>, which its account reads by name and the controller writes; the sweep judges each bucket against that
node's latest declaration, under the node's lock with the send, and an object the sweep marked is put again by the next send. Putting
that fails send after send raises a condition.
2026-10-21 15:43:36 +02:00
jschoubben 4771ee53d9 Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)
mesh/merge-gate out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@181b579ac6ea (merged as 44033f4b into main, walk plan-17926043…
2026-10-21 15:21:10 +02:00
jschoubben 2aa291f3b3 Pin mesh-sdk at its merge of the container-groups capability, on main (issue 1638)
mesh/repo-check head 2aa291f3b30b merged into trunk 98ffcb667fb2 — pass: its merge-check.sh passed
mesh/merge-gate head 2aa291f3b30b merged into trunk 98ffcb667fb2 — pass: group feat/r4d-groups composed: every machine composes with the change as it did w…
mesh/delivery delivered
mesh/delivery-group group feat/r4d-groups delivered: every member is delivered
2026-10-21 15:16:47 +02:00
jschoubben ceaa616d01 Send files over 64 KiB by digest through the bus's object bucket to nodes that read them, grant it, and warn at 75 percent (issues 1696, 1697, 1698; ADR 0486 step 3)
A node that reports file-references gets the digest and size of each file over 64 KiB that may travel so, and the content is put in
the bucket declaration-files before the declaration is sent; others get every content inline. The controller alone writes the
bucket and node-engines read an object by name (the four grants the node-engine's own fetch makes, no list, no write). An object
goes a day after no node's latest declaration names it. The near-limit condition warns at 75 percent, and a test holds the control
node's declaration, in the reference form, under half the bus's largest message.
2026-10-21 15:16:11 +02:00
jschoubben fa5b5e1c11 Take mesh-sdk at 863a981e, which carries the file references of ADR 0486 (issue 1696) 2026-10-21 15:07:36 +02:00
jschoubben 6ef78580ce Take the mesh-sdk that holds the container-groups capability rebased on main (issue 1638) 2026-10-21 15:04:37 +02:00
jschoubben d8d6d196a4 The controller holds a module whose container names groups on a node that does not report container-groups, naming the capability (issue 1638, ADR 0480 clause 7) 2026-10-21 14:30:26 +02:00
jschoubben a474762527 Pin route-proxy to a commit of mesh-sdk main, since the pinned one is on a feature branch only (issue 1669)
mesh/merge-gate head a47476252772 merged into trunk 98ffcb667fb2 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head a47476252772 merged into trunk 98ffcb667fb2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 14:25:34 +02:00
590 changed files with 107125 additions and 4435 deletions
+3 -12
View File
@@ -90,15 +90,6 @@ redis-provisioner-image:
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The proxy that turns a route grant into traffic reaching a workload.
PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
@@ -117,7 +108,7 @@ proxy-image:
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
@sh merge-check.sh ; status=$$? ; docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
@@ -135,7 +126,7 @@ fmt:
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker run -d --name $(PG_CONTAINER) -e POSTGRES_PASSWORD=check \
-p 127.0.0.1:$(PG_PORT):5432 $(PG_IMAGE) >/dev/null
@printf 'waiting for postgres'
@@ -147,7 +138,7 @@ postgres:
echo ' — never came up' ; docker logs $(PG_CONTAINER) | tail -20 ; exit 1
postgres-stop:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
clean:
rm -rf build/
+91 -2
View File
@@ -6,7 +6,11 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
@@ -73,14 +77,77 @@ func drainedTooLong(now time.Time) error {
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// inHand is true while this process has claimed a build and not yet released it. The mark in the temporary directory
// is only believed while it is: a mark with no build of this process behind it is stale (a crash, a fault in an
// earlier path) and is removed, because left alone it hands back every ask for ever (issue 1768).
var inHand atomic.Bool
// markRetry is how long an agent that cannot mark a build in hand takes no work, and markFault when it last failed to.
// Taking nothing is the point: an ask fetched and handed back at once, many times a second, spends all its deliveries.
const markRetry = 30 * time.Second
var (
markMu sync.Mutex
markFault time.Time
markSaid string
)
// markFaulty says the agent could not mark a build in hand and has not yet been able to since: it takes no work. Once
// markRetry has passed it tries the mark itself, with no ask in hand, so that an ask is fetched only when marking works.
func markFaulty(now time.Time) bool {
markMu.Lock()
failed := markFault
markMu.Unlock()
if failed.IsZero() {
return false
}
if now.Sub(failed) < markRetry {
return true
}
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil && errors.Is(err, os.ErrExist) && !inHand.Load() {
_ = os.Remove(inHandMark())
f, err = os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
}
if err != nil {
markFailed(err)
return true
}
_ = f.Close()
_ = os.Remove(inHandMark())
markMu.Lock()
markFault, markSaid = time.Time{}, ""
markMu.Unlock()
fmt.Fprintf(os.Stderr, "the mark of a build in hand can be made again (%s): taking builds\n", inHandMark())
return false
}
// markFailed notes that a build could not be marked in hand, and says it when it is a new reason.
func markFailed(err error) {
markMu.Lock()
defer markMu.Unlock()
markFault = time.Now()
if said := err.Error(); said != markSaid {
markSaid = said
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so this agent takes no build for %s and tries again: %v\n",
inHandMark(), markRetry, err)
}
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
if !inHand.Load() {
// Nothing of this process is in hand, so a mark already there is stale.
if err := os.Remove(inHandMark()); err == nil {
fmt.Fprintf(os.Stderr, "removed the mark of a build in hand (%s): no build of this agent is in hand\n", inHandMark())
}
}
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
@@ -89,7 +156,7 @@ func claim() bool {
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
if drained() {
@@ -97,6 +164,7 @@ func claim() bool {
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
inHand.Store(true)
return true
}
@@ -107,6 +175,7 @@ func holding(what string) {
// released ends the build in hand.
func released() {
inHand.Store(false)
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
@@ -144,3 +213,23 @@ func busy() (string, error) {
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
var (
roomMu sync.Mutex
roomSaid bool
)
// noRoom says this node is under the floor of free space (novox/hq issue 1834): it takes no work, so another build node
// takes the ask instead of this one taking it and ending it in an error. Said once when it starts and again when it ends.
func noRoom(workspace string) bool {
err := builder.RoomToWork("/", workspace)
roomMu.Lock()
defer roomMu.Unlock()
if err != nil && !roomSaid {
fmt.Fprintf(os.Stderr, "taking no builds: %v\n", err)
} else if err == nil && roomSaid {
fmt.Fprintf(os.Stderr, "there is room again: taking builds\n")
}
roomSaid = err != nil
return err != nil
}
+77
View File
@@ -1,6 +1,7 @@
package main
import (
"github.com/novox/mesh-controller/internal/builder"
"os"
"strings"
"sync"
@@ -160,3 +161,79 @@ func TestAnUnknownCommandFails(t *testing.T) {
t.Fatalf("help failed: %v", err)
}
}
// 1768/AC2: a mark naming no build of this agent (left by a crash or a fault) is stale: the next ask removes it and is
// run, never fetched and handed back in a loop.
func TestAStaleInHandMarkIsRemovedAndTheAskRuns(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
inHand.Store(false)
if err := undrained(); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(inHandMark(), []byte("2026-10-21T18:45:06Z\n"), 0o644); err != nil {
t.Fatal(err)
}
if !claim() {
t.Fatal("a stale in-hand mark made the agent hand back an ask")
}
// A live one — this agent's own build in hand — is never taken over.
if claim() {
t.Fatal("a second build was taken while one is in hand")
}
if _, err := busy(); err != nil {
t.Fatalf("the build in hand was lost: %v", err)
}
released()
}
// 1768/AC3: an agent that cannot mark a build in hand for another reason takes no work, instead of fetching asks and
// handing them back; it takes work again once marking works.
func TestAnAgentThatCannotMarkABuildTakesNoWorkUntilItCan(t *testing.T) {
dir := t.TempDir()
t.Setenv("TMPDIR", dir)
inHand.Store(false)
markFault, markSaid = time.Time{}, ""
t.Cleanup(func() { markFault, markSaid = time.Time{}, "" })
if err := undrained(); err != nil {
t.Fatal(err)
}
// The mark's place is a directory that cannot be removed or made over: a directory named as the mark.
if err := os.MkdirAll(inHandMark()+"/x", 0o755); err != nil {
t.Fatal(err)
}
if claim() {
t.Fatal("an agent that cannot mark a build took it")
}
if !markFaulty(time.Now()) {
t.Fatal("an agent that cannot mark a build still takes work")
}
// Still failing after the retry time: still takes nothing, and no ask was needed to find out.
if !markFaulty(time.Now().Add(2 * markRetry)) {
t.Fatal("an agent whose mark still cannot be made takes work")
}
if err := os.RemoveAll(inHandMark()); err != nil {
t.Fatal(err)
}
if markFaulty(time.Now().Add(4 * markRetry)) {
t.Fatal("an agent whose mark can be made again takes no work")
}
if !claim() {
t.Fatal("an agent that can mark takes no build")
}
released()
}
// 1834/AC4: given a node under the floor of free space, when the taking loop asks, then it takes nothing, and takes again
// once there is room.
func TestANodeUnderTheFloorOfFreeSpaceTakesNoWork(t *testing.T) {
was := builder.FreeBytes
t.Cleanup(func() { builder.FreeBytes = was })
builder.FreeBytes = func(string) (uint64, error) { return 2 << 30, nil }
if !noRoom(t.TempDir()) {
t.Error("a node with 2 GiB free took work")
}
builder.FreeBytes = func(string) (uint64, error) { return 200 << 30, nil }
if noRoom(t.TempDir()) {
t.Error("a node with 200 GiB free refused work")
}
}
+14
View File
@@ -36,3 +36,17 @@ func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
// 1826/AC2: the held values a check request carries reach the spec; a request that carries none leaves it empty.
func TestTheHeldValuesAnAskCarriesReachTheSpec(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog",
HeldConfig: []link.HeldConfig{{Pull: "novox/mesh-catalog#434", Head: "a", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}}})
if len(spec.Held) != 1 || spec.Held[0].Node != "ace" || spec.Held[0].Key != "media-gid" || string(spec.Held[0].Value) != "2000" ||
spec.Held[0].Approved != "the operator" {
t.Fatalf("the spec reads %+v", spec.Held)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Held != nil {
t.Fatalf("a request that carries none has %+v", plain.Held)
}
}
+2 -2
View File
@@ -91,7 +91,7 @@ func TestKillEndsTheBuildRunningHereAndNoOther(t *testing.T) {
t.Errorf("kill said %q", said)
}
// Removed at the kill and again once the build had ended: a container made in between is caught.
if len(removed) != 4 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[2], "label=mesh.build=build-1") {
if len(removed) != 6 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[3], "label=mesh.build=build-1") {
t.Errorf("removed %v", removed)
}
if c := h.current(); c.Running != nil {
@@ -160,7 +160,7 @@ func TestAHolderStartingRemovesWhatEarlierBuildsLeftHere(t *testing.T) {
}
return "", nil
}
if n := h.removeLeftBehind(); n != 2 || len(removed) != 2 || removed[1] != "docker rm -f c1 c2" {
if n := h.removeLeftBehind(); n != 2 || len(removed) != 3 || removed[2] != "docker rm -f -v c1 c2" {
t.Fatalf("removed %d: %v", n, removed)
}
h.remove = func(context.Context, string, string, ...string) (string, error) {
+11 -1
View File
@@ -64,6 +64,8 @@ is dialled except the broker.
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
MESH_WORKSPACE where to clone and build (default: a temporary directory)
MESH_BUILD_CGROUP_PARENT the slice every container a build or check starts joins, which carries the
machine's limits on builds (novox/hq issue 1798; default: the runtime's own)
It also builds one module and stops, which is how a mesh is raised — before there is a
broker to take work from or a registry to publish into:
@@ -137,6 +139,10 @@ func run() error {
if err != nil {
return err
}
// A build and a check run below the machine's own work (novox/hq issue 1798): the agent itself at the
// lowest priority, which its git and go children inherit, and every container in the build slice.
_ = syscall.Setpriority(syscall.PRIO_PROCESS, 0, 19)
builder.CgroupParent = strings.TrimSpace(os.Getenv("MESH_BUILD_CGROUP_PARENT"))
workspace := os.Getenv("MESH_WORKSPACE")
if workspace == "" {
workspace = os.TempDir() + "/mesh-builder"
@@ -184,7 +190,7 @@ func run() error {
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Paused: func() bool { return h.Paused() || drained() || markFaulty(time.Now()) || noRoom(workspace) },
Taking: func() { taking(seat) },
Claim: claim,
})
@@ -499,6 +505,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, h := range c.HeldConfig {
spec.Held = append(spec.Held, builder.HeldValue{Pull: h.Pull, Head: h.Head, Node: h.Node, Module: h.Module, Key: h.Key,
Value: h.Value, Approved: h.Approved, By: h.By, At: h.At})
}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
+21 -2
View File
@@ -408,12 +408,21 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
_, reads := m.OwnSecrets["broker"]
if !reads && !runsTheProvisionerClient(m) {
// A module whose bus accounts are its processes' alone has none of its own: only the processes that
// are not issued yet are issued (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
if !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return ""
} else if minted {
// Its own account stands and is kept; a process it gained is not issued yet (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
busAddress, err := broker.BusAddress()
if err == nil {
@@ -426,6 +435,16 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return fmt.Sprintf("its bus credential is issued and sealed to %s, and arrives with the push", node)
}
// processAccountsOnAssign issues the process accounts of an assigned module not issued yet, and says what it did.
func processAccountsOnAssign(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string) string {
line, err := issueMissingProcessAccounts(ctx, inv, m, node)
if err != nil {
return fmt.Sprintf("its process accounts are not issued (%v): `module issue %s --node %s` first — "+
"`push %s` refuses to send %s until it is", err, m.Module, node, node, m.Module)
}
return line
}
// refusedMovingTheController refuses assigning or unassigning the controller's module through a verb (review of
// novox/hq ADR 0272): the node it is assigned to is the control-node, and the control-node's operator account is
// the controller's terminal, so whoever moves the module chooses the terminal. At the terminal alone, as every
@@ -31,6 +31,7 @@ func TestALookComesBackForATierTheBusRefusedWhole(t *testing.T) {
t.Fatal(err)
}
bus := false
asksOfEveryModule(t) // the one ask left is the builder's own (ADR 0453 §5): this test is of what a walk does around it
was := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
if !bus {
@@ -95,6 +96,7 @@ func TestALookFailsTheTierTheBusRefusedPastTheBound(t *testing.T) {
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
asksOfEveryModule(t) // the one ask left is the builder's own (ADR 0453 §5): this test is of what a walk does around it
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
@@ -114,11 +116,6 @@ func TestALookFailsTheTierTheBusRefusedPastTheBound(t *testing.T) {
t.Fatalf("a bus silent past its bound did not fail the plan in words: %s %q", got.State, got.Note)
}
// Rebuilt by hand while the bus still reloads: said as waiting, not as a failure and not with an empty reason.
joined, said, err := joinAPlan(ctx, open, "a")
if err != nil || !joined || !strings.Contains(said, "not asked yet: the bus did not answer") {
t.Fatalf("a rebuild left for the bus is said as %v %q %v", joined, said, err)
}
// And retried by hand: the module is named as waiting, not as asked again.
got, _ = inv.PlanByID(ctx, "plan-bus")
if got.State == inventory.PlanFailed {
@@ -15,6 +15,15 @@ import (
// seconds) is not a failed build. The module is left unasked and the plan building, and the next look asks it again; only a
// module's own fault, or a bus silent for ten minutes, fails the plan.
// asksOfEveryModule makes every module a test names the builder's own, the one module a walk still asks build-agent
// for (ADR 0453 §5): what a test of that ask's waits reads.
func asksOfEveryModule(t *testing.T) {
t.Helper()
was := buildersOwn
buildersOwn = func(string) bool { return true }
t.Cleanup(func() { buildersOwn = was })
}
func aTierOfThree() (*inventory.Plan, map[string]inventory.Entry) {
p := &inventory.Plan{ID: "plan-1", State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b", "c"}},
Modules: map[string]*inventory.PlanModule{}}
@@ -26,6 +35,7 @@ func aTierOfThree() (*inventory.Plan, map[string]inventory.Entry) {
}
func TestABusThatDoesNotAnswerForAMomentDoesNotFailTheTier(t *testing.T) {
asksOfEveryModule(t)
p, byName := aTierOfThree()
bus := false
was := askABuild
@@ -68,6 +78,7 @@ func TestABusThatDoesNotAnswerForAMomentDoesNotFailTheTier(t *testing.T) {
}
func TestABusSilentForTenMinutesFailsTheTierAsBefore(t *testing.T) {
asksOfEveryModule(t)
p, byName := aTierOfThree()
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
@@ -119,6 +130,7 @@ func TestAPlanWithAModuleWaitingForTheBusIsOneTheLookComesBackTo(t *testing.T) {
}
func TestAModulesOwnFaultStillFailsThePlan(t *testing.T) {
asksOfEveryModule(t)
p, byName := aTierOfThree()
delete(byName, "a")
askModule(t.Context(), p, "a", byName)
+23
View File
@@ -0,0 +1,23 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A retry of a plan failed past the bus's bound, whose module the bus still does not answer, names that module as waiting:
// read while the plan still says failed, it was named nowhere, and the retry said "asked" with nothing after it (issue
// 1403, met again once the hand rebuild no longer resumed the plan first).
func TestARetryNamesTheModulesStillWaitingForTheBus(t *testing.T) {
since := time.Now().UTC()
p := inventory.Plan{ID: "plan-bus", State: inventory.PlanFailed, Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {BusWaitFrom: &since}}}
waiting := busWaitingOnRetry(p)
_, said := retryWords(p, []string{"a"}, nil, waiting)
if !strings.Contains(said, "a not asked yet: the bus did not answer") || strings.Contains(said, "asked ;") {
t.Fatalf("a retry left for the bus is said as %q", said)
}
}
+4
View File
@@ -645,6 +645,10 @@ func doesWords(act conditions.Action) string {
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case act.Verb == broker.ControllerSeat+".push" && act.Arguments["node"] != "":
return "the held builds are sent to " + act.Arguments["node"]
case act.Verb == broker.ControllerSeat+".upgrade" && act.Arguments["release-backlog"] != "":
return "the builds held after the failed delivery are released"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
@@ -68,6 +68,7 @@ func TestAPlanGoesOnWithoutAMachineAwayWhenItsTierIsSent(t *testing.T) {
var mu sync.Mutex
var asks []string
var sends [][]string
asksOfEveryModule(t) // the one ask left is the builder's own (ADR 0453 §5): this test is of what a walk does around it
wasAsk := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
mu.Lock()
+5 -2
View File
@@ -232,8 +232,8 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
fmt.Print(nothingReadsIt(repository, m))
return nil
}
// **A module the merge adds is asked for now** (novox/hq issue 1617), not when the batch is cut.
askNewModulesAtHearing(ctx, inv, m, entries, read, facts.from)
// A module the merge adds is registered from the builder's first done key of it, which the builder's write wakes
// (novox/hq ADR 0501 §6): nothing is asked here.
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, holding, err := groupWaitHolding(ctx, inv, m, deliverySeatHeld(entries))
@@ -800,6 +800,9 @@ type keptMerge struct {
// FirstBuild says the merge is a first merge into a repository the mesh builds nothing from, kept to ask the first
// build of its modules once mesh-delivery's word on it is kept (novox/hq ADR 0432, issue 1146).
FirstBuild bool `json:"first_build,omitempty"`
// Held says which first builds of the merge were refused for a word the running build-agent does not read: the merge
// waits for a build-agent that reads them, and is never failed (novox/hq ADR 0482 §5, issue 1173).
Held []heldFirstBuild `json:"held,omitempty"`
}
// announcedOf is a kept merge as the forge announced it, and what it moved; empty where the record says none.
+113 -296
View File
@@ -28,112 +28,6 @@ import (
// cheapest next step. That is how novox/hq ADR 0001 records `hal/sdk` reaching 34,636:
// nothing in it was wrong, and no one edit was the one that should have been a new file.
// buildCommand builds a module from its source and records what came out.
//
// **Run where there is a container runtime**, which is why it is a command rather than something
// the control plane does on its own: building needs to run things on a machine, and what the
// control plane may send a machine is bounded by the declaration language. This is the shape the
// builder module will take when it is given work over the broker; today a person runs it, and the
// mesh records the result the same way either way.
// buildOn rebuilds every module the mesh holds that stands on the named module's artifacts — the
// rebuild a changed base needs, which nothing else asks for: their sources did not move, and
// "behind" does not see a base that did (novox/hq 04-ISSUES/131). Bases first among them too.
func buildOn(ctx context.Context, base string, wait time.Duration) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
held, err := open.inventory.Catalogued(ctx)
if err != nil {
return err
}
against, err := open.inventory.BuiltAgainst(ctx)
if err != nil {
return err
}
var on []inventory.Entry
for _, e := range held {
if standsOnModule(e, base, against) {
on = append(on, e)
}
}
if len(on) == 0 {
fmt.Printf("nothing the mesh holds stands on %s\n", base)
return nil
}
on = orderByBases(on, against)
fmt.Printf("%d module(s) stand on %s:\n", len(on), base)
var failed []string
for _, e := range on {
fmt.Printf("--- %s\n", e.Manifest.Module)
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, wait); err != nil {
fmt.Printf(" %v\n", err)
failed = append(failed, e.Manifest.Module)
}
}
if len(failed) > 0 {
return fmt.Errorf("%d of %d could not be built: %s", len(failed), len(on), strings.Join(failed, ", "))
}
fmt.Printf("\n%d module(s) rebuilt on %s. `push --behind` sends them on\n", len(on), base)
return nil
}
func buildCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("build", flag.ContinueOnError)
ref := set.String("ref", "", "the branch, tag or commit to build")
// A module is a repository and a path within it (novox/hq ADR 0069). Empty is the repository's
// root, which is the ordinary case and why this is a flag rather than a second argument.
path := set.String("path", "", "the module's directory inside the repository")
wait := set.Duration("wait", 10*time.Minute, "how long to wait for a builder to answer")
dryRun := set.Bool("dry-run", false, "build and print the manifest, recording nothing")
// Every module whose source has moved, rather than one named repository.
//
// **The mirror of `push --behind`, and the same argument** (novox/hq ADR 0010): the mesh
// already knows which modules are behind their source, so making a person read that list and
// retype each repository is asking them to be the loop. Naming a repository and asking which
// ones need building are different requests, so they are not combined.
behind := set.Bool("behind", false, "every module the mesh holds older than its source has")
on := set.String("on", "", "rebuild every module that stands on this module's artifacts — the rebuild a changed base needs")
// A repository on the mesh's own forge, named by its path there (novox/hq ADR 0111). Without it
// the repository is external, cloned exactly as given — see source.go.
self := set.Bool("self", false, "the repository is a path on the forge holding the git seat")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if *on != "" {
if len(positionals) != 0 || *behind || *self {
return errors.New("build --on <module> names a base and nothing else")
}
return buildOn(ctx, *on, *wait)
}
if *behind {
if len(positionals) != 0 || *self {
return errors.New("build <repository> or build --behind, not both: one names a " +
"repository and the other asks which need building")
}
return buildBehind(ctx, *wait)
}
if len(positionals) != 1 {
return errors.New("build <repository> [--self] [--path P] [--ref R] [--wait D] [--dry-run] | build --behind | build --on <module>")
}
source := buildSource{Repository: positionals[0]}
if *self {
if err := onASeat(source.Repository); err != nil {
return err
}
source.Seat = gitSeat
}
if *dryRun {
return buildAndShow(ctx, source, *path, *ref, *wait)
}
return buildOne(ctx, source, *path, *ref, *wait)
}
// buildFrom turns what a builder said into what the mesh keeps.
//
// **By digest and path, never by where it was pushed** (novox/hq 04-ISSUES/102). The builder
@@ -355,86 +249,6 @@ func builderCommand(ctx context.Context, args []string) error {
return issueOnTheNewBus(ctx, inv, m, node, address)
}
// buildBehind builds every module the mesh holds older than its source has.
//
// **This is the loop novox/hq ADR 0010 replaced a pipeline with, closed.** The mesh already
// records where each module came from and what its source last had; until this, a person read
// that list and retyped each repository — which is a person being the loop, and the thing a
// pipeline was doing before it was taken away.
//
// Each is built and recorded on its own. **One failing does not stop the others**, for the same
// reason one broken module no longer blocks a machine's whole declaration: a mesh where one bad
// repository holds back nine good ones is a mesh where nobody dares add the tenth.
func buildBehind(ctx context.Context, wait time.Duration) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
inv := open.inventory
held, err := inv.Catalogued(ctx)
if err != nil {
return err
}
var stale []inventory.Entry
for _, e := range held {
if !e.Source.Current() {
stale = append(stale, e)
}
}
if len(stale) == 0 {
// Said rather than doing nothing quietly: "nothing needed building" and "this did not
// run" must never look the same.
fmt.Println("every module the mesh holds is what its source last had")
return nil
}
fmt.Printf("%d module(s) behind their source:\n", len(stale))
for _, e := range stale {
fmt.Printf(" %s %s < %s\n",
e.Manifest.Module, short(e.Source.BuiltFrom), short(e.Source.Head))
}
fmt.Println()
// Bases first: a module built before the module it stands on is built against the old one
// and reports success (novox/hq 04-ISSUES/131).
against, err := inv.BuiltAgainst(ctx)
if err != nil {
return err
}
stale = orderByBases(stale, against)
var failed []string
for _, e := range stale {
fmt.Printf("--- %s\n", e.Manifest.Module)
// Its own recorded ref, not its head commit: a module tracking a branch should be built
// from that branch, and pinning to the commit the mesh happened to notice would quietly
// turn a tracked branch into a pin.
source := buildSource{Repository: e.Source.Repository, Seat: e.Source.Seat}
if err := buildOne(ctx, source, e.Source.Path, e.Source.Ref, wait); err != nil {
fmt.Printf(" %v\n", err)
failed = append(failed, e.Manifest.Module)
}
}
if len(failed) > 0 {
return fmt.Errorf("%d of %d could not be built: %s",
len(failed), len(stale), strings.Join(failed, ", "))
}
fmt.Printf("\n%d module(s) built. `push --behind` sends them to the machines running them\n",
len(stale))
return nil
}
// buildOne asks a build machine for one repository and records everything that came back.
//
// Separated from the command so `--behind` can walk a list without a second path to the same act.
func buildOne(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
_, err := buildOneAsked(ctx, source, path, ref, wait, false, "build")
return err
}
// recordAsked keeps a build request once it is asked (novox/hq issue 325), with who asked it, in a store
// opened for the purpose: the asks reach here from processes that hold none.
func recordAsked(ctx context.Context, r inventory.BuildRequest) {
@@ -511,7 +325,6 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
Seats: seatBases(ctx),
DryRun: dryRun,
}
request.Reuse = reuseAsked(ctx, dryRun)
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
@@ -679,13 +492,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
recorded.Ref = followedBranch(was.Ref)
}
}
if err := namesNoInstallation(manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
result.On, result.Repository, short(result.Commit), err)
}
// **Only from the repository the catalogue builds the module from** (novox/hq ADR 0266): else the trunk
// below is the trunk of whatever repository was built, which may be one an agent made — and a module named
// `sudo` from it would be what the next push sends. Another repository is the operator's, at the terminal.
trunk := result.Trunk
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil && followedBranch(was.Ref) != "" {
trunk = followedBranch(was.Ref)
@@ -693,55 +499,111 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
if trunk == "" {
trunk = "main"
}
repoID, err := mayRegisterFrom(ctx, inv, manifest.Module, recorded, result.ID, result.Path, trunk)
j := takenBuild{kept: kept, manifest: manifest, recorded: recorded, ask: byBuild(result.ID), path: result.Path,
trunk: trunk, by: result.On, what: result.Repository, commit: result.Commit}
// The checks only an outcome of build-agent needs: a key is a trunk build by construction (ADR 0501 §2), in no
// scratch namespace, and carries no list of its own.
j.published = func() error {
// **Only a commit on the trunk is published** (novox/hq ADR 0238): a commit off its repository's
// default branch — a pull request's head, a feature branch built by hand, a `rebuild` or `replay
// --register` of one — is for checking, and is never a module's version; nothing could then send it.
// The branch the module already follows is its trunk — never the branch a build was asked at, or a
// build of a feature branch by name would make that branch its trunk.
follows := ""
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil {
follows = followedBranch(was.Ref)
}
if err := publishable(result, follows); err != nil {
return err
}
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return fmt.Errorf("%s is in the scratch namespace, which only a walk's copy leaves (ADR 0350)", scratch)
}
return nil
}
j.carried = func() error {
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept.
carried, err := carriedList(result)
if err != nil {
return err
}
if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
return nil
}
manifest, err = registerJudged(ctx, inv, j)
return manifest, kept, err
}
// takenBuild is a build the controller judges and registers, whatever told it of the build: build-agent's outcome (the
// builder's own module's, until R4g, ADR 0453 §5) or the builder's done key (ADR 0501 §1). **One judging for both**:
// the rules of who may register, the gate a build failed, the predefined keys and the rule that an unchanged build
// source moves nothing are the same whichever part built it (ADR 0501 §1, §3).
type takenBuild struct {
kept inventory.Build
manifest catalogue.Manifest
recorded inventory.Source
ask registrationAsk
path string
trunk string
// by and what name the build in what is said: who built it, and what was built ("ace", "novox/mesh-catalog").
by, what, commit string
// published refuses a build that may not become a module's version (off the trunk, in the scratch namespace);
// nil for a key, which is a trunk fingerprint's by construction (ADR 0501 §2).
published func() error
// carried publishes a list of predefined keys the build carried; nil when it carried none.
carried func() error
}
// registerJudged judges a recorded build and registers its module, or says why not.
func registerJudged(ctx context.Context, inv *inventory.Inventory, j takenBuild) (catalogue.Manifest, error) {
manifest, kept := j.manifest, j.kept
notRegistered := func(err error) (catalogue.Manifest, error) {
return manifest, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", j.by, manifest.Module,
short(j.commit), err)
}
if err := namesNoInstallation(manifest); err != nil {
return manifest, fmt.Errorf("%s built %s (%s), and the mesh does not register it: %w",
j.by, j.what, short(j.commit), err)
}
// **Only from the repository the catalogue builds the module from** (novox/hq ADR 0266): else the trunk
// below is the trunk of whatever repository was built, which may be one an agent made — and a module named
// `sudo` from it would be what the next push sends. Another repository is the operator's, at the terminal.
repoID, err := mayRegisterFrom(ctx, inv, manifest.Module, j.recorded, j.ask, j.path, j.trunk)
if err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
return notRegistered(err)
}
// **Only a commit on the trunk is published** (novox/hq ADR 0238): a commit off its repository's
// default branch — a pull request's head, a feature branch built by hand, a `rebuild` or `replay
// --register` of one — is for checking, and is never a module's version; nothing could then send it.
// The branch the module already follows is its trunk — never the branch a build was asked at, or a
// build of a feature branch by name would make that branch its trunk.
follows := ""
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil {
follows = followedBranch(was.Ref)
}
if err := publishable(result, follows); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %s is in the scratch "+
"namespace, which only a walk's copy leaves (ADR 0350)", result.On, manifest.Module, short(result.Commit), scratch)
if j.published != nil {
if err := j.published(); err != nil {
return notRegistered(err)
}
}
// **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard
// twice, or replayed, would otherwise make the build a rollback put back what the module is again,
// and the next push would send it.
if failed, err := inv.GateFailed(ctx, kept.ID); err != nil {
return manifest, kept, err
return manifest, err
} else if failed {
return manifest, kept, fmt.Errorf("%s built %s (%s), which failed its gate on its first machine and was put "+
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit))
return manifest, fmt.Errorf("%s built %s (%s), which failed its gate on its first machine and was put "+
"back: it is recorded and not registered again — a newer build is", j.by, manifest.Module,
short(j.commit))
}
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
// Then the module is judged against the list the mesh publishes, as every manifest read again is: one naming a
// key that list does not hold, or of another type, is refused here and never enters the catalogue.
if j.carried != nil {
if err := j.carried(); err != nil {
return manifest, err
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
return notRegistered(fmt.Errorf("judged against the list of predefined keys the mesh publishes, %w", err))
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
@@ -751,32 +613,34 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
if kept.SourceFingerprint != "" {
stands, raw, err := inv.StandingBuild(ctx, manifest.Module, kept.ID)
if err != nil {
return manifest, kept, err
return manifest, err
}
if stands != "" && stands != kept.ID && len(raw) > 0 {
if same, err := catalogue.ParseManifest(raw); err == nil && same.Module == manifest.Module {
if was, err := catalogue.ParseManifest(raw); err == nil && was.Module == manifest.Module {
fmt.Printf("%s at %s was made from the source %s was: registered with its artifacts, no move\n",
manifest.Module, short(result.Commit), stands)
manifest = same
manifest.Module, short(j.commit), stands)
manifest = was
}
}
}
if err := inv.RegisterModule(ctx, manifest, recorded); err != nil {
if err := inv.RegisterModule(ctx, manifest, j.recorded); err != nil {
if errors.Is(err, inventory.ErrSuperseded) {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w",
result.On, manifest.Module, short(result.Commit), err)
return notRegistered(err)
}
return manifest, kept, err
return manifest, err
}
// A bus process the new definition gains gets its account now, on each machine assigned the module, so the
// next push composes a real account (issue 1226).
issueGainedProcessAccounts(ctx, inv, manifest)
// Which repository it is registered from, by the forge's own id (novox/hq ADR 0266).
if err := inv.SetSourceIdentity(ctx, manifest.Module, repoID); err != nil {
return manifest, kept, err
return manifest, err
}
// The keep set just moved, and new bytes just landed (novox/hq ADR 0189). Asked here rather
// than on a timer of its own: this is the only moment either is true. Never fatal — the build
// worked and the module is registered.
collect(ctx, inv)
return manifest, kept, nil
return manifest, nil
}
// inScratch is the first artifact a build made into the scratch namespace; empty when none is.
@@ -817,64 +681,17 @@ func publishable(result link.BuildResult, follows string) error {
return nil
}
// buildAndShow builds and prints the manifest without recording anything.
func buildAndShow(ctx context.Context, source buildSource, path, ref string, wait time.Duration) error {
repository, err := cloneFrom(ctx, source)
if err != nil {
return err
}
if err := verbMayAsk(ctx, source, repository, path); err != nil {
return err
}
ident, err := openIdentity(ctx)
if err != nil {
return err
}
defer ident.Close()
server, err := connectLink(ctx, nil, nil, nil)
if err != nil {
return err
}
defer server.Close()
ask, err := askOverOn(buildSeatHeld(ctx))
if err != nil {
return err
}
defer ask.Close()
result, err := ask.Submit(ctx, link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository, Path: path, Ref: ref,
Held: heldBy(ctx), Seats: seatBases(ctx),
DryRun: true,
}, wait)
if err != nil {
return err
}
if result.Failed != "" {
return fmt.Errorf("%s could not build %s:\n%s", result.On, result.Repository, result.Failed)
}
manifest, err := catalogue.ParseManifest(result.Manifest)
if err != nil {
return fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
}
body, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
return nil
}
// answers is what the three questions came back with, read once.
type answers struct {
wrong []inventory.Doing
nodes []inventory.Node
quiet []inventory.Node
behind map[string][]string
sources map[string]inventory.Source
// notAnnounced are the owners some module reads the state of that have announced nothing yet (novox/hq ADR 0495,
// issue 1790): read from `reads` as before until they do; notAnnouncedWhy why that could not be read.
notAnnounced []string
notAnnouncedWhy string
wrong []inventory.Doing
nodes []inventory.Node
quiet []inventory.Node
behind map[string][]string
sources map[string]inventory.Source
// waiting is every machine not running what the mesh would send it.
waiting []inventory.Machine
// recordedWaits is, per machine, every move of a recorded module that waits there for a person (novox/hq ADR
+59 -59
View File
@@ -373,22 +373,7 @@ type protectionRule struct {
// branch, else the first glob rule, in the forge's order, that covers it (gitea's first matching rule). Never a
// later rule that happens to be stronger: the forge does not read it.
func judgedRule(rules []protectionRule, branch string) forgeFacts {
var applied *protectionRule
for i := range rules {
if rules[i].Rule == branch {
applied = &rules[i]
break
}
}
if applied == nil {
for i := range rules {
if ruleCovers(rules[i].Rule, branch) {
applied = &rules[i]
break
}
}
}
switch r := applied; {
switch r := appliedRule(rules, branch); {
case r == nil:
return forgeFacts{Why: fmt.Sprintf("no protection rule covers %s", branch)}
case r.Push:
@@ -401,6 +386,22 @@ func judgedRule(rules []protectionRule, branch string) forgeFacts {
return forgeFacts{Guarded: true}
}
// appliedRule is the one rule the forge applies to a branch: the rule named for it, else the first glob rule, in the
// forge's order, that covers it; nil when none does.
func appliedRule(rules []protectionRule, branch string) *protectionRule {
for i := range rules {
if rules[i].Rule == branch {
return &rules[i]
}
}
for i := range rules {
if ruleCovers(rules[i].Rule, branch) {
return &rules[i]
}
}
return nil
}
// ruleCovers says a protection rule's name — a branch, or a glob of them — covers a branch, as the forge reads it.
func ruleCovers(rule, branch string) bool {
if rule == branch {
@@ -469,8 +470,39 @@ func (f *sourceForms) onTheForge(repository, seat string) (owner, name string, o
// repository and path; or, for the build of a merge, when mesh-delivery announced it on the operator's Approve of
// that head (novox/hq ADR 0420 §7 for a moved module, ADR 0432 for a new one), the forge's checks still holding. path is the module's directory as built; branch the trunk it is registered from.
func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, buildID, path, branch)
ask registrationAsk, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, ask, path, branch)
}
// registrationAsk is what a registration's warrant is read from (novox/hq ADR 0501 §3): the build an outcome answers,
// whose kept request says whether it was asked at the terminal or for a merge (build-agent's outcome, the builder's own
// module until R4g); or, for a take-in from the builder's done key, the merge the indexer's trunk value names and
// whether the operator asked the registration at the controller's terminal (ADR 0501 §4). A key asks no build, so it
// keeps no request.
type registrationAsk struct {
// Build is the build whose kept request is read; empty for a take-in from a key.
Build string
// Merge is the commit of the merge a key's module is registered from (the indexer's trunk value); empty for an
// outcome, whose request names it.
Merge string
// Terminal is a key registration the operator asked at the controller's terminal for this repository and path.
Terminal bool
}
// byBuild is the warrant of a build's outcome: read from the request kept under its id.
func byBuild(id string) registrationAsk { return registrationAsk{Build: id} }
// String is the ask as the registration's words name it.
func (a registrationAsk) String() string {
switch {
case a.Build != "":
return a.Build
case a.Terminal:
return "the registration asked at the terminal"
case a.Merge != "":
return fmt.Sprintf("the merge %.8s", a.Merge)
}
return "a key"
}
// catalogueReader is the catalogue's manifests, as the router's records are found in them.
@@ -490,15 +522,19 @@ type registrations interface {
// mayRegisterIn is mayRegisterFrom on what it reads of the mesh handed in.
func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
ask registrationAsk, path, branch string) (int64, error) {
buildID := ask.String()
was, err := inv.SourceOf(ctx, module)
isNew := errors.Is(err, inventory.ErrNoSuchModule)
if err != nil && !isNew {
return 0, err
}
terminal, err := askedHereFor(ctx, inv, forms, buildID, built, path)
if err != nil {
return 0, err
terminal := ask.Terminal
if ask.Build != "" {
var err error
if terminal, err = askedHereFor(ctx, inv, forms, ask.Build, built, path); err != nil {
return 0, err
}
}
refuse := func(why string) (int64, error) {
return 0, fmt.Errorf("%w: %s. A module is registered from such a source only by a build asked at the "+
@@ -542,7 +578,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
// The forge's protection and the repository's identity are checked below as for every registration.
movedWhy := ""
if why != "" && !terminal {
accepted, err := approvedMove(ctx, inv, forms, buildID, built)
accepted, err := approvedMove(ctx, inv, forms, ask, built)
switch {
case err == nil:
movedWhy, why = accepted, ""
@@ -662,37 +698,6 @@ func sourceWords(repository, seat string) string {
return buildSource{Repository: repository, Seat: seat}.String()
}
// firstBuildWarrant is what the controller checked before it asks the first build of the modules a merge adds to a
// repository nothing is built from: the merge was made on the operator's Approve of its head (approvedAnnounced). It
// travels in the context of that ask alone, in this process, so no verb carries one.
type firstBuildWarrant struct {
repository string // owner/name on the git seat
paths map[string]bool
why string
}
type firstBuildWarrantKey struct{}
// withFirstBuildWarrant is ctx carrying the warrant for asking the first builds of paths in repository.
func withFirstBuildWarrant(ctx context.Context, repository string, paths []string, why string) context.Context {
w := &firstBuildWarrant{repository: repository, paths: map[string]bool{}, why: why}
for _, p := range paths {
w.paths[strings.Trim(p, "/")] = true
}
return context.WithValue(ctx, firstBuildWarrantKey{}, w)
}
func firstBuildWarrantIn(ctx context.Context) *firstBuildWarrant {
w, _ := ctx.Value(firstBuildWarrantKey{}).(*firstBuildWarrant)
return w
}
// covers says the warrant is for exactly this ask: the repository on the git seat, and a module path the merge holds.
func (w *firstBuildWarrant) covers(source buildSource, path string) bool {
return w != nil && source.Seat == gitSeat && strings.EqualFold(strings.Trim(source.Repository, "/"), w.repository) &&
w.paths[strings.Trim(path, "/")]
}
// verbMayAsk refuses, for a call through a verb, a build of a repository the catalogue builds no module from
// (novox/hq ADR 0266): the build node would run what an agent wrote, and its outcome could never be
// registered anyway. url is the repository as it is cloned. At the terminal anything may be asked.
@@ -700,11 +705,6 @@ func verbMayAsk(ctx context.Context, source buildSource, url, path string) error
if startedAtTheTerminal() {
return nil
}
// **Or the first build of a module a merge adds on the operator's Approve** (novox/hq ADR 0432, issue 1146): the
// controller's own ask, carrying the warrant it checked, for that repository and the module paths the merge holds.
if firstBuildWarrantIn(ctx).covers(source, path) {
return nil
}
open, err := openStores(ctx)
if err != nil {
return err
@@ -0,0 +1,139 @@
//go:build integration
package main
import (
"context"
"sync"
"testing"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// builderOnTheBus is the builder's and the indexer's buckets on a test's own bus, which the controller's connection for
// the moment (servingBus) reaches: what a take-in from a key reads (R4e, issue 1851).
type builderOnTheBus struct {
js *broker.JetStream
builds, modules, manifests jetstream.KeyValue
}
func aBuilderOnTheBus(t *testing.T) builderOnTheBus {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
ctx := t.Context()
b := builderOnTheBus{js: js}
for _, c := range []struct {
into *jetstream.KeyValue
bucket string
history uint8
}{
{&b.builds, sdkdelivery.BuilderBuildsBucket, sdkdelivery.BuilderBuildsHistory},
{&b.modules, sdkdelivery.ModulesBucket, sdkdelivery.ModulesHistory},
{&b.manifests, sdkdelivery.ManifestsBucket, 1},
} {
kv, err := api.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: c.bucket, History: c.history})
if err != nil {
t.Fatal(err)
}
*c.into = kv
}
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
withTrunkList(t, true)
// The indexer on this bus is read, not the "not held" a mesh with no bus reads (aMeshOnTheStore).
wasIndexed := indexedModule
indexedModule = theIndexedModule
t.Cleanup(func() { indexedModule = wasIndexed })
return b
}
// indexed writes the indexer's trunk value of module at commit, naming fp, and its manifest: a module of one archive.
func (b builderOnTheBus) indexed(t *testing.T, module, version, repository, path, commit, fp string) string {
t.Helper()
keys, trunk := aKeyFor(module, version, fp, repository, path, commit)
raw := keys.manifests[trunk.Manifest]
value, err := sdkdelivery.EncodeManifest(sdkdelivery.Manifest{Module: module, JSON: raw})
if err != nil {
t.Fatal(err)
}
if _, err := b.manifests.Put(context.Background(), trunk.Manifest, value); err != nil {
t.Fatal(err)
}
trunk.FP = "src1:" + fp[len("src2:"):]
encoded, err := sdkdelivery.EncodeModule(trunk)
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(context.Background(), module, encoded); err != nil {
t.Fatal(err)
}
return trunk.Manifest
}
// built writes the builder's key of module at fp: done, its one archive named with a reference, or in another state.
func (b builderOnTheBus) built(t *testing.T, module, version, fp, manifest, state string) {
t.Helper()
keys, _ := aKeyFor(module, version, fp, "", "", "")
v := keys.keys[module].Build
v.Manifest = manifest
if state != sdkdelivery.BuildDone {
v = sdkdelivery.BuilderBuild{State: state, By: "ace"}
if state == sdkdelivery.BuildFailed {
v.Error = "digests differ: ace sha256:1, g14 sha256:2"
}
}
value, err := sdkdelivery.EncodeBuilderBuild(v)
if err != nil {
t.Fatal(err)
}
key, err := sdkdelivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
if _, err := b.builds.Put(context.Background(), key, value); err != nil {
t.Fatal(err)
}
}
// noBuildAsked fails the test if anything asks build-agent for a build (ADR 0501 §6, ADR 0503 §6).
func noBuildAsked(t *testing.T) {
t.Helper()
was := askABuild
askABuild = func(_ context.Context, _ buildSource, path, _ string) (string, error) {
t.Errorf("a build was asked of build-agent, at %q", path)
return "", context.Canceled
}
t.Cleanup(func() { askABuild = was })
}
// asksWithPaths records every build a merge asks for, as repository, path and ref. The merge window's goroutine and
// the test's own may ask at once, so the record is written under a lock (issue 1543).
func asksWithPaths(t *testing.T) *[][3]string {
t.Helper()
asksOfEveryModule(t) // the one ask left is the builder's own (ADR 0453 §5): these tests are of what a walk does around it
var asked [][3]string
var mu sync.Mutex
was := askABuild
askABuild = func(_ context.Context, source buildSource, path, ref string) (string, error) {
mu.Lock()
defer mu.Unlock()
asked = append(asked, [3]string{source.Repository, path, ref})
return "b-" + path, nil
}
t.Cleanup(func() { askABuild = was })
return &asked
}
// theIndexedModule is the look at the indexer as the controller makes it, kept before any test replaces it.
var theIndexedModule = indexedModule
+112
View File
@@ -0,0 +1,112 @@
package main
import (
"context"
"errors"
"strings"
"testing"
)
// A12 (ADR 0453 §5, ADR 0498 §7; issue 1851): the one function every ask goes through refuses any module but the
// builder's own, naming the builder's key it is taken from instead; the builder's own module passes the refusal.
func TestBuildAgentIsAskedForTheBuildersOwnModuleAlone(t *testing.T) {
was := askedModuleOf
t.Cleanup(func() { askedModuleOf = was })
askedModuleOf = func(_ context.Context, _ buildSource, path string) (string, error) {
return strings.TrimPrefix(path, "modules/"), nil
}
_, err := askABuild(t.Context(), buildSource{Repository: "novox/mesh-catalog", Seat: "git"}, "modules/sudo", "main")
if !errors.Is(err, errNotTheBuildersOwn) || !strings.Contains(err.Error(), "sudo is taken from the builder's done key") ||
!strings.Contains(err.Error(), "mesh-delivery-builder_builds") {
t.Fatalf("an ask of another module: %v", err)
}
if err := onlyTheBuildersOwn(t.Context(), buildSource{Repository: "novox/mesh-delivery", Seat: "git"},
"modules/mesh-delivery-builder"); err != nil {
t.Fatalf("the builder's own module is refused: %v", err)
}
askedModuleOf = func(context.Context, buildSource, string) (string, error) { return "", errors.New("no store") }
if err := onlyTheBuildersOwn(t.Context(), buildSource{Repository: "novox/x"}, ""); !errors.Is(err, errNotTheBuildersOwn) {
t.Fatalf("an ask whose module cannot be read went on: %v", err)
}
}
// A12: a walk of the builder's own module and one other asks build-agent for the first alone; the other is taken from
// its key.
func TestAWalkAsksBuildAgentForTheBuildersOwnModuleAlone(t *testing.T) {
var asked []string
was := askABuild
askABuild = func(_ context.Context, _ buildSource, path, _ string) (string, error) {
asked = append(asked, path)
return "build-1", nil
}
t.Cleanup(func() { askABuild = was })
p, byName := aWalkOfOne("sudo")
p.Tiers = [][]string{{"mesh-delivery-builder", "sudo"}}
byName["mesh-delivery-builder"] = byName["sudo"]
e := byName["mesh-delivery-builder"]
e.Source.Repository, e.Source.Path = "novox/mesh-delivery", "modules/mesh-delivery-builder"
byName["mesh-delivery-builder"] = e
for _, m := range p.Tiers[0] {
askModule(t.Context(), p, m, byName)
}
if len(asked) != 1 || asked[0] != "modules/mesh-delivery-builder" {
t.Fatalf("build-agent was asked for %v, want the builder's own module alone", asked)
}
if s := p.Modules["mesh-delivery-builder"]; s.Key || s.Build != "build-1" {
t.Fatalf("the builder's own module is not asked: %+v", s)
}
if s := p.Modules["sudo"]; !s.Key || s.Build != "" {
t.Fatalf("the other module is not taken from its key: %+v", s)
}
}
// ADR 0520 §1 (issue 1851): until R4g build-agent and mesh-tools, the toolchain image its builds stand on, are build-agent's
// own, as the builder's module is: the builder writes no key of them, so a walk asks build-agent for them and reads no key;
// mesh-tools-go (its key from mesh-build-image) and any other module of the same tier are still taken from their keys.
func TestAWalkAsksBuildAgentForItselfAndMeshToolsAndReadsNoKeyOfThem(t *testing.T) {
own := []string{"build-agent", "mesh-tools"}
keyed := []string{"mesh-tools-go", "sudo"}
wasModule := askedModuleOf
askedModuleOf = func(_ context.Context, _ buildSource, path string) (string, error) {
return strings.TrimPrefix(path, "modules/"), nil
}
t.Cleanup(func() { askedModuleOf = wasModule })
for _, m := range own {
if err := onlyTheBuildersOwn(t.Context(), buildSource{Repository: "novox/x", Seat: "git"}, "modules/"+m); err != nil {
t.Fatalf("an ask of %s is refused: %v", m, err)
}
}
if err := onlyTheBuildersOwn(t.Context(), buildSource{Repository: "novox/x", Seat: "git"}, "modules/mesh-tools-go"); !errors.Is(err, errNotTheBuildersOwn) {
t.Fatalf("an ask of mesh-tools-go went on: %v", err)
}
var asked []string
was := askABuild
askABuild = func(_ context.Context, _ buildSource, path, _ string) (string, error) {
asked = append(asked, path)
return "build-" + strings.TrimPrefix(path, "modules/"), nil
}
t.Cleanup(func() { askABuild = was })
p, byName := aWalkOfOne("sudo")
p.Tiers = [][]string{append(append([]string{}, own...), keyed...)}
for _, m := range append(append([]string{}, own...), keyed...) {
e := byName["sudo"]
e.Source.Path = "modules/" + m
byName[m] = e
}
for _, m := range p.Tiers[0] {
askModule(t.Context(), p, m, byName)
}
if strings.Join(asked, " ") != "modules/build-agent modules/mesh-tools" {
t.Fatalf("build-agent was asked for %v", asked)
}
for _, m := range own {
if s := p.Modules[m]; s.Key || s.Build != "build-"+m || s.State != "asked" {
t.Fatalf("%s is not asked of build-agent, or is read from a key: %+v", m, s)
}
}
for _, m := range keyed {
if s := p.Modules[m]; !s.Key || s.Build != "" {
t.Fatalf("%s is not taken from its key: %+v", m, s)
}
}
}
+7
View File
@@ -32,6 +32,13 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err := broker.Raise(r, names); err != nil {
return nil, err
}
// And each node's bucket for the files of its declaration that travel by digest (novox/hq ADR 0486 §4), so a node finds its own
// whenever the bus is up.
if files, ok := r.(interface{ EnsureDeclarationFiles(nodes []string) error }); ok {
if err := files.EnsureDeclarationFiles(names); err != nil {
return nil, err
}
}
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
+3 -7
View File
@@ -241,10 +241,9 @@ func mergeAskPrefix(number int, head string) string {
return fmt.Sprintf("merge-%d-%s-", number, head)
}
// askersOfMerges are the askers a merge's warrant is read under: the teller, and mesh-delivery for a merge before release 3e
// (ADR 0437 §6). Release 4's controller work stops accepting mesh-delivery (ADR 0450 §8); a merge asked under it earlier is
// still found here until then.
func askersOfMerges() []string { return []string{sdkdelivery.TellerModule, theDelivery} }
// askersOfMerges are the askers a merge's warrant is read under: the teller alone (ADR 0437 §6), since R4e stopped taking
// mesh-delivery's (ADR 0450 §8; issue 1318).
func askersOfMerges() []string { return []string{sdkdelivery.TellerModule} }
// caughtUpWarrant judges a chosen merge ask of the router as the warrant of a merge: the ask is the operator's, the warrant
// answers that very ask as it was published, and it is the operator's Approve proved by the channel it came through (ADR 0437
@@ -391,9 +390,6 @@ var findMergeWarrant = func(ctx context.Context, inv catalogueReader, owner, rep
if asker == sdkdelivery.TellerModule && !strings.HasPrefix(id, sdkdelivery.AskKindMerge+"-") {
continue
}
if asker == theDelivery && !named {
continue
}
ask, w, err := closedRecordOf(ctx, inv, asker, id)
if err != nil {
return "", false, "", err
+6 -34
View File
@@ -69,12 +69,12 @@ func TestAForgeThatDoesNotAnswerIsAnErrorNotAnEmptyTrunk(t *testing.T) {
func TestACaughtUpMergeIsAcceptedOnTheSameWarrantAsAnAnnouncedOne(t *testing.T) {
ok := approvedMerge(t)
for _, asker := range []string{sdkdelivery.TellerModule, theDelivery} {
w := ok.warrant
w.Asker = asker
if _, err := caughtUpWarrant(asker, &ok.ask, &w); err != nil {
t.Errorf("the operator's Approve under %s was refused: %v", asker, err)
}
if _, err := caughtUpWarrant(sdkdelivery.TellerModule, &ok.ask, &ok.warrant); err != nil {
t.Errorf("the operator's Approve under the teller was refused: %v", err)
}
// 1318 (ADR 0450 §8): mesh-delivery's warrant of before the gate is no longer taken.
if _, err := caughtUpWarrant(theDelivery, &ok.ask, &ok.warrant); err == nil {
t.Error("a warrant under mesh-delivery was accepted")
}
if _, err := caughtUpWarrant("keycloak", &ok.ask, &ok.warrant); err == nil {
t.Error("a warrant of an asker the controller takes none from was accepted")
@@ -229,34 +229,6 @@ func TestAGroupsWarrantIsFoundByTheAsksWordsForThePullRequestAndItsHead(t *testi
}
}
// Keycloak's merge was asked as mesh-delivery.merge-378-b9104454f9a5-1, before release 3e: the id names the pull request and
// the first twelve digits of its head, and the warrant is found by them under mesh-delivery.
func TestAWarrantAskedUnderMeshDeliveryIsFoundByThePullRequestAndItsHead(t *testing.T) {
ok := approvedMerge(t)
head := "b9104454f9a5e0c4d3b2a1f0e9d8c7b6a5948372"
askID := "merge-378-b9104454f9a5-1"
ask, w := ok.ask, ok.warrant
ask.ID, w.Ask, w.Asker = askID, askID, theDelivery
w.AskDigest = ask.Digest()
withRouter(t, map[string]map[string]aRouterRecord{theDelivery: {
"merge-377-aaaaaaaaaaaa-1": {},
askID: {ask, w},
}})
why, found, reason, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head)
if err != nil || !found || !strings.Contains(why, askID) {
t.Fatalf("the warrant was not found: %q %v %q %v", why, found, reason, err)
}
// Another head names another ask, and none is found.
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, "ffffffffffff0000"); found {
t.Fatal("another head found a warrant")
}
// A router that cannot be read is an error, never "no warrant".
routerAskIDs = func(context.Context, catalogueReader, string) ([]string, error) { return nil, errors.New("no answer") }
if _, found, _, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head); found || err == nil {
t.Fatalf("an unreadable router read as no warrant: %v %v", found, err)
}
}
// A trunk shorter than a page, or with none, is read without a fault: the pages end when the forge's do.
func TestAShortTrunkIsReadWholeAndAnEmptyOneIsNoMerges(t *testing.T) {
was := trunkCommitsPage
+2 -44
View File
@@ -51,55 +51,13 @@ func attachTheChain(ctx context.Context, inv *inventory.Inventory, server *link.
fmt.Printf("chain: the builds are not laid out under their modules yet, and none is read until they are: %v\n", err)
go layBuildsOutSoon(ctx, inv)
}
// And the requests the move of delivery 1 found never handed over, asked again once (askAgain), once it is serving.
go askAgainSoon(ctx, inv)
// The requests the move of delivery 1 found never handed over are not asked again: the controller asks no build
// (novox/hq ADR 0501; issue 1851, A10), and the builder builds every fingerprint the indexer writes.
// And what delivery 1 recorded of what each machine was sent, copied onto the keys of the assignments (CopySentBuilds).
go copySentBuildsSoon(ctx, inv)
return nil
}
// askAgain asks once more for each build request the move found never handed over (novox/hq issue 981 note 7 Q5): the
// requests a merge could not ask for, which a person's hand `build` would otherwise have to repeat. **At most once**: a
// request is claimed before it is asked and settled after, so a crash between loses the ask, which is where it stood.
func askAgain(ctx context.Context, inv *inventory.Inventory) {
pending, err := inv.RequestsToAskAgain(ctx)
if err != nil {
fmt.Printf("chain: the requests never handed over cannot be read: %v\n", err)
return
}
for _, r := range pending {
if claimed, err := inv.ClaimToAskAgain(ctx, r.ID); err != nil || !claimed {
if err != nil {
fmt.Printf("chain: %s cannot be claimed to be asked again: %v\n", r.ID, err)
}
continue
}
id, err := askABuild(ctx, buildSource{Repository: r.Repository, Seat: r.Seat}, r.Path, r.Ref)
if err != nil {
if serr := inv.AskedAgainAs(ctx, r.ID, "not asked again: "+err.Error()); serr != nil {
fmt.Printf("chain: %s was not asked again, and that could not be kept: %v\n", r.ID, serr)
}
fmt.Printf("chain: %s was never handed over and cannot be asked again: %v\n", r.ID, err)
continue
}
recordBuildRequest(ctx, inv, inventory.BuildRequest{ID: id, Repository: r.Repository, Seat: r.Seat, Path: r.Path,
Ref: r.Ref, Commit: r.Commit, For: r.For})
if err := inv.AskedAgainAs(ctx, r.ID, id); err != nil {
fmt.Printf("chain: %s was asked again as %s, and that could not be kept: %v\n", r.ID, id, err)
}
fmt.Printf("chain: %s, never handed over, is asked again as %s\n", r.ID, id)
}
}
// askAgainSoon waits for the controller to be serving, which an ask needs, and then asks again.
func askAgainSoon(ctx context.Context, inv *inventory.Inventory) {
select {
case <-ctx.Done():
case <-time.After(time.Minute):
askAgain(ctx, inv)
}
}
// kindSentBuildsNotCopied is the condition of the one-time copy of node.sent_builds onto the assignments' keys that has not
// succeeded: until it has, a module an assignment pulled in may be missing from what a machine is read to run.
const kindSentBuildsNotCopied = "sent-builds-not-copied"
+1 -25
View File
@@ -5,7 +5,6 @@ import (
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
@@ -53,7 +52,7 @@ func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p invento
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
for _, pull := range pullsOfWalk(&p) {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
@@ -62,26 +61,3 @@ func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p invento
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
@@ -14,23 +14,12 @@ import (
"github.com/novox/mesh-controller/internal/link"
)
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
func TestAWalkNamesThePullRequestsOfItsMergesOnce(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(pullsOfWalk(p), want) {
t.Fatalf("the walk names %+v", pullsOfWalk(p))
}
}
-59
View File
@@ -104,44 +104,6 @@ func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
@@ -170,24 +132,3 @@ func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
-165
View File
@@ -1,165 +0,0 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
siblingsFrom(request, p, entries, lines, err)
}
// siblingsFrom is withSiblings from mesh-delivery's answer already had.
func siblingsFrom(request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry, lines []groupsLine, err error) {
if request.Check == nil {
return
}
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
// composedMember says the pull request is a member of a delivery group whose gate is the composed check's (novox/hq
// ADR 0403 clauses 1 and 7): the group has two members or more and mesh-delivery says so on its line. The group is
// found as siblingsFor finds it, by its head branch name, or, for a recheck that carries none, by the pull request's
// repository and number — not by siblings, which name only what the check reads beside it.
func composedMember(p link.PullUpdated, lines []groupsLine) bool {
own := strings.ToLower(p.Owner + "/" + p.Repo)
for _, l := range lines {
found := p.Head != "" && l.ID == p.Head
for _, m := range l.Details {
if p.Head == "" && p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
found = true
}
}
if found {
return l.ComposedGate && max(len(l.Details), len(l.Members)) >= 2
}
}
return false
}
-248
View File
@@ -1,248 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
+105 -58
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"io"
"os"
"path"
@@ -161,56 +162,77 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
return "", err
}
scope := pullScope(p, entries, read, edges)
// **A cycle of edges fails the check, naming every module and edge** (novox/hq ADR 0419 clause 6), a lone pull
// request's as a group's composed check is: no order exists for what it changes.
alone := orderMember{ID: p.Owner + "/" + p.Repo + "@" + twelve(p.Commit), Repository: p.Owner + "/" + p.Repo, Base: p.Base,
Head: p.Commit, Number: p.Number, Paths: p.Paths, PathsTruncated: p.PathsTruncated, Removed: p.Removed, BaseSHA: p.BaseSHA}
if failing, cyclic, err := cycleVerdict(ctx, "", []orderMember{alone},
map[string]orderReach{alone.ID: {Moved: scope.Modules}}, edges, entries); err != nil {
return "", err
} else if cyclic {
failing.Commit, failing.Trunk = p.Commit, twelve(p.BaseSHA)
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, failing.Summary)
sayChecked(ctx, w, failing)
// **The merge gate's checks stop** (novox/hq ADR 0503 §3, §5; issue 1299): the checker judges every repository whose
// trunk requires `mesh/check`. A repository whose trunk requires `mesh/repo-check` (hq), whatever else it requires
// (ADR 0513 §4), has its own merge-check.sh asked of build-agent, and said with that layer alone; any other is asked
// nothing and said nothing. What the forge's protection requires is read, never listed (lead decision L11).
// Mesh-config is still told what the head declares (ADR 0502), whatever judges it.
this := pullAtOf(p)
if this.Truncated {
this.Paths, this.Truncated = truncatedPaths(entries, p.Owner, p.Repo, scope.Modules), false
}
if scope.gated() || holdsModules(entries, p.Owner, p.Repo) {
tellPullDeclares(ctx, w, []pullAt{this})
}
required, err := requiredStatusesOf(ctx, p.Owner, p.Repo, p.Base)
switch {
case err != nil:
fmt.Fprintf(w, "%s/%s#%d (%.8s): what %s's protection requires cannot be read, so nothing is asked and nothing is "+
"said: %v\n", p.Owner, p.Repo, p.Number, p.Commit, p.Base, err)
return "", nil
case !asksRepoCheck(required):
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s does not require %s, and %s; nothing is asked\n", p.Owner, p.Repo, p.Number,
p.Commit, p.Base, sdkdelivery.StatusContextRepoCheck, checkerWords(required))
return "", nil
}
// **A member of a group whose gate is the composed check's runs its repository's check alone** (novox/hq ADR 0403
// clause 1): mesh-delivery is asked once, and says so on the group's line. Told nothing, the pull request is
// checked alone, gate included (clause 6).
lines, groupsErr := groupsAsked(ctx, deliverySeatHeld(entries))
composed := groupsErr == nil && composedMember(p, lines)
// The change plan of the commit at hand (ADR 0238): what a merge of it would build and send, posted
// with the verdict whatever the verdict is.
return f.repoCheckAlone(ctx, w, p, scope, entries)
}
// asksRepoCheck says what a trunk's protection has the controller ask: its repository's own merge-check.sh whenever the
// trunk requires `mesh/repo-check`, whatever else it requires (ADR 0513 §4: a trunk requiring both has the checker judge
// `mesh/check` and the repository check asked beside it), and nothing otherwise.
func asksRepoCheck(required []string) bool {
return slices.Contains(required, sdkdelivery.StatusContextRepoCheck)
}
// checkerWords says who judges a trunk that requires no `mesh/repo-check`.
func checkerWords(required []string) string {
if slices.Contains(required, sdkdelivery.StatusContextCheck) {
return "the checker judges its " + sdkdelivery.StatusContextCheck
}
return "does not require " + sdkdelivery.StatusContextCheck + " either, so nothing is said"
}
// repoCheckAlone asks build-agent for a repository's own merge-check.sh alone, for a pull request of a repository whose
// trunk requires `mesh/repo-check`, whether or not it also requires `mesh/check` (ADR 0503 §5, ADR 0513 §4): no gate, no module built, its verdict said with the
// repo-check layer alone.
func (f following) repoCheckAlone(ctx context.Context, w io.Writer, p link.PullUpdated, scope checkScope,
entries []inventory.Entry) (string, error) {
inv := f.open.inventory
plan := changePlanOf(p.Owner+"/"+p.Repo, p.Base, p.Commit, scope.Reach, entries, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
})
fmt.Fprint(w, planText(plan))
direct := directVerdict(p, plan)
if composed {
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched)
case !scope.Mesh:
direct.RepoCheck = &link.CheckLayer{Verdict: "warning", Summary: "the repository is not the mesh's: nothing run"}
fmt.Fprintf(w, "%s/%s#%d (%.8s): the repository is not the mesh's: said, nothing run\n", p.Owner, p.Repo, p.Number,
p.Commit)
sayChecked(ctx, w, direct)
return "", nil
case !scope.gated() && p.MergeCheckSaid && !p.MergeCheck:
case p.MergeCheckSaid && !p.MergeCheck:
direct.RepoCheck = &link.CheckLayer{Verdict: "warning", Summary: noMergeCheck}
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, noMergeCheck)
sayChecked(ctx, w, direct)
return "", nil
}
request, err := checkRequestFor(ctx, f.open, p, scope, entries)
request, err := checkRequestFor(ctx, f.open, p, checkScope{Mesh: true, From: scope.From}, entries)
if err != nil {
return "", err
}
request.Check.Plan = &plan
request.Check.Composed = composed
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
@@ -220,23 +242,42 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
if err := ask.Ask(ctx, request); err != nil {
return "", err
}
// **A newer request replaces the one still waiting, and a newer key stops the one running** (novox/hq ADR 0406 §3, §4,
// issue 974): the check queue keeps one request a pull request, and a running check watches its key. Nothing is asked
// of the forge here (issue 726).
what := "its own merge-check.sh alone: " + noModuleTouched
if composed {
what = "its own merge-check.sh alone: the gate is its group's composed check's"
}
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
strings.Join(append(append([]string{}, scope.Modules...), prefixedAll("new:", scope.New)...), ", "),
scope.Width, scope.Tiers)
}
fmt.Fprintf(w, "%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
p.Commit, seat, what, request.ID)
fmt.Fprintf(w, "%s/%s#%d (%.8s): asked %s to run its own merge-check.sh alone, as %s\n", p.Owner, p.Repo, p.Number,
p.Commit, seat, request.ID)
return request.ID, nil
}
// requiredStatusesOf is what the forge's protection of a repository's branch requires before a merge, as the forge
// picks the rule it applies (the one named for the branch, else the first glob covering it). A variable so a test needs
// no forge.
var requiredStatusesOf = func(ctx context.Context, owner, repo, branch string) ([]string, error) {
var rules struct {
Rules []protectionRule `json:"rules"`
}
js, err := aBus()
if err != nil {
return nil, err
}
defer js.Close()
raw, _ := json.Marshal(map[string]any{"owner": owner, "repo": repo})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_branch_protection_get", raw, 30*time.Second)
if err != nil {
return nil, err
}
if answer.Error != "" {
return nil, fmt.Errorf("gitea.gitea_branch_protection_get: %s", answer.Error)
}
if err := json.Unmarshal(answer.Result, &rules); err != nil {
return nil, err
}
r := appliedRule(rules.Rules, branch)
if r == nil {
// empty-on-error: not an error; no rule covers the branch, so it requires nothing, which the caller says
return nil, nil
}
return r.RequiredStatuses, nil
}
// composedWords is what a group member's own verdict says of its gate layer (novox/hq ADR 0403 clause 1).
const composedWords = "judged by its group's composed check"
@@ -365,15 +406,6 @@ func trunkOf(s inventory.Source) string {
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
@@ -594,11 +626,13 @@ func checkedOf(result link.BuildResult) link.Checked {
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
// Every member is named with its trunk commit (novox/hq issue 1106): the seat's word, else the one the forge
// announced with the head, so the forge's merge tool never meets a member whose verdict names none.
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref, Trunk: c.Trunk})
Commit: result.Ref, Trunk: firstTrunk(c.Trunk, g.Trunk)})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
Commit: m.Ref, Trunk: firstTrunk(memberTrunk(result.Check, m), m.Trunk)})
}
}
}
@@ -627,6 +661,16 @@ func checkedOf(result link.BuildResult) link.Checked {
return c
}
// firstTrunk is the first trunk commit named, as twelve characters; empty when none is.
func firstTrunk(named ...string) string {
for _, n := range named {
if n != "" {
return twelve(n)
}
}
return ""
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
@@ -651,6 +695,9 @@ func prefixedAll(prefix string, items []string) []string {
// sayChecked says a merge check's verdict on the bus, where the forge's holder hears it.
func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
// **No merge-gate layer from R4e, for any repository** (ADR 0503 §3, §5; issue 1299): the forge's holder writes
// `mesh/repo-check` from the repo-check layer alone, and the checker owes `mesh/check`.
c.Gate = nil
repo := "none"
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
@@ -659,8 +706,8 @@ func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Fprintf(w, "%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
fmt.Fprintf(w, "%s: %s/%s#%d at %.8s%s checked on %s: repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), repo)
if checkEvents == nil {
return
}
-97
View File
@@ -1,97 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Acceptance 5 of milestone 2 (novox/hq ADR 0406 §3, issue 974), live, with the store and a real bus: two merges into one
// repository within a minute, with two open pull requests, leave each pull request with exactly one queued check, by the
// stream's rule. Each merge moves the trunk, and mesh-delivery asks the controller to check both open pull requests again
// against it (`delivery check`, which is the controller's own path for a head announced); nothing moves a request, nothing
// asks the forge, and the queue is read back whole.
func TestTwoMergesWithinAMinuteLeaveEachOpenPullRequestOneQueuedCheck(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
holdTheDeliverySeat(t, open)
was := askDeliveryOwner
t.Cleanup(func() { askDeliveryOwner = was })
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, _ map[string]any) (json.RawMessage, error) {
if verb != "groups" {
t.Fatalf("asked the owner %s", verb)
}
return json.RawMessage(`[]`), nil
}
// The repository the two pull requests are into, as a module the mesh builds from it.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
heads := map[int]string{7: "7777777777777777777777777777777777777777", 8: "8888888888888888888888888888888888888888"}
check := func(number int, trunk string) {
t.Helper()
p := orderMember{ID: fmt.Sprintf("novox/mesh-controller@%.12s", heads[number]), Repository: "novox/mesh-controller",
Base: "main", Head: heads[number], Number: number, Paths: []string{"main.go"}, ModuleDirsSaid: true,
BaseSHA: trunk, Ready: number == 8}.pull()
p.MergeCheck, p.MergeCheckSaid = true, true
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
}
// Announced against the trunk as it stands, then asked again after each of two merges.
for _, trunk := range []string{"1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222",
"3333333333333333333333333333333333333333"} {
check(7, trunk)
check(8, trunk)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for _, a := range q.Asks {
c := a.Request.Check
if c == nil {
t.Fatalf("%s is no check", a.ID)
}
if _, twice := seen[c.Number]; twice {
t.Errorf("pull request %d has two queued checks", c.Number)
}
seen[c.Number] = c.Trunk
if a.State != link.AskWaiting {
t.Errorf("%s is %s", a.ID, a.State)
}
if want := "3333333333333333333333333333333333333333"; c.Trunk != want {
t.Errorf("pull request %d is queued against %.8s; want the newest trunk %.8s", c.Number, c.Trunk, want)
}
}
if len(seen) != 2 {
t.Fatalf("queued checks for %v; want one for each of the two open pull requests", seen)
}
// By the streams' own count: the ready one on its stream, the other on its.
for stream, want := range map[string]uint64{broker.ChecksReadyStream: 1, broker.ChecksOtherStream: 1} {
info, err := js.Context().StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != want {
t.Errorf("%s holds %d request(s); want %d", stream, info.State.Msgs, want)
}
}
}
@@ -0,0 +1,109 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// C1 (ADR 0503 §3, §5; issue 1299, lead decision L11), with the store and a real build queue: a pull request of a
// repository whose trunk requires mesh/check asks build-agent nothing and says nothing; one whose trunk requires
// mesh/repo-check alone (hq) has its own merge-check.sh asked, and every verdict said names no gate layer; a protection
// that cannot be read asks and says nothing.
func TestThePullRequestChecksAreAskedOnlyForARepositoryTheCheckerDoesNotJudge(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
said := &saidOnTheBus{}
checkEvents = said
t.Cleanup(func() { checkEvents = nil })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
// hq is cloned as a module built from it is, from its URL: no forge is held in this test.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "hq-docs", Version: "1"},
inventory.Source{Repository: "http://forge.invalid/novox/hq.git", Ref: "main", Path: "docs", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
required := map[string][]string{"novox/mesh-controller": {"mesh/check"}, "novox/hq": {"mesh/repo-check"}}
was := requiredStatusesOf
requiredStatusesOf = func(_ context.Context, owner, repo, _ string) ([]string, error) {
if r, ok := required[owner+"/"+repo]; ok {
return r, nil
}
return nil, errors.New("the forge did not answer")
}
t.Cleanup(func() { requiredStatusesOf = was })
queued := func() int {
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
return len(q.Asks)
}
pull := func(repo string, number int, mergeCheck bool) link.PullUpdated {
p := link.PullUpdated{Owner: "novox", Repo: repo, Number: number, Base: "main", Commit: "abcdefabcdefabcdefabcdefabcdefabcdefabcd",
BaseSHA: "1111111111111111111111111111111111111111", Paths: []string{"main.go"}, ModuleDirsSaid: true}
p.MergeCheck, p.MergeCheckSaid = mergeCheck, true
return p
}
f := following{open: open, oneShot: true}
if err := f.PullUpdated(ctx, pull("mesh-controller", 7, true)); err != nil {
t.Fatal(err)
}
if n := queued(); n != 0 || len(said.events) != 0 {
t.Fatalf("0503 §3: a repository the checker judges asked %d check(s) and said %v", n, said.events)
}
if err := f.PullUpdated(ctx, pull("unread", 3, true)); err != nil {
t.Fatal(err)
}
if n := queued(); n != 0 || len(said.events) != 0 {
t.Fatalf("L11: a protection that cannot be read asked %d check(s) and said %v", n, said.events)
}
if err := f.PullUpdated(ctx, pull("hq", 9, true)); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if len(q.Asks) != 1 || q.Asks[0].Request.Check == nil || len(q.Asks[0].Request.Check.Modules) != 0 ||
q.Asks[0].Request.Check.Repo != "hq" {
t.Fatalf("0503 §5: hq's repository check is not the one ask: %+v", q.Asks)
}
// A head with no merge-check.sh is said at once, with the repo-check layer alone.
if err := f.PullUpdated(ctx, pull("hq", 10, false)); err != nil {
t.Fatal(err)
}
if len(said.events) != 1 {
t.Fatalf("hq's head with no merge-check.sh was said %d time(s)", len(said.events))
}
var c link.Checked
if err := json.Unmarshal(said.bodies[0], &c); err != nil {
t.Fatal(err)
}
if c.Gate != nil || c.RepoCheck == nil || c.RepoCheck.Verdict != "warning" {
t.Fatalf("0503 §5: the verdict said names a gate layer, or no repo-check layer: %+v", c)
}
// A trunk requiring both (hq, ADR 0513 §4): the checker judges mesh/check, and the repository check is still asked.
required["novox/hq"] = []string{"mesh/check", "mesh/repo-check"}
if err := f.PullUpdated(ctx, pull("hq", 11, true)); err != nil {
t.Fatal(err)
}
if q, err := link.ReadQueue(ctx, js, link.TheBuildMachine); err != nil || len(q.Asks) != 2 ||
q.Asks[1].Request.Check == nil || q.Asks[1].Request.Check.Repo != "hq" {
t.Fatalf("0513 §4: a trunk requiring both had no repository check asked: %+v %v", q.Asks, err)
}
}
+43
View File
@@ -250,6 +250,31 @@ func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
}
}
// **A group's composed verdict names every member's trunk commit** (novox/hq issue 1106): a member the build seat named no
// trunk for (an SDK member whose base the seat did not merge, or a seat from before the rule) is named by the trunk the forge
// announced with its head, and the primary the same, so the merge tool never reads "names no trunk commit" on a member
// whose trunk did not move.
func TestAGroupVerdictNamesEveryMembersTrunkTheSDKMemberToo(t *testing.T) {
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 7, Base: "main", Group: "feat/x",
Trunk: "dddddddddddd0000",
Members: []link.GroupMember{
{Owner: "novox", Repo: "mesh-sdk", Number: 8, Ref: "cccccccccccc0000", Base: "main", Trunk: "eeeeeeeeeeee0000"},
{Owner: "novox", Repo: "mesh-host", Number: 9, Ref: "bbbbbbbbbbbb0000", Base: "main", Trunk: "ffffffffffff0000"}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "bbbbbbbbbbbb", Trunk: "222222222222"}}}})
want := []string{"dddddddddddd", "eeeeeeeeeeee", "222222222222"}
if len(c.Members) != 3 {
t.Fatalf("the composed check names its members %+v", c.Members)
}
for i, m := range c.Members {
if m.Trunk != want[i] {
t.Fatalf("member %d (%s/%s) is named with trunk %q, want %q (the seat's word first, else the announced one): %+v",
i, m.Owner, m.Repo, m.Trunk, want[i], c.Members)
}
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
@@ -291,3 +316,21 @@ func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
// A check stopped by its pull request's key moving on says no verdict (issue 974, review 2): an error said for it would name the
// live head, and the newer request is the one that judges. A check that failed for any other reason still says its error.
// C2 (ADR 0513 §4, ADR 0503 §5; issue 1299): a trunk requiring mesh/repo-check has its repository check asked, whether or
// not it also requires mesh/check (hq requires both); a trunk requiring mesh/check alone, or neither, is asked nothing.
func TestARepositoryCheckIsAskedWheneverTheTrunkRequiresIt(t *testing.T) {
for _, c := range []struct {
required []string
asks bool
}{
{[]string{"mesh/check", "mesh/repo-check"}, true},
{[]string{"mesh/repo-check"}, true},
{[]string{"mesh/check"}, false},
{nil, false},
} {
if got := asksRepoCheck(c.required); got != c.asks {
t.Fatalf("a trunk requiring %v asks its repository check: %v, want %v", c.required, got, c.asks)
}
}
}
+6
View File
@@ -233,6 +233,12 @@ func prepareSend(ctx context.Context, open *stores, node string) ([]string, erro
if !slices.Contains(in.BusMinted, "provisioner."+node) {
continue
}
default:
// A bus process's account nobody issued still gets the stand-in, until the composition leaves
// the module off (issue 1753): the warning names the verb (issue 1226).
if line := processAccountWarning(m, node, name, in.BusMinted); line != "" {
sayWarning(ctx, line)
}
}
if _, err := inv.SecretForModule(ctx, node, m.Module, name); err == nil {
did = true
+7 -1
View File
@@ -200,6 +200,7 @@ func composeQuestion(ctx context.Context, open *stores, node string, mode compos
}
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
if !withOrder {
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
}
@@ -238,7 +239,11 @@ func markUnresolvable(err error) error {
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
if err != nil {
return declared, err
}
// Never a declaration naming an image the store does not hold (novox/hq issue 1742): that module is left out.
return withoutMissingImages(ctx, open, node, declared), nil
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
@@ -281,6 +286,7 @@ func composeSendValidated(ctx context.Context, open *stores, node string) (senda
if len(left) == 0 && len(in.wouldMake) == 0 {
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
// The sender stamps the order it allots (order.stamp); a composition for a send carries none of its own.
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
here, _ := nodeOf(in.Input, node)
@@ -1,68 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A member of a group whose gate is the composed check's is asked of the build seat with no module: its repository's
// check alone (novox/hq ADR 0403 clause 1). With no word from mesh-delivery, or one that does not say the gate is
// composed, it is checked alone with its gate (clause 6).
func askedForMember(t *testing.T, answer []groupsLine, answerErr error) link.CheckRequest {
t.Helper()
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
was := groupsAsked
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, answerErr }
t.Cleanup(func() { groupsAsked = was })
p := link.PullUpdated{Owner: "novox", Repo: "mesh-controller", Number: 346, Base: "main", Head: "feat/x", Commit: headController,
Paths: []string{"main.go"}, ModuleDirsSaid: true, MergeCheck: true, MergeCheckSaid: true}
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil || len(q.Asks) != 1 {
t.Fatalf("asks %d: %v", len(q.Asks), err)
}
return *q.Asks[0].Request.Check
}
func TestAComposedMembersCheckIsItsRepositoryChecksAlone(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
line.ComposedGate = true
c := askedForMember(t, []groupsLine{line}, nil)
if len(c.Modules) != 0 || len(c.New) != 0 || len(c.Dependents) != 0 || !c.Composed {
t.Fatalf("the member's check is asked as %+v; want no module and composed", c)
}
}
func TestAMemberOfAGroupOfAnOlderMeshDeliveryKeepsItsOwnGate(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
c := askedForMember(t, []groupsLine{line}, nil)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed {
t.Fatalf("an older mesh-delivery's member is asked as %+v; want its own gate", c)
}
}
func TestAMemberWhoseGroupCannotBeLearnedIsCheckedAloneWithItsGate(t *testing.T) {
c := askedForMember(t, nil, context.DeadlineExceeded)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed ||
!strings.Contains(c.SiblingsUnknown, "reads no siblings") {
t.Fatalf("the member is asked as %+v; want its gate and why its siblings are unknown", c)
}
}
+5 -80
View File
@@ -1,50 +1,21 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery group's gate is the composed check's, and its members' own checks run their repository's check alone
// (novox/hq ADR 0403 clauses 1 and 7). Pure: no store, no bus.
// A member of a group whose line says composed_gate is a composed member; one whose group says nothing of it (an older
// mesh-delivery), one of a group of a single pull request and a lone pull request are not.
func TestAMemberOfAComposedGroupIsKnownByTheGroupsLine(t *testing.T) {
two := func(composed bool) groupsLine {
l := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))
l.ComposedGate = composed
return l
}
p := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
recheck := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Commit: headHost}
for name, c := range map[string]struct {
p link.PullUpdated
lines []groupsLine
want bool
}{
"composed": {p, []groupsLine{two(true)}, true},
"a recheck, by its number": {recheck, []groupsLine{two(true)}, true},
"an older mesh-delivery": {p, []groupsLine{two(false)}, false},
"another group": {p, []groupsLine{group("feat/y", member("novox/mesh-host", 1, headHost, "main", true))}, false},
"no group": {p, nil, false},
} {
if got := composedMember(c.p, c.lines); got != c.want {
t.Errorf("%s: composed %v, want %v", name, got, c.want)
}
}
alone := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))
alone.ComposedGate = true
if composedMember(p, []groupsLine{alone}) {
t.Error("a group of a single pull request is composed")
}
}
// The heads of a group's members, by repository.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
)
// A composed member's verdict never says its gate passed: the gate layer, and the verdict that is the gate's, read
// "composed"; a failure or an error stays what it was.
@@ -88,49 +59,3 @@ func TestTheComposedVerdictCarriesEveryMember(t *testing.T) {
}
}
}
// Members whose manifests order their modules in a cycle fail the composed check, naming every module and edge, and no
// ask is made of the build seat.
func TestAGroupInACycleFailsItsComposedCheckWithoutABuild(t *testing.T) {
members := []orderMember{
{ID: "novox/x@" + headHost[:12], Repository: "novox/x", Base: "main", Head: headHost, Number: 7, Paths: []string{"module.json"}},
{ID: "novox/y@" + headController[:12], Repository: "novox/y", Base: "main", Head: headController, Number: 9, Paths: []string{"module.json"}},
}
reach := map[string]orderReach{members[0].ID: {Moved: []string{"mx"}}, members[1].ID: {Moved: []string{"my"}}}
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, repo, _, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil // what the pull requests merge into holds no such manifest
}
if repo == "x" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
c, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, []inventory.Edge(nil), nil)
if err != nil || !cyclic {
t.Fatalf("no cycle found: %v", err)
}
if c.Verdict != "fail" || c.Group != "feat/cycle" || len(c.Members) != 2 || c.Gate == nil || c.Gate.Verdict != "fail" {
t.Fatalf("the verdict is %+v", c)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(c.Summary, w) {
t.Errorf("the cycle does not name %s: %s", w, c.Summary)
}
}
// Members with no edge between them are no cycle.
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
if _, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, nil, nil); err != nil || cyclic {
t.Fatalf("a cycle where there is none: %v %v", cyclic, err)
}
}
@@ -0,0 +1,65 @@
package main
import (
"io"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
)
// Issue 1638 (novox/hq ADR 0480 clause 7), taken in with mesh-sdk's container `groups`: the controller sends a container's
// groups only to a node-engine that reports the capability container-groups, and holds the module on any other, naming it.
const n8nWithAGroup = `{"module":"n8n","version":"1",
"config":{"media-gid":{"type":"number","minimum":1,"required":true,"description":"the group that owns the media library"}},
"resources":[{"id":"server","type":"container","name":"n8n",
"image":"n8nio/n8n@sha256:4846eb2f4b874ab04cde7fc1e249d2ddaec66e9aea64439beb2972cfea88e3c0",
"groups":["${config:media-gid}"]}]}`
// printedBy runs reportUnhostable and returns what it printed.
func printedBy(t *testing.T, node string, plan catalogue.Resolution) string {
t.Helper()
stdout := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
reportUnhostable(node, plan)
w.Close()
os.Stdout = stdout
out, _ := io.ReadAll(r)
return string(out)
}
// 1638/AC1: a node that reports the capability gets the module; one that does not has it held, the capability named.
func TestAModuleWithGroupsIsHeldOnANodeThatDoesNotReportTheCapability(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(n8nWithAGroup))
if err != nil {
t.Fatal(err)
}
shelf := catalogue.Shelf{"n8n": m}
reads := catalogue.Node{Name: "ace", Capabilities: map[string]bool{"container-runtime": true, "container-groups": true}}
old := catalogue.Node{Name: "g14", Capabilities: map[string]bool{"container-runtime": true}}
got, err := catalogue.Resolve(shelf, []string{"n8n"}, reads, catalogue.World{})
if err != nil || len(got.Unhostable) != 0 || len(got.Modules) != 1 {
t.Fatalf("a node that reads groups: %v, held %+v", err, got.Unhostable)
}
if out := printedBy(t, "ace", got); out != "" {
t.Errorf("a node that reads groups was told: %q", out)
}
got, err = catalogue.Resolve(shelf, []string{"n8n"}, old, catalogue.World{})
if err != nil {
t.Fatal(err)
}
if len(got.Modules) != 0 || len(got.Unhostable) != 1 {
t.Fatalf("n8n was sent to a node that does not report container-groups: hosted %d, held %+v", len(got.Modules), got.Unhostable)
}
if out := printedBy(t, "g14", got); !strings.Contains(out, "n8n") || !strings.Contains(out, "container-groups") {
t.Errorf("the hold does not name the module and the capability: %q", out)
}
}
@@ -0,0 +1,110 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/overlay"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// 1696/AC2, ADR 0486 §5: a node is sent a file by digest only when its profile reports the capability. A node whose profile lacks
// it, and a node the mesh has no profile of, are sent every content inline; a node that reports it is sent the reference. The test
// goes through composeQuestion and composeSend, the two places readsFileReferences is read (compose_sdk.go).
func TestAFileTravelsByDigestOnlyToANodeThatReportsTheCapability(t *testing.T) {
open := aMesh(t) // anchor and laptop report no file-references
ctx := t.Context()
big := strings.Repeat("0123456789abcdef", 8<<10) // 128 KiB
register(t, open, catalogue.Manifest{Module: "bigfile", Version: "1",
Resources: []map[string]any{{"id": "blob", "type": "file", "path": "/etc/bigfile/blob", "mode": "0644", "content": big}}})
// reader reports the capability; bare has never reported a profile at all.
for i, n := range []struct {
name string
profile map[string]any
}{
{"reader", map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true},
map[string]any{"name": sdkdecl.FileReferencesCapability, "present": true}}}},
{"bare", nil},
} {
record, err := open.inventory.AddNode(ctx, n.name)
if err != nil {
t.Fatal(err)
}
if err := open.inventory.SetPlace(ctx, n.name, n.name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+10)); err != nil {
t.Fatal(err)
}
if n.profile != nil {
if err := open.inventory.RecordProfile(ctx, record.ID, n.profile); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, n.name, overlay.Name); err != nil {
t.Fatal(err)
}
}
for _, node := range []string{"laptop", "reader", "bare"} {
if _, err := assign(ctx, open, node, "bigfile"); err != nil {
t.Fatalf("assign %s: %v", node, err)
}
}
byDigest := map[string]bool{"laptop": false, "bare": false, "reader": true}
for node, want := range byDigest {
asked, _, err := composeQuestion(ctx, open, node, compose.Reading, false)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
sent, err := composeSend(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
for what, s := range map[string]sendable{"question": asked, "send": sent} {
if s.fileReferences != want {
t.Fatalf("%s, %s: fileReferences %v, want %v", node, what, s.fileReferences, want)
}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
var found bool
for _, r := range got.Resources {
if r["path"] != "/etc/bigfile/blob" {
continue
}
found = true
inline := r["content"] == big
if want && (inline || r["digest"] == nil || r["size"] == nil) {
t.Fatalf("%s, %s: a node that reads references is sent the content inline", node, what)
}
if !want && (!inline || r["digest"] != nil) {
t.Fatalf("%s, %s: a node that does not report the capability is sent a reference", node, what)
}
}
if !found {
t.Fatalf("%s, %s: the file is not in the declaration", node, what)
}
}
}
}
+253
View File
@@ -0,0 +1,253 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/compose"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A declaration's large files travel by their digest to a node that can read one** (novox/hq ADR 0486 §1, §5, issue 1696).
//
// The declaration to a node that reports sdkdecl.FileReferencesCapability carries, in place of the content of every
// file resource of more than sdkdecl.FileReferenceThreshold bytes that may travel so (File.TravelsByDigest: plain content or
// bytes, not sealed, not filled with secrets, not written into another file), its sha256 digest and size. The content is put
// into the bucket of the bus under the hex of the digest before the declaration is sent. A node that does not report the
// capability is sent every content inline, as before: an older node-engine refuses a field it does not know, whole.
// readsFileReferences says the node reports the capability of reading a file given by digest (ADR 0486 §5). A node the mesh has
// no profile of reads none.
func readsFileReferences(in compose.Input, node string) bool {
n, ok := nodeOf(in, node)
return ok && n.Capabilities[sdkdecl.FileReferencesCapability]
}
// referencing is the resources as a node that reads references is sent them, and the content of each file that went by
// digest, by the name of its object (the hex of its digest). The resources it is given are not changed. A file that does not
// travel by digest is passed as it is.
func referencing(resources []map[string]any) ([]map[string]any, map[string][]byte, error) {
out := make([]map[string]any, len(resources))
var objects map[string][]byte
for i, r := range resources {
out[i] = r
if kind, _ := r["type"].(string); kind != string(sdkdecl.TypeFile) {
continue
}
// Cheap first: only a file with a content or bytes of the size is worth reading whole.
content, _ := r["content"].(string)
bytesField, _ := r["bytes"].(string)
if len(content) <= sdkdecl.FileReferenceThreshold && len(bytesField) <= sdkdecl.FileReferenceThreshold {
continue
}
raw, err := json.Marshal(r)
if err != nil {
return nil, nil, fmt.Errorf("resource %v: %w", r["id"], err)
}
var f sdkdecl.File
if err := json.Unmarshal(raw, &f); err != nil {
return nil, nil, fmt.Errorf("resource %v is not a file the controller can read: %w", r["id"], err)
}
if !f.TravelsByDigest() {
continue
}
ref, body, err := f.AsReference()
if err != nil {
return nil, nil, err
}
name, err := sdkdecl.ObjectName(ref.Digest)
if err != nil {
return nil, nil, err
}
with := make(map[string]any, len(r)+2)
for k, v := range r {
with[k] = v
}
delete(with, "content")
delete(with, "bytes")
with["digest"], with["size"] = ref.Digest, ref.Size
out[i] = with
if objects == nil {
objects = map[string][]byte{}
}
objects[name] = body
}
return out, objects, nil
}
// namedObjects are the objects of the bucket a body names: the object name of every file resource given by reference,
// with its size. It reads a declaration as the node does, so it also reads the latest one the bus holds.
func namedObjects(body []byte) (map[string]int64, error) {
var envelope struct {
Resources []struct {
Type string `json:"type"`
Digest string `json:"digest"`
Size int64 `json:"size"`
} `json:"resources"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
return nil, err
}
var out map[string]int64
for _, r := range envelope.Resources {
if r.Type != string(sdkdecl.TypeFile) || r.Digest == "" {
continue
}
name, err := sdkdecl.ObjectName(r.Digest)
if err != nil {
return nil, err
}
if out == nil {
out = map[string]int64{}
}
out[name] = r.Size
}
return out, nil
}
// objectStore is what putting the objects needs of a node's bucket.
type objectStore interface {
// Has says whether an object of that name is there, complete, and whether it carries the sweep's mark of being named by no
// node's latest declaration.
Has(ctx context.Context, name string) (have, marked bool, err error)
// Put writes the object under that name, replacing one of the same name (and so erasing its mark).
Put(ctx context.Context, name string, content []byte) error
}
// putObjects makes sure every object of the declaration is in the bucket before the declaration that names it is sent: a node
// that reads the declaration finds each object. An object already there (the same content, the same name) is not written again,
// **unless the sweep marked it unnamed**: then it is written again, so the mark that would have it removed is gone with the
// old object (issue 1696, review: a mark read just before a send must not take an object the send relies on). Refuses on the
// first it cannot put, naming it, so the declaration is not sent.
func putObjects(ctx context.Context, store objectStore, objects map[string][]byte) error {
names := make([]string, 0, len(objects))
for name := range objects {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
have, marked, err := store.Has(ctx, name)
if err != nil {
return fmt.Errorf("whether the object %s is in the bucket: %w", name[:12], err)
}
if have && !marked {
continue
}
if err := store.Put(ctx, name, objects[name]); err != nil {
return fmt.Errorf("putting the object %s (%d bytes) in the bucket: %w", name[:12], len(objects[name]), err)
}
}
return nil
}
// busObjects is a node's bucket on the controller's own connection.
type busObjects struct{ store jetstream.ObjectStore }
// openBucket opens the bucket of a node's files, making it first if it is not there (it is asserted for every node at the
// controller's start; a node adopted since is covered here).
func openBucket(ctx context.Context, conn *nats.Conn, node string) (busObjects, error) {
js, err := jetstream.New(conn)
if err != nil {
return busObjects{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(ctx, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
if err = broker.EnsureDeclarationFiles(ctx, js, node); err == nil {
store, err = js.ObjectStore(ctx, bucket)
}
}
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket %s: %w", bucket, err)
}
return busObjects{store}, nil
}
func (b busObjects) Has(ctx context.Context, name string) (bool, bool, error) {
info, err := b.store.GetInfo(ctx, name)
switch {
case errors.Is(err, jetstream.ErrObjectNotFound):
return false, false, nil
case err != nil:
return false, false, err
}
_, marked := info.Metadata[declarationUnnamed]
return !info.Deleted, marked, nil
}
func (b busObjects) Put(ctx context.Context, name string, content []byte) error {
_, err := b.store.PutBytes(ctx, name, content)
return err
}
// filesLocks hold a node's bucket for one act at a time in this process: a send puts its objects and publishes the declaration that
// names them under the node's lock, and the sweep judges the node's bucket under the same one, so the sweep never removes an
// object between a send's put and its declaration. Another process (a person's push) is covered by the put erasing a mark and by
// the sweep looking at an object again just before it removes it.
var filesLocks sync.Map
func lockFilesOf(node string) func() {
l, _ := filesLocks.LoadOrStore(node, &sync.Mutex{})
mu := l.(*sync.Mutex)
mu.Lock()
return mu.Unlock
}
// kindFilesNotPut is a node whose files that travel by digest could not be put in its bucket, send after send.
const kindFilesNotPut = "declaration-files-not-put"
// filesNotPutAfter is how many sends in a row may fail to put before the condition is raised: the first send to the machine holding
// the bus fails for want of the grant that very send delivers, and says so, and is not yet a fault.
const filesNotPutAfter = 2
var (
filesFailMu sync.Mutex
filesFailed = map[string]int{}
)
func filesNotPutObservation(node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindFilesNotPut, Token: kindFilesNotPut,
Machine: node, Severity: conditions.Warning, Source: "sends", Summary: summary}
}
// notePut records how putting a send's objects went: a failure counts, the second in a row raises the condition, a success
// clears the count and the condition.
func notePut(ctx context.Context, node string, put error) {
filesFailMu.Lock()
if put == nil {
delete(filesFailed, node)
} else {
filesFailed[node]++
}
n := filesFailed[node]
filesFailMu.Unlock()
keeper := conditionsFrom
if keeper == nil {
return
}
key := filesNotPutObservation(node, "").Key()
switch {
case put == nil:
if _, err := keeper.Clear(ctx, key, node+" was sent its files by digest"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
case n >= filesNotPutAfter:
summary := fmt.Sprintf("%s's files that travel by digest could not be put in its bucket %d sends in a row, so it is sent them "+
"inline: %v", node, n, put)
if _, err := keeper.Observe(ctx, filesNotPutObservation(node, summary)); err != nil {
fmt.Printf("the condition %s could not be raised: %v\n", key, err)
}
}
}
@@ -0,0 +1,246 @@
//go:build integration
package main
import (
"bytes"
"context"
"encoding/json"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
type sigOf struct{}
func (sigOf) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// 1696/AC7 (ADR 0486 §1): on a real bus with its default largest message of 1 MiB, a declaration whose files add up to more than
// 3 MiB is sent as references, its objects are in the bucket, each fetched back whole and verified against the digest and size the
// declaration names; the same declaration inline is refused by the controller, naming size and limit. A changed byte in an object
// fails the verification.
func TestADeclarationOfMoreThanThreeMiBIsSentByReferenceOnARealBus(t *testing.T) { // 1696/AC7
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
nc, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
if _, err := nc.AddStream(&nats.StreamConfig{Name: "NODES", Subjects: []string{"mesh.node.*.declare"},
MaxMsgsPerSubject: 1}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
api, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatal(err)
}
// Run twice: an assertion of an existing bucket is no error and no change.
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatalf("asserting the bucket again: %v", err)
}
var resources []map[string]any
for i := 0; i < 3; i++ {
resources = append(resources, aFileResource("big-"+string(rune('a'+i)), contentOf("big"+string(rune('a'+i)), 1200<<10)))
}
resources = append(resources, aFileResource("small", "tiny\n"))
inline := sendable{Resources: resources}
byDigest := sendable{fileReferences: true, Resources: resources}
bus := link.OverNATS{Conn: conn, JS: nc}
body, err := inline.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", body, 5*time.Second); err == nil ||
!strings.Contains(err.Error(), "1048576") || strings.Contains(err.Error(), "maximum payload exceeded") {
t.Fatalf("the inline declaration of more than 3 MiB: %v", err)
}
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
objects, err := byDigest.objects()
if err != nil || len(objects) != 3 {
t.Fatalf("objects: %d %v", len(objects), err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatal(err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatalf("putting them again: %v", err)
}
refBody, err := byDigest.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", refBody, 5*time.Second); err != nil {
t.Fatalf("the declaration by reference: %v", err)
}
// What the stream holds for the node is the reference form, and it names exactly the three objects.
latest, err := latestDeclarations(ctx, conn, []string{"novox", "ace"})
if err != nil || len(latest) != 1 {
t.Fatalf("the latest declarations: %v %v", len(latest), err)
}
named, err := namedByLatest("novox", latest["novox"])
if err != nil || len(named) != 3 {
t.Fatalf("what the latest declaration names: %v %v", named, err)
}
novoxBucket, err := broker.DeclarationFilesBucketOf("novox")
if err != nil {
t.Fatal(err)
}
store, err := api.ObjectStore(ctx, novoxBucket)
if err != nil {
t.Fatal(err)
}
var envelope struct {
Resources []sdkdecl.File `json:"resources"`
}
if err := json.Unmarshal(refBody, &envelope); err != nil {
t.Fatal(err)
}
for _, f := range envelope.Resources {
if !f.IsReference() {
continue
}
name, _ := sdkdecl.ObjectName(f.Digest)
got, err := store.GetBytes(ctx, name)
if err != nil {
t.Fatalf("the object of %s: %v", f.ID, err)
}
if err := f.Verify(got); err != nil {
t.Errorf("the object does not verify: %v", err)
}
got[len(got)/2] ^= 1
if err := f.Verify(got); err == nil {
t.Errorf("a changed byte in the object of %s passed the verification", f.ID)
}
}
// The sweep against the real bucket: all named, nothing marked; a fourth object named by none is marked, kept a day less a
// minute, and removed at the day.
extra := bytes.Repeat([]byte("x"), 100)
extraName := strings.Repeat("e", 64)
if err := putObjects(ctx, bucket, map[string][]byte{extraName: extra}); err != nil {
t.Fatal(err)
}
b := busSweptBucket{store}
t0 := time.Now()
if r, err := sweepDeclarationFiles(ctx, b, named, t0); err != nil || r.Marked != 1 || r.Removed != 0 || r.Kept != 3 {
t.Fatalf("first sweep: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour-time.Minute)); err != nil || r.Removed != 0 {
t.Fatalf("a minute short of a day: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour+time.Second)); err != nil || r.Removed != 1 {
t.Fatalf("a day after: %+v %v", r, err)
}
if infos, err := store.List(ctx); err != nil || len(infos) != 3 {
t.Fatalf("the bucket holds %d objects (%v), want the 3 the declaration names", len(infos), err)
}
}
// 1696/AC8 on a real bucket: the sweep removes an object only if it still carries the mark it judged it by. An object marked
// unnamed, then put again by a send (which erases the mark), is kept when the sweep deletes it by the old mark.
func TestTheSweepsDeleteLeavesAnObjectASendPutAgainAfterItWasMarked(t *testing.T) { // 1696/AC8
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
name := strings.Repeat("a", 64)
content := map[string][]byte{name: bytes.Repeat([]byte("y"), 100)}
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
b := busSweptBucket{bucket.store}
judged := time.Now().Add(-25 * time.Hour).Truncate(time.Second)
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if _, marked, err := bucket.Has(ctx, name); err != nil || !marked {
t.Fatalf("the object was not marked: %v %v", marked, err)
}
// The sweep has read the mark; a send now puts the object again, which erases it.
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
if have, marked, err := bucket.Has(ctx, name); err != nil || !have || marked {
t.Fatalf("a send did not put the marked object again: have %v marked %v %v", have, marked, err)
}
gone, err := b.Delete(ctx, name, judged)
if err != nil || gone {
t.Fatalf("the sweep deleted by an old mark: gone %v, %v", gone, err)
}
if have, _, err := bucket.Has(ctx, name); err != nil || !have {
t.Fatalf("the object a send relies on is gone: %v %v", have, err)
}
// And the mark still on it is honoured: marked again and deleted by that mark, it goes.
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if gone, err := b.Delete(ctx, name, judged); err != nil || !gone {
t.Fatalf("an object still carrying the mark it was judged by is kept: gone %v, %v", gone, err)
}
}
// 1696/AC9 on a real bus: the sweep of a node's bucket takes the node's lock first, so it waits for a send that has put its objects
// and not yet published the declaration that names them.
func TestTheSweepOfANodesBucketHoldsTheNodesLock(t *testing.T) { // 1696/AC9
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
const node = "sweep-lock-node"
if _, err := openBucket(ctx, conn, node); err != nil { // the bucket exists, or the sweep would return before the lock
t.Fatal(err)
}
release := lockFilesOf(node) // a send is between its put and its publication
done := make(chan struct{})
go func() {
_, _ = sweepNode(ctx, conn, node)
close(done)
}()
select {
case <-done:
t.Fatal("the sweep judged the node's bucket while a send held it")
case <-time.After(300 * time.Millisecond):
}
release()
select {
case <-done:
case <-time.After(20 * time.Second):
t.Fatal("the sweep did not go on once the send let go")
}
}
@@ -0,0 +1,534 @@
package main
import (
"context"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// contentOf is a deterministic content of n bytes that does not compress to nothing and holds no character JSON escapes.
func contentOf(seed string, n int) string {
var b strings.Builder
for i := 0; b.Len() < n; i++ {
fmt.Fprintf(&b, "%s-%07d-%x;", seed, i, i*2654435761%1000003)
}
return b.String()[:n]
}
func aFileResource(id, content string) map[string]any {
return map[string]any{"id": id, "type": "file", "path": "/etc/" + id, "mode": "0644", "content": content}
}
// controlNodeShape is a declaration shaped like the control node's on 2026-10-21 (issue 1689), the sizes measured then: a bus users
// file of 252,678 bytes, the vault's export of 138,475 bytes (compact), the tool runtime 21 KB, the packet filter 19 KB, the launcher
// 14 KB, and about 600 small resources for the rest (about 286 KB). `more` scales the count of small ones: 1 is the shape, 1.4 forty
// percent more of them.
func controlNodeShape(more float64) []map[string]any {
res := []map[string]any{
aFileResource("nats.bus-users", contentOf("users", 252678)),
aFileResource("mesh-vault.kept", contentOf("kept", 138475)),
aFileResource("node-tools.runtime", contentOf("runtime", 21<<10)),
aFileResource("nftables.filtering", contentOf("filter", 19<<10)),
aFileResource("mesh-host.launcher", contentOf("launcher", 14<<10)),
}
for i := 0; i < int(600*more); i++ {
res = append(res, aFileResource(fmt.Sprintf("module-%03d.config", i), contentOf(fmt.Sprintf("small%d", i), 380)))
}
return res
}
func wireSizeOf(t *testing.T, s sendable) int {
t.Helper()
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
// The signature the controller's key makes is 64 bytes.
msg, err := link.Envelope(body, make([]byte, 64))
if err != nil {
t.Fatal(err)
}
return len(msg)
}
// 1696/AC1: for a node that reports the capability, the two big files of a control-node-shaped declaration carry their digest and size
// and no content, and the declaration is far under the bus's 1 MiB.
func TestAFileOver64KiBTravelsByDigestToANodeThatReadsOne(t *testing.T) { // 1696/AC1
users, kept := contentOf("users", 252678), contentOf("kept", 138475)
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("nats.bus-users", users), aFileResource("mesh-vault.kept", kept),
aFileResource("small", "tiny\n"),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, want := range []string{users, kept} {
r := got.Resources[i]
digest := sdkdecl.DigestOf([]byte(want))
if _, has := r["content"]; has || r["digest"] != digest || r["size"] != float64(len(want)) {
t.Errorf("%v goes with content %v, digest %v, size %v; want no content, %s and %d", r["id"], r["content"] != nil,
r["digest"], r["size"], digest, len(want))
}
if r["path"] == nil || r["mode"] != "0644" {
t.Errorf("%v lost its path or mode: %v", r["id"], r)
}
}
if got.Resources[2]["content"] != "tiny\n" || got.Resources[2]["digest"] != nil {
t.Errorf("a small file is not inline: %v", got.Resources[2])
}
if strings.Contains(string(body), users[:200]) || strings.Contains(string(body), kept[:200]) {
t.Error("the content of a file that went by digest is in the body")
}
if n := wireSizeOf(t, s); n > 100<<10 {
t.Errorf("the declaration is %d bytes on the wire, want under 100 KB", n)
}
// The objects to put are the contents, each under the hex of its digest, and a reference verifies against its object.
objects, err := s.objects()
if err != nil || len(objects) != 2 {
t.Fatalf("objects: %v %v", objects, err)
}
for _, want := range []string{users, kept} {
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(want)))
if string(objects[name]) != want {
t.Errorf("the object %s does not hold the content", name[:12])
}
}
named, err := namedObjects(body)
if err != nil || len(named) != 2 {
t.Fatalf("the objects the body names: %v %v", named, err)
}
for name := range objects {
if _, ok := named[name]; !ok {
t.Errorf("the body does not name the object %s it was sent with", name[:12])
}
if _, err := hex.DecodeString(name); err != nil || len(name) != 64 {
t.Errorf("the object name %q is not 64 hex characters", name)
}
}
}
// 1696/AC2: a node that does not report the capability is sent every content inline, as before, and has no object put.
func TestANodeWithoutTheCapabilityIsSentEveryContentInline(t *testing.T) { // 1696/AC2
users := contentOf("users", 252678)
s := sendable{Resources: []map[string]any{aFileResource("nats.bus-users", users)}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if got.Resources[0]["content"] != users || got.Resources[0]["digest"] != nil || got.Resources[0]["size"] != nil {
t.Fatalf("a node that reads no references is not sent the content inline: %v", got.Resources[0]["digest"])
}
if objects, err := s.objects(); err != nil || len(objects) != 0 {
t.Fatalf("objects for a node that reads none: %v %v", len(objects), err)
}
}
// 1696/AC3: only a file that may travel by digest does: one of exactly 64 KiB, a sealed one, one filled with secrets, one written
// into another file, and a directory stay inline, whatever the node reads.
func TestOnlyAFileThatMayTravelByDigestDoes(t *testing.T) { // 1696/AC3
exact := contentOf("exact", 64<<10)
big := contentOf("big", 64<<10+1)
sealed := aFileResource("sealed", big)
sealed["sealed"], sealed["content"] = "box", ""
sealed["content"] = big // a sealed file's content is none; the body must pass it as it is, whatever it holds
secret := aFileResource("secret", big)
secret["secrets"] = map[string]any{"token": "x"}
into := aFileResource("into", big)
into["into"] = "/etc/other"
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("exact", exact), sealed, secret, into,
{"id": "dir", "type": "directory", "path": "/etc/d", "content": big},
aFileResource("big", big),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, id := range []string{"exact", "sealed", "secret", "into", "dir"} {
if got.Resources[i]["id"] != id || got.Resources[i]["digest"] != nil || got.Resources[i]["content"] == nil {
t.Errorf("%s went by digest or lost its content: %v", id, got.Resources[i]["digest"])
}
}
if big := got.Resources[5]; big["id"] != "big" || big["digest"] == nil || big["content"] != nil {
t.Errorf("a plain file of 64 KiB and a byte did not go by digest: %v", big["digest"])
}
}
// A file given as bytes goes by digest on its decoded length, with the decoded bytes in the object.
func TestAFileOfBytesTravelsByDigestOnItsDecodedLength(t *testing.T) {
raw := []byte(contentOf("png", 70<<10))
r := map[string]any{"id": "icon", "type": "file", "path": "/usr/share/icon.png", "bytes": stdBase64(raw)}
s := sendable{fileReferences: true, Resources: []map[string]any{r}}
objects, err := s.objects()
if err != nil || len(objects) != 1 {
t.Fatalf("%v %v", len(objects), err)
}
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf(raw))
if string(objects[name]) != string(raw) {
t.Fatal("the object is not the decoded bytes")
}
if r["bytes"] == nil {
t.Fatal("the composed resource was changed in place")
}
}
type memObjects struct {
have map[string][]byte
marked map[string]bool
puts []string
fail map[string]error
}
func (m *memObjects) Has(_ context.Context, name string) (bool, bool, error) {
_, ok := m.have[name]
return ok, m.marked[name], nil
}
func (m *memObjects) Put(_ context.Context, name string, c []byte) error {
if err := m.fail[name]; err != nil {
return err
}
m.puts = append(m.puts, name)
m.have[name] = c
delete(m.marked, name)
return nil
}
// 1696/AC4: an object already in the bucket is not put again, one absent is, and a put that fails stops the send, naming the object.
func TestObjectsAreInTheBucketBeforeTheDeclarationIsSent(t *testing.T) { // 1696/AC4
objects := map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")}
b := &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one")}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("b", 64) {
t.Fatalf("put %v (%v), want the one absent", b.puts, err)
}
// An object the sweep marked unnamed is put again, which erases the mark: a send relies on it.
b = &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")},
marked: map[string]bool{strings.Repeat("a", 64): true}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("a", 64) || b.marked[strings.Repeat("a", 64)] {
t.Fatalf("a marked object: put %v (%v) marked %v, want it put again and the mark gone", b.puts, err, b.marked)
}
b = &memObjects{have: map[string][]byte{}, fail: map[string]error{strings.Repeat("a", 64): errors.New("permissions violation")}}
err := putObjects(t.Context(), b, objects)
if err == nil || !strings.Contains(err.Error(), "aaaaaaaaaaaa") || !strings.Contains(err.Error(), "permissions violation") {
t.Fatalf("a failed put: %v", err)
}
}
// fallbackDelivery refuses the first declaration it is sent with the bucket's failure, as a bus that has not yet been given the
// controller's grant does, and takes the second.
type fallbackDelivery struct {
bodies [][]byte
first error
}
func (f *fallbackDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *fallbackDelivery) declare(_ context.Context, _ readyNode, body []byte) (string, error) {
f.bodies = append(f.bodies, body)
if len(f.bodies) == 1 && f.first != nil {
return "", fmt.Errorf("x: %w: %w", errFilesNotPut, f.first)
}
return "digest", nil
}
// 1696/AC5: when the bucket cannot be written yet (the grant travels in the declaration of the machine holding the bus), that send goes
// with every content inline and is made; a send that fails for another reason is not retried inline.
func TestASendWhoseFilesCannotBePutGoesInlineOnce(t *testing.T) { // 1696/AC5
users := contentOf("users", 100<<10)
s := readyNode{node: "novox", declared: sendable{fileReferences: true, Resources: []map[string]any{aFileResource("nats.bus-users", users)}}}
f := &fallbackDelivery{first: errors.New("timeout")}
digests, err := deliver(t.Context(), f, "novox", []readyNode{s})
if err != nil || digests["novox"] != "digest" || len(f.bodies) != 2 {
t.Fatalf("%v %v %d", digests, err, len(f.bodies))
}
if strings.Contains(string(f.bodies[0]), users[:100]) || !strings.Contains(string(f.bodies[1]), users[:100]) {
t.Error("the first body should name the file by digest and the second carry it inline")
}
other := &fallbackDelivery{}
other.first = nil
failing := &failingDelivery{err: errors.New("the bus is down")}
if _, err := deliver(t.Context(), failing, "novox", []readyNode{s}); err == nil || failing.calls != 1 {
t.Fatalf("a failure of another kind: %v after %d send(s)", err, failing.calls)
}
}
type failingDelivery struct {
err error
calls int
}
func (f *failingDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *failingDelivery) declare(context.Context, readyNode, []byte) (string, error) {
f.calls++
return "", f.err
}
// A bucket for the sweep.
type sweepBucketFake struct {
objects map[string]time.Time // zero: named at the last look
deleted []string
}
func (b *sweepBucketFake) Objects(context.Context) ([]bucketObject, error) {
var out []bucketObject
for n, at := range b.objects {
out = append(out, bucketObject{Name: n, UnnamedSince: at})
}
return out, nil
}
func (b *sweepBucketFake) MarkUnnamed(_ context.Context, n string, at time.Time) error {
b.objects[n] = at
return nil
}
func (b *sweepBucketFake) ClearUnnamed(_ context.Context, n string) error {
b.objects[n] = time.Time{}
return nil
}
func (b *sweepBucketFake) Delete(_ context.Context, n string, markedAt time.Time) (bool, error) {
if at, ok := b.objects[n]; !ok || !at.Equal(markedAt) {
return false, nil // put again since it was judged: its mark is gone
}
delete(b.objects, n)
b.deleted = append(b.deleted, n)
return true, nil
}
// 1696/AC6: an object a latest declaration still names is not removed, however long; one unnamed is marked at the first look and removed
// only when the mark is 24 hours old; one named again loses its mark.
func TestAnObjectIsRemovedADayAfterNoLatestDeclarationNamesIt(t *testing.T) { // 1696/AC6
named, unnamed, back := strings.Repeat("1", 64), strings.Repeat("2", 64), strings.Repeat("3", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{named: t0.Add(-100 * time.Hour), unnamed: {}, back: t0.Add(-5 * time.Hour)}}
named3 := map[string]bool{named: true, back: true}
r, err := sweepDeclarationFiles(t.Context(), b, named3, t0)
if err != nil || r.Marked != 1 || r.Cleared != 2 || r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("first look: %+v %v %v", r, err, b.deleted)
}
if !b.objects[named].IsZero() || !b.objects[back].IsZero() || !b.objects[unnamed].Equal(t0) {
t.Fatalf("marks after the first look: %v", b.objects)
}
// Still unnamed 23 hours 59 minutes later: kept. A day after the mark: removed. The named ones stay for ever.
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour-time.Minute)); err != nil ||
r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("a minute short of a day: %+v %v %v", r, err, b.deleted)
}
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour)); err != nil ||
r.Removed != 1 || len(b.deleted) != 1 || b.deleted[0] != unnamed {
t.Fatalf("a day after: %+v %v %v", r, err, b.deleted)
}
if _, ok := b.objects[named]; !ok {
t.Fatal("an object a latest declaration names was removed")
}
// An object named again before the day is up loses its mark and is not removed.
b = &sweepBucketFake{objects: map[string]time.Time{unnamed: t0}}
if _, err := sweepDeclarationFiles(t.Context(), b, map[string]bool{unnamed: true}, t0.Add(23*time.Hour)); err != nil || !b.objects[unnamed].IsZero() {
t.Fatalf("named again: %v %v", b.objects, err)
}
}
// What a node's latest declaration names is read from the signed envelope as stored on the bus; an unreadable one is an error and
// never "names none", which would mark an object a node needs.
func TestTheObjectsALatestDeclarationNamesAreReadFromTheBus(t *testing.T) {
s := sendable{fileReferences: true, Resources: []map[string]any{aFileResource("a", contentOf("a", 70<<10))}}
body, _ := s.Body()
msg, _ := link.Envelope(body, []byte("sig"))
named, err := namedByLatest("novox", msg)
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(contentOf("a", 70<<10))))
if err != nil || len(named) != 1 || !named[name] {
t.Fatalf("%v %v", named, err)
}
if none, err := namedByLatest("ace", []byte(`{"declaration":"e30=","signature":""}`)); err != nil || len(none) != 0 {
t.Fatalf("a declaration naming none: %v %v", none, err)
}
if _, err := namedByLatest("novox", []byte("not json")); err == nil {
t.Fatal("an unreadable latest declaration was read as naming none")
}
}
// 1698/AC2 (ADR 0486 §9): a declaration shaped like the control node's, for a node that reads references, stays under half of the
// bus's largest message on the wire; the inline form of the same does not fit at all; and the same shape with 40 percent more resources
// fails the test's own limit, so a change that eats the room fails in a pull request and not on the live mesh.
func TestTheControlNodesDeclarationInTheReferenceFormStaysUnderHalfTheBusLimit(t *testing.T) { // 1698/AC2
const limit = 1048576
shape := sendable{fileReferences: true, Resources: controlNodeShape(1)}
n := wireSizeOf(t, shape)
t.Logf("the control node's shape in the reference form: %d bytes on the wire, %.1f%% of %d", n, 100*float64(n)/limit, limit)
if n > limit/2 {
t.Errorf("the reference form is %d bytes on the wire, over half (%d) of the bus's largest message", n, limit/2)
}
if inline := wireSizeOf(t, sendable{Resources: controlNodeShape(1)}); inline < limit*3/4 {
t.Errorf("the inline form is %d bytes, which is not the shape that was over the limit", inline)
}
grown := wireSizeOf(t, sendable{fileReferences: true, Resources: controlNodeShape(1.4)})
if grown <= limit/2 {
t.Errorf("the shape with 40%% more resources is %d bytes, which the test would let pass", grown)
}
}
func stdBase64(b []byte) string { return base64.StdEncoding.EncodeToString(b) }
// A sweep that read an object's mark before a send put it again must not remove it: the object, now without the mark, is left.
func TestASweepDoesNotRemoveAnObjectASendPutAgainAfterItWasJudged(t *testing.T) { // 1696/AC8
n := strings.Repeat("4", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{n: t0}}
judged := bucketObject{Name: n, UnnamedSince: t0}
b.objects[n] = time.Time{} // the send put it again: no mark
if gone, err := b.Delete(t.Context(), judged.Name, judged.UnnamedSince); err != nil || gone || len(b.deleted) != 0 {
t.Fatalf("an object put again was removed: %v %v %v", gone, err, b.deleted)
}
}
// One node's bucket is held by one act at a time in this process: the sweep waits for a send that has put its objects and not yet
// published the declaration that names them (1696/AC9).
func TestTheSweepOfANodesBucketWaitsForASendThatHasPutItsObjects(t *testing.T) { // 1696/AC9
unlock := lockFilesOf("lock-test-node")
got := make(chan struct{})
go func() {
defer lockFilesOf("lock-test-node")()
close(got)
}()
select {
case <-got:
t.Fatal("the sweep took the node's bucket while a send held it")
case <-time.After(50 * time.Millisecond):
}
other := lockFilesOf("lock-test-other-node") // another node's bucket is not held
other()
unlock()
select {
case <-got:
case <-time.After(2 * time.Second):
t.Fatal("the sweep did not take the node's bucket once the send was done")
}
}
// A put that fails send after send raises a condition; the first failure does not (the grant it waits for travels in that very
// declaration), and a put that works clears it (1696/AC10).
func TestAPutThatFailsSendAfterSendRaisesAConditionAndClearsWhenItWorks(t *testing.T) { // 1696/AC10
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return time.Date(2026, 10, 21, 14, 0, 0, 0, time.UTC) }})
t.Cleanup(func() { k.Close(t.Context()) })
was := conditionsFrom
conditionsFrom = k
t.Cleanup(func() { conditionsFrom = was })
key := filesNotPutObservation("cond-node", "").Key()
open := func() bool {
_, found, err := conditions.ReadOne(t.Context(), store, key)
if err != nil {
t.Fatal(err)
}
return found
}
boom := errors.New("permissions violation")
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the first failure raised the condition")
}
notePut(t.Context(), "cond-node", boom)
if !open() {
t.Fatal("the second failure in a row did not raise the condition")
}
notePut(t.Context(), "cond-node", nil)
if open() {
t.Fatal("a put that worked did not clear the condition")
}
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the count did not start again after a put that worked")
}
}
// aBusThatSays is a bus that keeps the declarations published to it and, at each, asks whether the node's bucket is held.
type aBusThatSays struct {
link.Bus
published chan string
heldAtPublish func() bool
}
func (b *aBusThatSays) PublishDeclaration(_ context.Context, node string, _ []byte) error {
if b.heldAtPublish != nil && !b.heldAtPublish() {
return errors.New("published while the node's bucket was not held")
}
b.published <- node
return errors.New("stop here: nothing is recorded")
}
type aSignerOfZeros struct{}
func (aSignerOfZeros) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// whileHeld says whether the node's lock is held by someone right now, without keeping it.
func whileHeld(node string) bool {
got := make(chan struct{})
go func() {
defer lockFilesOf(node)()
close(got)
}()
select {
case <-got:
return false
case <-time.After(100 * time.Millisecond):
// held: the goroutine stays waiting and takes and releases the lock once it is let go.
return true
}
}
// 1696/AC9: a send holds the node's lock from the put to the publication, so the sweep never removes an object between them. The
// test goes through overTheBus.declare: while another holds the lock the declaration is not published, and at the publication the
// lock is held.
func TestASendHoldsTheNodesLockFromThePutToThePublication(t *testing.T) { // 1696/AC9
const node = "lock-send-node"
bus := &aBusThatSays{published: make(chan string, 1)}
bus.heldAtPublish = func() bool { return whileHeld(node) }
send := overTheBus{server: link.SendingOn(bus), signer: aSignerOfZeros{}}
release := lockFilesOf(node) // a sweep is judging this node's bucket
done := make(chan error, 1)
go func() {
_, err := send.declare(t.Context(), readyNode{node: node}, []byte(`{}`))
done <- err
}()
select {
case n := <-bus.published:
t.Fatalf("%s was sent its declaration while the sweep held the node's bucket", n)
case <-time.After(150 * time.Millisecond):
}
release()
select {
case <-bus.published:
case <-time.After(5 * time.Second):
t.Fatal("the send did not go on once the sweep let go")
}
if err := <-done; err == nil || !strings.Contains(err.Error(), "nothing is recorded") {
t.Fatalf("the send did not reach the publication (a lock held at it): %v", err)
}
}
+2 -2
View File
@@ -13,8 +13,8 @@ import (
// **A declaration is one bus message, and the bus's largest message is a cliff** (novox/hq issue 1689). On 2026-10-21
// the control node's declaration passed it by 2 KB and nothing reached that node until two assignments were undone.
// So the size is measured at every send: a warning condition from 90% of the limit, urgent over it, cleared when
// the declaration is back under 90%. The limit is the connection's own, not a constant.
// So the size is measured at every send: a warning condition from 75% of the limit (ADR 0486 §8, issue 1698: it was 90% while every file travelled inline), urgent over it, cleared when
// the declaration is back under 75%. The limit is the connection's own, not a constant.
// kindDeclarationNearLimit is a declaration at or near the bus's largest message.
const kindDeclarationNearLimit = "declaration-near-limit"
+8 -8
View File
@@ -33,7 +33,7 @@ func TestTheVaultsExportIsSentCompactAndNothingElseIsTouched(t *testing.T) {
}
}
// From 90% of the bus's largest message the declaration is a warning; over it, urgent; back under 90%, cleared.
// From 75% of the bus's largest message (ADR 0486 §8, issue 1698) the declaration is a warning; over it, urgent; back under 75%, cleared.
func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *testing.T) {
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
@@ -50,22 +50,22 @@ func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *test
return c, found
}
settleDeclarationRoom(t.Context(), k, "novox", 899, limit)
settleDeclarationRoom(t.Context(), k, "novox", 749, limit)
if _, found := read(); found {
t.Fatal("a declaration at 89.9% raised a condition")
t.Fatal("a declaration at 74.9% raised a condition")
}
settleDeclarationRoom(t.Context(), k, "novox", 900, limit)
settleDeclarationRoom(t.Context(), k, "novox", 750, limit)
c, found := read()
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "900 bytes") ||
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "750 bytes") ||
!strings.Contains(c.Summary, "1000") || c.Subject.Machine != "novox" {
t.Fatalf("at 90%% the condition is %v %+v", found, c)
t.Fatalf("at 75%% the condition is %v %+v", found, c)
}
settleDeclarationRoom(t.Context(), k, "novox", 1001, limit)
if c, _ := read(); c.Severity != conditions.Urgent {
t.Fatalf("over the limit the condition is %q, expected urgent", c.Severity)
}
settleDeclarationRoom(t.Context(), k, "novox", 850, limit)
settleDeclarationRoom(t.Context(), k, "novox", 700, limit)
if _, found := read(); found {
t.Fatal("the condition stayed after the declaration fell under 90%")
t.Fatal("the condition stayed after the declaration fell under 75%")
}
}
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **An object is removed a day after no node's latest declaration names it** (novox/hq ADR 0486 §3, issue 1696), so a node that is
// asleep and receives its latest declaration later still finds every object that declaration names. What each node's latest
// declaration names is read from the bus itself, the last message of that node's subject of the stream NODES: nothing else
// is kept of it. When an object is first found named by none, the time is written in its own metadata (declarationUnnamed);
// found named again, that is erased; found unnamed for a day, it is removed. Each node has a bucket of its own, judged against that
// node's latest declaration alone. Objects are immutable and regenerable from the
// controller's state, so a removal loses nothing.
// declarationUnnamed is the key of an object's metadata holding when it was first found named by no node's latest declaration.
const declarationUnnamed = "unnamed-since"
// sweepEvery is how often the bucket is judged: the removal is exact to within this.
const sweepEvery = time.Hour
// bucketObject is one object of the bucket as the sweep sees it.
type bucketObject struct {
Name string
// UnnamedSince is when it was first found named by none, zero when it was named at the last look.
UnnamedSince time.Time
}
// sweptBucket is what the sweep needs of the bucket.
type sweptBucket interface {
Objects(ctx context.Context) ([]bucketObject, error)
MarkUnnamed(ctx context.Context, name string, since time.Time) error
ClearUnnamed(ctx context.Context, name string) error
// Delete removes the object if it still carries the mark it was judged by, and says whether it did: a send that put it again
// since has erased the mark, and the object is left.
Delete(ctx context.Context, name string, markedAt time.Time) (bool, error)
}
// filesSweep says what a sweep did.
type filesSweep struct{ Kept, Marked, Cleared, Removed int }
// sweepDeclarationFiles judges every object against the names of the latest declarations: named ones are kept (and a
// mark of being unnamed is erased), unnamed ones are marked at the first look and removed once the mark is a day old.
func sweepDeclarationFiles(ctx context.Context, bucket sweptBucket, named map[string]bool, now time.Time) (filesSweep, error) {
var r filesSweep
objects, err := bucket.Objects(ctx)
if err != nil {
return r, err
}
sort.Slice(objects, func(i, j int) bool { return objects[i].Name < objects[j].Name })
for _, o := range objects {
switch {
case named[o.Name]:
r.Kept++
if !o.UnnamedSince.IsZero() {
if err := bucket.ClearUnnamed(ctx, o.Name); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Cleared++
}
case o.UnnamedSince.IsZero():
if err := bucket.MarkUnnamed(ctx, o.Name, now); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Marked++
case now.Sub(o.UnnamedSince) >= broker.DeclarationFilesKeptFor:
gone, err := bucket.Delete(ctx, o.Name, o.UnnamedSince)
if err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
if gone {
r.Removed++
} else {
r.Kept++
}
default:
r.Kept++
}
}
return r, nil
}
// namedByLatest is the name of every object this latest declaration of a node names. A body that cannot be read is an error, never
// "names none": an object the node still needs would be marked for removal.
func namedByLatest(node string, signed []byte) (map[string]bool, error) {
var s link.Signed
if err := json.Unmarshal(signed, &s); err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
names, err := namedObjects(s.Declaration)
if err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
named := make(map[string]bool, len(names))
for name := range names {
named[name] = true
}
return named, nil
}
// latestDeclarations are the last signed declarations of the nodes, by node, as the stream NODES holds them. A node with none has
// no entry.
func latestDeclarations(ctx context.Context, conn *nats.Conn, nodes []string) (map[string][]byte, error) {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return nil, err
}
out := map[string][]byte{}
for _, node := range nodes {
msg, err := js.GetLastMsg("NODES", link.DeclareSubject(node), nats.Context(ctx))
if errors.Is(err, nats.ErrMsgNotFound) {
continue
}
if err != nil {
return nil, fmt.Errorf("the latest declaration of %s: %w", node, err)
}
out[node] = msg.Data
}
return out, nil
}
// busSweptBucket is the bucket on the controller's connection.
type busSweptBucket struct{ store jetstream.ObjectStore }
func (b busSweptBucket) Objects(ctx context.Context) ([]bucketObject, error) {
infos, err := b.store.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return nil, nil
}
if err != nil {
return nil, err
}
out := make([]bucketObject, 0, len(infos))
for _, info := range infos {
o := bucketObject{Name: info.Name}
if at, err := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); err == nil {
o.UnnamedSince = at
}
out = append(out, o)
}
return out, nil
}
func (b busSweptBucket) meta(ctx context.Context, name string, change func(map[string]string)) error {
info, err := b.store.GetInfo(ctx, name)
if err != nil {
return err
}
meta := info.ObjectMeta
kept := map[string]string{}
for k, v := range meta.Metadata {
kept[k] = v
}
change(kept)
meta.Metadata = kept
return b.store.UpdateMeta(ctx, name, meta)
}
func (b busSweptBucket) MarkUnnamed(ctx context.Context, name string, since time.Time) error {
return b.meta(ctx, name, func(m map[string]string) { m[declarationUnnamed] = since.UTC().Format(time.RFC3339) })
}
func (b busSweptBucket) ClearUnnamed(ctx context.Context, name string) error {
return b.meta(ctx, name, func(m map[string]string) { delete(m, declarationUnnamed) })
}
func (b busSweptBucket) Delete(ctx context.Context, name string, markedAt time.Time) (bool, error) {
info, err := b.store.GetInfo(ctx, name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return false, nil
}
if err != nil {
return false, err
}
if at, perr := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); perr != nil || !at.Equal(markedAt) {
return false, nil
}
return true, b.store.Delete(ctx, name)
}
// sweepingDeclarationFiles judges the bucket every sweepEvery for as long as this controller serves and holds the lease.
func sweepingDeclarationFiles(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) {
tick := time.NewTicker(sweepEvery)
defer tick.Stop()
failing := ""
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
if epoch, err := theLease.epoch(ctx); err != nil || epoch == 0 {
continue
}
why := sweepOnce(ctx, inv, conn)
if why != failing {
if why != "" {
fmt.Printf("the bucket of declaration files cannot be swept: %s\n", why)
} else {
fmt.Printf("the bucket of declaration files is swept again\n")
}
failing = why
}
}
}
// sweepOnce judges the bucket of every node against that node's latest declaration, one bucket at a time under the node's lock, with
// the declaration read again under it: a send that put an object and published its declaration is seen, and one that has not yet
// is waiting for the lock.
func sweepOnce(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) string {
nodes, err := inv.Nodes(ctx)
if err != nil {
return fmt.Sprintf("the nodes cannot be read: %v", err)
}
var problems []string
var total filesSweep
for _, n := range nodes {
r, err := sweepNode(ctx, conn, n.Name)
total.Kept, total.Marked, total.Cleared, total.Removed = total.Kept+r.Kept, total.Marked+r.Marked, total.Cleared+r.Cleared, total.Removed+r.Removed
if err != nil {
problems = append(problems, fmt.Sprintf("%s: %v", n.Name, err))
}
}
if total.Removed+total.Marked+total.Cleared > 0 {
fmt.Printf("the buckets of declaration files: %d object(s) kept, %d found unnamed, %d named again, %d removed\n",
total.Kept, total.Marked, total.Cleared, total.Removed)
}
return strings.Join(problems, "; ")
}
func sweepNode(ctx context.Context, conn *nats.Conn, node string) (filesSweep, error) {
working, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
js, err := jetstream.New(conn)
if err != nil {
return filesSweep{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(working, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return filesSweep{}, nil // a node never sent a reference has no bucket yet
}
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket cannot be read: %w", err)
}
defer lockFilesOf(node)()
latest, err := latestDeclarations(working, conn, []string{node})
if err != nil {
return filesSweep{}, err
}
named := map[string]bool{}
if body, ok := latest[node]; ok {
if named, err = namedByLatest(node, body); err != nil {
return filesSweep{}, err
}
}
return sweepDeclarationFiles(working, busSweptBucket{store}, named, time.Now())
}
@@ -0,0 +1,89 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// 1840/AC5 (ADR 0507 §5): the source of the declared-packages mirror composes each node and names, for a package two modules
// declare and one declared absent, the modules that declare it; the same records give the same bytes on a second pass, so
// the mirror writes nothing; a node with no package has a key that says so.
func TestTheDeclaredPackagesMirrorNamesWhatEachNodesModulesDeclare(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "viewer", Version: "1", Resources: []map[string]any{
{"id": "pkg-curl", "type": "package", "package": "curl"},
{"id": "pkg-nano", "type": "package", "package": "nano", "absent": true}}})
register(t, open, catalogue.Manifest{Module: "fetcher", Version: "1", Resources: []map[string]any{
{"id": "pkg-curl", "type": "package", "package": "curl"}}})
for _, module := range []string{"viewer", "fetcher"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatalf("assign anchor %s: %v", module, err)
}
}
source := declaredPackagesMirror(open)
first, err := source(ctx)
if err != nil {
t.Fatal(err)
}
var got inventory.DeclaredPackagesMirrored
if err := json.Unmarshal(first["anchor"], &got); err != nil {
t.Fatalf("anchor has no value: %v", err)
}
// Every node also runs the mesh's own modules, which declare packages of their own: curl and nano are looked
// up by name, and no package names viewer or fetcher but those they declare.
byName := map[string]inventory.PackageDeclared{}
for _, p := range got.Packages {
byName[p.Package] = p
}
same := func(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
for _, p := range []inventory.PackageDeclared{{Package: "curl", Modules: []string{"fetcher", "viewer"}},
{Package: "nano", Modules: []string{"viewer"}, Absent: true}} {
if g, ok := byName[p.Package]; !ok || g.Absent != p.Absent || !same(g.Modules, p.Modules) {
t.Errorf("anchor declares %s as %+v, want %+v (all: %+v)", p.Package, g, p, got.Packages)
}
}
for _, p := range got.Packages {
if p.Package == "curl" || p.Package == "nano" {
continue
}
for _, m := range p.Modules {
if m == "viewer" || m == "fetcher" {
t.Errorf("%s is said to be declared by %s, which does not declare it", p.Package, m)
}
}
}
var laptop inventory.DeclaredPackagesMirrored
if err := json.Unmarshal(first["laptop"], &laptop); err != nil || laptop.Node != "laptop" || laptop.Packages == nil {
t.Errorf("a node assigned neither module has no value of its own: %s (%v)", first["laptop"], err)
}
for _, p := range laptop.Packages {
if p.Package == "curl" || p.Package == "nano" {
t.Errorf("laptop is assigned neither module and declares %+v", p)
}
}
second, err := source(ctx)
if err != nil {
t.Fatal(err)
}
for node, value := range first {
if string(second[node]) != string(value) {
t.Errorf("the value of %s changed between two passes over the same records:\n%s\n%s", node, value, second[node])
}
}
}
+15 -153
View File
@@ -815,16 +815,14 @@ func deliveryPlanDocument(m orderMember, entries []inventory.Entry, read map[str
// delivery walks [-n 50] [--plan <id>]
func deliveryCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("delivery plan|order|check|go|stop|walks")
return errors.New("delivery order|check|go|stop|walks")
}
sub, rest := args[0], args[1:]
if sub == "plan" {
// Removed (ADR 0450 §8; issue 1318): refused by name before any flag is read, naming its replacement.
return removedByName("delivery-plan")
}
set := flag.NewFlagSet("delivery "+sub, flag.ContinueOnError)
repository := set.String("repository", "", "owner/repository")
base := set.String("base", "main", "the branch it merges into")
head := set.String("head", "", "the commit at hand")
paths := set.String("paths", "", "the files it changes, comma-separated")
moduleDirs := set.String("module-dirs", "", "the directories holding a module.json at the head, comma-separated")
removed := set.String("removed", "", "the files it deletes, comma-separated")
membersJSON := set.String("members", "", "a group's members, as JSON")
membersFile := set.String("members-file", "", "a file of a group's members, as JSON: the same as --members, for a list "+
"too long for an argument (novox/hq issue 1272)")
@@ -872,19 +870,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
return enc.Encode(v)
}
switch sub {
case "plan":
if *repository == "" || *head == "" && *paths == "" {
return errors.New("delivery plan --repository owner/repo --head <commit> --paths a,b")
}
m := changeMember(*repository, *base, *head, splitList(*paths), splitList(*removed), moduleDirsOf(*moduleDirs))
entries, read, edges, err := theGraph(ctx, inv)
if err != nil {
return err
}
return answer(deliveryPlanDocument(m, entries, read, edges, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
}))
case "order":
if len(members) == 0 {
return errors.New("delivery order --members <json>: a group has members")
@@ -914,11 +899,9 @@ func deliveryCommand(ctx context.Context, args []string) error {
if *group == "" || len(members) < 2 {
return errors.New("delivery check --group <id> --members <json> (two heads or more), or --members <one head>")
}
id, err := askGroupCheck(ctx, open, *group, members)
if err != nil {
return err
}
return answer(map[string]any{"asked": id, "group": *group})
// **The group's composed check is the checker's** (novox/hq ADR 0503 §3; issue 1299, C2): nothing is asked of
// build-agent, and the answer says who judges it.
return answer(groupCheckAnswer(*group))
case "go":
if len(positionals) != 1 {
return errors.New("delivery go <plan> [--by <who>] [--why <text>]")
@@ -1093,38 +1076,6 @@ func theGraph(ctx context.Context, inv *inventory.Inventory) ([]inventory.Entry,
return entries, read, edges, nil
}
// cycleVerdict is the failing verdict of a group whose modules are ordered in a cycle, and whether there is one: the
// group's composed check, said without a build. The manifests' change is read from the forge; an answer it cannot give
// is an error, never read as no cycle.
func cycleVerdict(ctx context.Context, group string, members []orderMember, reach map[string]orderReach,
edges []inventory.Edge, entries []inventory.Entry) (link.Checked, bool, error) {
changes, err := changesOf(ctx, changeSourcesOf(members, entries, reach))
if err != nil {
return link.Checked{}, false, fmt.Errorf("what the group's members say of their manifests cannot be read: %w", err)
}
var moved []string
for _, m := range members {
moved = appendNew(moved, reach[m.ID].Moved...)
}
said := orderEdges(moved, changes, edges).CycleSaid()
if said == "" {
return link.Checked{}, false, nil
}
first := members[0].pull()
c := link.Checked{ID: link.NewBuildID(time.Now()), Owner: first.Owner, Repo: first.Repo, Number: first.Number,
Commit: first.Commit, Verdict: "fail", Summary: said, Report: said, Group: group,
Gate: &link.CheckLayer{Verdict: "fail", Summary: said}}
for _, m := range members {
if group == "" {
break // a lone pull request's verdict names no group and no member
}
p := m.pull()
c.Members = append(c.Members, link.CheckedMember{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
Trunk: twelve(p.BaseSHA)})
}
return c, true, nil
}
// twelve is a commit as twelve characters.
func twelve(c string) string {
if len(c) > 12 {
@@ -1133,93 +1084,6 @@ func twelve(c string) string {
return c
}
// groupPrimary is the head a group's composed check is run from: the one moving the controller, which then
// judges the group by itself; else the one moving the node-engine, whose validator judges; else the first.
func groupPrimary(members []orderMember, reach map[string]orderReach) int {
for _, want := range []string{"mesh-controller", "mesh-host"} {
for i, m := range members {
if slices.Contains(reach[m.ID].Moved, want) {
return i
}
}
}
return 0
}
// askGroupCheck asks the build seat for a group's composed check: every head laid over the mesh in turn,
// judged as one future state (novox/hq ADR 0239). The verdict comes back as `checked` with the group's id,
// for mesh-delivery; the forge's holder sets no status from it.
func askGroupCheck(ctx context.Context, open *stores, group string, members []orderMember) (string, error) {
entries, read, edges, err := theGraph(ctx, open.inventory)
if err != nil {
return "", err
}
reach := reachOfMembers(members, entries, read, edges)
// **A cycle of edges fails the group's composed check, naming every module and edge** (novox/hq ADR 0419 clause 6),
// before the build seat is asked: no composition of the heads has an order to be sent in.
if failing, cyclic, err := cycleVerdict(ctx, group, members, reach, edges, entries); err != nil {
return "", err
} else if cyclic {
sayChecked(ctx, os.Stderr, failing)
return failing.ID, nil
}
primary := groupPrimary(members, reach)
p := members[primary].pull()
scope := pullScope(p, entries, read, edges)
// The gate runs over what any member moves; a judge from the primary alone.
for _, m := range members {
r := reach[m.ID]
scope.Modules = appendNew(scope.Modules, r.Moved...)
scope.Dependents = appendNew(scope.Dependents, r.Dependents...)
}
request, err := checkRequestFor(ctx, open, p, scope, entries)
if err != nil {
return "", err
}
request.Check.Group = group
// A group's check is queued as ready when any member's pull request is marked ready.
request.Check.Ready = slices.ContainsFunc(members, func(m orderMember) bool { return m.Ready })
world, err := restOfTheMesh(ctx, open, shelfOf(entries), "")
if err != nil {
return "", err
}
for i, m := range members {
if i == primary {
continue
}
mp := m.pull()
// As the mesh clones a module built from it; a repository no module is built from, from the forge.
source := inventory.Source{Seat: gitSeat, Repository: mp.Owner + "/" + mp.Repo}
for _, e := range entries {
if !e.Provided && sameRepository(e.Source.Repository, link.SourceMoved{Owner: mp.Owner, Repo: mp.Repo}) {
source = e.Source
break
}
}
url := source.Repository
if source.Seat != "" {
url, err = clonedFromSeat(world, source.Seat, source.Repository)
}
if err != nil {
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return "", err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return "", err
}
fmt.Fprintf(os.Stderr, "group %s: asked %s to check %d head(s) composed together, as %s\n", group, seat,
len(members), request.ID)
return request.ID, nil
}
// appendNew appends what is not there yet.
func appendNew(to []string, items ...string) []string {
for _, i := range items {
@@ -1230,15 +1094,6 @@ func appendNew(to []string, items ...string) []string {
return to
}
// shelfOf is the catalogue's manifests by name.
func shelfOf(entries []inventory.Entry) map[string]catalogue.Manifest {
shelf := map[string]catalogue.Manifest{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
}
return shelf
}
// sayPlanMoved says a walk kept in a new state as the controller's `plan-moved`, the plan whole: what the
// delivery it walks reads its steps from. Never in the way of the walk: said beside it, and a save that could
// not be said is logged — the delivery's owner, which also asks for the walks it follows, finds it by
@@ -1402,3 +1257,10 @@ func recheckOneHead(member string, answer func(any) error, ask func() (string, e
}
return answer(recheckAnswer(asked, member))
}
// groupCheckAnswer is what `delivery check` of a group answers from R4e: who judges it, and that nothing was asked
// (novox/hq ADR 0503 §3; issue 1299, C2).
func groupCheckAnswer(group string) map[string]any {
return map[string]any{"asked": "", "group": group, "judged-by": checkerModule,
"said": "the checker judges a delivery group's heads merged together (ADR 0503 §3): nothing was asked"}
}
@@ -41,7 +41,7 @@ func TestSayCheckedWritesTheSameLineToItsWriter(t *testing.T) {
var w bytes.Buffer
sayChecked(context.Background(), &w, link.Checked{ID: "c1", Owner: "novox", Repo: "x", Number: 7, Commit: "0123456789abcdef",
Verdict: "pass", Summary: "fine"})
want := "c1: novox/x#7 at 01234567 checked on a machine that did not say which: gate PASS — fine; repository none\n"
want := "c1: novox/x#7 at 01234567 checked on a machine that did not say which: repository none\n"
if w.String() != want {
t.Fatalf("sayChecked said %q; want %q", w.String(), want)
}
@@ -58,8 +58,17 @@ type stalledLine struct {
// Severity is "urgent" for a group held or half merged and a refused member (novox/hq issues 1275, 1278); empty
// or anything else is a warning, as before.
Severity string `json:"severity,omitempty"`
// Needs is "new-head" for a delivery whose cure is its author's new head (novox/hq ADR 0505 §5), Author that author
// as the operator reads it ("the author of feat/x", with the session that pushed it when its pull request names one)
// and Branch its branch. A mesh-delivery from before says none, and the words are as before.
Needs string `json:"needs,omitempty"`
Author string `json:"author,omitempty"`
Branch string `json:"branch,omitempty"`
}
// needsNewHead is what a stalled line needs when its cure is its author's new head (ADR 0505 §5).
const needsNewHead = "new-head"
// severity is the condition's severity a stalled line asks: urgent when it says so, a warning otherwise.
func (l stalledLine) severity() conditions.Severity {
if l.Severity == string(conditions.Urgent) {
@@ -181,32 +181,3 @@ func TestATruncatedChangeIsReadAsChangingTheManifestsOfWhatItMoves(t *testing.T)
t.Fatalf("the member's change is read as %+v", src)
}
}
// A lone pull request's cycle fails its check, naming every module and edge, with no group and no members.
func TestALonePullRequestInACycleFailsWithNoGroup(t *testing.T) {
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
const head = "3333333333333333333333333333333333333333"
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, _, file, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil
}
if file == "modules/x/module.json" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
m := orderMember{ID: "novox/r@" + head[:12], Repository: "novox/r", Base: "main", Head: head, Number: 5,
Paths: []string{"modules/x/module.json", "modules/y/module.json"}}
c, cyclic, err := cycleVerdict(context.Background(), "", []orderMember{m}, map[string]orderReach{m.ID: {Moved: []string{"mx", "my"}}}, nil, nil)
if err != nil || !cyclic || c.Verdict != "fail" || c.Group != "" || len(c.Members) != 0 || !strings.Contains(c.Summary, "s.a") ||
!strings.Contains(c.Summary, "my") {
t.Fatalf("the verdict is %+v (cyclic %v, %v)", c, cyclic, err)
}
}
@@ -59,6 +59,18 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
}
}
// heardOf has this mesh's own view of the plans take in the plan as the bucket holds it. A verb of the
// controller (deliveryCommand, plansCommand) opens an inventory of its own and writes the plan there; this
// mesh's inventory learns of the write from the bucket's watch, a moment later, and a loaded machine makes the
// moment long enough for advanceHeld to read the plan as it was, still waiting for its word, and ask nothing
// (issue 1399). Reading the plan from the bucket puts it into the view at once, with no wait to bound.
heardOf := func(id string) {
t.Helper()
if _, err := inv.PlanByID(ctx, id); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
@@ -90,6 +102,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
heardOf(p.ID)
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
@@ -119,6 +132,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
heardOf(p.ID)
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
-2
View File
@@ -138,8 +138,6 @@ func TestTheDeliveryVerbsComposeTheirCommands(t *testing.T) {
{"delivery-order", map[string]any{"members": "[]"}, []string{"delivery", "order", "--members", "[]"}},
{"delivery-check", map[string]any{"group": "feat/x", "members": "[]"},
[]string{"delivery", "check", "--group", "feat/x", "--members", "[]"}},
{"delivery-plan", map[string]any{"repository": "novox/a", "paths": "x", "head": "c0"},
[]string{"delivery", "plan", "--repository", "novox/a", "--paths", "x", "--head", "c0"}},
{"plans", map[string]any{"go": "plan-1", "why": "down"}, []string{"plans", "go", "plan-1", "--why", "down"}},
} {
got, err := argvFor(c.verb, c.args)
-10
View File
@@ -35,16 +35,6 @@ func TestAMergeOfABaseTakesWhatStandsOnItAlong(t *testing.T) {
t.Fatalf("%s was rebuilt and stands on nothing that moved (%v)", n, names)
}
}
// The dependents come in base order when the merge orders them: the runtime, then shop, then
// the plugin that stands on shop.
ordered := orderByBases(append([]inventory.Entry{entry("mesh-tools")}, got...), against)
pos := map[string]int{}
for i, e := range ordered {
pos[e.Manifest.Module] = i
}
if !(pos["mesh-tools"] < pos["shop"] && pos["shop"] < pos["shop-plugin"]) {
t.Fatalf("not in base order: %v", ordered)
}
// Nothing moved: nothing follows.
if more := dependentsOf(nil, entries, against); len(more) != 0 {
t.Fatalf("with nothing moved, %d module(s) were rebuilt", len(more))
@@ -1,84 +0,0 @@
//go:build integration
package main
import (
"context"
"errors"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/identity"
"github.com/novox/mesh-controller/internal/link"
)
// aForgeAndABuildSeat gives a test what a real ask of a build needs: the forge holding the git seat, assigned to anchor, so
// the clone URL composes; the identity store; the build seat's queue on a real bus; and the environment of the serving controller, which is
// never the controller's terminal (a verb's, in effect: startedAtTheTerminal is false).
func aForgeAndABuildSeat(t *testing.T, open *stores) *broker.JetStream {
t.Helper()
js, _ := aBuildQueue(t)
// The ask signs what it sends with the controller's identity, as the serving controller does.
identity.ForTest(t)
forge, err := catalogue.ParseManifest([]byte(`{"module":"gitea","version":"1",
"claims":[{"name":"git","scope":"mesh"}],"provides":[{"name":"git","scope":"mesh"}],
"serves":{"git":{"scheme":"http","port":3000}}}`))
if err != nil {
t.Fatal(err)
}
register(t, open, forge)
if _, err := assign(t.Context(), open, "anchor", "gitea"); err != nil {
t.Fatal(err)
}
t.Setenv(servedVar, "1")
return js
}
// 1146/AC9, through the real ask (askABuild, buildOneAsked, verbMayAsk) and a real bus: the first build of a module a
// merge adds on the operator's Approve is queued on the build seat; the same ask without that warrant, and the warrant's
// ask for a path the merge does not hold, are refused by the guard of a build through a verb.
func TestTheApprovedFirstBuildPassesTheGuardOfAVerbAndIsQueued(t *testing.T) {
const head, commit = "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", "0177ea7a00000000"
open := aMesh(t)
ctx := t.Context()
js := aForgeAndABuildSeat(t, open)
a := aDeliveryMergeOf(t, head, map[string]string{"novox/mesh-delivery@a1a1a1a1a1a1": head})
wasKept, wasAsk := keptDeliveryMerge, routerAskOf
keptDeliveryMerge = func(context.Context, string, string, int, string) (deliveryMerge, bool, error) {
return a.merge, true, nil
}
routerAskOf = func(context.Context, catalogueReader, string, string) (*asks.Ask, *asks.Warrant, error) {
return &a.ask, &a.warrant, nil
}
t.Cleanup(func() { keptDeliveryMerge, routerAskOf = wasKept, wasAsk })
// Without the warrant the guard refuses, whatever the repository holds.
_, err := askABuild(ctx, buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "", "main")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("1146/AC9: a first build asked without the warrant was not refused: %v", err)
}
// And a path the merge does not hold is refused on the warrant.
warranted := withFirstBuildWarrant(ctx, "novox/mesh-delivery", []string{""}, "the Approve")
if _, err := askABuild(warranted, buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "other", "main"); !errors.As(err, &policy) {
t.Fatalf("1146/AC9: the warrant covered a path the merge does not hold: %v", err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-delivery", Base: "main", HeadSHA: head, Commit: commit, Number: 7,
MergedAt: time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano), Paths: []string{"module.json", "main.go"},
ModuleDirs: []string{"."}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
queue, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if len(queue.Asks) != 1 {
t.Fatalf("1146/AC9: the build queue holds %+v, want the one first build", queue)
}
}
@@ -1,29 +0,0 @@
package main
import "testing"
// 1146/AC9: the warrant the controller checked covers exactly the repository on the git seat and the module paths the
// merge holds; nothing else passes verbMayAsk on it.
func TestTheFirstBuildWarrantCoversOnlyItsRepositoryAndItsModulePaths(t *testing.T) {
ctx := withFirstBuildWarrant(t.Context(), "novox/mesh-delivery", []string{"", "view"}, "the Approve")
w := firstBuildWarrantIn(ctx)
for _, c := range []struct {
source buildSource
path string
covers bool
}{
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "", true},
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "/view/", true},
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "other", false},
{buildSource{Repository: "agent/mesh-delivery", Seat: "git"}, "", false},
{buildSource{Repository: "novox/mesh-delivery"}, "", false},
{buildSource{Repository: "novox/mesh-delivery", Seat: "elsewhere"}, "", false},
} {
if got := w.covers(c.source, c.path); got != c.covers {
t.Errorf("1146/AC9: %v at %q covered = %v, want %v", c.source, c.path, got, c.covers)
}
}
if firstBuildWarrantIn(t.Context()).covers(buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "") {
t.Error("1146/AC9: a context with no warrant covers an ask")
}
}
+77 -21
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"time"
@@ -46,7 +47,7 @@ func (f following) firstBuilds(ctx context.Context, m link.SourceMoved, entries
case len(added) == 0:
return false, nil
}
if err := askFirstBuilds(ctx, inv, m, now, added, why); err != nil {
if err := askFirstBuilds(ctx, f.open, m, now, added, why); err != nil {
return true, notNow(err)
}
return true, nil
@@ -67,27 +68,35 @@ func keepFirstMerge(ctx context.Context, inv *inventory.Inventory, m link.Source
return err
}
// askFirstBuilds keeps the merge, asks the builds, and marks the merge as no longer waiting for them.
func askFirstBuilds(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, now time.Time, added []string,
// askFirstBuilds keeps the merge as answered and registers its modules from the builder's done keys (novox/hq ADR 0501
// §3, §6): nothing asks their first builds. The merge kept is the registration's warrant (approvedMove reads it by the
// commit the indexer's trunk value names); a module whose key is not done yet is registered when the key's write wakes
// the take-in (new_modules.go).
func askFirstBuilds(ctx context.Context, open *stores, m link.SourceMoved, now time.Time, added []string,
why string) error {
inv := open.inventory
repository := m.Owner + "/" + m.Repo
if err := keepFirstMerge(ctx, inv, m, now, false); err != nil {
return err
}
fmt.Printf("%s merged into %s (%.8s) on the operator's Approve (%s): the mesh builds nothing from it yet, so the "+
"first build of %s is asked\n", repository, m.Base, m.Commit, why, strings.Join(added, ", "))
// The ask carries the warrant just checked, for this repository and these module paths alone: the guard of a build
// through a verb (verbMayAsk) lets exactly that through.
paths := make([]string, 0, len(added))
for _, dir := range added {
if dir == "." {
dir = ""
}
paths = append(paths, dir)
fmt.Printf("%s merged into %s (%.8s) on the operator's Approve (%s): the mesh builds nothing from it yet, so %s "+
"are registered from the builder's done keys, asking no build\n", repository, m.Base, m.Commit, why,
strings.Join(added, ", "))
// Answered: it no longer waits, so the word's arrival again does nothing twice.
if err := inv.AnswerMerge(ctx, repository, m.Commit, inventory.WaitingRecord, false); err != nil {
return err
}
askBuilds(withFirstBuildWarrant(ctx, repository, paths, why), inv, m, buildSource{Repository: repository, Seat: gitSeat}, added)
// Answered: it no longer waits, so the word's arrival again asks nothing twice.
return inv.AnswerMerge(ctx, repository, m.Commit, inventory.WaitingRecord, false)
modules, err := newModulesOf(ctx, repository)
switch {
case err != nil:
// Said, not failed: the merge is answered, and the look at every module when a controller starts, or the next
// write of the builder or the indexer for one of them, registers them.
fmt.Printf(" the indexer's modules of %s cannot be read now (%v): they are registered when the builder's next "+
"write of one wakes the take-in\n", repository, err)
case len(modules) > 0:
inBackground(func() { takeNewModules(context.WithoutCancel(ctx), open, modules...) })
}
return nil
}
// resumeFirstBuilds runs firstBuilds for the first merges waiting for the word mesh-delivery just announced for a pull
@@ -102,7 +111,7 @@ func (f following) resumeFirstBuilds(ctx context.Context, d deliveryMerge) error
}
for _, w := range waiting {
k := announcedOf(w)
if !k.FirstBuild || !strings.EqualFold(k.Owner+"/"+k.Repo, d.Owner+"/"+d.Repo) || k.Number != d.Number ||
if !k.FirstBuild || len(k.Held) > 0 || !strings.EqualFold(k.Owner+"/"+k.Repo, d.Owner+"/"+d.Repo) || k.Number != d.Number ||
k.HeadSHA != d.HeadSHA {
continue
}
@@ -125,7 +134,7 @@ func (f following) resumeFirstBuilds(ctx context.Context, d deliveryMerge) error
case err != nil:
return notNow(err)
case len(added) > 0:
if err := askFirstBuilds(ctx, inv, k.SourceMoved, time.Now().UTC(), added, why); err != nil {
if err := askFirstBuilds(ctx, f.open, k.SourceMoved, time.Now().UTC(), added, why); err != nil {
return notNow(err)
}
}
@@ -172,6 +181,29 @@ func firstBuildsOf(ctx context.Context, inv catalogueReader, m link.SourceMoved,
return nil, "", err
}
// heldFirstBuild says a first build was refused because its manifest uses a word the running build-agent does not read,
// and its merge is kept waiting for a build-agent that reads it (novox/hq ADR 0482 §5, issue 1173). Read from a merge kept
// before R4e, never written since: nothing asks a first build (ADR 0501 §6), and a controller that starts answers such a
// merge and registers its modules from their keys (answerHeldMerges).
type heldFirstBuild struct {
Module string `json:"module"`
Path string `json:"path"`
Word string `json:"word"`
At time.Time `json:"at"`
}
// unknownWord is how the strict reading of a manifest says a word is not one it knows: encoding/json's own words, which
// the build-agent's answer carries whole. The build's answer has no typed field for it (Failed is the builder's words).
var unknownWord = regexp.MustCompile(`json: unknown field "([^"]+)"`)
// refusedWord is the word a build's failure says the reader did not know; empty when it says none.
func refusedWord(failed string) string {
if m := unknownWord.FindStringSubmatch(failed); m != nil {
return m[1]
}
return ""
}
// probeFirstBuildsID is the probe that reads the first merges waiting for mesh-delivery's word.
const probeFirstBuildsID = "D18"
@@ -203,9 +235,10 @@ func firstBuildObservations(waiting []inventory.BatchedMerge, now time.Time) []c
Headline: fmt.Sprintf("The first build of %s is not asked: no Approve was announced", repository),
Explanation: fmt.Sprintf("The merge %.8s into %s adds modules the mesh builds nothing from yet. It registers them "+
"only when mesh-delivery announces that it merged the pull request on your Approve of that head, and no such "+
"announcement has come in %s. The merge is kept, and the builds are asked the moment the announcement is kept.",
"announcement has come in %s. The merge is kept, and the modules are registered from the builder's keys the moment "+
"the announcement is kept.",
k.Commit, repository, for_),
Resolved: "mesh-delivery announces the merge, or a hand `build` registers the modules",
Resolved: "mesh-delivery announces the merge, or the operator registers the modules at the controller's terminal",
})
}
return out
@@ -217,5 +250,28 @@ func probeFirstBuilds(ctx context.Context, d *doctor) ([]conditions.Observation,
if err != nil {
return nil, err
}
return firstBuildObservations(waiting, time.Now().UTC()), nil
now := time.Now().UTC()
return append(firstBuildObservations(waiting, now), newModuleObservations(now)...), nil
}
// answerHeldMerges answers every merge a controller of before held for a build-agent that reads its new modules'
// manifests (issue 1173): nothing asks their first builds any more, and the look at every module when this controller
// starts registers them from their keys (ADR 0501 §6, §7).
func answerHeldMerges(ctx context.Context, inv *inventory.Inventory) {
waiting, err := inv.WaitingMerges(ctx)
if err != nil {
fmt.Printf("the merges held for a newer build-agent could not be read: %v\n", err)
return
}
for _, w := range waiting {
if k := announcedOf(w); len(k.Held) > 0 {
if err := inv.AnswerMerge(ctx, w.Repository, w.Commit, inventory.WaitingRecord, false); err != nil {
fmt.Printf("the merge %.8s of %s held for a newer build-agent could not be answered: %v\n", w.Commit,
w.Repository, err)
continue
}
fmt.Printf("the merge %.8s of %s was held for a newer build-agent; its modules are registered from the "+
"builder's keys instead\n", w.Commit, w.Repository)
}
}
}
@@ -5,71 +5,48 @@ package main
import (
"context"
"encoding/json"
"slices"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1146/AC7, whole against a real bus and the store, in both orders of the two events (novox/hq ADR 0432, issue 1146):
// mesh-delivery #7 merged on the operator's Approve into a repository the mesh builds nothing from. The forge's merge,
// heard as the controller hears it (SourceMoved), and mesh-delivery's word, published on the subject the controller
// consumes with the router's record of the ask in its bucket, come in either order; each way the first build of the
// module the merge commit holds is asked exactly once, its outcome registers the module, and it is in the catalogue.
// Without the Approve in the record nothing is asked. Store-backed: skips without one.
func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegistersTheModule(t *testing.T) {
// 1146/AC7 on keys (ADR 0501 §3, §6; issue 1851, A5), whole against a real bus and the store, in both orders of the two
// events: mesh-delivery #7 merged on the operator's Approve into a repository the mesh builds nothing from. The forge's
// merge, heard as the controller hears it (SourceMoved), and mesh-delivery's word, published on the subject the
// controller consumes with the router's record of the ask in its bucket, come in either order; each way the module the
// merge commit holds is registered from the builder's done key of it, and nothing asks a build. Without the Approve in
// the record it is not registered. Store-backed: skips without one.
func TestAMergeOnTheApproveIntoANewRepositoryRegistersItsModuleFromItsKey(t *testing.T) {
const head, commit = "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", "0177ea7a00000000"
for _, c := range []struct {
what string
wordFirst bool
approved bool // the router's record holds the operator's Approve of the merged head
wantRegiter bool
what string
wordFirst bool
approved bool // the router's record holds the operator's Approve of the merged head
wantRegister bool
}{
{what: "the forge's merge before mesh-delivery's word", approved: true, wantRegiter: true},
{what: "mesh-delivery's word before the forge's merge", wordFirst: true, approved: true, wantRegiter: true},
{what: "the forge's merge before mesh-delivery's word", approved: true, wantRegister: true},
{what: "mesh-delivery's word before the forge's merge", wordFirst: true, approved: true, wantRegister: true},
{what: "a word whose record holds no Approve", wordFirst: true},
} {
t.Run(c.what, func(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
// The real ask, guard and queue included: only its answers are kept, so the build's outcome can be matched.
aForgeAndABuildSeat(t, open)
// Written by the controller's merge path on its own goroutine, read by the test: under one lock.
var mu sync.Mutex
var asks [][3]string
var ids []string
asked := func() [][3]string { mu.Lock(); defer mu.Unlock(); return slices.Clone(asks) }
realAsk := askABuild
askABuild = func(c context.Context, source buildSource, path, ref string) (string, error) {
id, err := realAsk(c, source, path, ref)
if err == nil {
mu.Lock()
asks = append(asks, [3]string{source.Repository, path, ref})
ids = append(ids, id)
mu.Unlock()
}
return id, err
}
t.Cleanup(func() { askABuild = realAsk })
noBuildAsked(t)
b := aBuilderOnTheBus(t)
js := b.js
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "operator-channel", Version: "1",
DefinesSeats: []catalogue.SeatDeclaration{{Name: broker.AsksSeat, Records: []string{"asks"}}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/operator-channel", BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
@@ -87,10 +64,14 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
a := aDeliveryMergeOf(t, head, map[string]string{"novox/mesh-delivery@a1a1a1a1a1a1": head})
if c.approved {
record, _ := json.Marshal(map[string]any{"state": "closed", "ask": a.ask, "warrant": a.warrant})
if _, err := records.Put(ctx, theDelivery+"."+a.merge.AskID, record); err != nil {
if _, err := records.Put(ctx, sdkdelivery.TellerModule+"."+a.merge.AskID, record); err != nil {
t.Fatal(err)
}
}
// The indexer's value and the builder's done key of the module the merge commit holds.
fp := fpOf('9')
manifest := b.indexed(t, "mesh-delivery-view", "1", "novox/mesh-delivery", ".", commit, fp)
b.built(t, "mesh-delivery-view", "1", fp, manifest, sdkdelivery.BuildDone)
following := following{open: open}
server := link.ConnectNats(js, nil, noListener{})
@@ -104,8 +85,6 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
done := make(chan struct{})
go func() { defer close(done); _ = server.Serve(serving) }()
t.Cleanup(func() { stop(); <-done })
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
word := func() {
t.Helper()
@@ -133,37 +112,27 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
forge()
} else {
forge()
if len(asked()) != 0 {
t.Fatalf("1146/AC7: the merge asked %v before mesh-delivery's word was kept", asked())
time.Sleep(300 * time.Millisecond)
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); known {
t.Fatal("1146/AC7: registered before mesh-delivery's word was kept")
}
word()
}
if !c.wantRegiter {
if len(asked()) != 0 {
t.Fatalf("1146/AC7: a merge whose record holds no Approve asked %v", asked())
if !c.wantRegister {
time.Sleep(time.Second)
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); known {
t.Fatal("1146/AC7: a merge whose record holds no Approve registered its module")
}
return
}
want := [3]string{"novox/mesh-delivery", "", "main"}
becomes(t, 10*time.Second, "the first build asked", func() bool { return len(asked()) >= 1 })
if got := asked(); len(got) != 1 || got[0] != want {
t.Fatalf("1146/AC7: the merge asked %v, want %v once", got, want)
}
mu.Lock()
first := ids[0]
mu.Unlock()
build := onTrunk(first, "novox/mesh-delivery", "git", "", map[string]any{"module": "mesh-delivery-view", "version": "1"})
build.Commit = commit
if _, _, err := takeIn(ctx, inv, build); err != nil {
t.Fatalf("1146/AC7: the outcome of the first build was refused: %v", err)
}
becomes(t, 10*time.Second, "the module registered", func() bool {
_, known := inventoryHolds(t, inv, "mesh-delivery-view")
return known
})
src, err := inv.SourceOf(ctx, "mesh-delivery-view")
if err != nil || src.Repository != "novox/mesh-delivery" {
t.Fatalf("1146/AC7: the module is built from %q (%v)", src.Repository, err)
}
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); !known {
t.Fatal("1146/AC7: the module is not in the catalogue")
}
})
}
}
+2 -2
View File
@@ -64,7 +64,7 @@ func registerFrom(t *testing.T, forms *sourceForms) (int64, error) {
mesh := &aMeshRegisteredFromTheURL{anEmptyMesh{asked: true,
request: inventory.BuildRequest{ID: "build-1", Repository: aRepoOnTheForge, For: "build"}}, aRepoOnTheForge}
return mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-view", inventory.Source{Repository: aRepoOnTheForge},
"build-1", "", "main")
byBuild("build-1"), "", "main")
}
// 1277/AC2: an address unread twice, then read, is waited out and the forge is asked.
@@ -112,7 +112,7 @@ func TestAKnownAddressAndAnotherHostIsStillNotOnTheForge(t *testing.T) {
const elsewhere = "http://elsewhere.example/x/y.git"
mesh := &aMeshRegisteredFromTheURL{anEmptyMesh{asked: true, request: inventory.BuildRequest{ID: "build-1", For: "build"}}, elsewhere}
_, err := mayRegisterIn(t.Context(), mesh, sourceFormsRead(context.Background(), time.Now, read), "mesh-delivery-view",
inventory.Source{Repository: elsewhere}, "build-1", "", "main")
inventory.Source{Repository: elsewhere}, byBuild("build-1"), "", "main")
if !errors.Is(err, errNotItsSource) || !strings.Contains(err.Error(), "is not on the mesh's forge") {
t.Fatalf("1277/AC4: the refusal changed: %v", err)
}
+1 -1
View File
@@ -68,7 +68,7 @@ func TestTheForgeNotAnsweringInTimeIsNotAskedAgain(t *testing.T) {
// An ask that hangs past the bound is cut off at the bound, not at its own wait, and the failure still says it
// waited: the receive loop is held no longer than the bound (novox/hq issue 486).
func TestAnAskHangingPastTheBoundIsCutOffAtIt(t *testing.T) {
w := waitOut{bound: 200 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
w := waitOut{bound: 100 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
now: time.Now, sleep: sleepOrDone}
asked := 0
began := time.Now()
+12
View File
@@ -0,0 +1,12 @@
package main
import "testing"
// C2 (ADR 0503 §3; issue 1299): a delivery group's composed check is answered as the checker's, and nothing is asked
// of build-agent: the command answers before any ask is composed.
func TestTheGroupCheckNamesTheCheckerAndAsksNothing(t *testing.T) {
a := groupCheckAnswer("feat/x")
if a["judged-by"] != "mesh-delivery-checker" || a["asked"] != "" || a["group"] != "feat/x" {
t.Fatalf("the group check answered %v", a)
}
}
+198
View File
@@ -0,0 +1,198 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A move held for a person is asked of the person (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
//
// **A wait only a person can end is said to that person at once.** ADR 0377 holds a recorded module's move that
// recreates a container or touches its data until a person pushes the machine. The walk said so in its note and
// ended "done", `status` named the machine behind, and nobody was asked: on 2026-10-21 mailu's new build waited on
// novox for 40 minutes behind a plan that read "done, 3 tier(s)", and on 2026-10-19 mssql waited on ace the same way.
// Row S24 raises one condition per machine that has such a move, for the operator, with the answer **Release**: the
// push of that machine, which the controller's own `push` verb performs on the operator's warrant (ADR 0259). It ends
// when the machine no longer waits, whoever pushed it.
//
// A hold a person chose is not asked again — a module they set to `record` with a why, and the bus, whose step has
// its own row (S17).
// kindMovesHeld is S24's kind: machine.<node>.moves-held.
const kindMovesHeld = "moves-held"
// heldFacts is one machine and the moves of recorded modules that wait there for a person.
type heldFacts struct {
node string
moves []heldMove
}
// gatherHeldMoves is, per machine that holds at least one move no person chose to hold, all its held moves, by machine.
func gatherHeldMoves(ctx context.Context, inv *inventory.Inventory) ([]heldFacts, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("the machines cannot be read: %w", err)
}
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
}
var out []heldFacts
for _, n := range nodes {
moves, err := heldMovesOn(ctx, inv, f, n.Name)
if err != nil {
return nil, err
}
h := heldFacts{node: n.Name, moves: moves}
if len(h.asked()) > 0 {
out = append(out, h)
}
}
return out, nil
}
// asked are the moves no person chose to hold, and chosen the ones a person did (a module set to record with a why, the bus).
func (h heldFacts) asked() []heldMove { return h.byChoice(false) }
func (h heldFacts) chosen() []heldMove { return h.byChoice(true) }
func (h heldFacts) byChoice(chosen bool) []heldMove {
var out []heldMove
for _, m := range h.moves {
if m.Chosen == chosen {
out = append(out, m)
}
}
return out
}
// modules are the held moves' modules, in order, of all of them.
func (h heldFacts) modules() []string { return moduleNames(h.moves) }
func moduleNames(moves []heldMove) []string {
out := make([]string, 0, len(moves))
for _, m := range moves {
out = append(out, m.Module)
}
return out
}
// releaseAction is the one answer a held move offers: the push of the machine, approved by the operator. The push
// names every module whose move waits there as one that may move its data: the operator's Approve is the word
// ADR 0217 asks for, and the ask says so. Offered only where no person's own hold is on the machine too: a push
// composes every recorded module at its current build, so it would send that move as well, which the ask never named
// (ADR 0259 §6), and hold the whole machine if that module's data would move.
func (h heldFacts) releaseAction() conditions.Action {
return conditions.Action{Label: "Release", Verb: "mesh-controller.push", Level: conditions.LevelApprove,
Arguments: map[string]string{"node": h.node, "move": strings.Join(moduleNames(h.asked()), ","), "why": "",
"cause": conditions.CauseOperatorAnswer}}
}
// heldWords are the plain words of a machine's held moves: which modules wait, why in general, and what an Approve does.
func heldWords(h heldFacts) (headline, explanation, needs string, resolved string) {
names := namesWords(moduleNames(h.asked()), 3)
headline = clipWords(conditions.Capital(names)+" waits for you on "+h.node, conditions.HeadlineMax)
recreates, data := false, false
for _, m := range h.asked() {
switch {
case strings.Contains(m.Why, "recreates"):
recreates = true
default:
data = true
}
}
why := "would change what it runs"
switch {
case recreates && data:
why = "would restart a container or touch its data"
case recreates:
why = "would restart one of its containers"
case data:
why = "would touch its files or data"
}
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. Release sends %s everything it is owed, these among it, and lets their data move where it "+
"has to. Nothing is sent until you answer.", names, why, h.node, h.node), conditions.ExplanationMax)
needs = "release them " + FromMeshMCPServer
if chosen := h.chosen(); len(chosen) > 0 {
// A push of the machine would send the move a person chose to hold as well: no answer is offered here.
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. %s is also held there by your own choice, and a push of %s would send that too, so "+
"nothing is offered here: the push is yours at the controller's terminal.",
names, why, h.node, conditions.Capital(namesWords(moduleNames(chosen), 3)), h.node), conditions.ExplanationMax)
needs = fmt.Sprintf("push %s at the controller's terminal, once you have decided about %s.", h.node,
namesWords(moduleNames(chosen), 3))
}
resolved = "Resolved: " + names + " is no longer waiting on " + h.node
return
}
// watchMovesHeld is S24: a machine that has moves of recorded modules waiting for a person, said to the operator with the
// answer that sends them. The condition ends when the machine no longer waits: it is no longer among the facts.
func watchMovesHeld(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, h := range f.held {
var said []string
for _, m := range h.moves {
said = append(said, fmt.Sprintf("%s %s → %s %s", m.Module, short(m.From), short(m.To), m.Why))
}
headline, explanation, needs, resolved := heldWords(h)
if c := h.chosen(); len(c) > 0 {
said = append(said, "held by a person's own choice: "+strings.Join(moduleNames(c), ", "))
}
var actions []conditions.Action
if len(h.chosen()) == 0 {
actions = []conditions.Action{h.releaseAction()}
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: h.node, Kind: kindMovesHeld,
Machine: h.node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s has %d move(s) of recorded modules waiting for a person's push: %s — `push %s` sends them",
h.node, len(h.moves), strings.Join(h.modules(), ", "), h.node),
Said: strings.Join(said, "; "),
Headline: headline, Explanation: explanation, Needs: needs, Resolved: resolved,
Actions: actions})
}
return out
}
// planHeld is, for a plan that ended or is running, the moves its modules carried that a machine still holds for a
// person: "module on machine", sorted. The plan's own record says the move waited when it was judged
// (PlanModule.Recorded); the machines say whether it still does, and for the build this plan carried, so a plan older
// than the build now held is not named.
func planHeld(p inventory.Plan, held []heldFacts) []string {
var out []string
for _, h := range held {
for _, m := range h.moves {
s := p.Modules[m.Module]
if s == nil || !strings.HasPrefix(s.Recorded[h.node], "waits:") || !sameCommit(s.Commit, m.To) {
continue
}
out = append(out, m.Module+" on "+h.node)
}
}
sort.Strings(out)
return out
}
// heldSaid is what a plan's line adds when it carries a move that a machine holds for a person (novox/hq issue 1732): a
// plan whose module waits for a push is not plainly "done".
func heldSaid(p inventory.Plan, held []heldFacts) string {
waiting := planHeld(p, held)
if len(waiting) == 0 {
return ""
}
return fmt.Sprintf(" — %d held for a person: %s (the operator is asked)", len(waiting), strings.Join(waiting, ", "))
}
// heldNow reads the machines' held moves for a plan's line, nil when they cannot be read: the line then reads as before.
func heldNow(ctx context.Context, inv *inventory.Inventory) []heldFacts {
held, err := gatherHeldMoves(ctx, inv)
if err != nil {
return nil // empty-on-error: a plan's line then reads as it did before; the watchdog (S24) says it cannot read
}
return held
}
@@ -0,0 +1,92 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func inventoryUpgrade(why string) inventory.Upgrade { return inventory.Upgrade{Why: why, By: "jochen"} }
func mustMoveFacts(t *testing.T, rm *recordedMesh) moveFacts {
t.Helper()
f, err := readMoveFacts(t.Context(), rm.open.inventory)
if err != nil {
t.Fatal(err)
}
return f
}
// 1732/AC8 (integration): a mesh running mailu at c1 on two machines, whose registered c2 recreates a container, is read
// as two machines each holding mailu's move for a person; a bundle-only c3 on top changes nothing of it; after a machine
// is sent c2 (as a push records it) it is no longer among them, and plans that carried c2 say nothing more of it.
func TestMachinesHoldingAMoveForAPersonAreReadFromTheRecords(t *testing.T) {
rm := aRecordedMesh(t, aMailu(true))
c2 := aMailu(true)
resourceOf(&c2, "imap")["image"] = "registry.invalid/imap@sha256:2"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
ctx := t.Context()
held, err := gatherHeldMoves(ctx, rm.open.inventory)
if err != nil {
t.Fatal(err)
}
if len(held) != 2 || held[0].node != "anchor" || held[1].node != "laptop" || len(held[0].moves) != 1 ||
held[0].moves[0].Module != "mailu" || held[0].moves[0].To != "c2" || !strings.Contains(held[0].moves[0].Why, "recreates") {
t.Fatalf("held %+v", held)
}
if err := rm.open.inventory.RecordSent(ctx, nodeID(t, rm.open, "anchor"), "d-anchor-c2", map[string]string{"mailu": "c2"}); err != nil {
t.Fatal(err)
}
held, err = gatherHeldMoves(ctx, rm.open.inventory)
if err != nil || len(held) != 1 || held[0].node != "laptop" {
t.Fatalf("after anchor was sent c2: %+v (%v)", held, err)
}
}
// 1732/AC9 (integration): a module a person set to record is no ask: they chose the hold.
func TestAHoldAPersonChoseIsNotAsked(t *testing.T) {
rm := aRecordedMesh(t, aMailu(false))
c2 := aMailu(false)
c2.Bundles[0].Digest = "sha256:bbb"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
if err := rm.open.inventory.SetUpgradeOf(t.Context(), "mailu", inventoryUpgrade("the migration week")); err != nil {
t.Fatal(err)
}
held, err := gatherHeldMoves(t.Context(), rm.open.inventory)
if err != nil || len(held) != 0 {
t.Fatalf("a machine holding only a person's own hold is asked: %+v (%v)", held, err)
}
if moves, _ := heldMovesOn(t.Context(), rm.open.inventory, mustMoveFacts(t, rm), "anchor"); len(moves) != 1 || !moves[0].Chosen {
t.Fatalf("status still names it, marked as chosen: %+v", moves)
}
}
// 1742/AC4 (integration): a registered build naming an image of the store that the store lacks is read as a missing
// image on each machine that runs the module; one the store holds is not; and what the store holds is not asked twice.
func TestARegisteredBuildNamingAnImageTheStoreLacksIsRead(t *testing.T) {
m := aMailu(true)
resourceOf(&m, "imap")["image"] = "artifact-store://mailu/imap@sha256:" + strings.Repeat("e", 64)
rm := aRecordedMesh(t, m)
asked := 0
lacking := func(_ context.Context, reference string) (bool, error) { asked++; return false, nil }
imagesHeld.Lock()
imagesHeld.held = map[string]bool{}
imagesHeld.Unlock()
got, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking)
if err != nil || len(got) != 2 || got[0].Module != "mailu" || got[0].Node != "anchor" || got[1].Node != "laptop" {
t.Fatalf("missing %+v (%v)", got, err)
}
holding := func(_ context.Context, reference string) (bool, error) { asked++; return true, nil }
if got, err := gatherMissingImages(t.Context(), rm.open.inventory, holding); err != nil || len(got) != 0 {
t.Fatalf("an image the store holds is missing: %+v (%v)", got, err)
}
before := asked
if _, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking); err != nil || asked != before {
t.Fatalf("an image the store was found to hold was asked about again (%d, %v)", asked-before, err)
}
}
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A move held for a person is asked of the person** (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
// aHeldMailu is novox holding mailu's new build because it would recreate a container: the walk's own words.
func aHeldMailu() heldFacts {
return heldFacts{node: "novox", moves: []heldMove{{Module: "mailu", From: "b2f5ebb2aaaa", To: "3259765dbbbb",
Why: "waits: recreates 1 of mailu's 12 container(s) (with a new image: automx)"}}}
}
func heldConditionOn(f *signalFacts) conditions.Condition {
o := watchMovesHeld(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1732/AC1: a machine holding a move of a recorded module for a person raises one condition for the operator, with
// Release as its only answer: the machine's push, approved, naming the modules whose moves wait there.
func TestAHeldMoveIsRaisedWithReleaseAsItsOnlyAction(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
got := watchMovesHeld(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "moves-held" || o.Scope != conditions.ScopeMachine || o.ID != "novox" || o.Severity != conditions.Warning ||
o.Resolver != conditions.ResolverOperator || o.Key() != "machine.novox.moves-held" {
t.Fatalf("kind, scope, subject, severity or resolver: %+v (%s)", o, o.Key())
}
if len(o.Actions) != 1 {
t.Fatalf("actions: %+v", o.Actions)
}
a := o.Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.push" || a.Level != conditions.LevelApprove ||
a.Arguments["node"] != "novox" || a.Arguments["move"] != "mailu" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action is not the push of novox: %+v", a)
}
if _, takes := a.Arguments["why"]; !takes {
t.Fatal("the push takes a why, which the asker fills in with the operator's answer")
}
for _, want := range []string{"mailu", "novox"} {
if !strings.Contains(o.Headline+o.Explanation, want) {
t.Errorf("the words do not name %q: %q / %q", want, o.Headline, o.Explanation)
}
}
if !strings.Contains(o.Said, "recreates 1 of mailu's 12") {
t.Errorf("the evidence does not carry the walk's own words: %q", o.Said)
}
if w := (conditions.Observation{Headline: o.Headline, Explanation: o.Explanation, Needs: o.Needs, Resolved: o.Resolved}); w.Headline == "" || len(o.Headline) > conditions.HeadlineMax {
t.Errorf("headline: %q", o.Headline)
}
}
// 1732/AC2: a machine with nothing held raises nothing, so the condition ends the look after the move is sent.
func TestAMachineHoldingNothingRaisesNoHeldMoveCondition(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
if got := watchMovesHeld(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
}
// 1732/AC3: the condition is asked on a channel at the approve level in words that say what Release does, and the
// operator's warrant calls the controller's own push with the machine and the modules, once.
func TestAHeldMovesReleaseIsAskedAndAWarrantPushesTheMachine(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
c := heldConditionOn(f)
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Release" || o.Level != asks.Approve || o.Does != "the held builds are sent to novox" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.push@ ") ||
!strings.Contains(r.called[0], `"node":"novox"`) || !strings.Contains(r.called[0], `"move":"mailu"`) ||
!strings.Contains(r.called[0], `"cause":"`+conditions.CauseOperatorAnswer+`"`) ||
!strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC4: the controller may call its own push and upgrade verbs on a warrant: the grant names them.
func TestTheControllerMayActOnAWarrantForPushAndUpgrade(t *testing.T) {
for _, verb := range []string{"push", "upgrade"} {
granted := false
for _, v := range broker.VerbsTheControllerActsOnAWarrant {
granted = granted || (v.Seat == broker.ControllerSeat && v.Verb == verb)
}
if !granted {
t.Errorf("the controller may not act on a warrant for %s", verb)
}
}
}
func aDoneMailuPlan() inventory.Plan {
return inventory.Plan{ID: "plan-1792585970446304628", Repository: "novox/mesh-catalog", State: inventory.PlanDone,
Tiers: [][]string{{"a"}, {"b"}, {"mailu"}},
Modules: map[string]*inventory.PlanModule{"mailu": {State: "built", Commit: "3259765dbbbbcccc",
Recorded: map[string]string{"novox": "waits: recreates 1 of mailu's 12 container(s)"}}}}
}
// 1732/AC5: a done plan that carries a move a machine still holds for a person says so on its line; once the machine
// no longer holds it, or holds another build of the module, the line says nothing more.
func TestADonePlanSaysWhatItHoldsForAPerson(t *testing.T) {
p := aDoneMailuPlan()
line := planLine(p, time.Date(2026, 10, 21, 16, 0, 0, 0, time.UTC)) + heldSaid(p, []heldFacts{aHeldMailu()})
if !strings.Contains(line, "done, 3 tier(s)") || !strings.Contains(line, "1 held for a person: mailu on novox") {
t.Errorf("the line: %q", line)
}
if said := heldSaid(p, nil); said != "" {
t.Errorf("a plan whose moves were sent says %q", said)
}
other := aHeldMailu()
other.moves[0].To = "9999999ffff"
if said := heldSaid(p, []heldFacts{other}); said != "" {
t.Errorf("a plan older than the build now held says %q", said)
}
p.Modules["mailu"].Recorded = map[string]string{"novox": recordedRollsWhy}
if said := heldSaid(p, []heldFacts{aHeldMailu()}); said != "" {
t.Errorf("a move the plan sent says %q", said)
}
}
// 1732/AC6: the builds held after a failed release are asked of the operator: the held-release condition offers
// Release, which calls the controller's own `upgrade` with release-backlog and nothing else, once, on a warrant.
func TestTheBuildsHeldAfterAFailedReleaseAreAskedAndReleasedOnAWarrant(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
backlogNow.held = "release-1792595552499982311 failed (failed on novox); what waits is released again by a person"
backlogNow.waiting = map[string][]inventory.CarriedMove{"novox": {{Module: "indexer"}, {Module: "nextcloud"}}}
got := backlogObservation()
if len(got) != 1 || len(got[0].Actions) != 1 {
t.Fatalf("raised %+v", got)
}
a := got[0].Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.upgrade" || a.Level != conditions.LevelApprove ||
a.Arguments["release-backlog"] != "true" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action: %+v", a)
}
o := got[0]
c := conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 || sent[0].Options[0].Does != "the builds held after the failed delivery are released" {
t.Fatalf("asked %+v", sent)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.upgrade@ ") ||
!strings.Contains(r.called[0], `"release-backlog":"true"`) || !strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC7: a plan whose send waits for a gate while the builds are held after a failed release reads as waiting for
// a person to release them, not "building ... LATE"; with nothing held it still reads as the wait it is.
func TestAPlanWaitingForAGateBehindAFailedReleaseIsNotLate(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
now := time.Date(2026, 10, 21, 18, 50, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Repository: "novox/mesh-sdk", Commit: "c0ffee00", State: inventory.PlanBuilding,
Tiers: [][]string{{"indexer"}}, TierEntered: now.Add(-100 * time.Minute), Created: now.Add(-110 * time.Minute),
Modules: map[string]*inventory.PlanModule{"indexer": {State: "built"}},
Note: refusalNote(0, errors.New("a build waits there for its gate: novox"))}
backlogNow.held = ""
if line := planLine(p, now); !strings.Contains(line, "a build waits there for its gate: novox") {
t.Errorf("the wait is not said: %q", line)
}
backlogNow.held = "release-1 failed (failed on novox); what waits is released again by a person"
line := planLine(p, now)
if strings.Contains(line, "LATE") || strings.Contains(line, "building") ||
!strings.Contains(line, "until a person releases it") {
t.Errorf("a plan held behind a failed release reads %q", line)
}
}
// 1732/AC10: a machine that also holds a move a person chose to hold gets no Release, since a push would send that move
// too; the condition names both and says the push is the controller's terminal. A machine with only asked moves keeps it.
func TestAMachineAlsoHoldingAPersonsOwnHoldIsOfferedNoRelease(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
h := aHeldMailu()
h.moves = append(h.moves, heldMove{Module: "nftables", From: "n1aaaa", To: "n2bbbb",
Why: "waits: a person set it to record, and holds every move of it until they push", Chosen: true})
f.held = []heldFacts{h}
got := watchMovesHeld(f)
if len(got) != 1 || len(got[0].Actions) != 0 {
t.Fatalf("%+v", got)
}
o := got[0]
text := o.Headline + o.Explanation + o.Needs + o.Summary + o.Said
for _, want := range []string{"mailu", "nftables", "terminal", "push novox"} {
if !strings.Contains(text, want) {
t.Errorf("does not say %q: %q", want, text)
}
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
f.held = []heldFacts{aHeldMailu()}
if got := watchMovesHeld(f); len(got) != 1 || len(got[0].Actions) != 1 || got[0].Actions[0].Arguments["move"] != "mailu" {
t.Fatalf("a machine with only asked moves: %+v", got)
}
}
+245
View File
@@ -0,0 +1,245 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// No declaration is sent that names an image the artifact store does not hold (novox/hq issue 1742).
//
// On 2026-10-21 at 17:13 a send to novox named `mailu/automx@sha256:c62150b8…`; the build record keeps that digest
// ("recent-build") and the store did not hold it. The node-engine removed the running container, then could not pull
// the replacement, and every later apply on novox failed for the next hour while the mail clients' autoconfiguration
// had no backend. The node-engine's half (pull first, replace after) is its own change; this is the controller's: **a
// module whose image the store lacks is left out of the declaration**, as a module whose setting nobody gave is
// (ADR 0163, rule 6) — the node-engine keeps what it has of it and touches none of its containers — and **it is said**,
// row S25: a condition naming the module, the machine and the image, until the store holds the image again (a new build
// of the module, or the copy put back).
//
// Judged only for an image the store's own address names, from the send's own resources; an image of another registry
// is no business of the store's. A store that cannot be asked judges nothing: a send is not refused for a question
// that did not answer.
// kindImageMissing is S25's kind: module.<module>.<machine>.image-missing.
const kindImageMissing = "image-missing"
// missingImage is a module held out of a machine's declaration because the store lacks an image it names.
type missingImage struct {
Node, Module, Image string
}
// storeHolds is whether the store holds the manifest an image reference at its address names.
type storeHolds func(ctx context.Context, reference string) (bool, error)
// imageInStore is the artifact-store reference of an image the store's address names, "" for any other.
func imageInStore(image, address string) string {
if address == "" || !strings.HasPrefix(image, address+"/") || !strings.Contains(image, "@sha256:") {
return ""
}
reference := catalogue.Recorded(image)
if _, ours := catalogue.InArtifactStore(reference); !ours {
return ""
}
return reference
}
// holdBackMissingImages leaves out of the declaration every module that names an image the store does not hold,
// and answers what it left out. The module's resources go, it joins LeftOut with the reason, and its build stays the
// one the machine was last sent (`was`).
func holdBackMissingImages(ctx context.Context, node string, s *sendable, address string, holds storeHolds,
was map[string]string) []missingImage {
lacks := map[string]bool{} // reference → the store does not hold it; absent: not asked, or no answer
asked := map[string]bool{}
lacking := map[string]string{} // module → the image
for _, r := range s.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
reference := imageInStore(image, address)
module := s.moduleOf[fmt.Sprint(r["id"])]
if reference == "" || module == "" || lacking[module] != "" {
continue
}
if !asked[reference] {
asked[reference] = true
held, err := holds(ctx, reference)
// empty-on-error: the store did not answer, so nothing is held back for a question unanswered
lacks[reference] = err == nil && !held
}
if lacks[reference] {
lacking[module] = image
}
}
if len(lacking) == 0 {
return nil
}
kept := s.Resources[:0:0]
for _, r := range s.Resources {
if _, out := lacking[s.moduleOf[fmt.Sprint(r["id"])]]; !out {
kept = append(kept, r)
}
}
s.Resources = kept
if s.leftOutWhy == nil {
s.leftOutWhy = map[string]string{}
}
var found []missingImage
for module, image := range lacking {
s.leftOutWhy[module] = "its image " + image + " is not in the artifact store"
if !slices.Contains(s.LeftOut, module) {
s.LeftOut = append(s.LeftOut, module)
}
if s.Builds != nil {
if prior, sent := was[module]; sent {
s.Builds[module] = prior
} else {
delete(s.Builds, module)
}
}
found = append(found, missingImage{Node: node, Module: module, Image: image})
}
sort.Strings(s.LeftOut)
sort.Slice(found, func(i, j int) bool { return found[i].Module < found[j].Module })
return found
}
// withoutMissingImages is composeSend's last step: the store asked about the images the declaration names, the modules
// whose image it lacks left out and kept in the ledger S25 reads. A store that cannot be found or asked leaves the
// declaration as composed.
func withoutMissingImages(ctx context.Context, open *stores, node string, s sendable) sendable {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return s
}
address, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil || address == "" {
return s
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
sent = nil
}
store := artifacts.Store{Address: address}
found := holdBackMissingImages(ctx, node, &s, address, store.HoldsManifest, sent)
for _, m := range found {
fmt.Printf("%s: %s is left out of the declaration: its image %s is not in the artifact store\n", node, m.Module, m.Image)
}
return s
}
// imageFacts are the images S25 raises: a module whose registered build names an image the store lacks, on each machine
// that runs it.
type imageFacts struct {
missingImage
}
// imagesHeld remembers the images the store was found to hold: a stored image is named by its digest and stays what it
// is, so it is asked about once; one it lacks is asked again at every look.
var imagesHeld = struct {
sync.Mutex
held map[string]bool
}{held: map[string]bool{}}
// gatherMissingImages is every machine's module whose registered build — the one a send would carry — names an image of
// the artifact store that the store does not hold. Read from the catalogue and the store, never from what a send did:
// a push is a command of its own, and the serving controller's watchdog hears nothing of it.
func gatherMissingImages(ctx context.Context, inv *inventory.Inventory, holds storeHolds) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
var out []imageFacts
for _, module := range sortedKeysOf(modulesOf(shelf)) {
for _, r := range shelf[module].Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
if _, ours := catalogue.InArtifactStore(image); !ours || !strings.Contains(image, "@sha256:") {
continue
}
imagesHeld.Lock()
known := imagesHeld.held[image]
imagesHeld.Unlock()
if !known {
held, err := holds(ctx, image)
if err != nil {
return nil, fmt.Errorf("the artifact store cannot be asked about %s: %w", image, err)
}
if held {
imagesHeld.Lock()
imagesHeld.held[image] = true
imagesHeld.Unlock()
continue
}
} else {
continue
}
running, err := inv.Running(ctx, module)
if err != nil {
return nil, err
}
for _, node := range running {
out = append(out, imageFacts{missingImage{Node: node, Module: module, Image: image}})
}
}
}
return out, nil
}
// modulesOf is the shelf's modules as a map of names, for sortedKeysOf.
func modulesOf(shelf map[string]catalogue.Manifest) map[string]string {
out := make(map[string]string, len(shelf))
for m := range shelf {
out[m] = m
}
return out
}
// watchMissingImages is S25: a module left out of a machine's declaration because the store lacks its image, said at
// once. The condition ends when the store holds the image, or a send names another.
func watchMissingImages(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.images {
out = append(out, conditions.Observation{Scope: conditions.ScopeModule, ID: m.Module + "." + m.Node, Kind: kindImageMissing,
Machine: m.Node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s is left out of every send to %s: its declaration names an image that the artifact store does "+
"not hold (the evidence names it); the running container is untouched", m.Module, m.Node),
Said: "image " + m.Image + " is not in the store",
Headline: clipWords(conditions.Capital(m.Module)+" is not updated on "+m.Node, conditions.HeadlineMax),
Explanation: clipWords(fmt.Sprintf("The new version of %s names a container image that the mesh's store does not "+
"hold, so it was not sent to %s. What runs there is untouched, and the rest of %s is sent as usual.",
m.Module, m.Node, m.Node), conditions.ExplanationMax),
Needs: fmt.Sprintf("build %s again; the details name the image.", m.Module),
Resolved: "Resolved: the store holds the image of " + m.Module + " again"})
}
return out
}
// gatherMissingImagesHere is gatherMissingImages asking the mesh's own artifact store; nothing to ask where the mesh has
// none on its network yet.
func gatherMissingImagesHere(ctx context.Context, inv *inventory.Inventory) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, nil
}
return gatherMissingImages(ctx, inv, artifacts.Store{Address: address}.HoldsManifest)
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
)
// **No declaration is sent that names an image the artifact store does not hold** (novox/hq issue 1742).
const storeAt = "novox.internal:5100"
// automx is the image the send of 2026-10-21 17:13 named and the store lacked; the others are held.
const (
automxImage = storeAt + "/mailu/automx@sha256:c62150b8e2cbcc18cb911b098c2877f928761a1e6251667f39ea18251ea35591"
imapImage = storeAt + "/mailu/imap@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
)
func aMailSend() sendable {
return sendable{
Resources: []map[string]any{
{"id": "mailu.automx", "type": "container", "image": automxImage},
{"id": "mailu.automx-config", "type": "file", "path": "/etc/automx.conf"},
{"id": "mailu.imap", "type": "container", "image": imapImage},
{"id": "other.web", "type": "container", "image": "docker.io/library/nginx@sha256:" + strings.Repeat("b", 64)},
{"id": "dns.dnsmasq", "type": "container", "image": storeAt + "/dnsmasq/main@sha256:" + strings.Repeat("c", 64)},
},
moduleOf: map[string]string{"mailu.automx": "automx", "mailu.automx-config": "automx", "mailu.imap": "mailu",
"other.web": "other", "dns.dnsmasq": "dnsmasq"},
Builds: map[string]string{"automx": "b2f5ebb2new", "mailu": "m2", "other": "o1", "dnsmasq": "d1"},
}
}
func storeHolding(missing ...string) (storeHolds, *[]string) {
var asked []string
return func(_ context.Context, reference string) (bool, error) {
asked = append(asked, reference)
for _, m := range missing {
if strings.Contains(reference, m) {
return false, nil
}
}
return true, nil
}, &asked
}
// 1742/AC1: given a send naming automx's image, which the store does not hold, when the declaration is judged, then
// every resource of automx is out of it, automx is among the modules left out with the reason, its build is the one the
// machine was last sent, and every other module — an image of another registry among them — is as composed.
func TestAModuleWhoseImageTheStoreLacksIsLeftOutOfTheSend(t *testing.T) {
s := aMailSend()
holds, asked := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, map[string]string{"automx": "b2f5ebb2old"})
if len(found) != 1 || found[0].Module != "automx" || found[0].Node != "novox" || found[0].Image != automxImage {
t.Fatalf("found %+v", found)
}
var ids []string
for _, r := range s.Resources {
ids = append(ids, r["id"].(string))
}
if got := strings.Join(ids, ","); got != "mailu.imap,other.web,dns.dnsmasq" {
t.Errorf("the declaration keeps %s", got)
}
if len(s.LeftOut) != 1 || s.LeftOut[0] != "automx" || !strings.Contains(s.leftOutWhy["automx"], "not in the artifact store") {
t.Errorf("left out %v because %v", s.LeftOut, s.leftOutWhy)
}
if s.Builds["automx"] != "b2f5ebb2old" || s.Builds["mailu"] != "m2" || s.Builds["other"] != "o1" {
t.Errorf("builds %v", s.Builds)
}
for _, a := range *asked {
if strings.Contains(a, "docker.io") {
t.Errorf("the store was asked about an image of another registry: %s", a)
}
}
}
// 1742/AC2: given a module never sent to the machine, when its image is missing, then it has no build in the record of the
// send; given the store does not answer, then nothing is held back for a question unanswered.
func TestAnUnansweredStoreHoldsNothingBackAndANewModuleHasNoBuild(t *testing.T) {
s := aMailSend()
holds, _ := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, nil)
if len(found) != 1 {
t.Fatalf("%+v", found)
}
if _, has := s.Builds["automx"]; has {
t.Errorf("a module never sent is recorded as sent: %v", s.Builds)
}
s = aMailSend()
silent := func(context.Context, string) (bool, error) { return false, errors.New("registry: connection refused") }
if found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, silent, nil); len(found) != 0 || len(s.LeftOut) != 0 ||
len(s.Resources) != 5 {
t.Errorf("an unanswered store held back %+v, left out %v", found, s.LeftOut)
}
}
// 1742/AC3: a module left out for its image raises a condition for the operator naming the module, the machine and the
// image in the evidence, plain words, and no answer to give; a mesh whose images are all held raises nothing.
func TestAMissingImageIsRaisedForTheOperator(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 40, 0, 0, time.UTC))
if got := watchMissingImages(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
f.images = []imageFacts{{missingImage{Node: "novox", Module: "automx", Image: automxImage}}}
got := watchMissingImages(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "image-missing" || o.Key() != "module.automx.novox.image-missing" || o.Resolver != conditions.ResolverOperator ||
o.Severity != conditions.Warning || o.Machine != "novox" {
t.Fatalf("%+v (%s)", o, o.Key())
}
if !strings.Contains(o.Said, "c62150b8") || !strings.Contains(o.Headline+o.Explanation, "automx") {
t.Errorf("words: %q / %q / %q", o.Headline, o.Explanation, o.Said)
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
}
+1 -1
View File
@@ -35,7 +35,7 @@ func TestAMergeSaysTheModulesWhoseBuildSourceItLeftUnchanged(t *testing.T) {
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "examples/route-proxy/"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "modules/route-proxy/cmd/route-proxy/"}},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
}
+460
View File
@@ -0,0 +1,460 @@
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"path"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/buildreads"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The take-in from the builder's done key (novox/hq ADR 0501, ADR 0498 §8, ADR 0453 §5; issue 1851).
//
// From R4e the controller asks no build but the builder's own module's: a module of a merge is taken from the builder's
// key for the fingerprint the indexer's trunk value of the module names (ADR 0501 §2). Only a done key whose every
// artifact names a reference is taken (ADR 0498 §8); a key not done yet, one waiting for a reference, a templates
// archive the store does not answer, or a list of predefined keys the forge does not answer is waited for and said,
// never failed and never delivered from. A failed key fails, in the builder's own words. The take-in then records the
// key as a build of today's record layout, so a controller of before reads it on a rollback, and judges it as every
// build is judged (registerJudged).
// errKeyWaits says a take-in from a key waits: for the builder, a reference, the store or the forge. Never a failure.
var errKeyWaits = errors.New("waits")
// errKeyFailed says the module's key failed: the builder's own words follow.
var errKeyFailed = errors.New("the builder's key failed")
// keyReads is what a take-in from a key reads of the builder and the indexer (builderkeys.Reader, or a test's).
type keyReads interface {
Build(ctx context.Context, module, fp string) (builderkeys.Key, error)
Manifest(ctx context.Context, digest string) (sdkdelivery.Manifest, error)
}
// keyBuildID is the id a key's build is recorded under: the key and its revision, so the same revision recorded twice
// is one record, and a rebuild of the same fingerprint (a new revision, ADR 0429 §18) another.
func keyBuildID(k builderkeys.Key) string {
part, err := sdkdelivery.FingerprintKeyPart(k.Fingerprint)
if err != nil {
part = strings.TrimPrefix(k.Fingerprint, sdkdelivery.SourcePrefix2)
}
return fmt.Sprintf("key-%s-%d", part, k.Revision)
}
// readArchive is the archive a key's reference names, read from the artifact store and checked against its digest. A
// variable so a test needs no store.
var readArchive = func(ctx context.Context, inv *inventory.Inventory, reference string) ([]byte, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is on no network the controller reaches")
}
return artifacts.Store{Address: address}.GetBlob(ctx, reference)
}
// predefinedHead is the list of predefined keys at the head of the catalogue repository's trunk, and that head.
type predefinedHead struct {
List catalogue.PredefinedList
Repository string
Commit string
// Generation orders two heads of the trunk: the head's commit time in seconds. A list kept from a build's
// generation (its commit count) is always older than a head read here.
Generation int
}
// predefinedAtTrunk is the list of predefined keys at the head of the catalogue repository's trunk (ADR 0501 §8), read
// from the forge. A variable so a test needs no forge.
var predefinedAtTrunk = func(ctx context.Context, inv *inventory.Inventory) (predefinedHead, error) {
entries, err := inv.Catalogued(ctx)
if err != nil {
return predefinedHead{}, err
}
repository := ""
for _, e := range entries {
if e.Source.Seat == gitSeat && path.Base(strings.Trim(e.Source.Repository, "/")) == catalogue.CatalogueRepository {
repository = strings.Trim(e.Source.Repository, "/")
break
}
}
if repository == "" {
return predefinedHead{}, fmt.Errorf("no module of the catalogue names the repository %s on the forge",
catalogue.CatalogueRepository)
}
var heads []struct {
SHA string `json:"sha"`
Commit struct {
Committer struct {
Date time.Time `json:"date"`
} `json:"committer"`
} `json:"commit"`
}
if err := askGitea(ctx, "gitea_api", map[string]any{"path": "/repos/" + repository +
"/commits?limit=1&stat=false&verification=false&files=false"}, &heads); err != nil {
return predefinedHead{}, err
}
if len(heads) == 0 || heads[0].SHA == "" {
return predefinedHead{}, fmt.Errorf("the forge named no head of %s's trunk", repository)
}
head := predefinedHead{Repository: repository, Commit: heads[0].SHA,
Generation: int(heads[0].Commit.Committer.Date.Unix())}
var file struct {
Content string `json:"content"`
Encoding string `json:"encoding"`
}
if err := askGitea(ctx, "gitea_api", map[string]any{"path": "/repos/" + repository + "/contents/" +
catalogue.PredefinedFile + "?ref=" + head.Commit}, &file); err != nil {
return predefinedHead{}, err
}
if file.Encoding != "base64" {
return predefinedHead{}, fmt.Errorf("the forge answered %s at %.8s in the encoding %q", catalogue.PredefinedFile,
head.Commit, file.Encoding)
}
raw, err := base64.StdEncoding.DecodeString(file.Content)
if err != nil {
return predefinedHead{}, err
}
if head.List, err = catalogue.ParsePredefined(raw); err != nil {
return predefinedHead{}, fmt.Errorf("%s at %.8s is refused: %w", catalogue.PredefinedFile, head.Commit, err)
}
return head, nil
}
// askGitea asks the forge's module one tool on the bus and reads its result into into.
func askGitea(ctx context.Context, tool string, args map[string]any, into any) error {
js, err := aBus()
if err != nil {
return err
}
defer js.Close()
raw, _ := json.Marshal(args)
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", tool, raw, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("gitea.%s: %s", tool, answer.Error)
}
var wrapped struct {
Result json.RawMessage `json:"result"`
}
if err := json.Unmarshal(answer.Result, &wrapped); err != nil {
return err
}
return json.Unmarshal(wrapped.Result, into)
}
// keyTake is one take-in from a key: the module, the indexer's trunk value naming the key's fingerprint, and the
// registration's warrant.
type keyTake struct {
Module string
Trunk sdkdelivery.Module
Ask registrationAsk
}
// takeInKey takes a module in from the builder's key for the fingerprint its trunk value names: records the key as a
// build and registers the module, or says why not. errKeyWaits when it waits, errKeyFailed when the key failed.
func takeInKey(ctx context.Context, inv *inventory.Inventory, keys keyReads, t keyTake) (
catalogue.Manifest, inventory.Build, error) {
head, err := predefinedAtTrunk(ctx, inv)
if err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("%w: the list of predefined keys at the catalogue's "+
"trunk head cannot be read, so %s is not judged: %v", errKeyWaits, t.Module, err)
}
keepPredefinedHead(ctx, inv, head)
previous, err := inv.Builds(ctx, t.Module, 5)
if err != nil {
return catalogue.Manifest{}, inventory.Build{}, err
}
kept, manifest, err := buildOfKey(ctx, keys, t, keyContext{list: head.List, previous: previous,
archive: func(ctx context.Context, reference string) ([]byte, error) { return readArchive(ctx, inv, reference) }})
if err != nil {
return catalogue.Manifest{}, kept, err
}
if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err
}
recorded := inventory.Source{Repository: t.Trunk.Repository, Seat: gitSeat, Path: keyPath(t.Trunk),
BuiltFrom: t.Trunk.Commit, Head: t.Trunk.Commit, Against: kept.Against}
trunk := "main"
if was, err := inv.SourceOf(ctx, manifest.Module); err == nil {
recorded.Ref = followedBranch(was.Ref)
if recorded.Ref != "" {
trunk = recorded.Ref
}
}
registered, err := registerJudged(ctx, inv, takenBuild{kept: kept, manifest: manifest, recorded: recorded,
ask: t.Ask, path: recorded.Path, trunk: trunk, by: "the builder", what: t.Trunk.Repository,
commit: t.Trunk.Commit})
return registered, kept, err
}
// keyContext is what a key's build is made with besides the key: the list of predefined keys its manifest is judged
// against, the module's recent builds (the bases its last one stood on are carried) and how an archive is read.
type keyContext struct {
list catalogue.PredefinedList
previous []inventory.Build
archive func(ctx context.Context, reference string) ([]byte, error)
}
// keyPath is a module's directory as the catalogue records it: empty for a module at its repository's root.
func keyPath(m sdkdelivery.Module) string {
if m.Path == sdkdelivery.RootPath {
return ""
}
return strings.Trim(m.Path, "/")
}
// buildOfKey reads the key of the trunk value's fingerprint and makes its build record and the manifest it resolves to
// (ADR 0498 §8, ADR 0501 §1, §8): the module's manifest by the key's digest, judged against the list of predefined keys
// at the catalogue's trunk head, resolved with the key's artifacts.
func buildOfKey(ctx context.Context, keys keyReads, t keyTake, kc keyContext) (
inventory.Build, catalogue.Manifest, error) {
waits := func(format string, a ...any) (inventory.Build, catalogue.Manifest, error) {
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("%w: "+format, append([]any{errKeyWaits}, a...)...)
}
if t.Trunk.FP2 == "" {
why := t.Trunk.None2
if why == "" {
why = "the indexer wrote no fingerprint of the new form"
}
return waits("%s at %.8s has no fingerprint to take a build by: %s", t.Module, t.Trunk.Commit, why)
}
k, err := keys.Build(ctx, t.Module, t.Trunk.FP2)
if err != nil {
return waits("%s's key cannot be read: %v", t.Module, err)
}
switch {
case k.Failed():
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("%w: %s", errKeyFailed, k.Says())
case !k.Done():
return waits("%s", k.Says())
case len(k.Unreferenced()) > 0:
return waits("%s", k.Says())
}
src, err := keys.Manifest(ctx, k.Build.Manifest)
if err != nil {
return waits("%s's manifest: %v", t.Module, err)
}
manifest, err := catalogue.ParseManifestWith([]byte(src.JSON), kc.list)
if err != nil {
// **A word the running controller does not read waits for the controller that reads it** (ADR 0501 §7): the
// key stays done, and the controller that reads the word registers it with no second build.
if word := refusedWord(err.Error()); word != "" {
return waits("%s's manifest uses the word %q, which this controller does not read: it is registered "+
"when a controller that reads it runs", t.Module, word)
}
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("the builder built %s and its manifest is "+
"refused: %w", t.Module, err)
}
if manifest.Module != t.Module {
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("the key of %s names the manifest of %s",
t.Module, manifest.Module)
}
built, err := builtOfKey(ctx, kc.archive, manifest, k)
if err != nil {
return waits("%v", err)
}
resolved, err := manifest.Resolve(built)
if err != nil {
return inventory.Build{}, catalogue.Manifest{}, fmt.Errorf("the builder built %s and it does not resolve: %w",
t.Module, err)
}
raw, err := json.Marshal(resolved)
if err != nil {
return inventory.Build{}, catalogue.Manifest{}, err
}
kept := inventory.Build{ID: keyBuildID(k), Repository: t.Trunk.Repository, Module: t.Module,
Commit: t.Trunk.Commit, On: k.Build.By, Path: keyPath(t.Trunk), Manifest: raw,
SourceFingerprint: k.Fingerprint, At: time.Now().UTC()}
// **Read from the manifest, fresh at every key** (the same rule build-agent says, internal/buildreads). Sources are
// left unsaid but for a module whose every artifact names its paths: the planner then reads the module's whole
// directory, over-planning and never under-planning, and a fingerprint that did not change moves nothing (R4e A9).
// **Its build source where its every artifact names its paths** (novox/hq ADR 0499 §1, §5; issue 1758, B1): the
// planner reads a merge's changed files against those paths and the manifest alone, so a change elsewhere in the
// module's directory touches nothing it builds from.
if own := pathsBuildSource(manifest, keyPath(t.Trunk)); own != nil {
kept.Sources = []inventory.BuildSource{{Paths: own}}
}
for _, c := range buildreads.Of(manifest) {
kept.Read = append(kept.Read, inventory.ReadRepository{Repository: c.Repository, Ref: c.Ref})
}
// **The bases its last build stood on, carried** (R4e, lead decision): what the collector keeps a mirrored base by.
// A key names none; the module's newest successful build's are carried so the keep set does not change.
for _, b := range kc.previous {
if b.Worked() && b.Module == t.Module && b.ID != kept.ID {
kept.Against = append([]string(nil), b.Against...)
break
}
}
for _, a := range k.Build.Artifacts {
kept.Made = append(kept.Made, inventory.Artifact{Name: a.Name, Kind: kindOf(manifest, a),
Reference: catalogue.Recorded(a.Reference)})
}
return kept, resolved, nil
}
// kindOf is an artifact's kind: the key's, else the manifest's.
func kindOf(m catalogue.Manifest, a sdkdelivery.BuiltArtifact) string {
if a.Kind != "" || m.Build == nil {
return a.Kind
}
for _, d := range m.Build.Artifacts {
if d.Name == a.Name {
return d.Kind
}
}
return ""
}
// builtOfKey is each of a done key's artifacts as Resolve reads it: its reference and digest, a TypeScript bundle's
// launchers by the build modules' one rule (mesh-sdk buildstep.Launchers), and, for the archive a module packs its
// templates into, each template's text read back from the archive the store holds under the key's reference.
func builtOfKey(ctx context.Context, archive func(context.Context, string) ([]byte, error), m catalogue.Manifest,
k builderkeys.Key) (
[]catalogue.Built, error) {
declared := map[string]catalogue.Artifact{}
if m.Build != nil {
for _, a := range m.Build.Artifacts {
declared[a.Name] = a
}
}
templates := ""
if m.Templated() {
templates = catalogue.TemplatesArtifact(m)
}
var out []catalogue.Built
for _, a := range k.Build.Artifacts {
d := declared[a.Name]
b := catalogue.Built{Name: a.Name, Kind: kindOf(m, a), Reference: catalogue.Recorded(a.Reference),
Digest: a.Digest}
if launchers := buildstep.Launchers(d.Language, d.Entrypoints); len(launchers) > 0 {
b.Launchers = launchers
}
if a.Name == templates && templates != "" {
body, err := archive(ctx, b.Reference)
if err != nil {
return nil, fmt.Errorf("the templates archive %q of %s cannot be read from the artifact store: %v",
a.Name, m.Module, err)
}
if b.Texts, err = unpackTexts(body); err != nil {
return nil, fmt.Errorf("the templates archive %q of %s cannot be read back: %v", a.Name, m.Module, err)
}
}
out = append(out, b)
}
return out, nil
}
// unpackTexts is every file of a packed templates archive (gzip over tar), by its path inside it.
func unpackTexts(body []byte) (map[string]string, error) {
zipped, err := gzip.NewReader(bytes.NewReader(body))
if err != nil {
return nil, err
}
defer zipped.Close()
reader := tar.NewReader(zipped)
texts := map[string]string{}
for {
header, err := reader.Next()
if errors.Is(err, io.EOF) {
return texts, nil
}
if err != nil {
return nil, err
}
if header.Typeflag != tar.TypeReg {
continue
}
text, err := io.ReadAll(io.LimitReader(reader, 16<<20))
if err != nil {
return nil, err
}
texts[strings.TrimPrefix(header.Name, "./")] = string(text)
}
}
// keepPredefinedHead keeps the list read at the catalogue's trunk head when it is newer than the one kept, as a build of
// the catalogue's did (ADR 0348 §7), and judges every manifest read again against it. Said, never fatal: the list read
// is the one this take-in judges against either way.
func keepPredefinedHead(ctx context.Context, inv *inventory.Inventory, head predefinedHead) {
kept, err := inv.KeepPredefined(ctx, inventory.PredefinedRecord{Keys: head.List, Repository: head.Repository,
Commit: head.Commit, Generation: head.Generation, Build: "the trunk head " + short(head.Commit)})
if err != nil {
fmt.Printf("the list of predefined keys at %s's trunk head %s is not kept: %v\n", head.Repository,
short(head.Commit), err)
return
}
if kept {
fmt.Printf("the list of predefined keys at %s's trunk head %s is published: %s\n", head.Repository,
short(head.Commit), strings.Join(head.List.Keys(), ", "))
}
if err := loadPredefined(ctx, inv); err != nil {
fmt.Printf("the list of predefined keys kept cannot be read: %v\n", err)
}
}
// openBuilderKeys is the builder's and the indexer's buckets, read on a connection for the moment.
func openBuilderKeys() (builderkeys.Reader, func(), error) {
js, err := aBus()
if err != nil {
return builderkeys.Reader{}, nil, err
}
api, err := jetstream.New(js.Conn())
if err != nil {
js.Close()
return builderkeys.Reader{}, nil, err
}
return builderkeys.Reader{JS: api}, js.Close, nil
}
// pathsBuildSource is the build source of a module whose every artifact names the paths it is built from (ADR 0499
// §1): its manifest and each path, relative to its repository's root, a path read as a file or everything below a
// directory of that name (`<path>/**` holds both). Nil when an artifact names none: the module reads its directory.
func pathsBuildSource(m catalogue.Manifest, dir string) []string {
if m.Build == nil || len(m.Build.Artifacts) == 0 {
return nil
}
in := func(p string) string {
if dir == "" {
return p
}
return dir + "/" + p
}
seen := map[string]bool{}
out := []string{in("module.json")}
for _, a := range m.Build.Artifacts {
if len(a.Paths) == 0 {
return nil
}
for _, p := range a.Paths {
entry := in(strings.Trim(path.Clean(p), "/")) + "/**"
if !seen[entry] {
seen[entry] = true
out = append(out, entry)
}
}
}
return out
}
@@ -0,0 +1,251 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/inventory"
)
// withTrunkList stands in for the catalogue's trunk head: an empty list of predefined keys, readable, or not.
func withTrunkList(t *testing.T, readable bool) {
t.Helper()
was := predefinedAtTrunk
predefinedAtTrunk = func(context.Context, *inventory.Inventory) (predefinedHead, error) {
if !readable {
return predefinedHead{}, errors.New("the forge did not answer")
}
list, err := catalogue.ParsePredefined([]byte(`{}`))
return predefinedHead{List: list, Repository: "novox/mesh-catalog", Commit: "abcdef0123456789",
Generation: 1 << 40}, err
}
t.Cleanup(func() { predefinedAtTrunk = was })
}
// aKeyFor is a done key of module at fp, of one archive with a reference, and the indexer's trunk value naming it at
// commit in repository/path.
func aKeyFor(module, version, fp, repository, path, commit string) (keyBuckets, sdkdelivery.Module) {
raw := `{"module":"` + module + `","version":"` + version + `","build":{"artifacts":[{"name":"a","kind":"archive","from":"files"}]},` +
`"resources":[{"id":"a","type":"archive","path":"/opt/` + module + `","artifact":"a"}]}`
digest := sdkdelivery.ManifestDigest([]byte(raw))
blob := shaOf(module + version)
k := builderkeys.Key{Module: module, Fingerprint: fp, Found: true, Revision: 1,
Build: sdkdelivery.BuilderBuild{State: sdkdelivery.BuildDone, By: "ace", Manifest: digest,
Artifacts: []sdkdelivery.BuiltArtifact{{Name: "a", Kind: "archive", Digest: blob,
Reference: "artifact-store://" + module + "/a/blobs/" + blob}}}}
return keyBuckets{keys: map[string]builderkeys.Key{module: k}, manifests: map[string]string{digest: raw}},
sdkdelivery.Module{FP2: fp, Manifest: digest, Repository: repository, Commit: commit, Path: path}
}
func fpOf(c byte) string { return "src2:" + strings.Repeat(string(c), 64) }
// A3, A4 (ADR 0498 §8, ADR 0501 §1-§3, §8): a done key of a trunk fingerprint records one build of today's layout,
// whose artifacts are the key's, and registers the module under the same rules of who may register as before.
func TestADoneKeyOfTheTrunkRegistersUnderTheSameRules(t *testing.T) {
open := theCatalogue(t) // sudo from novox/mesh-catalog modules/sudo, mesh-host from novox/mesh-host
ctx := t.Context()
inv := open.inventory
withTrunkList(t, true)
// A known module from the repository and path the catalogue builds it from: registered from its key.
keys, trunk := aKeyFor("sudo", "2", fpOf('a'), "novox/mesh-catalog", "modules/sudo", "c0ffee0123456789")
_, kept, err := takeInKey(ctx, inv, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}})
if err != nil {
t.Fatalf("0501 §1: a done key of sudo's trunk fingerprint was not taken in: %v", err)
}
if shelf, _ := inv.Catalogue(ctx); shelf["sudo"].Version != "2" {
t.Fatalf("0501 §1: sudo is %q", shelf["sudo"].Version)
}
got, found, err := inv.BuildByID(ctx, kept.ID)
if err != nil || !found || len(got.Made) != 1 || got.Made[0].Reference != kept.Made[0].Reference {
t.Fatalf("0498 §8: the recorded build is %+v (%v)", got, err)
}
if fps, err := inv.BuildSourceFingerprints(ctx, "sudo"); err != nil || fps["c0ffee0123456789"] != fpOf('a') {
t.Fatalf("0498 §8: the recorded build's fingerprint is %v (%v), not the key's", fps, err)
}
// Recording the same key again is one record.
if _, _, err := takeInKey(ctx, inv, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}}); err != nil {
t.Fatal(err)
}
builds, _ := inv.Builds(ctx, "sudo", 20)
n := 0
for _, b := range builds {
if b.ID == kept.ID {
n++
}
}
if n != 1 {
t.Fatalf("0498 §8: one key taken in twice is %d records", n)
}
// A known module from another repository: refused, as a build of it was (ADR 0266).
other, otherTrunk := aKeyFor("sudo", "evil", fpOf('b'), "agent/sudo", "modules/sudo", "d00d0123456789ab")
if _, _, err := takeInKey(ctx, inv, other, keyTake{Module: "sudo", Trunk: otherTrunk,
Ask: registrationAsk{Merge: otherTrunk.Commit}}); !errors.Is(err, errNotItsSource) {
t.Fatalf("0501 §3: a known module from another repository was taken in: %v", err)
}
// A new module from a repository the catalogue builds from: registered.
zram, zramTrunk := aKeyFor("zram", "1", fpOf('c'), "novox/mesh-catalog", "modules/zram", "c0ffee0123456789")
if _, _, err := takeInKey(ctx, inv, zram, keyTake{Module: "zram", Trunk: zramTrunk, Ask: registrationAsk{Merge: zramTrunk.Commit}}); err != nil {
t.Fatalf("0501 §3: a new module from a built repository was refused: %v", err)
}
// A new module from a repository nothing is built from, with no Approve: refused.
tools, toolsTrunk := aKeyFor("agent-tools", "1", fpOf('d'), "agent/tools", "", "e0e0e0e0e0e0e0e0")
if _, _, err := takeInKey(ctx, inv, tools, keyTake{Module: "agent-tools", Trunk: toolsTrunk,
Ask: registrationAsk{Merge: toolsTrunk.Commit}}); !errors.Is(err, errNotItsSource) {
t.Fatalf("0501 §3: a new repository's module was taken in with no warrant: %v", err)
}
// At the terminal the operator registers it (ADR 0501 §4).
if _, _, err := takeInKey(ctx, inv, tools, keyTake{Module: "agent-tools", Trunk: toolsTrunk,
Ask: registrationAsk{Terminal: true}}); err != nil {
t.Fatalf("0501 §4: the terminal's registration was refused: %v", err)
}
}
// ADR 0501 §2: a key only a candidate names registers nothing: the trunk value names another fingerprint, whose key is
// not there, and the take-in waits.
func TestAKeyOnlyACandidateNamesRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
withTrunkList(t, true)
keys, trunk := aKeyFor("sudo", "3", fpOf('e'), "novox/mesh-catalog", "modules/sudo", "c0ffee0123456789")
trunk.FP2 = fpOf('f') // the trunk names another fingerprint than the candidate's done key
_, _, err := takeInKey(ctx, open.inventory, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}})
if !errors.Is(err, errKeyWaits) {
t.Fatalf("0501 §2: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["sudo"].Version != "1" {
t.Fatalf("0501 §2: a candidate's key registered sudo %q", shelf["sudo"].Version)
}
}
// ADR 0501 §8: a take-in that cannot read the list of predefined keys at the catalogue's trunk head waits, says so and
// registers nothing.
func TestATakeInWithoutTheTrunksListWaits(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
withTrunkList(t, false)
keys, trunk := aKeyFor("sudo", "4", fpOf('1'), "novox/mesh-catalog", "modules/sudo", "c0ffee0123456789")
_, _, err := takeInKey(ctx, open.inventory, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}})
if !errors.Is(err, errKeyWaits) || !strings.Contains(err.Error(), "predefined keys") {
t.Fatalf("0501 §8: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["sudo"].Version != "1" {
t.Fatalf("0501 §8: registered %q", shelf["sudo"].Version)
}
}
// A3 (the lead's decision on Against and Read): a key's record keeps the collector's keep set as it was and its read
// repositories are its manifest's, fresh at every key.
func TestAKeysRecordKeepsTheKeepSetAndReadsItsManifest(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
withTrunkList(t, true)
before, err := inv.KeepSet(ctx)
if err != nil {
t.Fatal(err)
}
raw := `{"module":"sudo","version":"5","build":{"artifacts":[{"name":"img","kind":"image","from":"Dockerfile",` +
`"context":{"repository":"novox/elsewhere","seat":"git"}}]},"resources":[{"id":"run","type":"container","artifact":"img"}]}`
digest := sdkdelivery.ManifestDigest([]byte(raw))
k := builderkeys.Key{Module: "sudo", Fingerprint: fpOf('2'), Found: true, Revision: 1,
Build: sdkdelivery.BuilderBuild{State: sdkdelivery.BuildDone, By: "g14", Manifest: digest,
Artifacts: []sdkdelivery.BuiltArtifact{{Name: "img", Kind: "image", Digest: shaOf("img5"),
Reference: "artifact-store://sudo/img@" + shaOf("img5")}}}}
keys := keyBuckets{keys: map[string]builderkeys.Key{"sudo": k}, manifests: map[string]string{digest: raw}}
trunk := sdkdelivery.Module{FP2: fpOf('2'), Manifest: digest, Repository: "novox/mesh-catalog",
Commit: "c0ffee0123456789", Path: "modules/sudo"}
if _, _, err := takeInKey(ctx, inv, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}}); err != nil {
t.Fatal(err)
}
after, err := inv.KeepSet(ctx)
if err != nil {
t.Fatal(err)
}
for reference, why := range before {
if !reflect.DeepEqual(after[reference], why) && !strings.Contains(strings.Join(after[reference], ","), strings.Join(why, ",")) {
t.Errorf("the keep set changed for %s: %v, was %v", reference, after[reference], why)
}
}
read, err := inv.ReadRepositories(ctx)
if err != nil {
t.Fatal(err)
}
found := false
for _, r := range read["sudo"] {
if !r.Own && r.Repository == "novox/elsewhere" {
found = true
}
}
if !found {
text, _ := json.Marshal(read["sudo"])
t.Fatalf("the key's record does not read its manifest's context: %s", text)
}
}
// A9 (ADR 0453 §6, ADR 0499 consequences): at the switch, a key's build whose every artifact has the digest of the
// build-agent build the mesh holds (`src1:`) is no move: it is registered with that build's artifacts, and a walk sends
// nothing for it.
func TestAKeyWithTheHeldBuildsDigestsIsNoMove(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
withTrunkList(t, true)
keys, trunk := aKeyFor("sudo", "6", fpOf('6'), "novox/mesh-catalog", "modules/sudo", "c0ffee0123456789")
k := keys.keys["sudo"]
held := inventory.Build{ID: "build-sudo-old", Module: "sudo", Repository: "novox/mesh-catalog", Path: "modules/sudo",
Commit: "0ld0ld0ld0ld0ld0", SourceFingerprint: "src1:" + strings.Repeat("9", 64),
Made: []inventory.Artifact{{Name: "a", Kind: "archive", Reference: k.Build.Artifacts[0].Reference}}}
held.Manifest, _ = json.Marshal(catalogue.Manifest{Module: "sudo", Version: "held"})
if err := inv.RecordBuild(ctx, held); err != nil {
t.Fatal(err)
}
_, kept, err := takeInKey(ctx, inv, keys, keyTake{Module: "sudo", Trunk: trunk, Ask: registrationAsk{Merge: trunk.Commit}})
if err != nil {
t.Fatal(err)
}
same, err := inv.SameSourceCommits(ctx, "sudo", kept.ID)
if err != nil || !same["0ld0ld0ld0ld0ld0"] {
t.Fatalf("A9: the key's build does not stand on the held build of the same digests: %v %v", same, err)
}
if shelf, _ := inv.Catalogue(ctx); shelf["sudo"].Version != "held" {
t.Fatalf("A9: sudo is registered as %q, not with the held build's artifacts", shelf["sudo"].Version)
}
}
// A13 against the store: the switch's list reads the build the mesh holds of each module and the builder's key of its
// trunk fingerprint, and names each module unmoved, moved or not known.
func TestTheSwitchsListReadsTheHeldBuildsAndTheKeys(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
keys, trunk := aKeyFor("sudo", "1", fpOf('b'), "novox/mesh-catalog", "modules/sudo", "c0ffee0123456789")
k := keys.keys["sudo"]
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-sudo-held", Module: "sudo", Repository: "novox/mesh-catalog",
Commit: "c0ffee0123456789", Made: []inventory.Artifact{{Name: "a", Kind: "archive",
Reference: k.Build.Artifacts[0].Reference}}}); err != nil {
t.Fatal(err)
}
reads := planKeysOf{values: map[string]sdkdelivery.Module{"sudo@": trunk}, keys: map[string]builderkeys.Key{"sudo": k}}
lines, err := switchList(ctx, inv, reads)
if err != nil {
t.Fatal(err)
}
got := map[string]string{}
for _, l := range lines {
got[l.Module] = l.Verdict
}
if got["sudo"] != switchUnmoved || got["mesh-host"] != switchUnknown {
t.Fatalf("A13: the switch's list is %+v", lines)
}
}
+231
View File
@@ -0,0 +1,231 @@
package main
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/inventory"
)
// keyBuckets is the builder's and the indexer's buckets as a take-in reads them, for a test: keys by module, manifests by
// digest.
type keyBuckets struct {
keys map[string]builderkeys.Key
manifests map[string]string
}
func (k keyBuckets) Build(_ context.Context, module, fp string) (builderkeys.Key, error) {
if key, ok := k.keys[module]; ok && key.Fingerprint == fp {
return key, nil
}
return builderkeys.Key{Module: module, Fingerprint: fp}, nil
}
func (k keyBuckets) Manifest(_ context.Context, digest string) (sdkdelivery.Manifest, error) {
raw, ok := k.manifests[digest]
if !ok {
return sdkdelivery.Manifest{}, errors.New("the indexer holds no manifest " + digest)
}
return sdkdelivery.Manifest{Module: "x", JSON: raw}, nil
}
const keyFP = "src2:" + "1111111111111111111111111111111111111111111111111111111111111111"
// aKeyOf is a done key of module x at keyFP for the manifest raw, with these artifacts.
func aKeyOf(raw string, revision uint64, made ...sdkdelivery.BuiltArtifact) (keyBuckets, sdkdelivery.Module) {
digest := sdkdelivery.ManifestDigest([]byte(raw))
k := builderkeys.Key{Module: "x", Fingerprint: keyFP, Found: true, Revision: revision,
Build: sdkdelivery.BuilderBuild{State: sdkdelivery.BuildDone, By: "ace", Manifest: digest, Artifacts: made}}
trunk := sdkdelivery.Module{FP2: keyFP, Manifest: digest, Repository: "novox/mesh-catalog", Commit: "c0ffee0123456789",
Path: "modules/x"}
return keyBuckets{keys: map[string]builderkeys.Key{"x": k}, manifests: map[string]string{digest: raw}}, trunk
}
const tsBundle = `{"module":"x","version":"1","tools":["x_read"],
"build":{"artifacts":[{"name":"tools","kind":"bundle","language":"typescript","entrypoints":["dist/index.js"]},
{"name":"img","kind":"image","from":"Dockerfile","context":{"repository":"novox/other","seat":"git"}}]},
"resources":[{"id":"run","type":"container","artifact":"img"}]}`
func shaOf(b string) string {
sum := sha256.Sum256([]byte(b))
return "sha256:" + hex.EncodeToString(sum[:])
}
// A3 (ADR 0498 §8, ADR 0501 §1): a done key whose every artifact names a reference is one recorded build, its artifacts
// the key's, its launchers by the build modules' rule, its reads from its manifest, the bases its last build stood on
// carried; recorded twice it is one record.
func TestADoneKeyIsOneRecordedBuildOfItsArtifacts(t *testing.T) {
keys, trunk := aKeyOf(tsBundle, 3,
sdkdelivery.BuiltArtifact{Name: "tools", Kind: "bundle", Digest: shaOf("t"),
Reference: "artifact-store://x/tools/blobs/" + shaOf("t")},
sdkdelivery.BuiltArtifact{Name: "img", Kind: "image", Digest: shaOf("i"),
Reference: "artifact-store://x/img@" + shaOf("i")})
previous := []inventory.Build{{ID: "build-1", Module: "x", Against: []string{"artifact-store://base@sha256:aa"}}}
take := keyTake{Module: "x", Trunk: trunk}
kept, resolved, err := buildOfKey(t.Context(), keys, take, keyContext{previous: previous})
if err != nil {
t.Fatal(err)
}
if kept.ID != "key-1111111111111111111111111111111111111111111111111111111111111111-3" {
t.Errorf("the record's id is %s", kept.ID)
}
again, _, _ := buildOfKey(t.Context(), keys, take, keyContext{previous: previous})
if again.ID != kept.ID {
t.Errorf("one key recorded twice is two records: %s and %s", kept.ID, again.ID)
}
if kept.SourceFingerprint != keyFP || kept.Commit != trunk.Commit || kept.Path != "modules/x" || kept.On != "ace" ||
kept.Module != "x" || kept.Repository != "novox/mesh-catalog" {
t.Errorf("the record: %+v", kept)
}
if len(kept.Made) != 2 || kept.Made[0].Reference != "artifact-store://x/tools/blobs/"+shaOf("t") ||
kept.Made[1].Reference != "artifact-store://x/img@"+shaOf("i") || kept.Made[1].Kind != "image" {
t.Errorf("Made is not the key's artifacts: %+v", kept.Made)
}
if len(kept.Read) != 1 || kept.Read[0].Repository != "novox/other" {
t.Errorf("Read is not the manifest's contexts: %+v", kept.Read)
}
if kept.Sources != nil || kept.Mirrored != nil {
t.Errorf("a key's record says build sources or mirrored bases it cannot know: %+v %+v", kept.Sources, kept.Mirrored)
}
if len(kept.Against) != 1 || kept.Against[0] != "artifact-store://base@sha256:aa" {
t.Errorf("the bases the last build stood on are not carried: %v", kept.Against)
}
if len(resolved.Bundles) != 1 || resolved.Bundles[0].Launchers["dist/index.js"] != "dist/index.serve.mjs" {
t.Errorf("the bundle's launchers: %+v", resolved.Bundles)
}
var m map[string]any
if err := json.Unmarshal(kept.Manifest, &m); err != nil || !strings.Contains(string(kept.Manifest), shaOf("i")) {
t.Errorf("the recorded manifest names no image digest: %s", kept.Manifest)
}
}
// A3 (ADR 0498 §8): a key with an artifact naming no reference, a key not done and a missing key are waited for, said;
// a failed key fails in the builder's words; a module with no fingerprint of the new form waits, naming why.
func TestAKeyNotReadyWaitsAndAFailedKeyFails(t *testing.T) {
noRef, trunk := aKeyOf(tsBundle, 1,
sdkdelivery.BuiltArtifact{Name: "tools", Kind: "bundle", Digest: shaOf("t"),
Reference: "artifact-store://x/tools/blobs/" + shaOf("t")},
sdkdelivery.BuiltArtifact{Name: "img", Kind: "image", Digest: shaOf("i")})
take := keyTake{Module: "x", Trunk: trunk}
_, _, err := buildOfKey(t.Context(), noRef, take, keyContext{})
if !errors.Is(err, errKeyWaits) || !strings.Contains(err.Error(), "waits for the reference of img") {
t.Errorf("a key without a reference: %v", err)
}
building := noRef
k := building.keys["x"]
k.Build = sdkdelivery.BuilderBuild{State: sdkdelivery.BuildBuilding, By: "g14"}
building.keys = map[string]builderkeys.Key{"x": k}
if _, _, err := buildOfKey(t.Context(), building, take, keyContext{}); !errors.Is(err, errKeyWaits) ||
!strings.Contains(err.Error(), "building on g14") {
t.Errorf("a building key: %v", err)
}
if _, _, err := buildOfKey(t.Context(), keyBuckets{}, take, keyContext{}); !errors.Is(err, errKeyWaits) ||
!strings.Contains(err.Error(), "no build of x") {
t.Errorf("no key: %v", err)
}
k.Build = sdkdelivery.BuilderBuild{State: sdkdelivery.BuildFailed, Error: "digests differ: ace sha256:1, g14 sha256:2"}
failed := keyBuckets{keys: map[string]builderkeys.Key{"x": k}}
if _, _, err := buildOfKey(t.Context(), failed, take, keyContext{}); !errors.Is(err, errKeyFailed) ||
!strings.Contains(err.Error(), "digests differ: ace sha256:1, g14 sha256:2") {
t.Errorf("a failed key: %v", err)
}
none := take
none.Trunk.FP2, none.Trunk.None2 = "", "no lock"
if _, _, err := buildOfKey(t.Context(), noRef, none, keyContext{}); !errors.Is(err, errKeyWaits) ||
!strings.Contains(err.Error(), "no lock") {
t.Errorf("no fingerprint: %v", err)
}
}
// ADR 0501 §7: a manifest word this controller does not read waits for the controller that reads it; never fails.
func TestAManifestWordTheControllerDoesNotReadWaits(t *testing.T) {
raw := `{"module":"x","version":"1","not-a-word-yet":true,"build":{"artifacts":[{"name":"a","kind":"archive","from":"files"}]},
"resources":[{"id":"a","type":"archive","path":"/opt/x","artifact":"a"}]}`
keys, trunk := aKeyOf(raw, 1, sdkdelivery.BuiltArtifact{Name: "a", Kind: "archive", Digest: shaOf("a"),
Reference: "artifact-store://x/a/blobs/" + shaOf("a")})
_, _, err := buildOfKey(t.Context(), keys, keyTake{Module: "x", Trunk: trunk}, keyContext{})
if !errors.Is(err, errKeyWaits) || !strings.Contains(err.Error(), `"not-a-word-yet"`) {
t.Fatalf("an unread word: %v", err)
}
}
func packed(t *testing.T, files map[string]string) []byte {
t.Helper()
var buf bytes.Buffer
z := gzip.NewWriter(&buf)
w := tar.NewWriter(z)
for name, text := range files {
if err := w.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(text)), Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
if _, err := w.Write([]byte(text)); err != nil {
t.Fatal(err)
}
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
if err := z.Close(); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
// A templated module's texts are read back from the archive the store holds under the key's reference; an archive the
// store does not answer is waited for (the lead's Q1b).
func TestATemplatedModulesTextsAreReadFromItsArchive(t *testing.T) {
raw := `{"module":"x","version":"1","templates":"templates",
"build":{"artifacts":[{"name":"templates","kind":"archive","from":"templates"}]},
"resources":[{"id":"main","type":"file","path":"/etc/x.conf","mode":"0644","template":"templates/x.conf"}]}`
body := packed(t, map[string]string{"x.conf": "listen 1\n"})
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
keys, trunk := aKeyOf(raw, 1, sdkdelivery.BuiltArtifact{Name: "templates", Kind: "archive", Digest: digest,
Reference: "artifact-store://x/templates/blobs/" + digest})
asked := ""
archive := func(_ context.Context, reference string) ([]byte, error) { asked = reference; return body, nil }
_, resolved, err := buildOfKey(t.Context(), keys, keyTake{Module: "x", Trunk: trunk}, keyContext{archive: archive})
if err != nil {
t.Fatal(err)
}
if asked != "artifact-store://x/templates/blobs/"+digest {
t.Errorf("the archive asked: %q", asked)
}
if text, _ := json.Marshal(resolved.Resources); !strings.Contains(string(text), "listen 1") {
t.Errorf("the template's text is not in the file: %s", text)
}
gone := func(context.Context, string) ([]byte, error) { return nil, errors.New("the store does not hold it") }
if _, _, err := buildOfKey(t.Context(), keys, keyTake{Module: "x", Trunk: trunk}, keyContext{archive: gone}); !errors.Is(err, errKeyWaits) ||
!strings.Contains(err.Error(), "cannot be read from the artifact store") {
t.Errorf("an archive the store does not answer: %v", err)
}
}
// The launchers a key's bundle is resolved with are mesh-sdk's one rule (buildstep.Launchers), the names
// mesh-build-node writes: a TypeScript bundle's entrypoint x.js runs as x.serve.mjs, a Go bundle has none.
func TestALaunchersRuleIsTheBuildModules(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(tsBundle))
if err != nil {
t.Fatal(err)
}
k := builderkeys.Key{Build: sdkdelivery.BuilderBuild{Artifacts: []sdkdelivery.BuiltArtifact{{Name: "tools", Digest: "sha256:1",
Reference: "artifact-store://x/tools/blobs/sha256:1"}}}}
built, err := builtOfKey(t.Context(), nil, m, k)
if err != nil {
t.Fatal(err)
}
if built[0].Launchers["dist/index.js"] != "dist/index.serve.mjs" || built[0].Kind != "bundle" {
t.Fatalf("built: %+v", built[0])
}
}
+44 -14
View File
@@ -54,6 +54,8 @@ func run() error {
return fmt.Errorf("no command given")
}
// The bus's users are composed from what each module announced it keeps (novox/hq ADR 0495, issue 1790).
inventory.Announcements = readAnnouncements
// Every connection this process dials says what it is (novox/hq issue 327).
broker.ConnectionName = connectionName(args[0], os.Getenv(verbVar))
@@ -73,8 +75,13 @@ func run() error {
}
switch args[0] {
case "build":
return buildCommand(ctx, args[1:])
case "switch":
return switchCommand(ctx, args[1:])
case "register":
return registerCommand(ctx, args[1:])
case "build", "rebuild":
// **The hand build goes with the asks** (novox/hq ADR 0501 §5): refused by name, naming the replacement.
return removedByName(args[0])
case "builder":
return builderCommand(ctx, args[1:])
case "board":
@@ -98,14 +105,12 @@ func run() error {
return cancelCommand(ctx, args[1:])
case "clear":
return clearCommand(ctx, args[1:])
case "rebuild":
return rebuildCommand(ctx, args[1:])
case "replay":
return replayCommand(ctx, args[1:])
case "kill":
return killCommand(ctx, args[1:])
case "forget-checks":
return forgetChecksCommand(ctx, args[1:])
return removedByName("forget-checks")
case "try":
return tryCommand(ctx, args[1:])
case "pause", "resume":
@@ -232,8 +237,8 @@ func run() error {
}
}
func usage() {
fmt.Fprint(os.Stderr, `mesh-controller — the control plane
// usageText is the terminal's help: every command it has, and none it refuses as removed (removedByName).
var usageText = `mesh-controller — the control plane
On a node the operator types these as 'mesh-cli <command>' (in zsh, 'nox <command>'): asked
through the node's engine, and run as the controller's terminal only on the control-node
@@ -307,15 +312,14 @@ func usage() {
operator key make [--out <file>] make the operator's sealing key, off the mesh; private half to the file only
operator key set <public> [--replace] tell the mesh which operator key to seal to
operator key show the operator key, and what it can recover
build <repository> [--ref R] have a build machine build it, and record what came out
build --behind build every module the mesh holds older than its source
build --on <module> rebuild every module that stands on this module's artifacts, bases first
register <module> <owner>/<repository> <path> register a module from a repository the catalogue does not
build, from the builder's done key of its fingerprint (ADR 0501 §4)
switch [--json] the switch's list: each module, the build it runs, and what the builder holds
builds [<module>] what has been built lately, and what came of it
queue [--json] every ask in the build queue: waiting, in flight (where, how long), dead
cancel <id> drop a waiting or dead ask; recorded failed, cancelled by hand
clear [--dead] cancel every waiting ask (and the dead ones); never one in flight
rebuild <module|build-id> ask the module's source again, or that build's, under a new id
replay <build-id> [--register [--older]] that build's commit again; a dry run unless --register
replay <build-id> that build's commit again, a dry run: what a build of it makes
kill <id> end a build where it runs; recorded failed, killed by hand
pause [<node>] / resume [<node>] the build seat's holder there, or every holder, takes nothing new / again
plans retry <id> ask a failed plan's failed builds again, and carry the plan on
@@ -350,10 +354,13 @@ func usage() {
version what this binary is
Each context reaches its own store through its own credential (novox/hq ADR 0008), named
`+store.Variable("<context>")+` — or `+store.FileVariable("<context>")+`, naming a file that holds
` + store.Variable("<context>") + ` — or ` + store.FileVariable("<context>") + `, naming a file that holds
the same thing and keeps the password out of the environment. This process holds:
`)
`
func usage() {
fmt.Fprint(os.Stderr, usageText)
for _, c := range held {
fmt.Fprintf(os.Stderr, " %-12s database %-12s from %s\n",
c.name, store.Database(c.name), store.Variable(c.name))
@@ -475,3 +482,26 @@ func connectionName(command, verb string) string {
}
return "mesh-controller command " + command
}
// removedByName refuses a command the controller no longer has, naming what replaced it (novox/hq ADR 0501 §4, §5;
// ADR 0429 §18; ADR 0503 §3; issue 1851, A11, and issue 1318).
func removedByName(command string) error {
replacement := map[string]string{
"build": "the controller asks no build (ADR 0501 §5): the builder builds every fingerprint the indexer writes, a " +
"module is registered from its done key, and a branch is tried with `try`; a module from a repository the " +
"catalogue does not build is registered at the controller's terminal with `register`",
"rebuild": "the controller asks no build (ADR 0501 §5): a failed key is built again by the builder as a new " +
"revision (ADR 0429 §18), and a walk's Retry waits for it",
"replay --register": "a build is registered only from the builder's done key of a trunk fingerprint (ADR 0501 " +
"§1, §2); `replay` without --register still shows what a build of a recorded commit makes",
"forget-checks": "build-agent makes no check build from R4e (ADR 0503 §3), so there is nothing to let go of",
"delivery-plan": "a delivery's plan is read with `delivery-order` and the checker's verdict (ADR 0503 §3)",
}[command]
return &removedCommand{msg: fmt.Sprintf("`%s` is removed: %s", command, replacement)}
}
// removedCommand is a command or verb the controller no longer has: a refusal that is the verb's whole answer, not a
// verb of a newer build this one cannot run.
type removedCommand struct{ msg string }
func (r *removedCommand) Error() string { return r.msg }
+20 -2
View File
@@ -65,6 +65,9 @@ type mergeCheckInput struct {
// group are the other heads of a delivery group composed with this one (novox/hq ADR 0239), each laid
// over the mesh after the one before: the group judged as one future state.
group []groupChange
// held are the approved config values held for the keys the pull requests introduce (novox/hq ADR 0502 clause 4),
// laid over the composition with the change only.
held []heldValue
// admin is a PostgreSQL the gate may create and drop databases in.
admin string
// beside is the directory holding the core clones beside the check (MESH_CHECK_BESIDE), for finding moves.
@@ -189,6 +192,8 @@ func mergeGateCommand(ctx context.Context, args []string) error {
"--changed, for a list too long for an argument (novox/hq issue 1247)")
group := set.String("group", "", "a delivery group's other heads, as JSON [{repository, tree, changed}], composed "+
"with this one as one future state (novox/hq ADR 0239)")
heldFile := set.String("held", "", "a file of the config values approved for keys the change's pull requests introduce, as JSON "+
"[{pull, node, module, key, value, approved_by}], laid over the composition with the change (novox/hq ADR 0502)")
asJSON := set.Bool("json", false, "print the verdict as JSON")
if _, err := parseAround(set, args); err != nil {
return err
@@ -208,6 +213,10 @@ func mergeGateCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
held, err := readHeld(*heldFile)
if err != nil {
return err
}
out := io.Writer(os.Stdout)
stdout := os.Stdout
if *asJSON {
@@ -223,7 +232,7 @@ func mergeGateCommand(ctx context.Context, args []string) error {
return err
}
v, err := judgeChange(ctx, mergeCheckInput{facts: f, repository: *repository, tree: *tree, changed: changedList,
group: others, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
group: others, held: held, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
os.Stdout = stdout
if err != nil {
return err
@@ -387,7 +396,16 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
}
}
sort.Strings(added)
after, tried, notes, err := composeEveryMachine(ctx, in, change, changeSources, added)
// **The held values are laid in this composition only** (novox/hq ADR 0502 clause 4): the mesh as it is does not
// declare their keys, so the base composition would refuse them.
withHeld := in
var heldNotes []string
withHeld.facts, heldNotes = layHeld(in.facts, in.held, change)
for _, n := range heldNotes {
say("%s", n)
}
v.Notes = append(v.Notes, heldNotes...)
after, tried, notes, err := composeEveryMachine(ctx, withHeld, change, changeSources, added)
if err != nil {
return v, fmt.Errorf("the mesh with the change could not be raised: %w", err)
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"encoding/json"
"fmt"
"os"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
// Config values held for the keys an open pull request introduces (novox/hq ADR 0502 clause 4, issue 1826): the gate
// lays each approved one over its module's values on its node in the composition with the change, and in that one
// only. In the mesh as it is the key is not declared, so the value would be refused there.
// heldValue is one approved held value as the build agent writes it (--held): the value of Key for Module on Node,
// a machine's real name, which the snapshot holds only as a pseudonym.
type heldValue struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// readHeld reads the file --held names; none named, none held.
func readHeld(path string) ([]heldValue, error) {
if path == "" {
return nil, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var held []heldValue
if err := json.Unmarshal(raw, &held); err != nil {
return nil, fmt.Errorf("the held values are not readable: %w", err)
}
for _, h := range held {
if h.Node == "" || h.Module == "" || h.Key == "" || len(h.Value) == 0 {
return nil, fmt.Errorf("a held value of %s names no node, module, key or value", h.Pull)
}
}
return held, nil
}
// layHeld is the snapshot with each held value laid as the value of its key in its node's layer of its module, and what
// it said of them. The snapshot given is not changed. **A held value for a machine the snapshot does not hold, or a
// module the change does not define, is said and not laid**, never dropped silently; one for a module not assigned to
// its machine is laid and said. The notes name machines as the snapshot does (roles and pseudonym), not by their names.
func layHeld(f snapshot.Facts, held []heldValue, shelf map[string]catalogue.Manifest) (snapshot.Facts, []string) {
if len(held) == 0 {
return f, nil
}
out := f
out.Machines = make([]snapshot.Machine, len(f.Machines))
copy(out.Machines, f.Machines)
var notes, laid []string
pulls := map[string]bool{}
for _, h := range held {
at := -1
pseudonym := snapshot.Pseudonym("machine", h.Node)
for i, m := range out.Machines {
if m.Name == pseudonym {
at = i
}
}
if at < 0 {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the snapshot holds no machine %s",
h.Module, h.Pull, h.Key, h.Value, pseudonym))
continue
}
if _, defined := shelf[h.Module]; !defined {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the change defines no module %s",
h.Module, h.Pull, h.Key, h.Value, h.Module))
continue
}
var value any
if err := json.Unmarshal(h.Value, &value); err != nil {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s) is not composed: it is not a value: %v",
h.Module, h.Pull, h.Key, err))
continue
}
m := &out.Machines[at]
layers := make([]snapshot.Settings, len(m.Settings))
copy(layers, m.Settings)
found := false
for i := range layers {
if layers[i].Module == h.Module {
values := make(map[string]any, len(layers[i].Values)+1)
for k, v := range layers[i].Values {
values[k] = v
}
values[h.Key] = value
layers[i].Values = values
found = true
}
}
if !found {
layers = append(layers, snapshot.Settings{Module: h.Module, Values: map[string]any{h.Key: value}})
sort.Slice(layers, func(a, b int) bool { return layers[a].Module < layers[b].Module })
}
m.Settings = layers
assigned := false
for _, a := range m.Assigned {
assigned = assigned || a == h.Module
}
by := ""
if h.Approved != "" {
by = ", approved by " + h.Approved
}
line := fmt.Sprintf("%s on %s %s=%s%s", h.Module, m.Described(), h.Key, h.Value, by)
if !assigned {
line += " (not assigned there in the snapshot)"
}
laid = append(laid, line)
pulls[h.Pull] = true
}
if len(laid) > 0 {
var names []string
for p := range pulls {
names = append(names, p)
}
sort.Strings(names)
notes = append([]string{fmt.Sprintf("composed with %d value(s) held for %s: %s", len(laid), strings.Join(names, ", "),
strings.Join(laid, "; "))}, notes...)
}
return out, notes
}
@@ -0,0 +1,85 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
func heldGateFacts() snapshot.Facts {
return snapshot.Facts{Machines: []snapshot.Machine{
{Name: snapshot.Pseudonym("machine", "ace"), Roles: []string{"the build node"}, Assigned: []string{"n8n"},
Settings: []snapshot.Settings{{Module: "n8n", Values: map[string]any{"port": 5678.0}}}},
{Name: snapshot.Pseudonym("machine", "g14")},
}}
}
var heldShelf = map[string]catalogue.Manifest{"n8n": {Module: "n8n"}}
// 1826/AC3: a held value is laid in its node's layer of its module, beside what the layer holds, and said.
func TestAHeldValueIsLaidOverItsNodesLayerAndSaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}, heldShelf)
layer := got.Machines[0].Settings[0].Values
if layer["media-gid"] != 2000.0 || layer["port"] != 5678.0 {
t.Fatalf("the layer holds %v", layer)
}
if _, has := f.Machines[0].Settings[0].Values["media-gid"]; has {
t.Fatal("the snapshot given was changed")
}
if len(notes) != 1 || !strings.Contains(notes[0], "composed with 1 value(s) held for novox/mesh-catalog#434") ||
!strings.Contains(notes[0], "media-gid=2000") || !strings.Contains(notes[0], "approved by the operator") ||
strings.Contains(notes[0], "ace") {
t.Fatalf("said %q", notes)
}
}
// 1826/AC3: a machine with no layer of the module gets one.
func TestAHeldValueMakesTheLayerAMachineLacks(t *testing.T) {
got, _ := layHeld(heldGateFacts(), []heldValue{{Pull: "p", Node: "g14", Module: "n8n", Key: "k", Value: []byte(`"x"`)}}, heldShelf)
s := got.Machines[1].Settings
if len(s) != 1 || s[0].Module != "n8n" || s[0].Values["k"] != "x" {
t.Fatalf("layers %+v", s)
}
}
// 1826/AC3: a held value for a machine or a module the snapshot or the change does not hold is said, never laid.
func TestAHeldValueForWhatIsNotHeldIsSaidNotLaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{
{Pull: "p", Node: "nowhere", Module: "n8n", Key: "k", Value: []byte("1")},
{Pull: "p", Node: "ace", Module: "ghost", Key: "k", Value: []byte("1")},
}, heldShelf)
if len(notes) != 2 || !strings.Contains(notes[0], "holds no machine") || !strings.Contains(notes[1], "defines no module ghost") {
t.Fatalf("said %q", notes)
}
if _, has := got.Machines[0].Settings[0].Values["k"]; has {
t.Fatal("laid")
}
}
// 1826/AC3: none held lays nothing and says nothing; the file is read and refuses a value without a node.
func TestNoHeldValuesLayNothingAndTheFileIsRead(t *testing.T) {
if got, notes := layHeld(heldGateFacts(), nil, heldShelf); len(notes) != 0 || len(got.Machines) != 2 {
t.Fatalf("%v %v", got, notes)
}
dir := t.TempDir()
good := filepath.Join(dir, "good.json")
os.WriteFile(good, []byte(`[{"pull":"p","node":"ace","module":"n8n","key":"k","value":1}]`), 0o644)
if h, err := readHeld(good); err != nil || len(h) != 1 || h[0].Key != "k" {
t.Fatalf("%v %v", h, err)
}
bad := filepath.Join(dir, "bad.json")
os.WriteFile(bad, []byte(`[{"pull":"p","module":"n8n","key":"k","value":1}]`), 0o644)
if _, err := readHeld(bad); err == nil {
t.Fatal("a value without a node was read")
}
if h, err := readHeld(""); h != nil || err != nil {
t.Fatalf("%v %v", h, err)
}
}
@@ -526,3 +526,57 @@ func TestAMemberJudgedAloneSeesAMoveThroughTheClonesBeside(t *testing.T) {
t.Errorf("nothing beside, yet moves %v", got)
}
}
// 1826/AC3 (integration level, a throwaway store): a change whose container says ${config:media-gid} fails the gate for
// want of the value, passes it with the value the operator approved held for its pull request, and the mesh as it is
// composes the same either way.
func TestAChangeNeedingAHeldValueFailsWithoutItAndPassesWithIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
trunk := `{"module":"n8n","version":"1","resources":[{"id":"server","type":"container","name":"n8n",
"image":"n8nio/n8n@sha256:4846eb2f4b874ab04cde7fc1e249d2ddaec66e9aea64439beb2972cfea88e3c0"}]}`
m, err := catalogue.ParseManifest([]byte(trunk))
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog",
Path: "modules/n8n", BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "n8n"); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
for i := range f.Machines {
f.Machines[i].Capabilities = append(f.Machines[i].Capabilities, snapshot.Capability{Name: "container-groups", Present: true})
}
tree := aTree(t, map[string]string{"n8n": n8nWithAGroup})
judge := func(held []heldValue) mergeVerdict {
v, err := judgeChange(t.Context(), mergeCheckInput{facts: f, admin: os.Getenv("MESH_TEST_POSTGRES"),
repository: "novox/mesh-catalog", tree: tree, changed: []string{"modules/n8n/module.json"}, held: held})
if err != nil {
t.Fatal(err)
}
return v
}
without := judge(nil)
if without.Verdict != "fail" || !strings.Contains(without.Report(), "media-gid") {
t.Fatalf("a required value nothing sets passed, or failed for another reason:\n%s", without.Report())
}
with := judge([]heldValue{{Pull: "novox/mesh-catalog#434", Node: "laptop", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}})
if with.Verdict == "fail" || with.Verdict == "error" {
t.Fatalf("the held value did not make it compose:\n%s", with.Report())
}
if !strings.Contains(with.Report(), "composed with 1 value(s) held for novox/mesh-catalog#434") {
t.Fatalf("the value laid is not said:\n%s", with.Report())
}
for i := range with.Machines {
if with.Machines[i].Base.Composes != without.Machines[i].Base.Composes {
t.Fatalf("the mesh as it is composes differently with a held value: %+v / %+v", with.Machines[i].Base, without.Machines[i].Base)
}
}
}
+10
View File
@@ -13,6 +13,7 @@ import (
"github.com/novox/mesh-controller/internal/conditions"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
@@ -55,9 +56,18 @@ func aMeshOnTheStore(t *testing.T) *stores {
t.Fatal(err)
}
t.Cleanup(open.Close)
// The work this mesh started in the background ends before its stores close (cleanups run last first).
t.Cleanup(background.Wait)
// A condition store of its own, held in memory (novox/hq to-be 45 §2): status leads with what is
// open, and a mesh with no bus would otherwise read as one whose conditions cannot be read.
withConditionsInMemory(t)
// The indexer holds no module unless a test puts one on its bus (aBuilderOnTheBus): an assignment's look at it
// reads "not held" on a mesh with no bus of its own.
wasIndexed := indexedModule
indexedModule = func(context.Context, string) (sdkdelivery.Module, bool, error) {
return sdkdelivery.Module{}, false, nil
}
t.Cleanup(func() { indexedModule = wasIndexed })
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
+26 -20
View File
@@ -52,23 +52,22 @@ type deliveryMerge struct {
Asker string `json:"asker,omitempty"`
}
// askerOf is the asker whose warrant the merge is read under: the one the announcement names, mesh-delivery when it
// names none.
func (d deliveryMerge) askerOf() string {
if d.Asker == "" {
return theDelivery
}
return d.Asker
}
// askerOf is the asker whose warrant the merge is read under: the one the announcement names. One that names none is
// from mesh-delivery of before the gate, whose warrant is no longer taken (novox/hq ADR 0450 §8; issue 1318).
func (d deliveryMerge) askerOf() string { return d.Asker }
// askerAccepted is whether the controller takes a warrant from this asker for a merge: the teller, and mesh-delivery
// until the gate takes the seat over (novox/hq ADR 0437 §6). Any other registers no move.
// askerAccepted is whether the controller takes a warrant from this asker for a merge: the teller alone, since the gate
// took the delivery seat over (novox/hq ADR 0437 §6, ADR 0450 §8; issue 1318). Any other registers no move.
func askerAccepted(asker string) error {
if asker == theDelivery || asker == sdkdelivery.TellerModule {
if asker == sdkdelivery.TellerModule {
return nil
}
return fmt.Errorf("the merge names the asker %q, and the controller takes a warrant only from %s and %s",
asker, sdkdelivery.TellerModule, theDelivery)
if asker == "" {
return fmt.Errorf("the merge names no asker, as mesh-delivery's of before the gate did, and the controller takes "+
"a warrant only from %s (ADR 0450 §8)", sdkdelivery.TellerModule)
}
return fmt.Errorf("the merge names the asker %q, and the controller takes a warrant only from %s (ADR 0450 §8)",
asker, sdkdelivery.TellerModule)
}
// parseDeliveryMerge reads an announcement, refusing one that names no pull request, head or ask.
@@ -265,25 +264,32 @@ var errNotAMerge = errors.New("the build is not of a merge")
// (ADR 0420 §7): the build is of a merge, that merge is a pull request mesh-delivery announced it made, and the
// ask it made it on holds an Approve by the operator of that head. Answers why it was accepted; an error says why
// it was not, and the refusal of ADR 0266 stands.
func approvedMove(ctx context.Context, inv registrations, forms *sourceForms, buildID string,
func approvedMove(ctx context.Context, inv registrations, forms *sourceForms, ask registrationAsk,
built inventory.Source) (string, error) {
r, found, err := inv.BuildRequestByID(ctx, buildID)
if err != nil {
return "", err
commit := ask.Merge
if ask.Build != "" {
r, found, err := inv.BuildRequestByID(ctx, ask.Build)
if err != nil {
return "", err
}
if !found || r.For != "merge" || r.Commit == "" {
return "", errNotAMerge
}
commit = r.Commit
}
if !found || r.For != "merge" || r.Commit == "" {
if commit == "" {
return "", errNotAMerge
}
owner, name, onForge := forms.onTheForge(built.Repository, built.Seat)
if !onForge {
return "", errors.New("the repository is not on the mesh's forge")
}
merged, known, err := inv.MergeOf(ctx, owner+"/"+name, r.Commit)
merged, known, err := inv.MergeOf(ctx, owner+"/"+name, commit)
if err != nil {
return "", err
}
if !known {
return "", fmt.Errorf("the controller kept no merge of %s/%s at %.8s", owner, name, r.Commit)
return "", fmt.Errorf("the controller kept no merge of %s/%s at %.8s", owner, name, commit)
}
return approvedAnnounced(ctx, inv, owner, name, announcedOf(merged))
}
+8 -7
View File
@@ -48,18 +48,19 @@ func TestAMergeNamingTheTellerIsMovedOnTheTellersWarrant(t *testing.T) {
}
}
// 1213/AC2: an announcement naming mesh-delivery, or none (today's module): read under mesh-delivery, registered.
func TestAMergeNamingNoAskerIsMovedOnMeshDeliverysWarrant(t *testing.T) {
// 1318 (ADR 0450 §8): a merge naming no asker, or the asker mesh-delivery of before the gate, registers no move; the
// teller's is taken, and its warrant read under its name.
func TestAMergeOfMeshDeliveryOrNoAskerMovesNothing(t *testing.T) {
for _, named := range []string{"", theDelivery} {
a := askedBy(t, named, theDelivery)
if _, err := movedOnApprove(a.merge, &a.ask, &a.warrant); err != nil {
t.Errorf("asker %q: refused: %v", named, err)
if _, err := movedOnApprove(a.merge, &a.ask, &a.warrant); err == nil || !strings.Contains(err.Error(), "ADR 0450 §8") {
t.Errorf("asker %q: %v", named, err)
}
}
was := routerAskOf
wasKept := keptDeliveryMerge
t.Cleanup(func() { routerAskOf, keptDeliveryMerge = was, wasKept })
a := askedBy(t, "", theDelivery)
a := askedBy(t, sdkdelivery.TellerModule, sdkdelivery.TellerModule)
keptDeliveryMerge = func(context.Context, string, string, int, string) (deliveryMerge, bool, error) {
return a.merge, true, nil
}
@@ -71,8 +72,8 @@ func TestAMergeNamingNoAskerIsMovedOnMeshDeliverysWarrant(t *testing.T) {
if _, err := approvedAnnounced(t.Context(), nil, "novox", "mesh-controller", announcedPull(7, a.merge.HeadSHA)); err != nil {
t.Fatal(err)
}
if readUnder != theDelivery {
t.Errorf("with no asker named the warrant was read under %q, not %q", readUnder, theDelivery)
if readUnder != sdkdelivery.TellerModule {
t.Errorf("the warrant was read under %q, not %q", readUnder, sdkdelivery.TellerModule)
}
}
+2 -1
View File
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"testing"
"time"
@@ -68,7 +69,7 @@ func TestAModuleBuiltFromANewRepositoryIsRegisteredOnlyOnTheOperatorsApproveOnTh
a.merge.Owner, a.merge.Repo, a.merge.Number = "novox", "mesh-host", 7
if c.answer {
record, _ := json.Marshal(map[string]any{"state": "closed", "ask": a.ask, "warrant": a.warrant})
if _, err := records.Put(ctx, theDelivery+"."+a.merge.AskID, record); err != nil {
if _, err := records.Put(ctx, sdkdelivery.TellerModule+"."+a.merge.AskID, record); err != nil {
t.Fatal(err)
}
}
+3 -2
View File
@@ -1,6 +1,7 @@
package main
import (
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"strings"
"testing"
"time"
@@ -21,7 +22,7 @@ func approvedMerge(t *testing.T) anApprovedMerge {
heads := map[string]string{"novox/mesh-controller@aaaaaaaaaaaa": "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"novox/mesh-catalog@bbbbbbbbbbbb": "b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2"}
m := deliveryMerge{Owner: "novox", Repo: "mesh-controller", Number: 7, HeadSHA: "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
Group: "g1", AskID: "merge-g1", Option: optionApprove, Heads: heads}
Group: "g1", AskID: "merge-g1", Option: optionApprove, Heads: heads, Asker: sdkdelivery.TellerModule}
bind := func(option string, h map[string]string) string {
d, err := asks.ActDigest(mergeAct(m.AskID, option, h))
if err != nil {
@@ -35,7 +36,7 @@ func approvedMerge(t *testing.T) anApprovedMerge {
Options: []asks.Option{
{ID: optionApprove, Label: "Approve", Does: "merges the group", Level: asks.Approve, Binds: bind(optionApprove, heads)},
{ID: "decline", Label: "Decline", Does: "leaves it", Level: asks.Approve, Binds: bind("decline", heads)}}}
w := asks.Warrant{Ask: m.AskID, Asker: theDelivery, Outcome: asks.OutcomeChosen, Option: optionApprove, Label: "Approve",
w := asks.Warrant{Ask: m.AskID, Asker: sdkdelivery.TellerModule, Outcome: asks.OutcomeChosen, Option: optionApprove, Label: "Approve",
Level: asks.Approve, By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "1", Verified: "user id"},
At: at, AskDigest: ask.Digest()}
return anApprovedMerge{merge: m, ask: ask, warrant: w}
@@ -0,0 +1,40 @@
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/conditions"
)
// novox/hq ADR 0505 §5: a delivery whose cure is its author's new head reaches the operator in words that name its
// author and say a new head is needed, offer no Stop and never call it the operator's; the line as mesh-delivery-gate
// says it (ready.go readyLines: needs "new-head", author, branch).
func TestADeliveryThatNeedsANewHeadNamesItsAuthorAndOffersNoStop(t *testing.T) {
l := stalledLine{ID: "novox/mesh-controller@499ff5c21861", Number: 432, State: "rejected", For: "2h1m0s", Bound: "2h0m0s",
H2: "none: a new head from its author, or a recheck that passes", Needs: needsNewHead,
Author: "the author of feat/quiet-deliveries (quiet-deliveries on shanks)", Branch: "feat/quiet-deliveries",
Says: "novox/mesh-controller#432 passed its checks at 499ff5c21861 and is not merged: a conflict"}
o := stalledObservations([]stalledLine{l})[0]
for _, words := range []string{o.Headline, o.Explanation, o.Resolved, o.Needs} {
if why, ok := conditions.Plain(words); !ok {
t.Errorf("%q is not plain: %s", words, why)
}
for _, never := range []string{"stop it", "operator's"} {
if strings.Contains(words, never) {
t.Errorf("%q says %q", words, never)
}
}
}
if o.Headline != "Pull request 432 of mesh-controller needs a new head from its author" ||
!strings.Contains(o.Explanation, "until the author of feat/quiet-deliveries (quiet-deliveries on shanks) pushes a new head") ||
!strings.Contains(o.Explanation, "for 2 hours") || len(o.Actions) != 0 ||
o.Needs != "ask the author of feat/quiet-deliveries (quiet-deliveries on shanks) for a new head." {
t.Fatalf("words: %q / %q / %q / %v", o.Headline, o.Explanation, o.Needs, o.Actions)
}
// A branch in code is not said; the author is then "its author".
l.Author = "the author of fix/a1b2c3d4e5f6"
if o := stalledObservations([]stalledLine{l})[0]; !strings.Contains(o.Needs, "ask its author") {
t.Fatalf("a dotted branch was said: %q", o.Needs)
}
}
@@ -116,7 +116,7 @@ func TestANewModuleIsRegisteredOnTheOperatorsApproveOfItsMergeAndRefusedOtherwis
if c.built != nil {
src = *c.built
}
id, err := mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-view", src, "build-1", "", "main")
id, err := mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-view", src, byBuild("build-1"), "", "main")
switch {
case c.accepted && err != nil:
t.Fatalf("%s: the Approved merge's new module was refused: %v", c.ac, err)
@@ -157,7 +157,7 @@ func TestASecondNewModuleFromARepositoryIsJudgedAlongsideAndAsksNoWarrant(t *tes
request: inventory.BuildRequest{ID: "build-2", Repository: "novox/mesh-delivery", Seat: "git", For: "build"}}}
forms := &sourceForms{bases: map[string]string{}, base: func(string) string { return "http://forge.internal:20000" }}
built := inventory.Source{Repository: "novox/mesh-delivery", Seat: "git"}
if id, err := mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-watchman", built, "build-2", "", "main"); err != nil || id == 0 {
if id, err := mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-watchman", built, byBuild("build-2"), "", "main"); err != nil || id == 0 {
t.Fatalf("1146/AC5: a second module from a repository the catalogue builds from was refused (id %d): %v", id, err)
}
}
@@ -1,37 +0,0 @@
package main
import (
"reflect"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// 1617/AC2 (unit): a new module whose first build a merge's hearing asked is not asked again when its batch is cut,
// and one whose ask failed, was never handed over or is long lost is.
func TestAModuleWhoseBuildIsComingIsNotAskedAgain(t *testing.T) {
now := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
source := buildSource{Repository: "http://forge/novox/mesh-catalog.git", Seat: "git"}
asked := func(path string, mod func(*inventory.RequestOutcome)) inventory.RequestOutcome {
r := inventory.RequestOutcome{BuildRequest: inventory.BuildRequest{ID: "b-" + path, Repository: source.Repository,
Seat: "git", Path: path, For: "merge", At: now.Add(-3 * time.Minute)}}
if mod != nil {
mod(&r)
}
return r
}
requests := []inventory.RequestOutcome{
asked("modules/go", nil),
asked("modules/failed", func(r *inventory.RequestOutcome) { r.Heard, r.Failed = true, "no module.json" }),
asked("modules/notasked", func(r *inventory.RequestOutcome) { r.NotAsked = "the bus is silent" }),
asked("modules/lost", func(r *inventory.RequestOutcome) { r.At = now.Add(-2 * time.Hour) }),
asked("modules/byhand", func(r *inventory.RequestOutcome) { r.For = "build" }),
}
got := unaskedDirs([]string{"modules/go", "modules/failed", "modules/notasked", "modules/lost", "modules/byhand",
"modules/nodejs"}, requests, source, now)
want := []string{"modules/failed", "modules/notasked", "modules/lost", "modules/byhand", "modules/nodejs"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("asked again %v, want %v", got, want)
}
}
+2 -1
View File
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"testing"
"time"
@@ -69,7 +70,7 @@ func TestANewModuleFromANewRepositoryIsRegisteredOnlyOnTheOperatorsApproveOnTheB
map[string]string{"novox/mesh-delivery@a1a1a1a1a1a1": "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1"})
if c.answer {
record, _ := json.Marshal(map[string]any{"state": "closed", "ask": a.ask, "warrant": a.warrant})
if _, err := records.Put(ctx, theDelivery+"."+a.merge.AskID, record); err != nil {
if _, err := records.Put(ctx, sdkdelivery.TellerModule+"."+a.merge.AskID, record); err != nil {
t.Fatal(err)
}
}
+66 -90
View File
@@ -4,134 +4,110 @@ package main
import (
"context"
"sync"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// asksWithPaths records every build a merge asks for, as repository, path and ref. The merge window's goroutine and
// the test's own may ask at once, so the record is written under a lock (issue 1543).
func asksWithPaths(t *testing.T) *[][3]string {
t.Helper()
var asked [][3]string
var mu sync.Mutex
was := askABuild
askABuild = func(_ context.Context, source buildSource, path, ref string) (string, error) {
mu.Lock()
defer mu.Unlock()
asked = append(asked, [3]string{source.Repository, path, ref})
return "b-" + path, nil
}
t.Cleanup(func() { askABuild = was })
return &asked
}
// The live case of 2026-10-07 (novox/hq issue 300), replayed: a merge of the catalogue adds one module's
// directory and touches nothing else the mesh holds. Its delivery plan said "builds new: …, sent nowhere";
// the merge said it changed nothing any module the mesh holds is built from, and built nothing, so the
// module was never registered and `assign` refused it. The merge asks for its build, at the branch merged
// into, from the repository as the mesh spells it — and opens no plan, since nothing it holds moved.
func TestAMergeBuildsTheNewModuleItAdds(t *testing.T) {
// ADR 0501 §6 (issue 1851, A5): the live case of 2026-10-07 (novox/hq issue 300), on keys. A merge of the catalogue adds
// one module's directory and touches nothing else the mesh holds: nothing asks its first build, it opens no plan, and
// the module is registered from the builder's first done key of it, assigned nowhere, with no build request written.
func TestAMergeAddingAModuleEndsRegisteredFromItsKeyAndAsksNoBuild(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksWithPaths(t)
noBuildAsked(t)
b := aBuilderOnTheBus(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "networkmanager", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/networkmanager", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "3da80a4b00",
watching, stop := context.WithCancel(context.Background())
t.Cleanup(stop)
watchNewModules(watching, open)
const commit = "3da80a4b00000000"
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
Paths: []string{"modules/systemd-resolved/module.json", "modules/systemd-resolved/cmd/main.go"},
ModuleDirs: []string{"modules/systemd-resolved"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
want := [3]string{"novox/mesh-catalog", "modules/systemd-resolved", "main"}
if len(*asked) != 1 || (*asked)[0] != want {
t.Fatalf("the merge adding modules/systemd-resolved asked for %v, not %v", *asked, want)
}
if plans, err := inv.OpenPlans(ctx); err != nil || len(plans) != 0 {
t.Fatalf("a merge moving nothing the mesh holds opened a plan: %+v %v", plans, err)
}
// The indexer writes the module's trunk value; the builder builds it, then its key is done.
fp := fpOf('7')
manifest := b.indexed(t, "systemd-resolved", "1", "novox/mesh-catalog", "modules/systemd-resolved", commit, fp)
b.built(t, "systemd-resolved", "1", fp, manifest, sdkdelivery.BuildBuilding)
time.Sleep(300 * time.Millisecond)
if _, known := inventoryHolds(t, inv, "systemd-resolved"); known {
t.Fatal("0501 §1: registered from a key still building")
}
if obs := newModuleObservations(time.Now().Add(time.Hour)); len(obs) != 1 || obs[0].Kind != kindNewModuleWaits {
t.Fatalf("0501 §6: the new module waiting on its key past its bound is named by %+v", obs)
}
b.built(t, "systemd-resolved", "1", fp, manifest, sdkdelivery.BuildDone)
becomes(t, 10*time.Second, "the new module registered", func() bool {
_, known := inventoryHolds(t, inv, "systemd-resolved")
return known
})
// The take-in forgets the wait just after the registration it made is seen.
becomes(t, 5*time.Second, "the registered module no longer named waiting", func() bool {
return len(newModuleObservations(time.Now().Add(time.Hour))) == 0
})
requests, err := inv.RequestsNamed(ctx, inventory.BuildRequest{Repository: "novox/mesh-catalog",
Path: "modules/systemd-resolved"}.Name())
if err != nil || len(requests) != 0 {
t.Fatalf("0501 §6: a build request was written: %+v (%v)", requests, err)
}
}
// A merge that changes a held module and adds a new one does both: the held one walks its plan, the new
// one is asked for beside it. A merge adding nothing builds nothing new.
func TestAMergeBuildsItsNewModuleBesideItsPlan(t *testing.T) {
// ADR 0501 §7: a new module whose manifest uses a word this controller does not read is named waiting at once, never
// failed, and registered from the same key, with no second build, when a controller that reads it looks.
func TestANewModuleWithAnUnreadWordWaitsForTheControllerThatReadsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksWithPaths(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", Ref: "main",
noBuildAsked(t)
b := aBuilderOnTheBus(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "networkmanager", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/networkmanager", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "c1aaaaaaaa",
Paths: []string{"modules/app/index.ts", "modules/fresh/module.json"},
ModuleDirs: []string{"modules/app", "modules/fresh"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
fp := fpOf('8')
raw := `{"module":"watchman","version":"1","not-a-word-yet":"5m","build":{"artifacts":[{"name":"a","kind":"archive","from":"files"}]},` +
`"resources":[{"id":"a","type":"archive","path":"/opt/watchman","artifact":"a"}]}`
digest := sdkdelivery.ManifestDigest([]byte(raw))
value, _ := sdkdelivery.EncodeManifest(sdkdelivery.Manifest{Module: "watchman", JSON: raw})
if _, err := b.manifests.Put(ctx, digest, value); err != nil {
t.Fatal(err)
}
paths := map[string]bool{}
for _, a := range *asked {
paths[a[1]] = true
}
if !paths["modules/fresh"] || !paths["modules/app"] || len(*asked) != 2 {
t.Fatalf("asked %v; want the new module and the plan's first tier", *asked)
}
plans, err := inv.OpenPlans(ctx)
if err != nil || len(plans) != 1 || plans[0].Modules["fresh"] != nil || plans[0].Modules["app"] == nil {
t.Fatalf("the plan should walk app alone: %+v %v", plans, err)
}
*asked = nil
m2 := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "c2aaaaaaaa",
Paths: []string{"README.md"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m2); err != nil {
trunk, _ := sdkdelivery.EncodeModule(sdkdelivery.Module{FP: "src1:" + fp[5:], FP2: fp, Manifest: digest,
Repository: "novox/mesh-catalog", Commit: "4444444400000000", Path: "modules/watchman"})
if _, err := b.modules.Put(ctx, "watchman", trunk); err != nil {
t.Fatal(err)
}
if len(*asked) != 0 {
t.Fatalf("a merge adding no module asked for %v", *asked)
b.built(t, "watchman", "1", fp, digest, sdkdelivery.BuildDone)
takeNewModules(ctx, open, "watchman")
if _, known := inventoryHolds(t, inv, "watchman"); known {
t.Fatal("0501 §7: a manifest with an unread word was registered")
}
}
// 1617/AC1 (integration): a merge adding a module to a repository the mesh builds modules from, heard while a batch of
// that repository is still assembling (its window running), asks the module's first build when it is heard, so the
// module is registered while the batch waits; cutting the batch asks it no second time (1617/AC2).
func TestAMergeAddingAModuleAsksItsBuildBeforeItsBatchIsCut(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/c1")
asked := asksWithPaths(t)
catalog := func(commit string, made time.Time, paths ...string) link.SourceMoved {
return link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
Head: "pr/" + commit, MergedAt: made.Format(time.RFC3339Nano), Paths: paths}
obs := newModuleObservations(time.Now())
if len(obs) != 1 || obs[0].Kind != kindNewModuleWaits || !strings.Contains(obs[0].Summary, `"not-a-word-yet"`) {
t.Fatalf("0501 §7: the module waiting for a reader of its word is named by %+v", obs)
}
hear(t, open, catalog("c1", t0, "modules/app/index.ts"), t0)
if len(*asked) != 0 {
t.Fatalf("a merge adding no module asked %v", *asked)
}
hear(t, open, lone(catalog("c2", t0.Add(10*time.Second), "modules/go/packages.json", "modules/go/module.json")), t0.Add(10*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("the batch should still be assembling: %d walk(s), %d batch(es)", len(ws), len(bs))
}
want := [3]string{"novox/mesh-catalog", "modules/go", "main"}
if len(*asked) != 1 || (*asked)[0] != want {
t.Fatalf("heard with its batch assembling, the merge asked %v, not %v", *asked, want)
}
cutAt(t, open, t0.Add(10*time.Second+90*time.Second))
n := 0
for _, a := range *asked {
if a[1] == "modules/go" {
n++
builds, _ := inv.Builds(ctx, "watchman", 5)
for _, b := range builds {
if !b.Worked() {
t.Fatalf("0501 §7: a failed build was recorded: %+v", b)
}
}
if n != 1 {
t.Fatalf("modules/go was asked %d times: %v", n, *asked)
}
}
+274
View File
@@ -0,0 +1,274 @@
package main
import (
"context"
"errors"
"fmt"
"slices"
"sort"
"strings"
"sync"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A new module is registered from its first done key (novox/hq ADR 0501 §1, §6; issue 1851, A5).
//
// Nothing asks a new module's first build: the builder builds every fingerprint the indexer writes, a new module's
// included (ADR 0429 §5). The controller registers a module the catalogue does not hold from the builder's done key of the
// fingerprint the indexer's trunk value names, under the same rules of who may register as before (ADR 0501 §3): from a
// repository the catalogue builds another module from, or on the operator's Approve of the merge the trunk value names
// (a first merge, kept with mesh-delivery's word), or at the terminal (ADR 0501 §4). It is assigned nowhere.
//
// **No timer** (ADR 0426): a write of the builder or the indexer wakes it for that module (builderkeys.Watch), a first
// merge's word wakes it for that repository, and a controller that starts looks at every module once (ADR 0501 §7: a
// manifest the controller before could not read is registered when the controller that reads it runs).
// newModulesMu keeps one take-in of new modules at a time in this process.
var newModulesMu sync.Mutex
// newModuleWait is a new module the controller could not register yet, and why: what the first-builds condition names.
type newModuleWait struct {
Module, Repository, Commit, Why string
// Word is the manifest word this controller does not read, when that is why (ADR 0501 §7).
Word string
Since time.Time
}
// newModulesWaiting are the new modules waiting, by module, in this process: a restart looks at every module again.
var newModulesWaiting = struct {
sync.Mutex
by map[string]newModuleWait
}{by: map[string]newModuleWait{}}
// takeNewModules looks at the modules the indexer holds that the catalogue does not, all of them or those named, and
// registers each whose done key may register it. Each outcome is said once: a module still waiting for the same reason
// is not said again.
func takeNewModules(ctx context.Context, open *stores, only ...string) {
newModulesMu.Lock()
defer newModulesMu.Unlock()
keys, closeKeys, err := openBuilderKeys()
if err != nil {
fmt.Printf("new modules: the builder's keys cannot be read: %v\n", err)
return
}
defer closeKeys()
for _, module := range takeNewModulesWith(ctx, open.inventory, keys, only...) {
// An assignment waiting for it is made now (ADR 0501 §6; issue 1851, A10).
for _, line := range settlePendingOnKey(ctx, open, module) {
fmt.Println(line)
}
}
}
// takeNewModulesWith is takeNewModules on the buckets handed in; it answers the modules it registered.
func takeNewModulesWith(ctx context.Context, inv *inventory.Inventory, keys newModuleReads, only ...string) []string {
entries, err := inv.Catalogued(ctx)
if err != nil {
fmt.Printf("new modules: the catalogue cannot be read: %v\n", err)
// empty-on-error: said above; nothing is registered this time, and the next write of a key looks again
return nil
}
held := map[string]bool{}
for _, e := range entries {
held[e.Manifest.Module] = true
}
values, unread, err := keys.Modules(ctx)
if err != nil {
fmt.Printf("new modules: the indexer's modules cannot be read: %v\n", err)
// empty-on-error: said above; nothing is registered this time, and the next write of a key looks again
return nil
}
for module, err := range unread {
if !held[module] && (len(only) == 0 || slices.Contains(only, module)) {
sayNewModuleWait(newModuleWait{Module: module, Why: "the indexer's value of it cannot be read: " + err.Error()})
}
}
names := make([]string, 0, len(values))
for module := range values {
if !held[module] && (len(only) == 0 || slices.Contains(only, module)) {
names = append(names, module)
}
}
sort.Strings(names)
var registered []string
for _, module := range names {
v := values[module]
_, _, err := takeInKey(ctx, inv, keys, keyTake{Module: module, Trunk: v, Ask: registrationAsk{Merge: v.Commit}})
switch {
case err == nil:
forgetNewModuleWait(module)
registered = append(registered, module)
fmt.Printf("%s is new in %s at %.8s: registered from the builder's done key, assigned nowhere\n", module,
v.Repository, v.Commit)
case errors.Is(err, errKeyWaits):
w := newModuleWait{Module: module, Repository: v.Repository, Commit: v.Commit, Why: err.Error()}
if word := wordIn(err.Error()); word != "" {
w.Word = word
}
sayNewModuleWait(w)
default:
sayNewModuleWait(newModuleWait{Module: module, Repository: v.Repository, Commit: v.Commit,
Why: "not registered: " + err.Error()})
}
}
return registered
}
// newModuleReads is what the take-in of new modules reads of the builder and the indexer.
type newModuleReads interface {
keyReads
Modules(ctx context.Context) (map[string]sdkdelivery.Module, map[string]error, error)
}
var _ newModuleReads = builderkeys.Reader{}
// wordIn is the manifest word a take-in's words say this controller does not read.
func wordIn(said string) string {
_, rest, ok := strings.Cut(said, "uses the word \"")
if !ok {
return ""
}
word, _, _ := strings.Cut(rest, "\"")
return word
}
// sayNewModuleWait keeps a new module waiting and says it, once for each reason.
func sayNewModuleWait(w newModuleWait) {
newModulesWaiting.Lock()
defer newModulesWaiting.Unlock()
was, known := newModulesWaiting.by[w.Module]
if known && was.Why == w.Why && was.Commit == w.Commit {
return
}
w.Since = time.Now().UTC()
if known && was.Commit == w.Commit {
w.Since = was.Since
}
newModulesWaiting.by[w.Module] = w
fmt.Printf("%s is new at %.8s and not registered yet: %s\n", w.Module, w.Commit, firstLine(w.Why))
}
func forgetNewModuleWait(module string) {
newModulesWaiting.Lock()
defer newModulesWaiting.Unlock()
delete(newModulesWaiting.by, module)
}
// kindNewModuleWaits is what a new module waiting for this controller's reader of its manifest, or waiting past its
// bound, raises (ADR 0501 §6, §7: the first-builds condition reads the key, not a build request).
const kindNewModuleWaits = "new-module-waits"
// newModuleWaitBound is how long a new module waits for its key before it is said: the builder builds a fingerprint in
// minutes. Not a delay: its key's write registers it the moment it is done.
const newModuleWaitBound = 30 * time.Minute
// newModuleObservations names the new modules waiting: one whose manifest uses a word this controller does not read at
// once (nothing is lost, but the operator sees what waits for which controller), any other past the bound.
func newModuleObservations(now time.Time) []conditions.Observation {
newModulesWaiting.Lock()
defer newModulesWaiting.Unlock()
var out []conditions.Observation
for _, w := range newModulesWaiting.by {
if w.Word == "" && now.Sub(w.Since) <= newModuleWaitBound {
continue
}
head := fmt.Sprintf("%s is not registered: %s", w.Module, firstLine(w.Why))
resolved := "its key is done and its take-in registers it"
if w.Word != "" {
head = fmt.Sprintf("%s waits for a controller that reads the manifest word %q", w.Module, w.Word)
resolved = "a controller that reads the word runs and registers it from the same key, with no second build"
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMerge,
ID: fmt.Sprintf("%s_%s", w.Module, short(w.Commit)), Kind: kindNewModuleWaits, Severity: conditions.Warning,
Summary: head,
Said: fmt.Sprintf("%s at %s waits since %s: %s", w.Module, w.Commit, w.Since.Format(time.RFC3339), w.Why),
Headline: fmt.Sprintf("The new module %s is not registered yet", w.Module),
Explanation: fmt.Sprintf("The merge %.8s of %s adds %s. The mesh registers it from the builder's done key of its "+
"fingerprint, and has not yet: %s", w.Commit, w.Repository, w.Module, w.Why),
Resolved: resolved,
})
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
// watchNewModules wakes the take-in of new modules for each module the builder or the indexer writes, until ctx ends, and
// looks at every module once now. A bucket not there yet is said: the look at start and a first merge's word still work.
func watchNewModules(ctx context.Context, open *stores) {
inBackground(func() {
answerHeldMerges(ctx, open.inventory)
takeNewModules(ctx, open)
})
keys, closeKeys, err := openBuilderKeys()
if err != nil {
fmt.Printf("new modules: the builder's keys are not watched: %v\n", err)
return
}
if err := builderkeys.Watch(ctx, keys.JS, func(module string) { wakeNewModule(ctx, open, module) }); err != nil {
closeKeys()
fmt.Printf("new modules: the builder's keys are not watched: %v\n", err)
return
}
go func() {
<-ctx.Done()
closeKeys()
}()
}
// wakeNewModule takes module in when the catalogue does not hold it: never on the watch's own goroutine, which must keep
// draining.
func wakeNewModule(ctx context.Context, open *stores, module string) {
inBackground(func() {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return
}
for _, e := range entries {
if e.Manifest.Module == module {
// Known: a plan waiting on its key is woken by the plans' own look.
askToLook()
return
}
}
takeNewModules(ctx, open, module)
})
}
// background is the work the take-ins of keys and of announcements start beside the event that woke them, so a test
// ends only once the work its mesh started has.
var background sync.WaitGroup
// inBackground runs work beside the caller, counted in background.
func inBackground(work func()) {
background.Add(1)
go func() {
defer background.Done()
work()
}()
}
// newModulesOf are the modules the indexer holds in a repository: what a first merge's word wakes.
func newModulesOf(ctx context.Context, repository string) ([]string, error) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
return nil, err
}
defer closeKeys()
values, _, err := keys.Modules(ctx)
if err != nil {
return nil, err
}
var out []string
for module, v := range values {
if strings.EqualFold(v.Repository, repository) {
out = append(out, module)
}
}
sort.Strings(out)
return out, nil
}
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"strings"
"testing"
"time"
)
// ADR 0501 §6, §7 (issue 1851, A5b): the condition of a new module reads its key's take-in: one waiting for a controller
// that reads its manifest's word is named at once, one waiting for its key only past the bound, each reason said once.
func TestANewModuleWaitingIsNamedByWhatItWaitsFor(t *testing.T) {
t.Cleanup(func() {
newModulesWaiting.Lock()
newModulesWaiting.by = map[string]newModuleWait{}
newModulesWaiting.Unlock()
})
word := `waits: watchman's manifest uses the word "key-ttl", which this controller does not read: it is registered when a controller that reads it runs`
sayNewModuleWait(newModuleWait{Module: "watchman", Commit: "c1", Why: word, Word: wordIn(word)})
sayNewModuleWait(newModuleWait{Module: "view", Commit: "c1", Why: "waits: the build of view at aaaa is building on ace"})
now := time.Now()
obs := newModuleObservations(now)
if len(obs) != 1 || !strings.Contains(obs[0].Summary, `"key-ttl"`) || obs[0].Kind != kindNewModuleWaits {
t.Fatalf("at once, only the unread word is named: %+v", obs)
}
if obs := newModuleObservations(now.Add(newModuleWaitBound + time.Minute)); len(obs) != 2 {
t.Fatalf("past the bound, both are named: %+v", obs)
}
forgetNewModuleWait("watchman")
forgetNewModuleWait("view")
if obs := newModuleObservations(now.Add(time.Hour)); len(obs) != 0 {
t.Fatalf("registered modules are still named: %+v", obs)
}
}
-24
View File
@@ -1,24 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/chain"
)
// 1146: a merge's ask that failed is kept under an id the bus takes as a key, for a module at the repository's root
// (".") as for one in a folder.
func TestTheIdOfABuildNotAskedIsAKeyTheBusTakes(t *testing.T) {
for dir, want := range map[string]string{
".": "not-asked-0177ea7a",
"modules/mesh-delivery-view": "not-asked-0177ea7a-modules-mesh-delivery-view",
} {
id := notAskedID("0177ea7a00000000", dir)
if id != want {
t.Errorf("notAskedID(%q) = %q, want %q", dir, id, want)
}
if !chain.ValidKey("req." + id) {
t.Errorf("notAskedID(%q) = %q: the bus takes no key req.%s", dir, id, id)
}
}
}
+6 -11
View File
@@ -109,8 +109,8 @@ func TestAFeatureWithANewSeatVerbIsOneBatchAndOnePlanInOrderWithNoAfterLine(t *t
}
// Issue 1049: a move delivered half before its second merge. The controller adds a module the catalogue removes, in one
// group: the merges are one batch and one plan in either order, the module is built once and from the controller's
// repository, and the catalogue's removal does not forget it.
// group: the merges are one batch and one plan in either order, no build of the module is asked (it is registered from
// the builder's key, ADR 0501 §6), and the catalogue's removal does not forget it.
func TestAMoveOfAModuleAcrossTwoRepositoriesIsOnePlanAndBuildsItOnceFromTheNewOne(t *testing.T) {
for name, controllerFirst := range map[string]bool{"controller_first": true, "catalogue_first": false} {
t.Run(name, func(t *testing.T) {
@@ -149,17 +149,12 @@ func TestAMoveOfAModuleAcrossTwoRepositoriesIsOnePlanAndBuildsItOnceFromTheNewOn
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a move: %d walk(s), %d batch(es); want exactly one plan", len(ws), len(bs))
}
var fromController int
// **No build of it is asked from either repository** (ADR 0501 §6, issue 1851): the move is registered from the
// builder's done key of route-proxy in the controller's repository, on the operator's Approve of the group.
for _, a := range *asked {
if strings.Contains(a[0], "mesh-controller") && a[1] == "modules/route-proxy" {
fromController++
if a[1] == "modules/route-proxy" {
t.Errorf("a build of route-proxy was asked: %v", a)
}
if strings.Contains(a[0], "mesh-catalog") && a[1] == "modules/route-proxy" {
t.Errorf("route-proxy was built from the repository it left: %v", a)
}
}
if fromController != 1 {
t.Fatalf("route-proxy was built %d time(s) from the controller's repository: %v", fromController, *asked)
}
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
-42
View File
@@ -28,48 +28,6 @@ func stoodOn(edges map[string][]string) map[string][]string {
return out
}
// A module built before the module it stands on is built against the old one and reports success
// (novox/hq 04-ISSUES/131). So bases come first, however the set arrived.
func TestBasesAreBuiltBeforeWhatStandsOnThem(t *testing.T) {
in := []inventory.Entry{entry("app"), entry("runtime"), entry("other"), entry("base")}
edges := stoodOn(map[string][]string{"app": {"runtime"}, "runtime": {"base"}})
got := orderByBases(in, edges)
pos := map[string]int{}
for i, e := range got {
pos[e.Manifest.Module] = i
}
if !(pos["base"] < pos["runtime"] && pos["runtime"] < pos["app"]) {
t.Fatalf("bases not first: %v", pos)
}
if len(got) != 4 {
t.Fatalf("an entry was lost or doubled: %d", len(got))
}
// A base outside the set is not waited for: it is not being rebuilt.
got = orderByBases([]inventory.Entry{entry("app")}, stoodOn(map[string][]string{"app": {"elsewhere"}}))
if len(got) != 1 {
t.Fatalf("a dependency outside the set changed the set: %v", got)
}
}
// A module registered from its manifest and never built still names its bases there; once built,
// the recorded edge is what says so. Both are read, and a module never stands on itself.
func TestWhatStandsOnAModuleIsReadFromItsBuildOrItsManifest(t *testing.T) {
built := entry("gitea")
edges := stoodOn(map[string][]string{"gitea": {"mesh-tools"}})
if !standsOnModule(built, "mesh-tools", edges) {
t.Fatal("a recorded edge was not read")
}
if standsOnModule(built, "gitea", edges) || standsOnModule(built, "postgres", edges) {
t.Fatal("an edge was invented")
}
fresh := inventory.Entry{Manifest: catalogue.Manifest{Module: "plex", Build: &catalogue.Build{
On: []catalogue.BuildsOn{{Arg: "RUNTIME_BASE", Module: "mesh-tools", Artifact: "runtime"}},
}}}
if !standsOnModule(fresh, "mesh-tools", nil) {
t.Fatal("a manifest's own base was not read")
}
}
// A merge names a repository the way the forge does; a source is recorded the way a build was
// asked for. The two meet on owner/repo and branch, whichever form the record took.
func TestAMergeMatchesTheSourcesBuiltFromIt(t *testing.T) {
+69
View File
@@ -0,0 +1,69 @@
package main
import (
"context"
"fmt"
)
// A new controller's own grants reach the bus before it writes (novox/hq issue 1800).
//
// A controller's grants are composed by the controller binary itself (broker.PermissionsFor), so the controller that
// walks a delivery cannot put its successor's grants in the bus's user list: the list that carries them was sent only
// once the new controller passed its gate on its first machine (carryUserList). Until then the new controller wrote
// buckets its account was not granted, and the bus refused them: plans-building on 2026-10-21, declared-packages on
// 2026-10-22, and R4e's states-announced would be the third.
//
// So a controller that starts, holding the lease and before it starts anything that writes its buckets, sends the
// machine holding the bus its user list when the list it composes is not the one that machine was last sent: the grants
// step's send (release.go grantsStep), every build there kept at the one it runs, refused unsent when the declaration
// would change more than the user list (errNotGrantsOnly). No build of a running plan reaches that machine before the
// plan's own tier sends it (ADR 0503 §8). A send that cannot be made is said once, and the controller serves on.
//
// It does not ask heldMachines, as carryUserList does: every reason that gives is a build that would move with the
// list (a policy, a plan not yet sent there, no gate passed), and here none can. The running plan that brought this
// controller holds the bus's machine until its tier sends it; asking would leave the list behind exactly when 1800
// meets it. The refusal unsent (errNotGrantsOnly) is the guard instead.
// ownGrantsCarrier is what the start's send reads and does: whether the bus's list is behind and on which machine, and
// the grants-only send to it. Variables so a test needs no store and no bus.
type ownGrantsCarrier struct {
behind func(ctx context.Context) (holder string, behind bool, err error)
send func(ctx context.Context, holder string) error
}
// carry sends the machine holding the bus the controller's own user list when it is behind, and says what it did.
func (c ownGrantsCarrier) carry(ctx context.Context) string {
holder, behind, err := c.behind(ctx)
switch {
case err != nil:
return fmt.Sprintf("whether the bus's user list carries this controller's grants cannot be read, so it is not sent "+
"at start; a bucket it newly writes may be refused until the gate carries it: %v", err)
case holder == "" || !behind:
return ""
}
if err := c.send(ctx, holder); err != nil {
return fmt.Sprintf("this controller's grants could not be sent to %s, which holds the bus, at its start; a bucket "+
"it newly writes may be refused until the gate carries them: %v", holder, err)
}
return fmt.Sprintf("sent %s, which holds the bus, this controller's user list alone, every build there kept, "+
"before it writes", holder)
}
// startWithOwnGrants carries the controller's own grants to the bus and only then starts what writes (then).
func startWithOwnGrants(ctx context.Context, c ownGrantsCarrier, then func()) {
if said := c.carry(ctx); said != "" {
fmt.Println(said)
}
then()
}
// ownGrantsOf is the carrier of the serving controller: the grants step's reading and its grants-only send.
func ownGrantsOf(open *stores) ownGrantsCarrier {
return ownGrantsCarrier{
behind: func(ctx context.Context) (string, bool, error) { return brokerBehindOf(ctx, open, nil) },
send: func(ctx context.Context, holder string) error {
_, err := sendRollout(keepingEveryBuild(withScope(ctx, sendScope{}), holder), open, []string{holder})
return err
},
}
}
+73
View File
@@ -0,0 +1,73 @@
package main
import (
"context"
"errors"
"strings"
"testing"
)
// 1800 (1): a controller starting with the bus's list behind its own sends the bus's machine once, before anything
// that writes its buckets starts.
func TestAStartingControllerSendsItsGrantsBeforeItWrites(t *testing.T) {
var order []string
c := ownGrantsCarrier{
behind: func(context.Context) (string, bool, error) { return "novox", true, nil },
send: func(_ context.Context, holder string) error { order = append(order, "send "+holder); return nil },
}
startWithOwnGrants(t.Context(), c, func() { order = append(order, "writers") })
if strings.Join(order, ", ") != "send novox, writers" {
t.Fatalf("the start did %v", order)
}
}
// 1800 (3), (4): no send when the list is current or there is no bus with a list; one that cannot be read or sent is
// said once, and the controller goes on.
func TestAStartingControllerSendsNothingWhenTheListIsCurrentAndGoesOnWhenRefused(t *testing.T) {
sends := 0
send := func(context.Context, string) error { sends++; return nil }
for _, c := range []ownGrantsCarrier{
{behind: func(context.Context) (string, bool, error) { return "novox", false, nil }, send: send},
{behind: func(context.Context) (string, bool, error) { return "", true, nil }, send: send},
} {
if said := c.carry(t.Context()); said != "" || sends != 0 {
t.Fatalf("a current list sent %d and said %q", sends, said)
}
}
refused := ownGrantsCarrier{behind: func(context.Context) (string, bool, error) { return "novox", true, nil },
send: func(context.Context, string) error { return errNotGrantsOnly }}
went := false
startWithOwnGrants(t.Context(), refused, func() { went = true })
if !went {
t.Fatal("a refused send stopped the controller")
}
if said := refused.carry(t.Context()); !strings.Contains(said, "could not be sent to novox") ||
!strings.Contains(said, errNotGrantsOnly.Error()) {
t.Fatalf("a refused send is said as %q", said)
}
unread := ownGrantsCarrier{behind: func(context.Context) (string, bool, error) { return "", false, errors.New("no store") },
send: send}
if said := unread.carry(t.Context()); !strings.Contains(said, "cannot be read") || sends != 0 {
t.Fatalf("an unread list: sent %d, said %q", sends, said)
}
}
// 1800 (2): the start's send is the grants step's: composed with every build of the bus's machine kept at the one it
// runs, so the declaration differs from its last only by the user list (and is refused unsent otherwise,
// errNotGrantsOnly, release.go), whatever a running plan registered since.
func TestAStartingControllersSendKeepsEveryBuildOfTheBusMachine(t *testing.T) {
was := sendRollout
t.Cleanup(func() { sendRollout = was })
var kept bool
var named []string
sendRollout = func(ctx context.Context, _ *stores, names []string) ([]string, error) {
kept, named = everyBuildKept(ctx, "novox"), names
return names, nil
}
if err := ownGrantsOf(nil).send(t.Context(), "novox"); err != nil {
t.Fatal(err)
}
if !kept || len(named) != 1 || named[0] != "novox" {
t.Fatalf("the start's send keeps every build: %v, names %v", kept, named)
}
}
+40
View File
@@ -0,0 +1,40 @@
package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// B1 (ADR 0499 §1, §5; issue 1758): a module whose every artifact names the paths it is built from is read from those
// paths and its manifest alone. mesh-sdk's TypeScript package at its repository's root: a change to its go/ touches no
// module, one to its src/ or its package.json touches it. A module with an artifact naming no paths reads its directory.
func TestAModuleNamingItsPathsIsReadFromThemAlone(t *testing.T) {
m, err := catalogue.ParseManifest([]byte(`{"module":"mesh-sdk","version":"1","build":{"artifacts":[{"name":"lib",
"kind":"package","language":"typescript","paths":["src","package.json","package-lock.json","tsconfig.json"]}]},
"resources":[]}`))
if err != nil {
t.Fatal(err)
}
own := pathsBuildSource(m, "")
read := inventory.WithBuildSources(nil, []inventory.BuildSource{{Paths: own}})
e := inventory.Entry{Manifest: m, Source: inventory.Source{Repository: "novox/mesh-sdk", Path: ""}}
for file, reads := range map[string]bool{
"go/delivery/builder.go": false, "go/go.mod": false, "README.md": false,
"src/index.ts": true, "src/deep/x.ts": true, "package.json": true, "module.json": true,
} {
if got := readsFile(e, read, file); got != reads {
t.Errorf("%s: read %v, want %v", file, got, reads)
}
}
// In a module's directory, the paths are under it.
if got := pathsBuildSource(m, "modules/sdk"); got[0] != "modules/sdk/module.json" || got[1] != "modules/sdk/src/**" {
t.Errorf("paths under a directory: %v", got)
}
// An artifact naming no paths: the whole directory, as before.
m.Build.Artifacts = append(m.Build.Artifacts, catalogue.Artifact{Name: "files", Kind: "archive", From: "files"})
if got := pathsBuildSource(m, ""); got != nil {
t.Errorf("a module with an artifact naming no paths is narrowed to %v", got)
}
}
+116 -22
View File
@@ -9,6 +9,9 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -85,8 +88,30 @@ const (
unknownModule = iota
buildInFlight
knownNotBuilt
// keyComing is a module the indexer holds and the catalogue does not yet: registered from the builder's done key
// of it (ADR 0501 §6), which nobody asks.
keyComing
)
// keyPendingPrefix opens the build a pending assignment waits for when it waits for a module's key: no build is asked
// (ADR 0501 §5, §6; issue 1851, A10), so it names the module's key, not a build id.
const keyPendingPrefix = "key:"
// indexedModule is the indexer's trunk value of a module the catalogue does not hold, found when it holds one. A
// variable so a test needs no bus.
var indexedModule = func(ctx context.Context, module string) (sdkdelivery.Module, bool, error) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
return sdkdelivery.Module{}, false, err
}
defer closeKeys()
v, err := keys.ModuleAt(ctx, module, "")
if errors.Is(err, builderkeys.ErrNoValue) {
return sdkdelivery.Module{}, false, nil
}
return v, err == nil, err
}
// notHeld is where a module the catalogue does not hold stands: its case, the build request it was read
// from, and the sentence that says it.
type notHeld struct {
@@ -96,7 +121,24 @@ type notHeld struct {
}
// whereIs looks for a module the catalogue does not hold among the build requests the controller keeps.
func whereIs(ctx context.Context, inv *inventory.Inventory, module string, now time.Time) (notHeld, error) {
// notHeldReads is what whereIs reads of the records (*inventory.Inventory, or a test's).
type notHeldReads interface {
RequestsNamed(ctx context.Context, name string) ([]inventory.RequestOutcome, error)
Catalogue(ctx context.Context) (map[string]catalogue.Manifest, error)
RequestedNames(ctx context.Context) ([]string, error)
}
func whereIs(ctx context.Context, inv notHeldReads, module string, now time.Time) (notHeld, error) {
// **Its key is coming** (ADR 0501 §6): the indexer holds it, and the controller registers it from the builder's done
// key of it. Read before the build requests of before R4e, which nothing writes any more.
if v, held, err := indexedModule(ctx, module); err != nil {
return notHeld{}, fmt.Errorf("whether the indexer holds %s cannot be read: %w", module, err)
} else if held {
return notHeld{kind: keyComing, request: inventory.RequestOutcome{BuildRequest: inventory.BuildRequest{
ID: keyPendingPrefix + module, Repository: v.Repository, Seat: gitSeat, Path: keyPath(v)}},
said: fmt.Sprintf("%s is not registered yet: the indexer holds it in %s at %.8s, and it is registered from "+
"the builder's done key of it, which no one asks", module, v.Repository, v.Commit)}, nil
}
requests, err := inv.RequestsNamed(ctx, module)
if err != nil {
return notHeld{}, err
@@ -169,6 +211,12 @@ func notRegistered(ctx context.Context, open *stores, node, module string, opts
return "", err
}
switch where.kind {
case keyComing:
lead := where.said
if opts.Build {
lead += "\n build \"true\" asks nothing: the builder builds every fingerprint the indexer writes (ADR 0501 §6)"
}
return keepPending(ctx, open, node, module, where.request, waiting, lead)
case buildInFlight:
lead := where.said
if opts.Build {
@@ -183,30 +231,20 @@ func notRegistered(ctx context.Context, open *stores, node, module string, opts
also = fmt.Sprintf("\n %s already has a pending assignment of %s, waiting for build %s; build "+
"\"true\" asks for a new build and makes it wait for that one", node, module, waiting.Build)
}
return "", fmt.Errorf("%w: %s%s\n assign it again with build \"true\" to ask for its build and keep this "+
"assignment until the build registers it; or `build` it (repository %s, path %s) and assign it "+
"once it is registered", inventory.ErrNoSuchModule, where.said, also, last.Repository, orRoot(last.Path))
return "", fmt.Errorf("%w: %s%s\n it is registered from the builder's done key once a merge adds it to %s "+
"(path %s) and the indexer holds it, or at the controller's terminal; assign it then",
inventory.ErrNoSuchModule, where.said, also, last.Repository, orRoot(last.Path))
}
if waiting != nil && waiting.State == inventory.PendingApplying {
return "", fmt.Errorf("%s is being assigned %s now; nothing was asked", node, module)
}
source := buildSource{Repository: last.Repository, Seat: last.Seat}
id, err := askABuild(ctx, source, last.Path, last.Ref)
if err != nil {
return "", fmt.Errorf("%s\n its build could not be asked for: %w; nothing was assigned", where.said, err)
}
asked := inventory.RequestOutcome{BuildRequest: inventory.BuildRequest{ID: id, Repository: last.Repository,
Seat: last.Seat, Path: last.Path, Ref: last.Ref, For: "assign", At: now.UTC()}}
recordBuildRequest(ctx, inv, asked.BuildRequest)
lead := fmt.Sprintf("%s\n build %s of %s is asked for now; %s can be assigned once it registers it",
where.said, id, requestSource(asked), module)
return keepPending(ctx, open, node, module, asked, waiting, lead)
// **No build is asked** (ADR 0501 §5, §6): a module the indexer does not hold has no key coming.
return "", fmt.Errorf("%w: %s\n build \"true\" asks no build: the controller asks none (ADR 0501 §5); a module "+
"is registered from the builder's done key once a merge adds it to a repository the catalogue builds from, or "+
"at the controller's terminal, and the indexer holds no %s", inventory.ErrNoSuchModule, where.said, module)
}
answer := strings.TrimPrefix(where.said, inventory.ErrNoSuchModule.Error()) +
"\n a module in a repository is built with `build` (its repository and path), then assigned"
"\n a module in a repository is registered from the builder's done key of it, then assigned"
if opts.Build {
answer += "\n build \"true\" asks for nothing here: the controller has no build request saying where a " +
"module of that name is"
answer += "\n build \"true\" asks for nothing: the controller asks no build (ADR 0501 §5), and the indexer " +
"holds no module of that name"
}
return "", fmt.Errorf("%w%s", inventory.ErrNoSuchModule, answer)
}
@@ -316,7 +354,7 @@ func clock(t time.Time) string { return t.Local().Format("2006-01-02 15:04:05 MS
// closestNames is up to three names near the one asked: registered modules and directories the controller
// asked to build.
func closestNames(ctx context.Context, inv *inventory.Inventory, name string) []string {
func closestNames(ctx context.Context, inv notHeldReads, name string) []string {
var candidates []string
if shelf, err := inv.Catalogue(ctx); err == nil {
for m := range shelf {
@@ -535,6 +573,19 @@ func settlePending(ctx context.Context, open *stores, now time.Time) []string {
}
continue
}
if strings.HasPrefix(p.Build, keyPendingPrefix) {
why, err := keyPendingEnded(ctx, p.Module)
if err != nil {
say("the key %s waits for cannot be read: %v", p.Module, err)
continue
}
if why != "" {
if line := expirePending(ctx, inv, p, why); line != "" {
said = append(said, line)
}
}
continue
}
requests, err := inv.RequestsNamed(ctx, p.Module)
if err != nil {
say("the build requests for %s could not be read: %v", p.Module, err)
@@ -744,3 +795,46 @@ func withdrawPending(ctx context.Context, inv *inventory.Inventory, node, module
}
return "", false, nil
}
// keyPendingEnded is why a pending assignment waiting for a module's key ends: the indexer no longer holds the module,
// or the builder's key of its fingerprint failed; empty while it is coming. A variable so a test needs no bus.
var keyPendingEnded = func(ctx context.Context, module string) (string, error) {
keys, closeKeys, err := openBuilderKeys()
if err != nil {
return "", err
}
defer closeKeys()
v, err := keys.ModuleAt(ctx, module, "")
if errors.Is(err, builderkeys.ErrNoValue) {
return fmt.Sprintf("the indexer no longer holds %s", module), nil
}
if err != nil || v.FP2 == "" {
return "", err
}
k, err := keys.Build(ctx, module, v.FP2)
if err != nil {
return "", err
}
if k.Failed() {
return k.Says(), nil
}
return "", nil
}
// settlePendingOnKey assigns, now, every pending assignment of a module its done key just registered (ADR 0501 §6):
// what an outcome of a build did for one waiting on that build.
func settlePendingOnKey(ctx context.Context, open *stores, module string) []string {
waiting, err := open.inventory.Pending(ctx, time.Time{})
if err != nil {
return []string{fmt.Sprintf("the pending assignments could not be read after %s was registered: %v", module, err)}
}
var said []string
for _, p := range waiting {
if p.Module == module {
if line := applyPending(ctx, open, p, keyPendingPrefix+module); line != "" {
said = append(said, line)
}
}
}
return said
}
+37 -150
View File
@@ -3,7 +3,6 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
@@ -13,6 +12,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -36,11 +36,13 @@ func aCatalogueMesh(t *testing.T) *stores {
return open
}
// mergeAdding keeps the build request a merge adding a module asked before R4e (issue 300): nothing asks one since (ADR
// 0501 §6), and the pending assignments still read the ones kept. These tests are of what an assignment does with one.
func mergeAdding(t *testing.T, open *stores, dir, commit string) {
t.Helper()
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit,
Paths: []string{dir + "/module.json"}, ModuleDirs: []string{dir}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(t.Context(), m); err != nil {
if err := open.inventory.RecordBuildRequest(t.Context(), inventory.BuildRequest{ID: "b-" + dir,
Repository: "novox/mesh-catalog", Seat: "git", Path: dir, Ref: "main", Commit: commit, For: "merge",
At: time.Now().UTC()}); err != nil {
t.Fatal(err)
}
}
@@ -175,7 +177,7 @@ func TestAPendingAssignmentExpiresWhenItsBuildFails(t *testing.T) {
t.Fatalf("a module whose build failed: %v", err)
}
for _, want := range []string{"sensors is known and not registered", "its last build, b-modules/sensors",
"failed: go build: undefined: sensorsRead", "build \"true\"", "repository novox/mesh-catalog, path modules/sensors"} {
"failed: go build: undefined: sensorsRead", "registered from the builder's done key", "novox/mesh-catalog (path modules/sensors)"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q:\n%v", want, err)
}
@@ -225,100 +227,10 @@ func TestTheConditionClearsWhenTheModuleIsAssignedLater(t *testing.T) {
}
}
func TestAssignWithBuildAsksForAKnownModule(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-old", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/sensors", Ref: "main", For: "build", At: time.Now().Add(-time.Hour)}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-old", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/sensors", On: "anchor", Failed: "no space left on device"}); err != nil {
t.Fatal(err)
}
asked := asksWithPaths(t)
said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if err != nil {
t.Fatal(err)
}
if len(*asked) != 1 || (*asked)[0] != [3]string{"novox/mesh-catalog", "modules/sensors", "main"} {
t.Fatalf("asked %v", *asked)
}
for _, want := range []string{"known and not registered", "no space left on device",
"build b-modules/sensors of novox/mesh-catalog@main (modules/sensors) is asked for now", "kept as pending"} {
if !strings.Contains(said, want) {
t.Fatalf("the answer does not say %q:\n%s", want, said)
}
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].Build != "b-modules/sensors" {
t.Fatalf("pending: %+v", pending)
}
requests, _ := inv.RequestsNamed(ctx, "sensors")
if len(requests) != 2 || requests[0].ID != "b-modules/sensors" || requests[0].For != "assign" {
t.Fatalf("the request is not kept as assign's: %+v", requests)
}
}
func TestAssignWithBuildRepointsTheWaitingAssignment(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-old", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/sensors", Ref: "main", For: "merge", At: time.Now().Add(-time.Hour)}); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-old", Repository: "novox/mesh-catalog", Ref: "main",
Path: "modules/sensors", On: "anchor", Failed: "boom"}); err != nil {
t.Fatal(err)
}
// Waiting on the failed build, its expiry not yet settled.
if _, err := inv.RecordPending(ctx, inventory.PendingAssignment{Node: "laptop", Module: "sensors",
Build: "build-old", Repository: "novox/mesh-catalog", Path: "modules/sensors"}); err != nil {
t.Fatal(err)
}
_, err := assign(ctx, open, "laptop", "sensors")
if err == nil || !strings.Contains(err.Error(), "already has a pending assignment of sensors, waiting for build build-old") {
t.Fatalf("known and not built with a pending assignment waiting: %v", err)
}
asked := asksWithPaths(t)
said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if err != nil || !strings.Contains(said, "which waited for build build-old, now waits for build b-modules/sensors") {
t.Fatalf("assign with build: %q %v", said, err)
}
if len(*asked) != 1 {
t.Fatalf("asked %v", *asked)
}
pending := pendingOf(t, open)
if len(pending) != 1 || pending[0].Build != "b-modules/sensors" || pending[0].State != inventory.PendingWaiting {
t.Fatalf("pending: %+v", pending)
}
// The new build in flight now: build "true" again asks nothing.
if said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors"); err != nil ||
!strings.Contains(said, "no second build was asked") || len(*asked) != 1 {
t.Fatalf("a second build true while the build runs: %q %v, asked %v", said, err, *asked)
}
}
func TestAFailedAskIsNotABuildInFlight(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", errors.New("cannot submit a build: nats: timeout")
}
t.Cleanup(func() { askABuild = was })
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
_, err := assign(ctx, open, "laptop", "sensors")
if err == nil || !strings.Contains(err.Error(), "the merge that added it (3da80a4b) could not ask for its build: cannot submit") {
t.Fatalf("a merge that could not ask: %v", err)
}
if len(pendingOf(t, open)) != 0 {
t.Fatal("a pending assignment waits on a build never asked")
}
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-waited", Repository: "novox/mesh-catalog",
Seat: "git", Path: "modules/gauges", Ref: "main", For: "build"}); err != nil {
t.Fatal(err)
@@ -326,7 +238,7 @@ func TestAFailedAskIsNotABuildInFlight(t *testing.T) {
if err := inv.MarkNotAsked(ctx, "build-waited", "cannot submit a build: no responders"); err != nil {
t.Fatal(err)
}
_, err = assign(ctx, open, "laptop", "gauges")
_, err := assign(ctx, open, "laptop", "gauges")
if err == nil || !strings.Contains(err.Error(), "build build-waited, asked by build, was not handed over") {
t.Fatalf("a build not handed over: %v", err)
}
@@ -395,7 +307,7 @@ func TestAnUnknownModuleIsRefusedWithTheClosestNames(t *testing.T) {
}
for _, want := range []string{"no module of that name: sensors", "the catalogue holds no module of that name",
"no build request the controller kept (the last 30 days)", "the closest names it holds: sensord",
"asks for nothing here"} {
"asks for nothing: the controller asks no build"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("the refusal does not say %q:\n%v", want, err)
}
@@ -565,59 +477,6 @@ func TestAPendingAssignmentNeverWaitsSilently(t *testing.T) {
}
}
func TestASecondAssignmentNotMadeKeepsItsCondition(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
asked := 0
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
asked++
return fmt.Sprintf("b-%d", asked), nil
}
t.Cleanup(func() { askABuild = was })
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatal(err)
}
failedBuild(t, open, "b-1")
settlePending(ctx, open, time.Now())
first := pendingOf(t, open)[0]
if !conditionOpen(t, first) {
t.Fatal("the first assignment not made raised nothing")
}
if _, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors"); err != nil {
t.Fatal(err)
}
failedBuild(t, open, "b-2")
settlePending(ctx, open, time.Now())
var second inventory.PendingAssignment
for _, p := range pendingOf(t, open) {
if p.Build == "b-2" {
second = p
}
}
if second.State != inventory.PendingExpired {
t.Fatalf("the second: %+v", second)
}
if !conditionOpen(t, second) {
t.Fatal("the second assignment's condition was cleared in the tick that raised it")
}
if !conditionOpen(t, rowOf(t, open, first.ID)) {
t.Fatal("the first was judged answered by a second that itself was not made")
}
// unassign takes both back, and the next tick clears both.
if said, err := unassign(ctx, open, "laptop", "sensors"); err != nil || !strings.Contains(said, "b-1") ||
!strings.Contains(said, "b-2") {
t.Fatalf("unassign: %q %v", said, err)
}
settlePending(ctx, open, time.Now())
if conditionOpen(t, first) || conditionOpen(t, second) {
t.Fatal("a condition stayed open after both were taken back")
}
}
func TestARaisedRowIsKeptUntilItsConditionClears(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
@@ -735,3 +594,31 @@ func TestABuildNoLongerWaitedForIsStillInFlight(t *testing.T) {
t.Fatalf("past the bound: %+v", rows)
}
}
// A10 (ADR 0501 §6; issue 1851): an assignment of a module the indexer holds and the catalogue does not yet waits for
// the builder's done key of it, asking nothing, and is made when that key registers the module.
func TestAnAssignmentWaitsForTheKeyAndAsksNothing(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
noBuildAsked(t)
b := aBuilderOnTheBus(t)
fp := fpOf('a')
manifest := b.indexed(t, "sensors", "1", "novox/mesh-catalog", "modules/sensors", "3da80a4b00aa0000", fp)
b.built(t, "sensors", "1", fp, manifest, sdkdelivery.BuildBuilding)
said, err := assignWith(ctx, open, "laptop", assignOptions{Build: true}, "sensors")
if err != nil || !strings.Contains(said, "registered from the builder's done key") || !strings.Contains(said, "kept as pending") {
t.Fatalf("A10: the assignment of a module whose key is coming: %q %v", said, err)
}
if p := pendingOf(t, open); len(p) != 1 || p[0].Build != keyPendingPrefix+"sensors" {
t.Fatalf("A10: pending %+v", p)
}
settlePending(ctx, open, time.Now().Add(time.Hour))
if p := pendingOf(t, open); p[0].State != inventory.PendingWaiting {
t.Fatalf("A10: a pending assignment waiting for a building key expired: %+v", p)
}
b.built(t, "sensors", "1", fp, manifest, sdkdelivery.BuildDone)
takeNewModules(ctx, open, "sensors")
if !slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatalf("A10: the key registered sensors and the pending assignment was not made: %+v", pendingOf(t, open))
}
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"context"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
)
// A10 (ADR 0501 §6; issue 1851): a module the catalogue does not hold and the indexer does is one whose key is coming:
// an assignment of it waits for that key, named by the module, with no build asked and no build request read.
func TestAModuleTheIndexerHoldsIsWaitedForByItsKey(t *testing.T) {
was := indexedModule
indexedModule = func(_ context.Context, module string) (sdkdelivery.Module, bool, error) {
return sdkdelivery.Module{Repository: "novox/mesh-catalog", Commit: "3da80a4b00aa", Path: "modules/" + module}, true, nil
}
t.Cleanup(func() { indexedModule = was })
where, err := whereIs(t.Context(), noRequests{}, "sensors", time.Now())
if err != nil {
t.Fatal(err)
}
if where.kind != keyComing || where.request.ID != keyPendingPrefix+"sensors" || where.request.Path != "modules/sensors" ||
!strings.Contains(where.said, "registered from the builder's done key") {
t.Fatalf("a module whose key is coming is read as %+v", where)
}
}
// noRequests is records with no build request and no module: what a mesh after R4e reads of a new module.
type noRequests struct{}
func (noRequests) RequestsNamed(context.Context, string) ([]inventory.RequestOutcome, error) {
return nil, nil
}
func (noRequests) Catalogue(context.Context) (map[string]catalogue.Manifest, error) { return nil, nil }
func (noRequests) RequestedNames(context.Context) ([]string, error) { return nil, nil }
+46
View File
@@ -195,6 +195,14 @@ var plainWordings = map[string]func(conditions.Observation) words{
"there yet is sealed by the controller until the vault holds it.", m, m),
Resolved: "Resolved: the vault gave " + m + " the core's credentials"}
}),
kindFilesNotPut: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "The big files for " + m + " cannot be put on the bus",
Needs: "look at the bus's permissions for the controller; the details give the reason.",
Explanation: fmt.Sprintf("The mesh could not put the large files of %s's instructions in their place on the bus, "+
"send after send, so %s is sent them inside the instructions instead, which may make them too big.", m, m),
Resolved: "Resolved: the big files for " + m + " reach the bus again"}
}),
kindDeclarationNearLimit: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
if o.Severity == conditions.Urgent {
@@ -449,6 +457,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "A delivery did not finish. Nothing is tried again until you say so.",
Resolved: "Resolved: the delivery is no longer failed"}
}),
kindMovesHeld: worded(func(o conditions.Observation) words {
return words{Headline: "An update waits for you",
Explanation: "A new version of a module would restart a container or touch its data, so it waits for you.",
Resolved: "Resolved: the update is no longer waiting"}
}),
kindImageMissing: worded(func(o conditions.Observation) words {
return words{Headline: "A module is not updated",
Explanation: "The new version of a module names a container image that the mesh's store does not hold, so it " +
"was not sent. What runs is untouched.",
Needs: "build the module again; the details name the image.",
Resolved: "Resolved: the store holds the image again"}
}),
"ask-lost": worded(func(o conditions.Observation) words {
return words{Headline: "A build has no result",
Explanation: "A build was asked for and never answered. It is asked again with the next delivery.",
@@ -946,6 +966,9 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
if l.State == "unanswered" {
return unansweredWords(l, o)
}
if l.Needs == needsNewHead {
return newHeadWords(l, name)
}
held := l.State
if held == "" {
held = "held"
@@ -979,6 +1002,29 @@ func stalledWords(l stalledLine, o conditions.Observation) (headline, explanatio
fmt.Sprintf("Delivery of %s is no longer %s", name, held), needs, actions
}
// newHeadWords are the words of a delivery whose cure is its author's new head (novox/hq ADR 0505 §5): it names the
// author and says a new head is needed; it offers no Stop and never calls it the operator's, because the operator
// hears of it only once the author's own watch has failed, after two hours with no new head.
func newHeadWords(l stalledLine, name string) (headline, explanation, resolved, needs string, actions []conditions.Action) {
pull := "A pull request of " + name
if l.Number > 0 {
pull = fmt.Sprintf("Pull request %d of %s", l.Number, name)
}
author := l.Author
if _, plain := conditions.Plain(author); !plain {
author = "its author" // a branch named in code (a dotted name, a number) is not said
}
long := "for a long time"
if d, err := time.ParseDuration(l.For); err == nil {
long = "for " + humanDuration(d)
}
return pull + " needs a new head from its author",
fmt.Sprintf("%s cannot merge until %s pushes a new head: its check failed after its Approve, and no new head "+
"came %s.", pull, author, long),
pull + " has a new head, or is closed",
"ask " + author + " for a new head.", nil
}
// stoppedNeeds is what ends a delivery that is already stopped, from the stalled line's healer words ("none: a
// merge that delivers its modules ends it — the operator's"): the part that says what ends it.
func stoppedNeeds(l stalledLine) string {
+1 -1
View File
@@ -260,7 +260,7 @@ func TestUpdatesHeldNameWhatWaitsAndWhereItIsReleased(t *testing.T) {
plainExample(t, got[0], "Updates of openrazer and sensors wait for your release",
"Needs you: release them from the mesh MCP server; this notification cannot do it. Updates of openrazer and "+
"sensors on g14 and shanks wait for a person to release them, because the last walk failed. "+
"They are not delivered until then, and stay held if you leave them.")
"They are not delivered until then, and stay held if you leave them.", "Release")
}
// **What held them is said in the glossary's words** (2026-10-08 review): the backlog is held after any
+47
View File
@@ -78,3 +78,50 @@ func TestAPlansLineCountsFromItsTierNotFromItsLastSave(t *testing.T) {
// novox/hq issue 296: a plan whose tier stands still is not written again by every advance — its revision,
// its last save and its `plan-moved` stay where its last change left them.
// novox/hq issue 920: a building plan whose current tier is all built waits on its note (a send refused at the
// gate), and its line, `status` and the stalled condition say that, not "building for ... LATE".
func TestAPlanLineSaysTheRefusalOnceItsTierIsBuilt(t *testing.T) {
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
built := now.Add(-time.Hour)
refusal := "tier 1: a build waits there for its gate: novox (mesh-delivery-indexer d68d7ef3→a5aa69a4) — tried again"
mk := func(second string) inventory.Plan {
return inventory.Plan{ID: "plan-r", Repository: "novox/mesh-sdk", Commit: "b2f5ebb2", State: inventory.PlanBuilding,
Tier: 1, Created: now.Add(-2 * time.Hour), Updated: now, TierEntered: now.Add(-time.Hour), Note: refusal,
Tiers: [][]string{{"a"}, {"b", "c"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &built},
"b": {State: "built", BuiltAt: &built}, "c": {State: second}}}
}
// 920/AC1
p := mk("built")
line := planLine(p, now)
if !strings.Contains(line, "a build waits there for its gate: novox") || strings.Contains(line, "building for") {
t.Errorf("920/AC1: a plan with its tier built and a refusal reads %q", line)
}
st := planStatuses([]inventory.Plan{p}, now, pauseView{}, nil)
if len(st) != 1 || !st[0].Late {
t.Errorf("920/AC1: status: %+v", st)
}
// 920/AC2
if line := planLine(mk("asked"), now); !strings.Contains(line, "building for 1h0m0s") {
t.Errorf("920/AC2: a plan with a tier module not built reads %q", line)
}
}
// 920/AC3: a tier move writes the previous tier's summary into the note; with the new tier all built, the line still
// says "building for", for that note is no refusal.
func TestAPlanLineDoesNotHoldOnATiersSummary(t *testing.T) {
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
built := now.Add(-time.Hour)
for _, note := range []string{"failed or refused its send (apply-failing says it): novox", "tier 0 built; sent mesh-sdk to novox first, in one send"} {
p := inventory.Plan{ID: "plan-s", Repository: "novox/mesh-sdk", Commit: "b2f5ebb2", State: inventory.PlanBuilding,
Tier: 1, Created: now.Add(-2 * time.Hour), Updated: now, TierEntered: now.Add(-time.Hour), Note: note,
Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &built}, "b": {State: "built", BuiltAt: &built}}}
if line := planLine(p, now); !strings.Contains(line, "building for 1h0m0s") || strings.Contains(line, "held") {
t.Errorf("920/AC3: a plan with the note %q reads %q", note, line)
}
}
}
+280
View File
@@ -0,0 +1,280 @@
package main
import (
"context"
"errors"
"fmt"
"sync"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/inventory"
)
// A walk takes its modules from the builder's done keys (novox/hq ADR 0503 §6, ADR 0501 §1, ADR 0498 §8; issue 1851, A6
// and A7).
//
// From R4e a walk asks build-agent for no module but the builder's own (ADR 0453 §5). A module of its tier is marked
// taken from a key (PlanModule.Key) and settled by the look: its fingerprint is read from the indexer's trunk value
// written for the commit the walk carries of its repository; the builder's key of that fingerprint, done with every
// artifact referenced, is taken in (recorded and registered, off the plans' hold) and the module is built; a key not done
// yet, or waiting for a reference, holds the tier and says so; a failed key fails the walk in the builder's own words.
// **No timer**: the builder's and the indexer's writes wake the look (builderkeys.Watch, watchNewModules).
// isBuildersOwn says a module is build-agent's own: asked of build-agent on a merge and taken from its build record, never
// from the builder's key, until R4g (ADR 0453 §5, ADR 0498 §7, ADR 0520 §1).
func isBuildersOwn(module string) bool { return buildersOwn(module) }
// buildAgentsOwn are those modules, named (ADR 0520 §1): the builder's; build-agent itself, whose image's context is another
// repository's; and mesh-tools, the toolchain image build-agent's builds stand on. The builder writes no key of any of them,
// and a walk waiting for one would hold its tier for ever. mesh-tools-go is not among them: it builds from a Dockerfile in
// its own directory on a pinned base, and has its key from mesh-build-image.
var buildAgentsOwn = map[string]bool{sdkdelivery.BuilderModule: true, builder.AgentModule: true, "mesh-tools": true}
// buildersOwn is isBuildersOwn's rule: a variable only so a test of the asks left (their bus waits, their plan key) may
// name modules of its own as build-agent's.
var buildersOwn = func(module string) bool { return buildAgentsOwn[module] }
// planKeyReads is what a walk reads of the indexer and the builder (builderkeys.Reader, or a test's).
type planKeyReads interface {
ModuleAt(ctx context.Context, module, commit string) (sdkdelivery.Module, error)
Build(ctx context.Context, module, fp string) (builderkeys.Key, error)
}
// openPlanKeys opens what a walk reads of the indexer and the builder. A variable so a test needs no bus.
var openPlanKeys = func() (planKeyReads, func(), error) {
r, closeKeys, err := openBuilderKeys()
if err != nil {
return nil, nil, err
}
return r, closeKeys, nil
}
// takeFromKey marks a module of a walk's tier as taken from the builder's key: asked of nobody (ADR 0501 §6).
func takeFromKey(p *inventory.Plan, state *inventory.PlanModule, e inventory.Entry, now time.Time) {
state.BusWaitFrom = nil
state.State = "asked"
state.AskedAt = &now
state.Key = true
state.Build, state.Fingerprint = "", ""
state.KeyAt = p.CommitOn(e.Source.Repository, followedBranch(e.Source.Ref))
state.Why = "waits for the builder's key"
state.Commit, state.BuiltAt, state.Made = "", nil, ""
fmt.Printf(" tier %d: %s is taken from the builder's key, asking no build\n", p.Tier, e.Manifest.Module)
}
// keyTakes are the take-ins of keys a look started and has not seen end, by key build id, and what each said when it
// waits: kept in this process, so a look starts each once and a settle says why it waits.
var keyTakes = struct {
sync.Mutex
running map[string]bool
waits map[string]string
// taken are the keys whose take-in ended registered, or found the module already at a later build (superseded).
taken map[string]bool
}{running: map[string]bool{}, waits: map[string]string{}, taken: map[string]bool{}}
// settleFromKeys settles every module of the tier taken from a key: built from its key's record, failed with the key's
// words or the take-in's refusal, or still waiting, saying for what. It says whether it changed the plan. It reads only
// the buckets and the records; a done key not recorded yet is taken in off the plans' hold, and its end asks the look.
func settleFromKeys(ctx context.Context, open *stores, keys planKeyReads, p *inventory.Plan, tier []string) (bool, error) {
return settleKeys(ctx, open.inventory, keys, p, tier, func(module string, trunk sdkdelivery.Module, id string) {
inBackground(func() { takeKeyOffTheHold(context.WithoutCancel(ctx), open, module, trunk, id) })
})
}
// keyRecords is what settling from keys reads of the records (*inventory.Inventory, or a test's).
type keyRecords interface {
BuildByID(ctx context.Context, id string) (inventory.Build, bool, error)
RefusedOf(ctx context.Context, ids []string) (map[string]string, error)
SourceOf(ctx context.Context, module string) (inventory.Source, error)
}
// settleKeys is settleFromKeys on the records handed in, starting a take-in with take.
func settleKeys(ctx context.Context, inv keyRecords, keys planKeyReads, p *inventory.Plan, tier []string,
take func(module string, trunk sdkdelivery.Module, id string)) (bool, error) {
changed := false
set := func(s *inventory.PlanModule, why string) {
if s.Why != why {
s.Why = why
changed = true
}
}
for _, m := range tier {
s := p.Modules[m]
if s == nil || !s.Key || s.State != "asked" {
continue
}
trunk, err := keys.ModuleAt(ctx, m, s.KeyAt)
if errors.Is(err, builderkeys.ErrNoValue) {
set(s, fmt.Sprintf("waits for the indexer's value of %s at %s", m, orNow(s.KeyAt)))
continue
}
if err != nil {
return changed, err
}
if trunk.FP2 == "" {
why := trunk.None2
if why == "" {
why = "the indexer wrote no fingerprint of the new form"
}
set(s, fmt.Sprintf("waits: %s at %s has no fingerprint to take a build by: %s", m, orNow(s.KeyAt), why))
continue
}
if s.Fingerprint != trunk.FP2 {
s.Fingerprint, changed = trunk.FP2, true
}
k, err := keys.Build(ctx, m, trunk.FP2)
if err != nil {
return changed, err
}
switch {
case k.Failed():
// **A failed key fails the walk in the builder's own words** (`digests differ:`, `not stored:`, `not pinned:`,
// `version taken:`), and a rebuild of it is a new revision the next walk takes.
s.State, s.Why = "failed", k.Build.Error
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s failed to build in tier %d: %s", m, p.Tier, firstLine(k.Build.Error))
changed = true
continue
case !k.Done() || len(k.Unreferenced()) > 0:
set(s, "waits: "+k.Says())
continue
}
id := keyBuildID(k)
refused, err := inv.RefusedOf(ctx, []string{id})
if err != nil {
return changed, fmt.Errorf("the refused registrations cannot be read: %w", err)
}
for kept, why := range refusalsKeptFor([]string{id}) {
refused[kept] = why
}
if why, ok := refused[id]; ok {
s.State, s.Why, s.Build = "failed", why, id
p.State = inventory.PlanFailed
p.Note = fmt.Sprintf("%s failed to build in tier %d: %s", m, p.Tier, firstLine(why))
changed = true
continue
}
b, found, err := inv.BuildByID(ctx, id)
if err != nil {
return changed, err
}
if found && b.Worked() {
keyTakes.Lock()
taken := keyTakes.taken[id]
keyTakes.Unlock()
registered := taken
if !registered {
if registered, err = registeredAt(ctx, inv, m, trunk.Commit); err != nil {
return changed, err
}
}
if registered {
at := b.At
s.State, s.BuiltAt, s.Commit, s.Build = "built", &at, trunk.Commit, id
s.Made = fmt.Sprintf("taken from the builder's key %s (revision %d), no build asked", short(trunk.FP2), k.Revision)
s.Why = ""
fmt.Printf("%s: %s taken from the builder's key %s\n", p.ID, m, short(trunk.FP2))
changed = true
continue
}
}
keyTakes.Lock()
running, waits := keyTakes.running[id], keyTakes.waits[id]
if !running {
keyTakes.running[id] = true
}
keyTakes.Unlock()
if waits != "" {
set(s, firstLine(waits))
} else {
set(s, "taking in the builder's done key "+short(trunk.FP2))
}
if !running {
take(m, trunk, id)
}
}
return changed, nil
}
// registeredAt says the catalogue holds the module registered at the commit the walk takes it at: its key's take-in
// registered it there, whether with the key's artifacts or, unchanged, with the build it stands on (no move). A key
// recorded at an earlier commit (the same fingerprint at an older merge) is taken in again, to register it here.
func registeredAt(ctx context.Context, inv keyRecords, module, commit string) (bool, error) {
src, err := inv.SourceOf(ctx, module)
if errors.Is(err, inventory.ErrNoSuchModule) {
return false, nil
}
if err != nil {
return false, err
}
return sameCommitID(src.BuiltFrom, commit), nil
}
func sameCommitID(a, b string) bool {
if a == "" || b == "" {
return false
}
if a == b {
return true
}
n := min(len(a), len(b))
return n >= 7 && a[:n] == b[:n]
}
func orNow(commit string) string {
if commit == "" {
return "its value now"
}
return short(commit)
}
// takeKeyOffTheHold takes a walk's module in from its done key, off the plans' hold: a registration may sweep the
// artifact store, and the plans are held by one look at a time. Its end asks the look, which settles the module from the
// record; a refusal is kept by the key's build id, so the look fails the walk with it; a wait is said by the look.
func takeKeyOffTheHold(ctx context.Context, open *stores, module string, trunk sdkdelivery.Module, id string) {
defer func() {
keyTakes.Lock()
delete(keyTakes.running, id)
keyTakes.Unlock()
askToLook()
}()
keys, closeKeys, err := openBuilderKeys()
if err != nil {
keepKeyWait(id, "waits: the builder's keys cannot be read: "+err.Error())
return
}
defer closeKeys()
_, _, err = takeInKey(ctx, open.inventory, keys, keyTake{Module: module, Trunk: trunk,
Ask: registrationAsk{Merge: trunk.Commit}})
switch {
case err == nil, errors.Is(err, inventory.ErrSuperseded):
// Superseded is no failure: the module is already at what a later merge's key built, which a walk that carries
// this merge is answered by, as a build-agent outcome superseded was.
keepKeyWait(id, "")
keyTakes.Lock()
keyTakes.taken[id] = true
keyTakes.Unlock()
case errors.Is(err, errKeyWaits):
keepKeyWait(id, err.Error())
default:
why := err.Error()
if rerr := open.inventory.RecordRefused(ctx, id, module, why); rerr != nil {
fmt.Printf("plans: %s's refusal could not be kept in the store, only in this controller: %v\n", module, rerr)
}
keepRefusalFor(id, why, time.Now())
}
}
func keepKeyWait(id, why string) {
keyTakes.Lock()
defer keyTakes.Unlock()
if why == "" {
delete(keyTakes.waits, id)
return
}
keyTakes.waits[id] = why
}
+220
View File
@@ -0,0 +1,220 @@
package main
import (
"context"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/builderkeys"
"github.com/novox/mesh-controller/internal/inventory"
)
// planKeysOf is the indexer and the builder as a walk reads them, for a test: values by module and commit, keys by
// module.
type planKeysOf struct {
values map[string]sdkdelivery.Module // by module + "@" + commit
keys map[string]builderkeys.Key
}
func (k planKeysOf) ModuleAt(_ context.Context, module, commit string) (sdkdelivery.Module, error) {
if v, ok := k.values[module+"@"+commit]; ok {
return v, nil
}
return sdkdelivery.Module{}, builderkeys.ErrNoValue
}
func (k planKeysOf) Build(_ context.Context, module, fp string) (builderkeys.Key, error) {
if key, ok := k.keys[module]; ok && key.Fingerprint == fp {
return key, nil
}
return builderkeys.Key{Module: module, Fingerprint: fp}, nil
}
// recordsOf is the build records and refusals a settle reads, for a test.
type recordsOf struct {
builds map[string]inventory.Build
refused map[string]string
sources map[string]inventory.Source
}
func (r recordsOf) BuildByID(_ context.Context, id string) (inventory.Build, bool, error) {
b, ok := r.builds[id]
return b, ok, nil
}
func (r recordsOf) RefusedOf(_ context.Context, ids []string) (map[string]string, error) {
out := map[string]string{}
for _, id := range ids {
if why, ok := r.refused[id]; ok {
out[id] = why
}
}
return out, nil
}
func (r recordsOf) SourceOf(_ context.Context, module string) (inventory.Source, error) {
if s, ok := r.sources[module]; ok {
return s, nil
}
return inventory.Source{}, inventory.ErrNoSuchModule
}
func aWalkOfOne(module string) (*inventory.Plan, map[string]inventory.Entry) {
p := &inventory.Plan{ID: "plan-1", State: inventory.PlanBuilding, Tiers: [][]string{{module}},
Modules: map[string]*inventory.PlanModule{}, Repository: "novox/mesh-catalog", Commit: "c0ffee0123456789",
Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: "c0ffee0123456789"}}}}
byName := map[string]inventory.Entry{module: {Source: inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + module, Ref: "main"}}}
return p, byName
}
// A6 (ADR 0503 §6, ADR 0498 check §7): a walk whose module has a done key goes asked to built with no build asked: the
// ask stub is never called, the key is taken in once, and `plans` says it was taken from the builder's key.
func TestAWalkTakesADoneKeyAndAsksNoBuild(t *testing.T) {
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
t.Fatal("a walk asked build-agent for a module that is not the builder's own")
return "", nil
}
t.Cleanup(func() { askABuild = was })
p, byName := aWalkOfOne("sudo")
askModule(t.Context(), p, "sudo", byName)
s := p.Modules["sudo"]
if s.State != "asked" || !s.Key || s.Build != "" || s.KeyAt != "c0ffee0123456789" {
t.Fatalf("the module is not taken from a key: %+v", s)
}
fp := "src2:" + strings.Repeat("a", 64)
k := builderkeys.Key{Module: "sudo", Fingerprint: fp, Found: true, Revision: 2, Build: sdkdelivery.BuilderBuild{
State: sdkdelivery.BuildDone, Manifest: strings.Repeat("b", 64), Artifacts: []sdkdelivery.BuiltArtifact{{Name: "a",
Digest: "sha256:1", Reference: "artifact-store://sudo/a/blobs/sha256:1"}}}}
keys := planKeysOf{values: map[string]sdkdelivery.Module{"sudo@c0ffee0123456789": {FP2: fp, Commit: "c0ffee0123456789"}},
keys: map[string]builderkeys.Key{"sudo": k}}
records := recordsOf{builds: map[string]inventory.Build{}, sources: map[string]inventory.Source{}}
var taken []string
take := func(m string, _ sdkdelivery.Module, id string) { taken = append(taken, m+" "+id) }
t.Cleanup(func() {
keyTakes.Lock()
keyTakes.running, keyTakes.waits, keyTakes.taken = map[string]bool{}, map[string]string{}, map[string]bool{}
keyTakes.Unlock()
})
changed, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take)
if err != nil || !changed || len(taken) != 1 || s.State != "asked" || s.Fingerprint != fp {
t.Fatalf("a done key not recorded yet is taken in once: %v %v %v %+v", changed, err, taken, s)
}
// A second look while the take-in runs starts no second one.
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take); err != nil || len(taken) != 1 {
t.Fatalf("a second look took the key in again: %v %v", taken, err)
}
// The take-in recorded and registered it: the look settles it built from the record.
id := keyBuildID(k)
records.builds[id] = inventory.Build{ID: id, Module: "sudo", Commit: "c0ffee0123456789", At: time.Now()}
records.sources["sudo"] = inventory.Source{BuiltFrom: "c0ffee0123456789"}
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take); err != nil {
t.Fatal(err)
}
if s.State != "built" || s.Build != id || !strings.Contains(s.Made, "taken from the builder's key") {
t.Fatalf("the module is not built from its key: %+v", s)
}
if takesOutcome(*p, &inventory.PlanModule{Key: true, State: "asked", AskedAt: s.AskedAt}, time.Now(), "build-1") {
t.Fatal("a build-agent outcome answers a module taken from a key")
}
}
// A7: a key building waits and says so, then moves on when done, with no timer and no ask; a failed key fails the walk
// in the builder's words; a value the indexer has not written yet waits, named.
func TestAWalkWaitsForAKeyAndFailsWithItsWords(t *testing.T) {
asksOfNone := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
t.Fatal("a walk asked a build")
return "", nil
}
t.Cleanup(func() { askABuild = asksOfNone })
t.Cleanup(func() {
keyTakes.Lock()
keyTakes.running, keyTakes.waits, keyTakes.taken = map[string]bool{}, map[string]string{}, map[string]bool{}
keyTakes.Unlock()
})
p, byName := aWalkOfOne("view")
askModule(t.Context(), p, "view", byName)
s := p.Modules["view"]
records := recordsOf{builds: map[string]inventory.Build{}}
take := func(string, sdkdelivery.Module, string) {}
keys := planKeysOf{values: map[string]sdkdelivery.Module{}, keys: map[string]builderkeys.Key{}}
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take); err != nil ||
!strings.Contains(s.Why, "waits for the indexer's value of view at c0ffee01") {
t.Fatalf("no indexer value yet: %q %v", s.Why, err)
}
fp := "src2:" + strings.Repeat("c", 64)
keys.values["view@c0ffee0123456789"] = sdkdelivery.Module{FP2: fp}
keys.keys["view"] = builderkeys.Key{Module: "view", Fingerprint: fp, Found: true,
Build: sdkdelivery.BuilderBuild{State: sdkdelivery.BuildBuilding, By: "g14"}}
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take); err != nil ||
s.State != "asked" || !strings.Contains(s.Why, "building on g14") || p.State != inventory.PlanBuilding {
t.Fatalf("a building key: %+v %v", s, err)
}
keys.keys["view"] = builderkeys.Key{Module: "view", Fingerprint: fp, Found: true,
Build: sdkdelivery.BuilderBuild{State: sdkdelivery.BuildFailed, Error: "not stored: tools: the store answered 500"}}
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], take); err != nil {
t.Fatal(err)
}
if s.State != "failed" || p.State != inventory.PlanFailed || !strings.Contains(p.Note, "not stored: tools") {
t.Fatalf("a failed key does not fail the walk in its words: %+v %q", s, p.Note)
}
}
// A7: a take-in refused (a module from another repository, a manifest refused) fails the walk with the refusal.
func TestARefusedTakeInFailsTheWalk(t *testing.T) {
t.Cleanup(func() {
keyTakes.Lock()
keyTakes.running, keyTakes.waits, keyTakes.taken = map[string]bool{}, map[string]string{}, map[string]bool{}
keyTakes.Unlock()
})
p, byName := aWalkOfOne("sudo")
askModule(t.Context(), p, "sudo", byName)
fp := "src2:" + strings.Repeat("d", 64)
k := builderkeys.Key{Module: "sudo", Fingerprint: fp, Found: true, Revision: 1, Build: sdkdelivery.BuilderBuild{
State: sdkdelivery.BuildDone, Manifest: strings.Repeat("b", 64), Artifacts: []sdkdelivery.BuiltArtifact{{Name: "a",
Digest: "sha256:1", Reference: "artifact-store://sudo/a/blobs/sha256:1"}}}}
keys := planKeysOf{values: map[string]sdkdelivery.Module{"sudo@c0ffee0123456789": {FP2: fp}},
keys: map[string]builderkeys.Key{"sudo": k}}
records := recordsOf{refused: map[string]string{keyBuildID(k): "sudo is built from novox/mesh-catalog, and this build is of agent/sudo"}}
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], func(string, sdkdelivery.Module, string) {}); err != nil {
t.Fatal(err)
}
if p.State != inventory.PlanFailed || !strings.Contains(p.Note, "agent/sudo") {
t.Fatalf("a refused take-in did not fail the walk: %s %q", p.State, p.Note)
}
}
// A6: a walk's module whose fingerprint did not change since an earlier merge has its key recorded at that merge's
// commit: the walk takes it in again, so the module is registered at the commit the walk carries, before it is built.
func TestAKeyRecordedAtAnEarlierCommitIsTakenInAgain(t *testing.T) {
t.Cleanup(func() {
keyTakes.Lock()
keyTakes.running, keyTakes.waits, keyTakes.taken = map[string]bool{}, map[string]string{}, map[string]bool{}
keyTakes.Unlock()
})
p, byName := aWalkOfOne("sudo")
askModule(t.Context(), p, "sudo", byName)
fp := "src2:" + strings.Repeat("e", 64)
k := builderkeys.Key{Module: "sudo", Fingerprint: fp, Found: true, Revision: 1, Build: sdkdelivery.BuilderBuild{
State: sdkdelivery.BuildDone, Manifest: strings.Repeat("b", 64), Artifacts: []sdkdelivery.BuiltArtifact{{Name: "a",
Digest: "sha256:1", Reference: "artifact-store://sudo/a/blobs/sha256:1"}}}}
keys := planKeysOf{values: map[string]sdkdelivery.Module{"sudo@c0ffee0123456789": {FP2: fp, Commit: "c0ffee0123456789"}},
keys: map[string]builderkeys.Key{"sudo": k}}
id := keyBuildID(k)
records := recordsOf{builds: map[string]inventory.Build{id: {ID: id, Module: "sudo", Commit: "0ld0ld0ld0ld"}},
sources: map[string]inventory.Source{"sudo": {BuiltFrom: "0ld0ld0ld0ld"}}}
var taken []string
if _, err := settleKeys(t.Context(), records, keys, p, p.Tiers[0], func(m string, _ sdkdelivery.Module, _ string) {
taken = append(taken, m)
}); err != nil {
t.Fatal(err)
}
if s := p.Modules["sudo"]; s.State != "asked" || len(taken) != 1 {
t.Fatalf("a key recorded at an earlier commit settled the walk at that commit: %+v, taken %v", s, taken)
}
}
+15 -3
View File
@@ -192,7 +192,7 @@ func startRolloutEvents(ctx context.Context, open *stores, js *broker.JetStream)
if err := startProvAppliedLooks(ctx, js); err != nil {
fmt.Printf("plans: the provisioner clients' words could not be watched (%v); a walk waits for them until the next look\n", err)
}
if err := startPlanMovedLooks(ctx, js); err != nil {
if err := startPlanMovedLooks(ctx, js, open); err != nil {
fmt.Printf("plans: the plans moved by other processes could not be followed (%v)\n", err)
}
done := startPlanLooks(ctx, open)
@@ -206,6 +206,7 @@ func servingPlanSaved(ctx context.Context, bus link.Bus, open *stores) func(*inv
return func(_ *inventory.Inventory, p inventory.Plan) {
sayPlanMoved(ctx, bus, open.inventory, p)
planKept(p)
keepBuildingKey(ctx, p)
if p.WalkSaid == inventory.WalkEnded {
go forgetChecksOfWalk(context.WithoutCancel(ctx), open.inventory, p)
}
@@ -562,7 +563,7 @@ func startProvAppliedLooks(ctx context.Context, js *broker.JetStream) error {
// `plan-moved`, so a plan a person moved with a verb at the control node's terminal — a process this one did not spawn,
// whose saves it does not hear — is looked at all the same (novox/hq ADR 0406 §2, §5, issue 974). New messages only; the
// first look reads the rest. A look its own save asks is the same look: asks are one.
func startPlanMovedLooks(ctx context.Context, js *broker.JetStream) error {
func startPlanMovedLooks(ctx context.Context, js *broker.JetStream, open *stores) error {
api, err := jetstream.New(js.Conn())
if err != nil {
return err
@@ -577,7 +578,18 @@ func startPlanMovedLooks(ctx context.Context, js *broker.JetStream) error {
if err != nil {
return fmt.Errorf("following the plans moved: %w", err)
}
consuming, err := consumer.Consume(func(jetstream.Msg) { askToLook() })
var bus link.Bus
if core, err := js.Conn().JetStream(); err == nil {
bus = link.OverNATS{Conn: js.Conn(), JS: core}
}
consuming, err := consumer.Consume(func(m jetstream.Msg) {
askToLook()
if bus != nil && open != nil {
// A walk's start or end, said by this controller or a verb: said again with the machines whose
// declaration it changes (novox/hq ADR 0505 §2, walk_declared.go).
go sayDeclared(context.WithoutCancel(ctx), bus, open, m.Data())
}
})
if err != nil {
return fmt.Errorf("following the plans moved: %w", err)
}
@@ -174,7 +174,7 @@ func TestAPlanMovedByAnotherProcessAsksForALook(t *testing.T) {
ctx, stop := context.WithCancel(context.Background())
defer stop()
go l.run(ctx)
if err := startPlanMovedLooks(ctx, js); err != nil {
if err := startPlanMovedLooks(ctx, js, nil); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
@@ -31,7 +31,8 @@ func TestNoPlanIsWrittenButThroughSavePlansMoving(t *testing.T) {
return err
}
text := string(raw)
if write.MatchString(text) {
// The regular expression is the costly part of reading every source: only a file that names the table can match.
if strings.Contains(strings.ToLower(text), "release_plan") && write.MatchString(text) {
found = append(found, path+" writes the plan table")
}
// A plan record is written by the bucket's raw create and update, which only the plans' own file calls.

Some files were not shown because too many files have changed in this diff Show More