Compare commits

..
Author SHA1 Message Date
jschoubben 28a278c146 Take mesh-sdk at bd7731f: a machine that does not resolve keeps its name and its address in the mesh as well as its tunnel (issue 1833)
mesh/delivery-group group fix/1833-one-machine-alone checking: 0 of 2 member(s) ready
mesh/repo-check head 28a278c1462a merged into trunk 8f2d22db0caa — pass: its merge-check.sh passed
mesh/delivery checking
mesh/merge-gate building on ace since 16:03
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 28a278c1462a on b8f77577fed2 | unit 3353 run, 0 kept, 7 skipped | integration…
2026-10-22 15:27:27 +02:00
jschoubben c34ab0e43a Take mesh-sdk at c154bde: a machine that does not resolve keeps its place on the private network, so it alone is refused (issue 1833)
One machine whose modules could not be resolved left the private network uncomposable when it was the hub, and every
send to every machine was held. The SDK now keeps a refused machine at the place the controller already holds for it.
2026-10-22 15:04:47 +02:00
mesh-tools 8f2d22db0c Merge pull request 'Move route-proxy's program into its module and run it on the pinned alpine image (issues 1633, 1634, 1635, ADR 0480 §1-§4, §11)' (#433) from feat/r4d into main 2026-10-22 12:19:07 +00:00
mesh-tools aa9da94260 Merge pull request 'A build node leaves its builds' images behind no more, and takes no work under a floor of free space (issue 1834)' (#435) from fix/1834-build-node-disk into main 2026-10-22 12:05:14 +00:00
jschoubben 01f615aa0a Read the bus release from the nats module's upstream artifact, so the tests' bus is still held to the mesh's once the Dockerfile is gone (issue 1640)
mesh/merge-gate head 01f615aa0ad6 merged into trunk aa9da942606d — pass: group feat/r4d composed: every machine composes with the change as it did without …
mesh/delivery delivered
mesh/repo-check head 01f615aa0ad6 merged into trunk aa9da942606d — pass: its merge-check.sh passed
mesh/delivery-group group feat/r4d delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 01f615aa0ad6 on aa9da942606d | unit 3345 run, 0 kept, 7 skipped | integration…
mesh-catalog #434 runs the nats server as the upstream image named in module.json (ADR 0497) and
deletes modules/nats/Dockerfile, so the check failed opening it. The manifest's upstream nats
artifact is read first; the Dockerfile only for a catalogue from before the move (the captured copy).
2026-10-22 13:38:00 +02:00
jschoubben f501eaca43 Remove an image's pushed names with its build tag, and take no work under the free-space floor (issue 1834)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@01f615aa0ad6 (merged as 8f2d22db into main, walk plan-17926707…
mesh/merge-gate head f501eaca4305 merged into trunk 4463991e0609 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f501eaca4305 merged into trunk 4463991e0609 — pass: its merge-check.sh passed
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The registry push tags the image a second time, so removing only the build tag freed nothing.
The held image a comparison pulls goes after it is read, and a node under the floor leaves the ask
to another build node instead of taking it and failing.
2026-10-22 13:19:58 +02:00
jschoubben 995f8237ce Free a build node's disk after every build and refuse work under a floor of free space (issue 1834)
A full / made every check on shanks end 'store never became ready' with the reason erased by --rm.
Builds now remove their local image tag, narrowed source and context clone; a build or check
refuses under 20 GiB free on / and the workspace, saying so; the throwaway store is no longer
started with --rm and a store that does not come up reports its exit code and last 20 log lines.
2026-10-22 13:14:00 +02:00
jschoubben 707bfba4ea Check an image's Dockerfile place instead of refusing the image kind (ADR 0487 clause 13)
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 707bfba4ea36 merged into trunk 4463991e0609 beside mesh-catalog#434@68183ee4 — fail: its merge-check.sh failed: --- FAIL: TestTheBusTh…
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
2026-10-22 11:45:06 +02:00
mesh-tools 4463991e06 Merge pull request 'Compose a pull request's check with the config values held for it (issue 1826, ADR 0502)' (#434) from fix/1826-held-config into main 2026-10-22 09:38:35 +00:00
jschoubben 4357b3688f Tell mesh-config what a head declares at every head, so a head that reverts a key drops its held value (issue 1826)
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
mesh/repo-check head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: its merge-check.sh passed
mesh/merge-gate head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: group fix/1826-held-config composed: every machine composes with the change as it …
mesh/delivery delivered
mesh/delivery-group group fix/1826-held-config delivered: every member is delivered
2026-10-22 11:07:34 +02:00
jschoubben 03927a6f6c Carry a config value held for a pull request's new key into its composed check (issue 1826)
The controller tells mesh-config what a pull request declares, asks the approved held values with each check, and the build agent and merge gate lay them over the change's composition only: a key the trunk does not declare yet can then be approved before the merge (ADR 0502).
2026-10-22 11:01:12 +02:00
jschoubben 499ff5c218 Move the route proxy into its module and run it from the trust image, on mesh-sdk's public membership (issues 1633, 1634, 1635)
mesh/repo-check out of date: judged beside mesh-catalog#434 at c948f3a61c74, it is now at 7451e7edf4e5; judged again on a recheck or when it is merged
mesh/merge-gate group feat/r4d: composed again for the group's new heads
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
A bundle is built only from its module's directory (ADR 0480 clause 4), so the proxy leaves
examples/ and the controller's internal/broker; the broker keeps no second copy of what the
proxy shares with mesh-sdk. The image and its Dockerfile go, and a test holds that only
build-agent declares one (clause 11).
2026-10-22 09:53:56 +02:00
mesh-tools 3d4ad6fefb Merge pull request 'Limit what a build or check starts on a build node: a per-node build slice (issue 1798)' (#432) from fix/1798-limit-build-resources into main 2026-10-21 23:20:57 +00:00
jschoubben 6bf38072ca Say in the build-agent settings that changing one restarts the build slice (issue 1798)
mesh/merge-gate head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 01:01:56 +02:00
jschoubben b174d743c0 Name the build slice without a dash so it sits beside user.slice, and restart it when its limits change (issue 1798)
mesh/merge-gate building on ace since 00:57
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:56:48 +02:00
jschoubben 37119ea31f Fill the build slice's limits as the mesh fills a module's template today (issue 1798)
mesh/merge-gate building on ace since 00:53
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:38:02 +02:00
jschoubben af2c6b5e94 Run builds and checks in a per-node build slice with CPU, IO and memory limits (issue 1798) 2026-10-22 00:24:45 +02:00
mesh-tools a929df9b0e Merge pull request 'The checker runs beside today's check: checks wait while a delivery builds; a group's SDK member merges; a flaky test fixed' (#430) from feat/the-checker into main 2026-10-21 21:46:26 +00:00
mesh-tools a3c965b9a0 Merge pull request 'Remove a check's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)' (#431) from fix/1772-check-volumes into main 2026-10-21 21:15:07 +00:00
jschoubben b9ec1919b8 Name plans-building among the controller's readable states and pin the SDK that does (issue 1687, ADR 0483 clause 2)
mesh/repo-check head b9ec1919b865 merged into trunk a3c965b9a0d5 beside mesh-catalog#423@12ff1cfb — pass: its merge-check.sh passed
mesh/merge-gate head b9ec1919b865 merged into trunk a3c965b9a0d5 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker delivered: every member is delivered
2026-10-21 22:47:04 +02:00
jschoubben d491356f59 Remove a build's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)
mesh/merge-gate head d491356f59dd merged into trunk 1f3fdf311167 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head d491356f59dd merged into trunk 1f3fdf311167 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 22:40:16 +02:00
jschoubben 47e1819101 Merge remote-tracking branch 'origin/fix/1106-sdk-member-merges' into feat/the-checker
mesh/repo-check out of date: judged beside mesh-sdk#66 at 78364e318bdc, it is now at 580da61a9107; judged again on a recheck or when it is merged
mesh/delivery-group group feat/the-checker checking: 4 of 4 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 47e1819101a1 merged into trunk 1f3fdf311167 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
2026-10-21 22:01:42 +02:00
jschoubben 0e25ed7c5a Merge remote-tracking branch 'origin/fix/1399-person-word-test' into feat/the-checker 2026-10-21 22:01:42 +02:00
jschoubben 79e2923dc8 Merge remote-tracking branch 'origin/fix/1652-checks-wait' into feat/the-checker
# Conflicts:
#	internal/broker/testdata/composed.conf
#	internal/link/builds_nats.go
2026-10-21 22:01:42 +02:00
mesh-tools 1f3fdf3111 Merge pull request 'Judge a duplicate hand-out before marking a build in hand, and never hand every build back over a stale mark (issue 1768)' (#429) from fix/1768-in-hand-marker into main 2026-10-21 19:59:30 +00:00
jschoubben 9902a3f16c Name every group member's trunk in the composed verdict, the announced one when the seat names none (issue 1106) 2026-10-21 21:57:30 +02:00
jschoubben 11815c0fb3 Have the delivery test's view hear a plan a verb let go, so a loaded machine cannot read it still waiting (issue 1399) 2026-10-21 21:45:58 +02:00
mesh-tools f973cefd7f Merge pull request 'Ask the operator for a held move and a failed release, and send no image the store lacks (issues 1732, 1320, 1742)' (#428) from fix/1732-held-moves-asked into main 2026-10-21 19:07:51 +00:00
jschoubben 696300b9e2 Judge a duplicate hand-out before the in-hand claim, and treat a stale mark as stale (issue 1768)
mesh/merge-gate head 696300b9e280 merged into trunk f973cefd7fd3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 696300b9e280 merged into trunk f973cefd7fd3 — pass: its merge-check.sh passed
mesh/delivery delivered
A dropped duplicate left the in-hand marker, so every later claim failed and every ask spent its deliveries.
2026-10-21 20:57:12 +02:00
jschoubben e7cc315fd0 Guard the taking loop's subscriptions, and update the two tests that held a busy machine's pulls
The loop now lets go of and binds queues around every ask, so Close takes the lock the loop does;
the busy machine's test says it holds no pull, and the stray test keeps its long check alive
within its two-second ack wait.
2026-10-21 20:53:58 +02:00
mesh-tools 98155e327b Merge pull request 'Take mesh-sdk at 07a6592: an item's text places a config value (issue 1658), and fit the unit level in its bound (issue 1763)' (#420) from fix/1656-xterm-config into main 2026-10-21 18:36:39 +00:00
jschoubben 60253c82b7 Fix the review of the checks-wait milestone: redeliveries, deleted modules, the kept queue
A check pending redelivery is read before the let-through fetches (none is handed back);
a deleted module is done for the plan's key; the queue an ask came from is let go of when
it ends. Tries say they run under their own id.
2026-10-21 20:17:37 +02:00
jschoubben d6db334840 Offer no Release on a machine that also holds a person's own hold (issue 1732, review round 1)
mesh/repo-check head d6db334840a5 merged into trunk 98155e327b6d beside mesh-host#129@09c77d57 — pass: its merge-check.sh passed
mesh/merge-gate head d6db334840a5 merged into trunk 98155e327b6d — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/delivery delivered
mesh/delivery-group group fix/1732-held-moves-asked delivered: every member is delivered
2026-10-21 20:15:58 +02:00
jschoubben 69d016bfdf Merge remote-tracking branch 'origin/main' into fix/1732-held-moves-asked 2026-10-21 20:15:13 +02:00
jschoubben bb8f076cc0 Merge remote-tracking branch 'origin/main' into fix/1652-checks-wait 2026-10-21 20:14:12 +02:00
jschoubben a58b38cee3 Keep the missing image out of the condition's summary (issue 1742) 2026-10-21 20:06:10 +02:00
jschoubben 9d260412b3 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/repo-check head 9d260412b330 merged into trunk 44033f4b6d72 beside mesh-catalog#416@b9dcf19c — pass: its merge-check.sh passed
mesh/merge-gate head 9d260412b330 merged into trunk 44033f4b6d72 — pass: group fix/1656-xterm-config composed: every machine composes with the change as it…
mesh/delivery-group group fix/1656-xterm-config failed: a member failed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792609673527907956 (novox/mesh-controller at f973cefd into ma…
2026-10-21 19:59:59 +02:00
jschoubben 99610e9569 Cut the controller's slowest unit tests (a source scan that regexed every file, resolver stand-in timeouts, a wait bound) so mesh-controller's unit level fits its bound (issue 1763) 2026-10-21 19:59:58 +02:00
jschoubben bb4d2e1da4 Let go of the other queues while an ask runs, and read the let-through race right
A message that reaches a builder's subscription outside a fetch was held unheartbeated for
the whole run of another ask, behind which a delivery waited (issue 1747). The queues the
running ask did not come from are let go of and bound again after it.
2026-10-21 19:56:23 +02:00
mesh-tools 44033f4b6d Merge pull request 'Issue a bus process's account when its module is assigned or gains it, and warn of one still missing instead of composing it silently (issue 1226)' (#427) from fix/1226-issue-process-accounts into main 2026-10-21 17:53:47 +00:00
jschoubben dc06af1f5a Leave a module out of a send when the store lacks its image, and say it (issue 1742) 2026-10-21 19:43:36 +02:00
jschoubben b62fbeabbf Ask a held move and the builds held after a failed release, and say them on the plan's line (issue 1732, 1320) 2026-10-21 19:43:36 +02:00
jschoubben 373aefd14e Take mesh-sdk at 07a6592, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config checking: composed check out of date: novox/mesh-sdk@c8e35097e861 was judged against trunk 85b5a2687e6a, the tr…
mesh/merge-gate building on ace since 19:52
mesh/repo-check out of date: judged against trunk 22c9815f6522, the trunk is now 44033f4b6d72; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:43:32 +02:00
jschoubben f66458fe25 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config 2026-10-21 19:43:02 +02:00
mesh-tools 22c9815f65 Merge pull request 'Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)' (#423) from fix/1310-failed-check-summary into main 2026-10-21 17:39:15 +00:00
jschoubben a14dc04b7b Hold the build seat's checks while a delivery builds, and let one check that waited 30 minutes through
ADR 0483: the controller keeps a key per plan in its build phase; holders read it and
take no new check while one is there, but for the oldest check past 30 minutes, one
at a time by a let-through key. The queue says why checks wait.
2026-10-21 19:32:47 +02:00
jschoubben 181b579ac6 Assert the next compose carries the issued account, not a stand-in (issue 1226, review round 2)
mesh/merge-gate head 181b579ac6ea merged into trunk 22c9815f6522 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 181b579ac6ea merged into trunk 22c9815f6522 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:27:22 +02:00
jschoubben b08c6372e1 Assert the next compose carries the issued account (issue 1226, review round 2) 2026-10-21 19:26:57 +02:00
jschoubben 4e11965a2a Merge remote-tracking branch 'origin/main' into fix/1226-issue-process-accounts 2026-10-21 19:26:00 +02:00
jschoubben faf8beee84 Warn instead of holding the machine for an unissued process account, and drive the delivery path in the test (issue 1226, review round 1) 2026-10-21 19:25:44 +02:00
mesh-tools 3827c3eb60 Merge pull request 'Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920)' (#426) from fix/920-plan-line-says-the-refusal into main 2026-10-21 17:23:01 +00:00
jschoubben 6bcfbeb5b2 Issue a bus process's account when its module is assigned or gains it (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben deac3e232c Refuse a process account's stand-in, naming issue (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben 99da867a98 Merge origin/main into fix/1656-xterm-config 2026-10-21 19:16:11 +02:00
jschoubben 612f493c49 Say only a refusal as a plan's hold, not a tier's summary left in its note (issue 920, review round 1)
mesh/merge-gate head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:01:42 +02:00
jschoubben 2377d75ed8 Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920) 2026-10-21 18:57:59 +02:00
mesh-tools 5ba2ffe8e6 Merge pull request 'Send a declaration's large files by reference through a bucket per node, and hold no check twice on the build seat' (#425) from feat/1696-declaration-files into main 2026-10-21 16:51:05 +00:00
jschoubben 8fc0d135bb Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files
mesh/merge-gate head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: its merge-check.sh passed
mesh/delivery delivered
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-21 18:32:56 +02:00
mesh-tools d568d76b97 Merge pull request 'Take mesh-sdk 90b20f5, so node-files' contract names upload and download (issue 1718, design 60 F3 group)' (#424) from feat/files-through-the-store into main 2026-10-21 16:31:27 +00:00
jschoubben 610d88e6bf Merge commit '3217f097d38d9f1c4302ec827ad6b371fd8ba95b' into feat/1696-declaration-files 2026-10-21 18:21:37 +02:00
jschoubben 0b11b6c98d Take mesh-sdk at its merge 725590e0 of the per-node bucket (issue 1694, ADR 0488) 2026-10-21 18:03:47 +02:00
jschoubben 92a5193e31 Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files 2026-10-21 18:03:06 +02:00
jschoubben 3217f097d3 Make a fetch's pull request end before the fetch does, and drop the hand-back loop that could itself take asks (issue 1649) 2026-10-21 17:28:40 +02:00
jschoubben 69a67876c2 Take mesh-sdk at 6f63275, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 69a67876c269 merged into trunk d96c9f56a3f1 beside mesh-catalog#416@f178aab6 — fail: its merge-check.sh failed: --- FAIL: TestModuleCh…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 17:03:35 +02:00
jschoubben 650274d700 Read a pull's held asks with Fetch, hand them back all through a build, and drop a duplicate only while its builder's key is fresh (issue 1649) 2026-10-21 16:58:39 +02:00
jschoubben 678a517843 Tests first for issue 1649: a held ask really handed back, a stale running key no duplicate 2026-10-21 16:57:40 +02:00
jschoubben 516af253c6 Merge main into feat/1696-declaration-files, keeping mesh-sdk at 86f9a1a5, which holds main's pin 85b5a268 2026-10-21 16:46:21 +02:00
jschoubben 5157ccc3a8 Hand back any ask a pull holds before a build starts, and drop a hand-out of a check another builder runs, so one check is never run twice (issue 1649) 2026-10-21 16:35:09 +02:00
jschoubben 087e9266d1 Test the capability gate, the sweep's mark re-read and the lock use of a send and of the sweep, each failing under its mutation (issues 1696, 1697; review round 2) 2026-10-21 16:28:24 +02:00
jschoubben a3b4e0601d Take mesh-sdk at 86f9a1a5 and name each node's bucket with its FileBucketFor, refusing a node name that could carry a wildcard (issues 1696, 1697; ADR 0488) 2026-10-21 16:05:24 +02:00
jschoubben da364c2730 Give each node a bucket of its own for the files that travel by digest, and keep the sweep from removing an object a send relies on (issues 1696, 1697; review of ADR 0486 step 3)
A literal * in a published subject matches the grant's *, so in one shared bucket a node could read every node's files. Each node now
has declaration-files-<node>, which its account reads by name and the controller writes; the sweep judges each bucket against that
node's latest declaration, under the node's lock with the send, and an object the sweep marked is put again by the next send. Putting
that fails send after send raises a condition.
2026-10-21 15:43:36 +02:00
jschoubben 4771ee53d9 Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)
mesh/merge-gate out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@181b579ac6ea (merged as 44033f4b into main, walk plan-17926043…
2026-10-21 15:21:10 +02:00
jschoubben ceaa616d01 Send files over 64 KiB by digest through the bus's object bucket to nodes that read them, grant it, and warn at 75 percent (issues 1696, 1697, 1698; ADR 0486 step 3)
A node that reports file-references gets the digest and size of each file over 64 KiB that may travel so, and the content is put in
the bucket declaration-files before the declaration is sent; others get every content inline. The controller alone writes the
bucket and node-engines read an object by name (the four grants the node-engine's own fetch makes, no list, no write). An object
goes a day after no node's latest declaration names it. The near-limit condition warns at 75 percent, and a test holds the control
node's declaration, in the reference form, under half the bus's largest message.
2026-10-21 15:16:11 +02:00
jschoubben fa5b5e1c11 Take mesh-sdk at 863a981e, which carries the file references of ADR 0486 (issue 1696) 2026-10-21 15:07:36 +02:00
422 changed files with 97674 additions and 585 deletions
+3 -12
View File
@@ -90,15 +90,6 @@ redis-provisioner-image:
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The proxy that turns a route grant into traffic reaching a workload.
PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
@@ -117,7 +108,7 @@ proxy-image:
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
@sh merge-check.sh ; status=$$? ; docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
@@ -135,7 +126,7 @@ fmt:
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker run -d --name $(PG_CONTAINER) -e POSTGRES_PASSWORD=check \
-p 127.0.0.1:$(PG_PORT):5432 $(PG_IMAGE) >/dev/null
@printf 'waiting for postgres'
@@ -147,7 +138,7 @@ postgres:
echo ' — never came up' ; docker logs $(PG_CONTAINER) | tail -20 ; exit 1
postgres-stop:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
clean:
rm -rf build/
+91 -2
View File
@@ -6,7 +6,11 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
@@ -73,14 +77,77 @@ func drainedTooLong(now time.Time) error {
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// inHand is true while this process has claimed a build and not yet released it. The mark in the temporary directory
// is only believed while it is: a mark with no build of this process behind it is stale (a crash, a fault in an
// earlier path) and is removed, because left alone it hands back every ask for ever (issue 1768).
var inHand atomic.Bool
// markRetry is how long an agent that cannot mark a build in hand takes no work, and markFault when it last failed to.
// Taking nothing is the point: an ask fetched and handed back at once, many times a second, spends all its deliveries.
const markRetry = 30 * time.Second
var (
markMu sync.Mutex
markFault time.Time
markSaid string
)
// markFaulty says the agent could not mark a build in hand and has not yet been able to since: it takes no work. Once
// markRetry has passed it tries the mark itself, with no ask in hand, so that an ask is fetched only when marking works.
func markFaulty(now time.Time) bool {
markMu.Lock()
failed := markFault
markMu.Unlock()
if failed.IsZero() {
return false
}
if now.Sub(failed) < markRetry {
return true
}
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil && errors.Is(err, os.ErrExist) && !inHand.Load() {
_ = os.Remove(inHandMark())
f, err = os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
}
if err != nil {
markFailed(err)
return true
}
_ = f.Close()
_ = os.Remove(inHandMark())
markMu.Lock()
markFault, markSaid = time.Time{}, ""
markMu.Unlock()
fmt.Fprintf(os.Stderr, "the mark of a build in hand can be made again (%s): taking builds\n", inHandMark())
return false
}
// markFailed notes that a build could not be marked in hand, and says it when it is a new reason.
func markFailed(err error) {
markMu.Lock()
defer markMu.Unlock()
markFault = time.Now()
if said := err.Error(); said != markSaid {
markSaid = said
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so this agent takes no build for %s and tries again: %v\n",
inHandMark(), markRetry, err)
}
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
if !inHand.Load() {
// Nothing of this process is in hand, so a mark already there is stale.
if err := os.Remove(inHandMark()); err == nil {
fmt.Fprintf(os.Stderr, "removed the mark of a build in hand (%s): no build of this agent is in hand\n", inHandMark())
}
}
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
@@ -89,7 +156,7 @@ func claim() bool {
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
markFailed(err)
return false
}
if drained() {
@@ -97,6 +164,7 @@ func claim() bool {
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
inHand.Store(true)
return true
}
@@ -107,6 +175,7 @@ func holding(what string) {
// released ends the build in hand.
func released() {
inHand.Store(false)
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
@@ -144,3 +213,23 @@ func busy() (string, error) {
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
var (
roomMu sync.Mutex
roomSaid bool
)
// noRoom says this node is under the floor of free space (novox/hq issue 1834): it takes no work, so another build node
// takes the ask instead of this one taking it and ending it in an error. Said once when it starts and again when it ends.
func noRoom(workspace string) bool {
err := builder.RoomToWork("/", workspace)
roomMu.Lock()
defer roomMu.Unlock()
if err != nil && !roomSaid {
fmt.Fprintf(os.Stderr, "taking no builds: %v\n", err)
} else if err == nil && roomSaid {
fmt.Fprintf(os.Stderr, "there is room again: taking builds\n")
}
roomSaid = err != nil
return err != nil
}
+77
View File
@@ -1,6 +1,7 @@
package main
import (
"github.com/novox/mesh-controller/internal/builder"
"os"
"strings"
"sync"
@@ -160,3 +161,79 @@ func TestAnUnknownCommandFails(t *testing.T) {
t.Fatalf("help failed: %v", err)
}
}
// 1768/AC2: a mark naming no build of this agent (left by a crash or a fault) is stale: the next ask removes it and is
// run, never fetched and handed back in a loop.
func TestAStaleInHandMarkIsRemovedAndTheAskRuns(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
inHand.Store(false)
if err := undrained(); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(inHandMark(), []byte("2026-10-21T18:45:06Z\n"), 0o644); err != nil {
t.Fatal(err)
}
if !claim() {
t.Fatal("a stale in-hand mark made the agent hand back an ask")
}
// A live one — this agent's own build in hand — is never taken over.
if claim() {
t.Fatal("a second build was taken while one is in hand")
}
if _, err := busy(); err != nil {
t.Fatalf("the build in hand was lost: %v", err)
}
released()
}
// 1768/AC3: an agent that cannot mark a build in hand for another reason takes no work, instead of fetching asks and
// handing them back; it takes work again once marking works.
func TestAnAgentThatCannotMarkABuildTakesNoWorkUntilItCan(t *testing.T) {
dir := t.TempDir()
t.Setenv("TMPDIR", dir)
inHand.Store(false)
markFault, markSaid = time.Time{}, ""
t.Cleanup(func() { markFault, markSaid = time.Time{}, "" })
if err := undrained(); err != nil {
t.Fatal(err)
}
// The mark's place is a directory that cannot be removed or made over: a directory named as the mark.
if err := os.MkdirAll(inHandMark()+"/x", 0o755); err != nil {
t.Fatal(err)
}
if claim() {
t.Fatal("an agent that cannot mark a build took it")
}
if !markFaulty(time.Now()) {
t.Fatal("an agent that cannot mark a build still takes work")
}
// Still failing after the retry time: still takes nothing, and no ask was needed to find out.
if !markFaulty(time.Now().Add(2 * markRetry)) {
t.Fatal("an agent whose mark still cannot be made takes work")
}
if err := os.RemoveAll(inHandMark()); err != nil {
t.Fatal(err)
}
if markFaulty(time.Now().Add(4 * markRetry)) {
t.Fatal("an agent whose mark can be made again takes no work")
}
if !claim() {
t.Fatal("an agent that can mark takes no build")
}
released()
}
// 1834/AC4: given a node under the floor of free space, when the taking loop asks, then it takes nothing, and takes again
// once there is room.
func TestANodeUnderTheFloorOfFreeSpaceTakesNoWork(t *testing.T) {
was := builder.FreeBytes
t.Cleanup(func() { builder.FreeBytes = was })
builder.FreeBytes = func(string) (uint64, error) { return 2 << 30, nil }
if !noRoom(t.TempDir()) {
t.Error("a node with 2 GiB free took work")
}
builder.FreeBytes = func(string) (uint64, error) { return 200 << 30, nil }
if noRoom(t.TempDir()) {
t.Error("a node with 200 GiB free refused work")
}
}
+14
View File
@@ -36,3 +36,17 @@ func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
// 1826/AC2: the held values a check request carries reach the spec; a request that carries none leaves it empty.
func TestTheHeldValuesAnAskCarriesReachTheSpec(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog",
HeldConfig: []link.HeldConfig{{Pull: "novox/mesh-catalog#434", Head: "a", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}}})
if len(spec.Held) != 1 || spec.Held[0].Node != "ace" || spec.Held[0].Key != "media-gid" || string(spec.Held[0].Value) != "2000" ||
spec.Held[0].Approved != "the operator" {
t.Fatalf("the spec reads %+v", spec.Held)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Held != nil {
t.Fatalf("a request that carries none has %+v", plain.Held)
}
}
+2 -2
View File
@@ -91,7 +91,7 @@ func TestKillEndsTheBuildRunningHereAndNoOther(t *testing.T) {
t.Errorf("kill said %q", said)
}
// Removed at the kill and again once the build had ended: a container made in between is caught.
if len(removed) != 4 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[2], "label=mesh.build=build-1") {
if len(removed) != 6 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[3], "label=mesh.build=build-1") {
t.Errorf("removed %v", removed)
}
if c := h.current(); c.Running != nil {
@@ -160,7 +160,7 @@ func TestAHolderStartingRemovesWhatEarlierBuildsLeftHere(t *testing.T) {
}
return "", nil
}
if n := h.removeLeftBehind(); n != 2 || len(removed) != 2 || removed[1] != "docker rm -f c1 c2" {
if n := h.removeLeftBehind(); n != 2 || len(removed) != 3 || removed[2] != "docker rm -f -v c1 c2" {
t.Fatalf("removed %d: %v", n, removed)
}
h.remove = func(context.Context, string, string, ...string) (string, error) {
+11 -1
View File
@@ -64,6 +64,8 @@ is dialled except the broker.
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
MESH_WORKSPACE where to clone and build (default: a temporary directory)
MESH_BUILD_CGROUP_PARENT the slice every container a build or check starts joins, which carries the
machine's limits on builds (novox/hq issue 1798; default: the runtime's own)
It also builds one module and stops, which is how a mesh is raised — before there is a
broker to take work from or a registry to publish into:
@@ -137,6 +139,10 @@ func run() error {
if err != nil {
return err
}
// A build and a check run below the machine's own work (novox/hq issue 1798): the agent itself at the
// lowest priority, which its git and go children inherit, and every container in the build slice.
_ = syscall.Setpriority(syscall.PRIO_PROCESS, 0, 19)
builder.CgroupParent = strings.TrimSpace(os.Getenv("MESH_BUILD_CGROUP_PARENT"))
workspace := os.Getenv("MESH_WORKSPACE")
if workspace == "" {
workspace = os.TempDir() + "/mesh-builder"
@@ -184,7 +190,7 @@ func run() error {
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Paused: func() bool { return h.Paused() || drained() || markFaulty(time.Now()) || noRoom(workspace) },
Taking: func() { taking(seat) },
Claim: claim,
})
@@ -499,6 +505,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, h := range c.HeldConfig {
spec.Held = append(spec.Held, builder.HeldValue{Pull: h.Pull, Head: h.Head, Node: h.Node, Module: h.Module, Key: h.Key,
Value: h.Value, Approved: h.Approved, By: h.By, At: h.At})
}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
+21 -2
View File
@@ -408,12 +408,21 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
_, reads := m.OwnSecrets["broker"]
if !reads && !runsTheProvisionerClient(m) {
// A module whose bus accounts are its processes' alone has none of its own: only the processes that
// are not issued yet are issued (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
if !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return ""
} else if minted {
// Its own account stands and is kept; a process it gained is not issued yet (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
busAddress, err := broker.BusAddress()
if err == nil {
@@ -426,6 +435,16 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return fmt.Sprintf("its bus credential is issued and sealed to %s, and arrives with the push", node)
}
// processAccountsOnAssign issues the process accounts of an assigned module not issued yet, and says what it did.
func processAccountsOnAssign(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string) string {
line, err := issueMissingProcessAccounts(ctx, inv, m, node)
if err != nil {
return fmt.Sprintf("its process accounts are not issued (%v): `module issue %s --node %s` first — "+
"`push %s` refuses to send %s until it is", err, m.Module, node, node, m.Module)
}
return line
}
// refusedMovingTheController refuses assigning or unassigning the controller's module through a verb (review of
// novox/hq ADR 0272): the node it is assigned to is the control-node, and the control-node's operator account is
// the controller's terminal, so whoever moves the module chooses the terminal. At the terminal alone, as every
+4
View File
@@ -645,6 +645,10 @@ func doesWords(act conditions.Action) string {
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case act.Verb == broker.ControllerSeat+".push" && act.Arguments["node"] != "":
return "the held builds are sent to " + act.Arguments["node"]
case act.Verb == broker.ControllerSeat+".upgrade" && act.Arguments["release-backlog"] != "":
return "the builds held after the failed delivery are released"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
+3
View File
@@ -768,6 +768,9 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
}
return manifest, kept, err
}
// A bus process the new definition gains gets its account now, on each machine assigned the module, so the
// next push composes a real account (issue 1226).
issueGainedProcessAccounts(ctx, inv, manifest)
// Which repository it is registered from, by the forge's own id (novox/hq ADR 0266).
if err := inv.SetSourceIdentity(ctx, manifest.Module, repoID); err != nil {
return manifest, kept, err
+7
View File
@@ -32,6 +32,13 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err := broker.Raise(r, names); err != nil {
return nil, err
}
// And each node's bucket for the files of its declaration that travel by digest (novox/hq ADR 0486 §4), so a node finds its own
// whenever the bus is up.
if files, ok := r.(interface{ EnsureDeclarationFiles(nodes []string) error }); ok {
if err := files.EnsureDeclarationFiles(names); err != nil {
return nil, err
}
}
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
+27 -2
View File
@@ -191,6 +191,16 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
// **Mesh-config is told what the head declares at every announced head of a pull request in a repository that holds
// modules** (novox/hq ADR 0502 clauses 2 and 6, issue 1826), also when the head touches no module: a head that reverts
// the change must drop the value held for it. Asked of nothing else here.
this := pullAtOf(p)
if this.Truncated {
this.Paths, this.Truncated = truncatedPaths(entries, p.Owner, p.Repo, scope.Modules), false
}
if scope.gated() || holdsModules(entries, p.Owner, p.Repo) {
tellPullDeclares(ctx, w, []pullAt{this})
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
@@ -209,6 +219,9 @@ func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Wri
}
request.Check.Plan = &plan
request.Check.Composed = composed
if !composed {
carryHeld(ctx, w, &request, []pullAt{this})
}
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
@@ -594,11 +607,13 @@ func checkedOf(result link.BuildResult) link.Checked {
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
// Every member is named with its trunk commit (novox/hq issue 1106): the seat's word, else the one the forge
// announced with the head, so the forge's merge tool never meets a member whose verdict names none.
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref, Trunk: c.Trunk})
Commit: result.Ref, Trunk: firstTrunk(c.Trunk, g.Trunk)})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
Commit: m.Ref, Trunk: firstTrunk(memberTrunk(result.Check, m), m.Trunk)})
}
}
}
@@ -627,6 +642,16 @@ func checkedOf(result link.BuildResult) link.Checked {
return c
}
// firstTrunk is the first trunk commit named, as twelve characters; empty when none is.
func firstTrunk(named ...string) string {
for _, n := range named {
if n != "" {
return twelve(n)
}
}
return ""
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
+25
View File
@@ -250,6 +250,31 @@ func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
}
}
// **A group's composed verdict names every member's trunk commit** (novox/hq issue 1106): a member the build seat named no
// trunk for (an SDK member whose base the seat did not merge, or a seat from before the rule) is named by the trunk the forge
// announced with its head, and the primary the same, so the merge tool never reads "names no trunk commit" on a member
// whose trunk did not move.
func TestAGroupVerdictNamesEveryMembersTrunkTheSDKMemberToo(t *testing.T) {
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 7, Base: "main", Group: "feat/x",
Trunk: "dddddddddddd0000",
Members: []link.GroupMember{
{Owner: "novox", Repo: "mesh-sdk", Number: 8, Ref: "cccccccccccc0000", Base: "main", Trunk: "eeeeeeeeeeee0000"},
{Owner: "novox", Repo: "mesh-host", Number: 9, Ref: "bbbbbbbbbbbb0000", Base: "main", Trunk: "ffffffffffff0000"}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "bbbbbbbbbbbb", Trunk: "222222222222"}}}})
want := []string{"dddddddddddd", "eeeeeeeeeeee", "222222222222"}
if len(c.Members) != 3 {
t.Fatalf("the composed check names its members %+v", c.Members)
}
for i, m := range c.Members {
if m.Trunk != want[i] {
t.Fatalf("member %d (%s/%s) is named with trunk %q, want %q (the seat's word first, else the announced one): %+v",
i, m.Owner, m.Repo, m.Trunk, want[i], c.Members)
}
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
+6
View File
@@ -233,6 +233,12 @@ func prepareSend(ctx context.Context, open *stores, node string) ([]string, erro
if !slices.Contains(in.BusMinted, "provisioner."+node) {
continue
}
default:
// A bus process's account nobody issued still gets the stand-in, until the composition leaves
// the module off (issue 1753): the warning names the verb (issue 1226).
if line := processAccountWarning(m, node, name, in.BusMinted); line != "" {
sayWarning(ctx, line)
}
}
if _, err := inv.SecretForModule(ctx, node, m.Module, name); err == nil {
did = true
+7 -1
View File
@@ -200,6 +200,7 @@ func composeQuestion(ctx context.Context, open *stores, node string, mode compos
}
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
if !withOrder {
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
}
@@ -238,7 +239,11 @@ func markUnresolvable(err error) error {
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
if err != nil {
return declared, err
}
// Never a declaration naming an image the store does not hold (novox/hq issue 1742): that module is left out.
return withoutMissingImages(ctx, open, node, declared), nil
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
@@ -281,6 +286,7 @@ func composeSendValidated(ctx context.Context, open *stores, node string) (senda
if len(left) == 0 && len(in.wouldMake) == 0 {
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
// The sender stamps the order it allots (order.stamp); a composition for a send carries none of its own.
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
here, _ := nodeOf(in.Input, node)
@@ -0,0 +1,110 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/overlay"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// 1696/AC2, ADR 0486 §5: a node is sent a file by digest only when its profile reports the capability. A node whose profile lacks
// it, and a node the mesh has no profile of, are sent every content inline; a node that reports it is sent the reference. The test
// goes through composeQuestion and composeSend, the two places readsFileReferences is read (compose_sdk.go).
func TestAFileTravelsByDigestOnlyToANodeThatReportsTheCapability(t *testing.T) {
open := aMesh(t) // anchor and laptop report no file-references
ctx := t.Context()
big := strings.Repeat("0123456789abcdef", 8<<10) // 128 KiB
register(t, open, catalogue.Manifest{Module: "bigfile", Version: "1",
Resources: []map[string]any{{"id": "blob", "type": "file", "path": "/etc/bigfile/blob", "mode": "0644", "content": big}}})
// reader reports the capability; bare has never reported a profile at all.
for i, n := range []struct {
name string
profile map[string]any
}{
{"reader", map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true},
map[string]any{"name": sdkdecl.FileReferencesCapability, "present": true}}}},
{"bare", nil},
} {
record, err := open.inventory.AddNode(ctx, n.name)
if err != nil {
t.Fatal(err)
}
if err := open.inventory.SetPlace(ctx, n.name, n.name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+10)); err != nil {
t.Fatal(err)
}
if n.profile != nil {
if err := open.inventory.RecordProfile(ctx, record.ID, n.profile); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, n.name, overlay.Name); err != nil {
t.Fatal(err)
}
}
for _, node := range []string{"laptop", "reader", "bare"} {
if _, err := assign(ctx, open, node, "bigfile"); err != nil {
t.Fatalf("assign %s: %v", node, err)
}
}
byDigest := map[string]bool{"laptop": false, "bare": false, "reader": true}
for node, want := range byDigest {
asked, _, err := composeQuestion(ctx, open, node, compose.Reading, false)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
sent, err := composeSend(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
for what, s := range map[string]sendable{"question": asked, "send": sent} {
if s.fileReferences != want {
t.Fatalf("%s, %s: fileReferences %v, want %v", node, what, s.fileReferences, want)
}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
var found bool
for _, r := range got.Resources {
if r["path"] != "/etc/bigfile/blob" {
continue
}
found = true
inline := r["content"] == big
if want && (inline || r["digest"] == nil || r["size"] == nil) {
t.Fatalf("%s, %s: a node that reads references is sent the content inline", node, what)
}
if !want && (!inline || r["digest"] != nil) {
t.Fatalf("%s, %s: a node that does not report the capability is sent a reference", node, what)
}
}
if !found {
t.Fatalf("%s, %s: the file is not in the declaration", node, what)
}
}
}
}
+253
View File
@@ -0,0 +1,253 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/compose"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A declaration's large files travel by their digest to a node that can read one** (novox/hq ADR 0486 §1, §5, issue 1696).
//
// The declaration to a node that reports sdkdecl.FileReferencesCapability carries, in place of the content of every
// file resource of more than sdkdecl.FileReferenceThreshold bytes that may travel so (File.TravelsByDigest: plain content or
// bytes, not sealed, not filled with secrets, not written into another file), its sha256 digest and size. The content is put
// into the bucket of the bus under the hex of the digest before the declaration is sent. A node that does not report the
// capability is sent every content inline, as before: an older node-engine refuses a field it does not know, whole.
// readsFileReferences says the node reports the capability of reading a file given by digest (ADR 0486 §5). A node the mesh has
// no profile of reads none.
func readsFileReferences(in compose.Input, node string) bool {
n, ok := nodeOf(in, node)
return ok && n.Capabilities[sdkdecl.FileReferencesCapability]
}
// referencing is the resources as a node that reads references is sent them, and the content of each file that went by
// digest, by the name of its object (the hex of its digest). The resources it is given are not changed. A file that does not
// travel by digest is passed as it is.
func referencing(resources []map[string]any) ([]map[string]any, map[string][]byte, error) {
out := make([]map[string]any, len(resources))
var objects map[string][]byte
for i, r := range resources {
out[i] = r
if kind, _ := r["type"].(string); kind != string(sdkdecl.TypeFile) {
continue
}
// Cheap first: only a file with a content or bytes of the size is worth reading whole.
content, _ := r["content"].(string)
bytesField, _ := r["bytes"].(string)
if len(content) <= sdkdecl.FileReferenceThreshold && len(bytesField) <= sdkdecl.FileReferenceThreshold {
continue
}
raw, err := json.Marshal(r)
if err != nil {
return nil, nil, fmt.Errorf("resource %v: %w", r["id"], err)
}
var f sdkdecl.File
if err := json.Unmarshal(raw, &f); err != nil {
return nil, nil, fmt.Errorf("resource %v is not a file the controller can read: %w", r["id"], err)
}
if !f.TravelsByDigest() {
continue
}
ref, body, err := f.AsReference()
if err != nil {
return nil, nil, err
}
name, err := sdkdecl.ObjectName(ref.Digest)
if err != nil {
return nil, nil, err
}
with := make(map[string]any, len(r)+2)
for k, v := range r {
with[k] = v
}
delete(with, "content")
delete(with, "bytes")
with["digest"], with["size"] = ref.Digest, ref.Size
out[i] = with
if objects == nil {
objects = map[string][]byte{}
}
objects[name] = body
}
return out, objects, nil
}
// namedObjects are the objects of the bucket a body names: the object name of every file resource given by reference,
// with its size. It reads a declaration as the node does, so it also reads the latest one the bus holds.
func namedObjects(body []byte) (map[string]int64, error) {
var envelope struct {
Resources []struct {
Type string `json:"type"`
Digest string `json:"digest"`
Size int64 `json:"size"`
} `json:"resources"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
return nil, err
}
var out map[string]int64
for _, r := range envelope.Resources {
if r.Type != string(sdkdecl.TypeFile) || r.Digest == "" {
continue
}
name, err := sdkdecl.ObjectName(r.Digest)
if err != nil {
return nil, err
}
if out == nil {
out = map[string]int64{}
}
out[name] = r.Size
}
return out, nil
}
// objectStore is what putting the objects needs of a node's bucket.
type objectStore interface {
// Has says whether an object of that name is there, complete, and whether it carries the sweep's mark of being named by no
// node's latest declaration.
Has(ctx context.Context, name string) (have, marked bool, err error)
// Put writes the object under that name, replacing one of the same name (and so erasing its mark).
Put(ctx context.Context, name string, content []byte) error
}
// putObjects makes sure every object of the declaration is in the bucket before the declaration that names it is sent: a node
// that reads the declaration finds each object. An object already there (the same content, the same name) is not written again,
// **unless the sweep marked it unnamed**: then it is written again, so the mark that would have it removed is gone with the
// old object (issue 1696, review: a mark read just before a send must not take an object the send relies on). Refuses on the
// first it cannot put, naming it, so the declaration is not sent.
func putObjects(ctx context.Context, store objectStore, objects map[string][]byte) error {
names := make([]string, 0, len(objects))
for name := range objects {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
have, marked, err := store.Has(ctx, name)
if err != nil {
return fmt.Errorf("whether the object %s is in the bucket: %w", name[:12], err)
}
if have && !marked {
continue
}
if err := store.Put(ctx, name, objects[name]); err != nil {
return fmt.Errorf("putting the object %s (%d bytes) in the bucket: %w", name[:12], len(objects[name]), err)
}
}
return nil
}
// busObjects is a node's bucket on the controller's own connection.
type busObjects struct{ store jetstream.ObjectStore }
// openBucket opens the bucket of a node's files, making it first if it is not there (it is asserted for every node at the
// controller's start; a node adopted since is covered here).
func openBucket(ctx context.Context, conn *nats.Conn, node string) (busObjects, error) {
js, err := jetstream.New(conn)
if err != nil {
return busObjects{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(ctx, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
if err = broker.EnsureDeclarationFiles(ctx, js, node); err == nil {
store, err = js.ObjectStore(ctx, bucket)
}
}
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket %s: %w", bucket, err)
}
return busObjects{store}, nil
}
func (b busObjects) Has(ctx context.Context, name string) (bool, bool, error) {
info, err := b.store.GetInfo(ctx, name)
switch {
case errors.Is(err, jetstream.ErrObjectNotFound):
return false, false, nil
case err != nil:
return false, false, err
}
_, marked := info.Metadata[declarationUnnamed]
return !info.Deleted, marked, nil
}
func (b busObjects) Put(ctx context.Context, name string, content []byte) error {
_, err := b.store.PutBytes(ctx, name, content)
return err
}
// filesLocks hold a node's bucket for one act at a time in this process: a send puts its objects and publishes the declaration that
// names them under the node's lock, and the sweep judges the node's bucket under the same one, so the sweep never removes an
// object between a send's put and its declaration. Another process (a person's push) is covered by the put erasing a mark and by
// the sweep looking at an object again just before it removes it.
var filesLocks sync.Map
func lockFilesOf(node string) func() {
l, _ := filesLocks.LoadOrStore(node, &sync.Mutex{})
mu := l.(*sync.Mutex)
mu.Lock()
return mu.Unlock
}
// kindFilesNotPut is a node whose files that travel by digest could not be put in its bucket, send after send.
const kindFilesNotPut = "declaration-files-not-put"
// filesNotPutAfter is how many sends in a row may fail to put before the condition is raised: the first send to the machine holding
// the bus fails for want of the grant that very send delivers, and says so, and is not yet a fault.
const filesNotPutAfter = 2
var (
filesFailMu sync.Mutex
filesFailed = map[string]int{}
)
func filesNotPutObservation(node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindFilesNotPut, Token: kindFilesNotPut,
Machine: node, Severity: conditions.Warning, Source: "sends", Summary: summary}
}
// notePut records how putting a send's objects went: a failure counts, the second in a row raises the condition, a success
// clears the count and the condition.
func notePut(ctx context.Context, node string, put error) {
filesFailMu.Lock()
if put == nil {
delete(filesFailed, node)
} else {
filesFailed[node]++
}
n := filesFailed[node]
filesFailMu.Unlock()
keeper := conditionsFrom
if keeper == nil {
return
}
key := filesNotPutObservation(node, "").Key()
switch {
case put == nil:
if _, err := keeper.Clear(ctx, key, node+" was sent its files by digest"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
case n >= filesNotPutAfter:
summary := fmt.Sprintf("%s's files that travel by digest could not be put in its bucket %d sends in a row, so it is sent them "+
"inline: %v", node, n, put)
if _, err := keeper.Observe(ctx, filesNotPutObservation(node, summary)); err != nil {
fmt.Printf("the condition %s could not be raised: %v\n", key, err)
}
}
}
@@ -0,0 +1,246 @@
//go:build integration
package main
import (
"bytes"
"context"
"encoding/json"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
type sigOf struct{}
func (sigOf) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// 1696/AC7 (ADR 0486 §1): on a real bus with its default largest message of 1 MiB, a declaration whose files add up to more than
// 3 MiB is sent as references, its objects are in the bucket, each fetched back whole and verified against the digest and size the
// declaration names; the same declaration inline is refused by the controller, naming size and limit. A changed byte in an object
// fails the verification.
func TestADeclarationOfMoreThanThreeMiBIsSentByReferenceOnARealBus(t *testing.T) { // 1696/AC7
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
nc, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
if _, err := nc.AddStream(&nats.StreamConfig{Name: "NODES", Subjects: []string{"mesh.node.*.declare"},
MaxMsgsPerSubject: 1}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
api, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatal(err)
}
// Run twice: an assertion of an existing bucket is no error and no change.
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatalf("asserting the bucket again: %v", err)
}
var resources []map[string]any
for i := 0; i < 3; i++ {
resources = append(resources, aFileResource("big-"+string(rune('a'+i)), contentOf("big"+string(rune('a'+i)), 1200<<10)))
}
resources = append(resources, aFileResource("small", "tiny\n"))
inline := sendable{Resources: resources}
byDigest := sendable{fileReferences: true, Resources: resources}
bus := link.OverNATS{Conn: conn, JS: nc}
body, err := inline.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", body, 5*time.Second); err == nil ||
!strings.Contains(err.Error(), "1048576") || strings.Contains(err.Error(), "maximum payload exceeded") {
t.Fatalf("the inline declaration of more than 3 MiB: %v", err)
}
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
objects, err := byDigest.objects()
if err != nil || len(objects) != 3 {
t.Fatalf("objects: %d %v", len(objects), err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatal(err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatalf("putting them again: %v", err)
}
refBody, err := byDigest.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", refBody, 5*time.Second); err != nil {
t.Fatalf("the declaration by reference: %v", err)
}
// What the stream holds for the node is the reference form, and it names exactly the three objects.
latest, err := latestDeclarations(ctx, conn, []string{"novox", "ace"})
if err != nil || len(latest) != 1 {
t.Fatalf("the latest declarations: %v %v", len(latest), err)
}
named, err := namedByLatest("novox", latest["novox"])
if err != nil || len(named) != 3 {
t.Fatalf("what the latest declaration names: %v %v", named, err)
}
novoxBucket, err := broker.DeclarationFilesBucketOf("novox")
if err != nil {
t.Fatal(err)
}
store, err := api.ObjectStore(ctx, novoxBucket)
if err != nil {
t.Fatal(err)
}
var envelope struct {
Resources []sdkdecl.File `json:"resources"`
}
if err := json.Unmarshal(refBody, &envelope); err != nil {
t.Fatal(err)
}
for _, f := range envelope.Resources {
if !f.IsReference() {
continue
}
name, _ := sdkdecl.ObjectName(f.Digest)
got, err := store.GetBytes(ctx, name)
if err != nil {
t.Fatalf("the object of %s: %v", f.ID, err)
}
if err := f.Verify(got); err != nil {
t.Errorf("the object does not verify: %v", err)
}
got[len(got)/2] ^= 1
if err := f.Verify(got); err == nil {
t.Errorf("a changed byte in the object of %s passed the verification", f.ID)
}
}
// The sweep against the real bucket: all named, nothing marked; a fourth object named by none is marked, kept a day less a
// minute, and removed at the day.
extra := bytes.Repeat([]byte("x"), 100)
extraName := strings.Repeat("e", 64)
if err := putObjects(ctx, bucket, map[string][]byte{extraName: extra}); err != nil {
t.Fatal(err)
}
b := busSweptBucket{store}
t0 := time.Now()
if r, err := sweepDeclarationFiles(ctx, b, named, t0); err != nil || r.Marked != 1 || r.Removed != 0 || r.Kept != 3 {
t.Fatalf("first sweep: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour-time.Minute)); err != nil || r.Removed != 0 {
t.Fatalf("a minute short of a day: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour+time.Second)); err != nil || r.Removed != 1 {
t.Fatalf("a day after: %+v %v", r, err)
}
if infos, err := store.List(ctx); err != nil || len(infos) != 3 {
t.Fatalf("the bucket holds %d objects (%v), want the 3 the declaration names", len(infos), err)
}
}
// 1696/AC8 on a real bucket: the sweep removes an object only if it still carries the mark it judged it by. An object marked
// unnamed, then put again by a send (which erases the mark), is kept when the sweep deletes it by the old mark.
func TestTheSweepsDeleteLeavesAnObjectASendPutAgainAfterItWasMarked(t *testing.T) { // 1696/AC8
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
name := strings.Repeat("a", 64)
content := map[string][]byte{name: bytes.Repeat([]byte("y"), 100)}
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
b := busSweptBucket{bucket.store}
judged := time.Now().Add(-25 * time.Hour).Truncate(time.Second)
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if _, marked, err := bucket.Has(ctx, name); err != nil || !marked {
t.Fatalf("the object was not marked: %v %v", marked, err)
}
// The sweep has read the mark; a send now puts the object again, which erases it.
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
if have, marked, err := bucket.Has(ctx, name); err != nil || !have || marked {
t.Fatalf("a send did not put the marked object again: have %v marked %v %v", have, marked, err)
}
gone, err := b.Delete(ctx, name, judged)
if err != nil || gone {
t.Fatalf("the sweep deleted by an old mark: gone %v, %v", gone, err)
}
if have, _, err := bucket.Has(ctx, name); err != nil || !have {
t.Fatalf("the object a send relies on is gone: %v %v", have, err)
}
// And the mark still on it is honoured: marked again and deleted by that mark, it goes.
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if gone, err := b.Delete(ctx, name, judged); err != nil || !gone {
t.Fatalf("an object still carrying the mark it was judged by is kept: gone %v, %v", gone, err)
}
}
// 1696/AC9 on a real bus: the sweep of a node's bucket takes the node's lock first, so it waits for a send that has put its objects
// and not yet published the declaration that names them.
func TestTheSweepOfANodesBucketHoldsTheNodesLock(t *testing.T) { // 1696/AC9
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
const node = "sweep-lock-node"
if _, err := openBucket(ctx, conn, node); err != nil { // the bucket exists, or the sweep would return before the lock
t.Fatal(err)
}
release := lockFilesOf(node) // a send is between its put and its publication
done := make(chan struct{})
go func() {
_, _ = sweepNode(ctx, conn, node)
close(done)
}()
select {
case <-done:
t.Fatal("the sweep judged the node's bucket while a send held it")
case <-time.After(300 * time.Millisecond):
}
release()
select {
case <-done:
case <-time.After(20 * time.Second):
t.Fatal("the sweep did not go on once the send let go")
}
}
@@ -0,0 +1,534 @@
package main
import (
"context"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// contentOf is a deterministic content of n bytes that does not compress to nothing and holds no character JSON escapes.
func contentOf(seed string, n int) string {
var b strings.Builder
for i := 0; b.Len() < n; i++ {
fmt.Fprintf(&b, "%s-%07d-%x;", seed, i, i*2654435761%1000003)
}
return b.String()[:n]
}
func aFileResource(id, content string) map[string]any {
return map[string]any{"id": id, "type": "file", "path": "/etc/" + id, "mode": "0644", "content": content}
}
// controlNodeShape is a declaration shaped like the control node's on 2026-10-21 (issue 1689), the sizes measured then: a bus users
// file of 252,678 bytes, the vault's export of 138,475 bytes (compact), the tool runtime 21 KB, the packet filter 19 KB, the launcher
// 14 KB, and about 600 small resources for the rest (about 286 KB). `more` scales the count of small ones: 1 is the shape, 1.4 forty
// percent more of them.
func controlNodeShape(more float64) []map[string]any {
res := []map[string]any{
aFileResource("nats.bus-users", contentOf("users", 252678)),
aFileResource("mesh-vault.kept", contentOf("kept", 138475)),
aFileResource("node-tools.runtime", contentOf("runtime", 21<<10)),
aFileResource("nftables.filtering", contentOf("filter", 19<<10)),
aFileResource("mesh-host.launcher", contentOf("launcher", 14<<10)),
}
for i := 0; i < int(600*more); i++ {
res = append(res, aFileResource(fmt.Sprintf("module-%03d.config", i), contentOf(fmt.Sprintf("small%d", i), 380)))
}
return res
}
func wireSizeOf(t *testing.T, s sendable) int {
t.Helper()
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
// The signature the controller's key makes is 64 bytes.
msg, err := link.Envelope(body, make([]byte, 64))
if err != nil {
t.Fatal(err)
}
return len(msg)
}
// 1696/AC1: for a node that reports the capability, the two big files of a control-node-shaped declaration carry their digest and size
// and no content, and the declaration is far under the bus's 1 MiB.
func TestAFileOver64KiBTravelsByDigestToANodeThatReadsOne(t *testing.T) { // 1696/AC1
users, kept := contentOf("users", 252678), contentOf("kept", 138475)
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("nats.bus-users", users), aFileResource("mesh-vault.kept", kept),
aFileResource("small", "tiny\n"),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, want := range []string{users, kept} {
r := got.Resources[i]
digest := sdkdecl.DigestOf([]byte(want))
if _, has := r["content"]; has || r["digest"] != digest || r["size"] != float64(len(want)) {
t.Errorf("%v goes with content %v, digest %v, size %v; want no content, %s and %d", r["id"], r["content"] != nil,
r["digest"], r["size"], digest, len(want))
}
if r["path"] == nil || r["mode"] != "0644" {
t.Errorf("%v lost its path or mode: %v", r["id"], r)
}
}
if got.Resources[2]["content"] != "tiny\n" || got.Resources[2]["digest"] != nil {
t.Errorf("a small file is not inline: %v", got.Resources[2])
}
if strings.Contains(string(body), users[:200]) || strings.Contains(string(body), kept[:200]) {
t.Error("the content of a file that went by digest is in the body")
}
if n := wireSizeOf(t, s); n > 100<<10 {
t.Errorf("the declaration is %d bytes on the wire, want under 100 KB", n)
}
// The objects to put are the contents, each under the hex of its digest, and a reference verifies against its object.
objects, err := s.objects()
if err != nil || len(objects) != 2 {
t.Fatalf("objects: %v %v", objects, err)
}
for _, want := range []string{users, kept} {
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(want)))
if string(objects[name]) != want {
t.Errorf("the object %s does not hold the content", name[:12])
}
}
named, err := namedObjects(body)
if err != nil || len(named) != 2 {
t.Fatalf("the objects the body names: %v %v", named, err)
}
for name := range objects {
if _, ok := named[name]; !ok {
t.Errorf("the body does not name the object %s it was sent with", name[:12])
}
if _, err := hex.DecodeString(name); err != nil || len(name) != 64 {
t.Errorf("the object name %q is not 64 hex characters", name)
}
}
}
// 1696/AC2: a node that does not report the capability is sent every content inline, as before, and has no object put.
func TestANodeWithoutTheCapabilityIsSentEveryContentInline(t *testing.T) { // 1696/AC2
users := contentOf("users", 252678)
s := sendable{Resources: []map[string]any{aFileResource("nats.bus-users", users)}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if got.Resources[0]["content"] != users || got.Resources[0]["digest"] != nil || got.Resources[0]["size"] != nil {
t.Fatalf("a node that reads no references is not sent the content inline: %v", got.Resources[0]["digest"])
}
if objects, err := s.objects(); err != nil || len(objects) != 0 {
t.Fatalf("objects for a node that reads none: %v %v", len(objects), err)
}
}
// 1696/AC3: only a file that may travel by digest does: one of exactly 64 KiB, a sealed one, one filled with secrets, one written
// into another file, and a directory stay inline, whatever the node reads.
func TestOnlyAFileThatMayTravelByDigestDoes(t *testing.T) { // 1696/AC3
exact := contentOf("exact", 64<<10)
big := contentOf("big", 64<<10+1)
sealed := aFileResource("sealed", big)
sealed["sealed"], sealed["content"] = "box", ""
sealed["content"] = big // a sealed file's content is none; the body must pass it as it is, whatever it holds
secret := aFileResource("secret", big)
secret["secrets"] = map[string]any{"token": "x"}
into := aFileResource("into", big)
into["into"] = "/etc/other"
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("exact", exact), sealed, secret, into,
{"id": "dir", "type": "directory", "path": "/etc/d", "content": big},
aFileResource("big", big),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, id := range []string{"exact", "sealed", "secret", "into", "dir"} {
if got.Resources[i]["id"] != id || got.Resources[i]["digest"] != nil || got.Resources[i]["content"] == nil {
t.Errorf("%s went by digest or lost its content: %v", id, got.Resources[i]["digest"])
}
}
if big := got.Resources[5]; big["id"] != "big" || big["digest"] == nil || big["content"] != nil {
t.Errorf("a plain file of 64 KiB and a byte did not go by digest: %v", big["digest"])
}
}
// A file given as bytes goes by digest on its decoded length, with the decoded bytes in the object.
func TestAFileOfBytesTravelsByDigestOnItsDecodedLength(t *testing.T) {
raw := []byte(contentOf("png", 70<<10))
r := map[string]any{"id": "icon", "type": "file", "path": "/usr/share/icon.png", "bytes": stdBase64(raw)}
s := sendable{fileReferences: true, Resources: []map[string]any{r}}
objects, err := s.objects()
if err != nil || len(objects) != 1 {
t.Fatalf("%v %v", len(objects), err)
}
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf(raw))
if string(objects[name]) != string(raw) {
t.Fatal("the object is not the decoded bytes")
}
if r["bytes"] == nil {
t.Fatal("the composed resource was changed in place")
}
}
type memObjects struct {
have map[string][]byte
marked map[string]bool
puts []string
fail map[string]error
}
func (m *memObjects) Has(_ context.Context, name string) (bool, bool, error) {
_, ok := m.have[name]
return ok, m.marked[name], nil
}
func (m *memObjects) Put(_ context.Context, name string, c []byte) error {
if err := m.fail[name]; err != nil {
return err
}
m.puts = append(m.puts, name)
m.have[name] = c
delete(m.marked, name)
return nil
}
// 1696/AC4: an object already in the bucket is not put again, one absent is, and a put that fails stops the send, naming the object.
func TestObjectsAreInTheBucketBeforeTheDeclarationIsSent(t *testing.T) { // 1696/AC4
objects := map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")}
b := &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one")}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("b", 64) {
t.Fatalf("put %v (%v), want the one absent", b.puts, err)
}
// An object the sweep marked unnamed is put again, which erases the mark: a send relies on it.
b = &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")},
marked: map[string]bool{strings.Repeat("a", 64): true}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("a", 64) || b.marked[strings.Repeat("a", 64)] {
t.Fatalf("a marked object: put %v (%v) marked %v, want it put again and the mark gone", b.puts, err, b.marked)
}
b = &memObjects{have: map[string][]byte{}, fail: map[string]error{strings.Repeat("a", 64): errors.New("permissions violation")}}
err := putObjects(t.Context(), b, objects)
if err == nil || !strings.Contains(err.Error(), "aaaaaaaaaaaa") || !strings.Contains(err.Error(), "permissions violation") {
t.Fatalf("a failed put: %v", err)
}
}
// fallbackDelivery refuses the first declaration it is sent with the bucket's failure, as a bus that has not yet been given the
// controller's grant does, and takes the second.
type fallbackDelivery struct {
bodies [][]byte
first error
}
func (f *fallbackDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *fallbackDelivery) declare(_ context.Context, _ readyNode, body []byte) (string, error) {
f.bodies = append(f.bodies, body)
if len(f.bodies) == 1 && f.first != nil {
return "", fmt.Errorf("x: %w: %w", errFilesNotPut, f.first)
}
return "digest", nil
}
// 1696/AC5: when the bucket cannot be written yet (the grant travels in the declaration of the machine holding the bus), that send goes
// with every content inline and is made; a send that fails for another reason is not retried inline.
func TestASendWhoseFilesCannotBePutGoesInlineOnce(t *testing.T) { // 1696/AC5
users := contentOf("users", 100<<10)
s := readyNode{node: "novox", declared: sendable{fileReferences: true, Resources: []map[string]any{aFileResource("nats.bus-users", users)}}}
f := &fallbackDelivery{first: errors.New("timeout")}
digests, err := deliver(t.Context(), f, "novox", []readyNode{s})
if err != nil || digests["novox"] != "digest" || len(f.bodies) != 2 {
t.Fatalf("%v %v %d", digests, err, len(f.bodies))
}
if strings.Contains(string(f.bodies[0]), users[:100]) || !strings.Contains(string(f.bodies[1]), users[:100]) {
t.Error("the first body should name the file by digest and the second carry it inline")
}
other := &fallbackDelivery{}
other.first = nil
failing := &failingDelivery{err: errors.New("the bus is down")}
if _, err := deliver(t.Context(), failing, "novox", []readyNode{s}); err == nil || failing.calls != 1 {
t.Fatalf("a failure of another kind: %v after %d send(s)", err, failing.calls)
}
}
type failingDelivery struct {
err error
calls int
}
func (f *failingDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *failingDelivery) declare(context.Context, readyNode, []byte) (string, error) {
f.calls++
return "", f.err
}
// A bucket for the sweep.
type sweepBucketFake struct {
objects map[string]time.Time // zero: named at the last look
deleted []string
}
func (b *sweepBucketFake) Objects(context.Context) ([]bucketObject, error) {
var out []bucketObject
for n, at := range b.objects {
out = append(out, bucketObject{Name: n, UnnamedSince: at})
}
return out, nil
}
func (b *sweepBucketFake) MarkUnnamed(_ context.Context, n string, at time.Time) error {
b.objects[n] = at
return nil
}
func (b *sweepBucketFake) ClearUnnamed(_ context.Context, n string) error {
b.objects[n] = time.Time{}
return nil
}
func (b *sweepBucketFake) Delete(_ context.Context, n string, markedAt time.Time) (bool, error) {
if at, ok := b.objects[n]; !ok || !at.Equal(markedAt) {
return false, nil // put again since it was judged: its mark is gone
}
delete(b.objects, n)
b.deleted = append(b.deleted, n)
return true, nil
}
// 1696/AC6: an object a latest declaration still names is not removed, however long; one unnamed is marked at the first look and removed
// only when the mark is 24 hours old; one named again loses its mark.
func TestAnObjectIsRemovedADayAfterNoLatestDeclarationNamesIt(t *testing.T) { // 1696/AC6
named, unnamed, back := strings.Repeat("1", 64), strings.Repeat("2", 64), strings.Repeat("3", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{named: t0.Add(-100 * time.Hour), unnamed: {}, back: t0.Add(-5 * time.Hour)}}
named3 := map[string]bool{named: true, back: true}
r, err := sweepDeclarationFiles(t.Context(), b, named3, t0)
if err != nil || r.Marked != 1 || r.Cleared != 2 || r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("first look: %+v %v %v", r, err, b.deleted)
}
if !b.objects[named].IsZero() || !b.objects[back].IsZero() || !b.objects[unnamed].Equal(t0) {
t.Fatalf("marks after the first look: %v", b.objects)
}
// Still unnamed 23 hours 59 minutes later: kept. A day after the mark: removed. The named ones stay for ever.
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour-time.Minute)); err != nil ||
r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("a minute short of a day: %+v %v %v", r, err, b.deleted)
}
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour)); err != nil ||
r.Removed != 1 || len(b.deleted) != 1 || b.deleted[0] != unnamed {
t.Fatalf("a day after: %+v %v %v", r, err, b.deleted)
}
if _, ok := b.objects[named]; !ok {
t.Fatal("an object a latest declaration names was removed")
}
// An object named again before the day is up loses its mark and is not removed.
b = &sweepBucketFake{objects: map[string]time.Time{unnamed: t0}}
if _, err := sweepDeclarationFiles(t.Context(), b, map[string]bool{unnamed: true}, t0.Add(23*time.Hour)); err != nil || !b.objects[unnamed].IsZero() {
t.Fatalf("named again: %v %v", b.objects, err)
}
}
// What a node's latest declaration names is read from the signed envelope as stored on the bus; an unreadable one is an error and
// never "names none", which would mark an object a node needs.
func TestTheObjectsALatestDeclarationNamesAreReadFromTheBus(t *testing.T) {
s := sendable{fileReferences: true, Resources: []map[string]any{aFileResource("a", contentOf("a", 70<<10))}}
body, _ := s.Body()
msg, _ := link.Envelope(body, []byte("sig"))
named, err := namedByLatest("novox", msg)
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(contentOf("a", 70<<10))))
if err != nil || len(named) != 1 || !named[name] {
t.Fatalf("%v %v", named, err)
}
if none, err := namedByLatest("ace", []byte(`{"declaration":"e30=","signature":""}`)); err != nil || len(none) != 0 {
t.Fatalf("a declaration naming none: %v %v", none, err)
}
if _, err := namedByLatest("novox", []byte("not json")); err == nil {
t.Fatal("an unreadable latest declaration was read as naming none")
}
}
// 1698/AC2 (ADR 0486 §9): a declaration shaped like the control node's, for a node that reads references, stays under half of the
// bus's largest message on the wire; the inline form of the same does not fit at all; and the same shape with 40 percent more resources
// fails the test's own limit, so a change that eats the room fails in a pull request and not on the live mesh.
func TestTheControlNodesDeclarationInTheReferenceFormStaysUnderHalfTheBusLimit(t *testing.T) { // 1698/AC2
const limit = 1048576
shape := sendable{fileReferences: true, Resources: controlNodeShape(1)}
n := wireSizeOf(t, shape)
t.Logf("the control node's shape in the reference form: %d bytes on the wire, %.1f%% of %d", n, 100*float64(n)/limit, limit)
if n > limit/2 {
t.Errorf("the reference form is %d bytes on the wire, over half (%d) of the bus's largest message", n, limit/2)
}
if inline := wireSizeOf(t, sendable{Resources: controlNodeShape(1)}); inline < limit*3/4 {
t.Errorf("the inline form is %d bytes, which is not the shape that was over the limit", inline)
}
grown := wireSizeOf(t, sendable{fileReferences: true, Resources: controlNodeShape(1.4)})
if grown <= limit/2 {
t.Errorf("the shape with 40%% more resources is %d bytes, which the test would let pass", grown)
}
}
func stdBase64(b []byte) string { return base64.StdEncoding.EncodeToString(b) }
// A sweep that read an object's mark before a send put it again must not remove it: the object, now without the mark, is left.
func TestASweepDoesNotRemoveAnObjectASendPutAgainAfterItWasJudged(t *testing.T) { // 1696/AC8
n := strings.Repeat("4", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{n: t0}}
judged := bucketObject{Name: n, UnnamedSince: t0}
b.objects[n] = time.Time{} // the send put it again: no mark
if gone, err := b.Delete(t.Context(), judged.Name, judged.UnnamedSince); err != nil || gone || len(b.deleted) != 0 {
t.Fatalf("an object put again was removed: %v %v %v", gone, err, b.deleted)
}
}
// One node's bucket is held by one act at a time in this process: the sweep waits for a send that has put its objects and not yet
// published the declaration that names them (1696/AC9).
func TestTheSweepOfANodesBucketWaitsForASendThatHasPutItsObjects(t *testing.T) { // 1696/AC9
unlock := lockFilesOf("lock-test-node")
got := make(chan struct{})
go func() {
defer lockFilesOf("lock-test-node")()
close(got)
}()
select {
case <-got:
t.Fatal("the sweep took the node's bucket while a send held it")
case <-time.After(50 * time.Millisecond):
}
other := lockFilesOf("lock-test-other-node") // another node's bucket is not held
other()
unlock()
select {
case <-got:
case <-time.After(2 * time.Second):
t.Fatal("the sweep did not take the node's bucket once the send was done")
}
}
// A put that fails send after send raises a condition; the first failure does not (the grant it waits for travels in that very
// declaration), and a put that works clears it (1696/AC10).
func TestAPutThatFailsSendAfterSendRaisesAConditionAndClearsWhenItWorks(t *testing.T) { // 1696/AC10
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return time.Date(2026, 10, 21, 14, 0, 0, 0, time.UTC) }})
t.Cleanup(func() { k.Close(t.Context()) })
was := conditionsFrom
conditionsFrom = k
t.Cleanup(func() { conditionsFrom = was })
key := filesNotPutObservation("cond-node", "").Key()
open := func() bool {
_, found, err := conditions.ReadOne(t.Context(), store, key)
if err != nil {
t.Fatal(err)
}
return found
}
boom := errors.New("permissions violation")
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the first failure raised the condition")
}
notePut(t.Context(), "cond-node", boom)
if !open() {
t.Fatal("the second failure in a row did not raise the condition")
}
notePut(t.Context(), "cond-node", nil)
if open() {
t.Fatal("a put that worked did not clear the condition")
}
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the count did not start again after a put that worked")
}
}
// aBusThatSays is a bus that keeps the declarations published to it and, at each, asks whether the node's bucket is held.
type aBusThatSays struct {
link.Bus
published chan string
heldAtPublish func() bool
}
func (b *aBusThatSays) PublishDeclaration(_ context.Context, node string, _ []byte) error {
if b.heldAtPublish != nil && !b.heldAtPublish() {
return errors.New("published while the node's bucket was not held")
}
b.published <- node
return errors.New("stop here: nothing is recorded")
}
type aSignerOfZeros struct{}
func (aSignerOfZeros) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// whileHeld says whether the node's lock is held by someone right now, without keeping it.
func whileHeld(node string) bool {
got := make(chan struct{})
go func() {
defer lockFilesOf(node)()
close(got)
}()
select {
case <-got:
return false
case <-time.After(100 * time.Millisecond):
// held: the goroutine stays waiting and takes and releases the lock once it is let go.
return true
}
}
// 1696/AC9: a send holds the node's lock from the put to the publication, so the sweep never removes an object between them. The
// test goes through overTheBus.declare: while another holds the lock the declaration is not published, and at the publication the
// lock is held.
func TestASendHoldsTheNodesLockFromThePutToThePublication(t *testing.T) { // 1696/AC9
const node = "lock-send-node"
bus := &aBusThatSays{published: make(chan string, 1)}
bus.heldAtPublish = func() bool { return whileHeld(node) }
send := overTheBus{server: link.SendingOn(bus), signer: aSignerOfZeros{}}
release := lockFilesOf(node) // a sweep is judging this node's bucket
done := make(chan error, 1)
go func() {
_, err := send.declare(t.Context(), readyNode{node: node}, []byte(`{}`))
done <- err
}()
select {
case n := <-bus.published:
t.Fatalf("%s was sent its declaration while the sweep held the node's bucket", n)
case <-time.After(150 * time.Millisecond):
}
release()
select {
case <-bus.published:
case <-time.After(5 * time.Second):
t.Fatal("the send did not go on once the sweep let go")
}
if err := <-done; err == nil || !strings.Contains(err.Error(), "nothing is recorded") {
t.Fatalf("the send did not reach the publication (a lock held at it): %v", err)
}
}
+2 -2
View File
@@ -13,8 +13,8 @@ import (
// **A declaration is one bus message, and the bus's largest message is a cliff** (novox/hq issue 1689). On 2026-10-21
// the control node's declaration passed it by 2 KB and nothing reached that node until two assignments were undone.
// So the size is measured at every send: a warning condition from 90% of the limit, urgent over it, cleared when
// the declaration is back under 90%. The limit is the connection's own, not a constant.
// So the size is measured at every send: a warning condition from 75% of the limit (ADR 0486 §8, issue 1698: it was 90% while every file travelled inline), urgent over it, cleared when
// the declaration is back under 75%. The limit is the connection's own, not a constant.
// kindDeclarationNearLimit is a declaration at or near the bus's largest message.
const kindDeclarationNearLimit = "declaration-near-limit"
+8 -8
View File
@@ -33,7 +33,7 @@ func TestTheVaultsExportIsSentCompactAndNothingElseIsTouched(t *testing.T) {
}
}
// From 90% of the bus's largest message the declaration is a warning; over it, urgent; back under 90%, cleared.
// From 75% of the bus's largest message (ADR 0486 §8, issue 1698) the declaration is a warning; over it, urgent; back under 75%, cleared.
func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *testing.T) {
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
@@ -50,22 +50,22 @@ func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *test
return c, found
}
settleDeclarationRoom(t.Context(), k, "novox", 899, limit)
settleDeclarationRoom(t.Context(), k, "novox", 749, limit)
if _, found := read(); found {
t.Fatal("a declaration at 89.9% raised a condition")
t.Fatal("a declaration at 74.9% raised a condition")
}
settleDeclarationRoom(t.Context(), k, "novox", 900, limit)
settleDeclarationRoom(t.Context(), k, "novox", 750, limit)
c, found := read()
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "900 bytes") ||
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "750 bytes") ||
!strings.Contains(c.Summary, "1000") || c.Subject.Machine != "novox" {
t.Fatalf("at 90%% the condition is %v %+v", found, c)
t.Fatalf("at 75%% the condition is %v %+v", found, c)
}
settleDeclarationRoom(t.Context(), k, "novox", 1001, limit)
if c, _ := read(); c.Severity != conditions.Urgent {
t.Fatalf("over the limit the condition is %q, expected urgent", c.Severity)
}
settleDeclarationRoom(t.Context(), k, "novox", 850, limit)
settleDeclarationRoom(t.Context(), k, "novox", 700, limit)
if _, found := read(); found {
t.Fatal("the condition stayed after the declaration fell under 90%")
t.Fatal("the condition stayed after the declaration fell under 75%")
}
}
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **An object is removed a day after no node's latest declaration names it** (novox/hq ADR 0486 §3, issue 1696), so a node that is
// asleep and receives its latest declaration later still finds every object that declaration names. What each node's latest
// declaration names is read from the bus itself, the last message of that node's subject of the stream NODES: nothing else
// is kept of it. When an object is first found named by none, the time is written in its own metadata (declarationUnnamed);
// found named again, that is erased; found unnamed for a day, it is removed. Each node has a bucket of its own, judged against that
// node's latest declaration alone. Objects are immutable and regenerable from the
// controller's state, so a removal loses nothing.
// declarationUnnamed is the key of an object's metadata holding when it was first found named by no node's latest declaration.
const declarationUnnamed = "unnamed-since"
// sweepEvery is how often the bucket is judged: the removal is exact to within this.
const sweepEvery = time.Hour
// bucketObject is one object of the bucket as the sweep sees it.
type bucketObject struct {
Name string
// UnnamedSince is when it was first found named by none, zero when it was named at the last look.
UnnamedSince time.Time
}
// sweptBucket is what the sweep needs of the bucket.
type sweptBucket interface {
Objects(ctx context.Context) ([]bucketObject, error)
MarkUnnamed(ctx context.Context, name string, since time.Time) error
ClearUnnamed(ctx context.Context, name string) error
// Delete removes the object if it still carries the mark it was judged by, and says whether it did: a send that put it again
// since has erased the mark, and the object is left.
Delete(ctx context.Context, name string, markedAt time.Time) (bool, error)
}
// filesSweep says what a sweep did.
type filesSweep struct{ Kept, Marked, Cleared, Removed int }
// sweepDeclarationFiles judges every object against the names of the latest declarations: named ones are kept (and a
// mark of being unnamed is erased), unnamed ones are marked at the first look and removed once the mark is a day old.
func sweepDeclarationFiles(ctx context.Context, bucket sweptBucket, named map[string]bool, now time.Time) (filesSweep, error) {
var r filesSweep
objects, err := bucket.Objects(ctx)
if err != nil {
return r, err
}
sort.Slice(objects, func(i, j int) bool { return objects[i].Name < objects[j].Name })
for _, o := range objects {
switch {
case named[o.Name]:
r.Kept++
if !o.UnnamedSince.IsZero() {
if err := bucket.ClearUnnamed(ctx, o.Name); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Cleared++
}
case o.UnnamedSince.IsZero():
if err := bucket.MarkUnnamed(ctx, o.Name, now); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Marked++
case now.Sub(o.UnnamedSince) >= broker.DeclarationFilesKeptFor:
gone, err := bucket.Delete(ctx, o.Name, o.UnnamedSince)
if err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
if gone {
r.Removed++
} else {
r.Kept++
}
default:
r.Kept++
}
}
return r, nil
}
// namedByLatest is the name of every object this latest declaration of a node names. A body that cannot be read is an error, never
// "names none": an object the node still needs would be marked for removal.
func namedByLatest(node string, signed []byte) (map[string]bool, error) {
var s link.Signed
if err := json.Unmarshal(signed, &s); err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
names, err := namedObjects(s.Declaration)
if err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
named := make(map[string]bool, len(names))
for name := range names {
named[name] = true
}
return named, nil
}
// latestDeclarations are the last signed declarations of the nodes, by node, as the stream NODES holds them. A node with none has
// no entry.
func latestDeclarations(ctx context.Context, conn *nats.Conn, nodes []string) (map[string][]byte, error) {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return nil, err
}
out := map[string][]byte{}
for _, node := range nodes {
msg, err := js.GetLastMsg("NODES", link.DeclareSubject(node), nats.Context(ctx))
if errors.Is(err, nats.ErrMsgNotFound) {
continue
}
if err != nil {
return nil, fmt.Errorf("the latest declaration of %s: %w", node, err)
}
out[node] = msg.Data
}
return out, nil
}
// busSweptBucket is the bucket on the controller's connection.
type busSweptBucket struct{ store jetstream.ObjectStore }
func (b busSweptBucket) Objects(ctx context.Context) ([]bucketObject, error) {
infos, err := b.store.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return nil, nil
}
if err != nil {
return nil, err
}
out := make([]bucketObject, 0, len(infos))
for _, info := range infos {
o := bucketObject{Name: info.Name}
if at, err := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); err == nil {
o.UnnamedSince = at
}
out = append(out, o)
}
return out, nil
}
func (b busSweptBucket) meta(ctx context.Context, name string, change func(map[string]string)) error {
info, err := b.store.GetInfo(ctx, name)
if err != nil {
return err
}
meta := info.ObjectMeta
kept := map[string]string{}
for k, v := range meta.Metadata {
kept[k] = v
}
change(kept)
meta.Metadata = kept
return b.store.UpdateMeta(ctx, name, meta)
}
func (b busSweptBucket) MarkUnnamed(ctx context.Context, name string, since time.Time) error {
return b.meta(ctx, name, func(m map[string]string) { m[declarationUnnamed] = since.UTC().Format(time.RFC3339) })
}
func (b busSweptBucket) ClearUnnamed(ctx context.Context, name string) error {
return b.meta(ctx, name, func(m map[string]string) { delete(m, declarationUnnamed) })
}
func (b busSweptBucket) Delete(ctx context.Context, name string, markedAt time.Time) (bool, error) {
info, err := b.store.GetInfo(ctx, name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return false, nil
}
if err != nil {
return false, err
}
if at, perr := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); perr != nil || !at.Equal(markedAt) {
return false, nil
}
return true, b.store.Delete(ctx, name)
}
// sweepingDeclarationFiles judges the bucket every sweepEvery for as long as this controller serves and holds the lease.
func sweepingDeclarationFiles(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) {
tick := time.NewTicker(sweepEvery)
defer tick.Stop()
failing := ""
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
if epoch, err := theLease.epoch(ctx); err != nil || epoch == 0 {
continue
}
why := sweepOnce(ctx, inv, conn)
if why != failing {
if why != "" {
fmt.Printf("the bucket of declaration files cannot be swept: %s\n", why)
} else {
fmt.Printf("the bucket of declaration files is swept again\n")
}
failing = why
}
}
}
// sweepOnce judges the bucket of every node against that node's latest declaration, one bucket at a time under the node's lock, with
// the declaration read again under it: a send that put an object and published its declaration is seen, and one that has not yet
// is waiting for the lock.
func sweepOnce(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) string {
nodes, err := inv.Nodes(ctx)
if err != nil {
return fmt.Sprintf("the nodes cannot be read: %v", err)
}
var problems []string
var total filesSweep
for _, n := range nodes {
r, err := sweepNode(ctx, conn, n.Name)
total.Kept, total.Marked, total.Cleared, total.Removed = total.Kept+r.Kept, total.Marked+r.Marked, total.Cleared+r.Cleared, total.Removed+r.Removed
if err != nil {
problems = append(problems, fmt.Sprintf("%s: %v", n.Name, err))
}
}
if total.Removed+total.Marked+total.Cleared > 0 {
fmt.Printf("the buckets of declaration files: %d object(s) kept, %d found unnamed, %d named again, %d removed\n",
total.Kept, total.Marked, total.Cleared, total.Removed)
}
return strings.Join(problems, "; ")
}
func sweepNode(ctx context.Context, conn *nats.Conn, node string) (filesSweep, error) {
working, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
js, err := jetstream.New(conn)
if err != nil {
return filesSweep{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(working, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return filesSweep{}, nil // a node never sent a reference has no bucket yet
}
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket cannot be read: %w", err)
}
defer lockFilesOf(node)()
latest, err := latestDeclarations(working, conn, []string{node})
if err != nil {
return filesSweep{}, err
}
named := map[string]bool{}
if body, ok := latest[node]; ok {
if named, err = namedByLatest(node, body); err != nil {
return filesSweep{}, err
}
}
return sweepDeclarationFiles(working, busSweptBucket{store}, named, time.Now())
}
+7 -1
View File
@@ -1177,6 +1177,12 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", err
}
request.Check.Group = group
// **The values held for the keys the members introduce are told and carried** (novox/hq ADR 0502, issue 1826).
var pulls []pullAt
for i, s := range changeSourcesOf(members, entries, reach) {
pulls = append(pulls, pullAt{changeSource: s, Number: members[i].pull().Number})
}
withHeldConfig(ctx, os.Stderr, &request, pulls, true)
// A group's check is queued as ready when any member's pull request is marked ready.
request.Check.Ready = slices.ContainsFunc(members, func(m orderMember) bool { return m.Ready })
world, err := restOfTheMesh(ctx, open, shelfOf(entries), "")
@@ -1204,7 +1210,7 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base, Trunk: mp.BaseSHA})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
@@ -59,6 +59,18 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
}
}
// heardOf has this mesh's own view of the plans take in the plan as the bucket holds it. A verb of the
// controller (deliveryCommand, plansCommand) opens an inventory of its own and writes the plan there; this
// mesh's inventory learns of the write from the bucket's watch, a moment later, and a loaded machine makes the
// moment long enough for advanceHeld to read the plan as it was, still waiting for its word, and ask nothing
// (issue 1399). Reading the plan from the bucket puts it into the view at once, with no wait to bound.
heardOf := func(id string) {
t.Helper()
if _, err := inv.PlanByID(ctx, id); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
@@ -90,6 +102,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
heardOf(p.ID)
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
@@ -119,6 +132,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
heardOf(p.ID)
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
+1 -1
View File
@@ -68,7 +68,7 @@ func TestTheForgeNotAnsweringInTimeIsNotAskedAgain(t *testing.T) {
// An ask that hangs past the bound is cut off at the bound, not at its own wait, and the failure still says it
// waited: the receive loop is held no longer than the bound (novox/hq issue 486).
func TestAnAskHangingPastTheBoundIsCutOffAtIt(t *testing.T) {
w := waitOut{bound: 200 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
w := waitOut{bound: 100 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
now: time.Now, sleep: sleepOrDone}
asked := 0
began := time.Now()
+198
View File
@@ -0,0 +1,198 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A move held for a person is asked of the person (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
//
// **A wait only a person can end is said to that person at once.** ADR 0377 holds a recorded module's move that
// recreates a container or touches its data until a person pushes the machine. The walk said so in its note and
// ended "done", `status` named the machine behind, and nobody was asked: on 2026-10-21 mailu's new build waited on
// novox for 40 minutes behind a plan that read "done, 3 tier(s)", and on 2026-10-19 mssql waited on ace the same way.
// Row S24 raises one condition per machine that has such a move, for the operator, with the answer **Release**: the
// push of that machine, which the controller's own `push` verb performs on the operator's warrant (ADR 0259). It ends
// when the machine no longer waits, whoever pushed it.
//
// A hold a person chose is not asked again — a module they set to `record` with a why, and the bus, whose step has
// its own row (S17).
// kindMovesHeld is S24's kind: machine.<node>.moves-held.
const kindMovesHeld = "moves-held"
// heldFacts is one machine and the moves of recorded modules that wait there for a person.
type heldFacts struct {
node string
moves []heldMove
}
// gatherHeldMoves is, per machine that holds at least one move no person chose to hold, all its held moves, by machine.
func gatherHeldMoves(ctx context.Context, inv *inventory.Inventory) ([]heldFacts, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("the machines cannot be read: %w", err)
}
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
}
var out []heldFacts
for _, n := range nodes {
moves, err := heldMovesOn(ctx, inv, f, n.Name)
if err != nil {
return nil, err
}
h := heldFacts{node: n.Name, moves: moves}
if len(h.asked()) > 0 {
out = append(out, h)
}
}
return out, nil
}
// asked are the moves no person chose to hold, and chosen the ones a person did (a module set to record with a why, the bus).
func (h heldFacts) asked() []heldMove { return h.byChoice(false) }
func (h heldFacts) chosen() []heldMove { return h.byChoice(true) }
func (h heldFacts) byChoice(chosen bool) []heldMove {
var out []heldMove
for _, m := range h.moves {
if m.Chosen == chosen {
out = append(out, m)
}
}
return out
}
// modules are the held moves' modules, in order, of all of them.
func (h heldFacts) modules() []string { return moduleNames(h.moves) }
func moduleNames(moves []heldMove) []string {
out := make([]string, 0, len(moves))
for _, m := range moves {
out = append(out, m.Module)
}
return out
}
// releaseAction is the one answer a held move offers: the push of the machine, approved by the operator. The push
// names every module whose move waits there as one that may move its data: the operator's Approve is the word
// ADR 0217 asks for, and the ask says so. Offered only where no person's own hold is on the machine too: a push
// composes every recorded module at its current build, so it would send that move as well, which the ask never named
// (ADR 0259 §6), and hold the whole machine if that module's data would move.
func (h heldFacts) releaseAction() conditions.Action {
return conditions.Action{Label: "Release", Verb: "mesh-controller.push", Level: conditions.LevelApprove,
Arguments: map[string]string{"node": h.node, "move": strings.Join(moduleNames(h.asked()), ","), "why": "",
"cause": conditions.CauseOperatorAnswer}}
}
// heldWords are the plain words of a machine's held moves: which modules wait, why in general, and what an Approve does.
func heldWords(h heldFacts) (headline, explanation, needs string, resolved string) {
names := namesWords(moduleNames(h.asked()), 3)
headline = clipWords(conditions.Capital(names)+" waits for you on "+h.node, conditions.HeadlineMax)
recreates, data := false, false
for _, m := range h.asked() {
switch {
case strings.Contains(m.Why, "recreates"):
recreates = true
default:
data = true
}
}
why := "would change what it runs"
switch {
case recreates && data:
why = "would restart a container or touch its data"
case recreates:
why = "would restart one of its containers"
case data:
why = "would touch its files or data"
}
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. Release sends %s everything it is owed, these among it, and lets their data move where it "+
"has to. Nothing is sent until you answer.", names, why, h.node, h.node), conditions.ExplanationMax)
needs = "release them " + FromMeshMCPServer
if chosen := h.chosen(); len(chosen) > 0 {
// A push of the machine would send the move a person chose to hold as well: no answer is offered here.
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. %s is also held there by your own choice, and a push of %s would send that too, so "+
"nothing is offered here: the push is yours at the controller's terminal.",
names, why, h.node, conditions.Capital(namesWords(moduleNames(chosen), 3)), h.node), conditions.ExplanationMax)
needs = fmt.Sprintf("push %s at the controller's terminal, once you have decided about %s.", h.node,
namesWords(moduleNames(chosen), 3))
}
resolved = "Resolved: " + names + " is no longer waiting on " + h.node
return
}
// watchMovesHeld is S24: a machine that has moves of recorded modules waiting for a person, said to the operator with the
// answer that sends them. The condition ends when the machine no longer waits: it is no longer among the facts.
func watchMovesHeld(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, h := range f.held {
var said []string
for _, m := range h.moves {
said = append(said, fmt.Sprintf("%s %s → %s %s", m.Module, short(m.From), short(m.To), m.Why))
}
headline, explanation, needs, resolved := heldWords(h)
if c := h.chosen(); len(c) > 0 {
said = append(said, "held by a person's own choice: "+strings.Join(moduleNames(c), ", "))
}
var actions []conditions.Action
if len(h.chosen()) == 0 {
actions = []conditions.Action{h.releaseAction()}
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: h.node, Kind: kindMovesHeld,
Machine: h.node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s has %d move(s) of recorded modules waiting for a person's push: %s — `push %s` sends them",
h.node, len(h.moves), strings.Join(h.modules(), ", "), h.node),
Said: strings.Join(said, "; "),
Headline: headline, Explanation: explanation, Needs: needs, Resolved: resolved,
Actions: actions})
}
return out
}
// planHeld is, for a plan that ended or is running, the moves its modules carried that a machine still holds for a
// person: "module on machine", sorted. The plan's own record says the move waited when it was judged
// (PlanModule.Recorded); the machines say whether it still does, and for the build this plan carried, so a plan older
// than the build now held is not named.
func planHeld(p inventory.Plan, held []heldFacts) []string {
var out []string
for _, h := range held {
for _, m := range h.moves {
s := p.Modules[m.Module]
if s == nil || !strings.HasPrefix(s.Recorded[h.node], "waits:") || !sameCommit(s.Commit, m.To) {
continue
}
out = append(out, m.Module+" on "+h.node)
}
}
sort.Strings(out)
return out
}
// heldSaid is what a plan's line adds when it carries a move that a machine holds for a person (novox/hq issue 1732): a
// plan whose module waits for a push is not plainly "done".
func heldSaid(p inventory.Plan, held []heldFacts) string {
waiting := planHeld(p, held)
if len(waiting) == 0 {
return ""
}
return fmt.Sprintf(" — %d held for a person: %s (the operator is asked)", len(waiting), strings.Join(waiting, ", "))
}
// heldNow reads the machines' held moves for a plan's line, nil when they cannot be read: the line then reads as before.
func heldNow(ctx context.Context, inv *inventory.Inventory) []heldFacts {
held, err := gatherHeldMoves(ctx, inv)
if err != nil {
return nil // empty-on-error: a plan's line then reads as it did before; the watchdog (S24) says it cannot read
}
return held
}
@@ -0,0 +1,92 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func inventoryUpgrade(why string) inventory.Upgrade { return inventory.Upgrade{Why: why, By: "jochen"} }
func mustMoveFacts(t *testing.T, rm *recordedMesh) moveFacts {
t.Helper()
f, err := readMoveFacts(t.Context(), rm.open.inventory)
if err != nil {
t.Fatal(err)
}
return f
}
// 1732/AC8 (integration): a mesh running mailu at c1 on two machines, whose registered c2 recreates a container, is read
// as two machines each holding mailu's move for a person; a bundle-only c3 on top changes nothing of it; after a machine
// is sent c2 (as a push records it) it is no longer among them, and plans that carried c2 say nothing more of it.
func TestMachinesHoldingAMoveForAPersonAreReadFromTheRecords(t *testing.T) {
rm := aRecordedMesh(t, aMailu(true))
c2 := aMailu(true)
resourceOf(&c2, "imap")["image"] = "registry.invalid/imap@sha256:2"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
ctx := t.Context()
held, err := gatherHeldMoves(ctx, rm.open.inventory)
if err != nil {
t.Fatal(err)
}
if len(held) != 2 || held[0].node != "anchor" || held[1].node != "laptop" || len(held[0].moves) != 1 ||
held[0].moves[0].Module != "mailu" || held[0].moves[0].To != "c2" || !strings.Contains(held[0].moves[0].Why, "recreates") {
t.Fatalf("held %+v", held)
}
if err := rm.open.inventory.RecordSent(ctx, nodeID(t, rm.open, "anchor"), "d-anchor-c2", map[string]string{"mailu": "c2"}); err != nil {
t.Fatal(err)
}
held, err = gatherHeldMoves(ctx, rm.open.inventory)
if err != nil || len(held) != 1 || held[0].node != "laptop" {
t.Fatalf("after anchor was sent c2: %+v (%v)", held, err)
}
}
// 1732/AC9 (integration): a module a person set to record is no ask: they chose the hold.
func TestAHoldAPersonChoseIsNotAsked(t *testing.T) {
rm := aRecordedMesh(t, aMailu(false))
c2 := aMailu(false)
c2.Bundles[0].Digest = "sha256:bbb"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
if err := rm.open.inventory.SetUpgradeOf(t.Context(), "mailu", inventoryUpgrade("the migration week")); err != nil {
t.Fatal(err)
}
held, err := gatherHeldMoves(t.Context(), rm.open.inventory)
if err != nil || len(held) != 0 {
t.Fatalf("a machine holding only a person's own hold is asked: %+v (%v)", held, err)
}
if moves, _ := heldMovesOn(t.Context(), rm.open.inventory, mustMoveFacts(t, rm), "anchor"); len(moves) != 1 || !moves[0].Chosen {
t.Fatalf("status still names it, marked as chosen: %+v", moves)
}
}
// 1742/AC4 (integration): a registered build naming an image of the store that the store lacks is read as a missing
// image on each machine that runs the module; one the store holds is not; and what the store holds is not asked twice.
func TestARegisteredBuildNamingAnImageTheStoreLacksIsRead(t *testing.T) {
m := aMailu(true)
resourceOf(&m, "imap")["image"] = "artifact-store://mailu/imap@sha256:" + strings.Repeat("e", 64)
rm := aRecordedMesh(t, m)
asked := 0
lacking := func(_ context.Context, reference string) (bool, error) { asked++; return false, nil }
imagesHeld.Lock()
imagesHeld.held = map[string]bool{}
imagesHeld.Unlock()
got, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking)
if err != nil || len(got) != 2 || got[0].Module != "mailu" || got[0].Node != "anchor" || got[1].Node != "laptop" {
t.Fatalf("missing %+v (%v)", got, err)
}
holding := func(_ context.Context, reference string) (bool, error) { asked++; return true, nil }
if got, err := gatherMissingImages(t.Context(), rm.open.inventory, holding); err != nil || len(got) != 0 {
t.Fatalf("an image the store holds is missing: %+v (%v)", got, err)
}
before := asked
if _, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking); err != nil || asked != before {
t.Fatalf("an image the store was found to hold was asked about again (%d, %v)", asked-before, err)
}
}
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A move held for a person is asked of the person** (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
// aHeldMailu is novox holding mailu's new build because it would recreate a container: the walk's own words.
func aHeldMailu() heldFacts {
return heldFacts{node: "novox", moves: []heldMove{{Module: "mailu", From: "b2f5ebb2aaaa", To: "3259765dbbbb",
Why: "waits: recreates 1 of mailu's 12 container(s) (with a new image: automx)"}}}
}
func heldConditionOn(f *signalFacts) conditions.Condition {
o := watchMovesHeld(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1732/AC1: a machine holding a move of a recorded module for a person raises one condition for the operator, with
// Release as its only answer: the machine's push, approved, naming the modules whose moves wait there.
func TestAHeldMoveIsRaisedWithReleaseAsItsOnlyAction(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
got := watchMovesHeld(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "moves-held" || o.Scope != conditions.ScopeMachine || o.ID != "novox" || o.Severity != conditions.Warning ||
o.Resolver != conditions.ResolverOperator || o.Key() != "machine.novox.moves-held" {
t.Fatalf("kind, scope, subject, severity or resolver: %+v (%s)", o, o.Key())
}
if len(o.Actions) != 1 {
t.Fatalf("actions: %+v", o.Actions)
}
a := o.Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.push" || a.Level != conditions.LevelApprove ||
a.Arguments["node"] != "novox" || a.Arguments["move"] != "mailu" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action is not the push of novox: %+v", a)
}
if _, takes := a.Arguments["why"]; !takes {
t.Fatal("the push takes a why, which the asker fills in with the operator's answer")
}
for _, want := range []string{"mailu", "novox"} {
if !strings.Contains(o.Headline+o.Explanation, want) {
t.Errorf("the words do not name %q: %q / %q", want, o.Headline, o.Explanation)
}
}
if !strings.Contains(o.Said, "recreates 1 of mailu's 12") {
t.Errorf("the evidence does not carry the walk's own words: %q", o.Said)
}
if w := (conditions.Observation{Headline: o.Headline, Explanation: o.Explanation, Needs: o.Needs, Resolved: o.Resolved}); w.Headline == "" || len(o.Headline) > conditions.HeadlineMax {
t.Errorf("headline: %q", o.Headline)
}
}
// 1732/AC2: a machine with nothing held raises nothing, so the condition ends the look after the move is sent.
func TestAMachineHoldingNothingRaisesNoHeldMoveCondition(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
if got := watchMovesHeld(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
}
// 1732/AC3: the condition is asked on a channel at the approve level in words that say what Release does, and the
// operator's warrant calls the controller's own push with the machine and the modules, once.
func TestAHeldMovesReleaseIsAskedAndAWarrantPushesTheMachine(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
c := heldConditionOn(f)
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Release" || o.Level != asks.Approve || o.Does != "the held builds are sent to novox" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.push@ ") ||
!strings.Contains(r.called[0], `"node":"novox"`) || !strings.Contains(r.called[0], `"move":"mailu"`) ||
!strings.Contains(r.called[0], `"cause":"`+conditions.CauseOperatorAnswer+`"`) ||
!strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC4: the controller may call its own push and upgrade verbs on a warrant: the grant names them.
func TestTheControllerMayActOnAWarrantForPushAndUpgrade(t *testing.T) {
for _, verb := range []string{"push", "upgrade"} {
granted := false
for _, v := range broker.VerbsTheControllerActsOnAWarrant {
granted = granted || (v.Seat == broker.ControllerSeat && v.Verb == verb)
}
if !granted {
t.Errorf("the controller may not act on a warrant for %s", verb)
}
}
}
func aDoneMailuPlan() inventory.Plan {
return inventory.Plan{ID: "plan-1792585970446304628", Repository: "novox/mesh-catalog", State: inventory.PlanDone,
Tiers: [][]string{{"a"}, {"b"}, {"mailu"}},
Modules: map[string]*inventory.PlanModule{"mailu": {State: "built", Commit: "3259765dbbbbcccc",
Recorded: map[string]string{"novox": "waits: recreates 1 of mailu's 12 container(s)"}}}}
}
// 1732/AC5: a done plan that carries a move a machine still holds for a person says so on its line; once the machine
// no longer holds it, or holds another build of the module, the line says nothing more.
func TestADonePlanSaysWhatItHoldsForAPerson(t *testing.T) {
p := aDoneMailuPlan()
line := planLine(p, time.Date(2026, 10, 21, 16, 0, 0, 0, time.UTC)) + heldSaid(p, []heldFacts{aHeldMailu()})
if !strings.Contains(line, "done, 3 tier(s)") || !strings.Contains(line, "1 held for a person: mailu on novox") {
t.Errorf("the line: %q", line)
}
if said := heldSaid(p, nil); said != "" {
t.Errorf("a plan whose moves were sent says %q", said)
}
other := aHeldMailu()
other.moves[0].To = "9999999ffff"
if said := heldSaid(p, []heldFacts{other}); said != "" {
t.Errorf("a plan older than the build now held says %q", said)
}
p.Modules["mailu"].Recorded = map[string]string{"novox": recordedRollsWhy}
if said := heldSaid(p, []heldFacts{aHeldMailu()}); said != "" {
t.Errorf("a move the plan sent says %q", said)
}
}
// 1732/AC6: the builds held after a failed release are asked of the operator: the held-release condition offers
// Release, which calls the controller's own `upgrade` with release-backlog and nothing else, once, on a warrant.
func TestTheBuildsHeldAfterAFailedReleaseAreAskedAndReleasedOnAWarrant(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
backlogNow.held = "release-1792595552499982311 failed (failed on novox); what waits is released again by a person"
backlogNow.waiting = map[string][]inventory.CarriedMove{"novox": {{Module: "indexer"}, {Module: "nextcloud"}}}
got := backlogObservation()
if len(got) != 1 || len(got[0].Actions) != 1 {
t.Fatalf("raised %+v", got)
}
a := got[0].Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.upgrade" || a.Level != conditions.LevelApprove ||
a.Arguments["release-backlog"] != "true" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action: %+v", a)
}
o := got[0]
c := conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 || sent[0].Options[0].Does != "the builds held after the failed delivery are released" {
t.Fatalf("asked %+v", sent)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.upgrade@ ") ||
!strings.Contains(r.called[0], `"release-backlog":"true"`) || !strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC7: a plan whose send waits for a gate while the builds are held after a failed release reads as waiting for
// a person to release them, not "building ... LATE"; with nothing held it still reads as the wait it is.
func TestAPlanWaitingForAGateBehindAFailedReleaseIsNotLate(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
now := time.Date(2026, 10, 21, 18, 50, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Repository: "novox/mesh-sdk", Commit: "c0ffee00", State: inventory.PlanBuilding,
Tiers: [][]string{{"indexer"}}, TierEntered: now.Add(-100 * time.Minute), Created: now.Add(-110 * time.Minute),
Modules: map[string]*inventory.PlanModule{"indexer": {State: "built"}},
Note: refusalNote(0, errors.New("a build waits there for its gate: novox"))}
backlogNow.held = ""
if line := planLine(p, now); !strings.Contains(line, "a build waits there for its gate: novox") {
t.Errorf("the wait is not said: %q", line)
}
backlogNow.held = "release-1 failed (failed on novox); what waits is released again by a person"
line := planLine(p, now)
if strings.Contains(line, "LATE") || strings.Contains(line, "building") ||
!strings.Contains(line, "until a person releases it") {
t.Errorf("a plan held behind a failed release reads %q", line)
}
}
// 1732/AC10: a machine that also holds a move a person chose to hold gets no Release, since a push would send that move
// too; the condition names both and says the push is the controller's terminal. A machine with only asked moves keeps it.
func TestAMachineAlsoHoldingAPersonsOwnHoldIsOfferedNoRelease(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
h := aHeldMailu()
h.moves = append(h.moves, heldMove{Module: "nftables", From: "n1aaaa", To: "n2bbbb",
Why: "waits: a person set it to record, and holds every move of it until they push", Chosen: true})
f.held = []heldFacts{h}
got := watchMovesHeld(f)
if len(got) != 1 || len(got[0].Actions) != 0 {
t.Fatalf("%+v", got)
}
o := got[0]
text := o.Headline + o.Explanation + o.Needs + o.Summary + o.Said
for _, want := range []string{"mailu", "nftables", "terminal", "push novox"} {
if !strings.Contains(text, want) {
t.Errorf("does not say %q: %q", want, text)
}
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
f.held = []heldFacts{aHeldMailu()}
if got := watchMovesHeld(f); len(got) != 1 || len(got[0].Actions) != 1 || got[0].Actions[0].Arguments["move"] != "mailu" {
t.Fatalf("a machine with only asked moves: %+v", got)
}
}
+245
View File
@@ -0,0 +1,245 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// No declaration is sent that names an image the artifact store does not hold (novox/hq issue 1742).
//
// On 2026-10-21 at 17:13 a send to novox named `mailu/automx@sha256:c62150b8…`; the build record keeps that digest
// ("recent-build") and the store did not hold it. The node-engine removed the running container, then could not pull
// the replacement, and every later apply on novox failed for the next hour while the mail clients' autoconfiguration
// had no backend. The node-engine's half (pull first, replace after) is its own change; this is the controller's: **a
// module whose image the store lacks is left out of the declaration**, as a module whose setting nobody gave is
// (ADR 0163, rule 6) — the node-engine keeps what it has of it and touches none of its containers — and **it is said**,
// row S25: a condition naming the module, the machine and the image, until the store holds the image again (a new build
// of the module, or the copy put back).
//
// Judged only for an image the store's own address names, from the send's own resources; an image of another registry
// is no business of the store's. A store that cannot be asked judges nothing: a send is not refused for a question
// that did not answer.
// kindImageMissing is S25's kind: module.<module>.<machine>.image-missing.
const kindImageMissing = "image-missing"
// missingImage is a module held out of a machine's declaration because the store lacks an image it names.
type missingImage struct {
Node, Module, Image string
}
// storeHolds is whether the store holds the manifest an image reference at its address names.
type storeHolds func(ctx context.Context, reference string) (bool, error)
// imageInStore is the artifact-store reference of an image the store's address names, "" for any other.
func imageInStore(image, address string) string {
if address == "" || !strings.HasPrefix(image, address+"/") || !strings.Contains(image, "@sha256:") {
return ""
}
reference := catalogue.Recorded(image)
if _, ours := catalogue.InArtifactStore(reference); !ours {
return ""
}
return reference
}
// holdBackMissingImages leaves out of the declaration every module that names an image the store does not hold,
// and answers what it left out. The module's resources go, it joins LeftOut with the reason, and its build stays the
// one the machine was last sent (`was`).
func holdBackMissingImages(ctx context.Context, node string, s *sendable, address string, holds storeHolds,
was map[string]string) []missingImage {
lacks := map[string]bool{} // reference → the store does not hold it; absent: not asked, or no answer
asked := map[string]bool{}
lacking := map[string]string{} // module → the image
for _, r := range s.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
reference := imageInStore(image, address)
module := s.moduleOf[fmt.Sprint(r["id"])]
if reference == "" || module == "" || lacking[module] != "" {
continue
}
if !asked[reference] {
asked[reference] = true
held, err := holds(ctx, reference)
// empty-on-error: the store did not answer, so nothing is held back for a question unanswered
lacks[reference] = err == nil && !held
}
if lacks[reference] {
lacking[module] = image
}
}
if len(lacking) == 0 {
return nil
}
kept := s.Resources[:0:0]
for _, r := range s.Resources {
if _, out := lacking[s.moduleOf[fmt.Sprint(r["id"])]]; !out {
kept = append(kept, r)
}
}
s.Resources = kept
if s.leftOutWhy == nil {
s.leftOutWhy = map[string]string{}
}
var found []missingImage
for module, image := range lacking {
s.leftOutWhy[module] = "its image " + image + " is not in the artifact store"
if !slices.Contains(s.LeftOut, module) {
s.LeftOut = append(s.LeftOut, module)
}
if s.Builds != nil {
if prior, sent := was[module]; sent {
s.Builds[module] = prior
} else {
delete(s.Builds, module)
}
}
found = append(found, missingImage{Node: node, Module: module, Image: image})
}
sort.Strings(s.LeftOut)
sort.Slice(found, func(i, j int) bool { return found[i].Module < found[j].Module })
return found
}
// withoutMissingImages is composeSend's last step: the store asked about the images the declaration names, the modules
// whose image it lacks left out and kept in the ledger S25 reads. A store that cannot be found or asked leaves the
// declaration as composed.
func withoutMissingImages(ctx context.Context, open *stores, node string, s sendable) sendable {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return s
}
address, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil || address == "" {
return s
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
sent = nil
}
store := artifacts.Store{Address: address}
found := holdBackMissingImages(ctx, node, &s, address, store.HoldsManifest, sent)
for _, m := range found {
fmt.Printf("%s: %s is left out of the declaration: its image %s is not in the artifact store\n", node, m.Module, m.Image)
}
return s
}
// imageFacts are the images S25 raises: a module whose registered build names an image the store lacks, on each machine
// that runs it.
type imageFacts struct {
missingImage
}
// imagesHeld remembers the images the store was found to hold: a stored image is named by its digest and stays what it
// is, so it is asked about once; one it lacks is asked again at every look.
var imagesHeld = struct {
sync.Mutex
held map[string]bool
}{held: map[string]bool{}}
// gatherMissingImages is every machine's module whose registered build — the one a send would carry — names an image of
// the artifact store that the store does not hold. Read from the catalogue and the store, never from what a send did:
// a push is a command of its own, and the serving controller's watchdog hears nothing of it.
func gatherMissingImages(ctx context.Context, inv *inventory.Inventory, holds storeHolds) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
var out []imageFacts
for _, module := range sortedKeysOf(modulesOf(shelf)) {
for _, r := range shelf[module].Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
if _, ours := catalogue.InArtifactStore(image); !ours || !strings.Contains(image, "@sha256:") {
continue
}
imagesHeld.Lock()
known := imagesHeld.held[image]
imagesHeld.Unlock()
if !known {
held, err := holds(ctx, image)
if err != nil {
return nil, fmt.Errorf("the artifact store cannot be asked about %s: %w", image, err)
}
if held {
imagesHeld.Lock()
imagesHeld.held[image] = true
imagesHeld.Unlock()
continue
}
} else {
continue
}
running, err := inv.Running(ctx, module)
if err != nil {
return nil, err
}
for _, node := range running {
out = append(out, imageFacts{missingImage{Node: node, Module: module, Image: image}})
}
}
}
return out, nil
}
// modulesOf is the shelf's modules as a map of names, for sortedKeysOf.
func modulesOf(shelf map[string]catalogue.Manifest) map[string]string {
out := make(map[string]string, len(shelf))
for m := range shelf {
out[m] = m
}
return out
}
// watchMissingImages is S25: a module left out of a machine's declaration because the store lacks its image, said at
// once. The condition ends when the store holds the image, or a send names another.
func watchMissingImages(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.images {
out = append(out, conditions.Observation{Scope: conditions.ScopeModule, ID: m.Module + "." + m.Node, Kind: kindImageMissing,
Machine: m.Node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s is left out of every send to %s: its declaration names an image that the artifact store does "+
"not hold (the evidence names it); the running container is untouched", m.Module, m.Node),
Said: "image " + m.Image + " is not in the store",
Headline: clipWords(conditions.Capital(m.Module)+" is not updated on "+m.Node, conditions.HeadlineMax),
Explanation: clipWords(fmt.Sprintf("The new version of %s names a container image that the mesh's store does not "+
"hold, so it was not sent to %s. What runs there is untouched, and the rest of %s is sent as usual.",
m.Module, m.Node, m.Node), conditions.ExplanationMax),
Needs: fmt.Sprintf("build %s again; the details name the image.", m.Module),
Resolved: "Resolved: the store holds the image of " + m.Module + " again"})
}
return out
}
// gatherMissingImagesHere is gatherMissingImages asking the mesh's own artifact store; nothing to ask where the mesh has
// none on its network yet.
func gatherMissingImagesHere(ctx context.Context, inv *inventory.Inventory) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, nil
}
return gatherMissingImages(ctx, inv, artifacts.Store{Address: address}.HoldsManifest)
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
)
// **No declaration is sent that names an image the artifact store does not hold** (novox/hq issue 1742).
const storeAt = "novox.internal:5100"
// automx is the image the send of 2026-10-21 17:13 named and the store lacked; the others are held.
const (
automxImage = storeAt + "/mailu/automx@sha256:c62150b8e2cbcc18cb911b098c2877f928761a1e6251667f39ea18251ea35591"
imapImage = storeAt + "/mailu/imap@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
)
func aMailSend() sendable {
return sendable{
Resources: []map[string]any{
{"id": "mailu.automx", "type": "container", "image": automxImage},
{"id": "mailu.automx-config", "type": "file", "path": "/etc/automx.conf"},
{"id": "mailu.imap", "type": "container", "image": imapImage},
{"id": "other.web", "type": "container", "image": "docker.io/library/nginx@sha256:" + strings.Repeat("b", 64)},
{"id": "dns.dnsmasq", "type": "container", "image": storeAt + "/dnsmasq/main@sha256:" + strings.Repeat("c", 64)},
},
moduleOf: map[string]string{"mailu.automx": "automx", "mailu.automx-config": "automx", "mailu.imap": "mailu",
"other.web": "other", "dns.dnsmasq": "dnsmasq"},
Builds: map[string]string{"automx": "b2f5ebb2new", "mailu": "m2", "other": "o1", "dnsmasq": "d1"},
}
}
func storeHolding(missing ...string) (storeHolds, *[]string) {
var asked []string
return func(_ context.Context, reference string) (bool, error) {
asked = append(asked, reference)
for _, m := range missing {
if strings.Contains(reference, m) {
return false, nil
}
}
return true, nil
}, &asked
}
// 1742/AC1: given a send naming automx's image, which the store does not hold, when the declaration is judged, then
// every resource of automx is out of it, automx is among the modules left out with the reason, its build is the one the
// machine was last sent, and every other module — an image of another registry among them — is as composed.
func TestAModuleWhoseImageTheStoreLacksIsLeftOutOfTheSend(t *testing.T) {
s := aMailSend()
holds, asked := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, map[string]string{"automx": "b2f5ebb2old"})
if len(found) != 1 || found[0].Module != "automx" || found[0].Node != "novox" || found[0].Image != automxImage {
t.Fatalf("found %+v", found)
}
var ids []string
for _, r := range s.Resources {
ids = append(ids, r["id"].(string))
}
if got := strings.Join(ids, ","); got != "mailu.imap,other.web,dns.dnsmasq" {
t.Errorf("the declaration keeps %s", got)
}
if len(s.LeftOut) != 1 || s.LeftOut[0] != "automx" || !strings.Contains(s.leftOutWhy["automx"], "not in the artifact store") {
t.Errorf("left out %v because %v", s.LeftOut, s.leftOutWhy)
}
if s.Builds["automx"] != "b2f5ebb2old" || s.Builds["mailu"] != "m2" || s.Builds["other"] != "o1" {
t.Errorf("builds %v", s.Builds)
}
for _, a := range *asked {
if strings.Contains(a, "docker.io") {
t.Errorf("the store was asked about an image of another registry: %s", a)
}
}
}
// 1742/AC2: given a module never sent to the machine, when its image is missing, then it has no build in the record of the
// send; given the store does not answer, then nothing is held back for a question unanswered.
func TestAnUnansweredStoreHoldsNothingBackAndANewModuleHasNoBuild(t *testing.T) {
s := aMailSend()
holds, _ := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, nil)
if len(found) != 1 {
t.Fatalf("%+v", found)
}
if _, has := s.Builds["automx"]; has {
t.Errorf("a module never sent is recorded as sent: %v", s.Builds)
}
s = aMailSend()
silent := func(context.Context, string) (bool, error) { return false, errors.New("registry: connection refused") }
if found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, silent, nil); len(found) != 0 || len(s.LeftOut) != 0 ||
len(s.Resources) != 5 {
t.Errorf("an unanswered store held back %+v, left out %v", found, s.LeftOut)
}
}
// 1742/AC3: a module left out for its image raises a condition for the operator naming the module, the machine and the
// image in the evidence, plain words, and no answer to give; a mesh whose images are all held raises nothing.
func TestAMissingImageIsRaisedForTheOperator(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 40, 0, 0, time.UTC))
if got := watchMissingImages(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
f.images = []imageFacts{{missingImage{Node: "novox", Module: "automx", Image: automxImage}}}
got := watchMissingImages(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "image-missing" || o.Key() != "module.automx.novox.image-missing" || o.Resolver != conditions.ResolverOperator ||
o.Severity != conditions.Warning || o.Machine != "novox" {
t.Fatalf("%+v (%s)", o, o.Key())
}
if !strings.Contains(o.Said, "c62150b8") || !strings.Contains(o.Headline+o.Explanation, "automx") {
t.Errorf("words: %q / %q / %q", o.Headline, o.Explanation, o.Said)
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
}
+1 -1
View File
@@ -35,7 +35,7 @@ func TestAMergeSaysTheModulesWhoseBuildSourceItLeftUnchanged(t *testing.T) {
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "examples/route-proxy/"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"go.mod", "modules/route-proxy/cmd/route-proxy/"}},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
}
+20 -2
View File
@@ -65,6 +65,9 @@ type mergeCheckInput struct {
// group are the other heads of a delivery group composed with this one (novox/hq ADR 0239), each laid
// over the mesh after the one before: the group judged as one future state.
group []groupChange
// held are the approved config values held for the keys the pull requests introduce (novox/hq ADR 0502 clause 4),
// laid over the composition with the change only.
held []heldValue
// admin is a PostgreSQL the gate may create and drop databases in.
admin string
// beside is the directory holding the core clones beside the check (MESH_CHECK_BESIDE), for finding moves.
@@ -189,6 +192,8 @@ func mergeGateCommand(ctx context.Context, args []string) error {
"--changed, for a list too long for an argument (novox/hq issue 1247)")
group := set.String("group", "", "a delivery group's other heads, as JSON [{repository, tree, changed}], composed "+
"with this one as one future state (novox/hq ADR 0239)")
heldFile := set.String("held", "", "a file of the config values approved for keys the change's pull requests introduce, as JSON "+
"[{pull, node, module, key, value, approved_by}], laid over the composition with the change (novox/hq ADR 0502)")
asJSON := set.Bool("json", false, "print the verdict as JSON")
if _, err := parseAround(set, args); err != nil {
return err
@@ -208,6 +213,10 @@ func mergeGateCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
held, err := readHeld(*heldFile)
if err != nil {
return err
}
out := io.Writer(os.Stdout)
stdout := os.Stdout
if *asJSON {
@@ -223,7 +232,7 @@ func mergeGateCommand(ctx context.Context, args []string) error {
return err
}
v, err := judgeChange(ctx, mergeCheckInput{facts: f, repository: *repository, tree: *tree, changed: changedList,
group: others, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
group: others, held: held, admin: *admin, say: out, beside: os.Getenv("MESH_CHECK_BESIDE")})
os.Stdout = stdout
if err != nil {
return err
@@ -387,7 +396,16 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
}
}
sort.Strings(added)
after, tried, notes, err := composeEveryMachine(ctx, in, change, changeSources, added)
// **The held values are laid in this composition only** (novox/hq ADR 0502 clause 4): the mesh as it is does not
// declare their keys, so the base composition would refuse them.
withHeld := in
var heldNotes []string
withHeld.facts, heldNotes = layHeld(in.facts, in.held, change)
for _, n := range heldNotes {
say("%s", n)
}
v.Notes = append(v.Notes, heldNotes...)
after, tried, notes, err := composeEveryMachine(ctx, withHeld, change, changeSources, added)
if err != nil {
return v, fmt.Errorf("the mesh with the change could not be raised: %w", err)
}
+135
View File
@@ -0,0 +1,135 @@
package main
import (
"encoding/json"
"fmt"
"os"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
// Config values held for the keys an open pull request introduces (novox/hq ADR 0502 clause 4, issue 1826): the gate
// lays each approved one over its module's values on its node in the composition with the change, and in that one
// only. In the mesh as it is the key is not declared, so the value would be refused there.
// heldValue is one approved held value as the build agent writes it (--held): the value of Key for Module on Node,
// a machine's real name, which the snapshot holds only as a pseudonym.
type heldValue struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// readHeld reads the file --held names; none named, none held.
func readHeld(path string) ([]heldValue, error) {
if path == "" {
return nil, nil
}
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var held []heldValue
if err := json.Unmarshal(raw, &held); err != nil {
return nil, fmt.Errorf("the held values are not readable: %w", err)
}
for _, h := range held {
if h.Node == "" || h.Module == "" || h.Key == "" || len(h.Value) == 0 {
return nil, fmt.Errorf("a held value of %s names no node, module, key or value", h.Pull)
}
}
return held, nil
}
// layHeld is the snapshot with each held value laid as the value of its key in its node's layer of its module, and what
// it said of them. The snapshot given is not changed. **A held value for a machine the snapshot does not hold, or a
// module the change does not define, is said and not laid**, never dropped silently; one for a module not assigned to
// its machine is laid and said. The notes name machines as the snapshot does (roles and pseudonym), not by their names.
func layHeld(f snapshot.Facts, held []heldValue, shelf map[string]catalogue.Manifest) (snapshot.Facts, []string) {
if len(held) == 0 {
return f, nil
}
out := f
out.Machines = make([]snapshot.Machine, len(f.Machines))
copy(out.Machines, f.Machines)
var notes, laid []string
pulls := map[string]bool{}
for _, h := range held {
at := -1
pseudonym := snapshot.Pseudonym("machine", h.Node)
for i, m := range out.Machines {
if m.Name == pseudonym {
at = i
}
}
if at < 0 {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the snapshot holds no machine %s",
h.Module, h.Pull, h.Key, h.Value, pseudonym))
continue
}
if _, defined := shelf[h.Module]; !defined {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s=%s) is not composed: the change defines no module %s",
h.Module, h.Pull, h.Key, h.Value, h.Module))
continue
}
var value any
if err := json.Unmarshal(h.Value, &value); err != nil {
notes = append(notes, fmt.Sprintf("the value held for %s of %s (%s) is not composed: it is not a value: %v",
h.Module, h.Pull, h.Key, err))
continue
}
m := &out.Machines[at]
layers := make([]snapshot.Settings, len(m.Settings))
copy(layers, m.Settings)
found := false
for i := range layers {
if layers[i].Module == h.Module {
values := make(map[string]any, len(layers[i].Values)+1)
for k, v := range layers[i].Values {
values[k] = v
}
values[h.Key] = value
layers[i].Values = values
found = true
}
}
if !found {
layers = append(layers, snapshot.Settings{Module: h.Module, Values: map[string]any{h.Key: value}})
sort.Slice(layers, func(a, b int) bool { return layers[a].Module < layers[b].Module })
}
m.Settings = layers
assigned := false
for _, a := range m.Assigned {
assigned = assigned || a == h.Module
}
by := ""
if h.Approved != "" {
by = ", approved by " + h.Approved
}
line := fmt.Sprintf("%s on %s %s=%s%s", h.Module, m.Described(), h.Key, h.Value, by)
if !assigned {
line += " (not assigned there in the snapshot)"
}
laid = append(laid, line)
pulls[h.Pull] = true
}
if len(laid) > 0 {
var names []string
for p := range pulls {
names = append(names, p)
}
sort.Strings(names)
notes = append([]string{fmt.Sprintf("composed with %d value(s) held for %s: %s", len(laid), strings.Join(names, ", "),
strings.Join(laid, "; "))}, notes...)
}
return out, notes
}
@@ -0,0 +1,85 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
func heldGateFacts() snapshot.Facts {
return snapshot.Facts{Machines: []snapshot.Machine{
{Name: snapshot.Pseudonym("machine", "ace"), Roles: []string{"the build node"}, Assigned: []string{"n8n"},
Settings: []snapshot.Settings{{Module: "n8n", Values: map[string]any{"port": 5678.0}}}},
{Name: snapshot.Pseudonym("machine", "g14")},
}}
}
var heldShelf = map[string]catalogue.Manifest{"n8n": {Module: "n8n"}}
// 1826/AC3: a held value is laid in its node's layer of its module, beside what the layer holds, and said.
func TestAHeldValueIsLaidOverItsNodesLayerAndSaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}}, heldShelf)
layer := got.Machines[0].Settings[0].Values
if layer["media-gid"] != 2000.0 || layer["port"] != 5678.0 {
t.Fatalf("the layer holds %v", layer)
}
if _, has := f.Machines[0].Settings[0].Values["media-gid"]; has {
t.Fatal("the snapshot given was changed")
}
if len(notes) != 1 || !strings.Contains(notes[0], "composed with 1 value(s) held for novox/mesh-catalog#434") ||
!strings.Contains(notes[0], "media-gid=2000") || !strings.Contains(notes[0], "approved by the operator") ||
strings.Contains(notes[0], "ace") {
t.Fatalf("said %q", notes)
}
}
// 1826/AC3: a machine with no layer of the module gets one.
func TestAHeldValueMakesTheLayerAMachineLacks(t *testing.T) {
got, _ := layHeld(heldGateFacts(), []heldValue{{Pull: "p", Node: "g14", Module: "n8n", Key: "k", Value: []byte(`"x"`)}}, heldShelf)
s := got.Machines[1].Settings
if len(s) != 1 || s[0].Module != "n8n" || s[0].Values["k"] != "x" {
t.Fatalf("layers %+v", s)
}
}
// 1826/AC3: a held value for a machine or a module the snapshot or the change does not hold is said, never laid.
func TestAHeldValueForWhatIsNotHeldIsSaidNotLaid(t *testing.T) {
f := heldGateFacts()
got, notes := layHeld(f, []heldValue{
{Pull: "p", Node: "nowhere", Module: "n8n", Key: "k", Value: []byte("1")},
{Pull: "p", Node: "ace", Module: "ghost", Key: "k", Value: []byte("1")},
}, heldShelf)
if len(notes) != 2 || !strings.Contains(notes[0], "holds no machine") || !strings.Contains(notes[1], "defines no module ghost") {
t.Fatalf("said %q", notes)
}
if _, has := got.Machines[0].Settings[0].Values["k"]; has {
t.Fatal("laid")
}
}
// 1826/AC3: none held lays nothing and says nothing; the file is read and refuses a value without a node.
func TestNoHeldValuesLayNothingAndTheFileIsRead(t *testing.T) {
if got, notes := layHeld(heldGateFacts(), nil, heldShelf); len(notes) != 0 || len(got.Machines) != 2 {
t.Fatalf("%v %v", got, notes)
}
dir := t.TempDir()
good := filepath.Join(dir, "good.json")
os.WriteFile(good, []byte(`[{"pull":"p","node":"ace","module":"n8n","key":"k","value":1}]`), 0o644)
if h, err := readHeld(good); err != nil || len(h) != 1 || h[0].Key != "k" {
t.Fatalf("%v %v", h, err)
}
bad := filepath.Join(dir, "bad.json")
os.WriteFile(bad, []byte(`[{"pull":"p","module":"n8n","key":"k","value":1}]`), 0o644)
if _, err := readHeld(bad); err == nil {
t.Fatal("a value without a node was read")
}
if h, err := readHeld(""); h != nil || err != nil {
t.Fatalf("%v %v", h, err)
}
}
@@ -526,3 +526,57 @@ func TestAMemberJudgedAloneSeesAMoveThroughTheClonesBeside(t *testing.T) {
t.Errorf("nothing beside, yet moves %v", got)
}
}
// 1826/AC3 (integration level, a throwaway store): a change whose container says ${config:media-gid} fails the gate for
// want of the value, passes it with the value the operator approved held for its pull request, and the mesh as it is
// composes the same either way.
func TestAChangeNeedingAHeldValueFailsWithoutItAndPassesWithIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
trunk := `{"module":"n8n","version":"1","resources":[{"id":"server","type":"container","name":"n8n",
"image":"n8nio/n8n@sha256:4846eb2f4b874ab04cde7fc1e249d2ddaec66e9aea64439beb2972cfea88e3c0"}]}`
m, err := catalogue.ParseManifest([]byte(trunk))
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog",
Path: "modules/n8n", BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "n8n"); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
for i := range f.Machines {
f.Machines[i].Capabilities = append(f.Machines[i].Capabilities, snapshot.Capability{Name: "container-groups", Present: true})
}
tree := aTree(t, map[string]string{"n8n": n8nWithAGroup})
judge := func(held []heldValue) mergeVerdict {
v, err := judgeChange(t.Context(), mergeCheckInput{facts: f, admin: os.Getenv("MESH_TEST_POSTGRES"),
repository: "novox/mesh-catalog", tree: tree, changed: []string{"modules/n8n/module.json"}, held: held})
if err != nil {
t.Fatal(err)
}
return v
}
without := judge(nil)
if without.Verdict != "fail" || !strings.Contains(without.Report(), "media-gid") {
t.Fatalf("a required value nothing sets passed, or failed for another reason:\n%s", without.Report())
}
with := judge([]heldValue{{Pull: "novox/mesh-catalog#434", Node: "laptop", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}})
if with.Verdict == "fail" || with.Verdict == "error" {
t.Fatalf("the held value did not make it compose:\n%s", with.Report())
}
if !strings.Contains(with.Report(), "composed with 1 value(s) held for novox/mesh-catalog#434") {
t.Fatalf("the value laid is not said:\n%s", with.Report())
}
for i := range with.Machines {
if with.Machines[i].Base.Composes != without.Machines[i].Base.Composes {
t.Fatalf("the mesh as it is composes differently with a held value: %+v / %+v", with.Machines[i].Base, without.Machines[i].Base)
}
}
}
+20
View File
@@ -195,6 +195,14 @@ var plainWordings = map[string]func(conditions.Observation) words{
"there yet is sealed by the controller until the vault holds it.", m, m),
Resolved: "Resolved: the vault gave " + m + " the core's credentials"}
}),
kindFilesNotPut: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "The big files for " + m + " cannot be put on the bus",
Needs: "look at the bus's permissions for the controller; the details give the reason.",
Explanation: fmt.Sprintf("The mesh could not put the large files of %s's instructions in their place on the bus, "+
"send after send, so %s is sent them inside the instructions instead, which may make them too big.", m, m),
Resolved: "Resolved: the big files for " + m + " reach the bus again"}
}),
kindDeclarationNearLimit: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
if o.Severity == conditions.Urgent {
@@ -449,6 +457,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "A delivery did not finish. Nothing is tried again until you say so.",
Resolved: "Resolved: the delivery is no longer failed"}
}),
kindMovesHeld: worded(func(o conditions.Observation) words {
return words{Headline: "An update waits for you",
Explanation: "A new version of a module would restart a container or touch its data, so it waits for you.",
Resolved: "Resolved: the update is no longer waiting"}
}),
kindImageMissing: worded(func(o conditions.Observation) words {
return words{Headline: "A module is not updated",
Explanation: "The new version of a module names a container image that the mesh's store does not hold, so it " +
"was not sent. What runs is untouched.",
Needs: "build the module again; the details name the image.",
Resolved: "Resolved: the store holds the image again"}
}),
"ask-lost": worded(func(o conditions.Observation) words {
return words{Headline: "A build has no result",
Explanation: "A build was asked for and never answered. It is asked again with the next delivery.",
+1 -1
View File
@@ -260,7 +260,7 @@ func TestUpdatesHeldNameWhatWaitsAndWhereItIsReleased(t *testing.T) {
plainExample(t, got[0], "Updates of openrazer and sensors wait for your release",
"Needs you: release them from the mesh MCP server; this notification cannot do it. Updates of openrazer and "+
"sensors on g14 and shanks wait for a person to release them, because the last walk failed. "+
"They are not delivered until then, and stay held if you leave them.")
"They are not delivered until then, and stay held if you leave them.", "Release")
}
// **What held them is said in the glossary's words** (2026-10-08 review): the backlog is held after any
+47
View File
@@ -78,3 +78,50 @@ func TestAPlansLineCountsFromItsTierNotFromItsLastSave(t *testing.T) {
// novox/hq issue 296: a plan whose tier stands still is not written again by every advance — its revision,
// its last save and its `plan-moved` stay where its last change left them.
// novox/hq issue 920: a building plan whose current tier is all built waits on its note (a send refused at the
// gate), and its line, `status` and the stalled condition say that, not "building for ... LATE".
func TestAPlanLineSaysTheRefusalOnceItsTierIsBuilt(t *testing.T) {
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
built := now.Add(-time.Hour)
refusal := "tier 1: a build waits there for its gate: novox (mesh-delivery-indexer d68d7ef3→a5aa69a4) — tried again"
mk := func(second string) inventory.Plan {
return inventory.Plan{ID: "plan-r", Repository: "novox/mesh-sdk", Commit: "b2f5ebb2", State: inventory.PlanBuilding,
Tier: 1, Created: now.Add(-2 * time.Hour), Updated: now, TierEntered: now.Add(-time.Hour), Note: refusal,
Tiers: [][]string{{"a"}, {"b", "c"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &built},
"b": {State: "built", BuiltAt: &built}, "c": {State: second}}}
}
// 920/AC1
p := mk("built")
line := planLine(p, now)
if !strings.Contains(line, "a build waits there for its gate: novox") || strings.Contains(line, "building for") {
t.Errorf("920/AC1: a plan with its tier built and a refusal reads %q", line)
}
st := planStatuses([]inventory.Plan{p}, now, pauseView{}, nil)
if len(st) != 1 || !st[0].Late {
t.Errorf("920/AC1: status: %+v", st)
}
// 920/AC2
if line := planLine(mk("asked"), now); !strings.Contains(line, "building for 1h0m0s") {
t.Errorf("920/AC2: a plan with a tier module not built reads %q", line)
}
}
// 920/AC3: a tier move writes the previous tier's summary into the note; with the new tier all built, the line still
// says "building for", for that note is no refusal.
func TestAPlanLineDoesNotHoldOnATiersSummary(t *testing.T) {
now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC)
built := now.Add(-time.Hour)
for _, note := range []string{"failed or refused its send (apply-failing says it): novox", "tier 0 built; sent mesh-sdk to novox first, in one send"} {
p := inventory.Plan{ID: "plan-s", Repository: "novox/mesh-sdk", Commit: "b2f5ebb2", State: inventory.PlanBuilding,
Tier: 1, Created: now.Add(-2 * time.Hour), Updated: now, TierEntered: now.Add(-time.Hour), Note: note,
Tiers: [][]string{{"a"}, {"b"}},
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &built}, "b": {State: "built", BuiltAt: &built}}}
if line := planLine(p, now); !strings.Contains(line, "building for 1h0m0s") || strings.Contains(line, "held") {
t.Errorf("920/AC3: a plan with the note %q reads %q", note, line)
}
}
}
+1
View File
@@ -206,6 +206,7 @@ func servingPlanSaved(ctx context.Context, bus link.Bus, open *stores) func(*inv
return func(_ *inventory.Inventory, p inventory.Plan) {
sayPlanMoved(ctx, bus, open.inventory, p)
planKept(p)
keepBuildingKey(ctx, p)
if p.WalkSaid == inventory.WalkEnded {
go forgetChecksOfWalk(context.WithoutCancel(ctx), open.inventory, p)
}
@@ -31,7 +31,8 @@ func TestNoPlanIsWrittenButThroughSavePlansMoving(t *testing.T) {
return err
}
text := string(raw)
if write.MatchString(text) {
// The regular expression is the costly part of reading every source: only a file that names the table can match.
if strings.Contains(strings.ToLower(text), "release_plan") && write.MatchString(text) {
found = append(found, path+" writes the plan table")
}
// A plan record is written by the bucket's raw create and update, which only the plans' own file calls.
+2 -2
View File
@@ -52,7 +52,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"cmd/mesh-builder/", "internal/broker/"}, gomod...)}}
sourcesOf["route-proxy"] = []inventory.BuildSource{
{Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"examples/route-proxy/", "internal/broker/"}, gomod...)}}
{Repository: "novox/mesh-controller", Ref: "main", Paths: append([]string{"modules/route-proxy/cmd/route-proxy/", "internal/broker/"}, gomod...)}}
register(catalogue.Manifest{Module: "mesh-controller", Version: "1"}, "novox/mesh-controller", "", nil, nil)
register(agent, "novox/mesh-catalog", "modules/build-agent", nil, controllerRead)
register(catalogue.Manifest{Module: "route-proxy", Version: "1"}, "novox/mesh-catalog", "modules/route-proxy", nil, controllerRead)
@@ -133,7 +133,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{"a package only the controller builds from: the controller alone", "mesh-controller",
[]string{"internal/conditions/condition.go"}, "mesh-controller", true},
{"the route proxy's program: the route proxy alone", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, "route-proxy", true},
[]string{"modules/route-proxy/cmd/route-proxy/main.go"}, "route-proxy", true},
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, "mesh-controller,route-proxy | build-agent", false},
{"the route proxy's directory in the catalogue: it alone", "mesh-catalog",
+11 -11
View File
@@ -259,7 +259,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{Own: true, Paths: []string{"modules/build-agent/Dockerfile", "modules/build-agent/module.json"}},
},
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("examples/route-proxy/", "internal/broker/",
{Repository: "novox/mesh-controller", Ref: "main", Paths: with("modules/route-proxy/cmd/route-proxy/", "internal/broker/",
"vendor/github.com/nats-io/nats.go/")},
{Own: true, Paths: []string{"modules/route-proxy/Dockerfile", "modules/route-proxy/module.json"}},
},
@@ -288,7 +288,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{"a README of the controller's repository: no module (338)", "mesh-controller",
[]string{"README.md"}, read, "", "README.md"},
{"the route proxy's program alone: the route proxy alone (338)", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, read, "route-proxy", ""},
[]string{"modules/route-proxy/cmd/route-proxy/main.go"}, read, "route-proxy", ""},
{"the build seat's program alone: its holder alone", "mesh-controller",
[]string{"cmd/mesh-builder/main.go"}, read, "build-agent", ""},
{"a package the build seat's program and the controller build from: both", "mesh-controller",
@@ -305,7 +305,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
{"go.sum: all three", "mesh-controller",
[]string{"go.sum"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a file added to the proxy's package: the proxy", "mesh-controller",
[]string{"examples/route-proxy/new.go"}, read, "route-proxy", ""},
[]string{"modules/route-proxy/cmd/route-proxy/new.go"}, read, "route-proxy", ""},
// Before any build source is said: as before.
{"no build source said: a README moves all three, as before", "mesh-controller",
[]string{"README.md"}, unsaid, "mesh-controller,route-proxy | build-agent", ""},
@@ -345,7 +345,7 @@ func TestAModuleAPlanOvertookIsReadWhole(t *testing.T) {
built := time.Date(2026, 10, 10, 1, 0, 0, 0, time.UTC)
read := map[string][]inventory.ReadRepository{
"route-proxy": {
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/", "go.mod"}, Built: built},
{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "go.mod"}, Built: built},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}, Built: built},
},
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: built}},
@@ -396,11 +396,11 @@ func TestAMissedMergeMovingOnlyAPackagingModuleIsActedOnOnce(t *testing.T) {
}
read := map[string][]inventory.ReadRepository{
"mesh-controller": {{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/"}, Built: merged.Add(-time.Hour)}},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"examples/route-proxy/"},
"route-proxy": {{Repository: "novox/mesh-controller", Ref: "main", Paths: []string{"modules/route-proxy/cmd/route-proxy/"},
Built: merged.Add(-time.Hour), Looked: merged.Add(-time.Hour)}},
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "c1",
MergedAt: merged.Format(time.RFC3339), Paths: []string{"examples/route-proxy/main.go"}}
MergedAt: merged.Format(time.RFC3339), Paths: []string{"modules/route-proxy/cmd/route-proxy/main.go"}}
if got := wouldMove(m, entries, planningView(read, nil)); len(got) != 1 || got[0].Manifest.Module != "route-proxy" {
t.Fatalf("a missed merge of the proxy's program would move %v", got)
}
@@ -599,14 +599,14 @@ func TestTheGatePlansFromTheBuildSourcesTheSnapshotCarries(t *testing.T) {
Sources: []snapshot.BuildSource{{Own: true, Paths: []string{"module.json", "cmd/mesh-controller/", "internal/broker/"}}}},
{Name: "route-proxy", Repository: "novox/mesh-catalog", Path: "modules/route-proxy", Manifest: manifest("route-proxy"),
Reads: []string{"novox/mesh-controller"},
Sources: []snapshot.BuildSource{{Repository: "novox/mesh-controller", Paths: []string{"examples/route-proxy/", "internal/broker/"}},
Sources: []snapshot.BuildSource{{Repository: "novox/mesh-controller", Paths: []string{"modules/route-proxy/cmd/route-proxy/", "internal/broker/"}},
{Own: true, Paths: []string{"modules/route-proxy/module.json"}}}},
}}
for paths, want := range map[string]string{
"cmd/mesh-controller/main.go": "mesh-controller",
"examples/route-proxy/main.go": "route-proxy",
"internal/broker/broker.go": "mesh-controller,route-proxy",
"README.md": "",
"cmd/mesh-controller/main.go": "mesh-controller",
"modules/route-proxy/cmd/route-proxy/main.go": "route-proxy",
"internal/broker/broker.go": "mesh-controller,route-proxy",
"README.md": "",
} {
r, err := reachOfChange(facts, "novox/mesh-controller", []string{paths}, "")
if err != nil {
+238
View File
@@ -0,0 +1,238 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The plans building (novox/hq ADR 0483 clause 2, issue 1687).
//
// **A key per delivery plan in its build phase, so that the build seat takes no new check while a delivery builds.** The phase
// runs from the ask of the plan's first build until every build of its last tier is built, or the plan fails, stops or waits
// on the operator. It is derived from the plan alone (buildPhaseOf), so a controller that restarts finds the same answer. The
// key is put when the first build is asked (sayBuilding), kept current at every save of the plan (keepBuildingKey) and by a
// look every link.PlansBuildingRefresh while this controller holds the lease (keepingPlansBuilding), and deleted at the phase's
// end; the bucket's age removes the key of a controller that stopped, so checks are held at most that long.
//
// The layout is documented in internal/link/holds.go.
// buildPhaseOf says whether a plan is in its build phase: open, not waiting for the operator's word, with a build asked and a
// build of its tiers still to be built. A tier built and sent leaves the plan rolling while later tiers are still to be asked;
// it is in the phase still, which is what keeps a check from taking a place between two tiers.
func buildPhaseOf(p inventory.Plan) bool {
if !p.Open() || p.Waiting() || p.Release != nil {
return false
}
asked, unbuilt := false, false
for _, tier := range p.Tiers {
for _, m := range tier {
st := p.Modules[m]
if st != nil && st.State != "" {
asked = true
}
if st == nil || (st.State != "built" && st.State != planDeleted) {
unbuilt = true
}
}
}
return asked && unbuilt
}
// buildingSince is when the plan's first build was asked, or fallback when no module says.
func buildingSince(p inventory.Plan, fallback time.Time) time.Time {
since := time.Time{}
for _, st := range p.Modules {
if st != nil && st.AskedAt != nil && (since.IsZero() || st.AskedAt.Before(since)) {
since = *st.AskedAt
}
}
if since.IsZero() {
return fallback
}
return since
}
// buildingStore is the bucket of the plans building.
type buildingStore interface {
Put(ctx context.Context, plan string, value []byte) error
Delete(ctx context.Context, plan string) error
Keys(ctx context.Context) ([]string, error)
}
// syncBuilding makes the bucket say what the plans say: a key for each plan in its build phase, put again, and none for any
// other plan, deleted. It returns the faults, each once.
func syncBuilding(ctx context.Context, store buildingStore, plans []inventory.Plan, now time.Time) []error {
var faults []error
live := map[string]bool{}
for _, p := range plans {
if !buildPhaseOf(p) {
continue
}
live[p.ID] = true
if err := putBuilding(ctx, store, p.ID, buildingSince(p, now), now); err != nil {
faults = append(faults, err)
}
}
keys, err := store.Keys(ctx)
if err != nil {
return append(faults, err)
}
for _, key := range keys {
if live[key] {
continue
}
if err := store.Delete(ctx, key); err != nil {
faults = append(faults, fmt.Errorf("the key of %s could not be taken away: %w", key, err))
}
}
return faults
}
func putBuilding(ctx context.Context, store buildingStore, plan string, since, now time.Time) error {
body, _ := json.Marshal(link.PlanBuilding{Plan: plan, Since: since.UTC(), Seen: now.UTC()})
if err := store.Put(ctx, plan, body); err != nil {
return fmt.Errorf("the key of %s could not be put: %w", plan, err)
}
return nil
}
// busBuilding is the store on the controller's bucket.
type busBuilding struct {
conn *nats.Conn
mu sync.Mutex
kv jetstream.KeyValue
}
func (b *busBuilding) bucket(ctx context.Context) (jetstream.KeyValue, error) {
b.mu.Lock()
defer b.mu.Unlock()
if b.kv != nil {
return b.kv, nil
}
api, err := jetstream.New(b.conn)
if err != nil {
return nil, err
}
kv, err := api.KeyValue(ctx, broker.PlansBuildingBucket)
if err != nil {
return nil, fmt.Errorf("the plans building: %w", err)
}
b.kv = kv
return kv, nil
}
func (b *busBuilding) Put(ctx context.Context, plan string, value []byte) error {
kv, err := b.bucket(ctx)
if err != nil {
return err
}
_, err = kv.Put(ctx, plan, value)
return err
}
func (b *busBuilding) Delete(ctx context.Context, plan string) error {
kv, err := b.bucket(ctx)
if err != nil {
return err
}
if err := kv.Delete(ctx, plan); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return err
}
return nil
}
func (b *busBuilding) Keys(ctx context.Context) ([]string, error) {
kv, err := b.bucket(ctx)
if err != nil {
return nil, err
}
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
// buildingKeys is the store of the serving controller; nil in a process that is not it, which says nothing.
var buildingKeys atomic.Pointer[buildingStore]
func setBuildingStore(s buildingStore) {
if s == nil {
buildingKeys.Store(nil)
return
}
buildingKeys.Store(&s)
}
// sayBuilding puts a plan's key: its first build is being asked (or another of its builds). A key that could not be put is
// said; the look that follows puts it again.
func sayBuilding(ctx context.Context, plan string, since time.Time) {
s := buildingKeys.Load()
if s == nil {
return
}
if err := putBuilding(ctx, *s, plan, since, time.Now()); err != nil {
fmt.Printf("plans: %v; checks may take a place of %s until the next look\n", err, plan)
}
}
// keepBuildingKey is told each plan this controller keeps: its key put again while it builds and taken away once it does not.
func keepBuildingKey(ctx context.Context, p inventory.Plan) {
s := buildingKeys.Load()
if s == nil {
return
}
if buildPhaseOf(p) {
sayBuilding(ctx, p.ID, buildingSince(p, time.Now()))
return
}
if err := (*s).Delete(ctx, p.ID); err != nil {
fmt.Printf("plans: the key of %s could not be taken away; it ages out: %v\n", p.ID, err)
}
}
// keepingPlansBuilding looks at the plans every link.PlansBuildingRefresh for as long as this controller serves and holds the
// lease, so a key is put again before its age and the key of a plan that left the phase unseen is taken away.
func keepingPlansBuilding(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) {
store := &busBuilding{conn: conn}
setBuildingStore(store)
defer setBuildingStore(nil)
tick := time.NewTicker(broker.PlansBuildingRefresh)
defer tick.Stop()
failing := ""
for {
if epoch, err := theLease.epoch(ctx); err == nil && epoch != 0 {
why := ""
if plans, err := inv.OpenPlans(ctx); err != nil {
why = fmt.Sprintf("the plans cannot be read: %v", err)
} else if faults := syncBuilding(ctx, store, plans, time.Now()); len(faults) > 0 {
why = fmt.Sprint(faults)
}
if why != failing {
if why != "" {
fmt.Printf("plans: the keys of the plans building cannot be kept: %s\n", why)
} else {
fmt.Printf("plans: the keys of the plans building are kept again\n")
}
failing = why
}
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
+175
View File
@@ -0,0 +1,175 @@
package main
import (
"context"
"encoding/json"
"sort"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Issue 1687 (ADR 0483 clause 2): a delivery plan's key lives from the ask of its first build until every build of its last
// tier is built, or the plan fails, stops or waits on the operator.
type fakeBuildingStore struct {
mu sync.Mutex
keys map[string]link.PlanBuilding
puts map[string]int
}
func newFakeBuildingStore() *fakeBuildingStore {
return &fakeBuildingStore{keys: map[string]link.PlanBuilding{}, puts: map[string]int{}}
}
func (f *fakeBuildingStore) Put(_ context.Context, plan string, value []byte) error {
f.mu.Lock()
defer f.mu.Unlock()
var v link.PlanBuilding
if err := json.Unmarshal(value, &v); err != nil {
return err
}
f.keys[plan] = v
f.puts[plan]++
return nil
}
func (f *fakeBuildingStore) Delete(_ context.Context, plan string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.keys, plan)
return nil
}
func (f *fakeBuildingStore) Keys(context.Context) ([]string, error) {
f.mu.Lock()
defer f.mu.Unlock()
var out []string
for k := range f.keys {
out = append(out, k)
}
sort.Strings(out)
return out, nil
}
func aPlan(id, state string, tiers [][]string, states map[string]string) inventory.Plan {
p := inventory.Plan{ID: id, State: state, Tiers: tiers, Modules: map[string]*inventory.PlanModule{}}
now := time.Now().Add(-time.Minute)
for m, st := range states {
pm := &inventory.PlanModule{State: st}
if st != "" {
pm.AskedAt = &now
}
p.Modules[m] = pm
}
return p
}
func TestAPlanIsInItsBuildPhaseFromTheFirstAskToTheLastBuild(t *testing.T) {
tiers := [][]string{{"a", "b"}, {"c"}}
cases := []struct {
name string
plan inventory.Plan
want bool
}{
{"no build asked yet", aPlan("p", inventory.PlanBuilding, tiers, nil), false},
{"the first tier asked", aPlan("p", inventory.PlanBuilding, tiers, map[string]string{"a": "asked", "b": "asked"}), true},
// Between two tiers: the first built and sent (rolling), the second not asked yet.
{"between two tiers", aPlan("p", inventory.PlanRolling, tiers, map[string]string{"a": "built", "b": "built"}), true},
{"the last tier asked", aPlan("p", inventory.PlanBuilding, tiers, map[string]string{"a": "built", "b": "built", "c": "asked"}), true},
{"every build built", aPlan("p", inventory.PlanRolling, tiers, map[string]string{"a": "built", "b": "built", "c": "built"}), false},
// A module the plan deletes is done as far as its tiers go (release_plan.go), so it holds no key through the rollout.
{"a deleted module is done", aPlan("p", inventory.PlanRolling, tiers, map[string]string{"a": "built", "b": "built", "c": planDeleted}), false},
{"failed", aPlan("p", inventory.PlanFailed, tiers, map[string]string{"a": "failed", "b": "asked"}), false},
{"superseded", aPlan("p", inventory.PlanSuperseded, tiers, map[string]string{"a": "asked"}), false},
{"a batch whose window is open", aPlan("p", inventory.PlanAssembling, tiers, map[string]string{"a": "asked"}), false},
}
for _, c := range cases {
if got := buildPhaseOf(c.plan); got != c.want {
t.Errorf("%s: in the build phase = %v, want %v", c.name, got, c.want)
}
}
waiting := aPlan("p", inventory.PlanBuilding, tiers, map[string]string{"a": "asked"})
waiting.Delivery = &inventory.PlanDelivery{Awaits: "the operator's word"}
if buildPhaseOf(waiting) {
t.Error("a plan that waits on the operator is in its build phase")
}
}
// The key's life: put, refreshed, deleted at the phase's end; a plan that left the phase unseen loses its key at the next look.
func TestAPlanKeyIsPutRefreshedAndDeletedAtThePhasesEnd(t *testing.T) {
ctx := context.Background()
store := newFakeBuildingStore()
tiers := [][]string{{"a"}, {"b"}}
building := aPlan("plan-1", inventory.PlanBuilding, tiers, map[string]string{"a": "asked"})
if faults := syncBuilding(ctx, store, []inventory.Plan{building}, time.Now()); len(faults) > 0 {
t.Fatal(faults)
}
if _, ok := store.keys["plan-1"]; !ok {
t.Fatal("no key while the first build is asked")
}
first := store.keys["plan-1"]
// Refreshed: put again, the phase's start kept.
time.Sleep(2 * time.Millisecond)
syncBuilding(ctx, store, []inventory.Plan{building}, time.Now())
if store.puts["plan-1"] != 2 || !store.keys["plan-1"].Seen.After(first.Seen) || !store.keys["plan-1"].Since.Equal(first.Since) {
t.Errorf("the key was not refreshed with its start kept: %d puts, %+v then %+v", store.puts["plan-1"], first, store.keys["plan-1"])
}
// Between the tiers: still there.
between := aPlan("plan-1", inventory.PlanRolling, tiers, map[string]string{"a": "built"})
syncBuilding(ctx, store, []inventory.Plan{between}, time.Now())
if _, ok := store.keys["plan-1"]; !ok {
t.Error("the key was gone between two tiers")
}
// A key of another plan that is no longer in the phase.
store.keys["plan-0"] = link.PlanBuilding{Plan: "plan-0"}
done := aPlan("plan-1", inventory.PlanRolling, tiers, map[string]string{"a": "built", "b": "built"})
syncBuilding(ctx, store, []inventory.Plan{done}, time.Now())
if len(store.keys) != 0 {
t.Errorf("keys remain after the last build was built: %v", store.keys)
}
}
// keepBuildingKey, told of each save, puts the key while the plan builds and deletes it when the plan fails.
func TestASavedPlanPutsOrDeletesItsKey(t *testing.T) {
ctx := context.Background()
store := newFakeBuildingStore()
var s buildingStore = store
buildingKeys.Store(&s)
defer buildingKeys.Store(nil)
p := aPlan("plan-7", inventory.PlanBuilding, [][]string{{"a"}}, map[string]string{"a": "asked"})
keepBuildingKey(ctx, p)
if _, ok := store.keys["plan-7"]; !ok {
t.Fatal("the key was not put when the plan was saved building")
}
p.State = inventory.PlanFailed
keepBuildingKey(ctx, p)
if _, ok := store.keys["plan-7"]; ok {
t.Fatal("the key stayed after the plan failed")
}
}
// The plan's key is there before its first build is asked, so a check cannot take a place between the ask and the plan's save.
func TestThePlansKeyIsPutBeforeItsFirstBuildIsAsked(t *testing.T) {
store := newFakeBuildingStore()
var s buildingStore = store
buildingKeys.Store(&s)
defer buildingKeys.Store(nil)
p, byName := aTierOfThree()
keyThere := false
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
_, keyThere = store.keys["plan-1"]
return "build-x", nil
}
t.Cleanup(func() { askABuild = was })
askModule(t.Context(), p, "a", byName)
if !keyThere {
t.Fatal("the plan's key was not there when its first build was asked")
}
}
+1 -1
View File
@@ -153,7 +153,7 @@ func planWaitCommand(ctx context.Context, inv *inventory.Inventory, id, untilAsk
line := planLineSaid(p, now, pause, bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, one))
}, waitsOf(ctx, inv, one)) + heldSaid(p, heldNow(ctx, inv))
doc := planWaitedDoc(p, planStatuses(one, now, pause, bounds)[0], until, came, now.Sub(began), line)
return json.NewEncoder(os.Stdout).Encode(doc)
}
+130
View File
@@ -0,0 +1,130 @@
package main
import (
"context"
"fmt"
"io"
"os"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// processUser is the bus user of one process of a module on a node (novox/hq ADR 0436 §1).
func processUser(node, module, process string) string {
return broker.Principal{Kind: broker.KindProcess, Node: node, Module: module, Process: process}.Username()
}
// processAccountWarning is the warning line for the own secret broker-<process id> of a process that declares bus
// when nobody issued its account (novox/hq ADR 0436 §2, issue 1226): "" for any other secret and for an issued
// account. The send goes on with the stand-in until the composition leaves the module off (issue 1753), so the
// line is the only loud thing about it.
func processAccountWarning(m catalogue.Manifest, node, secret string, minted []string) string {
id, isProcess := strings.CutPrefix(secret, "broker-")
if !isProcess {
return ""
}
if _, declared := m.ProcessBuses()[id]; !declared {
return ""
}
user := processUser(node, m.Module, id)
for _, u := range minted {
if u == user {
return ""
}
}
return fmt.Sprintf("%s/%s process %s: its bus account is not issued — mesh-controller.issue %s %s",
node, m.Module, id, node, m.Module)
}
// sayWarning writes a line where the compose's other warnings go: the act's answer when it carries a writer,
// else the controller's journal.
func sayWarning(ctx context.Context, line string) {
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
fmt.Fprintln(w, line)
}
// processAccountsMissing is each process of m that declares bus and whose account is not minted yet on node.
func processAccountsMissing(m catalogue.Manifest, node string, minted []string) []broker.ProcessPrincipal {
have := map[string]bool{}
for _, u := range minted {
have[u] = true
}
var out []broker.ProcessPrincipal
for _, pr := range inventory.ProcessesOf(m) {
if !have[processUser(node, m.Module, pr.Process)] {
out = append(out, pr)
}
}
return out
}
// issueMissingProcessAccounts issues the accounts of m's bus processes on node that are not minted, and no
// other: an account already issued is rotated on purpose, never by a later delivery or assignment (issue 1226).
// It says what it issued, one line, and nothing when nothing was missing.
func issueMissingProcessAccounts(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string) (string, error) {
if len(m.ProcessBuses()) == 0 || mayIssue(m) != nil {
return "", nil
}
minted, err := inv.BusMinted(ctx)
if err != nil {
return "", err
}
missing := processAccountsMissing(m, node, minted)
if len(missing) == 0 {
return "", nil
}
known, err := broker.FromEnvironment()
if err != nil {
return "", fmt.Errorf("cannot deliver a credential without knowing where the bus is: %w", err)
}
reachable, err := brokerReachableAt(ctx, inv, known, node)
if err != nil {
return "", err
}
busAddress, err := broker.BusAddress()
if err != nil {
busAddress = ""
}
var ids []string
for _, pr := range missing {
if err := issueProcess(ctx, inv, m, pr, node, busAddress, known, reachable); err != nil {
return "", err
}
ids = append(ids, pr.Process)
}
return fmt.Sprintf("its process account(s) %s issued and sealed to %s, and arrive with the push",
strings.Join(ids, ", "), node), nil
}
// issueGainedProcessAccounts runs after a module's new definition is registered: on each machine assigned it, the
// accounts of its bus processes not yet minted are issued, so a process a delivery adds is not pushed with a
// random stand-in (issue 1226). Never fatal: the build worked and the module is registered, so a failure is said,
// and the push refuses the module until `mesh-controller.issue` is run.
func issueGainedProcessAccounts(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest) {
if len(m.ProcessBuses()) == 0 {
return
}
nodes, err := inv.Running(ctx, m.Module)
if err != nil {
fmt.Printf("%s: its machines are not read to issue its process accounts: %v\n", m.Module, err)
return
}
for _, node := range nodes {
line, err := issueMissingProcessAccounts(ctx, inv, m, node)
if err != nil {
fmt.Printf("%s on %s: its process accounts are not issued (%v): `mesh-controller.issue %s %s` first\n",
m.Module, node, err, node, m.Module)
continue
}
if line != "" {
fmt.Printf("%s on %s: %s\n", m.Module, node, line)
}
}
}
@@ -0,0 +1,135 @@
//go:build integration
package main
import (
"bytes"
"encoding/json"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func gainedAndPlain(t *testing.T) (plain, gained catalogue.Manifest) {
t.Helper()
plain, err := catalogue.ParseManifest([]byte(`{"module":"mesh-mail","version":"1",
"own-secrets":{"broker-send":{"path":"/var/lib/mesh-mail/broker-send","owner":"mesh-mail-send"}},
"resources":[{"id":"account","type":"user","name":"mesh-mail-send"}]}`))
if err != nil {
t.Fatal(err)
}
return plain, mailSenderWithProcess()
}
// 1226/AC2: given an assigned module whose new definition adds a bus process, when the delivery records it
// (takeIn, the function that records a build result), the process's account is minted; and 1226/AC3: the same
// build heard again replaces nothing.
func TestAGainedProcessIsIssuedWhenTheDeliveryRecordsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
dir := t.TempDir()
if err := busCertificate([]string{"--into", dir}); err != nil {
t.Fatal(err)
}
t.Setenv(broker.AddressVar, strings.TrimPrefix(testbus.URL(t), "nats://"))
t.Setenv(broker.CertificateVar, filepath.Join(dir, "tls.crt"))
plain, gained := gainedAndPlain(t)
register(t, open, plain)
if _, err := assign(ctx, open, "laptop", "mesh-mail"); err != nil {
t.Fatal(err)
}
if _, minted, _ := inv.BusUserHash(ctx, "laptop.mesh-mail.send"); minted {
t.Fatal("the account was minted before the module gained the process")
}
manifest, _ := json.Marshal(gained)
build := func(id, commit string) {
t.Helper()
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, link.BuildResult{
ID: id, Repository: "http://forge.internal:20000/novox/mesh-mail.git", Path: "modules/mesh-mail",
Ref: "main", On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-mail"},
})); err != nil {
t.Fatal(err)
}
}
build("b-1", "aaaaaaa1111")
hash, held, err := inv.BusUserHash(ctx, "laptop.mesh-mail.send")
if err != nil || !held {
t.Fatalf("1226/AC2: recording the gained definition did not mint the process's account: held %v, %v", held, err)
}
// And the next compose carries the account, with no warning (1226/AC2).
var said bytes.Buffer
if _, err := prepareSend(leftOutSaidBy(ctx, &said), open, "laptop"); err != nil {
t.Fatalf("1226/AC2: the next compose failed: %v", err)
}
if strings.Contains(said.String(), "bus account is not issued") {
t.Fatalf("1226/AC2: the next compose warns of an account just minted: %q", said.String())
}
// The row the compose reads is the account issueProcess accepted (origin accepted), never a value the mesh
// made for it (a stand-in).
in, err := inputFor(ctx, open)
if err != nil {
t.Fatal(err)
}
var row *compose.OwnSecretRow
for i, r := range in.OwnSecrets {
if r.Node == "laptop" && r.Module == "mesh-mail" && r.Name == "broker-send" {
row = &in.OwnSecrets[i]
}
}
if row == nil {
t.Fatal("1226/AC2: the next compose carries no row for the process's account")
}
if row.Origin != inventory.OriginAccepted {
t.Fatalf("1226/AC2: the process's account is a stand-in the mesh made (origin %q), not the issued account", row.Origin)
}
build("b-2", "bbbbbbb2222")
again, _, err := inv.BusUserHash(ctx, "laptop.mesh-mail.send")
if err != nil || again != hash {
t.Fatalf("1226/AC3: recording again replaced the issued account (%v)", err)
}
}
// 1226/AC1 and AC4 against the store: given two modules on a machine, one with an unissued process account, when
// the machine is composed, the rest of it is composed and sent, and only the warning in the push's answer names
// the first, with `mesh-controller.issue <node> <module>`.
func TestAnUnissuedProcessAccountIsWarnedOfAndTheRestIsComposed(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
_, gained := gainedAndPlain(t)
register(t, open, gained)
other := catalogue.Manifest{Module: "other", Version: "1",
OwnSecrets: catalogue.OwnSecrets{"token": {Path: "/var/lib/other/token"}}}
register(t, open, other)
for _, m := range []string{"mesh-mail", "other"} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
var said bytes.Buffer
if _, err := prepareSend(leftOutSaidBy(ctx, &said), open, "laptop"); err != nil {
t.Fatalf("1226/AC4: an unissued process account held the machine: %v", err)
}
want := "laptop/mesh-mail process send: its bus account is not issued — mesh-controller.issue laptop mesh-mail"
if !strings.Contains(said.String(), want) {
t.Fatalf("1226/AC1: the push says %q, want %q", said.String(), want)
}
if strings.Contains(said.String(), "other") {
t.Fatalf("1226/AC4: the other module is warned of: %q", said.String())
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "other", "token"); err != nil || !held {
t.Fatalf("1226/AC4: the other module's secret was not composed: held %v, %v", held, err)
}
}
@@ -0,0 +1,36 @@
package main
import "testing"
// 1226/AC1: given a module whose process declares bus and has no issued account, when a machine is composed, the
// push's warning names issue, and the machine's other modules are composed (AC4: nothing is refused).
func TestAProcessAccountNobodyIssuedIsWarnedOfNamingIssue(t *testing.T) {
m := mailSenderWithProcess()
got := processAccountWarning(m, "laptop", "broker-send", nil)
want := "laptop/mesh-mail process send: its bus account is not issued — mesh-controller.issue laptop mesh-mail"
if got != want {
t.Fatalf("1226/AC1: the warning is %q, want %q", got, want)
}
// Other names are not this rule's.
if processAccountWarning(m, "laptop", "broker", nil) != "" || processAccountWarning(m, "laptop", "broker-other", nil) != "" ||
processAccountWarning(m, "laptop", "token", nil) != "" {
t.Fatal("1226/AC1: a name that is no declared process's account was warned of")
}
if got := processAccountWarning(m, "laptop", "broker-send", []string{"laptop.mesh-mail.send"}); got != "" {
t.Fatalf("1226/AC3: an issued account was warned of: %q", got)
}
}
// 1226/AC3: an account already issued is not issued again by a later delivery or assignment.
func TestAnIssuedProcessAccountIsNotIssuedAgain(t *testing.T) {
m := mailSenderWithProcess()
if got := processAccountsMissing(m, "laptop", nil); len(got) != 1 || got[0].Process != "send" {
t.Fatalf("1226/AC3: an account not issued is not missing: %v", got)
}
if got := processAccountsMissing(m, "laptop", []string{"laptop.mesh-mail.send"}); len(got) != 0 {
t.Fatalf("1226/AC3: an issued account would be issued again: %v", got)
}
if got := processAccountsMissing(m, "ace", []string{"laptop.mesh-mail.send"}); len(got) != 1 {
t.Fatalf("1226/AC3: another machine's account stands in for this one's: %v", got)
}
}
+211
View File
@@ -0,0 +1,211 @@
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"path"
"reflect"
"slices"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// What the controller tells mesh-config of an open pull request, and asks it back (novox/hq ADR 0502 clauses 2 and 4,
// issue 1826): a config key a pull request introduces is not declared on the trunk, so a value for it can be judged
// only against the pull request's own declaration, which the core reads from the forge and mesh-config never does
// (ADR 0495 §1). The call is mesh-config's `config` verb in the controller's own name, actions `pull-declares` and
// `held`; the shapes are those of the interface the two sides agreed.
// meshConfigWait is how long the controller waits for mesh-config before it goes on without: a check is never held up
// by a module that does not answer.
const meshConfigWait = 10 * time.Second
// meshConfigAsk is one call of mesh-config's `config` verb in the controller's own name, answering its result. A
// variable so a test needs no bus.
var meshConfigAsk = func(ctx context.Context, args map[string]any) ([]byte, error) {
var result []byte
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, meshConfigWait)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("%s", answer.Error)
}
result = answer.Result
return nil
})
return result, err
}
// pullAt is an open pull request at a head: the change to read, and its number.
type pullAt struct {
changeSource
Number int
}
// pullAtOf is a pull request announced, as the change it makes over what it merges into.
func pullAtOf(p link.PullUpdated) pullAt {
old := p.BaseSHA
if old == "" {
old = p.Base
}
return pullAt{Number: p.Number, changeSource: changeSource{Owner: p.Owner, Repo: p.Repo, New: p.Commit, Old: old,
Paths: p.Paths, Truncated: p.PathsTruncated, Removed: p.Removed}}
}
func pullKey(owner, repo string, number int) string {
return fmt.Sprintf("%s/%s#%d", owner, repo, number)
}
func (p pullAt) key() string { return pullKey(p.Owner, p.Repo, p.Number) }
// pullDeclarations are the config declarations a pull request's head adds or changes over what it merges into, by
// module, in the shape of the declared-config mirror. The bool says the head could be read: false for a pull request whose
// paths are cut (the caller cuts them to the manifests of the modules it moves) or whose base is unknown, which is then
// told nothing, never an empty set.
func pullDeclarations(ctx context.Context, p pullAt) ([]inventory.DeclaredConfigMirrored, bool, error) {
var out []inventory.DeclaredConfigMirrored
if p.Truncated || p.Old == "" {
return nil, false, nil
}
for _, file := range p.Paths {
if path.Base(file) != moduleManifestFile || slices.Contains(p.Removed, file) {
continue
}
now, exists, err := manifestCached(ctx, p.Owner, p.Repo, file, p.New)
if err != nil {
return nil, false, err
}
if !exists || now.Module == "" {
continue
}
was, _, err := manifestCached(ctx, p.Owner, p.Repo, file, p.Old)
if err != nil {
return nil, false, err
}
have, err := inventory.DeclaredOf(now.Settings)
if err != nil {
return nil, false, fmt.Errorf("%s at %.8s: %w", file, p.New, err)
}
had, err := inventory.DeclaredOf(was.Settings)
if err != nil {
return nil, false, fmt.Errorf("%s at %.8s: %w", file, p.Old, err)
}
changed := map[string]inventory.DeclaredMirrored{}
for key, d := range have {
if before, there := had[key]; !there || !reflect.DeepEqual(before, d) {
changed[key] = d
}
}
if len(changed) > 0 {
out = append(out, inventory.DeclaredConfigMirrored{Module: now.Module, Settings: changed})
}
}
return out, true, nil
}
// tellPullDeclares tells mesh-config, for each pull request at every announced head, what the head declares, an empty set
// when it declares nothing new: mesh-config re-judges the values held at every head, so a head that reverts the change
// drops them (ADR 0502 clause 6). A
// failure is said to w and the check goes on: the held values are then none (ADR 0502 clause 2).
func tellPullDeclares(ctx context.Context, w io.Writer, pulls []pullAt) {
for _, p := range pulls {
mods, readable, err := pullDeclarations(ctx, p)
if err != nil {
fmt.Fprintf(w, "%s: what its head declares cannot be read, so mesh-config is told nothing: %v\n", p.key(), err)
continue
}
if !readable {
continue
}
declares := map[string]map[string]inventory.DeclaredMirrored{}
for _, m := range mods {
declares[m.Module] = m.Settings
}
if _, err := meshConfigAsk(ctx, map[string]any{"action": "pull-declares", "pull": p.key(), "head": p.New,
"declares": declares}); err != nil {
fmt.Fprintf(w, "%s: mesh-config cannot be told what its head declares (%v); the check goes on without held values\n",
p.key(), err)
}
}
}
// heldFor asks mesh-config for the approved held values of these pull requests at their heads (ADR 0502 clause 4). Only
// those of the pull requests asked about are carried. A failure is said to w and answers none.
func heldFor(ctx context.Context, w io.Writer, pulls []pullAt) []link.HeldConfig {
if len(pulls) == 0 {
return nil
}
type pullHead struct {
Pull string `json:"pull"`
Head string `json:"head"`
}
asked := make([]pullHead, 0, len(pulls))
for _, p := range pulls {
asked = append(asked, pullHead{p.key(), p.New})
}
raw, err := meshConfigAsk(ctx, map[string]any{"action": "held", "pulls": asked})
if err != nil {
fmt.Fprintf(w, "mesh-config cannot say what it holds for these pull requests (%v); the check goes on without held values\n", err)
return nil // empty-on-error: said above; a check without held values fails for want of the value and says so, it never passes on a guess
}
var answer struct {
Held []link.HeldConfig `json:"held"`
Skipped []struct {
Pull string `json:"pull"`
Why string `json:"why"`
} `json:"skipped"`
}
if len(raw) == 0 {
return nil
}
if err := json.Unmarshal(raw, &answer); err != nil {
fmt.Fprintf(w, "mesh-config's held values cannot be read (%v); the check goes on without them\n", err)
return nil // empty-on-error: said above; a check without held values fails for want of the value and says so, it never passes on a guess
}
for _, s := range answer.Skipped {
fmt.Fprintf(w, "%s: mesh-config holds a value it does not carry: %s\n", s.Pull, s.Why)
}
var out []link.HeldConfig
for _, h := range answer.Held {
for _, p := range pulls {
if h.Pull == p.key() {
out = append(out, h)
break
}
}
}
return out
}
// withHeldConfig tells mesh-config what these pull requests declare and puts the approved held values of them on the
// check request (ADR 0502 clause 4). With carry false, a member of a group whose composed check composes, only the
// declarations are told.
func withHeldConfig(ctx context.Context, w io.Writer, request *link.BuildRequest, pulls []pullAt, carry bool) {
tellPullDeclares(ctx, w, pulls)
if carry {
carryHeld(ctx, w, request, pulls)
}
}
// carryHeld puts the approved held values of these pull requests on the check request.
func carryHeld(ctx context.Context, w io.Writer, request *link.BuildRequest, pulls []pullAt) {
if request.Check != nil {
request.Check.HeldConfig = heldFor(ctx, w, pulls)
}
}
// holdsModules is whether a module of the mesh is built from this repository.
func holdsModules(entries []inventory.Entry, owner, repo string) bool {
return slices.ContainsFunc(entries, func(e inventory.Entry) bool {
return sameRepository(e.Source.Repository, link.SourceMoved{Owner: owner, Repo: repo})
})
}
+209
View File
@@ -0,0 +1,209 @@
package main
import (
"context"
"encoding/json"
"io"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
const (
headSHA = "2222222222222222222222222222222222222222"
baseSHA = "1111111111111111111111111111111111111111"
)
// fakeManifests serves manifests by ref, and counts what the forge was asked. Issue 1826/AC1 (unit level).
func fakeManifests(t *testing.T, at map[string]*catalogue.Manifest) {
t.Helper()
was := manifestAt
t.Cleanup(func() { manifestAt = was })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(_ context.Context, _, _, _, ref string) (catalogue.Manifest, bool, error) {
if m := at[ref]; m != nil {
return *m, true, nil
}
return catalogue.Manifest{}, false, nil
}
}
// fakeConfig records the calls to mesh-config and answers each action from answers.
func fakeConfig(t *testing.T, answers map[string]string) *[]map[string]any {
t.Helper()
var calls []map[string]any
was := meshConfigAsk
t.Cleanup(func() { meshConfigAsk = was })
meshConfigAsk = func(_ context.Context, args map[string]any) ([]byte, error) {
calls = append(calls, args)
return []byte(answers[args["action"].(string)]), nil
}
return &calls
}
func manifest(module string, keys map[string]catalogue.SettingDeclaration) *catalogue.Manifest {
return &catalogue.Manifest{Module: module, Settings: keys}
}
func thePull() pullAt {
return pullAt{Number: 434, changeSource: changeSource{Owner: "novox", Repo: "mesh-catalog", New: headSHA, Old: baseSHA,
Paths: []string{"modules/n8n/module.json", "modules/n8n/README.md"}}}
}
// 1826/AC1: a key the head adds is sent with its declaration, under the pull request and the head.
func TestAKeyThePullRequestAddsIsToldToMeshConfig(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{
baseSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}}),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"},
"media-gid": {Type: "number", Required: true}}),
})
calls := fakeConfig(t, map[string]string{"pull-declares": `{"kept":1}`})
tellPullDeclares(context.Background(), io.Discard, []pullAt{thePull()})
if len(*calls) != 1 {
t.Fatalf("mesh-config was asked %d times: %v", len(*calls), *calls)
}
c := (*calls)[0]
if c["action"] != "pull-declares" || c["pull"] != "novox/mesh-catalog#434" || c["head"] != headSHA {
t.Fatalf("the call is %v", c)
}
body, _ := json.Marshal(c["declares"])
var declares map[string]map[string]inventory.DeclaredMirrored
if err := json.Unmarshal(body, &declares); err != nil || len(declares) != 1 || declares["n8n"] == nil {
t.Fatalf("declares %s: %v", body, err)
}
mods := []inventory.DeclaredConfigMirrored{{Module: "n8n", Settings: declares["n8n"]}}
if _, has := mods[0].Settings["port"]; has || len(mods[0].Settings) != 1 {
t.Fatalf("only the added key is told, got %v", mods[0].Settings)
}
d := mods[0].Settings["media-gid"]
if d.Type != "number" || !d.Required {
t.Fatalf("the declaration is %+v", d)
}
}
// 1826/AC1: a key whose declaration the head changes is sent; one it leaves as it was is not.
func TestAnUnchangedKeyIsNotToldAndAChangedOneIs(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{
baseSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}, "x": {Type: "string"}}),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}, "x": {Type: "string", Pattern: "a+"}}),
})
mods, touched, err := pullDeclarations(context.Background(), thePull())
if err != nil || !touched || len(mods) != 1 || len(mods[0].Settings) != 1 || mods[0].Settings["x"].Pattern != "a+" {
t.Fatalf("%+v %v %v", mods, touched, err)
}
}
// 1826/AC1: a head that changes no module.json, or removes it, is still told, with declares {}: mesh-config re-judges at every
// head, so a head that reverts the change drops the value held for it (review round 1, finding 1).
func TestAHeadThatDeclaresNothingIsToldAsEmpty(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil), headSHA: manifest("n8n", nil)})
calls := fakeConfig(t, nil)
p := thePull()
p.Paths = []string{"modules/n8n/README.md"}
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
p = thePull()
p.Removed = p.Paths
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
if len(*calls) != 2 {
t.Fatalf("mesh-config was told %v", *calls)
}
for _, c := range *calls {
if d, _ := json.Marshal(c["declares"]); string(d) != "{}" || c["head"] != headSHA {
t.Fatalf("told %v", c)
}
}
}
// 1826/AC1: a head whose paths are cut, or whose base is unknown, cannot be read: nothing is told, never an empty set.
func TestAHeadThatCannotBeReadIsNotToldAsEmpty(t *testing.T) {
calls := fakeConfig(t, nil)
p := thePull()
p.Truncated = true
tellPullDeclares(context.Background(), io.Discard, []pullAt{p})
if len(*calls) != 0 {
t.Fatalf("told %v", *calls)
}
}
// 1826/AC1: a module.json changed without a declaration changing is told as an empty set, so mesh-config re-judges what it holds.
func TestAChangedManifestWithNoNewDeclarationIsToldAsEmpty(t *testing.T) {
m := manifest("n8n", map[string]catalogue.SettingDeclaration{"port": {Type: "number"}})
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: m, headSHA: m})
calls := fakeConfig(t, nil)
tellPullDeclares(context.Background(), io.Discard, []pullAt{thePull()})
if len(*calls) != 1 {
t.Fatalf("calls %v", *calls)
}
if d, _ := json.Marshal((*calls)[0]["declares"]); string(d) != "{}" {
t.Fatalf("declares %s", d)
}
}
// 1826/AC1: mesh-config not answering is said and goes on; the check is not stopped.
func TestMeshConfigNotAnsweringIsSaidAndGoneOn(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil),
headSHA: manifest("n8n", map[string]catalogue.SettingDeclaration{"k": {Type: "string"}})})
was := meshConfigAsk
t.Cleanup(func() { meshConfigAsk = was })
meshConfigAsk = func(context.Context, map[string]any) ([]byte, error) { return nil, io.ErrUnexpectedEOF }
var said strings.Builder
tellPullDeclares(context.Background(), &said, []pullAt{thePull()})
got := heldFor(context.Background(), &said, []pullAt{thePull()})
if len(got) != 0 || !strings.Contains(said.String(), "mesh-config") {
t.Fatalf("held %v, said %q", got, said.String())
}
}
// 1826/AC2: the held values mesh-config answers for the check's pull requests are what the check request carries.
func TestTheHeldValuesOfTheChecksPullRequestsAreAsked(t *testing.T) {
calls := fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#434","head":"` + headSHA +
`","node":"ace","module":"n8n","key":"media-gid","value":2000,"approved":"the operator"}]}`})
other := pullAt{Number: 7, changeSource: changeSource{Owner: "novox", Repo: "mesh-controller", New: baseSHA}}
got := heldFor(context.Background(), io.Discard, []pullAt{thePull(), other})
if len(got) != 1 || got[0].Node != "ace" || got[0].Key != "media-gid" || string(got[0].Value) != "2000" {
t.Fatalf("got %+v", got)
}
c := (*calls)[0]
pulls, _ := json.Marshal(c["pulls"])
if c["action"] != "held" || string(pulls) != `[{"pull":"novox/mesh-catalog#434","head":"`+headSHA+
`"},{"pull":"novox/mesh-controller#7","head":"`+baseSHA+`"}]` {
t.Fatalf("asked %v %s", c, pulls)
}
}
// 1826/AC2: a value held for a pull request the check is not of is never carried.
func TestAHeldValueOfAnotherPullRequestIsNotCarried(t *testing.T) {
fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#999","head":"x","node":"ace","module":"n8n","key":"k","value":1}]}`})
if got := heldFor(context.Background(), io.Discard, []pullAt{thePull()}); len(got) != 0 {
t.Fatalf("carried %+v", got)
}
}
// 1826/AC2: the check request carries what mesh-config answered; a member of a composed group asks none.
func TestTheCheckRequestCarriesTheHeldValues(t *testing.T) {
fakeManifests(t, map[string]*catalogue.Manifest{baseSHA: manifest("n8n", nil), headSHA: manifest("n8n", nil)})
fakeConfig(t, map[string]string{"held": `{"held":[{"pull":"novox/mesh-catalog#434","node":"ace","module":"n8n","key":"media-gid","value":2000}]}`})
request := link.BuildRequest{Check: &link.CheckRequest{}}
withHeldConfig(context.Background(), io.Discard, &request, []pullAt{thePull()}, true)
if len(request.Check.HeldConfig) != 1 || request.Check.HeldConfig[0].Key != "media-gid" {
t.Fatalf("the request carries %+v", request.Check.HeldConfig)
}
request = link.BuildRequest{Check: &link.CheckRequest{}}
withHeldConfig(context.Background(), io.Discard, &request, []pullAt{thePull()}, false)
if len(request.Check.HeldConfig) != 0 {
t.Fatalf("a composed member carries %+v", request.Check.HeldConfig)
}
}
// 1826/AC1: a repository that builds a module of the mesh holds modules; another does not.
func TestARepositoryThatBuildsAModuleHoldsModules(t *testing.T) {
entries := []inventory.Entry{{Manifest: catalogue.Manifest{Module: "a"}, Source: inventory.Source{Repository: "novox/mesh-catalog", Path: "modules/a"}}}
if !holdsModules(entries, "novox", "mesh-catalog") || holdsModules(entries, "novox", "hq") {
t.Fatal("wrong")
}
}
+57 -1
View File
@@ -282,6 +282,10 @@ func serve(ctx context.Context) (err error) {
// assignment.
inv.SetProvisioningPlans(inv.SentProvisioningPlans)
go mirroringRecords(ctx, inv, bus.Conn, bus)
// And the bucket of the files that travel by digest, judged against what each machine's latest declaration names (ADR 0486 §3).
go sweepingDeclarationFiles(ctx, inv, bus.Conn)
// And the key of each plan in its build phase, which holds the build seat's checks while a delivery builds (ADR 0483 clause 2).
go keepingPlansBuilding(ctx, inv, bus.Conn)
// Every call carries the lease's epoch, and its record is written only under the lease (novox/hq
// to-be 45 §6).
link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) })
@@ -979,6 +983,18 @@ func deliver(ctx context.Context, d delivery, holder string, sending []readyNode
return err
}
digest, err := d.declare(ctx, s, body)
if errors.Is(err, errFilesNotPut) && s.declared.fileReferences {
// **The grant that lets the controller write the bucket travels in the very declaration of the machine holding the
// bus** (novox/hq ADR 0486 §2): until the bus has it, a put is refused, and the declaration that would deliver the
// grant would wait for it. So that send, and only that, goes with every content inline, as to a node that reads no
// references, and says so; the next one tries the bucket again.
fmt.Printf("%s: the files that travel by digest could not be put in the bucket (%v): sending them inline\n", s.node, err)
s.declared.fileReferences = false
if body, err = s.declared.Body(); err != nil {
return err
}
digest, err = d.declare(ctx, s, body)
}
if err != nil {
return err
}
@@ -1040,6 +1056,35 @@ type overTheBus struct {
indent string
}
// putFiles puts the contents s names by digest in the bucket.
func (b overTheBus) putFiles(ctx context.Context, s readyNode) error {
objects, err := s.declared.objects()
if err != nil {
return fmt.Errorf("%s: the files that travel by digest cannot be read: %w", s.node, err)
}
if len(objects) == 0 {
return nil
}
bus, ok := b.server.Bus().(link.OverNATS)
if !ok {
return fmt.Errorf("%s: the declaration names files by digest and this bus has no object bucket", s.node)
}
putting, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
store, err := openBucket(putting, bus.Conn, s.node)
if err == nil {
err = putObjects(putting, store, objects)
}
notePut(ctx, s.node, err)
if err != nil {
return fmt.Errorf("%s was not sent its declaration: %w: %w", s.node, errFilesNotPut, err)
}
return nil
}
// errFilesNotPut marks a send that stopped because the files it names by digest could not be put in the bucket.
var errFilesNotPut = errors.New("the files it names by digest are not in the bucket")
func (b overTheBus) grant(ctx context.Context, sending []readyNode) error {
// The bus's objects first, which every declaration implies (novox/hq issue 208): said and raised
// when they cannot be, and never what holds the send back (assertOnSend).
@@ -1052,9 +1097,20 @@ func (b overTheBus) grant(ctx context.Context, sending []readyNode) error {
}
func (b overTheBus) declare(ctx context.Context, s readyNode, body []byte) (string, error) {
if err := link.Declare(ctx, b.server.Bus(), b.signer, s.node, body, 15*time.Second); err != nil {
// The contents the declaration names by digest are in the bucket before it is sent (novox/hq ADR 0486 §1, issue 1696): a node
// that applies it finds every one. A node reading no references has none.
// **Under the node's lock** from the put to the publication, so the sweep of its bucket cannot remove an object between them.
unlock := lockFilesOf(s.node)
err := b.putFiles(ctx, s)
if err != nil {
unlock()
return "", err
}
declared := link.Declare(ctx, b.server.Bus(), b.signer, s.node, body, 15*time.Second)
unlock()
if declared != nil {
return "", declared
}
// After it is away, not before. A digest recorded for something that failed to send would make
// the machine look current for a declaration it never received.
digest, err := recordSent(ctx, b.open.inventory, s.node, body, s.declared.Builds, s.declared.Epoch,
+40 -6
View File
@@ -91,7 +91,9 @@ type queueAnswer struct {
Dead int `json:"dead"`
MaxDeliver int `json:"max-deliver"`
Asks []queuedAnswer `json:"asks"`
Text string `json:"text"`
// HeldFor are the delivery plans building, while which the waiting checks are held (ADR 0483 clause 5); empty when none.
HeldFor []string `json:"held-for,omitempty"`
Text string `json:"text"`
}
// queuedAnswer is one ask as data. Never what the request carries beyond its source — not `held`, every
@@ -120,6 +122,9 @@ type queuedAnswer struct {
// Delivery is the how-manyth time an ask in flight was handed out; more than 1 is a restart
// (novox/hq issue 594).
Delivery int `json:"delivery,omitempty"`
// LetThrough is a check that waited past 30 minutes and is let through while a delivery builds: the next one, or the one
// that runs (ADR 0483 clause 5).
LetThrough bool `json:"let-through,omitempty"`
}
// aCommitRef is a ref that names a commit by its full hash (SHA-1 or SHA-256).
@@ -130,7 +135,8 @@ var aCommitRef = regexp.MustCompile(`^(?:[0-9a-f]{40}|[0-9a-f]{64})$`)
func queueAnswerOf(q link.Queue, now time.Time) queueAnswer {
waiting, running, dead := q.Of(link.AskWaiting), q.Of(link.AskInFlight), q.Of(link.AskDead)
out := queueAnswer{Seat: q.Seat, Waiting: len(waiting), InFlight: len(running), Dead: len(dead),
MaxDeliver: q.MaxDeliver, Asks: []queuedAnswer{}, Text: queueText(q, now)}
MaxDeliver: q.MaxDeliver, Asks: []queuedAnswer{}, HeldFor: q.Hold.Plans, Text: queueText(q, now)}
letNext, letRunning := q.LetThroughOf(now)
one := func(a link.QueuedAsk, position int) queuedAnswer {
kind := link.KindOf(a.Request)
since := a.AskedAt
@@ -143,7 +149,7 @@ func queueAnswerOf(q link.Queue, now time.Time) queueAnswer {
}
return queuedAnswer{ID: a.ID, State: a.State, Kind: kind, Repository: link.WithoutPassword(a.Repository), Path: a.Path,
Ref: a.Ref, Commit: commit, Machine: a.On, WasOn: a.Was, AskedAt: a.AskedAt, Since: since,
Position: position, Seq: a.Seq, Delivery: a.Delivery}
Position: position, Seq: a.Seq, Delivery: a.Delivery, LetThrough: a.ID != "" && (a.ID == letNext || a.ID == letRunning)}
}
for _, a := range running {
out.Asks = append(out.Asks, one(a, 0))
@@ -179,6 +185,11 @@ func queueText(q link.Queue, now time.Time) string {
}
return s
}
letNext, letRunning := q.LetThroughOf(now)
const passed = "waited past 30 minutes, let through"
if q.HoldFault != "" {
fmt.Fprintf(&b, "why checks wait cannot be read, so none is said held: %s\n", q.HoldFault)
}
if len(running) > 0 {
fmt.Fprintln(&b, "\nin flight:")
for _, a := range running {
@@ -193,14 +204,27 @@ func queueText(q link.Queue, now time.Time) string {
case a.Was != "":
where = fmt.Sprintf("handed back by %s, to be handed out again", a.Was)
}
fmt.Fprintf(&b, " %-26s %-8s %s — %s, %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), where,
ago(a.AskedAt), a.Seq)
note := ""
if a.ID != "" && a.ID == letRunning {
note = " — " + passed
}
fmt.Fprintf(&b, " %-26s %-8s %s — %s, %s (seq %d)%s\n", a.ID, link.KindOf(a.Request), what(a), where,
ago(a.AskedAt), a.Seq, note)
}
}
if len(waiting) > 0 {
fmt.Fprintln(&b, "\nwaiting, in the order they are handed out:")
// **Why the checks wait** (ADR 0483 clause 5): above them, the plans they are held for.
if len(q.Hold.Plans) > 0 && hasCheck(waiting) {
fmt.Fprintf(&b, " checks are held while delivery %s builds; the oldest check that waited 30 minutes goes first, one at a time\n",
strings.Join(q.Hold.Plans, ", "))
}
for _, a := range waiting {
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq)
note := ""
if a.ID != "" && a.ID == letNext {
note = " — " + passed + " next"
}
fmt.Fprintf(&b, " %-26s %-8s %s — %s (seq %d)%s\n", a.ID, link.KindOf(a.Request), what(a), ago(a.AskedAt), a.Seq, note)
}
}
if len(dead) > 0 {
@@ -218,6 +242,16 @@ func queueText(q link.Queue, now time.Time) string {
return b.String()
}
// hasCheck says a check is among the asks.
func hasCheck(asks []link.QueuedAsk) bool {
for _, a := range asks {
if a.Request.Check != nil {
return true
}
}
return false
}
// cancelCommand drops one waiting or dead ask.
func cancelCommand(ctx context.Context, args []string) error {
if len(args) != 1 {
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// Issue 1688 / 1652 AC5 (ADR 0483 clause 5): the queue names, above the waiting checks, the plan they are held for, and says of a
// check let through that it waited past 30 minutes.
func aHeldQueue(now time.Time) link.Queue {
check := func(id string, ago time.Duration, n int) link.QueuedAsk {
return link.QueuedAsk{Seq: uint64(n), ID: id, Repository: "ssh://forge/novox/r", AskedAt: now.Add(-ago), State: link.AskWaiting,
Request: link.BuildRequest{ID: id, Check: &link.CheckRequest{Owner: "novox", Repo: "r", Number: n}}}
}
return link.Queue{Seat: link.TheBuildMachine, Asks: []link.QueuedAsk{
check("build-young", 5*time.Minute, 1),
check("build-old", 45*time.Minute, 2),
check("build-older", 50*time.Minute, 3),
}, Hold: link.CheckHold{Plans: []string{"plan-42"}}}
}
func TestTheQueueNamesThePlanTheChecksAreHeldFor(t *testing.T) {
now := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
text := queueText(aHeldQueue(now), now)
heldAt := strings.Index(text, "held while delivery plan-42 builds")
firstAt := strings.Index(text, "build-young")
if heldAt < 0 || firstAt < 0 || heldAt > firstAt {
t.Fatalf("the plan the checks are held for is not named above the waiting checks:\n%s", text)
}
// Nothing building: nothing held.
q := aHeldQueue(now)
q.Hold = link.CheckHold{}
if strings.Contains(queueText(q, now), "held while") {
t.Fatal("checks are said held with no delivery building")
}
}
func TestACheckLetThroughIsSaidToHaveWaitedPastThirtyMinutes(t *testing.T) {
now := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
q := aHeldQueue(now)
text := queueText(q, now)
var saidOf []string
for _, l := range strings.Split(text, "\n") {
if strings.Contains(l, "waited past 30 minutes, let through") {
saidOf = append(saidOf, strings.Fields(l)[0])
}
}
if len(saidOf) != 1 || saidOf[0] != "build-older" {
t.Fatalf("the check let through next (the oldest) is not the one said: %v\n%s", saidOf, text)
}
answer := queueAnswerOf(q, now)
if len(answer.HeldFor) != 1 || answer.HeldFor[0] != "plan-42" {
t.Errorf("held-for = %v", answer.HeldFor)
}
for _, a := range answer.Asks {
if a.LetThrough != (a.ID == "build-older") {
t.Errorf("%s let-through = %v", a.ID, a.LetThrough)
}
}
// A check in flight under the let-through key is said too.
q.Asks = append(q.Asks, link.QueuedAsk{Seq: 9, ID: "build-running", State: link.AskInFlight, On: "ace", AskedAt: now.Add(-time.Hour),
Started: now.Add(-time.Minute), Request: link.BuildRequest{ID: "build-running", Check: &link.CheckRequest{Owner: "novox", Repo: "r", Number: 9}}})
q.Hold.LetThrough = &link.LetThrough{Builder: "ace"}
if !strings.Contains(queueText(q, now), "build-running") || !strings.Contains(queueText(q, now), "waited past 30 minutes, let through\n") {
t.Errorf("the check in flight let through is not said:\n%s", queueText(q, now))
}
// Younger than the bound: none is said.
young := aHeldQueue(now)
young.Asks = young.Asks[:1]
if strings.Contains(queueText(young, now), "waited past") {
t.Error("a check of 5 minutes is said to have waited past 30")
}
}
+28 -6
View File
@@ -181,15 +181,24 @@ func planModuleNames(p inventory.Plan) map[string]string {
return out
}
// recordedWaiting is, on one machine, every move of a recorded module that waits for a person, each said with why
// (ADR 0377 §6): what `status` names the machine behind for, with `push <node>` as the remedy. A move under a
// module's own `record` that recreates nothing and moves no data is not among them: it goes with its walk.
func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]string, error) {
// heldMove is one move of a recorded module that waits for a person on one machine (ADR 0377 §6).
type heldMove struct {
Module string
From, To string
// Why is the walk's own word for it: "waits: " and the recreate words, the data change, or the hold.
Why string
// Chosen is whether a person's own choice holds it — a module they set to record, or the bus's step — which the
// operator is not asked about again: they chose that hold, and the bus has its own condition (S17).
Chosen bool
}
// heldMovesOn is, on one machine, every move of a recorded module that waits for a person, in module order.
func heldMovesOn(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]heldMove, error) {
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return nil, err
}
var out []string
var out []heldMove
for _, m := range sortedKeysOf(sent) {
was := sent[m]
now, held := f.current[m]
@@ -197,12 +206,25 @@ func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts,
continue
}
if v := f.recordedMove(m, was); !v.rolls {
out = append(out, fmt.Sprintf("%s %s → %s %s", m, short(was), short(now.Commit), v.said))
out = append(out, heldMove{Module: m, From: was, To: now.Commit, Why: v.said,
Chosen: now.From == catalogue.FromPerson || now.From == catalogue.FromBus})
}
}
return out, nil
}
// recordedWaiting is, on one machine, every move of a recorded module that waits for a person, each said with why
// (ADR 0377 §6): what `status` names the machine behind for, with `push <node>` as the remedy. A move under a
// module's own `record` that recreates nothing and moves no data is not among them: it goes with its walk.
func recordedWaiting(ctx context.Context, inv *inventory.Inventory, f moveFacts, node string) ([]string, error) {
moves, err := heldMovesOn(ctx, inv, f, node)
var out []string
for _, h := range moves {
out = append(out, fmt.Sprintf("%s %s → %s %s", h.Module, short(h.From), short(h.To), h.Why))
}
return out, err
}
// stringsKeys is a map of lists as a map of its keys, for sortedKeysOf.
func stringsKeys(m map[string][]string) map[string]string {
out := map[string]string{}
+4
View File
@@ -1371,6 +1371,10 @@ func backlogObservation() []conditions.Observation {
"them, `upgrade release-backlog --why …` releases them", n, strings.Join(machines, ", "), backlogNow.held)}
w := releaseHeldWords(modules, machines)
o.Headline, o.Explanation, o.Resolved, o.Needs = w.Headline, w.Explanation, w.Resolved, w.Needs
// Asked of the operator, who releases them with an Approve (novox/hq issue 1732): the controller's own
// `upgrade release-backlog`, which opens the release plan that walks them through the machines, each judged.
o.Actions = []conditions.Action{{Label: "Release", Verb: "mesh-controller.upgrade", Level: conditions.LevelApprove,
Arguments: map[string]string{"release-backlog": "true", "why": "", "cause": conditions.CauseOperatorAnswer}}}
return []conditions.Observation{o}
}
+51 -3
View File
@@ -577,6 +577,9 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
}
// **The walk takes the check's build of the same key** (novox/hq ADR 0350 §2): its ask says so, with the pull
// requests it answers, whose checks name the part of a differing key.
// **The plan's key is there before its first build is** (ADR 0483 clause 2): a check taken between the ask and the plan's save
// would hold a place for its whole run.
sayBuilding(ctx, p.ID, now)
id, err := askABuild(withReuse(ctx, reuseWithVerdicts(ctx, p.ID, reuseAskOf(p), mergedHeadsOf(p))), source,
e.Source.Path, ref)
if err != nil {
@@ -851,6 +854,23 @@ func startedBeside(ctx context.Context, inv *inventory.Inventory, p inventory.Pl
return behind, why, nil
}
// refusalNotePrefix and refusalNoteSuffix frame the note of a step the walk was refused: "tier N: <refusal> — tried
// again". Only a note so framed is a hold (novox/hq issue 920); a tier's summary left in the note is not.
const (
refusalNotePrefix = "tier "
refusalNoteSuffix = " — tried again"
)
// refusalNote is the note of a plan whose step at tier was refused with err.
func refusalNote(tier int, err error) string {
return refusalNotePrefix + fmt.Sprint(tier) + ": " + err.Error() + refusalNoteSuffix
}
// refused says a plan's note is a refusal the walk wrote.
func refused(p inventory.Plan) bool {
return strings.HasPrefix(p.Note, refusalNotePrefix) && strings.HasSuffix(p.Note, refusalNoteSuffix)
}
// deferredNote begins the note of a walk deferred behind another.
const deferredNote = "let go; starts once "
@@ -863,6 +883,20 @@ func deferred(p inventory.Plan) bool {
p.Delivery.Awaits != "" && p.Delivery.Go != nil && strings.HasPrefix(p.Note, deferredNote)
}
// currentTierBuilt says every module of a plan's current tier is built (or deleted), so no build is what it waits on.
func currentTierBuilt(p inventory.Plan) bool {
if p.Tier < 0 || p.Tier >= len(p.Tiers) || len(p.Tiers[p.Tier]) == 0 {
return false
}
for _, m := range p.Tiers[p.Tier] {
s := p.Modules[m]
if s == nil || (s.State != "built" && s.State != planDeleted) {
return false
}
}
return true
}
// askedAny says a plan asked any module.
func askedAny(p inventory.Plan) bool {
for _, s := range p.Modules {
@@ -927,7 +961,7 @@ func advanceHeld(ctx context.Context, open *stores) (failed bool) {
// Kept in the plan, so `plans` says why it has not moved rather than the log alone;
// the state is left as it was and the step is tried again by the next look. Said and
// kept when it is new: the same refusal on every tick is one fact, not one per tick.
p.Note = "tier " + fmt.Sprint(p.Tier) + ": " + err.Error() + " — tried again"
p.Note = refusalNote(p.Tier, err)
// A refusal for the bus's planned step is a person's to end: S17 says it from its first moment.
if refusedForTheBus(err) {
busRefusedFirst.mark(p.ID, time.Now())
@@ -2025,6 +2059,19 @@ func planLineSaid(p inventory.Plan, now time.Time, pause pauseView, bound time.D
if p.State == inventory.PlanRolling {
what = p.Note
}
if p.State == inventory.PlanBuilding && refused(p) && currentTierBuilt(p) {
// Every build of the tier is in, so the plan waits on something else, and its note says what: a send
// refused ("a build waits there for its gate: <machine> (...)") is kept there. "building" with LATE
// hid that reason from `plans`, `status` and the stalled condition (novox/hq issue 920).
what = p.Note
// And when the builds are held after a failed release, they wait for a person, not for a build, and the
// operator is asked to release them (novox/hq issue 1732): said so, and not late.
if strings.Contains(p.Note, errUngated.Error()) && backlogNow.held != "" {
what += " — held after a failed delivery until a person releases it (the operator is asked)"
late = ""
}
return fmt.Sprintf("%s · %s, held: %s, for %s%s", name, where, what, since, late)
}
return fmt.Sprintf("%s · %s, %s for %s%s", name, where, what, since, late)
}
@@ -2230,7 +2277,7 @@ func plansCommand(ctx context.Context, args []string) error {
bounds.of(p.Repository), func(module string) []string {
on, _ := inv.Running(ctx, module)
return on
}, waitsOf(ctx, inv, []inventory.Plan{p})))
}, waitsOf(ctx, inv, []inventory.Plan{p}))+heldSaid(p, heldNow(ctx, inv)))
if r := p.Release; r != nil {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
@@ -2425,11 +2472,12 @@ func plansCommand(ctx context.Context, args []string) error {
return on
}
waits := waitsOf(ctx, inv, plans)
held := heldNow(ctx, inv)
for _, p := range plans {
if p.Batch() {
continue
}
fmt.Printf("%-28s %s\n", p.ID, planLineSaid(p, now, pause, bounds.of(p.Repository), running, waits))
fmt.Printf("%-28s %s%s\n", p.ID, planLineSaid(p, now, pause, bounds.of(p.Repository), running, waits), heldSaid(p, held))
}
return nil
}
+1 -1
View File
@@ -152,7 +152,7 @@ func answersRightly(q dnsmessage.Question) *dnsmessage.Message {
func quickResolvers(t *testing.T, port string) {
t.Helper()
before, within, pause := resolverPort, resolverWithin, resolverPause
resolverPort, resolverWithin, resolverPause = port, 150*time.Millisecond, 10*time.Millisecond
resolverPort, resolverWithin, resolverPause = port, 60*time.Millisecond, 5*time.Millisecond
t.Cleanup(func() { resolverPort, resolverWithin, resolverPause = before, within, pause })
}
+27 -3
View File
@@ -39,6 +39,10 @@ type sendable struct {
// ReadsModules says the machine's node-engine reads a module per resource and a build per module (novox/hq issue
// 1145): only then are they sent, because an older node-engine refuses a key it does not know, whole.
ReadsModules bool
// fileReferences says the machine's node-engine reports the capability of reading a file given as a digest and a size
// (declaration.FileReferencesCapability, novox/hq ADR 0486 §5): then every file over 64 KiB that may travel so does (Body,
// objects). Not on the wire as a field; the bodies differ.
fileReferences bool
// moduleOf is the composer's owner of each resource id, and buildIDs the build each module carries (its id):
// kept beside the resources, and on the wire only to a machine that reads them.
moduleOf map[string]string
@@ -115,12 +119,16 @@ type adoptionEnvelope struct {
// Body is the declaration's bytes, as sent and as digested.
func (s sendable) Body() ([]byte, error) {
envelope := map[string]any{"declaration": 1, "resources": s.Resources}
resources, _, err := s.byReference()
if err != nil {
return nil, err
}
envelope := map[string]any{"declaration": 1, "resources": resources}
if s.ReadsModules {
// The module of each resource the composer owns (what the mesh adds of its own has none), and the build of each
// module (novox/hq issue 1145): to a machine that said it reads them, and no other.
named := make([]map[string]any, len(s.Resources))
for n, r := range s.Resources {
named := make([]map[string]any, len(resources))
for n, r := range resources {
module, owned := s.moduleOf[fmt.Sprint(r["id"])]
if !owned {
named[n] = r
@@ -165,3 +173,19 @@ func (s sendable) Body() ([]byte, error) {
}
return json.Marshal(envelope)
}
// byReference is the resources as the machine is sent them: with the contents of its large files replaced by their digests when
// it reports it reads them (issue 1696), and the contents it replaced, by the name of the object that holds each.
func (s sendable) byReference() ([]map[string]any, map[string][]byte, error) {
if !s.fileReferences {
return s.Resources, nil, nil
}
return referencing(s.Resources)
}
// objects are the contents the body names by digest, by the name of each one's object: what is put in the bucket before the
// body is sent. Empty for a machine that is sent every content inline.
func (s sendable) objects() (map[string][]byte, error) {
_, objects, err := s.byReference()
return objects, err
}
+16
View File
@@ -244,6 +244,22 @@ var signalsTable = []signalRow{
newest: func(f *signalFacts) time.Time {
return newestOf(f.failed, func(p failedPlanFacts) time.Time { return p.since })
}},
{Row: "S24", Signal: "a move of a recorded module that waits for a person's push is asked of the operator",
Emitter: "controller's plan", Trigger: "each machine with a move of a recorded module held for a person (novox/hq ADR 0377, issue 1732)",
Bound: "none: raised on the first look that finds the machine holding such a move, naming the modules and the " +
"machine, with Release (the machine's push) as the answer; ended when the machine no longer waits, whoever " +
"pushed it. A hold a person chose (a module set to record with a why, the bus) is not asked.",
Kind: kindMovesHeld, Severity: conditions.Warning, Phase: 3,
needs: func(f *signalFacts) error { return f.heldErr }, watch: watchMovesHeld,
newest: func(f *signalFacts) time.Time { return time.Time{} }},
{Row: "S25", Signal: "a module left out of a send because the artifact store lacks an image it names",
Emitter: "controller's send", Trigger: "each send whose declaration names an image of the store that the store does not hold (novox/hq issue 1742)",
Bound: "none: raised on the first look at a registered build that names such an image, naming the module, each " +
"machine that runs it and the image; every send leaves that module out; the store is asked again at each " +
"look and the condition ends when it holds the image",
Kind: kindImageMissing, Severity: conditions.Warning, Phase: 3,
needs: func(f *signalFacts) error { return f.imagesErr }, watch: watchMissingImages,
newest: func(f *signalFacts) time.Time { return time.Time{} }},
{Row: "S20", Signal: "a send refused for the assignment generation it was composed from",
Emitter: "node-engine", Trigger: "each refusal (novox/hq issue 234)",
Bound: "none: raised at the first refusal, naming its sender, the generation it came from and the one the " +
+13
View File
@@ -171,6 +171,19 @@ var suppressions = map[string]suppression{
since: f.now.Add(-time.Second)}}
},
},
// A machine holding a move of a recorded module for a person is asked at its first look (novox/hq issue 1732).
// Inside: nothing waits there.
"S24": {
inside: func(f *signalFacts) { f.held = nil },
past: func(f *signalFacts) { f.held = []heldFacts{aHeldMailu()} },
},
// A registered build naming an image the artifact store lacks is raised at the first look (novox/hq issue 1742).
"S25": {
inside: func(f *signalFacts) { f.images = nil },
past: func(f *signalFacts) {
f.images = []imageFacts{{missingImage{Node: "anchor", Module: "automx", Image: automxImage}}}
},
},
// A batch queued behind an open walk past that walk's bound, named.
"S19": {
inside: func(f *signalFacts) {
+8
View File
@@ -58,6 +58,12 @@ type signalFacts struct {
// failed are the delivery plans that failed and are asked of the operator (S23, novox/hq ADR 0449).
failed []failedPlanFacts
failedErr error
// held are the machines with moves of recorded modules waiting for a person's push (S24, novox/hq issue 1732).
held []heldFacts
heldErr error
// images are the modules left out of a send because the artifact store lacks an image they name (S25, issue 1742).
images []imageFacts
imagesErr error
// refusedSends are the sends a machine refused within the hour for the generation they were composed
// from, each naming its sender (S20, novox/hq issue 234).
refusedSends []inventory.Send
@@ -357,6 +363,8 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
f.batches, f.plansErr = gatherBatches(ctx, inv, now)
}
f.failed, f.failedErr = gatherFailedPlans(ctx, inv, now)
f.held, f.heldErr = gatherHeldMoves(ctx, inv)
f.images, f.imagesErr = gatherMissingImagesHere(ctx, inv)
f.loop, f.loopErr = w.gatherLoop()
f.asks, f.asksErr = w.gatherAsks(ctx, inv)
if open, err := w.keeper.Open(ctx); err != nil {
+1 -1
View File
@@ -3,7 +3,7 @@ module github.com/novox/mesh-controller
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021144048-d4deecdb380b
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c
github.com/jackc/pgx/v5 v5.10.0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
+2 -2
View File
@@ -1,7 +1,7 @@
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5 h1:E5stfIeCN6uYgA9TCv+4K8JgaURg3zQppvUgJLaDMP4=
git.novox.be/novox/mesh-host v0.0.0-20261018171808-9d9e3a46c4f5/go.mod h1:4wyoCr6IIIIarEUuROThWDj7DAvJvIidqMjWcpSnOwE=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021144048-d4deecdb380b h1:5UlQ8ytlHNgqlm5AbwJEnZwuf1EWUiNyXX2hcOV9Yro=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261021144048-d4deecdb380b/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c h1:UKlwMw6fmesDqe0aL0lfI/ivhorsGjAvh9XefFjhNI4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261022132648-bd7731f3588c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+9
View File
@@ -46,6 +46,7 @@ func CheckStreams() []Stream {
Retention: RetentionWorkQueue,
MaxAge: 7 * 24 * 60 * 60,
MaxMsgsPerSubject: 1,
Direct: true,
Why: "the checks of pull requests marked ready, taken after a delivery build and before any other check " +
"(novox/hq ADR 0406 §3); one subject a pull request, the newest request kept, so a newer one replaces a " +
"request still waiting",
@@ -56,6 +57,7 @@ func CheckStreams() []Stream {
Retention: RetentionWorkQueue,
MaxAge: 7 * 24 * 60 * 60,
MaxMsgsPerSubject: 1,
Direct: true,
Why: "every other check, taken last and in the order asked (novox/hq ADR 0406 §3); one subject a pull " +
"request, the newest request kept; one left behind for a head already judged is dropped when taken",
},
@@ -98,6 +100,13 @@ func holderCheckGrants() []string {
"$KV."+CheckRunningBucket+".>",
"$JS.API.STREAM.INFO.KV_"+CheckRunningBucket,
"$JS.API.DIRECT.GET.KV_"+CheckRunningBucket+".>")
// And, to hold checks while a delivery builds (ADR 0483, issues 1687 and 1688): the bucket of the plans building, **read and
// never written** (the controller's alone, so no `$KV.` subject of it is granted), and the oldest waiting ask of each check
// queue, read by a direct get that takes nothing from the queue and so counts against no delivery of the ask.
pub = append(pub, readGrants(PlansBuildingBucket, "")...)
for _, w := range CheckWorkers() {
pub = append(pub, "$JS.API.DIRECT.GET."+w.Stream, "$JS.API.DIRECT.GET."+w.Stream+".>")
}
return pub
}
+34
View File
@@ -40,3 +40,37 @@ func TestTheBuildAgentReadsAndWatchesThePullRequestsKeys(t *testing.T) {
}
}
}
// Issue 1687 (ADR 0483 clause 2): a holder of the build seat reads the bucket of the plans building and writes none of it, and
// reads the oldest waiting ask of each check queue by a direct get, which takes nothing from the queue.
func TestTheBuildSeatsHoldersReadThePlansBuildingAndWriteNone(t *testing.T) {
seat := Seat{Name: TheBuildSeat, Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
p := Principal{Kind: KindModule, Node: "one", Module: "build-agent", Holds: []Seat{seat}, PasswordHash: "x"}
perms, err := PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
stream := "KV_" + PlansBuildingBucket
for _, s := range []string{
"$JS.API.STREAM.INFO." + stream,
"$JS.API.DIRECT.GET." + stream + ".>",
"$JS.API.CONSUMER.CREATE." + stream + ".>",
"$JS.API.DIRECT.GET.CHECKS_READY",
"$JS.API.DIRECT.GET.CHECKS_OTHER",
} {
has(t, perms.Publish, s)
}
for _, s := range perms.Publish {
if SubjectsOverlap(s, "$KV."+PlansBuildingBucket+".plan-1") {
t.Errorf("a holder of the build seat may write %s through %q", PlansBuildingBucket, s)
}
}
// And the controller alone writes it, as the writers table says.
for _, row := range WritersTable {
for _, s := range row.Subjects {
if s == "$KV."+PlansBuildingBucket+".>" && row.Writes(p, "$KV."+PlansBuildingBucket+".plan-1") {
t.Errorf("%q lets the build seat's holder write the plans building", row.State)
}
}
}
}
+39 -3
View File
@@ -63,6 +63,26 @@ var (
// repository: the controller reads the ask's warrant from the router and accepts only the operator's Approve of
// those heads. Written by the controller alone, from the announcement, when it holds the lease.
DeliveryMergesBucket = BucketName(ControllerSeat, "delivery-merges")
// PlansBuildingBucket holds one key per delivery plan in its build phase (novox/hq ADR 0483 clause 2, issue 1687): the key
// is `<plan id>` (`plan-<digits>`), its value the plan, the time its build phase began and the time of the last refresh.
// The controller (lease holder) puts it when the plan's first build is asked, puts it again about every thirty seconds
// while the plan builds, and deletes it when the phase ends (every build of the last tier built, or the plan fails, stops
// or waits on the operator); the bucket's age, PlansBuildingKeptFor, is how long a key outlives a controller that
// stopped. **Read by the build seat's holders and the checker, written by the controller alone**: a holder takes no new
// check while any key is there, but for one that waited past ADR 0483's bound. A reader lists the keys and reads none of
// their values to decide that (`link.BuildingPlans`).
PlansBuildingBucket = BucketName(ControllerSeat, PlansBuilding)
)
// PlansBuilding is the local name of the bucket of the plans building, as a reader names it in `reads`
// (`mesh-controller.plans-building`).
const PlansBuilding = "plans-building"
// PlansBuildingKeptFor is the age of a plan-building key: the controller puts it again at PlansBuildingRefresh, so a
// controller that stopped leaves checks held at most this long (ADR 0483 clause 2).
const (
PlansBuildingKeptFor = 2 * time.Minute
PlansBuildingRefresh = 30 * time.Second
)
// DeliveryMergesKeptFor is how long a merge mesh-delivery announced is kept: a week, as the events are.
@@ -133,7 +153,7 @@ func ChainBuckets() []string { return []string{BuildsBucket, RolloutsBucket, Ass
// ControllerMirrors are the local names of the controller's buckets a module may read: the record mirrors and the
// chain's buckets.
func ControllerMirrors() []string {
return []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined, ChainBuilds, ChainRollouts, ChecksRunning}
return []string{MirrorAssignments, MirrorDeclaredConfig, MirrorPredefined, ChainBuilds, ChainRollouts, ChecksRunning, PlansBuilding}
}
// MirrorsWrittenByAnother are the controller's mirrors whose writer is not the controller, each with its reason: the rule
@@ -204,14 +224,14 @@ func IsControllerBucket(bucket string) bool {
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket || bucket == PredefinedBucket ||
bucket == MergeWindowBucket || bucket == RolloutSilentBucket || bucket == CheckRunningBucket ||
bucket == DeliveryMergesBucket || bucket == BuildsBucket || bucket == RolloutsBucket ||
bucket == delivery.HaveBucket
bucket == PlansBuildingBucket || bucket == delivery.HaveBucket
}
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
func ControllerBuckets() []string {
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
AssignmentsBucket, DeclaredConfigBucket, PredefinedBucket, MergeWindowBucket, RolloutSilentBucket, CheckRunningBucket,
DeliveryMergesBucket, BuildsBucket, RolloutsBucket, delivery.HaveBucket}
DeliveryMergesBucket, PlansBuildingBucket, BuildsBucket, RolloutsBucket, delivery.HaveBucket}
}
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
@@ -354,6 +374,22 @@ func (j *JetStream) EnsureControllerBuckets() error {
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", CheckRunningBucket, err)
}
// **The plans building: keys that age** (ADR 0483 clause 2, issue 1687). One message a key, each refresh a new one, so the
// age of the bucket is how long a key outlives a controller that stopped refreshing it.
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: PlansBuildingBucket,
Description: "the delivery plans in their build phase (novox/hq ADR 0483): `<plan id>` holds the plan and when its " +
"build phase began; put when the plan's first build is asked and about every thirty seconds, deleted when the " +
"phase ends, gone within two minutes of a controller that stopped; read by the build seat's holders, which " +
"take no new check while a key is there; written by the controller alone",
History: 1,
TTL: PlansBuildingKeptFor,
MaxValueSize: 4 << 10,
MaxBytes: 8 << 20,
Storage: jetstream.FileStorage,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", PlansBuildingBucket, err)
}
// **The merges mesh-delivery announced: keys that age** (ADR 0420 §7, issue 1025). One message a key; a week.
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: DeliveryMergesBucket,
+123
View File
@@ -0,0 +1,123 @@
package broker
import (
"context"
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go/jetstream"
)
// The object buckets a declaration's large files travel in (novox/hq ADR 0486, issues 1696 and 1697): **one bucket for each
// node**, `declaration-files-<node>`.
//
// **The controller alone writes them, and a node-engine reads an object by its name from its own bucket** (ADR 0486 §2): its
// account may neither list the bucket nor write it, no account may touch another node's, and no other account has any access.
// Why a bucket each: a subject pattern cannot tell a name from a wildcard. A grant of `...M.*` matches a publish to the literal
// subject `...M.*` too (probed on nats-server 2.11.17 by the review of this issue: a node created a consumer filtered on
// `$O.<bucket>.C.*` and read every object of the bucket), so in one shared bucket a node could read every other node's files.
// With a bucket each, the same trick reads only the node's own files, which are the files of its own declaration.
//
// A bucket of the bus's object store is a stream, OBJ_<bucket>, whose messages are the objects' metadata
// (`$O.<bucket>.M.<base64 of the name>`) and chunks (`$O.<bucket>.C.<nuid>`).
// DeclarationFilesBucket is the prefix of the buckets' names, which mesh-sdk fixes for the node-engine and the controller alike.
const DeclarationFilesBucket = declaration.FileBucketPrefix
// DeclarationFilesBucketOf is the bucket a node's files are in: mesh-sdk's FileBucketFor, the one place the name is composed. A
// node name it refuses (one that could carry a subject wildcard or separator into a grant) is an error, never a bucket.
func DeclarationFilesBucketOf(node string) (string, error) { return declaration.FileBucketFor(node) }
// DeclarationFilesStreamOf is the stream under a node's bucket: mesh-sdk's FileStreamFor.
func DeclarationFilesStreamOf(node string) (string, error) { return declaration.FileStreamFor(node) }
// DeclarationFilesMaxBytes is a bucket's size: the objects are immutable and regenerable, and one node's hold well under a hundred
// megabytes.
const DeclarationFilesMaxBytes = 1 << 30
// DeclarationFilesKeptFor is how long an object stays after no node's latest declaration names it (ADR 0486 §3).
const DeclarationFilesKeptFor = 24 * time.Hour
// DeclarationFilesWrites is the one pattern the controller may publish to write a node's bucket: every object's metadata and
// chunks. The server's API for the stream is the controller's `$JS.API.>`.
func DeclarationFilesWrites(node string) (string, error) {
bucket, err := DeclarationFilesBucketOf(node)
if err != nil {
return "", err
}
return "$O." + bucket + ".>", nil
}
// DeclarationFilesReads is what a node-engine's account may publish to read an object of its own bucket by its name, and nothing
// more. The set is the one node-engine's own call (mesh-host internal/link FileObject) makes, for B the node's bucket and S its
// stream (issue 1697, notes 4 and 5):
//
// - the object's metadata, by a direct get of the last message of the one subject that names it. **Only the suffixed
// form**: the bare `$JS.API.DIRECT.GET.S` takes a sequence number and would read any message of the stream.
// - the ordered consumer on that object's chunks: `$JS.API.CONSUMER.CREATE.S.<name>.$O.B.C.<nuid>`. **Not `S.>`**:
// a consumer filtered on `$O.B.M.>` would read every object's name.
// - the flow-control replies of that consumer, without which a large object stalls at the flow-control window.
// - the consumer's deletion, best effort: the consumer's inactivity threshold removes it if this is refused.
//
// **Never granted:** STREAM.INFO (a `subjects_filter` of `>` lists every name), STREAM.MSG.GET (any message by its sequence), a
// publish to `$O.B.>` (a write), and any subject of another node's stream. A literal `*` in a subject matches the grant's `*`: it
// reaches the objects of this node's own bucket and no other (above).
func DeclarationFilesReads(node string) ([]string, error) {
bucket, err := DeclarationFilesBucketOf(node)
if err != nil {
return nil, err
}
stream, err := DeclarationFilesStreamOf(node)
if err != nil {
return nil, err
}
return []string{
"$JS.API.DIRECT.GET." + stream + ".$O." + bucket + ".M.*",
"$JS.API.CONSUMER.CREATE." + stream + ".*.$O." + bucket + ".C.*",
"$JS.API.CONSUMER.DELETE." + stream + ".*",
"$JS.FC." + stream + ".>",
}, nil
}
// EnsureDeclarationFiles creates the buckets of these nodes if absent and brings their options to match. A bucket exists whenever the
// bus does, so a node that is starting finds it (ADR 0486 §4). An update, never a delete.
func (j *JetStream) EnsureDeclarationFiles(nodes []string) error {
js, err := jetstream.New(j.conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
return EnsureDeclarationFiles(ctx, js, nodes...)
}
// EnsureDeclarationFiles is the same on a JetStream the caller holds.
func EnsureDeclarationFiles(ctx context.Context, js jetstream.JetStream, nodes ...string) error {
for _, node := range nodes {
cfg, err := DeclarationFilesConfig(node)
if err != nil {
return err
}
if _, err := js.CreateOrUpdateObjectStore(ctx, cfg); err != nil {
return fmt.Errorf("asserting the object bucket %s: %w", cfg.Bucket, err)
}
}
return nil
}
// DeclarationFilesConfig is a node's bucket as the controller asserts it.
func DeclarationFilesConfig(node string) (jetstream.ObjectStoreConfig, error) {
bucket, err := DeclarationFilesBucketOf(node)
if err != nil {
return jetstream.ObjectStoreConfig{}, err
}
return jetstream.ObjectStoreConfig{
Bucket: bucket,
Description: "the contents of the files over 64 KiB of the declaration of " + node + ", each under the hex of its sha256 " +
"digest (novox/hq ADR 0486): written by the controller alone, read by the node-engine of that node by name, removed " +
"a day after the node's latest declaration no longer names it",
MaxBytes: DeclarationFilesMaxBytes,
Storage: jetstream.FileStorage,
}, nil
}
@@ -0,0 +1,316 @@
//go:build integration
package broker
import (
"bytes"
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
)
// nodeEnginesFileObject is the node-engine's own call (mesh-host internal/link files.go, OverNATS.FileObject), as the
// engine makes it: the object's metadata by a direct get of one subject, then its chunks through an ordered consumer
// bound to the stream. It is copied here so the grants are judged by the call that has to work, and its subjects are the
// ones the grant names (issue 1697, notes 4 and 5); the node-engine's own integration test holds the other copy.
func nodeEnginesFileObject(ctx context.Context, nc *nats.Conn, node, name string) ([]byte, error) {
js, err := nc.JetStream(nats.MaxWait(10 * time.Second))
if err != nil {
return nil, err
}
stream, err := DeclarationFilesStreamOf(node)
if err != nil {
return nil, err
}
bucket, err := DeclarationFilesBucketOf(node)
if err != nil {
return nil, err
}
meta, err := js.GetLastMsg(stream, "$O."+bucket+".M."+base64.URLEncoding.EncodeToString([]byte(name)),
nats.DirectGet(), nats.Context(ctx))
if err != nil {
return nil, err
}
var info struct {
NUID string `json:"nuid"`
Size uint64 `json:"size"`
}
if err := json.Unmarshal(meta.Data, &info); err != nil {
return nil, err
}
sub, err := js.SubscribeSync("$O."+bucket+".C."+info.NUID, nats.OrderedConsumer(), nats.BindStream(stream))
if err != nil {
return nil, err
}
defer sub.Unsubscribe()
body := make([]byte, 0, info.Size)
for uint64(len(body)) < info.Size {
msg, err := sub.NextMsgWithContext(ctx)
if err != nil {
return nil, err
}
body = append(body, msg.Data...)
}
return body, nil
}
// **The buckets' grants on a real server, judged by the node-engine's own call** (novox/hq ADR 0486 §2, issue 1697, 1697/AC2): the
// controller's account writes an object into each node's bucket, a node-engine's account reads one of its own bucket by its name,
// a large object included, and is refused a list of the bucket (a stream info asking for every subject), a message get by
// sequence, a direct get by sequence, a consumer on the names, and a write; it is refused every one of those and the read by name on
// another node's bucket, **the literal-`*` forms included** (a published subject `...M.*` or `...C.*` matches the grant's `*`: on
// its own bucket that reaches its own files, on another's it is refused); another module's account is refused all of them.
func TestNatsANodeEngineReadsAnObjectOfItsOwnBucketByNameAndNothingElse(t *testing.T) { // 1697/AC2
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
users, err := Users(Records{Nodes: []string{"alpha", "beta"}, Assigned: map[string][]Declared{
"alpha": {{Module: RuntimeModule}, {Module: "other-module"}},
}})
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = string(hash)
}
accounts, err := ComposeAccounts(users)
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
connect := func(user string) (*nats.Conn, chan error) {
t.Helper()
refused := make(chan error, 16)
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused <- err }))
if err != nil {
t.Fatalf("%s cannot connect: %v", user, err)
}
t.Cleanup(nc.Close)
return nc, refused
}
// The controller makes the buckets (as it does at its start) and writes into each: for alpha a small object and one of
// several chunks, which needs the consumer's flow control; for beta one that holds a marker alpha must never read.
controller, _ := connect(ControllerName)
cjs, err := jetstream.New(controller)
if err != nil {
t.Fatal(err)
}
if err := EnsureDeclarationFiles(ctx, cjs, "alpha", "beta"); err != nil {
t.Fatalf("the controller cannot make the buckets: %v", err)
}
if err := EnsureDeclarationFiles(ctx, cjs, "alpha", "beta"); err != nil {
t.Fatalf("asserting them again: %v", err)
}
put := func(node string, content []byte) string {
t.Helper()
store, err := cjs.ObjectStore(ctx, mustBucket(t, node))
if err != nil {
t.Fatal(err)
}
sum := sha256.Sum256(content)
name := hex.EncodeToString(sum[:])
if _, err := store.PutBytes(ctx, name, content); err != nil {
t.Fatalf("the controller cannot write the object %s of %s: %v", name[:8], node, err)
}
return name
}
small := bytes.Repeat([]byte("small file\n"), 100)
large := make([]byte, 900<<10)
for i := range large {
large[i] = byte(i*7 + i>>8)
}
secret := bytes.Repeat([]byte("BETA-SECRET "), 50)
smallName, largeName, betaName := put("alpha", small), put("alpha", large), put("beta", secret)
alpha, alphaRefused := connect("node.alpha")
for name, want := range map[string][]byte{smallName: small, largeName: large} {
got, err := nodeEnginesFileObject(ctx, alpha, "alpha", name)
if err != nil {
t.Fatalf("a node-engine cannot read the object %s of its own bucket by its name: %v", name[:8], err)
}
if !bytes.Equal(got, want) {
t.Fatalf("the node-engine read %d bytes of the object %s, want %d", len(got), name[:8], len(want))
}
}
beta, _ := connect("node.beta")
if got, err := nodeEnginesFileObject(ctx, beta, "beta", betaName); err != nil || !bytes.Equal(got, secret) {
t.Fatalf("beta's node-engine reads its own object: %v", err)
}
select {
case err := <-alphaRefused:
t.Fatalf("a node-engine's own read was refused something: %v", err)
default:
}
// What each account is refused, judged by the server.
alphaStream, alphaBucket := mustStream(t, "alpha"), mustBucket(t, "alpha")
betaStream, betaBucket := mustStream(t, "beta"), mustBucket(t, "beta")
listing := []byte(`{"subjects_filter":">"}`)
bySeq := []byte(`{"seq":1}`)
type refusal struct {
what string
do func(nc *nats.Conn)
}
onBucket := func(stream, bucket, which string) []refusal {
return []refusal{
{which + ": a list of the bucket (stream info asking for every subject)", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.STREAM.INFO."+stream, listing, 300*time.Millisecond)
}},
{which + ": a message get by sequence", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.STREAM.MSG.GET."+stream, bySeq, 300*time.Millisecond)
}},
{which + ": a direct get by sequence", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.DIRECT.GET."+stream, bySeq, 300*time.Millisecond)
}},
{which + ": a consumer on every object's name", func(nc *nats.Conn) {
js, _ := nc.JetStream(nats.MaxWait(500 * time.Millisecond))
if sub, err := js.SubscribeSync("$O."+bucket+".M.>", nats.OrderedConsumer(), nats.BindStream(stream)); err == nil {
sub.Unsubscribe()
}
}},
{which + ": a write of an object's metadata", func(nc *nats.Conn) {
_ = nc.Publish("$O."+bucket+".M.abc", []byte(`{}`))
_ = nc.Flush()
}},
{which + ": a write of a chunk", func(nc *nats.Conn) {
_ = nc.Publish("$O."+bucket+".C.abc", []byte("x"))
_ = nc.Flush()
}},
}
}
refusedWith := func(who string, nc *nats.Conn, ch chan error, r refusal) {
t.Helper()
for len(ch) > 0 {
<-ch
}
r.do(nc)
_ = nc.Flush()
deadline := time.After(3 * time.Second)
for {
select {
case err := <-ch:
if errors.Is(err, nats.ErrPermissionViolation) {
return
}
case <-deadline:
t.Errorf("the server did not refuse %s %s", who, r.what)
return
}
}
}
// Alpha's account on its own bucket (list, sequence reads, names, writes) and on beta's (all of that, and the read by name,
// and the literal-`*` probes of the review).
reachBeta := []refusal{
{"beta's bucket: a read of one object by its name", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.DIRECT.GET."+betaStream+".$O."+betaBucket+".M."+
base64.URLEncoding.EncodeToString([]byte(betaName)), nil, 300*time.Millisecond)
}},
{"beta's bucket: a direct get with a literal `*` for the name", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.DIRECT.GET."+betaStream+".$O."+betaBucket+".M.*", nil, 300*time.Millisecond)
}},
{"beta's bucket: a consumer with a literal `*` for the chunks", func(nc *nats.Conn) {
js, _ := nc.JetStream(nats.MaxWait(500 * time.Millisecond))
if sub, err := js.SubscribeSync("$O."+betaBucket+".C.*", nats.OrderedConsumer(), nats.BindStream(betaStream)); err == nil {
sub.Unsubscribe()
}
}},
}
all := append(append(onBucket(alphaStream, alphaBucket, "its own bucket"), onBucket(betaStream, betaBucket, "beta's bucket")...), reachBeta...)
for _, r := range all {
nodeNC, nodeCh := connect("node.alpha")
refusedWith("a node-engine", nodeNC, nodeCh, r)
}
// Another module is refused all of that on both buckets, and the reads by name on both.
for _, r := range append(append(onBucket(alphaStream, alphaBucket, "alpha's bucket"), reachBeta...),
refusal{"alpha's bucket: a read of one object by its name", func(nc *nats.Conn) {
_, _ = nc.Request("$JS.API.DIRECT.GET."+alphaStream+".$O."+alphaBucket+".M."+
base64.URLEncoding.EncodeToString([]byte(smallName)), nil, 300*time.Millisecond)
}}) {
otherNC, otherCh := connect("alpha.other-module")
refusedWith("another module", otherNC, otherCh, r)
}
// The literal `*` on alpha's own bucket reaches alpha's own files and never beta's: the chunks it is given hold no marker of beta's.
js, err := alpha.JetStream(nats.MaxWait(2 * time.Second))
if err != nil {
t.Fatal(err)
}
if sub, err := js.SubscribeSync("$O."+alphaBucket+".C.*", nats.OrderedConsumer(), nats.BindStream(alphaStream)); err == nil {
for i := 0; i < 20; i++ {
msg, err := sub.NextMsg(300 * time.Millisecond)
if err != nil {
break
}
if bytes.Contains(msg.Data, []byte("BETA-SECRET")) {
t.Fatal("a literal `*` on alpha's bucket gave it a chunk of beta's file")
}
}
sub.Unsubscribe()
}
// Nothing was written by the refused: each bucket holds what the controller wrote.
for node, want := range map[string]int{"alpha": 2, "beta": 1} {
store, err := cjs.ObjectStore(ctx, mustBucket(t, node))
if err != nil {
t.Fatal(err)
}
infos, err := store.List(ctx)
if err != nil || len(infos) != want {
t.Errorf("the bucket of %s holds %d objects (%v), want the %d the controller wrote", node, len(infos), err, want)
}
}
}
func mustBucket(t *testing.T, node string) string {
t.Helper()
b, err := DeclarationFilesBucketOf(node)
if err != nil {
t.Fatal(err)
}
return b
}
func mustStream(t *testing.T, node string) string {
t.Helper()
b, err := DeclarationFilesStreamOf(node)
if err != nil {
t.Fatal(err)
}
return b
}
+131
View File
@@ -0,0 +1,131 @@
package broker
import (
"slices"
"strings"
"testing"
)
// 1697/AC1: the composed accounts give a node-engine exactly the four reads of one object by its name, on its own node's bucket and
// no other's; the controller the write of every node's bucket; and no other principal anything of any bucket.
func TestEachNodesBucketIsWrittenByTheControllerAndReadByNameByThatNodesEngine(t *testing.T) { // 1697/AC1
node, err := PermissionsFor(Principal{Kind: KindNode, Node: "alpha", PasswordHash: "x"})
if err != nil {
t.Fatal(err)
}
var mine []string
for _, s := range node.Publish {
if strings.Contains(s, "declaration-files") {
mine = append(mine, s)
}
}
want := []string{
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.*.$O.declaration-files-alpha.C.*",
"$JS.API.CONSUMER.DELETE.OBJ_declaration-files-alpha.*",
"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.*",
"$JS.FC.OBJ_declaration-files-alpha.>",
}
if !slices.Equal(mine, want) {
t.Fatalf("a node-engine may publish %q of the buckets, want exactly %q", mine, want)
}
// What would make a list, a write, or a read of another node's files: none of it, in any pattern a node holds. (The real
// server's judgement of the wildcard forms is the live test's.)
for _, refused := range []string{
"$JS.API.STREAM.INFO.OBJ_declaration-files-alpha",
"$JS.API.STREAM.MSG.GET.OBJ_declaration-files-alpha",
"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha",
"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.abc.def",
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.x.$O.declaration-files-alpha.M.YWJj",
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.x.$O.declaration-files-alpha.M.YWJj.more",
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha",
"$O.declaration-files-alpha.M.abc",
"$O.declaration-files-alpha.C.abc",
// another node's bucket, however it is named
"$JS.API.DIRECT.GET.OBJ_declaration-files-beta.$O.declaration-files-beta.M.YWJj",
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-beta.h4sh.$O.declaration-files-beta.C.NUID",
"$JS.API.CONSUMER.DELETE.OBJ_declaration-files-beta.h4sh",
"$JS.FC.OBJ_declaration-files-beta.x",
// and the old shared bucket
"$JS.API.DIRECT.GET.OBJ_declaration-files.$O.declaration-files.M.YWJj",
} {
if MayPublish(node, refused) {
t.Errorf("a node-engine may publish %s, which lists or writes a bucket or reads another node's", refused)
}
}
for _, kept := range []string{
"$JS.API.DIRECT.GET.OBJ_declaration-files-alpha.$O.declaration-files-alpha.M.YWJj",
"$JS.API.CONSUMER.CREATE.OBJ_declaration-files-alpha.h4sh.$O.declaration-files-alpha.C.NUID",
} {
if !MayPublish(node, kept) {
t.Errorf("a node-engine may not publish %s, which reads one object of its own bucket by its name", kept)
}
}
controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x", FilesFor: []string{"alpha", "beta"}})
if err != nil {
t.Fatal(err)
}
for _, s := range []string{"$O.declaration-files-alpha.M.abc", "$O.declaration-files-beta.C.nuid"} {
if !MayPublish(controller, s) {
t.Errorf("the controller may not write %s", s)
}
}
if MayPublish(controller, "$O.declaration-files-gamma.M.abc") {
t.Error("the controller may write the bucket of a node it was not given")
}
for _, p := range []Principal{
{Kind: KindModule, Module: "mesh-vault", PasswordHash: "x"},
{Kind: KindNodeTools, Node: "alpha", PasswordHash: "x"},
{Kind: KindPerson, Module: "jochen", Invokes: []string{"*"}, PasswordHash: "x"},
} {
perms, err := PermissionsFor(p)
if err != nil {
t.Fatal(err)
}
for _, s := range append(append([]string(nil), perms.Publish...), perms.Subscribe...) {
if strings.Contains(s, "declaration-files") {
t.Errorf("%s holds %s, and no account but the controller's and the node-engines' has any access", p.Username(), s)
}
}
}
}
// The users the controller composes carry each node's bucket to the controller and to that node alone.
func TestTheComposedUsersGiveEachNodeItsOwnBucketAndTheControllerAll(t *testing.T) {
users, err := Users(Records{Nodes: []string{"alpha", "beta"}})
if err != nil {
t.Fatal(err)
}
var controller Principal
for _, u := range users {
if u.Kind == KindController {
controller = u
}
}
if !slices.Equal(controller.FilesFor, []string{"alpha", "beta"}) {
t.Fatalf("the controller writes the buckets of %v", controller.FilesFor)
}
}
// A node name that could carry a subject wildcard or separator into a grant is refused with an error, never composed into a bucket
// (ADR 0488): the grants of such a name would reach another node's bucket.
func TestANodeNameThatCarriesAWildcardGetsNoBucketAndNoGrant(t *testing.T) {
for _, node := range []string{"", "*", ">", "a.b", "a b", "a*"} {
if b, err := DeclarationFilesBucketOf(node); err == nil {
t.Errorf("node %q got the bucket %q", node, b)
}
if _, err := DeclarationFilesReads(node); err == nil {
t.Errorf("node %q got read grants", node)
}
if _, err := DeclarationFilesWrites(node); err == nil {
t.Errorf("node %q got write grants", node)
}
if _, err := DeclarationFilesConfig(node); err == nil {
t.Errorf("node %q got a bucket config", node)
}
}
if b, err := DeclarationFilesBucketOf("g14"); err != nil || b != "declaration-files-g14" {
t.Errorf("g14: %q %v", b, err)
}
}
-22
View File
@@ -2,10 +2,7 @@ package broker
import (
"context"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/hex"
"errors"
"fmt"
"os"
@@ -112,25 +109,6 @@ func DialPinned(url, fingerprint string, opts ...nats.Option) (*JetStream, error
return Dial(url, opts...)
}
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other.
func PinnedToFingerprint(want string) *tls.Config {
return &tls.Config{
InsecureSkipVerify: true, //nolint:gosec // replaced by the pin below, which is stricter
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("the bus presented no certificate")
}
sum := sha256.Sum256(rawCerts[0])
got := "sha256:" + hex.EncodeToString(sum[:])
if got != want {
return fmt.Errorf("the bus presented a certificate this mesh does not know (%s…), expected %s…", got[:23], want[:23])
}
return nil
},
}
}
// Conn is the connection itself, for what the mesh keeps off JetStream on purpose — a heartbeat,
// a tool call — where a lost message is answered by the next one or by a timeout the caller
// already handles (design 25 §3).
-11
View File
@@ -98,17 +98,6 @@ func (l Log) Config() jetstream.StreamConfig {
}
}
// LogIssued is one log an assignment may reach, by the name its module uses for it (novox/hq ADR
// 0297): its stream, the subject its entries go under, and whether it may append. Only the owner's
// instances are issued a log, so Writes is always true today; it is said so the runtime need not
// assume it.
type LogIssued struct {
Name string `json:"name"`
Stream string `json:"stream"`
Subject string `json:"subject"`
Writes bool `json:"writes"`
}
// logsIssuedFor is every log a module's code may reach, as its membership lists them: its own.
func logsIssuedFor(d Declared) []LogIssued {
var out []LogIssued
-76
View File
@@ -1,7 +1,6 @@
package broker
import (
"encoding/json"
"slices"
"sort"
"strings"
@@ -16,81 +15,6 @@ import (
// grant is the same composition read the other way. The shape issued today is the shape the mesh
// already had, so nothing moves when a membership first arrives; only who decides it moves.
// Membership is one assignment's subjects: what this instance of a module on this machine serves,
// and what it may reach.
type Membership struct {
Node string `json:"node"`
Module string `json:"module"`
// Serves is every address a tool of this instance answers on. `{tool}` stands for the tool's
// own name, which the module knows and the mesh does not need to: the mesh issues the address,
// the runtime fills the name. An address with a queue is shared with the module's other
// instances, and the bus hands each call to one of them; an address without is this instance's.
Serves []Served `json:"serves"`
// Seats is every verb of a seat this instance holds, at the subject the seat's callers use.
Seats []SeatServed `json:"seats,omitempty"`
// Emits is where an event of this module lands; `{event}` stands for the event's name.
Emits string `json:"emits"`
// Reaches is each tool this module may call, `<module>.<tool>`, to the subjects that reach it:
// the first is whichever instance answers, when the mesh issued one; the rest name a machine.
Reaches map[string][]string `json:"reaches,omitempty"`
// Tools is where this instance answers what it serves — the runtime's one verb of its own.
Tools string `json:"tools"`
// Receives is what this assignment is given for each requirement it receives, by requirement:
// the contributions of every module that asked for it, as the catalogue composed them (novox/hq
// ADR 0167). The same list its received file is written from, so the two cannot disagree; a
// requirement nobody contributed to is an empty list, never absent. Kept as JSON because the
// catalogue owns the shape of a contribution and the bus only carries it.
Receives map[string]json.RawMessage `json:"receives,omitempty"`
// Mesh is every machine's address on the private network — what a rule saying "from the mesh"
// resolves to in the packet filter, issued here from the same list (novox/hq ADR 0167). A
// module that must tell the mesh from the world, the route proxy serving an internal name, reads
// it here rather than keeping a definition of its own.
Mesh []string `json:"mesh,omitempty"`
// State is every bucket this module's code may reach, by the name it uses for each, and whether
// it may write it (novox/hq ADR 0201): the runtime answers a bundle's state verbs from this list
// and refuses, with the reason, what is not on it — the bus enforces only the union over every
// module on the machine.
State []StateIssued `json:"state,omitempty"`
// Logs is every log this module's code may reach, by the name it uses for each (novox/hq ADR 0297):
// its own and no other's, since a log is its owner's alone. The runtime answers a bundle's log verbs
// from this list and refuses, with the reason, a log not on it.
Logs []LogIssued `json:"logs,omitempty"`
// SeatTraffic is what this module's code may submit, say, hear, take, ask, answer and read on seats
// that name their caller or their kind (novox/hq ADR 0259 §3). The runtime carrying the module
// publishes, takes and answers for it only what is listed here: the bus enforces only the union
// over every module on the machine, so one module's code reaching another's name or kind through
// the runtime is the runtime's to refuse.
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are this module's own streams (novox/hq ADR 0363 §2): an event one of them carries is published
// on that stream's subject, never on Emits. Hears are the module streams it hears, each through a durable
// consumer on that stream the runtime binds beside the one on EVENTS (§8).
Streams []StreamIssued `json:"streams,omitempty"`
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server, the tool runner's
// alone (novox/hq ADR 0367 §2): the subjects its account is granted, under the names a session asks for.
// The tool runner refuses any family not listed here.
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is one address a tool is answered on.
type Served struct {
Subject string `json:"subject"`
Queue string `json:"queue,omitempty"`
}
// SeatServed is one verb of a held seat, where its callers ask.
type SeatServed struct {
Seat string `json:"seat"`
Verb string `json:"verb"`
Subject string `json:"subject"`
}
// MembershipSubject is the one address a runtime derives for itself: where its own membership is
// published, from the two names its credential carries. Everything else is in the membership.
func MembershipSubject(node, module string) string {
return "mesh.assignment." + node + "." + module
}
// Placements is where every module runs, for deciding which instance answers for the module.
type Placements struct {
// Nodes is each module's machines.
-15
View File
@@ -260,21 +260,6 @@ func ownStreamGrants(module string, streams []ModuleStream) []string {
return out
}
// StreamIssued is one of a module's own streams as its membership names it (ADR 0363 §2): the events it
// carries, as the bus spells them, and the subject an event of them is published on, `{event}` the event.
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears, as its membership names it (ADR 0363 §8): the
// stream and the durable consumer on it the runtime binds.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// streamsIssuedFor is every stream of its own a module's membership names.
func streamsIssuedFor(d Declared) []StreamIssued {
var out []StreamIssued
+42 -5
View File
@@ -104,6 +104,10 @@ type Principal struct {
// Username and inbox are <node>.<module>.<process id>.
Process string
// FilesFor are the nodes whose declaration-file buckets this principal writes, for the controller alone (novox/hq ADR 0486 §2):
// every node of the mesh, one bucket each.
FilesFor []string
Emits []string
Consumes []string
Serves []string
@@ -357,9 +361,11 @@ var VaultNodeKey = SeatVerb{Seat: "mesh-vault", Verb: "node-key"}
// VerbsTheControllerActsOnAWarrant are the other seat verbs the controller calls when the operator's warrant
// chooses them (novox/hq ADR 0259): a machine's service restarted, and a walk started or stopped through the
// controller's own `plans`. Named one by one; a node seat's on any machine, a mesh seat's flat.
// controller's own `plans`, a machine's held builds sent by its own `push` and the builds held after a failed release
// sent by its own `upgrade release-backlog` (novox/hq issue 1732). Named one by one; a node seat's on any machine, a
// mesh seat's flat.
var VerbsTheControllerActsOnAWarrant = []SeatVerb{{Seat: "node-service-manager", Verb: "restart"},
{Seat: ControllerSeat, Verb: "plans"}}
{Seat: ControllerSeat, Verb: "plans"}, {Seat: ControllerSeat, Verb: "push"}, {Seat: ControllerSeat, Verb: "upgrade"}}
// VerbsTheControllerAsksTheOperatorChannel are the operator channel's verbs the controller calls (novox/hq
// ADR 0380): `tell`, one plain message with no condition behind it — a module's own secret given, named,
@@ -383,6 +389,16 @@ type DegradableGrant struct {
// works. Every other grant stays required — missing one of those is a controller that comes up and is
// refused on the first thing it tries, with no fallback that covers it.
var DegradableGrants = []DegradableGrant{
{
Subject: "mesh.seat." + ControllerSeat + ".tool.push",
Reason: "refused: the ask of a held move (Release) is answered, and the refusal told, while the operator's own " +
"push at the terminal still sends the machine (issue 1732; ADR 0259)",
},
{
Subject: "mesh.seat." + ControllerSeat + ".tool.upgrade",
Reason: "refused: the ask of the builds held after a failed release is answered, and the refusal told, while the " +
"operator's own `upgrade release-backlog` at the terminal still releases them (issue 1732; ADR 0259)",
},
{
Subject: "mesh.seat." + VerbsTheControllerAsksTheOperatorChannel[0].Seat + ".tool." + VerbsTheControllerAsksTheOperatorChannel[0].Verb,
Reason: "refused: the controller raises the secret-given condition as before (ADR 0380 §9)",
@@ -408,6 +424,11 @@ var DegradableGrants = []DegradableGrant{
Reason: "refused: the controller asserts this bucket through the API and writes none of its keys — the build seat's " +
"holders do (ADR 0406 §3, issue 974)",
},
{
Subject: "$KV." + PlansBuildingBucket + ".>",
Reason: "refused: the controller then says no plan builds, and no check is held while a delivery builds — a check run " +
"wrongly costs a delivery a place, a check held wrongly costs a pull request its turn (ADR 0483 clause 2, issue 1687)",
},
{
Subject: "mesh.seat." + ControllerSeat + ".event.sent",
Reason: "refused: the controller says a tier sent to a machine only for outsiders, and a send that could not " +
@@ -693,7 +714,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// And asking the operator (novox/hq ADR 0259): an ask and its cancel under its own name, its warrants
// heard under its own name, the record of its asks read under its own name — as any user of the seat,
// derived the same way, from the seat its holder declares.
tp, ts := SeatTrafficOf(ControllerSeat, nil, p.Uses, nil).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(ControllerSeat, nil, p.Uses, nil))
pub = append(pub, tp...)
sub = append(sub, ts...)
// And asks who answers (novox/hq to-be 45 §4, D3): the self-check finds every seat's holder by
@@ -736,6 +757,15 @@ func PermissionsFor(p Principal) (Permissions, error) {
}
pub = append(pub, "$KV."+bucket+".>")
}
// **And the object bucket a declaration's large files travel in** (novox/hq ADR 0486 §2, issue 1697): the
// controller alone writes it, as a publish to the bucket's own subjects, which `$JS.API.>` does not cover.
for _, node := range p.FilesFor {
writes, err := DeclarationFilesWrites(node)
if err != nil {
return Permissions{}, err
}
pub = append(pub, writes)
}
// **And what the bus says about itself, read-only** (novox/hq to-be 45 §3, S9): a durable
// consumer that gave up on a message, or one that was deleted. The server already publishes
// both in the mesh's own account; the controller says each as a condition in the mesh's words.
@@ -835,6 +865,13 @@ func PermissionsFor(p Principal) (Permissions, error) {
return Permissions{}, err
}
pub = append(pub, haveKeys...)
// And an object of the declaration bucket by its name, and nothing of the bucket's list or writing
// (novox/hq ADR 0486 §2, issue 1697).
fileReads, err := DeclarationFilesReads(p.Node)
if err != nil {
return Permissions{}, err
}
pub = append(pub, fileReads...)
// And, while the vault serves it, the vault's move-in in its own name and nobody else's (issue 676): the
// values sealed to this machine, which only it opens, handed in sealed to the vault's key for the call.
if p.MovesIntoTheVault {
@@ -1010,7 +1047,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// 6. Its traffic on seats that name their caller or their kind, ask proofs or keep records
// (novox/hq ADR 0259 §3).
tp, ts := SeatTrafficOf(p.Module, p.Holds, p.Uses, p.Watches).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(p.Module, p.Holds, p.Uses, p.Watches))
pub = append(pub, tp...)
sub = append(sub, ts...)
@@ -1134,7 +1171,7 @@ func PermissionsFor(p Principal) (Permissions, error) {
// not publish under another's name or kind through it is the runtime's to keep, from the seat
// traffic each module's membership lists.
for _, d := range p.Carries {
tp, ts := SeatTrafficOf(d.Module, d.Holds, d.Uses, d.Watches).grants()
tp, ts := seatTrafficGrants(SeatTrafficOf(d.Module, d.Holds, d.Uses, d.Watches))
pub = append(pub, tp...)
sub = append(sub, ts...)
}
-33
View File
@@ -35,39 +35,6 @@ func OnNATS() (address string, on bool, err error) {
return address, true, nil
}
// CredentialIn reads the user and password out of a bus address, and the address without them.
//
// The controller's own credential arrives in its address, the way the old bus's does. Split out so the
// controller can record a hash of what it is actually using: its user is created by the installer at a
// bootstrap password, before the controller exists to mint one, and a composition that left itself out
// would produce a bus the writer cannot connect to.
func CredentialIn(address string) (user, password, bare string) {
at := strings.LastIndex(address, "@")
if at < 0 {
return "", "", address
}
scheme := ""
rest := address[:at]
if i := strings.Index(rest, "://"); i >= 0 {
scheme, rest = rest[:i+3], rest[i+3:]
}
user, password, _ = strings.Cut(rest, ":")
return user, password, scheme + address[at+1:]
}
// BareAddress is a bus address with any credential stripped, for something that only needs to know
// whether a server is answering there.
func BareAddress(address string) string {
_, _, bare := CredentialIn(address)
if bare == "" {
return address
}
if strings.Contains(bare, "://") {
return bare
}
return "nats://" + bare
}
// BusAddress is where the mesh's bus is, with this process's credential, and refuses to be empty:
// there is one bus, and a control plane without it can hold records and answer nothing (novox/hq
// ADR 0131, design 28 task 5.5).
-28
View File
@@ -1,28 +0,0 @@
package broker
import (
"testing"
)
// Which bus the mesh is on is one fact, and being told about both is refused.
//
// The controller's own credential arrives in its address, and has to be readable out of it — its user
// is created by the installer at a bootstrap password, before the controller exists to mint one.
func TestACredentialIsReadOutOfABusAddress(t *testing.T) {
for _, c := range []struct{ in, user, password, bare string }{
{"nats://controller:secret@127.0.0.1:4222", "controller", "secret", "nats://127.0.0.1:4222"},
{"controller:secret@127.0.0.1:4222", "controller", "secret", "127.0.0.1:4222"},
{"nats://127.0.0.1:4222", "", "", "nats://127.0.0.1:4222"},
{"127.0.0.1:4222", "", "", "127.0.0.1:4222"},
// A password containing an at-sign: split on the last one, or the address becomes part of the
// credential and the connection goes somewhere nobody named.
{"nats://controller:a@b@127.0.0.1:4222", "controller", "a@b", "nats://127.0.0.1:4222"},
} {
user, password, bare := CredentialIn(c.in)
if user != c.user || password != c.password || bare != c.bare {
t.Errorf("%q read as %q/%q at %q; wanted %q/%q at %q",
c.in, user, password, bare, c.user, c.password, c.bare)
}
}
}
+134
View File
@@ -0,0 +1,134 @@
//go:build integration
package broker
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"golang.org/x/crypto/bcrypt"
)
// **A holder of the build seat reads the plans building and the server refuses its write** (novox/hq ADR 0483 clause 2, issue
// 1687): on a real server whose users are composed as the controller composes them, the controller makes the bucket and puts a
// plan's key; the build agent's credential lists the keys and gets one, and its put of a key is refused with a permissions
// violation. It also reads the first waiting ask of a check queue by a direct get, which takes nothing from the queue.
func TestNatsTheBuildSeatsHoldersReadThePlansBuildingAndTheServerRefusesTheirWrite(t *testing.T) { // 1687
hash, err := bcrypt.GenerateFromPassword([]byte("pw"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
seat := Seat{Name: TheBuildSeat, Scope: "node", Accepts: []string{"build"}, Emits: []string{"built", "started"}}
users, err := Users(Records{Nodes: []string{"alpha"}, Assigned: map[string][]Declared{
"alpha": {{Module: RuntimeModule}, {Module: "build-agent", Holds: []Seat{seat}}},
}})
if err != nil {
t.Fatal(err)
}
for i := range users {
users[i].PasswordHash = string(hash)
}
accounts, err := ComposeAccounts(users)
if err != nil {
t.Fatal(err)
}
conf := filepath.Join(t.TempDir(), "bus.conf")
if err := os.WriteFile(conf, []byte("listen: 127.0.0.1:-1\njetstream { store_dir: "+
`"`+t.TempDir()+`"`+" }\n"+accounts), 0o600); err != nil {
t.Fatal(err)
}
opts, err := server.ProcessConfigFile(conf)
if err != nil {
t.Fatalf("the composed accounts do not parse: %v", err)
}
opts.NoLog, opts.NoSigs = true, true
s, err := server.NewServer(opts)
if err != nil {
t.Fatal(err)
}
go s.Start()
if !s.ReadyForConnections(10 * time.Second) {
t.Fatal("the bus did not come up")
}
t.Cleanup(s.Shutdown)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
connect := func(user string) (*nats.Conn, chan error) {
t.Helper()
refused := make(chan error, 8)
nc, err := nats.Connect(s.ClientURL(), nats.UserInfo(user, "pw"), nats.CustomInboxPrefix("_INBOX."+user),
nats.ErrorHandler(func(_ *nats.Conn, _ *nats.Subscription, err error) { refused <- err }))
if err != nil {
t.Fatalf("%s cannot connect: %v", user, err)
}
t.Cleanup(nc.Close)
return nc, refused
}
controller, _ := connect(ControllerName)
cj := OnConn(controller)
if err := AssertMeshStreams(cj); err != nil {
t.Fatalf("the controller cannot make the streams: %v", err)
}
if err := cj.EnsureControllerBuckets(); err != nil {
t.Fatalf("the controller cannot make its buckets: %v", err)
}
cjs, _ := jetstream.New(controller)
ckv, err := cjs.KeyValue(ctx, PlansBuildingBucket)
if err != nil {
t.Fatal(err)
}
if _, err := ckv.Put(ctx, "plan-1", []byte(`{"plan":"plan-1"}`)); err != nil {
t.Fatalf("the controller cannot put a plan's key: %v", err)
}
holder, refused := connect("alpha.build-agent")
hjs, err := jetstream.New(holder)
if err != nil {
t.Fatal(err)
}
kv, err := hjs.KeyValue(ctx, PlansBuildingBucket)
if err != nil {
t.Fatalf("a holder of the build seat cannot bind %s: %v", PlansBuildingBucket, err)
}
keys, err := kv.Keys(ctx)
if err != nil || len(keys) != 1 || keys[0] != "plan-1" {
t.Fatalf("the holder lists %v (%v), want plan-1", keys, err)
}
if e, err := kv.Get(ctx, "plan-1"); err != nil || e == nil {
t.Fatalf("the holder cannot get the key: %v", err)
}
// Its write is refused by the server.
if err := holder.Publish("$KV."+PlansBuildingBucket+".plan-2", []byte(`{}`)); err != nil {
t.Fatal(err)
}
_ = holder.Flush()
select {
case err := <-refused:
if !errors.Is(err, nats.ErrPermissionViolation) {
t.Errorf("the server said %v, want a permissions violation", err)
}
case <-time.After(3 * time.Second):
t.Error("the server did not refuse the holder's put of a plan's key")
}
if _, err := ckv.Get(ctx, "plan-2"); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Errorf("the refused put left a key: %v", err)
}
// And the direct get of a check queue's first waiting ask, which is granted.
reply, err := holder.RequestWithContext(ctx, "$JS.API.DIRECT.GET."+ChecksOtherStream, []byte(`{"seq":1,"next_by_subj":"`+
CheckSubjectPrefix(TheBuildSeat, false)+`>"}`))
if err != nil {
t.Fatalf("the holder's direct get of %s was not answered: %v", ChecksOtherStream, err)
}
if reply.Header.Get("Status") != "404" {
t.Errorf("an empty queue answered %q, want 404", reply.Header.Get("Status"))
}
}
@@ -0,0 +1,30 @@
package broker
import (
"slices"
"testing"
)
// Issue 1687 (ADR 0483 clause 2): a module whose manifest reads mesh-controller.plans-building may name it in `reads`, and
// reads the bucket and writes none; the controller writes it, so it is no mirror written by another.
func TestAModuleReadingPlansBuildingReadsAndWritesNone(t *testing.T) {
if !slices.Contains(ControllerMirrors(), "plans-building") || BucketName(ControllerSeat, PlansBuilding) != PlansBuildingBucket {
t.Fatalf("a module cannot name %s in `reads`: %v", PlansBuildingBucket, ControllerMirrors())
}
if _, other := MirrorsWrittenByAnother[PlansBuilding]; other {
t.Error("the controller writes the plans building itself")
}
perms, err := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "mesh-delivery-checker", PasswordHash: "x",
Reads: []string{"mesh-controller.plans-building"}})
if err != nil {
t.Fatal(err)
}
stream := "KV_" + PlansBuildingBucket
has(t, perms.Publish, "$JS.API.STREAM.INFO."+stream)
has(t, perms.Publish, "$JS.API.DIRECT.GET."+stream+".>")
for _, p := range perms.Publish {
if SubjectsOverlap(p, "$KV."+PlansBuildingBucket+".plan-1") {
t.Errorf("the reading module may write %s through %q", PlansBuildingBucket, p)
}
}
}
+38
View File
@@ -0,0 +1,38 @@
package broker
import (
"git.novox.be/novox/mesh-sdk/go/membership"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The membership the mesh issues an assignment (novox/hq ADR 0160), its subject, its stream, the bare address and the
// pin are mesh-sdk's, public so that a module's program outside this Go module, the route proxy, reads them too
// (novox/hq ADR 0480 clause 4, issue 1633). This package keeps no second copy: what follows only names them where the
// controller already did.
type (
Membership = membership.Membership
Served = membership.Served
SeatServed = membership.SeatServed
StateIssued = membership.StateIssued
LogIssued = membership.LogIssued
Worker = membership.Worker
SeatTraffic = membership.SeatTraffic
KindHeld = membership.KindHeld
StreamIssued = membership.StreamIssued
HeardIssued = membership.HeardIssued
WaitIssued = membership.WaitIssued
)
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = membership.AssignmentsStream
var (
// MembershipSubject is where an assignment's own membership is published.
MembershipSubject = membership.Subject
// CredentialIn reads the user and password out of a bus address, and the address without them.
CredentialIn = membership.CredentialIn
// BareAddress is a bus address with any credential stripped.
BareAddress = membership.BareAddress
// PinnedToFingerprint accepts exactly the certificate with this SHA-256 and no other.
PinnedToFingerprint = process.PinnedToFingerprint
)
+2 -40
View File
@@ -24,44 +24,6 @@ import "sort"
// And one read: **a holder's records**, a bucket the seat names, read by each user under its own name only
// (`$KV.<bucket>.<module>.>`), so an asker reads the state of its own asks and no other asker's.
// Worker is one durable consumer a holder pulls a seat's work from.
type Worker struct {
Stream string
Consumer string
Filter string
}
// SeatTraffic is one module's seat traffic beyond tools. Publish and Subscribe are subject patterns, in the
// server's wildcards; the runtime that carries the module checks a bundle's request against them, since
// the runtime's own principal holds the union of every module it carries.
type SeatTraffic struct {
// Publish is what it submits (accepts of seats it uses), says (events of seats it holds) and asks
// (proofs of seats it holds a kind of).
Publish []string `json:"publish,omitempty"`
// Subscribe is what it hears: events of seats it uses that are named by caller, events of seats it
// watches, and accepts of seats it holds.
Subscribe []string `json:"subscribe,omitempty"`
// Answers is the proof subjects it answers, as a watcher of a kinded seat.
Answers []string `json:"answers,omitempty"`
// Workers are the work queues it takes from, as a holder.
Workers []Worker `json:"workers,omitempty"`
// Records is the direct-get subjects of the records it reads under its own name.
Records []string `json:"records,omitempty"`
// Kinds are the holders of every kinded bench it uses or watches, with what each promises: the one
// account of which channel is which, and what it can carry, that the router judges an answer by. The
// controller's, from the claims, never a channel's word (novox/hq ADR 0259 §5).
Kinds []KindHeld `json:"kinds,omitempty"`
}
// KindHeld is one kind of a kinded bench and who holds it.
type KindHeld struct {
Seat string `json:"seat"`
Kind string `json:"kind"`
Module string `json:"module"`
Node string `json:"node"`
Capabilities []string `json:"capabilities,omitempty"`
}
// KindedBenches are the seats that may be kinded (ADR 0234 §2): making another is a decision, recorded.
var KindedBenches = map[string]bool{"channel": true, "intake": true}
@@ -175,9 +137,9 @@ func SeatTrafficOf(module string, holds, uses, watches []Seat) SeatTraffic {
return t
}
// grants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a
// seatTrafficGrants is the bus permissions seat traffic needs: the subjects themselves, and the JetStream API a
// worker is pulled and acknowledged through and a record is read through.
func (t SeatTraffic) grants() (pub, sub []string) {
func seatTrafficGrants(t SeatTraffic) (pub, sub []string) {
pub = append(pub, t.Publish...)
sub = append(sub, t.Subscribe...)
sub = append(sub, t.Answers...)
-10
View File
@@ -186,16 +186,6 @@ func perMachineNames(buckets []Bucket) []string {
return out
}
// StateIssued is one bucket an assignment may reach, by the name its module uses for it: its own
// state by the local name, another's as `<module>.<name>` (novox/hq ADR 0201). Key, when set, is the
// one key it may reach (novox/hq ADR 0260): this machine's.
type StateIssued struct {
Name string `json:"name"`
Bucket string `json:"bucket"`
Writes bool `json:"writes,omitempty"`
Key string `json:"key,omitempty"`
}
// stateIssuedFor is every bucket a module's code may reach on a machine, as its membership lists them.
func stateIssuedFor(d Declared, node string) []StateIssued {
var out []StateIssued
-3
View File
@@ -80,9 +80,6 @@ const RestrictedMetadata = "mesh-restricted"
// it under: never set by a mesh-internal work-queue stream (a seat's, a check's), only by Stream.WorkQueue.
const WorkQueueMetadata = "mesh-work-queue"
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = "ASSIGNMENTS"
// What a durable consumer gave up on is kept (novox/hq issue 330, design 25 §3).
//
// **The server says it and keeps it; the controller copies it.** A consumer that handed a message over
+2 -2
View File
@@ -29,7 +29,7 @@ accounts {
]
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller-merges.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_builds.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_checks-running.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_delivery-merges.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$KV.mesh-controller_merge-window.>", "$KV.mesh-controller_plans-building.>", "$KV.mesh-controller_predefined.>", "$KV.mesh-controller_rollout-silent.>", "$KV.mesh-controller_rollouts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.predefined-changed", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.event.sent", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-controller.tool.push", "mesh.seat.mesh-controller.tool.upgrade", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.check.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*", "mesh.seat.operator-channel.tool.tell"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
@@ -38,7 +38,7 @@ accounts {
subscribe: { allow: ["_INBOX.enrol.one.>"] }
} }
{ user: "node.one", password: "$2a$11$nnnnnnnnnnnnnnnnnnnnnn", permissions: {
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.CREATE.KV_node-engine_have.*.$KV.node-engine_have.one.>", "$JS.API.CONSUMER.DELETE.KV_node-engine_have.>", "$JS.API.CONSUMER.INFO.NODES.one", "$JS.API.DIRECT.GET.KV_node-engine_have.$KV.node-engine_have.one.>", "$JS.API.STREAM.INFO.KV_node-engine_have", "$JS.FC.KV_node-engine_have.>", "$KV.node-engine_have.one.>", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
publish: { allow: ["$JS.ACK.NODES.one.>", "$JS.API.CONSUMER.CREATE.KV_node-engine_have.*.$KV.node-engine_have.one.>", "$JS.API.CONSUMER.CREATE.OBJ_declaration-files-one.*.$O.declaration-files-one.C.*", "$JS.API.CONSUMER.DELETE.KV_node-engine_have.>", "$JS.API.CONSUMER.DELETE.OBJ_declaration-files-one.*", "$JS.API.CONSUMER.INFO.NODES.one", "$JS.API.DIRECT.GET.KV_node-engine_have.$KV.node-engine_have.one.>", "$JS.API.DIRECT.GET.OBJ_declaration-files-one.$O.declaration-files-one.M.*", "$JS.API.STREAM.INFO.KV_node-engine_have", "$JS.FC.KV_node-engine_have.>", "$JS.FC.OBJ_declaration-files-one.>", "$KV.node-engine_have.one.>", "$SRV.PING.node-tools.one", "mesh.control.one.>"] }
subscribe: { allow: ["_DELIVER.one", "_DELIVER.one.>", "_INBOX.node.one.>", "mesh.node.one.ask.hand-over", "mesh.node.one.ask.judge-directories", "mesh.node.one.ask.move-secret", "mesh.node.one.ask.report", "mesh.node.one.ask.setuid-search", "mesh.node.one.declare"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+1 -1
View File
@@ -135,7 +135,7 @@ type Records struct {
// correct.
func Users(r Records) ([]Principal, error) {
out := []Principal{{Kind: KindController, Uses: asksSeatOf(r), AsksIssued: issuedOf(r), AsksTheVault: vaultVerbsOf(r),
AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig)}}
AsksTheConfig: servedOf(r, VerbsTheControllerAsksTheConfig), FilesFor: sortedCopy(r.Nodes)}}
for _, node := range sortedCopy(r.Nodes) {
witness := false
-6
View File
@@ -31,12 +31,6 @@ type Wait struct {
Last string
}
// WaitIssued is one family as a membership issues it: the name a session asks for, and its subject.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// WaitSubject is where a family's events land. Each token is checked before it becomes a subject: a
// wildcard, a dot inside a name or an empty token would widen the grant.
func WaitSubject(w Wait) (string, error) {
+11
View File
@@ -177,6 +177,10 @@ var WritersTable = []WriterRow{
KeptIn: "key-value " + CheckRunningBucket + ", aged out when not renewed", Others: "the build seat's holders read it to drop a duplicate request",
Subjects: kvOf(CheckRunningBucket), Writes: holdsTheBuildSeat,
Shared: "every machine holding the build seat puts the key of the check it runs and deletes it when the check ends"},
{State: "the plans-building mirror", Writer: "controller (lease holder), at the first build's ask and at each refresh",
KeptIn: "key-value " + PlansBuildingBucket + ", aged out when not refreshed",
Others: "the build seat's holders read it and take no new check while a key is there (ADR 0483)",
Subjects: kvOf(PlansBuildingBucket), Writes: isController},
{State: "a merge announced", Writer: "one announcer per forge (the hook, or the poll when the hook is absent — never both)",
KeptIn: "the bus", Others: "—", Subjects: []string{"mesh.mod.*.event.pull.merged"}, Writes: ownModule},
// A pull request's head, before it merges (novox/hq to-be 45 §9): the same one announcer.
@@ -219,6 +223,13 @@ var WritersTable = []WriterRow{
{State: "a machine's have", Writer: "the node-engine of the machine, under its own keys `<node>.>`",
KeptIn: "key-value " + delivery.HaveBucket + ", twenty revisions of each key",
Others: "read through `reads` as node-engine.have", Subjects: kvOf(delivery.HaveBucket), Writes: ownNodeKeys},
// The contents of a declaration's files over 64 KiB (novox/hq ADR 0486 §2): the controller alone writes the bucket, each
// object under the digest of its content; a node-engine reads one by its name, with no list and no write.
{State: "a declaration's large files", Writer: "controller, each object under the sha256 digest of its content",
KeptIn: "the object bucket " + DeclarationFilesBucket + "-<node> of each node, an object removed a day after the " +
"node's latest declaration no longer names it",
Others: "a node-engine reads an object of its own bucket by its name and may neither list the bucket nor write it",
Subjects: []string{"$O.*.>"}, Writes: isController},
{State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig,
Subjects: kvOf(DeclaredConfigBucket), Writes: isController},
+4
View File
@@ -29,6 +29,8 @@ var designRows = []string{
"builds and their outcomes",
"a pull request's check asked",
"a pull request's check running",
// The plans in their build phase, a bucket the build seat's holders read (ADR 0483 clause 2, issue 1687).
"the plans-building mirror",
"a merge announced",
"a pull request's head announced",
"a pull request's merge check",
@@ -43,6 +45,8 @@ var designRows = []string{
"the rollouts of the chain",
// What each machine has of each module, written by its node-engine under its own keys (ADR 0429 §1, issue 1136).
"a machine's have",
// The contents of a declaration's large files, in the bus's object bucket (ADR 0486 §2, issue 1697).
"a declaration's large files",
"the declared-config mirror",
"the predefined-keys mirror",
// The provisioner client's word on what it placed (novox/hq ADR 0349 §4, design 57 P6).
+6 -3
View File
@@ -86,11 +86,14 @@ func TestAnImageCompilingGoIsHandedItsBuildSourceAndSaysIt(t *testing.T) {
}
for file, want := range map[string]bool{"proxy/main.go": true, "internal/shared/s.go": true, "go.mod": true,
"cmd/ctl/main.go": false, "internal/only/o.go": false, "README.md": false} {
_, err := os.Stat(filepath.Join(workspace, "narrow-server", filepath.FromSlash(file)))
if (err == nil) != want {
t.Errorf("%s handed to the recipe: %v, wanted %v", file, err == nil, want)
held := slices.Contains(r.builtFrom, file)
if held != want {
t.Errorf("%s handed to the recipe: %v, wanted %v", file, held, want)
}
}
if _, err := os.Stat(filepath.Join(workspace, "narrow-server")); err == nil {
t.Error("the narrowed build source was left in the workspace")
}
// Said per repository: its own, the manifest and the recipe; the context's, the closure.
if len(got.Sources) != 2 {
+55 -1
View File
@@ -219,6 +219,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
if err := os.MkdirAll(workspace, 0o755); err != nil {
return Result{}, err
}
// **A node with no room takes no build** (novox/hq issue 1834).
if err := RoomToWork("/", workspace); err != nil {
say("clone", "refused: %v", err)
return Result{}, err
}
// The credential is a file git reads, never an argument: a URL carrying a password in argv
// would be readable by anything that can list processes for as long as a clone runs.
credentials, forget, err := storeCredential(workspace, forge)
@@ -738,11 +743,41 @@ func logging(log Log) func(step, format string, args ...any) {
}
}
// keepsImageLocally says the publisher leaves the image in the machine's runtime as the artifact itself.
func keepsImageLocally(publish Publisher) bool {
switch publish.(type) {
case Local, *Local:
return true
}
return false
}
// PushedNames is implemented by a publisher that tags the image under other names to push it: the names it leaves in the
// machine's runtime for a repository, which go with the build's own tag (novox/hq issue 1834).
type PushedNames interface {
PushedNames(repository string) []string
}
// releaseImage removes the names an image was given on the node, in one command: its layers go only with the last. It runs
// with a context of its own, the build's may be cancelled. A name never made (the build failed before it) is no fault;
// one that cannot be removed is said, and removal is said only when the image is gone.
func releaseImage(ctx context.Context, run Runner, names []string, say func(step, format string, args ...any)) {
removing, done := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
defer done()
if _, err := run(removing, "", "docker", append([]string{"rmi"}, names...)...); err != nil {
if !strings.Contains(err.Error(), "No such image") {
say("image", "the local image %s could not be removed, and goes with the week's sweep: %v", strings.Join(names, ", "), err)
}
return
}
say("image", "the local image %s is removed: the registry holds it", strings.Join(names, ", "))
}
// contextFrom clones an image artifact's own build context, when it names one apart from this
// module's own repository — a fresh tree, the same way the module's own is, keyed by artifact
// name so two artifacts of one module naming different contexts do not collide.
func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentials string,
from catalogue.ArtifactContext, seats map[string]string, say func(step, format string, args ...any)) (string, error) {
from catalogue.ArtifactContext, seats map[string]string, say func(step, format string, args ...any)) (_ string, err error) {
url, err := contextURL(from, seats)
if err != nil {
return "", err
@@ -752,6 +787,12 @@ func contextFrom(ctx context.Context, run Runner, workspace, artifact, credentia
if err := os.RemoveAll(dir); err != nil {
return "", err
}
// A clone that failed half-way is no context to leave behind (novox/hq issue 1834).
defer func() {
if err != nil {
os.RemoveAll(dir)
}
}()
if _, err := run(ctx, workspace, "git", cloneWith(credentials, "clone", "--quiet", url, dir)...); err != nil {
return "", fmt.Errorf("cannot clone %s: %w", url, err)
}
@@ -1143,6 +1184,8 @@ func one(ctx context.Context, run Runner, publish Publisher,
src.contexts[a.Name] = t
}
buildDir = cloned
// Let go of when the image is built, whatever became of it (novox/hq issue 1834).
defer os.RemoveAll(cloned)
if t, _ := trunkOf(ctx, run, cloned, "HEAD"); t == "" || !atTrunkHead(ctx, run, cloned, "HEAD", t) {
src.contextOffHead()
}
@@ -1180,6 +1223,7 @@ func one(ctx context.Context, run Runner, publish Publisher,
src.ownHas(paths...)
}
buildDir = narrowed
defer os.RemoveAll(narrowed)
} else if a.Context != nil {
src.readWhole(*a.Context)
}
@@ -1205,6 +1249,16 @@ func one(ctx context.Context, run Runner, publish Publisher,
}
invocation = append(invocation, ".")
say("image", "docker build -f %s", recipePath)
// **The local tag goes with the build** (novox/hq issue 1834): an image published to a registry is held there, and
// a tag per module and commit left on the node was the 96 GB of a full disk. Only a node publishing nowhere (a
// mesh being raised) keeps it, since the local image is then the artifact.
if !keepsImageLocally(publish) {
names := []string{local}
if p, ok := publish.(PushedNames); ok {
names = append(names, p.PushedNames(module+"/"+a.Name)...)
}
defer releaseImage(ctx, run, names, say)
}
if _, err := run(ctx, buildDir, "docker", invocation...); err != nil {
return catalogue.Built{}, fmt.Errorf("%s: building %s failed: %w", module, a.Name, err)
}
+13
View File
@@ -38,6 +38,9 @@ type recorded struct {
stamped time.Time
// tree is what git names as the tree of the module's directory; empty answers as for a commit.
tree string
// builtFrom is the files (slash-separated, relative) of the folder the last `docker build` ran in, read as it ran:
// the build removes that folder when it ends (novox/hq issue 1834).
builtFrom []string
}
func (r *recorded) run(_ context.Context, dir, name string, args ...string) (string, error) {
@@ -53,6 +56,16 @@ func (r *recorded) run(_ context.Context, dir, name string, args ...string) (str
break
}
}
if name == "docker" && len(args) > 0 && args[0] == "build" {
r.builtFrom = nil
_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
if err == nil && !d.IsDir() {
rel, _ := filepath.Rel(dir, p)
r.builtFrom = append(r.builtFrom, filepath.ToSlash(rel))
}
return nil
})
}
switch {
case name == "git" && isClone:
repository := args[len(args)-2]
+100 -5
View File
@@ -123,8 +123,41 @@ type CheckSpec struct {
// controller could not learn them, and the verdict says so.
Siblings []Sibling
SiblingsUnknown string
// Held are the approved held values of the check's pull requests (novox/hq ADR 0502 clause 4): written to a file and
// handed to the gate, which lays them over the change's composition only. None: no flag, no file.
Held []HeldValue
}
// HeldValue is one approved config value held for a key an open pull request introduces: the value of Key for Module on
// Node (a machine's real name; the gate finds it in the snapshot by its pseudonym). The gate reads this document.
type HeldValue struct {
Pull string `json:"pull"`
Head string `json:"head,omitempty"`
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value"`
Approved string `json:"approved,omitempty"`
By string `json:"by,omitempty"`
At string `json:"at,omitempty"`
}
// writeHeld writes the held values to held.json under root and answers its path; empty, and nothing written, for none.
func writeHeld(root string, held []HeldValue) (string, error) {
if len(held) == 0 {
return "", nil
}
raw, err := json.Marshal(held)
if err != nil {
return "", err
}
file := filepath.Join(root, "held.json")
return file, os.WriteFile(file, raw, 0o644)
}
// heldFlag is how the gate is given the held values: only when there are some, so a judge from before --held still runs.
const heldFlag = `${MESH_CHECK_HELD:+--held "$MESH_CHECK_HELD"}`
// Sibling is one other open pull request of a delivery group (novox/hq ADR 0382): the repository it is in as
// owner/repo, its number, its whole head, the branch it merges into, and Dir, the directory the check finds that
// repository under beside it (an entry of Beside).
@@ -272,6 +305,8 @@ const (
// EnvGroup is a delivery group's other heads, as JSON, for the gate (novox/hq ADR 0239); empty for a
// pull request checked alone.
EnvGroup = "MESH_CHECK_GROUP"
// EnvHeld is the file of the approved held values of the check's pull requests (novox/hq ADR 0502); empty for none.
EnvHeld = "MESH_CHECK_HELD"
// EnvTests is a `go test -run` pattern and EnvStoreOnly is 1 when a branch tried before its pull request
// asks for only those tests, or only the store tests (novox/hq issue 686). Never set for a pull request's
// check. A script that reads neither runs whole.
@@ -357,6 +392,12 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
ctx, stop := context.WithTimeout(ctx, CheckTimeout)
defer stop()
// **A node with no room takes no check** (novox/hq issue 1834): said as the check's verdict, never a throwaway store
// that dies at once on a full disk.
if err := RoomToWork("/", workspace); err != nil {
say("check", "refused: %v", err)
return CheckVerdict{}, err
}
root := filepath.Join(workspace, "check")
if err := os.RemoveAll(root); err != nil {
return CheckVerdict{}, err
@@ -572,6 +613,15 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
}
// The approved held values of the check's pull requests, for the gate (novox/hq ADR 0502 clause 4).
heldFile, err := writeHeld(root, spec.Held)
if err != nil {
return CheckVerdict{}, err
}
if heldFile != "" {
say("check", "with %d config value(s) held for its pull request(s)", len(spec.Held))
}
// Every clone is made: the credential is gone before anything of the check runs (novox/hq issue 462).
forget()
@@ -733,7 +783,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
EnvTestStore + "=" + storeURL, EnvTestBus + "=nats://" + bus, EnvRepository + "=" + spec.Owner + "/" + spec.Repo,
EnvBeside + "=" + root,
EnvModules + "=" + strings.Join(append(append([]string{}, spec.Modules...), spec.New...), ","),
EnvGroup + "=" + groupFile,
EnvGroup + "=" + groupFile, EnvHeld + "=" + heldFile,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")}
env = append(env, changedEnv...)
env = append(env, selection(spec)...)
@@ -1100,7 +1150,7 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
if err := running(inToolchain(tree, append(env, EnvVerdict+"="+verdictFile), "sh", "-c",
`"$MESH_GATE" merge-gate --facts "$MESH_FACTS" --store "$MESH_GATE_POSTGRES" `+
`--repository "$MESH_CHECK_REPOSITORY" --tree . `+gateChangedFlag(env)+` `+
`${MESH_CHECK_GROUP:+--group "$MESH_CHECK_GROUP"} --json > "$MESH_CHECK_VERDICT"`)); err != nil {
`${MESH_CHECK_GROUP:+--group "$MESH_CHECK_GROUP"} `+heldFlag+` --json > "$MESH_CHECK_VERDICT"`)); err != nil {
if ctx.Err() != nil {
return "error", "the check was ended during the merge gate"
}
@@ -1587,7 +1637,9 @@ func BusImage(version string) string {
// throwaway starts a container publishing one port on loopback, and answers where it is reached.
func throwaway(ctx context.Context, run, plain Runner, name, image string, port int, opts, args []string) (string, error) {
invocation := []string{"run", "-d", "--rm", "--name", name, "-p", fmt.Sprintf("127.0.0.1::%d", port)}
// Never with --rm (novox/hq issue 1834): a store that dies at once took its log with it. The check removes what it
// raised, with its volumes, after it has read why one stopped (RemoveContainersOf).
invocation := []string{"run", "-d", "--name", name, "-p", fmt.Sprintf("127.0.0.1::%d", port)}
invocation = append(invocation, opts...)
invocation = append(invocation, image)
invocation = append(invocation, args...)
@@ -1608,17 +1660,45 @@ func throwaway(ctx context.Context, run, plain Runner, name, image string, port
// waitFor runs a readiness command in a container until it answers, for a minute.
func waitFor(ctx context.Context, run Runner, name string, ready []string) error {
for i := 0; i < 60; i++ {
return waitReady(ctx, run, name, ready, 60)
}
// waitReady is waitFor for a number of tries, a second apart. A container that never answers is not left a mystery
// (novox/hq issue 1834): its state and exit code and the last 20 lines of its log are in the error, read before the
// check removes it.
func waitReady(ctx context.Context, run Runner, name string, ready []string, attempts int) error {
for i := 0; i < attempts; i++ {
if _, err := run(ctx, "", "docker", append([]string{"exec", name}, ready...)...); err == nil {
return nil
}
if i == attempts-1 {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Second):
}
}
return fmt.Errorf("%s never became ready", name)
return fmt.Errorf("%s never became ready%s", name, whyItStopped(ctx, run, name))
}
// whyItStopped is ": <state>, exit code N; its last log lines: …" for a container, or "" when docker says nothing.
func whyItStopped(ctx context.Context, run Runner, name string) string {
reading, done := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer done()
var said []string
if out, err := run(reading, "", "docker", "inspect", "--format",
"{{.State.Status}}, exit code {{.State.ExitCode}}", name); err == nil && strings.TrimSpace(out) != "" {
said = append(said, strings.TrimSpace(out))
}
if out, err := run(reading, "", "docker", "logs", "--tail", "20", name); err == nil && strings.TrimSpace(out) != "" {
said = append(said, "its last log lines: "+strings.TrimSpace(out))
}
if len(said) == 0 {
return ""
}
return ": " + strings.Join(said, "; ")
}
// dialable waits for an address to take a connection, for a minute.
@@ -1773,9 +1853,24 @@ func whatFailed(s string) string {
}
}
}
// The controller's module check refuses with a count line and one line per problem, then prints its
// warning counts: the refusal is what failed, not the count printed last (novox/hq issue 1310).
for i, line := range lines {
if line = strings.TrimSpace(line); problemsLine.MatchString(line) {
if i+1 < len(lines) {
if first := strings.TrimSpace(lines[i+1]); first != "" {
return line + ": " + first
}
}
return line
}
}
return lastLine(s)
}
// problemsLine is the module check's refusal: "mesh-controller: 3 problem(s) in 139 manifest(s)".
var problemsLine = regexp.MustCompile(`^\S+: [1-9]\d* problem\(s\) in \d+ manifest\(s\)$`)
func lastLine(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
return strings.TrimSpace(lines[len(lines)-1])
+33
View File
@@ -0,0 +1,33 @@
package builder
import (
"os"
"path/filepath"
"strings"
"testing"
)
// 1826/AC2: the held values are written whole to held.json, and the gate is given the flag only when there are some.
func TestHeldValuesAreWrittenForTheGateAndNoneWriteNothing(t *testing.T) {
root := t.TempDir()
if file, err := writeHeld(root, nil); err != nil || file != "" {
t.Fatalf("none wrote %q %v", file, err)
}
if _, err := os.Stat(filepath.Join(root, "held.json")); err == nil {
t.Fatal("none wrote a file")
}
file, err := writeHeld(root, []HeldValue{{Pull: "novox/mesh-catalog#434", Node: "ace", Module: "n8n", Key: "media-gid",
Value: []byte("2000"), Approved: "the operator"}})
if err != nil || file != filepath.Join(root, "held.json") {
t.Fatalf("wrote %q %v", file, err)
}
raw, _ := os.ReadFile(file)
for _, want := range []string{`"node":"ace"`, `"module":"n8n"`, `"key":"media-gid"`, `"value":2000`} {
if !strings.Contains(string(raw), want) {
t.Fatalf("%s lacks %s", raw, want)
}
}
if !strings.Contains(heldFlag, `--held "$MESH_CHECK_HELD"`) || !strings.HasPrefix(heldFlag, "${MESH_CHECK_HELD:+") {
t.Fatalf("the flag is %s", heldFlag)
}
}
+26
View File
@@ -199,3 +199,29 @@ func TestOnlyATrysSelectionReachesTheScript(t *testing.T) {
t.Fatalf("a try's selection: %v", env)
}
}
// **Issue 1310**: the controller's module check prints its warning counts after its refusal; a failed check
// is said by the refusal, never by the count printed last.
func TestAFailedModuleCheckIsSaidByItsRefusalNotItsWarningCount(t *testing.T) {
log := "mesh-controller: 3 problem(s) in 139 manifest(s)\n" +
"mesh-files claims node-files and says it serves upload, download, which that seat's protocol does not promise\n" +
"minio claims the seat \"object-store\", which no module declares and the mesh does not define\n" +
"only-office claims the seat \"document-converter\", which no module declares and the mesh does not define\n" +
"adwaita: ok, 4 tool(s)\nlong-running resources without health: 59\n" +
"files asking for a setting without saying whether it is trusted: 36\n"
// 1310/AC1
want := "mesh-controller: 3 problem(s) in 139 manifest(s): mesh-files claims node-files and says it serves upload, " +
"download, which that seat's protocol does not promise"
if got := whatFailed(log); got != want {
t.Errorf("1310/AC1: the captured log is said as %q, not %q", got, want)
}
// 1310/AC2
if got, want := whatFailed("ok \tx/a\t1s\n--- FAIL: TestX (0.01s)\n x_test.go:12: got 1\nFAIL\nlong-running resources without health: 59\n"),
"--- FAIL: TestX (0.01s)"; got != want {
t.Errorf("1310/AC2: a failing test is said as %q, not %q", got, want)
}
// 1310/AC3
if got, want := whatFailed("starting\nsomething odd happened\n"), "something odd happened"; got != want {
t.Errorf("1310/AC3: a log with no failure line is said as %q, not %q", got, want)
}
}
+216
View File
@@ -0,0 +1,216 @@
package builder
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
// Issue 1834: a build node's root disk filled and every check there ended "the store never became ready". A node
// frees what each build and check leaves, refuses to take work under a floor of free space, and a store that does
// not come up says why.
func init() {
// The disk the tests run on is none of their business; the tests of the floor say their own.
FreeBytes = func(string) (uint64, error) { return 500 << 30, nil }
}
func withFree(t *testing.T, bytes uint64) {
t.Helper()
was := FreeBytes
FreeBytes = func(string) (uint64, error) { return bytes, nil }
t.Cleanup(func() { FreeBytes = was })
}
// 1834/AC2: given 3 GiB free, when a check or a build is asked, then it refuses at once, saying so, running nothing.
func TestACheckUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
withFree(t, 3<<30)
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
_, err := Check(context.Background(), run, CheckSpec{ID: "build-1"}, t.TempDir(), "registry.invalid", GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "the build node has 3 GiB free on /, under the floor of 20 GiB") {
t.Fatalf("a check on a full disk said %v", err)
}
if len(ran) != 0 {
t.Errorf("a refused check ran %v", ran)
}
}
func TestABuildUnderTheFloorOfFreeSpaceRefusesSayingSo(t *testing.T) {
withFree(t, 3<<30)
r := &recorded{}
_, err := Build(context.Background(), r.run, r, "https://forge.invalid/x.git", "", "", t.TempDir(), nil, Npmrc{}, GitCredential{}, nil)
if err == nil || !strings.Contains(err.Error(), "under the floor of 20 GiB") {
t.Fatalf("a build on a full disk said %v", err)
}
if len(r.ran) != 0 {
t.Errorf("a refused build ran %v", r.ran)
}
}
// The workspace's filesystem is read too, and a workspace not made yet is read at its nearest folder.
func TestTheWorkspacesFilesystemIsReadToo(t *testing.T) {
was := FreeBytes
t.Cleanup(func() { FreeBytes = was })
missing := filepath.Join(t.TempDir(), "not", "yet")
FreeBytes = func(p string) (uint64, error) {
if p == "/" {
return 100 << 30, nil
}
if _, err := os.Stat(p); err != nil {
t.Errorf("read the free space of %s, which is not there", p)
}
return 1 << 30, nil
}
err := RoomToWork("/", missing)
if err == nil || !strings.Contains(err.Error(), "1 GiB free on "+missing) {
t.Errorf("the workspace's disk, under the floor, said %v", err)
}
FreeBytes = func(string) (uint64, error) { return 0, errors.New("no") }
if RoomToWork("/") == nil {
t.Error("a disk that cannot be read was taken as having room")
}
}
// 1834/AC3: given a store that stops at once, when the check waits for it, then its exit code and last log lines are
// in the error and the container is removed afterwards, not by --rm before anyone could read it.
func TestAStoreThatDiesSaysWhyItDied(t *testing.T) {
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
line := name + " " + strings.Join(args, " ")
ran = append(ran, line)
switch {
case args[0] == "exec":
return "", errors.New("Error response from daemon: container is not running")
case args[0] == "inspect":
return "exited exit code 1\n", nil
case args[0] == "logs":
return "initdb: error: could not create directory: No space left on device\n", nil
}
return "", nil
}
err := waitReady(context.Background(), run, "build-1-store", []string{"pg_isready"}, 1)
for _, want := range []string{"build-1-store never became ready", "exit code 1", "No space left on device"} {
if err == nil || !strings.Contains(err.Error(), want) {
t.Errorf("the verdict %q lacks %q", err, want)
}
}
logs := ""
for _, l := range ran {
if strings.HasPrefix(l, "docker logs") {
logs = l
}
}
if logs != "docker logs --tail 20 build-1-store" {
t.Errorf("the log was read as %q", logs)
}
}
func TestTheThrowawayStoreIsNotStartedWithRm(t *testing.T) {
var started []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
if args[0] == "run" {
started = args
}
return "127.0.0.1:40000\n", nil
}
if _, err := throwaway(context.Background(), run, run, "build-1-store", "postgres:17-alpine", 5432, nil, nil); err != nil {
t.Fatal(err)
}
for _, a := range started {
if a == "--rm" {
t.Errorf("started with --rm, which erases why it died: %v", started)
}
}
}
// 1834/AC1: given an image built for a check or a walk and published to the registry, when the build ends, then the
// local image tag, the narrowed source and the context clone it left are gone; an image published nowhere (a mesh
// being raised) is kept, since it is the artifact.
// pushedNamed is a publisher that tags the image as <registry>/<repository> to push it, as the registry one does.
type pushedNamed struct{ *recorded }
func (pushedNamed) PushedNames(repository string) []string {
return []string{"registry.invalid/" + repository}
}
func TestABuildRemovesWhatItLeftOnTheNode(t *testing.T) {
r := &recorded{
contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"},
secondary: map[string]map[string]string{"https://forge.invalid/ctl.git": {"go.mod": "module x\n"}},
}
workspace := t.TempDir()
if _, err := Build(context.Background(), r.run, pushedNamed{r}, "https://forge.invalid/c.git", "modules/m", "", workspace, nil, Npmrc{},
GitCredential{}, nil); err != nil {
t.Fatal(err)
}
// Both names go in one rmi: the layers stay while either is left (the push tagged the image with the registry's).
if !strings.Contains(strings.Join(r.ran, "\n"), "docker rmi m-server:c0ffeec0 registry.invalid/m/server") {
t.Errorf("the image built for it and its pushed name were not removed: %v", r.ran)
}
left, _ := filepath.Glob(filepath.Join(workspace, "*"))
if len(left) != 0 {
t.Errorf("the build left %v in its workspace", left)
}
}
const anImageWithAContext = `{"module":"m","version":"1",
"build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile",
"context":{"repository":"https://forge.invalid/ctl.git","ref":"main"}}]}}`
func TestAnImageKeptLocallyIsNotRemoved(t *testing.T) {
r := &recorded{contents: map[string]string{"modules/m/" + ManifestName: anImageWithAContext, "modules/m/Dockerfile": "FROM scratch\n"}}
local := Local{Run: func(ctx context.Context, dir, name string, args ...string) (string, error) {
if args[0] == "image" {
return "sha256:" + strings.Repeat("a", 64) + "\n", nil
}
return r.run(ctx, dir, name, args...)
}}
if _, err := Build(context.Background(), local.Run, local, "https://forge.invalid/c.git", "modules/m", "", t.TempDir(), nil, Npmrc{},
GitCredential{}, nil); err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(r.ran, "\n"), "docker rmi") {
t.Errorf("the artifact of a mesh being raised was removed: %v", r.ran)
}
}
// The held image the comparison pulls to read is let go of once read.
func TestTheHeldImageAComparisonPullsIsRemovedAfterReading(t *testing.T) {
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
if args[0] == "create" {
return "abcdef0123456789\n", nil
}
if args[0] == "image" {
return "", errors.New("stop here")
}
return "", nil
}
_, _ = imageContent(context.Background(), run, t.TempDir(), "reg.invalid/m/server@sha256:"+strings.Repeat("a", 64))
if got := ran[len(ran)-1]; !strings.HasPrefix(got, "docker rmi reg.invalid/m/server@sha256:") {
t.Errorf("the last command was %q, not the removal of the pulled image: %v", got, ran)
}
}
func TestAnImageThatCannotBeRemovedIsNotSaidRemoved(t *testing.T) {
var said []string
run := func(context.Context, string, string, ...string) (string, error) {
return "", errors.New("image is being used")
}
releaseImage(context.Background(), run, []string{"a:1"}, func(step, format string, args ...any) {
said = append(said, format)
})
for _, s := range said {
if strings.Contains(s, "is removed") {
t.Errorf("said removed: %q", s)
}
}
}
+46
View File
@@ -0,0 +1,46 @@
package builder
import (
"fmt"
"os"
"path/filepath"
"syscall"
)
// FreeSpaceFloor is the free space a build node must have, on / and on its workspace's filesystem, before it takes
// a build or a check (novox/hq issue 1834): under it, a throwaway store cannot initialise its data directory and
// dies at once, and the check ended "never became ready" instead of saying the disk was full.
const FreeSpaceFloor = 20 << 30
// FreeBytes reads the bytes an unprivileged writer can still use on the filesystem holding path. A variable so a
// test answers for a disk it does not have.
var FreeBytes = func(path string) (uint64, error) {
var s syscall.Statfs_t
if err := syscall.Statfs(path, &s); err != nil {
return 0, err
}
return s.Bavail * uint64(s.Bsize), nil
}
// RoomToWork refuses, saying how much is free, when any of the paths' filesystems is under FreeSpaceFloor. A path
// that does not exist yet (a workspace not made) is read at its nearest folder that does.
func RoomToWork(paths ...string) error {
for _, p := range paths {
at := p
for {
if _, err := os.Stat(at); err == nil || filepath.Dir(at) == at {
break
}
at = filepath.Dir(at)
}
free, err := FreeBytes(at)
if err != nil {
return fmt.Errorf("the free space on %s cannot be read: %w", p, err)
}
if free < FreeSpaceFloor {
return fmt.Errorf("the build node has %d GiB free on %s, under the floor of %d GiB",
free>>30, p, FreeSpaceFloor>>30)
}
}
return nil
}
+8 -8
View File
@@ -172,11 +172,11 @@ func TestANarrowedTreeHoldsTheBuildSourceAndNothingElse(t *testing.T) {
}
}
// This repository's own programs: the route proxy's build source is not the controller's, and neither
// holds the other's command.
// This repository's own programs: the route proxy lives in its module's own Go module, so its build source is
// not the controller's, and neither holds the other's command.
func TestThisRepositorysProgramsHaveBuildSourcesOfTheirOwn(t *testing.T) {
root := filepath.Join("..", "..")
proxy, err := GoBuildSource(root, "examples/route-proxy")
proxy, err := GoBuildSource(filepath.Join(root, "modules", "route-proxy"), "cmd/route-proxy")
if err != nil {
t.Fatal(err)
}
@@ -194,15 +194,15 @@ func TestThisRepositorysProgramsHaveBuildSourcesOfTheirOwn(t *testing.T) {
file string
held bool
}{
{proxy, "the route proxy", "examples/route-proxy/main.go", true},
{proxy, "the route proxy", "internal/broker/broker.go", true},
{proxy, "the route proxy", "cmd/mesh-controller/main.go", false},
{proxy, "the route proxy", "internal/conditions/condition.go", false},
{proxy, "the route proxy", "cmd/route-proxy/main.go", true},
{proxy, "the route proxy", "cmd/route-proxy/bus.go", true},
{proxy, "the route proxy", "go.mod", true},
{proxy, "the route proxy", "cmd/route-provider/main.go", false},
{proxy, "the route proxy", "README.md", false},
{controller, "the controller", "cmd/mesh-controller/main.go", true},
{controller, "the controller", "internal/conditions/condition.go", true},
{controller, "the controller", "internal/inventory/migrations/0001-nodes.sql", true},
{controller, "the controller", "examples/route-proxy/main.go", false},
{controller, "the controller", "modules/route-proxy/cmd/route-proxy/main.go", false},
{controller, "the controller", "cmd/mesh-builder/main.go", false},
{controller, "the controller", "README.md", false},
{builder, "the build seat's program", "cmd/mesh-builder/main.go", true},
+45 -3
View File
@@ -3,6 +3,7 @@ package builder
import (
"context"
"os/exec"
"regexp"
"strings"
"syscall"
"time"
@@ -41,6 +42,11 @@ func inItsOwnGroup(cmd *exec.Cmd) {
cmd.WaitDelay = KillWait
}
// CgroupParent is the slice every container a build or check starts joins (novox/hq issue 1798), so the node's own
// weights and cap bind it: the build-agent module declares the slice with the machine's CPU weight, IO weight and
// memory limits, and names it in MESH_BUILD_CGROUP_PARENT. Empty, a container joins the runtime's default.
var CgroupParent string
// Labelled is a Runner that marks every container a build starts with the build's id, so a kill
// finds what outlived the docker client (novox/hq ADR 0219). Applied at the one place every command
// passes rather than at each `docker run` the builder composes: a run added later is labelled too.
@@ -55,8 +61,11 @@ func LabelledArgs(name string, args []string, id string) []string {
if name != "docker" || len(args) == 0 || args[0] != "run" || id == "" {
return args
}
out := make([]string, 0, len(args)+2)
out := make([]string, 0, len(args)+4)
out = append(out, "run", "--label", BuildLabel+"="+id)
if CgroupParent != "" {
out = append(out, "--cgroup-parent", CgroupParent)
}
return append(out, args[1:]...)
}
@@ -71,12 +80,45 @@ func RemoveContainersOf(ctx context.Context, run Runner, id string) (int, error)
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
if err := removeWithVolumes(ctx, run, ids); err != nil {
return 0, err
}
return len(ids), nil
}
// MountedVolumes is the template `docker inspect --format` reads the names of a container's volumes with.
const MountedVolumes = `{{range .Mounts}}{{if eq .Type "volume"}}{{println .Name}}{{end}}{{end}}`
// anonymousVolume is the name docker gives a volume it makes for an image's VOLUME line: 64 hex digits.
var anonymousVolume = regexp.MustCompile(`^[0-9a-f]{64}$`)
// removeWithVolumes removes containers together with their anonymous volumes (novox/hq issue 1772). A throwaway
// store declares its data directory a VOLUME, docker makes an anonymous volume for it, and `docker rm -f`
// without -v leaves that volume for ever: 1,629 of them, 107 GB, on one build node. So the volumes the
// containers mount are read first, the containers are removed with -v, and an anonymous volume still there
// afterwards is removed by name. Only a volume one of these containers mounted, and only one with an anonymous
// volume's name, is touched: a named volume — an operator's own data — never is. A volume already gone is
// no failure.
func removeWithVolumes(ctx context.Context, run Runner, ids []string) error {
var volumes []string
if out, err := run(ctx, "", "docker", append([]string{"inspect", "--format", MountedVolumes}, ids...)...); err == nil {
seen := map[string]bool{}
for _, name := range strings.Fields(out) {
if anonymousVolume.MatchString(name) && !seen[name] {
seen[name] = true
volumes = append(volumes, name)
}
}
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f", "-v"}, ids...)...); err != nil {
return err
}
if len(volumes) > 0 {
_, _ = run(ctx, "", "docker", append([]string{"volume", "rm"}, volumes...)...)
}
return nil
}
// A holder that stops mid-build leaves its containers running (novox/hq issue 306).
//
// A build's containers are removed by the build itself, by a kill, or by the next delivery of the same
@@ -106,7 +148,7 @@ func RemoveLeftBehind(ctx context.Context, run Runner) (int, error) {
if len(ids) == 0 {
return 0, nil
}
if _, err := run(ctx, "", "docker", append([]string{"rm", "-f"}, ids...)...); err != nil {
if err := removeWithVolumes(ctx, run, ids); err != nil {
return 0, err
}
return len(ids), nil
+76 -2
View File
@@ -2,6 +2,7 @@ package builder
import (
"context"
"errors"
"reflect"
"strings"
"testing"
@@ -49,7 +50,8 @@ func TestAKillRemovesTheContainersLabelledWithTheBuild(t *testing.T) {
if err != nil || n != 2 {
t.Fatalf("%d %v", n, err)
}
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker rm -f c1 c2"}; !reflect.DeepEqual(ran, want) {
if want := []string{"docker ps -aq --filter label=mesh.build=build-3", "docker inspect --format " + MountedVolumes + " c1 c2",
"docker rm -f -v c1 c2"}; !reflect.DeepEqual(ran, want) {
t.Errorf("ran %v", ran)
}
}
@@ -73,7 +75,7 @@ func TestAStartingHolderRemovesWhatEarlierHoldersLeftAndNothingElse(t *testing.T
if n != 3 {
t.Errorf("removed %d", n)
}
if len(ran) != 2 || !reflect.DeepEqual(ran[1], []string{"docker", "rm", "-f", "c1", "c2", "c7"}) {
if len(ran) != 3 || !reflect.DeepEqual(ran[2], []string{"docker", "rm", "-f", "-v", "c1", "c2", "c7"}) {
t.Fatalf("ran %v", ran)
}
if !strings.Contains(strings.Join(ran[0], " "), "--filter label=mesh.build") {
@@ -90,3 +92,75 @@ func TestAStartingHolderRemovesWhatEarlierHoldersLeftAndNothingElse(t *testing.T
t.Errorf("with nothing of the seat's: %d %v %v", n, err, ran)
}
}
// **Issue 1772**: a throwaway store declares its data directory a VOLUME, so docker makes an anonymous volume for
// it, and `docker rm -f` without -v leaves that volume behind (1,629 of them, 107 GB, on one build node). Every
// container a build or check removes is removed with its anonymous volumes (-v), and any volume the container
// mounted that is still there afterwards is removed by name — those, and only those, never a named volume.
func TestARemovedContainerTakesItsAnonymousVolumesWithIt(t *testing.T) {
anon := strings.Repeat("a", 64)
other := strings.Repeat("b", 64)
var ran []string
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
switch args[0] {
case "ps":
return "c1\n", nil
case "inspect":
return anon + "\n" + "compose_data\n" + other + "\n", nil
}
return "", nil
}
if n, err := RemoveContainersOf(context.Background(), run, "build-4"); err != nil || n != 1 {
t.Fatalf("%d %v", n, err)
}
last := ran[len(ran)-1]
if want := "docker volume rm " + anon + " " + other; last != want {
t.Errorf("last command %q, want %q", last, want)
}
if !strings.Contains(strings.Join(ran, "\n"), "docker rm -f -v c1") {
t.Errorf("ran %v", ran)
}
if strings.Contains(last, "compose_data") {
t.Errorf("a named volume was removed: %q", last)
}
}
// A volume already gone when the sweep looks (rm -v took it) is no failure.
func TestAVolumeAlreadyGoneIsNoFailure(t *testing.T) {
run := func(_ context.Context, _ string, name string, args ...string) (string, error) {
switch args[0] {
case "ps":
return "c1\n", nil
case "inspect":
return strings.Repeat("c", 64) + "\n", nil
case "volume":
return "", errors.New("no such volume")
}
return "", nil
}
if n, err := RemoveContainersOf(context.Background(), run, "build-5"); err != nil || n != 1 {
t.Fatalf("%d %v", n, err)
}
}
// A node's build slice limits everything a build or check starts (novox/hq issue 1798): with a parent
// set, every `docker run` joins it, so the node's CPU and IO weights and memory cap bind the container;
// with none set the arguments are as they always were.
func TestEveryContainerJoinsTheNodesBuildSlice(t *testing.T) {
was := CgroupParent
defer func() { CgroupParent = was }()
CgroupParent = "meshbuild.slice"
got := LabelledArgs("docker", []string{"run", "--rm", "img", "sh"}, "build-1")
want := []string{"run", "--label", "mesh.build=build-1", "--cgroup-parent", "meshbuild.slice", "--rm", "img", "sh"}
if !reflect.DeepEqual(got, want) {
t.Errorf("1798/AC1: docker run became %v", got)
}
if got := LabelledArgs("docker", []string{"build", "."}, "build-1"); !reflect.DeepEqual(got, []string{"build", "."}) {
t.Errorf("a docker build became %v", got)
}
CgroupParent = ""
if got := LabelledArgs("docker", []string{"run", "img"}, "build-1"); !reflect.DeepEqual(got, []string{"run", "--label", "mesh.build=build-1", "img"}) {
t.Errorf("1798/AC1: with no slice, docker run became %v", got)
}
}
+29 -4
View File
@@ -43,6 +43,11 @@ type Registry struct {
HTTP *http.Client
}
// PushedNames are the names PublishImage leaves the image under on the node.
func (r Registry) PushedNames(repository string) []string {
return []string{r.Address + "/" + repository}
}
// PublishImage pushes a locally built image and returns a reference pinned by digest.
//
// The digest is read back from the registry's own answer rather than computed here. What matters
@@ -59,15 +64,35 @@ func (r Registry) PublishImage(ctx context.Context, localTag, repository string)
}); err != nil {
return "", err
}
out, err := r.Run(ctx, "", "docker", "inspect", "--format", "{{index .RepoDigests 0}}", remote)
// Every digest the daemon lists for the image, one per line: the image may have been pushed under other names before
// (a check's scratch copy, another registry), and the first of them is not the one this push made (novox/hq issue
// 1742: a build recorded a digest the store never held).
out, err := r.Run(ctx, "", "docker", "inspect", "--format", "{{range .RepoDigests}}{{println .}}{{end}}", remote)
if err != nil {
return "", fmt.Errorf("pushed %s and cannot read back what it was pinned as: %w", remote, err)
}
pinned := strings.TrimSpace(out)
if !strings.Contains(pinned, "@sha256:") {
first, pinned := "", ""
for _, line := range strings.Split(out, "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
if first == "" {
first = line
}
if strings.HasPrefix(line, remote+"@sha256:") {
pinned = line
break
}
}
if pinned == "" {
if strings.Contains(first, "@sha256:") {
return "", fmt.Errorf("pushed %s, but the daemon lists no digest for that name (only %q): the digest of another "+
"repository is not what this registry serves", remote, first)
}
// A tag is not a pin. It can be made to point at something else, and this file is applied
// on machines with no mesh to ask about anything (novox/hq ADR 0006).
return "", fmt.Errorf("%s came back as %q, which is not pinned by digest", remote, pinned)
return "", fmt.Errorf("%s came back as %q, which is not pinned by digest", remote, first)
}
return pinned, nil
}
+29
View File
@@ -252,3 +252,32 @@ func TestAPushedImageComesBackPinned(t *testing.T) {
t.Fatalf("it did not tag and then push: %v", ran)
}
}
// 1742: an image pushed under other names before lists their digests too, and the digest recorded is the one of the name
// this push made — never the first listed, which may be another repository's.
func TestThePinnedDigestIsTheOneOfTheNamePushed(t *testing.T) {
want := "registry.invalid/shell/server@sha256:" + strings.Repeat("b", 64)
earlier := "registry.invalid/scratch/shell/server@sha256:" + strings.Repeat("a", 64)
r := Registry{Address: "registry.invalid", Run: func(
_ context.Context, _, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
return earlier + "\n" + want + "\n", nil
}
return "", nil
}}
got, err := r.PublishImage(context.Background(), "local", "shell/server")
if err != nil || got != want {
t.Fatalf("pinned %q (%v); want %q", got, err, want)
}
// Only another repository's digest is listed: the push is not taken for pinned.
r.Run = func(_ context.Context, _, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "inspect" {
return earlier + "\n", nil
}
return "", nil
}
if _, err := r.PublishImage(context.Background(), "local", "shell/server"); err == nil ||
!strings.Contains(err.Error(), "no digest for that name") {
t.Fatalf("a digest of another repository was taken for the push's own: %v", err)
}
}
+7
View File
@@ -253,6 +253,13 @@ func (s scratchPublisher) PublishImage(ctx context.Context, localTag, repository
return s.to.PublishImage(ctx, localTag, ScratchPrefix+repository)
}
func (s scratchPublisher) PushedNames(repository string) []string {
if p, ok := s.to.(PushedNames); ok {
return p.PushedNames(ScratchPrefix + repository)
}
return nil
}
func (s scratchPublisher) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
return s.to.PublishArchive(ctx, ScratchPrefix+repository, body, digest)
}

Some files were not shown because too many files have changed in this diff Show More