Compare commits

...
Author SHA1 Message Date
mesh-toolsandClaude Opus 5.5 f877c33c7c Read the borrowed-words log without the logger's clock, so the test passes at any time of day (issue 752)
mesh/merge-gate head f877c33c7c18 merged into trunk 0b7e740b299c — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head f877c33c7c18 merged into trunk 0b7e740b299c — pass: its merge-check.sh passed
mesh/delivery ready: it delivers once merged
The test forbids "23:3" in the log line, and the logger's own timestamp held it from 23:30
to 23:39 every night, so the check went red for no reason in that window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:51:05 +00:00
mesh-tools 0b7e740b29 Merge pull request 'Let the tool runner's account hear the event families its manifest names (issue 745, ADR 0367)' (#342) from 745-mesh-wait into main 2026-10-14 23:14:41 +00:00
mesh-toolsandClaude Opus 5.5 755457cc37 Let the tool runner's account hear the event families its manifest names (issue 745, ADR 0367)
mesh/merge-gate head 755457cc379e merged into trunk 4cc6b6c566a5 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 755457cc379e merged into trunk 4cc6b6c566a5 — pass: its merge-check.sh passed
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
A session on the mesh MCP server polls for every verdict, answer and delivery because the tool
runner's account hears no event. ADR 0367 lets the tool runner's module name, one by one, the
families a session may wait on: the manifest's `waits`, refused for any other module and for a
wildcard. The controller grants the runtime's account a plain subscription on those subjects
alone, makes no durable consumer for them, and issues the same list in the tool runner's
membership, so the grant and what `mesh_wait` offers come from one composition and no call or
setting widens it. A family on a seat whose events end in a kind or an asker must name it, and is
refused at composition without it, so it is never granted a subject nothing publishes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:05:14 +00:00
mesh-admin 4cc6b6c566 Merge pull request 'Say a read-only mesh-cli call in no line of its own, and count them hourly (hq issue 762)' (#341) from fix/762-a-read-call-writes-no-journal-line into main 2026-10-14 22:33:08 +00:00
jschoubben 9e11790bd7 Say a read-only mesh-cli call in no line of its own, and count them hourly (hq issue 762)
mesh/merge-gate head 9e11790bd74a merged into trunk 3ed08c3464e3 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 9e11790bd74a merged into trunk 3ed08c3464e3 — pass: its merge-check.sh passed
mesh/delivery delivered
A page polling plans every few seconds wrote about 14,000 journal lines a day. A line is judged a
read by the generic command verb's reading forms (ADR 0266), on its every word, so plans go,
conditions silence and settings set are still said; a refused or failed read is said too.
2026-10-15 00:28:17 +02:00
mesh-admin 3ed08c3464 Merge pull request 'Take no rebuild into a waiting rollout, and ask what a tier left unasked (hq issue 754)' (#340) from fix/754-a-rebuild-in-the-wait-is-not-the-walks-build into main 2026-10-14 22:06:58 +00:00
jschoubben 9f072c110c Take no rebuild into a waiting walk, and ask what a tier left unasked (hq issue 754)
mesh/merge-gate head 9f072c110c14 merged into trunk 60cc9ec363df — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 9f072c110c14 merged into trunk 60cc9ec363df — pass: its merge-check.sh passed
mesh/delivery delivered
A rebuild that landed while a walk waited for mesh-delivery's word was taken as
the walk's build of that module; once let go, the walk asked a tier only when
every module of it was unasked, so it never asked the rest and stood still.

A walk now takes only the outcomes of builds it asked, and none while it waits;
a rebuild does not join a waiting walk and says it runs beside it; and a tier
asked in part is asked the rest, including a module held built that the walk
never asked for, which is somebody else's build and not sent as the walk's.
2026-10-15 00:02:28 +02:00
mesh-admin 60cc9ec363 Merge pull request 'End the check of a pull request's old head when a new head is checked (hq issue 726)' (#339) from fix/726-a-new-push-ends-the-old-check into main 2026-10-14 20:28:16 +00:00
mesh-admin 5d22d7df86 Merge pull request 'A module moves only when its program changes (hq issue 721)' (#336) from fix/a-module-moves-only-when-its-program-changes into main 2026-10-14 20:21:05 +00:00
mesh-admin 86057f4551 Merge pull request 'A module may declare a stream of its own, heard only by the consumers its trusted config value grants (hq issue 718, ADR 0363)' (#338) from fix-718-module-streams-and-grants into main 2026-10-14 20:13:33 +00:00
jschoubben e4acab32d6 End the check of a pull request's old head when a new head is checked
mesh/merge-gate head e4acab32d6a2 merged into trunk 60cc9ec363df — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head e4acab32d6a2 merged into trunk 60cc9ec363df — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792009731456564734 (novox/mesh-controller at 60cc9ec3 into ma…
A check judges one head; once the pull request has another, its verdict
answers nobody and its run holds a build machine the new head waits
behind. Asking a check of R#N at H now drops every waiting check of R#N
at another head and kills every one in flight, saying each in the log.
Announcements are unordered and a recheck asks a recorded head, so this
happens only when the forge confirms H is the pull request's current
head; otherwise nothing ends and that is said. A one-shot recheck ends
them within the call. Tries, walk builds, group checks, other pull
requests' checks and rechecks of the same head are left alone.

novox/hq issue 726
2026-10-14 22:10:15 +02:00
jochen 743e6478a7 Remove a restricted stream's consumer when the bus refuses to narrow it, so its wider filters deliver nothing more (ADR 0363 §6, review of PR 338)
mesh/merge-gate head 743e6478a76c merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 743e6478a76c merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 22:05:47 +02:00
jochen ede2772b3c Deliver a module stream's given-up message again only while its consumer is still granted it (ADR 0363 §7)
mesh/merge-gate head ede2772b3c90 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head ede2772b3c90 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:55:55 +02:00
jschoubben b34055f580 Register the compared module after the fixture's own registration (review of #336)
mesh/merge-gate head b34055f580d4 merged into trunk 86057f455194 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head b34055f580d4 merged into trunk 86057f455194 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792009731456564734 (novox/mesh-controller at 60cc9ec3 into ma…
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes delivering: 1 of 2 delivered; novox/mesh-catalog@d94764f00168, stopped for a member …
2026-10-14 21:53:50 +02:00
jochen ee13e75bf3 Let a module declare a stream of its own and restrict who hears it by a trusted config value (hq issue 718, ADR 0363)
mesh-mail's events must live on a stream of their own and reach only the modules the operator granted
an account (ADR 0359 §2, §4). The controller now creates a module's declared stream, publishes its events
on mesh.stream.<module>.event.<event>, and composes a consumer's durable consumer, filters and bus grants
on a restricted stream only where its consumes asks and the provider's trusted value names it; a consumer
that loses its grant has its consumer removed on the next assertion. Nothing changes while no manifest
declares a stream.
2026-10-14 21:52:45 +02:00
jschoubben 480657d95b Register the module a walk's store test compares with, as the mesh runs it (review of #336)
mesh/merge-gate head 480657d95b9b merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 480657d95b9b merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestAWalkWhoseEveryBuildCameOutTheSameIsDone …
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:43:20 +02:00
jschoubben 38c87a7e47 Compare a build with the one the mesh registered, not the newest, which may be one put back (review of #336, hq issue 721)
mesh/merge-gate head 38c87a7e4714 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 38c87a7e4714 merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestAWalkWhoseEveryBuildCameOutTheSameIsDone …
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:30:09 +02:00
jschoubben 6acce60f8b List the store test of a walk whose every build came out the same (hq issues 611, 721)
mesh/merge-gate head 6acce60f8bd3 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6acce60f8bd3 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:23:46 +02:00
jschoubben 4390589c61 Leave out a dependent of a base that came out the same, and end a walk with every build the same as done (ADR 0267 rule 3; hq issues 607, 611, 721)
mesh/merge-gate head 4390589c61a4 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4390589c61a4 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:19:14 +02:00
jschoubben e6a28a2d28 Say why the walk moves every module when the checks' builds cannot be read (lint, hq issue 721)
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes checking: 1 of 2 member(s) ready
mesh/merge-gate head e6a28a2d2814 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head e6a28a2d2814 merged into trunk c2ec0a601acb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 21:15:24 +02:00
jschoubben 32bd8b5d5f Leave a module out of a walk when its check built it the same as what runs (hq issue 721)
mesh/merge-gate head 32bd8b5d5fb2 merged into trunk c2ec0a601acb — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 32bd8b5d5fb2 merged into trunk c2ec0a601acb — fail: its merge-check.sh failed: --- FAIL: TestNoReaderAnswersEmptyForAnError (2.20s)
mesh/delivery-group group fix/a-module-moves-only-when-its-program-changes rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
A one-sentence change in a package mesh-builder imports but never reaches moved
build-agent. A merge check now compares each module it builds with the build the
mesh holds, the whole image without file times, labels or creation time, and a
walk is cut without a module every check of its pull requests found the same,
while the mesh still holds that build. A comparison that fails moves the module
as before.
2026-10-14 21:07:25 +02:00
mesh-admin c2ec0a601a Merge pull request 'A one-machine walk says its report was its run (hq issue 720)' (#335) from fix/720-a-one-machine-walk-says-its-report into main 2026-10-14 18:59:22 +00:00
jschoubben 34050dc6a3 Say a one-machine walk's report was its run: no gate reads it since ADR 0360 (hq issue 720)
mesh/merge-gate head 34050dc6a3c5 merged into trunk a5500f732f77 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 34050dc6a3c5 merged into trunk a5500f732f77 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 20:55:04 +02:00
mesh-admin a5500f732f Merge pull request 'Run the slow test packages in shards and build judges from a warm Go cache (hq issue 717)' (#334) from fix/717-the-suite-runs-in-under-two-minutes into main 2026-10-14 18:46:47 +00:00
jschoubben 3ec4473b29 Run the slow test packages in shards and stop compiling the standard library per judge test (hq issue 717)
mesh/merge-gate head 3ec4473b29b2 merged into trunk df8c1ccf1fff — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 3ec4473b29b2 merged into trunk df8c1ccf1fff — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
The controller's check spent 3-5 minutes in cmd/mesh-controller, whose few hundred store tests run one
after another, while most packages end in seconds. merge-check.sh now compiles each slow package once and
deals its tests round-robin into processes that run beside each other and beside the rest of the suite;
every test still runs once, under the race detector, through its package's TestMain. The listed store
tests run beside the suite instead of after it.

internal/builder's judge tests ran a real go build with a fresh Go cache each, compiling the standard
library every time: half the suite's CPU. The fake runtime fills an empty cache from one warmed per
process. The lease tests run in parallel. The link test of slow work now sets its one-second window after
the serving loop asserted the controller's thirty-second one, so it proves the heartbeat (it passed
without it) in half the time.
2026-10-14 20:38:35 +02:00
mesh-admin df8c1ccf1f Merge pull request 'Build a check's builds where the host's engine sees them, a tier at once and beside the tests (hq issues 713, 716)' (#333) from fix/713-the-walk-reuses-its-checks-builds into main 2026-10-14 18:36:18 +00:00
jschoubben bbee20adf9 Build a merge's modules of a tier at once, and beside the check's tests where the judge cache allows (hq issue 716)
mesh/merge-gate head bbee20adf9b9 merged into trunk 622fb1e9cbac — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head bbee20adf9b9 merged into trunk 622fb1e9cbac — pass: its merge-check.sh passed
mesh/delivery delivered
A check built its images one after another, after merge-check.sh. With a judge cache the
builds run during the tests, out of their reach; a failed check lets go of them.
2026-10-14 20:20:49 +02:00
jschoubben afed887217 Build a check's builds for its merge where the host's container engine sees them (hq issue 713)
The check built in a temporary directory of the build holder's container, which the
host's engine mounted empty: every check build failed and every walk built again.
2026-10-14 20:14:20 +02:00
mesh-admin 622fb1e9cb Merge pull request 'Judge a pull request into a release branch by that branch's controller and repositories (issue 704, ADR 0358)' (#329) from fix/704-release-branch-judge into main 2026-10-14 18:09:03 +00:00
mesh-adminandClaude Opus 5.5 5d5bdf9e32 Refuse the replays without a judge cache rather than run them from a volume
mesh/merge-gate head 5d5bdf9e32d1 merged into trunk 4af9f1147de6 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 5d5bdf9e32d1 merged into trunk 4af9f1147de6 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/704-release-branch-judge delivered: every member is delivered
A check's own volume is no place for the replays: the store replay binds a
directory it makes there into other containers by path, which the container
runtime would create on the build node's own disk (review 12308). Without the
build agent's judge cache the replays are refused and the gate says so as an
error; replays the runtime cannot start are the check's error, not the
change's; and an id that cannot name a place in the judge cache is refused
(hq issue 709, issue 704).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:58:07 +00:00
mesh-admin 4af9f1147d Merge pull request 'Send a merge's builds to every machine at once, with no first-machine gate (hq ADR 0360)' (#332) from feat/lean-rollout into main 2026-10-14 17:53:34 +00:00
jschoubben 704b21b3c5 Carry the new controller's user list to the bus when its walk ends, as the gate's pass did (hq ADR 0236 §34, ADR 0360)
mesh/merge-gate head 704b21b3c547 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 704b21b3c547 merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 19:36:28 +02:00
jschoubben 5b455b8a24 Keep the first-machine gate behind a switch for walks begun before hq ADR 0360, and test both paths
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 5b455b8a2438 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 5b455b8a2438 merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestARenewalWrittenThoughItsAnswerWasLostIsSt…
2026-10-14 19:35:07 +02:00
jschoubben 0e97413875 Assert the backlog walk sends every machine at once and passes on the report (hq ADR 0360)
mesh/merge-gate head 0e974138751e merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 0e974138751e merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestAReusedCheckBuildIsRegisteredByItsDigestA…
mesh/delivery superseded: a newer head of the same pull request
2026-10-14 19:21:42 +02:00
jschoubben da8a5e9bc0 Send a merge's builds to every machine at once, with no first-machine gate (hq ADR 0360)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head da8a5e9bc0f7 merged into trunk 76304d9db817 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head da8a5e9bc0f7 merged into trunk 76304d9db817 — fail: its merge-check.sh failed: --- FAIL: TestAReusedCheckBuildIsRegisteredByItsDigestA…
The operator chose the leanest rollout: no gate, no one machine first, failures loud and fixed forward.
A walk sends every rolled-out module to every machine running it in one step, built-by orders no tier,
and a walk ends on its machines' reports. The backlog walk sends every machine at once and passes each
on its applied report.
2026-10-14 19:19:49 +02:00
mesh-adminandClaude Opus 5.5 847ffd6d12 Never run the replays from the workspace, with or without a judge cache
mesh/merge-gate out of date: judged against trunk 76304d9db817, the trunk is now 4af9f1147de6; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 76304d9db817, the trunk is now 4af9f1147de6; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Without a judge cache the replays still ran mesh-lab's checkout in the workspace
with the container runtime's socket, so a self-judging pull request could write
a test into them (hq issue 709, review 12303). They now run from a copy in a
container volume of the check's own, made before anything of the change runs,
with a fresh home and throwaway caches, removed after the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:34:00 +00:00
mesh-adminandClaude Opus 5.5 0214b45469 Run mesh-lab's replays where no pull request reaches
mesh/merge-gate head 0214b45469ec merged into trunk 76304d9db817 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 0214b45469ec merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The replays hold the container runtime's socket, yet took their HOME, Go
environment, build cache and module cache from the workspace a pull request's
own script writes, so a planted -toolexec hook ran as root on the build node
(hq issue 709). They now run from a copy of mesh-lab made before anything of
the change runs, with a fresh home and the judge cache's caches, as a judge is
built (ADR 0358 §6).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 14:15:19 +00:00
mesh-adminandClaude Opus 5.5 2bee290eea Keep judge builds warm in the build agent's judge cache, and leave nothing on the host
mesh/delivery-group group fix/704-release-branch-judge rejected: its order contradicts itself: novox/mesh-catalog@47ef2644aca1 and novox/mesh-controller@2bee29…
mesh/merge-gate head 2bee290eea56 merged into trunk 76304d9db817 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 2bee290eea56 merged into trunk 76304d9db817 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Review of #329: a judge's home made with a temporary directory inside the build
agent is resolved on the host by the container runtime, leaving a cold cache
behind on every check, and a kept judge was lost on every redeploy. Judge builds
now use the judge cache the build agent declares (MESH_JUDGE_CACHE), mounted into
judge builds only: one warm Go cache, the kept judges, and a fresh home per check
removed after it (hq ADR 0358 §6, issue 704, issue 482).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:53:05 +00:00
mesh-admin 76304d9db8 Merge pull request 'Keep the controller lease through a stalled renewal until its TTL passes (hq issue 477)' (#331) from fix-477-lease-survives-stall into main 2026-10-14 13:35:23 +00:00
mesh-adminandClaude Opus 5.5 b7084a19e9 Build every judge where no pull request can reach, and never read mesh-lab at a release
mesh/merge-gate out of date: judged against trunk c975dbf7b053, the trunk is now 76304d9db817; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk c975dbf7b053, the trunk is now 76304d9db817; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #329: a judge built with HOME in the workspace runs git under a
configuration a pull request's script can plant, so a hook could rewrite the
judge's sources before they compile and the result be kept under an honest key
(issue 482, issue 704). Every judge now builds with its own HOME, configuration
and Go cache outside the workspace, no git configuration and no VCS stamping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:30:46 +00:00
mesh-admin c975dbf7b0 Merge pull request 'A module may declare its unhealthy urgent, with why (hq ADR 0355, issue 691)' (#330) from adr-355-urgent-unhealthy into main 2026-10-14 13:21:56 +00:00
jochen 98e8568749 Keep the controller lease through a renewal the bus did not answer, until its TTL passes
mesh/merge-gate head 98e856874913 merged into trunk c975dbf7b053 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 98e856874913 merged into trunk c975dbf7b053 — pass: its merge-check.sh passed
mesh/delivery delivered
A bus stall of a few seconds made a renewal time out, and the controller
called that a lost lease, exited and raised the urgent lease-lost alert,
though no other controller could take a key that still lived. Now only
another holder seen, or the key's TTL passed unrenewed, loses the lease;
new acts stop at half the TTL unrenewed and resume on the next renewal
(novox/hq issue 477).
2026-10-14 15:20:19 +02:00
jochen 58889235d1 Raise a module's unhealthy urgent when its manifest says why (hq ADR 0355, issue 691)
mesh/merge-gate head 58889235d163 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 58889235d163 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
mesh/delivery delivered
A broken service people rely on, such as mail, was said only as a warning on
the desktop. A module may now declare unhealthy urgent with what people lose;
the module check refuses it without why or with another severity.
2026-10-14 15:13:57 +02:00
mesh-adminandClaude Opus 5.5 48b2906b5c Keep mesh-lab at its trunk and build the release judge with a cache of its own
mesh/merge-gate out of date: judged against trunk 535c7641ac41, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 535c7641ac41, the trunk is now c975dbf7b053; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of #329: mesh-lab's replays run with the container runtime's socket, so
a same-named branch must never be what runs there; the shared Go cache and HOME
are writable by a pull request's own script, so the judge kept for a head must
not be built from them (issue 704, issue 482). A forge answer naming no branch
is an error, and a trunk is the branch a repository follows, not a name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:13:30 +00:00
mesh-adminandClaude Opus 5.5 ea7d91bff1 Say which validator judged a release-branch check, and prove clause 5
mesh/merge-gate head ea7d91bff190 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head ea7d91bff190 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ADR 0358 version 4 asks the verdict to say whether mesh-host's release-branch
validator or the vendored one judged, and checks that a release branch's own
pull request into its trunk is judged as any trunk pull request (issue 704).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:56:55 +00:00
mesh-adminandClaude Opus 5.5 4a75cc51dd Judge a pull request into a release branch by that branch's controller and repositories
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 4a75cc51dda6 merged into trunk 535c7641ac41 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 4a75cc51dda6 merged into trunk 535c7641ac41 — pass: its merge-check.sh passed
A release branch could never carry a change that needs another repository's
release-branch change: its pull requests were judged by the running controller
and against the other repositories' main (hq issue 704, ADR 0358).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:48:44 +00:00
mesh-admin 535c7641ac Merge pull request 'A fresh check keeps a build whose key is unchanged instead of building it again (hq issue 703)' (#328) from fix/703-a-fresh-check-keeps-an-unchanged-build into main 2026-10-14 08:15:56 +00:00
jschoubben c707bf18d8 Say which check made a build the walk does not take, and test the merged heads
mesh/merge-gate head c707bf18d8bf merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head c707bf18d8bf merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery delivered
A build of another pull request or of a head the forge did not merge is
in no verdict the walk is handed; the walk now names the check that made
it, so its "built:" line says which head it was (novox/hq issue 703).
2026-10-14 10:05:31 +02:00
jschoubben 060f2a9ca5 Build for the merge from a copy its tests never touch, and keep or take only the same head's build
mesh/merge-gate head 060f2a9ca5fd merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 060f2a9ca5fd merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The gate and the repository's check mount the workspace read-write, so a
build from that tree could hold what the tests wrote under a key that
describes the merge. A check now clones the merge outside the workspace
before anything of the change runs, and builds from that. A check keeps,
and a walk takes, only a build an earlier check of the same pull request
at the same head made: another pull request's build, or an earlier
head's, is never registered. The walk reads the verdicts once
(novox/hq issue 703, review of #328).
2026-10-14 09:56:23 +02:00
jschoubben 1666747a22 Keep a check build whose key is unchanged instead of building it again
mesh/merge-gate head 1666747a2246 merged into trunk 04730b54a8f2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 1666747a2246 merged into trunk 04730b54a8f2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A fresh check asked when only the trunk moved rebuilt every image, though
the reuse key said nothing it is built from changed: a controller check
took about 12 minutes instead of 5.5. The check now keeps a build kept
under its key when a verdict of the build seat names that key and every
digest, names it as the build's own check so a walk takes it, and the
cleanup keeps it while any check names it (novox/hq issue 703).
2026-10-14 09:41:39 +02:00
mesh-admin 04730b54a8 Merge pull request 'S9 raises every bus user's slow consumer and closed connection, from the bus module's events (hq issue 270, ADR 0357)' (#326) from fix/270-the-bus-module-says-every-users-slow-consumer into main 2026-10-14 07:14:33 +00:00
jochen 45870ab31c Bound the read of bus-module events, keep S9 seeing when it fails, count the controller's own slow consumer once (hq issue 270, review of #326)
mesh/merge-gate head 45870ab31cc5 merged into trunk 3479a35462b2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 45870ab31cc5 merged into trunk 3479a35462b2 — pass: its merge-check.sh passed
mesh/delivery delivered
A storm of closures must not spend a watchdog tick, so the read stops at 2000
events and says the count is a floor. A failed read is its own condition
rather than blinding S9, so the controller's own signals are still said. The
controller's fall behind, told to its client and closed by the server, is one
fault, counted once.
2026-10-14 09:02:47 +02:00
jochen fd9308d328 Raise every bus user's slow consumer and closed connection as S9, from the bus module's events (hq issue 270, ADR 0357)
mesh/delivery-group group fix/270-the-bus-module-says-every-users-slow-consumer rejected: a member's own check failed
mesh/merge-gate head fd9308d3287e merged into trunk 3479a35462b2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fd9308d3287e merged into trunk 3479a35462b2 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The controller heard a slow consumer only on its own connection: the server
says another user's only in its record of closed connections. The bus module
now says that record as events; the watchdogs read the last hour back on every
tick and raise one condition per user and kind, naming the module and machine.
The module check lets the bus module emit, since its tool runner publishes.
2026-10-14 08:41:47 +02:00
mesh-admin 3479a35462 Merge pull request 'List the provider-hold tests among the store tests (hq issue 145, to-be 48 Phase C)' (#319) from fix/145-a-consumer-is-held-under-its-provider into main
mesh/merge-gate head 3479a35462b2 merged into trunk 221fe5a60d1f — fail: head 3479a35462b2 does not merge cleanly into trunk 221fe5a60d1f: in conflict test…
mesh/repo-check head 3479a35462b2 merged into trunk 221fe5a60d1f — fail: head 3479a35462b2 does not merge cleanly into trunk 221fe5a60d1f: in conflict test…
mesh/delivery stopped: the pull request closed unmerged
2026-10-14 05:18:00 +00:00
jochen cebc593627 List the provider-hold tests among the store tests, so a skip cannot pass unseen (hq issue 145)
mesh/merge-gate head cebc593627d3 merged into trunk 2959c8143536 — pass: builds build-agent → ace, g14, novox, shanks; no bus step; every machine composes …
mesh/repo-check head cebc593627d3 merged into trunk 2959c8143536 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/145-a-consumer-is-held-under-its-provider delivered: every member is delivered
The rule 5 test of ADR 0240 (one provider down, three consumers, one
condition) and its healthy-provider twin need the store and skip without
one; unlisted, the merge check passed them without running them.
2026-10-14 03:21:50 +02:00
mesh-admin 2959c81435 Merge pull request 'Link a build's bases into its repository before pushing it (issue 694)' (#318) from fix/694-a-push-holds-its-bases into main 2026-10-14 01:19:24 +00:00
jochen 8933864a49 Link a build's bases into its repository before pushing it (issue 694)
mesh/merge-gate head 8933864a49d0 merged into trunk df9e5728a925 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 8933864a49d0 merged into trunk df9e5728a925 — pass: its merge-check.sh passed
mesh/delivery delivered
A runtime on containerd's image store, the default on a fresh docker 29,
need not hold the compressed layers of a base it built FROM: a layer it
had unpacked already is never fetched. Its push then fails on that layer
although the registry holds it under the base's mirror. A push reads
locally only what the target repository lacks, so mount the base's blobs
there first.
2026-10-14 02:59:53 +02:00
mesh-admin df9e5728a9 Merge pull request 'Try a pushed branch on the build seat before its pull request (hq issue 686)' (#314) from fix/686-a-branch-is-tested-before-its-pull-request into main 2026-10-14 00:57:46 +00:00
mesh-admin 2d18f438c5 Merge pull request 'A no-move tier waits for no new report, so its walk asks the next tier (issue 648)' (#309) from fix/648-a-no-move-tier-waits-for-nothing into main 2026-10-14 00:32:47 +00:00
jschoubben 3ce5a0239b Keep a try off the forge and bound how many run (hq issue 686)
mesh/merge-gate head 3ce5a0239bb1 merged into trunk 2d18f438c516 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 3ce5a0239bb1 merged into trunk 2d18f438c516 — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery delivered
Review of #314: a try's start named its commit, which the forge's holder
matches to an open pull request's head and marks pending. Its start now
names no repository or commit. One try per commit and four in all, a try
never ages ahead of deliveries or pull request checks, and its id is
checked before it is a bus subject.
2026-10-14 02:29:28 +02:00
jochen 70885ab3c4 Share one prefix between the no-move Why's writer and its reader (review of #309, issue 648)
mesh/delivery delivered
mesh/merge-gate head 70885ab3c4a3 merged into trunk ca3f43c400ba — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 70885ab3c4a3 merged into trunk ca3f43c400ba — pass: its merge-check.sh passed
2026-10-14 02:22:25 +02:00
jschoubben 65508d9df9 Try a pushed branch on the build seat before its pull request (hq issue 686)
mesh/merge-gate head 65508d9df9f4 merged into trunk ca3f43c400ba — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 65508d9df9f4 merged into trunk ca3f43c400ba — pass: THE CHANGE ALTERS ITS OWN CHECK (merge-check.sh): main's version judged it; the ch…
mesh/delivery superseded: a newer head of the same pull request
Store tests skip without a store and no agent may start one, so each failure
cost a 15-30 minute review round. The try verb asks the build seat for the
repository's merge-check.sh on the ref merged into the trunk, as a check in
the build queue, and reads its outcome from the build's log: no status on
the forge, nothing registered, nothing built for a merge.
2026-10-14 02:20:37 +02:00
jschoubben 8123ca5654 Test that a branch is tried before its pull request (hq issue 686)
A builder's store tests first ran in the pull request's merge check, a review
round late. These fail on main: a try asks the repository's own check alone,
its outcome sets no status and registers nothing, an unknown ref is refused
before anything is asked, and the answer carries the failing lines.
2026-10-14 02:20:37 +02:00
jochen d9b86376d3 A no-move tier waits for no new report, so its walk asks the next tier (issue 648)
A build made from the source every machine running it runs is marked sent
and sent nothing, yet the tier's wait asked each machine for a report after
that marking. None follows, and the walk waited for ever. Each machine is
now judged by whether it answered its last send, the one whose builds made
the build no move.
2026-10-14 02:18:53 +02:00
mesh-admin ca3f43c400 Merge pull request 'The rest go at once after the first passes, and a gate survives a new controller (hq ADR 0353)' (#312) from fix/the-rest-in-parallel-and-gates-survive-a-new-controller into main 2026-10-13 23:43:02 +00:00
jschoubben b482122d66 Say a read error of a silent rest, and keep the node-engine one at a time in a kept step (hq ADR 0353)
mesh/delivery delivered
mesh/merge-gate head b482122d6646 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head b482122d6646 merged into trunk 1601cb17432e — pass: its merge-check.sh passed
An empty answer on a read error would clear release-silent unseen.
2026-10-14 01:26:00 +02:00
jschoubben 2f5cc146fe Keep a walk's sends to what was kept, and say a silent rest (hq ADR 0353 review)
mesh/merge-gate head 2f5cc146fe46 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 2f5cc146fe46 merged into trunk 1601cb17432e — fail: its merge-check.sh failed: --- FAIL: TestAFirstMachineThatFailsStillStopsTheWalk (…
mesh/delivery superseded: a newer head of the same pull request
A release's own gates held its other sends back for ever; an adopted send
lost what the machine ran before; a kept send could grow a build no gate
passed; a take-over matched by commit alone; a rest that went silent ended
done. The node-engine and the resolver still go one machine at a time, since
a fault that cuts a machine off shows only as silence.
2026-10-14 01:18:51 +02:00
jschoubben c2ea9b4c51 Send the rest at once after the first passes, and resume a stored gate (hq ADR 0353)
mesh/repo-check head c2ea9b4c5187 merged into trunk 1601cb17432e — fail: its merge-check.sh failed: --- FAIL: TestAReleaseSendsTheRestAtOnceAfterTheFirstPa…
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head c2ea9b4c5187 merged into trunk 1601cb17432e — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
A release now sends every other machine in one step once its first machine
passed, each judged by its own gate; a failing one is put back without
stopping the others, a silent one is left. A send is kept before it is made,
and a machine already sent what a gate is for is not sent again: the gate
counts from that send and takes over another walk's readings of the same
build there.
2026-10-14 01:06:49 +02:00
jschoubben b03d257f81 Test that the rest go at once after the first passes and a gate survives a new controller (hq ADR 0353)
The release walked four machines one after another in eleven minutes, and a
new controller re-sent and re-judged a build its machine already ran.
2026-10-14 01:06:49 +02:00
126 changed files with 11307 additions and 447 deletions
+63 -14
View File
@@ -26,6 +26,7 @@ import (
"os/signal"
"sort"
"strings"
"sync"
"syscall"
"time"
@@ -272,7 +273,12 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
@@ -313,25 +319,36 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Seats: request.Seats}
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built)}
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Release: v.Release, JudgedBy: v.JudgedBy, BesideAt: v.BesideAt, Validator: v.Validator}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -346,15 +363,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
for _, b := range request.Reuse.Builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
reuse.Named = append(reuse.Named, named)
}
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
@@ -457,9 +466,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)}
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, Release: c.Release}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref, Else: b.Else}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
@@ -472,11 +482,50 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String()}
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
+27
View File
@@ -0,0 +1,27 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// **A pull request into a release branch reaches the check as one** (novox/hq ADR 0358): the build seat hands the
// check the release branch and, beside it, the trunk each repository is read at when it has no branch of that name;
// and it hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckItsReleaseBranch(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{ID: "b", Repository: "r", Ref: "abc", Check: &link.CheckRequest{
Owner: "novox", Repo: "mesh-catalog", Base: "feat/57", Release: "feat/57",
Beside: map[string]link.CheckedOut{"mesh-host": {Repository: "h", Ref: "feat/57", Else: "main"}}}})
if spec.Release != "feat/57" || spec.Beside["mesh-host"].Ref != "feat/57" || spec.Beside["mesh-host"].Else != "main" {
t.Fatalf("the check was handed release %q and mesh-host %+v", spec.Release, spec.Beside["mesh-host"])
}
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
+31
View File
@@ -0,0 +1,31 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
+5 -2
View File
@@ -26,8 +26,11 @@ import (
//
// - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so
// its service manager restarts it as a candidate (serve, in push.go).
// Another holder seen, or the key's age passed with no renewal made, is the lease lost: the gate
// closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again.
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
+1 -1
View File
@@ -140,7 +140,7 @@ func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testin
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx)
announced, err := following{open: open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
+8 -3
View File
@@ -688,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Merged: m.Merged, Heard: m.Heard}
Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -1000,10 +1000,15 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld
// **Left where it is, a module whose program did not change** (novox/hq issue 721): its check found it the same as
// the build the mesh holds.
for n, why := range leaveSameAsRunning(ctx, inv, &plan, moved, append(append([]inventory.Edge{}, edges...), ordered...)) {
unchanged[n] = why
}
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 {
if len(moved) == 0 || len(plan.Modules) == 0 {
plan.State = inventory.PlanDone
plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk"
@@ -1044,7 +1049,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold.
if err := askTier(ctx, inv, &plan); err != nil {
if err := askTier(ctx, inv, &plan, plan.Tiers[plan.Tier]); err != nil {
return err
}
if err := inv.SavePlan(ctx, &plan); err != nil {
+1 -1
View File
@@ -758,7 +758,7 @@ func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
return entries, read, err
}
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil {
(following{open: open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err)
}
got, _ := open.inventory.PlanByID(ctx, w.ID)
+28
View File
@@ -899,6 +899,34 @@ type answers struct {
//
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull.
// runningBy is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module, as a
// builder reaches the artifact store (novox/hq issue 721): what a check compares its builds with. Empty when it cannot
// be read, said, and then nothing is said unchanged.
func runningBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
defer open.Close()
running, err := open.inventory.RegisteredHeld(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil || address == "" {
return running
}
routed := make(map[string]string, len(running))
for key, reference := range running {
routed[key] = catalogue.Rerouted(reference, address)
}
return routed
}
func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
@@ -0,0 +1,92 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
+30 -1
View File
@@ -60,9 +60,19 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil {
return nil, err
}
// And every module's own stream (novox/hq ADR 0363), from the catalogue, before the consumers on it.
modStreams, err := inv.DeclaredStreams(ctx)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, s := range broker.ModuleStreams(modStreams) {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the module stream %s: %w", s.Name, err))
}
}
for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
@@ -73,11 +83,30 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
}
}
restricted := map[string]bool{}
for _, s := range modStreams {
restricted[s.Name] = s.Restricted
}
for _, c := range consumers {
if err := r.EnsureConsumer(c.Consumer); err != nil {
if err := broker.EnsureHeard(r, c.Consumer, restricted[c.Consumer.Stream]); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
}
}
// **And a consumer no longer granted loses what it reads through** (novox/hq ADR 0363 §6): every consumer
// on a module's stream the composition no longer asks for is removed, on this very send.
if p, prunes := r.(broker.Pruner); prunes && len(modStreams) > 0 {
var wanted []broker.Consumer
for _, c := range consumers {
wanted = append(wanted, c.Consumer)
}
removed, err := broker.PruneHeard(p, modStreams, wanted)
if err != nil {
failed = append(failed, err)
}
for _, gone := range removed {
fmt.Printf(" removed the consumer %s: its module is no longer granted that stream (novox/hq ADR 0363)\n", gone)
}
}
if len(failed) > 0 {
return nil, errors.Join(failed...)
}
+77 -13
View File
@@ -125,6 +125,7 @@ func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
@@ -190,27 +191,90 @@ func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.
}
}
// A walk's leave names the builds its own pull requests' verdicts made, and none when they cannot be read: the
// builder takes nothing a verdict does not name (novox/hq ADR 0350 §2).
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
var asked []link.PullRef
named := []link.CheckedBuild{{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 7}, Check: "build-1",
Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}}
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = pulls
return named, nil
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}
if got := reuseWithVerdicts(t.Context(), &link.ReuseAsk{Pulls: pulls}); !reflect.DeepEqual(got.Builds, named) ||
!reflect.DeepEqual(asked, pulls) {
t.Fatalf("the leave names %+v, read for %+v", got.Builds, asked)
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), &link.ReuseAsk{Pulls: pulls}); got.Builds != nil {
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
+193 -25
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"path"
"regexp"
@@ -188,6 +189,8 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
if err := ask.Ask(ctx, request); err != nil {
return err
}
// A new head ends the check of the old one (novox/hq issue 726).
endingSupersededChecks(ctx, seat, p, request.ID, f.oneShot)
what := "its own merge-check.sh alone: " + noModuleTouched
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
@@ -248,31 +251,16 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil {
return link.BuildRequest{}, err
}
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it.
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return link.BuildRequest{}, err
}
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
beside[sibling] = link.CheckedOut{Repository: url, Ref: refs[sibling]}
}
}
}
}
// **A pull request into a release branch is judged by that branch** (novox/hq ADR 0358): asked of the forge once.
release, err := releaseBranchOf(ctx, p, scope, entries)
if err != nil {
return link.BuildRequest{}, err
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone, release)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
@@ -282,10 +270,188 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge, Builds: checkBuildsOf(p, scope, entries, clone)},
Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: namedForTheCheck(ctx, p, builds),
Release: release, Running: runningBy(ctx)},
}, nil
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs — or, for a pull request into a release branch (novox/hq ADR
// 0358 §3), at its branch of that name where it has one and at its trunk otherwise, the controller's main kept for
// its judge, and mesh-lab at its trunk whatever the base: its replays run with the container runtime's socket. running
// is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error), release string) (map[string]link.CheckedOut, error) {
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return nil, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
at := func(dir, trunk string) link.CheckedOut {
if release == "" {
return link.CheckedOut{Repository: url, Ref: refs[dir]}
}
if trunk == "" {
trunk = "main"
}
return link.CheckedOut{Repository: url, Ref: release, Else: trunk}
}
beside[dir] = at(dir, followedBranch(e.Source.Ref))
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if release != "" {
// mesh-controller has the release branch: that is what makes it one (ADR 0358 §1).
beside[dir] = link.CheckedOut{Repository: url, Ref: release}
}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
b := link.CheckedOut{Repository: url, Ref: refs[sibling]}
// **mesh-lab is read at its trunk, always** (ADR 0358): its replays run with the container
// runtime's socket, so they are only ever reviewed code — never a branch anyone may create.
if release != "" && sibling != "mesh-lab" {
b = link.CheckedOut{Repository: url, Ref: release, Else: "main"}
}
beside[sibling] = b
}
}
}
}
}
return beside, nil
}
// releaseBranchOf is the release branch a pull request merges into (novox/hq ADR 0358 §1): its base, when that is not
// its repository's trunk and mesh-controller has a branch of that name; empty otherwise, and the pull request is
// checked as before it. **A base that is a trunk asks nothing**: the branch a module built from the repository follows
// (main when it records none), or mesh-controller's own trunk. Any other base is asked of the forge once, and an
// answer it cannot give is an error, never read as "no release branch".
func releaseBranchOf(ctx context.Context, p link.PullUpdated, scope checkScope, entries []inventory.Entry) (string, error) {
base := p.Base
if base == "" {
return "", nil
}
if scope.From != nil && base == trunkOf(scope.From.Source) {
return "", nil
}
var controller *inventory.Entry
for i, e := range entries {
if e.Manifest.Module == "mesh-controller" && !e.Provided && e.Source.Repository != "" {
controller = &entries[i]
break
}
}
if controller == nil || base == trunkOf(controller.Source) {
return "", nil
}
if !tryRef.MatchString(base) || strings.Contains(base, "..") {
return "", nil // no branch the forge could hold: checked as before
}
owner, repo := sourceOwner(controller.Source.Repository), sourceName(controller.Source.Repository)
if owner == "" || repo == "" {
return "", nil
}
has, err := forgeHasBranch(ctx, owner, repo, base)
if err != nil {
return "", fmt.Errorf("whether %s/%s#%d merges into a release branch cannot be told — the forge did not say "+
"whether %s/%s has a branch %s (novox/hq ADR 0358): %w", p.Owner, p.Repo, p.Number, owner, repo, base, err)
}
if !has {
return "", nil
}
return base, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// forgeHasBranch asks the forge, through its module's tools on the bus, whether a repository has a branch. A
// variable so a test needs no forge.
var forgeHasBranch = func(ctx context.Context, owner, repo, branch string) (bool, error) {
js, err := aBus()
if err != nil {
return false, err
}
defer js.Close()
raw, _ := json.Marshal(map[string]any{"path": "/repos/" + owner + "/" + repo + "/branches/" + branch})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return false, err
}
return branchAnswered(answer, branch)
}
// branchAnswered reads the forge's answer about a branch: a 404 is "no such branch"; an answer naming the branch is
// "it has it"; anything else — another error, an unreadable answer, one naming no branch or another — is an error,
// never read as "no release branch".
func branchAnswered(answer link.Answer, branch string) (bool, error) {
if answer.Error != "" {
if strings.Contains(answer.Error, " 404 ") {
return false, nil
}
return false, errors.New(answer.Error)
}
var found struct {
Result struct {
Name string `json:"name"`
} `json:"result"`
}
if err := json.Unmarshal(answer.Result, &found); err != nil {
return false, fmt.Errorf("the forge's answer about branch %s cannot be read: %w", branch, err)
}
if found.Result.Name != branch {
return false, fmt.Errorf("the forge answered about branch %s naming %q", branch, found.Result.Name)
}
return true, nil
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
@@ -432,6 +598,8 @@ func checkedOf(result link.BuildResult) link.Checked {
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Release, c.JudgedBy, c.BesideAt = result.Check.Release, result.Check.JudgedBy, result.Check.BesideAt
c.Validator = result.Check.Validator
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
+1 -1
View File
@@ -952,7 +952,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked.
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open: open, oneShot: true}).PullUpdated(ctx, members[0].pull()) }); err != nil {
return err
}
return answer(map[string]any{"rechecked": members[0].ID})
+3
View File
@@ -22,6 +22,7 @@ func foundDirectory(module string, since time.Time) inventory.ResourceHealth {
}
func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
now := time.Now()
sent := now.Add(-time.Minute)
f := gateFacts{now: now, health: map[string]inventory.NodeHealth{"laptop": {Node: "laptop", HeardAt: now,
@@ -48,6 +49,7 @@ func TestADirectoryFoundBeforeTheSendIsAWaitForAPerson(t *testing.T) {
// The whole walk: a module whose directory was used as found long before still gets its fix to every machine,
// its pass kept and the wait said; a directory this very send found holds it and puts it back.
func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
for _, c := range []struct {
name string
found time.Duration // when the directory was found, against now
@@ -116,6 +118,7 @@ func TestAFixGoesThroughPastADirectoryFoundBefore(t *testing.T) {
// The condition says the wait in its own kind: the operator's, never urgent, and cleared once handed over.
func TestADirectoryUsedAsFoundIsItsOwnCondition(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
k, _ := withConditionsInMemory(t)
ctx := t.Context()
rs := map[string][]inventory.ResourceHealth{"notes": {foundDirectory("notes", time.Now().Add(-time.Hour))}}
+12
View File
@@ -1443,6 +1443,12 @@ func probeGates(ctx context.Context, d *doctor) ([]conditions.Observation, error
}
// A release held after a failed one, while builds still wait for a gate (ADR 0236).
out = append(out, backlogObservation()...)
// And a rest that went silent (novox/hq ADR 0353 §4).
silent, err := silentObservation(ctx, d.open.inventory)
if err != nil {
return nil, err
}
out = append(out, silent...)
return sortedFound(dedupeObservations(out)), nil
}
@@ -1532,6 +1538,12 @@ func gateLine(g *inventory.PlanGate) string {
case inventory.GateFailed:
what = "FAILED"
}
switch {
case g.Verdict == "" && g.Pending:
what = "kept, not yet sent" // novox/hq ADR 0353 §8
case g.Verdict == "" && g.Silent:
what = "silent on its send, left" // ADR 0353 §4
}
if g.Verdict == "" && g.Waiting() {
what = "waits for a busy container on " + g.Busy.Machine
}
@@ -57,6 +57,7 @@ func healthyContainer(module string) inventory.ResourceHealth {
// tools' witness put its build back at once, and the send waits for the healthy module's own verdict — a pass,
// counted over the gate's spacing — before it says its own. The broken one alone is put back.
func TestAFaultBeforeTheSettleTimeWaitsForTheModuleBesideIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
@@ -99,6 +100,7 @@ const broker_runtime = "node-tools"
// **A pass is counted only the gate's spacing after the one before** (review (c)): a send judged every tick
// while a module beside it is not yet healthy counts the healthy one once.
func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
withGateBounds(t, time.Hour, time.Hour, time.Hour)
@@ -123,6 +125,7 @@ func TestAPassIsCountedNoFasterThanTheGatesSpacing(t *testing.T) {
// **A carried build's pass carries its own wait, never another's** (review (e)).
func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
@@ -147,6 +150,7 @@ func TestACarriedPassCarriesOnlyItsOwnWait(t *testing.T) {
// carried its own module, healthy, and a build waiting on that machine that never became healthy. At the
// bound the waiting one is put back and marked; the plan's own module keeps its pass, so no walk waits on it.
func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
@@ -227,6 +231,7 @@ func TestThePlansOwnModuleKeepsItsPassWhenItsSendFails(t *testing.T) {
// **A provider waiting for a login says who waits on it** (review (g)): its consumer, failing a check that
// needs it, is held under it, and the provider's relogin-needed condition lists it and is urgent.
func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
@@ -289,6 +294,7 @@ func TestAProviderWaitingForALoginSaysWhoWaitsOnIt(t *testing.T) {
// tools are marked and put back in the very judging that found them broken — their registered build is the
// one before, and the laptop is sent it — while `app` goes on being judged, recovers, and keeps its own pass.
func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
open := aMesh(t)
ctx := t.Context()
@@ -431,6 +437,7 @@ func TestABrokenModuleIsPutBackAtOnceAndTheOneBesideItGetsItsOwnVerdict(t *testi
// **A move from a build not known excuses no wait** (issue 318 review): a plan's own module whose previous
// build was not recorded, or whose previous manifest is not kept, cannot be shown to have added the group.
func TestAMoveFromAnUnknownBuildExcusesNoWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
+32
View File
@@ -36,6 +36,10 @@ func aGateMesh(t *testing.T) *gateMesh {
ctx := t.Context()
inv := open.inventory
g := &gateMesh{open: open, health: map[string]health{}}
// The gate's own tests judge a first machine, as walks a controller before novox/hq ADR 0360 started still are.
wasGated := gateFirstMachine
gateFirstMachine = true
t.Cleanup(func() { gateFirstMachine = wasGated })
g.keeper, _ = withConditionsInMemory(t)
g.told = &conditions.Told{}
was := doctorFrom
@@ -271,6 +275,34 @@ func TestABuildThatPassesItsGateRollsEverywhereUnattended(t *testing.T) {
}
}
// A build rolls everywhere with no hand and no gate (novox/hq ADR 0360 §1): both machines in one send, no health
// judged, and the walk done once both reported on the send. Before, anchor was judged first, then laptop sent.
func TestABuildRollsToEveryMachineAtOnceWithNoGate(t *testing.T) {
g := aGateMesh(t)
gateFirstMachine = false
ctx := t.Context()
judged := false
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
judged = true
return wasGather(ctx, open, component)
}
t.Cleanup(func() { gatherGateFacts = wasGather })
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p := g.plan(t)
if !reflect.DeepEqual(g.sent, [][]string{{"anchor", "laptop"}}) {
t.Fatalf("sent %v, not both machines in one send", g.sent)
}
if s := p.Modules["app"]; p.State != inventory.PlanDone || s.Gate != nil || s.FirstAt != nil || judged {
t.Fatalf("the plan is %s (%s), app %+v, judged %v", p.State, p.Note, s, judged)
}
if obs, err := probeGates(ctx, doctorFrom); err != nil || len(obs) != 0 {
t.Fatalf("a rolled build left a condition: %+v %v", obs, err)
}
}
// A leaf module passes its gate on one reading, as soon as its first machine is healthy, without waiting out the
// core's settle time (novox/hq ADR 0282 decision 4 as ADR 0304 decision 3 changed it, issue 524).
func TestALeafModulePassesOnOneHealthyReading(t *testing.T) {
+1 -1
View File
@@ -151,7 +151,7 @@ func heldMachines(ctx context.Context, open *stores, names []string) (map[string
for _, mv := range f.moves(node, modules, same, known, false) {
if planStillToSend(plans, mv.Module, node) == "" {
why = append(why, fmt.Sprintf("%s would move from %s to %s, which has passed no gate yet — a release "+
"plan sends it, one machine at a time, judged", mv.Module, buildName(mv.From), buildName(mv.To)))
"plan sends it, one machine first and then the rest at once, judged", mv.Module, buildName(mv.From), buildName(mv.To)))
}
}
if len(why) > 0 {
+8
View File
@@ -25,6 +25,7 @@ import (
// another send reaches anchor, unreported; the gate still passes. And a send that moves a judged module
// to another build supersedes the judging: no verdict, nothing put back.
func TestReplay352(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
t.Run("a newer send to the judged machine does not unreport the gate's", testANewerSendDoesNotUnreportTheGatesOwn)
t.Run("a send that moves the module supersedes the judging", testASendThatMovesTheModuleSupersedesTheJudging)
@@ -67,6 +68,7 @@ func testANewerSendDoesNotUnreportTheGatesOwn(t *testing.T) {
// A plan's own first send waits while a release judges the same module on that machine with another build.
func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
@@ -82,6 +84,7 @@ func TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere(t *testing.T) {
// A merge plan's judging is superseded the same way: another send moved its module on the first machine.
func TestAPlansJudgingIsSupersededByASendThatMovesItsModule(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
@@ -140,6 +143,7 @@ func testASendThatMovesTheModuleSupersedesTheJudging(t *testing.T) {
// A report is on the gate's own send: the declaration itself, or one sequenced after it; a gate kept
// without its send reads the report against the send made last, as before. Pure.
func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
sent := inventory.SentDeclaration{Digest: "d-490", Sequence: 490}
for _, c := range []struct {
r inventory.Reported
@@ -185,6 +189,7 @@ func TestAReportIsHeldAgainstTheGatesOwnSend(t *testing.T) {
// A gate judges only the build the machine was last sent: last sent another build of the module, the
// judging is superseded, whatever the machine reports. Pure.
func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
at := time.Now()
f := gateFacts{now: at, reports: map[string]inventory.Reported{"anchor": {Node: "anchor", Outcome: inventory.OutcomeApplied,
At: &at, Current: true}}, engines: map[string]string{}, served: map[string]served{}, rolledBack: map[string][]lease.Rollback{},
@@ -212,6 +217,7 @@ func TestAGateJudgesOnlyTheBuildTheMachineWasLastSent(t *testing.T) {
// A move of another build of a module to a machine where a release or a plan is judging that module
// waits for that judging; the same build to that machine is already there. Pure.
func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
at := time.Now()
release := inventory.Plan{ID: "release-1", State: inventory.PlanRolling, Release: &inventory.PlanRelease{
Gate: &inventory.PlanGate{Machines: []string{"novox"}, Carried: []inventory.CarriedMove{
@@ -249,6 +255,7 @@ func TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine(t *testing.T) {
// The controller is never put back to a build that reaches less of the store's schema than the store
// has, or to one that never said what it reaches: the current build is kept, and the condition says so.
func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
@@ -345,6 +352,7 @@ func TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema(t *testing.T)
// The lease's holder names the build the declaration told it it is, and the version stamp only without one.
func TestTheHolderNamesTheBuildTheDeclarationToldIt(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
t.Setenv(RunningBuildVar, " ad62528c47c7 ")
if h := holderOf("x"); h.Build != "ad62528c47c7" {
t.Fatalf("the holder's build is %q", h.Build)
+88
View File
@@ -0,0 +1,88 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A tier whose module is no move — made from the source every machine running it runs, so nothing is
// sent — is not waited on for a report after its marking (novox/hq issue 648). Its machines answered the
// send that last carried that build before the walk got there, and nothing will make them report again:
// the walk asks the next tier in the same step, rather than "waiting for app on anchor, laptop" for ever.
func TestANoMoveTierWaitsForNoNewReport(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
// app holds the build seat, so a module of the next tier is built by it; since novox/hq ADR 0360 §3 that gates
// nothing.
for _, m := range []catalogue.Manifest{
{Module: "app", Version: "c2", Claims: []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}},
{Module: "web", Version: "w1"},
} {
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + m.Module, BuiltFrom: m.Version, Head: m.Version, Asked: time.Now().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
// What anchor and laptop were last sent (c1) was made from the source the plan's build of c2 is.
for _, b := range []inventory.Build{
{ID: "build-1s", Module: "app", Commit: "c1", SourceFingerprint: "src1:app",
Asked: time.Now().Add(-30 * time.Second), At: time.Now().Add(-30 * time.Second)},
{ID: "build-2s", Module: "app", Commit: "c2", SourceFingerprint: "src1:app", Asked: time.Now(), At: time.Now()},
} {
b.Manifest, _ = json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
// Each machine answered that send, applied, before the walk's tier was built.
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.RecordDoing(ctx, nodeID(t, g.open, n), inventory.Doing{Node: n,
Outcome: inventory.OutcomeApplied, Declared: "d-" + n + "-c1", Applied: 1,
At: time.Now().Add(-10 * time.Second)}); err != nil {
t.Fatal(err)
}
}
built := time.Now().UTC()
p := g.plan(t)
p.Tiers = [][]string{{"app"}, {"web"}}
p.Modules["app"].Build, p.Modules["app"].BuiltAt = "build-2s", &built
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
if len(g.sent) != 0 {
t.Fatalf("a no-move build was sent: %v", g.sent)
}
p = g.plan(t)
if s := p.Modules["app"]; s == nil || !strings.HasPrefix(s.Why, "no move") {
t.Fatalf("the build was not marked no move: %+v", s)
}
if p.Tier < 1 || strings.Contains(p.Note, "waiting for app") {
t.Fatalf("the walk waits on a no-move tier for a report nothing will cause: tier %d, %s %q", p.Tier, p.State, p.Note)
}
// A machine that has not answered its last send is not waited on either: a built-by edge gates no tier
// (novox/hq ADR 0360 §3). Before, the walk waited for app on laptop.
p.Tier, p.State, p.Note = 0, inventory.PlanRolling, ""
if err := inv.RecordSent(ctx, nodeID(t, g.open, "laptop"), "d-laptop-later", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
p = g.plan(t)
if p.Tier < 1 || strings.Contains(p.Note, "waiting for app") {
t.Fatalf("the walk waits on a built-by edge: tier %d, %q", p.Tier, p.Note)
}
}
+188
View File
@@ -0,0 +1,188 @@
package main
import (
"context"
"slices"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq ADR 0360 §4, §5: a walk ends on its machines' reports — every machine sent has reported on its send,
// or is silent past ApplySilentAfter — and names a failed or refused apply; it never waits for a health reading.
func TestAWalkEndsOnItsMachinesReports(t *testing.T) {
now := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
sent := now.Add(-time.Minute)
long := now.Add(-inventory.ApplySilentAfter - time.Minute)
at := now.Add(-10 * time.Second)
p := inventory.Plan{Modules: map[string]*inventory.PlanModule{
"app": {Rest: map[string]inventory.SentDeclaration{
"ace": {Digest: "d-ace", At: sent},
"novox": {Digest: "d-novox", At: sent},
"shanks": {Digest: "d-shanks", At: sent},
"laptop": {Digest: "d-laptop", At: long},
}},
"proxy": {Rest: map[string]inventory.SentDeclaration{"ace": {Digest: "d-ace", At: sent}}},
}}
reports := []inventory.Reported{
{Node: "ace", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-ace"},
{Node: "novox", Outcome: inventory.OutcomeFailed, At: &at, Declared: "d-novox"},
{Node: "shanks", Outcome: inventory.OutcomeApplied, At: &at, Declared: "d-before"},
}
waiting, failed, silent := walkReports(p, reports, now)
if !slices.Equal(waiting, []string{"shanks"}) || !slices.Equal(failed, []string{"novox"}) ||
!slices.Equal(silent, []string{"laptop"}) {
t.Fatalf("waiting %v, failed %v, silent %v", waiting, failed, silent)
}
reports[2].Declared = "d-shanks"
if waiting, _, _ := walkReports(p, reports, now); len(waiting) != 0 {
t.Fatalf("the walk still waits once every machine reported: %v", waiting)
}
}
// withTheGate turns the first-machine gate on for a test of its own path: kept for the walks a controller before
// novox/hq ADR 0360 started.
func withTheGate(t *testing.T) {
t.Helper()
was := gateFirstMachine
gateFirstMachine = true
t.Cleanup(func() { gateFirstMachine = was })
}
// Every machine heard from is sent at once and passed on its applied report, with no judging (novox/hq ADR 0360 §7);
// before, the first machine was judged before the rest were sent (ADR 0353 §1).
func TestAReleaseSendsEveryMachineAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
said := events(t)
for i := 0; i < 12; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanDone || len(p.Release.Done) != 3 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
for _, n := range []string{"desk", "laptop", "anchor"} {
if !slices.Contains(*said, "send "+n) {
t.Fatalf("%s was not sent: %v", n, *said)
}
}
if slices.Contains(*said, "judge") {
t.Fatalf("a machine was judged by its health: %v", *said)
}
if v, found, err := b.open.inventory.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
}
// A machine that reports its apply failed is named and fails the release, with nothing put back, and every other
// machine goes on to its own report (novox/hq ADR 0360 §5, §7); before, it was put back (ADR 0353 §3).
func TestAMachineThatFailsItsApplyIsNamedAndNothingIsPutBack(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
inv := b.open.inventory
b.broken["desk"] = true // desk fails what it is sent
advancePlans(ctx, b.open)
for i := 0; i < 12 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Failed, []string{"desk"}) {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if !slices.Contains(p.Release.Done, "laptop") || !slices.Contains(p.Release.Done, "anchor") {
t.Fatalf("the others have no verdict of their own: %+v", p.Release)
}
if v, found, err := inv.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
if sent, _, _ := inv.SentBuilds(ctx, "desk"); sent["app"] != "c2" {
t.Fatalf("desk was put back: %v", sent)
}
}
// A release's send kept before it was made, and made, is judged by the controller that takes over without being
// made again (ADR 0353 §8), with every machine sent at once (ADR 0360 §7).
func TestAReleaseSendKeptAndMadeIsNotMadeAgainAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
p, err := releaseBacklog(ctx, b.open, "")
if err != nil || p == nil {
t.Fatalf("no release opened: %v", err)
}
if _, err := advanceRelease(ctx, b.open, p); err != nil || p.Release.Gates["anchor"] == nil || !p.Release.Gates["anchor"].Pending {
t.Fatalf("the send was not kept before it was made: %+v %v", p.Release, err)
}
if err := inv.SavePlan(ctx, p); err != nil {
t.Fatal(err)
}
// The send is made, and the controller is replaced before it keeps it.
if _, err := sendRollout(ctx, b.open, []string{"anchor"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if n := len(slices.DeleteFunc(slices.Clone(b.sent), func(s []string) bool { return !slices.Equal(s, []string{"anchor"}) })); n != 1 {
t.Fatalf("anchor was sent again: %v", b.sent)
}
g := b.release(t).Release.Gates["anchor"]
if g == nil || g.Pending || g.Since == nil || len(g.Carried) == 0 {
t.Fatalf("anchor is not judged on the send made: %+v", g)
}
}
// A machine of the rest that says nothing on its send is left silent after fifteen minutes: not failed, not put
// back; a rest that went silent whole ends failed, said as a condition (ADR 0353 §4); its step also sent anchor, so the release is done, not failed (ADR 0360 §7).
func TestAMachineSilentPastItsBoundIsLeftAndSaidAtOnce(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
reporting := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if slices.Equal(names, []string{"laptop"}) { // laptop sleeps: sent, and says nothing
b.sent = append(b.sent, names)
current, _ := open.inventory.CurrentBuilds(ctx)
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "laptop"), "d-laptop-asleep",
map[string]string{"app": current["app"].Commit, "late": current["late"].Commit, "same": current["same"].Commit})
}
return reporting(ctx, open, names)
}
t.Cleanup(func() { sendRollout = reporting })
for i := 0; i < 8; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
g := p.Release.Gates["laptop"]
if !p.Open() || g == nil || g.Verdict != "" || g.Silent {
t.Fatalf("laptop is judged before fifteen minutes: %s %s %+v", p.State, p.Note, g)
}
since := time.Now().Add(-inventory.ApplySilentAfter - time.Minute)
g.Since = &since
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
p = b.release(t)
// anchor applied in the same step, so the release is done, laptop named silent (novox/hq ADR 0360 §7).
if p.State != inventory.PlanDone || !slices.Equal(p.Release.Silent, []string{"laptop"}) || len(p.Release.Failed) > 0 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if failed, _ := inv.GateFailed(ctx, "build-late-c2"); failed {
t.Fatal("a silent machine's build was marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "laptop"); sent["late"] != "c2" {
t.Fatalf("laptop was put back: %v", sent)
}
obs, err := silentObservation(ctx, inv)
if err != nil {
t.Fatal(err)
}
if len(obs) != 1 || obs[0].Kind != "release-silent" || !slices.Equal(obs[0].Also, []string{"laptop"}) {
t.Fatalf("the silence is not said: %+v", obs)
}
}
+9
View File
@@ -106,6 +106,8 @@ func run() error {
return killCommand(ctx, args[1:])
case "forget-checks":
return forgetChecksCommand(ctx, args[1:])
case "try":
return tryCommand(ctx, args[1:])
case "pause", "resume":
return pauseCommand(ctx, args[0], args[1:])
case "collection":
@@ -381,6 +383,13 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// **A merge check builds nothing** (novox/hq to-be 45 §9): its verdict is said, and nothing of it is
// recorded or registered.
if result.Check != nil || result.Checked != nil {
if isTry(result) {
// A branch tried before its pull request (novox/hq issue 686): its asker reads the outcome from the
// build's log. Never said as `checked`, so the forge's holder sets no status.
fmt.Printf("%s: a try of %s at %.8s, read with `try --id`: said to nobody, nothing taken in\n", result.ID,
result.Repository, result.Ref)
return nil
}
checked(ctx, result)
return nil
}
+96 -3
View File
@@ -7,7 +7,10 @@ import (
"fmt"
"os/exec"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -123,8 +126,82 @@ func answerMeshCLI(inv *inventory.Inventory) link.CLIHandler {
// cliJournal says one line in the controller's journal (its standard output); a variable so a test can read it.
var cliJournal = func(line string) { fmt.Println(line) }
// runForMeshCLI runs a judged line and answers what it said. Every line is said in the journal first: its call,
// who asked on which node, its command word only, and how it runs (review of ADR 0272).
// onlyReads says a line only reads, judged by the generic `command` verb's reading forms (novox/hq ADR 0266) — the
// same allow list that decides what an ordinary call may run, never a second list kept here — judged on the line's
// every word, so `plans go`, `conditions silence` and `settings set` are not reads while `plans`, `conditions` and
// `settings show` are. A line those forms do not name is not judged a read, and is said (novox/hq issue 762).
func onlyReads(asked link.CLIAsked, v cliVerdict) bool {
if v.refused != "" || len(asked.Line) == 0 || len(asked.Stdin) > 0 || cliServers[asked.Line[0]] {
return false
}
return commandReads(asked.Line) == nil && terminalOnly(asked.Line) == nil
}
// readCalls counts the read calls said in no line, per caller, until the next hourly line says them.
type readCalls struct {
mu sync.Mutex
counts map[string]int
}
// cliReads are this controller's read calls not yet said.
var cliReads = &readCalls{}
func (r *readCalls) add(caller string) {
r.mu.Lock()
defer r.mu.Unlock()
if r.counts == nil {
r.counts = map[string]int{}
}
r.counts[caller]++
}
// said is the line that says the read calls counted since the last, and starts counting again; empty when there
// were none, so an idle hour says nothing.
func (r *readCalls) said(over time.Duration) string {
r.mu.Lock()
counts := r.counts
r.counts = nil
r.mu.Unlock()
if len(counts) == 0 {
return ""
}
callers := make([]string, 0, len(counts))
for c := range counts {
callers = append(callers, c)
}
sort.Strings(callers)
parts := make([]string, len(callers))
for i, c := range callers {
parts[i] = fmt.Sprintf("%s %d", c, counts[c])
}
return fmt.Sprintf("mesh-cli read calls in the last %s, said in no line of their own: %s", over,
strings.Join(parts, ", "))
}
// readCallsEvery is how often the count of read calls is said.
const readCallsEvery = time.Hour
// sayReadCalls says, every readCallsEvery until ctx ends, how many read calls each caller made — when any did — so
// a runaway poller stays visible while its calls write no line each (novox/hq issue 762).
func sayReadCalls(ctx context.Context) {
tick := time.NewTicker(readCallsEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
if line := cliReads.said(readCallsEvery); line != "" {
cliJournal(line)
}
}
}
}
// runForMeshCLI runs a judged line and answers what it said. Every line that changes something, is refused or fails
// is said in the journal: its call, who asked on which node, its command word only, and how it runs (review of ADR
// 0272). A line that only reads and succeeds is counted instead, and the count said hourly (novox/hq issue 762): a
// page polling `plans` every few seconds would otherwise write thousands of lines a day.
func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer {
how := "as an ordinary call"
switch {
@@ -137,7 +214,23 @@ func runForMeshCLI(ctx context.Context, node string, asked link.CLIAsked, v cliV
if call == "" {
call = "(no call)"
}
cliJournal(fmt.Sprintf("mesh-cli %s: %s on %s asked %q, %s", call, asked.Account, node, asked.Line[0], how))
line := fmt.Sprintf("mesh-cli %s: %s on %s asked %q, %s", call, asked.Account, node, asked.Line[0], how)
if !onlyReads(asked, v) {
cliJournal(line)
return runJudgedLine(ctx, node, asked, v)
}
answer := runJudgedLine(ctx, node, asked, v)
if answer.Exit != 0 || answer.Refused != "" {
// A read that was refused or failed is said, as a change is.
cliJournal(fmt.Sprintf("%s, and it failed (exit %d)", line, answer.Exit))
} else {
cliReads.add(fmt.Sprintf("%s on %s", asked.Account, node))
}
return answer
}
// runJudgedLine runs a line runForMeshCLI has said or counted.
func runJudgedLine(ctx context.Context, node string, asked link.CLIAsked, v cliVerdict) link.CLIAnswer {
if v.refused != "" {
return link.CLIRefusal(v.refused)
}
+123
View File
@@ -0,0 +1,123 @@
package main
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// echoFails makes the echoing test binary end with a failure, as a read that failed would.
const echoFails = "MESH_TEST_ECHO_FAILS"
// journalled runs lines through runForMeshCLI and answers what the journal said, with a fresh count of read calls.
func journalled(t *testing.T) *[]string {
t.Helper()
t.Setenv(echoEnvironment, "1")
said := &[]string{}
was, wasReads := cliJournal, cliReads
cliJournal = func(line string) { *said = append(*said, line) }
cliReads = &readCalls{}
t.Cleanup(func() { cliJournal, cliReads = was, wasReads })
return said
}
// **A call that only reads writes no journal line; one that changes something does** (novox/hq issue 762).
func TestAReadCallWritesNoJournalLine(t *testing.T) {
said := journalled(t)
ctx := link.WithCallID(context.Background(), "call-1")
ordinary := cliVerdict{why: "not the terminal"}
for _, line := range [][]string{{"plans"}, {"status"}, {"conditions"}, {"node", "list"}, {"builds"}, {"queue"},
{"settings", "show", "zsh"}, {"plans", "--json"}} {
if a := runForMeshCLI(ctx, "g14", cliAsked("jochen", 1000, line...), ordinary); a.Exit != 0 || a.Refused != "" {
t.Fatalf("%q did not run: %+v", line, a)
}
}
runForMeshCLI(ctx, "control", cliAsked("jochen", 1000, "plans"), cliVerdict{terminal: true, why: "the terminal"})
if len(*said) != 0 {
t.Fatalf("read calls wrote journal lines: %q", *said)
}
runForMeshCLI(ctx, "control", cliAsked("jochen", 1000, "assign", "g14", "zsh"), cliVerdict{terminal: true, why: "the terminal"})
if len(*said) != 1 || !strings.Contains((*said)[0], `asked "assign"`) {
t.Fatalf("a call that changes something was not said: %q", *said)
}
}
// **A refused or failed read keeps its line** (novox/hq issue 762).
func TestAFailedOrRefusedReadIsSaid(t *testing.T) {
said := journalled(t)
runForMeshCLI(context.Background(), "g14", cliAsked("root", 0, "plans"), cliVerdict{refused: "root is not answered"})
t.Setenv(echoFails, "1")
a := runForMeshCLI(context.Background(), "g14", cliAsked("jochen", 1000, "plans"), cliVerdict{why: "not the terminal"})
if a.Exit == 0 {
t.Fatalf("the read did not fail: %+v", a)
}
if len(*said) != 2 || !strings.Contains((*said)[0], "refused") || !strings.Contains((*said)[1], "failed (exit 3)") {
t.Fatalf("the journal said %q", *said)
}
if line := cliReads.said(readCallsEvery); line != "" {
t.Fatalf("a failed read was counted as a read: %q", line)
}
}
// **A verb is classed by what the call does, not its name** (novox/hq issue 762): the same command word reads with
// some arguments and writes with others, and a line no reading form names is said.
func TestAVerbIsClassedByItsArguments(t *testing.T) {
ordinary := cliVerdict{why: "not the terminal"}
for _, c := range []struct {
line []string
reads bool
}{
{[]string{"settings", "show", "zsh"}, true},
{[]string{"settings", "preferences"}, true},
{[]string{"settings", "set", "zsh", `{"a":1}`}, false},
{[]string{"settings", "clear", "zsh"}, false},
{[]string{"plans"}, true},
{[]string{"plans", "p-1"}, true},
{[]string{"plans", "go", "p-1"}, false},
{[]string{"plans", "--json", "stop", "p-1"}, false},
{[]string{"plans", "close", "p-1"}, false},
{[]string{"plans", "retry", "p-1"}, false},
{[]string{"conditions"}, true},
{[]string{"conditions", "show", "x"}, true},
{[]string{"conditions", "silence", "x"}, false},
{[]string{"doctor", "run"}, false},
{[]string{"node", "account", "g14", "x"}, false},
{[]string{"assign", "g14", "zsh"}, false},
{[]string{"a-command-added-later"}, false},
{[]string{"serve"}, false},
} {
if got := onlyReads(cliAsked("jochen", 1000, c.line...), ordinary); got != c.reads {
t.Errorf("%q judged a read: %v, want %v", c.line, got, c.reads)
}
}
if onlyReads(cliAsked("jochen", 1000, "plans"), cliVerdict{refused: "no"}) {
t.Error("a refused line judged a read")
}
stdin := cliAsked("jochen", 1000, "status")
stdin.Stdin = []byte("x")
if onlyReads(stdin, cliVerdict{terminal: true}) {
t.Error("a line carrying standard input judged a read")
}
}
// **Once an hour the read calls are counted per caller, only if there were any** (novox/hq issue 762).
func TestTheReadCallsAreCountedPerCaller(t *testing.T) {
said := journalled(t)
ordinary := cliVerdict{why: "not the terminal"}
for range 3 {
runForMeshCLI(context.Background(), "g14", cliAsked("jochen", 1000, "plans"), ordinary)
}
runForMeshCLI(context.Background(), "ace", cliAsked("jochen", 1000, "status"), ordinary)
if len(*said) != 0 {
t.Fatalf("read calls wrote lines: %q", *said)
}
line := cliReads.said(readCallsEvery)
if !strings.Contains(line, "jochen on ace 1, jochen on g14 3") || !strings.Contains(line, "1h0m0s") {
t.Fatalf("the hourly line said %q", line)
}
if again := cliReads.said(readCallsEvery); again != "" {
t.Fatalf("an hour with no read calls said %q", again)
}
}
+1 -1
View File
@@ -86,7 +86,7 @@ func owedToTheRecord(ctx context.Context, announced merges, m link.SourceMoved,
// catchingUpOnMerges reads back the forge's announcements on a timer, until the context ends.
func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
f := following{open}
f := following{open: open}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
+15
View File
@@ -240,6 +240,7 @@ func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditi
continue
}
o := moduleUnhealthyObservation(m, node, unhealthy[m])
declaredSeverity(&o, hold, m)
if hold != nil {
if by, held := hold.heldWith(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits. The
@@ -347,6 +348,7 @@ func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p cata
} else {
_, rest := besideAWait(p.Module, rs)
o = moduleUnhealthyObservation(p.Module, p.Node, rest)
declaredSeverity(&o, hold, p.Module)
if said, waits := personWait(p.Module, p.Node, rest); waits {
o = reloginObservation(p.Module, p.Node, said, operatorOn(ctx, hold.inv, p.Node), rest)
}
@@ -481,6 +483,19 @@ func sayWaitingOn(o *conditions.Observation, waiters []string) {
o.Explanation += fmt.Sprintf(" %d module(s) that depend on it wait for it.", len(waiters))
}
// declaredSeverity raises a module's unhealthy condition urgent when its manifest declares `unhealthy` urgent
// (novox/hq ADR 0355 §1). Only that condition changes (ADR 0355 §4): a wait for a new login, a module that needs
// the operator and a directory used as found keep their severities, and without the field, or without a manifest
// to read it from, it stays the warning it was.
func declaredSeverity(o *conditions.Observation, hold *holding, module string) {
if o.Kind != kindModuleUnhealthy || hold == nil {
return
}
if m, ok := hold.manifestOf(module); ok && m.UnhealthyIsUrgent() {
o.Severity = conditions.Urgent
}
}
// moduleUnhealthyObservation is a module unhealthy on a machine, in words: the summary names the module,
// the machine and what is wrong with each resource; the detail — targets, streaks, since — is evidence.
func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHealth) conditions.Observation {
+7
View File
@@ -30,6 +30,7 @@ func userUnit(module, account string) inventory.ResourceHealth {
// Only what depends on the new login alone is a wait; anything else unhealthy of the module is judged as
// before, so that a fault is never excused.
func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
notInGroup := relogin("lights", "operator")
notInGroup.Reason = "not in the group lights: the apply has not put operator there; its outcome says why"
systemUnit := userUnit("lights", "operator")
@@ -74,6 +75,7 @@ func TestOnlyWhatDependsOnTheNewLoginIsAWait(t *testing.T) {
// The gate reads a wait for a person as its own reading, never a fault, and passes with it; the same module
// with a container down beside it is not yet healthy, as before.
func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
now := time.Now()
since := now.Add(-time.Minute)
account := relogin("lights", "operator")
@@ -102,6 +104,7 @@ func TestAWaitForAPersonIsAPassCarriedAlong(t *testing.T) {
// The module's condition says the wait in one sentence for a person, and clears on the first statement that
// no longer says it — said as the module's own fault when its unit still fails after the new login.
func TestTheReloginConditionSaysTheWaitAndClearsAfterTheLogin(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
k, _ := withConditionsInMemory(t)
@@ -158,6 +161,7 @@ func TestTheReloginConditionSaysTheWaitAndClearsAfterTheLogin(t *testing.T) {
// A module that is healthy on its own keeps its pass when another module of the same send fails at the
// bound: it is neither put back nor left without a verdict for every other walk to wait on.
func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
@@ -207,6 +211,7 @@ func TestAModuleHealthyOnItsOwnKeepsItsPassWhenItsSendFails(t *testing.T) {
// What the operator reads of the wait (ADR 0253, ADR 0254): it needs them, so it is never quiet, and it offers
// no button, since nothing but their own new login can do it.
func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
o := reloginObservation("openrazer", "g14", "relogin needed on g14: …", "operator", []inventory.ResourceHealth{
relogin("openrazer", "operator"), userUnit("openrazer", "operator")})
plainExample(t, o, "openrazer waits for a new login on g14",
@@ -238,6 +243,7 @@ func TestTheReloginConditionNeedsTheOperatorAndOffersNoButton(t *testing.T) {
// When a module not working turns into one waiting for a new login, and back, the clearing line says what it
// became, never that it works again (issue 318 review).
func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
store := conditions.NewInMemory()
@@ -286,6 +292,7 @@ func TestAConditionThatBecameTheOtherKindSaysSoWhenItClears(t *testing.T) {
// Which moves add an account group, read from the builds' manifests (issue 318 review).
func TestOnlyAMoveThatAddsAnAccountGroupCanBringAWait(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
user := func(groups ...any) catalogue.Manifest {
return catalogue.Manifest{Module: "lights", Resources: []map[string]any{{"id": "operator", "type": "user",
"name": "operator", "groups": groups}}}
+28
View File
@@ -456,6 +456,9 @@ var plainWordings = map[string]func(conditions.Observation) words{
"release-held": worded(func(o conditions.Observation) words {
return releaseHeldWords(nil, o.Also)
}),
"release-silent": worded(func(o conditions.Observation) words {
return releaseSilentWords(o.Also)
}),
"facts-stale": worded(func(o conditions.Observation) words {
return words{Headline: "Merge checks use outdated facts",
Explanation: "The facts a merge check judges a change against are out of date; they are gathered again " +
@@ -617,6 +620,18 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The bus reports a listener too slow to keep up, so messages to it are late.",
Resolved: "The listener keeps up again"}
}),
kindBusFaultsUnread: worded(func(o conditions.Observation) words {
return words{Headline: "Slow clients of the bus cannot be checked",
Explanation: "The controller could not read what the bus said about its clients' connections, so a " +
"client the bus cut off would go unnoticed. It tries again at every check.",
Resolved: "Resolved: the bus's clients are checked again"}
}),
"connection-closed": worded(func(o conditions.Observation) words {
return words{Headline: "The bus cut a client off",
Explanation: "The bus closed a client's connection because it broke the bus's rules, so what it was " +
"sending was lost.",
Resolved: "Resolved: the client keeps to the bus's rules"}
}),
kindBusReconnects: worded(func(o conditions.Observation) words {
return words{Headline: "A client keeps losing the bus",
Explanation: "One of the mesh's clients lost its connection to the bus again and again in the last hour. " +
@@ -841,6 +856,19 @@ func releaseHeldWords(modules, machines []string) words {
Resolved: "Resolved: the held updates are released"}
}
// releaseSilentWords are the plain words of machines that said nothing on an update they were sent (ADR 0353 §4).
func releaseSilentWords(machines []string) words {
where := "Some machines"
if len(machines) > 0 {
where = conditions.Capital(namesWords(machines, 4))
}
return words{Headline: "Machines did not answer an update",
Needs: "check that they are awake and on the mesh; the next walk sends them what waits.",
Explanation: where + " were sent module updates and said nothing about them within fifteen minutes, so " +
"they were left as they were. What they run is not known to be the update.",
Resolved: "Resolved: the next walk reached them"}
}
// reloginNeeds is what an account waiting for its groups needs (ADR 0252).
func reloginNeeds(node string) string {
return fmt.Sprintf("log out of %s completely and log in again, or restart it.", node)
+7
View File
@@ -372,6 +372,13 @@ func joinAPlan(ctx context.Context, open *stores, module string) (bool, string,
if !found {
return false, "", nil
}
// **A walk that waits for its delivery's word asks nothing** (novox/hq ADR 0239, issue 754): a rebuild does not
// join it, and its outcome is not the walk's. The walk asks its own tier once it has its word.
if p.Waiting() {
return false, fmt.Sprintf("not joining %s (%s at %s): it waits for %s's word, so this build is not its "+
"outcome; once let go, the walk asks %s itself", p.ID, p.Repository, short(p.Commit), p.Delivery.Awaits,
module), nil
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return false, "", err
+1 -1
View File
@@ -133,7 +133,7 @@ func TestASharedRepositoryIsPlannedFromTheRecordedBuildSources(t *testing.T) {
{"the route proxy's program: the route proxy alone", "mesh-controller",
[]string{"examples/route-proxy/main.go"}, "route-proxy", true},
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, "mesh-controller | build-agent | route-proxy", false},
[]string{"internal/broker/broker.go"}, "mesh-controller,route-proxy | build-agent", false},
{"the route proxy's directory in the catalogue: it alone", "mesh-catalog",
[]string{"modules/route-proxy/module.json"}, "route-proxy", false},
{"the build agent's directory: it alone, nothing it builds", "mesh-catalog",
+11 -9
View File
@@ -145,8 +145,9 @@ func TestAPlanIsWhatTheChangeTouchedAndWhatIsBuiltOnIt(t *testing.T) {
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"},
{what: "built-by never widens", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "built-by orders", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a | x"},
// Whichever builder runs builds it (novox/hq ADR 0360 §3): built-by orders nothing, as before it ordered.
{what: "built-by does not order", edges: []inventory.Edge{dep("x", builtBy, "a")},
repo: "one", paths: []string{"modules/a/x", "modules/x/y"}, want: "a,x"},
{what: "worker-of never widens", edges: []inventory.Edge{dep("x", workerOf, "a")},
repo: "one", paths: []string{"modules/a/x"}, want: "a"},
{what: "worker-of orders", edges: []inventory.Edge{dep("x", workerOf, "a")},
@@ -296,18 +297,18 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
[]string{"internal/inventory/migrations/0088-a-build-says-its-build-source.sql"}, read,
"mesh-controller | build-agent", ""},
// A package all three build from: all three; the build seat's holder after the controller whose worker
// it binds (worker-of, issue 206), the proxy after the holder that builds it (built-by).
// it binds (worker-of, issue 206), the proxy beside the controller: built-by orders nothing (ADR 0360 §3).
{"a package all three build from: all three", "mesh-controller",
[]string{"internal/broker/broker.go"}, read, "mesh-controller | build-agent | route-proxy", ""},
[]string{"internal/broker/broker.go"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a vendored package all three build from: all three", "mesh-controller",
[]string{"vendor/github.com/nats-io/nats.go/nats.go"}, read, "mesh-controller | build-agent | route-proxy", ""},
[]string{"vendor/github.com/nats-io/nats.go/nats.go"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"go.sum: all three", "mesh-controller",
[]string{"go.sum"}, read, "mesh-controller | build-agent | route-proxy", ""},
[]string{"go.sum"}, read, "mesh-controller,route-proxy | build-agent", ""},
{"a file added to the proxy's package: the proxy", "mesh-controller",
[]string{"examples/route-proxy/new.go"}, read, "route-proxy", ""},
// Before any build source is said: as before.
{"no build source said: a README moves all three, as before", "mesh-controller",
[]string{"README.md"}, unsaid, "mesh-controller | build-agent | route-proxy", ""},
[]string{"README.md"}, unsaid, "mesh-controller,route-proxy | build-agent", ""},
// In the catalogue, where they live, each by its own build source.
{"the route proxy's recipe: it alone", "mesh-catalog",
[]string{"modules/route-proxy/Dockerfile"}, read, "route-proxy", ""},
@@ -331,7 +332,7 @@ func TestASharedRepositoryMovesOnlyWhatItsBuildSourceHolds(t *testing.T) {
// Files not all said: everything the repository builds, as before.
m := link.SourceMoved{Owner: "novox", Repo: "mesh-controller", Base: "main", Commit: "head",
Paths: []string{"README.md"}, PathsTruncated: true}
if got := tiered(reachOfMerge(m, entries, read, edges).Plan.Tiers); got != "mesh-controller | build-agent | route-proxy" {
if got := tiered(reachOfMerge(m, entries, read, edges).Plan.Tiers); got != "mesh-controller,route-proxy | build-agent" {
t.Errorf("a merge whose files were not all said: planned %q, wanted all three", got)
}
}
@@ -452,8 +453,9 @@ func TestAPlanIsTheTouchedModulesAndWhatIsReachableAlongTheWideningEdges(t *test
kinds := []string{inventory.EdgeStandsOn, inventory.EdgeDeclared, inventory.EdgePackages,
inventory.EdgeBuiltBy, inventory.EdgeWorkerOf}
widens := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true}
// Built-by orders nothing since novox/hq ADR 0360 §3.
orders := map[string]bool{inventory.EdgeStandsOn: true, inventory.EdgeDeclared: true,
inventory.EdgeBuiltBy: true, inventory.EdgeWorkerOf: true}
inventory.EdgeWorkerOf: true}
// Directory names drawn from one pool, so two repositories hold directories of the same name, and one
// is a prefix of another.
dirs := []string{"a", "ab", "b", "c", "lib/x", "lib/y", "modules/a", "modules/a/sub"}
+88 -3
View File
@@ -17,6 +17,7 @@ import (
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
@@ -194,7 +195,7 @@ func serve(ctx context.Context) (err error) {
// And what the catalogue decided a build meant. The builder's own result is already handled
// above; this is the other half — the control plane is the only one of the three that knows
// which machines run the thing, so it is the one that acts (novox/hq ADR 0072).
if err := server.Follows(following{open}); err != nil {
if err := server.Follows(following{open: open}); err != nil {
return err
}
// And the merges the bus announced and never handed over, read back on a timer and acted on late
@@ -202,7 +203,7 @@ func serve(ctx context.Context) (err error) {
go catchingUpOnMerges(ctx, open, server)
// And a catalogue that has just started, asking for what it missed. The same type answers
// both: what a build meant and what the builds were are two questions about one record.
if err := server.Answers(following{open}); err != nil {
if err := server.Answers(following{open: open}); err != nil {
return err
}
// And what providers say about consumers they keep failing, kept for `status` (novox/hq ADR
@@ -256,7 +257,7 @@ func serve(ctx context.Context) (err error) {
go forgetChecksOfWalk(context.WithoutCancel(ctx), open.inventory, p)
}
}
if err := server.Checks(following{open}); err != nil {
if err := server.Checks(following{open: open}); err != nil {
return err
}
// Composed now and kept current, before the verb that answers from it is served.
@@ -296,6 +297,8 @@ func serve(ctx context.Context) (err error) {
return err
}
defer stopCLI()
// Its read calls write no line each; how many each caller made is said once an hour (novox/hq issue 762).
go sayReadCalls(ctx)
// And says so on the bus (novox/hq ADR 0197): what it serves, as the NATS services protocol asks.
stopAnnouncing, err := bus.Announce(seatAnnouncement(handlers), log.New(os.Stdout, "", log.LstdFlags))
if err != nil {
@@ -1386,6 +1389,13 @@ func issueMemberships(ctx context.Context, open *stores, server *link.Server, se
if issued > 0 {
fmt.Printf(" issued %d membership(s)\n", issued)
}
// **And what every other machine hears of a module's stream** (novox/hq ADR 0363 §6): a grant the provider's
// setting changed reaches a consumer on a machine this send does not reach. Its membership is issued again
// as last issued, with only what it hears changed, so nothing else of it moves before its own send.
reheard := reissueHearing(ctx, bus, records, where, sent)
if reheard > 0 {
fmt.Printf(" issued %d membership(s) again for what they hear of a module's stream\n", reheard)
}
if len(refused) > 0 {
// Returned, never passed over (novox/hq issue 249): the declarations of the machines they are
// for are not sent, and the rollout that asked is tried again rather than waiting for a push.
@@ -1744,3 +1754,78 @@ func schemaReach() (int, error) {
}
return reach, nil
}
// reissueHearing issues again, on every machine not just sent, the membership of each assignment whose
// `consumes` names a module owning a stream, when what it hears changed: read as last issued from the bus,
// its `hears` replaced, and published. One that was never issued is left to its own send. Failures are said
// and passed over: the send that carries the grant is not the consumer's, and its own next send issues it.
func reissueHearing(ctx context.Context, bus link.OverNATS, records broker.Records, where broker.Placements,
sent []readyNode) int {
reached := map[string]bool{}
for _, s := range sent {
reached[s.node] = true
}
issued := 0
for node, declared := range records.Assigned {
if reached[node] {
continue
}
for _, d := range declared {
if !d.HearsAStream {
continue
}
subject := broker.MembershipSubject(node, d.Module)
last, err := bus.JS.GetLastMsg(broker.AssignmentsStream, subject)
if err != nil {
if !errors.Is(err, nats.ErrMsgNotFound) {
fmt.Printf(" %s on %s: its membership could not be read to issue what it hears again: %v\n",
d.Module, node, err)
}
continue
}
body, changed, err := withHearing(last.Data, broker.HeardIssuedFor(node, d))
if err != nil {
fmt.Printf(" %s on %s: its membership could not be read to issue what it hears again: %v\n",
d.Module, node, err)
continue
}
if !changed {
continue
}
if err := bus.PublishMembership(ctx, node, d.Module, body); err != nil {
fmt.Printf(" %s on %s: what it hears could not be issued again: %v\n", d.Module, node, err)
continue
}
issued++
}
}
return issued
}
// withHearing is a membership as issued with its `hears` replaced, and whether that changed it. Pure.
func withHearing(issued []byte, hears []broker.HeardIssued) ([]byte, bool, error) {
var fields map[string]json.RawMessage
if err := json.Unmarshal(issued, &fields); err != nil {
return nil, false, err
}
var had []broker.HeardIssued
if raw, ok := fields["hears"]; ok {
if err := json.Unmarshal(raw, &had); err != nil {
return nil, false, err
}
}
if slices.Equal(had, hears) {
return issued, false, nil
}
if len(hears) == 0 {
delete(fields, "hears")
} else {
raw, err := json.Marshal(hears)
if err != nil {
return nil, false, err
}
fields["hears"] = raw
}
body, err := json.Marshal(fields)
return body, true, err
}
+12 -2
View File
@@ -340,7 +340,10 @@ func cancelAsk(ctx context.Context, js *broker.JetStream, open *stores, seat str
return "", fmt.Errorf("%s is marked cancelled and could not be deleted from the queue (seq %d): %w — a "+
"holder taking it ends it as cancelled", ask.ID, ask.Seq, err)
}
recordCancelled(ctx, open, ask)
// A check ended because its pull request has a newer head records nothing (novox/hq issue 726): open is nil.
if open != nil {
recordCancelled(ctx, open, ask)
}
return cancelSaid(ask, seat, removed), nil
}
@@ -518,7 +521,9 @@ func rebuildCommand(ctx context.Context, args []string) error {
}
// **A module a plan holds unbuilt, or failed, joins that plan** rather than running beside it: the
// plan would ask it again, or stay failed on an outcome a rebuild has replaced.
// plan would ask it again, or stay failed on an outcome a rebuild has replaced. A walk waiting for its word
// is not joined (novox/hq issue 754): the rebuild runs beside it, and says so.
var beside string
if module != "" {
joined, said, err := joinAPlan(ctx, open, module)
if err != nil {
@@ -528,6 +533,7 @@ func rebuildCommand(ctx context.Context, args []string) error {
fmt.Println(said)
return nil
}
beside = said
}
id, err := askABuild(ctx, source, path, ref)
if err != nil {
@@ -535,6 +541,10 @@ func rebuildCommand(ctx context.Context, args []string) error {
}
recordAsked(ctx, inventory.BuildRequest{ID: id, Repository: source.Repository, Seat: source.Seat, Path: path,
Ref: ref, For: "rebuild"})
if beside != "" {
fmt.Printf("rebuild asked as %s, beside the walk: %s\n", id, beside)
return nil
}
fmt.Printf("rebuild asked as %s\n", id)
return nil
}
+4 -2
View File
@@ -616,6 +616,7 @@ func TestKillAndPauseAskTheHolderOnTheMachine(t *testing.T) {
// A plan that stopped at its first machine is retried: the module sent to that machine again, the
// send recorded as the first anew, and the plan goes on — unless a newer plan holds the module.
func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
@@ -636,8 +637,9 @@ func TestAPlanStoppedAtItsFirstMachineIsRetried(t *testing.T) {
stopped := inventory.Plan{ID: "plan-rollout", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: long, State: inventory.PlanFailed, Tiers: [][]string{{"a"}, {"b"}},
Note: "a stopped at its first machine in tier 0: laptop refused what it was sent",
Modules: map[string]*inventory.PlanModule{"a": {State: "built", BuiltAt: &long, Commit: "c0ffee",
First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}}
// Asked by the plan, as every module a plan holds built is (novox/hq issue 754).
Modules: map[string]*inventory.PlanModule{"a": {State: "built", AskedAt: &long, Build: "build-a", BuiltAt: &long,
Commit: "c0ffee", First: []string{"laptop"}, FirstAt: &long, Why: "laptop refused what it was sent"}}}
if err := open.inventory.SavePlan(ctx, &stopped); err != nil {
t.Fatal(err)
}
@@ -0,0 +1,39 @@
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/broker"
)
// A membership issued again for what it hears changes nothing else of it (novox/hq ADR 0363 §6).
func TestAMembershipIssuedAgainChangesOnlyWhatItHears(t *testing.T) {
issued := []byte(`{"node":"ace","module":"reader","receives":{"x":[1]},"emits":"mesh.mod.reader.event.{event}"}`)
hears := []broker.HeardIssued{{Stream: "MAIL", Consumer: "ace_reader"}}
body, changed, err := withHearing(issued, hears)
if err != nil || !changed {
t.Fatalf("changed %v, %v", changed, err)
}
var got map[string]any
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if got["receives"] == nil || got["emits"] != "mesh.mod.reader.event.{event}" || got["hears"] == nil {
t.Fatalf("issued again as %s", body)
}
if _, again, _ := withHearing(body, hears); again {
t.Fatal("the same hearing issued again counts as a change")
}
gone, changed, err := withHearing(body, nil)
if err != nil || !changed {
t.Fatalf("a withdrawal: %v, %v", changed, err)
}
got = nil
if err := json.Unmarshal(gone, &got); err != nil {
t.Fatal(err)
}
if _, still := got["hears"]; still {
t.Fatalf("a withdrawn hearing is still issued: %s", gone)
}
}
+568 -42
View File
@@ -39,8 +39,8 @@ import (
// from the same manifest is no move.
//
// **The release plan** is what walks the waiting moves through: whenever moves wait for a gate that no
// started plan is walking, the mesh opens one — every such machine, one at a time, the control node last,
// each sent and judged before the next. A release plan that fails stops the next one opening on its own:
// started plan is walking, the mesh opens one — every such machine, the first sent and judged, then the rest
// at once, each judged by its own gate (novox/hq ADR 0353, in place of one at a time). A release plan that fails stops the next one opening on its own:
// a person releases it again (`upgrade release-backlog --why`), after the condition says why.
// sendScope is what a send may carry that no gate has seen.
@@ -81,6 +81,15 @@ type moveFacts struct {
srcs map[string]map[string]string
passed map[string]map[string]bool
plans []inventory.Plan
// self is the release walking these moves, whose own gates are no other walk's (novox/hq ADR 0353).
self string
}
type walkingKey struct{}
// withWalking says a release plan is the one sending: its own gates never hold its own sends back.
func withWalking(ctx context.Context, id string) context.Context {
return context.WithValue(ctx, walkingKey{}, id)
}
func readMoveFacts(ctx context.Context, inv *inventory.Inventory) (moveFacts, error) {
@@ -99,6 +108,7 @@ func readMoveFacts(ctx context.Context, inv *inventory.Inventory) (moveFacts, er
return f, err
}
f.plans, err = inv.OpenPlans(ctx)
f.self, _ = ctx.Value(walkingKey{}).(string)
return f, err
}
@@ -193,17 +203,24 @@ func (f moveFacts) moves(node string, modules []string, sent map[string]string,
// feet. A release keeps no module records: its open gate's carried moves are its walk.
func (f moveFacts) walkedBy(module, node, to string) string {
for _, p := range f.plans {
if !p.Open() {
if !p.Open() || (p.Release != nil && p.ID == f.self) {
continue
}
if p.Release != nil {
g := p.Release.Gate
if g == nil || g.Verdict != "" {
continue
// The first machine's gate, and every gate of the rest still judging (novox/hq ADR 0353).
gates := []*inventory.PlanGate{p.Release.Gate}
for _, g := range p.Release.Gates {
gates = append(gates, g)
}
for _, c := range g.Carried {
if c.Module == module && !(slices.Contains(g.Machines, node) && sameCommit(c.To, to)) {
return p.ID
for _, g := range gates {
// A send kept and not yet made walks nothing yet (ADR 0353 §8).
if g == nil || g.Verdict != "" || g.Silent || g.Pending {
continue
}
for _, c := range g.Carried {
if c.Module == module && !(slices.Contains(g.Machines, node) && sameCommit(c.To, to)) {
return p.ID
}
}
}
continue
@@ -303,8 +320,8 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
}
}
if len(refused) > 0 {
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine at a time, each judged (novox/hq ADR "+
"0236); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine first and then the rest at once, each judged (novox/hq ADR "+
"0236, ADR 0353); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
}
return kept, nil
}
@@ -320,19 +337,35 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
// And the grants step first, when the send changes what the bus's user list must say (novox/hq issue 490):
// answered for the gate to keep, nil when there was none.
func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.CarriedMove) ([]inventory.CarriedMove, []string, *inventory.GrantsStep, error) {
inv := open.inventory
f, err := readMoveFacts(ctx, inv)
moves, err := gatedMoves(ctx, open, node, owns)
if err != nil || len(moves) == 0 {
return nil, nil, nil, err
}
sayRecreations(ctx, open, moves)
sent, grants, err := sendJudged(ctx, open, node)
if err != nil {
return nil, nil, nil, err
}
return moves, sent, grants, nil
}
// gatedMoves is what a gated send to one machine would carry, with the build each moves to, made of nothing:
// gatedSend's reading, which a walk also keeps before it sends (novox/hq ADR 0353 §8). Refused the same way
// while a plan that has started walks one of those builds elsewhere.
func gatedMoves(ctx context.Context, open *stores, node string, owns []inventory.CarriedMove) ([]inventory.CarriedMove, error) {
inv := open.inventory
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
}
moves, err := machineMoves(ctx, open, f, node, true)
if err != nil {
return nil, nil, nil, err
return nil, err
}
// A move the send keeps is not made, and is no move for the gate to judge (novox/hq issue 201): a core module
// kept over an older record judged as moving back to it would read the gate as superseded by its own send.
if moves, err = withoutKept(ctx, open, node, moves); err != nil {
return nil, nil, nil, err
return nil, err
}
own := func(module string) bool {
return slices.ContainsFunc(owns, func(o inventory.CarriedMove) bool { return o.Module == module })
@@ -342,7 +375,7 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
continue
}
if id := f.walkedBy(mv.Module, node, mv.To); id != "" {
return nil, nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
return nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
mv.Module, short(mv.To), node, id)
}
}
@@ -350,7 +383,7 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
// plan, or a release, on this machine is not crossed with a newer build; this send waits for its gate.
for _, o := range owns {
if id := f.walkedBy(o.Module, node, o.To); id != "" {
return nil, nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
return nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
o.Module, short(o.To), node, id)
}
}
@@ -363,30 +396,26 @@ func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.
moves[i].Build = o.Build
}
}
if len(moves) == 0 && len(owns) == 0 {
return nil, nil, nil, nil
if len(moves) == 0 {
return nil, nil
}
for i := range moves {
if moves[i].Build == "" {
if moves[i].Build, err = inv.BuildOf(ctx, moves[i].Module, moves[i].To); err != nil {
return nil, nil, nil, err
return nil, err
}
}
}
sort.Slice(moves, func(i, j int) bool { return moves[i].Module < moves[j].Module })
sayRecreations(ctx, open, moves)
sent, grants, err := sendJudged(ctx, open, node)
if err != nil {
return nil, nil, nil, err
}
return moves, sent, grants, nil
return moves, nil
}
// passCarried keeps a pass as the verdict of every build the gate judged beside its own module.
func passCarried(ctx context.Context, open *stores, p *inventory.Plan, g *inventory.PlanGate, except string) {
seen := map[string]bool{}
for _, c := range g.Carried {
if c.Module == except || c.Build == "" || seen[c.Build] {
// A build put back with another machine's failure is not this gate's to pass (novox/hq ADR 0353 §3).
if c.Module == except || c.Build == "" || seen[c.Build] || slices.Contains(g.Returned, c.Module) {
continue
}
seen[c.Build] = true
@@ -695,7 +724,7 @@ func releaseBacklog(ctx context.Context, open *stores, by string) (*inventory.Pl
p := inventory.Plan{ID: fmt.Sprintf("release-%d", now.UnixNano()), Repository: releaseRepository,
Created: now, State: inventory.PlanRolling, Tiers: [][]string{names}, Modules: map[string]*inventory.PlanModule{},
Release: &inventory.PlanRelease{Order: order, By: by},
Note: fmt.Sprintf("%d build(s) wait for a gate on %s; one machine at a time, each judged", len(names),
Note: fmt.Sprintf("%d build(s) wait for a gate on %s; one machine first, then the rest at once, each judged", len(names),
strings.Join(order, ", "))}
if err := inv.SavePlan(ctx, &p); err != nil {
return nil, err
@@ -704,14 +733,28 @@ func releaseBacklog(ctx context.Context, open *stores, by string) (*inventory.Pl
return &p, nil
}
// advanceRelease takes one step of a release plan: the next machine sent everything waiting there under a
// gate, or the machine being judged judged once more; the plan done when every machine is.
// advanceRelease takes one step of a release plan: the first machine sent everything waiting there under a
// gate and judged; once it passed, every other machine sent at once, each under its own gate (novox/hq ADR
// 0353 §1), and judged; the plan done when every machine is.
//
// **A send is kept before it is made** (ADR 0353 §8): the step that chooses a machine keeps its gate pending with
// what it will carry, and the next step makes the send. A controller replaced between the send and keeping it
// finds the gate pending, reads the machine's last send, and neither sends again nor leaves the machine unjudged.
func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool, error) {
r := p.Release
if r.Settings != nil {
return advanceSettingsWalk(ctx, open, p)
}
ctx = withWalking(ctx, p.ID)
now := time.Now().UTC()
if r.Rest {
return advanceRest(ctx, open, p, now)
}
if r.Gate == nil && !gateFirstMachine {
// **No first machine** (novox/hq ADR 0360 §7): every machine heard from is sent at once, in one step.
r.Rest = true
return advanceRest(ctx, open, p, now)
}
if r.Gate == nil {
heard, err := releaseHeard(ctx, open)
if err != nil {
@@ -724,7 +767,7 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Next++
continue
}
moves, sent, grants, err := gatedSend(ctx, open, node, nil)
moves, err := gatedMoves(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
note := fmt.Sprintf("waiting before %s: %v", node, err)
changed := p.Note != note
@@ -732,23 +775,15 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return changed, nil
}
if err != nil {
return false, fmt.Errorf("sending %s what waits there: %w", node, err)
return false, fmt.Errorf("reading what waits on %s: %w", node, err)
}
if len(moves) == 0 {
r.Done = append(r.Done, node)
r.Next++
continue
}
r.Gate = &inventory.PlanGate{Machines: sent, Since: &now, Carried: moves, Sent: sentNow(ctx, open.inventory, sent),
Grants: grants}
p.Note = fmt.Sprintf("sent %s %d build(s) that waited for a gate; judging them there", node, len(moves))
if said := grantsSaid(grants); said != "" {
p.Note += "; " + said
}
if said := recreationsSaid(moves); said != "" {
p.Note += "; " + said
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
r.Gate = &inventory.PlanGate{Machines: []string{node}, Carried: moves, Pending: true}
p.Note = fmt.Sprintf("sending %s %d build(s) that wait for a gate, first", node, len(moves))
return true, nil
}
p.State, p.Tier = inventory.PlanDone, len(p.Tiers)
@@ -759,6 +794,19 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return true, nil
}
g := r.Gate
if g.Pending {
node := firstOf(g.Machines)
out, err := sendKept(ctx, open, p, node, g, now)
if err != nil || out == keptWaiting {
return out == keptWaiting, err
}
if out == keptNothing {
r.Gate = nil
r.Done = append(r.Done, node)
r.Next++
}
return true, nil
}
verdict, err := judgeMoves(ctx, open, g, nil, now)
if err != nil {
return false, err
@@ -777,6 +825,11 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Done = append(r.Done, firstOf(g.Machines))
r.Next++
r.Gate = nil
// **Then the rest at once** (novox/hq ADR 0353 §1): every other machine heard from, each judged by its own
// gate, in place of one machine at a time (ADR 0236 §22).
r.Rest = true
p.Note = fmt.Sprintf("passed on %s; the rest are sent at once, each judged", firstOf(g.Machines))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
case inventory.GateSuperseded:
// Another send moved a judged module on the judged machine (novox/hq issue 352): no verdict on what
@@ -797,6 +850,479 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return true, nil
}
// What sendKept did with a kept send.
const (
keptSent = iota // sent now
keptAlready // the machine was already sent it: judged from that send, nothing sent
keptNothing // nothing waits there any more: no gate
keptWaiting // another walk judges a build it would carry: it waits
)
// sendKept makes a send a gate kept before it was made (novox/hq ADR 0353 §8) — unless the machine's last send
// already carries what the gate is for, made before a controller was replaced and before it was kept: then
// nothing is sent, and the gate judges from that send (§6).
func sendKept(ctx context.Context, open *stores, p *inventory.Plan, node string, g *inventory.PlanGate,
now time.Time) (int, error) {
inv := open.inventory
moves, sent, grants, err := gatedSend(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
p.Note = fmt.Sprintf("waiting before %s: %v", node, err)
return keptWaiting, nil
}
if err != nil {
return keptSent, fmt.Errorf("sending %s what waits there: %w", node, err)
}
if len(moves) > 0 {
g.Machines, g.Carried, g.Since, g.Grants, g.Pending = sent, moves, &now, grants, false
g.Sent = sentNow(ctx, inv, sent)
p.Note = fmt.Sprintf("sent %s %d build(s) that waited for a gate; judging them there", node, len(moves))
if said := grantsSaid(grants); said != "" {
p.Note += "; " + said
}
if said := recreationsSaid(moves); said != "" {
p.Note += "; " + said
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
return keptSent, nil
}
if !alreadySent(ctx, inv, node, g.Carried) {
return keptNothing, nil
}
since := now
if s, found, err := inv.SentTo(ctx, node); err == nil && found {
if !s.At.IsZero() {
since = s.At
}
g.Sent = map[string]inventory.SentDeclaration{node: s}
}
g.Since, g.Pending = &since, false
p.Note = fmt.Sprintf("%s was already sent the %d build(s) kept for it, at %s: judged from that send, nothing sent again",
node, len(g.Carried), since.Local().Format("15:04:05"))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return keptAlready, nil
}
// alreadySent says a machine's last send carries every build of the moves: a send made and not kept.
func alreadySent(ctx context.Context, inv *inventory.Inventory, node string, moves []inventory.CarriedMove) bool {
if len(moves) == 0 {
return false
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return false
}
for _, mv := range moves {
if !sameCommit(sent[mv.Module], mv.To) {
return false
}
}
return true
}
// advanceRest takes one step of a release plan's rest (novox/hq ADR 0353): the sends kept for a step made, every
// machine of the step judged by its own gate, one that fails put back by itself while the others go on to their
// verdicts, and the next step — or the end — once every gate of the step has its verdict or is silent.
func advanceRest(ctx context.Context, open *stores, p *inventory.Plan, now time.Time) (bool, error) {
r := p.Release
// The sends kept for this step, in the plan's order: the control node, last in it, last (§1).
var pending []string
for _, n := range r.Order {
if g := r.Gates[n]; g != nil && g.Pending {
pending = append(pending, n)
}
}
if len(pending) > 0 {
// **A kept send carries only what was kept** (ADR 0353 §2): a build that passed no gate and came to wait on
// a machine since its step was chosen could reach a second machine in it; the step is chosen again.
if grew, err := keptStepGrew(ctx, open, r, pending); err != nil {
return false, err
} else if grew != "" {
for _, n := range pending {
delete(r.Gates, n)
}
opened, waiting, err := openRestStep(ctx, open, p)
if err != nil {
return true, err
}
p.Note = "chosen again: " + grew
if !opened && waiting != "" {
p.Note += "; " + waiting
}
return true, nil
}
var sent, already, waiting []string
for _, node := range pending {
out, err := sendKept(ctx, open, p, node, r.Gates[node], now)
if err != nil {
return true, err
}
switch out {
case keptSent:
sent = append(sent, node)
case keptAlready:
already = append(already, node)
case keptNothing:
delete(r.Gates, node)
r.Done = append(r.Done, node)
case keptWaiting:
waiting = append(waiting, node)
}
}
var said []string
if len(sent) > 0 {
said = append(said, "sent "+strings.Join(sent, ", ")+" at once, each judged there")
}
if len(already) > 0 {
said = append(said, strings.Join(already, ", ")+" already sent it: judged from that send")
}
if len(waiting) > 0 {
said = append(said, "waiting before "+strings.Join(waiting, ", ")+": another walk judges a build it carries")
}
if len(said) > 0 {
p.Note = strings.Join(said, "; ")
}
return len(sent)+len(already) > 0 || len(waiting) < len(pending), nil
}
reports, err := open.inventory.LastReports(ctx)
if err != nil {
return false, err
}
byNode := map[string]inventory.Reported{}
for _, rep := range reports {
byNode[rep.Node] = rep
}
changed, noteWas := false, p.Note
var notes, judging []string
batched, back := batchingRollbacks(ctx)
for _, node := range r.Order {
g := r.Gates[node]
if g == nil || g.Verdict != "" || g.Silent {
continue
}
// What another machine's failure already put back is not this gate's to judge (ADR 0353 §3).
if len(g.Carried) > 0 && len(g.Returned) >= len(g.Carried) {
decide(g, inventory.GateFailed, "what it was sent was put back with another machine's failure", now)
r.Failed = append(r.Failed, node)
changed = true
continue
}
// **A machine that says nothing on its send is left silent, not failed** (ADR 0353 §4): it is not judged
// until it reports, and past ApplySilentAfter it is listed and left.
if !reportedOnSend(byNode[node], node, g) {
if now.Sub(*g.Since) > inventory.ApplySilentAfter {
g.Silent = true
r.Silent = append(r.Silent, node)
changed = true
notes = append(notes, fmt.Sprintf("%s said nothing on its send within %s: left silent", node,
inventory.ApplySilentAfter))
continue
}
judging = append(judging, node+": not yet reported on its send")
continue
}
if !gateFirstMachine {
// **Judged by its report alone** (novox/hq ADR 0360 §7): applied passes; failed or refused is named and
// fails the walk, with nothing put back — apply-failing says it.
switch rep := byNode[node]; rep.Outcome {
case inventory.OutcomeApplied:
decide(g, inventory.GatePassed, node+" reported it applied its send (novox/hq ADR 0360)", now)
passCarried(ctx, open, p, g, "")
r.Done = append(r.Done, node)
changed = true
case inventory.OutcomeFailed, inventory.OutcomeRefused:
decide(g, inventory.GateFailed, fmt.Sprintf("%s %s its send", node, rep.Outcome), now)
r.Failed = append(r.Failed, node)
changed = true
notes = append(notes, fmt.Sprintf("%s %s its send: nothing put back, apply-failing says it", node, rep.Outcome))
default:
judging = append(judging, node+": not yet reported on its send")
}
continue
}
verdict, err := judgeMoves(ctx, open, g, nil, now)
if err != nil {
return changed, err
}
switch verdict {
case "":
if putBackBroken(ctx, open, p, g, "", nil) {
changed = true
notes = append(notes, p.Note)
}
judging = append(judging, node+": "+gateLine(g))
case inventory.GatePassed:
passCarried(ctx, open, p, g, "")
r.Done = append(r.Done, node)
changed = true
case inventory.GateSuperseded:
// Another send moved a judged module there (novox/hq issue 352): nothing judged, nothing put back on
// it; what still waits is released again.
r.Done = append(r.Done, node)
changed = true
notes = append(notes, fmt.Sprintf("superseded on %s: %s", node, g.Why))
default:
// **Put back by itself** (ADR 0353 §3): what was found wanting goes back at once, and every other
// machine of the step goes on to its own verdict.
stateWas := p.State
p.Note = ""
failCarried(batched, open, p, g, "")
p.State = stateWas
r.Failed = append(r.Failed, node)
changed = true
notes = append(notes, fmt.Sprintf("failed its gate on %s: %s — %s", node, g.Why, p.Note))
returnedElsewhere(ctx, open, r, node, g)
}
}
p.Note = ""
sendRollbacks(ctx, open, p, back)
if p.Note != "" {
notes = append(notes, strings.TrimPrefix(p.Note, "; "))
}
if len(judging) > 0 {
note := strings.Join(append(notes, "judging "+strings.Join(judging, "; ")), "; ")
p.Note = note
return changed || noteWas != note, nil
}
// Every gate of the step has its verdict or is silent: the next step, unless a machine failed (§3).
if len(r.Failed) == 0 {
opened, waiting, err := openRestStep(ctx, open, p)
if err != nil {
return changed, err
}
if opened {
return true, nil
}
if waiting != "" {
note := strings.Join(append(notes, waiting), "; ")
p.Note = note
return changed || noteWas != note, nil
}
}
p.Tier = len(p.Tiers)
end := fmt.Sprintf("released on %s", orNone(strings.Join(r.Done, ", ")))
if len(r.Silent) > 0 {
end += "; silent on their send, left: " + strings.Join(r.Silent, ", ")
}
if len(r.Skipped) > 0 {
end += "; not heard from, left as they were: " + strings.Join(r.Skipped, ", ")
}
p.State = inventory.PlanDone
if len(r.Failed) > 0 {
p.State = inventory.PlanFailed
end = "failed on " + strings.Join(r.Failed, ", ") + "; " + end
} else if len(r.Silent) > 0 && len(r.Gates) == len(r.Silent) {
// **A rest that said nothing is no success** (ADR 0353 §4): failure is loud; the condition says it.
p.State = inventory.PlanFailed
end = "no machine of the rest reported on its send; " + end
}
p.Note = strings.Join(append(notes, end), "; ")
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
}
// keptStepGrew says why a step's kept sends would now carry a build no gate passed that they were not kept with;
// empty when none would.
func keptStepGrew(ctx context.Context, open *stores, r *inventory.PlanRelease, pending []string) (string, error) {
if !gateFirstMachine {
return "", nil // every machine goes at once (novox/hq ADR 0360 §7): a send carries what waits when it is made
}
f, err := readMoveFacts(ctx, open.inventory)
if err != nil {
return "", err
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
for _, node := range pending {
moves, err := gatedMoves(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
continue // sendKept says it waits
}
if err != nil {
return "", err
}
kept := r.Gates[node].Carried
for _, mv := range moves {
// The node-engine and the resolver keep one machine at a time, passed or not (ADR 0353 §9).
if (f.gated(mv.Module, mv.To) && !oneAtATime(mv.Module, shelf[mv.Module])) || slices.ContainsFunc(kept, func(c inventory.CarriedMove) bool {
return c.Module == mv.Module && sameCommit(c.To, mv.To)
}) {
continue
}
return fmt.Sprintf("%s's build %s, which this step did not keep for it, came to wait on %s", mv.Module, short(mv.To), node), nil
}
}
return "", nil
}
// reportedOnSend says a machine's last report is about the send its gate judges (novox/hq issue 352): the
// declaration kept on the gate, or one sequenced after it; a gate that kept none reads the send made last.
func reportedOnSend(r inventory.Reported, node string, g *inventory.PlanGate) bool {
if r.At == nil {
return false
}
if s, kept := g.Sent[node]; kept {
return s.ReportsOn(r)
}
return r.Current
}
// returnedElsewhere marks, on every other gate of the rest still judging, the builds a failed machine's
// rollback put back wherever they were sent (ADR 0236 §5): no longer theirs to judge (ADR 0353 §3).
func returnedElsewhere(ctx context.Context, open *stores, r *inventory.PlanRelease, failed string, g *inventory.PlanGate) {
for _, c := range g.Carried {
if c.Build == "" {
continue
}
if marked, err := open.inventory.GateFailed(ctx, c.Build); err != nil || !marked {
continue
}
for node, other := range r.Gates {
if node == failed || other.Verdict != "" || slices.Contains(other.Returned, c.Module) {
continue
}
for _, oc := range other.Carried {
if oc.Module == c.Module && sameCommit(oc.To, c.To) {
other.Returned = append(other.Returned, c.Module)
}
}
}
}
}
// openRestStep chooses the next step of a release plan's rest and keeps its sends (novox/hq ADR 0353 §1, §2):
// every machine left that is heard from and has something waiting, except that a build no gate has passed goes
// to one machine of the step only. Answers whether a step was kept, or why the walk waits.
func openRestStep(ctx context.Context, open *stores, p *inventory.Plan) (bool, string, error) {
r := p.Release
heard, err := releaseHeard(ctx, open)
if err != nil {
return false, "", err
}
f, err := readMoveFacts(ctx, open.inventory)
if err != nil {
return false, "", err
}
waiting := map[string][]inventory.CarriedMove{}
var nodes, walked []string
for _, node := range r.Order[min(r.Next, len(r.Order)):] {
if r.Gates[node] != nil || slices.Contains(r.Done, node) || slices.Contains(r.Skipped, node) ||
slices.Contains(r.Failed, node) || slices.Contains(r.Silent, node) {
continue
}
if !heard[node] {
r.Skipped = append(r.Skipped, node)
continue
}
moves, err := gatedMoves(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
walked = append(walked, fmt.Sprintf("%s (%v)", node, err))
continue
}
if err != nil {
return false, "", fmt.Errorf("reading what waits on %s: %w", node, err)
}
if len(moves) == 0 {
r.Done = append(r.Done, node)
continue
}
waiting[node] = moves
nodes = append(nodes, node)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return false, "", err
}
// **Every machine at once** (novox/hq ADR 0360 §2, §7): no build waits for one machine to pass first.
step := nodes
if gateFirstMachine {
step = restStep(nodes, waiting, f.gated, func(module string) bool { return oneAtATime(module, shelf[module]) })
}
if len(step) == 0 {
if len(walked) > 0 {
return false, "waiting before " + strings.Join(walked, "; "), nil
}
return false, "", nil
}
if r.Gates == nil {
r.Gates = map[string]*inventory.PlanGate{}
}
for _, node := range step {
r.Gates[node] = &inventory.PlanGate{Machines: []string{node}, Carried: waiting[node], Pending: true}
}
p.Note = fmt.Sprintf("sending %s at once, each judged there", strings.Join(step, ", "))
if left := len(nodes) - len(step); left > 0 {
p.Note += fmt.Sprintf("; %d machine(s) wait for a build that passed no gate yet to pass on one", left)
}
return true, "", nil
}
// oneAtATime is a module whose builds go to the rest one machine at a time (novox/hq ADR 0353 §9): the
// node-engine, which applies every send, and a module holding the mesh's resolver, which names the bus.
func oneAtATime(module string, m catalogue.Manifest) bool {
if module == hostModule {
return true
}
return slices.ContainsFunc(m.Claims, func(c catalogue.Claim) bool { return c.Name == resolverSeat })
}
// restStep is the machines of one step of the rest (novox/hq ADR 0353 §2), in the order given: each machine, unless
// a build it would carry has passed no gate and an earlier machine of the step already carries it — that build's
// first machine. Pure.
//
// **The node-engine and the mesh's resolver still go one machine at a time** (ADR 0353 §9): a step carries a
// build of either to one machine only, judged before the next, since a fault that cuts a machine off the bus
// shows only as its silence and no rollback reaches it.
func restStep(nodes []string, waiting map[string][]inventory.CarriedMove, passed func(module, commit string) bool,
alone func(module string) bool) []string {
first := map[string]string{}
var out []string
for _, n := range nodes {
var unpassed []string
for _, mv := range waiting[n] {
if !passed(mv.Module, mv.To) || alone(mv.Module) {
unpassed = append(unpassed, mv.Module+"@"+mv.To)
}
}
if slices.ContainsFunc(unpassed, func(k string) bool { _, taken := first[k]; return taken }) {
continue
}
for _, k := range unpassed {
first[k] = n
}
out = append(out, n)
}
return out
}
// silentObservation is what the gate's probe says of the newest release whose rest went silent (novox/hq ADR 0353
// §4): machines that never reported on their send, left as they were — loud, never a success said by nothing.
func silentObservation(ctx context.Context, inv *inventory.Inventory) ([]conditions.Observation, error) {
recent, err := inv.RecentPlans(ctx, 20)
if err != nil {
return nil, err
}
for _, p := range recent {
if p.Release == nil || p.Release.Settings != nil || p.Open() {
continue
}
if len(p.Release.Silent) == 0 {
return nil, nil
}
machines := append([]string(nil), p.Release.Silent...)
sort.Strings(machines)
o := conditions.Observation{Scope: conditions.ScopeMesh, ID: "release", Token: "silent", Kind: "release-silent",
Severity: conditions.Warning, Resolver: conditions.ResolverOperator, Also: machines,
Summary: fmt.Sprintf("%s: %s said nothing on the send of the builds that waited for a gate within %s, and were "+
"left as they were", p.ID, strings.Join(machines, ", "), inventory.ApplySilentAfter)}
w := releaseSilentWords(machines)
o.Headline, o.Explanation, o.Resolved, o.Needs = w.Headline, w.Explanation, w.Resolved, w.Needs
return []conditions.Observation{o}, nil
}
return nil, nil
}
// backlogObservation is what the gate's probe says of a release held after a failure.
func backlogObservation() []conditions.Observation {
if backlogNow.held == "" || len(backlogNow.waiting) == 0 {
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A pull request into a release branch is judged by that branch** (novox/hq ADR 0358, issue 704): a base that is
// not its repository's trunk, where mesh-controller has a branch of that name, is a release branch; the check then
// reads every repository beside it at its branch of that name where it has one, else at its trunk. Any other base is
// checked exactly as before.
const aRelease = "feat/57-config-and-provisioning"
// theCore is the mesh's core as a check reads it beside: the controller, the node-engine, the tool runner and the
// catalogue's bus, on the git seat.
func theCore() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, onSeat("gitea", "novox/mesh-catalog")}
}
// aForgeWithBranches stands in for the forge: which branches each repository has, and how often it was asked.
func aForgeWithBranches(t *testing.T, branches map[string][]string, failing error) *int {
t.Helper()
asked := 0
was := forgeHasBranch
t.Cleanup(func() { forgeHasBranch = was })
forgeHasBranch = func(_ context.Context, owner, repo, branch string) (bool, error) {
asked++
if failing != nil {
return false, failing
}
for _, b := range branches[owner+"/"+repo] {
if b == branch {
return true, nil
}
}
return false, nil
}
return &asked
}
func TestABaseIsAReleaseBranchWhenTheControllerHasABranchOfThatName(t *testing.T) {
entries := theCore()
scopeOf := func(p link.PullUpdated) checkScope { return pullScope(p, entries, nil, nil) }
pull := func(repo, base string) link.PullUpdated {
return link.PullUpdated{Owner: "novox", Repo: repo, Number: 286, Base: base, Commit: "abc",
Paths: []string{"modules/gitea/index.ts"}}
}
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease}}, nil)
for _, c := range []struct {
what string
p link.PullUpdated
release string
asks int
}{
{"a pull request into the trunk asks nothing and is no release", pull("mesh-catalog", "main"), "", 0},
{"a catalogue pull request into the release branch", pull("mesh-catalog", aRelease), aRelease, 1},
{"a controller pull request into the release branch", pull("mesh-controller", aRelease), aRelease, 1},
{"a base mesh-controller has no branch of is checked as today", pull("mesh-catalog", "feat/other"), "", 1},
{"no base: as today", pull("mesh-catalog", ""), "", 0},
} {
*asked = 0
got, err := releaseBranchOf(t.Context(), c.p, scopeOf(c.p), entries)
if err != nil {
t.Fatalf("%s: %v", c.what, err)
}
if got != c.release || *asked != c.asks {
t.Errorf("%s: release %q after %d question(s) to the forge, not %q after %d", c.what, got, *asked,
c.release, c.asks)
}
}
// The branch a module built from the repository follows is its trunk: never asked about.
photos := fromRepo("photos", "novox/photos", "")
photos.Source.Ref = "nox-mesh"
withPhotos := append(theCore(), photos)
p := link.PullUpdated{Owner: "novox", Repo: "photos", Base: "nox-mesh", Commit: "abc", Paths: []string{"x.js"}}
*asked = 0
if got, err := releaseBranchOf(t.Context(), p, pullScope(p, withPhotos, nil, nil), withPhotos); err != nil ||
got != "" || *asked != 0 {
t.Fatalf("a module's own trunk read as release %q (%v) after %d question(s)", got, err, *asked)
}
// A forge that cannot answer is an error, never "no release branch".
aForgeWithBranches(t, nil, errors.New("nothing serves gitea.gitea_api"))
q := pull("mesh-catalog", aRelease)
if _, err := releaseBranchOf(t.Context(), q, scopeOf(q), entries); err == nil ||
!strings.Contains(err.Error(), "cannot be told") {
t.Fatalf("an unanswered forge was read as an answer: %v", err)
}
}
func TestBesideAReleaseBranchCheckEachRepositoryIsReadAtThatBranchElseItsTrunk(t *testing.T) {
entries := theCore()
clone := func(s inventory.Source) (string, error) { return "http://git/" + s.Repository + ".git", nil }
running := func(module string) string { return "running-" + module }
beside, err := besideOf(entries, running, clone, aRelease)
if err != nil {
t.Fatal(err)
}
at := func(repo string, ref, orElse string) link.CheckedOut {
return link.CheckedOut{Repository: "http://git/novox/" + repo + ".git", Ref: ref, Else: orElse}
}
want := map[string]link.CheckedOut{
// The judge: mesh-controller's release branch, which it has.
"mesh-controller": at("mesh-controller", aRelease, ""),
"mesh-controller-main": at("mesh-controller", "main", ""),
// A controller pull request into the release branch reads mesh-host at that branch where it has one.
"mesh-host": at("mesh-host", aRelease, "main"),
"mesh-tools": at("mesh-tools", aRelease, "main"),
"mesh-catalog": at("mesh-catalog", aRelease, "main"),
// mesh-lab at its trunk, always: its replays run with the container runtime's socket.
"mesh-lab": at("mesh-lab", "main", ""),
"mesh-sdk": at("mesh-sdk", aRelease, "main"),
}
if !reflect.DeepEqual(beside, want) {
t.Fatalf("beside a release-branch check:\n got %v\nwant %v", beside, want)
}
// Any other base: exactly as before — what the mesh runs, the catalogue's and the lab's main.
beside, err = besideOf(entries, running, clone, "")
if err != nil {
t.Fatal(err)
}
want = map[string]link.CheckedOut{
"mesh-controller": at("mesh-controller", "running-mesh-controller", ""),
"mesh-controller-main": at("mesh-controller", "main", ""),
"mesh-host": at("mesh-host", "running-mesh-host", ""),
"mesh-tools": at("mesh-tools", "running-node-tools", ""),
"mesh-catalog": at("mesh-catalog", "main", ""),
"mesh-lab": at("mesh-lab", "main", ""),
"mesh-sdk": at("mesh-sdk", sdkPinned(), ""),
}
if !reflect.DeepEqual(beside, want) {
t.Fatalf("beside a trunk check:\n got %v\nwant %v", beside, want)
}
}
func TestAReleaseBranchVerdictNamesTheBranchTheJudgeAndWhatItRead(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: aRelease, Release: aRelease}
c := checkedOf(link.BuildResult{ID: "b", Ref: "aaaaaaaaaaaa0000", Checked: asked, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Trunk: "111111111111", Gate: &link.CheckLayer{Verdict: "pass"},
RepoCheck: &link.CheckLayer{Verdict: "pass"}, Release: aRelease, JudgedBy: "fcdecb436117",
Beside: map[string]string{"mesh-host": "222222222222", "mesh-lab": "333333333333"},
BesideAt: map[string]string{"mesh-host": aRelease, "mesh-lab": "main"},
Validator: "mesh-host's validator of " + aRelease + " at 222222222222"}})
if c.Release != aRelease || c.JudgedBy != "fcdecb436117" {
t.Fatalf("the verdict names release %q judged by %q", c.Release, c.JudgedBy)
}
if c.BesideAt["mesh-host"] != aRelease || c.BesideAt["mesh-lab"] != "main" || c.Beside["mesh-host"] != "222222222222" ||
c.Validator != "mesh-host's validator of "+aRelease+" at 222222222222" {
t.Fatalf("the verdict names what it read beside it as %v at %v", c.Beside, c.BesideAt)
}
}
// **Clause 5** (ADR 0358 §5): a release branch's own pull request into its trunk is no release-branch check: the forge
// is asked nothing, the request names no release branch, and the repositories beside it are read as for any trunk
// pull request. Its delivery group's composed check is asked as for any other (askGroupCheck, unchanged).
func TestAReleaseBranchsPullRequestIntoItsTrunkIsCheckedAsAnyTrunkPullRequest(t *testing.T) {
entries := theCore()
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease}}, nil)
for _, repo := range []string{"mesh-catalog", "mesh-controller", "mesh-host"} {
p := link.PullUpdated{Owner: "novox", Repo: repo, Number: 400, Head: aRelease, Base: "main", Commit: "abc",
Paths: []string{"modules/gitea/index.ts"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, entries, nil, nil), entries)
if err != nil || got != "" || *asked != 0 {
t.Fatalf("%s's release branch into main read as release %q (%v) after %d question(s)", repo, got, err, *asked)
}
}
clone := func(s inventory.Source) (string, error) { return "http://git/" + s.Repository + ".git", nil }
beside, err := besideOf(entries, func(module string) string { return "running-" + module }, clone, "")
if err != nil {
t.Fatal(err)
}
for dir, b := range beside {
if b.Ref == aRelease || b.Else != "" {
t.Fatalf("beside a trunk pull request %s is read at %+v", dir, b)
}
}
}
// **A forge answer that names no branch is an error**, never "no release branch"; a 404 is "no such branch".
func TestTheForgesAnswerAboutABranchIsReadStrictly(t *testing.T) {
for _, c := range []struct {
what string
answer link.Answer
has bool
fails bool
}{
{"the branch named", link.Answer{Result: []byte(`{"result":{"name":"` + aRelease + `"}}`)}, true, false},
{"not found", link.Answer{Error: "Gitea API /repos/novox/mesh-controller/branches/x: 404 {\"message\":\"not found\"}"},
false, false},
{"an empty branch name", link.Answer{Result: []byte(`{"result":{"name":""}}`)}, false, true},
{"no result", link.Answer{Result: []byte(`{}`)}, false, true},
{"another branch named", link.Answer{Result: []byte(`{"result":{"name":"main"}}`)}, false, true},
{"unreadable", link.Answer{Result: []byte(`not json`)}, false, true},
{"another error", link.Answer{Error: "Gitea API: 500 internal"}, false, true},
} {
has, err := branchAnswered(c.answer, aRelease)
if has != c.has || (err != nil) != c.fails {
t.Errorf("%s: has %v, error %v", c.what, has, err)
}
}
}
// **The trunk is the repository's own, not a name**: a repository whose trunk is not main, one that follows main, and
// a controller whose trunk is not main.
func TestATrunkIsTheBranchTheRepositoryFollows(t *testing.T) {
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease, "master"}}, nil)
legacy := fromRepo("legacy", "novox/legacy", "")
legacy.Source.Ref = "master"
onMain := fromRepo("photos", "novox/photos", "")
onMain.Source.Ref = "main"
entries := append(theCore(), legacy, onMain)
for _, c := range []struct {
what, repo, base, release string
asks int
}{
{"a repository whose trunk is master, into master", "legacy", "master", "", 0},
{"a repository whose trunk is master, into the release branch", "legacy", aRelease, aRelease, 1},
{"a repository whose module follows main, into main", "photos", "main", "", 0},
{"a repository whose module follows main, into the release branch", "photos", aRelease, aRelease, 1},
} {
*asked = 0
p := link.PullUpdated{Owner: "novox", Repo: c.repo, Base: c.base, Commit: "abc", Paths: []string{"x"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, entries, nil, nil), entries)
if err != nil || got != c.release || *asked != c.asks {
t.Errorf("%s: release %q (%v) after %d question(s), not %q after %d", c.what, got, err, *asked, c.release, c.asks)
}
}
// A controller whose trunk is not main: main may be a release branch there, and its own trunk never is.
core := theCore()
core[0].Source.Ref = "trunk"
for _, c := range []struct {
base, release string
asks int
}{{"trunk", "", 0}, {"main", "main", 1}} {
*asked = 0
p := link.PullUpdated{Owner: "novox", Repo: "hq", Base: c.base, Commit: "abc", Paths: []string{"x"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, core, nil, nil), core)
if err != nil || got != c.release || *asked != c.asks {
t.Errorf("hq into %s, the controller following trunk: release %q (%v) after %d question(s)", c.base, got, err,
*asked)
}
}
}
// trunkOf: the branch a source follows, and main when it records none or records a commit (issue 215's rule).
func TestATrunkIsMainWhenTheSourceRecordsNone(t *testing.T) {
for ref, want := range map[string]string{"": "main", "main": "main", "master": "master", "nox-mesh": "nox-mesh",
"0123456789abcdef0123456789abcdef01234567": "main", " ": "main"} {
if got := trunkOf(inventory.Source{Ref: ref}); got != want {
t.Errorf("a source recording %q follows %q, not %q", ref, got, want)
}
}
}
+413 -94
View File
@@ -9,11 +9,13 @@ import (
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -26,6 +28,10 @@ import (
// alone cannot (a tier waiting for machines to report); a controller replaced mid-plan finds the
// plan where it left it.
// gateFirstMachine sends a module that rolls out to one machine first and judges it there (ADR 0236). Off since
// novox/hq ADR 0360: the gate's code is kept for the walks a controller before it started, and its tests turn it on.
var gateFirstMachine = false
// planWaitBound is how long a plan may wait on one thing before `status` names it red.
const planWaitBound = 30 * time.Minute
@@ -42,29 +48,15 @@ func tiersOf(set []string, edges []inventory.Edge) [][]string {
for _, m := range set {
deps[m] = map[string]bool{}
}
// The build seat's holders follow the controller that defines their worker (EdgeWorkerOf,
// novox/hq issue 206), so the built-by edge from that controller to such a holder yields: the
// controller is built by whichever build machine is running, as the runtime image always was.
worker := map[string]map[string]bool{}
for _, e := range edges {
if e.Kind == inventory.EdgeWorkerOf && in[e.From] && in[e.To] {
if worker[e.To] == nil {
worker[e.To] = map[string]bool{}
}
worker[e.To][e.From] = true
}
}
for _, e := range edges {
// A code dependency — B packages A's source — rebuilds B with A, in the same tier: B's
// build needs nothing of A's first. The other kinds order: stands-on and declared after
// the base is built, built-by after the build machine is built and running — except for
// what the build machine itself stands on, and for the controller whose worker the build
// machine binds. The runtime image is built by the builder and the builder is built on the
// runtime image; the image comes first, built by the builder that is running.
// build needs nothing of A's first. Stands-on and declared order after the base is built, and the
// build seat's holder follows the controller that defines its worker (EdgeWorkerOf, issue 206).
if !in[e.From] || !in[e.To] || e.From == e.To || e.Kind == inventory.EdgePackages {
continue
}
if e.Kind == inventory.EdgeBuiltBy && (isBaseOf(e.From, e.To, edges, in) || worker[e.From][e.To]) {
// **Being built by the builder orders nothing** (novox/hq ADR 0360 §3): whichever builder runs builds it.
if e.Kind == inventory.EdgeBuiltBy {
continue
}
deps[e.From][e.To] = true
@@ -105,29 +97,6 @@ func tiersOf(set []string, edges []inventory.Edge) [][]string {
return tiers
}
// isBaseOf says whether `to` stands on `base`, directly or through other bases in the set, along
// the build edges alone.
func isBaseOf(base, to string, edges []inventory.Edge, in map[string]bool) bool {
seen := map[string]bool{}
var walk func(string) bool
walk = func(m string) bool {
if m == base {
return true
}
if seen[m] {
return false
}
seen[m] = true
for _, e := range edges {
if e.From == m && in[e.To] && (e.Kind == inventory.EdgeStandsOn || e.Kind == inventory.EdgeDeclared) && walk(e.To) {
return true
}
}
return false
}
return walk(to)
}
// reachableFrom is the moved modules plus everything that depends on them, through every layer:
// what a merge rebuilds. Along the code and build edges only: a module *built by* the build machine
// is not changed by a new build machine, so a built-by edge orders and gates a plan and never
@@ -163,7 +132,7 @@ func reachableFrom(moved []string, edges []inventory.Edge) []string {
}
// hasCycle says whether the tiers' last tier holds modules that still depend on each other. A packages
// edge orders nothing (tiersOf), so a module and what packages its source share a tier by rule: that is
// or built-by edge orders nothing (tiersOf), so a module and what packages its source share a tier by rule: that is
// no cycle, and saying one was is a false report in every such plan's log.
func hasCycle(tiers [][]string, edges []inventory.Edge) bool {
if len(tiers) == 0 {
@@ -174,7 +143,7 @@ func hasCycle(tiers [][]string, edges []inventory.Edge) bool {
last[m] = true
}
for _, e := range edges {
if e.Kind == inventory.EdgePackages {
if e.Kind == inventory.EdgePackages || e.Kind == inventory.EdgeBuiltBy {
continue
}
if last[e.From] && last[e.To] {
@@ -220,12 +189,6 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
return nil
}
inTier := map[string]bool{}
all := map[string]bool{}
for _, tier := range p.Tiers {
for _, m := range tier {
all[m] = true
}
}
for _, m := range p.Tiers[p.Tier] {
inTier[m] = true
}
@@ -240,8 +203,8 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
for _, e := range edges {
// A delivery group's order inside a walk (novox/hq ADR 0276 decision 5) gates as built-by does: the
// member before is running on its machines before the member after is asked.
ordered := e.Kind == edgeGroupOrder ||
e.Kind == inventory.EdgeBuiltBy && !isBaseOf(e.From, e.To, edges, all)
// A built-by edge waits for nothing since novox/hq ADR 0360 §3.
ordered := e.Kind == edgeGroupOrder
if later[e.From] && inTier[e.To] && ordered && !seen[e.To] && rollsOut(e.To) {
seen[e.To] = true
out = append(out, e.To)
@@ -251,6 +214,20 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
return out
}
// noMoveWhy is what a plan says of a module of its tier whose build was made from the source every machine
// running it runs (novox/hq issue 280): nothing is sent for it.
// noMovePrefix opens it, and is what noMove reads: one prefix for the writer and the reader.
const (
noMovePrefix = "no move"
noMoveWhy = noMovePrefix + ": made from the source every machine running it runs"
)
// noMove is whether the plan marked the module no move and sent it nothing. Read from what the plan says,
// so a plan kept before the mark had a name of its own is read the same (novox/hq issue 648).
func noMove(s *inventory.PlanModule) bool {
return s != nil && s.FirstAt == nil && s.Gate == nil && strings.HasPrefix(s.Why, noMovePrefix)
}
// applied says whether every machine running the module has reported since the module was built.
// appliedEach is applied with a moment of its own for each machine: the reports that count are the ones
// after that machine was sent the build (novox/hq issue 256).
@@ -282,10 +259,11 @@ func applied(module string, builtAt time.Time, running []string, reports []inven
return len(waiting) == 0, waiting
}
// askTier asks the build machine for every module of the tier, and marks each asked. A module
// askTier asks the build machine for the named modules of the plan's tier — all of it, or what of it is still
// unasked (novox/hq issue 754) — and marks each asked. A module
// the catalogue no longer holds, or whose ask could not be made, is a failure of the plan: a tier
// half asked is a tier that will never complete.
func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) error {
func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan, names []string) error {
entries, err := inv.Catalogued(ctx)
if err != nil {
return err
@@ -294,12 +272,62 @@ func askTier(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan) e
for _, e := range entries {
byName[e.Manifest.Module] = e
}
for _, name := range p.Tiers[p.Tier] {
for _, name := range names {
askModule(ctx, p, name, byName)
}
return nil
}
// unaskedOf are the modules of a plan's current tier it has still to ask: those it never asked, and one it holds
// built that it never asked for (novox/hq issue 754) — a build taken in by a controller from before a walk took
// only its own asks, somebody else's, which the walk would otherwise send as its own (ADR 0219). A tier partly asked is asked the rest, not
// waited on for ever: a walk that waits for every build of a tier it never asked whole never moves again.
func unaskedOf(p inventory.Plan) []string {
if p.Tier >= len(p.Tiers) {
return nil
}
var out []string
for _, m := range p.Tiers[p.Tier] {
s := p.Modules[m]
switch {
case s == nil || s.State == "":
out = append(out, m)
case s.State == "built" && s.Build == "" && s.AskedAt == nil:
out = append(out, m)
}
}
return out
}
// askedInTier says the plan asked a module of its current tier itself: under an id, or at a time it kept.
func askedInTier(p inventory.Plan) bool {
if p.Tier >= len(p.Tiers) {
return false
}
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && (s.Build != "" || s.AskedAt != nil) {
return true
}
}
return false
}
// takesOutcome says whether a build's outcome is the plan's for a module of its current tier (novox/hq ADR 0219,
// issues 219 and 754). **A walk that waits for its delivery's word asks nothing, and so is answered by nothing**
// (ADR 0239): a rebuild that lands in the wait is somebody else's build. **Nor is a module the plan never asked
// answered**: taken, the plan held a module built that it never asked, and waited for the rest of the tier for
// ever. Asked under an id, the plan takes that id's outcome and no other; asked before ids were kept, a build
// asked at or after the plan's ask.
func takesOutcome(p inventory.Plan, state *inventory.PlanModule, asked time.Time, id string) bool {
if p.Waiting() || state == nil || (state.Build == "" && state.AskedAt == nil) {
return false
}
if state.Build != "" {
return state.Build == id
}
return !askedBefore(asked, state.AskedAt)
}
// askABuild is how a plan asks for one build, not waited for, and learns the id it asked under. A
// variable so a test of what a plan does around an ask needs no build machine.
//
@@ -372,15 +400,73 @@ var checkedBuildsOf = func(ctx context.Context, pulls []link.PullRef) ([]link.Ch
return link.ReadCheckBuilds(ctx, js, buildSeatHeld(ctx), pulls, time.Now().Add(-builder.CheckBuildsKept))
}
// mergedHeadsOf are the head commits the walk's merges merged, by pull request (novox/hq issue 703): a pull request
// merged twice in one walk, or whose head the announcement did not say, has none, and takes no check's build.
func mergedHeadsOf(p *inventory.Plan) map[link.PullRef]string {
heads := map[link.PullRef]string{}
if p.Delivery == nil {
return heads
}
seen := map[link.PullRef]bool{}
for _, m := range p.Delivery.Merges {
pull, ok := pullOf(m)
if !ok {
continue
}
if seen[pull] && heads[pull] != m.Head {
heads[pull] = ""
continue
}
seen[pull] = true
heads[pull] = m.Head
}
return heads
}
// walkVerdicts are the checks' builds a walk read, by the walk's plan: read once per walk (novox/hq issue 649), not
// once per module it asks for. Kept a day; a walk lasts minutes.
var walkVerdicts = struct {
sync.Mutex
read map[string]verdictsRead
}{read: map[string]verdictsRead{}}
type verdictsRead struct {
at time.Time
builds []link.CheckedBuild
}
// reuseWithVerdicts is a walk's leave with the builds its pull requests' verdicts name: the only builds the builder
// may take (ADR 0350 §2). When they cannot be read the leave names none, and the walk builds, saying why.
func reuseWithVerdicts(ctx context.Context, r *link.ReuseAsk) *link.ReuseAsk {
builds, err := checkedBuildsOf(ctx, r.Pulls)
if err != nil {
fmt.Printf(" the checks' builds of %d pull request(s) could not be read, so the walk builds: %v\n", len(r.Pulls), err)
return r
// may take (ADR 0350 §2), and **only those of a check of the head each merge merged** (novox/hq issue 703): a build
// a check made at an earlier head of the same pull request is never registered. Read once per walk. When they
// cannot be read the leave names none, and the walk builds, saying why.
func reuseWithVerdicts(ctx context.Context, walk string, r *link.ReuseAsk, heads map[link.PullRef]string) *link.ReuseAsk {
walkVerdicts.Lock()
read, known := walkVerdicts.read[walk]
walkVerdicts.Unlock()
if !known || walk == "" {
builds, err := checkedBuildsOf(ctx, r.Pulls)
if err != nil {
fmt.Printf(" the checks' builds of %d pull request(s) could not be read, so the walk builds: %v\n", len(r.Pulls), err)
return r
}
read = verdictsRead{at: time.Now(), builds: builds}
if walk != "" {
walkVerdicts.Lock()
for id, w := range walkVerdicts.read {
if time.Since(w.at) > 24*time.Hour {
delete(walkVerdicts.read, id)
}
}
walkVerdicts.read[walk] = read
walkVerdicts.Unlock()
}
}
r.Builds = nil
for _, b := range read.builds {
if link.SameHead(b.Head, heads[b.Pull]) {
r.Builds = append(r.Builds, b)
}
}
r.Builds = builds
return r
}
@@ -423,7 +509,8 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
}
// **The walk takes the check's build of the same key** (novox/hq ADR 0350 §2): its ask says so, with the pull
// requests it answers, whose checks name the part of a differing key.
id, err := askABuild(withReuse(ctx, reuseWithVerdicts(ctx, reuseAskOf(p))), source, e.Source.Path, ref)
id, err := askABuild(withReuse(ctx, reuseWithVerdicts(ctx, p.ID, reuseAskOf(p), mergedHeadsOf(p))), source,
e.Source.Path, ref)
if err != nil {
state.State = "failed"
state.Why = err.Error()
@@ -490,22 +577,15 @@ func planBuiltSaying(ctx context.Context, open *stores, module, commit, failed s
if !inTier {
continue
}
state := p.Modules[module]
if state == nil {
state = &inventory.PlanModule{}
p.Modules[module] = state
}
// **The plan's own ask is its outcome, by id** (novox/hq ADR 0219); another build of the module
// is, as before, when it was asked at or after the plan's ask (issue 219).
// **A module asked under an id is answered by that id's outcome and no other** (novox/hq ADR
// 0219): a replay, a rebuild beside the plan, or an older ask finishing late is somebody else's
// build, made from other source, and settling the plan with it would send that. A plan from
// before ids were kept is matched as it was: by when the build was asked (issue 219).
if state.Build != "" {
if state.Build != id {
continue
}
} else if askedBefore(asked, state.AskedAt) {
// before ids were kept is matched as it was: by when the build was asked (issue 219). A walk
// waiting for its word, or a module it never asked, takes no outcome at all (issue 754).
state := p.Modules[module]
if !takesOutcome(*p, state, asked, id) {
continue
}
if failed != "" && deletedAtSource(failed) {
@@ -595,7 +675,7 @@ func advancePlans(ctx context.Context, open *stores) {
return
}
defer release()
// What waits for a gate, released one machine at a time (ADR 0236).
// What waits for a gate, released one machine first, then the rest at once (ADR 0236, ADR 0353).
if _, err := releaseBacklog(ctx, open, ""); err != nil {
fmt.Printf("plans: what waits for a gate could not be looked at: %v\n", err)
}
@@ -711,17 +791,11 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return true, nil
}
tier := p.Tiers[p.Tier]
// Not yet asked: ask.
unasked := 0
for _, m := range tier {
if s := p.Modules[m]; s == nil || s.State == "" {
unasked++
}
}
if unasked == len(tier) {
// Not yet asked: ask — the whole tier, or what of it is still unasked (novox/hq issue 754).
if unasked := unaskedOf(*p); len(unasked) > 0 {
// **One walk at a time** (novox/hq ADR 0276): a walk about to ask its first tier while another started
// walk is open waits for that one to end, let go or not, and says so.
if p.Tier == 0 {
if p.Tier == 0 && !askedInTier(*p) {
if behind, err := startedBeside(ctx, inv, p.ID, p.Created); err != nil {
return false, err
} else if behind != "" {
@@ -731,7 +805,10 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return changed, nil
}
}
if err := askTier(ctx, inv, p); err != nil {
if len(unasked) < len(tier) {
fmt.Printf("%s: tier %d was asked in part; asking the rest: %s\n", p.ID, p.Tier, strings.Join(unasked, ", "))
}
if err := askTier(ctx, inv, p, unasked); err != nil {
return false, err
}
return true, nil
@@ -805,7 +882,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
} else if same {
now := time.Now().UTC()
state.SentAt = &now
state.Why = "no move: made from the source every machine running it runs"
state.Why = noMoveWhy
fmt.Printf("%s: %s built from the source every machine running it runs: no move, nothing sent\n", p.ID, m)
continue
}
@@ -814,6 +891,12 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if err != nil {
return false, err
}
// **Every machine at once** (novox/hq ADR 0360 §1, §2): a module that rolls out is sent to every machine
// running it in one step, with no first machine and no gate, whatever module it is. A module a controller
// before ADR 0360 already sent to its first machine finishes under its gate.
if !gateFirstMachine && state.FirstAt == nil {
policy.Together = true
}
var reports []inventory.Reported
if !policy.Together {
// Read for the choice of the first machine as well as for its report.
@@ -995,6 +1078,25 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if state.BuiltAt != nil {
since = *state.BuiltAt
}
// **A build that is no move is waited on by the send that last carried it** (novox/hq issue 648).
// Nothing was sent for it, so no report follows its marking or its build: a machine that does not
// happen to report again held the walk for ever. Each machine runs it once it answered its last
// send — the one whose builds made it no move — whenever that was.
if noMove(state) {
byNode := map[string]inventory.Reported{}
for _, r := range reports {
byNode[r.Node] = r
}
if ok, on := appliedEach(m, func(node string) time.Time {
if s := byNode[node].Sent; s != nil {
return *s
}
return time.Time{}
}, running, reports); !ok {
waiting = append(waiting, fmt.Sprintf("%s on %s", m, strings.Join(on, ", ")))
}
continue
}
// **Each machine from its own send** (novox/hq issue 256). With one machine first (ADR 0218)
// a module is sent twice — the first machine, then the rest — and SentAt is the second.
// Asked of every machine, the first machine's report, made between the two sends, read as
@@ -1024,15 +1126,92 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
return changed, nil
}
}
// **A walk ends on its machines' reports** (novox/hq ADR 0360 §4, §5): after its last tier, every machine it sent
// has reported on that send, or is silent past ApplySilentAfter; never a health reading.
end := ""
if p.Tier == len(p.Tiers)-1 {
reports, err := inv.LastReports(ctx)
if err != nil {
return false, err
}
waiting, failed, silent := walkReports(*p, reports, time.Now().UTC())
if len(waiting) > 0 {
note := "sent; waiting for " + strings.Join(waiting, ", ") + " to report on their send"
changed := p.State != inventory.PlanRolling || p.Note != note
p.State = inventory.PlanRolling
p.Note = note
return changed, nil
}
var said []string
if len(failed) > 0 {
said = append(said, "failed or refused its send (apply-failing says it): "+strings.Join(failed, ", "))
}
if len(silent) > 0 {
said = append(said, fmt.Sprintf("said nothing on its send within %s: %s", inventory.ApplySilentAfter,
strings.Join(silent, ", ")))
}
end = strings.Join(said, "; ")
// **The new controller's user list follows its walk** (ADR 0236 §34, read without the gate since ADR 0360):
// once a walk that moved the controller ends, the bus's machine is sent the list it composes, if behind.
if s := p.Modules[catalogue.ControllerSeatName]; s != nil && s.SentAt != nil && s.Gate == nil && !noMove(s) {
carryUserList(ctx, open, p)
}
}
p.Tier++
p.State = inventory.PlanBuilding
p.Note = ""
p.Note = end
if p.Tier < len(p.Tiers) {
fmt.Printf("%s: tier %d done; asking tier %d: %s\n", p.ID, p.Tier-1, p.Tier, strings.Join(p.Tiers[p.Tier], ", "))
}
return true, nil
}
// walkReports is, for every machine a walk sent its rest, whether it reported on that send (novox/hq ADR 0360 §4):
// those still waiting, those that reported a failed or refused apply, and those silent past ApplySilentAfter,
// each sorted. Pure.
func walkReports(p inventory.Plan, reports []inventory.Reported, now time.Time) (waiting, failed, silent []string) {
byNode := map[string]inventory.Reported{}
for _, r := range reports {
byNode[r.Node] = r
}
seen := map[string]bool{}
for _, s := range p.Modules {
if s == nil {
continue
}
for node, sent := range s.Rest {
if seen[node] {
continue
}
r := byNode[node]
at := sent.At
if at.IsZero() && s.SentAt != nil {
at = *s.SentAt
}
switch {
case r.At != nil && sent.ReportsOn(r):
seen[node] = true
if r.Outcome == inventory.OutcomeFailed || r.Outcome == inventory.OutcomeRefused {
failed = append(failed, node)
}
case !at.IsZero() && now.Sub(at) > inventory.ApplySilentAfter:
seen[node] = true
silent = append(silent, node)
default:
if !slices.Contains(waiting, node) {
waiting = append(waiting, node)
}
}
}
}
// A machine one module's send was reported on and another's not yet is the same send: reported.
waiting = slices.DeleteFunc(waiting, func(n string) bool { return seen[n] })
sort.Strings(waiting)
sort.Strings(failed)
sort.Strings(silent)
return waiting, failed, silent
}
// restOf is, for one module, every machine of its rest that the send reached and what it carried there.
func restOf(to, sent []string, what map[string]inventory.SentDeclaration) map[string]inventory.SentDeclaration {
out := map[string]inventory.SentDeclaration{}
@@ -1056,10 +1235,36 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
runningOf map[string][]string) error {
inv := open.inventory
before, beforeKnown, _ := inv.SentBuilds(ctx, node)
// **A machine already sent what this send is for is not sent again** (novox/hq ADR 0353 §6): sent by this plan
// before a controller was replaced and before it was kept, or by another walk. Judged from that send, and from
// the readings of a gate already judging it there.
// Only a send the machine reported it applied is one it has; one it never reported on is sent, as before
// (issue 280: what it ran before is then that build, and a failure leaves it as it was).
if beforeKnown && slices.IndexFunc(modules, func(m string) bool { return !sameCommit(before[m], p.Modules[m].Commit) }) < 0 &&
appliedLastSend(ctx, inv, node) {
if others, err := gatedMoves(ctx, open, node, nil); err == nil && len(others) == 0 {
adoptSend(ctx, open, p, node, modules, before, runningOf)
return nil
}
}
// **What the machine ran before is kept before the send is made** (novox/hq ADR 0353 §6): a controller replaced
// between the send and keeping it reads it here, and a failure still puts the machine back.
kept := false
for _, m := range modules {
s := p.Modules[m]
if beforeKnown && (s.Previous == "" || !sameCommit(before[m], s.Commit)) && s.Previous != before[m] {
s.Previous, kept = before[m], true
}
}
if kept {
if err := inv.SavePlan(ctx, p); err != nil {
return fmt.Errorf("what %s ran before is not kept, so nothing is sent: %w", node, err)
}
}
owns := make([]inventory.CarriedMove, 0, len(modules))
for _, m := range modules {
s := p.Modules[m]
owns = append(owns, inventory.CarriedMove{Module: m, Node: node, From: before[m], To: s.Commit, Build: s.Build})
owns = append(owns, inventory.CarriedMove{Module: m, Node: node, From: s.Previous, To: s.Commit, Build: s.Build})
}
// A gated send (ADR 0236): everything waiting on the machine goes with the tier, and the gate judges
// all of it there.
@@ -1074,10 +1279,7 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
sentWhat := sentNow(ctx, inv, sent)
for _, m := range modules {
s := p.Modules[m]
// What the first machine ran before: what a failed gate puts back (ADR 0236).
if beforeKnown {
s.Previous = before[m]
}
// What the first machine ran before (ADR 0236), kept above before the send.
s.First, s.FirstAt = sent, &now
s.Gate = &inventory.PlanGate{Component: coreComponent(m), Machines: firstRunning(sent, runningOf[m]),
From: s.Previous, To: s.Commit, Since: &now, Sent: sentWhat}
@@ -1106,6 +1308,113 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
return nil
}
// adoptSend records a first send the machine already had (novox/hq ADR 0353 §6, §7): nothing is sent; the gate counts
// from the machine's last send, and takes over the readings of an open gate of another walk judging the same build
// there — whose carried move also says what the machine ran before, for a rollback.
func adoptSend(ctx context.Context, open *stores, p *inventory.Plan, node string, modules []string,
before map[string]string, runningOf map[string][]string) {
inv := open.inventory
since := time.Now().UTC()
last, found, err := inv.SentTo(ctx, node)
if err == nil && found && !last.At.IsZero() {
since = last.At
}
lead := modules[0]
other, id, from := judgingThere(ctx, inv, p.ID, lead, node, p.Modules[lead].Commit, p.Modules[lead].Build)
for _, m := range modules {
s := p.Modules[m]
// What it ran before: kept by this walk before its send was made (§6); else what the other walk's send
// moved it from; else the build itself, so a failure leaves it as it was (issue 280).
switch {
case s.Previous != "":
case m == lead && from != "":
s.Previous = from
default:
s.Previous = before[m]
}
at := since
s.First, s.FirstAt = []string{node}, &at
s.Gate = &inventory.PlanGate{Component: coreComponent(m), Machines: firstRunning([]string{node}, runningOf[m]),
From: s.Previous, To: s.Commit, Since: &at}
if found {
s.Gate.Sent = map[string]inventory.SentDeclaration{node: last}
}
s.GatedBy = ""
if m == lead {
s.Gate.TakeReadings(other, modules, id)
} else {
s.GatedBy = lead
}
}
p.State = inventory.PlanRolling
p.Note = fmt.Sprintf("tier %d built; %s already runs %s's build: nothing sent again, judged from that send",
p.Tier, node, strings.Join(modules, ", "))
if id != "" {
p.Note += ", going on from the readings of " + id
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
}
// appliedLastSend says a machine reported it applied the declaration it was sent last.
func appliedLastSend(ctx context.Context, inv *inventory.Inventory, node string) bool {
reports, err := inv.LastReports(ctx)
if err != nil {
return false
}
for _, r := range reports {
if r.Node == node {
return r.Current && r.At != nil && r.Outcome == inventory.OutcomeApplied
}
}
return false
}
// judgingThere is an open gate of another walk judging a module's build on a machine (novox/hq ADR 0353 §7), the
// walk's id, and what the machine ran before; nil when none does.
func judgingThere(ctx context.Context, inv *inventory.Inventory, self, module, node, commit, build string) (*inventory.PlanGate, string, string) {
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, "", ""
}
// The same build, by its id where both name one, else by its commit.
same := func(b, c string) bool {
if b != "" && build != "" {
return b == build
}
return sameCommit(c, commit)
}
usable := func(g *inventory.PlanGate) bool {
return g != nil && !g.Pending && g.Since != nil && (g.Verdict == "" || g.Verdict == inventory.GatePassed) &&
slices.Contains(g.Machines, node)
}
for _, q := range plans {
if q.ID == self {
continue
}
if r := q.Release; r != nil {
gates := []*inventory.PlanGate{r.Gate}
for _, g := range r.Gates {
gates = append(gates, g)
}
for _, g := range gates {
if !usable(g) {
continue
}
for _, c := range g.Carried {
if c.Module == module && c.Node == node && same(c.Build, c.To) {
return g, q.ID, c.From
}
}
}
continue
}
if s := q.Modules[module]; s != nil && usable(s.Gate) && same(s.Build, s.Commit) {
return s.Gate, q.ID, s.Previous
}
}
return nil, "", ""
}
// passedWith keeps, on a module sent with others, the verdict of the gate that judged the send: the gate
// on the first of them passed, and its pass was kept for every build it carried (passCarried).
func passedWith(module string, s *inventory.PlanModule, lead string, g *inventory.PlanGate) {
@@ -1594,9 +1903,19 @@ func plansCommand(ctx context.Context, args []string) error {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
orNone(strings.Join(r.Done, ", ")), orNone(strings.Join(r.Skipped, ", ")))
if r.Gate != nil {
fmt.Printf(" %s\n", gateLine(r.Gate))
for _, c := range r.Gate.Carried {
if len(r.Failed)+len(r.Silent) > 0 {
fmt.Printf(" failed: %s; silent: %s\n", orNone(strings.Join(r.Failed, ", ")), orNone(strings.Join(r.Silent, ", ")))
}
gates := []*inventory.PlanGate{r.Gate}
for _, n := range r.Order {
gates = append(gates, r.Gates[n])
}
for _, g := range gates {
if g == nil {
continue
}
fmt.Printf(" %s\n", gateLine(g))
for _, c := range g.Carried {
fmt.Printf(" %-22s %s → %s\n", c.Module, short(c.From), short(c.To))
if c.Recreates != "" {
fmt.Printf(" %-22s %s\n", "", c.Recreates)
+14 -12
View File
@@ -1,6 +1,7 @@
package main
import (
"slices"
"testing"
"time"
@@ -49,8 +50,10 @@ func TestAMergeIsPlannedInTiersAlongTheThreeKindsOfDependency(t *testing.T) {
if pos["mesh-tools"] != 0 || pos["builder"] != 1 {
t.Fatalf("the runtime then the builder: %v", tiers)
}
if !(pos["shop"] > pos["builder"] && pos["postgres"] > pos["builder"]) {
t.Fatalf("what the builder builds comes after the builder: %v", tiers)
// Built-by orders nothing (novox/hq ADR 0360 §3): what the builder builds is beside it, after the runtime it
// stands on.
if pos["shop"] != 1 || pos["postgres"] != 1 {
t.Fatalf("what the builder builds waits for the builder: %v", tiers)
}
if pos["shop-plugin"] <= pos["shop"] {
t.Fatalf("a plugin after what it is declared on: %v", tiers)
@@ -67,24 +70,23 @@ func TestAMergeIsPlannedInTiersAlongTheThreeKindsOfDependency(t *testing.T) {
// A change to a package all three build from moves all three. The builder holds
// the build seat, whose worker the controller defines, so it follows the controller (worker-of,
// novox/hq issue 206), and the controller's built-by edge to it yields: the controller is built by the
// build machine that is running. The proxy is built by the new builder: the controller, the builder,
// the proxy — the live plan of every controller merge. (This read "the builder, then the controller
// and the proxy together" before issue 206, and the fixture had no worker-of edge.)
// build machine that is running. The proxy is built by whichever builder runs, so it goes beside the
// controller (novox/hq ADR 0360 §3; it read "the controller, the builder, the proxy" before).
small := reachableFrom([]string{"mesh-controller", "builder", "route-proxy"}, edges)
smallTiers := tiersOf(small, edges)
if got := tiered(smallTiers); got != "mesh-controller | builder | route-proxy" {
t.Fatalf("the controller, then the builder, then the proxy: %v", smallTiers)
if got := tiered(smallTiers); got != "mesh-controller,route-proxy | builder" {
t.Fatalf("the controller and the proxy, then the builder: %v", smallTiers)
}
// The builder alone moved: the builder, and nothing it builds.
if only := reachableFrom([]string{"builder"}, edges); len(only) != 1 {
t.Fatalf("a build machine change rebuilds the build machine alone: %v", only)
}
// Only a runtime dependency gates on deployment, and only when the module rolls out.
// Nothing waits on deployment for a built-by edge (novox/hq ADR 0360 §3): the builder gates no tier after it.
p := planOfMerge(link.SourceMoved{Owner: "novox", Repo: "mesh-tools", Commit: "abc"}, []string{"mesh-tools"}, edges)
p.Tier = pos["builder"]
rollsOut := func(m string) bool { return m == "builder" }
if g := gates(p, edges, rollsOut); len(g) != 1 || g[0] != "builder" {
if g := gates(p, edges, rollsOut); len(g) != 0 {
t.Fatalf("the builder gates the tier after it: %v", g)
}
p.Tier = 0
@@ -253,10 +255,10 @@ func TestAChangeToTheBuildAgentRebuildsTheBuildAgentAlone(t *testing.T) {
t.Fatalf("%v planned %v: the build agent alone", paths, p.Tiers)
}
}
// With what it builds moved beside it, the agent comes first and they follow: an order, not a widening.
// With what it builds moved beside it, both go in one tier (novox/hq ADR 0360 §3): built-by orders nothing.
p := planOfMerge(link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Commit: "abc"},
[]string{"build-agent", "redis"}, edges)
if len(p.Tiers) != 2 || p.Tiers[0][0] != "build-agent" || p.Tiers[1][0] != "redis" || len(p.Modules) != 2 {
t.Fatalf("the agent, then what moved beside it: %v", p.Tiers)
if len(p.Tiers) != 1 || !slices.Equal(p.Tiers[0], []string{"build-agent", "redis"}) || len(p.Modules) != 2 {
t.Fatalf("the agent and what moved beside it, in one tier: %v", p.Tiers)
}
}
+8 -1
View File
@@ -111,8 +111,13 @@ func aBacklog(t *testing.T) *backlogMesh {
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest, carried); err != nil {
return nil, err
}
// A broken machine reports its apply failed (novox/hq ADR 0360 §7: the report is the verdict).
outcome := inventory.OutcomeApplied
if b.broken[node] {
outcome = inventory.OutcomeFailed
}
if _, err := open.inventory.RecordDoing(ctx, nodeID(t, open, node), inventory.Doing{Node: node,
Outcome: inventory.OutcomeApplied, Declared: digest, Applied: 1, At: time.Now()}); err != nil {
Outcome: outcome, Declared: digest, Applied: 1, At: time.Now()}); err != nil {
return nil, err
}
}
@@ -164,6 +169,7 @@ func (b *backlogMesh) release(t *testing.T) inventory.Plan {
// The backlog goes out one machine at a time: the first judged before the second is sent, each build's
// pass kept; a rebuild with the same bytes is no move at all; and a send outside a gate is refused.
func TestTheBacklogIsReleasedOneMachineAtATimeEachJudged(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
@@ -219,6 +225,7 @@ func TestTheBacklogIsReleasedOneMachineAtATimeEachJudged(t *testing.T) {
// A release that fails on its first machine puts back what it carried there, goes no further, and the
// next one waits for a person, said as a condition; a person releases it with why.
func TestAFailedReleaseIsPutBackAndTheNextWaitsForAPerson(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
+1
View File
@@ -31,6 +31,7 @@ import (
// nothing is put back, the walk goes on to the workstation, and openrazer's condition says, in one sentence
// for the operator, that a new login is needed on the laptop.
func TestReplay318(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
+2
View File
@@ -16,6 +16,7 @@ import (
// at its gate with the pass's own words, and put it back. Written with only what the controller had before
// its fix, so it is laid over the commit before.
func TestReplay335(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
t.Run("the first machine's later reports", testAPassedGateIsNotJudgedAgainFromTheFirstMachinesLaterReports)
t.Run("a walk stopped after the pass", testAWalkStoppedAfterItsGatePassedKeepsThePass)
}
@@ -77,6 +78,7 @@ func testAWalkStoppedAfterItsGatePassedKeepsThePass(t *testing.T) {
// the first send is older than the wait for a first machine's report. Read before the pass is taken over, the
// carried module's step said the first machine never applied it, and the passed build was put back.
func TestACarriedModuleTakesItsLeadsPassBeforeItsStepIsRead(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
tm := aTierMesh(t, "m01", "m02")
ctx := t.Context()
+1
View File
@@ -19,6 +19,7 @@ import (
// had before its fix — the stores, register, assign, a plan saved and advanced, the watchdogs' gathering and
// seeing — so it is laid over the commit before.
func TestReplay336(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
+1
View File
@@ -139,6 +139,7 @@ func TestReplay273AConsumerBesideItsStoreStaysBoundToIt(t *testing.T) {
// one, kept below. Heard as bytes, so an older controller reads them as it reads any report — this file is
// written only with what the controller had before the judging, for the prover to lay over that commit.
func TestReplayCrashLoopFailsItsGateOnTheFirstMachine(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
+382
View File
@@ -0,0 +1,382 @@
package main
import (
"context"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq ADR 0353, kept behind gateFirstMachine for walks a controller before ADR 0360 started: after its first machine passes, a build goes to every other machine at once, each judged by
// its own gate and a failing one put back by itself; and a gate's readings outlive the controller that took them —
// a machine already sent what a send is for is not sent again, and its judging goes on from what was read.
//
// Seen 2026-10-14: release-1791930194184991192 sent ace, g14, shanks and the control node one after another in
// eleven minutes; the new controller its last send started sent the control node again the controller build it
// had just applied, and judged it from its first reading.
// aBacklogOnThree is the backlog fixture with a third machine, desk, running app at c1 like the others.
func aBacklogOnThree(t *testing.T) *backlogMesh {
t.Helper()
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
if _, err := inv.AddNode(ctx, "desk"); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "desk", "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, b.open, "desk"), "d-desk", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
wasHeard := releaseHeard
releaseHeard = func(context.Context, *stores) (map[string]bool, error) {
return map[string]bool{"anchor": true, "laptop": true, "desk": true}, nil
}
t.Cleanup(func() { releaseHeard = wasHeard })
return b
}
// events records every send and every judging, in order.
func events(t *testing.T) *[]string {
t.Helper()
var said []string
wasSend, wasGather := sendRollout, gatherGateFacts
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
said = append(said, "send "+strings.Join(names, ","))
return wasSend(ctx, open, names)
}
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
said = append(said, "judge")
return wasGather(ctx, open, component)
}
t.Cleanup(func() { sendRollout, gatherGateFacts = wasSend, wasGather })
return &said
}
// Once the first machine passed, the rest are both sent before either is judged (ADR 0353 §1); before, the second
// was judged to its verdict before the third was sent.
func TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
said := events(t)
gateEvery = 0
for i := 0; i < 12; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanDone || len(p.Release.Done) != 3 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
desk, laptop := slices.Index(*said, "send desk"), slices.Index(*said, "send laptop")
if desk < 0 || laptop < 0 || slices.Index(*said, "send anchor") > min(desk, laptop) {
t.Fatalf("sends and judgings: %v", *said)
}
if slices.Contains((*said)[min(desk, laptop):max(desk, laptop)], "judge") {
t.Fatalf("the rest were sent one at a time, each judged before the next: %v", *said)
}
if v, found, err := b.open.inventory.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
}
// A machine of the rest whose gate fails is put back at once, and the other machine of the step goes on to its own
// verdict (ADR 0353 §3); before, the walk stopped at the failure and the machine after it was never sent.
func TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
inv := b.open.inventory
gateEvery = 0
b.broken["desk"] = true // desk fails what it is sent in the rest's step
advancePlans(ctx, b.open)
for i := 0; i < 12 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Failed, []string{"desk"}) {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if !slices.ContainsFunc(b.sent, func(s []string) bool { return slices.Equal(s, []string{"laptop"}) }) {
t.Fatalf("laptop was never sent: %v", b.sent)
}
if !slices.Contains(p.Release.Done, "laptop") {
t.Fatalf("laptop has no verdict of its own: %+v", p.Release)
}
// late waited on laptop alone and was healthy there: its pass is kept, whatever desk did.
if v, found, err := inv.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
// app failed on desk: marked, and put back wherever it was sent (ADR 0236).
if failed, _ := inv.GateFailed(ctx, "build-app-c2"); !failed {
t.Fatal("app's build is not marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "desk"); sent["app"] != "c1" {
t.Fatalf("desk was not put back: %v", sent)
}
}
// A walk whose first machine already runs its build under another walk's gate sends nothing and goes on from that
// gate's readings (ADR 0353 §7); before, it sent the machine again and judged from its first reading.
func TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
advancePlans(ctx, b.open) // the release sends anchor app c2, and reads it once
release := b.release(t)
judging := release.Release.Gate
if judging == nil || judging.Pending || judging.Passes == 0 {
t.Fatalf("the release is not judging anchor: %+v", judging)
}
built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-app", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2",
Created: built, State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {State: "built", BuiltAt: &built, Commit: "c2",
Build: "build-app-c2"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if len(b.sent) != 1 {
t.Fatalf("anchor was sent again what it runs: %v", b.sent)
}
p, err := inv.PlanByID(ctx, "plan-app")
if err != nil {
t.Fatal(err)
}
g := p.Modules["app"].Gate
if g == nil || g.Since == nil || !g.Since.Equal(*judging.Since) || g.Passes < judging.Passes || g.TookOver != release.ID {
t.Fatalf("the walk's gate starts from zero: %+v (the release's: %+v)", g, judging)
}
if p.Modules["app"].Previous != "c1" {
t.Fatalf("what anchor ran before is %q, not c1", p.Modules["app"].Previous)
}
}
// A walk's first send that was made and not kept — the controller replaced between the two — is not made again
// by the controller that takes over: the gate counts from that send (ADR 0353 §5, §6).
func TestAFirstSendMadeAndNotKeptIsNotSentAgain(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
// The old controller sends anchor app c2 and is replaced before it keeps the send.
made := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if _, err := made(ctx, open, names); err != nil {
return nil, err
}
return nil, context.Canceled
}
advancePlans(ctx, g.open)
sendRollout = made
if len(g.sent) != 1 || g.plan(t).Modules["app"].FirstAt != nil {
t.Fatalf("the fixture's send was not made, or was kept: %v", g.sent)
}
last, _, err := g.open.inventory.SentTo(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
if len(g.sent) != 1 {
t.Fatalf("the send was made again: %v", g.sent)
}
s := g.plan(t).Modules["app"]
if s.FirstAt == nil || s.Gate == nil || s.Gate.Since == nil || !s.Gate.Since.Equal(last.At) {
t.Fatalf("the gate does not count from the send made: %+v %+v", s, s.Gate)
}
// What anchor ran before was kept before the send, so a failure still puts it back (§5).
if s.Previous != "c1" || s.Gate.From != "c1" {
t.Fatalf("what anchor ran before is %q (gate from %q), not c1", s.Previous, s.Gate.From)
}
}
// A release's send kept before it was made, and made, is judged by the controller that takes over without being
// made again (ADR 0353 §8).
func TestAReleaseSendKeptAndMadeIsNotMadeAgain(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
p, err := releaseBacklog(ctx, b.open, "")
if err != nil || p == nil {
t.Fatalf("no release opened: %v", err)
}
if _, err := advanceRelease(ctx, b.open, p); err != nil || p.Release.Gate == nil || !p.Release.Gate.Pending {
t.Fatalf("the send was not kept before it was made: %+v %v", p.Release, err)
}
if err := inv.SavePlan(ctx, p); err != nil {
t.Fatal(err)
}
// The send is made, and the controller is replaced before it keeps it.
if _, err := sendRollout(ctx, b.open, []string{"anchor"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if len(b.sent) != 1 {
t.Fatalf("anchor was sent again: %v", b.sent)
}
g := b.release(t).Release.Gate
if g == nil || g.Pending || g.Since == nil || len(g.Carried) == 0 {
t.Fatalf("anchor is not judged on the send made: %+v", g)
}
}
// A build no gate has passed goes to one machine of a step, its first; a machine that would be its second waits
// for the next step, and one whose every build passed goes now (ADR 0353 §2).
func TestABuildThatPassedNoGateGoesToOneMachineOfAStep(t *testing.T) {
withTheGate(t)
waiting := map[string][]inventory.CarriedMove{
"desk": {{Module: "app", To: "c2"}, {Module: "new", To: "n1"}},
"laptop": {{Module: "app", To: "c2"}, {Module: "new", To: "n1"}},
"shanks": {{Module: "app", To: "c2"}},
"novox": {{Module: "app", To: "c2"}, {Module: "ctl", To: "k1"}},
}
passed := func(module, commit string) bool { return module == "app" && commit == "c2" }
never := func(string) bool { return false }
step := restStep([]string{"desk", "laptop", "shanks", "novox"}, waiting, passed, never)
if !slices.Equal(step, []string{"desk", "shanks", "novox"}) {
t.Fatalf("the step is %v: laptop would be new's second machine before it passed", step)
}
if step := restStep([]string{"laptop"}, waiting, func(string, string) bool { return true }, never); !slices.Equal(step, []string{"laptop"}) {
t.Fatalf("a machine whose every build passed waits: %v", step)
}
}
// The node-engine and the mesh's resolver go to the rest one machine at a time, even passed (ADR 0353 §9): a step
// carries either to one machine only, while what else passed goes everywhere at once.
func TestTheNodeEngineAndTheResolverGoToTheRestOneAtATime(t *testing.T) {
withTheGate(t)
waiting := map[string][]inventory.CarriedMove{
"desk": {{Module: hostModule, To: "h2"}},
"laptop": {{Module: hostModule, To: "h2"}, {Module: "app", To: "c2"}},
"shanks": {{Module: "app", To: "c2"}},
"novox": {{Module: "dns", To: "d2"}},
"ace": {{Module: "dns", To: "d2"}},
}
resolver := catalogue.Manifest{Module: "dns", Claims: []catalogue.Claim{{Name: resolverSeat}}}
alone := func(m string) bool {
if m == "dns" {
return oneAtATime(m, resolver)
}
return oneAtATime(m, catalogue.Manifest{Module: m})
}
step := restStep([]string{"desk", "laptop", "shanks", "novox", "ace"}, waiting,
func(string, string) bool { return true }, alone)
if !slices.Equal(step, []string{"desk", "shanks", "novox"}) {
t.Fatalf("the step is %v: the node-engine and the resolver each reach one machine of it", step)
}
if oneAtATime("app", catalogue.Manifest{Module: "app"}) {
t.Fatal("a leaf module is held to one machine at a time")
}
}
// A first machine that fails still stops the walk (ADR 0236 §5, kept by ADR 0353 §3): the rest are never sent.
func TestAFirstMachineThatFailsStillStopsTheWalk(t *testing.T) {
withTheGate(t)
b := aBacklogOnThree(t)
ctx := t.Context()
gateEvery = 0
b.broken["anchor"] = true
advancePlans(ctx, b.open)
for i := 0; i < 6 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
if p := b.release(t); p.State != inventory.PlanFailed || p.Release.Rest {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
for _, s := range b.sent {
if !slices.Equal(s, []string{"anchor"}) {
t.Fatalf("sent %v: the rest after a failed first machine", b.sent)
}
}
}
// The control node is sent last in a step of the rest (ADR 0353 §1): the machine holding the controller is
// ordered after every other.
func TestTheControlNodeIsSentLastInAStep(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
register(t, b.open, catalogue.Manifest{Module: catalogue.ControllerSeatName, Version: "1"})
if _, err := b.open.inventory.Assign(ctx, "anchor", catalogue.ControllerSeatName); err != nil {
t.Fatal(err)
}
said := events(t)
gateEvery = 0
for i := 0; i < 12 && (len(b.sent) == 0 || b.release(t).Open()); i++ {
advancePlans(ctx, b.open)
}
if p := b.release(t); !slices.Equal(p.Release.Order, []string{"desk", "laptop", "anchor"}) {
t.Fatalf("the order is %v", p.Release.Order)
}
laptop, anchor := slices.Index(*said, "send laptop"), slices.Index(*said, "send anchor")
if laptop < 0 || anchor < laptop || slices.Contains((*said)[laptop:anchor], "judge") {
t.Fatalf("the control node is not sent last in the rest's step: %v", *said)
}
}
// A machine of the rest that says nothing on its send is left silent after fifteen minutes: not failed, not put
// back; a rest that went silent whole ends failed, said as a condition (ADR 0353 §4).
func TestARestMachineSilentPastItsBoundIsLeftAndSaid(t *testing.T) {
withTheGate(t)
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
reporting := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if slices.Equal(names, []string{"laptop"}) { // laptop sleeps: sent, and says nothing
b.sent = append(b.sent, names)
current, _ := open.inventory.CurrentBuilds(ctx)
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "laptop"), "d-laptop-asleep",
map[string]string{"app": current["app"].Commit, "late": current["late"].Commit, "same": current["same"].Commit})
}
return reporting(ctx, open, names)
}
t.Cleanup(func() { sendRollout = reporting })
gateEvery = 0
for i := 0; i < 8; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
g := p.Release.Gates["laptop"]
if !p.Open() || g == nil || g.Verdict != "" || g.Silent {
t.Fatalf("laptop is judged before fifteen minutes: %s %s %+v", p.State, p.Note, g)
}
since := time.Now().Add(-inventory.ApplySilentAfter - time.Minute)
g.Since = &since
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
p = b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Silent, []string{"laptop"}) || len(p.Release.Failed) > 0 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if failed, _ := inv.GateFailed(ctx, "build-late-c2"); failed {
t.Fatal("a silent machine's build was marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "laptop"); sent["late"] != "c2" {
t.Fatalf("laptop was put back: %v", sent)
}
obs, err := silentObservation(ctx, inv)
if err != nil {
t.Fatal(err)
}
if len(obs) != 1 || obs[0].Kind != "release-silent" || !slices.Equal(obs[0].Also, []string{"laptop"}) {
t.Fatalf("the silence is not said: %+v", obs)
}
}
+176
View File
@@ -0,0 +1,176 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A module moves only when its program changes** (novox/hq issue 721). A merge check builds every module the
// delivery plan moves and compares each with the build the mesh holds of it; a walk is cut without a module whose
// check found its whole image the same as that build, while the mesh still holds it, and says so among the modules
// it left unchanged. A merge of one sentence in a package a program imports but never reaches moves no build of it.
// sameAsRunning are the modules of a walk its checks found the same as the build the mesh holds, each with what its
// check said. **Only when every pull request the walk answers was checked at the head it merged, and every one of
// those checks built the module and found it the same as the build the mesh still holds**: a merge the walk answers
// without a pull request, a check of another head, a check that built it differently, or a held build that moved
// since, leaves the module in the walk.
func sameAsRunning(plan *inventory.Plan, builds []link.CheckedBuild, held map[string]string) map[string]string {
if plan.Delivery == nil || len(plan.Delivery.Merges) == 0 {
return nil
}
for _, m := range plan.Delivery.Merges {
if _, ok := pullOf(m); !ok {
return nil
}
}
pulls := reuseAskOf(plan).Pulls
heads := mergedHeadsOf(plan)
out := map[string]string{}
for name := range plan.Modules {
said := ""
for _, pull := range pulls {
b, ok := checkedBuildOf(builds, pull, heads[pull], name)
if !ok || !stillHeld(b.Built, name, held) {
said = ""
break
}
if said == "" {
said = b.Built.Same
}
}
if said != "" {
out[name] = said
}
}
if len(out) == 0 {
return nil
}
return out
}
// checkedBuildOf is the newest build a check of the pull request at head made of the module; false when none did.
func checkedBuildOf(builds []link.CheckedBuild, pull link.PullRef, head, module string) (link.CheckedBuild, bool) {
if head == "" {
return link.CheckedBuild{}, false
}
for _, b := range builds {
if b.Pull == pull && b.Built.Module == module && link.SameHead(b.Head, head) {
return b, true
}
}
return link.CheckedBuild{}, false
}
// stillHeld is whether a check found every artifact of the module the same as a build the mesh still holds.
func stillHeld(b link.CheckBuilt, module string, held map[string]string) bool {
if b.Same == "" || b.NotBuilt != "" || len(b.Made) == 0 || len(b.SameAs) != len(b.Made) {
return false
}
for _, m := range b.Made {
was := pinnedDigest(b.SameAs[m.Name])
if was == "" || was != pinnedDigest(held[module+"/"+m.Name]) {
return false
}
}
return true
}
// pinnedDigest is a reference's pinned digest, empty when it is not pinned.
func pinnedDigest(ref string) string {
if i := strings.LastIndex(ref, "@"); i >= 0 {
return ref[i+1:]
}
return ""
}
// leaveSameAsRunning takes from a walk being cut every module its checks found the same as the build the mesh holds
// (sameAsRunning), and every dependent in the walk only for one of those (withDependentsOf): what was taken, with why. **Anything not read leaves the walk as it was**, said.
func leaveSameAsRunning(ctx context.Context, inv *inventory.Inventory, plan *inventory.Plan, moved []string,
edges []inventory.Edge) map[string]string {
if plan.Delivery == nil || len(plan.Modules) == 0 {
return nil
}
pulls := reuseAskOf(plan).Pulls
if len(pulls) == 0 {
return nil
}
builds, err := checkedBuildsOf(ctx, pulls)
if err != nil {
fmt.Printf(" the checks' builds could not be read, so every module moves: %v\n", err)
// empty-on-error: said above; nothing left out of the walk is the safe answer, every module moves as before
return nil
}
held, err := inv.RegisteredHeld(ctx)
if err != nil {
fmt.Printf(" what the mesh holds could not be read, so every module moves: %v\n", err)
// empty-on-error: said above; nothing left out of the walk is the safe answer, every module moves as before
return nil
}
same := withDependentsOf(sameAsRunning(plan, builds, held), plan, moved, edges)
if len(same) == 0 {
return nil
}
names := make([]string, 0, len(same))
for n := range same {
names = append(names, n)
}
sort.Strings(names)
withhold(plan, names, nil)
for _, n := range names {
fmt.Printf(" %s left out: %s\n", n, same[n])
}
return same
}
// withDependentsOf adds to the modules found the same every module of the walk that is in it only as a dependent of
// one of them (novox/hq ADR 0267 rule 3): **a dependent is left out when its base comes out the same**, since the
// base it stands on stays the build the mesh holds. A module a merge moves itself, or a dependent of a base that
// still moves, stays. moved are the modules the walk's merges move; edges the graph the walk was planned along.
func withDependentsOf(same map[string]string, plan *inventory.Plan, moved []string, edges []inventory.Edge) map[string]string {
if len(same) == 0 {
return same
}
var still []string
for _, n := range moved {
if same[n] == "" {
still = append(still, n)
}
}
keep := map[string]bool{}
for _, n := range reachableFrom(still, edges) {
keep[n] = true
}
// Until nothing more is found: a dependent of a dependent left out is left out too.
for grew := true; grew; {
grew = false
for n := range plan.Modules {
if keep[n] || same[n] != "" {
continue
}
var bases []string
for _, e := range edges {
ordering := e.Kind == inventory.EdgeBuiltBy || e.Kind == inventory.EdgeWorkerOf ||
e.Kind == inventory.EdgePackages || e.Kind == edgeGroupOrder
if !ordering && e.From == n && same[e.To] != "" && !slices.Contains(bases, e.To) {
bases = append(bases, e.To)
}
}
sort.Strings(bases)
if len(bases) == 0 {
// In the walk for no base found the same: whatever brought it in is not said here, so it moves.
continue
}
same[n] = "unchanged: it is in the walk only for " + strings.Join(bases, ", ") +
", which came out the same as the build the mesh holds (ADR 0267 rule 3)"
grew = true
}
}
return same
}
+141
View File
@@ -0,0 +1,141 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// TestAWalkLeavesOutAModuleWhoseProgramDidNotChange is novox/hq issue 721: mesh-controller #335 changed one sentence
// that mesh-builder never reaches; its check found build-agent the same as the build the mesh holds, so the walk moves
// mesh-controller alone and says build-agent is unchanged.
func TestAWalkLeavesOutAModuleWhoseProgramDidNotChange(t *testing.T) {
pull := link.PullRef{Owner: "novox", Repo: "mesh-controller", Number: 335}
head := "34050dc6aaaabbbbccccdddd"
aPlan := func() *inventory.Plan {
return &inventory.Plan{Tiers: [][]string{{"mesh-controller", "build-agent"}},
Modules: map[string]*inventory.PlanModule{"mesh-controller": {}, "build-agent": {}},
Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{{Repository: "novox/mesh-controller",
Commit: "c2ec0a60", Number: 335, Head: head}}}}
}
held := map[string]string{"build-agent/agent": "store/build-agent@sha256:aaaa",
"mesh-controller/controller": "store/mesh-controller@sha256:cccc"}
builds := []link.CheckedBuild{
{Pull: pull, Head: head[:12], Built: link.CheckBuilt{Module: "build-agent", Key: "k1",
Made: []link.MadeArtifact{{Name: "agent", Kind: "image", Reference: "scratch/build-agent@sha256:bbbb"}},
Same: "unchanged: its program is the same as running build agent aaaa",
SameAs: map[string]string{"agent": "10.0.0.1:5000/build-agent@sha256:aaaa"}}},
{Pull: pull, Head: head[:12], Built: link.CheckBuilt{Module: "mesh-controller", Key: "k2",
Made: []link.MadeArtifact{{Name: "controller", Kind: "image", Reference: "scratch/mesh-controller@sha256:dddd"}}}},
}
p := aPlan()
same := sameAsRunning(p, builds, held)
if len(same) != 1 || same["build-agent"] == "" {
t.Fatalf("the walk leaves out %v, want build-agent alone", same)
}
withhold(p, []string{"build-agent"}, nil)
if len(p.Tiers) != 1 || len(p.Tiers[0]) != 1 || p.Tiers[0][0] != "mesh-controller" {
t.Fatalf("the walk moves %v, want mesh-controller alone", p.Tiers)
}
// **The held build moved since the check**: what it was compared with no longer runs.
moved := map[string]string{"build-agent/agent": "store/build-agent@sha256:eeee"}
if same := sameAsRunning(aPlan(), builds, moved); len(same) != 0 {
t.Fatalf("a module compared with a build the mesh no longer holds is left out: %v", same)
}
// **A check of another head** says nothing of this merge.
other := aPlan()
other.Delivery.Merges[0].Head = "ffffffffffffffff"
if same := sameAsRunning(other, builds, held); len(same) != 0 {
t.Fatalf("a check of another head left out %v", same)
}
// **Another pull request the walk answers** that did not find it the same leaves it in.
two := aPlan()
two.Delivery.Merges = append(two.Delivery.Merges, inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: "1234", Number: 9, Head: "abcdefabcdefabcdef"})
if same := sameAsRunning(two, builds, held); len(same) != 0 {
t.Fatalf("a module another merge moves is left out: %v", same)
}
// **A merge with no pull request** leaves the walk as it is.
direct := aPlan()
direct.Delivery.Merges[0].Number = 0
if same := sameAsRunning(direct, builds, held); len(same) != 0 {
t.Fatalf("a walk answering a merge without a pull request left out %v", same)
}
}
// TestADependentOfABaseThatCameOutTheSameIsLeftOut is ADR 0267 rule 3 (novox/hq issue 607): a dependent is in a walk
// for its base; when the base comes out the same as the build the mesh holds, the dependent stands on that build
// still, and leaves the walk with it — and so does a dependent of that dependent. One a merge moves itself stays.
func TestADependentOfABaseThatCameOutTheSameIsLeftOut(t *testing.T) {
edges := []inventory.Edge{{From: "app", To: "base", Kind: inventory.EdgeDeclared},
{From: "app2", To: "app", Kind: inventory.EdgeDeclared}, {From: "own", To: "base", Kind: inventory.EdgeDeclared}}
plan := &inventory.Plan{Modules: map[string]*inventory.PlanModule{"base": {}, "app": {}, "app2": {}, "own": {}}}
same := withDependentsOf(map[string]string{"base": "unchanged: its program is the same"}, plan,
[]string{"base", "own"}, edges)
if !strings.Contains(same["app"], "only for base") || !strings.Contains(same["app2"], "only for app") {
t.Fatalf("the dependents of a base that came out the same are not left out: %v", same)
}
if _, left := same["own"]; left {
t.Fatalf("a module its merge moves itself is left out for its base: %v", same)
}
// A dependent of a base that still moves stays.
edges = append(edges, inventory.Edge{From: "app", To: "own", Kind: inventory.EdgeDeclared})
same = withDependentsOf(map[string]string{"base": "unchanged"}, plan, []string{"base", "own"}, edges)
if _, left := same["app"]; left {
t.Fatalf("a dependent of a base that still moves is left out: %v", same)
}
}
// Against the store, the path a real merge takes (novox/hq issues 721 and 611): a merge whose check built its one
// module the same as the build the mesh holds is cut into a walk done with nothing to walk, whose record names the
// module unchanged and why — the walk mesh-delivery reads as delivered, never stranded.
func TestAWalkWhoseEveryBuildCameOutTheSameIsDone(t *testing.T) {
open := windowed(t)
endsItsLine(t)
asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-one", Repository: "novox/one", Ref: "main",
Module: "one", Commit: "c0", On: "builder", Asked: t0.Add(-time.Hour),
Made: []inventory.Artifact{{Name: "one", Kind: "image", Reference: "store/one@sha256:aaaa"}},
Sources: []inventory.BuildSource{{Paths: []string{"main.go", "go.mod"}}}}); err != nil {
t.Fatal(err)
}
// What runs: the module registered with that build's image (RegisteredHeld reads the module table).
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "one", Version: "1", Resources: []map[string]any{
{"id": "app", "type": "container", "name": "one", "image": "store/one@sha256:aaaa"}}},
inventory.Source{Repository: "novox/one", Ref: "main", BuiltFrom: "c0", Asked: time.Now()}); err != nil {
t.Fatal(err)
}
pull := link.PullRef{Owner: "novox", Repo: "one", Number: 7}
was := checkedBuildsOf
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return []link.CheckedBuild{{Pull: pull, Head: "headaaaaaaaa", Built: link.CheckBuilt{Module: "one", Key: "k",
Made: []link.MadeArtifact{{Name: "one", Kind: "image", Reference: "scratch/one@sha256:bbbb"}},
Same: "unchanged: its program is the same as running build one aaaa",
SameAs: map[string]string{"one": "10.0.0.1:5000/one@sha256:aaaa"}}}}, nil
}
t.Cleanup(func() { checkedBuildsOf = was })
code := repoMerge("one", "one1aaaaaaaa", t0)
code.Number, code.HeadSHA = 7, "headaaaaaaaa1111"
hear(t, open, code, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("walks: %+v", ws)
}
w := ws[0]
if w.State != inventory.PlanDone || len(w.Tiers) != 0 || len(w.Modules) != 0 {
t.Fatalf("the walk reads %s, tiers %v, modules %v; want done with nothing to walk", w.State, w.Tiers, w.Modules)
}
if w.Delivery == nil || !strings.Contains(w.Delivery.Unchanged["one"], "its program is the same as running build") {
t.Fatalf("the walk does not say one matched what runs: %+v", w.Delivery)
}
}
+3
View File
@@ -43,6 +43,9 @@ func TestMain(m *testing.M) {
}
if os.Getenv(echoEnvironment) != "" {
fmt.Printf("verb=%q caller=%q terminal=%v\n", os.Getenv("MESH_VERB"), os.Getenv("MESH_CALLER"), startedAtTheTerminal())
if os.Getenv(echoFails) != "" {
os.Exit(3)
}
os.Exit(0)
}
conditions.Unsayable = func(o conditions.Observation, field string, r outward.Refusal) {
+28
View File
@@ -457,6 +457,28 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, err
}
return []string{"forget-checks", str("owner"), str("repo"), str("number")}, nil
case "try":
// A branch tried before its pull request (novox/hq issue 686): asked, or read by the id it answered.
if id := str("id"); id != "" {
return []string{"try", "--id", id}, nil
}
if a.given["repository"] == "" && a.given["ref"] == "" {
// Neither shape named: the command says its usage, which names both, as `build` does.
return []string{"try"}, nil
}
if err := need("repository", "ref"); err != nil {
return nil, err
}
argv := []string{"try", str("repository"), str("ref")}
for _, flag := range []string{"base", "tests"} {
if v := str(flag); v != "" {
argv = append(argv, "--"+flag, v)
}
}
if on("store-only") {
argv = append(argv, "--store-only")
}
return argv, nil
case "cancel", "kill":
if err := need("id"); err != nil {
return nil, err
@@ -1018,6 +1040,8 @@ var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "co
"delivery": true,
// The build queue, each ask with its position among the waiting (novox/hq issue 455).
"queue": true,
// A branch tried before its pull request (novox/hq issue 686).
"try": true,
// A plan followed (novox/hq issue 229): `plans wait` answers JSON; its other forms say text, and so answer no data.
"plans": true}
@@ -1308,6 +1332,10 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
link.AnswersWithin(ctx, bound+planWaitStart)
}
}
if verb == "try" && len(argv) > 1 && argv[1] == "--id" {
// Reading a try holds its answer for the verdict, up to its bound (novox/hq issue 686, 229).
link.AnswersWithin(ctx, tryWait+planWaitStart)
}
if answer, read := readHere(ctx, argv); read {
return answer, nil
}
+59 -3
View File
@@ -153,11 +153,15 @@ var signalsTable = []signalRow{
return newestOf(f.standings, func(c conditions.Condition) time.Time { return c.LastObserved })
}},
{Row: "S9", Signal: "bus advisories: maximum deliveries, consumer deleted; the controller's own slow " +
"consumer and refused subjects", Emitter: "bus server's advisory subjects; the controller's connection",
"consumer and refused subjects; every user's connection the server closed as a slow consumer or for " +
"breaking the bus's rules (ADR 0357)",
Emitter: "bus server's advisory subjects; the controller's connection; the bus module's watch of its " +
"server (bus.slow-consumer, bus.connection-closed)",
Trigger: "any", Bound: "any occurrence; clears after an hour without another, a deleted consumer " +
"once it exists again or the mesh no longer expects it, and a message given up on once DEAD_LETTERS " +
"no longer holds it (novox/hq issue 330)",
Kind: "slow-consumer, max-deliveries, refused, consumer-lost", Severity: conditions.Warning, Phase: 1,
Kind: "slow-consumer, connection-closed, max-deliveries, refused, consumer-lost", Severity: conditions.Warning,
Phase: 1,
needs: func(f *signalFacts) error { return f.advisoriesErr }, watch: watchAdvisories,
newest: func(f *signalFacts) time.Time {
return newestOf(f.advisories, func(a link.Advisory) time.Time { return a.Last })
@@ -569,6 +573,9 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
// The advisory is remembered and read again on every look for an hour: the condition counts
// what the bus said and when, not the looks (novox/hq issue 402).
Happened: a.Last, Times: a.Count}
if a.User != "" && a.Reason != "" {
busUserClosed(&o, a, f.host)
}
if a.Kind == link.AdvisoryMaxDeliveries && a.Token == link.AdvisoryNotKept {
o.Machine = consumerMachine(a.Stream, a.Consumer)
o.Headline = clip(conditions.Capital(fmt.Sprintf("%s gave up on a message, not kept",
@@ -579,9 +586,58 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
}
out = append(out, o)
}
if f.busFaultsErr != nil {
out = append(out, busFaultsUnread(f.busFaultsErr))
}
return append(out, watchDeadLetters(f)...)
}
// kindBusFaultsUnread is the bus module's events not read back (novox/hq ADR 0357): every other user's
// slow consumer is unheard while it lasts, and the controller's own signals are still said.
const kindBusFaultsUnread = "bus-faults-unread"
func busFaultsUnread(err error) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeBus, ID: "bus-module", Kind: kindBusFaultsUnread,
Severity: conditions.Warning, Confirm: true,
Summary: "what the bus module said of its users' connections could not be read back, so a slow consumer " +
"or a connection the bus closed is not said until it can",
Said: firstLine(err.Error())}
}
// busUserClosed words an advisory about a bus user's connection the server closed (novox/hq ADR 0357): the
// user as the operator says it and the machine it is on, the dotted user name kept in the evidence only.
func busUserClosed(o *conditions.Observation, a link.Advisory, host string) {
who, machine := busUserWords(a.User)
if machine == "" && a.User == "controller" {
machine = host
}
o.Machine = machine
times, count := "once", "once"
if a.Count > 1 {
times = fmt.Sprintf("%d times since %s", a.Count, a.First.UTC().Format("15:04 MST"))
count = fmt.Sprintf("%d times", a.Count)
}
switch a.Kind {
case link.AdvisorySlowConsumer:
o.Summary = fmt.Sprintf("the bus closed the connection of %s as a slow consumer %s in the last hour, "+
"newest at %s: it could not take what it was sent fast enough, and what was on its way to it was lost",
who, times, a.Last.UTC().Format("15:04 MST"))
o.Headline = clip(conditions.Capital(who)+" was too slow for the bus", 60)
o.Explanation = conditions.Capital(fmt.Sprintf("%s could not keep up with what the bus sent it, so the bus "+
"closed its connection (%s in the last hour) and what was on its way was lost. It connects again "+
"by itself; this clears an hour after the last time.", who, count))
o.Resolved = "Resolved: " + who + " kept up for an hour"
default:
o.Summary = fmt.Sprintf("the bus closed the connection of %s for breaking its rules %s in the last hour, "+
"newest at %s: %s", who, times, a.Last.UTC().Format("15:04 MST"), strings.ToLower(a.Reason))
o.Headline = clip(conditions.Capital(who)+" was cut off by the bus", 60)
o.Explanation = conditions.Capital(fmt.Sprintf("The bus closed the connection of %s because it broke "+
"the bus's rules (%s, %s in the last hour), so what it was sending was lost. This clears an hour "+
"after the last time.", who, strings.ToLower(a.Reason), count))
o.Resolved = "Resolved: " + who + " kept to the bus's rules for an hour"
}
}
// watchDeadLetters says each consumer that DEAD_LETTERS holds a message for (novox/hq issue 330): open
// while it holds any, so it clears when they are delivered again or dropped, never because the server
// stopped saying it.
@@ -806,7 +862,7 @@ func watchLease(f *signalFacts) []conditions.Observation {
}
}
if len(lost) > 0 {
how := "lost it: its renewal was refused or could not be made"
how := "lost it: another holder wrote its key, or no renewal was made within the key's age"
if newest.How == inventory.EpochExpired {
how = "stopped renewing it without giving it back, and was taken over"
}
+157
View File
@@ -0,0 +1,157 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// A new push to a pull request ends the check of its old head (novox/hq issue 726).
//
// A check judges one head; once the pull request has another, its verdict answers nobody and its run holds a build
// machine for minutes the new head waits behind. So when a check is asked for pull request N of a repository at a
// head, **every check of that pull request at another head ends**: a waiting one is dropped from the queue as
// `cancel` drops it, one in flight is ended on its machine as `kill` ends it. Each is said in the log.
//
// **Never anything else**: a try (`try`, asked for no pull request), a walk's build (no check), a delivery group's
// composed check (it answers several pull requests), another pull request's check, a dead ask (left for `clear
// --dead`), and a check of the same head — a recheck of that head is not another head.
// **Only for the pull request's current head.** Announcements of a pull request are not ordered, one is heard again
// after a restart, and a recheck asks for the head it recorded: a check asked for a head the pull request no longer
// has must never end the check of the head it has. So the forge is asked first which head the pull request has; only
// when it names this one does anything end, and when it cannot be asked, or names another, nothing ends and that is
// said.
// forgePullHead asks the forge, through its module's tools on the bus, which commit a pull request's head is. A
// variable so a test needs no forge.
var forgePullHead = func(ctx context.Context, js *broker.JetStream, owner, repo string, number int) (string, error) {
raw, _ := json.Marshal(map[string]any{"path": fmt.Sprintf("/repos/%s/%s/pulls/%d", owner, repo, number)})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return "", err
}
if answer.Error != "" {
return "", errors.New(answer.Error)
}
var pull struct {
Result struct {
Head struct {
SHA string `json:"sha"`
} `json:"head"`
} `json:"result"`
}
if err := json.Unmarshal(answer.Result, &pull); err != nil {
return "", fmt.Errorf("the forge's answer about the pull request cannot be read: %w", err)
}
if pull.Result.Head.SHA == "" {
return "", errors.New("the forge named no head for the pull request")
}
return pull.Result.Head.SHA, nil
}
// supersededChecks are the checks in the queue that a check of owner/repo#number at head ends: of the same pull
// request, at another head, waiting or in flight, and not the ask keep.
func supersededChecks(q link.Queue, owner, repo string, number int, head, keep string) []link.QueuedAsk {
if number <= 0 || head == "" {
return nil
}
var out []link.QueuedAsk
for _, a := range q.Asks {
c := a.Request.Check
switch {
case c == nil, link.IsTry(a.Request), c.Group != "", c.Number != number, a.ID == keep,
!strings.EqualFold(c.Owner, owner), !strings.EqualFold(c.Repo, repo),
a.State != link.AskWaiting && a.State != link.AskInFlight:
continue
}
ref := a.Request.Ref
if ref == "" {
ref = a.Ref
}
if ref == "" || ref == head || link.SameHead(ref, head) {
continue
}
out = append(out, a)
}
return out
}
// endSupersededChecks ends the checks a check of p at its head supersedes, keep being the new check's own ask, and
// says each in the log. A waiting one is cancelled — nothing recorded: a check builds nothing, and the forge's status
// of the old head is no longer the pull request's — and one a machine builds is killed there.
func endSupersededChecks(ctx context.Context, js *broker.JetStream, seat string, p link.PullUpdated, keep string) []string {
if p.Number <= 0 || p.Commit == "" {
return nil
}
current, err := forgePullHead(ctx, js, p.Owner, p.Repo, p.Number)
if err != nil {
return []string{fmt.Sprintf("%s/%s#%d (%.8s): nothing ended — the forge could not say the pull request's "+
"current head, so checks of its other heads keep running: %v", p.Owner, p.Repo, p.Number, p.Commit, err)}
}
if current != p.Commit && !link.SameHead(current, p.Commit) {
return []string{fmt.Sprintf("%s/%s#%d (%.8s): nothing ended — the pull request's head is %.12s now, not "+
"this one", p.Owner, p.Repo, p.Number, p.Commit, current)}
}
q, err := link.ReadQueue(ctx, js, seat)
if err != nil {
return []string{fmt.Sprintf("%s/%s#%d (%.8s): could not read the %s queue for checks of its earlier heads: %v",
p.Owner, p.Repo, p.Number, p.Commit, seat, err)}
}
var said []string
for _, a := range supersededChecks(q, p.Owner, p.Repo, p.Number, p.Commit, keep) {
old := a.Request.Ref
if old == "" {
old = a.Ref
}
what := fmt.Sprintf("check of %s/%s#%d at %.12s", p.Owner, p.Repo, p.Number, old)
var err error
if a.State == link.AskInFlight && a.On != "" {
var answer link.Answer
answer, err = link.AskSeatTool(ctx, js.Conn(), seat, "kill", a.On, map[string]string{"id": a.ID}, killAnswer)
if err == nil && answer.Error != "" {
err = fmt.Errorf("%s refused: %s", a.On, answer.Error)
}
} else {
_, err = cancelAsk(ctx, js, nil, seat, a)
}
if err != nil {
said = append(said, fmt.Sprintf("not ended: %s (%s) — superseded by %.12s, but: %v", what, a.ID, p.Commit, err))
continue
}
said = append(said, fmt.Sprintf("ended: %s — superseded by %.12s (%s, %s)", what, p.Commit, a.ID, a.State))
}
return said
}
// endingSupersededChecks ends them beside the announcement, so a kill waiting on its machine never holds the
// handler: on its own connection to the bus, within twice a kill's wait. A one-shot process (oneShot) ends them
// within the call instead, as its exit would cut them off.
func endingSupersededChecks(ctx context.Context, seat string, p link.PullUpdated, keep string, oneShot bool) {
end := func() {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*killAnswer)
defer cancel()
js, err := dialTheBus()
if err != nil {
fmt.Fprintf(os.Stderr, "%s/%s#%d (%.8s): could not reach the bus to end checks of its earlier heads: %v\n",
p.Owner, p.Repo, p.Number, p.Commit, err)
return
}
defer js.Close()
for _, line := range endSupersededChecks(ctx, js, seat, p, keep) {
fmt.Println(line)
}
}
if oneShot {
end()
return
}
go end()
}
+192
View File
@@ -0,0 +1,192 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// A new push to a pull request ends the check of its old head (novox/hq issue 726): the waiting one is dropped from
// the queue, the one in flight is killed on its machine, and each is said. A try, a walk's build, another pull
// request's check, a group's check and a recheck of the new head are left alone. On the bus alone: no store.
func TestANewHeadEndsTheChecksOfTheOldOnes(t *testing.T) {
js, _ := aBuildQueue(t)
ctx := t.Context()
const (
oldHead = "1111111111111111111111111111111111111111"
olderHead = "2222222222222222222222222222222222222222"
newHead = "3333333333333333333333333333333333333333"
)
publish := func(r link.BuildRequest) {
t.Helper()
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(link.BuildWork(), body); err != nil {
t.Fatal(err)
}
}
check := func(id string, number int, head string) link.BuildRequest {
return link.BuildRequest{ID: id, Repository: "https://forge.example/novox/a.git", Ref: head,
Check: &link.CheckRequest{Owner: "novox", Repo: "a", Number: number}}
}
// In flight first: taken by a holder, started on ace.
publish(check("check-running", 7, olderHead))
worker, _ := broker.HolderConsumerFor("", "", broker.DeclaredSeat{Name: link.TheBuildMachine, Accepts: []string{"build"}})
sub, err := js.Context().PullSubscribe(worker.Filters[0], worker.Name, nats.Bind(worker.Stream, worker.Name), nats.ManualAck())
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
if _, err := sub.Fetch(1, nats.MaxWait(3*time.Second)); err != nil {
t.Fatal(err)
}
started, _ := json.Marshal(link.BuildStart{ID: "check-running", On: "ace", At: time.Now().UTC().Format(time.RFC3339Nano)})
if _, err := js.Context().Publish(link.BuildStarted(), started); err != nil {
t.Fatal(err)
}
publish(check("check-old", 7, oldHead))
publish(check("check-other-pull", 8, oldHead))
publish(check("check-recheck", 7, newHead))
publish(check("check-new", 7, newHead))
try := check("try-old", 7, oldHead)
try.Check.Number, try.Check.Try = 0, true
publish(try)
group := check("group-old", 7, oldHead)
group.Check.Group = "g1"
publish(group)
publish(link.BuildRequest{ID: "walk-build", Repository: "https://forge.example/novox/a.git", Ref: oldHead})
var mu sync.Mutex
var killed []string
stop, err := link.OverNATS{Conn: js.Conn()}.ServeNodeSeatTools(link.TheBuildMachine, "ace", map[string]link.ToolHandler{
"kill": func(_ context.Context, raw json.RawMessage) (any, error) {
var args struct{ ID string }
_ = json.Unmarshal(raw, &args)
mu.Lock()
killed = append(killed, args.ID)
mu.Unlock()
return map[string]any{"said": "killed " + args.ID}, nil
}}, nil)
if err != nil {
t.Fatal(err)
}
defer stop()
forgeSays(t, newHead, nil)
said := endSupersededChecks(ctx, js, link.TheBuildMachine,
link.PullUpdated{Owner: "novox", Repo: "a", Number: 7, Commit: newHead}, "check-new")
mu.Lock()
if len(killed) != 1 || killed[0] != "check-running" {
t.Errorf("the holder was asked to kill %v, want only the in-flight check of the older head", killed)
}
mu.Unlock()
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if _, found := q.Find("check-old"); found {
t.Error("the waiting check of the old head is still in the queue")
}
for _, id := range []string{"check-other-pull", "check-recheck", "check-new", "try-old", "group-old", "walk-build"} {
if _, found := q.Find(id); !found {
t.Errorf("%s was ended; only checks of the same pull request at another head end", id)
}
}
text := strings.Join(said, "\n")
for _, want := range []string{
"ended: check of novox/a#7 at " + oldHead[:12] + " — superseded by " + newHead[:12],
"ended: check of novox/a#7 at " + olderHead[:12] + " — superseded by " + newHead[:12],
} {
if !strings.Contains(text, want) {
t.Errorf("the log does not say %q:\n%s", want, text)
}
}
if len(said) != 2 {
t.Errorf("said %d lines, want 2:\n%s", len(said), text)
}
}
// Which asks a check supersedes, read from the queue alone.
func TestSupersededChecksAreOnlyThePullRequestsOtherHeads(t *testing.T) {
ask := func(id, state, ref string, c *link.CheckRequest) link.QueuedAsk {
return link.QueuedAsk{ID: id, State: state, Ref: ref, Request: link.BuildRequest{ID: id, Ref: ref, Check: c}}
}
pr := func(n int) *link.CheckRequest { return &link.CheckRequest{Owner: "Novox", Repo: "A", Number: n} }
head := "abcdefabcdefabcdef0123456789abcdef012345"
q := link.Queue{Asks: []link.QueuedAsk{
ask("old", link.AskWaiting, "0123456789ab0123456789ab0123456789ab0123", pr(7)),
ask("short-same", link.AskWaiting, head[:12], pr(7)),
ask("dead", link.AskDead, "0123456789ab0123456789ab0123456789ab0123", pr(7)),
ask("other", link.AskWaiting, "0123456789ab0123456789ab0123456789ab0123", pr(8)),
ask("build", link.AskWaiting, "0123456789ab0123456789ab0123456789ab0123", nil),
ask("keep", link.AskWaiting, "0123456789ab0123456789ab0123456789ab0123", pr(7)),
}}
got := supersededChecks(q, "novox", "a", 7, head, "keep")
if len(got) != 1 || got[0].ID != "old" {
t.Fatalf("superseded %+v, want only old", got)
}
if got := supersededChecks(q, "novox", "a", 0, head, ""); got != nil {
t.Errorf("a check of no pull request superseded %+v", got)
}
}
// forgeSays makes the forge name head as every pull request's current head, or fail with err.
func forgeSays(t *testing.T, head string, err error) {
t.Helper()
was := forgePullHead
forgePullHead = func(context.Context, *broker.JetStream, string, string, int) (string, error) { return head, err }
t.Cleanup(func() { forgePullHead = was })
}
// An older head heard after a newer one — an announcement out of order, one heard again after a restart, a recheck
// of a recorded head — ends nothing: the forge says the pull request's head is the newer one. Nor does a check whose
// pull request's head the forge cannot say.
func TestAnOlderHeadHeardLateEndsNothing(t *testing.T) {
js, _ := aBuildQueue(t)
ctx := t.Context()
const (
oldHead = "1111111111111111111111111111111111111111"
newHead = "3333333333333333333333333333333333333333"
)
for _, id := range []string{"check-new", "check-old-late"} {
head := newHead
if id == "check-old-late" {
head = oldHead
}
body, _ := json.Marshal(link.BuildRequest{ID: id, Repository: "https://forge.example/novox/a.git", Ref: head,
Check: &link.CheckRequest{Owner: "novox", Repo: "a", Number: 7}})
if _, err := js.Context().Publish(link.BuildWork(), body); err != nil {
t.Fatal(err)
}
}
late := link.PullUpdated{Owner: "novox", Repo: "a", Number: 7, Commit: oldHead}
forgeSays(t, newHead, nil)
said := endSupersededChecks(ctx, js, link.TheBuildMachine, late, "check-old-late")
if len(said) != 1 || !strings.Contains(said[0], "nothing ended") {
t.Errorf("an older head heard late said %q", said)
}
forgeSays(t, "", errors.New("nothing serves gitea.gitea_api"))
said = endSupersededChecks(ctx, js, link.TheBuildMachine,
link.PullUpdated{Owner: "novox", Repo: "a", Number: 7, Commit: newHead}, "check-new")
if len(said) != 1 || !strings.Contains(said[0], "nothing ended") {
t.Errorf("a head the forge could not confirm said %q", said)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
for _, id := range []string{"check-new", "check-old-late"} {
if _, found := q.Find(id); !found {
t.Errorf("%s was ended", id)
}
}
}
+7
View File
@@ -171,6 +171,7 @@ func (tm *tierMesh) raise(machine, kind, summary string) {
// The incident's tier, small: a dozen modules whose first machine is the same are sent there in one
// send, judged by one gate, and the rest of their machines are sent once — two sends, not two dozen.
func TestATierOfManyModulesIsSentToEachMachineOnce(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
var modules []string
for i := 1; i <= 12; i++ {
@@ -223,6 +224,7 @@ func TestATierOfManyModulesIsSentToEachMachineOnce(t *testing.T) {
// settling, is the node tools' — inside their settle window, the gate's bound — and no failure of the
// modules sent with them: at the bound only the node tools are put back.
func TestACoreBehindWhileTheNodeToolsSettleFailsNoOtherModule(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
tm := aTierMesh(t, "app1", "app2", broker.RuntimeModule)
ctx := t.Context()
inv := tm.open.inventory
@@ -276,6 +278,7 @@ func TestACoreBehindWhileTheNodeToolsSettleFailsNoOtherModule(t *testing.T) {
// A machine-level condition about anything else holds the machine back as a whole: everything the send
// moved there fails together, once, with that reason — not one module for it.
func TestAMachineUnhealthyAsAWholeFailsTheWholeSendTogether(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
tm := aTierMesh(t, "app1", "app2", "app3")
ctx := t.Context()
@@ -307,6 +310,7 @@ func TestAMachineUnhealthyAsAWholeFailsTheWholeSendTogether(t *testing.T) {
// The machine-level conditions, sorted: one naming a module is that module's; the core behind is the
// core's when the send moved it and nobody's when it did not; anything else is the machine's as a whole.
func TestTheConditionsAboutAMachineItself(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
since := time.Now().Add(-time.Minute)
about := func(kind, id string) conditions.Condition {
return conditions.Condition{Key: "machine." + id + "." + kind, Kind: kind, Raised: time.Now(),
@@ -339,6 +343,7 @@ func TestTheConditionsAboutAMachineItself(t *testing.T) {
// rollback that failed; and `plans retry` asks it again rather than refusing a build that was never
// judged (the incident's "NOT put back: no build kept").
func TestAModuleItsSendChangedNothingOfIsLeftAsItWasAndRetried(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
@@ -394,6 +399,7 @@ func TestAModuleItsSendChangedNothingOfIsLeftAsItWasAndRetried(t *testing.T) {
// them, they are registered back and marked in the judging that finds them broken, the plan goes on judging
// the other module to its own pass, and only then fails.
func TestATierPutsABrokenModuleBackAtOnce(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
judgeLeavesAsTheCore(t)
for _, order := range [][]string{{broker.RuntimeModule, "app1"}, {"app1", broker.RuntimeModule}} {
t.Run("gate kept on "+order[0], func(t *testing.T) {
@@ -454,6 +460,7 @@ func TestATierPutsABrokenModuleBackAtOnce(t *testing.T) {
// decision 4): the leaf beside the node tools has its one reading at once, but the send is judged by the tool
// runner's rule: one reading, not before the node-engine's witness window for it has ended.
func TestASendCarryingTheToolRunnerWaitsForItsWitnessWindow(t *testing.T) {
withTheGate(t) // the first-machine gate\'s own path (novox/hq ADR 0360 keeps it for earlier walks)
tm := aTierMesh(t, broker.RuntimeModule, "app1")
ctx := t.Context()
// One reading counted per advance, as every test with a gate left open does: a gate judged with no time
+353
View File
@@ -0,0 +1,353 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"net/url"
"os"
"regexp"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// A branch is tried before its pull request (novox/hq issue 686).
//
// A repository's store tests skip without a store, and no agent may start a database on its machine. So a
// builder pushed, a reviewer read, and only the pull request's merge check ran those tests for the first time:
// each failure there cost a round of fifteen to thirty minutes (mesh-controller #283, #310, #312).
//
// `try <owner>/<repository> <ref>` asks the build seat for the repository's own check of a pushed branch or
// commit, as the merge check runs it (internal/builder/check.go): the ref merged into the trunk's head, the
// trunk's merge-check.sh, each part in its toolchain's container, against a throwaway store and bus. It is a
// check and nothing more:
//
// - **no pull request, so no status**: its outcome is never said as `checked`, which is what the forge's
// holder sets statuses from (isTry, in Built);
// - **nothing registered, nothing built**: it names no module of the graph, so the merge gate does not run and
// no build is made for a merge (novox/hq ADR 0350's scratch builds are a pull request's check's alone);
// - **a check in the build queue**: after deliveries (link.KindOf, novox/hq issue 558), and bounded by the
// check's own bound (checkTimeout);
// - **the code a pull request's check would run anyway**: only a repository of the mesh, whose every pull
// request the build seat already checks, in containers that hold no runtime socket.
//
// A check outlasts a seat verb's one answer, so the ask answers its id at once and `try --id <id>` reads what
// came of it from the build's log, where the build seat says it whole (link.StepTried): holding its answer up
// to tryWait for a verdict, like `plans wait` (novox/hq issue 229). Nothing is kept beside the log.
// tryWait is how long reading a try holds its answer for the verdict. A variable so a test need not wait.
var tryWait = 20 * time.Second
// A ref as the forge is asked for it — a branch or a commit, never an option — and a `go test -run` pattern.
var (
tryRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._/-]*$`)
tryTests = regexp.MustCompile(`^[A-Za-z0-9_^$|().*+?/\[\]-]+$`)
)
// tryAnswer is what `try` answers, asking or reading.
type tryAnswer struct {
ID string `json:"id"`
// State is asked (just now), waiting (nothing said of it yet: in the build queue), running, or the verdict:
// pass, warning (the repository declares no merge-check.sh), fail, or error (it could not run).
State string `json:"state"`
Said string `json:"said"`
Repository string `json:"repository,omitempty"`
Ref string `json:"ref,omitempty"`
Commit string `json:"commit,omitempty"`
Base string `json:"base,omitempty"`
Trunk string `json:"trunk,omitempty"`
On string `json:"on,omitempty"`
Took string `json:"took,omitempty"`
Failed []string `json:"failed,omitempty"`
// Last is the last thing the build seat said of one still running.
Last string `json:"last,omitempty"`
}
// isTry says a check's outcome is a try's: marked so, or — echoed by a build seat from before the mark — a
// check of no pull request and no group, which is nobody's status either.
func isTry(result link.BuildResult) bool {
c := result.Checked
return c != nil && (c.Try || (c.Number == 0 && c.Group == ""))
}
// asATry makes a check's ask a try's: the repository's own script alone, for no pull request.
func asATry(request link.BuildRequest, tests string, storeOnly bool) link.BuildRequest {
c := *request.Check
c.Number, c.Try, c.Tests, c.StoreOnly = 0, true, tests, storeOnly
c.Paths, c.Modules, c.New, c.Dependents, c.Manifests, c.Builds = nil, nil, nil, nil, nil, nil
c.Judge, c.Plan, c.Group, c.Members = "", nil, "", nil
request.Check = &c
return request
}
// maxTries is how many tries may wait or run at once, mesh-wide: each holds a build machine for minutes.
const maxTries = 4
// triesIn reads the build queue for a new try of one commit: the id of a try of that commit already waiting or
// running — the answer to a second ask — and whether there is room for another.
func triesIn(q link.Queue, owner, repo, commit string) (same string, room bool) {
n := 0
for _, a := range q.Asks {
if !link.IsTry(a.Request) || (a.State != link.AskWaiting && a.State != link.AskInFlight) {
continue
}
n++
c := a.Request.Check
if same == "" && strings.EqualFold(c.Owner, owner) && strings.EqualFold(c.Repo, repo) && a.Request.Ref == commit {
same = a.ID
}
}
return same, n < maxTries
}
// forgeCommit asks the forge, through its module's tools on the bus, which commit a branch or commit of a
// repository is, and the repository's default branch. A variable so a test needs no forge.
var forgeCommit = func(ctx context.Context, owner, repo, ref string) (commit, trunk string, err error) {
js, err := aBus()
if err != nil {
return "", "", err
}
defer js.Close()
bus := link.OverNATS{Conn: js.Conn()}
ask := func(path string, into any) error {
raw, _ := json.Marshal(map[string]any{"path": path})
answer, err := link.Ask(ctx, bus, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
return json.Unmarshal(answer.Result, into)
}
var commits struct {
Result []struct {
SHA string `json:"sha"`
} `json:"result"`
}
if err := ask("/repos/"+owner+"/"+repo+"/commits?limit=1&stat=false&verification=false&files=false&sha="+
url.QueryEscape(ref), &commits); err != nil {
return "", "", err
}
if len(commits.Result) == 0 || commits.Result[0].SHA == "" {
return "", "", errors.New("the forge named no commit for it")
}
var found struct {
Result struct {
DefaultBranch string `json:"default_branch"`
} `json:"result"`
}
if err := ask("/repos/"+owner+"/"+repo, &found); err != nil {
return "", "", err
}
return commits.Result[0].SHA, found.Result.DefaultBranch, nil
}
// tryCommand is `try <owner>/<repository> <ref> [--base <branch>] [--tests <pattern>] [--store-only]`, and
// `try --id <id>` to read what came of one.
func tryCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("try", flag.ContinueOnError)
base := set.String("base", "", "the branch it is merged into (the repository's default branch when none)")
tests := set.String("tests", "", "only the tests this `go test -run` pattern names")
storeOnly := set.Bool("store-only", false, "only the repository's store tests")
id := set.String("id", "", "a try's id: what came of it")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
usage := errors.New("try <owner>/<repository> <branch or commit> [--base <branch>] [--tests <pattern>] " +
"[--store-only], or try --id <id>")
if *id != "" {
if len(positionals) != 0 {
return usage
}
// Before it is a token of a bus subject: an id, never a pattern.
if !link.SafeBuildID(*id) {
return fmt.Errorf("%q is not a try's id", *id)
}
return tryRead(ctx, *id)
}
if len(positionals) != 2 {
return usage
}
owner, repo, isPath := strings.Cut(positionals[0], "/")
ref := positionals[1]
switch {
case !isPath || !tryRef.MatchString(owner) || !tryRef.MatchString(repo) || strings.Contains(repo, "/"):
return fmt.Errorf("%q is not a repository on the forge, owner/name; nothing was asked", positionals[0])
case !tryRef.MatchString(ref):
return fmt.Errorf("%q is not a branch or a commit; nothing was asked", ref)
case *base != "" && !tryRef.MatchString(*base):
return fmt.Errorf("%q is not a branch; nothing was asked", *base)
case *tests != "" && !tryTests.MatchString(*tests):
return fmt.Errorf("%q is not a `go test -run` pattern of test names; nothing was asked", *tests)
}
// **An unknown ref is refused here, loudly** — not sent to a machine to fail at its clone minutes later.
commit, trunk, err := forgeCommit(ctx, owner, repo, ref)
if err != nil {
return fmt.Errorf("the forge holds no branch or commit %s in %s/%s — is it pushed? — so nothing was asked: %w",
ref, owner, repo, err)
}
if *base != "" {
trunk = *base
}
if trunk == "" {
return fmt.Errorf("the forge named no default branch for %s/%s, and no --base says what %s is merged into; "+
"nothing was asked", owner, repo, ref)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
entries, read, edges, err := theGraph(ctx, open.inventory)
if err != nil {
return err
}
p := link.PullUpdated{Owner: owner, Repo: repo, Base: trunk, Commit: commit}
// Whose repository it is, as a pull request's check reckons it: only the mesh's is tried, the code the
// build seat runs for every pull request already.
mesh := pullScope(p, entries, read, edges)
if !mesh.Mesh {
return fmt.Errorf("%s/%s is no repository of the mesh — no module is built from it and its owner is not "+
"the core's — and only the mesh's are tried on the build seat; nothing was asked", owner, repo)
}
request, err := checkRequestFor(ctx, open, p, checkScope{Mesh: true, From: mesh.From}, entries)
if err != nil {
return err
}
request = asATry(request, *tests, *storeOnly)
seat := buildSeatHeld(ctx)
// **Bounded in number** (novox/hq issue 686): one try of a commit at a time, and maxTries in all.
js, err := dialTheBus()
if err != nil {
return err
}
queue, err := link.ReadQueue(ctx, js, seat)
js.Close()
if err != nil {
return err
}
same, room := triesIn(queue, owner, repo, commit)
if same != "" {
return printTry(tryAnswer{ID: same, State: "asked", Repository: owner + "/" + repo, Ref: ref, Commit: commit,
On: seat, Said: fmt.Sprintf("a try of %s/%s at %.12s already waits or runs, as %s: nothing new was asked. "+
"Read it with try id=%s", owner, repo, commit, same, same)})
}
if !room {
return fmt.Errorf("%d tries already wait or run on the build seat, the most at once (`queue` lists them); "+
"nothing was asked — read one of yours, or ask again when one has ended", maxTries)
}
ask, err := askOverOn(seat)
if err != nil {
return err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return err
}
return printTry(tryAnswer{ID: request.ID, State: "asked", Repository: owner + "/" + repo, Ref: ref, Commit: commit,
Base: trunk, On: seat, Said: fmt.Sprintf("asked %s to try %s/%s at %.12s (%s) merged into %s: its own "+
"merge-check.sh, as a check in the build queue, bounded at %s; no status is set and nothing is registered. "+
"Read it with try id=%s — a full run of this kind takes minutes", seat, owner, repo, commit, ref, trunk,
checkTimeout, request.ID)})
}
// tryRead answers what came of a try, holding the answer up to tryWait for its verdict.
func tryRead(ctx context.Context, id string) error {
until := time.Now().Add(tryWait)
for {
lines, err := buildLines(ctx, id)
if err != nil {
return err
}
a := tryAnswerOf(id, lines)
if (a.State != "waiting" && a.State != "running") || !time.Now().Add(2*time.Second).Before(until) {
return printTry(a)
}
if err := sleepOrDone(ctx, 2*time.Second); err != nil {
return err
}
}
}
// tryAnswerOf reads a try's answer from what the build seat said of it: its outcome under StepTried; `failed`
// for one that could not run — an error, never a pass; else how far it is.
func tryAnswerOf(id string, lines []link.BuildLine) tryAnswer {
a := tryAnswer{ID: id, State: "waiting", Said: "nothing is said of " + id + " yet: it waits in the build queue " +
"(`queue` says where), or no try has that id; ask again"}
for _, l := range lines {
switch l.Step {
case link.StepTried:
var t link.Tried
if err := json.Unmarshal([]byte(l.Message), &t); err != nil || t.Verdict == "" {
return tryAnswer{ID: id, State: "error", Said: "the build seat's word on it cannot be read: " + l.Message}
}
return tryAnswer{ID: id, State: t.Verdict, Said: t.Summary, Took: t.Took, Commit: t.Head, Trunk: t.Trunk,
Failed: t.Failed}
case "failed":
return tryAnswer{ID: id, State: "error", Said: "it could not run: " + l.Message}
case "checked":
// A check that is no try, or a build seat from before the verb: its verdict is not a try's.
return tryAnswer{ID: id, State: "error", Said: "the build seat checked it and said no try's outcome — " +
"it predates the verb (novox/hq issue 686), or " + id + " is no try: " + l.Message + "; `builds` with " +
"the id says every line"}
}
a.State, a.Last = "running", l.Message
a.Said = "still running; ask again. `builds` with the id says every line so far"
}
return a
}
func printTry(a tryAnswer) error {
fmt.Fprintln(os.Stderr, a.Said)
for _, l := range a.Failed {
fmt.Fprintln(os.Stderr, " "+l)
}
out, err := json.MarshalIndent(a, "", " ")
if err != nil {
return err
}
fmt.Println(string(out))
return nil
}
// buildLines is everything the build seat said of one build, as the bus keeps it, in order.
func buildLines(ctx context.Context, id string) ([]link.BuildLine, error) {
js, err := aBus()
if err != nil {
return nil, fmt.Errorf("cannot reach the bus to read a build's log: %w", err)
}
defer js.Close()
subject := link.BuildLogOf("*", id)
sub, err := js.Context().PullSubscribe(subject, "",
nats.BindStream(broker.EventsStream), nats.DeliverAll(), nats.AckNone())
if err != nil {
return nil, fmt.Errorf("cannot read %s from the bus: %w", subject, err)
}
defer func() { _ = sub.Unsubscribe() }()
var lines []link.BuildLine
for {
batch, err := sub.Fetch(200, nats.MaxWait(2*time.Second))
if err != nil && !errors.Is(err, nats.ErrTimeout) && !errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("reading a build's log: %w", err)
}
for _, msg := range batch {
var line link.BuildLine
if json.Unmarshal(msg.Data, &line) == nil {
lines = append(lines, line)
}
}
if len(batch) < 200 {
return lines, nil
}
}
}
+194
View File
@@ -0,0 +1,194 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// A branch is tried before its pull request (novox/hq issue 686): the repository's own check asked for a pushed
// branch or commit, with no pull request.
// What a try asks is a check — after deliveries in the build queue, bounded as one — of the repository's own
// script alone: no module of the graph, so no gate; nothing built for a merge, so no reusable image.
func TestATryAsksACheckOfTheRepositorysOwnScriptAlone(t *testing.T) {
request := link.BuildRequest{ID: "build-1", Ref: "c0ffee", Check: &link.CheckRequest{Owner: "novox", Repo: "a",
Number: 7, Base: "main", Modules: []string{"a"}, New: []string{"b"}, Manifests: []string{"module.json"},
Dependents: []string{"c"}, Builds: []link.CheckBuild{{Module: "a"}}, Judge: link.JudgeSelf,
Plan: &link.ChangePlan{}}}
tried := asATry(request, "^TestStore", true)
c := tried.Check
if !c.Try || c.Number != 0 || c.Tests != "^TestStore" || !c.StoreOnly {
t.Fatalf("not marked a try, or it kept a pull request's number: %+v", c)
}
if len(c.Modules)+len(c.New)+len(c.Manifests)+len(c.Dependents)+len(c.Builds) != 0 || c.Judge != "" || c.Plan != nil {
t.Fatalf("a try runs the gate or builds for a merge: %+v", c)
}
if c.Base != "main" || c.Owner != "novox" || tried.Ref != "c0ffee" {
t.Fatalf("a try lost what it merges into what: %+v", tried)
}
if link.KindOf(tried) != link.KindCheck {
t.Fatal("a try does not count as a check in the build queue")
}
}
// A try's outcome is heard and left there: no `checked` is said, so the forge's holder sets no status, and
// nothing is recorded or registered (the handler holds no store here, and would fail reaching for one).
func TestATrysOutcomeSetsNoStatusAndRegistersNothing(t *testing.T) {
said := &saidOnTheBus{}
checkEvents = said
defer func() { checkEvents = nil }()
for name, result := range map[string]link.BuildResult{
"a try": {ID: "build-1", Ref: "c0ffee", Checked: &link.CheckRequest{Owner: "novox", Repo: "a", Try: true},
Check: &link.CheckOutcome{Verdict: "pass", RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "failed"}}},
"a try that could not run": {ID: "build-2", Checked: &link.CheckRequest{Owner: "novox", Repo: "a", Try: true},
Failed: "cannot clone"},
// A build seat from before the mark echoes the ask without it: a check of no pull request and no
// group is still nobody's status.
"a try echoed by an older build seat": {ID: "build-3", Checked: &link.CheckRequest{Owner: "novox", Repo: "a"},
Check: &link.CheckOutcome{Verdict: "pass"}},
} {
if err := (builds{}).Built(context.Background(), result); err != nil {
t.Fatalf("%s: %v", name, err)
}
}
if len(said.events) != 0 {
t.Fatalf("a try's outcome was said for the forge: %v", said.events)
}
// A pull request's check is still said.
_ = (builds{}).Built(context.Background(), link.BuildResult{ID: "build-4", Checked: &link.CheckRequest{
Owner: "novox", Repo: "a", Number: 7}, Check: &link.CheckOutcome{Verdict: "pass"}})
if len(said.events) != 1 {
t.Fatalf("a pull request's check was said %d time(s)", len(said.events))
}
}
// saidOnTheBus keeps the seat events said, and nothing else.
type saidOnTheBus struct {
link.Bus
events []string
}
func (s *saidOnTheBus) PublishSeatEvent(_ context.Context, seat, event string, _ []byte) error {
s.events = append(s.events, seat+" "+event)
return nil
}
// A ref the forge does not hold is refused before anything is asked, naming it.
func TestATryOfAnUnknownRefIsRefusedLoudly(t *testing.T) {
was := forgeCommit
defer func() { forgeCommit = was }()
forgeCommit = func(_ context.Context, owner, repo, ref string) (string, string, error) {
return "", "", errors.New("404 object does not exist [id: " + ref + "]")
}
err := tryCommand(context.Background(), []string{"novox/mesh-controller", "fix/no-such-branch"})
if err == nil {
t.Fatal("a try of a ref the forge does not hold was asked")
}
for _, want := range []string{"fix/no-such-branch", "novox/mesh-controller", "nothing was asked", "404"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
for _, args := range [][]string{{"mesh-controller", "main"}, {"novox/mesh-controller"}, {"novox/a", "--upload-pack=x"},
{"novox/a", "main", "--tests", "x; rm -rf /"}, {"novox/a", "main", "--id", "build-1"}} {
if err := tryCommand(context.Background(), args); err == nil {
t.Errorf("try %v was taken", args)
}
}
}
// The answer of a try carries the lines that name what failed (novox/hq issue 460's shape) and how long it took.
func TestATrysAnswerCarriesTheFailingLines(t *testing.T) {
tried, _ := json.Marshal(link.Tried{Verdict: "fail", Summary: "its merge-check.sh failed: TestX", Took: "5m3s",
Head: "c0ffee", Trunk: "ca3f43c", Failed: []string{"--- FAIL: TestX (0.01s)", " x_test.go:12: got 1, want 2",
"FAIL\tgithub.com/novox/a/cmd\t3.1s"}})
lines := []link.BuildLine{{Seq: 1, Step: "check", Message: "novox/a#0 at c0ffee"},
{Seq: 2, Step: "output", Message: "ok github.com/novox/a/internal 1.2s"},
{Seq: 3, Step: link.StepTried, Message: string(tried)}, {Seq: 4, Step: "checked", Message: "pass: x"}}
a := tryAnswerOf("build-1", lines)
if a.State != "fail" || a.Took != "5m3s" || len(a.Failed) != 3 || a.Failed[0] != "--- FAIL: TestX (0.01s)" ||
a.Trunk != "ca3f43c" || a.ID != "build-1" {
t.Fatalf("the answer lost the verdict, the time or the failing lines: %+v", a)
}
// Not answered yet: said as running, with how far it is — never as a pass.
if a := tryAnswerOf("build-1", lines[:2]); a.State != "running" || a.Last == "" {
t.Fatalf("a try still running: %+v", a)
}
if a := tryAnswerOf("build-1", nil); a.State != "waiting" {
t.Fatalf("a try nothing was said of: %+v", a)
}
// One that could not run is an error with the build seat's word, never a pass.
if a := tryAnswerOf("build-1", append(lines[:1:1], link.BuildLine{Seq: 2, Step: "failed",
Message: "cannot clone"})); a.State != "error" || !strings.Contains(a.Said, "cannot clone") {
t.Fatalf("a try that could not run: %+v", a)
}
}
// One try of a commit at a time, and four in all (novox/hq issue 686): a second ask is answered the first's id,
// a fifth is refused; a try that ended, and a pull request's check, count for nothing.
func TestTriesAreBoundedInNumber(t *testing.T) {
ask := func(id, repo, ref, state string, try bool) link.QueuedAsk {
return link.QueuedAsk{ID: id, State: state, Request: link.BuildRequest{ID: id, Ref: ref,
Check: &link.CheckRequest{Owner: "novox", Repo: repo, Try: try}}}
}
q := link.Queue{Asks: []link.QueuedAsk{
ask("t1", "a", "c1", link.AskInFlight, true), ask("t2", "a", "c2", link.AskWaiting, true),
ask("t3", "b", "c1", link.AskDead, true), ask("p1", "a", "c9", link.AskWaiting, false)}}
if same, room := triesIn(q, "novox", "a", "c2"); same != "t2" || !room {
t.Fatalf("a second try of a commit: %q %v", same, room)
}
if same, room := triesIn(q, "novox", "b", "c1"); same != "" || !room {
t.Fatalf("a dead try, or another repository's, was taken for this one: %q %v", same, room)
}
if same, _ := triesIn(q, "novox", "a", "c9"); same != "" {
t.Fatalf("a pull request's check was taken for a try: %q", same)
}
q.Asks = append(q.Asks, ask("t4", "c", "c3", link.AskWaiting, true), ask("t5", "d", "c4", link.AskWaiting, true))
if _, room := triesIn(q, "novox", "e", "c5"); room {
t.Fatal("a fifth try was let in")
}
// An id is checked before it is a token of a bus subject.
for _, id := range []string{"x.>", "*", "a b"} {
if err := tryCommand(context.Background(), []string{"--id", id}); err == nil ||
!strings.Contains(err.Error(), "not a try's id") {
t.Errorf("try --id %q: %v", id, err)
}
}
}
// The verb runs its command, and is declared with the shell habit it replaces (novox/hq ADR 0245).
func TestTheTryVerbRunsItsCommand(t *testing.T) {
for _, c := range []struct {
args map[string]any
want []string
}{
{map[string]any{"repository": "novox/a", "ref": "fix/x"}, []string{"try", "novox/a", "fix/x"}},
{map[string]any{"repository": "novox/a", "ref": "fix/x", "base": "main", "tests": "^TestX$", "store-only": "true"},
[]string{"try", "novox/a", "fix/x", "--base", "main", "--tests", "^TestX$", "--store-only"}},
{map[string]any{"id": "build-1"}, []string{"try", "--id", "build-1"}},
} {
got, err := argvFor("try", c.args)
if err != nil || !reflect.DeepEqual(got, c.want) {
t.Errorf("try %v: %v %v, want %v", c.args, got, err, c.want)
}
}
if _, err := argvFor("try", map[string]any{"repository": "novox/a"}); err == nil {
t.Error("a try of no ref was taken")
}
for _, v := range catalogue.ControllerVerbs {
if v.Name == "try" {
if len(v.Replaces) == 0 {
t.Error("try names no shell habit it replaces")
}
return
}
}
t.Error("try is not a verb of the controller seat")
}
+7 -2
View File
@@ -23,7 +23,12 @@ import (
// **It holds the stores, not a copy of the decision.** What to do about an upgrade is read when
// one arrives, so changing it takes effect on the next upgrade rather than on the next restart of
// the control plane.
type following struct{ open *stores }
type following struct {
open *stores
// oneShot is a process that ends after the call (a recheck through `delivery check`): what it would do beside
// the call it does within it, or the process's exit would cut it off (novox/hq issue 726).
oneShot bool
}
// Upgraded sends the machines running a module the version the catalogue now considers current —
// or records that they are behind, which is the default and needs no record.
@@ -79,7 +84,7 @@ func (f following) Upgraded(ctx context.Context, u link.Upgraded) error {
// `rebuild`, a `replay` registered — was sent here one machine after another without any judging; it
// now waits for a gate like any other build, and the release plan walks it through the machines, one
// at a time, each judged.
fmt.Printf("%s moved to %s outside a plan; a release plan sends it to %s, one machine at a time, each judged "+
fmt.Printf("%s moved to %s outside a plan; a release plan sends it to %s, one machine first and then the rest at once, each judged "+
"(`upgrade backlog` lists what waits)\n", u.Module, shortCommit(u.Commit), readableList(on))
return nil
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module may declare its unhealthy urgent (novox/hq ADR 0355): its module-unhealthy condition is then raised
// urgent; without the field it stays a warning; a wait for a new login stays a warning whatever the module says.
// judgedTwice judges one machine's statement twice, over a catalogue in which mail declares its unhealthy
// urgent and notes says nothing, and returns what is open.
func judgedTwice(t *testing.T, unhealthy map[string][]inventory.ResourceHealth) []conditions.Condition {
t.Helper()
ctx := t.Context()
k, _ := withConditionsInMemory(t)
was := readHoldingFor
t.Cleanup(func() { readHoldingFor = was })
readHoldingFor = func(_ context.Context, _ *inventory.Inventory, open []conditions.Condition) (*holding, error) {
return &holding{ctx: context.Background(), open: open, providers: map[string]providerLookup{},
shelf: map[string]catalogue.Manifest{
"mail": {Module: "mail", Version: "1", Unhealthy: &catalogue.UnhealthyPolicy{
Severity: catalogue.UnhealthyUrgent, Why: "mail is refused or stuck for everyone with a mailbox"}},
"notes": {Module: "notes", Version: "1"},
}}, nil
}
streaks := map[string]int{}
for look := 1; look <= 2; look++ {
for m := range unhealthy {
streaks[m] = look
}
if err := judgeModuleHealth(ctx, nil, k, "anchor", unhealthy, streaks, time.Now()); err != nil {
t.Fatal(err)
}
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
return open
}
func containerDown(module string) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: module, Resource: module + ".web", Kind: "container", Target: module,
State: link.StateUnhealthy, Reason: "down"}
}
func TestAModuleThatDeclaresItsUnhealthyUrgentIsRaisedUrgent(t *testing.T) {
open := judgedTwice(t, map[string][]inventory.ResourceHealth{"mail": {containerDown("mail")}})
c := conditionOf(open, "module.mail.anchor.unhealthy")
if c == nil {
t.Fatalf("mail's unhealthy condition was not raised: %+v", open)
}
if c.Severity != conditions.Urgent {
t.Errorf("mail declares its unhealthy urgent, and it was raised %s", c.Severity)
}
}
func TestAModuleThatSaysNothingOfItsUnhealthyStaysAWarning(t *testing.T) {
open := judgedTwice(t, map[string][]inventory.ResourceHealth{"notes": {containerDown("notes")}})
c := conditionOf(open, "module.notes.anchor.unhealthy")
if c == nil {
t.Fatalf("notes' unhealthy condition was not raised: %+v", open)
}
if c.Severity != conditions.Warning {
t.Errorf("notes says nothing of its unhealthy, and it was raised %s", c.Severity)
}
}
// Only the module-unhealthy condition changes (ADR 0355 §4): a wait for a new login (ADR 0254) of a module that
// declares its unhealthy urgent stays a warning.
func TestAWaitForANewLoginStaysAWarningWhenTheModuleDeclaresItsUnhealthyUrgent(t *testing.T) {
open := judgedTwice(t, map[string][]inventory.ResourceHealth{"mail": {relogin("mail", "operator"),
userUnit("mail", "operator")}})
c := conditionOf(open, "module.mail.anchor.relogin-needed")
if c == nil {
t.Fatalf("mail's relogin-needed condition was not raised: %+v", open)
}
if c.Severity != conditions.Warning {
t.Errorf("a wait for a new login was raised %s; want a warning", c.Severity)
}
if conditionOf(open, "module.mail.anchor.unhealthy") != nil {
t.Errorf("a wait for a new login raised mail's unhealthy condition beside it: %+v", open)
}
}
+6
View File
@@ -177,6 +177,12 @@ func walkReached(p inventory.Plan, running func(string) []string) map[string][]s
reached = appendNew(reached, r.Gate.Machines...)
carried(r.Gate)
}
for _, g := range r.Gates {
if g != nil && !g.Pending {
reached = appendNew(reached, g.Machines...)
carried(g)
}
}
for _, tier := range p.Tiers {
for _, m := range tier {
for _, n := range running(m) {
@@ -0,0 +1,122 @@
package main
import (
"reflect"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 754: a walk waiting for its delivery's word takes no build's outcome, nor does a walk take one for
// a module it never asked; asked under an id, it takes that id's alone.
func TestAWalkTakesOnlyTheOutcomesOfWhatItAsked(t *testing.T) {
asked := time.Date(2026, 10, 14, 23, 30, 0, 0, time.UTC)
before := asked.Add(-time.Minute)
waiting := inventory.Plan{Delivery: &inventory.PlanDelivery{Awaits: "mesh-delivery"}}
letGo := inventory.Plan{Delivery: &inventory.PlanDelivery{Awaits: "mesh-delivery", Go: &asked}}
own := &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: "build-own"}
old := &inventory.PlanModule{State: "asked", AskedAt: &asked}
for _, c := range []struct {
what string
p inventory.Plan
state *inventory.PlanModule
at time.Time
id string
takes bool
}{
{"a rebuild while the walk waits", waiting, nil, asked, "build-rebuild", false},
{"its own id while it waits", waiting, own, asked, "build-own", false},
{"a module it never asked", letGo, nil, asked, "build-rebuild", false},
{"a module it never asked, state kept empty", letGo, &inventory.PlanModule{}, asked, "build-rebuild", false},
{"another id", letGo, own, asked, "build-rebuild", false},
{"its own id", letGo, own, asked, "build-own", true},
{"a plan from before ids, asked after it", letGo, old, asked.Add(time.Second), "x", true},
{"a plan from before ids, asked before it", letGo, old, before, "x", false},
{"a plan with no delivery, its own id", inventory.Plan{}, own, asked, "build-own", true},
} {
if got := takesOutcome(c.p, c.state, c.at, c.id); got != c.takes {
t.Errorf("%s: takes %v, want %v", c.what, got, c.takes)
}
}
}
// novox/hq issue 754: a tier partly asked asks the rest; a module held built that the plan never asked — the live
// shape, a hand rebuild taken in during the wait, built after the plan was made — is asked again; one asked is kept.
func TestATierPartlyAskedAsksTheRest(t *testing.T) {
created := time.Date(2026, 10, 14, 23, 30, 0, 0, time.UTC)
earlier, later := created.Add(-time.Hour), created.Add(7*time.Minute)
p := inventory.Plan{Created: created, Tiers: [][]string{{"build-agent", "gitea", "route-proxy", "stale", "kept", "own"}},
Modules: map[string]*inventory.PlanModule{
"gitea": {},
"route-proxy": {State: "built", BuiltAt: &later, Commit: "c51357f8"},
"stale": {State: "built", BuiltAt: &earlier},
"kept": {State: "asked", AskedAt: &created, Build: "build-kept"},
"own": {State: "built", AskedAt: &created, BuiltAt: &later, Build: "build-own"},
}}
if got, want := unaskedOf(p), []string{"build-agent", "gitea", "route-proxy", "stale"}; !reflect.DeepEqual(got, want) {
t.Fatalf("unasked %v, want %v", got, want)
}
if !askedInTier(p) {
t.Fatal("a tier with a module asked under an id reads as never asked")
}
p.Modules["kept"], p.Modules["own"] = nil, nil
if askedInTier(p) {
t.Fatal("a tier the plan never asked reads as asked")
}
}
// novox/hq issue 754, as seen on 2026-10-14: a rebuild lands while the walk waits for its word; the walk is let go;
// it asks every module of its tier, and completes it once those builds come in.
func TestARebuildInTheWaitIsNotTheWalksBuild(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
twoTiers(t, open)
created := time.Now().UTC().Add(-10 * time.Minute)
plan := inventory.Plan{ID: "plan-754", Repository: "novox/a", Branch: "main", Commit: "c0ffee", Created: created,
State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b"}}, Modules: map[string]*inventory.PlanModule{},
Delivery: &inventory.PlanDelivery{Awaits: "mesh-delivery"}}
if err := open.inventory.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
// A rebuild of a, beside the waiting walk: not joined, and its outcome not taken.
joined, said, err := joinAPlan(ctx, open, "a")
if err != nil || joined || said == "" {
t.Fatalf("a rebuild joined a waiting walk: %v %q %v", joined, said, err)
}
planBuilt(ctx, open, "a", "c0ffee", "", time.Now().UTC(), "build-rebuild")
p, _ := open.inventory.PlanByID(ctx, plan.ID)
if s := p.Modules["a"]; s != nil && s.State != "" {
t.Fatalf("the waiting walk took a rebuild as its build: %+v", s)
}
if len(*asked) != 0 {
t.Fatalf("a waiting walk asked %v", *asked)
}
// Let go: both are asked, and their outcomes complete the tier.
if _, err := letGo(ctx, open.inventory, plan.ID, "operator", "test"); err != nil {
t.Fatal(err)
}
advancePlans(ctx, open)
p, _ = open.inventory.PlanByID(ctx, plan.ID)
for _, m := range []string{"a", "b"} {
if s := p.Modules[m]; s == nil || s.State != "asked" || s.Build == "" {
t.Fatalf("let go, %s is %+v", m, s)
}
}
if len(*asked) != 2 {
t.Fatalf("asked %v", *asked)
}
for _, m := range []string{"a", "b"} {
id := p.Modules[m].Build
at, _ := link.BuildAskedAt(id)
planBuilt(ctx, open, m, "c0ffee", "", at, id)
}
advancePlans(ctx, open)
if p, _ = open.inventory.PlanByID(ctx, plan.ID); p.Open() && p.Tier == 0 {
t.Fatalf("the walk did not get past its tier: %s at %d, %q", p.State, p.Tier, p.Note)
}
}
+11
View File
@@ -92,6 +92,8 @@ type signalFacts struct {
// consumers: the condition counts the messages given up on, not the looks (novox/hq issue 440).
deadLettersNewest map[string]time.Time
advisoriesErr error
// busFaultsErr is why the bus module's events could not be read this tick (novox/hq ADR 0357).
busFaultsErr error
selfCheck selfCheckFacts
@@ -358,6 +360,15 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
}
f.advisories = link.Advisories.Since(now.Add(-advisoryQuiet))
f.lostConsumers, f.advisoriesErr = w.lostConsumers(ctx, f.advisories)
// And what the bus did to every other user's connection, as the bus module said it (novox/hq ADR 0357):
// read back for the same hour, and one advisory with the controller's own where both are its.
// A read that fails is its own condition, never S9 blind: the controller's own signals are still said.
if w.server != nil && w.js != nil {
var faults []link.Advisory
if faults, f.busFaultsErr = w.server.BusFaults(ctx, now.Add(-advisoryQuiet)); f.busFaultsErr == nil {
f.advisories = link.MergeAdvisories(f.advisories, faults)
}
}
if f.advisoriesErr == nil && w.js != nil {
f.deadLetters, f.advisoriesErr = link.HeldDeadLetters(w.js.Context())
}
+3 -2
View File
@@ -38,8 +38,9 @@ func TestTheBuildSeatsHolderFollowsTheControllerThatDefinesItsWorker(t *testing.
if pos["build-agent"] <= pos["mesh-controller"] {
t.Fatalf("the build machine after the controller that defines its worker: %v", tiers)
}
if pos["route-proxy"] <= pos["build-agent"] {
t.Fatalf("what the build machine builds comes after it: %v", tiers)
// Built-by orders nothing (novox/hq ADR 0360 §3): whichever builder runs builds the proxy, beside the controller.
if pos["route-proxy"] != 0 {
t.Fatalf("what the builder builds waits for a new one: %v", tiers)
}
if hasCycle(tiers, edges) {
t.Fatalf("no cycle here: %v", tiers)
+34
View File
@@ -0,0 +1,34 @@
package broker
import (
"slices"
"testing"
)
// **The bus module's watch speaks through the runtime that carries it** (novox/hq ADR 0357): the node's
// runtime is granted the events the bus module declares, under the module's name, while the bus's own
// account keeps the snapshot API and nothing else (ADR 0235).
func TestTheRuntimeCarryingTheBusModuleMayPublishItsEventsAndTheBussAccountMayNot(t *testing.T) {
events := []string{"bus.slow-consumer", "bus.connection-closed"}
runtime := Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule,
Carries: []Declared{{Module: "nats", Emits: events}}}
p, err := PermissionsFor(runtime)
if err != nil {
t.Fatal(err)
}
for _, e := range events {
if !slices.Contains(p.Publish, "mesh.mod.nats.event."+e) {
t.Errorf("the runtime carrying the bus module may not publish %s: %v", e, p.Publish)
}
}
account := Principal{Kind: KindModule, Node: "anchor", Module: "nats", Emits: events, SnapshotsTheBus: true}
a, err := PermissionsFor(account)
if err != nil {
t.Fatal(err)
}
for _, s := range a.Publish {
if s == "mesh.mod.nats.event.bus.slow-consumer" || s == "mesh.mod.nats.event.bus.connection-closed" {
t.Errorf("the bus's own account is granted %s", s)
}
}
}
+21 -4
View File
@@ -128,7 +128,13 @@ func ConsumerFor(p Principal) (Consumer, bool) {
// connected, and its graph stayed empty with nothing anywhere reporting why.
// And one that hears its own answers on a seat it uses (novox/hq ADR 0259 §3): an asker's warrants.
hearsItsOwn := len(SeatTrafficOf(p.Module, nil, p.Uses, nil).Subscribe) > 0
if p.Kind != KindModule || (len(p.Consumes) == 0 && len(p.Watches) == 0 && !hearsItsOwn) {
hearsAStream := false
for _, h := range p.Hears {
if len(h.Filters) > 0 {
hearsAStream = true
}
}
if p.Kind != KindModule || (len(p.Consumes) == 0 && len(p.Watches) == 0 && !hearsItsOwn && !hearsAStream) {
return Consumer{}, false
}
perms, err := PermissionsFor(p)
@@ -140,11 +146,14 @@ func ConsumerFor(p Principal) (Consumer, bool) {
// business, which is why this is a filter and not the whole list.
var filters []string
for _, s := range perms.Subscribe {
if strings.Contains(s, ".event.") {
// Not a module stream's, which its own consumer on that stream hears (novox/hq ADR 0363).
if strings.Contains(s, ".event.") && !strings.HasPrefix(s, moduleStreamPrefix) {
filters = append(filters, s)
}
}
if len(filters) == 0 {
// **A module that hears only a module's stream still has one here** (novox/hq ADR 0363 §7): what it
// gives up on there is delivered again under `mesh.again.<consumer>.` on EVENTS, and only this filters it.
if len(filters) == 0 && !hearsAStream {
return Consumer{}, false
}
sort.Strings(filters)
@@ -181,6 +190,13 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
}
seen[p.Node+"/"+p.Module] = true
out = append(out, ModuleConsumer{Node: p.Node, Module: p.Module, Consumer: c})
// And one on each module stream it hears (novox/hq ADR 0363 §5).
for _, h := range p.Hears {
if len(h.Filters) == 0 {
continue
}
out = append(out, ModuleConsumer{Node: p.Node, Module: p.Module, Consumer: heardConsumer(p, h)})
}
}
for _, p := range users {
if p.Kind == KindModule {
@@ -193,7 +209,8 @@ func ConsumersOf(users []Principal) []ModuleConsumer {
}
for _, d := range p.Carries {
add(Principal{Kind: KindModule, Node: p.Node, Module: d.Module, Emits: d.Emits,
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches})
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches,
Streams: d.Streams, Hears: d.Hears})
}
}
return out
+22
View File
@@ -486,3 +486,25 @@ func (j *JetStream) LogStreams() ([]string, error) {
}
return out, lister.Err()
}
// ConsumerNames is every durable consumer on a stream; none for a stream that does not exist yet.
func (j *JetStream) ConsumerNames(stream string) ([]string, error) {
if _, err := j.js.StreamInfo(stream); errors.Is(err, nats.ErrStreamNotFound) {
return nil, nil
} else if err != nil {
return nil, err
}
var out []string
for name := range j.js.ConsumerNames(stream) {
out = append(out, name)
}
return out, nil
}
// RemoveConsumer deletes one durable consumer (novox/hq ADR 0363 §6): one the composition no longer asks for.
func (j *JetStream) RemoveConsumer(stream, name string) error {
if err := j.js.DeleteConsumer(stream, name); err != nil && !errors.Is(err, nats.ErrConsumerNotFound) {
return err
}
return nil
}
+12
View File
@@ -61,6 +61,15 @@ type Membership struct {
// over every module on the machine, so one module's code reaching another's name or kind through
// the runtime is the runtime's to refuse.
SeatTraffic *SeatTraffic `json:"seat-traffic,omitempty"`
// Streams are this module's own streams (novox/hq ADR 0363 §2): an event one of them carries is published
// on that stream's subject, never on Emits. Hears are the module streams it hears, each through a durable
// consumer on that stream the runtime binds beside the one on EVENTS (§8).
Streams []StreamIssued `json:"streams,omitempty"`
Hears []HeardIssued `json:"hears,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server, the tool runner's
// alone (novox/hq ADR 0367 §2): the subjects its account is granted, under the names a session asks for.
// The tool runner refuses any family not listed here.
Waits []WaitIssued `json:"waits,omitempty"`
}
// Served is one address a tool is answered on.
@@ -126,6 +135,9 @@ func MembershipFor(node string, d Declared, where Placements) Membership {
}
m.State = stateIssuedFor(d, node)
m.Logs = logsIssuedFor(d)
m.Streams = streamsIssuedFor(d)
m.Hears = HeardIssuedFor(node, d)
m.Waits = waitsIssuedFor(d)
t := SeatTrafficOf(d.Module, d.Holds, d.Uses, d.Watches)
for _, s := range append(append([]Seat{}, d.Uses...), d.Watches...) {
if !s.Kinded {
+243
View File
@@ -0,0 +1,243 @@
package broker
import (
"fmt"
"sort"
"strings"
)
// A module's own stream on the bus (novox/hq ADR 0363).
//
// A module declares a stream as it declares a log: named locally, created by the controller from the
// catalogue on every raise and every send, never removed by the mesh. Its events are published on
// `mesh.stream.<module>.event.<event>`, a subject space the shared EVENTS stream (`mesh.mod.*.event.>`)
// does not take — the bus refuses two streams claiming one subject, so a module's stream cannot share
// EVENTS' — and a consumer hears it through a durable consumer of its own on that stream, named as on
// EVENTS. Pure; the controller's raise asserts what is composed here.
// moduleStreamPrefix starts every subject a module's own stream takes.
const moduleStreamPrefix = "mesh.stream."
// ModuleStreamSubject is where a module's event on its own stream lands: `mesh.stream.<module>.event.<event>`.
// The token before `.event.` is the emitter, as on EVENTS, so a runtime reads the same key from it.
func ModuleStreamSubject(module, event string) string {
return moduleStreamPrefix + module + ".event." + event
}
// OnAModuleStream says a subject is one a module's own stream takes.
func OnAModuleStream(subject string) bool { return strings.HasPrefix(subject, moduleStreamPrefix) }
// WirePattern is a local event pattern as the bus spells it: `**` is `>`.
func WirePattern(local string) string {
parts := strings.Split(local, ".")
for i, p := range parts {
if p == "**" {
parts[i] = ">"
}
}
return strings.Join(parts, ".")
}
// A ModuleStream is one module's declared stream as the bus holds it.
type ModuleStream struct {
Module string
// Name is its name on the bus, `MAIL`.
Name string
// Events are the events it carries, as the bus spells them (WirePattern).
Events []string
MaxAgeDays int
MaxMiB int
// Restricted says a trusted config value of the module grants who hears it (ADR 0363 §4).
Restricted bool
}
// Subjects are the subjects the stream takes.
func (s ModuleStream) Subjects() []string {
var out []string
for _, e := range s.Events {
out = append(out, ModuleStreamSubject(s.Module, e))
}
sort.Strings(out)
return out
}
// Stream is the stream the controller asserts: kept by limits, its age and size the module's, dropping
// its oldest when full — a consumer that was away catches up within the age, and the module's own record
// answers for anything older.
func (s ModuleStream) Stream() Stream {
why := fmt.Sprintf("%s's own stream (novox/hq ADR 0363): its events, kept %d days and at most %d MiB so a "+
"consumer that was away catches up", s.Module, s.MaxAgeDays, s.MaxMiB)
if s.Restricted {
why += "; heard only by the consumers its trusted config value grants"
}
return Stream{
Name: s.Name,
Subjects: s.Subjects(),
Retention: RetentionLimits,
MaxAge: s.MaxAgeDays * 24 * 60 * 60,
MaxBytes: int64(s.MaxMiB) * 1024 * 1024,
Why: why,
}
}
// ModuleStreams is every declared module stream as the controller asserts it, in name order.
func ModuleStreams(declared []ModuleStream) []Stream {
sorted := append([]ModuleStream(nil), declared...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Name < sorted[j].Name })
out := make([]Stream, 0, len(sorted))
for _, s := range sorted {
out = append(out, s.Stream())
}
return out
}
// Heard is a module's stream another module hears, and the subjects it hears of it: its `consumes`
// within the stream and, for a restricted stream, within what the provider's trusted config value grants it
// (ADR 0363 §5). Composed only with at least one subject.
type Heard struct {
Stream string
Emitter string
Filters []string
}
// heardConsumer is the durable consumer a module hears a module's stream through: named as its consumer
// on EVENTS, on that stream, with the subjects it was granted as its filters.
func heardConsumer(p Principal, h Heard) Consumer {
filters := append([]string(nil), h.Filters...)
sort.Strings(filters)
return Consumer{
Name: consumerDurable(p),
Stream: h.Stream,
Filters: filters,
AckWaitSeconds: 30,
MaxDeliver: 5,
Why: "what " + p.Module + " consumes of " + h.Emitter + "'s stream " + h.Stream + " and is granted " +
"(novox/hq ADR 0363); after max-deliver it gives an event up, and the controller keeps it in " +
"DEAD_LETTERS until a person delivers it again or drops it",
}
}
// heardGrants are what a principal publishes to read and acknowledge the consumers it hears module streams
// through: asking about each, pulling from it, acknowledging it — its own name on each stream, no other's.
func heardGrants(p Principal, hears []Heard) []string {
var out []string
durable := consumerDurable(p)
for _, h := range hears {
if !safeSubject.MatchString(h.Stream) || len(h.Filters) == 0 {
continue
}
out = append(out,
"$JS.API.CONSUMER.INFO."+h.Stream+"."+durable,
"$JS.API.CONSUMER.MSG.NEXT."+h.Stream+"."+durable,
"$JS.ACK."+h.Stream+"."+durable+".>")
}
return out
}
// ownStreamGrants are the subjects a module publishes its own streams' events on, and no other's.
func ownStreamGrants(module string, streams []ModuleStream) []string {
var out []string
for _, s := range streams {
if s.Module != module {
continue
}
out = append(out, s.Subjects()...)
}
return out
}
// StreamIssued is one of a module's own streams as its membership names it (ADR 0363 §2): the events it
// carries, as the bus spells them, and the subject an event of them is published on, `{event}` the event.
type StreamIssued struct {
Stream string `json:"stream"`
Events []string `json:"events"`
Subject string `json:"subject"`
}
// HeardIssued is one module stream an assignment hears, as its membership names it (ADR 0363 §8): the
// stream and the durable consumer on it the runtime binds.
type HeardIssued struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
}
// streamsIssuedFor is every stream of its own a module's membership names.
func streamsIssuedFor(d Declared) []StreamIssued {
var out []StreamIssued
for _, s := range d.Streams {
if s.Module != d.Module || !safeSubject.MatchString(d.Module) {
continue
}
out = append(out, StreamIssued{Stream: s.Name, Events: append([]string(nil), s.Events...),
Subject: ModuleStreamSubject(d.Module, "{event}")})
}
return out
}
// HeardIssuedFor is every module stream an assignment hears, as its membership names it.
func HeardIssuedFor(node string, d Declared) []HeardIssued {
var out []HeardIssued
for _, h := range d.Hears {
if len(h.Filters) == 0 {
continue
}
out = append(out, HeardIssued{Stream: h.Stream, Consumer: node + "_" + d.Module})
}
sort.Slice(out, func(i, j int) bool { return out[i].Stream < out[j].Stream })
return out
}
// Pruner is the part of a JetStream connection that removes a module's consumer on a module's stream it
// may no longer hear (ADR 0363 §6). Optional: what only records what it is asked to assert has none.
type Pruner interface {
ConsumerNames(stream string) ([]string, error)
RemoveConsumer(stream, name string) error
}
// PruneHeard removes, from each module stream, every durable consumer the composition no longer asks for,
// and answers what it removed as `<stream>/<consumer>`. A consumer that loses its grant loses what it reads
// through, whatever its runtime still holds; a stream that does not exist yet has nothing to remove.
func PruneHeard(p Pruner, streams []ModuleStream, wanted []Consumer) ([]string, error) {
keep := map[string]bool{}
for _, c := range wanted {
keep[c.Stream+"/"+c.Name] = true
}
var removed []string
for _, s := range streams {
names, err := p.ConsumerNames(s.Name)
if err != nil {
return removed, fmt.Errorf("listing the consumers on %s: %w", s.Name, err)
}
for _, n := range names {
if keep[s.Name+"/"+n] {
continue
}
if err := p.RemoveConsumer(s.Name, n); err != nil {
return removed, fmt.Errorf("removing %s's consumer on %s, which is no longer granted: %w", n, s.Name, err)
}
removed = append(removed, s.Name+"/"+n)
}
}
return removed, nil
}
// EnsureHeard asserts a consumer on a module's stream (ADR 0363 §5, §6). **On a restricted stream, a consumer
// the bus will not bring to match is removed**: a pull consumer delivers to whoever pulls it, so one left with
// its wider filters after a narrowing was refused would keep handing over what the grant no longer names,
// whatever the user list now says. The next assertion makes it again with the granted filters. The refusal is
// returned either way, saying what was done, so the send says it and raises it.
func EnsureHeard(r Raiser, c Consumer, restricted bool) error {
err := r.EnsureConsumer(c)
if err == nil || !restricted {
return err
}
p, prunes := r.(Pruner)
if !prunes {
return fmt.Errorf("%w; it could not be removed here, and may still deliver more than it is granted", err)
}
if rerr := p.RemoveConsumer(c.Stream, c.Name); rerr != nil {
return fmt.Errorf("%w; REMOVING IT FAILED TOO, so it may still deliver more than it is granted: %v", err, rerr)
}
return fmt.Errorf("%w; REMOVED, so it delivers nothing it is no longer granted: the next assertion makes it "+
"again with its granted filters (novox/hq ADR 0363 §6)", err)
}
+68
View File
@@ -0,0 +1,68 @@
package broker
import (
"slices"
"testing"
"time"
"github.com/novox/mesh-controller/internal/testbus"
)
// On a real bus: a module's stream is made beside EVENTS without a clash, keeps what the module publishes on
// its own subjects, a granted consumer hears only its part, a narrowed grant narrows it, and a withdrawn one
// is removed (novox/hq ADR 0363).
//
// go test ./internal/broker/ -run TestAModuleStreamOnARealBus
func TestAModuleStreamOnARealBus(t *testing.T) {
js, err := Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer js.Close()
for _, s := range []Stream{MeshStreams()[3], mailStream.Stream()} {
_ = js.js.DeleteStream(s.Name)
if err := js.EnsureStream(s); err != nil {
t.Fatalf("%s: %v", s.Name, err)
}
defer func(name string) { _ = js.js.DeleteStream(name) }(s.Name)
}
for _, account := range []string{"mesh", "work"} {
if _, err := js.js.Publish(ModuleStreamSubject("mesh-mail", "mail."+account+".message.added"), []byte("{}")); err != nil {
t.Fatalf("publishing %s's mail: %v", account, err)
}
}
if info, _ := js.js.StreamInfo(EventsStream); info.State.Msgs != 0 {
t.Fatalf("EVENTS kept %d of the module stream's events", info.State.Msgs)
}
reader := Principal{Kind: KindModule, Node: "ace", Module: "reader", Hears: []Heard{mailHeard}}
c := heardConsumer(reader, mailHeard)
if err := js.EnsureConsumer(c); err != nil {
t.Fatal(err)
}
time.Sleep(50 * time.Millisecond)
if info, _ := js.js.ConsumerInfo("MAIL", c.Name); info.NumPending != 1 {
t.Fatalf("the reader granted `mesh` has %d pending, want 1", info.NumPending)
}
narrowed := c
narrowed.Filters = []string{"mesh.stream.mesh-mail.event.mail.mesh.message.changed"}
if err := EnsureHeard(js, narrowed, true); err != nil {
t.Fatalf("narrowing the grant: %v", err)
}
if info, _ := js.js.ConsumerInfo("MAIL", c.Name); info.Config.FilterSubject != narrowed.Filters[0] || len(info.Config.FilterSubjects) != 0 {
t.Fatalf("narrowed to %q %v, want only %s", info.Config.FilterSubject, info.Config.FilterSubjects, narrowed.Filters[0])
}
if info, _ := js.js.ConsumerInfo("MAIL", c.Name); info.NumPending != 0 {
t.Fatalf("after narrowing, %d of the reader's former grant are still pending", info.NumPending)
}
removed, err := PruneHeard(js, []ModuleStream{mailStream}, nil)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(removed, []string{"MAIL/ace_reader"}) {
t.Fatalf("removed %v", removed)
}
if names, _ := js.ConsumerNames("MAIL"); len(names) != 0 {
t.Fatalf("still on MAIL: %v", names)
}
}
+210
View File
@@ -0,0 +1,210 @@
package broker
import (
"errors"
"reflect"
"slices"
"strings"
"testing"
)
// mailStream is mesh-mail's own stream as the bus holds it (novox/hq ADR 0363).
var mailStream = ModuleStream{Module: "mesh-mail", Name: "MAIL",
Events: []string{"mail.*.message.added", "mail.*.message.changed", "mail.*.message.removed"},
MaxAgeDays: 7, MaxMiB: 1024, Restricted: true}
// mailHeard is what a reader granted the account `mesh` hears of it.
var mailHeard = Heard{Stream: "MAIL", Emitter: "mesh-mail", Filters: []string{
"mesh.stream.mesh-mail.event.mail.mesh.message.added", "mesh.stream.mesh-mail.event.mail.mesh.message.changed"}}
// A module's stream is its own: subjects EVENTS does not claim, its age and size, kept by limits (ADR 0363 §1, §2).
func TestAModuleStreamTakesSubjectsEventsDoesNot(t *testing.T) {
s := mailStream.Stream()
if s.Name != "MAIL" || s.Retention != RetentionLimits || s.MaxAge != 7*24*60*60 || s.MaxBytes != 1024<<20 || s.DiscardNew {
t.Fatalf("composed as %+v", s)
}
want := []string{"mesh.stream.mesh-mail.event.mail.*.message.added", "mesh.stream.mesh-mail.event.mail.*.message.changed",
"mesh.stream.mesh-mail.event.mail.*.message.removed"}
if !reflect.DeepEqual(s.Subjects, want) {
t.Fatalf("subjects %v", s.Subjects)
}
for _, own := range MeshStreams() {
for _, a := range own.Subjects {
for _, b := range s.Subjects {
if SubjectsOverlap(a, b) {
t.Fatalf("%s claims %s, which overlaps %s of %s: the bus would refuse the stream", s.Name, b, a, own.Name)
}
}
}
}
if WirePattern("mail.**") != "mail.>" {
t.Fatal("`**` is spelled `>` on the wire")
}
}
// The provider publishes its stream's events on their subjects and nowhere on EVENTS for them; a consumer
// granted a part hears it through its own consumer on the stream, and one not granted has nothing there
// (ADR 0363 §2, §5).
func TestTheProviderPublishesItsStreamAndOnlyAGrantedConsumerHearsIt(t *testing.T) {
provider, err := PermissionsFor(Principal{Kind: KindModule, Node: "novox", Module: "mesh-mail",
Streams: []ModuleStream{mailStream}})
if err != nil {
t.Fatal(err)
}
if !MayPublish(provider, "mesh.stream.mesh-mail.event.mail.work.message.added") {
t.Fatal("the provider may not publish its own stream's event")
}
if MayPublish(provider, "mesh.mod.mesh-mail.event.mail.work.message.added") {
t.Fatal("the provider may publish its stream's event on EVENTS")
}
if MayPublish(provider, "mesh.stream.other.event.mail.work.message.added") {
t.Fatal("the provider may publish on another module's stream")
}
reader := Principal{Kind: KindModule, Node: "ace", Module: "reader", Hears: []Heard{mailHeard}}
perms, err := PermissionsFor(reader)
if err != nil {
t.Fatal(err)
}
for _, s := range []string{"$JS.API.CONSUMER.MSG.NEXT.MAIL.ace_reader", "$JS.API.CONSUMER.INFO.MAIL.ace_reader",
"$JS.ACK.MAIL.ace_reader.1"} {
if !MayPublish(perms, s) {
t.Errorf("the granted reader may not publish %s", s)
}
}
if !MaySubscribe(perms, "mesh.stream.mesh-mail.event.mail.mesh.message.added") ||
MaySubscribe(perms, "mesh.stream.mesh-mail.event.mail.work.message.added") {
t.Fatal("the reader's subscription is not what it was granted")
}
if MayPublish(perms, "$JS.API.CONSUMER.MSG.NEXT.MAIL.ace_other") {
t.Fatal("the reader may pull another's consumer")
}
stranger, err := PermissionsFor(Principal{Kind: KindModule, Node: "ace", Module: "stranger",
Consumes: []string{"billing.paid"}})
if err != nil {
t.Fatal(err)
}
if MayPublish(stranger, "$JS.API.CONSUMER.MSG.NEXT.MAIL.ace_stranger") ||
MaySubscribe(stranger, "mesh.stream.mesh-mail.event.mail.mesh.message.added") {
t.Fatal("a module not granted reaches the stream")
}
}
// A granted consumer has a durable consumer on the stream, with its granted subjects as filters, and an EVENTS
// consumer that hears what it gives up on delivered again (ADR 0363 §5, §7); the runtime carrying it may read both.
func TestAGrantedConsumerHasAConsumerOnTheStream(t *testing.T) {
records := Records{Nodes: []string{"ace", "novox"}, Assigned: map[string][]Declared{
"novox": {{Module: "mesh-mail", Streams: []ModuleStream{mailStream}}},
"ace": {{Module: RuntimeModule}, {Module: "reader", Hears: []Heard{mailHeard}, HearsAStream: true}, {Module: "stranger", HearsAStream: true}},
}}
users, err := Users(records)
if err != nil {
t.Fatal(err)
}
var onMail, onEvents []Consumer
for _, c := range ConsumersOf(users) {
switch c.Consumer.Stream {
case "MAIL":
onMail = append(onMail, c.Consumer)
case EventsStream:
onEvents = append(onEvents, c.Consumer)
}
}
if len(onMail) != 1 || onMail[0].Name != "ace_reader" || !slices.Equal(onMail[0].Filters, mailHeard.Filters) {
t.Fatalf("on MAIL: %+v", onMail)
}
if len(onEvents) != 1 || onEvents[0].Name != "ace_reader" || !slices.Equal(onEvents[0].Filters, []string{AgainFilter("ace_reader")}) {
t.Fatalf("on EVENTS: %+v", onEvents)
}
for _, u := range users {
if u.Kind != KindNodeTools || u.Node != "ace" {
continue
}
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
if !MayPublish(perms, "$JS.API.CONSUMER.MSG.NEXT.MAIL.ace_reader") || !MayPublish(perms, "$JS.ACK.MAIL.ace_reader.1") {
t.Fatal("the runtime carrying the reader may not read its consumer on MAIL")
}
if MayPublish(perms, "$JS.API.CONSUMER.MSG.NEXT.MAIL.ace_stranger") {
t.Fatal("the runtime may read a consumer no module it carries is granted")
}
}
}
// The membership says where a module's stream events go and which consumers on module streams to bind
// (ADR 0363 §2, §8).
func TestTheMembershipNamesTheStreamAndWhatIsHeard(t *testing.T) {
where := Placements{Nodes: map[string][]string{}}
provider := MembershipFor("novox", Declared{Module: "mesh-mail", Streams: []ModuleStream{mailStream}}, where)
if len(provider.Streams) != 1 || provider.Streams[0].Stream != "MAIL" ||
provider.Streams[0].Subject != "mesh.stream.mesh-mail.event.{event}" ||
!slices.Equal(provider.Streams[0].Events, mailStream.Events) {
t.Fatalf("the provider's membership: %+v", provider.Streams)
}
reader := MembershipFor("ace", Declared{Module: "reader", Hears: []Heard{mailHeard}}, where)
if !reflect.DeepEqual(reader.Hears, []HeardIssued{{Stream: "MAIL", Consumer: "ace_reader"}}) {
t.Fatalf("the reader's membership: %+v", reader.Hears)
}
if none := MembershipFor("ace", Declared{Module: "stranger", HearsAStream: true}, where); none.Hears != nil {
t.Fatalf("a module not granted is told to bind %+v", none.Hears)
}
}
type fakePruner struct {
on map[string][]string
removed []string
}
func (f *fakePruner) ConsumerNames(stream string) ([]string, error) { return f.on[stream], nil }
func (f *fakePruner) RemoveConsumer(stream, name string) error {
f.removed = append(f.removed, stream+"/"+name)
return nil
}
// A consumer that lost its grant has its consumer on the stream removed by the next assertion (ADR 0363 §6).
func TestAConsumerThatLostItsGrantIsRemoved(t *testing.T) {
f := &fakePruner{on: map[string][]string{"MAIL": {"ace_reader", "ace_former"}}}
wanted := []Consumer{{Name: "ace_reader", Stream: "MAIL"}, {Name: "ace_former", Stream: EventsStream}}
removed, err := PruneHeard(f, []ModuleStream{mailStream}, wanted)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(removed, []string{"MAIL/ace_former"}) || !slices.Equal(f.removed, removed) {
t.Fatalf("removed %v", removed)
}
}
type refusingRaiser struct {
fakePruner
refuse bool
}
func (r *refusingRaiser) EnsureStream(Stream) error { return nil }
func (r *refusingRaiser) EnsureConsumer(Consumer) error {
if r.refuse {
return errors.New("consumer update refused")
}
return nil
}
// A narrowing the bus refuses on a restricted stream removes the consumer, so its wider filters deliver nothing
// more; on an unrestricted stream the refusal is only said (review of PR 338, ADR 0363 §6).
func TestARefusedNarrowingOnARestrictedStreamRemovesTheConsumer(t *testing.T) {
c := Consumer{Name: "ace_reader", Stream: "MAIL", Filters: mailHeard.Filters[:1]}
r := &refusingRaiser{refuse: true}
err := EnsureHeard(r, c, true)
if err == nil || !strings.Contains(err.Error(), "REMOVED") || !slices.Equal(r.removed, []string{"MAIL/ace_reader"}) {
t.Fatalf("restricted: %v, removed %v", err, r.removed)
}
r = &refusingRaiser{refuse: true}
if err := EnsureHeard(r, c, false); err == nil || len(r.removed) != 0 {
t.Fatalf("unrestricted: %v, removed %v", err, r.removed)
}
r = &refusingRaiser{}
if err := EnsureHeard(r, c, true); err != nil || len(r.removed) != 0 {
t.Fatalf("accepted: %v, removed %v", err, r.removed)
}
}
+30 -1
View File
@@ -141,6 +141,10 @@ type Principal struct {
KeyedReads []KeyedRead
// Logs is the local names of the logs this principal's module keeps (novox/hq ADR 0297).
Logs []string
// Streams are the streams this principal's module owns, and Hears the module streams it hears with the
// subjects it was granted (novox/hq ADR 0363).
Streams []ModuleStream
Hears []Heard
// SnapshotsTheBus is the bus's own module, the one holding mesh-broker (novox/hq ADR 0235). Its
// whole authority is BusSnapshotGrants: it copies the streams for the night's backup and nothing
@@ -637,6 +641,8 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, e := range p.Emits {
pub = append(pub, own+".event."+e)
}
// And its own streams' events, on their own subjects (novox/hq ADR 0363 §2).
pub = append(pub, ownStreamGrants(p.Module, p.Streams)...)
// Every tool under its own name, not a list: the tools a module serves are what its code
// answers, and a second copy of that list in the manifest would be a second source of
// truth for the mesh to keep in step (2026-09-28: every module that served a tool was
@@ -678,6 +684,16 @@ func PermissionsFor(p Principal) (Permissions, error) {
sub = append(sub, subject)
}
// 2a. What it hears of a module's own stream, as it was granted (novox/hq ADR 0363 §5): the subjects,
// and its own consumer on that stream, asked about, pulled from and acknowledged.
for _, h := range p.Hears {
if len(h.Filters) == 0 {
continue
}
sub = append(sub, h.Filters...)
}
pub = append(pub, heardGrants(p, p.Hears)...)
// 2b. Events of a role it watches, under the seat's own namespace. Subscribe only: watching a
// role is hearing what it announced, not taking part in it.
for _, w := range p.Watches {
@@ -800,6 +816,8 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, e := range d.Emits {
pub = append(pub, own+".event."+e)
}
// And its own streams' events (novox/hq ADR 0363 §2).
pub = append(pub, ownStreamGrants(d.Module, d.Streams)...)
for _, s := range d.Holds {
if servedOnlyByTheController(s) {
continue
@@ -838,7 +856,10 @@ func PermissionsFor(p Principal) (Permissions, error) {
// consumer is still the controller's to make, from the module's own principal.
for _, d := range p.Carries {
own := Principal{Kind: KindModule, Node: p.Node, Module: d.Module, Emits: d.Emits,
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches}
Consumes: d.Consumes, Serves: d.Serves, Holds: d.Holds, Uses: d.Uses, Watches: d.Watches,
Streams: d.Streams, Hears: d.Hears}
// And the module streams it hears, each through its own consumer there (novox/hq ADR 0363 §8).
pub = append(pub, heardGrants(own, d.Hears)...)
if _, consumes := ConsumerFor(own); !consumes {
continue
}
@@ -862,6 +883,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
for _, d := range p.Carries {
pub = append(pub, logGrants(d.Module, logNames(d.Logs))...)
}
// **And the event families a session may wait on** (novox/hq ADR 0367 §2): the tool runner's module's own,
// named one by one in its manifest, as a plain subscription — no consumer is made for them, and no
// call or setting adds one.
waits, err := waitGrants(p.Carries)
if err != nil {
return Permissions{}, err
}
sub = append(sub, waits...)
// **Never the traffic of a trusted holder** (novox/hq ADR 0259 §8): the machine's runtime runs as the
// operator's account, which every agent runs as, so a module saying warrants or speaking for a kind
// that proves its sender is never composed into it — refused here, naming it, whatever registration
@@ -0,0 +1,23 @@
package broker
import (
"slices"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The catalogue's list of the mesh's own stream names, which a module's stream may not take, is the broker's
// own list (novox/hq ADR 0363 §1): read from MeshStreams, so a stream added there cannot be forgotten there.
func TestTheCataloguesReservedStreamNamesAreTheMeshsOwn(t *testing.T) {
var own []string
for _, s := range MeshStreams() {
own = append(own, s.Name)
}
reserved := append([]string(nil), catalogue.MeshStreamNames...)
slices.Sort(own)
slices.Sort(reserved)
if !slices.Equal(own, reserved) {
t.Fatalf("the broker's own streams are %v, the catalogue reserves %v", own, reserved)
}
}
+11 -1
View File
@@ -36,10 +36,20 @@ type Declared struct {
Watches []Seat
// Invokes are the tools it calls, `<module>.<tool>` or `*` (novox/hq ADR 0152).
Invokes []string
// Waits are the event families a session may wait on through the tool runner, the tool runner's module's
// alone (novox/hq ADR 0367 §2): its account subscribes them for a call, through no consumer.
Waits []Wait
// State is the state it keeps, each a bucket its instances write (novox/hq ADR 0201).
State []Bucket
// Logs are the logs it keeps, each a stream its instances append to (novox/hq ADR 0297).
Logs []Log
// Streams are the streams it owns, each its events' one place (novox/hq ADR 0363).
Streams []ModuleStream
// Hears are the module streams it hears on this machine, each with the subjects it was granted
// (novox/hq ADR 0363 §5); HearsAStream says its `consumes` names a module that owns a stream, granted
// or not, so its membership is issued again when a grant changes.
Hears []Heard
HearsAStream bool
// Reads are other modules' state it reads, each `<module>.<name>` (novox/hq ADR 0201).
Reads []string
// KeyedReads are keys of other modules' state it reads, each one key alone: what a seat's holder is
@@ -129,7 +139,7 @@ func Users(r Records) ([]Principal, error) {
Holds: d.Holds, Uses: d.Uses, Watches: d.Watches, Invokes: d.Invokes,
State: stateNames(d.State), Reads: d.Reads, SnapshotsTheBus: d.SnapshotsTheBus,
PerMachine: perMachineNames(d.State), KeyedReads: d.KeyedReads,
Logs: logNames(d.Logs),
Logs: logNames(d.Logs), Streams: d.Streams, Hears: d.Hears,
})
}
if runtimeHere {
+100
View File
@@ -0,0 +1,100 @@
package broker
import (
"fmt"
"strings"
)
// What a session may wait on through the mesh MCP server (novox/hq ADR 0367).
//
// **A grant apart from `consumes`, and the tool runner's alone.** The tool runner's module names in its
// manifest the event families a session may wait on, one by one; the runtime's account is granted a plain
// subscription on those subjects and nothing wider — no durable consumer, no queue of its own — and its
// membership names the same list, from which `mesh_wait` refuses any other. Both are read from this one
// composition, so the grant and what the tool runner offers cannot disagree, and nothing but the manifest
// adds a family (§6).
// Wait is one event family the tool runner's module declares: its name as the manifest gives it, and the
// emitter — a seat or a module — and the event, as the inventory resolved them.
type Wait struct {
// Name is the family as the manifest names it, `<emitter>.<event>`: what a session asks for.
Name string
Emitter string
// Seat says the emitter is a seat, whose events live under the seat's namespace (novox/hq ADR 0121).
Seat bool
// Event is the event, with any further token the subject carries: `decided.claude-code`, the asker of a
// seat that names it last (ADR 0259 §3).
Event string
// Last names what the seat's event subject ends in — "kind" for a kinded bench, "asker" for a seat naming
// its caller (ADR 0259 §3) — and is empty for an event that ends at its name. A family needing it is waited
// on with it given, and refused without.
Last string
}
// WaitIssued is one family as a membership issues it: the name a session asks for, and its subject.
type WaitIssued struct {
Event string `json:"event"`
Subject string `json:"subject"`
}
// WaitSubject is where a family's events land. Each token is checked before it becomes a subject: a
// wildcard, a dot inside a name or an empty token would widen the grant.
func WaitSubject(w Wait) (string, error) {
if !safeSubject.MatchString(w.Emitter) {
return "", fmt.Errorf("%q cannot name an emitter of a waited event: it would widen the subject", w.Emitter)
}
for _, t := range strings.Split(w.Event, ".") {
if !safeSubject.MatchString(t) {
return "", fmt.Errorf("%q cannot name a waited event: each of its names is plain, never a wildcard "+
"(novox/hq ADR 0367 §2)", w.Event)
}
}
if w.Last != "" && !strings.Contains(w.Event, ".") {
return "", fmt.Errorf("%s names no %s: %s's events are said for each %s, so a family is %s.%s.<%s>; "+
"without it no event would ever be heard (novox/hq ADR 0367 §5)",
w.Name, w.Last, w.Emitter, w.Last, w.Emitter, w.Event, w.Last)
}
if w.Seat {
return "mesh.seat." + w.Emitter + ".event." + w.Event, nil
}
return "mesh.mod." + w.Emitter + ".event." + w.Event, nil
}
// waitGrants is the subscription the tool runner's account is granted for its module's families, and an
// error for a carried module other than the tool runner's that declares any (ADR 0367 §2).
func waitGrants(carries []Declared) ([]string, error) {
var sub []string
for _, d := range carries {
if len(d.Waits) == 0 {
continue
}
if d.Module != RuntimeModule {
return nil, fmt.Errorf("%s declares what a session waits on; only %s, the tool runner's module, "+
"does (novox/hq ADR 0367 §2)", d.Module, RuntimeModule)
}
for _, w := range d.Waits {
s, err := WaitSubject(w)
if err != nil {
return nil, err
}
sub = append(sub, s)
}
}
return sub, nil
}
// waitsIssuedFor is the families a membership names: the tool runner's module's, and nobody else's.
func waitsIssuedFor(d Declared) []WaitIssued {
if d.Module != RuntimeModule {
return nil
}
var out []WaitIssued
for _, w := range d.Waits {
s, err := WaitSubject(w)
if err != nil {
continue // refused where the account is composed, which names it
}
out = append(out, WaitIssued{Event: w.Name, Subject: s})
}
return out
}
+135
View File
@@ -0,0 +1,135 @@
package broker
import (
"reflect"
"sort"
"strings"
"testing"
)
// The families ADR 0367 §5 starts the grant with, resolved as the inventory resolves a manifest: a seat's
// event under the seat's namespace, a module's under the module's.
var firstFamilies = []Wait{
{Name: "mesh-knowledge.reviewed", Emitter: "mesh-knowledge", Event: "reviewed"},
{Name: "mesh-delivery.transition", Emitter: "mesh-delivery", Event: "transition"},
{Name: "gitea.forge.status", Emitter: "gitea", Event: "forge.status"},
{Name: "mesh-controller.checked", Emitter: "mesh-controller", Seat: true, Event: "checked"},
{Name: "operator-channel.decided.claude-code", Emitter: "operator-channel", Seat: true, Event: "decided.claude-code"},
}
var firstSubjects = []string{
"mesh.mod.gitea.event.forge.status",
"mesh.mod.mesh-delivery.event.transition",
"mesh.mod.mesh-knowledge.event.reviewed",
"mesh.seat.mesh-controller.event.checked",
"mesh.seat.operator-channel.event.decided.claude-code",
}
func runtimeCarrying(waits []Wait) Principal {
return Principal{Kind: KindNodeTools, Node: "desk", Module: RuntimeModule, PasswordHash: "x", Carries: []Declared{
{Module: "shop", Emits: []string{"order.placed"}, Consumes: []string{"billing.paid"}, Serves: []string{"price"}},
{Module: RuntimeModule, Invokes: []string{"*"}, Waits: waits},
}}
}
func without(all, some []string) []string {
drop := map[string]bool{}
for _, s := range some {
drop[s] = true
}
var out []string
for _, s := range all {
if !drop[s] {
out = append(out, s)
}
}
return out
}
// novox/hq ADR 0367 "How it is checked": the account's wait grant equals the manifest's families. The tool
// runner's account subscribes exactly the families' subjects beyond what it had without them — nothing
// published, nothing wider — and the membership it is issued names the same subjects.
func TestTheToolRunnersWaitGrantIsTheManifestsFamiliesAlone(t *testing.T) {
before, err := PermissionsFor(runtimeCarrying(nil))
if err != nil {
t.Fatal(err)
}
after, err := PermissionsFor(runtimeCarrying(firstFamilies))
if err != nil {
t.Fatal(err)
}
added := without(after.Subscribe, before.Subscribe)
sort.Strings(added)
if !reflect.DeepEqual(added, firstSubjects) {
t.Errorf("the wait grant is %v, want the families' subjects %v", added, firstSubjects)
}
if extra := without(after.Publish, before.Publish); len(extra) > 0 {
t.Errorf("waiting granted a publish: %v", extra)
}
for _, s := range after.Subscribe {
if strings.HasPrefix(s, "mesh.mod.*.event") || s == "mesh.>" || s == ">" || strings.HasSuffix(s, ".event.>") {
t.Errorf("the runtime may subscribe every event: %s", s)
}
}
for _, other := range []string{"mesh.seat.operator-channel.event.decided.shop",
"mesh.mod.mesh-knowledge.event.proposed", "mesh.mod.gitea.event.forge.push"} {
if MaySubscribe(after, other) {
t.Errorf("the runtime may subscribe %s, which no family names", other)
}
}
issued := MembershipFor("desk", runtimeCarrying(firstFamilies).Carries[1], Placements{})
var subjects []string
for _, w := range issued.Waits {
subjects = append(subjects, w.Subject)
}
sort.Strings(subjects)
if !reflect.DeepEqual(subjects, firstSubjects) {
t.Errorf("the membership names %v, want the grant %v", subjects, firstSubjects)
}
if issued.Waits[0].Event != "mesh-knowledge.reviewed" {
t.Errorf("a family is issued under its manifest name: %+v", issued.Waits[0])
}
if shop := MembershipFor("desk", runtimeCarrying(firstFamilies).Carries[0], Placements{}); len(shop.Waits) > 0 {
t.Errorf("a module beside the runtime is issued waits: %+v", shop.Waits)
}
}
// **A subscription for the call, never a durable consumer** (ADR 0367 §2): composing the users makes no
// consumer that filters a family, and none for the runtime's module at all.
func TestAWaitMakesNoDurableConsumer(t *testing.T) {
for _, c := range ConsumersOf([]Principal{runtimeCarrying(firstFamilies)}) {
if c.Module == RuntimeModule {
t.Errorf("the runtime's module has a durable consumer: %+v", c.Consumer)
}
for _, f := range c.Consumer.Filters {
for _, s := range firstSubjects {
if f == s {
t.Errorf("%s's consumer filters the family %s", c.Module, s)
}
}
}
}
}
// Only the tool runner's module waits (ADR 0367 §2): a carried module declaring waits is refused at
// composition, naming it, and no name that would widen a subject is composed.
func TestAWaitIsComposedOnlyForTheToolRunnerAndOnlyWhole(t *testing.T) {
p := runtimeCarrying(nil)
p.Carries[0].Waits = []Wait{{Name: "mesh-knowledge.reviewed", Emitter: "mesh-knowledge", Event: "reviewed"}}
if _, err := PermissionsFor(p); err == nil || !strings.Contains(err.Error(), "shop") {
t.Errorf("a carried module's waits were composed: %v", err)
}
for _, w := range []Wait{
{Name: "x", Emitter: "mesh-knowledge", Event: "*"},
{Name: "x", Emitter: "mesh-knowledge", Event: "a.>"},
{Name: "x", Emitter: "*", Event: "reviewed"},
{Name: "x", Emitter: "mesh.knowledge", Event: "reviewed"},
{Name: "x", Emitter: "mesh-knowledge", Event: ""},
{Name: "x", Emitter: "mesh-knowledge", Event: "a..b"},
} {
if _, err := PermissionsFor(runtimeCarrying([]Wait{w})); err == nil {
t.Errorf("%+v was composed", w)
}
}
}
+74 -1
View File
@@ -162,6 +162,10 @@ type Reuse struct {
// keyFirst is a check's build (§1): its key is taken before anything is built, and a module with none is not
// built at all — nothing could ever take it.
keyFirst bool
// pull and head are the check's own pull request and the head it judged (novox/hq issue 703): it keeps only a
// build an earlier check of the same pull request at the same head made.
pull PullRef
head string
}
// errNoKey is a check's build of a module whose source does not pin it: not built (ADR 0350 §1).
@@ -374,7 +378,7 @@ func build(ctx context.Context, run Runner, publish Publisher,
// only when asked — a check, a dry run, `build --ref` and `rebuild` never are. Its key is taken by the same
// steps a build takes, stopped before anything is built; what it takes is copied by digest.
if reuse != nil && reuse.keyFirst {
_, key, err := keyFor(ctx, run, publish, manifest, within, workspace, commit, credentials, artifacts, args,
keyed, key, err := keyFor(ctx, run, publish, manifest, within, workspace, commit, credentials, artifacts, args,
held, npmrcPath, npmrc, seatBases, src, say)
if err == nil && key.Key == "" {
err = fmt.Errorf("%w: %s", errNoKey, key.Why)
@@ -383,6 +387,11 @@ func build(ctx context.Context, run Runner, publish Publisher,
say("reuse", "not built for the merge: %v", err)
return Result{Commit: commit, NoKey: err.Error()}, err
}
// **A check keeps a build of the same key a verdict names** (novox/hq issue 703): asked again when only
// the trunk moved, it builds nothing the earlier check built of an unchanged source.
if kept := keptCheckBuild(ctx, *reuse, manifest.Module, key.Key, len(artifacts), say); kept != nil {
built, src, reused = kept.Made, keyed, kept
}
} else if reuse != nil && reuse.Scratch != nil {
taken, keyed, why := takeACheckBuild(ctx, run, publish, *reuse, manifest, within, workspace, commit,
credentials, artifacts, args, held, npmrcPath, npmrc, seatBases, src, onTrunk, say)
@@ -492,6 +501,47 @@ func takeACheckBuild(ctx context.Context, run Runner, publish Publisher, reuse R
return taken, keyed, ""
}
// keptCheckBuild is the build a check keeps instead of building (novox/hq issue 703): the one kept under the
// module's key by an earlier check of the same pull request at the same head — only the trunk moved — when a verdict
// of the build seat names its key and every digest — the record alone is never trusted (ADR 0350 §2) — and it holds
// every artifact the module declares. Another pull request's build, or an earlier head's, is never kept. Nil, and
// the check builds, otherwise.
func keptCheckBuild(ctx context.Context, reuse Reuse, module, key string, artifacts int,
say func(step, format string, args ...any)) *Reused {
if reuse.Scratch == nil || reuse.head == "" {
return nil
}
var same []NamedBuild
for _, n := range reuse.Named {
if n.Pull == reuse.pull {
same = append(same, n)
}
}
if len(same) == 0 {
return nil
}
found, ok, err := reuse.Scratch.Find(ctx, module, key)
switch {
case err != nil:
say("reuse", "the scratch namespace could not be read, so the check builds: %v", err)
return nil
case !ok:
return nil
case found.Pull != reuse.pull || found.Head == "" || twelve(found.Head) != twelve(reuse.head):
say("reuse", "the build kept under %s is %s, not this pull request's at this head: built", twelve(key[3:]),
found.Said())
return nil
case unnamed(same, module, key, found) != "":
say("reuse", "a build is kept under %s and no verdict names it with its digests: built", twelve(key[3:]))
return nil
case len(found.Made) != artifacts:
return nil
}
kept := &Reused{From: found, Made: found.Made}
say("reuse", "kept, not built: %s", strings.Replace(kept.Words(), "reused from", "the build of", 1))
return kept
}
// keyFor takes a build's reuse key by the steps a build takes, each artifact stopped before anything is built or
// published (ADR 0350 §2): its own copy of the source inputs, so the build's are untouched.
func keyFor(ctx context.Context, run Runner, publish Publisher, manifest catalogue.Manifest,
@@ -1076,6 +1126,16 @@ func one(ctx context.Context, run Runner, publish Publisher,
return catalogue.Built{}, fmt.Errorf("%s: building %s failed: %w", module, a.Name, err)
}
say("image", "built, publishing")
// Its bases linked into its repository first, so the push needs none of their blobs from this
// machine's store, which may not hold them (novox/hq issue 694). Said and not fatal: a push
// from a store that holds them works without, and one that fails says which blob it lacked.
if lender, can := publish.(BaseLender); can {
if n, err := lender.LendBases(ctx, module+"/"+a.Name, buildArgValues(args)); err != nil {
say("image", "its bases could not all be linked into %s/%s before the push: %v", module, a.Name, err)
} else if n > 0 {
say("image", "%d blob(s) of its bases linked into %s/%s before the push", n, module, a.Name)
}
}
reference, err := publish.PublishImage(ctx, local, module+"/"+a.Name)
if err != nil {
return catalogue.Built{}, err
@@ -1427,6 +1487,19 @@ func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[strin
return args, resolved, nil
}
// buildArgValues is what each --build-arg of a build was handed: the bases, as this mesh holds them.
func buildArgValues(args []string) []string {
var values []string
for i := 0; i+1 < len(args); i += 2 {
if args[i] == "--build-arg" {
if _, value, has := strings.Cut(args[i+1], "="); has {
values = append(values, value)
}
}
}
return values
}
// compile runs a module's own code through its toolchain, and says where the result is.
//
// **In the module's own directory, under the path the toolchain expects.** A module is compiled
+639 -40
View File
@@ -13,7 +13,9 @@ import (
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/artifacts"
@@ -87,6 +89,19 @@ type CheckSpec struct {
// Judge is who judges the gate: "" the controller the mesh runs, "self" the change's own, "validator"
// the running one with the change's validator.
Judge string
// Release is the release branch the pull request merges into (novox/hq ADR 0358), empty for any other base. The
// gate is then judged by the controller beside it as mesh-controller — mesh-controller's release branch, never
// the controller the mesh runs nor its main — with mesh-host's validator from beside it when mesh-host was read at
// the release branch too; a change to the controller still judges itself and a change to the node-engine brings
// its own validator. The verdict names the branch, the judge's commit and the branch each beside was read at.
Release string
// JudgeCache is the judge cache (novox/hq ADR 0358 §6): a directory of the build agent's own, outside the
// workspace, that only judge builds mount — never a check's containers, so no pull request's script can write it.
// It holds the Go build cache every judge is built with (judges stay warm), the release-branch judges kept by the
// commits they are built from (judges/, one per release-branch head), and each judge build's fresh home (homes/,
// removed after the build). Empty: each judge is built cold under the check's own root, which every check empties
// before it clones, and nothing is kept.
JudgeCache string
// Toolchain is the image a check's Go runs in: the mesh's own Go toolchain, as it holds it.
// Toolchains is every toolchain the mesh holds, by language, for a script that declares another.
Toolchain string
@@ -103,6 +118,23 @@ type CheckSpec struct {
// for the walk to take. None for a group's composed check, whose members' own checks build.
Builds []CheckBuildSpec
BuildWith *CheckBuilding
// Tests and StoreOnly narrow what the repository's own script runs, for a branch tried before its pull
// request (novox/hq issue 686): handed to the script as EnvTests and EnvStoreOnly, which it may read.
Tests string
StoreOnly bool
}
// selection is what a check hands the repository's script to narrow what it runs (novox/hq issue 686): nothing
// for a pull request's check, which always runs whole.
func selection(spec CheckSpec) []string {
var env []string
if spec.Tests != "" {
env = append(env, EnvTests+"="+spec.Tests)
}
if spec.StoreOnly {
env = append(env, EnvStoreOnly+"=1")
}
return env
}
// GroupHead is one other head of a delivery group's composed check.
@@ -136,10 +168,12 @@ func ToolchainsOf(held map[string]string) map[string]string {
return out
}
// Beside is one repository cloned next to the one checked.
// Beside is one repository cloned next to the one checked: at Ref, or at Else when Else is said and the repository
// has no branch Ref (novox/hq ADR 0358 §3).
type Beside struct {
Repository string
Ref string
Else string
}
// CheckVerdict is what came of one check. Verdict and Summary are the gate's; Gate and Repo each layer.
@@ -162,6 +196,16 @@ type CheckVerdict struct {
Members []JudgedHead
// Built are the builds it made for its merge (novox/hq ADR 0350 §1), each with its key, or why not.
Built []CheckBuilt
// Release is the release branch it was judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the controller
// that judged its gate (twelve characters), and BesideAt the branch each repository beside it was read at, by
// directory. Empty for a check of any other base.
Release string
JudgedBy string
BesideAt map[string]string
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2):
// mesh-host's of the release branch with its commit, the one the release branch's controller vendors, the
// change's own, or the one the change's own controller vendors. Empty for a check of any other base.
Validator string
}
// Layer is one layer of a check, judged.
@@ -193,6 +237,11 @@ const (
// EnvGroup is a delivery group's other heads, as JSON, for the gate (novox/hq ADR 0239); empty for a
// pull request checked alone.
EnvGroup = "MESH_CHECK_GROUP"
// EnvTests is a `go test -run` pattern and EnvStoreOnly is 1 when a branch tried before its pull request
// asks for only those tests, or only the store tests (novox/hq issue 686). Never set for a pull request's
// check. A script that reads neither runs whole.
EnvTests = "MESH_CHECK_TESTS"
EnvStoreOnly = "MESH_CHECK_STORE_ONLY"
)
// CheckTimeout bounds one check; a check that runs past it is an error, not a pass.
@@ -252,7 +301,17 @@ func ScriptToolchain(script []byte) string {
// Check runs one merge check. An error is that it could not run; the verdict is then "error".
func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry string, forge GitCredential,
log Log) (CheckVerdict, error) {
// Everything a check says goes to the build's log, which the bus keeps: said redacted (novox/hq issue 462).
// Everything a check says goes to the build's log, which the bus keeps: said redacted (novox/hq issue 462). A line
// at a time: its builds for the merge say theirs while its tests run (novox/hq issue 716).
if log != nil {
var saying sync.Mutex
inner := log
log = func(step, message string) {
saying.Lock()
defer saying.Unlock()
inner(step, message)
}
}
redact := redactorFor(forge)
say := logging(func(step, message string) {
if log != nil {
@@ -281,13 +340,32 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
if err := forgetLeftCredentials(workspace); err != nil {
return CheckVerdict{}, err
}
clone := func(repository, ref, dir string) error {
// **mesh-lab's replays run where no pull request reaches** (novox/hq issue 709, ADR 0358 §6): they hold the
// container runtime's socket, so their sources, home, Go cache and module cache are none of the workspace's.
replay, err := replayDirs(spec.JudgeCache, workspace, root, spec.ID)
if err != nil {
return CheckVerdict{}, err
}
if replay.base != "" {
defer os.RemoveAll(replay.base)
}
// cloneAt clones a repository and checks out ref — or, when the repository has no branch ref and orElse is said,
// orElse (novox/hq ADR 0358 §3); readAt keeps which, by directory.
readAt := map[string]string{}
cloneAt := func(repository, ref, orElse, dir string) error {
if _, err := run(ctx, root, "git", cloneWith(credentials, "clone", "--quiet", repository, dir)...); err != nil {
return fmt.Errorf("cannot clone %s: %s", redact.redact(repository), redact.redact(err.Error()))
}
if err := recordedWithoutUserinfo(ctx, run, filepath.Join(root, dir), repository); err != nil {
return err
}
if orElse != "" && ref != "" {
if _, err := run(ctx, filepath.Join(root, dir), "git", "rev-parse", "--verify", "--quiet",
"refs/remotes/origin/"+ref); err != nil {
ref = orElse
}
}
readAt[dir] = ref
if ref != "" {
if _, err := run(ctx, filepath.Join(root, dir), "git", "checkout", "--quiet", ref); err != nil {
return fmt.Errorf("%s has no %s: %w", redact.redact(repository), ref, err)
@@ -295,6 +373,9 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
return nil
}
clone := func(repository, ref, dir string) error {
return cloneAt(repository, ref, "", dir)
}
name := spec.Repo
if name == "" || !safeName.MatchString(name) {
name = "checked"
@@ -330,8 +411,28 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
named := func(v CheckVerdict) CheckVerdict {
v.Head, v.Trunk, v.Merge, v.Beside, v.Members = judged.Head, judged.Trunk, judged.Merge, judged.Beside, judged.Members
v.Release, v.JudgedBy, v.BesideAt, v.Validator = spec.Release, judged.JudgedBy, judged.BesideAt, judged.Validator
return v
}
// **What the merge would move is built from a checkout nothing of the check touches** (novox/hq issue 703): the
// gate and the repository's own check run in containers that mount the workspace read-write, so they can change
// its tree, its objects and its caches. The builds are made from a copy of the merge's objects, cloned before
// anything of the change runs, outside the workspace — so what is kept under a key is what the key describes. Each
// build's working copy is cloned from it into the workspace only after the check has ended (forTheMerge): the
// host's container engine sees no other directory of the holder's (novox/hq issue 713).
buildingTree := ""
if len(spec.Builds) > 0 && spec.BuildWith != nil && len(spec.Group) == 0 {
at := judged.Merge
if at == "" {
at = spec.Ref
}
var building string
building, buildingTree, err = checkoutForTheBuilds(ctx, run, tree, at, workspace)
if err != nil {
return CheckVerdict{}, err
}
defer os.RemoveAll(building)
}
// **The base branch's check judges the change** (novox/hq issue 310), never the one the change brings.
judging, err := TheCheckThatJudges(ctx, tree, spec.Base)
if err != nil {
@@ -354,9 +455,35 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
if dir == name || !safeName.MatchString(dir) {
continue
}
if err := clone(b.Repository, b.Ref, dir); err != nil {
if dir == "mesh-lab" {
// **mesh-lab is never read at a release branch** (novox/hq ADR 0358): its replays run with the container
// runtime's socket, so a branch of a release's name — which anyone may create — is never what runs there,
// whether it is asked with a fallback or without.
if b.Else != "" || (spec.Release != "" && b.Ref == spec.Release) {
b.Ref = b.Else
if b.Ref == "" {
b.Ref = "main"
}
}
b.Else = ""
}
if err := cloneAt(b.Repository, b.Ref, b.Else, dir); err != nil {
return CheckVerdict{}, fmt.Errorf("beside it, %w", err)
}
if dir == "mesh-lab" {
// The copy the replays run from, made before anything of the change runs, outside every check container
// (novox/hq issue 709) — and whether it holds replays, read now, so nothing written later decides it.
if err := replay.copyLab(ctx, run, filepath.Join(root, dir)); err != nil {
return CheckVerdict{}, fmt.Errorf("mesh-lab could not be copied for its replays: %w", err)
}
}
if spec.Release != "" {
// The branch each was read at is named with its commit (ADR 0358 §4).
if judged.BesideAt == nil {
judged.BesideAt = map[string]string{}
}
judged.BesideAt[dir] = readAt[dir]
}
// The commit each one read is named in the verdict (ADR 0323 §3): its move is issue 568's question.
if at := headAt(ctx, run, filepath.Join(root, dir)); at != "" {
if judged.Beside == nil {
@@ -364,7 +491,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
judged.Beside[dir] = at
}
say("check", "beside it %s at %s", dir, short(b.Ref))
say("check", "beside it %s at %s", dir, refWords(readAt[dir]))
}
// A delivery group's other heads (novox/hq ADR 0239), each at its head, for the gate to compose with this.
@@ -416,6 +543,25 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
// Every clone is made: the credential is gone before anything of the check runs (novox/hq issue 462).
forget()
// **What the merge moves is built beside the tests** (novox/hq issue 716) when the build agent has a judge cache to
// build it in, where no container of the check reaches; without one, after the check has judged.
logged := func(step, message string) {
if log != nil {
log(step, redact.redact(message))
}
}
var beside *besideTheTests
if buildingTree != "" {
place, err := buildsBesideTheTests(spec.JudgeCache, workspace, spec.ID)
if err != nil {
return CheckVerdict{}, err
}
if place != "" {
beside = startBesideTheTests(ctx, run, spec, place, buildingTree, judged, forge, logged, say)
defer beside.abandon()
}
}
// The facts, and the versions they say the mesh runs.
if registry == "" {
return CheckVerdict{}, errors.New("no artifact store to read the facts snapshot from")
@@ -478,21 +624,56 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
user = fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid())
}
in := func(image, dir string, env []string, command ...string) []string {
home := workspace
for _, e := range env {
if strings.HasPrefix(e, "HOME=") {
home = strings.TrimPrefix(e, "HOME=")
}
}
// As the builder itself: what a check writes into the workspace is the builder's to remove.
args := []string{"run", "--rm", "--network", "host", "--volume", workspace + ":" + workspace, "--workdir", dir,
"--user", user, "--env", "HOME=" + workspace,
"--user", user, "--env", "HOME=" + home,
// The check's own checkouts, whoever cloned them: git in a container of another user than the
// one that cloned refuses a repository it does not own ("dubious ownership"), and Go's build
// stamps the version from git — a judge that would not build for want of it.
"--env", "GIT_CONFIG_COUNT=1", "--env", "GIT_CONFIG_KEY_0=safe.directory", "--env", "GIT_CONFIG_VALUE_0=*"}
for _, e := range env {
args = append(args, "--env", e)
if !strings.HasPrefix(e, "HOME=") {
args = append(args, "--env", e)
}
}
return append(append(args, image), command...)
}
inToolchain := func(dir string, env []string, command ...string) []string {
return in(spec.Toolchain, dir, env, command...)
}
// **Every judge is built where no pull request can reach** (novox/hq issue 482, ADR 0358 review): the workspace's
// Go cache, HOME and git configuration are written by pull requests' own scripts, and git runs during a Go build
// to stamp its version — a planted hook would rewrite the judge's sources before they compile. So a judge is built
// with a HOME and configuration made fresh for this check and a Go cache only judge builds mount (judgeBuildDirs),
// no global or system git configuration, no Go environment file, and no version stamping from git.
judgeHome, judgeGoCache, err := judgeBuildDirs(spec.JudgeCache, workspace, root, spec.ID)
if err != nil {
return CheckVerdict{}, err
}
defer os.RemoveAll(judgeHome)
inJudgeBuild := func(dir string, _ []string, command ...string) []string {
args := in(spec.Toolchain, dir, JudgeBuildEnv(judgeHome, judgeGoCache), command...)
if spec.JudgeCache == "" {
return args // under the check's root, in the workspace mounted already
}
// The judge cache's parts this build needs, mounted into judge builds alone (ADR 0358 §6).
return append([]string{args[0], "--volume", judgeHome + ":" + judgeHome, "--volume",
judgeGoCache + ":" + judgeGoCache}, args[1:]...)
}
inReplays := func(env []string, command ...string) []string {
args := in(spec.Toolchain, filepath.Join(replay.lab, "replays"), append(ReplayEnv(replay.home, replay.goCache,
replay.modCache), env...), command...)
// The socket goes to the replays alone, never to the change's own code.
mounts := []string{"--volume", "/var/run/docker.sock:/var/run/docker.sock", "--volume", replay.base + ":" +
replay.base, "--volume", replay.goCache + ":" + replay.goCache, "--volume", replay.modCache + ":" + replay.modCache}
return append(append([]string{args[0]}, mounts...), args[1:]...)
}
var out tail
// running runs one container of the check, its output into the report, in a process group of its own.
running := func(args []string) error {
@@ -504,10 +685,23 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
// The judge. Its failing to build is the change's fault when the change is the judge or its validator,
// and the check's when it is the controller the mesh runs.
gate, judgeFault, err := judgeFor(ctx, labelled, run, spec, root, tree, inToolchain, say)
gate, judgeFault, err := judgeFor(ctx, labelled, run, spec, root, tree, judged.BesideAt, inJudgeBuild, say)
if err != nil {
return CheckVerdict{}, err
}
if spec.Release != "" {
// **A release-branch judgement says so** (novox/hq ADR 0358 §4): the branch, the controller's commit that
// judged, and each repository read beside it at its branch and commit.
judgedBy := filepath.Join(root, "mesh-controller")
if spec.Judge == "self" {
judgedBy = tree
}
judged.JudgedBy = headAt(ctx, run, judgedBy)
judged.Validator = releaseValidator(spec, judged.BesideAt, judged.Beside)
said := releaseWords(spec.Release, judged.JudgedBy, judged.Validator, judged.BesideAt, judged.Beside)
fmt.Fprintf(&out, "--- %s\n", said)
say("check", "%s", said)
}
verdictFile := filepath.Join(root, "verdict.json")
env := []string{EnvFacts + "=" + factsFile, EnvGate + "=" + gate, EnvGateStore + "=" + storeURL,
EnvTestStore + "=" + storeURL, EnvTestBus + "=nats://" + bus, EnvRepository + "=" + spec.Owner + "/" + spec.Repo,
@@ -515,6 +709,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
EnvModules + "=" + strings.Join(append(append([]string{}, spec.Modules...), spec.New...), ","),
EnvGroup + "=" + groupFile,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")}
env = append(env, selection(spec)...)
v := CheckVerdict{}
modules := append(append([]string{}, spec.Modules...), prefixed("new:", spec.New)...)
@@ -532,7 +727,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
say("check", "the gate: %d module(s) of the graph touched — %s", len(modules), strings.Join(modules, ", "))
fmt.Fprintf(&out, "--- the gate: %s\n", strings.Join(modules, ", "))
v.Gate.Verdict, v.Gate.Summary = gateLayer(ctx, spec, tree, root, gate, verdictFile, env, inToolchain,
running, &out, bus, workspace, name, say)
running, &out, bus, name, replay.hasReplays, replay.refused(), inReplays, say)
}
if timedOut() {
v.Gate.Verdict, v.Gate.Summary = "error", fmt.Sprintf("the check ran past %s and was ended", CheckTimeout)
@@ -572,6 +767,11 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
}
if spec.Release != "" {
// The statuses name the release branch and the judge first, so the forge's short description keeps them.
mark := releaseMark(spec.Release, judged.JudgedBy)
v.Gate.Summary, v.Repo.Summary = mark+v.Gate.Summary, mark+v.Repo.Summary
}
// What the check printed travels in the verdict to the forge and the controller: redacted as the log is.
v.Gate.Summary, v.Repo.Summary, v.Repo.Failed = redact.redact(v.Gate.Summary), redact.redact(v.Repo.Summary),
redact.redact(v.Repo.Failed)
@@ -582,24 +782,68 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
// **What the merge would move, built once, here** (novox/hq ADR 0350 §1): after the judging, from the merge it
// judged, into the scratch namespace — so the walk of the merge commit takes it rather than building again.
if len(spec.Builds) > 0 && spec.BuildWith != nil && len(spec.Group) == 0 {
if why := skipsTheBuilds(v); why != "" {
say("check", "%s", why)
} else {
began := time.Now()
v.Built = buildForTheMerge(ctx, run, spec, *spec.BuildWith, workspace, tree, judged.Merge, judged, forge,
func(step, message string) {
if log != nil {
log(step, redact.redact(message))
}
}, say)
say("check", "built for the merge: %d of %d module(s) (%s)", builtCount(v.Built), len(v.Built),
time.Since(began).Round(time.Second))
}
switch {
case beside != nil:
v.Built = beside.settle(ctx, v)
case len(spec.Builds) > 0 && spec.BuildWith != nil && len(spec.Group) == 0:
v.Built = forTheMerge(ctx, run, spec, v, workspace, buildingTree, judged, forge, logged, say)
}
return named(v), nil
}
// forTheMerge builds what the merge moves once the check has judged it, from merged — the copy of the merge
// checkoutForTheBuilds made outside the workspace — **in the builder's own workspace** (novox/hq issue 713). A build's
// containers are run by the host's container engine, which mounts a path as the host names it: the build seat's
// holder runs in a container that shares only its workspace with the host, at the same path, so a working copy in
// its own temporary directory reaches the toolchain as an empty directory ("go.mod file not found"), every check
// build failed, and every walk built again. The copy stays outside the workspace, so nothing the tests ran reaches it;
// the working copy cloned from it is made only now, after every container of the check has ended.
func forTheMerge(ctx context.Context, run Runner, spec CheckSpec, verdict CheckVerdict, workspace, merged string,
judged CheckVerdict, forge GitCredential, log Log, say func(step, format string, args ...any)) []CheckBuilt {
if why := skipsTheBuilds(verdict); why != "" {
say("check", "%s", why)
return nil
}
began := time.Now()
built := buildForTheMerge(ctx, run, spec, *spec.BuildWith, workspace, merged, judged.Merge, judged, forge, log, say)
say("check", "built for the merge: %d of %d module(s) (%s)", builtCount(built), len(built),
time.Since(began).Round(time.Second))
return built
}
// checkoutForTheBuilds is a clone of the commit a check builds for its merge, made before anything of the check runs,
// outside the workspace its containers mount (novox/hq issue 703): the directory to remove, and the clone in it.
// Cloned over git's own transport, never by linking the objects: what the check's containers later write into the
// workspace's objects cannot reach it.
func checkoutForTheBuilds(ctx context.Context, run Runner, tree, at, workspace string) (string, string, error) {
dir, err := os.MkdirTemp("", "mesh-check-builds-")
if err != nil {
return "", "", err
}
if withinDir(workspace, dir) {
os.RemoveAll(dir)
return "", "", fmt.Errorf("the temporary directory %s is inside the workspace %s, which a check's containers "+
"mount: the merge is not built from where its tests could change it", dir, workspace)
}
commit, err := run(ctx, tree, "git", "rev-parse", "--verify", at+"^{commit}")
if err != nil {
os.RemoveAll(dir)
return "", "", fmt.Errorf("the commit to build for the merge, %s: %w", at, err)
}
commit = strings.TrimSpace(commit)
if _, err := run(ctx, tree, "git", "update-ref", "refs/heads/mesh-check-builds", commit); err != nil {
os.RemoveAll(dir)
return "", "", err
}
cloned := filepath.Join(dir, "merged")
if _, err := run(ctx, dir, "git", "clone", "--quiet", "--no-local", "--no-checkout", "--branch",
"mesh-check-builds", tree, cloned); err != nil {
os.RemoveAll(dir)
return "", "", fmt.Errorf("cannot copy the merge to build it: %w", err)
}
return dir, cloned, nil
}
// builtCount is how many of a check's builds for its merge were built.
func builtCount(built []CheckBuilt) int {
n := 0
@@ -771,8 +1015,9 @@ const SomeManifestsEnv = "MESH_MODULE_CHECK_SOME=1"
// gateLayer runs the gate: the touched manifests through `module check`, every machine composed with the
// change, and the replays of what the mesh runs. It answers the gate's verdict and summary.
func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFile string, env []string,
inToolchain func(string, []string, ...string) []string, running func([]string) error, out *tail, bus, workspace,
name string, say func(step, format string, args ...any)) (string, string) {
inToolchain func(string, []string, ...string) []string, running func([]string) error, out *tail, bus,
name string, hasReplays bool, replaysRefused string, inReplays func([]string, ...string) []string,
say func(step, format string, args ...any)) (string, string) {
// 1. The manifests the change touches, as the judge reads them: a manifest it cannot read, or one with a
// problem the change brings, fails here. **What was already so on the base branch is said and fails
// nothing** — the gate's own rule: a module whose manifest the running controller already finds fault
@@ -864,24 +1109,37 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
// 3. **The replays of what the mesh runs** (to-be 45 §9, M9): mesh-lab's, from its main — reviewed code,
// so given the container runtime the resolver replay raises containers with — against the bus of the
// release the mesh runs and the change's own catalogue when the change is to the catalogue.
if lab := filepath.Join(root, "mesh-lab", "replays"); ctx.Err() == nil {
if _, err := os.Stat(lab); err == nil {
if ctx.Err() == nil && hasReplays && replaysRefused != "" {
// **Never from the workspace** (novox/hq issue 709): without the judge cache there is no place outside it.
say("check", "%s", replaysRefused)
fmt.Fprintf(out, "--- the replays: %s\n", replaysRefused)
return "error", replaysRefused
}
if ctx.Err() == nil {
if hasReplays {
catalogue := filepath.Join(root, "mesh-catalog")
if name == "mesh-catalog" {
catalogue = tree
}
say("check", "the replays of what the mesh runs, from mesh-lab")
fmt.Fprintln(out, "--- the replays (mesh-lab replays/)")
args := inToolchain(lab, []string{EnvTestBus + "=nats://" + bus, "MESH_REPLAY_CATALOGUE=" + catalogue,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")},
// In the replays' own environment (inReplays): reviewed code, the socket, and nothing the workspace holds
// but the catalogue they read.
args := inReplays([]string{EnvTestBus + "=nats://" + bus, "MESH_REPLAY_CATALOGUE=" + catalogue},
"go", "test", "-count=1", "./...")
// The socket goes to the replays alone, never to the change's own code above.
args = append([]string{args[0], "--volume", "/var/run/docker.sock:/var/run/docker.sock"}, args[1:]...)
if err := running(args); err != nil {
if ctx.Err() != nil {
return "error", "the check was ended during the replays"
}
return "fail", "a replay of a core incident fails with this change: " + lastLine(out.String())
if replaysNotStarted(err) {
// The container runtime could not start them (a mount, an image): the check's fault, never the change's.
said := "the replays could not be started, so the change is not judged by them: " + lastLine(out.String())
say("check", "%s", said)
return "error", said
}
said := "a replay of a core incident fails with this change: " + lastLine(out.String())
say("check", "%s", said)
return "fail", said
}
}
}
@@ -896,21 +1154,32 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
// the mesh runs. It answers the judge's path; or, when the change makes its own judge unbuildable, why —
// the change's fault, a failing gate; or an error when the check cannot build a judge at all. A check
// whose gate does not run builds a judge only to hand its scripts one, and goes on without when it cannot.
func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, tree string,
//
// **A pull request into a release branch is judged by that branch's controller** (novox/hq ADR 0358 §2): the one
// beside it as mesh-controller, read at the release branch — with mesh-host's validator from beside it when mesh-host
// has that branch too, or with the change's own validator for a change to the node-engine — never the controller the
// mesh runs. besideAt is the branch each repository beside it was read at.
func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, tree string, besideAt map[string]string,
inToolchain func(string, []string, ...string) []string, say func(step, format string, args ...any)) (string, string, error) {
gated := spec.Gated()
switch spec.Judge {
case "self":
bin := filepath.Join(root, "bin", "judge-of-itself")
if _, err := labelled(ctx, root, "docker", inToolchain(tree,
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
if _, err := labelled(ctx, root, "docker", inToolchain(tree, nil,
"go", "build", "-o", bin, "./cmd/mesh-controller")...); err != nil {
return "", "the change's controller, which judges itself, does not build: " + firstLine(err.Error()), nil
}
say("check", "judged by the change's own controller")
return bin, "", nil
}
bin, judgeTree, err := judge(ctx, labelled, run, root, inToolchain, say)
var bin, judgeTree string
var err error
if spec.Release != "" {
withHost := spec.Judge != "validator" && besideAt["mesh-host"] == spec.Release
bin, judgeTree, err = releaseJudge(ctx, labelled, run, spec, root, withHost, inToolchain, say)
} else {
bin, judgeTree, err = judge(ctx, labelled, run, root, inToolchain, say)
}
if err != nil {
if !gated {
say("check", "no judge for its script: %v", err)
@@ -929,15 +1198,346 @@ func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, t
}
}
withValidator := filepath.Join(root, "bin", "judge-with-this-validator")
if _, err := labelled(ctx, root, "docker", inToolchain(filepath.Join(root, judgeTree),
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
if _, err := labelled(ctx, root, "docker", inToolchain(filepath.Join(root, judgeTree), nil,
"go", "build", "-o", withValidator, "./cmd/mesh-controller")...); err != nil {
if spec.Release != "" {
return "", spec.Release + "'s controller does not build with this change's validator: " +
firstLine(err.Error()), nil
}
return "", "the controller the mesh runs does not build with this change's validator: " + firstLine(err.Error()), nil
}
if spec.Release != "" {
say("check", "judged by %s's controller, with this change's validator", spec.Release)
return withValidator, "", nil
}
say("check", "judged by the controller the mesh runs, with this change's validator")
return withValidator, "", nil
}
// releaseJudge builds the controller that judges a pull request into a release branch (novox/hq ADR 0358 §2): the
// one beside the check as mesh-controller, read at that branch — with mesh-host's validator from beside it in place
// of the one it vendors when withHost. **Built once per release-branch head**: kept in the spec's judge cache by the
// commits it is built from and taken from there while they stand. It answers the binary and the directory it was
// built from.
func releaseJudge(ctx context.Context, run, plain Runner, spec CheckSpec, root string, withHost bool,
inToolchain func(string, []string, ...string) []string, say func(step, format string, args ...any)) (string, string, error) {
const dir = "mesh-controller"
if _, err := os.Stat(filepath.Join(root, dir)); err != nil {
return "", "", fmt.Errorf("no controller of %s beside the check to judge it with", spec.Release)
}
key := fullHeadAt(ctx, plain, filepath.Join(root, dir))
if key == "" {
return "", "", fmt.Errorf("the controller of %s beside the check names no commit", spec.Release)
}
with := ""
if withHost {
host := fullHeadAt(ctx, plain, filepath.Join(root, "mesh-host"))
if host == "" {
return "", "", fmt.Errorf("mesh-host of %s beside the check names no commit", spec.Release)
}
key += "-with-mesh-host-" + host
with = ", with mesh-host's validator of " + spec.Release + " at " + twelve(host)
}
bin := filepath.Join(root, "bin", "mesh-controller")
if took := judgeKept(keptJudges(spec.JudgeCache), key, bin); took {
say("check", "judged by %s's controller at %s%s, built for an earlier check of that head", spec.Release,
twelve(key), with)
return bin, dir, nil
}
if withHost {
vendored := filepath.Join(root, dir, "vendor", "github.com", "novox", "mesh-host")
for _, pkg := range []string{"validate", filepath.Join("internal", "declaration")} {
if err := replaceGoFiles(filepath.Join(root, "mesh-host", pkg), filepath.Join(vendored, pkg)); err != nil {
return "", "", fmt.Errorf("mesh-host's validator of %s could not be put in its controller: %w", spec.Release, err)
}
}
}
// Built where no pull request reaches (JudgeBuildEnv): a judge kept for this head is the head's, nothing else.
if _, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir), nil,
"go", "build", "-o", bin, "./cmd/mesh-controller")...); err != nil {
return "", "", fmt.Errorf("the controller of %s does not build%s: %w", spec.Release, with, err)
}
// Kept before anything of the change runs: nothing a pull request's script writes can reach the cache.
keepJudge(keptJudges(spec.JudgeCache), key, bin, say)
say("check", "judged by %s's controller at %s%s", spec.Release, twelve(key), with)
return bin, dir, nil
}
// JudgeBuildEnv is the environment every judge is built in (novox/hq ADR 0358 §6, issue 482): home, made fresh for the
// build, as HOME and configuration; goCache, the judge cache's Go cache that no check's container mounts; no global
// or system git configuration; no Go environment file; and no version stamped from git, so no git runs at all.
func JudgeBuildEnv(home, goCache string) []string {
return []string{"HOME=" + home, "XDG_CONFIG_HOME=" + filepath.Join(home, ".config"),
"GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "GOENV=off",
"GOCACHE=" + goCache, "GOPATH=" + filepath.Join(home, "go"),
"CGO_ENABLED=0", "GOFLAGS=-mod=vendor -buildvcs=false", "GOPROXY=off"}
}
// judgeBuildDirs are the home and the Go cache a check's judge builds use (ADR 0358 §6). With a judge cache: a home
// made fresh under its homes/ for this check, and its go-cache/, kept warm between checks — both on the host path the
// build agent declares, so the container runtime mounts what the agent sees, and nothing is left in a temporary
// directory of the host. Without one: both under the check's root, which every check empties before it clones. A
// judge cache inside the workspace is refused: a check's containers mount the workspace.
func judgeBuildDirs(cache, workspace, root, id string) (string, string, error) {
if cache == "" {
home, goCache := filepath.Join(root, "judge-home"), filepath.Join(root, "judge-go-cache")
for _, d := range []string{home, goCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return "", "", err
}
}
return home, goCache, nil
}
if withinDir(workspace, cache) {
return "", "", fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers mount: "+
"no judge is built where a pull request could reach it", cache, workspace)
}
if !safeName.MatchString(id) {
return "", "", fmt.Errorf("the check's id %q cannot name its place in the judge cache", id)
}
home, goCache := filepath.Join(cache, "homes", id), filepath.Join(cache, "go-cache")
if err := os.RemoveAll(home); err != nil {
return "", "", err
}
for _, d := range []string{home, goCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return "", "", err
}
}
return home, goCache, nil
}
// replayPlace is where mesh-lab's replays run from and with (novox/hq issue 709), never anything a pull request can
// write: base, the directory made for this check under the judge cache's homes/ (the same path on the host and in the
// build agent), removed after it; lab, the copy of mesh-lab they run; home, their HOME and configuration; and the judge
// cache's Go build and module caches. Empty without a judge cache: the replays are then refused. hasReplays says whether
// mesh-lab held replays when it was cloned, before anything of the change ran.
type replayPlace struct {
base, lab, home, goCache, modCache string
hasReplays bool
}
// replaysNeedTheJudgeCache is the gate's error when mesh-lab holds replays and the build agent has no judge cache.
const replaysNeedTheJudgeCache = "the replays need the build agent's judge cache (mesh-catalog#297); refused rather " +
"than run from the workspace"
// refused is why the replays cannot run here, empty when they can.
func (p replayPlace) refused() string {
if p.base == "" {
return replaysNeedTheJudgeCache
}
return ""
}
// replayDirs makes the replays' place (issue 709, ADR 0358 §6). The replays' build is as trusted as a judge's: it
// compiles mesh-lab's trunk, reviewed code, and nothing of the pull request, which hands them only its catalogue, as
// data. Never the workspace: the gate runs before the replays, a pull request's own code among its containers. Without
// a judge cache there is no place: the build agent declares no other host path, and the replays are refused.
func replayDirs(cache, workspace, root, id string) (replayPlace, error) {
if cache == "" {
return replayPlace{}, nil
}
if withinDir(workspace, cache) {
return replayPlace{}, fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers "+
"mount: no replay is run where a pull request could reach it", cache, workspace)
}
if !safeName.MatchString(id) {
return replayPlace{}, fmt.Errorf("the check's id %q cannot name its place in the judge cache", id)
}
base := filepath.Join(cache, "homes", id+"-replays")
p := replayPlace{base: base, lab: filepath.Join(base, "mesh-lab"), home: filepath.Join(base, "home"),
goCache: filepath.Join(cache, "go-cache"), modCache: filepath.Join(cache, "go-modules")}
if err := os.RemoveAll(base); err != nil {
return replayPlace{}, err
}
for _, d := range []string{p.home, p.goCache, p.modCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return replayPlace{}, err
}
}
return p, nil
}
// copyLab reads whether mesh-lab's checkout, freshly cloned and before anything of the change runs, holds replays, and
// copies it over git's own transport to where the replays run it — when there is such a place.
func (p *replayPlace) copyLab(ctx context.Context, run Runner, from string) error {
_, err := os.Stat(filepath.Join(from, "replays"))
p.hasReplays = err == nil
if p.lab == "" {
return nil
}
return copyCheckout(ctx, run, from, p.lab)
}
// replaysNotStarted says the container runtime could not start the replays' container — its own exit status 125, or
// 126 and 127 for a command it could not run — rather than that a replay failed.
func replaysNotStarted(err error) bool {
var exit *exec.ExitError
if !errors.As(err, &exit) {
return true
}
code := exit.ExitCode()
return code == 125 || code == 126 || code == 127
}
// ReplayEnv is the environment mesh-lab's replays run in (novox/hq issue 709): home as HOME and configuration, the Go
// build and module caches given, no global or system git configuration, no Go environment file, and no version
// stamped from git — as a judge is built (JudgeBuildEnv), with the modules the replays fetch.
func ReplayEnv(home, goCache, modCache string) []string {
return []string{"HOME=" + home, "XDG_CONFIG_HOME=" + filepath.Join(home, ".config"),
"GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "GOENV=off",
"GOCACHE=" + goCache, "GOMODCACHE=" + modCache, "GOPATH=" + filepath.Join(home, "go"),
"GOFLAGS=-buildvcs=false", "GOWORK=off"}
}
// copyCheckout copies a checkout at the commit it stands at into dir, over git's own transport — never by linking
// objects — so what is written into the source afterwards does not reach the copy.
func copyCheckout(ctx context.Context, run Runner, from, dir string) error {
commit := fullHeadAt(ctx, run, from)
if commit == "" {
return fmt.Errorf("%s names no commit", from)
}
if _, err := run(ctx, filepath.Dir(dir), "git", "clone", "--quiet", "--no-local", "--no-checkout", from, dir); err != nil {
return err
}
_, err := run(ctx, dir, "git", "checkout", "--quiet", "--detach", commit)
return err
}
// keptJudges is where a judge cache keeps the release-branch judges, empty when there is no judge cache.
func keptJudges(cache string) string {
if cache == "" {
return ""
}
return filepath.Join(cache, "judges")
}
// judgesKept is how many release-branch judges a judge cache keeps: a head moved past is not built from again.
const judgesKept = 4
// judgeKept puts the judge kept under key in the cache at bin, and says whether there was one.
func judgeKept(cache, key, bin string) bool {
if cache == "" {
return false
}
body, err := os.ReadFile(filepath.Join(cache, key, "mesh-controller"))
if err != nil {
return false
}
if err := os.MkdirAll(filepath.Dir(bin), 0o755); err != nil {
return false
}
if err := os.WriteFile(bin, body, 0o755); err != nil {
return false
}
now := time.Now()
_ = os.Chtimes(filepath.Join(cache, key), now, now)
return true
}
// keepJudge keeps a judge just built under key, and lets go of all but the judgesKept most recently used. A judge
// that cannot be kept is said and built again next time: the cache only saves time.
func keepJudge(cache, key, bin string, say func(step, format string, args ...any)) {
if cache == "" {
return
}
body, err := os.ReadFile(bin)
if err == nil {
err = os.MkdirAll(filepath.Join(cache, key), 0o755)
}
if err == nil {
kept := filepath.Join(cache, key, "mesh-controller")
if err = os.WriteFile(kept+".part", body, 0o755); err == nil {
err = os.Rename(kept+".part", kept)
}
}
if err != nil {
say("check", "the judge could not be kept for the next check, which builds it again: %v", err)
return
}
entries, err := os.ReadDir(cache)
if err != nil || len(entries) <= judgesKept {
return
}
type used struct {
name string
at time.Time
}
var all []used
for _, e := range entries {
if info, err := e.Info(); err == nil && e.IsDir() {
all = append(all, used{e.Name(), info.ModTime()})
}
}
sort.Slice(all, func(i, j int) bool { return all[i].at.After(all[j].at) })
for _, u := range all[min(judgesKept, len(all)):] {
if u.name != key {
_ = os.RemoveAll(filepath.Join(cache, u.name))
}
}
}
// fullHeadAt is the whole commit a checkout stands at, empty when it names none.
func fullHeadAt(ctx context.Context, run Runner, tree string) string {
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", "HEAD^{commit}")
if err != nil {
return ""
}
return strings.TrimSpace(out)
}
// releaseValidator says which node-engine validator a release-branch check's judge is built with (novox/hq ADR 0358
// §2), by the rule judgeFor builds it by.
func releaseValidator(spec CheckSpec, at, commits map[string]string) string {
switch {
case spec.Judge == "self":
return "the validator the change's own controller vendors"
case spec.Judge == "validator":
return "the change's own validator"
case at["mesh-host"] == spec.Release:
return fmt.Sprintf("mesh-host's validator of %s at %s", spec.Release, commits["mesh-host"])
}
return fmt.Sprintf("the validator %s's controller vendors (mesh-host has no branch %s)", spec.Release, spec.Release)
}
// releaseWords is how a release-branch check says what judged it and what it read (novox/hq ADR 0358 §4).
func releaseWords(release, judgedBy, validator string, at, commits map[string]string) string {
var read []string
for _, dir := range sortedKeys(at) {
read = append(read, fmt.Sprintf("%s %s at %s", dir, refWords(at[dir]), commits[dir]))
}
said := fmt.Sprintf("judged for the release branch %s by its controller at %s, with %s", release, judgedBy, validator)
if len(read) > 0 {
said += "; read beside it: " + strings.Join(read, ", ")
}
return said
}
// releaseMark opens each layer's summary of a release-branch check, so a status says it first.
func releaseMark(release, judgedBy string) string {
return fmt.Sprintf("[release %s, judged by its controller %s] ", release, judgedBy)
}
// refWords is a ref as a line says it: a commit by its first eight characters, a branch whole.
func refWords(ref string) string {
if commitLike.MatchString(ref) {
return short(ref)
}
return ref
}
// commitLike is a ref that names a commit.
var commitLike = regexp.MustCompile(`^[0-9a-f]{7,40}$`)
func sortedKeys(m map[string]string) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// replaceGoFiles puts a package's Go files — never its tests — in place of another copy's.
func replaceGoFiles(from, into string) error {
old, err := filepath.Glob(filepath.Join(into, "*.go"))
@@ -1172,8 +1772,7 @@ func judge(ctx context.Context, run, plain Runner, root string, inToolchain func
say func(step, format string, args ...any)) (string, string, error) {
bin := filepath.Join(root, "bin", "mesh-controller")
build := func(dir string) error {
_, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir),
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
_, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir), nil,
"go", "build", "-o", bin, "./cmd/mesh-controller")...)
return err
}
+269 -65
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"os"
"path/filepath"
"sync"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
@@ -35,8 +36,17 @@ type CheckBuilding struct {
Publish Publisher
Scratch Scratch
Held map[string]string
// Running is each artifact of the build the mesh has registered for its module, keyed "<module>/<artifact>"
// (novox/hq issue 721): what a build for the merge is compared with — never Held's newest build, which may be one
// its gate failed and the mesh put back.
Running map[string]string
Npmrc Npmrc
Seats map[string]string
// Named are the builds the build seat's verdicts of earlier checks of this pull request at this head name
// (novox/hq issue 703): a build kept under a module's key is kept again, instead of built, only when one of them
// names its key and every digest, and the build is this pull request's at this head. The record alone is never
// enough.
Named []NamedBuild
}
// CheckBuilt is one module a check built for its merge, as its verdict lists it (§1): its key and its artifacts, or
@@ -48,6 +58,14 @@ type CheckBuilt struct {
// NotBuilt says why it was not: no key, a build that failed, or a module after one of those.
NotBuilt string
Took time.Duration
// Kept is the check whose build this check kept instead of building, its key unchanged (novox/hq issue 703):
// the verdict names that build, by key and digest, as that check's. Empty when this check built it.
Kept string
// Same says every artifact came out the same as the build the mesh holds of the module (novox/hq issue 721), and
// SameAs names, per artifact, the held reference it was compared with: the walk leaves the module where it is
// while the mesh still holds that build. Empty when it differs, or could not be compared.
Same string
SameAs map[string]string
}
// skipsTheBuilds is why a check builds nothing for its merge; empty when it builds.
@@ -60,13 +78,45 @@ func skipsTheBuilds(v CheckVerdict) string {
return ""
}
// buildForTheMerge builds each module the plan moves, tier by tier, into the scratch namespace, and keeps what it
// built under its keys. A module standing after one not built is not built either: what it stands on in the walk
// is not what it would stand on here. tree is the check's clone and merge the merge commit it made there.
// buildForTheMerge builds each module the plan moves into the scratch namespace (makeForTheMerge), and keeps what it
// built under its keys (keepForTheMerge): the order of a check that builds after it has judged.
func buildForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with CheckBuilding, workspace, tree, merge string,
judged CheckVerdict, forge GitCredential, log Log, say func(step, format string, args ...any)) []CheckBuilt {
out, kept := makeForTheMerge(ctx, run, spec, with, workspace, tree, merge, judged, forge, log, say)
return keepForTheMerge(ctx, spec, with, out, kept, say)
}
// makeForTheMerge builds each module the plan moves, tier by tier, into the scratch namespace — **the modules of a
// tier at the same time** (novox/hq issue 716): none of them stands on another, and a tier waits only for the ones
// before it, whose builds it is handed. A module standing after one not built is not built either: what it stands on
// in the walk is not what it would stand on here. tree is the check's copy of the merge and merge the merge commit it
// made there; workspace is where each build's working copy is made, one directory per module, which the host's
// container engine sees at the same path (novox/hq issue 713). Answers what each module came to, in the plan's order,
// and the builds to keep — nothing is kept here.
func makeForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with CheckBuilding, workspace, tree, merge string,
judged CheckVerdict, forge GitCredential, log Log, say func(step, format string, args ...any)) ([]CheckBuilt, []CheckBuild) {
if len(spec.Builds) == 0 || with.Publish == nil || with.Scratch == nil {
return nil
return nil, nil
}
// What an earlier check left, ended before it could remove it, is not built on.
place := filepath.Join(workspace, "for-the-merge")
_ = os.RemoveAll(place)
defer os.RemoveAll(place)
// One log, said to by builds running at once: a line at a time.
var saying sync.Mutex
if log != nil {
inner := log
log = func(step, message string) {
saying.Lock()
defer saying.Unlock()
inner(step, message)
}
}
inner := say
say = func(step, format string, args ...any) {
saying.Lock()
defer saying.Unlock()
inner(step, format, args...)
}
ref := merge
if ref == "" {
@@ -76,76 +126,230 @@ func buildForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with Chec
for k, v := range with.Held {
held[k] = v
}
var out []CheckBuilt
out := make([]CheckBuilt, len(spec.Builds))
results := make([]Result, len(spec.Builds))
var kept []CheckBuild
stopAt, stopWhy := -1, ""
for _, b := range spec.Builds {
began := time.Now()
if stopAt >= 0 && b.Tier > stopAt {
out = append(out, CheckBuilt{Module: b.Module, NotBuilt: stopWhy})
continue
for start := 0; start < len(spec.Builds); {
tier := spec.Builds[start].Tier
end := start
for end < len(spec.Builds) && spec.Builds[end].Tier == tier {
end++
}
if b.Skip != "" {
say("build", "%s not built for the merge: %s", b.Module, b.Skip)
out = append(out, CheckBuilt{Module: b.Module, NotBuilt: b.Skip})
if stopAt < 0 || b.Tier < stopAt {
stopAt, stopWhy = b.Tier, "it stands after "+b.Module+", which was not built for the merge"
// What this tier's builds are handed: the builds of the tiers before it, read alone while they run.
handed := map[string]string{}
for k, v := range held {
handed[k] = v
}
var building sync.WaitGroup
for i := start; i < end; i++ {
b := spec.Builds[i]
if stopAt >= 0 && b.Tier > stopAt {
out[i] = CheckBuilt{Module: b.Module, NotBuilt: stopWhy}
continue
}
continue
}
repository, at := b.Repository, b.Ref
if b.Merged {
repository, at = tree, ref
}
say("build", "for the merge: %s (tier %d)", b.Module, b.Tier)
var mirrored []string
result, err := build(ctx, run, scratchPublisher{to: with.Publish}, repository, b.Path, at,
filepath.Join(workspace, "for-the-merge"), held, with.Npmrc, forge, log, &mirrored, &Reuse{keyFirst: true},
with.Seats)
built := CheckBuilt{Module: b.Module, Key: result.ReuseKey, Took: time.Since(began)}
switch {
case errors.Is(err, errNoKey):
built.NotBuilt = err.Error()
case err != nil:
built.NotBuilt = "its build failed: " + err.Error()
case result.ReuseKey == "":
built.NotBuilt = "it has no reuse key: " + result.NoKey
}
if built.NotBuilt != "" {
say("build", "%s not built for the merge: %s", b.Module, built.NotBuilt)
out = append(out, built)
if stopAt < 0 || b.Tier < stopAt {
stopAt, stopWhy = b.Tier, "it stands after "+b.Module+", which was not built for the merge"
if b.Skip != "" {
say("build", "%s not built for the merge: %s", b.Module, b.Skip)
out[i] = CheckBuilt{Module: b.Module, NotBuilt: b.Skip}
continue
}
continue
}
built.Made = result.Built
for _, m := range result.Built {
held[b.Module+"/"+m.Name] = m.Reference
}
say("build", "%s built for the merge under %s (%s)", b.Module, twelve(result.ReuseKey[3:]), built.Took.Round(time.Second))
out = append(out, built)
if len(result.Built) > 0 {
kept = append(kept, CheckBuild{Key: result.ReuseKey, Module: b.Module, Parts: result.ReuseParts,
Made: result.Built, Pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number},
Head: judged.Head, Trunk: judged.Trunk, Check: spec.ID, At: time.Now().UTC()})
}
}
_ = os.RemoveAll(filepath.Join(workspace, "for-the-merge"))
if len(kept) > 0 {
if err := with.Scratch.Keep(ctx, PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, spec.ID,
time.Now().UTC(), kept); err != nil {
// Built and not findable: the walk builds, and says no check built it. Said, never the check's verdict.
say("build", "the builds for the merge could not be kept, and the walk will build: %v", err)
if err := with.Scratch.Discard(ctx, kept); err != nil {
say("build", "and what they made could not all be let go of: %v", err)
repository, at := b.Repository, b.Ref
if b.Merged {
repository, at = tree, ref
}
for i := range out {
if out[i].NotBuilt == "" {
out[i].NotBuilt = fmt.Sprintf("built, and could not be kept: %v", err)
name := b.Module
if !safeName.MatchString(name) {
name = fmt.Sprintf("module-%d", i)
}
building.Add(1)
go func(i int, b CheckBuildSpec, repository, at, dir string) {
defer building.Done()
began := time.Now()
say("build", "for the merge: %s (tier %d)", b.Module, b.Tier)
var mirrored []string
result, err := build(ctx, run, scratchPublisher{to: with.Publish}, repository, b.Path, at, dir, handed,
with.Npmrc, forge, log, &mirrored, &Reuse{keyFirst: true, Scratch: with.Scratch, Named: with.Named,
pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, head: judged.Head},
with.Seats)
built := CheckBuilt{Module: b.Module, Key: result.ReuseKey, Took: time.Since(began)}
switch {
case errors.Is(err, errNoKey):
built.NotBuilt = err.Error()
case err != nil:
built.NotBuilt = "its build failed: " + err.Error()
case result.ReuseKey == "":
built.NotBuilt = "it has no reuse key: " + result.NoKey
}
if built.NotBuilt == "" {
// **Compared with what runs** (novox/hq issue 721): a failed comparison says nothing, and the
// module moves as before.
same, against, err := sameAsHeld(ctx, run, dir, b.Module, result.Built, with.Running)
switch {
case err != nil:
say("build", "%s could not be compared with the build the mesh holds, so it moves: %v", b.Module, err)
case same != "":
built.Same, built.SameAs = same, against
say("build", "%s %s", b.Module, same)
}
}
out[i], results[i] = built, result
}(i, b, repository, at, filepath.Join(place, name))
}
building.Wait()
for i := start; i < end; i++ {
b, built, result := spec.Builds[i], &out[i], results[i]
if built.NotBuilt != "" {
if built.Key != "" || built.Took > 0 {
say("build", "%s not built for the merge: %s", b.Module, built.NotBuilt)
}
if stopAt < 0 || b.Tier < stopAt {
stopAt, stopWhy = b.Tier, "it stands after "+b.Module+", which was not built for the merge"
}
continue
}
built.Made = result.Built
for _, m := range result.Built {
held[b.Module+"/"+m.Name] = m.Reference
}
if result.Reused != nil {
// **Kept, not built** (novox/hq issue 703): the earlier check's build, named as that check's, so the
// walk takes it as the verdict names it; this check's document names it too, so it stays while this
// check does (ADR 0350 §5).
built.Kept = result.Reused.From.Check
say("build", "%s kept for the merge under %s: %s", b.Module, twelve(result.ReuseKey[3:]),
result.Reused.From.Said())
kept = append(kept, result.Reused.From)
continue
}
say("build", "%s built for the merge under %s (%s)", b.Module, twelve(result.ReuseKey[3:]),
built.Took.Round(time.Second))
if len(result.Built) > 0 {
kept = append(kept, CheckBuild{Key: result.ReuseKey, Module: b.Module, Parts: result.ReuseParts,
Made: result.Built, Pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number},
Head: judged.Head, Trunk: judged.Trunk, Check: spec.ID, At: time.Now().UTC()})
}
}
start = end
}
return out, kept
}
// keepForTheMerge keeps a check's builds under their keys, its own document naming them, so the walk of its merge
// finds them; what cannot be kept is let go of, said, and each module says it — never the check's verdict.
func keepForTheMerge(ctx context.Context, spec CheckSpec, with CheckBuilding, out []CheckBuilt, kept []CheckBuild,
say func(step, format string, args ...any)) []CheckBuilt {
if len(kept) == 0 {
return out
}
if err := with.Scratch.Keep(ctx, PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, spec.ID,
time.Now().UTC(), kept); err != nil {
// Built and not findable: the walk builds, and says no check built it. Said, never the check's verdict.
say("build", "the builds for the merge could not be kept, and the walk will build: %v", err)
if err := with.Scratch.Discard(ctx, kept); err != nil {
say("build", "and what they made could not all be let go of: %v", err)
}
for i := range out {
if out[i].NotBuilt == "" {
out[i].NotBuilt = fmt.Sprintf("built, and could not be kept: %v", err)
}
}
}
return out
}
// discardForTheMerge lets go of builds a check made beside its tests when the check did not pass (novox/hq issue 716):
// a change that fails is not merged, and nothing it built is kept. A build another check names stays (Discard).
func discardForTheMerge(ctx context.Context, with CheckBuilding, kept []CheckBuild, why string,
say func(step, format string, args ...any)) {
say("check", "%s: the %d build(s) made beside it are let go of", why, len(kept))
if len(kept) == 0 {
return
}
if err := with.Scratch.Discard(ctx, kept); err != nil {
say("build", "what they made could not all be let go of, and goes with the week's sweep: %v", err)
}
}
// besideTheTests are a check's builds for its merge running while its gate and its own check run (novox/hq issue
// 716): started once every clone is made, waited for when the check has judged, kept when it passed and let go of
// when it did not.
type besideTheTests struct {
spec CheckSpec
with CheckBuilding
cancel context.CancelFunc
done chan struct{}
place string
out []CheckBuilt
kept []CheckBuild
began time.Time
settled bool
say func(step, format string, args ...any)
}
// buildsBesideTheTests is where a check's builds run beside its tests: a directory made for this check under the judge
// cache's homes/ (ADR 0358 §6), on the host path the build agent declares — so the host's container engine mounts what
// the agent sees (issue 713) — and never mounted by a check's container, so nothing a pull request's script runs can
// change a build while it is made (issue 703). Empty, and the builds wait for the check, without a judge cache.
func buildsBesideTheTests(cache, workspace, id string) (string, error) {
if cache == "" {
return "", nil
}
if withinDir(workspace, cache) {
return "", fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers mount: "+
"nothing is built for the merge where a pull request could reach it", cache, workspace)
}
if !safeName.MatchString(id) {
return "", fmt.Errorf("the check's id %q cannot name its place in the judge cache", id)
}
place := filepath.Join(cache, "homes", id+"-builds")
if err := os.RemoveAll(place); err != nil {
return "", err
}
if err := os.MkdirAll(place, 0o700); err != nil {
return "", err
}
return place, nil
}
// startBesideTheTests starts the builds for the merge in place, from merged, the check's copy of the merge.
func startBesideTheTests(ctx context.Context, run Runner, spec CheckSpec, place, merged string, judged CheckVerdict,
forge GitCredential, log Log, say func(step, format string, args ...any)) *besideTheTests {
building, cancel := context.WithCancel(ctx)
b := &besideTheTests{spec: spec, with: *spec.BuildWith, cancel: cancel, done: make(chan struct{}), place: place,
began: time.Now(), say: say}
say("check", "building what the merge moves (%d module(s)) beside its tests", len(spec.Builds))
go func() {
defer close(b.done)
b.out, b.kept = makeForTheMerge(building, run, spec, b.with, place, merged, judged.Merge, judged, forge, log, say)
}()
return b
}
// settle waits for the builds and keeps them when the check passed; a check that did not pass lets go of them, and
// its verdict names none.
func (b *besideTheTests) settle(ctx context.Context, verdict CheckVerdict) []CheckBuilt {
<-b.done
b.settled = true
defer os.RemoveAll(b.place)
if why := skipsTheBuilds(verdict); why != "" {
discardForTheMerge(ctx, b.with, b.kept, why, b.say)
return nil
}
built := keepForTheMerge(ctx, b.spec, b.with, b.out, b.kept, b.say)
b.say("check", "built for the merge: %d of %d module(s), beside the tests (%s)", builtCount(built), len(built),
time.Since(b.began).Round(time.Second))
return built
}
// abandon ends builds a check that ended without a verdict leaves, and lets go of what they made.
func (b *besideTheTests) abandon() {
if b.settled {
return
}
b.cancel()
<-b.done
defer os.RemoveAll(b.place)
letGo, done := context.WithTimeout(context.Background(), time.Minute)
defer done()
discardForTheMerge(letGo, b.with, b.kept, "the check ended without a verdict", b.say)
}
+50
View File
@@ -210,3 +210,53 @@ func TestACheckMergeRunsNoHook(t *testing.T) {
t.Fatalf("a hook ran when the check made its merge: %s", said)
}
}
// What a check's tests change in the workspace never reaches what it builds for the merge (novox/hq issue 703): the
// builds are made from a copy of the merge, outside the workspace, cloned before anything of the change runs.
func TestWhatTheTestsChangeInTheWorkspaceNeverReachesTheBuild(t *testing.T) {
repo, head, _ := aTrunkAndABranch(t, map[string]string{"main.go": "package main // the merge\n"},
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
workspace := t.TempDir()
tree := filepath.Join(workspace, "check", "checked")
if out, err := exec.Command("git", "clone", "--quiet", repo, tree).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
dir, cloned, err := checkoutForTheBuilds(t.Context(), Command, tree, head, workspace)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)
if withinDir(workspace, cloned) {
t.Fatalf("the builds' checkout %s is inside the workspace %s the tests mount", cloned, workspace)
}
// The tests rewrite the tree, its branch and every object it holds.
if err := os.WriteFile(filepath.Join(tree, "main.go"), []byte("package main // planted\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := filepath.WalkDir(filepath.Join(tree, ".git", "objects"), func(path string, d os.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
_ = os.Chmod(path, 0o644)
return os.WriteFile(path, []byte("planted"), 0o644)
}); err != nil {
t.Fatal(err)
}
// The build clones the merge from its own checkout, as buildForTheMerge does, and reads what the merge holds.
build := filepath.Join(t.TempDir(), "source")
if out, err := exec.Command("git", "clone", "--quiet", cloned, build).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
if out, err := exec.Command("git", "-C", build, "checkout", "--quiet", head).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
body, err := os.ReadFile(filepath.Join(build, "main.go"))
if err != nil {
t.Fatal(err)
}
if string(body) != "package main // the merge\n" {
t.Fatalf("the build read %q: what the tests wrote reached it", body)
}
}
+12
View File
@@ -590,3 +590,15 @@ func TestARepositoryInTwoLanguagesIsCheckedInBoth(t *testing.T) {
t.Errorf("a part in a toolchain the mesh does not hold answered %+v — never a pass", layer)
}
}
// A branch tried before its pull request may narrow what the repository's script runs (novox/hq issue 686); a
// pull request's check is handed nothing, and runs whole.
func TestOnlyATrysSelectionReachesTheScript(t *testing.T) {
if env := selection(CheckSpec{Number: 7}); len(env) != 0 {
t.Fatalf("a pull request's check was handed a selection: %v", env)
}
env := selection(CheckSpec{Tests: "^TestX$", StoreOnly: true})
if len(env) != 2 || env[0] != "MESH_CHECK_TESTS=^TestX$" || env[1] != "MESH_CHECK_STORE_ONLY=1" {
t.Fatalf("a try's selection: %v", env)
}
}
+178
View File
@@ -0,0 +1,178 @@
package builder
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"runtime"
"strings"
)
// **A built image's bases are linked into its repository before it is pushed** (novox/hq issue 694).
//
// A push reads every blob of the image out of this machine's store, unless the repository it goes to
// already holds that blob: the runtime asks the registry first, blob by blob, and reads locally only
// what the registry lacks. A runtime keeping its images in containerd's store — the default for a
// fresh install from docker 29 on — does not always hold the compressed layers of the base an image
// was built FROM: a layer whose unpacked form it already had is never fetched. The push of such an
// image then fails on a layer this machine never had ("not all of them are available locally"), while
// the very bytes sit in this registry, under the repository the base was mirrored into.
//
// So the base's blobs are mounted into the image's repository first. A mount moves no bytes, and once
// the repository holds a blob the push asks this machine for nothing of it — whichever store the
// runtime keeps. A runtime that holds every layer (the classic store) is not changed by this: its push
// finds the same blobs already there.
// BaseLender links what a build's bases hold into the repository its image is about to be pushed to.
type BaseLender interface {
// LendBases mounts every blob of each base this registry holds — for the platform this machine
// builds — into `repository`, and answers how many it mounted.
LendBases(ctx context.Context, repository string, bases []string) (int, error)
}
// platformed is an index's entry: a manifest and the platform it is for.
type platformed struct {
descriptor
Platform *struct {
OS string `json:"os"`
Architecture string `json:"architecture"`
} `json:"platform"`
}
// LendBases mounts each base's blobs into the repository. A base this registry does not hold under its
// repository — named by tag, or held somewhere else — lends nothing and is no error: the push reads it
// from this machine's store as before.
func (r Registry) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
mounted := 0
for _, base := range bases {
name, digest, pinned := strings.Cut(base, "@")
if !pinned || !strings.HasPrefix(digest, "sha256:") {
continue
}
where, err := parseReference(name + "@" + digest)
if err != nil {
continue
}
from := where.repository
if from == repository {
continue
}
blobs, held, err := r.blobsOf(ctx, from, digest)
if err != nil {
return mounted, fmt.Errorf("reading %s: %w", base, err)
}
if !held {
continue
}
for _, blob := range blobs {
did, err := r.mount(ctx, repository, from, blob)
if err != nil {
return mounted, fmt.Errorf("linking %s of %s into %s: %w", blob, base, repository, err)
}
if did {
mounted++
}
}
}
return mounted, nil
}
// blobsOf is the config and layers of the manifest under repository@digest — for an index, of each
// manifest it names for this machine's platform. False when the registry holds no such manifest there.
func (r Registry) blobsOf(ctx context.Context, repository, digest string) ([]string, bool, error) {
body, held, err := r.manifest(ctx, repository, digest)
if err != nil || !held {
return nil, held, err
}
var document struct {
Manifests []platformed `json:"manifests"`
Config *descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
if err := json.Unmarshal(body, &document); err != nil {
return nil, false, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
var blobs []string
if document.Config != nil {
blobs = append(blobs, document.Config.Digest)
}
for _, l := range document.Layers {
blobs = append(blobs, l.Digest)
}
for _, m := range document.Manifests {
if m.Platform == nil || m.Platform.OS != runtime.GOOS || m.Platform.Architecture != runtime.GOARCH {
continue
}
more, held, err := r.blobsOf(ctx, repository, m.Digest)
if err != nil {
return nil, false, err
}
if held {
blobs = append(blobs, more...)
}
}
return blobs, true, nil
}
// manifest reads one manifest document from this registry; false when it is not there.
func (r Registry) manifest(ctx context.Context, repository, digest string) ([]byte, bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet,
"http://"+r.Address+"/v2/"+repository+"/manifests/"+digest, nil)
if err != nil {
return nil, false, err
}
request.Header.Set("Accept", manifestAccept)
response, err := r.client().Do(request)
if err != nil {
return nil, false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
}
defer response.Body.Close()
if response.StatusCode == http.StatusNotFound {
return nil, false, nil
}
if response.StatusCode != http.StatusOK {
return nil, false, fmt.Errorf("%s/%s@%s: %s", r.Address, repository, digest, response.Status)
}
body, err := io.ReadAll(io.LimitReader(response.Body, 4<<20))
return body, err == nil, err
}
// mount links one blob from one repository into another; false when the repository already held it.
func (r Registry) mount(ctx context.Context, repository, from, digest string) (bool, error) {
base := "http://" + r.Address + "/v2/" + repository
if there, err := r.has(ctx, base+"/blobs/"+digest); err != nil {
return false, err
} else if there {
return false, nil
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
base+"/blobs/uploads/?mount="+digest+"&from="+from, nil)
if err != nil {
return false, err
}
begun, err := r.client().Do(start)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
}
begun.Body.Close()
if begun.StatusCode == http.StatusCreated {
return true, nil
}
if begun.StatusCode == http.StatusAccepted {
// Not mounted, and an upload begun in its place: let go of it, so nothing is left half-open.
if location := begun.Header.Get("Location"); location != "" {
if strings.HasPrefix(location, "/") {
location = "http://" + r.Address + location
}
if cancel, err := http.NewRequestWithContext(ctx, http.MethodDelete, location, nil); err == nil {
if done, err := r.client().Do(cancel); err == nil {
done.Body.Close()
}
}
}
return false, fmt.Errorf("%s did not mount it from %s", r.Address, from)
}
return false, fmt.Errorf("%s answered %s when asked to mount it from %s", r.Address, begun.Status, from)
}
+140
View File
@@ -0,0 +1,140 @@
package builder
import (
"context"
"encoding/json"
"net/http/httptest"
"runtime"
"strings"
"testing"
)
// A built image's bases are linked into its repository before it is pushed (novox/hq issue 694): a
// runtime keeping containerd's image store may not hold a base layer it never had to fetch, and its push
// asks this machine only for what the repository lacks.
// aBaseForThisMachine puts an index into the registry under `repository`: one manifest for this
// machine's platform, one for another, and answers the index's digest and each manifest's blobs.
func aBaseForThisMachine(t *testing.T, dst *aRegistryOfRepositories, repository string) (string, []string, []string) {
t.Helper()
put := func(arch string) (string, []string) {
config := []byte(`{"architecture":"` + arch + `"}`)
layer := []byte(arch + " layer bytes")
for _, b := range [][]byte{config, layer} {
dst.blobs[digestOf(b)] = b
dst.link(repository, digestOf(b))
}
m, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaManifestOCI,
"config": map[string]any{"mediaType": "application/vnd.oci.image.config.v1+json", "digest": digestOf(config), "size": len(config)},
"layers": []map[string]any{{"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": digestOf(layer), "size": len(layer)}},
})
dst.manifests[repository+"@"+digestOf(m)] = m
return digestOf(m), []string{digestOf(config), digestOf(layer)}
}
other := "s390x"
if runtime.GOARCH == other {
other = "amd64"
}
mine, mineBlobs := put(runtime.GOARCH)
theirs, theirBlobs := put(other)
index, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaIndexOCI,
"manifests": []map[string]any{
{"mediaType": mediaManifestOCI, "digest": mine, "size": 1, "platform": map[string]string{"os": runtime.GOOS, "architecture": runtime.GOARCH}},
{"mediaType": mediaManifestOCI, "digest": theirs, "size": 1, "platform": map[string]string{"os": "linux", "architecture": other}},
},
})
dst.manifests[repository+"@"+digestOf(index)] = index
return digestOf(index), mineBlobs, theirBlobs
}
func TestABuildsBaseIsLinkedIntoItsRepositoryBeforeThePush(t *testing.T) {
dst := newRegistryOfRepositories()
server := httptest.NewServer(dst.handler())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
r := Registry{Address: address}
const from = "upstream/docker.io/library/golang"
index, mine, theirs := aBaseForThisMachine(t, dst, from)
bases := []string{
address + "/" + from + "@" + index,
// Not bases this registry holds: a package version, a tag, an image held nowhere here.
"@novox/mesh-sdk@1.2.3",
"golang:1-alpine",
"elsewhere.invalid/thing@sha256:" + strings.Repeat("e", 64),
}
n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases)
if err != nil {
t.Fatal(err)
}
// Every blob of this machine's platform is in the image's repository, so a push asks this machine's
// store for none of them; another platform's are not, since nothing built here stands on them.
for _, blob := range mine {
if !dst.links["mesh-tools-go/build"][blob] {
t.Fatalf("the base's blob %s is not in the image's repository", blob)
}
}
for _, blob := range theirs {
if dst.links["mesh-tools-go/build"][blob] {
t.Fatalf("another platform's blob %s was linked", blob)
}
}
if n != len(mine) || dst.mounts != len(mine) || dst.uploads != 0 {
t.Fatalf("%d linked, %d mounts, %d uploads; want %d mounts and no bytes moved", n, dst.mounts, dst.uploads, len(mine))
}
// Asked again, nothing is mounted twice.
if n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases); err != nil || n != 0 {
t.Fatalf("a second lending mounted %d: %v", n, err)
}
}
// A registry that will not mount is said, not hidden: the push is still tried, from the store.
func TestABaseTheRegistryWillNotMountIsAnError(t *testing.T) {
dst := newRegistryOfRepositories()
dst.noMount = true
server := httptest.NewServer(dst.handler())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
index, _, _ := aBaseForThisMachine(t, dst, "upstream/thing")
_, err := Registry{Address: address}.LendBases(context.Background(), "hello/server",
[]string{address + "/upstream/thing@" + index})
if err == nil || !strings.Contains(err.Error(), "did not mount") {
t.Fatalf("a refused mount passed silently: %v", err)
}
}
// lending records what a build lent, and when, against the push.
type lending struct {
*mirroring
order []string
}
func (l *lending) LendBases(_ context.Context, repository string, bases []string) (int, error) {
l.order = append(l.order, "lend "+repository+" "+strings.Join(bases, " "))
return len(bases), nil
}
func (l *lending) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
l.order = append(l.order, "push "+repository)
return l.mirroring.PublishImage(ctx, localTag, repository)
}
func TestAnImageIsPushedOnlyAfterItsBasesAreLent(t *testing.T) {
r, workspace := aRepository(t, anImage, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
r.contents["modules/bus/module.json"] = anImage
l := &lending{mirroring: &mirroring{recorded: r, at: "127.0.0.1:5000"}}
if _, err := Build(context.Background(), r.run, l, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
nil, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatal(err)
}
want := []string{
"lend bus/server 127.0.0.1:5000/upstream/docker.io/vendor/server@sha256:" + strings.Repeat("1", 64),
"push bus/server",
}
if strings.Join(l.order, "\n") != strings.Join(want, "\n") {
t.Fatalf("the build did:\n%s\nwant:\n%s", strings.Join(l.order, "\n"), strings.Join(want, "\n"))
}
}
+870
View File
@@ -0,0 +1,870 @@
package builder
import (
"fmt"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// **A pull request into a release branch is judged by the controller of that branch** (novox/hq ADR 0358, issue
// 704): the build seat's check builds the judge from mesh-controller read at the release branch — with mesh-host's
// validator of that branch when mesh-host has one — reads every repository beside it at that branch where it has one
// and at its trunk otherwise, and names the branch, the judge's commit and each repository read with its commit.
const releaseBranch = "feat/57-config-and-provisioning"
// gitInRepo runs git in a repository made for a test.
func gitInRepo(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
// commitFiles writes files into a repository and commits them on the branch checked out.
func commitFiles(t *testing.T, dir string, files map[string]string) string {
t.Helper()
for name, body := range files {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
gitInRepo(t, dir, "add", "-A")
gitInRepo(t, dir, "commit", "--quiet", "-m", "x")
return gitInRepo(t, dir, "rev-parse", "HEAD")
}
// aRepositoryWithARelease is a repository whose main holds mainFiles and, when releaseFiles is not nil, a release
// branch over them holding releaseFiles. It answers the repository, main's head and the release branch's head.
func aRepositoryWithARelease(t *testing.T, mainFiles, releaseFiles map[string]string) (string, string, string) {
t.Helper()
dir := t.TempDir()
gitInRepo(t, dir, "init", "--quiet", "-b", "main")
main := commitFiles(t, dir, mainFiles)
if releaseFiles == nil {
return dir, main, ""
}
gitInRepo(t, dir, "checkout", "--quiet", "-b", releaseBranch)
head := commitFiles(t, dir, releaseFiles)
gitInRepo(t, dir, "checkout", "--quiet", "main")
return dir, main, head
}
// aJudgeOf is a controller that judges as told: its gate passes, saying who it is and the word of the node-engine
// validator it vendors — the one it was built with.
func aJudgeOf(who string) map[string]string {
return map[string]string{
"go.mod": "module example.org/judge\n\ngo 1.22\n\nrequire github.com/novox/mesh-host v0.0.0\n",
"vendor/modules.txt": "# github.com/novox/mesh-host v0.0.0\n## explicit; go 1.22\ngithub.com/novox/mesh-host/validate\n",
"vendor/github.com/novox/mesh-host/validate/v.go": "package validate\n\nconst Word = \"the vendored validator\"\n",
"vendor/github.com/novox/mesh-host/internal/declaration/d.go": "package declaration\n",
"cmd/mesh-controller/main.go": `package main
import (
"fmt"
"os"
"github.com/novox/mesh-host/validate"
)
func main() {
if len(os.Args) == 2 && os.Args[1] == "merge-gate" {
fmt.Println("usage: merge-gate --facts <file> --store <postgres>")
os.Exit(2)
}
fmt.Printf("{\"verdict\":\"pass\",\"summary\":\"%s with %s\"}\n", "` + who + `", validate.Word)
}
`,
}
}
// aHostSaying is the node-engine's validator, saying a word.
func aHostSaying(word string) map[string]string {
return map[string]string{
"validate/v.go": "package validate\n\nconst Word = \"" + word + "\"\n",
"internal/declaration/d.go": "package declaration\n",
"validate/v_test.go": "package validate\n",
"internal/declaration/x_test.go": "package declaration\n",
}
}
// aRuntimeThatBuildsTheJudge puts a container runtime on PATH that runs every container's command on this machine,
// in the directory and with the environment the check gives it — the judge's build too — and a store and a bus that
// answer. It answers the file each `go` the check runs is written to.
func aRuntimeThatBuildsTheJudge(t *testing.T) string {
t.Helper()
if _, err := exec.LookPath("go"); err != nil {
t.Skip("no Go toolchain on this machine to build a judge with")
}
bus, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bus.Close() })
go func() {
for {
c, err := bus.Accept()
if err != nil {
return
}
c.Close()
}
}()
dir := t.TempDir()
built := filepath.Join(dir, "built")
runtime := `#!/bin/sh
cmd="$1"; shift
case "$cmd" in
ps|rm|exec) exit 0 ;;
volume)
if [ -n "$VOLUMES" ]; then echo "$*" >> "$VOLUMES"; fi
exit 0 ;;
port) echo "` + bus.Addr().String() + `"; exit 0 ;;
run)
if [ -n "$RUNS" ]; then echo "$*" >> "$RUNS"; fi
workdir=""; detached=""
while [ $# -gt 0 ]; do
case "$1" in
-d) detached=1; shift ;;
--rm) shift ;;
--workdir) workdir="$2"; shift 2 ;;
--env) export "$2"; shift 2 ;;
--network|--volume|--user|--label|--name|-p|-e) shift 2 ;;
*) break ;;
esac
done
if [ -n "$detached" ]; then echo a-container; exit 0; fi
shift
if [ "$1" = go ]; then echo "GOCACHE=$GOCACHE GOENV=$GOENV $*" >> "` + built + `"; fi
if [ "$1" = go ] && [ "$2" = build ] && [ -n "$JUDGE_HOMES" ]; then echo "$HOME" >> "$JUDGE_HOMES"; fi
if [ "$1" = go ] && [ "$2" = test ] && [ -n "$REPLAYS_CANNOT_START" ]; then
echo "docker: Error response from daemon: invalid mount config for type bind" >&2; exit 125
fi
if [ "$1" = go ] && [ "$2" = test ] && [ -n "$REPLAYS" ]; then
echo "HOME=$HOME GOCACHE=$GOCACHE GOMODCACHE=$GOMODCACHE GOENV=$GOENV GOFLAGS=$GOFLAGS DIR=$workdir" >> "$REPLAYS"
fi
if [ "$1" = go ] && [ -n "$WARM_GO_CACHE" ] && [ -n "$GOCACHE" ] && [ -z "$(ls -A "$GOCACHE" 2>/dev/null)" ]; then
mkdir -p "$GOCACHE" && cp -R "$WARM_GO_CACHE/." "$GOCACHE/"
fi
cd "$workdir" && exec "$@" ;;
esac
exit 1
`
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(runtime), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
t.Setenv("GOTOOLCHAIN", "local")
// An empty Go cache the check names is filled from a warm one before `go` runs (warm_go_cache_test.go).
t.Setenv("WARM_GO_CACHE", warmGoCache(t))
return built
}
// builds is how many times the check ran go to build a judge.
func builds(t *testing.T, file string) int {
t.Helper()
body, err := os.ReadFile(file)
if err != nil {
return 0
}
return strings.Count(string(body), "go build")
}
// theReleasedMesh is the core beside a release-branch check: a controller whose main and release branch judge
// differently, a node-engine with a validator of each, and a lab with no release branch.
func theReleasedMesh(t *testing.T) (map[string]Beside, string, string, string) {
t.Helper()
controller, _, controllerRelease := aRepositoryWithARelease(t, aJudgeOf("main's controller"),
aJudgeOf("the release branch's controller"))
host, _, hostRelease := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{"README": "the lab"}, nil)
return map[string]Beside{
"mesh-controller": {Repository: controller, Ref: releaseBranch},
"mesh-controller-main": {Repository: controller, Ref: "main"},
"mesh-host": {Repository: host, Ref: releaseBranch, Else: "main"},
"mesh-lab": {Repository: lab, Ref: releaseBranch, Else: "main"},
}, controllerRelease, hostRelease, labMain
}
// aPullRequestInto is a repository whose pull request's head is cut from base, with a change.
func aPullRequestInto(t *testing.T, files map[string]string, base string, change map[string]string) (string, string) {
t.Helper()
repo, _, _ := aRepositoryWithARelease(t, files, map[string]string{"released": "r"})
gitInRepo(t, repo, "checkout", "--quiet", "-b", "change", base)
head := commitFiles(t, repo, change)
gitInRepo(t, repo, "checkout", "--quiet", "main")
return repo, head
}
func TestACataloguePullRequestIntoAReleaseBranchIsJudgedByThatBranchsController(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, controllerRelease, hostRelease, labMain := theReleasedMesh(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
cache := t.TempDir()
check := func(id string) CheckVerdict {
t.Helper()
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: catalogue, Ref: head, Owner: "novox",
Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch, JudgeCache: cache,
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside, Toolchain: "golang"},
t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
return v
}
v := check("check-704-catalogue")
// The release branch's controller judged, with mesh-host's validator of the release branch.
if v.Gate == nil || v.Gate.Verdict != "pass" ||
!strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the release branch's validator") {
t.Fatalf("the gate was not judged by the release branch's controller with its node-engine's validator: %+v\n%s",
v.Gate, v.Report)
}
// It names the release branch, the judge's commit, and each repository read beside it at its branch and commit.
mark := "[release " + releaseBranch + ", judged by its controller " + controllerRelease[:12] + "] "
if !strings.HasPrefix(v.Gate.Summary, mark) || !strings.HasPrefix(v.Repo.Summary, mark) {
t.Fatalf("the statuses do not name the release branch and its judge first:\n%s\n%s", v.Gate.Summary, v.Repo.Summary)
}
if v.Release != releaseBranch || v.JudgedBy != controllerRelease[:12] {
t.Fatalf("the verdict names release %q judged by %q, not %s by %s", v.Release, v.JudgedBy, releaseBranch,
controllerRelease[:12])
}
if v.BesideAt["mesh-host"] != releaseBranch || v.Beside["mesh-host"] != hostRelease[:12] {
t.Fatalf("mesh-host was read at %q (%s), not its release branch at %s", v.BesideAt["mesh-host"],
v.Beside["mesh-host"], hostRelease[:12])
}
// A repository with no release branch is read at its trunk, and says so.
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] {
t.Fatalf("the lab, with no release branch, was read at %q (%s), not main at %s", v.BesideAt["mesh-lab"],
v.Beside["mesh-lab"], labMain[:12])
}
if !strings.Contains(v.Report, "read beside it: ") || !strings.Contains(v.Report, "mesh-host "+releaseBranch+" at "+hostRelease[:12]) {
t.Fatalf("the report does not say what was read beside it:\n%s", v.Report)
}
// It says which validator judged: mesh-host's of the release branch, with its commit.
if want := "mesh-host's validator of " + releaseBranch + " at " + hostRelease[:12]; v.Validator != want ||
!strings.Contains(v.Report, want) {
t.Fatalf("the verdict says the validator was %q, not %q:\n%s", v.Validator, want, v.Report)
}
// Each repository read beside it is named with the branch and the commit read.
for dir := range beside {
if v.BesideAt[dir] == "" || v.Beside[dir] == "" ||
!strings.Contains(v.Report, dir+" "+v.BesideAt[dir]+" at "+v.Beside[dir]) {
t.Fatalf("%s is not named with its branch and commit: at %q, commit %q\n%s", dir, v.BesideAt[dir],
v.Beside[dir], v.Report)
}
}
if n := builds(t, built); n != 1 {
t.Fatalf("the judge was built %d times for one check", n)
}
// **The judge is built with the judge cache's Go cache** (ADR 0358 §6), which no check's container mounts, and no
// Go environment file.
said, _ := os.ReadFile(built)
if !strings.Contains(string(said), "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build") {
t.Fatalf("the release judge was not built with the judge cache's Go cache:\n%s", said)
}
// **Clause 5**: the head was merged onto the release branch's head, which the verdict names as its trunk commit,
// so the release branch's move puts it out of date as a trunk's move does (ADR 0323 §4, the forge's holder).
if releaseHead := gitInRepo(t, catalogue, "rev-parse", releaseBranch); v.Trunk != releaseHead[:12] {
t.Fatalf("the verdict names trunk %q, not the release branch's head %s", v.Trunk, releaseHead[:12])
}
// **Built once per release-branch head**: a second check while that head stands takes the judge kept.
v = check("check-704-catalogue-again")
if !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the release branch's validator") {
t.Fatalf("the kept judge did not judge: %+v\n%s", v.Gate, v.Report)
}
if n := builds(t, built); n != 1 {
t.Fatalf("the judge of the same release-branch head was built again: %d builds", n)
}
}
func TestAControllerPullRequestIntoAReleaseBranchReadsTheNodeEngineAtThatBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, _, hostRelease, _ := theReleasedMesh(t)
delete(beside, "mesh-controller")
controller, head := aPullRequestInto(t, aJudgeOf("the change's controller"), releaseBranch,
map[string]string{"cmd/x.txt": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-controller", Repository: controller, Ref: head,
Owner: "novox", Repo: "mesh-controller", Number: 315, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"cmd/x.txt"}, Modules: []string{"mesh-controller"}, Judge: "self",
Beside: beside, Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the change's controller with the vendored validator") {
t.Fatalf("a controller change into a release branch did not judge itself: %+v\n%s", v.Gate, v.Report)
}
if v.BesideAt["mesh-host"] != releaseBranch || v.Beside["mesh-host"] != hostRelease[:12] {
t.Fatalf("mesh-host was read beside it at %q (%s), not at the release branch's %s", v.BesideAt["mesh-host"],
v.Beside["mesh-host"], hostRelease[:12])
}
if v.Release != releaseBranch || v.JudgedBy == "" || v.JudgedBy != v.Merge[:12] {
t.Fatalf("the verdict names release %q judged by %q, not its own merge %s", v.Release, v.JudgedBy, v.Merge)
}
}
func TestAPullRequestIntoTheTrunkIsJudgedAsBefore(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("main's controller"),
aJudgeOf("the release branch's controller"))
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-trunk", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: t.TempDir(),
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain},
"mesh-host": {Repository: host, Ref: hostMain}}}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "main's controller with the vendored validator" {
t.Fatalf("a trunk pull request was not judged by the controller beside it as before: %+v\n%s", v.Gate, v.Report)
}
if v.Release != "" || v.JudgedBy != "" || v.BesideAt != nil || strings.Contains(v.Report, "release branch") {
t.Fatalf("a trunk pull request's verdict speaks of a release branch: %q %q %v", v.Release, v.JudgedBy, v.BesideAt)
}
if v.Beside["mesh-host"] != hostMain[:12] {
t.Fatalf("mesh-host was read at %s, not the commit asked %s", v.Beside["mesh-host"], hostMain[:12])
}
}
// **Without mesh-host's release branch, the release branch's controller keeps the validator it vendors** (ADR 0358
// §2), and the verdict says so.
func TestWithoutTheNodeEnginesReleaseBranchTheJudgeKeepsItsVendoredValidator(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, controllerRelease, _, _ := theReleasedMesh(t)
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"), nil)
beside["mesh-host"] = Beside{Repository: host, Ref: releaseBranch, Else: "main"}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-vendored", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the vendored validator") {
t.Fatalf("without mesh-host's release branch the judge did not keep its vendored validator: %+v\n%s", v.Gate, v.Report)
}
if v.JudgedBy != controllerRelease[:12] || v.BesideAt["mesh-host"] != "main" || v.Beside["mesh-host"] != hostMain[:12] {
t.Fatalf("judged by %q, mesh-host read at %q (%s)", v.JudgedBy, v.BesideAt["mesh-host"], v.Beside["mesh-host"])
}
if !strings.Contains(v.Validator, "controller vendors") || !strings.Contains(v.Report, v.Validator) {
t.Fatalf("the verdict does not say the vendored validator judged: %q\n%s", v.Validator, v.Report)
}
}
// **Clause 7** (ADR 0358 §7): a release branch's own pull request into its trunk is checked as any pull request into a
// trunk — by the controller beside it as the mesh runs it, the repositories at what a trunk check reads — and its
// verdict names no release branch.
func TestAReleaseBranchsPullRequestIntoItsTrunkIsJudgedByTheControllerTheMeshRuns(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"),
aJudgeOf("the release branch's controller"))
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
catalogue, _, releaseHead := aRepositoryWithARelease(t, map[string]string{
"modules/gitea/module.json": `{"module":"gitea"}`, CheckScript: "true\n"},
map[string]string{"modules/gitea/index.ts": "released"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-release-into-trunk", Repository: catalogue,
Ref: releaseHead, Owner: "novox", Repo: "mesh-catalog", Number: 300, Base: "main", JudgeCache: t.TempDir(),
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain},
"mesh-host": {Repository: host, Ref: hostMain}}}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "the controller the mesh runs with the vendored validator" {
t.Fatalf("the release branch's pull request into its trunk was not judged by the controller the mesh runs: %+v\n%s",
v.Gate, v.Report)
}
if v.Release != "" || v.JudgedBy != "" || v.BesideAt != nil || v.Validator != "" ||
strings.Contains(v.Report, "release branch") || strings.Contains(v.Gate.Summary, "[release") {
t.Fatalf("its verdict names a release branch: %q %q %v %q", v.Release, v.JudgedBy, v.BesideAt, v.Validator)
}
if v.Beside["mesh-host"] != hostMain[:12] || v.Beside["mesh-controller"] != controllerMain[:12] {
t.Fatalf("beside it were read %v, not the trunk's %s and %s", v.Beside, controllerMain[:12], hostMain[:12])
}
}
// **mesh-lab is read at its trunk, always** (ADR 0358): its replays run with the container runtime's socket, so a
// branch of the release's name in mesh-lab — which anyone may create — is never what a check reads.
func TestAReleaseBranchCheckReadsMeshLabAtItsTrunkEvenWhenItHasThatBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, _, _, _ := theReleasedMesh(t)
lab, labMain, labRelease := aRepositoryWithARelease(t, map[string]string{"README": "the lab"},
map[string]string{"README": "a branch anyone could push"})
beside["mesh-lab"] = Beside{Repository: lab, Ref: releaseBranch, Else: "main"}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-lab", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] || v.Beside["mesh-lab"] == labRelease[:12] {
t.Fatalf("mesh-lab was read at %q (%s), not its trunk main at %s", v.BesideAt["mesh-lab"], v.Beside["mesh-lab"],
labMain[:12])
}
}
// aPlantedWorkspace is a workspace a pull request's own script has written into, as it can: a git configuration in its
// HOME whose fsmonitor hook says each directory git runs in, a git configuration under XDG_CONFIG_HOME doing the same,
// and a Go environment file that would hand the judge's build a tool of its own. It answers the workspace and the
// file the hook writes to.
func aPlantedWorkspace(t *testing.T) (string, string) {
t.Helper()
workspace, logs := t.TempDir(), t.TempDir()
ran := filepath.Join(logs, "hook-ran")
hook := filepath.Join(workspace, "hook.sh")
files := map[string]string{
hook: "#!/bin/sh\necho \"$PWD\" >> " + ran + "\n",
filepath.Join(workspace, ".gitconfig"): "[core]\n\tfsmonitor = " + hook + "\n",
filepath.Join(workspace, ".config", "git", "config"): "[core]\n\tfsmonitor = " + hook + "\n",
filepath.Join(workspace, ".config", "go", "env"): "GOFLAGS=-toolexec=" + hook + "\n",
}
for name, body := range files {
if err := os.MkdirAll(filepath.Dir(name), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(name, []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
return workspace, ran
}
// judgeRanTheHook says whether the planted hook ran in a judge's tree: the controller beside the check, or the
// change's own when it judges itself.
func judgeRanTheHook(t *testing.T, ran string) string {
t.Helper()
body, _ := os.ReadFile(ran)
for _, line := range strings.Split(string(body), "\n") {
if strings.HasSuffix(line, "/check/mesh-controller") {
return string(body)
}
}
return ""
}
// **Every judge is built where no pull request reaches** (issue 482, ADR 0358 review): a git configuration with a
// hook, and a Go environment file, planted in the workspace by an earlier pull request's script, do not run during
// the release judge's build — while the pull request's own script, in the workspace, still meets them.
func TestAHookPlantedInTheWorkspaceDoesNotRunDuringTheReleaseJudgesBuild(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
homes := filepath.Join(t.TempDir(), "homes")
t.Setenv("JUDGE_HOMES", homes)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
beside, _, _, _ := theReleasedMesh(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "git status >/dev/null 2>&1; true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-planted", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" {
t.Fatalf("the check did not judge: %+v\n%s", v.Gate, v.Report)
}
if body, _ := os.ReadFile(ran); !strings.Contains(string(body), "/check/mesh-catalog") {
t.Fatalf("the planted hook never ran, so this test proves nothing: %q", body)
}
if said := judgeRanTheHook(t, ran); said != "" {
t.Fatalf("the hook planted in the workspace ran during the judge's build:\n%s", said)
}
if body, _ := os.ReadFile(homes); len(body) == 0 || strings.Contains(string(body), workspace) {
t.Fatalf("the judge was built with a HOME in the workspace: %q", body)
}
}
// The same for the judge built with a node-engine change's own validator, into a release branch (ADR 0358 §2).
func TestAHookPlantedInTheWorkspaceDoesNotRunDuringTheValidatorJudgesBuild(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
homes := filepath.Join(t.TempDir(), "homes")
t.Setenv("JUDGE_HOMES", homes)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
beside, _, _, _ := theReleasedMesh(t)
delete(beside, "mesh-host")
files := aHostSaying("the release branch's validator")
files["module.json"] = `{"module":"mesh-host"}`
files[CheckScript] = "git status >/dev/null 2>&1; true\n"
host, head := aPullRequestInto(t, files, releaseBranch, aHostSaying("this change's validator"))
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-planted-validator", Repository: host, Ref: head,
Owner: "novox", Repo: "mesh-host", Number: 111, Base: releaseBranch, Release: releaseBranch, Judge: "validator",
JudgeCache: t.TempDir(), Paths: []string{"validate/v.go"}, Modules: []string{"mesh-host"}, Beside: beside,
Toolchain: "golang"}, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with this change's validator") {
t.Fatalf("a node-engine change into a release branch was not judged by that branch's controller with its own "+
"validator: %+v\n%s", v.Gate, v.Report)
}
if body, _ := os.ReadFile(ran); !strings.Contains(string(body), "/check/mesh-host") {
t.Fatalf("the planted hook never ran, so this test proves nothing: %q", body)
}
if said := judgeRanTheHook(t, ran); said != "" {
t.Fatalf("the hook planted in the workspace ran during a judge's build:\n%s", said)
}
if body, _ := os.ReadFile(homes); strings.Count(string(body), "\n") != 2 || strings.Contains(string(body), workspace) {
t.Fatalf("the judges were not both built with a HOME outside the workspace: %q", body)
}
}
// **mesh-lab is never read at a release branch**, asked with a fallback or without (ADR 0358).
func TestTheBuildSeatNeverReadsMeshLabAtAReleaseBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{"README": "the lab"},
map[string]string{"README": "a branch anyone could push"})
for _, asked := range []Beside{{Repository: lab, Ref: releaseBranch}, {Repository: lab, Ref: releaseBranch, Else: "main"}} {
beside, _, _, _ := theReleasedMesh(t)
beside["mesh-lab"] = asked
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-lab-asked", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] {
t.Fatalf("mesh-lab asked as %+v was read at %q (%s), not main at %s", asked, v.BesideAt["mesh-lab"],
v.Beside["mesh-lab"], labMain[:12])
}
}
}
// **Judges are built warm and where no pull request reaches** (ADR 0358 §6): every judge build uses the judge cache's
// one Go cache, kept between checks; its home is made fresh under the judge cache and removed after; nothing is left in
// a temporary directory of the host; and no container a check runs for the pull request mounts the judge cache.
func TestJudgesAreBuiltWarmFromAJudgeCacheNoCheckContainerMounts(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
// Not filled from the test's warm cache: what the first judge's build leaves in the judge cache is what is proved.
t.Setenv("WARM_GO_CACHE", "")
runs := filepath.Join(t.TempDir(), "runs")
t.Setenv("RUNS", runs)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
registry := aFactsRegistry(t)
cache := t.TempDir()
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
for i := range 2 {
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": fmt.Sprint(i)})
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-704-warm-%d", i), Repository: catalogue,
Ref: head, Owner: "novox", Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache,
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang", Beside: beside},
t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "the controller the mesh runs with the vendored validator" {
t.Fatalf("check %d was not judged: %+v\n%s", i, v.Gate, v.Report)
}
if i == 0 {
if entries, _ := os.ReadDir(filepath.Join(cache, "go-cache")); len(entries) == 0 {
t.Fatal("the judge's build left nothing in the judge cache's Go cache to build the next judge warm")
}
}
}
// Both judges built with the one Go cache of the judge cache.
said, _ := os.ReadFile(built)
if n := strings.Count(string(said), "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build"); n != 2 {
t.Fatalf("%d of 2 judge builds used the judge cache's Go cache:\n%s", n, said)
}
// Nothing left behind: no home under the judge cache, nothing in the host's temporary directory.
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
t.Fatalf("judge builds left %d home(s) under the judge cache", len(left))
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory, e.g. %s", len(left), left[0].Name())
}
// Only a judge build mounts the judge cache: no container the check runs for the pull request does.
body, _ := os.ReadFile(runs)
judgeBuilds := 0
for _, line := range strings.Split(strings.TrimSpace(string(body)), "\n") {
judging := strings.Contains(line, " go build ")
if judging {
judgeBuilds++
if !strings.Contains(line, "--volume "+filepath.Join(cache, "go-cache")+":") {
t.Fatalf("a judge build did not mount the judge cache's Go cache: %s", line)
}
continue
}
if strings.Contains(line, cache) {
t.Fatalf("a container the check runs for the pull request names the judge cache: %s", line)
}
}
if judgeBuilds != 2 {
t.Fatalf("%d judge builds were run, not 2", judgeBuilds)
}
}
// Without a judge cache, a judge is built cold under the check's own root — never in a temporary directory of the
// host — and a judge cache inside the workspace is refused.
func TestWithoutAJudgeCacheAJudgeIsBuiltColdUnderTheChecksRoot(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
workspace := t.TempDir()
spec := CheckSpec{ID: "check-704-cold", Repository: catalogue, Ref: head, Owner: "novox", Repo: "mesh-catalog",
Number: 1, Base: "main", Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"},
Toolchain: "golang", Beside: beside}
v, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" {
t.Fatalf("the check was not judged: %+v\n%s", v.Gate, v.Report)
}
said, _ := os.ReadFile(built)
if !strings.Contains(string(said), "GOCACHE="+filepath.Join(workspace, "check", "judge-go-cache")+" GOENV=off go build") {
t.Fatalf("without a judge cache the judge was not built under the check's root:\n%s", said)
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
}
spec.JudgeCache = filepath.Join(workspace, "judge-cache")
if _, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil); err == nil ||
!strings.Contains(err.Error(), "inside the workspace") {
t.Fatalf("a judge cache inside the workspace was taken: %v", err)
}
}
// aLabWithReplays is mesh-lab with replays: a Go module of its own whose one test passes, needing nothing fetched.
func aLabWithReplays(t *testing.T) (string, string) {
t.Helper()
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{
"replays/go.mod": "module example.org/replays\n\ngo 1.22\n",
"replays/replay_test.go": "package replays\n\nimport \"testing\"\n\nfunc TestAReplay(t *testing.T) {}\n",
}, nil)
return lab, labMain
}
// **mesh-lab's replays run where no pull request reaches** (novox/hq issue 709, ADR 0358 §6): they hold the container
// runtime's socket, so a Go environment file with a -toolexec hook and a Go cache and module cache planted in the
// workspace by a pull request's script are neither run nor read — the replays run from a copy of mesh-lab outside
// the workspace with a fresh home and the judge cache's caches, all removed or kept outside every check container.
func TestTheReplaysRunWhereNoPullRequestReaches(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
replays := filepath.Join(t.TempDir(), "replays")
t.Setenv("REPLAYS", replays)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
for _, poisoned := range []string{filepath.Join(workspace, "go-cache", "README"),
filepath.Join(workspace, "go-modules", "cache", "download", "planted")} {
if err := os.MkdirAll(filepath.Dir(poisoned), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(poisoned, []byte("planted by a pull request"), 0o644); err != nil {
t.Fatal(err)
}
}
cache := t.TempDir()
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
lab, labMain := aLabWithReplays(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709", Repository: catalogue, Ref: head, Owner: "novox",
Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache, Paths: []string{"modules/gitea/index.ts"},
Modules: []string{"gitea"}, Toolchain: "golang", Beside: map[string]Beside{
"mesh-controller": {Repository: controller, Ref: controllerMain}, "mesh-lab": {Repository: lab, Ref: labMain}}},
workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
}
body, _ := os.ReadFile(replays)
said := strings.TrimSpace(string(body))
if said == "" || strings.Count(said, "\n") != 0 {
t.Fatalf("the replays ran %q, not once", said)
}
for _, want := range []string{"GOCACHE=" + filepath.Join(cache, "go-cache") + " ",
"GOMODCACHE=" + filepath.Join(cache, "go-modules") + " ", "GOENV=off ", "GOFLAGS=-buildvcs=false ",
"HOME=" + filepath.Join(cache, "homes", "check-709-replays", "home") + " ",
"DIR=" + filepath.Join(cache, "homes", "check-709-replays", "mesh-lab", "replays")} {
if !strings.Contains(said, want) {
t.Fatalf("the replays did not run with %q: %s", want, said)
}
}
if strings.Contains(said, workspace) {
t.Fatalf("the replays ran with something of the workspace: %s", said)
}
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
}
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
t.Fatalf("the replays left %d entr(ies) under the judge cache's homes", len(left))
}
}
// aSelfJudgingControllerThatWritesIntoTheReplays is a controller pull request whose own judge, while the gate runs it,
// writes a test into the replays beside the check — the reviewer's case for issue 709: code of the pull request's,
// run before the replays, reaching what they run.
func aSelfJudgingControllerThatWritesIntoTheReplays(t *testing.T, marker string) (string, string) {
t.Helper()
files := aJudgeOf("the change's controller")
evil := "package replays\n\nimport (\n\t\"os\"\n\t\"testing\"\n)\n\nfunc TestEvil(t *testing.T) {\n\t_ = os.WriteFile(" +
fmt.Sprintf("%q", marker) + ", []byte(\"ran\"), 0o644)\n}\n"
files["cmd/mesh-controller/main.go"] = strings.Replace(files["cmd/mesh-controller/main.go"], "func main() {",
"func main() {\n\tif len(os.Args) > 2 && os.Args[1] == \"merge-gate\" {\n\t\t_ = os.WriteFile(os.Getenv(\"MESH_CHECK_BESIDE\")+"+
"\"/mesh-lab/replays/evil_test.go\", []byte("+fmt.Sprintf("%q", evil)+"), 0o644)\n\t\t_ = os.WriteFile("+
fmt.Sprintf("%q", marker+".wrote")+", nil, 0o644)\n\t}", 1)
return aPullRequestInto(t, files, "main", map[string]string{"cmd/x.txt": "x"})
}
// **The replays never read what a pull request wrote** (issue 709, reviews 12303 and 12308): with a judge cache they
// run from a copy under it, and a self-judging controller pull request writing a test into the replays beside the
// check, a planted GOFLAGS hook and a poisoned cache entry in the workspace neither run nor are read. Without one the
// replays are refused — the gate an error that says why, never a pass, never the workspace — they never start, and
// nothing is made on the host.
func TestTheReplaysNeverReadWhatAPullRequestWroteAndNeedAJudgeCache(t *testing.T) {
for _, withCache := range []bool{true, false} {
t.Run(fmt.Sprintf("judge cache %v", withCache), func(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
replays, volumes := filepath.Join(t.TempDir(), "replays"), filepath.Join(t.TempDir(), "volumes")
t.Setenv("REPLAYS", replays)
t.Setenv("VOLUMES", volumes)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
if err := os.MkdirAll(filepath.Join(workspace, "go-cache"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(workspace, "go-cache", "README"), []byte("planted"), 0o644); err != nil {
t.Fatal(err)
}
evilRan := filepath.Join(t.TempDir(), "evil-ran")
controller, head := aSelfJudgingControllerThatWritesIntoTheReplays(t, evilRan)
lab, labMain := aLabWithReplays(t)
cache := ""
if withCache {
cache = t.TempDir()
}
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709-self", Repository: controller, Ref: head,
Owner: "novox", Repo: "mesh-controller", Number: 330, Base: "main", Judge: "self", JudgeCache: cache,
Paths: []string{"cmd/x.txt"}, Modules: []string{"mesh-controller"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-lab": {Repository: lab, Ref: labMain}}}, workspace, registry,
GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(evilRan + ".wrote"); err != nil {
t.Fatalf("the self-judging pull request did not write into the replays beside it, so this proves nothing: %v", err)
}
if _, err := os.Stat(evilRan); err == nil {
t.Fatal("the test the pull request wrote into the replays ran with the socket")
}
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
}
body, _ := os.ReadFile(replays)
said := strings.TrimSpace(string(body))
if vols, _ := os.ReadFile(volumes); len(vols) != 0 {
t.Fatalf("the check made a container volume: %q", vols)
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
}
if !withCache {
if v.Gate == nil || v.Gate.Verdict != "error" || !strings.Contains(v.Gate.Summary, replaysNeedTheJudgeCache) ||
v.Verdict != "error" {
t.Fatalf("without a judge cache the gate said %+v, not an error saying the replays need it", v.Gate)
}
if said != "" {
t.Fatalf("without a judge cache the replays started: %s", said)
}
return
}
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
}
base := filepath.Join(cache, "homes", "check-709-self-replays")
if said == "" || strings.Contains(said, workspace) || !strings.Contains(said, "GOENV=off ") ||
!strings.Contains(said, "DIR="+filepath.Join(base, "mesh-lab", "replays")) {
t.Fatalf("the replays did not run from the copy under %s, apart from the workspace: %q", base, said)
}
if _, err := os.Stat(base); err == nil {
t.Fatalf("the replays' place %s was left after the check", base)
}
})
}
}
// **Replays the container runtime could not start are the check's error, never the change's failure**, said in the
// build's log (review 12308); and a check whose id cannot name a place in the judge cache is refused, never given a
// shared one.
func TestReplaysThatCannotStartAreAnErrorAndAnUnsafeIDIsRefused(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
t.Setenv("REPLAYS_CANNOT_START", "1")
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
lab, labMain := aLabWithReplays(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
spec := CheckSpec{ID: "check-709-start", Repository: catalogue, Ref: head, Owner: "novox", Repo: "mesh-catalog",
Number: 1, Base: "main", JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"},
Modules: []string{"gitea"}, Toolchain: "golang", Beside: map[string]Beside{
"mesh-controller": {Repository: controller, Ref: controllerMain}, "mesh-lab": {Repository: lab, Ref: labMain}}}
var logged []string
v, err := Check(t.Context(), Command, spec, t.TempDir(), registry, GitCredential{}, func(step, message string) {
logged = append(logged, message)
})
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "error" || !strings.Contains(v.Gate.Summary, "could not be started") ||
!strings.Contains(v.Gate.Summary, "invalid mount config") {
t.Fatalf("replays that could not start were said %+v, not an error naming why", v.Gate)
}
if !strings.Contains(strings.Join(logged, "\n"), "the replays could not be started") {
t.Fatalf("the build's log does not say what failed:\n%s", strings.Join(logged, "\n"))
}
spec.ID = "Check 1/../../x"
if _, err := Check(t.Context(), Command, spec, t.TempDir(), registry, GitCredential{}, nil); err == nil ||
!strings.Contains(err.Error(), "cannot name its place in the judge cache") {
t.Fatalf("a check with an unsafe id was taken: %v", err)
}
}
+125
View File
@@ -0,0 +1,125 @@
package builder
import (
"context"
"errors"
"path/filepath"
"strings"
"testing"
)
// novox/hq issue 713: after ADR 0350 the walk of novox/mesh-controller#331 built mesh-controller, build-agent and
// route-proxy again, each "built: no check of novox/mesh-controller#331 built it". The check had built none: its
// build log said "built for the merge: 0 of 3 module(s)", mesh-controller's go bundle failing with "go.mod file not
// found". The build seat's holder runs in a container sharing only its workspace with the host, and a build's
// toolchain container is run by the host's container engine, which mounts a path as the host names it. The check
// built in a temporary directory of the holder's own (issue 703's copy), which the engine mounted as an empty one.
// aHostsEngine runs as the host's container engine does for a holder in a container: a mount of a path outside the
// workspace the holder shares with the host is an empty directory, and a Go toolchain finds no go.mod in it.
type aHostsEngine struct {
onTrunk
shared string
}
func (h aHostsEngine) run(ctx context.Context, dir, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "run" {
for i, a := range args {
if a != "--volume" || i+1 >= len(args) {
continue
}
from, _, _ := strings.Cut(args[i+1], ":")
if !withinDir(h.shared, from) {
return "", errors.New("go: go.mod file not found in current directory or any parent directory")
}
}
}
return h.onTrunk.run(ctx, dir, name, args...)
}
// aGoModule is a module of one Go bundle built from a repository's root.
func aGoModule(module string) map[string]string {
files := aSharedRepository("one", "1")
files[ManifestName] = `{"module":"` + module + `","version":"1","build":{"artifacts":[
{"name":"server","kind":"bundle","language":"go","system":"arch","from":"cmd/ctl","binary":"ctl"}]},
"resources":[{"id":"server","type":"process","name":"` + module + `","artifact":"server","run":["./ctl"]}]}`
return files
}
// A controller change whose walk moves mesh-controller, build-agent and route-proxy (#331's) has its check build all
// three, in the workspace the host's engine sees, and the walk of its merge takes each from the check by its key.
func TestReplay713TheWalkOfAControllerChangeTakesAllThreeFromItsCheck(t *testing.T) {
withVersion(t, "builder-1")
pull := PullRef{Owner: "novox", Repo: "mesh-controller", Number: 331}
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
repositories := map[string]string{"build-agent": "https://forge.invalid/mesh-catalog-build-agent.git",
"route-proxy": "https://forge.invalid/mesh-catalog-route-proxy.git"}
checked := func(shared string) (*recorded, aHostsEngine) {
r := &recorded{tree: "tree-of-the-merge", contents: aGoModule("mesh-controller"),
secondary: map[string]map[string]string{}}
for module, repository := range repositories {
r.secondary[repository] = aGoModule(module)
}
return r, aHostsEngine{onTrunk: onTrunk{recorded: r}, shared: shared}
}
spec := CheckSpec{ID: "build-331", Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number,
Builds: []CheckBuildSpec{{Module: "mesh-controller", Merged: true, Tier: 0},
{Module: "build-agent", Repository: repositories["build-agent"], Ref: "main", Tier: 1},
{Module: "route-proxy", Repository: repositories["route-proxy"], Ref: "main", Tier: 2}}}
judged := CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053", Merge: "2d22279b1d07"}
passed := CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "pass"}}
// As it ran: the builds made in the holder's own temporary directory, which the host's engine does not see.
shared, outside := t.TempDir(), t.TempDir()
merged := filepath.Join(outside, "merged")
r, engine := checked(shared)
asItRan := buildForTheMerge(t.Context(), engine.run, spec, CheckBuilding{Publish: r, Scratch: newScratch(), Held: held},
outside, merged, judged.Merge, judged, GitCredential{}, nil, logging(nil))
if builtCount(asItRan) != 0 {
t.Fatalf("the engine of this test sees the holder's temporary directory: %+v", asItRan)
}
// The check as it builds now: every module its merge moves, kept under its key.
scratch := newScratch()
spec.BuildWith = &CheckBuilding{Publish: r, Scratch: scratch, Held: held}
built := forTheMerge(t.Context(), engine.run, spec, passed, shared, merged, judged, GitCredential{}, nil,
logging(nil))
if len(built) != 3 || builtCount(built) != 3 {
t.Fatalf("the check built %d of %d module(s) its merge moves: %+v", builtCount(built), len(built), built)
}
var names []NamedBuild
for _, b := range built {
n := NamedBuild{Pull: pull, Check: spec.ID, Module: b.Module, Key: b.Key, Digests: map[string]string{}}
for _, m := range b.Made {
d := m.Digest
if d == "" {
d = DigestOf(m.Reference)
}
n.Digests[m.Name] = d
}
names = append(names, n)
}
// The walk of the merge commit, tier by tier: each module taken from the check, none built.
walked := map[string]string{"mesh-controller": "https://forge.invalid/mesh-controller.git"}
for module, repository := range repositories {
walked[module] = repository
}
for _, module := range []string{"mesh-controller", "build-agent", "route-proxy"} {
w, walk := checked(shared)
w.secondary[walked["mesh-controller"]] = aGoModule("mesh-controller")
got, err := BuildReusing(t.Context(), walk.run, w, Reuse{Scratch: scratch, Pulls: []PullRef{pull}, Named: names},
walked[module], "", "main", t.TempDir(), held, Npmrc{}, GitCredential{}, nil)
if err != nil {
t.Fatalf("%s: %v", module, err)
}
if got.Reused == nil || got.Why != "" {
t.Fatalf("the walk built %s again: %s", module, got.Why)
}
for _, line := range w.ran {
if strings.HasPrefix(line, "docker run") && strings.Contains(line, "go build") {
t.Fatalf("the walk compiled %s: %s", module, line)
}
}
}
}
+158
View File
@@ -0,0 +1,158 @@
package builder
import (
"context"
"errors"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
// novox/hq issue 716: a check built the images its merge moves one after another, and only after its merge-check.sh
// had passed — 3 to 6 minutes on a check of about 5. The modules of a tier stand on none of each other and are built
// at the same time; with a judge cache, beside the tests; a check that does not pass lets go of them.
// atOnce is a fake engine whose compiles wait until every one of a tier's has started: builds one after another never
// get there, and the first one's compile says so.
type atOnce struct {
mu sync.Mutex
r *recorded
want int
started int
all chan struct{}
shared string
}
func (a *atOnce) run(ctx context.Context, dir, name string, args ...string) (string, error) {
if name == "docker" && len(args) > 0 && args[0] == "run" && strings.Contains(strings.Join(args, " "), "go build") {
a.mu.Lock()
a.started++
if a.started == a.want {
close(a.all)
}
a.mu.Unlock()
select {
case <-a.all:
case <-time.After(5 * time.Second):
return "", errors.New("built alone: no other module of its tier was being built")
}
}
a.mu.Lock()
defer a.mu.Unlock()
return onTrunk{recorded: a.r}.run(ctx, dir, name, args...)
}
func (a *atOnce) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
a.mu.Lock()
defer a.mu.Unlock()
return a.r.PublishImage(ctx, localTag, repository)
}
func (a *atOnce) PublishArchive(ctx context.Context, repository string, body []byte, digest string) (string, error) {
a.mu.Lock()
defer a.mu.Unlock()
return a.r.PublishArchive(ctx, repository, body, digest)
}
// aLockedScratch is the scratch namespace in memory, said to by builds at once.
type aLockedScratch struct {
mu sync.Mutex
*aScratch
discarded []CheckBuild
}
func (s *aLockedScratch) Keep(ctx context.Context, p PullRef, c string, at time.Time, b []CheckBuild) error {
s.mu.Lock()
defer s.mu.Unlock()
return s.aScratch.Keep(ctx, p, c, at, b)
}
func (s *aLockedScratch) Find(ctx context.Context, module, key string) (CheckBuild, bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
return s.aScratch.Find(ctx, module, key)
}
func (s *aLockedScratch) Discard(_ context.Context, b []CheckBuild) error {
s.mu.Lock()
defer s.mu.Unlock()
s.discarded = append(s.discarded, b...)
return nil
}
// threeAtOnce are three Go modules in one tier, each from its own repository.
func threeAtOnce(t *testing.T) (*atOnce, CheckSpec, map[string]string) {
t.Helper()
r := &recorded{tree: "tree-of-the-merge", contents: aGoModule("mesh-controller"),
secondary: map[string]map[string]string{}}
spec := CheckSpec{ID: "build-716", Owner: "novox", Repo: "mesh-controller", Number: 716,
Builds: []CheckBuildSpec{{Module: "mesh-controller", Merged: true, Tier: 0}}}
for _, module := range []string{"build-agent", "route-proxy"} {
repository := "https://forge.invalid/" + module + ".git"
r.secondary[repository] = aGoModule(module)
spec.Builds = append(spec.Builds, CheckBuildSpec{Module: module, Repository: repository, Ref: "main", Tier: 0})
}
held := map[string]string{"mesh-tools-go/build": "registry.invalid/mesh-tools-go/build@sha256:" + strings.Repeat("b", 64)}
return &atOnce{r: r, want: 3, all: make(chan struct{})}, spec, held
}
// The modules of one tier are built at the same time, each in a working copy of its own.
func TestReplay716TheModulesOfATierAreBuiltAtTheSameTime(t *testing.T) {
withVersion(t, "builder-1")
engine, spec, held := threeAtOnce(t)
scratch := &aLockedScratch{aScratch: newScratch()}
built := buildForTheMerge(t.Context(), engine.run, spec, CheckBuilding{Publish: engine, Scratch: scratch, Held: held},
t.TempDir(), filepath.Join(t.TempDir(), "merged"), "2d22279b1d07",
CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053"}, GitCredential{}, nil, logging(nil))
if builtCount(built) != 3 {
t.Fatalf("the check built %d of 3 module(s) of one tier: %+v", builtCount(built), built)
}
if len(scratch.builds) != 3 {
t.Fatalf("the check kept %d build(s), not 3", len(scratch.builds))
}
}
// With a judge cache, the builds run while the tests run, where no container of the check reaches; a check that passed
// keeps them, and one that failed lets go of every one and names none.
func TestReplay716TheBuildsRunBesideTheTestsAndAFailedCheckLetsGoOfThem(t *testing.T) {
withVersion(t, "builder-1")
workspace := t.TempDir()
if _, err := buildsBesideTheTests(filepath.Join(workspace, "judge-cache"), workspace, "build-716"); err == nil {
t.Fatal("a judge cache inside the workspace was built in, where a pull request's script reaches")
}
for _, passed := range []bool{true, false} {
cache := t.TempDir()
place, err := buildsBesideTheTests(cache, workspace, "build-716")
if err != nil || !withinDir(cache, place) || withinDir(workspace, place) {
t.Fatalf("the builds' place %q (%v) is not the judge cache's, outside the workspace", place, err)
}
engine, spec, held := threeAtOnce(t)
scratch := &aLockedScratch{aScratch: newScratch()}
spec.BuildWith = &CheckBuilding{Publish: engine, Scratch: scratch, Held: held}
judged := CheckVerdict{Head: "98e856874913", Trunk: "c975dbf7b053", Merge: "2d22279b1d07"}
beside := startBesideTheTests(t.Context(), engine.run, spec, place, filepath.Join(t.TempDir(), "merged"),
judged, GitCredential{}, nil, logging(nil))
// The tests: the builds are under way before the check has a verdict.
select {
case <-engine.all:
case <-time.After(5 * time.Second):
t.Fatal("nothing was built while the tests ran")
}
verdict := CheckVerdict{Gate: &Layer{Verdict: "pass"}, Repo: &Layer{Verdict: "pass"}}
if !passed {
verdict.Repo.Verdict = "fail"
}
built := beside.settle(t.Context(), verdict)
beside.abandon()
switch {
case passed && (builtCount(built) != 3 || len(scratch.builds) != 3 || len(scratch.discarded) != 0):
t.Fatalf("a check that passed: built %+v, kept %d, let go of %d", built, len(scratch.builds),
len(scratch.discarded))
case !passed && (built != nil || len(scratch.builds) != 0 || len(scratch.discarded) != 3):
t.Fatalf("a check that failed: named %+v, kept %d, let go of %d", built, len(scratch.builds),
len(scratch.discarded))
}
}
}
+19 -4
View File
@@ -257,6 +257,13 @@ func (s scratchPublisher) PublishArchive(ctx context.Context, repository string,
return s.to.PublishArchive(ctx, ScratchPrefix+repository, body, digest)
}
func (s scratchPublisher) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
if l, can := s.to.(BaseLender); can {
return l.LendBases(ctx, ScratchPrefix+repository, bases)
}
return 0, nil
}
func (s scratchPublisher) MirrorImage(ctx context.Context, from, repository string) (string, error) {
m, can := s.to.(Mirrorer)
if !can {
@@ -487,11 +494,15 @@ func (r Registry) Discard(ctx context.Context, builds []CheckBuild) error {
func (r Registry) letGo(ctx context.Context, checks, all map[string]checkIndex) (int, error) {
store := r.store()
stays := map[string]bool{}
// recorded are the records under a key another check still names (novox/hq issue 703): a later check that kept
// the build names it as the check's that built it, and the walk finds it by that record.
recorded := map[string]bool{}
for tag, c := range all {
if _, going := checks[tag]; going {
continue
}
for _, b := range c.Builds {
recorded[b.Module+" "+b.Key+" "+b.Check] = true
for _, m := range b.Made {
stays[catalogue.Recorded(m.Reference)] = true
}
@@ -516,8 +527,10 @@ func (r Registry) letGo(ctx context.Context, checks, all map[string]checkIndex)
}
}
// The record under the key, only while it is still this check's: a later check of the same key
// replaced it, and that one is its own check's to let go of.
if kept, found, err := r.Find(ctx, b.Module, b.Key); err == nil && found && kept.Check == c.Check {
// replaced it, and that one is its own check's to let go of. A build a check kept is that build's
// check's record, and goes with the last check naming it (novox/hq issue 703).
if kept, found, err := r.Find(ctx, b.Module, b.Key); err == nil && found &&
(kept.Check == c.Check || kept.Check == b.Check) && !recorded[b.Module+" "+b.Key+" "+kept.Check] {
if err := store.DeleteTagged(ctx, ScratchPrefix+b.Module, tagOf(b.Key)); err != nil && !errors.Is(err, artifacts.Gone) {
ok = false
failed = append(failed, fmt.Sprintf("%s's record: %v", b.Module, err))
@@ -603,6 +616,8 @@ func unnamed(named []NamedBuild, module, key string, found CheckBuild) string {
return ""
}
}
return "a build is kept under this key, and no check verdict of the walk's pull requests names it with its " +
"digests: not taken"
// A build of another pull request, or of a head the forge did not merge, is in no verdict the walk is handed
// (novox/hq issue 703): said with the check that made it, whose head names which head it was.
return "a build is kept under this key, by " + found.Said() + ", and no check verdict of the walk's pull requests " +
"at the head the forge merged names it with its digests: not taken"
}
+153 -1
View File
@@ -6,6 +6,7 @@ import (
"errors"
"net/http"
"net/http/httptest"
"reflect"
"slices"
"strconv"
"strings"
@@ -215,7 +216,8 @@ func TestARecordNoVerdictNamesIsRefusedAndBuilt(t *testing.T) {
if got.Reused != nil || len(scratch.took) > 0 || !dockerBuilt(r) {
t.Fatalf("a record no verdict names was taken: %+v, copied %v", got.Reused, scratch.took)
}
if !strings.Contains(got.Why, "no check verdict of the walk's pull requests names it") {
if !strings.Contains(got.Why, "no check verdict of the walk's pull requests at the head the forge merged names it") ||
!strings.Contains(got.Why, "at head abcdefabcdef") {
t.Fatalf("the walk says %q", got.Why)
}
})
@@ -625,3 +627,153 @@ func TestABuildTwoChecksShareGoesWithTheLastOfThem(t *testing.T) {
t.Fatal("the build stayed after the last check naming it went")
}
}
// aCheckAgain is a later check (build-2) of aPull building the module from a tree, with the builds the build seat's
// verdicts name for it (novox/hq issue 703).
func aCheckAgain(t *testing.T, scratch *aScratch, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
t.Helper()
return aCheckOf(t, scratch, aPull, "abcdefabcdef", tree, leave)
}
// aCheckOf is a later check (build-2) of a pull request at a head, the trunk moved since build-1.
func aCheckOf(t *testing.T, scratch *aScratch, pull PullRef, head, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
t.Helper()
r, run, workspace := aModuleAt(t, tree)
spec := CheckSpec{ID: "build-2", Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number,
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
Ref: "main"}}}
built := buildForTheMerge(t.Context(), run, spec, CheckBuilding{Publish: r, Scratch: scratch, Named: leave},
workspace, "", "", CheckVerdict{Head: head, Trunk: "654321654321"}, GitCredential{}, nil, logging(nil))
if len(built) != 1 || built[0].NotBuilt != "" || built[0].Key == "" {
t.Fatalf("the second check did not answer the module for its merge: %+v", built)
}
return built[0], r
}
// A check asked again when only the trunk moved, whose key is unchanged, builds nothing: it keeps the build an
// earlier verdict names and names it by key and digest, so the walk can take it (novox/hq issue 703).
func TestASecondCheckWhoseKeyIsUnchangedBuildsNothingAndNamesTheEarlierBuild(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
again, r := aCheckAgain(t, scratch, "tree-one", named(first))
if dockerBuilt(r) || len(r.images) > 0 || len(r.archives) > 0 {
t.Fatalf("the second check built again: ran %v, published %v %v", r.ran, r.images, r.archives)
}
if again.Key != first.Key || again.Kept != "build-1" || !reflect.DeepEqual(again.Made, first.Made) {
t.Fatalf("the second check names %+v, not the first check's build %+v", again, first)
}
// The walk takes it by what the second verdict names.
leave := named(again)
leave[0].Check = again.Kept
w, run, workspace := aModuleAt(t, "tree-one")
got, err := BuildReusing(t.Context(), run, w, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: leave},
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil || got.Reused == nil || dockerBuilt(w) {
t.Fatalf("the walk did not take the kept build: %v, why %q", err, got.Why)
}
}
// A check whose key changed builds, as before.
func TestASecondCheckWhoseKeyChangedBuilds(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
again, r := aCheckAgain(t, scratch, "tree-two", named(first))
if !dockerBuilt(r) || again.Kept != "" || again.Key == first.Key {
t.Fatalf("a check whose key changed did not build: %+v", again)
}
}
// A scratch record no verdict names is never kept: the store takes writes from anything on its network, so a
// check builds rather than name a planted record (novox/hq issue 703, ADR 0350 §2).
func TestASecondCheckKeepsNoRecordAVerdictDoesNotName(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
for name, leave := range map[string][]NamedBuild{
"no verdict": nil,
"another digest": func() []NamedBuild {
n := named(first)
n[0].Digests["server"] = "sha256:" + strings.Repeat("e", 64)
return n
}(),
"another key": func() []NamedBuild { n := named(first); n[0].Key = "r1:" + strings.Repeat("f", 64); return n }(),
} {
t.Run(name, func(t *testing.T) {
again, r := aCheckAgain(t, scratch, "tree-one", leave)
if !dockerBuilt(r) || again.Kept != "" {
t.Fatalf("a record no verdict names was kept: %+v", again)
}
})
}
}
// A build a later check kept stays when the check that built it goes: its artifacts and its record, which the later
// check's verdict names (novox/hq issue 703, ADR 0350 §5).
func TestABuildALaterCheckKeptStaysWhenTheCheckThatBuiltItGoes(t *testing.T) {
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
server := httptest.NewServer(m.serve())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
registry := Registry{Address: address}
ctx := t.Context()
now := time.Date(2026, 10, 14, 9, 0, 0, 0, time.UTC)
body := `{"mediaType":"` + mediaManifestOCI + `","layers":[]}`
digest := digestOf([]byte(body))
m.manifests["scratch/meshboard/server@"+digest] = []byte(body)
key := "r1:" + strings.Repeat("1", 64)
b := CheckBuild{Key: key, Module: "meshboard", Pull: aPull, Check: "build-1", At: now,
Made: []catalogue.Built{{Name: "server", Kind: "image", Reference: address + "/scratch/meshboard/server@" + digest}}}
if err := registry.Keep(ctx, aPull, "build-1", now, []CheckBuild{b}); err != nil {
t.Fatal(err)
}
// Another pull request's check keeps it: its own document names the build, which stays the first check's.
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
if err := registry.Keep(ctx, other, "build-2", now, []CheckBuild{b}); err != nil {
t.Fatal(err)
}
held := func() bool { _, ok := m.manifests["scratch/meshboard/server@"+digest]; return ok }
found := func() bool { _, ok, _ := registry.Find(ctx, "meshboard", key); return ok }
if n, err := registry.Forget(ctx, aPull); err != nil || n != 1 {
t.Fatalf("forgot %d (%v)", n, err)
}
if !held() || !found() {
t.Fatalf("the build a later check names went with the check that built it: held %v, record %v", held(), found())
}
if n, err := registry.Forget(ctx, other); err != nil || n != 1 {
t.Fatalf("forgot %d (%v)", n, err)
}
if held() || found() {
t.Fatalf("the build stayed after the last check naming it went: held %v, record %v", held(), found())
}
}
// Only an earlier check of the same pull request at the same head is kept from: another head of it, or another pull
// request whose verdict names the same key and digests, builds (novox/hq issue 703).
func TestACheckKeepsNoBuildOfAnotherHeadOrPullRequest(t *testing.T) {
withVersion(t, "builder-1")
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
for name, c := range map[string]struct {
pull PullRef
head string
}{
"another head": {aPull, "999999999999"},
"another pull request": {other, "abcdefabcdef"},
} {
t.Run(name, func(t *testing.T) {
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
leave := named(first)
leave[0].Pull = c.pull // as if its own verdict named it
again, r := aCheckOf(t, scratch, c.pull, c.head, "tree-one", leave)
if !dockerBuilt(r) || again.Kept != "" {
t.Fatalf("a build of %s was kept: %+v", name, again)
}
})
}
}
+180
View File
@@ -0,0 +1,180 @@
package builder
import (
"archive/tar"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
)
// A module moves only when its program changes (novox/hq issue 721). A merge check builds every module the delivery
// plan moves; a module whose every artifact comes out the same as the build the mesh holds of it — the same files,
// the same modes and owners, the same entrypoint and environment — is said unchanged, and the walk leaves it where it
// is. **The whole image is compared, never its main binary alone**: a module whose image differs in any file moves.
// What is ignored is only what a rebuild cannot help changing: each file's time, the image's labels and its creation
// time. **A comparison that cannot be made says nothing**, and the module is built and moved as before.
// sameAsHeld says whether every artifact made is the same as the build the mesh holds of the module: the sentence
// that says so and, per artifact, the held reference it was compared with. Empty when any artifact differs, is not
// an image, has no held build, or could not be compared.
func sameAsHeld(ctx context.Context, run Runner, dir, module string, made []catalogue.Built,
held map[string]string) (string, map[string]string, error) {
if len(made) == 0 {
return "", nil, nil
}
against := map[string]string{}
for _, m := range made {
if m.Kind != "image" || m.Reference == "" {
return "", nil, nil
}
was := held[module+"/"+m.Name]
if was == "" {
return "", nil, nil
}
if digestPart(was) != "" && digestPart(was) == digestPart(m.Reference) {
against[m.Name] = was
continue
}
now, err := imageContent(ctx, run, dir, m.Reference)
if err != nil {
return "", nil, fmt.Errorf("%s's new image: %w", m.Name, err)
}
before, err := imageContent(ctx, run, dir, was)
if err != nil {
return "", nil, fmt.Errorf("%s's held image: %w", m.Name, err)
}
if now != before {
return "", nil, nil
}
against[m.Name] = was
}
var names []string
for name, ref := range against {
names = append(names, name+" "+shortRef(ref))
}
sort.Strings(names)
return "unchanged: its program is the same as running build " + strings.Join(names, ", "), against, nil
}
// digestPart is a reference's "sha256:…", empty when it is not pinned.
func digestPart(ref string) string {
if i := strings.LastIndex(ref, "@"); i >= 0 {
return ref[i+1:]
}
return ""
}
// shortRef names a reference by its digest's first twelve characters when it has one.
func shortRef(ref string) string {
if d := strings.TrimPrefix(digestPart(ref), "sha256:"); d != "" {
return twelve(d)
}
return ref
}
// imageContent is an image's content as one digest: its platform, its configuration without labels, and every file
// of its filesystem by path, type, mode, owner, link, extended attributes and bytes — never by time.
func imageContent(ctx context.Context, run Runner, dir, ref string) (string, error) {
if _, err := run(ctx, dir, "docker", "pull", "--quiet", ref); err != nil {
return "", fmt.Errorf("pulling %s: %w", ref, err)
}
config, err := run(ctx, dir, "docker", "image", "inspect", "--format",
"{{.Os}}/{{.Architecture}}/{{.Variant}} {{json .Config}}", ref)
if err != nil {
return "", fmt.Errorf("reading %s's configuration: %w", ref, err)
}
said, err := configContent(config)
if err != nil {
return "", fmt.Errorf("reading %s's configuration: %w", ref, err)
}
id, err := run(ctx, dir, "docker", "create", "--entrypoint", "/", ref)
if err != nil {
return "", fmt.Errorf("opening %s's filesystem: %w", ref, err)
}
id = strings.TrimSpace(id)
if id == "" {
return "", errors.New("the container engine named no container for " + ref)
}
defer func() { _, _ = run(context.WithoutCancel(ctx), dir, "docker", "rm", "--force", id) }()
if err := os.MkdirAll(dir, 0o700); err != nil {
return "", err
}
file := filepath.Join(dir, "content-"+twelve(id)+".tar")
defer os.Remove(file)
if _, err := run(ctx, dir, "docker", "export", "--output", file, id); err != nil {
return "", fmt.Errorf("reading %s's filesystem: %w", ref, err)
}
f, err := os.Open(file)
if err != nil {
return "", err
}
defer f.Close()
files, err := treeContent(f)
if err != nil {
return "", fmt.Errorf("reading %s's filesystem: %w", ref, err)
}
sum := sha256.Sum256([]byte(said + "\n" + files))
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// configContent is an image's platform and configuration as compared: Labels, the parent's id and the hostname the
// builder ran under are left out — the first holds the commit, the others no part of what runs.
func configContent(inspected string) (string, error) {
platform, raw, ok := strings.Cut(strings.TrimSpace(inspected), " ")
if !ok {
return "", fmt.Errorf("unexpected inspection %q", inspected)
}
var config map[string]json.RawMessage
if err := json.Unmarshal([]byte(raw), &config); err != nil {
return "", err
}
for _, k := range []string{"Labels", "Image", "Hostname", "Domainname"} {
delete(config, k)
}
out, err := json.Marshal(config) // map keys are sorted
if err != nil {
return "", err
}
return platform + " " + string(out), nil
}
// treeContent is a filesystem tarball's entries, sorted by path, each with everything but its times.
func treeContent(r io.Reader) (string, error) {
tr := tar.NewReader(r)
var lines []string
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return "", err
}
sum := sha256.New()
if _, err := io.Copy(sum, tr); err != nil {
return "", err
}
var xattrs []string
for k, v := range h.PAXRecords {
if strings.HasPrefix(k, "SCHILY.xattr.") {
xattrs = append(xattrs, k+"="+hex.EncodeToString([]byte(v)))
}
}
sort.Strings(xattrs)
lines = append(lines, fmt.Sprintf("%q %c %o %d:%d %q %d,%d %d %x %s",
strings.TrimPrefix(h.Name, "./"), h.Typeflag, h.Mode, h.Uid, h.Gid, h.Linkname, h.Devmajor, h.Devminor,
h.Size, sum.Sum(nil), strings.Join(xattrs, ";")))
}
sort.Strings(lines)
return strings.Join(lines, "\n"), nil
}
+126
View File
@@ -0,0 +1,126 @@
package builder
import (
"archive/tar"
"bytes"
"context"
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// aFilesystem is a filesystem tarball of files, each written at the time given.
func aFilesystem(t *testing.T, at time.Time, files map[string]string) []byte {
t.Helper()
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
for _, name := range []string{"usr/local/bin/mesh-builder", "etc/apk/world"} {
body, ok := files[name]
if !ok {
continue
}
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(body)), ModTime: at,
Typeflag: tar.TypeReg}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write([]byte(body)); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
// anEngine answers the container engine's commands for images by reference: each image's filesystem and labels.
func anEngine(images map[string][]byte, labels map[string]string, failing string) Runner {
created := map[string]string{}
return func(_ context.Context, _ string, name string, args ...string) (string, error) {
joined := strings.Join(args, " ")
ref := args[len(args)-1]
if failing != "" && strings.Contains(joined, failing) {
return "", fmt.Errorf("the engine refused %s", joined)
}
switch {
case args[0] == "pull", args[0] == "rm":
return "", nil
case args[0] == "image":
return fmt.Sprintf(`linux/amd64/ {"Entrypoint":["/usr/local/bin/mesh-builder"],"Labels":{"commit":%q}}`,
labels[ref]), nil
case args[0] == "create":
id := fmt.Sprintf("container%04d", len(created))
created[id] = ref
return id + "\n", nil
case args[0] == "export":
return "", os.WriteFile(args[2], images[created[ref]], 0o600)
}
return "", fmt.Errorf("unexpected %s %s", name, joined)
}
}
// TestAModuleWhoseProgramDidNotChangeIsSaidUnchanged is novox/hq issue 721: mesh-controller #335 changed one sentence
// in a package mesh-builder imports and never reaches, and build-agent moved. Built again, its image differs only in
// the files' times and its commit label, and the check says it is the same as the build the mesh holds.
func TestAModuleWhoseProgramDidNotChangeIsSaidUnchanged(t *testing.T) {
held := map[string]string{"build-agent/agent": "store/build-agent@sha256:aaaa"}
made := []catalogue.Built{{Name: "agent", Kind: "image", Reference: "scratch/build-agent@sha256:bbbb"}}
files := map[string]string{"usr/local/bin/mesh-builder": "the same program", "etc/apk/world": "docker-cli\ngit\n"}
images := map[string][]byte{
"store/build-agent@sha256:aaaa": aFilesystem(t, time.Unix(1000, 0), files),
"scratch/build-agent@sha256:bbbb": aFilesystem(t, time.Unix(2000, 0), files),
}
labels := map[string]string{"store/build-agent@sha256:aaaa": "a5500f73", "scratch/build-agent@sha256:bbbb": "c2ec0a60"}
same, against, err := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent",
made, held)
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(same, "unchanged: its program is the same as running build agent") ||
against["agent"] != held["build-agent/agent"] {
t.Fatalf("an image the same but for its times and labels is not said unchanged: %q %v", same, against)
}
// **Any file differing moves it**, not only its program.
changed := map[string]string{"usr/local/bin/mesh-builder": "the same program", "etc/apk/world": "docker-cli\ngit\ncurl\n"}
images["scratch/build-agent@sha256:bbbb"] = aFilesystem(t, time.Unix(2000, 0), changed)
if same, _, err := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent", made,
held); err != nil || same != "" {
t.Fatalf("an image with another file is said unchanged: %q %v", same, err)
}
// **A comparison that fails says nothing**, and the module moves.
images["scratch/build-agent@sha256:bbbb"] = aFilesystem(t, time.Unix(2000, 0), files)
if same, _, err := sameAsHeld(context.Background(), anEngine(images, labels, "export"), t.TempDir(), "build-agent",
made, held); err == nil || same != "" {
t.Fatalf("a comparison that failed said %q, %v", same, err)
}
// No held build, or an artifact that is not an image: nothing compared.
if same, _, _ := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent", made,
nil); same != "" {
t.Fatalf("a module the mesh holds no build of is said unchanged: %q", same)
}
archive := []catalogue.Built{{Name: "agent", Kind: "archive", Reference: "scratch/build-agent@sha256:bbbb"}}
if same, _, _ := sameAsHeld(context.Background(), anEngine(images, labels, ""), t.TempDir(), "build-agent",
archive, held); same != "" {
t.Fatalf("an archive is said unchanged without being compared: %q", same)
}
}
// TestAnImageWithAnotherEntrypointIsNotTheSame: the configuration counts, but for its labels.
func TestAnImageWithAnotherEntrypointIsNotTheSame(t *testing.T) {
a, err := configContent(`linux/amd64/ {"Entrypoint":["/a"],"Labels":{"commit":"1"}}`)
if err != nil {
t.Fatal(err)
}
b, _ := configContent(`linux/amd64/ {"Labels":{"commit":"2"},"Entrypoint":["/a"]}`)
c, _ := configContent(`linux/amd64/ {"Entrypoint":["/b"],"Labels":{"commit":"1"}}`)
if a != b || a == c {
t.Fatalf("labels count, or the entrypoint does not: %q %q %q", a, b, c)
}
}
+80
View File
@@ -0,0 +1,80 @@
package builder
import (
"os"
"os/exec"
"path/filepath"
"sync"
"testing"
)
// **A judge's build in a test starts from a warm Go cache** (novox/hq issue 717). The runtime these tests put on
// PATH (aRuntimeThatBuildsTheJudge) runs the check's `go build` and `go test` on this machine with the Go cache the
// check names — a fresh directory in every test — so each of them compiled the standard library from nothing: some
// ten CPU-seconds a build, under twenty tests, half of the CPU the whole suite spent. The runtime now fills an empty
// Go cache it is given from one this process warmed once, before it runs `go`; what the check names, mounts and
// keeps is unchanged, and so is what each test reads of it. A cache entry is keyed by everything the compile
// depends on, flags included, so a build with a planted -toolexec hook still misses it and still runs its hook.
//
// The warm cache lives beside the system's temporary directory as it was when the package started, never under a
// TMPDIR a test set and then asserts left empty, and is removed when the package's tests end.
var (
warmCacheBase = os.TempDir()
warmCacheOnce sync.Once
warmCacheDir string
)
func TestMain(m *testing.M) {
code := m.Run()
if warmCacheDir != "" {
_ = os.RemoveAll(warmCacheDir)
}
os.Exit(code)
}
// warmGoCache is a Go cache holding what a judge's build and the replays' `go test` compile of the standard
// library, or "" when it cannot be warmed — a test then builds cold, as before, and is only slower.
func warmGoCache(t *testing.T) string {
t.Helper()
warmCacheOnce.Do(func() {
dir, err := os.MkdirTemp(warmCacheBase, "judge-warm-go-cache-")
if err != nil {
return
}
src := filepath.Join(dir, "src")
files := map[string]string{
"go.mod": "module example.org/warm\n\ngo 1.22\n",
"main.go": "package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n)\n\nfunc main() { fmt.Fprintln(os.Stdout, len(os.Args)) }\n",
"main_test.go": "package main\n\nimport \"testing\"\n\nfunc TestWarm(t *testing.T) {}\n",
}
for name, body := range files {
if err := os.MkdirAll(src, 0o755); err != nil {
return
}
if err := os.WriteFile(filepath.Join(src, name), []byte(body), 0o644); err != nil {
return
}
}
cache := filepath.Join(dir, "go-cache")
env := append(os.Environ(), "GOCACHE="+cache, "GOENV=off", "GOTOOLCHAIN=local", "GOPROXY=off", "GOWORK=off",
"GOFLAGS=-buildvcs=false", "TMPDIR="+warmCacheBase)
// As the check builds a judge (CGO off) and runs the replays (CGO as the toolchain has it).
for _, run := range [][]string{
{"env", "CGO_ENABLED=0", "go", "build", "-o", filepath.Join(dir, "judge"), "."},
{"go", "test", "-count=1", "."},
} {
cmd := exec.Command(run[0], run[1:]...)
cmd.Dir, cmd.Env = src, env
if out, err := cmd.CombinedOutput(); err != nil {
t.Logf("the warm Go cache could not be built, so judges build cold: %v\n%s", err, out)
_ = os.RemoveAll(dir)
return
}
}
warmCacheDir = dir
})
if warmCacheDir == "" {
return ""
}
return filepath.Join(warmCacheDir, "go-cache")
}
+10
View File
@@ -21,6 +21,16 @@ func TestTheBussAccountSaysNothingOnTheBus(t *testing.T) {
t.Fatalf("a bus module declaring what it emits was not refused, or not for the reason: %v", err)
}
// Its tools bundle says what it emits (novox/hq ADR 0357): the node's runtime carries the bundle and is
// granted the module's events, so a bus module that ships one may declare them.
bundled := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"emits":["bus.slow-consumer"],"tools":["bus_streams"],
"build":{"artifacts":[{"name":"tools","kind":"bundle","language":"go","system":"arch","from":"cmd/t","binary":"t"}]}}`)
if _, err := ParseManifest(bundled); err != nil {
t.Fatalf("a bus module whose tools bundle declares what it emits was refused: %v", err)
}
quiet := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"tools":["bus_streams"]}`)
+82 -2
View File
@@ -513,6 +513,13 @@ type Manifest struct {
// person's account (design 25 §7) already had the same shape.
Invokes []string `json:"invokes,omitempty"`
// Waits are the event families a session may wait on through the mesh MCP server's `mesh_wait`, each
// `<emitter>.<event>` as `consumes` names one, a seat's or a module's (novox/hq ADR 0367 §2). Only the tool
// runner's module declares them, one by one and never by a wildcard. **A grant apart from `consumes`**: the
// controller grants the tool runner's account a plain subscription on those subjects alone, with no
// durable consumer, and issues the same list in its membership; no call or setting adds one (§6).
Waits []string `json:"waits,omitempty"`
// State is the current state this module keeps on the bus, by local name: each a key-value
// bucket the controller creates, which every instance of the module writes and reads
// (novox/hq ADR 0201). Not history — that is an event — and never a secret, sealed or not.
@@ -523,6 +530,12 @@ type Manifest struct {
// (novox/hq ADR 0297). A log is its owner's alone: no other module reads it.
Logs []LogDeclaration `json:"logs,omitempty"`
// Streams are the streams this module owns on the bus (novox/hq ADR 0363): each named locally, the
// events it carries named as `emits` names them but with `*` and `**`, and bounded in age and size; the
// controller creates each. One may be restricted by a trusted setting of this module, so a consumer
// hears a part of it only where that setting's value names it.
Streams []StreamDeclaration `json:"streams,omitempty"`
// Settings are the defaults this module gives its settings (novox/hq ADR 0262): each key a file,
// a contribution or a served fact asks for as `${setting:<key>}`, its default, and why that
// default. Only a preference has one — a font size, a width, a number of workers — and a value
@@ -552,6 +565,10 @@ type Manifest struct {
// why. A person's choice through the `upgrade` verb stands over it; the bus records whatever it says.
Upgrade *UpgradePolicy `json:"upgrade,omitempty"`
// Unhealthy is how loud this module's unhealthy condition is (novox/hq ADR 0355): a warning when unsaid;
// `urgent`, with why, for a module whose failure costs people something they rely on.
Unhealthy *UnhealthyPolicy `json:"unhealthy,omitempty"`
// Reads are other modules' state this module reads and watches, each `<module>.<name>`
// (novox/hq ADR 0201). Read-only: only the owner's instances write.
Reads []string `json:"reads,omitempty"`
@@ -1630,16 +1647,22 @@ func ParseManifest(raw []byte) (Manifest, error) {
// **The bus's own account copies the bus and says nothing on it** (novox/hq ADR 0235). The module
// holding mesh-broker is granted the snapshot API and nothing else, so anything it declared to say
// or hear on the bus would be granted nothing — refused here rather than silently dropped.
//
// **But its tools bundle may say what it declares** (novox/hq ADR 0357): the node's runtime carries
// the bundle and is granted the events the module emits, under the module's name, as for any module it
// carries — the bus's account is not who publishes them. So emits are refused only for a bus module
// with no bundle to say them.
if _, account := m.OwnSecrets["broker"]; account && m.ClaimsSeat(BrokerSeat) {
var said []string
if len(m.EmitsAll()) > 0 {
if len(m.EmitsAll()) > 0 && !m.shipsABundle() {
said = append(said, "emits")
}
for _, f := range []struct {
name string
n int
}{{"consumes", len(m.Consumes)}, {"uses", len(m.Uses)}, {"invokes", len(m.Invokes)},
{"state", len(m.State)}, {"logs", len(m.Logs)}, {"reads", len(m.Reads)}} {
{"state", len(m.State)}, {"logs", len(m.Logs)}, {"reads", len(m.Reads)},
{"streams", len(m.Streams)}} {
if f.n > 0 {
said = append(said, f.name)
}
@@ -1783,6 +1806,8 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, StateProblems(m)...)
// And what it may call its logs, and how large it may ask them to grow (logs.go, novox/hq ADR 0297).
problems = append(problems, LogProblems(m)...)
// And the streams it owns, and who may hear them (streams.go, novox/hq ADR 0363).
problems = append(problems, StreamProblems(m)...)
// And the defaults it gives its settings (setting_defaults.go, novox/hq ADR 0262).
problems = append(problems, SettingProblems(m)...)
// And what it requires optionally, and the files it delivers only while bound (novox/hq ADR 0330).
@@ -1953,6 +1978,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
}
problems = append(problems, invokeProblems(m)...)
problems = append(problems, waitProblems(m)...)
problems = append(problems, replacesProblems(m)...)
problems = append(problems, UnitSettingProblems(m)...)
problems = append(problems, OwnerSettingProblems(m)...)
@@ -2321,6 +2347,7 @@ func ParseManifest(raw []byte) (Manifest, error) {
problems = append(problems, m.placementProblems()...)
problems = append(problems, m.dataProblems()...)
problems = append(problems, m.upgradeProblems()...)
problems = append(problems, m.unhealthyProblems()...)
for i, r := range m.Resources {
id, _ := r["id"].(string)
@@ -2675,6 +2702,46 @@ func invokeProblems(m Manifest) []string {
return problems
}
// waitToken is one name of an event family's subject: never a wildcard, never a dot, never empty.
var waitToken = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]*$`)
// waitProblems judges the event families a module says a session may wait on (novox/hq ADR 0367 §2, §6):
// the tool runner's alone, each named whole — emitter and event, with no wildcard — and each once. Refused
// here, in the manifest's words, rather than at the composition of the bus's users.
func waitProblems(m Manifest) []string {
if len(m.Waits) == 0 {
return nil
}
if m.Module != RuntimeModule {
return []string{fmt.Sprintf("%s says what a session waits on; only %s, the tool runner's module, "+
"declares waits (novox/hq ADR 0367 §2)", m.Module, RuntimeModule)}
}
var problems []string
seen := map[string]bool{}
for _, w := range m.Waits {
if strings.ContainsAny(w, "*>") {
problems = append(problems, fmt.Sprintf("%s waits on %q: a family is named one by one, never a "+
"wildcard (novox/hq ADR 0367 §2)", m.Module, w))
continue
}
tokens := strings.Split(w, ".")
whole := len(tokens) >= 2 && name.MatchString(tokens[0])
for _, t := range tokens[1:] {
whole = whole && waitToken.MatchString(t)
}
if !whole {
problems = append(problems, fmt.Sprintf("%s waits on %q, which does not name an emitter and an "+
"event: a family is <emitter>.<event>, as consumes names one", m.Module, w))
continue
}
if seen[w] {
problems = append(problems, fmt.Sprintf("%s waits on %q twice", m.Module, w))
}
seen[w] = true
}
return problems
}
// How a module takes one of its own secrets (ADR 0114): read from the file when it starts, or
// applied by its own code to a backend that takes it once.
const (
@@ -2897,3 +2964,16 @@ func whileStoppedProblems(m Manifest, r map[string]any, scheduled bool) []string
}
return problems
}
// shipsABundle says the module builds a tools bundle, which the node's runtime carries and speaks for.
func (m Manifest) shipsABundle() bool {
if m.Build == nil {
return false
}
for _, a := range m.Build.Artifacts {
if a.Kind == ArtifactBundle {
return true
}
}
return false
}
+2
View File
@@ -245,6 +245,8 @@ func IsUndeclaredSeat(problem string) bool { return strings.HasSuffix(problem, U
func CatalogueProblems(shelf Shelf) []string {
var problems []string
// A stream name has one owner (novox/hq ADR 0363).
problems = append(problems, StreamNameProblems(shelf)...)
// Who declares what, and who declared it first.
declaredBy := map[string]string{}
+479
View File
@@ -0,0 +1,479 @@
package catalogue
import (
"bytes"
"encoding/json"
"fmt"
"slices"
"strings"
)
// A module's own stream (novox/hq ADR 0363).
//
// A module declares a stream as it declares a log (ADR 0297) or a bucket (ADR 0201): named locally, the
// events it carries named locally as `emits` names them, its age and its size within the mesh's bounds.
// The controller creates it and stays the only writer of stream definitions. A stream may be restricted
// by a trusted config value of its own module: then a consumer hears a part of it only where its own
// `consumes` asks and that value names it.
// The mesh's bounds on a module's stream: its age in days and its size in MiB, and what it gets when it
// says nothing.
const (
StreamLeastDays = 1
StreamMostDays = 30
StreamDefaultDays = 7
StreamLeastMiB = 1
StreamMostMiB = 8192
StreamDefaultMiB = 1024
)
// GrantKey is the name in a grant's pattern that each key of the config value fills.
const GrantKey = "{key}"
// ReservedStreamPrefixes are the starts of the stream names the mesh derives, which no module's stream
// may take: a seat's work queue, a log, a bucket, an object store, and the dead letters.
var ReservedStreamPrefixes = []string{"SEAT_", "LOG_", "KV_", "OBJ_", "DEAD_"}
// MeshStreamNames are the mesh's own streams by name. Written here as well as in the broker, which this
// package cannot import; a test keeps them agreeing.
var MeshStreamNames = []string{"CONTROL", "NODES", "ASSIGNMENTS", "EVENTS", "DEAD_LETTER_NOTICES", "DEAD_LETTERS"}
// StreamDeclaration is one stream a module owns.
type StreamDeclaration struct {
Name string `json:"name"`
// Events are the events it carries, named locally; `*` is one name and `**`, last only, the rest.
Events []string `json:"events"`
// MaxAgeDays and MaxMiB bound it; zero is the default.
MaxAgeDays int `json:"max-age-days,omitempty"`
MaxMiB int `json:"max-mib,omitempty"`
// Grant restricts it, when said.
Grant *StreamGrant `json:"grant,omitempty"`
}
// StreamGrant says which trusted config value of the module grants a consumer a part of the stream, and
// which part each of its keys names.
type StreamGrant struct {
// Config is the trusted config value of the module that grants it (ADR 0348 §8): set only at the
// controller's terminal or on the operator's Approve.
Config string `json:"config"`
// Events is a pattern holding GrantKey exactly once, which each key of the config value fills.
Events string `json:"events"`
}
// UnmarshalJSON reads a stream strictly: an unknown key is refused here, not passed over.
func (s *StreamDeclaration) UnmarshalJSON(raw []byte) error {
type plain StreamDeclaration
var p plain
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&p); err != nil {
return fmt.Errorf("a stream is {name, events, max-age-days, max-mib, grant}: %w", typedUnknown(err))
}
*s = StreamDeclaration(p)
return nil
}
// UnmarshalJSON reads a grant strictly.
func (g *StreamGrant) UnmarshalJSON(raw []byte) error {
type plain StreamGrant
var p plain
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&p); err != nil {
return fmt.Errorf("a stream's grant is {config, events}: %w", typedUnknown(err))
}
*g = StreamGrant(p)
return nil
}
// Days is the stream's age in days, the default where none is said.
func (s StreamDeclaration) Days() int {
if s.MaxAgeDays == 0 {
return StreamDefaultDays
}
return s.MaxAgeDays
}
// Cap is the stream's size in MiB, the default where none is said.
func (s StreamDeclaration) Cap() int {
if s.MaxMiB == 0 {
return StreamDefaultMiB
}
return s.MaxMiB
}
// BusName is the stream's name on the bus: its local name in capitals, a hyphen an underscore.
func (s StreamDeclaration) BusName() string { return StreamBusName(s.Name) }
// StreamBusName is a local stream name as the bus holds it.
func StreamBusName(local string) string {
return strings.ToUpper(strings.ReplaceAll(local, "-", "_"))
}
// Carries says whether an event, or every event a pattern names, is one this stream carries: within one
// of its patterns.
func (s StreamDeclaration) Carries(pattern string) bool {
for _, p := range s.Events {
if within(pattern, p) {
return true
}
}
return false
}
// StreamProblems is what is wrong with a manifest's streams, judged on the one manifest.
func StreamProblems(m Manifest) []string {
if len(m.Streams) == 0 {
return nil
}
var problems []string
if !stateName.MatchString(m.Module) {
problems = append(problems, fmt.Sprintf(
"%s declares a stream, and a module's name is part of its stream's subjects, which take one plain "+
"name — no dot (novox/hq ADR 0363)", m.Module))
}
seen := map[string]bool{}
terminal := TerminalKeys(m)
for i, s := range m.Streams {
switch {
case !stateName.MatchString(s.Name):
problems = append(problems, fmt.Sprintf(
"%s declares stream %q: a stream is named locally — lower-case letters, digits and hyphens "+
"(novox/hq ADR 0363)", m.Module, s.Name))
case seen[s.BusName()]:
problems = append(problems, fmt.Sprintf("%s declares stream %q twice", m.Module, s.Name))
case slices.Contains(MeshStreamNames, s.BusName()):
problems = append(problems, fmt.Sprintf(
"%s declares stream %q, which is the mesh's own stream %s (novox/hq ADR 0363)", m.Module, s.Name, s.BusName()))
default:
for _, prefix := range ReservedStreamPrefixes {
if strings.HasPrefix(s.BusName(), prefix) {
problems = append(problems, fmt.Sprintf(
"%s declares stream %q, whose name on the bus starts with %s, which the mesh derives names "+
"with (novox/hq ADR 0363)", m.Module, s.Name, prefix))
}
}
}
seen[s.BusName()] = true
if s.MaxAgeDays != 0 && (s.MaxAgeDays < StreamLeastDays || s.MaxAgeDays > StreamMostDays) {
problems = append(problems, fmt.Sprintf("%s keeps stream %q %d days; a stream keeps between %d and %d "+
"(novox/hq ADR 0363)", m.Module, s.Name, s.MaxAgeDays, StreamLeastDays, StreamMostDays))
}
if s.MaxMiB != 0 && (s.MaxMiB < StreamLeastMiB || s.MaxMiB > StreamMostMiB) {
problems = append(problems, fmt.Sprintf("%s caps stream %q at %d MiB; a stream holds between %d and %d "+
"(novox/hq ADR 0363)", m.Module, s.Name, s.MaxMiB, StreamLeastMiB, StreamMostMiB))
}
if len(s.Events) == 0 {
problems = append(problems, fmt.Sprintf("%s declares stream %q with no events", m.Module, s.Name))
}
for _, e := range s.Events {
if err := eventPattern(e, false); err != nil {
problems = append(problems, fmt.Sprintf("%s's stream %q carries %q: %v", m.Module, s.Name, e, err))
}
}
// **An event goes to one place** (ADR 0363 §2): an emitted event a stream carries, or two streams
// carrying one event, would leave the module's runtime to choose.
for _, e := range m.EmitsAll() {
if s.Carries(e) {
problems = append(problems, fmt.Sprintf(
"%s emits %q, which its stream %q carries: an event goes to EVENTS or to the module's stream, "+
"so name it in one place (novox/hq ADR 0363)", m.Module, e, s.Name))
}
}
for _, other := range m.Streams[i+1:] {
for _, a := range s.Events {
for _, b := range other.Events {
if patternsOverlap(a, b) {
problems = append(problems, fmt.Sprintf(
"%s's streams %q and %q both carry %q and %q: an event goes to one stream (novox/hq "+
"ADR 0363)", m.Module, s.Name, other.Name, a, b))
}
}
}
}
if g := s.Grant; g != nil {
if !slices.Contains(terminal, g.Config) {
problems = append(problems, fmt.Sprintf(
"%s's stream %q is granted by %q, which is not a trusted config value of the module, set only at "+
"the controller's terminal or on the operator's Approve: a grant of what a module hears is the "+
"operator's (novox/hq ADR 0348 §8, ADR 0363)", m.Module, s.Name, g.Config))
}
if strings.Count(g.Events, GrantKey) != 1 {
problems = append(problems, fmt.Sprintf(
"%s's stream %q grants %q, which must hold %s exactly once, as a name of its own (novox/hq ADR 0363)",
m.Module, s.Name, g.Events, GrantKey))
} else if err := eventPattern(g.Events, true); err != nil {
problems = append(problems, fmt.Sprintf("%s's stream %q grants %q: %v", m.Module, s.Name, g.Events, err))
}
}
}
return problems
}
// eventPattern checks one local event pattern: names, `*` for one, `**` last for the rest; with key,
// GrantKey is a name too.
func eventPattern(pattern string, key bool) error {
if pattern == "" {
return fmt.Errorf("a pattern needs a name")
}
parts := strings.Split(pattern, ".")
for i, part := range parts {
switch {
case part == TheRest:
if i != len(parts)-1 {
return fmt.Errorf("%q stands for the rest of a name, so nothing may follow it", TheRest)
}
case part == OneName:
case key && part == GrantKey:
case !eventName.MatchString(part):
return fmt.Errorf("%q is not a usable name: lower-case letters, digits and dashes", part)
}
}
return nil
}
// StreamsHearing is every stream of a module that carries some of a consumed pattern's events, and
// whether one of them carries all of them.
func (m Manifest) StreamsHearing(pattern string) (hearing []StreamDeclaration, whole bool) {
for _, s := range m.Streams {
for _, p := range s.Events {
if patternsOverlap(pattern, p) {
hearing = append(hearing, s)
break
}
}
if s.Carries(pattern) {
whole = true
}
}
return hearing, whole
}
// within says every event pattern a names is one b names too.
func within(a, b string) bool {
x, y := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; ; i++ {
switch {
case i == len(y):
return i == len(x)
case y[i] == TheRest:
return i < len(x)
case i == len(x):
return false
case x[i] == TheRest:
return false
case y[i] == OneName:
continue
case x[i] == OneName:
return false
case x[i] != y[i]:
return false
}
}
}
// patternsOverlap says some event both patterns name.
func patternsOverlap(a, b string) bool {
x, y := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; ; i++ {
switch {
case i == len(x) && i == len(y):
return true
case i == len(x) || i == len(y):
return false
case x[i] == TheRest || y[i] == TheRest:
return true
case x[i] == OneName || y[i] == OneName || x[i] == y[i]:
continue
default:
return false
}
}
}
// StreamNameProblems are the rules about stream names no single manifest can be judged against: two
// modules declaring one name.
func StreamNameProblems(shelf Shelf) []string {
var problems []string
declaredBy := map[string]string{}
for _, module := range shelfOrder(shelf) {
for _, s := range shelf[module].Streams {
name := s.BusName()
if first, taken := declaredBy[name]; taken && first != module {
problems = append(problems, fmt.Sprintf(
"%s declares the stream %s, which %s already declares; a stream has one owner (novox/hq ADR 0363)",
module, name, first))
continue
}
declaredBy[name] = module
}
}
return problems
}
// GrantedKeys reads a restricting config value (novox/hq ADR 0363 §4): each key whose list names the
// module wherever it runs, or the module on this machine as `<machine>.<module>`. A value of another shape
// grants nothing; a key that is not one usable name grants nothing.
func GrantedKeys(value any, module, machine string) []string {
if text, isText := value.(string); isText {
// A layer may hold the object as its JSON text.
var parsed any
if json.Unmarshal([]byte(text), &parsed) != nil {
return nil
}
value = parsed
}
entries, ok := value.(map[string]any)
if !ok {
return nil
}
var out []string
for key, listed := range entries {
if !eventName.MatchString(key) {
continue
}
names, ok := listed.([]any)
if !ok {
continue
}
for _, n := range names {
s, _ := n.(string)
if s == module || (machine != "" && s == machine+"."+module) {
out = append(out, key)
break
}
}
}
slices.Sort(out)
return out
}
// Granted is a grant's pattern with its key filled.
func (g StreamGrant) Granted(key string) string { return strings.Replace(g.Events, GrantKey, key, 1) }
// narrow is the events both patterns name, as one pattern; false when they name none in common.
func narrow(a, b string) (string, bool) {
x, y := strings.Split(a, "."), strings.Split(b, ".")
var out []string
for i := 0; ; i++ {
switch {
case i == len(x) && i == len(y):
return strings.Join(out, "."), true
case i == len(x) || i == len(y):
return "", false
case x[i] == TheRest:
return strings.Join(append(out, y[i:]...), "."), true
case y[i] == TheRest:
return strings.Join(append(out, x[i:]...), "."), true
case x[i] == OneName:
out = append(out, y[i])
case y[i] == OneName || x[i] == y[i]:
out = append(out, x[i])
default:
return "", false
}
}
}
// HeardPart is what a consumer hears of one module's stream: the patterns, local, it is granted.
type HeardPart struct {
Emitter string
Stream StreamDeclaration
Patterns []string
}
// Hearing is what a consumer's `consumes` hears of the module streams the shelf declares, on one machine
// (novox/hq ADR 0363 §5): for each stream, its `consumes` narrowed to the stream's events and, for a
// restricted stream, to the parts the provider's trusted config value grants it. values answers a restricting config
// value on each machine running the provider. Answered too: whether any `consumes` names a module owning a
// stream, granted or not, and the `consumes` left for EVENTS — those naming something the provider emits
// there.
//
// Only an entry naming its emitter literally reaches a module's stream: a wildcard emitter, `**` among them,
// is EVENTS' alone, so nothing that hears every event hears a restricted stream by saying so.
func Hearing(consumer Manifest, shelf map[string]Manifest, machine string,
values func(provider, setting string) []any) (parts []HeardPart, touches bool, onEvents []string) {
byStream := map[string]*HeardPart{}
var order []string
for _, c := range consumer.Consumes {
emitter, rest, named := strings.Cut(c, ".")
provider, known := shelf[emitter]
if !named || emitter == OneName || emitter == TheRest || !known || len(provider.Streams) == 0 {
onEvents = append(onEvents, c)
continue
}
hearing, whole := provider.StreamsHearing(rest)
if len(hearing) > 0 {
touches = true
}
// Kept for EVENTS while the provider says something there it names: a stream carrying it whole, or
// a provider that says nothing of it on EVENTS, leaves it nothing to hear there.
if !whole && slices.ContainsFunc(provider.EmitsAll(), func(e string) bool { return patternsOverlap(rest, e) }) {
onEvents = append(onEvents, c)
}
for _, s := range hearing {
var granted []string
if s.Grant != nil {
for _, v := range values(emitter, s.Grant.Config) {
for _, key := range GrantedKeys(v, consumer.Module, machine) {
granted = append(granted, s.Grant.Granted(key))
}
}
}
var got []string
for _, p := range s.Events {
x, ok := narrow(rest, p)
if !ok {
continue
}
if s.Grant == nil {
got = append(got, x)
continue
}
for _, g := range granted {
if y, ok := narrow(x, g); ok {
got = append(got, y)
}
}
}
if len(got) == 0 {
continue
}
id := emitter + "/" + s.BusName()
if byStream[id] == nil {
byStream[id] = &HeardPart{Emitter: emitter, Stream: s}
order = append(order, id)
}
byStream[id].Patterns = append(byStream[id].Patterns, got...)
}
}
for _, id := range order {
h := byStream[id]
h.Patterns = uncovered(h.Patterns)
parts = append(parts, *h)
}
return parts, touches, onEvents
}
// uncovered is a list of patterns without those another names whole, and without repeats: the bus refuses
// a consumer whose filters overlap.
func uncovered(patterns []string) []string {
var out []string
for i, p := range patterns {
covered := false
for j, q := range patterns {
if i == j {
continue
}
if within(p, q) && (!within(q, p) || j < i) {
covered = true
break
}
}
if !covered {
out = append(out, p)
}
}
slices.Sort(out)
return out
}
+206
View File
@@ -0,0 +1,206 @@
package catalogue
import (
"encoding/json"
"slices"
"strings"
"testing"
)
// mailManifest is a module owning a restricted stream, as mesh-mail declares it (novox/hq ADR 0363), with
// the trusted mergeable file that takes its grants setting; extra is spliced in at the top.
func mailManifest(t *testing.T, streams, extra string) string {
t.Helper()
return `{"module":"mesh-mail","version":"1",` + extra + `"streams":` + streams + `,` +
`"resources":[{"id":"state","type":"directory","mode":"0700","place":"."},` +
`{"id":"grants","trusted":true,"type":"file","path":"${dir:state}/grants.json","mode":"0600",` +
`"merge":"json","takes-settings":["grants"],"content":"{\"grants\": {}}"}]}`
}
const mailStream = `[{"name":"mail","events":["mail.*.message.added","mail.*.message.changed",` +
`"mail.*.message.removed"],"max-age-days":7,"grant":{"config":"grants","events":"mail.{key}.**"}}]`
// A module declares a stream of its own (ADR 0363 §1): its events with a variable name (§3), its age and
// size with defaults, and a grant by a trusted setting of its own (§4).
func TestAModuleDeclaresAStreamOfItsOwn(t *testing.T) {
m, err := ParseManifest([]byte(mailManifest(t, mailStream, "")))
if err != nil {
t.Fatal(err)
}
if len(m.Streams) != 1 {
t.Fatalf("streams not read: %+v", m.Streams)
}
s := m.Streams[0]
if s.BusName() != "MAIL" || s.Days() != 7 || s.Cap() != StreamDefaultMiB || s.Grant == nil ||
s.Grant.Config != "grants" || s.Grant.Granted("mesh") != "mail.mesh.**" {
t.Fatalf("stream read as %+v", s)
}
if !s.Carries("mail.work.message.added") || s.Carries("mail.work.body") || !s.Carries("mail.*.message.removed") {
t.Fatal("the stream's patterns do not say what it carries")
}
plain, err := ParseManifest([]byte(`{"module":"feed","version":"1","streams":[{"name":"feed-items","events":["item.**"]}]}`))
if err != nil {
t.Fatalf("a stream without a grant was refused: %v", err)
}
if got := plain.Streams[0]; got.BusName() != "FEED_ITEMS" || got.Days() != StreamDefaultDays || got.Grant != nil {
t.Fatalf("defaults read as %+v", got)
}
}
// Registration refuses a stream the mesh could not hold or would leave ambiguous (ADR 0363 §1, §2, §4).
func TestAStreamIsNamedBoundedAndGrantedByATrustedConfigValue(t *testing.T) {
grants := `"grant":{"config":"grants","events":"mail.{key}.**"}`
for _, c := range []struct{ manifest, says string }{
{mailManifest(t, `[{"name":"Mail","events":["mail.**"]}]`, ""), `a stream is named locally`},
{mailManifest(t, `[{"name":"mail.x","events":["mail.**"]}]`, ""), `a stream is named locally`},
{mailManifest(t, `[{"name":"events","events":["mail.**"]}]`, ""), `the mesh's own stream EVENTS`},
{mailManifest(t, `[{"name":"log-x","events":["mail.**"]}]`, ""), `starts with LOG_`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"]},{"name":"mail","events":["x.**"]}]`, ""), `twice`},
{mailManifest(t, `[{"name":"mail","events":[]}]`, ""), `with no events`},
{mailManifest(t, `[{"name":"mail","events":["mail.**.x"]}]`, ""), `nothing may follow it`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"max-age-days":31}]`, ""), `between 1 and 30`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"max-mib":9000}]`, ""), `between 1 and 8192`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"stream":"MAIL"}]`, ""), `{name, events`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"]}]`, `"emits":["mail.mesh.message.added"],`),
`which its stream "mail" carries`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"]},{"name":"more","events":["mail.x.*"]}]`, ""),
`both carry`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"grant":{"config":"tags","events":"mail.{key}.**"}}]`, ""),
`is not a trusted config value`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"grant":{"config":"grants","events":"mail.**"}}]`, ""),
`exactly once`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],"grant":{"config":"grants","events":"{key}.{key}"}}]`, ""),
`exactly once`},
{mailManifest(t, `[{"name":"mail","events":["mail.**"],`+grants+`,"why":1}]`, ""), `{name, events`},
{`{"module":"a.b","version":"1","streams":[{"name":"x","events":["x"]}]}`, `no dot`},
} {
_, err := ParseManifest([]byte(c.manifest))
if err == nil {
t.Errorf("%s was accepted", c.manifest)
continue
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("%s refused for the wrong reason: %v", c.manifest, err)
}
}
}
// A stream name has one owner across the catalogue (ADR 0363 §1).
func TestTwoModulesCannotDeclareOneStream(t *testing.T) {
a := Manifest{Module: "a", Streams: []StreamDeclaration{{Name: "mail", Events: []string{"x"}}}}
b := Manifest{Module: "b", Streams: []StreamDeclaration{{Name: "mail", Events: []string{"y"}}}}
got := strings.Join(CatalogueProblems(Shelf{"a": a, "b": b}), "; ")
if !strings.Contains(got, "b declares the stream MAIL, which a already declares") {
t.Fatalf("a second owner of MAIL was not refused: %s", got)
}
if got := StreamNameProblems(Shelf{"a": a}); len(got) != 0 {
t.Fatalf("one owner refused: %v", got)
}
}
// The setting's value names who hears each key: a module wherever it runs, or on one machine (ADR 0363 §4).
func TestARestrictingSettingNamesAModuleOrAModuleOnAMachine(t *testing.T) {
var value any
if err := json.Unmarshal([]byte(`{"mesh":["reader","person.jochen"],"work":["ace.reader"],"home":["other"],`+
`"Bad.Key":["reader"]}`), &value); err != nil {
t.Fatal(err)
}
if got := GrantedKeys(value, "reader", "ace"); !slices.Equal(got, []string{"mesh", "work"}) {
t.Fatalf("on ace reader is granted %v", got)
}
if got := GrantedKeys(value, "reader", "shanks"); !slices.Equal(got, []string{"mesh"}) {
t.Fatalf("on shanks reader is granted %v", got)
}
if got := GrantedKeys(`{"mesh":["reader"]}`, "reader", "ace"); !slices.Equal(got, []string{"mesh"}) {
t.Fatalf("a value held as its JSON text grants %v", got)
}
if got := GrantedKeys([]any{"reader"}, "reader", "ace"); len(got) != 0 {
t.Fatalf("a value of another shape granted %v", got)
}
}
// A consumer hears a restricted stream only where its consumes asks and the setting grants it, and only
// by naming the emitter (ADR 0363 §5); what a stream carries whole leaves EVENTS.
func TestAConsumerHearsOnlyWhatItAsksAndIsGranted(t *testing.T) {
provider, err := ParseManifest([]byte(mailManifest(t, mailStream, `"emits":["account.added"],`)))
if err != nil {
t.Fatal(err)
}
shelf := map[string]Manifest{"mesh-mail": provider}
var value any
_ = json.Unmarshal([]byte(`{"mesh":["reader"],"work":["shanks.reader"]}`), &value)
values := func(module, setting string) []any {
if module == "mesh-mail" && setting == "grants" {
return []any{value}
}
return nil
}
reader := Manifest{Module: "reader", Consumes: []string{"mesh-mail.mail.**", "mesh-mail.account.added", "billing.paid"}}
parts, touches, onEvents := Hearing(reader, shelf, "ace", values)
if !touches || len(parts) != 1 || parts[0].Stream.BusName() != "MAIL" {
t.Fatalf("heard %+v", parts)
}
want := []string{"mail.mesh.message.added", "mail.mesh.message.changed", "mail.mesh.message.removed"}
if !slices.Equal(parts[0].Patterns, want) {
t.Fatalf("on ace reader hears %v, want %v", parts[0].Patterns, want)
}
if !slices.Equal(onEvents, []string{"mesh-mail.account.added", "billing.paid"}) {
t.Fatalf("left on EVENTS: %v", onEvents)
}
parts, _, _ = Hearing(reader, shelf, "shanks", values)
if len(parts) != 1 || len(parts[0].Patterns) != 6 {
t.Fatalf("on shanks reader hears %+v", parts)
}
narrow := Manifest{Module: "reader", Consumes: []string{"mesh-mail.mail.*.message.added"}}
parts, _, _ = Hearing(narrow, shelf, "ace", values)
if len(parts) != 1 || !slices.Equal(parts[0].Patterns, []string{"mail.mesh.message.added"}) {
t.Fatalf("a narrower consumes heard %+v", parts)
}
stranger := Manifest{Module: "stranger", Consumes: []string{"mesh-mail.mail.**"}}
parts, touches, onEvents = Hearing(stranger, shelf, "ace", values)
if len(parts) != 0 || !touches || len(onEvents) != 0 {
t.Fatalf("a module not granted heard %+v (touches %v, on EVENTS %v)", parts, touches, onEvents)
}
everything := Manifest{Module: "audit", Consumes: []string{"**", "*.mail.**"}}
parts, touches, _ = Hearing(everything, shelf, "ace", values)
if len(parts) != 0 || touches {
t.Fatalf("a wildcard emitter reached a module's stream: %+v", parts)
}
parts, _, _ = Hearing(reader, shelf, "ace", func(string, string) []any { return nil })
if len(parts) != 0 {
t.Fatalf("with no setting reader heard %+v", parts)
}
}
// A stream without a grant is heard by whoever asks, within its patterns.
func TestAnUnrestrictedStreamIsHeardByWhoeverAsks(t *testing.T) {
feed := Manifest{Module: "feed", Streams: []StreamDeclaration{{Name: "items", Events: []string{"item.*.added", "item.*.gone"}}}}
reader := Manifest{Module: "reader", Consumes: []string{"feed.item.**", "feed.item.x.added"}}
parts, touches, onEvents := Hearing(reader, map[string]Manifest{"feed": feed}, "ace", nil)
if !touches || len(onEvents) != 0 || len(parts) != 1 ||
!slices.Equal(parts[0].Patterns, []string{"item.*.added", "item.*.gone"}) {
t.Fatalf("heard %+v, on EVENTS %v", parts, onEvents)
}
}
func TestNarrowingTwoPatterns(t *testing.T) {
for _, c := range []struct{ a, b, want string }{
{"mail.**", "mail.*.message.added", "mail.*.message.added"},
{"mail.*.message.added", "mail.mesh.**", "mail.mesh.message.added"},
{"a.*", "*.b", "a.b"},
{"**", "a.**", "a.**"},
{"a.b", "a.c", ""},
{"a", "a.**", ""},
} {
got, ok := narrow(c.a, c.b)
if (c.want == "") == ok || got != c.want {
t.Errorf("narrow(%s, %s) = %q, %v; want %q", c.a, c.b, got, ok, c.want)
}
}
if got := uncovered([]string{"a.**", "a.b", "a.b", "c"}); !slices.Equal(got, []string{"a.**", "c"}) {
t.Fatalf("uncovered: %v", got)
}
}
+55
View File
@@ -0,0 +1,55 @@
package catalogue
import (
"fmt"
"strings"
)
// How loud a module's unhealthy condition is (novox/hq ADR 0355, extending ADR 0240).
//
// **A module may declare its unhealthy urgent.** A module whose failure costs people something they rely on
// (mail refused or stuck for everyone with a mailbox) says so with `unhealthy`, and its condition
// `module.<module>.<machine>.unhealthy` is raised urgent instead of a warning. **Urgent needs a why**, as an
// `upgrade` policy other than roll does: a person reading the catalogue sees what is lost while it is broken.
// The severities a module may declare for its unhealthy condition.
const (
// UnhealthyUrgent raises the module's unhealthy condition urgent, on every machine it runs on.
UnhealthyUrgent = "urgent"
// UnhealthyWarning is what a module's unhealthy condition is without the field, said out loud.
UnhealthyWarning = "warning"
)
// UnhealthyPolicy is what a module says about how loud its unhealthy condition is: `unhealthy` in its manifest
// (novox/hq ADR 0355).
type UnhealthyPolicy struct {
Severity string `json:"severity"`
// Why names what people lose while the module is broken. It is required: a module that says nothing of
// why it is loud is refused.
Why string `json:"why,omitempty"`
}
// unhealthyProblems refuses an `unhealthy` without a why, and any severity but urgent or warning (novox/hq ADR
// 0355 §2).
func (m Manifest) unhealthyProblems() []string {
if m.Unhealthy == nil {
return nil
}
var problems []string
switch m.Unhealthy.Severity {
case UnhealthyUrgent, UnhealthyWarning:
default:
problems = append(problems, fmt.Sprintf("unhealthy is %q or %q, not %q", UnhealthyUrgent, UnhealthyWarning,
m.Unhealthy.Severity))
}
if strings.TrimSpace(m.Unhealthy.Why) == "" {
problems = append(problems, "unhealthy says why: a module that declares how loud its failure is names "+
"what people lose while it is broken")
}
return problems
}
// UnhealthyIsUrgent is whether the module declares its unhealthy condition urgent (novox/hq ADR 0355 §1).
func (m Manifest) UnhealthyIsUrgent() bool {
return m.Unhealthy != nil && m.Unhealthy.Severity == UnhealthyUrgent
}
+53
View File
@@ -0,0 +1,53 @@
package catalogue
import (
"strings"
"testing"
)
// A module may declare its unhealthy urgent, with why; the module check refuses it without why and with any
// severity but urgent or warning (novox/hq ADR 0355).
func TestAnUnhealthyDeclaredUrgentIsUrgent(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"mail","version":"1",` +
`"unhealthy":{"severity":"urgent","why":"mail is refused or stuck for everyone with a mailbox"}}`))
if err != nil {
t.Fatalf("a manifest declaring its unhealthy urgent with why was refused: %v", err)
}
if !m.UnhealthyIsUrgent() {
t.Errorf("a manifest declaring its unhealthy urgent reads as not urgent: %+v", m.Unhealthy)
}
}
func TestAManifestWithoutUnhealthyIsNotUrgent(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"notes","version":"1"}`))
if err != nil {
t.Fatal(err)
}
if m.UnhealthyIsUrgent() {
t.Errorf("a manifest saying nothing of its unhealthy reads as urgent")
}
warning := Manifest{Module: "notes", Unhealthy: &UnhealthyPolicy{Severity: UnhealthyWarning, Why: "notes wait"}}
if warning.UnhealthyIsUrgent() || len(warning.unhealthyProblems()) != 0 {
t.Errorf("a warning said with why reads as urgent or is refused: %v", warning.unhealthyProblems())
}
}
func TestAnUnhealthyWithoutWhyIsRefused(t *testing.T) {
for _, why := range []string{"", " "} {
if problems := (Manifest{Module: "mail", Unhealthy: &UnhealthyPolicy{Severity: UnhealthyUrgent, Why: why}}).unhealthyProblems(); len(problems) == 0 {
t.Errorf("urgent with why %q was accepted", why)
}
}
if _, err := ParseManifest([]byte(`{"module":"mail","version":"1","unhealthy":{"severity":"urgent"}}`)); err == nil ||
!strings.Contains(err.Error(), "unhealthy says why") {
t.Errorf("a manifest declaring its unhealthy urgent without why parsed: %v", err)
}
}
func TestAnUnhealthySeverityOtherThanUrgentOrWarningIsRefused(t *testing.T) {
if _, err := ParseManifest([]byte(`{"module":"mail","version":"1","unhealthy":{"severity":"loud","why":"mail stops"}}`)); err == nil ||
!strings.Contains(err.Error(), `not "loud"`) {
t.Errorf("a manifest declaring its unhealthy loud parsed: %v", err)
}
}
+22
View File
@@ -362,6 +362,28 @@ var ControllerVerbs = []Verb{
"register": "\"true\" to register what it builds",
"older": "\"true\", with register: even though a newer build of the module is registered",
}, []string{"id"}, "register", "older")},
// novox/hq issue 686: a builder ran a branch's store tests for the first time in its pull request's merge check.
{Name: "try", Description: "Try a pushed branch or commit before its pull request (novox/hq issue 686): the build " +
"seat merges it into the repository's trunk and runs the repository's own merge-check.sh on the merge, in the " +
"toolchain containers and against the throwaway store and bus a pull request's merge check gets. Answers at " +
"once with an id; asked again with that id it waits up to 20 s and answers pass, fail or error, how long it " +
"took and the lines naming what failed — or that it is still waiting or running, to ask again. Sets no " +
"status on the forge, registers and builds nothing, runs no merge gate, counts as a check in the build queue " +
"(after deliveries and after every pull request's check, however long it waits) and is bounded as one. One " +
"try of a commit at a time — a second ask answers the first's id — and at most four waiting or running in " +
"all: a fifth is refused. Only for a repository of the mesh; the ref must be on the forge.",
Input: schema(map[string]string{
"repository": "the repository on the forge, owner/name; with ref",
"ref": "the branch or commit to try, as pushed to the forge",
"base": "the branch it is merged into (the repository's default branch when none is named). The " +
"merge-check.sh that runs is that branch's, never the ref's own: the same trust as a pull request's base",
"tests": "only the tests this `go test -run` pattern names, where the repository's merge-check.sh reads " +
"MESH_CHECK_TESTS; a script that does not runs whole",
"store-only": "\"true\": only the repository's store tests, where its merge-check.sh reads " +
"MESH_CHECK_STORE_ONLY",
"id": "instead of a repository: the id a try answered, to read what came of it",
}, nil, "store-only"),
Replaces: []string{"sh merge-check.sh", "docker run postgres"}},
// novox/hq ADR 0350 §5: asked by the delivery's owner when the forge says a pull request closed unmerged.
{Name: "forget-checks", Description: "Let go of every build a pull request's merge checks made for its merge in " +
"the artifact store's scratch namespace (novox/hq ADR 0350 §5): asked when it closed unmerged; a merged one's go " +
+43
View File
@@ -0,0 +1,43 @@
package catalogue
import (
"strings"
"testing"
)
// novox/hq ADR 0367 §2: the tool runner's module names, one by one, the event families a session may wait on.
func TestTheToolRunnerMayNameTheEventsASessionWaitsOn(t *testing.T) {
m, err := ParseManifest([]byte(`{"module":"node-tools","version":"1",` +
`"waits":["mesh-knowledge.reviewed","operator-channel.decided.claude-code"]}`))
if err != nil {
t.Fatal(err)
}
if len(m.Waits) != 2 || m.Waits[1] != "operator-channel.decided.claude-code" {
t.Fatalf("waits not read: %v", m.Waits)
}
}
// A family is named one by one, never by a wildcard, and only by the tool runner's module (ADR 0367 §2, §6):
// each refusal is said in the manifest's words, at parse, rather than at the composition of the bus's users.
func TestAWaitIsRefusedUnlessItNamesOneFamilyOfTheToolRunner(t *testing.T) {
for manifest, want := range map[string]string{
`{"module":"shop","version":"1","waits":["mesh-knowledge.reviewed"]}`: "only node-tools",
`{"module":"node-tools","version":"1","waits":["mesh-knowledge.*"]}`: "never a wildcard",
`{"module":"node-tools","version":"1","waits":["**"]}`: "never a wildcard",
`{"module":"node-tools","version":"1","waits":["mesh-knowledge.>"]}`: "never a wildcard",
`{"module":"node-tools","version":"1","waits":["*.reviewed"]}`: "never a wildcard",
`{"module":"node-tools","version":"1","waits":["reviewed"]}`: "does not name an emitter and an event",
`{"module":"node-tools","version":"1","waits":["mesh-knowledge..reviewed"]}`: "does not name an emitter and an event",
`{"module":"node-tools","version":"1","waits":["Mesh-Knowledge.reviewed x"]}`: "does not name an emitter and an event",
`{"module":"node-tools","version":"1","waits":["a.b","a.b"]}`: "twice",
} {
_, err := ParseManifest([]byte(manifest))
if err == nil {
t.Errorf("accepted: %s", manifest)
continue
}
if !strings.Contains(err.Error(), want) {
t.Errorf("%s refused for the wrong reason (want %q): %v", manifest, want, err)
}
}
}
+4 -2
View File
@@ -258,9 +258,11 @@ func TestBorrowedWordsAreLogged(t *testing.T) {
loggedUnworded.Clear()
t.Cleanup(loggedUnworded.Clear)
var out bytes.Buffer
writer := log.Writer()
writer, flags := log.Writer(), log.Flags()
log.SetOutput(&out)
t.Cleanup(func() { log.SetOutput(writer) })
// Without the logger's own date and time, which would hold "23:3" itself from 23:30 to 23:39 (issue 752).
log.SetFlags(0)
t.Cleanup(func() { log.SetOutput(writer); log.SetFlags(flags) })
// A time that changes each observation, and a hash-shaped word: one line, quoting neither.
for _, at := range []string{"23:32", "23:33", "23:34"} {
if _, err := k.Observe(t.Context(), Observation{Scope: ScopeMachine, ID: "ace", Kind: "test-clock", Machine: "ace",
+46
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5"
@@ -301,6 +302,51 @@ func (i *Inventory) Held(ctx context.Context) (map[string]string, error) {
return held, nil
}
// RegisteredHeld is Held narrowed to the build each module has registered (novox/hq issue 721): an artifact is kept
// only when its reference's digest is in the manifest the module table holds for it, which is what its machines are
// sent. After a gate failed and the mesh put the earlier build back (RestoreModule), the newest successful build is
// not what runs, and its artifacts are left out here, so nothing is compared with them.
func (i *Inventory) RegisteredHeld(ctx context.Context) (map[string]string, error) {
held, err := i.Held(ctx)
if err != nil {
return nil, err
}
rows, err := i.store.Pool().Query(ctx, `select name, coalesce(manifest::text, '') from module`)
if err != nil {
return nil, err
}
defer rows.Close()
registered := map[string]string{}
for rows.Next() {
var name, manifest string
if err := rows.Scan(&name, &manifest); err != nil {
return nil, err
}
registered[name] = manifest
}
if err := rows.Err(); err != nil {
return nil, err
}
return registeredOnly(held, registered), nil
}
// registeredOnly keeps of held each "<module>/<artifact>" whose reference's digest the module's registered manifest
// names.
func registeredOnly(held, registered map[string]string) map[string]string {
out := map[string]string{}
for key, ref := range held {
module, _, ok := strings.Cut(key, "/")
at := strings.LastIndex(ref, "@")
if !ok || at < 0 || len(ref) <= at+1 {
continue
}
if strings.Contains(registered[module], ref[at+1:]) {
out[key] = ref
}
}
return out
}
// BuiltAgainst is what each module's newest successful build stood on, as recorded — the build
// edges (ADR 0009). A module whose last build recorded no bases is absent, which is also what a
// module standing on nothing looks like: an edge the mesh has not derived is not an edge.
+143 -1
View File
@@ -61,12 +61,23 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
out := broker.Records{Assigned: map[string][]broker.Declared{}, People: map[string][]string{},
Interchangeable: map[string]bool{}}
assigned := map[string][]string{}
for _, n := range nodes {
out.Nodes = append(out.Nodes, n.Name)
modules, err := i.Assigned(ctx, n.Name)
if err != nil {
return broker.Records{}, fmt.Errorf("cannot read what %s runs: %w", n.Name, err)
}
assigned[n.Name] = modules
}
// What each restricting config value says on each machine running its module (novox/hq ADR 0363 §5): read
// once, for every consumer composed below.
grants, err := i.streamGrantValues(ctx, declared, nodes, assigned)
if err != nil {
return broker.Records{}, err
}
for _, n := range nodes {
out.Nodes = append(out.Nodes, n.Name)
modules := assigned[n.Name]
for _, module := range modules {
m, known := declared[module]
if !known {
@@ -89,6 +100,9 @@ func (i *Inventory) BusRecords(ctx context.Context) (broker.Records, error) {
d.KeyedReads = append(d.KeyedReads, broker.KeyedRead{Read: sr.Read, Key: sr.Key})
}
d.Holds = heldHere(d.Holds, holdings, n.Name, module)
// And the module streams it hears on this machine, as granted (novox/hq ADR 0363 §5): what a
// stream carries whole leaves EVENTS' consumes.
d.Hears, d.HearsAStream, d.Consumes = hearsOf(m, declared, n.Name, grants, d.Consumes)
out.Assigned[n.Name] = append(out.Assigned[n.Name], d)
if m.Instances == catalogue.InstancesInterchangeable {
out.Interchangeable[m.Module] = true
@@ -164,11 +178,15 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
Serves: m.Tools,
// And what it calls (novox/hq ADR 0152) — the console's `*`, nothing else's.
Invokes: m.Invokes,
// And the event families a session may wait on, the tool runner's alone (novox/hq ADR 0367 §2).
Waits: waitsOf(m, seats),
// And the state it keeps and reads (novox/hq ADR 0201).
State: bucketsOf(m),
Reads: m.Reads,
// And the logs it keeps (novox/hq ADR 0297).
Logs: logsOf(m),
// And the streams it owns (novox/hq ADR 0363).
Streams: streamsOf(m),
// And the tools its health asks (novox/hq ADR 0240): the machine's node-engine is granted them.
Checks: catalogue.HealthChecks(m),
// And whether it runs as an account of its own (novox/hq ADR 0259 §8).
@@ -203,6 +221,32 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
return d
}
// waitsOf is the event families a manifest says a session may wait on, each resolved as a consumed name is
// (novox/hq ADR 0121, ADR 0239): a seat's event when the seat says it, else the event of the module of that
// name. A seat whose event subjects end in a kind (a kinded bench) or in the asker (ADR 0259 §3) is waited on
// with that last name given — `decided.claude-code` — and the family says which it needs, so one given without
// it is refused where it is composed rather than granted a subject nothing publishes on.
func waitsOf(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaration) []broker.Wait {
var out []broker.Wait
for _, w := range m.Waits {
emitter, event, _ := strings.Cut(w, ".")
first, _, _ := strings.Cut(event, ".")
wait := broker.Wait{Name: w, Emitter: emitter, Event: event}
if s, isASeat := seats[emitter]; isASeat {
switch {
case s.Kinded && catalogue.SeatSays(s.Emits, first):
wait.Seat, wait.Last = true, "kind"
case s.NamedByCaller(first) && catalogue.SeatSays(s.Emits, first):
wait.Seat, wait.Last = true, "asker"
case catalogue.SeatSays(s.Emits, event):
wait.Seat = true
}
}
out = append(out, wait)
}
return out
}
// issuedTools are the tools through which a module answers the credentials its software issues (novox/hq ADR 0305).
func issuedTools(m catalogue.Manifest) []string {
var out []string
@@ -362,3 +406,101 @@ func (i *Inventory) DeclaredTrafficSeats(ctx context.Context) ([]broker.Seat, er
}
return out, nil
}
// streamsOf is the streams a module owns, as the bus holds them (novox/hq ADR 0363).
func streamsOf(m catalogue.Manifest) []broker.ModuleStream {
var out []broker.ModuleStream
for _, s := range m.Streams {
events := make([]string, 0, len(s.Events))
for _, e := range s.Events {
events = append(events, broker.WirePattern(e))
}
out = append(out, broker.ModuleStream{Module: m.Module, Name: s.BusName(), Events: events,
MaxAgeDays: s.Days(), MaxMiB: s.Cap(), Restricted: s.Grant != nil})
}
return out
}
// DeclaredStreams is every module stream the catalogue declares, registered modules assigned or not: a
// stream exists from registration, as a log does (novox/hq ADR 0363 §1).
func (i *Inventory) DeclaredStreams(ctx context.Context) ([]broker.ModuleStream, error) {
declared, err := i.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("cannot read the catalogue: %w", err)
}
var out []broker.ModuleStream
for _, m := range declared {
out = append(out, streamsOf(m)...)
}
return out, nil
}
// grantValues is each restricting config value on each machine running its module, by `<module>/<key>`.
type grantValues map[string][]any
// streamGrantValues reads, for every module owning a restricted stream, the config value that restricts it on every
// machine the module runs on: the mesh's layer under that machine's, the machine's winning (novox/hq ADR 0363 §5).
func (i *Inventory) streamGrantValues(ctx context.Context, declared map[string]catalogue.Manifest,
nodes []Node, assigned map[string][]string) (grantValues, error) {
out := grantValues{}
for _, n := range nodes {
for _, module := range assigned[n.Name] {
m, known := declared[module]
if !known {
continue
}
var settings []string
for _, s := range m.Streams {
if s.Grant != nil && !slices.Contains(settings, s.Grant.Config) {
settings = append(settings, s.Grant.Config)
}
}
if len(settings) == 0 {
continue
}
layers, err := i.SettingsFor(ctx, n.Name, module)
if err != nil {
return nil, fmt.Errorf("cannot read %s's settings on %s, which grant who hears its stream: %w",
module, n.Name, err)
}
for _, setting := range settings {
if v, set := settingValue(layers, setting); set {
out[module+"/"+setting] = append(out[module+"/"+setting], v)
}
}
}
}
return out, nil
}
// settingValue is one key's value over a machine's layers, the last that sets it winning. Pure.
func settingValue(layers []catalogue.Layer, key string) (any, bool) {
var value any
set := false
for _, l := range layers {
if v, ok := l.Values[key]; ok {
value, set = v, true
}
}
return value, set
}
// hearsOf is what a module hears of the module streams on one machine, and the consumes left for EVENTS
// (novox/hq ADR 0363 §5). Pure.
func hearsOf(m catalogue.Manifest, declared map[string]catalogue.Manifest, node string, grants grantValues,
consumes []string) ([]broker.Heard, bool, []string) {
reading := m
reading.Consumes = consumes
parts, touches, onEvents := catalogue.Hearing(reading, declared, node, func(provider, setting string) []any {
return grants[provider+"/"+setting]
})
var out []broker.Heard
for _, p := range parts {
h := broker.Heard{Stream: p.Stream.BusName(), Emitter: p.Emitter}
for _, pattern := range p.Patterns {
h.Filters = append(h.Filters, broker.ModuleStreamSubject(p.Emitter, broker.WirePattern(pattern)))
}
out = append(out, h)
}
return out, touches, onEvents
}

Some files were not shown because too many files have changed in this diff Show More