Compare commits

...
Author SHA1 Message Date
jschoubben 8831c74f70 Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:38:51 +02:00
jschoubben a96bd2ff38 Mark the packages whose tests read files outside their directory, so mesh-build-go never keeps their verdict (issue 1935) 2026-10-24 04:00:41 +02:00
jschoubben 1c50608246 Merge pull request 'Give route-proxy a build the fixed controller takes in, dropping the incident note #450 added (issue 1933)' (#452) from fix/1933-rebuild-route-proxy-2 into main
Reviewed-on: #452
2026-10-24 01:00:18 +00:00
jschoubben 760351b4ae Merge pull request 'The sweep keeps a fresh build's files, and a take-in holds every file before it records a build (issue 1933)' (#451) from fix/1933-sweep-keeps-fresh-builds into main
Reviewed-on: #451
2026-10-24 00:54:55 +00:00
jschoubben 899389c710 Reword route-provider's stdout line, so route-proxy's tree differs from the 01:52 build whose files are gone (issue 1933)
mesh/delivery delivered
2026-10-24 02:50:12 +02:00
jschoubben a203f605d4 Give route-proxy a build the fixed controller takes in, dropping the incident note #450 added (issue 1933) 2026-10-24 02:49:56 +02:00
jschoubben 0741880d31 Keep a fresh build's archives from the controller's own sweep (issue 1933)
mesh/delivery delivered
A sweep read what it keeps before what was ever recorded, so a take-in's
build recorded between the two reads, by a concurrent take-in off the
plans' hold, was eligible and let go of seconds after the take-in read
it; a new module's build is unkept until it is registered. Now the sweep
reads the records first and keeps what a build recorded within the hour
or an open plan names, and names every reference it lets go of; a take-in
holds every artifact in the store before recording, failing on one the
store lacks; and RecordBuild clears artifact_collected as RecordMirrored
does.
2026-10-24 02:44:20 +02:00
jschoubben fa734d3098 Merge pull request 'Give route-proxy a new build: the controller's sweep removed its last build's files from the store, so novox and ace cannot apply (issue 1933)' (#450) from fix/1933-rebuild-route-proxy into main
Reviewed-on: #450
2026-10-24 00:39:47 +00:00
jschoubben f13d42abb4 Give route-proxy a new build: the controller's sweep removed its last build's files from the store, so novox and ace cannot apply (issue 1933)
mesh/delivery delivered
2026-10-24 02:37:06 +02:00
jschoubben fdf26a1fe2 Merge pull request 'A module the catalogue no longer holds ends deleted instead of failing the plan; vendor/ goes; tests name verbs that exist (issues 1922, 1305, 1931)' (#449) from fix/1922-removed-module into main
Reviewed-on: #449
2026-10-23 23:47:05 +00:00
jschoubben 343e271ec7 Integration tests stop naming the verbs R4g removed (issue 1931)
mesh/delivery delivered
mesh/check checking 343e271ec794 on fdf26a1fe29f
2026-10-23 23:21:39 +02:00
jschoubben 05c872485a Delete the committed dependencies; builds resolve by go.sum (issue 1305) 2026-10-23 22:16:09 +02:00
jschoubben 12dd2bb3f0 A module the catalogue no longer holds ends deleted instead of failing the plan; the unsent hint names release-backlog (issue 1922) 2026-10-23 22:15:18 +02:00
jschoubben 5831eec770 Merge pull request 'R4g+R5 (mesh-controller): the controller asks no build, build-agent and the merge gate go, register by reference (design 47)' (#448) from feat/r4g into main
Reviewed-on: #448
2026-10-23 17:50:22 +00:00
jschoubben c9a7102bc5 Set aside a retired seat's consume the trunk catalogue still holds, and say it, so a group that retires a seat passes its check
mesh/delivery failed: its walk failed, carrying its own commit: a gate on a first machine (what it carried put back), a build, a machine
mesh/delivery-group group feat/r4g failed: a member failed
mesh/check checking c9a7102bc53b on 791786d5b87f | novox/mesh-controller#448 in group feat/r4g
2026-10-23 18:05:06 +02:00
jschoubben 67279199e9 Say the trunk rule as ADR 0534 §6 has it in the registration's comment
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: github.com/novox/mesh-controller/internal/broker.TestTheCataloguesDerivedSubjectsMeet: agreement_catalogue_test.go:130: judge…
2026-10-23 17:19:39 +02:00
jschoubben 9d2a9e62d1 Merge remote-tracking branch 'origin/fix/forge-requires-no-status' into feat/r4g 2026-10-23 17:18:58 +02:00
jschoubben b345d10bec Pin mesh-sdk at feat/r4g's head a2c813a6, its test fixture fix, with go.sum tidied
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: git.novox.be/novox/mesh-catalog/checks/config.TestTheUnmovedListHoldsNoModuleTheTrunksDoesNot: realtree_test.go:14: mesh-cata…
2026-10-23 17:06:08 +02:00
jschoubben 13842a7234 Grant the builder the build seats' streams' information and messages by name, so its take-over at start can tell an ask its key names is on its stream (issue 1910)
mesh/delivery-group group feat/r4g rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: g14: novox/mesh-catalog#464: checks/health/go.mod names mesh-sdk a9560890ee26, which is neither on mesh-sdk's trunk nor the group's…
2026-10-23 16:59:03 +02:00
jschoubben 4238318601 Pin mesh-sdk at feat/r4g's head a9560890, re-vendor it and capture its conformance events there (R4g) 2026-10-23 16:54:07 +02:00
jschoubben 758c8e84e5 Bound the build seats' workers' deliveries, so an ask no run ends is offered at most five times even when the holder that says it is poison dies too (issue 1907) 2026-10-23 16:42:13 +02:00
jschoubben 9222575ae9 Vendor the dependencies again for R4g's go.mod (mesh-sdk 9205222a, the mesh-host replace as before), so build-agent compiles the controller offline in the group's first tier (issue 1305) 2026-10-23 16:38:31 +02:00
jschoubben c5b8fca8bd Revert "Delete vendor/ and the route proxy's: the build module fetches the modules, the root image builds with -mod=mod (issue 1305, C6)"
This reverts commit 9c4bf1f08b.
2026-10-23 16:37:40 +02:00
jschoubben ff543d3def Take the watchman's grant on the build seats back: a waiting ask has no pulse, a taken one is read from the steps (issue 1908) 2026-10-23 16:35:28 +02:00
jschoubben ef211faefb Prove the registration by hand reads back as a build and is held by the sweep, and name the store check as the test's limit (issue 1888, C8b) 2026-10-23 16:34:29 +02:00
jschoubben 3c247e80af Fix the view grant test's consumer subjects (issue 1887 A8) 2026-10-23 16:32:43 +02:00
jschoubben bb0d5e17db Grant the watchman the build seats' accept subjects and the view the seat workers' information, each by name, so the backlog and the waiting-ask pulse read live (issue 1887 A8) 2026-10-23 16:32:32 +02:00
jschoubben 24ad859629 Record the checksums of mesh-sdk feat/r4g 9205222a for the group's modules (R4g) 2026-10-23 16:26:11 +02:00
jschoubben 5ccc87f4d0 Capture mesh-catalog at feat/r4g (2246ccc4) with the templates its manifests name and its predefined.json, and lay the copy out as a repository's root
The catalogue's manifests moved their content into template files and fall back to predefined keys, so a copy of
module.json alone no longer passes the catalogue's own check; the bus's Dockerfile is gone from the catalogue.
2026-10-23 16:24:14 +02:00
jschoubben 68c1220e87 Pin mesh-sdk at feat/r4g's head 9205222a, the merge of lanes A, B, S8 and F1, and capture its conformance events there
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:22:07 +02:00
jschoubben 5b5cd6160a Merge branch 'feat/r4g-asks' into feat/r4g 2026-10-23 16:20:32 +02:00
jschoubben d7d70cd539 Merge branch 'feat/r4g-assigner' into feat/r4g 2026-10-23 16:20:32 +02:00
jschoubben df70ca43a7 Take the controller's publish grants on node-build-agent and mesh-build-machine away, with the cancelled sets and the redelivery of its own asks: it asks no build seat (ADR 0525 §10, C3a) 2026-10-23 16:13:43 +02:00
jschoubben f60c742d93 Prove the grant through the send of the bus's machine (overTheBus.declare), so removing the write from the live path fails the test (issue 1394, review round 1 item 4) 2026-10-23 16:13:42 +02:00
jschoubben 2db5dcdcb4 Write the hand act of register --reference after the registration was made, not before, and test the why and the record (issue 1888, C8b) 2026-10-23 16:12:16 +02:00
jschoubben d9cdb3615d Test that the builder's record names its key and fingerprint after a plan that moves it (issue 1851, C5 criterion) 2026-10-23 16:11:07 +02:00
jschoubben dec3b83e66 Test that the builder, a build module and the controller are taken from the builder's keys and nothing reaches node-build-agent (issue 1851, C2 criterion) 2026-10-23 16:11:07 +02:00
jschoubben 2e36a63096 Pin mesh-sdk at feat/r4g's head c3915ffb, the merge of lane B's layouts, so every r4g module builds against one SDK
go.sum gains its lines from go mod tidy once mesh-sdk feat/r4g is pushed.
2026-10-23 16:01:15 +02:00
jschoubben eb8286fc68 Merge feat/r4g-assigner into feat/r4g: lane B's grants bucket and its writer beside lane C's deletions
The controller's readable states, its bucket lists and the composed golden file keep lane C's removal of the
checks-running and plans-building buckets and the build agent's check subjects, and add lane B's grants bucket.
2026-10-23 15:59:52 +02:00
jschoubben 5540fd4f85 Find the catalogue's repository by its URL or seat source, and fail a plan whose catalogue repository is nowhere, so a take-in from a key no longer waits for ever (issue 1902) 2026-10-23 15:57:30 +02:00
jschoubben e419d0182f Delete the builder, the build asks, the build queue and the merge gate: the builder's keys are the only way a module is built (issues 1851, 1299, 1318; C2b-C5, C10)
The controller asks no build of anybody (ADR 0501 §5, ADR 0525 clause 6, ADR 0503 §3). Every module of a walk is taken from the
builder's done key; a key not done yet waits, named in the plan, and never fails it.

- C2b/C2c: internal/builder and cmd/mesh-controller's asks go (askABuild, onlyTheBuildersOwn, buildOneAsked, takeIn from an
  outcome, buildAgentsOwn, the builder's say and its fallback, the daemon hearing an outcome). The server follows the retired
  seats' built subjects only to let go of them, because the installer's carried user list names them.
- C2d: replay, queue, cancel, clear, kill, pause, resume, delivery-check, check-here and merge-gate are refused by name.
- C3: the check queues (CHECKS_READY, CHECKS_OTHER) and their workers, the checks-running and plans-building buckets and
  their grants, the build queue and its hold (plans_building.go, queue.go, link/holds.go, builds_nats.go, checks_nats.go).
- C4: merge_gate*.go, the pull request's repo-check path, the Checked event and its layers; a pull request asks the
  controller nothing but to tell mesh-config what its head declares.
- C5: the builds bucket is written only by key_takein.
- C10 (help and refusals): the terminal's help lists only what is served.

Integration tests of the plan's tiers, sends, stops and retries now run on a stand-in builder with a done key for every
module (walk_keys_scaffold_integration_test.go); the tests of the asks, the queue, the merge gate and the replays are deleted.
2026-10-23 15:55:15 +02:00
jschoubben 5ca4f14479 Derive a worker per kind for the build seats and grant their users to put asks, so a build node pulls delivery, check and recheck in order and a session cannot ask (issue 1887 A6, ADR 0525 §2) 2026-10-23 15:54:45 +02:00
jschoubben c43883910d Write grants.<module>.<fp> once the bus's user list carrying a module's manifest at that fingerprint is sent, so the assigner's sixth rule reads that its credential exists (issue 1394, subtask C9b; ADR 0528 clause 2, ADR 0429 §6) 2026-10-23 15:39:40 +02:00
jschoubben 548681cc4e Declare the controller's grants bucket, written by the controller alone and readable through mesh-controller.grants, so the assigner can read its sixth rule's key (issue 1394, subtask C9a; ADR 0528 clause 2); the SDK pin moves to the layouts' commit, captured beside 2026-10-23 15:39:40 +02:00
jschoubben 9c4bf1f08b Delete vendor/ and the route proxy's: the build module fetches the modules, the root image builds with -mod=mod (issue 1305, C6) 2026-10-23 15:14:58 +02:00
jschoubben 6300a5b31c Delete merge-check.sh, check-nested-modules.sh, the store-tests list and RELEASE-57: the checker runs the build module's steps, nested modules by convention (issue 1294, C7a and C7b) 2026-10-23 15:14:01 +02:00
jschoubben 505694c424 Register a module at artifacts the store holds, by hand, recorded as a build and a hand act, so the controller can be delivered without the builder (issue 1888, C8a and C8b) 2026-10-23 15:13:17 +02:00
jschoubben 72186815e3 Delete the build-agent module and the builder program: the build module builds, the checker checks (issue 1851, C2a) 2026-10-23 15:07:57 +02:00
jschoubben 526eed99fa Drop the toolchain image from the facts and the dependency edges: no module is compiled in one any more (issue 1851, C1b) 2026-10-23 15:07:29 +02:00
jschoubben b506852388 Move the Go build source reader out of internal/builder so the builder can go (issue 1851, C1a) 2026-10-23 15:06:31 +02:00
mesh-tools 791786d5b8 Merge pull request 'Wait out a resolver that does not answer in a build's registry step, and say it was the resolver (issue 1748)' (#447) from fix/1748-registry-wait-retries-a-lookup into main 2026-10-23 11:20:15 +00:00
jschoubben 0b312218f9 Wait out a failed lookup of the registry's name in a build, and say it was the resolver (issue 1748)
mesh/delivery delivered
mesh/check pass 0b312218f90f on c834e4f82137 | unit 1762 run, 0 kept, 4 skipped | integration 3040 run, 0 kept, 16 skipped | over bound: integration (…
2026-10-23 13:03:17 +02:00
mesh-tools c834e4f821 Merge pull request 'Keep every refused build by its id, so a plan never settles a refused build built and sends the old version (issue 1713)' (#446) from fix/1713-keep-every-refusal into main 2026-10-23 10:58:08 +00:00
jschoubben 149f89e8f9 Keep every refused registration by its build id, so a Retry never settles it built
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792753090890815254 (novox/mesh-controller at c834e4f8 into ma…
mesh/check pass 149f89e8f9bf on 0173d9c70e59 | unit 1759 run, 0 kept, 4 skipped | integration 3037 run, 0 kept, 16 skipped | over bound: integration (…
A build heard and recorded and not registered kept its refusal only when a
plan was open or the plans were held. A tier's first failure fails its plan,
so later refusals were lost; a Retry then settled those builds built from
their records, sent the modules' old versions and reported the plan done.
A build without a readable manifest never reached its plan at all.

Keep the refusal in the store and the process whatever the plans' state, and
fail the asking plan by the build's id when the module is not known.

Issue 1713.
2026-10-23 12:13:33 +02:00
jschoubben b2af3d654f Trust a trunk whose whitelist adds the operator's own login, never an agent's (ADR 0534 §2, §6)
The gitea module keeps the operator's login beside mesh-tools on every main so
the operator can merge by hand; an agent's account there is still refused.
2026-10-23 12:10:11 +02:00
jschoubben 7676e67c5e Trust a module's trunk for who can merge into it, not for a required status (ADR 0534 §6)
The forge requires no status to merge any more (ADR 0534 §1); a trunk is
trusted when it takes no push, its merge whitelist names mesh-tools alone and
no administrator merges past its rule, so only mesh-delivery lands a commit
there, on a passing mesh/check and the operator's Approve. The old rule failed
the 11:51 delivery plan once mesh-catalog's status was unticked.
2026-10-23 12:02:00 +02:00
mesh-tools 0173d9c70e Merge pull request 'Keep the admin address's query in a test database's address, so store tests reach the checker's socket store (issue 1889)' (#445) from fix/1889-identity-store-ready into main 2026-10-23 07:37:06 +00:00
mesh-tools 89abce5df9 Merge pull request 'Judge each machine on a plan's last send and resume a plan lost to silence with plans retry, so a resumed plan is not failed by its own old sends (issue 1886, ADR 0531)' (#444) from fix/1886-resume-times into main 2026-10-23 07:26:31 +00:00
jschoubben 185d0294eb Keep the store address's query in every test database's address, so store tests reach a socket-addressed store (issue 1889)
mesh/merge-gate head 185d0294ebd4 merged into trunk 89abce5df955 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; the ch…
mesh/repo-check head 185d0294ebd4 merged into trunk 89abce5df955 — pass: its merge-check.sh passed
mesh/delivery covered after its stop: a later merge that contains it was delivered: novox/mesh-controller@0b312218f90f (merged as 791786d5 into main, wal…
mesh/check pass 185d0294ebd4 on 89abce5df955 | unit 1756 run, 0 kept, 4 skipped | integration 3034 run, 0 kept, 16 skipped | over bound: integration (…
The checker's integration store answers on a unix socket named in the query of MESH_TEST_POSTGRES
(?host=<socket dir>&sslmode=disable). The identity, inventory and licences fixtures cut the address at
its last slash and appended ?sslmode=disable, dropping host=, so pgx dialled its default socket where no
server is: every store test waited out its 20 s readiness bound, each package ran into go test's
10-minute timeout, and the verdict named only the first failure,
identity.TestACertificateChainsToTheMeshsOwnAuthority.

store.DatabaseForTest now builds that address for every fixture, keeping the query as the template raise
already did; identity's fresh is ForTest rather than a second copy. The readiness bound and its error,
which wraps the driver's cause, are unchanged.
2026-10-23 08:55:50 +02:00
jschoubben 4d4d51eb01 Pin the resume of a lost walk's refusals beside a newer or a related plan, and cite ADR 0360 §7 and ADR 0531 (issue 1886)
mesh/merge-gate head 4d4d51eb017f merged into trunk 219fdcd7f00c — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; the ch…
mesh/repo-check head 4d4d51eb017f merged into trunk 219fdcd7f00c — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-23 08:10:08 +02:00
jschoubben b7d996878b Judge a walk's machine on its last send, enter a resumed tier anew, and resume a walk lost to silence (issue 1886)
R4e, resumed at tier 1 after a stop of five hours, named ace and shanks silent within seconds of tier 1's send: the
walk judged each machine on every send it had made it, tier 0's of 00:23 too, held against tier 1's templates, and
measured the bound from it. A send carries the whole declaration, so the last one is what a machine is judged on.
The resume kept the tier's entry of before the stop, so the plan read waiting for 5h48m. And plans retry refused a
walk failed only by the gate's silence, though its machines had since applied.
2026-10-23 08:02:11 +02:00
mesh-tools 219fdcd7f0 Merge pull request 'Resume a delivery plan stopped by hand with plans retry, so a stop is not the end of what its tier waited on (issue 1886)' (#443) from fix/1886-retry-stopped into main 2026-10-23 05:40:52 +00:00
jschoubben 55ba7f2cb8 Hold every refusal of the stopped path in the test ADR 0523 names: a newer open plan of the branch, a newer plan of a build not sent (issue 1886)
mesh/merge-gate head 55ba7f2cb83c merged into trunk 296e11d499cb — pass: builds mesh-controller → novox; no bus step; the change touches no module of the m…
mesh/repo-check head 55ba7f2cb83c merged into trunk 296e11d499cb — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check checking 55ba7f2cb83c on 219fdcd7f00c
2026-10-23 02:44:58 +02:00
jschoubben 102f8dc267 Judge a resume only by the builds its tier holds, and ask build-agent's own of build-agent (issue 1886, review round 1)
mesh/merge-gate head 102f8dc2674e merged into trunk 296e11d499cb — pass: builds mesh-controller → novox; no bus step; the change touches no module of the m…
mesh/repo-check head 102f8dc2674e merged into trunk 296e11d499cb — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-23 02:34:59 +02:00
jschoubben 296e11d499 Merge pull request 'Ask build-agent until the builder says it writes references, so a plan never waits on a key nothing can reference (issue 1886, ADR 0522)' (#442) from fix/1886-builder-says-references into main
Reviewed-on: #442
2026-10-23 00:27:00 +00:00
jschoubben eed4976483 Resume a walk stopped by hand with plans retry, so a stop is not the end of what its tier waited on (issue 1886) 2026-10-23 02:05:16 +02:00
jschoubben bbadbb2af0 Judge the ask and the settle by one say of the builder (issue 1886, merge check hang)
mesh/merge-gate head bbadbb2af021 merged into trunk 289bbf68e4f9 — pass: builds mesh-controller → novox; no bus step; the change touches no module of the m…
mesh/repo-check head bbadbb2af021 merged into trunk 289bbf68e4f9 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/1886-builder-says-references delivered: every member is delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The settle read the say from the builder's bucket while the ask read it through
builderSaysReferences; where they differed (a test's say against a bus with no
bucket) a module was handed back and taken from its key again at every turn,
and the look's loop never ended.
2026-10-23 01:59:18 +02:00
jschoubben 1414ed7f7b Judge an ask by the builder's say the walk read, never a second read (issue 1886, review round 1)
mesh/delivery-group group fix/1886-builder-says-references checking: 0 of 2 member(s) ready
mesh/merge-gate building on ace since 01:38
mesh/delivery superseded: a newer head of the same pull request
A second read that failed, or that turned true in between, refused the ask and
failed the whole plan.
2026-10-23 01:03:19 +02:00
jschoubben f67342f0d8 Ask build-agent for the build modules that write references, whatever the builder says (issue 1886, ADR 0522 §6)
They push and answer each reference, so a key they write themselves can never
carry their own delivery; until R4g build-agent delivers them as it does the
builder.
2026-10-23 00:53:25 +02:00
jschoubben fae7710bed Ask build-agent for every module until a running builder says it writes references (issue 1886)
R4e's own delivery plan waited for ever in tier 1: the controller took every
module but build-agent's own from a referenced builder key, and only R4e's
builder and build modules, delivered by that same plan, write references.
The builder now says so in its bucket of instances; until it does, a walk asks
build-agent as before R4e, and an open walk waiting on a key the running
builder cannot reference is judged again at the next look.
2026-10-23 00:47:15 +02:00
mesh-tools 289bbf68e4 Merge pull request 'R4e (mesh-controller): deliver from the builder's keys, ask build-agent only for its own modules, follow the forge's protection for checks, announce the controller's states' (#441) from feat/r4e into main 2026-10-22 22:12:58 +00:00
mesh-tools f2a4363a1a Merge pull request 'Clone from the forge again when the resolver does not answer, so a passing lookup failure does not fail a build (issue 1773)' (#440) from fix/1773-clone-retries-a-lookup into main 2026-10-22 19:44:36 +00:00
jschoubben d51324df45 Format the help's test (issue 1851)
mesh/repo-check head d51324df4518 merged into trunk f2a4363a1a77 beside mesh-catalog#457@896bd883 — pass: its merge-check.sh passed
mesh/merge-gate head d51324df4518 merged into trunk f2a4363a1a77 — pass: group feat/r4e composed: every machine composes with the change as it did without …
mesh/delivery covered after its stop: a later walk that contains it was delivered: plan-1792707174627079192 (novox/mesh-controller at 289bbf68 into main,…
mesh/delivery-group group feat/r4e delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:989 | d51324df4518 on f2a4363a1a77 | unit 1 run, 0 kept | integration 0 run, 0 kept…
2026-10-22 21:20:06 +02:00
jschoubben 58512a9c6d Pin mesh-sdk at the R4e member's final head 104533e8 and vendor it (R4e E)
The SDK's S1 and S2 review fixes landed after 875c680d.
2026-10-22 21:19:46 +02:00
jschoubben 1fd833bfe8 List the store tests R4e adds, and drop those of the hand builds and the stopped checks it deleted (issue 1851, issue 1299)
The merge check refused the eight deleted tests as listed and not run.
2026-10-22 21:18:58 +02:00
jschoubben fd4d26b42c Keep TestReplay867 off the build node's real free space (issue 1834, note 12)
mesh/merge-gate head fd4d26b42cd7 merged into trunk 96441ea28a84 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head fd4d26b42cd7 merged into trunk 96441ea28a84 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The check's floor of free space reads the disk the test runs on, so
every mesh-controller check failed while its build node had under
20 GiB free, whatever the change (seen in the try of issue 1773's fix).
The replay stubs the free space, as internal/builder's tests do.
2026-10-22 21:17:42 +02:00
jschoubben f9014452d3 List register and switch in the help, and none of the removed commands (issue 1851 A11a, A11b)
The help still offered build, rebuild and replay --register, each now
refused as removed, and did not name the terminal's register.
2026-10-22 21:10:20 +02:00
jschoubben 61449249a8 Name build-agent's own as the builder's module, build-agent and mesh-tools (ADR 0520 §1, issue 1851)
ADR 0520 narrowed after its review: mesh-tools-go builds from its own
directory on a pinned base and has its key from mesh-build-image, and
mesh-tools-python is no module, so the set is named, not derived from
the toolchain bases.
2026-10-22 21:08:47 +02:00
jschoubben 61f7e89ad5 Ask a repository check whenever the trunk requires it, beside mesh/check (issue 1299 C1, ADR 0513 §4)
A trunk requiring both contexts (hq) matched mesh/check first and had
its own merge-check.sh never asked.
2026-10-22 21:07:55 +02:00
jschoubben 9713ceb84d Ask build-agent for itself and its toolchain images, never a builder key (ADR 0520, issue 1851)
The builder writes no key of build-agent or of the toolchain bases its
builds stand on, so a walk moving one would hold its tier for ever.
Until R4g they are build-agent's own, as the builder's module is; the
bases are read from the toolchains, not copied.
2026-10-22 21:06:33 +02:00
jschoubben 93fc2f48a1 Name a retry's modules still waiting for the bus, read as the plan it resumes (issue 1851 A11a, issue 1403)
With the hand rebuild gone, a retry of a plan failed past the bus's bound
read the waiting modules while the plan still said failed, named none,
and answered "asked" with nothing after it.
2026-10-22 21:05:01 +02:00
jschoubben acaf8830ec Try a forge clone again when its name lookup fails (issue 1773)
A build or check failed now and then on 'Could not resolve host:
novox.internal' between clones that resolved the same name, and its
whole delivery failed with it. A clone whose lookup failed reached
nothing, so it is cloned again with a growing pause for up to 60 s,
each retry said in the build's log as the resolver not answering;
past the bound it fails saying so. Any other git failure fails at
once, as before. Covers a module's own clone, an artifact's context
and a check's repositories.
2026-10-22 20:56:46 +02:00
jschoubben d5d53893a4 Send the bus its user list before a starting controller writes (issue 1800)
A new controller's grants reached the bus only after its gate, so the
buckets it newly writes were refused until then. At start it now sends
the bus's machine the user list alone, every build there kept.
2026-10-22 20:51:08 +02:00
jschoubben a0b2d98002 Merge main (upgrade safety's declared packages, issue 1840) and pin the R4e mesh-sdk member's final head 875c680d, re-vendored here and in modules/route-proxy; the controller announces declared-packages readable (issue 1790, D1; E) 2026-10-22 20:44:59 +02:00
jschoubben bc80683b9e Count the work the take-ins start in the background, so a test's mesh closes only once that work has ended (issue 1851, A5-A7; issue 1790, D3) 2026-10-22 20:41:30 +02:00
mesh-tools 96441ea28a Merge pull request 'Keep each node's declared packages in a state the pacman module reads (upgrade safety, issue 1840, ADR 0507 §5)' (#439) from feat/upgrade-safety into main 2026-10-22 18:34:19 +00:00
jschoubben 420ae3d0e7 Test the group check's answer without swapping standard output, and wait for a take-in to forget a new module's wait (issue 1299, C2; issue 1851, A5) 2026-10-22 20:19:57 +02:00
jschoubben 7c0e1d223f Serve a removed verb as its refusal, not as a verb this build is behind on, and fit the tests of the verbs and the stopped checks (issue 1851, A11a; issue 1299, C1; issue 1318, C3) 2026-10-22 19:38:02 +02:00
jschoubben a10ef8995b Vendor what go.mod pins after the rebase on main (issue 1790; the SDK's 1833 fix returns with E's re-pin) 2026-10-22 19:05:21 +02:00
jschoubben dfc2f2aa98 Take a walk's key in again when it was recorded at an earlier merge, so the module is registered at the commit the walk carries before it is built (issue 1851, A6) 2026-10-22 19:05:05 +02:00
jschoubben ab33f015ec Say again what the controller keeps at every push, so a word refused before its grant is kept once a push carries it (issue 1790, D1) 2026-10-22 19:05:05 +02:00
jschoubben 791c2a3d48 Refuse a removed verb by name before reading its arguments, as the SDK member's table now offers none (issue 1851, A11a; issue 1318, C3) 2026-10-22 19:05:05 +02:00
jschoubben d36e0ea278 Announce the controller's states, keep every module's word, and grant a read only where its owner announced the state readable; the copied list of the controller's mirrors goes (issue 1790, D1-D4, ADR 0495, ADR 0513 §3) 2026-10-22 19:05:04 +02:00
jschoubben b6ed18197b Vendor the R4e mesh-sdk member's head with the states announcement (issue 1790; re-pinned to the final head in E) 2026-10-22 19:05:04 +02:00
jschoubben 45d091778c Refuse delivery-plan and forget-checks by name and take a merge's warrant from the teller alone, no longer mesh-delivery's (issue 1318, C3, ADR 0450 §8) 2026-10-22 19:05:04 +02:00
jschoubben 3a1937914e Ask build-agent no composed check of a delivery group, and answer that the checker judges it (issue 1299, C2, ADR 0503 §3) 2026-10-22 19:05:04 +02:00
jschoubben 25fdf9920e Read a commit as the same as itself however short, and fit the pending assignments' tests to a mesh that asks no build (issue 1851, A9, A10) 2026-10-22 19:05:04 +02:00
jschoubben 323e1270cc Ask no pull request check of build-agent but the own merge-check.sh of a repository whose trunk requires mesh/repo-check alone, and say every verdict with no gate layer (issue 1299, C1, ADR 0503 §3, §5) 2026-10-22 19:05:04 +02:00
jschoubben e7e23c57cc Read a module whose every artifact names its paths from those paths and its manifest alone, so a change elsewhere in its directory moves nothing (issue 1758, B1, ADR 0499 §5) 2026-10-22 19:05:04 +02:00
jschoubben a8387ebad8 Name, before the switch, which modules it moves: the held build's digests against the builder's done key, images and packages said as what cannot be compared (issue 1851, A13) 2026-10-22 19:05:04 +02:00
jschoubben 1963d84a7a Refuse, in the one function every ask goes through, an ask of build-agent for any module but the builder's own (issue 1851, A12, ADR 0453 §5) 2026-10-22 19:05:04 +02:00
jschoubben b71425867f Register a module from a repository the catalogue does not build at the controller's terminal alone, from its done trunk key, naming exactly its repository and path (issue 1851, A11b, ADR 0501 §4) 2026-10-22 19:05:04 +02:00
jschoubben c7039d81e1 Refuse build, rebuild and replay --register by name, naming what replaced them, and delete the hand builds (issue 1851, A11a, ADR 0501 §5) 2026-10-22 19:05:04 +02:00
jschoubben f76da2ce9c Ask the checker for a try of a branch head, not build-agent, and read its word in today's answer (issue 1851, A11, ADR 0501 §5) 2026-10-22 19:05:04 +02:00
jschoubben 7f6d418205 Mark the walks the integration tests drive around the one ask left as the builder's own, and expect no ask of a moved module (issue 1851, A5, A6) 2026-10-22 19:05:04 +02:00
jschoubben 0aec975d19 Keep an assignment of a module whose key is coming pending on that key and ask no build; nothing asks a request again at start (issue 1851, A10) 2026-10-22 19:05:04 +02:00
jschoubben 9da79ef8c3 Move no module whose fingerprint is unchanged, and let a key's build stand on the held build-agent build of the same digests at the switch (issue 1851, A9) 2026-10-22 19:05:04 +02:00
jschoubben 7743bd0d4b Delete the walk's reuse of a check's build: a walk takes keys and asks build-agent only for the builder's own module (issue 1851, A8) 2026-10-22 19:05:04 +02:00
jschoubben 62268cf43d Take a walk's modules from the builder's done keys and ask no build but the builder's own: a key building or unreferenced holds the tier and says so, a failed key fails the walk in its words (issue 1851, A6, A7) 2026-10-22 19:05:04 +02:00
jschoubben ea3c8ad7f6 Register a new module from the builder's first done key and ask no first build; its condition reads the key and names a manifest word this controller does not read (issue 1851, A5a, A5b) 2026-10-22 19:05:04 +02:00
jschoubben d984f0a55d Take a module in from the builder's done key of its trunk fingerprint: one record of the key's artifacts, its manifest judged against the catalogue trunk's predefined keys (issue 1851, A3, A4b, A4c) 2026-10-22 19:05:04 +02:00
jschoubben f00b6aff8b Judge a recorded build, not build-agent's outcome, and read a registration's warrant from the merge a key names, so a done key registers under the same rules (issue 1851, A4a) 2026-10-22 19:05:04 +02:00
jschoubben eff0d5cbf4 Vendor the R4e mesh-sdk member's head for its shared launcher rule and the manifest word paths (issue 1851; re-pinned to the final head in E) 2026-10-22 19:05:04 +02:00
jschoubben 1ed7e34155 Read the builder's keys, the indexer's values by commit and its manifests, so the controller can take a module from a done key (issue 1851, A1, A2) 2026-10-22 19:05:04 +02:00
jschoubben abe356a262 Merge main and take the SDK at the milestone's mesh-sdk head 8892a117, which holds main's SDK and the milestone's (issue 1840)
mesh/merge-gate head abe356a262e8 merged into trunk 41c96287578e — pass: group feat/upgrade-safety composed: every machine composes with the change as it d…
mesh/repo-check head abe356a262e8 merged into trunk 41c96287578e beside mesh-catalog#450@86ba54b0 — pass: its merge-check.sh passed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792694058124013462 (novox/mesh-controller at 96441ea2 into ma…
mesh/delivery-group group feat/upgrade-safety delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | abe356a262e8 on 41c96287578e | unit 3372 run, 0 kept, 7 skipped | integration…
2026-10-22 18:14:14 +02:00
mesh-tools 41c9628757 Merge pull request 'Pin mesh-sdk at the fix that refuses an unresolvable machine alone (issue 1833)' (#437) from fix/1833-one-machine-alone into main 2026-10-22 16:10:20 +00:00
jschoubben 3e3400f510 Take the SDK at the milestone's mesh-sdk head f8b8f3a6, which the delivery group's members must pin exactly (issue 1840) 2026-10-22 18:07:38 +02:00
jschoubben e022a3fd6f issue 1840/S3: look the declared packages up by name in the mirror's integration test, since every node also runs the mesh's own modules and their packages 2026-10-22 17:52:40 +02:00
jschoubben 684a8783e5 Merge remote-tracking branch 'origin/main' into feat/upgrade-safety 2026-10-22 17:40:00 +02:00
mesh-tools 11e364bd43 Merge pull request 'Hold a first merge whose new module the running build-agent refuses for an unknown manifest word, name it, and ask it again when build-agent is delivered (issue 1173)' (#438) from fix/1173-held-first-build into main 2026-10-22 15:28:11 +00:00
jschoubben 1e9b87e03a Hold every refused module of a first merge and ask each again by its own path (issue 1173)
mesh/merge-gate head 1e9b87e03acd merged into trunk b8f77577fed2 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 1e9b87e03acd merged into trunk b8f77577fed2 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
A held merge was asked again by the rule of a first merge, which finds nothing once a sibling module of the
same merge registered, so it stayed held forever; and a second refusal in a merge already held fell through to
a failed build. The hold now keeps a list of module, path and word, a refusal in a held waiting merge joins it,
and the delivery of build-agent asks each held path again on the merge's Approve warrant.
2026-10-22 16:50:06 +02:00
jschoubben 45a09f90e3 Carry a held merge's new announcement into its record (issue 1173)
HoldMerge replaced the event on the flag only; the record kept the old one.
2026-10-22 16:33:28 +02:00
jschoubben 4538d05732 Ask a held first build again when a delivery of build-agent ends (issue 1173)
ADR 0482 section 5: the build waiting for a build-agent that reads its manifest is asked once the plan delivering it ends.
2026-10-22 16:19:45 +02:00
jschoubben 980d396d40 Keep a first merge whose manifest word the running build-agent does not read, and name it (issue 1173)
ADR 0482 section 5: a merged manifest the running reader refuses waits for the newer reader, never failed.
2026-10-22 16:18:39 +02:00
mesh-tools b8f77577fe Merge pull request 'Say a delivery's start and end with the machines its merges declare, and word a needed new head as its author's (ADR 0505)' (#436) from feat/quiet-deliveries into main 2026-10-22 14:01:22 +00:00
jschoubben 28a278c146 Take mesh-sdk at bd7731f: a machine that does not resolve keeps its name and its address in the mesh as well as its tunnel (issue 1833)
mesh/repo-check head 28a278c1462a merged into trunk 11e364bd439f — pass: its merge-check.sh passed
mesh/merge-gate head 28a278c1462a merged into trunk 11e364bd439f — pass: group fix/1833-one-machine-alone composed: every machine composes with the change …
mesh/delivery delivered
mesh/delivery-group group fix/1833-one-machine-alone delivered: every member is delivered
mesh/check failed: g14: go.mod of novox/mesh-controller#437 cannot be read to place the mesh-sdk its tests read: nats: no responders available for req…
2026-10-22 15:27:27 +02:00
jschoubben c34ab0e43a Take mesh-sdk at c154bde: a machine that does not resolve keeps its place on the private network, so it alone is refused (issue 1833)
One machine whose modules could not be resolved left the private network uncomposable when it was the hub, and every
send to every machine was held. The SDK now keeps a refused machine at the place the controller already holds for it.
2026-10-22 15:04:47 +02:00
jschoubben 771cc24fcb Take the SDK whose process can delete a key (issue 1840/S1) 2026-10-22 14:37:18 +02:00
jschoubben b3786cc67e Merge remote-tracking branch 'origin/main' into feat/upgrade-safety
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-22 14:25:33 +02:00
jschoubben b7d03b6897 Word a delivery that needs its author's new head as its author's, with no Stop (ADR 0505 §5)
mesh/repo-check head b7d03b68974d merged into trunk 8f2d22db0caa beside mesh-catalog#444@09e749bc — pass: its merge-check.sh passed
mesh/merge-gate head b7d03b68974d merged into trunk 8f2d22db0caa — pass: group feat/quiet-deliveries composed: every machine composes with the change as it…
mesh/delivery delivered
mesh/delivery-group group feat/quiet-deliveries delivered: every member is delivered
mesh/check checking b7d03b68974d on 8f2d22db0caa
2026-10-22 14:25:26 +02:00
jschoubben 21954ad1bc issue 1840/S3b: prove a process may watch its own state's bucket and no other (AC9) 2026-10-22 14:23:58 +02:00
jschoubben 485eb44de4 issue 1840/S3: keep each node's declared packages on the bus, so the pacman verbs can say what a node declares 2026-10-22 14:23:58 +02:00
jschoubben 8d562f9b22 Say a walk's start and end again with the machines whose declaration its merges change (ADR 0505 §2) 2026-10-22 14:23:54 +02:00
mesh-tools 8f2d22db0c Merge pull request 'Move route-proxy's program into its module and run it on the pinned alpine image (issues 1633, 1634, 1635, ADR 0480 §1-§4, §11)' (#433) from feat/r4d into main 2026-10-22 12:19:07 +00:00
jschoubben cc8a74fce4 issue 1840/S3: take the SDK that names declared-packages 2026-10-22 14:18:39 +02:00
mesh-tools aa9da94260 Merge pull request 'A build node leaves its builds' images behind no more, and takes no work under a floor of free space (issue 1834)' (#435) from fix/1834-build-node-disk into main 2026-10-22 12:05:14 +00:00
jschoubben 01f615aa0a Read the bus release from the nats module's upstream artifact, so the tests' bus is still held to the mesh's once the Dockerfile is gone (issue 1640)
mesh/merge-gate head 01f615aa0ad6 merged into trunk aa9da942606d — pass: group feat/r4d composed: every machine composes with the change as it did without …
mesh/delivery delivered
mesh/repo-check head 01f615aa0ad6 merged into trunk aa9da942606d — pass: its merge-check.sh passed
mesh/delivery-group group feat/r4d delivered: every member is delivered
mesh/check failed: unit: ..test: node:internal/modules/esm/resolve:241 | 01f615aa0ad6 on aa9da942606d | unit 3345 run, 0 kept, 7 skipped | integration…
mesh-catalog #434 runs the nats server as the upstream image named in module.json (ADR 0497) and
deletes modules/nats/Dockerfile, so the check failed opening it. The manifest's upstream nats
artifact is read first; the Dockerfile only for a catalogue from before the move (the captured copy).
2026-10-22 13:38:00 +02:00
jschoubben f501eaca43 Remove an image's pushed names with its build tag, and take no work under the free-space floor (issue 1834)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@01f615aa0ad6 (merged as 8f2d22db into main, walk plan-17926707…
mesh/merge-gate head f501eaca4305 merged into trunk 4463991e0609 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head f501eaca4305 merged into trunk 4463991e0609 — pass: its merge-check.sh passed
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
The registry push tags the image a second time, so removing only the build tag freed nothing.
The held image a comparison pulls goes after it is read, and a node under the floor leaves the ask
to another build node instead of taking it and failing.
2026-10-22 13:19:58 +02:00
jschoubben 995f8237ce Free a build node's disk after every build and refuse work under a floor of free space (issue 1834)
A full / made every check on shanks end 'store never became ready' with the reason erased by --rm.
Builds now remove their local image tag, narrowed source and context clone; a build or check
refuses under 20 GiB free on / and the workspace, saying so; the throwaway store is no longer
started with --rm and a store that does not come up reports its exit code and last 20 log lines.
2026-10-22 13:14:00 +02:00
jschoubben 707bfba4ea Check an image's Dockerfile place instead of refusing the image kind (ADR 0487 clause 13)
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 707bfba4ea36 merged into trunk 4463991e0609 beside mesh-catalog#434@68183ee4 — fail: its merge-check.sh failed: --- FAIL: TestTheBusTh…
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
2026-10-22 11:45:06 +02:00
mesh-tools 4463991e06 Merge pull request 'Compose a pull request's check with the config values held for it (issue 1826, ADR 0502)' (#434) from fix/1826-held-config into main 2026-10-22 09:38:35 +00:00
jschoubben 4357b3688f Tell mesh-config what a head declares at every head, so a head that reverts a key drops its held value (issue 1826)
mesh/repo-check head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: its merge-check.sh passed
mesh/merge-gate head 4357b3688f58 merged into trunk 3d4ad6fefb7c — pass: group fix/1826-held-config composed: every machine composes with the change as it …
mesh/delivery delivered
mesh/delivery-group group fix/1826-held-config delivered: every member is delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 11:07:34 +02:00
jschoubben 03927a6f6c Carry a config value held for a pull request's new key into its composed check (issue 1826)
The controller tells mesh-config what a pull request declares, asks the approved held values with each check, and the build agent and merge gate lay them over the change's composition only: a key the trunk does not declare yet can then be approved before the merge (ADR 0502).
2026-10-22 11:01:12 +02:00
jschoubben 499ff5c218 Move the route proxy into its module and run it from the trust image, on mesh-sdk's public membership (issues 1633, 1634, 1635)
mesh/repo-check out of date: judged beside mesh-catalog#434 at c948f3a61c74, it is now at 7451e7edf4e5; judged again on a recheck or when it is merged
mesh/merge-gate group feat/r4d: composed again for the group's new heads
mesh/delivery superseded: a newer head of the same pull request
mesh/delivery-group group feat/r4d rejected: a member's own check failed
mesh/check failed: unit: i3/cmd/i3-tools.TestTheMainFileAndEveryModulesContributionLoadTogether: manifest_test.go:247: expected the launcher, clipboar…
A bundle is built only from its module's directory (ADR 0480 clause 4), so the proxy leaves
examples/ and the controller's internal/broker; the broker keeps no second copy of what the
proxy shares with mesh-sdk. The image and its Dockerfile go, and a test holds that only
build-agent declares one (clause 11).
2026-10-22 09:53:56 +02:00
mesh-tools 3d4ad6fefb Merge pull request 'Limit what a build or check starts on a build node: a per-node build slice (issue 1798)' (#432) from fix/1798-limit-build-resources into main 2026-10-21 23:20:57 +00:00
jschoubben 6bf38072ca Say in the build-agent settings that changing one restarts the build slice (issue 1798)
mesh/merge-gate head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 6bf38072cacf merged into trunk a929df9b0ec3 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 01:01:56 +02:00
jschoubben b174d743c0 Name the build slice without a dash so it sits beside user.slice, and restart it when its limits change (issue 1798)
mesh/merge-gate building on ace since 00:57
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:56:48 +02:00
jschoubben 37119ea31f Fill the build slice's limits as the mesh fills a module's template today (issue 1798)
mesh/merge-gate building on ace since 00:53
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: github.com/novox/mesh-controller/internal/identity.TestACertificateChainsToTheMeshsOwnAuthority: authority_test.go:45:…
2026-10-22 00:38:02 +02:00
jschoubben af2c6b5e94 Run builds and checks in a per-node build slice with CPU, IO and memory limits (issue 1798) 2026-10-22 00:24:45 +02:00
mesh-tools a929df9b0e Merge pull request 'The checker runs beside today's check: checks wait while a delivery builds; a group's SDK member merges; a flaky test fixed' (#430) from feat/the-checker into main 2026-10-21 21:46:26 +00:00
mesh-tools a3c965b9a0 Merge pull request 'Remove a check's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)' (#431) from fix/1772-check-volumes into main 2026-10-21 21:15:07 +00:00
jschoubben b9ec1919b8 Name plans-building among the controller's readable states and pin the SDK that does (issue 1687, ADR 0483 clause 2)
mesh/repo-check head b9ec1919b865 merged into trunk a3c965b9a0d5 beside mesh-catalog#423@12ff1cfb — pass: its merge-check.sh passed
mesh/merge-gate head b9ec1919b865 merged into trunk a3c965b9a0d5 — pass: group feat/the-checker composed: every machine composes with the change as it did …
mesh/delivery delivered
mesh/delivery-group group feat/the-checker delivered: every member is delivered
2026-10-21 22:47:04 +02:00
jschoubben d491356f59 Remove a build's throwaway containers with their anonymous volumes, so a build node's disk does not fill (issue 1772)
mesh/merge-gate head d491356f59dd merged into trunk 1f3fdf311167 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head d491356f59dd merged into trunk 1f3fdf311167 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 22:40:16 +02:00
jschoubben 47e1819101 Merge remote-tracking branch 'origin/fix/1106-sdk-member-merges' into feat/the-checker
mesh/repo-check out of date: judged beside mesh-sdk#66 at 78364e318bdc, it is now at 580da61a9107; judged again on a recheck or when it is merged
mesh/delivery-group group feat/the-checker checking: 4 of 4 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 47e1819101a1 merged into trunk 1f3fdf311167 — fail: group feat/the-checker composed: 1 problem(s) the change brings; the first: mesh-d…
2026-10-21 22:01:42 +02:00
jschoubben 0e25ed7c5a Merge remote-tracking branch 'origin/fix/1399-person-word-test' into feat/the-checker 2026-10-21 22:01:42 +02:00
jschoubben 79e2923dc8 Merge remote-tracking branch 'origin/fix/1652-checks-wait' into feat/the-checker
# Conflicts:
#	internal/broker/testdata/composed.conf
#	internal/link/builds_nats.go
2026-10-21 22:01:42 +02:00
mesh-tools 1f3fdf3111 Merge pull request 'Judge a duplicate hand-out before marking a build in hand, and never hand every build back over a stale mark (issue 1768)' (#429) from fix/1768-in-hand-marker into main 2026-10-21 19:59:30 +00:00
jschoubben 9902a3f16c Name every group member's trunk in the composed verdict, the announced one when the seat names none (issue 1106) 2026-10-21 21:57:30 +02:00
jschoubben 11815c0fb3 Have the delivery test's view hear a plan a verb let go, so a loaded machine cannot read it still waiting (issue 1399) 2026-10-21 21:45:58 +02:00
mesh-tools f973cefd7f Merge pull request 'Ask the operator for a held move and a failed release, and send no image the store lacks (issues 1732, 1320, 1742)' (#428) from fix/1732-held-moves-asked into main 2026-10-21 19:07:51 +00:00
jschoubben 696300b9e2 Judge a duplicate hand-out before the in-hand claim, and treat a stale mark as stale (issue 1768)
mesh/merge-gate head 696300b9e280 merged into trunk f973cefd7fd3 — pass: builds build-agent, mesh-controller → ace, g14, novox, shanks; no bus step; every …
mesh/repo-check head 696300b9e280 merged into trunk f973cefd7fd3 — pass: its merge-check.sh passed
mesh/delivery delivered
A dropped duplicate left the in-hand marker, so every later claim failed and every ask spent its deliveries.
2026-10-21 20:57:12 +02:00
jschoubben e7cc315fd0 Guard the taking loop's subscriptions, and update the two tests that held a busy machine's pulls
The loop now lets go of and binds queues around every ask, so Close takes the lock the loop does;
the busy machine's test says it holds no pull, and the stray test keeps its long check alive
within its two-second ack wait.
2026-10-21 20:53:58 +02:00
mesh-tools 98155e327b Merge pull request 'Take mesh-sdk at 07a6592: an item's text places a config value (issue 1658), and fit the unit level in its bound (issue 1763)' (#420) from fix/1656-xterm-config into main 2026-10-21 18:36:39 +00:00
jschoubben 60253c82b7 Fix the review of the checks-wait milestone: redeliveries, deleted modules, the kept queue
A check pending redelivery is read before the let-through fetches (none is handed back);
a deleted module is done for the plan's key; the queue an ask came from is let go of when
it ends. Tries say they run under their own id.
2026-10-21 20:17:37 +02:00
jschoubben d6db334840 Offer no Release on a machine that also holds a person's own hold (issue 1732, review round 1)
mesh/repo-check head d6db334840a5 merged into trunk 98155e327b6d beside mesh-host#129@09c77d57 — pass: its merge-check.sh passed
mesh/merge-gate head d6db334840a5 merged into trunk 98155e327b6d — pass: group fix/1732-held-moves-asked composed: every machine composes with the change a…
mesh/delivery delivered
mesh/delivery-group group fix/1732-held-moves-asked delivered: every member is delivered
2026-10-21 20:15:58 +02:00
jschoubben 69d016bfdf Merge remote-tracking branch 'origin/main' into fix/1732-held-moves-asked 2026-10-21 20:15:13 +02:00
jschoubben bb8f076cc0 Merge remote-tracking branch 'origin/main' into fix/1652-checks-wait 2026-10-21 20:14:12 +02:00
jschoubben a58b38cee3 Keep the missing image out of the condition's summary (issue 1742) 2026-10-21 20:06:10 +02:00
jschoubben 9d260412b3 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config
mesh/repo-check head 9d260412b330 merged into trunk 44033f4b6d72 beside mesh-catalog#416@b9dcf19c — pass: its merge-check.sh passed
mesh/merge-gate head 9d260412b330 merged into trunk 44033f4b6d72 — pass: group fix/1656-xterm-config composed: every machine composes with the change as it…
mesh/delivery-group group fix/1656-xterm-config failed: a member failed
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792609673527907956 (novox/mesh-controller at f973cefd into ma…
2026-10-21 19:59:59 +02:00
jschoubben 99610e9569 Cut the controller's slowest unit tests (a source scan that regexed every file, resolver stand-in timeouts, a wait bound) so mesh-controller's unit level fits its bound (issue 1763) 2026-10-21 19:59:58 +02:00
jschoubben bb4d2e1da4 Let go of the other queues while an ask runs, and read the let-through race right
A message that reaches a builder's subscription outside a fetch was held unheartbeated for
the whole run of another ask, behind which a delivery waited (issue 1747). The queues the
running ask did not come from are let go of and bound again after it.
2026-10-21 19:56:23 +02:00
mesh-tools 44033f4b6d Merge pull request 'Issue a bus process's account when its module is assigned or gains it, and warn of one still missing instead of composing it silently (issue 1226)' (#427) from fix/1226-issue-process-accounts into main 2026-10-21 17:53:47 +00:00
jschoubben dc06af1f5a Leave a module out of a send when the store lacks its image, and say it (issue 1742) 2026-10-21 19:43:36 +02:00
jschoubben b62fbeabbf Ask a held move and the builds held after a failed release, and say them on the plan's line (issue 1732, 1320) 2026-10-21 19:43:36 +02:00
jschoubben 373aefd14e Take mesh-sdk at 07a6592, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config checking: composed check out of date: novox/mesh-sdk@c8e35097e861 was judged against trunk 85b5a2687e6a, the tr…
mesh/merge-gate building on ace since 19:52
mesh/repo-check out of date: judged against trunk 22c9815f6522, the trunk is now 44033f4b6d72; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 19:43:32 +02:00
jschoubben f66458fe25 Merge remote-tracking branch 'origin/main' into fix/1656-xterm-config 2026-10-21 19:43:02 +02:00
mesh-tools 22c9815f65 Merge pull request 'Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)' (#423) from fix/1310-failed-check-summary into main 2026-10-21 17:39:15 +00:00
jschoubben a14dc04b7b Hold the build seat's checks while a delivery builds, and let one check that waited 30 minutes through
ADR 0483: the controller keeps a key per plan in its build phase; holders read it and
take no new check while one is there, but for the oldest check past 30 minutes, one
at a time by a let-through key. The queue says why checks wait.
2026-10-21 19:32:47 +02:00
jschoubben 181b579ac6 Assert the next compose carries the issued account, not a stand-in (issue 1226, review round 2)
mesh/merge-gate head 181b579ac6ea merged into trunk 22c9815f6522 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 181b579ac6ea merged into trunk 22c9815f6522 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:27:22 +02:00
jschoubben b08c6372e1 Assert the next compose carries the issued account (issue 1226, review round 2) 2026-10-21 19:26:57 +02:00
jschoubben 4e11965a2a Merge remote-tracking branch 'origin/main' into fix/1226-issue-process-accounts 2026-10-21 19:26:00 +02:00
jschoubben faf8beee84 Warn instead of holding the machine for an unissued process account, and drive the delivery path in the test (issue 1226, review round 1) 2026-10-21 19:25:44 +02:00
mesh-tools 3827c3eb60 Merge pull request 'Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920)' (#426) from fix/920-plan-line-says-the-refusal into main 2026-10-21 17:23:01 +00:00
jschoubben 6bcfbeb5b2 Issue a bus process's account when its module is assigned or gains it (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben deac3e232c Refuse a process account's stand-in, naming issue (issue 1226) 2026-10-21 19:21:11 +02:00
jschoubben 99da867a98 Merge origin/main into fix/1656-xterm-config 2026-10-21 19:16:11 +02:00
jschoubben 612f493c49 Say only a refusal as a plan's hold, not a tier's summary left in its note (issue 920, review round 1)
mesh/merge-gate head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: builds mesh-controller → novox; no bus step; every machine composes with the chang…
mesh/repo-check head 612f493c4926 merged into trunk 5ba2ffe8e608 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-21 19:01:42 +02:00
jschoubben 2377d75ed8 Say a building plan's refusal in its line, not 'building' with LATE, once its tier is built (issue 920) 2026-10-21 18:57:59 +02:00
mesh-tools 5ba2ffe8e6 Merge pull request 'Send a declaration's large files by reference through a bucket per node, and hold no check twice on the build seat' (#425) from feat/1696-declaration-files into main 2026-10-21 16:51:05 +00:00
jschoubben 8fc0d135bb Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files
mesh/merge-gate head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 8fc0d135bbe2 merged into trunk d568d76b9710 — pass: its merge-check.sh passed
mesh/delivery delivered
# Conflicts:
#	go.mod
#	go.sum
#	testdata/beside/CAPTURED
#	vendor/modules.txt
2026-10-21 18:32:56 +02:00
mesh-tools d568d76b97 Merge pull request 'Take mesh-sdk 90b20f5, so node-files' contract names upload and download (issue 1718, design 60 F3 group)' (#424) from feat/files-through-the-store into main 2026-10-21 16:31:27 +00:00
jschoubben 610d88e6bf Merge commit '3217f097d38d9f1c4302ec827ad6b371fd8ba95b' into feat/1696-declaration-files 2026-10-21 18:21:37 +02:00
jschoubben 0b11b6c98d Take mesh-sdk at its merge 725590e0 of the per-node bucket (issue 1694, ADR 0488) 2026-10-21 18:03:47 +02:00
jschoubben 92a5193e31 Merge remote-tracking branch 'origin/main' into feat/1696-declaration-files 2026-10-21 18:03:06 +02:00
jschoubben 3217f097d3 Make a fetch's pull request end before the fetch does, and drop the hand-back loop that could itself take asks (issue 1649) 2026-10-21 17:28:40 +02:00
jschoubben 69a67876c2 Take mesh-sdk at 6f63275, which counts an item's text as a placement of a config value (issue 1658)
mesh/delivery-group group fix/1656-xterm-config rejected: a member's own check failed
mesh/merge-gate waits for its group's composed check
mesh/repo-check head 69a67876c269 merged into trunk d96c9f56a3f1 beside mesh-catalog#416@f178aab6 — fail: its merge-check.sh failed: --- FAIL: TestModuleCh…
mesh/delivery superseded: a newer head of the same pull request
2026-10-21 17:03:35 +02:00
jschoubben 650274d700 Read a pull's held asks with Fetch, hand them back all through a build, and drop a duplicate only while its builder's key is fresh (issue 1649) 2026-10-21 16:58:39 +02:00
jschoubben 678a517843 Tests first for issue 1649: a held ask really handed back, a stale running key no duplicate 2026-10-21 16:57:40 +02:00
jschoubben 516af253c6 Merge main into feat/1696-declaration-files, keeping mesh-sdk at 86f9a1a5, which holds main's pin 85b5a268 2026-10-21 16:46:21 +02:00
jschoubben 5157ccc3a8 Hand back any ask a pull holds before a build starts, and drop a hand-out of a check another builder runs, so one check is never run twice (issue 1649) 2026-10-21 16:35:09 +02:00
jschoubben 087e9266d1 Test the capability gate, the sweep's mark re-read and the lock use of a send and of the sweep, each failing under its mutation (issues 1696, 1697; review round 2) 2026-10-21 16:28:24 +02:00
jschoubben a3b4e0601d Take mesh-sdk at 86f9a1a5 and name each node's bucket with its FileBucketFor, refusing a node name that could carry a wildcard (issues 1696, 1697; ADR 0488) 2026-10-21 16:05:24 +02:00
jschoubben da364c2730 Give each node a bucket of its own for the files that travel by digest, and keep the sweep from removing an object a send relies on (issues 1696, 1697; review of ADR 0486 step 3)
A literal * in a published subject matches the grant's *, so in one shared bucket a node could read every node's files. Each node now
has declaration-files-<node>, which its account reads by name and the controller writes; the sweep judges each bucket against that
node's latest declaration, under the node's lock with the send, and an object the sweep marked is put again by the next send. Putting
that fails send after send raises a condition.
2026-10-21 15:43:36 +02:00
jschoubben 4771ee53d9 Name the failing line in a failed check's summary, not the warning count printed last (issue 1310)
mesh/merge-gate out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 5ba2ffe8e608, the trunk is now 22c9815f6522; judged again on a recheck or when it is merged
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@181b579ac6ea (merged as 44033f4b into main, walk plan-17926043…
2026-10-21 15:21:10 +02:00
jschoubben ceaa616d01 Send files over 64 KiB by digest through the bus's object bucket to nodes that read them, grant it, and warn at 75 percent (issues 1696, 1697, 1698; ADR 0486 step 3)
A node that reports file-references gets the digest and size of each file over 64 KiB that may travel so, and the content is put in
the bucket declaration-files before the declaration is sent; others get every content inline. The controller alone writes the
bucket and node-engines read an object by name (the four grants the node-engine's own fetch makes, no list, no write). An object
goes a day after no node's latest declaration names it. The near-limit condition warns at 75 percent, and a test holds the control
node's declaration, in the reference form, under half the bus's largest message.
2026-10-21 15:16:11 +02:00
jschoubben fa5b5e1c11 Take mesh-sdk at 863a981e, which carries the file references of ADR 0486 (issue 1696) 2026-10-21 15:07:36 +02:00
2147 changed files with 26014 additions and 701378 deletions
+3 -1
View File
@@ -2,9 +2,11 @@
# A built binary. The lab writes one here when pointed at the repository root by mistake;
# built artifacts belong in build/, which is already ignored.
/mesh-builder
/mesh-controller
/postgres-provisioner
/redis-provisioner
# And where `go build` in cmd/mesh-controller leaves it (novox/hq issue 233).
/cmd/mesh-controller/mesh-controller
# No repository commits its dependencies (issue 1305).
vendor/
+3 -5
View File
@@ -21,17 +21,15 @@ ARG GO_BASE=golang@sha256:8ac98ca534ac3f51e1f420a1dd2c15e74c75cfa0f23f3ad27eb5d7
FROM ${GO_BASE} AS build
WORKDIR /src
# **Nothing is fetched** (novox/hq to-be 45 Phase 1): every dependency is in vendor/, committed, so
# the image builds from this repository alone — the host's validator among them, whose module no
# public proxy is asked for. Dependencies first, so a change to the source does not re-copy them.
# **No dependency is committed** (issue 1305): go.sum names each one's version and checksum, and the build
# resolves them with -mod=mod. Dependencies first, so a change to the source does not re-copy them.
COPY go.mod go.sum ./
COPY vendor/ vendor/
# `COPY . .` brings `.git` along, so Go is told not to stamp the commit: the version is said by -X, and a
# rebuild of an unchanged program is the same bytes (novox/hq ADR 0267 rule 6, issue 363).
COPY . .
ARG VERSION=development
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath -buildvcs=false \
RUN CGO_ENABLED=0 GOFLAGS=-mod=mod go build -trimpath -buildvcs=false \
-ldflags "-s -w -X main.version=${VERSION}" \
-o /mesh-controller ./cmd/mesh-controller
+3 -44
View File
@@ -46,17 +46,6 @@ image:
@echo
@docker image inspect $(IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The builder ships as an image too, because it is a module the mesh assigns rather than a program
# somebody starts on a machine by hand.
BUILDER_IMAGE ?= mesh-builder:$(VERSION)
BUILDER_DEV_TAG ?= mesh-builder:development
builder-image:
@test -n "$(GO_BASE)" || { echo "no GO_BASE; pass GO_BASE=<image>"; exit 1; }
docker build --build-arg GO_BASE=$(GO_BASE) -f cmd/mesh-builder/Dockerfile -t $(BUILDER_IMAGE) -t $(BUILDER_DEV_TAG) .
@echo
@docker image inspect $(BUILDER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The provisioner ships as an image too, because it is the thing that makes a sealed credential
# true on a machine -- and the mesh cannot, having discarded the plaintext.
PROVISIONER_IMAGE ?= mesh-provision-postgres:$(VERSION)
@@ -90,35 +79,6 @@ redis-provisioner-image:
@echo
@docker image inspect $(REDIS_PROVISIONER_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The proxy that turns a route grant into traffic reaching a workload.
PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
# The whole check. Raises a database, runs the repository's own check against it -- merge-check.sh,
# the very script `mesh/repo-check` runs -- and takes the database down again, including when the
# check fails, which is why the teardown is not conditional and the script's exit status is the
# target's.
#
# **It calls the script rather than restating it** (novox/hq issue 431): this target used to list its
# own steps, its formatting step walked vendor/ where the script's does not, and it failed on
# third-party code no change could fix -- so a developer's check and the gate disagreed, and the steps
# after formatting never ran through it. The script is the one list of steps; this target only gives it
# a database (MESH_TEST_POSTGRES, exported above).
#
# **Two levels** (issue 895): the unit level, `go test ./...` untagged, timed and held to a bound; and the integration
# level, `-tags integration`, the tests that start a bus, a store or another process. Each is said with its time.
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
# store's tests would skip rather than fail anyway.
@@ -128,14 +88,13 @@ test:
vet:
go vet ./...
# Quick steps for a developer, not part of `check`. The directories gofmt reads must be the ones
# merge-check.sh lists, never `.`, which walks vendor/ (novox/hq issue 431).
# A quick step for a developer: the checker runs the build module's own formatting and vet (ADR 0501 §5, D7).
fmt:
@unformatted=$$(gofmt -l cmd internal examples) ; \
if [ -n "$$unformatted" ] ; then echo "not gofmt'd:" ; echo "$$unformatted" ; exit 1 ; fi
postgres:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker run -d --name $(PG_CONTAINER) -e POSTGRES_PASSWORD=check \
-p 127.0.0.1:$(PG_PORT):5432 $(PG_IMAGE) >/dev/null
@printf 'waiting for postgres'
@@ -147,7 +106,7 @@ postgres:
echo ' — never came up' ; docker logs $(PG_CONTAINER) | tail -20 ; exit 1
postgres-stop:
@docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true
@docker rm -f -v $(PG_CONTAINER) >/dev/null 2>&1 || true
clean:
rm -rf build/
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
-29
View File
@@ -1,29 +0,0 @@
#!/bin/sh
# The unit tests of every Go module of its own below this repository's root, run from the root, for merge-check.sh
# and for the test that tries it on a tree (merge_check_nested_test.go). Its own file so that the test needs no
# particular merge-check.sh: the check that judges a change to merge-check.sh is main's.
set -eu
cgo="${cgo:-0}"
race="${race:-}"
# nested_modules vets and tests every Go module of its own below the root, `modules/<name>/go.mod` (novox/hq ADR 0420
# §5, issue 1025): `./...` stops at a nested go.mod, so a module that lives whole in this repository would have its
# unit tests run by nobody. None is a pass. Each runs in its own directory, outside the vendor directory this
# repository's own packages are built from; a nested module vendors what the forge holds (its own `vendor/`, so the
# check runs offline), as this repository's root module does (issue 1019). A failing module fails the check; the others are still run.
nested_modules() {
nested=0
for gomod in modules/*/go.mod; do
[ -e "$gomod" ] || continue
dir=$(dirname "$gomod")
echo "nested module $dir:"
if ! (cd "$dir" && GOFLAGS= CGO_ENABLED=0 go vet ./... &&
GOFLAGS= CGO_ENABLED="${cgo:-0}" go test ${race:-} -count=1 -timeout 30m ./...); then
echo "nested module $dir FAILED"
nested=1
fi
done
return $nested
}
nested_modules
-22
View File
@@ -1,22 +0,0 @@
ARG ALPINE_BASE=alpine:3
ARG GO_BASE=golang:1.25-alpine
# The builder, as a module ships one.
#
# Not FROM scratch, unlike the control plane: this one runs git and a container client, so it
# needs a filesystem with them on it. That is the honest cost of a machine whose job is to build —
# and it is why building is a MODULE on a machine that has a runtime rather than something the
# control plane does (novox/hq ADR 0005).
FROM ${GO_BASE} AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -buildvcs=false -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
# is the machine's, reached through its socket — a build machine shares the runtime it was given
# rather than running one inside itself.
RUN apk add --no-cache git docker-cli
COPY --from=build /mesh-builder /usr/local/bin/mesh-builder
ENTRYPOINT ["/usr/local/bin/mesh-builder"]
-146
View File
@@ -1,146 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
// container declares `busy` as `mesh-builder busy`: the node-engine runs it inside the container before it
// replaces it. It exits 0 while a build is in hand and 1 when none is; and from the first ask the agent takes
// no new build request, so its answer ends once the build in hand ends — a replaced builder finishes the check
// it runs, instead of the build queue handing it out again from the start (issue 594).
//
// Two marks in the container's own temporary directory, as the ready mark (ready.go): a new container starts
// without them, and a restarted one removes them before it takes work again.
//
// - in hand: written by the agent when it claims a request, before it reads whether it is drained; removed
// when the build ends.
// - drained: written by the ask, before it reads whether a build is in hand.
//
// Each side writes its own mark before it reads the other's, so an ask racing a fetch sees the build in hand,
// or the agent sees the drain and hands the request back; never neither.
func inHandMark() string { return filepath.Join(os.TempDir(), "mesh-builder.in-hand") }
func drainMark() string { return filepath.Join(os.TempDir(), "mesh-builder.drained") }
// The drain lapses when no ask has come for drainLapse: the node-engine asks every 20 seconds for as long as it
// really waits, so an agent whose replacement was withdrawn — the declaration put back, the walk stopped —
// takes work again rather than staying drained for ever (review of ADR 0335). And an agent drained for longer
// than the node-engine's bound and the slack says it is not ready: the replacement it was drained for never
// came, which is a fault to see.
const (
drainLapse = 2 * time.Minute
drainTooLong = 15*time.Minute + 5*time.Minute
)
// drainedAt is when the drain began and when it was last asked, from its mark; false when there is no mark.
func drainedAt() (began, asked time.Time, ok bool) {
info, err := os.Stat(drainMark())
if err != nil {
return time.Time{}, time.Time{}, false
}
asked = info.ModTime()
began = asked
if raw, err := os.ReadFile(drainMark()); err == nil {
if t, err := time.Parse(time.RFC3339, strings.TrimSpace(string(raw))); err == nil && !t.After(asked) {
began = t
}
}
return began, asked, true
}
// drained says the agent was asked whether it is busy within drainLapse: it takes no new build request.
func drained() bool { return drainedSince(time.Now()) }
func drainedSince(now time.Time) bool {
_, asked, ok := drainedAt()
return ok && now.Sub(asked) < drainLapse
}
// drainedTooLong is the health check's second half: nil unless the agent has been drained, asked all along, for
// longer than its node-engine's bound and the slack.
func drainedTooLong(now time.Time) error {
began, asked, ok := drainedAt()
if !ok || now.Sub(asked) >= drainLapse || now.Sub(began) <= drainTooLong {
return nil
}
return fmt.Errorf("drained since %s and still asked: the replacement its node-engine waits for has not come "+
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
if closeErr := f.Close(); err == nil {
err = closeErr
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
if drained() {
_ = os.Remove(inHandMark())
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
return true
}
// holding says what is in hand, for the ask to print: written once the request is read.
func holding(what string) {
_ = os.WriteFile(inHandMark(), []byte(time.Now().UTC().Format(time.RFC3339)+" "+what+"\n"), 0o644)
}
// released ends the build in hand.
func released() {
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
}
// undrained clears both marks at the start of a holder: a container started again is asked afresh.
func undrained() error {
for _, mark := range []string{drainMark(), inHandMark()} {
if err := os.Remove(mark); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove %s: %w", mark, err)
}
}
return nil
}
// busy is the ask: it drains the agent, then says whether a build is in hand. Nil — exit 0 — while one is.
func busy() (string, error) {
// The mark keeps when the drain began while it is asked within drainLapse, and is written again on every
// ask: its time is the last ask.
began := time.Now().UTC()
if was, _, ok := drainedAt(); ok && drained() {
began = was.UTC()
}
if err := os.WriteFile(drainMark(), []byte(began.Format(time.RFC3339)+"\n"), 0o644); err != nil {
// Not drained, it could take the next build: said, and answered busy only for what is in hand now.
fmt.Fprintf(os.Stderr, "cannot mark this agent drained (%s): %v\n", drainMark(), err)
}
held, err := os.ReadFile(inHandMark())
if err != nil {
return "", errors.New("idle: no build in hand, and this agent takes no new one")
}
since, what, _ := strings.Cut(strings.TrimSpace(string(held)), " ")
if what == "" {
what = "a build"
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
-162
View File
@@ -1,162 +0,0 @@
package main
import (
"os"
"strings"
"sync"
"testing"
"time"
)
// `mesh-builder busy` answers 0 while a build is in hand and non-zero when idle, and after its first ask the
// agent takes no new build request (novox/hq ADR 0335 §5).
func TestBusyAnswersWhileABuildIsInHandAndDrainsTheAgent(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("an agent never asked is drained")
}
// A build is claimed and runs.
if !claim() {
t.Fatal("an agent never asked refused a build")
}
holding("novox/mesh-controller (check-280)")
said, err := busy()
if err != nil || !strings.Contains(said, "novox/mesh-controller (check-280)") {
t.Fatalf("an agent with a build in hand did not answer busy with what it holds: %q %v", said, err)
}
// From that ask on it takes nothing new: the fetch is not made, and one fetched is handed back.
if !drained() {
t.Fatal("the ask did not drain the agent")
}
if claim() {
t.Fatal("a drained agent took a new build")
}
// The build in hand is still in hand after the refused claim, and busy goes on saying so.
if _, err := busy(); err != nil {
t.Fatalf("a refused claim ended the build in hand: %v", err)
}
// It ends: idle, and still drained.
released()
if _, err := busy(); err == nil {
t.Fatal("an agent with nothing in hand answered busy")
}
if claim() {
t.Fatal("a drained agent took a build after its last one ended")
}
// Started again, it is asked afresh.
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() || !claim() {
t.Fatal("a holder started again is still drained")
}
released()
}
// An ask racing a fetch sees the build in hand, or the agent sees the drain and hands the request back: never
// neither, which would replace the container under a build just taken.
func TestAnAskRacingAFetchNeverMissesTheBuild(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for i := 0; i < 200; i++ {
if err := undrained(); err != nil {
t.Fatal(err)
}
var claimed bool
var askErr error
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); claimed = claim() }()
go func() { defer wg.Done(); _, askErr = busy() }()
wg.Wait()
if claimed && askErr != nil {
t.Fatalf("round %d: the agent took a build and the ask said idle", i)
}
released()
}
}
// A drain no longer asked lapses: an agent whose replacement was withdrawn after the first ask (a put-back, a
// revert, a stopped walk) takes work again after drainLapse, rather than staying drained with its health green.
func TestADrainNoLongerAskedLapses(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if !drained() || claim() {
t.Fatal("an agent just asked takes work")
}
// The last ask was three minutes ago.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("a drain no longer asked for three minutes still holds")
}
if !claim() {
t.Fatal("an agent whose drain lapsed takes no work")
}
released()
}
// An agent drained, and asked all along, for longer than its node-engine's bound and the slack is not ready: the
// replacement it was drained for never came. One asked again after a lapse starts a new drain.
func TestAnAgentDrainedTooLongIsNotReady(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
taking("mesh-build-machine")
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
now := time.Now()
if err := drainedTooLong(now); err != nil {
t.Fatalf("an agent drained a moment ago is not ready: %v", err)
}
// The drain began 25 minutes ago, and the node-engine asked a moment ago.
began := now.Add(-25 * time.Minute).UTC().Format(time.RFC3339)
if err := os.WriteFile(drainMark(), []byte(began+"\n"), 0o644); err != nil {
t.Fatal(err)
}
err := drainedTooLong(time.Now())
if err == nil || !strings.Contains(err.Error(), "has not come") {
t.Fatalf("an agent drained for 25 minutes reads ready: %v", err)
}
// Each ask keeps when the drain began.
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if err := drainedTooLong(time.Now()); err == nil {
t.Fatal("an ask started the drain again")
}
// The asks stopped: the drain lapsed, and so did the fault.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if err := drainedTooLong(time.Now()); err != nil {
t.Fatalf("a lapsed drain still reads not ready: %v", err)
}
}
// An ask this build does not know exits non-zero: an older image asked `busy` reads idle, not busy for the
// node-engine's whole bound.
func TestAnUnknownCommandFails(t *testing.T) {
was := os.Args
t.Cleanup(func() { os.Args = was })
os.Args = []string{"mesh-builder", "some-later-word"}
if err := run(); err == nil {
t.Fatal("an unknown command exited 0")
}
os.Args = []string{"mesh-builder", "help"}
if err := run(); err != nil {
t.Fatalf("help failed: %v", err)
}
}
-38
View File
@@ -1,38 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// The build seat hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckTheJudgeCacheItDeclares(t *testing.T) {
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
// The siblings a check is asked to read reach the build seat's spec and, as read, the outcome (novox/hq ADR 0382).
func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main",
Siblings: []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", Dir: "mesh-controller"}},
SiblingsUnknown: "why"}})
if len(spec.Siblings) != 1 || spec.Siblings[0] != (builder.Sibling{Repository: "novox/mesh-controller", Number: 346,
Head: "a", Base: "main", Dir: "mesh-controller"}) || spec.SiblingsUnknown != "why" {
t.Fatalf("the spec reads %+v", spec)
}
out := siblingsOut([]builder.JudgedSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", BaseCommit: "b"}})
if len(out) != 1 || out[0].BaseCommit != "b" || out[0].Number != 346 {
t.Fatalf("the outcome reads %+v", out)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Siblings != nil {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
-433
View File
@@ -1,433 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/micro"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// What a holder of the build seat answers for, on its own machine (novox/hq ADR 0219).
//
// **The queue is the controller's; the build running here is this machine's.** The controller can
// see and change what waits in the seat's queue, but an ask a holder already took is a process tree
// on this machine and containers in this machine's runtime, and only this machine can end them. So
// the holder serves four verbs on the seat's subjects for this machine: what it is building, kill
// it, pause, resume.
//
// **One build at a time** (ADR 0190), which is also what builder.Said assumes — a package global
// set per build — so "the build running here" is one or none, and kill names it by id so a call
// that arrives as one build ends and the next begins cannot end the wrong one.
// holder is this machine's state as a holder of the build seat.
type holder struct {
on, seat string
// workspace is where the paused flag is kept, so a holder restarted while paused stays paused
// rather than silently taking work again.
workspace string
// say publishes this machine's state: whether it takes work (link.HolderState).
say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner
// scratch is where check builds are kept (novox/hq ADR 0350 §5): what the week's sweep lets go of. Nil where
// this machine publishes to no artifact store.
scratch builder.Scratch
mu sync.Mutex
paused bool
running *running
}
// running is the build this machine is doing.
type running struct {
request link.BuildRequest
step string
started time.Time
cancel context.CancelFunc
killed bool
// returned is the build's own work having ended, before a kill or not; outcome is what was then
// announced, and announced whether it went out.
returned bool
outcome string
announced bool
done chan struct{}
}
// pausedFile is where the flag lives in the workspace.
func pausedFile(workspace string) string { return filepath.Join(workspace, ".mesh-builder-paused") }
// newHolder reads the paused flag the workspace keeps.
func newHolder(on, seat, workspace string, say func(link.HolderState) error) *holder {
h := &holder{on: on, seat: seat, workspace: workspace, say: say, remove: plainRun}
if _, err := os.Stat(pausedFile(workspace)); err == nil {
h.paused = true
}
return h
}
// Paused is asked by the taking loop before every fetch.
func (h *holder) Paused() bool {
h.mu.Lock()
defer h.mu.Unlock()
return h.paused
}
// setPaused records the flag in the workspace first and then in memory, so what this holder says
// it is and what it would be after a restart never differ.
func (h *holder) setPaused(paused bool) error {
path := pausedFile(h.workspace)
if paused {
if err := os.MkdirAll(h.workspace, 0o755); err != nil {
return err
}
if err := os.WriteFile(path, []byte(time.Now().UTC().Format(time.RFC3339)+"\n"), 0o644); err != nil {
return fmt.Errorf("cannot keep the paused flag in %s: %w", path, err)
}
} else if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the paused flag %s: %w", path, err)
}
h.mu.Lock()
h.paused = paused
h.mu.Unlock()
h.announce()
return nil
}
// announce says whether this machine takes work. Never fatal: the flag is kept either way, and the
// controller reading an older state is a plan read as late rather than a build lost.
func (h *holder) announce() {
if h.say == nil {
return
}
if err := h.say(link.HolderState{On: h.on, Paused: h.Paused(), At: time.Now().UTC().Format(time.RFC3339Nano)}); err != nil {
fmt.Fprintf(os.Stderr, "cannot say whether this machine takes builds: %v\n", err)
}
}
// stateWhilePaused says this machine's state at start, and again every while it stays paused, so
// a pause outlives the events stream's retention.
func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
h.announce()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
if h.Paused() {
h.announce()
}
}
}
}
// expireChecks lets go of every check build older than a week (novox/hq ADR 0350 §5), at start and every while
// after. Every holder sweeps the one store; what one let go of, another finds gone.
func (h *holder) expireChecks(ctx context.Context, every time.Duration, now func() time.Time) {
sweep := func() {
if h.scratch == nil {
return
}
n, err := h.scratch.Expire(ctx, builder.CheckBuildsKept, now())
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d let go of, and not all: %v\n", builder.CheckBuildsKept, n, err)
case n > 0:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d check(s)' builds let go of\n", builder.CheckBuildsKept, n)
}
}
sweep()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
sweep()
}
}
}
// begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
h.mu.Lock()
h.running = r
h.mu.Unlock()
return r
}
// end clears it, and lets a kill waiting on it know it is over.
func (h *holder) end(r *running) {
h.mu.Lock()
if h.running == r {
h.running = nil
}
h.mu.Unlock()
close(r.done)
}
// stepped records the step a build is at, for `current`.
func (h *holder) stepped(r *running, step string) {
h.mu.Lock()
r.step = step
h.mu.Unlock()
}
// currentBuild is what `current` answers.
type currentBuild struct {
On string `json:"on"`
Paused bool `json:"paused"`
Running *struct {
ID string `json:"id"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Step string `json:"step,omitempty"`
Started string `json:"started"`
Elapsed string `json:"elapsed"`
} `json:"running,omitempty"`
Said string `json:"said"`
}
func (h *holder) current() currentBuild {
h.mu.Lock()
defer h.mu.Unlock()
out := currentBuild{On: h.on, Paused: h.paused}
taking := "taking builds"
if h.paused {
taking = "paused, taking no new build"
}
if h.running == nil {
out.Said = fmt.Sprintf("%s is building nothing; %s", h.on, taking)
return out
}
r := h.running
elapsed := time.Since(r.started).Round(time.Second)
out.Running = &struct {
ID string `json:"id"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Step string `json:"step,omitempty"`
Started string `json:"started"`
Elapsed string `json:"elapsed"`
}{r.request.ID, r.request.Repository, r.request.Path, r.request.Ref, r.step,
r.started.UTC().Format(time.RFC3339), elapsed.String()}
out.Said = fmt.Sprintf("%s is building %s (%s) at %s for %s; %s",
h.on, r.request.Repository, r.request.ID, orNothing(r.step), elapsed, taking)
return out
}
// The bounds a kill keeps: each pass removing containers, and the wait for the build to end between
// them. The worst case — 15s, 20s, 15s — is inside what the controller waits for the answer
// (killAnswer in the controller's queue.go, 75s).
var (
killRemoves = 15 * time.Second
killWaits = 20 * time.Second
)
// kill ends the build with this id, if it is the one running here and still working: its context
// cancelled — which kills each command's process group — and the containers it started removed by
// their label, once at once and again after the build has ended, so one created while it was being
// killed is not left. The build's own goroutine announces it failed, killed by hand, and settles the
// ask so it is not redelivered; the answer says whether that happened.
func (h *holder) kill(id string) (string, error) {
h.mu.Lock()
r := h.running
if r == nil || r.request.ID != id {
doing := "nothing"
if r != nil {
doing = r.request.ID
}
h.mu.Unlock()
return "", fmt.Errorf("%s is not building %s; it is building %s. `queue` says where an ask is", h.on, id, doing)
}
if r.returned {
h.mu.Unlock()
return "", fmt.Errorf("%s on %s has already ended on its own and is saying how; `builds` shows it", id, h.on)
}
r.killed = true
cancel, done := r.cancel, r.done
h.mu.Unlock()
cancel()
removed, removeErr := h.removeContainers(id)
ended := false
select {
case <-done:
ended = true
case <-time.After(killWaits):
}
again, againErr := h.removeContainers(id)
removed += again
if removeErr == nil {
removeErr = againErr
}
containers := fmt.Sprintf("%d container(s) it started removed", removed)
if removeErr != nil {
containers = "its containers could not all be listed or removed: " + removeErr.Error()
}
if !ended {
return fmt.Sprintf("killed %s on %s: %s; the build has not finished ending yet — `builds` says when "+
"its outcome is in", id, h.on, containers), nil
}
h.mu.Lock()
outcome, announced := r.outcome, r.announced
h.mu.Unlock()
if !announced {
return fmt.Sprintf("killed %s (%s) on %s: its commands ended and %s, and its outcome could not be "+
"announced — the ask is not settled and will be handed out again", id, r.request.Repository, h.on,
containers), nil
}
return fmt.Sprintf("killed %s (%s) on %s: its commands ended, %s, and its outcome announced as failed, %s — "+
"settled, so it is not handed to another machine", id, r.request.Repository, h.on, containers, outcome), nil
}
// removeContainers is one pass of removing what the build left, bounded.
func (h *holder) removeContainers(id string) (int, error) {
cleanup, stop := context.WithTimeout(context.Background(), killRemoves)
defer stop()
return builder.RemoveContainersOf(cleanup, h.remove, id)
}
// removeLeftBehind removes, once, what earlier holders on this machine left — called before anything is
// taken, so none of it is this holder's — and says what it did. A runtime that cannot be asked is said
// and the holder takes work all the same: what is left is waste, not a reason to stop building.
func (h *holder) removeLeftBehind() int {
cleanup, stop := context.WithTimeout(context.Background(), killRemoves)
defer stop()
n, err := builder.RemoveLeftBehind(cleanup, h.remove)
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "could not look for containers earlier builds left on %s: %v\n", h.on, err)
case n > 0:
fmt.Fprintf(os.Stderr, "removed %d container(s) earlier builds left on %s\n", n, h.on)
}
return n
}
// returned records that the build's work ended, and says whether a kill came first — only then is
// the build killed; an error it ended with on its own is its own outcome.
func (h *holder) returned(r *running) bool {
h.mu.Lock()
defer h.mu.Unlock()
r.returned = true
return r.killed
}
// said records the outcome announced, and whether it went out, for a kill to answer with.
func (h *holder) said(r *running, outcome string, announced bool) {
h.mu.Lock()
r.outcome, r.announced = outcome, announced
h.mu.Unlock()
}
// handlers are the seat's verbs, as this machine answers them.
func (h *holder) handlers() map[string]link.ToolHandler {
return map[string]link.ToolHandler{
"current": func(context.Context, json.RawMessage) (any, error) { return h.current(), nil },
"kill": func(_ context.Context, raw json.RawMessage) (any, error) {
var args struct {
ID string `json:"id"`
}
if err := json.Unmarshal(raw, &args); err != nil || strings.TrimSpace(args.ID) == "" {
return nil, errors.New("kill needs the build's id")
}
said, err := h.kill(strings.TrimSpace(args.ID))
if err != nil {
return nil, err
}
return map[string]any{"said": said}, nil
},
"pause": func(context.Context, json.RawMessage) (any, error) {
if err := h.setPaused(true); err != nil {
return nil, err
}
said := h.on + " is paused: it takes no new build until resumed"
if c := h.current(); c.Running != nil {
said += "; " + c.Running.ID + " runs on and finishes"
}
return map[string]any{"said": said, "paused": true}, nil
},
"resume": func(context.Context, json.RawMessage) (any, error) {
if err := h.setPaused(false); err != nil {
return nil, err
}
return map[string]any{"said": h.on + " takes builds again", "paused": false}, nil
},
}
}
func orNothing(s string) string {
if s == "" {
return "its start"
}
return s
}
// plainRun runs a command outside any build: no line reaches a build's log, because the build whose
// log it would be is the one being ended.
func plainRun(ctx context.Context, dir, name string, args ...string) (string, error) {
cmd := exec.CommandContext(ctx, name, args...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
if err != nil {
return string(out), fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, strings.TrimSpace(string(out)))
}
return string(out), nil
}
// announcement is what this holder answers discovery with (novox/hq ADR 0195, ADR 0197): this
// machine's verbs of the seat, in the shape every tool runtime announces a seat's verb — kind seat,
// the module answering, the seat, scope node, the machine, its description and argument schema — so
// the console finds `<node>/node-build-agent.kill` by searching, as it finds any seat's verb.
//
// One service per machine, named for the seat and identified by the machine, so the answer is this
// machine's four verbs and nothing more. The verbs are the compiled seat row's: a build machine has no
// store, and what it serves is what this binary was built to serve.
func announcement(seat, module, node string) micro.Info {
s, _ := catalogue.SeatNamed(seat)
var endpoints []micro.EndpointInfo
for _, v := range s.Serves {
schema, _ := json.Marshal(v.Input)
endpoints = append(endpoints, micro.EndpointInfo{
Name: seat + "__" + v.Name,
Subject: link.NodeSeatToolSubject(seat, v.Name, node),
// The queue group the verbs are served in, as every runtime announces its own.
QueueGroup: "seat." + seat,
Metadata: map[string]string{
"kind": "seat", "module": module, "tool": v.Name, "seat": seat, "scope": "node",
"node": node, "interchangeable": "false", "description": v.Description, "schema": string(schema),
},
})
}
return micro.Info{
ServiceIdentity: micro.ServiceIdentity{Name: seat, ID: node, Version: "0.1.0",
Metadata: map[string]string{"seat": seat, "scope": "node", "node": node, "module": module}},
Description: "what the build running on " + node + " is, and ending, pausing and resuming it (novox/hq ADR 0219)",
Endpoints: endpoints,
}
}
// moduleOf is the module a credential was issued for: its user is `<node>.<module>`.
func moduleOf(user string) string {
if _, module, ok := strings.Cut(user, "."); ok && module != "" {
return module
}
return "build-agent"
}
-172
View File
@@ -1,172 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A holder paused stays paused across its restart: the flag is kept in its workspace (novox/hq ADR
// 0219), and what it says about itself follows.
func TestAPausedHolderStaysPausedAcrossARestart(t *testing.T) {
workspace := t.TempDir()
var said []link.HolderState
h := newHolder("ace", link.TheBuildMachine, workspace, func(s link.HolderState) error {
said = append(said, s)
return nil
})
if h.Paused() {
t.Fatal("a new holder starts paused")
}
if _, err := h.handlers()["pause"](context.Background(), json.RawMessage(`{}`)); err != nil {
t.Fatal(err)
}
if !h.Paused() || len(said) != 1 || !said[0].Paused || said[0].On != "ace" {
t.Fatalf("paused: %v, said %+v", h.Paused(), said)
}
again := newHolder("ace", link.TheBuildMachine, workspace, nil)
if !again.Paused() {
t.Fatal("restarted, the holder forgot it was paused")
}
if _, err := again.handlers()["resume"](context.Background(), json.RawMessage(`{}`)); err != nil {
t.Fatal(err)
}
if newHolder("ace", link.TheBuildMachine, workspace, nil).Paused() {
t.Fatal("resumed, the holder came back paused")
}
}
// kill ends the build with that id — its context, which ends its commands — removes what it left by
// label, and refuses an id it is not building.
func TestKillEndsTheBuildRunningHereAndNoOther(t *testing.T) {
h := newHolder("ace", link.TheBuildMachine, t.TempDir(), nil)
var removed []string
h.remove = func(_ context.Context, _ string, name string, args ...string) (string, error) {
removed = append(removed, name+" "+strings.Join(args, " "))
if args[0] == "ps" {
return "c1\n", nil
}
return "", nil
}
kill := h.handlers()["kill"]
if _, err := kill(context.Background(), json.RawMessage(`{"id":"build-1"}`)); err == nil {
t.Fatal("killed a build while none ran")
}
building, cancel := context.WithCancel(context.Background())
r := h.begin(link.BuildRequest{ID: "build-1", Repository: "novox/a"}, cancel)
h.stepped(r, "image")
if c := h.current(); c.Running == nil || c.Running.ID != "build-1" || c.Running.Step != "image" {
t.Fatalf("current says %+v", c)
}
if _, err := kill(context.Background(), json.RawMessage(`{"id":"build-2"}`)); err == nil ||
!strings.Contains(err.Error(), "build-1") {
t.Fatalf("killed another id: %v", err)
}
// The build's own goroutine: it ends when its context does, as a build's commands do, and
// announces what came of it.
go func() {
<-building.Done()
if h.returned(r) {
h.said(r, link.KilledByHand, true)
}
h.end(r)
}()
answer, err := kill(context.Background(), json.RawMessage(`{"id":"build-1"}`))
if err != nil {
t.Fatal(err)
}
if building.Err() == nil {
t.Fatal("the build's context was not cancelled")
}
if !r.killed {
t.Error("the build is not marked killed, so it would be announced as an ordinary failure")
}
said := answer.(map[string]any)["said"].(string)
if !strings.Contains(said, "2 container(s)") || !strings.Contains(said, "announced as failed") || !strings.Contains(said, link.KilledByHand) {
t.Errorf("kill said %q", said)
}
// Removed at the kill and again once the build had ended: a container made in between is caught.
if len(removed) != 4 || !strings.Contains(removed[0], "label=mesh.build=build-1") || !strings.Contains(removed[2], "label=mesh.build=build-1") {
t.Errorf("removed %v", removed)
}
if c := h.current(); c.Running != nil {
t.Errorf("after the kill current says %+v", c)
}
}
// A holder announces this machine's verbs of the seat as the console reads a seat's verb, and no more.
func TestAHolderAnnouncesItsMachinesVerbsForTheConsole(t *testing.T) {
info := announcement(link.TheBuildMachine, moduleOf("ace.build-agent"), "ace")
if info.Name != "node-build-agent" || info.ID != "ace" || len(info.Endpoints) != 4 {
t.Fatalf("announced %s/%s with %d endpoints", info.Name, info.ID, len(info.Endpoints))
}
kill := info.Endpoints[1]
md := kill.Metadata
if kill.Subject != "mesh.seat.node-build-agent.tool.kill.ace" || kill.QueueGroup != "seat.node-build-agent" || md["kind"] != "seat" || md["seat"] != "node-build-agent" ||
md["scope"] != "node" || md["node"] != "ace" || md["tool"] != "kill" || md["module"] != "build-agent" ||
!strings.Contains(md["description"], "killed by hand") || !strings.Contains(md["schema"], `"id"`) {
t.Fatalf("kill is announced as %+v", kill)
}
body, err := json.Marshal(info)
if err != nil || len(body) > 8*1024 {
t.Fatalf("the answer is %d bytes (%v)", len(body), err)
}
}
// A build that ended on its own as the kill arrived says what it did: the kill is refused, and its
// own error is its outcome. One whose outcome could not be announced is not said to be settled.
func TestAKillArrivingAfterTheBuildEndedIsRefusedAndAnUnannouncedKillSaysSo(t *testing.T) {
h := newHolder("ace", link.TheBuildMachine, t.TempDir(), nil)
h.remove = func(context.Context, string, string, ...string) (string, error) { return "", nil }
_, cancel := context.WithCancel(context.Background())
r := h.begin(link.BuildRequest{ID: "build-1"}, cancel)
if killed := h.returned(r); killed {
t.Fatal("a build nobody killed reads as killed")
}
if _, err := h.kill("build-1"); err == nil || !strings.Contains(err.Error(), "ended on its own") {
t.Fatalf("killed a build that had ended: %v", err)
}
h.end(r)
building, cancel := context.WithCancel(context.Background())
r = h.begin(link.BuildRequest{ID: "build-2"}, cancel)
go func() {
<-building.Done()
if h.returned(r) {
h.said(r, link.KilledByHand, false)
}
h.end(r)
}()
said, err := h.kill("build-2")
if err != nil || strings.Contains(said, "announced as failed") || !strings.Contains(said, "could not be announced") {
t.Fatalf("kill said %q (%v)", said, err)
}
}
// **Issue 306**: a holder starting removes the containers earlier builds left on its machine, through
// the runner a kill uses, and a runtime that cannot be asked does not stop it.
func TestAHolderStartingRemovesWhatEarlierBuildsLeftHere(t *testing.T) {
h := newHolder("novox", link.TheBuildMachine, t.TempDir(), nil)
var removed []string
h.remove = func(_ context.Context, _ string, name string, args ...string) (string, error) {
removed = append(removed, name+" "+strings.Join(args, " "))
if args[0] == "ps" {
return "c1 build-1\nc2 build-1\nc3 check-here-2\n", nil
}
return "", nil
}
if n := h.removeLeftBehind(); n != 2 || len(removed) != 2 || removed[1] != "docker rm -f c1 c2" {
t.Fatalf("removed %d: %v", n, removed)
}
h.remove = func(context.Context, string, string, ...string) (string, error) {
return "", errors.New("no runtime")
}
if n := h.removeLeftBehind(); n != 0 {
t.Errorf("a runtime that cannot be asked removed %d", n)
}
}
-936
View File
@@ -1,936 +0,0 @@
// mesh-builder — the thing a build machine runs.
//
// It takes work from the mesh, turns a repository into artifacts, publishes them, and says what
// came out. It is **not** the control plane and it is **not** the host:
//
// - the control plane decides and never touches a machine. Building runs commands on one, and
// what the control plane may send a machine is bounded by the declaration language
// (novox/hq ADR 0005). "Run this build" is not in it, and widening the language so it could
// be would make the control plane able to run anything anywhere.
// - the host applies declarations and holds no opinion about what they contain. A host that
// also built things would need a container runtime and git, on every machine, to do something
// almost none of them will ever do.
//
// So it is a module: a program a machine runs because the mesh told it to, holding its own broker
// credential and nothing else. Compromise of a build machine is compromise of a build machine.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// version is set at build time.
var version = "development"
func main() {
if err := run(); err != nil {
fmt.Fprintf(os.Stderr, "mesh-builder: %v\n", err)
os.Exit(1)
}
}
const usage = `mesh-builder — builds modules for the mesh
It consumes build requests and answers with what it made. Nothing is listened on and nothing
is dialled except the broker.
MESH_BROKER_FILE a file the mesh sealed to this machine holding the same
MESH_REGISTRY host:port to publish artifacts to, when the mesh has not said
MESH_BINDING a file the mesh wrote saying where the artifact store is
MESH_PACKAGE_BINDING a file the mesh wrote saying where the package registry is
MESH_NPM_TOKEN_FILE a file the mesh sealed holding the token for it
MESH_GENESIS_PACKAGE_REGISTRY the document the installer sealed at genesis, read while no
binding is present (novox/hq ADR 0336)
MESH_NPM_REGISTRY a package registry URL, when the mesh has not said (a person, the bootstrap)
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
MESH_WORKSPACE where to clone and build (default: a temporary directory)
It also builds one module and stops, which is how a mesh is raised — before there is a
broker to take work from or a registry to publish into:
mesh-builder build <repository> [--path P] [--ref COMMIT] [--registry HOST:PORT]
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
mesh-builder busy
exits 0 while the agent running in this container has a build in hand, and 1 when it has none;
from the first ask the agent takes no new build: its container's busy command, asked before the
container is replaced (novox/hq ADR 0335).
`
func run() error {
if len(os.Args) > 1 {
switch os.Args[1] {
case "version":
fmt.Println(version)
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work, and
// once it has been drained longer than its node-engine would wait to replace it (ADR 0335).
if err := ready(); err != nil {
return err
}
return drainedTooLong(time.Now())
case "busy":
// The container's busy command (novox/hq ADR 0335): drains the agent, and exits non-zero once
// nothing is in hand.
said, err := busy()
if err != nil {
return err
}
fmt.Println(said)
return nil
case "help", "-h", "--help":
fmt.Print(usage)
return nil
default:
// **An ask this build does not know fails** (novox/hq ADR 0335): an image older than `busy`, asked
// it, must read as idle — exiting 0 would hold its container for the node-engine's whole bound.
fmt.Fprint(os.Stderr, usage)
return fmt.Errorf("%q is not a command this mesh-builder knows", os.Args[1])
}
}
if err := notReady(); err != nil {
return err
}
// **The builder's own version is part of every reuse key** (novox/hq ADR 0350 §1): a builder that builds
// differently never takes a build another one made.
builder.Version = builder.SelfVersion(version)
if err := undrained(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
}
registry, err := whereToPublish()
if err != nil {
return err
}
workspace := os.Getenv("MESH_WORKSPACE")
if workspace == "" {
workspace = os.TempDir() + "/mesh-builder"
}
// **The mesh's name for this machine, not the container's.** A build is reported to the rest
// of the mesh, and a report whose origin reads `104cb10e105b` names something no other module
// can look up. The mesh already knows the answer and has a way to say it — `${machine:name}`
// in the environment file this module is handed — so the hostname is only what is left when
// nobody said.
on := os.Getenv("MESH_NODE")
if on == "" {
hostname, err := os.Hostname()
if err != nil {
return fmt.Errorf("this build machine has no name: nothing said MESH_NODE and the host would not say either: %w", err)
}
on = hostname
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
js, seat, err := dialFor(credential)
if err != nil {
return err
}
defer js.Close()
// **This machine's holder: paused or not, and the build it is running** (novox/hq ADR 0219). The
// paused flag is read from the workspace before anything is taken, so a holder restarted while
// paused takes nothing.
h := newHolder(on, seat, workspace, func(state link.HolderState) error {
body, err := json.Marshal(state)
if err != nil {
return err
}
_, err = js.Context().Publish(link.BuildPausedOf(seat, on), body)
return err
})
if h.Paused() {
fmt.Fprintf(os.Stderr, "paused (kept in %s): taking no build until resumed\n", pausedFile(workspace))
}
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Taking: func() { taking(seat) },
Claim: claim,
})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
// one serves none, and a subscription its holder has no grant for would be refused for ever.
if seat == link.TheBuildMachine {
stopServing, err := link.OverNATS{Conn: js.Conn()}.ServeNodeSeatTools(seat, on, h.handlers(),
log.New(os.Stderr, "", 0))
if err != nil {
return err
}
defer stopServing()
// And says so, for the console to find (novox/hq ADR 0197).
stopAnnouncing, err := link.OverNATS{Conn: js.Conn()}.Announce(
announcement(seat, moduleOf(credential.User), on), log.New(os.Stderr, "", 0))
if err != nil {
return err
}
defer stopAnnouncing()
go h.stateWhilePaused(ctx, time.Hour)
}
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command}
// Check builds are let go of after a week (novox/hq ADR 0350 §5).
h.scratch = publisher
go h.expireChecks(ctx, time.Hour, time.Now)
return machine.Take(ctx, func(ctx context.Context, work link.Build) {
defer released()
holding(work.Request().Repository + " (" + work.Request().ID + ")")
answer(ctx, publisher, on, workspace, work, h)
})
}
// dialFor opens this machine's link to whichever bus the mesh is on, and says which build seat it
// holds.
//
// **One place chooses**, as everywhere else the bus change went (novox/hq ADR 0116 step 5): a build
// machine told about both would take work from one and answer on the other, and every log line would
// say it was fine.
func dialFor(credential Credential) (*broker.JetStream, string, error) {
// **The credential names the bus, and there is one** (novox/hq ADR 0131, design 28 task 5.5).
// A credential for the mesh's bus carries user, password and fingerprint beside the address,
// and that is enough to dial it, pinned.
if !credential.onTheNewBus() {
return nil, "", fmt.Errorf("the credential at hand names %q, which is not the mesh's bus", credential.URL)
}
// Named for what it is, not the controller whose code dials it (novox/hq issue 327).
host, _ := os.Hostname()
js, err := broker.DialPinned(credential.natsURL(), credential.Fingerprint, nats.Name("build agent on "+host))
if err != nil {
return nil, "", err
}
// **The seat this machine serves is the one its credential claims** (novox/hq ADR 0190, the
// handover): the mesh issues a build machine's credential naming the seat its module claims,
// and one binary serves the old role as `builder` and the new as `build-agent` from that alone.
seat := link.BuildSeatClaimed(credential.seatsClaimed())
fmt.Fprintf(os.Stderr, "taking build work as a holder of %s\n", seat)
return js, seat, nil
}
// answer does one build and says what happened, whichever way it went.
func answer(ctx context.Context, publisher builder.Publisher, on, workspace string, work link.Build, h *holder) {
request := work.Request()
// **Its own context, so it can be killed alone** (novox/hq ADR 0219): cancelled by `kill`, it ends
// this build's commands and nothing else; the machine's own context ending — a SIGTERM — still
// reaches it through the parent, and that keeps today's meaning below.
building, cancel := context.WithCancel(ctx)
defer cancel()
var mine *running
if h != nil {
mine = h.begin(request, cancel)
defer h.end(mine)
}
// **First thing, and to stdout.** A build request that arrives and produces no visible line until
// it either finishes or fails is indistinguishable from one that never arrived — which cost a long
// diagnosis against a running mesh, chasing "the handler never fired" when the truth was only that
// the handler said nothing until the end.
fmt.Fprintf(os.Stderr, "a build request arrived for %s (%s)\n", request.Repository, request.ID)
// **Everything a build says goes two ways**: to stderr, as always, and onto the bus as the
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
h.stepped(mine, step)
}
}
builder.Said = say
defer func() { builder.Said = nil }()
if err := work.Began(ctx); err != nil {
fmt.Fprintf(os.Stderr, "cannot say a build started: %v\n", err)
}
// **A running check watches its pull request's key** (novox/hq ADR 0406 §4, issue 974): when the head or the trunk
// commit in it moves, the check is for a state nobody wants any more, and ends itself — nothing is asked of the forge
// (issue 726). A group's check, a try and a build watch nothing.
var superseded atomic.Bool
if w, ok := work.(interface {
WatchPull(context.Context, func(link.PullKey)) (func(), error)
}); ok && request.Check != nil {
stopWatching, err := w.WatchPull(building, func(k link.PullKey) {
superseded.Store(true)
say("superseded", fmt.Sprintf("the pull request's key moved to %.12s on %.12s while this checked %.12s on %.12s: stopped",
k.Head, k.Trunk, request.Ref, request.Check.Trunk))
cancel()
})
if err != nil {
fmt.Fprintf(os.Stderr, "the key of %s/%s#%d could not be watched, so its check runs on if the key moves: %v\n",
request.Check.Owner, request.Check.Repo, request.Check.Number, err)
} else {
defer stopWatching()
}
}
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
Ref: request.Ref, On: on, Source: request.Source, DryRun: request.DryRun,
}
what := "building " + request.Repository
if request.Path != "" {
what += " at " + request.Path
}
if request.Ref != "" {
what += " on " + request.Ref
}
say("build", what)
npmrc, err := packagesFrom(on)
if err == nil {
say("packages", saysRegistry(npmrc))
}
registryHere, isRegistry := publisher.(builder.Registry)
var built builder.Result
if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
// published or registered; the verdict is the outcome.
result.Checked = request.Check
registry := ""
if r, ok := publisher.(builder.Registry); ok {
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Validator: v.Validator, Siblings: siblingsOut(v.Siblings), SiblingsUnknown: v.SiblingsUnknown, JudgeFrom: v.JudgeFrom}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
// build that could not have resolved its dependencies is refused in front of the reason, not
// after a clone that then fails at npm ci.
// Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219).
if request.Reuse != nil && isRegistry && !request.DryRun {
// **A walk's build takes the check's build of the same key** (novox/hq ADR 0350 §2).
reuse := builder.Reuse{Scratch: registryHere}
for _, p := range request.Reuse.Pulls {
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
} else {
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
}
// What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored
// Only a build the kill ended: the kill came before its work did. One that finished — built, or
// failed on its own — in the moment the kill arrived says what it did, and the kill is refused.
killed := false
if mine != nil {
killed = h.returned(mine) && err != nil
}
if superseded.Load() && err != nil {
// Ended by its own key moving on (ADR 0406 §4): said as the check's failure, as a kill is.
result.Failed = link.SupersededByKey
say("failed", link.SupersededByKey)
} else if killed {
// **Killed by hand is the outcome, whatever the build was doing** (novox/hq ADR 0219): the
// error it ended with is the kill's consequence, not a fault of the source.
result.Failed = link.KilledByHand
say("failed", link.KilledByHand)
} else if err != nil {
// A failure is a result. A build that fails and says nothing is indistinguishable from a
// builder that is not running, and those want completely different responses.
result.Failed = err.Error()
say("failed", err.Error())
} else if request.Check != nil {
say("checked", result.Check.Verdict+": "+result.Check.Summary)
} else {
manifest, marshalErr := json.Marshal(built.Manifest)
if marshalErr != nil {
result.Failed = marshalErr.Error()
} else {
result.Commit = built.Commit
result.Manifest = manifest
for _, made := range built.Built {
result.Made = append(result.Made, link.MadeArtifact{
Name: made.Name, Kind: made.Kind, Reference: made.Reference,
})
}
result.Against = built.Against
result.SourceFingerprint = built.Source
result.SourceInputs = built.SourceInputs
// What the walk did instead of building, or why it built (novox/hq ADR 0350 §3, §4).
result.ReuseKey = built.ReuseKey
if built.Reused != nil {
f := built.Reused.From
result.Reused = built.Reused.Words()
result.ReusedFrom = &link.ReusedFrom{Pull: link.PullRef{Owner: f.Pull.Owner, Repo: f.Pull.Repo,
Number: f.Pull.Number}, Head: f.Head, Trunk: f.Trunk, Check: f.Check, Key: f.Key}
} else if request.Reuse != nil {
result.BuiltWhy = built.Why
if result.BuiltWhy == "" {
result.BuiltWhy = "it has no artifact to build"
}
}
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
// A killed build is announced on a context of its own: the build's was the one cancelled, and the
// outcome must go out and the ask be settled — acknowledged, never redelivered to another machine
// to be built again. A machine being stopped is the other case and keeps its meaning: the
// parent's context is gone, nothing is announced or settled, and the ask is redelivered.
announcing := ctx
if killed {
fresh, stop := context.WithTimeout(context.Background(), 30*time.Second)
defer stop()
announcing = fresh
}
announceErr := work.Announce(announcing, result)
if mine != nil {
h.said(mine, result.Failed, announceErr == nil)
}
if err := announceErr; err != nil {
// Said, not fatal: the build happened. A build reported as failed because announcing it
// failed is a lie about work that was done — and the request stays unsettled below only if
// nothing was said at all, so another machine can try.
fmt.Fprintf(os.Stderr, "cannot say what came of a build: %v\n", err)
return
}
// Settled only once the outcome is away, so a machine that dies before answering leaves the work
// for another rather than losing it.
if err := work.Done(); err != nil {
fmt.Fprintf(os.Stderr, "the outcome is away and the request could not be settled: %v\n", err)
}
}
// checkSpecOf is a check request as the builder runs it.
func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
c := request.Check
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
Ref: b.Ref, Merged: b.Merged, Tier: b.Tier, Skip: b.Skip})
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
}
return spec
}
// siblingsOut is the siblings a check read as the outcome carries them (novox/hq ADR 0382 §4).
func siblingsOut(read []builder.JudgedSibling) []link.CheckSibling {
var out []link.CheckSibling
for _, s := range read {
out = append(out, link.CheckSibling{Repository: s.Repository, Number: s.Number, Head: s.Head, Base: s.Base,
BaseCommit: s.BaseCommit})
}
return out
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
out = append(out, c)
}
return out
}
// layerOf is one layer of a check as the outcome carries it.
func layerOf(l *builder.Layer) *link.CheckLayer {
if l == nil {
return nil
}
return &link.CheckLayer{Verdict: l.Verdict, Summary: l.Summary, Modules: l.Modules}
}
// packagesFrom is where a build resolves the mesh's own published packages — the SDK above all
// (novox/hq ADR 0076, issue 053) — on this machine.
//
// **The binding wins, then the genesis document** (novox/hq ADR 0336 §3, §4). The build agent's
// npm-package-registry requirement is optional: on genesis's node, before the gitea module is assigned,
// nothing provides it, and the installer sealed the forge it raised to this agent as one document
// instead. Once the requirement binds, the binding names the endpoint the way the artifact store's
// does and a sealed token file the credential, and the genesis document is left unread. A binding
// file named and absent is a requirement not bound here, never a fault: the environment names the
// path whether or not the mesh wrote it.
//
// The environment variables remain for a builder run by a person. With none of the three the result
// is disabled and says this machine, so a build that needs the registry is refused naming it (§6),
// and a build that needs none builds anyway.
func packagesFrom(machine string) (builder.Npmrc, error) {
scope := strings.TrimSpace(os.Getenv("MESH_NPM_SCOPE"))
if scope == "" {
scope = "@novox"
}
none := builder.Npmrc{Machine: machine}
if path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not bound on this machine: the genesis document, or nothing.
case err != nil:
return none, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
default:
return fromBinding(path, raw, scope, machine)
}
}
if path := strings.TrimSpace(os.Getenv("MESH_GENESIS_PACKAGE_REGISTRY")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not this mesh's genesis node, or the installer has not sealed it yet.
case err != nil:
return none, fmt.Errorf("cannot read the genesis document for the package registry: %w", err)
default:
return fromGenesis(path, raw, scope, machine)
}
}
// A builder run by a person.
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
username := strings.TrimSpace(os.Getenv("MESH_NPM_USER"))
if registry == "" && secret == "" {
return none, nil
}
n := builder.Npmrc{Scope: scope, Registry: registry, From: builder.FromEnvironment, Machine: machine}
if username != "" {
n.Username, n.Password = username, secret
} else {
n.Token = secret
}
return n, nil
}
// fromBinding composes the registry from the npm-package-registry binding the mesh wrote and the
// credential it sealed beside it. The mesh authenticates the ordinary way — a generated password the
// provider only applies (novox/hq ADR 0048) — so with a username the secret is a password (basic
// auth); without one it is a bearer token a provider minted.
func fromBinding(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return none, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return none, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return none, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return none, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
raw, err := os.ReadFile(file)
if err != nil {
return none, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
}
secret = strings.TrimSpace(string(raw))
}
n := builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath),
From: builder.FromBinding, Machine: machine}
if told.As != "" {
n.Username, n.Password = told.As, secret
} else {
n.Token = secret
}
return n, nil
}
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): what the installer sealed to the
// build agent on its node at genesis's packages step, as the `genesis-package-registry` own-secret —
// the forge it raised, the port this node gave the package registry, its npm path, and the account it
// made with that account's password. mesh-host writes it; the field names are the two repositories'
// agreement.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// fromGenesis composes the registry from the genesis document.
func fromGenesis(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var g GenesisRegistry
if err := json.Unmarshal(raw, &g); err != nil {
return none, fmt.Errorf("%s is not the genesis document for the package registry: %w", path, err)
}
var missing []string
for name, value := range map[string]string{"at": g.At, "npm-path": g.NpmPath, "as": g.As, "password": g.Password} {
if strings.TrimSpace(value) == "" {
missing = append(missing, name)
}
}
if g.Port <= 0 {
missing = append(missing, "port")
}
if len(missing) > 0 {
sort.Strings(missing)
return none, fmt.Errorf("%s, the genesis document for the package registry, says nothing of %s",
path, strings.Join(missing, ", "))
}
scheme := g.Scheme
if scheme == "" {
scheme = "http"
}
return builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%d%s", scheme, g.At, g.Port, g.NpmPath),
Username: g.As, Password: g.Password, From: builder.FromGenesis, Machine: machine}, nil
}
// saysRegistry is the line each build's log carries about the package registry (novox/hq ADR 0336
// §4): which of the binding and the genesis document it resolved from, or that it has neither here.
func saysRegistry(n builder.Npmrc) string {
if n.Enabled() {
return "the package registry is resolved from " + n.From
}
machine := n.Machine
if machine == "" {
machine = "this machine"
}
return fmt.Sprintf("no package registry: %s is unbound on %s and no genesis document is here; "+
"a build needing it is refused", builder.RegistryRequirement, machine)
}
// forgeFrom is the git credential this builder may offer a clone, composed from the same binding
// and sealed secret its package-registry half already reads: the forge that answers npm is the
// forge that hosts the repositories, and its provisioner applies one password to one user for
// both. Anything missing means no credential, and every clone stays anonymous — which is all a
// mesh of public repositories ever needs.
//
// The URL names the binding's own address — the machine the mesh says the forge is on — so a
// private repository is registered and built by that address, and a clone of anything else is
// never shown this credential (git's credential store matches the whole origin).
func forgeFrom() builder.GitCredential {
path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING"))
if path == "" {
return builder.GitCredential{}
}
raw, err := os.ReadFile(path)
if err != nil {
return builder.GitCredential{}
}
var told struct {
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil || told.At == "" || told.As == "" {
return builder.GitCredential{}
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
if raw, err := os.ReadFile(file); err == nil {
secret = strings.TrimSpace(string(raw))
}
}
if secret == "" {
return builder.GitCredential{}
}
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
host := told.At
if port, ok := told.Serves["port"]; ok {
host = fmt.Sprintf("%s:%v", told.At, port)
}
made := url.URL{Scheme: scheme, User: url.UserPassword(told.As, secret), Host: host}
return builder.GitCredential{URL: made.String()}
}
func short(commit string) string {
if len(commit) > 8 {
return commit[:8]
}
return commit
}
// whereToPublish is the artifact store this builder uses.
//
// **Preferably from the mesh.** A builder that is a module requires an artifact store, and the
// mesh writes it a file saying which machine answers that and on what port — the same binding any
// consumer of any provision gets. Reading it means the address is not a setting somebody keeps in
// step by hand, and moving the store is an ordinary reassignment rather than an edit on every
// build machine.
//
// The environment variable remains for a builder run by a person, which is how this started and
// how it is still run while being developed.
func whereToPublish() (string, error) {
binding := strings.TrimSpace(os.Getenv("MESH_BINDING"))
if binding == "" {
registry := strings.TrimSpace(os.Getenv("MESH_REGISTRY"))
if registry == "" {
return "", fmt.Errorf("neither MESH_BINDING nor MESH_REGISTRY: a built artifact " +
"nobody can fetch is not built")
}
return registry, nil
}
raw, err := os.ReadFile(binding)
if err != nil {
return "", fmt.Errorf("cannot read what the mesh said about the artifact store: %w", err)
}
var told struct {
From string `json:"from"`
At string `json:"at"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return "", fmt.Errorf("%s is not a binding: %w", binding, err)
}
if told.At == "" {
// The provider is not on the private network, so there is no name to reach it by. Said
// rather than falling back to the machine's own name, which would publish to a store on
// the wrong machine and be found out much later.
return "", fmt.Errorf(
"%s says the artifact store is on %q and gives no address for it", binding, told.From)
}
port, ok := told.Serves["port"]
if !ok {
return "", fmt.Errorf("%s says nothing about which port the artifact store answers on",
binding)
}
return fmt.Sprintf("%s:%v", told.At, port), nil
}
// brokerFrom is where this builder connects, and with what.
//
// **Preferably from a file the mesh sealed to this machine.** A builder that is a module is given
// its credential the way every other module is given one: generated or accepted centrally, sealed
// to the machine, written by the host. Putting it in an environment variable instead would mean
// the one copy that matters passing through a terminal and a process listing.
//
// The variable remains for a builder run by a person.
func brokerFrom() (Credential, error) {
if path := strings.TrimSpace(os.Getenv("MESH_BROKER_FILE")); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
return Credential{}, fmt.Errorf("cannot read this builder's credential: %w", err)
}
said := strings.TrimSpace(string(raw))
if said == "" {
// An empty credential file is a machine that will connect as nobody and be refused,
// with the reason three layers away.
return Credential{}, fmt.Errorf("%s is empty, so this builder has no credential", path)
}
var held Credential
if err := json.Unmarshal([]byte(said), &held); err == nil && held.URL != "" {
return held, nil
}
// A file holding only a URL, which is what a person writing one by hand produces. The
// broker is then verified against whatever this machine already trusts.
return Credential{URL: said}, nil
}
return Credential{}, fmt.Errorf(
"no MESH_BROKER_FILE: a build machine with no credential for the bus has nothing to build")
}
// Credential is what a build machine is given so it can reach the broker.
//
// Two things, because reaching a broker over TLS needs both: who to connect as, and what to check
// the certificate against. A mesh's broker presents a certificate of the mesh's own, which is in
// no public trust store, so a URL alone can only connect to a broker somebody else vouches for.
//
// **The same shape a node gets, for the same reason** (novox/hq ADR 0004): the fingerprint travels
// out of band — here, sealed with the credential — and the endpoint is verified once at connect.
type Credential struct {
URL string `json:"url"`
Fingerprint string `json:"fingerprint,omitempty"`
// User and Password ride beside the address on the bus being built (design 25): a credential
// embedded in a URL leaks into every log line that prints a connection, so the mesh seals them
// as two fields and this machine joins them once, here, to dial.
User string `json:"user,omitempty"`
Password string `json:"password,omitempty"`
// Claims are the seats the module this credential was issued for claims, as the mesh writes
// them beside the credential (novox/hq ADR 0159). The first is the build role this machine
// serves; a credential naming none is from before claims travelled in it.
Claims []struct {
Seat string `json:"seat"`
} `json:"claims,omitempty"`
}
// seatsClaimed is the seats the credential names, in order.
func (c Credential) seatsClaimed() []string {
out := make([]string, 0, len(c.Claims))
for _, claim := range c.Claims {
out = append(out, claim.Seat)
}
return out
}
// onTheNewBus is whether a credential is for the bus being built: its address says so, and the
// mesh only ever seals such a credential with the user and password beside it.
func (c Credential) onTheNewBus() bool { return strings.HasPrefix(strings.TrimSpace(c.URL), "nats://") }
// natsURL is the address with this machine's credential in it, for the one dial that needs it.
func (c Credential) natsURL() string {
rest := strings.TrimPrefix(strings.TrimSpace(c.URL), "nats://")
if c.User == "" {
return "nats://" + rest
}
return "nats://" + c.User + ":" + c.Password + "@" + rest
}
-188
View File
@@ -1,188 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"github.com/novox/mesh-controller/internal/builder"
)
// buildOnce is the builder doing one build and stopping, with no broker and no mesh.
//
// **This is how a mesh is raised** (novox/hq ADR 0073). The installer carries this program and runs
// it once, before anything else exists, to produce the control plane from the same repository and
// path that every later rebuild of it will use. What raises the mesh is therefore the same thing
// that maintains it — not a second mechanism that has to be kept in step with the first and is
// exercised once per new mesh, which is how often enough to rot.
//
// It takes no work from a queue and answers nobody: there is no broker yet, and the only thing
// waiting for the answer is the installer that started it. So the result goes to standard output as
// JSON, which is what the installer reads.
//
// With no --registry it publishes nowhere and the image stays in this machine's container runtime,
// named by the digest of its own configuration (see builder.Local). That is the genesis case. With
// one, it publishes as it always does — the same command is how an operator builds a module by hand
// on a mesh that already exists.
func buildOnce(ctx context.Context, args []string) error {
set := flag.NewFlagSet("build", flag.ContinueOnError)
path := set.String("path", "", "the module's directory inside the repository (default: its root)")
ref := set.String("ref", "", "the commit to build; a branch is a moving target somebody else controls")
registry := set.String("registry", "",
"host:port to publish to. Without it the artifacts stay in this machine's container runtime, which is the genesis case")
workspace := set.String("workspace", "", "where to clone and build (default: a temporary directory)")
on := multiple(set, "on",
"a base this build stands on, as <module>/<artifact>=<reference>. Repeatable")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 1 {
return errors.New("mesh-builder build <repository> [--path P] [--ref COMMIT] [--registry HOST:PORT]")
}
repository := positionals[0]
where := *workspace
if where == "" {
where = os.TempDir() + "/mesh-builder-once"
}
// What this build may stand on, said on the command line because there is no mesh to ask.
//
// **This is the genesis case and it is meant to be awkward.** On a running mesh the control
// plane answers this, because only it knows what this mesh holds. Here nothing has been built
// yet, so whoever runs this says what to use — and for the control plane, which is the one
// module raised before anything else exists, the answer is ordinarily nothing at all.
bases := map[string]string{}
for _, pair := range *on {
key, reference, found := strings.Cut(pair, "=")
if !found || key == "" || reference == "" {
return fmt.Errorf(
"--on takes <module>/<artifact>=<reference>, and %q is not that", pair)
}
bases[key] = reference
}
// Local unless told otherwise, because the moment this exists for has nowhere to publish. A
// default pointing at a registry would mean genesis failing at a push to something that is not
// there yet, one step away from the thing that could explain it.
var publisher builder.Publisher = builder.Local{Run: builder.Command}
if *registry != "" {
publisher = builder.Registry{Address: *registry, Run: builder.Command}
}
fmt.Fprintf(os.Stderr, "building %s", repository)
if *path != "" {
fmt.Fprintf(os.Stderr, " at %s", *path)
}
if *ref != "" {
fmt.Fprintf(os.Stderr, " at %s", *ref)
}
fmt.Fprintln(os.Stderr)
machine := os.Getenv("MESH_NODE")
if machine == "" {
machine, _ = os.Hostname()
}
npmrc, err := packagesFrom(machine)
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, " [packages] %s\n", saysRegistry(npmrc))
built, buildErr := builder.Build(ctx, builder.Command, publisher, repository, *path, *ref, where, bases, npmrc,
forgeFrom(),
func(step, message string) { fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) })
if buildErr != nil {
return buildErr
}
// To standard output, and everything else to standard error, so the caller can read this
// without having to separate it from progress.
out := onceResult{
Module: built.Manifest.Module,
Commit: built.Commit,
Repository: repository,
Path: *path,
Ref: *ref,
Manifest: built.Manifest,
Against: built.Against,
Source: built.Source,
}
for _, r := range built.Read {
out.Read = append(out.Read, readRepository{Repository: r.Repository, Ref: r.Ref})
}
for _, made := range built.Built {
out.Made = append(out.Made, madeArtifact{Name: made.Name, Kind: made.Kind, Reference: made.Reference})
}
body, err := json.MarshalIndent(out, "", " ")
if err != nil {
return err
}
fmt.Println(string(body))
fmt.Fprintf(os.Stderr, " built %s from %s\n", built.Manifest.Module, short(built.Commit))
return nil
}
// onceResult is what one build reports to whoever started it.
//
// The same fields the mesh records for a build, so a reader comparing a genesis build against an
// ordinary one is comparing the same thing said the same way.
type onceResult struct {
Module string `json:"module"`
Commit string `json:"commit"`
Repository string `json:"repository"`
Path string `json:"path,omitempty"`
Ref string `json:"ref,omitempty"`
Manifest any `json:"manifest"`
Made []madeArtifact `json:"made"`
Against []string `json:"against,omitempty"`
Read []readRepository `json:"read,omitempty"`
Source string `json:"source-fingerprint,omitempty"`
}
// readRepository is a repository this build read source from besides the module's own.
type readRepository struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
}
type madeArtifact struct {
Name string `json:"name"`
Kind string `json:"kind"`
Reference string `json:"reference"`
}
// parseAround lets flags appear on either side of the repository, because a person writing this by
// hand will put them wherever reads best and the standard parser stops at the first thing that is
// not a flag. The same helper the control plane's commands use, for the same reason.
func parseAround(set *flag.FlagSet, args []string) ([]string, error) {
var positionals []string
rest := args
for {
if err := set.Parse(rest); err != nil {
return nil, err
}
rest = set.Args()
if len(rest) == 0 {
return positionals, nil
}
positionals = append(positionals, rest[0])
rest = rest[1:]
}
}
// multiple is a flag that may be given more than once.
type repeated []string
func (r *repeated) String() string { return strings.Join(*r, ", ") }
func (r *repeated) Set(v string) error { *r = append(*r, v); return nil }
func multiple(set *flag.FlagSet, name, usage string) *[]string {
var values repeated
set.Var(&values, name, usage)
return (*[]string)(&values)
}
-102
View File
@@ -1,102 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/builder"
)
// The build agent's package registry (novox/hq ADR 0336, issue 121): its npm-package-registry
// requirement is optional, so the binding may be absent, and on genesis's node the installer seals
// the forge it raised as one genesis document instead.
// registryFiles names the three paths the build agent's environment always names, writing only those
// given.
func registryFiles(t *testing.T, binding, token, genesis string) {
t.Helper()
dir := t.TempDir()
for env, content := range map[string]string{
"MESH_PACKAGE_BINDING": binding,
"MESH_NPM_TOKEN_FILE": token,
"MESH_GENESIS_PACKAGE_REGISTRY": genesis,
} {
path := filepath.Join(dir, env)
t.Setenv(env, path)
if content != "" {
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
}
}
for _, env := range []string{"MESH_NPM_REGISTRY", "MESH_NPM_TOKEN", "MESH_NPM_USER", "MESH_NPM_SCOPE"} {
t.Setenv(env, "")
}
}
const (
aBinding = `{"from":"gitea","at":"anchor.mesh","as":"anchor-agent",
"serves":{"scheme":"http","port":3000,"npm-path":"/api/packages/novox/npm/"}}`
aGenesisDocument = `{"scheme":"http","at":"127.0.0.1","port":3100,"npm-path":"/api/packages/novox/npm/",
"as":"mesh-builder","password":"genesis-secret"}`
)
func TestUntilTheRequirementBindsTheGenesisDocumentIsRead(t *testing.T) {
registryFiles(t, "", "", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatalf("an absent binding beside a genesis document was refused: %v", err)
}
if n.From != builder.FromGenesis || n.Registry != "http://127.0.0.1:3100/api/packages/novox/npm/" ||
n.Username != "mesh-builder" || n.Password != "genesis-secret" || n.Scope != "@novox" {
t.Fatalf("the genesis document was not what the registry came from: %+v", n)
}
file, err := n.File()
if err != nil || !strings.Contains(file, "//127.0.0.1:3100/api/packages/novox/npm/:username=mesh-builder") {
t.Fatalf("the .npmrc is not written from the genesis document: %v\n%s", err, file)
}
if got := saysRegistry(n); !strings.Contains(got, "genesis document") {
t.Errorf("the log does not say the genesis document: %s", got)
}
}
func TestARealBindingWinsOverTheGenesisDocument(t *testing.T) {
registryFiles(t, aBinding, "bound-secret\n", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatal(err)
}
if n.From != builder.FromBinding || n.Registry != "http://anchor.mesh:3000/api/packages/novox/npm/" ||
n.Username != "anchor-agent" || n.Password != "bound-secret" {
t.Fatalf("the binding did not win over the genesis document: %+v", n)
}
if got := saysRegistry(n); !strings.Contains(got, "binding") {
t.Errorf("the log does not say the binding: %s", got)
}
}
func TestWithNeitherTheAgentHasNoRegistryAndSaysWhere(t *testing.T) {
registryFiles(t, "", "", "")
n, err := packagesFrom("workstation")
if err != nil {
t.Fatalf("an unbound optional requirement was a fault: %v", err)
}
if n.Enabled() || n.Machine != "workstation" {
t.Fatalf("a registry came from nowhere: %+v", n)
}
got := saysRegistry(n)
for _, want := range []string{"npm-package-registry", "unbound", "workstation"} {
if !strings.Contains(got, want) {
t.Errorf("the log does not say %q: %s", want, got)
}
}
}
func TestAGenesisDocumentMissingAFieldIsRefusedByName(t *testing.T) {
registryFiles(t, "", "", `{"at":"127.0.0.1","port":3000,"npm-path":"/api/packages/novox/npm/","as":"mesh-builder"}`)
if _, err := packagesFrom("anchor"); err == nil || !strings.Contains(err.Error(), "password") {
t.Fatalf("a genesis document with no password was taken: %v", err)
}
}
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
-32
View File
@@ -1,32 +0,0 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
-27
View File
@@ -1,27 +0,0 @@
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// The seat a build machine serves comes from its credential (novox/hq ADR 0190 handover).
func TestTheCredentialSaysWhichBuildRoleThisMachineServes(t *testing.T) {
var held Credential
if err := json.Unmarshal([]byte(`{"url":"nats://bus:4222","user":"anchor.builder","password":"x",
"claims":[{"seat":"mesh-build-machine","scope":"mesh","serves":[]}]}`), &held); err != nil {
t.Fatal(err)
}
if got := link.BuildSeatClaimed(held.seatsClaimed()); got != "mesh-build-machine" {
t.Errorf("the old builder's credential serves %q", got)
}
var bare Credential
if err := json.Unmarshal([]byte(`{"url":"nats://bus:4222","user":"anchor.build-agent","password":"x"}`), &bare); err != nil {
t.Fatal(err)
}
if got := link.BuildSeatClaimed(bare.seatsClaimed()); got != link.TheBuildMachine {
t.Errorf("a credential without claims serves %q, want %s", got, link.TheBuildMachine)
}
}
-45
View File
@@ -1,45 +0,0 @@
package main
import (
"os"
"strings"
"testing"
)
// **The genesis path writes its result to stdout and nothing else there.** The installer captures
// stdout alone and parses it as JSON; one stray log line makes that fail with "invalid character"
// — exactly what a builder fmt.Printf caused, breaking a build that had worked. This keeps
// diagnostics on stderr so a future print cannot repeat it silently.
func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
allowed := map[string]bool{
"string(body)": true, // once.go: the result JSON, which IS stdout
"version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
"said)": true, // busy: the answer its node-engine reads (novox/hq ADR 0335)
}
for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file)
if err != nil {
t.Fatal(err)
}
for i, raw := range strings.Split(string(src), "\n") {
line := strings.TrimSpace(raw)
if !strings.HasPrefix(line, "fmt.Printf(") &&
!strings.HasPrefix(line, "fmt.Println(") &&
!strings.HasPrefix(line, "fmt.Print(") {
continue
}
ok := false
for token := range allowed {
if strings.Contains(line, token) {
ok = true
}
}
if !ok {
t.Errorf("%s:%d writes a diagnostic to stdout, which the installer parses as JSON:\n %s",
file, i+1, line)
}
}
}
}
-31
View File
@@ -1,31 +0,0 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
-225
View File
@@ -1,225 +0,0 @@
package main
import (
"crypto/ed25519"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"math/big"
"net"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
)
// Where a builder publishes.
//
// Preferably from the mesh: a builder that is a module requires an artifact store, and the mesh
// writes it a binding saying which machine answers and on what port. Reading it means the address
// is not a setting somebody keeps in step by hand.
func binding(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "artifact-store.json")
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
return path
}
func TestTheMeshSaysWhereToPublish(t *testing.T) {
t.Setenv("MESH_BINDING", binding(t, `{"binding":1,"provision":"artifact-store",
"from":"anchor","at":"anchor.internal","serves":{"port":5000,"scheme":"http"}}`))
where, err := whereToPublish()
if err != nil {
t.Fatal(err)
}
if where != "anchor.internal:5000" {
t.Fatalf("got %q", where)
}
}
func TestABindingWithNoAddressIsRefused(t *testing.T) {
// The provider is not on the private network, so there is no name to reach it by. Falling
// back to anything would publish to a store on the wrong machine and be found out much later.
t.Setenv("MESH_BINDING", binding(t, `{"binding":1,"from":"anchor","serves":{"port":5000}}`))
_, err := whereToPublish()
if err == nil {
t.Fatal("a binding with nowhere to reach was accepted")
}
if !strings.Contains(err.Error(), "anchor") {
t.Fatalf("the failure does not name the machine: %v", err)
}
}
func TestABindingWithNoPortIsRefused(t *testing.T) {
t.Setenv("MESH_BINDING", binding(t, `{"binding":1,"from":"a","at":"a.internal","serves":{}}`))
if _, err := whereToPublish(); err == nil {
t.Fatal("a binding saying nothing about a port was accepted")
}
}
func TestTheVariableStillWorksForABuilderRunByAPerson(t *testing.T) {
// Which is how this started and how it is still run while being developed.
t.Setenv("MESH_BINDING", "")
t.Setenv("MESH_REGISTRY", "127.0.0.1:5000")
where, err := whereToPublish()
if err != nil {
t.Fatal(err)
}
if where != "127.0.0.1:5000" {
t.Fatalf("got %q", where)
}
}
func TestNeitherIsRefusedRatherThanGuessed(t *testing.T) {
t.Setenv("MESH_BINDING", "")
t.Setenv("MESH_REGISTRY", "")
if _, err := whereToPublish(); err == nil {
t.Fatal("a builder with nowhere to publish reported somewhere")
}
}
func TestTheCredentialComesFromAFileTheMeshSealed(t *testing.T) {
// A builder that is a module is given its credential the way every module is: sealed to the
// machine and written by the host. An environment variable instead would put the one copy
// that matters through a terminal and a process listing.
path := filepath.Join(t.TempDir(), "broker")
if err := os.WriteFile(path, []byte("amqps://a-builder:secret@broker.internal:5671/\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("MESH_BROKER_FILE", path)
t.Setenv("MESH_BROKER_AMQP", "amqp://should-not-be-used@nowhere/")
got, err := brokerFrom()
if err != nil {
t.Fatal(err)
}
if got.URL != "amqps://a-builder:secret@broker.internal:5671/" {
t.Fatalf("got %q", got.URL)
}
}
// The credential the mesh seals carries what to check the broker's certificate against, because a
// mesh's broker presents a certificate of the mesh's own and no public trust store has it. A URL
// alone can only reach a broker somebody else vouches for.
func TestTheSealedCredentialCarriesWhatVerifiesTheBroker(t *testing.T) {
path := filepath.Join(t.TempDir(), "broker")
if err := os.WriteFile(path, []byte(
`{"url":"amqps://a-builder:secret@broker.internal:5671/","fingerprint":"abc123"}`),
0o600); err != nil {
t.Fatal(err)
}
t.Setenv("MESH_BROKER_FILE", path)
t.Setenv("MESH_BROKER_AMQP", "")
got, err := brokerFrom()
if err != nil {
t.Fatal(err)
}
if got.URL != "amqps://a-builder:secret@broker.internal:5671/" {
t.Fatalf("the url was lost: %q", got.URL)
}
if got.Fingerprint != "abc123" {
t.Fatal("the builder was given nothing to check the broker against, so it can only " +
"connect to a broker some public authority vouches for")
}
}
func TestAnEmptyCredentialFileIsRefused(t *testing.T) {
// Otherwise the builder connects as nobody and is refused, with the reason three layers away.
path := filepath.Join(t.TempDir(), "broker")
if err := os.WriteFile(path, []byte("\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("MESH_BROKER_FILE", path)
t.Setenv("MESH_BROKER_AMQP", "")
if _, err := brokerFrom(); err == nil {
t.Fatal("an empty credential was accepted")
}
}
func TestABuilderWithNoCredentialAtAllSaysSo(t *testing.T) {
t.Setenv("MESH_BROKER_FILE", "")
t.Setenv("MESH_BROKER_AMQP", "")
if _, err := brokerFrom(); err == nil {
t.Fatal("a builder with no broker reported one")
}
}
// The pin is compared in the spelling the mesh writes it.
//
// A bare digest against a written fingerprint never matches, and the failure is indistinguishable
// from being pointed at the wrong broker — which is the one thing this check exists to report
// truthfully. It cost a lab run.
func TestThePinIsComparedInTheSpellingTheMeshWritesIt(t *testing.T) {
certificate, key := aServerCertificate(t)
der := certificate.Certificate[0]
sum := sha256.Sum256(der)
written := "sha256:" + hex.EncodeToString(sum[:])
listener, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{
Certificates: []tls.Certificate{certificate}, MinVersion: tls.VersionTLS12,
})
if err != nil {
t.Fatal(err)
}
defer listener.Close()
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
_ = conn.(*tls.Conn).Handshake()
conn.Close()
}
}()
_ = key
// The pin the mesh wrote must be accepted.
if err := handshakeWith(listener.Addr().String(), written); err != nil {
t.Fatalf("the broker this builder was told about was refused: %v", err)
}
// And a different one refused, or the check reports nothing.
other := "sha256:" + strings.Repeat("ab", 32)
if err := handshakeWith(listener.Addr().String(), other); err == nil {
t.Fatal("a broker this builder was not told about was accepted")
}
}
// handshakeWith runs the pin check the builder dials with against an address.
func handshakeWith(address, pin string) error {
conn, err := tls.Dial("tcp", address, broker.PinnedToFingerprint(pin))
if err != nil {
return err
}
return conn.Close()
}
func aServerCertificate(t *testing.T) (tls.Certificate, ed25519.PrivateKey) {
t.Helper()
public, private, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
template := &x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "a broker"},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
}
der, err := x509.CreateCertificate(rand.Reader, template, template, public, private)
if err != nil {
t.Fatal(err)
}
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: private}, private
}
+21 -2
View File
@@ -408,12 +408,21 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
_, reads := m.OwnSecrets["broker"]
if !reads && !runsTheProvisionerClient(m) {
// A module whose bus accounts are its processes' alone has none of its own: only the processes that
// are not issued yet are issued (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
if !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
if _, minted, err := inv.BusUserHash(ctx, user); err != nil {
return ""
} else if minted {
// Its own account stands and is kept; a process it gained is not issued yet (issue 1226).
return processAccountsOnAssign(ctx, inv, m, node)
}
busAddress, err := broker.BusAddress()
if err == nil {
@@ -426,6 +435,16 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return fmt.Sprintf("its bus credential is issued and sealed to %s, and arrives with the push", node)
}
// processAccountsOnAssign issues the process accounts of an assigned module not issued yet, and says what it did.
func processAccountsOnAssign(ctx context.Context, inv *inventory.Inventory, m catalogue.Manifest, node string) string {
line, err := issueMissingProcessAccounts(ctx, inv, m, node)
if err != nil {
return fmt.Sprintf("its process accounts are not issued (%v): `module issue %s --node %s` first — "+
"`push %s` refuses to send %s until it is", err, m.Module, node, node, m.Module)
}
return line
}
// refusedMovingTheController refuses assigning or unassigning the controller's module through a verb (review of
// novox/hq ADR 0272): the node it is assigned to is the control-node, and the control-node's operator account is
// the controller's terminal, so whoever moves the module chooses the terminal. At the terminal alone, as every
@@ -221,8 +221,4 @@ func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
-59
View File
@@ -1,12 +1,9 @@
package main
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// An address is read from the node's settings where it is used, never recorded with a port
@@ -14,62 +11,6 @@ import (
var aDigest = "sha256:" + strings.Repeat("e", 64)
// **A build is recorded by digest and path**, whatever address the builder pushed to — what it
// made and what it stood on both; an image the module runs from elsewhere is left where it says.
func TestABuildIsRecordedWithoutTheStoresAddress(t *testing.T) {
manifest, _ := json.Marshal(map[string]any{
"module": "gitea", "version": "1",
"resources": []map[string]any{
{"id": "server", "type": "container", "name": "mesh-gitea",
"image": "anchor.internal:5100/gitea/server@" + aDigest},
{"id": "config", "type": "archive", "path": "/etc/gitea", "digest": aDigest,
"source": "http://anchor.internal:5100/v2/gitea/config/blobs/" + aDigest},
{"id": "cache", "type": "container", "name": "mesh-gitea-cache",
"image": "valkey/valkey@" + aDigest},
},
})
kept := buildFrom(link.BuildResult{
ID: "b1", Repository: "https://forge.example/gitea.git", Commit: "abc", On: "laptop",
Manifest: manifest,
Made: []link.MadeArtifact{
{Name: "server", Kind: "image", Reference: "anchor.internal:5100/gitea/server@" + aDigest},
{Name: "config", Kind: "archive", Reference: "http://anchor.internal:5100/v2/gitea/config/blobs/" + aDigest},
},
Against: []string{"anchor.internal:5100/mesh-tools/runtime@" + aDigest},
})
if kept.Module != "gitea" {
t.Fatalf("the module was not read from the recorded manifest: %q", kept.Module)
}
if kept.Made[0].Reference != catalogue.ArtifactStoreScheme+"gitea/server@"+aDigest {
t.Errorf("the image is recorded as %q, address and all", kept.Made[0].Reference)
}
if kept.Made[1].Reference != catalogue.ArtifactStoreScheme+"gitea/config/blobs/"+aDigest {
t.Errorf("the archive is recorded as %q, address and all", kept.Made[1].Reference)
}
recorded, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
t.Fatal(err)
}
if got := recorded.Resources[0]["image"]; got != catalogue.ArtifactStoreScheme+"gitea/server@"+aDigest {
t.Errorf("the recorded manifest's image is %v", got)
}
if got := recorded.Resources[1]["source"]; got != catalogue.ArtifactStoreScheme+"gitea/config/blobs/"+aDigest {
t.Errorf("the recorded manifest's archive is %v", got)
}
if got := recorded.Resources[2]["image"]; got != "valkey/valkey@"+aDigest {
t.Errorf("an image the build did not make was rewritten: %v", got)
}
if strings.Contains(string(kept.Manifest), "anchor.internal:5100") {
t.Errorf("the recorded manifest still carries the store's address:\n%s", kept.Manifest)
}
// What the build stood on is an edge to another module's artifact, and it is recorded the way
// that artifact is: by path in the store, so the edge still names the same thing when the
// store answers at another address.
if kept.Against[0] != catalogue.ArtifactStoreScheme+"mesh-tools/runtime@"+aDigest {
t.Errorf("what the build stood on was recorded by address: %v", kept.Against)
}
}
// aStore is a module offering the artifact store on 5000, published the long way as the
// distribution module does, so a node may be given another number for it.
func aStore() catalogue.Manifest {
@@ -1,134 +0,0 @@
//go:build integration
package main
import (
"context"
"fmt"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// Issue 1334, the look: when the bus did not answer an ask of any module of a tier, no build is in flight and no event follows,
// so the look itself comes back (it fails, and the looker retries it), asks the modules again, and past the bound fails the plan.
func TestALookComesBackForATierTheBusRefusedWhole(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, name := range []string{"a", "b"} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: name, Version: "1"},
inventory.Source{Repository: "novox/" + name, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
p := inventory.Plan{ID: "plan-bus", Repository: "novox/a", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b"}}, Modules: map[string]*inventory.PlanModule{"a": {}, "b": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
bus := false
was := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
if !bus {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
return "build-" + source.Repository, nil
}
t.Cleanup(func() { askABuild = was })
look := func() bool {
t.Helper()
release, err := inv.HoldPlans(ctx, true)
if err != nil {
t.Fatal(err)
}
defer release()
return advanceHeld(ctx, open)
}
read := func() inventory.Plan {
t.Helper()
got, err := inv.PlanByID(ctx, "plan-bus")
if err != nil {
t.Fatal(err)
}
return got
}
// Every module refused: the look says it failed, which has the looker come back; the plan is building and says why.
if !look() {
t.Fatal("a look that left the whole tier unasked for the bus did not fail, so nothing would come back to it")
}
if got := read(); got.State != inventory.PlanBuilding || !strings.Contains(got.Note, "bus did not answer") ||
got.Modules["a"].State != "" || got.Modules["b"].State != "" {
t.Fatalf("the plan after a refused tier: %s %q %+v %+v", got.State, got.Note, got.Modules["a"], got.Modules["b"])
}
// Again, still refused: the same fact is not saved again, and the look still comes back.
before := read().Revision
if !look() || read().Revision != before {
t.Fatalf("a second refused look failed to come back or saved the plan again (revision %d, was %d)", read().Revision, before)
}
// The bus answers at the next look: both are asked, and nothing waits any more.
bus = true
if look() {
t.Fatal("a look that asked the whole tier failed")
}
if got := read(); got.Modules["a"].State != "asked" || got.Modules["b"].State != "asked" || got.Modules["a"].BusWaitFrom != nil {
t.Fatalf("the tier was not asked once the bus answered: %+v %+v", got.Modules["a"], got.Modules["b"])
}
}
func TestALookFailsTheTierTheBusRefusedPastTheBound(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "a", Version: "1"},
inventory.Source{Repository: "novox/a", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
long := time.Now().UTC().Add(-busWaitBound - time.Minute)
p := inventory.Plan{ID: "plan-bus", Repository: "novox/a", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {BusWaitFrom: &long}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
t.Cleanup(func() { askABuild = was })
release, err := inv.HoldPlans(ctx, true)
if err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
release()
got, err := inv.PlanByID(ctx, "plan-bus")
if err != nil {
t.Fatal(err)
}
if got.State != inventory.PlanFailed || !strings.Contains(got.Note, "the bus did not answer an ask for 10m0s (since ") {
t.Fatalf("a bus silent past its bound did not fail the plan in words: %s %q", got.State, got.Note)
}
// Rebuilt by hand while the bus still reloads: said as waiting, not as a failure and not with an empty reason.
joined, said, err := joinAPlan(ctx, open, "a")
if err != nil || !joined || !strings.Contains(said, "not asked yet: the bus did not answer") {
t.Fatalf("a rebuild left for the bus is said as %v %q %v", joined, said, err)
}
// And retried by hand: the module is named as waiting, not as asked again.
got, _ = inv.PlanByID(ctx, "plan-bus")
if got.State == inventory.PlanFailed {
got.Modules["a"].State, got.Modules["a"].Why = "failed", "x"
if err := inv.SavePlan(ctx, &got); err != nil {
t.Fatal(err)
}
said, err := retryPlan(ctx, open, "plan-bus")
if err != nil || !strings.Contains(said, "not asked yet: the bus did not answer") {
t.Fatalf("a retry left for the bus is said as %q %v", said, err)
}
}
}
-139
View File
@@ -1,139 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Issue 1334: a bus that does not answer for a moment (it reloads its user list, and JetStream's API has no responder for a few
// seconds) is not a failed build. The module is left unasked and the plan building, and the next look asks it again; only a
// module's own fault, or a bus silent for ten minutes, fails the plan.
func aTierOfThree() (*inventory.Plan, map[string]inventory.Entry) {
p := &inventory.Plan{ID: "plan-1", State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b", "c"}},
Modules: map[string]*inventory.PlanModule{}}
byName := map[string]inventory.Entry{}
for _, m := range []string{"a", "b", "c"} {
byName[m] = inventory.Entry{Source: inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main"}}
}
return p, byName
}
func TestABusThatDoesNotAnswerForAMomentDoesNotFailTheTier(t *testing.T) {
p, byName := aTierOfThree()
bus := false
was := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
if source.Repository == "novox/b" && !bus {
return "", fmt.Errorf("%w: the bucket mesh-controller_builds is not on the bus: nats: no responders available for request",
inventory.ErrNoChain)
}
return "build-" + source.Repository, nil
}
t.Cleanup(func() { askABuild = was })
for _, m := range []string{"a", "b", "c"} {
askModule(t.Context(), p, m, byName)
}
if p.State != inventory.PlanBuilding {
t.Fatalf("a bus that did not answer for a moment failed the plan: %s, %q", p.State, p.Note)
}
if p.Modules["a"].State != "asked" || p.Modules["c"].State != "asked" {
t.Fatalf("the modules the bus did answer for were not asked: %+v %+v", p.Modules["a"], p.Modules["c"])
}
if s := p.Modules["b"]; s.State != "" {
t.Fatalf("the module the bus did not answer for is %q, not left unasked", s.State)
}
if got := unaskedOf(*p); len(got) != 1 || got[0] != "b" {
t.Fatalf("the next look would ask %v, not b", got)
}
if !strings.Contains(p.Note, "bus did not answer") {
t.Fatalf("the plan says nothing of the wait: %q", p.Note)
}
// The next look, the bus answering: b is asked and the plan goes on.
bus = true
for _, m := range unaskedOf(*p) {
askModule(t.Context(), p, m, byName)
}
if p.State != inventory.PlanBuilding || p.Modules["b"].State != "asked" || p.Modules["b"].Build == "" {
t.Fatalf("b was not asked once the bus answered: %s %+v", p.State, p.Modules["b"])
}
}
func TestABusSilentForTenMinutesFailsTheTierAsBefore(t *testing.T) {
p, byName := aTierOfThree()
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
t.Cleanup(func() { askABuild = was })
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanBuilding {
t.Fatalf("the first refusal failed the plan: %q", p.Note)
}
long := time.Now().Add(-busWaitBound - time.Minute)
p.Modules["a"].BusWaitFrom = &long
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || p.Modules["a"].State != "failed" {
t.Fatalf("a bus silent past its bound did not fail the plan: %s %+v", p.State, p.Modules["a"])
}
// Said in plain words: the bus, the wait, and since when.
want := "a could not be asked for: the bus did not answer an ask for 10m0s (since " + long.UTC().Format(time.RFC3339) +
"): this kind of record lives on the bus"
if !strings.HasPrefix(p.Note, want) || p.Modules["a"].Why != strings.TrimPrefix(want, "a could not be asked for: ")+p.Note[len(want):] {
t.Fatalf("the plan says %q, not words starting %q", p.Note, want)
}
// The wait has ended: a Retry made while the bus still reloads waits again, and does not fail at its first refusal.
if p.Modules["a"].BusWaitFrom != nil {
t.Fatalf("a failed module still holds the start of its wait: %v", p.Modules["a"].BusWaitFrom)
}
p.State = inventory.PlanBuilding
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanBuilding || p.Modules["a"].State != "" || p.Modules["a"].BusWaitFrom == nil {
t.Fatalf("a retry made while the bus is silent did not wait afresh: %s %+v", p.State, p.Modules["a"])
}
}
func TestAPlanWithAModuleWaitingForTheBusIsOneTheLookComesBackTo(t *testing.T) {
p, _ := aTierOfThree()
if busWaiting(*p) {
t.Fatal("a plan whose modules are not asked yet waits for the bus")
}
now := time.Now()
p.Modules["b"] = &inventory.PlanModule{BusWaitFrom: &now}
p.Modules["a"] = &inventory.PlanModule{State: "asked"}
if got := busWaitingIn(*p); len(got) != 1 || got[0] != "b" {
t.Fatalf("the modules waiting for the bus are %v, not b", got)
}
p.State = inventory.PlanFailed
if busWaiting(*p) {
t.Fatal("a plan that is not open is waited for")
}
}
func TestAModulesOwnFaultStillFailsThePlan(t *testing.T) {
p, byName := aTierOfThree()
delete(byName, "a")
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || !strings.Contains(p.Note, "no longer in the catalogue") {
t.Fatalf("a module gone from the catalogue did not fail the plan: %s %q", p.State, p.Note)
}
p, byName = aTierOfThree()
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", errors.New("git.example/novox/a is no repository the catalogue builds a module from")
}
t.Cleanup(func() { askABuild = was })
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || p.Modules["a"].State != "failed" {
t.Fatalf("a module's own refusal did not fail the plan: %s %+v", p.State, p.Modules["a"])
}
}
-35
View File
@@ -1,16 +1,12 @@
package main
import (
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -18,37 +14,6 @@ import (
// (issue 1334).
const busWaitBound = 10 * time.Minute
// busDidNotAnswer says an ask failed because the bus's JetStream had no responder at that moment, as it has for a few seconds
// when the bus reloads its user list (novox/hq issue 353), and not because of the module: the same ask is made again.
func busDidNotAnswer(err error) bool {
if err == nil {
return false
}
if errors.Is(err, nats.ErrNoResponders) || errors.Is(err, nats.ErrJetStreamNotEnabled) ||
errors.Is(err, jetstream.ErrJetStreamNotEnabled) || errors.Is(err, inventory.ErrNoChain) {
return true
}
// An error that crossed a boundary as text only.
text := err.Error()
return strings.Contains(text, "no responders available") || strings.Contains(text, "jetstream not enabled")
}
// waitForTheBus leaves a module whose ask the bus did not answer unasked, the plan building, and says so in the plan's note: the
// next look asks the tier's unasked modules again. It says false, so the caller fails the plan as for any refusal, once the bus
// has been silent for busWaitBound.
func waitForTheBus(p *inventory.Plan, state *inventory.PlanModule, name string, err error, now time.Time) bool {
if state.BusWaitFrom == nil {
state.BusWaitFrom = &now
}
if now.Sub(*state.BusWaitFrom) > busWaitBound {
return false
}
state.State, state.Why = "", ""
p.Note = fmt.Sprintf("tier %d: %s not asked: the bus did not answer (%v); asked again at the next look, failing after %s",
p.Tier, name, firstLine(err.Error()), busWaitBound)
return true
}
// busWaiting says an open plan has a module of its current tier that the bus did not answer an ask of: the look must come back
// for it, as nothing else will when no build of the tier is in flight (issue 1334).
func busWaiting(p inventory.Plan) bool {
+23
View File
@@ -0,0 +1,23 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A retry of a plan failed past the bus's bound, whose module the bus still does not answer, names that module as waiting:
// read while the plan still says failed, it was named nowhere, and the retry said "asked" with nothing after it (issue
// 1403, met again once the hand rebuild no longer resumed the plan first).
func TestARetryNamesTheModulesStillWaitingForTheBus(t *testing.T) {
since := time.Now().UTC()
p := inventory.Plan{ID: "plan-bus", State: inventory.PlanFailed, Tiers: [][]string{{"a"}},
Modules: map[string]*inventory.PlanModule{"a": {BusWaitFrom: &since}}}
waiting := busWaitingOnRetry(p)
_, said := retryWords(p, []string{"a"}, nil, waiting)
if !strings.Contains(said, "a not asked yet: the bus did not answer") || strings.Contains(said, "asked ;") {
t.Fatalf("a retry left for the bus is said as %q", said)
}
}
+4
View File
@@ -645,6 +645,10 @@ func doesWords(act conditions.Action) string {
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case act.Verb == broker.ControllerSeat+".push" && act.Arguments["node"] != "":
return "the held builds are sent to " + act.Arguments["node"]
case act.Verb == broker.ControllerSeat+".upgrade" && act.Arguments["release-backlog"] != "":
return "the builds held after the failed delivery are released"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
@@ -33,7 +33,7 @@ func TestAWarrantForRetryRetriesAFailedPlanInTheStoreAndEndsItsCondition(t *test
Note: "a failed to build in tier 0",
Delivery: &inventory.PlanDelivery{},
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}},
Why: "killed by hand"}},
Times: &inventory.PlanTimes{Cut: &before, Started: &before, Ended: &ended, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
+2 -17
View File
@@ -66,17 +66,8 @@ func TestAPlanGoesOnWithoutAMachineAwayWhenItsTierIsSent(t *testing.T) {
// The build machine and the machines, as far as the plan meets them: an ask is counted, a send is recorded and kept.
var mu sync.Mutex
var asks []string
var sends [][]string
wasAsk := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
mu.Lock()
defer mu.Unlock()
id := link.NewBuildID(time.Now().Add(time.Duration(len(asks)) * time.Millisecond))
asks = append(asks, source.Repository+"#"+id)
return id, nil
}
t.Cleanup(func() { askABuild = wasAsk })
everyKeyDone(t) // the builds of the tiers: each module is taken from a done key as soon as the walk reaches it
digests := 0
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
@@ -198,13 +189,7 @@ func TestAPlanGoesOnWithoutAMachineAwayWhenItsTierIsSent(t *testing.T) {
if !strings.Contains(moved, "laptop was away: sent when it is heard again") || strings.Contains(moved, "anchor") {
t.Errorf("1161/AC2: the plan says %q; want laptop named away and not anchor", moved)
}
// The ask for tier 1's build is the next step, and records its build id on the module.
eventually(t, "tier 1 asked", func() bool { m := plan().Modules["b"]; return m != nil && m.Build != "" })
b := plan().Modules["b"]
askedAt, _ := link.BuildAskedAt(b.Build)
record(b.Build, "b", "c2", askedAt)
register("b", "c2", askedAt)
planBuilt(ctx, open, "b", "c2", "", askedAt, b.Build)
// Tier 1's module is taken from its key, which the walk's look does by itself.
eventually(t, "tier 1 sent to anchor alone", func() bool {
mu.Lock()
defer mu.Unlock()
+5 -2
View File
@@ -232,8 +232,8 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
fmt.Print(nothingReadsIt(repository, m))
return nil
}
// **A module the merge adds is asked for now** (novox/hq issue 1617), not when the batch is cut.
askNewModulesAtHearing(ctx, inv, m, entries, read, facts.from)
// A module the merge adds is registered from the builder's first done key of it, which the builder's write wakes
// (novox/hq ADR 0501 §6): nothing is asked here.
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, holding, err := groupWaitHolding(ctx, inv, m, deliverySeatHeld(entries))
@@ -800,6 +800,9 @@ type keptMerge struct {
// FirstBuild says the merge is a first merge into a repository the mesh builds nothing from, kept to ask the first
// build of its modules once mesh-delivery's word on it is kept (novox/hq ADR 0432, issue 1146).
FirstBuild bool `json:"first_build,omitempty"`
// Held says which first builds of the merge were refused for a word the running build-agent does not read: the merge
// waits for a build-agent that reads them, and is never failed (novox/hq ADR 0482 §5, issue 1173).
Held []heldFirstBuild `json:"held,omitempty"`
}
// announcedOf is a kept merge as the forge announced it, and what it moved; empty where the record says none.
+1 -1
View File
@@ -264,7 +264,7 @@ func TestAPlanLineNamesWhatItMoves(t *testing.T) {
}
return nil
}
if got, want := planLineOn(p, now, pauseView{}, tierAtLeast, running),
if got, want := planLineOn(p, now, tierAtLeast, running),
"mesh-catalog #175 a tap shows its outcome · messenger, telegram → novox · tier 1 of 1, building for 2m0s"; got != want {
t.Fatalf("the line reads %q, want %q", got, want)
}
File diff suppressed because it is too large Load Diff
-43
View File
@@ -1,43 +0,0 @@
package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func claiming(module, seat string, on ...string) inventory.Entry {
return inventory.Entry{
Manifest: catalogue.Manifest{Module: module, Claims: []catalogue.Claim{{Name: seat}}},
On: on,
}
}
// The controller asks the build role that has a holder (novox/hq ADR 0190 handover): the retired
// one while only the builder is assigned, the current one from the first build-agent on, and the
// current one when nothing holds either — where every ask goes once the handover is done.
func TestTheControllerAsksTheBuildRoleThatHasAHolder(t *testing.T) {
onlyTheBuilder := []inventory.Entry{
claiming("builder", link.TheBuildMachineBefore, "anchor"),
claiming("build-agent", link.TheBuildMachine), // registered, assigned nowhere yet
}
if got := buildSeatAmong(onlyTheBuilder); got != link.TheBuildMachineBefore {
t.Errorf("with only the builder assigned, asked %q", got)
}
bothHeld := []inventory.Entry{
claiming("builder", link.TheBuildMachineBefore, "anchor"),
claiming("build-agent", link.TheBuildMachine, "home-server"),
}
if got := buildSeatAmong(bothHeld); got != link.TheBuildMachine {
t.Errorf("with a build-agent assigned anywhere, asked %q", got)
}
neither := []inventory.Entry{claiming("builder", link.TheBuildMachineBefore)}
if got := buildSeatAmong(neither); got != link.TheBuildMachine {
t.Errorf("with no holder of either, asked %q, want the current role", got)
}
if got := buildSeatAmong(nil); got != link.TheBuildMachine {
t.Errorf("an empty catalogue asks %q", got)
}
}
+86 -105
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"os"
"regexp"
"slices"
"strings"
"time"
@@ -188,25 +189,13 @@ func (f *sourceForms) same(aRepository, aSeat, bRepository, bSeat string) bool {
return f.canonical(aRepository, aSeat) == f.canonical(bRepository, bSeat)
}
// buildsFrom says the catalogue builds some module from this repository.
func (f *sourceForms) buildsFrom(entries []inventory.Entry, repository, seat string) bool {
for _, e := range entries {
if e.Provided || e.Source.Repository == "" {
continue
}
if f.same(e.Source.Repository, e.Source.Seat, repository, seat) {
return true
}
}
return false
}
// forgeFacts is what the mesh's forge says of a repository a module is registered from (novox/hq ADR 0266).
type forgeFacts struct {
// ID is the forge's own id of the repository: what tells it from one deleted and made again by its name.
ID int64
// Guarded is whether the branch is protected as a module's trunk must be: no direct push, at least one
// required status, and no administrator merging past one. Why says what is missing when it is not.
// Guarded is whether the branch is protected as a module's trunk must be: no direct push, a merge whitelist naming
// the tools' account and no agent's account, and no administrator merging past the rule (novox/hq ADR 0534 §6; no
// required status, which the forge no longer asks, §1). Why says what is missing when it is not.
Guarded bool
Why string
}
@@ -361,46 +350,68 @@ func askTheForgeOnce(ctx context.Context, owner, repo, branch string) (forgeFact
return facts, nil
}
// forgeToolsAccount is the forge tools' account, which main's merge whitelist names for mesh-delivery (novox/hq ADR 0398
// §2, the gitea module's tools-account.ts TOOLS_ACCOUNT); beside it the whitelist names only people's own logins, the
// operator's (ADR 0534 §2), never an agent's account.
const forgeToolsAccount = "mesh-tools"
// isAgentAccount is whether a forge login is a node's agent's account, agent-<node> (novox/hq ADR 0398 §1): one that
// may never merge into a module's trunk by itself.
func isAgentAccount(login string) bool { return strings.HasPrefix(login, "agent-") }
// protectionRule is a branch protection rule as the forge's tool summarises it.
type protectionRule struct {
Rule string `json:"rule"`
Push bool `json:"push"`
RequiredStatuses []string `json:"required_statuses"`
AdminMayOverride bool `json:"admin_may_override"`
// MergeWhitelist is who may merge into the branch; nil when the rule keeps no merge whitelist (anyone who writes).
MergeWhitelist []string `json:"merge_whitelist"`
}
// judgedRule judges the one rule the forge applies to a branch, as the forge picks it: the rule named for the
// branch, else the first glob rule, in the forge's order, that covers it (gitea's first matching rule). Never a
// later rule that happens to be stronger: the forge does not read it.
// later rule that happens to be stronger: the forge does not read it. A trunk is trusted for who can land a commit on
// it, not for a status it requires (novox/hq ADR 0534 §6): no push, a merge whitelist naming the tools' account and
// beside it only people's own logins, never an agent's, and no administrator merging past the rule; then a commit lands
// there through mesh-delivery, on a passing mesh/check and the operator's Approve (ADR 0534 §4), or by the operator's
// own hand at the forge (§2).
func judgedRule(rules []protectionRule, branch string) forgeFacts {
var applied *protectionRule
for i := range rules {
if rules[i].Rule == branch {
applied = &rules[i]
break
}
}
if applied == nil {
for i := range rules {
if ruleCovers(rules[i].Rule, branch) {
applied = &rules[i]
break
}
}
}
switch r := applied; {
switch r := appliedRule(rules, branch); {
case r == nil:
return forgeFacts{Why: fmt.Sprintf("no protection rule covers %s", branch)}
case r.Push:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets a person push to %s directly", r.Rule, branch)}
case len(r.RequiredStatuses) == 0:
return forgeFacts{Why: fmt.Sprintf("the rule %s requires no status before a merge into %s", r.Rule, branch)}
case r.MergeWhitelist == nil || len(r.MergeWhitelist) == 0:
return forgeFacts{Why: fmt.Sprintf("the rule %s keeps no merge whitelist naming %s alone for %s", r.Rule, forgeToolsAccount, branch)}
case !slices.Contains(r.MergeWhitelist, forgeToolsAccount):
return forgeFacts{Why: fmt.Sprintf("the rule %s lets %s merge into %s, and not %s", r.Rule,
strings.Join(r.MergeWhitelist, ", "), branch, forgeToolsAccount)}
case slices.ContainsFunc(r.MergeWhitelist, isAgentAccount):
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an agent's account merge into %s: %s", r.Rule, branch,
strings.Join(r.MergeWhitelist, ", "))}
case r.AdminMayOverride:
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an administrator merge into %s past a status", r.Rule, branch)}
return forgeFacts{Why: fmt.Sprintf("the rule %s lets an administrator merge into %s past it", r.Rule, branch)}
}
return forgeFacts{Guarded: true}
}
// appliedRule is the one rule the forge applies to a branch: the rule named for it, else the first glob rule, in the
// forge's order, that covers it; nil when none does.
func appliedRule(rules []protectionRule, branch string) *protectionRule {
for i := range rules {
if rules[i].Rule == branch {
return &rules[i]
}
}
for i := range rules {
if ruleCovers(rules[i].Rule, branch) {
return &rules[i]
}
}
return nil
}
// ruleCovers says a protection rule's name — a branch, or a glob of them — covers a branch, as the forge reads it.
func ruleCovers(rule, branch string) bool {
if rule == branch {
@@ -463,14 +474,42 @@ func (f *sourceForms) onTheForge(repository, seat string) (owner, name string, o
// - from the module's registered repository — the same repository by the forge's own id, not only its name; or,
// for a module new to the catalogue, from a repository the catalogue builds another module from;
// - and from a repository on the mesh's forge whose trunk is protected as a trunk must be: no direct push, at
// least one required status, no administrator merging past one.
// a merge whitelist naming the tools' account and no agent's, no administrator merging past the rule (ADR 0534 §6).
//
// Anything else only when the build request was kept as asked at the controller's terminal, for this very
// repository and path; or, for the build of a merge, when mesh-delivery announced it on the operator's Approve of
// that head (novox/hq ADR 0420 §7 for a moved module, ADR 0432 for a new one), the forge's checks still holding. path is the module's directory as built; branch the trunk it is registered from.
func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, buildID, path, branch)
ask registrationAsk, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, ask, path, branch)
}
// registrationAsk is what a registration's warrant is read from (novox/hq ADR 0501 §3): the build an outcome answers,
// whose kept request says whether it was asked at the terminal or for a merge (build-agent's outcome, the builder's own
// module until R4g); or, for a take-in from the builder's done key, the merge the indexer's trunk value names and
// whether the operator asked the registration at the controller's terminal (ADR 0501 §4). A key asks no build, so it
// keeps no request.
type registrationAsk struct {
// Build is the build whose kept request is read; empty for a take-in from a key.
Build string
// Merge is the commit of the merge a key's module is registered from (the indexer's trunk value); empty for an
// outcome, whose request names it.
Merge string
// Terminal is a key registration the operator asked at the controller's terminal for this repository and path.
Terminal bool
}
// String is the ask as the registration's words name it.
func (a registrationAsk) String() string {
switch {
case a.Build != "":
return a.Build
case a.Terminal:
return "the registration asked at the terminal"
case a.Merge != "":
return fmt.Sprintf("the merge %.8s", a.Merge)
}
return "a key"
}
// catalogueReader is the catalogue's manifests, as the router's records are found in them.
@@ -490,15 +529,19 @@ type registrations interface {
// mayRegisterIn is mayRegisterFrom on what it reads of the mesh handed in.
func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
ask registrationAsk, path, branch string) (int64, error) {
buildID := ask.String()
was, err := inv.SourceOf(ctx, module)
isNew := errors.Is(err, inventory.ErrNoSuchModule)
if err != nil && !isNew {
return 0, err
}
terminal, err := askedHereFor(ctx, inv, forms, buildID, built, path)
if err != nil {
return 0, err
terminal := ask.Terminal
if ask.Build != "" {
var err error
if terminal, err = askedHereFor(ctx, inv, forms, ask.Build, built, path); err != nil {
return 0, err
}
}
refuse := func(why string) (int64, error) {
return 0, fmt.Errorf("%w: %s. A module is registered from such a source only by a build asked at the "+
@@ -542,7 +585,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
// The forge's protection and the repository's identity are checked below as for every registration.
movedWhy := ""
if why != "" && !terminal {
accepted, err := approvedMove(ctx, inv, forms, buildID, built)
accepted, err := approvedMove(ctx, inv, forms, ask, built)
switch {
case err == nil:
movedWhy, why = accepted, ""
@@ -661,65 +704,3 @@ func sourceWords(repository, seat string) string {
}
return buildSource{Repository: repository, Seat: seat}.String()
}
// firstBuildWarrant is what the controller checked before it asks the first build of the modules a merge adds to a
// repository nothing is built from: the merge was made on the operator's Approve of its head (approvedAnnounced). It
// travels in the context of that ask alone, in this process, so no verb carries one.
type firstBuildWarrant struct {
repository string // owner/name on the git seat
paths map[string]bool
why string
}
type firstBuildWarrantKey struct{}
// withFirstBuildWarrant is ctx carrying the warrant for asking the first builds of paths in repository.
func withFirstBuildWarrant(ctx context.Context, repository string, paths []string, why string) context.Context {
w := &firstBuildWarrant{repository: repository, paths: map[string]bool{}, why: why}
for _, p := range paths {
w.paths[strings.Trim(p, "/")] = true
}
return context.WithValue(ctx, firstBuildWarrantKey{}, w)
}
func firstBuildWarrantIn(ctx context.Context) *firstBuildWarrant {
w, _ := ctx.Value(firstBuildWarrantKey{}).(*firstBuildWarrant)
return w
}
// covers says the warrant is for exactly this ask: the repository on the git seat, and a module path the merge holds.
func (w *firstBuildWarrant) covers(source buildSource, path string) bool {
return w != nil && source.Seat == gitSeat && strings.EqualFold(strings.Trim(source.Repository, "/"), w.repository) &&
w.paths[strings.Trim(path, "/")]
}
// verbMayAsk refuses, for a call through a verb, a build of a repository the catalogue builds no module from
// (novox/hq ADR 0266): the build node would run what an agent wrote, and its outcome could never be
// registered anyway. url is the repository as it is cloned. At the terminal anything may be asked.
func verbMayAsk(ctx context.Context, source buildSource, url, path string) error {
if startedAtTheTerminal() {
return nil
}
// **Or the first build of a module a merge adds on the operator's Approve** (novox/hq ADR 0432, issue 1146): the
// controller's own ask, carrying the warrant it checked, for that repository and the module paths the merge holds.
if firstBuildWarrantIn(ctx).covers(source, path) {
return nil
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return err
}
forms := newSourceForms(ctx, open.inventory)
if forms.buildsFrom(entries, source.Repository, source.Seat) || forms.buildsFrom(entries, url, "") {
return nil
}
return terminalRefusal("%s is no repository the catalogue builds a module from, and a build of any other is "+
"asked at the controller's terminal only, never through a verb: a build node runs what the repository "+
"says, and its outcome would register a module the next push sends — whoever may call a verb includes "+
"agents (novox/hq ADR 0266). Nothing was asked", source)
}
@@ -129,27 +129,6 @@ func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
}
}
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git", "")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git", ""); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "", ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
+38 -4
View File
@@ -102,16 +102,17 @@ func init() {
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
guarded := protectionRule{Rule: "*", RequiredStatuses: []string{"mesh/merge-gate"}}
open := protectionRule{Rule: "main", Push: true, RequiredStatuses: []string{"mesh/merge-gate"}}
tools := []string{forgeToolsAccount}
guarded := protectionRule{Rule: "*", MergeWhitelist: tools}
open := protectionRule{Rule: "main", Push: true, MergeWhitelist: tools}
if f := judgedRule([]protectionRule{guarded, open}, "main"); f.Guarded {
t.Error("an exact rule letting pushes was passed over for a glob that guards")
}
if f := judgedRule([]protectionRule{{Rule: "ma*", RequiredStatuses: nil}, {Rule: "*", RequiredStatuses: []string{"x"}}},
if f := judgedRule([]protectionRule{{Rule: "ma*", MergeWhitelist: nil}, {Rule: "*", MergeWhitelist: tools}},
"main"); f.Guarded || !strings.Contains(f.Why, "ma*") {
t.Errorf("the first glob covering the branch was not the one judged: %+v", f)
}
if f := judgedRule([]protectionRule{{Rule: "release/*"}, {Rule: "main", RequiredStatuses: []string{"x"}}}, "main"); !f.Guarded {
if f := judgedRule([]protectionRule{{Rule: "release/*"}, {Rule: "main", MergeWhitelist: tools}}, "main"); !f.Guarded {
t.Errorf("the rule named for the branch was not judged: %+v", f)
}
if f := judgedRule(nil, "main"); f.Guarded {
@@ -119,6 +120,39 @@ func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
}
}
// **A module's trunk is trusted when it takes no push, only the tools' account and people's own logins merge into it,
// and no administrator merges past its rule** (novox/hq ADR 0534 §2, §6): the forge requires no status to merge (§1), so
// a rule requiring none is trusted; one with no merge whitelist, no tools' account in it, an agent's account in it, a
// push allowed or an administrator's override is not.
func TestATrunkIsTrustedForItsMergersNotForAStatus(t *testing.T) {
tools := []string{forgeToolsAccount}
for _, c := range []struct {
name string
rule protectionRule
want bool
why string
}{
{"no status, no push, the tools alone merge", protectionRule{Rule: "main", MergeWhitelist: tools}, true, ""},
{"a status required as well", protectionRule{Rule: "main", MergeWhitelist: tools, RequiredStatuses: []string{"mesh/check"}}, true, ""},
{"no merge whitelist", protectionRule{Rule: "main"}, false, "merge whitelist"},
{"an empty merge whitelist", protectionRule{Rule: "main", MergeWhitelist: []string{}}, false, "merge whitelist"},
{"the operator's own login beside the tools", protectionRule{Rule: "main", MergeWhitelist: []string{forgeToolsAccount, "jschoubben"}}, true, ""},
{"an agent's account in the whitelist", protectionRule{Rule: "main", MergeWhitelist: []string{forgeToolsAccount, "agent-ace"}}, false, "agent-ace"},
{"an agent's account alone", protectionRule{Rule: "main", MergeWhitelist: []string{"agent-ace"}}, false, "agent-ace"},
{"a person's login without the tools", protectionRule{Rule: "main", MergeWhitelist: []string{"jschoubben"}}, false, forgeToolsAccount},
{"a push allowed", protectionRule{Rule: "main", Push: true, MergeWhitelist: tools}, false, "push"},
{"an administrator's override", protectionRule{Rule: "main", MergeWhitelist: tools, AdminMayOverride: true}, false, "administrator"},
} {
f := judgedRule([]protectionRule{c.rule}, "main")
if f.Guarded != c.want || !strings.Contains(f.Why, c.why) {
t.Errorf("%s: %+v, want guarded %v saying %q", c.name, f, c.want, c.why)
}
if strings.Contains(f.Why, "requires no status") {
t.Errorf("%s: a trunk is still judged on a required status: %s", c.name, f.Why)
}
}
}
// An id the forge could not give keeps the id already recorded.
// **The operator's Approve of a reviewed group registers a moved module from its new repository** (novox/hq ADR 0420 §7,
-11
View File
@@ -2,12 +2,10 @@ package main
import (
"bytes"
"reflect"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -68,12 +66,3 @@ func TestBuildsSaysTheSourceFingerprint(t *testing.T) {
t.Errorf("the other lines changed:\n%s\nwant\n%s", plain.String(), want)
}
}
// 1215/AC4: what the builder said of the inputs is what the controller keeps of the build.
func TestTheControllerKeepsTheInputsABuilderSaid(t *testing.T) {
in := &sdkdelivery.SourceInputs{Module: "mailu", Tree: "t", Toolchains: map[string]string{"code": "ts sha256:aa 01"}}
kept := buildFrom(link.BuildResult{ID: "build-1", Repository: "r", Commit: "c", SourceFingerprint: "src1:x", SourceInputs: in})
if !reflect.DeepEqual(kept.SourceInputs, in) {
t.Errorf("the controller keeps %+v, the builder said %+v", kept.SourceInputs, in)
}
}
@@ -0,0 +1,109 @@
//go:build integration
package main
import (
"context"
"testing"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// builderOnTheBus is the builder's and the indexer's buckets on a test's own bus, which the controller's connection for
// the moment (servingBus) reaches: what a take-in from a key reads (R4e, issue 1851).
type builderOnTheBus struct {
js *broker.JetStream
builds, modules, manifests jetstream.KeyValue
}
func aBuilderOnTheBus(t *testing.T) builderOnTheBus {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
ctx := t.Context()
b := builderOnTheBus{js: js}
for _, c := range []struct {
into *jetstream.KeyValue
bucket string
history uint8
}{
{&b.builds, sdkdelivery.BuilderBuildsBucket, sdkdelivery.BuilderBuildsHistory},
{&b.modules, sdkdelivery.ModulesBucket, sdkdelivery.ModulesHistory},
{&b.manifests, sdkdelivery.ManifestsBucket, 1},
} {
kv, err := api.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: c.bucket, History: c.history})
if err != nil {
t.Fatal(err)
}
*c.into = kv
}
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
withTrunkList(t, true)
// The indexer on this bus is read, not the "not held" a mesh with no bus reads (aMeshOnTheStore).
wasIndexed := indexedModule
indexedModule = theIndexedModule
t.Cleanup(func() { indexedModule = wasIndexed })
return b
}
// indexed writes the indexer's trunk value of module at commit, naming fp, and its manifest: a module of one archive.
func (b builderOnTheBus) indexed(t *testing.T, module, version, repository, path, commit, fp string) string {
t.Helper()
keys, trunk := aKeyFor(module, version, fp, repository, path, commit)
raw := keys.manifests[trunk.Manifest]
value, err := sdkdelivery.EncodeManifest(sdkdelivery.Manifest{Module: module, JSON: raw})
if err != nil {
t.Fatal(err)
}
if _, err := b.manifests.Put(context.Background(), trunk.Manifest, value); err != nil {
t.Fatal(err)
}
trunk.FP = "src1:" + fp[len("src2:"):]
encoded, err := sdkdelivery.EncodeModule(trunk)
if err != nil {
t.Fatal(err)
}
if _, err := b.modules.Put(context.Background(), module, encoded); err != nil {
t.Fatal(err)
}
return trunk.Manifest
}
// built writes the builder's key of module at fp: done, its one archive named with a reference, or in another state.
func (b builderOnTheBus) built(t *testing.T, module, version, fp, manifest, state string) {
t.Helper()
keys, _ := aKeyFor(module, version, fp, "", "", "")
v := keys.keys[module].Build
v.Manifest = manifest
if state != sdkdelivery.BuildDone {
v = sdkdelivery.BuilderBuild{State: state, By: "ace"}
if state == sdkdelivery.BuildFailed {
v.Error = "digests differ: ace sha256:1, g14 sha256:2"
}
}
value, err := sdkdelivery.EncodeBuilderBuild(v)
if err != nil {
t.Fatal(err)
}
key, err := sdkdelivery.BuilderBuildKey(module, fp)
if err != nil {
t.Fatal(err)
}
if _, err := b.builds.Put(context.Background(), key, value); err != nil {
t.Fatal(err)
}
}
// theIndexedModule is the look at the indexer as the controller makes it, kept before any test replaces it.
var theIndexedModule = indexedModule
+7
View File
@@ -32,6 +32,13 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err := broker.Raise(r, names); err != nil {
return nil, err
}
// And each node's bucket for the files of its declaration that travel by digest (novox/hq ADR 0486 §4), so a node finds its own
// whenever the bus is up.
if files, ok := r.(interface{ EnsureDeclarationFiles(nodes []string) error }); ok {
if err := files.EnsureDeclarationFiles(names); err != nil {
return nil, err
}
}
// The work queues of the mesh's own roles (novox/hq ADR 0121). The queue before the holder,
// deliberately: work queues until somebody arrives to do it, so assigning a build machine a week
// after something started asking for builds flushes the backlog instead of having lost it.
+3 -7
View File
@@ -241,10 +241,9 @@ func mergeAskPrefix(number int, head string) string {
return fmt.Sprintf("merge-%d-%s-", number, head)
}
// askersOfMerges are the askers a merge's warrant is read under: the teller, and mesh-delivery for a merge before release 3e
// (ADR 0437 §6). Release 4's controller work stops accepting mesh-delivery (ADR 0450 §8); a merge asked under it earlier is
// still found here until then.
func askersOfMerges() []string { return []string{sdkdelivery.TellerModule, theDelivery} }
// askersOfMerges are the askers a merge's warrant is read under: the teller alone (ADR 0437 §6), since R4e stopped taking
// mesh-delivery's (ADR 0450 §8; issue 1318).
func askersOfMerges() []string { return []string{sdkdelivery.TellerModule} }
// caughtUpWarrant judges a chosen merge ask of the router as the warrant of a merge: the ask is the operator's, the warrant
// answers that very ask as it was published, and it is the operator's Approve proved by the channel it came through (ADR 0437
@@ -391,9 +390,6 @@ var findMergeWarrant = func(ctx context.Context, inv catalogueReader, owner, rep
if asker == sdkdelivery.TellerModule && !strings.HasPrefix(id, sdkdelivery.AskKindMerge+"-") {
continue
}
if asker == theDelivery && !named {
continue
}
ask, w, err := closedRecordOf(ctx, inv, asker, id)
if err != nil {
return "", false, "", err
+6 -34
View File
@@ -69,12 +69,12 @@ func TestAForgeThatDoesNotAnswerIsAnErrorNotAnEmptyTrunk(t *testing.T) {
func TestACaughtUpMergeIsAcceptedOnTheSameWarrantAsAnAnnouncedOne(t *testing.T) {
ok := approvedMerge(t)
for _, asker := range []string{sdkdelivery.TellerModule, theDelivery} {
w := ok.warrant
w.Asker = asker
if _, err := caughtUpWarrant(asker, &ok.ask, &w); err != nil {
t.Errorf("the operator's Approve under %s was refused: %v", asker, err)
}
if _, err := caughtUpWarrant(sdkdelivery.TellerModule, &ok.ask, &ok.warrant); err != nil {
t.Errorf("the operator's Approve under the teller was refused: %v", err)
}
// 1318 (ADR 0450 §8): mesh-delivery's warrant of before the gate is no longer taken.
if _, err := caughtUpWarrant(theDelivery, &ok.ask, &ok.warrant); err == nil {
t.Error("a warrant under mesh-delivery was accepted")
}
if _, err := caughtUpWarrant("keycloak", &ok.ask, &ok.warrant); err == nil {
t.Error("a warrant of an asker the controller takes none from was accepted")
@@ -229,34 +229,6 @@ func TestAGroupsWarrantIsFoundByTheAsksWordsForThePullRequestAndItsHead(t *testi
}
}
// Keycloak's merge was asked as mesh-delivery.merge-378-b9104454f9a5-1, before release 3e: the id names the pull request and
// the first twelve digits of its head, and the warrant is found by them under mesh-delivery.
func TestAWarrantAskedUnderMeshDeliveryIsFoundByThePullRequestAndItsHead(t *testing.T) {
ok := approvedMerge(t)
head := "b9104454f9a5e0c4d3b2a1f0e9d8c7b6a5948372"
askID := "merge-378-b9104454f9a5-1"
ask, w := ok.ask, ok.warrant
ask.ID, w.Ask, w.Asker = askID, askID, theDelivery
w.AskDigest = ask.Digest()
withRouter(t, map[string]map[string]aRouterRecord{theDelivery: {
"merge-377-aaaaaaaaaaaa-1": {},
askID: {ask, w},
}})
why, found, reason, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head)
if err != nil || !found || !strings.Contains(why, askID) {
t.Fatalf("the warrant was not found: %q %v %q %v", why, found, reason, err)
}
// Another head names another ask, and none is found.
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, "ffffffffffff0000"); found {
t.Fatal("another head found a warrant")
}
// A router that cannot be read is an error, never "no warrant".
routerAskIDs = func(context.Context, catalogueReader, string) ([]string, error) { return nil, errors.New("no answer") }
if _, found, _, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head); found || err == nil {
t.Fatalf("an unreadable router read as no warrant: %v %v", found, err)
}
}
// A trunk shorter than a page, or with none, is read without a fault: the pages end when the forge's do.
func TestAShortTrunkIsReadWholeAndAnEmptyOneIsNoMerges(t *testing.T) {
was := trunkCommitsPage
+2 -44
View File
@@ -51,55 +51,13 @@ func attachTheChain(ctx context.Context, inv *inventory.Inventory, server *link.
fmt.Printf("chain: the builds are not laid out under their modules yet, and none is read until they are: %v\n", err)
go layBuildsOutSoon(ctx, inv)
}
// And the requests the move of delivery 1 found never handed over, asked again once (askAgain), once it is serving.
go askAgainSoon(ctx, inv)
// The requests the move of delivery 1 found never handed over are not asked again: the controller asks no build
// (novox/hq ADR 0501; issue 1851, A10), and the builder builds every fingerprint the indexer writes.
// And what delivery 1 recorded of what each machine was sent, copied onto the keys of the assignments (CopySentBuilds).
go copySentBuildsSoon(ctx, inv)
return nil
}
// askAgain asks once more for each build request the move found never handed over (novox/hq issue 981 note 7 Q5): the
// requests a merge could not ask for, which a person's hand `build` would otherwise have to repeat. **At most once**: a
// request is claimed before it is asked and settled after, so a crash between loses the ask, which is where it stood.
func askAgain(ctx context.Context, inv *inventory.Inventory) {
pending, err := inv.RequestsToAskAgain(ctx)
if err != nil {
fmt.Printf("chain: the requests never handed over cannot be read: %v\n", err)
return
}
for _, r := range pending {
if claimed, err := inv.ClaimToAskAgain(ctx, r.ID); err != nil || !claimed {
if err != nil {
fmt.Printf("chain: %s cannot be claimed to be asked again: %v\n", r.ID, err)
}
continue
}
id, err := askABuild(ctx, buildSource{Repository: r.Repository, Seat: r.Seat}, r.Path, r.Ref)
if err != nil {
if serr := inv.AskedAgainAs(ctx, r.ID, "not asked again: "+err.Error()); serr != nil {
fmt.Printf("chain: %s was not asked again, and that could not be kept: %v\n", r.ID, serr)
}
fmt.Printf("chain: %s was never handed over and cannot be asked again: %v\n", r.ID, err)
continue
}
recordBuildRequest(ctx, inv, inventory.BuildRequest{ID: id, Repository: r.Repository, Seat: r.Seat, Path: r.Path,
Ref: r.Ref, Commit: r.Commit, For: r.For})
if err := inv.AskedAgainAs(ctx, r.ID, id); err != nil {
fmt.Printf("chain: %s was asked again as %s, and that could not be kept: %v\n", r.ID, id, err)
}
fmt.Printf("chain: %s, never handed over, is asked again as %s\n", r.ID, id)
}
}
// askAgainSoon waits for the controller to be serving, which an ask needs, and then asks again.
func askAgainSoon(ctx context.Context, inv *inventory.Inventory) {
select {
case <-ctx.Done():
case <-time.After(time.Minute):
askAgain(ctx, inv)
}
}
// kindSentBuildsNotCopied is the condition of the one-time copy of node.sent_builds onto the assignments' keys that has not
// succeeded: until it has, a module an assignment pulled in may be missing from what a machine is read to run.
const kindSentBuildsNotCopied = "sent-builds-not-copied"
+4 -17
View File
@@ -14,7 +14,6 @@ import (
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 10 of design 47: one read of an assignment names the commit, the build and the rollout that put it there,
@@ -80,7 +79,7 @@ func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRollou
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
for try := 0; try < 6 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == "" || asking.Modules["notes"].State == "asked"); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
@@ -90,13 +89,6 @@ func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRollou
if buildID == "" || len(*asked) != 1 || !strings.HasSuffix((*asked)[0], "#"+buildID) {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("a", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.ReuseKey = "rk-notes"
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// The walk sends the tier to its first machine.
for try := 0; try < 3 && len(sent) == 0; try++ {
@@ -116,9 +108,9 @@ func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRollou
if wireguard, pulled := assignmentOf(on, "mesh-wireguard"); pulled && (wireguard.Build != "" || wireguard.Rollout != "") {
t.Fatalf("a module this rollout did not send names a build or a rollout: %+v", wireguard)
}
if a.Module != "notes" || a.Commit != "c-notes" || a.Build != buildID || a.Rollout != rollout || a.Artifact != digest ||
if a.Module != "notes" || a.Commit != "c-notes" || a.Build != buildID || a.Rollout != rollout ||
a.Since.IsZero() || a.Assigned.IsZero() || !a.Stands() {
t.Fatalf("the assignment names %+v, not commit c-notes, build %s, artifact %s, rollout %s", a, buildID, digest, rollout)
t.Fatalf("the assignment names %+v, not commit c-notes, build %s, rollout %s", a, buildID, rollout)
}
// One read of the rollout.
p, err := inv.PlanByID(ctx, rollout)
@@ -130,16 +122,11 @@ func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRollou
p.Steps[0].Node != "anchor" || p.Steps[0].Sent == nil {
t.Fatalf("the rollout lists changed %+v, builds %v, steps %+v; want %+v", p.Changed, p.Builds, p.Steps, wantChange)
}
// And one read of the build names the same commit, with what it published and its reuse key.
// And one read of the build names the same commit.
built, found, err := inv.BuildByID(ctx, buildID)
if err != nil || !found || built.Commit != "c-notes" || built.Module != "notes" {
t.Fatalf("the build: %+v %v %v", built, found, err)
}
lately, err := inv.Builds(ctx, "notes", 1)
if err != nil || len(lately) != 1 || lately[0].ReuseKey != "rk-notes" || lately[0].Published == nil ||
lately[0].Published.Artifact != digest {
t.Fatalf("what the build published: %+v %v", lately, err)
}
// The verbs say the links.
if lines := changedLines(p); len(lines) != 2 || !strings.Contains(lines[1], "anchor/notes") ||
!strings.Contains(lines[1], buildID) {
@@ -6,8 +6,6 @@ import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 12 of design 47: no table in the controller's store holds builds, rollouts or assignments** (novox/hq ADR
@@ -34,7 +32,7 @@ func TestNoTableOfTheControllersStoreHoldsBuildsRolloutsOrAssignments(t *testing
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
for try := 0; try < 6 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == "" || asking.Modules["notes"].State == "asked"); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
@@ -44,12 +42,6 @@ func TestNoTableOfTheControllersStoreHoldsBuildsRolloutsOrAssignments(t *testing
if buildID == "" || len(*asked) != 1 {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("b", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// What the live path made is in the chain.
if built, found, err := inv.BuildByID(ctx, buildID); err != nil || !found || built.Commit != "c-notes" {
-29
View File
@@ -190,32 +190,3 @@ func summaryOf(p link.ChangePlan) string {
}
return strings.Join(parts, "; ")
}
// planText is a change plan as a person reads it, for the pull request's comment.
func planText(p link.ChangePlan) string {
var b strings.Builder
fmt.Fprintf(&b, "change plan of %s at %.8s into %s: %s\n", p.Repository, p.Head, p.Base, p.Summary)
for i, tier := range p.Tiers {
fmt.Fprintf(&b, " tier %d: %s\n", i, strings.Join(tier, ", "))
}
for _, m := range p.Machines {
line := " " + m.Machine + ": "
if len(m.Receives) > 0 {
line += "receives " + strings.Join(m.Receives, ", ")
}
if len(m.Waits) > 0 {
if len(m.Receives) > 0 {
line += "; "
}
line += "waits for a person: " + strings.Join(m.Waits, ", ")
}
b.WriteString(line + "\n")
}
for _, s := range p.Steps {
b.WriteString(" - " + s + "\n")
}
if len(p.Unread) > 0 {
fmt.Fprintf(&b, " read by no module's build: %s\n", strings.Join(p.Unread, ", "))
}
return b.String()
}
-3
View File
@@ -72,7 +72,4 @@ func TestAChangePlanSaysWhatEachMachineReceives(t *testing.T) {
if !strings.HasPrefix(p.Summary, "builds nothing") || len(p.Machines) != 0 || strings.Join(p.Unread, ",") != "merge-check.sh" {
t.Errorf("a root file's plan reads %+v", p)
}
if !strings.Contains(planText(p), "read by no module's build: merge-check.sh") {
t.Errorf("the plan's text does not say why nothing is built:\n%s", planText(p))
}
}
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
)
// A pull request's check builds go when it closes (novox/hq ADR 0350 §5): the build seat keeps them in the artifact
// store's scratch namespace for the walk of its merge to take. **Let go of where the moment is heard**: the
// controller ends the walk answering a merge — after that walk took what it reuses — and the delivery's owner, which
// hears the forge say a pull request closed unmerged, asks the `forget-checks` verb; both let go through the store,
// as the controller's own collection does. The build seat's holders let go of anything a week old on their own,
// so a word lost here leaves a week's builds, never more.
// checkBuildsIn is the scratch namespace as the controller reaches the artifact store.
func checkBuildsIn(ctx context.Context, inv *inventory.Inventory) (builder.Scratch, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is not on the mesh's network")
}
return builder.Registry{Address: address}, nil
}
// forgetChecks lets go of a pull request's check builds and answers what it did.
var forgetChecks = func(ctx context.Context, inv *inventory.Inventory, pull builder.PullRef) (string, error) {
scratch, err := checkBuildsIn(ctx, inv)
if err != nil {
return "", err
}
n, err := scratch.Forget(ctx, pull)
if err != nil {
return "", fmt.Errorf("%s: %d check(s)' builds let go of, and not all: %w", pull, n, err)
}
return fmt.Sprintf("%s: %d check(s)' builds let go of", pull, n), nil
}
// forgetChecksOfWalk lets go, on the save that ends a walk, of the check builds of every pull request whose merge
// it answers: the walk took what it reuses. Said, never fatal.
func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) {
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
continue
}
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
}
}
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
gateEvery = 0
digest := "sha256:" + strings.Repeat("c", 64)
reused := func(id, reference string) link.BuildResult {
manifest := mustJSON(t, catalogue.Manifest{Module: "app", Version: "c3", Resources: []map[string]any{
{"id": "svc", "type": "container", "name": "app", "image": reference}}})
return link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: "modules/app", Ref: "main", On: "anchor",
Commit: "c3", Manifest: manifest, Trunk: "main", OnTrunk: true, Branches: []string{"main"},
Made: []link.MadeArtifact{{Name: "server", Kind: "image", Reference: reference}},
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"},
Reused: "reused from the check of novox/mesh-catalog#9 at head abcdefabcdef (merged into trunk 123456123456): server " + digest,
ReusedFrom: &link.ReusedFrom{Pull: link.PullRef{Owner: "novox", Repo: "mesh-catalog", Number: 9},
Head: "abcdefabcdef", Trunk: "123456123456", Key: "r1:" + strings.Repeat("1", 64)}}
}
if _, _, err := takeIn(ctx, inv, reused(link.NewBuildID(time.Now()), "store.internal:5000/scratch/app/server@"+digest)); err == nil ||
!strings.Contains(err.Error(), "scratch") {
t.Fatalf("a build in the scratch namespace was registered: %v", err)
}
id := link.NewBuildID(time.Now().Add(time.Second))
result := reused(id, "store.internal:5000/app/server@"+digest)
if _, _, err := takeIn(ctx, inv, result); err != nil {
t.Fatal(err)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil || current["app"].Commit != "c3" {
t.Fatalf("the reused build is not what app is registered at: %+v %v", current["app"], err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(shelf["app"].Resources)
if !strings.Contains(string(raw), "app/server@"+digest) {
t.Fatalf("app is registered as %s, not by the check's digest in its own namespace", raw)
}
// The walk asked it: the walk keeps what its build did, and its gate judges the registered build.
asked := time.Now().UTC()
p := g.plan(t)
p.Commit = "c3"
p.Modules["app"] = &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: id}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
planBuiltSaying(ctx, g.open, "app", "c3", "", asked, id, result.MadeWords())
p = g.plan(t)
if m := p.Modules["app"]; m.State != "built" || !strings.HasPrefix(m.Made, "reused from the check of novox/mesh-catalog#9") {
t.Fatalf("the walk keeps %+v", m)
}
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p = g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.To != "c3" || gate.Verdict != inventory.GatePassed {
t.Fatalf("the gate judged %+v, not the reused build", gate)
}
if sent, _, _ := inv.SentBuilds(ctx, "anchor"); sent["app"] != "c3" {
t.Fatalf("the first machine was sent %v, not the reused build", sent)
}
}
-193
View File
@@ -1,193 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// One build per change (novox/hq ADR 0350), as the controller asks it and takes it in.
// The check of a change is asked to build what a merge of it would move, in the walk's order: a module of the
// repository checked from the merge, one standing on it from its own repository as the walk builds it, and one
// that reads the repository through a context not at all. A change that moves no module builds nothing.
func TestACheckIsAskedToBuildWhatTheMergeWouldMove(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "app"}, Source: inventory.Source{Repository: "novox/shop", Seat: "git", Path: "modules/app"}},
{Manifest: catalogue.Manifest{Module: "web"}, Source: inventory.Source{Repository: "novox/web", Seat: "git", Ref: "main"}},
{Manifest: catalogue.Manifest{Module: "docs"}, Source: inventory.Source{Repository: "novox/docs", Seat: "git"}},
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Base: "main", Commit: "abc"}
scope := checkScope{Reach: mergeReach{Packaging: entries[2:], Plan: inventory.Plan{Tiers: [][]string{{"app", "docs"}, {"web"}}}}}
clone := func(s inventory.Source) (string, error) { return "http://forge.invalid/" + s.Repository + ".git", nil }
got := checkBuildsOf(p, scope, entries, clone)
want := []link.CheckBuild{
{Module: "app", Path: "modules/app", Merged: true, Tier: 0},
{Module: "docs", Tier: 0, Skip: "its build reads novox/shop through a context, which a check cannot hand the merge"},
{Module: "web", Repository: "http://forge.invalid/novox/web.git", Ref: "main", Tier: 1},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("the check is asked to build\n%+v\nwant\n%+v", got, want)
}
if got := checkBuildsOf(p, checkScope{}, entries, clone); got != nil {
t.Fatalf("a change that moves no module asks for builds: %+v", got)
}
}
// A walk's ask carries its leave to take a check's build, naming the pull requests of the merges it answers; a dry
// run never does, and neither does an ask that is not a walk's.
// A walk ended lets go of its merges' check builds — only once it ended, after it took what it reuses.
func TestAWalkEndedLetsGoOfItsMergesCheckBuilds(t *testing.T) {
var forgot []builder.PullRef
was := forgetChecks
forgetChecks = func(_ context.Context, _ *inventory.Inventory, pull builder.PullRef) (string, error) {
forgot = append(forgot, pull)
return "", nil
}
t.Cleanup(func() { forgetChecks = was })
p := inventory.Plan{ID: "plan-1", Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}}}}
p.WalkSaid = inventory.WalkStarted
forgetChecksOfWalk(t.Context(), nil, p)
p.WalkSaid = ""
forgetChecksOfWalk(t.Context(), nil, p)
if len(forgot) != 0 {
t.Fatalf("a walk not ended let go of %v, which it may still take", forgot)
}
p.WalkSaid = inventory.WalkEnded
forgetChecksOfWalk(t.Context(), nil, p)
if want := []builder.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}; !reflect.DeepEqual(forgot, want) {
t.Fatalf("the walk ended let go of %v", forgot)
}
}
// A build in the scratch namespace is never registered: only the walk's copy in the module's own namespace is.
func TestABuildInTheScratchNamespaceIsNeverRegistered(t *testing.T) {
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: "store.internal:5000/app/server@sha256:aa"}}); got != "" {
t.Fatalf("a module's own build reads as a scratch build: %s", got)
}
scratch := "store.internal:5000/scratch/app/server@sha256:aa"
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: scratch}}); got != scratch {
t.Fatalf("a scratch build was not found: %q", got)
}
if got := inScratch([]link.MadeArtifact{{Name: "look", Reference: "http://store.internal:5000/v2/scratch/app/look/blobs/sha256:bb"}}); got == "" {
t.Fatal("a scratch archive was not found")
}
}
// The walk says per module what it did: "reused from …" or "built: <why>" — and a build seat from before says
// neither, which is said.
func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
if got := (link.BuildResult{Reused: "reused from the check of novox/shop#7"}).MadeWords(); got != "reused from the check of novox/shop#7" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{BuiltWhy: "no check built it"}).MadeWords(); got != "built: no check built it" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{}).MadeWords(); !strings.Contains(got, "predates ADR 0350") {
t.Fatalf("said %q", got)
}
}
// A walk's module the check's build stood in for is registered by the digest the check built — the copy in the
// module's namespace — and the walk keeps "reused from …" for it; its first-node gate judges that registered
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
-203
View File
@@ -1,203 +0,0 @@
package main
import (
"context"
"errors"
"flag"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// `check-here` is a pull request's merge check run on the machine at hand **exactly as the build seat
// runs it** (novox/hq issue 286): the same code (builder.Check), the same ask the controller would make of
// the seat — the gate's modules and judge by the planner's own answer over the facts snapshot, the
// repositories beside it at the refs the snapshot says the seat clones them at — in the toolchain image
// the mesh holds, as the user the build seat runs as, against a throwaway store and bus of the versions
// the mesh runs.
//
// **The build seat is the reference.** A merge-check.sh that passed on an agent's machine failed on the
// seat for three reasons that were each the agent's environment, never the change: a newer Go whose gofmt
// lays a file out differently, a sibling checkout at the agent's feature branch where the seat had the
// commit the mesh runs, and a different user. Run here, a check sees what the seat will.
//
// check-here [--tree <checkout>] [--base main] [--registry <artifact store>] [--forge <url of the owner>]
// [--number <n>] [--user uid:gid] [--facts store|<file>] [--keep]
//
// The checkout's HEAD is what is checked, and it must be committed: the seat checks a commit, never a
// working tree.
func checkHereCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("check-here", flag.ContinueOnError)
tree := set.String("tree", ".", "the change's checkout; its HEAD is checked")
base := set.String("base", "main", "the branch the change would merge into")
registry := set.String("registry", os.Getenv("MESH_REGISTRY"), "the artifact store holding the facts and the toolchains")
forge := set.String("forge", "", "where the repositories beside it are cloned from, as <forge>/<repository>.git; "+
"the checkout's origin without its own name when not given")
number := set.Int("number", 0, "the pull request's number, when there is one")
// The build seat's service runs as root, and its check containers run as the builder does.
user := set.String("user", "0:0", "the user the check's containers run as: the build seat's")
keep := set.Bool("keep", false, "keep the workspace afterwards")
factsFrom := set.String("facts", "store", "the facts snapshot: `store`, the one the artifact store holds — "+
"what the seat reads — or a file")
if _, err := parseAround(set, args); err != nil {
return err
}
if *registry == "" {
return errors.New("check-here reads the facts and the toolchains from the artifact store: --registry <host:port> or MESH_REGISTRY")
}
dir, err := filepath.Abs(*tree)
if err != nil {
return err
}
git := func(args ...string) (string, error) {
cmd := exec.CommandContext(ctx, "git", args...)
cmd.Dir = dir
out, err := cmd.Output()
return strings.TrimSpace(string(out)), err
}
if dirty, err := git("status", "--porcelain", "--untracked-files=no"); err != nil {
return fmt.Errorf("%s is not a checkout: %w", dir, err)
} else if dirty != "" {
return errors.New("the checkout has changes not committed: the build seat checks a commit, so commit first")
}
head, err := git("rev-parse", "HEAD")
if err != nil {
return err
}
origin, err := git("remote", "get-url", "origin")
if err != nil {
return fmt.Errorf("the checkout has no origin to say which repository it is: %w", err)
}
owner, repo, prefix := ownerRepoOf(origin)
if *forge == "" {
*forge = prefix
}
if _, err := git("fetch", "--quiet", "origin", *base); err != nil {
return fmt.Errorf("cannot fetch %s to say what the change touches: %w", *base, err)
}
// Without rename detection, so a file moved out of a build source is said under its old name as well:
// its going is a change to the build that held it (novox/hq ADR 0267).
changedText, err := git("diff", "--name-only", "--no-renames", "origin/"+*base+"...HEAD")
if err != nil {
return err
}
var paths, removed []string
for _, p := range strings.Split(changedText, "\n") {
if p = strings.TrimSpace(p); p == "" {
continue
}
paths = append(paths, p)
if _, err := os.Stat(filepath.Join(dir, p)); os.IsNotExist(err) {
removed = append(removed, p)
}
}
_ = os.Setenv("MESH_REGISTRY", *registry)
f, err := readFacts(ctx, *factsFrom)
if err != nil {
return fmt.Errorf("the facts snapshot cannot be read: %w", err)
}
entries, read, edges, err := graphOfFacts(f)
if err != nil {
return err
}
p := link.PullUpdated{Owner: owner, Repo: repo, Number: *number, Base: *base, Commit: head, Paths: paths,
Removed: removed, ModuleDirs: moduleDirsIn(dir, paths), ModuleDirsSaid: true}
scope := pullScope(p, entries, read, edges)
_, scriptErr := os.Stat(filepath.Join(dir, builder.CheckScript))
if !scope.gated() && !scope.Mesh {
fmt.Printf("%s: %s, and the repository is not the mesh's: the build seat runs nothing for it\n",
owner+"/"+repo, noModuleTouched)
return nil
}
if !scope.gated() && scriptErr != nil {
fmt.Printf("%s: %s; %s\n", owner+"/"+repo, noModuleTouched, noMergeCheck)
return nil
}
toolchains := map[string]string{}
for language, reference := range f.Versions.Toolchains {
toolchains[language] = catalogue.Rerouted(reference, *registry)
}
if len(toolchains) == 0 {
return errors.New("the facts snapshot names no toolchain: it was taken by a controller from before " +
"issue 286, and the seat's toolchain cannot be known here")
}
beside := map[string]builder.Beside{}
for d, ref := range f.Beside {
from := d
if d == "mesh-controller-main" {
from = "mesh-controller"
}
beside[d] = builder.Beside{Repository: strings.TrimSuffix(*forge, "/") + "/" + from + ".git", Ref: ref}
}
id := fmt.Sprintf("check-here-%d", time.Now().UnixNano())
spec := builder.CheckSpec{ID: id, Repository: dir, Ref: head, Owner: owner, Repo: repo, Number: *number,
Paths: paths, Beside: beside, Modules: scope.Modules, New: scope.New, Manifests: scope.Manifests,
Base: *base, Judge: scope.Judge, Toolchain: toolchains["go"], Toolchains: toolchains, User: *user}
workspace, err := os.MkdirTemp("", "mesh-check-here-")
if err != nil {
return err
}
if !*keep {
defer removeWorkspace(spec.Toolchain, workspace, *user)
}
fmt.Fprintf(os.Stderr, "checking %s/%s at %.8s as the build seat would, against the facts of %s, in %s\n",
owner, repo, head, f.Taken.Format(time.RFC3339), workspace)
v, err := builder.Check(ctx, builder.Command, spec, workspace, *registry, builder.GitCredential{},
func(step, message string) {
if step != "output" {
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
}
})
if err != nil {
return fmt.Errorf("the check could not run — on the seat an error, never a pass: %w", err)
}
fmt.Println(v.Report)
fmt.Println()
fmt.Printf("mesh/merge-gate: %s — %s\n", strings.ToUpper(v.Gate.Verdict), v.Gate.Summary)
if v.Repo != nil {
fmt.Printf("mesh/repo-check: %s — %s\n", strings.ToUpper(v.Repo.Verdict), v.Repo.Summary)
}
for _, l := range []*builder.Layer{v.Gate, v.Repo} {
if l != nil && l.Verdict != "pass" && l.Verdict != "warning" {
return errors.New("the build seat would not pass this change")
}
}
return nil
}
// ownerRepoOf reads owner, repository and the owner's URL from a remote: ssh://git@host:222/novox/mesh-host.git
// → novox, mesh-host, ssh://git@host:222/novox.
func ownerRepoOf(remote string) (string, string, string) {
trimmed := strings.TrimSuffix(strings.TrimSuffix(remote, "/"), ".git")
cut := strings.LastIndexAny(trimmed, "/:")
if cut < 0 {
return "", trimmed, ""
}
repo, prefix := trimmed[cut+1:], trimmed[:cut]
owner := prefix
if at := strings.LastIndexAny(prefix, "/:"); at >= 0 {
owner = prefix[at+1:]
}
return owner, repo, prefix
}
// removeWorkspace removes what the check left, written as the seat's user: by a container of that user
// when it is not this one.
func removeWorkspace(image, workspace, user string) {
if image != "" && user != fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid()) {
// Everything in it — the check's HOME is the workspace, so the toolchain's own files are there too.
_ = exec.Command("docker", "run", "--rm", "--user", user, "--volume", workspace+":/workspace", image,
"sh", "-c", "rm -rf /workspace/* /workspace/.[!.]*").Run()
}
_ = os.RemoveAll(workspace)
}
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"reflect"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// 1212/AC1 and AC2: the check request names the pulls bucket of each module claiming the delivery seat, in the order
// of the claimants' names.
func TestACheckNamesThePullsBucketsOfTheSeatsClaimants(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), namedOnly("mesh-delivery")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery"), namedOnly("gitea")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
"none": {[]inventory.Entry{namedOnly("gitea")}, nil},
} {
if got := pullsBucketsOf(c.entries); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: %v, want %v", name, got, c.want)
}
}
}
// 1212/AC1 and AC2: the check a request carries names those buckets (the wiring, not only the helper).
func TestTheCheckRequestCarriesThePullsBuckets(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
} {
got := checkOf(link.PullUpdated{Owner: "novox", Repo: "a", Number: 7}, checkScope{}, c.entries, nil, nil, nil, nil)
if !reflect.DeepEqual(got.Pulls, c.want) || got.Owner != "novox" || got.Number != 7 {
t.Errorf("%s: %+v, want pulls %v", name, got, c.want)
}
}
}
-165
View File
@@ -1,165 +0,0 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
siblingsFrom(request, p, entries, lines, err)
}
// siblingsFrom is withSiblings from mesh-delivery's answer already had.
func siblingsFrom(request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry, lines []groupsLine, err error) {
if request.Check == nil {
return
}
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
// composedMember says the pull request is a member of a delivery group whose gate is the composed check's (novox/hq
// ADR 0403 clauses 1 and 7): the group has two members or more and mesh-delivery says so on its line. The group is
// found as siblingsFor finds it, by its head branch name, or, for a recheck that carries none, by the pull request's
// repository and number — not by siblings, which name only what the check reads beside it.
func composedMember(p link.PullUpdated, lines []groupsLine) bool {
own := strings.ToLower(p.Owner + "/" + p.Repo)
for _, l := range lines {
found := p.Head != "" && l.ID == p.Head
for _, m := range l.Details {
if p.Head == "" && p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
found = true
}
}
if found {
return l.ComposedGate && max(len(l.Details), len(l.Members)) >= 2
}
}
return false
}
-248
View File
@@ -1,248 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
+22 -517
View File
@@ -2,8 +2,6 @@ package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"path"
@@ -12,46 +10,18 @@ import (
"slices"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A pull request's merge check (novox/hq to-be 45 §9, ADR 0237 as amended 2026-10-06): the forge
// announces every pull request's new head, the controller decides what is checked, asks the build seat to
// check it, and says the verdict as `checked`, which the forge's holder sets as the pull request's
// statuses. **Before merge, never after**: every check the mesh had ran after a merge, on a machine.
//
// **The mesh's module graph decides, not the repository.** The controller holds the graph — every module,
// the repository and directory it is built from — and maps the pull request's changed paths onto it by
// the planner's own answer (reachOfMerge, touchedBy — the one place a changed file is mapped onto modules,
// for the merge handler, the release planner, the merge gate and this check; issue 280): **a changed file
// touches exactly the modules whose build reads it** — a module's own directory (the whole repository for
// one built from its root), or a repository its recipe packages. A file no build reads — a script at the
// root, a README — touches no module. A directory the change adds a module.json in, which the graph does
// not hold yet (said by the head, issue 278), is a new module and is checked too.
//
// - touches a module: the build seat runs **the gate** — `mesh/merge-gate`, the touched manifests, every
// machine composed with the change, the replays — and the repository's own merge-check.sh beside it;
// - touches none, in a repository that is the mesh's (it sources a module on some branch, or shares the
// core's owner): the gate is a pass that says so — a fact, not a missing check — and the repository's
// own merge-check.sh runs as `mesh/repo-check`, a warning when it has none;
// - touches none, anywhere else: the gate is a pass that says so, and nothing more is said.
//
// What is checked is decided here and run there (internal/builder/check.go).
// checkTimeout is how long one check may run on the build seat. Said here so the ask's watchdog (S6)
// and the builder agree on what late means.
const checkTimeout = 45 * time.Minute
// noModuleTouched is the gate's word for a change that touches nothing of the graph.
const noModuleTouched = "the change touches no module of the mesh's graph"
// noMergeCheck is the repository layer's word for a repository of the mesh with no merge-check.sh.
const noMergeCheck = "the repository declares no merge-check.sh: none of its own tests run before it merges"
// What a pull request reaches of the mesh's graph (novox/hq to-be 45 §9, issue 280). The controller no longer checks a
// pull request or says a verdict on it (R4g, ADR 0503 §3, ADR 0501 §5): the checker judges the heads, and mesh-delivery
// asks the operator's Approve on its verdict. What stays here is the mapping of a pull request's changed paths onto the
// mesh's module graph by the planner's own answer (reachOfMerge, touchedBy), which the delivery verbs (`delivery order`,
// the change plan) and the head's declared config (tellPullDeclares) read: **a changed file touches exactly the modules
// whose build reads it** — a module's own directory (the whole repository for one built from its root), or a repository
// its recipe packages — and a directory the change adds a module.json in is a new module.
// coreModules are the modules whose repositories are the mesh's core, by the directory a check finds
// each beside it — and whose owner is the mesh's own.
@@ -75,8 +45,6 @@ type checkScope struct {
// Mesh says the repository is the mesh's: modules are built from it on some branch, its owner is the
// core's, or the change adds a module to it.
Mesh bool
// Judge is who judges the gate (link.JudgeSelf, or the running controller).
Judge string
// From is a module built from the repository, for how the mesh clones it; nil when none is.
From *inventory.Entry
// Reach is the planner's whole answer, which the change plan is made from.
@@ -98,10 +66,6 @@ func pullScope(p link.PullUpdated, entries []inventory.Entry, read map[string][]
Width: len(r.Plan.Modules), Tiers: len(r.Plan.Tiers), Reach: r}
for _, e := range append(append([]inventory.Entry{}, r.Touched...), r.Deleted...) {
s.Manifests = append(s.Manifests, path.Join(strings.Trim(e.Source.Path, "/"), moduleManifestFile))
switch e.Manifest.Module {
case "mesh-controller":
s.Judge = link.JudgeSelf
}
}
for _, d := range r.Added {
s.Manifests = append(s.Manifests, path.Join(d, moduleManifestFile))
@@ -137,315 +101,37 @@ func sourceOwner(repository string) string {
return strings.ToLower(parts[len(parts)-2])
}
// PullUpdated decides a pull request's merge check, and asks for it when there is something to run.
// PullUpdated hears a pull request's new head. It asks no check and says no verdict (pullUpdated).
func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
_, err := f.pullUpdated(ctx, p, os.Stdout)
return err
return f.pullUpdated(ctx, p, os.Stdout)
}
// pullUpdated is PullUpdated saying what it did to w. It returns the id of the check it asked of the build seat, or
// "" when it said a verdict itself and asked nothing. A command that answers JSON on standard output (`delivery
// check`) gives it standard error, so its output stays the document alone (novox/hq issue 1311).
func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Writer) (string, error) {
// pullUpdated is PullUpdated saying what it did to w. It asks nothing of anybody and says no verdict (R4g; ADR 0503 §3, §5,
// ADR 0527): the checker judges every repository whose trunk requires `mesh/check`, hq included. All it still does is tell
// mesh-config what the head declares (ADR 0502), whatever judges it.
func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Writer) error {
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
return "", err
return err
}
read, err := readForPlanning(ctx, inv)
if err != nil {
return "", err
return err
}
edges, err := inv.Dependencies(ctx)
if err != nil {
return "", err
return err
}
scope := pullScope(p, entries, read, edges)
// **A cycle of edges fails the check, naming every module and edge** (novox/hq ADR 0419 clause 6), a lone pull
// request's as a group's composed check is: no order exists for what it changes.
alone := orderMember{ID: p.Owner + "/" + p.Repo + "@" + twelve(p.Commit), Repository: p.Owner + "/" + p.Repo, Base: p.Base,
Head: p.Commit, Number: p.Number, Paths: p.Paths, PathsTruncated: p.PathsTruncated, Removed: p.Removed, BaseSHA: p.BaseSHA}
if failing, cyclic, err := cycleVerdict(ctx, "", []orderMember{alone},
map[string]orderReach{alone.ID: {Moved: scope.Modules}}, edges, entries); err != nil {
return "", err
} else if cyclic {
failing.Commit, failing.Trunk = p.Commit, twelve(p.BaseSHA)
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, failing.Summary)
sayChecked(ctx, w, failing)
return "", nil
this := pullAtOf(p)
if this.Truncated {
this.Paths, this.Truncated = truncatedPaths(entries, p.Owner, p.Repo, scope.Modules), false
}
// **A member of a group whose gate is the composed check's runs its repository's check alone** (novox/hq ADR 0403
// clause 1): mesh-delivery is asked once, and says so on the group's line. Told nothing, the pull request is
// checked alone, gate included (clause 6).
lines, groupsErr := groupsAsked(ctx, deliverySeatHeld(entries))
composed := groupsErr == nil && composedMember(p, lines)
// The change plan of the commit at hand (ADR 0238): what a merge of it would build and send, posted
// with the verdict whatever the verdict is.
plan := changePlanOf(p.Owner+"/"+p.Repo, p.Base, p.Commit, scope.Reach, entries, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
})
fmt.Fprint(w, planText(plan))
direct := directVerdict(p, plan)
if composed {
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
if scope.gated() || holdsModules(entries, p.Owner, p.Repo) {
tellPullDeclares(ctx, w, []pullAt{this})
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched)
sayChecked(ctx, w, direct)
return "", nil
case !scope.gated() && p.MergeCheckSaid && !p.MergeCheck:
direct.RepoCheck = &link.CheckLayer{Verdict: "warning", Summary: noMergeCheck}
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
sayChecked(ctx, w, direct)
return "", nil
}
request, err := checkRequestFor(ctx, f.open, p, scope, entries)
if err != nil {
return "", err
}
request.Check.Plan = &plan
request.Check.Composed = composed
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return "", err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return "", err
}
// **A newer request replaces the one still waiting, and a newer key stops the one running** (novox/hq ADR 0406 §3, §4,
// issue 974): the check queue keeps one request a pull request, and a running check watches its key. Nothing is asked
// of the forge here (issue 726).
what := "its own merge-check.sh alone: " + noModuleTouched
if composed {
what = "its own merge-check.sh alone: the gate is its group's composed check's"
}
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
strings.Join(append(append([]string{}, scope.Modules...), prefixedAll("new:", scope.New)...), ", "),
scope.Width, scope.Tiers)
}
fmt.Fprintf(w, "%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
p.Commit, seat, what, request.ID)
return request.ID, nil
}
// composedWords is what a group member's own verdict says of its gate layer (novox/hq ADR 0403 clause 1).
const composedWords = "judged by its group's composed check"
// gateComposed is a gate layer's verdict when no gate was run for the member alone: never a pass. Mesh-delivery reads it
// as the composed check's to judge, and the forge's holder sets no mesh/merge-gate from it.
const gateComposed = "composed"
// composedGate says a group member's own verdict has no gate of its own: its gate layer, and the verdict that is
// the gate's, read "composed" in place of a pass.
func composedGate(c *link.Checked) {
c.Verdict, c.Summary = gateComposed, composedWords
c.Gate = &link.CheckLayer{Verdict: gateComposed, Summary: composedWords}
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
// the mesh's or the head holds no merge-check.sh. **It names the trunk commit it holds for** (novox/hq ADR 0323 §3):
// the base branch's head the forge announced with the head, so the merge tool can weigh it against the trunk's head
// and does not refuse it for ever (a verdict naming no trunk is never the trunk's head's). Nothing was built to
// judge: the trunk commit is the one the forge said the change was announced against.
func directVerdict(p link.PullUpdated, plan link.ChangePlan) link.Checked {
trunk := p.BaseSHA
if len(trunk) > 12 {
trunk = trunk[:12]
}
if trunk != "" {
plan.Trunk = trunk
}
return link.Checked{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
ID: link.NewBuildID(time.Now()), Verdict: "pass", Summary: noModuleTouched,
Gate: &link.CheckLayer{Verdict: "pass", Summary: noModuleTouched}, Plan: &plan, Trunk: trunk}
}
// checkRequestFor is the ask for one pull request's head: the repository as the mesh clones it, the head,
// and what is read beside it.
func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scope checkScope,
entries []inventory.Entry) (link.BuildRequest, error) {
shelf := map[string]catalogue.Manifest{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
}
world, err := restOfTheMesh(ctx, open, shelf, "")
if err != nil {
return link.BuildRequest{}, err
}
clone := func(s inventory.Source) (string, error) {
if s.Seat == "" {
return s.Repository, nil
}
return clonedFromSeat(world, s.Seat, s.Repository)
}
// As the mesh clones a module built from it; a repository no module is built from, from the forge.
source := inventory.Source{Seat: gitSeat, Repository: p.Owner + "/" + p.Repo}
if scope.From != nil {
source = scope.From.Source
}
repository, err := clone(source)
if err != nil {
return link.BuildRequest{}, err
}
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return link.BuildRequest{}, err
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: checkOf(p, scope, entries, beside, builds, namedForTheCheck(ctx, p, builds), runningBy(ctx)),
}, nil
}
// checkOf is the check a request carries: the pull request, what the change touches, and the buckets of the pull
// requests' keys of the modules claiming the delivery seat (novox/hq ADR 0437 §3, issue 1212).
func checkOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry, beside map[string]link.CheckedOut,
builds []link.CheckBuild, named []link.CheckedBuild, running map[string]string) *link.CheckRequest {
return &link.CheckRequest{Pulls: pullsBucketsOf(entries), Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Trunk: p.BaseSHA, Ready: p.Ready, Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents,
New: scope.New, Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: named, Running: running}
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs. running is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error)) (map[string]link.CheckedOut, error) {
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return nil, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
beside[sibling] = link.CheckedOut{Repository: url, Ref: refs[sibling]}
}
}
}
}
}
return beside, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
// branch it follows, as the walk builds it. A module whose build reads the repository checked through a context,
// or that the merge deletes, cannot be built as the walk would: said, and nothing after it is built. A change that
// moves no module builds nothing.
func checkBuildsOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry,
clone func(inventory.Source) (string, error)) []link.CheckBuild {
if len(scope.Reach.Plan.Tiers) == 0 {
return nil
}
m := link.SourceMoved{Owner: p.Owner, Repo: p.Repo, Base: p.Base, CloneURL: p.CloneURL}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
skip := map[string]string{}
for _, e := range scope.Reach.Packaging {
skip[e.Manifest.Module] = "its build reads " + p.Owner + "/" + p.Repo + " through a context, which a check " +
"cannot hand the merge"
}
for _, e := range scope.Reach.Deleted {
skip[e.Manifest.Module] = "the merge deletes it"
}
var out []link.CheckBuild
for tier, names := range scope.Reach.Plan.Tiers {
for _, name := range names {
e, known := byName[name]
b := link.CheckBuild{Module: name, Tier: tier, Skip: skip[name]}
switch {
case !known:
b.Skip = "the catalogue does not hold it"
case b.Skip != "":
case sourceIs(e.Source, m):
b.Merged, b.Path = true, e.Source.Path
default:
url, err := clone(e.Source)
if err != nil {
b.Skip = "its repository cannot be cloned: " + err.Error()
break
}
b.Repository, b.Path, b.Ref = url, e.Source.Path, followedBranch(e.Source.Ref)
}
out = append(out, b)
}
}
return out
return nil
}
// besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under:
@@ -501,186 +187,5 @@ func sdkPinned() string {
return "main"
}
// siblingOf is another repository of the same owner: novox/mesh-controller → novox/mesh-lab.
func siblingOf(repository, name string) string {
if cut := strings.LastIndex(repository, "/"); cut >= 0 {
return repository[:cut+1] + name
}
return name
}
// sourceOnSeat is a source's seat form, nil for one on no seat.
func sourceOnSeat(s inventory.Source) *link.SourceOnSeat {
if s.Seat == "" {
return nil
}
return &link.SourceOnSeat{Seat: s.Seat, Repository: s.Repository}
}
// checkEvents is where the serving controller says a check's verdict; nil in a command.
var checkEvents link.Bus
// maxCheckReport is how much of a check's report travels in its verdict: enough for the failures and
// the machines, never a log.
const maxCheckReport = 60 << 10
// checked says a merge check's verdict as the controller's `checked`. Nothing is recorded or
// registered: a check builds nothing (issue 240's rule for a dry run, kept for a check).
func checked(ctx context.Context, result link.BuildResult) {
// **A check stopped by its key says nothing** (novox/hq ADR 0406 §4, issue 974, review 2): it stopped because the head or
// the trunk commit moved on, and the newer request will judge. Said as an error it would name the live head — red
// statuses on the forge, a delivery refused — and written to the key it would stand in the newer request's way.
if result.Failed == link.SupersededByKey {
fmt.Printf("%s: stopped, its pull request's key moved on; the newer check will judge\n", result.ID)
return
}
sayChecked(ctx, os.Stdout, checkedOf(result))
}
// checkedOf is what a check's outcome says: each layer, and an error — never a pass — for a check that
// could not run.
func checkedOf(result link.BuildResult) link.Checked {
c := link.Checked{ID: result.ID, On: result.On, Commit: result.Ref}
if result.Checked != nil {
c.Owner, c.Repo, c.Number = result.Checked.Owner, result.Checked.Repo, result.Checked.Number
}
switch {
case result.Check != nil:
c.Verdict, c.Summary, c.Report = result.Check.Verdict, result.Check.Summary, result.Check.Report
c.Gate, c.RepoCheck = result.Check.Gate, result.Check.RepoCheck
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Validator = result.Check.Validator
c.Siblings, c.SiblingsUnknown, c.JudgeFrom = result.Check.Siblings, result.Check.SiblingsUnknown, result.Check.JudgeFrom
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
}
case result.Failed != "":
// The check could not run: an error, never read as a pass — on both layers it was asked for.
c.Verdict, c.Summary = "error", "the check could not run: "+firstLine(result.Failed)
default:
c.Verdict, c.Summary = "error", "the build seat answered the check with no verdict"
}
if c.Verdict == "" {
c.Verdict = "error"
}
if result.Check == nil {
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
c.RepoCheck = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
}
// **A group member's own check has no gate of its own** (novox/hq ADR 0403 clause 1): what ran was the repository's
// check alone, and its gate layer is "composed", never a pass. A failure or an error stays what it was.
if result.Checked != nil && result.Checked.Composed && result.Check != nil && c.Gate != nil &&
(c.Gate.Verdict == "pass" || c.Gate.Verdict == "warning") {
composedGate(&c)
}
if result.Checked != nil && c.Gate != nil && len(c.Gate.Modules) == 0 {
c.Gate.Modules = append(append([]string{}, result.Checked.Modules...), prefixedAll("new:", result.Checked.New)...)
}
if result.Checked != nil && c.Gate != nil && len(c.Gate.Dependents) == 0 {
c.Gate.Dependents = result.Checked.Dependents
}
if result.Checked != nil {
c.Plan = result.Checked.Plan
if c.Plan != nil && c.Trunk != "" {
// The kept delivery plan records the trunk commit with its other inputs (ADR 0323 §3).
plan := *c.Plan
plan.Trunk = c.Trunk
c.Plan = &plan
}
// A delivery group's composed check (novox/hq ADR 0239): every head it judged, this one first.
if g := result.Checked; g.Group != "" {
c.Group = g.Group
c.Members = append(c.Members, link.CheckedMember{Owner: g.Owner, Repo: g.Repo, Number: g.Number,
Commit: result.Ref, Trunk: c.Trunk})
for _, m := range g.Members {
c.Members = append(c.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo, Number: m.Number,
Commit: m.Ref, Trunk: memberTrunk(result.Check, m)})
}
}
}
for _, l := range []*link.CheckLayer{c.Gate, c.RepoCheck} {
if l != nil && l.Verdict == "" {
l.Verdict = "error"
}
}
// **A check run again says so** (novox/hq issue 594). The summaries are the lines the pull request
// shows, clipped by the forge to a few words, so they get only a short marker after the verdict's own
// words; the report gets the whole sentence. Layers copied, so the build's own outcome is left as it came.
if mark := link.RunMarker(result.Delivery); mark != "" {
c.Summary += " " + mark
for _, l := range []**link.CheckLayer{&c.Gate, &c.RepoCheck} {
if *l != nil {
layer := **l
layer.Summary += " " + mark
*l = &layer
}
}
c.Report = link.Restarted(result.Delivery) + "\n\n" + c.Report
}
if len(c.Report) > maxCheckReport {
c.Report = "…" + c.Report[len(c.Report)-maxCheckReport:]
}
return c
}
// memberTrunk is the trunk commit the build seat merged one of a group's heads into (novox/hq ADR 0323 §6), empty
// when it said none.
func memberTrunk(outcome *link.CheckOutcome, m link.GroupMember) string {
if outcome == nil {
return ""
}
for _, j := range outcome.Members {
if strings.EqualFold(j.Owner, m.Owner) && strings.EqualFold(j.Repo, m.Repo) && strings.HasPrefix(m.Ref, j.Commit) {
return j.Trunk
}
}
return ""
}
func prefixedAll(prefix string, items []string) []string {
out := make([]string, 0, len(items))
for _, i := range items {
out = append(out, prefix+i)
}
return out
}
// sayChecked says a merge check's verdict on the bus, where the forge's holder hears it.
func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
repo := "none"
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
}
judged := ""
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Fprintf(w, "%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
if checkEvents == nil {
return
}
body, err := json.Marshal(c)
if err != nil {
return
}
stating, stop := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer stop()
if err := checkEvents.PublishSeatEvent(stating, link.MeshControllerSeat, link.KeyChecked, body); err != nil {
fmt.Fprintf(w, "%s: the verdict could not be said, so the pull request is not told it: %v\n", c.ID, err)
}
}
// pullsBucketsOf are the buckets of the pull requests' keys a check names for the build agent: one per module
// claiming the delivery seat, in the order of their names (novox/hq ADR 0437 §3, issue 1212).
func pullsBucketsOf(entries []inventory.Entry) []string {
var out []string
for _, m := range claimantsOf(entries, catalogue.DeliverySeat) {
out = append(out, broker.BucketName(m, "pulls"))
}
return out
}
@@ -1,56 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestACheckStoppedByItsKeyIsAnnouncedAsNoVerdict(t *testing.T) {
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
was := checkEvents
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = was })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyChecked), heard)
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 3}
checked(t.Context(), link.BuildResult{ID: "stopped", Ref: "abc", Checked: asked, Failed: link.SupersededByKey})
checked(t.Context(), link.BuildResult{ID: "broken", Ref: "abc", Checked: asked, Failed: "the facts snapshot cannot be read"})
select {
case m := <-heard:
var c link.Checked
_ = json.Unmarshal(m.Data, &c)
if c.ID != "broken" || c.Verdict != "error" {
t.Fatalf("the first verdict said is %+v; want only the broken check's error", c)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing was said")
}
select {
case m := <-heard:
t.Fatalf("a second verdict was said: %s", m.Data)
case <-time.After(300 * time.Millisecond):
}
}
-97
View File
@@ -1,97 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Acceptance 5 of milestone 2 (novox/hq ADR 0406 §3, issue 974), live, with the store and a real bus: two merges into one
// repository within a minute, with two open pull requests, leave each pull request with exactly one queued check, by the
// stream's rule. Each merge moves the trunk, and mesh-delivery asks the controller to check both open pull requests again
// against it (`delivery check`, which is the controller's own path for a head announced); nothing moves a request, nothing
// asks the forge, and the queue is read back whole.
func TestTwoMergesWithinAMinuteLeaveEachOpenPullRequestOneQueuedCheck(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
holdTheDeliverySeat(t, open)
was := askDeliveryOwner
t.Cleanup(func() { askDeliveryOwner = was })
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, _ map[string]any) (json.RawMessage, error) {
if verb != "groups" {
t.Fatalf("asked the owner %s", verb)
}
return json.RawMessage(`[]`), nil
}
// The repository the two pull requests are into, as a module the mesh builds from it.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
heads := map[int]string{7: "7777777777777777777777777777777777777777", 8: "8888888888888888888888888888888888888888"}
check := func(number int, trunk string) {
t.Helper()
p := orderMember{ID: fmt.Sprintf("novox/mesh-controller@%.12s", heads[number]), Repository: "novox/mesh-controller",
Base: "main", Head: heads[number], Number: number, Paths: []string{"main.go"}, ModuleDirsSaid: true,
BaseSHA: trunk, Ready: number == 8}.pull()
p.MergeCheck, p.MergeCheckSaid = true, true
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
}
// Announced against the trunk as it stands, then asked again after each of two merges.
for _, trunk := range []string{"1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222",
"3333333333333333333333333333333333333333"} {
check(7, trunk)
check(8, trunk)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for _, a := range q.Asks {
c := a.Request.Check
if c == nil {
t.Fatalf("%s is no check", a.ID)
}
if _, twice := seen[c.Number]; twice {
t.Errorf("pull request %d has two queued checks", c.Number)
}
seen[c.Number] = c.Trunk
if a.State != link.AskWaiting {
t.Errorf("%s is %s", a.ID, a.State)
}
if want := "3333333333333333333333333333333333333333"; c.Trunk != want {
t.Errorf("pull request %d is queued against %.8s; want the newest trunk %.8s", c.Number, c.Trunk, want)
}
}
if len(seen) != 2 {
t.Fatalf("queued checks for %v; want one for each of the two open pull requests", seen)
}
// By the streams' own count: the ready one on its stream, the other on its.
for stream, want := range map[string]uint64{broker.ChecksReadyStream: 1, broker.ChecksOtherStream: 1} {
info, err := js.Context().StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != want {
t.Errorf("%s holds %d request(s); want %d", stream, info.State.Msgs, want)
}
}
}
+2 -143
View File
@@ -1,7 +1,6 @@
package main
import (
"errors"
"strings"
"testing"
@@ -76,8 +75,8 @@ func TestThePullRequestIsMappedOntoTheModuleGraph(t *testing.T) {
t.Errorf("%s: %s %q, wanted %q", c.what, what, got[i], want[i])
}
}
if s.Mesh != c.mesh || s.Judge != c.judge {
t.Errorf("%s: the mesh's %v judged by %q, wanted %v by %q", c.what, s.Mesh, s.Judge, c.mesh, c.judge)
if s.Mesh != c.mesh {
t.Errorf("%s: the mesh's %v, wanted %v", c.what, s.Mesh, c.mesh)
}
if s.gated() != (c.modules != "" || c.new != "") {
t.Errorf("%s: gated %v", c.what, s.gated())
@@ -95,28 +94,6 @@ func TestAPullRequestTouchesWhatPackagesItsRepository(t *testing.T) {
}
}
// Each layer is said; a check that could not run is an error on both, never a pass; a build seat from
// before the layers is read as the gate.
func TestAChecksLayersAreEachSaidAndAnErrorIsNeverAPass(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 3, Modules: []string{"gitea"}}
c := checkedOf(link.BuildResult{ID: "b", Ref: "abc", Checked: asked, Failed: "the facts snapshot cannot be read"})
if c.Verdict != "error" || c.Gate == nil || c.Gate.Verdict != "error" || c.RepoCheck == nil || c.RepoCheck.Verdict != "error" {
t.Fatalf("a check that could not run said %+v", c)
}
if strings.Join(c.Gate.Modules, ",") != "gitea" {
t.Errorf("the gate names %v", c.Gate.Modules)
}
c = checkedOf(link.BuildResult{ID: "b", Ref: "abc", Checked: asked, Check: &link.CheckOutcome{Verdict: "warning", Summary: "wide"}})
if c.Gate == nil || c.Gate.Verdict != "warning" || c.RepoCheck != nil {
t.Fatalf("an outcome without layers said %+v", c)
}
c = checkedOf(link.BuildResult{ID: "b", Ref: "abc", Checked: asked, Check: &link.CheckOutcome{Verdict: "pass",
Gate: &link.CheckLayer{Verdict: "pass"}, RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "its merge-check.sh failed"}}})
if c.RepoCheck.Verdict != "fail" || c.Gate.Verdict != "pass" {
t.Fatalf("the layers said %+v / %+v", c.Gate, c.RepoCheck)
}
}
// **The check asks the planner, never a mapping of its own** (novox/hq ADR 0238): what a pull request
// reaches is reachOfMerge's answer — the same that plans a merge — so a file at the root touches no
// module in both, and what stands on a touched module is named as its dependents in both.
@@ -171,123 +148,5 @@ func TestAPullRequestReachesWhatAMergeOfItWouldPlan(t *testing.T) {
}
}
// **Only a commit on the trunk is published** (novox/hq ADR 0238): a build of a commit off its repository's
// default branch — a pull request's head, a branch built by hand, a rebuild or replay of one — is recorded
// and never registered, so nothing can send it; a check or a dry run never is either.
func TestABuildOffTheTrunkIsNeverPublished(t *testing.T) {
on := link.BuildResult{ID: "b", Commit: "abc", Trunk: "main", OnTrunk: true}
if err := publishable(on, ""); err != nil {
t.Errorf("a build on the trunk was refused: %v", err)
}
off := link.BuildResult{ID: "b", Commit: "abc", Trunk: "main"}
if err := publishable(off, ""); !errors.Is(err, errOffTheTrunk) || !strings.Contains(err.Error(), "main") {
t.Errorf("a build off the trunk was let through: %v", err)
}
for _, r := range []link.BuildResult{
{ID: "c", Trunk: "main", OnTrunk: true, Check: &link.CheckOutcome{Verdict: "pass"}},
{ID: "d", Trunk: "main", OnTrunk: true, Checked: &link.CheckRequest{}},
{ID: "e", Trunk: "main", OnTrunk: true, DryRun: true},
} {
if publishable(r, "") == nil {
t.Errorf("%s, a check or a dry run, was publishable", r.ID)
}
}
// A module that follows a branch other than the default: that branch is its trunk, and the default
// is not.
follows := link.BuildResult{ID: "g", Commit: "abc", Trunk: "master", Branches: []string{"nox-mesh"}}
if err := publishable(follows, "nox-mesh"); err != nil {
t.Errorf("a commit on the branch a module follows was refused: %v", err)
}
if err := publishable(link.BuildResult{ID: "h", Commit: "abc", Trunk: "master", OnTrunk: true,
Branches: []string{"master"}}, "nox-mesh"); err == nil {
t.Error("a commit on the default but not on the branch the module follows was published")
}
if err := publishable(link.BuildResult{ID: "i", Commit: "abc", Trunk: "main", Branches: []string{"feat/x"}}, ""); err == nil {
t.Error("a feature branch's commit was published")
}
// A build seat older than the rule says nothing: let through and said, so the rule can reach the mesh.
if err := publishable(link.BuildResult{ID: "f", Commit: "abc"}, ""); err != nil {
t.Errorf("a build seat that said nothing was refused: %v", err)
}
}
// **A verdict names the trunk commit it judged** (novox/hq ADR 0323 §3, issue 341): the controller's `checked`
// carries the head and the trunk commit the build seat merged it into, the commits of the repositories read beside
// it, how long it took, and the kept delivery plan records the trunk commit with its other inputs. A group's
// composed check names, per member, the trunk commit that head was merged into (§6).
func TestAVerdictNamesTheTrunkCommitItJudged(t *testing.T) {
plan := &link.ChangePlan{Repository: "novox/mesh-host", Base: "main", Head: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
Summary: "builds mesh-host"}
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 66, Base: "main", Plan: plan}
c := checkedOf(link.BuildResult{ID: "b", Ref: "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", Checked: asked,
Check: &link.CheckOutcome{Verdict: "pass", Took: "4m12s", Head: "dfb1415aaaaa", Trunk: "d28ca95bbbbb",
Gate: &link.CheckLayer{Verdict: "pass"}, RepoCheck: &link.CheckLayer{Verdict: "pass"},
Beside: map[string]string{"mesh-catalog": "0123456789ab", "mesh-lab": "ba9876543210"}}})
if c.Trunk != "d28ca95bbbbb" || c.Commit != "dfb1415aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" {
t.Fatalf("the verdict names head %s and trunk %q", c.Commit, c.Trunk)
}
if c.Beside["mesh-catalog"] != "0123456789ab" || c.Beside["mesh-lab"] != "ba9876543210" {
t.Fatalf("the verdict names the repositories read beside it as %v", c.Beside)
}
if c.Took != "4m12s" {
t.Fatalf("the verdict says it took %q", c.Took)
}
if c.Plan == nil || c.Plan.Trunk != "d28ca95bbbbb" || c.Plan.Head != plan.Head {
t.Fatalf("the kept plan records %+v", c.Plan)
}
if plan.Trunk != "" {
t.Fatal("the plan the controller asked with was changed: the verdict's is a copy")
}
group := &link.CheckRequest{Owner: "novox", Repo: "mesh-controller", Number: 7, Base: "main", Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-host", Number: 8, Ref: "cccccccccccc0000", Base: "main"}}}
c = checkedOf(link.BuildResult{ID: "g", Ref: "aaaaaaaaaaaa0000", Checked: group, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Trunk: "111111111111", Gate: &link.CheckLayer{Verdict: "pass"},
Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-host", Commit: "cccccccccccc", Trunk: "222222222222"}}}})
if len(c.Members) != 2 || c.Members[0].Trunk != "111111111111" || c.Members[1].Trunk != "222222222222" ||
c.Members[1].Commit != "cccccccccccc0000" {
t.Fatalf("the composed check names its members %+v", c.Members)
}
}
// **The merge made for a check is never registered** (novox/hq ADR 0323 §2): a build of it — a commit on no branch
// of the forge — is refused, naming the trunk. The rule of ADR 0238 §1 stands unchanged; this holds it for the
// merge commit a check makes.
func TestABuildOfACheckMergeIsRefusedNamingTheTrunk(t *testing.T) {
merge := link.BuildResult{ID: "b", Commit: "4b5c6d7e8f90aabbccddeeff0011223344556677", Trunk: "main"}
err := publishable(merge, "")
if !errors.Is(err, errOffTheTrunk) || !strings.Contains(err.Error(), "not on main") {
t.Fatalf("a build of a check's merge was answered %v", err)
}
}
// **A verdict no build judges names its trunk commit too** (novox/hq ADR 0323 §3): a change touching no module of a
// repository with no merge-check.sh (novox/mesh-tools-go) is answered by the controller itself. Naming no trunk, the
// merge tool refused it, asked again, got the same verdict, and refused for ever.
func TestADirectVerdictNamesTheTrunkTheForgeAnnounced(t *testing.T) {
p := link.PullUpdated{Owner: "novox", Repo: "mesh-tools-go", Number: 3, Base: "main", Commit: "1a2b3c4d5e6f7a8b9c0d",
BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
c := directVerdict(p, link.ChangePlan{Repository: "novox/mesh-tools-go", Base: "main", Head: p.Commit})
if c.Trunk != "7a8b9c0d1e2f" || c.Plan == nil || c.Plan.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("the direct verdict names trunk %q, its plan %+v", c.Trunk, c.Plan)
}
if c.Verdict != "pass" || c.Commit != p.Commit {
t.Fatalf("the direct verdict is %+v", c)
}
if c := directVerdict(link.PullUpdated{Owner: "novox", Repo: "x", Commit: "abc"}, link.ChangePlan{}); c.Trunk != "" {
t.Fatalf("a forge that said no base head gave the trunk %q", c.Trunk)
}
}
// A head checked again through mesh-delivery (`delivery check --members`, a recheck) reaches the same direct verdict:
// it names the base head the forge last announced, kept on the member, or the merge tool would refuse it for ever.
func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
m := orderMember{ID: "novox/mesh-tools-go@1a2b3c4d5e6f", Repository: "novox/mesh-tools-go", Base: "main",
Head: "1a2b3c4d5e6f7a8b9c0d", Number: 3, BaseSHA: "7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b"}
if c := directVerdict(m.pull(), link.ChangePlan{}); c.Trunk != "7a8b9c0d1e2f" {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
// A check stopped by its pull request's key moving on says no verdict (issue 974, review 2): an error said for it would name the
// live head, and the newer request is the one that judges. A check that failed for any other reason still says its error.
+16 -7
View File
@@ -4,7 +4,9 @@ import (
"context"
"errors"
"fmt"
"io"
"os"
"strings"
"time"
"github.com/novox/mesh-controller/internal/artifacts"
@@ -215,27 +217,34 @@ func collect(ctx context.Context, inv *inventory.Inventory) {
}
r := sweep(ctx, inv, artifacts.Store{Address: address}, references, kept, afterBuild)
sayCollected(os.Stderr, r)
}
// sayCollected says what one sweep after a build did.
func sayCollected(w io.Writer, r sweepResult) {
if r.HoldersWritten > 0 {
fmt.Fprintf(os.Stderr, "the artifact store now holds %d more kept archive(s) by a manifest\n", r.HoldersWritten)
fmt.Fprintf(w, "the artifact store now holds %d more kept archive(s) by a manifest\n", r.HoldersWritten)
}
if r.Missing > 0 {
fmt.Fprintf(os.Stderr, "%d archive(s) the mesh keeps are not in the artifact store at all; "+
fmt.Fprintf(w, "%d archive(s) the mesh keeps are not in the artifact store at all; "+
"`collection` lists them\n", r.Missing)
}
if len(r.LetGo) > 0 {
fmt.Fprintf(os.Stderr, "the artifact store let go of %d artifact(s) the mesh no longer keeps\n", len(r.LetGo))
// **Each one named** (novox/hq issue 1933): a count could not say which archive a machine's 404 was let go of.
fmt.Fprintf(w, "the artifact store let go of %d artifact(s) the mesh no longer keeps: %s\n", len(r.LetGo),
strings.Join(r.LetGo, ", "))
}
if r.Stopped != "" && !r.Bounded {
fmt.Fprintln(os.Stderr, r.Stopped)
fmt.Fprintln(w, r.Stopped)
}
if r.Held && r.Left > 0 {
fmt.Fprintf(os.Stderr, "%d more to collect; the next build asks again\n", r.Left)
fmt.Fprintf(w, "%d more to collect; the next build asks again\n", r.Left)
}
if r.Skipped > 0 {
fmt.Fprintf(os.Stderr, "%d artifact(s) the sweep will not address were skipped\n", r.Skipped)
fmt.Fprintf(w, "%d artifact(s) the sweep will not address were skipped\n", r.Skipped)
}
if r.Indexes > 0 {
fmt.Fprintf(os.Stderr, "%d eligible index(es) wait for a collect a person confirms, which takes their platforms too\n", r.Indexes)
fmt.Fprintf(w, "%d eligible index(es) wait for a collect a person confirms, which takes their platforms too\n", r.Indexes)
}
}
+6
View File
@@ -233,6 +233,12 @@ func prepareSend(ctx context.Context, open *stores, node string) ([]string, erro
if !slices.Contains(in.BusMinted, "provisioner."+node) {
continue
}
default:
// A bus process's account nobody issued still gets the stand-in, until the composition leaves
// the module off (issue 1753): the warning names the verb (issue 1226).
if line := processAccountWarning(m, node, name, in.BusMinted); line != "" {
sayWarning(ctx, line)
}
}
if _, err := inv.SecretForModule(ctx, node, m.Module, name); err == nil {
did = true
+7 -1
View File
@@ -200,6 +200,7 @@ func composeQuestion(ctx context.Context, open *stores, node string, mode compos
}
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
if !withOrder {
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
}
@@ -238,7 +239,11 @@ func markUnresolvable(err error) error {
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
if err != nil {
return declared, err
}
// Never a declaration naming an image the store does not hold (novox/hq issue 1742): that module is left out.
return withoutMissingImages(ctx, open, node, declared), nil
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
@@ -281,6 +286,7 @@ func composeSendValidated(ctx context.Context, open *stores, node string) (senda
if len(left) == 0 && len(in.wouldMake) == 0 {
sayFound(ctx, d)
out := sendableOf(d)
out.fileReferences = readsFileReferences(in.Input, node)
// The sender stamps the order it allots (order.stamp); a composition for a send carries none of its own.
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
here, _ := nodeOf(in.Input, node)
@@ -1,68 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A member of a group whose gate is the composed check's is asked of the build seat with no module: its repository's
// check alone (novox/hq ADR 0403 clause 1). With no word from mesh-delivery, or one that does not say the gate is
// composed, it is checked alone with its gate (clause 6).
func askedForMember(t *testing.T, answer []groupsLine, answerErr error) link.CheckRequest {
t.Helper()
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
was := groupsAsked
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, answerErr }
t.Cleanup(func() { groupsAsked = was })
p := link.PullUpdated{Owner: "novox", Repo: "mesh-controller", Number: 346, Base: "main", Head: "feat/x", Commit: headController,
Paths: []string{"main.go"}, ModuleDirsSaid: true, MergeCheck: true, MergeCheckSaid: true}
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil || len(q.Asks) != 1 {
t.Fatalf("asks %d: %v", len(q.Asks), err)
}
return *q.Asks[0].Request.Check
}
func TestAComposedMembersCheckIsItsRepositoryChecksAlone(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
line.ComposedGate = true
c := askedForMember(t, []groupsLine{line}, nil)
if len(c.Modules) != 0 || len(c.New) != 0 || len(c.Dependents) != 0 || !c.Composed {
t.Fatalf("the member's check is asked as %+v; want no module and composed", c)
}
}
func TestAMemberOfAGroupOfAnOlderMeshDeliveryKeepsItsOwnGate(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
c := askedForMember(t, []groupsLine{line}, nil)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed {
t.Fatalf("an older mesh-delivery's member is asked as %+v; want its own gate", c)
}
}
func TestAMemberWhoseGroupCannotBeLearnedIsCheckedAloneWithItsGate(t *testing.T) {
c := askedForMember(t, nil, context.DeadlineExceeded)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed ||
!strings.Contains(c.SiblingsUnknown, "reads no siblings") {
t.Fatalf("the member is asked as %+v; want its gate and why its siblings are unknown", c)
}
}
@@ -1,136 +0,0 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery group's gate is the composed check's, and its members' own checks run their repository's check alone
// (novox/hq ADR 0403 clauses 1 and 7). Pure: no store, no bus.
// A member of a group whose line says composed_gate is a composed member; one whose group says nothing of it (an older
// mesh-delivery), one of a group of a single pull request and a lone pull request are not.
func TestAMemberOfAComposedGroupIsKnownByTheGroupsLine(t *testing.T) {
two := func(composed bool) groupsLine {
l := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))
l.ComposedGate = composed
return l
}
p := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
recheck := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Commit: headHost}
for name, c := range map[string]struct {
p link.PullUpdated
lines []groupsLine
want bool
}{
"composed": {p, []groupsLine{two(true)}, true},
"a recheck, by its number": {recheck, []groupsLine{two(true)}, true},
"an older mesh-delivery": {p, []groupsLine{two(false)}, false},
"another group": {p, []groupsLine{group("feat/y", member("novox/mesh-host", 1, headHost, "main", true))}, false},
"no group": {p, nil, false},
} {
if got := composedMember(c.p, c.lines); got != c.want {
t.Errorf("%s: composed %v, want %v", name, got, c.want)
}
}
alone := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))
alone.ComposedGate = true
if composedMember(p, []groupsLine{alone}) {
t.Error("a group of a single pull request is composed")
}
}
// A composed member's verdict never says its gate passed: the gate layer, and the verdict that is the gate's, read
// "composed"; a failure or an error stays what it was.
func TestAComposedMembersVerdictHasNoPassingGate(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Composed: true}
pass := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "no module touched",
Gate: &link.CheckLayer{Verdict: "pass", Summary: "no module touched"}, RepoCheck: &link.CheckLayer{Verdict: "pass", Summary: "ok"}}}
c := checkedOf(pass)
if c.Gate == nil || c.Gate.Verdict != "composed" || c.Verdict != "composed" || c.RepoCheck == nil || c.RepoCheck.Verdict != "pass" {
t.Fatalf("the member's verdict says %+v gate %+v", c, c.Gate)
}
asked.Composed = false
if c := checkedOf(pass); c.Gate.Verdict != "pass" || c.Verdict != "pass" {
t.Fatalf("a member of no composed group has its own gate: %+v", c.Gate)
}
asked.Composed = true
failing := pass
failing.Check = &link.CheckOutcome{Verdict: "fail", Summary: "conflict", Gate: &link.CheckLayer{Verdict: "fail", Summary: "conflict"},
RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "conflict"}}
if c := checkedOf(failing); c.Gate.Verdict != "fail" || c.Verdict != "fail" {
t.Fatalf("a failure was read as composed: %+v", c.Gate)
}
broken := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Failed: "the facts snapshot cannot be read"}
if c := checkedOf(broken); c.Verdict != "error" || c.Gate.Verdict != "error" {
t.Fatalf("an error was read as composed: %+v", c)
}
}
// The composed verdict on the bus carries every member, each with its trunk commit, for the forge's holder to set each
// member's mesh/merge-gate from it.
func TestTheComposedVerdictCarriesEveryMember(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-controller", Number: 346, Ref: headController}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok",
Trunk: "aaaaaaaaaaaa", Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-controller", Commit: headController[:12], Trunk: "bbbbbbbbbbbb"}}}})
body, _ := json.Marshal(c)
for _, want := range []string{`"group":"feat/x"`, `"members":[{"owner":"novox","repo":"mesh-host","number":114,"commit":"` + headHost + `","trunk":"aaaaaaaaaaaa"}`,
`{"owner":"novox","repo":"mesh-controller","number":346,"commit":"` + headController + `","trunk":"bbbbbbbbbbbb"}`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the composed verdict lacks %s:\n%s", want, body)
}
}
}
// Members whose manifests order their modules in a cycle fail the composed check, naming every module and edge, and no
// ask is made of the build seat.
func TestAGroupInACycleFailsItsComposedCheckWithoutABuild(t *testing.T) {
members := []orderMember{
{ID: "novox/x@" + headHost[:12], Repository: "novox/x", Base: "main", Head: headHost, Number: 7, Paths: []string{"module.json"}},
{ID: "novox/y@" + headController[:12], Repository: "novox/y", Base: "main", Head: headController, Number: 9, Paths: []string{"module.json"}},
}
reach := map[string]orderReach{members[0].ID: {Moved: []string{"mx"}}, members[1].ID: {Moved: []string{"my"}}}
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, repo, _, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil // what the pull requests merge into holds no such manifest
}
if repo == "x" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
c, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, []inventory.Edge(nil), nil)
if err != nil || !cyclic {
t.Fatalf("no cycle found: %v", err)
}
if c.Verdict != "fail" || c.Group != "feat/cycle" || len(c.Members) != 2 || c.Gate == nil || c.Gate.Verdict != "fail" {
t.Fatalf("the verdict is %+v", c)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(c.Summary, w) {
t.Errorf("the cycle does not name %s: %s", w, c.Summary)
}
}
// Members with no edge between them are no cycle.
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
if _, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, nil, nil); err != nil || cyclic {
t.Fatalf("a cycle where there is none: %v %v", cyclic, err)
}
}
@@ -0,0 +1,110 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/overlay"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// 1696/AC2, ADR 0486 §5: a node is sent a file by digest only when its profile reports the capability. A node whose profile lacks
// it, and a node the mesh has no profile of, are sent every content inline; a node that reports it is sent the reference. The test
// goes through composeQuestion and composeSend, the two places readsFileReferences is read (compose_sdk.go).
func TestAFileTravelsByDigestOnlyToANodeThatReportsTheCapability(t *testing.T) {
open := aMesh(t) // anchor and laptop report no file-references
ctx := t.Context()
big := strings.Repeat("0123456789abcdef", 8<<10) // 128 KiB
register(t, open, catalogue.Manifest{Module: "bigfile", Version: "1",
Resources: []map[string]any{{"id": "blob", "type": "file", "path": "/etc/bigfile/blob", "mode": "0644", "content": big}}})
// reader reports the capability; bare has never reported a profile at all.
for i, n := range []struct {
name string
profile map[string]any
}{
{"reader", map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true},
map[string]any{"name": sdkdecl.FileReferencesCapability, "present": true}}}},
{"bare", nil},
} {
record, err := open.inventory.AddNode(ctx, n.name)
if err != nil {
t.Fatal(err)
}
if err := open.inventory.SetPlace(ctx, n.name, n.name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+10)); err != nil {
t.Fatal(err)
}
if n.profile != nil {
if err := open.inventory.RecordProfile(ctx, record.ID, n.profile); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, n.name, overlay.Name); err != nil {
t.Fatal(err)
}
}
for _, node := range []string{"laptop", "reader", "bare"} {
if _, err := assign(ctx, open, node, "bigfile"); err != nil {
t.Fatalf("assign %s: %v", node, err)
}
}
byDigest := map[string]bool{"laptop": false, "bare": false, "reader": true}
for node, want := range byDigest {
asked, _, err := composeQuestion(ctx, open, node, compose.Reading, false)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
sent, err := composeSend(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
for what, s := range map[string]sendable{"question": asked, "send": sent} {
if s.fileReferences != want {
t.Fatalf("%s, %s: fileReferences %v, want %v", node, what, s.fileReferences, want)
}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
var found bool
for _, r := range got.Resources {
if r["path"] != "/etc/bigfile/blob" {
continue
}
found = true
inline := r["content"] == big
if want && (inline || r["digest"] == nil || r["size"] == nil) {
t.Fatalf("%s, %s: a node that reads references is sent the content inline", node, what)
}
if !want && (!inline || r["digest"] != nil) {
t.Fatalf("%s, %s: a node that does not report the capability is sent a reference", node, what)
}
}
if !found {
t.Fatalf("%s, %s: the file is not in the declaration", node, what)
}
}
}
}
+253
View File
@@ -0,0 +1,253 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"sync"
"git.novox.be/novox/mesh-sdk/go/compose"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
)
// **A declaration's large files travel by their digest to a node that can read one** (novox/hq ADR 0486 §1, §5, issue 1696).
//
// The declaration to a node that reports sdkdecl.FileReferencesCapability carries, in place of the content of every
// file resource of more than sdkdecl.FileReferenceThreshold bytes that may travel so (File.TravelsByDigest: plain content or
// bytes, not sealed, not filled with secrets, not written into another file), its sha256 digest and size. The content is put
// into the bucket of the bus under the hex of the digest before the declaration is sent. A node that does not report the
// capability is sent every content inline, as before: an older node-engine refuses a field it does not know, whole.
// readsFileReferences says the node reports the capability of reading a file given by digest (ADR 0486 §5). A node the mesh has
// no profile of reads none.
func readsFileReferences(in compose.Input, node string) bool {
n, ok := nodeOf(in, node)
return ok && n.Capabilities[sdkdecl.FileReferencesCapability]
}
// referencing is the resources as a node that reads references is sent them, and the content of each file that went by
// digest, by the name of its object (the hex of its digest). The resources it is given are not changed. A file that does not
// travel by digest is passed as it is.
func referencing(resources []map[string]any) ([]map[string]any, map[string][]byte, error) {
out := make([]map[string]any, len(resources))
var objects map[string][]byte
for i, r := range resources {
out[i] = r
if kind, _ := r["type"].(string); kind != string(sdkdecl.TypeFile) {
continue
}
// Cheap first: only a file with a content or bytes of the size is worth reading whole.
content, _ := r["content"].(string)
bytesField, _ := r["bytes"].(string)
if len(content) <= sdkdecl.FileReferenceThreshold && len(bytesField) <= sdkdecl.FileReferenceThreshold {
continue
}
raw, err := json.Marshal(r)
if err != nil {
return nil, nil, fmt.Errorf("resource %v: %w", r["id"], err)
}
var f sdkdecl.File
if err := json.Unmarshal(raw, &f); err != nil {
return nil, nil, fmt.Errorf("resource %v is not a file the controller can read: %w", r["id"], err)
}
if !f.TravelsByDigest() {
continue
}
ref, body, err := f.AsReference()
if err != nil {
return nil, nil, err
}
name, err := sdkdecl.ObjectName(ref.Digest)
if err != nil {
return nil, nil, err
}
with := make(map[string]any, len(r)+2)
for k, v := range r {
with[k] = v
}
delete(with, "content")
delete(with, "bytes")
with["digest"], with["size"] = ref.Digest, ref.Size
out[i] = with
if objects == nil {
objects = map[string][]byte{}
}
objects[name] = body
}
return out, objects, nil
}
// namedObjects are the objects of the bucket a body names: the object name of every file resource given by reference,
// with its size. It reads a declaration as the node does, so it also reads the latest one the bus holds.
func namedObjects(body []byte) (map[string]int64, error) {
var envelope struct {
Resources []struct {
Type string `json:"type"`
Digest string `json:"digest"`
Size int64 `json:"size"`
} `json:"resources"`
}
if err := json.Unmarshal(body, &envelope); err != nil {
return nil, err
}
var out map[string]int64
for _, r := range envelope.Resources {
if r.Type != string(sdkdecl.TypeFile) || r.Digest == "" {
continue
}
name, err := sdkdecl.ObjectName(r.Digest)
if err != nil {
return nil, err
}
if out == nil {
out = map[string]int64{}
}
out[name] = r.Size
}
return out, nil
}
// objectStore is what putting the objects needs of a node's bucket.
type objectStore interface {
// Has says whether an object of that name is there, complete, and whether it carries the sweep's mark of being named by no
// node's latest declaration.
Has(ctx context.Context, name string) (have, marked bool, err error)
// Put writes the object under that name, replacing one of the same name (and so erasing its mark).
Put(ctx context.Context, name string, content []byte) error
}
// putObjects makes sure every object of the declaration is in the bucket before the declaration that names it is sent: a node
// that reads the declaration finds each object. An object already there (the same content, the same name) is not written again,
// **unless the sweep marked it unnamed**: then it is written again, so the mark that would have it removed is gone with the
// old object (issue 1696, review: a mark read just before a send must not take an object the send relies on). Refuses on the
// first it cannot put, naming it, so the declaration is not sent.
func putObjects(ctx context.Context, store objectStore, objects map[string][]byte) error {
names := make([]string, 0, len(objects))
for name := range objects {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
have, marked, err := store.Has(ctx, name)
if err != nil {
return fmt.Errorf("whether the object %s is in the bucket: %w", name[:12], err)
}
if have && !marked {
continue
}
if err := store.Put(ctx, name, objects[name]); err != nil {
return fmt.Errorf("putting the object %s (%d bytes) in the bucket: %w", name[:12], len(objects[name]), err)
}
}
return nil
}
// busObjects is a node's bucket on the controller's own connection.
type busObjects struct{ store jetstream.ObjectStore }
// openBucket opens the bucket of a node's files, making it first if it is not there (it is asserted for every node at the
// controller's start; a node adopted since is covered here).
func openBucket(ctx context.Context, conn *nats.Conn, node string) (busObjects, error) {
js, err := jetstream.New(conn)
if err != nil {
return busObjects{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(ctx, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
if err = broker.EnsureDeclarationFiles(ctx, js, node); err == nil {
store, err = js.ObjectStore(ctx, bucket)
}
}
if err != nil {
return busObjects{}, fmt.Errorf("the object bucket %s: %w", bucket, err)
}
return busObjects{store}, nil
}
func (b busObjects) Has(ctx context.Context, name string) (bool, bool, error) {
info, err := b.store.GetInfo(ctx, name)
switch {
case errors.Is(err, jetstream.ErrObjectNotFound):
return false, false, nil
case err != nil:
return false, false, err
}
_, marked := info.Metadata[declarationUnnamed]
return !info.Deleted, marked, nil
}
func (b busObjects) Put(ctx context.Context, name string, content []byte) error {
_, err := b.store.PutBytes(ctx, name, content)
return err
}
// filesLocks hold a node's bucket for one act at a time in this process: a send puts its objects and publishes the declaration that
// names them under the node's lock, and the sweep judges the node's bucket under the same one, so the sweep never removes an
// object between a send's put and its declaration. Another process (a person's push) is covered by the put erasing a mark and by
// the sweep looking at an object again just before it removes it.
var filesLocks sync.Map
func lockFilesOf(node string) func() {
l, _ := filesLocks.LoadOrStore(node, &sync.Mutex{})
mu := l.(*sync.Mutex)
mu.Lock()
return mu.Unlock
}
// kindFilesNotPut is a node whose files that travel by digest could not be put in its bucket, send after send.
const kindFilesNotPut = "declaration-files-not-put"
// filesNotPutAfter is how many sends in a row may fail to put before the condition is raised: the first send to the machine holding
// the bus fails for want of the grant that very send delivers, and says so, and is not yet a fault.
const filesNotPutAfter = 2
var (
filesFailMu sync.Mutex
filesFailed = map[string]int{}
)
func filesNotPutObservation(node, summary string) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindFilesNotPut, Token: kindFilesNotPut,
Machine: node, Severity: conditions.Warning, Source: "sends", Summary: summary}
}
// notePut records how putting a send's objects went: a failure counts, the second in a row raises the condition, a success
// clears the count and the condition.
func notePut(ctx context.Context, node string, put error) {
filesFailMu.Lock()
if put == nil {
delete(filesFailed, node)
} else {
filesFailed[node]++
}
n := filesFailed[node]
filesFailMu.Unlock()
keeper := conditionsFrom
if keeper == nil {
return
}
key := filesNotPutObservation(node, "").Key()
switch {
case put == nil:
if _, err := keeper.Clear(ctx, key, node+" was sent its files by digest"); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
case n >= filesNotPutAfter:
summary := fmt.Sprintf("%s's files that travel by digest could not be put in its bucket %d sends in a row, so it is sent them "+
"inline: %v", node, n, put)
if _, err := keeper.Observe(ctx, filesNotPutObservation(node, summary)); err != nil {
fmt.Printf("the condition %s could not be raised: %v\n", key, err)
}
}
}
@@ -0,0 +1,246 @@
//go:build integration
package main
import (
"bytes"
"context"
"encoding/json"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
type sigOf struct{}
func (sigOf) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// 1696/AC7 (ADR 0486 §1): on a real bus with its default largest message of 1 MiB, a declaration whose files add up to more than
// 3 MiB is sent as references, its objects are in the bucket, each fetched back whole and verified against the digest and size the
// declaration names; the same declaration inline is refused by the controller, naming size and limit. A changed byte in an object
// fails the verification.
func TestADeclarationOfMoreThanThreeMiBIsSentByReferenceOnARealBus(t *testing.T) { // 1696/AC7
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
nc, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
if _, err := nc.AddStream(&nats.StreamConfig{Name: "NODES", Subjects: []string{"mesh.node.*.declare"},
MaxMsgsPerSubject: 1}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
api, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatal(err)
}
// Run twice: an assertion of an existing bucket is no error and no change.
if err := broker.EnsureDeclarationFiles(ctx, api, "novox"); err != nil {
t.Fatalf("asserting the bucket again: %v", err)
}
var resources []map[string]any
for i := 0; i < 3; i++ {
resources = append(resources, aFileResource("big-"+string(rune('a'+i)), contentOf("big"+string(rune('a'+i)), 1200<<10)))
}
resources = append(resources, aFileResource("small", "tiny\n"))
inline := sendable{Resources: resources}
byDigest := sendable{fileReferences: true, Resources: resources}
bus := link.OverNATS{Conn: conn, JS: nc}
body, err := inline.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", body, 5*time.Second); err == nil ||
!strings.Contains(err.Error(), "1048576") || strings.Contains(err.Error(), "maximum payload exceeded") {
t.Fatalf("the inline declaration of more than 3 MiB: %v", err)
}
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
objects, err := byDigest.objects()
if err != nil || len(objects) != 3 {
t.Fatalf("objects: %d %v", len(objects), err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatal(err)
}
if err := putObjects(ctx, bucket, objects); err != nil {
t.Fatalf("putting them again: %v", err)
}
refBody, err := byDigest.Body()
if err != nil {
t.Fatal(err)
}
if err := link.Declare(ctx, bus, sigOf{}, "novox", refBody, 5*time.Second); err != nil {
t.Fatalf("the declaration by reference: %v", err)
}
// What the stream holds for the node is the reference form, and it names exactly the three objects.
latest, err := latestDeclarations(ctx, conn, []string{"novox", "ace"})
if err != nil || len(latest) != 1 {
t.Fatalf("the latest declarations: %v %v", len(latest), err)
}
named, err := namedByLatest("novox", latest["novox"])
if err != nil || len(named) != 3 {
t.Fatalf("what the latest declaration names: %v %v", named, err)
}
novoxBucket, err := broker.DeclarationFilesBucketOf("novox")
if err != nil {
t.Fatal(err)
}
store, err := api.ObjectStore(ctx, novoxBucket)
if err != nil {
t.Fatal(err)
}
var envelope struct {
Resources []sdkdecl.File `json:"resources"`
}
if err := json.Unmarshal(refBody, &envelope); err != nil {
t.Fatal(err)
}
for _, f := range envelope.Resources {
if !f.IsReference() {
continue
}
name, _ := sdkdecl.ObjectName(f.Digest)
got, err := store.GetBytes(ctx, name)
if err != nil {
t.Fatalf("the object of %s: %v", f.ID, err)
}
if err := f.Verify(got); err != nil {
t.Errorf("the object does not verify: %v", err)
}
got[len(got)/2] ^= 1
if err := f.Verify(got); err == nil {
t.Errorf("a changed byte in the object of %s passed the verification", f.ID)
}
}
// The sweep against the real bucket: all named, nothing marked; a fourth object named by none is marked, kept a day less a
// minute, and removed at the day.
extra := bytes.Repeat([]byte("x"), 100)
extraName := strings.Repeat("e", 64)
if err := putObjects(ctx, bucket, map[string][]byte{extraName: extra}); err != nil {
t.Fatal(err)
}
b := busSweptBucket{store}
t0 := time.Now()
if r, err := sweepDeclarationFiles(ctx, b, named, t0); err != nil || r.Marked != 1 || r.Removed != 0 || r.Kept != 3 {
t.Fatalf("first sweep: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour-time.Minute)); err != nil || r.Removed != 0 {
t.Fatalf("a minute short of a day: %+v %v", r, err)
}
if r, err := sweepDeclarationFiles(ctx, b, named, t0.Add(24*time.Hour+time.Second)); err != nil || r.Removed != 1 {
t.Fatalf("a day after: %+v %v", r, err)
}
if infos, err := store.List(ctx); err != nil || len(infos) != 3 {
t.Fatalf("the bucket holds %d objects (%v), want the 3 the declaration names", len(infos), err)
}
}
// 1696/AC8 on a real bucket: the sweep removes an object only if it still carries the mark it judged it by. An object marked
// unnamed, then put again by a send (which erases the mark), is kept when the sweep deletes it by the old mark.
func TestTheSweepsDeleteLeavesAnObjectASendPutAgainAfterItWasMarked(t *testing.T) { // 1696/AC8
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
bucket, err := openBucket(ctx, conn, "novox")
if err != nil {
t.Fatal(err)
}
name := strings.Repeat("a", 64)
content := map[string][]byte{name: bytes.Repeat([]byte("y"), 100)}
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
b := busSweptBucket{bucket.store}
judged := time.Now().Add(-25 * time.Hour).Truncate(time.Second)
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if _, marked, err := bucket.Has(ctx, name); err != nil || !marked {
t.Fatalf("the object was not marked: %v %v", marked, err)
}
// The sweep has read the mark; a send now puts the object again, which erases it.
if err := putObjects(ctx, bucket, content); err != nil {
t.Fatal(err)
}
if have, marked, err := bucket.Has(ctx, name); err != nil || !have || marked {
t.Fatalf("a send did not put the marked object again: have %v marked %v %v", have, marked, err)
}
gone, err := b.Delete(ctx, name, judged)
if err != nil || gone {
t.Fatalf("the sweep deleted by an old mark: gone %v, %v", gone, err)
}
if have, _, err := bucket.Has(ctx, name); err != nil || !have {
t.Fatalf("the object a send relies on is gone: %v %v", have, err)
}
// And the mark still on it is honoured: marked again and deleted by that mark, it goes.
if err := b.MarkUnnamed(ctx, name, judged); err != nil {
t.Fatal(err)
}
if gone, err := b.Delete(ctx, name, judged); err != nil || !gone {
t.Fatalf("an object still carrying the mark it was judged by is kept: gone %v, %v", gone, err)
}
}
// 1696/AC9 on a real bus: the sweep of a node's bucket takes the node's lock first, so it waits for a send that has put its objects
// and not yet published the declaration that names them.
func TestTheSweepOfANodesBucketHoldsTheNodesLock(t *testing.T) { // 1696/AC9
s := testbus.Start(t)
conn, err := nats.Connect(s.ClientURL())
if err != nil {
t.Fatal(err)
}
defer conn.Close()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
const node = "sweep-lock-node"
if _, err := openBucket(ctx, conn, node); err != nil { // the bucket exists, or the sweep would return before the lock
t.Fatal(err)
}
release := lockFilesOf(node) // a send is between its put and its publication
done := make(chan struct{})
go func() {
_, _ = sweepNode(ctx, conn, node)
close(done)
}()
select {
case <-done:
t.Fatal("the sweep judged the node's bucket while a send held it")
case <-time.After(300 * time.Millisecond):
}
release()
select {
case <-done:
case <-time.After(20 * time.Second):
t.Fatal("the sweep did not go on once the send let go")
}
}
@@ -0,0 +1,534 @@
package main
import (
"context"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
sdkdecl "git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// contentOf is a deterministic content of n bytes that does not compress to nothing and holds no character JSON escapes.
func contentOf(seed string, n int) string {
var b strings.Builder
for i := 0; b.Len() < n; i++ {
fmt.Fprintf(&b, "%s-%07d-%x;", seed, i, i*2654435761%1000003)
}
return b.String()[:n]
}
func aFileResource(id, content string) map[string]any {
return map[string]any{"id": id, "type": "file", "path": "/etc/" + id, "mode": "0644", "content": content}
}
// controlNodeShape is a declaration shaped like the control node's on 2026-10-21 (issue 1689), the sizes measured then: a bus users
// file of 252,678 bytes, the vault's export of 138,475 bytes (compact), the tool runtime 21 KB, the packet filter 19 KB, the launcher
// 14 KB, and about 600 small resources for the rest (about 286 KB). `more` scales the count of small ones: 1 is the shape, 1.4 forty
// percent more of them.
func controlNodeShape(more float64) []map[string]any {
res := []map[string]any{
aFileResource("nats.bus-users", contentOf("users", 252678)),
aFileResource("mesh-vault.kept", contentOf("kept", 138475)),
aFileResource("node-tools.runtime", contentOf("runtime", 21<<10)),
aFileResource("nftables.filtering", contentOf("filter", 19<<10)),
aFileResource("mesh-host.launcher", contentOf("launcher", 14<<10)),
}
for i := 0; i < int(600*more); i++ {
res = append(res, aFileResource(fmt.Sprintf("module-%03d.config", i), contentOf(fmt.Sprintf("small%d", i), 380)))
}
return res
}
func wireSizeOf(t *testing.T, s sendable) int {
t.Helper()
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
// The signature the controller's key makes is 64 bytes.
msg, err := link.Envelope(body, make([]byte, 64))
if err != nil {
t.Fatal(err)
}
return len(msg)
}
// 1696/AC1: for a node that reports the capability, the two big files of a control-node-shaped declaration carry their digest and size
// and no content, and the declaration is far under the bus's 1 MiB.
func TestAFileOver64KiBTravelsByDigestToANodeThatReadsOne(t *testing.T) { // 1696/AC1
users, kept := contentOf("users", 252678), contentOf("kept", 138475)
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("nats.bus-users", users), aFileResource("mesh-vault.kept", kept),
aFileResource("small", "tiny\n"),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, want := range []string{users, kept} {
r := got.Resources[i]
digest := sdkdecl.DigestOf([]byte(want))
if _, has := r["content"]; has || r["digest"] != digest || r["size"] != float64(len(want)) {
t.Errorf("%v goes with content %v, digest %v, size %v; want no content, %s and %d", r["id"], r["content"] != nil,
r["digest"], r["size"], digest, len(want))
}
if r["path"] == nil || r["mode"] != "0644" {
t.Errorf("%v lost its path or mode: %v", r["id"], r)
}
}
if got.Resources[2]["content"] != "tiny\n" || got.Resources[2]["digest"] != nil {
t.Errorf("a small file is not inline: %v", got.Resources[2])
}
if strings.Contains(string(body), users[:200]) || strings.Contains(string(body), kept[:200]) {
t.Error("the content of a file that went by digest is in the body")
}
if n := wireSizeOf(t, s); n > 100<<10 {
t.Errorf("the declaration is %d bytes on the wire, want under 100 KB", n)
}
// The objects to put are the contents, each under the hex of its digest, and a reference verifies against its object.
objects, err := s.objects()
if err != nil || len(objects) != 2 {
t.Fatalf("objects: %v %v", objects, err)
}
for _, want := range []string{users, kept} {
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(want)))
if string(objects[name]) != want {
t.Errorf("the object %s does not hold the content", name[:12])
}
}
named, err := namedObjects(body)
if err != nil || len(named) != 2 {
t.Fatalf("the objects the body names: %v %v", named, err)
}
for name := range objects {
if _, ok := named[name]; !ok {
t.Errorf("the body does not name the object %s it was sent with", name[:12])
}
if _, err := hex.DecodeString(name); err != nil || len(name) != 64 {
t.Errorf("the object name %q is not 64 hex characters", name)
}
}
}
// 1696/AC2: a node that does not report the capability is sent every content inline, as before, and has no object put.
func TestANodeWithoutTheCapabilityIsSentEveryContentInline(t *testing.T) { // 1696/AC2
users := contentOf("users", 252678)
s := sendable{Resources: []map[string]any{aFileResource("nats.bus-users", users)}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
if got.Resources[0]["content"] != users || got.Resources[0]["digest"] != nil || got.Resources[0]["size"] != nil {
t.Fatalf("a node that reads no references is not sent the content inline: %v", got.Resources[0]["digest"])
}
if objects, err := s.objects(); err != nil || len(objects) != 0 {
t.Fatalf("objects for a node that reads none: %v %v", len(objects), err)
}
}
// 1696/AC3: only a file that may travel by digest does: one of exactly 64 KiB, a sealed one, one filled with secrets, one written
// into another file, and a directory stay inline, whatever the node reads.
func TestOnlyAFileThatMayTravelByDigestDoes(t *testing.T) { // 1696/AC3
exact := contentOf("exact", 64<<10)
big := contentOf("big", 64<<10+1)
sealed := aFileResource("sealed", big)
sealed["sealed"], sealed["content"] = "box", ""
sealed["content"] = big // a sealed file's content is none; the body must pass it as it is, whatever it holds
secret := aFileResource("secret", big)
secret["secrets"] = map[string]any{"token": "x"}
into := aFileResource("into", big)
into["into"] = "/etc/other"
s := sendable{fileReferences: true, Resources: []map[string]any{
aFileResource("exact", exact), sealed, secret, into,
{"id": "dir", "type": "directory", "path": "/etc/d", "content": big},
aFileResource("big", big),
}}
body, err := s.Body()
if err != nil {
t.Fatal(err)
}
var got struct {
Resources []map[string]any `json:"resources"`
}
if err := json.Unmarshal(body, &got); err != nil {
t.Fatal(err)
}
for i, id := range []string{"exact", "sealed", "secret", "into", "dir"} {
if got.Resources[i]["id"] != id || got.Resources[i]["digest"] != nil || got.Resources[i]["content"] == nil {
t.Errorf("%s went by digest or lost its content: %v", id, got.Resources[i]["digest"])
}
}
if big := got.Resources[5]; big["id"] != "big" || big["digest"] == nil || big["content"] != nil {
t.Errorf("a plain file of 64 KiB and a byte did not go by digest: %v", big["digest"])
}
}
// A file given as bytes goes by digest on its decoded length, with the decoded bytes in the object.
func TestAFileOfBytesTravelsByDigestOnItsDecodedLength(t *testing.T) {
raw := []byte(contentOf("png", 70<<10))
r := map[string]any{"id": "icon", "type": "file", "path": "/usr/share/icon.png", "bytes": stdBase64(raw)}
s := sendable{fileReferences: true, Resources: []map[string]any{r}}
objects, err := s.objects()
if err != nil || len(objects) != 1 {
t.Fatalf("%v %v", len(objects), err)
}
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf(raw))
if string(objects[name]) != string(raw) {
t.Fatal("the object is not the decoded bytes")
}
if r["bytes"] == nil {
t.Fatal("the composed resource was changed in place")
}
}
type memObjects struct {
have map[string][]byte
marked map[string]bool
puts []string
fail map[string]error
}
func (m *memObjects) Has(_ context.Context, name string) (bool, bool, error) {
_, ok := m.have[name]
return ok, m.marked[name], nil
}
func (m *memObjects) Put(_ context.Context, name string, c []byte) error {
if err := m.fail[name]; err != nil {
return err
}
m.puts = append(m.puts, name)
m.have[name] = c
delete(m.marked, name)
return nil
}
// 1696/AC4: an object already in the bucket is not put again, one absent is, and a put that fails stops the send, naming the object.
func TestObjectsAreInTheBucketBeforeTheDeclarationIsSent(t *testing.T) { // 1696/AC4
objects := map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")}
b := &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one")}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("b", 64) {
t.Fatalf("put %v (%v), want the one absent", b.puts, err)
}
// An object the sweep marked unnamed is put again, which erases the mark: a send relies on it.
b = &memObjects{have: map[string][]byte{strings.Repeat("a", 64): []byte("one"), strings.Repeat("b", 64): []byte("two")},
marked: map[string]bool{strings.Repeat("a", 64): true}}
if err := putObjects(t.Context(), b, objects); err != nil || len(b.puts) != 1 || b.puts[0] != strings.Repeat("a", 64) || b.marked[strings.Repeat("a", 64)] {
t.Fatalf("a marked object: put %v (%v) marked %v, want it put again and the mark gone", b.puts, err, b.marked)
}
b = &memObjects{have: map[string][]byte{}, fail: map[string]error{strings.Repeat("a", 64): errors.New("permissions violation")}}
err := putObjects(t.Context(), b, objects)
if err == nil || !strings.Contains(err.Error(), "aaaaaaaaaaaa") || !strings.Contains(err.Error(), "permissions violation") {
t.Fatalf("a failed put: %v", err)
}
}
// fallbackDelivery refuses the first declaration it is sent with the bucket's failure, as a bus that has not yet been given the
// controller's grant does, and takes the second.
type fallbackDelivery struct {
bodies [][]byte
first error
}
func (f *fallbackDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *fallbackDelivery) declare(_ context.Context, _ readyNode, body []byte) (string, error) {
f.bodies = append(f.bodies, body)
if len(f.bodies) == 1 && f.first != nil {
return "", fmt.Errorf("x: %w: %w", errFilesNotPut, f.first)
}
return "digest", nil
}
// 1696/AC5: when the bucket cannot be written yet (the grant travels in the declaration of the machine holding the bus), that send goes
// with every content inline and is made; a send that fails for another reason is not retried inline.
func TestASendWhoseFilesCannotBePutGoesInlineOnce(t *testing.T) { // 1696/AC5
users := contentOf("users", 100<<10)
s := readyNode{node: "novox", declared: sendable{fileReferences: true, Resources: []map[string]any{aFileResource("nats.bus-users", users)}}}
f := &fallbackDelivery{first: errors.New("timeout")}
digests, err := deliver(t.Context(), f, "novox", []readyNode{s})
if err != nil || digests["novox"] != "digest" || len(f.bodies) != 2 {
t.Fatalf("%v %v %d", digests, err, len(f.bodies))
}
if strings.Contains(string(f.bodies[0]), users[:100]) || !strings.Contains(string(f.bodies[1]), users[:100]) {
t.Error("the first body should name the file by digest and the second carry it inline")
}
other := &fallbackDelivery{}
other.first = nil
failing := &failingDelivery{err: errors.New("the bus is down")}
if _, err := deliver(t.Context(), failing, "novox", []readyNode{s}); err == nil || failing.calls != 1 {
t.Fatalf("a failure of another kind: %v after %d send(s)", err, failing.calls)
}
}
type failingDelivery struct {
err error
calls int
}
func (f *failingDelivery) grant(context.Context, []readyNode) error { return nil }
func (f *failingDelivery) declare(context.Context, readyNode, []byte) (string, error) {
f.calls++
return "", f.err
}
// A bucket for the sweep.
type sweepBucketFake struct {
objects map[string]time.Time // zero: named at the last look
deleted []string
}
func (b *sweepBucketFake) Objects(context.Context) ([]bucketObject, error) {
var out []bucketObject
for n, at := range b.objects {
out = append(out, bucketObject{Name: n, UnnamedSince: at})
}
return out, nil
}
func (b *sweepBucketFake) MarkUnnamed(_ context.Context, n string, at time.Time) error {
b.objects[n] = at
return nil
}
func (b *sweepBucketFake) ClearUnnamed(_ context.Context, n string) error {
b.objects[n] = time.Time{}
return nil
}
func (b *sweepBucketFake) Delete(_ context.Context, n string, markedAt time.Time) (bool, error) {
if at, ok := b.objects[n]; !ok || !at.Equal(markedAt) {
return false, nil // put again since it was judged: its mark is gone
}
delete(b.objects, n)
b.deleted = append(b.deleted, n)
return true, nil
}
// 1696/AC6: an object a latest declaration still names is not removed, however long; one unnamed is marked at the first look and removed
// only when the mark is 24 hours old; one named again loses its mark.
func TestAnObjectIsRemovedADayAfterNoLatestDeclarationNamesIt(t *testing.T) { // 1696/AC6
named, unnamed, back := strings.Repeat("1", 64), strings.Repeat("2", 64), strings.Repeat("3", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{named: t0.Add(-100 * time.Hour), unnamed: {}, back: t0.Add(-5 * time.Hour)}}
named3 := map[string]bool{named: true, back: true}
r, err := sweepDeclarationFiles(t.Context(), b, named3, t0)
if err != nil || r.Marked != 1 || r.Cleared != 2 || r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("first look: %+v %v %v", r, err, b.deleted)
}
if !b.objects[named].IsZero() || !b.objects[back].IsZero() || !b.objects[unnamed].Equal(t0) {
t.Fatalf("marks after the first look: %v", b.objects)
}
// Still unnamed 23 hours 59 minutes later: kept. A day after the mark: removed. The named ones stay for ever.
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour-time.Minute)); err != nil ||
r.Removed != 0 || len(b.deleted) != 0 {
t.Fatalf("a minute short of a day: %+v %v %v", r, err, b.deleted)
}
if r, err = sweepDeclarationFiles(t.Context(), b, map[string]bool{named: true, back: true}, t0.Add(24*time.Hour)); err != nil ||
r.Removed != 1 || len(b.deleted) != 1 || b.deleted[0] != unnamed {
t.Fatalf("a day after: %+v %v %v", r, err, b.deleted)
}
if _, ok := b.objects[named]; !ok {
t.Fatal("an object a latest declaration names was removed")
}
// An object named again before the day is up loses its mark and is not removed.
b = &sweepBucketFake{objects: map[string]time.Time{unnamed: t0}}
if _, err := sweepDeclarationFiles(t.Context(), b, map[string]bool{unnamed: true}, t0.Add(23*time.Hour)); err != nil || !b.objects[unnamed].IsZero() {
t.Fatalf("named again: %v %v", b.objects, err)
}
}
// What a node's latest declaration names is read from the signed envelope as stored on the bus; an unreadable one is an error and
// never "names none", which would mark an object a node needs.
func TestTheObjectsALatestDeclarationNamesAreReadFromTheBus(t *testing.T) {
s := sendable{fileReferences: true, Resources: []map[string]any{aFileResource("a", contentOf("a", 70<<10))}}
body, _ := s.Body()
msg, _ := link.Envelope(body, []byte("sig"))
named, err := namedByLatest("novox", msg)
name, _ := sdkdecl.ObjectName(sdkdecl.DigestOf([]byte(contentOf("a", 70<<10))))
if err != nil || len(named) != 1 || !named[name] {
t.Fatalf("%v %v", named, err)
}
if none, err := namedByLatest("ace", []byte(`{"declaration":"e30=","signature":""}`)); err != nil || len(none) != 0 {
t.Fatalf("a declaration naming none: %v %v", none, err)
}
if _, err := namedByLatest("novox", []byte("not json")); err == nil {
t.Fatal("an unreadable latest declaration was read as naming none")
}
}
// 1698/AC2 (ADR 0486 §9): a declaration shaped like the control node's, for a node that reads references, stays under half of the
// bus's largest message on the wire; the inline form of the same does not fit at all; and the same shape with 40 percent more resources
// fails the test's own limit, so a change that eats the room fails in a pull request and not on the live mesh.
func TestTheControlNodesDeclarationInTheReferenceFormStaysUnderHalfTheBusLimit(t *testing.T) { // 1698/AC2
const limit = 1048576
shape := sendable{fileReferences: true, Resources: controlNodeShape(1)}
n := wireSizeOf(t, shape)
t.Logf("the control node's shape in the reference form: %d bytes on the wire, %.1f%% of %d", n, 100*float64(n)/limit, limit)
if n > limit/2 {
t.Errorf("the reference form is %d bytes on the wire, over half (%d) of the bus's largest message", n, limit/2)
}
if inline := wireSizeOf(t, sendable{Resources: controlNodeShape(1)}); inline < limit*3/4 {
t.Errorf("the inline form is %d bytes, which is not the shape that was over the limit", inline)
}
grown := wireSizeOf(t, sendable{fileReferences: true, Resources: controlNodeShape(1.4)})
if grown <= limit/2 {
t.Errorf("the shape with 40%% more resources is %d bytes, which the test would let pass", grown)
}
}
func stdBase64(b []byte) string { return base64.StdEncoding.EncodeToString(b) }
// A sweep that read an object's mark before a send put it again must not remove it: the object, now without the mark, is left.
func TestASweepDoesNotRemoveAnObjectASendPutAgainAfterItWasJudged(t *testing.T) { // 1696/AC8
n := strings.Repeat("4", 64)
t0 := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC)
b := &sweepBucketFake{objects: map[string]time.Time{n: t0}}
judged := bucketObject{Name: n, UnnamedSince: t0}
b.objects[n] = time.Time{} // the send put it again: no mark
if gone, err := b.Delete(t.Context(), judged.Name, judged.UnnamedSince); err != nil || gone || len(b.deleted) != 0 {
t.Fatalf("an object put again was removed: %v %v %v", gone, err, b.deleted)
}
}
// One node's bucket is held by one act at a time in this process: the sweep waits for a send that has put its objects and not yet
// published the declaration that names them (1696/AC9).
func TestTheSweepOfANodesBucketWaitsForASendThatHasPutItsObjects(t *testing.T) { // 1696/AC9
unlock := lockFilesOf("lock-test-node")
got := make(chan struct{})
go func() {
defer lockFilesOf("lock-test-node")()
close(got)
}()
select {
case <-got:
t.Fatal("the sweep took the node's bucket while a send held it")
case <-time.After(50 * time.Millisecond):
}
other := lockFilesOf("lock-test-other-node") // another node's bucket is not held
other()
unlock()
select {
case <-got:
case <-time.After(2 * time.Second):
t.Fatal("the sweep did not take the node's bucket once the send was done")
}
}
// A put that fails send after send raises a condition; the first failure does not (the grant it waits for travels in that very
// declaration), and a put that works clears it (1696/AC10).
func TestAPutThatFailsSendAfterSendRaisesAConditionAndClearsWhenItWorks(t *testing.T) { // 1696/AC10
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return time.Date(2026, 10, 21, 14, 0, 0, 0, time.UTC) }})
t.Cleanup(func() { k.Close(t.Context()) })
was := conditionsFrom
conditionsFrom = k
t.Cleanup(func() { conditionsFrom = was })
key := filesNotPutObservation("cond-node", "").Key()
open := func() bool {
_, found, err := conditions.ReadOne(t.Context(), store, key)
if err != nil {
t.Fatal(err)
}
return found
}
boom := errors.New("permissions violation")
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the first failure raised the condition")
}
notePut(t.Context(), "cond-node", boom)
if !open() {
t.Fatal("the second failure in a row did not raise the condition")
}
notePut(t.Context(), "cond-node", nil)
if open() {
t.Fatal("a put that worked did not clear the condition")
}
notePut(t.Context(), "cond-node", boom)
if open() {
t.Fatal("the count did not start again after a put that worked")
}
}
// aBusThatSays is a bus that keeps the declarations published to it and, at each, asks whether the node's bucket is held.
type aBusThatSays struct {
link.Bus
published chan string
heldAtPublish func() bool
}
func (b *aBusThatSays) PublishDeclaration(_ context.Context, node string, _ []byte) error {
if b.heldAtPublish != nil && !b.heldAtPublish() {
return errors.New("published while the node's bucket was not held")
}
b.published <- node
return errors.New("stop here: nothing is recorded")
}
type aSignerOfZeros struct{}
func (aSignerOfZeros) Sign(context.Context, []byte) ([]byte, error) { return make([]byte, 64), nil }
// whileHeld says whether the node's lock is held by someone right now, without keeping it.
func whileHeld(node string) bool {
got := make(chan struct{})
go func() {
defer lockFilesOf(node)()
close(got)
}()
select {
case <-got:
return false
case <-time.After(100 * time.Millisecond):
// held: the goroutine stays waiting and takes and releases the lock once it is let go.
return true
}
}
// 1696/AC9: a send holds the node's lock from the put to the publication, so the sweep never removes an object between them. The
// test goes through overTheBus.declare: while another holds the lock the declaration is not published, and at the publication the
// lock is held.
func TestASendHoldsTheNodesLockFromThePutToThePublication(t *testing.T) { // 1696/AC9
const node = "lock-send-node"
bus := &aBusThatSays{published: make(chan string, 1)}
bus.heldAtPublish = func() bool { return whileHeld(node) }
send := overTheBus{server: link.SendingOn(bus), signer: aSignerOfZeros{}}
release := lockFilesOf(node) // a sweep is judging this node's bucket
done := make(chan error, 1)
go func() {
_, err := send.declare(t.Context(), readyNode{node: node}, []byte(`{}`))
done <- err
}()
select {
case n := <-bus.published:
t.Fatalf("%s was sent its declaration while the sweep held the node's bucket", n)
case <-time.After(150 * time.Millisecond):
}
release()
select {
case <-bus.published:
case <-time.After(5 * time.Second):
t.Fatal("the send did not go on once the sweep let go")
}
if err := <-done; err == nil || !strings.Contains(err.Error(), "nothing is recorded") {
t.Fatalf("the send did not reach the publication (a lock held at it): %v", err)
}
}
+2 -2
View File
@@ -13,8 +13,8 @@ import (
// **A declaration is one bus message, and the bus's largest message is a cliff** (novox/hq issue 1689). On 2026-10-21
// the control node's declaration passed it by 2 KB and nothing reached that node until two assignments were undone.
// So the size is measured at every send: a warning condition from 90% of the limit, urgent over it, cleared when
// the declaration is back under 90%. The limit is the connection's own, not a constant.
// So the size is measured at every send: a warning condition from 75% of the limit (ADR 0486 §8, issue 1698: it was 90% while every file travelled inline), urgent over it, cleared when
// the declaration is back under 75%. The limit is the connection's own, not a constant.
// kindDeclarationNearLimit is a declaration at or near the bus's largest message.
const kindDeclarationNearLimit = "declaration-near-limit"
+8 -8
View File
@@ -33,7 +33,7 @@ func TestTheVaultsExportIsSentCompactAndNothingElseIsTouched(t *testing.T) {
}
}
// From 90% of the bus's largest message the declaration is a warning; over it, urgent; back under 90%, cleared.
// From 75% of the bus's largest message (ADR 0486 §8, issue 1698) the declaration is a warning; over it, urgent; back under 75%, cleared.
func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *testing.T) {
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
@@ -50,22 +50,22 @@ func TestADeclarationNearTheBusLimitRaisesAConditionAndClearsWhenSmaller(t *test
return c, found
}
settleDeclarationRoom(t.Context(), k, "novox", 899, limit)
settleDeclarationRoom(t.Context(), k, "novox", 749, limit)
if _, found := read(); found {
t.Fatal("a declaration at 89.9% raised a condition")
t.Fatal("a declaration at 74.9% raised a condition")
}
settleDeclarationRoom(t.Context(), k, "novox", 900, limit)
settleDeclarationRoom(t.Context(), k, "novox", 750, limit)
c, found := read()
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "900 bytes") ||
if !found || c.Severity != conditions.Warning || !strings.Contains(c.Summary, "750 bytes") ||
!strings.Contains(c.Summary, "1000") || c.Subject.Machine != "novox" {
t.Fatalf("at 90%% the condition is %v %+v", found, c)
t.Fatalf("at 75%% the condition is %v %+v", found, c)
}
settleDeclarationRoom(t.Context(), k, "novox", 1001, limit)
if c, _ := read(); c.Severity != conditions.Urgent {
t.Fatalf("over the limit the condition is %q, expected urgent", c.Severity)
}
settleDeclarationRoom(t.Context(), k, "novox", 850, limit)
settleDeclarationRoom(t.Context(), k, "novox", 700, limit)
if _, found := read(); found {
t.Fatal("the condition stayed after the declaration fell under 90%")
t.Fatal("the condition stayed after the declaration fell under 75%")
}
}
+273
View File
@@ -0,0 +1,273 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **An object is removed a day after no node's latest declaration names it** (novox/hq ADR 0486 §3, issue 1696), so a node that is
// asleep and receives its latest declaration later still finds every object that declaration names. What each node's latest
// declaration names is read from the bus itself, the last message of that node's subject of the stream NODES: nothing else
// is kept of it. When an object is first found named by none, the time is written in its own metadata (declarationUnnamed);
// found named again, that is erased; found unnamed for a day, it is removed. Each node has a bucket of its own, judged against that
// node's latest declaration alone. Objects are immutable and regenerable from the
// controller's state, so a removal loses nothing.
// declarationUnnamed is the key of an object's metadata holding when it was first found named by no node's latest declaration.
const declarationUnnamed = "unnamed-since"
// sweepEvery is how often the bucket is judged: the removal is exact to within this.
const sweepEvery = time.Hour
// bucketObject is one object of the bucket as the sweep sees it.
type bucketObject struct {
Name string
// UnnamedSince is when it was first found named by none, zero when it was named at the last look.
UnnamedSince time.Time
}
// sweptBucket is what the sweep needs of the bucket.
type sweptBucket interface {
Objects(ctx context.Context) ([]bucketObject, error)
MarkUnnamed(ctx context.Context, name string, since time.Time) error
ClearUnnamed(ctx context.Context, name string) error
// Delete removes the object if it still carries the mark it was judged by, and says whether it did: a send that put it again
// since has erased the mark, and the object is left.
Delete(ctx context.Context, name string, markedAt time.Time) (bool, error)
}
// filesSweep says what a sweep did.
type filesSweep struct{ Kept, Marked, Cleared, Removed int }
// sweepDeclarationFiles judges every object against the names of the latest declarations: named ones are kept (and a
// mark of being unnamed is erased), unnamed ones are marked at the first look and removed once the mark is a day old.
func sweepDeclarationFiles(ctx context.Context, bucket sweptBucket, named map[string]bool, now time.Time) (filesSweep, error) {
var r filesSweep
objects, err := bucket.Objects(ctx)
if err != nil {
return r, err
}
sort.Slice(objects, func(i, j int) bool { return objects[i].Name < objects[j].Name })
for _, o := range objects {
switch {
case named[o.Name]:
r.Kept++
if !o.UnnamedSince.IsZero() {
if err := bucket.ClearUnnamed(ctx, o.Name); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Cleared++
}
case o.UnnamedSince.IsZero():
if err := bucket.MarkUnnamed(ctx, o.Name, now); err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
r.Marked++
case now.Sub(o.UnnamedSince) >= broker.DeclarationFilesKeptFor:
gone, err := bucket.Delete(ctx, o.Name, o.UnnamedSince)
if err != nil {
return r, fmt.Errorf("the object %s: %w", o.Name[:12], err)
}
if gone {
r.Removed++
} else {
r.Kept++
}
default:
r.Kept++
}
}
return r, nil
}
// namedByLatest is the name of every object this latest declaration of a node names. A body that cannot be read is an error, never
// "names none": an object the node still needs would be marked for removal.
func namedByLatest(node string, signed []byte) (map[string]bool, error) {
var s link.Signed
if err := json.Unmarshal(signed, &s); err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
names, err := namedObjects(s.Declaration)
if err != nil {
return nil, fmt.Errorf("%s's latest declaration is not readable: %w", node, err)
}
named := make(map[string]bool, len(names))
for name := range names {
named[name] = true
}
return named, nil
}
// latestDeclarations are the last signed declarations of the nodes, by node, as the stream NODES holds them. A node with none has
// no entry.
func latestDeclarations(ctx context.Context, conn *nats.Conn, nodes []string) (map[string][]byte, error) {
js, err := conn.JetStream(nats.Context(ctx))
if err != nil {
return nil, err
}
out := map[string][]byte{}
for _, node := range nodes {
msg, err := js.GetLastMsg("NODES", link.DeclareSubject(node), nats.Context(ctx))
if errors.Is(err, nats.ErrMsgNotFound) {
continue
}
if err != nil {
return nil, fmt.Errorf("the latest declaration of %s: %w", node, err)
}
out[node] = msg.Data
}
return out, nil
}
// busSweptBucket is the bucket on the controller's connection.
type busSweptBucket struct{ store jetstream.ObjectStore }
func (b busSweptBucket) Objects(ctx context.Context) ([]bucketObject, error) {
infos, err := b.store.List(ctx)
if errors.Is(err, jetstream.ErrNoObjectsFound) {
return nil, nil
}
if err != nil {
return nil, err
}
out := make([]bucketObject, 0, len(infos))
for _, info := range infos {
o := bucketObject{Name: info.Name}
if at, err := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); err == nil {
o.UnnamedSince = at
}
out = append(out, o)
}
return out, nil
}
func (b busSweptBucket) meta(ctx context.Context, name string, change func(map[string]string)) error {
info, err := b.store.GetInfo(ctx, name)
if err != nil {
return err
}
meta := info.ObjectMeta
kept := map[string]string{}
for k, v := range meta.Metadata {
kept[k] = v
}
change(kept)
meta.Metadata = kept
return b.store.UpdateMeta(ctx, name, meta)
}
func (b busSweptBucket) MarkUnnamed(ctx context.Context, name string, since time.Time) error {
return b.meta(ctx, name, func(m map[string]string) { m[declarationUnnamed] = since.UTC().Format(time.RFC3339) })
}
func (b busSweptBucket) ClearUnnamed(ctx context.Context, name string) error {
return b.meta(ctx, name, func(m map[string]string) { delete(m, declarationUnnamed) })
}
func (b busSweptBucket) Delete(ctx context.Context, name string, markedAt time.Time) (bool, error) {
info, err := b.store.GetInfo(ctx, name)
if errors.Is(err, jetstream.ErrObjectNotFound) {
return false, nil
}
if err != nil {
return false, err
}
if at, perr := time.Parse(time.RFC3339, info.Metadata[declarationUnnamed]); perr != nil || !at.Equal(markedAt) {
return false, nil
}
return true, b.store.Delete(ctx, name)
}
// sweepingDeclarationFiles judges the bucket every sweepEvery for as long as this controller serves and holds the lease.
func sweepingDeclarationFiles(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) {
tick := time.NewTicker(sweepEvery)
defer tick.Stop()
failing := ""
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
if epoch, err := theLease.epoch(ctx); err != nil || epoch == 0 {
continue
}
why := sweepOnce(ctx, inv, conn)
if why != failing {
if why != "" {
fmt.Printf("the bucket of declaration files cannot be swept: %s\n", why)
} else {
fmt.Printf("the bucket of declaration files is swept again\n")
}
failing = why
}
}
}
// sweepOnce judges the bucket of every node against that node's latest declaration, one bucket at a time under the node's lock, with
// the declaration read again under it: a send that put an object and published its declaration is seen, and one that has not yet
// is waiting for the lock.
func sweepOnce(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn) string {
nodes, err := inv.Nodes(ctx)
if err != nil {
return fmt.Sprintf("the nodes cannot be read: %v", err)
}
var problems []string
var total filesSweep
for _, n := range nodes {
r, err := sweepNode(ctx, conn, n.Name)
total.Kept, total.Marked, total.Cleared, total.Removed = total.Kept+r.Kept, total.Marked+r.Marked, total.Cleared+r.Cleared, total.Removed+r.Removed
if err != nil {
problems = append(problems, fmt.Sprintf("%s: %v", n.Name, err))
}
}
if total.Removed+total.Marked+total.Cleared > 0 {
fmt.Printf("the buckets of declaration files: %d object(s) kept, %d found unnamed, %d named again, %d removed\n",
total.Kept, total.Marked, total.Cleared, total.Removed)
}
return strings.Join(problems, "; ")
}
func sweepNode(ctx context.Context, conn *nats.Conn, node string) (filesSweep, error) {
working, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
js, err := jetstream.New(conn)
if err != nil {
return filesSweep{}, err
}
bucket, err := broker.DeclarationFilesBucketOf(node)
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket of node %q: %w", node, err)
}
store, err := js.ObjectStore(working, bucket)
if errors.Is(err, jetstream.ErrBucketNotFound) {
return filesSweep{}, nil // a node never sent a reference has no bucket yet
}
if err != nil {
return filesSweep{}, fmt.Errorf("the bucket cannot be read: %w", err)
}
defer lockFilesOf(node)()
latest, err := latestDeclarations(working, conn, []string{node})
if err != nil {
return filesSweep{}, err
}
named := map[string]bool{}
if body, ok := latest[node]; ok {
if named, err = namedByLatest(node, body); err != nil {
return filesSweep{}, err
}
}
return sweepDeclarationFiles(working, busSweptBucket{store}, named, time.Now())
}
@@ -0,0 +1,89 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// 1840/AC5 (ADR 0507 §5): the source of the declared-packages mirror composes each node and names, for a package two modules
// declare and one declared absent, the modules that declare it; the same records give the same bytes on a second pass, so
// the mirror writes nothing; a node with no package has a key that says so.
func TestTheDeclaredPackagesMirrorNamesWhatEachNodesModulesDeclare(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "viewer", Version: "1", Resources: []map[string]any{
{"id": "pkg-curl", "type": "package", "package": "curl"},
{"id": "pkg-nano", "type": "package", "package": "nano", "absent": true}}})
register(t, open, catalogue.Manifest{Module: "fetcher", Version: "1", Resources: []map[string]any{
{"id": "pkg-curl", "type": "package", "package": "curl"}}})
for _, module := range []string{"viewer", "fetcher"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatalf("assign anchor %s: %v", module, err)
}
}
source := declaredPackagesMirror(open)
first, err := source(ctx)
if err != nil {
t.Fatal(err)
}
var got inventory.DeclaredPackagesMirrored
if err := json.Unmarshal(first["anchor"], &got); err != nil {
t.Fatalf("anchor has no value: %v", err)
}
// Every node also runs the mesh's own modules, which declare packages of their own: curl and nano are looked
// up by name, and no package names viewer or fetcher but those they declare.
byName := map[string]inventory.PackageDeclared{}
for _, p := range got.Packages {
byName[p.Package] = p
}
same := func(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
for _, p := range []inventory.PackageDeclared{{Package: "curl", Modules: []string{"fetcher", "viewer"}},
{Package: "nano", Modules: []string{"viewer"}, Absent: true}} {
if g, ok := byName[p.Package]; !ok || g.Absent != p.Absent || !same(g.Modules, p.Modules) {
t.Errorf("anchor declares %s as %+v, want %+v (all: %+v)", p.Package, g, p, got.Packages)
}
}
for _, p := range got.Packages {
if p.Package == "curl" || p.Package == "nano" {
continue
}
for _, m := range p.Modules {
if m == "viewer" || m == "fetcher" {
t.Errorf("%s is said to be declared by %s, which does not declare it", p.Package, m)
}
}
}
var laptop inventory.DeclaredPackagesMirrored
if err := json.Unmarshal(first["laptop"], &laptop); err != nil || laptop.Node != "laptop" || laptop.Packages == nil {
t.Errorf("a node assigned neither module has no value of its own: %s (%v)", first["laptop"], err)
}
for _, p := range laptop.Packages {
if p.Package == "curl" || p.Package == "nano" {
t.Errorf("laptop is assigned neither module and declares %+v", p)
}
}
second, err := source(ctx)
if err != nil {
t.Fatal(err)
}
for node, value := range first {
if string(second[node]) != string(value) {
t.Errorf("the value of %s changed between two passes over the same records:\n%s\n%s", node, value, second[node])
}
}
}
+9 -200
View File
@@ -815,20 +815,21 @@ func deliveryPlanDocument(m orderMember, entries []inventory.Entry, read map[str
// delivery walks [-n 50] [--plan <id>]
func deliveryCommand(ctx context.Context, args []string) error {
if len(args) == 0 {
return errors.New("delivery plan|order|check|go|stop|walks")
return errors.New("delivery order|go|stop|walks")
}
sub, rest := args[0], args[1:]
if sub == "plan" {
// Removed (ADR 0450 §8; issue 1318): refused by name before any flag is read, naming its replacement.
return removedByName("delivery-plan")
}
if sub == "check" {
// Removed (ADR 0503 §3, ADR 0527; issue 1318): the checker judges every head, so the controller asks nothing.
return removedByName("delivery-check")
}
set := flag.NewFlagSet("delivery "+sub, flag.ContinueOnError)
repository := set.String("repository", "", "owner/repository")
base := set.String("base", "main", "the branch it merges into")
head := set.String("head", "", "the commit at hand")
paths := set.String("paths", "", "the files it changes, comma-separated")
moduleDirs := set.String("module-dirs", "", "the directories holding a module.json at the head, comma-separated")
removed := set.String("removed", "", "the files it deletes, comma-separated")
membersJSON := set.String("members", "", "a group's members, as JSON")
membersFile := set.String("members-file", "", "a file of a group's members, as JSON: the same as --members, for a list "+
"too long for an argument (novox/hq issue 1272)")
group := set.String("group", "", "a delivery group's id")
ready := set.Bool("ready", false, "the head's pull request is marked ready: its check is queued before every other check "+
"(novox/hq ADR 0406 §3)")
by := set.String("by", catalogue.DeliverySeat, "who says it")
@@ -872,19 +873,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
return enc.Encode(v)
}
switch sub {
case "plan":
if *repository == "" || *head == "" && *paths == "" {
return errors.New("delivery plan --repository owner/repo --head <commit> --paths a,b")
}
m := changeMember(*repository, *base, *head, splitList(*paths), splitList(*removed), moduleDirsOf(*moduleDirs))
entries, read, edges, err := theGraph(ctx, inv)
if err != nil {
return err
}
return answer(deliveryPlanDocument(m, entries, read, edges, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
}))
case "order":
if len(members) == 0 {
return errors.New("delivery order --members <json>: a group has members")
@@ -898,27 +886,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
return fmt.Errorf("what the members say of their manifests cannot be read: %w", err)
}
return answer(orderOf(members, reachOfMembers(members, entries, read, edges), edges, changes))
case "check":
if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked.
return recheckOneHead(members[0].ID, answer, func() (string, error) {
var asked string
err := sayingOnTheBus(ctx, inv, func() (err error) {
asked, err = (following{open: open, oneShot: true}).pullUpdated(ctx, members[0].pull(), os.Stderr)
return err
})
return asked, err
})
}
if *group == "" || len(members) < 2 {
return errors.New("delivery check --group <id> --members <json> (two heads or more), or --members <one head>")
}
id, err := askGroupCheck(ctx, open, *group, members)
if err != nil {
return err
}
return answer(map[string]any{"asked": id, "group": *group})
case "go":
if len(positionals) != 1 {
return errors.New("delivery go <plan> [--by <who>] [--why <text>]")
@@ -1093,133 +1060,6 @@ func theGraph(ctx context.Context, inv *inventory.Inventory) ([]inventory.Entry,
return entries, read, edges, nil
}
// cycleVerdict is the failing verdict of a group whose modules are ordered in a cycle, and whether there is one: the
// group's composed check, said without a build. The manifests' change is read from the forge; an answer it cannot give
// is an error, never read as no cycle.
func cycleVerdict(ctx context.Context, group string, members []orderMember, reach map[string]orderReach,
edges []inventory.Edge, entries []inventory.Entry) (link.Checked, bool, error) {
changes, err := changesOf(ctx, changeSourcesOf(members, entries, reach))
if err != nil {
return link.Checked{}, false, fmt.Errorf("what the group's members say of their manifests cannot be read: %w", err)
}
var moved []string
for _, m := range members {
moved = appendNew(moved, reach[m.ID].Moved...)
}
said := orderEdges(moved, changes, edges).CycleSaid()
if said == "" {
return link.Checked{}, false, nil
}
first := members[0].pull()
c := link.Checked{ID: link.NewBuildID(time.Now()), Owner: first.Owner, Repo: first.Repo, Number: first.Number,
Commit: first.Commit, Verdict: "fail", Summary: said, Report: said, Group: group,
Gate: &link.CheckLayer{Verdict: "fail", Summary: said}}
for _, m := range members {
if group == "" {
break // a lone pull request's verdict names no group and no member
}
p := m.pull()
c.Members = append(c.Members, link.CheckedMember{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
Trunk: twelve(p.BaseSHA)})
}
return c, true, nil
}
// twelve is a commit as twelve characters.
func twelve(c string) string {
if len(c) > 12 {
return c[:12]
}
return c
}
// groupPrimary is the head a group's composed check is run from: the one moving the controller, which then
// judges the group by itself; else the one moving the node-engine, whose validator judges; else the first.
func groupPrimary(members []orderMember, reach map[string]orderReach) int {
for _, want := range []string{"mesh-controller", "mesh-host"} {
for i, m := range members {
if slices.Contains(reach[m.ID].Moved, want) {
return i
}
}
}
return 0
}
// askGroupCheck asks the build seat for a group's composed check: every head laid over the mesh in turn,
// judged as one future state (novox/hq ADR 0239). The verdict comes back as `checked` with the group's id,
// for mesh-delivery; the forge's holder sets no status from it.
func askGroupCheck(ctx context.Context, open *stores, group string, members []orderMember) (string, error) {
entries, read, edges, err := theGraph(ctx, open.inventory)
if err != nil {
return "", err
}
reach := reachOfMembers(members, entries, read, edges)
// **A cycle of edges fails the group's composed check, naming every module and edge** (novox/hq ADR 0419 clause 6),
// before the build seat is asked: no composition of the heads has an order to be sent in.
if failing, cyclic, err := cycleVerdict(ctx, group, members, reach, edges, entries); err != nil {
return "", err
} else if cyclic {
sayChecked(ctx, os.Stderr, failing)
return failing.ID, nil
}
primary := groupPrimary(members, reach)
p := members[primary].pull()
scope := pullScope(p, entries, read, edges)
// The gate runs over what any member moves; a judge from the primary alone.
for _, m := range members {
r := reach[m.ID]
scope.Modules = appendNew(scope.Modules, r.Moved...)
scope.Dependents = appendNew(scope.Dependents, r.Dependents...)
}
request, err := checkRequestFor(ctx, open, p, scope, entries)
if err != nil {
return "", err
}
request.Check.Group = group
// A group's check is queued as ready when any member's pull request is marked ready.
request.Check.Ready = slices.ContainsFunc(members, func(m orderMember) bool { return m.Ready })
world, err := restOfTheMesh(ctx, open, shelfOf(entries), "")
if err != nil {
return "", err
}
for i, m := range members {
if i == primary {
continue
}
mp := m.pull()
// As the mesh clones a module built from it; a repository no module is built from, from the forge.
source := inventory.Source{Seat: gitSeat, Repository: mp.Owner + "/" + mp.Repo}
for _, e := range entries {
if !e.Provided && sameRepository(e.Source.Repository, link.SourceMoved{Owner: mp.Owner, Repo: mp.Repo}) {
source = e.Source
break
}
}
url := source.Repository
if source.Seat != "" {
url, err = clonedFromSeat(world, source.Seat, source.Repository)
}
if err != nil {
return "", fmt.Errorf("%s cannot be cloned for the group's check: %w", m.ID, err)
}
request.Check.Members = append(request.Check.Members, link.GroupMember{Owner: mp.Owner, Repo: mp.Repo,
Number: mp.Number, Repository: url, Ref: mp.Commit, Paths: mp.Paths, Base: mp.Base})
}
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return "", err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return "", err
}
fmt.Fprintf(os.Stderr, "group %s: asked %s to check %d head(s) composed together, as %s\n", group, seat,
len(members), request.ID)
return request.ID, nil
}
// appendNew appends what is not there yet.
func appendNew(to []string, items ...string) []string {
for _, i := range items {
@@ -1230,15 +1070,6 @@ func appendNew(to []string, items ...string) []string {
return to
}
// shelfOf is the catalogue's manifests by name.
func shelfOf(entries []inventory.Entry) map[string]catalogue.Manifest {
shelf := map[string]catalogue.Manifest{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
}
return shelf
}
// sayPlanMoved says a walk kept in a new state as the controller's `plan-moved`, the plan whole: what the
// delivery it walks reads its steps from. Never in the way of the walk: said beside it, and a save that could
// not be said is logged — the delivery's owner, which also asks for the walks it follows, finds it by
@@ -1380,25 +1211,3 @@ func overNATS(conn *nats.Conn) (link.Bus, error) {
}
return link.OverNATS{Conn: conn, JS: js}, nil
}
// recheckAnswer is what `delivery check` answers for one head: the id of the check asked of the build seat (empty when
// the controller said the verdict itself and asked nothing), which mesh-delivery reads, and the head rechecked.
func recheckAnswer(asked, member string) map[string]any {
return map[string]any{"asked": asked, "rechecked": member}
}
// recheckOneHead runs a one-head `delivery check` and answers it as data. Standard output is the answer's JSON alone,
// or the seat verb reads no answer from it (novox/hq issue 1311): while the check runs, whatever it prints to standard
// output (the controller's own path prints its prose there, as do the stores it reads) goes to standard error.
func recheckOneHead(member string, answer func(any) error, ask func() (string, error)) error {
real := os.Stdout
os.Stdout = os.Stderr
asked, err := func() (string, error) {
defer func() { os.Stdout = real }()
return ask()
}()
if err != nil {
return err
}
return answer(recheckAnswer(asked, member))
}
@@ -1,85 +0,0 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// `delivery check` with one head answers the id of the check it asked, as data (novox/hq issue 1311): the prose
// the check says goes to standard error, so standard output is the document alone and answerOf reads it.
func TestDeliveryCheckOfOneHeadAnswersTheRequestID(t *testing.T) {
argv := []string{"delivery", "check", "--members", "[]"}
doc, err := json.Marshal(recheckAnswer("check-42", "novox/x@abc"))
if err != nil {
t.Fatal(err)
}
// Seen failing first: the prose the command used to print on standard output before the document.
polluted := append([]byte("novox/x#7 (abc): asked the build seat as check-42\n"), doc...)
if got := answerOf(argv, polluted, "", true); got.Answer != nil {
t.Fatalf("prose before the document should leave no answer; got %v", got.Answer)
}
got := answerOf(argv, doc, "novox/x#7 (abc): asked the build seat as check-42\n", true)
m, ok := got.Answer.(map[string]any)
if !ok || m["asked"] != "check-42" || m["rechecked"] != "novox/x@abc" {
t.Fatalf("the answer is %v; want asked check-42 and rechecked novox/x@abc", got.Answer)
}
if !strings.Contains(got.Output, "asked the build seat") {
t.Fatalf("the prose is still said in the text: %q", got.Output)
}
}
// The text of the other callers does not change: sayChecked writes the same line to the writer it is given, and
// the controller's own path still gives PullUpdated standard output.
func TestSayCheckedWritesTheSameLineToItsWriter(t *testing.T) {
var w bytes.Buffer
sayChecked(context.Background(), &w, link.Checked{ID: "c1", Owner: "novox", Repo: "x", Number: 7, Commit: "0123456789abcdef",
Verdict: "pass", Summary: "fine"})
want := "c1: novox/x#7 at 01234567 checked on a machine that did not say which: gate PASS — fine; repository none\n"
if w.String() != want {
t.Fatalf("sayChecked said %q; want %q", w.String(), want)
}
}
// Whatever the check prints to standard output while it runs (checks.go namedForTheCheck, the inventory's pointer
// repair) must not reach the answer's standard output: it is exactly one JSON document (novox/hq issue 1311).
func TestOneHeadRecheckLeavesStandardOutputTheAnswerAlone(t *testing.T) {
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
e, ew, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
realOut, realErr := os.Stdout, os.Stderr
os.Stdout, os.Stderr = w, ew
runErr := recheckOneHead("novox/x@abc", func(v any) error { return json.NewEncoder(os.Stdout).Encode(v) },
func() (string, error) {
fmt.Println(" the earlier checks' builds could not be read, so the check builds")
fmt.Fprintln(os.Stdout, "repaired a pointer")
return "check-42", nil
})
os.Stdout, os.Stderr = realOut, realErr
w.Close()
ew.Close()
if runErr != nil {
t.Fatal(runErr)
}
var out, errOut bytes.Buffer
_, _ = out.ReadFrom(r)
_, _ = errOut.ReadFrom(e)
var doc map[string]any
if err := json.Unmarshal(out.Bytes(), &doc); err != nil || doc["asked"] != "check-42" || doc["rechecked"] != "novox/x@abc" {
t.Fatalf("standard output %q is not the answer alone (%v)", out.String(), err)
}
if !strings.Contains(errOut.String(), "repaired a pointer") {
t.Fatalf("the prose should be on standard error: %q", errOut.String())
}
}
+10 -1
View File
@@ -41,7 +41,7 @@ type stalledLine struct {
// waiting for the owner's word and what holds it (novox/hq issues 362, 529).
Walk string `json:"walk,omitempty"`
// Waiting are the merge checks a line of the state `unanswered` waits on, as mesh-delivery says each:
// "mesh/merge-gate pending since <UTC time>", "mesh/repo-check never set" (novox/hq issue 438).
// "mesh/check pending since <UTC time>", "mesh/check never set" (novox/hq issue 438).
Waiting []string `json:"waiting,omitempty"`
// Checks are the same merge checks as data, which the controller words in the operator's own time (novox/hq
// issue 443): a time inside a finished sentence cannot be said again in another zone. A mesh-delivery from before
@@ -58,8 +58,17 @@ type stalledLine struct {
// Severity is "urgent" for a group held or half merged and a refused member (novox/hq issues 1275, 1278); empty
// or anything else is a warning, as before.
Severity string `json:"severity,omitempty"`
// Needs is "new-head" for a delivery whose cure is its author's new head (novox/hq ADR 0505 §5), Author that author
// as the operator reads it ("the author of feat/x", with the session that pushed it when its pull request names one)
// and Branch its branch. A mesh-delivery from before says none, and the words are as before.
Needs string `json:"needs,omitempty"`
Author string `json:"author,omitempty"`
Branch string `json:"branch,omitempty"`
}
// needsNewHead is what a stalled line needs when its cure is its author's new head (ADR 0505 §5).
const needsNewHead = "new-head"
// severity is the condition's severity a stalled line asks: urgent when it says so, a warning otherwise.
func (l stalledLine) severity() conditions.Severity {
if l.Severity == string(conditions.Urgent) {
@@ -181,32 +181,3 @@ func TestATruncatedChangeIsReadAsChangingTheManifestsOfWhatItMoves(t *testing.T)
t.Fatalf("the member's change is read as %+v", src)
}
}
// A lone pull request's cycle fails its check, naming every module and edge, with no group and no members.
func TestALonePullRequestInACycleFailsWithNoGroup(t *testing.T) {
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
const head = "3333333333333333333333333333333333333333"
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, _, file, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil
}
if file == "modules/x/module.json" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
m := orderMember{ID: "novox/r@" + head[:12], Repository: "novox/r", Base: "main", Head: head, Number: 5,
Paths: []string{"modules/x/module.json", "modules/y/module.json"}}
c, cyclic, err := cycleVerdict(context.Background(), "", []orderMember{m}, map[string]orderReach{m.ID: {Moved: []string{"mx", "my"}}}, nil, nil)
if err != nil || !cyclic || c.Verdict != "fail" || c.Group != "" || len(c.Members) != 0 || !strings.Contains(c.Summary, "s.a") ||
!strings.Contains(c.Summary, "my") {
t.Fatalf("the verdict is %+v (cyclic %v, %v)", c, cyclic, err)
}
}
@@ -59,6 +59,18 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
}
}
// heardOf has this mesh's own view of the plans take in the plan as the bucket holds it. A verb of the
// controller (deliveryCommand, plansCommand) opens an inventory of its own and writes the plan there; this
// mesh's inventory learns of the write from the bucket's watch, a moment later, and a loaded machine makes the
// moment long enough for advanceHeld to read the plan as it was, still waiting for its word, and ask nothing
// (issue 1399). Reading the plan from the bucket puts it into the view at once, with no wait to bound.
heardOf := func(id string) {
t.Helper()
if _, err := inv.PlanByID(ctx, id); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
@@ -90,6 +102,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
heardOf(p.ID)
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
@@ -119,6 +132,7 @@ func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testin
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
heardOf(p.ID)
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
-4
View File
@@ -136,10 +136,6 @@ func TestTheDeliveryVerbsComposeTheirCommands(t *testing.T) {
{"delivery-walks", map[string]any{}, []string{"delivery", "walks"}},
{"delivery-walks", map[string]any{"plan": "plan-1"}, []string{"delivery", "walks", "--plan", "plan-1"}},
{"delivery-order", map[string]any{"members": "[]"}, []string{"delivery", "order", "--members", "[]"}},
{"delivery-check", map[string]any{"group": "feat/x", "members": "[]"},
[]string{"delivery", "check", "--group", "feat/x", "--members", "[]"}},
{"delivery-plan", map[string]any{"repository": "novox/a", "paths": "x", "head": "c0"},
[]string{"delivery", "plan", "--repository", "novox/a", "--paths", "x", "--head", "c0"}},
{"plans", map[string]any{"go": "plan-1", "why": "down"}, []string{"plans", "go", "plan-1", "--why", "down"}},
} {
got, err := argvFor(c.verb, c.args)
-10
View File
@@ -35,16 +35,6 @@ func TestAMergeOfABaseTakesWhatStandsOnItAlong(t *testing.T) {
t.Fatalf("%s was rebuilt and stands on nothing that moved (%v)", n, names)
}
}
// The dependents come in base order when the merge orders them: the runtime, then shop, then
// the plugin that stands on shop.
ordered := orderByBases(append([]inventory.Entry{entry("mesh-tools")}, got...), against)
pos := map[string]int{}
for i, e := range ordered {
pos[e.Manifest.Module] = i
}
if !(pos["mesh-tools"] < pos["shop"] && pos["shop"] < pos["shop-plugin"]) {
t.Fatalf("not in base order: %v", ordered)
}
// Nothing moved: nothing follows.
if more := dependentsOf(nil, entries, against); len(more) != 0 {
t.Fatalf("with nothing moved, %d module(s) were rebuilt", len(more))
+1 -1
View File
@@ -669,7 +669,7 @@ func localDoctor(ctx context.Context) (*doctor, func(), error) {
if err != nil {
return nil, nil, err
}
js, err := dialTheBus()
js, err := aBus()
if err != nil {
open.Close()
return nil, nil, err
@@ -1,20 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
func TestADryRunsOutcomeIsTakenInByNothing(t *testing.T) {
err := builds{}.Built(t.Context(), link.BuildResult{
ID: "build-1", Repository: "ssh://forge/app.git", Ref: "unreviewed", Module: "app", DryRun: true,
Manifest: json.RawMessage(`{"module":"app","version":"1"}`),
})
if err != nil {
t.Fatalf("a dry run's outcome was not simply set aside: %v", err)
}
}
-29
View File
@@ -1,29 +0,0 @@
package main
import (
"encoding/json"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// A dry run's outcome is looked at, never taken in (novox/hq issue 240). The daemon here holds no
// store at all, so anything that tried to record or register would fail rather than pass quietly.
// The mark survives the wire both ways: asked as a dry run, answered as one.
func TestTheDryRunMarkTravelsWithTheBuild(t *testing.T) {
raw, _ := json.Marshal(link.BuildRequest{ID: "build-1", Repository: "r", DryRun: true})
var asked link.BuildRequest
if err := json.Unmarshal(raw, &asked); err != nil || !asked.DryRun {
t.Fatalf("the request lost its dry-run mark: %s", raw)
}
raw, _ = json.Marshal(link.BuildResult{ID: "build-1", DryRun: true})
var answered link.BuildResult
if err := json.Unmarshal(raw, &answered); err != nil || !answered.DryRun {
t.Fatalf("the outcome lost its dry-run mark: %s", raw)
}
raw, _ = json.Marshal(link.BuildResult{ID: "build-2"})
if string(raw) != `{"id":"build-2","repository":"","on":""}` {
t.Fatalf("an ordinary outcome carries a dry-run mark: %s", raw)
}
}
-20
View File
@@ -12,7 +12,6 @@ import (
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// What the core's bounds are set from (novox/hq to-be 45 Phase 0).
@@ -24,25 +23,6 @@ import (
// controller as it hears each — a send's first report, a machine's next word, a plan leaving a tier, a
// build's outcome — and summarised here per machine, repository or module.
// recordBuildDuration measures one build from its ask to its outcome heard.
func recordBuildDuration(ctx context.Context, inv *inventory.Inventory, result link.BuildResult, asked time.Time) {
if asked.IsZero() || result.ID == "" {
return
}
subject := result.Module
if subject == "" {
subject = result.Repository
}
detail := "built"
if result.Failed != "" {
detail = "failed: " + firstLine(result.Failed)
}
if err := inv.RecordDuration(ctx, inventory.Duration{Kind: inventory.DurationBuild, Subject: subject,
Node: result.On, Ref: result.ID, Started: asked, Took: time.Since(asked), Detail: detail}); err != nil {
fmt.Fprintf(os.Stderr, "%s: how long it took could not be recorded: %v\n", result.ID, err)
}
}
// durationSummary is one subject's measurements of one kind.
type durationSummary struct {
Kind string `json:"kind"`
-12
View File
@@ -19,7 +19,6 @@ import (
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -241,17 +240,6 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
scrub.Site(o.Site)
}
// What a merge check runs in and reads beside it, as the build seat would be asked for it.
if held, err := inv.Held(ctx); err == nil {
for language, reference := range builder.ToolchainsOf(held) {
if f.Versions.Toolchains == nil {
f.Versions.Toolchains = map[string]string{}
}
f.Versions.Toolchains[language] = catalogue.Recorded(reference)
}
} else {
return snapshot.Facts{}, err
}
for _, e := range entries {
if dir, core := coreModules[e.Manifest.Module]; core && !e.Provided && e.Source.Repository != "" {
for d, ref := range besideRefs(dir, current[e.Manifest.Module].Commit) {
@@ -1,84 +0,0 @@
//go:build integration
package main
import (
"context"
"errors"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/identity"
"github.com/novox/mesh-controller/internal/link"
)
// aForgeAndABuildSeat gives a test what a real ask of a build needs: the forge holding the git seat, assigned to anchor, so
// the clone URL composes; the identity store; the build seat's queue on a real bus; and the environment of the serving controller, which is
// never the controller's terminal (a verb's, in effect: startedAtTheTerminal is false).
func aForgeAndABuildSeat(t *testing.T, open *stores) *broker.JetStream {
t.Helper()
js, _ := aBuildQueue(t)
// The ask signs what it sends with the controller's identity, as the serving controller does.
identity.ForTest(t)
forge, err := catalogue.ParseManifest([]byte(`{"module":"gitea","version":"1",
"claims":[{"name":"git","scope":"mesh"}],"provides":[{"name":"git","scope":"mesh"}],
"serves":{"git":{"scheme":"http","port":3000}}}`))
if err != nil {
t.Fatal(err)
}
register(t, open, forge)
if _, err := assign(t.Context(), open, "anchor", "gitea"); err != nil {
t.Fatal(err)
}
t.Setenv(servedVar, "1")
return js
}
// 1146/AC9, through the real ask (askABuild, buildOneAsked, verbMayAsk) and a real bus: the first build of a module a
// merge adds on the operator's Approve is queued on the build seat; the same ask without that warrant, and the warrant's
// ask for a path the merge does not hold, are refused by the guard of a build through a verb.
func TestTheApprovedFirstBuildPassesTheGuardOfAVerbAndIsQueued(t *testing.T) {
const head, commit = "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", "0177ea7a00000000"
open := aMesh(t)
ctx := t.Context()
js := aForgeAndABuildSeat(t, open)
a := aDeliveryMergeOf(t, head, map[string]string{"novox/mesh-delivery@a1a1a1a1a1a1": head})
wasKept, wasAsk := keptDeliveryMerge, routerAskOf
keptDeliveryMerge = func(context.Context, string, string, int, string) (deliveryMerge, bool, error) {
return a.merge, true, nil
}
routerAskOf = func(context.Context, catalogueReader, string, string) (*asks.Ask, *asks.Warrant, error) {
return &a.ask, &a.warrant, nil
}
t.Cleanup(func() { keptDeliveryMerge, routerAskOf = wasKept, wasAsk })
// Without the warrant the guard refuses, whatever the repository holds.
_, err := askABuild(ctx, buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "", "main")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("1146/AC9: a first build asked without the warrant was not refused: %v", err)
}
// And a path the merge does not hold is refused on the warrant.
warranted := withFirstBuildWarrant(ctx, "novox/mesh-delivery", []string{""}, "the Approve")
if _, err := askABuild(warranted, buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "other", "main"); !errors.As(err, &policy) {
t.Fatalf("1146/AC9: the warrant covered a path the merge does not hold: %v", err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-delivery", Base: "main", HeadSHA: head, Commit: commit, Number: 7,
MergedAt: time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano), Paths: []string{"module.json", "main.go"},
ModuleDirs: []string{"."}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
queue, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
if len(queue.Asks) != 1 {
t.Fatalf("1146/AC9: the build queue holds %+v, want the one first build", queue)
}
}
@@ -1,29 +0,0 @@
package main
import "testing"
// 1146/AC9: the warrant the controller checked covers exactly the repository on the git seat and the module paths the
// merge holds; nothing else passes verbMayAsk on it.
func TestTheFirstBuildWarrantCoversOnlyItsRepositoryAndItsModulePaths(t *testing.T) {
ctx := withFirstBuildWarrant(t.Context(), "novox/mesh-delivery", []string{"", "view"}, "the Approve")
w := firstBuildWarrantIn(ctx)
for _, c := range []struct {
source buildSource
path string
covers bool
}{
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "", true},
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "/view/", true},
{buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "other", false},
{buildSource{Repository: "agent/mesh-delivery", Seat: "git"}, "", false},
{buildSource{Repository: "novox/mesh-delivery"}, "", false},
{buildSource{Repository: "novox/mesh-delivery", Seat: "elsewhere"}, "", false},
} {
if got := w.covers(c.source, c.path); got != c.covers {
t.Errorf("1146/AC9: %v at %q covered = %v, want %v", c.source, c.path, got, c.covers)
}
}
if firstBuildWarrantIn(t.Context()).covers(buildSource{Repository: "novox/mesh-delivery", Seat: "git"}, "") {
t.Error("1146/AC9: a context with no warrant covers an ask")
}
}
+77 -21
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"regexp"
"strings"
"time"
@@ -46,7 +47,7 @@ func (f following) firstBuilds(ctx context.Context, m link.SourceMoved, entries
case len(added) == 0:
return false, nil
}
if err := askFirstBuilds(ctx, inv, m, now, added, why); err != nil {
if err := askFirstBuilds(ctx, f.open, m, now, added, why); err != nil {
return true, notNow(err)
}
return true, nil
@@ -67,27 +68,35 @@ func keepFirstMerge(ctx context.Context, inv *inventory.Inventory, m link.Source
return err
}
// askFirstBuilds keeps the merge, asks the builds, and marks the merge as no longer waiting for them.
func askFirstBuilds(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, now time.Time, added []string,
// askFirstBuilds keeps the merge as answered and registers its modules from the builder's done keys (novox/hq ADR 0501
// §3, §6): nothing asks their first builds. The merge kept is the registration's warrant (approvedMove reads it by the
// commit the indexer's trunk value names); a module whose key is not done yet is registered when the key's write wakes
// the take-in (new_modules.go).
func askFirstBuilds(ctx context.Context, open *stores, m link.SourceMoved, now time.Time, added []string,
why string) error {
inv := open.inventory
repository := m.Owner + "/" + m.Repo
if err := keepFirstMerge(ctx, inv, m, now, false); err != nil {
return err
}
fmt.Printf("%s merged into %s (%.8s) on the operator's Approve (%s): the mesh builds nothing from it yet, so the "+
"first build of %s is asked\n", repository, m.Base, m.Commit, why, strings.Join(added, ", "))
// The ask carries the warrant just checked, for this repository and these module paths alone: the guard of a build
// through a verb (verbMayAsk) lets exactly that through.
paths := make([]string, 0, len(added))
for _, dir := range added {
if dir == "." {
dir = ""
}
paths = append(paths, dir)
fmt.Printf("%s merged into %s (%.8s) on the operator's Approve (%s): the mesh builds nothing from it yet, so %s "+
"are registered from the builder's done keys, asking no build\n", repository, m.Base, m.Commit, why,
strings.Join(added, ", "))
// Answered: it no longer waits, so the word's arrival again does nothing twice.
if err := inv.AnswerMerge(ctx, repository, m.Commit, inventory.WaitingRecord, false); err != nil {
return err
}
askBuilds(withFirstBuildWarrant(ctx, repository, paths, why), inv, m, buildSource{Repository: repository, Seat: gitSeat}, added)
// Answered: it no longer waits, so the word's arrival again asks nothing twice.
return inv.AnswerMerge(ctx, repository, m.Commit, inventory.WaitingRecord, false)
modules, err := newModulesOf(ctx, repository)
switch {
case err != nil:
// Said, not failed: the merge is answered, and the look at every module when a controller starts, or the next
// write of the builder or the indexer for one of them, registers them.
fmt.Printf(" the indexer's modules of %s cannot be read now (%v): they are registered when the builder's next "+
"write of one wakes the take-in\n", repository, err)
case len(modules) > 0:
inBackground(func() { takeNewModules(context.WithoutCancel(ctx), open, modules...) })
}
return nil
}
// resumeFirstBuilds runs firstBuilds for the first merges waiting for the word mesh-delivery just announced for a pull
@@ -102,7 +111,7 @@ func (f following) resumeFirstBuilds(ctx context.Context, d deliveryMerge) error
}
for _, w := range waiting {
k := announcedOf(w)
if !k.FirstBuild || !strings.EqualFold(k.Owner+"/"+k.Repo, d.Owner+"/"+d.Repo) || k.Number != d.Number ||
if !k.FirstBuild || len(k.Held) > 0 || !strings.EqualFold(k.Owner+"/"+k.Repo, d.Owner+"/"+d.Repo) || k.Number != d.Number ||
k.HeadSHA != d.HeadSHA {
continue
}
@@ -125,7 +134,7 @@ func (f following) resumeFirstBuilds(ctx context.Context, d deliveryMerge) error
case err != nil:
return notNow(err)
case len(added) > 0:
if err := askFirstBuilds(ctx, inv, k.SourceMoved, time.Now().UTC(), added, why); err != nil {
if err := askFirstBuilds(ctx, f.open, k.SourceMoved, time.Now().UTC(), added, why); err != nil {
return notNow(err)
}
}
@@ -172,6 +181,29 @@ func firstBuildsOf(ctx context.Context, inv catalogueReader, m link.SourceMoved,
return nil, "", err
}
// heldFirstBuild says a first build was refused because its manifest uses a word the running build-agent does not read,
// and its merge is kept waiting for a build-agent that reads it (novox/hq ADR 0482 §5, issue 1173). Read from a merge kept
// before R4e, never written since: nothing asks a first build (ADR 0501 §6), and a controller that starts answers such a
// merge and registers its modules from their keys (answerHeldMerges).
type heldFirstBuild struct {
Module string `json:"module"`
Path string `json:"path"`
Word string `json:"word"`
At time.Time `json:"at"`
}
// unknownWord is how the strict reading of a manifest says a word is not one it knows: encoding/json's own words, which
// the build-agent's answer carries whole. The build's answer has no typed field for it (Failed is the builder's words).
var unknownWord = regexp.MustCompile(`json: unknown field "([^"]+)"`)
// refusedWord is the word a build's failure says the reader did not know; empty when it says none.
func refusedWord(failed string) string {
if m := unknownWord.FindStringSubmatch(failed); m != nil {
return m[1]
}
return ""
}
// probeFirstBuildsID is the probe that reads the first merges waiting for mesh-delivery's word.
const probeFirstBuildsID = "D18"
@@ -203,9 +235,10 @@ func firstBuildObservations(waiting []inventory.BatchedMerge, now time.Time) []c
Headline: fmt.Sprintf("The first build of %s is not asked: no Approve was announced", repository),
Explanation: fmt.Sprintf("The merge %.8s into %s adds modules the mesh builds nothing from yet. It registers them "+
"only when mesh-delivery announces that it merged the pull request on your Approve of that head, and no such "+
"announcement has come in %s. The merge is kept, and the builds are asked the moment the announcement is kept.",
"announcement has come in %s. The merge is kept, and the modules are registered from the builder's keys the moment "+
"the announcement is kept.",
k.Commit, repository, for_),
Resolved: "mesh-delivery announces the merge, or a hand `build` registers the modules",
Resolved: "mesh-delivery announces the merge, or the operator registers the modules at the controller's terminal",
})
}
return out
@@ -217,5 +250,28 @@ func probeFirstBuilds(ctx context.Context, d *doctor) ([]conditions.Observation,
if err != nil {
return nil, err
}
return firstBuildObservations(waiting, time.Now().UTC()), nil
now := time.Now().UTC()
return append(firstBuildObservations(waiting, now), newModuleObservations(now)...), nil
}
// answerHeldMerges answers every merge a controller of before held for a build-agent that reads its new modules'
// manifests (issue 1173): nothing asks their first builds any more, and the look at every module when this controller
// starts registers them from their keys (ADR 0501 §6, §7).
func answerHeldMerges(ctx context.Context, inv *inventory.Inventory) {
waiting, err := inv.WaitingMerges(ctx)
if err != nil {
fmt.Printf("the merges held for a newer build-agent could not be read: %v\n", err)
return
}
for _, w := range waiting {
if k := announcedOf(w); len(k.Held) > 0 {
if err := inv.AnswerMerge(ctx, w.Repository, w.Commit, inventory.WaitingRecord, false); err != nil {
fmt.Printf("the merge %.8s of %s held for a newer build-agent could not be answered: %v\n", w.Commit,
w.Repository, err)
continue
}
fmt.Printf("the merge %.8s of %s was held for a newer build-agent; its modules are registered from the "+
"builder's keys instead\n", w.Commit, w.Repository)
}
}
}
@@ -5,71 +5,48 @@ package main
import (
"context"
"encoding/json"
"slices"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1146/AC7, whole against a real bus and the store, in both orders of the two events (novox/hq ADR 0432, issue 1146):
// mesh-delivery #7 merged on the operator's Approve into a repository the mesh builds nothing from. The forge's merge,
// heard as the controller hears it (SourceMoved), and mesh-delivery's word, published on the subject the controller
// consumes with the router's record of the ask in its bucket, come in either order; each way the first build of the
// module the merge commit holds is asked exactly once, its outcome registers the module, and it is in the catalogue.
// Without the Approve in the record nothing is asked. Store-backed: skips without one.
func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegistersTheModule(t *testing.T) {
// 1146/AC7 on keys (ADR 0501 §3, §6; issue 1851, A5), whole against a real bus and the store, in both orders of the two
// events: mesh-delivery #7 merged on the operator's Approve into a repository the mesh builds nothing from. The forge's
// merge, heard as the controller hears it (SourceMoved), and mesh-delivery's word, published on the subject the
// controller consumes with the router's record of the ask in its bucket, come in either order; each way the module the
// merge commit holds is registered from the builder's done key of it, and nothing asks a build. Without the Approve in
// the record it is not registered. Store-backed: skips without one.
func TestAMergeOnTheApproveIntoANewRepositoryRegistersItsModuleFromItsKey(t *testing.T) {
const head, commit = "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1", "0177ea7a00000000"
for _, c := range []struct {
what string
wordFirst bool
approved bool // the router's record holds the operator's Approve of the merged head
wantRegiter bool
what string
wordFirst bool
approved bool // the router's record holds the operator's Approve of the merged head
wantRegister bool
}{
{what: "the forge's merge before mesh-delivery's word", approved: true, wantRegiter: true},
{what: "mesh-delivery's word before the forge's merge", wordFirst: true, approved: true, wantRegiter: true},
{what: "the forge's merge before mesh-delivery's word", approved: true, wantRegister: true},
{what: "mesh-delivery's word before the forge's merge", wordFirst: true, approved: true, wantRegister: true},
{what: "a word whose record holds no Approve", wordFirst: true},
} {
t.Run(c.what, func(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
// The real ask, guard and queue included: only its answers are kept, so the build's outcome can be matched.
aForgeAndABuildSeat(t, open)
// Written by the controller's merge path on its own goroutine, read by the test: under one lock.
var mu sync.Mutex
var asks [][3]string
var ids []string
asked := func() [][3]string { mu.Lock(); defer mu.Unlock(); return slices.Clone(asks) }
realAsk := askABuild
askABuild = func(c context.Context, source buildSource, path, ref string) (string, error) {
id, err := realAsk(c, source, path, ref)
if err == nil {
mu.Lock()
asks = append(asks, [3]string{source.Repository, path, ref})
ids = append(ids, id)
mu.Unlock()
}
return id, err
}
t.Cleanup(func() { askABuild = realAsk })
noBuildAsked(t)
b := aBuilderOnTheBus(t)
js := b.js
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "operator-channel", Version: "1",
DefinesSeats: []catalogue.SeatDeclaration{{Name: broker.AsksSeat, Records: []string{"asks"}}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/operator-channel", BuiltFrom: "c0ffee"}); err != nil {
t.Fatal(err)
}
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
@@ -87,10 +64,14 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
a := aDeliveryMergeOf(t, head, map[string]string{"novox/mesh-delivery@a1a1a1a1a1a1": head})
if c.approved {
record, _ := json.Marshal(map[string]any{"state": "closed", "ask": a.ask, "warrant": a.warrant})
if _, err := records.Put(ctx, theDelivery+"."+a.merge.AskID, record); err != nil {
if _, err := records.Put(ctx, sdkdelivery.TellerModule+"."+a.merge.AskID, record); err != nil {
t.Fatal(err)
}
}
// The indexer's value and the builder's done key of the module the merge commit holds.
fp := fpOf('9')
manifest := b.indexed(t, "mesh-delivery-view", "1", "novox/mesh-delivery", ".", commit, fp)
b.built(t, "mesh-delivery-view", "1", fp, manifest, sdkdelivery.BuildDone)
following := following{open: open}
server := link.ConnectNats(js, nil, noListener{})
@@ -104,8 +85,6 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
done := make(chan struct{})
go func() { defer close(done); _ = server.Serve(serving) }()
t.Cleanup(func() { stop(); <-done })
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
word := func() {
t.Helper()
@@ -133,37 +112,27 @@ func TestAMergeOnTheApproveIntoANewRepositoryAsksTheFirstBuildAndItsOutcomeRegis
forge()
} else {
forge()
if len(asked()) != 0 {
t.Fatalf("1146/AC7: the merge asked %v before mesh-delivery's word was kept", asked())
time.Sleep(300 * time.Millisecond)
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); known {
t.Fatal("1146/AC7: registered before mesh-delivery's word was kept")
}
word()
}
if !c.wantRegiter {
if len(asked()) != 0 {
t.Fatalf("1146/AC7: a merge whose record holds no Approve asked %v", asked())
if !c.wantRegister {
time.Sleep(time.Second)
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); known {
t.Fatal("1146/AC7: a merge whose record holds no Approve registered its module")
}
return
}
want := [3]string{"novox/mesh-delivery", "", "main"}
becomes(t, 10*time.Second, "the first build asked", func() bool { return len(asked()) >= 1 })
if got := asked(); len(got) != 1 || got[0] != want {
t.Fatalf("1146/AC7: the merge asked %v, want %v once", got, want)
}
mu.Lock()
first := ids[0]
mu.Unlock()
build := onTrunk(first, "novox/mesh-delivery", "git", "", map[string]any{"module": "mesh-delivery-view", "version": "1"})
build.Commit = commit
if _, _, err := takeIn(ctx, inv, build); err != nil {
t.Fatalf("1146/AC7: the outcome of the first build was refused: %v", err)
}
becomes(t, 10*time.Second, "the module registered", func() bool {
_, known := inventoryHolds(t, inv, "mesh-delivery-view")
return known
})
src, err := inv.SourceOf(ctx, "mesh-delivery-view")
if err != nil || src.Repository != "novox/mesh-delivery" {
t.Fatalf("1146/AC7: the module is built from %q (%v)", src.Repository, err)
}
if _, known := inventoryHolds(t, inv, "mesh-delivery-view"); !known {
t.Fatal("1146/AC7: the module is not in the catalogue")
}
})
}
}
+2 -2
View File
@@ -64,7 +64,7 @@ func registerFrom(t *testing.T, forms *sourceForms) (int64, error) {
mesh := &aMeshRegisteredFromTheURL{anEmptyMesh{asked: true,
request: inventory.BuildRequest{ID: "build-1", Repository: aRepoOnTheForge, For: "build"}}, aRepoOnTheForge}
return mayRegisterIn(t.Context(), mesh, forms, "mesh-delivery-view", inventory.Source{Repository: aRepoOnTheForge},
"build-1", "", "main")
registrationAsk{Build: "build-1"}, "", "main")
}
// 1277/AC2: an address unread twice, then read, is waited out and the forge is asked.
@@ -112,7 +112,7 @@ func TestAKnownAddressAndAnotherHostIsStillNotOnTheForge(t *testing.T) {
const elsewhere = "http://elsewhere.example/x/y.git"
mesh := &aMeshRegisteredFromTheURL{anEmptyMesh{asked: true, request: inventory.BuildRequest{ID: "build-1", For: "build"}}, elsewhere}
_, err := mayRegisterIn(t.Context(), mesh, sourceFormsRead(context.Background(), time.Now, read), "mesh-delivery-view",
inventory.Source{Repository: elsewhere}, "build-1", "", "main")
inventory.Source{Repository: elsewhere}, registrationAsk{Build: "build-1"}, "", "main")
if !errors.Is(err, errNotItsSource) || !strings.Contains(err.Error(), "is not on the mesh's forge") {
t.Fatalf("1277/AC4: the refusal changed: %v", err)
}
+1 -1
View File
@@ -68,7 +68,7 @@ func TestTheForgeNotAnsweringInTimeIsNotAskedAgain(t *testing.T) {
// An ask that hangs past the bound is cut off at the bound, not at its own wait, and the failure still says it
// waited: the receive loop is held no longer than the bound (novox/hq issue 486).
func TestAnAskHangingPastTheBoundIsCutOffAtIt(t *testing.T) {
w := waitOut{bound: 200 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
w := waitOut{bound: 100 * time.Millisecond, first: 10 * time.Millisecond, most: 50 * time.Millisecond,
now: time.Now, sleep: sleepOrDone}
asked := 0
began := time.Now()
+6 -2
View File
@@ -444,8 +444,12 @@ func TestAMergeThatDeletesAModulePlansNothingToBuildForIt(t *testing.T) {
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
planBuilt(ctx, open, "kept", "", "http://forge/novox/mesh-catalog.git has no module.json at modules/kept, so "+
"there is nothing saying what it is: open …/module.json: no such file or directory", asked0, "build-k")
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-k", Module: "kept", Repository: "novox/mesh-catalog",
At: time.Now().UTC(), Asked: asked0, Failed: "http://forge/novox/mesh-catalog.git has no module.json at modules/kept, so " +
"there is nothing saying what it is: open …/module.json: no such file or directory"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, open)
p, _ := inv.PlanByID(ctx, "plan-deleted")
if p.State == inventory.PlanFailed || p.Modules["kept"].State != planDeleted {
t.Fatalf("a module deleted at its source failed the plan: %s %+v", p.State, p.Modules["kept"])
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"strings"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// The grants issued (novox/hq ADR 0528 clause 2, ADR 0429 §6 rule 6, issue 1394): what the assigner's sixth rule reads,
// that a module's bus credential for a fingerprint exists. A module's permissions on the bus are what the bus's user list
// composes from its registered manifest (inv.BusRecords), and a module's manifest is registered at a fingerprint when the
// controller takes it in from the builder's key; so once the list is sent to the machine that runs the bus, the
// credential of every module it carried is issued at the fingerprint of the build it is registered at. The controller
// writes `<module>.<fingerprint hex>` then, naming the accounts it composed for the module, never removed. A module one
// of whose accounts has no password yet (the list leaves it out) has no grant until a later send carries it.
// grantsFrom is the grant of every module whose accounts all have a password in hashes, at the fingerprint current names
// for it; a module whose current build names no fingerprint of the new form has none. principals are the bus's users
// as broker.Users composes them.
func grantsFrom(principals []broker.Principal, hashes map[string]string, current map[string]string) map[string]sdkdelivery.ControllerGrant {
accounts, missing := map[string][]string{}, map[string]bool{}
for _, p := range principals {
if p.Module == "" || (p.Kind != broker.KindModule && p.Kind != broker.KindProcess) {
continue
}
if hashes[p.Username()] == "" {
missing[p.Module] = true
continue
}
accounts[p.Module] = append(accounts[p.Module], p.Username())
}
out := map[string]sdkdelivery.ControllerGrant{}
for module, fp := range current {
if missing[module] || !strings.HasPrefix(fp, sdkdelivery.SourcePrefix2) {
continue
}
key, err := sdkdelivery.ControllerGrantKey(module, fp)
if err != nil {
continue
}
names := accounts[module]
sort.Strings(names)
out[key] = sdkdelivery.ControllerGrant{Accounts: names}
}
return out
}
// currentFingerprints is, by module, the fingerprint of the build each registered module is registered at: its
// manifest's commit (built_from), that commit's newest successful build, and the fingerprint that build names.
func currentFingerprints(ctx context.Context, inv *inventory.Inventory) (map[string]string, error) {
builds, err := inv.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
out := map[string]string{}
for module, b := range builds {
if b.Commit == "" {
continue
}
id, err := inv.BuildOf(ctx, module, b.Commit)
if err != nil || id == "" {
continue
}
build, found, err := inv.BuildByID(ctx, id)
if err != nil || !found {
continue
}
out[module] = build.SourceFingerprint
}
return out, nil
}
// grantsKV is what writing the grants needs of their bucket.
type grantsKV interface {
Get(ctx context.Context, key string) (jetstream.KeyValueEntry, error)
Create(ctx context.Context, key string, value []byte, opts ...jetstream.KVCreateOpt) (uint64, error)
Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error)
}
// keepGrants writes every grant the list just sent issued that the bucket does not hold as it is: a create for a new one,
// a compare-and-set for one whose accounts changed. It answers how many it wrote.
func keepGrants(ctx context.Context, kv grantsKV, grants map[string]sdkdelivery.ControllerGrant) (int, error) {
keys := make([]string, 0, len(grants))
for k := range grants {
keys = append(keys, k)
}
sort.Strings(keys)
wrote := 0
var errs []error
for _, key := range keys {
value, err := sdkdelivery.EncodeControllerGrant(grants[key])
if err != nil {
errs = append(errs, fmt.Errorf("%s: %w", key, err))
continue
}
e, err := kv.Get(ctx, key)
switch {
case errors.Is(err, jetstream.ErrKeyNotFound):
_, err = kv.Create(ctx, key, value)
case err != nil:
default:
if before, derr := sdkdelivery.DecodeControllerGrant(e.Value()); derr == nil && sdkdelivery.SameControllerGrant(before, withVersion(grants[key])) {
continue
}
_, err = kv.Update(ctx, key, value, e.Revision())
}
if err != nil {
errs = append(errs, fmt.Errorf("%s: %w", key, err))
continue
}
wrote++
}
return wrote, errors.Join(errs...)
}
func withVersion(g sdkdelivery.ControllerGrant) sdkdelivery.ControllerGrant {
g.V = sdkdelivery.Version
return g
}
// grantsIssued writes the grants the bus's user list just sent to its machine issued (ADR 0528 clause 2). Said, never
// fatal: a grant not written now is written at the next send of the list, and the assigner's wants wait for it.
func grantsIssued(ctx context.Context, inv *inventory.Inventory, js jetstream.JetStream, indent string) {
kv, err := js.KeyValue(ctx, broker.GrantsBucket)
if err != nil {
fmt.Printf("%sthe grants issued are not written: the bucket %s: %v\n", indent, broker.GrantsBucket, err)
return
}
records, err := inv.BusRecords(ctx)
if err != nil {
fmt.Printf("%sthe grants issued are not written: %v\n", indent, err)
return
}
principals, err := broker.Users(records)
if err != nil {
fmt.Printf("%sthe grants issued are not written: %v\n", indent, err)
return
}
kept, err := inv.BusUsers(ctx)
if err != nil {
fmt.Printf("%sthe grants issued are not written: %v\n", indent, err)
return
}
hashes := make(map[string]string, len(kept))
for name, u := range kept {
hashes[name] = u.PasswordHash
}
current, err := currentFingerprints(ctx, inv)
if err != nil {
fmt.Printf("%sthe grants issued are not written: %v\n", indent, err)
return
}
wrote, err := keepGrants(ctx, kv, grantsFrom(principals, hashes, current))
if wrote > 0 {
fmt.Printf("%s%d grant(s) issued written for the assigner\n", indent, wrote)
}
if err != nil {
fmt.Printf("%ssome grants issued are not written: %v\n", indent, err)
}
}
@@ -0,0 +1,91 @@
//go:build integration
package main
import (
"context"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1394/AC1 of C9 (ADR 0528 clause 2): when the controller has issued mesh-x's credential for fingerprint F (mesh-x
// registered at a build of F, its account minted, the bus's machine sent its declaration with the user list, through
// overTheBus.declare as a push sends it), then grants.mesh-x.F exists on the bus,
// and a second pass writes it again as nothing new.
func TestTheGrantOfAnIssuedCredentialIsOnTheBus(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
fp := grantFP('d')
m := catalogue.Manifest{Module: "mesh-x", Version: "1", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/var/lib/mesh-x/broker"}}}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-x1", Repository: "novox/mesh-x", Module: "mesh-x", Commit: "c0ffee1",
SourceFingerprint: fp}); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-x", BuiltFrom: "c0ffee1", Head: "c0ffee1"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-x"); err != nil {
t.Fatal(err)
}
user := broker.Principal{Kind: broker.KindModule, Node: "anchor", Module: "mesh-x"}.Username()
if _, err := inv.MintBusPassword(ctx, inventory.BusUser{Username: user, Kind: inventory.BusModule, Node: "anchor", Module: "mesh-x"}); err != nil {
t.Fatal(err)
}
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
js, err := jetstream.New(conn)
if err != nil {
t.Fatal(err)
}
if err := broker.OnConn(conn).EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
// The live path: a send of the bus's machine's declaration carrying the user list (overTheBus.declare), which records
// the list sent and then writes the grants it issued.
legacy, err := conn.JetStream()
if err != nil {
t.Fatal(err)
}
_ = js.DeleteStream(ctx, "GRANTS_DECLARED")
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "GRANTS_DECLARED", Subjects: []string{link.DeclareSubject("anchor")}}); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.DeleteStream(context.Background(), "GRANTS_DECLARED") })
send := overTheBus{open: open, server: link.SendingOn(link.OverNATS{Conn: conn, JS: legacy}), signer: aSignerOfZeros{}}
ready := readyNode{node: "anchor", declared: sendable{BusUsers: "the user list the bus's machine is sent"}}
if _, err := send.declare(ctx, ready, []byte(`{}`)); err != nil {
t.Fatalf("the send of the bus's machine: %v", err)
}
kv, err := js.KeyValue(ctx, broker.GrantsBucket)
if err != nil {
t.Fatal(err)
}
key, _ := sdkdelivery.ControllerGrantKey("mesh-x", fp)
e, err := kv.Get(ctx, key)
if err != nil {
t.Fatalf("no grant %s after the list was issued: %v", key, err)
}
g, err := sdkdelivery.DecodeControllerGrant(e.Value())
if err != nil || len(g.Accounts) != 1 || g.Accounts[0] != user {
t.Fatalf("the grant reads %+v, %v; want the account %s", g, err, user)
}
if _, err := send.declare(ctx, ready, []byte(`{}`)); err != nil {
t.Fatalf("the second send: %v", err)
}
if again, _ := kv.Get(ctx, key); again.Revision() != e.Revision() {
t.Errorf("a second pass wrote the grant again (revision %d, was %d)", again.Revision(), e.Revision())
}
}
-34
View File
@@ -9,9 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The grants reach the bus before the gate (novox/hq issue 490).
@@ -176,35 +174,3 @@ func TestTheGrantsStepKeepsOneMachinesBuilds(t *testing.T) {
t.Error("the grants step keeps the builds of a machine it does not send")
}
}
// The delivery plan says the step before the merge.
func TestAChangePlanSaysTheGrantsStep(t *testing.T) {
const catalogue_ = "http://forge.internal:20000/novox/mesh-catalog.git"
entry := func(name string, on ...string) inventory.Entry {
e := fromRepo(name, catalogue_, "modules/"+name)
e.Source.BuiltFrom = "old"
e.On = on
return e
}
nats := entry("nats", "novox")
nats.Manifest.BusUsers = "/etc/nats/users.conf"
nats.Manifest.Claims = []catalogue.Claim{{Name: catalogue.BrokerSeat, Scope: catalogue.ScopeMesh}}
restic := entry("restic", "ace", "novox", "shanks")
only := entry("bus-tools", "novox")
entries := []inventory.Entry{nats, restic, only}
rollsOut := func(string) (inventory.Upgrade, bool) { return inventory.Upgrade{RollOut: true}, true }
plan := func(paths ...string) link.ChangePlan {
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "head", Paths: paths}
return changePlanOf("novox/mesh-catalog", "main", "head", reachOfMerge(m, entries, nil, nil), entries, rollsOut)
}
text := planText(plan("modules/restic/module.json"))
if !strings.Contains(text, "grants step: if this changes what the bus's user list says (a new tool, subject or "+
"user), novox is sent that list alone, every build there kept, before restic is judged on its first machine") {
t.Errorf("the change plan does not say the grants step:\n%s", text)
}
// A module only on the bus's machine goes there with the list in its own send: no step of its own.
if text := planText(plan("modules/bus-tools/module.json")); strings.Contains(text, "grants step") {
t.Errorf("a module on the bus's machine alone reads:\n%s", text)
}
}
+106
View File
@@ -0,0 +1,106 @@
package main
import (
"context"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
func grantFP(c byte) string { return sdkdelivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
// 1394/AC (unit half): the list sent issues the credential of every module whose accounts all have a password, at the
// fingerprint its build names; a module with an account the list leaves out, and one registered at a fingerprint not of
// the new form, get no grant; a module with no account of its own is granted with none.
func TestTheGrantsAListIssues(t *testing.T) {
x := broker.Principal{Kind: broker.KindModule, Node: "novox", Module: "mesh-x"}
xp := broker.Principal{Kind: broker.KindProcess, Node: "novox", Module: "mesh-x", Process: "run"}
y := broker.Principal{Kind: broker.KindModule, Node: "ace", Module: "mesh-y"}
node := broker.Principal{Kind: broker.KindNode, Node: "novox"}
hashes := map[string]string{x.Username(): "h1", xp.Username(): "h2", node.Username(): "h3"}
got := grantsFrom([]broker.Principal{x, xp, y, node}, hashes,
map[string]string{"mesh-x": grantFP('a'), "mesh-y": grantFP('b'), "web": grantFP('c'), "old": "src1:abc"})
xKey, _ := sdkdelivery.ControllerGrantKey("mesh-x", grantFP('a'))
webKey, _ := sdkdelivery.ControllerGrantKey("web", grantFP('c'))
if len(got) != 2 {
t.Fatalf("grants = %v, want mesh-x's and web's", got)
}
if g := got[xKey]; len(g.Accounts) != 2 || g.Accounts[0] != x.Username() && g.Accounts[1] != x.Username() {
t.Errorf("mesh-x's grant = %+v, want both its accounts", g)
}
if g, ok := got[webKey]; !ok || len(g.Accounts) != 0 {
t.Errorf("web, with no account of its own, is granted %+v, %v", g, ok)
}
}
// fakeGrants is the grants bucket in memory.
type fakeGrants struct {
values map[string][]byte
creates int
updates int
}
type fakeEntry struct {
key string
value []byte
rev uint64
}
func (e fakeEntry) Bucket() string { return broker.GrantsBucket }
func (e fakeEntry) Key() string { return e.key }
func (e fakeEntry) Value() []byte { return e.value }
func (e fakeEntry) Revision() uint64 { return e.rev }
func (e fakeEntry) Created() time.Time { return time.Time{} }
func (e fakeEntry) Delta() uint64 { return 0 }
func (e fakeEntry) Operation() jetstream.KeyValueOp { return jetstream.KeyValuePut }
func (f *fakeGrants) Get(_ context.Context, key string) (jetstream.KeyValueEntry, error) {
v, ok := f.values[key]
if !ok {
return nil, jetstream.ErrKeyNotFound
}
return fakeEntry{key: key, value: v, rev: 1}, nil
}
func (f *fakeGrants) Create(_ context.Context, key string, value []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
if _, ok := f.values[key]; ok {
return 0, jetstream.ErrKeyExists
}
f.values[key] = value
f.creates++
return 1, nil
}
func (f *fakeGrants) Update(_ context.Context, key string, value []byte, _ uint64) (uint64, error) {
f.values[key] = value
f.updates++
return 2, nil
}
// 1394/AC (unit half): a grant is created once; a second send of the same list writes nothing; a reissue for other
// accounts is a new revision of the same key.
func TestAGrantIsWrittenOnceAndMovedOnAReissue(t *testing.T) {
kv := &fakeGrants{values: map[string][]byte{}}
key, _ := sdkdelivery.ControllerGrantKey("mesh-x", grantFP('a'))
grants := map[string]sdkdelivery.ControllerGrant{key: {Accounts: []string{"novox.mesh-x"}}}
for range 2 {
if _, err := keepGrants(t.Context(), kv, grants); err != nil {
t.Fatal(err)
}
}
if kv.creates != 1 || kv.updates != 0 {
t.Errorf("creates %d, updates %d, want one create", kv.creates, kv.updates)
}
grants[key] = sdkdelivery.ControllerGrant{Accounts: []string{"ace.mesh-x"}}
if wrote, err := keepGrants(t.Context(), kv, grants); err != nil || wrote != 1 || kv.updates != 1 {
t.Errorf("a reissue wrote %d (%v), updates %d", wrote, err, kv.updates)
}
if g, err := sdkdelivery.DecodeControllerGrant(kv.values[key]); err != nil || g.Accounts[0] != "ace.mesh-x" {
t.Errorf("the grant reads %+v, %v", g, err)
}
}
+3
View File
@@ -64,6 +64,9 @@ var handActVerbs = []handActVerb{
{Verb: "plans stop", Decision: "the operator's answer to an ask is their decision, not a repair (ADR 0259)",
DecidedFor: []string{conditions.CauseOperatorAnswer}},
{Verb: "plans close"},
// A module registered at references by hand, because the builder had no key for it (ADR 0526 clause 3): a repair
// the builder should have made, so it counts.
{Verb: "register"},
// A walk started by a person instead of its delivery's owner (novox/hq ADR 0239): the owner down, or
// not trusted with it — either is a repair the owner should have made. Unless the operator chose it on
// a warrant (ADR 0259).
+198
View File
@@ -0,0 +1,198 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// A move held for a person is asked of the person (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
//
// **A wait only a person can end is said to that person at once.** ADR 0377 holds a recorded module's move that
// recreates a container or touches its data until a person pushes the machine. The walk said so in its note and
// ended "done", `status` named the machine behind, and nobody was asked: on 2026-10-21 mailu's new build waited on
// novox for 40 minutes behind a plan that read "done, 3 tier(s)", and on 2026-10-19 mssql waited on ace the same way.
// Row S24 raises one condition per machine that has such a move, for the operator, with the answer **Release**: the
// push of that machine, which the controller's own `push` verb performs on the operator's warrant (ADR 0259). It ends
// when the machine no longer waits, whoever pushed it.
//
// A hold a person chose is not asked again — a module they set to `record` with a why, and the bus, whose step has
// its own row (S17).
// kindMovesHeld is S24's kind: machine.<node>.moves-held.
const kindMovesHeld = "moves-held"
// heldFacts is one machine and the moves of recorded modules that wait there for a person.
type heldFacts struct {
node string
moves []heldMove
}
// gatherHeldMoves is, per machine that holds at least one move no person chose to hold, all its held moves, by machine.
func gatherHeldMoves(ctx context.Context, inv *inventory.Inventory) ([]heldFacts, error) {
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, fmt.Errorf("the machines cannot be read: %w", err)
}
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
}
var out []heldFacts
for _, n := range nodes {
moves, err := heldMovesOn(ctx, inv, f, n.Name)
if err != nil {
return nil, err
}
h := heldFacts{node: n.Name, moves: moves}
if len(h.asked()) > 0 {
out = append(out, h)
}
}
return out, nil
}
// asked are the moves no person chose to hold, and chosen the ones a person did (a module set to record with a why, the bus).
func (h heldFacts) asked() []heldMove { return h.byChoice(false) }
func (h heldFacts) chosen() []heldMove { return h.byChoice(true) }
func (h heldFacts) byChoice(chosen bool) []heldMove {
var out []heldMove
for _, m := range h.moves {
if m.Chosen == chosen {
out = append(out, m)
}
}
return out
}
// modules are the held moves' modules, in order, of all of them.
func (h heldFacts) modules() []string { return moduleNames(h.moves) }
func moduleNames(moves []heldMove) []string {
out := make([]string, 0, len(moves))
for _, m := range moves {
out = append(out, m.Module)
}
return out
}
// releaseAction is the one answer a held move offers: the push of the machine, approved by the operator. The push
// names every module whose move waits there as one that may move its data: the operator's Approve is the word
// ADR 0217 asks for, and the ask says so. Offered only where no person's own hold is on the machine too: a push
// composes every recorded module at its current build, so it would send that move as well, which the ask never named
// (ADR 0259 §6), and hold the whole machine if that module's data would move.
func (h heldFacts) releaseAction() conditions.Action {
return conditions.Action{Label: "Release", Verb: "mesh-controller.push", Level: conditions.LevelApprove,
Arguments: map[string]string{"node": h.node, "move": strings.Join(moduleNames(h.asked()), ","), "why": "",
"cause": conditions.CauseOperatorAnswer}}
}
// heldWords are the plain words of a machine's held moves: which modules wait, why in general, and what an Approve does.
func heldWords(h heldFacts) (headline, explanation, needs string, resolved string) {
names := namesWords(moduleNames(h.asked()), 3)
headline = clipWords(conditions.Capital(names)+" waits for you on "+h.node, conditions.HeadlineMax)
recreates, data := false, false
for _, m := range h.asked() {
switch {
case strings.Contains(m.Why, "recreates"):
recreates = true
default:
data = true
}
}
why := "would change what it runs"
switch {
case recreates && data:
why = "would restart a container or touch its data"
case recreates:
why = "would restart one of its containers"
case data:
why = "would touch its files or data"
}
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. Release sends %s everything it is owed, these among it, and lets their data move where it "+
"has to. Nothing is sent until you answer.", names, why, h.node, h.node), conditions.ExplanationMax)
needs = "release them " + FromMeshMCPServer
if chosen := h.chosen(); len(chosen) > 0 {
// A push of the machine would send the move a person chose to hold as well: no answer is offered here.
explanation = clipWords(fmt.Sprintf("The new version of %s %s on %s, so it does not go out with its delivery; "+
"it waits for you. %s is also held there by your own choice, and a push of %s would send that too, so "+
"nothing is offered here: the push is yours at the controller's terminal.",
names, why, h.node, conditions.Capital(namesWords(moduleNames(chosen), 3)), h.node), conditions.ExplanationMax)
needs = fmt.Sprintf("push %s at the controller's terminal, once you have decided about %s.", h.node,
namesWords(moduleNames(chosen), 3))
}
resolved = "Resolved: " + names + " is no longer waiting on " + h.node
return
}
// watchMovesHeld is S24: a machine that has moves of recorded modules waiting for a person, said to the operator with the
// answer that sends them. The condition ends when the machine no longer waits: it is no longer among the facts.
func watchMovesHeld(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, h := range f.held {
var said []string
for _, m := range h.moves {
said = append(said, fmt.Sprintf("%s %s → %s %s", m.Module, short(m.From), short(m.To), m.Why))
}
headline, explanation, needs, resolved := heldWords(h)
if c := h.chosen(); len(c) > 0 {
said = append(said, "held by a person's own choice: "+strings.Join(moduleNames(c), ", "))
}
var actions []conditions.Action
if len(h.chosen()) == 0 {
actions = []conditions.Action{h.releaseAction()}
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: h.node, Kind: kindMovesHeld,
Machine: h.node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s has %d move(s) of recorded modules waiting for a person's push: %s — `push %s` sends them",
h.node, len(h.moves), strings.Join(h.modules(), ", "), h.node),
Said: strings.Join(said, "; "),
Headline: headline, Explanation: explanation, Needs: needs, Resolved: resolved,
Actions: actions})
}
return out
}
// planHeld is, for a plan that ended or is running, the moves its modules carried that a machine still holds for a
// person: "module on machine", sorted. The plan's own record says the move waited when it was judged
// (PlanModule.Recorded); the machines say whether it still does, and for the build this plan carried, so a plan older
// than the build now held is not named.
func planHeld(p inventory.Plan, held []heldFacts) []string {
var out []string
for _, h := range held {
for _, m := range h.moves {
s := p.Modules[m.Module]
if s == nil || !strings.HasPrefix(s.Recorded[h.node], "waits:") || !sameCommit(s.Commit, m.To) {
continue
}
out = append(out, m.Module+" on "+h.node)
}
}
sort.Strings(out)
return out
}
// heldSaid is what a plan's line adds when it carries a move that a machine holds for a person (novox/hq issue 1732): a
// plan whose module waits for a push is not plainly "done".
func heldSaid(p inventory.Plan, held []heldFacts) string {
waiting := planHeld(p, held)
if len(waiting) == 0 {
return ""
}
return fmt.Sprintf(" — %d held for a person: %s (the operator is asked)", len(waiting), strings.Join(waiting, ", "))
}
// heldNow reads the machines' held moves for a plan's line, nil when they cannot be read: the line then reads as before.
func heldNow(ctx context.Context, inv *inventory.Inventory) []heldFacts {
held, err := gatherHeldMoves(ctx, inv)
if err != nil {
return nil // empty-on-error: a plan's line then reads as it did before; the watchdog (S24) says it cannot read
}
return held
}
@@ -0,0 +1,92 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func inventoryUpgrade(why string) inventory.Upgrade { return inventory.Upgrade{Why: why, By: "jochen"} }
func mustMoveFacts(t *testing.T, rm *recordedMesh) moveFacts {
t.Helper()
f, err := readMoveFacts(t.Context(), rm.open.inventory)
if err != nil {
t.Fatal(err)
}
return f
}
// 1732/AC8 (integration): a mesh running mailu at c1 on two machines, whose registered c2 recreates a container, is read
// as two machines each holding mailu's move for a person; a bundle-only c3 on top changes nothing of it; after a machine
// is sent c2 (as a push records it) it is no longer among them, and plans that carried c2 say nothing more of it.
func TestMachinesHoldingAMoveForAPersonAreReadFromTheRecords(t *testing.T) {
rm := aRecordedMesh(t, aMailu(true))
c2 := aMailu(true)
resourceOf(&c2, "imap")["image"] = "registry.invalid/imap@sha256:2"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
ctx := t.Context()
held, err := gatherHeldMoves(ctx, rm.open.inventory)
if err != nil {
t.Fatal(err)
}
if len(held) != 2 || held[0].node != "anchor" || held[1].node != "laptop" || len(held[0].moves) != 1 ||
held[0].moves[0].Module != "mailu" || held[0].moves[0].To != "c2" || !strings.Contains(held[0].moves[0].Why, "recreates") {
t.Fatalf("held %+v", held)
}
if err := rm.open.inventory.RecordSent(ctx, nodeID(t, rm.open, "anchor"), "d-anchor-c2", map[string]string{"mailu": "c2"}); err != nil {
t.Fatal(err)
}
held, err = gatherHeldMoves(ctx, rm.open.inventory)
if err != nil || len(held) != 1 || held[0].node != "laptop" {
t.Fatalf("after anchor was sent c2: %+v (%v)", held, err)
}
}
// 1732/AC9 (integration): a module a person set to record is no ask: they chose the hold.
func TestAHoldAPersonChoseIsNotAsked(t *testing.T) {
rm := aRecordedMesh(t, aMailu(false))
c2 := aMailu(false)
c2.Bundles[0].Digest = "sha256:bbb"
rm.register(t, c2, "c2", time.Now().Add(-time.Minute))
if err := rm.open.inventory.SetUpgradeOf(t.Context(), "mailu", inventoryUpgrade("the migration week")); err != nil {
t.Fatal(err)
}
held, err := gatherHeldMoves(t.Context(), rm.open.inventory)
if err != nil || len(held) != 0 {
t.Fatalf("a machine holding only a person's own hold is asked: %+v (%v)", held, err)
}
if moves, _ := heldMovesOn(t.Context(), rm.open.inventory, mustMoveFacts(t, rm), "anchor"); len(moves) != 1 || !moves[0].Chosen {
t.Fatalf("status still names it, marked as chosen: %+v", moves)
}
}
// 1742/AC4 (integration): a registered build naming an image of the store that the store lacks is read as a missing
// image on each machine that runs the module; one the store holds is not; and what the store holds is not asked twice.
func TestARegisteredBuildNamingAnImageTheStoreLacksIsRead(t *testing.T) {
m := aMailu(true)
resourceOf(&m, "imap")["image"] = "artifact-store://mailu/imap@sha256:" + strings.Repeat("e", 64)
rm := aRecordedMesh(t, m)
asked := 0
lacking := func(_ context.Context, reference string) (bool, error) { asked++; return false, nil }
imagesHeld.Lock()
imagesHeld.held = map[string]bool{}
imagesHeld.Unlock()
got, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking)
if err != nil || len(got) != 2 || got[0].Module != "mailu" || got[0].Node != "anchor" || got[1].Node != "laptop" {
t.Fatalf("missing %+v (%v)", got, err)
}
holding := func(_ context.Context, reference string) (bool, error) { asked++; return true, nil }
if got, err := gatherMissingImages(t.Context(), rm.open.inventory, holding); err != nil || len(got) != 0 {
t.Fatalf("an image the store holds is missing: %+v (%v)", got, err)
}
before := asked
if _, err := gatherMissingImages(t.Context(), rm.open.inventory, lacking); err != nil || asked != before {
t.Fatalf("an image the store was found to hold was asked about again (%d, %v)", asked-before, err)
}
}
+237
View File
@@ -0,0 +1,237 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A move held for a person is asked of the person** (novox/hq issue 1732, issue 1320; ADR 0377, ADR 0259).
// aHeldMailu is novox holding mailu's new build because it would recreate a container: the walk's own words.
func aHeldMailu() heldFacts {
return heldFacts{node: "novox", moves: []heldMove{{Module: "mailu", From: "b2f5ebb2aaaa", To: "3259765dbbbb",
Why: "waits: recreates 1 of mailu's 12 container(s) (with a new image: automx)"}}}
}
func heldConditionOn(f *signalFacts) conditions.Condition {
o := watchMovesHeld(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1732/AC1: a machine holding a move of a recorded module for a person raises one condition for the operator, with
// Release as its only answer: the machine's push, approved, naming the modules whose moves wait there.
func TestAHeldMoveIsRaisedWithReleaseAsItsOnlyAction(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
got := watchMovesHeld(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "moves-held" || o.Scope != conditions.ScopeMachine || o.ID != "novox" || o.Severity != conditions.Warning ||
o.Resolver != conditions.ResolverOperator || o.Key() != "machine.novox.moves-held" {
t.Fatalf("kind, scope, subject, severity or resolver: %+v (%s)", o, o.Key())
}
if len(o.Actions) != 1 {
t.Fatalf("actions: %+v", o.Actions)
}
a := o.Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.push" || a.Level != conditions.LevelApprove ||
a.Arguments["node"] != "novox" || a.Arguments["move"] != "mailu" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action is not the push of novox: %+v", a)
}
if _, takes := a.Arguments["why"]; !takes {
t.Fatal("the push takes a why, which the asker fills in with the operator's answer")
}
for _, want := range []string{"mailu", "novox"} {
if !strings.Contains(o.Headline+o.Explanation, want) {
t.Errorf("the words do not name %q: %q / %q", want, o.Headline, o.Explanation)
}
}
if !strings.Contains(o.Said, "recreates 1 of mailu's 12") {
t.Errorf("the evidence does not carry the walk's own words: %q", o.Said)
}
if w := (conditions.Observation{Headline: o.Headline, Explanation: o.Explanation, Needs: o.Needs, Resolved: o.Resolved}); w.Headline == "" || len(o.Headline) > conditions.HeadlineMax {
t.Errorf("headline: %q", o.Headline)
}
}
// 1732/AC2: a machine with nothing held raises nothing, so the condition ends the look after the move is sent.
func TestAMachineHoldingNothingRaisesNoHeldMoveCondition(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
if got := watchMovesHeld(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
}
// 1732/AC3: the condition is asked on a channel at the approve level in words that say what Release does, and the
// operator's warrant calls the controller's own push with the machine and the modules, once.
func TestAHeldMovesReleaseIsAskedAndAWarrantPushesTheMachine(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
f.held = []heldFacts{aHeldMailu()}
c := heldConditionOn(f)
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Release" || o.Level != asks.Approve || o.Does != "the held builds are sent to novox" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.push@ ") ||
!strings.Contains(r.called[0], `"node":"novox"`) || !strings.Contains(r.called[0], `"move":"mailu"`) ||
!strings.Contains(r.called[0], `"cause":"`+conditions.CauseOperatorAnswer+`"`) ||
!strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC4: the controller may call its own push and upgrade verbs on a warrant: the grant names them.
func TestTheControllerMayActOnAWarrantForPushAndUpgrade(t *testing.T) {
for _, verb := range []string{"push", "upgrade"} {
granted := false
for _, v := range broker.VerbsTheControllerActsOnAWarrant {
granted = granted || (v.Seat == broker.ControllerSeat && v.Verb == verb)
}
if !granted {
t.Errorf("the controller may not act on a warrant for %s", verb)
}
}
}
func aDoneMailuPlan() inventory.Plan {
return inventory.Plan{ID: "plan-1792585970446304628", Repository: "novox/mesh-catalog", State: inventory.PlanDone,
Tiers: [][]string{{"a"}, {"b"}, {"mailu"}},
Modules: map[string]*inventory.PlanModule{"mailu": {State: "built", Commit: "3259765dbbbbcccc",
Recorded: map[string]string{"novox": "waits: recreates 1 of mailu's 12 container(s)"}}}}
}
// 1732/AC5: a done plan that carries a move a machine still holds for a person says so on its line; once the machine
// no longer holds it, or holds another build of the module, the line says nothing more.
func TestADonePlanSaysWhatItHoldsForAPerson(t *testing.T) {
p := aDoneMailuPlan()
line := planLine(p, time.Date(2026, 10, 21, 16, 0, 0, 0, time.UTC)) + heldSaid(p, []heldFacts{aHeldMailu()})
if !strings.Contains(line, "done, 3 tier(s)") || !strings.Contains(line, "1 held for a person: mailu on novox") {
t.Errorf("the line: %q", line)
}
if said := heldSaid(p, nil); said != "" {
t.Errorf("a plan whose moves were sent says %q", said)
}
other := aHeldMailu()
other.moves[0].To = "9999999ffff"
if said := heldSaid(p, []heldFacts{other}); said != "" {
t.Errorf("a plan older than the build now held says %q", said)
}
p.Modules["mailu"].Recorded = map[string]string{"novox": recordedRollsWhy}
if said := heldSaid(p, []heldFacts{aHeldMailu()}); said != "" {
t.Errorf("a move the plan sent says %q", said)
}
}
// 1732/AC6: the builds held after a failed release are asked of the operator: the held-release condition offers
// Release, which calls the controller's own `upgrade` with release-backlog and nothing else, once, on a warrant.
func TestTheBuildsHeldAfterAFailedReleaseAreAskedAndReleasedOnAWarrant(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
backlogNow.held = "release-1792595552499982311 failed (failed on novox); what waits is released again by a person"
backlogNow.waiting = map[string][]inventory.CarriedMove{"novox": {{Module: "indexer"}, {Module: "nextcloud"}}}
got := backlogObservation()
if len(got) != 1 || len(got[0].Actions) != 1 {
t.Fatalf("raised %+v", got)
}
a := got[0].Actions[0]
if a.Label != "Release" || a.Verb != "mesh-controller.upgrade" || a.Level != conditions.LevelApprove ||
a.Arguments["release-backlog"] != "true" || a.Arguments["cause"] != conditions.CauseOperatorAnswer {
t.Fatalf("the action: %+v", a)
}
o := got[0]
c := conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
r := newAskerRig(t)
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 || sent[0].Options[0].Does != "the builds held after the failed delivery are released" {
t.Fatalf("asked %+v", sent)
}
w := r.warrantFor(t, c.Key, "Release")
answerWith(t, r, w)
answerWith(t, r, w)
if len(r.called) != 1 || !strings.HasPrefix(r.called[0], "mesh-controller.upgrade@ ") ||
!strings.Contains(r.called[0], `"release-backlog":"true"`) || !strings.Contains(r.called[0], "chose Release") {
t.Fatalf("called %v", r.called)
}
}
// 1732/AC7: a plan whose send waits for a gate while the builds are held after a failed release reads as waiting for
// a person to release them, not "building ... LATE"; with nothing held it still reads as the wait it is.
func TestAPlanWaitingForAGateBehindAFailedReleaseIsNotLate(t *testing.T) {
saved := backlogNow
t.Cleanup(func() { backlogNow = saved })
now := time.Date(2026, 10, 21, 18, 50, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Repository: "novox/mesh-sdk", Commit: "c0ffee00", State: inventory.PlanBuilding,
Tiers: [][]string{{"indexer"}}, TierEntered: now.Add(-100 * time.Minute), Created: now.Add(-110 * time.Minute),
Modules: map[string]*inventory.PlanModule{"indexer": {State: "built"}},
Note: refusalNote(0, errors.New("a build waits there for its gate: novox"))}
backlogNow.held = ""
if line := planLine(p, now); !strings.Contains(line, "a build waits there for its gate: novox") {
t.Errorf("the wait is not said: %q", line)
}
backlogNow.held = "release-1 failed (failed on novox); what waits is released again by a person"
line := planLine(p, now)
if strings.Contains(line, "LATE") || strings.Contains(line, "building") ||
!strings.Contains(line, "until a person releases it") {
t.Errorf("a plan held behind a failed release reads %q", line)
}
}
// 1732/AC10: a machine that also holds a move a person chose to hold gets no Release, since a push would send that move
// too; the condition names both and says the push is the controller's terminal. A machine with only asked moves keeps it.
func TestAMachineAlsoHoldingAPersonsOwnHoldIsOfferedNoRelease(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 30, 0, 0, time.UTC))
h := aHeldMailu()
h.moves = append(h.moves, heldMove{Module: "nftables", From: "n1aaaa", To: "n2bbbb",
Why: "waits: a person set it to record, and holds every move of it until they push", Chosen: true})
f.held = []heldFacts{h}
got := watchMovesHeld(f)
if len(got) != 1 || len(got[0].Actions) != 0 {
t.Fatalf("%+v", got)
}
o := got[0]
text := o.Headline + o.Explanation + o.Needs + o.Summary + o.Said
for _, want := range []string{"mailu", "nftables", "terminal", "push novox"} {
if !strings.Contains(text, want) {
t.Errorf("does not say %q: %q", want, text)
}
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
f.held = []heldFacts{aHeldMailu()}
if got := watchMovesHeld(f); len(got) != 1 || len(got[0].Actions) != 1 || got[0].Actions[0].Arguments["move"] != "mailu" {
t.Fatalf("a machine with only asked moves: %+v", got)
}
}
+245
View File
@@ -0,0 +1,245 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/artifacts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// No declaration is sent that names an image the artifact store does not hold (novox/hq issue 1742).
//
// On 2026-10-21 at 17:13 a send to novox named `mailu/automx@sha256:c62150b8…`; the build record keeps that digest
// ("recent-build") and the store did not hold it. The node-engine removed the running container, then could not pull
// the replacement, and every later apply on novox failed for the next hour while the mail clients' autoconfiguration
// had no backend. The node-engine's half (pull first, replace after) is its own change; this is the controller's: **a
// module whose image the store lacks is left out of the declaration**, as a module whose setting nobody gave is
// (ADR 0163, rule 6) — the node-engine keeps what it has of it and touches none of its containers — and **it is said**,
// row S25: a condition naming the module, the machine and the image, until the store holds the image again (a new build
// of the module, or the copy put back).
//
// Judged only for an image the store's own address names, from the send's own resources; an image of another registry
// is no business of the store's. A store that cannot be asked judges nothing: a send is not refused for a question
// that did not answer.
// kindImageMissing is S25's kind: module.<module>.<machine>.image-missing.
const kindImageMissing = "image-missing"
// missingImage is a module held out of a machine's declaration because the store lacks an image it names.
type missingImage struct {
Node, Module, Image string
}
// storeHolds is whether the store holds the manifest an image reference at its address names.
type storeHolds func(ctx context.Context, reference string) (bool, error)
// imageInStore is the artifact-store reference of an image the store's address names, "" for any other.
func imageInStore(image, address string) string {
if address == "" || !strings.HasPrefix(image, address+"/") || !strings.Contains(image, "@sha256:") {
return ""
}
reference := catalogue.Recorded(image)
if _, ours := catalogue.InArtifactStore(reference); !ours {
return ""
}
return reference
}
// holdBackMissingImages leaves out of the declaration every module that names an image the store does not hold,
// and answers what it left out. The module's resources go, it joins LeftOut with the reason, and its build stays the
// one the machine was last sent (`was`).
func holdBackMissingImages(ctx context.Context, node string, s *sendable, address string, holds storeHolds,
was map[string]string) []missingImage {
lacks := map[string]bool{} // reference → the store does not hold it; absent: not asked, or no answer
asked := map[string]bool{}
lacking := map[string]string{} // module → the image
for _, r := range s.Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
reference := imageInStore(image, address)
module := s.moduleOf[fmt.Sprint(r["id"])]
if reference == "" || module == "" || lacking[module] != "" {
continue
}
if !asked[reference] {
asked[reference] = true
held, err := holds(ctx, reference)
// empty-on-error: the store did not answer, so nothing is held back for a question unanswered
lacks[reference] = err == nil && !held
}
if lacks[reference] {
lacking[module] = image
}
}
if len(lacking) == 0 {
return nil
}
kept := s.Resources[:0:0]
for _, r := range s.Resources {
if _, out := lacking[s.moduleOf[fmt.Sprint(r["id"])]]; !out {
kept = append(kept, r)
}
}
s.Resources = kept
if s.leftOutWhy == nil {
s.leftOutWhy = map[string]string{}
}
var found []missingImage
for module, image := range lacking {
s.leftOutWhy[module] = "its image " + image + " is not in the artifact store"
if !slices.Contains(s.LeftOut, module) {
s.LeftOut = append(s.LeftOut, module)
}
if s.Builds != nil {
if prior, sent := was[module]; sent {
s.Builds[module] = prior
} else {
delete(s.Builds, module)
}
}
found = append(found, missingImage{Node: node, Module: module, Image: image})
}
sort.Strings(s.LeftOut)
sort.Slice(found, func(i, j int) bool { return found[i].Module < found[j].Module })
return found
}
// withoutMissingImages is composeSend's last step: the store asked about the images the declaration names, the modules
// whose image it lacks left out and kept in the ledger S25 reads. A store that cannot be found or asked leaves the
// declaration as composed.
func withoutMissingImages(ctx context.Context, open *stores, node string, s sendable) sendable {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
if err != nil {
return s
}
address, err := artifactStoreAddress(ctx, inv, shelf, node)
if err != nil || address == "" {
return s
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
sent = nil
}
store := artifacts.Store{Address: address}
found := holdBackMissingImages(ctx, node, &s, address, store.HoldsManifest, sent)
for _, m := range found {
fmt.Printf("%s: %s is left out of the declaration: its image %s is not in the artifact store\n", node, m.Module, m.Image)
}
return s
}
// imageFacts are the images S25 raises: a module whose registered build names an image the store lacks, on each machine
// that runs it.
type imageFacts struct {
missingImage
}
// imagesHeld remembers the images the store was found to hold: a stored image is named by its digest and stays what it
// is, so it is asked about once; one it lacks is asked again at every look.
var imagesHeld = struct {
sync.Mutex
held map[string]bool
}{held: map[string]bool{}}
// gatherMissingImages is every machine's module whose registered build — the one a send would carry — names an image of
// the artifact store that the store does not hold. Read from the catalogue and the store, never from what a send did:
// a push is a command of its own, and the serving controller's watchdog hears nothing of it.
func gatherMissingImages(ctx context.Context, inv *inventory.Inventory, holds storeHolds) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
var out []imageFacts
for _, module := range sortedKeysOf(modulesOf(shelf)) {
for _, r := range shelf[module].Resources {
if fmt.Sprint(r["type"]) != "container" {
continue
}
image, _ := r["image"].(string)
if _, ours := catalogue.InArtifactStore(image); !ours || !strings.Contains(image, "@sha256:") {
continue
}
imagesHeld.Lock()
known := imagesHeld.held[image]
imagesHeld.Unlock()
if !known {
held, err := holds(ctx, image)
if err != nil {
return nil, fmt.Errorf("the artifact store cannot be asked about %s: %w", image, err)
}
if held {
imagesHeld.Lock()
imagesHeld.held[image] = true
imagesHeld.Unlock()
continue
}
} else {
continue
}
running, err := inv.Running(ctx, module)
if err != nil {
return nil, err
}
for _, node := range running {
out = append(out, imageFacts{missingImage{Node: node, Module: module, Image: image}})
}
}
}
return out, nil
}
// modulesOf is the shelf's modules as a map of names, for sortedKeysOf.
func modulesOf(shelf map[string]catalogue.Manifest) map[string]string {
out := make(map[string]string, len(shelf))
for m := range shelf {
out[m] = m
}
return out
}
// watchMissingImages is S25: a module left out of a machine's declaration because the store lacks its image, said at
// once. The condition ends when the store holds the image, or a send names another.
func watchMissingImages(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, m := range f.images {
out = append(out, conditions.Observation{Scope: conditions.ScopeModule, ID: m.Module + "." + m.Node, Kind: kindImageMissing,
Machine: m.Node, Severity: conditions.Warning, Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("%s is left out of every send to %s: its declaration names an image that the artifact store does "+
"not hold (the evidence names it); the running container is untouched", m.Module, m.Node),
Said: "image " + m.Image + " is not in the store",
Headline: clipWords(conditions.Capital(m.Module)+" is not updated on "+m.Node, conditions.HeadlineMax),
Explanation: clipWords(fmt.Sprintf("The new version of %s names a container image that the mesh's store does not "+
"hold, so it was not sent to %s. What runs there is untouched, and the rest of %s is sent as usual.",
m.Module, m.Node, m.Node), conditions.ExplanationMax),
Needs: fmt.Sprintf("build %s again; the details name the image.", m.Module),
Resolved: "Resolved: the store holds the image of " + m.Module + " again"})
}
return out
}
// gatherMissingImagesHere is gatherMissingImages asking the mesh's own artifact store; nothing to ask where the mesh has
// none on its network yet.
func gatherMissingImagesHere(ctx context.Context, inv *inventory.Inventory) ([]imageFacts, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, fmt.Errorf("the catalogue cannot be read: %w", err)
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, nil
}
return gatherMissingImages(ctx, inv, artifacts.Store{Address: address}.HoldsManifest)
}
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
)
// **No declaration is sent that names an image the artifact store does not hold** (novox/hq issue 1742).
const storeAt = "novox.internal:5100"
// automx is the image the send of 2026-10-21 17:13 named and the store lacked; the others are held.
const (
automxImage = storeAt + "/mailu/automx@sha256:c62150b8e2cbcc18cb911b098c2877f928761a1e6251667f39ea18251ea35591"
imapImage = storeAt + "/mailu/imap@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
)
func aMailSend() sendable {
return sendable{
Resources: []map[string]any{
{"id": "mailu.automx", "type": "container", "image": automxImage},
{"id": "mailu.automx-config", "type": "file", "path": "/etc/automx.conf"},
{"id": "mailu.imap", "type": "container", "image": imapImage},
{"id": "other.web", "type": "container", "image": "docker.io/library/nginx@sha256:" + strings.Repeat("b", 64)},
{"id": "dns.dnsmasq", "type": "container", "image": storeAt + "/dnsmasq/main@sha256:" + strings.Repeat("c", 64)},
},
moduleOf: map[string]string{"mailu.automx": "automx", "mailu.automx-config": "automx", "mailu.imap": "mailu",
"other.web": "other", "dns.dnsmasq": "dnsmasq"},
Builds: map[string]string{"automx": "b2f5ebb2new", "mailu": "m2", "other": "o1", "dnsmasq": "d1"},
}
}
func storeHolding(missing ...string) (storeHolds, *[]string) {
var asked []string
return func(_ context.Context, reference string) (bool, error) {
asked = append(asked, reference)
for _, m := range missing {
if strings.Contains(reference, m) {
return false, nil
}
}
return true, nil
}, &asked
}
// 1742/AC1: given a send naming automx's image, which the store does not hold, when the declaration is judged, then
// every resource of automx is out of it, automx is among the modules left out with the reason, its build is the one the
// machine was last sent, and every other module — an image of another registry among them — is as composed.
func TestAModuleWhoseImageTheStoreLacksIsLeftOutOfTheSend(t *testing.T) {
s := aMailSend()
holds, asked := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, map[string]string{"automx": "b2f5ebb2old"})
if len(found) != 1 || found[0].Module != "automx" || found[0].Node != "novox" || found[0].Image != automxImage {
t.Fatalf("found %+v", found)
}
var ids []string
for _, r := range s.Resources {
ids = append(ids, r["id"].(string))
}
if got := strings.Join(ids, ","); got != "mailu.imap,other.web,dns.dnsmasq" {
t.Errorf("the declaration keeps %s", got)
}
if len(s.LeftOut) != 1 || s.LeftOut[0] != "automx" || !strings.Contains(s.leftOutWhy["automx"], "not in the artifact store") {
t.Errorf("left out %v because %v", s.LeftOut, s.leftOutWhy)
}
if s.Builds["automx"] != "b2f5ebb2old" || s.Builds["mailu"] != "m2" || s.Builds["other"] != "o1" {
t.Errorf("builds %v", s.Builds)
}
for _, a := range *asked {
if strings.Contains(a, "docker.io") {
t.Errorf("the store was asked about an image of another registry: %s", a)
}
}
}
// 1742/AC2: given a module never sent to the machine, when its image is missing, then it has no build in the record of the
// send; given the store does not answer, then nothing is held back for a question unanswered.
func TestAnUnansweredStoreHoldsNothingBackAndANewModuleHasNoBuild(t *testing.T) {
s := aMailSend()
holds, _ := storeHolding("mailu/automx@")
found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, holds, nil)
if len(found) != 1 {
t.Fatalf("%+v", found)
}
if _, has := s.Builds["automx"]; has {
t.Errorf("a module never sent is recorded as sent: %v", s.Builds)
}
s = aMailSend()
silent := func(context.Context, string) (bool, error) { return false, errors.New("registry: connection refused") }
if found := holdBackMissingImages(t.Context(), "novox", &s, storeAt, silent, nil); len(found) != 0 || len(s.LeftOut) != 0 ||
len(s.Resources) != 5 {
t.Errorf("an unanswered store held back %+v, left out %v", found, s.LeftOut)
}
}
// 1742/AC3: a module left out for its image raises a condition for the operator naming the module, the machine and the
// image in the evidence, plain words, and no answer to give; a mesh whose images are all held raises nothing.
func TestAMissingImageIsRaisedForTheOperator(t *testing.T) {
f := calm(time.Date(2026, 10, 21, 17, 40, 0, 0, time.UTC))
if got := watchMissingImages(f); len(got) != 0 {
t.Fatalf("%+v", got)
}
f.images = []imageFacts{{missingImage{Node: "novox", Module: "automx", Image: automxImage}}}
got := watchMissingImages(f)
if len(got) != 1 {
t.Fatalf("%+v", got)
}
o := got[0]
if o.Kind != "image-missing" || o.Key() != "module.automx.novox.image-missing" || o.Resolver != conditions.ResolverOperator ||
o.Severity != conditions.Warning || o.Machine != "novox" {
t.Fatalf("%+v (%s)", o, o.Key())
}
if !strings.Contains(o.Said, "c62150b8") || !strings.Contains(o.Headline+o.Explanation, "automx") {
t.Errorf("words: %q / %q / %q", o.Headline, o.Explanation, o.Said)
}
w := conditions.Words{Headline: o.Headline, Explanation: o.Explanation, Resolved: o.Resolved, Needs: o.Needs}
if why, ok := conditions.PlainWords(w, "novox"); !ok {
t.Errorf("not plain: %s", why)
}
}

Some files were not shown because too many files have changed in this diff Show More