Compare commits

..
Author SHA1 Message Date
jschoubben 604ccd10d4 Merge remote-tracking branch 'origin/main' into feat/try-speed 2026-10-25 03:17:47 +01:00
jschoubben 68f7431d98 Merge pull request 'The provisioner client fills every module again when the socket returns and when its node's own composition moves (issue 1944)' (#476) from fix/r6-node-composition into main
Reviewed-on: #476
2026-10-25 00:18:25 +00:00
jschoubben 52d22f07d7 mesh-build-go: fail a change that adds an executable or a file over 5 MB in the lint step unless .binaries-allowed lists it; ignore every module's in-place build binary, and remove the two that were committed (issue 1921, try speed S13) 2026-10-25 02:06:05 +02:00
jschoubben 53749833bf Fill every module again when the node-engine's socket returns and when the node's own composition moves, so a file an apply left empty is filled (issue 1945)
mesh/check checking 53749833bfbd on 2167b95a3699 | novox/mesh-catalog#476 in group fix/r6-node-composition
mesh/delivery-group group fix/r6-node-composition held: a member is held: novox/mesh-sdk@10b6e202320f (merged without a passing check: only a person decides wh…
mesh/delivery held for a person: merged without a passing check: only a person decides what follows
2026-10-25 02:02:02 +02:00
jschoubben 8f6a2b1f82 mesh-build-go: tests that a workspace is left with only its tree leave out the ask's own cache layer and slots, every such test at once (issue 1921, try speed S2, S10) 2026-10-25 01:46:01 +02:00
jschoubben a0fbd6bcb5 mesh-build-go: the real-go integration tests expect a lint finding as the lint step's, beside the tests (ADR 0542 §2; issue 1921, try speed S4) 2026-10-25 01:40:02 +02:00
jschoubben 5ce46cd197 mesh-build-go: report a lint finding as the lint step's, not the unit level's (ADR 0542 §2), and pin mesh-sdk's try speed-up head with main in it in both Go and Node build modules (issue 1921, try speed S4, S11) 2026-10-25 01:31:10 +02:00
jschoubben 8943a290f5 Merge remote-tracking branch 'origin/main' into feat/try-speed 2026-10-25 01:29:43 +02:00
jschoubben 537e2ef31f mesh-build-go: check with CGO_ENABLED=0 as production builds do, and let the lint's go vet, a compile alone, warm the shared cache for later checks (issue 1921, try speed S11) 2026-10-25 01:29:21 +02:00
jschoubben 545a172bbf mesh-build-go: bound the processes of an ask's levels together, the processors divided by the build capacity, through lock files in its workspace, now that the levels run side by side (issue 1921, try speed S10) 2026-10-25 01:26:50 +02:00
jschoubben 391095ccb9 mesh-build-go: run in the integration level only the tests its tags add, read from the source, so the unit tests are not run twice (issue 1921, try speed S7) 2026-10-25 01:13:21 +02:00
jschoubben 8536021f82 mesh-build-go: bring the mirror to the trunk commit so the merge preview deepens from it, and place the siblings side by side (issue 1921, try speed S6) 2026-10-25 01:10:19 +02:00
jschoubben 23307935b9 mesh-build-go: start the integration level's store and bus beside its compile, and compile the go.mod directories side by side without running a test binary (issue 1921, try speed S5) 2026-10-25 01:06:53 +02:00
jschoubben d5d6fd7412 mesh-build-go: run a unit level's lint beside its tests under one bound, so its wall time is the longer of the two; a lint failure still fails the level (issue 1921, try speed S4) 2026-10-25 01:04:35 +02:00
jschoubben 2e69099354 mesh-build-go: lint only the packages a unit level runs, in one vet pass with every tag unless a tag leaves a file out, so a small change's lint costs a few packages (issue 1921, try speed S3) 2026-10-25 01:02:08 +02:00
jschoubben 5c65b1f3dc mesh-build-go: give each ask a cache layer in its workspace, so what its vet compiles its unit tests and integration level find compiled (issue 1921, try speed S2) 2026-10-25 01:00:32 +02:00
mesh-tools 2167b95a36 Merge pull request 'R6: the gate keeps only the merge's states; walk, plan and step code and their verbs go (issue 1946)' (#475) from feat/r6 into main 2026-10-24 23:00:25 +00:00
jschoubben 4f074e87c7 mesh-build-go: say how long each phase of a fetch ran, so a check's "clone" time splits into mirror, fetch, preview, repack, siblings and modules (issue 1921, try speed S1) 2026-10-25 00:54:55 +02:00
jschoubben d09cbf1523 Ignore the binaries a local go build leaves in three module roots (issue 1943)
mesh/delivery-group group feat/r6 merged: every member is merged
mesh/check pass d09cbf152317 on 6fb0ceeef817 | unit 4378 run, 4719 kept, 15 skipped | integration 5060 run, 4917 kept, 23 skipped | over bound: unit, …
mesh/delivery delivering: 0 machine step(s) passed
2026-10-25 00:02:45 +02:00
jschoubben ac6712cbb5 Pin mesh-sdk to feat/r6 748c564, which carries main's composer (issue 1943) 2026-10-25 00:02:45 +02:00
jschoubben 1968aba6e0 mesh-build-go, mesh-build-node: run every package of a check's level after a failure (issue 1921)
Pins mesh-sdk fix/1921-every-failure, whose check ask no longer stops a
level at its first failing test step, so one check runs every package
and names every failure. A captured fixture with a failing test in each
of two packages proves the build module's test step names both.
2026-10-24 23:47:27 +02:00
jschoubben 5b32342ac2 Merge main into feat/r6: keep R6's gate, which already carries #474's catch-up at start, and drop the seat and walk adoption R6 removed
The reads-outside markers, mesh-build-go's compose job and gitea's forge-account fix come in from main.
2026-10-24 23:03:42 +02:00
jschoubben 6fb0ceeef8 Merge pull request 'R4g finish: mesh-build-go's compose job, the reads-outside marker in the digest, merge-check.sh gone (issues 1295, 1621-1623, 1935)' (#473) from feat/r4g-finish into main
Reviewed-on: #473
2026-10-24 20:56:49 +00:00
mesh-tools c651161787 Merge pull request 'gitea's forge-account reads the forge's own host key file before scanning its sshd, so a forge restart no longer refuses every forge account (issue 1939)' (#472) from fix/1939-forge-host-key into main 2026-10-24 20:05:39 +00:00
jschoubben ee216e4e63 Merge pull request 'The gate keeps a merge it hears before it holds its seat and catches source.main up when it does, so a missed merge no longer leaves the checker on an old trunk (issue 1726)' (#474) from fix/1726-trunk-catch-up into main
Reviewed-on: #474
2026-10-24 19:47:47 +00:00
jschoubben b67b5af5e6 Pin the gate to mesh-sdk feat/r6 at d7a9b968 (issue 1946) 2026-10-24 21:18:26 +02:00
jschoubben d700ceca20 Pin the gate to mesh-sdk feat/r6 at 7f9cc0e5 (issue 1946) 2026-10-24 20:43:41 +02:00
jschoubben 3a1017f559 Pin the gate to mesh-sdk feat/r6 at e45bdc08 (issue 1946) 2026-10-24 20:22:07 +02:00
jschoubben a17d0c8e93 The messenger holds and tells what the delivery view says is in motion, not the controller's plans (issue 1946) 2026-10-24 18:25:38 +02:00
jschoubben 9fa6710296 Pin the gate to mesh-sdk feat/r6 at a523a8cb (issue 1946) 2026-10-24 18:18:06 +02:00
jschoubben 05f849979c Match a fix to a delivered revision by what the gate says of its merge; keep #474's source catch-up at start (issues 1946, 1726) 2026-10-24 18:13:40 +02:00
jschoubben 4dee77df23 Pin the gate to the head of mesh-sdk feat/r6 (issue 1946) 2026-10-24 17:54:30 +02:00
jschoubben 8a79877a15 A newer delivered revision covers the older merges of its repository, as the forge says ancestry (issue 1946) 2026-10-24 17:03:20 +02:00
jschoubben 2139e83a8a Remove a binary built by mistake 2026-10-24 16:58:33 +02:00
jschoubben e7ca8f7af1 The backup module says on the bus when a backup asked for now has ended, so no one polls backed-up (issue 1944, ADR 0539 §16) 2026-10-24 16:58:19 +02:00
jschoubben eb79996336 The tracker derives a fix's delivered from the delivery view's event, not the gate's gone states (issue 1946) 2026-10-24 16:50:50 +02:00
jschoubben 1a271a1fb1 The group's integration test hears the forge's merges once both members are handed (issue 1946, R6-C3) 2026-10-24 16:43:48 +02:00
jschoubben 30d04f6bef Pin the gate to mesh-sdk's release 6 verb tables and prove it holds the seat as they have it (issue 1946, R6-C3)
The integration tests hear the forge's own word that a pull request merged, as the real flow does, and fail on any call of the controller.
2026-10-24 16:35:15 +02:00
jschoubben 5ff337e0dc Gate keeps the merge's six states and asks the controller nothing (issue 1946, R6-C1, C2, C4)
Release 6 deletes the plans: the walk half of the holder, the controller port, the carry-over of the former module and the verbs times, what-if, release, stop and retire-history go; the table holds proposed, ready, rejected, merged, held and closed; the delivery's status and view name the source revision the merge moved.
2026-10-24 16:26:36 +02:00
jschoubben 2fabbe556e Pin to the group's new mesh-sdk head bfc3883a, whose composer reads a fallback to a predefined key as the indexer does (ADR 0348 §7)
mesh/check checking 2fabbe556e73 on 6fb0ceeef817 | novox/mesh-catalog#473 in group feat/r4g-finish
mesh/delivery delivered
mesh/delivery-group group feat/r4g-finish merged: every member is merged
2026-10-24 14:57:35 +02:00
jschoubben 89b9212c4d Pin to the group's new mesh-sdk head 19c1d359, whose composer names the reader's pinned SDK instead of a bare (devel) (ADR 0482 clause 5)
mesh/check failed: shanks: novox/mesh-catalog#473: modules/mesh-build-go/go.mod names mesh-sdk 19c1d35939bc, which is neither on mesh-sdk's trunk nor …
mesh/delivery superseded: a newer head of the same pull request
2026-10-24 14:12:09 +02:00
jschoubben 40da368ee3 Date an adopted merge once, by the walk it was adopted from, so a later walk it follows never makes it look newer than a forge merge heard after it (issue 1726, review)
mesh/check pass 40da368ee34e on ee216e4e63f1 | unit 246 run, 4189 kept | integration 228 run, 4249 kept | over bound: unit, integration (slowest unit …
mesh/delivery delivered
2026-10-24 13:55:10 +02:00
jschoubben 865f3b6455 Say what pacman-tools reads outside: the root predefined.json (issue 1935)
mesh/delivery-group group feat/r4g-finish rejected: a member's own check failed
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: shanks: novox/mesh-catalog#473: modules/mesh-build-go/go.mod names mesh-sdk bc0c25e51052, which is neither on mesh-sdk's trunk nor …
2026-10-24 13:48:18 +02:00
jschoubben 03524ecd13 Mark pacman-tools as reading outside its directory: its upgrade-order test reads the root predefined.json, which its digest does not cover (issue 1935) 2026-10-24 13:48:09 +02:00
jschoubben 15101c9ce6 Move source.main for a merge the gate caught up after a gap, so the checker re-checks open heads on the trunk that is (issue 1726)
The checker takes a repository's trunk head only from the gate's source.main. On 2026-10-24 gitea's
restarted poll announced mesh-catalog #471's merge while the gate was starting and could not yet tell
whether it held the seat; the gate acknowledged the event unread. It then adopted the controller's walk
of the merge commit, which carries no merge time, so every older merge of the repository looked newer
and source.main was never moved: #472 was checked on the trunk before #471 and its merge refused.

- An event heard before the gate can tell whether it holds the seat is offered again, not taken.
- An adopted merge is dated by its walk's creation, so it settles into source.main as a forge merge does.
- Each time the gate comes to hold the seat it weighs the newest merge of each repository against
  source.main once more, so a merge marked there and never written (as #471) is written; never backwards.
2026-10-24 13:41:56 +02:00
jschoubben f910b0ceb8 Pin mesh-build-go to the group's mesh-sdk head after main's merge, so it builds on the SDK the group delivers 2026-10-24 13:28:16 +02:00
jschoubben 91c3e954ae Merge main into feat/r4g-finish (#471) 2026-10-24 13:26:24 +02:00
jschoubben 545c72bb59 Merge pull request 'gitea's watcher answers its handshake without waiting for the forge, so a slow forge at a tool-runner restart no longer kills its tools and events (issue 1726)' (#471) from fix/1726-gitea-handshake-not-on-forge into main
Reviewed-on: #471
2026-10-24 11:10:39 +00:00
jschoubben 7afcd0282d Set up the forge's webhook without holding the bundle's import, so a slow forge never costs gitea its handshake and its tools (issue 1726)
At 12:37 on 2026-10-24 node-tools on novox was replaced; gitea's watcher bundle awaited the webhook's set-up on the forge's admin API at the top of its import, outlasted the 10 s handshake and was killed, never to start again. gitea_commit_status, gitea_merge_pull_request and gitea_events were then served nowhere: the forge bridge's calls found no responders and no new pull request was announced.
2026-10-24 13:00:57 +02:00
jschoubben 75156fce00 gitea: forge-account reads the forge's own host key file before scanning its sshd (issue 1939)
mesh/check pass 75156fce007e on c65116178720 | unit 246 run, 4192 kept | integration 228 run, 4252 kept | over bound: unit, integration (slowest unit …
mesh/delivery delivered
2026-10-24 12:50:20 +02:00
jschoubben 16d7b1898d Mark the packages whose tests read files outside their directory, found by the review (issue 1935) 2026-10-24 04:38:49 +02:00
jschoubben f04465537b Say in realtree_test that a missing trunk fails, not skips (issue 1295) 2026-10-24 04:37:52 +02:00
jschoubben e870a7f65f Pin to the finish group's mesh-sdk head dd0d156b, so every member names one SDK (issue 1622) 2026-10-24 04:28:14 +02:00
jschoubben e8a59171c7 Merge remote-tracking branch 'origin/feat/r4g-finish-composer' into HEAD 2026-10-24 04:26:03 +02:00
jschoubben ebb25757a2 Merge remote-tracking branch 'origin/feat/r4g-finish-1935' into HEAD 2026-10-24 04:26:02 +02:00
jschoubben 6a70735489 Run a compose ask on the Go build seat: build mesh-sdk's composer at the asked commit once, fill its manifests from the indexer and run it shut off (issue 1622) 2026-10-24 04:14:21 +02:00
jschoubben a715c8821f mesh-build-go: a catalogue module's non-Go files enter its packages' digests; markers where tests read other modules (issue 1935) 2026-10-24 03:59:08 +02:00
jschoubben b6ed8599c8 mesh-build-go: a package marked reads-outside is never kept, so tests that read files outside their directory run on every check (issue 1935) 2026-10-24 03:46:45 +02:00
jschoubben ae6f7d5b54 Make a missing trunk copy a failure again, now the checker places it (issue 1295) 2026-10-24 03:43:49 +02:00
jschoubben 25176e1ccd Merge pull request 'mesh-build-go: lint the unit level's directories as many at once as its tests, so a large repository's check ends within its bound (issue 1930)' (#470) from fix/1930-parallel-lint into main
Reviewed-on: #470
2026-10-24 00:16:55 +00:00
jschoubben 7e1f3de77a mesh-build-go: lint directories side by side like the tests, so a large repository's check stays inside its bound (issue 1930)
mesh/delivery delivered
mesh/check failed: mesh-build-go: the check of novox/mesh-catalog#470: the ask mvlnph6fd84a7b's step on shanks did not end within 30m0s of its start |…
2026-10-24 02:01:36 +02:00
jschoubben 4b1bed4b9d Merge pull request 'mesh-mail-reader: read messenger's sent_on as a list, so an unchanged set of mails is signalled once, not every 160 s (issue 1926)' (#469) from fix/1926-mail-signal-once into main
Reviewed-on: #469
2026-10-23 22:36:10 +00:00
jschoubben 25d69406b7 Merge pull request 'mesh-build-go: a failed test keeps its last lines and its cause, not its first fifteen (issue 1901, with mesh-delivery)' (#466) from fix/1901-rebased into main
Reviewed-on: #466
2026-10-23 22:35:57 +00:00
jschoubben f836c021e1 Merge pull request 'gitea: the checker's owed results are taken at once and written in the background, so the runner keeps the watch and every mesh/check reaches the forge (issue 1726)' (#468) from fix/1726-owed-watch-not-blocked into main
Reviewed-on: #468
2026-10-23 22:35:33 +00:00
jschoubben 450f0bbe79 Merge pull request 'The build modules pin mesh-sdk's list-form level digest, so a check of a repository with a package named token ends (issue 1930, with mesh-sdk and mesh-delivery)' (#467) from fix/1930-digest-shape into main
Reviewed-on: #467
2026-10-23 22:08:24 +00:00
jschoubben 5a7076a6a2 Take the checker's owed without waiting on the forge, so a large owed never times its watch out (issue 1726)
mesh/delivery delivered
mesh/check checking 5a7076a6a25e on f836c021e14f
The runtime gives a watch's current values 10 s and waits for each to be taken; a forge write per key made ~200 keys outlast it at every attempt, so after node-tools restarted on novox at 23:25:58 the watch never held and no verdict reached mesh/check. Writes now run in the background, a few at a time, each key's in revision order.
2026-10-23 23:52:52 +02:00
jschoubben 30b22c198e Pin the build modules to mesh-sdk 403a5d7, whose level digest is a list, so a check of a repository with a package named token ends (issue 1930)
mesh/check checking 30b22c198e6a on 450f0bbe7984 | novox/mesh-catalog#467 in group fix/1930-digest-shape
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792875397682979170 (novox/mesh-catalog at 6fb0ceee into main,…
mesh/delivery-group group fix/1930-digest-shape merged: every member is merged
2026-10-23 23:36:31 +02:00
jschoubben 753e8fc202 mesh-mail-reader: read tell's sent_on as the list it is, so a told set is not sent again; leave past-due and duplicate mails out (issue 1926)
mesh/delivery delivered
mesh/check checking 753e8fc202c3 on 4b1bed4b9d5b
2026-10-23 23:27:23 +02:00
mesh-tools e5202445ba Merge pull request 'mesh-build-go: build_go_test takes packages as a typed list too, so checks run again (issue 1925)' (#465) from fix/1925-packages-shape into main 2026-10-23 20:44:09 +00:00
jschoubben dcae699323 mesh-build-go: a clipped failure starts with its cause, not the marker, for an older checker (issue 1901/AC3)
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-catalog@753e8fc202c3 (merged as 4b1bed4b into main, walk plan-17927949853…
mesh/check checking dcae699323c8 on 25d69406b740 | novox/mesh-catalog#466 in group fix/1901-rebased
mesh/delivery-group group fix/1901-rebased merged: every member is merged
2026-10-23 22:26:29 +02:00
jschoubben 70ef99595a mesh-build-go: build_go_test takes packages as a typed list too (issue 1925)
mesh/delivery delivered
mesh/check failed: shanks: the check of novox/mesh-catalog#465 failed: the unit level: packages is a list of package patterns | 70ef99595aa9 on e52024…
2026-10-23 22:21:41 +02:00
jschoubben 746eb885a3 Test 1901/AC3: an older pair of build module and checker still agree (issue 1901) 2026-10-23 22:14:28 +02:00
jschoubben 2cdb9b8fc4 mesh-build-go: a failed test keeps its last lines and its error line, not its first fifteen (issue 1901) 2026-10-23 22:13:41 +02:00
jschoubben e830045545 Merge pull request 'R4g+R5 (mesh-catalog): the build modules pull asks, scripts become tests, build-agent references go (design 47)' (#464) from feat/r4g into main
Reviewed-on: #464
2026-10-23 17:49:43 +00:00
jschoubben 454fb8a918 Write a pull request's check verdict into its key after the key is created: the status event can be taken first (R4g, issue 1887)
mesh/delivery covered after its failure: a later walk that contains it was delivered: plan-1792777718506798022 (novox/mesh-catalog at e8300455 into main,…
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check checking 454fb8a918ad on 3bec357968da | novox/mesh-catalog#464 in group feat/r4g
2026-10-23 19:42:54 +02:00
jschoubben c32474ee40 Judge the delivery-before-check order from the step records, not a poll: at capacity 1 the check starts the instant the delivery ends (R4g, issue 1887)
mesh/delivery superseded: a newer head of the same pull request
mesh/check failed: integration: mesh-delivery-gate/cmd/mesh-d… | c32474ee40f4 on 3bec357968da | judged by the candidate's mesh-build-go at 2e667ac15ce7
2026-10-23 19:19:45 +02:00
jschoubben f95f3f430e Give the delivery-ask integration test a bus of its own: the level shares one bus across directories, so a named stream there could be taken by another test (R4g, issue 1887) 2026-10-23 19:15:52 +02:00
218 changed files with 4974 additions and 10927 deletions
+308
View File
@@ -29,3 +29,311 @@ modules/mesh-build-go/cmd/mesh-build-go/mesh-build-go
modules/mesh-config/mesh-config modules/mesh-config/mesh-config
modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
*.test *.test
# binaries a local go build leaves in a module root
/checks/paths/paths
/modules/mesh-build-go/mesh-build-go
/modules/mesh-delivery-gate/mesh-delivery-gate
# A binary built in place, named like its module directory or a cmd directory, in the module root or in that cmd directory
# (issue 1921, try speed S13), for every module; the lint step of every check refuses one a change adds, a module added
# later included.
/modules/adwaita/adwaita
/modules/adwaita/adwaita-tools
/modules/adwaita/cmd/adwaita-tools/adwaita-tools
/modules/artifact-store-tools/artifact-store-tools
/modules/artifact-store-tools/cmd/artifact-store-tools/artifact-store-tools
/modules/asus-zephyrus-g14/asus-zephyrus-g14
/modules/asus-zephyrus-g14/zephyrus
/modules/asus-zephyrus-g14/cmd/zephyrus/zephyrus
/modules/audit-logger/audit-logger
/modules/avahi/avahi
/modules/avahi/avahi-tools
/modules/avahi/cmd/avahi-tools/avahi-tools
/modules/baserow/baserow
/modules/blueman/blueman
/modules/blueman/blueman-tools
/modules/blueman/cmd/blueman-tools/blueman-tools
/modules/bluetooth/bluetooth
/modules/bluetooth/bluetooth-tools
/modules/bluetooth/cmd/bluetooth-tools/bluetooth-tools
/modules/brother/brother
/modules/brother/cmd/brother/brother
/modules/buildah/buildah
/modules/ca-trust/ca-trust
/modules/claude-code/claude-code
/modules/claude-code/cmd/claude-code/claude-code
/modules/claude-licence-manager/claude-licence-manager
/modules/claude-licence-manager/cmd/claude-licence-manager/claude-licence-manager
/modules/clipmenu/clipmenu
/modules/clipmenu/clipmenu-tools
/modules/clipmenu/cmd/clipmenu-tools/clipmenu-tools
/modules/confluence/confluence
/modules/cups/cups
/modules/cups/cups-tools
/modules/cups/cmd/cups-tools/cups-tools
/modules/dbus/dbus
/modules/dbus/dbus-tools
/modules/dbus/cmd/dbus-tools/dbus-tools
/modules/desk-channel/desk-channel
/modules/desk-channel/cmd/desk-channel/desk-channel
/modules/dev-tools/dev-tools
/modules/disk-load/disk-load
/modules/disk-load/cmd/disk-load/disk-load
/modules/distribution/distribution
/modules/dmenu/dmenu
/modules/dmenu/dmenu-tools
/modules/dmenu/cmd/dmenu-tools/dmenu-tools
/modules/dnsmasq/dnsmasq
/modules/dnsmasq/dnsmasq-tools
/modules/dnsmasq/cmd/dnsmasq-tools/dnsmasq-tools
/modules/docker/docker
/modules/docker/docker-tools
/modules/docker/cmd/docker-tools/docker-tools
/modules/docker-compose/docker-compose
/modules/docker-compose/docker-compose-tools
/modules/docker-compose/cmd/docker-compose-tools/docker-compose-tools
/modules/dunst/dunst
/modules/dunst/dunst-tools
/modules/dunst/cmd/dunst-tools/dunst-tools
/modules/fail2ban/fail2ban
/modules/fail2ban/fail2ban-tools
/modules/fail2ban/cmd/fail2ban-tools/fail2ban-tools
/modules/feh/feh
/modules/feh/feh-tools
/modules/feh/cmd/feh-tools/feh-tools
/modules/flatpak/flatpak
/modules/flatpak/flatpak-tools
/modules/flatpak/cmd/flatpak-tools/flatpak-tools
/modules/fonts/fonts
/modules/fonts/fonts-tools
/modules/fonts/cmd/fonts-tools/fonts-tools
/modules/forticlient/forticlient
/modules/forticlient/forticlient-tools
/modules/forticlient/cmd/forticlient-tools/forticlient-tools
/modules/gitea/gitea
/modules/gitea/agent-teams
/modules/gitea/cmd/agent-teams/agent-teams
/modules/gitea/agent-token
/modules/gitea/cmd/agent-token/agent-token
/modules/gitea/forge
/modules/gitea/cmd/forge/forge
/modules/gitea/forge-account-provider
/modules/gitea/cmd/forge-account-provider/forge-account-provider
/modules/gitea/npm-registry
/modules/gitea/cmd/npm-registry/npm-registry
/modules/gitea/repositories
/modules/gitea/cmd/repositories/repositories
/modules/gitlab/gitlab
/modules/gnome-keyring/gnome-keyring
/modules/gnome-keyring/gnome-keyring-tools
/modules/gnome-keyring/cmd/gnome-keyring-tools/gnome-keyring-tools
/modules/go/go
/modules/grafana/grafana
/modules/hello-web/hello-web
/modules/home-assistant/home-assistant
/modules/hostname/hostname
/modules/hostname/hostname-tools
/modules/hostname/cmd/hostname-tools/hostname-tools
/modules/i3/i3
/modules/i3/i3-tools
/modules/i3/cmd/i3-tools/i3-tools
/modules/i3status-rust/i3status-rust
/modules/i3status-rust/i3status-rust-tools
/modules/i3status-rust/cmd/i3status-rust-tools/i3status-rust-tools
/modules/icecast/icecast
/modules/influxdb/influxdb
/modules/invoicing/invoicing
/modules/jira/jira
/modules/kernel/kernel
/modules/kernel/kernel-tools
/modules/kernel/cmd/kernel-tools/kernel-tools
/modules/keycloak/keycloak
/modules/keycloak/keycloak-provider
/modules/keycloak/cmd/keycloak-provider/keycloak-provider
/modules/lab/lab
/modules/lemurs/lemurs
/modules/lemurs/lemurs-tools
/modules/lemurs/cmd/lemurs-tools/lemurs-tools
/modules/letta/letta
/modules/localization/localization
/modules/localization/localization-tools
/modules/localization/cmd/localization-tools/localization-tools
/modules/local-model-consumer/local-model-consumer
/modules/logrotate/logrotate
/modules/logrotate/logrotate-tools
/modules/logrotate/cmd/logrotate-tools/logrotate-tools
/modules/mailu/mailu
/modules/mailu/automx
/modules/mailu/cmd/automx/automx
/modules/marrytts/marrytts
/modules/matrix/matrix
/modules/memory-pressure/memory-pressure
/modules/memory-pressure/cmd/memory-pressure/memory-pressure
/modules/mesh-build-go/mesh-build-go
/modules/mesh-build-go/cmd/mesh-build-go/mesh-build-go
/modules/mesh-build-image/mesh-build-image
/modules/mesh-build-image/cmd/mesh-build-image/mesh-build-image
/modules/mesh-build-node/mesh-build-node
/modules/mesh-build-node/cmd/mesh-build-node/mesh-build-node
/modules/mesh-catalog/mesh-catalog
/modules/mesh-cli/mesh-cli
/modules/mesh-cli/cmd/mesh-cli/mesh-cli
/modules/mesh-config/mesh-config
/modules/mesh-config/cmd/mesh-config/mesh-config
/modules/mesh-delivery-gate/mesh-delivery-gate
/modules/mesh-delivery-gate/cmd/mesh-delivery-gate/mesh-delivery-gate
/modules/mesh-files/mesh-files
/modules/mesh-files/cmd/mesh-files/mesh-files
/modules/mesh-issues/mesh-issues
/modules/mesh-issues/cmd/mesh-issues/mesh-issues
/modules/mesh-knowledge/mesh-knowledge
/modules/mesh-knowledge/cmd/mesh-knowledge/mesh-knowledge
/modules/mesh-mail/mesh-mail
/modules/mesh-mail/cmd/mesh-mail/mesh-mail
/modules/mesh-mail-notifier/mesh-mail-notifier
/modules/mesh-mail-notifier/cmd/mesh-mail-notifier/mesh-mail-notifier
/modules/mesh-mail-reader/mesh-mail-reader
/modules/mesh-mail-reader/cmd/mesh-mail-reader/mesh-mail-reader
/modules/mesh-mail-web/mesh-mail-web
/modules/mesh-mail-web/cmd/mesh-mail-web/mesh-mail-web
/modules/mesh-provisioner/mesh-provisioner
/modules/mesh-provisioner/cmd/mesh-provisioner/mesh-provisioner
/modules/mesh-provisioner-client/mesh-provisioner-client
/modules/mesh-provisioner-client/cmd/mesh-provisioner-client/mesh-provisioner-client
/modules/mesh-reads/mesh-reads
/modules/mesh-reads/cmd/mesh-reads/mesh-reads
/modules/mesh-vault/mesh-vault
/modules/mesh-vault/cmd/mesh-vault/mesh-vault
/modules/mesh-watcher/mesh-watcher
/modules/mesh-watcher/cmd/mesh-watcher/mesh-watcher
/modules/messenger/messenger
/modules/messenger/cmd/messenger/messenger
/modules/minio/minio
/modules/minio/object-store
/modules/minio/cmd/object-store/object-store
/modules/model-usage/model-usage
/modules/mongodb/mongodb
/modules/mosquitto/mosquitto
/modules/mounts/mounts
/modules/mounts/cmd/mounts/mounts
/modules/mssql/mssql
/modules/n8n/n8n
/modules/nats/nats
/modules/nats/mesh-nats-snapshot
/modules/nats/cmd/mesh-nats-snapshot/mesh-nats-snapshot
/modules/nats/nats-tools
/modules/nats/cmd/nats-tools/nats-tools
/modules/netcheck/netcheck
/modules/netcheck/cmd/netcheck/netcheck
/modules/networkmanager/networkmanager
/modules/networkmanager/uplink-tools
/modules/networkmanager/cmd/uplink-tools/uplink-tools
/modules/nextcloud/nextcloud
/modules/nextcloud-client/nextcloud-client
/modules/nextcloud-client/nextcloud-client-tools
/modules/nextcloud-client/cmd/nextcloud-client-tools/nextcloud-client-tools
/modules/nfs-server/nfs-server
/modules/nfs-server/cmd/nfs-server/nfs-server
/modules/nftables/nftables
/modules/nm-applet/nm-applet
/modules/nm-applet/nm-applet-tools
/modules/nm-applet/cmd/nm-applet-tools/nm-applet-tools
/modules/node-env/node-env
/modules/nodejs/nodejs
/modules/nodered/nodered
/modules/ollama/ollama
/modules/only-office/only-office
/modules/only-office/cmd/only-office/only-office
/modules/openai-consumer/openai-consumer
/modules/openrazer/openrazer
/modules/openrazer/openrazer-tools
/modules/openrazer/cmd/openrazer-tools/openrazer-tools
/modules/pacman/pacman
/modules/pacman/pacman-tools
/modules/pacman/cmd/pacman-tools/pacman-tools
/modules/photos-eef/photos-eef
/modules/photos-filip/photos-filip
/modules/picom/picom
/modules/picom/picom-tools
/modules/picom/cmd/picom-tools/picom-tools
/modules/polychromatic/polychromatic
/modules/polychromatic/polychromatic-tools
/modules/polychromatic/cmd/polychromatic-tools/polychromatic-tools
/modules/postgres/postgres
/modules/postgres/postgres-provider
/modules/postgres/cmd/postgres-provider/postgres-provider
/modules/power/power
/modules/power/cmd/power/power
/modules/powerlevel10k/powerlevel10k
/modules/records/records
/modules/records/cmd/records/records
/modules/redis/redis
/modules/restic/restic
/modules/restic/restic-backups
/modules/restic/cmd/restic-backups/restic-backups
/modules/rofi/rofi
/modules/rofi/rofi-tools
/modules/rofi/cmd/rofi-tools/rofi-tools
/modules/route-adapter/route-adapter
/modules/route-adapter/route-provider
/modules/route-adapter/cmd/route-provider/route-provider
/modules/screen-lock/screen-lock
/modules/screen-lock/screen-lock-tools
/modules/screen-lock/cmd/screen-lock-tools/screen-lock-tools
/modules/searxng/searxng
/modules/sensors/sensors
/modules/sensors/cmd/sensors/sensors
/modules/showcase/showcase
/modules/slack/slack
/modules/slack/slack-tools
/modules/slack/cmd/slack-tools/slack-tools
/modules/snapd/snapd
/modules/snapd/snapd-tools
/modules/snapd/cmd/snapd-tools/snapd-tools
/modules/ssh-client/ssh-client
/modules/ssh-client/ssh-client-tools
/modules/ssh-client/cmd/ssh-client-tools/ssh-client-tools
/modules/sshd/sshd
/modules/step-ca/step-ca
/modules/sudo/sudo
/modules/sudo/sudo-tools
/modules/sudo/cmd/sudo-tools/sudo-tools
/modules/supabase/supabase
/modules/systemd/systemd
/modules/systemd/systemd-tools
/modules/systemd/cmd/systemd-tools/systemd-tools
/modules/systemd-networkd/systemd-networkd
/modules/systemd-networkd/uplink-tools
/modules/systemd-networkd/cmd/uplink-tools/uplink-tools
/modules/systemd-resolved/systemd-resolved
/modules/systemd-resolved/resolver-tools
/modules/systemd-resolved/cmd/resolver-tools/resolver-tools
/modules/telegram/telegram
/modules/telegram/cmd/telegram/telegram
/modules/time-sync/time-sync
/modules/time-sync/time-sync-tools
/modules/time-sync/cmd/time-sync-tools/time-sync-tools
/modules/triggerhappy/triggerhappy
/modules/triggerhappy/triggerhappy-tools
/modules/triggerhappy/cmd/triggerhappy-tools/triggerhappy-tools
/modules/umami/umami
/modules/unifi/unifi
/modules/website/website
/modules/xclip/xclip
/modules/xclip/xclip-tools
/modules/xclip/cmd/xclip-tools/xclip-tools
/modules/xdg/xdg
/modules/xdg/xdg-tools
/modules/xdg/cmd/xdg-tools/xdg-tools
/modules/xorg/xorg
/modules/xorg/cmd/xorg/xorg
/modules/xterm/xterm
/modules/xterm/xterm-tools
/modules/xterm/cmd/xterm-tools/xterm-tools
/modules/zfs/zfs
/modules/zfs/zfs-tools
/modules/zfs/cmd/zfs-tools/zfs-tools
/modules/zsh/zsh
/modules/zsh-autosuggestions/zsh-autosuggestions
/modules/zsh-syntax-highlighting/zsh-syntax-highlighting
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+2 -1
View File
@@ -251,7 +251,8 @@ func main() {
} }
var grown []string var grown []string
if baseDir == "" { if baseDir == "" {
fmt.Println("NOT COMPARED: no base branch's lists were given, so additions to them were not judged") fmt.Fprintln(os.Stderr, "NOT COMPARED: no base branch's lists were given (-base <dir>), so additions to them were not judged; that is a failure, not a pass (issue 1295)")
os.Exit(2)
} else { } else {
baseShared, _, err1 := list(filepath.Join(baseDir, "shared")) baseShared, _, err1 := list(filepath.Join(baseDir, "shared"))
baseKnown, _, err2 := list(filepath.Join(baseDir, "known")) baseKnown, _, err2 := list(filepath.Join(baseDir, "known"))
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+2 -2
View File
@@ -32,8 +32,8 @@ func TestNoModuleNamesAnothersState(t *testing.T) {
} }
} }
// And neither list gains a line the trunk's copies at the merge base do not hold, unless the line says why after `# allow:`. Skipped, // And neither list gains a line the trunk's copies at the merge base do not hold, unless the line says why after `# allow:`. Fails, saying
// and said, where no trunk is beside. // NOT COMPARED, where no trunk is beside (issue 1295).
func TestTheListsOfPathsHoldNoLineTheTrunksDoNot(t *testing.T) { func TestTheListsOfPathsHoldNoLineTheTrunksDoNot(t *testing.T) {
sharedPath, okShared := trunk.Copy(t, "checks/paths/shared") sharedPath, okShared := trunk.Copy(t, "checks/paths/shared")
knownPath, okKnown := trunk.Copy(t, "checks/paths/known") knownPath, okKnown := trunk.Copy(t, "checks/paths/known")
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+14 -18
View File
@@ -3,14 +3,14 @@
// //
// The checker fetches the change's own repository at its base commit beside the change, as it fetches a sibling, and says where // The checker fetches the change's own repository at its base commit beside the change, as it fetches a sibling, and says where
// in MESH_CHECK_BESIDE; the copy is under the repository's name (the same convention the other repositories' tests use). With // in MESH_CHECK_BESIDE; the copy is under the repository's name (the same convention the other repositories' tests use). With
// MESH_CHECK_BESIDE unset, or set without the catalogue's copy in it, the comparison is skipped and the test says NOT COMPARED // MESH_CHECK_BESIDE unset, or set without the catalogue's copy in it, the test fails saying NOT COMPARED WITH THE TRUNK and
// WITH THE TRUNK, naming why: the checker that runs before R4g's delivery places siblings but not the change's own repository, // why: the checker places the change's own repository since R4g (issue 1295), so a missing copy is a fault, never a pass. A trunk that holds no such file had
// so R4g's own check cannot hold the copy (issue 1295 notes that a missing copy becomes a failure once that checker is gone). A trunk that holds no such file had
// none: the list's first copy, which nothing was added to. // none: the list's first copy, which nothing was added to.
package trunk package trunk
import ( import (
"errors" "errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
@@ -23,13 +23,11 @@ const Repository = "mesh-catalog"
const BesideEnv = "MESH_CHECK_BESIDE" const BesideEnv = "MESH_CHECK_BESIDE"
// Copy is the trunk's copy of the file at rel (a path from the catalogue's root), written to a file of the test's own: its path, // Copy is the trunk's copy of the file at rel (a path from the catalogue's root), written to a file of the test's own: its path,
// and whether the trunk holds one (false: its first copy). It skips the test, saying so, when no copy of the catalogue is beside. // and whether the trunk holds one (false: its first copy). It fails the test when no copy of the catalogue is beside.
func Copy(t testing.TB, rel string) (string, bool) { func Copy(t testing.TB, rel string) (string, bool) {
t.Helper() t.Helper()
raw, found, skip, err := Read(rel) raw, found, err := Read(rel)
switch { switch {
case skip != "":
t.Skip(skip)
case err != nil: case err != nil:
t.Fatal(err) t.Fatal(err)
case !found: case !found:
@@ -43,27 +41,25 @@ func Copy(t testing.TB, rel string) (string, bool) {
return path, true return path, true
} }
// Read is the trunk's copy of rel and whether the trunk holds one. skip is the words to skip the comparison with, when no trunk is // Read is the trunk's copy of rel and whether the trunk holds one. err is a trunk not beside the check (MESH_CHECK_BESIDE unset,
// named; err is a trunk named and not there, or unreadable. // or without the catalogue's directory), or a copy unreadable: a comparison that cannot be made is never a pass.
func Read(rel string) (content []byte, found bool, skip string, err error) { func Read(rel string) (content []byte, found bool, err error) {
beside := os.Getenv(BesideEnv) beside := os.Getenv(BesideEnv)
if beside == "" { if beside == "" {
return nil, false, "NOT COMPARED WITH THE TRUNK: " + BesideEnv + " is not set, so the trunk's copy of " + rel + return nil, false, fmt.Errorf("NOT COMPARED WITH THE TRUNK: %s is not set, so the trunk's copy of %s is not beside this run; the checker places the change's own repository there (issue 1295)", BesideEnv, rel)
" is not beside this run; the checker places it", nil
} }
dir := filepath.Join(beside, Repository) dir := filepath.Join(beside, Repository)
if _, err := os.Stat(dir); errors.Is(err, os.ErrNotExist) { if _, err := os.Stat(dir); errors.Is(err, os.ErrNotExist) {
return nil, false, "NOT COMPARED WITH THE TRUNK: " + Repository + " is not beside this check in " + BesideEnv + "=" + beside + return nil, false, fmt.Errorf("NOT COMPARED WITH THE TRUNK: %s is not beside this check in %s=%s, so %s is not judged against the trunk's copy; the checker places the change's own repository there at its trunk (issue 1295)", Repository, BesideEnv, beside, rel)
", so " + rel + " is not judged against the trunk's copy (the checker before R4g places no copy of the change's own repository)", nil
} else if err != nil { } else if err != nil {
return nil, false, "", err return nil, false, err
} }
raw, err := os.ReadFile(filepath.Join(dir, rel)) raw, err := os.ReadFile(filepath.Join(dir, rel))
if errors.Is(err, os.ErrNotExist) { if errors.Is(err, os.ErrNotExist) {
return nil, false, "", nil return nil, false, nil
} }
if err != nil { if err != nil {
return nil, false, "", err return nil, false, err
} }
return raw, true, "", nil return raw, true, nil
} }
+14 -14
View File
@@ -17,27 +17,27 @@ func TestTheCloneBesideTheCheckIsTheTrunk(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
t.Setenv(BesideEnv, beside) t.Setenv(BesideEnv, beside)
raw, found, skip, err := Read("checks/x/known") raw, found, err := Read("checks/x/known")
if err != nil || skip != "" || !found || string(raw) != "a\nb\n" { if err != nil || !found || string(raw) != "a\nb\n" {
t.Fatalf("read %q found %v skip %q err %v", raw, found, skip, err) t.Fatalf("read %q found %v err %v", raw, found, err)
} }
if _, found, skip, err := Read("checks/x/absent"); err != nil || skip != "" || found { if _, found, err := Read("checks/x/absent"); err != nil || found {
t.Fatalf("a file the trunk does not hold: found %v skip %q err %v; wanted its first copy", found, skip, err) t.Fatalf("a file the trunk does not hold: found %v err %v; wanted its first copy", found, err)
} }
} }
func TestWithNothingBesideTheComparisonIsSkippedAndSaidNotMade(t *testing.T) { func TestWithNothingBesideTheComparisonFailsAndSaysNotMade(t *testing.T) {
t.Setenv(BesideEnv, "") t.Setenv(BesideEnv, "")
_, found, skip, err := Read("checks/x/known") _, found, err := Read("checks/x/known")
if found || err != nil || len(skip) < 12 || skip[:12] != "NOT COMPARED" { if found || err == nil || !strings.Contains(err.Error(), "NOT COMPARED") {
t.Fatalf("found %v, skip %q, err %v; wanted NOT COMPARED", found, skip, err) t.Fatalf("found %v, err %v; wanted a failure saying NOT COMPARED", found, err)
} }
} }
func TestACloneBesideWithoutTheCatalogueIsSkippedAndSaidNotMade(t *testing.T) { func TestACloneBesideWithoutTheCatalogueFailsAndSaysNotMade(t *testing.T) {
t.Setenv(BesideEnv, t.TempDir()) // the siblings are there, the catalogue's own base is not (the checker before R4g) t.Setenv(BesideEnv, t.TempDir()) // the siblings are there, the catalogue's own base is not
_, found, skip, err := Read("checks/x/known") _, found, err := Read("checks/x/known")
if found || err != nil || len(skip) < 12 || skip[:12] != "NOT COMPARED" || !strings.Contains(skip, "not beside") { if found || err == nil || !strings.Contains(err.Error(), "NOT COMPARED") || !strings.Contains(err.Error(), "not beside") {
t.Fatalf("found %v, skip %q, err %v; wanted NOT COMPARED naming the missing copy", found, skip, err) t.Fatalf("found %v, err %v; wanted a failure naming the missing copy", found, err)
} }
} }
+1
View File
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+28 -9
View File
@@ -83,6 +83,9 @@ export function checkStatusOf(change: OwedChange): { owner: string; repo: string
return { owner, repo, sha, revision, status }; return { owner, repo, sha, revision, status };
} }
/** How many mesh/check writes this module has at the forge at once. */
export const WRITES_AT_ONCE = 4;
/** The one writer of mesh/check: each owed revision of the checker's, in revision order per key. */ /** The one writer of mesh/check: each owed revision of the checker's, in revision order per key. */
export class CheckStatuses { export class CheckStatuses {
/** The newest revision written of each key, and the newest seen; the oldest keys go past what any forge holds open. */ /** The newest revision written of each key, and the newest seen; the oldest keys go past what any forge holds open. */
@@ -91,6 +94,9 @@ export class CheckStatuses {
/** The turn each key's writes wait on, and the asks again still running. */ /** The turn each key's writes wait on, and the asks again still running. */
private turns = new Map<string, Promise<unknown>>(); private turns = new Map<string, Promise<unknown>>();
private again = new Set<Promise<void>>(); private again = new Set<Promise<void>>();
/** Writes at the forge at once, at most WRITES_AT_ONCE: a watch's start hands over every key together (issue 1726). */
private writing = 0;
private queued: (() => void)[] = [];
private forge: CheckForge; private forge: CheckForge;
private say: (line: string) => void; private say: (line: string) => void;
private sleep: (ms: number) => Promise<void>; private sleep: (ms: number) => Promise<void>;
@@ -123,7 +129,7 @@ export class CheckStatuses {
this.say(`[gitea] ${asked.owner}/${asked.repo}@${asked.sha.slice(0, 12)}: mesh/check at revision ${asked.revision} is older than ${newest}, already written, and is dropped`); this.say(`[gitea] ${asked.owner}/${asked.repo}@${asked.sha.slice(0, 12)}: mesh/check at revision ${asked.revision} is older than ${newest}, already written, and is dropped`);
return "dropped"; return "dropped";
} }
await this.forge.setCommitStatus(asked.owner, asked.repo, asked.sha, asked.status); await this.atTheForge(() => this.forge.setCommitStatus(asked.owner, asked.repo, asked.sha, asked.status));
this.remember(this.written, key, asked.revision); this.remember(this.written, key, asked.revision);
return "written"; return "written";
}); });
@@ -133,19 +139,32 @@ export class CheckStatuses {
return turn; return turn;
} }
/** What the watch hands over: written, or when the forge refuses, asked again on a bound in the background until it is /** What the watch hands over: taken at once and written in the background, or when the forge refuses, asked again on a
* written or a newer revision of the key was seen. Never throws: a watch's handler that throws loses the change. */ * bound until it is written or a newer revision of the key was seen. Never throws: a watch's handler that throws loses
* the change. It never waits on the forge (novox/hq issue 1726): the runtime hands a watch's current values one at a
* time, waits for each to be taken and gives the whole set 10 s; a forge write per key made a large owed outlast that
* bound at every attempt, so the watch never held and no verdict reached mesh/check. Each key's writes still take
* turns in revision order (apply). */
async take(change: OwedChange): Promise<void> { async take(change: OwedChange): Promise<void> {
const run = this.apply(change).then(() => {}, (err) => this.askAgain(change, err));
this.again.add(run);
void run.finally(() => this.again.delete(run));
}
/** Runs one forge write when fewer than WRITES_AT_ONCE are running, else after one ends. */
private async atTheForge<T>(write: () => Promise<T>): Promise<T> {
if (this.writing >= WRITES_AT_ONCE) await new Promise<void>((r) => this.queued.push(r));
else this.writing++;
try { try {
await this.apply(change); return await write();
} catch (err) { } finally {
const run = this.askAgain(change, err); const next = this.queued.shift();
this.again.add(run); if (next) next();
void run.finally(() => this.again.delete(run)); else this.writing--;
} }
} }
/** Resolves when no ask again is running. */ /** Resolves when no write or ask again is running. */
async settled(): Promise<void> { async settled(): Promise<void> {
while (this.again.size) await Promise.all([...this.again]); while (this.again.size) await Promise.all([...this.again]);
} }
@@ -65,6 +65,7 @@ func configFromEnv(getenv func(string) string) (Config, error) {
HostFile: getenv("MESH_FORGE_HOST_FILE"), HostFile: getenv("MESH_FORGE_HOST_FILE"),
OriginsFile: getenv("MESH_GITEA_ORIGINS_FILE"), OriginsFile: getenv("MESH_GITEA_ORIGINS_FILE"),
Port: port, Port: port,
HostKeyFile: getenv("MESH_GITEA_HOST_KEY_FILE"), // optional
Forge: forgehttp.Client{URL: strings.TrimRight(getenv("MESH_GITEA_URL"), "/"), User: getenv("MESH_GITEA_ADMIN_USER"), Forge: forgehttp.Client{URL: strings.TrimRight(getenv("MESH_GITEA_URL"), "/"), User: getenv("MESH_GITEA_ADMIN_USER"),
PasswordFile: getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"), Timeout: 20 * time.Second}, PasswordFile: getenv("MESH_GITEA_ADMIN_PASSWORD_FILE"), Timeout: 20 * time.Second},
Runner: execRunner{}, Runner: execRunner{},
@@ -44,10 +44,13 @@ type Config struct {
// origins, one per line. // origins, one per line.
HostFile, OriginsFile string HostFile, OriginsFile string
// Port is where the forge's sshd is published on the mesh. // Port is where the forge's sshd is published on the mesh.
Port int Port int
Forge forgehttp.Client // HostKeyFile is the forge's own ed25519 host key (the .pub file in its data directory). It is read first; when it
Runner Runner // is unset or unreadable the forge's sshd is asked with ssh-keyscan instead (issue 1939).
Log func(format string, a ...any) HostKeyFile string
Forge forgehttp.Client
Runner Runner
Log func(format string, a ...any)
} }
// Service answers the consumers of the provision `forge-account` (novox/hq ADR 0399, ADR 0400; issue 955): for each // Service answers the consumers of the provision `forge-account` (novox/hq ADR 0399, ADR 0400; issue 955): for each
@@ -448,14 +451,30 @@ func (s *Service) httpsBase() (string, error) {
return "", fmt.Errorf("%s names no https origin", s.cfg.OriginsFile) return "", fmt.Errorf("%s names no https origin", s.cfg.OriginsFile)
} }
// scan reads the forge's ed25519 host key from its own sshd and writes it as the machines' known_hosts line. // scan gives the forge's ed25519 host key as the machines' known_hosts line. The key the forge itself holds is read
// from its file first; only when that cannot be read is the forge's sshd asked over the network, and when that fails
// too (the forge's container is restarting) the key read before is kept, as a host key does not change (issue 1939).
func (s *Service) scan(ctx context.Context) (string, error) { func (s *Service) scan(ctx context.Context) (string, error) {
host, err := s.hostName() host, err := s.hostName()
if err != nil { if err != nil {
return "", err return "", err
} }
if s.cfg.HostKeyFile != "" {
raw, ferr := os.ReadFile(s.cfg.HostKeyFile)
if ferr == nil {
if line, perr := knownHostLine("127.0.0.1 "+strings.TrimSpace(string(raw)), host, s.cfg.Port); perr == nil {
return line, nil
}
}
}
out, err := s.cfg.Runner.Run(ctx, "ssh-keyscan", "-t", "ed25519", "-p", fmt.Sprint(s.cfg.Port), "127.0.0.1") out, err := s.cfg.Runner.Run(ctx, "ssh-keyscan", "-t", "ed25519", "-p", fmt.Sprint(s.cfg.Port), "127.0.0.1")
if err != nil { if err != nil {
s.mu.Lock()
kept := s.known
s.mu.Unlock()
if kept != "" {
return kept, nil
}
return "", fmt.Errorf("the forge's ssh host key cannot be read: %w", err) return "", fmt.Errorf("the forge's ssh host key cannot be read: %w", err)
} }
return knownHostLine(string(out), host, s.cfg.Port) return knownHostLine(string(out), host, s.cfg.Port)
@@ -696,3 +696,42 @@ func TestNothingIsSaidForAPerNodeUserWithoutMeshKeys(t *testing.T) {
} }
} }
} }
// The forge's own host key file is read first: no scan over the network is needed (issue 1939).
func TestTheHostKeyIsReadFromTheForgesOwnFile(t *testing.T) {
r := newRig(t)
f := filepath.Join(t.TempDir(), "ssh_host_ed25519_key.pub")
if err := os.WriteFile(f, []byte("ssh-ed25519 AAAAC3FILEKEY forge@host\n"), 0o644); err != nil {
t.Fatal(err)
}
r.svc.cfg.HostKeyFile = f
r.runner.err = errors.New("sshd down")
got, err := r.svc.scan(context.Background())
if err != nil || got != "[novox.internal]:222 ssh-ed25519 AAAAC3FILEKEY" {
t.Fatalf("scan = %q, %v", got, err)
}
if r.runner.args != nil {
t.Errorf("the forge's sshd was asked: %v", r.runner.args)
}
}
// A file that cannot be read falls back to the forge's sshd.
func TestAnUnreadableHostKeyFileFallsBackToTheScan(t *testing.T) {
r := newRig(t)
r.svc.cfg.HostKeyFile = filepath.Join(t.TempDir(), "missing.pub")
got, err := r.svc.scan(context.Background())
if err != nil || got != "[novox.internal]:222 ssh-ed25519 AAAAC3HOSTKEY" {
t.Fatalf("scan = %q, %v", got, err)
}
}
// While the forge restarts, a key read before is kept: the sshd is not up, the key has not changed.
func TestARestartingForgeKeepsTheKeyReadBefore(t *testing.T) {
r := newRig(t)
r.svc.known = "[novox.internal]:222 ssh-ed25519 AAAAC3HOSTKEY"
r.runner.err = errors.New("connection refused")
got, err := r.svc.scan(context.Background())
if err != nil || got != r.svc.known {
t.Fatalf("scan = %q, %v", got, err)
}
}
+5 -1
View File
@@ -589,7 +589,11 @@ if (gitea) {
}; };
// **The forge's events first, the polls as the safety net** (novox/hq issue 729): while verified deliveries arrive the // **The forge's events first, the polls as the safety net** (novox/hq issue 729): while verified deliveries arrive the
// polls slow to every five minutes and announce only what no event said (Pace); otherwise they keep 30 s. // polls slow to every five minutes and announce only what no event said (Pace); otherwise they keep 30 s.
await listenForForge(client); // **Not awaited** (novox/hq issue 1726): the receiver's set-up asks the forge's admin API, and the runtime kills a bundle
// whose import has not ended in its 10 s handshake and never starts it again; a forge slow to answer while every bundle
// of the node starts left gitea_commit_status, gitea_merge_pull_request and gitea_events served nowhere. Until it is
// listening, Pace keeps the polls at 30 s, so nothing is missed meanwhile; listenForForge says its own failure.
void listenForForge(client);
// Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is // Main takes no push and only the tools' account merges, in every repository, kept and not only when a token is
// minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2). // minted: at the start and every ten minutes after, so a repository made later gets it too (issue 948, ADR 0398 §2).
// Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set. // Only while the setting keep-main-rules is true (mainrules.ts); false, the default, leaves every repository's rule as set.
+1
View File
@@ -537,6 +537,7 @@
"MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret", "MESH_GITEA_ADMIN_PASSWORD_FILE": "${dir:state}/admin.secret",
"MESH_GITEA_ORIGINS_FILE": "${dir:mesh-state}/origins", "MESH_GITEA_ORIGINS_FILE": "${dir:mesh-state}/origins",
"MESH_GITEA_SSH_PORT": "222", "MESH_GITEA_SSH_PORT": "222",
"MESH_GITEA_HOST_KEY_FILE": "${dir:data}/ssh/ssh_host_ed25519_key.pub",
"MESH_FORGE_HOST_FILE": "/etc/gitea/forge-host" "MESH_FORGE_HOST_FILE": "/etc/gitea/forge-host"
} }
} }
+50
View File
@@ -109,3 +109,53 @@ test("a bundle whose read of the checker's owed is granted late writes the lates
assert.match(got.statuses[0], /"state":"success".*pass at 4/); assert.match(got.statuses[0], /"state":"success".*pass at 4/);
assert.match(err, /mesh-delivery-checker\.owed could not be bound/, "the refusal is said"); assert.match(err, /mesh-delivery-checker\.owed could not be bound/, "the refusal is said");
}); });
// **The bundle's import does not wait on the forge** (novox/hq issue 1726, 2026-10-24 12:37). The webhook receiver's set-up
// asks the forge's admin API for its system webhook; awaited at the top of index.ts, a forge slow to answer (busy while
// every bundle of the node starts) held the import past node-tools' 10 s handshake, the bundle was killed and never
// started again, and gitea_commit_status, gitea_merge_pull_request and gitea_events were served nowhere: the forge
// bridge's calls found no responders and no new pull request was announced. Here the forge stand-in takes every
// request and never answers: the import resolves at once all the same, with the bundle's tools registered.
test("importing the bundle resolves, its tools registered, while the forge never answers the webhook's set-up", async () => {
const index = pathToFileURL(join(here, "..", "dist", "index.js")).href;
const script = `
const http = await import("node:http");
const fs = await import("node:fs");
const os = await import("node:os");
const path = await import("node:path");
const forge = http.createServer(() => {});
await new Promise((r) => forge.listen(0, "127.0.0.1", r));
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "gitea-1726-"));
fs.writeFileSync(path.join(dir, "admin.password"), "pw");
process.env.MESH_GITEA_URL = "http://127.0.0.1:" + forge.address().port;
process.env.MESH_GITEA_STATE_DIR = dir;
process.env.MESH_GITEA_ADMIN_PASSWORD_FILE = path.join(dir, "admin.password");
process.env.MESH_GITEA_HOOK_PORT = "0";
process.env.MESH_GITEA_HOOK_HOST = "127.0.0.1";
const { useBroker } = await import("@novox/mesh-sdk/messaging");
useBroker(() => ({
subscribe: async () => () => {}, publish: async () => {}, request: async () => { throw new Error("no responders"); },
handle: async () => () => {}, close: async () => {},
}));
await import(${JSON.stringify(index)});
const { collectTools } = await import("@novox/mesh-sdk/tools");
console.log(JSON.stringify({ imported: true, tools: collectTools().flatMap((r) => r.tools.map((t) => t.name)) }));
process.exit(0);`;
const env = { ...process.env, MESH_GITEA_TOKEN: "x", MESH_GITEA_ADMIN_USER: "admin", MESH_GITEA_HOOK_URL: "http://127.0.0.1:1" };
for (const k of ["MESH_SERVED_MODULE", "MESH_GITEA_ADMIN_PASSWORD", "MESH_GITEA_CONFIG_FILE"]) delete env[k];
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { cwd: join(here, ".."), env, stdio: ["ignore", "pipe", "pipe"] });
let out = "";
let err = "";
child.stdout.on("data", (d) => (out += d));
child.stderr.on("data", (d) => (err += d));
const ended = new Promise<void>((r) => child.on("exit", () => r()));
const timer = setTimeout(() => child.kill("SIGKILL"), 3000);
await ended;
clearTimeout(timer);
const line = out.split("\n").find((l) => l.startsWith("{"));
assert.ok(line, `the import did not resolve within 3 s while the forge did not answer; stdout: ${out.slice(0, 400)} stderr: ${err.slice(0, 600)}`);
const got = JSON.parse(line!) as { tools: string[] };
for (const t of ["gitea_events", "gitea_commit_status", "gitea_merge_pull_request"]) {
assert.ok(got.tools.includes(t), `${t} is registered: ${got.tools}`);
}
});
+42
View File
@@ -133,3 +133,45 @@ test("no tool writes mesh/check: gitea_check_status is gone and gitea_commit_sta
const index = readFileSync(new URL("../index.ts", import.meta.url), "utf8"); const index = readFileSync(new URL("../index.ts", import.meta.url), "utf8");
assert.ok(!/gitea_check_status|checkStatusTool/.test(index), "index.ts serves no gitea_check_status"); assert.ok(!/gitea_check_status|checkStatusTool/.test(index), "index.ts serves no gitea_check_status");
}); });
test("the watch's current values are taken without waiting on the forge, so a slow forge never times the watch out (issue 1726)", async () => {
// The tool runtime hands a watch's current values one at a time and waits for each to be taken; it gives the whole set
// 10 s (mesh-tools StateTimeout) and otherwise ends the watch. On 2026-10-23 the owed held ~200 keys, each written to the
// forge before the next was handed over: the set took longer than 10 s, the watch ended at every attempt, and no
// verdict reached the forge's mesh/check again.
const { CheckStatuses, watchOwed } = await import("../check.ts");
const sent: string[] = [];
let release: () => void = () => {};
const slow = new Promise<void>((r) => { release = r; });
const f = { setCommitStatus: async (_o: string, _r: string, s: string, st: any) => { await slow; sent.push(`${s.slice(0, 4)} ${st.state}`); } };
const checks = new CheckStatuses(f, () => {}, noSleep);
const keys = Array.from({ length: 200 }, (_, i) => key(i.toString(16).padStart(4, "0") + sha.slice(4), "novox/mesh-sdk", i + 1));
let handedOver = 0;
let attempts = 0;
const watch = async (handler: (c: any) => Promise<void>) => {
if (++attempts > 1) return; // a second attempt is not what this test waits for: the first must hold
// The runtime's bound on the current values: the forge has answered none of them yet.
const timedOut = new Promise<"timeout">((r) => setTimeout(() => r("timeout"), 200));
const delivered = (async () => { for (const [i, k] of keys.entries()) { await handler(put(k, owed(), i + 1, true)); handedOver++; } return "delivered" as const; })();
if (await Promise.race([delivered, timedOut]) === "timeout") throw new Error("the current values of mesh-delivery-checker.owed did not arrive in 10s");
};
const said: string[] = [];
await watchOwed(watch, checks, noSleep, (l) => said.push(l));
assert.equal(handedOver, 200, "every current value was taken before the forge answered");
assert.ok(!said.some((l) => /could not be bound/.test(l)), said.join(" | "));
assert.equal(sent.length, 0, "nothing is written before the forge answers");
release();
await checks.settled();
assert.equal(sent.length, 200, "every owed status is written once the forge answers");
});
test("the writes a watch's start hands over go to the forge a few at a time, never all at once (issue 1726)", async () => {
const { CheckStatuses, WRITES_AT_ONCE } = await import("../check.ts");
let running = 0, most = 0, done = 0;
const f = { setCommitStatus: async () => { running++; most = Math.max(most, running); await new Promise((r) => setTimeout(r, 1)); running--; done++; } };
const checks = new CheckStatuses(f, () => {}, noSleep);
for (let i = 0; i < 50; i++) await checks.take(put(key(i.toString(16).padStart(4, "0") + sha.slice(4), "novox/mesh-sdk", i + 1), owed(), i + 1, true));
await checks.settled();
assert.equal(done, 50);
assert.equal(most, WRITES_AT_ONCE);
});
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
@@ -0,0 +1 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
+5 -1
View File
@@ -102,7 +102,11 @@ integration level, the postgres major and the nats-server version; and, when the
workspace (`beside`), each one's repository and commit, which the answer echoes as `beside` (the checker keeps a verdict only workspace (`beside`), each one's repository and commit, which the answer echoes as `beside` (the checker keeps a verdict only
for the siblings it was judged with). The text's form is `DigestVersion` 3. The Go version is not in it: the toolchain enters a for the siblings it was judged with). The text's form is `DigestVersion` 3. The Go version is not in it: the toolchain enters a
fingerprint once, as the version the build nodes agree on (ADR 0477 §3), and the standard library's packages are that fingerprint once, as the version the build nodes agree on (ADR 0477 §3), and the standard library's packages are that
version's. A package with no tests at the level has no digest and is named in `without_tests`. A directory that vendors version's. A package with no tests at the level has no digest and is named in `without_tests`. A catalogue module's packages also hash the module's non-Go files (the nearest ancestor directory holding `module.json`: its manifest, README, `files/` and the rest, so `../../module.json` is covered; digest form 5). A package whose directory holds a file named `reads-outside` (any content; say what the tests read) is never kept (issue 1935): its tests read
files outside it, which a digest from `go list` cannot name, so its digest is a fresh random one on every call, the ask finds no kept
verdict for it and runs it, and the answer names it in `reads_outside` and the module's log says why. The checker still writes a
verdict under that digest, which nothing ever reads again. The marker is the plain way to say it until each verdict keeps the files
its tests opened (Go's test log). A directory that vendors
is listed and tested from its vendor directory (`-mod=vendor`). is listed and tested from its vendor directory (`-mod=vendor`).
## A change to this module is judged by its candidate ## A change to this module is judged by its candidate
@@ -100,6 +100,9 @@ func (m *Module) Puller(steps []StepTool, source buildstep.AskSource) *buildstep
if err := m.room(); err != nil { if err := m.room(); err != nil {
return nil, err return nil, err
} }
if a.Compose != nil {
return m.compose(ctx, ask.Fetch, a)
}
return buildstep.RunAsk(ctx, ask, a) return buildstep.RunAsk(ctx, ask, a)
}} }}
} }
@@ -4,7 +4,6 @@ package main
import ( import (
"context" "context"
"os"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -14,16 +13,25 @@ import (
"git.novox.be/novox/mesh-sdk/go/buildstep" "git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/testbus"
) )
// checkCommit is the commit the check ask fetches, apart from the delivery ask's, so the fetch order names which ran first.
const checkCommit = "89abcdef0123456789abcdef0123456789abcdef"
// 1887/AC-order, 1887/AC-pull against the real bus: a check ask published before a delivery ask runs after it, and at // 1887/AC-order, 1887/AC-pull against the real bus: a check ask published before a delivery ask runs after it, and at
// capacity 1 the second waits in the stream until the first ends. // capacity 1 the second waits in the stream until the first ends.
//
// **A bus of its own.** The integration level runs every directory's tests at once on one bus, and this test names the
// build seat's stream and its fixed workers (SEAT_NODE_BUILD_GO and its workers) and deletes the stream at its end, so
// anything else on that bus, a puller or a leftover consumer, could hold or take its delivery ask and let the check
// ask run first. It starts from testbus (mesh-sdk, novox/hq ADR 0495 clause 5): a bus for this test alone, gone after it.
//
// **The order is read from the record.** At capacity 1 the puller starts the check the instant the delivery ask's step
// ends, well before any poll, so "the check key holds a state" is true on correct behaviour (issue 1887). The test
// waits for both asks to end and compares the step records' own Started and Ended times, and the fetch order by commit.
func TestTheModuleTakesADeliveryAskBeforeAnEarlierCheck(t *testing.T) { func TestTheModuleTakesADeliveryAskBeforeAnEarlierCheck(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS") nc, err := nats.Connect(testbus.URL(t))
if url == "" {
t.Skip("MESH_TEST_NATS names no bus: the integration level starts one")
}
nc, err := nats.Connect(url)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -40,6 +48,7 @@ func TestTheModuleTakesADeliveryAskBeforeAnEarlierCheck(t *testing.T) {
t.Cleanup(func() { _ = js.DeleteStream(context.Background(), "SEAT_NODE_BUILD_GO") }) t.Cleanup(func() { _ = js.DeleteStream(context.Background(), "SEAT_NODE_BUILD_GO") })
check := deliveryAsk() check := deliveryAsk()
check.Kind = delivery.AskCheck check.Kind = delivery.AskCheck
check.Build.Fetch["commit"] = checkCommit
first := deliveryAsk() first := deliveryAsk()
for _, a := range []delivery.StepAsk{check, first} { for _, a := range []delivery.StepAsk{check, first} {
raw, _ := delivery.EncodeStepAsk(a) raw, _ := delivery.EncodeStepAsk(a)
@@ -68,30 +77,36 @@ func TestTheModuleTakesADeliveryAskBeforeAnEarlierCheck(t *testing.T) {
p := m.Puller(steps, &buildstep.JetStreamAsks{JS: js, Language: delivery.AskGo}) p := m.Puller(steps, &buildstep.JetStreamAsks{JS: js, Language: delivery.AskGo})
p.PullWait = 200 * time.Millisecond p.PullWait = 200 * time.Millisecond
go p.Pull(ctx) go p.Pull(ctx)
for _, a := range []delivery.StepAsk{first, check} { var ran map[string][]delivery.Step
key, _ := delivery.AskStepKey("ace", a.ID) for deadline := time.Now().Add(10 * time.Second); ; time.Sleep(10 * time.Millisecond) {
for deadline := time.Now().Add(10 * time.Second); ; time.Sleep(10 * time.Millisecond) { ran = map[string][]delivery.Step{}
states.mu.Lock() ended := 0
v := states.steps[key]
states.mu.Unlock()
if len(v) > 0 && v[len(v)-1].State != delivery.StepRunning {
break
}
if time.Now().After(deadline) {
t.Fatalf("the ask %s (%s) never ended", a.ID, a.Kind)
}
}
states.mu.Lock() states.mu.Lock()
checkKey, _ := delivery.AskStepKey("ace", check.ID) for _, a := range []delivery.StepAsk{first, check} {
if a.ID == first.ID && len(states.steps[checkKey]) > 0 { key, _ := delivery.AskStepKey("ace", a.ID)
states.mu.Unlock() v := append([]delivery.Step(nil), states.steps[key]...)
t.Fatal("the check ask started before the delivery ask ended at capacity 1") ran[a.ID] = v
if len(v) > 0 && v[len(v)-1].State != delivery.StepRunning {
ended++
}
} }
states.mu.Unlock() states.mu.Unlock()
if ended == 2 {
break
}
if time.Now().After(deadline) {
t.Fatalf("the asks never both ended: %v", ran)
}
}
// The check's step may start the moment the delivery's ends, so the order is read from the records' own times.
deliveryEnded := ran[first.ID][len(ran[first.ID])-1].Ended
checkStarted := ran[check.ID][0].Started
if checkStarted.Before(deliveryEnded) {
t.Fatalf("the check ask started at %v, before the delivery ask ended at %v at capacity 1", checkStarted, deliveryEnded)
} }
mu.Lock() mu.Lock()
defer mu.Unlock() defer mu.Unlock()
if len(order) != 2 { if len(order) != 2 || order[0] != askCommit || order[1] != checkCommit {
t.Fatalf("fetched %v", order) t.Fatalf("fetched %v, want the delivery ask's commit %s first, then the check's %s", order, askCommit, checkCommit)
} }
} }
@@ -174,7 +174,7 @@ func TestACheckAsksDigestIsTheToolsAnswerInTheCall(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if l := out.(delivery.CheckAnswer).Levels; len(l) != 1 || !strings.Contains(string(l[0].Digest), `"x/a":"d1"`) { if l := out.(delivery.CheckAnswer).Levels; len(l) != 1 || l[0].Digest.Map()["x/a"] != "d1" {
t.Fatalf("levels = %+v, want the digest's packages", l) t.Fatalf("levels = %+v, want the digest's packages", l)
} }
} }
@@ -116,7 +116,7 @@ func TestTheIntegrationLevelDigestsUnderBothTags(t *testing.T) {
t.Errorf("the integration digest listed %q", s.command) t.Errorf("the integration digest listed %q", s.command)
} }
} }
if DigestVersion != "4" { if DigestVersion != "5" {
t.Errorf("the digest's form changed in kind and is still version %s", DigestVersion) t.Errorf("the digest's form changed in kind and is still version %s", DigestVersion)
} }
} }
@@ -0,0 +1,90 @@
package main
// Stray build output (issue 1921, try speed S13): the lint step fails a change that adds an executable (ELF or Mach-O) or
// a file over MaxAddedFile, naming each, unless the repository's .binaries-allowed lists its path. Build binaries were
// committed three times in one night; the forge keeps them for ever.
import (
"bufio"
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// MaxAddedFile is the largest file a change may add without .binaries-allowed naming it.
const MaxAddedFile = 5 << 20
// BinariesAllowed is the file in a repository's root listing, one path per line ("#" starts a comment), the executables
// and large files it keeps on purpose.
const BinariesAllowed = ".binaries-allowed"
// executableMagic are the first bytes of an ELF and of each Mach-O form (32 and 64 bits, both byte orders, universal).
var executableMagic = [][]byte{[]byte("\x7fELF"), {0xfe, 0xed, 0xfa, 0xce}, {0xfe, 0xed, 0xfa, 0xcf}, {0xce, 0xfa, 0xed, 0xfe},
{0xcf, 0xfa, 0xed, 0xfe}, {0xca, 0xfe, 0xba, 0xbe}}
// addedBinaries is the lint finding of the files the merge adds to the trunk that are executables or over MaxAddedFile,
// nil when there is none. The paths come from git (the merge commit against its first parent, the trunk); a tree with no
// merge (no first parent) adds nothing to judge.
func (m *Module) addedBinaries(ctx context.Context, run forgekey.Runner, tree string) []Failure {
r := run(ctx, []string{"GIT_TERMINAL_PROMPT=0"}, "git", "-C", tree, "diff", "--name-only", "--no-renames", "--diff-filter=A", "HEAD^1", "HEAD")
if r.Status != 0 || r.Err != "" {
return nil
}
allowed := map[string]bool{}
if raw, err := os.ReadFile(filepath.Join(tree, BinariesAllowed)); err == nil {
scan := bufio.NewScanner(bytes.NewReader(raw))
for scan.Scan() {
if l := strings.TrimSpace(scan.Text()); l != "" && !strings.HasPrefix(l, "#") {
allowed[filepath.ToSlash(l)] = true
}
}
}
var found []string
for _, p := range strings.Split(r.Stdout, "\n") {
if p = strings.TrimSpace(p); p == "" || allowed[p] {
continue
}
path := filepath.Join(tree, filepath.FromSlash(p))
info, err := os.Lstat(path)
if err != nil || !info.Mode().IsRegular() {
continue // narrowed away, or not a file
}
if info.Size() > MaxAddedFile {
found = append(found, fmt.Sprintf("%s (%d bytes, over %d)", p, info.Size(), MaxAddedFile))
continue
}
if isExecutable(path) {
found = append(found, p+" (an executable)")
}
}
if len(found) == 0 {
return nil
}
sort.Strings(found)
return []Failure{{Package: ".", Test: "binaries", Step: StepLint,
Output: "the change adds build output or a large file (remove it, or list its path in " + BinariesAllowed + "): " + strings.Join(found, ", ")}}
}
func isExecutable(path string) bool {
f, err := os.Open(path)
if err != nil {
return false
}
defer f.Close()
head := make([]byte, 4)
if n, _ := f.Read(head); n < 4 {
return false
}
for _, magic := range executableMagic {
if bytes.Equal(head, magic) {
return true
}
}
return false
}
@@ -0,0 +1,67 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// Issue 1921, try speed S13: the lint step fails a change that adds an executable (ELF or Mach-O) or a file over 5 MB,
// naming it, unless .binaries-allowed in the repository's root lists its path; a file the change does not add is not
// judged, and neither is an allowed one.
func TestTheLintStepFailsAChangeThatAddsABinaryOrALargeFile(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "slow")
tree := filepath.Join(m.Work, ws, "tree")
write := func(name string, body []byte) {
os.MkdirAll(filepath.Dir(filepath.Join(tree, name)), 0o755)
if err := os.WriteFile(filepath.Join(tree, name), body, 0o644); err != nil {
t.Fatal(err)
}
}
write("go.mod", []byte("module fixture/slow\n"))
elf := append([]byte("\x7fELF"), make([]byte, 64)...)
macho := append([]byte{0xcf, 0xfa, 0xed, 0xfe}, make([]byte, 64)...)
write("modules/x/cmd/x/x", elf)
write("modules/y/y", macho)
write("big.tar", make([]byte, 5<<20+1))
write("tools/allowed-tool", elf)
write("old/binary", elf) // there before the change
write("modules/x/doc.md", []byte("text\n"))
write(".binaries-allowed", []byte("# kept on purpose\ntools/allowed-tool\n"))
inner := m.Run
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if name == "git" && strings.Contains(strings.Join(args, " "), "--diff-filter=A") {
return forgekey.Ran{Stdout: "modules/x/cmd/x/x\nmodules/y/y\nbig.tar\ntools/allowed-tool\nmodules/x/doc.md\n.binaries-allowed\n"}
}
return inner(ctx, env, name, args...)
}
res, st := runUnit(t, m, states, ws)
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
var found []Failure
for _, f := range res.Failed {
if f.Test == "binaries" {
found = append(found, f)
}
}
if res.Passed || len(found) != 1 || found[0].Step != StepLint {
t.Fatalf("failed = %+v, want one lint finding of the binaries", res.Failed)
}
out := found[0].Output
for _, want := range []string{"modules/x/cmd/x/x", "modules/y/y", "big.tar"} {
if !strings.Contains(out, want) {
t.Errorf("the finding does not name %s: %s", want, out)
}
}
for _, not := range []string{"allowed-tool", "old/binary", "doc.md"} {
if strings.Contains(out, not) {
t.Errorf("the finding names %s: %s", not, out)
}
}
}
@@ -0,0 +1,40 @@
package main
import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1921/S11 (the lead's decision): a check compiles and runs with CGO_ENABLED=0, as a production build does, so it tests
// what ships and the warm cache matches; the lint's go vet, a compile alone, writes the shared cache, while a go test that
// runs a test binary reads it beneath the ask's own layer.
func TestACheckRunsWithoutCgoAndOnlyItsCompileWritesTheSharedCache(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "failing")
var seen []shutOff
m.Run = throughBubblewrap(t, m.Run, &seen)
if _, st := runUnit(t, m, states, ws); st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
for _, c := range *calls {
if strings.HasPrefix(c.line, "go ") && !slices.Contains(c.env, "CGO_ENABLED=0") {
t.Errorf("%s ran without CGO_ENABLED=0: %v", c.line, c.env)
}
}
vets, tests := 0, 0
for _, s := range seen {
switch {
case strings.Contains(s.command, " vet "):
vets++
checkShutOff(t, m, s, true) // the shared caches bound read-write
case strings.Contains(s.command, " test "):
tests++
checkShutOff(t, m, s, false)
}
}
if vets == 0 || tests == 0 {
t.Errorf("%d vets and %d tests ran shut off, want both", vets, tests)
}
}
@@ -0,0 +1,210 @@
package main
// **A compose ask runs mesh-sdk's composer at a commit** (novox/hq ADR 0482 clause 4, issues 1621 and 1622): the
// assigner asks it, to compose a group with its mesh-sdk member's head, and to read the group's manifests with the
// mesh-sdk the running controller and node-engine pin (ADR 0482 clause 5, issue 1623). The module fetches the commit
// (its fetch step: the forge key and the network, none of the change's code), builds the composer package in the Go
// module the ask names, in a run step shut off with nothing fetched, and keeps the program under its key
// (delivery.ComposerKey), so each commit is built once on a node. It fills the request's manifests from the delivery
// indexer's manifests state, by digest, and runs the program shut off: no network, none of the account, the request a
// file of a workspace of its own. The answer is the composer's verdict whole (delivery.ComposeAnswer); the module
// judges nothing of it.
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// ManifestsState is the delivery indexer's manifests by digest, which a compose ask's request names (the module reads
// it, module.json reads).
const ManifestsState = delivery.IndexerModule + ".manifests"
// composerLocks hold one mutex per composer key, so a composer two asks need at once is built once.
var composerLocks sync.Map
// composerPath is where the composer of key is kept: the cache directory, beside Go's own caches.
func (m *Module) composerPath(key string) string {
return filepath.Join(m.Cache, "composers", key, "mesh-compose")
}
// compose runs a compose ask: the composer of its commit, built once, over its request with the manifests filled.
func (m *Module) compose(ctx context.Context, fetch buildstep.StepFunc, a delivery.StepAsk) (any, error) {
job := *a.Compose
repository, _ := job.Fetch["repository"].(string)
commit, _ := job.Fetch["commit"].(string)
installed, err := m.Probe(ctx)
if err != nil {
return nil, err
}
key := delivery.ComposerKey(repository, commit, job.Dir, job.Package, delivery.ToolchainUsed{Version: installed, Recipe: m.Recipe.Version()})
exe, built, err := m.composerOf(ctx, fetch, job, key)
if err != nil {
return nil, err
}
req, err := composer.Decode(job.Request)
if err != nil {
return nil, buildstep.Refuse("the compose ask %s: %v", a.ID, err)
}
if err := composer.Fill(&req, m.manifestOf); err != nil {
return nil, err
}
verdict, err := m.runComposer(ctx, exe, req)
if err != nil {
return nil, err
}
return delivery.ComposeAnswer{Built: built, Key: key, Verdict: verdict}, nil
}
// manifestOf is the manifest the indexer keeps under digest, as its repository holds it.
func (m *Module) manifestOf(digest string) ([]byte, error) {
if m.ReadManifest == nil {
return nil, errors.New("this module reads no manifests state")
}
key, err := delivery.ManifestsKey(digest)
if err != nil {
return nil, err
}
raw, err := m.ReadManifest(key)
if err != nil {
return nil, err
}
if raw == nil {
return nil, fmt.Errorf("the indexer keeps no manifest of digest %s", digest)
}
man, err := delivery.DecodeManifest(raw)
if err != nil {
return nil, err
}
return []byte(man.JSON), nil
}
// composerOf is the composer kept under key, or, when none is, the one built now from the fetched commit (built true).
func (m *Module) composerOf(ctx context.Context, fetch buildstep.StepFunc, job delivery.ComposeJob, key string) (string, bool, error) {
exe := m.composerPath(key)
mu, _ := composerLocks.LoadOrStore(exe, &sync.Mutex{})
mu.(*sync.Mutex).Lock()
defer mu.(*sync.Mutex).Unlock()
if st, err := os.Stat(exe); err == nil && st.Mode().IsRegular() {
return exe, false, nil
}
if fetch == nil {
return "", false, buildstep.Refuse("this build module fetches nothing")
}
out, err := fetch(ctx, job.Fetch)
if err != nil {
return "", false, err
}
raw, err := json.Marshal(out)
if err != nil {
return "", false, err
}
var f struct {
Workspace string `json:"workspace"`
}
if err := json.Unmarshal(raw, &f); err != nil || f.Workspace == "" {
return "", false, fmt.Errorf("the fetch answered no workspace: %s", raw)
}
defer m.releaseSaid(f.Workspace)
done := m.spaces().Hold(f.Workspace)
defer done()
dir := filepath.Join(m.Work, f.Workspace, "tree", job.Dir)
if _, err := os.Stat(filepath.Join(dir, "go.mod")); err != nil {
return "", false, fmt.Errorf("%s at %s holds no go.mod in %s, so no composer can be built there", job.Fetch["repository"], job.Fetch["commit"], job.Dir)
}
into := filepath.Join(m.Work, f.Workspace, "composer", "mesh-compose")
if err := os.MkdirAll(filepath.Dir(into), 0o700); err != nil {
return "", false, err
}
run, err := m.stepRunner(f.Workspace, false)
if err != nil {
return "", false, err
}
env := append(m.goEnvFor(dir), "GOPROXY=off", "GOTOOLCHAIN=local", "CGO_ENABLED=0")
r := run(ctx, env, "go", "-C", dir, "build", "-trimpath", "-o", into, job.Package)
if r.Status != 0 || r.Err != "" {
// The words Go says when the module holds no such package (a mesh-sdk from before the composer) are kept whole,
// so the asker tells that apart from a fault.
return "", false, fmt.Errorf("the composer %s does not build at %s: %s", job.Package, job.Fetch["commit"], compileError(r.Err+"\n"+r.Stderr+"\n"+r.Stdout))
}
if err := copyExecutable(into, exe); err != nil {
return "", false, fmt.Errorf("the composer could not be kept: %w", err)
}
return exe, true, nil
}
// copyExecutable copies a built program into place through a file beside it, so a half-written program is never kept.
func copyExecutable(from, to string) error {
if err := os.MkdirAll(filepath.Dir(to), 0o700); err != nil {
return err
}
src, err := os.Open(from)
if err != nil {
return err
}
defer src.Close()
tmp := to + ".new"
dst, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o700)
if err != nil {
return err
}
if _, err := io.Copy(dst, src); err != nil {
dst.Close()
return err
}
if err := dst.Close(); err != nil {
return err
}
return os.Rename(tmp, to)
}
// releaseSaid removes a workspace once its steps ended, saying a failure: the sweep removes it later.
func (m *Module) releaseSaid(workspace string) {
if err := m.spaces().Release(workspace); err != nil && m.Say != nil {
m.Say("the compose ask's workspace %s: %v", workspace, err)
}
}
// runComposer runs the composer over req in a workspace of its own, shut off, and answers its verdict: the last line of
// its standard output, a composer.Verdict of the version this module reads.
func (m *Module) runComposer(ctx context.Context, exe string, req composer.Request) (json.RawMessage, error) {
id := buildstep.NewID()
done := m.spaces().Hold(id)
defer m.releaseSaid(id)
defer done()
ws := filepath.Join(m.Work, id)
if err := os.MkdirAll(ws, 0o700); err != nil {
return nil, err
}
raw, err := json.Marshal(req)
if err != nil {
return nil, err
}
file := filepath.Join(ws, "request.json")
if err := os.WriteFile(file, raw, 0o600); err != nil {
return nil, err
}
run := m.Run
if !m.Inside {
iso := m.isolation(id, false)
iso.ReadOnly = append(iso.ReadOnly, exe)
run = buildstep.Isolated(m.Run, iso)
}
r := run(ctx, nil, exe, "-request", file)
line := strings.TrimSpace(lastLine(r.Stdout))
var v composer.Verdict
if r.Status != 0 || r.Err != "" || json.Unmarshal([]byte(line), &v) != nil || v.V != composer.Version {
return nil, fmt.Errorf("the composer answered no verdict (status %d %s): %s", r.Status, r.Err, firstLine(r.Stderr+"\n"+r.Stdout))
}
return json.RawMessage(line), nil
}
@@ -0,0 +1,156 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
"git.novox.be/novox/mesh-sdk/go/testbus"
)
// sdkPin is the mesh-sdk version this module's go.mod requires: the composer the test builds is that version's.
func sdkPin(t *testing.T) string {
t.Helper()
raw, err := os.ReadFile("../../go.mod")
if err != nil {
t.Fatal(err)
}
for _, l := range strings.Split(string(raw), "\n") {
if f := strings.Fields(l); len(f) >= 2 && f[0] == "git.novox.be/novox/mesh-sdk/go" {
return f[1]
}
}
t.Fatal("go.mod requires no mesh-sdk")
return ""
}
// goEnvOf is one value of this machine's go env.
func goEnvOf(t *testing.T, name string) string {
t.Helper()
out, err := exec.Command("go", "env", name).Output()
if err != nil {
t.Fatalf("go env %s: %v", name, err)
}
return strings.TrimSpace(string(out))
}
// 1622/AC1, AC3 against the real bus: a compose ask put on the Go build seat's check stream is pulled by the module
// started whole on that bus, which fetches the commit (here a tree of a module requiring mesh-sdk at this module's pin,
// as a reader repository is), builds mesh-sdk's composer there with nothing fetched, fills the request's manifests from
// the indexer's state and runs the composer; the step key <node>.<ask id> passes with the composer's verdict. A second
// ask of the same commit runs the kept composer: fetched and built once.
func TestAComposeAskIsBuiltOnceAndAnsweredOnTheBus(t *testing.T) {
nc, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer nc.Close()
js, err := jetstream.New(nc)
if err != nil {
t.Fatal(err)
}
ctx, stop := context.WithCancel(context.Background())
defer stop()
if err := buildstep.DeclareAskWorkers(ctx, js, delivery.AskGo, 2*time.Minute); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = js.DeleteStream(context.Background(), "SEAT_NODE_BUILD_GO") })
states := newStates()
m := moduleWith(states, "go1.26.0")
m.Work, m.Cache, m.Inside = t.TempDir(), t.TempDir(), true
m.Run = StepRunner(t.TempDir())
m.GoModCache = goEnvOf(t, "GOMODCACHE") // the module cache this module's own build filled, read and not fetched
run, gocache := m.Run, goEnvOf(t, "GOCACHE") // this machine's build cache, so the composer's packages compile once
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
return run(ctx, append(env, "GOCACHE="+gocache), name, args...)
}
read, digests := indexerHolds(t, `{"module":"demo","version":"1"}`)
m.ReadManifest = read
gosum, err := os.ReadFile("../../go.sum")
if err != nil {
t.Fatal(err)
}
fetches := 0
steps := fakeSteps(new([]string))
for i := range steps {
if steps[i].Name == "build_go_fetch" {
steps[i].Run = func(ctx context.Context, args map[string]any) (any, error) {
fetches++
ws := buildstep.NewID()
tree := filepath.Join(m.Work, ws, "tree")
if err := os.MkdirAll(tree, 0o700); err != nil {
return nil, err
}
mod := "module reader\n\ngo 1.26.0\n\nrequire git.novox.be/novox/mesh-sdk/go " + sdkPin(t) + "\n"
if err := os.WriteFile(filepath.Join(tree, "go.mod"), []byte(mod), 0o600); err != nil {
return nil, err
}
return map[string]any{"workspace": ws}, os.WriteFile(filepath.Join(tree, "go.sum"), gosum, 0o600)
}
}
}
p := m.Puller(steps, &buildstep.JetStreamAsks{JS: js, Language: delivery.AskGo})
p.PullWait = 200 * time.Millisecond
go p.Pull(ctx)
ask := func() delivery.Step {
a := composeAskOf(t, digests[0])
a.Compose.Fetch["repository"], a.Compose.Dir = "novox/mesh-controller", delivery.RootPath
raw, err := delivery.EncodeStepAsk(a)
if err != nil {
t.Fatal(err)
}
subject, _ := delivery.AskSubject(delivery.AskGo, delivery.AskCheck)
if _, err := js.Publish(ctx, subject, raw); err != nil {
t.Fatal(err)
}
key, _ := delivery.AskStepKey("ace", a.ID)
for deadline := time.Now().Add(90 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
states.mu.Lock()
v := states.steps[key]
states.mu.Unlock()
if len(v) > 0 && v[len(v)-1].State != delivery.StepRunning {
return v[len(v)-1]
}
}
t.Fatalf("the compose ask %s never ended", a.ID)
return delivery.Step{}
}
for n, wantBuilt := range []bool{true, false} {
step := ask()
if step.State != delivery.StepPassed {
t.Fatalf("ask %d: %s: %s", n+1, step.State, step.Reason)
}
var ans delivery.ComposeAnswer
var v composer.Verdict
if err := json.Unmarshal(step.Result, &ans); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(ans.Verdict, &v); err != nil {
t.Fatal(err)
}
if ans.Built != wantBuilt || !strings.HasPrefix(ans.Key, "composer-") || len(v.Cases) != 1 || !v.Cases[0].Manifests["demo"].Accepted ||
!strings.Contains(v.SDK, sdkPin(t)) {
t.Fatalf("ask %d: answer built %v key %s, verdict %+v; want built %v and demo accepted by mesh-sdk %s", n+1, ans.Built, ans.Key, v, wantBuilt, sdkPin(t))
}
}
if fetches != 1 {
t.Errorf("two asks of one commit fetched %d times, want once", fetches)
}
}
@@ -0,0 +1,125 @@
package main
import (
"context"
"encoding/json"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/composer"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const composeCommit = "4444444444444444444444444444444444444444"
// indexerHolds is the delivery indexer's manifests state holding each body under its digest, as the indexer writes it.
func indexerHolds(t *testing.T, bodies ...string) (func(string) (json.RawMessage, error), []string) {
t.Helper()
kept, digests := map[string]json.RawMessage{}, []string{}
for _, b := range bodies {
d := delivery.ManifestDigest([]byte(b))
key, _ := delivery.ManifestsKey(d)
raw, err := delivery.EncodeManifest(delivery.Manifest{Module: "demo", JSON: b})
if err != nil {
t.Fatal(err)
}
kept[key], digests = raw, append(digests, d)
}
return func(key string) (json.RawMessage, error) { return kept[key], nil }, digests
}
func composeAskOf(t *testing.T, digest string) delivery.StepAsk {
t.Helper()
req, _ := json.Marshal(composer.Request{V: composer.Version, Cases: []composer.Case{{Name: "change",
Manifests: map[string]composer.Manifest{"demo": {Digest: digest}}}}})
return delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: "mesh-delivery-assigner",
At: time.Now(), Compose: &delivery.ComposeJob{Fetch: map[string]any{"repository": "novox/mesh-sdk", "commit": composeCommit},
Dir: "go", Package: composer.Package, Request: req}}
}
// composeModule is a module with work and cache directories of its own, running its commands as they are (Inside: a
// unit test has no bubblewrap to shut them off in).
func composeModule(t *testing.T) *Module {
t.Helper()
m := moduleWith(newStates(), "go1.26.0")
m.Work, m.Cache, m.Inside, m.Run = t.TempDir(), t.TempDir(), true, StepRunner(t.TempDir())
return m
}
// 1622/AC3: a composer kept under the ask's key is run again, nothing fetched or built: the answer says it was not
// built, names the key, and carries the composer's verdict whole; the request it read had its manifests filled from the
// indexer's state.
func TestAKeptComposerIsRunWithTheManifestsFilled(t *testing.T) {
m := composeModule(t)
read, digests := indexerHolds(t, `{"module":"demo","version":"1"}`)
m.ReadManifest = read
a := composeAskOf(t, digests[0])
key := delivery.ComposerKey("novox/mesh-sdk", composeCommit, "go", composer.Package, delivery.ToolchainUsed{Version: "go1.26.0", Recipe: m.Recipe.Version()})
exe := m.composerPath(key)
if err := os.MkdirAll(filepath.Dir(exe), 0o700); err != nil {
t.Fatal(err)
}
// A composer that answers which manifests its request held, filled or not, as a verdict.
script := "#!/bin/sh\n" +
`if grep -q '"digest"' "$2"; then echo '{"v":1,"sdk":"stand-in","cases":[{"name":"unfilled","manifests":{}}]}'; ` +
`else echo '{"v":1,"sdk":"stand-in","cases":[{"name":"change","manifests":{"demo":{"accepted":true}}}]}'; fi` + "\n"
if err := os.WriteFile(exe, []byte(script), 0o700); err != nil {
t.Fatal(err)
}
fetched := 0
fetch := func(context.Context, map[string]any) (any, error) { fetched++; return nil, nil }
out, err := m.compose(context.Background(), fetch, a)
if err != nil {
t.Fatal(err)
}
ans := out.(delivery.ComposeAnswer)
var v composer.Verdict
if err := json.Unmarshal(ans.Verdict, &v); err != nil {
t.Fatal(err)
}
if ans.Built || ans.Key != key || fetched != 0 || len(v.Cases) != 1 || v.Cases[0].Name != "change" || !v.Cases[0].Manifests["demo"].Accepted {
t.Fatalf("answer = %+v (fetched %d), verdict %+v", ans, fetched, v)
}
if entries, _ := os.ReadDir(m.Work); len(entries) != 0 {
t.Errorf("the run's workspace was left: %v", entries)
}
}
// 1622: a manifest the indexer does not keep fails the ask, naming its digest; a fetched commit with no go.mod where
// the ask builds fails it naming the directory, and its workspace is released.
func TestAComposeAskFailsLoudly(t *testing.T) {
m := composeModule(t)
m.ReadManifest, _ = indexerHolds(t)
a := composeAskOf(t, "feedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeedfeed")
key := delivery.ComposerKey("novox/mesh-sdk", composeCommit, "go", composer.Package, delivery.ToolchainUsed{Version: "go1.26.0", Recipe: m.Recipe.Version()})
os.MkdirAll(filepath.Dir(m.composerPath(key)), 0o700)
os.WriteFile(m.composerPath(key), []byte("#!/bin/sh\nexit 0\n"), 0o700)
if _, err := m.compose(context.Background(), nil, a); err == nil || !strings.Contains(err.Error(), "feedfeed") {
t.Errorf("an unkept manifest: %v", err)
}
m2 := composeModule(t)
fetch := func(context.Context, map[string]any) (any, error) {
if err := os.MkdirAll(filepath.Join(m2.Work, "ws0000000000a", "tree", "go"), 0o700); err != nil {
return nil, err
}
return map[string]any{"workspace": "ws0000000000a"}, nil
}
if _, err := m2.compose(context.Background(), fetch, composeAskOf(t, "x")); err == nil || !strings.Contains(err.Error(), "no go.mod in go") {
t.Errorf("a commit with no go.mod: %v", err)
}
if entries, _ := os.ReadDir(m2.Work); len(entries) != 0 {
t.Errorf("the fetched workspace was left: %v", entries)
}
}
// 1622: the module reads the delivery indexer's manifests, which a compose ask names by digest.
func TestTheModuleReadsTheIndexersManifests(t *testing.T) {
if m := readManifest(t); !slices.Contains(m.Reads, ManifestsState) {
t.Errorf("reads = %v, lacks %s", m.Reads, ManifestsState)
}
}
@@ -7,6 +7,7 @@ package main
import ( import (
"bytes" "bytes"
"context" "context"
"crypto/rand"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
@@ -24,12 +25,20 @@ import (
// 2: no Go version in the text; the toolchain enters a fingerprint as the agreed version (ADR 0477 §3). 3: the siblings' // 2: no Go version in the text; the toolchain enters a fingerprint as the agreed version (ADR 0477 §3). 3: the siblings'
// commits (the lead's decision L5) and the integration level's tags integration,isolation (the checker's agreement). // commits (the lead's decision L5) and the integration level's tags integration,isolation (the checker's agreement).
// 4: the siblings' commits enter only the digest of a package whose tests reach a file naming MESH_CHECK_BESIDE (ADR 0536 §1). // 4: the siblings' commits enter only the digest of a package whose tests reach a file naming MESH_CHECK_BESIDE (ADR 0536 §1).
const DigestVersion = "4" // 5: a catalogue module's non-Go files enter the digest of its packages (issue 1935).
const DigestVersion = "5"
// besideVariable is the name a file bears when it reads where the siblings lie; a package whose tests reach such a file // besideVariable is the name a file bears when it reads where the siblings lie; a package whose tests reach such a file
// depends on the siblings' commits. // depends on the siblings' commits.
const besideVariable = "MESH_CHECK_BESIDE" const besideVariable = "MESH_CHECK_BESIDE"
// ReadsOutsideMarker is the name of a file in a package's directory that says its tests read files outside it (a
// repository's documents, a sibling directory): such a package is never kept (issue 1935). The digest hashes only what
// the Go toolchain lists, so it cannot name what a test opens elsewhere; a marker file, beside the Go files where a
// reader of the package sees it, is the plain way to say so until the checker keeps the files a test opened (the Go
// test log) with each verdict.
const ReadsOutsideMarker = "reads-outside"
// integrationTags are the build tags of the integration level, its run and its digest alike: integration, and isolation // integrationTags are the build tags of the integration level, its run and its digest alike: integration, and isolation
// for the bubblewrap proofs (mesh-sdk's and the build modules'), which a build node can run. // for the bubblewrap proofs (mesh-sdk's and the build modules'), which a build node can run.
var integrationTags = []string{"integration", "isolation"} var integrationTags = []string{"integration", "isolation"}
@@ -89,6 +98,9 @@ type DigestResult struct {
// only when they are the siblings it gave the fetch. // only when they are the siblings it gave the fetch.
Beside []string `json:"beside,omitempty"` Beside []string `json:"beside,omitempty"`
WithoutTests []string `json:"without_tests,omitempty"` WithoutTests []string `json:"without_tests,omitempty"`
// ReadsOutside names the packages that carry the marker: each has a digest that no kept verdict can have (a fresh
// random one), so the check runs it every time.
ReadsOutside []string `json:"reads_outside,omitempty"`
// JudgedBy is the program that ran the digest (ADR 0535 §5): "installed", or "candidate <merge commit>" under the // JudgedBy is the program that ran the digest (ADR 0535 §5): "installed", or "candidate <merge commit>" under the
// runner workspace. // runner workspace.
JudgedBy string `json:"judged_by,omitempty"` JudgedBy string `json:"judged_by,omitempty"`
@@ -199,6 +211,12 @@ func (m *Module) digestHere(ctx context.Context, workspace, level string) (Diges
} }
} }
sort.Strings(res.WithoutTests) sort.Strings(res.WithoutTests)
sort.Strings(res.ReadsOutside)
for _, p := range res.ReadsOutside {
if m.Say != nil {
m.Say("the %s level of %s always runs: its directory carries %s, so its tests read files outside it that no digest names", level, p, ReadsOutsideMarker)
}
}
return res, nil return res, nil
} }
@@ -256,6 +274,17 @@ func digestModule(res *DigestResult, pkgs map[string]*goPackage, tree string, he
if testMain == nil { if testMain == nil {
return fmt.Errorf("go list gave no test binary for %s", path) return fmt.Errorf("go list gave no test binary for %s", path)
} }
if _, err := os.Stat(filepath.Join(root.Dir, ReadsOutsideMarker)); err == nil {
// The digest of a package that reads outside itself is one nothing can have kept: the key is a fresh random
// hex, so the ask finds no kept verdict and runs it, and what the checker writes under it is never read again.
var nonce [32]byte
if _, err := rand.Read(nonce[:]); err != nil {
return err
}
res.Packages[path] = hex.EncodeToString(nonce[:])
res.ReadsOutside = append(res.ReadsOutside, path)
continue
}
d, err := packageDigest(root, testMain, pkgs, tree, header, beside) d, err := packageDigest(root, testMain, pkgs, tree, header, beside)
if err != nil { if err != nil {
return fmt.Errorf("%s: %w", path, err) return fmt.Errorf("%s: %w", path, err)
@@ -303,6 +332,15 @@ func packageDigest(root, testMain *goPackage, pkgs map[string]*goPackage, tree s
} }
} }
} }
// A catalogue module's tests read its manifest, its README and its files from the package's directory (../../module.json):
// every non-Go file of the module's directory is an input (issue 1935). The module is the nearest ancestor with a module.json.
if modDir := enclosingModule(tree, root.Dir); modDir != "" {
more, err := moduleFileLines(tree, root, modDir)
if err != nil {
return "", err
}
lines = append(lines, more...)
}
data := filepath.Join(root.Dir, "testdata") data := filepath.Join(root.Dir, "testdata")
if _, err := os.Stat(data); err == nil { if _, err := os.Stat(data); err == nil {
err := filepath.WalkDir(data, func(path string, e fs.DirEntry, err error) error { err := filepath.WalkDir(data, func(path string, e fs.DirEntry, err error) error {
@@ -328,6 +366,41 @@ func packageDigest(root, testMain *goPackage, pkgs map[string]*goPackage, tree s
return hex.EncodeToString(h.Sum(nil)), nil return hex.EncodeToString(h.Sum(nil)), nil
} }
// enclosingModule is the nearest ancestor of dir, inside tree, that holds a module.json, or "" when there is none.
func enclosingModule(tree, dir string) string {
tree = filepath.Clean(tree)
for d := filepath.Clean(dir); d != tree && strings.HasPrefix(d, tree+string(filepath.Separator)); d = filepath.Dir(d) {
if st, err := os.Stat(filepath.Join(d, "module.json")); err == nil && !st.IsDir() {
return d
}
}
return ""
}
// moduleFileLines are the content lines of every file of a module's directory that is not Go source (Go source is the
// packages' own, listed by go list), skipping .git and node_modules.
func moduleFileLines(tree string, p *goPackage, modDir string) ([]string, error) {
var lines []string
err := filepath.WalkDir(modDir, func(path string, e fs.DirEntry, err error) error {
if err != nil {
return err
}
if e.IsDir() {
if n := e.Name(); n == ".git" || n == "node_modules" {
return filepath.SkipDir
}
return nil
}
if strings.HasSuffix(path, ".go") {
return nil
}
l, err := contentLine(tree, p, path)
lines = append(lines, l)
return err
})
return lines, err
}
func fileLine(tree string, p *goPackage, name string) (string, error) { func fileLine(tree string, p *goPackage, name string) (string, error) {
return contentLine(tree, p, filepath.Join(p.Dir, name)) return contentLine(tree, p, filepath.Join(p.Dir, name))
} }
@@ -189,3 +189,71 @@ func TestAModuleOutsideTheTreeCountsByVersionAndChecksum(t *testing.T) {
t.Errorf("a new version left the digest as it was") t.Errorf("a new version left the digest as it was")
} }
} }
// Issue 1935: a package whose directory carries the reads-outside marker is never kept: its digest is one no kept verdict
// can have (it differs on every call, though nothing changed), and the answer names it. A package without the marker
// keeps its digest.
func TestAPackageWithTheReadsOutsideMarkerIsNeverKept(t *testing.T) {
t.Parallel()
m, tree := digestFixture(t)
plain := digests(t, m, "unit")
if len(plain.ReadsOutside) != 0 {
t.Fatalf("a package carries the marker in the fixture: %v", plain.ReadsOutside)
}
edit(t, tree, "c/"+ReadsOutsideMarker, "the tests read ../docs\n")
first, second := digests(t, m, "unit"), digests(t, m, "unit")
c := "fixture/digest/c"
if len(first.Packages[c]) != 64 || first.Packages[c] == second.Packages[c] || first.Packages[c] == plain.Packages[c] {
t.Errorf("c carries the marker and its digest is stable: %s, %s, before %s", first.Packages[c], second.Packages[c], plain.Packages[c])
}
if strings.Join(first.ReadsOutside, ",") != c {
t.Errorf("the answer does not name the package that reads outside: %v", first.ReadsOutside)
}
a := "fixture/digest/a"
if first.Packages[a] != plain.Packages[a] || first.Packages[a] != second.Packages[a] {
t.Errorf("a has no marker and its digest moved: %s, %s, before %s", first.Packages[a], second.Packages[a], plain.Packages[a])
}
}
// Issue 1935: a catalogue module's packages read ../../module.json and the module's files: a change to them changes the
// digest of the module's package, and a change to another module's files does not.
func TestACatalogueModulesFilesChangeTheDigestOfItsPackagesOnly(t *testing.T) {
t.Parallel()
m, tree := digestFixture(t)
for _, mod := range []string{"mod1", "mod2"} {
dir := filepath.Join(tree, mod, "cmd", "p")
if err := os.MkdirAll(filepath.Join(tree, mod, "files"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
edit(t, tree, mod+"/module.json", `{"module":"`+mod+`"}`)
edit(t, tree, mod+"/README.md", "# "+mod+"\n")
edit(t, tree, mod+"/files/f.env", "A=1\n")
edit(t, tree, mod+"/cmd/p/p.go", "package p\n")
edit(t, tree, mod+"/cmd/p/p_test.go", "package p\n\nimport (\n\t\"os\"\n\t\"testing\"\n)\n\nfunc TestReads(t *testing.T) {\n\tif _, err := os.ReadFile(\"../../module.json\"); err != nil {\n\t\tt.Fatal(err)\n\t}\n}\n")
}
p1, p2 := "fixture/digest/mod1/cmd/p", "fixture/digest/mod2/cmd/p"
before := digests(t, m, "unit")
if len(before.Packages[p1]) != 64 || len(before.Packages[p2]) != 64 {
t.Fatalf("no digest for the module packages: %+v", before.Packages)
}
if before.Packages[p1] == before.Packages[p2] {
t.Errorf("two modules, one digest")
}
for _, file := range []string{"mod1/module.json", "mod1/README.md", "mod1/files/f.env"} {
edit(t, tree, file, "changed "+file+"\n")
after := digests(t, m, "unit")
if after.Packages[p1] == before.Packages[p1] {
t.Errorf("a change to %s did not change the module's digest", file)
}
if after.Packages[p2] != before.Packages[p2] {
t.Errorf("a change to %s changed the other module's digest", file)
}
before = after
}
if got := digests(t, m, "unit"); got.Packages["fixture/digest/c"] == "" {
t.Errorf("a package outside any module lost its digest")
}
}
@@ -1,6 +1,7 @@
package main package main
import ( import (
"encoding/json"
"os" "os"
"strings" "strings"
"testing" "testing"
@@ -27,7 +28,7 @@ func TestAFailedTestKeepsItsFatalMessageAfterItsLogLines(t *testing.T) {
} }
} }
if want := "3 subscription(s) derive to a subject no emitter owns:"; f.Cause != want && !strings.HasSuffix(f.Cause, want) { if want := "3 subscription(s) derive to a subject no emitter owns:"; f.Cause != want && !strings.HasSuffix(f.Cause, want) {
t.Errorf("1901/AC2: cause = %q, want the t.Fatalf line, not the t.Logf line", f.Cause) t.Errorf("1901/AC1: cause = %q, want the t.Fatalf line, not the t.Logf line", f.Cause)
} }
} }
@@ -38,11 +39,74 @@ func TestAFailureLongerThanTheBoundKeepsItsLastLines(t *testing.T) {
lines = append(lines, " x_test.go:1: log line "+strings.Repeat("y", 40)) lines = append(lines, " x_test.go:1: log line "+strings.Repeat("y", 40))
} }
lines = append(lines, " x_test.go:9: the real cause", "--- FAIL: TestX (0.00s)") lines = append(lines, " x_test.go:9: the real cause", "--- FAIL: TestX (0.00s)")
out := clip(lines) out := clip(lines, "")
if len(out) > outputBytes+100 { if len(out) > outputBytes+100 {
t.Errorf("1901/AC1: kept %d bytes, bound is %d", len(out), outputBytes) t.Errorf("1901/AC1: kept %d bytes, bound is %d", len(out), outputBytes)
} }
if !strings.HasSuffix(out, "the real cause\n--- FAIL: TestX (0.00s)") || !strings.HasPrefix(out, "... (") { if !strings.HasSuffix(out, "the real cause\n--- FAIL: TestX (0.00s)") || !strings.Contains(out, "\n... (") {
t.Errorf("1901/AC1: out = %q...%q", out[:40], out[len(out)-60:]) t.Errorf("1901/AC1: out = %q...%q", out[:40], out[len(out)-60:])
} }
} }
// An older checker, which knows no "cause" field, still reads the new build module's answer, and the output it gets
// holds at least what the older build module kept (its first lines, for an output shorter than the bound).
func TestAnOlderCheckerReadsTheNewFailureAndGetsAtLeastWhatItKept(t *testing.T) { // 1901/AC3
raw, err := os.ReadFile("testdata/captured/go-test-json-log-then-fatal.jsonl")
if err != nil {
t.Fatal(err)
}
res := &TestResult{}
rd := newReader(res, ".")
rd.read(string(raw))
rd.finish()
b, err := json.Marshal(res.Failed)
if err != nil {
t.Fatal(err)
}
var old []struct {
Package string `json:"package"`
Test string `json:"test,omitempty"`
Output string `json:"output,omitempty"`
}
if err := json.Unmarshal(b, &old); err != nil || len(old) != 1 {
t.Fatalf("1901/AC3: an older checker refused the answer: %v, %d failures", err, len(old))
}
first := strings.SplitN(old[0].Output, "\n", 2)[0]
if first == "" || !strings.Contains(old[0].Output, "judged against") || !strings.Contains(old[0].Output, "3 subscription(s)") {
t.Errorf("1901/AC3: older checker's output = %q", old[0].Output)
}
}
// A long failure, whose earlier lines are left out, still starts with a real line for an older checker, which keeps only the
// first non-empty line of the output cut to 120 characters, and not with the marker.
func TestALongFailureStillStartsWithARealLineForAnOlderChecker(t *testing.T) { // 1901/AC3
var lines []string
for i := 0; i < 400; i++ {
lines = append(lines, " x_test.go:1: log line "+strings.Repeat("y", 40))
}
lines = append(lines, " x_test.go:9: the real cause", "--- FAIL: TestX (0.00s)")
oldFirst := func(out string) string { // the older checker's rule (judge.go firstLine)
for _, l := range strings.Split(out, "\n") {
if l = strings.TrimSpace(l); l != "" {
if len(l) > 120 {
l = l[:120]
}
return l
}
}
return ""
}
for _, cause := range []string{"the real cause", ""} {
out := clip(lines, cause)
got := oldFirst(out)
if strings.HasPrefix(got, "... (") || got == "" {
t.Errorf("1901/AC3: cause %q: an older checker would read %q", cause, got)
}
if cause != "" && got != cause {
t.Errorf("1901/AC3: older checker reads %q, want the cause", got)
}
if !strings.HasSuffix(out, "--- FAIL: TestX (0.00s)") {
t.Errorf("1901/AC3: the last lines are lost: %q", out[len(out)-40:])
}
}
}
@@ -49,6 +49,51 @@ type FetchResult struct {
// Beside are the sibling repositories placed as plain directories under the workspace's beside directory, which // Beside are the sibling repositories placed as plain directories under the workspace's beside directory, which
// build_go_test hands to the tests as MESH_CHECK_BESIDE (ADR 0484 clause 4). // build_go_test hands to the tests as MESH_CHECK_BESIDE (ADR 0484 clause 4).
Beside []Sibling `json:"beside,omitempty"` Beside []Sibling `json:"beside,omitempty"`
// Took is how long each phase of the fetch ran, in seconds (issue 1921, the try speed-up's first measurement): what
// the checker reports as one "clone" time, split.
Took *FetchTook `json:"took,omitempty"`
}
// FetchTook is the time of each phase of a fetch, in seconds; its phases add up to Total. Mirror is bringing the
// repository's mirror to the commit, Fetch the fetch and checkout into the tree, Merge the merge preview, Dissociate
// the repack that makes the tree independent of the mirror, Beside the placing of the siblings (their own mirror,
// fetch and repack), Other the narrowing and the listing of the go.mod directories, and Modules the download and
// verify of the Go modules.
type FetchTook struct {
Mirror float64 `json:"mirror"`
Fetch float64 `json:"fetch"`
Merge float64 `json:"preview"` // named for the merge preview, apart from the result's merge commit
Dissociate float64 `json:"dissociate"`
Beside float64 `json:"siblings"`
Other float64 `json:"other"`
Modules float64 `json:"modules"`
Total float64 `json:"total"`
}
// phaseClock stamps the phases of a fetch: each lap is the time since the one before, so the laps add up to the total.
type phaseClock struct {
now func() time.Time
start, since time.Time
}
func newPhaseClock(now func() time.Time) *phaseClock {
t := now()
return &phaseClock{now: now, start: t, since: t}
}
// lap adds the time since the last lap to *into.
func (c *phaseClock) lap(into *float64) {
t := c.now()
*into += t.Sub(c.since).Seconds()
c.since = t
}
func (c *phaseClock) total() float64 { return c.since.Sub(c.start).Seconds() }
// String is the one log line a fetch says of its phases.
func (t FetchTook) String() string {
return fmt.Sprintf("mirror %.1fs, fetch %.1fs, merge %.1fs, dissociate %.1fs, beside %.1fs, other %.1fs, modules %.1fs, total %.1fs",
t.Mirror, t.Fetch, t.Merge, t.Dissociate, t.Beside, t.Other, t.Modules, t.Total)
} }
// fetchTool is the fetch step tool. // fetchTool is the fetch step tool.
@@ -109,43 +154,77 @@ func (m *Module) fetch(ctx context.Context, args map[string]any) (res any, err e
} }
func (m *Module) fetchInto(ctx context.Context, req buildstep.FetchRequest, id string, siblings []siblingArg) (any, error) { func (m *Module) fetchInto(ctx context.Context, req buildstep.FetchRequest, id string, siblings []siblingArg) (any, error) {
res, err := m.fetchPhases(ctx, req, id, siblings)
if err != nil {
return nil, err
}
if m.Say != nil && res.Took != nil {
m.Say("the fetch of %s at %s took %s", req.Repository, short(req.Commit), res.Took)
}
return res, nil
}
func short(commit string) string {
if len(commit) > 12 {
return commit[:12]
}
return commit
}
func (m *Module) fetchPhases(ctx context.Context, req buildstep.FetchRequest, id string, siblings []siblingArg) (FetchResult, error) {
repository, commit, ref, base := req.Repository, req.Commit, req.Ref, req.Base repository, commit, ref, base := req.Repository, req.Commit, req.Ref, req.Base
var err error var err error
took := &FetchTook{}
clock := newPhaseClock(m.now)
defer func() { took.Total = clock.total() }()
root := filepath.Join(m.Work, id) root := filepath.Join(m.Work, id)
tree := filepath.Join(root, "tree") tree := filepath.Join(root, "tree")
if err := os.MkdirAll(tree, 0o700); err != nil { if err := os.MkdirAll(tree, 0o700); err != nil {
return nil, err return FetchResult{}, err
} }
remote := m.remote(repository) remote := m.remote(repository)
if err := buildstep.FetchCommit(ctx, m.Run, tree, remote, commit, ref, m.fetchOptions(ctx, repository, commit, ref)...); err != nil { opts := m.fetchOptions(ctx, repository, commit, ref)
return nil, err mergeOpts := m.FetchOptions
}
res := FetchResult{Workspace: id, Commit: commit}
if base != "" { if base != "" {
if err := m.merge(ctx, &res, tree, remote, base, commit); err != nil { // The mirror is brought to the trunk commit too, so the merge preview deepens the tree from it and asks the forge
return nil, err // for no history the mirror holds (issue 1921, try speed S6).
mergeOpts = m.fetchOptions(ctx, repository, base, "")
}
clock.lap(&took.Mirror)
if err := buildstep.FetchCommit(ctx, m.Run, tree, remote, commit, ref, opts...); err != nil {
return FetchResult{}, err
}
clock.lap(&took.Fetch)
res := FetchResult{Workspace: id, Commit: commit, Took: took}
if base != "" {
if err := m.merge(ctx, &res, tree, remote, base, commit, mergeOpts); err != nil {
return FetchResult{}, err
} }
} }
clock.lap(&took.Merge)
if err := m.dissociate(ctx, tree); err != nil { if err := m.dissociate(ctx, tree); err != nil {
return nil, err return FetchResult{}, err
} }
clock.lap(&took.Dissociate)
if res.Beside, err = m.placeSiblings(ctx, root, siblings); err != nil { if res.Beside, err = m.placeSiblings(ctx, root, siblings); err != nil {
return nil, err return FetchResult{}, err
} }
clock.lap(&took.Beside)
if (base == "" || res.Merge != "") && len(req.Paths) > 0 { if (base == "" || res.Merge != "") && len(req.Paths) > 0 {
if err := buildstep.Narrow(tree, req); err != nil { if err := buildstep.Narrow(tree, req); err != nil {
return nil, err return FetchResult{}, err
} }
// Said beside the tree, so the steps that list the tree's go.mod files leave out those narrowed away. // Said beside the tree, so the steps that list the tree's go.mod files leave out those narrowed away.
if err := os.WriteFile(filepath.Join(root, narrowedMark), nil, 0o600); err != nil { if err := os.WriteFile(filepath.Join(root, narrowedMark), nil, 0o600); err != nil {
return nil, err return FetchResult{}, err
} }
} }
dirs, err := m.goModDirs(ctx, tree) dirs, err := m.goModDirs(ctx, tree)
if err != nil { if err != nil {
return nil, err return FetchResult{}, err
} }
res.Modules = dirs res.Modules = dirs
clock.lap(&took.Other)
if base != "" && res.Merge == "" { if base != "" && res.Merge == "" {
return res, nil // a conflict: the tree is at the trunk, nothing of the head is built or tested return res, nil // a conflict: the tree is at the trunk, nothing of the head is built or tested
} }
@@ -168,9 +247,10 @@ func (m *Module) fetchInto(ctx context.Context, req buildstep.FetchRequest, id s
}() }()
} }
wg.Wait() wg.Wait()
clock.lap(&took.Modules)
for _, err := range errs { for _, err := range errs {
if err != nil { if err != nil {
return nil, err return FetchResult{}, err
} }
} }
return res, nil return res, nil
@@ -288,10 +368,15 @@ func inTestdata(path string) bool {
// git, through the account's rewrite of the forge's address to ssh), and nothing that asks a question. For a directory // git, through the account's rewrite of the forge's address to ssh), and nothing that asks a question. For a directory
// that vendors, see goEnvFor. // that vendors, see goEnvFor.
func (m *Module) goEnv() []string { func (m *Module) goEnv() []string {
modcache := filepath.Join(m.Cache, "gomod")
if m.GoModCache != "" {
modcache = m.GoModCache
}
return []string{ return []string{
"CGO_ENABLED=0", // as a production build compiles (build.go), so a check tests what ships (issue 1921, S11)
"GOTOOLCHAIN=local", "GOTOOLCHAIN=local",
"GOFLAGS=-mod=mod", "GOFLAGS=-mod=mod",
"GOMODCACHE=" + filepath.Join(m.Cache, "gomod"), "GOMODCACHE=" + modcache,
"GOCACHE=" + filepath.Join(m.Cache, "gobuild"), "GOCACHE=" + filepath.Join(m.Cache, "gobuild"),
"GOPRIVATE=git.novox.be", "GOPRIVATE=git.novox.be",
"GIT_TERMINAL_PROMPT=0", "GIT_TERMINAL_PROMPT=0",
@@ -3,6 +3,7 @@ package main
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -302,3 +303,61 @@ func TestWorkAndCacheAreTheDirectoriesTheManifestPlacesOutsideTheHome(t *testing
} }
} }
} }
// 1921/S1: the fetch says how long each phase ran (mirror, fetch, merge, dissociate, beside, other, modules), and the
// phases add up to its total, so a try's "clone" time is split into its parts.
func TestTheFetchSaysTheTimeOfEachPhaseAndTheyAddUpToTheTotal(t *testing.T) {
var mu sync.Mutex
now := time.Date(2026, 10, 25, 9, 0, 0, 0, time.UTC)
advance := func(s int) {
mu.Lock()
now = now.Add(time.Duration(s) * time.Second)
mu.Unlock()
}
f := &fakeStep{gomods: "go.mod\n", files: map[string]string{"go.mod": "module x\n", "go.sum": ""}, rule: func(line string) (forgekey.Ran, bool) {
switch {
case strings.Contains(line, "/mirror/") && strings.Contains(line, " fetch "):
advance(1)
case strings.Contains(line, " fetch --no-tags --depth 1 "):
advance(2)
case strings.Contains(line, " merge --no-ff "):
advance(3)
case strings.Contains(line, " repack "):
advance(4)
case strings.HasSuffix(line, " mod download"):
advance(5)
}
return forgekey.Ran{}, false
}}
m, states := fetchModule(t, f)
said := []string{}
m.Say = func(format string, a ...any) { said = append(said, fmt.Sprintf(format, a...)) }
m.Now = func() time.Time { mu.Lock(); defer mu.Unlock(); return now }
args := map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA, "base": strings.Repeat("b", 40)}
if _, err := fetchTool(t, m)(args); err != nil {
t.Fatal(err)
}
st := states.waitEnd(t)
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
var res FetchResult
if err := json.Unmarshal(st.Result, &res); err != nil {
t.Fatal(err)
}
if res.Took == nil {
t.Fatalf("the result says no phase times: %s", st.Result)
}
got := *res.Took
// the mirror is brought to the head and to the base (1 s each), and the preview deepens from it (1 s) before merging
want := FetchTook{Mirror: 2, Fetch: 2, Merge: 4, Dissociate: 4, Modules: 5, Total: 17}
if got != want {
t.Errorf("phases = %+v, want %+v", got, want)
}
if sum := got.Mirror + got.Fetch + got.Merge + got.Dissociate + got.Beside + got.Other + got.Modules; sum != got.Total {
t.Errorf("the phases add up to %v, not the total %v", sum, got.Total)
}
if !strings.Contains(strings.Join(said, "\n"), "dissociate 4.0s") {
t.Errorf("no log line names the phases: %q", said)
}
}
@@ -10,8 +10,10 @@ import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"sort"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"git.novox.be/novox/mesh-sdk/go/buildstep" "git.novox.be/novox/mesh-sdk/go/buildstep"
@@ -62,34 +64,99 @@ func (m *Module) integrationHere(ctx context.Context, workspace, tree string, pa
if err != nil { if err != nil {
return nil, err return nil, err
} }
postgres, stopStore, err := m.startStore(ctx, run)
if err != nil {
return nil, fmt.Errorf("the integration level's store: %w", err)
}
defer stopStore()
nats, stopBus, err := m.startBus(ctx, filepath.Join(run, "nats"))
if err != nil {
return nil, fmt.Errorf("the integration level's bus: %w", err)
}
defer stopBus()
env := append([]string{"MESH_TEST_POSTGRES=" + postgres, "MESH_TEST_NATS=" + nats}, m.besideEnv(workspace)...)
// Compiling counts as setup: the tests are built once, here, with the addresses a TestMain may read, and the run
// below finds them built. Whether they compile is for the run to say; its failure is not read here.
// Only the directories the selected packages lie in compile (ADR 0536 §3); no selection is every directory. // Only the directories the selected packages lie in compile (ADR 0536 §3); no selection is every directory.
byDir, err := packagesByDir(tree, dirs, packages) byDir, err := packagesByDir(tree, dirs, packages)
if err != nil { if err != nil {
return nil, err return nil, err
} }
// The tests the tags add, not the unit tests again (issue 1921, try speed S7): only the packages holding one compile
// and run, with a -run pattern naming them. A try that names its own pattern runs it as asked.
var runByDir map[string]string
if pattern == "" {
tagged, err := taggedTests(tree, dirs, packages, integrationTags)
if err != nil {
return nil, err
}
if len(tagged) == 0 {
res := &TestResult{Level: LevelIntegration, Packages: []PackageResult{}}
res.judge(m.integrationBounds())
return res, nil // no test the tags add: nothing to start or run
}
byDir, runByDir, packages = map[string][]string{}, map[string]string{}, nil
for d, tr := range tagged {
byDir[d], runByDir[d] = tr.packages, tr.pattern
packages = append(packages, tr.packages...)
}
sort.Strings(packages)
}
// The store, the bus and the compile start together (issue 1921, try speed S5); the setup is the longest of them.
// The compile runs no test binary (-exec true), so it needs neither server: whether the tests compile is for the run
// to say, and its failure is not read here; the run below finds them built.
var (
wg sync.WaitGroup
postgres, nats string
stopStore, stopBus func()
storeErr, busErr error
crash any
crashMu sync.Mutex
)
guard := func() {
if p := recover(); p != nil {
crashMu.Lock()
crash = p
crashMu.Unlock()
}
}
wg.Add(2)
go func() { defer wg.Done(); defer guard(); postgres, stopStore, storeErr = m.startStore(ctx, run) }()
go func() {
defer wg.Done()
defer guard()
nats, stopBus, busErr = m.startBus(ctx, filepath.Join(run, "nats"))
}()
slots := m.slots(workspace) // shared with the ask's unit level (issue 1921, try speed S10)
var slotErr error
for _, d := range dirs { for _, d := range dirs {
if len(byDir[d]) == 0 { if len(byDir[d]) == 0 {
continue continue
} }
m.Run(ctx, append(m.goEnvFor(filepath.Join(tree, d)), env...), "go", append([]string{"-C", filepath.Join(tree, d), "test", "-tags", strings.Join(integrationTags, ","), "-run", "^$", "-count=1"}, byDir[d]...)...) release, err := slots.acquire(ctx)
if err != nil {
slotErr = err
break
}
wg.Add(1)
go func() {
defer wg.Done()
defer release()
defer guard()
m.Run(ctx, m.goEnvFor(filepath.Join(tree, d)), "go", append([]string{"-C", filepath.Join(tree, d), "test", "-tags", strings.Join(integrationTags, ","), "-exec", "true", "-count=1"}, byDir[d]...)...)
}()
} }
wg.Wait()
if stopStore != nil {
defer stopStore()
}
if stopBus != nil {
defer stopBus()
}
if crash != nil {
panic(crash)
}
if slotErr != nil {
return nil, slotErr
}
if storeErr != nil {
return nil, fmt.Errorf("the integration level's store: %w", storeErr)
}
if busErr != nil {
return nil, fmt.Errorf("the integration level's bus: %w", busErr)
}
env := append([]string{"MESH_TEST_POSTGRES=" + postgres, "MESH_TEST_NATS=" + nats}, m.besideEnv(workspace)...)
setup := m.clock().Sub(began) setup := m.clock().Sub(began)
return m.runLevel(ctx, tree, levelOpts{ return m.runLevel(ctx, tree, levelOpts{
workspace: workspace, run: pattern, workspace: workspace, run: pattern, runByDir: runByDir,
level: LevelIntegration, tags: integrationTags, packages: packages, bounds: m.integrationBounds(), level: LevelIntegration, tags: integrationTags, packages: packages, bounds: m.integrationBounds(),
env: env, env: env,
setup: setup, wall: true, setup: setup, wall: true,
@@ -6,6 +6,7 @@ import (
"net/url" "net/url"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -35,6 +36,8 @@ type integFake struct {
// command runs, with its line. // command runs, with its line.
gomods string gomods string
onTest func(line string) onTest func(line string)
// onCompile is called, unlocked, as each compile of the setup runs (issue 1921, try speed S5).
onCompile func(line string)
} }
func newIntegFake(t *testing.T, captured string) *integFake { func newIntegFake(t *testing.T, captured string) *integFake {
@@ -71,7 +74,10 @@ func (f *integFake) run(ctx context.Context, env []string, name string, args ...
return forgekey.Ran{Stdout: f.gomods} return forgekey.Ran{Stdout: f.gomods}
} }
return forgekey.Ran{Stdout: "go.mod\n"} return forgekey.Ran{Stdout: "go.mod\n"}
case name == "go" && strings.Contains(line, "-run ^$"): case name == "go" && strings.Contains(line, "-exec true"):
if f.onCompile != nil {
f.onCompile(line)
}
f.tick("compile") f.tick("compile")
case name == "go": case name == "go":
if f.onTest != nil { if f.onTest != nil {
@@ -123,9 +129,32 @@ func integModule(t *testing.T, f *integFake) (*Module, *keptStates) {
if err := os.MkdirAll(filepath.Join(m.Work, "ws1", "tree"), 0o755); err != nil { if err := os.MkdirAll(filepath.Join(m.Work, "ws1", "tree"), 0o755); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// A test the integration tag adds, so the level has a package to run (issue 1921, try speed S7).
writeTaggedTest(t, filepath.Join(m.Work, "ws1", "tree"), "fixture/slow")
return m, states return m, states
} }
// taggedTestFile is the file writeTaggedTest writes: one test the integration tag adds.
const taggedTestFile = "level_integration_test.go"
// writeTaggedTest gives the go.mod directory dir the module path mp and one test the integration tag adds, in its root
// package and in its package x.
func writeTaggedTest(t *testing.T, dir, mp string) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module "+mp+"\n\ngo 1.25.0\n"), 0o644); err != nil {
t.Fatal(err)
}
body := "//go:build integration\n\npackage x\n\nimport \"testing\"\n\nfunc TestTakesAWhile(t *testing.T) {}\n"
for _, d := range []string{dir, filepath.Join(dir, "x")} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(d, taggedTestFile), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
}
func runLevelTool(t *testing.T, m *Module, states *keptStates, level string) (TestResult, delivery.Step) { func runLevelTool(t *testing.T, m *Module, states *keptStates, level string) (TestResult, delivery.Step) {
t.Helper() t.Helper()
for _, tool := range m.Tools(m.StepTools()) { for _, tool := range m.Tools(m.StepTools()) {
@@ -314,11 +343,11 @@ func TestAStoreThatDoesNotStartEndsTheStepFailedAndLeavesNothing(t *testing.T) {
if !strings.Contains(f.lines(), " stop") { if !strings.Contains(f.lines(), " stop") {
t.Errorf("the half-started store was not stopped:\n%s", f.lines()) t.Errorf("the half-started store was not stopped:\n%s", f.lines())
} }
if f.busDir != "" { if f.busDir != "" && !f.busDown {
t.Errorf("the bus was started after the store failed") t.Errorf("the bus, started beside the store, was not stopped after the store failed")
} }
if entries, _ := os.ReadDir(filepath.Join(m.Work, "ws1")); len(entries) != 1 || entries[0].Name() != "tree" { if left := workspaceLeft(filepath.Join(m.Work, "ws1")); len(left) != 1 || left[0] != "tree" {
t.Errorf("the workspace holds %v, want only its tree", entries) t.Errorf("the workspace holds %v, want only its tree", left)
} }
} }
@@ -377,3 +406,138 @@ func TestTheSocketPathStaysUnderTheLimitForALongRunDirectory(t *testing.T) {
t.Errorf("the short socket directory %s outlived the store (%v)", sock, err) t.Errorf("the short socket directory %s outlived the store (%v)", sock, err)
} }
} }
// 1921/S5: the setup compiles the go.mod directories side by side, up to the bound, beside the store's and the bus's
// start, and compiles without running a test binary, so it needs neither; the setup stays one number.
func TestTheSetupCompilesTheDirectoriesAtOnceBesideTheStoreAndTheBus(t *testing.T) {
f := newIntegFake(t, "slow")
f.gomods = "a/go.mod\nb/go.mod\nc/go.mod\nd/go.mod\ne/go.mod\n"
var mu sync.Mutex
running, most := 0, 0
storeStarting, storeUp, overlap := false, false, false
f.onCompile = func(line string) {
mu.Lock()
running++
most = max(most, running)
mu.Unlock()
time.Sleep(40 * time.Millisecond)
mu.Lock()
running--
mu.Unlock()
}
m, states := integModule(t, f)
m.Parallel = 3
tree := filepath.Join(m.Work, "ws1", "tree")
for _, d := range []string{"a", "b", "c", "d", "e"} {
os.MkdirAll(filepath.Join(tree, d), 0o755)
writeTaggedTest(t, filepath.Join(tree, d), "x/"+d)
}
inner := m.Run
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
line := strings.Join(args, " ")
if name == "pg_ctl" && strings.HasSuffix(line, " start") {
mu.Lock()
storeStarting = true
mu.Unlock()
time.Sleep(100 * time.Millisecond) // the store takes a while to start
defer func() { mu.Lock(); storeUp = true; mu.Unlock() }()
}
if name == "go" && strings.Contains(line, "-exec true") {
time.Sleep(10 * time.Millisecond)
mu.Lock()
overlap = overlap || (storeStarting && !storeUp)
mu.Unlock()
for _, e := range env {
if strings.HasPrefix(e, "MESH_TEST_") {
t.Errorf("the compile was given %s: it runs no test binary", e)
}
}
}
return inner(ctx, env, name, args...)
}
res, st := runLevelTool(t, m, states, "integration")
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
if most != 3 {
t.Errorf("%d compiles ran at once, want the bound of 3", most)
}
mu.Lock()
defer mu.Unlock()
if !overlap {
t.Errorf("no compile ran while the store was starting:\n%s", f.lines())
}
if res.Setup < 0 {
t.Errorf("setup = %v", res.Setup)
}
}
// 1921/S7: the integration level runs the tests its tags add, not the unit tests again: only the packages holding a
// tagged test compile and run, with a -run pattern naming those tests; a tree with no tagged test starts no store.
func TestTheIntegrationLevelRunsOnlyTheTestsItsTagsAdd(t *testing.T) {
f := newIntegFake(t, "slow")
m, states := integModule(t, f)
tree := filepath.Join(m.Work, "ws1", "tree")
os.RemoveAll(filepath.Join(tree, "x"))
os.Remove(filepath.Join(tree, taggedTestFile))
for name, body := range map[string]string{
"go.mod": "module fixture/slow\n",
"unit/u_test.go": "package unit\n\nimport \"testing\"\n\nfunc TestUnit(t *testing.T) {}\n",
"store/s_test.go": "package store\n\nimport \"testing\"\n\nfunc TestPlain(t *testing.T) {}\n",
"store/s_integration_test.go": "//go:build integration\n\npackage store\n\nimport \"testing\"\n\nfunc TestMain(m *testing.M) {}\nfunc TestReal(t *testing.T) {}\nfunc Testable() {}\nfunc helper() {}\n",
"store/fake_test.go": "//go:build !integration\n\npackage store\n\nimport \"testing\"\n\nfunc TestFake(t *testing.T) {}\n",
"iso/iso_test.go": "//go:build integration && isolation && linux\n\npackage iso\n\nimport \"testing\"\n\nfunc TestShutOff(t *testing.T) {}\n",
"plat/p_linux_test.go": "//go:build linux\n\npackage plat\n\nimport \"testing\"\n\nfunc TestLinux(t *testing.T) {}\n",
} {
os.MkdirAll(filepath.Dir(filepath.Join(tree, name)), 0o755)
if err := os.WriteFile(filepath.Join(tree, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
if _, st := runLevelTool(t, m, states, "integration"); st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
var compiled, ran []string
for _, c := range f.calls {
switch {
case strings.HasPrefix(c.line, "go ") && strings.Contains(c.line, "-exec true"):
compiled = append(compiled, c.line)
case strings.HasPrefix(c.line, "go ") && strings.Contains(c.line, " -json"):
ran = append(ran, c.line)
}
}
if len(ran) != 1 || !strings.Contains(ran[0], "-run ^(TestReal|TestShutOff)$") ||
!strings.HasSuffix(ran[0], " fixture/slow/iso fixture/slow/store") {
t.Errorf("the run = %q, want the tagged tests of iso and store alone", ran)
}
if len(compiled) != 1 || !strings.HasSuffix(compiled[0], " fixture/slow/iso fixture/slow/store") {
t.Errorf("the compile = %q, want iso and store alone", compiled)
}
g := newIntegFake(t, "slow")
m, states = integModule(t, g)
tree = filepath.Join(m.Work, "ws1", "tree")
os.RemoveAll(filepath.Join(tree, "x"))
os.Remove(filepath.Join(tree, taggedTestFile))
os.WriteFile(filepath.Join(tree, "u_test.go"), []byte("package x\n\nimport \"testing\"\n\nfunc TestUnit(t *testing.T) {}\n"), 0o644)
res, st := runLevelTool(t, m, states, "integration")
if st.State != delivery.StepPassed || !res.Passed || !res.NoTests {
t.Fatalf("a tree with no tagged test: %s %+v", st.State, res)
}
if strings.Contains(g.lines(), "pg_ctl") || strings.Contains(g.lines(), "go -C") {
t.Errorf("a tree with no tagged test started the store or ran go:\n%s", g.lines())
}
}
// workspaceLeft is what a workspace holds besides the directories its ask keeps there (askDirs: its cache layer and its
// bound on processes, gone with the workspace; issue 1921, S2 and S10).
func workspaceLeft(dir string) []string {
var left []string
entries, _ := os.ReadDir(dir)
for _, e := range entries {
if !slices.Contains(askDirs, e.Name()) {
left = append(left, e.Name())
}
}
return left
}
@@ -29,21 +29,30 @@ func (m *Module) caches() []string {
} }
// isolation is how a run step of the workspace is shut off: the workspace (its tree, its siblings, its run directories, // isolation is how a run step of the workspace is shut off: the workspace (its tree, its siblings, its run directories,
// its packed artifacts) read-write, the caches beneath a throwaway layer unless write, and an environment of a PATH and // its packed artifacts) read-write, the caches beneath the ask's own layer unless write, and an environment of a PATH and
// a locale alone (the step's HOME is its own, inside its /tmp). // a locale alone (the step's HOME is its own, inside its /tmp).
func (m *Module) isolation(workspace string, write bool) buildstep.Isolation { func (m *Module) isolation(workspace string, write bool) buildstep.Isolation {
path := os.Getenv("PATH") path := os.Getenv("PATH")
if path == "" { if path == "" {
path = defaultPath path = defaultPath
} }
return buildstep.Isolation{ iso := buildstep.Isolation{
Workspace: filepath.Join(m.Work, workspace), Workspace: filepath.Join(m.Work, workspace),
Caches: m.caches(), Caches: m.caches(),
WriteCaches: write, WriteCaches: write,
Env: []string{"PATH=" + path, "LC_ALL=C"}, Env: []string{"PATH=" + path, "LC_ALL=C"},
} }
if !write {
// What one command of the ask compiles is kept for its later commands (the vet, the unit tests, the integration
// level's compile), in the workspace and gone with it (issue 1921, try speed S2).
iso.CacheLayer = filepath.Join(iso.Workspace, cacheLayerDir)
}
return iso
} }
// cacheLayerDir is the directory in a workspace's root where an ask keeps what its commands wrote into the caches.
const cacheLayerDir = "cachelayer"
// stepRunner is the runner of a run step of the workspace: buildstep.Isolated over m.Run, so every command it runs is // stepRunner is the runner of a run step of the workspace: buildstep.Isolated over m.Run, so every command it runs is
// shut off; write is true for a trunk build alone (the shared caches are written by trunk builds only). Inside is the // shut off; write is true for a trunk build alone (the shared caches are written by trunk builds only). Inside is the
// integration level's own process, already running shut off (level.go): it runs its commands as they are. // integration level's own process, already running shut off (level.go): it runs its commands as they are.
@@ -20,6 +20,7 @@ import (
type shutOff struct { type shutOff struct {
flags []string flags []string
binds, roBinds, overlays []string binds, roBinds, overlays []string
uppers map[string]string // the writable layer each --overlay gives a cache: cache -> upper
env []string env []string
command string command string
chdir string chdir string
@@ -27,7 +28,7 @@ type shutOff struct {
// bwrapArity is how many words each bubblewrap option the SDK writes takes. // bwrapArity is how many words each bubblewrap option the SDK writes takes.
var bwrapArity = map[string]int{"--unshare-all": 0, "--die-with-parent": 0, "--new-session": 0, "--clearenv": 0, var bwrapArity = map[string]int{"--unshare-all": 0, "--die-with-parent": 0, "--new-session": 0, "--clearenv": 0,
"--proc": 1, "--dev": 1, "--tmpfs": 1, "--dir": 1, "--chdir": 1, "--overlay-src": 1, "--tmp-overlay": 1, "--proc": 1, "--dev": 1, "--tmpfs": 1, "--dir": 1, "--chdir": 1, "--overlay-src": 1, "--tmp-overlay": 1, "--overlay": 3,
"--ro-bind": 2, "--ro-bind-try": 2, "--bind": 2, "--symlink": 2, "--setenv": 2} "--ro-bind": 2, "--ro-bind-try": 2, "--bind": 2, "--symlink": 2, "--setenv": 2}
// throughBubblewrap is the fake runner of a step's tests: a bubblewrap command is read back (recorded in seen) and its // throughBubblewrap is the fake runner of a step's tests: a bubblewrap command is read back (recorded in seen) and its
@@ -56,6 +57,11 @@ func throughBubblewrap(t *testing.T, run forgekey.Runner, seen *[]shutOff) forge
s.roBinds = append(s.roBinds, args[i+1]) s.roBinds = append(s.roBinds, args[i+1])
case "--overlay-src": case "--overlay-src":
s.overlays = append(s.overlays, args[i+1]) s.overlays = append(s.overlays, args[i+1])
case "--overlay":
if s.uppers == nil {
s.uppers = map[string]string{}
}
s.uppers[args[i+3]] = args[i+1]
case "--setenv": case "--setenv":
s.env = append(s.env, args[i+1]+"="+args[i+2]) s.env = append(s.env, args[i+1]+"="+args[i+2])
case "--chdir": case "--chdir":
@@ -105,6 +111,9 @@ func checkShutOff(t *testing.T, m *Module, s shutOff, write bool) {
t.Errorf("a trunk build's %s does not write the shared cache %s (bound %v, layered %v)", s.command, c, bound, layered) t.Errorf("a trunk build's %s does not write the shared cache %s (bound %v, layered %v)", s.command, c, bound, layered)
case !write && (bound || !layered): case !write && (bound || !layered):
t.Errorf("%s may write the shared cache %s (bound %v, layered %v): only a trunk build does", s.command, c, bound, layered) t.Errorf("%s may write the shared cache %s (bound %v, layered %v): only a trunk build does", s.command, c, bound, layered)
case !write && !strings.HasPrefix(s.uppers[c], filepath.Join(m.Work, "ws1", "cachelayer")+"/"):
// 1921/S2: what one command of the ask compiles is kept for its later commands, in the workspace
t.Errorf("%s writes the cache %s into %q, not the ask's cache layer in its workspace", s.command, c, s.uppers[c])
} }
} }
if want := filepath.Join(m.Work, "ws1"); !slices.Contains(s.binds, want) { if want := filepath.Join(m.Work, "ws1"); !slices.Contains(s.binds, want) {
@@ -131,7 +140,8 @@ func TestTheUnitLevelRunsEveryGoTestShutOff(t *testing.T) {
t.Fatalf("%d go commands ran, %d of them shut off: %v", ran, len(seen), *calls) t.Fatalf("%d go commands ran, %d of them shut off: %v", ran, len(seen), *calls)
} }
for _, s := range seen { for _, s := range seen {
checkShutOff(t, m, s, false) // the lint's go vet, a compile alone, writes the shared cache (issue 1921, S11): cgo_test.go
checkShutOff(t, m, s, strings.Contains(s.command, " vet "))
} }
} }
@@ -41,6 +41,8 @@ type levelSpec struct {
Run string `json:"run,omitempty"` Run string `json:"run,omitempty"`
Setup time.Duration `json:"setup"` Setup time.Duration `json:"setup"`
Bound time.Duration `json:"bound"` Bound time.Duration `json:"bound"`
// Parallel is the ask's bound on processes, which the process shares with the ask's other level (issue 1921, S10).
Parallel int `json:"parallel,omitempty"`
} }
// levelAnswer is the one line the process prints: the level's result or the digest, or why it could not run. // levelAnswer is the one line the process prints: the level's result or the digest, or why it could not run.
@@ -83,7 +85,7 @@ func (m *Module) integrationShutOff(ctx context.Context, workspace string, packa
// workspace read-write, the program alone read-only, the shared caches beneath a throwaway layer, the Node build // workspace read-write, the program alone read-only, the shared caches beneath a throwaway layer, the Node build
// module's toolchains read-only for the integration level. // module's toolchains read-only for the integration level.
func (m *Module) shutOff(ctx context.Context, exe string, spec levelSpec) (levelAnswer, error) { func (m *Module) shutOff(ctx context.Context, exe string, spec levelSpec) (levelAnswer, error) {
spec.Work, spec.Cache = m.Work, m.Cache spec.Work, spec.Cache, spec.Parallel = m.Work, m.Cache, m.parallel()
raw, err := json.Marshal(spec) raw, err := json.Marshal(spec)
if err != nil { if err != nil {
return levelAnswer{}, err return levelAnswer{}, err
@@ -165,7 +167,7 @@ func runLevelProcess(args []string) int {
return say(levelAnswer{Error: fmt.Sprintf("%q is not a workspace id", spec.Workspace)}) return say(levelAnswer{Error: fmt.Sprintf("%q is not a workspace id", spec.Workspace)})
} }
m := &Module{Inside: true, Recipe: Current(), Work: spec.Work, Cache: spec.Cache, Home: os.Getenv("HOME"), UnitBound: spec.Bound, m := &Module{Inside: true, Recipe: Current(), Work: spec.Work, Cache: spec.Cache, Home: os.Getenv("HOME"), UnitBound: spec.Bound,
IntegrationBounds: Bounds{Setup: spec.Setup, Run: spec.Bound}, Say: func(string, ...any) {}} IntegrationBounds: Bounds{Setup: spec.Setup, Run: spec.Bound}, Parallel: spec.Parallel, Say: func(string, ...any) {}}
m.Run = StepRunner(m.Home) m.Run = StepRunner(m.Home)
return say(m.answer(context.Background(), spec)) return say(m.answer(context.Background(), spec))
} }
@@ -57,8 +57,14 @@ type Failure struct {
Test string `json:"test,omitempty"` Test string `json:"test,omitempty"`
Output string `json:"output,omitempty"` Output string `json:"output,omitempty"`
Cause string `json:"cause,omitempty"` Cause string `json:"cause,omitempty"`
// Step is StepLint for a finding of the lint step (gofmt, go vet), which runs beside the unit level and is not one of
// the levels (ADR 0542 §2): the checker reports it as the lint step's failure. Empty for a test's failure.
Step string `json:"step,omitempty"`
} }
// StepLint is the Step of a lint finding.
const StepLint = "lint"
// TestResult is what a test step passes with. // TestResult is what a test step passes with.
type TestResult struct { type TestResult struct {
Level string `json:"level"` Level string `json:"level"`
@@ -73,7 +79,7 @@ type TestResult struct {
Elapsed float64 `json:"elapsed"` Elapsed float64 `json:"elapsed"`
// Setup is the level's setup time in seconds, when it has a setup. // Setup is the level's setup time in seconds, when it has a setup.
Setup float64 `json:"setup,omitempty"` Setup float64 `json:"setup,omitempty"`
// Lint is the unit level's gofmt and go vet time in seconds, before its tests (ADR 0529 §1). // Lint is the unit level's gofmt and go vet time in seconds, run beside its tests (ADR 0529 §1; issue 1921).
Lint float64 `json:"lint,omitempty"` Lint float64 `json:"lint,omitempty"`
// Bound and SetupBound are the bounds the level was judged by, in seconds. // Bound and SetupBound are the bounds the level was judged by, in seconds.
Bound float64 `json:"bound"` Bound float64 `json:"bound"`
@@ -207,15 +213,17 @@ func (r *reader) event(ev event) {
r.times = append(r.times, TestTime{Package: ev.Package, Test: ev.Test, Elapsed: ev.Elapsed}) r.times = append(r.times, TestTime{Package: ev.Package, Test: ev.Test, Elapsed: ev.Elapsed})
if ev.Action == "fail" { if ev.Action == "fail" {
r.failedIn[ev.Package] = true r.failedIn[ev.Package] = true
r.res.Failed = append(r.res.Failed, Failure{Package: ev.Package, Test: ev.Test, Output: clip(r.testOut[key]), Cause: r.causes[key]}) r.res.Failed = append(r.res.Failed, Failure{Package: ev.Package, Test: ev.Test, Output: clip(r.testOut[key], r.causes[key]), Cause: r.causes[key]})
} }
} }
} }
} }
// clip joins the last lines of a failure's output that fit outputBytes, each cut to outputLineBytes, and says how many // clip joins the last lines of a failure's output that fit outputBytes, each cut to outputLineBytes, and says how many
// earlier lines it left out. // earlier lines it left out. **When it leaves lines out, its first line is still a real one** (issue 1901/AC3): the failure's
func clip(lines []string) string { // cause, or failing that the first non-empty line of the output, then the marker. An older checker takes only the first
// non-empty line of the output, and a marker there would be less than the older pair kept.
func clip(lines []string, cause string) string {
size, from := 0, len(lines) size, from := 0, len(lines)
for from > 0 { for from > 0 {
n := len(cutLine(lines[from-1])) + 1 n := len(cutLine(lines[from-1])) + 1
@@ -227,6 +235,13 @@ func clip(lines []string) string {
} }
kept := make([]string, 0, len(lines)-from+1) kept := make([]string, 0, len(lines)-from+1)
if from > 0 { if from > 0 {
head := strings.TrimSpace(cause)
for i := 0; head == "" && i < from; i++ {
head = strings.TrimSpace(lines[i])
}
if head != "" {
kept = append(kept, cutLine(head))
}
kept = append(kept, fmt.Sprintf("... (%d earlier lines)", from)) kept = append(kept, fmt.Sprintf("... (%d earlier lines)", from))
} }
for _, l := range lines[from:] { for _, l := range lines[from:] {
@@ -256,7 +271,7 @@ func (r *reader) finish() {
for _, name := range r.order { for _, name := range r.order {
p := r.pkgs[name] p := r.pkgs[name]
if !p.Passed && !r.failedIn[name] { if !p.Passed && !r.failedIn[name] {
r.res.Failed = append(r.res.Failed, Failure{Package: name, Output: clip(r.pkgOut[name])}) r.res.Failed = append(r.res.Failed, Failure{Package: name, Output: clip(r.pkgOut[name], "")})
} }
if !p.NoTests && p.Elapsed > r.res.Elapsed { if !p.NoTests && p.Elapsed > r.res.Elapsed {
r.res.Elapsed = p.Elapsed r.res.Elapsed = p.Elapsed
+141 -24
View File
@@ -12,6 +12,7 @@ import (
"regexp" "regexp"
"sort" "sort"
"strings" "strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/forgekey" "git.novox.be/novox/mesh-sdk/go/forgekey"
) )
@@ -59,17 +60,23 @@ func lintable(dir string, each func(path string) error) error {
}) })
} }
// unformatted are the Go files of a directory gofmt would change, relative to the tree, and the build tags they name. // unformatted are the Go files of a directory gofmt would change, relative to the tree, the build tags the directory's
func unformatted(tree, d string) (files, tags []string, err error) { // files name, and whether a file is left out by one of them (a negated tag, //go:build !integration). Only files whose
// directory (relative to the go.mod directory, "." for its own) is in only are checked for gofmt; nil checks every file.
func unformatted(tree, d string, only map[string]bool) (files, tags []string, negated bool, err error) {
seen := map[string]bool{} seen := map[string]bool{}
err = lintable(filepath.Join(tree, d), func(path string) error { base := filepath.Join(tree, d)
err = lintable(base, func(path string) error {
src, err := os.ReadFile(path) src, err := os.ReadFile(path)
if err != nil { if err != nil {
return err return err
} }
if out, ferr := format.Source(src); ferr != nil || !bytes.Equal(out, src) { pkgDir, _ := filepath.Rel(base, filepath.Dir(path))
rel, _ := filepath.Rel(tree, path) if only == nil || only[filepath.ToSlash(pkgDir)] {
files = append(files, filepath.ToSlash(rel)) if out, ferr := format.Source(src); ferr != nil || !bytes.Equal(out, src) {
rel, _ := filepath.Rel(tree, path)
files = append(files, filepath.ToSlash(rel))
}
} }
scan := bufio.NewScanner(bytes.NewReader(src)) scan := bufio.NewScanner(bytes.NewReader(src))
for scan.Scan() { for scan.Scan() {
@@ -78,8 +85,15 @@ func unformatted(tree, d string) (files, tags []string, err error) {
break break
} }
if expr, ok := strings.CutPrefix(line, "//go:build "); ok { if expr, ok := strings.CutPrefix(line, "//go:build "); ok {
for _, tag := range buildTag.FindAllString(expr, -1) { for _, at := range buildTag.FindAllStringIndex(expr, -1) {
if !notATag[tag] && !strings.HasPrefix(tag, "go1") && !seen[tag] { tag := expr[at[0]:at[1]]
if notATag[tag] || strings.HasPrefix(tag, "go1") {
continue
}
if before := strings.TrimRight(expr[:at[0]], " ("); strings.HasSuffix(before, "!") {
negated = true
}
if !seen[tag] {
seen[tag] = true seen[tag] = true
tags = append(tags, tag) tags = append(tags, tag)
} }
@@ -90,40 +104,143 @@ func unformatted(tree, d string) (files, tags []string, err error) {
}) })
sort.Strings(files) sort.Strings(files)
sort.Strings(tags) sort.Strings(tags)
return files, tags, err return files, tags, negated, err
} }
// lint is gofmt and go vet of one go.mod directory d of the tree, run with run in the directory's go environment; a // boundOf is a bound of n processes of this process alone.
// failure of either is a Failure of the level naming what failed. func boundOf(n int) func(context.Context) (func(), error) {
func (m *Module) lint(ctx context.Context, run forgekey.Runner, tree, d string, env []string) ([]Failure, error) { slots := make(chan struct{}, n)
files, tags, err := unformatted(tree, d) return func(context.Context) (func(), error) {
slots <- struct{}{}
return func() { <-slots }, nil
}
}
// lintScope is what of a go.mod directory d the lint covers, from the patterns the level runs there: every package
// (./..., and the directories nil) when a pattern is relative or none is given, else the import paths given and the
// directories of their files, relative to d.
func lintScope(tree, d string, patterns []string) (targets []string, dirs map[string]bool) {
if len(patterns) == 0 {
return []string{"./..."}, nil
}
raw, _ := os.ReadFile(filepath.Join(tree, d, "go.mod"))
mp := modulePath(string(raw))
dirs = map[string]bool{}
for _, p := range patterns {
if strings.HasPrefix(p, "./") || p == "." {
return []string{"./..."}, nil
}
rel, ok := strings.CutPrefix(p, mp)
if mp == "" || !ok || (rel != "" && !strings.HasPrefix(rel, "/")) {
return []string{"./..."}, nil // a package of no module here: the whole directory, as before
}
if rel = strings.TrimPrefix(rel, "/"); rel == "" {
rel = "."
}
dirs[rel] = true
targets = append(targets, p)
}
return targets, dirs
}
// lint is gofmt and go vet of one go.mod directory d of the tree, run with run in the directory's go environment, over
// the packages the level runs there (patterns, as packagesByDir gives them; issue 1921, try speed S3); a failure of either
// is a Failure of the level naming what failed. go vet runs once with every tag the directory names, which vets the
// untagged files too; only a directory where a tag leaves a file out vets also without tags, both passes at once.
func (m *Module) lint(ctx context.Context, run forgekey.Runner, tree, d string, env, patterns []string) ([]Failure, error) {
targets, only := lintScope(tree, d, patterns)
files, tags, negated, err := unformatted(tree, d, only)
if err != nil { if err != nil {
return nil, fmt.Errorf("reading the Go files of %s: %w", d, err) return nil, fmt.Errorf("reading the Go files of %s: %w", d, err)
} }
var failed []Failure var failed []Failure
if len(files) > 0 { if len(files) > 0 {
failed = append(failed, Failure{Package: d, Test: "gofmt", failed = append(failed, Failure{Package: d, Test: "gofmt", Step: StepLint,
Output: "not as gofmt writes them (run gofmt -w): " + strings.Join(files, ", ")}) Output: "not as gofmt writes them (run gofmt -w): " + strings.Join(files, ", ")})
} }
sets := [][]string{nil} sets := [][]string{tags}
if len(tags) > 0 { if len(tags) > 0 && negated {
sets = append(sets, tags) sets = [][]string{nil, tags}
} }
for _, set := range sets { type vetRun struct {
name string
r forgekey.Ran
}
runs := make([]vetRun, len(sets))
var wg sync.WaitGroup
for i, set := range sets {
args := []string{"-C", filepath.Join(tree, d), "vet"} args := []string{"-C", filepath.Join(tree, d), "vet"}
name := "go vet" name := "go vet"
if len(set) > 0 { if len(set) > 0 {
args = append(args, "-tags", strings.Join(set, ",")) args = append(args, "-tags", strings.Join(set, ","))
name += " -tags " + strings.Join(set, ",") name += " -tags " + strings.Join(set, ",")
} }
args = append(args, "./...") args = append(args, targets...)
r := run(ctx, env, "go", args...) runs[i].name = name
if r.Err != "" { wg.Add(1)
return nil, fmt.Errorf("%s in %s could not be run: %s", name, d, firstLine(r.Err+"\n"+r.Stderr)) go func() {
defer wg.Done()
runs[i].r = run(ctx, env, "go", args...)
}()
}
wg.Wait()
for _, v := range runs {
if v.r.Err != "" {
return nil, fmt.Errorf("%s in %s could not be run: %s", v.name, d, firstLine(v.r.Err+"\n"+v.r.Stderr))
} }
if r.Status != 0 { if v.r.Status != 0 {
failed = append(failed, Failure{Package: d, Test: name, Output: strings.TrimSpace(r.Stderr + "\n" + r.Stdout)}) failed = append(failed, Failure{Package: d, Test: v.name, Step: StepLint, Output: strings.TrimSpace(v.r.Stderr + "\n" + v.r.Stdout)})
} }
} }
return failed, nil return failed, nil
} }
// lintAll lints each directory, up to m.parallel() at once like the level's tests; the failures join in directory order
// and the error is the first directory's, so the answer does not depend on which ended first (issue 1930). byDir names
// the packages the level runs in each directory (packagesByDir); nil lints each directory whole. acquire takes a slot of
// the bound the lint shares with the ask's tests; nil gives it one of its own.
func (m *Module) lintAll(ctx context.Context, run forgekey.Runner, tree string, dirs []string, env func(d string) []string, byDir map[string][]string, acquire func(context.Context) (func(), error)) ([]Failure, error) {
type answer struct {
failed []Failure
err error
}
answers := make([]answer, len(dirs))
if acquire == nil {
acquire = boundOf(m.parallel())
}
var wg sync.WaitGroup
var crash any // a panic of a lint goes on in the caller, as a panic of a test run does
var crashMu sync.Mutex
for i, d := range dirs {
release, err := acquire(ctx)
if err != nil {
answers[i].err = err
break
}
wg.Add(1)
go func() {
defer wg.Done()
defer release()
defer func() {
if p := recover(); p != nil {
crashMu.Lock()
crash = p
crashMu.Unlock()
}
}()
answers[i].failed, answers[i].err = m.lint(ctx, run, tree, d, env(d), byDir[d])
}()
}
wg.Wait()
if crash != nil {
panic(crash)
}
var failed []Failure
for _, a := range answers {
if a.err != nil {
return nil, a.err
}
failed = append(failed, a.failed...)
}
return failed, nil
}
@@ -4,9 +4,13 @@ import (
"context" "context"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"sync"
"testing" "testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey" "git.novox.be/novox/mesh-sdk/go/forgekey"
) )
@@ -26,8 +30,9 @@ func writeTree(t *testing.T, files map[string]string) string {
} }
// 1887/D7-go, ADR 0529 §1: an unformatted file fails the unit level naming the file (testdata and vendor are left // 1887/D7-go, ADR 0529 §1: an unformatted file fails the unit level naming the file (testdata and vendor are left
// alone), and go vet runs with no tag and with every tag the files name together, integration and isolation included, so // alone), and go vet runs with every tag the files name together, integration and isolation included, so a vet error
// a vet error behind isolation fails the level naming the run. // behind isolation fails the level naming the run. One pass with every tag vets the untagged files too (issue 1921, try
// speed S3), as no file here is left out by a tag.
func TestTheUnitLevelRunsGofmtAndVetWithEveryTag(t *testing.T) { func TestTheUnitLevelRunsGofmtAndVetWithEveryTag(t *testing.T) {
tree := writeTree(t, map[string]string{ tree := writeTree(t, map[string]string{
"go.mod": "module x\n\ngo 1.26\n", "go.mod": "module x\n\ngo 1.26\n",
@@ -49,11 +54,11 @@ func TestTheUnitLevelRunsGofmtAndVetWithEveryTag(t *testing.T) {
} }
return forgekey.Ran{} return forgekey.Ran{}
} }
failed, err := (&Module{}).lint(context.Background(), run, tree, ".", nil) failed, err := (&Module{}).lint(context.Background(), run, tree, ".", nil, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if strings.Join(vetted, " | ") != "vet ./... | vet -tags integration,isolation ./..." { if strings.Join(vetted, " | ") != "vet -tags integration,isolation ./..." {
t.Errorf("vet ran %v", vetted) t.Errorf("vet ran %v", vetted)
} }
if len(failed) != 2 || failed[0].Test != "gofmt" || !strings.Contains(failed[0].Output, "b.go") || if len(failed) != 2 || failed[0].Test != "gofmt" || !strings.Contains(failed[0].Output, "b.go") ||
@@ -65,14 +70,14 @@ func TestTheUnitLevelRunsGofmtAndVetWithEveryTag(t *testing.T) {
} }
// A formatted directory that vets passes. // A formatted directory that vets passes.
clean := writeTree(t, map[string]string{"go.mod": "module y\n", "a.go": "package y\n"}) clean := writeTree(t, map[string]string{"go.mod": "module y\n", "a.go": "package y\n"})
if failed, err := (&Module{}).lint(context.Background(), func(context.Context, []string, string, ...string) forgekey.Ran { return forgekey.Ran{} }, clean, ".", nil); err != nil || len(failed) != 0 { if failed, err := (&Module{}).lint(context.Background(), func(context.Context, []string, string, ...string) forgekey.Ran { return forgekey.Ran{} }, clean, ".", nil, nil); err != nil || len(failed) != 0 {
t.Errorf("a clean directory: %+v, %v", failed, err) t.Errorf("a clean directory: %+v, %v", failed, err)
} }
} }
// 1887/D7-go: through the step, the unit level of a tree holding an unformatted file fails, naming the file. // 1887/D7-go: through the step, the unit level of a tree holding an unformatted file fails, naming the file.
func TestAnUnformattedFileFailsTheUnitLevel(t *testing.T) { func TestAnUnformattedFileFailsTheUnitLevel(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "empty") m, states, ws, _ := workspaceFor(t, "empty")
tree := filepath.Join(m.Work, ws, "tree") tree := filepath.Join(m.Work, ws, "tree")
for name, body := range map[string]string{"go.mod": "module x\n", "x.go": "package x\nfunc X() {}\n"} { for name, body := range map[string]string{"go.mod": "module x\n", "x.go": "package x\nfunc X() {}\n"} {
if err := os.WriteFile(filepath.Join(tree, name), []byte(body), 0o644); err != nil { if err := os.WriteFile(filepath.Join(tree, name), []byte(body), 0o644); err != nil {
@@ -83,10 +88,117 @@ func TestAnUnformattedFileFailsTheUnitLevel(t *testing.T) {
if res.Passed || len(res.Failed) != 1 || res.Failed[0].Test != "gofmt" || !strings.Contains(res.Failed[0].Output, "x.go") { if res.Passed || len(res.Failed) != 1 || res.Failed[0].Test != "gofmt" || !strings.Contains(res.Failed[0].Output, "x.go") {
t.Fatalf("result = %+v, want the level failed naming x.go", res) t.Fatalf("result = %+v, want the level failed naming x.go", res)
} }
// Review round 2 item 5: before the level's tests, which do not run once gofmt failed. }
for _, c := range *calls {
if strings.Contains(c.line, " test -json") { // 1921/S4: the lint runs beside the unit level's tests, not before them: both have started before either ends, a lint
t.Errorf("the tests ran after gofmt failed: %s", c.line) // failure still fails the level, and the result reports the lint's time and the tests' failures both.
func TestTheLintRunsBesideTheUnitTestsAndItsFailureStillFailsTheLevel(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "failing")
tree := filepath.Join(m.Work, ws, "tree")
for name, body := range map[string]string{"go.mod": "module fixture/failing\n", "x.go": "package x\nfunc X() {}\n"} {
if err := os.WriteFile(filepath.Join(tree, name), []byte(body), 0o644); err != nil {
t.Fatal(err)
} }
} }
m.Parallel = 4
inner := m.Run
var vetStarted, testStarted sync.Once
vet, test := make(chan struct{}), make(chan struct{})
timeout := time.After(2 * time.Second)
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
line := strings.Join(args, " ")
switch {
case strings.Contains(line, " vet "):
vetStarted.Do(func() { close(vet) })
select { // the vet ends only once a test has started
case <-test:
case <-timeout:
t.Error("no test started while the vet ran")
}
case strings.Contains(line, " test -json"):
testStarted.Do(func() { close(test) })
select {
case <-vet:
case <-timeout:
t.Error("no vet started while the tests ran")
}
}
return inner(ctx, env, name, args...)
}
res, st := runUnit(t, m, states, ws)
if st.State != delivery.StepPassed {
t.Fatalf("the step ended %s: %s", st.State, st.Reason)
}
if res.Passed || len(res.Failed) < 2 || res.Failed[0].Test != "gofmt" || res.Failed[1].Test != "TestThreeIsFour" {
t.Fatalf("failed = %+v, want gofmt's failure, then the test's", res.Failed)
}
// ADR 0542 §2: the lint's failure is the lint step's, not the unit level's.
if res.Failed[0].Step != StepLint || res.Failed[1].Step != "" {
t.Errorf("steps = %q, %q; want the gofmt failure the lint step's and the test's the level's", res.Failed[0].Step, res.Failed[1].Step)
}
}
// 1921/S3: the lint of a level that runs some packages vets those packages alone, in one pass with every tag their
// directory names, and reports a gofmt fault only in their files: a fault in a package the level keeps was reported
// when that package changed.
func TestTheLintOfSomePackagesVetsThemInOnePassAndFormatsOnlyTheirFiles(t *testing.T) {
tree := writeTree(t, map[string]string{
"go.mod": "module x\n\ngo 1.26\n",
"a/a.go": "package a\n\nfunc A() {}\n",
"a/a_integration_test.go": "//go:build integration\n\npackage a\n",
"b/b.go": "package b\nfunc B() {}\n",
"a/deeper/d.go": "package deeper\nfunc D() {}\n",
})
var vetted []string
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
vetted = append(vetted, strings.Join(args[2:], " "))
return forgekey.Ran{}
}
failed, err := (&Module{}).lint(context.Background(), run, tree, ".", nil, []string{"x/a"})
if err != nil {
t.Fatal(err)
}
if strings.Join(vetted, " | ") != "vet -tags integration x/a" {
t.Errorf("vet ran %v, want one pass over x/a with its tag", vetted)
}
if len(failed) != 0 {
t.Errorf("the faults of packages the level keeps were reported: %+v", failed)
}
vetted = nil
failed, err = (&Module{}).lint(context.Background(), run, tree, ".", nil, []string{"x/b"})
if err != nil {
t.Fatal(err)
}
if len(failed) != 1 || failed[0].Test != "gofmt" || !strings.Contains(failed[0].Output, "b/b.go") || strings.Contains(failed[0].Output, "d.go") {
t.Errorf("failed = %+v, want gofmt naming b/b.go alone", failed)
}
if strings.Join(vetted, " | ") != "vet -tags integration x/b" {
t.Errorf("vet ran %v", vetted)
}
}
// 1921/S3: a file a tag leaves out (//go:build !integration) is vetted by no pass with the tag, so a directory holding
// one keeps the pass without tags beside the pass with them.
func TestADirectoryWithAFileATagLeavesOutKeepsThePassWithoutTags(t *testing.T) {
tree := writeTree(t, map[string]string{
"go.mod": "module x\n",
"a.go": "package x\n",
"fake.go": "//go:build !integration\n\npackage x\n",
"real_test.go": "//go:build integration\n\npackage x\n",
})
var mu sync.Mutex
var vetted []string
run := func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
mu.Lock()
vetted = append(vetted, strings.Join(args[2:], " "))
mu.Unlock()
return forgekey.Ran{}
}
if _, err := (&Module{}).lint(context.Background(), run, tree, ".", nil, nil); err != nil {
t.Fatal(err)
}
slices.Sort(vetted)
if strings.Join(vetted, " | ") != "vet -tags integration ./... | vet ./..." {
t.Errorf("vet ran %v, want the pass without tags and the pass with them", vetted)
}
} }
@@ -54,6 +54,7 @@ func main() {
m.RegistryFile = m.Home + "/registry" m.RegistryFile = m.Home + "/registry"
m.Settings = m.Home + "/settings.json" m.Settings = m.Home + "/settings.json"
m.ReadKept = readOf(stdio.State(KeptState)).Get m.ReadKept = readOf(stdio.State(KeptState)).Get
m.ReadManifest = readOf(stdio.State(ManifestsState)).Get
m.Probe = ProbeGo(m.Home, m.Cache) m.Probe = ProbeGo(m.Home, m.Cache)
m.Bubblewrap = ProbeBubblewrap m.Bubblewrap = ProbeBubblewrap
// Outside a launch by the runtime — a build, a check — there is no node and no state to write. // Outside a launch by the runtime — a build, a check — there is no node and no state to write.
@@ -12,6 +12,7 @@ import (
"time" "time"
"git.novox.be/novox/mesh-sdk/go/delivery" "git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
) )
func fetchOnce(t *testing.T, m *Module, states *keptStates, args map[string]any) FetchResult { func fetchOnce(t *testing.T, m *Module, states *keptStates, args map[string]any) FetchResult {
@@ -188,3 +189,61 @@ func TestTheWorkspaceIsSelfContainedAfterAFetchThroughTheMirror(t *testing.T) {
t.Errorf("the workspace is not whole: %v %s", err, out) t.Errorf("the workspace is not whole: %v %s", err, out)
} }
} }
// 1921/S6: with a base, the mirror is brought to the trunk commit too, and the merge preview deepens the tree from the
// mirror: it asks the forge for no history the mirror holds.
func TestTheMirrorIsBroughtToTheBaseAndThePreviewFetchesFromIt(t *testing.T) {
f := &fakeStep{gomods: "go.mod\n", rule: previewRules(false)}
m, states := fetchModule(t, f)
mirror := filepath.Join(m.Cache, "mirror", "novox", "mesh-sdk.git")
f.files = map[string]string{"modules/m/module.json": "{}\n", "modules/m/go.sum": "sum\n", "go.mod": "module x\n", "go.sum": ""}
fetchOnce(t, m, states, map[string]any{"repository": "novox/mesh-sdk", "commit": commitSHA, "base": baseSHA})
lines := strings.Join(f.lines(), "\n")
if !strings.Contains(lines, "-C "+mirror+" fetch --no-tags origin "+baseSHA) {
t.Errorf("the mirror was not brought to the base:\n%s", lines)
}
if !strings.Contains(lines, "fetch --no-tags --unshallow "+mirror+" "+baseSHA) || strings.Contains(lines, "--unshallow origin") {
t.Errorf("the preview fetched the trunk's history from the forge:\n%s", lines)
}
}
// 1921/S6: the siblings are placed side by side: each sibling's fetch has started before any of them ends.
func TestTheSiblingsArePlacedAtOnce(t *testing.T) {
var mu sync.Mutex
started := map[string]bool{}
both := make(chan struct{})
f := &fakeStep{gomods: "go.mod\n"}
f.rule = func(line string) (forgekey.Ran, bool) {
if !strings.Contains(line, "/beside/") || !strings.Contains(line, " fetch --no-tags --depth 1 ") {
return forgekey.Ran{}, false
}
mu.Lock()
for _, name := range []string{"mesh-catalog", "mesh-host"} {
if strings.Contains(line, "/beside/"+name) {
started[name] = true
}
}
if len(started) == 2 {
select {
case <-both:
default:
close(both)
}
}
mu.Unlock()
select {
case <-both:
case <-time.After(2 * time.Second):
return forgekey.Ran{Status: 1, Stderr: "the other sibling's fetch never started"}, true
}
return forgekey.Ran{}, false
}
m, states := fetchModule(t, f)
res := fetchOnce(t, m, states, map[string]any{"repository": "novox/mesh-controller", "commit": commitSHA, "beside": []any{
map[string]any{"repository": "novox/mesh-host", "commit": commitSHA},
map[string]any{"repository": "novox/mesh-catalog", "commit": commitSHA},
}})
if len(res.Beside) != 2 || res.Beside[0].Dir != "mesh-catalog" || res.Beside[1].Dir != "mesh-host" {
t.Errorf("beside = %+v, want both, in name order", res.Beside)
}
}
@@ -82,6 +82,10 @@ type Module struct {
Settings string Settings string
Capacity func() int Capacity func() int
ReadKept func(key string) (json.RawMessage, error) ReadKept func(key string) (json.RawMessage, error)
// ReadManifest reads a key of the delivery indexer's manifests state (ManifestsState), which a compose ask's request
// names its manifests by (compose.go); GoModCache, when set, is the Go module cache in place of Cache's (a test).
ReadManifest func(key string) (json.RawMessage, error)
GoModCache string
spacesOnce sync.Once spacesOnce sync.Once
workspaces *buildstep.Workspaces workspaces *buildstep.Workspaces
@@ -0,0 +1,79 @@
package main
import (
"context"
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// 1925/AC1: packagesArg takes the list as the bus's JSON shape ([]any) and as the typed list an ask hands in process
// ([]string), and refuses a bad element in either.
func TestPackagesArgTakesBothShapes(t *testing.T) {
cases := []struct {
name string
in any
want []string
refused bool
}{
{"absent", nil, nil, false},
{"typed", []string{"x/a", "x/b/..."}, []string{"x/a", "x/b/..."}, false},
{"json", []any{"x/a", "x/b/..."}, []string{"x/a", "x/b/..."}, false},
{"typed empty", []string{}, []string{}, false},
{"typed dotdot", []string{"x/../y"}, nil, true},
{"json dotdot", []any{"x/../y"}, nil, true},
{"typed bad", []string{"x a"}, nil, true},
{"json bad", []any{"x a"}, nil, true},
{"json not a string", []any{3}, nil, true},
{"a string", "x/a", nil, true},
}
for _, c := range cases {
got, err := packagesArg(c.in)
if c.refused != (err != nil) {
t.Errorf("%s: err = %v, refused want %v", c.name, err, c.refused)
continue
}
if !c.refused && !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: got %#v, want %#v", c.name, got, c.want)
}
}
}
// 1925/AC2: a check ask through the module's real build_go_test body gets past the packages argument: the digest names
// a package, buildstep.RunAsk hands it in as a typed list, and the step fails only later, on the workspace nobody fetched.
func TestACheckAskReachesTheTestStepPastItsPackages(t *testing.T) {
m := &Module{Node: "ace", Work: t.TempDir()}
var calls []string
steps := fakeSteps(&calls)
for i := range steps {
switch steps[i].Name {
case "build_go_digest":
steps[i].Run = func(ctx context.Context, args map[string]any) (any, error) {
return map[string]any{"level": "unit", "packages": map[string]string{"x/a": "d1"}}, nil
}
case "build_go_test":
for _, real := range m.StepTools() {
if real.Name == "build_go_test" {
steps[i].Run = real.Run
}
}
}
}
a := delivery.StepAsk{ID: buildstep.NewID(), Kind: delivery.AskCheck, Language: delivery.AskGo, Asker: "mesh-delivery-checker",
At: time.Now(), Check: &delivery.CheckJob{Fetch: map[string]any{"repository": "novox/x", "commit": askCommit}, Levels: []string{delivery.LevelUnit}}}
out, err := buildstep.RunAsk(context.Background(), m.askSteps(steps), a)
got := ""
if err != nil {
got = err.Error()
}
if strings.Contains(got, "packages is a list") {
t.Fatalf("the test step refused the ask's packages: %s", got)
}
if !strings.Contains(got, "no workspace") {
t.Fatalf("want the step to fail on the unfetched workspace, past its packages; err=%v out=%+v", err, out)
}
}
@@ -13,6 +13,7 @@ import (
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/buildstep" "git.novox.be/novox/mesh-sdk/go/buildstep"
) )
@@ -34,8 +35,9 @@ func (m *Module) remote(repository string) string {
} }
// merge makes the merge preview of the head onto base and writes the answer into res. // merge makes the merge preview of the head onto base and writes the answer into res.
func (m *Module) merge(ctx context.Context, res *FetchResult, tree, remote, base, commit string) error { // opts are the fetch's options, with the mirror when it holds base.
mr, err := buildstep.MergeInto(ctx, m.Run, tree, remote, base, commit, m.FetchOptions...) func (m *Module) merge(ctx context.Context, res *FetchResult, tree, remote, base, commit string, opts []buildstep.Option) error {
mr, err := buildstep.MergeInto(ctx, m.Run, tree, remote, base, commit, opts...)
if err != nil { if err != nil {
return err return err
} }
@@ -89,23 +91,22 @@ func (m *Module) besideDir(workspace string) string {
// placeSiblings fetches each sibling into beside/<name> of the workspace root, as a plain directory. A symlink anywhere // placeSiblings fetches each sibling into beside/<name> of the workspace root, as a plain directory. A symlink anywhere
// in a placed tree fails the step naming it: the mesh creates none, and a test must not be led out of its sibling. // in a placed tree fails the step naming it: the mesh creates none, and a test must not be led out of its sibling.
func (m *Module) placeSiblings(ctx context.Context, root string, siblings []siblingArg) ([]Sibling, error) { func (m *Module) placeSiblings(ctx context.Context, root string, siblings []siblingArg) ([]Sibling, error) {
var placed []Sibling // The siblings are placed side by side (issue 1921, try speed S6); the first failure in the order given is the step's.
for _, s := range siblings { placed := make([]Sibling, len(siblings))
name := s.repository[strings.Index(s.repository, "/")+1:] errs := make([]error, len(siblings))
dir := filepath.Join(root, "beside", name) var wg sync.WaitGroup
if err := os.MkdirAll(dir, 0o700); err != nil { for i, s := range siblings {
wg.Add(1)
go func() {
defer wg.Done()
placed[i], errs[i] = m.placeSibling(ctx, root, s)
}()
}
wg.Wait()
for _, err := range errs {
if err != nil {
return nil, err return nil, err
} }
if err := buildstep.FetchCommit(ctx, m.Run, dir, m.remote(s.repository), s.commit, s.ref, m.fetchOptions(ctx, s.repository, s.commit, s.ref)...); err != nil {
return nil, fmt.Errorf("beside %s: %w", s.repository, err)
}
if err := m.dissociate(ctx, dir); err != nil {
return nil, fmt.Errorf("beside %s: %w", s.repository, err)
}
if link := firstSymlink(dir); link != "" {
return nil, fmt.Errorf("beside %s holds a symlink, %s", s.repository, link)
}
placed = append(placed, Sibling{Repository: s.repository, Commit: s.commit, Dir: name})
} }
sort.Slice(placed, func(i, j int) bool { return placed[i].Dir < placed[j].Dir }) sort.Slice(placed, func(i, j int) bool { return placed[i].Dir < placed[j].Dir })
if len(placed) > 0 { if len(placed) > 0 {
@@ -121,6 +122,25 @@ func (m *Module) placeSiblings(ctx context.Context, root string, siblings []sibl
return placed, nil return placed, nil
} }
// placeSibling fetches one sibling into beside/<name> of the workspace root.
func (m *Module) placeSibling(ctx context.Context, root string, s siblingArg) (Sibling, error) {
name := s.repository[strings.Index(s.repository, "/")+1:]
dir := filepath.Join(root, "beside", name)
if err := os.MkdirAll(dir, 0o700); err != nil {
return Sibling{}, err
}
if err := buildstep.FetchCommit(ctx, m.Run, dir, m.remote(s.repository), s.commit, s.ref, m.fetchOptions(ctx, s.repository, s.commit, s.ref)...); err != nil {
return Sibling{}, fmt.Errorf("beside %s: %w", s.repository, err)
}
if err := m.dissociate(ctx, dir); err != nil {
return Sibling{}, fmt.Errorf("beside %s: %w", s.repository, err)
}
if link := firstSymlink(dir); link != "" {
return Sibling{}, fmt.Errorf("beside %s holds a symlink, %s", s.repository, link)
}
return Sibling{Repository: s.repository, Commit: s.commit, Dir: name}, nil
}
// besideRecord is the file in a workspace's root naming the siblings the fetch placed beside it. // besideRecord is the file in a workspace's root naming the siblings the fetch placed beside it.
const besideRecord = "beside.json" const besideRecord = "beside.json"
@@ -85,7 +85,8 @@ func TestFetchWithABaseMergesAndAnswersThePreview(t *testing.T) {
t.Errorf("trees = %+v", res.Trees) t.Errorf("trees = %+v", res.Trees)
} }
lines := strings.Join(f.lines(), "\n") lines := strings.Join(f.lines(), "\n")
for _, want := range []string{"fetch --no-tags --unshallow origin " + baseSHA, "checkout -q --detach " + baseSHA, mirror := filepath.Join(m.Cache, "mirror", "novox", "mesh-sdk.git")
for _, want := range []string{"fetch --no-tags --unshallow " + mirror + " " + baseSHA, "checkout -q --detach " + baseSHA,
"core.hooksPath=/dev/null", "merge --no-ff --no-edit --no-verify"} { "core.hooksPath=/dev/null", "merge --no-ff --no-edit --no-verify"} {
if !strings.Contains(lines, want) { if !strings.Contains(lines, want) {
t.Errorf("git was not asked %q:\n%s", want, lines) t.Errorf("git was not asked %q:\n%s", want, lines)
@@ -290,8 +291,8 @@ func TestTheIntegrationLevelIsGivenTheBesideDirectoryOnlyWhenSiblingsWerePlaced(
} }
tested := 0 tested := 0
for _, c := range f.calls { for _, c := range f.calls {
if !strings.HasPrefix(c.line, "go ") || !strings.Contains(c.line, "-tags integration") { if !strings.HasPrefix(c.line, "go ") || !strings.Contains(c.line, "-tags integration") || strings.Contains(c.line, "-exec true") {
continue continue // the setup's compile runs no test binary, so it is given nothing a test reads (issue 1921, S5)
} }
tested++ tested++
got := "" got := ""
@@ -0,0 +1 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
@@ -24,9 +24,7 @@ func threeModules(t *testing.T, f *integFake) (*Module, *keptStates) {
if err := os.MkdirAll(dir, 0o755); err != nil { if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if err := os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module fixture/"+d+"\n\ngo 1.25.0\n"), 0o644); err != nil { writeTaggedTest(t, dir, "fixture/"+d)
t.Fatal(err)
}
} }
return m, states return m, states
} }
@@ -72,7 +70,7 @@ func TestOnlyTheSelectedDirectoryCompilesAndRuns(t *testing.T) {
if !strings.HasPrefix(l, "go -C "+filepath.Join(tree, "b")+" test ") { if !strings.HasPrefix(l, "go -C "+filepath.Join(tree, "b")+" test ") {
t.Errorf("go test ran outside the selected directory: %s", l) t.Errorf("go test ran outside the selected directory: %s", l)
} }
if strings.Contains(l, "-run ^$") { if strings.Contains(l, "-exec true") {
compiled++ compiled++
} else { } else {
ran++ ran++
@@ -91,7 +89,7 @@ func TestWithoutASelectionEveryDirectoryCompilesAndRuns(t *testing.T) {
compiled, ran := 0, 0 compiled, ran := 0, 0
for _, l := range strings.Split(f.lines(), "\n") { for _, l := range strings.Split(f.lines(), "\n") {
if strings.HasPrefix(l, "go ") && strings.Contains(l, " test ") { if strings.HasPrefix(l, "go ") && strings.Contains(l, " test ") {
if strings.Contains(l, "-run ^$") { if strings.Contains(l, "-exec true") {
compiled++ compiled++
} else { } else {
ran++ ran++
@@ -0,0 +1,81 @@
package main
// The bound on the processes one ask starts (issue 1921, try speed S10; ADR 0541): the levels of an ask run side by side,
// the unit level in this process and the integration level in a shut-off process of its own, and they share one bound,
// the processors divided by the node's build capacity. The bound lives in the ask's workspace as lock files, one per
// slot, so both processes count against it.
import (
"context"
"fmt"
"os"
"path/filepath"
"runtime"
"strconv"
"sync/atomic"
"syscall"
)
// slotsDir is the directory in a workspace's root holding its slots.
const slotsDir = "slots"
// askDirs are the directories an ask keeps in its workspace's root beside its tree, gone with the workspace: its cache
// layer and its bound on processes.
var askDirs = []string{cacheLayerDir, slotsDir}
// numCPU is the node's processor count.
func numCPU() int { return runtime.NumCPU() }
// slotPool is the bound of one workspace: n slots, each a lock file under dir.
type slotPool struct {
dir string
n int
next *atomic.Uint32
}
// slotTurns spreads the waits of the commands that find every slot taken over the slots.
var slotTurns atomic.Uint32
// slots is the bound of the workspace's processes, m.parallel() slots.
func (m *Module) slots(workspace string) slotPool {
return slotPool{dir: filepath.Join(m.Work, workspace, slotsDir), n: m.parallel(), next: &slotTurns}
}
// acquire takes a slot of the pool, waiting for one when every slot is taken, and answers what frees it. A free slot is
// taken at once; when none is, the wait is on one slot, taken in turn, until its holder frees it.
func (p slotPool) acquire(ctx context.Context) (func(), error) {
if err := os.MkdirAll(p.dir, 0o700); err != nil {
return nil, fmt.Errorf("the workspace's bound on processes: %w", err)
}
take := func(i int, how int) (func(), bool, error) {
f, err := os.OpenFile(filepath.Join(p.dir, "slot-"+strconv.Itoa(i)), os.O_CREATE|os.O_RDWR, 0o600)
if err != nil {
return nil, false, err
}
if err := syscall.Flock(int(f.Fd()), how); err != nil {
f.Close()
if err == syscall.EWOULDBLOCK {
return nil, false, nil
}
return nil, false, err
}
return func() { syscall.Flock(int(f.Fd()), syscall.LOCK_UN); f.Close() }, true, nil
}
for i := 0; i < p.n; i++ {
release, ok, err := take(i, syscall.LOCK_EX|syscall.LOCK_NB)
if err != nil {
return nil, fmt.Errorf("the workspace's bound on processes: %w", err)
}
if ok {
return release, nil
}
}
if err := ctx.Err(); err != nil {
return nil, err
}
release, _, err := take(int(p.next.Add(1))%p.n, syscall.LOCK_EX)
if err != nil {
return nil, fmt.Errorf("the workspace's bound on processes: %w", err)
}
return release, nil
}
@@ -0,0 +1,61 @@
package main
import (
"context"
"strings"
"sync/atomic"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/forgekey"
)
// 1921/S10 (ADR 0541): the levels of one ask, running side by side, share one bound on the processes they start: the
// bound is the workspace's, so a slot another level of the ask holds is one this level waits for.
func TestTheLevelsOfAnAskShareOneBoundOnProcesses(t *testing.T) {
m, states, ws, _ := workspaceFor(t, "failing")
m.Parallel = 1
held, err := m.slots(ws).acquire(context.Background()) // the other level holds the one slot
if err != nil {
t.Fatal(err)
}
var ran atomic.Int32
inner := m.Run
m.Run = func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
if strings.Contains(strings.Join(args, " "), " go ") || name == "go" {
ran.Add(1)
}
return inner(ctx, env, name, args...)
}
done := make(chan delivery.Step, 1)
go func() { _, st := runUnit(t, m, states, ws); done <- st }()
time.Sleep(150 * time.Millisecond)
if n := ran.Load(); n != 0 {
t.Errorf("%d go commands ran while the ask's one slot was held by its other level", n)
}
held()
select {
case st := <-done:
if st.State != delivery.StepPassed {
t.Errorf("the step ended %s: %s", st.State, st.Reason)
}
case <-time.After(3 * time.Second):
t.Fatal("the level never ran after the slot was freed")
}
if ran.Load() == 0 {
t.Error("no go command ran")
}
}
// 1921/S10 (ADR 0541): the bound is the processors divided by the node's build capacity, not each level half of them.
func TestTheBoundIsTheProcessorsDividedByTheBuildCapacity(t *testing.T) {
m := &Module{Capacity: func() int { return 3 }}
if got, want := m.parallel(), max(1, numCPU()/3); got != want {
t.Errorf("parallel = %d, want %d", got, want)
}
m = &Module{}
if got, want := m.parallel(), max(1, numCPU()/2); got != want {
t.Errorf("with no capacity known parallel = %d, want %d", got, want)
}
}
@@ -0,0 +1,161 @@
package main
// The integration level runs the tests its tags add, not the unit tests again (issue 1921, try speed S7): `go test
// -tags integration` without -run would run every untagged test too, and the unit level ran those already. Which tests a
// tag adds is read from the source: a test function in a _test.go file whose //go:build line holds with the level's
// tags and does not hold without them.
import (
"go/ast"
"go/build/constraint"
"go/parser"
"go/token"
"os"
"path/filepath"
"runtime"
"sort"
"strings"
"unicode"
"unicode/utf8"
)
// taggedRun is what the integration level runs in one go.mod directory: the packages holding a test the tags add, by
// import path, and the -run pattern naming those tests.
type taggedRun struct {
packages []string
pattern string
}
// taggedTests reads, for each go.mod directory, the tests that tags add. selected are the packages the level was asked
// to run (import paths, as build_go_digest keys them); none, or a relative pattern, is every package. A directory with no
// test the tags add is left out of the answer.
func taggedTests(tree string, dirs, selected, tags []string) (map[string]taggedRun, error) {
want := map[string]bool{}
for _, p := range selected {
if strings.HasPrefix(p, "./") || p == "." {
want = nil
break
}
want[p] = true
}
if len(selected) == 0 {
want = nil
}
with := map[string]bool{}
for _, t := range tags {
with[t] = true
}
out := map[string]taggedRun{}
for _, d := range dirs {
raw, _ := os.ReadFile(filepath.Join(tree, d, "go.mod"))
mp := modulePath(string(raw))
base := filepath.Join(tree, d)
pkgs := map[string]bool{}
names := map[string]bool{}
err := lintable(base, func(path string) error {
if !strings.HasSuffix(path, "_test.go") {
return nil
}
src, err := os.ReadFile(path)
if err != nil {
return err
}
expr := buildLine(src)
if expr == nil || !expr.Eval(func(tag string) bool { return with[tag] || platformTag(tag) }) || expr.Eval(platformTag) {
return nil // no constraint, or one the tags do not change: the unit level's
}
rel, _ := filepath.Rel(base, filepath.Dir(path))
pkg := mp
if rel != "." {
pkg = mp + "/" + filepath.ToSlash(rel)
}
if want != nil && !want[pkg] {
return nil
}
found := testNames(path, src)
for _, n := range found {
names[n] = true
}
if len(found) > 0 {
pkgs[pkg] = true
}
return nil
})
if err != nil {
return nil, err
}
if len(pkgs) == 0 {
continue
}
run := taggedRun{packages: sortedKeys(pkgs)}
run.pattern = "^(" + strings.Join(sortedKeys(names), "|") + ")$"
out[d] = run
}
return out, nil
}
func sortedKeys(m map[string]bool) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// buildLine is a file's //go:build constraint, nil when it has none.
func buildLine(src []byte) constraint.Expr {
for _, line := range strings.Split(string(src), "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "package ") {
break
}
if constraint.IsGoBuild(line) {
if e, err := constraint.Parse(line); err == nil {
return e
}
}
}
return nil
}
// platformTag is how a tag other than the level's evaluates, the same with the level's tags and without them: the
// machine's platform, unix, cgo and every Go version hold; any other tag does not.
func platformTag(tag string) bool {
switch tag {
case runtime.GOOS, runtime.GOARCH, "unix", "cgo", "gc":
return true
}
return strings.HasPrefix(tag, "go1.")
}
// testNames are the top-level test functions of a test file: func TestXxx, as go test runs them, TestMain left out.
func testNames(path string, src []byte) []string {
f, err := parser.ParseFile(token.NewFileSet(), path, src, parser.SkipObjectResolution)
if err != nil {
return nil // a file that does not parse fails the compile, which the run reports
}
var out []string
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Recv != nil || fn.Name.Name == "TestMain" || !isTestName(fn.Name.Name) {
continue
}
out = append(out, fn.Name.Name)
}
return out
}
// isTestName says whether go test runs a function of this name as a test: Test, then nothing or a rune that is not
// lower case.
func isTestName(name string) bool {
rest, ok := strings.CutPrefix(name, "Test")
if !ok {
return false
}
if rest == "" {
return true
}
r, _ := utf8.DecodeRuneInString(rest)
return !unicode.IsLower(r)
}
+80 -31
View File
@@ -10,7 +10,6 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
"runtime"
"strings" "strings"
"sync" "sync"
"time" "time"
@@ -127,17 +126,25 @@ func (m *Module) besideEnv(workspace string) []string {
return []string{"MESH_CHECK_BESIDE=" + dir} return []string{"MESH_CHECK_BESIDE=" + dir}
} }
// packagesArg reads the packages argument: a list of package patterns, as []any when it came over the bus as JSON and
// as []string when an ask handed it in process (issue 1925).
func packagesArg(v any) ([]string, error) { func packagesArg(v any) ([]string, error) {
if v == nil { var list []string
switch t := v.(type) {
case nil:
return nil, nil return nil, nil
} case []string:
list, ok := v.([]any) list = t
if !ok { case []any:
for _, e := range t {
s, _ := e.(string)
list = append(list, s)
}
default:
return nil, fmt.Errorf("packages is a list of package patterns") return nil, fmt.Errorf("packages is a list of package patterns")
} }
out := []string{} out := []string{}
for _, e := range list { for _, s := range list {
s, _ := e.(string)
if !packageOK.MatchString(s) || strings.Contains(strings.TrimSuffix(s, "/..."), "..") { if !packageOK.MatchString(s) || strings.Contains(strings.TrimSuffix(s, "/..."), "..") {
return nil, fmt.Errorf("%q is not a package pattern", s) return nil, fmt.Errorf("%q is not a package pattern", s)
} }
@@ -163,6 +170,9 @@ type levelOpts struct {
// parallelism, where the unit level's is the slowest package's. // parallelism, where the unit level's is the slowest package's.
setup time.Duration setup time.Duration
wall bool wall bool
// runByDir is the -run pattern of each go.mod directory, over run: the tests the integration level's tags add there
// (issue 1921, try speed S7).
runByDir map[string]string
// compose is a state of the level to put in the result (the store and the bus it ran against). // compose is a state of the level to put in the result (the store and the bus it ran against).
store, bus string store, bus string
} }
@@ -197,37 +207,56 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
todo = append(todo, d) todo = append(todo, d)
} }
} }
// The level's commands share one bound on processes with each other and with the ask's other level: the lint's and
// the tests' (issue 1921, try speed S4, S10).
slots := m.slots(o.workspace)
var lintFailed []Failure
var lintErr error
var lintDone chan struct{}
var crash any // a panic of a run or of the lint goes on in the caller, which cleans up after it
var crashMu sync.Mutex
if o.level == LevelUnit { if o.level == LevelUnit {
// gofmt and go vet of each directory the level runs come before its tests (ADR 0529 §1; issue 1887 D7-go): a // gofmt and go vet of each directory the level runs, beside its tests (ADR 0529 §1; issue 1887 D7-go; issue 1921,
// directory that fails them fails the level at once, and their time is the result's Lint. // try speed S4): a directory that fails them fails the level, and their time is the result's Lint.
linted := m.clock() lintDone = make(chan struct{})
var failed []Failure go func() {
for _, d := range todo { defer close(lintDone)
f, err := m.lint(ctx, run, tree, d, append(m.goEnvFor(filepath.Join(tree, d)), o.env...)) defer func() {
if p := recover(); p != nil {
crashMu.Lock()
crash = p
crashMu.Unlock()
}
}()
linted := m.clock()
// go vet compiles and runs nothing of the change, so it warms the shared cache for every later check, a pull
// request's head included (issue 1921, try speed S11; the operator's decision); the tests run beneath the
// ask's own layer.
compile, err := m.stepRunner(o.workspace, true)
if err != nil { if err != nil {
return nil, err lintErr = err
return
} }
failed = append(failed, f...) lintFailed, lintErr = m.lintAll(ctx, compile, tree, todo, func(d string) []string {
} return append(m.goEnvFor(filepath.Join(tree, d)), o.env...)
res.Lint = m.clock().Sub(linted).Seconds() }, byDir, slots.acquire)
if len(failed) > 0 { lintFailed = append(m.addedBinaries(ctx, m.Run, tree), lintFailed...) // git alone, no code of the change
res.Failed = failed res.Lint = m.clock().Sub(linted).Seconds()
res.judge(o.bounds) }()
return res, nil
}
began = m.clock() // the level's running time is its tests'
} }
runs := make([]dirRun, len(todo)) runs := make([]dirRun, len(todo))
slots := make(chan struct{}, m.parallel())
var wg sync.WaitGroup var wg sync.WaitGroup
var crash any // a panic of a run goes on in the caller, which cleans up after it var slotErr error
var crashMu sync.Mutex
for i, d := range todo { for i, d := range todo {
release, err := slots.acquire(ctx)
if err != nil {
slotErr = err
break
}
wg.Add(1) wg.Add(1)
slots <- struct{}{}
go func() { go func() {
defer wg.Done() defer wg.Done()
defer func() { <-slots }() defer release()
defer func() { defer func() {
if p := recover(); p != nil { if p := recover(); p != nil {
crashMu.Lock() crashMu.Lock()
@@ -239,7 +268,9 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
if len(o.tags) > 0 { if len(o.tags) > 0 {
args = append(args, "-tags", strings.Join(o.tags, ",")) args = append(args, "-tags", strings.Join(o.tags, ","))
} }
if o.run != "" { if r := o.runByDir[d]; r != "" {
args = append(args, "-run", r)
} else if o.run != "" {
args = append(args, "-run", o.run) args = append(args, "-run", o.run)
} }
args = append(args, byDir[d]...) args = append(args, byDir[d]...)
@@ -252,9 +283,19 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
}() }()
} }
wg.Wait() wg.Wait()
if lintDone != nil {
<-lintDone
}
if crash != nil { if crash != nil {
panic(crash) panic(crash)
} }
if lintErr != nil {
return nil, lintErr
}
if slotErr != nil {
return nil, slotErr
}
res.Failed = append(res.Failed, lintFailed...) // the lint's failures first, then the tests'
for i, d := range todo { for i, d := range todo {
dr := runs[i] dr := runs[i]
if (dr.r.Status != 0 || dr.r.Err != "") && len(dr.rd.pkgs) == 0 { if (dr.r.Status != 0 || dr.r.Err != "") && len(dr.rd.pkgs) == 0 {
@@ -280,12 +321,20 @@ func (m *Module) runLevel(ctx context.Context, tree string, o levelOpts) (*TestR
// runAnswer is what a step runner answers. // runAnswer is what a step runner answers.
type runAnswer = forgekey.Ran type runAnswer = forgekey.Ran
// parallel is how many directories a level runs at once. // parallel is how many processes the levels of one ask start at once, together (ADR 0541; issue 1921, try speed S10):
// the processors divided by the node's build capacity, half of them when the capacity is not known.
func (m *Module) parallel() int { func (m *Module) parallel() int {
if m.Parallel > 0 { if m.Parallel > 0 {
return m.Parallel return m.Parallel
} }
return max(1, runtime.NumCPU()/2) capacity := 0
if m.Capacity != nil {
capacity = m.Capacity()
}
if capacity <= 0 {
capacity = 2
}
return max(1, numCPU()/capacity)
} }
// packagesByDir says which patterns run in which go.mod directory. A relative pattern (./...) runs in every directory, as // packagesByDir says which patterns run in which go.mod directory. A relative pattern (./...) runs in every directory, as
@@ -66,7 +66,9 @@ func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
"tagged": {passed: true, noTests: true}, "tagged": {passed: true, noTests: true},
"empty": {passed: true, noTests: true}, "empty": {passed: true, noTests: true},
"slow": {passed: true}, "slow": {passed: true},
"broken": {failed: "."}, // go vet, before the tests since R4g (ADR 0529 §1), names the directory and the compiler's line // go vet, the lint step beside the tests (ADR 0542 §2), names the directory and the compiler's line; the test run
// of the package that does not compile fails beside it
"broken": {failed: "."},
} }
for name, w := range want { for name, w := range want {
home := t.TempDir() home := t.TempDir()
@@ -84,14 +86,16 @@ func TestUnitLevelRunsTheRealGoOnTheFixtureRepositories(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("%s: %v", name, err) t.Fatalf("%s: %v", name, err)
} }
if res.Passed != w.passed || res.NoTests != w.noTests || (w.failed != "" && (len(res.Failed) != 1 || res.Failed[0].Package != w.failed)) { lintFirst := w.failed != "." || (len(res.Failed) >= 1 && res.Failed[0].Step == StepLint)
if res.Passed != w.passed || res.NoTests != w.noTests || !lintFirst ||
(w.failed != "" && (len(res.Failed) == 0 || res.Failed[0].Package != w.failed)) {
t.Errorf("%s: result = %+v", name, res) t.Errorf("%s: result = %+v", name, res)
} }
} }
} }
// 1887/D7-go against the real go: a tree holding an unformatted file fails the unit level naming the file, before any // 1887/D7-go against the real go: a tree holding an unformatted file fails the check naming the file, as the lint step's
// test runs, and the level's result carries the lint's time. // failure (ADR 0542 §2), and the level's result carries the lint's time.
func TestAnUnformattedFileFailsTheRealUnitLevel(t *testing.T) { func TestAnUnformattedFileFailsTheRealUnitLevel(t *testing.T) {
needTools(t, "the unit level's gofmt and vet", "go", "git", "bwrap") needTools(t, "the unit level's gofmt and vet", "go", "git", "bwrap")
home := t.TempDir() home := t.TempDir()
@@ -111,8 +115,9 @@ func TestAnUnformattedFileFailsTheRealUnitLevel(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if res.Passed || len(res.Failed) == 0 || res.Failed[0].Test != "gofmt" || !strings.Contains(res.Failed[0].Output, "ugly.go") || res.Tests != 0 { if res.Passed || len(res.Failed) == 0 || res.Failed[0].Test != "gofmt" || res.Failed[0].Step != StepLint ||
t.Fatalf("result = %+v, want the level failed by gofmt naming ugly.go and no test run", res) !strings.Contains(res.Failed[0].Output, "ugly.go") || res.Lint <= 0 {
t.Fatalf("result = %+v, want the lint step failed by gofmt naming ugly.go, with its time", res)
} }
} }
@@ -156,9 +161,8 @@ func TestIntegrationLevelRunsAgainstARealStoreAndBusAndLeavesNothing(t *testing.
if !res.Passed || res.Tests != 1 || res.Setup <= 0 || res.Setup > IntegrationSetup.Seconds() { if !res.Passed || res.Tests != 1 || res.Setup <= 0 || res.Setup > IntegrationSetup.Seconds() {
t.Errorf("result = %+v", res) t.Errorf("result = %+v", res)
} }
entries, _ := os.ReadDir(filepath.Join(work, "ws1")) if left := workspaceLeft(filepath.Join(work, "ws1")); len(left) != 1 || left[0] != "tree" {
if len(entries) != 1 || entries[0].Name() != "tree" { t.Errorf("the workspace holds %v besides the ask's own directories, want only its tree: the run directory is gone", left)
t.Errorf("the workspace holds %v, want only its tree: the run directory is gone", entries)
} }
for deadline := time.Now().Add(5 * time.Second); len(processesNamed(work)) > 0 && time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) { for deadline := time.Now().Add(5 * time.Second); len(processesNamed(work)) > 0 && time.Now().Before(deadline); time.Sleep(100 * time.Millisecond) {
} }
@@ -6,6 +6,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
@@ -25,8 +26,11 @@ func capturedRunner(t *testing.T, captured string, calls *[]cmdRecord) forgekey.
if strings.Contains(string(raw), `"Action":"fail"`) { if strings.Contains(string(raw), `"Action":"fail"`) {
status = 1 // go test exits 1 when a test or a build failed status = 1 // go test exits 1 when a test or a build failed
} }
var mu sync.Mutex // the lint and the tests of a level run at once (issue 1921, try speed S4)
return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran { return func(ctx context.Context, env []string, name string, args ...string) forgekey.Ran {
mu.Lock()
*calls = append(*calls, cmdRecord{env: env, line: name + " " + strings.Join(args, " ")}) *calls = append(*calls, cmdRecord{env: env, line: name + " " + strings.Join(args, " ")})
mu.Unlock()
switch { switch {
case name == "git" && args[2] == "ls-files": case name == "git" && args[2] == "ls-files":
return forgekey.Ran{Stdout: "go.mod\n"} return forgekey.Ran{Stdout: "go.mod\n"}
@@ -103,6 +107,33 @@ func TestUnitLevelNamesTheFailingTestAndFailsTheLevel(t *testing.T) {
} }
} }
// Issue 1921: a fixture with a failing test in each of two packages: one level's run names both, each with its own
// cause, and counts every test, the passing one beside a failure too. go test runs every package to its end; nothing
// stops at the first failure.
func TestUnitLevelNamesEveryFailingTestOfEveryPackage(t *testing.T) {
m, states, ws, calls := workspaceFor(t, "twofailing")
res, st := runUnit(t, m, states, ws)
if st.State != delivery.StepPassed || res.Passed {
t.Fatalf("step %s (%s), result %+v: want the step done and the level failed", st.State, st.Reason, res)
}
var got []string
for _, f := range res.Failed {
got = append(got, f.Package+"."+f.Test+": "+f.Cause)
}
want := []string{"fixture/twofailing/one.TestOneIsTwo: One() = 1, want 2", "fixture/twofailing/two.TestTwoIsThree: Two() = 2, want 3"}
if strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("failed\n%s\nwant\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
}
if res.Tests != 3 {
t.Errorf("%d tests counted, want 3", res.Tests)
}
for _, c := range *calls {
if strings.Contains(c.line, " test ") && strings.Contains(c.line, "-failfast") {
t.Errorf("go test ran with -failfast: %s", c.line)
}
}
}
// 1290/AC2: a fixture whose tests take more than the bound (lowered by the module's option, not by waiting five // 1290/AC2: a fixture whose tests take more than the bound (lowered by the module's option, not by waiting five
// seconds): the level fails naming the bound and the slowest tests; under the real bound the same run passes. // seconds): the level fails naming the bound and the slowest tests; under the real bound the same run passes.
func TestUnitLevelOverItsBoundFailsNamingTheBoundAndTheSlowestTests(t *testing.T) { func TestUnitLevelOverItsBoundFailsNamingTheBoundAndTheSlowestTests(t *testing.T) {
@@ -1,6 +1,7 @@
Each file is the standard output of `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go test -count=1 -json ./...` (go1.27.1) Each file is the standard output of `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go test -count=1 -json ./...` (go1.27.1)
run in the fixture module of the same name under ../repos, captured on 2026-10-20 for issue 1290. The unit run in the fixture module of the same name under ../repos, captured on 2026-10-20 for issue 1290. The unit
tests feed them to the parser through a fake runner, so no test here runs `go test`. tests feed them to the parser through a fake runner, so no test here runs `go test`. twofailing.json was captured the
same way on 2026-10-24 for issue 1921: two packages with a failing test each, and a passing test beside one of them.
golist-external.json is `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go list -json=ImportPath,Standard,DepOnly,Module golist-external.json is `GOTOOLCHAIN=go1.27.1 GOFLAGS=-mod=mod go list -json=ImportPath,Standard,DepOnly,Module
git.novox.be/novox/mesh-sdk/go/buildstep` run in this module (go1.27.1, 2026-10-20 for issue 1291): a package of a module git.novox.be/novox/mesh-sdk/go/buildstep` run in this module (go1.27.1, 2026-10-20 for issue 1291): a package of a module
@@ -0,0 +1,23 @@
{"Time":"2026-10-24T23:45:37.721718705+02:00","Action":"start","Package":"fixture/twofailing/one"}
{"Time":"2026-10-24T23:45:37.723685152+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsTwo"}
{"Time":"2026-10-24T23:45:37.723726472+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"=== RUN TestOneIsTwo\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723743682+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:6: checking One\n"}
{"Time":"2026-10-24T23:45:37.723750622+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":" one_test.go:8: One() = 1, want 2\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.723759302+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Output":"--- FAIL: TestOneIsTwo (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723889303+02:00","Action":"start","Package":"fixture/twofailing/two"}
{"Time":"2026-10-24T23:45:37.723762972+02:00","Action":"fail","Package":"fixture/twofailing/one","Test":"TestOneIsTwo","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723952274+02:00","Action":"run","Package":"fixture/twofailing/one","Test":"TestOneIsOne"}
{"Time":"2026-10-24T23:45:37.723955484+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"=== RUN TestOneIsOne\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723959304+02:00","Action":"output","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Output":"--- PASS: TestOneIsOne (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.723964914+02:00","Action":"pass","Package":"fixture/twofailing/one","Test":"TestOneIsOne","Elapsed":0}
{"Time":"2026-10-24T23:45:37.723967704+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724225126+02:00","Action":"output","Package":"fixture/twofailing/one","Output":"FAIL\tfixture/twofailing/one\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.724231936+02:00","Action":"fail","Package":"fixture/twofailing/one","Elapsed":0.003}
{"Time":"2026-10-24T23:45:37.725752949+02:00","Action":"run","Package":"fixture/twofailing/two","Test":"TestTwoIsThree"}
{"Time":"2026-10-24T23:45:37.725796309+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"=== RUN TestTwoIsThree\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.725808239+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":" two_test.go:7: Two() = 2, want 3\n","OutputType":"error"}
{"Time":"2026-10-24T23:45:37.72581671+02:00","Action":"output","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Output":"--- FAIL: TestTwoIsThree (0.00s)\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.72582026+02:00","Action":"fail","Package":"fixture/twofailing/two","Test":"TestTwoIsThree","Elapsed":0}
{"Time":"2026-10-24T23:45:37.72582549+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726319344+02:00","Action":"output","Package":"fixture/twofailing/two","Output":"FAIL\tfixture/twofailing/two\t0.002s\n","OutputType":"frame"}
{"Time":"2026-10-24T23:45:37.726362544+02:00","Action":"fail","Package":"fixture/twofailing/two","Elapsed":0.002}
@@ -0,0 +1,3 @@
module fixture/twofailing
go 1.25.0
@@ -0,0 +1,3 @@
package one
func One() int { return 1 }
@@ -0,0 +1,16 @@
package one
import "testing"
func TestOneIsTwo(t *testing.T) {
t.Log("checking One")
if One() != 2 {
t.Fatalf("One() = %d, want 2", One())
}
}
func TestOneIsOne(t *testing.T) {
if One() != 1 {
t.Fatal("one")
}
}
@@ -0,0 +1,3 @@
package two
func Two() int { return 2 }
@@ -0,0 +1,9 @@
package two
import "testing"
func TestTwoIsThree(t *testing.T) {
if Two() != 3 {
t.Errorf("Two() = %d, want 3", Two())
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildgo
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
+10 -2
View File
@@ -1,5 +1,13 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024214154-bc16740feb0d h1:oJREw0ggKqJWJ5DEISSL0i2ZC5XqdXAidxo9NJ74TCc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024214154-bc16740feb0d/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024225846-cb548a6a588c h1:GWHP9ozfkFXHVEL2gE4Xa5kc5e8uPO+UL9+mau6iaIs=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024225846-cb548a6a588c/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024230839-4a12ccb78d19 h1:NqU4hkk/jHdTEcGFnnXvdOJwD8dH5u+qWaCdQQ+JYMc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024230839-4a12ccb78d19/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232250-9b408db24a58 h1:4Damtdqlf6u6nJhvMA99peDy1jmjixtcwFXJ4bWEkLk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232250-9b408db24a58/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95 h1:I8zbgGzRjmgYydPTWo8hmaZW/P/g+qGDUfu3E32/pkU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+2 -1
View File
@@ -20,7 +20,8 @@
} }
], ],
"reads": [ "reads": [
"mesh-delivery-checker.kept" "mesh-delivery-checker.kept",
"mesh-delivery-indexer.manifests"
], ],
"config": { "config": {
"capacity": { "capacity": {
@@ -0,0 +1 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildimage
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93
github.com/nats-io/nats-server/v2 v2.11.17 github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93 h1:FAiFY2KBKEoZSH49OZTt7/zRBIQxyl5g/KhYrpggbN4=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023213200-403a5d72ab93/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo= github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -0,0 +1 @@
These tests read other catalogue modules' files, outside this module (issue 1935): never kept.
+1 -1
View File
@@ -3,7 +3,7 @@ module meshbuildnode
go 1.26.0 go 1.26.0
require ( require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95
github.com/distribution/distribution/v3 v3.1.2 github.com/distribution/distribution/v3 v3.1.2
github.com/nats-io/nats.go v1.54.0 github.com/nats-io/nats.go v1.54.0
) )
+4 -2
View File
@@ -1,5 +1,7 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468 h1:UcMee+IV2ii5E4dRZdQUaSaeh9IedTFYumuQbBiB0k0= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024214154-bc16740feb0d h1:oJREw0ggKqJWJ5DEISSL0i2ZC5XqdXAidxo9NJ74TCc=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261023150321-a2c813a6b468/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY= git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024214154-bc16740feb0d/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95 h1:I8zbgGzRjmgYydPTWo8hmaZW/P/g+qGDUfu3E32/pkU=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024232922-99407c37df95/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI= github.com/alicebob/miniredis/v2 v2.35.0 h1:QwLphYqCEAo1eu1TqPRN2jgVMPBweeQcR21jeqDCONI=
github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= github.com/alicebob/miniredis/v2 v2.35.0/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0= github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
+12 -27
View File
@@ -10,15 +10,15 @@ Each directory or archive this module declares says how it treats a file it did
- `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2). - `${dir:state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
## A backup that failed before the send ## What it keeps (ADR 0539 §17)
The controller backs up the data of the modules a delivery moves before it asks or sends anything (ADR 0404). When its The gate keeps the merge's states: proposed, ready, rejected, merged, held (a merge that reached the trunk without a passing
walk's record, in `plan-moved`, names a backup that failed or stayed silent and nothing of the plan was sent, the check) and closed. It asks the operator's Approve and merges once, moving `source.main`. It asks the controller nothing, follows
delivery goes from delivering to held (the observed row `backup failed`); the held reason names the machine, the no plan event and derives nothing after the merge: whether a merged change reached the machines is the view's, from
module and the backup's words, and the pull request's comment shows it. The hold does not lift by itself. `release` `source.main`, the wants and the haves. The seat `mesh-delivery` keeps the verbs `deliveries`, `show`, `groups`, `checks`,
of such a delivery asks the controller (`delivery-backups-again`) to take the backups again and the delivery goes on `table`, `stalled` (for a merge or a check past its bound), `recheck` and `close`; `times`, `what-if`, `release`, `stop` and
delivering; a record from before the controller cleared the failure does not hold it again. A held delivery already `retire-history` went with the plans, so the gate stays a bundle serving the seat. A delivery an earlier owner kept in a state that
ends as covered or delivered when a later merge covers it. is gone is read as the state it came to (merged, closed or proposed) when the gate starts.
## When a refused or rejected merge is said (ADR 0505 §4, §5) ## When a refused or rejected merge is said (ADR 0505 §4, §5)
@@ -33,23 +33,8 @@ names every refusal and rejection at once, and the pull request is told at once;
`needs: new-head` and names its author: the branch, and the session a `Pushed-by: <who@node>` line in the pull `needs: new-head` and names its author: the branch, and the session a `Pushed-by: <who@node>` line in the pull
request's description names. request's description names.
## Taking the seat over from today's module ## The seat's hand-over
The gate claims the seat `mesh-delivery` and took it over from the module `mesh-delivery`, which the catalogue no longer holds. The gate took the seat `mesh-delivery` over from the module `mesh-delivery` (ADR 0449 §4, ADR 0450 §6, ADR 0454) and carried its
The controller refuses a merge that removes a module that is assigned, so today's module stayed in the catalogue until it was deliveries, groups and pull keys once; the carry, and the reading of the controller's assignments and the node-engine's have keys
unassigned, and went by a handover of verbs after the merge, never by hand on a machine (novox/hq ADR 0449 §4, ADR 0450 §6, ADR 0454): that waited for it, went with release 6. `catalogue_test.go` keeps the catalogue to one claimant of the seat.
1. assign `mesh-delivery-gate` to the machine that holds the seat (`mesh-controller.assign`);
2. check that `mesh-controller.seats` still names `mesh-delivery`, held by the running module;
3. unassign `mesh-delivery` from that machine (`mesh-controller.unassign`) and push it (`mesh-controller.push`);
4. the seat passes: the gate reads the controller's assignments and the node-engine's have keys, finds the module on no machine,
carries its deliveries, groups and pull keys over once (the key `carried` in its `deliveries` says it did), reads the controller's
walks and the forge's open pull requests, and acts.
Until then the gate is silent: it takes no event, writes no key, asks nothing and answers no verb but `delivery_status`, which
says why (`holds-the-seat`, `seat-waits`).
The removal of `modules/mesh-delivery` followed the handover as a pull request of its own, merged by the gate (ADR 0454); the
catalogue has one claimant of the seat, and no manifest but the gate's and the issue tracker's names the old module but the names
the handover needed: the three buckets the gate carries over, and the event the issue tracker and the tool runner hear
(`catalogue_test.go`).
@@ -1,50 +1,32 @@
package main package main
import ( import (
"strings"
"testing" "testing"
"time" "time"
) )
// One build per change (novox/hq ADR 0350): a pull request closed unmerged has its check builds let go of, and a const (
// walk's record says per module whether its build reused a check's build or built, and why. cat = "novox/mesh-catalog"
headCC = "cc00000000000001"
headDu = "dd00000000000002"
)
// A pull request closed unmerged is stopped. Since issue 1318 (ADR 0503 §3) the controller is no longer asked to let go var atCC = time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)
// of what its checks built (§5's forget-checks goes with the merge gate's checks); a controller that is down changes
// nothing. // A pull request closed unmerged is closed. Since issue 1318 (ADR 0503 §3) the controller is no longer asked to let go
func TestAPullRequestClosedUnmergedIsStoppedAndAsksTheControllerNothing(t *testing.T) { // of what its checks built, and since ADR 0539 §17 the gate asks it nothing at all.
w := newWorldOver(t, newMemStore(), newFakeController(), newFakeForge(), atCC) func TestAPullRequestClosedUnmergedIsClosedAndAsksNoOne(t *testing.T) {
w := newWorldOver(t, newMemStore(), newFakeForge(), atCC)
cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json") cc := pr(cat, 101, headCC, "feat/claude-code", "modules/claude-code/module.json")
w.h.PullUpdated(cc) w.h.PullUpdated(cc)
w.h.PullClosed(cc) w.h.PullClosed(cc)
if len(w.ctl.forgot) != 0 { if s := w.state(IDOf(cat, headCC)); s != Closed {
t.Fatalf("closed unmerged, the controller was asked to let go of %v", w.ctl.forgot)
}
if s := w.state(IDOf(cat, headCC)); s != Stopped {
t.Fatalf("closed unmerged it is %s", s) t.Fatalf("closed unmerged it is %s", s)
} }
w.ctl.down = true
du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json") du := pr(cat, 102, headDu, "feat/dunst", "modules/dunst/module.json")
w.h.PullUpdated(du) w.h.PullUpdated(du)
w.h.PullClosed(du) w.h.PullClosed(du)
if s := w.state(IDOf(cat, headDu)); s != Stopped { if s := w.state(IDOf(cat, headDu)); s != Closed {
t.Fatalf("with the controller down, a closed delivery is %s", s) t.Fatalf("a closed delivery is %s", s)
}
}
// The walk's record says per module what its build did, and the delivery keeps it beside its walk (§4).
func TestAWalksRecordSaysWhetherEachModuleWasReusedOrBuilt(t *testing.T) {
walk := catWalk("plan-1", mergeCC, atCC.Add(time.Minute), "claude-code", "dunst")
walk.Modules["claude-code"].Made = "reused from the check of novox/mesh-catalog#101 at head cccccccccccc " +
"(merged into trunk 553b71910000): tools sha256:aa"
walk.Modules["dunst"].Made = "built: no check of novox/mesh-catalog#101 built it"
seen := walk.Seen()
if !strings.HasPrefix(seen.Built["claude-code"], "reused from the check of novox/mesh-catalog#101") ||
seen.Built["dunst"] != "built: no check of novox/mesh-catalog#101 built it" {
t.Fatalf("the delivery keeps %v", seen.Built)
}
if catWalk("plan-2", mergeCC, atCC, "dunst").Seen().Built != nil {
t.Fatal("a walk from a controller that says nothing of its builds reads as if it said something")
} }
} }
@@ -1,14 +1,11 @@
package main package main
import ( import (
"encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"time" "time"
stdio "git.novox.be/novox/mesh-sdk/go"
) )
// The tests of novox/hq ADR 0505 "How it is checked" for the gate: a refusal the gate is already answering is no // The tests of novox/hq ADR 0505 "How it is checked" for the gate: a refusal the gate is already answering is no
@@ -76,7 +73,7 @@ func refusedForAMovedTrunk(t *testing.T) (*world, string) {
t.Helper() t.Helper()
w := announced(t) w := announced(t)
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
w.forge.refuse = errAskedAgain w.forge.refuse = errAskedAgain
approve(w) approve(w)
if len(w.forge.merges) != 1 || w.delivery(id).Ready.RefusedSince == nil { if len(w.forge.merges) != 1 || w.delivery(id).Ready.RefusedSince == nil {
@@ -125,7 +122,7 @@ func TestATrunkMovedRefusalIsStalledAtTwiceTheLastCheckTimeAtLeastTheFloor(t *te
func TestAnyOtherRefusalIsStalledAtOnce(t *testing.T) { func TestAnyOtherRefusalIsStalledAtOnce(t *testing.T) {
w := announced(t) w := announced(t)
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
w.forge.refuse = refusedWords("novox/app#7") w.forge.refuse = refusedWords("novox/app#7")
approve(w) approve(w)
if got := linesFor(w, id); len(got) != 1 { if got := linesFor(w, id); len(got) != 1 {
@@ -146,7 +143,7 @@ func rejectedAfterItsApprove(t *testing.T, pushedBy string) (*world, string) {
w.h.PullUpdated(p) w.h.PullUpdated(p)
w.settleAll() w.settleAll()
id := IDOf("novox/app", readyHead) id := IDOf("novox/app", readyHead)
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "pass", "t1t1t1t1t1t1"))
// Judged again (the table's recheck, which the checker's verdict on a new trunk makes): proposed once more. // Judged again (the table's recheck, which the checker's verdict on a new trunk makes): proposed once more.
w.h.mu.Lock() w.h.mu.Lock()
d := w.h.deliveries[id] d := w.h.deliveries[id]
@@ -156,7 +153,7 @@ func rejectedAfterItsApprove(t *testing.T, pushedBy string) (*world, string) {
} }
d.Check = nil d.Check = nil
w.h.mu.Unlock() w.h.mu.Unlock()
hear(t, w, ControllerSeat+".checked", judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1")) hear(t, w, shimCheckedKey, judgedAt("novox/app", 7, readyHead, "fail", "t1t1t1t1t1t1"))
approve(w) approve(w)
w.h.React() w.h.React()
if w.state(id) != Rejected { if w.state(id) != Rejected {
@@ -220,48 +217,3 @@ func TestANewHeadWithinTwoHoursKeepsTheRejectedHeadFromTheOperator(t *testing.T)
t.Fatalf("the superseded head reached the operator: %+v", got) t.Fatalf("the superseded head reached the operator: %+v", got)
} }
} }
// ADR 0505 §2: a walk's end said again by the controller with its declared machines (`declared_read`, a touch marked
// declared) acts once: no transition, comment or ask of the controller the first saying did not make.
func TestAWalkSaidAgainWithItsDeclaredMachinesActsOnceInTheGate(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
done := w.ctl.walk(plan)
sent := time.Date(2026, 10, 6, 12, 40, 0, 0, time.UTC)
done.Modules["app"] = &WalkModule{State: "built", First: []string{"anchor"}, FirstAt: &sent, SentAt: &sent, Build: "build-1",
Gate: &WalkGate{Machines: []string{"anchor"}, Since: &sent, Passes: 1, Verdict: "passed", JudgedAt: &sent}}
done.State = walkDone
done = w.ctl.put(done)
body := func(again bool) []byte {
raw, err := json.Marshal(done)
if err != nil {
t.Fatal(err)
}
var said map[string]any
_ = json.Unmarshal(raw, &said)
said["walk"], said["phase"], said["outcome"] = plan, "ended", "delivered"
said["touches"] = []map[string]any{{"machine": "anchor", "modules": []string{"app"}}}
if again {
said["touches"] = append(said["touches"].([]map[string]any),
map[string]any{"machine": "shanks", "modules": []string{}, "declared": true})
said["declared_read"] = true
}
out, _ := json.Marshal(said)
return out
}
if err := take(w.h, stdio.Envelope{Key: ControllerSeat + ".plan-moved", Body: body(false)}); err != nil {
t.Fatal(err)
}
w.settleAll()
if s := w.state(id); s != Delivered {
t.Fatalf("the walk's end left it %s", s)
}
transitions, comments, delivers := len(w.delivery(id).Transitions), len(w.forge.comments), len(w.ctl.delivers)
if err := take(w.h, stdio.Envelope{Key: ControllerSeat + ".plan-moved", Body: body(true)}); err != nil {
t.Fatal(err)
}
w.settleAll()
if len(w.delivery(id).Transitions) != transitions || len(w.forge.comments) != comments || len(w.ctl.delivers) != delivers {
t.Fatalf("said again it acted again: transitions %d→%d, comments %d→%d, asks of the controller %d→%d",
transitions, len(w.delivery(id).Transitions), comments, len(w.forge.comments), delivers, len(w.ctl.delivers))
}
}
@@ -1,672 +0,0 @@
package main
import (
"fmt"
"sort"
"strings"
"time"
)
// Which merges a walk answers (novox/hq issue 362, ADR 0276).
//
// On 2026-10-10 two merges of the catalogue 18 seconds apart were walked by one walk at the later merge's
// commit, which superseded the first. This owner found the walk a delivery follows by the walk's id or by the
// commit it was merged as, final or not: the surviving walk matched the dunst delivery, which the supersession
// had already ended, so nothing gave it its word, and the claude-code delivery, merged as another commit,
// matched no walk and was held for a person after ten minutes, for a reason that named neither.
//
// A walk answers every merge it carries: the merges it names (`delivery.merges`, said by a controller that
// batches merges), the merge of its own commit, a merge whose walk it took over, and a merge of its repository
// and branch the forge says its commit contains. A delivery follows the newest walk that answers it, and only a
// delivery not yet ended is ever tied to a walk by what it answers: an ended delivery is moved only by the
// record of its own walk, which no longer moves it.
// seenWalk is a walk's last record and when it was heard.
type seenWalk struct {
walk Walk
at time.Time
}
// keptWalkFor is how long a walk no longer open is kept for a merge heard after it ended; a batch is kept as
// long, past its window's latest close.
const keptWalkFor = 10 * time.Minute
// answer is a delivery a walk answers and the commit of its repository the walk carries for it.
type answer struct {
d *Delivery
carried string
}
// walkAnswers is whether a walk answers a delivery's merge, and the commit it carries for it.
func walkAnswers(w Walk, d *Delivery) (string, bool) {
if w.Release != nil || w.IsBatch() || d.MergedAs == "" {
return "", false
}
if c, ok := w.carriedFor(d.Repository, d.MergedAs); ok {
return c, true
}
if d.Walk != nil && d.Walk.TakenOverBy == w.ID && namesNoMerges(w) && strings.EqualFold(w.Repository, d.Repository) &&
sameBranch(w.Branch, d.Base) {
// The walk that took its walk over, from a controller that names no merges: it answers every merge the one
// it took over answered, by its commit of the same repository and branch. A walk that names its merges
// answers only those (carriedFor, above).
return w.Commit, true
}
return "", false
}
// containedAnswer is, for a controller that does not name the merges a walk answers, whether an open walk
// answers a ready merged delivery by containment: of its repository and branch, made for a merge after it, the
// forge says its commit contains this merge, and it moves every module this merge's plan moves. A later walk that
// moves only what its own merge moved does not deliver this one, however much of the trunk it contains.
func (h *Holder) containedAnswer(w Walk, d *Delivery) (string, bool) {
if !h.mayContain(w, d) || !d.ContainedIn[w.Commit] {
return "", false
}
return w.Commit, true
}
// mayContain is whether an open walk is one to ask the forge about for a ready merged delivery (containedAnswer).
func (h *Holder) mayContain(w Walk, d *Delivery) bool {
if w.Release != nil || w.IsBatch() || !w.Open() || !namesNoMerges(w) || d.State != Ready || d.MergedAs == "" || d.Walk != nil ||
d.Plan == nil || len(d.Plan.Moved) == 0 || !strings.EqualFold(w.Repository, d.Repository) ||
!sameBranch(w.Branch, d.Base) || sameCommit(w.Commit, d.MergedAs) {
return false
}
for _, m := range d.Plan.Moved {
if _, moves := w.Modules[m]; !moves {
return false
}
}
// Made for a merge after this one: the walk's own merge, where this owner holds it, or else the walk itself.
after := w.Created
for _, o := range h.deliveries {
if strings.EqualFold(o.Repository, w.Repository) && sameCommit(o.MergedAs, w.Commit) && !o.MergedAt.IsZero() {
after = o.MergedAt
}
}
return after.After(d.MergedAt)
}
// namesNoMerges is whether a walk is from a controller that does not name the merges it answers.
func namesNoMerges(w Walk) bool { return w.Delivery == nil || len(w.Delivery.Merges) == 0 }
func sameBranch(a, b string) bool { return a == "" || b == "" || strings.EqualFold(a, b) }
// follows is whether a delivery takes a walk that answers it as the one it follows: it has none, it is that walk,
// that walk took its own over, or that walk is newer than the one it follows. A walk record kept before its
// creation was read is followed past only once it is superseded or failed.
func follows(d *Delivery, w Walk) bool {
switch {
case d.Walk == nil || d.Walk.ID == w.ID || d.Walk.TakenOverBy == w.ID:
return true
case d.Walk.Created.IsZero():
return d.Walk.State == walkSuperseded || d.Walk.State == walkFailed
}
return w.Created.After(d.Walk.Created)
}
// deliveriesWalking are the deliveries a walk record moves: every delivery that already follows it — ended or
// not, its own record — and every delivery not ended that it answers and that follows it now. In id order.
func (h *Holder) deliveriesWalking(w Walk) []answer {
var out []answer
for _, d := range h.deliveries {
if d.Walk != nil && d.Walk.ID == w.ID {
carried := d.AnsweredBy
if c, ok := walkAnswers(w, d); ok {
carried = c
}
out = append(out, answer{d: d, carried: carried})
continue
}
if d.State.Final() || !follows(d, w) {
continue
}
if c, ok := walkAnswers(w, d); ok {
out = append(out, answer{d: d, carried: c})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].d.ID < out[j].d.ID })
return out
}
// accounted is whether any delivery, ended or not, follows the walk or is answered by it: a walk known for
// what it answers is never made a delivery of its own.
func (h *Holder) accounted(w Walk) bool {
for _, d := range h.deliveries {
if d.Walk != nil && d.Walk.ID == w.ID {
return true
}
if _, ok := walkAnswers(w, d); ok {
return true
}
}
return false
}
// held is whether a live delivery follows the walk, or a live delivery it answers follows a newer one. A walk whose
// word only stops that lapsed give (lapsedOn, novox/hq issue 753) is not held: it stays under the orphans' timer, so
// past the grace without its word it is listed by `stalled`, saying why it was not given.
func (h *Holder) heldWalk(w Walk) bool {
for _, d := range h.deliveries {
if d.State.Final() {
continue
}
if d.Walk != nil && d.Walk.ID == w.ID {
return true
}
if _, ok := walkAnswers(w, d); ok {
return true
}
}
return false
}
// pickWalk is the newest walk heard that answers a merged delivery with no walk yet; nil for none.
func (h *Holder) pickWalk(d *Delivery) (Walk, string, bool) {
var best Walk
var carried string
found := false
for _, s := range h.walks {
if !follows(d, s.walk) {
continue
}
c, ok := walkAnswers(s.walk, d)
if !ok {
c, ok = h.containedAnswer(s.walk, d)
}
if !ok {
continue
}
if !found || s.walk.Created.After(best.Created) || s.walk.Created.Equal(best.Created) && s.walk.ID > best.ID {
best, carried, found = s.walk, c, true
}
}
return best, carried, found
}
// linkAnswering ties a merged delivery with no walk to the newest walk heard that answers it; under the lock.
func (h *Holder) linkAnswering(d *Delivery) bool {
if d.State.Final() || d.MergedAs == "" || d.Walk != nil {
return false
}
w, carried, ok := h.pickWalk(d)
if !ok {
return false
}
h.linkWalk(d, w, carried)
delete(h.unmatched, w.ID)
return true
}
// containsAsks are the forge's asks whether an open walk of a merged delivery's repository and branch, at
// another commit, contains its merge: for a controller that does not name the merges a walk answers (issue 362),
// asked once per pair, the answer kept on the delivery.
func (h *Holder) containsAsks(d *Delivery, now time.Time) []func() {
var asks []func()
for _, s := range h.walks {
w := s.walk
if !h.mayContain(w, d) {
continue
}
if _, known := d.ContainedIn[w.Commit]; known {
continue
}
if a := h.askContains(d, w.Commit, now); a != nil {
asks = append(asks, a)
}
}
return asks
}
// batchOf is the batch a merged delivery's merge is in, when the controller said one; false for none.
func (h *Holder) batchOf(d *Delivery) (Walk, bool) {
if d.MergedAs == "" {
return Walk{}, false
}
for _, s := range h.walks {
if !s.walk.IsBatch() || s.walk.Delivery == nil {
continue
}
for _, m := range s.walk.Delivery.Merges {
if strings.EqualFold(m.Repository, d.Repository) && sameCommit(m.Commit, d.MergedAs) {
return s.walk, true
}
}
}
return Walk{}, false
}
// batchSaid is what a batch holding a delivery's merge waits for, in words.
func batchSaid(b Walk, now time.Time) string {
said := "its batch " + b.ID + ": " + b.State
bt := b.Delivery.Batch
switch {
case b.State == walkQueued && bt != nil && bt.Behind != "":
said += " behind " + bt.Behind
case b.State == walkAssembling && bt != nil && bt.ClosesAt != nil:
left := bt.ClosesAt.Sub(now).Round(time.Second)
if left < 0 {
left = 0
}
said += fmt.Sprintf(", %s left", left)
if bt.AtMost != nil {
said += " (at the latest " + bt.AtMost.Local().Format("15:04:05") + ")"
}
}
return said + "; plan not yet calculated"
}
// unwalkedWhy is why a merged delivery no walk answers is held: the open walks of its repository and that none
// answers its merge, or that the controller holds none — what happened, not what might have.
func (h *Holder) unwalkedWhy(d *Delivery) string {
var open []string
for _, s := range h.walks {
w := s.walk
if w.Release == nil && !w.IsBatch() && w.Open() && strings.EqualFold(w.Repository, d.Repository) {
open = append(open, fmt.Sprintf("%s at %s", w.ID, shortOf(w.Commit, 8)))
}
}
sort.Strings(open)
head := fmt.Sprintf("merged as %s, and no walk answers its merge within %s: ", shortOf(d.MergedAs, 8), unwalked)
if len(open) == 0 {
return head + "the controller holds no open walk of " + d.Repository + " — its merge was not heard, or " +
"nothing the mesh holds follows " + orNone(d.Base) + ": a person decides"
}
return head + "the open walk(s) of " + d.Repository + " — " + strings.Join(open, ", ") + " — name no merge of " +
"it and the forge did not say they contain it: a person decides"
}
func orNone(s string) string {
if s == "" {
return "its branch"
}
return s
}
// orphanLines are the waiting walks no live delivery holds, past the grace a merge is given to be heard, as
// stalled lines in the state Unheld: nothing would ever give such a walk its word (issue 362). Under the lock.
func (h *Holder) orphanLines(now time.Time) []StalledLine {
var out []StalledLine
for id, first := range h.orphans {
s, known := h.walks[id]
if !known || now.Sub(first) < matchGrace {
continue
}
w := s.walk
if len(h.lapsedOn(w)) > 0 {
// Owed its word by stops that lapsed, and not given it past the grace (review of issue 753): said, why.
says := fmt.Sprintf("the walk %s of %s at %s waits for this owner's word, which the stopped deliveries whose "+
"merge it answers owe it (novox/hq issue 753), and it has not started", w.ID, w.Repository, shortOf(w.Commit, 8))
if why := h.lapsedSaid(w); why != "" {
says += ": " + why
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: w.ID,
For: now.Sub(first).Round(time.Second).String(), Bound: matchGrace.String(),
H2: "none: its word is given at its turn — the operator's past the bound", Says: says})
continue
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: w.ID,
For: now.Sub(first).Round(time.Second).String(), Bound: matchGrace.String(),
H2: "none: no delivery holds the walk, so nothing gives it its word — the operator's",
Says: fmt.Sprintf("the walk %s of %s at %s waits for this owner's word, and no delivery that has not ended "+
"follows it or is answered by it: `mesh-delivery.show` for the deliveries of %s, and `plans go %s --why …` "+
"starts it by hand", w.ID, w.Repository, shortOf(w.Commit, 8), w.Repository, w.ID)})
out[len(out)-1].Says += h.endedSaid(w)
if hold := h.coverHeld(id); hold != "" {
out[len(out)-1].Says += "; it is held back meanwhile: " + hold
}
}
return out
}
// Unheld is a waiting walk no live delivery holds: not a state of the table — only what a stalled line says.
const Unheld State = "unheld"
// seeOrphans keeps when each waiting walk no live delivery holds was first seen so, and forgets the rest; a
// walk still unmatched waits for its merge or its adoption instead. Under the lock.
func (h *Holder) seeOrphans(now time.Time) {
for id, s := range h.walks {
w := s.walk
_, unmatched := h.unmatched[id]
if unmatched || w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.heldWalk(w) {
delete(h.orphans, id)
continue
}
if _, was := h.orphans[id]; !was {
h.orphans[id] = now
h.Logf("[mesh-delivery] the walk %s waits for this owner's word and no live delivery holds it: said once "+
"it is %s so", id, matchGrace)
}
}
for id := range h.orphans {
if _, known := h.walks[id]; !known {
delete(h.orphans, id)
}
}
}
// forgetWalks drops a walk record no longer open after keptWalkFor, and a batch heard of that long past its
// window's latest close. Under the lock.
func (h *Holder) forgetWalks(now time.Time) {
for id, s := range h.walks {
w := s.walk
switch {
case w.IsBatch():
end := s.at
if w.Delivery != nil && w.Delivery.Batch != nil && w.Delivery.Batch.AtMost != nil && w.Delivery.Batch.AtMost.After(end) {
end = *w.Delivery.Batch.AtMost
}
if now.Sub(end) > 3*time.Hour {
delete(h.walks, id)
}
case !w.Open() && now.Sub(s.at) > keptWalkFor:
delete(h.walks, id)
delete(h.deliverErr, id)
}
}
}
// --- the word, once per walk ------------------------------------------------------------------------------
// turnWant is a walk whose turn came for some of the deliveries it carries, and why.
type turnWant struct {
ids []string
whys []string
}
// endedSaid names the ended deliveries a waiting walk answers or that follow it, with their states, so a walk no live
// delivery holds says what it carries instead of only that nothing holds it (novox/hq issue 753); empty for none.
func (h *Holder) endedSaid(w Walk) string {
var ended []string
for _, d := range h.deliveries {
if !d.State.Final() {
continue
}
_, answers := walkAnswers(w, d)
if answers || d.Walk != nil && d.Walk.ID == w.ID {
ended = append(ended, fmt.Sprintf("%s (%s)", d.ID, d.State))
}
}
if len(ended) == 0 {
return ""
}
sort.Strings(ended)
return "; every delivery it carries ended: " + strings.Join(ended, ", ")
}
// lapsedOn are the deliveries, by id, whose word an open walk waiting for this owner's word takes from a stop that
// lapsed (novox/hq issue 753): each was stopped by its group for a member delivered since, so its own change waits for
// a later walk of its repository that contains its merge (issue 747), and this walk answers that merge — it names it,
// or is made at it — and has not left it out. Such a walk is the one that delivery waits for: it gives it its word, so
// a walk whose every other delivery ended is never left without one. A walk at the merge of a delivery stopped for
// itself, or by a person, gets nothing from it. Under the lock.
//
// No member is ordered before another (novox/hq ADR 0418 §2): each such delivery gives its word at once.
func (h *Holder) lapsedOn(w Walk) []string {
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits {
return nil
}
var out []string
for _, d := range h.deliveries {
if h.lapsedAnswered(w, d) {
out = append(out, d.ID)
}
}
sort.Strings(out)
return out
}
// lapsedAnswered is whether a delivery's stop lapsed and the walk answers its merge, not left out.
func (h *Holder) lapsedAnswered(w Walk, d *Delivery) bool {
if d.MergedAs == "" || !h.stopLapsed(d) {
return false
}
if _, ok := walkAnswers(w, d); !ok {
return false
}
_, left := w.leftOut(d.Repository, d.MergedAs)
return !left
}
// lapsedSaid is why a walk whose word stops that lapsed give still waits: the members of their groups not yet
// delivered before them, what the last look held it for, or the controller's refusal of the word. Under the lock.
func (h *Holder) lapsedSaid(w Walk) string {
var parts []string
if why := h.waitsWhy[w.ID]; why != "" {
parts = append(parts, "not let go yet: "+why)
}
if err := h.deliverErr[w.ID]; err != "" {
parts = append(parts, "the controller refused its word: "+err)
}
for _, id := range h.lapsedOn(w) {
if d := h.deliveries[id]; d != nil && strings.Contains(d.LetGoWhy, lapsedWord(w.ID)) {
parts = append(parts, fmt.Sprintf("its word was given for %s at %s, and the controller has not started it",
id, d.LetGo.Local().Format("15:04:05")))
}
}
return strings.Join(parts, "; ")
}
// lapsedOwed is whether a walk's word is owed by stops that lapsed.
func (h *Holder) lapsedOwed(w Walk) bool {
return len(h.lapsedOn(w)) > 0
}
// lapsedTurns registers, for every open walk waiting for this owner's word, the turn of each delivery whose stop
// lapsed and whose merge it answers (lapsedOn, novox/hq issue 753). The word is then given as for any walk: once,
// when every live delivery it carries is at its turn too and it carries no stopped change still held back (words).
// Given once per walk: the walk's id is in the delivery's LetGoWhy once its word was given. Under the lock.
func (h *Holder) lapsedTurns() {
for id, s := range h.walks {
for _, did := range h.lapsedOn(s.walk) {
d := h.deliveries[did]
if strings.Contains(d.LetGoWhy, lapsedWord(id)) {
continue
}
if h.turning == nil {
h.turning = map[string]*turnWant{}
}
want := h.turning[id]
if want == nil {
want = &turnWant{}
h.turning[id] = want
}
if !containsString(want.ids, d.ID) {
want.ids = append(want.ids, d.ID)
want.whys = append(want.whys, fmt.Sprintf("%s: %s its merge %s, stopped by its group for %s, whose change "+
"was delivered since; it waits for a later walk of %s that contains its merge, and this is that "+
"walk (novox/hq issue 753)", d.ID, lapsedWord(id), shortOf(d.MergedAs, 8), d.stoppedFor(), d.Repository))
}
}
}
}
// lapsedWord is how the word a lapsed stop gives a walk names that walk.
func lapsedWord(walk string) string { return "the walk " + walk + " answers" }
// words are the asks of the word of every walk whose turn came for every live delivery it carries: one ask per
// walk, however many deliveries it carries, and none while one of them is not at its turn (ADR 0276 decision 5).
// Under the lock.
func (h *Holder) words() []func() {
walks := make([]string, 0, len(h.turning))
for walk := range h.turning {
walks = append(walks, walk)
}
sort.Strings(walks)
var asks []func()
h.waitsWhy = map[string]string{}
for _, walk := range walks {
want := h.turning[walk]
var waiting []string
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != walk || containsString(want.ids, d.ID) {
continue
}
waiting = append(waiting, fmt.Sprintf("%s (%s)", d.ID, d.State))
}
if len(waiting) > 0 {
sort.Strings(waiting)
h.waitsWhy[walk] = strings.Join(waiting, ", ") + " carried by it not at its turn"
h.Logf("[mesh-delivery] the walk %s is not let go yet: %s", walk, h.waitsWhy[walk])
continue
}
ids, why := append([]string(nil), want.ids...), strings.Join(want.whys, "; ")
// **Never the word for a walk that carries a stopped change** (ADR 0299, review of #221): it is asked to
// leave the merge out first (leaveStopped).
if stopped, _ := h.carriesStopped(walk); len(stopped) > 0 {
h.waitsWhy[walk] = "it carries " + strings.Join(stopped, ", ") + ", whose delivery was stopped: it is " +
"asked to leave that merge out first"
if hold := h.coverHeld(walk); hold != "" {
h.waitsWhy[walk] = hold
}
h.Logf("[mesh-delivery] the walk %s is not let go: it carries %s, whose delivery was stopped", walk,
strings.Join(stopped, ", "))
continue
}
if h.asking[walk] {
continue // its word is being asked
}
h.asking[walk] = true
walk := walk
asks = append(asks, func() {
err := h.Controller.Deliver(walk, why)
h.mu.Lock()
defer h.mu.Unlock()
delete(h.asking, walk)
if err != nil {
if h.deliverErr == nil {
h.deliverErr = map[string]string{}
}
h.deliverErr[walk] = err.Error() // said by `stalled` past the grace (review of novox/hq issue 753)
h.Logf("[mesh-delivery] the walk %s could not be let go yet: %v", walk, err)
h.again("a walk's word the controller refused or did not answer")
return
}
delete(h.deliverErr, walk)
for _, id := range ids {
d := h.deliveries[id]
if d == nil {
continue
}
d.LetGo, d.LetGoWhy = h.Now(), why
if d.Walk != nil && d.Walk.ID == walk {
d.Walk.Waits, d.Walk.GoBy = false, byOwner
}
h.settleAndKeep(d)
}
})
}
h.turning = nil
return asks
}
// endedMovingNothing is whether a delivery ended delivered with nothing for a walk to move: merged, its plan
// moving no module the mesh holds, and no walk followed.
func endedMovingNothing(d *Delivery) bool {
return d.State == Delivered && d.MergedAs != "" && d.Walk == nil && d.MovesNothing()
}
// endedAsNothingToMove are the deliveries a walk answers that ended delivered as nothing for a walk to move (novox/hq
// issues 362, 529: the nzbgeek case); empty when it answers any other. Such a walk is held by nothing: said, never
// let go by this owner. Under the lock.
func (h *Holder) endedAsNothingToMove(w Walk) []string {
var out []string
for _, d := range h.deliveries {
_, answers := walkAnswers(w, d)
if !answers && (d.Walk == nil || d.Walk.ID != w.ID) {
continue
}
if !endedMovingNothing(d) {
return nil
}
out = append(out, d.ID)
}
sort.Strings(out)
return out
}
// WalksWaiting is `stalled` asked with `walks` (novox/hq issue 529): for each walk waiting for this owner's word,
// what holds it, in the shape of a stalled line naming the walk — each held delivery it carries (State held, its
// reason and the verb that releases it), else why its word was not given at the last look (State published), else
// that no live delivery holds it (State unheld). A walk whose word is being given is not listed. The controller's
// `plans` says these instead of pointing at a hand start; the self-check never raises a condition for them.
func (h *Holder) WalksWaiting() []StalledLine {
h.mu.Lock()
defer h.mu.Unlock()
now := h.Now()
var out []StalledLine
for id, s := range h.walks {
w := s.walk
if w.Release != nil || w.IsBatch() || !w.Open() || !w.Seen().Waits || h.asking[id] {
continue
}
waited := now.Sub(w.Created).Round(time.Second).String()
var held []StalledLine
live := false
for _, d := range h.deliveries {
if d.State.Final() || d.Walk == nil || d.Walk.ID != id {
continue
}
live = true
if d.State == Held {
held = append(held, StalledLine{ID: d.ID, Number: d.Number, State: Held, Walk: id, For: waited,
Bound: "0s", H2: "none: held for a person — the operator's", Says: HeldLine(d)})
}
}
switch {
case len(held) > 0:
out = append(out, held...)
case h.coverHeld(id) != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: coverAskBound.String(), H2: "none: it is asked to leave the merge out past the bound",
Says: "held back: " + h.coverHeld(id)})
case h.waitsWhy[id] != "":
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: "not let go yet: " + h.waitsWhy[id]})
case !live && h.lapsedOwed(w):
// Its word comes from the stopped deliveries whose merge it answers (novox/hq issue 753): said, never
// unheld, while the controller has not taken it, with what holds it.
lapsed := h.lapsedOn(w)
says := "its word is owed by the stopped deliveries whose merge it answers, each stopped for a member " +
"delivered since (novox/hq issue 753)"
if len(lapsed) > 0 {
says = "its word is given for " + strings.Join(lapsed, ", ") + ", stopped for a member delivered since, " +
"whose merge it answers (novox/hq issue 753)"
}
if why := h.lapsedSaid(w); why != "" {
says += "; " + why
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Published, Walk: id, For: waited,
Bound: "0s", H2: "none: it goes at its turn", Says: says})
case !live:
says := "no delivery that has not ended follows it or is answered by it, so nothing gives it its word: " +
"`mesh-delivery.show` for the deliveries of " + w.Repository
if ended := h.endedAsNothingToMove(w); len(ended) > 0 {
// The nzbgeek case (novox/hq issues 362, 529): its merges' deliveries ended as nothing for a walk to
// move, yet the walk moves modules — nothing judged those, so nothing here lets it go.
says = "held by nothing: its merges' deliveries ended as nothing-to-move (" + strings.Join(ended, ", ") +
"), yet the walk moves " + strings.Join(walkModules(w), ", ") + " — nothing gives it its word"
}
out = append(out, StalledLine{ID: IDOf(w.Repository, w.Commit), State: Unheld, Walk: id, For: waited,
Bound: "0s", H2: "none: no delivery holds the walk — the operator's", Says: says + h.endedSaid(w)})
}
}
sort.Slice(out, func(i, j int) bool {
if out[i].Walk != out[j].Walk {
return out[i].Walk < out[j].Walk
}
return out[i].ID < out[j].ID
})
return out
}
// walkModules are the modules a walk's record names, sorted.
func walkModules(w Walk) []string {
out := make([]string, 0, len(w.Modules))
for m := range w.Modules {
out = append(out, m)
}
sort.Strings(out)
return out
}
@@ -140,6 +140,14 @@ func repoName(repository string) string {
func pullSaid(d *Delivery) string { return fmt.Sprintf("%s#%d", repoName(d.Repository), d.Number) } func pullSaid(d *Delivery) string { return fmt.Sprintf("%s#%d", repoName(d.Repository), d.Number) }
// touchesSaid is the modules the delivery's check says its change touches, for the ask; the gate plans no delivery.
func touchesSaid(d *Delivery) string {
if d.Check == nil || len(d.Check.Modules) == 0 {
return "no module the check named"
}
return strings.Join(d.Check.Modules, ", ")
}
// composeMergeAsk is the ask for a delivery, or for a group's members together. // composeMergeAsk is the ask for a delivery, or for a group's members together.
func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery, heads map[string]string, now time.Time) asks.Ask { func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery, heads map[string]string, now time.Time) asks.Ask {
var a asks.Ask var a asks.Ask
@@ -155,13 +163,10 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
unreviewed = pullSaid(m) + " is " + words unreviewed = pullSaid(m) + " is " + words
} }
reviewers = append(reviewers, fmt.Sprintf("%s: %s", pullSaid(m), words)) reviewers = append(reviewers, fmt.Sprintf("%s: %s", pullSaid(m), words))
plan := "no delivery plan yet" touches := touchesSaid(m)
if m.Plan != nil && m.Plan.Summary != "" { reaches = append(reaches, touches)
plan = m.Plan.Summary details = append(details, fmt.Sprintf("%s %q at %s: issues: %s; review: %s; touches: %s",
} pullSaid(m), m.Title, shortOf(m.Commit, 12), issuesSaid([]*Delivery{m}), words, touches))
reaches = append(reaches, plan)
details = append(details, fmt.Sprintf("%s %q at %s: issues: %s; review: %s; reaches: %s",
pullSaid(m), m.Title, shortOf(m.Commit, 12), issuesSaid([]*Delivery{m}), words, plan))
} }
if group == "" { if group == "" {
a.Headline = phone("Merge into main: "+pullSaid(d)+" "+d.Title, asks.HeadlineLength) a.Headline = phone("Merge into main: "+pullSaid(d)+" "+d.Title, asks.HeadlineLength)
@@ -174,7 +179,7 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
phone(summary, asks.SummaryLength), phone(summary, asks.SummaryLength),
phone("Issues: "+issuesSaid(members), asks.PointLength), phone("Issues: "+issuesSaid(members), asks.PointLength),
phone("Review: "+reviewers[0], asks.PointLength), phone("Review: "+reviewers[0], asks.PointLength),
phone("Reaches: "+reaches[0], asks.PointLength), phone("Touches: "+reaches[0], asks.PointLength),
}, "\n") }, "\n")
} else { } else {
var names []string var names []string
@@ -191,7 +196,7 @@ func composeMergeAsk(id string, d *Delivery, group string, members []*Delivery,
phone(summary, asks.SummaryLength), phone(summary, asks.SummaryLength),
phone("Issues: "+issuesSaid(members), asks.PointLength), phone("Issues: "+issuesSaid(members), asks.PointLength),
phone("Review: "+strings.Join(reviewers, "; "), asks.PointLength), phone("Review: "+strings.Join(reviewers, "; "), asks.PointLength),
phone("Reaches: "+strings.Join(reaches, "; "), asks.PointLength), phone("Touches: "+strings.Join(reaches, "; "), asks.PointLength),
}, "\n") }, "\n")
} }
a.Details = phone(strings.Join(details, "\n"), 1500) a.Details = phone(strings.Join(details, "\n"), 1500)
@@ -43,7 +43,7 @@ func TestAMarkedGreenDeliveryIntoMainIsAskedAndNotMergedUntilApproved(t *testing
len(a.Options) != 2 || a.Options[0].ID != OptionApprove || a.Options[1].ID != OptionDecline { len(a.Options) != 2 || a.Options[0].ID != OptionApprove || a.Options[1].ID != OptionDecline {
t.Fatalf("the ask is not the merge ask: %+v", a) t.Fatalf("the ask is not the merge ask: %+v", a)
} }
for _, want := range []string{"issue 948", "issue 12", "reviewed by reviewer on shanks", "builds app"} { for _, want := range []string{"issue 948", "issue 12", "reviewed by reviewer on shanks", "Touches: "} {
if !strings.Contains(a.Explanation, want) { if !strings.Contains(a.Explanation, want) {
t.Fatalf("the ask does not say %q: %s", want, a.Explanation) t.Fatalf("the ask does not say %q: %s", want, a.Explanation)
} }
@@ -72,7 +72,6 @@ func TestAMarkedGreenDeliveryIntoMainIsAskedAndNotMergedUntilApproved(t *testing
t.Fatalf("the warrant is not kept: %+v", d.Ready) t.Fatalf("the warrant is not kept: %+v", d.Ready)
} }
w.h.PullMerged(merged("novox/app", 7, readyHead, "m7m7m7m7m7m7")) w.h.PullMerged(merged("novox/app", 7, readyHead, "m7m7m7m7m7m7"))
w.h.WalkMoved(w.ctl.put(aWalk("plan-7", "novox/app", "m7m7m7m7m7m7", true)))
w.settleAll() w.settleAll()
var last Transition var last Transition
for _, tr := range w.delivery(id).Transitions { for _, tr := range w.delivery(id).Transitions {
@@ -373,7 +372,7 @@ func TestIssue1025AnOwedMergedIsSaidAfterARestartAndOnce(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
w.h.mu.Unlock() w.h.mu.Unlock()
again := newWorldOver(t, w.store, w.ctl, w.forge, w.now.Add(time.Minute)) again := newWorldOver(t, w.store, w.forge, w.now.Add(time.Minute))
again.settleAll() again.settleAll()
again.later(time.Hour) again.later(time.Hour)
again.settleAll() again.settleAll()
@@ -1,154 +0,0 @@
package main
import (
"strings"
"testing"
)
// A failed backup before the send holds the delivery, and a release asks the backups again (novox/hq ADR 0404
// clause 4, issue 1068). The walk's backups field is the controller's contract.
func failedBackup() *WalkBackups {
return &WalkBackups{State: "failed", Asked: []WalkBackupAsked{
{Machine: "ace", Module: "mail", State: "done"},
{Machine: "ace", Module: "pg", State: "failed", Words: "no space left on device"}}}
}
// aDeliveringWalk is a delivery whose walk was let go and runs, nothing sent yet.
func aDeliveringWalk(t *testing.T) (*world, string, string) {
t.Helper()
w := newWorld(t)
id := IDOf("novox/app", head)
w.h.PullUpdated(pr("novox/app", 7, head, "feat/x", "src/a.go"))
w.h.Checked(verdict("novox/app", 7, head, "pass"))
w.h.PullMerged(merged("novox/app", 7, head, merge))
w.h.WalkMoved(w.ctl.put(aWalk("plan-1", "novox/app", merge, true)))
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("setup: %s, want delivering", s)
}
return w, id, "plan-1"
}
func TestABackupFailedTakesADeliveringDeliveryToHeldOnlyWhileNothingWasSent(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
sent := w.ctl.walk(plan)
sent.Backups = failedBackup()
sent.Modules["app"].SentAt = &w.now
w.h.WalkMoved(w.ctl.put(sent))
w.settleAll()
if s := w.state(id); s == Held {
t.Fatal("held for a failed backup after a module of the plan was sent")
}
w, id, plan = aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
d := w.delivery(id)
if d.State != Held {
t.Fatalf("with a failed backup and nothing sent it is %s", d.State)
}
for _, want := range []string{"ace", "pg", "no space left on device"} {
if !strings.Contains(d.HeldWhy, want) {
t.Errorf("the held reason %q does not name %q", d.HeldWhy, want)
}
}
// It does not lift by itself: the same facts read again leave it held.
w.h.WalkMoved(w.ctl.put(w.ctl.walk(plan)))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("read again it is %s, want it still held", s)
}
}
func TestAReleaseOfADeliveryHeldForAFailedBackupAsksTheControllerAgainAndGoesToDelivering(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("setup: %s, want held", s)
}
lettingGo := len(w.ctl.delivers)
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
if got := w.ctl.backupsAgain; len(got) != 1 || got[0] != plan {
t.Fatalf("the controller was asked to take the backups again %v, want once for %s", got, plan)
}
if len(w.ctl.delivers) != lettingGo {
t.Errorf("a started walk was let go again: %v", w.ctl.delivers)
}
if s := w.state(id); s != Delivering {
t.Fatalf("released, it is %s", s)
}
// The record the controller cleared is read: it stays delivering.
w.h.WalkMoved(w.ctl.walk(plan))
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("with the backups cleared it is %s", s)
}
// Another failure, in a later record, holds it again.
again := w.ctl.walk(plan)
again.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(again))
w.settleAll()
if s := w.state(id); s != Held {
t.Fatalf("a failed backup in a later record left it %s", s)
}
}
func TestARecordThatStillHoldsTheFailureAfterAReleaseDoesNotHoldItAgain(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
stale := w.ctl.put(held)
w.h.WalkMoved(stale)
w.settleAll()
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
w.h.WalkMoved(stale) // the record as it was before the controller cleared it
w.settleAll()
if s := w.state(id); s != Delivering {
t.Fatalf("a record from before the release held it again: %s", s)
}
}
func TestADeliveryHeldForAFailedBackupIsNotLetGoByItsWalkAlone(t *testing.T) {
// The controller's own path starts a walk with nobody's word: the held-to-delivering row for that must not
// undo the backup hold.
d := &Delivery{ID: "novox/app@c0", State: Held, MergedAs: "aaaa", Walk: &WalkSeen{ID: "plan-1", State: walkBuilding, Revision: 3,
BackupsFailed: []WalkBackupFailure{{Machine: "ace", Module: "pg", Words: "no space left on device"}}}}
if _, err := Apply(d, EvGo, false, Facts{}); err == nil {
t.Fatal("a walk started on the controller's own path lifted the backup hold")
}
}
func TestAReleaseWhoseBackupsTheControllerRefusedToAskAgainLeavesTheDeliveryHeld(t *testing.T) {
w, id, plan := aDeliveringWalk(t)
held := w.ctl.walk(plan)
held.Backups = failedBackup()
w.h.WalkMoved(w.ctl.put(held))
w.settleAll()
w.ctl.refuseBackupsAgain = true
_, err := w.h.Release(id, "freed some space", "jochen")
if err == nil || !strings.Contains(err.Error(), "held again") {
t.Fatalf("the release of a refused word answered %v, want an error saying it is held again", err)
}
w.settleAll()
d := w.delivery(id)
if d.State != Held || d.BackupsAgainAfter != 0 || !strings.Contains(d.HeldWhy, "no space left on device") {
t.Fatalf("after the refusal it is %s (again-after %d, why %q), want held for the backup", d.State, d.BackupsAgainAfter, d.HeldWhy)
}
// The controller takes the word the next time: the delivery goes on.
w.ctl.refuseBackupsAgain = false
if _, err := w.h.Release(id, "freed some space", "jochen"); err != nil {
t.Fatal(err)
}
if s := w.state(id); s != Delivering {
t.Fatalf("released again, it is %s", s)
}
}

Some files were not shown because too many files have changed in this diff Show More